In Java, is it possible to view multiple response headers on a HttpURLConnection if they have the same name?
In the Oracle documentation for "GetHeaderField", it states:
If called on a connection that sets the same header multiple times
with possibly different values, only the last value is returned.
My question is, how do I view all the different values for a header that is set multiple times?
Use getHeaderFields
List<String> values = conn.getHeaderFields().get("X-Header-Of-Interest");
Complete example
import java.io.IOException;
import java.net.URL;
import java.net.URLConnection;
public class UrlConnectionTest {
public static void main (String[] args) throws IOException {
URL url = new URL("http://localhost:8888/");
URLConnection conn = url.openConnection();
conn.getContent(); // Force request
System.out.println(conn.getHeaderFields().get("X-Funky-Header"));
}
}
On Linux you can create a simple single-request server with netcat for testing
$ echo -e 'HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nX-Funky-Header: value1\r\nX-Funky-Header: value2\r\n\r\nContent' | nc -l 8888 &
Related
So I'm trying to read a JSON file in from a website (fortniteapi.com), every time I try to download the file to my local computer it does not download. I've been at this for about a week and I just can't figure out why it won't work.
also i'm using Gson
Here is my code so far:
package sample;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
import javafx.application.Application;
import javafx.fxml.FXMLLoader;
import javafx.scene.Parent;
import javafx.scene.Scene;
import javafx.stage.Stage;
import java.io.*;
import java.net.URL;
import java.net.URLConnection;
public class Main extends Application {
#Override
public void start(Stage primaryStage) throws Exception{
Parent root = FXMLLoader.load(getClass().getResource("sample.fxml"));
primaryStage.setTitle("Fortnite");
primaryStage.setScene(new Scene(root, 300, 275));
primaryStage.show();
ReadJson();
}
public static void main(String[] args) {
launch(args);
}
public void ReadJson()
{
try {
// read url
String sURL = "https://fortnite-public-api.theapinetwork.com/prod09/users/id?username=Ninja"; //just a string
// Connect to the URL using java's native library
URL url = new URL(sURL);
URLConnection request = url.openConnection();
request.connect();
// Convert to a JSON object
JsonParser jp = new JsonParser(); //from gson
JsonElement root = jp.parse(new InputStreamReader((InputStream) request.getContent())); //Convert the input stream to a json element
JsonObject rootobj = root.getAsJsonObject();
String output = rootobj.get("username").getAsString(); //just grab the username value
// print out the result/output
System.out.println(output);
} catch (IOException e) {
System.out.println("Unexpected Error.");
// JOptionPane.showMessageDialog(null, "Oh no something went wrong.", "Unexpected Error", JOptionPane.ERROR_MESSAGE);
System.exit(1);
}
}
}
The error
After reading the errorStream() of the request (after casting it to HttpURLConnection) HTML is printed and states:
Access denied | fortnite-public-api.theapinetwork.com used Cloudflare
to restrict access
and
The owner of this website (fortnite-public-api.theapinetwork.com) has
banned your access based on your browser's signature
(mybrowsersignature).
What does this mean
Cloudflare states that that error means that:
the domain owner is blocking this request based on the client's web
browser signature.
and that the feature is called "Browser Integrity Check", from there we can find What does the Browser Integrity Check do?:
Cloudflare's Browser Integrity Check (BIC) is similar to Bad Behavior
and looks for common HTTP headers abused most commonly by spammers and
denies access to your page. It will also challenge visitors that do
not have a user agent or a non standard user agent (also commonly used
by abuse bots, crawlers or visitors).
Solution
We can change the User-Agent of request to something that should be valid before request.connect(); like so (user agent copied from User-Agent | MDN):
request.setRequestProperty("User-Agent",
"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0");
The expected output is printed:
Ninja
For example i block my site for Pakistan.Then how do i automate from Pakistan end,to check whether my site is blocked or not .
We used TOR for this case via silvertunnel lib https://mvnrepository.com/artifact/org.silvertunnel-ng.
import java.io.IOException;
import java.net.Socket;
import net.freehaven.tor.control.TorControlConnection;
public class Main {
public static void main(String[] args) throws IOException {
//Connecting to TOR's control port
//TOR must be running and following line must be added to the torrc:
//ControlPort 9100
Socket s = new Socket("127.0.0.1", 9100);
TorControlConnection conn = TorControlConnection.getConnection(s);
//no authentification at the moment
conn.authenticate(new byte[0]);
//write some changes to torrc (like setting exit node)
//ExitNodes {US}
//StrictExitNodes 1
//send "RELOAD" signal - reloading torrc from disk
conn.signal("RELOAD");
}
}
And these files: https://github.com/guardianproject/jtorctl
It was long time ago so I have no more code but here are a couple of links for reference:
http://www.yasith.me/2017/01/selenium-webdriver-with-tor-network.html
https://tor.stackexchange.com/questions/652/how-to-call-tor-browser-using-selenium-using-java
I am using Java JRE 1.8.0_141 and I am trying to access a specific URL and store the HTML into a String so that I can manipulate the data later in the code, but I keep getting error 405 whenever I call getInputStream().
The code seems to work with other URLs without problems. The trouble URL is:
http://www.streeteasy.com/for-rent/nyc/status:open%7Cprice:1750-2900%7Carea:104,116,119,143,141%7Camenities:pool?page=2&refined_search=true
Here is Eclipse 4.6.3's specific error:
<terminated, exit value: 1>C:\Program Files\Java\jre1.8.0_141\bin\javaw.exe (Aug 6, 2017, 10:53:37 PM)
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Server returned HTTP response code: 405 for URL: http://www.streeteasy.com/for-rent/nyc/status:open%7Cprice:1750-2900%7Carea:104,116,119,143,141%7Camenities:pool?page=2&refined_search=true
at RunMe.getHTMLFromURL(RunMe.java:52)
at RunMe.main(RunMe.java:18)
Caused by: java.io.IOException: Server returned HTTP response code: 405 for URL: http://www.streeteasy.com/for-rent/nyc/status:open%7Cprice:1750-2900%7Carea:104,116,119,143,141%7Camenities:pool?page=2&refined_search=true
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at RunMe.getHTMLFromURL(RunMe.java:36)
... 1 more
My RunMe.java code is below:
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;
import java.util.LinkedList;
public class RunMe {
public static void main(String[] args) throws IOException {
// TODO Auto-generated method stub
System.out.println(getHTMLFromURL("http://www.streeteasy.com/for-rent/nyc/status:open%7Cprice:1750-2900%7Carea:104,116,119,143,141%7Camenities:pool?page=2&refined_search=true"));
}
public static String getHTMLFromURL(String url){
try{
URL urlObj = new URL(url);
URLConnection con = urlObj.openConnection();
con.setDoOutput(false);
con.connect();
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
// CODE FAILS HERE ^
StringBuilder response = new StringBuilder();
String inputLine;
String newLine = System.getProperty("line.separator");
while ((inputLine = in.readLine()) != null){
response.append(inputLine + newLine);
}
in.close();
return response.toString();
}
catch (Exception e){
throw new RuntimeException(e);
}
}
}
Any idea how I can pull the HTML from this URL if not via this method? Thanks in advance!
I executed a curl command against the URL and it seems like the site is trying to run JavaScript for rendering the page.
curl -v -L -H "User-Agent: Mozilla/5.0" -H "Accept: text/html" "http://www.streeteasy.com/for-rent/nyc/status:open%7Cprice:1750-2900%7Carea:104,116,119,143,141%7Camenities:pool?page=2"
> GET /for-rent/nyc/status:open%7Cprice:1750-2900%7Carea:104,116,119,143,141%7Camenities:pool?page=2 HTTP/1.1
> Host: www.streeteasy.com
> User-Agent: Mozilla/5.0
> Accept: text/html
>
< HTTP/1.1 405 Not Allowed
// elided
<h1>Pardon Our Interruption...</h1>
<p>As you were browsing <strong>www.streeteasy.com</strong> something about your browser made us think you were a bot. There are a few reasons this might happen:</p>
<ul>
<li>You're a power user moving through this website with super-human speed.</li>
<li>You've disabled JavaScript in your web browser.</li>
<li>A third-party browser plugin, such as Ghostery or NoScript, is preventing JavaScript from running. Additional information is available in this <a title='Third party browser plugins that block javascript' href='http://ds.tl/help-third-party-plugins' target='_blank'>support article</a>.</li>
</ul>
<p>After completing the CAPTCHA below, you will immediately regain access to www.streeteasy.com.</p>
You may be outta luck here unless you can programmatically fill in the captcha.
Edit:
The problem is clearly cookies as indicated in the discussion below.
I need to set source IP address (for something like IP Spoofing) before sending out an http request. Class used for setting up http connection is HTTPURLConnection. I found below link on stackoverflow which is really useful.
Registering and using a custom java.net.URL protocol
As in the post, I have already created 3 classes extending URLConnection , URLStreamHandler and implementing URLStreamHandlerFactory. This looks to be working fine; however I am getting exception which I think is because I have not implemented getInputStream for URLConnection as was mentioned in above post.
I have couple of questions
1> I am extending custom URLConnection class from HTTPURLConnection, so what's the need of implementing getInputStream as anyway it's a virtual method
2> If I have to do it, can someone provide sample implementation of this method?
JMeter already provides the IP Spoofing feature.
In Http Request Defaults, select (in version 3.0 of JMeter) advanced tab :
See http://jmeter.apache.org/usermanual/component_reference.html#HTTP_Request_parms1:
Source address field
[Only for HTTP Request with HTTPClient implementation]
This property is used to enable IP Spoofing. It overrides the default local IP address for this sample. The JMeter host must have multiple IP addresses (i.e. IP aliases, network interfaces, devices). The value can be a host name, IP address, or a network interface device such as "eth0" or "lo" or "wlan0".
If the property httpclient.localaddress is defined, that is used for all HttpClient requests.
Based on UBIK LOAD PACK's answer. Here is the same code using Aapche http client 3.x. (The variable for setting source IP is args[1])
import org.apache.commons.httpclient.*;
import org.apache.commons.httpclient.methods.*;
import org.apache.commons.httpclient.protocol.Protocol;
public class HC3Test {
public static void main(String[] args) throws Exception {
String url = args[0];
java.net.URL uri = new java.net.URL(url);
HostConfiguration hc = new HostConfiguration();
hc.setHost(uri.getHost(), uri.getPort(), Protocol.getProtocol(uri.getProtocol()));
hc.setLocalAddress(java.net.InetAddress.getByName(args[1]));//for pseudo 'ip spoofing'
HttpClient client = new HttpClient(new SimpleHttpConnectionManager());
client.setHostConfiguration(hc);
GetMethod method = new GetMethod(url);
client.executeMethod(method);
method.releaseConnection();
}
}
Sample code for http client 4.x:
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.conn.params.ConnRoutePNames;
public class HC4Test {
public static void main(String[] args) throws Exception {
DefaultHttpClient httpClient = new DefaultHttpClient();
org.apache.http.params.HttpParams params = httpClient.getParams();
params.setParameter(ConnRoutePNames.LOCAL_ADDRESS,
java.net.InetAddress.getByName(args[1]));//for pseudo 'ip spoofing'
httpClient.execute(new HttpGet(args[0]));
}
}
To accomplish to have a different source address via an http request you could use a local proxy.
import java.io.IOException;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.MalformedURLException;
import java.net.Proxy;
import java.net.URL;
import java.net.URLConnection;
import java.net.UnknownHostException;
public class Huhu {
public static void main(String[] args) throws IOException {
URL url = new URL("http://google.com");
Proxy proxy = new Proxy(Proxy.Type.DIRECT,
new InetSocketAddress(
InetAddress.getByAddress(
new byte[]{your, ip, interface, here}), yourTcpPortHere));
URLConnection conn = url.openConnection(proxy);
}
}
You do not have to override anything this way.
I have collected text of about 2 * 10^5 rows, i have to expand the shortened URLs in the text. the problem is, that some expanded URLs redirect to again a shorten URL, which means that the original URL is 2 or more times shortened. how to handle that efficiently? because if i make a while loop, it is taking too much time.
The problem with the code below, is about 2*10^5 context switches from my MySQL DB. I extract the URL from each text and then expand it. This will make an http request, and then it has to be entered into my DB. it is taking too much time.
import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.Proxy;
import java.net.URL;
public class urlExpander {
public static void main(String[] args) throws IOException {
String shortenedUrl = "https://www.youtube.com/watch?v=4oZTPGvG3s8&feature=youtu.be";
String expandedURL = expandUrl(shortenedUrl);
System.out.println(shortenedUrl + "-->" + expandedURL);
}
public static String expandUrl(String shortenedUrl) throws IOException {
URL url = new URL(shortenedUrl);
// open connection
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection(Proxy.NO_PROXY);
// stop following browser redirect
httpURLConnection.setInstanceFollowRedirects(false);
// extract location header containing the actual destination URL
String expandedURL = httpURLConnection.getHeaderField("Location");
httpURLConnection.disconnect();
return expandedURL;
}
}