I am trying to make a post request to a https address and set up fiddler to return a standard response. I have two rules set up in Fiddler and the process works from both Internet Explorer and Postman (but not Chrome) and I cannot get it to work from the java application I am trying to write even when I have created an executeable jar file and run from the cmd. I have been using this example as the base for this work. I have the sendGet() working (ish) but I cannot get sendPost() to work getting a java.net.UnknownHostException.
I think the problem may be that I am not hitting Fiddler as the proxy from Eclipse. For the sendGet() from browser and Postman I get the contents of 200_SimpleHTML.dat as required but from eclipse the same rule has no affect and I get the content from the actual URL (Our TeamForge in this case)
My organisation uses a proxy which is set in IE and I have set the java configuration to "Use browser settings" and also tried "Use automatic proxy configuration script" (pointing to the proxy.pac file) and neither seems to have any affect. I have the following in Window -> Preferences -> Network Connections:
but I have no idea how, or even if, I can point to Fiddler as the proxy here. I am not setting up any authentication from the working routes.
The current state of my sendPost is below:
USER_AGENT = "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
which I copied from Fiddler after one of the successful request.
private void sendPost() throws Exception
{
String url = "<Actual URL removed>";
URL obj = new URL(url);
HttpURLConnection http = (HttpURLConnection) obj.openConnection();
// add request header
http.setRequestMethod("POST");
http.setDoOutput(true);
http.setRequestProperty("User-Agent", USER_AGENT);
http.setRequestProperty("Accept-Language", "en-GB,en;q=0.5");
OutputStream out = http.getOutputStream();
int responseCode = http.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(http.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null)
{
response.append(inputLine);
}
in.close();
// print result
System.out.println(response.toString());
}
Does anyone have any ideas as to how I can get this to work from my Java app?
Related
I'm validating links by trying to hit them and getting the response codes(in Java). But I get invalid response codes(403 or 404) from code but from browser, I get 200 status code when I inspect the network activity. Here's my code that gets the response code. [I do basic validations on urls beforehand, like making it lowercase, etc.]
static int getResponseCode(String link) throws IOException {
URL url = new URL(link);
HttpURLConnection http = (HttpURLConnection) url.openConnection();
return http.getResponseCode();
}
For link like http://science.sciencemag.org/content/220/4599/868, I am getting 403 status when I run this code. But on browser(chrome), I am getting 200 status. Also, if I use the below curl command, I am getting 200 status code.
curl -Is http://science.sciencemag.org/content/220/4599/868
The only way to overcome that is to:
check what are the HTTP headers sent by your program (for instance, by sending queries to http://scooterlabs.com/echo and check the response)
check what are the HTTP headers sent by your browser (for instance, by visiting https://www.whatismybrowser.com/detect/what-http-headers-is-my-browser-sending )
spot the differences
change your program to send the same headers as your browser (the ones that work)
I made this analysis for you, and it turns out this website requires an Accept header that resemble the Accept headers of an existing browser. By default Java sends something valid, but not resembling that.
You just need to change your program as so:
static int getResponseCode(String link) throws IOException {
URL url = new URL(link);
HttpURLConnection http = (HttpURLConnection) url.openConnection();
http.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
return http.getResponseCode();
}
(Or any other value that an actual browser uses)
i'm trying to read the best price from the skyscanner website using a normal get request, but i'm not getting the content that i want by using this code.
private void getRequest() throws Exception {
StringBuilder result = new StringBuilder();
URL url = new URL(URL);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.addRequestProperty("User-Agent","Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0");
System.out.println(conn.getURL());
conn.setInstanceFollowRedirects(true);
HttpURLConnection.setFollowRedirects(true);
conn.setRequestMethod("GET");
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line;
while ((line = rd.readLine()) != null) {
result.append(line);
}
System.out.println(conn.getURL());
rd.close();
response = result.toString();
}
The requested URL is the following:
https://www.skyscanner.com/transport/flights/fra/txl/181220/?adults=1&children=0&adultsv2=1&childrenv2=&infants=0&cabinclass=economy&rtn=0&preferdirects=false&outboundaltsenabled=false&inboundaltsenabled=false¤cy=EUR&market=DE&locale=en-US
Response from the code above looks like this:
https://pastebin.com/YKh17RKE
By going to the mentioned skyscanner link in chrome i can click on inspect element and voila under
fqs-opts-container -> <span class="fqs-price">42 €</span>
i can see the cheapest price.
How to get this information using java? What am i doing wrong here?
Thanks in advance.
Inspect shows the current HTML DOM (Document Object Model) resulting from:
the static HTML page (see right-click + View page source) plus
dynamic modifications by JavaScript.
If you do Inspect, tab Network and reload the page, you can see which files (and their contents) are all requested by the browser to display the page.
In this particular case, it seems that you could get the data as JSON:
In the tab Network filter for conductor/v1/fps3/search/. The query is an HTTP post request with the URL https://www.skyscanner.de/g/conductor/v1/fps3/search/?geo_schema=skyscanner&carrier_schema=skyscanner&response_include=query%3Bdeeplink%3Bsegment%3Bstats%3Bfqs%3Bpqs%3B_flights_availability. The answer is in JSON and includes a session_id which is required as part of the URL for subsequent requests for details.
Please note that even if it is technically possible to receive the data, it is in most cases forbidden to use them commercially.
I'm working on a pet project to scrape fantasy football stats from MY own fantasy league on ESPN. The problem that I'm running into that I can't seem to get past is the login which is needed before I can make requests for my league's page.
The URL I hit is
http://games.espn.com/ffl/leaguesetup/ownerinfo?leagueId=123456&seasonId=2016
and by looking at the GET requests it looks like I get redirected to
http://games.espn.com/ffl/signin?redir=http://games.espn.com/ffl/leaguesetup/ownerinfo?leagueId=123456&seasonId=2016
Which immediately gets me to a login prompt window. When I log in I inspect the POST request and note down all the Request Header. Looks like the requested URL on the POST is
https://registerdisney.go.com/jgc/v5/client/ESPN-FANTASYLM-PROD/guest/login?langPref=en-US
additionally I noted the following JSON objected is passed along:
{"loginValue":"myusername","password":"mypassword"}
using the Request Headers and JSON object I did the following:
String url = "http://games.espn.com/ffl/leaguesetup/ownerinfo?leagueId=123456&seasonId=2016";
String rawData = "{\"loginValue\":\"myusername\",\"password\":\"mypassword\"}";
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
con.setRequestProperty("Accept-Encoding", "gzip, deflate");
con.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
con.setRequestProperty("Authorization", "APIKEY 8IYGqTgmpFTX51iF1ldp6MBtWrdQ0BxNUf8bg5/empOdV4u16KUSrnkJqy1DXy+QxV8RaxKq45o2sM8Omos/DlHYhQ==");
con.setRequestProperty("Cache-Control", "no-cache");
con.setRequestProperty("Content-Length", "52");
con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
con.setRequestProperty("Expires", "-1");
con.setRequestProperty("Host", "registerdisney.go.com");
con.setRequestProperty("Origin", "https://cdn.registerdisney.go.com");
con.setRequestProperty("Pragma", "no-cache");
con.setRequestProperty("Referer", "https://cdn.registerdisney.go.com/v2/ESPN-ESPNCOM-PROD/en-US?include=config,l10n,js,html&scheme=http&postMessageOrigin=http%3A%2F%2Fwww.espn.com%2F&cookieDomain=www.espn.com&config=PROD&logLevel=INFO&topHost=www.espn.com&ageBand=ADULT&countryCode=US&cssOverride=https%3A%2F%2Fsecure.espncdn.com%2Fcombiner%2Fc%3Fcss%3Ddisneyid%2Fcore.css&responderPage=https%3A%2F%2Fwww.espn.com%2Flogin%2Fresponder%2Findex.html&buildId=157599bfa88");
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0");
con.setRequestProperty("conversation-id", "5a4572f4-c940-454c-8f86-9af27345c894, adffddd3-8c31-41a0-84d7-7a0401cd2ad0");
con.setRequestProperty("correlation-id", "4d9ddc78-b00e-4c5a-8eec-87622961fd34")
con.setDoOutput(true);`
OutputStreamWriter w = new OutputStreamWriter(con.getOutputStream(), "UTF-8");
w.write(rawData);
w.close();
int responseCode = con.getResponseCode();
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
Assuming I'm on the right track what I'm currently getting back from the server is server is
returned HTTP response code: 400 for URL: https://registerdisney.go.com/jgc/v5/client/ESPN-FANTASYLM-PROD/guest/login?langPref=en-US
Any ideas what is happening or if i'm taking the complete wrong approach here? I tried to use JSoup but had no luck either and I believe underneath JSoup uses HttpUrlConnection as well.
Do I need to do some sort of GET request first, save something then do the POST request? How should it work?
You are trying to emulate the behaviour of a Web Browser with JSoup. As you have experienced this is quite complicated and JSoup is not made for to impersonate a browser. When you start with crafting HTTP headers, then it's better to go another way.
The solution for your problem is to use a browser that can be programmatically manipulated. Selenium is more or less the defacto standard in Java.
Selenium starts your favorite browser (Firefox, Chrome, ..) and let you control it from your Java program. You can also retrieve the content of the web pages in order to scrap them with JSoup. Selenium is well documented, you will have no difficulty to find the required documentation/tutorial.
Another answer to your problem. While it is impossible for me to reproduce your issue (don't have football fantasy account and I have no intent to create one), I can still try to give some methodology help.
I would tackle the problem by using the network inspector from my browser, copy in a file all the exchanges between the browser and the server and try to reproduce this in my code.
The API key value in the Authorization header can only be reused for a limited time. If it is expired, the registration response body will contain an "API_KEY_INVALID" error.
I have some code, which is meant to send a GET request via HTTP to a server, and fetch the data there. I haven't yet coded the part that does stuff with the response, as I first wanted to test whether the GET request worked. And it didn't:
private static String fetch() throws UnsupportedEncodingException, MalformedURLException, IOException {
// Set the parameters
String url = "http://www.futhead.com";
String charset = "UTF-8";
//Fire the request
try {
URLConnection connection = new URL(url).openConnection();
connection.setRequestProperty("Accept-Charset", charset);
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
// ^^^ I tried this, and it doesn't help!
InputStream response = connection.getInputStream();
HttpURLConnection httpConnection = (HttpURLConnection) new URL(url).openConnection();
httpConnection.setRequestMethod("GET");
System.out.println("Status: " + httpConnection.getResponseCode());
} catch (UnknownHostException e) {
// stuff
}
return null;
// ^^^ I haven't coded the fetching itself yet
}
With that code in mind, fetch() prints Status: 403. Why is this happening? My guess is that this particular server doesn't let non-browser clients access it (because the code works with http://www.google.com), but is there a workaround?
There are some answers out there already, but some of them are either irrelevant to me (they talk about a problem with HTTPS) or incomprehensible. I've tried those that I can understand, to no avail.
You might have Browser Integrity Check enabled https://support.cloudflare.com/hc/en-us/articles/200170086-What-does-the-Browser-Integrity-Check-do-
I disabled Browser Integrity Check and it works fine now. Another solution would be to set User-Agent, if possible.
I experienced the problem from Scala, which eventually uses java.net.URL
I'm trying to get an xml file from a distant server.
The server asks a username and password.
My code works fine on Android 4.x but on Android 2.x i get an Error 400.
400 Bad Request
Your browser sent a request that this server could not understand.
here is my code :
URL url = new URL("http://username:password#xx.xx.xx.xx:80/pathToXML/file.xml");
URLConnection uc = url.openConnection();
String val = (new StringBuffer("username").append(":").append("password")).toString();
byte[] base = val.getBytes();
String authorizationString = "Basic " + new String(Base64.encode(base, Base64.DEFAULT));
uc.setRequestProperty("Authorization", authorizationString);
uc.connect();
InputStream ins = uc.getInputStream(); // it fails on this line
I tried many things (adding Content-Length, SetRequestMethod to POST, set user-Agent) but it doesn't work.
If someone has any idea i'm open to it.
Thanks a lot for your help.
Cheers