I'm trying to post some login data to a form in order to grab the cookies from the response.
The url is: https://www.deviantart.com/users/login
However I can not get the server to return FOUND 302 but only 200, so I think I'm b0rking my querystring or headers in some manner:
try {
String query = URLEncoder.encode("&username="+user+"&password="+password+"&remember_me=1", "UTF-8");
URL url = new URL("https://www.deviantart.com/users/login");
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
HttpsURLConnection.setFollowRedirects(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("Host", "www.deviantart.com");
conn.setRequestProperty("User-Agent", "Mozilla 4.0");
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
conn.setRequestProperty("Content-Length", String.valueOf(query.length()));
conn.setRequestProperty("Content-Language", "en-US");
conn.setUseCaches(false);
conn.setDoOutput(true);
conn.setDoInput(true);
PrintWriter out = new PrintWriter(conn.getOutputStream(), false);
out.write(query);
out.flush();
As far as I know, its normal to get response "200", means that your request was recognized and properly answered.
But when I need to know about HTTP code responses and some details I usually use this link: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html.
Hope it helps...
A 302 response is a redirect ... telling the browser to go to another page. A sites login page will often do this to send the user to a "login succeeded" page, or back to the home page or the page that the user was originally looking at. But it doesn't have to.
A 200 response means "Succeeded", and for a POST request will result in the browser staying at the login page.
I'd not worry about getting a 200 instead of a 302. If you are sending requests from a Java app, you probably don't care where the site redirects you after login. The thing that matters is whether your credentials have been accepted, and you can only determine that by trying a request that needs authentication. (You need to make sure that your code captures the cookies set by the response to your login POST. They need to be supplied in follow-on requests.)
If you are worried that you have "borked" the login URL query String, fetch the login page and look at its source to see if there are any hidden inputs in the form.
Related
I'm working on a pet project to scrape fantasy football stats from MY own fantasy league on ESPN. The problem that I'm running into that I can't seem to get past is the login which is needed before I can make requests for my league's page.
The URL I hit is
http://games.espn.com/ffl/leaguesetup/ownerinfo?leagueId=123456&seasonId=2016
and by looking at the GET requests it looks like I get redirected to
http://games.espn.com/ffl/signin?redir=http://games.espn.com/ffl/leaguesetup/ownerinfo?leagueId=123456&seasonId=2016
Which immediately gets me to a login prompt window. When I log in I inspect the POST request and note down all the Request Header. Looks like the requested URL on the POST is
https://registerdisney.go.com/jgc/v5/client/ESPN-FANTASYLM-PROD/guest/login?langPref=en-US
additionally I noted the following JSON objected is passed along:
{"loginValue":"myusername","password":"mypassword"}
using the Request Headers and JSON object I did the following:
String url = "http://games.espn.com/ffl/leaguesetup/ownerinfo?leagueId=123456&seasonId=2016";
String rawData = "{\"loginValue\":\"myusername\",\"password\":\"mypassword\"}";
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
con.setRequestProperty("Accept-Encoding", "gzip, deflate");
con.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
con.setRequestProperty("Authorization", "APIKEY 8IYGqTgmpFTX51iF1ldp6MBtWrdQ0BxNUf8bg5/empOdV4u16KUSrnkJqy1DXy+QxV8RaxKq45o2sM8Omos/DlHYhQ==");
con.setRequestProperty("Cache-Control", "no-cache");
con.setRequestProperty("Content-Length", "52");
con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
con.setRequestProperty("Expires", "-1");
con.setRequestProperty("Host", "registerdisney.go.com");
con.setRequestProperty("Origin", "https://cdn.registerdisney.go.com");
con.setRequestProperty("Pragma", "no-cache");
con.setRequestProperty("Referer", "https://cdn.registerdisney.go.com/v2/ESPN-ESPNCOM-PROD/en-US?include=config,l10n,js,html&scheme=http&postMessageOrigin=http%3A%2F%2Fwww.espn.com%2F&cookieDomain=www.espn.com&config=PROD&logLevel=INFO&topHost=www.espn.com&ageBand=ADULT&countryCode=US&cssOverride=https%3A%2F%2Fsecure.espncdn.com%2Fcombiner%2Fc%3Fcss%3Ddisneyid%2Fcore.css&responderPage=https%3A%2F%2Fwww.espn.com%2Flogin%2Fresponder%2Findex.html&buildId=157599bfa88");
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0");
con.setRequestProperty("conversation-id", "5a4572f4-c940-454c-8f86-9af27345c894, adffddd3-8c31-41a0-84d7-7a0401cd2ad0");
con.setRequestProperty("correlation-id", "4d9ddc78-b00e-4c5a-8eec-87622961fd34")
con.setDoOutput(true);`
OutputStreamWriter w = new OutputStreamWriter(con.getOutputStream(), "UTF-8");
w.write(rawData);
w.close();
int responseCode = con.getResponseCode();
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
Assuming I'm on the right track what I'm currently getting back from the server is server is
returned HTTP response code: 400 for URL: https://registerdisney.go.com/jgc/v5/client/ESPN-FANTASYLM-PROD/guest/login?langPref=en-US
Any ideas what is happening or if i'm taking the complete wrong approach here? I tried to use JSoup but had no luck either and I believe underneath JSoup uses HttpUrlConnection as well.
Do I need to do some sort of GET request first, save something then do the POST request? How should it work?
You are trying to emulate the behaviour of a Web Browser with JSoup. As you have experienced this is quite complicated and JSoup is not made for to impersonate a browser. When you start with crafting HTTP headers, then it's better to go another way.
The solution for your problem is to use a browser that can be programmatically manipulated. Selenium is more or less the defacto standard in Java.
Selenium starts your favorite browser (Firefox, Chrome, ..) and let you control it from your Java program. You can also retrieve the content of the web pages in order to scrap them with JSoup. Selenium is well documented, you will have no difficulty to find the required documentation/tutorial.
Another answer to your problem. While it is impossible for me to reproduce your issue (don't have football fantasy account and I have no intent to create one), I can still try to give some methodology help.
I would tackle the problem by using the network inspector from my browser, copy in a file all the exchanges between the browser and the server and try to reproduce this in my code.
The API key value in the Authorization header can only be reused for a limited time. If it is expired, the registration response body will contain an "API_KEY_INVALID" error.
URL url = new URL("http://myserver.com/myendpoint");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
//connection.setRequestMethod("POST") <- this didn't help either
connection.setDoOutput(true);
OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());
out.write("string=test");
out.close();
connection.close()
The code above WORKS on desktop JVM, sends a post request, parsed on server-side successfully with response 200, however on android, the request method stays GET (yes I checked it IS false) and results in a 404 exception. Official docs say that setting doOutput to true triggers setting the request method to POST but that doesn't seem the case.
404 is not an exception. It is a HTTP status code returned by the server you make the request to and it means that the url you make the request to is not found. That has nothing to do with POST being set or not.
Things to check:
If the url you are making a request to is right.
If the server has a POST controller/handler mapped to the url you are making the request to.
Ask the guy who develops the server if he is handling the cases right ans if he's sending the correct response codes for the relevant scenarios.
Extra info: if the url is registered on the service but a POST request is not allowed you would get a 415 response code.
When posting data to a server, I'm setting some additional request header:
String query = "string=test";
URL url = new URL("http://myserver.com/myendpoint");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("POST");
connection.setRequestProperty("Accept-Charset", "UTF-8");
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded; charset=UTF-8");
connection.setFixedLengthStreamingMode(query.getBytes("UTF-8").length);
OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());
out.write(query);
But as suggested, the 404 exception usually means, that the endpoint, you're trying to access, isn't available.
Try it:
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("GET");
I made an app that basically just posts a form to login then gets data using an API. Now the website it's for changed the way it works so that when you load the login page it redirects you to one with a random string in the query string and a different random string in the form action.
Since I can't just do an HTTPS POST request to the submit page anymore I need a way to load the login page, wait for the redirect and then wait for load again and then use the form action as HTTPS POST URL.
How do I load a page, specifically wait for it to redirect and then be done loading?
What I'm using now is pretty much this:
HttpsURLConnection urlConnection = (HttpsURLConnection) url.openConnection();
urlConnection.setRequestMethod("POST");
urlConnection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
urlConnection.setRequestProperty("User-Agent", USER_AGENT);
urlConnection.setConnectTimeout(5000);
String urlParameters = "ctl00$ContentPlaceHolder1$UsernameTextBox="+ URLEncoder.encode(inumber, "UTF-8") + "&ctl00$ContentPlaceHolder1$PasswordTextBox="+ URLEncoder.encode(password, "UTF-8");
urlConnection.setRequestProperty("Content-Length", "" +
Integer.toString(urlParameters.getBytes().length));
urlConnection.setUseCaches(false);
urlConnection.setDoInput(true);
urlConnection.setDoOutput(true);
urlConnection.setChunkedStreamingMode(0);
DataOutputStream out = new DataOutputStream(urlConnection.getOutputStream());
out.writeBytes(urlParameters);
out.flush();
out.close();
You have a couple of options:
use HTTPURLConnection#setInstanceFollowRedirects(boolean) and ensure that you request the initial page with the same protocol as the redirect (it won't follow otherwise)
use [Google HTTP Client] which supports arbitrary redirects
I stuck in amazing situation while consuming REST operation.
When i run simple java program in IDE then i am able to create HTTP request: 201/Created. SUCCESS. Server receives request
When i put same program in Runtime environment(say Mule), i get 405/Method not Allowed. FAILED. Server does not receive anything
When i run it through Rest Client(some mozilla or chrome plugin), It accepts request 201/created. SUCCESS. Server receives request
On that particular URL, i am getting following Allowed:
Allow: GET, POST
Is this issue with Runtime?
or any settings to be done in Runtime environment?
or issue in Code?
Following is Code:
`URL urlRequest = new URL(url);
HttpURLConnection conn = (HttpURLConnection) urlRequest.openConnection();
conn.setRequestProperty("Host", "10.91.17.170:81");
conn.setRequestProperty("User-Agent", "Mozilla/4.0");
conn.setRequestProperty("Accept", "application/atom+xml;q=0.9,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-us,en;q=0.5");
conn.setRequestProperty("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
conn.setRequestProperty("Keep-Alive", "115");
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty("Access-Control-Allow-Origin", "*");
conn.setRequestProperty("Content-Type", "application/atom+xml");
conn.setDoOutput(true);
conn.setRequestMethod("POST");
DataOutputStream wr = new DataOutputStream(conn.getOutputStream());
wr.writeBytes(requestXML);
wr.flush();
wr.close();
System.out.println("Created Request");
System.out.println("Response Code: "+conn.getResponseCode());
System.out.println("Response Status: "+conn.getResponseMessage());
System.out.println("Response Status: "+conn.getContentType());
System.out.println("Request Method: "+conn.getRequestMethod());
`
Unable to get answer on google. Did hit and trials but not successful.
405 errors often arise with the POST method. You may be trying to introduce some kind of input form on the Web site, but not all ISPs allow the POST method necessary to process the form.
All 405 errors can be traced to configuration of the Web server and security governing access to the content of the Web site, so should easily be explained by your ISP.
for details you can see this:
jQuery .ajax() POST Request throws 405 (Method Not Allowed) on RESTful WCF
Don't know how to explain it better but i'm trying to get a response from an URL containing a function (right?).
I'm working on this for a lot of hours and progressing a little every time but can't get this finally working.
This is the request and response headers from chrome dev tools:
Headers
My code is:
String params = "{\"prefixText\":\"" + city
+ "\",\"count\":10,\"contextKey\":\"he\"}";
conn = (HttpURLConnection) new URL(
"http://bus.gov.il/WebForms/wfrmMain.aspx/GetCompletionList")
.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("POST");
conn.setChunkedStreamingMode(0);
// conn.setFixedLengthStreamingMode(params.length());
conn.addRequestProperty("Accept", "*/*");
conn.addRequestProperty("Content-Type", "application/json; charset=UTF-8");
conn.addRequestProperty("Content-Length", String.valueOf(params.length()));
conn.addRequestProperty("Host", "bus.gov.il");
conn.addRequestProperty("Origin", "http://bus.gov.il");
conn.addRequestProperty("X-Requested-With", "XMLHttpRequest");
conn.addRequestProperty("Referer",
"http://bus.gov.il/WebForms/wfrmMain.aspx?width=1024&company=1&language=he&state=");
OutputStream os = new BufferedOutputStream(conn.getOutputStream());
os.write(params.getBytes());
String answer = readStream(conn.getInputStream());
I get the exception (I see in the stack trace) when calling "getinputstream" on this line:
String answer = readStream(conn.getInputStream());
before entering the readStream function!
I don't know how to solve it...
Tried searching about xmlhttprequest but understood that it's only in JS.
Also: I know I have a lot of unnecessary request properties but I can't figure out which are unnecessary until the code will work.
Thanks in advance :)
Sadly, it used to be (and probably still is) that the HttpURLConnection throws a FileNotFoundException when you get a 404 error. When you are doing the getInputStream() that's when it's first connecting, so any error from the server will show up there.
Get Wireshark or something if you want to see what's really going on in HTTP land as you make the request.