I'm using Apache HttpComponents to create an http connection with a website. I have made some methods to get website content using post/get, to send cookies, recieve them and store them in a class I created called CookieManager. Everything works fine, but when I try to get a page content using the GET method the program keeps running but it doesn't do anything.
public HttpResponse sendRequestGet(String url, List<NameValuePair> headers) throws IOException{
HttpGet get = new HttpGet(url);
for (NameValuePair header : headers){
get.setHeader(header.getName(), header.getValue());
}
HttpResponse response = client.execute(get);
System.out.println("----------- STATUS CODE -------------");
System.out.println(response.getStatusLine().getStatusCode() + ": " + url);
System.out.println("-------------------------------------");
return response;
}
The code is the one shown above. The string url contains the url which I want to access, lets say http://mylink.com/market and the headers parameters is made like this:
List<NameValuePair> headerList = new ArrayList<NameValuePair>();
headerList.add((NameValuePair) new BasicNameValuePair("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"));
headerList.add((NameValuePair) new BasicNameValuePair("Accept-Language", "en-US;q=1,en;q=0.8"));
headerList.add((NameValuePair) new BasicNameValuePair("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36"));
headerList.add((NameValuePair) new BasicNameValuePair("Cookie", getCookies()));
headerList.add((NameValuePair) new BasicNameValuePair("Referer", "http://mylink.com/profile/"));
headerList.add((NameValuePair) new BasicNameValuePair("Upgrade-Insecure-Requests", "1"));
If I call the function with the getCookies() returning an empty string, it works , but the problem is that I have to send my session Id which is in the cookies. I tried debugging and I found that since the line HttpResponse response = client.execute(get); it doesn't do anything, the program is still executing but it gets stuck in that line. Also, I should mention that I can get other pages sending the needed cookies but http://mylink.com/market/ gives me this problem.
I have already used chrome 'network' tab to see the interaction between the browser and the server, the only thing is that I don't include some headers (like Host).
Does someone know what I'm doing wrong?
Thanks
I was able to fix it by adding this line inside the function, before the client.execute(...) call : client = HttpClientBuilder.create().build();
Related
i am trying to get list of stops(stations) for a train by passing train number and other required parameters(got from web developer tools-firefox) with the url(POST method), but i get 404-page not found error code. when i tried with POSTMAN, it gets the webpage with the requested data, what is wrong with the code?
Document doc= Jsoup.connect("https://enquiry.indianrail.gov.in/mntes/q?")
.data("opt","TrainRunning")
.data("subOpt","FindStationList")
.data("trainNo",trainNumber)
.data("jStation","")
.data("jDate","25-Aug-2021")
.data("jDateDay","Wed")
.userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0")
.referrer("https://enquiry.indianrail.gov.in/mntes/")
.ignoreHttpErrors(true)
.post();
System.out.println(doc.text());
thank you in advance
I've tried to make the request work with Jsoup but to no avail. An odd way of sending form data is being used. Form data is passed as URL query parameters in a POST request.
Jsoup uses a simplified HTTP API in which this particular use case was not foreseen. It is debatable whether it is appropriate to send form parameters the way https://enquiry.indianrail.gov.in/mntes expects them to be sent.
If you're using Java 11 or later, you could simply fetch the response of your POST request via the modern Java HTTP Client. It fully supports the HTTP protocol. You can then feed the returned String into Jsoup.
Here's what you could do:
// 1. Get the response
HttpClient client = HttpClient.newHttpClient();
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create("https://enquiry.indianrail.gov.in/mntes/q?opt=TrainRunning&subOpt=ShowRunC&trainNo=07482&jDate=25-Aug-2021&jDateDay=Wed&jStation=DVD%23false"))
.POST(BodyPublishers.noBody())
.build();
HttpResponse<String> response =
client.send(request, BodyHandlers.ofString());
// 2. Parse the response via Jsoup
Document doc = Jsoup.parse(response.body());
System.out.println(doc.text());
I've simply copy-pasted the proper URL from Postman. You might want to build your query string in a more robust way. See:
Java URL encoding of query string parameters
How to convert map to url query string?
I have some code, which is meant to send a GET request via HTTP to a server, and fetch the data there. I haven't yet coded the part that does stuff with the response, as I first wanted to test whether the GET request worked. And it didn't:
private static String fetch() throws UnsupportedEncodingException, MalformedURLException, IOException {
// Set the parameters
String url = "http://www.futhead.com";
String charset = "UTF-8";
//Fire the request
try {
URLConnection connection = new URL(url).openConnection();
connection.setRequestProperty("Accept-Charset", charset);
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
// ^^^ I tried this, and it doesn't help!
InputStream response = connection.getInputStream();
HttpURLConnection httpConnection = (HttpURLConnection) new URL(url).openConnection();
httpConnection.setRequestMethod("GET");
System.out.println("Status: " + httpConnection.getResponseCode());
} catch (UnknownHostException e) {
// stuff
}
return null;
// ^^^ I haven't coded the fetching itself yet
}
With that code in mind, fetch() prints Status: 403. Why is this happening? My guess is that this particular server doesn't let non-browser clients access it (because the code works with http://www.google.com), but is there a workaround?
There are some answers out there already, but some of them are either irrelevant to me (they talk about a problem with HTTPS) or incomprehensible. I've tried those that I can understand, to no avail.
You might have Browser Integrity Check enabled https://support.cloudflare.com/hc/en-us/articles/200170086-What-does-the-Browser-Integrity-Check-do-
I disabled Browser Integrity Check and it works fine now. Another solution would be to set User-Agent, if possible.
I experienced the problem from Scala, which eventually uses java.net.URL
I am trying to send data using POST method from my android apps. However in
the server it is always recognized as GET. I am using Rails apps as the web
service. Here is the snippet of my Android code:
URI uri = new URI(hostName);
HttpPost httpRequest = new HttpPost(uri);
httpRequest.addHeader("Accept", "application/json");
httpRequest.addHeader("Content-Type", "application/json");
List<NameValuePair> pairs = new ArrayList<NameValuePair>();
pairs.add(new BasicNameValuePair("key1", "value1"));
httpRequest.setEntity(new UrlEncodedFormEntity(pairs));
HttpClient httpClient = new DefaultHttpClient();
HttpResponse httpResponse = httpClient.execute(httpRequest);
Have I done anything wrong? Thanks for your help.
You're android code looks fine, make sure your log doesn't show a 301 redirect code for POST despite showing a 200 code for GET. Strangely, this can be the case depending on your host configuration.
e.g. You might see something like this :
123.156.189.123 - - [21/Oct/2011:09:03:34 -0700] "POST /server_script.php HTTP/1.1" 301 532 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1"
123.156.189.123 - - [21/Oct/2011:09:03:34 -0700] "GET /server_script.php HTTP/1.1" 200 250 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1"
Here the GET was not redirected (code 200), but the POST was (code 301). If this is happening then you need to override your redirect settings using a .htaccess or other configuration options.
Were you being redirected? I suspect that if you try to POST to domain A that redirects you to domain B, your request will be turned in to a GET request. I had the same problem until I decided to use the server's IP address in the POST request directly, instead of using a alphabet name that redirects to the IP.
hey people, I am stuck trying to integrate java and paypal together.
I have posted here:
http://forum.springsource.org/showthread.php?p=316498#post316498
any help would be greatly appreciated, please advise.
Update:
well my dilemma is that if I was to use a paypal button as mentioned, when do I actually "save" the form information to my local database? In other words I need to save the "user" information etc. only AFTER payment confirmation, it would be useless to do it before.
My understanding is that the "return" URL only will be triggered if there is a successful transaction. If that is the case, then there must be a way to identify this incoming response from paypal so that it was associated with the form post for "this" user. In other words for "this" session. In that way, the user data can then be saved. I know there is something called IPN? Could that be incorpated in the same form initially so we get back a confirmation? I am just trying to figure out how to be in the same session so I can persist the user information. AND if the user's credit card is rejected, does it still go back to the RETURN URL regardless? Need to distinguish somehow.
Lastly, I was playing around with the HTTPPOST code, basically I am tring to simulate a browser here.
public void testPost15() throws ClientProtocolException, IOException {
HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost("https://www.paypal.com/cgi-bin/webscr");
post.setHeader("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3");
//conn.setRequestProperty( "User-Agent", "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.2; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0");
//httpPost.setHeader( "Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
//post.setHeader("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
//httpPost.setHeader( "Accept-Language", "en-us,en;q=0.5");
//post.setHeader("Accept-Language","en-gb,en;q=0.5");
List<NameValuePair> params = new ArrayList<NameValuePair>();
//params.add(new BasicNameValuePair("cmd", "_s-xclick"));
params.add(new BasicNameValuePair("cmd", "_xclick"));
//params.add(new BasicNameValuePair("cmd", "_ext-enter"));
//params.add(new BasicNameValuePair("redirect_cmd", "_xclick-subscriptions"));
params.add(new BasicNameValuePair("business", "abc#logixplayer.com"));
params.add(new BasicNameValuePair("currency_code", "USD"));
params.add(new BasicNameValuePair("amount", "4"));
params.add(new BasicNameValuePair("item_name", "PiT words"));
params.add(new BasicNameValuePair("no_note", "1"));
params.add(new BasicNameValuePair("return", "http://localhost:8080/pit-web-0.0.1-SNAPSHOT/welcome"));
post.setEntity(new UrlEncodedFormEntity(params));
HttpResponse response = client.execute(post);
InputStream is = response.getEntity().getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
StringBuilder str = new StringBuilder();
String line = null;
while((line = reader.readLine()) != null){
str.append(line + "\n");
}
is.close();
String responseText = str.toString();
System.out.println("response: "+responseText);
}
and I think I am onto something, it says: You have requested an outdated version of PayPal. This error often results from the use of bookmarks.
It seems to me it is possible to post to paypal via java...its just that I think I might be missing something. This error is a common error.
Please advise. Thank you.
You cannot use an URLConnection nor an apache HttpPost because these will initiate a connection between your web server and paypal, not between the client's browser and paypal.
Since paypal requires a POST, not a GET, you cannot use a redirect either, so the only option that's left, is returning an HTML page to the client, with a form with all the paypal parameters as hidden <input>s , and a bit of javascript to submit the form immediately.
I am writing an android application which uses a REST-based API on the server. So far the login works perfectly using HttpGet = I send the credentials, it sends me back a JSON response object containing session id or failure. I then moved onto using another get api (this one is passed the sessionid) and the response I get back looks like a valid one "200 - Ok" but the response body contains nothing - 0 text.
If I take the same URL and drop it into a browser, I get all the JSON text I expect displayed in the browser window. So what is the difference between a browser request/response and that of HttpGet? Any clues as to why my HttpGet might return a 'valid' nothing?
I had the same problem. Setting user agent solved my problem:
HttpParams params = new BasicHttpParams();
...
params.setParameter(CoreProtocolPNames.USER_AGENT, "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.71");
Thats my pull() I have written
mHttpGet.setURI(url.toURI());
mResponse = mHttpClient.execute(mHttpGet);
mResponse.getEntity().getContent(); // returns inputstream
How did you do yours?!
It turned out to be a server-side issue. They were actually sending me empty strings when the requester was not a browser. Too bad I can't delete a question. :(