We Keep Coding

How to handle 419 http response in Java

I am trying to parse through a particular website and am getting an HTTP response code of 419 when my java code calls it. I need to parse through the response to find content and I am stuck on the response code.
I have tried putting together a Java program using apache http client(version 4.5.6) to call a website that I need to parse. The http response code I get back is 419.
try (CloseableHttpClient httpclient = HttpClients.createDefault()) {
HttpGet httpGet = new HttpGet("http://www.website.com");
try (CloseableHttpResponse response1 = httpclient.execute(httpGet)) {
System.out.println(response1.getStatusLine());
HttpEntity entity1 = response1.getEntity();
EntityUtils.consume(entity1);
}
}
The result that it prints out is this:
HTTP/1.1 419 status code 419
I am expecting a 200
HTTP/1.1 200 OK
I get that when I change the website to google or other sites.

I was making a get request through HttpClient library as well as from POSTMAN and facing same 419 error. To solve this 419 error we need to add csrf token while making form submission.
However, In-case if you are still wondering how to find csrf token even when you are making a GET request and facing status 419. In my case I solved the problem by adding the user-agent: xxxx token in header.
Example:
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36
HttpClient Code:
connectionManager = new PoolingHttpClientConnectionManager();
...
...
...
httpClient = HttpClients.custom()
.setConnectionManager(connectionManager)
.setRedirectStrategy(new LaxRedirectStrategy())
.setUserAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36")
.build();

Apache HttpComponents in java HttpClient.execute(get) not doing anything

I'm using Apache HttpComponents to create an http connection with a website. I have made some methods to get website content using post/get, to send cookies, recieve them and store them in a class I created called CookieManager. Everything works fine, but when I try to get a page content using the GET method the program keeps running but it doesn't do anything.
public HttpResponse sendRequestGet(String url, List<NameValuePair> headers) throws IOException{
HttpGet get = new HttpGet(url);
for (NameValuePair header : headers){
get.setHeader(header.getName(), header.getValue());
}
HttpResponse response = client.execute(get);
System.out.println("----------- STATUS CODE -------------");
System.out.println(response.getStatusLine().getStatusCode() + ": " + url);
System.out.println("-------------------------------------");
return response;
}
The code is the one shown above. The string url contains the url which I want to access, lets say http://mylink.com/market and the headers parameters is made like this:
List<NameValuePair> headerList = new ArrayList<NameValuePair>();
headerList.add((NameValuePair) new BasicNameValuePair("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"));
headerList.add((NameValuePair) new BasicNameValuePair("Accept-Language", "en-US;q=1,en;q=0.8"));
headerList.add((NameValuePair) new BasicNameValuePair("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36"));
headerList.add((NameValuePair) new BasicNameValuePair("Cookie", getCookies()));
headerList.add((NameValuePair) new BasicNameValuePair("Referer", "http://mylink.com/profile/"));
headerList.add((NameValuePair) new BasicNameValuePair("Upgrade-Insecure-Requests", "1"));
If I call the function with the getCookies() returning an empty string, it works , but the problem is that I have to send my session Id which is in the cookies. I tried debugging and I found that since the line HttpResponse response = client.execute(get); it doesn't do anything, the program is still executing but it gets stuck in that line. Also, I should mention that I can get other pages sending the needed cookies but http://mylink.com/market/ gives me this problem.
I have already used chrome 'network' tab to see the interaction between the browser and the server, the only thing is that I don't include some headers (like Host).
Does someone know what I'm doing wrong?
Thanks

I was able to fix it by adding this line inside the function, before the client.execute(...) call : client = HttpClientBuilder.create().build();

How to crawl mobile website java?

I want to read mobile version of the website but my program reads the normal website.
I am using this property
connection.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt)");
What am I supposed to do?

The decision on which page to serve by the server is made based upon the "User-Agent" property of the request.
To get the mobile version of the page, take a look at this chrome dev article detailing chrome on android user agent strings, and set the "User-Agent" string in your header to be that of a mobile client; it doesn't look like the User-Agent string you have used in your question is that of a mobile client.
For example,
HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost(url);
String userAgent = "Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.133 Mobile Safari/535.19";
try {
httppost.setHeader("User-Agent", userAgent);
// Add your data
// Execute HTTP Post Request
HttpResponse response = httpclient.execute(httppost);
// ....
} catch ... {
This should give you the mobile version of a page, as would be seen by a Galaxy Nexus device.

Here is a list of a ton of mobile browser user agent strings: http://www.useragentstring.com/pages/Mobile%20Browserlist/
Maybe try a different user agent string like:
Opera/9.80 (J2ME/MIDP; Opera Mini/9 (Compatible; MSIE:9.0; iPhone; BlackBerry9700; AppleWebKit/24.746; U; en) Presto/2.5.25 Version/10.54

java.io.IOException: Server returned HTTP response code: 403 for URL: <url>

I am trying to check if a service is available and always returns the same error:
 
java.io.IOException: Server returned HTTP response code: 403 for URL
Internet browsing proposed that it was necessary to indicate the "USER-AGENT" and so I did, but the error remains the same:
openConnection.addRequestProperty("User-Agent", "Mozilla / 5.0 (Windows NT 6.1; WOW64) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 40.0.2214.91 Safari / 537.36");
The complete code is:
url = cadenaURL + cadenaEndpoint;
URLConnection openConnection = new URL(url).openConnection();
openConnection.connect();
is = openConnection.getInputStream();
if ("gzip".equals(openConnection.getContentEncoding()))
is = new GZIPInputStream(is);
and the error is in:
is = openConnection.getInputStream();
Someone could help me?
Thank You,
a greeting,

I am trying to check if a service is available
Simply put returning 403 means the service is not available. There's your answer.
There's a possibility you're using the wrong url and attempting to connect to the wrong service.
However it seems also that
is = openConnection.getInputStream();
This may not be enough to connect to an HTTP server. You need to properly format the request, you need to show more of the code and how you're using it for us to help you more.

Finally I changed URLConnection by HttpURLConnection and works perfect.
Thank You

Jakarta commons HTTPClient can't send a post request on my apache server with php

I have set up a Apache server, with a php script running to handle various post requests. the script looks as followed:
$uploaddir = realpath('mypath');
//realpath('./') . '/';
$uploadfile = $uploaddir . '/' . basename($_FILES['file_contents']['name']);
echo '<pre>';
if (move_uploaded_file($_FILES['file_contents']['tmp_name'], $uploadfile)) {
echo "File is valid, and was successfully uploaded.\n";
} else {
echo "Possible file upload attack!\n";
}
echo 'Here is some more debugging info:';
print_r($_FILES);
echo "\n<hr />\n";
print_r($_POST);
print "</pr" . "e>\n";
and this works great, it handles the request and moves my file to where i want it.
But when i try this with following java client, i dont get anything:
String URL = "http://localhost/accept.php";
File file = new File("C:/Ozer/ANPRProject/MDT/Export/test.txt");
HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost(URL);
try{
System.out.println(""+file.exists());
InputStreamEntity reqEntity = new InputStreamEntity(new FileInputStream(file),-1);
reqEntity.setContentType("application/octet-stream");
reqEntity.setChunked(true);
httppost.setEntity(reqEntity);
System.out.println("execute request " + httppost.getRequestLine());
HttpResponse response = httpclient.execute(httppost);
HttpEntity resEntity = response.getEntity();
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
if (resEntity != null) {
System.out.println("Response content length: " + resEntity.getContentLength());
System.out.println("Chunked: " + resEntity.isChunked());
}
EntityUtils.consume(resEntity);
} catch(Exception e) {
} finally {
httpclient.getConnectionManager().shutdown();
}
}
What I get as system out put is:
true
execute request POST http://localhost/accept.php HTTP/1.1
----------------------------------------
HTTP/1.1 200 OK
Response content length: 330
Chunked: false
Which I find weird, because the chunked has not been set to true? While i specifically asked for it in my code. But anyhow, the uploading does not work. I Have set the contentType to "application/octet-stream" because the php script I used to request a post gave me as feedback: [type] => application/octet-stream
So the logs of the apache show me that the connection was successful and he indeed did get the request:
127.0.0.1 - - [08/Aug/2013:16:10:04 +0200] "POST /accept.php HTTP/1.1" 200 330 "-" ""Apache-HttpClient/4.2.5 (java 1.5)"
this is what i get when i use my Php script to request a post:
127.0.0.1 - - [08/Aug/2013:15:54:23 +0200] "POST /accept.php HTTP/1.1" 200 419 "-" "-" ::1 - - [08/Aug/2013:15:54:23 +0200] "GET /testSendDat.php HTTP/1.1" 200 3330 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36"
and the error log from the php script shows me:
[08-Aug-2013 16:10:04 Europe/Berlin] PHP Notice: Undefined index: file_contents
So I know what that means, the array where i'm trying to read my files from are empty. But why? I have no clue. I'm guessing I'm using the wrong way to post a request but have been looking for a while now and haven't really been successful in finding what the problem could be. Help would be much obliged.
Oh and would it perhaps be easier to set up a apache tomcat server and work with httpservlets? Or would'nt that matter?

You do not create a HTTP-Request that simulate a filled in form with a file upload element in it. You create a HTTP POST Request with the file as the request body.
The body of a filled in form POST HTTP Request looks like this:
user=Fritz&age=12
It looks sumilar like the URL params of a GET HTTP Request.
A filled in form POST with a file upload in it is a multipart HTTP Request with multiple bodies. One of it has the form fields the other the upload file content.
see: how-does-http-file-upload-work
I cannot provide sample to set up your request, but this information may help you.