We Keep Coding

HttpGet sends request with the wrong encoding

I'm trying to get the text response from the following URL:
http://translate.google.cn/translate_a/single?client=t&sl=zh-CN&tl=en&dt=t&tk=265632.142896&q=%E4%BD%A0%E5%A5%BD
The response is the following:
[[["Hello there","你好",,,1]],,"zh-CN"]
(You can verify this response by entering the address into your browser.)
Here is a simplified version of my code that tries to download this text:
import org.apache.http.client.HttpClient;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.BasicResponseHandler;
import org.apache.http.impl.client.DefaultHttpClient;
public class Test {
public static String downloadString() {
String url = "http://translate.google.cn/translate_a/single?client=t&sl=zh-CN&tl=en&dt=t&tk=265632.142896&q=%E4%BD%A0%E5%A5%BD";
HttpClient client = new DefaultHttpClient();
HttpGet request = new HttpGet(url);
ResponseHandler<String> handler = new BasicResponseHandler();
try {
return client.execute(request, handler);
} catch (Exception e) {
return "GET request failed.";
}
}
}
When I call Test.downloadString(), I get the following (incorrect) response:
[[["Huan Chai Sunsolt","浣犲ソ",,,0]],,"zh-CN"]
I'm guessing that there is some sort of encoding problem behind the scenes somewhere in the request process (there are six bytes that should be interpreted as two Chinese characters, but are instead interpreted as three Japanese characters), but I can't seem to pinpoint the exact cause. What am I doing wrong in my code?

It's strange, but adding the User-Agent header fixed the problem:
request.addHeader("User-Agent", "Mozilla/5.0 (X11; Linux x86_64; rv:33.0) Gecko/20100101 Firefox/33.0");

Android 6.0 release removes support for the Apache HTTP client. If your app is using this client and targets Android 2.3 (API level 9) or higher, use the HttpURLConnection class instead.
here: http://developer.android.com/about/versions/marshmallow/android-6.0-changes.html#behavior-apache-http-client

Apache HttpComponents in java HttpClient.execute(get) not doing anything

I'm using Apache HttpComponents to create an http connection with a website. I have made some methods to get website content using post/get, to send cookies, recieve them and store them in a class I created called CookieManager. Everything works fine, but when I try to get a page content using the GET method the program keeps running but it doesn't do anything.
public HttpResponse sendRequestGet(String url, List<NameValuePair> headers) throws IOException{
HttpGet get = new HttpGet(url);
for (NameValuePair header : headers){
get.setHeader(header.getName(), header.getValue());
}
HttpResponse response = client.execute(get);
System.out.println("----------- STATUS CODE -------------");
System.out.println(response.getStatusLine().getStatusCode() + ": " + url);
System.out.println("-------------------------------------");
return response;
}
The code is the one shown above. The string url contains the url which I want to access, lets say http://mylink.com/market and the headers parameters is made like this:
List<NameValuePair> headerList = new ArrayList<NameValuePair>();
headerList.add((NameValuePair) new BasicNameValuePair("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"));
headerList.add((NameValuePair) new BasicNameValuePair("Accept-Language", "en-US;q=1,en;q=0.8"));
headerList.add((NameValuePair) new BasicNameValuePair("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36"));
headerList.add((NameValuePair) new BasicNameValuePair("Cookie", getCookies()));
headerList.add((NameValuePair) new BasicNameValuePair("Referer", "http://mylink.com/profile/"));
headerList.add((NameValuePair) new BasicNameValuePair("Upgrade-Insecure-Requests", "1"));
If I call the function with the getCookies() returning an empty string, it works , but the problem is that I have to send my session Id which is in the cookies. I tried debugging and I found that since the line HttpResponse response = client.execute(get); it doesn't do anything, the program is still executing but it gets stuck in that line. Also, I should mention that I can get other pages sending the needed cookies but http://mylink.com/market/ gives me this problem.
I have already used chrome 'network' tab to see the interaction between the browser and the server, the only thing is that I don't include some headers (like Host).
Does someone know what I'm doing wrong?
Thanks

I was able to fix it by adding this line inside the function, before the client.execute(...) call : client = HttpClientBuilder.create().build();

java.io.IOException: Server returned HTTP response code: 403 for URL: <url>

I am trying to check if a service is available and always returns the same error:
 
java.io.IOException: Server returned HTTP response code: 403 for URL
Internet browsing proposed that it was necessary to indicate the "USER-AGENT" and so I did, but the error remains the same:
openConnection.addRequestProperty("User-Agent", "Mozilla / 5.0 (Windows NT 6.1; WOW64) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 40.0.2214.91 Safari / 537.36");
The complete code is:
url = cadenaURL + cadenaEndpoint;
URLConnection openConnection = new URL(url).openConnection();
openConnection.connect();
is = openConnection.getInputStream();
if ("gzip".equals(openConnection.getContentEncoding()))
is = new GZIPInputStream(is);
and the error is in:
is = openConnection.getInputStream();
Someone could help me?
Thank You,
a greeting,

I am trying to check if a service is available
Simply put returning 403 means the service is not available. There's your answer.
There's a possibility you're using the wrong url and attempting to connect to the wrong service.
However it seems also that
is = openConnection.getInputStream();
This may not be enough to connect to an HTTP server. You need to properly format the request, you need to show more of the code and how you're using it for us to help you more.

Finally I changed URLConnection by HttpURLConnection and works perfect.
Thank You

Java HTTP GET request gives 403 Forbidden, but works in browser

I have some code, which is meant to send a GET request via HTTP to a server, and fetch the data there. I haven't yet coded the part that does stuff with the response, as I first wanted to test whether the GET request worked. And it didn't:
private static String fetch() throws UnsupportedEncodingException, MalformedURLException, IOException {
// Set the parameters
String url = "http://www.futhead.com";
String charset = "UTF-8";
//Fire the request
try {
URLConnection connection = new URL(url).openConnection();
connection.setRequestProperty("Accept-Charset", charset);
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
// ^^^ I tried this, and it doesn't help!
InputStream response = connection.getInputStream();
HttpURLConnection httpConnection = (HttpURLConnection) new URL(url).openConnection();
httpConnection.setRequestMethod("GET");
System.out.println("Status: " + httpConnection.getResponseCode());
} catch (UnknownHostException e) {
// stuff
}
return null;
// ^^^ I haven't coded the fetching itself yet
}
With that code in mind, fetch() prints Status: 403. Why is this happening? My guess is that this particular server doesn't let non-browser clients access it (because the code works with http://www.google.com), but is there a workaround?
There are some answers out there already, but some of them are either irrelevant to me (they talk about a problem with HTTPS) or incomprehensible. I've tried those that I can understand, to no avail.

You might have Browser Integrity Check enabled https://support.cloudflare.com/hc/en-us/articles/200170086-What-does-the-Browser-Integrity-Check-do-
I disabled Browser Integrity Check and it works fine now. Another solution would be to set User-Agent, if possible.
I experienced the problem from Scala, which eventually uses java.net.URL

Why do I get the proper URL response from a browser, but empty within Android's "HttpGet"?

I am writing an android application which uses a REST-based API on the server. So far the login works perfectly using HttpGet = I send the credentials, it sends me back a JSON response object containing session id or failure. I then moved onto using another get api (this one is passed the sessionid) and the response I get back looks like a valid one "200 - Ok" but the response body contains nothing - 0 text.
If I take the same URL and drop it into a browser, I get all the JSON text I expect displayed in the browser window. So what is the difference between a browser request/response and that of HttpGet? Any clues as to why my HttpGet might return a 'valid' nothing?

I had the same problem. Setting user agent solved my problem:
HttpParams params = new BasicHttpParams();
...
params.setParameter(CoreProtocolPNames.USER_AGENT, "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.71");

Thats my pull() I have written
mHttpGet.setURI(url.toURI());
mResponse = mHttpClient.execute(mHttpGet);
mResponse.getEntity().getContent(); // returns inputstream
How did you do yours?!

It turned out to be a server-side issue. They were actually sending me empty strings when the requester was not a browser. Too bad I can't delete a question. :(