I am using HttpClient (version 3.1) on several different (but apparently identical) computers to read a UTF-8 encoded JSON data from a URL.
On all the machines, save one, it works fine. I have some Spanish language words and they come through with accents and tildes intact.
One computer stubbornly refuses to cooperate. It is apparently treating the data as ISO-8859-1, despite a Content-Type: application/json;charset=utf-8 header.
If I use curl to access that URL from that computer, it works correctly. On every other computer, both curl and my HttpClient-based program work correctly.
I did an md5sum on the common-httpclient.jar file on each machine: the same.
Is there some setting, deep in Linux, that might be different and be messing with me? Any other theories, or even places to look?
EDIT: some people asked for more details.
Originally I had the problem deep in the bowels of a complex Tomcat app, but I lightly adapted the sample to just retrieve the URL in question, and (fortunately) had the same problem.
These are Linux 2.6 machines running jdk1.7.0_45.
An env command yields a bunch of variables. The only one that looks remotely on point is LANG=en_US.UTF-8.
How do you get the json response data from HttpClient?
If you get it back in binary form (through getResponseBodyAsStream() for example), and then convert it to a String without specifying charset, then the result depends on your JVM's default charset.
You can check the value of JVM default charset by:
Charset.defaultCharset().name()
This might give "UTF-8" on all machines except the one failing.
Without seeing your code, it is difficult to say what's wrong, but here is a "correct" way of doing this (using HttpClient 3.1.0 for request and Jackson 2.1.3 to parse the JSON).
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.http.HttpStatus;
import java.io.IOException;
import java.io.InputStreamReader;
HttpClient hc = new HttpClient();
GetMethod get = new GetMethod(uri);
int status = hc.executeMethod(get);
if (status != HttpStatus.SC_OK) throw new RuntimeException("http status " + status);
ObjectMapper jsonParser = new ObjectMapper(new JsonFactory());
// we use an InputStreamReader with explicit charset to read the response body
JsonNode json = jsonParser.readTree(
new InputStreamReader(get.getResponseBodyAsStream(), get.getResponseCharSet())
);
I already faced this issue and this was because of the encoding type configured in the client. So I had to make a "work around" like the one below:
String encmsg = new String(respStr.getBytes("ISO-8859-1"), java.nio.charset.Charset.forName("UTF-8"));
It reads the String as ISO-8859-1 and convert to UTF-8.
Related
I was trying to call an api like below. It works in my IDE(Intellij), where as it fails in hackerrank IDE while getting input stream.
import java.io.*;
import java.util.*;
import java.net.*;
URL url = new URL("https://jsonmock.hackerrank.com/api/article_users?page=1");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("GET");
connection.addRequestProperty("Content-Type", "application/json");
InputStream is = connection.getInputStream();
Getting an error:
error while calling: java.net.UnknownHostException: jsonmock.hackerrank.com
inside hacker rank IDE.
Please let me know your suggestions
just remove s from https
URL url = new URL("http://jsonmock.hackerrank.com/api/article_users?page=1");
Are you sure Hackerrank IDE has api call feature?
Perhaps for security or some other reason, this feature may be blocked or not supported.
It may be the case that Hackerrank does not let Java code make outbound network requests. Similar questions here and here suggest that you may need to choose a different language that Hackerrank supports. I imagine they should list it somewhere on their site, or on the IDE page directly.
Above is my code i have written to hit a web service whose endpoint is expecting a Byte Stream object.
I am able to do that but i am not getting any response.
I have to test the response.
Though i am getting 200 ok but a string is sent in response that i am not getting.
And the response is blank
How can I get the response ?
In order to read server's response you need to use URLConnection.getInputStream() method, not OutputStream
In order to convert stream to string you can use IOUtils.toString() method
In order to return data you can use return keyword
Minimal working code is below, adjust as per your needs:
import java.net.HttpURLConnection;
import java.net.URL;
import java.nio.charset.StandardCharsets;
import org.apache.commons.io.IOUtils;
URL url = new URL("http://example.com");
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("GET");
String response = IOUtils.toString(con.getInputStream(), StandardCharsets.UTF_8);
return response;
Be aware of fact that JMeter is built on top of Apache HttpComponents so you can use the power of these libraries in order to create HTTP requests, see QuickStart wiki page to get ramped up in minutes
Be aware that starting from JMeter 3.1 it is recommended to use JSR223 Test Elements and Groovy language for scripting, check out Apache Groovy - Why and How You Should Use It for comprehensive explanation, benchmarks, code samples, etc.
You can add your output by using the SampleResult object:
String output = "...";
SampleResult.setResponseData( output );
SampleResult.setDataType( org.apache.jmeter.samplers.SampleResult.TEXT );
I am stuck in a strange issue, I am posting some image data to a server, now I created the requested using curl and then traced back it.
Next was to create similar request in java.
The code posts raw binary image data to server, but when I compare the binary data in java to that posted by curl, there is a minute difference due to which I am getting 400 response code from server.
The difference I think is in few dots.
Below is the request generate by curl (linux).
Generate by curl
Now here is the request generate by Java, when I read bytes.
Click here
Java code looks something like this:
PrintWriter out = new PrintWriter(os);
out.println("POST /1izb0rx1 HTTP/1.1");
out.println("User-Agent: curl/7.35.0");
out.println("Host: requestb.in");
out.println("Accept: */*");
out.println("Content-Disposition:inline; filename=test.png");
out.println("Authorization: Basic YW5kcm9pZDpUZXN0dGVzdDExISE=");
out.println("Content-Length: "+"24143");
out.println("Content-Type: application/x-www-form-urlencoded");
out.println();
out.println("."+imgBytes);
Any idea what can be causing this issue ?
Thanks
So,
I got it working, the problem was that certain classes on Android are broken and not behaving as the way they behave on core Java.
The same code that was working on Java, wasn't working here, reason being, a change in header occurring here (On Android).
This issue is very well mentioned here also:
https://github.com/WP-API/WP-API/issues/1862
Thus I was facing a similar issue, and adding external updated jars were conflicting with the ones on Android.
Finally I used a small HTTP Request library: https://github.com/kevinsawicki/http-request
The code is below:
HttpRequest request = HttpRequest.post(url);
request.authorization("Basic "+ah);
request.part("file", fName+".png", "image/png", new File(file));
request.part("title", "test");
if(request.code()==201) {
StringWriter sw = new StringWriter();
request.receive(sw);
onMedia(Media.parse(new JsonParser().parse(sw.toString()).getAsJsonObject()));
}
Thanks
Do not use PrintWriter to send raw bytes.
I'm having bad times when posting JSON via AJAX.
For testing purposes, the code below reproduces the problem, and was created in the Firefox Scratchpad (I'm using Backbone in the application layer):
var xhr = new XMLHttpRequest();
xhr.open("POST", "/my/api/url"); // could be PUT too
xhr.setRequestHeader("Content-Type", "application/json");
xhr.send(JSON.stringify({ test: "é" }));
Actually, no matter what accented string I use in my JSON, it's always sent incorrectly (I've tested in Firefox and Chrome, both latest versions) - I can see the request data wrong in Chrome DevTools/Firebug. What is sent in this case is é.
I have found an workaround in Java. I would not bother using it once or twice, however, seems like this is not going to be the case. There are still lots of stuff to do.
The workaround is the following:
test = new String(test.getBytes("ISO-8859-1"), "UTF-8");
Any further help would be appreciated.
My setup (if anyone needs):
Windows 7 x64
JDK 1.7 x64
JBoss 4.2.3
RESTEasy
MyEclipse 9.1
I'll be often updating this question with interesting data whenever they appear.
You are not passing the json properly
xhr.send(JSON.stringify({ test: "é" }));
also there should be qoutes for key like this { "test": "e" }
you can use variable
var jsonStr={ "test": "é" };
xhr.send(JSON.stringify(jsonStr));
I don't think you need to stringify json since you have set Content-Type to json, not sure about it though.
Hope it helped you...
Try sending the parameters this way
var params='json='+jsonStr;
xhr.send(params);
I've resolved my own problem. As I am using RESTEasy, one of my methods were using an interceptor MessageBodyReader. In this class, I was reading the body without using the request's encoding...
Was:
String body = new String(IOUtils.toByteArray(inputStream));
Become:
String body = new String(IOUtils.toByteArray(inputStream), request.getCharacterEncoding());
I am calling a restful service that returns JSON using the Apache HttpClient.
The problem is I am getting different results in the encoding of the response when I run the code on different platforms.
Here is my code:
GetMethod get = new GetMethod("http://urltomyrestservice");
get.addRequestHeader("Content-Type", "text/html; charset=UTF-8");
...
HttpResponse response = httpexecutor.execute(request, conn, context);
response.setParams(params);
httpexecutor.postProcess(response, httpproc, context);
StringWriter writer = new StringWriter();
IOUtils.copy(response.getEntity().getContent(), writer);
When I run this on OSX, asian characters etc return fine e.g. 張惠妹 in the response. But when I run this on a linux server the same code displays the characters as ???
The linux server is an Amazon EC2 instance running Java 1.6.0_26-b03
My local OSX is running 1.6.0_29-b11
Any ideas really appreciated!!!!!
If you look at the javadoc of org.apache.commons.io.IOUtils.copy(InputStream, Writer):
Copy bytes from an InputStream to chars on a Writer using the default
character encoding of the platform.
So that will give different answers depending on the client (which is what you're seeing)
Also, Content-Type is usually a response header (unless you're using POST or PUT). The server is likely to ignore it (though you might have more luck with the Accept-Charset request header).
You need to parse the content type's charset-encoding parameter of the response header, and use that to convert the response into a String (if it's a String you're actually after). I expect Commons HTTP has code that will do that automatically for you. If it doesn't, Spring's RESTTemplate definitely does.
I believe that the problem is not in the HTTP encoding but elsewhere (e.g. while reading or forming the answer). Where do you get the content from and how? Is this stored in a DB or file?