till now I've used the following code snippet in order to send and recieve JSON strings:
static private String sendJson(String json,String url){
HttpClient httpClient = new DefaultHttpClient();
String responseString = "";
try {
HttpPost request = new HttpPost(url);
StringEntity params =new StringEntity(json, "UTF-8");
request.addHeader("content-type", "application/json");
request.setEntity(params);
HttpResponse response = httpClient.execute(request);
HttpEntity entity = response.getEntity();
responseString = EntityUtils.toString(entity, "UTF-8");
}catch (Exception ex) {
ex.printStackTrace();
// handle exception here
} finally {
httpClient.getConnectionManager().shutdown();
}
return responseString;
}
The code above worked perfect even if the json string contained UTF-8 chars, and everything worked fine.
For several reasons I had to change the way I send HTTP post requests and use HttpURLConnection instead apache's HttpClient. Here's my code:
static private String sendJson(String json,String url){
String responseString = "";
try {
URL m_url = new URL(url);
HttpURLConnection conn = (HttpURLConnection)m_url.openConnection();
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("content-type", "application/json");
DataOutputStream outputStream = new DataOutputStream(conn.getOutputStream());
outputStream.writeBytes(json);
outputStream.flush();
outputStream.close();
BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream()));
StringBuilder sb = new StringBuilder();
String line;
while ((line = br.readLine()) != null) {
sb.append(line+"\n");
}
br.close();
responseString = sb.toString();
} catch (MalformedURLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return responseString;
}
This code works well for normal English characters, but doesn't seem to support UTF-8 characters in the json string, since it fails each time. (when sending json to server, server crushes saying that utf8 cant decode a certain byte, but when recieving utf8 json from server I think it does work since I manage to view the special characters).
Server didn't change at all and worked fine with previous code, so the problem is 100% on this new code snippet.
Any idea how to fix the json string sending so it would support UTF 8? Thanks
I think the problem is in this part:
DataOutputStream outputStream = new DataOutputStream(conn.getOutputStream());
outputStream.writeBytes(json);
outputStream.flush();
outputStream.close();
Instead of doing this you need to encode json as UTF-8
and send those bytes which represent the UTF-8 encoding.
Try using this:
Charset.forName("UTF-8").encode(json)
See:
Charset.encode
An even simpler approach is to use e.g. a BufferedWriter wrapping an
OutputStreamWriter. The OutputStreamWriter knows about its own encoding
and so it will do the work for you (the encoding work of the json String).
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(conn.getOutputStream(), "UTF-8"));
bw.write(json);
bw.flush();
bw.close();
When writing a String to an output stream (bytes), you need to specify the encoding to do the conversion.
One way to do is to wrap the output stream in a OutputStreamWriter that will use UTF-8 charset for the encoding.
conn.setRequestProperty("content-type", "application/json; charset=utf-8");
Writer writer = new BufferedWriter(new OutputStreamWriter(conn.getOutputStream(), "UTF-8"));
writer.write(json);
writer.close();
The flush() is also optional if you call close().
Another option, as mentionned by peter.petrov is to first convert your String to bytes (in memory) and then output the byte array to your output stream.
And to make it obvious on the server side, you can pass the charset used in the content-type header ("content-type", "application/json; charset=utf-8").
StringEntity uses a Charset to make sure the encoding is right. It does ~that:
byte[] content = s.getBytes(charset);
Without much change in your code, your write can be:
outputStream.write(json.getBytes("UTF-8"));
As for your read, there is no point in using a BufferedReader with readLine, except for normalizing the end of line. It is much slower than other methods, as it requires to read each byte individually.
EntityUtils does mostly that:
Reader reader = new InputStreamReader(conn.getInputStream(), "UTF-8");
StringBuilder buffer = new StringBuilder();
char[] tmp = new char[1024];
int l;
while((l = reader.read(tmp)) != -1) {
buffer.append(tmp, 0, l);
}
responseString = buffer.toString();
Related
I have a textmessage/string with letters like ä,ü,ß. I want everything to be UTF-8 encoded. When I write to a file or print the string to console, everything is fine. But when I want to send the same string to a web service, I get instead of ä,ü,ß the following �
I read the file from a Servlet.
Do I really have to use the following 2 lines to get a UTF-8 encoded text?
byte [] bray = text.getBytes("UTF-8");
text = new String(bray);
.
public static String readAsStream_UTF8(String filePathName){
String text ="";
InputStream input = Thread.currentThread().getContextClassLoader().getResourceAsStream("resources/"+filePathName);
if(input == null){
System.out.println("Inputstream null.");
}else{
InputStreamReader isr = null;
try {
isr = new InputStreamReader((InputStream)input, "UTF-8");
BufferedReader reader = new BufferedReader(isr);
StringBuilder sb = new StringBuilder();
String sCurrentLine;
while ((sCurrentLine = reader.readLine()) != null) {
sb.append(sCurrentLine);
}
text= sb.toString();
//it works only if I use the following 2 lines
byte [] bray = text.getBytes("UTF-8");
text = new String(bray);
} catch (Exception e1) {
e1.printStackTrace();
}
}
return text;
}
My sendPOST method looks something like the following:
String charset = "UTF-8";
OutputStreamWriter writer = null;
HttpURLConnection con = null;
String response_txt ="";
InputStream iss = null;
try {
URL url = new URL(urlService);
con = (HttpURLConnection)url.openConnection();
con.setDoOutput(true); //triggers POST
con.setDoInput(true);
con.setRequestMethod("POST");
con.setRequestProperty("accept-charset", charset);
//con.setRequestProperty("Content-Type", "application/soap+xml");
con.setRequestProperty("Content-Type", "application/soap+xml;charset=UTF-8");
writer = new OutputStreamWriter(con.getOutputStream());
writer.write(msg); //send POST data string
writer.flush();
writer.close();
What do I have to do to force the msg, that will be sent to the web service, to really be UTF-8 encoded.
If you know the encoding of the file which you want to send you don't need to convert it to an intermediary string. Simply copy its bytes to the output:
// inputstream to a UTF-8 encoded resource file
InputStream in = Thread.currentThread().getContextClassLoader().getResourceAsStream("resources/"+filePathName);
HttpURLConnection con = ...
// set contenttype and encoding
con.setRequestProperty("Content-Type", "application/soap+xml;charset=UTF-8");
// copy input to output
copy(in, con.getOutputStream());
using some copy function.
Additionally you could also set the Content-Length header to the size of the resource file.
I tried to do post call and to pass input with this value - "ä€愛لآहที่"
I got error message
{"error":{"code":"","message":{"lang":"en-US","value":{"type":"ODataInputError","message":"Bad Input: Invalid JSON format"}}}}
This is my code
conn.setRequestMethod(ConnectionMethod.POST.toString());
conn.setRequestProperty(CONTENT_LENGTH, Integer.toString(content.getBytes().length));
conn.setRequestProperty("Accept-Charset", "UTF-8");
conn.setUseCaches(false);
conn.setDoInput(true);
conn.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(conn.getOutputStream());
wr.writeBytes(content);
wr.flush();
wr.close();
InputStream resultContentIS;
String resultContent;
try {
resultContentIS = conn.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(resultContentIS));
StringBuilder sb = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
sb.append(line);
}
it falied on conn.getInputStream();
The value of content is
{ "input" : "ä€愛لآहที่" }
It is working where the input is String or integer
When I added the statement
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded; charset=utf-8");
I got different message
{"error":{"code":"","message":{"lang":"en-US","value":{"type":"Error","message":"Internal server error"}}}}
Please try this code below:
DataOutputStream wr = new DataOutputStream(conn.getOutputStream());
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(wr, "UTF-8"));
writer.write(content);
writer.close();
wr.close();
You should use JSONObject to pass params
The input, please try
BufferedReader reader = new BufferedReader(new InputStreamReader(resultContentIS, "UTF-8"));
If the out put is: ???????, so do not worry because your output console do not support UTF-8
It seems that your variable content does already have the wrong data because you may have converted a String without any attention to the required encoding.
Setting the correct enconding on the writer and use write() instead of writeBytes() should be worth a try.
You have to send content via byte array
DataOutputStream outputStream= new DataOutputStream(conn.getOutputStream());
outputStream.write(content.toString().getBytes());
This is completely solution for your file name character problems. The imported point is string sending via byte array. Every character changing via byte character. This is prevent your character encoding problems.
I'm working on Yahoo boss API. The URL supposed to return JSON, I need to store it in a string then parse it. http://developer.yahoo.com/java/howto-parseRestJava.html
My question: How can I save URL response in a string ??
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpResponse response = (HttpResponse) httpclient.execute(httpPostRequest);//send a request and receive a response
System.out.println("HTTPResponse received in [" + (System.currentTimeMillis()-t) + "ms]");
HttpEntity entity = response.getEntity();
if (entity != null) {
// Read the content stream
InputStream instream = entity.getContent();
// convert content stream to a String
String resultString= convertStreamToString(instream);
instream.close();
resultString = resultString.substring(1,resultString.length()-1); // remove wrapping "[" and "]"
and here is the function convertStreamToString
private static String convertStreamToString(InputStream is) {
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
StringBuilder sb = new StringBuilder();
try {
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sb.toString();
}
Technically, you want to wrap an appropriately configured InputStreamReader around the URL InputStream and copy the Reader to a StringWriter (apache commons IO has a "copy Reader to String" utility method). However, in order to determine the correct character set for the InputStreamReader, you need to parse the ContentType header. In which case you might be better off working with a higher level library like apache commons HttpClient.
Or, you could wrap a JSONTokener around the URL InputStream and parse the JSONObject directly from the JSONTokener (although i'm not entirely sure how the tokener determines the correct character set, so you might be safer using something like HttpClient).
So I have 2 codes which supposedly do the same. The one I'm using on Android however, returns the wrong HTML data. The stock Java one returns the correct data after sending the request. I have both codes here. Can you tell me (EVEN THOUGH I GAVE INTERNET PERMISSION TO ANDROID) why the Android one isn't working, while the stock Java one is working? This is the Android code:
EDIT: I FOUND THE FIX. If you're going to use a StringEntity to send such a String to the Server, you have to set the content to application/x-www-form-urlencoded. I've edited my code to show this:
public static String sendNamePostRequest(String urlString) {
HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost(urlString);
StringBuffer sb = new StringBuffer();
try {
StringEntity se = new StringEntity(
"__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPDwULLTE3NDM5MzMwMzRkZA%3D%3D&__EVENTVALIDATION=%2FwEWBAL%2B%2B4CfBgK52%2BLYCQK1gpH7BAL0w%2FPHAQ%3D%3D&_nameTextBox=John&_zoekButton=Zoek&numberOfLettersField=3");
se.setContent("application/x-www-form-urlencoded");
post.setEntity();
HttpResponse response = client.execute(post);
HttpEntity entity = response.getEntity();
BufferedReader br = new BufferedReader(new InputStreamReader(
entity.getContent()));
String in = "";
while ((in = br.readLine()) != null) {
sb.append(in + "\n");
}
br.close();
} catch (Exception e) {
e.printStackTrace();
}
return sb.toString();
}
This is the stock Java code:
public String sendNamePostRequest(String urlString) {
StringBuffer sb = null;
try {
String data = "__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPDwULLTE3NDM5MzMwMzRkZA%3D%3D&__EVENTVALIDATION=%2FwEWBAL%2B%2B4CfBgK52%2BLYCQK1gpH7BAL0w%2FPHAQ%3D%3D&_nameTextBox=John&_zoekButton=Zoek&numberOfLettersField=3";
// String data = "";
URL requestUrl = new URL(urlString);
HttpURLConnection conn = (HttpURLConnection) requestUrl
.openConnection();
conn.setDoOutput(true);
conn.setDoInput(true);
conn.setRequestMethod("POST");
DataOutputStream dos = new DataOutputStream(conn.getOutputStream());
dos.writeBytes(data);
dos.flush();
BufferedReader br = new BufferedReader(new InputStreamReader(
conn.getInputStream()));
String in = "";
sb = new StringBuffer();
while ((in = br.readLine()) != null) {
sb.append(in + "\n");
}
dos.close();
br.close();
} catch (Exception e) {
e.printStackTrace();
}
return sb.toString();
}
If the data is getting to the server, you might want to see what is happening there (logs, errors, exceptions, etc.) Other than that:
use can use HttpURLConnection, so you can have the exact same code
For HttpClient, not sure whey you are encoding the entity yourself. Use NameValuePair to set parameters, and HttpClient will encode them for you (correctly).
You can use NameValuePair and UrlEncodedFormEntity:
List<BasicNameValuePair> nvps = new ArrayList<BasicNameValuePair>();
nvps.add(new BasicNameValuePair(KEY1, VALUE1));
nvps.add(new BasicNameValuePair(KEY2, VALUE2));
UrlEncodedFormEntity p_entity = new UrlEncodedFormEntity(nvps,HTTP.UTF_8);
post.setEntity(p_entity);
What is the best way to use preemptive basic http authentication using HttpUrlConnection. (Assume for now I can't use HttpClient).
EDIT for clarification: I'm setting the un/pw correctly in the request header using Base64 encoding. Are there any additional flags or properties that need to be set, or is the fact that I'm setting the basic auth headers for the request all that is needed for preemptive basic auth?
If you are using Java 8 or later, java.util.Base64 is usable:
HttpURLConnection connection = (HttpURLConnection) new URL(url).openConnection();
String encoded = Base64.getEncoder().encodeToString((username+":"+password).getBytes(StandardCharsets.UTF_8)); //Java 8
connection.setRequestProperty("Authorization", "Basic "+encoded);
Then use the connection as normal.
If you're using Java 7 or lower, you'll need a method to encode a String to Base64, such as:
byte[] message = (username+":"+password).getBytes("UTF-8");
String encoded = javax.xml.bind.DatatypeConverter.printBase64Binary(message);
Yes, that's all you have to do in order to use Basic Auth. The code above to set the Request Property should be done immediately after opening the connection and before getting the Input or Output streams.
Incidentally, in case someone else runs into the same, the android problem, is also present if you use org.apache.commons.codec.binary.Base64 and do Base64.encodeBase64String(). You need to do Base64.encodeBase64() and get a byte[] then construct the string.
It caught me offguard entirely that the results would be different for the line ending between those two methods.
You can use java.net.Authenticator to configure basic auth. globally for every request send by your application, see :
http://docs.oracle.com/javase/7/docs/technotes/guides/net/http-auth.html
http://developer.android.com/reference/java/net/Authenticator.html#getPasswordAuthentication()
you need to do this just copy paste it be happy
HttpURLConnection urlConnection;
String url;
// String data = json;
String result = null;
try {
String username ="danish.hussain#gmail.com";
String password = "12345678";
String auth =new String(username + ":" + password);
byte[] data1 = auth.getBytes(UTF_8);
String base64 = Base64.encodeToString(data1, Base64.NO_WRAP);
//Connect
urlConnection = (HttpURLConnection) ((new URL(urlBasePath).openConnection()));
urlConnection.setDoOutput(true);
urlConnection.setRequestProperty("Content-Type", "application/json");
urlConnection.setRequestProperty("Authorization", "Basic "+base64);
urlConnection.setRequestProperty("Accept", "application/json");
urlConnection.setRequestMethod("POST");
urlConnection.setConnectTimeout(10000);
urlConnection.connect();
JSONObject obj = new JSONObject();
obj.put("MobileNumber", "+97333746934");
obj.put("EmailAddress", "danish.hussain#dhl.com");
obj.put("FirstName", "Danish");
obj.put("LastName", "Hussain");
obj.put("Country", "BH");
obj.put("Language", "EN");
String data = obj.toString();
//Write
OutputStream outputStream = urlConnection.getOutputStream();
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(outputStream, "UTF-8"));
writer.write(data);
writer.close();
outputStream.close();
int responseCode=urlConnection.getResponseCode();
if (responseCode == HttpsURLConnection.HTTP_OK) {
//Read
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream(), "UTF-8"));
String line = null;
StringBuilder sb = new StringBuilder();
while ((line = bufferedReader.readLine()) != null) {
sb.append(line);
}
bufferedReader.close();
result = sb.toString();
}else {
// return new String("false : "+responseCode);
new String("false : "+responseCode);
}
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (JSONException e) {
e.printStackTrace();
}
I was having this problem too.
And Now I have solved this problem.
My code is :
URL url = new URL(stringUrl);
String authStr = "MyAPIKey"+":"+"Password";
System.out.println("Original String is " + authStr);
// encode data on your side using BASE64
byte[] bytesEncoded = Base64.encodeBase64(authStr .getBytes());
String authEncoded = new String(bytesEncoded);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty("Authorization", "Basic "+authEncoded);
It may help many others.
Best of luck.
Regarding the Base64 encoding problem, I found this library: http://sourceforge.net/projects/migbase64/
I have not fully vetted it but I am using it for the Basic Authentication solution shown above (as well as for image encoding/decoding), and it works well. It provides a parameter for whether or not to include the newline.