We Keep Coding

Sending special characters using HttpURLConnection in Java

I have a web service and I want to invoke that with "application/x-www-form-urlencoded" content type. The request sometimes contains special characters such as + * - and .... The problem is that destination web service doesn't receive the request perfectly. It receives something like this: "////////////////w==" almost all characters are turned to / . What is the problem?
Here is my code:
URL url = new URL("a-web-service-url");
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
httpURLConnection.setDoInput(true);
httpURLConnection.setDoOutput(true);
httpURLConnection.setUseCaches(false);
httpURLConnection.setRequestMethod("POST");
httpURLConnection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
OutputStreamWriter outputStreamWriter = new OutputStreamWriter(httpURLConnection.getOutputStream(), "UTF-8");
outputStreamWriter.write("test=/-+*=!##$%^&*()_");
outputStreamWriter.flush();
InputStream inputStream = httpURLConnection.getInputStream();
InputStreamReader inputStreamReader = new InputStreamReader(inputStream, "UTF-8");
BufferedReader bufferedReader = new BufferedReader(inputStreamReader);
StringBuilder stringBuilder;
String line;
for (stringBuilder = new StringBuilder(); (line = bufferedReader.readLine()) != null; stringBuilder = stringBuilder.append(line)) {
;
}
bufferedReader.close();
httpURLConnection.disconnect();
String response = stringBuilder.toString().trim();
The web service receives:
test=////////////////w==

Use URLEncoder to encode the string before sending.
URLEncoder.encode(message, "UTF-8" );
In this case it will be
outputStreamWriter.write(URLEncoder.encode("test=/-+*=!##$%^&*()_", "UTF-8" ));

Sending UTF-8 string using HttpURLConnection

till now I've used the following code snippet in order to send and recieve JSON strings:
static private String sendJson(String json,String url){
HttpClient httpClient = new DefaultHttpClient();
String responseString = "";
try {
HttpPost request = new HttpPost(url);
StringEntity params =new StringEntity(json, "UTF-8");
request.addHeader("content-type", "application/json");
request.setEntity(params);
HttpResponse response = httpClient.execute(request);
HttpEntity entity = response.getEntity();
responseString = EntityUtils.toString(entity, "UTF-8");
}catch (Exception ex) {
ex.printStackTrace();
// handle exception here
} finally {
httpClient.getConnectionManager().shutdown();
}
return responseString;
}
The code above worked perfect even if the json string contained UTF-8 chars, and everything worked fine.
For several reasons I had to change the way I send HTTP post requests and use HttpURLConnection instead apache's HttpClient. Here's my code:
static private String sendJson(String json,String url){
String responseString = "";
try {
URL m_url = new URL(url);
HttpURLConnection conn = (HttpURLConnection)m_url.openConnection();
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("content-type", "application/json");
DataOutputStream outputStream = new DataOutputStream(conn.getOutputStream());
outputStream.writeBytes(json);
outputStream.flush();
outputStream.close();
BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream()));
StringBuilder sb = new StringBuilder();
String line;
while ((line = br.readLine()) != null) {
sb.append(line+"\n");
}
br.close();
responseString = sb.toString();
} catch (MalformedURLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return responseString;
}
This code works well for normal English characters, but doesn't seem to support UTF-8 characters in the json string, since it fails each time. (when sending json to server, server crushes saying that utf8 cant decode a certain byte, but when recieving utf8 json from server I think it does work since I manage to view the special characters).
Server didn't change at all and worked fine with previous code, so the problem is 100% on this new code snippet.
Any idea how to fix the json string sending so it would support UTF 8? Thanks

I think the problem is in this part:
DataOutputStream outputStream = new DataOutputStream(conn.getOutputStream());
outputStream.writeBytes(json);
outputStream.flush();
outputStream.close();
Instead of doing this you need to encode json as UTF-8
and send those bytes which represent the UTF-8 encoding.
Try using this:
Charset.forName("UTF-8").encode(json)
See:
Charset.encode
An even simpler approach is to use e.g. a BufferedWriter wrapping an
OutputStreamWriter. The OutputStreamWriter knows about its own encoding
and so it will do the work for you (the encoding work of the json String).
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(conn.getOutputStream(), "UTF-8"));
bw.write(json);
bw.flush();
bw.close();

When writing a String to an output stream (bytes), you need to specify the encoding to do the conversion.
One way to do is to wrap the output stream in a OutputStreamWriter that will use UTF-8 charset for the encoding.
conn.setRequestProperty("content-type", "application/json; charset=utf-8");
Writer writer = new BufferedWriter(new OutputStreamWriter(conn.getOutputStream(), "UTF-8"));
writer.write(json);
writer.close();
The flush() is also optional if you call close().
Another option, as mentionned by peter.petrov is to first convert your String to bytes (in memory) and then output the byte array to your output stream.
And to make it obvious on the server side, you can pass the charset used in the content-type header ("content-type", "application/json; charset=utf-8").

StringEntity uses a Charset to make sure the encoding is right. It does ~that:
byte[] content = s.getBytes(charset);
Without much change in your code, your write can be:
outputStream.write(json.getBytes("UTF-8"));
As for your read, there is no point in using a BufferedReader with readLine, except for normalizing the end of line. It is much slower than other methods, as it requires to read each byte individually.
EntityUtils does mostly that:
Reader reader = new InputStreamReader(conn.getInputStream(), "UTF-8");
StringBuilder buffer = new StringBuilder();
char[] tmp = new char[1024];
int l;
while((l = reader.read(tmp)) != -1) {
buffer.append(tmp, 0, l);
}
responseString = buffer.toString();

when servlet receive xml data , there is a error

servelt code
System.out.println(" ================servlet==================");
InputStream in = request.getInputStream();
int a = in.available();
byte[] b = new byte[a];
in.read(b);
String stringValue = new String(b,"utf-8");
System.out.println("receive data==="+stringValue);
OutputStream dataOut = response.getOutputStream();
String responseData = "<test>test</test>";
System.out.println("response datea==="+responseData);
dataOut.write(responseData.getBytes("utf-8"));
dataOut.flush();
dataOut.close();
client code
System.out.println("================client======================");
java.net.URL url = new java.net.URL("test address");
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setUseCaches(false);
con.setDoOutput(true);
con.setDoInput(true);
con.setRequestMethod("POST");
String sendData = "<data>send</data>";
System.out.println("send data="+sendData);
OutputStream dataOut = con.getOutputStream();
dataOut.write(sendData.getBytes("utf-8"));
dataOut.flush();
dataOut.close();
InputStream in = con.getInputStream();
int a = in.available();
byte[] b = new byte[a];
in.read(b);
String stringValue = new String(b,"utf-8");
in.close();
System.out.println("receive data="+stringValue);
I get the print results
servlet console
================servlet==================
receive data===
response datea===test
client console
================client======================
send data=<data>send</data>
receive data=<test>test</test>
My question is that servlet can't receive the data from the client
who can help me?

My question is that servlet can't receive the data from the client
It may not be the only problem, but this code is completely broken:
int a = in.available();
byte[] b = new byte[a];
in.read(b);
You're assuming that all the data is available right at the start. You should instead be reading from the stream until it runs out of data. Given that you want the result as text, I'd wrap the stream in an InputStreamReader and read from there. For example:
BufferdReader reader = new BufferedReader(new InputStreamReader(in, "UTF-8"));
String line;
while ((line = reader.readLine()) != null) {
System.out.println("Servlet read line: " + line);
}
If you actually want to read it as XML, you should be able to pass the InputStream (or Reader) to an XML parser library to create a DOM.
You should be doing the same thing in the client code too, by the way. Basically:
Never ignore the return value of InputStream.read
Avoid using available(); it's rarely appropriate
Use an InputStreamReader to read text from a stream, rather than constructing it yourself from the bytes
Use an XML API to read XML rather than handling it as raw text

As of now I can see that the value of int b is 0 so it is not reading any data from the input stream.
According to this documentation
available
will always return 0 for InputStream which has been extended byt the
ServletInputStream.
As told by Jon or
Edit:
InputStream is=request.getInputStream();
OutputStream os=response.getOutputStream();
byte[] buf = new byte[1024];
int chunk = is.read(buf);

Corrupted JSON encoding?

I'm sending a JSON object of the same class from a servlet to an applet, but
all strings variables in this class are missing some characters like: 'ą', 'ę', 'ś', 'ń', 'ł'.
However, 'ó' is displayed normally (?). For example:
"Zaznacz prawid?ow? operacj? porównywania dwóch zmiennych typu"
Solution
I wish I could explain it more thoroughly, but as Henry noticed, it's IDE causing this issue. I solved it using farmer1992's class from the google ticket. It prints escaped unicode characters (\u...) - the only way my applet could encode characters correctly. Also I have to restart NetBeans IDE from time to time to force the Tomcat servlet to work correctly (I have no idea why :) ).
Servlet code (updated with solution):
//begin of the servlet code extract
public void sendToApplet(HttpServletResponse response, String path) throws IOException
{
TestServlet x = new TestServlet();
x.load(path);
String json = new Gson().toJson(x);
response.setCharacterEncoding("UTF-8");
response.setContentType("application/json;charset=UTF-8");
PrintWriter out = response.getWriter();
//out.print(json);
//out.flush();
GhettoAsciiWriter out2 = new GhettoAsciiWriter(out);
out2.write(json);
out2.flush();
}
//end of the servlet code extract
Applet code:
//begin of the applet code extract
public void retrieveFromServlet(String path) throws MalformedURLException, IOException
{
String encoder = URLEncoder.encode(path, "UTF-8");
URL urlServlet = new URL("http://localhost:8080/ProjektServlet?action=" + encoder);
URLConnection connection = urlServlet.openConnection();
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setUseCaches(false);
connection.setRequestProperty("Content-Type", "application/json;charset=UTF-8");
InputStream inputStream = connection.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(inputStream));
String json = br.readLine();
Test y = new Gson().fromJson(json, Test.class);
inputStream.close();
}
//end of the applet code extract

those chars should encode in \uxxxx form
you can see this ticket
http://code.google.com/p/google-gson/issues/detail?id=388#c4

With this line
BufferedReader br = new BufferedReader(new InputStreamReader(inputStream));
the platform character encoding will be used (which may or may not be UTF-8). Try to set the encoding explicitly with
BufferedReader br = new BufferedReader(new InputStreamReader(inputStream, "UTF-8"));

How can I send POST data through url.openStream()?

i'm looking for tutorial or quick example, how i can send POST data throw openStream.
My code is:
URL url = new URL("http://localhost:8080/test");
InputStream response = url.openStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(response, "UTF-8"));
Could you help me ?

URL url = new URL(urlSpec);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod(method);
connection.setDoOutput(true);
connection.setDoInput(true);
// important: get output stream before input stream
OutputStream out = connection.getOutputStream();
out.write(content);
out.close();
// now you can get input stream and read.
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line = null;
while ((line = reader.readLine()) != null) {
writer.println(line);
}

Use Apache HTTP Compoennts http://hc.apache.org/httpcomponents-client-ga/
tutorial: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/fundamentals.html
Look for HttpPost - there are some examples of sending dynamic data, text, files and form data.

Apache HTTP Components in particular, the Client would be the best way to go.
It absracts a lot of that nasty coding you would normally have to do by hand