We Keep Coding

How can I be sure that an HttpClient PostMethod contains UTF-8 strings in the parameters?

In our webapp, we have to send a POST request via HttpClient to an endpoint on our network, which will receive this and do some work with it. We are having trouble with character encoding, and I am having difficulties finding an answer to my question.
We have used the postMethod.getParams().setContentCharset("UTF-8") method when sending the request, but on the receiving end, it seems like the characters are still encoded in ISO 8859-1. I have determined this because when I inspect the String on the receiving side, it has garbage characters in it that go away once I follow the steps found at https://stackoverflow.com/a/16549329/1130549. Is there any extra steps I need to take on the sending end to ensure that I am actually writing characters in UTF-8 as expected? All we are doing now is using postMethod.addParameter(paramKey, paramValue) with native String objects.
Edit: Here is a very simple example of how we're sending the POST request. For what it's worth, the values are being taken from an XMLBeans object.
PostMethod postMethod = new PostMethod(url);
postMethod.getParams().setContentCharset("UTF-8");
postMethod.addParameter("key1", "value1");
postMethod.addParameter("key2", "value2");
HttpClient httpClient = new HttpClient();
int status = httpClient.executeMethod(postMethod);

EDIT
Simpler solution is to encode the value
postMethod.addParameter("key1", URLEncoder.encode("value1","UTF-8"));
To encode properly UTF-8, you can execute differently, using StringEntity and NameValuePair, e.g.:
try (CloseableHttpClient httpClient = HttpClients.custom().build()) {
URIBuilder uriBuilder = new URIBuilder(url);
HttpHost target = new HttpHost(uriBuilder.getHost(), uriBuilder.getPort(), uriBuilder.getScheme());
List<NameValuePair> nameValuePairs = new ArrayList<>();
nameValuePairs.add(new BasicNameValuePair("key1", "value1"));
nameValuePairs.add(new BasicNameValuePair("key2", "value2"));
String entityValue = URLEncodedUtils.format(nameValuePairs, StandardCharsets.UTF_8.name());
StringEntity entity = new StringEntity(entityValue, StandardCharsets.UTF_8.name());
post.setEntity(entity);
httpClient.execute(target, post);

First of all, you do need to make sure that the string that you are actually writing is encoded in UTF-8. I realized that you already know that but still double-check that it is so, as it would be the prime suspect of your problem. Also, I would recommend trying a much simpler HTTP client. Apache HTTP client (I believe that's the library that you are using) is an excellent library. But due to covering a very wide range of options it tends to be a bit bulky. So, or simple requests I would suggest a lightweight HTTP client that maybe not that comprehensive as Apache library but offers simplicity as a trade-off. Here how your code may look like:
private static void testHttpClient() {
HttpClient client = new HttpClient();
// client.setContentType("text/html; charset=utf-8");
client.setContentType("application/json; charset=utf-8");
client.setConnectionUrl("http://www.my-url.com");
String content = null;
try {
String myMessage = getMyMessage() // get the string that you want to send
content = client.sendHttpRequest(HttpMethod.POST, myMessage);
} catch (IOException e) {
content = client.getLastResponseMessage() + TextUtils.getStacktrace(e, false);
}
System.out.println(content);
}
It looks much more simple, I think. Also in the same library, there is another utility that allows you to convert any string in any language into a sequence of unicodes and vice-versa. This helped me numerous times to diagnose encoding thorny issues. For instance, if you see some gibberish symbols that could be a wrong display of a valid character or actual character loss. Here is an example of how it works:
result = "Hello World";
result = StringUnicodeEncoderDecoder.encodeStringToUnicodeSequence(result);
System.out.println(result);
result = StringUnicodeEncoderDecoder.decodeUnicodeSequenceToString(result);
System.out.println(result);
The output of this code is:
\u0048\u0065\u006c\u006c\u006f\u0020\u0057\u006f\u0072\u006c\u0064
Hello World
That might help you to check if the string you passed is valid or not. The library is called MgntUtils and could be found at Maven Central or at Github It comes as maven artifact and with sources and Javadoc. Javadoc could be found separately here
Disclaimer: The MgntUtils library is written by me

StringEntity object from JSONObject gives java.io.UnsupportedEncodingException

I am new to Java and trying to make native android application which includes making HTTP Calls to API Server. Now My issue is that for making HTTP POST (apache httpPost and httpClient) call with some JSON data. So to make StringEntity out of JSONObject I am writing this line of code:
StringEntity userDataStringEntity = new StringEntity(userDataString);
Where StringEntity is imported from import org.apache.http.entity.StringEntity;.
I have tried searching for this issue and I am finding same method with same "string" parameter.
Here are some links, but it didn't help me:
How to send a JSON object over HttpClient Request with Android?
How to send a JSON object over Request with Android?

That's definitely weird, by default the StringEntity goes for the charset "ISO-8859-1" which tells me that the userDataString is in another charset.
Either way, try:
StringEntity userDataStringEntity = new StringEntity(userDataString, "UTF-8");
This will work for utf-8 encoded strings.

Perhaps unrelated, but I was getting an error at compile time, as the new StringEntity(str) wasn't wrapped in a try catch.
Might be of use to someone tho :)

android HTTP POST fail

I'm trying to connect and post to a simple java webservice, running the post's URL from chrome succeeded, but android code skip the following lines (without throwing errors), but the webservice doesn't accept the post
HttpPost post = new HttpPost(setFacebookEventsAddress+userId+"/"+accesstoken);
post.setHeader("Accept", "application/json");
post.setHeader("Content-type", "application/json");
HttpResponse response = client.execute(post);
the webservice method signature handling the above request:
#GET
#Path("setData/{user_id}/{accessToken}")
#Produces(MediaType.APPLICATION_JSON+ ";charset=utf-8")
public String setData(#PathParam("user_id") String user_id,
#PathParam("accessToken") String accessToken) {
since I manage to post throw my browser, anyone can help with what's wrong with my android code?

URL url = new URL(setFacebookEventsAddress+userId+"/"+accesstoken);
HttpURLConnection con = (HttpURLConnection) url
.openConnection();
ja = readStream(con.getInputStream());
Using HttpURLConnection instead of HttpPost did the trick for me, thanks for all the helpers!

It is not possible to say with any certainty (given the evidence), but my guess would be that the expression
setFacebookEventsAddress + userId + "/" + accesstoken
is evaluating to a different URL to the one you are using from the web browser.
I suggest that you try the following:
Turn on request logging on your server, and compare the URLs in the requests being sent.
Modify your client to print out the response status code and the response body. The latter is likely to be an error page that will give you more clues.
Another possible problem is that your code doesn't appear to be sending any body with the POST request.
On revisiting this, the problem was that you were using / trying to do a POST to a web service that you had configured to support GET only. I expect that if you had looked at the status code you would have found that the response code was "Method not supported".

"Illegal Characters" in URL for HttpGet in Android get double-encoded

I am trying to find a solution to this the whole evening now...
I write an app which requests data from a web server. The Server answers in JSON format.
Everything works well except when I enter a umlaut like ä into my App.
In the following I assume the request URL is http://example.com/?q= and I am searching for "Jäger"
The correct call would then be h++p://example.com/?q=J%C3%A4ger
(Sorry for plus-signs but the spam protection doesnt let me post it correctly.)
So my problem is now:
When I give my URL String encoded or unencoded over to HttpGet it will always result in a doublee-encoded URL.
The Request to my Server is then http://example.com/?q=J%25C3%25A4ger (It encodes the percent signs)
which leads to the server searching in database for J%C3%A4ger what is obviously wrong.
So my question is how can I achive that if the user enters "Jäger" my app calls the correctly encoded URL?
Thanks for any help!
Here is the currently used code... Ist probably the worst possible idea I had...
URI url = new URI("http", "//example.com/?q=" + ((EditText)findViewById(R.id.input)).getText().toString(), null);
Log.v("MyLogTag", "API Request: " + url);
HttpGet httpGetRequest = new HttpGet(url);
// Execute the request in the client
HttpResponse httpResponse;
httpResponse = defaultClient.execute(httpGetRequest);

Update: Sorry, HttpParams isn't meant for request parameters but for configuring HttpClient.
On Android, you might want to use Uri.Builder, like suggested in this other SO answer:
Uri uri = new Uri.Builder()
.scheme("http")
.authority("example.com")
.path("someservlet")
.appendQueryParameter("param1", foo)
.appendQueryParameter("param2", bar)
.build();
HttpGet request = new HttpGet(uri.toString());
// This looks very tempting but does NOT set request parameters
// but just HttpClient configuration parameters:
// HttpParams params = new BasicHttpParams();
// params.setParameter("q", query);
// request.setParams(params);
HttpResponse response = defaultClient.execute(request);
String json = EntityUtils.toString(response.getEntity());
Outside of Android, your best bet is building the query string manually (with all the encoding hassles) or finding something similar to Android's Uri.Builder.

Wrapper class for HttpGet / Post in Java?

Sorry, I'm quite new to Java.
I've stumbled across HttpGet and HttpPost which seem to be perfect for my needs, but a little long winded. I have written a rather bad wrapper class, but does anyone know of where to get a better one?
Ideally, I'd be able to do
String response = fetchContent("http://url/", postdata);
where postdata is optional.
Thanks!

HttpClient sounds like what you want. You certainly can't do stuff like the above in one line, but it's a fully-fledged HTTP library that wraps up Get/Post requests (and the rest).

I would consider using the HttpClient library. From their documentation, you can generate a POST like this:
PostMethod post = new PostMethod("http://jakarata.apache.org/");
NameValuePair[] data = {
new NameValuePair("user", "joe"),
new NameValuePair("password", "bloggs")
};
post.setRequestBody(data);
// execute method and handle any error responses.
...
InputStream in = post.getResponseBodyAsStream();
// handle response.
There are a number of advanced options for configuring the client should you eventually required those.