We Keep Coding

How can I be sure that an HttpClient PostMethod contains UTF-8 strings in the parameters?

In our webapp, we have to send a POST request via HttpClient to an endpoint on our network, which will receive this and do some work with it. We are having trouble with character encoding, and I am having difficulties finding an answer to my question.
We have used the postMethod.getParams().setContentCharset("UTF-8") method when sending the request, but on the receiving end, it seems like the characters are still encoded in ISO 8859-1. I have determined this because when I inspect the String on the receiving side, it has garbage characters in it that go away once I follow the steps found at https://stackoverflow.com/a/16549329/1130549. Is there any extra steps I need to take on the sending end to ensure that I am actually writing characters in UTF-8 as expected? All we are doing now is using postMethod.addParameter(paramKey, paramValue) with native String objects.
Edit: Here is a very simple example of how we're sending the POST request. For what it's worth, the values are being taken from an XMLBeans object.
PostMethod postMethod = new PostMethod(url);
postMethod.getParams().setContentCharset("UTF-8");
postMethod.addParameter("key1", "value1");
postMethod.addParameter("key2", "value2");
HttpClient httpClient = new HttpClient();
int status = httpClient.executeMethod(postMethod);

EDIT
Simpler solution is to encode the value
postMethod.addParameter("key1", URLEncoder.encode("value1","UTF-8"));
To encode properly UTF-8, you can execute differently, using StringEntity and NameValuePair, e.g.:
try (CloseableHttpClient httpClient = HttpClients.custom().build()) {
URIBuilder uriBuilder = new URIBuilder(url);
HttpHost target = new HttpHost(uriBuilder.getHost(), uriBuilder.getPort(), uriBuilder.getScheme());
List<NameValuePair> nameValuePairs = new ArrayList<>();
nameValuePairs.add(new BasicNameValuePair("key1", "value1"));
nameValuePairs.add(new BasicNameValuePair("key2", "value2"));
String entityValue = URLEncodedUtils.format(nameValuePairs, StandardCharsets.UTF_8.name());
StringEntity entity = new StringEntity(entityValue, StandardCharsets.UTF_8.name());
post.setEntity(entity);
httpClient.execute(target, post);

First of all, you do need to make sure that the string that you are actually writing is encoded in UTF-8. I realized that you already know that but still double-check that it is so, as it would be the prime suspect of your problem. Also, I would recommend trying a much simpler HTTP client. Apache HTTP client (I believe that's the library that you are using) is an excellent library. But due to covering a very wide range of options it tends to be a bit bulky. So, or simple requests I would suggest a lightweight HTTP client that maybe not that comprehensive as Apache library but offers simplicity as a trade-off. Here how your code may look like:
private static void testHttpClient() {
HttpClient client = new HttpClient();
// client.setContentType("text/html; charset=utf-8");
client.setContentType("application/json; charset=utf-8");
client.setConnectionUrl("http://www.my-url.com");
String content = null;
try {
String myMessage = getMyMessage() // get the string that you want to send
content = client.sendHttpRequest(HttpMethod.POST, myMessage);
} catch (IOException e) {
content = client.getLastResponseMessage() + TextUtils.getStacktrace(e, false);
}
System.out.println(content);
}
It looks much more simple, I think. Also in the same library, there is another utility that allows you to convert any string in any language into a sequence of unicodes and vice-versa. This helped me numerous times to diagnose encoding thorny issues. For instance, if you see some gibberish symbols that could be a wrong display of a valid character or actual character loss. Here is an example of how it works:
result = "Hello World";
result = StringUnicodeEncoderDecoder.encodeStringToUnicodeSequence(result);
System.out.println(result);
result = StringUnicodeEncoderDecoder.decodeUnicodeSequenceToString(result);
System.out.println(result);
The output of this code is:
\u0048\u0065\u006c\u006c\u006f\u0020\u0057\u006f\u0072\u006c\u0064
Hello World
That might help you to check if the string you passed is valid or not. The library is called MgntUtils and could be found at Maven Central or at Github It comes as maven artifact and with sources and Javadoc. Javadoc could be found separately here
Disclaimer: The MgntUtils library is written by me

Apache HttpClient. How to properly produce multipart requests with Encoding

Similar to the already existing question Apache HttpClient making multipart form post
want to produce a http request, holding a file and a key=val pair.
Currently the code looks like:
HttpPost post = new HttpPost("http://localhost/mainform.cgi/auto_config.htm");
HttpEntity ent = MultipartEntityBuilder.create()
.addTextBody("TYPE", "6",ContentType.TEXT_BINARY)
.addBinaryBody("upname", new File("factory.cfg"),ContentType.APPLICATION_OCTET_STREAM,"factory.cfg")
.build();
This is simply applied to HttpPost object as entity and passed to the client.
Which is sent to a linux-type device (blackbox) with a lighthttp service running. The problem is that when I send this request I do not see a response from the device(physical, and the HttpEntity always returns a default 200 OK).
Through Wireshark I've noticed two differences, on which i would really appreciate to get some help:
1. The multipart elements have an additional header(comparing to the original request) - Content-transfer-encoding, which i assume may be the reason of the fail.
2. Content length differs drastically.
So the first question would be - how to deal with the Encoding?

Found the point, on which it was failing. Needed this addition:
.addTextBody("TYPE", "6",ContentType.WILDCARD)
.setMode(HttpMultipartMode.BROWSER_COMPATIBLE).setCharset(Charset.forName("UTF-8"))

Cannot get URL (with sql query) to be handled by DefaultHttpClient in java

I've been struggling for half a day trying to learn this and am stuck. My goal is to query finance data from yql yahoo finance tables. I have set up some code from an AndroidHive example and am able to get it running correctly for their sample query. But that sample query just grabs a JSON object directly from the main URL that they provide. To do this for yql, I need to convert the SQL query into a format that the httpClient will recognize, and my app keeps hanging and never returning a response.
First, I tried taking the exact query string from the yql to replicate their search, which for me was this:
https://query.yahooapis.com/v1/public/yql?q=select%20symbol%2CChange%20from%20yahoo.finance.quote%20where%20symbol%20in%20(%22SH%22%2C%22DOG%22%2C%22RWM%22)&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=
That gives me a null result when I set url = to the above, and then run the following:
DefaultHttpClient httpClient = new DefaultHttpClient();
.
.
.
HttpGet httpGet = new HttpGet(url);
httpResponse = httpClient.execute(httpGet);
If I do this with the AndroidHive example URL in my code, it works fine.
If I enter the above URL in browser, it works fine. So clearly, my URL is not being entered correctly.
So then I read online that I need to use a URLEncode to convert syntax of URL into the correct format. Here's what I did:
private String url = "https://query.yahooapis.com/v1/public/yql?q=" +URLEncoder.encode("select symbol, Change from yahoo.finance.quote where symbol in (\"SH\",\"DOG\".\"RWM\")")+"&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=";
That gives me 2 problems. First, it tells me that encode is deprecated, and that I am supposed to use the new syntax that adds a "UTF-8" as a second string parameter, but when I do that, it is a compiler error of java.io.UnsupportedEncodingException (ugh), so then I just tried running it with the deprecated form, but that just leaves my app hanging forever like the original format.
I must be doing something really obvious incorrect here...
okay - updated.
HTTPS!!! versus HTTP. the sample query that i did for the AndroidHive example was HTTP and this yql query had HTTPS. if i simply change it to HTTP it works, and i didn't even need to do the URLEncoder business. The string directly from yql worked.
Now I need to start solving my JSON parsing bugs, but I've got my data!

Yahoo YQL bad request

I'm trying to use Yahoo Content Analysis using a file containing text as input. So every character and length is possible.
This code works with a simple text String (no special characters, short text) however when I use longer texts or special characters I get a Bad Request error (HTTP 400) sometimes with an error message like "no viable alternative at character '['" or without an error message.
I encode every request and HTTP Post shouldn't have any limit as to the length.
Does the Yahoo service place a limit on the length of the request and/or are there any characters that it can't handle?
Any help to help this work is appreciated!
Here's my code (using commons-httpclient):
String fileInput = FileUtils.readFileToString(f);
StringBuilder builder = new StringBuilder();
builder.append("http://query.yahooapis.com/v1/public/yql?");
System.out.println(fileInput);
builder.append("q=")
.append(URLEncoder.encode("select * from contentanalysis.analyze where text='"+ fileInput +"'" , "UTF-8"))
.append("&format=json");
final String postUrl = builder.toString();
HttpClient client = new HttpClient();
PostMethod method = new PostMethod(postUrl);
// Send POST request
int statusCode = client.executeMethod(method);

I think the problem is that while you are sending the request as an HTTP POST, the YQL query and text are all included in the URL. YQL does not really have a way for you to make HTTP POST requests directly, so I can think of a couple options:
Directly use the Content Analysis web service with an HTTP POST (docs)
Create a custom YQL data table which uses the <execute> tag to run custom JavaScript which could do the POST (example)
Of these options I think the former would be easier.

Using InputStreamEntity for buidling a http put request with httpclient won't work unless I pass the content length explicitely

I am trying to do a http request in scala using httpclient from org.apache.httpcomponents version 4.23. In particular I want to do a put using an InputStreamEntity to build the request in order to avoid copying over a large (~100Mb) byte array in memory. Here is the snippet:
val req = new HttpPut(url)
req setEntity new InputStreamEntity(contentStream, -1/*contentlength*/)
val client = new DefaultHttpClient(connManager, httpParams)
val resp = client execute req
In the code url, connManager, httpParams are defined elsewhere. The result of the code is the creation of a file on the desired location with NO content. I am testing with a contentStream which has 3 bytes. Creating the InputStreamEntity with content length as argument set explicitly to 3 will result in the code to create the file the right way. For good reasons in production I won't know the length of the stream hence I want to use negative numbers to make sure the entire stream is sent until, as advertized by the api of InputStreamEntity, the end of the stream is reached.
What am I doing wrong? Why am I getting an empty file when not explicitely setting the content length?

Not setting the content-length, will result in HTTP Client switching to chunked transfer-encoding
For this to work, the http server you are posting to must be HTTP 1.1 compliant. Is it ?