We Keep Coding

Apache HttpClient. How to properly produce multipart requests with Encoding

Similar to the already existing question Apache HttpClient making multipart form post
want to produce a http request, holding a file and a key=val pair.
Currently the code looks like:
HttpPost post = new HttpPost("http://localhost/mainform.cgi/auto_config.htm");
HttpEntity ent = MultipartEntityBuilder.create()
.addTextBody("TYPE", "6",ContentType.TEXT_BINARY)
.addBinaryBody("upname", new File("factory.cfg"),ContentType.APPLICATION_OCTET_STREAM,"factory.cfg")
.build();
This is simply applied to HttpPost object as entity and passed to the client.
Which is sent to a linux-type device (blackbox) with a lighthttp service running. The problem is that when I send this request I do not see a response from the device(physical, and the HttpEntity always returns a default 200 OK).
Through Wireshark I've noticed two differences, on which i would really appreciate to get some help:
1. The multipart elements have an additional header(comparing to the original request) - Content-transfer-encoding, which i assume may be the reason of the fail.
2. Content length differs drastically.
So the first question would be - how to deal with the Encoding?

Found the point, on which it was failing. Needed this addition:
.addTextBody("TYPE", "6",ContentType.WILDCARD)
.setMode(HttpMultipartMode.BROWSER_COMPATIBLE).setCharset(Charset.forName("UTF-8"))

HttpClient redirecting

I am trying to write a video downloader from kissanime.to . I am using HttpClient library. This site is using cloudflare. It redirects after 5 secs. How can I set so my application will go to the redirected link? My code below isnt working. Where am I going wrong and how can I fix it.
HttpGet request = new HttpGet(url);
HttpClient httpClient = HttpClientBuilder.create()
.setRedirectStrategy(new LaxRedirectStrategy()).build();
HttpResponse response = httpClient.execute(request);
HttpEntity entity = response.getEntity();
String entityContents = EntityUtils.toString(entity);`

It means that given site is under DDOS protection mode (maybe you try to open it too often?) to workaround you would need to stop hitting it that much (e.g. wait some time between tries). Or if you insist use some javascript executing library (Rhino?) that would execute javascript that cloudflare is using.

HttpClient can't get response from server

This problem has blocked our whole team half a day!
We use apache httpclient 4.3.x to post and get data from an storage server which provides http api. In order to improve performance, we used PoolingHttpClientConnectionManager:
public HttpClient createHttpClient() {
Registry registry = RegistryBuilder.create()....build();
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager(registry);
connectionManager.setMaxTotal(50);
connectionManager.setDefaultMaxPerRoute(50);
CloseableHttpClient httpClient = HttpClients.custom()
.setConnectionManager(connectionManager)
.build();
return httpClient;
}
Then we hold an instance of the httpClient in our program, reuse it with every http request:
Global httpClient:
HttpClient httpClient = createHttpClient();
Post some data:
HttpPost httpPut = new HttpPost("...");
HttpResponse response = httpClient.execute(httpPut);
// Notice we get the response content here!
String content = EntityUtils.toString(response.getEntity());
System.out.println(content);
httpPut.releaseConnection();
response.close();
Then get:
HttpGet httpGet = new HttpGet("...");
// Blocked at this line !!!!
HttpResponse response = httpClient.execute(httpGet);
String content = EntityUtils.toString(response.getEntity());
System.out.println(content);
httpPut.releaseConnection();
response.close();
Please notice the line: // Blocked at this line !!!!
The program has blocked at that line and never go to next line. In debugging mode, I can see it has been blocked at:
SocketInputStream.socketRead0()
I've searched for a lot of questions and documents, but no lucky.
My colleage just fix it by setting NoConnectionReuseStrategy.INSTANCE:
HttpClients.custom()
.setConnectionManager(connectionManager)
// Following line fixed the problem, but why?
.setConnectionReuseStrategy(NoConnectionReuseStrategy.INSTANCE)
.build();
Now it doens't blocked, but why?
What does "reuse connection" mean? And is there performance issue by using NoConnectionReuseStrategy?
Thank you, guys~

I tried to reproduce the blocking http-get (also as an exercise for myself) but even without closing responses I could not get it to block. The ONLY time I managed to make the http-get block is by doing a response.getEntity().getContent() without reading from the returned InputStream and without closing the returned InputStream.
For my tests I used Tomcat 7.0.47 with two very simple servlets (one responding "OK" to a get, the other echoing a post) as a server. The client started 50 threads with each thread performing 30 alternating http-get and http-post request (total of 1500 requests). The client did not use the RegistryBuilder, instead the default one is used (created by the PoolingHttpClientConnectionManager itself).
About the NoConnectionReuseStrategy: by default (HttpClient created with HttpClients.createDefault(), I used org.apache.httpcomponents:httpclient:4.3.1) a connection pool is used with a maximum of 2 connections to 1 server. E.g. even if 5 threads are doing all kinds of requests at the same time to 1 server, the connection pool opens only 2 connections, re-uses them for all requests and ensures that 1 connection is used by 1 thread at any given time. This can have a very positive impact on client performance and significantly reduces load on the server. The only thing you must make sure is to call response.close() in a finally-block (this ensures the connection is returned to the connection pool). By using the NoConnectionReuseStrategy you basically disable the connection pool: for each request a new connection will be created. I recommend you enable debug-logging for category org.apache.http.impl.conn.PoolingHttpClientConnectionManager, it is very informative.
A note about httpPut.releaseConnection(): this does not actually release a connection, it only ensures that you can re-use the "httpPut" object in a next request (see the apidocs, follow the shown link). Also note that in your code for the "httpGet", you call releaseConnection() on "httpPut" instead of "httpGet".

Ran into this problem just a while back. In case someone else comes across this problem, this post might be useful.
I am using a Java Servlet to service my requests. When I wrote to the response stream using the PrintWriter instance my client blocked. Tried writing to the OutputStream directlyresponse.getOutputStream.write("myresponse") and it worked.

Polling data from restful webservice with "keepalive" connection

i'm developing an android application that fetches data from a restful web service.
i was thinking about the best way to implement this, my code is working but i'm sure it can be optimized.
here's my current code state:
i'm calling the restResource from a service using a scheduler.
the bad part is opening and closing connection on every request.
(the rest resource keeps realtime data( i have to fetch it every second), when it's empty it returns an empty array)
httpResponse = client.execute(request);
responseCode = httpResponse.getStatusLine().getStatusCode();
message = httpResponse.getStatusLine().getReasonPhrase();
HttpEntity entity = httpResponse.getEntity();
if (entity != null) {
InputStream instream = entity.getContent();
response = convertStreamToString(instream);
instream.close();
}
how to make use of the keepalive header returned with the response?

I have done this before using long-polling, in which my RESTful call blocks until either there is a timeout or it has something to reply with. Either way, the calling client would re-call the RESTful service again on either response (timeout or valid).
I am not sure how to you could use the keepalive header, but it seems like what you would do is send it, and then simply keep the connection open. In your server-side java code, you would get the output buffer for the response and start sending stuff down. On the client side, get the input buffer (not sure how this happens in Android, but I am under the impression its Java code, so there are loads of ways you can do any of this) and start reading. At some point I would think you would need to force the connection close, just because you don't want to keep it open indefinately.
Again, I have never tried it this way, so I don't know. Also, I am not sure if this breaks some unspoken laws of RESTful services. I have never read anything to contradict this approach.

Multiple Difficulties with HttpURLConnection class in Java for Android

--Update--
Apologies for those who helped me, it turns out this is just a problem with Eclipse's debugger. After suspecting that it was leading me wrong, I placed down a couple of System.out.println to watch the variables, and according to them they ARE being changed, and that the debugger was just showing me old information for whatever reason. No clue why that's happening, but the important thing is that the code does apparently actually work.
I'm working on a method to share with twitter for an Android application, and I'm having errors when setting up the HttpURLConnection. I create the connection object as per usual, using the openconnection function of a url then casting it to a HttpURLConnection, and when I subsequently run SetRequestMethod("POST") on the connection, it does absolutely nothing. When I run the code in the debugger line by line, as I go through that line the request method just remains as the default ("GET"). Anyone have any idea as to why this may be happening? I'm getting the same problem with setDoOutput(true) also not changing anything. However, adding a request property does still work. I've been searching around and haven't been able to find anything on this problem, not even another person reporting these problems.

I am not sur whether using HttpURLConnection is the best here.
Did you try the following way?
// Building the POST request
final BasicNameValuePair message = new BasicNameValuePair("yourField", "yourContent");
final List<NameValuePair> list = new ArrayList<NameValuePair>(1);
list.add(message);
final HttpPost httppost = createHttpPost(UrlEncodedFormEntity(list));
// Building the HTTP client
final HttpParams httpParameters = new BasicHttpParams();
HttpConnectionParams.setConnectionTimeout(httpParameters, YOUR_CHOSEN_CONN_TIMEOUT);
HttpConnectionParams.setSoTimeout (httpParameters, YOUR_CHOSEN_SO_TIMEOUT);
final HttpClient httpClient = new DefaultHttpClient(httpParameters);
// Execution of the POST request
final HttpResponse response = httpClient.execute(httppost);
This is the way I usually do, with no problems.
[EDIT: 04-25-2014] Apache's HttpClient was the best approach for Froyo and former versions. Now, according to this article from Android Developers Blog (written after this Q&A), it is better to use URLConnection.