We Keep Coding

Why does Apache CloseableHttpResponse not consume the entity on close?

Looking at the quick start guide it gives the following code example:
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpGet = new HttpGet("http://targethost/homepage");
CloseableHttpResponse response1 = httpclient.execute(httpGet);
// The underlying HTTP connection is still held by the response object
// to allow the response content to be streamed directly from the network socket.
// In order to ensure correct deallocation of system resources
// the user MUST call CloseableHttpResponse#close() from a finally clause.
// Please note that if response content is not fully consumed the underlying
// connection cannot be safely re-used and will be shut down and discarded
// by the connection manager.
try {
System.out.println(response1.getStatusLine());
HttpEntity entity1 = response1.getEntity();
// do something useful with the response body
// and ensure it is fully consumed
EntityUtils.consume(entity1);
} finally {
response1.close();
}
The two comments in the code above say that we must close the response object for
"correct deallocation of system resources"
and
"if response content is not fully consumed the underlying connection cannot be safely re-used and will be shut down and discarded by the connection manager".
Now Apache have very kindly implementend a CloseableHttpResponse for us, which means we can use a try-with-resources block. But the close method only closes the response object, why doesn't it also consume the entity?

Because it is hard to say at that point whether or not the caller intends to re-use the underlying connection. In some cases one may want to read just a small chunk from a large response body and immediately terminate the connection.
In other words, the same thing happens over and over again: there is no one way that can make everyone happy.
The code snippet will do ensure proper de-allocation of resources while trying to keep the underlying connection alive.
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpGet = new HttpGet("http://targethost/homepage");
CloseableHttpResponse response1 = httpclient.execute(httpGet);
try {
System.out.println(response1.getStatusLine());
} finally {
EntityUtils.consume(response1.getEntity());
}

HTTPClient never leaves socketRead() when executing GET on stream - workaround?

I am using Apache HttpClient (from Apache HTTP Components 4.3) in order to execute a GET against a ShoutCast stream:
CloseableHttpClient client = HttpClients.createDefault();
HttpGet request = new HttpGet("http://relay3.181.fm:8062/");
CloseableHttpResponse response = client.execute(request);
The call to client.execute() never returns, and according to the debugger it is a nested invocation to java.net.SocketInputStream#socketRead0() which is the last node in the call stack. From profiling the code, my only conclusion (based on a steadily rising number of char[] allocations) is that it simply "latches on" to the stream and keeps pulling bytes from the socket indefinitely.
What I would like is for the client to simply work normally and give me a HTTPResponse which I can use to pull what I want from the stream. As a matter of fact, I have been able to do so with other ShoutCast streams, but not this one.
Is there any way to work around this? Could I for example tell the client to break off after a certain number of bytes?

That site is very particular. If you don't specify a supported User-Agent (like Mozilla), the server keep streaming bytes. I don't know what these bytes are meant to represent, audio perhaps.
If you print out the bytes that you receive, you will see
ICY 200 OK
icy-notice1:<BR>This stream requires Winamp<BR>
icy-notice2:SHOUTcast Distributed Network Audio Server/Linux v1.9.8<BR>
icy-name:181.FM - The Beatles Channel
icy-genre:Oldies
icy-url:http://www.181.fm
content-type:audio/mpeg
icy-pub:1
icy-br:128
which indicates that the response is not a valid HTTP response. It is an ICY response from the ICY protocol.
Now the default HttpClient you are using uses a DefaultHttpResponseParser which is a
Lenient HTTP response parser implementation that can skip malformed
data until a valid HTTP response message head is encountered.
In other words, it keeps reading the bytes the server is sending until it finds a valid HTTP response header, which will never happen, thus the infinite read.
I don't think you will be able to accomplish what you want with the Http Components library. Either look for an ICY client implementation in Java or spin your own.

Polling data from restful webservice with "keepalive" connection

i'm developing an android application that fetches data from a restful web service.
i was thinking about the best way to implement this, my code is working but i'm sure it can be optimized.
here's my current code state:
i'm calling the restResource from a service using a scheduler.
the bad part is opening and closing connection on every request.
(the rest resource keeps realtime data( i have to fetch it every second), when it's empty it returns an empty array)
httpResponse = client.execute(request);
responseCode = httpResponse.getStatusLine().getStatusCode();
message = httpResponse.getStatusLine().getReasonPhrase();
HttpEntity entity = httpResponse.getEntity();
if (entity != null) {
InputStream instream = entity.getContent();
response = convertStreamToString(instream);
instream.close();
}
how to make use of the keepalive header returned with the response?

I have done this before using long-polling, in which my RESTful call blocks until either there is a timeout or it has something to reply with. Either way, the calling client would re-call the RESTful service again on either response (timeout or valid).
I am not sure how to you could use the keepalive header, but it seems like what you would do is send it, and then simply keep the connection open. In your server-side java code, you would get the output buffer for the response and start sending stuff down. On the client side, get the input buffer (not sure how this happens in Android, but I am under the impression its Java code, so there are loads of ways you can do any of this) and start reading. At some point I would think you would need to force the connection close, just because you don't want to keep it open indefinately.
Again, I have never tried it this way, so I don't know. Also, I am not sure if this breaks some unspoken laws of RESTful services. I have never read anything to contradict this approach.

How to use HttpClient to download continous data stream?

I'm currently using HttpURLConnection to stream live content such as a radio broadcast. However it seems that using HttpClient is a better option since it's well supported by Android and it's a better implementation. Also, there seems to be a logic for automatic reconnection from a lost connection.
My problem is that I can't get this to work. It's always hanging when calling httpclient.execute(...).
What am I doing wrong?
HttpClient httpclient = new DefaultHttpClient();
HttpGet httpget = new HttpGet("http://208.76.243.123:7100");
HttpResponse response = httpclient.execute(httpget);
HttpEntity entity = response.getEntity();

Run it in debugger and when it hangs, call break. Then find the thread that is executing your code and see in stack trace where exactly it blocked. You will see if it blocked on IO or something else is happening. With that data it will be easier to identify the problem.

Are you sure your server understands the HTTP protocol? (I assume yes, it sounds like you had a different client working). It is possible the execute method is blocking because it has not seen a valid Response header yet.
You probably want entity.getContent() which will return a handle to a stream. See this question.

Why does HttpClient throw a SocketTimeOutException when executing POST

I have code similar to the following:
try {
HttpPost post = new HttpPost(httpsUrl);
setHeaders(post);
HttpEntity entity = new StringEntity(request, "UTF-8");
post.setEntity(entity);
HttpResponse response = httpclient.execute(post);
String result = EntityReader.readContent(response.getEntity());
checkAnswer(result);
return result;
} catch (Exception e) {
throw new ZapException("Error executing the http post request: "+e.getMessage(), e);
}
It sends the content of request to a server via POST using a httpclient instance that might have already been used before (it has persistent connections turned on, since we're sending quite some requests to the same server...).
This sometimes fails with a SocketTimeoutException with "Read timed out" as the message.
It's not clear to us, why it only fails at some times, when most times it doesn't. What gives?

In the following, I assume you are using Apache Commons HttpClient (org.apache.commons.httpclient.HttpClient).
Maybe you get thrown a SocketTimeoutException simply because, occasionally, the host your HttpClient instance is communicating with takes too long to respond, triggering HttpClient's cancellation routine.
You can increase the connection timeout and the socket timeout with the following
HttpConnectionParams params = httpclient.getHttpConnectionManager().getParams();
params.setConnectionTimeout(20000);
params.setSoTimeout(15000);
Aditionally, if you still face timeouts despite increasing the timeout limits, it is a good practice to handle the SocketTimeoutException gracefully - for example by retrying the connection a second and third time.