We Keep Coding

Jersey: Close I/O resources after HTTP response

My Setup: I have created a REST service (Jersey/Dropwizard) that streams large content from a database. During a GET operation, the service taps into the database through a connection pool, wraps the data into a stream and performs some on-the-fly transformation to render the requested data in various encodings (CSV, JSON, ...). The life time of the database connection is tied to the life time of the stream and only when the stream is closed, the database connection is released.
The stream transformation is performed by an Encoder class that returns a StreamingOutput which is then passed to the Response object. The Encoder currently handles resource closing when the stream is fully consumed.
My Problem: Since StreamingOutput does not implement AutoCloseable, connection leaks may occur when the output is only partially consumed.
I sometimes observe that stale active connections are piling up in the connection pool, and I suspect that they arise from aborted HTTP connections. As you can see below, the current code handles exceptions that occur in the try block. What I cannot handle are Exceptions that occur after the return statement and I don't know how to attach any instructions for resource closing to the Response object.
My Question: How can I inform the Response object to close particular resources after the request has terminated (regularly or due to an error)? Or: Is there a better way to safely close any associated resources when the request context ends?
#GET
//#Produces(...)
public Response streamData(
#PathParam("key") String key,
// ... other params
) {
//decode and validate params
Stream<Pojo> ps = null;
try {
// connect to db and obtain data stream for <key>
ps = loadData(db, key);
// apply detailed encoding instrunctions and create a StreamingOutput
final StreamingOutput stream = Encoder.encodeData(ps, encodingArgs);
return Response.ok(stream).build();
} catch (Exception e) {
closeOnException(ps); // wrapper for ps.close();
throw e;
}
}

I received a good answer from the dropwizard mailing list that solves my problem and I want to reference here it in case somebody encounters the same problem.
https://groups.google.com/forum/#!topic/dropwizard-user/62GoLDBrQuo
Citing from Shawn's response:
Jersey supports CloseableService that lets you register Closeable objects to be closed when the request is complete:
public Response streamData(..., #Context CloseableService closer) {
...
closer.add(closeable);
return Response.ok(...).build();
}

You can add to your method HttpServletResponse:
#Produces(MediaType.APPLICATION_OCTET_STREAM)
public Object streamData(
#PathParam("key") String key,
#Context HttpServletResponse response,
// ... other params
) {
...
response.getOutputStream().write(....)
response.flushBuffer();
response.getOutputStream().close();
return null;
}

Releasing connections when using PoolingClientConnectionManager?

I'm using an Apache DefaultHttpClient with a PoolingClientConnectionManager and BasicResponseHandler. These are shared between different threads, and each thread creates its own HttpRequestBase extension.
Do I need to manually tell the manager that I'm done using the connection when using BasicResponseHandlers? Do I need to wrap it in a finally so exceptions don't cause a connection leak?
In other words, do I need to do this
HttpGet get = new HttpGet(address);
try {
httpclient.execute(get, new BasicResponseHandler());
} finally {
get.reset();
}
or is this enough ?
HttpGet get = new HttpGet(address);
httpclient.execute(get, new BasicResponseHandler());
I didn't see a clear answer in the Apache documentation.

This is enough and is recommended.
HttpClient#execute methods are guaranteed to automatically release all resources associated with the request in case of an exception (either I/O or runtime). When an HTTP response is processed using a ResponseHandler the underlying connection gets automatically released back to the connection manager is all cases.

HttpClient stuck without any exception

I'm developing a long-running application that heavily uses the HttpClient from apache.
On my first test run, the application worked perfectly until it just got stuck. It wasn't stopped, it didn't throw any exception, it just sits there doing nothing.
I did a second run just now and stopped the time and it stopped after approx. 24 hours of contant running. Additionally I noticed that the internet connection of my laptop on which I had it running was terminated at the exact moment the application got stuck. I had to reboot my WLAN adapter in order to the the net running again.
The application though, didn't return to working after the connection was up again. And now, it's stuck again.
Is there any timeout controller I'm not aware of in the HttpClient? Why doesn't my application throw an exception when the connection is down?
The part that uses the client looks as follows;
public HttpUtil(ConfigUtil config) {
this.config = config;
client = new DefaultHttpClient();
client.getParams().setParameter(HttpProtocolParams.USER_AGENT, this.config.getProperty("httputil.userAgent"));
}
public String getContentAsString(String url) throws ParseException, ClientProtocolException, IOException {
return EntityUtils.toString(
client.execute(
new HttpGet(url)).getEntity());
}
The application repeatedly calls httputil.getContentAsString() on the URLs it needs.

This code is now deprecated (get HttpParams, etc). A better way is:
RequestConfig defaultRequestConfig = RequestConfig.custom()
.setCookieSpec(CookieSpecs.BEST_MATCH)
.setExpectContinueEnabled(true)
.setStaleConnectionCheckEnabled(true)
.setTargetPreferredAuthSchemes(Arrays.asList(AuthSchemes.NTLM, AuthSchemes.DIGEST))
.setProxyPreferredAuthSchemes(Arrays.asList(AuthSchemes.BASIC))
.build();
HttpGet httpGet = new HttpGet(url);
RequestConfig requestConfig = RequestConfig.copy(defaultRequestConfig)
.setSocketTimeout(5000)
.setConnectTimeout(5000)
.setConnectionRequestTimeout(5000)
.build();
httpGet.setConfig(requestConfig);

As of version 4.4, both answers by users user2393012 and Stephen C have been deprecated. I'm not sure if there is another way of doing it, but the way I do it is by using a builder paradigm, HTTPClientBuilder.
Ex.
HttpClients.custom().setConnectionTimeToLive(1, TimeUnit.MINUTES).build()
A very similar (it actually might have been OP's problem) problem to what OP mentioned also happens but is due to Apache setting the default concurrent connections to only two connections per client. The solution to this would be to increase the max connections or close them if you can.
To increase the max connections:
HttpClients.custom().setMaxConnPerRoute(100000).build()
To close connections, you can use a BasicHttpClientConnectionManager and call on the close method for

I gave a similar answer in another thread (HttpClient hangs on socketRead0 with successfully executed method)
In my case, I was setting the connectionTimeout and socketTimeout on the request, but not on the connection socket used during the establishment of the SSL connection. As a result, I would sometime hang during the SSL handshake. Below is some code that sets all 3 timeouts using the v4.4 (also tested in v4.5)
// Configure the socket timeout for the connection, incl. ssl tunneling
connManager = new PoolingHttpClientConnectionManager();
connManager.setMaxTotal(200);
connManager.setDefaultMaxPerRoute(100);
SocketConfig sc = SocketConfig.custom()
.setSoTimeout(soTimeoutMs)
.build();
connManager.setDefaultSocketConfig(sc);
HttpClient client = HttpClients.custom()
.setConnectionManager(connManager)
.setConnectionManagerShared(true)
.build();
// configure the timeouts (socket and connection) for the request
RequestConfig.Builder config = = RequestConfig.copy(RequestConfig.DEFAULT);
config.setConnectionRequestTimeout(connectionTimeoutMs);
config.setSocketTimeout(socketTimeoutMs);
HttpRequestBase req = new HttpGet(uri);
req.setConfig(config.build());
client.execute(req);

You haven't said which version of HttpClient you are using, but assuming that it is version 4, this blog article explains what to do.
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpParams params = httpClient.getParams();
HttpConnectionParams.setConnectionTimeout(httpParams, connectionTimeoutMillis);
HttpConnectionParams.setSoTimeout(httpParams, socketTimeoutMillis);

I have all the timeouts setup just fine but I found out we have on url that does http chunking but sends no results(works fine in chrome, but in http client it hangs forever even with the timeout set). Luckily I own the server and just return some garbage and it no longer hangs. This seems like a very unique bug in that http client does not handle some kind of empty chunking case well(though I could be way off)....I just know it hangs every time on that same url with empty data and that url is http chunking csv download back to our http client.

I had the same issue, it stuck cause I didn't close DefaultHttpClient.
So this is wrong:
try{
DefaultHttpClient httpclient = new DefaultHttpClient();
...
} catch (Exception e){
e.PrintStackTrace();
}
And this is right:
try (DefaultHttpClient httpclient = new DefaultHttpClient()){
...
} catch (Exception e){
e.PrintStackTrace();
}
Hope it helps someone.

By default, HttpClient does not timeout (which causes more problem than it helps). What you are describing could be a hardware issue, if your network adapter died, the HttpClient will hang.
Here are the parameters set to HttpParams as part of the constructor to DefaultHttpClient including
http.socket.timeout: defines the socket timeout (SO_TIMEOUT) in
milliseconds, which is the timeout for waiting for data or, put
differently, a maximum period inactivity between two consecutive data
packets). A timeout value of zero is interpreted as an infinite
timeout. This parameter expects a value of type java.lang.Integer. If
this parameter is not set, read operations will not time out (infinite
timeout).
That will set a timeout on the connection, so after the set timeout an exception will be thrown.

Safe use of HttpURLConnection

When using HttpURLConnection does the InputStream need to be closed if we do not 'get' and use it?
i.e. is this safe?
HttpURLConnection conn = (HttpURLConnection) uri.getURI().toURL().openConnection();
conn.connect();
// check for content type I don't care about
if (conn.getContentType.equals("image/gif") return;
// get stream and read from it
InputStream is = conn.getInputStream();
try {
// read from is
} finally {
is.close();
}
Secondly, is it safe to close an InputStream before all of it's content has been fully read?
Is there a risk of leaving the underlying socket in ESTABLISHED or even CLOSE_WAIT state?

According to http://docs.oracle.com/javase/6/docs/technotes/guides/net/http-keepalive.html
and OpenJDK source code.
(When keepAlive == true)
If client called HttpURLConnection.getInputSteam().close(), the later call to HttpURLConnection.disconnect() will NOT close the Socket. i.e. The Socket is reused (cached)
If client does not call close(), call disconnect() will close the InputStream and close the Socket.
So in order to reuse the Socket, just call InputStream.close(). Do not call HttpURLConnection.disconnect().

is it safe to close an InputStream
before all of it's content has been
read
You need to read all of the data in the input stream before you close it so that the underlying TCP connection gets cached. I have read that it should not be required in latest Java, but it was always mandated to read the whole response for connection re-use.
Check this post: keep-alive in java6

Here is some information regarding the keep-alive cache. All of this information pertains Java 6, but is probably also accurate for many prior and later versions.
From what I can tell, the code boils down to:
If the remote server sends a "Keep-Alive" header with a "timeout" value that can be parsed as a positive integer, that number of seconds is used for the timeout.
If the remote server sends a "Keep-Alive" header but it doesn't have a "timeout" value that can be parsed as a positive integer and "usingProxy" is true, then the timeout is 60 seconds.
In all other cases, the timeout is 5 seconds.
This logic is split between two places: around line 725 of sun.net.www.http.HttpClient (in the "parseHTTPHeader" method), and around line 120 of sun.net.www.http.KeepAliveCache (in the "put" method).
So, there are two ways to control the timeout period:
Control the remote server and configure it to send a Keep-Alive header with the proper timeout field
Modify the JDK source code and build your own.
One would think that it would be possible to change the apparently arbitrary five-second default without recompiling internal JDK classes, but it isn't. A bug was filed in 2005 requesting this ability, but Sun refused to provide it.

If you really want to make sure that the connection is close you should call conn.disconnect().
The open connections you observed are because of the HTTP 1.1 connection keep alive feature (also known as HTTP Persistent Connections).
If the server supports HTTP 1.1 and does not send a Connection: close in the response header Java does not immediately close the underlaying TCP connection when you close the input stream. Instead it keeps it open and tries to reuse it for the next HTTP request to the same server.
If you don't want this behaviour at all you can set the system property http.keepAlive to false:
System.setProperty("http.keepAlive","false");

When using HttpURLConnection does the InputStream need to be closed if we do not 'get' and use it?
Yes, it always needs to be closed.
i.e. is this safe?
Not 100%, you run the risk of getting a NPE. Safer is:
InputStream is = null;
try {
is = conn.getInputStream()
// read from is
} finally {
if (is != null) {
is.close();
}
}

You also have to close error stream if the HTTP request fails (anything but 200):
try {
...
}
catch (IOException e) {
connection.getErrorStream().close();
}
If you don't do it, all requests that don't return 200 (e.g. timeout) will leak one socket.

Since Java 7 the recommended way is
try (InputStream is = conn.getInputStream()) {
// read from is
// ...
}
as for all other classes implementing Closable. close() is called at the end of the try {...} block.
Closing the input stream also means you are done with reading. Otherwise the connection hangs around until the finalizer closes the stream.
Same applies to the output stream, if you are sending data.
There is no need to get an close the ErrorStream. Even if it implements the InputStream interface: It's using the InputStream in combination with a buffer. Closing the InputStream is sufficient.

java.net.SocketException: Software caused connection abort: recv failed

I haven't been able to find an adequate answer to what exactly the following error means:
java.net.SocketException: Software caused connection abort: recv failed
Notes:
This error is infrequent and unpredictable; although getting this error means that all future requests for URIs will also fail.
The only solution that works (also, only occasionally) is to reboot Tomcat and/or the actual machine (Windows in this case).
The URI is definitely available (as confirmed by asking the browser to do the fetch).
Relevant code:
BufferedReader reader;
try {
URL url = new URL(URI);
reader = new BufferedReader(new InputStreamReader(url.openStream())));
} catch( MalformedURLException e ) {
throw new IOException("Expecting a well-formed URL: " + e);
}//end try: Have a stream
String buffer;
StringBuilder result = new StringBuilder();
while( null != (buffer = reader.readLine()) ) {
result.append(buffer);
}//end while: Got the contents.
reader.close();

This also happens if your TLS client is unable to be authenticate by the server configured to require client authentication.

This usually means that there was a network error, such as a TCP timeout. I would start by placing a sniffer (wireshark) on the connection to see if you can see any problems. If there is a TCP error, you should be able to see it. Also, you can check your router logs, if this is applicable. If wireless is involved anywhere, that is another source for these kind of errors.

This error occurs when a connection is closed abruptly (when a TCP connection is reset while there is still data in the send buffer). The condition is very similar to a much more common 'Connection reset by peer'. It can happen sporadically when connecting over the Internet, but also systematically if the timing is right (e.g. with keep-alive connections on localhost).
An HTTP client should just re-open the connection and retry the request. It is important to understand that when a connection is in this state, there is no way out of it other than to close it. Any attempt to send or receive will produce the same error.
Don't use URL.open(), use Apache-Commons HttpClient which has a retry mechanism, connection pooling, keep-alive and many other features.
Sample usage:
HttpClient httpClient = HttpClients.custom()
.setConnectionTimeToLive(20, TimeUnit.SECONDS)
.setMaxConnTotal(400).setMaxConnPerRoute(400)
.setDefaultRequestConfig(RequestConfig.custom()
.setSocketTimeout(30000).setConnectTimeout(5000).build())
.setRetryHandler(new DefaultHttpRequestRetryHandler(5, true))
.build();
// the httpClient should be re-used because it is pooled and thread-safe.
HttpGet request = new HttpGet(uri);
HttpResponse response = httpClient.execute(request);
reader = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
// handle response ...

Are you accessing http data? Can you use the HttpClient library instead of the standard library? The library has more options and will provide better error messages.
http://hc.apache.org/httpclient-3.x/

The only time I've seen something like this happen is when I have a bad connection, or when somebody is closing the socket that I am using from a different thread context.

Try adding 'autoReconnect=true' to the jdbc connection string

This will happen from time to time either when a connection times out or when a remote host terminates their connection (closed application, computer shutdown, etc). You can avoid this by managing sockets yourself and handling disconnections in your application via its communications protocol and then calling shutdownInput and shutdownOutput to clear up the session.

Look if you have another service or program running on the http port. It happened to me when I tried to use the port and it was taken by another program.

If you are using Netbeans to manage Tomcat, try to disable HTTP monitor in Tools - Servers

I too had this problem. My solution was:
sc.setSoLinger(true, 10);
COPY FROM A WEBSITE -->By using the setSoLinger() method, you can explicitly set a delay before a reset is sent, giving more time for data to be read or send.
Maybe it is not the answer to everybody but to some people.