I'm using an Apache DefaultHttpClient with a PoolingClientConnectionManager and BasicResponseHandler. These are shared between different threads, and each thread creates its own HttpRequestBase extension.
Do I need to manually tell the manager that I'm done using the connection when using BasicResponseHandlers? Do I need to wrap it in a finally so exceptions don't cause a connection leak?
In other words, do I need to do this
HttpGet get = new HttpGet(address);
try {
httpclient.execute(get, new BasicResponseHandler());
} finally {
get.reset();
}
or is this enough ?
HttpGet get = new HttpGet(address);
httpclient.execute(get, new BasicResponseHandler());
I didn't see a clear answer in the Apache documentation.
This is enough and is recommended.
HttpClient#execute methods are guaranteed to automatically release all resources associated with the request in case of an exception (either I/O or runtime). When an HTTP response is processed using a ResponseHandler the underlying connection gets automatically released back to the connection manager is all cases.
Related
I have a webservice which is accepting a POST method with XML. It is working fine then at some random occasion, it fails to communicate to the server throwing IOException with message The target server failed to respond. The subsequent calls work fine.
It happens mostly, when i make some calls and then leave my application idle for like 10-15 min. the first call which I make after that returns this error.
I tried couple of things ...
I setup the retry handler like
HttpRequestRetryHandler retryHandler = new HttpRequestRetryHandler() {
public boolean retryRequest(IOException e, int retryCount, HttpContext httpCtx) {
if (retryCount >= 3){
Logger.warn(CALLER, "Maximum tries reached, exception would be thrown to outer block");
return false;
}
if (e instanceof org.apache.http.NoHttpResponseException){
Logger.warn(CALLER, "No response from server on "+retryCount+" call");
return true;
}
return false;
}
};
httpPost.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, retryHandler);
but this retry never got called. (yes I am using right instanceof clause). While debugging this class never being called.
I even tried setting up HttpProtocolParams.setUseExpectContinue(httpClient.getParams(), false); but no use. Can someone suggest what I can do now?
IMPORTANT
Besides figuring out why I am getting the exception, one of the important concerns I have is why isn't the retryhandler working here?
Most likely persistent connections that are kept alive by the connection manager become stale. That is, the target server shuts down the connection on its end without HttpClient being able to react to that event, while the connection is being idle, thus rendering the connection half-closed or 'stale'. Usually this is not a problem. HttpClient employs several techniques to verify connection validity upon its lease from the pool. Even if the stale connection check is disabled and a stale connection is used to transmit a request message the request execution usually fails in the write operation with SocketException and gets automatically retried. However under some circumstances the write operation can terminate without an exception and the subsequent read operation returns -1 (end of stream). In this case HttpClient has no other choice but to assume the request succeeded but the server failed to respond most likely due to an unexpected error on the server side.
The simplest way to remedy the situation is to evict expired connections and connections that have been idle longer than, say, 1 minute from the pool after a period of inactivity. For details please see the 2.5. Connection eviction policy of the HttpClient 4.5 tutorial.
Accepted answer is right but lacks solution. To avoid this error, you can add setHttpRequestRetryHandler (or setRetryHandler for apache components 4.4) for your HTTP client like in this answer.
HttpClient 4.4 suffered from a bug in this area relating to validating possibly stale connections before returning to the requestor. It didn't validate whether a connection was stale, and this then results in an immediate NoHttpResponseException.
This issue was resolved in HttpClient 4.4.1. See this JIRA and the release notes
Solution: change the ReuseStrategy to never
Since this problem is very complex and there are so many different factors which can fail I was happy to find this solution in another post: How to solve org.apache.http.NoHttpResponseException
Never reuse connections:
configure in org.apache.http.impl.client.AbstractHttpClient:
httpClient.setReuseStrategy(new NoConnectionReuseStrategy());
The same can be configured on a org.apache.http.impl.client.HttpClientBuilder builder:
builder.setConnectionReuseStrategy(new NoConnectionReuseStrategy());
Although accepted answer is right, but IMHO is just a workaround.
To be clear: it's a perfectly normal situation that a persistent connection may become stale. But unfortunately it's very bad when the HTTP client library cannot handle it properly.
Since this faulty behavior in Apache HttpClient was not fixed for many years, I definitely would prefer to switch to a library that can easily recover from a stale connection problem, e.g. OkHttp.
Why?
OkHttp pools http connections by default.
It gracefully recovers from situations when http connection becomes stale and request cannot be retried due to being not idempotent (e.g. POST). I cannot say it about Apache HttpClient (mentioned NoHttpResponseException).
Supports HTTP/2.0 from early drafts and beta versions.
When I switched to OkHttp, my problems with NoHttpResponseException disappeared forever.
Nowadays, most HTTP connections are considered persistent unless declared otherwise. However, to save server ressources the connection is rarely kept open forever, the default connection timeout for many servers is rather short, for example 5 seconds for the Apache httpd 2.2 and above.
The org.apache.http.NoHttpResponseException error comes most likely from one persistent connection that was closed by the server.
It's possible to set the maximum time to keep unused connections open in the Apache Http client pool, in milliseconds.
With Spring Boot, one way to achieve this:
public class RestTemplateCustomizers {
static public class MaxConnectionTimeCustomizer implements RestTemplateCustomizer {
#Override
public void customize(RestTemplate restTemplate) {
HttpClient httpClient = HttpClientBuilder
.create()
.setConnectionTimeToLive(1000, TimeUnit.MILLISECONDS)
.build();
restTemplate.setRequestFactory(
new HttpComponentsClientHttpRequestFactory(httpClient));
}
}
}
// In your service that uses a RestTemplate
public MyRestService(RestTemplateBuilder builder ) {
restTemplate = builder
.customizers(new RestTemplateCustomizers.MaxConnectionTimeCustomizer())
.build();
}
This can happen if disableContentCompression() is set on a pooling manager assigned to your HttpClient, and the target server is trying to use gzip compression.
Same problem for me on apache http client 4.5.5
adding default header
Connection: close
resolve the problem
Use PoolingHttpClientConnectionManager instead of BasicHttpClientConnectionManager
BasicHttpClientConnectionManager will make an effort to reuse the connection for subsequent requests with the same route. It will, however, close the existing connection and re-open it for the given route.
I have faced same issue, I resolved by adding "connection: close" as extention,
Step 1: create a new class ConnectionCloseExtension
import com.github.tomakehurst.wiremock.common.FileSource;
import com.github.tomakehurst.wiremock.extension.Parameters;
import com.github.tomakehurst.wiremock.extension.ResponseTransformer;
import com.github.tomakehurst.wiremock.http.HttpHeader;
import com.github.tomakehurst.wiremock.http.HttpHeaders;
import com.github.tomakehurst.wiremock.http.Request;
import com.github.tomakehurst.wiremock.http.Response;
public class ConnectionCloseExtension extends ResponseTransformer {
#Override
public Response transform(Request request, Response response, FileSource files, Parameters parameters) {
return Response.Builder
.like(response)
.headers(HttpHeaders.copyOf(response.getHeaders())
.plus(new HttpHeader("Connection", "Close")))
.build();
}
#Override
public String getName() {
return "ConnectionCloseExtension";
}
}
Step 2: set extension class in wireMockServer like below,
final WireMockServer wireMockServer = new WireMockServer(options()
.extensions(ConnectionCloseExtension.class)
.port(httpPort));
I'm developing a long-running application that heavily uses the HttpClient from apache.
On my first test run, the application worked perfectly until it just got stuck. It wasn't stopped, it didn't throw any exception, it just sits there doing nothing.
I did a second run just now and stopped the time and it stopped after approx. 24 hours of contant running. Additionally I noticed that the internet connection of my laptop on which I had it running was terminated at the exact moment the application got stuck. I had to reboot my WLAN adapter in order to the the net running again.
The application though, didn't return to working after the connection was up again. And now, it's stuck again.
Is there any timeout controller I'm not aware of in the HttpClient? Why doesn't my application throw an exception when the connection is down?
The part that uses the client looks as follows;
public HttpUtil(ConfigUtil config) {
this.config = config;
client = new DefaultHttpClient();
client.getParams().setParameter(HttpProtocolParams.USER_AGENT, this.config.getProperty("httputil.userAgent"));
}
public String getContentAsString(String url) throws ParseException, ClientProtocolException, IOException {
return EntityUtils.toString(
client.execute(
new HttpGet(url)).getEntity());
}
The application repeatedly calls httputil.getContentAsString() on the URLs it needs.
This code is now deprecated (get HttpParams, etc). A better way is:
RequestConfig defaultRequestConfig = RequestConfig.custom()
.setCookieSpec(CookieSpecs.BEST_MATCH)
.setExpectContinueEnabled(true)
.setStaleConnectionCheckEnabled(true)
.setTargetPreferredAuthSchemes(Arrays.asList(AuthSchemes.NTLM, AuthSchemes.DIGEST))
.setProxyPreferredAuthSchemes(Arrays.asList(AuthSchemes.BASIC))
.build();
HttpGet httpGet = new HttpGet(url);
RequestConfig requestConfig = RequestConfig.copy(defaultRequestConfig)
.setSocketTimeout(5000)
.setConnectTimeout(5000)
.setConnectionRequestTimeout(5000)
.build();
httpGet.setConfig(requestConfig);
As of version 4.4, both answers by users user2393012 and Stephen C have been deprecated. I'm not sure if there is another way of doing it, but the way I do it is by using a builder paradigm, HTTPClientBuilder.
Ex.
HttpClients.custom().setConnectionTimeToLive(1, TimeUnit.MINUTES).build()
A very similar (it actually might have been OP's problem) problem to what OP mentioned also happens but is due to Apache setting the default concurrent connections to only two connections per client. The solution to this would be to increase the max connections or close them if you can.
To increase the max connections:
HttpClients.custom().setMaxConnPerRoute(100000).build()
To close connections, you can use a BasicHttpClientConnectionManager and call on the close method for
I gave a similar answer in another thread (HttpClient hangs on socketRead0 with successfully executed method)
In my case, I was setting the connectionTimeout and socketTimeout on the request, but not on the connection socket used during the establishment of the SSL connection. As a result, I would sometime hang during the SSL handshake. Below is some code that sets all 3 timeouts using the v4.4 (also tested in v4.5)
// Configure the socket timeout for the connection, incl. ssl tunneling
connManager = new PoolingHttpClientConnectionManager();
connManager.setMaxTotal(200);
connManager.setDefaultMaxPerRoute(100);
SocketConfig sc = SocketConfig.custom()
.setSoTimeout(soTimeoutMs)
.build();
connManager.setDefaultSocketConfig(sc);
HttpClient client = HttpClients.custom()
.setConnectionManager(connManager)
.setConnectionManagerShared(true)
.build();
// configure the timeouts (socket and connection) for the request
RequestConfig.Builder config = = RequestConfig.copy(RequestConfig.DEFAULT);
config.setConnectionRequestTimeout(connectionTimeoutMs);
config.setSocketTimeout(socketTimeoutMs);
HttpRequestBase req = new HttpGet(uri);
req.setConfig(config.build());
client.execute(req);
You haven't said which version of HttpClient you are using, but assuming that it is version 4, this blog article explains what to do.
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpParams params = httpClient.getParams();
HttpConnectionParams.setConnectionTimeout(httpParams, connectionTimeoutMillis);
HttpConnectionParams.setSoTimeout(httpParams, socketTimeoutMillis);
I have all the timeouts setup just fine but I found out we have on url that does http chunking but sends no results(works fine in chrome, but in http client it hangs forever even with the timeout set). Luckily I own the server and just return some garbage and it no longer hangs. This seems like a very unique bug in that http client does not handle some kind of empty chunking case well(though I could be way off)....I just know it hangs every time on that same url with empty data and that url is http chunking csv download back to our http client.
I had the same issue, it stuck cause I didn't close DefaultHttpClient.
So this is wrong:
try{
DefaultHttpClient httpclient = new DefaultHttpClient();
...
} catch (Exception e){
e.PrintStackTrace();
}
And this is right:
try (DefaultHttpClient httpclient = new DefaultHttpClient()){
...
} catch (Exception e){
e.PrintStackTrace();
}
Hope it helps someone.
By default, HttpClient does not timeout (which causes more problem than it helps). What you are describing could be a hardware issue, if your network adapter died, the HttpClient will hang.
Here are the parameters set to HttpParams as part of the constructor to DefaultHttpClient including
http.socket.timeout: defines the socket timeout (SO_TIMEOUT) in
milliseconds, which is the timeout for waiting for data or, put
differently, a maximum period inactivity between two consecutive data
packets). A timeout value of zero is interpreted as an infinite
timeout. This parameter expects a value of type java.lang.Integer. If
this parameter is not set, read operations will not time out (infinite
timeout).
That will set a timeout on the connection, so after the set timeout an exception will be thrown.
I use Apache http client 4.1 and always close the response InputStream (e.g. by calling EntityUtils.consume(response.getEntity)) after the HttpGet execution.
Now I wonder whether the connection is properly released if the HttpGet execution throws an exception. Should I release the connection explicitly in this case?
Always close I/O objects in the finally clause if you acquired them.
InputStrem input = acquireStream( );
try
{
useStream( input );
}
finally
{
input.close( );
}
SimpleHttpConnectionManager being used incorrectly. Be sure that HttpMethod.releaseConnection() is always called and that only one thread and/or method is using this connection manager at a time.
Does Anyone know why this error shows up and is causes the files I want to download or to fail and retry or to download uncompleted
Thank you !
Make sure that you don't use SimpleHttpConnectionManager to create and use connections from multiple threads. The simple connection manager is not designed for it - it returns always the same connection, and this is not thread safe.
In a multi-threaded environment, use a different manager that uses a pool of connections. See MultiThreadedHttpConnectionManager.
Prefer to take no credit for this, but as per Eyal Schneider's answer, find more info on using MultiThreadedHttpConnectionManager in Vincent de Villers excellent blog.
Code snippet copied in case the link ever disappears:
HttpClient httpclient = new HttpClient(new MultiThreadedHttpConnectionManager());
GetMethod httpget = new GetMethod("http://www.myhost.com/");
try {
httpclient.executeMethod(httpget);
Reader reader = new InputStreamReader(
httpget.getResponseBodyAsStream(), httpget.getResponseCharSet());
// consume the response entity
} finally {
httpget.releaseConnection();
}
I haven't been able to find an adequate answer to what exactly the following error means:
java.net.SocketException: Software caused connection abort: recv failed
Notes:
This error is infrequent and unpredictable; although getting this error means that all future requests for URIs will also fail.
The only solution that works (also, only occasionally) is to reboot Tomcat and/or the actual machine (Windows in this case).
The URI is definitely available (as confirmed by asking the browser to do the fetch).
Relevant code:
BufferedReader reader;
try {
URL url = new URL(URI);
reader = new BufferedReader(new InputStreamReader(url.openStream())));
} catch( MalformedURLException e ) {
throw new IOException("Expecting a well-formed URL: " + e);
}//end try: Have a stream
String buffer;
StringBuilder result = new StringBuilder();
while( null != (buffer = reader.readLine()) ) {
result.append(buffer);
}//end while: Got the contents.
reader.close();
This also happens if your TLS client is unable to be authenticate by the server configured to require client authentication.
This usually means that there was a network error, such as a TCP timeout. I would start by placing a sniffer (wireshark) on the connection to see if you can see any problems. If there is a TCP error, you should be able to see it. Also, you can check your router logs, if this is applicable. If wireless is involved anywhere, that is another source for these kind of errors.
This error occurs when a connection is closed abruptly (when a TCP connection is reset while there is still data in the send buffer). The condition is very similar to a much more common 'Connection reset by peer'. It can happen sporadically when connecting over the Internet, but also systematically if the timing is right (e.g. with keep-alive connections on localhost).
An HTTP client should just re-open the connection and retry the request. It is important to understand that when a connection is in this state, there is no way out of it other than to close it. Any attempt to send or receive will produce the same error.
Don't use URL.open(), use Apache-Commons HttpClient which has a retry mechanism, connection pooling, keep-alive and many other features.
Sample usage:
HttpClient httpClient = HttpClients.custom()
.setConnectionTimeToLive(20, TimeUnit.SECONDS)
.setMaxConnTotal(400).setMaxConnPerRoute(400)
.setDefaultRequestConfig(RequestConfig.custom()
.setSocketTimeout(30000).setConnectTimeout(5000).build())
.setRetryHandler(new DefaultHttpRequestRetryHandler(5, true))
.build();
// the httpClient should be re-used because it is pooled and thread-safe.
HttpGet request = new HttpGet(uri);
HttpResponse response = httpClient.execute(request);
reader = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
// handle response ...
Are you accessing http data? Can you use the HttpClient library instead of the standard library? The library has more options and will provide better error messages.
http://hc.apache.org/httpclient-3.x/
The only time I've seen something like this happen is when I have a bad connection, or when somebody is closing the socket that I am using from a different thread context.
Try adding 'autoReconnect=true' to the jdbc connection string
This will happen from time to time either when a connection times out or when a remote host terminates their connection (closed application, computer shutdown, etc). You can avoid this by managing sockets yourself and handling disconnections in your application via its communications protocol and then calling shutdownInput and shutdownOutput to clear up the session.
Look if you have another service or program running on the http port. It happened to me when I tried to use the port and it was taken by another program.
If you are using Netbeans to manage Tomcat, try to disable HTTP monitor in Tools - Servers
I too had this problem. My solution was:
sc.setSoLinger(true, 10);
COPY FROM A WEBSITE -->By using the setSoLinger() method, you can explicitly set a delay before a reset is sent, giving more time for data to be read or send.
Maybe it is not the answer to everybody but to some people.