I use lot of http connection with Apache HttpClient (org.apache.httpcomponents 4.3.6) to test servers and I cannot force connections to close, even when I schedule HttpConnectionManager to httpClient.getConnectionManager().shutdown(); after 10 seconds in another thread.
httpClient.close() also doesn't help.
Connections can last for minutes or even hours.
I have tried custom SocketConfig and this also not helps:
SocketConfig socketConfig = SocketConfig.custom()
.setSoKeepAlive(false)
.setSoLinger(5)
.setSoReuseAddress(true)
.setSoTimeout(5000)
.setTcpNoDelay(true).build();
The way I am fetching the content:
HttpUriRequest request = new HttpGet(url);
HttpResponse response = httpClient.execute(request);
try (InputStream in = response.getEntity().getContent()) {
String result = IOUtils.toString(in, StandardCharsets.UTF_8);
httpClient.close();
return result;
}
The way I am building HTTP Client:
SocketConfig socketConfig = SocketConfig.custom()
.setSoKeepAlive(false)
.setSoLinger(configuration.getInt("proxy.list.socket.so.linger"))
.setSoReuseAddress(true)
.setSoTimeout(configuration.getInt("proxy.list.socket.so.timeout"))
.setTcpNoDelay(true).build();
HttpClientBuilder builder = HttpClientBuilder.create();
builder.disableAutomaticRetries();
builder.disableContentCompression();
builder.disableCookieManagement();
builder.disableRedirectHandling();
builder.setConnectionReuseStrategy(new NoConnectionReuseStrategy());
builder.setDefaultSocketConfig(socketConfig);
One of my prototypes of doing shutdown:
shutdownExecutor.schedule(() -> {
httpClient.getConnectionManager().closeExpiredConnections();
httpClient.getConnectionManager().closeIdleConnections(1, TimeUnit.SECONDS);
httpClient.getConnectionManager().shutdown();
httpClient.notifyAll();
try {
httpClient.close();
} catch (IOException ex) {
}
}, configuration.getInt("proxy.test.forced.timeout.seconds"), TimeUnit.SECONDS);
String content = HttpContentFetcher.getAndCloseClient(url, httpClient);
RequestConfig has helped. Now it looks like all the connections are discarded in specified limits.
RequestConfig config= RequestConfig.custom()
.setCircularRedirectsAllowed(false)
.setConnectionRequestTimeout(4000)
.setConnectTimeout(4000)
.setMaxRedirects(0)
.setRedirectsEnabled(false)
.setSocketTimeout(4000)
.setStaleConnectionCheckEnabled(true).build();
request.setConfig(config);
Related
I am using following version of httpclient api to call the rest webservice.
httpcore4.4.9
httpclient 4.5.9
I am using the ExecutorService to execute the task where it is calling my following methods to get the data from rest API. ExecutorService execute about 30+ rest call in one transaction and i observed that, httpclient HUNG or SLOW for few of the rest call and it hit the performance issue. Can you please check if below i am using httpclient correct way ?
String output = null;
CloseableHttpClient httpClient = getHttpClients();
try (CloseableHttpResponse response = httpClient.execute(httpGet)) {
HttpEntity entity = response.getEntity();
output = EntityUtils.toString(entity);
EntityUtils.consume(response.getEntity());
} catch (Exception e) {
log.error("error during call to REST API " + httpGet.getURI() + " ", e);
} finally {
httpClient.close();
httpGet.releaseConnection();
httpGet.abort();
}
return output;
}
private CloseableHttpClient getHttpClients() {
long startTime = System.currentTimeMillis();
long endTime = 0;
int timeout = 12000;
RequestConfig.Builder requestBuilder = RequestConfig.custom();
requestBuilder.setConnectTimeout(timeout);
requestBuilder.setConnectionRequestTimeout(timeout);
SocketConfig socketConfig = SocketConfig.custom().setSoTimeout(timeout).build();
HttpClientBuilder builder = HttpClientBuilder.create();
builder.setDefaultRequestConfig(requestBuilder.build());
builder.disableContentCompression();
builder.setDefaultSocketConfig(socketConfig);
CloseableHttpClient httpClient = builder
.build();
endTime = System.currentTimeMillis();
log.debug("Total time took to build client = " + (endTime - startTime));
return httpClient;
} ```
As per my experience, this problem is occurring because of EntityUtils.consume(response.getEntity()) .
You could find what it actually does here.
Try removing and running your code to match the speed and decide if you want it or not.
Below is the code used by me:
CloseableHttpClient httpClient = HttpClients.createDefault();
try {
HttpGet request = new HttpGet(url);
CloseableHttpResponse response = httpClient.execute(request);
String responseString = EntityUtils.toString(response.getEntity());
} catch (Exception ex) {
log.error(ex.toString());
}
Try to adjust MaxTotal and MaxPerRoute variable in PoolingHttpClientConnectionManager class.
connectionManager.setMaxTotal(maxTotalConnections);
connectionManager.setDefaultMaxPerRoute(maxConnectionsPerRoute);
I create a CloseableHttpClient with intent to connect to our Tomcat to execute some services.
After the request i call close on the client but I still see that there a active Session in Tomcat.
Should not close invalidate the session?
RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(1000 * 60 * 2)
.setConnectTimeout(4000)
.build();
HttpClientBuilder cl = HttpClients.custom();
cl.setDefaultRequestConfig(requestConfig);
cl.setSSLSocketFactory(makeSSLConnectionSocketFactory());
HttpGet postMethod = new HttpGet(url);
try (CloseableHttpClient client = cl.build();
CloseableHttpResponse httpResponse = client.execute(postMethod);) {
} catch (IOException e) {
//e.printStackTrace();
}
In my Xpages application I am calling an external service to collect data.
Users are complaining that they sometimes get a timeout error message:
Connect to customerbank.acme.se:20543 [customerbank.acme.se/127.17.27.172] failed: Connection timed out: connect
I assumed the timeout would result in an IOException but apparently not. How can I catch this error?
Below is part of my code. The logic of handling the response I have left out.
private CloseableHttpClient httpclient;
try{
HttpClientBuilder cb = HttpClientBuilder.create();
RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(30 * 1000)
.setConnectTimeout(30 * 1000)
.setConnectionRequestTimeout(30 * 1000)
.build();
cb.setDefaultRequestConfig(requestConfig);
httpclient = cb.build();
HttpPost httpPost = new HttpPost(urlFromConfiguration);
httpPost.setHeader("Content-Type", "application/json");
HttpEntity entity;
entity = new ByteArrayEntity(JSONobj.toString().getBytes("UTF-8"));
httpPost.setEntity(entity);
CloseableHttpResponse response = httpclient.execute(httpPost);
if (200 == response.getStatusLine().getStatusCode()){//response received
//perform some logic with the response...
}
} catch (IOException e) {
OpenLogUtil.logError(e);
FacesContext.getCurrentInstance().addMessage(null, new javax.faces.application.FacesMessage(javax.faces.application.FacesMessage.SEVERITY_ERROR, "some IO exception occurred", ""));
} catch (Exception e) {
OpenLogUtil.logError(e);
FacesContext.getCurrentInstance().addMessage(null, new javax.faces.application.FacesMessage(javax.faces.application.FacesMessage.SEVERITY_ERROR, "some general error has occured" , ""));
}
I think this Baeldung page can help you:
"Note that the connection timeout will result in an
org.apache.http.conn.ConnectTimeoutException being thrown, while
socket timeout will result in a java.net.SocketTimeoutException."
Apache Http client that you are using is a great utility. But it could be a bit heavy and cumbersome for a relatively simple task that you are running. There is a much simpler Http client provided in MgntUtils Open source library (written by me). It may be not as comprehensive as Apache one, but is much simpler in use. It does throw IOException upon connection or time-out error. In your case it could be an alternative to use. Take a look at Javadoc. Library itself provided as Maven artifacts and on Git (including source code and Javadoc). All in all your code may look like this:
private static void testHttpClient() {
HttpClient client = new HttpClient();
client.setContentType("application/json");
String content = null;
try {
content = client.sendHttpRequest("http://yourUrl.com", HttpMethod.POST, JSONobj.toString());
//content holds the response. Do your logic here
} catch (IOException e) {
//Error Handling is here
content = TextUtils.getStacktrace(e, false);
}
}
I have given the connectiontimeout 5000 milliseconds(5 second) but actually it takes 10127 milliseconds(10.127 second)
if connectiontimeout=10000 milliseconds(10 second) then it is taking 20032 milliseconds(20 second) for connection time out
below is the code which i tried.
public static void getTest()
{
long start=0;
try {
CloseableHttpClient httpClient = HttpClients.createDefault();
HttpGet httpGet = new HttpGet("http://testing url");
RequestConfig config=null;
config = RequestConfig.custom()
.setConnectTimeout(5000)
.setConnectionRequestTimeout(5000)
.setSocketTimeout(5000)
.build();
httpGet.setConfig(config);
start = System.currentTimeMillis();
httpClient.execute(httpGet);
} catch (Exception e) {
long end=System.currentTimeMillis();
System.out.println("total time in Milliseconds:="+(end-start));
}
}
The reason is unsuccessful HTTP POST request will be automatically re-sent to the server. Unsuccessful post means, in this case, the server did not send a valid HTTP response or an IOException occurred and HTTP PostRetry default value is true in JVM. There are servral ways to prevent from silent HttpRetry please refer below-mentioned table.
That's the main reason for doubled the exact timeout.
I have machine with 4 internet IP's and I want to know if I can make apache http client to make requests from specific ip/network interface
Using HttpClient 4.3 APIs
RequestConfig config = RequestConfig.custom()
.setLocalAddress(InetAddress.getByAddress(new byte[] {127,0,0,1}))
.build();
HttpGet httpGet = new HttpGet("/stuff");
httpGet.setConfig(config);
CloseableHttpClient httpClient = HttpClients.createDefault();
try {
CloseableHttpResponse response = httpClient.execute(httpGet);
try {
// do something useful
} finally {
response.close();
}
} finally {
httpClient.close();
}
Never did this, but there is a ClientConnectionOperator interface (and some factories too) in the API to create the socket. Maybe you can implement your own and create the socket with a concrete interface.