We Keep Coding

How to use RestTemplate in Spring Boot application in multithreaded environment?

I am having a problem in my Spring Boot application where I am using a RestTemplate which is autowired for making POST calls to a REST service. The code where I am initializing the REstTemplate bean is as follows:
#Bean(name = "template")
public RestTemplate restTemplate(RestTemplateBuilder builder) throws NoSuchAlgorithmException, KeyManagementException {
TrustManager[] trustAllCerts = new TrustManager[] {
new X509TrustManager() {
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return new X509Certificate[0];
}
public void checkClientTrusted(
java.security.cert.X509Certificate[] certs, String authType) {
}
public void checkServerTrusted(
java.security.cert.X509Certificate[] certs, String authType) {
}
}
};
SSLContext sslContext = SSLContext.getInstance("SSL");
sslContext.init(null, trustAllCerts, new java.security.SecureRandom());
HostnameVerifier hostnameVerifier = SSLConnectionSocketFactory.getDefaultHostnameVerifier();
// Conn Pool
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager(
RegistryBuilder.<ConnectionSocketFactory>create()
.register("http", PlainConnectionSocketFactory.getSocketFactory())
.register("https", new SSLConnectionSocketFactory(sslContext, hostnameVerifier))
.build());
cm.setMaxTotal(Integer.MAX_VALUE);
cm.setDefaultMaxPerRoute(50);
CloseableHttpClient httpClient = HttpClients.custom()
.setSSLContext(sslContext)
.setConnectionManager(cm)
.setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE)
.build();
HttpComponentsClientHttpRequestFactory customRequestFactory = new HttpComponentsClientHttpRequestFactory();
customRequestFactory.setHttpClient(httpClient);
return builder.requestFactory(() -> customRequestFactory).build();
}
The RestTemplate is then autowired:
#Autowired
#Qualifier("template")
private RestTemplate restTemplate;
And the RestTemplate is used thus:
restResponse = restTemplate.postForObject(targetUrl, obj, CustomRestResponse.class);
I am deploying the application to PCF where it is running with 6 instances. It's then giving me an error like this:
org.springframework.web.client.RestClientException: Error while extracting response for type [class com.test.model.CustomRestResponse] and content type [application/json;charset=UTF-8]; nested exception is org.apache.http.ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected
To replicate the issue in my local environment, I created a multithreaded application which will spawn multiple threads and call the REST service simultaneously:
ExecutorService service = Executors.newFixedThreadPool(10);
IntStream.range(0, 100).forEach(i -> service.submit(new RestTemplateTesterRunnable(i)));
And I was able to get the exception in my local system as well (Note that I don't have this multithreading code in the actual application. This is just for replicating the problem in my local since I can't run 6 instances automatically). The full stacktrace is as follows:
org.springframework.web.client.RestClientException: Error while extracting response for type [class com.test.model.CustomRestResponse] and content type [application/json;charset=UTF-8]; nested exception is org.apache.http.ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected
at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:119)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:744)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:677)
at org.springframework.web.client.RestTemplate.postForObject(RestTemplate.java:421)
at com.test.RestTemplateTester$RestTemplateTesterRunnable.run(RestTemplateTester.java:237)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.http.ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected
at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:263)
at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:222)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:183)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:210)
at org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:312)
at org.apache.http.impl.execchain.ResponseEntityProxy.streamClosed(ResponseEntityProxy.java:142)
at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)
at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)
at java.io.PushbackInputStream.close(PushbackInputStream.java:379)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._closeInput(UTF8StreamJsonParser.java:252)
at com.fasterxml.jackson.core.base.ParserBase.close(ParserBase.java:369)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4210)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3258)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:239)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:227)
at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:104)
... 9 more
However, I am not being able to solve the problem. I referred this question:
How to efficiently use RestTemplate in multithreading application?
and this one:
How to use RestTemplate efficiently in Multithreaded environment?
and in both of them, people are saying that RestTemplate is thread-safe and there shouldn't be a problem in using it in a multithreaded environment. However, what I am seeing is that this exception is still happening for me. I thought at first that this was happening due to my use of the CloseableHttpClient, as is given by this post:
ConnectionClosedException: Premature end of chunk coded message body with apache hhtpclient 4.5.5
However, as I am not directly using the client, which is internally used by Spring Boot, I didn't have much to do about it. But I looked into the internal working of the RestTemplate's postForObject() method in any case, which led me to this method:
#Nullable
protected <T> T doExecute(URI url, #Nullable HttpMethod method, #Nullable RequestCallback requestCallback,
#Nullable ResponseExtractor<T> responseExtractor) throws RestClientException {
Assert.notNull(url, "URI is required");
Assert.notNull(method, "HttpMethod is required");
ClientHttpResponse response = null;
try {
ClientHttpRequest request = createRequest(url, method);
if (requestCallback != null) {
requestCallback.doWithRequest(request);
}
response = request.execute();
handleResponse(url, method, response);
return (responseExtractor != null ? responseExtractor.extractData(response) : null);
}
catch (IOException ex) {
String resource = url.toString();
String query = url.getRawQuery();
resource = (query != null ? resource.substring(0, resource.indexOf('?')) : resource);
throw new ResourceAccessException("I/O error on " + method.name() +
" request for \"" + resource + "\": " + ex.getMessage(), ex);
}
finally {
if (response != null) {
response.close();
}
}
}
I am seeing that before returning, the response object is being closed, as was being told in the previous StackOverflow post I linked. Can this be the reason for the ConnectionClosedException in a multithreaded environment? If so, what is the solution? I tried to use evictIdleConnections() as given by the Javadocs:
https://hc.apache.org/httpcomponents-client-4.5.x/current/httpclient/apidocs/org/apache/http/impl/client/HttpClientBuilder.html#evictIdleConnections(long,%20java.util.concurrent.TimeUnit)
So the code became:
CloseableHttpClient httpClient = HttpClients.custom()
.setSSLContext(sslContext)
.setConnectionManager(cm)
.setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE)
.evictIdleConnections(10, TimeUnit.MILLISECONDS)
.build();
However, this didn't help. I also saw that some people are using org.apache.commons.httpclient.MultiThreadedHttpConnectionManager, but I found that this is a deprecated connection manager, and the HttpComponentsClientHttpRequestFactory does not support it. Nevertheless, I somehow used it also, but with no results. Please hellp me in getting the actual reason for this exception, and how to resolve it.

Apache HttpClient throws java.net.SocketException: Connection reset if I use it as singletone

I created an Apache HTTP client
CloseableHttpClient client = HttpClients.custom()
.setMaxConnPerRoute(25)
.setMaxConnTotal(50)
.setDefaultRequestConfig(RequestConfig.custom()
.setConnectionRequestTimeout(20_000)
.build())
.build();
I am using it as a singleton. I have this method for sending requests:
public RestResponse sendPost(String serverUrl, String body) throws RestException {
try {
HttpPost httpPost = new HttpPost(serverUrl);
httpPost.setEntity(new StringEntity(body));
try (CloseableHttpResponse response = client.execute(httpPost)) {
RestResponse restResponse = new RestResponse();
restResponse.setCode(response.getStatusLine().getStatusCode());
restResponse.setBody(EntityUtils.toString(response.getEntity()));
return restResponse;
}
} catch (Exception e) {
throw new RestException(e);
}
}
It works fine. But after some time (5-6 min) of idle, if I send a request I get "java.net.SocketException: Connection reset"
I have 2 questions
How can I find a place where this time is outset? those parameters don't work for me
.setSocketTimeout(25_000)
.setConnectTimeout(26_000)
What is the best way to fix this problem? (I mean retry request or change timeout or reconnect after or before each request)

This is a general limitation of the classic (blocking) i/o: blocking connections cannot react to any i/o events when idle (not engaged in an i/o operations). Likewise, timeout settings have no effect on idle connections.
There are several defensive measures one can employ to minimize the chances of a persistent connection reset:
Validate persistent connections upon lease from the connection pool after a certain period of inactivity (for example, after 1s)
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
cm.setValidateAfterInactivity(1000);
CloseableHttpClient httpclient = HttpClients.custom()
.setConnectionManager(cm)
.build();
Proactively evict expired connections and those that have been idle over a certain period of time (for example, 5s) from the connection pool
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
cm.setValidateAfterInactivity(1000);
CloseableHttpClient httpclient = HttpClients.custom()
.setConnectionManager(cm)
.evictExpiredConnections()
.evictIdleConnections(5L, TimeUnit.SECONDS)
.build();
If everything else fails, retry requests that are safe to retry. One might want to use a custom HttpRequestRetryHandler implementation instead of the default one when more control is necessary.
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
cm.setValidateAfterInactivity(1000);
CloseableHttpClient httpclient = HttpClients.custom()
.setConnectionManager(cm)
.setRetryHandler(DefaultHttpRequestRetryHandler.INSTANCE)
.build();
Evict connections from the pool manually when expecting a long period inactivity. This is what HttpClient does for you automatically when configured with evictIdleConnections method at the cost of an extra monitor thread.
cm.closeIdleConnections(0, TimeUnit.MICROSECONDS);

Adding SSLContext in CloseableHttpAsyncClient at Runtime

We have a generic application which delivers message to different POST endpoints. And we are using
CloseableHttpAsyncClient for this purpose. Its been built/initialized as follows,
private static CloseableHttpAsyncClient get() {
CloseableHttpAsyncClient lInstance;
IOReactorConfig ioReactorConfig = IOReactorConfig.custom()
.setIoThreadCount(100)
.setConnectTimeout(10000)
.setSoTimeout(10000).build();
ConnectingIOReactor ioReactor = null;
try {
ioReactor = new DefaultConnectingIOReactor(ioReactorConfig);
} catch (IOReactorException e) {
logger_.logIfEnabled(Level.ERROR, e);
}
PoolingNHttpClientConnectionManager connManager = new PoolingNHttpClientConnectionManager(ioReactor);
connManager.setDefaultMaxPerRoute(50);
connManager.setMaxTotal(5000);
connManager.closeIdleConnections(10000, TimeUnit.MILLISECONDS);
baseRequestConfig = RequestConfig.custom().setConnectTimeout(10000)
.setConnectionRequestTimeout(10000)
.setSocketTimeout(10000).build();
lInstance = HttpAsyncClients.custom().setDefaultRequestConfig(baseRequestConfig)
.setConnectionManager(connManager).build();
lInstance.start();
return lInstance;
}
This is prebuilt and initialized. As an when a new request arrives to our application, based on message, authentication type, a new postRequest is built httpPost = new HttpPost(builder.build());
After setting the required header, payload etc. exiting httpClient is used to send the request.
httpClient.execute(httpPost, httpContext, null);
Now, the question is based on the our new requirement to support client certificate based authentication. And since our current approach is to create httpClient in the beginning, the question is how to change the behaviour of httpClient to send client certificate to some endpoints and work as it is for other endpoints which doesn't require certificate to be send?
I know I can introduce SSLContext to CloseableHttpAsyncClient while creating, but at the time of creating I don't have any information that we have any endpoint which requires certificate based authentication. And we can have many endpoints which would be supporting client certificate and that would be known at runtime.

Using Apache HttpClient how to set the TIMEOUT on a request and response

I need to set time out for the Http Request we make to a service (not a web service). We are using Apache HTTP Client. I have added these 2 lines of code to set the time out on request and response to the service.
HttpConnectionParams.setConnectionTimeout(params, 10000);
HttpConnectionParams.setSoTimeout(params, 10000);
1) Currently I have set 10 seconds as the timeout since I see the response coming from the service almost instantaneously. Should I increase or decrease the timing?
2) What will happen when response is takes more than 10 seconds? Will it throw exception and what exception will it be? Is there any thing else I need to add to set the time out in the below code.
public HashMap<String, Object> getJSONData(String url) throw Exception{
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpParams params = httpClient.getParams();
HttpConnectionParams.setConnectionTimeout(params, 10000);
HttpConnectionParams.setSoTimeout(params, 10000);
HttpHost proxy = new HttpHost(getProxy(), getProxyPort());
ConnRouteParams.setDefaultProxy(params, proxy);
URI uri;
InputStream data = null;
try {
uri = new URI(url);
HttpGet method = new HttpGet(uri);
HttpResponse response = httpClient.execute(method);
data = response.getEntity().getContent();
}
catch (Exception e) {
e.printStackTrace();
}
Reader r = new InputStreamReader(data);
HashMap<String, Object> jsonObj = (HashMap<String, Object>) GenericJSONUtil.fromJson(r);
return jsonObj;
}

I am guessing many people come here because of the title and because the HttpConnectionParams API is deprecated.
Using a recent version of Apache HTTP Client, you can set these timeouts using the request params:
HttpPost request = new HttpPost(url);
RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(TIMEOUT_MILLIS)
.setConnectTimeout(TIMEOUT_MILLIS)
.setConnectionRequestTimeout(TIMEOUT_MILLIS)
.build();
request.setConfig(requestConfig);
Alternatively, you can also set this when you create your HTTP Client, using the builder API for the HTTP client, but you'll also need to build a custom connection manager with a custom socket config.
The configuration example file is an excellent resource to find out about how to configure Apache HTTP Client.

The exceptions you'll see will be ConnectTimeoutException and SocketTimeoutException. The actual timeout values you use should be the maximum time your application is willing to wait. One important note about the read timeout is that it corresponds to the timeout on a socket read. So it's not the time allowed for the full response to arrive, but rather the time given to a single socket read. So if there are 4 socket reads, each taking 9 seconds, your total read time is 9 * 4 = 36 seconds.
If you want to specify a total time for the response to arrive (including connect and total read time), you can wrap the call in a thread and use a thread timeout for that. For example, I usually do something like this:
Future<T> future = null;
future = pool.submit(new Callable<T>() {
public T call() {
return executeImpl(url);
}
});
try {
return future.get(10, TimeUnit.SECONDS);
}
catch (InterruptedException e) {
log.warn("task interrupted", name);
}
catch (ExecutionException e) {
log.error(name + " execution exception", e);
}
catch (TimeoutException e) {
log.debug("future timed out", name);
}
Some assumptions made in the code above are: 1) this is in a function with a url parameter, 2) it's in a class with a name variable, 3) log is a log4j instance, and 4) pool is a some thread pool executor. Note that even if you use a thread timeout, you should also specify a connect and socket timeout on the HttpClient, so that slow requests don't eat up the resources in the thread pool. Also note that I use a thread pool because typically I use this in a web service so the thread pool is shared across a bunch of tomcat threads. You're environment may be different, and you may prefer to simply spawn a new thread for each call.
Also, I've usually see the timeouts set via member functions of the params, like this:
params.setConnectionTimeout(10000);
params.setSoTimeout(10000);
But perhaps your syntax works as well (not sure).

"Time out" when multithreading requests to a webservice with java and axis2

I'm working with a slow webservice (about 4 minutes each request) and I need to do about 100 requests in two hours, so I've decided to use multiple threads. The problem is that I can only have 2 threads, as the stub rejects all the other ones. Here I've found an explanation and possible solution:
I had the same problem. It seems that
the source of it is
defaultMaxConnectionsPerHost value in
MultiThreadedHttpConnectionManager
equals 2. Workaround for me was to
create own instance of
MultiThreadedHttpConnectionManager and
use it in service stub, something like
in example below
I've done as the author said, and passed a HttpClient to the stub with higher setMaxTotalConnections and setDefaultMaxConnectionsPerHost values, but the problem is that now the application freezes (well, it does not really freezes, but It does nothing).
Thats my code:
public ReportsStub createReportsStub(String url, HttpTransportProperties.Authenticator auth){
ReportsStub stub = null;
HttpClient httpClient = null;
try {
stub = new ReportsStub(url);
httpClient = createHttpClient(10,5);
stub._getServiceClient().getOptions().setTimeOutInMilliSeconds(10000000);
stub._getServiceClient().getOptions().setProperty(org.apache.axis2.transport.http.HTTPConstants.AUTHENTICATE, auth);
stub._getServiceClient().getOptions().setProperty(org.apache.axis2.transport.http.HTTPConstants.CHUNKED, false);
stub._getServiceClient().getServiceContext().getConfigurationContext().setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
return stub;
} catch (AxisFault e) {
e.printStackTrace();
}
return stub;
}
protected HttpClient createHttpClient(int maxTotal, int maxPerHost) {
MultiThreadedHttpConnectionManager httpConnectionManager = new MultiThreadedHttpConnectionManager();
HttpConnectionManagerParams params = httpConnectionManager.getParams();
if (params == null) {
params = new HttpConnectionManagerParams();
httpConnectionManager.setParams(params);
}
params.setMaxTotalConnections(maxTotal);
params.setDefaultMaxConnectionsPerHost(maxPerHost);
HttpClient httpClient = new HttpClient(httpConnectionManager);
return httpClient;
}
Then I pass that stub and the request to each one of threads and run them. If I don't set the HttpClient and use the default, only two threads execute, and if I set it, the application does not work. Any idea?

If anyone wants to create a dynamic REST client in WSO2 Axis2, the following code worked for me...
// Set the max connections to 20 and the timeout to 20 seconds
MultiThreadedHttpConnectionManager multiThreadedHttpConnectionManager = new MultiThreadedHttpConnectionManager();
HttpConnectionManagerParams params = new HttpConnectionManagerParams();
params.setDefaultMaxConnectionsPerHost(20);
params.setMaxTotalConnections(20);
params.setSoTimeout(20000);
params.setConnectionTimeout(20000);
multiThreadedHttpConnectionManager.setParams(params);
HttpClient httpClient = new HttpClient(multiThreadedHttpConnectionManager);
// Create the service client
ServiceClient serviceClient = new ServiceClient();
Options options = new Options();
options.setTo(new EndpointReference(endpoint));
options.setProperty(Constants.Configuration.ENABLE_REST, Constants.VALUE_TRUE);
options.setProperty(Constants.Configuration.HTTP_METHOD, Constants.Configuration.HTTP_METHOD_POST);
serviceClient.getServiceContext().getConfigurationContext().setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
serviceClient.setOptions(options);
// Blocking call
OMElement result = serviceClient.sendReceive(ClientUtils.getRestPayload()); // just a dummy payload <root></root>
// Cleanup Transport after each call, this is needed to otherwise the HTTP gets blocked
serviceClient.cleanupTransport();
I put the Max Connections to 20 and the Timeout to 20 seconds.
Also my 'endpoint' contains all the REST arguments, I'm just using a dummy payload "<root></root>" in the serviceClient.sendReceive() method.

I noticed this in a corporate web application that called a back-end service that could take a long period to respond. The web application would lock up because a limit of 2 connections to a single host would take hold.
You call httpConnectionManager.setParams( params ) before you call params.setDefaultMaxConnectionsPerHost(). Have you tried calling these functions in the opposite order to confirm that application of params doesn't take place within the httpConnectionManager.setParams function itself?