We Keep Coding

Why try-with-resources causes Truncated chunk error with CloseableHttpClient?

Could you please help me with the following:
I have the following method:
public static CloseableHttpResponse getRequest (String url) {
try (CloseableHttpClient httpClient = HttpClients.createDefault();){
HttpGet httpget = new HttpGet(url); //http get request (create get connection with particular url)
return httpClient.execute(httpget);
} catch (Exception e) {
throw new RuntimeException(e.getMessage(), e);
}
}
Where I use CloseableHttpClient with try-with-resources
I invoke method in some simple test:
CloseableHttpResponse closeableHttpResponse = RestClient.getRequest("https://reqres.in/api/users?page=2");
String responseString = EntityUtils.toString(closeableHttpResponse.getEntity(), "UTF-8");
JSONObject responseJson = new JSONObject(responseString);
System.out.println(responseJson);
And I am getting error: org.apache.http.TruncatedChunkException: Truncated chunk (expected size: 379; actual size: 358)
When I am not using try-with-resources like that:
public static CloseableHttpResponse getRequest (String url) throws IOException {
CloseableHttpClient httpClient = HttpClients.createDefault();
HttpGet httpget = new HttpGet(url); //http get request (create get connection with particular url)
return httpClient.execute(httpget);
}
I have no error at all! Could you please explain - what the wrong? I am newbie and have no clue - some examples from internet are working good...

The try-with-resources block will automatically call close() on the object, so the return from one of those getRequest calls is a closed CloseableHttpClient instance.
The call without try-with-resources will return a working (not closed) CloseableHttpClient.

While running test case ClientHttpResponse is throwing 401 error

I have a problem with my unit test. In my unit test I am getting 401 Unauthorised as response status and I don't know how to solve this problem. This is not a Spring project.
My Test class
#RunWith(MockitoJUnitRunner.class)
public class LTest {
#Test
public void test_retrieve() throws Exceptions{
CloseableHttpClient mockHttpClient = mock(CloseableHttpClient.class);
CloseableHttpResponse mockHttpResponse = mock(CloseableHttpResponse.class);
HttpEntity mockEntity = mock(HttpEntity.class);
StatusLine mockStatusLine = mock(StatusLine.class);
when(mockHttpClient.execute(new HttpGet(new URIBuilder(anyString()).build()))).thenReturn(mockHttpResponse);
when(mockHttpResponse.getEntity()).thenReturn(mockEntity);
when(mockHttpResponse.getStatusLine()).thenReturn(mockStatusLine);
when(mockStatusLine.getStatusCode()).thenReturn(HttpStatus.SC_OK);
Map<String, Employee> map = sample.retrieve();
assertNotNull(map);
assertEquals(1,map.size());
}
source code for the above test case
CloseableHttpClient httpClient = HttpClientUtils.setupClient(HttpClientBuilder.create()).build();
String url = "http://someexample.com";
UriBuilder builder = new URIBuilder(url)
.setParameter("limit",5)
.setParameter("centre",centre);
CloseableHttpResponse httpResponse = httpClient.execute(new HttpGet(builder.build()));
if(httpResponse.getStatusLine().getStatusCode() == HttpStatus.SC_OK){
try{
String entity = EntityUtils.toString(httpResponse.getEntity());
ObjectNode node = new ObjectMapper().readValue(entity,ObjectNode.class);
} catch (IOException e){
e.printStackTrace();
}
}
While running the test case it's showing assertion error because it's going through catch block due below line is throwing 401 instead of 200
CloseableHttpResponse httpResponse = httpClient.execute(new HttpGet(builder.build()));
Can anyone please help me with the above error I am getting?

In the test, HTTPClient is not mocked and that is the reason for the failure.
To mock the HTTPClient we can follow the below strategy
Extract getHttpClient() in the ClassToBeTested as
public class HttpClientToBeTested {
public Map retrieve() throws URISyntaxException, IOException {
CloseableHttpClient httpClient = getHttpClient();
String url = "http://someexample.com";
URIBuilder builder = new URIBuilder(url)
.setParameter("limit","5")
.setParameter("centre","centre");
CloseableHttpResponse httpResponse = httpClient.execute(new HttpGet(builder.build()));
if(httpResponse.getStatusLine().getStatusCode() == HttpStatus.SC_OK){
try{
String entity = EntityUtils.toString(httpResponse.getEntity());
//ObjectNode node = new ObjectMapper().readValue(entity,ObjectNode.class);
Map node = new ObjectMapper().readValue(entity, Map.class); // Assume ObjectNode is a custom class, so for demo using Map.
return node;
} catch (IOException e){
e.printStackTrace();
throw e;
}
}
return null;
}
// New extracted method that will be mocked in the test case
protected CloseableHttpClient getHttpClient() {
return HttpClientBuilder.create().build();
}
}
Next in the test class, we can inject the mock by subclassing the ClassToBeTested in an anonymous class as follows.
#Test
public void test_retrieve() throws Exception {
CloseableHttpClient mockHttpClient = mock(CloseableHttpClient.class);
CloseableHttpResponse mockHttpResponse = mock(CloseableHttpResponse.class);
//HttpEntity mockEntity = mock(HttpEntity.class); Not required since we will pass actual entity
StatusLine mockStatusLine = mock(StatusLine.class);
when(mockHttpClient.execute(new HttpGet(new URIBuilder(anyString()).build()))).thenReturn(mockHttpResponse);
when(mockHttpResponse.getEntity()).thenReturn(new StringEntity("{\"key\":\"value\"}")); // Important: Pass your actual response as string here.
when(mockHttpResponse.getStatusLine()).thenReturn(mockStatusLine);
when(mockStatusLine.getStatusCode()).thenReturn(HttpStatus.SC_OK);
// Code to mock the http client
HttpClientToBeTested sample = new HttpClientToBeTested() {
#Override
protected CloseableHttpClient getHttpClient() {
return mockHttpClient;
}
};
Map map = sample.retrieve();
assertNotNull(map);
assertEquals(1,map.size());
}
UPDATE: After mocking the httpClient, httpClient.execute() should return SC_OK. However, post that deserializing the response will fail since the mock HttpEntity will return null. To avoid it, we will rather send a StringEntity. Updated the actual and test class with the details.

Right way to close CloseableHttpResponse/CloseableHttpClient [duplicate]

This question already has answers here:
What is the difference between CloseableHttpClient and HttpClient in Apache HttpClient API?
(8 answers)
Closed 2 years ago.
I'm using CloseableHttpResponse (from apache-httpclient-4.5.3) and I'm not sure I'm using it right, I saw an answer with no votes to use EntityUtils.consume on finally:
CloseableHttpResponse response1 = httpclient.execute(httpGet);
try {
System.out.println(response1.getStatusLine());
} finally {
EntityUtils.consume(response1.getEntity());
CloseableHttpClient is abstract and has no close method to call although in this answer it's used:
CloseableHttpResponse response = httpclient.execute(httpget);
try {
//do something
} finally {
response.close();
}
Currently I'm using try with resources for CloseableHttpClient and CloseableHttpResponse inside of send method.
Am I not missing any resource open or using it in a wrong way?
private CloseableHttpResponse send()
throws URISyntaxException, UnsupportedEncodingException, IOException, ClientProtocolException {
URIBuilder uriBuilder = new URIBuilder(BASE_URL);
HttpHost target = new HttpHost(uriBuilder.getHost(), uriBuilder.getPort(), uriBuilder.getScheme());
HttpPost post = new HttpPost(uriBuilder.build());
try (CloseableHttpClient httpClient = HttpClients.custom().build(); CloseableHttpResponse response = httpClient.execute(target, post)) {
return response;
}

It has been explained in detail in the docs here.
Quoting the pseudo code from the docs here's a typical way to allocate/deallocate an instance of CloseableHttpClient:
try (CloseableHttpClient httpclient = HttpClients.createDefault()) {
<...>
}
The same applies to CloseableHttpResponse :
try (CloseableHttpResponse response = httpclient.execute(httpget)) {
<...>
}
Now, about the close method in CloseableHttpClient. CloseableHttpClient is an abstract class that implements Closeable interface. That is, although it doesn't have a close method itself the classes that extend it are required to implement the close method. One class is InternalHttpClient. You can check the source code for the details.
Before Java7, explicit close would be required:
CloseableHttpClient httpclient = HttpClients.createDefault();
try {
<...>
} finally {
httpclient.close();
}
CloseableHttpResponse response = httpclient.execute(httpget);
try {
<...>
} finally {
response.close();
}

You can avoid the finally by using the try(resource)
try (CloseableHttpResponse response = httpclient.execute(httpGet)) {
... }

HttpClient connection reusing with 4.3.x

I'm trying to use HttpClient and am having trouble deciphering the meaning of 1.1.5. Ensuring release of low level resources.
Are these how closing the content stream and closing the response are interpreted?
Closing the content stream: (keeps the underlying connection alive)
CloseableHttpClient httpclient = HttpClients.createDefault();
try {
HttpGet httpget = new HttpGet("http://localhost/");
// do multiple times on the same connection
for (...) {
HttpResponse response = httpclient.execute(httpget);
HttpEntity entity = response.getEntity();
if (entity != null) {
try {
// do something useful
} finally {
EntityUtils.consume(entity); // <-- ensures reuse
}
}
}
} finally {
httpclient.close();
}
Closing the response: (immediately shuts down and discards the connection)
CloseableHttpClient httpclient = HttpClients.createDefault();
try {
HttpGet httpget = new HttpGet("http://localhost/");
// do multiple times on different connections
for (...) {
ClosableHttpResponse response = httpclient.execute(httpget);
try {
HttpEntity entity = response.getEntity();
if (entity != null) {
// do something useful
}
} finally {
response.close(); // <-- ensures reconnect
}
}
} finally {
httpclient.close();
}

entityUtils.consume closes the stream for you...
if (entity.isStreaming()) {
final InputStream instream = entity.getContent();
if (instream != null) {
instream.close();
}
}
You just 'release' your client back to the pool...
Then, you should wrap your HttpClient in a runnable...
public void run() {
handler.sendMessage(Message.obtain(handler, HttpConnection.DID_START));
CloseableHttpClient httpClient = HttpClients.custom()
.setConnectionManager(YourConnectionMgr.getInstance())
.addInterceptorLast(new HttpRequestInterceptor() {
public void process(
final HttpRequest request,
final HttpContext context) throws HttpException, IOException {
}
})
.build();
} //end runnable
At the endof runnable, the client just gets released back to the ConnectionPool and you dont have to worry about resources or cleanup.
Use a manager that extends PoolingClientConnectionManager
newInstance = new MyConnectionManager(schemeRegistry);
instance.setMaxTotal(15);
instance.setDefaultMaxPerRoute(15);
HttpHost localhost = new HttpHost("api.parse.com", 443);
instance.setMaxPerRoute(new HttpRoute(localhost), 10);
Then at the end , i think you do need to shutdown the pool.
YourConnectionMgr.getInstance().shutdown();
YourConnectionMgr.reset();
More details here

In general, once you're done with the entity you want to discard it so that system resources aren't tied up with objects that are no longer meaningful. In my opinion, the only distinction here is use. That chapter on fundamentals is basically describing that point. However you implement it, make sure that you use resources only for as long as you need them. The low level resource is the InputStream in the entity, the high level resource is the connection. If you're implementing something that doesn't need to read the full InputStream in order to make a determination, for example, just terminate the response and the cleanup will be handled for you efficiently.

Just need someone familiar with HTTPClient to check over a piece of code

here are two little helper methods I have made for downloading files. I have had to mix and match different tutorials of the web to get what I have here.
Now is there anything that I have done blatantly wrong here?
public static InputStream simplePostRequest(URL url, List<NameValuePair> postData) throws ClientProtocolException, IOException {
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpPost postMethod=new HttpPost(url.toExternalForm());
postMethod.setEntity(new UrlEncodedFormEntity(postData, HTTP.UTF_8));
HttpResponse response = httpclient.execute(postMethod);
HttpEntity entity = response.getEntity();
return entity.getContent();
}
public static InputStream simpleGetRequest(URL url, List<NameValuePair> queryString) throws ClientProtocolException, IOException {
Uri.Builder uri = new Uri.Builder();
uri.path(url.getPath());
for(NameValuePair nvp: queryString) {
uri.appendQueryParameter(nvp.getName(), nvp.getValue());
}
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpHost host = new HttpHost(url.getHost());
HttpResponse response = httpClient.execute(host, new HttpGet(uri.build().toString()));
HttpEntity entity = response.getEntity();
return entity.getContent();
}

I wouldn't expect a huge response to a such a vague question. Why not write a pair of unit tests to try your code out instead?
Anyway, the one thing that stands out for me based on my experience with HttpClient is that if subject to heavy load (large number of concurrent threads), your code seems unsafe - there seems to be no upper bound to the number of concurrent connections that could get created.
If you think this might be relevant in your case, you could try something like this:
class X {
private static final HttpClient httpClient;
static {
SchemeRegistry defaultRegistery = new DefaultHttpClient().getConnectionManager()
.getSchemeRegistry();
ThreadSafeClientConnManager connMgr = new ThreadSafeClientConnManager(defaultRegistery);
connMgr.setMaxTotalConnections(10);
connMgr.setDefaultMaxPerRoute(10);
httpClient = new DefaultHttpClient(connMgr);
httpClient.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 30000);
}
public static InputStream simpleGetRequest(URL url, List<NameValuePair> queryString) throws ClientProtocolException, IOException {...}
public static InputStream simpleGetRequest(URL url, List<NameValuePair> queryString) throws ClientProtocolException, IOException {...}
}
... and just use the static httpClient in your methods instead of instantiating a new one every time.
I copied and modified this from some code I've written before, but don't consider this necessarily correct. The unit tests will be your friends here.
EDIT: With regards to your comment about mixing URL and the HttpClient library class NameValuePair (is this your concern?), why not just a Map<String, String> in the method signatures?