This is scenario I am testing. The steps are as follows
Create an instance of HttpClient(org.apache.commons.httpclient.HttpClient)3.1
Use the instance to get response from a url (http://www.example.com/submitData)
Bring down the server hosting the url mentioned in step 2.
Bring up the url
Wait till the url is up
Use the instance of HttpClient created in step 1 to get response from the url (http://www.example.com/getData)
I get an error - Connection Reset.
Can anyone help me understand.
I have used the following params in the HttpClient
httpClient.getParams().setParameter("http.socket.timeout", new Integer(0));
httpClient.getParams().setParameter("http.connection.stalecheck", new Boolean(true));
Edited after comments from Peter
This is how I create the HttpClient
HttpClient httpClient = new HttpClient();
AuthScope authScope = new AuthScope("www.example.com", 80, AuthScope.ANY_REALM);
Credentials defaultcreds = new UsernamePasswordCredentials("username", "pwd");
httpClient.getState().setCredentials(authScope, defaultcreds);
httpClient.getParams().setParameter("http.socket.timeout", new Integer(0));
httpClient.getParams().setParameter("http.connection.stalecheck", new Boolean(true));
To establish connection, I use the following
GetMethod getMethod = new GetMethod("http://"+httpClient.getHostConfiguration().getHost()+"/getData");
int statusCode = httpClient.executeMethod(getMethod);
I want the httpclient instance to preserve the hostname and port and credentials.
This is exact execption message that I get - "Connection reset".
More info - I am executing the test from eclipse. If I run the test case in debug mode, I dont get this exception.
I think the behaviour is reasonable because when the server goes down the client socket on the server goes away too. The client still holds old socket but this socket is gone on the server. You should reconnect from client.
AFAIK the stale parameter allows the client to close the connection the clean way (without Exception) but not prevent from disconnection.
This looks suspicious
httpClient.getParams().setParameter("http.socket.timeout", new Integer(0));
Having a timeout set to 0ms would likely terminate the request prematurely?
Related
I am using httpClient version 4.5 to connect with our external vendor site. We do not need any connection pool or persistent connection, so I am using the BasicHttpClientConnectionManager to create the HttpClient.
This works fine for minimal number of requests, but if I test this for 1TPS for 1 hour, by the end of the test, we start seeing intermittent connection resets. (guessing request count > 100)
I/O exception (java.net.SocketException) caught when processing request to {s}->https://apiURL:443: Connection reset
Please find below the code snippet for making connection.
SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(kmf.getKeyManagers(), new X509TrustManager[] { new DefaultTrustManager() }, new SecureRandom());
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslContext, new String[] { "TLSv1.2" }, null,SSLConnectionSocketFactory.getDefaultHostnameVerifier());
Registry<ConnectionSocketFactory> socketFactoryRegistry = RegistryBuilder.<ConnectionSocketFactory>create()
.register("https", sslsf).register("http", new PlainConnectionSocketFactory()).build();
HttpClientConnectionManager connectionManager = new BasicHttpClientConnectionManager(socketFactoryRegistry);
HttpRequestRetryHandler retryHandler = new DefaultHttpRequestRetryHandler(1, false);
RequestConfig defaultRequestConfig = RequestConfig.custom().setSocketTimeout(Integer.parseInt(30000)).setConnectTimeout(Integer.parseInt(30000)).setConnectionRequestTimeout(30000).setCookieSpec(CookieSpecs.STANDARD).build();
CloseableHttpClient httpClient = HttpClients.custom().setConnectionManager(connectionManager).setDefaultRequestConfig(defaultRequestConfig).setRetryHandler(retryHandler).evictExpiredConnections().build();
HttpPost httpPost = new HttpPost(<endpoint>);
httpPost.setEntity(new UrlEncodedFormEntity(requestData));
httpResponse = httpClient.execute(httpPost);
I saw that fix for a similar issue reported is already available with version 4.5. (Ref: https://issues.apache.org/jira/browse/HTTPCLIENT-1655) provided by Oleg
If thats the case, not sure why I am still facing this issue. Could someone please help on this.
Thanks!
Hi Oleg,
I had been using the httpclient version 4.5.3 when I was still seeing the connection reset errors as posted above.
Later noticed that the fix for reset issue was committed to the version 4.5.1 (https://issues.apache.org/jira/browse/HTTPCLIENT-1655). So, just tried updating that particular version, ran a test and not seeing the connection reset errors any more.
I had expected that this fix should also be available in the higher versions starting from 4.5.1. But, I guess its somehow missed out in the higher versions, have verified that its still an issue with version 4.5.3 for sure.
So, conclusion is that the connection reset error was fixed using httpclient 4.5.1 jar.
Thanks!
This problem has blocked our whole team half a day!
We use apache httpclient 4.3.x to post and get data from an storage server which provides http api. In order to improve performance, we used PoolingHttpClientConnectionManager:
public HttpClient createHttpClient() {
Registry registry = RegistryBuilder.create()....build();
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager(registry);
connectionManager.setMaxTotal(50);
connectionManager.setDefaultMaxPerRoute(50);
CloseableHttpClient httpClient = HttpClients.custom()
.setConnectionManager(connectionManager)
.build();
return httpClient;
}
Then we hold an instance of the httpClient in our program, reuse it with every http request:
Global httpClient:
HttpClient httpClient = createHttpClient();
Post some data:
HttpPost httpPut = new HttpPost("...");
HttpResponse response = httpClient.execute(httpPut);
// Notice we get the response content here!
String content = EntityUtils.toString(response.getEntity());
System.out.println(content);
httpPut.releaseConnection();
response.close();
Then get:
HttpGet httpGet = new HttpGet("...");
// Blocked at this line !!!!
HttpResponse response = httpClient.execute(httpGet);
String content = EntityUtils.toString(response.getEntity());
System.out.println(content);
httpPut.releaseConnection();
response.close();
Please notice the line: // Blocked at this line !!!!
The program has blocked at that line and never go to next line. In debugging mode, I can see it has been blocked at:
SocketInputStream.socketRead0()
I've searched for a lot of questions and documents, but no lucky.
My colleage just fix it by setting NoConnectionReuseStrategy.INSTANCE:
HttpClients.custom()
.setConnectionManager(connectionManager)
// Following line fixed the problem, but why?
.setConnectionReuseStrategy(NoConnectionReuseStrategy.INSTANCE)
.build();
Now it doens't blocked, but why?
What does "reuse connection" mean? And is there performance issue by using NoConnectionReuseStrategy?
Thank you, guys~
I tried to reproduce the blocking http-get (also as an exercise for myself) but even without closing responses I could not get it to block. The ONLY time I managed to make the http-get block is by doing a response.getEntity().getContent() without reading from the returned InputStream and without closing the returned InputStream.
For my tests I used Tomcat 7.0.47 with two very simple servlets (one responding "OK" to a get, the other echoing a post) as a server. The client started 50 threads with each thread performing 30 alternating http-get and http-post request (total of 1500 requests). The client did not use the RegistryBuilder, instead the default one is used (created by the PoolingHttpClientConnectionManager itself).
About the NoConnectionReuseStrategy: by default (HttpClient created with HttpClients.createDefault(), I used org.apache.httpcomponents:httpclient:4.3.1) a connection pool is used with a maximum of 2 connections to 1 server. E.g. even if 5 threads are doing all kinds of requests at the same time to 1 server, the connection pool opens only 2 connections, re-uses them for all requests and ensures that 1 connection is used by 1 thread at any given time. This can have a very positive impact on client performance and significantly reduces load on the server. The only thing you must make sure is to call response.close() in a finally-block (this ensures the connection is returned to the connection pool). By using the NoConnectionReuseStrategy you basically disable the connection pool: for each request a new connection will be created. I recommend you enable debug-logging for category org.apache.http.impl.conn.PoolingHttpClientConnectionManager, it is very informative.
A note about httpPut.releaseConnection(): this does not actually release a connection, it only ensures that you can re-use the "httpPut" object in a next request (see the apidocs, follow the shown link). Also note that in your code for the "httpGet", you call releaseConnection() on "httpPut" instead of "httpGet".
Ran into this problem just a while back. In case someone else comes across this problem, this post might be useful.
I am using a Java Servlet to service my requests. When I wrote to the response stream using the PrintWriter instance my client blocked. Tried writing to the OutputStream directlyresponse.getOutputStream.write("myresponse") and it worked.
My program is supposed to make multiple POST requests to a https site and I need it to do a SSL-handshake each time it does a new request.
However it seems to only do the handshake the first time and then use the existing connection to do the other requests without a new handshake. I'm sure it doesn't do the handshake later times, because the first time it takes about 700 ms to do the request and receive a response, but later ones only take about 30 ms.
Here's how I initialize the client: (Am I missing some property here?)
SSLContext context = SSLContext.getInstance("SSL");
context.init(kms, trustAllCerts, null);
SSLContext.setDefault(context);
ClientConfig config = new DefaultClientConfig();
config.getProperties().put(ClientConfig.PROPERTY_READ_TIMEOUT, timeout * 1000);
config.getProperties().put(ClientConfig.PROPERTY_FOLLOW_REDIRECTS, false);
config.getFeatures().put(ClientConfig.FEATURE_DISABLE_XML_SECURITY, true);
client = Client.create(config);
And here's how I make the actual request:
ClientResponse cr = service.type(MediaType.APPLICATION_XML).accept(MediaType.APPLICATION_XML).post(ClientResponse.class, batch);
(The service variable is the Builder class, which creates the ClientRequest. It's where the URL is specified.)
Any ideas, please?
HTTP has persistent connections, and SSL has resumable sessions, both of which are specifically intended to prevent what you are trying to accomplish.
You probably turn them both off somehow for your testing purposes, if you delve into the innards of your servers, but I really don't see the point of testing a configuration you would be mad to deploy in production.
I am trying to use httpclient to verify if a specific endpoint is reachable. It seems that it is only possible to check if the server is up but cannot verify if the actual resource is available.
Here is my code:
HttpClient client = new Default HttpClient();
client.execute(new HttpOptions(url)).getStatusLine().getStatusCode();
According to the protocol (http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html) it should have been checking the status of the specific resource but I always get a 200 response as long the the actual host is reachable. What am I doing wrong here?
Thank you.
I'm trying to use the Apache/Jakarta HttpClient 4.1.1 to connect to an arbitrary web page using the given credentials. To test this, I have a minimal install of IIS 7.5 on my dev machine running where only one authentication mode is active at a time. Basic authentication works fine, but Digest and NTLM return 401 error messages whenever I try to log in. Here is my code:
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpContext localContext = new BasicHttpContext();
HttpGet httpget = new HttpGet("http://localhost/");
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(AuthScope.ANY,
new NTCredentials("user", "password", "", "localhost"));
if (!new File(System.getenv("windir") + "\\krb5.ini").exists()) {
List<String> authtypes = new ArrayList<String>();
authtypes.add(AuthPolicy.NTLM);
authtypes.add(AuthPolicy.DIGEST);
authtypes.add(AuthPolicy.BASIC);
httpclient.getParams().setParameter(AuthPNames.PROXY_AUTH_PREF,
authtypes);
httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF,
authtypes);
}
localContext.setAttribute(ClientContext.CREDS_PROVIDER, credsProvider);
HttpResponse response = httpclient.execute(httpget, localContext);
System.out.println("Response code: " + response.getStatusLine());
The one thing I've noticed in Fiddler is that the hashes sent by Firefox versus by HttpClient are different, making me think that maybe IIS 7.5 is expecting stronger hashing than HttpClient provides? Any ideas? It'd be great if I could verify that this would work with NTLM. Digest would be nice too, but I can live without that if necessary.
I am not an expert on the subject but during the NTLM authentication using http components I have seen that the client needs 3 attempts in order to connect to an NTML endpoint in my case. It is kinda described here for Spnego but it is a bit different for the NTLM authentication.
For NTLM in the first attempt client will make a request with Target auth state: UNCHALLENGED and Web server returns HTTP 401 status and a header: WWW-Authenticate: NTLM
Client will check for the configured Authentication schemes, NTLM should be configured in client code.
Second attempt, client will make a request with Target auth state: CHALLENGED, and will send an authorization header with a token encoded in base64 format: Authorization: NTLM TlRMTVNTUAABAAAAAYIIogAAAAAoAAAAAAAAACgAAAAFASgKAAAADw==
Server again returns HTTP 401 status but the header: WWW-Authenticate: NTLM now is populated with encoded information.
3rd Attempt Client will use the information from WWW-Authenticate: NTLM header and will make the final request with Target auth state: HANDSHAKE and an authorisation header Authorization: NTLM which contains more information for the server.
In my case I receive an HTTP/1.1 200 OK after that.
In order to avoid all this in every request documentation at chapter 4.7.1 states that the same execution token must be used for logically related requests. For me it did not worked.
My code:
I initialize the client once in a #PostConstruct method of an EJB
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
cm.setMaxTotal(18);
cm.setDefaultMaxPerRoute(6);
RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(30000)
.setConnectTimeout(30000)
.setTargetPreferredAuthSchemes(Arrays.asList(AuthSchemes.NTLM))
.setProxyPreferredAuthSchemes(Arrays.asList(AuthSchemes.BASIC))
.build();
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
new NTCredentials(userName, password, hostName, domainName));
// Finally we instantiate the client. Client is a thread safe object and can be used by several threads at the same time.
// Client can be used for several request. The life span of the client must be equal to the life span of this EJB.
this.httpclient = HttpClients.custom()
.setConnectionManager(cm)
.setDefaultCredentialsProvider(credentialsProvider)
.setDefaultRequestConfig(requestConfig)
.build();
Use the same client instance in every request:
HttpPost httppost = new HttpPost(endPoint.trim());
// HttpClientContext is not thread safe, one per request must be created.
HttpClientContext context = HttpClientContext.create();
response = this.httpclient.execute(httppost, context);
Deallocate the resources and return the connection back to connection manager, at the #PreDestroy method of my EJB:
this.httpclient.close();
I had the same problem with HttpClient4.1.X After upgrading it to
HttpClient 4.2.6 it woked like charm. Below is my code
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpContext localContext = new BasicHttpContext();
HttpGet httpget = new HttpGet("url");
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(AuthScope.ANY,
new NTCredentials("username", "pwd", "", "domain"));
List<String> authtypes = new ArrayList<String>();
authtypes.add(AuthPolicy.NTLM);
httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF,authtypes);
localContext.setAttribute(ClientContext.CREDS_PROVIDER, credsProvider);
HttpResponse response = httpclient.execute(httpget, localContext);
HttpEntity entity=response.getEntity();
The easiest way troubleshoot such situations I found is Wireshark. It is a very big hammer, but it really will show you everything. Install it, make sure your server is on another machine (does not work with Localhost) and start logging.
Run your request that fails, run one that works. Then, filter by http (just put http in the filter field), find the first GET request, find the other GET request and compare. Identify meaningful difference, you now have specific keywords or issues to search code/net for. If not enough, narrow down to first TCP conversation and look at full request/response. Same with the other one.
I solved an unbelievable number of problems with that approach. And Wireshark is very useful tool to know. Lots of super-advanced functions to make your network debugging easier.
You can also run it on either client or server end. Whatever will show you both requests to allow you to compare.
I had a similar problem with HttpClient 4.1.2. For me, it was resolved by reverting to HttpClient 4.0.3. I could never get NTLM working with 4.1.2 using either the built-in implementation or using JCIFS.
Updating our application to use the jars in the httpcomponents-client-4.5.1 resolved this issue for me.
I finally figured it out. Digest authentication requires that if you use a full URL in the request, the proxy also needs to use the full URL. I did not leave the proxy code in the sample, but it was directed to "localhost", which caused it to fail. Changing this to 127.0.0.1 made it work.