I am using httpClient version 4.5 to connect with our external vendor site. We do not need any connection pool or persistent connection, so I am using the BasicHttpClientConnectionManager to create the HttpClient.
This works fine for minimal number of requests, but if I test this for 1TPS for 1 hour, by the end of the test, we start seeing intermittent connection resets. (guessing request count > 100)
I/O exception (java.net.SocketException) caught when processing request to {s}->https://apiURL:443: Connection reset
Please find below the code snippet for making connection.
SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(kmf.getKeyManagers(), new X509TrustManager[] { new DefaultTrustManager() }, new SecureRandom());
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslContext, new String[] { "TLSv1.2" }, null,SSLConnectionSocketFactory.getDefaultHostnameVerifier());
Registry<ConnectionSocketFactory> socketFactoryRegistry = RegistryBuilder.<ConnectionSocketFactory>create()
.register("https", sslsf).register("http", new PlainConnectionSocketFactory()).build();
HttpClientConnectionManager connectionManager = new BasicHttpClientConnectionManager(socketFactoryRegistry);
HttpRequestRetryHandler retryHandler = new DefaultHttpRequestRetryHandler(1, false);
RequestConfig defaultRequestConfig = RequestConfig.custom().setSocketTimeout(Integer.parseInt(30000)).setConnectTimeout(Integer.parseInt(30000)).setConnectionRequestTimeout(30000).setCookieSpec(CookieSpecs.STANDARD).build();
CloseableHttpClient httpClient = HttpClients.custom().setConnectionManager(connectionManager).setDefaultRequestConfig(defaultRequestConfig).setRetryHandler(retryHandler).evictExpiredConnections().build();
HttpPost httpPost = new HttpPost(<endpoint>);
httpPost.setEntity(new UrlEncodedFormEntity(requestData));
httpResponse = httpClient.execute(httpPost);
I saw that fix for a similar issue reported is already available with version 4.5. (Ref: https://issues.apache.org/jira/browse/HTTPCLIENT-1655) provided by Oleg
If thats the case, not sure why I am still facing this issue. Could someone please help on this.
Thanks!
Hi Oleg,
I had been using the httpclient version 4.5.3 when I was still seeing the connection reset errors as posted above.
Later noticed that the fix for reset issue was committed to the version 4.5.1 (https://issues.apache.org/jira/browse/HTTPCLIENT-1655). So, just tried updating that particular version, ran a test and not seeing the connection reset errors any more.
I had expected that this fix should also be available in the higher versions starting from 4.5.1. But, I guess its somehow missed out in the higher versions, have verified that its still an issue with version 4.5.3 for sure.
So, conclusion is that the connection reset error was fixed using httpclient 4.5.1 jar.
Thanks!
Related
public HttpResponseBean get(String url, Map<String, String> headers) throws Exception {
logger.debug("Sending get request...");
HttpClient httpClient = null;
try {
int timeout = 30 * 1000; // 30 seconds
RequestConfig requestConfig = RequestConfig.custom()
.setConnectTimeout(timeout)
.setConnectionRequestTimeout(timeout)
.setSocketTimeout(timeout).build();
httpClient = HttpClients.custom()
.setDefaultRequestConfig(requestConfig).build();
HttpGet httpGetRequest = new HttpGet(url);
if (headers != null) {
for (Entry<String, String> entry: headers.entrySet()) {
httpGetRequest.addHeader(new BasicHeader(entry.getKey(), entry.getValue()));
}
}
HttpResponse response = httpClient.execute(httpGetRequest);
HttpResponseBean hrb = new HttpResponseBean(response);
logger.debug("Get response: Response: " + hrb.toString());
return hrb;
} finally {
closeConnection(httpClient);
}
}
This works well most of the time...but once in a while it gets stuck on handshake and will take forever until the server(tomcat) is restarted.
As per this link looks like a bug -> apache httpclient 4.3 not timing out
Is there a way out of this? I am using httpclient 4.4.1
Is there a way out of this? I am using httpclient 4.4.1
Here's the associated bug on the Apache site. It looks like people have had problems with the 4.4.1 version:
I had this issue with version 4.4.1. At first I ignored this thread because it was flagged as resolved in 4.3.
But this seemed to have been resolved in the 4.5.1 version. I'd encourage you to upgrade if possible.
I encountered this issue on version 4.3.4. I upgraded to 4.5.1 and the issue was fixed there.
Here's additional information:
In version 4.3.4, http:// worked fine and timed out after 1 second. With https, the request hangs beyond 1 seconds and hangs until the [...] server closes the TCP connection.
In version 4.5.1, the [correct] http behaviour was identical, and on https I got the following exception: [...] org.apache.http.conn.ConnectTimeoutException: Connect to localhost:6171 [localhost/127.0.0.1] failed: Read timed out
Hope this helps.
I have an http2 server at https://ec2-52-57-54-142.eu-central-1.compute.amazonaws.com/ with a self-signed cert. And I have a jetty http2 client that simply posts things to it. For some reason, I'm not getting ignoring self-signed cert to work. Here's a snippet of relevant code
SslContextFactory factory = new SslContextFactory(true);
factory.setTrustAll(true);
factory.setValidateCerts(false);
factory.setValidatePeerCerts(false);
factory.setEndpointIdentificationAlgorithm(null);
SSLContext sslContext = factory.getSslContext();
if(null == sslContext) {
sslContext = SSLContext.getInstance("TLS");
}
TrustManager[] verifiers = new TrustManager[] {...// some dummy trust manager that always passes};
sslContext.init(null, verifiers, null);
factory.setSslContext(sslContext);
HttpClientTransportOverHTTP2 httpClientTransportOverHTTP2
= new HttpClientTransportOverHTTP2(new HTTP2Client());
HttpClient httpClient = new HttpClient(httpClientTransportOverHTTP2, factory);
Request request = httpClient.POST(destination);
ContentProvider contentProvider = new InputStreamContentProvider(new StringInputStream(payload));
request.content(contentProvider);
ContentResponse response = request.send();
And I get these stacktrace
Caused by: java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException
at org.eclipse.jetty.client.util.FutureResponseListener.getResult(FutureResponseListener.java:118)
at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:101)
at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:652)
at my code
... 34 more
Caused by: java.nio.channels.ClosedChannelException
at org.eclipse.jetty.io.WriteFlusher.onClose(WriteFlusher.java:498)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onIncompleteFlush(SslConnection.java:409)
at org.eclipse.jetty.io.AbstractEndPoint$2.onIncompleteFlush(AbstractEndPoint.java:54)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:322)
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:140)
at org.eclipse.jetty.http2.HTTP2Flusher.process(HTTP2Flusher.java:243)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at org.eclipse.jetty.util.IteratingCallback.succeeded(IteratingCallback.java:365)
at org.eclipse.jetty.http2.HTTP2Flusher.succeeded(HTTP2Flusher.java:258)
at org.eclipse.jetty.io.WriteFlusher$PendingState.complete(WriteFlusher.java:269)
at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:394)
at org.eclipse.jetty.io.ssl.SslConnection$1.run(SslConnection.java:101)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
... 1 more
When I step through things with a debugger, I see something about NOT_HANDSHAKE in one of the connection objects.
I really don't care about verifying anything. I just want to connect over HTTP2/TLS. I searched for many different terms, but they all end up with more or less the same thing (setTrustAll, custom TrustManager, etc)
Any help? Thanks!
P.S. Jetty version 9.3.12
Your client code is correct, although redundant.
It is enough to do:
SslContextFactory sslContextFactory = new SslContextFactory(true);
HTTP2Client http2Client = new HTTP2Client();
HttpClient httpClient = new HttpClient(new HttpClientTransportOverHTTP2(http2Client), sslContextFactory);
httpClient.start();
ContentResponse response = httpClient.GET("https://ec2-52-57-54-142.eu-central-1.compute.amazonaws.com/");
If you enable DEBUG logging on the Jetty HTTP/2 client, you will see that the client receives:
2016-10-05 09:20:33.102:DBUG:oejhp.Parser:qtp1897115967-15: Parsed GO_AWAY frame header from java.nio.HeapByteBuffer[pos=9 lim=35 cap=16384]
2016-10-05 09:20:33.103:DBUG:oejh.HTTP2Session:qtp1897115967-15: Received GoAwayFrame#3bc447d3,0/INADEQUATE_SECURITY_ERROR/Unknown error code
So the problem is that the server thinks that the security is inadequate (the GOAWAY frame arrives with error code INADEQUATE_SECURITY_ERROR).
At this point, the problem is on the server. You have to figure out why the server thinks the security is inadequate. Probably just a matter of configuration on the server.
Try
transport.setUseALPN(false);
This is scenario I am testing. The steps are as follows
Create an instance of HttpClient(org.apache.commons.httpclient.HttpClient)3.1
Use the instance to get response from a url (http://www.example.com/submitData)
Bring down the server hosting the url mentioned in step 2.
Bring up the url
Wait till the url is up
Use the instance of HttpClient created in step 1 to get response from the url (http://www.example.com/getData)
I get an error - Connection Reset.
Can anyone help me understand.
I have used the following params in the HttpClient
httpClient.getParams().setParameter("http.socket.timeout", new Integer(0));
httpClient.getParams().setParameter("http.connection.stalecheck", new Boolean(true));
Edited after comments from Peter
This is how I create the HttpClient
HttpClient httpClient = new HttpClient();
AuthScope authScope = new AuthScope("www.example.com", 80, AuthScope.ANY_REALM);
Credentials defaultcreds = new UsernamePasswordCredentials("username", "pwd");
httpClient.getState().setCredentials(authScope, defaultcreds);
httpClient.getParams().setParameter("http.socket.timeout", new Integer(0));
httpClient.getParams().setParameter("http.connection.stalecheck", new Boolean(true));
To establish connection, I use the following
GetMethod getMethod = new GetMethod("http://"+httpClient.getHostConfiguration().getHost()+"/getData");
int statusCode = httpClient.executeMethod(getMethod);
I want the httpclient instance to preserve the hostname and port and credentials.
This is exact execption message that I get - "Connection reset".
More info - I am executing the test from eclipse. If I run the test case in debug mode, I dont get this exception.
I think the behaviour is reasonable because when the server goes down the client socket on the server goes away too. The client still holds old socket but this socket is gone on the server. You should reconnect from client.
AFAIK the stale parameter allows the client to close the connection the clean way (without Exception) but not prevent from disconnection.
This looks suspicious
httpClient.getParams().setParameter("http.socket.timeout", new Integer(0));
Having a timeout set to 0ms would likely terminate the request prematurely?
I have an application connecting to sites that require basic authentication. The sites are provided at run time and not known at compile time.
I am using HttpClient 4.2.
I am not sure if the code below is how I am supposed to specify basic authentication, but the documentation would suggest it is. However, I don't know what to pass in the constructor of AuthScope. I had thought that a null parameter meant that the credentials supplied should be used for all URLs, but it throws a NullPointerException, so clearly I am wrong.
m_client = new DefaultHttpClient();
UsernamePasswordCredentials credentials = new UsernamePasswordCredentials(m_userName, m_password);
((DefaultHttpClient)m_client).getCredentialsProvider().setCredentials(new AuthScope((HttpHost)null), credentials);
AuthScope.ANY is what you're after: http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/auth/AuthScope.html
Try this:
final HttpClient client = new HttpClient();
client.getParams().setAuthenticationPreemptive(true);
client.getState().setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user.getUsername(), user.getPassword()));
final GetMethod method = new GetMethod(uri);
client.executeMethod(method);
From at least version 4.2.3 (I guess after version 3.X), the accepted answer is no longer valid. Instead, do something like:
private HttpClient createClient() {
HttpParams params = new BasicHttpParams();
HttpProtocolParams.setContentCharset(params, "UTF-8");
Credentials credentials = new UsernamePasswordCredentials("user", "password");
DefaultHttpClient httpclient = new DefaultHttpClient(params);
httpclient.getCredentialsProvider().setCredentials(AuthScope.ANY, credentials);
return httpclient;
}
The JavaDoc for AuthScope.ANY says In the future versions of HttpClient the use of this parameter will be discontinued, so use it at your own risk. A better option would be to use one of the constructors defined in AuthScope.
For a discussion on how to make requests preemptive, see:
Preemptive Basic authentication with Apache HttpClient 4
I'm trying to use the Apache/Jakarta HttpClient 4.1.1 to connect to an arbitrary web page using the given credentials. To test this, I have a minimal install of IIS 7.5 on my dev machine running where only one authentication mode is active at a time. Basic authentication works fine, but Digest and NTLM return 401 error messages whenever I try to log in. Here is my code:
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpContext localContext = new BasicHttpContext();
HttpGet httpget = new HttpGet("http://localhost/");
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(AuthScope.ANY,
new NTCredentials("user", "password", "", "localhost"));
if (!new File(System.getenv("windir") + "\\krb5.ini").exists()) {
List<String> authtypes = new ArrayList<String>();
authtypes.add(AuthPolicy.NTLM);
authtypes.add(AuthPolicy.DIGEST);
authtypes.add(AuthPolicy.BASIC);
httpclient.getParams().setParameter(AuthPNames.PROXY_AUTH_PREF,
authtypes);
httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF,
authtypes);
}
localContext.setAttribute(ClientContext.CREDS_PROVIDER, credsProvider);
HttpResponse response = httpclient.execute(httpget, localContext);
System.out.println("Response code: " + response.getStatusLine());
The one thing I've noticed in Fiddler is that the hashes sent by Firefox versus by HttpClient are different, making me think that maybe IIS 7.5 is expecting stronger hashing than HttpClient provides? Any ideas? It'd be great if I could verify that this would work with NTLM. Digest would be nice too, but I can live without that if necessary.
I am not an expert on the subject but during the NTLM authentication using http components I have seen that the client needs 3 attempts in order to connect to an NTML endpoint in my case. It is kinda described here for Spnego but it is a bit different for the NTLM authentication.
For NTLM in the first attempt client will make a request with Target auth state: UNCHALLENGED and Web server returns HTTP 401 status and a header: WWW-Authenticate: NTLM
Client will check for the configured Authentication schemes, NTLM should be configured in client code.
Second attempt, client will make a request with Target auth state: CHALLENGED, and will send an authorization header with a token encoded in base64 format: Authorization: NTLM TlRMTVNTUAABAAAAAYIIogAAAAAoAAAAAAAAACgAAAAFASgKAAAADw==
Server again returns HTTP 401 status but the header: WWW-Authenticate: NTLM now is populated with encoded information.
3rd Attempt Client will use the information from WWW-Authenticate: NTLM header and will make the final request with Target auth state: HANDSHAKE and an authorisation header Authorization: NTLM which contains more information for the server.
In my case I receive an HTTP/1.1 200 OK after that.
In order to avoid all this in every request documentation at chapter 4.7.1 states that the same execution token must be used for logically related requests. For me it did not worked.
My code:
I initialize the client once in a #PostConstruct method of an EJB
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
cm.setMaxTotal(18);
cm.setDefaultMaxPerRoute(6);
RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(30000)
.setConnectTimeout(30000)
.setTargetPreferredAuthSchemes(Arrays.asList(AuthSchemes.NTLM))
.setProxyPreferredAuthSchemes(Arrays.asList(AuthSchemes.BASIC))
.build();
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
new NTCredentials(userName, password, hostName, domainName));
// Finally we instantiate the client. Client is a thread safe object and can be used by several threads at the same time.
// Client can be used for several request. The life span of the client must be equal to the life span of this EJB.
this.httpclient = HttpClients.custom()
.setConnectionManager(cm)
.setDefaultCredentialsProvider(credentialsProvider)
.setDefaultRequestConfig(requestConfig)
.build();
Use the same client instance in every request:
HttpPost httppost = new HttpPost(endPoint.trim());
// HttpClientContext is not thread safe, one per request must be created.
HttpClientContext context = HttpClientContext.create();
response = this.httpclient.execute(httppost, context);
Deallocate the resources and return the connection back to connection manager, at the #PreDestroy method of my EJB:
this.httpclient.close();
I had the same problem with HttpClient4.1.X After upgrading it to
HttpClient 4.2.6 it woked like charm. Below is my code
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpContext localContext = new BasicHttpContext();
HttpGet httpget = new HttpGet("url");
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(AuthScope.ANY,
new NTCredentials("username", "pwd", "", "domain"));
List<String> authtypes = new ArrayList<String>();
authtypes.add(AuthPolicy.NTLM);
httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF,authtypes);
localContext.setAttribute(ClientContext.CREDS_PROVIDER, credsProvider);
HttpResponse response = httpclient.execute(httpget, localContext);
HttpEntity entity=response.getEntity();
The easiest way troubleshoot such situations I found is Wireshark. It is a very big hammer, but it really will show you everything. Install it, make sure your server is on another machine (does not work with Localhost) and start logging.
Run your request that fails, run one that works. Then, filter by http (just put http in the filter field), find the first GET request, find the other GET request and compare. Identify meaningful difference, you now have specific keywords or issues to search code/net for. If not enough, narrow down to first TCP conversation and look at full request/response. Same with the other one.
I solved an unbelievable number of problems with that approach. And Wireshark is very useful tool to know. Lots of super-advanced functions to make your network debugging easier.
You can also run it on either client or server end. Whatever will show you both requests to allow you to compare.
I had a similar problem with HttpClient 4.1.2. For me, it was resolved by reverting to HttpClient 4.0.3. I could never get NTLM working with 4.1.2 using either the built-in implementation or using JCIFS.
Updating our application to use the jars in the httpcomponents-client-4.5.1 resolved this issue for me.
I finally figured it out. Digest authentication requires that if you use a full URL in the request, the proxy also needs to use the full URL. I did not leave the proxy code in the sample, but it was directed to "localhost", which caused it to fail. Changing this to 127.0.0.1 made it work.