I use a httpClient of apache in java to call REST APIs which is configured as below in a static block of class, what I want is to change the connection timeout per request. is it possible? how?
static {
PoolingHttpClientConnectionManager pool = new PoolingHttpClientConnectionManager();
pool.setMaxTotal(ChatSettings.HTTP_CLIENT_THREAD_POOL_SIZE);
httpClient = HttpClients
.custom()
.setConnectionManager(pool)
.setDefaultRequestConfig(
RequestConfig
.custom()
.setConnectTimeout(DEFAULT_HTTP_TIMEOUT)
.setSocketTimeout(DEFAULT_HTTP_TIMEOUT)
.build()
)
.build();
}
One thing you can implement for sure is to follow this example: https://github.com/apache/httpcomponents-client/blob/5.1.x/httpclient5/src/test/java/org/apache/hc/client5/http/examples/ClientAbortMethod.java
It demonstrates how a client can fire a request, but eventually change it's mind and tell the server it no longer wants to receive the response.
Combine this with a 'watchdog' thread that fires after some time/when the time runs out.
I am instantiating an Apache HTTP Components HttpClient using the following code:
CloseableHttpClient httpClient = HttpClients.custom()
.setProxy(new HttpHost(proxyServerAddress, proxyServerPort))
.disableConnectionState()
.disableCookieManagement()
.build();
But I would like to set the proxy only if an property (e.g. useProxy) is set to true. I can use an if-then-else pair of blocks based on the property value, but I was wondering if there is a better way to achieve this? My goal is to externalize the control of whether or not to use a proxy, using a configuration file property or via JAVA_OPTS.
How about:
HttpClientBuilder builder = HttpClients.custom()
.disableConnectionState()
.disableCookieManagement();
if( useProxy )
builder = builder.setProxy(new HttpHost(proxyServerAddress, proxyServerPort));
CloseableHttpClient httpClient = builder.build();
Today I was trying to configure the HttpComponentsMessageSender which uses Apache's org.apache.httpcomponents.httpclient library. Before I used the standard Java classes (I think java.net.HttpUrlMessageSender or something along those lines).
My HTTP methods need to authenticate using NTLM and the software is running under Linux so I cannot use the default Windows mechanism to do authentication, but instead I must configure authentication inside the Java application.
Originally I was just using a custom implementation of java.net.Authenticator which I used to set the username and password
for the NTLM access. But when I switched to HttpComponentsMessageSender this approach did not work anymore. When I tried to setup a custom HttpClient configuration with the HttpComponentsMessageSender I ran into various issues which I thought I would document here.
I am going to post my own answer but if anyone has a better solution, please feel free to add your solution.
I solved this adding a custom build HttpClient object which I built with the HttpClientBuilder. However adding this caused the following exception to appear:
org.apache.http.ProtocolException: Content-Length header already present
The solution which I found in this answer was to add a HttpRequestInterceptor.
Using the setConnectionTimeout on the HttpComponentsMessageSender did not work anymore with my custom HttpClient object. Instead I had to inject a RequestConfig object into the HttpClientBuilder.
The NTCredentials object required the domain name as a parameters. Earlier when I was using the Authenticator interface from java.net it was enough to just supply the username and password.
This is the code I am using currently:
HttpComponentsMessageSender messageSender = new HttpComponentsMessageSender ();
AuthScope authscope;
NTCredentials credentials;
CredentialsProvider credentialsProvider;
Registry<AuthSchemeProvider> registry;
RequestConfig requestConfig;
authscope = new AuthScope (HOST_IP, HOST_PORT);
credentials = new NTCredentials ("user", "pass", null, "domain");
credentialsProvider = new BasicCredentialsProvider ();
credentialsProvider.setCredentials (authscope, credentials);
registry = RegistryBuilder.<AuthSchemeProvider>create ()
.register(AuthSchemes.NTLM, new NTLMSchemeFactory ())
.build ();
HttpRequestInterceptor interceptor
= (request, context) -> request.removeHeaders(HTTP.CONTENT_LEN);
requestConfig = RequestConfig.custom ()
.setConnectTimeout (3000)
.build ();
HttpClient httpClient
= HttpClientBuilder.create ()
.setDefaultRequestConfig (requestConfig)
.setDefaultAuthSchemeRegistry (registry)
.setDefaultCredentialsProvider (credentialsProvider)
.addInterceptorFirst (interceptor)
.build ();
messageSender.setHttpClient (httpClient);
I am having a problem getting the Apache HttpClient to connect to a service external to my virtualised development environment.
To access the internet (e.g. api.twitter.com) I need to call a local URL (e.g. api.twitter.com.dev.mycompany.net), which then forwards the request to real host.
The problem is, that to whatever request I send, I get a 404 Not Found response.
I have tried debugging it using wget, and it appears the problem is, that the destination server identifies the desired resource by using both the request URL and the hostname in the Host header. Since the hostname does not match, it is unable to locate the resource.
I have (unsuccessfully) tried to override the Host header by setting the http.virtual-host parameter on the client like this:
HttpClient client = new DefaultHttpClient();
if (envType.isWithProxy()) {
client.getParams().setParameter(ClientPNames.VIRTUAL_HOST, "api.twitter.com");
}
Technical details:
Client is used as an executor in RESTeasy to call the REST API. So "manually" setting the virtual host (as described here) is not an option.
Everything is done via HTTPS/SSL - not that I think it makes a difference.
Edit 1: Using a HttpHost instead of a String does not have the desired effect either:
HttpClient client = new DefaultHttpClient();
if (envType.isWithProxy()) {
HttpHost realHost = new HttpHost("api.twitter.com", port, scheme);
client.getParams().setParameter(ClientPNames.VIRTUAL_HOST, realHost);
}
Edit 2: Further investigation has revealed, that the parameter needs to be set on the request object. The following is the code v. 4.2-aplha1 of HttpClient setting the virtual host:
HttpRequest orig = request;
RequestWrapper origWrapper = wrapRequest(orig);
origWrapper.setParams(params);
HttpRoute origRoute = determineRoute(target, origWrapper, context);
virtualHost = (HttpHost) orig.getParams().getParameter(
ClientPNames.VIRTUAL_HOST);
paramsare the parameters passed from the client. But the value for 'virtualHost' is read from the request parameters.
So this changes the nature of the question to: How do I set the VIRTUAL_HOST property on the requests?
ClientPNames.VIRTUAL_HOST is the right parameter for overriding physical host name in HTTP requests. I would just recommend setting this parameter on the request object instead of the client object. If that does not produce the desired effect please post the complete wire / context log of the session (see logging guide for instructions) either here or to the HttpClient user list.
Follow-up
OK. Let's take a larger sledge hammer. One can override content of the Host header using an interceptor.
DefaultHttpClient client = new DefaultHttpClient();
client.addRequestInterceptor(new HttpRequestInterceptor() {
public void process(
final HttpRequest request,
final HttpContext context) throws HttpException, IOException {
request.setHeader(HTTP.TARGET_HOST, "www.whatever.com");
}
});
One can make the interceptor clever enough to override the header selectively, only for specific hosts.
I'm trying to use the Apache/Jakarta HttpClient 4.1.1 to connect to an arbitrary web page using the given credentials. To test this, I have a minimal install of IIS 7.5 on my dev machine running where only one authentication mode is active at a time. Basic authentication works fine, but Digest and NTLM return 401 error messages whenever I try to log in. Here is my code:
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpContext localContext = new BasicHttpContext();
HttpGet httpget = new HttpGet("http://localhost/");
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(AuthScope.ANY,
new NTCredentials("user", "password", "", "localhost"));
if (!new File(System.getenv("windir") + "\\krb5.ini").exists()) {
List<String> authtypes = new ArrayList<String>();
authtypes.add(AuthPolicy.NTLM);
authtypes.add(AuthPolicy.DIGEST);
authtypes.add(AuthPolicy.BASIC);
httpclient.getParams().setParameter(AuthPNames.PROXY_AUTH_PREF,
authtypes);
httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF,
authtypes);
}
localContext.setAttribute(ClientContext.CREDS_PROVIDER, credsProvider);
HttpResponse response = httpclient.execute(httpget, localContext);
System.out.println("Response code: " + response.getStatusLine());
The one thing I've noticed in Fiddler is that the hashes sent by Firefox versus by HttpClient are different, making me think that maybe IIS 7.5 is expecting stronger hashing than HttpClient provides? Any ideas? It'd be great if I could verify that this would work with NTLM. Digest would be nice too, but I can live without that if necessary.
I am not an expert on the subject but during the NTLM authentication using http components I have seen that the client needs 3 attempts in order to connect to an NTML endpoint in my case. It is kinda described here for Spnego but it is a bit different for the NTLM authentication.
For NTLM in the first attempt client will make a request with Target auth state: UNCHALLENGED and Web server returns HTTP 401 status and a header: WWW-Authenticate: NTLM
Client will check for the configured Authentication schemes, NTLM should be configured in client code.
Second attempt, client will make a request with Target auth state: CHALLENGED, and will send an authorization header with a token encoded in base64 format: Authorization: NTLM TlRMTVNTUAABAAAAAYIIogAAAAAoAAAAAAAAACgAAAAFASgKAAAADw==
Server again returns HTTP 401 status but the header: WWW-Authenticate: NTLM now is populated with encoded information.
3rd Attempt Client will use the information from WWW-Authenticate: NTLM header and will make the final request with Target auth state: HANDSHAKE and an authorisation header Authorization: NTLM which contains more information for the server.
In my case I receive an HTTP/1.1 200 OK after that.
In order to avoid all this in every request documentation at chapter 4.7.1 states that the same execution token must be used for logically related requests. For me it did not worked.
My code:
I initialize the client once in a #PostConstruct method of an EJB
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
cm.setMaxTotal(18);
cm.setDefaultMaxPerRoute(6);
RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(30000)
.setConnectTimeout(30000)
.setTargetPreferredAuthSchemes(Arrays.asList(AuthSchemes.NTLM))
.setProxyPreferredAuthSchemes(Arrays.asList(AuthSchemes.BASIC))
.build();
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
new NTCredentials(userName, password, hostName, domainName));
// Finally we instantiate the client. Client is a thread safe object and can be used by several threads at the same time.
// Client can be used for several request. The life span of the client must be equal to the life span of this EJB.
this.httpclient = HttpClients.custom()
.setConnectionManager(cm)
.setDefaultCredentialsProvider(credentialsProvider)
.setDefaultRequestConfig(requestConfig)
.build();
Use the same client instance in every request:
HttpPost httppost = new HttpPost(endPoint.trim());
// HttpClientContext is not thread safe, one per request must be created.
HttpClientContext context = HttpClientContext.create();
response = this.httpclient.execute(httppost, context);
Deallocate the resources and return the connection back to connection manager, at the #PreDestroy method of my EJB:
this.httpclient.close();
I had the same problem with HttpClient4.1.X After upgrading it to
HttpClient 4.2.6 it woked like charm. Below is my code
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpContext localContext = new BasicHttpContext();
HttpGet httpget = new HttpGet("url");
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(AuthScope.ANY,
new NTCredentials("username", "pwd", "", "domain"));
List<String> authtypes = new ArrayList<String>();
authtypes.add(AuthPolicy.NTLM);
httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF,authtypes);
localContext.setAttribute(ClientContext.CREDS_PROVIDER, credsProvider);
HttpResponse response = httpclient.execute(httpget, localContext);
HttpEntity entity=response.getEntity();
The easiest way troubleshoot such situations I found is Wireshark. It is a very big hammer, but it really will show you everything. Install it, make sure your server is on another machine (does not work with Localhost) and start logging.
Run your request that fails, run one that works. Then, filter by http (just put http in the filter field), find the first GET request, find the other GET request and compare. Identify meaningful difference, you now have specific keywords or issues to search code/net for. If not enough, narrow down to first TCP conversation and look at full request/response. Same with the other one.
I solved an unbelievable number of problems with that approach. And Wireshark is very useful tool to know. Lots of super-advanced functions to make your network debugging easier.
You can also run it on either client or server end. Whatever will show you both requests to allow you to compare.
I had a similar problem with HttpClient 4.1.2. For me, it was resolved by reverting to HttpClient 4.0.3. I could never get NTLM working with 4.1.2 using either the built-in implementation or using JCIFS.
Updating our application to use the jars in the httpcomponents-client-4.5.1 resolved this issue for me.
I finally figured it out. Digest authentication requires that if you use a full URL in the request, the proxy also needs to use the full URL. I did not leave the proxy code in the sample, but it was directed to "localhost", which caused it to fail. Changing this to 127.0.0.1 made it work.