We are currently using Apache HttpClient (5) to send a POST request to a server. The response is sent back to us using server side events (SSE) with multiple payloads using the standard format:
data: {...}
Currently we have code like this which sends the request and receives the response:
// Set the socket timeout
final ConnectionConfig connConfig = ConnectionConfig.custom()
.setSocketTimeout(socketTimeout, TimeUnit.MILLISECONDS)
.build();
// Custom config
final BasicHttpClientConnectionManager cm = new BasicHttpClientConnectionManager();
cm.setConnectionConfig(connConfig);
// Build the client
try (final CloseableHttpClient client = HttpClientBuilder.create().setConnectionManager(cm).build()) {
// Execute the request
return client.execute(request.getRequest(),
// Get and process the response
response -> HttpResponse.builder()
.withCode(response.getCode())
.withContent(EntityUtils.toByteArray(response.getEntity()))
.build()
);
}
This all works great except I need to access the individual incoming response payloads (data {...}) in the response as they arrive instead of waiting for them all to finish before being able to access the response.
If this isn't possible using Apache I am open to other options providing they can send a normal HTTP(S) POST.
Ok I couldn't get it to work with the Apache Client but I did find this video that showed how to do it using the native HttpClient.
// Create the client
final HttpClient client = HttpClient.newHttpClient();
// Make the request
final HttpResponse<Stream<String>> response = client.send(request, HttpResponse.BodyHandlers.ofLines());
// Status code check
if (response.statusCode() != 200) ...;
// This will consume individual events as they arrive
response.body().forEach(System.out::println);
I would like to use HTTP authentication with a Jersey 2 client using ApacheConnectorProvider, but I want to set it for each request (not as ClientConfig property). The reason is that I use the client for multiple connections, only some of which require HTTP authentication. I assume it is better to not recreate the Client object each time I want to send an HTTP request.
What I found so far:
1) from https://github.com/jersey/jersey/blob/master/connectors/apache-connector/src/test/java/org/glassfish/jersey/apache/connector/AuthTest.java
CredentialsProvider credentialsProvider = new org.apache.http.impl.client.BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
new UsernamePasswordCredentials("name", "password")
);
ClientConfig cc = new ClientConfig();
cc.property(ApacheClientProperties.CREDENTIALS_PROVIDER, credentialsProvider).property(ApacheClientProperties.PREEMPTIVE_BASIC_AUTHENTICATION, true);
cc.connectorProvider(new ApacheConnectorProvider());
Client client = ClientBuilder.newClient(cc);
WebTarget r = client.target(getBaseUri());
r.request().get(String.class);
This would probably work, but as I reuse the same Client for multiple HTTP requests, not all using HTTP authentication, I'd need to recreate a Client object each time I send a request. I do not know if that is an expensive operation, so it could be a solution.
2) from https://jersey.java.net/documentation/latest/client.html#d0e4833
Response response = client.target("http://localhost:8080/rest/homer/contact").request()
.property(HttpAuthenticationFeature.HTTP_AUTHENTICATION_BASIC_USERNAME, "homer")
.property(HttpAuthenticationFeature.HTTP_AUTHENTICATION_BASIC_PASSWORD, "p1swd745").get();
This does not seem to work with ApacheConnectorProvider.
What is the right solution to my problem?
Thanks!
I am having a problem getting the Apache HttpClient to connect to a service external to my virtualised development environment.
To access the internet (e.g. api.twitter.com) I need to call a local URL (e.g. api.twitter.com.dev.mycompany.net), which then forwards the request to real host.
The problem is, that to whatever request I send, I get a 404 Not Found response.
I have tried debugging it using wget, and it appears the problem is, that the destination server identifies the desired resource by using both the request URL and the hostname in the Host header. Since the hostname does not match, it is unable to locate the resource.
I have (unsuccessfully) tried to override the Host header by setting the http.virtual-host parameter on the client like this:
HttpClient client = new DefaultHttpClient();
if (envType.isWithProxy()) {
client.getParams().setParameter(ClientPNames.VIRTUAL_HOST, "api.twitter.com");
}
Technical details:
Client is used as an executor in RESTeasy to call the REST API. So "manually" setting the virtual host (as described here) is not an option.
Everything is done via HTTPS/SSL - not that I think it makes a difference.
Edit 1: Using a HttpHost instead of a String does not have the desired effect either:
HttpClient client = new DefaultHttpClient();
if (envType.isWithProxy()) {
HttpHost realHost = new HttpHost("api.twitter.com", port, scheme);
client.getParams().setParameter(ClientPNames.VIRTUAL_HOST, realHost);
}
Edit 2: Further investigation has revealed, that the parameter needs to be set on the request object. The following is the code v. 4.2-aplha1 of HttpClient setting the virtual host:
HttpRequest orig = request;
RequestWrapper origWrapper = wrapRequest(orig);
origWrapper.setParams(params);
HttpRoute origRoute = determineRoute(target, origWrapper, context);
virtualHost = (HttpHost) orig.getParams().getParameter(
ClientPNames.VIRTUAL_HOST);
paramsare the parameters passed from the client. But the value for 'virtualHost' is read from the request parameters.
So this changes the nature of the question to: How do I set the VIRTUAL_HOST property on the requests?
ClientPNames.VIRTUAL_HOST is the right parameter for overriding physical host name in HTTP requests. I would just recommend setting this parameter on the request object instead of the client object. If that does not produce the desired effect please post the complete wire / context log of the session (see logging guide for instructions) either here or to the HttpClient user list.
Follow-up
OK. Let's take a larger sledge hammer. One can override content of the Host header using an interceptor.
DefaultHttpClient client = new DefaultHttpClient();
client.addRequestInterceptor(new HttpRequestInterceptor() {
public void process(
final HttpRequest request,
final HttpContext context) throws HttpException, IOException {
request.setHeader(HTTP.TARGET_HOST, "www.whatever.com");
}
});
One can make the interceptor clever enough to override the header selectively, only for specific hosts.
I am developing a web application with struts2&spring3 and one of the last things I need to do is the communication between my server and another server where I have to send a XML file and after wait for its response.
As someone said me, I have implemented the sending of the XML file by HTTP with the library HttpClient4 (from Apache):
File file = new File(fileName);
FileEntity entity = new FileEntity(file, "text/xml; charset=\"UTF-8\"");
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpPost method = new HttpPost(server);
method.setEntity(entity);
HttpResponse response = httpclient.execute(method);
But now I have to implement the waiting and getting of the response that the other server will send me. The problem is that the other server is still not developed, so, which way you think it would be the best to get that answer? by HTTP also would be perfect but I don't know which library and how to do it.
Thank you very much in advance for all your help,
Aleix
Here I find two options
Send response immediately.
Develop a module [client side] which accepts response from server after request is received and added to queue [ Asynchronous mode ]
I'm trying to use the Apache/Jakarta HttpClient 4.1.1 to connect to an arbitrary web page using the given credentials. To test this, I have a minimal install of IIS 7.5 on my dev machine running where only one authentication mode is active at a time. Basic authentication works fine, but Digest and NTLM return 401 error messages whenever I try to log in. Here is my code:
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpContext localContext = new BasicHttpContext();
HttpGet httpget = new HttpGet("http://localhost/");
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(AuthScope.ANY,
new NTCredentials("user", "password", "", "localhost"));
if (!new File(System.getenv("windir") + "\\krb5.ini").exists()) {
List<String> authtypes = new ArrayList<String>();
authtypes.add(AuthPolicy.NTLM);
authtypes.add(AuthPolicy.DIGEST);
authtypes.add(AuthPolicy.BASIC);
httpclient.getParams().setParameter(AuthPNames.PROXY_AUTH_PREF,
authtypes);
httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF,
authtypes);
}
localContext.setAttribute(ClientContext.CREDS_PROVIDER, credsProvider);
HttpResponse response = httpclient.execute(httpget, localContext);
System.out.println("Response code: " + response.getStatusLine());
The one thing I've noticed in Fiddler is that the hashes sent by Firefox versus by HttpClient are different, making me think that maybe IIS 7.5 is expecting stronger hashing than HttpClient provides? Any ideas? It'd be great if I could verify that this would work with NTLM. Digest would be nice too, but I can live without that if necessary.
I am not an expert on the subject but during the NTLM authentication using http components I have seen that the client needs 3 attempts in order to connect to an NTML endpoint in my case. It is kinda described here for Spnego but it is a bit different for the NTLM authentication.
For NTLM in the first attempt client will make a request with Target auth state: UNCHALLENGED and Web server returns HTTP 401 status and a header: WWW-Authenticate: NTLM
Client will check for the configured Authentication schemes, NTLM should be configured in client code.
Second attempt, client will make a request with Target auth state: CHALLENGED, and will send an authorization header with a token encoded in base64 format: Authorization: NTLM TlRMTVNTUAABAAAAAYIIogAAAAAoAAAAAAAAACgAAAAFASgKAAAADw==
Server again returns HTTP 401 status but the header: WWW-Authenticate: NTLM now is populated with encoded information.
3rd Attempt Client will use the information from WWW-Authenticate: NTLM header and will make the final request with Target auth state: HANDSHAKE and an authorisation header Authorization: NTLM which contains more information for the server.
In my case I receive an HTTP/1.1 200 OK after that.
In order to avoid all this in every request documentation at chapter 4.7.1 states that the same execution token must be used for logically related requests. For me it did not worked.
My code:
I initialize the client once in a #PostConstruct method of an EJB
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
cm.setMaxTotal(18);
cm.setDefaultMaxPerRoute(6);
RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(30000)
.setConnectTimeout(30000)
.setTargetPreferredAuthSchemes(Arrays.asList(AuthSchemes.NTLM))
.setProxyPreferredAuthSchemes(Arrays.asList(AuthSchemes.BASIC))
.build();
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
new NTCredentials(userName, password, hostName, domainName));
// Finally we instantiate the client. Client is a thread safe object and can be used by several threads at the same time.
// Client can be used for several request. The life span of the client must be equal to the life span of this EJB.
this.httpclient = HttpClients.custom()
.setConnectionManager(cm)
.setDefaultCredentialsProvider(credentialsProvider)
.setDefaultRequestConfig(requestConfig)
.build();
Use the same client instance in every request:
HttpPost httppost = new HttpPost(endPoint.trim());
// HttpClientContext is not thread safe, one per request must be created.
HttpClientContext context = HttpClientContext.create();
response = this.httpclient.execute(httppost, context);
Deallocate the resources and return the connection back to connection manager, at the #PreDestroy method of my EJB:
this.httpclient.close();
I had the same problem with HttpClient4.1.X After upgrading it to
HttpClient 4.2.6 it woked like charm. Below is my code
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpContext localContext = new BasicHttpContext();
HttpGet httpget = new HttpGet("url");
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(AuthScope.ANY,
new NTCredentials("username", "pwd", "", "domain"));
List<String> authtypes = new ArrayList<String>();
authtypes.add(AuthPolicy.NTLM);
httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF,authtypes);
localContext.setAttribute(ClientContext.CREDS_PROVIDER, credsProvider);
HttpResponse response = httpclient.execute(httpget, localContext);
HttpEntity entity=response.getEntity();
The easiest way troubleshoot such situations I found is Wireshark. It is a very big hammer, but it really will show you everything. Install it, make sure your server is on another machine (does not work with Localhost) and start logging.
Run your request that fails, run one that works. Then, filter by http (just put http in the filter field), find the first GET request, find the other GET request and compare. Identify meaningful difference, you now have specific keywords or issues to search code/net for. If not enough, narrow down to first TCP conversation and look at full request/response. Same with the other one.
I solved an unbelievable number of problems with that approach. And Wireshark is very useful tool to know. Lots of super-advanced functions to make your network debugging easier.
You can also run it on either client or server end. Whatever will show you both requests to allow you to compare.
I had a similar problem with HttpClient 4.1.2. For me, it was resolved by reverting to HttpClient 4.0.3. I could never get NTLM working with 4.1.2 using either the built-in implementation or using JCIFS.
Updating our application to use the jars in the httpcomponents-client-4.5.1 resolved this issue for me.
I finally figured it out. Digest authentication requires that if you use a full URL in the request, the proxy also needs to use the full URL. I did not leave the proxy code in the sample, but it was directed to "localhost", which caused it to fail. Changing this to 127.0.0.1 made it work.