I'm using Java's HttpUrlConnection to hit foo.com
foo.com has multiple A-Records that point to different IP addresses (1.1.1.1 and 1.1.1.2)
If my first connect call resolves to 1.1.1.1, but then that machine goes down, will a subsequent connect call recognize this and try to connect on 1.1.1.2 instead?
Or do I need to implement this sort of logic myself using the INetAddress api?
I was able to resolve this by using Apache Commons HttpClient, see the code snippet below.
Like I feared, the URLConnection provided by java.net is a very simplistic implementation and will only try the first IP address from the resolved list. If you really are not allowed to use another library, you will have to write your own error handling. It's kinda messy, since you will need to resolve all IPs before hand using InetAddress, and connect to each IP passing the "Host: domain.name" header to the HTTP stack yourself until one of the IPs responds.
The Apache library is greatly more robust and allows for a great deal of customization. You can control how many times it will retry and, most importantly, it will automatically try all IP addresses resolved to the same name until one of them responds successfully.
HttpRequestRetryHandler myRetryHandler = new HttpRequestRetryHandler() {
#Override
public boolean retryRequest(IOException exception, int count, HttpContext context) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
}
return count < 30;
}
};
ConnectionKeepAliveStrategy keepAlive = new ConnectionKeepAliveStrategy() {
#Override
public long getKeepAliveDuration(HttpResponse response, HttpContext context) {
return 500;
}
};
DefaultHttpClient httpclient = new DefaultHttpClient();
httpclient.getParams().setParameter("http.socket.timeout", new Integer(2000));
httpclient.getParams().setParameter("http.connection.timeout", new Integer(2000));
httpclient.setHttpRequestRetryHandler(myRetryHandler);
httpclient.setKeepAliveStrategy(keepAlive);
HttpGet httpget = new HttpGet("http://remotehost.com");
HttpResponse httpres = httpclient.execute(httpget);
InputStream is = httpres.getEntity().getContent();
I hope this helps!
Related
This question is worded the same way as another question on SO (HttpClient: how do I obtain the underlying socket from an existing connection?), but that question is actually about tunneling other protocols through HTTP(S) and thus the answer is quite a bit different.
What I'm trying to do is make an HTTPS connection, then find out the details of that connection. Java's SSLSocket class will give me what I need, but I need to be able to get a hold of the Socket itself in order to interrogate it.
Is there a way to get to the underlying Socket? httpclient/httpcore has become a maze of factories and private/protected implementations of things, so it's really difficult to poke-around the API to figure out how to actually get things once they have been configured.
HttpClient intentionally makes it difficult to get hold of the underlying connection object and the socket it is bound to, primarily to ensure the connection state is consistent and persistent connections in the connection pool are safe to be re-used by another transaction.
However, one can get hold of the underlying connection from a response interceptor.
CloseableHttpClient httpclient = HttpClients.custom()
.addInterceptorLast(new HttpResponseInterceptor() {
#Override
public void process(
final HttpResponse response,
final HttpContext context) throws HttpException, IOException {
HttpClientContext clientContext = HttpClientContext.adapt(context);
ManagedHttpClientConnection connection = clientContext.getConnection(ManagedHttpClientConnection.class);
// Can be null if the response encloses no content
if (connection != null) {
Socket socket = connection.getSocket();
System.out.println(socket);
}
}
})
.build();
try (CloseableHttpResponse response = httpclient.execute(new HttpGet("http://www.google.com/"))) {
System.out.println(response.getStatusLine());
EntityUtils.consume(response.getEntity());
}
I ended up using a somewhat different technique, but #oleg got me on the right track. Here's my one-time code:
HttpClientContext ctx = HttpClientContext.create();
HttpResponse response = getHttpClient().execute(method, ctx);
if(log.isDebugEnabled())
{
ManagedHttpClientConnection connection = ctx.getConnection(ManagedHttpClientConnection.class);
// Can be null if the response encloses no content
if(null != connection)
{
Socket socket = connection.getSocket();
if(socket instanceof SSLSocket)
{
SSLSocket sslSock = (SSLSocket)socket;
log.debug("Connected to " + getEndpointURL()
+ " using " + sslSock.getSession().getProtocol()
+ " and suite " + sslSock.getSession().getCipherSuite());
}
}
}
I am working on a Http Proxy in java. Basically I have 3 applications:
a client application, where I just submit a request to a server VIA a proxy
a proxy that captures the request, modifies it and then forwards it to the web server
the web server
Here is my code for the Client (taken from the apache httpcore examples, but works well):
public class ClientExecuteProxy () {
public static void main(String[] args)throws Exception {
HttpHost proxy = new HttpHost("127.0.0.1", 8080, "http");
DefaultHttpClient httpclient = new DefaultHttpClient();
try {
httpclient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
HttpHost target = new HttpHost("issues.apache.org", 443, "https");
HttpGet req = new HttpGet("/");
System.out.println("executing request to " + target + " via " + proxy);
HttpResponse rsp = httpclient.execute(target, req);
HttpEntity entity = rsp.getEntity();
System.out.println("----------------------------------------");
System.out.println(rsp.getStatusLine());
Header[] headers = rsp.getAllHeaders();
for (int i = 0; i<headers.length; i++) {
System.out.println(headers[i]);
}
System.out.println("----------------------------------------");
if (entity != null) {
System.out.println(EntityUtils.toString(entity));
}
} finally {
// When HttpClient instance is no longer needed,
// shut down the connection manager to ensure
// immediate deallocation of all system resources
httpclient.getConnectionManager().shutdown();
}
}
}
If I do a direct execution of the request to the server (if I comment the line "httpclient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);"), it works without any problem. But if I leave it like that, it will pass by the proxy. Here is the part that I do not know how to handle for the proxy:
The proxy listens for the requests, reads its content and verifies if it respects certain policies. If OK it will forward it to the server, else it will drop the request and it will send a HttpResponse with an error. The problem is when the request is OK and it needs to be forwarded. How does the proxy know to what address to forward it? My question is: How do I get the information from the request entered at the line "HttpHost target = new HttpHost("issues.apache.org", 443, "https");"?
I've googled for a couple of hours but found nothing. Can anybody help me please?
When you define an HTTP proxy to an application or browser, either:
There will be a preceding CONNECT request to form a tunnel, that tells you the target host:port, or
The entire target URL is placed in the middle of the GET/POST/... request line. Normally, without a proxy, this is just a relative URL, relative to the host:port of the TCP connection.
I need to set time out for the Http Request we make to a service (not a web service). We are using Apache HTTP Client. I have added these 2 lines of code to set the time out on request and response to the service.
HttpConnectionParams.setConnectionTimeout(params, 10000);
HttpConnectionParams.setSoTimeout(params, 10000);
1) Currently I have set 10 seconds as the timeout since I see the response coming from the service almost instantaneously. Should I increase or decrease the timing?
2) What will happen when response is takes more than 10 seconds? Will it throw exception and what exception will it be? Is there any thing else I need to add to set the time out in the below code.
public HashMap<String, Object> getJSONData(String url) throw Exception{
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpParams params = httpClient.getParams();
HttpConnectionParams.setConnectionTimeout(params, 10000);
HttpConnectionParams.setSoTimeout(params, 10000);
HttpHost proxy = new HttpHost(getProxy(), getProxyPort());
ConnRouteParams.setDefaultProxy(params, proxy);
URI uri;
InputStream data = null;
try {
uri = new URI(url);
HttpGet method = new HttpGet(uri);
HttpResponse response = httpClient.execute(method);
data = response.getEntity().getContent();
}
catch (Exception e) {
e.printStackTrace();
}
Reader r = new InputStreamReader(data);
HashMap<String, Object> jsonObj = (HashMap<String, Object>) GenericJSONUtil.fromJson(r);
return jsonObj;
}
I am guessing many people come here because of the title and because the HttpConnectionParams API is deprecated.
Using a recent version of Apache HTTP Client, you can set these timeouts using the request params:
HttpPost request = new HttpPost(url);
RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(TIMEOUT_MILLIS)
.setConnectTimeout(TIMEOUT_MILLIS)
.setConnectionRequestTimeout(TIMEOUT_MILLIS)
.build();
request.setConfig(requestConfig);
Alternatively, you can also set this when you create your HTTP Client, using the builder API for the HTTP client, but you'll also need to build a custom connection manager with a custom socket config.
The configuration example file is an excellent resource to find out about how to configure Apache HTTP Client.
The exceptions you'll see will be ConnectTimeoutException and SocketTimeoutException. The actual timeout values you use should be the maximum time your application is willing to wait. One important note about the read timeout is that it corresponds to the timeout on a socket read. So it's not the time allowed for the full response to arrive, but rather the time given to a single socket read. So if there are 4 socket reads, each taking 9 seconds, your total read time is 9 * 4 = 36 seconds.
If you want to specify a total time for the response to arrive (including connect and total read time), you can wrap the call in a thread and use a thread timeout for that. For example, I usually do something like this:
Future<T> future = null;
future = pool.submit(new Callable<T>() {
public T call() {
return executeImpl(url);
}
});
try {
return future.get(10, TimeUnit.SECONDS);
}
catch (InterruptedException e) {
log.warn("task interrupted", name);
}
catch (ExecutionException e) {
log.error(name + " execution exception", e);
}
catch (TimeoutException e) {
log.debug("future timed out", name);
}
Some assumptions made in the code above are: 1) this is in a function with a url parameter, 2) it's in a class with a name variable, 3) log is a log4j instance, and 4) pool is a some thread pool executor. Note that even if you use a thread timeout, you should also specify a connect and socket timeout on the HttpClient, so that slow requests don't eat up the resources in the thread pool. Also note that I use a thread pool because typically I use this in a web service so the thread pool is shared across a bunch of tomcat threads. You're environment may be different, and you may prefer to simply spawn a new thread for each call.
Also, I've usually see the timeouts set via member functions of the params, like this:
params.setConnectionTimeout(10000);
params.setSoTimeout(10000);
But perhaps your syntax works as well (not sure).
I have been trying to use a custom SocketFactory in the httpclient library from the Apache HTTPComponents project. So far without luck. I was expecting that I could just set a socket factory for a HttpClient instance, but it is obviously not so easy.
The documentation for HttpComponents at http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html does mention socket factories, but does not say how to use them.
Does anybody know how this is done?
oleg's answer is of course correct, I just wanted to put the information directly here, in case the link goes bad. In the code that creates a HttpClient, I use this code to let it use my socket factory:
CustomSocketFactory socketFactory = new CustomSocketFactory();
Scheme scheme = new Scheme("http", 80, socketFactory);
httpclient.getConnectionManager().getSchemeRegistry().register(scheme);
CustomSocketFactory is my own socket factory, and I want to use it for normal HTTP traffic, that's why I use "http" and 80 as parameters.
My CustomSchemeSocketFactory looks similar to this:
public class CustomSchemeSocketFactory implements SchemeSocketFactory {
#Override
public Socket connectSocket( Socket socket, InetSocketAddress remoteAddress, InetSocketAddress localAddress, HttpParams params ) throws IOException, UnknownHostException, ConnectTimeoutException {
if (localAddress != null) {
socket.setReuseAddress(HttpConnectionParams.getSoReuseaddr(params));
socket.bind(localAddress);
}
int connTimeout = HttpConnectionParams.getConnectionTimeout(params);
int soTimeout = HttpConnectionParams.getSoTimeout(params);
try {
socket.setSoTimeout(soTimeout);
socket.connect(remoteAddress, connTimeout );
} catch (SocketTimeoutException ex) {
throw new ConnectTimeoutException("Connect to " + remoteAddress + " timed out");
}
return socket;
}
#Override
public Socket createSocket( HttpParams params ) throws IOException {
// create my own socket and return it
}
#Override
public boolean isSecure( Socket socket ) throws IllegalArgumentException {
return false;
}
}
We use a custom socket factory to allow HttpClient connections to connect to HTTPS URLs with untrusted certificates.
Here is how we did it:
We adapted implementations of both the 'EasySSLProtocolSocketFactory' and 'EasyX509TrustManager' classes from the examples source directory referenced by Oleg.
In our HttpClient startup code, we do the following to enable the new socket factory:
HttpClient httpClient = new HttpClient();
Protocol easyhttps = new Protocol("https", new EasySSLProtocolSocketFactory(), 443);
Protocol.registerProtocol("https", easyhttps);
So that any time we request an https:// URL, this socket factory is used.
I'm working with a slow webservice (about 4 minutes each request) and I need to do about 100 requests in two hours, so I've decided to use multiple threads. The problem is that I can only have 2 threads, as the stub rejects all the other ones. Here I've found an explanation and possible solution:
I had the same problem. It seems that
the source of it is
defaultMaxConnectionsPerHost value in
MultiThreadedHttpConnectionManager
equals 2. Workaround for me was to
create own instance of
MultiThreadedHttpConnectionManager and
use it in service stub, something like
in example below
I've done as the author said, and passed a HttpClient to the stub with higher setMaxTotalConnections and setDefaultMaxConnectionsPerHost values, but the problem is that now the application freezes (well, it does not really freezes, but It does nothing).
Thats my code:
public ReportsStub createReportsStub(String url, HttpTransportProperties.Authenticator auth){
ReportsStub stub = null;
HttpClient httpClient = null;
try {
stub = new ReportsStub(url);
httpClient = createHttpClient(10,5);
stub._getServiceClient().getOptions().setTimeOutInMilliSeconds(10000000);
stub._getServiceClient().getOptions().setProperty(org.apache.axis2.transport.http.HTTPConstants.AUTHENTICATE, auth);
stub._getServiceClient().getOptions().setProperty(org.apache.axis2.transport.http.HTTPConstants.CHUNKED, false);
stub._getServiceClient().getServiceContext().getConfigurationContext().setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
return stub;
} catch (AxisFault e) {
e.printStackTrace();
}
return stub;
}
protected HttpClient createHttpClient(int maxTotal, int maxPerHost) {
MultiThreadedHttpConnectionManager httpConnectionManager = new MultiThreadedHttpConnectionManager();
HttpConnectionManagerParams params = httpConnectionManager.getParams();
if (params == null) {
params = new HttpConnectionManagerParams();
httpConnectionManager.setParams(params);
}
params.setMaxTotalConnections(maxTotal);
params.setDefaultMaxConnectionsPerHost(maxPerHost);
HttpClient httpClient = new HttpClient(httpConnectionManager);
return httpClient;
}
Then I pass that stub and the request to each one of threads and run them. If I don't set the HttpClient and use the default, only two threads execute, and if I set it, the application does not work. Any idea?
If anyone wants to create a dynamic REST client in WSO2 Axis2, the following code worked for me...
// Set the max connections to 20 and the timeout to 20 seconds
MultiThreadedHttpConnectionManager multiThreadedHttpConnectionManager = new MultiThreadedHttpConnectionManager();
HttpConnectionManagerParams params = new HttpConnectionManagerParams();
params.setDefaultMaxConnectionsPerHost(20);
params.setMaxTotalConnections(20);
params.setSoTimeout(20000);
params.setConnectionTimeout(20000);
multiThreadedHttpConnectionManager.setParams(params);
HttpClient httpClient = new HttpClient(multiThreadedHttpConnectionManager);
// Create the service client
ServiceClient serviceClient = new ServiceClient();
Options options = new Options();
options.setTo(new EndpointReference(endpoint));
options.setProperty(Constants.Configuration.ENABLE_REST, Constants.VALUE_TRUE);
options.setProperty(Constants.Configuration.HTTP_METHOD, Constants.Configuration.HTTP_METHOD_POST);
serviceClient.getServiceContext().getConfigurationContext().setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
serviceClient.setOptions(options);
// Blocking call
OMElement result = serviceClient.sendReceive(ClientUtils.getRestPayload()); // just a dummy payload <root></root>
// Cleanup Transport after each call, this is needed to otherwise the HTTP gets blocked
serviceClient.cleanupTransport();
I put the Max Connections to 20 and the Timeout to 20 seconds.
Also my 'endpoint' contains all the REST arguments, I'm just using a dummy payload "<root></root>" in the serviceClient.sendReceive() method.
I noticed this in a corporate web application that called a back-end service that could take a long period to respond. The web application would lock up because a limit of 2 connections to a single host would take hold.
You call httpConnectionManager.setParams( params ) before you call params.setDefaultMaxConnectionsPerHost(). Have you tried calling these functions in the opposite order to confirm that application of params doesn't take place within the httpConnectionManager.setParams function itself?