We Keep Coding

How to override DNS in HTTP connections in Java

Curl has a feature for manually specifying which IP to resolve a host to. For example:
curl https://google.com --resolve "google.com:443:173.194.72.113"
This is particularly useful when using HTTPS. If it was just a HTTP request, I could have achieved the same by specifying the IP address directly, and adding a host header. But in HTTPS that would break the connection since the SSL certificate host would be compared to the IP address and not the host header.
My question is, how can I achieve the same thing in Java?

If using Apache's HttpClient, you can create a custom DNS resolver to detect the host you'd like to redirect, and then provide a substitute IP address.
Note: Just changing the Host header for HTTPS requests doesn't work. It will
throw "javax.net.ssl.SSLPeerUnverifiedException", forcing you to trust bad
certificates, stop SNI from working, etc., so really not an option. A
custom DnsResolver is the only clean way I've found to get these requests to work
with HTTPS in Java.
Example:
/* Custom DNS resolver */
DnsResolver dnsResolver = new SystemDefaultDnsResolver() {
#Override
public InetAddress[] resolve(final String host) throws UnknownHostException {
if (host.equalsIgnoreCase("my.host.com")) {
/* If we match the host we're trying to talk to,
return the IP address we want, not what is in DNS */
return new InetAddress[] { InetAddress.getByName("127.0.0.1") };
} else {
/* Else, resolve it as we would normally */
return super.resolve(host);
}
}
};
/* HttpClientConnectionManager allows us to use custom DnsResolver */
BasicHttpClientConnectionManager connManager = new BasicHttpClientConnectionManager(
/* We're forced to create a SocketFactory Registry. Passing null
doesn't force a default Registry, so we re-invent the wheel. */
RegistryBuilder.<ConnectionSocketFactory>create()
.register("http", PlainConnectionSocketFactory.getSocketFactory())
.register("https", SSLConnectionSocketFactory.getSocketFactory())
.build(),
null, /* Default ConnectionFactory */
null, /* Default SchemePortResolver */
dnsResolver /* Our DnsResolver */
);
/* build HttpClient that will use our DnsResolver */
HttpClient httpClient = HttpClientBuilder.create()
.setConnectionManager(connManager)
.build();
/* build our request */
HttpGet httpRequest = new HttpGet("https://my.host.com/page?and=stuff");
/* Executing our request should now hit 127.0.0.1, regardless of DNS */
HttpResponse httpResponse = httpClient.execute(httpRequest);

I don't have the code close at hand, but you can also write your own SSL handler/checker that could adapt or flat-out just ignore all the security. Using the JDK base networking, we had to totally ignore SSL certificates internally for testing. Should be easy to find examples.

HttpsUrlConnection and keep-alive

I am using com.sun.net.httpserver.HttpsServer in my current project which deals with client-authentification etc.. Currently it only prints out the clients address/port, so that I can check if one TCP-connection is used for multiple requests (keep-alive) or if a new connection is established for every request (and thus a new SSL-handshake is made every time). When I use FireFox to make multiple request against the server I can see that keep-alive is working. So the server part works fine with GET and POST-requests.
If I use HttpURLConnection to make a request against the Server (in this case using no SSL) keep-alive works, too: Only one connection is established for multiple sequentially started requests.
But if I use HttpsURLConnection (using exactly the same code, but using SSL) then keep-alive is not working anymore. So for each request a new connection is established, although I am using the same SSLContext (and SSLSocketFactory):
// URL myUrl = ...
// SSLContext mySsl = ...
HttpsURLConnection conn = (HttpsURLConnection) myUrl.openConnection();
conn.setUseCaches(false);
conn.setSSLSocketFactory(mySsl.getSocketFactory());
conn.setRequestMethod("POST");
// send Data
// receive Data
How do I force HttpsURLConnection to use keep-alive because many requests will lead to many SSL-handshakes which is a real performance issue?
Update (2012-04-02):
Instead of calling mySsl.getSocketFactory() each time, I tried to cache the SSLSocketFactory. But nothing changed. The problem still exists.

I ran into this exact same problem and finally have a solution after some in-depth debugging.
Http(s)UrlConnection does handle Keep-Alive by default but sockets must be in a very specific condition in order to be reused.
These are:
Input streams must be fully consumed. You must call read on the input stream until it returns -1 and also close it.
Settings on the underlying socket must use the exact same objects.
You should call disconnect (yes this is counter-intuitive) on the Http(s)URLConnection when done with it.
In the above code, the problem is:
conn.setSSLSocketFactory(mySsl.getSocketFactory());
Saving the result of getSocketFactory() to a static variable during initialization and then passing that in to conn.setSSLSocketFactory should allow the socket to be reused.

I couldn't get it working with HttpsUrlConnection. But Apache's HTTP client handles keep-alive with SSL connections very well.

SSL connection establishment is really expensive either for service calls or when getting many resources from a browser.
Java Http(s)UrlConnection handles HTTP(S) Keep-Alive by default.
I have not found the source code of the default SSLSocketFactory and probably the keep-alive mechanism is implemented there. As a confirmation, disable your own SSLSocketFactory implementation for a test, with a custom trust store in javax.net.ssl.trustStore so that your self-signed certificate is accepted.
According to OpenJDK 7 ServerImpl implementation which uses ServerConfig the HttpsServer you used emits a keep-alive with 5 minutes timeout per default.
I propose you set the property sun.net.httpserver.debug to true server-side to get details.
Take care your code does not add the header Connection: close which disables keep-alive mechanism.

As far as I can understand HTTP/1.1 and HTTPS protocol, also documented here, Keep-Alive is not an end-to-end header but a hop-to-hop header. Since SSL involves multiple steps of handshaking among "different hops" (e.g. CA and the server) for each new connection, I think Keep-Alive may not be applicable in SSL context. So, that can be why Keep-Alive header is ignored using HTTPS connections. Based on this this question, you may need to ensure one instance of HTTP connection is used to guarantee Keep-Alive observation. Also, in the question, it seems that Apache HTTPClient has been a better solution.

We may setup an Apache Webserver, add following directives to see whether the Apache's access.log has a keep-alive connection for the http client.
LogFormat "%k %v %h %l %u %t \"%r\" %>s %b" common
CustomLog "logs/access.log" common
http://httpd.apache.org/docs/current/mod/mod_log_config.html
"%k" Number of keepalive requests handled on this connection. Interesting if KeepAlive is being used, so that, for example, a '1' means the first keepalive request after the initial one, '2' the second, etc...; otherwise this is always 0 (indicating the initial request).

I faced the same problem, and Bill Healey is right.
I tested my example code below with few https libraries.
HttpsURLConnection and OKHTTP are exact same behavior.
Volley is a bit different when session resumption, but almost same behavior.
I hope this will be some help.
public class SampleActivity extends Activity implements OnClickListener {
// Keep default context and factory
private SSLContext mDefaultSslContext;
private SSLSocketFactory mDefaultSslFactory;
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
findViewById(R.id.button_id).setOnClickListener(this);
try {
// Initialize context and factory
mDefaultSslContext = SSLContext.getInstance("TLS");
mDefaultSslContext.init(null, null, null);
mDefaultSslFactory = mDefaultSslContext.getSocketFactory();
} catch (NoSuchAlgorithmException | KeyManagementException e) {
Log.e(TAG, e.getMessage(), e);
}
}
#Override
public void onClick(View v){
SSLContext sslcontext;
SSLSocketFactory sslfactory;
try {
// If using this factory, enable Keep-Alive
sslfactory = mDefaultSslFactory;
// If using this factory, enable session resumption (abbreviated handshake)
sslfactory = mDefaultSslContext.getSocketFactory();
// If using this factory, enable full handshake each time
sslcontext = SSLContext.getInstance("TLS");
sslcontext.init(null, null, null);
sslfactory = sslcontext.getSocketFactory();
} catch (NoSuchAlgorithmException | KeyManagementException e) {
Log.e(TAG, e.getMessage(), e);
}
URL url = new URL("https://example.com");
HttpsURLConnection = conn = (HttpsURLConnection) url.openConnection();
conn.setSSLSocketFactory(sslfactory);
conn.connect();
}
}
Update:
Sharing SSLSocketFactory enables keep-alive. Sharing SSLContext and getting facotry each request enable session resumption. I don't know how TLS stack works, but just confirmed these connection behaviors with some mobile devices.
If you want to enable keep-alive among multiple classes, you should share the instance of SSLSocketFactory using singleton pattern.
If you want to enable session resumption, make sure the session timeout settings is long enough on server side, such as SSLSessionCacheTimeout(apache), ssl_session_timeout(nginx).

In addition to #Bill Healey answer the HostnameVerifier also must be declared static.
I've tried several patterns with and without closing input stream and connection they make no change for me. The only thing that matters is the static declarations of mentioned properties.
/**
SSLSocketFactory and HostnameVerifier must be declared static in order to be able to use keep-alive option
*/
private static SSLSocketFactory factory = null;
private static HostnameVerifier hostnameVerifier = new HostnameVerifier() {
#Override
public boolean verify(String s, SSLSession sslSession) {
return true;
}
};
public static void prepareForCustomTrustIfNeeded(HttpsURLConnection connection) {
try {
if(factory == null) {
SSLContext sslc = SSLContext.getInstance("TLS");
sslc.init(null, customTrustedCerts, new SecureRandom());
factory = sslc.getSocketFactory();
}
connection.setSSLSocketFactory(factory);
connection.setHostnameVerifier(hostnameVerifier);
} catch (Exception e) {
e.printStackTrace();
}
}

try to add the following code:
con.setRequestProperty("Connection", "Keep-Alive");
con.setRequestProperty("Keep-Alive", "header");

Can I override the Host header where using java's HttpUrlConnection class?

I'm using the following code to open a http connection in java:
URL url = new URL("http://stackoverflow.com");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("GET");
conn.setRequestProperty("Host", "Test:8080");
conn.getOutputStream();
However calling conn.setRequestProperty("Host", "Test:8080") appears to have no effect regardless of what order I call the methods and the Host is reset to the destination server. Is there any way to override the Host header without using a different library?
TIA Matt

This used to work in the past, but it has been disabled as part of a security-fix. Apparently without a note in the changelog. There are even bugs like #7022056 for this at bugs.sun.com.
There is a similar question for another header, where the answer goes more into the details, so I just link it instead of writing it myself. :-)
The only workarounds seem to be setting sun.net.http.allowRestrictedHeaders to true or use another http-library like the already mentioned http components.

The Host header is filled by the HttpURLConnection based on the URL. You can't open foo.com with Host=bar.com. From the RFC
The Host request-header field specifies the Internet host and port number of the resource being requested, as obtained from the original URI given by the user or referring resource (generally an HTTP URL)
Btw, you can also try apache http components.

This is an issue with how volley handles HTTPUrlConnection and retry policy.
A Quick fix for it is to extend "HurlStack" class and override the "createConnection" function to return a HTTPUrlConnection with ChunkStreamMode of 0
public class CustomHurlStack extends HurlStack {
public CustomHurlStack(){
super();
}
#Override
protected HttpURLConnection createConnection(URL url) throws IOException {
HttpURLConnection connection = super.createConnection(url);
connection.setChunkedStreamingMode(0);
return connection;
}
}

ProxySelector changes URL's scheme from https:// to socket://

I need to access Facebook but all outgoing communication is blocked on our server so I have to use proxy.
I initialize proxies with:
ProxySelector.setDefault(new ConfigurableProxySelector(mapping));
Proxy type is HTTP, proxy host and port are working (confirmed by simple wget test).
I'm trying to do this:
HttpClient httpClient = new HttpClient();
HttpMethod method = new GetMethod("https://graph.facebook.com:443");
int status = httpClient.executeMethod(method);
Now, in my class ConfigurableProxySelector I have select method on which I have breakpoint:
public List<Proxy> select(URI uri) {
...
}
So, using HttpClient I make an request, which should be proxied and code stops at breakpoint in select() method in ConfigurableProxySelector.
But what is strange is that uri.scheme = "socket" and .toString() gives "socket://graph.facebook.com:443" instead of "https://graph.facebook.com:443".
Because ProxySelector have mapping for "https://" and not for "socket://", it does not find it and it ends with "Connection refused". What is strange is that select() method is called 4 times before execution ends with "Connection refused".
Any help would be appreciated.

Apache HTTP Client 3.1 will not natively honor HTTP Proxies returned from the default ProxySelector or user implementations.
Quick Summary of ProxySelector
ProxySelector is a service class which selects and returns a suitable Proxy for a given URL based on its scheme. For example, a request for http://somehost will try to provide an HTTP proxy if one is defined. The default ProxySelector can be configured at runtime using System Properties, such as http.proxyHost and http.proxyPort.
HTTPUrlConnection
An instance of HTTPUrlConnection will check against the default ProxySelector multiple times: 1st to select for http or https, then later when it builds the raw tcp socket, using the socket scheme. A SOCKS proxy could be used to proxy a raw tcp socket but are not often found in corporate environments, so a raw tcp socket will usually receive no proxy.
HTTP Client 3.1
HC 3.1, on the other hand, will never check the default ProxySelector for the http/https schemes. It will check, however, at a later points for the socket scheme when it eventually builds the raw socket - This is the request you are seeing. This means the System Properties http.proxyHost and http.proxyPort are ineffective. This is obviously not ideal for most people who only have an HTTP/HTTPS proxy.
To work around this, you have two options: define a proxy on each HC 3.1 connection or implement your own HC 3.1 HTTPConnectionManager.
HTTPConnectionManager
The HTTPConnectionManager is responsible for building connections for the HC 3.1 client.
The default HC 3.1 HTTPConnectionManager can be extended so that it looks for a suitable proxy from a ProxySelector (default or custom) when building the request in the same way HTTPUrlConnection does:
public class MyHTTPConnectionManager extends SimpleHttpConnectionManager {
#Override
public HttpConnection getConnectionWithTimeout(
HostConfiguration hostConfiguration, long timeout) {
HttpConnection hc = super.getConnectionWithTimeout(hostConfiguration, timeout);
try {
URI uri = new URI( hostConfiguration.getHostURL());
List<Proxy> hostProxies = ProxySelector.getDefault().select(uri);
Proxy Proxy = hostProxies.get(0);
InetSocketAddress sa = (InetSocketAddress) Proxy.address();
hc.setProxyHost(sa.getHostName());
hc.setProxyPort(sa.getPort());
} catch (URISyntaxException e) {
return hc;
}
return hc;
}
}
Then, when you create an HC 3.1 client, use your new connection manager:
HttpClient client = new HttpClient(new MyHTTPConnectionManager() );

It's not the ProxySelector that changes the scheme, but the SocketFactory opening a Socket.
If the SocketFactory is null a SOCKS socket will be created by default which only allows SOCKS proxies. I don't know anything about Sockets and cannot tell you if there's a way to make it work with HTTP proxies.
But using another approach may help, since Apache HttpClient seems to have its own way to configure proxies.
client.getHostConfiguration().setProxy(proxyHost, proxyPort);
if (proxyUser != null) {
client.getState().setProxyCredentials(new AuthScope(proxyHost, proxyPort),
new UsernamePasswordCredentials(proxyUser, proxyPassword));
}

How can I override the "Host" header in the request when using Apache commons HttpClient

I am using Jakarta Commons HttpClient 3.1 writing a load test tool that needs to target different servers and pretend like it targeted the correct virtual host in the HTTP server. For that I need to be able to set the "Host" HTTP header in the request to a different host name then the actual host name that I'm connecting to.
It seemed pretty obvious that I should use Method.setRequestHeader("Host","fakehostname"), but HttpClient just ignores this and always sends the real host name I'm connecting to in the "Host" header (I've enabled debug logging for "httpclient.wire" and I can it does this specifically).
How can I override the header so that HttpClient takes heed?

After searching some more, and taking a hint from Oleg's answer, I've found the method HttpMethodParams::setVirtualHost().
when HttpClient formats a request, it always creates the "Host" header itself just before sending the request - so it cannot be overridden as a standard header. But before the host name for the "Host" header is generated from the URL, HttpClient checks the HttpMethodParams object to see if the user wants to override the host name. This only overrides the host name and not the port so it would be easier to use, though not as intuitive as I'd like.
The code to use this can look like this:
Method m = new GetMethod("http://some-site/some/path");
m.getParams().setVirtualHost("some-other-site");
client.executeMethod(m);
Because I like one liners, this can also be written as:
client.executeMethod(new GetMethod("http://some-site/some/path") {{
getParams().setVirtualHost("some-other-site"); }});

I believe you want http://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/HttpHost.html: this lets you configure the host for a specific connection. If I understand it correctly, you can either use the execute method (see http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/AbstractHttpClient.html#execute(org.apache.http.HttpHost,%20org.apache.http.HttpRequest)) and pass it a custom HttpHost object, or do this:
Construct an HttpHost instance, passing it your Host header.
Use that to create an HttpRoute instance (see http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/routing/HttpRoute.html)
Pass that to the connection manager when you request a connection (see http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/ClientConnectionManager.html#requestConnection(org.apache.http.conn.routing.HttpRoute,%20java.lang.Object)).
Use the connection with your method: see http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html for more details.
Let me know how that works.
EDIT: principle remains the same.
1. Construct an HttpHost instance, passing it your Host header (see http://hc.apache.org/httpclient-legacy/apidocs/index.html?org/apache/commons/httpclient/HttpHost.html).
2. Create an HttpConfiguration instance and then pass it the HttpHost you created (see http://hc.apache.org/httpclient-legacy/apidocs/index.html?org/apache/commons/httpclient/HostConfiguration.html).
3. Use the execute method on HttpClient with that configuration (see http://hc.apache.org/httpclient-legacy/apidocs/org/apache/commons/httpclient/HttpClient.html#executeMethod(org.apache.commons.httpclient.HostConfiguration,%20org.apache.commons.httpclient.HttpMethod))

Following works on android:
System.setProperty("sun.net.http.allowRestrictedHeaders", "true");
InputStream stream_content=null;
try
{URL url=new URL("http://74.125.28.103/");
HttpURLConnection conn=(HttpURLConnection)url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("GET");
conn.setRequestProperty("Host", "www.google.com");
stream_content=conn.getInputStream();
}
catch (Exception e) {}
for https url:
System.setProperty("sun.net.http.allowRestrictedHeaders", "true");
InputStream stream_content=null;
try
{URL url=new URL("https://74.125.28.103/");
HttpsURLConnection conn=(HttpsURLConnection)url.openConnection();
conn.setHostnameVerifier(SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER );
conn.setDoOutput(true);
conn.setRequestMethod("GET");
conn.setRequestProperty("Host", "www.google.com");
stream_content=conn.getInputStream();
}
catch (Exception e) {}

One can use the 'http.virtual-host' parameter in order to force an arbitrary (virtual) hostname and port as a value of the Host request header instead of those derived from the actual request URI. This works with the 4.x API only, though.