We Keep Coding

HttpsURLConnections Default Hostname Verifier

I'm using a HttpURLConnection in order create a POST request (for fetching a token at some OAuth2 token endpoint). The token endpoint uses HTTPS. I wonder how the hostname verification with regards to HTTPS works. The default hostname verifier of HttpsURLConnection seems to be the following [1]:
/**
* HostnameVerifier provides a callback mechanism so that
* implementers of this interface can supply a policy for
* handling the case where the host to connect to and
* the server name from the certificate mismatch.
*
* The default implementation will deny such connections.
*/
private static HostnameVerifier defaultHostnameVerifier =
new HostnameVerifier() {
public boolean verify(String urlHostname, String certHostname) {
return false;
}
};
I expected my POST request to fail as this verifier always returns false. This is not the case. The comment already states that there is some kind of callback mechanism. What I do not know is: Does the defaultHostnameVerifier verify the hostname of the connection and the certificate or is it rather a dummy implementation?
My current coding looks like the following piece:
private HttpURLConnection openConnection(String url) throws IOException {
URL urly = new URL(url);
final HttpURLConnection con;
Proxy proxy = getProxy();
if (proxy == null) {
con = (HttpURLConnection) urly.openConnection();
} else {
con = (HttpURLConnection) urly.openConnection(proxy);
}
if (con instanceof HttpsURLConnection) {
HostnameVerifier verifier = ((HttpsURLConnection) con).getHostnameVerifier(); // there is a default set
System.out.println(verifier.getClass().getName());
}
return con;
}
I've found some explanation with regards to the AsyncHttpClient [2]. As I do not use it at this point of time am I safe going with the default implementation?
[1] https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/com/sun/net/ssl/HttpsURLConnection.java#L76
[2] https://kevinlocke.name/bits/2012/10/03/ssl-certificate-verification-in-dispatch-and-asynchttpclient/

As the comments at the top say, that class is no longer used; it was the abstract user-visible class that is now replaced by javax.net.HttpsURLConnection which you will observe has the same code. But the implementation class for https URL is sun.net.www.protocol.https.HttpsURLConnectionImpl which just wraps (delegates to) sun.net.www.protocol.https.DelegateHttpsURLConnection which subclasses sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection which to make the actual connection uses sun.net.www.protocol.HttpsClient and in particular .afterConnect(). As background, in earlier versions of Java SSLSocket did not do hostname verification, so the implementation of HttpsURLConnection had to add it. In Java 7 up, SSLSocket does support 'endpoint identification', so when afterConnect() recognizes that the hostnameVerifier on this URLconnection was the default one, it turns off needToCheckSpoofing and sets the SSLSocket to do endpoint identification.
javax.net.ssl.SSLSocket is similarly an abstract class that is actually implemented by sun.security.ssl.SSLSocketImpl and several dozen related classes including sun.security.ssl.ClientHandshaker.serverCertificate() which calls the configurable trustmanager which by default is sun.security.ssl.X509TrustManagerImpl which in checkTrusted() since endpoint identification was requested calls checkIdentity() which calls sun.security.util.HostnameChecker with TYPE_TLS (actually meaning HTTPS RFC 2818), and that does the actual checking.
Glad you asked?
PS: the analysis on that webpage, that HttpsURLConnection.DefaultHostnameVerifier is called only for mismatch, is quite wrong. As above it is bypassed and never called by the actual implementation.
Also I assume you realize java 7 has not been supported for years unless you pay. Although this area hasn't changed that I know of in more recent versions. Java 11 does add a new java.net.http.HttpClient which functionally supersedes [Http,Https]URLConnection.

Understanding SSL in Apache HttpClient (Java)

Here there is an example for custom SSL:
https://hc.apache.org/httpcomponents-client-ga/httpclient/examples/org/apache/http/examples/client/ClientCustomSSL.java
/**
* This example demonstrates how to create secure connections with a custom SSL
* context.
*/
public class ClientCustomSSL {
public final static void main(String[] args) throws Exception {
// Trust own CA and all self-signed certs
SSLContext sslcontext = SSLContexts.custom()
.loadTrustMaterial(new File("my.keystore"), "nopassword".toCharArray(),
new TrustSelfSignedStrategy())
.build();
// Allow TLSv1 protocol only
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(
sslcontext,
new String[] { "TLSv1" },
null,
SSLConnectionSocketFactory.getDefaultHostnameVerifier());
CloseableHttpClient httpclient = HttpClients.custom()
.setSSLSocketFactory(sslsf)
.build();
try {
HttpGet httpget = new HttpGet("https://httpbin.org/");
System.out.println("Executing request " + httpget.getRequestLine());
CloseableHttpResponse response = httpclient.execute(httpget);
try {
HttpEntity entity = response.getEntity();
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
EntityUtils.consume(entity);
} finally {
response.close();
}
} finally {
httpclient.close();
}
}
}
Why we need that? I've tested an HttpClient request without any SSL thing on it and I'm getting the correct response from HTTPS urls without errors.
What is the problem if I don't add any SSLContext?
And if it's important to make it more secure, what is this line?:
.loadTrustMaterial(new File("my.keystore"), "nopassword".toCharArray(),
it seems we need some file and also some password?

If you don't specify (a factory using) a context, Java (JSSE) uses a default context containing the default truststore, which defaults to the file JRE/lib/security/cacerts (or jssecacerts if present) unless overridden with system properties; see https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#CustomizingStores . Depending on whether you are using an Oracle-was-Sun Java package, an IBM or Apple package or Android system, or OpenJDK, this default truststore usually contains more or less the same set of public CAs as most OSes and browsers, like Verisign Symantec Digicert and GoDaddy and LetsEncrypt/Identrust. Whether you consider the default cacerts 'secure' is a choice for you to make; if not you can either change the contents of the default file, or have your code use a different file and to do the latter yes you must specify the filename of the keystore file and its password.
That example uses a custom store because it is an example of custom SSL. If it used the defaults, it would be an example of default SSL not an example of custom SSL. For many actual applications using the defaults is fine.
Aside: specifying only TLSv1 (meaning 1.0) for protocol is way out of date, and is likely to be considered insecure or at least borderline. It hasn't actually been broken outright like SSLv3 (and long ago SSLv2), because BEAST proved tamer than feared, but TLSv1.1 and 1.2 are now widely implemented and used, and 1.3 hopefully not too far away, so using 1.0 is widely considered substandard and for one example applicable to many people TLSv1.0 for payment-card transactions is prohibited outright as of last weekend.

HttpsUrlConnection and keep-alive

I am using com.sun.net.httpserver.HttpsServer in my current project which deals with client-authentification etc.. Currently it only prints out the clients address/port, so that I can check if one TCP-connection is used for multiple requests (keep-alive) or if a new connection is established for every request (and thus a new SSL-handshake is made every time). When I use FireFox to make multiple request against the server I can see that keep-alive is working. So the server part works fine with GET and POST-requests.
If I use HttpURLConnection to make a request against the Server (in this case using no SSL) keep-alive works, too: Only one connection is established for multiple sequentially started requests.
But if I use HttpsURLConnection (using exactly the same code, but using SSL) then keep-alive is not working anymore. So for each request a new connection is established, although I am using the same SSLContext (and SSLSocketFactory):
// URL myUrl = ...
// SSLContext mySsl = ...
HttpsURLConnection conn = (HttpsURLConnection) myUrl.openConnection();
conn.setUseCaches(false);
conn.setSSLSocketFactory(mySsl.getSocketFactory());
conn.setRequestMethod("POST");
// send Data
// receive Data
How do I force HttpsURLConnection to use keep-alive because many requests will lead to many SSL-handshakes which is a real performance issue?
Update (2012-04-02):
Instead of calling mySsl.getSocketFactory() each time, I tried to cache the SSLSocketFactory. But nothing changed. The problem still exists.

I ran into this exact same problem and finally have a solution after some in-depth debugging.
Http(s)UrlConnection does handle Keep-Alive by default but sockets must be in a very specific condition in order to be reused.
These are:
Input streams must be fully consumed. You must call read on the input stream until it returns -1 and also close it.
Settings on the underlying socket must use the exact same objects.
You should call disconnect (yes this is counter-intuitive) on the Http(s)URLConnection when done with it.
In the above code, the problem is:
conn.setSSLSocketFactory(mySsl.getSocketFactory());
Saving the result of getSocketFactory() to a static variable during initialization and then passing that in to conn.setSSLSocketFactory should allow the socket to be reused.

I couldn't get it working with HttpsUrlConnection. But Apache's HTTP client handles keep-alive with SSL connections very well.

SSL connection establishment is really expensive either for service calls or when getting many resources from a browser.
Java Http(s)UrlConnection handles HTTP(S) Keep-Alive by default.
I have not found the source code of the default SSLSocketFactory and probably the keep-alive mechanism is implemented there. As a confirmation, disable your own SSLSocketFactory implementation for a test, with a custom trust store in javax.net.ssl.trustStore so that your self-signed certificate is accepted.
According to OpenJDK 7 ServerImpl implementation which uses ServerConfig the HttpsServer you used emits a keep-alive with 5 minutes timeout per default.
I propose you set the property sun.net.httpserver.debug to true server-side to get details.
Take care your code does not add the header Connection: close which disables keep-alive mechanism.

As far as I can understand HTTP/1.1 and HTTPS protocol, also documented here, Keep-Alive is not an end-to-end header but a hop-to-hop header. Since SSL involves multiple steps of handshaking among "different hops" (e.g. CA and the server) for each new connection, I think Keep-Alive may not be applicable in SSL context. So, that can be why Keep-Alive header is ignored using HTTPS connections. Based on this this question, you may need to ensure one instance of HTTP connection is used to guarantee Keep-Alive observation. Also, in the question, it seems that Apache HTTPClient has been a better solution.

We may setup an Apache Webserver, add following directives to see whether the Apache's access.log has a keep-alive connection for the http client.
LogFormat "%k %v %h %l %u %t \"%r\" %>s %b" common
CustomLog "logs/access.log" common
http://httpd.apache.org/docs/current/mod/mod_log_config.html
"%k" Number of keepalive requests handled on this connection. Interesting if KeepAlive is being used, so that, for example, a '1' means the first keepalive request after the initial one, '2' the second, etc...; otherwise this is always 0 (indicating the initial request).

I faced the same problem, and Bill Healey is right.
I tested my example code below with few https libraries.
HttpsURLConnection and OKHTTP are exact same behavior.
Volley is a bit different when session resumption, but almost same behavior.
I hope this will be some help.
public class SampleActivity extends Activity implements OnClickListener {
// Keep default context and factory
private SSLContext mDefaultSslContext;
private SSLSocketFactory mDefaultSslFactory;
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
findViewById(R.id.button_id).setOnClickListener(this);
try {
// Initialize context and factory
mDefaultSslContext = SSLContext.getInstance("TLS");
mDefaultSslContext.init(null, null, null);
mDefaultSslFactory = mDefaultSslContext.getSocketFactory();
} catch (NoSuchAlgorithmException | KeyManagementException e) {
Log.e(TAG, e.getMessage(), e);
}
}
#Override
public void onClick(View v){
SSLContext sslcontext;
SSLSocketFactory sslfactory;
try {
// If using this factory, enable Keep-Alive
sslfactory = mDefaultSslFactory;
// If using this factory, enable session resumption (abbreviated handshake)
sslfactory = mDefaultSslContext.getSocketFactory();
// If using this factory, enable full handshake each time
sslcontext = SSLContext.getInstance("TLS");
sslcontext.init(null, null, null);
sslfactory = sslcontext.getSocketFactory();
} catch (NoSuchAlgorithmException | KeyManagementException e) {
Log.e(TAG, e.getMessage(), e);
}
URL url = new URL("https://example.com");
HttpsURLConnection = conn = (HttpsURLConnection) url.openConnection();
conn.setSSLSocketFactory(sslfactory);
conn.connect();
}
}
Update:
Sharing SSLSocketFactory enables keep-alive. Sharing SSLContext and getting facotry each request enable session resumption. I don't know how TLS stack works, but just confirmed these connection behaviors with some mobile devices.
If you want to enable keep-alive among multiple classes, you should share the instance of SSLSocketFactory using singleton pattern.
If you want to enable session resumption, make sure the session timeout settings is long enough on server side, such as SSLSessionCacheTimeout(apache), ssl_session_timeout(nginx).

In addition to #Bill Healey answer the HostnameVerifier also must be declared static.
I've tried several patterns with and without closing input stream and connection they make no change for me. The only thing that matters is the static declarations of mentioned properties.
/**
SSLSocketFactory and HostnameVerifier must be declared static in order to be able to use keep-alive option
*/
private static SSLSocketFactory factory = null;
private static HostnameVerifier hostnameVerifier = new HostnameVerifier() {
#Override
public boolean verify(String s, SSLSession sslSession) {
return true;
}
};
public static void prepareForCustomTrustIfNeeded(HttpsURLConnection connection) {
try {
if(factory == null) {
SSLContext sslc = SSLContext.getInstance("TLS");
sslc.init(null, customTrustedCerts, new SecureRandom());
factory = sslc.getSocketFactory();
}
connection.setSSLSocketFactory(factory);
connection.setHostnameVerifier(hostnameVerifier);
} catch (Exception e) {
e.printStackTrace();
}
}

try to add the following code:
con.setRequestProperty("Connection", "Keep-Alive");
con.setRequestProperty("Keep-Alive", "header");

Configuring Apache HttpClient to access service through proxy/load-balancer (overriding Host header)

I am having a problem getting the Apache HttpClient to connect to a service external to my virtualised development environment.
To access the internet (e.g. api.twitter.com) I need to call a local URL (e.g. api.twitter.com.dev.mycompany.net), which then forwards the request to real host.
The problem is, that to whatever request I send, I get a 404 Not Found response.
I have tried debugging it using wget, and it appears the problem is, that the destination server identifies the desired resource by using both the request URL and the hostname in the Host header. Since the hostname does not match, it is unable to locate the resource.
I have (unsuccessfully) tried to override the Host header by setting the http.virtual-host parameter on the client like this:
HttpClient client = new DefaultHttpClient();
if (envType.isWithProxy()) {
client.getParams().setParameter(ClientPNames.VIRTUAL_HOST, "api.twitter.com");
}
Technical details:
Client is used as an executor in RESTeasy to call the REST API. So "manually" setting the virtual host (as described here) is not an option.
Everything is done via HTTPS/SSL - not that I think it makes a difference.
Edit 1: Using a HttpHost instead of a String does not have the desired effect either:
HttpClient client = new DefaultHttpClient();
if (envType.isWithProxy()) {
HttpHost realHost = new HttpHost("api.twitter.com", port, scheme);
client.getParams().setParameter(ClientPNames.VIRTUAL_HOST, realHost);
}
Edit 2: Further investigation has revealed, that the parameter needs to be set on the request object. The following is the code v. 4.2-aplha1 of HttpClient setting the virtual host:
HttpRequest orig = request;
RequestWrapper origWrapper = wrapRequest(orig);
origWrapper.setParams(params);
HttpRoute origRoute = determineRoute(target, origWrapper, context);
virtualHost = (HttpHost) orig.getParams().getParameter(
ClientPNames.VIRTUAL_HOST);
paramsare the parameters passed from the client. But the value for 'virtualHost' is read from the request parameters.
So this changes the nature of the question to: How do I set the VIRTUAL_HOST property on the requests?

ClientPNames.VIRTUAL_HOST is the right parameter for overriding physical host name in HTTP requests. I would just recommend setting this parameter on the request object instead of the client object. If that does not produce the desired effect please post the complete wire / context log of the session (see logging guide for instructions) either here or to the HttpClient user list.
Follow-up
OK. Let's take a larger sledge hammer. One can override content of the Host header using an interceptor.
DefaultHttpClient client = new DefaultHttpClient();
client.addRequestInterceptor(new HttpRequestInterceptor() {
public void process(
final HttpRequest request,
final HttpContext context) throws HttpException, IOException {
request.setHeader(HTTP.TARGET_HOST, "www.whatever.com");
}
});
One can make the interceptor clever enough to override the header selectively, only for specific hosts.

ProxySelector changes URL's scheme from https:// to socket://

I need to access Facebook but all outgoing communication is blocked on our server so I have to use proxy.
I initialize proxies with:
ProxySelector.setDefault(new ConfigurableProxySelector(mapping));
Proxy type is HTTP, proxy host and port are working (confirmed by simple wget test).
I'm trying to do this:
HttpClient httpClient = new HttpClient();
HttpMethod method = new GetMethod("https://graph.facebook.com:443");
int status = httpClient.executeMethod(method);
Now, in my class ConfigurableProxySelector I have select method on which I have breakpoint:
public List<Proxy> select(URI uri) {
...
}
So, using HttpClient I make an request, which should be proxied and code stops at breakpoint in select() method in ConfigurableProxySelector.
But what is strange is that uri.scheme = "socket" and .toString() gives "socket://graph.facebook.com:443" instead of "https://graph.facebook.com:443".
Because ProxySelector have mapping for "https://" and not for "socket://", it does not find it and it ends with "Connection refused". What is strange is that select() method is called 4 times before execution ends with "Connection refused".
Any help would be appreciated.

Apache HTTP Client 3.1 will not natively honor HTTP Proxies returned from the default ProxySelector or user implementations.
Quick Summary of ProxySelector
ProxySelector is a service class which selects and returns a suitable Proxy for a given URL based on its scheme. For example, a request for http://somehost will try to provide an HTTP proxy if one is defined. The default ProxySelector can be configured at runtime using System Properties, such as http.proxyHost and http.proxyPort.
HTTPUrlConnection
An instance of HTTPUrlConnection will check against the default ProxySelector multiple times: 1st to select for http or https, then later when it builds the raw tcp socket, using the socket scheme. A SOCKS proxy could be used to proxy a raw tcp socket but are not often found in corporate environments, so a raw tcp socket will usually receive no proxy.
HTTP Client 3.1
HC 3.1, on the other hand, will never check the default ProxySelector for the http/https schemes. It will check, however, at a later points for the socket scheme when it eventually builds the raw socket - This is the request you are seeing. This means the System Properties http.proxyHost and http.proxyPort are ineffective. This is obviously not ideal for most people who only have an HTTP/HTTPS proxy.
To work around this, you have two options: define a proxy on each HC 3.1 connection or implement your own HC 3.1 HTTPConnectionManager.
HTTPConnectionManager
The HTTPConnectionManager is responsible for building connections for the HC 3.1 client.
The default HC 3.1 HTTPConnectionManager can be extended so that it looks for a suitable proxy from a ProxySelector (default or custom) when building the request in the same way HTTPUrlConnection does:
public class MyHTTPConnectionManager extends SimpleHttpConnectionManager {
#Override
public HttpConnection getConnectionWithTimeout(
HostConfiguration hostConfiguration, long timeout) {
HttpConnection hc = super.getConnectionWithTimeout(hostConfiguration, timeout);
try {
URI uri = new URI( hostConfiguration.getHostURL());
List<Proxy> hostProxies = ProxySelector.getDefault().select(uri);
Proxy Proxy = hostProxies.get(0);
InetSocketAddress sa = (InetSocketAddress) Proxy.address();
hc.setProxyHost(sa.getHostName());
hc.setProxyPort(sa.getPort());
} catch (URISyntaxException e) {
return hc;
}
return hc;
}
}
Then, when you create an HC 3.1 client, use your new connection manager:
HttpClient client = new HttpClient(new MyHTTPConnectionManager() );

It's not the ProxySelector that changes the scheme, but the SocketFactory opening a Socket.
If the SocketFactory is null a SOCKS socket will be created by default which only allows SOCKS proxies. I don't know anything about Sockets and cannot tell you if there's a way to make it work with HTTP proxies.
But using another approach may help, since Apache HttpClient seems to have its own way to configure proxies.
client.getHostConfiguration().setProxy(proxyHost, proxyPort);
if (proxyUser != null) {
client.getState().setProxyCredentials(new AuthScope(proxyHost, proxyPort),
new UsernamePasswordCredentials(proxyUser, proxyPassword));
}