I need to access Facebook but all outgoing communication is blocked on our server so I have to use proxy.
I initialize proxies with:
ProxySelector.setDefault(new ConfigurableProxySelector(mapping));
Proxy type is HTTP, proxy host and port are working (confirmed by simple wget test).
I'm trying to do this:
HttpClient httpClient = new HttpClient();
HttpMethod method = new GetMethod("https://graph.facebook.com:443");
int status = httpClient.executeMethod(method);
Now, in my class ConfigurableProxySelector I have select method on which I have breakpoint:
public List<Proxy> select(URI uri) {
...
}
So, using HttpClient I make an request, which should be proxied and code stops at breakpoint in select() method in ConfigurableProxySelector.
But what is strange is that uri.scheme = "socket" and .toString() gives "socket://graph.facebook.com:443" instead of "https://graph.facebook.com:443".
Because ProxySelector have mapping for "https://" and not for "socket://", it does not find it and it ends with "Connection refused". What is strange is that select() method is called 4 times before execution ends with "Connection refused".
Any help would be appreciated.
Apache HTTP Client 3.1 will not natively honor HTTP Proxies returned from the default ProxySelector or user implementations.
Quick Summary of ProxySelector
ProxySelector is a service class which selects and returns a suitable Proxy for a given URL based on its scheme. For example, a request for http://somehost will try to provide an HTTP proxy if one is defined. The default ProxySelector can be configured at runtime using System Properties, such as http.proxyHost and http.proxyPort.
HTTPUrlConnection
An instance of HTTPUrlConnection will check against the default ProxySelector multiple times: 1st to select for http or https, then later when it builds the raw tcp socket, using the socket scheme. A SOCKS proxy could be used to proxy a raw tcp socket but are not often found in corporate environments, so a raw tcp socket will usually receive no proxy.
HTTP Client 3.1
HC 3.1, on the other hand, will never check the default ProxySelector for the http/https schemes. It will check, however, at a later points for the socket scheme when it eventually builds the raw socket - This is the request you are seeing. This means the System Properties http.proxyHost and http.proxyPort are ineffective. This is obviously not ideal for most people who only have an HTTP/HTTPS proxy.
To work around this, you have two options: define a proxy on each HC 3.1 connection or implement your own HC 3.1 HTTPConnectionManager.
HTTPConnectionManager
The HTTPConnectionManager is responsible for building connections for the HC 3.1 client.
The default HC 3.1 HTTPConnectionManager can be extended so that it looks for a suitable proxy from a ProxySelector (default or custom) when building the request in the same way HTTPUrlConnection does:
public class MyHTTPConnectionManager extends SimpleHttpConnectionManager {
#Override
public HttpConnection getConnectionWithTimeout(
HostConfiguration hostConfiguration, long timeout) {
HttpConnection hc = super.getConnectionWithTimeout(hostConfiguration, timeout);
try {
URI uri = new URI( hostConfiguration.getHostURL());
List<Proxy> hostProxies = ProxySelector.getDefault().select(uri);
Proxy Proxy = hostProxies.get(0);
InetSocketAddress sa = (InetSocketAddress) Proxy.address();
hc.setProxyHost(sa.getHostName());
hc.setProxyPort(sa.getPort());
} catch (URISyntaxException e) {
return hc;
}
return hc;
}
}
Then, when you create an HC 3.1 client, use your new connection manager:
HttpClient client = new HttpClient(new MyHTTPConnectionManager() );
It's not the ProxySelector that changes the scheme, but the SocketFactory opening a Socket.
If the SocketFactory is null a SOCKS socket will be created by default which only allows SOCKS proxies. I don't know anything about Sockets and cannot tell you if there's a way to make it work with HTTP proxies.
But using another approach may help, since Apache HttpClient seems to have its own way to configure proxies.
client.getHostConfiguration().setProxy(proxyHost, proxyPort);
if (proxyUser != null) {
client.getState().setProxyCredentials(new AuthScope(proxyHost, proxyPort),
new UsernamePasswordCredentials(proxyUser, proxyPassword));
}
Related
Curl has a feature for manually specifying which IP to resolve a host to. For example:
curl https://google.com --resolve "google.com:443:173.194.72.113"
This is particularly useful when using HTTPS. If it was just a HTTP request, I could have achieved the same by specifying the IP address directly, and adding a host header. But in HTTPS that would break the connection since the SSL certificate host would be compared to the IP address and not the host header.
My question is, how can I achieve the same thing in Java?
If using Apache's HttpClient, you can create a custom DNS resolver to detect the host you'd like to redirect, and then provide a substitute IP address.
Note: Just changing the Host header for HTTPS requests doesn't work. It will
throw "javax.net.ssl.SSLPeerUnverifiedException", forcing you to trust bad
certificates, stop SNI from working, etc., so really not an option. A
custom DnsResolver is the only clean way I've found to get these requests to work
with HTTPS in Java.
Example:
/* Custom DNS resolver */
DnsResolver dnsResolver = new SystemDefaultDnsResolver() {
#Override
public InetAddress[] resolve(final String host) throws UnknownHostException {
if (host.equalsIgnoreCase("my.host.com")) {
/* If we match the host we're trying to talk to,
return the IP address we want, not what is in DNS */
return new InetAddress[] { InetAddress.getByName("127.0.0.1") };
} else {
/* Else, resolve it as we would normally */
return super.resolve(host);
}
}
};
/* HttpClientConnectionManager allows us to use custom DnsResolver */
BasicHttpClientConnectionManager connManager = new BasicHttpClientConnectionManager(
/* We're forced to create a SocketFactory Registry. Passing null
doesn't force a default Registry, so we re-invent the wheel. */
RegistryBuilder.<ConnectionSocketFactory>create()
.register("http", PlainConnectionSocketFactory.getSocketFactory())
.register("https", SSLConnectionSocketFactory.getSocketFactory())
.build(),
null, /* Default ConnectionFactory */
null, /* Default SchemePortResolver */
dnsResolver /* Our DnsResolver */
);
/* build HttpClient that will use our DnsResolver */
HttpClient httpClient = HttpClientBuilder.create()
.setConnectionManager(connManager)
.build();
/* build our request */
HttpGet httpRequest = new HttpGet("https://my.host.com/page?and=stuff");
/* Executing our request should now hit 127.0.0.1, regardless of DNS */
HttpResponse httpResponse = httpClient.execute(httpRequest);
I don't have the code close at hand, but you can also write your own SSL handler/checker that could adapt or flat-out just ignore all the security. Using the JDK base networking, we had to totally ignore SSL certificates internally for testing. Should be easy to find examples.
I am using com.sun.net.httpserver.HttpsServer in my current project which deals with client-authentification etc.. Currently it only prints out the clients address/port, so that I can check if one TCP-connection is used for multiple requests (keep-alive) or if a new connection is established for every request (and thus a new SSL-handshake is made every time). When I use FireFox to make multiple request against the server I can see that keep-alive is working. So the server part works fine with GET and POST-requests.
If I use HttpURLConnection to make a request against the Server (in this case using no SSL) keep-alive works, too: Only one connection is established for multiple sequentially started requests.
But if I use HttpsURLConnection (using exactly the same code, but using SSL) then keep-alive is not working anymore. So for each request a new connection is established, although I am using the same SSLContext (and SSLSocketFactory):
// URL myUrl = ...
// SSLContext mySsl = ...
HttpsURLConnection conn = (HttpsURLConnection) myUrl.openConnection();
conn.setUseCaches(false);
conn.setSSLSocketFactory(mySsl.getSocketFactory());
conn.setRequestMethod("POST");
// send Data
// receive Data
How do I force HttpsURLConnection to use keep-alive because many requests will lead to many SSL-handshakes which is a real performance issue?
Update (2012-04-02):
Instead of calling mySsl.getSocketFactory() each time, I tried to cache the SSLSocketFactory. But nothing changed. The problem still exists.
I ran into this exact same problem and finally have a solution after some in-depth debugging.
Http(s)UrlConnection does handle Keep-Alive by default but sockets must be in a very specific condition in order to be reused.
These are:
Input streams must be fully consumed. You must call read on the input stream until it returns -1 and also close it.
Settings on the underlying socket must use the exact same objects.
You should call disconnect (yes this is counter-intuitive) on the Http(s)URLConnection when done with it.
In the above code, the problem is:
conn.setSSLSocketFactory(mySsl.getSocketFactory());
Saving the result of getSocketFactory() to a static variable during initialization and then passing that in to conn.setSSLSocketFactory should allow the socket to be reused.
I couldn't get it working with HttpsUrlConnection. But Apache's HTTP client handles keep-alive with SSL connections very well.
SSL connection establishment is really expensive either for service calls or when getting many resources from a browser.
Java Http(s)UrlConnection handles HTTP(S) Keep-Alive by default.
I have not found the source code of the default SSLSocketFactory and probably the keep-alive mechanism is implemented there. As a confirmation, disable your own SSLSocketFactory implementation for a test, with a custom trust store in javax.net.ssl.trustStore so that your self-signed certificate is accepted.
According to OpenJDK 7 ServerImpl implementation which uses ServerConfig the HttpsServer you used emits a keep-alive with 5 minutes timeout per default.
I propose you set the property sun.net.httpserver.debug to true server-side to get details.
Take care your code does not add the header Connection: close which disables keep-alive mechanism.
As far as I can understand HTTP/1.1 and HTTPS protocol, also documented here, Keep-Alive is not an end-to-end header but a hop-to-hop header. Since SSL involves multiple steps of handshaking among "different hops" (e.g. CA and the server) for each new connection, I think Keep-Alive may not be applicable in SSL context. So, that can be why Keep-Alive header is ignored using HTTPS connections. Based on this this question, you may need to ensure one instance of HTTP connection is used to guarantee Keep-Alive observation. Also, in the question, it seems that Apache HTTPClient has been a better solution.
We may setup an Apache Webserver, add following directives to see whether the Apache's access.log has a keep-alive connection for the http client.
LogFormat "%k %v %h %l %u %t \"%r\" %>s %b" common
CustomLog "logs/access.log" common
http://httpd.apache.org/docs/current/mod/mod_log_config.html
"%k" Number of keepalive requests handled on this connection. Interesting if KeepAlive is being used, so that, for example, a '1' means the first keepalive request after the initial one, '2' the second, etc...; otherwise this is always 0 (indicating the initial request).
I faced the same problem, and Bill Healey is right.
I tested my example code below with few https libraries.
HttpsURLConnection and OKHTTP are exact same behavior.
Volley is a bit different when session resumption, but almost same behavior.
I hope this will be some help.
public class SampleActivity extends Activity implements OnClickListener {
// Keep default context and factory
private SSLContext mDefaultSslContext;
private SSLSocketFactory mDefaultSslFactory;
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
findViewById(R.id.button_id).setOnClickListener(this);
try {
// Initialize context and factory
mDefaultSslContext = SSLContext.getInstance("TLS");
mDefaultSslContext.init(null, null, null);
mDefaultSslFactory = mDefaultSslContext.getSocketFactory();
} catch (NoSuchAlgorithmException | KeyManagementException e) {
Log.e(TAG, e.getMessage(), e);
}
}
#Override
public void onClick(View v){
SSLContext sslcontext;
SSLSocketFactory sslfactory;
try {
// If using this factory, enable Keep-Alive
sslfactory = mDefaultSslFactory;
// If using this factory, enable session resumption (abbreviated handshake)
sslfactory = mDefaultSslContext.getSocketFactory();
// If using this factory, enable full handshake each time
sslcontext = SSLContext.getInstance("TLS");
sslcontext.init(null, null, null);
sslfactory = sslcontext.getSocketFactory();
} catch (NoSuchAlgorithmException | KeyManagementException e) {
Log.e(TAG, e.getMessage(), e);
}
URL url = new URL("https://example.com");
HttpsURLConnection = conn = (HttpsURLConnection) url.openConnection();
conn.setSSLSocketFactory(sslfactory);
conn.connect();
}
}
Update:
Sharing SSLSocketFactory enables keep-alive. Sharing SSLContext and getting facotry each request enable session resumption. I don't know how TLS stack works, but just confirmed these connection behaviors with some mobile devices.
If you want to enable keep-alive among multiple classes, you should share the instance of SSLSocketFactory using singleton pattern.
If you want to enable session resumption, make sure the session timeout settings is long enough on server side, such as SSLSessionCacheTimeout(apache), ssl_session_timeout(nginx).
In addition to #Bill Healey answer the HostnameVerifier also must be declared static.
I've tried several patterns with and without closing input stream and connection they make no change for me. The only thing that matters is the static declarations of mentioned properties.
/**
SSLSocketFactory and HostnameVerifier must be declared static in order to be able to use keep-alive option
*/
private static SSLSocketFactory factory = null;
private static HostnameVerifier hostnameVerifier = new HostnameVerifier() {
#Override
public boolean verify(String s, SSLSession sslSession) {
return true;
}
};
public static void prepareForCustomTrustIfNeeded(HttpsURLConnection connection) {
try {
if(factory == null) {
SSLContext sslc = SSLContext.getInstance("TLS");
sslc.init(null, customTrustedCerts, new SecureRandom());
factory = sslc.getSocketFactory();
}
connection.setSSLSocketFactory(factory);
connection.setHostnameVerifier(hostnameVerifier);
} catch (Exception e) {
e.printStackTrace();
}
}
try to add the following code:
con.setRequestProperty("Connection", "Keep-Alive");
con.setRequestProperty("Keep-Alive", "header");
I am having a problem getting the Apache HttpClient to connect to a service external to my virtualised development environment.
To access the internet (e.g. api.twitter.com) I need to call a local URL (e.g. api.twitter.com.dev.mycompany.net), which then forwards the request to real host.
The problem is, that to whatever request I send, I get a 404 Not Found response.
I have tried debugging it using wget, and it appears the problem is, that the destination server identifies the desired resource by using both the request URL and the hostname in the Host header. Since the hostname does not match, it is unable to locate the resource.
I have (unsuccessfully) tried to override the Host header by setting the http.virtual-host parameter on the client like this:
HttpClient client = new DefaultHttpClient();
if (envType.isWithProxy()) {
client.getParams().setParameter(ClientPNames.VIRTUAL_HOST, "api.twitter.com");
}
Technical details:
Client is used as an executor in RESTeasy to call the REST API. So "manually" setting the virtual host (as described here) is not an option.
Everything is done via HTTPS/SSL - not that I think it makes a difference.
Edit 1: Using a HttpHost instead of a String does not have the desired effect either:
HttpClient client = new DefaultHttpClient();
if (envType.isWithProxy()) {
HttpHost realHost = new HttpHost("api.twitter.com", port, scheme);
client.getParams().setParameter(ClientPNames.VIRTUAL_HOST, realHost);
}
Edit 2: Further investigation has revealed, that the parameter needs to be set on the request object. The following is the code v. 4.2-aplha1 of HttpClient setting the virtual host:
HttpRequest orig = request;
RequestWrapper origWrapper = wrapRequest(orig);
origWrapper.setParams(params);
HttpRoute origRoute = determineRoute(target, origWrapper, context);
virtualHost = (HttpHost) orig.getParams().getParameter(
ClientPNames.VIRTUAL_HOST);
paramsare the parameters passed from the client. But the value for 'virtualHost' is read from the request parameters.
So this changes the nature of the question to: How do I set the VIRTUAL_HOST property on the requests?
ClientPNames.VIRTUAL_HOST is the right parameter for overriding physical host name in HTTP requests. I would just recommend setting this parameter on the request object instead of the client object. If that does not produce the desired effect please post the complete wire / context log of the session (see logging guide for instructions) either here or to the HttpClient user list.
Follow-up
OK. Let's take a larger sledge hammer. One can override content of the Host header using an interceptor.
DefaultHttpClient client = new DefaultHttpClient();
client.addRequestInterceptor(new HttpRequestInterceptor() {
public void process(
final HttpRequest request,
final HttpContext context) throws HttpException, IOException {
request.setHeader(HTTP.TARGET_HOST, "www.whatever.com");
}
});
One can make the interceptor clever enough to override the header selectively, only for specific hosts.
I am using Jakarta Commons HttpClient 3.1 writing a load test tool that needs to target different servers and pretend like it targeted the correct virtual host in the HTTP server. For that I need to be able to set the "Host" HTTP header in the request to a different host name then the actual host name that I'm connecting to.
It seemed pretty obvious that I should use Method.setRequestHeader("Host","fakehostname"), but HttpClient just ignores this and always sends the real host name I'm connecting to in the "Host" header (I've enabled debug logging for "httpclient.wire" and I can it does this specifically).
How can I override the header so that HttpClient takes heed?
After searching some more, and taking a hint from Oleg's answer, I've found the method HttpMethodParams::setVirtualHost().
when HttpClient formats a request, it always creates the "Host" header itself just before sending the request - so it cannot be overridden as a standard header. But before the host name for the "Host" header is generated from the URL, HttpClient checks the HttpMethodParams object to see if the user wants to override the host name. This only overrides the host name and not the port so it would be easier to use, though not as intuitive as I'd like.
The code to use this can look like this:
Method m = new GetMethod("http://some-site/some/path");
m.getParams().setVirtualHost("some-other-site");
client.executeMethod(m);
Because I like one liners, this can also be written as:
client.executeMethod(new GetMethod("http://some-site/some/path") {{
getParams().setVirtualHost("some-other-site"); }});
I believe you want http://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/HttpHost.html: this lets you configure the host for a specific connection. If I understand it correctly, you can either use the execute method (see http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/AbstractHttpClient.html#execute(org.apache.http.HttpHost,%20org.apache.http.HttpRequest)) and pass it a custom HttpHost object, or do this:
Construct an HttpHost instance, passing it your Host header.
Use that to create an HttpRoute instance (see http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/routing/HttpRoute.html)
Pass that to the connection manager when you request a connection (see http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/ClientConnectionManager.html#requestConnection(org.apache.http.conn.routing.HttpRoute,%20java.lang.Object)).
Use the connection with your method: see http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html for more details.
Let me know how that works.
EDIT: principle remains the same.
1. Construct an HttpHost instance, passing it your Host header (see http://hc.apache.org/httpclient-legacy/apidocs/index.html?org/apache/commons/httpclient/HttpHost.html).
2. Create an HttpConfiguration instance and then pass it the HttpHost you created (see http://hc.apache.org/httpclient-legacy/apidocs/index.html?org/apache/commons/httpclient/HostConfiguration.html).
3. Use the execute method on HttpClient with that configuration (see http://hc.apache.org/httpclient-legacy/apidocs/org/apache/commons/httpclient/HttpClient.html#executeMethod(org.apache.commons.httpclient.HostConfiguration,%20org.apache.commons.httpclient.HttpMethod))
Following works on android:
System.setProperty("sun.net.http.allowRestrictedHeaders", "true");
InputStream stream_content=null;
try
{URL url=new URL("http://74.125.28.103/");
HttpURLConnection conn=(HttpURLConnection)url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("GET");
conn.setRequestProperty("Host", "www.google.com");
stream_content=conn.getInputStream();
}
catch (Exception e) {}
for https url:
System.setProperty("sun.net.http.allowRestrictedHeaders", "true");
InputStream stream_content=null;
try
{URL url=new URL("https://74.125.28.103/");
HttpsURLConnection conn=(HttpsURLConnection)url.openConnection();
conn.setHostnameVerifier(SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER );
conn.setDoOutput(true);
conn.setRequestMethod("GET");
conn.setRequestProperty("Host", "www.google.com");
stream_content=conn.getInputStream();
}
catch (Exception e) {}
One can use the 'http.virtual-host' parameter in order to force an arbitrary (virtual) hostname and port as a value of the Host request header instead of those derived from the actual request URI. This works with the 4.x API only, though.
I need to establish and send/read over/from an https connection (to a website of course) but through an http proxy or SOCKS proxy. A few other requirements
supports blocking (I can't use non-blocking/nio)
isn't set as an environment or some other global scope property (there are multiple threads accessing)
I was looking into HttpCore components but I did not see any support for blocking https.
Look at the java.net.Proxy class. That does what you need. You create one, and then pass it to the URLConnection to create the connection.
To support per-thread proxy, your best bet is Apache HttpClient 4 (Http Components Client). Get the source code,
http://hc.apache.org/downloads.cgi
It comes with examples for both HTTP proxy and SOCKS proxy,
ClientExecuteProxy.java
ClientExecuteSOCKS.java
Did you look at Apache HTTP Client? Haven't used it in ages but I did use it to pick a proxy server dynamically. Example from site here:
HttpClient httpclient = new HttpClient();
httpclient.getHostConfiguration().setProxy("myproxyhost", 8080);
httpclient.getState().setProxyCredentials("my-proxy-realm", " myproxyhost",
new UsernamePasswordCredentials("my-proxy-username", "my-proxy-password"));
GetMethod httpget = new GetMethod("https://www.verisign.com/");
try {
httpclient.executeMethod(httpget);
System.out.println(httpget.getStatusLine());
} finally {
httpget.releaseConnection();
}
System.setProperty("http.proxyHost", "proxy.com");
System.setPropery("http.proxyPort", "8080");
URL url = new URL("http://java.sun.com/");
InputStream in = url.openStream();
http://java.sun.com/javase/6/docs/technotes/guides/net/proxies.html