I need to establish and send/read over/from an https connection (to a website of course) but through an http proxy or SOCKS proxy. A few other requirements
supports blocking (I can't use non-blocking/nio)
isn't set as an environment or some other global scope property (there are multiple threads accessing)
I was looking into HttpCore components but I did not see any support for blocking https.
Look at the java.net.Proxy class. That does what you need. You create one, and then pass it to the URLConnection to create the connection.
To support per-thread proxy, your best bet is Apache HttpClient 4 (Http Components Client). Get the source code,
http://hc.apache.org/downloads.cgi
It comes with examples for both HTTP proxy and SOCKS proxy,
ClientExecuteProxy.java
ClientExecuteSOCKS.java
Did you look at Apache HTTP Client? Haven't used it in ages but I did use it to pick a proxy server dynamically. Example from site here:
HttpClient httpclient = new HttpClient();
httpclient.getHostConfiguration().setProxy("myproxyhost", 8080);
httpclient.getState().setProxyCredentials("my-proxy-realm", " myproxyhost",
new UsernamePasswordCredentials("my-proxy-username", "my-proxy-password"));
GetMethod httpget = new GetMethod("https://www.verisign.com/");
try {
httpclient.executeMethod(httpget);
System.out.println(httpget.getStatusLine());
} finally {
httpget.releaseConnection();
}
System.setProperty("http.proxyHost", "proxy.com");
System.setPropery("http.proxyPort", "8080");
URL url = new URL("http://java.sun.com/");
InputStream in = url.openStream();
http://java.sun.com/javase/6/docs/technotes/guides/net/proxies.html
Related
There is a file that will be downloaded when I make a get request to particular URL. I am able to get InputStream from both ways.
Method 1
Using URL class in java.net package.
java.net.URL url = new URL(downloadFileUrl);
InputStream inputStream = url.openStream();
Method 2
Using Apache's HttpClient class.
org.apache.http.impl.client.CloseableHttpClient httpclient = new CloseableHttpClient();
HttpGet request = new HttpGet(url);
CloseableHttpResponse response = httpclient.execute((HttpUriRequest)request);
InputStream inputStream = response.getEntity().getContent();
Are these methods the same? If not how? Which method is preferred generally or in a specific situation?
The examples I provided are simplistic. Assume I did the neccessary
congifurations with the URL and HttpClient objects to get successful response.
Both methods returns the input stream to read from the connection. There isn't difference between these methods. Since HttpClient is third party library, you need to keep a check for any vulnerabilities and keep updating the library.
Only difference is HttpClient supports only HTTP(s) protocol, whereas URLConnection can be used for other protocols too like FTP
In terms of functionalities, Apache HttpClient has more fine tuning options than URLConnection
I am very new to Jersey and I did a search but unable to figure out whether Is there a way in jersey client to use connection pooling instead of creating a connection each and every time we are sending a new request.
The whole idea is to reuse set of connection from the pool, which will save lots or resource.
FYI I'm not looking for Connection: keep-alive.
This is what I'm doing now
public void postData()
{
Client client = new Client();
WebResource webResource = client.resource("http://SomeService.com/..");
ClientResponse response = webResource.accept("text/plain").get(ClientResponse.class);
System.out.println(response.getStatus());
System.out.println(response.getEntity(String.class));
}
Any help is highly appreciable,Expecting code snippet. Thanks in advance.
You can configure Jersey client to use Apache HttpClient with connection pooling. Details of how to do so can be found on this blog post. Note that the post itself covers Jersey 2.x, but there is a gist for Jersey 1.x mentioned in the comments.
I'm trying to make a jersey client call using NTLM proxy? is that possible as i was not able to get any clear information on the same. Did anyone tried before?
Yes it is possible to configure the Jersey Client to connect through a proxy server that requires NTLM authentication.
Here is a simplified code snippet that prepares a suitable ClientConfig that should work with Jersey v2.5+:
final ClientConfig config = new ClientConfig();
config.property(ClientProperties.PROXY_URI, "http://myproxy.com:8000");
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
final AuthScope ntlmAuthScope =
new AuthScope("myproxy.com", 8000, AuthScope.ANY_REALM, "NTLM");
credentialsProvider.setCredentials(
ntlmAuthScope,
new NTCredentials("user", "password", "hostname", "domain") );
config.property(
ApacheClientProperties.CREDENTIALS_PROVIDER, credentialsProvider);
config.connectorProvider(new ApacheConnectorProvider());
Client client = ClientBuilder.newClient(config);
Please note: I am using the Apache HttpClient connector with Jersey Client - you may require slightly different code if you are using another client transport connector.
You may also need to add the following line to your code if you want your POST/PUT requests to be buffered (and therefore repeatable) in response to any 407 authentication challenges that come back from your proxy server:
config.property(ClientProperties.REQUEST_ENTITY_PROCESSING,
RequestEntityProcessing.BUFFERED);
I need to access Facebook but all outgoing communication is blocked on our server so I have to use proxy.
I initialize proxies with:
ProxySelector.setDefault(new ConfigurableProxySelector(mapping));
Proxy type is HTTP, proxy host and port are working (confirmed by simple wget test).
I'm trying to do this:
HttpClient httpClient = new HttpClient();
HttpMethod method = new GetMethod("https://graph.facebook.com:443");
int status = httpClient.executeMethod(method);
Now, in my class ConfigurableProxySelector I have select method on which I have breakpoint:
public List<Proxy> select(URI uri) {
...
}
So, using HttpClient I make an request, which should be proxied and code stops at breakpoint in select() method in ConfigurableProxySelector.
But what is strange is that uri.scheme = "socket" and .toString() gives "socket://graph.facebook.com:443" instead of "https://graph.facebook.com:443".
Because ProxySelector have mapping for "https://" and not for "socket://", it does not find it and it ends with "Connection refused". What is strange is that select() method is called 4 times before execution ends with "Connection refused".
Any help would be appreciated.
Apache HTTP Client 3.1 will not natively honor HTTP Proxies returned from the default ProxySelector or user implementations.
Quick Summary of ProxySelector
ProxySelector is a service class which selects and returns a suitable Proxy for a given URL based on its scheme. For example, a request for http://somehost will try to provide an HTTP proxy if one is defined. The default ProxySelector can be configured at runtime using System Properties, such as http.proxyHost and http.proxyPort.
HTTPUrlConnection
An instance of HTTPUrlConnection will check against the default ProxySelector multiple times: 1st to select for http or https, then later when it builds the raw tcp socket, using the socket scheme. A SOCKS proxy could be used to proxy a raw tcp socket but are not often found in corporate environments, so a raw tcp socket will usually receive no proxy.
HTTP Client 3.1
HC 3.1, on the other hand, will never check the default ProxySelector for the http/https schemes. It will check, however, at a later points for the socket scheme when it eventually builds the raw socket - This is the request you are seeing. This means the System Properties http.proxyHost and http.proxyPort are ineffective. This is obviously not ideal for most people who only have an HTTP/HTTPS proxy.
To work around this, you have two options: define a proxy on each HC 3.1 connection or implement your own HC 3.1 HTTPConnectionManager.
HTTPConnectionManager
The HTTPConnectionManager is responsible for building connections for the HC 3.1 client.
The default HC 3.1 HTTPConnectionManager can be extended so that it looks for a suitable proxy from a ProxySelector (default or custom) when building the request in the same way HTTPUrlConnection does:
public class MyHTTPConnectionManager extends SimpleHttpConnectionManager {
#Override
public HttpConnection getConnectionWithTimeout(
HostConfiguration hostConfiguration, long timeout) {
HttpConnection hc = super.getConnectionWithTimeout(hostConfiguration, timeout);
try {
URI uri = new URI( hostConfiguration.getHostURL());
List<Proxy> hostProxies = ProxySelector.getDefault().select(uri);
Proxy Proxy = hostProxies.get(0);
InetSocketAddress sa = (InetSocketAddress) Proxy.address();
hc.setProxyHost(sa.getHostName());
hc.setProxyPort(sa.getPort());
} catch (URISyntaxException e) {
return hc;
}
return hc;
}
}
Then, when you create an HC 3.1 client, use your new connection manager:
HttpClient client = new HttpClient(new MyHTTPConnectionManager() );
It's not the ProxySelector that changes the scheme, but the SocketFactory opening a Socket.
If the SocketFactory is null a SOCKS socket will be created by default which only allows SOCKS proxies. I don't know anything about Sockets and cannot tell you if there's a way to make it work with HTTP proxies.
But using another approach may help, since Apache HttpClient seems to have its own way to configure proxies.
client.getHostConfiguration().setProxy(proxyHost, proxyPort);
if (proxyUser != null) {
client.getState().setProxyCredentials(new AuthScope(proxyHost, proxyPort),
new UsernamePasswordCredentials(proxyUser, proxyPassword));
}
I am using Jakarta Commons HttpClient 3.1 writing a load test tool that needs to target different servers and pretend like it targeted the correct virtual host in the HTTP server. For that I need to be able to set the "Host" HTTP header in the request to a different host name then the actual host name that I'm connecting to.
It seemed pretty obvious that I should use Method.setRequestHeader("Host","fakehostname"), but HttpClient just ignores this and always sends the real host name I'm connecting to in the "Host" header (I've enabled debug logging for "httpclient.wire" and I can it does this specifically).
How can I override the header so that HttpClient takes heed?
After searching some more, and taking a hint from Oleg's answer, I've found the method HttpMethodParams::setVirtualHost().
when HttpClient formats a request, it always creates the "Host" header itself just before sending the request - so it cannot be overridden as a standard header. But before the host name for the "Host" header is generated from the URL, HttpClient checks the HttpMethodParams object to see if the user wants to override the host name. This only overrides the host name and not the port so it would be easier to use, though not as intuitive as I'd like.
The code to use this can look like this:
Method m = new GetMethod("http://some-site/some/path");
m.getParams().setVirtualHost("some-other-site");
client.executeMethod(m);
Because I like one liners, this can also be written as:
client.executeMethod(new GetMethod("http://some-site/some/path") {{
getParams().setVirtualHost("some-other-site"); }});
I believe you want http://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/HttpHost.html: this lets you configure the host for a specific connection. If I understand it correctly, you can either use the execute method (see http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/AbstractHttpClient.html#execute(org.apache.http.HttpHost,%20org.apache.http.HttpRequest)) and pass it a custom HttpHost object, or do this:
Construct an HttpHost instance, passing it your Host header.
Use that to create an HttpRoute instance (see http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/routing/HttpRoute.html)
Pass that to the connection manager when you request a connection (see http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/ClientConnectionManager.html#requestConnection(org.apache.http.conn.routing.HttpRoute,%20java.lang.Object)).
Use the connection with your method: see http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html for more details.
Let me know how that works.
EDIT: principle remains the same.
1. Construct an HttpHost instance, passing it your Host header (see http://hc.apache.org/httpclient-legacy/apidocs/index.html?org/apache/commons/httpclient/HttpHost.html).
2. Create an HttpConfiguration instance and then pass it the HttpHost you created (see http://hc.apache.org/httpclient-legacy/apidocs/index.html?org/apache/commons/httpclient/HostConfiguration.html).
3. Use the execute method on HttpClient with that configuration (see http://hc.apache.org/httpclient-legacy/apidocs/org/apache/commons/httpclient/HttpClient.html#executeMethod(org.apache.commons.httpclient.HostConfiguration,%20org.apache.commons.httpclient.HttpMethod))
Following works on android:
System.setProperty("sun.net.http.allowRestrictedHeaders", "true");
InputStream stream_content=null;
try
{URL url=new URL("http://74.125.28.103/");
HttpURLConnection conn=(HttpURLConnection)url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("GET");
conn.setRequestProperty("Host", "www.google.com");
stream_content=conn.getInputStream();
}
catch (Exception e) {}
for https url:
System.setProperty("sun.net.http.allowRestrictedHeaders", "true");
InputStream stream_content=null;
try
{URL url=new URL("https://74.125.28.103/");
HttpsURLConnection conn=(HttpsURLConnection)url.openConnection();
conn.setHostnameVerifier(SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER );
conn.setDoOutput(true);
conn.setRequestMethod("GET");
conn.setRequestProperty("Host", "www.google.com");
stream_content=conn.getInputStream();
}
catch (Exception e) {}
One can use the 'http.virtual-host' parameter in order to force an arbitrary (virtual) hostname and port as a value of the Host request header instead of those derived from the actual request URI. This works with the 4.x API only, though.