I am trying to set proxy for a request I am making using HttpClientBuilder as follows:
CredentialsProvider credsProvider = new BasicCredentialsProvider();
UsernamePasswordCredentials usernamePasswordCredentials = new UsernamePasswordCredentials(proxyUser, proxyPassword);
credsProvider.setCredentials(new AuthScope(proxyHost, proxyPort), usernamePasswordCredentials);
builder.useSystemProperties();
builder.setProxy(new HttpHost(proxyHost, proxyPort));
builder.setDefaultCredentialsProvider(credsProvider);
builder.setProxyAuthenticationStrategy(new ProxyAuthenticationStrategy());
where builder is:
HttpClientBuilder builder = HttpClientBuilder.create();
However, I get this exception when I execute this request:
java.lang.RuntimeException: org.apache.http.conn.UnsupportedSchemeException: http protocol is not supported
Caused by: org.apache.http.conn.UnsupportedSchemeException: http protocol is not supported
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:108) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:338) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:388) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.5.1.jar:4.5.1]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[httpclient-4.5.1.jar:4.5.1]
(exception shortened for brevity)
Since this is an HTTP proxy, I don't want to change the scheme to HTTPS, which anyways won't work. How do I get this working?
java.lang.RuntimeException:
org.apache.http.conn.UnsupportedSchemeException: http protocol is not
supported
Why this problem occurs?
Ans: This actually happens because you forget to register a connection socket factory for the 'http' scheme.
Plain 'http' scheme must be used to establish an intermediate connection
to the proxy itself before 'https' tunneling could be employed.
For operational purpose, you can try this code:
CloseableHttpClient client = HttpClients.custom()
.setRoutePlanner(new
SystemDefaultRoutePlanner(ProxySelector.getDefault()))
.build();
I would also suggest simple code for your research. Hope it can save you.
ClientExecuteProxy.java
package org.apache.http.examples.client;
import org.apache.http.HttpHost;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
/**
* How to send a request via proxy.
*
* #since 4.0
*/
public class ClientExecuteProxy {
public static void main(String[] args)throws Exception {
CloseableHttpClient httpclient = HttpClients.createDefault();
try {
HttpHost target = new HttpHost("httpbin.org", 443, "https");
HttpHost proxy = new HttpHost("127.0.0.1", 8080, "http");
RequestConfig config = RequestConfig.custom()
.setProxy(proxy)
.build();
HttpGet request = new HttpGet("/");
request.setConfig(config);
System.out.println("Executing request " + request.getRequestLine() + " to " + target + " via " + proxy);
CloseableHttpResponse response = httpclient.execute(target, request);
try {
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
System.out.println(EntityUtils.toString(response.getEntity()));
} finally {
response.close();
}
} finally {
httpclient.close();
}
}
}
Are you using using CloudantClient java API for Cloudant DB?
Ans:
If YES, then It turned out the issue with HTTP when setting a proxy was a bug at our end (sorry about that). We released 1.2.1 with the fix for this problem. You can download jar file from here. (Collected from mike-rhodes's answer)
UPDATE
How do I specify the credentials for the proxy here?
From HTTP authentication,
By default, httpclient will not provide credentials preemptively, it will first create a HTTP request without authentication parameters. This is by design, as a security precaution, and as part of the spec. But, this causes issues if you don't retry the connection, or wherever you're connecting to expects you to send authentication details on the first connection. It also causes extra latency to a request, as you need to make multiple calls, and causes 401s to appear in the logs.
The workaround is to use an authentication cache to pretend that you've already connected to the server once. This means you'll only make one HTTP call.
CloseableHttpClient httpclient = HttpClientBuilder.create().build();
HttpHost targetHost = new HttpHost("localhost", 80, "http");
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(
new AuthScope(targetHost.getHostName(), targetHost.getPort()),
new UsernamePasswordCredentials("username", "password"));
// Create AuthCache instance
AuthCache authCache = new BasicAuthCache();
// Generate BASIC scheme object and add it to the local auth cache
BasicScheme basicAuth = new BasicScheme();
authCache.put(targetHost, basicAuth);
// Add AuthCache to the execution context
HttpClientContext context = HttpClientContext.create();
context.setCredentialsProvider(credsProvider);
context.setAuthCache(authCache);
HttpGet httpget = new HttpGet("/");
for (int i = 0; i < 3; i++) {
CloseableHttpResponse response = httpclient.execute(
targetHost, httpget, context);
try {
HttpEntity entity = response.getEntity();
} finally {
response.close();
}
}
N.B: You need to trust the host you're connecting to, and if you're
using HTTP, your username and password will be sent in cleartext
(well, base64, but that doesn't count).
You should also be using a much more specific Authscope rather than
relying on AuthScope.ANY_HOST and AuthScope.ANY_PORT like in your
example.
Credit goes to Cetra
Related Links:
HttpClientBuilder basic auth
Apache HttpClient 4.1 - Proxy Authentication
What you have should be very close to working. I would make the following simple changes:
builder.useSystemProperties();
Delete the call to useSystemProperties. It isn't documented well, but when you set the Proxy (as you do in the next line), it overrides this, so just remove that line.
builder.setProxy(new HttpHost(proxyHost, proxyPort));
Call the HttpHost constructor with the explicit 'scheme' parameter. This is where you are getting the error, so make it explicit:
String proxyScheme = "http";
builder.setProxy(new HttpHost(proxyHost, proxyPort, proxyScheme));
Note: you did not say, but based on the usage of "BasicCredentialsProvider", this is only giving you "Basic" authentication. Basic is only encoded and is not really secure. For Digest or NTLM or Kerberos you will need different code.
I think the problem is with your HttpClient, not the proxy. Did you try to create your HttpClient by using HttpClientBuilder.build()
HttpClient client = builder.build();
ChallengeState.PROXY would provide proxy-authorization header.
However since v4.3, the code is deprecated. It still works in v4.5.
HttpHost proxyHost = this.getProxyHttpHost(config);
authCache.put(proxyHost, new BasicScheme(ChallengeState.PROXY));
Another way to have proxy-authorization header
credsProvider.setCredentials(new AuthScope("127.0.0.1","8080"),
new UsernamePasswordCredentials("username", "password"));
builder.setProxyAuthenticationStrategy(new ProxyAuthenticationStrategy());
builder.setDefaultCredentialsProvider(credsProvider);
Related
I'm trying to hit a REST API link using Apache HttpClient but I keep getting a 401 error returned. I can login when I go to the URL in browser, after being prompted for a password. The code I'm using is below:
CredentialsProvider provider = new BasicCredentialsProvider();
UsernamePasswordCredentials credentials = new UsernamePasswordCredentials(creds.get(0), creds.get(1));
provider.setCredentials(AuthScope.ANY, credentials);
AuthCache authCache = new BasicAuthCache();
authCache.put(new HttpHost(uri.getHost(), uri.getPort(), "https"), new BasicScheme());
BasicHttpContext context = new BasicHttpContext();
context.setAttribute(ClientContext.CREDS_PROVIDER, provider);
context.setAttribute(ClientContext.AUTH_CACHE, authCache);
DefaultHttpClient client = new DefaultHttpClient();
client.setHttpRequestRetryHandler(new DefaultHttpRequestRetryHandler());
client.setCredentialsProvider(provider);
HttpResponse response = null;
try
{
// response = client.execute(new HttpGet(uri));
response = client.execute(new HttpGet(uri), context);
}
catch(IOException e)
{
logger.error("Error running authenticated get request: " + e);
}
I'm using HttpClient 4.2.3 and unfortunately I'm not able to upgrade this.
Any help would be appreciated! Thanks!
EDIT: turns out I need to supply the certificate, like using -cacert in curl, however I can't find an example of this!
Since you need to provide a certificate maybe this can help:
http://hc.apache.org/httpcomponents-client-4.2.x/httpclient/examples/org/apache/http/examples/client/ClientCustomSSL.java
I think that example complies with 4.2.3 .
This is the exception that I'm having
Host name 'bla bla bla.com' does not match the certificate subject provided by the peer (CN=*.bla bla bla.com, OU=PositiveSSL Wildcard, OU=Domain Control Validated)
I already saw this question:
Ignoring SSL certificate in Apache HttpClient 4.3
and I did as it suggests, but it didn't work. I have seen many question related to the problem but they are all deprecated.
This is my code:
SSLContextBuilder builder = new SSLContextBuilder();
builder.loadTrustMaterial(null, new TrustSelfSignedStrategy());
SSLConnectionSocketFactory sslsf =
new SSLConnectionSocketFactory(builder.build());
CloseableHttpClient httpclient =
HttpClients.custom().setSSLSocketFactory(sslsf).build();
HttpGet httpGet = new HttpGet("https://b.blablabla.com");
CloseableHttpResponse response1 = httpclient.execute(httpGet);
try {
System.out.println(response1.getStatusLine());
HttpEntity entity1 = response1.getEntity();
// do something useful with the response body
// and ensure it is fully consumed
EntityUtils.consume(entity1);
} finally {
response1.close();
}
How can I bypass this certificate thing? This is just for testing; it is not a real production environment.
Beside localhost you can add own custom hostnames to your development-machine. Use the C:/windows/system32/etc/hosts to add the hostname anjadavid.blablabla.com in example.
Now open your browser and go to https://anjadavid.blablabla.com and the error disappears.
I am using Apache Http client 4.0.1 for communicating with the server. I already have a secure/non secure client code that works just fine.
Recently the new addition being to add proxy to this code, so i added the following piece of code to do that (currently non secure proxy),
HttpHost proxy = new HttpHost("localhost", 5555);
httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
This has worked fine with a non secure request. However i am having trouble with a secure (https) request with the same code.
Get the below exception (it tries a few time before failing),
Mar 12, 2014 11:14:27 AM org.apache.http.impl.client.DefaultRequestDirector tryConnect
INFO: I/O exception (org.apache.http.NoHttpResponseException) caught when connecting to the target host: The target server failed to respond
Mar 12, 2014 11:14:27 AM org.apache.http.impl.client.DefaultRequestDirector tryConnect
INFO: Retrying connect
org.apache.http.NoHttpResponseException: The target server failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:95)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.createTunnelToTarget(DefaultRequestDirector.java:899)
at org.apache.http.impl.client.DefaultRequestDirector.establishRoute(DefaultRequestDirector.java:818)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:644)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at com.poc.test.SSLTest.main(SSLTest.java:88)
Tried following things,
For https requests, i added both "http" as well as "https" to the schema registry, using the same SSLFactory as the one used for "https".
Changed the proxy to,
HttpHost proxy = new HttpHost("localhost", 5555, "https");
However in both cases it failed with,
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:397)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:572)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at com.poc.test.SSLTest.main(SSLTest.java:89)
Note - I am running a non secure proxy on my localhost via tcpmon.
EDIT: Here is the code i am using for the SSL with proxy communication,
DefaultHttpClient httpClient = new DefaultHttpClient();
try {
SSLContext ctx = SSLContext.getInstance("TLSv1.1");
TrustManager[] trustManagers = getTrustManagers("jks", new FileInputStream(new File("C:\\SSLKeyStore.ks")), "changeit");
ctx.init(null, trustManagers, new SecureRandom());
HttpGet httpget = new HttpGet("https://localhost:8844/Channels/HTTP/getData");
System.out.println("executing request" + httpget.getRequestLine());
SSLSocketFactory factory = new SSLSocketFactory(ctx);
factory.setHostnameVerifier(SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
ClientConnectionManager manager = httpClient.getConnectionManager();
manager.getSchemeRegistry().register(new Scheme("https", 443, factory));
manager.getSchemeRegistry().register(new Scheme("http", 80, PlainSocketFactory.getSocketFactory()));
HttpHost proxy = new HttpHost("localhost", 5555, "http");
httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
HttpResponse response = httpClient.execute(httpget);
HttpEntity entity = response.getEntity();
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
if (entity != null) {
System.out.println("Response content length: " + entity.getContentLength());
}
EntityUtils.consume(entity);
} catch (Exception exception) {
exception.printStackTrace();
} finally {
httpClient.getConnectionManager().shutdown();
}
Any ideas of what is happening, what am i missing with respect to https and proxy.
Latest EDIT - Even tried their example code (ClientExecuteProxy.java) as is, that too failed with proxy. Is this functionality broken?
HttpHost proxy = new HttpHost("127.0.0.1", 8080, "http");
DefaultHttpClient httpclient = new DefaultHttpClient();
try {
httpclient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
HttpHost target = new HttpHost("issues.apache.org", 443, "https");
HttpGet req = new HttpGet("/");
System.out.println("executing request to " + target + " via " + proxy);
HttpResponse rsp = httpclient.execute(target, req);
HttpEntity entity = rsp.getEntity();
System.out.println("----------------------------------------");
System.out.println(rsp.getStatusLine());
Header[] headers = rsp.getAllHeaders();
for (int i = 0; i<headers.length; i++) {
System.out.println(headers[i]);
}
System.out.println("----------------------------------------");
if (entity != null) {
System.out.println(EntityUtils.toString(entity));
}
} finally {
// When HttpClient instance is no longer needed,
// shut down the connection manager to ensure
// immediate deallocation of all system resources
httpclient.getConnectionManager().shutdown();
}
}
Thanks,
Vicky
Off the top of my head I'd say that you are dealing with a certificate trust issue.
Without seeing how you are setting up the connection specifically I can say with no certainty though.
The "peer not authenticated" means that somewhere along the line the verification of the certificates as presented by one or more of the servers cannot be verified.
EDIT
Since the proxy is under your control you have a load of flexibility at this time.
Please see this SO article, it may suit your needs.
I'm trying to send a GET via a proxy and some sites have the header: Content-Encoding: none, which causes Apache to throw an exception. I'm wondering if this is the intended behavior, and whether I should treat this as a bug or not:
Caused by: org.apache.http.HttpException: Unsupported Content-Coding: none
at org.apache.http.client.protocol.ResponseContentEncoding.process(ResponseContentEncoding.java:98)
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:139)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:199)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
... 10 more
My code:
public CloseableHttpResponse getViaProxy(String url, String ip, int port, String username,
String password) {
CloseableHttpClient httpClient;
if (username == null) {
httpClient = HttpClients.custom()
.setSSLSocketFactory(getCustomSslConnectionSocketFactory())
.build();
} else {
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(
new AuthScope(ip, port),
new UsernamePasswordCredentials(username, password));
httpClient = HttpClients.custom()
.setDefaultCredentialsProvider(credsProvider)
.setSSLSocketFactory(getCustomSslConnectionSocketFactory())
.build();
}
RequestConfig config = RequestConfig.custom()
.setProxy(new HttpHost(ip, port))
.build();
HttpGet httpGet = new HttpGet(url);
httpGet.setConfig(config);
return httpClient.execute(httpGet);
}
The error I'm referring to is from here. It seems like this method only supports headers with Content-Encoding: gzip, deflate, or identity.
http://hc.apache.org/httpcomponents-client-ga/httpclient/xref/org/apache/http/client/protocol/ResponseContentEncoding.html
HTTP protocol specification defines gzip, compress, deflate and identity as valid content coding schemes. One should be using identity instead to signal that no content transformation is expected.
If you only want to process the request with the Content-Encoding is none, you can use the custom method of HttpClent like so:
CloseableHttpClient httpClient = HttpClients.custom().disableContentCompression().build();
this code disable the two interceptors RequestAcceptEncoding and ResponseContentEncoding.
There is a possibility to fix that issue on client-side with interceptors: http://adamscheller.com/java/httpexception-unsupported-content-coding-none-solution/
I am using jersey client with ApacheConnection Provider.
Builder builder = RequestConfig.custom().setConnectTimeout(timeout);
List<Proxy> proxies = ProxyManager.getInstance().select(baseUrl.toURI());
if (useProxy) {
...
builder.setProxy(new HttpHost(proxyUri.getHost(), proxyUri.getPort()));
}
RequestConfig requestConfig = builder.build();
final ClientConfig clientConfig = new ClientConfig();
clientConfig.property(ApacheClientProperties.REQUEST_CONFIG, requestConfig);
clientConfig.connectorProvider(new ApacheConnectorProvider());
client = ClientBuilder.newBuilder().withConfig(clientConfig).sslContext(getSSLContext()).build();
client.property(ClientProperties.CONNECT_TIMEOUT, 5000);
But how to add username and password for Proxy authentication?
Seems like apache connection provider does not use the standard java proxy selector mechanisms.
I finally found the solution by myself. Unfortunately this is documented nowhere:
HttpHost proxyhost = new HttpHost(host,pw);
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(new AuthScope(proxyhost), new UsernamePasswordCredentials(user, pw));
clientConfig.property(ApacheClientProperties.CREDENTIALS_PROVIDER, credsProvider);
builder.setProxy(proxyhost);
I think you should add few more lines of code
builder.setProxy(proxyhost).setDefaultCredentialsProvider(credsProvider)
.setProxyAuthenticationStrategy(new ProxyAuthenticationStrategy());
otherwise it wont really authenticate the proxy host I feel. In your case it might be bypassing the proxy. ?