We Keep Coding

Http call in java not sending client certificate

I'm using the Apache HTTP client (version 4.5.13) in Java 8 to perform a POST call that requires the client to authenticate using a certificate certificate, that I have stored in a .PFX file.
This is the code I'm using:
public static void performClientRequest() throws Exception {
//Trust Strategy to accept any server certificate
TrustStrategy trustStrategy = new TrustStrategy() {
public boolean isTrusted(X509Certificate[] chain, String authType)
throws CertificateException {
return true;
}
};
//Load PFX client certificate
KeyStore clientStore = KeyStore.getInstance("PKCS12");
InputStream instream = new FileInputStream("C:\\client.pfx");
try {
clientStore.load(instream, null);
} finally {
instream.close();
}
//Create ssl context with key store and trust strategy
SSLContext sslContext = SSLContexts.custom()
.loadKeyMaterial(clientStore, null)
.loadTrustMaterial(trustStrategy)
.build();
//Create ssl socket factory from context
SSLConnectionSocketFactory sslSocketFactory = new SSLConnectionSocketFactory(sslContext, NoopHostnameVerifier.INSTANCE);
//Create HTTP client
HttpClient httpClient = HttpClients.custom()
.setSSLSocketFactory(sslSocketFactory)
.build();
//Perform call
URI url = new URI("https://mysite.foo");
HttpPost request = new HttpPost(url);
request.setHeader("Content-Type","application/json");
request.setHeader("Accept", "application/json");
String body = "...";
StringEntity bodyEntity = new StringEntity(body);
request.setEntity(bodyEntity);
HttpResponse response = httpClient.execute(request);
HttpEntity entity = response.getEntity();
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
EntityUtils.consume(entity);
}
I've used this code in in the past and it worked perfectly back then, but now I'm trying to reuse it and it just doesn't send the certificate, the server replies with:
HTTP/1.1 403 No client certificate supplied
How can I debug this and discover why the certificate is not being sent?
Note: I implemented a similar call both in C# and using Postman, and in both cases it works perfectly, so the client certificate authentication to the server is working, it's just not working in my Java implementation.

So, I don't know if this is a bug or intended behavior (if so, why?), but apparently the PFX file must be password-protected, then it gets sent correctly. I could not make this work with a non-protected PFX file and passing null as the password like I was doing in the code I posted in the question.
So the problem is solved, but I would be curious if anyone could comment on WHY this happens.

Malformed reply from SOCKS server whereas I use a HTTP proxy (with Apache HTTP library)

I know that there are a lot of questions regarding the error Malformed reply from SOCKS server which mostly point to a wrong configuration for the proxy.
However, in my case, I'm using the system HTTP(!) proxy for a POST request with the apache httpclient library 4.3.5 (httpcore 4.3.2) like this:
SystemDefaultRoutePlanner routePlanner = new SystemDefaultRoutePlanner(
ProxySelector.getDefault());
CloseableHttpClient httpclient = HttpClients.custom()
.setRoutePlanner(routePlanner)
.build();
This is what has also been advised in https://stackoverflow.com/a/20743545/1128689
In most situations this httpclient and the following http POST request work very well. At one customer however, it fails with the following log error:
Mrz 05, 2015 10:11:04 AM org.apache.http.impl.execchain.RetryExec execute
Information: I/O exception (java.net.SocketException) caught when processing request to {}->http://proxy.local:80->http://my-webservice.tld:80: Malformed reply from SOCKS server
Mrz 05, 2015 10:11:04 AM org.apache.http.impl.execchain.RetryExec execute
Information: Retrying request to {}->http://proxy.local:80->http://my-webservice.tld:80
The local system proxy settings (on Windows 7) aren't configured as a SOCKS proxy, but as an HTTP proxy! I confirmed this by logging different RoutePlanner and Proxy parameters:
TunnelType: PLAIN
TargetHost: http://my-webservice.tld:80
ProxyHost: http://proxy.local:80
ProxyPort: 80
ProxyType: HTTP
However, my POST request isn't beeing sent correctly. In the proxy logfiles it appears as follows:
2015-03-05 10:11:04 119973 192.168.124.111 0 TCP_ERR_MISS 0 4 unknown - - / - - - - - - PROXIED - - xxx.xxx.xxx.xxx SG-HTTP-Service
This is my uploadFile() method which uses the mentioned SystemDefaultRoutePlanner and creates and executes the HttpPost:
private String uploadFile(File fileToUpload) throws Exception {
SystemDefaultRoutePlanner routePlanner = new SystemDefaultRoutePlanner(ProxySelector.getDefault());
CloseableHttpClient httpclient = HttpClients.custom()
.setRoutePlanner(routePlanner)
.build();
try {
HttpPost httppost = new HttpPost(webserviceURL);
MultipartEntityBuilder requestEntity = MultipartEntityBuilder.create();
requestEntity.addPart("project", new StringBody(paramProjekt, ContentType.TEXT_PLAIN));
requestEntity.addPart("param1", new StringBody(param1, ContentType.TEXT_PLAIN));
requestEntity.addPart("param2", new StringBody(param2, ContentType.TEXT_PLAIN));
requestEntity.addPart("debug", new StringBody(paramDebug, ContentType.TEXT_PLAIN));
requestEntity.addPart("xmlFile", new FileBody(fileToUpload));
ProgressHttpEntityWrapper.ProgressCallback progressCallback = new ProgressHttpEntityWrapper.ProgressCallback() {
#Override
public void progress(float progress) {
int min = 5;
int max = 40;
int diff = max - min;
gui.updateProgress(min + (int)(diff * progress / 100));
if (progress == 100) {
gui.setStep(GUI.STEP.PROCESSING);
}
}
};
httppost.setEntity(new ProgressHttpEntityWrapper(requestEntity.build(), progressCallback));
logger.info("executing request " + httppost.getRequestLine());
CloseableHttpResponse response = httpclient.execute(httppost);
try {
logger.info("Response Status: " + response.getStatusLine());
HttpEntity resEntity = response.getEntity();
if (resEntity != null) {
String responseString = EntityUtils.toString(resEntity);
return responseString;
}
EntityUtils.consume(resEntity);
} catch (Exception e) {
e.printStackTrace();
return null;
} finally {
response.close();
}
} finally {
httpclient.close();
}
return null;
}
The last thing in my logfile is the executing request log info but the Response Status: log information never shows up...
Mrz 05, 2015 10:09:04 AM de.company.product.WebserviceClient.WebserviceClient uploadFile
Information: executing request POST http://my-webservice.tld HTTP/1.1
Mrz 05, 2015 10:11:04 AM org.apache.http.impl.execchain.RetryExec execute
Information: I/O exception (java.net.SocketException) caught when processing request to {}->http://proxy.local:80->http://my-webservice.tld:80: Malformed reply from SOCKS server
Mrz 05, 2015 10:11:04 AM org.apache.http.impl.execchain.RetryExec execute
Information: Retrying request to {}->http://proxy.local:80->http://my-webservice.tld:80
Does anyone has a clue why
it is showing the Malformed reply from SOCKS server error when there is a HTTP proxy defined in the system and I use the system proxy settings?
how to configure either the httpclient or the httppost in the right way to recognize system HTTP proxy settings and work well with and without HTTP proxy?

Apache Http Client 4.0.1 SSL Proxy

I am using Apache Http client 4.0.1 for communicating with the server. I already have a secure/non secure client code that works just fine.
Recently the new addition being to add proxy to this code, so i added the following piece of code to do that (currently non secure proxy),
HttpHost proxy = new HttpHost("localhost", 5555);
httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
This has worked fine with a non secure request. However i am having trouble with a secure (https) request with the same code.
Get the below exception (it tries a few time before failing),
Mar 12, 2014 11:14:27 AM org.apache.http.impl.client.DefaultRequestDirector tryConnect
INFO: I/O exception (org.apache.http.NoHttpResponseException) caught when connecting to the target host: The target server failed to respond
Mar 12, 2014 11:14:27 AM org.apache.http.impl.client.DefaultRequestDirector tryConnect
INFO: Retrying connect
org.apache.http.NoHttpResponseException: The target server failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:95)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.createTunnelToTarget(DefaultRequestDirector.java:899)
at org.apache.http.impl.client.DefaultRequestDirector.establishRoute(DefaultRequestDirector.java:818)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:644)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at com.poc.test.SSLTest.main(SSLTest.java:88)
Tried following things,
For https requests, i added both "http" as well as "https" to the schema registry, using the same SSLFactory as the one used for "https".
Changed the proxy to,
HttpHost proxy = new HttpHost("localhost", 5555, "https");
However in both cases it failed with,
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:397)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:572)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at com.poc.test.SSLTest.main(SSLTest.java:89)
Note - I am running a non secure proxy on my localhost via tcpmon.
EDIT: Here is the code i am using for the SSL with proxy communication,
DefaultHttpClient httpClient = new DefaultHttpClient();
try {
SSLContext ctx = SSLContext.getInstance("TLSv1.1");
TrustManager[] trustManagers = getTrustManagers("jks", new FileInputStream(new File("C:\\SSLKeyStore.ks")), "changeit");
ctx.init(null, trustManagers, new SecureRandom());
HttpGet httpget = new HttpGet("https://localhost:8844/Channels/HTTP/getData");
System.out.println("executing request" + httpget.getRequestLine());
SSLSocketFactory factory = new SSLSocketFactory(ctx);
factory.setHostnameVerifier(SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
ClientConnectionManager manager = httpClient.getConnectionManager();
manager.getSchemeRegistry().register(new Scheme("https", 443, factory));
manager.getSchemeRegistry().register(new Scheme("http", 80, PlainSocketFactory.getSocketFactory()));
HttpHost proxy = new HttpHost("localhost", 5555, "http");
httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
HttpResponse response = httpClient.execute(httpget);
HttpEntity entity = response.getEntity();
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
if (entity != null) {
System.out.println("Response content length: " + entity.getContentLength());
}
EntityUtils.consume(entity);
} catch (Exception exception) {
exception.printStackTrace();
} finally {
httpClient.getConnectionManager().shutdown();
}
Any ideas of what is happening, what am i missing with respect to https and proxy.
Latest EDIT - Even tried their example code (ClientExecuteProxy.java) as is, that too failed with proxy. Is this functionality broken?
HttpHost proxy = new HttpHost("127.0.0.1", 8080, "http");
DefaultHttpClient httpclient = new DefaultHttpClient();
try {
httpclient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
HttpHost target = new HttpHost("issues.apache.org", 443, "https");
HttpGet req = new HttpGet("/");
System.out.println("executing request to " + target + " via " + proxy);
HttpResponse rsp = httpclient.execute(target, req);
HttpEntity entity = rsp.getEntity();
System.out.println("----------------------------------------");
System.out.println(rsp.getStatusLine());
Header[] headers = rsp.getAllHeaders();
for (int i = 0; i<headers.length; i++) {
System.out.println(headers[i]);
}
System.out.println("----------------------------------------");
if (entity != null) {
System.out.println(EntityUtils.toString(entity));
}
} finally {
// When HttpClient instance is no longer needed,
// shut down the connection manager to ensure
// immediate deallocation of all system resources
httpclient.getConnectionManager().shutdown();
}
}
Thanks,
Vicky

Off the top of my head I'd say that you are dealing with a certificate trust issue.
Without seeing how you are setting up the connection specifically I can say with no certainty though.
The "peer not authenticated" means that somewhere along the line the verification of the certificates as presented by one or more of the servers cannot be verified.
EDIT
Since the proxy is under your control you have a load of flexibility at this time.
Please see this SO article, it may suit your needs.

Apache HttpAsyncClient

I am testing the Apache HttpAsyncClient, in particular I want to make an asynchronous HTTP POST Request where authentication is needed. I use this example as reference. So far I found out how to set Application type and body but can't find out how to set the credentials.
I try to add Authentication credentials with
HttpAsyncClientBuilder create = HttpAsyncClientBuilder.create();
create.setTargetAuthenticationStrategy(new TargetAuthenticationStrategy());
BasicCredentialsProvider basicCredentialsProvider = new BasicCredentialsProvider();
Credentials defaultcreds = new UsernamePasswordCredentials("user", "password");
basicCredentialsProvider.setCredentials(new AuthScope("http://localhost", 7351), defaultcreds);
create.setDefaultCredentialsProvider(basicCredentialsProvider);
final CloseableHttpAsyncClient httpclient = create.build();
httpclient.start();
...
But I always get
Sep 11, 2013 4:21:35 PM org.apache.http.impl.auth.HttpAuthenticator handleAuthChallenge
WARNING: Malformed challenge: Authentication challenge is empty
I have not found an example which explains how to set authentication data for the CloseableHttpAsyncClient. Anyone can help me out?

You can set a credentials provider either at the client level if you want it to be shared by all requests by default
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
CloseableHttpAsyncClient httpclient = HttpAsyncClients.custom()
.setDefaultCredentialsProvider(credentialsProvider)
.build();
httpclient.start();
try {
HttpGet request = new HttpGet("http://www.apache.org/");
Future<HttpResponse> future = httpclient.execute(request, null);
HttpResponse response = future.get();
System.out.println("Response: " + response.getStatusLine());
System.out.println("Shutting down");
} finally {
httpclient.close();
}
System.out.println("Done");
or set it at the request level, if you want it to apply to a particular request only
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
CloseableHttpAsyncClient httpclient = HttpAsyncClients.createDefault();
httpclient.start();
try {
HttpGet request = new HttpGet("http://www.apache.org/");
HttpClientContext context = HttpClientContext.create();
context.setCredentialsProvider(credentialsProvider);
Future<HttpResponse> future = httpclient.execute(request, context, null);
HttpResponse response = future.get();
System.out.println("Response: " + response.getStatusLine());
System.out.println("Shutting down");
} finally {
httpclient.close();
}
System.out.println("Done");
Please also note that Malformed challenge: Authentication challenge is empty warning is likely caused by the server sending a malformed (empty) auth challenge rather than HttpClient configuration. Providing user credentials for the request may not necessarily resolve the issue.

Apache httpComponents HttpClient 4.2.2 unable connect to a web page via Webscarab proxy

I have a method which uses the Apache httpComponents HttpClient class to connect to a web page via the Webscarab proxy. I get the method from the Apache Software Foundation here. below is my hole method:
public void HTTPGet(){
//HttpHost proxy = new HttpHost("localhost", 8008);
System.setProperty("http.proxyHost", "localhost");
System.setProperty("http.proxyPort", "8008");
HttpHost proxy = new HttpHost(System.getProperty("http.proxyHost", "localhost"),Integer.parseInt(System.getProperty("http.proxyPort", "8008")));
DefaultHttpClient httpclient = new DefaultHttpClient();
try {
httpclient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY,
proxy);
HttpHost target = new HttpHost("www.google.gr/", 80);
HttpGet req = new HttpGet("/");
System.out.println("executing request to " + target + " via "
+ proxy);
HttpResponse rsp = httpclient.execute(target, req); //here I get the exception as below
HttpEntity entity = rsp.getEntity();
System.out.println("----------------------------------------");
System.out.println(rsp.getStatusLine());
Header[] headers = rsp.getAllHeaders();
for (int i = 0; i < headers.length; i++) {
System.out.println(headers[i]);
}
System.out.println("----------------------------------------");
if (entity != null) {
System.out.println(EntityUtils.toString(entity));
}
} catch (IOException ex) {
Logger.getLogger(WebBrowser.class.getName()).log(Level.SEVERE, null, ex);
}
finally {
// When HttpClient instance is no longer needed,
// shut down the connection manager to ensure
// immediate deallocation of all system resources
httpclient.getConnectionManager().shutdown();
}
}
The println before the HttpResponse rsp = httpclient.execute(target, req);,
return this : executing request to http://www.google.gr/:80 via http://localhost:8008
Then the below exception occurs.
Nov 28, 2012 1:23:09 AM student.WebBrowser HTTPGet
SEVERE: null
org.apache.http.conn.HttpHostConnectException: Connection to http://localhost:8008 refused
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:827)
Webscarab interfere correctly when I try to reach a page from another browser, like Firefox, Chrome or JavaFX webengine.
Thanks in advance.

Since the connection is being refused, I can only assume that WebScarab is not listening on the port you are trying to connect to (at the time that you are trying to connect).
e.g. WebScarab is not running, or you have configured it to listen on a different port than the default 8008, or on a different interface to localhost (127.0.0.1).
Is your code running on the same machine as WebScarab? If not, you need to make WebScarab listen to the actual network interface that it is reachable on over the network, and update your code to reference that IP address instead of localhost.