We Keep Coding

CloseableHttpClient and How to use it from connection pool?

I have a creating a connectionManager for the pool as this -->>
public static PoolingHttpClientConnectionManager getCm() {
return cm;
}
public static void setCm(PoolingHttpClientConnectionManager cm) {
ClassName.cm = cm;
}
public static PoolingHttpClientConnectionManager createPool()
{
System.out.println("Generating new connection pool");
setCm( new PoolingHttpClientConnectionManager());
getCm().setMaxTotal(10);
getCm().setDefaultMaxPerRoute(5);
return getCm();
}
I am creating an instance of CloseableHttpClient using this -->>
public static CloseableHttpClient createConnection(){
if(getCm()!=null)
return HttpClients.custom().setConnectionManager(getCm()).setConnectionManagerShared(true).setDefaultRequestConfig(getConfig()).build();
else
return HttpClients.custom().setConnectionManager(createPool()).setConnectionManagerShared(true).setDefaultRequestConfig(getConfig()).build();
}
But here each time I am creating an instance of CloseableHttpClient using the PoolingHttpClientConnectionManager. I am confused if this is the correct way or not?
In the Class where I make a HTTP call, I am doing it in this way--->>
private static CloseableHttpClient client; //class level variable. static or final? Which one to prefer? I want this to handle more than 50 concurrent request
Inside My method--->>
client = createConnection();
String endPoint = //someendpoint
HttpPost httpPost = new HttpPost(endPoint);
httpPost.setHeader("Content-type", "application/json");
httpPost.setHeader("Accept-API-Version", "resource=2.0, protocol=1.0");
CloseableHttpResponse response1 = null;
try{
response1 = client.execute(httpPost);
String responseString;
int statusCode;
responseString = EntityUtils.toString(response1.getEntity());
//doing something useful
}catch(){
} finally(){ //Another big question comes here. How to close and what should I close?
/Just to be on the safe side, I am closing everything
httpPost.releaseConnection();
if(response1!= null)
response1.close();
if (client != null)
client.close();
}
Please suggest the best way or the alternative? Also I am a newbie and doing this to learn so forgive and correct me If I made any mistake.

Android: NTLM Authentication, ksoap, and persistent connections

After working with iOS and dealing with auth challenges without much of a learning curve, I've found that Windows Authentication is much more complicated of a process in Java/Android.
I tried multiple different approaches, so without getting too much into those, I will get to the one that worked for the most part. I'm now using the class created for NTLM and ksoap called NtlmTransport
I'm now successfully authenticating in the following way:
NtlmTransport httpTransport = new NtlmTransport();
httpTransport.setCredentials(serverURL, Login.username, Login.password, deviceIp, "DOMAINNAME");
httpTransport.call(SOAP_ACTION, envelope);
If you take a look at the NtlmTransport class, you'll see that it's returning the following headers from the setupNtlm():
status Line HTTP/1.1 200 OK
Setup Cache-Control:private, max-age=0
Setup Content-Type:text/html; charset=utf-8
Setup Server:Microsoft-IIS/8.0
Setup X-AspNet-Version:4.0.30319
Setup Persistent-Auth:true
Setup X-Powered-By:ASP.NET
Setup Date:Tue, 17 Sep 2013 20:57:45 GMT
Setup Content-Length:11549
The "Persistent-Auth:true is the main one I'm concerned about at this time. I'm getting the SoapObjects just fine and can get the data I need from that one connection, but as soon as I try to access the web service again, which is presumably able to be hit after the successful authentication, I can't access a different method using HttpTransportSE:
private void setSomething() {
xml = null;
final String SOAP_ACTION = "http://this.ismy.org/AWebServiceMethod";
final String METHOD_NAME = "AWebServiceMethod";
final String URL = protocol + "://" + host + ":" + port + "/WebService.asmx";
SoapObject request = new SoapObject(NAMESPACE, METHOD_NAME);
SoapSerializationEnvelope envelope = new SoapSerializationEnvelope(SoapEnvelope.VER11);
envelope.dotNet = true;
envelope.setOutputSoapObject(request);
envelope.implicitTypes = true;
envelope.setAddAdornments(false);
try
{
HttpTransportSE transport = new HttpTransportSE(URL);
transport.debug = true;
transport.call(SOAP_ACTION, envelope);
xml = transport.responseDump.toString();
Log.d(TAG, xml);
}
catch(SocketException ex)
{
Log.e("SocketException : " , "Error on setSomething() " + ex.getMessage());
}
catch (Exception e)
{
Log.e("Exception : " , "Error on setSomething() " + e.getMessage());
}
}
This all works just fine as the background task of an AsyncTask, which then passes the "xml" to an XMLPullParser method.
The main question here is why am I getting a:
Error on setSomething() No authentication challenges found
??
After IIS successfully validates the user with a 200, why is it asking me to authenticate again? How can I persist that first authenticated challenge to hit whatever method I want inside WebService.asmx? What are the headers that need to be added/changed to create a session if necessary? What am I missing that makes this whole NTLM process work and persist for more than the WS method that needs to pass the authentication challenges?
EDIT : Adding the Library code
Here's the link to the JCIFS from Apache
public static final class JCIFSEngine implements NTLMEngine {
private static final int TYPE_1_FLAGS =
NtlmFlags.NTLMSSP_NEGOTIATE_56 |
NtlmFlags.NTLMSSP_NEGOTIATE_128 |
NtlmFlags.NTLMSSP_NEGOTIATE_NTLM2 |
NtlmFlags.NTLMSSP_NEGOTIATE_ALWAYS_SIGN |
NtlmFlags.NTLMSSP_REQUEST_TARGET;
public String generateType1Msg(final String domain, final String workstation)
throws NTLMEngineException {
final Type1Message type1Message = new Type1Message(TYPE_1_FLAGS, domain, workstation);
return jcifs.util.Base64.encode(type1Message.toByteArray());
}
public String generateType3Msg(final String username, final String password,
final String domain, final String workstation, final String challenge)
throws NTLMEngineException {
Type2Message type2Message;
try {
type2Message = new Type2Message(jcifs.util.Base64.decode(challenge));
} catch (final IOException exception) {
throw new NTLMEngineException("Invalid NTLM type 2 message", exception);
}
final int type2Flags = type2Message.getFlags();
final int type3Flags = type2Flags
& (0xffffffff ^ (NtlmFlags.NTLMSSP_TARGET_TYPE_DOMAIN | NtlmFlags.NTLMSSP_TARGET_TYPE_SERVER));
final Type3Message type3Message = new Type3Message(type2Message, Login.password, "",
Login.username, deviceIp, type3Flags);
System.out.println("type3Message: " + type3Message.toByteArray());
return jcifs.util.Base64.encode(type3Message.toByteArray());
}
}
So is the "NtlmFlags.NTLMSSP_NEGOTIATE_ALWAYS_SIGN" causing this problem? Is there another flag I'm supposed to set for the keep-alive? Also, I found a great resource for a list of NTLM flags and more: http://fossies.org/dox/jcifs-1.3.17/interfacejcifs_1_1ntlmssp_1_1NtlmFlags.html

I was also struggling about windows authentication from Android.
I found android-ntlm-master on https://github.com/masconsult/android-ntlm. Add this class as library in your project.
Change is in NtlmTransport.java class.I made change in call method of NtlmTransport class =>
public List call(String soapAction, SoapEnvelope envelope,
List headers, File outputFile)
throws IOException, XmlPullParserException {
HttpResponse resp = null;
try {
//setupNtlm(urlString, user, password);
DefaultHttpClient httpclient = new DefaultHttpClient();
httpclient.getAuthSchemes().register("ntlm", new NTLMSchemeFactory());
httpclient.getCredentialsProvider().setCredentials(
new AuthScope(AuthScope.ANY_HOST, AuthScope.ANY_PORT),
new NTCredentials(user, password, "", "")
);
HttpPost httpget = new HttpPost(urlString);
httpget.addHeader("soapaction", soapAction);
httpget.addHeader("Content-Type", "text/xml; charset=utf-8");
byte[] requestData = null;
try {
requestData = createRequestData(envelope);
} catch (IOException iOException) {
}
ByteArrayEntity byteArrayEntity = new ByteArrayEntity(requestData);
httpget.setEntity(byteArrayEntity);
resp = httpclient.execute(httpget);
if(resp == null) {
System.out.println("Response is null");
}
HttpEntity respEntity = resp.getEntity();
InputStream is = respEntity.getContent();
if(is == null) {
System.out.println("InputStream is null");
}
parseResponse(envelope, is);
} catch (Exception ex) {
// ex.printStackTrace();
}
if (resp != null) {
return Arrays.asList(resp.getAllHeaders());
} else {
return null;
}
}
And below is the code how I make call:
SoapObject request = new SoapObject(NAMESPACE, PRODUCT_DETAILS_METHOD_NAME);
request.addProperty("ListingID", Integer.parseInt(Product_ID));
NtlmTransport httpTransport = new NtlmTransport();
httpTransport.setCredentials(URL, USERNAME, PASSWORD, "","");
SoapSerializationEnvelope envelope = new SoapSerializationEnvelope(SoapEnvelope.VER11);
envelope.dotNet = true;
envelope.implicitTypes = true;
envelope.setOutputSoapObject(request);
httpTransport.call(PRODUCT_DETAILS_SOAP_ACTION, envelope);
SoapObject response = (SoapObject) envelope.getResponse();
It worked for me.
More you can find here: https://suhas1989.wordpress.com/2015/01/28/ntlm-authentication-in-android/

How to use a custom socketfactory in Apache HttpComponents

I have been trying to use a custom SocketFactory in the httpclient library from the Apache HTTPComponents project. So far without luck. I was expecting that I could just set a socket factory for a HttpClient instance, but it is obviously not so easy.
The documentation for HttpComponents at http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html does mention socket factories, but does not say how to use them.
Does anybody know how this is done?

oleg's answer is of course correct, I just wanted to put the information directly here, in case the link goes bad. In the code that creates a HttpClient, I use this code to let it use my socket factory:
CustomSocketFactory socketFactory = new CustomSocketFactory();
Scheme scheme = new Scheme("http", 80, socketFactory);
httpclient.getConnectionManager().getSchemeRegistry().register(scheme);
CustomSocketFactory is my own socket factory, and I want to use it for normal HTTP traffic, that's why I use "http" and 80 as parameters.
My CustomSchemeSocketFactory looks similar to this:
public class CustomSchemeSocketFactory implements SchemeSocketFactory {
#Override
public Socket connectSocket( Socket socket, InetSocketAddress remoteAddress, InetSocketAddress localAddress, HttpParams params ) throws IOException, UnknownHostException, ConnectTimeoutException {
if (localAddress != null) {
socket.setReuseAddress(HttpConnectionParams.getSoReuseaddr(params));
socket.bind(localAddress);
}
int connTimeout = HttpConnectionParams.getConnectionTimeout(params);
int soTimeout = HttpConnectionParams.getSoTimeout(params);
try {
socket.setSoTimeout(soTimeout);
socket.connect(remoteAddress, connTimeout );
} catch (SocketTimeoutException ex) {
throw new ConnectTimeoutException("Connect to " + remoteAddress + " timed out");
}
return socket;
}
#Override
public Socket createSocket( HttpParams params ) throws IOException {
// create my own socket and return it
}
#Override
public boolean isSecure( Socket socket ) throws IllegalArgumentException {
return false;
}
}

We use a custom socket factory to allow HttpClient connections to connect to HTTPS URLs with untrusted certificates.
Here is how we did it:
We adapted implementations of both the 'EasySSLProtocolSocketFactory' and 'EasyX509TrustManager' classes from the examples source directory referenced by Oleg.
In our HttpClient startup code, we do the following to enable the new socket factory:
HttpClient httpClient = new HttpClient();
Protocol easyhttps = new Protocol("https", new EasySSLProtocolSocketFactory(), 443);
Protocol.registerProtocol("https", easyhttps);
So that any time we request an https:// URL, this socket factory is used.

Java SSLException: hostname in certificate didn't match

I have been using the following code to connect to one of google's service. This code worked fine on my local machine :
HttpClient client=new DefaultHttpClient();
HttpPost post = new HttpPost("https://www.google.com/accounts/ClientLogin");
post.setEntity(new UrlEncodedFormEntity(myData));
HttpResponse response = client.execute(post);
I put this code in a production environment, which had blocked Google.com. On request, they allowed communication with Google server by allowing me to accessing an IP : 74.125.236.52 - which is one of Google's IPs. I edited my hosts file to add this entry too.
Still I could not access the URL, which I wonder why. So I replaced the above code with :
HttpPost post = new HttpPost("https://74.125.236.52/accounts/ClientLogin");
Now I get an error like this :
javax.net.ssl.SSLException: hostname in certificate didn't match:
<74.125.236.52> != <www.google.com>
I guess this is because Google has multiple IPs. I cant ask the network admin to allow me access to all those IPs - I may not even get this entire list.
What should I do now ? Is there a workaround at Java level ? Or is it totally in hands of the network guy ?

You can also try to set a HostnameVerifier as described here. This worked for me to avoid this error.
// Do not do this in production!!!
HostnameVerifier hostnameVerifier = org.apache.http.conn.ssl.SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER;
DefaultHttpClient client = new DefaultHttpClient();
SchemeRegistry registry = new SchemeRegistry();
SSLSocketFactory socketFactory = SSLSocketFactory.getSocketFactory();
socketFactory.setHostnameVerifier((X509HostnameVerifier) hostnameVerifier);
registry.register(new Scheme("https", socketFactory, 443));
SingleClientConnManager mgr = new SingleClientConnManager(client.getParams(), registry);
DefaultHttpClient httpClient = new DefaultHttpClient(mgr, client.getParams());
// Set verifier
HttpsURLConnection.setDefaultHostnameVerifier(hostnameVerifier);
// Example send http request
final String url = "https://encrypted.google.com/";
HttpPost httpPost = new HttpPost(url);
HttpResponse response = httpClient.execute(httpPost);

The certificate verification process will always verify the DNS name of the certificate presented by the server, with the hostname of the server in the URL used by the client.
The following code
HttpPost post = new HttpPost("https://74.125.236.52/accounts/ClientLogin");
will result in the certificate verification process verifying whether the common name of the certificate issued by the server, i.e. www.google.com matches the hostname i.e. 74.125.236.52. Obviously, this is bound to result in failure (you could have verified this by browsing to the URL https://74.125.236.52/accounts/ClientLogin with a browser, and seen the resulting error yourself).
Supposedly, for the sake of security, you are hesitant to write your own TrustManager (and you musn't unless you understand how to write a secure one), you ought to look at establishing DNS records in your datacenter to ensure that all lookups to www.google.com will resolve to 74.125.236.52; this ought to be done either in your local DNS servers or in the hosts file of your OS; you might need to add entries to other domains as well. Needless to say, you will need to ensure that this is consistent with the records returned by your ISP.

I had similar problem. I was using Android's DefaultHttpClient. I have read that HttpsURLConnection can handle this kind of exception. So I created custom HostnameVerifier which uses the verifier from HttpsURLConnection. I also wrapped the implementation to custom HttpClient.
public class CustomHttpClient extends DefaultHttpClient {
public CustomHttpClient() {
super();
SSLSocketFactory socketFactory = SSLSocketFactory.getSocketFactory();
socketFactory.setHostnameVerifier(new CustomHostnameVerifier());
Scheme scheme = (new Scheme("https", socketFactory, 443));
getConnectionManager().getSchemeRegistry().register(scheme);
}
Here is the CustomHostnameVerifier class:
public class CustomHostnameVerifier implements org.apache.http.conn.ssl.X509HostnameVerifier {
#Override
public boolean verify(String host, SSLSession session) {
HostnameVerifier hv = HttpsURLConnection.getDefaultHostnameVerifier();
return hv.verify(host, session);
}
#Override
public void verify(String host, SSLSocket ssl) throws IOException {
}
#Override
public void verify(String host, X509Certificate cert) throws SSLException {
}
#Override
public void verify(String host, String[] cns, String[] subjectAlts) throws SSLException {
}
}

A cleaner approach ( only for test environment) in httpcliet4.3.3 is as follows.
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslContext,SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
CloseableHttpClient httpclient = HttpClients.custom().setSSLSocketFactory(sslsf).build();

In httpclient-4.3.3.jar, there is another HttpClient to use:
public static void main (String[] args) throws Exception {
// org.apache.http.client.HttpClient client = new DefaultHttpClient();
org.apache.http.client.HttpClient client = HttpClientBuilder.create().build();
System.out.println("HttpClient = " + client.getClass().toString());
org.apache.http.client.methods.HttpPost post = new HttpPost("https://www.rideforrainbows.org/");
org.apache.http.HttpResponse response = client.execute(post);
java.io.InputStream is = response.getEntity().getContent();
java.io.BufferedReader rd = new java.io.BufferedReader(new java.io.InputStreamReader(is));
String line;
while ((line = rd.readLine()) != null) {
System.out.println(line);
}
}
This HttpClientBuilder.create().build() will return org.apache.http.impl.client.InternalHttpClient. It can handle the this hostname in certificate didn't match issue.

Thanks Vineet Reynolds. The link you provided held a lot of user comments - one of which I tried in desperation and it helped. I added this method :
// Do not do this in production!!!
HttpsURLConnection.setDefaultHostnameVerifier( new HostnameVerifier(){
public boolean verify(String string,SSLSession ssls) {
return true;
}
});
This seems fine for me now, though I know this solution is temporary. I am working with the network people to identify why my hosts file is being ignored.

The concern is we should not use ALLOW_ALL_HOSTNAME_VERIFIER.
How about I implement my own hostname verifier?
class MyHostnameVerifier implements org.apache.http.conn.ssl.X509HostnameVerifier
{
#Override
public boolean verify(String host, SSLSession session) {
String sslHost = session.getPeerHost();
System.out.println("Host=" + host);
System.out.println("SSL Host=" + sslHost);
if (host.equals(sslHost)) {
return true;
} else {
return false;
}
}
#Override
public void verify(String host, SSLSocket ssl) throws IOException {
String sslHost = ssl.getInetAddress().getHostName();
System.out.println("Host=" + host);
System.out.println("SSL Host=" + sslHost);
if (host.equals(sslHost)) {
return;
} else {
throw new IOException("hostname in certificate didn't match: " + host + " != " + sslHost);
}
}
#Override
public void verify(String host, X509Certificate cert) throws SSLException {
throw new SSLException("Hostname verification 1 not implemented");
}
#Override
public void verify(String host, String[] cns, String[] subjectAlts) throws SSLException {
throw new SSLException("Hostname verification 2 not implemented");
}
}
Let's test against https://www.rideforrainbows.org/ which is hosted on a shared server.
public static void main (String[] args) throws Exception {
//org.apache.http.conn.ssl.SSLSocketFactory sf = org.apache.http.conn.ssl.SSLSocketFactory.getSocketFactory();
//sf.setHostnameVerifier(new MyHostnameVerifier());
//org.apache.http.conn.scheme.Scheme sch = new Scheme("https", 443, sf);
org.apache.http.client.HttpClient client = new DefaultHttpClient();
//client.getConnectionManager().getSchemeRegistry().register(sch);
org.apache.http.client.methods.HttpPost post = new HttpPost("https://www.rideforrainbows.org/");
org.apache.http.HttpResponse response = client.execute(post);
java.io.InputStream is = response.getEntity().getContent();
java.io.BufferedReader rd = new java.io.BufferedReader(new java.io.InputStreamReader(is));
String line;
while ((line = rd.readLine()) != null) {
System.out.println(line);
}
}
SSLException:
Exception in thread "main" javax.net.ssl.SSLException: hostname in certificate didn't match: www.rideforrainbows.org != stac.rt.sg OR stac.rt.sg OR www.stac.rt.sg
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:231)
...
Do with MyHostnameVerifier:
public static void main (String[] args) throws Exception {
org.apache.http.conn.ssl.SSLSocketFactory sf = org.apache.http.conn.ssl.SSLSocketFactory.getSocketFactory();
sf.setHostnameVerifier(new MyHostnameVerifier());
org.apache.http.conn.scheme.Scheme sch = new Scheme("https", 443, sf);
org.apache.http.client.HttpClient client = new DefaultHttpClient();
client.getConnectionManager().getSchemeRegistry().register(sch);
org.apache.http.client.methods.HttpPost post = new HttpPost("https://www.rideforrainbows.org/");
org.apache.http.HttpResponse response = client.execute(post);
java.io.InputStream is = response.getEntity().getContent();
java.io.BufferedReader rd = new java.io.BufferedReader(new java.io.InputStreamReader(is));
String line;
while ((line = rd.readLine()) != null) {
System.out.println(line);
}
}
Shows:
Host=www.rideforrainbows.org
SSL Host=www.rideforrainbows.org
At least I have the logic to compare (Host == SSL Host) and return true.
The above source code is working for httpclient-4.2.3.jar and httpclient-4.3.3.jar.

Updating the java version from 1.8.0_40 to 1.8.0_181 resolved the issue.

SSLConnectionSocketFactory sslConnectionSocketFactory = new SSLConnectionSocketFactory(
SSLContexts.custom().loadTrustMaterial(null, new TrustSelfSignedStrategy()).build(),
SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
CloseableHttpClient httpClient = HttpClients.custom().setSSLSocketFactory(sslConnectionSocketFactory).build();

Implementing custom HTTP methods with HttpCore

I'm new to Java and am hoping for some direction with Apache HttpCore library.
I've written a simple server, and would like to implement a few custom HTTP methods. I've gone through the docs a few time but haven't been able to figure it out.
It looks like the 501 Not Implemented is raised in HttpService.doService(), but overriding that method doesn't work. My request handler doesn't get called.
Any help will be appreciated.
Thanks.
Here's the gist of what I've got:
ServerSocket serverSocket;
HttpParams params;
HttpService httpService;
serverSocket = new ServerSocket(8000);
params = new BasicHttpParams();
params.setIntParameter(CoreConnectionPNames.SO_TIMEOUT, 5000);
params.setIntParameter(CoreConnectionPNames.SOCKET_BUFFER_SIZE, 8 * 1024);
params.setBooleanParameter(CoreConnectionPNames.STALE_CONNECTION_CHECK, false);
params.setBooleanParameter(CoreConnectionPNames.TCP_NODELAY, true);
params.setParameter(CoreProtocolPNames.ORIGIN_SERVER, "?");
BasicHttpProcessor httpproc = new BasicHttpProcessor();
httpproc.addInterceptor(new ResponseDate());
httpproc.addInterceptor(new ResponseServer());
httpproc.addInterceptor(new ResponseContent());
httpproc.addInterceptor(new ResponseConnControl());
HttpRequestHandlerRegistry registry = new HttpRequestHandlerRegistry();
registry.register("*", new HttpRequestHandler() {
public void handle(HttpRequest request, HttpResponse response,
HttpContext context) throws HttpException, IOException {
System.out.println(request.getRequestLine().toString());
}
});
httpService = new HttpService(httpproc, new DefaultConnectionReuseStrategy(), new DefaultHttpResponseFactory());
httpService.setParams(params);
httpService.setHandlerResolver(registry);
Socket socket = serverSocket.accept();
DefaultHttpServerConnection conn = new DefaultHttpServerConnection();
conn.bind(socket, params);
HttpContext context = new BasicHttpContext();
httpService.handleRequest(conn, context);
socket.close();
conn.shutdown();
serverSocket.close();
Response:
# curl -X FOO -i http://127.0.0.1:8000
HTTP/1.0 501 Not Implemented
Content-Length: 26
Content-Type: text/plain; charset=US-ASCII
Connection: Close
FOO method not supported
Request line isn't written to System.out unless method is GET, POST, etc.
SOLUTION: I needed to implement a HttpRequestFactory.
e.g.:
DefaultHttpServerConnection conn = new DefaultHttpServerConnection() {
#Override
public DefaultHttpRequestFactory createHttpRequestFactory() {
return new DefaultHttpRequestFactory() {
#Override
public HttpRequest newHttpRequest(final RequestLine requestline) {
return new BasicHttpRequest(requestline);
}
#Override
public HttpRequest newHttpRequest(final String method, final String uri) {
return new BasicHttpRequest(method, uri);
}
};
}
};

It doesn't look like you need to override doService(). You rather need to implement a handler for you method and make sure
handler = this.handlerResolver.lookup(requestURI);
returns your handler. I take you did that, but for some reason your handler is not found. I bet you didn't register it properly.