We Keep Coding

How to Request an External JSON URL From Within Java Servlet or Class When the JSON URL sits behind NTLM Authentication On Another Server/Domain

Accessing the following JSON URL from within the web browser is simple on a Windows machine as this pops up an authentication box asking for the username and password which when entered displays the JSON data correctly.
www.json-behind-ntlm-authentication.com/view-data
I am now trying to move this into a Java Servlet.
I have tested the HttpClient library, http://hc.apache.org, and every example I have tried from their documentation, doesn't work. Most of the code I've tried doesn't even compile correctly.
I have also tested Jsoup, https://jsoup.org/, as that is a very good library for web scraping, but this doesn't seem to support accessing pages behind NTLM authentication.
I have also tested the code found here, https://blogs.msdn.microsoft.com/freddyk/2010/01/19/connecting-to-nav-web-services-from-java/, which is the only code sample I can find related to accessing a JSON URL that sits behind NTLM authentication. This is actually what I'm looking to achieve, a Java web application accessing Microsoft Nav data through their web services - and even this official example doesn't compile.
Any pointers / options? There must be a Java library somewhere that has this problem solved? The access is currently over HTTP, but ultimately is going to be over SSL for security reasons, so any solution must also support SSL handshakes.
I would really like not to build a separate C# application using LINQ, https://blogs.msdn.microsoft.com/freddyk/2009/04/20/using-linq-with-nav-web-services/, which I would hope works, but I'm not hopeful that the C# example would work in this scenario based on the Java examples not compiling.
UPDATE
After an awful lot of searching, I've found the following code below which seems to be close to working, but not quite - See the comments in the code where this is breaking. Thanks for the pointers in the comments already.
DefaultHttpClient httpclient = new DefaultHttpClient();
List<String> authpref = new ArrayList<String>();
authpref.add(AuthPolicy.NTLM);
httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF, authpref); //ERROR - This causes an error: java.lang.VerifyError: Cannot inherit from final class
NTCredentials creds = new NTCredentials(username, password, "", domain);
httpclient.getCredentialsProvider().setCredentials(AuthScope.ANY, creds);
HttpHost target = new HttpHost(baseURL);
// Make sure the same context is used to execute logically related requests
HttpContext localContext = new BasicHttpContext();
// Execute a cheap method first. This will trigger NTLM authentication
HttpGet httpget = new HttpGet(baseURL);
HttpResponse response1 = httpclient.execute(target, httpget, localContext); //ERROR - This line is throwing an error: java.lang.VerifyError: Cannot inherit from final class
HttpEntity entity = response1.getEntity();
System.out.println(EntityUtils.toString(entity));
I'm still unsure how to actually solve this problem. Any additional pointers?

org.apache.http.auth has NTCredentials which you can use in a HttpComponentsMessageSender in a spring boot #Configuration
HttpComponentsMessageSender httpComponentsMessageSender = new HttpComponentsMessageSender();
NTCredentials credentials = new NTCredentials("username", "password", null, "domain");

HttpClient redirecting

I am trying to write a video downloader from kissanime.to . I am using HttpClient library. This site is using cloudflare. It redirects after 5 secs. How can I set so my application will go to the redirected link? My code below isnt working. Where am I going wrong and how can I fix it.
HttpGet request = new HttpGet(url);
HttpClient httpClient = HttpClientBuilder.create()
.setRedirectStrategy(new LaxRedirectStrategy()).build();
HttpResponse response = httpClient.execute(request);
HttpEntity entity = response.getEntity();
String entityContents = EntityUtils.toString(entity);`

It means that given site is under DDOS protection mode (maybe you try to open it too often?) to workaround you would need to stop hitting it that much (e.g. wait some time between tries). Or if you insist use some javascript executing library (Rhino?) that would execute javascript that cloudflare is using.

Android login and redirect

I am trying to get some data from a webpage that needs login in a page. I could do with htmlunit but gave me problems import these libraries on Android. So I am trying to do it with apache http client.
HttpClient httpClient = new DefaultHttpClient();
HttpPost httpPost = new HttpPost("https://inside.cineca.it/cgi-bin/uinside/marcature.pl");
BasicNameValuePair usernameBasicNameValuePair = new BasicNameValuePair("j_username", "user");
BasicNameValuePair passwordBasicNameValuePAir = new BasicNameValuePair("j_password", "pass");
List<NameValuePair> nameValuePairList = new ArrayList<NameValuePair>();
nameValuePairList.add(usernameBasicNameValuePair);
nameValuePairList.add(passwordBasicNameValuePAir);
UrlEncodedFormEntity urlEncodedFormEntity = new UrlEncodedFormEntity(nameValuePairList);
HttpResponse httpResponse = httpClient.execute(httpPost);
System.out.println(EntityUtils.toString(httpResponse.getEntity()));
But is giving me error. I am affraid i should manage the redirect but i don't know how. I would be happy if someone could give me some advice.
Best regards

You're performing an http POST to a login page. Maybe the server that serves you back the login page doesn't know to handle the post. Anyway, I am sure your intention is to perform the login actually. You need to figure out what is the server API where you should post the credentials and process the result. Performing a quick investigation with Chrome Network profiler, the login API for above link actually resides at: https://idp-is.cineca.it/idp/Authn/Multilogin. That is the URL that you should be using and there is where you need to POST.
Also, make sure the request you're making is done on a non-UI thread as this is a common mistake done by beginners.

Android Rest Client

I found so many samples for requesting a REST API, but all together are confusing, can some one please explain me a way to use http requests.
My Requirement is, I want to get data from a REST API by providing username, pwd and a key.
What I have Used was,
HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost("REST API url");
post.setHeader("Content-type", "application/json");
JSONObject obj = new JSONObject();
obj.put("username", "un");
obj.put("pwd", "password");
obj.put("key","123456");
post.setEntity(new StringEntity(obj.toString(), "UTF-8"));
HttpResponse response = client.execute(post);
But the response is always null and these working fine when tested with browser tool by posting the same data.Is some thing wrong with my approach? please suggest me the correct way. Thank you

(1) Google I/O video session for developing REST clients
(2) search in android developer blog
(3) https://github.com/darko1002001/android-rest-client
Please try after that post your question,
I can share code snippet from my rest client developed based on (1) & (2)
Do not use Cloud to Device Messaging, instead use the latest cloud approach with android application development.
There is new library called Volley, which looks better than AsyncTask. It should be useful in developing RESTful clients.

You probably forgot to add the internet permission to the manifest file.
Add the following line.
<uses-permission android:name="android.permission.INTERNET" />

I thing you should try this,
HttpContext localContext = new BasicHttpContext();
HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost("REST API url");
post.setHeader("Content-type", "application/json");
JSONObject obj = new JSONObject();
obj.put("username", "un");
obj.put("pwd", "password");
obj.put("key","123456");
post.setEntity(new StringEntity(obj.toString(), "UTF-8"));
HttpResponse response = client.execute(post,localContext);
Hope this will help.

By any chance, is the server expecting a GET request for this operation? If so, you may want to use HttpGet instead of HttpPost.

HttpClient 4.1.1 returns 401 when authenticating with NTLM, browsers work fine

I'm trying to use the Apache/Jakarta HttpClient 4.1.1 to connect to an arbitrary web page using the given credentials. To test this, I have a minimal install of IIS 7.5 on my dev machine running where only one authentication mode is active at a time. Basic authentication works fine, but Digest and NTLM return 401 error messages whenever I try to log in. Here is my code:
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpContext localContext = new BasicHttpContext();
HttpGet httpget = new HttpGet("http://localhost/");
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(AuthScope.ANY,
new NTCredentials("user", "password", "", "localhost"));
if (!new File(System.getenv("windir") + "\\krb5.ini").exists()) {
List<String> authtypes = new ArrayList<String>();
authtypes.add(AuthPolicy.NTLM);
authtypes.add(AuthPolicy.DIGEST);
authtypes.add(AuthPolicy.BASIC);
httpclient.getParams().setParameter(AuthPNames.PROXY_AUTH_PREF,
authtypes);
httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF,
authtypes);
}
localContext.setAttribute(ClientContext.CREDS_PROVIDER, credsProvider);
HttpResponse response = httpclient.execute(httpget, localContext);
System.out.println("Response code: " + response.getStatusLine());
The one thing I've noticed in Fiddler is that the hashes sent by Firefox versus by HttpClient are different, making me think that maybe IIS 7.5 is expecting stronger hashing than HttpClient provides? Any ideas? It'd be great if I could verify that this would work with NTLM. Digest would be nice too, but I can live without that if necessary.

I am not an expert on the subject but during the NTLM authentication using http components I have seen that the client needs 3 attempts in order to connect to an NTML endpoint in my case. It is kinda described here for Spnego but it is a bit different for the NTLM authentication.
For NTLM in the first attempt client will make a request with Target auth state: UNCHALLENGED and Web server returns HTTP 401 status and a header: WWW-Authenticate: NTLM
Client will check for the configured Authentication schemes, NTLM should be configured in client code.
Second attempt, client will make a request with Target auth state: CHALLENGED, and will send an authorization header with a token encoded in base64 format: Authorization: NTLM TlRMTVNTUAABAAAAAYIIogAAAAAoAAAAAAAAACgAAAAFASgKAAAADw==
Server again returns HTTP 401 status but the header: WWW-Authenticate: NTLM now is populated with encoded information.
3rd Attempt Client will use the information from WWW-Authenticate: NTLM header and will make the final request with Target auth state: HANDSHAKE and an authorisation header Authorization: NTLM which contains more information for the server.
In my case I receive an HTTP/1.1 200 OK after that.
In order to avoid all this in every request documentation at chapter 4.7.1 states that the same execution token must be used for logically related requests. For me it did not worked.
My code:
I initialize the client once in a #PostConstruct method of an EJB
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
cm.setMaxTotal(18);
cm.setDefaultMaxPerRoute(6);
RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(30000)
.setConnectTimeout(30000)
.setTargetPreferredAuthSchemes(Arrays.asList(AuthSchemes.NTLM))
.setProxyPreferredAuthSchemes(Arrays.asList(AuthSchemes.BASIC))
.build();
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
new NTCredentials(userName, password, hostName, domainName));
// Finally we instantiate the client. Client is a thread safe object and can be used by several threads at the same time.
// Client can be used for several request. The life span of the client must be equal to the life span of this EJB.
this.httpclient = HttpClients.custom()
.setConnectionManager(cm)
.setDefaultCredentialsProvider(credentialsProvider)
.setDefaultRequestConfig(requestConfig)
.build();
Use the same client instance in every request:
HttpPost httppost = new HttpPost(endPoint.trim());
// HttpClientContext is not thread safe, one per request must be created.
HttpClientContext context = HttpClientContext.create();
response = this.httpclient.execute(httppost, context);
Deallocate the resources and return the connection back to connection manager, at the #PreDestroy method of my EJB:
this.httpclient.close();

I had the same problem with HttpClient4.1.X After upgrading it to
HttpClient 4.2.6 it woked like charm. Below is my code
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpContext localContext = new BasicHttpContext();
HttpGet httpget = new HttpGet("url");
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(AuthScope.ANY,
new NTCredentials("username", "pwd", "", "domain"));
List<String> authtypes = new ArrayList<String>();
authtypes.add(AuthPolicy.NTLM);
httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF,authtypes);
localContext.setAttribute(ClientContext.CREDS_PROVIDER, credsProvider);
HttpResponse response = httpclient.execute(httpget, localContext);
HttpEntity entity=response.getEntity();

The easiest way troubleshoot such situations I found is Wireshark. It is a very big hammer, but it really will show you everything. Install it, make sure your server is on another machine (does not work with Localhost) and start logging.
Run your request that fails, run one that works. Then, filter by http (just put http in the filter field), find the first GET request, find the other GET request and compare. Identify meaningful difference, you now have specific keywords or issues to search code/net for. If not enough, narrow down to first TCP conversation and look at full request/response. Same with the other one.
I solved an unbelievable number of problems with that approach. And Wireshark is very useful tool to know. Lots of super-advanced functions to make your network debugging easier.
You can also run it on either client or server end. Whatever will show you both requests to allow you to compare.

I had a similar problem with HttpClient 4.1.2. For me, it was resolved by reverting to HttpClient 4.0.3. I could never get NTLM working with 4.1.2 using either the built-in implementation or using JCIFS.

Updating our application to use the jars in the httpcomponents-client-4.5.1 resolved this issue for me.

I finally figured it out. Digest authentication requires that if you use a full URL in the request, the proxy also needs to use the full URL. I did not leave the proxy code in the sample, but it was directed to "localhost", which caused it to fail. Changing this to 127.0.0.1 made it work.