We Keep Coding

How to Request an External JSON URL From Within Java Servlet or Class When the JSON URL sits behind NTLM Authentication On Another Server/Domain

Accessing the following JSON URL from within the web browser is simple on a Windows machine as this pops up an authentication box asking for the username and password which when entered displays the JSON data correctly.
www.json-behind-ntlm-authentication.com/view-data
I am now trying to move this into a Java Servlet.
I have tested the HttpClient library, http://hc.apache.org, and every example I have tried from their documentation, doesn't work. Most of the code I've tried doesn't even compile correctly.
I have also tested Jsoup, https://jsoup.org/, as that is a very good library for web scraping, but this doesn't seem to support accessing pages behind NTLM authentication.
I have also tested the code found here, https://blogs.msdn.microsoft.com/freddyk/2010/01/19/connecting-to-nav-web-services-from-java/, which is the only code sample I can find related to accessing a JSON URL that sits behind NTLM authentication. This is actually what I'm looking to achieve, a Java web application accessing Microsoft Nav data through their web services - and even this official example doesn't compile.
Any pointers / options? There must be a Java library somewhere that has this problem solved? The access is currently over HTTP, but ultimately is going to be over SSL for security reasons, so any solution must also support SSL handshakes.
I would really like not to build a separate C# application using LINQ, https://blogs.msdn.microsoft.com/freddyk/2009/04/20/using-linq-with-nav-web-services/, which I would hope works, but I'm not hopeful that the C# example would work in this scenario based on the Java examples not compiling.
UPDATE
After an awful lot of searching, I've found the following code below which seems to be close to working, but not quite - See the comments in the code where this is breaking. Thanks for the pointers in the comments already.
DefaultHttpClient httpclient = new DefaultHttpClient();
List<String> authpref = new ArrayList<String>();
authpref.add(AuthPolicy.NTLM);
httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF, authpref); //ERROR - This causes an error: java.lang.VerifyError: Cannot inherit from final class
NTCredentials creds = new NTCredentials(username, password, "", domain);
httpclient.getCredentialsProvider().setCredentials(AuthScope.ANY, creds);
HttpHost target = new HttpHost(baseURL);
// Make sure the same context is used to execute logically related requests
HttpContext localContext = new BasicHttpContext();
// Execute a cheap method first. This will trigger NTLM authentication
HttpGet httpget = new HttpGet(baseURL);
HttpResponse response1 = httpclient.execute(target, httpget, localContext); //ERROR - This line is throwing an error: java.lang.VerifyError: Cannot inherit from final class
HttpEntity entity = response1.getEntity();
System.out.println(EntityUtils.toString(entity));
I'm still unsure how to actually solve this problem. Any additional pointers?

org.apache.http.auth has NTCredentials which you can use in a HttpComponentsMessageSender in a spring boot #Configuration
HttpComponentsMessageSender httpComponentsMessageSender = new HttpComponentsMessageSender();
NTCredentials credentials = new NTCredentials("username", "password", null, "domain");

HttpComponentMessageSender NTLM

Today I was trying to configure the HttpComponentsMessageSender which uses Apache's org.apache.httpcomponents.httpclient library. Before I used the standard Java classes (I think java.net.HttpUrlMessageSender or something along those lines).
My HTTP methods need to authenticate using NTLM and the software is running under Linux so I cannot use the default Windows mechanism to do authentication, but instead I must configure authentication inside the Java application.
Originally I was just using a custom implementation of java.net.Authenticator which I used to set the username and password
for the NTLM access. But when I switched to HttpComponentsMessageSender this approach did not work anymore. When I tried to setup a custom HttpClient configuration with the HttpComponentsMessageSender I ran into various issues which I thought I would document here.
I am going to post my own answer but if anyone has a better solution, please feel free to add your solution.

I solved this adding a custom build HttpClient object which I built with the HttpClientBuilder. However adding this caused the following exception to appear:
org.apache.http.ProtocolException: Content-Length header already present
The solution which I found in this answer was to add a HttpRequestInterceptor.
Using the setConnectionTimeout on the HttpComponentsMessageSender did not work anymore with my custom HttpClient object. Instead I had to inject a RequestConfig object into the HttpClientBuilder.
The NTCredentials object required the domain name as a parameters. Earlier when I was using the Authenticator interface from java.net it was enough to just supply the username and password.
This is the code I am using currently:
HttpComponentsMessageSender messageSender = new HttpComponentsMessageSender ();
AuthScope authscope;
NTCredentials credentials;
CredentialsProvider credentialsProvider;
Registry<AuthSchemeProvider> registry;
RequestConfig requestConfig;
authscope = new AuthScope (HOST_IP, HOST_PORT);
credentials = new NTCredentials ("user", "pass", null, "domain");
credentialsProvider = new BasicCredentialsProvider ();
credentialsProvider.setCredentials (authscope, credentials);
registry = RegistryBuilder.<AuthSchemeProvider>create ()
.register(AuthSchemes.NTLM, new NTLMSchemeFactory ())
.build ();
HttpRequestInterceptor interceptor
= (request, context) -> request.removeHeaders(HTTP.CONTENT_LEN);
requestConfig = RequestConfig.custom ()
.setConnectTimeout (3000)
.build ();
HttpClient httpClient
= HttpClientBuilder.create ()
.setDefaultRequestConfig (requestConfig)
.setDefaultAuthSchemeRegistry (registry)
.setDefaultCredentialsProvider (credentialsProvider)
.addInterceptorFirst (interceptor)
.build ();
messageSender.setHttpClient (httpClient);

Webwervice security header in java client code

<soapenv:Header>
<UsernameToken xmlns="http://siebel.com/webservices">uname</UsernameToken>
<PasswordText xmlns="http://siebel.com/webservices">pass</PasswordText>
<SessionType xmlns="http://siebel.com/webservices">None</SessionType>
</soapenv:Header>
I have generated client code using axis runtime and I am getting this exception:Operation 'QueryByExample' with no authentication cannot be executed in an anonymous session pool. Please associate an authentication type with the web service operation.(SBL-EAI-04552).
I looked at others answers from stackoverflow for hours and nothing worked out for me. Please guide me....
Service code:
SOAPHeaderElement wsseSecurity = new SOAPHeaderElement(new PrefixedQName("http://schemas.xmlsoap.org/ws/2002/04/secext","Security", "wsse"));
MessageElement username = new MessageElement("uname", "UsernameToken");
MessageElement password = new MessageElement("pass", "PasswordText");
username.setObjectValue("username");
password.setObjectValue("password");
wsseSecurity.addChild(username);
wsseSecurity.addChild(password);
stub.setHeader(wsseSecurity);

You need to add few more things and the main problem is how you are deifning and using your password. You are on the correct path but here is how you would make a SOAP Axis 1.4 client:
InputStream inConfig = BaseTestCase.class.getClassLoader().getResourceAsStream("axis_client_config.xml");
EngineConfiguration config = new FileProvider(inConfig);
PartnerAPILocator locator = new PartnerAPILocator(config);
inConfig.close();
stub = locator.getSoap();
Stub axisPort = (Stub) stub;
axisPort._setProperty(UsernameToken.PASSWORD_TYPE, WSConstants.PASSWORD_TEXT);
axisPort._setProperty(WSHandlerConstants.USER, "ET USERNAME");
axisPort._setProperty(WSHandlerConstants.PW_CALLBACK_REF, new PasswordTokenHandler());
here is my source and try to follow the tutorial and you can even download the code. Hope this helps you.

Is there any way with apache httpclient to set credentials based of a URI

Currently I have some uris that I call using resteasy that That looks something like this:
http://host.com/api/project1/getsomestuff
and
http://host.com/api/project2/getsomestuff
Both use digest authentication but require a different username and password for authentication. Currently in code I have to handle this by creating a different client instance for each project like this:
DefaultHttpClient project1Client = new DefaultHttpClient();
Credentials project1Credentials = new UsernamePasswordCredentials("user1", "password1");
project1Client.getCredentialsProvider().setCredentials(AuthScope.ANY, project1Credentials);
ClientExecutor executor1 = new ApacheHttpClient4Executor(project1Client);
MyService project1Proxy = ProxyFactory.create(MyService.class, executor1);
project1Proxy.getSomeStuff("project1");
DefaultHttpClient project2Client = new DefaultHttpClient();
Credentials project2Credentials = new UsernamePasswordCredentials("user2", "password2");
project2Client.getCredentialsProvider().setCredentials(AuthScope.ANY, project1Credentials);
ClientExecutor executor2 = new ApacheHttpClient4Executor(project1Client);
MyService project2Proxy = ProxyFactory.create(MyService.class, executor2);
project2Proxy.getSomeStuff("project2");
I have looked through AuthScope and CredentialsProvider and I just can't see any way how this would be possible. Does anybody know of any way to use both sets of credentials using the same client or am I stuck using multiple clients?
Also, please note that I do not control the server side of this rest call and both calls use the same realm so I am unable to set the different credentials per realm.

If those applications have a different security contexts they are also likely to have different authentication realms (or at least they are expected to). One can explicitly set the realm attribute on the AuthScope object to explicitly match it to a specific authentication context with the given realm name.

I think you are out of luck. Looking through the docs you cannot create an Authscope which sends a different password to different paths on the same host and port.

Java Authenticator on a per connection basis?

I'm building an Eclipse plugin that talks to a REST interface which uses Basic Authentication. When the authentication fails I would like to popup my plugin's settings dialog and retry. Normally I could use the static Authenticator.setDefault() to setup an authenticator for all HttpURLConnection's for this, but since I am writing a plugin I don't want to overwrite Eclipse's default Authenticator (org.eclipse.ui.internal.net.auth);
I thought of setting my custom Authenticator before loading and putting Eclipse's default back afterwards, but I imagine this will cause all sorts of race issues with multithreading so I quickly lost that notion.
Google searches yield all sorts of results basically telling me it's not possible:
The Java URLConnection API should have a setAuthenticator(Authenticator) method for making it easier to use this class in multi-threaded context where authentication is required.
Source
If applications contains few third party plugins and each plugin use its own Authenticator what we should do? Each invocation of "Authenticator.setDefault()" method rewrite previously defined Authenticator...
Source
Are there any different approaches that might help me overcome this issue?

If it is not possible with HttpURLConnection I would suggest using the httpclient library from Apache.
A quick example:
HttpClient client = new HttpClient();
client.getState().setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("test","test"));
GetMethod getMethod = new GetMethod("http://www.example.com/mylogin");
client.executeMethod(getMethod);
System.out.println(getMethod.getResponseBodyAsString());

Another approach would be to perform the basic authentication yourself on the connection.
final byte[] encodedBytes = Base64.encodeData((username + ':' + new String(password)).getBytes("iso-8859-1"));
final String encoded = new String(encodedBytes, "iso-8859-1");
connection.setRequestProperty("Authorization", "Basic " + encoded);
This would also have the advantage of not requiring an unauthenticated request to receive a 401 before providing the credential on a subsequent request. Similar behavior can be leveraged in the apache http-client by requesting preemptive authentication.