We Keep Coding

How to Request an External JSON URL From Within Java Servlet or Class When the JSON URL sits behind NTLM Authentication On Another Server/Domain

Accessing the following JSON URL from within the web browser is simple on a Windows machine as this pops up an authentication box asking for the username and password which when entered displays the JSON data correctly.
www.json-behind-ntlm-authentication.com/view-data
I am now trying to move this into a Java Servlet.
I have tested the HttpClient library, http://hc.apache.org, and every example I have tried from their documentation, doesn't work. Most of the code I've tried doesn't even compile correctly.
I have also tested Jsoup, https://jsoup.org/, as that is a very good library for web scraping, but this doesn't seem to support accessing pages behind NTLM authentication.
I have also tested the code found here, https://blogs.msdn.microsoft.com/freddyk/2010/01/19/connecting-to-nav-web-services-from-java/, which is the only code sample I can find related to accessing a JSON URL that sits behind NTLM authentication. This is actually what I'm looking to achieve, a Java web application accessing Microsoft Nav data through their web services - and even this official example doesn't compile.
Any pointers / options? There must be a Java library somewhere that has this problem solved? The access is currently over HTTP, but ultimately is going to be over SSL for security reasons, so any solution must also support SSL handshakes.
I would really like not to build a separate C# application using LINQ, https://blogs.msdn.microsoft.com/freddyk/2009/04/20/using-linq-with-nav-web-services/, which I would hope works, but I'm not hopeful that the C# example would work in this scenario based on the Java examples not compiling.
UPDATE
After an awful lot of searching, I've found the following code below which seems to be close to working, but not quite - See the comments in the code where this is breaking. Thanks for the pointers in the comments already.
DefaultHttpClient httpclient = new DefaultHttpClient();
List<String> authpref = new ArrayList<String>();
authpref.add(AuthPolicy.NTLM);
httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF, authpref); //ERROR - This causes an error: java.lang.VerifyError: Cannot inherit from final class
NTCredentials creds = new NTCredentials(username, password, "", domain);
httpclient.getCredentialsProvider().setCredentials(AuthScope.ANY, creds);
HttpHost target = new HttpHost(baseURL);
// Make sure the same context is used to execute logically related requests
HttpContext localContext = new BasicHttpContext();
// Execute a cheap method first. This will trigger NTLM authentication
HttpGet httpget = new HttpGet(baseURL);
HttpResponse response1 = httpclient.execute(target, httpget, localContext); //ERROR - This line is throwing an error: java.lang.VerifyError: Cannot inherit from final class
HttpEntity entity = response1.getEntity();
System.out.println(EntityUtils.toString(entity));
I'm still unsure how to actually solve this problem. Any additional pointers?

org.apache.http.auth has NTCredentials which you can use in a HttpComponentsMessageSender in a spring boot #Configuration
HttpComponentsMessageSender httpComponentsMessageSender = new HttpComponentsMessageSender();
NTCredentials credentials = new NTCredentials("username", "password", null, "domain");

How to instantiate GoogleIdTokenVerifier properly / what does .setAudience() do?

My Guidelines
If followed this Google documentation about verifying Google-Account-Tokens on the server side, but I am kinda confused.
My Problem
GoogleIdTokenVerifier googleIdTokenVerifier = new GoogleIdTokenVerifier.Builder(new NetHttpTransport(), new JacksonFactory())
.setAudience(Collections.singletonList(CLIENT_ID))
.build();
In this piece of code I figured out that the transport and jsonFactory arguments can be filled as new NetHttpTransport() and new JacksonFactory() here. It also describes how to get AudienceString, but I couldn't figure out what it is for. I couldn't test it, but my question is if I can use it without .setAudience() or if I need it and what it is for.

In .setAudience() you have to pass all client ID's. You can get the ID for your client from the Credentials Page. It's explained here.
Thanks to #StevenSoneff.
If you didn't get the basic concept
For every client you want your server to accept, you need to create a project in the `Developer Console`. Clients are differentiated by their `SHA-1` fingerprint. You can for example have a debug project (will take your debug fingerprint) and a release one. To make both work, you have to add both `ID`'s to your server's `GoogleIdTokenVerifier`'s `.setAudience()`.

In my case, If you're using Firebase to get the id token on Android or iOS. You should follow these instructions to verify it on your backend server.
Verify ID tokens using a third-party JWT library
For me, I'm using Google OAuth Client as the third-party library so it's easy to use.
But it's a little bit different from this document.
Verify the Google ID token on your server side
The CLIENT_ID is your firebase project ID.
The Issuer has to be set as https://securetoken.google.com/<projectId>.
You need to use GooglePublicKeysManager and call setPublicCertsEncodedUrl to set it as https://www.googleapis.com/robot/v1/metadata/x509/securetoken#system.gserviceaccount.com
GooglePublicKeysManager manager = new GooglePublicKeysManager.Builder(HTTP_TRANSPORT, JSON_FACTORY)
.setPublicCertsEncodedUrl(PUBLIC_KEY_URL)
.build();
GoogleIdTokenVerifier verifier = new GoogleIdTokenVerifier.Builder(manager)
.setAudience(Collections.singletonList(FIREBASE_PROJECT_ID))
.setIssuer(ISSUER)
.build();
If you have multiple issuers, then you have to create GoogleIdTokenVerifier for each one.

Java Web service client can't connect to BPEL webservice when SoapUI can

I am experiencing the following error and after extensive research on the matter with a few similar issues, none seem to help.
My error:
com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: java.net.ConnectException: Connection refused: connect
at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:121)
at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:142)
at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:83)
at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:105)
at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Fiber.java:587)
at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Fiber.java:546)
at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Fiber.java:531)
at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Fiber.java:428)
at com.sun.xml.internal.ws.client.Stub.process(Stub.java:211)
at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(SEIStub.java:124)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:98)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:78)
at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:107)
at com.sun.proxy.$Proxy33.onboardPerson(Unknown Source)
at com.paychex.corp.fileloader.FileloaderMain.callWebService(FileloaderMain.java:202)
at com.paychex.corp.fileloader.FileloaderMain.main(FileloaderMain.java:104)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:411)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:525)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:208)
at sun.net.www.http.HttpClient.New(HttpClient.java:291)
at sun.net.www.http.HttpClient.New(HttpClient.java:310)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:987)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:966)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:841)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1031)
at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:109)
My client is generated from wsimport using the WSDL file that IS visible from the active service. Code connection sample: URL and service name, etc are correct, can't list for security
URL url = new URL("http://correctserviceurl?WSDL");
QName qname = new QName("http://correct servicelocation","nameofService");
CorporateEmployeeOnboardingService service = new CorporateEmployeeOnboardingService(url, qname);
//CorporateEmployeeOnboardingService service = new CorporateEmployeeOnboardingService();
CorporateEmployeeOnboardingProcess process = service.getICorporateEmployeeOnboardingProcess();
OnboardRequestType onboardRequest = new OnboardRequestType();
/**
* TESTING SECTION FOR PROCESS CALL
*/
//create necessary complex types:
GregorianCalendar gregorianCalendar = new GregorianCalendar();
DatatypeFactory datatypeFactory;
datatypeFactory = DatatypeFactory.newInstance();
XMLGregorianCalendar testDate =
datatypeFactory.newXMLGregorianCalendar(gregorianCalendar);
testDate.setYear(1991);
testDate.setMonth(4);
testDate.setDay(3);
AssignmentComplexType testAssignment = new AssignmentComplexType();
testAssignment.setHiringManagerID("123445");
//set person to send
person.setSSN("333-33-3333");
person.setFirstName("testFname");
person.setLastName("testLname");
person.setDOB(testDate);
person.setHireDate(testDate);
person.setAssignment(testAssignment);
/**
* end testing section of the person creation
*/
onboardRequest.getPerson().add(person);//add a person to the array of people for the request
ServiceResponse response = process.onboardPerson(onboardRequest);
As stated SoapUI can send this same test information and get a perfectly valid request from the service. All information the same. I have been digging around for awhile and confirmed that there is no proxy, both are same bit (32/64 in this case both 32), service name and WSDL are correct and objects are named properly, etc. I did further debugging trying to get around this awful error code that has seemingly no usable information and using wireshark compared the SoapUI request to the java request to find that SoapUI was using POST and the java client was using GET which has me leaning on this right now as the source of the generic refused connection error.
The service is using BPEL to run and was created using BPEL so all code was basically generated. I will be looking at the Get/post find as of now but wanted to get this out there as I am doubtful it could be the cause.
Any help is greatly appreciated. Let me know if I forgot any key information I might be able to share.
EDIT
I wanted to add that I am able to access the endpoint URL in the web browser from the machine that I am using to try and connect to the BPEL service. It seems the issue is solely with the client generated from the WSDL stubs I got using wsimport for the service's WSDL.
Edit 2: Development/Progress
The issue seems to be that the client is sending the response to local host even after setting up the qname to the correct address. I am now trying to figure out where this local host is being set. Any ideas on this development would be greatly appreciated.

It's been a while, but I believe this line: URL url = new URL("http://correctserviceurl?WSDL");
Needs to read: URL url = new URL("http://correctserviceurl");

K! This one was an issue with BPEL I believe. Quite frustrating. Apparently after initializing the service in the following lines:
URL url = new URL("http://serviceaddresshere?WSDL");
QName qname = new QName("http://servicelocationaddress","NameOfService");
CorporateEmployeeOnboardingService service = new CorporateEmployeeOnboardingService(url, qname);
CorporateEmployeeOnboardingProcess process = service.getICorporateEmployeeOnboardingProcess();
BPEL gets the request with the service WSDL and the CORRECT address. However, BPEL seems to think "O? you want this service, here you go I'll give you that, only the WSDL you want is wrapper, so here's the REAL WSDL." In plain english, you get back the BPEL WSDL that it thinks is for the service, and this has the Localhost address in it. Now I tried changing the address in this WSDL but BPEL through errors up the wazzoo. So the work around was adding in this line after the process variable is make: (AKA THE FIX)
((BindingProvider) process).getRequestContext().put(Dispatch.ENDPOINT_ADDRESS_PROPERTY, "http://the serviceaddress");
Bassically you set the endpoint AGAIN after you make the process, because BPEL overwrites what you sent it the first time. This is of course a sort of work around, but the process now works good. If you can think of a way to bypass this work around and dynamically fix the service so we don't have to keep changing the service WSDL when we move ti to new enviornments like Dev, etc. I am all ears. Thanks and as always I will read comments for suggestions. =)

You can use bpel configuration plans to switch between environments.
Configuration plans are the recommended option to use for making composites environment specific.
Efficient use of configuration plans can greatly reduce the work required per process.
A configuration plan can be generated from the composite.xml file.
Please consider this link:
http://biemond.blogspot.com.br/2010/02/web-service-references-and-soa.html

Simple Kerberos client in Java?

Applications such a Google's Chrome and IE can transparently handle Kerberos authentication; however I can not find a "simple" Java solution to match this transparency. All of the solutions I have found require the presence of a krb5.conf file and a login.conf file which nether of the above apps seem to require.
What is the best way to build a Java application with Kerberos SSO capabilities that just work?
[update]: to be clear I need a CLIENT side solution for creating tickets not validating them. Also, it seems that SPNEGO is the default "wrapper" protocol that will eventually delegate to Kerberos but I need to be able to handle the SPNEGO protocol as well.

There is now a simple solution for this using the Apache HTTP Components Client 4.5 or greater. This is still marked as experimental in 4.5 so your milage may vary, but this is working fine for me in an enterprise context.
In addition to the HC 4.5 client jars you will need to have the httpclient-win, jna and jna-platform jars on your classpath, as provided with http-component-client. You then construct a Kerberos enabled HC-client as follows:
CloseableHttpClient httpclient = WinHttpClients.createDefault();
Or using the builder:
HttpClientBuilder clientBuilder = WinHttpClients.custom();
Which can then be customised as required before building the client:
CloseableHttpClient client = clientBuilder.build();
This solution works without any external configuration, and most importantly solves the issue where the in-built JRE mechanism breaks for users with local Admin rights on Windows 7+. This is possible because the Kerberos ticket is being retrieved directly from the SSPI API via JNA, rather than going through the GSSAPI provided by the JRE.
Example code from the http-components team
This was all made possible by the good work of Daniel Doubrovkine Timothy Wall
and Ryan McKinley

Adding to David Roussels answer on url specific http based kerberos authentication:-
The reason why your code works is because your target SPN(server side principal) is configured to with HTTP/serverhostname.realm.com#DOMAIN.COM. In that case it will work because you are not explicitly setting the token. URLConnection internally sets a token with that SPN
1 Perform steps(from my previous answer) to get a subject
2 Use gss api init sec context to generate a context token. There are numerous tutorials out there for this step
3 Base 64 encode the token
4 Attach the token to urlconnection:-
URL url = new URL("http://myhost/myapp")
HttpURLConnection urlConn = (HttpURLConnection)url.openConnection(); =
urlConn.setRequestProperty("Authorization", "Negotiate " + encodedToken);
5 Implement a priviledged action:-
//this internally calls the getInputStream
public class PrivilegedGetInputStream implements PrivilegedExceptionAction<InputStream>
6 Wrap the whole thing in Subject.doAs
//use prev answer instructions to get subject
Subject.doAs(subject, new PrivilegedGetInputStream(urlConnection)

Oracle has an example using Java's SaslClient. I'm not a Java programmer, but when I pointed this out once to someone who is, they were able to make it work pretty quickly. It may still require a "conf" file somewhere (n.b. Kerberos uses environment variables, often starting with KRB5_, to know where to look for such files). Also note that Kerberos itself does not include a transport of any kind--your app needs to know how to send and receive the Kerberos payloads the way the server expects (and this is different depending on the server you are trying to authenticate with).
Edit: you edited your question, so here's a link related to SPNEGO in Java which might be of some use:
http://download.oracle.com/javase/6/docs/technotes/guides/security/jgss/lab/part5.html

You don't actually need to do anything. In Java 6, on a Windows client machine you can do this:
new URL("http://myhost/myapp").openStream();
And negotiate authentication just works. At least it does for me. And the server I tested on only supports Negotiate, not NTLM auth.

Ok if you want to avoid using a login.conf file you need to code differently:-
//define your own configuration
import javax.security.auth.login.Configuration;
public class CustomLoginConfiguration extends Configuration
//pass certain parameters to its constructor
//define an config entry
import javax.security.auth.login.AppConfigurationEntry;
private AppConfigurationEntry configEntry;
//define a map of params you wish to pass and fill them up
//the map contains entries similar to one you have in login.conf
Map<String, String> params = new HashMap<String, String>();
//define the configuration
configEntry = new AppConfigurationEntry(
"com.sun.security.auth.module.Krb5LoginModule",
AppConfigurationEntry.LoginModuleControlFlag.REQUIRED, params);
//implement getappconfig method
public AppConfigurationEntry[] getAppConfigurationEntry() {
return new AppConfigurationEntry[] { configEntry };
}
Now once you are done with this definition you can use this in you use this to fetch tickets from kdc
//get ticket in login context
LoginContext lc = null;
lc = new LoginContext("lc", null, callback, new CustomLoginConfiguration(argumentlist));
lc.login();
Now from here on you can fetch jaas subject and can basically do a ton of authentication stuff.
In case you need further pointers just leave a comment.

You can use system properties instead of config files to specify the KDC hostname and service name, but those things (at least) are mandatory....
Waffle will actually give you the information you need to set most of the properties, even if it won't get you a ticket. Look at the WindowsAuthProviderImpl class (the Waffle.chm help file shows the API).
I use JAAS do obtain a service ticket from Active Directory in two steps:
Use Krb5LoginModule to retrieve the cached TGT and add it to the Subject.
Use the Subject and GSS-API to retrieve a service ticket from the KDC.
There's a lot of good information and example code at The Java Way of Active Directory.

I created a small tool to simplify connecting with httpclient to kerberos, you might want to give it a try.
https://github.com/DovAmir/httpclientAuthHelper
DefaultHttpClient httpclient = new DefaultHttpClient();
AuthUtils.securityLogging(SecurityLogType.KERBEROS,true);
CredentialsUtils.setKerberosCredentials(client, new UsernamePasswordCredentials("xxx", "xxx"), "domain", "kdc");
client.executeMethod(httpget);

Use WAFFLE

Here's a good blog post on having a java client to use with Kerberos
http://sachithdhanushka.blogspot.com/2014/02/kerberos-java-client-configuration.html

Using Java, Need to establish an https connection via proxy

I need to establish and send/read over/from an https connection (to a website of course) but through an http proxy or SOCKS proxy. A few other requirements
supports blocking (I can't use non-blocking/nio)
isn't set as an environment or some other global scope property (there are multiple threads accessing)
I was looking into HttpCore components but I did not see any support for blocking https.

Look at the java.net.Proxy class. That does what you need. You create one, and then pass it to the URLConnection to create the connection.

To support per-thread proxy, your best bet is Apache HttpClient 4 (Http Components Client). Get the source code,
http://hc.apache.org/downloads.cgi
It comes with examples for both HTTP proxy and SOCKS proxy,
ClientExecuteProxy.java
ClientExecuteSOCKS.java

Did you look at Apache HTTP Client? Haven't used it in ages but I did use it to pick a proxy server dynamically. Example from site here:
HttpClient httpclient = new HttpClient();
httpclient.getHostConfiguration().setProxy("myproxyhost", 8080);
httpclient.getState().setProxyCredentials("my-proxy-realm", " myproxyhost",
new UsernamePasswordCredentials("my-proxy-username", "my-proxy-password"));
GetMethod httpget = new GetMethod("https://www.verisign.com/");
try {
httpclient.executeMethod(httpget);
System.out.println(httpget.getStatusLine());
} finally {
httpget.releaseConnection();
}

System.setProperty("http.proxyHost", "proxy.com");
System.setPropery("http.proxyPort", "8080");
URL url = new URL("http://java.sun.com/");
InputStream in = url.openStream();
http://java.sun.com/javase/6/docs/technotes/guides/net/proxies.html