We Keep Coding

Implications of using NetHttpTransport() over GoogleNetHttpTransport.newTrustedTransport()

I'm using google-api-services-admin-directory in a project, and the docs recommend using GoogleNetHttpTransport.newTrustedTransport() as the HTTP transport, but I kept getting certificate errors that I think were cause by my proxy. I switched over to new NetHttpTransport() instead, and everything now works fine, but I just want to make sure I'm not compromising on security by doing so.
Any thoughts?
Thanks,
Rhys

Hope you don't mind the late reply. I was having the same issue and did some research.
According to the Java Docs of Google API the GoogleNetHttpTransport.newTrustedTransport() takes into account website certificates to use for Google APIs. However:
This helper method doesn't provide for customization of the NetHttpTransport, such as the ability to specify a proxy.
This confirms what you were saying about the certificates errors you are getting being caused by your proxy.
It is recommended, security wise, that instead of new NetHttpTransport() to use the NetHttpTransport.Builder in order to get a Trusted Transport for a specified proxy, as show on the link's example:
static HttpTransport newProxyTransport() throws GeneralSecurityException, IOException {
NetHttpTransport.Builder builder = new NetHttpTransport.Builder();
builder.trustCertificates(GoogleUtils.getCertificateTrustStore());
builder.setProxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress("127.0.0.1", 3128)));
return builder.build();
}

How to Request an External JSON URL From Within Java Servlet or Class When the JSON URL sits behind NTLM Authentication On Another Server/Domain

Accessing the following JSON URL from within the web browser is simple on a Windows machine as this pops up an authentication box asking for the username and password which when entered displays the JSON data correctly.
www.json-behind-ntlm-authentication.com/view-data
I am now trying to move this into a Java Servlet.
I have tested the HttpClient library, http://hc.apache.org, and every example I have tried from their documentation, doesn't work. Most of the code I've tried doesn't even compile correctly.
I have also tested Jsoup, https://jsoup.org/, as that is a very good library for web scraping, but this doesn't seem to support accessing pages behind NTLM authentication.
I have also tested the code found here, https://blogs.msdn.microsoft.com/freddyk/2010/01/19/connecting-to-nav-web-services-from-java/, which is the only code sample I can find related to accessing a JSON URL that sits behind NTLM authentication. This is actually what I'm looking to achieve, a Java web application accessing Microsoft Nav data through their web services - and even this official example doesn't compile.
Any pointers / options? There must be a Java library somewhere that has this problem solved? The access is currently over HTTP, but ultimately is going to be over SSL for security reasons, so any solution must also support SSL handshakes.
I would really like not to build a separate C# application using LINQ, https://blogs.msdn.microsoft.com/freddyk/2009/04/20/using-linq-with-nav-web-services/, which I would hope works, but I'm not hopeful that the C# example would work in this scenario based on the Java examples not compiling.
UPDATE
After an awful lot of searching, I've found the following code below which seems to be close to working, but not quite - See the comments in the code where this is breaking. Thanks for the pointers in the comments already.
DefaultHttpClient httpclient = new DefaultHttpClient();
List<String> authpref = new ArrayList<String>();
authpref.add(AuthPolicy.NTLM);
httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF, authpref); //ERROR - This causes an error: java.lang.VerifyError: Cannot inherit from final class
NTCredentials creds = new NTCredentials(username, password, "", domain);
httpclient.getCredentialsProvider().setCredentials(AuthScope.ANY, creds);
HttpHost target = new HttpHost(baseURL);
// Make sure the same context is used to execute logically related requests
HttpContext localContext = new BasicHttpContext();
// Execute a cheap method first. This will trigger NTLM authentication
HttpGet httpget = new HttpGet(baseURL);
HttpResponse response1 = httpclient.execute(target, httpget, localContext); //ERROR - This line is throwing an error: java.lang.VerifyError: Cannot inherit from final class
HttpEntity entity = response1.getEntity();
System.out.println(EntityUtils.toString(entity));
I'm still unsure how to actually solve this problem. Any additional pointers?

org.apache.http.auth has NTCredentials which you can use in a HttpComponentsMessageSender in a spring boot #Configuration
HttpComponentsMessageSender httpComponentsMessageSender = new HttpComponentsMessageSender();
NTCredentials credentials = new NTCredentials("username", "password", null, "domain");

Accessing kerberos secured WebHDFS without SPnego

I have a working application for managing HDFS using WebHDFS.
I need to be able to do this on a Kerberos secured cluster.
The problem is, that there is no library or extension to negotiate the ticket for my app, I only have a basic HTTP client.
Would it be possible to create a Java service which would handle the ticket exchange and once it gets the Service ticket to just pass it to the app for use in a HTTP request?
In other words, my app would ask the Java service to negotiate the tickets and it would return the Service ticket back to my app in a string or raw string and the app would just attach it to the HTTP request?
EDIT: Is there a similar elegant solution like #SamsonScharfrichter described for HTTPfs? (To my knowledge, it does not support delegation tokens)
EDIT2: Hi guys, I am still completly lost. Im trying to figure out the Hadoop-auth client without any luck. Could you please help me out again? I already spent hours reading upon it without luck.
The examples say to do this:
* // establishing an initial connection
*
* URL url = new URL("http://foo:8080/bar");
* AuthenticatedURL.Token token = new AuthenticatedURL.Token();
* AuthenticatedURL aUrl = new AuthenticatedURL();
* HttpURLConnection conn = new AuthenticatedURL(url, token).openConnection();
* ....
* // use the 'conn' instance
* ....
Im lost already here. What initial connection do I need? How can
new AuthenticatedURL(url, token).openConnection();
take two parameters? there is no constructor for such a case. (im getting error because of this). Shouldnt a principal be somewhere specified? It is probably not going to be this easy.
URL url = new URL("http://<host>:14000/webhdfs/v1/?op=liststatus");
AuthenticatedURL.Token token = new AuthenticatedURL.Token();
HttpURLConnection conn = new AuthenticatedURL(url, token).openConnection(url, token);

Using Java code plus the Hadoop Java API to open a Kerberized session, get the Delegation Token for the session, and pass that Token to the other app -- as suggested by #tellisnz -- has a drawback: the Java API requires quite a lot of dependencies (i.e. a lot of JARs, plus Hadoop native libraries). If you run you app on Windows, in particular, it will be a tough ride.
Another option is to use Java code plus WebHDFS to run a single SPNEGOed query and GET the Delegation Token, then pass it to the other app -- that option requires absolutely no Hadoop library on your server. The barebones version would be sthg like
URL urlGetToken = new URL("http://<host>:<port>/webhdfs/v1/?op=GETDELEGATIONTOKEN") ;
HttpURLConnection cnxGetToken =(HttpURLConnection) urlGetToken.openConnection() ;
BufferedReader httpMessage = new BufferedReader( new InputStreamReader(cnxGetToken.getInputStream()), 1024) ;
Pattern regexHasToken =Pattern.compile("urlString[\": ]+(.[^\" ]+)") ;
String httpMessageLine ;
while ( (httpMessageLine =httpMessage.readLine()) != null)
{ Matcher regexToken =regexHasToken.matcher(httpMessageLine) ;
if (regexToken.find())
{ System.out.println("Use that template: http://<Host>:<Port>/webhdfs/v1%AbsPath%?delegation=" +regexToken.group(1) +"&op=...") ; }
}
httpMessage.close() ;
That's what I use to access HDFS from a Windows Powershell script (or even an Excel macro). Caveat: with Windows you have to create your Kerberos TGT on the fly, by passing to the JVM a JAAS config pointing to the appropriate keytab file. But that caveat also applies to the Java API, anyway.

You could take a look at the hadoop-auth client and create a service which does the first connection, then you might be able to grab the 'Authorization' and 'X-Hadoop-Delegation-Token' headers and cookie from it and add it to your basic client's requests.
First you'll need to have either used kinit to authenticate your user for application before running. Otherwise, you're going to have to do a JAAS login for your user, this tutorial provides a pretty good overview on how to do that.
Then, to do the login to WebHDFS/HttpFS, we'll need to do something like:
URL url = new URL("http://youhost:8080/your-kerberised-resource");
AuthenticatedURL.Token token = new AuthenticatedURL.Token();
HttpURLConnection conn = new AuthenticatedURL().openConnection(url, token);
String authorizationTokenString = conn.getRequestProperty("Authorization");
String delegationToken = conn.getRequestProperty("X-Hadoop-Delegation-Token");
...
// do what you have to to get your basic client connection
...
myBasicClientConnection.setRequestProperty("Authorization", authorizationTokenString);
myBasicClientConnection.setRequestProperty("Cookie", "hadoop.auth=" + token.toString());
myBasicClientConnection.setRequestProperty("X-Hadoop-Delegation-Token", delegationToken);

Is there any way with apache httpclient to set credentials based of a URI

Currently I have some uris that I call using resteasy that That looks something like this:
http://host.com/api/project1/getsomestuff
and
http://host.com/api/project2/getsomestuff
Both use digest authentication but require a different username and password for authentication. Currently in code I have to handle this by creating a different client instance for each project like this:
DefaultHttpClient project1Client = new DefaultHttpClient();
Credentials project1Credentials = new UsernamePasswordCredentials("user1", "password1");
project1Client.getCredentialsProvider().setCredentials(AuthScope.ANY, project1Credentials);
ClientExecutor executor1 = new ApacheHttpClient4Executor(project1Client);
MyService project1Proxy = ProxyFactory.create(MyService.class, executor1);
project1Proxy.getSomeStuff("project1");
DefaultHttpClient project2Client = new DefaultHttpClient();
Credentials project2Credentials = new UsernamePasswordCredentials("user2", "password2");
project2Client.getCredentialsProvider().setCredentials(AuthScope.ANY, project1Credentials);
ClientExecutor executor2 = new ApacheHttpClient4Executor(project1Client);
MyService project2Proxy = ProxyFactory.create(MyService.class, executor2);
project2Proxy.getSomeStuff("project2");
I have looked through AuthScope and CredentialsProvider and I just can't see any way how this would be possible. Does anybody know of any way to use both sets of credentials using the same client or am I stuck using multiple clients?
Also, please note that I do not control the server side of this rest call and both calls use the same realm so I am unable to set the different credentials per realm.

If those applications have a different security contexts they are also likely to have different authentication realms (or at least they are expected to). One can explicitly set the realm attribute on the AuthScope object to explicitly match it to a specific authentication context with the given realm name.

I think you are out of luck. Looking through the docs you cannot create an Authscope which sends a different password to different paths on the same host and port.

Simple Kerberos client in Java?

Applications such a Google's Chrome and IE can transparently handle Kerberos authentication; however I can not find a "simple" Java solution to match this transparency. All of the solutions I have found require the presence of a krb5.conf file and a login.conf file which nether of the above apps seem to require.
What is the best way to build a Java application with Kerberos SSO capabilities that just work?
[update]: to be clear I need a CLIENT side solution for creating tickets not validating them. Also, it seems that SPNEGO is the default "wrapper" protocol that will eventually delegate to Kerberos but I need to be able to handle the SPNEGO protocol as well.

There is now a simple solution for this using the Apache HTTP Components Client 4.5 or greater. This is still marked as experimental in 4.5 so your milage may vary, but this is working fine for me in an enterprise context.
In addition to the HC 4.5 client jars you will need to have the httpclient-win, jna and jna-platform jars on your classpath, as provided with http-component-client. You then construct a Kerberos enabled HC-client as follows:
CloseableHttpClient httpclient = WinHttpClients.createDefault();
Or using the builder:
HttpClientBuilder clientBuilder = WinHttpClients.custom();
Which can then be customised as required before building the client:
CloseableHttpClient client = clientBuilder.build();
This solution works without any external configuration, and most importantly solves the issue where the in-built JRE mechanism breaks for users with local Admin rights on Windows 7+. This is possible because the Kerberos ticket is being retrieved directly from the SSPI API via JNA, rather than going through the GSSAPI provided by the JRE.
Example code from the http-components team
This was all made possible by the good work of Daniel Doubrovkine Timothy Wall
and Ryan McKinley

Adding to David Roussels answer on url specific http based kerberos authentication:-
The reason why your code works is because your target SPN(server side principal) is configured to with HTTP/serverhostname.realm.com#DOMAIN.COM. In that case it will work because you are not explicitly setting the token. URLConnection internally sets a token with that SPN
1 Perform steps(from my previous answer) to get a subject
2 Use gss api init sec context to generate a context token. There are numerous tutorials out there for this step
3 Base 64 encode the token
4 Attach the token to urlconnection:-
URL url = new URL("http://myhost/myapp")
HttpURLConnection urlConn = (HttpURLConnection)url.openConnection(); =
urlConn.setRequestProperty("Authorization", "Negotiate " + encodedToken);
5 Implement a priviledged action:-
//this internally calls the getInputStream
public class PrivilegedGetInputStream implements PrivilegedExceptionAction<InputStream>
6 Wrap the whole thing in Subject.doAs
//use prev answer instructions to get subject
Subject.doAs(subject, new PrivilegedGetInputStream(urlConnection)

Oracle has an example using Java's SaslClient. I'm not a Java programmer, but when I pointed this out once to someone who is, they were able to make it work pretty quickly. It may still require a "conf" file somewhere (n.b. Kerberos uses environment variables, often starting with KRB5_, to know where to look for such files). Also note that Kerberos itself does not include a transport of any kind--your app needs to know how to send and receive the Kerberos payloads the way the server expects (and this is different depending on the server you are trying to authenticate with).
Edit: you edited your question, so here's a link related to SPNEGO in Java which might be of some use:
http://download.oracle.com/javase/6/docs/technotes/guides/security/jgss/lab/part5.html

You don't actually need to do anything. In Java 6, on a Windows client machine you can do this:
new URL("http://myhost/myapp").openStream();
And negotiate authentication just works. At least it does for me. And the server I tested on only supports Negotiate, not NTLM auth.

Ok if you want to avoid using a login.conf file you need to code differently:-
//define your own configuration
import javax.security.auth.login.Configuration;
public class CustomLoginConfiguration extends Configuration
//pass certain parameters to its constructor
//define an config entry
import javax.security.auth.login.AppConfigurationEntry;
private AppConfigurationEntry configEntry;
//define a map of params you wish to pass and fill them up
//the map contains entries similar to one you have in login.conf
Map<String, String> params = new HashMap<String, String>();
//define the configuration
configEntry = new AppConfigurationEntry(
"com.sun.security.auth.module.Krb5LoginModule",
AppConfigurationEntry.LoginModuleControlFlag.REQUIRED, params);
//implement getappconfig method
public AppConfigurationEntry[] getAppConfigurationEntry() {
return new AppConfigurationEntry[] { configEntry };
}
Now once you are done with this definition you can use this in you use this to fetch tickets from kdc
//get ticket in login context
LoginContext lc = null;
lc = new LoginContext("lc", null, callback, new CustomLoginConfiguration(argumentlist));
lc.login();
Now from here on you can fetch jaas subject and can basically do a ton of authentication stuff.
In case you need further pointers just leave a comment.

You can use system properties instead of config files to specify the KDC hostname and service name, but those things (at least) are mandatory....
Waffle will actually give you the information you need to set most of the properties, even if it won't get you a ticket. Look at the WindowsAuthProviderImpl class (the Waffle.chm help file shows the API).
I use JAAS do obtain a service ticket from Active Directory in two steps:
Use Krb5LoginModule to retrieve the cached TGT and add it to the Subject.
Use the Subject and GSS-API to retrieve a service ticket from the KDC.
There's a lot of good information and example code at The Java Way of Active Directory.

I created a small tool to simplify connecting with httpclient to kerberos, you might want to give it a try.
https://github.com/DovAmir/httpclientAuthHelper
DefaultHttpClient httpclient = new DefaultHttpClient();
AuthUtils.securityLogging(SecurityLogType.KERBEROS,true);
CredentialsUtils.setKerberosCredentials(client, new UsernamePasswordCredentials("xxx", "xxx"), "domain", "kdc");
client.executeMethod(httpget);

Use WAFFLE

Here's a good blog post on having a java client to use with Kerberos
http://sachithdhanushka.blogspot.com/2014/02/kerberos-java-client-configuration.html