We Keep Coding

Accessing kerberos secured WebHDFS without SPnego

I have a working application for managing HDFS using WebHDFS.
I need to be able to do this on a Kerberos secured cluster.
The problem is, that there is no library or extension to negotiate the ticket for my app, I only have a basic HTTP client.
Would it be possible to create a Java service which would handle the ticket exchange and once it gets the Service ticket to just pass it to the app for use in a HTTP request?
In other words, my app would ask the Java service to negotiate the tickets and it would return the Service ticket back to my app in a string or raw string and the app would just attach it to the HTTP request?
EDIT: Is there a similar elegant solution like #SamsonScharfrichter described for HTTPfs? (To my knowledge, it does not support delegation tokens)
EDIT2: Hi guys, I am still completly lost. Im trying to figure out the Hadoop-auth client without any luck. Could you please help me out again? I already spent hours reading upon it without luck.
The examples say to do this:
* // establishing an initial connection
*
* URL url = new URL("http://foo:8080/bar");
* AuthenticatedURL.Token token = new AuthenticatedURL.Token();
* AuthenticatedURL aUrl = new AuthenticatedURL();
* HttpURLConnection conn = new AuthenticatedURL(url, token).openConnection();
* ....
* // use the 'conn' instance
* ....
Im lost already here. What initial connection do I need? How can
new AuthenticatedURL(url, token).openConnection();
take two parameters? there is no constructor for such a case. (im getting error because of this). Shouldnt a principal be somewhere specified? It is probably not going to be this easy.
URL url = new URL("http://<host>:14000/webhdfs/v1/?op=liststatus");
AuthenticatedURL.Token token = new AuthenticatedURL.Token();
HttpURLConnection conn = new AuthenticatedURL(url, token).openConnection(url, token);

Using Java code plus the Hadoop Java API to open a Kerberized session, get the Delegation Token for the session, and pass that Token to the other app -- as suggested by #tellisnz -- has a drawback: the Java API requires quite a lot of dependencies (i.e. a lot of JARs, plus Hadoop native libraries). If you run you app on Windows, in particular, it will be a tough ride.
Another option is to use Java code plus WebHDFS to run a single SPNEGOed query and GET the Delegation Token, then pass it to the other app -- that option requires absolutely no Hadoop library on your server. The barebones version would be sthg like
URL urlGetToken = new URL("http://<host>:<port>/webhdfs/v1/?op=GETDELEGATIONTOKEN") ;
HttpURLConnection cnxGetToken =(HttpURLConnection) urlGetToken.openConnection() ;
BufferedReader httpMessage = new BufferedReader( new InputStreamReader(cnxGetToken.getInputStream()), 1024) ;
Pattern regexHasToken =Pattern.compile("urlString[\": ]+(.[^\" ]+)") ;
String httpMessageLine ;
while ( (httpMessageLine =httpMessage.readLine()) != null)
{ Matcher regexToken =regexHasToken.matcher(httpMessageLine) ;
if (regexToken.find())
{ System.out.println("Use that template: http://<Host>:<Port>/webhdfs/v1%AbsPath%?delegation=" +regexToken.group(1) +"&op=...") ; }
}
httpMessage.close() ;
That's what I use to access HDFS from a Windows Powershell script (or even an Excel macro). Caveat: with Windows you have to create your Kerberos TGT on the fly, by passing to the JVM a JAAS config pointing to the appropriate keytab file. But that caveat also applies to the Java API, anyway.

You could take a look at the hadoop-auth client and create a service which does the first connection, then you might be able to grab the 'Authorization' and 'X-Hadoop-Delegation-Token' headers and cookie from it and add it to your basic client's requests.
First you'll need to have either used kinit to authenticate your user for application before running. Otherwise, you're going to have to do a JAAS login for your user, this tutorial provides a pretty good overview on how to do that.
Then, to do the login to WebHDFS/HttpFS, we'll need to do something like:
URL url = new URL("http://youhost:8080/your-kerberised-resource");
AuthenticatedURL.Token token = new AuthenticatedURL.Token();
HttpURLConnection conn = new AuthenticatedURL().openConnection(url, token);
String authorizationTokenString = conn.getRequestProperty("Authorization");
String delegationToken = conn.getRequestProperty("X-Hadoop-Delegation-Token");
...
// do what you have to to get your basic client connection
...
myBasicClientConnection.setRequestProperty("Authorization", authorizationTokenString);
myBasicClientConnection.setRequestProperty("Cookie", "hadoop.auth=" + token.toString());
myBasicClientConnection.setRequestProperty("X-Hadoop-Delegation-Token", delegationToken);

Java HttpURLConnection class Program

I am learning Java through use of a textbook, which contains the following code describing the use of a HttpURLConnection ...
class HttpURLDemo {
public static void main(String args[]) throws Exception {
URL hp = new URL("http://www.google.com");
HttpURLConnection hpCon = (HttpURLConnection) hp.openConnection();
// Display request method.
System.out.println("Request method is " + hpCon.getRequestMethod());
}
}
Could someone please explain why the hpCon object is declared in the following way...
HttpURLConnection hpCon = (HttpURLConnection) hp.openConnection();
instead of declaring it like this...
HttpURLConnection hpCon = new HttpURLConnection();
The textbook author provided the following explanation, which I don't really understand...
Java provides a subclass of URLConnection that provides support for HTTP connections.
This class is called HttpURLConnection. You obtain an HttpURLConnection in the same
way just shown, by calling openConnection( ) on a URL object, but you must cast the result
to HttpURLConnection. (Of course, you must make sure that you are actually opening an
HTTP connection.) Once you have obtained a reference to an HttpURLConnection object,
you can use any of the methods inherited from URLConnection

The declaration that you don't understand why not to use:
HttpURLConnection hpCon = new HttpURLConnection();
Does not provide information about the URL to which you want to open the connection. This is the reason why you should use:
HttpURLConnection hpCon = new HttpURLConnection(hp);
Because this way the constructor knows that you want to open a connection to the url "http://www.google.com".

java.net.URLConnection is an abstract class that facilitates in communication with various types of servers via various protocols (ftp http etc).
The protocol specific subclasses are hidden inside SUN's packages and these hidden classes are responsible for the concrete implementation of the protocols.
In your example since your URL is a http://www.google.com by parsing the URL the internals of the URL class knows that an HTTP handler/subclass must be used.
So when you open a connection to the server hp.openConnection(); you get a concrete instance of a class that implements the HTTP protocol.
That class is an instance of HttpURLConnection (actually a subclass since HTTPURLConnection is also abstract and that is why you can do:
HttpURLConnection hpCon = (HttpURLConnection) hp.openConnection(); and not get class cast exception.
So with Java's design you can't do HttpURLConnection hpCon = new HttpURLConnection(hp); as you ask, since that is not how the designers want you to use these APIs.
You are expected to work arround URLs and URLConnections and only worry about input/output.
You shouldn't worry about the rest

java http authentication reuse between servlets

not sure how to formulate the question correctly.
i've got two web-apps: A with a servlet and B with two servlets, both protected by basic authentication in web.xml (deployed to weblogic server).
a user authenticates to A and to one of B's servlets (not sure if what i say here is total rubbish) using browser-native login/password window (that's managed by weblogic).
however, the servlet of application A should as well call the other of B's servlets and that requires authentication also.
the question is: can it be avoided? the user has already authenticated to both of web-apps, so i'd like to just somehow reuse this authentication (i'm really not good at all these http session things terminology, don't throw rocks at me :)).
cookies can't be used it seems, as it is really server-side communication.

following should fix the problem:
create javax.servlet.Filter that extracts jsessionid from requests and stores it per user somehow (in my case i put it to org.springframework.security.core.Authentication as details, making sure this filter runs after spring-security's one).
add it to each servlet-servlet request as cookie header:
org.apache.commons.httpclient.HttpMethodBase.setRequestHeader("Cookie",
"jsessionid="+org.springframework.security.core.Authentication.getDetails().toString());
where appropriate.

Do this in your servlet code in app A:
URLConnection conn = new URL("http://hostname/appB/servlet1");
String authorizationHeader = request.getHeader("authorization");
if (null != authorizationHeader) {
conn.setRequestProperty("authorization", authorizationHeader);
}
InputStream inStream = conn.getInputStream();
//Read from inStream in the usual way

Multiple Http requests and responses using HttpURLConnection

I am developing a very simple http bot. I am using the javax.net.ssl.HttpsURLConnection class and I have to make multiple requests.
Snippet of code :
HttpURLConnection urlConnection =
(HttpURLConnection) new URL(url+"?"+firstParameters).openConnection();
urlConnection.setRequestProperty("Accept-Charset", "UTF-8");
headerFields = urlConnection.getHeaderFields();
keys = headerFields.keySet();
for(String key : keys){
if(key != null && key.contains("ookie")){
cookies = urlConnection.getHeaderField(key);
break;
}
}
for(String cookie : cookies.split(";")){
if(cookie.contains("JSESSION")){
JSESSION = cookie.split("=")[1];
break;
}
}
document = new InputSource(urlConnection.getInputStream());
parser.setDocument(document);
attributesId.put("name",new ArrayList<String>(Arrays.asList(attributesNames)));
elementsIds.put("INPUT",attributesId);
elements = parser.getValues(elementsIds);
for(String attr : attributesNames){
secondParameters = secondParameters.replaceAll("#r"+index,elements.get(attr));
}
urlConnection.getInputStream().close();
//Second call
urlConnection = (HttpURLConnection) new URL(url2).openConnection();
urlConnection.setRequestMethod("POST");
urlConnection.setRequestProperty("Cookie", "JSESSIONID="+JSESSION);
urlConnection.setDoInput(true);
urlConnection.setDoOutput(true);
payload = new PrintWriter(urlConnection.getOutputStream());
payload.print(secondParameters);
payload.flush();
payload.close();
Summarizing the code above, first i do a request without any payload and i am able to see the correct response from the server, but the problem is when i make the second request (now with payload and with the JSESSION cookie), what i receive it his the same response that i received in the first request, it looks like i am making the first request again.
So my question is , what i am doing wrong ?
I just need to open one connection, and then change the headers and payload ?
There is any tutorial related with multiple http requests(with mixed methods , post and get)?
Thanks in advance

I've never used HttpURLConnection before. I usually use Apache's HTTPClient code. There are a lot of docs and tutorials about it on their home page.
Couple of things that I noticed about your code:
You code does not handle multiple Cookie headers on the response. Mine seems to handle that better.
Are you sure that all you need is JSESSION? Maybe there are other cookies you are missing?
Have you debugged your code to make sure that your JSESSION cookie gets set appropriately? I added some trim() calls in my cookie processing code to make sure some spaces didn't slip in there.
I can't see the real value of your secondParameters. I have no idea if they are valid. Have you debugged your code to verify the secondParamters value looks good. You can see in my code what I'm posting to the server. Btw, I'd use a StringBuilder instead of + to build them.
Hope this helps.

Using java class HttpsURLConnection

I have a small piece of code which basically impements a HTTP-Client,
i.e. it POSTS request and works with re RESPONSE. As long as HTTP is
concenerned everthing work well. For some reason I now have to support
HTTPS too. So here is briefly what I do in order to get a connection opened:
URL url = new URL(serverAddress);
HttpsURLConnection httpsConn = (HttpsURLConnection) url.openConnection();
This fails, stating:
sun.net.www.protocol.https.HttpsURLConnectionImpl cannot be cast to com.sun.net.ssl.HttpsURLConnection
I guess this is kinda trivial, but I just don't get what I'm doing wrong in this one...
Googled it, and the code just looks right - not?
any ideas are appreciated!

Just keep it java.net.URLConnection or cast it to java.net.HttpURLConnection instead. Both offers methods to do the desired task as good.
A side remark unrelated to the technical problem: you should never explicitly import/use Sun Java SE implementation specific classes in your code. Those are undocumented classes and are subject to changes which may cause your code break when you upgrade the JVM. On the other hand, your code may also break when you run it at a different brand JVM.
Update: since you seem to accidentally have imported it, go to Window > Preferences > Java > Appearance > Type Filters and Add com.sun.* and sun.* to the list. This way you won't ever import them accidentally:

Your url's protocol should also be https and not http. Check your url.

The above problem is only caused by two issues
Using wrong import
Using http in the string you create url from use instead https

Instead of creating a URL object using standard constructor like
URL wsURL = new URL(url);
Use
java.net.URL wsURL = new URL(null, url,new sun.net.www.protocol.https.Handler());
which would solve this problem

Check your imports, you should be using
java.net.HttpURLConnection
or
javax.net.ssl.HttpsURLConnection

Check value of your "serverAddress" variable. It should https and not http

Change:
import javax.net.ssl.HttpsURLConnection;
and
URL url = new URL(serverAddress);
HttpsURLConnection httpsConn = (HttpsURLConnection) url.openConnection();
Change To:
import java.net.HttpURLConnection;
and
URL url = new URL(serverAddress);
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();

Hard to tell without seeing the whole file, but it looks like you're importing com.sun.net.ssl.HttpsURLConnection when you really want javax.net.ssl.HttpsURLConnection.

In my case, the protocol and port were not correct while invoking the httpsUrlConnection.
Port and protocol were defined as static class variables. And the step prior to the failed step, was invoking an httpUrlConnection. That method changed the port/protocol to 80/http, but didn't set it back to /https at the end. So eventhough httpsUrlConnection was invoked, it was still using http/80. Once I reset those at the end of the httpUrlConnection method, the error disappeared.