We Keep Coding

Accessing kerberos secured WebHDFS without SPnego

I have a working application for managing HDFS using WebHDFS.
I need to be able to do this on a Kerberos secured cluster.
The problem is, that there is no library or extension to negotiate the ticket for my app, I only have a basic HTTP client.
Would it be possible to create a Java service which would handle the ticket exchange and once it gets the Service ticket to just pass it to the app for use in a HTTP request?
In other words, my app would ask the Java service to negotiate the tickets and it would return the Service ticket back to my app in a string or raw string and the app would just attach it to the HTTP request?
EDIT: Is there a similar elegant solution like #SamsonScharfrichter described for HTTPfs? (To my knowledge, it does not support delegation tokens)
EDIT2: Hi guys, I am still completly lost. Im trying to figure out the Hadoop-auth client without any luck. Could you please help me out again? I already spent hours reading upon it without luck.
The examples say to do this:
* // establishing an initial connection
*
* URL url = new URL("http://foo:8080/bar");
* AuthenticatedURL.Token token = new AuthenticatedURL.Token();
* AuthenticatedURL aUrl = new AuthenticatedURL();
* HttpURLConnection conn = new AuthenticatedURL(url, token).openConnection();
* ....
* // use the 'conn' instance
* ....
Im lost already here. What initial connection do I need? How can
new AuthenticatedURL(url, token).openConnection();
take two parameters? there is no constructor for such a case. (im getting error because of this). Shouldnt a principal be somewhere specified? It is probably not going to be this easy.
URL url = new URL("http://<host>:14000/webhdfs/v1/?op=liststatus");
AuthenticatedURL.Token token = new AuthenticatedURL.Token();
HttpURLConnection conn = new AuthenticatedURL(url, token).openConnection(url, token);

Using Java code plus the Hadoop Java API to open a Kerberized session, get the Delegation Token for the session, and pass that Token to the other app -- as suggested by #tellisnz -- has a drawback: the Java API requires quite a lot of dependencies (i.e. a lot of JARs, plus Hadoop native libraries). If you run you app on Windows, in particular, it will be a tough ride.
Another option is to use Java code plus WebHDFS to run a single SPNEGOed query and GET the Delegation Token, then pass it to the other app -- that option requires absolutely no Hadoop library on your server. The barebones version would be sthg like
URL urlGetToken = new URL("http://<host>:<port>/webhdfs/v1/?op=GETDELEGATIONTOKEN") ;
HttpURLConnection cnxGetToken =(HttpURLConnection) urlGetToken.openConnection() ;
BufferedReader httpMessage = new BufferedReader( new InputStreamReader(cnxGetToken.getInputStream()), 1024) ;
Pattern regexHasToken =Pattern.compile("urlString[\": ]+(.[^\" ]+)") ;
String httpMessageLine ;
while ( (httpMessageLine =httpMessage.readLine()) != null)
{ Matcher regexToken =regexHasToken.matcher(httpMessageLine) ;
if (regexToken.find())
{ System.out.println("Use that template: http://<Host>:<Port>/webhdfs/v1%AbsPath%?delegation=" +regexToken.group(1) +"&op=...") ; }
}
httpMessage.close() ;
That's what I use to access HDFS from a Windows Powershell script (or even an Excel macro). Caveat: with Windows you have to create your Kerberos TGT on the fly, by passing to the JVM a JAAS config pointing to the appropriate keytab file. But that caveat also applies to the Java API, anyway.

You could take a look at the hadoop-auth client and create a service which does the first connection, then you might be able to grab the 'Authorization' and 'X-Hadoop-Delegation-Token' headers and cookie from it and add it to your basic client's requests.
First you'll need to have either used kinit to authenticate your user for application before running. Otherwise, you're going to have to do a JAAS login for your user, this tutorial provides a pretty good overview on how to do that.
Then, to do the login to WebHDFS/HttpFS, we'll need to do something like:
URL url = new URL("http://youhost:8080/your-kerberised-resource");
AuthenticatedURL.Token token = new AuthenticatedURL.Token();
HttpURLConnection conn = new AuthenticatedURL().openConnection(url, token);
String authorizationTokenString = conn.getRequestProperty("Authorization");
String delegationToken = conn.getRequestProperty("X-Hadoop-Delegation-Token");
...
// do what you have to to get your basic client connection
...
myBasicClientConnection.setRequestProperty("Authorization", authorizationTokenString);
myBasicClientConnection.setRequestProperty("Cookie", "hadoop.auth=" + token.toString());
myBasicClientConnection.setRequestProperty("X-Hadoop-Delegation-Token", delegationToken);

Accessing OneNote API from Java

I am interested in writing a Java application that can access my OneNote notebooks via the OneNote API. I am not sure how to gain access to that API from within Java. Can anybody point me to an example of how to get started here? I use Eclipse as my development environment.

This as straightforward process.
The 3 steps would be:
1) create a OneNote application on the OneNote developper's page. More info here https://dev.onedrive.com/app-registration.htm. This is a one time action.
2) your java application should then provide an authentification mechanism and a tolken-refresh mechanism.
See this post for more info on the authentification mechanism part : Getting a OneNote token with Java. This post is about the OAuth 2.0 flow 'Authorization code grant flow'. More info here https://msdn.microsoft.com/en-us/library/hh243647.aspx#flows
3) your java application calls adhoc API Rest methods to retreive the needed informations.
Example to retrieve all your notebooks (using OkHttp for Http requests):
private final static String NOTEBOOKS_ENDPOINT = "https://www.onenote.com/api/v1.0/me/notes/notebooks";
public Notebooks readAllNoteBooks() {
try {
if (client == null)
client = new OkHttpClient();
Request request = createOneNoteRequest(a_valid_tolken, NOTEBOOKS_ENDPOINT);
Response response = client.newCall(request).execute();
JsonObject content = UrlHelper.parseResponse(response);
System.out.println(content);
return Notebooks.build(content.get("value"));
} catch (Exception ex) {
ex.printStackTrace();
}
return null;
}
public static Request createOneNoteRequest(String mAccessToken, String url) {
Request.Builder reqBuilder = new Request.Builder();
reqBuilder.url(url);
reqBuilder.header("Authorization", "Bearer " + mAccessToken);
return reqBuilder.build();
}
NoteBooks and NoteBook are 2 tiny classes matching the key attributes from the OneNote objects.

Microsoft has provided REST apis for accessing One note functionalities like creating and accessing notes. See OneNote Rest API reference.

singrass,
In addition to the above replies, the Android OneNote API sample may also help you. There is no OneNote application class that you can create (unless you want to create your own). You simply call the API through the HttpClient. If you are unfamiliar on how to call REST APIs in Java in general, this thread may help you.
-- James

Iterating through SOAP requests from a properties file

I am writing a java plugin that I plan to use to test a number of web services. These SOAPs for the web services are located in a properties file, and are grouped under which WSDL they fall under (Subscriber, Network, User, etc...). Also, there are some regexs associated with each web service to test the response against.
Properties Example
#Web services to be tested and regexes to test responses
#List of web service groups used (WSDLs)
webservice.list = SubscriberMgmt,NetworkMgmt
# < -- SubscriberMgmt -- >
#getSubscriberDevices
webservice.subscriber = <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ws="http://ws.blah.blah.com"><soapenv:Header/><soapenv:Body><ws:getSubscriberDevices><PhoneNumber></PhoneNumber><LastName></LastName><MACAddress></MACAddress><ExternalId></ExternalId><AccountExternalId>john</AccountExternalId><IPAddress></IPAddress></ws:getSubscriberDevices></soapenv:Body></soapenv:Envelope>
webservice.SubscriberMgmt.regex = subscriberId="(.+?)"
webservice.SubscriberMgmt.regex.1 = externalId="(.+?)"
#getMpegResultsById
webservice.subscriber.1 = <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ws="http://ws.blah.blah.com"><soapenv:Header/><soapenv:Body><ws:getMpegResultsById><SubscriberId>100016</SubscriberId><Duration>2880</Duration></ws:getMpegResultsById></soapenv:Body></soapenv:Envelope>
webservice.SubscriberMgmt.1.regex = id="(.+?)"
webservice.SubscriberMgmt.1.regex.1 = externalId="(.+?)"
I currently have code to connect using each WSDL from the properties file, so say when the 'webservicegroup' variable is SubscriberMgmt, I'd like to test the .subscriber web service(s) and check the responses if it contains the corresponding regex(es). (the 'data' variable only corresponds to one SOAP request from the property file at the moment)
//Soap Request
try
{
for(String webservicegroup : webserviceList)
{
URL url = new URL("http://" + server + "/webservices/" + webservicegroup);
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
conn.setRequestProperty("Content-type", "text/xml; charset=utf-8");
conn.setRequestProperty("SOAPAction", "\"\"");
String loginEnc = new BASE64Encoder().encodeBuffer((username + ":" + password).getBytes());
loginEnc = loginEnc.replaceAll("\n", "");
conn.setRequestProperty("Authorization", "Basic " + loginEnc);
conn.setConnectTimeout(timeout);
conn.setReadTimeout(timeout);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
//Send request
wr.write(data);
wr.flush();
wr.close();
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
//Save response
String line;
while ((line = in.readLine()) != null)
{
response += line;
}
in.close();
}
}
Any ideas on the best way of doing this? Any help is greatly appreciated

Assuming your connection and POST/GET code is working:
Step 1: Get the entire response as a string:
String response = new String(ByteStreams.toByteArray(inputStream), "UTF-8");
In the above line of code, the ByteStreams class is part of google's guava library. Similar code can be found in apache commons-io, if you prefer that.
Step 2: Test Regular expressions:
if( response.matches(regex) ) { ... }

Don't reinvent the wheel, building a custom testing software mini empire from scratch.
Use SOAPUI open source version to test web services. It allows you to:
generate SOAP tests from web service endpoint WSDL, saving manual labour
avoid processing "pre-canned" SOAP strings and parsing files
automatically generate mocks of your service endpoints, including ability to programmatically control custom responses.
implement test step logic including loops, branches, invoking other test steps, running scripts, and reading/writing parameters from/to property files and data sources (although you must programatically navigate & modify XmlHolder instances via scripts if you wish to "data-drive" your tests)
Execute SOAP, REST, security & load testing of web services, and also JDBC and HTTP test calls.
Integrates with the common build and continuous integration tools for test automation (TDD & continuous delivery).
Use SOAPUI operations within your IDE, via plugins.
It's considered fairly standard "best practice" for testing web services.
To checks SOAP response messages for valid content, using the open source version of SOAPUI:
you can use XPath or XQuery expressions to validate the XML.
you can use script assertions
E.g. if your SOAP response is:
<soap:Body>
<GetEnumResponse xmlns="http://www.xyz.com/">
<GetEnumResult>
<ErrorCode>0</ErrorCode>
<StatusId>0</StatusId>
</GetEnumResult>
<enumsInformation>
<EnumInformation>
<TransactionId>0</TransactionId>
<ConstraintId>5000006</ConstraintId>
<EnumValue>xyz</EnumValue>
<Index>10</Index>
</EnumInformation>
</enumsInformation>
</GetEnumResponse>
</soap:Body>
You can script:
import com.eviware.soapui.support.XmlHolder
def holder = new XmlHolder(messageExchange.responseContentAsXml)
holder.namespaces["tal"]="http://www.xyz.com/"
def node = holder.getNodeValue("//tal:ConstraintId[1]");
log.info(node);
assert node == "5000006";
You can even use the maximum power of standard java regex processing.
Create java classes that do the regex processing and put them into a jar file and place in soapUIinstallation/bin/ext as explained here.
Or wrap your SOAPUI Test inside a JUnit test method, and add standard java code at end to check regexs. This also eases test automation & allows any non-web service tests to be executed as well. This approach works with SOAPUI open source version, whereas the alternative of using SOAPUI assertion steps requires the Pro version.
Steps:
If you choose, install the SOAPUI plugin in your IDE
Use SOAPUI to create a test suite
Use SOAPUI to create test case(s) within the test suite
Use SOAPUI to create test step(s) within the test suite. This is the core of using SOAPUI.
Create a Java project in your IDE. Within this project, add a JUnit test case.
Add all JARs from SoapUI bin and lib directories to Java Build Path.
Within the Junit Test case, add code to execute a SOAPUI test step
Obtain the MessageExchange object, get the response from it, and then get headers, content or attachments. Run a regex check on result.
The following is indicative only. Not intended to be a working example.
package com.example;
import org.junit.Test;
import com.eviware.soapui.tools.SoapUITestCaseRunner;
public class SoapUIProject {
// runs an entire SOAPUI test suite
#Test
public void soapTest1() throws Exception {
SoapUITestCaseRunner runner = new SoapUITestCaseRunner();
runner.setProjectFile("/path/to/your/W3Schools-Tutorial-soapui-project.xml");
runner.run();
}
// runs a single SOAPUI test step - and checks response matches a regex
#Test
public void soapTest2() throws Exception {
WsdlProject project = new WsdlProject( "src/dist/sample-soapui-project.xml" );
TestSuite testSuite = project.getTestSuiteByName("My Test Suite");
TestCase testCase = testSuite.getTestCaseByName("My Test Case");
TestCaseRunner testCaseRunner = new WsdlTestCaseRunner(testCase,null);
// Must have test step setup as WsdlMessageExchange for cast to work
WsdlMessageExchangeTestStepResult testStepResult = (WsdlMessageExchangeTestStepResult)testStep.runTestStepByName("My Test Step");
// TestStep testStep = testCase.getTestStepByName("My Test Step");
// TestCaseRunContext testCaseRunContext = new WsdlTestRunContext(testStep);
// testStep.prepare(testCaseRunner, testCaseRunContext);
// WsdlMessageExchangeTestStepResult testStepResult = (WsdlMessageExchangeTestStepResult)testStep.run(testCaseRunner, testCaseRunContext);
MessageExchange[] messageExchanges = testStepResult.getMessageExchanges();
for (MessageExchange me : messageExchanges) {
String response = me.getResponseContentAsXML();
// do any desired regex processing
// can use any desired assertions
}
assertEquals( Status.FINISHED, runner.getStatus() );
}
}
Further refs: http://www.soapui.org/Scripting-Properties/tips-a-tricks.html#3-xml-nodes
http://books.google.com.au/books?id=DkWx7xZ263gC&printsec=frontcover#v=onepage&q&f=false

Can web service testing done using SOAPUI tool be done in Java completely?

In SOAP UI Web service testing,
User imports the Project in to the work space and mentions the End point. Enters the required data in the Request xml and runs to get the resulting response in xml format.
Is there a way we can achieve this using Java only without using the SoapUI tool. I guess the steps should be:
Create a Wsdl Project.
Create a xml request (in the required format)
Send the request to the end point (How to do this?)
Receive response and verify it.
Please help me how to do this using Java only(without using SOAP UI tool). Any links/code will be greatly helpful.
Thanks,
Mike

Use soapUI API.
;
Here are some useful links:
http://www.soapui.org/Developers-Corner/integrating-with-soapui.html
http://pritikaur23.wordpress.com/2013/06/16/saving-a-soapui-project-and-sending-requests-using-soapui-api/
I used the following code to create a project:
File projectFile = new File(filePath);
SoapUI.setSoapUICore(new StandaloneSoapUICore(true));
WsdlProject project = new WsdlProject();
project.setName(projectName);
WsdlInterface[] wsdls = WsdlImporter.importWsdl(project, url);
for (WsdlInterface wsdl : wsdls) {
int c = wsdl.getOperationCount();
String reqContent = "";
for (int j = 0; j < c; j++) {
WsdlOperation op = wsdl.getOperationAt(j);
reqContent = op.createRequest(true);
WsdlRequest req = op.addNewRequest(requestName);
req.setRequestContent(reqContent );
}
}
project.saveIn(projectFile);
SoapUI.shutdown();

You can create client and pass in HTTP Request test request populated with needed parameter for testing purpose, below mention question has some useful insights.
Java Web service testing

Calling a Web Service from Seam

A simple question, but could someone provide sample code as to how would someone call a web service from within the JBoss Seam framework, and process the results?
I need to be able to integrate with a search platform being provided by a private vendor who is exposing his functionality as a web service. So, I'm just looking for some guidance as to what the code for calling a given web service would look like.
(Any sample web service can be chosen as an example.)

There's roughly a gajillion HTTP client libraries (Restlet is quite a bit more than that, but I already had that code snippet for something else), but they should all provide support for sending GET requests. Here's a rather less featureful snippet that uses HttpClient from Apache Commons:
HttpClient client = new HttpClient();
HttpMethod method = new GetMethod("http://api.search.yahoo.com/WebSearchService/V1/webSearch?appid=restbook&query=HttpClient");
client.executeMethod(method);

import org.restlet.Client;
import org.restlet.data.Protocol;
import org.restlet.data.Reference;
import org.restlet.data.Response;
import org.restlet.resource.DomRepresentation;
import org.w3c.dom.Node;
/**
* Uses YAHOO!'s RESTful web service with XML.
*/
public class YahooSearch {
private static final String BASE_URI = "http://api.search.yahoo.com/WebSearchService/V1/webSearch";
public static void main(final String[] args) {
if (1 != args.length) {
System.err.println("You need to pass a search term!");
} else {
final String term = Reference.encode(args[0]);
final String uri = BASE_URI + "?appid=restbook&query=" + term;
final Response response = new Client(Protocol.HTTP).get(uri);
final DomRepresentation document = response.getEntityAsDom();
document.setNamespaceAware(true);
document.putNamespace("y", "urn:yahoo:srch");
final String expr = "/y:ResultSet/y:Result/y:Title/text()";
for (final Node node : document.getNodes(expr)) {
System.out.println(node.getTextContent());
}
}
}
}
This code uses Restlet to make a request to Yahoo's RESTful search service. Obviously, the details of the web service you are using will dictate what your client for it looks like.

final Response response = new Client(Protocol.HTTP).get(uri);
So, if I understand this correctly, the above line is where the actual call to the web service is being made, with the response being converted to an appropriate format and manipulated after this line.
Assuming I were not using Restlet, how would this line differ?
(Of course, the actual processing code would be significantly different as well, so that's a given.)