We Keep Coding

Enriching headers via a proxy using java and Selenium

I'm working on an automated web test stack using Selenium, Java and testNG.
For authentication and safety reasons, I need to enrich the headers of the website I am accessing through Kubernetes.
For example, I can successfully use this CURL command in terminal to retrieve the page I want to access: curl -H 'Host: staging.myapp.com' -H 'X-Forwarded-Proto: https' http://nginx.myapp.svc.cluster.local.
So as you can see, I only need to add 2 headers for Host and X-Forwarded-Proto.
I'm trying to create a proxy that will enrich headers in my #BeforeMethod method for a couple of days, but I'm still stuck, and there are so many shadow areas that I can't find a way to debug anything and understand what's wrong. For now, no matter my code, I keep getting a "No internet" (ERR_PROXY_CONNECTION_FAILED) error page in my driver when I launch it.
For example, one version of my code:
BrowserMobProxy browserMobProxy = new BrowserMobProxyServer();
browserMobProxy.setTrustAllServers(true);
browserMobProxy.addHeader("Host", "staging.myapp.com");
browserMobProxy.addHeader("X-Forwarded-Proto", "https");
browserMobProxy.start();
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.setProxy(ClientUtil.createSeleniumProxy(browserMobProxy));
driver = new ChromeDriver(chromeOptions);
driver.get("http://nginx.myapp.svc.cluster.local");
I tried several other code structures like:
defining browserMobProxyServer.addRequestFilter to add headers in requests
using only org.openqa.selenium.Proxy
setting up proxy with setHttpProxy("http://nginx.myapp.svc.cluster.local:8888");
But nothing works, I always get ERR_PROXY_CONNECTION_FAILED.
Anybody have any clue about that? Thanks!

OK so, after days of research, I found out 2 things:
Due to whatever configuration on my Mac, I need to force host Address (other people running the same code had no issue...):
proxy.setHttpProxy(Inet4Address.getLocalHost().getHostAddress() + ":" + browserMobProxyServer.getPort());
I have to manually alter headers via a response filter instead of using .addHeader method:
browserMobProxyServer.addResponseFilter((response, content, messageInfo)->{
//Do something here related to response.headers()
});
I hope it will help some lost souls here.

Java Web service client can't connect to BPEL webservice when SoapUI can

I am experiencing the following error and after extensive research on the matter with a few similar issues, none seem to help.
My error:
com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: java.net.ConnectException: Connection refused: connect
at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:121)
at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:142)
at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:83)
at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:105)
at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Fiber.java:587)
at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Fiber.java:546)
at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Fiber.java:531)
at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Fiber.java:428)
at com.sun.xml.internal.ws.client.Stub.process(Stub.java:211)
at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(SEIStub.java:124)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:98)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:78)
at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:107)
at com.sun.proxy.$Proxy33.onboardPerson(Unknown Source)
at com.paychex.corp.fileloader.FileloaderMain.callWebService(FileloaderMain.java:202)
at com.paychex.corp.fileloader.FileloaderMain.main(FileloaderMain.java:104)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:411)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:525)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:208)
at sun.net.www.http.HttpClient.New(HttpClient.java:291)
at sun.net.www.http.HttpClient.New(HttpClient.java:310)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:987)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:966)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:841)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1031)
at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:109)
My client is generated from wsimport using the WSDL file that IS visible from the active service. Code connection sample: URL and service name, etc are correct, can't list for security
URL url = new URL("http://correctserviceurl?WSDL");
QName qname = new QName("http://correct servicelocation","nameofService");
CorporateEmployeeOnboardingService service = new CorporateEmployeeOnboardingService(url, qname);
//CorporateEmployeeOnboardingService service = new CorporateEmployeeOnboardingService();
CorporateEmployeeOnboardingProcess process = service.getICorporateEmployeeOnboardingProcess();
OnboardRequestType onboardRequest = new OnboardRequestType();
/**
* TESTING SECTION FOR PROCESS CALL
*/
//create necessary complex types:
GregorianCalendar gregorianCalendar = new GregorianCalendar();
DatatypeFactory datatypeFactory;
datatypeFactory = DatatypeFactory.newInstance();
XMLGregorianCalendar testDate =
datatypeFactory.newXMLGregorianCalendar(gregorianCalendar);
testDate.setYear(1991);
testDate.setMonth(4);
testDate.setDay(3);
AssignmentComplexType testAssignment = new AssignmentComplexType();
testAssignment.setHiringManagerID("123445");
//set person to send
person.setSSN("333-33-3333");
person.setFirstName("testFname");
person.setLastName("testLname");
person.setDOB(testDate);
person.setHireDate(testDate);
person.setAssignment(testAssignment);
/**
* end testing section of the person creation
*/
onboardRequest.getPerson().add(person);//add a person to the array of people for the request
ServiceResponse response = process.onboardPerson(onboardRequest);
As stated SoapUI can send this same test information and get a perfectly valid request from the service. All information the same. I have been digging around for awhile and confirmed that there is no proxy, both are same bit (32/64 in this case both 32), service name and WSDL are correct and objects are named properly, etc. I did further debugging trying to get around this awful error code that has seemingly no usable information and using wireshark compared the SoapUI request to the java request to find that SoapUI was using POST and the java client was using GET which has me leaning on this right now as the source of the generic refused connection error.
The service is using BPEL to run and was created using BPEL so all code was basically generated. I will be looking at the Get/post find as of now but wanted to get this out there as I am doubtful it could be the cause.
Any help is greatly appreciated. Let me know if I forgot any key information I might be able to share.
EDIT
I wanted to add that I am able to access the endpoint URL in the web browser from the machine that I am using to try and connect to the BPEL service. It seems the issue is solely with the client generated from the WSDL stubs I got using wsimport for the service's WSDL.
Edit 2: Development/Progress
The issue seems to be that the client is sending the response to local host even after setting up the qname to the correct address. I am now trying to figure out where this local host is being set. Any ideas on this development would be greatly appreciated.

It's been a while, but I believe this line: URL url = new URL("http://correctserviceurl?WSDL");
Needs to read: URL url = new URL("http://correctserviceurl");

K! This one was an issue with BPEL I believe. Quite frustrating. Apparently after initializing the service in the following lines:
URL url = new URL("http://serviceaddresshere?WSDL");
QName qname = new QName("http://servicelocationaddress","NameOfService");
CorporateEmployeeOnboardingService service = new CorporateEmployeeOnboardingService(url, qname);
CorporateEmployeeOnboardingProcess process = service.getICorporateEmployeeOnboardingProcess();
BPEL gets the request with the service WSDL and the CORRECT address. However, BPEL seems to think "O? you want this service, here you go I'll give you that, only the WSDL you want is wrapper, so here's the REAL WSDL." In plain english, you get back the BPEL WSDL that it thinks is for the service, and this has the Localhost address in it. Now I tried changing the address in this WSDL but BPEL through errors up the wazzoo. So the work around was adding in this line after the process variable is make: (AKA THE FIX)
((BindingProvider) process).getRequestContext().put(Dispatch.ENDPOINT_ADDRESS_PROPERTY, "http://the serviceaddress");
Bassically you set the endpoint AGAIN after you make the process, because BPEL overwrites what you sent it the first time. This is of course a sort of work around, but the process now works good. If you can think of a way to bypass this work around and dynamically fix the service so we don't have to keep changing the service WSDL when we move ti to new enviornments like Dev, etc. I am all ears. Thanks and as always I will read comments for suggestions. =)

You can use bpel configuration plans to switch between environments.
Configuration plans are the recommended option to use for making composites environment specific.
Efficient use of configuration plans can greatly reduce the work required per process.
A configuration plan can be generated from the composite.xml file.
Please consider this link:
http://biemond.blogspot.com.br/2010/02/web-service-references-and-soa.html

Open an authenticated image served by django from java using Apache http client

I Am serving an authenticated image using django. The image is behind a view which require login, and in the end I have to check more things than just the authentication.
Because of a reason to complicated to explain here, I cannot use the real url to the image, but I Am serving it with a custom url leading to the authenticated view.
From java the image must be reachable, to save or display. For this part I use Apache httpclient.
In Apacahe I tried a lot of things (every example and combination of examples...) but can't seem to get it working.
For other parts of the webapp I use django-rest-framwork, which I succesfully connected to from java (and c and curl).
I use the login_reuired decorator in django, which makes the attempt to get to the url redirect to a login page first.
Trying the link and the login in a webviewer, I see the 200 code (OK) in the server console.
Trying the link with the httpclient, I get a 302 Found in the console.... (looking up 302, it means a redirect..)
this is what I do in django:
in urls.py:
url(r'^photolink/(?P<filename>.*)$', 'myapp.views.photolink',name='photolink'),
in views.py:
import mimetypes
import os
#login_required
def photolink(request, filename):
# from the filename I get the image object, for this question not interesting
# there is a good reason for this complicated way to reach a photo, but not the point here
filename_photo = some_image_object.url
base_filename=os.path.basename(filename_photo)
# than this is the real path and filename to the photo:
path_filename=os.path.join(settings.MEDIA_ROOT,'photos',mac,base_filename)
mime = mimetypes.guess_type(filename_photot)[0]
logger.debug("mimetype response = %s" % mime)
image_data = open(path_filename, 'rb').read()
return HttpResponse(image_data, mimetype=mime)
by the way, if i get this working i need another decorator to pass some other tests....
but i first need to get this thing working....
for now it's not a secured url.... plain http.
in java i tried a lot of things... using apache's httpclient 4.2.1
proxy, cookies, authentication negociation, with follow redirects... and so on...
Am I overlooking some basic thing here?...
it seems the login of the website client is not suitable for automated login...
so the problem can be in my code in django....or in the java code....

In the end the problem was, using HTTP authorization.
Which is not by default used in the login_required decorator.
adding a custom decorator that checks for HTTP authorization did the trick:
see this example: http://djangosnippets.org/snippets/243/

Simple Kerberos client in Java?

Applications such a Google's Chrome and IE can transparently handle Kerberos authentication; however I can not find a "simple" Java solution to match this transparency. All of the solutions I have found require the presence of a krb5.conf file and a login.conf file which nether of the above apps seem to require.
What is the best way to build a Java application with Kerberos SSO capabilities that just work?
[update]: to be clear I need a CLIENT side solution for creating tickets not validating them. Also, it seems that SPNEGO is the default "wrapper" protocol that will eventually delegate to Kerberos but I need to be able to handle the SPNEGO protocol as well.

There is now a simple solution for this using the Apache HTTP Components Client 4.5 or greater. This is still marked as experimental in 4.5 so your milage may vary, but this is working fine for me in an enterprise context.
In addition to the HC 4.5 client jars you will need to have the httpclient-win, jna and jna-platform jars on your classpath, as provided with http-component-client. You then construct a Kerberos enabled HC-client as follows:
CloseableHttpClient httpclient = WinHttpClients.createDefault();
Or using the builder:
HttpClientBuilder clientBuilder = WinHttpClients.custom();
Which can then be customised as required before building the client:
CloseableHttpClient client = clientBuilder.build();
This solution works without any external configuration, and most importantly solves the issue where the in-built JRE mechanism breaks for users with local Admin rights on Windows 7+. This is possible because the Kerberos ticket is being retrieved directly from the SSPI API via JNA, rather than going through the GSSAPI provided by the JRE.
Example code from the http-components team
This was all made possible by the good work of Daniel Doubrovkine Timothy Wall
and Ryan McKinley

Adding to David Roussels answer on url specific http based kerberos authentication:-
The reason why your code works is because your target SPN(server side principal) is configured to with HTTP/serverhostname.realm.com#DOMAIN.COM. In that case it will work because you are not explicitly setting the token. URLConnection internally sets a token with that SPN
1 Perform steps(from my previous answer) to get a subject
2 Use gss api init sec context to generate a context token. There are numerous tutorials out there for this step
3 Base 64 encode the token
4 Attach the token to urlconnection:-
URL url = new URL("http://myhost/myapp")
HttpURLConnection urlConn = (HttpURLConnection)url.openConnection(); =
urlConn.setRequestProperty("Authorization", "Negotiate " + encodedToken);
5 Implement a priviledged action:-
//this internally calls the getInputStream
public class PrivilegedGetInputStream implements PrivilegedExceptionAction<InputStream>
6 Wrap the whole thing in Subject.doAs
//use prev answer instructions to get subject
Subject.doAs(subject, new PrivilegedGetInputStream(urlConnection)

Oracle has an example using Java's SaslClient. I'm not a Java programmer, but when I pointed this out once to someone who is, they were able to make it work pretty quickly. It may still require a "conf" file somewhere (n.b. Kerberos uses environment variables, often starting with KRB5_, to know where to look for such files). Also note that Kerberos itself does not include a transport of any kind--your app needs to know how to send and receive the Kerberos payloads the way the server expects (and this is different depending on the server you are trying to authenticate with).
Edit: you edited your question, so here's a link related to SPNEGO in Java which might be of some use:
http://download.oracle.com/javase/6/docs/technotes/guides/security/jgss/lab/part5.html

You don't actually need to do anything. In Java 6, on a Windows client machine you can do this:
new URL("http://myhost/myapp").openStream();
And negotiate authentication just works. At least it does for me. And the server I tested on only supports Negotiate, not NTLM auth.

Ok if you want to avoid using a login.conf file you need to code differently:-
//define your own configuration
import javax.security.auth.login.Configuration;
public class CustomLoginConfiguration extends Configuration
//pass certain parameters to its constructor
//define an config entry
import javax.security.auth.login.AppConfigurationEntry;
private AppConfigurationEntry configEntry;
//define a map of params you wish to pass and fill them up
//the map contains entries similar to one you have in login.conf
Map<String, String> params = new HashMap<String, String>();
//define the configuration
configEntry = new AppConfigurationEntry(
"com.sun.security.auth.module.Krb5LoginModule",
AppConfigurationEntry.LoginModuleControlFlag.REQUIRED, params);
//implement getappconfig method
public AppConfigurationEntry[] getAppConfigurationEntry() {
return new AppConfigurationEntry[] { configEntry };
}
Now once you are done with this definition you can use this in you use this to fetch tickets from kdc
//get ticket in login context
LoginContext lc = null;
lc = new LoginContext("lc", null, callback, new CustomLoginConfiguration(argumentlist));
lc.login();
Now from here on you can fetch jaas subject and can basically do a ton of authentication stuff.
In case you need further pointers just leave a comment.

You can use system properties instead of config files to specify the KDC hostname and service name, but those things (at least) are mandatory....
Waffle will actually give you the information you need to set most of the properties, even if it won't get you a ticket. Look at the WindowsAuthProviderImpl class (the Waffle.chm help file shows the API).
I use JAAS do obtain a service ticket from Active Directory in two steps:
Use Krb5LoginModule to retrieve the cached TGT and add it to the Subject.
Use the Subject and GSS-API to retrieve a service ticket from the KDC.
There's a lot of good information and example code at The Java Way of Active Directory.

I created a small tool to simplify connecting with httpclient to kerberos, you might want to give it a try.
https://github.com/DovAmir/httpclientAuthHelper
DefaultHttpClient httpclient = new DefaultHttpClient();
AuthUtils.securityLogging(SecurityLogType.KERBEROS,true);
CredentialsUtils.setKerberosCredentials(client, new UsernamePasswordCredentials("xxx", "xxx"), "domain", "kdc");
client.executeMethod(httpget);

Use WAFFLE

Here's a good blog post on having a java client to use with Kerberos
http://sachithdhanushka.blogspot.com/2014/02/kerberos-java-client-configuration.html

Using java class HttpsURLConnection

I have a small piece of code which basically impements a HTTP-Client,
i.e. it POSTS request and works with re RESPONSE. As long as HTTP is
concenerned everthing work well. For some reason I now have to support
HTTPS too. So here is briefly what I do in order to get a connection opened:
URL url = new URL(serverAddress);
HttpsURLConnection httpsConn = (HttpsURLConnection) url.openConnection();
This fails, stating:
sun.net.www.protocol.https.HttpsURLConnectionImpl cannot be cast to com.sun.net.ssl.HttpsURLConnection
I guess this is kinda trivial, but I just don't get what I'm doing wrong in this one...
Googled it, and the code just looks right - not?
any ideas are appreciated!

Just keep it java.net.URLConnection or cast it to java.net.HttpURLConnection instead. Both offers methods to do the desired task as good.
A side remark unrelated to the technical problem: you should never explicitly import/use Sun Java SE implementation specific classes in your code. Those are undocumented classes and are subject to changes which may cause your code break when you upgrade the JVM. On the other hand, your code may also break when you run it at a different brand JVM.
Update: since you seem to accidentally have imported it, go to Window > Preferences > Java > Appearance > Type Filters and Add com.sun.* and sun.* to the list. This way you won't ever import them accidentally:

Your url's protocol should also be https and not http. Check your url.

The above problem is only caused by two issues
Using wrong import
Using http in the string you create url from use instead https

Instead of creating a URL object using standard constructor like
URL wsURL = new URL(url);
Use
java.net.URL wsURL = new URL(null, url,new sun.net.www.protocol.https.Handler());
which would solve this problem

Check your imports, you should be using
java.net.HttpURLConnection
or
javax.net.ssl.HttpsURLConnection

Check value of your "serverAddress" variable. It should https and not http

Change:
import javax.net.ssl.HttpsURLConnection;
and
URL url = new URL(serverAddress);
HttpsURLConnection httpsConn = (HttpsURLConnection) url.openConnection();
Change To:
import java.net.HttpURLConnection;
and
URL url = new URL(serverAddress);
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();

Hard to tell without seeing the whole file, but it looks like you're importing com.sun.net.ssl.HttpsURLConnection when you really want javax.net.ssl.HttpsURLConnection.

In my case, the protocol and port were not correct while invoking the httpsUrlConnection.
Port and protocol were defined as static class variables. And the step prior to the failed step, was invoking an httpUrlConnection. That method changed the port/protocol to 80/http, but didn't set it back to /https at the end. So eventhough httpsUrlConnection was invoked, it was still using http/80. Once I reset those at the end of the httpUrlConnection method, the error disappeared.