We Keep Coding

JavaFX webengine: Use http proxy for https websites

I'm trying to connect to a website with javafx webengine/webview. I use a proxy that does not support the https protocol (so only http). When I try to connect to a http website, it works. However, when I try to load a https website, it ignores the proxy and uses my internet connection.
When I use the same proxy in my browser I'm also able to connect to https websites and I wonder if it's possible to change my code so that the webengine will still use my http proxy instead of switching to my normal internet connection.
My code:
final String[] pArray = line.split(":");
System.out.println("IP:" + pArray[0] + " Port:" + pArray[1]);
System.setProperty("http.proxyHost",pArray[0]);
System.setProperty("http.proxyPort",pArray[1]);
System.setProperty("http.proxyUser",pArray[2]);
System.setProperty("http.proxyPassword",pArray[3]);
Authenticator.setDefault(
new Authenticator() {
#Override
public PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication(pArray[2], pArray[3].toCharArray());
}
}
);
System.setProperty("java.net.useSystemProxies", "true");
System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");
When I load my webengine:
engine.load("http://whatismyip.host");
The website (a http only website) displays the proxy IP.
But when I do:
engine.load("https://myip.is");
It shows the IP of my home connection, meaning it ignores the http proxy...
How can I change that?
Regards

Play java application accessing a Web Service through an Authenticating Proxy

I am developing a Play Java application and it needs to connect to another REST service using the Play WS API (JavaWS) through an Authenticating Proxy (i.e. the proxy requires username/password).
First I tried using the JVM options given below when starting the Play app.
-Dhttp.proxyHost=<proxy_server_hostname> -Dhttp.proxyPort=<proxy_server_port> -Dhttp.proxyUser=<username> -Dhttp.proxyPassword=<password>
Above didn't work fully. Application was able to connect to the proxy server without any issue, but proxy server is returning PROXY_AUTH_REQUIRED error which suggests that -Dhttp.proxyUser and -Dhttp.proxyPassword JVM options didn't work.
I searched and found the following two links which shows how to do this in a typical Java application.
http://memorynotfound.com/configure-http-proxy-settings-java/
http://rolandtapken.de/blog/2012-04/java-process-httpproxyuser-and-httpproxypassword
As suggested in these two links, I modified my Play application's onStart method in Global.java as below,
#Override
public void onStart(Application application) {
//Proxy authentication begin
System.setProperty("http.proxyHost", "<proxy_server_hostname>");
System.setProperty("http.proxyPort", "<proxy_server_port>");
System.setProperty("http.proxyUser", "<username>");
System.setProperty("http.proxyPassword", "<password>");
Authenticator.setDefault(new Authenticator() {
#Override
protected PasswordAuthentication getPasswordAuthentication() {
if (getRequestorType() == RequestorType.PROXY) {
String prot = getRequestingProtocol().toLowerCase();
String host = System.getProperty(prot + ".proxyHost", "");
String port = System.getProperty(prot + ".proxyPort", "");
String user = System.getProperty(prot + ".proxyUser", "");
String password = System.getProperty(prot + ".proxyPassword", "");
if (getRequestingHost().equalsIgnoreCase(host)) {
if (Integer.parseInt(port) == getRequestingPort()) {
return new PasswordAuthentication(user, password.toCharArray());
}
}
}
return null;
}
});
//Proxy authentication end
}
With the above modification I started the Play application *without specifying the previously mentioned JVM options (as I an now giving the same in the code). But the result remained the same. The proxy server still returns PROXY_AUTH_REQUIRED error message to the application. Again the application is connecting to the proxy server with the above code modification, but Java Authenticator seems not submitting the proxy username and password to the proxy server.
Or is there a different approach to do this in a Play Java application?
Thanks

Network Authentication : Class Authenticator
- The class Authenticator represents an object that knows how to obtain authentication for a network connection. Usually, it will do this by prompting
the user for information.
- Can be used when credential need to be sent over network.
Authenticator

NTLM authentication with httpclient 4.2.1

I need to do a HTTP GET to a URL that needs NTLM authentication. I can access the URL using Firefox or Chrome on a MacBook Pro. The browser asks for the username/password combo and it works. I am now trying to do the same from Groovy using HttpClient. I followed the NTLM support guide, but I always get a 401 Unauthorized back. There is also this sentence in the response:
You do not have permission to view this directory or page using the
credentials that you supplied because your Web browser is sending a
WWW-Authenticate header field that the Web server is not configured to
accept.
I tried all kinds of combinations for the servername and domain (the remote windows pc is not on a domain) in this piece of code, but I always get the same response.
httpclient.getCredentialsProvider().setCredentials(
new AuthScope("myserver", -1),
new NTCredentials("username", "password", "MYSERVER", "MYDOMAIN"));
Anybody had the same problem and managed to solve it? Note that this is an external program that uses IIS under the hood, so I don't think I can change any settings there.
EDIT:
Unlike what I have said, I managed to change the security settings in IIS to accept BASIC authentation, so I don't have the problem anymore.

EDIT:
In my experience with setting up Kerberos or NTLM (both are single sign on), you don't have to enter username/password at all when you are already logged in to your system.
I am pretty sure that when the browser asked for username/password combo, that's not an NTLM authentication at all. Most likely the server side application has a fallback scheme to HTTP Basic Digest (that why it displayed the username/password combo). With NTLM you'll never have to enter your username/password (principal/credentials) at all, as the server will recognize who you are through the negotiation mechanism between your browser, your operating system, server and Active Directory server.
If your MacBook Pro is running on OS/X, you also need to add your OS/X to the domain.
Your server also needs to be in the same domain where the client OS/X being added.
This may not be a trivial case. Some external tools/driver may be needed. This one may be a good candidate (but I haven't tried that).
NTLM needs both the client to be a member of the same domain as the server, hence both needs to be registered in the Active Directory domain. If your server is not in the domain, than that will be another set of problem.
In order to get your browser works with NTLM, you need to install plugin (ntlmauth-plugin?). But I have never try that on MacOS/X yet. Even in Windows you still need a plugin in order to run Firefox successfully with NTLM.

HttpClient did not work for me but finally the code below worked.
Reference - http://docs.oracle.com/javase/7/docs/technotes/guides/net/http-auth.html
For quick reference -
public static String getResponse(String url, String userName, String password) throws IOException {
Authenticator.setDefault(new Authenticator() {
#Override
public PasswordAuthentication getPasswordAuthentication() {
System.out.println(getRequestingScheme() + " authentication");
return new PasswordAuthentication(userName, password.toCharArray());
}
});
URL urlRequest = new URL(url);
HttpURLConnection conn = (HttpURLConnection) urlRequest.openConnection();
conn.setDoOutput(true);
conn.setDoInput(true);
conn.setRequestMethod("GET");
StringBuilder response = new StringBuilder();
InputStream stream = conn.getInputStream();
BufferedReader in = new BufferedReader(new InputStreamReader(stream));
String str = "";
while ((str = in.readLine()) != null) {
response.append(str);
}
in.close();
return response.toString();
}

Java Proxy Authentication

I have a Java webapp, running in Tomcat 6, that loads RSS feeds from remote URLs.
I use Rome to handle the RSS feeds and different formats for me. The connection part looks like like that :
try{
feedSource = new URL(rssObject.getAsset());
}catch(MalformedURLException mue){
logger.error(...);
throw mue;
}
try{
URLConnection connection = feedSource.openConnection();
feed = new SyndFeedInput().build(new XmlReader(connection));
}catch(Exception){handle...}
The code works fine, except at this new client, where they use a proxy.
In order to use the proxy, I set the http.proxyHost and proxyPort system properties :
System.setProperty("http.proxyHost", proxyHost);
System.setProperty("http.proxyPort", proxyPort);
System.setProperty("https.proxyHost", proxyHost);
System.setProperty("https.proxyPort", proxyPort);
HTTP GET is made to the proxy alright, but now I get a HTTP 502 error (bad gateway or something similar).
Analysing the HTTP exchange with Wireshark, I noticed that the proxy is requiring authentication. It sends a HTTP 507. Java is somehow trying to authenticate but it uses the wrong username and passwords. It seems to use the host name as the username, as for the password I don't know.
So I tried to implement the Authenticator method of specifying a username+password :
Authenticator.setDefault(new Authenticator() {
#Override
protected PasswordAuthentication getPasswordAuthentication() {
logger.info(MessageFormat.format("Generating PasswordAuthentitcation for proxy authentication, using username={0} and password={1}.", username, password));
return new PasswordAuthentication(username, password.toCharArray());
}
});
Now my problem is that it is ignored. The getPasswordAuthentication method is never called. I don't see the logging statement in the log file and using Wireshark I can see that it still uses the host name as the user name.
Why ? It seems that java somehow tries to authenticate by itself without consulting the Authenticator.
The proxy seems to be a MS device that uses NTLM for authentication. Is there some built-in mechanism in java to handle this ? The machine on which the app runs is Win Server 2008 R2.

We did the same here for authenticating on a NTLM based proxy.
The authentication on the proxy is actually a normal HTTP Basic Authentication.
We used the following method:
protected URLConnection newURLConnection(URL pURL) throws IOException {
URLConnection urlConnection = super.newURLConnection(pURL);
String auth = new String(Base64.base64Encode(new String("username:password").getBytes()));
auth = "Basic " + auth;
urlConnection.setRequestProperty("Proxy-Connection","Keep-Alive");
urlConnection.setRequestProperty("Proxy-Authorization",auth);
return urlConnection;
}
That, together with the proxy jvm settings, did the trick.
See http://en.wikipedia.org/wiki/Basic_access_authentication.

Using HttpProxy to connect to a host with preemtive authentication

I am using HttpClient to connect to a host which requires BasicAUTH. But the proxy doesn't require any authentication. I have set it up as follows:
private final HttpClient httpClient; // Spring injected
Setting Basic auth:
private void setBasicAuth(final String username, final String password) {
httpClient.getState().setCredentials(new AuthScope(AuthScope.ANY_HOST, AuthScope.ANY_PORT, AuthScope.ANY_REALM), new UsernamePasswordCredentials(username, password));
httpClient.getParams().setAuthenticationPreemptive(true);
}
Setting proxy:
private void setProxy(final String proxyHost, final int proxyPort) {
hostConfiguration hostConfiguration = httpClient.getHostConfiguration();
hostConfiguration.setProxy(proxyHost, proxyPort);
}
But I get the following warnings when running the code. Everything works, but I want to get rid of the warnings as well (or at least understand why they appears)
WARN o.a.c.httpclient.HttpMethodDirector - Required proxy credentials not available for BASIC <any realm>#proxy.XXXXXX.no:3128
WARN o.a.c.httpclient.HttpMethodDirector - Preemptive authentication requested but no default proxy credentials available
Any ideas?

Here's an example from the Apache site for a proxy w/o credentials:
http://hc.apache.org/httpcomponents-client-ga/httpclient/examples/org/apache/http/examples/client/ClientExecuteProxy.java
(From http://hc.apache.org/httpcomponents-client-ga/examples.html)
You are getting the error because you are passing in a username/password and don't need to.
The WARN messages are coming from the logger (http://hc.apache.org/httpcomponents-client-ga/logging.html) - depending on how you have your logger set up you could just ignore that.
Having spent WAY too much time dealing with trying to make a Java application deal with proxy servers, I can tell you that using a tool such as Proxifier ( http://www.proxifier.com/ for Mac OS X and Windows) or CNTLM ( http://cntlm.sourceforge.net/) was much easier, more flexible, easier to debug, and kept the code clean.