I'm trying to connect to a website with javafx webengine/webview. I use a proxy that does not support the https protocol (so only http). When I try to connect to a http website, it works. However, when I try to load a https website, it ignores the proxy and uses my internet connection.
When I use the same proxy in my browser I'm also able to connect to https websites and I wonder if it's possible to change my code so that the webengine will still use my http proxy instead of switching to my normal internet connection.
My code:
final String[] pArray = line.split(":");
System.out.println("IP:" + pArray[0] + " Port:" + pArray[1]);
System.setProperty("http.proxyHost",pArray[0]);
System.setProperty("http.proxyPort",pArray[1]);
System.setProperty("http.proxyUser",pArray[2]);
System.setProperty("http.proxyPassword",pArray[3]);
Authenticator.setDefault(
new Authenticator() {
#Override
public PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication(pArray[2], pArray[3].toCharArray());
}
}
);
System.setProperty("java.net.useSystemProxies", "true");
System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");
When I load my webengine:
engine.load("http://whatismyip.host");
The website (a http only website) displays the proxy IP.
But when I do:
engine.load("https://myip.is");
It shows the IP of my home connection, meaning it ignores the http proxy...
How can I change that?
Regards
We have a java client side application deployed in our customer (a java application, not an applet).
This application checks connectivity with an url.openConnection() and calls web services (with CXF/JAX-WS) through internet.
Some of our customer network use proxies to access to the external world.
The client side application sets the proxy parameter in java system properties :
System.setProperty("proxySet", "true"); //Obsolete ?
System.setProperty("http.keepAlive", "false");
System.setProperty("java.net.useSystemProxies", "false");
System.setProperty("https.proxyHost", httpsProxyHost);
System.setProperty("https.proxyPort", httpsProxyPort);
System.setProperty("https.proxyUser", httpsProxyUser);
System.setProperty("https.proxyPassword", httpsProxyPassword);
System.setProperty("http.proxyHost", httpProxyHost);
System.setProperty("http.proxyPort", httpProxyPort);
System.setProperty("http.proxyUser", httpProxyUser);
System.setProperty("http.proxyPassword", httpProxyPassword);
Authenticator.setDefault(new NtlmAuthenticator(httpsProxyUser, httpsProxyPassword));
The NtlmAuthenticator class :
public class NtlmAuthenticator extends Authenticator {
private final String username;
private final char[] password;
public NtlmAuthenticator(final String username, final String password) {
super();
this.username = username;
this.password = password.toCharArray();
}
public PasswordAuthentication getPasswordAuthentication() {
return (new PasswordAuthentication (username, password));
}
}
We're using Java 6 (client side application embbed a JRE 1.6.0_39), and application is deployed on Windows (XP / Seven). I read that NTLM protocol is supported since 1.4.2 on Windows platform.
So we made tests with a Trend proxy and succeed to perform NTLM proxy authentication (we see the 3 packets with Wireshark NTLMSSP_NEGOCIATE (from app) / NTLMSSP_CHALLENGE (from proxy) / NTLMSSP_AUTH (from app))
But with one of our customers, who use a Bluecoat proxy, NTLM authentication failed after NTLMSSP_CHALLENGE. With Wireshark, we only see the 2 first packets NTLMSSP_NEGOCIATE (from app) and NTLMSSP_CHALLENGE (from proxy), NTLMSSP_AUTH is never sent by our application.
In the application we catch a SocketException : socket is closed
We also try to use jCIFS HttpUrlNltmHandler, but authentication failed too (same diagnostic).
I found this thread with similar issue but it doesn't provide any clues.
I found also this thread about NTLM session security
Any ideas ?
Thanks.
Find the solution just by setting http.keepalive to true :
System.setProperty("http.keepAlive", "true");
But i don't know why, with false value, it works with our Trend proxy and doesn't work with our customer's bluecoat proxy
It's due to a bug on the underlying implementation. It's described on Java 6 NTLM proxy authentication and HTTPS - has anyone got it to work?
I need to do a HTTP GET to a URL that needs NTLM authentication. I can access the URL using Firefox or Chrome on a MacBook Pro. The browser asks for the username/password combo and it works. I am now trying to do the same from Groovy using HttpClient. I followed the NTLM support guide, but I always get a 401 Unauthorized back. There is also this sentence in the response:
You do not have permission to view this directory or page using the
credentials that you supplied because your Web browser is sending a
WWW-Authenticate header field that the Web server is not configured to
accept.
I tried all kinds of combinations for the servername and domain (the remote windows pc is not on a domain) in this piece of code, but I always get the same response.
httpclient.getCredentialsProvider().setCredentials(
new AuthScope("myserver", -1),
new NTCredentials("username", "password", "MYSERVER", "MYDOMAIN"));
Anybody had the same problem and managed to solve it? Note that this is an external program that uses IIS under the hood, so I don't think I can change any settings there.
EDIT:
Unlike what I have said, I managed to change the security settings in IIS to accept BASIC authentation, so I don't have the problem anymore.
EDIT:
In my experience with setting up Kerberos or NTLM (both are single sign on), you don't have to enter username/password at all when you are already logged in to your system.
I am pretty sure that when the browser asked for username/password combo, that's not an NTLM authentication at all. Most likely the server side application has a fallback scheme to HTTP Basic Digest (that why it displayed the username/password combo). With NTLM you'll never have to enter your username/password (principal/credentials) at all, as the server will recognize who you are through the negotiation mechanism between your browser, your operating system, server and Active Directory server.
If your MacBook Pro is running on OS/X, you also need to add your OS/X to the domain.
Your server also needs to be in the same domain where the client OS/X being added.
This may not be a trivial case. Some external tools/driver may be needed. This one may be a good candidate (but I haven't tried that).
NTLM needs both the client to be a member of the same domain as the server, hence both needs to be registered in the Active Directory domain. If your server is not in the domain, than that will be another set of problem.
In order to get your browser works with NTLM, you need to install plugin (ntlmauth-plugin?). But I have never try that on MacOS/X yet. Even in Windows you still need a plugin in order to run Firefox successfully with NTLM.
HttpClient did not work for me but finally the code below worked.
Reference - http://docs.oracle.com/javase/7/docs/technotes/guides/net/http-auth.html
For quick reference -
public static String getResponse(String url, String userName, String password) throws IOException {
Authenticator.setDefault(new Authenticator() {
#Override
public PasswordAuthentication getPasswordAuthentication() {
System.out.println(getRequestingScheme() + " authentication");
return new PasswordAuthentication(userName, password.toCharArray());
}
});
URL urlRequest = new URL(url);
HttpURLConnection conn = (HttpURLConnection) urlRequest.openConnection();
conn.setDoOutput(true);
conn.setDoInput(true);
conn.setRequestMethod("GET");
StringBuilder response = new StringBuilder();
InputStream stream = conn.getInputStream();
BufferedReader in = new BufferedReader(new InputStreamReader(stream));
String str = "";
while ((str = in.readLine()) != null) {
response.append(str);
}
in.close();
return response.toString();
}
I have a Java webapp, running in Tomcat 6, that loads RSS feeds from remote URLs.
I use Rome to handle the RSS feeds and different formats for me. The connection part looks like like that :
try{
feedSource = new URL(rssObject.getAsset());
}catch(MalformedURLException mue){
logger.error(...);
throw mue;
}
try{
URLConnection connection = feedSource.openConnection();
feed = new SyndFeedInput().build(new XmlReader(connection));
}catch(Exception){handle...}
The code works fine, except at this new client, where they use a proxy.
In order to use the proxy, I set the http.proxyHost and proxyPort system properties :
System.setProperty("http.proxyHost", proxyHost);
System.setProperty("http.proxyPort", proxyPort);
System.setProperty("https.proxyHost", proxyHost);
System.setProperty("https.proxyPort", proxyPort);
HTTP GET is made to the proxy alright, but now I get a HTTP 502 error (bad gateway or something similar).
Analysing the HTTP exchange with Wireshark, I noticed that the proxy is requiring authentication. It sends a HTTP 507. Java is somehow trying to authenticate but it uses the wrong username and passwords. It seems to use the host name as the username, as for the password I don't know.
So I tried to implement the Authenticator method of specifying a username+password :
Authenticator.setDefault(new Authenticator() {
#Override
protected PasswordAuthentication getPasswordAuthentication() {
logger.info(MessageFormat.format("Generating PasswordAuthentitcation for proxy authentication, using username={0} and password={1}.", username, password));
return new PasswordAuthentication(username, password.toCharArray());
}
});
Now my problem is that it is ignored. The getPasswordAuthentication method is never called. I don't see the logging statement in the log file and using Wireshark I can see that it still uses the host name as the user name.
Why ? It seems that java somehow tries to authenticate by itself without consulting the Authenticator.
The proxy seems to be a MS device that uses NTLM for authentication. Is there some built-in mechanism in java to handle this ? The machine on which the app runs is Win Server 2008 R2.
We did the same here for authenticating on a NTLM based proxy.
The authentication on the proxy is actually a normal HTTP Basic Authentication.
We used the following method:
protected URLConnection newURLConnection(URL pURL) throws IOException {
URLConnection urlConnection = super.newURLConnection(pURL);
String auth = new String(Base64.base64Encode(new String("username:password").getBytes()));
auth = "Basic " + auth;
urlConnection.setRequestProperty("Proxy-Connection","Keep-Alive");
urlConnection.setRequestProperty("Proxy-Authorization",auth);
return urlConnection;
}
That, together with the proxy jvm settings, did the trick.
See http://en.wikipedia.org/wiki/Basic_access_authentication.
Many times, a Java app needs to connect to the Internet. The most common example happens when it is reading an XML file and needs to download its schema.
I am behind a proxy server. How can I set my JVM to use the proxy ?
From the Java documentation (not the javadoc API):
http://download.oracle.com/javase/6/docs/technotes/guides/net/proxies.html
Set the JVM flags http.proxyHost and http.proxyPort when starting your JVM on the command line.
This is usually done in a shell script (in Unix) or bat file (in Windows). Here's the example with the Unix shell script:
JAVA_FLAGS=-Dhttp.proxyHost=10.0.0.100 -Dhttp.proxyPort=8800
java ${JAVA_FLAGS} ...
When using containers such as JBoss or WebLogic, my solution is to edit the start-up scripts supplied by the vendor.
Many developers are familiar with the Java API (javadocs), but many times the rest of the documentation is overlooked. It contains a lot of interesting information: http://download.oracle.com/javase/6/docs/technotes/guides/
Update : If you do not want to use proxy to resolve some local/intranet hosts, check out the comment from #Tomalak:
Also don't forget the http.nonProxyHosts property!
-Dhttp.nonProxyHosts="localhost|127.0.0.1|10.*.*.*|*.example.com|etc"
To use the system proxy setup:
java -Djava.net.useSystemProxies=true ...
Or programatically:
System.setProperty("java.net.useSystemProxies", "true");
Source: http://docs.oracle.com/javase/7/docs/api/java/net/doc-files/net-properties.html
To set an HTTP/HTTPS and/or SOCKS proxy programmatically:
...
public void setProxy() {
if (isUseHTTPProxy()) {
// HTTP/HTTPS Proxy
System.setProperty("http.proxyHost", getHTTPHost());
System.setProperty("http.proxyPort", getHTTPPort());
System.setProperty("https.proxyHost", getHTTPHost());
System.setProperty("https.proxyPort", getHTTPPort());
if (isUseHTTPAuth()) {
String encoded = new String(Base64.encodeBase64((getHTTPUsername() + ":" + getHTTPPassword()).getBytes()));
con.setRequestProperty("Proxy-Authorization", "Basic " + encoded);
Authenticator.setDefault(new ProxyAuth(getHTTPUsername(), getHTTPPassword()));
}
}
if (isUseSOCKSProxy()) {
// SOCKS Proxy
System.setProperty("socksProxyHost", getSOCKSHost());
System.setProperty("socksProxyPort", getSOCKSPort());
if (isUseSOCKSAuth()) {
System.setProperty("java.net.socks.username", getSOCKSUsername());
System.setProperty("java.net.socks.password", getSOCKSPassword());
Authenticator.setDefault(new ProxyAuth(getSOCKSUsername(), getSOCKSPassword()));
}
}
}
...
public class ProxyAuth extends Authenticator {
private PasswordAuthentication auth;
private ProxyAuth(String user, String password) {
auth = new PasswordAuthentication(user, password == null ? new char[]{} : password.toCharArray());
}
protected PasswordAuthentication getPasswordAuthentication() {
return auth;
}
}
...
Remember that HTTP proxies and SOCKS proxies operate at different levels in the network stack, so you can use one or the other or both.
You can set those flags programmatically this way:
if (needsProxy()) {
System.setProperty("http.proxyHost",getProxyHost());
System.setProperty("http.proxyPort",getProxyPort());
} else {
System.setProperty("http.proxyHost","");
System.setProperty("http.proxyPort","");
}
Just return the right values from the methods needsProxy(), getProxyHost() and getProxyPort() and you can call this code snippet whenever you want.
JVM uses the proxy to make HTTP calls
System.getProperties().put("http.proxyHost", "someProxyURL");
System.getProperties().put("http.proxyPort", "someProxyPort");
This may use user setting proxy
System.setProperty("java.net.useSystemProxies", "true");
Combining Sorter's and javabrett/Leonel's answers:
java -Dhttp.proxyHost=10.10.10.10 -Dhttp.proxyPort=8080 -Dhttp.proxyUser=username -Dhttp.proxyPassword=password -jar myJar.jar
Set the java.net.useSystemProxies property to true. You can set it, for example, through the JAVA_TOOL_OPTIONS environmental variable. In Ubuntu, you can, for example, add the following line to .bashrc:
export JAVA_TOOL_OPTIONS+=" -Djava.net.useSystemProxies=true"
You can set some properties about the proxy server as jvm parameters
-Dhttp.proxyPort=8080, proxyHost, etc.
but if you need pass through an authenticating proxy, you need an authenticator like this example:
ProxyAuthenticator.java
import java.net.*;
import java.io.*;
public class ProxyAuthenticator extends Authenticator {
private String userName, password;
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication(userName, password.toCharArray());
}
public ProxyAuthenticator(String userName, String password) {
this.userName = userName;
this.password = password;
}
}
Example.java
import java.net.Authenticator;
import ProxyAuthenticator;
public class Example {
public static void main(String[] args) {
String username = System.getProperty("proxy.authentication.username");
String password = System.getProperty("proxy.authentication.password");
if (username != null && !username.equals("")) {
Authenticator.setDefault(new ProxyAuthenticator(username, password));
}
// here your JVM will be authenticated
}
}
Based on this reply:
http://mail-archives.apache.org/mod_mbox/jakarta-jmeter-user/200208.mbox/%3C494FD350388AD511A9DD00025530F33102F1DC2C#MMSX006%3E
The following shows how to set in Java a proxy with proxy user and proxy password from the command line, which is a very common case. You should not save passwords and hosts in the code, as a rule in the first place.
Passing the system properties in command line with -D and setting them in the code with System.setProperty("name", "value") is equivalent.
But note this
Example that works:
C:\temp>java -Dhttps.proxyHost=host -Dhttps.proxyPort=port -Dhttps.proxyUser=user -Dhttps.proxyPassword="password" -Djavax.net.ssl.trustStore=c:/cacerts -Djavax.net.ssl.trustStorePassword=changeit com.andreas.JavaNetHttpConnection
But the following does not work:
C:\temp>java com.andreas.JavaNetHttpConnection -Dhttps.proxyHost=host -Dhttps.proxyPort=port -Dhttps=proxyUser=user -Dhttps.proxyPassword="password" -Djavax.net.ssl.trustStore=c:/cacerts -Djavax.net.ssl.trustStorePassword=changeit
The only difference is the position of the system properties! (before and after the class)
If you have special characters in password, you are allowed to put it in quotes "#MyPass123%", like in the above example.
If you access an HTTPS service, you have to use https.proxyHost, https.proxyPort etc.
If you access an HTTP service, you have to use http.proxyHost, http.proxyPort etc.
reading an XML file and needs to download its schema
If you are counting on retrieving schemas or DTDs over the internet, you're building a slow, chatty, fragile application. What happens when that remote server hosting the file takes planned or unplanned downtime? Your app breaks. Is that OK?
See http://xml.apache.org/commons/components/resolver/resolver-article.html#s.catalog.files
URL's for schemas and the like are best thought of as unique identifiers. Not as requests to actually access that file remotely. Do some google searching on "XML catalog". An XML catalog allows you to host such resources locally, resolving the slowness, chattiness and fragility.
It's basically a permanently cached copy of the remote content. And that's OK, since the remote content will never change. If there's ever an update, it'd be at a different URL. Making the actual retrieval of the resource over the internet especially silly.
If you want "Socks Proxy", inform the "socksProxyHost" and "socksProxyPort" VM arguments.
e.g.
java -DsocksProxyHost=127.0.0.1 -DsocksProxyPort=8080 org.example.Main
I am also behind firewall, this worked for me!!
System.setProperty("http.proxyHost", "proxy host addr");
System.setProperty("http.proxyPort", "808");
Authenticator.setDefault(new Authenticator() {
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication("domain\\user","password".toCharArray());
}
});
URL url = new URL("http://www.google.com/");
URLConnection con = url.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(
con.getInputStream()));
// Read it ...
String inputLine;
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
in.close();
Add this before you connect to a URL behind a proxy.
System.getProperties().put("http.proxyHost", "someProxyURL");
System.getProperties().put("http.proxyPort", "someProxyPort");
System.getProperties().put("http.proxyUser", "someUserName");
System.getProperties().put("http.proxyPassword", "somePassword");
This is a minor update, but since Java 7, proxy connections can now be created programmatically rather than through system properties. This may be useful if:
Proxy needs to be dynamically rotated during the program's runtime
Multiple parallel proxies need to be used
Or just make your code cleaner :)
Here's a contrived example in groovy:
// proxy configuration read from file resource under "proxyFileName"
String proxyFileName = "proxy.txt"
String proxyPort = "1234"
String url = "http://www.promised.land"
File testProxyFile = new File(proxyFileName)
URLConnection connection
if (!testProxyFile.exists()) {
logger.debug "proxyFileName doesn't exist. Bypassing connection via proxy."
connection = url.toURL().openConnection()
} else {
String proxyAddress = testProxyFile.text
connection = url.toURL().openConnection(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyAddress, proxyPort)))
}
try {
connection.connect()
}
catch (Exception e) {
logger.error e.printStackTrace()
}
Full Reference:
http://docs.oracle.com/javase/7/docs/technotes/guides/net/proxies.html
Recently I've discovered the way to allow JVM to use browser proxy settings. What you need to do is to add ${java.home}/lib/deploy.jar to your project and to init the library like the following:
import com.sun.deploy.net.proxy.DeployProxySelector;
import com.sun.deploy.services.PlatformType;
import com.sun.deploy.services.ServiceManager;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
public abstract class ExtendedProxyManager {
private static final Log logger = LogFactory.getLog(ExtendedProxyManager.class);
/**
* After calling this method, proxy settings can be magically retrieved from default browser settings.
*/
public static boolean init() {
logger.debug("Init started");
// Initialization code was taken from com.sun.deploy.ClientContainer:
ServiceManager
.setService(System.getProperty("os.name").toLowerCase().indexOf("windows") != -1 ? PlatformType.STANDALONE_TIGER_WIN32
: PlatformType.STANDALONE_TIGER_UNIX);
try {
// This will call ProxySelector.setDefault():
DeployProxySelector.reset();
} catch (Throwable throwable) {
logger.error("Unable to initialize extended dynamic browser proxy settings support.", throwable);
return false;
}
return true;
}
}
Afterwards the proxy settings are available to Java API via java.net.ProxySelector.
The only problem with this approach is that you need to start JVM with deploy.jar in bootclasspath e.g. java -Xbootclasspath/a:"%JAVA_HOME%\jre\lib\deploy.jar" -jar my.jar. If somebody knows how to overcome this limitation, let me know.
That works for me:
public void setHttpProxy(boolean isNeedProxy) {
if (isNeedProxy) {
System.setProperty("http.proxyHost", getProxyHost());
System.setProperty("http.proxyPort", getProxyPort());
} else {
System.clearProperty("http.proxyHost");
System.clearProperty("http.proxyPort");
}
}
P/S: I base on GHad's answer.
As is pointed out in other answers, if you need to use Authenticated proxies, there's no reliable way to do this purely using command-line variables - which is annoying if you're using someone else's application and don't want to mess with the source code.
Will Iverson makes the helpful suggestion over at Using HttpProxy to connect to a host with preemtive authentication to use a Proxy-management tool such as Proxifier ( http://www.proxifier.com/ for Mac OS X and Windows) to handle this.
For example with Proxifier you can set it up to only intercept java commands to be managed and redirected through its (authenticated) proxy. You're going to want to set the proxyHost and proxyPort values to blank in this case though, e.g. pass in -Dhttp.proxyHost= -Dhttp.proxyPort= to your java commands.
This is a complete example that worked for me - note that for HTTPS there are separate properties (as per https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html).
Code below sends a request to https://api.myip.com API and prints the response.
public static void main(String[] args) throws IOException {
System.setProperty("java.net.useSystemProxies", "true");
final String proxyUser = "proxy-user";
final String proxyPass = "password123";
final String host = "some.proxy.io";
final Integer port = 50201;
// http
System.setProperty("http.proxyHost",host);
System.setProperty("http.proxyPort", String.valueOf(port));
System.setProperty("http.proxyUser", proxyUser);
System.setProperty("http.proxyPassword", proxyPass);
// https
System.setProperty("https.proxyHost",host);
System.setProperty("https.proxyPort", String.valueOf(port));
System.setProperty("https.proxyUser", proxyUser);
System.setProperty("https.proxyPassword", proxyPass);
System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");
System.setProperty("jdk.https.auth.tunneling.disabledSchemes", "");
Authenticator.setDefault(new Authenticator() {
#Override
public PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication(proxyUser, proxyPass.toCharArray());
}
}
);
// create and send a https request to myip.com API
URL url = new URL("https://api.myip.com");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
int status = connection.getResponseCode();
// read the response
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String responseLine;
StringBuffer responseContent = new StringBuffer();
while ((responseLine = in.readLine()) != null)
responseContent.append(responseLine);
in.close();
connection.disconnect();
// print the response
System.out.println(status);
System.out.println(responseContent);
}
You can utilize the http.proxy* JVM variables if you're within a standalone JVM but you SHOULD NOT modify their startup scripts and/or do this within your application server (except maybe jboss or tomcat). Instead you should utilize the JAVA Proxy API (not System.setProperty) or utilize the vendor's own configuration options. Both WebSphere and WebLogic have very defined ways of setting up the proxies that are far more powerful than the J2SE one. Additionally, for WebSphere and WebLogic you will likely break your application server in little ways by overriding the startup scripts (particularly the server's interop processes as you might be telling them to use your proxy as well...).
I think configuring WINHTTP will also work.
Many programs including Windows Updates are having problems behind proxy. By setting up WINHTTP will always fix this kind of problems