I have a very simple code that uses HttpURLConnection to access some web site via proxy
System.setProperty("java.net.useSystemProxies", "true");
System.out.println("Proxy: " + ProxySelector.getDefault().select(new URI(urlS)));
URL url = new URL(urlS);
HttpURLConnection ic = (HttpURLConnection)url.openConnection();
ic.connect();
For some reason, Java thinks that I need SOCKS proxy, not http, throwing the following exception:
ERROR: Can't connect to SOCKS proxy:Connection timed out: connect
If you are having this issues on Windows, you may run into a Java bug.
Java treats any system proxy setting as SOCKS. You have to either disable useSystemProxies or don't use proxy in Windows.
If proxy is needed, try to uncheck "Use the same proxy server for all protocols", making sure the field for the SOCKS proxy is blank. That fixed our problem.
The real problem is that Java assumes that the "Use the same proxy server for all protocols" check affects SOCKS proxy too (I don't know the logic behind this dialog in Windows, but it is, at least, confusing)
If the check is set, you get proxies enabled for both HTTP and SOCKS, wich is very unlikely to be the desired configuration.
One way to solve it is unchecking the check and leaving blank the SOCKS field.
I finally solved it creating a ProxySelector wich first calls the default selector and if it finds the same configuration for HTTP and SOCKS connections, it omits the SOCKS proxy.
public class SocksFixerProxySelector extends ProxySelector {
ProxySelector base;
public SocksFixerProxySelector() {
base = ProxySelector.getDefault();
}
#Override
public List<Proxy> select(URI uri) {
List<Proxy> baseList = base.select(uri);
try {
if (uri.getScheme().equals("socket")) {
Proxy socksProxy = findByType(baseList, Type.SOCKS);
if (socksProxy != null) {
URI httpTestUri = new URI("http", uri.getHost(), uri.getPath(), uri.getFragment());
Proxy httpProxy = findByType(base.select(httpTestUri), Type.HTTP);
if (httpProxy != null && socksProxy.address().equals(httpProxy.address())) {
// Quitamos SOCKS
List<Proxy> filteredList = new ArrayList<>(baseList);
filteredList.remove(socksProxy);
return filteredList;
}
}
}
} catch (Exception e) {
}
return baseList;
}
#Override
public void connectFailed(URI uri, SocketAddress sa, IOException ioe) {
base.connectFailed(uri, sa, ioe);
}
private Proxy findByType(List<Proxy> proxies, Proxy.Type type) {
for (Proxy proxy : proxies) {
if (proxy.type() == type)
return proxy;
}
return null;
}
Maybe a better solution would be to inspect the registry and detect the right settings, but I didn't want to mess with Windows specific code (and all those script settings looked bad, too )
You need to use the http.proxyHost system property instead. See http://java.sun.com/javase/6/docs/technotes/guides/net/proxies.html for details.
java -Dhttp.proxyHost=webcache.mydomain.com GetURL
Check that something has not set the "socksProxyHost" property in the Systems properties.
EDIT
The "useSystemProxies" property is described thus:
"On recent Windows systems and on Gnome 2.x platforms it is possible to tell the default ProxySelector to use the system proxy settings (both recent versions of Windows and Gnome 2.x let you set proxies globally through their user interface). If the system property java.net.useSystemProxies is set to true (by default it is set to false for compatibility sake), then the default ProxySelector will try to use these settings."
So, assuming that you have not supplied your own ProxySelector class, you should also check the system proxy settings to ensure that they don't say to use SOCKS.
Related
Using grpc from either nodejs or java, what are the properties or configuration necessary to get a grpc client to connect to a server through a proxy?
I have been unable to find either an example or a document explaining the settings. Do I need to do something in the code itself?
I am behind a proxy and I am not sure if the issue is that my settings are incorrect or that my proxy does not support grpc. It supports http/2 as a protocol upgrade.
My proxy settings in java are:
-Dhttp.proxyHost=xxx.xxx.xxx
-Dhttp.proxyPort=8888
-Dhttp.nonProxyHosts="*.nowhere.nothing"
-Dhttps.proxyHost=xxx.xxx.com
-Dhttps.proxyPort=8888
-Dhttps.nonProxyHosts="*.nowhere.nothing"
-Dsocks.proxyHost=xxx.xxx.xxx
-Dsocks.proxyPort=8888
-Dsocks.nonProxyHosts="*.nowhere.nothing"
Since grpc-java 1.0.3 you can specify the environment variable GRPC_PROXY_EXP with a value in the form host:port. The "EXP" means experimental, as it will be removed after grpc-java observes the normal Java settings (like https.proxyHost).
In later releases (I think since 1.8.0+) you need:
System.setProperty("http.proxyHost", "http-ip-address-hostname");
System.setProperty("http.proxyPort", "http-port-value");
System.setProperty("https.proxyHost", "https-ip-address-hostname");
System.setProperty("https.proxyPort", "https-port-value");
If you prefer to not use the global https.proxyHost, https.proxyPort properties, you could use the StubSettings of your client to specify a ChannelConfigurator. It might then look like this:
InetSocketAddress proxyAddress = new InetSocketAddress("my.proxy.local", 8080);
InstantiatingGrpcChannelProvider transportProvider = SessionsStubSettings.defaultGrpcTransportProviderBuilder()
.setChannelConfigurator(new ApiFunction<ManagedChannelBuilder, ManagedChannelBuilder>() {
#Override
public ManagedChannelBuilder apply(ManagedChannelBuilder input) {
return input.proxyDetector(new ProxyDetector() {
#Override
public ProxiedSocketAddress proxyFor(SocketAddress targetServerAddress) throws IOException {
if (!(targetServerAddress instanceof InetSocketAddress) || targetServerAddress == null) {
return null;
}
return HttpConnectProxiedSocketAddress.newBuilder()
.setTargetAddress((InetSocketAddress) targetServerAddress)
.setProxyAddress(proxyAddress)
.build();
}
});
}
})
.build();
and then you could use the stubSettings bellow to create your GRPC client:
stubSettings = XYZStubSettings.newBuilder().setTransportChannelProvider(transportProvider);
I was wondering if it's possible to provide a custom implementation for DNS lookups on java.net.URL - my hosting provider's DNS gets flaky at certain times of day and then DNS lookups fail for a few minutes, but if I manually configure the relevant domains in my hosts file, they work fine, so what I want to do is have some sort of DNS cache at a software level, if DNS lookup succeed, update the cache, if it fails, fall back to the cached IP address and open the URLConnection on that IP address.
This is my URL connection implementation:
URL endpoint = new URL(null, url, new URLStreamHandler() {
#Override
protected URLConnection openConnection(URL url)
throws IOException {
URL target = new URL(url.toString());
URLConnection connection = target.openConnection();
// Connection settings
connection.setConnectTimeout(connectionTimeout);
connection.setReadTimeout(readTimeout);
return (connection);
}
});
I was looking at proxies on Oracle, but can't see any immediate way to do custom DNS lookups at the software level.
Limitations:
1: It needs to work in Java6 (maybe Java7, but the client won't be switching to Java8 anytime soon)
2: Can't add JVM args
3: I don't own these endpoints, so substituting the hostname with an IP address is not a solution since load balancers will serve different content / APIs depending on whether you come from a hostname or an IP address. As an example: mail.google.com resolves to 216.58.223.37, going to that IP address will serve google.com content and not mail.google.com content, since both services are sitting behind the same load balancer using a single IP address.
4: I don't know how many URLs' DNS resolutions I'll need to cache, but I do know it won't be more than a 1000. Ideal solution would be to have the DNS resolutions in a static hashmap, if any DNS resolution succeed, update the hashmap, if it fails, use the DNS resolution in the hashmap.
5: If there's a native java solution, I'd prefer that over using JNI - Understanding host name resolution and DNS behavior in Java
You could create a custom method to check whether the host resolves to an IP. Prior to opening the connection if the host does not resolve then do your lookup and use the IP directly to build the URL instead:
At class level:
private Map<String,InetAddress> cacheMap = new HashMap<String,InetAddress>();
....then a couple of methods to build your URL:
private URL buildUrl (String host) throws MalformedURLException {
InetAddress ia = resolveHostToIp(host);
URL url = null;
if (ia != null) {
url = new URL(ia.getHostAddress());
} else {
// Does not resolve and is not in cache....dunno
}
return url;
}
private InetAddress resolveHostToIp(String host) {
InetAddress ia = null;
try {
ia = InetAddress.getByName(host);
// Update your cache
cacheMap.put(host, ia);
} catch (UnknownHostException uhe) {
System.out.println("\"" + host + "\" does not resolve to an IP! Looking for it in the cacheMap....");
// Head off to your cache and return the InetAddress from there.....
ia = cacheMap.get(host);
}
return ia;
}
You could simply construct another URL:
URL target = new URL(
url.getProtocol(),
customDns.resolve(url.getHost()),
url.getFile());
You can implement customDns.resolve(String) using whatever strategy you need.
I have set up a local proxy server for request logging but my java code ignores it and connects directly (Windows XP, JDK 1.7). Web browsers work with it. So I wrote test code for discussion that seems to connect directly even if a (bogus) proxy is specified. With the bogus proxy, I would expect connection failure but the code succeeds, connecting directly:
System.setProperty("http.proxyHost", "localhost");
System.setProperty("http.proxyPort", "12345");
System.setProperty("http.nonProxyHosts", "noNonProxyHost.com");
URL url = new URL("http://docs.oracle.com/javase/7/docs/technotes/guides/net/proxies.html");
InputStream in = url.openStream();
System.out.println("Connection via bogus proxy succeeded");
The code is run as standalone Java, no Maven, no applet, no container. I have a direct internet connection.
In your case using java.net.URL(), if the proxy server cannot be reached at http.proxyHost and http.proxyPort then it simply falls back and tries to do a direct connect. If that succeeds, you'll see no exception thrown which is why your code works without error. You should see a pause while it tries to find the proxy though.
This sample code below happily fetches the URL and displays it, without error, even when run with bogus proxy settings. -Dhttp.proxyHost=bogus -Dhttp.proxyPort=2345 but will talk to my local proxy localhost port 8888 if set correctly
import java.io.*;
import java.net.URL;
import java.util.*;
public class URLClient {
private static String sUrl = "http://www.apache.org/";
public static void main(String[] args) {
try {
URL url = new URL(sUrl);
InputStream is = url.openStream();
java.util.Scanner s = new java.util.Scanner(is).useDelimiter("\\A");
String output = s.hasNext() ? s.next() : "";
System.out.println(output);
} catch(Throwable e) {
System.err.println("exception");
}
}
}
The problem I was originally having with http.proxyHost and http.proxyPort being ignored (Google led me to your question) was that those settings are completely ignored by apache.commons.httpClient because it uses its own sockets, as described here.
http://cephas.net/blog/2007/11/14/java-commons-http-client-and-http-proxies/
I have faced a similar problem recently. First of all, one part of the above answer from Daemon42 explains pretty well, why the bogus proxy server didn't lead to a failure of the program:
if the proxy server cannot be reached at http.proxyHost and http.proxyPort then it simply falls back and tries to do a direct connect. If that succeeds, you'll see no exception thrown which is why your code works without error. You should see a pause while it tries to find the proxy though.
Still, your actual question was, why the proxy server configured via the operating system is not used by the Java application. As stated in the Oracle documentation (https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html), the system proxy settings are not evaluated by Java by default. To do so, you have to set the value of the system property "java.net.useSystemProxies" to the value "true".
You can set that system property on the command line, or you can edit the JRE installation file jre/lib/net.properties, that way you have to change it only once on a given system.
I want to write a ContainerRequestFilter for a Jersey webapp that will filter out all remote calls.
So only requests from same machine (where webapp is running) are allowed.
I get a context object of type ContainerRequestContext where I get the host name via ctx.getUriInfo().getRequestUri().getHost().
How can I check if this host name (in form of IPv4, IPv6 or domain name) is an address of the local machine?
I'd go with something like this, once you stripped the host name from the request. It should work with inputs like localhost and such as well.
public boolean isLocalAddress(String domain) {
try {
InetAddress address = InetAddress.getByName(domain);
return address.isAnyLocalAddress()
|| address.isLoopbackAddress()
|| NetworkInterface.getByInetAddress(address) != null;
} catch (UnknownHostException | SocketException e) {
// ignore
}
return false;
}
But please keep in mind, as it's not straightforward to determine if a request is originated from a local client, and there is also performance implications, I'd suggest to bind the container's listen address only to a locally accessible interface (127.0.0.1, ::1), or implement some sort of authentication. This approach - where you trying to determine this info from the request is also insecure.
I'm trying to write a small java program that connects to a twitter search URL (which returns a JSON list of tweets) using the URL connection libary.
My code which is taken from the java tutorials looks like :
public static void main(String[] args) throws Exception {
URL oracle = new URL("http://search.twitter.com/search.json?q=hi");
URLConnection yc = oracle.openConnection();
BufferedReader in = new BufferedReader(
new InputStreamReader(
yc.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
in.close();
}
but for some reason I keep getting the following Exception:
in thread "main" java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
I don't know if this is something due to the way I've written the code, and eclipse setting or something to do with my network. I do have a proxy server configured for internet access. As far as I know this is properly configured because I am getting updates and can install new software through eclipse. Do I need to put the proxy information in the URL method somehow or is something else the problem.
URL rely on System properties for proxy, try setting proxy like this:
System.setProperty("http.proxyHost", "yourproxyserver");
System.setProperty("http.proxyPort", "portnumber");
Unfortunately, a correct proxy setup in Eclipse seems not to help with proxying Java programs started in Eclipse. Similarly, setting the Java system settings to use the systemwide proxy settings doesn't either. Not when you have a proxy that requires authentication, anyway.
As Thomas Johan Eggum said, if you have a "normal," non-authenticating proxy then setting the two JVM variables http.proxyHost and http.proxyPort either in the command line via -D or programmatically (see below) will take care of things.
For an authenticating proxy server, i.e. one that wants to see a user ID and password, many people recommend setting http.proxyUser and http.proxyPassword. That's bad advice, because these don't work. Apparently they are not defined in the Java docs.
Unfortunately, it looks like the way to "do" authentication is to use an Authenticator, programmatically. If you're going to do that, you might as well do the whole thing programmatically, i.e. including host and port. Here's how I got that to work:
public static void main(String[] args) {
try {
System.setProperty("http.proxyHost", "my.proxy.host");
System.setProperty("http.proxyPort", "8080-or-whatever-proxy-port");
Authenticator.setDefault(new DummyAuthenticator());
/* do your main program stuff */
} catch (Exception e) {
e.printStackTrace();
}
}
private static class DummyAuthenticator extends Authenticator {
public PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication(
"my-user-id", "my-password".toCharArray()
);
}
}