I was wondering if it's possible to provide a custom implementation for DNS lookups on java.net.URL - my hosting provider's DNS gets flaky at certain times of day and then DNS lookups fail for a few minutes, but if I manually configure the relevant domains in my hosts file, they work fine, so what I want to do is have some sort of DNS cache at a software level, if DNS lookup succeed, update the cache, if it fails, fall back to the cached IP address and open the URLConnection on that IP address.
This is my URL connection implementation:
URL endpoint = new URL(null, url, new URLStreamHandler() {
#Override
protected URLConnection openConnection(URL url)
throws IOException {
URL target = new URL(url.toString());
URLConnection connection = target.openConnection();
// Connection settings
connection.setConnectTimeout(connectionTimeout);
connection.setReadTimeout(readTimeout);
return (connection);
}
});
I was looking at proxies on Oracle, but can't see any immediate way to do custom DNS lookups at the software level.
Limitations:
1: It needs to work in Java6 (maybe Java7, but the client won't be switching to Java8 anytime soon)
2: Can't add JVM args
3: I don't own these endpoints, so substituting the hostname with an IP address is not a solution since load balancers will serve different content / APIs depending on whether you come from a hostname or an IP address. As an example: mail.google.com resolves to 216.58.223.37, going to that IP address will serve google.com content and not mail.google.com content, since both services are sitting behind the same load balancer using a single IP address.
4: I don't know how many URLs' DNS resolutions I'll need to cache, but I do know it won't be more than a 1000. Ideal solution would be to have the DNS resolutions in a static hashmap, if any DNS resolution succeed, update the hashmap, if it fails, use the DNS resolution in the hashmap.
5: If there's a native java solution, I'd prefer that over using JNI - Understanding host name resolution and DNS behavior in Java
You could create a custom method to check whether the host resolves to an IP. Prior to opening the connection if the host does not resolve then do your lookup and use the IP directly to build the URL instead:
At class level:
private Map<String,InetAddress> cacheMap = new HashMap<String,InetAddress>();
....then a couple of methods to build your URL:
private URL buildUrl (String host) throws MalformedURLException {
InetAddress ia = resolveHostToIp(host);
URL url = null;
if (ia != null) {
url = new URL(ia.getHostAddress());
} else {
// Does not resolve and is not in cache....dunno
}
return url;
}
private InetAddress resolveHostToIp(String host) {
InetAddress ia = null;
try {
ia = InetAddress.getByName(host);
// Update your cache
cacheMap.put(host, ia);
} catch (UnknownHostException uhe) {
System.out.println("\"" + host + "\" does not resolve to an IP! Looking for it in the cacheMap....");
// Head off to your cache and return the InetAddress from there.....
ia = cacheMap.get(host);
}
return ia;
}
You could simply construct another URL:
URL target = new URL(
url.getProtocol(),
customDns.resolve(url.getHost()),
url.getFile());
You can implement customDns.resolve(String) using whatever strategy you need.
Related
For a project in Java, i use a users ip address as a unique identifier for a website. It all works fine when a user only has a IPv4 address.
But here's my problem, when a user has a IPv6 address, Java will report a IPv4 address, while their browser will report a IPv6 address (In the $_SERVER['REMOTE_ADDR'] call in PHP).
So how do i make Java report it's IPv6 address insted of IPv4 ?
Edit: Some clarification:
This is only used to allow one person to download a file. It does not matter if several people use the same PC, not really important.
Say a user goes to http://whatismyipaddress.com/ with chrome, it will display a IPv6 address.
But if i now use Java to fetch http://whatismyipaddress.com/ it will show a IPv4 address. Why is this and how can i make it show the IPv6 address shown in Chrome ?
Edit 2: The code i use
public static String get(String url) throws WebFetchException {
String result = "";
try {
URL url1 = new URL(url);
HttpsURLConnection urlConn = (HttpsURLConnection) url1.openConnection();
urlConn.setConnectTimeout(8 * 1000);
urlConn.setRequestMethod("GET");
BufferedReader in = new BufferedReader(new InputStreamReader(urlConn.getInputStream()));
String text;
while ((text = in.readLine()) != null) {
result = result + text;
}
in.close();
}
} catch (Exception e) {
throw new WebFetchException("There was an exception while fetching the requested page: " + url);
}
return result;
}
The assumption that an ip address can be used as an identifier is wrong. More and more ISPs are deploying some form of NAT so many users will share a pool of IPv4 assesses. The address that a user is coming from can change from one connection to the other, so you can't rely on it being stable.
On the other hand more users also get IPv6, and devices have multiple IPv6 addresses that will change over time.
It also happens that because of connectivity issues users will switch between IPv4 and IPv6. Well, the user won't even notice, their device will just do it.
So relying on ip addresses just won't work.
I want to write a ContainerRequestFilter for a Jersey webapp that will filter out all remote calls.
So only requests from same machine (where webapp is running) are allowed.
I get a context object of type ContainerRequestContext where I get the host name via ctx.getUriInfo().getRequestUri().getHost().
How can I check if this host name (in form of IPv4, IPv6 or domain name) is an address of the local machine?
I'd go with something like this, once you stripped the host name from the request. It should work with inputs like localhost and such as well.
public boolean isLocalAddress(String domain) {
try {
InetAddress address = InetAddress.getByName(domain);
return address.isAnyLocalAddress()
|| address.isLoopbackAddress()
|| NetworkInterface.getByInetAddress(address) != null;
} catch (UnknownHostException | SocketException e) {
// ignore
}
return false;
}
But please keep in mind, as it's not straightforward to determine if a request is originated from a local client, and there is also performance implications, I'd suggest to bind the container's listen address only to a locally accessible interface (127.0.0.1, ::1), or implement some sort of authentication. This approach - where you trying to determine this info from the request is also insecure.
Here's the method-
public static String getHostByAddr(byte[] addr) throws UnknownHostException {
Name name = ReverseMap.fromAddress(InetAddress.getByAddress(addr));
final String[] servers = new String[] {"208.67.220.220", "208.67.222.222"};
final Resolver res = new ExtendedResolver(servers);
final Lookup lookUp = new Lookup(name, Type.PTR);
lookUp.setResolver(res);
Record[] records = lookUp.run();
if (records == null) {
throw new UnknownHostException();
}
return ((PTRRecord) records[0]).getTarget().toString();
}
And here's the call to the above method-
final InetAddress ip = InetAddress.getByName("198.154.218.168");
final byte[] bytes = ip.getAddress();
final String host = getHostByAddr(bytes);
System.out.println("Host - " + host);
Works fine for most of the cases, but fails when the IP is mapped to multiple domains (???)
Here's the example-
Get IP of securonix.com from here, it is 198.154.218.168
If I pass this IP to the above method it gives error
But if I try the same IP here, it lists down the 4 domains
Is it possible to do this with DNSJava?
In general, you can't. Just as the owner of the domain securonix.com created an entry on their DNS server that translates securonix.com to 198.154.218.168, the owner of the IP address 198.154.218.168 maintains a "reverse" DNS record (usually on a separate server) that maps the IP address to a default DNS name. See the page for "Reverse DNS lookup" on wikipedia for more information on reverse DNS lookups, and note that the reverse DNS lookup -- from IP address to name -- is often maintained by a different owner than the domain name owner and it is NOT just created "automatically" by swapping the name and IP address.
Furthermore, the site you mention doesn't seem to have reverse DNS entries for that IP address, so you won't ever get an answer for 198.154.218.168 from a standard DNS query; there simply is no (reverse) DNS entry for 198.154.218.168 and the code above fails.
As an example of the difference between the forward and reverse DNS lookups, when I was first allocated my static IP address, my provider (Comcast) mapped the IP address to some generic name like 75-148-###-###-Houston.hfc.comcastbusiness.net (random IP address example) and I had ask them to modify the reverse DNS entry to map to the name of my domain instead so that a forward and reverse lookup of the IP address and domain name matched. They maintained the lookup of the IP address on their DNS servers, and I maintained the lookup of the domain name on my DNS servers.
The page at yougetsignal.com must have been doing DNS (forward) lookups of names and storing them in the huge database available for purchase, which allows the web page to find all the names with the same IP. But there is no easy way to do that for an arbitrary IP address by querying DNS servers, unless you have considerable additional information or have already done the millions of lookups like that site.
I found that there are two websites providing this functionality. However I don't know how they implement that functionality.
http://reverseip.domaintools.com/
http://ipaddress.com/reverse_ip/
update:
domaintools provide api calls. maybe you can call domaintools api to get multiple domain names for the single IP.
I have a very simple code that uses HttpURLConnection to access some web site via proxy
System.setProperty("java.net.useSystemProxies", "true");
System.out.println("Proxy: " + ProxySelector.getDefault().select(new URI(urlS)));
URL url = new URL(urlS);
HttpURLConnection ic = (HttpURLConnection)url.openConnection();
ic.connect();
For some reason, Java thinks that I need SOCKS proxy, not http, throwing the following exception:
ERROR: Can't connect to SOCKS proxy:Connection timed out: connect
If you are having this issues on Windows, you may run into a Java bug.
Java treats any system proxy setting as SOCKS. You have to either disable useSystemProxies or don't use proxy in Windows.
If proxy is needed, try to uncheck "Use the same proxy server for all protocols", making sure the field for the SOCKS proxy is blank. That fixed our problem.
The real problem is that Java assumes that the "Use the same proxy server for all protocols" check affects SOCKS proxy too (I don't know the logic behind this dialog in Windows, but it is, at least, confusing)
If the check is set, you get proxies enabled for both HTTP and SOCKS, wich is very unlikely to be the desired configuration.
One way to solve it is unchecking the check and leaving blank the SOCKS field.
I finally solved it creating a ProxySelector wich first calls the default selector and if it finds the same configuration for HTTP and SOCKS connections, it omits the SOCKS proxy.
public class SocksFixerProxySelector extends ProxySelector {
ProxySelector base;
public SocksFixerProxySelector() {
base = ProxySelector.getDefault();
}
#Override
public List<Proxy> select(URI uri) {
List<Proxy> baseList = base.select(uri);
try {
if (uri.getScheme().equals("socket")) {
Proxy socksProxy = findByType(baseList, Type.SOCKS);
if (socksProxy != null) {
URI httpTestUri = new URI("http", uri.getHost(), uri.getPath(), uri.getFragment());
Proxy httpProxy = findByType(base.select(httpTestUri), Type.HTTP);
if (httpProxy != null && socksProxy.address().equals(httpProxy.address())) {
// Quitamos SOCKS
List<Proxy> filteredList = new ArrayList<>(baseList);
filteredList.remove(socksProxy);
return filteredList;
}
}
}
} catch (Exception e) {
}
return baseList;
}
#Override
public void connectFailed(URI uri, SocketAddress sa, IOException ioe) {
base.connectFailed(uri, sa, ioe);
}
private Proxy findByType(List<Proxy> proxies, Proxy.Type type) {
for (Proxy proxy : proxies) {
if (proxy.type() == type)
return proxy;
}
return null;
}
Maybe a better solution would be to inspect the registry and detect the right settings, but I didn't want to mess with Windows specific code (and all those script settings looked bad, too )
You need to use the http.proxyHost system property instead. See http://java.sun.com/javase/6/docs/technotes/guides/net/proxies.html for details.
java -Dhttp.proxyHost=webcache.mydomain.com GetURL
Check that something has not set the "socksProxyHost" property in the Systems properties.
EDIT
The "useSystemProxies" property is described thus:
"On recent Windows systems and on Gnome 2.x platforms it is possible to tell the default ProxySelector to use the system proxy settings (both recent versions of Windows and Gnome 2.x let you set proxies globally through their user interface). If the system property java.net.useSystemProxies is set to true (by default it is set to false for compatibility sake), then the default ProxySelector will try to use these settings."
So, assuming that you have not supplied your own ProxySelector class, you should also check the system proxy settings to ensure that they don't say to use SOCKS.
I am trying to get hostname/computer name using this method. Unfortunately i only can get localhost but not other computer.
private String getHostName(String _strIP) {
try {
InetAddress inetAddress = InetAddress.getByName(_strIP);
System.out.println("getHostAddress : " + inetAddress.getHostAddress());
System.out.println("getHostName : " + inetAddress.getHostName());
System.out.println("getCanonicalHostName : " + inetAddress.getCanonicalHostName());
return inetAddress.getHostName();
} catch (UnknownHostException e) {
e.printStackTrace();
}
return strDefaultHostName;
}
the result (not localhost)
getHostAddress : 192.168.2.139
getHostName : 192.168.2.139
getCanonicalHostName : 192.168.2.139
the result (localhost)
getHostAddress : 127.0.0.1
getHostName : localhost
getCanonicalHostName : localhost
Thank you
We've established roughly what the problem is in tangens' answer.
I think you can fix the problem pretty simply by putting host names into your hosts file.
%SystemRoot%\system32\drivers\etc\hosts
is the file you're looking for; localhost is defined here. You want to put a name and address line in it for every host you want to resolve.
I've never tried this. If it doesn't work, you get your money back.
Update
The above is the "quick hack" solution. This essentially entails that whenever someone manually changes the IP address of a host you're interested in, someone must at the same time change the hosts files on any machines that want to access those hosts.
The other alternative is to operate your own DNS server. You still need to update IP addresses when a host's address changes, but you only need to do so in one place, and you get both forward and reverse name resolution throughout your network. This takes more setting up but is easier to maintain in the long run.
Here is a very useful reference: http://www.dns.net/dnsrd/servers/windows.html
They mention that the "built in" Microsoft DNS server is a terrible solution (up until the one in Windows 2003 Server) but mention at least two alternatives, one commercial and one free. BIND is what is currently holding much of the Internet together, DNS-wise, and it's great that they have a Windows port too.
Looking at the source for InetAddress.getHostName() (Sun JDK8)...
The method performs the following logic:
Loops through the available sun.net.spi.nameservice.NameService's
Performs a reverse DNS lookup - e.g. 192.168.0.23 -> frodo.baggins.com.au
*Checks with the java.lang.SecurityManager, to see if "we have permission to connect" to hostname
*Performs a forward DNS lookup on the hostname, to prevent spoofing - e.g. frodo.baggins.com.au -> 192.168.0.99
If forward lookup result matches the original address (e.g. 192.168.0.23 == 192.168.0.99?), return hostname, otherwise return getHostAddress()
*If step 3 or 4 throws a SecurityException/UnknownHostException, return getHostAddress()
For me, step #2 successfully resolved the hostname, but failed at step #4 with an UnknownHostException.
TLDR; you must fulfill ALL of the following requirements:
the SecurityManager must provide permission to access the host
you must be able to forward AND reverse DNS lookup your InetAddress
the forward lookup details MUST match the reverse lookup details
Only then will Java give you the hostname.
OR, you could bypass these steps with the following method, and just get the hostname.
#SuppressWarnings("unchecked")
public static String getHostName(InetAddress addr) {
String host = null;
List<NameService> nameServicesImpl = new ArrayList<>();
try {
// do naughty things...
Field nameServices = InetAddress.class.getDeclaredField("nameServices");
nameServices.setAccessible(true);
nameServicesImpl = (List<NameService>) nameServices.get(null);
} catch (Throwable t) {
throw new RuntimeException("Got caught doing naughty things.", t);
}
for (NameService nameService : nameServicesImpl) {
try {
// lookup the hostname...
host = nameService.getHostByAddr(addr.getAddress());
} catch (Throwable t) {
// NOOP: problem getting hostname from this name service, continue looping...
}
}
return host != null ? host : addr.getHostAddress();
}
Your DNS is broken. Then IP-numbers are returned instead.
The javadoc of InetAddress.getCanonicalHostName() says:
Gets the fully qualified domain name for this IP address. Best effort method, meaning we may not be able to return the FQDN depending on the underlying system configuration.
If there is a security manager, this method first calls its checkConnect method with the hostname and -1 as its arguments to see if the calling code is allowed to know the hostname for this IP address, i.e., to connect to the host. If the operation is not allowed, it will return the textual representation of the IP address.
I looks like your system configuration isn't correct. Are you running from within an applet?
Reply Feedback for Carl Smotricz
Great answer, but we still don't know if the host name has been updated or not...
This is something like we hardcode.
Anyway thank you so much
# Copyright (c) 1993-1999 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a '#' symbol.
#
# For example:
#
# 102.54.94.97 rhino.acme.com # source server
# 38.25.63.10 x.acme.com # x client host
127.0.0.1 localhost
192.168.2.139 dev-testing
The problem can be caused by multiple reasons.
Reason 1: the IP address doesn't have a hostname
This is probably the most common reason, and has nothing to do with security managers.
If an IP address doesn't resolve to a hostname, because there is no hostname, then you would expect getHostName() to return null or throw a UnknownHostException, but this doesn't happen. Instead getHostName() simply returns the IP address as a string back again. For reasons unknown to me, this common situation is undocumented.
So if the IP address is the same as the result returned by getHostName(), then the hostname doesn't exist.
Detailed explanation
The following JDK code is the cause of this undocumented problem:
https://github.com/openjdk/jdk/blob/jdk-17+35/src/java.base/share/classes/java/net/InetAddress.java#L697
public class InetAddress implements java.io.Serializable {
private static String getHostFromNameService(InetAddress addr, boolean check) {
String host = null;
try {
// first lookup the hostname
host = nameService.getHostByAddr(addr.getAddress());
/* check to see if calling code is allowed to know
* the hostname for this IP address, ie, connect to the host
*/
if (check) {
#SuppressWarnings("removal")
SecurityManager sec = System.getSecurityManager();
if (sec != null) {
sec.checkConnect(host, -1);
}
}
/* now get all the IP addresses for this hostname,
* and make sure one of them matches the original IP
* address. We do this to try and prevent spoofing.
*/
InetAddress[] arr = InetAddress.getAllByName0(host, check);
boolean ok = false;
if(arr != null) {
for(int i = 0; !ok && i < arr.length; i++) {
ok = addr.equals(arr[i]);
}
}
//XXX: if it looks a spoof just return the address?
if (!ok) {
host = addr.getHostAddress();
return host;
}
} catch (SecurityException e) {
host = addr.getHostAddress();
} catch (UnknownHostException e) {
host = addr.getHostAddress();
// let next provider resolve the hostname
}
return host;
}
}
So what happens is that the IP-address is passed to NameService.getHostByAddr() (NameService is a private interface), which has this (private) documentation in the source code:
Lookup the host corresponding to the IP address provided
#param addr byte array representing an IP address
#return {#code String} representing the host name mapping
#throws UnknownHostException if no host found for the specified IP address
So NameService.getHostByAddr() throws an UnknownHostException if the IP doesn't have a hostname, but InetAddress.getHostFromNameService() swallows this exception and instead, it returns the provided IP-address itself!!! IMO it should have let the exception be thrown instead of swallowing it, because swallowing it makes it more difficult for the client to determine whether a hostname exists.
You can check if the IP address has a hostname by using the nslookup commandline tool: nslookup 192.168.2.139. If it returns something like:
** server can't find 139.2.168.192.in-addr.arpa: NXDOMAIN (Linux) or *** can't find 192.168.2.139: Non-existent domain (Windows) then there is no hostname.
Reason 2: a security manager is applied
By default, Java doesn't have a security manager enabled. In that case, this reason doesn't apply.
A security manager is an object that defines a security policy for an application. If you have a security manager and want to find out if it is the cause of your problem, then you should check whether it is allowing you to open a socket to the resolved hostname (if any). To do so, first use nslookup 192.168.2.139 and verify if a hostname is resolved. If no hostname is resolved, then your problem is caused by "Reason 1". If it does resolve to a hostname, for example myhostname, then try this:
SecurityManager sec = System.getSecurityManager();
if (sec != null) {
sec.checkConnect("myhostname", -1);
}
If checkConnect() throws a SecurityException, then a SecurityManager is active and is causing the problem. So then you could look into how you can configure your securityManager to solve the problem.