Let's say, within a program, there are capabilities to connect to a socket and server(which would be my own computer - localhost) and has all the handling for communicating to the server and back (receiving and sending messages like a chat service). If i send this program as a .jar file to two other users their respective machines, as the users executes the program, it would connect to this socket and server. If all was set up correctly (correctly writes and reads from the socket inputstream and outputstreams) and displays the messages on a GUI chat box, would the two users be able to communicate with this chat service?
Yes, but the main barrier that would prevent users from doing this is a little something called Network Address Translation (NAT).
In short, the vast majority of computers connected to the Internet do not have their own, unique, public IPv4 address (and the vast majority do not have IPv6 addresses either, because ISPs are dragging their feet in rolling out IPv6, but that's another matter entirely).
Dedicated servers often do have (usually multiple) unique IPv4 addresses, but significant numbers of IP addresses are usually only available to hosting companies that operate their own Autonomous System -- and even then, purchasing IPs these days is very expensive, because they're a valuable and rare commodity.
Anyway, two random users on some sort of home Internet connection (like cable, DSL, or fiber) almost certainly have a router or modem of some sort that performs NAT, giving each user device (computer, tablet, etc.) their own private LAN IP, instead of giving each device a unique public IP.
This means that inbound connections from the public Internet are, by default, ignored by the NAT gateway. So if you try to open a socket to your friend's public IP, unless that gateway (modem/router) has been configured correctly, it will just silently ignore your connection attempt, and you'll get a timeout or "connection refused" type message.
There are a few techniques for getting around this, but most of them either require much more complicated application network code, or require the user to mess around in their router configuration pages (which casual users usually aren't willing to do).
Port forwarding: Users must manually configure their router to "forward" public IP ports to a specific private IP, thus allowing the connection attempt on the public IP to pass through the Network Address Translation table to the private LAN. There is no general-purpose way of automating this without user intervention due to the differences between router/gateway devices.
NAT Traversal, which requires one of a variety of NAT-T protocols to be implemented in your program, and requires the explicit support of the desired NAT-T protocol for the routers involved (yours and your friend's). There are loads of different NAT-T protocols available; it'd be up to you to determine which one to use, based on compatibility, availability, etc.
Using an interposing server as an indirect way of routing packets between the two: In this case, you create a server on the public Internet that does have a unique IP (or has ports forwarded to it), and have each of your two client computers initiate an outbound connection to that server. The server would accept connections from both clients, then process the packets and pass them along, like talking to someone through a middleman. This doesn't require any special configuration on the clients, as long as they can make outbound connections on the desired ports (by default, most can, except in restricted enterprise environments where your ports may be limited to a few like 80 and 443.)
You'd need to adopt one of these techniques if your clients are using NAT. If one of your clients has a static IP directly connected to their computer and can accept incoming connections directly (without NAT), your program concept would work without modification, as long as the user behind the NAT initiates the connection to the one with a static IP.
Related
I am building an IoT solution where I will have multiple devices connected to my local network that I want to connect to a hub that controls my solution.
Once I connect a new device I want to connect it with the hub and give it a name to identify it. I would like to implement the detection in an automatic way (so I don´t have to enter IP addresses manually). As an example when a Chromecast is present in a network I can see it in my streaming applications in my phone. So I want to do something similar to connect the hub with the devices.
My ideas so far is that it is two ways of doing this:
The hub scans the network for new devices (either periodically or when I say there is a new device present).
The devices scan the network to find the hub once connected.
Is any of these approaches to prefer over the other and in that case why?
When doing a scan, in whatever direction I choose, what is the most effective way to do the scan? I am doing an implementation using Java and what I have so far is this:
int timeout = 100;
for (int i = 1; i < 255; i++)
{
String host = subnet + "." + i;
if (InetAddress.getByName(host).isReachable(timeout))
{
String hostname = InetAddress.getByName(host).getHostName();
String canonicalHostName = InetAddress.getByName(host).getCanonicalHostName();
System.out.println(host + " is reachable. Hostname: " + hostname + ", CanonicalHostName: " + canonicalHostName);
}
}
What I have seen here is that the hostname returned is for most stuff in my network just the IP address, and not the name that I see as the hostname in my router. I thought I could use the hostname as an identifyer to detect specific devices and understand what they where - but with this small poc that does not seem to work. So how can I identify the devices in a good way?
Is there any library/solution for Java (or Javascript) and ESP8266 to do this already? (Feels like a common problem if implementing "smart home" devices).
There is no single way that we discover devices on a local area network.
Rather than scanning a network, devices generally either use a multicast or broadcast protocol to announce their presence, or they rendezvous at a (usually external) preconfigured server.
Some devices use mDNS - based loosely on the DNS protocol, they multicast packets to advertise their presence on the network. Apple uses mDNS in their products. It's poorly supported on Android, and requires additional software on Windows. mDNS names are normally in the .local domain.
Some devices use UPNP and SSDP - Belkin's Wemo line of products does this. UPNP and SSDP are overly complicated protocols based on XML and SOAP, which are poor choices for devices with limited RAM and processing power like the ESP8266 and ESP32.
Some devices just roll their own protocol. Haiku's "Big Ass Fan" line does this - they broadcast UDP packets with a homegrown protocol that at least initially was vulnerable to a variety of problems. I don't recommend going this route unless you really know what you're doing. Other, established protocols have already had a chance to have the bugs ironed out. Unless you're experienced in protocol design you're more likely to reinvent problems other protocols have had than a wonderful shiny new discoverability protocol.
These kinds of devices will either periodically broadcast or multicast a packet announcing themselves, or the thing you're calling a "hub" would broadcast or multicast a request and the devices would respond to that request.
Not all devices present an interface for direct control over the LAN they're connected to. Some simply rendezvous with a remote server. In this case you discover them by asking the server to enumerate them, and you control them via that server. Google's Nest products work this way - initial provisioning is done via Bluetooth; after that apps communicate with the devices through remote servers.
Devices rendezvousing in this manner are generally preconfigured with the name of the rendezvous server, but they might also be configured with the name of the server during network provisioning (generally an app communicates with them to share wifi credentials; it might also share information about a rendezvous server as well).
We generally do not scan for names in IP address blocks or actively probe for new devices unless we're debugging a network or doing some kind of security sweep.
The process of scanning an IP address block that you described is problematic and unreliable. The only reason it works is that some routers pick up a device's name from the device's DHCP request (or the router may be configured to know the device's name). The router also handles DNS for the devices on the network, generally by forwarding them to the ISP's DNS servers or to DNS servers that the network's owner configured it to use. It intercepts DNS requests for devices whose names it knows and replies to them itself, instead of forwarding them to an external DNS server.
You also have to know about the network configuration to do this properly. Is the network only a /24? What if it's /22? or /16? Are you prepared to scan 2^24 IP addresses if the network is configured as a /8?
Though the router may intercept requests for names and return addresses, it won't necessarily intercept names for addresses and return names.
Scanning also generates an unnecessary network traffic. While your one "hub" scanning may not seem like much, if you have multiple scanners from different makers running it doesn't scale well.
If your "hub" bypasses the router for DNS requests then it also will not be able to resolve names provided by the router.
Also not all routers do this. It's not part of the Internet architecture, it's a convenience feature that some routers provide. You cannot depend on it working.
You could also attempt an active scan of the network, trying to communicate with each IP address on it. We do this for network debugging, but running this continually to detect new devices would be a hostile way to interact with the network.
Network infrastructure devices - switches and routers - of course maintain lists of all active devices on the network. You can access these lists using SNMP (Simple Network Management Protocol), but few consumer switches and routers support SNMP. And allowing access to your network infrastructure to a random piece of software is a network security nightmare.
Your best bet is a simple multicast/broadcast protocol like mDNS. Don't scan, announce or request. It's responsive, doesn't place a burden on the network, doesn't depend on peculiarities of routers, and won't make network administrators hate you.
Long time ago I implemented a similar solution to discover clients connecting to the network. My strategy consisted in taking advantage of the DHCP configuration.
If your devices must ask IP addresses from a Linux/Unix DHCP server under your control, you could configure it to let you know when a device connects or disconnects from the network.
We were using Linux DHCP server, and in the dhcpd.conf there is a section about events. Basically it says that when the DHCP commits to a particular IP address lease for a client it raises an event, and when that happens, it can run a listener you could have defined for that event. In the listener we can ask the DHCP to execute a command, e.g. write the client information (e.g. MAC address) and its IP address lease to a named pipe, and your Java application could just be reading from this queue. (See also dhcp-eval for further details). A similar event can be run when a client releases an IP lease or when a lease expires.
Here's an example
subnet 192.168.1.0 netmask 255.255.255.0 {
option routers 192.168.1.2;
on commit {
set clip = binary-to-ascii(10, 8, ".", leased-address);
set clhw = binary-to-ascii(16, 8, ":", substring(hardware, 1, 6));
execute("/usr/local/sbin/dhcpevent", "commit", clip, clhw, host-decl-name);
}
...
If you can control your DHCP config there is so much you can extract from it.
I am doing an android app allowing users to play online.
Currently, I use a TCP server: when two persons are connected, the server takes care of forwarding the packets between the two clients.
I would like to replace my server by a java servlet with google app engine. This new server will just be used to connect the two players.
It would work in that way:
Player A opens a server socket and then post to the server the connection details.
When a player B wants to play against A, he asks to the server the port number of A and he connects directly to A.
The problem is that I am not sure that it will work if player A is behind a NAT. When player A opens a server socket, that opens one port of its 192.168.x.y address, but does it ask to the box a port forwarding? I assume it doesn't...
So two questions:
Is it possible to make a direct connection TCP between two devices even when there is a NAT or a firewall (I don't know how firewalls work on Android...)
If it isn't possible, what is the best solution: Is it possible to make a TCP server to ensure the exchange of the messages with app engine?
Thank you by advance.
game
Creating direct TCP connection between users under different NAT is mostly possible. There are 4 types of NAT. FC, ARC, PRC, Symmetric. If one of player A or B has symmetric NAT then it is impossible to create TCP P2P connection. In this case you will have to use a server in the middle for exchanging data between two players.
For other types of NAT combinations it is very much possible but not guaranteed. The technique that is used to create TCP P2P connection is called TCP hole punching. Read this answer to know in details about this technique.
Also creating TCP P2P connection is not related to any platform.
First, the device itself is probably not going to be the main problem. If they are at home and using WiFi, you will probably have to deal with a cable modem/DSL modem, which typically includes a firewall. Also if they are at work (or a hotel, conference center, etc.), there may be a corporate firewall to deal with.
I believe most home cable/DSL modems support uPnP (Universal Plug and Play), which includes the Internet Gateway Device Protocol (IGD) designed to let devices determine the external IP address and set up port mappings. In general you can look up NAT traversal for ways to handle connections through a home modem/firewall. I will note that corporate firewalls are a different matter and many of these techniques won't work.
So probably I would recommend you be ready for at least the following four scenarios
Direct connection with nothing creating problems. You can test this by having the server do a test connection when the player first contacts the server. If this works, things are simple.
Home NAT device that understands uPnP. If you have a 10.x.x.x, 172.16.x.x-172.31.x.x, or 192.168.x.x number (typical home WiFi), then you can try to set up the NAT traversal and if that works you can send the appropriate information to your server. It probably would be worthwhile for the server to do a test connection just to be sure that things work.
If you have a firewall that you can't get around, then make a note on the server regarding player A, and when B tries to join A's game, look and see if B will accept connections, and if so then arrange for A to connect to B instead.
If none of the above work, then have A and B both connect to the server and have the server relay messages between A and B.
If you don't want to program all those possibilities, then option 4 is the one that is most likely to work, even if it does mean extra traffic going to/from your server. But note that for corporate networks, they may simply have a rule blocking unknown connections, and there may not be much you can do.
Edit: I was able to get a simple TCP server working on Android without anything special regarding Android itself, so removed a comment saying I didn't know about that.
As I am programming a network chat (java, but should not make a difference for the question), and wanted to use UDP, I ran into the problem of it not working over the internet. After a little research I found out that you have to have port forwarding for the specific port activated. So now it comes to my question:
Does UDP work over the Internet in a not configurable way?
For example, if I would program a whole Network Game would it make sense to use UDP? Or would I require the Player to activate Portforwarding and open the Port etc?
When would it make sense to use UDP then? And why?
I'm actually not understanding the whole point of UDP then.
For my programming point of view I would like to have a way to use it intuitive.
Like creating the DatagramSocket and the DatagramPacket, configure the Packet with the Data and the Destination and send it away over the internet.
As for my Users I don't want them to have to configure any specific things like opening the exact port they want to use etc. I just want them to use the program (server and client) and it should work.
The problem you've run into is not one of UDP vs TCP (although using the unreliable, unordered UDP as the basis of a chat application seems like an odd choice to me).
The problem is that of NAT traversal. In a nutshell, home routers perform a network function called NAT - Network Address Translation. They do it in order to use a single public IP address for all machines inside the NAT (which are given private addresses - usually 10.0.0.0 or 192.168.0.0). The router then switches the source IP address in all packets sent from inside the LAN from the private address to the public one. It uses port numbers to "remember" which machine sent what to what address, in order to perform the backwards translation when the response arrives.
The problem arises when someone wants to initiate a connection to a machine behind a NAT. Without seeing an outgoing connection first, the NAT doesn't know to which internal computer and port it should forward the packet. This is what happens to you.
There are various fixes for this issue, with the simplest one being manual port forwarding (as you've discovered), but it's a well known problem faced by any peer-to-peer application. If you need to contact a machine behind NAT (i.e. contact most home users) and you want your application to work out-of-the box (without your users fiddling with their routers) you need to research NAT traversal techniques, implement them in your application, and hope that the user's home routers support them. It's a huge pain in the neck.
EDITED: with Joachim Pileborg's correct suggestions!
UDP is often a better choice for action-based games, where it's vitally important to have updates to the client or server with the latest data about a player, player input, or the game world.
TCP begins with a 3-way handshake to establish a connection (which takes time). If your game communication protocol is via TCP, all packets in a message have to arrive before the message becomes available. Even a small amount of Internet congestion could cause your game to lag.
TCP is good for communications that MUST arrive in full.
With UDP, the client or server can send the latest player/game state in individual packets that do not depend on arriving in order. If a packet is late, or arrives out of order... it should be ignored.
UDP is good for communications that need to be fast, but losing individual packets is OK.
Both should be available in your Java platform.
Here's some further reading:
http://gafferongames.com/networking-for-game-programmers/udp-vs-tcp/
I'm writing a Java application like AIM where I want a lookup server to help two clients connect to each other with the ServerSocket class. Unfortunately a serversocket needs an open port or it will be blocked by your firewall. Without having the clients manually change their firewall settings is there a way I can either find an open port, open a port (it's okay if it asks for permission) I just don't want every client to manually have to change firewall settings. Thanks!
Creating a ServerSocket is what opens the port on the server end (assuming the port is NOT also blocked on the firewall). On many systems, port numbers <1024 are not blocked by the firewall, and therefore often don't require custom configuration. However, since many ISPs have been more and more stringent with what ports they block within their network, many pieces of software have moved to operating over port 80. Why port 80? It's the port used for HTTP, and therefore pretty much open (at firewalls) 100% of the time for any service that operates on the web.
To answer the question of how to find an open port, port scanners perform this function. Basically all they do is try to establish a connection on a range of ports (say port 1-1024). When a connection is successful (in Java you would see this as a good connection over a Socket), you can consider that port "open". You don't need to exchange any data in order to make the connection, you just need to establish the connection, and then close it.
Also, if there is no server software of any kind running on the server, it won't appear "open", even if it isn't blocked. Without a successful Socket (TCP) connection, you won't know what ports are not blocked by user's firewall.
To reiterate, I'm switching back and forth between two related, but separate concepts. Firewalls can block/open ports, or have port forwarding. That's not the same as a port being "open" for connections on a given machine. In order for a machine to accept connections on a given port, there must be a piece of software listening for a connection on that port, otherwise no connection can be made.
I would suggest to look at a couple of alternative solutions that are less cumbersome - scanning for open ports can take quite some time, can cause panicky reactions from firewalls that feel attacked, and so on. There are a couple of techniques that are in active use and have been developed exactly to solve the problem of servers behind firewalls.
UDP Hole Punching, zero user side configuration needed. Simple explanation on how Skype uses this technique can be found here
UPnP / IGD could be used as an alternative, though less devices support it out of the box nowadays due to security problems.
STUN with a Java implementation of client and serverside called JSTUN
Whatever solution you choose, test it thoroughly with different internet service providers, there's a plethora of limitations you can expect wrt blocked ports.
I created a small chat program, that works flawlessly when client & server are run on the same computer (and probably network, too).
However, as soon as I try to connect to another computer over the internet, the socket connection simply times out.
Is this because of firewalls / routers, etc?
And how can I connect a ServerSocket & Socket over the internet?
However, as soon as I try to connect to another computer over the internet, the socket connection simply times out. Is this because of firewalls / routers, etc?
Yes, most likely. You're running into the NAT problem: essentially, the same externally visible IP address maps to many internally visible endpoints, and external endpoint doesn't know which internal endpoint to give your socket request to.
The easiest way around this is to have both your clients connect to a third party which both of them can see, and then have the third party mediate the communication. This is how most instant-messaging protocols work, for example.
If you have no way to control a third-party entity like that, an alternative to directly connect two clients is to have both clients open up an agreed-upon port, and then map communications on that port to their own internal endpoint. This provides the missing link that the externally visible endpoint (e.g. your home router) needs to deliver the communication to its intended destination.
If your server is behind a NAT router box (and most home computers are, especially if you use WiFi), then it won't be reachable from the outside unless you set up your router to port forward to that server.
What's the IP of your server computer? If it's 192.168.x.x or 10.x.x.x, then it's a non-routable address and can't be reached from outside.
Assuming with running on the same computer you mean that you tell the client the server is at 127.0.0.1 / localhost, it shouldn't be a problem in your code but because of firewalls or routers. If your server is behind a router performing masquerading (i.e., the server doesn't have a public but private IP address like 192.168.x.y for instance), you have to configure the router to pass a connection from the internet to the computer running the server.
Another reason why it doesn't work might be the way you bind your server to the interface. If you specify 127.0.0.1 there, the server will only listen for requests coming from the same system. If you don't specify an address, it will listen on all interfaces.
Edit Your comment indicates that you indeed have the NAT problem like others said. Configuring your router accordingly is probably the easiest solution.
First, test to see if it really works on a LAN; it sounds like you're just assuming it does.
If it works on your local network, perhaps it's failing because the server lacks a public IP, and is running behind a NAT'ing router.