I am building an IoT solution where I will have multiple devices connected to my local network that I want to connect to a hub that controls my solution.
Once I connect a new device I want to connect it with the hub and give it a name to identify it. I would like to implement the detection in an automatic way (so I don´t have to enter IP addresses manually). As an example when a Chromecast is present in a network I can see it in my streaming applications in my phone. So I want to do something similar to connect the hub with the devices.
My ideas so far is that it is two ways of doing this:
The hub scans the network for new devices (either periodically or when I say there is a new device present).
The devices scan the network to find the hub once connected.
Is any of these approaches to prefer over the other and in that case why?
When doing a scan, in whatever direction I choose, what is the most effective way to do the scan? I am doing an implementation using Java and what I have so far is this:
int timeout = 100;
for (int i = 1; i < 255; i++)
{
String host = subnet + "." + i;
if (InetAddress.getByName(host).isReachable(timeout))
{
String hostname = InetAddress.getByName(host).getHostName();
String canonicalHostName = InetAddress.getByName(host).getCanonicalHostName();
System.out.println(host + " is reachable. Hostname: " + hostname + ", CanonicalHostName: " + canonicalHostName);
}
}
What I have seen here is that the hostname returned is for most stuff in my network just the IP address, and not the name that I see as the hostname in my router. I thought I could use the hostname as an identifyer to detect specific devices and understand what they where - but with this small poc that does not seem to work. So how can I identify the devices in a good way?
Is there any library/solution for Java (or Javascript) and ESP8266 to do this already? (Feels like a common problem if implementing "smart home" devices).
There is no single way that we discover devices on a local area network.
Rather than scanning a network, devices generally either use a multicast or broadcast protocol to announce their presence, or they rendezvous at a (usually external) preconfigured server.
Some devices use mDNS - based loosely on the DNS protocol, they multicast packets to advertise their presence on the network. Apple uses mDNS in their products. It's poorly supported on Android, and requires additional software on Windows. mDNS names are normally in the .local domain.
Some devices use UPNP and SSDP - Belkin's Wemo line of products does this. UPNP and SSDP are overly complicated protocols based on XML and SOAP, which are poor choices for devices with limited RAM and processing power like the ESP8266 and ESP32.
Some devices just roll their own protocol. Haiku's "Big Ass Fan" line does this - they broadcast UDP packets with a homegrown protocol that at least initially was vulnerable to a variety of problems. I don't recommend going this route unless you really know what you're doing. Other, established protocols have already had a chance to have the bugs ironed out. Unless you're experienced in protocol design you're more likely to reinvent problems other protocols have had than a wonderful shiny new discoverability protocol.
These kinds of devices will either periodically broadcast or multicast a packet announcing themselves, or the thing you're calling a "hub" would broadcast or multicast a request and the devices would respond to that request.
Not all devices present an interface for direct control over the LAN they're connected to. Some simply rendezvous with a remote server. In this case you discover them by asking the server to enumerate them, and you control them via that server. Google's Nest products work this way - initial provisioning is done via Bluetooth; after that apps communicate with the devices through remote servers.
Devices rendezvousing in this manner are generally preconfigured with the name of the rendezvous server, but they might also be configured with the name of the server during network provisioning (generally an app communicates with them to share wifi credentials; it might also share information about a rendezvous server as well).
We generally do not scan for names in IP address blocks or actively probe for new devices unless we're debugging a network or doing some kind of security sweep.
The process of scanning an IP address block that you described is problematic and unreliable. The only reason it works is that some routers pick up a device's name from the device's DHCP request (or the router may be configured to know the device's name). The router also handles DNS for the devices on the network, generally by forwarding them to the ISP's DNS servers or to DNS servers that the network's owner configured it to use. It intercepts DNS requests for devices whose names it knows and replies to them itself, instead of forwarding them to an external DNS server.
You also have to know about the network configuration to do this properly. Is the network only a /24? What if it's /22? or /16? Are you prepared to scan 2^24 IP addresses if the network is configured as a /8?
Though the router may intercept requests for names and return addresses, it won't necessarily intercept names for addresses and return names.
Scanning also generates an unnecessary network traffic. While your one "hub" scanning may not seem like much, if you have multiple scanners from different makers running it doesn't scale well.
If your "hub" bypasses the router for DNS requests then it also will not be able to resolve names provided by the router.
Also not all routers do this. It's not part of the Internet architecture, it's a convenience feature that some routers provide. You cannot depend on it working.
You could also attempt an active scan of the network, trying to communicate with each IP address on it. We do this for network debugging, but running this continually to detect new devices would be a hostile way to interact with the network.
Network infrastructure devices - switches and routers - of course maintain lists of all active devices on the network. You can access these lists using SNMP (Simple Network Management Protocol), but few consumer switches and routers support SNMP. And allowing access to your network infrastructure to a random piece of software is a network security nightmare.
Your best bet is a simple multicast/broadcast protocol like mDNS. Don't scan, announce or request. It's responsive, doesn't place a burden on the network, doesn't depend on peculiarities of routers, and won't make network administrators hate you.
Long time ago I implemented a similar solution to discover clients connecting to the network. My strategy consisted in taking advantage of the DHCP configuration.
If your devices must ask IP addresses from a Linux/Unix DHCP server under your control, you could configure it to let you know when a device connects or disconnects from the network.
We were using Linux DHCP server, and in the dhcpd.conf there is a section about events. Basically it says that when the DHCP commits to a particular IP address lease for a client it raises an event, and when that happens, it can run a listener you could have defined for that event. In the listener we can ask the DHCP to execute a command, e.g. write the client information (e.g. MAC address) and its IP address lease to a named pipe, and your Java application could just be reading from this queue. (See also dhcp-eval for further details). A similar event can be run when a client releases an IP lease or when a lease expires.
Here's an example
subnet 192.168.1.0 netmask 255.255.255.0 {
option routers 192.168.1.2;
on commit {
set clip = binary-to-ascii(10, 8, ".", leased-address);
set clhw = binary-to-ascii(16, 8, ":", substring(hardware, 1, 6));
execute("/usr/local/sbin/dhcpevent", "commit", clip, clhw, host-decl-name);
}
...
If you can control your DHCP config there is so much you can extract from it.
Related
Is there any way to set the dns for mobile network on android?
I've searched everywhere and only found how to set dns for wifi using wifimanager.
It may not be what you are referring to, but just in case - in a mobile network there is a DNS that is entirely separate from the public internet DNS.
Its purpose is to provide domain name resolution within the mobile networks - for GPRS roaming, networking between different operators IMS system, delivery MMS between operators etc.
Its kept completely separate, largely for security reasons, and the network operator or the GRX/IPX operator sets the DNS - IP Packet exchange (IPX) is a network connecting IP network providers and GPRS Roaming exchange (GRX) is a hub for exchanging traffic between GPRS providers.
The mobile or 3GPP DNS is not set by the mobile device or by an app on the mobile device (unless the app happens to be admin app for a member of the GRX/IPX operator team...).
As a simple example, to try to show how this DNS is actually used, an SGSN node in an operators core IP network may want to get the IP address of a GGSN node in the network that it needs to communicate with - it uses this mobile network DNS system to resolve the domain name of the GGSN. Its fairly clear why you would not want a mobile device to be able to influence this process - otherwise a rogue mobile device could change how the elements of the core network communicate with each other.
Let's say, within a program, there are capabilities to connect to a socket and server(which would be my own computer - localhost) and has all the handling for communicating to the server and back (receiving and sending messages like a chat service). If i send this program as a .jar file to two other users their respective machines, as the users executes the program, it would connect to this socket and server. If all was set up correctly (correctly writes and reads from the socket inputstream and outputstreams) and displays the messages on a GUI chat box, would the two users be able to communicate with this chat service?
Yes, but the main barrier that would prevent users from doing this is a little something called Network Address Translation (NAT).
In short, the vast majority of computers connected to the Internet do not have their own, unique, public IPv4 address (and the vast majority do not have IPv6 addresses either, because ISPs are dragging their feet in rolling out IPv6, but that's another matter entirely).
Dedicated servers often do have (usually multiple) unique IPv4 addresses, but significant numbers of IP addresses are usually only available to hosting companies that operate their own Autonomous System -- and even then, purchasing IPs these days is very expensive, because they're a valuable and rare commodity.
Anyway, two random users on some sort of home Internet connection (like cable, DSL, or fiber) almost certainly have a router or modem of some sort that performs NAT, giving each user device (computer, tablet, etc.) their own private LAN IP, instead of giving each device a unique public IP.
This means that inbound connections from the public Internet are, by default, ignored by the NAT gateway. So if you try to open a socket to your friend's public IP, unless that gateway (modem/router) has been configured correctly, it will just silently ignore your connection attempt, and you'll get a timeout or "connection refused" type message.
There are a few techniques for getting around this, but most of them either require much more complicated application network code, or require the user to mess around in their router configuration pages (which casual users usually aren't willing to do).
Port forwarding: Users must manually configure their router to "forward" public IP ports to a specific private IP, thus allowing the connection attempt on the public IP to pass through the Network Address Translation table to the private LAN. There is no general-purpose way of automating this without user intervention due to the differences between router/gateway devices.
NAT Traversal, which requires one of a variety of NAT-T protocols to be implemented in your program, and requires the explicit support of the desired NAT-T protocol for the routers involved (yours and your friend's). There are loads of different NAT-T protocols available; it'd be up to you to determine which one to use, based on compatibility, availability, etc.
Using an interposing server as an indirect way of routing packets between the two: In this case, you create a server on the public Internet that does have a unique IP (or has ports forwarded to it), and have each of your two client computers initiate an outbound connection to that server. The server would accept connections from both clients, then process the packets and pass them along, like talking to someone through a middleman. This doesn't require any special configuration on the clients, as long as they can make outbound connections on the desired ports (by default, most can, except in restricted enterprise environments where your ports may be limited to a few like 80 and 443.)
You'd need to adopt one of these techniques if your clients are using NAT. If one of your clients has a static IP directly connected to their computer and can accept incoming connections directly (without NAT), your program concept would work without modification, as long as the user behind the NAT initiates the connection to the one with a static IP.
We are working on a Android project with the below requirements.
The application should be able to send data to all the devices which are running our application which exists in the WiFi LAN.
Some payloads are expected to be of size >= 5MB.
The data shouldn't be lost and if lost the client should know the failure.
All the devices should be able to communicate with all other. There will be no message targeted to a specific device instead all the messages should be reached all the devices in the N/W.
No internet hence no remote server.
Study we have done:-
UDP Broadcasting - UDP doesn't guarantee the message delivery but this is a prime requirement in our case. Hence not an option.
TCP - TCP guarantees the message delivery but requires the receiver IP address to be known before hand and in our case we need to send the message to all the devices inside the LAN. Hence not a straight option.
Solutions we are looking into:-
A Hybrid approach - Name one of the devices in the N/W as Server. Post all the messages to a local Server. The Server keeps a open socket to all the devices(which have our application) & when there is a message from a device then it routes the message to all the devices. The disadvantages of this approach are,
Server having multiple sockets open each per device. But in our case we are expecting devices <=5 in LAN.
Server discovery using continuous UDP broadcast.
We want to have all the data in all the devices. So if we newly introduce any device into the LAN then that device needs to get all the data from the server.
So my question, have you any time worked on these kind of hybrid approaches? Or can you suggest any other approaches?
Your hybrid approach is the way to go.
Cleanly split your problem into parts and solve them independently:
Discovery: Devices need to be able to discover the server, if there is any.
Select server: Decide which of your devices assumes the server role.
Server implementation: The server distributes all data to all devices and sends notifications as necessary. Push or pull with notifications does not matter.
Client implementation: Clients only talk to the server. The device which contains the server should also contain a normal client, potentially passing data to the server directly, but using the same abstract protocol.
You could use mDNS (aka Bonjour or zeroconf) for the discovery, but I would not even recommend that. It often createsmore problems than it solves, and it does not solve your 'I need one server' problem. I would suggest you handcraft a simple UDP broadcast protocol for the discovery, which already tells you who the server is, if there is any.
Select server: One approach is to use network meta data which you have anyway, for example 'use the device with the highest IP address'. This often works better than fancy arbitration algorithms. Once you established a server new devices would use this, rather than switching the server role.
Use UDP broadcast for the discovery, with manual heuristic repeats. No fancy logic, just make your protocol resilient against repeated packets and repeat your packets. (Your WLAN router may repeat your packets without your knowledge anyway.)
You may want to use two TCP connections per client, potentially to two different server ports, but that does not matter much: One control connection (always very responsive, no big amounts of data, just a few hundred bytes per message) and one data connection (for the bulk of the data, your > 5 MB chunks). This is so that everything stays responsive.
The system I am developing potentially has a very large number of clients (lets say one million) that need to periodically update a central server with some information. Clients are written in Java.
The specific use-case is that the server backend needs to have an up to date mapping of IP address to clients. But the client IPs are dynamic and subject to (effectively random) change.
The solution I have in mind requires the clients to ping the server to update their IP. The period ideally should be once every minute, but even 1 ping/10 mins is acceptable.
My questions, in sequence:
1M pings per 1 min is over 10k/sec. So first off I want to know
the approaches can scale to handle such a load. This is to know the options available.
Assuming you have more than one solution in mind, which of these
would be the most economical? The cost effectiveness is critically important. I don't have my own data center or
static and fat end-point on the net, so the server application will
need to run on some sort of provider or ultimately on the cloud.
Notes:
I considered running the server from home using my own ISP provided connection, but I am neither sure of the performance issues, nor what my ISP will think about a constant stream of pings.
I can't see how the server can auto-discover these IP changes.
Erik, your problem is much simpler than it seems to have been made to sound.
This problem been around for a decade maybe two. No need to re-invent the wheel here.
Why Polling/Pinging is a Bad Idea
The dynamic IPs provided by ISPs can have a variable lease time, but will often be at least 24-72 hours. Pinging your server every 1-10m will be a horrible waist of resources potentially making over a 4,320 useless HTTP requests PER CLIENT in a 72 hour period. Each request will be say around 300 bytes * 4,320 wasted http requests equals 1.3mb wasted bandwidth multiplied by your target client count of 1 million clients, you are talking about a monthly wasted bandwidth of ~1.2 TB! And that's just the wasted bandwidth, not the other bandwidth you might need to run your app and provide useful info.
The clients need to be smarter than just pinging frequently. Rather they should be able to check if their IP address matches the DNS on startup, then only when the IP changes, send a notification to the server. This will cut down your bandwidth and server processing requirements by thousands of times.
What you are describing is Dynamic DNS
What you are talking about is "Dynamic DNS" (both a descriptive name for the technology and also the name of one company that provides a SaaS solution).
Dynamic DNS is quite simply a DNS server that allows you to very rapidly change the mapping between a name and an IP address. Normally this is useful for devices using an ISP which only provides dynamic IPs. Whenever the IP changes for the router/server on a dynamic IP it will inform the Dynamic DNS server of the change.
The defacto standard protocol for dynamic DNS is well documented. Start here: DNS Update API, I think the specifics you are looking for are here: DynDNS Perform Update. Most commercial implementations out there are very close to the same protocol due to the fact that router hardware usually has a built in DynDNS client which everyone wants to use.
Most routers (even cheap ones) already have Dynamic DNS clients built into them. (You can write your own soft client, but the router is likely the most efficient location for this as your clients are likely being NAT'd with a private IP - you can still do it but at a cost of more bandwidth for public IP discovery)
A quick google search for "dynamic DNS java client" brings up full source projects like this one: Java DynDNS client (untested, just illustrating the power of search)
Other Considerations for your System Architecture
Lets say the IP-client mapping thing gets resolved. You figured it all out and it works perfectly, you always knows the IP for each client. Would you then have a nice reliable system for transferring files to clients from mobile devices? I would say no.
Both mobiles and home computers can have multiple connection types, Wi-Fi, Cellular Data, maybe wired data. Each of these networks may have different security systems in place. So a connection from a cellular data mobile to a wifi laptop behind a home router is going to look very different than a wifi mobile device connecting to laptop on the same wifi network.
You may have physical router firewalls to contend with. Also home computers may have windows firewall enabled, maybe norton internet security, maybe symantec, maybe AVG, maybe zone alarm, etc... Do you know the firewall considerations for all these potential clients?
Maybe you could use SIP as protocol for that purpose ?
Probably the java SIP libs already solved your problem.
Nice app by the way.
I would suggest better tweak you java program to know the IP change and then only hit the web service.
You can do it like,
on your java program initiation extract the IP of machine and store
it in Global variable or better some property file.
Run a batch process/scheduler which will check your IP every 30sec/1 minute for change.Java Quartz Scheduler will come very handy for you.
Invoke the web service in case of a change of IP.
This way it reduces your server role and thus traffic and connections.
You could create your own protocol on top of UDP, for example XML based. Define 3 messages:
request - client requests a challenge from server
challenge - server replies with challenge (basically a random number)
response - client sends username and hashed password + challenge back to the server
It's lightweight and not too traffic-heavy. You can load-balance it to multiple servers at any layer or using load-balancer.
Any average PC could handle million such hits per minute, provided you do server-side in C/C++ (I don't know about java network performance)
Please have a look at how no-ip works. Your requirement is exactly same as what it does.
Do I have the use case right? A community of users all want to receive pictures from each other? You don't want to host the images on the server but broadcast them directly to all the users?
There are two questions here. The first question is "how to know if my own WAN IP address has changed."
If you are not NATed then:
InetAddress.getLocalHost()
will tell you your IP address.
If you are NATed, then using dynamic DNS and resolving your own host name will work.
The second question is something like "How to share pictures between hosts which come and go on the internet".
The possible solution space includes:
IP Multicast, probably with Forward Error Correction and Carouseling, e.g. FLUTE.
File Swarming - e.g. bittorrent.
A Publish/Subscribe message bus solution using Jabber, AMQP, JMS, STOMP or similar. Suitable implementations include RabbitMQ, ActiveMQ, etc. JMS Topics are a key concept here.
The solution should avoid the massive overheads of doing things at the IP level.
I'm writting a java application, and I need to quickly discover any other running clients on any wired or wireless local networks in order to establish a TCP connection.
what's the best way of doing this? Are there libraries or code snippets that would do this?
Multicast UDP is a good way of doing this. It's used in a couple of technologies that support automatic discovery of networked devices over local IP networks (UPnP and ZeroConf).
Multicast UDP is not TCP, but it is still based on IP and, so, uses the same addressing mechanism i.e. IP addresses. Quite often it is compared to radio broadcasting i.e. a multicast sender only needs to send 1 message (i.e. it is like a broadcast) but only clients that are "tuned-in" to the multicast channel will receive it.
You can do a quick search on google or wikipedia for these as a starter, but the basic idea is as follows:
when a client starts, it sends out a multicast UDP "hello" message to some pre-specified multicast address and port (e.g. UPnP uses 239.255.255.250:1900)
existing clients are listening for incoming multicast "hello" messages on the specified address and port - when a client receives one, it sends a response to the sender
the client sending the "hello" message receives a response from each existing client on the network and is now aware of the IP address of each client
If you are looking for libraries to use, UPnP libraries can tend to be a bit heavyweight and a lot of folk generally don't like them, so ZeroConf might be a little more suitable. I don't know of any java implementations of such things but I'm sure you can find some with a little digging around.
A network scan can be very long, even longer on wireless networks. If you need them quickly thru Java you may implement a "meeting point" server on your network. This server listen to a predefined port, clients register on the server on startup and the server can distribute information about the clients on request.
HTH.
I guess you need to do a scan on your application's port on all IPs in your subnet.
Just what are the available IPs - or what is your subnet for that matter?
I'm afraid determining that could turn out to be impossible as the network is designed to be transparent to your application.
So, i'd use brute force: pick your IP and change the last byte. Might be too much, might be not enough though.
Or you send a broadcast (which usually would be targeted at x.x.x.255) and see who answers.
See Datagram Broadcasting and Multicasts. But i think that's not TCP/IP anymore.
There is a JGroups toolkit for reliable multicast communications. It allows automatic discovery of additional clients using Multicast techniques as described in other answers.
It also provides communication APIs on top of multicast sockets.
It is used in a number of projects such as JBoss, Tomcat and more to provide an infrastructure for distributed cache. See more here.