We Keep Coding

Apache Mina, How to detect when you're sending messages using an invalid socket to the client side?

I have a server setup using MINA version 2.
I don't have much experience with sockets and tcp.
The problem is if I make a connection to my server, and then unplug my internet and close the connection, (Server doesn't get notification of the connection being closed) the server will forever think that my connection is still active and valid.
The server will continue to send messages to my connection, and doesn't throw any exceptions even though there is nothing on my computer binded to the local port.
How can I test that the connection still exists?
I've tried running MINA logging in debug mode, and logging the
IoSession.isConnected() IoSession.isActive IoSession.isClosing
They always return true, true, false. Also, in debug mode, there was no useful information stating that the connection was lost. It just logged the regular "sent message" stuff, as if there was nothing wrong.
From using Flash actionscript, I have had experiences where flash will throw errors that it's operating on an invalid socket. That leads me to believe that it's saying the socket on the server is no longer valid for the connection. So in other words if flash can detect invalid sockets, a Java server should be able to detect it too correct?
If there is truly no way to detect dead connections, I can always setup a connection keep alive routine where the client is constantly sending an "I'm here" message to the server, and the server closes sessions that havent had an incoming message for a period of seconds.
EDIT: After learning that "sockets" are private and never shared over the network I managed to find better results for my issue and I found this SO thread.
Java socket API: How to tell if a connection has been closed?
Unfortunately
IOException 'Connection reset by peer' Doesn't occur when I write to
the IoSession in MINA.
Edit:
Is there any way at all in Java to detect when an ACK to a TCP packet was not received after sending a packet? An ACK Timeout?
Edit:
Yet apparantly, my computer should send a RST to the server? According to this answer. https://stackoverflow.com/a/1434592/4425643
But that seems like a bad way of port scanning. Is this how port scanning works? Port scanners send data to a port and the victim's service responds with a RST? Sorry I think I need a new question for all this. But it's odd that MINA doesn't throw connection reset by peer when it sends data. So then my computer doesn't send a RST.

The concept of socket or connection in Internet protocols is an illusion. It's a convenient abstraction that is provided to you by the operating system and the TCP stack, but in reality, it's all fake.
Under the hood, everything on the Internet takes the form of individual packets.
From the perspective of a computer sending packets to another computer, there is no built-in way to know whether that computer is actually receiving the packets, unless that computer (or some other computer in between, like a router) tells you that the packets were, or were not, received.
From the perspective of a computer expecting to receive packets from another computer, there is no way to know in advance whether any packets are coming, will ever come, or in what order -- until they actually arrive. And once they arrive, just the fact that you received one packet does not mean you'll receive any more in the future.
That's why I say connections or sockets are an illusion. The way that the operating system determines whether a connection is "alive" or not, is simply by waiting an arbitrary amount of time. After that amount of time -- called a timeout -- if one side of the TCP connection doesn't hear back from the other side, it will just assume that the other end has been disconnected, and arbitrarily set the connection status to "closed", "dead" or "terminated" ("timed out").
So:
Your server has no clue that you've pulled the plug on your Internet connection. It has no way of knowing that.
Your server's TCP stack has been configured a certain way to wait an arbitrary amount of time before "giving up" on the other end if no response is received. If this timeout is set to a very large period of time, it may appear to you that your server is hanging on to connections that are no longer valid. If this bothers you, you should look into ways to decrease the timeout interval.
Analogy: If you are on a phone call with someone, and there's a very real risk of them being hurt or killed, and you are talking to them and getting them to answer, and then the phone suddenly goes dead..... Well, how long do you wait? At what point do you assume the other person has been hurt or killed? If you wait a couple milliseconds, in most cases that's too short of a "timeout", because the other person could just be listening and thinking of how to respond. If you wait for 50 years, the person might be long dead by then. So you have to set a reasonable timeout value that makes sense.

What you want is a KeepAlive, heartbeat, or ping.
As per #allquicatic's answer, there's no completely reliable built-in method to do this in TCP. You'll have to implement a method to explicitly ask the client "Are you still there?" and await an answer for a specified amount of time.
https://en.wikipedia.org/wiki/Keepalive
A keepalive (KA) is a message sent by one device to another to check that the link between the two is operating, or to prevent this link from being broken.
https://en.wikipedia.org/wiki/Heartbeat_(computing)
In computer science, a heartbeat is a periodic signal generated by hardware or software to indicate normal operation or to synchronize other parts of a system.[1] Usually a heartbeat is sent between machines at a regular interval in the order of seconds. If a heartbeat isn't received for a time—usually a few heartbeat intervals—the machine that should have sent the heartbeat is assumed to have failed.[2]
The easiest way to implement one is to periodically send an arbitrary piece of data - e.g. a null command. A properly programmed TCP stack will timeout if an ACK is not received within its specified timeout period, and then you'll get a IOException 'Connection reset by peer'
You may have to manually tune the TCP parameters, or implement your own functionality if you want more fine-grained control than the default timeout.

The TCP framework is not exposed to Java. And Java does not provide a means to edit TCP configuration that exists on the OS level.
This means we cannot use TCP keep alive in Java efficiently because we can't change its default configuration values. Furthermore we can't set the timeout for not receiving an ACK for a message sent. (Learn about TCP to discover that every message sent will wait for an ACK (acknowledgement) from the peer that the message has been successfully delivered.)
Java can only throw exceptions for cases such as a timeout for not completing the TCP handshake in a custom amount of time, a 'Connection Reset by Peer' exception when a RST is received from the peer, and an exception for an ACK timeout after whatever period of time that may be.
To dependably track connection status, you must implement your own Ping/Pong, Keep Alive, or Heartbeat system as #Dog suggested in his answer. (The server must poll the client to see if it's still there, or the client has to continuosly let the server know it's still there.)
For example, configure your client to send a small packet every 10 seconds.
In MINA, you can set a session reader idle timeout, which will send an event when a session reader has been idle for a period of time. You can terminate that connection on delivery of this event. Setting the reader timeout to be a bit longer than the small packet interval will account for random high latency between the client and server. For example, a reader idle timeout of 15 seconds would be lenient in this case.
If your server will rarely experience session idling, and you think you can save bandwidth by polling the client when the session has gone idle, look into using the Apache MINA Keep Alive Filter.
https://mina.apache.org/mina-project/apidocs/org/apache/mina/filter/keepalive/KeepAliveFilter.html

Java Sockets Local Connection Between 2 clients - Design

I have an application where the user (client #1) enters a local ip and a port and the application sends a picture to client #2 (who is also using the same application). However for the final application, I do not want the user to enter the local ip because they will not know this information, and I want my program to automatically figure this out.
My first idea:
Originally, I thought that I could scan all the local ip's for an open port, but this would take way too long.
My second idea:
My next idea was to have the clients send their local hostnames to a remote server which then swaps them and sends them back to the clients.
However, I do not want to run a dedicated server for my second idea.
Because this is more of a design question, I am not including any code but I will do so if necessary.
Do you guys have any ideas on how I should design my application to automatically figure out the local ips?
I did try to google this but couldn't figure out a solution, and so I gave up after an hour and just put my question here.

you can use something like jgroups (allowing discovery based on multicast [lan] etc) or some peer-to-peer implementatons for that, although the latter require at least some servers for initial discovery.
in principle that works the way, that the clients send a message out to "the world" using some well known address and wait for someone to answer. each client itself waits meanwhile for such a message and replies it with information how to "connect" to the current client. This can be done via a so called blackboard, where this bb is either a special tcp-address for multicast-messages (the os/tcp sends the message to all clients listening concurrently) or one or more servers (seeds) that take and coordinate the request and the "membership" lists. anyway, there are some tools ;)

How do I handle a single text io stream with multiple inputs and outputs?

So I'm working through a bit of a problem, and some advice would be nice. First a little background, please excuse the length.
I am working on a management system that queries network devices via the TL1 protocol. For those unfamiliar with the protocol, the short answer is that is is a "human readable" language that communicates via a text based IO stream.
I am using Spring and Jsch to open a port to the remote NE (network element), login, run the command, then close the connection. There are two kinds of ways to get into the remote NE's, either directly (via the ssh gateway) if the element has a tcp/ip address (many are osi only), or through an ems (management system) of some type using what is called a "northbound interface".
Either way, the procedure is the same.
Use Jsch to open a port to the NE or ems.
Send login command for the NE ex. "act-user<tid>:<username>:UniqueId::<password>;"
Send command ex. "rtrv-alm-all:<tid>:ALL:uniqueid::,,,,;"
Retrieve and process results. The results of the above for example might look something like this...
RTRV-ALM-ALL:foo:ALL:uniqueid;
CMPSW205 02-01-11 18:33:05
M uniqueid COMPLD
"01-01-06:MJ,BOARDOUT-ALM,SA,01-10,12-53-58,,:\"OPA_C__LRX:BOARD EXTRACTED\","
;
The ; is important because it signals the end of the response.
Lastly logout, and close the port.
With Spring I have been using the ThreadPoolTaskExecutor quite effectively to do this.
Until this issue came up ...
With one particular ems platform (Hitachi) I ran across a roadblock with my approach. This ems handles as many as 80 nodes through it. You connect to the port, then issue a command to login to the ems, then run commands pointing to the various NE's. Same procedure as before, but here is the problem...
After you login into the ems, the next command, no matter what it is, will take up to 10 minutes to complete. until that happens, all other commands are blocked. After this initial wait all other commands work quickly. There appears to be no way to defeat this behaviour (my suspicion is that there is some NE auto-discovery happening during this period).
Now the thrust of my question...
So my next approach for this platform would be to connect to the ems, login to it, and keep the connection open, and just pass commands to the various NE's. That would mean a 10 minute delay after the application (web based) first loads, but would be fine after this point.
The problem I have is how best to do this. Having a single text based iostream for passing this stuff through looks like a large bottleneck, plus multiple users will be using the application, how do I handle multiple commands and responses against this single iostream? I can open a few iostreams (maybe up to 6) on this ems, but that also complicates sorting out what goes where.
Any advice on direction would be appreciated.

Look at using one process per ems so that communication to each ems is separated. This will at least ensure that communications with other ems's are unaffected by the problems with this one.
You're going to have to build some sort of a command queuing system so that commands sent to the Hitachi ems don't block the user interface until they are completed. Either that, or you're going to have to put a 10 minute delay into the client software before they can begin using it, or a 10 minute delay into the part of the interface that would handle the Hitachi.
Perhaps it would be a good policy to bring up the connection and immediately send some sort of ping or station keeping idle command - something benign that you don't care about the response, or gives no response, but will trigger the 10 minute delay to get it over with. Your users can become familiar with this 10 minute delay and at least start the application up before getting their coffee or something.
If you can somehow isolate the Hitachi from the other ems's in the application's design, this would really ensure that the 10 minute delay only exists while interfacing with the Hitachi. You can connect and issue a dummy command, and put the Hitachi in some sort of "connecting" state where commands cannot be used until the result comes in, and then you change the status to ready so the user can interact with it.
One other approach would be to develop some sort of middleware component - I don't know if you've already done this. If the clients are all web-based, you could run a communications piece on the webserver which takes connections from the clients and pipes them through one piece on the webserver which communicates with all of the ems's. When this piece starts up on the webserver, it can connect to each ems and send some initial ping command which starts the 10 minute timer. Once this is complete, the piece on the webserver could send keepalive messages every so often, again some sort of dummy command, to keep the socket alive so it wouldn't have to reset and go through the 10-minute wait time again. When the user brings up the website, they can communicate with this middleware server piece which would forward the requests to the appropriate ems and forward the response back to the client -- all through the already open connection.

Client Server communication in Java - which approach to use?

I have a typical client server communication - Client sends data to the server, server processes that, and returns data to the client. The problem is that the process operation can take quite some time - order of magnitude - minutes. There are a few approaches that could be used to solve this.
Establish a connection, and keep it alive, until the operation is finished and the client receives the response.
Establish connection, send data, close the connection. Now the processing takes place and once it is finished the server could establish a connection to the client to send the data.
Establish a connection, send data, close the connection. Processing takes place. client asks server, every n minutes/seconds if the operation is finished. If the processing is finished the client fetches the data.
I was wondering which approach would be the best way to use. Is there maybe some "de facto" standard for solving this problem? How "expensive" is opening a socket in Java? Solution 1. seems pretty nasty to me, but 2. and 3. could do. The problem with solution 2. is that the server needs to know on which port the client is listening, while solution 3. adds some network overhead.

is good enought
will not work at many situations, for example wne client is under firewall, NAT, and so on. Server usually accepts incoming connections from everywhere, desktops usualy not
better than 1 just because you will haven't problems when connection is lost
solutions 1+3 - make long waiting connections, with periodical sleep and reconnect after. I mean: connect to server, wait 30 sec for data, if no data received, sleep for 10 sec, loop.
Opening sockets is sometimes expensive, but not so expensive that your data processing.

I see an immediate problem with option 2. If the client is behind a firewall, he might very well be allowed to connect and do the request, but the server might be prevented to connect back to the cilent.
As you say, option 1 looks a bit nasty (not too nasty though, could work well), so among the options listed, I would go for option 3. Perhaps the server could estimate the time that's left of the processing, and hint the client, in each poll, of when it's about time to check back.

polling a HTTP server from J2ME client

I have a J2ME app running on my mobile phone(client),
I would like to open an HTTP connection with the server and keep polling for updated information on the server.
Every poll performed will use up GPRS bytes and would turn out expensive in the long run, as GPRS billing is based on packets sent and received.
Is there a byte efficient way of polling using the HTTP protocol?.
I have also heard of long polling, But I am not sure how it works and how efficient it would be.
Actually the preffered way would be for the Server to tell the phone app that new data is ready to be used that way polling won't be needed to be done, however I don't know of these techniques especially in J2ME.

If you want solve this problem using HTTP only, long polling would be the best way. It's fairly easy. First you need to setup an URL on server side for notification (e.g. http://example.com/notify), and define a notification protocol. The protocol can be as simply as some text lines and each line is an event. For example,
MSG user1
PHOTO user2 album1
EMAIL user1
HEARTBEAT 300
The polling thread on the phone works like this,
Make a HTTP connection to notification URL. In J2ME, you can use GCF HttpConnection.
The server will block if no events to push.
If the server responds, get each line and spawn a new thread to notify the application and loopback to #1.
If the connection closes for any reason, sleep for a while and go back to step 1.
You have to pay attention to following implementation details,
Tune HTTP timeouts on both client and server. The longer the timeout, the more efficient. Timed out connection will cause a reconnect.
Enable HTTP keepalive on both the phone and the server. TCP's 3-way handshake is expensive in GPRS term so try to avoid it.
Detect stale connections. In mobile environments, it's very easy to get stale HTTP connections (connection is gone but polling thread is still waiting). You can use heartbeats to recover. Say heartbeat rate is 5 minutes. Server should send a notification in every 5 minutes. If no data to push, just send HEARTBEAT. On the phone, the polling thread should try to close and reopen the polling connection if nothing received for 5 minutes.
Handling connectivity errors carefully. Long polling doesn't work well when there are connectivity issues. If not handled properly, it can be the deal-breaker. For example, you can waste lots of packets on Step 4 if the sleep is not long enough. If possible, check GPRS availability on the phone and put the polling thread on hold when GPRS is not available to save battery.
Server cost can be very high if not implemented properly. For example, if you use Java servlet, every running application will have at least one corresponding polling connection and its thread. Depending on the number of users, this can kill a Tomcat quickly :) You need to use resource efficient technologies, like Apache Mina.
I was told there are other more efficient ways to push notifications to the phone, like using SMS and some IP-level tricks. But you either have to do some low level non-portable programming or run into risks of patent violations. Long polling is probably the best you can get with a HTTP only solution.

I don't know exactly what you mean by "polling", do you mean something like IMAP IDLE?
A connection stays open and there is no overhead for building up the connection itself again and again. As stated, another possible solution is the HEAD Header of a HTTP Request (forgot it, thanks!).
Look into this tutorial for the basic of HTTP Connections in J2ME.
Pushing data to an application/device without Push Support (like a Blackberry) is not possible.

The HEAD HTTP request is the method that HTTP provides if you want to check if a page has changed or not, it is used by browsers and proxy servers to check whether a page has been updated or not without consuming much bandwidth.
In HTTP terms, the HEAD request is the same as GET without the body, I assume this would be only a couple hundred bytes at most which looks acceptable if your polls are not very frequent.

The best way to do this is to use socket connection. Many application like GMail use them.