I have a single-threaded non-blocking socket IO server written in Java using nio.
When I have finished writing to a connection, I want to close it.
Does the closing of the channel mean blocking until all buffered writes have been acknowledged by the recipient?
It would be useful to know if, when asynchronously closing, it succeeded or not, but I could live with any errors in the closing being ignored.
Is there any way to configure this, e.g. with setSoLinger() (and what settings would be appropriate?)
(A general discussion beyond Java about Linux and other OS in this respect would be useful to)
Closing in non-blocking mode is non-blocking.
You could put the channel into blocking mode, set a positive linger timeout, and close, and that would block for up to the linger timeout while the socket send buffer was being emptied, but alas Java doesn't throw an exception if the linger timeout expires, so you can't know whether all the data has gone. I reported this bug ten or more years ago and it came back 'will not fix' because of compatiblity concerns. If you can wait until Java 7 comes out I believe the nio2 stuff has this fixed, I certainly requested it, but who knows when that will be?
And even if you have all that, all you know is that the data was sent. You don't know anything about it being received or processed by the recipient application. If you need that you have to build it into your application protocol.
I'm not sure what really happens but I know that close() includes flush() (except in PrintStream and PrintWriter...).
So my approach would be to add the connections to close to a queue and process that queue in a second thread (including error handling).
I understand that your server is single-threaded but a second thread doesn't cost that much, the complexity of the problem is low and the solution will be easy to understand any maintain.
Related
By guarantee I mean the commonly understood feature of TCP that if a packet gets damaged or lost, then it will go unacknowledged and be resent.
Consider the situation (in java) where the sender sends some data, and immediately closes the socket. If that data goes missing or corrupts itself en-route, will the reciever never be able to get it? Or does the senders socket wait and not actually close until all of the appropriate ACKs have been read back in?
I've tried tracing the socket.close() method back myself, but its hard as not only are there are many internal socket implementations but the functionality splits down several paths with ambiguous method names.
Most resource management happens at the OS level below the JVM.
In the case the of TCP protocol it is completely managed by the OS what happens is
Application tells JVM to send data on TCP Stream
JVM access OS functions to perform data send task.
OS manages TCP protocol.
data gets sent over the network.
The data gets copied out of the JVM into the OS memory space so even if the JVM aborted
the TCP contracts should still be handled appropriately.
If the JVM halts or requests the OS to close the socket it should hold the socket
open until the other endpoint acknowledges all data has arrived or the other
endpoint closes.
As the JVM only interfaces into the OS you won't see this management code in the Java
libraries, they assume that management is handled correctly by the OS and the JVM.
Consider the situation (in java) where the sender sends some data, and immediately closes the socket. If that data goes missing or corrupts itself en-route, will the reciever never be able to get it?
TCP will retransmit, at least until its retry timers or counters expire, after which it would give up and rest the connection.
Or does the senders socket wait and not actually close until all of the appropriate ACKs have been read back in?
Yes. This takes place in TCP, not Java.
You should use Socket.shutdownOutput() and then wait for the closing event, as if the remote host closed the connection. This method is a wrapper on the shutdown() C socket function, which accepts a parameter to decide wich network flow to close.
Is there any way of preventing my application from temporarily crashing as soon as I create a socket? This program unfreezes as soon as it receives a connection but it can be misleading to users.
I have tried putting the socket creation methods etc. in a thread and running it from there but that did not work.
Edit: Unless it would be feasible to give the socket a set amount of time and then disconnect once it expires?
Running it in a separate thread is the right way, since the Socket.accept() call is blocking (i.e. it blocks the thread you call it on, until it gets a connection).
There must be something wrong with your thread architecture. Post some code and maybe I can tell You what exactly.
EDIT: Giving the socket a short timeout will either not avoid the blocking, or timeout before someone connects, while getting a connection before the timeout will not have any difference from the current setup.
You program does not crash, it is simply waiting for a connection because the java.io library is blocking.
To prevent waiting for a connection, you can either use the non-blocking java.nio classes or start a new Thread and let this new thread be the one that is waiting for the connection.
I have a typical java client and a server. The client sends some request to the server and waits for the response. The client reads up to say 100 bytes of data from the contained input stream into an array of bytes. It waits for the complete response of 100 bytes to be read within a specified timeout period of say 3 secs. The problem here is to identify if the server went down or crashed while/before writing the response. Basically, we need to identify if the socket was broken or the peer disconnected for some reason. Is there a way to identify this?
How to identify a broken socket connection in Java immediately?
You can't detect it immediately, in Java or any other language. TCP/IP doesn't know, so Java can't know. The only sure way to detect a broken TCP connection is by writing to it and catching IOExceptions, and they won't happen immediately.
The best way to identity the connection is down is to timeout the connection. i.e. you expect a response in a given amount of time and flag if that response does not come as you expect.
When you have a graceful disconnection (.e.g the other end calls close()) the read on the connection will let you know once the buffer has been drained.
However, if there some other type of failure, you might not be notified until the OS times out the connection (e.g. after 3 minutes) and indeed, you may want to keep the connection. e.g. if you pull the network cable out for 10 seconds and put it back in, that doesn't need to be a failure.
EDIT: I don't believe its a good idea to be too aggressive in automatically handling connection/service "failures". This is usually better handled by a planned fix to the system, based on investigation of the true cause. e.g. increased bandwidth, redundant connectivity, faster servers, code fixes.
If connection is broken abnormally, you will receieve IOException when reading; that normally happens quite fast, but there is no guarantees about time - all depends on the OS, network hardware, etc. If remote end gracefully closes the socket, you'll read -1 as next byte.
Assuming everything else works, if the remote peer - the TCP server - was killed then the TCP client will normally receive a TCP RST (reset) and you'll get an IOException in your client application.
However, there are lots of other things that can go wrong besides a process being killed. Basically anything on the network path between the two processes: a cable is yanked, a router dies, a firewall dies, etc. All of this will not immediately be detected.
For the above reasons the general rule is - as pointed out in the answer from EJP - that a broken connection can only be detected by writing to it. This is why it is always recommended that a TCP client and TCP server exchange some type of heartbeat messages at regular intervals. There are different ways to do this. I like best the method where the TCP client will - in the absence of data being received from the TCP server - send a heartbeat message to the server and expect a reply back within a certain time period. This way heartbeat messages will only be sent when really needed.
A sub-optimal approach - if you cannot implement true heartbeating - is to always read with a timeout. Set the timeout on the socket and then catch java.net.SocketTimeoutException. This will allow you to know that no data has been received on socket during x milliseconds.
It should be mentioned that there's one scenario where you don't have to use heartbeating, nor using the socket timeout: if the TCP client and the TCP server communicate over a loopback interface then a broken connection will always be propagated to both the TCP client application and the TCP server application. This is because, in this case, there's really no network infrastructure between the two processes. So if you have an existing application which isn't well-designed with respect to its TCP communication (i.e. it doesn't implement some form of heartbeating or at least reading with a timeout), then as a last resort you may 'fix' the problem by moving the two application onto the same host and let them communicate over the loopback interface.
I have a Java and Tomcat-based server application which initiates many outbound HTTP requests to other web sites. We use Jakarta's HTTP Core/Client libraries, very latest versions.
The server locks up at some point since all its worker threads are stuck trying to close completed HTTP connections. Using 'lsof' reveals a bunch of sockets stuck in TCP CLOSE_WAIT state.
This doesn't happen for all, or even most connections. In fact, I saw it before and resolved it by making sure to set the Connection: Close response header. So that makes me think it may be bad behavior of remote servers.
It may have come up again since I moved the app to a totally new service provider -- different OS, network situation.
But, I am still at a loss as to what I could do, if anything, to work around this. Some poking around on the internet didn't turn up anything I'm not already doing. Just thought I'd ask if anyone has seen and solved this?
I'm not sure how much you know about TCP. A TCP client ends up in the CLOSE_WAIT state when it is in the ESTABLISHED state and receives a FIN packet. The CLOSE_WAIT state means that it is waiting to receive a close command from the application layer - in this case, that means it's waiting for close() to be called on the socket.
So, my guess would be that calling close() in the worker threads will fix the problem. Or are you already doing this?
I believe I might have found a solution -- at least, these changes, together, seem to have made the problem go away.
Call HttpEntity.consumeContent() after you're done reading from its InputStream, to double-check the content is consumed and the framework releases the connection
For good measure, I call ClientConnectionManager.closeExpiredConnections() and ClientConnectionManager.closeIdleConnections(0L, TimeUnit.MILLISECONDS) just after this, to press the framework to release anything it's done with right away. This might be overkill.
OK the above didn't quite do the trick. In the end, I had to use a SingleClientConnManager, and create and shut it down for every request. This did the trick. I assume that this is what's needed to make sure the connection is closed.
I'm working through the 'Simple Point-to-Point Example' section of the Sun JMS tutorial (sender source, receiver source), using Glassfish as my JMS provider. I've set up the QueueConnectionFactory and Queue in the Glassfish admin UI, and added the relevant JARs to my classpath and the receiver is receiving the messages sent by the sender.
However, neither sender nor receiver terminate. The main thread exits normally (after successfully calling queueConnection.close()) but two non-daemon threads are left hanging around:
iMQReadChannel-0
imqConnectionFlowControl-0
It seems (from this java.net thread) that the reason is that queueConnection.close() just returns the connection to the pool, rather than really closing it. I can't find any way to tell the pool to shutdown, so the only option I'm left with is System.exit(), which feels wrong.
I've tried setting the minimum pool size to 0, the maximum pool size to 1 and the idle timeout to 10 seconds but it seems to make no difference. Even when I just lookup the connection factory and don't ask for a connection, these two threads are still started and don't terminate.
Any help much appreciated!
Why don't you simply terminate with a System.exit(0)? Given the sample, the current behavior is correct (a Java program terminates when all non-daemon threads end).
Maybe you can have the samples shutting down properly by playing with client library's properties (idle time, etc...), but it seems others ( http://www.nabble.com/Simple-JMS-Client-doesn%27t-quit-td15662753.html) still experience the very same problem (and, anyway, i still don't understand what the point is).
Good news for us. "Will not fixed"
http://java.net/jira/browse/GLASSFISH-1429?focusedCommentId=85555&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_85555