now this problem is bugging me for a while.
In a working application that i work on, i use SocketChannel in non-blocking mode to communicate with embedded devices.
Now i receive sporadically corrupted data.
On some PCs it does not happen, now it happens on mine.
But when I change too much in the program, the problem disappears.
So much might have effects. The timing, the network interface hardware, win7, the java version, the company firewall, ...
The data reading boils down to this code:
byteBuffer.compact();
socketChannel.read(byteBuffer); // <<< problem here ?
byteBuffer.flip();
if( byteBuffer.hasRemaining() ){
handleData( byteBuffer );
}
This is run in the same thread as the writing, when the selector wakes up and the interest op OP_READ is set.
This code is the only place where byteBuffer is referenced. socketChannel is used only from the same thread when writing.
I instrumented the code, so i can printout the content of the last few read() calls, when the error happens. At the same time I analyze the network traffic on Wireshark. I added lots of asserts to check the bytebuffer integrity.
In Wireshark, the received stream looks good. No DUP-ACK or something else suspicious. The last read() calls match exactly with the data in Wireshark.
In Wireshark, i see many small TCP frames receiving with 90 bytes of payload data in intervals like 10ms arriving. Normally the Java thread reads the data as well all 10ms when it is just arrived.
When it comes to the problem, the Java thread is a bit delay, as the reading happens after 300ms, and the read returns with like ~3000 bytes which is plausible. But the data is corrupted.
The data looks like, if it was copied into the buffer and concurrently received data has overwritten the first data.
Now I don't know how to proceed. I cannot create a small example, as this only rarely happens and I don't know the exact condition which is needed.
Can someone give a hint?
How can I prove, it is the Java lib or not?
What conditions may be important to look at, too?
thanks
Frank
29-June-2015:
Now i was able to build a example for reproduction.
There is one Sender and a Receiver program.
The Sender is using blocking IO, first waiting for a connection, then sending 90 byte blocks every 2ms. The first 4 byte are a running counter, the remaining is not set. The Sender uses setNoTcpDelay(true).
The Receiver is using non-blocking IO. First it connects to the Sender, then it read the channel whenever the selection key is ready for it. Sometime, the read loop does a Thread.sleep(300).
If they run on the same PC over the loopback, this works for me all the time. If I put the Sender onto another PC, directly connected over LAN, it triggers the error. Checking with Wireshark, the traffic and the data sent looks good.
To run, first start the Sender on one PC, then (after editing the hostaddress) start the Receiver.
As long as it works, it prints a line about every 2 seconds. If it fails, it prints information about the last 5 read() calls.
What i found to be the trigger:
The sender has configured setNoTcpDelay(true)
The receiver has sometimes a Thread.sleep(300) before doing the read().
thanks
Frank
buf.order(ByteOrder.BIG_ENDIAN);
This is the default. Remove this.
buf.clear();
The buffer is already empty, because you just allocated it. Remove this.
buf.limit(0);
The limit is already zero after the clear() and also after the initial allocation. Remove this.
while( true ) {
There should be a select() call here.
Iterator<SelectionKey> it = selector.selectedKeys().iterator();
// ...
if( key == keyData && key.isConnectable() ) {
ch.finishConnect();
This method can return false. You're not handling that case.
// ...
if( key == keyData && key.isReadable() ) {
// ...
readPos += ch.read(buf);
Completely incorrect. You are entirely overlooking the case where read() returns -1, meaning the peer has disconnected. In this case you must close the channel.
// without this Thread.sleep, it would not trigger the error
So? Hasn't the penny dropped? Remove the sleep. It is completely and utterly pointless. The select() will block until data arrives. It doesn't need your help. This sleep is literally a waste of time.
if( rnd.nextInt(20) == 0 ) {
Thread.sleep(300);
}
Remove this.
selector.select();
This should be at the top of the loop, not the bottom.
I turned out to be a driver problem, at least it seems so.
I used the USB to Ethernet adapater "D-Link E-DUB100 Rev A".
Because of wireshark showing the correct data, i thought to eliminate the hardware a possible failure cause.
But meanwhile i tried "D-Link E-DUB100 Rev C1" and the problem disappeared.
So I assume it is a problem in the delivered drivers from D-Link for Rev A. And with Rev C1 it might use a system driver that does not have this problem.
thx for all taking the time to read my question.
Related
I am trying to figure out why the below close function is being called in netty's(3.10.5) NioWorker. The full Class can be found here
if (ret < 0 || failure) {
k.cancel(); // Some JDK implementations run into an infinite loop without this.
close(channel, succeededFuture(channel));
return false;
}
I am thinking there may be a couple of reasons like the host going down or host closing the channel after some time but I was thinking someone else who has worked with NioWorker might know better.
I have two different systems and this code is called a few time per day but the other system has like 100 per day with the same number of traffic. I am trying to find out why this might be the case.
There are 2 conditions for when this function is called:
There was an error condition when reading from the channel, this is signalled by the failure variable
The channel has reached end of stream, this happens if the other sides closes the channel cleanly (ie, using the TCP FIN flag)
I have a custom NIO server but I am seeing a behavior that I can not reproduce nor explain: the selector keeps triggering on a certain selection key with isReadable() == true but when I read from the channel, there is no data.
I have:
triple checked that the EOS are respected in all scenarios
built custom clients that try all kinds of funky combinations of invalid data to trigger the bug but can not trigger it myself
looked through the apache mina code to see if they do anything special as compared to my server
tried different versions of the JDK (8_111 and 8_121)
triple checked that the selection key is removed from the iterator of the selected key set in a finally block that wraps around everything after iterator.next() so it should not be a ghost key
Everything turns up empty, every weird thing I try on the client side is correctly handled by the server but nonetheless approximately every four hours (you can almost set a clock to it), an IP from Russia connects to the server and triggers the bug.
At that point the selector goes into overdrive continuously triggering the channel and the read process attached to it which keeps reporting 0 bytes incoming.
So two questions:
apart from actual data and an EOS, what else can trigger a read operation on a selector?
if I can not find the actual problem, is it ok to simply check for x amount of subsequent read triggers that turn up with no data and conclude that I should close the socket? There are read timeouts in place but due to the CPU-intensive nature of the bug they are too long for my comfort
UPDATE:
it is not the server socket channel triggering the reads but one of the accepted client channels, as such it can't (shouldn't?) be an incoming connection
by default only OP_READ is registered, OP_WRITE is registered sporadically if the internal buffers still contain data but are unregistered once the data has been sent
the read timeouts mentioned are custom timeouts, the parser will keep track of when the first data for a message comes in and if it takes too long to complete, it will trigger a read timeout
If I knew where the problem was I could provide some code of that part but the whole server is too big to paste here.
UPDATE 2
In the debug things that I have added, I print out the following state:
selectionKey.isReadable() + "/" + selectionKey.isValid() + "/" + selectionKey.channel().isOpen()
All three booleans are always true.
Impossible to answer this properly without some code, but:
apart from actual data and an EOS, what else can trigger a read operation on a selector?
If it's a ServerSocketChannel, an incoming connection.
if I can not find the actual problem, is it ok to simply check for x amount of subsequent read triggers that turn up with no data and conclude that I should close the socket?
No. Find the bug in your code.
There are read timeouts in place
There can't be read timeouts in place on non-blocking sockets, and as you're using a Selector you must be using non-blocking sockets.
but due to the CPU-intensive nature of the bug they are too long for my comfort
I don't know what this means.
The same issue happened to me also, check if you have kept the connection opened with the client. If it is so then selection key will be triggered with isReadable() is true, even though client is not actually sending any data.
Suppose a simple network model: A has successfully created a TCP connection to B, and they are communicating with each other like this
A <----------> B
I know that if the program on A dies (such as core dump), that will cause a RST packet to B. So any read attempt of B will lead to an EOF, and any write attempt of B will lead to SIGPIPE. Am I right?
If, however, suppose the network has broken down (such as cable/router failure) on A, what happens to the read/write attempt of B? In my situation, all the sockets has been set to non-blocking. As a result, is it impossible for me to detect network error?
By the way, I notice that there is an option SO_KEEPALIVE in socket which may be useful to me http://tldp.org/HOWTO/html_single/TCP-Keepalive-HOWTO/. But I wonder how much the cost will be if I set the probing interval to 2~3 second (which by default is 75 seoncd)? And it seems interval configuration is a global one, so is this gonna affect all the sockets on the machine?
Final question...
Say the network has broken down and any write attempt would cause EPIPE some time later. If, however, instead of trying to write, I put this socket into epoll device, what will happend then? Will epoll_wait return EPOLLHUP or EPOLLERR event?
There's numerous other ways a TCP connection can go dead undetected
someone yanks out a network cable inbetween.
the computer at the other end gets nuked.
a nat gateway inbetween silently drops the connection
the OS at the other end crashes hard.
the FIN packets gets lost.
undetectable errors: A router in-between the endpoints may drops packets.(including control packets)
reff
In all cases you can know about it when you try to write on socket this cause through SIGPIPE error in your program and terminate it.
By read() it can't be know whether other-side live or not. Thants Why SO_KEEPALIVE useful. Keepalive is non-invasive, and in most cases, if you're in doubt, you can turn it on without the risk of doing something wrong. But do remember that it generates extra network traffic, which can have an impact on routers and firewalls.
And this affects all sockets on your machine too!(you are correct). And Because SO_KEEPALIVE increase traffic and consume CPU. It's best to set the SIGPIPE handle, if there is a chance application will ever write to a broken connection.
Also use SO_KEEPALIVE at reasonable place in the application. It's poor to use it for whole connection duration (i.e do use so_keepalive when server works for long on client query).
Setting the probing interval Dependends on your application or say
Application layer protocol.
Though enabling TCP keepalive, you'll detect it eventually - at least during a couple of hours.
Say if the network has broken down and however, instead of trying to write, socket is puted into some epoll device :
The second argument in epoll:
n = epoll_wait (efd, events, MAXEVENTS, -1);
Set with correct event-related code, Good practice is to check this code for
caution as follow.
n = epoll_wait (efd, events, MAXEVENTS, -1);
for (i = 0; i < n; i++)
{
if ((events[i].events & EPOLLERR) ||
(events[i].events & EPOLLHUP) ||
(!(events[i].events & EPOLLIN)))
{
/* An error has occured on this fd, or the socket is not
ready for reading (why were we notified then?) */
fprintf (stderr, "epoll error\n");
close (events[i].data.fd);
continue;
}
else if (sfd == events[i].data.fd)
{
/* We have a notification on the listening socket, which
means one or more incoming connections. */
// Do what you wants
}
}
Where EPOLLRDHUP means is:
Stream socket peer closed connection, or shut down writing half of connection. (This flag is especially useful for writing simple code to detect peer shutdown when using Edge Triggered monitoring.)
I know that if the program on A dies (such as core dump), that will cause a RST packet to B. So any read attempt of B will lead to an EOF, and any write attempt of B will lead to SIGPIPE. Am I right?
Partially. RST causes an ECONNRESET, not an EOF, when reading, and EPIPE when writing.
If, however, suppose the network has broken down (such as cable/router failure) on A, what happens to the read/write attempt of B? In my situation, all the sockets has been set to non-blocking. As a result, is it impossible for me to detect network error?
Impossible on read alone, unless you use a read timeout, e.g. via select(), and take a timeout as a failure, which it mightn't be. On write you will eventually get an EPIPE, but it could take some time and several attempts due to buffering and retries.
As some background:
I have a connection to a server with a SocketChannel, SelectionKey...etc. On the client end, if I want to send something to the server, I just write my data into a ByteBuffer and send it through the socket channel. If all of it was written, I'm done and can return to OP_READ. If not all of it was written, I take the bytes left over, store them in a "to send" buffer somewhere and mark OP_WRITE on the key (is it a good idea to replace OP_READ so its only write?).
Therefore, the next time I call selectNow(), I'm assuming it will recognize OP_WRITE and attempt to flush more data through (which I will attempt to do by entering another writing loop with the data to write, and repeating the previous if needed).
This leads me to two questions:
Am I supposed to leave it in OP_WRITE until all the data has been flushed through? Or should I change to OP_READ and attempt any reads in between?
If the writing channel is full and I can't write, do I just keep looping until I can start writing stuff through? If the connection suddenly gets choked, I'm unsure if I'm supposed to just write what I can, flip back to OP_READ, attempt to read, then flip back to OP_WRITE. From what I've read, this appears to not be the correct way to do things (and may cause large overhead constantly switching back and forth?).
What is the optimal way to handle reading and writing bulk data when the buffers both may become full?
Reading sounds easy cause you just loop through until the data is consumed, but with writing... the server may be only writing and not reading. This would leave you with quite a full send buffer, and cycling around forever on OP_WRITE without reading would be bad. How do you avoid that situation? Do you set a timer on which you just stop attempting to write and start reading again if the send buffer is not clearing up? If so, do you remove OP_WRITE and remember it for later?
Side question: Do you even need OP_READ to read from the network? I'm unsure if it's like OP_WRITE where you only mark it in a specific case (just in case I', doing it wrong, since I have it on OP_READ 99.9% of the time).
Currently I just set my key to OP_READ and then leave it in that mode, waiting for data, and then go to OP_WRITE if and only if writing fails to send all the data (with a write() value of 0).
Am I supposed to leave it in OP_WRITE until all the data has been flushed through? Or should I change to OP_READ and attempt any reads in between?
There are differing views about that. Mine is that the peer should be reading every part of the response you're sending before he sends a new request, and if he doesn't he is just misbehaving, and this you shouldn't encourage by reading ahead. Otherwise you just run out of memory eventually, and you shouldn't let a client do that to you. Of course that assumes you're the server in a request-response protocol. Other situations have their own requirements.
If the writing channel is full and I can't write, do I just keep looping until I can start writing stuff through?
No, you wait for OP_WRITE to fire.
If the connection suddenly gets choked, I'm unsure if I'm supposed to just write what I can, flip back to OP_READ, attempt to read, then flip back to OP_WRITE. From what I've read, this appears to not be the correct way to do things (and may cause large overhead constantly switching back and forth?).
The overhead isn't significant, but it's the wrong thing to do in the situation I described above.
What is the optimal way to handle reading and writing bulk data when the buffers both may become full?
In general, read when OP_READ fires; write whenever you need to; and use OP_WRITE to tell you when an outbound stall has relieved itself.
Do you even need OP_READ to read from the network?
Yes, otherwise you just smoke the CPU.
Whenever you need to write just set interested operation to (OP_READ || OP_WRITE). When you finish writing just set the interested operation to OP_READ.
that's all you have to do.
To be more specific, i have written a server with java NIO, and it works quiet well, after some testing i have found out that for some reason, in average a call to the SocketChannels write method takes 1ms, the read method on the other hand takes 0.22ms in average.
Now at first i was thinking that setting the sent/receive buffer values on Socket might help a bit, but after thinking about it, all the messages are very short(a few bytes) and i send a message about every 2 seconds on a single connection. Both sent and receive buffers are well over 1024 bytes in size so this can't really be the problem, i do have several thousand clients connected at once thou.
Now i am a bit out of ideas on this, is this normal and if it is, why ?
I would start by using Wireshark to eliminate variables.
#Nuoji i am using nonblocikng-io and yes i am using a Selector, as for when i write to a channel i do the following:
Since what i wrote in the second paragraph in my post is true, i assume that the channel is ready for writing in most cases, hence i do not at first set the interest set on the key to write, but rather try to write to the channel directly. In case however that, i can not write everything to the channel (or anything at all for that matter), i set the interest set on the key to write(that way the next time i try to write to the channel it is ready to write). Although in my testing where i got the results mentioned in the original post, this happens very rarely.
And yes i can give you samples of the code, although i didn't really want to bother anyone with it. What parts in particular would you like to see, the selector thread or the write thread ?