When transmitting a file through socket in blocking mode
bytesTransferred = fileIChannel.transferTo(0, fileIChannel.size(), socketChannel);
// or using a buffer
ByteBuffer byteBuffer = ByteBuffer.allocateDirect(1024*8);
while (fileIChannel.read(byteBuffer) != -1) {
byteBuffer.flip();
bytesTransferred += socketChannel.write(byteBuffer);
byteBuffer.clear();
In the event of a connection failure, I need to keep the number of bytes transferred.
I can do this by waiting for a response every from the server when it receives a particular number of bytes. Or, when the connection is restored, I send a request for the number of bytes received.
Which of the options will be more correct? How is this problem usually resolved?
And the second question. Is data integrity guaranteed when sending large data through a socket?
WIth this code you cannot tell. If you want to know that the peer application has received and processed sent data, the peer application has to tell you. TCP does buffering at both ends, so the API alone cannot tell you.
NB Your copy loop is wrong. It should be:
while ((fileIChannel.read(byteBuffer) != -1 && byteBuffer.position() > 0)
{
byteBuffer.flip();
bytesTransferred += socketChannel.write(byteBuffer);
byteBuffer.compact();
}
and there should also be an error test on the write. At present you are assuming that everything got written to the SocketChannel on every write, which isn't guaranteed in non-blocking mode.
The code using transferTo() is also wrong, as transferTo() isn't guaranteed to perform the entire transfer: that's why it returns a count. You have to loop.
Related
In my app I'm using socket connection to communicate with a server. Everything works fine, I'm able to send/receive from/to the server with no issues. None, but one. There's a certain option/response from the server than leaves the connection open for around 30 seconds. Server sends the reply right away, but it keeps the connection open and as such the app hangs there showing the 'waiting' message, because I can't seem to figure out how to close the connection/inputStream without it waiting for the server to close it.
Is there a way to read each character received by the buffer and as soon as a character is found ('*' for example) the buffer should close and also the connection to the server.
Any help?
you need to manage it between client and server. One approach is HTTP chunked. HTTP chunked send first length of message, then message data. Or, if as it is your case, if you know a "magic" byte notifies client that connection can close, you can read data one by one, and when you reach the finalizer byte, you can complete your reading and close connection. Otherwise, application hangs in input.read() until connection reaches timeout or new byte arrives
InputStream input = ....;
ByteArrayOutputStream bo = new ByteArrayOutputStream();
while (true){
int singlebyte = input.read();
if (singlebyte == -1) break; //stream ends
bo.write(singlebyte);
if (singlebyte == '*'){
//the byte you are wating. at this point, you can break loop. or continue to read
bo.write(singlebyte);
byte data[] = bo.toByteArray();
}
}
You do 'read each character as it is found'. Your problem here is not reading characters, it is the server not closing the connection, so you're blocked waiting for end of stream. Such a protocol is either broken, in which case it needs redesigning, or else it includes a length indicator or some other means of knowing when the message is complete,mwhich you're not taking proper notice of.
I am sending a file over a socket in Java.
This works pretty well if I close the socket connection after I sent the file, so that the read method returns -1.
But I don't want to close the socket, so I need a termination condition.
I tried to use inputStream.available, but its not returning the exact number of bytes.
int number;
while((number = inputStream.read(buffer)) != -1) {
fileStream.write(buffer, 0, number);
}
How can I do this?
A standard pattern is to send the length first as say an int or long. e.g. DataInput/OutputStream can help you do this.
When you have read this amount of data, you have finished, but the connection is still open and can continue to be used.
I have a client/server application written in Java using non-blocking IO.
There are several message types which are transferred as Json encoding and a message delimiter appended at the end of each message.
The client reads bytes and merges the messages which are coming in chunks. On regular cases it is working but in heavy load cases i get a chunk which includes messages which are not in right order. I mean, lets say I have a message m1="AAABBBCCCDDD" and m2="EEEFFF" and delimiter is "||". When the message is received it is supposed to be "AAABBBCCCDDD||EEEFFF||". But it is received "AAABBBEEEFFF||CCCDDD||". As a result it fails to parse the message.
Actually, I would like to hear the ideas that should be considered while developing network applications using non-blocking IO. what can be the reason of being in the wrong order..?
Reader code is like this:
ByteBuffer buffer = ByteBuffer.allocate(20000);
count = 0;
while ((count = channel.read(buffer)) > 0) {
buffer.flip();
processSocketData(Charset.defaultCharset().decode(buffer));
}
processSocketData() method is like that:
socketData.append(newData);
delIndex = socketData.indexOf(cGlobals.delimiterSequence);
if (delIndex > -1) {
processRawMessage(socketData.substring(0, delIndex));
socketData.delete(0, delIndex + cGlobals.delimiterSize);
}
You need to flip() before processing, as you are doing, and you also need to either compact() or clear() the buffer after you process it.
I'm using a java server to connect to a browser with secure websockets. All works fine with the connect, but many times i get an unexpected -1 result from socket.in.read(buffer,off,len), this happens also in the middle of a frame. Normally i close a socket directly upon reception of -1, since it is end of stream. However i noted that it can also happen on a connection reset. I have come over many cases where in my tests the socket whould return valuable data after read returned -1. I even have the feeling this is more often than not. My problem arrises when sometimes i just get some scrambled data out of the socket after such a case. Another problem is that the other side is not notified when a frame cannot be delivered... So what good is TCP/SSL than? if you need to consider it an unreliable connection for transporting websocket frames in java?
I have some schemes to use that are used to deal with unreliable connections for making shure a packet arrives. But i hope that somebody knows what to do after read returns -1.
Sorry for the somewhat vague description in this one... i'm getting tired with solving this issue.
Just an example of some rubbish comming in (only text frames are submitted containing JSON data):
16-06-13 22:43:13.918;WebSocket;7: Read frame from websocket: 377, opcode:UNKNOWN
data: null
16-06-13 22:43:13.918;WebSocket;7: Read frame from websocket: 377, opcode:PONG_FRAME
data: null
16-06-13 22:43:13.918;WebSocket;7: Read frame from websocket: 377, opcode:TEXT_FRAME
data: =,6GiGGV7C6_TfPHg\~\c
Here another example of a received frame that is just a bit malformed!? how is this possible with a TCP/TLS connection???:
17-06-13 09:42:37.510;WebSocket;7: Read frame from websocket: 15, opcode:TEXT_FRAME
data: "kep-aiveY:"d613Nb2-N24eV463K-808-fJb30I9e3M02
It is supposed to read {"keep-alive":"[UUID]"}
Meanwhilst i have done some more testing and found that 9 out of 10 times it works if you continue reading after reception of -1. So even if you are reading halfway the frame and receive a -1 then you should test somehow if the socket is closed or not, i now use: socket.isInputShutdown(). if this is not the case then just continue filling up the buffer. To do so i now use the following code where socket is the SSLSocket:
public static int readFully(Socket socket, InputStream is, byte[] buffer, int off, int len) throws IOException
{
int read = 0;
while(read < len)
{
int b = is.read();
if(b < 0)
{
Logger.log(TAG, "readFully read returned: " + b + " testing if connection is reset or closed.", Logger.WARNING);
if(socket.isInputShutdown())
{
throw new IOException("InputStream closed before data could be fully read! (readFully read returned -1 and socket.isInputShutdown() is true");
}
}
else
{
buffer[off + (read++)] = (byte) b;
}
}
return read;
}
It is still not a hundred % correct but at leas i get more reliable results then before.
i get an unexpected -1 result from socket.in.read(buffer,off,len)
You have already reached EOS (end of stream) before you called this method.
this happens also in the middle of a frame.
There is no such thing as a 'frame' in TCP. If you mean it happens in the middle of an application message, you have an application protocol error.
Normally i close a socket directly upon reception of -1, since it is end of stream.
Correct.
However i noted that it can also happen on a connection reset
No it doesn't. If it did, you could not possibly have detected the reset. The statement is self-contradictory.
I have come over many cases where in my tests the socket whould return valuable data after read returned -1.
No you haven't. A socket can't return anything but -1 after it first does so. You can't be getting any data at all, let alone 'valuable' data, unless you are ignoring the -1 somewhere.
My problem arrises when sometimes i just get some scrambled data out of the socket after such a case.
Only if you ignore the -1, as you are doing.
Another problem is that the other side is not notified when a frame cannot be delivered.
Of course it isn't. If you could deliver a notification to the other side, you could deliver the packet. This doesn't make sense either. If you mean that the other side doesn't get notified when it couldn't deliver the packet, you are up against the fact that TCP sends are asyncrhonous, so you won't normally get a send error on the send that caused it. You will get it on a later send. If you need per-send acknowledgements, you need to build them into your application protocol.
So what good is TCP/SSL then?
TCP is a reliable data-stream protocol, and SSL is a secure reliable data-stream protocol. That's what use they are.
if you need to consider it an unreliable connection for transporting websocket frames in java?
Neither of them is unreliable.
I hope that somebody knows what to do after read returns -1.
Close the socket.
Meanwhilst i have done some more testing and found that 9 out of 10 times it works if you continue reading after reception of -1.
No it doesn't. 1000 times of 1000 it continues to return -1. All you are seeing here is the effect of other bugs in your code.
So even if you are reading halfway the frame and receive a -1 then you should test somehow if the socket is closed or not
You can't. The socket isn't closed. Proof: you just read from it without getting an exception. You can't test whether the connection is closed either, other than by read() returning -1.
I now use: socket.isInputShutdown().
Pointless. That tells you whether you have called Socket.shutdownInput() on your own socket. It doesn't tell you diddly-squat about the state of the connection. There is no TCP API that can do that, other than reading or writing.
if this is not the case then just continue filling up the buffer.
I.e. reading gargabe by ignoring the -1 that read() is returning.
To do so i now use the following code where socket is the SSLSocket:
Why? DataInputStream.readFully() already exists. Re-implementing it won't help.
if(b < 0)
{
Logger.log(TAG, "readFully read returned: " + b + " testing if connection is reset or closed.", Logger.WARNING);
if(socket.isInputShutdown())
At this point it is 100% irrelevant whether your Socket is shutdown for input. read() has returned -1, which means the peer has closed the connection. Period.
{
throw new IOException("InputStream closed before data could be fully read! (readFully read returned -1 and socket.isInputShutdown() is true");
}
This is all nonsense.
}
else
{
buffer[off + (read++)] = (byte) b;
}
Here you are adding the low byte of -1, which is 0xff, to the buffer. This is also nonsense.
it's not my first time trying to understand this issue but i hope it will be the last one:
some background:
i have a Java SocketChannel NIO server working in non-blocking mode.
this server has multiple clients which send and receive messages from it.
each client maintain its connection to the server with "keepalive" messages every once in a while.
The main idea with the server is that the clients will remain connect "all the time" and receive messages from it in "push" mode.
now to my question:
in Java NIO read() function - when the read() return -1 - it means that its EOS.
in the question i've asked here i realized that it means that the socket has finished its current stream and doesn't need to be closed..
when searching in google a bit more about this i found out that it does mean that the connection is closed on the other side..
what does the word "stream" exactly means? is it the current message being sent from the client? is it the ability of the client side connection to send anymore messages ?
why would a SocketChannel be closed on the client side if the client never told him to be closed ?
what is the difference between read() return -1 and connection reset by peer I/O error ?
this is how i read from SocketChannel:
private JSONObject readIncomingData(SocketChannel socketChannel)
throws JSONException, InvalidKeyException, IllegalBlockSizeException, BadPaddingException, IOException {
JSONObject returnObject = null;
ByteBuffer buffer = ByteBuffer.allocate(1024);
Charset charset = Charset.forName("UTF-8");
String endOfMesesage = "\"}";
String message = "";
StringBuilder input = new StringBuilder();
boolean continueReading = true;
while (continueReading && socketChannel.isOpen())
{
buffer.clear();
int bytesRead = socketChannel.read(buffer);
if (bytesRead == -1)
{
continueReading = false;
continue;
}
buffer.flip();
input.append(charset.decode(buffer));
message = input.toString();
if (message.contains(endOfMesesage))
continueReading = false;
}
if (input.length() > 0 && message.contains(endOfMesesage))
{
JSONObject messageJson = new JSONObject(input.toString());
returnObject = new JSONObject(encrypter.decrypt(messageJson.getString("m")));
}
return returnObject;
}
What does the word "stream" exactly means? is it the current message being sent from the client? is it the ability of the client side connection to send anymore messages ?
The stream means the data that is flowing between two locations, usually between the client and the server but effectively it's any kind of data flowing. E.g. if you read a file from your hard disc you use a FileInputStream which represents data flowing from the file on disc to your program. It's a very generic concept. Think of it as a river where the water is the data. Plus it's a very cool kind of river which allows you to control how the water/data is flowing.
Why would a SocketChannel be closed on the client side if the client never told him to be closed ?
That can happen if the connection between client and server is reset or interrupted. Your program should never assume that connections just live and are never interrupted. Connections are interrupted for all kinds of reasons, may it be a flaky network component, someone pulling a plug that should better be left where it was or the wireless network is going down. Also the server might close the connection, e.g. if the server program goes down, has a bug or the connection runs into a timeout. Always remember that open connections are a limited resource so servers might decide to close them if they are idle for too long.
What is the difference between read() return -1 and connection reset by peer I/O error ?
When the read() returns -1 this simply means that there is currently no more data in the stream. A connection reset means, there was probably more data, but the connection no longer exists and therefore this data cannot be read anymore. Again taking the river anology: Think of the data as some quantity of water being sent from a village upstream (aka Serverville) to a village downstream (aka Clientville) using a riverbed that connects the two villages (the connection). Now someone at Serverville pulls the big lever and the water (the data) flows down from Serverville to Clientville. After Serverville has sent all the water it wanted to send, it closes the lever and the riverbed will be empty again (and actually destroyed as the connection got closed). This is where Clientville get's the -1. Now imagine some bulldozer interrupting the riverbed and some of the water never makes it to Clientville. This is the "connection reset" situation.
Hope this helps :)
what does the word "stream" exactly means? is it the current message being sent from the client?
It is a stream of bytes, not messages. You can use those bytes to form a message but the stream has no idea you are doing this, nor does it support messages in any way.
why would a SocketChannel be closed on the client side if the client never told him to be closed ?
It can only be closed with a -1 if the other end closed it.
what is the difference between read() return -1 and connection reset by peer I/O error ?
You can close or drop a connection other ways such as closing it from the same side, or a timeout in the connection e.g.you pulled out the network cable.
BTW: The way you have written the code is better suited to blocking NIO. For example, if you receive more than one whole message, anything after the first one is discarded. If you use blocking IO and keep everything you read you will not get corrupted or dropped messages.
What does the word "stream" exactly means? is it the current message being sent from the client?
It basically means one side of the connection, which is full-duplex. TCP is a byte-stream protocol, providing two independent byte streams, one in each direction.
Why would a SocketChannel be closed on the client side if the client never told him to be closed?
It wouldn't. The client did close the connection. That's what read() returning -1 means.
What is the difference between read() return -1 and connection reset by peer I/O error ?
read() returning -1 means the peer closed the connection properly. 'Connection reset by peer' indicates a protocol error of some kind, usually that you have written data to a connection that had already been closed by the peer.
Re your code, if read() returns -1 you must close the channel. There is no other sensible way to proceed.