I am sending a file over a socket in Java.
This works pretty well if I close the socket connection after I sent the file, so that the read method returns -1.
But I don't want to close the socket, so I need a termination condition.
I tried to use inputStream.available, but its not returning the exact number of bytes.
int number;
while((number = inputStream.read(buffer)) != -1) {
fileStream.write(buffer, 0, number);
}
How can I do this?
A standard pattern is to send the length first as say an int or long. e.g. DataInput/OutputStream can help you do this.
When you have read this amount of data, you have finished, but the connection is still open and can continue to be used.
Related
When transmitting a file through socket in blocking mode
bytesTransferred = fileIChannel.transferTo(0, fileIChannel.size(), socketChannel);
// or using a buffer
ByteBuffer byteBuffer = ByteBuffer.allocateDirect(1024*8);
while (fileIChannel.read(byteBuffer) != -1) {
byteBuffer.flip();
bytesTransferred += socketChannel.write(byteBuffer);
byteBuffer.clear();
In the event of a connection failure, I need to keep the number of bytes transferred.
I can do this by waiting for a response every from the server when it receives a particular number of bytes. Or, when the connection is restored, I send a request for the number of bytes received.
Which of the options will be more correct? How is this problem usually resolved?
And the second question. Is data integrity guaranteed when sending large data through a socket?
WIth this code you cannot tell. If you want to know that the peer application has received and processed sent data, the peer application has to tell you. TCP does buffering at both ends, so the API alone cannot tell you.
NB Your copy loop is wrong. It should be:
while ((fileIChannel.read(byteBuffer) != -1 && byteBuffer.position() > 0)
{
byteBuffer.flip();
bytesTransferred += socketChannel.write(byteBuffer);
byteBuffer.compact();
}
and there should also be an error test on the write. At present you are assuming that everything got written to the SocketChannel on every write, which isn't guaranteed in non-blocking mode.
The code using transferTo() is also wrong, as transferTo() isn't guaranteed to perform the entire transfer: that's why it returns a count. You have to loop.
In my app I'm using socket connection to communicate with a server. Everything works fine, I'm able to send/receive from/to the server with no issues. None, but one. There's a certain option/response from the server than leaves the connection open for around 30 seconds. Server sends the reply right away, but it keeps the connection open and as such the app hangs there showing the 'waiting' message, because I can't seem to figure out how to close the connection/inputStream without it waiting for the server to close it.
Is there a way to read each character received by the buffer and as soon as a character is found ('*' for example) the buffer should close and also the connection to the server.
Any help?
you need to manage it between client and server. One approach is HTTP chunked. HTTP chunked send first length of message, then message data. Or, if as it is your case, if you know a "magic" byte notifies client that connection can close, you can read data one by one, and when you reach the finalizer byte, you can complete your reading and close connection. Otherwise, application hangs in input.read() until connection reaches timeout or new byte arrives
InputStream input = ....;
ByteArrayOutputStream bo = new ByteArrayOutputStream();
while (true){
int singlebyte = input.read();
if (singlebyte == -1) break; //stream ends
bo.write(singlebyte);
if (singlebyte == '*'){
//the byte you are wating. at this point, you can break loop. or continue to read
bo.write(singlebyte);
byte data[] = bo.toByteArray();
}
}
You do 'read each character as it is found'. Your problem here is not reading characters, it is the server not closing the connection, so you're blocked waiting for end of stream. Such a protocol is either broken, in which case it needs redesigning, or else it includes a length indicator or some other means of knowing when the message is complete,mwhich you're not taking proper notice of.
I'm using a java server to connect to a browser with secure websockets. All works fine with the connect, but many times i get an unexpected -1 result from socket.in.read(buffer,off,len), this happens also in the middle of a frame. Normally i close a socket directly upon reception of -1, since it is end of stream. However i noted that it can also happen on a connection reset. I have come over many cases where in my tests the socket whould return valuable data after read returned -1. I even have the feeling this is more often than not. My problem arrises when sometimes i just get some scrambled data out of the socket after such a case. Another problem is that the other side is not notified when a frame cannot be delivered... So what good is TCP/SSL than? if you need to consider it an unreliable connection for transporting websocket frames in java?
I have some schemes to use that are used to deal with unreliable connections for making shure a packet arrives. But i hope that somebody knows what to do after read returns -1.
Sorry for the somewhat vague description in this one... i'm getting tired with solving this issue.
Just an example of some rubbish comming in (only text frames are submitted containing JSON data):
16-06-13 22:43:13.918;WebSocket;7: Read frame from websocket: 377, opcode:UNKNOWN
data: null
16-06-13 22:43:13.918;WebSocket;7: Read frame from websocket: 377, opcode:PONG_FRAME
data: null
16-06-13 22:43:13.918;WebSocket;7: Read frame from websocket: 377, opcode:TEXT_FRAME
data: =,6GiGGV7C6_TfPHg\~\c
Here another example of a received frame that is just a bit malformed!? how is this possible with a TCP/TLS connection???:
17-06-13 09:42:37.510;WebSocket;7: Read frame from websocket: 15, opcode:TEXT_FRAME
data: "kep-aiveY:"d613Nb2-N24eV463K-808-fJb30I9e3M02
It is supposed to read {"keep-alive":"[UUID]"}
Meanwhilst i have done some more testing and found that 9 out of 10 times it works if you continue reading after reception of -1. So even if you are reading halfway the frame and receive a -1 then you should test somehow if the socket is closed or not, i now use: socket.isInputShutdown(). if this is not the case then just continue filling up the buffer. To do so i now use the following code where socket is the SSLSocket:
public static int readFully(Socket socket, InputStream is, byte[] buffer, int off, int len) throws IOException
{
int read = 0;
while(read < len)
{
int b = is.read();
if(b < 0)
{
Logger.log(TAG, "readFully read returned: " + b + " testing if connection is reset or closed.", Logger.WARNING);
if(socket.isInputShutdown())
{
throw new IOException("InputStream closed before data could be fully read! (readFully read returned -1 and socket.isInputShutdown() is true");
}
}
else
{
buffer[off + (read++)] = (byte) b;
}
}
return read;
}
It is still not a hundred % correct but at leas i get more reliable results then before.
i get an unexpected -1 result from socket.in.read(buffer,off,len)
You have already reached EOS (end of stream) before you called this method.
this happens also in the middle of a frame.
There is no such thing as a 'frame' in TCP. If you mean it happens in the middle of an application message, you have an application protocol error.
Normally i close a socket directly upon reception of -1, since it is end of stream.
Correct.
However i noted that it can also happen on a connection reset
No it doesn't. If it did, you could not possibly have detected the reset. The statement is self-contradictory.
I have come over many cases where in my tests the socket whould return valuable data after read returned -1.
No you haven't. A socket can't return anything but -1 after it first does so. You can't be getting any data at all, let alone 'valuable' data, unless you are ignoring the -1 somewhere.
My problem arrises when sometimes i just get some scrambled data out of the socket after such a case.
Only if you ignore the -1, as you are doing.
Another problem is that the other side is not notified when a frame cannot be delivered.
Of course it isn't. If you could deliver a notification to the other side, you could deliver the packet. This doesn't make sense either. If you mean that the other side doesn't get notified when it couldn't deliver the packet, you are up against the fact that TCP sends are asyncrhonous, so you won't normally get a send error on the send that caused it. You will get it on a later send. If you need per-send acknowledgements, you need to build them into your application protocol.
So what good is TCP/SSL then?
TCP is a reliable data-stream protocol, and SSL is a secure reliable data-stream protocol. That's what use they are.
if you need to consider it an unreliable connection for transporting websocket frames in java?
Neither of them is unreliable.
I hope that somebody knows what to do after read returns -1.
Close the socket.
Meanwhilst i have done some more testing and found that 9 out of 10 times it works if you continue reading after reception of -1.
No it doesn't. 1000 times of 1000 it continues to return -1. All you are seeing here is the effect of other bugs in your code.
So even if you are reading halfway the frame and receive a -1 then you should test somehow if the socket is closed or not
You can't. The socket isn't closed. Proof: you just read from it without getting an exception. You can't test whether the connection is closed either, other than by read() returning -1.
I now use: socket.isInputShutdown().
Pointless. That tells you whether you have called Socket.shutdownInput() on your own socket. It doesn't tell you diddly-squat about the state of the connection. There is no TCP API that can do that, other than reading or writing.
if this is not the case then just continue filling up the buffer.
I.e. reading gargabe by ignoring the -1 that read() is returning.
To do so i now use the following code where socket is the SSLSocket:
Why? DataInputStream.readFully() already exists. Re-implementing it won't help.
if(b < 0)
{
Logger.log(TAG, "readFully read returned: " + b + " testing if connection is reset or closed.", Logger.WARNING);
if(socket.isInputShutdown())
At this point it is 100% irrelevant whether your Socket is shutdown for input. read() has returned -1, which means the peer has closed the connection. Period.
{
throw new IOException("InputStream closed before data could be fully read! (readFully read returned -1 and socket.isInputShutdown() is true");
}
This is all nonsense.
}
else
{
buffer[off + (read++)] = (byte) b;
}
Here you are adding the low byte of -1, which is 0xff, to the buffer. This is also nonsense.
Is there a way to detect the EOF when reading from a TCP Socket whilst the socket connection stays open?
Most of the examples I have seen are something along the lines of:
int n=0;
While((n = read.inStream(data)) != -1){
destType.write(data, 0, n);
}
However this means that you are forced to create a new connection every time you want to receive a new piece of data.
In my case this is a constant stream of images that are sent across the socket as bytes and I would like to process each image without having to close the connection between so that I can have all the images associated with a single user, aswell as it is just more efficient to not constantly open and close connections at a high frequency.
So is there a way to do this or some information on a possible alternative?
No - if the connection stays open, the stream hasn't reached its end. The idea of a stream reporting EOF and then having more data later goes against the principle of a stream.
If you want to send multiple messages across a TCP stream, the simplest way is to prefix each message with its length:
HEADER BODY
HEADER BODY
HEADER BODY
Then the client will read the header, find out how long the message is, read it, then read the next header, etc.
When working with Sockets in Java, how can you tell whether the client has finished sending all (binary) data, before you could start processing them. Consider for example:
istream = new BufferedInputStream (socket.getInputStream());
ostream = new BufferedOutputStream(socket.getOutputStream());
byte[] buffer = new byte[BUFFER_SIZE];
int count;
while(istream.available() > 0 && (count = istream.read(buffer)) != -1)
{
// do something..
}
// assuming all input has been read
ostream.write(getResponse());
ostream.flush();
I've read similar posts on SO such as this, but couldn't find a conclusive answer. While my solution above works, my understanding is that you can never really tell if the client has finished sending all data. If for instance the client socket sends a few chunks of data and then blocks waiting for data from another data source before it could send more data, the code above may very well assume that the client has finished sending all data since istream.available() will return 0 for the current stream of bytes.
Yes, you're right - using available() like this is unreliable. Personally I very rarely use available(). If you want to read until you reach the end of the stream (as per the question title), keep calling read() until it returns -1. That's the easy bit. The hard bit is if you don't want the end of the stream, but the end of "what the server wants to send you at the moment."
As the others have said, if you need to have a conversation over a socket, you must make the protocol explain where the data finishes. Personally I prefer the "length prefix" solution to the "end of message token" solution where it's possible - it generally makes the reading code a lot simpler. However, it can make the writing code harder, as you need to work out the length before you send anything. This is a pain if you could be sending a lot of data.
Of course, you can mix and match solutions - in particular, if your protocol deals with both text and binary data, I would strongly recommend length-prefixing strings rather than null-terminating them (or anything similar). Decoding string data tends to be a lot easier if you can pass the decoder a complete array of bytes and just get a string back - you don't need to worry about reading to half way through a character, for example. You could use this as part of your protocol but still have overall "records" (or whatever you're transmitting) with an "end of data" record to let the reader process the data and respond.
Of course, all of this protocol design stuff is moot if you're not in control of the protocol :(
I think this is the task more of a protocol, assuming that you are the man who writes both the transmitting and receiving sides of application.
For example you could implement some simple logic protocol and divide you data into packets. Then divide packets into two parts: the head and the body. And then to say that your head consists of a predefined starting sequence and contains number of bytes in the body. Of forget about starting sequence and simpy transfer number of bytes in the bofy as a first byte of the packet.
Then you've could solve you problem.
As some ppl already said you can't avoid some kind of protocol for communication.
It should look like this for example:
On the server side you have:
void sendMSG(PrintWriter out){
try {
//just for example..
Process p = Runtime.getRuntime().exec("cmd /c dir C:");
BufferedReader br = new BufferedReader(new InputStreamReader(
p.getInputStream()));
//and then send all this crap to the client
String s = "";
while ((s = br.readLine()) != null) {
out.println("MSG");
out.println(s);
}
} catch (Exception e) {
System.out.println("Command incorrect!");
}
out.println("END");
}
//You are not supposed to close the stream or the socket, because you might want to send smth else later..
On the client side you have:
void recieveMSG(BufferedReader in) {
try {
while (in.readLine().equals("MSG")) {
System.out.println(in.readLine());
}
} catch (IOException e) {
System.out.println("Connection closed!");
}
}
as Nikita said this is more of task of protocol. Either you can go by header and body approach or you can send a special character or symbol for end of stream to break processing loop. Something like if you send say '[[END]]' on socket to denote end of stream.