I have a server which reads from the SocketChannel like below:
private void readDataFromSocket(SocketChannel socketChannel) throws IOException {
BUFFER.clear();
int count = 0;
while ((count = socketChannel.read(BUFFER)) > 0) {
BUFFER.flip();
int limit = BUFFER.limit();
while (limit > 0) {
System.out.print((char) BUFFER.get());
limit--;
}
}
if (count < 0) {
System.out.println("closing the socket!!!");
socketChannel.close();
}
}
And below is the client where client writes to the SocketChannel:
private void write(String str, SocketChannel channel) throws IOException{
byte[] b = str.getBytes();
buffer.clear();
buffer.put(b);
buffer.flip();
while(buffer.hasRemaining()){
channel.write(buffer);
}
}
So my question:
when exactly in the server code the count value will be 0 ( while ((count = socketChannel.read(BUFFER)) > 0))?
is it possible that the count will be 0 if the server has read half of the message that client has sent i.e.
Suppose client wrote: stack overflow, is it possible that in the server count will be 0 after reading stack i.e. half of the message that the client has sent (think that the message can be of 1MB size)?
When using blocking mode, you will always get at least 1 byte. Note: you might only get 1 byte, it doesn't read "messages".
When using non-blocking mode, you will get 0 most of the time, in fact whenever there is not a packet waiting for you.
In TCP, data is sent in packets, not messages. This means if you send 1 MB, most likely it will be broken into packets of your MTU size e.g. ~1500 bytes. If you read this socket, you will most likely see blocks of this size or a multiple if multiple packets came in since the last read. You will never see part of a packet, unless you read less than the available data. e.g. if 1500 bytes is waiting and you read just 8 bytes, you get part of that packet.
Related
I am using C# to create a server software for Windows and Java to create the client software.
It works fine most of the time, except for those few exceptions that I don't understand.
I am generally using .ReadLine() and .WriteLine() on both ends to communicate, unless I try to send binary data. That's when I write and read the bytes directly.
This is how the software is supposed work:
Client requests the binary data
Server responds with the length of the binary data as a string
Client receives the length and converts it into an integer and starts reading (length) bytes
Server starts writing (length) bytes
It works in most cases, but sometimes the client app doesn't receive the full data and blocks. The server always immediately flushes after writing data, so flushing is not the problem.
Furthermore I've noticed this usually happens with larger files, small files (up to ~1 MB) usually are not a problem.
NOTE It seems like the C# server does send the data completely, so the problem is most likely somewhere in the Java code.
EDIT - Here are some logs from the client side
Working download: pastebin.com/hFd5TvrF
Failing download: pastebin.com/Q3zFWRLB
It seems like the client is waiting for 2048 bytes at the end (as it should be, as length - processed = 2048 in this case), but for some reason the client blocks.
Any ideas what I'm doing wrong? Below are the source codes of both server and client:
C# Server:
public void Write(BinaryWriter str, byte[] data)
{
int BUFFER = 2048;
int PROCESSED = 0;
// WriteString sends the String using a StreamWriter (+ flushing)
WriteString(data.Length.ToString());
while (PROCESSED < data.Length)
{
if (PROCESSED + BUFFER > data.Length)
BUFFER = data.Length - PROCESSED;
str.Write(data, PROCESSED, BUFFER);
str.Flush();
PROCESSED += BUFFER;
}
}
Java Client:
public byte[] ReadBytes(int length){
byte[] buffer = new byte[length];
int PROCESSED = 0;
int READBUF = 2048;
TOTAL = length;
progress.setMax(TOTAL);
InputStream m;
try {
m = clientSocket.getInputStream();
while(PROCESSED < length){
if(PROCESSED + READBUF > length)
READBUF = length - PROCESSED;
try {
PROCESSED += m.read(buffer, PROCESSED, READBUF);
} catch (IOException e) {
}
XPROCESSED = PROCESSED;
}
} catch (IOException e1) {
// Removed because of sensitive data
}
return decryptData(buffer);
}
I've found a fix. As of now, the server sends the length and right after sends the byte array. For some reason this does not work.
So what I've changed is:
Send length and wait for the client to respond with "OK"
Start writing bytes
Not sure why, but it works. Ran it in a while(true) loop and it's been sending data 1000 times in 4 minutes straight and no problems, so I guess it's fixed.
I'm developing Server-Client application and I have a problem with waiting for input data on input stream.
I have thread dedicated to reading input data. Currently it uses while loop to hold until data is available. (N.B. protocol is as follow: send size of packet, say N, as int then send N bytes).
public void run(){
//some initialization
InputStream inStream = sock.getInputStream();
byte[] packetData;
//some more stuff
while(!interrupted){
while(inStream.available()==0);
packetData = new byte[inStream.read()];
while(inStream.available()<packetData.length);
inStream.read(packetData,0,packetData.length);
//send packet for procession in other thread
}
}
It works but blocking the thread by while loop is IMO a bad idea. I could use Thread.sleep(X) to prevent resources being continously consumed by the loop, but there surely must be a better way.
Also I can not rely on InputStream.read to block the thread as part of the data may be sent by the server with delays. I have tried but it always resulted in unexpected behaviour.
I'd appreciate any ideas :)
You can use DataInputStream.readFully()
DataInputStream in = new DataInputStream(sock.getInputStream());
//some more stuff
while(!interrupted) {
// readInt allows lengths of up to 2 GB instead of limited to 127 bytes.
byte[] packetData = new byte[in.readInt()];
in.readFully(packetData);
//send packet for procession in other thread
}
I prefer to use blocking NIO which supports re-usable buffers.
SocketChannel sc =
ByteBuffer bb = ByteBuffer.allocateDirect(1024 *1024); // off heap memory.
while(!Thread.currentThread.isInterrupted()) {
readLength(bb, 4);
int length = bb.getInt(0);
if (length > bb.capacity())
bb = ByteBuffer.allocateDirect(length);
readLength(bb, length);
bb.flip();
// process buffer.
}
static void readLength(ByteBuffer bb, int length) throws EOFException {
bb.clear();
bb.limit(length);
while(bb.remaining() > 0 && sc.read(bb) > 0);
if (bb.remaining() > 0) throw new EOFException();
}
As UmNyobe said, available() is meant to be used if you dont want to block as the default behaviour is blocking.
Just use the normal read to read whatever is available but only send packet for processing in other thread once you have packetData.length bytes in your buffer...
It would seem that the Client - Server application i wrote does work however it seems that not all data is processed every time.
I am testing it on a local machine in Eclipse env.
Server:
private void sendData() throws Exception
{
DatagramPacket data = new DatagramPacket(outgoingData, outgoingData.length, clientAddress, clientPort);
InputStream fis = new FileInputStream(responseData);
int a;
while((a = fis.read(outgoingData,0,512)) != -1)
{
serverSocket.send(data);
}
}
Client:
private void receiveData() throws Exception
{
DatagramPacket receiveData = new DatagramPacket(incomingData, incomingData.length);
OutputStream fos = new FileOutputStream(new File("1"+data));
while(true)
{
clientSocket.receive(receiveData);
fos.write(incomingData);
}
}
I used to have if else in the while(true) loop to check if packet length is less than 512 bytes so it knew when to break;
I was thinking there was a problem whit that but seems that was oke for now i just wait few minutes and then stop the Client.java app
The file does transfer but the original file is 852kb and so far i got 777, 800, 850,.. but never all of it.
The fundamental problem with your approach is that UDP does not guarantee delivery. If you have to use UDP (rather than, say, TCP), you have to implement a scheme that would detect and deal with packets that got lost, arrive out of order, or are delivered multiple times.
See When is it appropriate to use UDP instead of TCP?
You should probably use TCP to transfer files. You are probably losing packets because you are sending them so fast in that while loop.
int a;
while((a = fis.read(outgoingData,0,512)) != -1)
{
serverSocket.send(data);
}
since you're sending so fast I highly doubt it will have a chance to be received in the right order. some packets will probably be lost because of it too.
Also since your sending a fixed size of 512 bytes the last packet you send will probably not be exactly that size, so you will see the end of the file "look wierd."
The problem I am having is that when I use an InputStream to read bytes, it blocks until the connection is finished. EG:
InputStream is = socket.getInputStream();
byte[] buffer = new byte[20000];
while (is.read(buffer) != -1) {
System.out.println("reading");
}
System.out.println("socket read");
"socket read" doesn't print out until the FYN packet is actually recieved, thus closing the connection. What is the proper way to receive all the bytes in without blocking and waiting for the connection to drop?
Take a look at java.nio which has non-blocking IO support.
Reading till you get -1 means that you want to read until EOS. If you don't want to read until EOS, don't loop till the -1: stop sooner. The question is 'when?'
If you want to read a complete 'message' and no more, you must send the message in such a way that the reader can find its end: for example, a type-length-value protocol, or more simply a size word before each message, or a self-describing protocol such as XML.
With traditional sockets the point is that usually you do want them to block: what you do when logically you don't want your program to block is you put your reading/writing code in another thread, so that the separate read/write thread blocks, but not your whole program.
Failing that, you can use the available() method to see if there is actually any input available before reading. But then you need to be careful not to sit in a loop burning CPU by constantly calling available().
Edit: if the problem is that you're happy to block until the bytes have arrived, but not until the connection has dropped (and that is what is happeningh), then you need to make the client at the other end call flush() on its output stream after it has sent the bytes.
Try this:
InputStream is = socket.getInputStream();
byte[] buffer = new byte[20000];
int bytesRead;
do {
System.out.println("reading");
bytesRead = is.read(buffer);
}
while (is.available() > 0 && bytesRead != -1);
System.out.println("socket read");
More info: https://docs.oracle.com/javase/1.5.0/docs/api/java/io/InputStream.html#available()
Example taken from exampledepot on java.nio
// Create a direct buffer to get bytes from socket.
// Direct buffers should be long-lived and be reused as much as possible.
ByteBuffer buf = ByteBuffer.allocateDirect(1024);
try {
// Clear the buffer and read bytes from socket
buf.clear();
int numBytesRead = socketChannel.read(buf);
if (numBytesRead == -1) {
// No more bytes can be read from the channel
socketChannel.close();
} else {
// To read the bytes, flip the buffer
buf.flip();
// Read the bytes from the buffer ...;
// see Getting Bytes from a ByteBuffer
}
} catch (IOException e) {
// Connection may have been closed
}
Be sure to understand buffer flipping because it causes a lot of headache. Basically, you have to reverse your buffer to read from it. If you are to reuse that buffer to have the socket to write in it, you have to flip it again. However clear() resets the buffer direction.
the code is probably not doing what you think it does.
read(buffer) returns the number of bytes it read, in other words: it is not guaranties to fill up your buffer anyway.
See DataInputStream.readFully() for code that fill up the entire array:
or you can use this functions (which are based on DataInputStream.readFully()) :
public final void readFully(InputStream in, byte b[]) throws IOException
{
readFully(in, b, 0, b.length);
}
public final void readFully(InputStream in, byte b[], int off, int len) throws IOException
{
if (len < 0) throw new IndexOutOfBoundsException();
int n = 0;
while (n < len)
{
int count = in.read(b, off + n, len - n);
if (count < 0) throw new EOFException();
n += count;
}
}
Your code would look like:
InputStream is = socket.getInputStream();
byte[] buffer = new byte[20000];
readFully(is, buffer);
System.out.println("socket read");
I created a SocketChannel to a remote server to send and receive messages on Tomcat. To receive messages from a remote computer, I used a thread dedicated to task (only this thread will read from the socket, nothing else).
When some bytes are received at the SocketChannel (I keep polling the SocketChannel on non-blocking mode for new data), I first read 4 bytes to get the length of the next message, then allocate and read x bytes from the SocketChannel, which is then decoded and reconstructed into a message.
Below is my code for the receiving thread:
#Override
public void run() {
while (true) { //Don't exit thread
//Attempt to read the size of the incoming message
ByteBuffer buf = ByteBuffer.allocate(4);
int bytesread = 0;
try {
while (buf.remaining() > 0) {
bytesread = schannel.read(buf);
if (bytesread == -1) { //Socket was terminated
}
if (quitthread) break;
}
} catch (IOException ex) {
}
if (buf.remaining() == 0) {
//Read the header
byte[] header = buf.array();
int msgsize = (0xFF & (int)header[0]) + ((0xFF & (int)header[1]) << 8)
+ ((0xFF & (int)header[2]) << 16) + ((0xFF & (int)header[3]) << 24);
//Read the message coming from the pipeline
buf = ByteBuffer.allocate(msgsize);
try {
while (buf.remaining() > 0) {
bytesread = schannel.read(buf);
if (bytesread == -1) { //Socket was terminated
}
if (quitthread) break;
}
} catch (IOException ex) {
}
parent.recvMessage(buf.array());
}
if (quitthread) {
break;
}
}
}
The first bytes I received from the SocketChannel is fine, and I successfully decoded the message. However, the next time I read from the SocketChannel, the socket skipped ahead about 100 bytes, which caused the wrong bytes to be read and interpreted as length, causing everything to become corrupted.
What is wrong with the code? No other thread is reading from the SocketChannel.
Your parenthesis are off, the code is:
(0xFF & ((int)header[1] << 8))
which is always 0 (same with << 16 and << 24), my guess is you meant:
((0xFF & ((int)header[1])) << 8)
This would lead to reading not enough message bytes, also leading to a mismatch in synchronisation (as opposed to reading too many.)
Edit: now you fixed the above, I cannot see anything wrong. Could you tell us the relation between the length of the first message and the exact number of bytes that are eaten?
Based on the code shown, my only guess is that you edited some of the behaviour out of the sample shown which might influence the schannel, is the schannel referenced elsewhere?
If the line:
ByteBuffer buf = ByteBuffer.allocate(4);
would be outside of the while that would result in behaviour you describe, but in your sample code it isn't.
I presume when you say you're polling the socket in non-blocking mode you mean you're using a the "standard" Selector.select() approach?
When select returns and indicates that there's data available for reading from the socket you should only read the bytes that are available before re-entering the call to select(). If read() returns -1 it indicates that no more bytes are available for immediate reading in the buffer - It does not mean that the socket has been closed. Hence I suspect your of attempting to completely fill the buffer before returning is incorrect. Even if it does work your I/O thread will be constantly spinning whilst data arrives. In particular, it looks like you're simply ignoring a return value of -1.
Consider re-architecting your code to use a finite state machine approach. For example, I've implemented this in the past using a 3-state model: IDLE, READ_MESSAGE_LENGTH and READ_MESSAGE.