I am trying to write an App for Android that exchanges data in between 2 phones. (Android 4.2.1 and 4.4.2)
As a basis I used the BluetoothChat example from the android tutorial section.
My problem is this: when writing larger amounts of data (>1kB) the receiving side often gets the wrong data. Sometimes it receives seemingly unrelated data, often it receives sections of many consecutive 0s, or sometimes it even receives parts that where received before in the same message.
The way the transmissions are handled is as follows:
First I send a 4 byte header, which provides the length of the following message and it's messagetype (just a single byte indicating how the message is to be understood by the program)
Then the actual message.
I used the ConnectedThread from the BluetoothChat but implemented as an AsyncTask, otherwise it is exactly the same apart from the receiving which is handled as follows: (and as mentioned above)
//code inside the doInBackground() section, and enclosed in a while + try block
byte[] header = new byte[4];
inputStream.read(header);
int messagelength = readHeader(header); // readHeader is the function that analyzes the 4 bytes header
byte[] message = new byte[messagelength];
inputStream.read(message); // read exactly that amount of bytes
readMessage(message); // this function takes the byte[] and transfers it into whatever is needed.
// in this example simply calling new String(message) and printing to the Log
The sending side is exactly the same
write(byte[] bytes) {
outputStream.write(bytes); // I excluded the try/catch here
}
Testing multiple Strings and converting by simply calling string.getBytes(), resulted in:
130 byte String being transmitted correctly,
1443 byte String being transmitted correctly,
51948 byte String failing after about ~900 bytes. the Log would write the String correctly up until a point where it would only print the weird "question mark in a diamond" - symbol.
The moment of failure is variable though.
Is there a limit to how many bytes the outputStream.write(bytes) can handle? Has someone got an idea why this simple sending of bytes through Streams can fail? Some internal buffer problems? Maybe a cyclic buffer (as sometimes old message parts got repeated)?
Thank you very much for the help, I've lost hours due to this issue (tried writing and reading smaller chunks, but that didn't help a lot)
Related
In my system, I am using tcp to transfer messages between an android application, and a desktop application (that is developed in Qt).
I am using msgpack library to serialize/deserialize structures in both applications. In order to make sure that the structure is always received as a whole, I always send the number of bytes (encoded as a 32-bit big endian unsigned integer not using msgpack so that it always takes the first 4 bytes) before the message itself (that is encoded in msgpack). If there is a better way please tell me.
In the android app, I read the first 4 bytes into a byte[] and decode it into a long msgSize, next I need a way for the thread to keep blocking until the next msgSize bytes are received. After reading other questions and answers, I think I can write something like this:
InputStream is= sock.getInputStream();
byte[] msgSizeBuff= new byte[4];
is.read(msgSizeBuff, 0, 4);
long msgSize= MyDecodeFunction(msgSizeBuff);
DataInputStream dis = new DataInputStream(is);
byte[] msg = new byte[msgSize];
dis.readFully(msg);
After that I can use the msg array with msgpack, and I am sure that the whole message is received.
So:
is my usage of DataInputStream class alright, and am I guranteed that I will wait until I receive the specified number of bytes? because in the reference they say:
If insufficient bytes are available, EOFException is thrown.
I want a way to tell this thread to cancel the operation, so what would happen if another thread calls socket.close() or socket.shutdownInput(), will I get an exception?
I am currently using java.net.Socket to send messages from the client and reading messages from the server. All my messages are fairly short so far, and I have never had any problems.
One of my friends noticed that I was not handling message fragmentation, where the data could come in pieces, and has advised that I should create a buffer to handle this. I insisted that TCP handles this for me, but I'm not 100% sure.
Who is right?
Also, I plan on creating a client in C as well in the future. Do Berkeley sockets handle message fragmentation?
Details: Currently, in Java, the server creates a socket and reads the first byte from the message with InputStream#read(). That first byte determines the length of the entire message, and creates a byte array of the appropriate length, and calls InputStream#read(byte[]) once and assumes that the entire message has been read.
If you are talking about WebSockets,you may be mixing different concepts.
One thing is TCP/IP message fragmentation.
Other thing is how buffering works. You read buffers of data, and you need a framing protocol that tells you when you have a complete "message" (or frame). Basically you:
Read buffer.
Has complete header? No-> Goto 1, Yes-> continue
Read until having all the bytes that the head indicates as message
length.
Has complete message? No-> Goto 3, Yes -> continue
Yield message.
Goto 1.
Other different thing is WebSocket message fragmentation. WebSocket has already a framing protocol and messages can be split in different data frames, and control frames can be interleaved with data frames: https://developer.mozilla.org/en-US/docs/WebSockets/Writing_WebSocket_servers#Message_Fragmentation
If you are writing a WebSocket client or server you have to be ready for this situation.
Expanding on what nos said, TCP will break up large messages into smaller chunks, if the message is large enough. Often, it isn't. Often, the data you write is already split into parts (by you), into meaningful chunks like discrete messages.
The stuff about the reads/writes taking different amounts of calls comes from how the data is written, how it travels over the wire, and how you read it.
If you write 2 bytes 100 times, and then 20 seconds later go to read, it will say there is 200 bytes to be read, which you can read all at once if you want. If you pass a massive 2mb buffer to be written (I dont even know if thats possible), it would take longer to write out, giving more of a chance to the reading program to get different read calls.
Details: Currently, in Java, the server creates a socket and reads the first byte from the message with InputStream#read(). That first byte determines the length of the entire message, and creates a byte array of the appropriate length, and calls InputStream#read(byte[]) once and assumes that the entire message has been read.
That won't work. Have a look at the contract for InputStream.read(byte[]). It isn't obliged to transfer more than one byte. The correct technique is to read the length byte and then use DataInputStream.readFully(), which has the obligation to fill the buffer.
i wanted to report this error directly, but did not find any possibility yet from main page netty.io
I have noticed an error while sending data to channel. It happens not always, in 10-20% of cases, but it happens.
Following,
as example, if I get first connection with message of 1024 byte data, everything is fine till now, than I create socket to forwarded address doing it with HexDumpProxyInboundHandler
and here is fine everything, except one thing, I have created an listener on forwarded address with traffic logging, where i get the messages which were sent by Netty. I would expect the data of 1024 bytes on it, but it happens not always, not also in 100% of cases.
sometimes...
exactly sometimes the nightmare begins here,
if i get on same channel next message after 1024 bytes message, the data gets written in following possible forms:
3.1 either the first message and second message are merged and the data that i get on my port listener is correct, 1024 + 72(as example) and in correct byte order too (but in merged form, what is already not correct for me)
3.2 or the first message and second message are merged too, but with one little difference, with different order, 72(as example) + 1024 bytes, even if data was correctly received by server socket, and in correct order.. the sending order was incorrect also.
3.3 or finally the first message of 1024 gets send as is following by second message which gets send as is too, so everything is fine here too, and that is the correct and expected behavior..
Also, the error happens not always, but it happens, and always, if it happens, it happens only if by first connection the first message was 1024 byte long and the second message was sent immediately after first message without the received data before.
Now the question to community, is that possible to switch off such strange buffering behavior in Netty? So that all messages received on server socket are sent exactly in same way to the client socket channel without merging the data.
Thank you in advance!
This "strange" behavior has nothing todo with netty. Its up to the network layer how much bytes get transfered at once, so its really expected to see this. if you need to have all 1024 bytes you will need t buffer them until you receive enough.
Ok, after long night I have finally solved my problem. It seems, the Netty project is still buggy in this way and will accept incoming messages for sending in the wrong order.
So, what I do, I fill the buffer with incoming messages until the remote connection by client gets opened, so I send than the full correct buffer instead to let things do by Netty.
I have written a small program which send/receives files from one client to another. I've set up progressbars for both the receiver and the client, but the problem is that the sender seems to have his progressbar finish much quicker than the actual transfer. The problem lies with the how it calculates how many bytes that have been written. I'm assuming it's counting how many bytes I've read into buffer, rather than bytes that were sent through the network, so how can I find a solution to this problem? The receiver is calculating his received bytes at a correct rate, but the sender is not doing his part correctly.
Setting a lower buffersize offsets the difference a bit, but it's still not correct. I've tried wrapping the outputstream with a CountingOutputStream, but it returns the same result as the code snippet below. The transfer eventually completes correctly, but I need the proper "sent" values to update my progressbar, as in what was actually received and written to disc at the receiver side. I've included a very stripped down code snippet which represents my way of calculating transferred bytes. Any examples of a solution would be very helpful.
try
{
int sent = 0;
Socket sk = new Socket(ip, port);
OutputStream output = sk.getOutputStream();
FileInputStream file = new FileInputStream(filepath);
byte[] buffer = new byte[8092];
while ((bytesRead = file.read(buffer)) > 0)
{
output.write(buffer, 0, bytesRead);
sent += bytesRead;
System.out.println(sent); // Shows incorrect values for the actual speed.
}
}
In short, I don't think you can get the sort of accurate visibility you're looking for solely from the "sender" side, given the number of buffers between you and the "wire" itself. But also, I don't think that matters. Here's why:
Bytes count as "sent" when they are handed to the network stack. When you are sending a small number of bytes (such as your 8K example) those bytes are going to be buffered & the write() calls will return quickly.
Once you're reached network saturation, your write() calls will start to block as the various network buffers become full - and thus then you'll get a real sense of the timings.
If you really must have some sort of "how many bytes have you received?" you'll have to have the receiving end send that data back periodically via an out-of-band mechanism (such as suggested by glowcoder)
Get the input stream from the socket, and on the other side, when you've written a selection of bytes to disk, write the result to the output stream. Spawn a second thread to handle the reading of this information, and link it to your counter.
Your variable is sent - it is accurate. What you need is a received or processed variable, and for that you will need two-way communication.
I'm programming a server (Java) - client (Android/Java) application. The server is a W7. All the communication goes well until one read in the client that freezes and stops reading data until I send it 2 times more.
The data not read is a byte array. I repeat that all the communication goes well until this point.
Here's the code that I use to send the data:
Long lLength = new Long(length);
byte [] bLength = this.longToBytes(lLength.longValue());
dos.write(bLength);
dos.flush();
dos.write(bLength);
dos.flush();
dos.write(bLength);
dos.flush();
This code transforms a long value into an 8 bytes array. As I said, when the first write is executed (and the client is waiting for data), the read is not done. It is not done until I execute the last write().
And here's the code to read it:
byte length[] = {0,0,0,0,0,0,0,0};
dis.read(length);
I've used Wireshark to sniff the traffic, and I can see that the byte array is send, and the client answers with an ACK, but the read is not done.
In the client and the server, the sockets are setup like this:
socket = new Socket(sIP, oiPort.intValue());
dos = new DataOutputStream(socket.getOutputStream());
dis = new DataInputStream(socket.getInputStream());
This is driving me mad... I don't know why, at one moment, the application stops reading the data, when I send it the same way as always.
I suppose that the problem may be in the input buffers of the client socket... But I don't know what to do or what to try...
Say that I've also test the server in a WXPSP3 and it still doesn't work.
First thing I'd look at is the code for your longToBytes method. Is it really creating a byte array of 8 bytes? If it is generating an array of less than 8 bytes, then that explains the problem. (Your client is expecting 8 bytes, and will block until they all arrive.)
Next thing I'd ask myself is why I'm not just using writeLong and readLong. It would simplify your code, and quite possibly cure the problem.