I have a client/server application that sends/receives data using BufferedOutputStream / BufferedInputStream . The protocol of communication is the following:
Send part :
first byte is the action to perform
next 4 bytes are the length of the message
next x bytes (x=length of message) are the message itself
Receive part :
read first byte to get the action
read the next 4 bytes to get the message length
read the x (obtained on prev step) bytes to get the message
Now the problem is that sometimes when i sent the length of the message (ex : 23045) on server part when i receive it i get a huge int (ex: 123106847).
A important clue is that this happens only when message exceeds a number of characters (in my case > 10K ) , if i sent a smaller message (ex 4-5k) everything works as expected.
Client send part (outputStream/inputStream are the type BufferedXXXStream):
private String getResponseFromServer( NormalizerActionEnum action, String message) throws IOException{
writeByte( action.id());
writeString( message);
flush(;
return read();
}
private String read() throws IOException{
byte[] msgLen = new byte[4];
inputStream.read(msgLen);
int len = ByteBuffer.wrap(msgLen).getInt();
byte[] bytes = new byte[len];
inputStream.read(bytes);
return new String(bytes);
}
private void writeByte( byte msg) throws IOException{
outputStream.write(msg);
}
private void writeString( String msg) throws IOException{
byte[] msgLen = ByteBuffer.allocate(4).putInt(msg.length()).array();
outputStream.write(msgLen);
outputStream.write(msg.getBytes());
}
private void flush() throws IOException{
outputStream.flush();
}
Server part (_input/_output are the type BufferedXXXStream)
private byte readByte() throws IOException, InterruptedException {
int b = _input.read();
while(b==-1){
Thread.sleep(1);
b = _input.read();
}
return (byte) b;
}
private String readString() throws IOException, InterruptedException {
byte[] msgLen = new byte[4];
int s = _input.read(msgLen);
while(s==-1){
Thread.sleep(1);
s = _input.read(msgLen);
}
int len = ByteBuffer.wrap(msgLen).getInt();
byte[] bytes = new byte[len];
s = _input.read(bytes);
while(s==-1){
Thread.sleep(1);
s = _input.read(bytes);
}
return new String(bytes);
}
private void writeString(String message) throws IOException {
byte[] msgLen = ByteBuffer.allocate(4).putInt(message.length()).array();
_output.write(msgLen);
_output.write(message.getBytes());
_output.flush();
}
....
byte cmd = readByte();
String message = readString();
Any help will be greatly appreciated. If you need additional details let me know.
UPDATE: Due to comments from Jon Skeet and EJP i realized that the read part on the server was having some pointless operations but letting this aside i finally got what the problem was: the key thing is that i keep the streams opened for the full length of the app and the first several times i sent the message length i'm able to read it on the server side BUT as Jon Skeet pointed out the data doesn't arrive all at once so when i try to read the message length again i'm actually reading from the message itself that is why i have bogus message lengths .
~ instead of sending the data length and then reading it all at once i sent it without the length and i read one byte at a time till the end of the string which works perfectly
private String readString() throws IOException, InterruptedException {
StringBuilder sb = new StringBuilder();
byte[] bytes = new byte[100];
int s = 0;
int index=0;
while(true){
s = _input.read();
if(s == 10){
break;
}
bytes[index++] = (byte) (s);
if(index == bytes.length){
sb.append(new String(bytes));
bytes = new byte[100];
index=0;
}
}
if(index > 0){
sb.append(new String(Arrays.copyOfRange(bytes, 0, index)));
}
return sb.toString();
}
Look at this:
byte[] bytes = new byte[len];
s = _input.read(bytes);
while(s==-1){
Thread.sleep(1);
s = _input.read(bytes);
}
return new String(bytes);
Firstly, the loop is pointless: the only time read will return -1 is if it's closed, in which case looping isn't going to help you.
Secondly, you're ignoring the possibility that the data will come in more than one chunk. You're assuming that if you've managed to get any data, you've got all the data. Instead, you should loop something like this:
int bytesRead = 0;
while (bytesRead < bytes.length) {
int chunk = _input.read(bytes, bytesRead, bytes.length - bytesRead);
if (chunk == -1) {
throw new IOException("Didn't get as much data as we should have");
}
bytesRead += chunk;
}
Note that all your other InputStream.read calls also assume that you've managed to read data, and indeed that you've read all the data you need.
Oh, and you're using the platform-default encoding to convert between binary data and text data - not a good idea.
Is there any reason you're not using DataInputStream and DataOutputStream for this? Currently you're reinventing the wheel, and doing so with bugs.
You sending code is bugged:
byte[] msgLen = ByteBuffer.allocate(4).putInt(message.length()).array();
_output.write(msgLen);
_output.write(message.getBytes());
You send the number of characters as the message length, but after that you convert the message to bytes. Depending on the platform encoding String.getBytes() can give you much more bytes than there are characters.
You should never assume that String.length() has any relationship with String.getBytes().length! Those are different concepts and should never be mixed.
Related
First post, usually I find what Im looking for in other threads but not this time:
Im using javas Deflater and Inflater to compress/ decompress some data I send between a server and client application that Im working on.
It works just fine for 99% of my tests. However there is one particular dataset that when inflated throws this exception from the inflater.inflate() method:
DataFormatException: incorrect header check
There is nothing special about the data compared to the other runs. Its just a bunch of numbers seperated by commas "encoded" as a String and then done .getBytes() to. The only thing I know is that its a bit bigger this time. There is not encoding happening anywhere between the compression -> decompression steps.
This is the code to send something to either the client or the server. The code is shared.
OutputStream outputStream = new DataOutputStream(socket.getOutputStream());
byte[] uncompressed = SOMEJSON.toString().getBytes();
int realLength = uncompressed.length;
// compress data
byte[] compressedData = ByteCompression.compress(uncompressed);
int compressedLength = compressedData.length;
outputStream.write(ByteBuffer.allocate(Integer.BYTES).putInt(compressedLength).array());
outputStream.write(ByteBuffer.allocate(Integer.BYTES).putInt(realLength).array());
outputStream.write(compressedData);
outputStream.flush();
This is the code to receive data (either client or server) also shared:
DataInputStream dataIn = new DataInputStream(socket.getInputStream());
int compressedLength = dataIn.readInt();
int realLength = dataIn.readInt();
errorhandling.info("Packet Reader", "Expecting " + compressedLength + " (" + realLength + ") bytes.");
byte[] compressedData = new byte[compressedLength];
int readBytes = 0;
while (readBytes < compressedLength) {
int newByteAmount = dataIn.read(compressedData);
// catch nothing being read or end of line
if (newByteAmount <= 0) {
break;
}
readBytes += newByteAmount;
}
if (readBytes != compressedLength) {
errorhandling.info("Packet Reader", "Read byte amount differs from expected bytes.");
return new ErrorPacket("Read byte amount differs from expected bytes.").create();
}
byte[] uncompressedData = ByteCompression.decompress(compressedData, realLength);
String packetData = new String(uncompressedData);
Here are the methods to compress and decompress a byteArray (you guessed right its shared):
public static byte[] compress(byte[] uncompressed) {
Deflater deflater = new Deflater(Deflater.BEST_COMPRESSION);
deflater.setInput(uncompressed);
deflater.finish();
byte[] compressed = new byte[uncompressed.length];
int compressedSize = 0;
while (!deflater.finished()) {
compressedSize += deflater.deflate(compressed);
}
deflater.end();
return Arrays.copyOfRange(compressed, 0, compressedSize);
}
public static byte[] decompress(byte[] compressed, int realLength) throws DataFormatException {
Inflater inflater = new Inflater(true);
inflater.setInput(compressed);
byte[] uncompressed = new byte[realLength];
while (!inflater.finished()) {
inflater.inflate(uncompressed); // throws DataFormatException: incorrect header check (but only super rarely)
}
inflater.end();
return uncompressed;
}
So far Ive tried differnt compression levels and messing with the "nowrap" option for both Deflater and Inflater (all combinations):
// [...]
Deflater deflater = new Deflater(Deflater.BEST_COMPRESSION, true);
// [...]
Inflater inflater = new Inflater(true);
But that just results in these exceptions (but again only for that one particulat dataset):
DataFormatException: invalid stored block lengths
DataFormatException: invalid distance code
Im sorry for this wall of text but at this point I really dont know anymore what could be causing this issue.
Alright here is the solution:
My assumption was that this loop would APPEND new read data to the byte array where it last stopped THIS IS NOT THE CASE (it seems to stop reading after 2^16 bytes so thats why I dont get this issue with smaller packets).
This is wrong:
int readBytes = 0;
while (readBytes < compressedLength) {
int newByteAmount = dataIn.read(compressedData); // focus here!
readBytes += newByteAmount;
}
So whats happening is that the data is read correctly however the output array is overwriting itself!! Thats why I see wrong data at the start and a bunch of 00 00 at the end (because it never actually reached that part of the array)!
Using this instead fixed my issue:
dataIn.readFully(compressedData);
What concerns me is that I see the first variant of the code A LOT. Thats what I found when googling it.
I wanted to use Base64.java to encode and decode files. Encode.wrap(InputStream) and decode.wrap(InputStream) worked but runned slowly. So I used following code.
public static void decodeFile(String inputFileName,
String outputFileName)
throws FileNotFoundException, IOException {
Base64.Decoder decoder = Base64.getDecoder();
InputStream in = new FileInputStream(inputFileName);
OutputStream out = new FileOutputStream(outputFileName);
byte[] inBuff = new byte[BUFF_SIZE]; //final int BUFF_SIZE = 1024;
byte[] outBuff = null;
while (in.read(inBuff) > 0) {
outBuff = decoder.decode(inBuff);
out.write(outBuff);
}
out.flush();
out.close();
in.close();
}
However, it always throws
Exception in thread "AWT-EventQueue-0" java.lang.IllegalArgumentException: Input byte array has wrong 4-byte ending unit
at java.util.Base64$Decoder.decode0(Base64.java:704)
at java.util.Base64$Decoder.decode(Base64.java:526)
at Base64Coder.JavaBase64FileCoder.decodeFile(JavaBase64FileCoder.java:69)
...
After I changed final int BUFF_SIZE = 1024; into final int BUFF_SIZE = 3*1024;, the code worked. Since "BUFF_SIZE" is also used to encode file, I believe there were something wrong with the file encoded (1024 % 3 = 1, which means paddings are added in the middle of the file).
Also, as #Jon Skeet and #Tagir Valeev mentioned, I should not ignore the return value from InputStream.read(). So, I modified the code as below.
(However, I have to mention that the code does run much faster than using wrap(). I noticed the speed difference because I had coded and intensively used Base64.encodeFile()/decodeFile() long before jdk8 was released. Now, my buffed jdk8 code runs as fast as my original code. So, I do not know what is going on with wrap()... )
public static void decodeFile(String inputFileName,
String outputFileName)
throws FileNotFoundException, IOException
{
Base64.Decoder decoder = Base64.getDecoder();
InputStream in = new FileInputStream(inputFileName);
OutputStream out = new FileOutputStream(outputFileName);
byte[] inBuff = new byte[BUFF_SIZE];
byte[] outBuff = null;
int bytesRead = 0;
while (true)
{
bytesRead = in.read(inBuff);
if (bytesRead == BUFF_SIZE)
{
outBuff = decoder.decode(inBuff);
}
else if (bytesRead > 0)
{
byte[] tempBuff = new byte[bytesRead];
System.arraycopy(inBuff, 0, tempBuff, 0, bytesRead);
outBuff = decoder.decode(tempBuff);
}
else
{
out.flush();
out.close();
in.close();
return;
}
out.write(outBuff);
}
}
Special thanks to #Jon Skeet and #Tagir Valeev.
I strongly suspect that the problem is that you're ignoring the return value from InputStream.read, other than to check for the end of the stream. So this:
while (in.read(inBuff) > 0) {
// This always decodes the *complete* buffer
outBuff = decoder.decode(inBuff);
out.write(outBuff);
}
should be
int bytesRead;
while ((bytesRead = in.read(inBuff)) > 0) {
outBuff = decoder.decode(inBuff, 0, bytesRead);
out.write(outBuff);
}
I wouldn't expect this to be any faster than using wrap though.
Try to use decode.wrap(new BufferedInputStream(new FileInputStream(inputFileName))). With buffering it should be at least as fast as your manually crafted version.
As for why your code doesn't work: that's because the last chunk is likely to be shorter than 1024 bytes, but you try to decode the whole byte[] array. See the #JonSkeet answer for details.
Well, I changed
"final int BUFF_SIZE = 1024;"
into
"final int BUFF_SIZE = 1024 * 3;"
It worked!
So, I guess probabaly there is something wrong with padding... I mean, when encoding the file, (since 1024 % 3 = 1) there must be paddings. And those might raise problems when decoding...
You should records the number of bytes you have read, beside this,
You should be sure that your buffer size is divisible for 3, cause in Base64, every 3 bytes have four output(64 is 2^6, and 3*8 equals 4*6), by doing this, you can avoid padding problems.( In this way your output will not have the wrong ending of "=")
I'm trying to read all bytes from a web site but I think I don't get all bytes. I give a high value for bytes array length. I used this method but it always returns an exception.
Here is the code:
DataInputStream dis = new DataInputStream(s2.getInputStream());
byte[] bytes = new byte[900000];
// Read in the bytes
int offset = 0;
int numRead = 0;
while (offset < bytes.length
&& (numRead=dis.read(bytes, offset, bytes.length-offset)) >= 0) {
offset += numRead;
}
// Ensure all the bytes have been read in
if (offset < bytes.length) {
throw new IOException("Could not completely read website");
}
out.write(bytes);
Edited Version:
ByteArrayOutputStream bais = new ByteArrayOutputStream();
InputStream is = null;
try {
is = s2.getInputStream();
byte[] byteChunk = new byte[4096]; // Or whatever size you want to read in at a time.
int n;
while ( (n = is.read(byteChunk)) > 0 ) {
bais.write(byteChunk, 0, n);
}
}
catch (IOException e) {
System.err.printf ("Failed while reading bytes");
e.printStackTrace ();
// Perform any other exception handling that's appropriate.
}
finally {
if (is != null) { is.close(); }
}
byte[] asd = bais.toByteArray();
out.write(asd);
This is the problem:
if (offset < bytes.length)
You'll only trigger that if the original data is more than 900,000 bytes. If the response is entirely complete in less than that, read() will return -1 correctly to indicate the end of the stream.
You should actually be throwing an exception if offset is equal to bytes.length, as that indicates that you might have truncated data :)
It's not clear where you got the 900,000 value from, mind you...
I would suggest that if you want to stick with the raw stream, you use Guava's ByteStreams.toByteArray method to read all the data. Alternatively, you could keep looping round, reading into a smaller buffer, writing into a ByteArrayOutputStream on each iteration.
I realise this doesn't answer your specific question. However I really wouldn't hand-code this sort of thing, when libraries such as HttpClient exist and are debugged/profiled etc.
e.g. here's how to use the fluent interface
Request.Get("http://targethost/homepage").execute().returnContent();
JSoup is an alternative if you're dealing with grabbing and scraping HTML.
I use the following code (from Bluetooth Chat sample app) to read the incoming data and construct a string out of the bytes read. I want to read until this string has arrived <!MSG>. How to insert this condition with read() function?
The whole string looks like this <MSG><N>xxx<!N><V>yyy<!V><!MSG>. But the read() function does not read entire string at once. When I display the characters, I cannot see all the characters in the same line. It looks like:
Sender: <MS
Sender: G><N>xx
Sender: x<V
.
.
.
I display the characters on my phone (HTC Desire) and I send the data using windows hyperterminal.
How to make sure all the characters are displayed in a single line? I have tried using StringBuilder and StringBuffer instead of new String() but the problem is read() function does not read all the characters sent. The length of the input stream (bytes) is not equal to actual length of the string sent. The construction of string from the read bytes is happening alright.
Thank you for any suggestions and time spent on this. Also please feel free to suggest other mistakes or better way of doing below things, if any.
Cheers,
Madhu
public void run() {
Log.i(TAG, "BEGIN mConnectedThread");
//Writer writer = new StringWriter();
byte[] buffer = new byte[1024];
int bytes;
//String end = "<!MSG>";
//byte compare = new Byte(Byte.parseByte(end));
// Keep listening to the InputStream while connected
while (true) {
try {
//boolean result = buffer.equals(compare);
//while(true) {
// Read from the InputStream
bytes = mmInStream.read(buffer);
//Reader reader = new BufferedReader(new InputStreamReader(mmInStream, "UTF-8"));
//int n;
//while ((bytes = reader.read(buffer)) != -1) {
//writer.write(buffer, 0, bytes);
//StringBuffer sb = new StringBuffer();
//sb = sb.append(buffer);
//String readMsg = writer.toString();
String readMsg = new String(buffer, 0, bytes);
//if (readMsg.endsWith(end))
// Send the obtained bytes to the UI Activity
mHandler.obtainMessage(BluetoothChat.MESSAGE_READ, bytes, -1, readMsg)
.sendToTarget();
//}
} catch (IOException e) {
Log.e(TAG, "disconnected", e);
connectionLost();
break;
}
}
}
The read function does not make any guarantee about the number of bytes it returns (it generally tries to return as many bytes from the stream as it can, without blocking). Therefore, you have to buffer the results, and keep them aside until you have your full message. Notice that you could receive something after the "<!MSG>" message, so you have to take care not to throw it away.
You can try something along these lines:
byte[] buffer = new byte[1024];
int bytes;
String end = "<!MSG>";
StringBuilder curMsg = new StringBuilder();
while (-1 != (bytes = mmInStream.read(buffer))) {
curMsg.append(new String(buffer, 0, bytes, Charset.forName("UTF-8")));
int endIdx = curMsg.indexOf(end);
if (endIdx != -1) {
String fullMessage = curMsg.substring(0, endIdx + end.length());
curMsg.delete(0, endIdx + end.length());
// Now send fullMessage
}
}
I have a TCP socket client receiving messages (data) from a server.
messages are of the type length (2 bytes) + data (length bytes), delimited by STX & ETX characters.
I'm using a bufferedReader to retrieve the two first bytes, decode the length, then read again from the same bufferedReader the appropriate length and put the result in a char array.
most of the time, I have no problem, but SOMETIMES (1 out of thousands of messages received), when attempting to read (length) bytes from the reader, I get only part of it, the rest of my array being filled with "NUL" characters. I imagine it's because the buffer has not yet been filled.
char[] bufLen = new char[2];
_bufferedReader.read(bufLen);
int len = decodeLength(bufLen);
char[] _rawMsg = new char[len];
_bufferedReader.read(_rawMsg);
return _rawMsg;
I solved the problem in several iterative ways:
first I tested the last char of my array: if it wasn't ETX I would read chars from the bufferedReader one by one until I would reach ETX, then start over my regular routine. the
consequence is that I would basically DROP one message.
then, in order to still retrieve that message, I would find the first occurence of the NUL char in my "truncated" message, read & store additional characters one at a time until I reached ETX, and append them to my "truncated" messages, confirming length is ok.
it works also, but I'm really thinking there's something I could do better, like checking if the total number of characters I need are available in the buffer before reading it, but can't find the right way to do it...
any idea / pointer ?
thanks !
The InputStream read method may return short reads; you must check the return value to determine how many characters were read, and continue reading in a loop until you get the number you wanted. The method may block, but it only blocks until some data is available, not necessarily all the data you requested.
Most people end up writing a "readFully" method, like DataInputStream, which reads the amount of data expected, or throws an IOException:
static public int readFully(InputStream inp, byte[] arr, int ofs, int len) throws IOException {
int rmn,cnt;
for(rmn=len; rmn>0; ofs+=cnt, rmn-=cnt) {
if((cnt=inp.read(arr,ofs,rmn))==-1) {
throw new IOException("End of stream encountered before reading at least "+len+" bytes from input stream");
}
}
return len;
}
Here is a sample server that I have used for testing
The main rcv is structured like
while((chars_read = from_server.read(buffer)) != -1)
{
to_user.write(buffer,0,chars_read);
to_user.flush();
}
The actual whole server is below ...
public static void main(String[] args) throws IOException
{
try
{
if (args.length != 2)
throw new IllegalArgumentException("Wrong number of Args");
String host = args[0];
int port = Integer.parseInt(args[1]);
Socket s = new Socket(host,port);
final Reader from_server = new InputStreamReader(s.getInputStream());
PrintWriter to_server = new PrintWriter(new OutputStreamWriter(s.getOutputStream()));
BufferedReader from_user = new BufferedReader(new InputStreamReader(System.in));
final PrintWriter to_user = new PrintWriter(new OutputStreamWriter(System.out));
to_user.println("Connected to " + s.getInetAddress() + ":" + s.getPort());
Thread t = new Thread()
{
public void run()
{
char [] buffer = new char[1024];
int chars_read;
try
{
while((chars_read = from_server.read(buffer)) != -1)
{
to_user.write(buffer,0,chars_read);
to_user.flush();
}
}
catch(IOException e)
{
to_user.println(e);
}
to_user.println("Connection closed by server");
to_user.flush();
System.exit(0);
}
};
t.setPriority(Thread.currentThread().getPriority() + 1);
t.start();
String line;
while ((line = from_user.readLine()) != null)
{
to_server.println(line);
to_server.flush();
}
//t.stop();
s.close();
to_user.println("Connection closed by client");
to_user.flush();
}
catch(Throwable e)
{
e.printStackTrace();
System.err.println("Usage : java TCPClient <hostname> <port>");
}
}