I have a question about reading bytes from server socket.
I'm using the default values for literacy 8192b bytes on the network.
The problem is that there are times you need to send files larger than the buffer capacity.
To avoid increasing the buffer as you can know that those packets arriving at different times are related?
Client side snippet:
{
String c = xxxxx //imagine that it an string format JSON with 64000bites length
OutputStream wsOS = socket.getOutputStream();
wsOS.write(new String(data,"UTF-8"));
wsOS.flush();
}
When server side received the JSON string:
{
byte[] buffer = new byte[8192];
int size = 0;
StringBuilder str = new StringBuilder();
size = wsIS.read(buffer);
if (size > 0) {
str.append(new String(buffer, "UTF-8")
.substring(0, size));
while (wsIS.available() > 0) {
size = wsIS.read(buffer);
str.append(new String(buffer, "UTF-8")
.substring(0, size));
}
}
}
Problem:
All string arrived in the server but in block 8192b - I can not concatenate the string because I don't know if the last string JSON is part of the previews.
Even if you increase the buffer size there is no guarantee that the whole string will be read in a single call to wsIS.read.
What this means is that you must have some mechanism to know where the previous string has ends and new one begins. Some choices are as follows:
Use ObjectOutputStream / ObjectInputStream
Write number of characters before actually writing the characters. On the reading side, read the length and then those many chars
Related
I am trying the following:
C# Client:
string stringToSend = "Hello man";
BinaryWriter writer = new BinaryWriter(mClientSocket.GetStream(),Encoding.UTF8);
//write number of bytes:
byte[] headerBytes = BitConverter.GetBytes(stringToSend.Length);
mClientSocket.GetStream().Write(headerBytes, 0, headerBytes.Length);
//write text:
byte[] textBytes = System.Text.Encoding.UTF8.GetBytes(stringToSend);
writer.Write(textBytes, 0, textBytes.Length);
Java Server:
Charset utf8 = Charset.forName("UTF-8");
BufferedReader in = new BufferedReader(new InputStreamReader(clientSocket.getInputStream(), utf8));
while (true) {
//we read header first
int headerSize = in.read();
int bytesRead = 0;
char[] input = new char[headerSize];
while (bytesRead < headerSize)
{
bytesRead += in.read(input, bytesRead, headerSize - bytesRead);
}
String resString = new String(input);
System.out.println(resString);
if (resString.equals("!$$$")) {
break;
}
}
The string size equals 9.That's correct on both sides.But, when I am reading the string iteself on the Java side, the data looks wrong.The char buffer ('input' variable)content looks like this:
",",",'H','e','l','l','o',''
I tried to change endianness with reversing the byte array.Also tried changing string encoding format between ASCII and UTF-8.I still feel like it relates to the endianness problem,but can not figure out how to solve it.I know I can use other types of writers in order to write text data to the steam,but I am trying using raw byte arrays for the sake of learning.
These
byte[] headerBytes = BitConverter.GetBytes(stringToSend.Length);
are 4 bytes. And they aren't character data so it makes no sense to read them with a BufferedReader. Just read the bytes directly.
byte[] headerBytes = new byte[4];
// shortcut, make sure 4 bytes were actually read
in.read(headerBytes);
Now extract your text's length and allocate enough space for it
int length = ByteBuffer.wrap(headerBytes).getInt();
byte[] textBytes = new byte[length];
Then read the text
int remaining = length;
int offset = 0;
while (remaining > 0) {
int count = in.read(textBytes, offset, remaining);
if (-1 == count) {
// deal with it
break;
}
remaining -= count;
offset += count;
}
Now decode it as UTF-8
String text = new String(textBytes, StandardCharsets.UTF_8);
and you are done.
Endianness will have to match for those first 4 bytes. One way of ensuring that is to use "network order" (big-endian). So:
C# Client
byte[] headerBytes = BitConverter.GetBytes(IPAddress.HostToNetworkOrder(stringToSend.Length));
Java Server
int length = ByteBuffer.wrap(headerBytes).order(ByteOrder.BIG_ENDIAN).getInt();
At first glance it appears you have a problem with your indexes.
You C# code is sending an integer converted to 4 bytes.
But you Java Code is only reading a single byte as the length of the string.
The next 3 bytes sent from C# are going to the three zero bytes from your string length.
You Java code is reading those 3 zero bytes and converting them to empty characters which represent the first 3 empty characters of your input[] array.
C# Client:
string stringToSend = "Hello man";
BinaryWriter writer = new BinaryWriter(mClientSocket.GetStream(),Encoding.UTF8);
//write number of bytes: Original line was sending the entire string here. Optionally if you string is longer than 255 characters, you'll need to send another data type, perhaps an integer converted to 4 bytes.
byte[] textBytes = System.Text.Encoding.UTF8.GetBytes(stringToSend);
mClientSocket.GetStream().Write((byte)textBytes.Length);
//write text the entire buffer
writer.Write(textBytes, 0, textBytes.Length);
Java Server:
Charset utf8 = Charset.forName("UTF-8");
BufferedReader in = new BufferedReader(new InputStreamReader(clientSocket.getInputStream(), utf8));
while (true) {
//we read header first
// original code was sending an integer as 4 bytes but was only reading a single char here.
int headerSize = in.read();// read a single byte from the input
int bytesRead = 0;
char[] input = new char[headerSize];
// no need foe a while statement here:
bytesRead = in.read(input, 0, headerSize);
// if you are going to use a while statement, then in each loop
// you should be processing the input but because it will get overwritten on the next read.
String resString = new String(input, utf8);
System.out.println(resString);
if (resString.equals("!$$$")) {
break;
}
}
How to read first 2 bytes from input stream and convert 2 bytes data into actual int length value, then read and copy the rest of message into byte array.
The rest of data array should be defined after reading first 2 bytes from the stream, does anyone know efficient logic?
Use a DataInputStream. Use the readUnsignedShort() method to return the length word, then the readFully() method to read the following data.
This creates a string from a byte array. Adapt as needed.
InputStream in;
try {
in = socket.getInputStream();
DataInputStream dis = new DataInputStream(in);
int len = dis.readInt();
byte[] data = new byte[len];
if (len > 0) {
dis.readFully(data);
}
String sReturn = new String(data);
}
I am trying to first read 4 bytes(int) specifying the size of the message and then read the remaining bytes based on the byte count. I am using the following code to accomplish this:
DataInputStream dis = new DataInputStream(
mClientSocket.getInputStream());
// read the message length
int len = dis.readInt();
Log.i(TAG, "Reading bytes of length:" + len);
// read the message data
byte[] data = new byte[len];
if (len > 0) {
dis.readFully(data);
} else {
return "";
}
return new String(data);
Is there a better/efficient way of doing this?
From JavaDocs of readUTF:
First, two bytes are read and used to construct an unsigned 16-bit
*integer* in exactly the manner of the readUnsignedShort method . This
integer value is called the UTF length and specifies the number of
additional bytes to be read. These bytes are then converted to
characters by considering them in groups. The length of each group is
computed from the value of the first byte of the group. The byte
following a group, if any, is the first byte of the next group.
The only problem with this is that your protocol seems to only send 4 bytes for the payload length. Perhaps you can do a similar method but increase the size of length sentinel read to 4 bytes/32-bits.
Also, I see that you are just doing new String(bytes) which works fine as long as the encoding of the data is the same as "the platform's default charset." See javadoc So it would be much safer to just ensure that you are encoding it correctly(e.g. if you know that the sender sends it as UTF-8 then do new String(bytes,"UTF-8") instead).
How about
DataInputStream dis = new DataInputStream(new BufferedInputStream(
mClientSocket.getInputStream()));
return dis.readUTF();
You can use read(byte[] b, int off, int len) like this
byte[] data = new byte[len];
dis.read(data,0,len);
i have a java code that SOMETIMES hangs when i run it.Its a put command that puts through sockets( and input stream etc a file from a server to a client) . Sometimes this works but sometimes it doesnt and i get an error message. i tried every possible path that can be taken through debugging and i can never get it to hang. Is there a way to inspect the thing when its hang through eclipse?`
if (sentence.length() > 3 && sentence.substring(0, 3).equals("put")) {
File checkFile = new File(dir.getCurrentPath(), sentence.substring(4));
if (checkFile.isFile() && checkFile.exists()) {
try {
outToServer.writeBytes(sentence + "\n");
boolean cont = false;
String x;
while (!cont) {
if ((x = inFromServer.readLine()).equals("continue")) {
cont = true;
}
}
String name = sentence.substring(4);
copy.copyFile(name);
// outToServer.writeBytes("continue" + "\n");
this is the client code that recieves the PUT request(i.e. put test.txt takes the file test.txt and puts it in the server's local dir.
Copy file: (the thing that copies the data)
File checkFile = new File(dir.getCurrentPath(), file);
if (checkFile.isFile() && checkFile.exists()) {
DataOutputStream outToClient = new DataOutputStream(socket.getOutputStream());
// byte[] receivedData = new byte[8192];
File inputFile = new File(dir.getCurrentPath(), file);
byte[] receivedData = new byte[(int) inputFile.length()];
// String theLength = "" + inputFile.length();
outToClient.writeBytes("" + inputFile.length() + "\n");
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(getCopyPath(file)));
// if (extension.equals("")) {
// extension = "txt";
// }
// BufferedReader inFromClient = new BufferedReader(new InputStreamReader(socket.getInputStream()));
// boolean cont = false;
// while (!cont) {
// if (inFromClient.readLine().equals("continue")) {
// cont = true;
//
// }
// }
// outToClient.writeBytes("continue" + "\n");
bis.read(receivedData, 0, receivedData.length);
OutputStream os = socket.getOutputStream();
os.write(receivedData, 0, receivedData.length);
// outToClient.writeBytes("finish" + "\n");
os.flush();
Protocol on server(copies stuff in the file)
if (get.equals("put")) {
//so the client sends: the put request
//then sends the length
try {
BufferedReader inFromClient = new BufferedReader(new InputStreamReader(socket.getInputStream()));
DataOutputStream outToClient = new DataOutputStream(socket.getOutputStream());
outToClient.writeBytes("continue" + "\n");
int length = Integer.parseInt(inFromClient.readLine());
// String x = "";
// boolean cont = false;
// while (!cont) {
// String temp = inFromClient.readLine();
// if (temp.equals("finish")) {
// cont = true;
// }
// else {
// x += temp;
// }
// }
byte[] recieveFile = new byte[length];
InputStream is = socket.getInputStream();
FileOutputStream fos = new FileOutputStream("Copy " + input.substring(4));
BufferedOutputStream bos = new BufferedOutputStream(fos);
int bytesRead;
int current = 0;
bytesRead = is.read(recieveFile, 0, recieveFile.length);
current = bytesRead;
If you cant answer the question in specific just tell me how i can debug the code that hangs or how i can debug concurrent code.(by the way the way that the signal passing is done is by passing a token as you can see i.e. the server sents a continue token that tells the client to start sending data, i havent done this with threads methods notify and wait since i cant use , since any single object has only 1 method.
when your process hangs, get the current stack dump for the process. this will show you why the process is hung.
note, you have at least one bug in your program, in that you are not handling the return value from the InputStream.read() method which will, at the very least, cause you to have busted data on the server side. (see #rk2010's answer for more details).
you main bug, though, is that when you wrap the BufferedReader around the socket InputStream, you are probably going to end up "stealing" more bytes from the stream than the just length value. when a BufferedReader reads data from the underlying stream, it can read more data than it actually returns in the readLine() method (so, it may read 1000 chars internally, but the first "line" may only contain 20 chars). if you continue to use the BufferedReader, everything is fine, but if you discard the BufferedReader and try to read more data from the underlying stream, you will have less data there than you expect. so, when you go to read the file contents, there aren't enough bytes available.
instead, you should be using DataOutputStream/DataInputStream exclusively. write the length as a long value (i.e. DataOutputStream.writeLong(long)), then write the bytes after (i.e. DataOutputStream.write(byte[])). then read the data using the corresponding methods in DataInputStream. since you can use the DataInputStream exclusively when reading (first to read the file size and then to read the actual file bytes), you don't risk losing bytes when switching read "modes" (additionally, DataInputStream does not do any internal buffering like BufferedReader). don't use Readers/Writers at all.
add logging after every read and write command. Give each thread a name.
run the program, see how the log matches up with what you expect.
note that you can't always rely on read method like the way you are. Safe way to read is by looping until you get a negative length.
int len = -1;
byte[] buff = new byte[1024]; // for 4KB, use: 4* 1024
while ( (len = is.read(buff, 0, buff.length)) > -1){
// copy len number of bytes from buff array into some other place.
}
Check out IOUtils.copy method
from Apache Commons IOUtils class
I use the following code (from Bluetooth Chat sample app) to read the incoming data and construct a string out of the bytes read. I want to read until this string has arrived <!MSG>. How to insert this condition with read() function?
The whole string looks like this <MSG><N>xxx<!N><V>yyy<!V><!MSG>. But the read() function does not read entire string at once. When I display the characters, I cannot see all the characters in the same line. It looks like:
Sender: <MS
Sender: G><N>xx
Sender: x<V
.
.
.
I display the characters on my phone (HTC Desire) and I send the data using windows hyperterminal.
How to make sure all the characters are displayed in a single line? I have tried using StringBuilder and StringBuffer instead of new String() but the problem is read() function does not read all the characters sent. The length of the input stream (bytes) is not equal to actual length of the string sent. The construction of string from the read bytes is happening alright.
Thank you for any suggestions and time spent on this. Also please feel free to suggest other mistakes or better way of doing below things, if any.
Cheers,
Madhu
public void run() {
Log.i(TAG, "BEGIN mConnectedThread");
//Writer writer = new StringWriter();
byte[] buffer = new byte[1024];
int bytes;
//String end = "<!MSG>";
//byte compare = new Byte(Byte.parseByte(end));
// Keep listening to the InputStream while connected
while (true) {
try {
//boolean result = buffer.equals(compare);
//while(true) {
// Read from the InputStream
bytes = mmInStream.read(buffer);
//Reader reader = new BufferedReader(new InputStreamReader(mmInStream, "UTF-8"));
//int n;
//while ((bytes = reader.read(buffer)) != -1) {
//writer.write(buffer, 0, bytes);
//StringBuffer sb = new StringBuffer();
//sb = sb.append(buffer);
//String readMsg = writer.toString();
String readMsg = new String(buffer, 0, bytes);
//if (readMsg.endsWith(end))
// Send the obtained bytes to the UI Activity
mHandler.obtainMessage(BluetoothChat.MESSAGE_READ, bytes, -1, readMsg)
.sendToTarget();
//}
} catch (IOException e) {
Log.e(TAG, "disconnected", e);
connectionLost();
break;
}
}
}
The read function does not make any guarantee about the number of bytes it returns (it generally tries to return as many bytes from the stream as it can, without blocking). Therefore, you have to buffer the results, and keep them aside until you have your full message. Notice that you could receive something after the "<!MSG>" message, so you have to take care not to throw it away.
You can try something along these lines:
byte[] buffer = new byte[1024];
int bytes;
String end = "<!MSG>";
StringBuilder curMsg = new StringBuilder();
while (-1 != (bytes = mmInStream.read(buffer))) {
curMsg.append(new String(buffer, 0, bytes, Charset.forName("UTF-8")));
int endIdx = curMsg.indexOf(end);
if (endIdx != -1) {
String fullMessage = curMsg.substring(0, endIdx + end.length());
curMsg.delete(0, endIdx + end.length());
// Now send fullMessage
}
}