I have a Java application that is reading data from a TCP socket that is receiving XML of varying size. The first 5 bytes of a given packet are supposed to indicate the size of the remaining message. I can read the message and xml successfully if I manually create a large byte[] and read the data.
Here are the instructions from the manual for the application that is generating the data:
Each message is preceded by the message size indicator which is a
32-bit unsinged integer using the network bytes order method. For
example: \x05\x00\x00\x00\x30\x31\x30\x32\x00 indicates the message
size of an ack which is 5 bytes included the fifth message byte '\0'. The
size indicator specifies everything following the size indicator
itself.
However I can't figure out how to decode the first 5 bytes to an integer that I can use to correctly size a byte[] for reading the rest of the message. I get random results:
Here is the code I'm using to parse the message:
DataOutputStream out = new DataOutputStream(clientSocket.getOutputStream());
BufferedInputStream inFromServer = new BufferedInputStream(clientSocket.getInputStream());
byte[] data = new byte[10];
inFromServer.read(data);
String result = new String(data, "ISO-8859-1");
Logger.info(data+"");
//PROBLEM AREA: Tried reading different byte lengths but no joy
//This should be a number but it never is. Often strange symbols
byte[] numeric = Arrays.copyOfRange(data,1,5);
String numericString = new String(numeric, "ISO-8859-1");
//Create a huge array to make sure everything gets captured.
//Want to use the parsed value from the start here
byte[] message = new byte[1000000];
inFromServer.read(message);
//This works as expected and returns correctly formatted XML
String fullMessage = new String(message, "ISO-8859-1");
Logger.info("Result "+result+ " Full message "+fullMessage);
The length looks like it's little endian. You can still use DataInputStream but you have to swap the bytes. If you used NIO's SocketChannel and a ByteBuffer you could set the byte order, but this is likely to be harder to use.
// only do this once per socket.
DataInputStream in = new DataInputStream(
new BufferedInputStream(clientSocket.getInputStream()));
// for each message.
int len0 = in.readInt();
int len = Integer.reverseBytes(len0);
assert len < 1 << 24;
byte[] bytes = new byte[len];
in.readFully(bytes);
String text = new String(bytes, "ISO-8859-1").trim();
int number = Integer.parseInt(text);
Network byte order is aka big-endian. But seeing your data it seems, that actually little-endian is used. At least 5 will look like those first 4 bytes in little-endian, but not in big-endian. So you need to read those bytes, consider little-endian and convert to long to consider "unsigned-ness".
public static void main(String[] args) throws IOException {
DataInputStream inFromServer = new DataInputStream(new BufferedInputStream(null));
int iSize = inFromServer.readInt();
iSize = Integer.reverseBytes(iSize); //read as little-endian
long count = Integer.toUnsignedLong(iSize); //unsigned int
}
Related
Trying to send a byte[] through TCP Socket in Java, it sends a specific array as two separate messages, while other arrays are sending as one message.
In more details, I convert a hex String to a byte[] using the following function:
public static byte[] hexStringToByteArray(String s) {
byte[] b = new byte[s.length() / 2];
for (int i = 0; i < b.length; i++) {
int index = i * 2;
int v = Integer.parseInt(s.substring(index, index + 2), 16);
b[i] = (byte) v;
}
return b;
}
And then I send it through Java TCP Socket:
ServerSocket server = new ServerSocket(3030);
Socket socket = server.accept();
OutputStream out = socket.getOutputStream();
// sample hex string
String msg = "F0700F8000F42400001544952414E00000000000000000000000662000E00060000";
out.write(HexUtiles.hexStringToByteArray(msg));
In debugging the code I found out it separate the array in byte number 1024. Also, increasing and decreasing the socket buffer size made no differences.
In addition, there is no 0A (Hex String of \n) in the message! I guess there is some strange behavior in the write method that sends a byte[] array as two messages! How can I send the byte[] as only one message?
There are no "messages" in TCP. Everything is sent as a stream and it's undefined how it will be chunked on the other end. Chunks might be split or joined together.
Send the byte size before each array using DataOutputStream. Read the size on the other end using DataInputStream and then that many bytes. This way you will be able to recreate your arrays at the other end.
Another option is to use ObjectOutputStream and ObjectInputStream and send the arrays as objects.
I am reading TCP packets that I use to display an image. Each packet is supposed to have a 1025 length. And each time, I get 128 lines, I draw an image.
SO I begin by initializing
s = new Socket(ip, port);
stream = s.getInputStream();
byte[] tmpBuffer = new byte[1025];
byte[] finalBuffer = new byte[128 * 1025];
int count_lines =0
and then I read stream by stream of length 1025,
while((read = stream.read(tmpBuffer, 0, tmpBuffer.length)) != -1){
System.arraycopy(tmpBuffer, 1, finalBuffer, count_lines * 1025, 1025);
count_lines++;
if(count_lines == 128) break;
}
The problem is when I log the read integer, I get a bunch of 1025 but sometimes (apparently randomly) 423 or 602 (noticing that 423+602=1025)
Am I going wrong with the TCP reading or is there a problem on the server side ?
Am I going wrong with the TCP reading or is there a problem on the server side ?
In TCP you have only a stream of bytes. You have messages nor control over packets which are typically no more than 1532 bytes long.
You have to have your own protocol to handle sending of messages.
inputStream.read(buffer) will read between 1 byte and buffer.length and you can't ignore the actual length read as you are doing.
If you want to read into a buffer of a known length, you can use
DataInputStream dis = new DataInputStream(socket.getInputStream());
byte[] finalBuffer = new byte[128 * 1025];
dis.readFully(finalBuffer);
The readFully reads as much as it can at a time until the byte[] has been fully read.
A more flexible protocol is to send the length first.
public static void write(DataOutputStream out, byte[] bytes) {
out.writeInt(bytes.length());
out.write(bytes);
}
public static byte[] read(DataInputStream in) {
int length = in.readInt();
byte[] bytes = new byte[length];
in.readFully(bytes);
return bytes;
}
This way, whatever length of data you send, you will recieve in the same manner.
I'm trying to write some string (as ascii) by using network stream (tcp/ssl) to server, i.e., local C# console app. The code looks like this:
public void SendMessage(String message) throws IOException {
OutputStream stream = client.getOutputStream();
// message is a String of 27 characters.
stream.write(message.getBytes("US-ASCII"), 0, message.getBytes("US-ASCII").length);
stream.flush();
}
This works fine on emulator, however, when I run app on my device (Galaxy A5 on Lolipop 5.0.2), it sends only one byte. What is the problem here ?
My receiving code on server side, I first try to read first four bytes (C#):
var messageData = new StringBuilder();
var buffer = new byte[4];
var bytes = sslStream.Read(buffer, 0, 4);
var len = Encoding.ASCII.GetString(buffer);
var chars = new char[Encoding.ASCII.GetCharCount(buffer, 0, bytes)];
// Use Decoder class to convert from bytes to ACII
// in case a character spans two buffers.
Encoding.ASCII.GetChars(buffer, 0, bytes, chars, 0);
messageData.Append(chars);
Try to use the standart StandardCharsets.US_ASCII instead of your hard coded String, which is always a bad idea.
yourString.getBytes(StandardCharsets.US_ASCII);
i have to send a short string as text from client to server and then after that send a binary file.
how would I send both binary file and the string using the same socket connection?
the server is a java desktop application and the client is an Android tablet. i have already set it up to send text messages between the client and server in both directions. i have not yet done the binary file sending part.
one idea is to set up two separate servers running at the same time. I think this is possible if i use two different port numbers and set up the servers on two different threads in the application. and i would have to set up two concurrent clients running on two services in the Android app.
the other idea is to somehow use an if else statement to determine which of the two types of files is being sent, either text of binary, and use the appropriate method to receive the file for the file type being sent.
example code for sending text
PrintWriter out;
BufferedReader in;
out = new PrintWriter(new BufferedWriter
(new OutputStreamWriter(Socket.getOutputStream())) true,);
in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
out.println("test out");
String message = in.readLine();
example code for sending binary file
BufferedOutputStream out;
BufferedInputStream in;
byte[] buffer = new byte[];
int length = 0;
out = new BufferedOutputStream(new FileOutputStream("test.pdf));
in = new BufferedInputStream(new FileOutputStream("replacement.pdf"));
while((length = in.read(buffer)) > 0 ){
out.write(buffer, 0, length);
}
I don't think using two threads would be necessary in your case. Simply use the socket's InputStream and OutputStream in order to send binary data after you have sent your text messages.
Server Code
OutputStream stream = socket.getOutputStream();
PrintWriter out = new PrintWriter(
new BufferedWriter(
new OutputStreamWriter(stream)
)
);
out.println("test output");
out.flush(); // ensure that the string is not buffered by the BufferedWriter
byte[] data = getBinaryDataSomehow();
stream.write(data);
Client Code
InputStream stream = socket.getInputStream();
String message = readLineFrom(stream);
int dataSize = getSizeOfBinaryDataSomehow();
int totalBytesRead = 0;
byte[] data = new byte[dataSize];
while (totalBytesRead < dataSize) {
int bytesRemaining = dataSize - totalBytesRead;
int bytesRead = stream.read(data, totalBytesRead, bytesRemaining);
if (bytesRead == -1) {
return; // socket has been closed
}
totalBytesRead += bytesRead;
}
In order to determine the correct dataSize on the client side you have to transmit the size of the binary block somehow. You could send it as a String right before out.flush() in the Server Code or make it part of your binary data. In the latter case the first four or eight bytes could hold the actual length of the binary data in bytes.
Hope this helps.
Edit
As #EJP correctly pointed out, using a BufferedReader on the client side will probably result in corrupted or missing binary data because the BufferedReader "steals" some bytes from the binary data to fill its buffer. Instead you should read the string data yourself and either look for a delimiter or have the length of the string data transmitted by some other means.
/* Reads all bytes from the specified stream until it finds a line feed character (\n).
* For simplicity's sake I'm reading one character at a time.
* It might be better to use a PushbackInputStream, read more bytes at
* once, and push the surplus bytes back into the stream...
*/
private static String readLineFrom(InputStream stream) throws IOException {
InputStreamReader reader = new InputStreamReader(stream);
StringBuffer buffer = new StringBuffer();
for (int character = reader.read(); character != -1; character = reader.read()) {
if (character == '\n')
break;
buffer.append((char)character);
}
return buffer.toString();
}
You can read about how HTTP protocol works which essentially sends 'ascii and human readable' headers (so to speak) and after that any content can be added with appropriate encoding like base64 for example. You may create sth similar yourself.
You need to first send the String, then the size of the byte array then the byte array, use String.startsWith() method to check what is being send.
my assignment includes sending an image file using UDP service (using java I implemented that successfully). My professor asked to include:
"The exchanged data messages must also have a header part for the sender to include 16-bit message sequence number for duplicate filtering at the receiver end"
How to do this?
I assume to create your UDP packet, you are using a ByteArrayOutputStream to generate the data. If that is the case, just Wrap a DataOutputStream on top of that ByteArrayOutputStream, and call writeInt(somesequenceNumber) before writing the image data to the stream.
on the receive side, do the opposite, wrap a DataInputStream around a ByteArrayInputStream, and call readInt() to get the sequence number. From there you can check whether you have already received this packet.
Something like
Write Side
ByteArrayOutputStream baos = new ByteArrayOutputStream();
DataOutputStream dos = new DataOutputStream(baos);
dos.writeInt(sequenceNumber++);
dos.writeInt(imageDataLength);
dos.write(imageData);
dos.flush();
byte[] udpPacketBytes = baos.toByteArray();
Read Side
ByteArrayInputStream bais = new ByteArrayInputStream(udpPacketBytes);
DataInputStream dis = new DataInputStream(bais);
int sequenceNumber = dis.readInt();
if (seenSequenceNumbers.add(Integer.valueOf(sequenceNumber)))
{
int imageLength = dis.readInt();
byte[] imageData = new byte[imageLength];
dis.read(imageData);
}
where seenSequenceNumbers is some Set
For a 16-bit value I would use DataOutputStream.writeShort() and DataInputSTream readShort()/readUnsignedShort(). writeInt() and readInt() are for 32-bit values. If you want to avoid duplicates, a 32-bit value may be a better choice in any case. ;)