We Keep Coding

Setting a Java Byte Array Size from Packet Header

A piece of Java code is residing on a server expecting about 64 bytes of information from a piece of hardware, sent via TCP. The packet has a 10 byte header. The first byte is a protocol identifier, the second two bytes gives the total number of bytes in the packet, including all the header bytes and checksum. The last 7 bytes are a UID.
Server Code:
public void run () throws Exception
{
//Open a socket on localhost at port 11111
ServerSocket welcomeSocket = new ServerSocket(11111);
while(true) {
//Open and Accept on Socket
Socket connectionSocket = welcomeSocket.accept();
//Alt Method
DataInputStream dis = new DataInputStream(connectionSocket.getInputStream());
int len = dis.readInt();
byte[] data = new byte[len];
if (len > 0) {
dis.readFully(data);
}
System.out.println("Recv[HEX]: " + StringTools.toHexString(data));
}
}
The issue is my readInt() line, that takes the first four bytes, however I need to determine the length based on the second two bytes. How can this be achieved?
And secondly, is my stringTools.toHexString(data) correct to dump the received buffer which I know should be readable as a HEX string?
Note: This question has its root here: Java TCP Socket Byte Heap Memory Issue

Only use DataInputStream if the other side is using DataOutputStream or it's exact format. The integers, for example, may be encoded big-endian or little-endian - DataOutputStream uses big-endian notation, if the other side uses different encoding, you cannot use DataInputStream. Using InputStream.read() gives you more control if you need it.
Now, since the format of message as you stated starts with one byte for protocol identifier, you first need to read that as a byte (dis.readByte() or InputStream.read()) and either check that the protocol is what you expect or handle different protocols. Then you read the message length, etc.

You can use ByteBuffer to read the int in the last two bytes
import static java.lang.System.out;
import java.nio.ByteBuffer;
class B {
public static void main( String ... args ) {
// test value
int a = 1238098;
// convert it into an arrays of bytes
ByteBuffer b = ByteBuffer.allocate(4);
b.putInt(a);
byte [] r = b.array();
// read last two
int size = ByteBuffer.wrap(new byte[]{0x0,0x0, r[2], r[3]}).getInt();
// print it
out.println("Original: " + String.format("%32s%n" , Integer.toString(a,2)).replace(' ', '0'));
out.printf("Last two: %32s%n" , Integer.toString(size,2));
out.printf("Decimal : %d%n" , size );
}
}
Output:
Original: 00000000000100101110010001010010
Last two: 1110010001010010
Decimal : 58450
However I would recommend to follow #Jiri answer about read using InputStream.read() instead of DateInputStream

processing bytes received from UDP port in java

I am trying to read data from UDP port on localhost using Java. I'm pretty good with Java, but I can't solve this for quite a while now...
The thing is, after I connect using DatagramSocket and receive a packet with DatagramPacket, I get some bytes that have no sence, I can't see connection with the data I expect. Printout looks like this:
$őZAŇ"¤E€^ĽxΕ’M#ŢúCîS5;Ń8†8Ŕ$5»ôxŕ¸Ţf+?’Ť;Ů%>ż?>żA€ĹĽ‘_
so, I'm obviously handlig something in the wrong way. I've also read some signed/unsigned data problems with Java.
About a year ago I've created a similar app using C#, everything went pretty smooth.
Really hope someone can help.
Here is the code (one of the versions, I've tried a lot of different solutions)
DatagramSocket mySocket = new DatagramSocket(null);
InetSocketAddress addr = new InetSocketAddress("127.0.0.1", 20777);
mySocket.bind(addr);
byte[] receiveData = new byte[152];
while(true)
{
DatagramPacket receivePacket = new DatagramPacket(receiveData, 0, receiveData.length);
mySocket.receive(receivePacket);
byte[] barray = receivePacket.getData();
ByteArrayInputStream inputStream = new ByteArrayInputStream(barray);
DataInputStream dInputStream = new DataInputStream(inputStream);
float a = dInputStream.readFloat();
System.out.println(a);
}

Using this method you can convert a byte array to hexadecimal string representation.
private String bytesToHex(byte[] bytes) {
char[] hexArray = "0123456789ABCDEF".toCharArray();
char[] hexChars = new char[bytes.length * 2];
for ( int j = 0; j < bytes.length; j++ ) {
int v = bytes[j] & 0xFF;
hexChars[j * 2] = hexArray[v >>> 4];
hexChars[j * 2 + 1] = hexArray[v & 0x0F];
}
return new String(hexChars);
}
Hope it helps.

I won't flag your question as a duplicate because it is your first one, but I think you should refer to this other exchange. A very elegant and clear solution to your problem is available.
By the way, a citation of the code reading the section you printed would have been welcome. Good luck...

You need:
A specification of the packet format you are receiving.
A DataInputStreamwrapped around a ByteArrayInputStream wrapped around the byte array you used to build the DatagramPacket, not forgetting to use the constructor that takes an offset and length, which you get from the DatagramPacket.
Code that calls the appropriate DataInputStream methods corresponding to (1).
At the moment you don't even appear to have (1). Without that, you haven't got a hope. Just trying to 'make sense' of binary data, especially by just printing it, is a complete waste of your time.
EDIT If, as per your comment, all the fields are floats, just loop over the datagram calling DataInputStream.readFloat() until it throws EOFException:
try
{
while (true)
{
float f = dataInputStream.readFloat();
System.out.println(f);
}
}
catch (EOFException exc)
{
// expected
}
If that doesn't work (i.e produce recognizable value), you will have to switch to DatagramSocketChannel and ByteBuffer and experiment with the different byte-order possibilites.
Why you were trying to print floating-point data as though it was text remains a mystery.

How to send java byte[] on tcp socket to c++ char[] on server?

I want to send a byte[] array from a java client to a server that receives the data in C++. The byte array contains characters and integers that are converted to bytes (its a wave header). The server doesn't receive the values correctly. How can I send the byte[] so that the server socket can write it to a char[]? I am using the following code:
Client.java:
//Some example values in byte[]
byte[] bA = new byte[44];
bA[0]='R';
...
bA[4]=(byte)(2048 & 0xff);
...
bA[16] = 16;
....
//Write byte[] on socket
DataOutputStream out = new DataOutputStream(socket.getOutputStream());
out.write(bA,0,44);
Server.cpp
int k = 0,n = 0;
char buffer[100];
ofstream wav("out.wav", ios::out | ios::binary);
while(k<44){//receive 44 values
memset(buffer ,0 , 100);
n = recv(sock , buffer , 100 , 0);
k += n;
buffer[99] = '\0';
wav.write(buffer,n);
}

One issue I see is if you receive 100 characters, you're corrupting the data with this line:
buffer[99] = '\0';
If there is a character other than NULL at that position, you've corrupted the data. Since the data is binary, there is no need to null terminate the buffer. Remove that line from your loop.
Instead, rely on the return value of recv to determine the number of characters to copy to the stream. Which brings up another point -- you're not checking if recv returns an error.

How to read raw bytes from socket in Python?

I have an android java app sending bytes over a socket which is connected to a host machine running a server in Python. I need to receive these bytes as they were sent from the python socket. I see that in Python 'socket.recv' only returns a string. When I send an ASCII string from the java app, I am able to receive the data correctly in the python server, but when I send binary data using java byte, I see the data received is not same. I need to receive raw bytes in Python for my protocol to work correctly. Please point me in right direction.
Code snippet for Sending data on socket:
private void sendFrameMessage(byte[] data) {
byte[] lengthInfo = new byte[4];
Log.v(TAG, "sendFrameMessage");
for(int i=0; i<data.length; i++) {
Log.v(TAG, String.format("data[%d] = %d", i, data[i]));
}
try {
lengthInfo[0] = (byte) data.length;
lengthInfo[1] = (byte) (data.length >> 8);
lengthInfo[2] = (byte) (data.length >> 16);
lengthInfo[3] = (byte) (data.length >> 24);
DataOutputStream dos;
dos = new DataOutputStream(mSocket.getOutputStream());
dos.write(lengthInfo, 0, 4);
dos.write(data, 0, data.length);
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
Python Code on receiver side
def recvFrameMessage(self, s):
recv_count = 4;
data = s.recv(recv_count)
if data == 0:
return None
total_rx = len(data)
lenInfo = data
while total_rx < recv_count:
data = s.recv(recv_count - total_rx)
if data == 0:
return None
total_rx += len(data)
lenInfo = lenInfo + data
recv_count = self.decodeFrameLen(lenInfo)
logger.info("length = %d" % recv_count)
data = s.recv(recv_count)
total_rx = len(data)
msg = data
while total_rx < recv_count:
data = s.recv(recv_count - total_rx)
if data == 0:
return None
total_rx += len(data)
msg = msg + data
logger.info("msg = " + msg)
for i in range(0, len(msg)-1):
logger.info("msg[%d] = %s" % (i, msg[i]))
return msg

#SteveP makes good points for binary data "with some structure", but if this is a plain stream of bytes, in Python 2 simply apply the ord() function to each "character" you get from the socket. For example, if the Java end sends a NUL byte, that will show up on the Python end as the character "\x00", and then:
>>> ord("\x00")
0
To convert a whole string s,
map(ord, s)
returns a list of the corresponding 8-bit unsigned integers.
I'm assuming Python 2 here.

Reading binary data is perfectly doable, but what if the binary representation from your android app is different than the byte representation on the Python server? From the Python documentation:
It is perfectly possible to send binary data over a socket. The major
problem is that not all machines use the same formats for binary data.
For example, a Motorola chip will represent a 16 bit integer with the
value 1 as the two hex bytes 00 01. Intel and DEC, however, are
byte-reversed - that same 1 is 01 00. Socket libraries have calls for
converting 16 and 32 bit integers - ntohl, htonl, ntohs, htons where
“n” means network and “h” means host, “s” means short and “l” means
long. Where network order is host order, these do nothing, but where
the machine is byte-reversed, these swap the bytes around
appropriately.
Without code and example input/output, this question is going to be really difficult to answer. I assume the issue is that the representation is different. The most likely issue is that Java uses big endian, whereas Python adheres to whatever machine you are running it off of. If your server uses little endian, then you need to account for that. See here for a more thorough explanation on endianness.

Efficiently send large int[] over sockets in Java

I am working on a Java application where I need to send an array of 500,000 integers from one Android phone to another Android phone over a socket connection as quickly as possible. The main bottleneck seems to be converting the integers so the socket can take them, whether I use ObjectOutputStreams, ByteBuffers, or a low level mask-and-shift conversion. What is the fastest way to send an int[] over a socket from one Java app to another?
Here is the code for everything I've tried so far, with benchmarks on the LG Optimus V I'm testing on (600 MHz ARM processor, Android 2.2).
Low level mask-and-shift: 0.2 seconds
public static byte[] intToByte(int[] input)
{
byte[] output = new byte[input.length*4];
for(int i = 0; i < input.length; i++) {
output[i*4] = (byte)(input[i] & 0xFF);
output[i*4 + 1] = (byte)((input[i] & 0xFF00) >>> 8);
output[i*4 + 2] = (byte)((input[i] & 0xFF0000) >>> 16);
output[i*4 + 3] = (byte)((input[i] & 0xFF000000) >>> 24);
}
return output;
}
Using ByteBuffer and IntBuffer: 0.75 seconds
public static byte[] intToByte(int[] input)
{
ByteBuffer byteBuffer = ByteBuffer.allocate(input.length * 4);
IntBuffer intBuffer = byteBuffer.asIntBuffer();
intBuffer.put(input);
byte[] array = byteBuffer.array();
return array;
}
ObjectOutputStream: 3.1 seconds (I tried variations of this using DataOutPutStream and writeInt() instead of writeObject(), but it didn't make much of a difference)
public static void sendSerialDataTCP(String address, int[] array) throws IOException
{
Socket senderSocket = new Socket(address, 4446);
OutputStream os = senderSocket.getOutputStream();
BufferedOutputStream bos = new BufferedOutputStream (os);
ObjectOutputStream oos = new ObjectOutputStream(bos);
oos.writeObject(array);
oos.flush();
bos.flush();
os.flush();
oos.close();
os.close();
bos.close();
senderSocket.close();
}
Lastly, the code I used to send byte[]: takes an addition 0.2 seconds over the intToByte() functions
public static void sendDataTCP(String address, byte[] data) throws IOException
{
Socket senderSocket = new Socket(address, 4446);
OutputStream os = senderSocket.getOutputStream();
os.write(data, 0, data.length);
os.flush();
senderSocket.close();
}
I'm writing the code on both sides of the socket so I can try any kind of endianness, compression, serialization, etc. There's got to be a way to do this conversion more efficiently in Java. Please help!

As I noted in a comment, I think you're banging against the limits of your processor. As this might be helpful to others, I'll break it down. Here's your loop to convert integers to bytes:
for(int i = 0; i < input.length; i++) {
output[i*4] = (byte)(input[i] & 0xFF);
output[i*4 + 1] = (byte)((input[i] & 0xFF00) >>> 8);
output[i*4 + 2] = (byte)((input[i] & 0xFF0000) >>> 16);
output[i*4 + 3] = (byte)((input[i] & 0xFF000000) >>> 24);
}
This loop executes 500,000 times. You 600Mhz processor can process roughly 600,000,000 operations per second. So each iteration of the loop will consume roughly 1/1200 of a second for every operation.
Again, using very rough numbers (I don't know the ARM instruction set, so there may be more or less per action), here's an operation count:
Test/branch: 5 (retrieve counter, retrieve array length, compare, branch, increment counter)
Mask and shift: 10 x 4 (retrieve counter, retrieve input array base, add, retrieve mask, and, shift, multiply counter, add offset, add to output base, store)
OK, so in rough numbers, this loop takes at best 55/1200 of a second, or 0.04 seconds. However, you're not dealing with best case scenario. For one thing, with an array this large you're not going to benefit from a processor cache, so you'll introduce wait states into every array store and load.
Plus, the basic operations that I described may or may not translate directly into machine code. If not (and I suspect not), the loop will cost more than I've described.
Finally, if you're really unlucky, the JVM hasn't JIT-ed your code, so for some portion (or all) of the loop it's interpreting bytecode rather than executing native instructions. I don't know enough about Dalvik to comment on that.

Java was IMO never intended to be able efficiently reinterpret a memory region from int[] to byte[] like you could do in C. It doesn't even have such a memory address model.
You either need to go native to send the data or you can try to find some micro optimizations. But I doubt you will gain a lot.
E.g. this could be slightly faster than your version (if it works at all)
public static byte[] intToByte(int[] input)
{
byte[] output = new byte[input.length*4];
for(int i = 0; i < input.length; i++) {
int position = i << 2;
output[position | 0] = (byte)((input[i] >> 0) & 0xFF);
output[position | 1] = (byte)((input[i] >> 8) & 0xFF);
output[position | 2] = (byte)((input[i] >> 16) & 0xFF);
output[position | 3] = (byte)((input[i] >> 24) & 0xFF);
}
return output;
}

I would do it like this:
Socket senderSocket = new Socket(address, 4446);
OutputStream os = senderSocket.getOutputStream();
BufferedOutputStream bos = new BufferedOutputStream(os);
DataOutputStream dos = new DataOutputStream(bos);
dos.writeInt(array.length);
for(int i : array) dos.writeInt(i);
dos.close();
On the other side, read it like:
Socket recieverSocket = ...;
InputStream is = recieverSocket.getInputStream();
BufferedInputStream bis = new BufferedInputStream(is);
DataInputStream dis = new DataInputStream(bis);
int length = dis.readInt();
int[] array = new int[length];
for(int i = 0; i < length; i++) array[i] = dis.readInt();
dis.close();

If you're not adverse to using a library, you might want to check out Protocol Buffers from Google. It's built for much more complex object serialization, but I'd bet that they worked hard to figure out how to quickly serialize an array of integers in Java.
EDIT: I looked in the Protobuf source code, and it uses something similar to your low-level mask and shift.