We Keep Coding

processing bytes received from UDP port in java

I am trying to read data from UDP port on localhost using Java. I'm pretty good with Java, but I can't solve this for quite a while now...
The thing is, after I connect using DatagramSocket and receive a packet with DatagramPacket, I get some bytes that have no sence, I can't see connection with the data I expect. Printout looks like this:
$őZAŇ"¤E€^ĽxΕ’M#ŢúCîS5;Ń8†8Ŕ$5»ôxŕ¸Ţf+?’Ť;Ů%>ż?>żA€ĹĽ‘_
so, I'm obviously handlig something in the wrong way. I've also read some signed/unsigned data problems with Java.
About a year ago I've created a similar app using C#, everything went pretty smooth.
Really hope someone can help.
Here is the code (one of the versions, I've tried a lot of different solutions)
DatagramSocket mySocket = new DatagramSocket(null);
InetSocketAddress addr = new InetSocketAddress("127.0.0.1", 20777);
mySocket.bind(addr);
byte[] receiveData = new byte[152];
while(true)
{
DatagramPacket receivePacket = new DatagramPacket(receiveData, 0, receiveData.length);
mySocket.receive(receivePacket);
byte[] barray = receivePacket.getData();
ByteArrayInputStream inputStream = new ByteArrayInputStream(barray);
DataInputStream dInputStream = new DataInputStream(inputStream);
float a = dInputStream.readFloat();
System.out.println(a);
}

Using this method you can convert a byte array to hexadecimal string representation.
private String bytesToHex(byte[] bytes) {
char[] hexArray = "0123456789ABCDEF".toCharArray();
char[] hexChars = new char[bytes.length * 2];
for ( int j = 0; j < bytes.length; j++ ) {
int v = bytes[j] & 0xFF;
hexChars[j * 2] = hexArray[v >>> 4];
hexChars[j * 2 + 1] = hexArray[v & 0x0F];
}
return new String(hexChars);
}
Hope it helps.

I won't flag your question as a duplicate because it is your first one, but I think you should refer to this other exchange. A very elegant and clear solution to your problem is available.
By the way, a citation of the code reading the section you printed would have been welcome. Good luck...

You need:
A specification of the packet format you are receiving.
A DataInputStreamwrapped around a ByteArrayInputStream wrapped around the byte array you used to build the DatagramPacket, not forgetting to use the constructor that takes an offset and length, which you get from the DatagramPacket.
Code that calls the appropriate DataInputStream methods corresponding to (1).
At the moment you don't even appear to have (1). Without that, you haven't got a hope. Just trying to 'make sense' of binary data, especially by just printing it, is a complete waste of your time.
EDIT If, as per your comment, all the fields are floats, just loop over the datagram calling DataInputStream.readFloat() until it throws EOFException:
try
{
while (true)
{
float f = dataInputStream.readFloat();
System.out.println(f);
}
}
catch (EOFException exc)
{
// expected
}
If that doesn't work (i.e produce recognizable value), you will have to switch to DatagramSocketChannel and ByteBuffer and experiment with the different byte-order possibilites.
Why you were trying to print floating-point data as though it was text remains a mystery.

How to read (all available) data from serial connection when using JSSC?

I'm trying to work with JSSC.
I built my app according to this link:
https://code.google.com/p/java-simple-serial-connector/wiki/jSSC_examples
My event handler looks like:
static class SerialPortReader implements SerialPortEventListener {
public void serialEvent(SerialPortEvent event) {
if(event.isRXCHAR()){//If data is available
try {
byte buffer[] = serialPort.readBytes();
}
catch (SerialPortException ex) {
System.out.println(ex);
}
}
}
}
}
The problem is that I'm always not getting the incoming data in one piece. (I the message has a length of 100 bytes, Im getting 48 and 52 bytes in 2 separates calls)
- The other side send me messages in different lengths.
- In the ICD Im working with, there is a field which tell us the length of the message. (from byte #10 to byte #13)
- I cant read 14 bytes:
(serialPort.readBytes(14);,
parse the message length and read the rest of the message:
(serialPort.readBytes(messageLength-14);
But if I will do it, I will not have the message in once piece (I will have 2 separates byte[] and I need it in one piece (byte[]) without the work of copy function.
Is it possible ?
When working with Ethernet (SocketChannel) we can read data using ByteBuffer. But with JSSC we cant.
Is there a good alternative to JSSC ?
Thanks

You can't rely on any library to give you all the content you need at once because :
the library dont know how many data you need
the library will give you data as it comes and also depending on buffers, hardware, etc
You must develop your own business logic to handle your packets reception. It will of course depend on how your packets are defined : are they always the same length, are they separated with same ending character, etc.
Here is an example that should work with your system (note you should take this as a start, not a full solution, it doesn't include timeout for example) :
static class SerialPortReader implements SerialPortEventListener
{
private int m_nReceptionPosition = 0;
private boolean m_bReceptionActive = false;
private byte[] m_aReceptionBuffer = new byte[2048];
#Override
public void serialEvent(SerialPortEvent p_oEvent)
{
byte[] aReceiveBuffer = new byte[2048];
int nLength = 0;
int nByte = 0;
switch(p_oEvent.getEventType())
{
case SerialPortEvent.RXCHAR:
try
{
aReceiveBuffer = serialPort.readBytes();
for(nByte = 0;nByte < aReceiveBuffer.length;nByte++)
{
//System.out.print(String.format("%02X ",aReceiveBuffer[nByte]));
m_aReceptionBuffer[m_nReceptionPosition] = aReceiveBuffer[nByte];
// Buffer overflow protection
if(m_nReceptionPosition >= 2047)
{
// Reset for next packet
m_bReceptionActive = false;
m_nReceptionPosition = 0;
}
else if(m_bReceptionActive)
{
m_nReceptionPosition++;
// Receive at least the start of the packet including the length
if(m_nReceptionPosition >= 14)
{
nLength = (short)((short)m_aReceptionBuffer[10] & 0x000000FF);
nLength |= ((short)m_aReceptionBuffer[11] << 8) & 0x0000FF00;
nLength |= ((short)m_aReceptionBuffer[12] << 16) & 0x00FF0000;
nLength |= ((short)m_aReceptionBuffer[13] << 24) & 0xFF000000;
//nLength += ..; // Depending if the length in the packet include ALL bytes from the packet or only the content part
if(m_nReceptionPosition >= nLength)
{
// You received at least all the content
// Reset for next packet
m_bReceptionActive = false;
m_nReceptionPosition = 0;
}
}
}
// Start receiving only if this is a Start Of Header
else if(m_aReceptionBuffer[0] == '\0')
{
m_bReceptionActive = true;
m_nReceptionPosition = 1;
}
}
}
catch(Exception e)
{
e.printStackTrace();
}
break;
default:
break;
}
}
}

After writing data to serial port it need to be flushed. Check the timing and pay attention to the fact that read should occur only after other end has written. read size is just an indication to read system call and is not guaranteed. The data may have arrived and is buffered in serial port hardware buffer but may not have been transferred to operating system buffer hence not to application. Consider using scm library, it flushes data after each write http://www.embeddedunveiled.com/

Try this:
Write your data to the serial port (using serialPort.writeBytes()) and if you are expecting a response, use this:
byte[] getData() throws SerialPortException, IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] b;
try {
while ((b = serialPort.readBytes(1, 100)) != null) {
baos.write(b);
// System.out.println ("Wrote: " + b.length + " bytes");
}
// System.out.println("Returning: " + Arrays.toString(baos.toByteArray()));
} catch (SerialPortTimeoutException ex) {
; //don't want to catch it, it just means there is no more data to read
}
return baos.toByteArray();
}
Do what you want with the returned byte array; in my case I just display it for testing.
I found it works just fine if you read one byte at a time, using a 100ms timeout, and when it does time out, you've read all data in the buffer.
Source: trying to talk to an Epson serial printer using jssc and ESC/POS.

How to read raw bytes from socket in Python?

I have an android java app sending bytes over a socket which is connected to a host machine running a server in Python. I need to receive these bytes as they were sent from the python socket. I see that in Python 'socket.recv' only returns a string. When I send an ASCII string from the java app, I am able to receive the data correctly in the python server, but when I send binary data using java byte, I see the data received is not same. I need to receive raw bytes in Python for my protocol to work correctly. Please point me in right direction.
Code snippet for Sending data on socket:
private void sendFrameMessage(byte[] data) {
byte[] lengthInfo = new byte[4];
Log.v(TAG, "sendFrameMessage");
for(int i=0; i<data.length; i++) {
Log.v(TAG, String.format("data[%d] = %d", i, data[i]));
}
try {
lengthInfo[0] = (byte) data.length;
lengthInfo[1] = (byte) (data.length >> 8);
lengthInfo[2] = (byte) (data.length >> 16);
lengthInfo[3] = (byte) (data.length >> 24);
DataOutputStream dos;
dos = new DataOutputStream(mSocket.getOutputStream());
dos.write(lengthInfo, 0, 4);
dos.write(data, 0, data.length);
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
Python Code on receiver side
def recvFrameMessage(self, s):
recv_count = 4;
data = s.recv(recv_count)
if data == 0:
return None
total_rx = len(data)
lenInfo = data
while total_rx < recv_count:
data = s.recv(recv_count - total_rx)
if data == 0:
return None
total_rx += len(data)
lenInfo = lenInfo + data
recv_count = self.decodeFrameLen(lenInfo)
logger.info("length = %d" % recv_count)
data = s.recv(recv_count)
total_rx = len(data)
msg = data
while total_rx < recv_count:
data = s.recv(recv_count - total_rx)
if data == 0:
return None
total_rx += len(data)
msg = msg + data
logger.info("msg = " + msg)
for i in range(0, len(msg)-1):
logger.info("msg[%d] = %s" % (i, msg[i]))
return msg

#SteveP makes good points for binary data "with some structure", but if this is a plain stream of bytes, in Python 2 simply apply the ord() function to each "character" you get from the socket. For example, if the Java end sends a NUL byte, that will show up on the Python end as the character "\x00", and then:
>>> ord("\x00")
0
To convert a whole string s,
map(ord, s)
returns a list of the corresponding 8-bit unsigned integers.
I'm assuming Python 2 here.

Reading binary data is perfectly doable, but what if the binary representation from your android app is different than the byte representation on the Python server? From the Python documentation:
It is perfectly possible to send binary data over a socket. The major
problem is that not all machines use the same formats for binary data.
For example, a Motorola chip will represent a 16 bit integer with the
value 1 as the two hex bytes 00 01. Intel and DEC, however, are
byte-reversed - that same 1 is 01 00. Socket libraries have calls for
converting 16 and 32 bit integers - ntohl, htonl, ntohs, htons where
“n” means network and “h” means host, “s” means short and “l” means
long. Where network order is host order, these do nothing, but where
the machine is byte-reversed, these swap the bytes around
appropriately.
Without code and example input/output, this question is going to be really difficult to answer. I assume the issue is that the representation is different. The most likely issue is that Java uses big endian, whereas Python adheres to whatever machine you are running it off of. If your server uses little endian, then you need to account for that. See here for a more thorough explanation on endianness.

Efficiently send large int[] over sockets in Java

I am working on a Java application where I need to send an array of 500,000 integers from one Android phone to another Android phone over a socket connection as quickly as possible. The main bottleneck seems to be converting the integers so the socket can take them, whether I use ObjectOutputStreams, ByteBuffers, or a low level mask-and-shift conversion. What is the fastest way to send an int[] over a socket from one Java app to another?
Here is the code for everything I've tried so far, with benchmarks on the LG Optimus V I'm testing on (600 MHz ARM processor, Android 2.2).
Low level mask-and-shift: 0.2 seconds
public static byte[] intToByte(int[] input)
{
byte[] output = new byte[input.length*4];
for(int i = 0; i < input.length; i++) {
output[i*4] = (byte)(input[i] & 0xFF);
output[i*4 + 1] = (byte)((input[i] & 0xFF00) >>> 8);
output[i*4 + 2] = (byte)((input[i] & 0xFF0000) >>> 16);
output[i*4 + 3] = (byte)((input[i] & 0xFF000000) >>> 24);
}
return output;
}
Using ByteBuffer and IntBuffer: 0.75 seconds
public static byte[] intToByte(int[] input)
{
ByteBuffer byteBuffer = ByteBuffer.allocate(input.length * 4);
IntBuffer intBuffer = byteBuffer.asIntBuffer();
intBuffer.put(input);
byte[] array = byteBuffer.array();
return array;
}
ObjectOutputStream: 3.1 seconds (I tried variations of this using DataOutPutStream and writeInt() instead of writeObject(), but it didn't make much of a difference)
public static void sendSerialDataTCP(String address, int[] array) throws IOException
{
Socket senderSocket = new Socket(address, 4446);
OutputStream os = senderSocket.getOutputStream();
BufferedOutputStream bos = new BufferedOutputStream (os);
ObjectOutputStream oos = new ObjectOutputStream(bos);
oos.writeObject(array);
oos.flush();
bos.flush();
os.flush();
oos.close();
os.close();
bos.close();
senderSocket.close();
}
Lastly, the code I used to send byte[]: takes an addition 0.2 seconds over the intToByte() functions
public static void sendDataTCP(String address, byte[] data) throws IOException
{
Socket senderSocket = new Socket(address, 4446);
OutputStream os = senderSocket.getOutputStream();
os.write(data, 0, data.length);
os.flush();
senderSocket.close();
}
I'm writing the code on both sides of the socket so I can try any kind of endianness, compression, serialization, etc. There's got to be a way to do this conversion more efficiently in Java. Please help!

As I noted in a comment, I think you're banging against the limits of your processor. As this might be helpful to others, I'll break it down. Here's your loop to convert integers to bytes:
for(int i = 0; i < input.length; i++) {
output[i*4] = (byte)(input[i] & 0xFF);
output[i*4 + 1] = (byte)((input[i] & 0xFF00) >>> 8);
output[i*4 + 2] = (byte)((input[i] & 0xFF0000) >>> 16);
output[i*4 + 3] = (byte)((input[i] & 0xFF000000) >>> 24);
}
This loop executes 500,000 times. You 600Mhz processor can process roughly 600,000,000 operations per second. So each iteration of the loop will consume roughly 1/1200 of a second for every operation.
Again, using very rough numbers (I don't know the ARM instruction set, so there may be more or less per action), here's an operation count:
Test/branch: 5 (retrieve counter, retrieve array length, compare, branch, increment counter)
Mask and shift: 10 x 4 (retrieve counter, retrieve input array base, add, retrieve mask, and, shift, multiply counter, add offset, add to output base, store)
OK, so in rough numbers, this loop takes at best 55/1200 of a second, or 0.04 seconds. However, you're not dealing with best case scenario. For one thing, with an array this large you're not going to benefit from a processor cache, so you'll introduce wait states into every array store and load.
Plus, the basic operations that I described may or may not translate directly into machine code. If not (and I suspect not), the loop will cost more than I've described.
Finally, if you're really unlucky, the JVM hasn't JIT-ed your code, so for some portion (or all) of the loop it's interpreting bytecode rather than executing native instructions. I don't know enough about Dalvik to comment on that.

Java was IMO never intended to be able efficiently reinterpret a memory region from int[] to byte[] like you could do in C. It doesn't even have such a memory address model.
You either need to go native to send the data or you can try to find some micro optimizations. But I doubt you will gain a lot.
E.g. this could be slightly faster than your version (if it works at all)
public static byte[] intToByte(int[] input)
{
byte[] output = new byte[input.length*4];
for(int i = 0; i < input.length; i++) {
int position = i << 2;
output[position | 0] = (byte)((input[i] >> 0) & 0xFF);
output[position | 1] = (byte)((input[i] >> 8) & 0xFF);
output[position | 2] = (byte)((input[i] >> 16) & 0xFF);
output[position | 3] = (byte)((input[i] >> 24) & 0xFF);
}
return output;
}

I would do it like this:
Socket senderSocket = new Socket(address, 4446);
OutputStream os = senderSocket.getOutputStream();
BufferedOutputStream bos = new BufferedOutputStream(os);
DataOutputStream dos = new DataOutputStream(bos);
dos.writeInt(array.length);
for(int i : array) dos.writeInt(i);
dos.close();
On the other side, read it like:
Socket recieverSocket = ...;
InputStream is = recieverSocket.getInputStream();
BufferedInputStream bis = new BufferedInputStream(is);
DataInputStream dis = new DataInputStream(bis);
int length = dis.readInt();
int[] array = new int[length];
for(int i = 0; i < length; i++) array[i] = dis.readInt();
dis.close();

If you're not adverse to using a library, you might want to check out Protocol Buffers from Google. It's built for much more complex object serialization, but I'd bet that they worked hard to figure out how to quickly serialize an array of integers in Java.
EDIT: I looked in the Protobuf source code, and it uses something similar to your low-level mask and shift.

Socket communication between Java and C++

I'm trying to have a connection between a Java server and a C++ client. But when I read the data in my client I always have the same strange character (’). I tried to change the encoding in both side but nothing work.
Here is my Java code :
public class Serveur
{
public static void main(String[] args) throws Exception
{
final int PORT = 13370;
try
{
ServerSocket service= new ServerSocket(PORT);
Socket connection = service.accept();
PrintWriter pw = new PrintWriter(connection.getOutputStream());
String s = Integer.toString(5);
while(true)
{
pw.print(s.getBytes("UTF-8"));
pw.flush();
pw.close();
}
connection.close();
}
}
I also tried to use an OutputStream, a DataOutputStream and a BufferedOutputStream.
And here is the C++ code :
int main(int argc, char* argv[])
{
WSADATA WSAData;
WSAStartup(MAKEWORD(2,0), &WSAData);
SOCKET sock;
SOCKADDR_IN sin;
char buffer[512];
sin.sin_addr.s_addr = inet_addr("127.0.0.1");
sin.sin_family = AF_INET;
sin.sin_port = htons(13370);
sock = socket(AF_INET,SOCK_STREAM,0);
if(connect(sock, (SOCKADDR*)&sin, sizeof(sin)) != SOCKET_ERROR)
{
cout<<"connection"<<endl;
if(recv(sock, buffer, sizeof(buffer), 0) != SOCKET_ERROR)
{
string s = buffer;
wchar_t *pwchello = L"Hi";
wchar_t *pwc = (wchar_t *)malloc( sizeof( wchar_t ));
char *pmbhello = buffer;
int i = mbstowcs(pwc,pmbhello, MB_CUR_MAX);
cout << i << endl;
cout<<"cout : "<<pwc<<endl;
cout <<buffer<<endl;
printf("printf : %s\n", buffer);
cout << "wsagetlasterror() : "<<WSAGetLastError();
closesocket(sock);
WSACleanup();
free(m_pBuffer);
}
return 0;
}
As you can see, I tried different solution but without success.
Thanks in advance, and sorry for my english it may be not very good

You are mixing up lots of different encoding conversions and I/O strategies. You should try out the following simplified version:
if(connect(sock, (SOCKADDR*)&sin, sizeof(sin)) != SOCKET_ERROR)
{
cout << "connection" << endl;
// the result of 'recv()' is either SOCKET_ERROR or
// the number of bytes received. don't though away
// the return value.
const int result = recv(sock, buffer, sizeof(buffer), 0);
if(result != SOCKET_ERROR)
{
// use length (in bytes) returned by 'recv()'
// since buffer is not null terminated.
string s(buffer,result);
// 's' is in UTF-8 no converstion to wide strings
// should be necessary.
cout << "message: '" << s << "'." << endl;
}
closesocket(sock);
}
WSACleanup();
However, note that the standard output is in the current code page and usually UTF-8 is not the default code page. Outputing Unicode data to the console in windows requires a few other library calls to configure.

recv does not turn its destination buffer into null-terminated string. It fills in a number of bytes in the buffer, but does not append a 0.
You need top do this (with error checking, of course):
ssize_t bytesRead = recv(buffer, ...);
string str(buffer, bytesRead);
Also, be aware that recv does not guarantee that something sent in one call gets received in one call (unless you're doing UDP).

You're only allocating room for a single wchar_t here:
wchar_t *pwc = (wchar_t *)malloc( sizeof( wchar_t ));
You also assign buffer to string s, but never seem to use s

I have been having the same problem since last night. Finally figured out that encoding is not recognized by my server (written in C). Therefore, I changed in my client
someOutputStream.writeUTF(someSillyString);
to
someOutputStream.write(someSillyString.getBytes());
This way, I did not even need to typecast on the server side.