I have an android java app sending bytes over a socket which is connected to a host machine running a server in Python. I need to receive these bytes as they were sent from the python socket. I see that in Python 'socket.recv' only returns a string. When I send an ASCII string from the java app, I am able to receive the data correctly in the python server, but when I send binary data using java byte, I see the data received is not same. I need to receive raw bytes in Python for my protocol to work correctly. Please point me in right direction.
Code snippet for Sending data on socket:
private void sendFrameMessage(byte[] data) {
byte[] lengthInfo = new byte[4];
Log.v(TAG, "sendFrameMessage");
for(int i=0; i<data.length; i++) {
Log.v(TAG, String.format("data[%d] = %d", i, data[i]));
}
try {
lengthInfo[0] = (byte) data.length;
lengthInfo[1] = (byte) (data.length >> 8);
lengthInfo[2] = (byte) (data.length >> 16);
lengthInfo[3] = (byte) (data.length >> 24);
DataOutputStream dos;
dos = new DataOutputStream(mSocket.getOutputStream());
dos.write(lengthInfo, 0, 4);
dos.write(data, 0, data.length);
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
Python Code on receiver side
def recvFrameMessage(self, s):
recv_count = 4;
data = s.recv(recv_count)
if data == 0:
return None
total_rx = len(data)
lenInfo = data
while total_rx < recv_count:
data = s.recv(recv_count - total_rx)
if data == 0:
return None
total_rx += len(data)
lenInfo = lenInfo + data
recv_count = self.decodeFrameLen(lenInfo)
logger.info("length = %d" % recv_count)
data = s.recv(recv_count)
total_rx = len(data)
msg = data
while total_rx < recv_count:
data = s.recv(recv_count - total_rx)
if data == 0:
return None
total_rx += len(data)
msg = msg + data
logger.info("msg = " + msg)
for i in range(0, len(msg)-1):
logger.info("msg[%d] = %s" % (i, msg[i]))
return msg
#SteveP makes good points for binary data "with some structure", but if this is a plain stream of bytes, in Python 2 simply apply the ord() function to each "character" you get from the socket. For example, if the Java end sends a NUL byte, that will show up on the Python end as the character "\x00", and then:
>>> ord("\x00")
0
To convert a whole string s,
map(ord, s)
returns a list of the corresponding 8-bit unsigned integers.
I'm assuming Python 2 here.
Reading binary data is perfectly doable, but what if the binary representation from your android app is different than the byte representation on the Python server? From the Python documentation:
It is perfectly possible to send binary data over a socket. The major
problem is that not all machines use the same formats for binary data.
For example, a Motorola chip will represent a 16 bit integer with the
value 1 as the two hex bytes 00 01. Intel and DEC, however, are
byte-reversed - that same 1 is 01 00. Socket libraries have calls for
converting 16 and 32 bit integers - ntohl, htonl, ntohs, htons where
“n” means network and “h” means host, “s” means short and “l” means
long. Where network order is host order, these do nothing, but where
the machine is byte-reversed, these swap the bytes around
appropriately.
Without code and example input/output, this question is going to be really difficult to answer. I assume the issue is that the representation is different. The most likely issue is that Java uses big endian, whereas Python adheres to whatever machine you are running it off of. If your server uses little endian, then you need to account for that. See here for a more thorough explanation on endianness.
Related
My app sends small lines of text (commands to the server) or 32 bytes chunks of voice data. Right now I'm just using the socket's OutputStream's write. However, the problem is that Android Java seems to like to send the first byte by itself. Example:
Send: "Call Iron Man"
Received: "C", "all Iron Man"
To work around this splitting I prefix each line with a "throw away" character #. So the previous example is sent as:
Send: "#Call Iron Man"
Received: "#", "Call Iron Man" --> "Call Iron Man" will be used and "#" is ignored.
The problem becomes when I want to send 32 bytes of voice, it is sent as one packet of 1 byte, and then one packet of 31 byte. These 1 byte packets waste a lot of data because of the TCP/IP overhead. According to Sizing Source that means I will use (64+1) + (64+31) = 160 bytes for my 32 byte voice chunk when I could be using 64+32 = 96. That means I will be using 1.67X more LTE data than I should which (in Canada) will cost me a very pretty penny.
Is there a way to force all 32 bytes to be sent as one packet?
Here is the Android code for sending:
int totalRead = 0, dataRead;
while (totalRead < WAVBUFFERSIZE)
{//although unlikely to be necessary, buffer the mic input
dataRead = wavRecorder.read(wavbuffer, totalRead, WAVBUFFERSIZE - totalRead);
totalRead = totalRead + dataRead;
}
int encodeLength = AmrEncoder.encode(AmrEncoder.Mode.MR122.ordinal(), wavbuffer, amrbuffer);
try
{
Vars.mediaSocket.getOutputStream().write(amrbuffer, 0, encodeLength);
}
catch (Exception e)
{
Utils.logcat(Const.LOGE, encTag, "Cannot send amr out the media socket");
}
Here is the C/C++ code for receiving:
mediaRead = 0; //to know how much media is ACTUALLY received. don't always assume MAXMEDIA amount was received
bzero(bufferMedia, MAXMEDIA+1);
alarm(ALARMTIMEOUT);
do
{//wait for the media chunk to come in first before doing something
returnValue = SSL_read(sdssl, bufferMedia, MAXMEDIA-mediaRead);
if(returnValue > 0)
{
mediaRead = mediaRead + returnValue;
}
int sslerr = SSL_get_error(sdssl, returnValue);
switch (sslerr)
{
case SSL_ERROR_NONE:
waiting = false;
eventful = true; //an ssl operation completed this round, something did happen
break;
//other cases when necessary. right now only no error signals a successful read
}
} while(waiting && SSL_pending(sdssl));
alarm(0);
if(alarmKilled)
{
alarmKilled = false;
cout << "Alarm killed SSL read of media socket\n";
}
where MAXMEDIA = 1024 so there is definetly enough place for the 32 bytes of voice data
I am trying to read data from UDP port on localhost using Java. I'm pretty good with Java, but I can't solve this for quite a while now...
The thing is, after I connect using DatagramSocket and receive a packet with DatagramPacket, I get some bytes that have no sence, I can't see connection with the data I expect. Printout looks like this:
$őZAŇ"¤E€^ĽxΕ’M#ŢúCîS5;Ń8†8Ŕ$5»ôxŕ¸Ţf+?’Ť;Ů%>ż?>żA€ĹĽ‘_
so, I'm obviously handlig something in the wrong way. I've also read some signed/unsigned data problems with Java.
About a year ago I've created a similar app using C#, everything went pretty smooth.
Really hope someone can help.
Here is the code (one of the versions, I've tried a lot of different solutions)
DatagramSocket mySocket = new DatagramSocket(null);
InetSocketAddress addr = new InetSocketAddress("127.0.0.1", 20777);
mySocket.bind(addr);
byte[] receiveData = new byte[152];
while(true)
{
DatagramPacket receivePacket = new DatagramPacket(receiveData, 0, receiveData.length);
mySocket.receive(receivePacket);
byte[] barray = receivePacket.getData();
ByteArrayInputStream inputStream = new ByteArrayInputStream(barray);
DataInputStream dInputStream = new DataInputStream(inputStream);
float a = dInputStream.readFloat();
System.out.println(a);
}
Using this method you can convert a byte array to hexadecimal string representation.
private String bytesToHex(byte[] bytes) {
char[] hexArray = "0123456789ABCDEF".toCharArray();
char[] hexChars = new char[bytes.length * 2];
for ( int j = 0; j < bytes.length; j++ ) {
int v = bytes[j] & 0xFF;
hexChars[j * 2] = hexArray[v >>> 4];
hexChars[j * 2 + 1] = hexArray[v & 0x0F];
}
return new String(hexChars);
}
Hope it helps.
I won't flag your question as a duplicate because it is your first one, but I think you should refer to this other exchange. A very elegant and clear solution to your problem is available.
By the way, a citation of the code reading the section you printed would have been welcome. Good luck...
You need:
A specification of the packet format you are receiving.
A DataInputStreamwrapped around a ByteArrayInputStream wrapped around the byte array you used to build the DatagramPacket, not forgetting to use the constructor that takes an offset and length, which you get from the DatagramPacket.
Code that calls the appropriate DataInputStream methods corresponding to (1).
At the moment you don't even appear to have (1). Without that, you haven't got a hope. Just trying to 'make sense' of binary data, especially by just printing it, is a complete waste of your time.
EDIT If, as per your comment, all the fields are floats, just loop over the datagram calling DataInputStream.readFloat() until it throws EOFException:
try
{
while (true)
{
float f = dataInputStream.readFloat();
System.out.println(f);
}
}
catch (EOFException exc)
{
// expected
}
If that doesn't work (i.e produce recognizable value), you will have to switch to DatagramSocketChannel and ByteBuffer and experiment with the different byte-order possibilites.
Why you were trying to print floating-point data as though it was text remains a mystery.
I'm trying to work with JSSC.
I built my app according to this link:
https://code.google.com/p/java-simple-serial-connector/wiki/jSSC_examples
My event handler looks like:
static class SerialPortReader implements SerialPortEventListener {
public void serialEvent(SerialPortEvent event) {
if(event.isRXCHAR()){//If data is available
try {
byte buffer[] = serialPort.readBytes();
}
catch (SerialPortException ex) {
System.out.println(ex);
}
}
}
}
}
The problem is that I'm always not getting the incoming data in one piece. (I the message has a length of 100 bytes, Im getting 48 and 52 bytes in 2 separates calls)
- The other side send me messages in different lengths.
- In the ICD Im working with, there is a field which tell us the length of the message. (from byte #10 to byte #13)
- I cant read 14 bytes:
(serialPort.readBytes(14);,
parse the message length and read the rest of the message:
(serialPort.readBytes(messageLength-14);
But if I will do it, I will not have the message in once piece (I will have 2 separates byte[] and I need it in one piece (byte[]) without the work of copy function.
Is it possible ?
When working with Ethernet (SocketChannel) we can read data using ByteBuffer. But with JSSC we cant.
Is there a good alternative to JSSC ?
Thanks
You can't rely on any library to give you all the content you need at once because :
the library dont know how many data you need
the library will give you data as it comes and also depending on buffers, hardware, etc
You must develop your own business logic to handle your packets reception. It will of course depend on how your packets are defined : are they always the same length, are they separated with same ending character, etc.
Here is an example that should work with your system (note you should take this as a start, not a full solution, it doesn't include timeout for example) :
static class SerialPortReader implements SerialPortEventListener
{
private int m_nReceptionPosition = 0;
private boolean m_bReceptionActive = false;
private byte[] m_aReceptionBuffer = new byte[2048];
#Override
public void serialEvent(SerialPortEvent p_oEvent)
{
byte[] aReceiveBuffer = new byte[2048];
int nLength = 0;
int nByte = 0;
switch(p_oEvent.getEventType())
{
case SerialPortEvent.RXCHAR:
try
{
aReceiveBuffer = serialPort.readBytes();
for(nByte = 0;nByte < aReceiveBuffer.length;nByte++)
{
//System.out.print(String.format("%02X ",aReceiveBuffer[nByte]));
m_aReceptionBuffer[m_nReceptionPosition] = aReceiveBuffer[nByte];
// Buffer overflow protection
if(m_nReceptionPosition >= 2047)
{
// Reset for next packet
m_bReceptionActive = false;
m_nReceptionPosition = 0;
}
else if(m_bReceptionActive)
{
m_nReceptionPosition++;
// Receive at least the start of the packet including the length
if(m_nReceptionPosition >= 14)
{
nLength = (short)((short)m_aReceptionBuffer[10] & 0x000000FF);
nLength |= ((short)m_aReceptionBuffer[11] << 8) & 0x0000FF00;
nLength |= ((short)m_aReceptionBuffer[12] << 16) & 0x00FF0000;
nLength |= ((short)m_aReceptionBuffer[13] << 24) & 0xFF000000;
//nLength += ..; // Depending if the length in the packet include ALL bytes from the packet or only the content part
if(m_nReceptionPosition >= nLength)
{
// You received at least all the content
// Reset for next packet
m_bReceptionActive = false;
m_nReceptionPosition = 0;
}
}
}
// Start receiving only if this is a Start Of Header
else if(m_aReceptionBuffer[0] == '\0')
{
m_bReceptionActive = true;
m_nReceptionPosition = 1;
}
}
}
catch(Exception e)
{
e.printStackTrace();
}
break;
default:
break;
}
}
}
After writing data to serial port it need to be flushed. Check the timing and pay attention to the fact that read should occur only after other end has written. read size is just an indication to read system call and is not guaranteed. The data may have arrived and is buffered in serial port hardware buffer but may not have been transferred to operating system buffer hence not to application. Consider using scm library, it flushes data after each write http://www.embeddedunveiled.com/
Try this:
Write your data to the serial port (using serialPort.writeBytes()) and if you are expecting a response, use this:
byte[] getData() throws SerialPortException, IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] b;
try {
while ((b = serialPort.readBytes(1, 100)) != null) {
baos.write(b);
// System.out.println ("Wrote: " + b.length + " bytes");
}
// System.out.println("Returning: " + Arrays.toString(baos.toByteArray()));
} catch (SerialPortTimeoutException ex) {
; //don't want to catch it, it just means there is no more data to read
}
return baos.toByteArray();
}
Do what you want with the returned byte array; in my case I just display it for testing.
I found it works just fine if you read one byte at a time, using a 100ms timeout, and when it does time out, you've read all data in the buffer.
Source: trying to talk to an Epson serial printer using jssc and ESC/POS.
I'm using AES/CTR/NoPadding algorithm to encrypt data sent using socket between PC and Android.
I wrote unit-test, it sends [1;512] bytes to Android device and receive back the same data - echo service. Received data must be equal to data that was sent.
Test client:
for (int n = 1; n <= 512; n++) {
... skip ...
try {
Object connection = socketFilter.openConnection(socket);
in = new CipherInputStream(socket.getInputStream(), encryptor);
out = new CipherOutputStream(socket.getOutputStream(), decryptor);
byte buf[] = new byte[n];
byte received[] = new byte[n];
TestUtils.numbers(buf);
out.write(buf, 0, buf.length);
socket.shutdownOutput();
int len = in.read(received, 0, received.length);
if (buf.length != len) {
System.err.println("Expected: " + buf.length + " but was: " + len);
}
}
finally {
... skip close streams ...
}
}
Echo server:
Socket clientSocket = socket.accept();
CipherInputStream in = new CipherInputStream(clientSocket.getInputStream(), decryptor);
CipherOutputStream out = new CipherOutputStream(clientSocket.getOutputStream(), encryptor);
try {
byte buf[] = new byte[512];
int len;
if ((len = in.read(buf)) > 0) {
out.write(buf, 0, len);
out.close();
}
}
finally {
in.close();
out.close();
}
I tested this code with localhost - all works fine.
When i testing it with Android device, the last block is lost if it's not full.
So, if it was 30 bytes, then only 16 bytes received.
Messages from the test:
... skip ...
Expected: 30 but was: 16
Expected: 31 but was: 16
Expected: 33 but was: 32
... skip ...
Expected: 207 but was: 192
Expected: 209 but was: 208
Expected: 210 but was: 208
... skip ...
What can be wrong?
It seems that the problem is caused by the fact that Android and Hotspot JVM uses different Cipher Provider.
Android uses one called Bouncy Castle which has a known 'bug' in AES/CTR mode. It will miss the last block when doing encryption/decryption. (see many other stackoverflow questions)
If you want only CTR mode. Known work around is that you implement it yourself on Android by generating blocks of keystream repeatedly "on the fly"(by encrypt byte array of 0's), and XOR them with your buffer.
Hope this helps
Have you completely flushed the encryption streams before you close them? AES processes data in block sized chunks, in CTR mode it is the keystream. If you don't fully flush the stream before closing it, on either encryption or decryption, you will probably lose the last block.
Similarly, you need to be sure that you have written/read everything from the transfer file streams between Android and PC. Your last pieces of data may have been sitting in a file transfer buffer somewhere waiting to be written when the buffer was closed.
Android does differ from Java, so I suspect that your error is probably on the Android side. Perhaps try Android -> Android as well as PC -> PC, just to make sure that everything on the Android side is fine.
Sorry, but I think I am going mad. I have this in C++:
std::stringstream message;
protoMsg.SerializeToOstream(&message);
boost::system::error_code ignored_error;
std::cout << "Writing response to socket. Byte size " << protoMsg.ByteSize() << " Buffer " << message.str().length() << std::endl;
for(int i(0);i<(int)message.str().length();i++)
std::cout << (int)message.str().at(i);
std::cout << std::endl;
boost::asio::write(socket, boost::asio::buffer(message.str()), ignored_error);
std::cout << "Done writing." << std::endl;
Which yields this output
Writing response to socket. Byte size 88 Buffer 88
1886811891161111001113278971091012500000000320400480560640730000000081000000008859621061211611110011132115121109789710910111416691201161011141109710832114101113117101115116
Done writing.
And this in Java:
try {
System.out.println("Parsing");
int lenbytes = is.read(buffer);
System.out.println("Read bytes " + lenbytes);
for(int i=0;i<lenbytes;i++)
System.out.print(Integer.toString((int)buffer[i]));
System.out.println("");
EnvInfoProto envInfoProto = EnvInfoProto.parseFrom(buffer);
System.out.println("Done");
return envInfoProto;
} catch(IOException ignore) { System.out.println("Ex: " + ignore.toString(); }
Which yields
Parsing
Read bytes 88
1886811891161111001113278971091012500000000320400480560640730000000081000000008859621061211611110011132115121109789710910111416691201161011141109710832114101113117101115116
Ex: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
The binary data is the same. I checked that I am using the right version of the proto files. I am a bit at a loss tbh. Any help appreciated.
You're asking the message to be parsed from the whole of buffer - and my guess is that buffer is more than 88 bytes long.
I can't remember offhand whether parseFrom allows you to specify the maximum amount of data to read, but an alternative would be:
ByteArrayIntputStream stream = new ByteArrayInputStream(buffer, 0, lenbytes);
EnvInfoProto envInfoProto = EnvInfoProto.parseFrom(stream);
Note that this still has the problem that you're assuming you can read all the data from the stream in a single call to read, which is never a good idea - but that's a very separate issue. If you're going to close the socket after writing, you could just parse stream from the socket's InputStream in the Java code, of course. If you're not going to close the stream, I would suggest writing the message length to the socket first (as a 32-bit integer) so you can then read that in Java, and read exactly the right amount of data, knowing when you're finished.