We Keep Coding

How to handle byte stream without corrupting data

I got a problem of how to handle byte data without corrupting them. Here is my code
...
byte[] b = new byte[1000];
// read input stream
BufferedInputStream inData = new BufferedInputStream(socket.getInputStream());
int length = inData.read(b);
String data = new String(b, 0, length);
if (Log4j.log.isEnabledFor(Level.INFO)) {
Log4j.log.info("Data Length: " + length
+ ", Received data: " + data);
}
...
// start a new socket to other server
...
BufferedOutputStream out = new BufferedOutputStream(remote.getOutputStream());
out.write(data.getBytes());
out.flush();
...
It seem like nothing problem here. But if I got a hex string like
F8 F0 F0 C2 20 00 00 80 00 00
few data like C2 will be turned into 3F. I could see this in my log & remote server's log too.
At first, I suspect it will be the overflow. But since those data will be treat as Hex String and send to another server, so this suspicion will be crossed.
I got not clue about what is going on about this, so I could really use some help if anyone knows about this issue.

Right now you are converting the bytes into a String with the platform default charset, and then calling getBytes() back later. If the bytes do not represent a valid string in that charset, data will be lost, e.g. the invalid bytes will be replaced with the character '?'.
Stop that. If you have bytes, pass them around as a byte[]. Do not at any point convert them into a String.

InputStream reads extra bytes which are not exist in source

I have the Java server that receives the RTMP packets that are sent from client app. The server reads the packet header using InputStream, recognizes how big the packet body is, then creates byte array with that size, and then reads that body from InputStream in that array.
The problem is: the received set of bytes are modified - there are neccessary bytes (that exist in source) standing with extra bytes that don't exist in the source packet (I watch the content of the source packet via WireShark and compare them with those bytes that I received on the server).
These extra bytes are 0xc6 bytes that meet periodically by the way...
It looks like this:
Source: ... 75 f1 f5 55 73 .... fc a9 47 14 ... 40 ca d5 75 ... fe 30 a7
Received: ... 75 f1 f5 55 73 c6 .... fc a9 47 14 c6 ... 40 ca d5 75 c6 ... fe 30 a7
... - means "some quantity of bytes here"
As a result, I can't receive neccessary data because it's stretched, it's bigger than it have to be, than the body size that I received from rtmp header. And most importantly, that modified data is not what I had to receive!
My questions are: how can it be fixed? What's wrong with InputStream? Why does it insert those 0xc6 bytes to the receiving array?
I understand that I can simply parse received array and exclude those extra bytes, but this is bad solution, since speed and performance are neccessary (and, in this case, it's not clear that it's an extra byte or byte from source, without the comparison of whole arrays) ...
enter code here
public static void getRtmpPacket(InputStream in) throws Exception {
byte[] rtmpHeader = new byte[8];
byte[] rtmpBody;
int bodySize = 0;
//reading rtmp header:
in.read(rtmpHeader);
//reading the body size. This method works fine
bodySize = Server.bigEndianBytesToInt(rtmpHeader, 4, 3);
rtmpBody = new byte[bodySize];
in.read(rtmpBody);
//printing received data:
System.out.println("Packet:");
System.out.println("Body size: " + bodySize);
System.out.print(bytesToString(rtmpHeader) + " ");
System.out.print(bytesToString(rtmpBody));
System.out.println();
}

According to the RTMP spec, it behaves normally. You need to "unchunk" the incoming data, so reading it all at once in a single read() will not work.
Something along these lines (pseudocode):
int remaining = payloadSize;
int totalRead = 0;
int totalReadForChunk = 0;
while (true) {
int num = read(buf, 0, min(remaining, chunkSize - totalReadForChunk))
if (num < 0) break; // i/o error
appendData(<buf>, 0, num)
totalReadForChunk += num
remaining -= num
if (remaining == 0) break; // end of payload
if (totalReadForChunk == chunkSize) {
totalReadForChunk = 0;
// read the chunk header (it's not neccessarily 0xc6)
int header = read()
if (header != currentStreamEmptyHeader) { // 0xc6
// ... parse the new rtmp message according to header value
// (usually invoke the upper-level message reading method "recursively")
}
}
}

Probably, you should see (and use) code of Red5 Media Server and other open-source solutions that implement RTMP protocol.

InputStream.read(byte[]) is only guarenteed to read one byte, and it return the length as an int of the actual length read.
in.read(rtmpHeader); // might read 1, 2, 3, .. 8 bytes.
//reading the body size. This method works fine
bodySize = Server.bigEndianBytesToInt(rtmpHeader, 4, 3);
rtmpBody = new byte[bodySize];
in.read(rtmpBody); // might read 1, 2, 3, ... bodySize bytes.
If you don't check the actual length, and assume the byte[] is full, you get whatever bytes where there before you called read().
What you intended is available using DataInputStream
DataInputStream dis = new DataInputStream(in);
int len = dis.readInt(); // read an int in big endian.
byte[]] bytes = new byte[len];
dis.readFully(bytes); // read the whole byte[] or throw an IOException.

The problem is resolved.
Those extra 0xc6 bytes were the chunking bytes of RTMP packet, which were not visible from the WireShark.
More than this, received header says the actual body size and WireShark "confirms" it, but in fact the body size will be bigger, and should be calculated.
https://www.wireshark.org/lists/wireshark-bugs/200801/msg00011.html
http://red5.osflash.narkive.com/LYumrzr4/rtmp-video-packets-and-streaming-thereof#post12

Java Socket handling in Android different than on other platforms?

I have a very strange situation. I connect my Java software with a device, let´s call it "Black Box" (because I cannot look into it or make traces within it). I am adressing a specific port (5550) and send commands as byte sequences on a socket. As a result, I get an answer from the Black Box on the same socket.
Both my commands and the replies are prefixed in a pre-defined way (according to the API) and have an XOR checksum.
When I run the code from Windows, all is fine: Command 1 gets its Reply 1 and Command 2 gets its Reply 2.
When I run the code from Android (which is actually my target - Windows came into play to track down the error) it gets STRANGE: Command 1 gets its Reply 1 but Command 2 does not. When I play with Command 2 (change the prefix illegally, violate the checksum) the Black Box reacts as expected (with an error reply). But with the correct Command 2 being issued from Android, the Reply is totally mis-formed: Wrong prefix and missing checksum.
In the try to analyse the error I tried WireShark and this shows that on the network interface, the Black Box is sending the RIGHT Reply 2, but evaluating this reply in Java from the socket, it is wrong. How can this be when all is fine for Command/Reply 1???
Strange is, that parts of the expected data are present:
Expected: ff fe e4 04 00 11 00 f1
Received: fd fd fd 04 00 11 00 // byte 8 missing
I am attaching the minimalistic code to force the problem. What could falsify the bytes which I receive? Is there a "raw" access in Java to the socket which could reveal the problem?
I am totally confused so any help would be appreciated:
String address = "192.168.1.10";
int port = 5550;
Socket socket;
OutputStream out;
BufferedReader in;
try {
socket = new Socket(address, port);
out = socket.getOutputStream();
in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
// This is "Command 1" which is receiving the right reply
// byte[] allesAn = new byte[] {(byte)0xff, (byte)0xfe, (byte)0x21, (byte)0x81, (byte)0xa0};
// out.write(allesAn);
// This is "Command 2" which will not receive a right reply
byte[] getLokInfo3 = new byte[] {(byte)0xff, (byte)0xfe, (byte)0xe3, (byte)0, (byte)0, (byte)3, (byte)0xe0};
out.write(getLokInfo3);
out.flush();
while (true) {
String received = "";
final int BufSize = 1000;
char[] buffer = new char[BufSize];
int charsRead = 0;
charsRead = in.read(buffer, 0, BufSize);
// Convert to hex presentation
for (int i=0; i < charsRead; i++) {
byte b = (byte)buffer[i];
received += hexByte((b + 256) % 256) + " ";
}
String result = charsRead + ">" + received + "<";
Log.e("X", "Read: " + result);
}
} catch (Exception e) {
Log.e("X", e.getMessage() + "");
}
with
private static String hexByte(int value) {
String s = Integer.toHexString(value);
return s.length() % 2 == 0 ? s : "0" + s;
}
Here is what wireshark says, showing the expected 8 bytes:

How ObjectDeletion works in java card?

Below, you see a program that I wrote to see the state of different fields and memory allocations after calling requestObjectDeletion() method:
public class ReqObjDel extends Applet {
static byte[] buffer = new byte[2];
static boolean isNull = false;
private ReqObjDel() {
}
public static void install(byte bArray[], short bOffset, byte bLength)
throws ISOException {
new ReqObjDel().register();
}
public void process(APDU arg0) throws ISOException {
if (selectingApplet()) {
return;
}
if (buffer != null && (short) buffer.length == (short) 10) {
return;
}
byte[] oldBuffer = buffer;
buffer = new byte[10];
JCSystem.requestObjectDeletion();
if (oldBuffer == null)
isNull = true;
if (isNull) {
ISOException.throwIt((short) 0x1111);
} else
ISOException.throwIt((short) 0x0000);
}
}
As far as I know,this method reclaims memory which is being used by “unreachable” objects. To be “unreachable”, an object can neither be pointed to by a static field nor by an object field. So calling requestObjectDeletion() in the above program reclaims the part of EEPROM that oldBuffer is refer to (As far as I know, oldBuffer is neither class field nor object field,right?). In this situation I expect that oldBuffer == null and therefore the JCRE must return 0x1111. But the output is 0x0000 unexpectedly :
OSC: opensc-tool -s 00a404000b0102030405060708090000 -s 00000000
Using reader with a card: ACS CCID USB Reader 0
Sending: 00 A4 04 00 0B 01 02 03 04 05 06 07 08 09 00 00
Received (SW1=0x90, SW2=0x00)
Sending: 00 00 00 00
Received (SW1=0x00, SW2=0x00)
Q1 : What can I conclude?
That part of memory is not reclaimed?
That part of memory is reclaimed but oldBuffer is a reference to it still?
something else?
Q2 : Is there any way to obtain the free memory size before and after of calling this method? (i.e. is there any method that return the size of free memory[not allocated]?)
Update 1 : Trying JCSystem.getAvailableMemory()
Based on #vojta answer, I changed my program in a way that the line byte[] oldBuffer = buffer; runs only once (using a flag named isFirstInvocation) and return the free available memory in two consecutive process() method invocation :
public class ReqObjDel extends Applet {
static byte[] buffer = new byte[10];
static boolean isFirstInvocation = true;
private ReqObjDel() {
}
public static void install(byte bArray[], short bOffset, byte bLength)
throws ISOException {
new ReqObjDel().register();
}
public void process(APDU arg0) throws ISOException {
if (selectingApplet()) {
return;
}
short availableMem1 = JCSystem
.getAvailableMemory(JCSystem.MEMORY_TYPE_PERSISTENT);
if (isFirstInvocation) {
byte[] oldBuffer = buffer;
buffer = new byte[10];
JCSystem.requestObjectDeletion();
firstInvocation = false;
}
short availableMem2 = JCSystem
.getAvailableMemory(JCSystem.MEMORY_TYPE_PERSISTENT);
short availableMemory = (short) (availableMem1 + availableMem2);
ISOException.throwIt(availableMemory);
}
}
And this is the output :
OSC: osc -s 00a404000b0102030405060708090000 -s 00000000 -s 00000000
Using reader with a card: ACS CCID USB Reader 0
Sending: 00 A4 04 00 0B 01 02 03 04 05 06 07 08 09 00 00
Received (SW1=0x90, SW2=0x00)
Sending: 00 00 00 00
Received (SW1=0xFF, SW2=0xFE)
Sending: 00 00 00 00
Received (SW1=0xFF, SW2=0xFE)
As both invocations return an equal value, I think the JCRE reclaims that part of memory immediately after calling requestObjectDeletion(), right?

First of all, a rule based on my personal experience: if possible, do not use the garbage collector at all. GC is very slow and could be even dangerous (see Javacard - power loss during garbage collection).
Q1:
If you really have to use GC, read the documentation:
This method is invoked by the applet to trigger the object deletion
service of the Java Card runtime environment. If the Java Card runtime
environment implements the object deletion mechanism, the request is
merely logged at this time. The Java Card runtime environment must
schedule the object deletion service prior to the next invocation of
the Applet.process() method.
Shortly speaking, JCSystem.requestObjectDeletion(); has no immediate effect. That is why your local variable oldBuffer remains unchanged.
Q2: To find out how much persistent memory is available to your applet, use:
JCSystem.getAvailableMemory(JCSystem.MEMORY_TYPE_PERSISTENT)
ANSWER to UPDATE 1: JCSystem.getAvailableMemory(JCSystem.MEMORY_TYPE_PERSISTENT) may be confusing for cards with more than 32767 bytes of persistent memory. Such cards usually offer their own proprietary ways to find out available memory.
If the number of available bytes is greater than 32767, then this
method returns 32767.

How to read raw bytes from socket in Python?

I have an android java app sending bytes over a socket which is connected to a host machine running a server in Python. I need to receive these bytes as they were sent from the python socket. I see that in Python 'socket.recv' only returns a string. When I send an ASCII string from the java app, I am able to receive the data correctly in the python server, but when I send binary data using java byte, I see the data received is not same. I need to receive raw bytes in Python for my protocol to work correctly. Please point me in right direction.
Code snippet for Sending data on socket:
private void sendFrameMessage(byte[] data) {
byte[] lengthInfo = new byte[4];
Log.v(TAG, "sendFrameMessage");
for(int i=0; i<data.length; i++) {
Log.v(TAG, String.format("data[%d] = %d", i, data[i]));
}
try {
lengthInfo[0] = (byte) data.length;
lengthInfo[1] = (byte) (data.length >> 8);
lengthInfo[2] = (byte) (data.length >> 16);
lengthInfo[3] = (byte) (data.length >> 24);
DataOutputStream dos;
dos = new DataOutputStream(mSocket.getOutputStream());
dos.write(lengthInfo, 0, 4);
dos.write(data, 0, data.length);
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
Python Code on receiver side
def recvFrameMessage(self, s):
recv_count = 4;
data = s.recv(recv_count)
if data == 0:
return None
total_rx = len(data)
lenInfo = data
while total_rx < recv_count:
data = s.recv(recv_count - total_rx)
if data == 0:
return None
total_rx += len(data)
lenInfo = lenInfo + data
recv_count = self.decodeFrameLen(lenInfo)
logger.info("length = %d" % recv_count)
data = s.recv(recv_count)
total_rx = len(data)
msg = data
while total_rx < recv_count:
data = s.recv(recv_count - total_rx)
if data == 0:
return None
total_rx += len(data)
msg = msg + data
logger.info("msg = " + msg)
for i in range(0, len(msg)-1):
logger.info("msg[%d] = %s" % (i, msg[i]))
return msg

#SteveP makes good points for binary data "with some structure", but if this is a plain stream of bytes, in Python 2 simply apply the ord() function to each "character" you get from the socket. For example, if the Java end sends a NUL byte, that will show up on the Python end as the character "\x00", and then:
>>> ord("\x00")
0
To convert a whole string s,
map(ord, s)
returns a list of the corresponding 8-bit unsigned integers.
I'm assuming Python 2 here.

Reading binary data is perfectly doable, but what if the binary representation from your android app is different than the byte representation on the Python server? From the Python documentation:
It is perfectly possible to send binary data over a socket. The major
problem is that not all machines use the same formats for binary data.
For example, a Motorola chip will represent a 16 bit integer with the
value 1 as the two hex bytes 00 01. Intel and DEC, however, are
byte-reversed - that same 1 is 01 00. Socket libraries have calls for
converting 16 and 32 bit integers - ntohl, htonl, ntohs, htons where
“n” means network and “h” means host, “s” means short and “l” means
long. Where network order is host order, these do nothing, but where
the machine is byte-reversed, these swap the bytes around
appropriately.
Without code and example input/output, this question is going to be really difficult to answer. I assume the issue is that the representation is different. The most likely issue is that Java uses big endian, whereas Python adheres to whatever machine you are running it off of. If your server uses little endian, then you need to account for that. See here for a more thorough explanation on endianness.