We Keep Coding

InputStream skip bytes

I'm communicating with peers using TCP Sockets and I see that when I read the inputStream for the first incoming message, all goes well. Then when I read the inputStream for the second incoming message, the inputStream skip the first n bytes (n is a different positive number in each run).
How do I know that inputStream skip n bytes? Using Wireshark, I can see that the second message received well but Java TCP socket still ignore the first n bytes.
Moreover, Wireshark itself show me something strange - by looking at the first message in Wireshark, it contains at the end: the start of the second message. And by looking at the second message in Wireshark, the start of the message appears here also.
I can't understand what is going on.
Technical details + Wireshark photos:
The first message I receive is a 'Handshake' message.
The second message I receive is different each time but most of the time it's 'extended' message.
I checked in my code and I only read the same InputStream in 2 places: When I'm waiting for 'Handshake' and when I'm waiting for the rest of the messages which is not equal to 'Handshake' message.
The first message I receive:
* Offset Size Name value
* 0 8-bit byte pstrLength
* 1 pstrlen-bit bytes pstr
* 1+pstrlen 64-bit byte reserved
* 9+pstrlen 20-bit String torrentInfoHash
* 29+pstrlen 20-bit String peerId
* 49+pstrlen
public HandShake(InputStream dataInputStream) throws IOException {
byte[] data = new byte[1];
dataInputStream.read(data);
byte pstrLength = ByteBuffer.wrap(data).get();
data = new byte[pstrLength + 48];// how much we need to read more.
dataInputStream.read(data);
ByteBuffer byteBuffer = ByteBuffer.allocate(1 + pstrLength + 48);
byteBuffer.put(pstrLength);
byteBuffer.put(data);
HandShake handShake = HandShake.createObjectFromPacket(byteBuffer.array());
Details: 13 until 45 is the content of the first message - Handshake. 00 until 3a is the first n bytes fo the second message which will appear also in here:
The second message I receive:
public static PeerMessage create(Peer from, Peer to, InputStream inputStream) throws IOException {
byte[] data = new byte[4];
boolean isPeerClosedConnection = (inputStream.read(data) == -1);
if (isPeerClosedConnection)
throw new IOException("the peer closed the socket:" + from.toString());
int lengthOfTheRest = ByteBuffer.wrap(data).getInt(); // how much do we need to read more
data = new byte[lengthOfTheRest];
isPeerClosedConnection = (inputStream.read(data) == -1);
if (isPeerClosedConnection)
throw new IOException("the peer closed the socket:" + from.toString());
ByteBuffer byteBuffer = ByteBuffer.allocate(4 + lengthOfTheRest);;
byteBuffer.putInt(lengthOfTheRest);
byteBuffer.put(data);
return create(from, to, byteBuffer.array()); // initialize message object from byte[]
}
Details: 00 until 3a is the first n bytes of the second message.
When I read the InputStream, I get the following bytes: from 6d to 65.
Why Wireshark shows the same data twice and why my InputStream skip the first n bytes of the second message?

You wrote:
I calculate how much to read and I use every byte.
You coded:
data = new byte[pstrLength + 48];// how much we need to read more.
dataInputStream.read(data);
This code does not conform with your description. The second read() is not guaranteed to fill the buffer. See the Javadoc. Change it to readFully().
NB There is another problem, in your isPeerConnected test. You are reading a byte of input and throwing it away. This will cause you to lose synchronization with the peer if it is still connected.

InputStream reads extra bytes which are not exist in source

I have the Java server that receives the RTMP packets that are sent from client app. The server reads the packet header using InputStream, recognizes how big the packet body is, then creates byte array with that size, and then reads that body from InputStream in that array.
The problem is: the received set of bytes are modified - there are neccessary bytes (that exist in source) standing with extra bytes that don't exist in the source packet (I watch the content of the source packet via WireShark and compare them with those bytes that I received on the server).
These extra bytes are 0xc6 bytes that meet periodically by the way...
It looks like this:
Source: ... 75 f1 f5 55 73 .... fc a9 47 14 ... 40 ca d5 75 ... fe 30 a7
Received: ... 75 f1 f5 55 73 c6 .... fc a9 47 14 c6 ... 40 ca d5 75 c6 ... fe 30 a7
... - means "some quantity of bytes here"
As a result, I can't receive neccessary data because it's stretched, it's bigger than it have to be, than the body size that I received from rtmp header. And most importantly, that modified data is not what I had to receive!
My questions are: how can it be fixed? What's wrong with InputStream? Why does it insert those 0xc6 bytes to the receiving array?
I understand that I can simply parse received array and exclude those extra bytes, but this is bad solution, since speed and performance are neccessary (and, in this case, it's not clear that it's an extra byte or byte from source, without the comparison of whole arrays) ...
enter code here
public static void getRtmpPacket(InputStream in) throws Exception {
byte[] rtmpHeader = new byte[8];
byte[] rtmpBody;
int bodySize = 0;
//reading rtmp header:
in.read(rtmpHeader);
//reading the body size. This method works fine
bodySize = Server.bigEndianBytesToInt(rtmpHeader, 4, 3);
rtmpBody = new byte[bodySize];
in.read(rtmpBody);
//printing received data:
System.out.println("Packet:");
System.out.println("Body size: " + bodySize);
System.out.print(bytesToString(rtmpHeader) + " ");
System.out.print(bytesToString(rtmpBody));
System.out.println();
}

According to the RTMP spec, it behaves normally. You need to "unchunk" the incoming data, so reading it all at once in a single read() will not work.
Something along these lines (pseudocode):
int remaining = payloadSize;
int totalRead = 0;
int totalReadForChunk = 0;
while (true) {
int num = read(buf, 0, min(remaining, chunkSize - totalReadForChunk))
if (num < 0) break; // i/o error
appendData(<buf>, 0, num)
totalReadForChunk += num
remaining -= num
if (remaining == 0) break; // end of payload
if (totalReadForChunk == chunkSize) {
totalReadForChunk = 0;
// read the chunk header (it's not neccessarily 0xc6)
int header = read()
if (header != currentStreamEmptyHeader) { // 0xc6
// ... parse the new rtmp message according to header value
// (usually invoke the upper-level message reading method "recursively")
}
}
}

Probably, you should see (and use) code of Red5 Media Server and other open-source solutions that implement RTMP protocol.

InputStream.read(byte[]) is only guarenteed to read one byte, and it return the length as an int of the actual length read.
in.read(rtmpHeader); // might read 1, 2, 3, .. 8 bytes.
//reading the body size. This method works fine
bodySize = Server.bigEndianBytesToInt(rtmpHeader, 4, 3);
rtmpBody = new byte[bodySize];
in.read(rtmpBody); // might read 1, 2, 3, ... bodySize bytes.
If you don't check the actual length, and assume the byte[] is full, you get whatever bytes where there before you called read().
What you intended is available using DataInputStream
DataInputStream dis = new DataInputStream(in);
int len = dis.readInt(); // read an int in big endian.
byte[]] bytes = new byte[len];
dis.readFully(bytes); // read the whole byte[] or throw an IOException.

The problem is resolved.
Those extra 0xc6 bytes were the chunking bytes of RTMP packet, which were not visible from the WireShark.
More than this, received header says the actual body size and WireShark "confirms" it, but in fact the body size will be bigger, and should be calculated.
https://www.wireshark.org/lists/wireshark-bugs/200801/msg00011.html
http://red5.osflash.narkive.com/LYumrzr4/rtmp-video-packets-and-streaming-thereof#post12

Send byte array on serial port out put stream

when I want to send byte array on serial port stream with java,on destination device I receive different result !!!
byte[] sendingPack = new byte[7];
sendingPack[0] = 0x6E;
sendingPack[1] = 0x55;
sendingPack[2] = (byte) 0x0D;
sendingPack[3] = (byte) (1 & 0x000000FF);
sendingPack[4] = 0x01;
sendingPack[5] = 0x0D;
sendingPack[6] = (byte) 0xAA;
getSendBuffer().getOutputStream().write(sendingPack);
sending array : byte[]{0x6E,0x55,0x0D,0x01,0x01,0x0D,0xAA}
receive result array : 6E 55 0D 0A 01 01 0D 0A AA
on CodeVisionAVR terminal I receive "0A"!!
how can I solve this problem??

The terminal is probably in text reading mode and not in binary read mode.
The 0x0A which is inserted after every 0x0D you send is a carriage return conversion.
The terminal converts "\r" to "\r\n". It adds a line feed char to every carriage return.
The terminal converts every 0D to 0D 0A.
This same feature can be found in the ftp protocol. You tell your client how to transfer files: in text or binary mode.

Java Socket handling in Android different than on other platforms?

I have a very strange situation. I connect my Java software with a device, let´s call it "Black Box" (because I cannot look into it or make traces within it). I am adressing a specific port (5550) and send commands as byte sequences on a socket. As a result, I get an answer from the Black Box on the same socket.
Both my commands and the replies are prefixed in a pre-defined way (according to the API) and have an XOR checksum.
When I run the code from Windows, all is fine: Command 1 gets its Reply 1 and Command 2 gets its Reply 2.
When I run the code from Android (which is actually my target - Windows came into play to track down the error) it gets STRANGE: Command 1 gets its Reply 1 but Command 2 does not. When I play with Command 2 (change the prefix illegally, violate the checksum) the Black Box reacts as expected (with an error reply). But with the correct Command 2 being issued from Android, the Reply is totally mis-formed: Wrong prefix and missing checksum.
In the try to analyse the error I tried WireShark and this shows that on the network interface, the Black Box is sending the RIGHT Reply 2, but evaluating this reply in Java from the socket, it is wrong. How can this be when all is fine for Command/Reply 1???
Strange is, that parts of the expected data are present:
Expected: ff fe e4 04 00 11 00 f1
Received: fd fd fd 04 00 11 00 // byte 8 missing
I am attaching the minimalistic code to force the problem. What could falsify the bytes which I receive? Is there a "raw" access in Java to the socket which could reveal the problem?
I am totally confused so any help would be appreciated:
String address = "192.168.1.10";
int port = 5550;
Socket socket;
OutputStream out;
BufferedReader in;
try {
socket = new Socket(address, port);
out = socket.getOutputStream();
in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
// This is "Command 1" which is receiving the right reply
// byte[] allesAn = new byte[] {(byte)0xff, (byte)0xfe, (byte)0x21, (byte)0x81, (byte)0xa0};
// out.write(allesAn);
// This is "Command 2" which will not receive a right reply
byte[] getLokInfo3 = new byte[] {(byte)0xff, (byte)0xfe, (byte)0xe3, (byte)0, (byte)0, (byte)3, (byte)0xe0};
out.write(getLokInfo3);
out.flush();
while (true) {
String received = "";
final int BufSize = 1000;
char[] buffer = new char[BufSize];
int charsRead = 0;
charsRead = in.read(buffer, 0, BufSize);
// Convert to hex presentation
for (int i=0; i < charsRead; i++) {
byte b = (byte)buffer[i];
received += hexByte((b + 256) % 256) + " ";
}
String result = charsRead + ">" + received + "<";
Log.e("X", "Read: " + result);
}
} catch (Exception e) {
Log.e("X", e.getMessage() + "");
}
with
private static String hexByte(int value) {
String s = Integer.toHexString(value);
return s.length() % 2 == 0 ? s : "0" + s;
}
Here is what wireshark says, showing the expected 8 bytes:

how to transfer the records in rms(j2me) to j2se through bluetooth

NOW here is the coding for j2me mobile for sending the string:
String s="hai";
try{
String url = "btspp://001F81000250:1;authenticate=false;encrypt=false;master=false";
StreamConnection stream = null;
InputStream in;
OutputStream out;
stream = (StreamConnection) Connector.open(url);
out=stream.openOutputStream();
String s=tf.getString();
byte size=(byte) s.length();
out.write(size);
out.write(s.getBytes());
out.flush();
out.close();
stream.close();
}
catch(Exception e){
}
NOW the coding for j2se for receiving the String :
StreamConnectionNotifier notifier=null;
try{
String url = "btspp://localhost:"+new UUID("1101", true).toString()+";name=PCServerCOMM;authenticate=false";
System.out.println(LocalDevice.getLocalDevice().getBluetoothAddress()+"\nCreate server by uri: " + url);
notifier= (StreamConnectionNotifier) Connector.open(url);
while(true){
System.out.println("waiting....");
StreamConnection con = notifier.acceptAndOpen();
System.out.println("Got connection..");
InputStream is=con.openInputStream();
//byte b[]=new byte[40];
/*
while(is.available()>0){
System.out.print((char)is.read());
}*/
//is.read(b, 0, 40);
int size=is.read();
byte b[]=new byte[size];
is.read(b, 0, size);
File f=new File("d://test.xml");
FileOutputStream fo=new FileOutputStream(f);
fo.write(b,0,b.length);
fo.close();
con.close();
System.out.println(new String (b));
}
//printing(f);
} catch(Exception e){
JOptionPane.showConfirmDialog(new JFrame(), e.getMessage());
}
I tried this coding for data transfer but it is not a successful one because when the string which we sent is too long then there is problem in receiving side. How can I solve this?
Is there any other way to transfer the data in rms to j2se, if so please help me.... please make your reply quick...

The way you are writing and reading here, only strings up to 255 characters in length, which additionally only take the same number of bytes in your default encoding, are written right.
On the writing side:
The statement byte size=(byte) s.length(); converts the length of the string in a byte, thus only takes the lower 8 bits of the length. So, only lengths up to 255 are written right.
Then you are converting the String to a byte array with s.getBytes() - this array could be longer (in bytes) than the original string in characters. This conversion uses the default encoding of your sending device.
On the reading side:
The statement int size=is.read(); reads the length written before, then you are creating a byte array.
is.read(b, 0, size); reads some bytes into this array - it does not necessarily fills the complete array.
Then you are converting your byte array (which may not even be filled completely) to a string, using the default encoding of the receiving device.
So, we have:
All strings longer than 255 characters are written wrongly.
If sending and receiving side are using different encodings, you may get a wrong output.
If the sending side uses an encoding like UTF-8 where some characters take more than one byte, the string is cut off at the end (if such characters occur).
How to solve this:
If you can use a DataInputStream and DataOutputStream on both sides (I don't know anything about J2ME), use them there, with their readUTF and writeUTF methods. They solve all your problems (if your strings take at most 65535 bytes in the modified UTF-8 encoding used here).
If not:
make a decision on how long the strings can be, and encode your length with the right number of bytes. 4 bytes are enough for every Java String.
measure the length after converting to a byte[], not before.
use a loop for reading into the array, to be sure to capture the whole string.
for the getBytes() and new String(...), use the variants which take an explicit encoding name and give them the same encoding (I recommend "UTF-8").