We Keep Coding

How to handle byte stream without corrupting data

I got a problem of how to handle byte data without corrupting them. Here is my code
...
byte[] b = new byte[1000];
// read input stream
BufferedInputStream inData = new BufferedInputStream(socket.getInputStream());
int length = inData.read(b);
String data = new String(b, 0, length);
if (Log4j.log.isEnabledFor(Level.INFO)) {
Log4j.log.info("Data Length: " + length
+ ", Received data: " + data);
}
...
// start a new socket to other server
...
BufferedOutputStream out = new BufferedOutputStream(remote.getOutputStream());
out.write(data.getBytes());
out.flush();
...
It seem like nothing problem here. But if I got a hex string like
F8 F0 F0 C2 20 00 00 80 00 00
few data like C2 will be turned into 3F. I could see this in my log & remote server's log too.
At first, I suspect it will be the overflow. But since those data will be treat as Hex String and send to another server, so this suspicion will be crossed.
I got not clue about what is going on about this, so I could really use some help if anyone knows about this issue.

Right now you are converting the bytes into a String with the platform default charset, and then calling getBytes() back later. If the bytes do not represent a valid string in that charset, data will be lost, e.g. the invalid bytes will be replaced with the character '?'.
Stop that. If you have bytes, pass them around as a byte[]. Do not at any point convert them into a String.

First byte drops when writing over Java SSL Socket

I am sending multiple commands in a byte array format and receive responses based on the commands that were sent in a typical client/server communication. My Java app is the client running on Windows 7 and I know what commands to send and what I am expected to receive in the response. However, I do not have any control or have any knowledge of the server source code.
The problem I am having is that on the second command I sent or any commands after that, the first byte of the array is being dropped. When I send the first command, I get the proper response since the first byte is not dropped. When sending the next command or any commands after that, the first byte is dropped which the server does not respond since the command is not in the proper format for the server to recognize.
I am sending these commands over a Java SSLSocket DataOutputStream and of course I am receiving the responses on a DataInputStream. I first perform a handshake with the server and proceed on after the handshake is successful. At this point is when I send the first command and receive the response shown here in hex.:
Sending: 01 03 03
Receive: 01 0e fd 85 02 09 01 01 04 01 06
The next command being sent:
Sending: 01 48 65 6c 6c 6f
But this is where I do not receive a response from the server.
When printing out the javax.net.debug output, I can see that the first byte '01' does drop or went missing somehow.
Padded plaintext before ENCRYPTION: len = 32
0000: 48 65 6C 6C 6F FE 57 F9 4A 29 13 8F 2B AB 71 A3 Hello.W.J)..+.q.
0010: 16 12 29 FF D5 DE 12 48 8B 06 06 06 06 06 06 06 ..)....H........
main, WRITE: TLSv1 Application Data, length = 32
[Raw write]: length = 37
0000: 17 03 01 00 20 34 42 ED 88 FC 41 2D 13 1A FD BA .... 4B...A-....
0010: 64 0E 9D C7 FE 11 76 96 48 09 A6 BC B2 BC 0E FA d.....v.H.......
0020: C8 5B 79 4B 82 .[yK.
The following is my source code:
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;
import javax.net.ssl.SSLSocketFactory;
import javax.net.ssl.TrustManagerFactory;
import javax.net.ssl.HandshakeCompletedEvent;
import javax.net.ssl.HandshakeCompletedListener;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLSocket;
import javax.net.ssl.X509TrustManager;
import javax.net.ssl.TrustManager;
import java.security.KeyStore;
public class SSLSocketTest
{
private SSLSocket sslSocket = null;
private SSLSocketFactory sslSocketFactory = null;
private String ipAddress = "192.168.100.99";
private int port = 9999;
DataOutputStream dataOS = null;
DataInputStream dataIS = null;
private boolean handshakeSuccessful = false;
public static void main(String[] args)
{
SSLSocketTest sslSocketTest = new SSLSocketTest();
sslSocketTest.sslSocketConnect();
}
SSLSocketTest()
{
System.setProperty("javax.net.debug", "all");
try{
File certFile = new File("cacerts");
KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
char[] certPassword = "changeit".toCharArray();
InputStream fileIS = new FileInputStream(certFile);
keyStore.load(fileIS, certPassword);
fileIS.close();
SSLContext sslContext = SSLContext.getInstance("TLSv1");
TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trustManagerFactory.init(keyStore);
X509TrustManager defaultTrustManager = (X509TrustManager)trustManagerFactory.getTrustManagers()[0];
sslContext.init(null, new TrustManager[] {defaultTrustManager}, null);
sslSocketFactory = sslContext.getSocketFactory();
}catch(Exception e){
e.printStackTrace();
}
}
public void sslSocketConnect()
{
try{
sslSocket = (SSLSocket) sslSocketFactory.createSocket(ipAddress, port);
dataOS = new DataOutputStream(sslSocket.getOutputStream());
dataIS = new DataInputStream(sslSocket.getInputStream());
sslSocket.setSoTimeout(15000);
//Handshake
sslSocket.addHandshakeCompletedListener(new MyHandshakeListener());
sslSocket.startHandshake();
while(!handshakeSuccessful)
{
Thread.sleep(100);
}
//Sending commands
byte[] firstCommand = new byte[]{(byte)0x01, (byte)0x03, (byte)0x03};
String[] firstCommandResponse = processCommand(firstCommand);
byte[] secondCommand = new byte[]{(byte)0x01, (byte)0x48, (byte)0x65, (byte)0x6C, (byte)0x6C, (byte)0x6F};
String[] secondCommandResponse = processCommand(secondCommand);
disconnect();
}catch(Exception e){
e.printStackTrace();
}
}
public void disconnect()
{
try{
byte[] endConnection = new byte[]{(byte)0x01, (byte)0x01, (byte)0x02, (byte)0x03};
processCommand(endConnection);
dataOS.close();
dataIS.close();
sslSocket.close();
}catch (Exception e){
e.printStackTrace();
}
}
public String[] processCommand(byte[] command)
{
String[] returnResponse = null;
byte[] commandResponse = new byte[120];
byte[] trimCommandResponse;
try{
int commandResponseLength = -1;
int errorCount = 0;
while(commandResponseLength == -1)
{
StringBuilder cmdStr = new StringBuilder();
cmdStr.append("Sending: ");
for(int i=0; i<command.length; i++)
{
cmdStr.append(fixHexStringData(Integer.toHexString(command[i])) + " ");
}
System.out.println(cmdStr.toString());
dataOS.write(command, 0, command.length);
dataOS.flush();
commandResponseLength = dataIS.read(commandResponse);
errorCount++;
if(errorCount == 3)
{
throw new Exception();
}
}
returnResponse = new String[commandResponseLength];
trimCommandResponse = new byte[commandResponseLength];
//Getting Reponse Data
for(int i=0; i<commandResponseLength; i++)
{
returnResponse[i] = fixHexStringData(Integer.toHexString(commandResponse[i]));
trimCommandResponse[i] = commandResponse[i];
}
StringBuilder rcvStr = new StringBuilder();
rcvStr.append("Receive: ");
for(int i=0; i<returnResponse.length; i++)
{
rcvStr.append(returnResponse[i] + " ");
}
System.out.println(rcvStr.toString());
}catch(Exception e){
e.printStackTrace();
}
return returnResponse;
}
private String fixHexStringData(String dataByte)
{
if(dataByte.length() < 2)
{
dataByte = "0" + dataByte;
}
else if(dataByte.length() > 2)
{
dataByte = dataByte.substring(dataByte.length()-2);
}
return dataByte;
}
class MyHandshakeListener implements HandshakeCompletedListener
{
public void handshakeCompleted(HandshakeCompletedEvent e)
{
System.out.println("Handshake succesful!");
handshakeSuccessful = true;
}
}
}
The questions I have are:
Am I missing a step in writing out the byte array? I have done this over a standard Java Socket with no issues, so is writing over a SSL Socket different from a standard Socket? I looked for this but did not see anything different.
Could this be a certificate issue? If the handshake and the first command are successful, would this mean that the communication has been established at this point and it is beyond the certificate?
Could the server be affecting this? If so, what could be the reason behind this? If writing the byte array to the DataOutputStream is on the client side and the first byte is dropped, how could the server have any affect on the client side?
Could this be a JVM bug?

It looks like this is actually a function of the JSSE implementation which splits the data between two packets. The first byte goes in one packet and the rest in the next. More detail at this answer
You should be able to override this functionality by including
System.setProperty("jsse.enableCBCProtection", "false");

InputStream reads extra bytes which are not exist in source

I have the Java server that receives the RTMP packets that are sent from client app. The server reads the packet header using InputStream, recognizes how big the packet body is, then creates byte array with that size, and then reads that body from InputStream in that array.
The problem is: the received set of bytes are modified - there are neccessary bytes (that exist in source) standing with extra bytes that don't exist in the source packet (I watch the content of the source packet via WireShark and compare them with those bytes that I received on the server).
These extra bytes are 0xc6 bytes that meet periodically by the way...
It looks like this:
Source: ... 75 f1 f5 55 73 .... fc a9 47 14 ... 40 ca d5 75 ... fe 30 a7
Received: ... 75 f1 f5 55 73 c6 .... fc a9 47 14 c6 ... 40 ca d5 75 c6 ... fe 30 a7
... - means "some quantity of bytes here"
As a result, I can't receive neccessary data because it's stretched, it's bigger than it have to be, than the body size that I received from rtmp header. And most importantly, that modified data is not what I had to receive!
My questions are: how can it be fixed? What's wrong with InputStream? Why does it insert those 0xc6 bytes to the receiving array?
I understand that I can simply parse received array and exclude those extra bytes, but this is bad solution, since speed and performance are neccessary (and, in this case, it's not clear that it's an extra byte or byte from source, without the comparison of whole arrays) ...
enter code here
public static void getRtmpPacket(InputStream in) throws Exception {
byte[] rtmpHeader = new byte[8];
byte[] rtmpBody;
int bodySize = 0;
//reading rtmp header:
in.read(rtmpHeader);
//reading the body size. This method works fine
bodySize = Server.bigEndianBytesToInt(rtmpHeader, 4, 3);
rtmpBody = new byte[bodySize];
in.read(rtmpBody);
//printing received data:
System.out.println("Packet:");
System.out.println("Body size: " + bodySize);
System.out.print(bytesToString(rtmpHeader) + " ");
System.out.print(bytesToString(rtmpBody));
System.out.println();
}

According to the RTMP spec, it behaves normally. You need to "unchunk" the incoming data, so reading it all at once in a single read() will not work.
Something along these lines (pseudocode):
int remaining = payloadSize;
int totalRead = 0;
int totalReadForChunk = 0;
while (true) {
int num = read(buf, 0, min(remaining, chunkSize - totalReadForChunk))
if (num < 0) break; // i/o error
appendData(<buf>, 0, num)
totalReadForChunk += num
remaining -= num
if (remaining == 0) break; // end of payload
if (totalReadForChunk == chunkSize) {
totalReadForChunk = 0;
// read the chunk header (it's not neccessarily 0xc6)
int header = read()
if (header != currentStreamEmptyHeader) { // 0xc6
// ... parse the new rtmp message according to header value
// (usually invoke the upper-level message reading method "recursively")
}
}
}

Probably, you should see (and use) code of Red5 Media Server and other open-source solutions that implement RTMP protocol.

InputStream.read(byte[]) is only guarenteed to read one byte, and it return the length as an int of the actual length read.
in.read(rtmpHeader); // might read 1, 2, 3, .. 8 bytes.
//reading the body size. This method works fine
bodySize = Server.bigEndianBytesToInt(rtmpHeader, 4, 3);
rtmpBody = new byte[bodySize];
in.read(rtmpBody); // might read 1, 2, 3, ... bodySize bytes.
If you don't check the actual length, and assume the byte[] is full, you get whatever bytes where there before you called read().
What you intended is available using DataInputStream
DataInputStream dis = new DataInputStream(in);
int len = dis.readInt(); // read an int in big endian.
byte[]] bytes = new byte[len];
dis.readFully(bytes); // read the whole byte[] or throw an IOException.

The problem is resolved.
Those extra 0xc6 bytes were the chunking bytes of RTMP packet, which were not visible from the WireShark.
More than this, received header says the actual body size and WireShark "confirms" it, but in fact the body size will be bigger, and should be calculated.
https://www.wireshark.org/lists/wireshark-bugs/200801/msg00011.html
http://red5.osflash.narkive.com/LYumrzr4/rtmp-video-packets-and-streaming-thereof#post12

Send byte array on serial port out put stream

when I want to send byte array on serial port stream with java,on destination device I receive different result !!!
byte[] sendingPack = new byte[7];
sendingPack[0] = 0x6E;
sendingPack[1] = 0x55;
sendingPack[2] = (byte) 0x0D;
sendingPack[3] = (byte) (1 & 0x000000FF);
sendingPack[4] = 0x01;
sendingPack[5] = 0x0D;
sendingPack[6] = (byte) 0xAA;
getSendBuffer().getOutputStream().write(sendingPack);
sending array : byte[]{0x6E,0x55,0x0D,0x01,0x01,0x0D,0xAA}
receive result array : 6E 55 0D 0A 01 01 0D 0A AA
on CodeVisionAVR terminal I receive "0A"!!
how can I solve this problem??

The terminal is probably in text reading mode and not in binary read mode.
The 0x0A which is inserted after every 0x0D you send is a carriage return conversion.
The terminal converts "\r" to "\r\n". It adds a line feed char to every carriage return.
The terminal converts every 0D to 0D 0A.
This same feature can be found in the ftp protocol. You tell your client how to transfer files: in text or binary mode.

Java Code to get output of running program

I want to run the following command from a java program:
socat -x -u /dev/ttyFTDI0,raw,echo=0,crnl /dev/ttyFTDI1,raw,echo=0,crnl
This program is supposed to run for an indefinite time while outputting hex strings like this:
b4 03 03 92 00 01 3f c6 b4 03 10 03 00 01 6a af
My current testing code is:
public void testOutput() {
try {
List<String> command = new ArrayList<String>();
command.add("socat");
command.add("-x");
command.add("-u");
command.add("/dev/ttyFTDI0,raw,echo=0,crnl");
command.add("/dev/ttyFTDI1,raw,echo=0,crnl");
ProcessBuilder proBui = new ProcessBuilder(command);
Process process = proBui.start();
//process.waitFor();
BufferedReader r = new BufferedReader(new InputStreamReader(process.getInputStream()));
while (true) {
String temp = r.readLine();
if (temp != null && !temp.isEmpty()) {
//do something with data
System.out.print(temp);
}
Thread.sleep(1);
}
} catch (Exception ex) {
System.out.println(ex);
}
}
While this code works for commands that finish after a given time, commands like the one given or watch ls won't work.

According to the socat documentation, the -x option
Writes the transferred data not only to their target streams, but also to stderr.
You are reading from process.getInputStream() which is connected to the native process stdout - to get stderr you should read process.getErrorStream() instead.