I am sending multiple commands in a byte array format and receive responses based on the commands that were sent in a typical client/server communication. My Java app is the client running on Windows 7 and I know what commands to send and what I am expected to receive in the response. However, I do not have any control or have any knowledge of the server source code.
The problem I am having is that on the second command I sent or any commands after that, the first byte of the array is being dropped. When I send the first command, I get the proper response since the first byte is not dropped. When sending the next command or any commands after that, the first byte is dropped which the server does not respond since the command is not in the proper format for the server to recognize.
I am sending these commands over a Java SSLSocket DataOutputStream and of course I am receiving the responses on a DataInputStream. I first perform a handshake with the server and proceed on after the handshake is successful. At this point is when I send the first command and receive the response shown here in hex.:
Sending: 01 03 03
Receive: 01 0e fd 85 02 09 01 01 04 01 06
The next command being sent:
Sending: 01 48 65 6c 6c 6f
But this is where I do not receive a response from the server.
When printing out the javax.net.debug output, I can see that the first byte '01' does drop or went missing somehow.
Padded plaintext before ENCRYPTION: len = 32
0000: 48 65 6C 6C 6F FE 57 F9 4A 29 13 8F 2B AB 71 A3 Hello.W.J)..+.q.
0010: 16 12 29 FF D5 DE 12 48 8B 06 06 06 06 06 06 06 ..)....H........
main, WRITE: TLSv1 Application Data, length = 32
[Raw write]: length = 37
0000: 17 03 01 00 20 34 42 ED 88 FC 41 2D 13 1A FD BA .... 4B...A-....
0010: 64 0E 9D C7 FE 11 76 96 48 09 A6 BC B2 BC 0E FA d.....v.H.......
0020: C8 5B 79 4B 82 .[yK.
The following is my source code:
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;
import javax.net.ssl.SSLSocketFactory;
import javax.net.ssl.TrustManagerFactory;
import javax.net.ssl.HandshakeCompletedEvent;
import javax.net.ssl.HandshakeCompletedListener;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLSocket;
import javax.net.ssl.X509TrustManager;
import javax.net.ssl.TrustManager;
import java.security.KeyStore;
public class SSLSocketTest
{
private SSLSocket sslSocket = null;
private SSLSocketFactory sslSocketFactory = null;
private String ipAddress = "192.168.100.99";
private int port = 9999;
DataOutputStream dataOS = null;
DataInputStream dataIS = null;
private boolean handshakeSuccessful = false;
public static void main(String[] args)
{
SSLSocketTest sslSocketTest = new SSLSocketTest();
sslSocketTest.sslSocketConnect();
}
SSLSocketTest()
{
System.setProperty("javax.net.debug", "all");
try{
File certFile = new File("cacerts");
KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
char[] certPassword = "changeit".toCharArray();
InputStream fileIS = new FileInputStream(certFile);
keyStore.load(fileIS, certPassword);
fileIS.close();
SSLContext sslContext = SSLContext.getInstance("TLSv1");
TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trustManagerFactory.init(keyStore);
X509TrustManager defaultTrustManager = (X509TrustManager)trustManagerFactory.getTrustManagers()[0];
sslContext.init(null, new TrustManager[] {defaultTrustManager}, null);
sslSocketFactory = sslContext.getSocketFactory();
}catch(Exception e){
e.printStackTrace();
}
}
public void sslSocketConnect()
{
try{
sslSocket = (SSLSocket) sslSocketFactory.createSocket(ipAddress, port);
dataOS = new DataOutputStream(sslSocket.getOutputStream());
dataIS = new DataInputStream(sslSocket.getInputStream());
sslSocket.setSoTimeout(15000);
//Handshake
sslSocket.addHandshakeCompletedListener(new MyHandshakeListener());
sslSocket.startHandshake();
while(!handshakeSuccessful)
{
Thread.sleep(100);
}
//Sending commands
byte[] firstCommand = new byte[]{(byte)0x01, (byte)0x03, (byte)0x03};
String[] firstCommandResponse = processCommand(firstCommand);
byte[] secondCommand = new byte[]{(byte)0x01, (byte)0x48, (byte)0x65, (byte)0x6C, (byte)0x6C, (byte)0x6F};
String[] secondCommandResponse = processCommand(secondCommand);
disconnect();
}catch(Exception e){
e.printStackTrace();
}
}
public void disconnect()
{
try{
byte[] endConnection = new byte[]{(byte)0x01, (byte)0x01, (byte)0x02, (byte)0x03};
processCommand(endConnection);
dataOS.close();
dataIS.close();
sslSocket.close();
}catch (Exception e){
e.printStackTrace();
}
}
public String[] processCommand(byte[] command)
{
String[] returnResponse = null;
byte[] commandResponse = new byte[120];
byte[] trimCommandResponse;
try{
int commandResponseLength = -1;
int errorCount = 0;
while(commandResponseLength == -1)
{
StringBuilder cmdStr = new StringBuilder();
cmdStr.append("Sending: ");
for(int i=0; i<command.length; i++)
{
cmdStr.append(fixHexStringData(Integer.toHexString(command[i])) + " ");
}
System.out.println(cmdStr.toString());
dataOS.write(command, 0, command.length);
dataOS.flush();
commandResponseLength = dataIS.read(commandResponse);
errorCount++;
if(errorCount == 3)
{
throw new Exception();
}
}
returnResponse = new String[commandResponseLength];
trimCommandResponse = new byte[commandResponseLength];
//Getting Reponse Data
for(int i=0; i<commandResponseLength; i++)
{
returnResponse[i] = fixHexStringData(Integer.toHexString(commandResponse[i]));
trimCommandResponse[i] = commandResponse[i];
}
StringBuilder rcvStr = new StringBuilder();
rcvStr.append("Receive: ");
for(int i=0; i<returnResponse.length; i++)
{
rcvStr.append(returnResponse[i] + " ");
}
System.out.println(rcvStr.toString());
}catch(Exception e){
e.printStackTrace();
}
return returnResponse;
}
private String fixHexStringData(String dataByte)
{
if(dataByte.length() < 2)
{
dataByte = "0" + dataByte;
}
else if(dataByte.length() > 2)
{
dataByte = dataByte.substring(dataByte.length()-2);
}
return dataByte;
}
class MyHandshakeListener implements HandshakeCompletedListener
{
public void handshakeCompleted(HandshakeCompletedEvent e)
{
System.out.println("Handshake succesful!");
handshakeSuccessful = true;
}
}
}
The questions I have are:
Am I missing a step in writing out the byte array? I have done this over a standard Java Socket with no issues, so is writing over a SSL Socket different from a standard Socket? I looked for this but did not see anything different.
Could this be a certificate issue? If the handshake and the first command are successful, would this mean that the communication has been established at this point and it is beyond the certificate?
Could the server be affecting this? If so, what could be the reason behind this? If writing the byte array to the DataOutputStream is on the client side and the first byte is dropped, how could the server have any affect on the client side?
Could this be a JVM bug?
It looks like this is actually a function of the JSSE implementation which splits the data between two packets. The first byte goes in one packet and the rest in the next. More detail at this answer
You should be able to override this functionality by including
System.setProperty("jsse.enableCBCProtection", "false");
Related
Hope you are doing well.
The strange thing happened to me since yesterday.
I have the following code for saving the JSON.
public static boolean saveCacheJson(String pathToCache, JSONObject json) {
boolean isSaveSuccess = false;
ObjectOutputStream outputStream = null;
FileOutputStream fileOutputStream = null;
try {
File file = new File(pathToCache);
if (!file.exists()) {
file.getParentFile().mkdirs();
new FileWriter(file);
} else {
file.delete();
}
fileOutputStream = new FileOutputStream(pathToCache);
outputStream = new ObjectOutputStream(fileOutputStream);
outputStream.writeObject(json.toString());
isSaveSuccess = true;
} catch (IOException e) {
IsoGame.$().crossPlatformManager.getCrossPlatformUtilsInstance().sendLog(e);
e.printStackTrace();
} finally {
try {
outputStream.close();
} catch (Exception e1) {
IsoGame.$().crossPlatformManager.getCrossPlatformUtilsInstance().sendLog(e1);
e1.printStackTrace();
}
try {
fileOutputStream.close();
} catch (Exception e1) {
IsoGame.$().crossPlatformManager.getCrossPlatformUtilsInstance().sendLog(e1);
e1.printStackTrace();
}
}
return isSaveSuccess;
}
and the following code for reading it.
public static JSONObject getCacheJson(String pathToCache, boolean throwException) throws Exception {
JSONObject result = null;
String resultString;
ObjectInputStream inputStream = null;
FileInputStream fileInputStream = null;
try {
fileInputStream = new FileInputStream(pathToCache);
inputStream = new ObjectInputStream(fileInputStream);
resultString = (String) inputStream.readObject();
result = new JSONObject(resultString);
} catch (ClassNotFoundException | IOException | JSONException e) {
e.printStackTrace();
if (throwException && e instanceof FileNotFoundException) {
throw e;
}
} finally {
try {
if (inputStream != null) {
inputStream.close();
}
} catch (Exception e1) {
e1.printStackTrace();
}
try {
if (fileInputStream != null) {
fileInputStream.close();
}
} catch (Exception e1) {
e1.printStackTrace();
}
}
return result;
}
I'm writing the JSON via saveCacheJson and then putting the file to my Server (Linux) and then my 'front end' part downloads it and tries to read.
From yesterday I started receiving the following exception.
java.io.StreamCorruptedException: invalid stream header:
After some research and trying to understand what's going on I found the following.
There is a MAGIC_NUMBER in the ObjectOutputStream class, that used for a header.
protected void writeStreamHeader() throws IOException {
this.bout.writeShort(-21267);
this.bout.writeShort(5);
}
Then, in the ObjectInputStream class the following method is called.
protected void readStreamHeader() throws IOException, StreamCorruptedException {
short var1 = this.bin.readShort();
short var2 = this.bin.readShort();
if (var1 != -21267 || var2 != 5) {
throw new StreamCorruptedException(String.format("invalid stream header: %04X%04X", var1, var2));
}
}
And the exception is thrown here.
So, I opened the file that I write on my local machine (Mac OS) and found the first 2 bytes that are the following
¨Ì
Then I tried the following in the terminal.
hexdump myfile
and found that the first 2 bytes are right.
Here is it.
ac ed ...
So, I tried to do the same for the downloaded file (from Linux server, that I put there before) and found that the first 2 bytes are wrong.
So, I checked the flow and found that they have changed during copying the file to the server.
This is strange overall, but the strangest thing is that it worked for about 10 days ago. As I remember, nothing changed in my server.
So, my main question is
ANY IDEAS WHY THE STREAM HEADER IS CHANGED DURING UPLOAD TO LINUX SERVER???
Thanks.
UPDATE
The first 2 shorts in the file BEFORE UPLOAD are.
ac ed 00 05 ...
Which is right.
The first 2 shorts in the file AFTER UPLOADING to linux from Mac are.
bfef efbd ...
Which is wrong.
UPDATE 2.
The result from the right file.
0000000 ac ed 00 05 7c 00 00 00 00 00 0f 26 fe 7b 22 64
0000010 65 22 3a 7b 22 6e 70 63 22 3a 5b 7b 22 63 6f 64
0000020 65 22 3a 22 64 65 22 2c 22 6e 70 63 5f 69 64 22
0000030 3a 32 2c 22 6c 61 6e 67 5f 69 64 22 3a 36 2c 22
0000040 69 64 22 3a 31 32 2c 22 6c 61 6e 67 75 61 67 65
0000050 5f 69 64 22 3a 36 2c 22 64 69 61 6c 6f 67 73 22
0000060 3a 5b 22 53 63 68 61 75 20 6d 61 6c 2c 20 64 61
0000070 73 20 69 73 74 20 49 72 69 73 2c 20 64 65 72 20
0000080 42 6f 74 65 20 64 65 72 20 47 c3 b6 74 74 65 72
0000090 2e 20 53 69 65 20 68 61 74 20 75 6e 73 20 6d 61
..........
And from the Wrong file.
0000000 ef bf bd ef bf bd 00 05 7c 00 00 00 00 00 0f 26
0000010 ef bf bd 7b 22 64 65 22 3a 7b 22 6e 70 63 22 3a
0000020 5b 7b 22 63 6f 64 65 22 3a 22 64 65 22 2c 22 6e
0000030 70 63 5f 69 64 22 3a 32 2c 22 6c 61 6e 67 5f 69
0000040 64 22 3a 36 2c 22 69 64 22 3a 31 32 2c 22 6c 61
0000050 6e 67 75 61 67 65 5f 69 64 22 3a 36 2c 22 64 69
0000060 61 6c 6f 67 73 22 3a 5b 22 53 63 68 61 75 20 6d
0000070 61 6c 2c 20 64 61 73 20 69 73 74 20 49 72 69 73
0000080 2c 20 64 65 72 20 42 6f 74 65 20 64 65 72 20 47
0000090 c3 b6 74 74 65 72 2e 20 53 69 65 20 68 61 74 20
It looks like whatever you use to transfer the data from Mac to Linux is attempting to do some kind of re-encoding, possibly involving UTF-8 or UTF-16, but I cannot identify exactly what it's trying to do.
There is also a byte-order issue in that it's reversing the order of bytes in 16-bit words. Note on the first line of the correct hex the sequence 0x0f26. In the garbled file the sequence is 0x260f.
I suggest you carefully examine the process used to transfer the file from Mac to Linux and try to find any options controlling the file encoding and byte order.
UPDATE
I figured it out. First, the converter is deciding that some characters are invalid and replacing them with the Unicode code point 0xfffd or Replacement Character. Then, it is encoding the resulting stream as UTF-8. The encoding of 0xfffd in UTF-8 is 0xefbfbd.
AC ED --(replace)-> FFFD FFFD --(utf8 encode)-> EF BF BD EF BF BD
Whatever process is handling the transfer is assuming the input is UTF-8, which is incorrect, the data stream is binary. Byte values 0xAC and 0xED are not valid in UTF-8, so they are being replaced with the Unicode Replacement Character, encoded in UTF-8. Other binary values that happen to represent invalid UTF-8 bytes are also being replaced.
OK, I found what causes the problem. Not sure why yet, but found how to solve.
I used mypath/myfile.txt as a pathToCache parameter for the saveCacheJson method. So, I found that the problem because of the .txt extension.
I removed an extension at all and the problem resolved.
It seems the .txt extension is the reason, that Mac's system doesn't understand that the file is binary and trying to encode its content during copying.
Not sure why the system trying to change the file's content at all during the copy ))
when I want to send byte array on serial port stream with java,on destination device I receive different result !!!
byte[] sendingPack = new byte[7];
sendingPack[0] = 0x6E;
sendingPack[1] = 0x55;
sendingPack[2] = (byte) 0x0D;
sendingPack[3] = (byte) (1 & 0x000000FF);
sendingPack[4] = 0x01;
sendingPack[5] = 0x0D;
sendingPack[6] = (byte) 0xAA;
getSendBuffer().getOutputStream().write(sendingPack);
sending array : byte[]{0x6E,0x55,0x0D,0x01,0x01,0x0D,0xAA}
receive result array : 6E 55 0D 0A 01 01 0D 0A AA
on CodeVisionAVR terminal I receive "0A"!!
how can I solve this problem??
The terminal is probably in text reading mode and not in binary read mode.
The 0x0A which is inserted after every 0x0D you send is a carriage return conversion.
The terminal converts "\r" to "\r\n". It adds a line feed char to every carriage return.
The terminal converts every 0D to 0D 0A.
This same feature can be found in the ftp protocol. You tell your client how to transfer files: in text or binary mode.
I have a very strange situation. I connect my Java software with a device, let´s call it "Black Box" (because I cannot look into it or make traces within it). I am adressing a specific port (5550) and send commands as byte sequences on a socket. As a result, I get an answer from the Black Box on the same socket.
Both my commands and the replies are prefixed in a pre-defined way (according to the API) and have an XOR checksum.
When I run the code from Windows, all is fine: Command 1 gets its Reply 1 and Command 2 gets its Reply 2.
When I run the code from Android (which is actually my target - Windows came into play to track down the error) it gets STRANGE: Command 1 gets its Reply 1 but Command 2 does not. When I play with Command 2 (change the prefix illegally, violate the checksum) the Black Box reacts as expected (with an error reply). But with the correct Command 2 being issued from Android, the Reply is totally mis-formed: Wrong prefix and missing checksum.
In the try to analyse the error I tried WireShark and this shows that on the network interface, the Black Box is sending the RIGHT Reply 2, but evaluating this reply in Java from the socket, it is wrong. How can this be when all is fine for Command/Reply 1???
Strange is, that parts of the expected data are present:
Expected: ff fe e4 04 00 11 00 f1
Received: fd fd fd 04 00 11 00 // byte 8 missing
I am attaching the minimalistic code to force the problem. What could falsify the bytes which I receive? Is there a "raw" access in Java to the socket which could reveal the problem?
I am totally confused so any help would be appreciated:
String address = "192.168.1.10";
int port = 5550;
Socket socket;
OutputStream out;
BufferedReader in;
try {
socket = new Socket(address, port);
out = socket.getOutputStream();
in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
// This is "Command 1" which is receiving the right reply
// byte[] allesAn = new byte[] {(byte)0xff, (byte)0xfe, (byte)0x21, (byte)0x81, (byte)0xa0};
// out.write(allesAn);
// This is "Command 2" which will not receive a right reply
byte[] getLokInfo3 = new byte[] {(byte)0xff, (byte)0xfe, (byte)0xe3, (byte)0, (byte)0, (byte)3, (byte)0xe0};
out.write(getLokInfo3);
out.flush();
while (true) {
String received = "";
final int BufSize = 1000;
char[] buffer = new char[BufSize];
int charsRead = 0;
charsRead = in.read(buffer, 0, BufSize);
// Convert to hex presentation
for (int i=0; i < charsRead; i++) {
byte b = (byte)buffer[i];
received += hexByte((b + 256) % 256) + " ";
}
String result = charsRead + ">" + received + "<";
Log.e("X", "Read: " + result);
}
} catch (Exception e) {
Log.e("X", e.getMessage() + "");
}
with
private static String hexByte(int value) {
String s = Integer.toHexString(value);
return s.length() % 2 == 0 ? s : "0" + s;
}
Here is what wireshark says, showing the expected 8 bytes:
Below, you see a program that I wrote to see the state of different fields and memory allocations after calling requestObjectDeletion() method:
public class ReqObjDel extends Applet {
static byte[] buffer = new byte[2];
static boolean isNull = false;
private ReqObjDel() {
}
public static void install(byte bArray[], short bOffset, byte bLength)
throws ISOException {
new ReqObjDel().register();
}
public void process(APDU arg0) throws ISOException {
if (selectingApplet()) {
return;
}
if (buffer != null && (short) buffer.length == (short) 10) {
return;
}
byte[] oldBuffer = buffer;
buffer = new byte[10];
JCSystem.requestObjectDeletion();
if (oldBuffer == null)
isNull = true;
if (isNull) {
ISOException.throwIt((short) 0x1111);
} else
ISOException.throwIt((short) 0x0000);
}
}
As far as I know,this method reclaims memory which is being used by “unreachable” objects. To be “unreachable”, an object can neither be pointed to by a static field nor by an object field. So calling requestObjectDeletion() in the above program reclaims the part of EEPROM that oldBuffer is refer to (As far as I know, oldBuffer is neither class field nor object field,right?). In this situation I expect that oldBuffer == null and therefore the JCRE must return 0x1111. But the output is 0x0000 unexpectedly :
OSC: opensc-tool -s 00a404000b0102030405060708090000 -s 00000000
Using reader with a card: ACS CCID USB Reader 0
Sending: 00 A4 04 00 0B 01 02 03 04 05 06 07 08 09 00 00
Received (SW1=0x90, SW2=0x00)
Sending: 00 00 00 00
Received (SW1=0x00, SW2=0x00)
Q1 : What can I conclude?
That part of memory is not reclaimed?
That part of memory is reclaimed but oldBuffer is a reference to it still?
something else?
Q2 : Is there any way to obtain the free memory size before and after of calling this method? (i.e. is there any method that return the size of free memory[not allocated]?)
Update 1 : Trying JCSystem.getAvailableMemory()
Based on #vojta answer, I changed my program in a way that the line byte[] oldBuffer = buffer; runs only once (using a flag named isFirstInvocation) and return the free available memory in two consecutive process() method invocation :
public class ReqObjDel extends Applet {
static byte[] buffer = new byte[10];
static boolean isFirstInvocation = true;
private ReqObjDel() {
}
public static void install(byte bArray[], short bOffset, byte bLength)
throws ISOException {
new ReqObjDel().register();
}
public void process(APDU arg0) throws ISOException {
if (selectingApplet()) {
return;
}
short availableMem1 = JCSystem
.getAvailableMemory(JCSystem.MEMORY_TYPE_PERSISTENT);
if (isFirstInvocation) {
byte[] oldBuffer = buffer;
buffer = new byte[10];
JCSystem.requestObjectDeletion();
firstInvocation = false;
}
short availableMem2 = JCSystem
.getAvailableMemory(JCSystem.MEMORY_TYPE_PERSISTENT);
short availableMemory = (short) (availableMem1 + availableMem2);
ISOException.throwIt(availableMemory);
}
}
And this is the output :
OSC: osc -s 00a404000b0102030405060708090000 -s 00000000 -s 00000000
Using reader with a card: ACS CCID USB Reader 0
Sending: 00 A4 04 00 0B 01 02 03 04 05 06 07 08 09 00 00
Received (SW1=0x90, SW2=0x00)
Sending: 00 00 00 00
Received (SW1=0xFF, SW2=0xFE)
Sending: 00 00 00 00
Received (SW1=0xFF, SW2=0xFE)
As both invocations return an equal value, I think the JCRE reclaims that part of memory immediately after calling requestObjectDeletion(), right?
First of all, a rule based on my personal experience: if possible, do not use the garbage collector at all. GC is very slow and could be even dangerous (see Javacard - power loss during garbage collection).
Q1:
If you really have to use GC, read the documentation:
This method is invoked by the applet to trigger the object deletion
service of the Java Card runtime environment. If the Java Card runtime
environment implements the object deletion mechanism, the request is
merely logged at this time. The Java Card runtime environment must
schedule the object deletion service prior to the next invocation of
the Applet.process() method.
Shortly speaking, JCSystem.requestObjectDeletion(); has no immediate effect. That is why your local variable oldBuffer remains unchanged.
Q2: To find out how much persistent memory is available to your applet, use:
JCSystem.getAvailableMemory(JCSystem.MEMORY_TYPE_PERSISTENT)
ANSWER to UPDATE 1: JCSystem.getAvailableMemory(JCSystem.MEMORY_TYPE_PERSISTENT) may be confusing for cards with more than 32767 bytes of persistent memory. Such cards usually offer their own proprietary ways to find out available memory.
If the number of available bytes is greater than 32767, then this
method returns 32767.
I want to run the following command from a java program:
socat -x -u /dev/ttyFTDI0,raw,echo=0,crnl /dev/ttyFTDI1,raw,echo=0,crnl
This program is supposed to run for an indefinite time while outputting hex strings like this:
b4 03 03 92 00 01 3f c6 b4 03 10 03 00 01 6a af
My current testing code is:
public void testOutput() {
try {
List<String> command = new ArrayList<String>();
command.add("socat");
command.add("-x");
command.add("-u");
command.add("/dev/ttyFTDI0,raw,echo=0,crnl");
command.add("/dev/ttyFTDI1,raw,echo=0,crnl");
ProcessBuilder proBui = new ProcessBuilder(command);
Process process = proBui.start();
//process.waitFor();
BufferedReader r = new BufferedReader(new InputStreamReader(process.getInputStream()));
while (true) {
String temp = r.readLine();
if (temp != null && !temp.isEmpty()) {
//do something with data
System.out.print(temp);
}
Thread.sleep(1);
}
} catch (Exception ex) {
System.out.println(ex);
}
}
While this code works for commands that finish after a given time, commands like the one given or watch ls won't work.
According to the socat documentation, the -x option
Writes the transferred data not only to their target streams, but also to stderr.
You are reading from process.getInputStream() which is connected to the native process stdout - to get stderr you should read process.getErrorStream() instead.