We Keep Coding

Python byte object vs java

Good morning, I have a quick question regarding the differences between a byte object in python (denoted b'') and how to replicate it in java.
The project I am working on is some personal work on an emulation server for a dead game to better my reversing skills. I have a working rendition of the project in python, but would like to switch over to java as I am better with the language and it comes with many additional tools included that are useful for a project like this.
I am using a ServerSocket to capture TCP data in the java project.
When data comes over the network from the Python project it looks a little something like this:
When I capture the same data over the java ServerSocket I get something like this:
My question is how can I reformat this ASCII text to get the proper data as seen in the python version of the software.
Currently I am able to get an output like this:
By converting the byte[] data from the ServerSocket as such
while(true) {
try {
Socket socket = serverSocket.accept();
onConnection(socket);
byte[] incomingData = new byte[0];
byte[] temp = new byte[1024];
int k = -1;
//this is due to the client of said game not sending EOL (readLine() does not work here)
while((k = socket.getInputStream().read(temp, 0, temp.length)) > -1) {
byte[] tbuff = new byte[incomingData.length + k];
System.arraycopy(incomingData, 0, tbuff, 0, incomingData.length);
System.arraycopy(temp, 0, tbuff, incomingData.length, k);
incomingData = tbuff;
receiveData(socket, incomingData); <--- this is the important bit
}
} catch (IOException e) {
e.printStackTrace();
}
}
public void receiveData(Socket socket, byte[] data) {
int lenLo = (int) (data[0]);
int lenHi = (int) (data[1]);
int length = lenHi * 256 + lenLo;
if(lenHi < 0) {
System.out.println("Invalid Packet Length");
}
if(data.length != length) {
System.out.println("Incomplete Packet Received");
}
try {
String test = new String(data, "UTF-8");
serverGUI.serverDebug(test); //produces the string in a jframe (pic 2)
serverGUI.debugByteArray(test.getBytes(StandardCharsets.UTF_8)); //produces the byte[] in jframe (pic 3 -- all bytes in this array are & 0xff prior to being printed out)
} catch (UnsupportedEncodingException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
}
However this has obviously not produced the desired outcome. Any advice is appreciated or any resources that can be put forth are also appreciated.
Thanks in advance!

How to read (all available) data from serial connection when using JSSC?

I'm trying to work with JSSC.
I built my app according to this link:
https://code.google.com/p/java-simple-serial-connector/wiki/jSSC_examples
My event handler looks like:
static class SerialPortReader implements SerialPortEventListener {
public void serialEvent(SerialPortEvent event) {
if(event.isRXCHAR()){//If data is available
try {
byte buffer[] = serialPort.readBytes();
}
catch (SerialPortException ex) {
System.out.println(ex);
}
}
}
}
}
The problem is that I'm always not getting the incoming data in one piece. (I the message has a length of 100 bytes, Im getting 48 and 52 bytes in 2 separates calls)
- The other side send me messages in different lengths.
- In the ICD Im working with, there is a field which tell us the length of the message. (from byte #10 to byte #13)
- I cant read 14 bytes:
(serialPort.readBytes(14);,
parse the message length and read the rest of the message:
(serialPort.readBytes(messageLength-14);
But if I will do it, I will not have the message in once piece (I will have 2 separates byte[] and I need it in one piece (byte[]) without the work of copy function.
Is it possible ?
When working with Ethernet (SocketChannel) we can read data using ByteBuffer. But with JSSC we cant.
Is there a good alternative to JSSC ?
Thanks

You can't rely on any library to give you all the content you need at once because :
the library dont know how many data you need
the library will give you data as it comes and also depending on buffers, hardware, etc
You must develop your own business logic to handle your packets reception. It will of course depend on how your packets are defined : are they always the same length, are they separated with same ending character, etc.
Here is an example that should work with your system (note you should take this as a start, not a full solution, it doesn't include timeout for example) :
static class SerialPortReader implements SerialPortEventListener
{
private int m_nReceptionPosition = 0;
private boolean m_bReceptionActive = false;
private byte[] m_aReceptionBuffer = new byte[2048];
#Override
public void serialEvent(SerialPortEvent p_oEvent)
{
byte[] aReceiveBuffer = new byte[2048];
int nLength = 0;
int nByte = 0;
switch(p_oEvent.getEventType())
{
case SerialPortEvent.RXCHAR:
try
{
aReceiveBuffer = serialPort.readBytes();
for(nByte = 0;nByte < aReceiveBuffer.length;nByte++)
{
//System.out.print(String.format("%02X ",aReceiveBuffer[nByte]));
m_aReceptionBuffer[m_nReceptionPosition] = aReceiveBuffer[nByte];
// Buffer overflow protection
if(m_nReceptionPosition >= 2047)
{
// Reset for next packet
m_bReceptionActive = false;
m_nReceptionPosition = 0;
}
else if(m_bReceptionActive)
{
m_nReceptionPosition++;
// Receive at least the start of the packet including the length
if(m_nReceptionPosition >= 14)
{
nLength = (short)((short)m_aReceptionBuffer[10] & 0x000000FF);
nLength |= ((short)m_aReceptionBuffer[11] << 8) & 0x0000FF00;
nLength |= ((short)m_aReceptionBuffer[12] << 16) & 0x00FF0000;
nLength |= ((short)m_aReceptionBuffer[13] << 24) & 0xFF000000;
//nLength += ..; // Depending if the length in the packet include ALL bytes from the packet or only the content part
if(m_nReceptionPosition >= nLength)
{
// You received at least all the content
// Reset for next packet
m_bReceptionActive = false;
m_nReceptionPosition = 0;
}
}
}
// Start receiving only if this is a Start Of Header
else if(m_aReceptionBuffer[0] == '\0')
{
m_bReceptionActive = true;
m_nReceptionPosition = 1;
}
}
}
catch(Exception e)
{
e.printStackTrace();
}
break;
default:
break;
}
}
}

After writing data to serial port it need to be flushed. Check the timing and pay attention to the fact that read should occur only after other end has written. read size is just an indication to read system call and is not guaranteed. The data may have arrived and is buffered in serial port hardware buffer but may not have been transferred to operating system buffer hence not to application. Consider using scm library, it flushes data after each write http://www.embeddedunveiled.com/

Try this:
Write your data to the serial port (using serialPort.writeBytes()) and if you are expecting a response, use this:
byte[] getData() throws SerialPortException, IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] b;
try {
while ((b = serialPort.readBytes(1, 100)) != null) {
baos.write(b);
// System.out.println ("Wrote: " + b.length + " bytes");
}
// System.out.println("Returning: " + Arrays.toString(baos.toByteArray()));
} catch (SerialPortTimeoutException ex) {
; //don't want to catch it, it just means there is no more data to read
}
return baos.toByteArray();
}
Do what you want with the returned byte array; in my case I just display it for testing.
I found it works just fine if you read one byte at a time, using a 100ms timeout, and when it does time out, you've read all data in the buffer.
Source: trying to talk to an Epson serial printer using jssc and ESC/POS.

IndexOutOfBoundException when reading a file

I am working on a TFTP server application. I managed to process a successful file transfer from server to client however the other way around is bugged.
Client instead of transmitting the entire file simply terminated whit compiler returning no errors. Debugger shows IOBE exception on the marked code referring that the array is out of range.
The whole transfer process goes like so:
Client transmits a file name and requested operation WRQ - Write Request
Server received the packet and determines the operation if WRQ is gives the new file appropriate name.
Server now starts executing receiveData() until it gets a packet < 512 indicationg EOT
Client keeps transferring data it read from the file.
Key code:
Client:
private void sendWRQ() throws Exception
{
String rrq = "WRQ-" + data;
outgoingData = rrq.getBytes();
DatagramPacket output = new DatagramPacket(outgoingData, outgoingData.length, serverAddress, serverPort);
clientSocket.send(output);
//Thread.sleep(50);
sendData();
}
byte outgoingData = new byte[512];
private void sendData() throws Exception
{
DatagramPacket dataTransfer = new DatagramPacket(outgoingData, outgoingData.length, serverAddress, serverPort);
InputStream fis = new FileInputStream(new File(data));
int x;
while((x = fis.read(outgoingData,0,512)) != -1) // << Debugged gives IOBE
{
dataTransfer.setLength(x);
clientSocket.send(dataTransfer);
Thread.sleep(5);
}
fis.close();
}
Server:
private void listen() throws Exception
{
DatagramPacket incTransfer = new DatagramPacket(incomingData, incomingData.length);
serverSocket.receive(incTransfer);
clientAddress = incTransfer.getAddress();
clientPort = incTransfer.getPort();
String output = new String(incTransfer.getData());
if(output.substring(0, 3).equals("RRQ"))
{
File test = new File(output.substring(4));
responseData = output.substring(4);
if(test.exists())
{
sendResponse("Y");
} else {
sendResponse("N");
}
} else if (output.substring(0, 3).equals("WRQ"))
{
File test = new File(output.substring(4));
if(test.exists())
{
Calendar cal = Calendar.getInstance();
SimpleDateFormat prefix = new SimpleDateFormat(date_format);
String date = prefix.format(cal.getTime()).toString();
responseData = date + output.substring(4);
receiveData();
} else {
responseData = output.substring(4);
receiveData();
}
}
}
private void receiveData() throws Exception
{
DatagramPacket receiveData = new DatagramPacket(incomingData, incomingData.length);
OutputStream fos = new FileOutputStream(new File(responseData));
while(true)
{
serverSocket.receive(receiveData);
if(receiveData.getLength() == 512)
{
fos.write(receiveData.getData());
} else {
fos.write(receiveData.getData(), receiveData.getOffset(), receiveData.getLength());
break;
}
}
fos.close();
}

The only way that can happen is if the offset or length parameters violate the constraints specified for InputStream.read(byte[], int, int); in this case probably the buffer isn't 512 bytes long. There's no need to specify the 2nd nd third parameters in this case, just omit them, then it becomes read(buffer, 0, buffer.length) internally, which can't be wrong.

Okay, the way this is coded, the 'outgoingData' field is:
1) Initialized to a length of 512
2) Then, in sendWRQ(), 'outgoingData' is re-initialized to whatever rrq.getBytes() sends back.
3) Then, in sendData(), 'outgoingData' is used as the intermediate buffer to read data from file and put it in the dataTransfer object.
However, since 'outgoingData' is re-initialized in step #2, the assumption in step #3 that 'outgoingData' is still 512 bytes in length is false.
So while EJP was correct in saying that using read(outgoingData, 0, outgoingData.length()) will work, there are some architecture issues that if you address, you'll clean up a lot of potential errors.
For instance:
WIth the code provided, there is seemingly no reason to have outgoingData declared at the class level and shared among two functions. Depending on the rest of the app, this could end up being a Threading issue.
Perhaps byte[] buffer = rrq.getBytes(); in sendWRQ() and byte[] buffer = new byte[1024]; in sendData().
Also, the 'data' parameter is at the class level.... for what reason? Might be better able to be controlled if its a passed in parameter.
Lastly, I've had good luck using the do{} while() loop in network situations. Ensures that the send() gets at least one chance to send the data AND it keeps the code a bit more readable.

Is there a difference in Java's writeInt when executed on Windows vs an Intel based Mac

I currently writing a Java TCP server to handle the communication with a client (which I didn't write). When the server, hosted on windows, responds to the client with the number of records received the client doesn't read the integer correctly, and instead reads it as an empty packet. When the same server code, hosted on my Mac, responds to the client with the number of records received the client reads the packet and responds correctly. Through my research I haven't found an explanation that seems to solve the issue. I have tried reversing the bytes (Integer.reverseBytes) before calling the writeInt method and that didn't seem to resolve the issue. Any ideas are appreciated.
Brian
After comparing the pcap files there are no obvious differences in how they are sent. The first byte is sent followed by the last 3. Both systems send the correct number of records.
Yes I'm referring to the DataOutputStream.writeInt() method. //Code added
public void run() {
try {
InputStream in = socket.getInputStream();
DataOutputStream datOut = new DataOutputStream(socket.getOutputStream());
datOut.writeByte(1); //sends correctly and read correctly by client
datOut.flush();
//below is used to read bytes to determine length of message
int bytesRead=0;
int bytesToRead=25;
byte[] input = new byte[bytesToRead];
while (bytesRead < bytesToRead) {
int result = in.read(input, bytesRead, bytesToRead - bytesRead);
if (result == -1) break;
bytesRead += result;
}
try {
inputLine = getHexString(input);
String hexLength = inputLine.substring(46, 50);
System.out.println("hexLength: " + hexLength);
System.out.println(inputLine);
//used to read entire sent message
bytesRead = 0;
bytesToRead = Integer.parseInt(hexLength, 16);
System.out.println("bytes to read " + bytesToRead);
byte[] dataInput = new byte[bytesToRead];
while (bytesRead < bytesToRead) {
int result = in.read(dataInput, bytesRead, bytesToRead - bytesRead);
if (result == -1) break;
bytesRead += result;
}
String data = getHexString(dataInput);
System.out.println(data);
//Sends received data to class to process
ProcessTel dataValues= new ProcessTel(data);
String[] dataArray = new String[10];
dataArray = dataValues.dataArray();
//assigns returned number of records to be written to client
int towrite = Integer.parseInt(dataArray[0].trim());
//Same write method on Windows & Mac...works on Mac but not Windows
datOut.writeInt(towrite);
System.out.println("Returned number of records: " + Integer.parseInt(dataArray[0].trim()) );
datOut.flush();
} catch (Exception ex) {
Logger.getLogger(ServerThread.class.getName()).log(Level.SEVERE, null, ex);
}
datOut.close();
in.close();
socket.close();
} catch (IOException e) {
e.printStackTrace();
}
}

As described in its Javadoc, DataOutputStream.writeInt() uses network byte order as per the TCP/IP RFCs. Is that the method you are referring to?

No, x86 processors only support little-endian byte order, it doesn't vary with the OS. Something else is wrong.
I suggest using wireshark to capture the stream from a working Mac server and a non-working Windows server and compare.

Some general comments on your code:
int bytesRead=0;
int bytesToRead=25;
byte[] input = new byte[bytesToRead];
while (bytesRead < bytesToRead) {
int result = in.read(input, bytesRead, bytesToRead - bytesRead);
if (result == -1) break;
bytesRead += result;
}
This EOF handling is hokey. It means that you don't know whether or not you've actually read the full 25 bytes. And if you don't, you'll assume that the bytes-to-send is 0.
Worse, you copy-and-paste this code lower down, relying on proper initialization of the same variables. If there's a typo, you'll never know it. You could refactor it into its own method (with tests), or you could call DataInputStream.readFully().
inputLine = getHexString(input);
String hexLength = inputLine.substring(46, 50);
You're converting to hex in order to extract an integer? Why? And more important, if you have any endianness issues this is probably the reason
I was originally going to recommend using a ByteBuffer to extract values, but on a second look I think you should wrap your input stream with a DataInputStream. That would allow you to read complete byte[] buffers without the need for a loop, and it would let you get rid of the byte-to-hex-to-integer conversions: you'd simply call readInt().
But, continuing on:
String[] dataArray = new String[10];
dataArray = dataValues.dataArray();
Do you realize that the new String[10] is being thrown away by the very next line? Is that what you want?
int towrite = Integer.parseInt(dataArray[0].trim());
datOut.writeInt(towrite);
System.out.println("Returned number of records: " + Integer.parseInt(dataArray[0].trim()) );
If you're using logging statements, print what you're actually using (towrite). Don't recalculate it. There's too much of a chance to make a mistake.
} catch (Exception ex) {
Logger.getLogger(ServerThread.class.getName()).log(Level.SEVERE, null, ex);
}
// ...
} catch (IOException e) {
e.printStackTrace();
}
Do either or both of these catch blocks get invoked? And why do they send their output to different places? For that matter, if you have a logger, why are you inserting System.out.println() statements?

Sending an int from Java to C using sockets

I was just wondering how to send an int from a Java application to a C application using sockets. I have got different C programs communicating with each other and have got the Java application retrieving data from the C application, but I can't work out sending.
The C application is acting as database, the Java application then sends a user id (a 4 digit number) to the C application, if it exists it returns that record's details.
In Java I have tried using a printWriter and DataOutputStream to send the data, printWriter produces weird symbols and DataOutputStream produces "prof_agent.so".
Any help would be appreciated as I don't have a good grasp of sockets at the moment.

You can use DataOutputStream.writeInt. It writes an int already in network byte order by contract.
On a C side you can call recv, or read to fill in the 4-byte buffer, and then you can use ntohl ( Network-TO-Host-Long ) to convert the value you've just read to your platform int representation.

You can send the textual representation. So the number 123 would be sent as 3 bytes '1' '2' '3'.

It's a bit too late but let this answer be here. Using UDP sockets:
Java code:
public void runJavaSocket() {
System.out.println("Java Sockets Program has started."); int i=0;
try {
DatagramSocket socket = new DatagramSocket();
System.out.println("Sending the udp socket...");
// Send the Message "HI"
socket.send(toDatagram("HI",InetAddress.getByName("127.0.0.1"),3800));
while (true)
{
System.out.println("Sending hi " + i);
Thread.currentThread();
Thread.sleep(1000);
socket.send(toDatagram("HI " + String.valueOf(i),InetAddress.getByName("127.0.0.1"),3800));
i++;
}
}
catch (Exception e)
{
e.printStackTrace();
}
}
public DatagramPacket toDatagram(
String s, InetAddress destIA, int destPort) {
// Deprecated in Java 1.1, but it works:
byte[] buf = new byte[s.length() + 1];
s.getBytes(0, s.length(), buf, 0);
// The correct Java 1.1 approach, but it's
// Broken (it truncates the String):
// byte[] buf = s.getBytes();
return new DatagramPacket(buf, buf.length,
destIA, destPort);
}
C# code:
string returnData;
byte[] receiveBytes;
//ConsoleKeyInfo cki = new ConsoleKeyInfo();
using (UdpClient udpClient = new UdpClient(new IPEndPoint(IPAddress.Parse("127.0.0.1"), 3800)))
{
IPEndPoint remoteIpEndPoint = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 3800);
while (true)
{
receiveBytes = udpClient.Receive(ref remoteIpEndPoint);
returnData = Encoding.ASCII.GetString(receiveBytes);
Console.WriteLine(returnData);
}
}

Try this:
Socket s = ...;
DataOutputStream out = null;
try {
out = new DataOutputStream( s.getOutputStream() );
out.writeInt( 123456 );
} catch ( IOException e ) {
// TODO Handle exception
} finally {
if ( out != null ) {
try {
out.close();
} catch ( IOException e ) {
// TODO Handle exception
}
}
}
It whould help if you could explain a little more what your problem is.