We Keep Coding

Socket / InputStream Dropping Data

I have been attempting to set up a basic server using Java's ServerSocket, Socket, and InputStream. In reading the InputStream, the expected result was a repeating series of byte 0x0b and 10 bytes of associated data (0x0b-data-0x0b-data repeating). The issue is that a small amount of the bytes are entirely dropped somewhere within the Java application, leaving only 9 bytes of data in some packets (after checking with Wireshark, the bytes are present in the original packets, just not the output of the InputStream).
The context in which this is happening is during a sequence of around a hundred packets sent in quick succession in response to certain behavior. I believe this is simply because there are more bytes that have an opportunity to be dropped and not the speed at which it is recieved.
After some searching, I found the same issue at Java Socket InputStream read missing bytes, but that thread died with requests for further information (and hence no useful answers).
The entirety of the code causing this problem is below. The most important sections are the while true loop and the readData function (excluding the else if chain).
To clarify, the question is the cause of this weird behaviour.
package com.kevycat.minerria;
import java.io.IOException;
import java.io.InputStream;
import java.net.ServerSocket;
import java.net.Socket;
import java.util.Arrays;
public class Minerria {
private static Socket client;
public static void main(String[] args) throws IOException {
ServerSocket socket = new ServerSocket(7777);
System.out.println("Listening");
client = socket.accept();
InputStream stream = client.getInputStream();
System.out.println("Connected");
byte[] extraData = new byte[0];
while (true) {
int available = stream.available();
byte[] data = new byte[available + extraData.length];
stream.read(data, extraData.length, available);
if (extraData.length > 0) {
for (int i = 0; i < extraData.length; i++) {
data[i] = extraData[i];
}
}
if (data.length > 0) {
for (int i = 0; i < data.length; i++) {
System.out.print(data[i] + " ");
}
System.out.println(" ");
}
if (data.length > 0) {
extraData = readData(data);
}
}
}
private static byte[] readData(byte[] data) throws IOException {
if (data.length < 3) {
return data;
}
int length = data[0] + data[1] * 256;
int type = data[2];
String payload = new String(Arrays.copyOfRange(data, 4, length));
System.out.println(length + " " + type + " " + payload);
if (type == 1) {
client.getOutputStream().write(new byte[] { 5, 0, 3, 0, 0 });
} else if (type == 4) {
client.getOutputStream().write(data);
} else if (type == 5) {
client.getOutputStream().write(data);
} else if (type == 68) {
client.getOutputStream().write(data);
} else if (type == 16) {
client.getOutputStream().write(data);
} else if (type == 42) {
client.getOutputStream().write(data);
} else if (type == 50) {
client.getOutputStream().write(data);
} else if (type == 6) {
byte[] b = new byte[80];
b[0] = 80;
b[2] = 7;
client.getOutputStream().write(b);
} else if (type == 8) {
client.getOutputStream().write(new byte[] { 11, 0, 9, 0, 1, 0, 0, 0, 'e', 'e', 'e' });
}
return data.length > length ? Arrays.copyOfRange(data, length, data.length - 1) : new byte[0];
}
}

int available = stream.available();
Don't do this. available() does nothing useful. If you don't believe me, I shall quote the javadoc:
Returns an estimate of the number of bytes that can be read (or skipped over) from this input stream without blocking by the next invocation of a method for this input stream.
'estimate'. That's programmer jargon. In plain english it translates as 'mostly useless'.
stream.read(data, extraData.length, available);
There's your error. You can't ignore the returned value of a read call. Read the javadoc: That read call will guarantee:
It reads at least 1 byte, unless the stream is closed / ended (then it reads nothing, and returns -1).
It will never read more than available.
But that is where it ends. It is perfectly legimitate for this method to only read half of available.
The ACTUAL # of bytes read is returned, unless it read nothing (only possible if stream is closed), then it returns -1.
The reason it's so convoluted is to get it to you as fast as possible. If packet arrives on your network card with 6 bytes and you ask for 10, it'll give you 6.
Use .readFully() if you want to just read X bytes (such as, 10 bytes, in your protocol that sounds useful), and ask the stream to wait as long as is needed (specifically, return only until either the stream ends are all 10 bytes are read).
For your protocol, I see two easy options:
Wrap the stream into a BufferedInputStream, and invoke only read(), the no-args one. That is a much simpler call: It returns -1 if stream ends, and a byte otherwise, easy peasy. It'll wait as long as needed until there's either data, or the stream is closed.
Alternatively, use .readFully. If you know that the data arrives in exact chunks of 11 every time, that'll work just as well. Although, calling a 'short' read (11 bytes is very short) on a non-buffered stream can be rather inefficient. Depends on the underlying stream.
Door #1 is less messy. It definitely does not suffer from inefficiency due to asking for too few bytes at a time, and it's hard to mess up your code.

Python byte object vs java

Good morning, I have a quick question regarding the differences between a byte object in python (denoted b'') and how to replicate it in java.
The project I am working on is some personal work on an emulation server for a dead game to better my reversing skills. I have a working rendition of the project in python, but would like to switch over to java as I am better with the language and it comes with many additional tools included that are useful for a project like this.
I am using a ServerSocket to capture TCP data in the java project.
When data comes over the network from the Python project it looks a little something like this:
When I capture the same data over the java ServerSocket I get something like this:
My question is how can I reformat this ASCII text to get the proper data as seen in the python version of the software.
Currently I am able to get an output like this:
By converting the byte[] data from the ServerSocket as such
while(true) {
try {
Socket socket = serverSocket.accept();
onConnection(socket);
byte[] incomingData = new byte[0];
byte[] temp = new byte[1024];
int k = -1;
//this is due to the client of said game not sending EOL (readLine() does not work here)
while((k = socket.getInputStream().read(temp, 0, temp.length)) > -1) {
byte[] tbuff = new byte[incomingData.length + k];
System.arraycopy(incomingData, 0, tbuff, 0, incomingData.length);
System.arraycopy(temp, 0, tbuff, incomingData.length, k);
incomingData = tbuff;
receiveData(socket, incomingData); <--- this is the important bit
}
} catch (IOException e) {
e.printStackTrace();
}
}
public void receiveData(Socket socket, byte[] data) {
int lenLo = (int) (data[0]);
int lenHi = (int) (data[1]);
int length = lenHi * 256 + lenLo;
if(lenHi < 0) {
System.out.println("Invalid Packet Length");
}
if(data.length != length) {
System.out.println("Incomplete Packet Received");
}
try {
String test = new String(data, "UTF-8");
serverGUI.serverDebug(test); //produces the string in a jframe (pic 2)
serverGUI.debugByteArray(test.getBytes(StandardCharsets.UTF_8)); //produces the byte[] in jframe (pic 3 -- all bytes in this array are & 0xff prior to being printed out)
} catch (UnsupportedEncodingException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
}
However this has obviously not produced the desired outcome. Any advice is appreciated or any resources that can be put forth are also appreciated.
Thanks in advance!

Reusing the inputstream of a socket

i would like to know how to keep the input stream of a socket and reuse it until the application is close.
What i do for now is creating a thread in the main method. This thread is supposed to keep running for the all time the application runs. In this thread i read data from the server using the socket input stream. But i'm able to read only one time what the server is sending. After that i think the thread is dead or i cannot read from the input stream. How can i do to keep the input stream reading what is coming from the server.
Thanks.
int length = readInt(input);
byte[] msg = new byte[length];
input.read(msg);
ByteArrayInputStream bs = new ByteArrayInputStream(msg);
DataInputStream in = new DataInputStream(bs);
int cmd = readInt(in);
switch(cmd) {
case 1: Msg msg = readMsg(cmd, msg);
}
I put here everything, but in my code things happen in different methods.
The readInt method:
public static int readInt(InputStream in) throws IOException {
int byte1 = in.read();
int byte2 = in.read();
int byte3 = in.read();
int byte4 = in.read();
if (byte4 == -1) {
throw new EOFException();
}
return (byte4 << 24)
+ ((byte3 << 24) >>> 8)
+ ((byte2 << 24) >>> 16)
+ ((byte1 << 24) >>> 24);
}
Used for little-endian conversion.

your socket might well be blocking. If you encounter such a problem one good way around is to design your software for a polling method rather than being interrupt driven. Then again, the software design pattern will be done around what you are trying to achieve.
Hope it helps! Cheers!

You need to call input.read() in a loop such as this:
try {
while(running) {
int length = readInt(input);
byte[] msg = new byte[length];
input.read(msg);
ByteArrayInputStream bs = new ByteArrayInputStream(msg);
DataInputStream in = new DataInputStream(bs);
int cmd = readInt(in);
switch(cmd) {
case 1: Msg msg = readMsg(cmd, msg);
}
}
} catch (IOException e) {
//Handle error
}
Set running to false when you are finished with what ever your thread needs to be doing. Remember input.read() will block until the socket has received something. I hope this helps.

Is there a difference in Java's writeInt when executed on Windows vs an Intel based Mac

I currently writing a Java TCP server to handle the communication with a client (which I didn't write). When the server, hosted on windows, responds to the client with the number of records received the client doesn't read the integer correctly, and instead reads it as an empty packet. When the same server code, hosted on my Mac, responds to the client with the number of records received the client reads the packet and responds correctly. Through my research I haven't found an explanation that seems to solve the issue. I have tried reversing the bytes (Integer.reverseBytes) before calling the writeInt method and that didn't seem to resolve the issue. Any ideas are appreciated.
Brian
After comparing the pcap files there are no obvious differences in how they are sent. The first byte is sent followed by the last 3. Both systems send the correct number of records.
Yes I'm referring to the DataOutputStream.writeInt() method. //Code added
public void run() {
try {
InputStream in = socket.getInputStream();
DataOutputStream datOut = new DataOutputStream(socket.getOutputStream());
datOut.writeByte(1); //sends correctly and read correctly by client
datOut.flush();
//below is used to read bytes to determine length of message
int bytesRead=0;
int bytesToRead=25;
byte[] input = new byte[bytesToRead];
while (bytesRead < bytesToRead) {
int result = in.read(input, bytesRead, bytesToRead - bytesRead);
if (result == -1) break;
bytesRead += result;
}
try {
inputLine = getHexString(input);
String hexLength = inputLine.substring(46, 50);
System.out.println("hexLength: " + hexLength);
System.out.println(inputLine);
//used to read entire sent message
bytesRead = 0;
bytesToRead = Integer.parseInt(hexLength, 16);
System.out.println("bytes to read " + bytesToRead);
byte[] dataInput = new byte[bytesToRead];
while (bytesRead < bytesToRead) {
int result = in.read(dataInput, bytesRead, bytesToRead - bytesRead);
if (result == -1) break;
bytesRead += result;
}
String data = getHexString(dataInput);
System.out.println(data);
//Sends received data to class to process
ProcessTel dataValues= new ProcessTel(data);
String[] dataArray = new String[10];
dataArray = dataValues.dataArray();
//assigns returned number of records to be written to client
int towrite = Integer.parseInt(dataArray[0].trim());
//Same write method on Windows & Mac...works on Mac but not Windows
datOut.writeInt(towrite);
System.out.println("Returned number of records: " + Integer.parseInt(dataArray[0].trim()) );
datOut.flush();
} catch (Exception ex) {
Logger.getLogger(ServerThread.class.getName()).log(Level.SEVERE, null, ex);
}
datOut.close();
in.close();
socket.close();
} catch (IOException e) {
e.printStackTrace();
}
}

As described in its Javadoc, DataOutputStream.writeInt() uses network byte order as per the TCP/IP RFCs. Is that the method you are referring to?

No, x86 processors only support little-endian byte order, it doesn't vary with the OS. Something else is wrong.
I suggest using wireshark to capture the stream from a working Mac server and a non-working Windows server and compare.

Some general comments on your code:
int bytesRead=0;
int bytesToRead=25;
byte[] input = new byte[bytesToRead];
while (bytesRead < bytesToRead) {
int result = in.read(input, bytesRead, bytesToRead - bytesRead);
if (result == -1) break;
bytesRead += result;
}
This EOF handling is hokey. It means that you don't know whether or not you've actually read the full 25 bytes. And if you don't, you'll assume that the bytes-to-send is 0.
Worse, you copy-and-paste this code lower down, relying on proper initialization of the same variables. If there's a typo, you'll never know it. You could refactor it into its own method (with tests), or you could call DataInputStream.readFully().
inputLine = getHexString(input);
String hexLength = inputLine.substring(46, 50);
You're converting to hex in order to extract an integer? Why? And more important, if you have any endianness issues this is probably the reason
I was originally going to recommend using a ByteBuffer to extract values, but on a second look I think you should wrap your input stream with a DataInputStream. That would allow you to read complete byte[] buffers without the need for a loop, and it would let you get rid of the byte-to-hex-to-integer conversions: you'd simply call readInt().
But, continuing on:
String[] dataArray = new String[10];
dataArray = dataValues.dataArray();
Do you realize that the new String[10] is being thrown away by the very next line? Is that what you want?
int towrite = Integer.parseInt(dataArray[0].trim());
datOut.writeInt(towrite);
System.out.println("Returned number of records: " + Integer.parseInt(dataArray[0].trim()) );
If you're using logging statements, print what you're actually using (towrite). Don't recalculate it. There's too much of a chance to make a mistake.
} catch (Exception ex) {
Logger.getLogger(ServerThread.class.getName()).log(Level.SEVERE, null, ex);
}
// ...
} catch (IOException e) {
e.printStackTrace();
}
Do either or both of these catch blocks get invoked? And why do they send their output to different places? For that matter, if you have a logger, why are you inserting System.out.println() statements?

File transfer from C++ client to Java server

I have a c++ client which needs to send a file to a c++ server. I'm splitting the file to chunks of PACKET_SIZE (=1024) bytes and send them over a TCP socket. At the server side I read at most PACKET_SIZE bytes to a buffer. When the client sends files which are less than PACKET_SIZE, the server receives more bytes than sent. Even when I limit the number of bytes to be exactly the size of the file, the files differ. I know the problem does not have to do with the client because I've tested it with a c++ server and it works flawlessly.
Thanks.
Server:
public void run() {
DataInputStream input = null;
PrintWriter output = null;
try {
input = new DataInputStream (_client.getInputStream());
}
catch (Exception e) {/* Error handling code */}
FileHeader fh = recvHeader(input);
size = fh._size;
filename = fh._name;
try {
output = new PrintWriter(_client.getOutputStream(), true);
}
catch (Exception e) {/* Error handling code */}
output.write(HEADER_ACK);
output.flush();
FileOutputStream file = null;
try {
file = new FileOutputStream(filename);
}
catch (FileNotFoundException fnfe) {/* Error handling code */}
int total_bytes_rcvd = 0, bytes_rcvd = 0, packets_rcvd = 0;
byte [] buf = new byte [PACKET_DATA_SIZE];
try {
int max = (size > PACKET_DATA_SIZE)? PACKET_DATA_SIZE: size;
bytes_rcvd = input.read(buf,0, max);
while (total_bytes_rcvd < size) {
if (-1 == bytes_rcvd) {...}
++packets_rcvd;
total_bytes_rcvd += bytes_rcvd;
file.write (buf,0, bytes_rcvd);
if (total_bytes_rcvd < size)
bytes_rcvd = input.read(buf);
}
file.close();
}
catch (Exception e) {/* Error handling code */}
}
Client:
char packet [PACKET_SIZE] ;
file.open (filename, ios::in | ios::binary);//fopen (file_path , "rb");
int max = 0;
if (file.is_open()) {
if (size > PACKET_SIZE)
max = PACKET_SIZE;
else
max = size;
file.read (packet , max);
}
else {...}
int sent_packets = 0;
while (sent_packets < (int) ceil (((float)size)/PACKET_SIZE) ) {
_write=send(_sd , packet, max,0);
if (_write <0) {...}
else {
++sent_packets;
if (size > PACKET_SIZE* sent_packets) {
if (size - PACKET_SIZE* sent_packets >= PACKET_SIZE)
max = PACKET_SIZE;
else
max = size - PACKET_SIZE* sent_packets;
file.read (packet , max);
}
}
}

Is the sending socket closed at the end of the file, or is the next file streamed over the same socket? If more than one file is streamed, you could pick up data from the next file if you have the endedness wrong for the file size in recvHeader(), i.e. you send a file of length 0x0102 and try to read one of length 0x0201.
Other question, why do you provide a max for the first read, but not for the following reads on the same file?

One issue I see is that it appears that you assume that if the send returns a non-error, that it sent the entire chunk you requested it to send. This is not necessarily true, especially with stream sockets. How large are the packets you are sending, and how many? The most likely reason this could occur would be if the sndbuf for the socket filled, and your socket _sd is set to non-blocking. I'm not positive (depends on stack implementation), but I believe it could also likely occur if the TCP transmit window was full for your connection, and tcp couldn't enqueue your entire packet.
You should probably loop on the send until max is sent.
Thusly:
int send_ct=0;
while( (_write = send(_sd, packet + send_ct, max-send_ct, 0)) > 0) {
send_ct += _write;
if(send_ct >= max) {
break;
} else {
// Had to do another send
}
}

the code is not complete. E.g. you have omitted the sending of the filename and the filesize, as well as the parsing of those values. Are those values correct? If not first ensure that these values are the right ones before investigating further.