We Keep Coding

Socket / InputStream Dropping Data

I have been attempting to set up a basic server using Java's ServerSocket, Socket, and InputStream. In reading the InputStream, the expected result was a repeating series of byte 0x0b and 10 bytes of associated data (0x0b-data-0x0b-data repeating). The issue is that a small amount of the bytes are entirely dropped somewhere within the Java application, leaving only 9 bytes of data in some packets (after checking with Wireshark, the bytes are present in the original packets, just not the output of the InputStream).
The context in which this is happening is during a sequence of around a hundred packets sent in quick succession in response to certain behavior. I believe this is simply because there are more bytes that have an opportunity to be dropped and not the speed at which it is recieved.
After some searching, I found the same issue at Java Socket InputStream read missing bytes, but that thread died with requests for further information (and hence no useful answers).
The entirety of the code causing this problem is below. The most important sections are the while true loop and the readData function (excluding the else if chain).
To clarify, the question is the cause of this weird behaviour.
package com.kevycat.minerria;
import java.io.IOException;
import java.io.InputStream;
import java.net.ServerSocket;
import java.net.Socket;
import java.util.Arrays;
public class Minerria {
private static Socket client;
public static void main(String[] args) throws IOException {
ServerSocket socket = new ServerSocket(7777);
System.out.println("Listening");
client = socket.accept();
InputStream stream = client.getInputStream();
System.out.println("Connected");
byte[] extraData = new byte[0];
while (true) {
int available = stream.available();
byte[] data = new byte[available + extraData.length];
stream.read(data, extraData.length, available);
if (extraData.length > 0) {
for (int i = 0; i < extraData.length; i++) {
data[i] = extraData[i];
}
}
if (data.length > 0) {
for (int i = 0; i < data.length; i++) {
System.out.print(data[i] + " ");
}
System.out.println(" ");
}
if (data.length > 0) {
extraData = readData(data);
}
}
}
private static byte[] readData(byte[] data) throws IOException {
if (data.length < 3) {
return data;
}
int length = data[0] + data[1] * 256;
int type = data[2];
String payload = new String(Arrays.copyOfRange(data, 4, length));
System.out.println(length + " " + type + " " + payload);
if (type == 1) {
client.getOutputStream().write(new byte[] { 5, 0, 3, 0, 0 });
} else if (type == 4) {
client.getOutputStream().write(data);
} else if (type == 5) {
client.getOutputStream().write(data);
} else if (type == 68) {
client.getOutputStream().write(data);
} else if (type == 16) {
client.getOutputStream().write(data);
} else if (type == 42) {
client.getOutputStream().write(data);
} else if (type == 50) {
client.getOutputStream().write(data);
} else if (type == 6) {
byte[] b = new byte[80];
b[0] = 80;
b[2] = 7;
client.getOutputStream().write(b);
} else if (type == 8) {
client.getOutputStream().write(new byte[] { 11, 0, 9, 0, 1, 0, 0, 0, 'e', 'e', 'e' });
}
return data.length > length ? Arrays.copyOfRange(data, length, data.length - 1) : new byte[0];
}
}

int available = stream.available();
Don't do this. available() does nothing useful. If you don't believe me, I shall quote the javadoc:
Returns an estimate of the number of bytes that can be read (or skipped over) from this input stream without blocking by the next invocation of a method for this input stream.
'estimate'. That's programmer jargon. In plain english it translates as 'mostly useless'.
stream.read(data, extraData.length, available);
There's your error. You can't ignore the returned value of a read call. Read the javadoc: That read call will guarantee:
It reads at least 1 byte, unless the stream is closed / ended (then it reads nothing, and returns -1).
It will never read more than available.
But that is where it ends. It is perfectly legimitate for this method to only read half of available.
The ACTUAL # of bytes read is returned, unless it read nothing (only possible if stream is closed), then it returns -1.
The reason it's so convoluted is to get it to you as fast as possible. If packet arrives on your network card with 6 bytes and you ask for 10, it'll give you 6.
Use .readFully() if you want to just read X bytes (such as, 10 bytes, in your protocol that sounds useful), and ask the stream to wait as long as is needed (specifically, return only until either the stream ends are all 10 bytes are read).
For your protocol, I see two easy options:
Wrap the stream into a BufferedInputStream, and invoke only read(), the no-args one. That is a much simpler call: It returns -1 if stream ends, and a byte otherwise, easy peasy. It'll wait as long as needed until there's either data, or the stream is closed.
Alternatively, use .readFully. If you know that the data arrives in exact chunks of 11 every time, that'll work just as well. Although, calling a 'short' read (11 bytes is very short) on a non-buffered stream can be rather inefficient. Depends on the underlying stream.
Door #1 is less messy. It definitely does not suffer from inefficiency due to asking for too few bytes at a time, and it's hard to mess up your code.

Protocol buffer failed to parse single codeInputStream having multiple large size of message

Here is the message schema:>
message ServerResponse {
optional string ReferenceCode = 1;
optional NestedMessageProto.NestedMessage NestedMessage = 2;//Huge size data in response
optional bool Success = 3 [default = false];
repeated Errors Errors = 4;
}
Below is code for getting the response from serve and calling the proto response method.
String apiResponse = Server Response
protoResponseClass.parseFrom(apiResponse.getBytes())
its failing when reading the NestedMessage response on below bold line
public int pushLimit(int byteLimit) throws InvalidProtocolBufferException {
if (byteLimit < 0) {
throw InvalidProtocolBufferException.negativeSize();
}
byteLimit += totalBytesRetired + bufferPos;
if (byteLimit > currentLimit) {
currentLimit = byteLimit + currentLimit;
}
final int oldLimit = currentLimit;
**if (byteLimit > oldLimit) {
throw InvalidProtocolBufferException.truncatedMessage();
}**
currentLimit = byteLimit;
recomputeBufferSizeAfterLimit();
return oldLimit;
}
When its reading nested message the byte limit becoming greater than old limit.
What could be the solution?
Thanks

This is almost certainly the problem:
String apiResponse = Server Response
protoResponseClass.parseFrom(apiResponse.getBytes())
Protocol buffer messages are binary data. They're not text, and shouldn't be handled as text. You're taking the binary response from the server, interpreting it as text in some way (we can't tell how), then converting it back to a byte array using the platform default encoding (which is almost always a bad idea - never call getBytes() without specifying a charset, even when calling getBytes() is appropriate, which it's not here). The conversion to text and back is almost certainly losing information - quite possibly making the parsing code expect more data than is actually present in the message.
You should be handling the server response as binary data from the start - ideally just passing an InputStream from the server response into parseFrom, but at least reading it as a byte array instead of a String.

Instead of converting httpresponse to string and then to byte array for parsing, use EntityUtils.toByteArray directly.
private String readResponse(HttpResponse httpresponse) throws IOException {
int responseCode = httpresponse.getStatusLine().getStatusCode();
String mimeType = httpresponse.getFirstHeader(CONTENT_TYPE_KEY).getValue();
if (responseCode != HttpURLConnection.HTTP_OK) {
String cause = String.format("Bad HTTP response code: %d\nOr MIME type: %s", responseCode, mimeType);
throw new IOException(cause);
}
return EntityUtils.toString(httpresponse.getEntity());
}

How to send java byte[] on tcp socket to c++ char[] on server?

I want to send a byte[] array from a java client to a server that receives the data in C++. The byte array contains characters and integers that are converted to bytes (its a wave header). The server doesn't receive the values correctly. How can I send the byte[] so that the server socket can write it to a char[]? I am using the following code:
Client.java:
//Some example values in byte[]
byte[] bA = new byte[44];
bA[0]='R';
...
bA[4]=(byte)(2048 & 0xff);
...
bA[16] = 16;
....
//Write byte[] on socket
DataOutputStream out = new DataOutputStream(socket.getOutputStream());
out.write(bA,0,44);
Server.cpp
int k = 0,n = 0;
char buffer[100];
ofstream wav("out.wav", ios::out | ios::binary);
while(k<44){//receive 44 values
memset(buffer ,0 , 100);
n = recv(sock , buffer , 100 , 0);
k += n;
buffer[99] = '\0';
wav.write(buffer,n);
}

One issue I see is if you receive 100 characters, you're corrupting the data with this line:
buffer[99] = '\0';
If there is a character other than NULL at that position, you've corrupted the data. Since the data is binary, there is no need to null terminate the buffer. Remove that line from your loop.
Instead, rely on the return value of recv to determine the number of characters to copy to the stream. Which brings up another point -- you're not checking if recv returns an error.

Sending ints, receives wrong data

I'm trying to send the player's position (x, y) to the client from the Server. Everything else seems to work, but not these 2 ints. I can send other integers, for example 3456, and it will receive 3456. But these wont work.
Server Java code:
public void sendAccountInformation() throws IOException {
dos.write(player.id);
main.log(player.id);
dos.println(player.username);
main.log(player.username);
dos.write(player.x);
main.log(player.x);
dos.write(player.y);
main.log(player.y);
dos.write(player.mapid);
main.log(player.mapid);
dos.flush();
}
Server output:
0
sdfsdffsd
544
384
0
The above is the correct information that should be sent.
Client Java code:
public void loadAccount() throws IOException {
int id = dis.read();
main.player = new Player(id, main);
main.log(id);
main.player.username = dis.readLine();
main.log(main.player.username);
main.player.x = dis.read();
main.log(main.player.x);
main.player.y = dis.read();
main.log(main.player.y);
main.player.mapid = dis.read();
main.log(main.player.mapid);
}
Client output:
0
sdfsdffsd
63
63
0
As you can see, the two integers (544 and 384) was changed into (63 and 63). But EVERYTHING else sends and is received correctly?

I believe you should try using writeInt() and readInt() to write and read int.

Safari 5 / iOS, WebSocket handshake only works sometimes

I have made a class in java that allows me to wrap an existing socket with the WebSocket protocols. I have everything working for the RFC6445 protocol working and everything works in chrome and FF. However Safari and iOS is using the hixie76 / HyBi00 protocol (according to Wikipedia).
I have everything working and Safari and iOS correctly handshake and start sending/receiving messages... well, at least most of the time.
About 20-30% of the time, the handshake fails and Safari closes the connection. (Java reads a -1 byte upon trying to read first frame). Safari does not report any errors in the console, but just calls the onclose event handler.
Why would the handshakes only work part of the time?
Here is my handshake code:
Note: No exceptions are thrown and the "Handshake Complete" is written to the console. But then upon trying to read the first frame the connection is closed. (Java returns -1 on inst.read())
// Headers are read in a previous method which wraps the socket using RFC6445
// protocol. If it detects 2 keys it will call this and pass in the headers.
public static MessagingWebSocket wrapOldProtocol(HashMap<String, String> headers, PushbackInputStream pin, Socket sock) throws IOException, NoSuchAlgorithmException {
// SPEC
// https://datatracker.ietf.org/doc/html/draft-hixie-thewebsocketprotocol-76#page-32
// Read the "key3" value. This is 8 random bytes after the headers.
byte[] key3 = new byte[8];
for ( int i=0; i<key3.length; i++ ) {
key3[i] = (byte)pin.read();
}
// Grab the two keys we need to use for the handshake
String key1 = headers.get("Sec-WebSocket-Key1");
String key2 = headers.get("Sec-WebSocket-Key2");
// Count the spaces in both keys
// Abort the connection is either key has 0 spaces
int spaces1 = StringUtils.countMatches(key1, " ");
int spaces2 = StringUtils.countMatches(key2, " ");
if ( spaces1 == 0 || spaces2 == 0 ) {
throw new IOException("Bad Handshake Request, Possible Cross-protocol attack");
}
// Strip all non-digit characters from each key
// Use the remaining value as a base-10 integer.
// Abort if either number is not a multiple of it's #spaces counterpart
// Need to use long because the values are unsigned
long num1 = Long.parseLong( key1.replaceAll("\\D", "") );
long num2 = Long.parseLong( key2.replaceAll("\\D", "") );
if ( !(num1 % spaces1 == 0) || !(num2 % spaces2 == 0) ) {
throw new IOException("Bad Handshake Request. Possible non-conforming client");
}
// Part1/2 is key num divided by the # of spaces
int part1 = (int)(num1 / spaces1);
int part2 = (int)(num2 / spaces2);
// Now calculate the challenge response
// MD5( num1 + num2 + key3 ) ... concat, not add
MessageDigest md = MessageDigest.getInstance("MD5");
md.update(ByteBuffer.allocate(4).putInt(part1));
md.update(ByteBuffer.allocate(4).putInt(part2));
md.update(key3);
byte[] response = md.digest();
// Now build the server handshake response
// Ignore Sec-WebSocket-Protocol (we don't use this)
String origin = headers.get("Origin");
String location = "ws://" + headers.get("Host") + "/";
StringBuilder sb = new StringBuilder();
sb.append("HTTP/1.1 101 WebSocket Protocol Handshake").append("\r\n");
sb.append("Upgrade: websocket").append("\r\n");
sb.append("Connection: Upgrade").append("\r\n");
sb.append("Sec-WebSocket-Origin: ").append(origin).append("\r\n");
sb.append("Sec-WebSocket-Location: ").append(location).append("\r\n");
sb.append("\r\n");
// Anything left in the buffer?
if ( pin.available() > 0 ) {
throw new IOException("Unexpected bytes after handshake!");
}
// Send the handshake & challenge response
OutputStream out = sock.getOutputStream();
out.write(sb.toString().getBytes());
out.write(response);
out.flush();
System.out.println("[MessagingWebSocket] Handshake Complete.");
// Return the wrapper socket class.
MessagingWebSocket ws = new MessagingWebSocket(sock);
ws.oldProtocol = true;
return ws;
}
Thanks!
Note: I am not looking for third-party alternatives for WebSockets such at jWebSocket, Jetty and Socket.IO. I already know about many of these.

Your MD5 digest method has a bug:
protocol described as below: https://datatracker.ietf.org/doc/html/draft-hixie-thewebsocketprotocol-76#section-5.2
byte[] bytes = new byte[16];
BytesUtil.fillBytesWithArray(bytes, 0, 3, BytesUtil.intTobyteArray(part1));
BytesUtil.fillBytesWithArray(bytes, 4, 7, BytesUtil.intTobyteArray(part2));
BytesUtil.fillBytesWithArray(bytes, 8, 15, key3);
I think your problem is caused by Little Endian and Big Endian.