Hi let me get straight to the problem. I have a big JSON packet that the server sends to this client once the client is authenticated
But the packet comes back in a weird way like it's split or something example:
The JSON should be:
Received: {"UserID":1,"PlayerID":2,"EXP":0,"Lvl":1,"Coins":0,"ItemSlots":30}
When it comes through:
Received: {"UserID":1,"PlayerID":2,"EXP":0,"Lvl":1,"Coins":0,
Received: "ItemSlots":30}
Why does it split the packet or something when it comes to the client and how can I fix this anyway?
Java Receive Code:
private class ClientThread extends Thread {
public void run() {
try {
while (selector.select() > 0) {
for (SelectionKey sk : selector.selectedKeys()) {
selector.selectedKeys().remove(sk);
if (sk.isReadable()) {
SocketChannel sc = (SocketChannel)sk.channel();
ByteBuffer buff = ByteBuffer.allocate(1024);
String content = "";
while (sc.read(buff) > 0) {
sc.read(buff);
buff.flip();
content += charset.decode(buff);
buff.clear();
}
System.out.println("Recieved: " + content);
sk.interestOps(SelectionKey.OP_READ);
}
}
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
Thanks have a wonderful day.
Hi lemme get straight to the problem so i got a big JSON packet that the server sends to this client once the client is authenticated
You mean you have a big JSON message. Packets are things that network protocols used to exchange information.
But the packet comes back in a weird way like its split or something example:
Unless you're looking at the wire, you aren't looking at packets. You're looking at the bytes you got from your end of the TCP connection.
The JSON should be:
Recieved: {"UserID":1,"PlayerID":2,"EXP":0,"Lvl":1,"Coins":0,"ItemSlots":30}
When it comes through:
Recieved: {"UserID":1,"PlayerID":2,"EXP":0,"Lvl":1,"Coins":0,
Recieved: "ItemSlots":30}
Excellent. You got the same bytes. Now make a JSON parser that figures out where the message ends and parses it.
Why does it split the packet or something when it comes to the client
It splits the message into packets because that's how TCP gets the message to the other side. TCP is not a message protocol and it doesn't know or care what the application considers to be a message -- that's the application's job.
and how i can i fix this anyway?
Write a JSON parser to figure out where the messages end. You haven't implemented any code to receive JSON over TCP yet, so that won't work until you do.
TL;DR: If you want an application-level message protocol, you need to implement one. TCP is not one.
TCP protocol does not maintain message boundaries. It is not guaranteed that what the server sends is received as-is by the client and vice-versa.
If the server sends 1000 bytes data, the client application can receive the same across multiple recv or single recv. TCP does not guarantee any behaviour. "Split" can happen, it is upto the application to handle the data coming in multiple chunks, coalesce it to one unit of application data for further processing. One can see this particularly with large data sizes.
It looks like you've got a non-blocking socket channel, meaning that the while (sc.read(buff) > 0) { loop is terminating due to sc.read(buff) returning 0 since only a portion of the data sent has, at this point, been received.
Why does it split the packet or something when it comes to the client
Most likely the data is being split into two or more packets.
and how i can i fix this anyway?
Keep filling your buffer until the socket is closed by the server (read should return -1 rather than 0 in that case). You need to maintain a separate buffer per channel. If the server doesn't close its end after sending the data, you'll need to delineate in some other way; you could prefix the JSON blob with a size header, for instance.
Related
Deplyment environment:
I have created a TCP server using JAVA over windows 10 OS. My TCP client program is written in VC++ and runs on windows 7 OS (I don't have any control over this part of the code, it is a black box to me).
My TCP server code is like this:
Socket s = ss.accept();
s.setReceiveBufferSize(2000);
s.setSendBufferSize(2000);
s.setTcpNoDelay(true);
s.setKeepAlive(true);
new TcpConnectionHandler(s,this.packetHandler);
Following is the TCP connection handler snippet:
InputStream incomingPacketBuffer = this.clientSocket.getInputStream();
OutputStream outgoingPacketBuffer = this.clientSocket.getOutputStream();
int bufferLen=0;
byte inBuffer[] = new byte[this.clientSocket.getReceiveBufferSize()];
byte outBuffer[] = new byte[this.clientSocket.getSendBufferSize()];
while(this.clientSocket.isConnected())
{
bufferLen = incomingPacketBuffer.read(inBuffer);
if(bufferLen>0)
{
outBuffer = (byte[]) this.packetHandlerModule.invoke(this.packetHandler,Arrays.copyOf(inBuffer, bufferLen));
}
if(outBuffer != null)
{
if(this.clientSocket.isConnected())
{
outgoingPacketBuffer.write(outBuffer);
outgoingPacketBuffer.flush();
}
}
}
this.clientSocket.close();
The communication is packet based and the protocol/parsing is handled by packetHandler.
Two more variant I've tried:
I have tried to close the socket as and when a reply is sent back to the client. That is, after receiving one packet of data, I reply to the client and close the connection.
I used inputStream.available before using the read method.
The problem I face:
Most of the time the TCP server replies to incoming packets within a second. If the server receives a packet after some idle time, the server doesn't reply to the packet. Sometimes even when there is active communication is going on, the reply is not being transmitted. Secondly, the isConnected function returns true even when the client socket closed the connection.
Debugging attempts:
I used teraterm to send packets and checked it. The behavior is same. As long as I send packets one after another, I don't have an issue. If one packet doesn't get a reply, then every packet sent after that does not get reply from the server.
When I press Ctrl+C in server console, all the packets sent from teraterm is processed by TCP server and reply is sent back. After this the server works properly for some duration.
I checked the packet flow with wireshark. When the replies are sent back normally, it is sent along with the ACK of client request (SYN, SYN+ACK, ACK, PSH, PSH+ACK, FYN, FYN+ACK, ACK). When the reply gets staled (may not be the right term, it is stuck in inputStream.available or inputStream.read), only ACK packet is sent by server (SYN, SYN+ACK, ACK, PSH, ACK).
I checked many forums and other threads in stackexchange, learned about Nagle's algorithm, applicaion must take care of packetization in TCP, TCP may receive 10+10 packets as 8+12 or 15+5 or any such manner. The server code takes care of packetization, setKeepAlive is set to true (there is no problem when a packet is sent from server).
Problem in short: "At times, TCP read call is getting blocked for a long duration even when there is incoming packets. When Ctrl+C is pressed, they are getting processed."
PS: I just started posting queries on stackexchange, so kindly let me know if there is any issues in the way of formulating the query.
PPS: Sorry for such a long post.
UPDATE
The comment from EJB helped me to identify the peer disconnect.
I made another setup with Ubuntu 16.04 as operating system for server. It has been 3 days, windows system had the issue occasionally. Ubuntu 16.04 never staled.
Some things to consider;
the TCP buffer sizes are usually 8K at least and I don't think you can skink them to 2000 bytes, or if you can, I don't think it's a good idea.
the size of the byte[] doesn't really matter over about 2K, you may as well pick a value.
you can't need to be creating a buffer more than once.
So in short I would try.
Socket s = ss.accept();
s.setTcpNoDelay(true);
s.setKeepAlive(true);
new TcpConnectionHandler(s,this.packetHandler);
and
try {
InputStream in = this.clientSocket.getInputStream();
OutputStream out = this.clientSocket.getOutputStream();
int bufferLen = 0;
byte[] buffer = new byte[2048];
while ((bufferLen = in.read(buffer)) > 0) {
out.write(buffer, 0, bufferLen); // not buffered so no need to flush
}
} finally {
this.clientSocket.close();
}
At times, TCP read call is getting blocked for a long duration even when there is incoming packets.
Would write a test Java client to see that this is not due to behaviour in Java.
I made a server in which the sending of the data itself is made like this:
PrintWriter writer = new PrintWriter(new OutputStreamWriter(clientSocket.getOutputStream(), "UTF-8"));
writer.write(json);
writer.close();
How can i know if the server has indeed sent the entire message?
In my client (written in C#), when the json string is long, then i get a certain amount of bytes which is less then the length of the json string (for some reason always a max length of 183 bytes).
The client is using asynchronous connection. How can i know if the server indeed sent the entire message? Is it possible i need to divide a long message into chunks and send them one after the other? I'm just not sure if the problem is in the server or in the client.
What I did in the end was adding an header to the message with the size of the message and then checked on the C# side if the size of the received message is the size of the message and if it's not, ask again the socket for data until I got the full message.
Is there a way to detect the EOF when reading from a TCP Socket whilst the socket connection stays open?
Most of the examples I have seen are something along the lines of:
int n=0;
While((n = read.inStream(data)) != -1){
destType.write(data, 0, n);
}
However this means that you are forced to create a new connection every time you want to receive a new piece of data.
In my case this is a constant stream of images that are sent across the socket as bytes and I would like to process each image without having to close the connection between so that I can have all the images associated with a single user, aswell as it is just more efficient to not constantly open and close connections at a high frequency.
So is there a way to do this or some information on a possible alternative?
No - if the connection stays open, the stream hasn't reached its end. The idea of a stream reporting EOF and then having more data later goes against the principle of a stream.
If you want to send multiple messages across a TCP stream, the simplest way is to prefix each message with its length:
HEADER BODY
HEADER BODY
HEADER BODY
Then the client will read the header, find out how long the message is, read it, then read the next header, etc.
I'm trying to write a WebSocket Server in both, java and C++ but I'm stuck right now.
Using java and java.net.ServerSocket/java.net.Socket I managed to get a connection and succesfully do the handshake but the data sent by the WebSocket to the Java Server is not quite what I expected.
When sending messages from javascript like this:
var count = 0;
function loop(){
websocket.send("loop: " + count + "\n");
count++;
setTimeout(loop, 100);
}
The Java server receives this, with line feeds every now and then but not for every websocket.send() that has been invoked.
?‡½÷"˜Ñ˜Mè‡×?‡AÎ3-¡C{îN?‡ŒÍ[Uà¢4%¶íi?‡$ÍåøH¢ŠˆíÖ?‡·†ÞžÛé±î?¦ê?‡'½Ø…KÒ·õ?í?‡dÒÛ‘½´á^òí?‡+ù?YG–â)Ùº?‡›?
Ë÷àb»¡¯5?‡mÉŒQ¦ã!Wéµ?ˆ:J FV%f6
The Java server retrieves values from the socket using BufferedReader.readLine()
BufferedReader socketReader = new BufferedReader(new InputStreamReader(socket.getInputStream()));
String line =socketReader.readLine();
This works fine for the handshake and all handshake data is readable but it does not work after the handshake is done.
Is the data after the handshake somehow encrypted? How can I read it?
EDIT :
The program files:
SocketConnectTest.html
ServerTest.java
ClientSessionTest.java
ResponseGenerator.java
output
Just run ServerTest.java and then open SocketConnectTest.html.
ClientSessionTest.initClientListener() handles the messages from the client.
SOLUTION :
For the solution see pimvdbs post below and his answer at How to (de)construct data frames in WebSockets hybi 08+?
The data coming across web sockets is raw, not string encoded data.
I'd suggest not wrapping a BufferedReader around the incoming data as packets are framed with 0x00 bytes. The crazy characters you are seeing are a result of Java not understanding the encoding that the data is in.
You will need to be responsible for splitting up the data into character and control parts. Once you've split the data up into the appropriate areas, then you can decode the data as a string.
Looking to make a Java stop-and-wait UDP server and client but I'm running into some problems starting off. I've made a simple UDP client and server without the stop-and-wait part, but I would now like to learn how to change it. How can I send ACKs and implement timeouts using java sockets ?
Could someone please post up some examples for me to use in my implementation ?
If you're implementing this in UDP, sending and receiving acknowledgements is up to you. This seems to be what you want for this stop and wait protocol. In terms of pseudocode, you would want something like:
int Send(msg)
{
char rcvBuf[];
sentBytes = sock.send(msg);
sock.rcv(rcvBuf);
return sentBytes;
}
int Recv(rcvBuf)
{
String ackMsg = "ACK";
length = sock.rcv(rcvBuf);
sock.send(ackMsg);
return length;
}
After every send, you wait for an acknowledgement message to come in, and every time you receive, you send an acknowledgement.