Why doesn't channelRead() give me the full message I send to the server? Fragmentation sometimes occur when messages are getting above 140 bytes (Roughly, sometimes more and sometimes less). I'm using a TCP socket using the NioServerSocketChannel class.
I'm using 4.1.0.Beta5.
Isn't there a way to read the full message when it has arrived?
this.serverBootstrap = new ServerBootstrap();
this.serverBootstrap.group(new NioEventLoopGroup(1), new NioEventLoopGroup(6))
.channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer<SocketChannel>()
{
#Override
public void initChannel(SocketChannel ch) throws Exception
{
ch.pipeline().addLast(new TestServerHandler());
}
})
.option(ChannelOption.SO_BACKLOG, (int)Short.MAX_VALUE)
.option(ChannelOption.SO_RCVBUF, (int) Short.MAX_VALUE)
.option(ChannelOption.SO_KEEPALIVE, true)
.option(ChannelOption.TCP_NODELAY, true);
this.serverBootstrap.bind(this.host, this.port);
And class TestServerHandler extends ChannelInboundHandlerAdapter:
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
String s = buffer.toString(CharsetUtil.UTF_8);
for(int i = 0; i < 20; i++)
{
s = s.replace("[" + ((char)i) + "]", i + "");
}
System.out.println(s.length() + "");
System.out.println();
System.out.println();
System.out.println(s);
}
I need a way to get the full bytebuf / bytearray when it has fully arrived at the server and get notified of that so my application can respond in a correct way according to the data the client has send.
So in short: How can I prevent fragmentation and have the channelRead event output the whole message / bytebuf.
The basic data type used by Netty is Channel Buffers or ByteBuf. This is simply a collection of bytes and nothing else. In your code you have simply used a custom handler to handle the raw incoming data. This is generally not a good practice. A very basic netty pipeline should look something like the following
So a pipeline consists of a decoder / encoder and then we have our custom handlers or logging handlers. We never really handle any raw data as is. TCP is a stream protocol. It does not identify when a specific packet ends and a new packet starts. Even if we send a very very large packet or say two individual packets, they will simply be treated as a set of bytes and when we try to read the raw set of bytes, fragmentation might happen.
So properly implement a channel pipeline which consists of a String decoder / encoder (whatever you need) and this problem will go away.
TCP provides a stream of bytes, so you can't rely on receiving a complete message in one packet. You will need a handler in your pipeline that knows how your messages are framed. Netty provides some built-in handlers that you can adapt for your protocol. See Dealing with a Stream-based Transport in the Netty User Guide.
Related
When trying to write with netty, the written data never ends up at the remote side, confirmed with Wireshark.
I have tried:
//Directly using writeAndFlush
channel.writeAndFlush(new Packet());
//Manually flushing
channel.write(new Packet());
channel.flush();
// Even sending bytes won't work:
channel.writeAndFlush(new byte[]{1,2,3});
No exception is caught when I wrap it in try{...}catch(Throwable e){e.printStackTrace();}
What can I do to debug this problem?
Netty is asynchronous, meaning that it won't throw exceptions when a write failed. Instead of throwing exceptions, it returns a Future<?> that will be updated when the request is done. Make sure to log any exceptions coming from this as your first debugging steps:
channel.writeAndFlush(...).addListener(new GenericFutureListener<Future<Object>>() {
#Override
public void operationComplete(Future<Object> future) {
// TODO: Use proper logger in production here
if (future.isSuccess()) {
System.out.println("Data written succesfully");
} else {
System.out.println("Data failed to write:");
future.cause().printStackTrace();
}
}
});
Or more simply:
channel.writeAndFlush(...).addListener(ChannelFutureListener.FIRE_EXCEPTION_ON_FAILURE);
After you get the root cause of the exception, there could be multiple problems:
java.lang.UnsupportedOperationException:unsupported message type: <type> (expected: ...)
Notice: This also throws when using an ObjectEncoder, but your object does not implements Serializable
A default Netty channel can only send ByteBufs and FileRegions. You need to convert your objects to these types either by adding more handlers to the pipeline, or converting them manually to ByteBufs.
A ByteBuf is the Netty variant of a byte array, but has the potential for performance because it can be stored in the direct memory space.
The following handlers are commonly used:
To convert a String use a StringEncoder
To convert a Serializable use a ObjectEncoder (warning, not compatible with normal Java object streams)
To convert a byte[] use a ByteArrayEncoder
Notice: Since TCP is a stream based protocol, you usually want some form of packet sizes attached, since you may not receive exact packets that you write. See Dealing with a Stream-based Transport in the Netty wiki for more information.
I have the following Server and Client initializers (both have extremely similar code where only sch changes to cch for the client, both representing their respective handlers).
#Override
public void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast("handler", sch);
ch.pipeline().addLast(new CommonClassHandler());
ch.pipeline().addLast("frameDecoder",
new ProtobufVarint32FrameDecoder());
ch.pipeline().addLast("protobufDecoder",
new ProtobufDecoder(Server.MyMessage.getDefaultInstance()));
ch.pipeline().addLast("frameEncoder", new ProtobufVarint32LengthFieldPrepender());
ch.pipeline().addLast("protobufEncoder", new ProtobufEncoder());
}
I wish to use a binary format when sending commands/actions to the client or the server, therefore, I'm using Google's Protocol Buffers.
Here is where I create a builder when dealing with the client's input:
while (channel.isOpen()) {
Client.MyMessage.Builder builder = Client.MyMessage.newBuilder();
String input = in.readLine(); // Save console input
builder.setKeyword(input); // Set the value of keyword to said input
channel.writeAndFlush(builder.build()); // Send the build to the server
}
And finally here is the method automatically called when the server / client receive a message:
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
ByteBuf bb = (ByteBuf)msg;
String order = bb.toString(Charset.defaultCharset());
System.out.println(order); // Displays received data
Server.MyMessage.Builder builder = Server.MyMessage.newBuilder();
builder.setKeyword("301");
ctx.writeAndFlush(builder.build());
}
1) When displaying the contents of my ByteBuf it displays two unknown characters and a "\n" prior to my message; maybe I should handle my received data another way to have a normal display?
2) After displaying the received data, my server should send the answer "301" to my client, but to no use, as my client does not display anything (the method is not even called in the Client handler), is there an apparent reason?
Please excuse my questions, but there is very little documentation concerning the use of Protocol Buffers with Netty 4.1.6.
You are adding your handler at the start of your pipeline ch.pipeline().addLast("handler", sch);, but you should be putting it at the end of the pipeline, after your protobufDecoder.
Once you make that change you should start receiving MyMessage as your msg instead of a ByteBuf. I'm guessing that the unknown characters you are seeing right now are the frame lengths that get stripped by the frame decoder you have, but it won't run until after your handler the way you have things setup right now.
Hi let me get straight to the problem. I have a big JSON packet that the server sends to this client once the client is authenticated
But the packet comes back in a weird way like it's split or something example:
The JSON should be:
Received: {"UserID":1,"PlayerID":2,"EXP":0,"Lvl":1,"Coins":0,"ItemSlots":30}
When it comes through:
Received: {"UserID":1,"PlayerID":2,"EXP":0,"Lvl":1,"Coins":0,
Received: "ItemSlots":30}
Why does it split the packet or something when it comes to the client and how can I fix this anyway?
Java Receive Code:
private class ClientThread extends Thread {
public void run() {
try {
while (selector.select() > 0) {
for (SelectionKey sk : selector.selectedKeys()) {
selector.selectedKeys().remove(sk);
if (sk.isReadable()) {
SocketChannel sc = (SocketChannel)sk.channel();
ByteBuffer buff = ByteBuffer.allocate(1024);
String content = "";
while (sc.read(buff) > 0) {
sc.read(buff);
buff.flip();
content += charset.decode(buff);
buff.clear();
}
System.out.println("Recieved: " + content);
sk.interestOps(SelectionKey.OP_READ);
}
}
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
Thanks have a wonderful day.
Hi lemme get straight to the problem so i got a big JSON packet that the server sends to this client once the client is authenticated
You mean you have a big JSON message. Packets are things that network protocols used to exchange information.
But the packet comes back in a weird way like its split or something example:
Unless you're looking at the wire, you aren't looking at packets. You're looking at the bytes you got from your end of the TCP connection.
The JSON should be:
Recieved: {"UserID":1,"PlayerID":2,"EXP":0,"Lvl":1,"Coins":0,"ItemSlots":30}
When it comes through:
Recieved: {"UserID":1,"PlayerID":2,"EXP":0,"Lvl":1,"Coins":0,
Recieved: "ItemSlots":30}
Excellent. You got the same bytes. Now make a JSON parser that figures out where the message ends and parses it.
Why does it split the packet or something when it comes to the client
It splits the message into packets because that's how TCP gets the message to the other side. TCP is not a message protocol and it doesn't know or care what the application considers to be a message -- that's the application's job.
and how i can i fix this anyway?
Write a JSON parser to figure out where the messages end. You haven't implemented any code to receive JSON over TCP yet, so that won't work until you do.
TL;DR: If you want an application-level message protocol, you need to implement one. TCP is not one.
TCP protocol does not maintain message boundaries. It is not guaranteed that what the server sends is received as-is by the client and vice-versa.
If the server sends 1000 bytes data, the client application can receive the same across multiple recv or single recv. TCP does not guarantee any behaviour. "Split" can happen, it is upto the application to handle the data coming in multiple chunks, coalesce it to one unit of application data for further processing. One can see this particularly with large data sizes.
It looks like you've got a non-blocking socket channel, meaning that the while (sc.read(buff) > 0) { loop is terminating due to sc.read(buff) returning 0 since only a portion of the data sent has, at this point, been received.
Why does it split the packet or something when it comes to the client
Most likely the data is being split into two or more packets.
and how i can i fix this anyway?
Keep filling your buffer until the socket is closed by the server (read should return -1 rather than 0 in that case). You need to maintain a separate buffer per channel. If the server doesn't close its end after sending the data, you'll need to delineate in some other way; you could prefix the JSON blob with a size header, for instance.
Please give me advice how to increase my ByteBuf initial capacity. In situation like:
#Override
protected void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception {
byte[] byteinput = new byte[in.readableBytes()];
in.readBytes(byteinput);
//further handling...
}
If income message more than max capacity of ByteBuf - i get cutted data. Its vital for this project to get whole, non chunked message.
I suppose i need to set initial capacity of ByteBuf somewhere in bootstraps childOptions, or in cannel.config()... inside of ChannelInitializer.
And i tried different ways like setting
ch.config().setReceiveBufferSize(1024)
but i still have same value of ByteBuf capacity(e.g. 496).
UPD
I discovered my protocol traffic with wireshark, and packets up to 1,4k going uncorrupted out and in from my test user client. This issue is only matter of netty settings. Operating system socket buffer do not cuts messages.
That was easy as pie.
ServerBootstrap b = new ServerBootstrap(); // (2)
b.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.class) // (3)
.childHandler(new ChannelInitializer<SocketChannel>() { // (4)
#Override
public void initChannel(SocketChannel ch) throws Exception {
//decrypt //checknum
ch.config().setRecvByteBufAllocator(new FixedRecvByteBufAllocator(2048)); //set buf size here
ch.pipeline().addLast(new InboundDecryptor());
.
.
.
You may be able to configure Netty's buffer allocation sizes but there are likely more general limitations you are subject to. Netty is an asynchronous framework. This means it will read what ever is made available to it by the OS and pass that on to you. Netty has no control over network conditions, networking hardware behavior, OS behavior, or anything else in between your producer of data and your Netty application. If your application logic requires complete application level messages you may have to aggregate the data before you invoke this application logic. Netty has some convenience methods to help with this see MessageAggregator.java and for an HTTP specific implementation see HttpObjectAggregator.java.
Scott is right. Increasing only the size of buffer does not solve the problem.
ch.config().setRecvByteBufAllocator(new FixedRecvByteBufAllocator(2048));
For HTTP requests it also should use HTTPObjectAggregator. It worked for me.
I know that a server sends MP3 stream after connecting to it and sending few bytes. How to read it's transmission with Apache MINA? Can you provide any examples please?
You need a client to read data from server. If it is possible to make a TCP connection with the server you can get help from this tutorial on Apache MINA TCP client
[UPDATE]
Data will be received in ClientSessionHandler's messageReceived. You can override this function according to you need. You may go through SumUp example to understand it fully.
[UPDATE 2]
To receive bytes in your case, you will have to update messageReceived of your session handler a bit. You can use IoBuffer to read byte. Something like this :
public void messageReceived(IoSession session, Object message) {
if (message instanceof IoBuffer) {
IoBuffer buffer = (IoBuffer) message;
byte[] b = new byte[buffer.remaining()];
buffer.get(b);
}
}