We Keep Coding

Decoding and Encoding multiple packet types in Netty

So I am working on a netty server that handles a specific type packet. This netty server was not made by me and I am trying to add another type of packet handler, decoder, and encoder to it. Because of poor implementation of the existing handlers, I have to add mine before like so.
channel.pipeline()
.addLast(new custom.PacketDecoder().server()) // My Packet Decoder
.addLast(new custom.PacketEncoder().server()) // My Packet Encoder
.addLast(new custom.NetworkHandler().server()) // My Packet Handler
.addLast("timeout", new ReadTimeoutHandler(30)) // Not made by me
.addLast("legacy_query", new LegacyPingHandler(ServerConnection.this)) // Not made by me
.addLast("splitter", new PacketSplitter()) // Not made by me
.addLast("decoder", new PacketDecoder(EnumProtocolDirection.SERVERBOUND)) // Not made by me
.addLast("prepender", new PacketPrepender()) // Not made by me
.addLast("encoder", new PacketEncoder(EnumProtocolDirection.CLIENTBOUND)) // Not made by me
.addLast("packet_handler", networkmanager); // Not made by me
As of now this sort of works but the issue is that if it recives the packets that arent mine that it should be able to handle, my handlers stop those packets from going through and being handled correctly. How do I pass packets that are not mine through to other handlers without touching them.

You handler should check if an inbound packet is of specific type, that the handler can process. If a packet is "not yours", handler should pass it upstream - to the next handler in pipeline (for netty 4.x - ChannelHandlerContext.fireChannelRead(Object)).
Maybe this documentation can help.

C# TcpClient to netty not always arriving

Me and a friend are working on a project which requires us to communicate between a C#.NET application and a Java application.
We're using the TcpClient and BinaryWriter classes on the .NET side of things to send and receive things. We're using code similar to this to send things:
byte[] content = //we're getting our content here
Writer.Write(new byte[9]); //this is the BinaryWriter with the NetworkStream of the TcpClient
Writer.Flush();
On the Java side of things, we're using netty to handle our networking. To receive the content we send from the .NET side, we add a ChannelInboundHandlerAdapter to the pipeline and use the channelRead method to read the content:
public void channelRead(ChannelHandlerContext ctx, Object received)
{
ByteBuf receivedByteBuf = (ByteBuf)received;
this.bytesRead = receivedByteBuf.readableBytes();
System.out.println("Received " + this.bytesRead + " bytes.");
final byte[] buffer = new byte[this.bytesRead];
receivedByteBuf.markReaderIndex();
receivedByteBuf.readBytes(buffer);
receivedByteBuf.resetReaderIndex();
}
Now the strange thing is, that when we try sending content, it doesn't always arrive in one piece. Sometimes we only receive all but some bytes we originally sent, which arrive in a new call of channelRead. In this example, only 6-8 bytes would arrive. This is very strange, as this only happens when using .NET. We tried sending content using python and everything worked fine and it arrived in one channelRead call.
import socket
import string, random
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("127.0.0.1", 8888))
s.send(''.join(random.choice(string.lowercase) for x in range(500)))
s.close()
Unfortunately, the nature of our project prevents us from changing our Java networking library, so we're stuck with netty.
Did we miss some setting in netty or does this have to do with the nature of the .NET TCP libraries? We would appreciate any help we can get.

Netty channelRead not getting full message

Why doesn't channelRead() give me the full message I send to the server? Fragmentation sometimes occur when messages are getting above 140 bytes (Roughly, sometimes more and sometimes less). I'm using a TCP socket using the NioServerSocketChannel class.
I'm using 4.1.0.Beta5.
Isn't there a way to read the full message when it has arrived?
this.serverBootstrap = new ServerBootstrap();
this.serverBootstrap.group(new NioEventLoopGroup(1), new NioEventLoopGroup(6))
.channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer<SocketChannel>()
{
#Override
public void initChannel(SocketChannel ch) throws Exception
{
ch.pipeline().addLast(new TestServerHandler());
}
})
.option(ChannelOption.SO_BACKLOG, (int)Short.MAX_VALUE)
.option(ChannelOption.SO_RCVBUF, (int) Short.MAX_VALUE)
.option(ChannelOption.SO_KEEPALIVE, true)
.option(ChannelOption.TCP_NODELAY, true);
this.serverBootstrap.bind(this.host, this.port);
And class TestServerHandler extends ChannelInboundHandlerAdapter:
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
String s = buffer.toString(CharsetUtil.UTF_8);
for(int i = 0; i < 20; i++)
{
s = s.replace("[" + ((char)i) + "]", i + "");
}
System.out.println(s.length() + "");
System.out.println();
System.out.println();
System.out.println(s);
}
I need a way to get the full bytebuf / bytearray when it has fully arrived at the server and get notified of that so my application can respond in a correct way according to the data the client has send.
So in short: How can I prevent fragmentation and have the channelRead event output the whole message / bytebuf.

The basic data type used by Netty is Channel Buffers or ByteBuf. This is simply a collection of bytes and nothing else. In your code you have simply used a custom handler to handle the raw incoming data. This is generally not a good practice. A very basic netty pipeline should look something like the following
So a pipeline consists of a decoder / encoder and then we have our custom handlers or logging handlers. We never really handle any raw data as is. TCP is a stream protocol. It does not identify when a specific packet ends and a new packet starts. Even if we send a very very large packet or say two individual packets, they will simply be treated as a set of bytes and when we try to read the raw set of bytes, fragmentation might happen.
So properly implement a channel pipeline which consists of a String decoder / encoder (whatever you need) and this problem will go away.

TCP provides a stream of bytes, so you can't rely on receiving a complete message in one packet. You will need a handler in your pipeline that knows how your messages are framed. Netty provides some built-in handlers that you can adapt for your protocol. See Dealing with a Stream-based Transport in the Netty User Guide.

In Java, must non-serializable Sockets prevent use of a client-server Observer pattern?

In my Java Sockets program, I have implemented a client-server Observer pattern. That is, the server subject publishes its server events to client observers that have subscribed to the server. That's the theory.
In practice, I am unable to send the client observer through the socket to subscribe to the server. The error that comes up is: "java.io.NotSerializableException: java.net.Socket." I think this means Java is complaining that the client observer contains a Socket which, as we all know, is not Serializable.
However, the Socket is the means of communication between the client and the server!
How can I implement a client-server Observer pattern when the client appears to contain a non-Serializable roadblock?
Here is a code overview to give you an understanding of what is happening:
Server
public class Server implements ServerSocketPublisher {
// traditional Observer publisher methods implemented here, such as register,
// deregister, notifySubscribers
// ServerSocket implemented here. Waiting on accept()
}
Client
public class Client implements ClientSocketSubscriber, Serializable {
// traditional Observer subscriber methods implemented here, i.e. updateClient
Socket connectingSocket = null; //I SUSPECT THIS VARIABLE IS THE PROBLEM
try {
connectingSocket = new Socket();
// set SocketAddress and timeout
connectingSocket.connect(sockAddr, timeout)
if (connectingSocket.isConnected()) {
ObjectOutputStream oos = new ObjectOutputStream
(connectingSocket.getOutputStream());
oos.writeObject(this); // THIS LINE THROWS THE ERROR in STACKTRACES
oos.flush();
oos.close();
}
} catch (/*various exceptions*/) {
}
// close connectingSocket
}

You have couple of ways to get this fixed:
Mark your socket as transient
transient Socket connectingSocket = null;
Instead of implementing Serializable implement Externalizable and then in your implementation of read and write object ignore the Socket.
Along with this you should also read
About transient:
Post on SO
About Externalizable :
Javabeat

you cannot write the Client to the output stream socket since it contains a Socket. If you serialize the Client, you serialize all non-transient vars in it, and thats when you get the exception.
However, the server already has the socket on its side, so you don't need to send it and the client across. If all clients are observers once the connection has occurred you can pretty much at that point start waiting for data from the socket on the client side. The server will need to keep a list of sockets its ready to broadcast to, and when it gets an event to send, loop over all sockets and send the register, deregister, notifySubscriber messages
Alternatively if you wish to treat the client as an object on the server side and call methods on it (which it looks like you might be trying to do), maybe you need to look into RMI - where the server holds stubs of the client and invoking the stub sends messages to the client.

How to read binary data from socket with Apache MINA?

I know that a server sends MP3 stream after connecting to it and sending few bytes. How to read it's transmission with Apache MINA? Can you provide any examples please?

You need a client to read data from server. If it is possible to make a TCP connection with the server you can get help from this tutorial on Apache MINA TCP client
[UPDATE]
Data will be received in ClientSessionHandler's messageReceived. You can override this function according to you need. You may go through SumUp example to understand it fully.
[UPDATE 2]
To receive bytes in your case, you will have to update messageReceived of your session handler a bit. You can use IoBuffer to read byte. Something like this :
public void messageReceived(IoSession session, Object message) {
if (message instanceof IoBuffer) {
IoBuffer buffer = (IoBuffer) message;
byte[] b = new byte[buffer.remaining()];
buffer.get(b);
}
}