We Keep Coding

Netty doesn't write

When trying to write with netty, the written data never ends up at the remote side, confirmed with Wireshark.
I have tried:
//Directly using writeAndFlush
channel.writeAndFlush(new Packet());
//Manually flushing
channel.write(new Packet());
channel.flush();
// Even sending bytes won't work:
channel.writeAndFlush(new byte[]{1,2,3});
No exception is caught when I wrap it in try{...}catch(Throwable e){e.printStackTrace();}
What can I do to debug this problem?

Netty is asynchronous, meaning that it won't throw exceptions when a write failed. Instead of throwing exceptions, it returns a Future<?> that will be updated when the request is done. Make sure to log any exceptions coming from this as your first debugging steps:
channel.writeAndFlush(...).addListener(new GenericFutureListener<Future<Object>>() {
#Override
public void operationComplete(Future<Object> future) {
// TODO: Use proper logger in production here
if (future.isSuccess()) {
System.out.println("Data written succesfully");
} else {
System.out.println("Data failed to write:");
future.cause().printStackTrace();
}
}
});
Or more simply:
channel.writeAndFlush(...).addListener(ChannelFutureListener.FIRE_EXCEPTION_ON_FAILURE);
After you get the root cause of the exception, there could be multiple problems:
java.lang.UnsupportedOperationException:unsupported message type: <type> (expected: ...)
Notice: This also throws when using an ObjectEncoder, but your object does not implements Serializable
A default Netty channel can only send ByteBufs and FileRegions. You need to convert your objects to these types either by adding more handlers to the pipeline, or converting them manually to ByteBufs.
A ByteBuf is the Netty variant of a byte array, but has the potential for performance because it can be stored in the direct memory space.
The following handlers are commonly used:
To convert a String use a StringEncoder
To convert a Serializable use a ObjectEncoder (warning, not compatible with normal Java object streams)
To convert a byte[] use a ByteArrayEncoder
Notice: Since TCP is a stream based protocol, you usually want some form of packet sizes attached, since you may not receive exact packets that you write. See Dealing with a Stream-based Transport in the Netty wiki for more information.

Netty channelRead not getting full message

Why doesn't channelRead() give me the full message I send to the server? Fragmentation sometimes occur when messages are getting above 140 bytes (Roughly, sometimes more and sometimes less). I'm using a TCP socket using the NioServerSocketChannel class.
I'm using 4.1.0.Beta5.
Isn't there a way to read the full message when it has arrived?
this.serverBootstrap = new ServerBootstrap();
this.serverBootstrap.group(new NioEventLoopGroup(1), new NioEventLoopGroup(6))
.channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer<SocketChannel>()
{
#Override
public void initChannel(SocketChannel ch) throws Exception
{
ch.pipeline().addLast(new TestServerHandler());
}
})
.option(ChannelOption.SO_BACKLOG, (int)Short.MAX_VALUE)
.option(ChannelOption.SO_RCVBUF, (int) Short.MAX_VALUE)
.option(ChannelOption.SO_KEEPALIVE, true)
.option(ChannelOption.TCP_NODELAY, true);
this.serverBootstrap.bind(this.host, this.port);
And class TestServerHandler extends ChannelInboundHandlerAdapter:
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
String s = buffer.toString(CharsetUtil.UTF_8);
for(int i = 0; i < 20; i++)
{
s = s.replace("[" + ((char)i) + "]", i + "");
}
System.out.println(s.length() + "");
System.out.println();
System.out.println();
System.out.println(s);
}
I need a way to get the full bytebuf / bytearray when it has fully arrived at the server and get notified of that so my application can respond in a correct way according to the data the client has send.
So in short: How can I prevent fragmentation and have the channelRead event output the whole message / bytebuf.

The basic data type used by Netty is Channel Buffers or ByteBuf. This is simply a collection of bytes and nothing else. In your code you have simply used a custom handler to handle the raw incoming data. This is generally not a good practice. A very basic netty pipeline should look something like the following
So a pipeline consists of a decoder / encoder and then we have our custom handlers or logging handlers. We never really handle any raw data as is. TCP is a stream protocol. It does not identify when a specific packet ends and a new packet starts. Even if we send a very very large packet or say two individual packets, they will simply be treated as a set of bytes and when we try to read the raw set of bytes, fragmentation might happen.
So properly implement a channel pipeline which consists of a String decoder / encoder (whatever you need) and this problem will go away.

TCP provides a stream of bytes, so you can't rely on receiving a complete message in one packet. You will need a handler in your pipeline that knows how your messages are framed. Netty provides some built-in handlers that you can adapt for your protocol. See Dealing with a Stream-based Transport in the Netty User Guide.

Erlang - Data not received when issuing ssl:send(Socket, Data)

I'm using Erlang with SSL,
My server socket listens to incoming client connections and spawns a new thread for every incoming connection (assume the looping function called clientroutine())
This thread is designed based on this tutorial I found on web: http://erlycoder.com/89/erlang-ssl-sockets-example-ssl-echo-server-ssl-client-
so basically clientroutine() waits in receive, gets data from client, does some action based on received data and recursively calls itself again
Now, the problem is that when I issue ssl:send(Socket, Data), the client (Java-based) does not get anything from inputstream
Interestingly, this happens only when I recursively call clientroutine() after ssl:send like this (I skip socket close and default cases for simplicity):
clientroutine(Socket) ->
ssl:setopts(Socket, [{active, once}]),
receive
{ssl, Sock , Data} ->
ok = ssl:send(Sock, "~100 bytes list goes to client"),
clientroutine(Socket)
end.
The following works correctly (i.e. no recursion takes places and thread finishes) and my Java-client receives the string from inputstream:
clientroutine(Socket) ->
ssl:setopts(Socket, [{active, once}]),
receive
{ssl, Sock , Data} ->
ok = ssl:send(Sock, "~100 bytes list goes to client")
end.
Java-client launches inputstream listener in a separate thread like this (BufferedReader in has been declared above, among class fields):
new Thread(new Runnable(){
#Override
public void run() {
String msg;
try {
while((msg=in.readLine())!=null)
System.out.println("user received: " + msg);
} catch (IOException e) {
System.out.println("user: exception occured - inputstream reader");
}
}}).start();
I haven't yet checked if this works with Erlang client or not, I will update my post when I check it as well, but anyhow I need it to work with Java client
Any ideas why this happens?
Probably I should use some other BufferedReader routine instead of readLine(), or maybe BufferedReader requires some special character to be pushed into outputstream after the transferred message?
UPDATE. Erlang client receives everything correctly, with and without recursive call. Seems that this is somewhat related to Java inputstream

I found out that the newline character required for readLine() to fetch a line from inputstream, was (strangely) not included when I was sending my message in "recursive" version of the program, so everything goes well after I append \n to the transferred message

How to put other ongoing streamreader thread on hold

For an example if I try to build a file transfer mechanism on the server program to whack one file bytes from sender A to user B by capitalizing on their own existing connecting socket, so how could avoid it interleaving with all the ongoing chat stream coming in, i.e to keep clear wait and notify between this two threads apart from creating an extra serversocket at server side …
Many thanks
Kev

create a message protocol for your app which distinguishes between various message types. then create a "file chunk" message and a "chat" message and whatever other messages you need.
pseudo code:
// sending method
public synchronized void send(Message msg) {
// write message to socket ...
}
// receiving method
while(true) {
Message msg = readMessage(); // read message from socket
handleMessage(msg); // may want to push this work to another thread
}

Forwarding incoming TCP "Commands" to a function?

In Java, how would you set up a socket listener that listened to a socket for a series of bytes that represented a command and on recieving called a method which parsed the incoming data and invoked the appropriate command?
Clarification:
My issue is not with handling the commands (Which might also be error codes or responses to commands from the server) but with creating the socket and listening to it.
More Clarification:
What I want to do is mimic the following line of .Net (C#) code:
_stream.BeginRead(_data,0, _data.Length, new
AsyncCallback(this.StreamEventHandler), _stream);
Where:
_stream is a network stream created from a socket
_data is an array of Byte of length 9
this.StreamHandler is a delegate (function pointer) which get executed when data is read.
I am rewriting a library from C# into Java and the component I am currently writing passes commands to a server over TCPIP but also has to be able to bubble up events/responses to the layer above it.
In C# this seems to be trivial and it's looking less and less so in Java.

Starting from my other answer: The specific part you request is the one that goes into the section: "Magic goes here". It can be done in ohh so many ways, but one is:
final InputStream in = socket.getInputStream();
// This creates a new thread to service the request.
new Thread(new Runnable(){
public void run(){
byte[] retrievedData= new byte[ITEM_LENGTH];
in.read(retrievedData, 0, ITEM_LENGTH);
in.close();
// Here call your delegate or something to process the data
callSomethingWithTheData(retrievedData);
}
}).start();

Have a small main method which sets up the socket and listens for incoming connections. Pass each connection to a worker object (possibly in its own thread).
The worker object should have two APIs: The server and the client. The client API gets a connection and reads data from it, the server API takes a connection and writes data to it.
I like to keep these two in a single class because that makes it much more simple to keep the two in sync. Use a helper class to encode/decode the data for transmission, so you have single point to decide how to transmit integers, commands, options, etc.
If you want to go further, define a command class and write code to serialize that to a socket connection and read it from it. This way, you worker objects just need to declare which command class they handle and the server/client API gets even more simple (at the expense of the command class).

I would
put each command into a class of its own, where each class implements a specific interface (e.g. Command)
create a Map<String,Command> which contains a lookup table from each command string to an instance of the class that implements that command

This should help.
Lesson 1: Socket Communications

The TCP connection provides you with one InputStream and one OutputStream. You could just poll the InputStream continuously for the next command (and its inputs) on a dedicated thread. ByteBuffer.wrap(byte[] array) may be useful in interpreting the bytes as chars, ints, longs, etc. You could also pass objects around using serialization.

Any naive approach most likely will not scale well.
Consider using a REST-approach with a suitable small web-server. Jetty is usually a good choice.

To create an listen to a socket, in a very naive way:
mServerSocket = new ServerSocket(port);
listening = true;
while (listening) {
// This call blocks until a connection is made
Socket socket = serverSocket.accept();
OutputStream out = socket.getOutputStream();
InputStream in = socket.getInputStream();
// Here you do your magic, reading and writing what you need from the streams
// You would set listening to true if you have some command to close the server
// remotely
out.close();
in.close();
socket.close();
}
Normally it is a good idea to delegate the processing of the input stream to some other thread, so you can answer the next request. Otherwise, you will answer all requests serially.
You also need to define some kind of protocol of what bytes you expect on the input and output streams, but from your question it looks like you already have one.

You could create an enum with one member per command
interface Comamnd {
// whatever you expect all command to know to perform their function
void perform(Context context);
}
enum Commands implements Command{
ACTIONONE() {
void perform(Context context) {
System.out.println("Action One");
}
},
ACTIONTWO() {
void perform(Context context) {
System.out.println("Action Two");
}
}
}
// initialise
DataInputStream in = new DataInputStream(socket.getInputStream());
// in a loop
byte[] retrievedData= new byte[ITEM_LENGTH];
in.readFully(retrievedData);
String command = new String(retrievedData, 0);
Commands.valueOf(command).perform(context);