We Keep Coding

Netty UDP Performance Issue

I've implemented three small UDP server. One with a plain Java DatagramSocket (threaded), one with Netty and the last one also with Netty but with a threaded message handling (because Netty doesn't support multiple threads with UDP).
After some measurements I got the following results for requests per second:
DatagramSocket ~30.000 requests/second
Netty ~1.500 requests/second
Netty (threaded): ~8.000 requests/second
The real application I have to implement must handle > 25.000 requests/second. So my question is if I make something wrong with Netty or if Netty is not designed to handle that much of connections per second?
Here are the implementations
DatagramSocket Main
public static void main(String... args) throws Exception {
final int port = Integer.parseInt(args[0]);
final int threads = Integer.parseInt(args[1]);
final int work = Integer.parseInt(args[2]);
DATAGRAM_SOCKET = new DatagramSocket(port);
for (int i = 0; i < threads; i++) {
new Thread(new Handler(work)).start();
}
}
DatagramSocket Handler
private static final class Handler implements Runnable {
private final int work;
public Handler(int work) throws SocketException {
this.work = work;
}
#Override
public void run() {
try {
while (!DATAGRAM_SOCKET.isClosed()) {
final DatagramPacket receivePacket = new DatagramPacket(new byte[1024], 1024);
DATAGRAM_SOCKET.receive(receivePacket);
final InetAddress ip = receivePacket.getAddress();
final int port = receivePacket.getPort();
final byte[] sendData = "Hey there".getBytes();
Thread.sleep(RANDOM.nextInt(work));
final DatagramPacket sendPacket = new DatagramPacket(sendData, sendData.length, ip, port);
DATAGRAM_SOCKET.send(sendPacket);
}
} catch (Exception e) {
System.out.println("ERROR: " + e.getMessage());
}
}
}
Netty Main
public static void main(String[] args) throws Exception
{
final int port = Integer.parseInt(args[0]);
final int sleep = Integer.parseInt(args[1]);
final Bootstrap bootstrap = new Bootstrap();
bootstrap.group(new NioEventLoopGroup());
bootstrap.channel(NioDatagramChannel.class);
bootstrap.handler(new MyNettyUdpHandler(sleep));
bootstrap.bind(port).sync().channel().closeFuture().sync();
}
Netty Handler (threaded)
public class MyNettyUdpHandler extends MessageToMessageDecoder<DatagramPacket> {
private final Random random = new Random(System.currentTimeMillis());
private final int sleep;
public MyNettyUdpHandler(int sleep) {
this.sleep = sleep;
}
#Override
protected void decode(ChannelHandlerContext channelHandlerContext, DatagramPacket datagramPacket, List list) throws Exception {
new Thread(() -> {
try {
Thread.sleep(random.nextInt(sleep));
} catch (InterruptedException e) {
System.out.println("ERROR while sleeping");
}
final ByteBuf buffer = Unpooled.buffer(64);
buffer.writeBytes("Hey there".getBytes());
channelHandlerContext.channel().writeAndFlush(new DatagramPacket(buffer, datagramPacket.sender()));
}).start();
}
}
The non threaded Netty Handler is the same but without the thread.

You can change your Netty decode() method like so to make it equivalent to the DatagramSocket code:
#Override
protected void decode(ChannelHandlerContext channelHandlerContext, DatagramPacket datagramPacket, List list) throws Exception {
final Channel channel = channelHandlerContext.channel();
channel.eventLoop().schedule(() -> {
final ByteBuf buffer = Unpooled.buffer(64);
buffer.writeBytes("Hey there".getBytes());
channel.writeAndFlush(new DatagramPacket(buffer, datagramPacket.sender()));
}, random.nextInt(sleep), TimeUnit.MILLISECONDS);
}
But I'm guessing the sleep() code is simulating business code you will later execute.
If that is the case make sure you don't run blocking code inside the handler.
EDIT:
To answer your question below:
You got a bit confused with the channels. You create a pipeline in the bootstrap, and you bind to some port. The returned channel is the server channel. The channel in the handlers method (your decode method in your case), is like the socket you get when you accept() in traditional socket programming. Note that port you extracted from the incoming DatagramPacket - it's roughly the same. So you send data to the client back on this channel.
The code I wrote that schedules the response is simply doing the same as what your DatagramSocket code, and the threaded netty code you wrote.
I wasn't sure why you did that, and simply assumed you have a business requirement to delay the response.
If this isn't the case, you can remove the schedule call, and your code will run much faster.
If your business logic is non-blocking, and runs in a few millis, you're done. If it's blocking, you need to try to find a non-blocking alternative, or run it in an executor, i.e. not on the event loop.
Hope this helps, even though this wasn't part of your original question. Netty is awesome, and I hate seeing bad examples and bad vibes about it so it's worth my time I guess ;)

Creating a thread in every decode() is inefficient.
You can submit the task to channel.eventLoop() as Eran said if the task is simple and won't block(In fact decode() in MesaggeToMessageDecoders is executed by the channel's EventLoop,so you need not submit it manually unless you want to shedule it).
Or you can submit the task to a ThreadPoolExecutor or EventExecutorGroup.
The latter is better because you can add listeners to the Future returned by EventExecutorGroup.submit() so you don't have to wait for the task to be completed.
My English is poor,hope these can help you.
Edit:
You can write as following,just executing the simple logic code in the EventLoop(ie.I/O thread):
#Override
protected void decode(ChannelHandlerContext channelHandlerContext, DatagramPacket datagramPacket, List list) throws Exception {
//do something simple with datagramPacket
...
final ByteBuf buffer = Unpooled.buffer(64);
buffer.writeBytes("Hey there".getBytes());
channelHandlerContext.channel().writeAndFlush(new DatagramPacket(buffer, datagramPacket.sender()));
}

Stop HttpServer after request are served

I am using a simple Http server from com.sun.net.httpserver, as described in simple HTTP server in Java using only Java SE API.
Everything works fine, however I am unsure how can I cleanly shutdown the server once I no longer need it. I do not want to stop the server while it is still sending the data for some request, only once it is idle.
My particular scenario is a desktop application performing OAuth dance. I use a local web server embedded in the application to provide a callback response, the application launches a desktop browser to show the server response using java.awt.Desktop.browse API.
I have tried calling HttpServer.stop(0) directly from my HttpHandler.handle function, after I have written the response page into the HttpExchange response output stream, but this is too early, the browser shows ERR_EMPTY_RESPONSE.
The same happens when I stop the server once my main application has finished its work - this is often too early, the HttpServer has not completed sending the data out yet at that moment.
I could provide a few seconds delay value to stop, but I would like to achieve a proper and clean synchronization.
What is a proper solution to this?

I can offer a solution based on one CountDownLatch + custom ExecutorService that is set up as executor for HttpServer:
public void runServer() throws Exception {
final ExecutorService ex = Executors.newSingleThreadExecutor();
final CountDownLatch c = new CountDownLatch(1);
HttpServer server = HttpServer.create(new InetSocketAddress(8888), 0);
server.createContext("/test", (HttpExchange h) -> {
StringBuilder resp = new StringBuilder();
for (int i = 0; i < 1_000_000; i++)
resp.append(i).append(", ");
String response = resp.toString();
h.sendResponseHeaders(200, response.length());
OutputStream os = h.getResponseBody();
os.write(response.getBytes());
os.close();
c.countDown(); // count down, letting `c.await()` to return
});
server.setExecutor(ex); // set up a custom executor for the server
server.start(); // start the server
System.out.println("HTTP server started");
c.await(); // wait until `c.countDown()` is invoked
ex.shutdown(); // send shutdown command to executor
// wait until all tasks complete (i. e. all responses are sent)
ex.awaitTermination(1, TimeUnit.HOURS);
server.stop(0);
System.out.println("HTTP server stopped");
}
I did test this on our work network environment, and it seems to work correctly. HttpServer stops no earlier than response is fully sent, so I think this is exactly what you need.
Another approach may be not shutting down the executor ex, but sending there a new task containing server.stop() after the response is written to a stream. As ex is constructed single-threaded, such task will execute not earlier than previous task completes, i. e. a response is fully sent:
public void run() throws Exception {
final ExecutorService ex = Executors.newSingleThreadExecutor();
final HttpServer server = HttpServer.create(new InetSocketAddress(8888), 0);
server.createContext("/test", (HttpExchange h) -> {
// ... generate and write a response
ex.submit(() -> {
server.stop(0);
System.out.println("HTTP server stopped");
});
});
server.setExecutor(ex);
server.start();
System.out.println("HTTP server started");
}
For more information, see ExecutorService

Netty 4.0.23 multiple hosts single client

My question is about creating multiple TCP clients to multiple hosts using the same event loop group in Netty 4.0.23 Final, I must admit that I don't quite understand Netty 4's client threading business, especially with the loads of confusing references to Netty 3.X.X implementations I hit through my research on the internet.
with the following code, I establish a connection with a single server, and send random commands using a command queue:
public class TCPsocket {
private static final CircularFifoQueue CommandQueue = new CircularFifoQueue(20);
private final EventLoopGroup workerGroup;
private final TcpClientInitializer tcpHandlerInit; // all handlers shearable
public TCPsocket() {
workerGroup = new NioEventLoopGroup();
tcpHandlerInit = new TcpClientInitializer();
}
public void connect(String host, int port) throws InterruptedException {
try {
Bootstrap b = new Bootstrap();
b.group(workerGroup);
b.channel(NioSocketChannel.class);
b.remoteAddress(host, port);
b.handler(tcpHandlerInit);
Channel ch = b.connect().sync().channel();
ChannelFuture writeCommand = null;
for (;;) {
if (!CommandQueue.isEmpty()) {
writeCommand = ch.writeAndFlush(CommandExecute()); // commandExecute() fetches a command form the commandQueue and encodes it into a byte array
}
if (CommandQueue.isFull()) { // this will never happen ... or should never happen
ch.closeFuture().sync();
break;
}
}
if (writeCommand != null) {
writeCommand.sync();
}
} finally {
workerGroup.shutdownGracefully();
}
}
public static void main(String args[]) throws InterruptedException {
TCPsocket socket = new TCPsocket();
socket.connect("192.168.0.1", 2101);
}
}
in addition to executing commands off of the command queue, this client keeps receiving periodic responses from the serve as a response to an initial command that is sent as soon as the channel becomes active, in one of the registered handlers (in TCPClientInitializer implementation), I have:
#Override
public void channelActive(ChannelHandlerContext ctx) {
ctx.writeAndFlush(firstMessage);
System.out.println("sent first message\n");
}
which activates a feature in the connected-to server, triggering a periodic packet that is returned from the server through the life span of my application.
The problem comes when I try to use this same setup to connect to multiple servers,
by looping through a string array of known server IPs:
public static void main(String args[]) throws InterruptedException {
String[] hosts = new String[]{"192.168.0.2", "192.168.0.4", "192.168.0.5"};
TCPsocket socket = new TCPsocket();
for (String host : hosts) {
socket.connect(host, 2101);
}
}
once the first connection is established, and the server (192.168.0.2) starts sending the designated periodic packets, no other connection is attempted, which (I think) is the result of the main thread waiting on the connection to die, hence never running the second iteration of the for loop, the discussion in this question leads me to think that the connection process is started in a separate thread, allowing the main thread to continue executing, but that's not what I see here, So what is actually happening? And how would I go about implementing multiple hosts connections using the same client in Netty 4.0.23 Final?
Thanks in advance

Netty Connection Retries

Retry Connection in Netty
I am building a client socket system. The requirements are:
First attemtp to connect to the remote server
When the first attempt fails keep on trying until the server is online.
I would like to know whether there is such feature in netty to do it or how best can I solve that.
Thank you very much
This is the code snippet I am struggling with:
protected void connect() throws Exception {
this.bootstrap = new ClientBootstrap(new NioClientSocketChannelFactory(
Executors.newCachedThreadPool(),
Executors.newCachedThreadPool()));
// Configure the event pipeline factory.
bootstrap.setPipelineFactory(new SmpPipelineFactory());
bootstrap.setOption("writeBufferHighWaterMark", 10 * 64 * 1024);
bootstrap.setOption("sendBufferSize", 1048576);
bootstrap.setOption("receiveBufferSize", 1048576);
bootstrap.setOption("tcpNoDelay", true);
bootstrap.setOption("keepAlive", true);
// Make a new connection.
final ChannelFuture connectFuture = bootstrap
.connect(new InetSocketAddress(config.getRemoteAddr(), config
.getRemotePort()));
channel = connectFuture.getChannel();
connectFuture.addListener(new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture future)
throws Exception {
if (connectFuture.isSuccess()) {
// Connection attempt succeeded:
// Begin to accept incoming traffic.
channel.setReadable(true);
} else {
// Close the connection if the connection attempt has
// failed.
channel.close();
logger.info("Unable to Connect to the Remote Socket server");
}
}
});
}

Assuming netty 3.x the simplest example would be:
// Configure the client.
ClientBootstrap bootstrap = new ClientBootstrap(
new NioClientSocketChannelFactory(
Executors.newCachedThreadPool(),
Executors.newCachedThreadPool()));
ChannelFuture future = null;
while (true)
{
future = bootstrap.connect(new InetSocketAddress("127.0.0.1", 80));
future.awaitUninterruptibly();
if (future.isSuccess())
{
break;
}
}
Obviously you'd want to have your own logic for the loop that set a max number of tries, etc. Netty 4.x has a slightly different bootstrap but the logic is the same. This is also synchronous, blocking, and ignores InterruptedException; in a real application you might register a ChannelFutureListener with the Future and be notified when the Future completes.
Add after OP edited question:
You have a ChannelFutureListener that is getting notified. If you want to then retry the connection you're going to have to either have that listener hold a reference to the bootstrap, or communicate back to your main thread that the connection attempt failed and have it retry the operation. If you have the listener do it (which is the simplest way) be aware that you need to limit the number of retries to prevent an infinite recursion - it's being executed in the context of the Netty worker thread. If you exhaust your retries, again, you'll need to communicate that back to your main thread; you could do that via a volatile variable, or the observer pattern could be used.
When dealing with async you really have to think concurrently. There's a number of ways to skin that particular cat.

Thank you Brian Roach. The connected variable is a volatile and can be accessed outside the code or further processing.
final InetSocketAddress sockAddr = new InetSocketAddress(
config.getRemoteAddr(), config.getRemotePort());
final ChannelFuture connectFuture = bootstrap
.connect(sockAddr);
channel = connectFuture.getChannel();
connectFuture.addListener(new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture future)
throws Exception {
if (future.isSuccess()) {
// Connection attempt succeeded:
// Begin to accept incoming traffic.
channel.setReadable(true);
connected = true;
} else {
// Close the connection if the connection attempt has
// failed.
channel.close();
if(!connected){
logger.debug("Attempt to connect within " + ((double)frequency/(double)1000) + " seconds");
try {
Thread.sleep(frequency);
} catch (InterruptedException e) {
logger.error(e.getMessage());
}
bootstrap.connect(sockAddr).addListener(this);
}
}
}
});

Netty slower than Tomcat

We just finished building a server to store data to disk and fronted it with Netty. During load testing we were seeing Netty scaling to about 8,000 messages per second. Given our systems, this looked really low. For a benchmark, we wrote a Tomcat front-end and run the same load tests. With these tests we were getting roughly 25,000 messages per second.
Here are the specs for our load testing machine:
Macbook Pro Quad core
16GB of RAM
Java 1.6
Here is the load test setup for Netty:
10 threads
100,000 messages per thread
Netty server code (pretty standard) - our Netty pipeline on the server is two handlers: a FrameDecoder and a SimpleChannelHandler that handles the request and response.
Client side JIO using Commons Pool to pool and reuse connections (the pool was sized the same as the # of threads)
Here is the load test setup for Tomcat:
10 threads
100,000 messages per thread
Tomcat 7.0.16 with default configuration using a Servlet to call the server code
Client side using URLConnection without any pooling
My main question is why such a huge different in performance? Is there something obvious with respect to Netty that can get it to run faster than Tomcat?
Edit: Here is the main Netty server code:
NioServerSocketChannelFactory factory = new NioServerSocketChannelFactory();
ServerBootstrap server = new ServerBootstrap(factory);
server.setPipelineFactory(new ChannelPipelineFactory() {
public ChannelPipeline getPipeline() {
RequestDecoder decoder = injector.getInstance(RequestDecoder.class);
ContentStoreChannelHandler handler = injector.getInstance(ContentStoreChannelHandler.class);
return Channels.pipeline(decoder, handler);
}
});
server.setOption("child.tcpNoDelay", true);
server.setOption("child.keepAlive", true);
Channel channel = server.bind(new InetSocketAddress(port));
allChannels.add(channel);
Our handlers look like this:
public class RequestDecoder extends FrameDecoder {
#Override
protected ChannelBuffer decode(ChannelHandlerContext ctx, Channel channel, ChannelBuffer buffer) {
if (buffer.readableBytes() < 4) {
return null;
}
buffer.markReaderIndex();
int length = buffer.readInt();
if (buffer.readableBytes() < length) {
buffer.resetReaderIndex();
return null;
}
return buffer;
}
}
public class ContentStoreChannelHandler extends SimpleChannelHandler {
private final RequestHandler handler;
#Inject
public ContentStoreChannelHandler(RequestHandler handler) {
this.handler = handler;
}
#Override
public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) {
ChannelBuffer in = (ChannelBuffer) e.getMessage();
in.readerIndex(4);
ChannelBuffer out = ChannelBuffers.dynamicBuffer(512);
out.writerIndex(8); // Skip the length and status code
boolean success = handler.handle(new ChannelBufferInputStream(in), new ChannelBufferOutputStream(out), new NettyErrorStream(out));
if (success) {
out.setInt(0, out.writerIndex() - 8); // length
out.setInt(4, 0); // Status
}
Channels.write(e.getChannel(), out, e.getRemoteAddress());
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e) {
Throwable throwable = e.getCause();
ChannelBuffer out = ChannelBuffers.dynamicBuffer(8);
out.writeInt(0); // Length
out.writeInt(Errors.generalException.getCode()); // status
Channels.write(ctx, e.getFuture(), out);
}
#Override
public void channelOpen(ChannelHandlerContext ctx, ChannelStateEvent e) {
NettyContentStoreServer.allChannels.add(e.getChannel());
}
}
UPDATE:
I've managed to get my Netty solution to within 4,000/second. A few weeks back I was testing a client side PING in my connection pool as a safe guard against idle sockets but I forgot to remove that code before I started load testing. This code effectively PINGed the server every time a Socket was checked out from the pool (using Commons Pool). I commented that code out and I'm now getting 21,000/second with Netty and 25,000/second with Tomcat.
Although, this is great news on the Netty side, I'm still getting 4,000/second less with Netty than Tomcat. I can post my client side (which I thought I had ruled out but apparently not) if anyone is interested in seeing that.

The method messageReceived is executed using a worker thread that is possibly getting blocked by RequestHandler#handle which may be busy doing some I/O work.
You could try adding into the channel pipeline an OrderdMemoryAwareThreadPoolExecutor (recommended) for executing the handlers or alternatively, try dispatching your handler work to a new ThreadPoolExecutor and passing a reference to the socket channel for later writing the response back to client. Ex.:
#Override
public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) {
executor.submit(new Runnable() {
processHandlerAndRespond(e);
});
}
private void processHandlerAndRespond(MessageEvent e) {
ChannelBuffer in = (ChannelBuffer) e.getMessage();
in.readerIndex(4);
ChannelBuffer out = ChannelBuffers.dynamicBuffer(512);
out.writerIndex(8); // Skip the length and status code
boolean success = handler.handle(new ChannelBufferInputStream(in), new ChannelBufferOutputStream(out), new NettyErrorStream(out));
if (success) {
out.setInt(0, out.writerIndex() - 8); // length
out.setInt(4, 0); // Status
}
Channels.write(e.getChannel(), out, e.getRemoteAddress());
}