We Keep Coding

How is Nettys FileDescriptor usage on OS X

in the PLC4X project we are using Netty for the clients to connect to PLCs which act as server. Sometimes, either by user error or by PLC error the connections are not accepted but rejected. If we retry to build up the connection ASAP multiple times, we run into the error message Too many open files.
I try to clean up everything in my code, so I would assume that there are no filedescriptors that could leak:
try {
final NioEventLoopGroup workerGroup = new NioEventLoopGroup();
Bootstrap bootstrap = new Bootstrap();
bootstrap.group(workerGroup);
bootstrap.channel(NioSocketChannel.class);
bootstrap.option(ChannelOption.SO_KEEPALIVE, true);
bootstrap.option(ChannelOption.TCP_NODELAY, true);
// TODO we should use an explicit (configurable?) timeout here
// bootstrap.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 1000);
bootstrap.handler(channelHandler);
// Start the client.
final ChannelFuture f = bootstrap.connect(address, port);
f.addListener(new GenericFutureListener<Future<? super Void>>() {
#Override public void operationComplete(Future<? super Void> future) throws Exception {
if (!future.isSuccess()) {
logger.info("Unable to connect, shutting down worker thread.");
workerGroup.shutdownGracefully();
}
}
});
// Wait for sync
f.sync();
f.awaitUninterruptibly(); // jf: unsure if we need that
// Wait till the session is finished initializing.
return f.channel();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new PlcConnectionException("Error creating channel.", e);
} catch (Exception e) {
throw new PlcConnectionException("Error creating channel.", e);
}
From my understanding, the Listener should always shutdown the group and free up all descriptors used.
But in reality, when running it on macOS Catalina I see that about 1% of the fails that its not due to "rejection" but due to "Too many open files".
Is this a ulimit thing, as Netty (on macOS) simply needs a number of fd's to use? Or am I leaking something?
Thanks for clarification!

I found out the solution, kind of myself.
There are 2 issues (probably even 3) in an original implementation, which are not really related to Mac OS X:
connect and addListener should be chained
workerGroup.shutdownGracefully() is triggered in another thread, so the main (called) thread already finishes
its not awaited that the workerGroup really finishes.
This together can lead to situations as it seems, where new groups are spawned faster than old groups are closed.
Thus, I changed the implementation to
try {
final NioEventLoopGroup workerGroup = new NioEventLoopGroup();
Bootstrap bootstrap = new Bootstrap();
bootstrap.group(workerGroup);
bootstrap.channel(NioSocketChannel.class);
bootstrap.option(ChannelOption.SO_KEEPALIVE, true);
bootstrap.option(ChannelOption.TCP_NODELAY, true);
// TODO we should use an explicit (configurable?) timeout here
// bootstrap.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 1000);
bootstrap.handler(channelHandler);
// Start the client.
logger.trace("Starting connection attempt on tcp layer to {}:{}", address.getHostAddress(), port);
final ChannelFuture f = bootstrap.connect(address, port);
// Wait for sync
try {
f.sync();
} catch (Exception e) {
// Shutdown worker group here and wait for it
logger.info("Unable to connect, shutting down worker thread.");
workerGroup.shutdownGracefully().awaitUninterruptibly();
logger.debug("Worker Group is shutdown successfully.");
throw new PlcConnectionException("Unable to Connect on TCP Layer to " + address.getHostAddress() + ":" + port, e);
}
// Wait till the session is finished initializing.
return f.channel();
}
catch (Exception e) {
throw new PlcConnectionException("Error creating channel.", e);
}
which adresses the issues above. Thus, the call only finishes when its properly cleaned up.
My tests now show a constant number of open file descriptors.

Netty - how to survive DDOS?

I'm using netty 4.1 as NIO socket server for MMORPG game. It was running perfectly for years but recently we are suffering from DDOS attacks. I was fighting it for a long time but currently, I don't have any more ideas on how could I improve it. Ddoser is spamming with new connections from thousands of ips from all over the world. It's difficult to cut it on the network level because attacks look very similar to normal players. Attacks are not very big compared to attacks on HTTP servers but big enough to crash our game.
How i'm using netty:
public void startServer() {
bossGroup = new NioEventLoopGroup(1);
workerGroup = new NioEventLoopGroup();
try {
int timeout = (Settings.SOCKET_TIMEOUT*1000);
bootstrap = new ServerBootstrap();
int bufferSize = 65536;
bootstrap.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.class)
.childOption(ChannelOption.SO_KEEPALIVE, true)
.childOption(ChannelOption.SO_TIMEOUT, timeout)
.childOption(ChannelOption.SO_RCVBUF, bufferSize)
.childOption(ChannelOption.SO_SNDBUF, bufferSize)
.handler(new LoggingHandler(LogLevel.INFO))
.childHandler(new CustomInitalizer(sslCtx));
ChannelFuture bind = bootstrap.bind(DrServerAdmin.port);
bossChannel = bind.sync();
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
bossGroup.shutdownGracefully();
workerGroup.shutdownGracefully();
}
}
Initalizer:
public class CustomInitalizer extends ChannelInitializer<SocketChannel> {
public static DefaultEventExecutorGroup normalGroup = new DefaultEventExecutorGroup(16);
public static DefaultEventExecutorGroup loginGroup = new DefaultEventExecutorGroup(8);
public static DefaultEventExecutorGroup commandsGroup = new DefaultEventExecutorGroup(4);
private final SslContext sslCtx;
public CustomInitalizer(SslContext sslCtx) {
this.sslCtx = sslCtx;
}
#Override
public void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
if (sslCtx != null) {
pipeline.addLast(sslCtx.newHandler(ch.alloc()));
}
pipeline.addLast(new CustomFirewall()); //it is AbstractRemoteAddressFilter<InetSocketAddress>
int limit = 32768;
pipeline.addLast(new DelimiterBasedFrameDecoder(limit, Delimiters.nulDelimiter()));
pipeline.addLast("decoder", new StringDecoder(CharsetUtil.UTF_8));
pipeline.addLast("encoder", new StringEncoder(CharsetUtil.UTF_8));
pipeline.addLast(new CustomReadTimeoutHandler(Settings.SOCKET_TIMEOUT));
int id = DrServerNetty.getDrServer().getIdClient();
CustomHandler normalHandler = new CustomHandler();
FlashClientNetty client = new FlashClientNetty(normalHandler,id);
normalHandler.setClient(client);
pipeline.addLast(normalGroup,"normalHandler",normalHandler);
CustomLoginHandler loginHandler = new CustomLoginHandler(client);
pipeline.addLast(loginGroup,"loginHandler",loginHandler);
CustomCommandsHandler commandsHandler = new CustomCommandsHandler(loginHandler.client);
pipeline.addLast(commandsGroup, "commandsHandler", commandsHandler);
}
}
I'm using 5 groups:
bootstrap bossGroup - for new connections
bootstrap workerGroup - for delivering messages
normalGroup - for most messages
loginGroup - for heavy login process
commands group - for some heavy logic
I'm monitoring the number of new connections and messages so I can immediately find out if there is an attack going. During the attack I'm not accepting new connections anymore: I'm returning false in the custom firewall ( AbstractRemoteAddressFilter ).
protected boolean accept(ChannelHandlerContext ctx, InetSocketAddress remoteAddress) throws Exception {
if(ddosDetected())
return false;
else
return true;
}
But even that I'm dropping new connections right away my workgroup is getting overloaded. PendingTasks for worker group (all other groups are fine) are growing which causes longer and longer communications for normal players and finally, they get kicked by socket_timeouts. I'm not sure why is it happen. During normal server usage, the busiest groups are login and normal group. On network level server is fine - it's using just ~10% of its bandwidth limit. CPU and RAM usage also isn't very high during the attack. But after a few minutes of such an attack, all my players are kicked out from the game and are not able to connect anymore.
Is there any better way to instantly drop all incoming connections and protect users that are aready connected?

I think you will need to "fix this" on the kernel level via for example iptables. Otherwise you can only close the connection after you already accept it which sounds not good enough in this case.

Netty Add Channel to ServerBootstrap

I have a ServerBootstrap accepting data from clients. Most of them are from any endpoint that connects to it, however I also want to handle data coming from a specific endpoint.
I'm reading and writing strings from n+1 connections basically. If the one specific connection ever closes, I would need to reopen it again.
Currently I'm trying to get a Bootstrap connected to the specific endpoint, and a ServerBootstrap handling all of the incoming connections, but the sync() that starts one of the Bootstraps blocks the rest of the application and I can't run the other one.
Or is it possible to just create a channel from scratch, connect to it, and add it to the EventLoopGroup?
Here's an example of what I have so far. Currently startServer() blocks at channelfuture.channel().closeFuture().sync()
private Channel mChannel;
private EventLoopGroup mListeningGroup;
private EventLoopGroup mSpeakingGroup;
public void startServer() {
try {
ServerBootstrap bootstrap = new ServerBootstrap()
.group(mListeningGroup, mSpeakingGroup)
.channel(NioServerSocketChannel.class)
.option(ChannelOption.SO_BACKLOG, 1024)
.childOption(ChannelOption.SO_KEEPALIVE, true)
.childHandler(new ServerInitializer());
ChannelFuture channelFuture = bootstrap.bind(mListeningPort).sync();
channelFuture.channel().closeFuture().sync();
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
mListeningGroup.shutdownGracefully();
mSpeakingGroup.shutdownGracefully();
}
}
public void startClient() throws InterruptedException {
Bootstrap bootstrap = new Bootstrap()
.group(mSpeakingGroup)
.channel(NioSocketChannel.class)
.option(ChannelOption.SO_BACKLOG, 1024)
.option(ChannelOption.TCP_NODELAY, true)
.option(ChannelOption.SO_KEEPALIVE, true)
.handler(new ClientInitializer());
ChannelFuture future = bootstrap.connect(mAddress,mPort).sync();
mChannel = future.channel();
mChannel.closeFuture().addListener((ChannelFutureListener) futureListener -> mChannel = null).sync();
}
Once data is read by any of the n+1 sockets it puts it's message into a PriorityQueue and a while loops continuously pops off the queue and writes the data to every Channel. Does anyone have any ideas in regards to the best way to approach this?

Netty: channel.write hangs on disconnect

I have the following situation:
A new channel connection is opened in this way:
ClientBootstrap bootstrap = new ClientBootstrap(
new OioClientSocketChannelFactory(Executors.newCachedThreadPool()));
icapClientChannelPipeline = new ICAPClientChannelPipeline();
bootstrap.setPipelineFactory(icapClientChannelPipeline);
ChannelFuture future = bootstrap.connect(new InetSocketAddress(host, port));
channel = future.awaitUninterruptibly().getChannel();
This is working as expected.
Stuff is written to the channel in the following way:
channel.write(chunk)
This also works as expected when the connection to the server is still alive. But if the server goes down (machine goes offline), the call hangs and doesn't return.
I confirmed this by adding log statements before and after the channel.write(chunk). When the connection is broken, only the log statement before is displayed.
What is causing this? I thought these calls are all async and return immediately? I also tried with NioClientSocketChannelFactory, same behavior.
I tried to use channel.getCloseFuture() but the listener never gets called, I tried to check the channel before writing with channel.isOpen(), channel.isConnected() and channel.isWritable() and they are always true...
How to work around this? No exception is thrown and nothing really happens... Some questions like this one and this one indicate that it isn't possible to detect a channel disconnect without a heartbeat. But I can't implement a heartbeat because I can't change the server side.
Environment: Netty 3, JDK 1.7

Ok, I solved this one on my own last week so I'll add the answer for completness.
I was wrong in 3. because I thought I'll have to change both the client and the server side for a heartbeat. As described in this question you can use the IdleStateAwareHandler for this purpose. I implemented it like this:
The IdleStateAwareHandler:
public class IdleStateAwareHandler extends IdleStateAwareChannelHandler {
#Override
public void channelIdle(ChannelHandlerContext ctx, IdleStateEvent e) {
if (e.getState() == IdleState.READER_IDLE) {
e.getChannel().write("heartbeat-reader_idle");
}
else if (e.getState() == IdleState.WRITER_IDLE) {
Logger.getLogger(IdleStateAwareHandler.class.getName()).log(
Level.WARNING, "WriteIdle detected, closing channel");
e.getChannel().close();
e.getChannel().write("heartbeat-writer_idle");
}
else if (e.getState() == IdleState.ALL_IDLE) {
e.getChannel().write("heartbeat-all_idle");
}
}
}
The PipeLine:
public class ICAPClientChannelPipeline implements ICAPClientPipeline {
ICAPClientHandler icapClientHandler;
ChannelPipeline pipeline;
public ICAPClientChannelPipeline(){
icapClientHandler = new ICAPClientHandler();
pipeline = pipeline();
pipeline.addLast("idleStateHandler", new IdleStateHandler(new HashedWheelTimer(10, TimeUnit.MILLISECONDS), 5, 5, 5));
pipeline.addLast("idleStateAwareHandler", new IdleStateAwareHandler());
pipeline.addLast("encoder",new IcapRequestEncoder());
pipeline.addLast("chunkSeparator",new IcapChunkSeparator(1024*4));
pipeline.addLast("decoder",new IcapResponseDecoder());
pipeline.addLast("chunkAggregator",new IcapChunkAggregator(1024*4));
pipeline.addLast("handler", icapClientHandler);
}
#Override
public ChannelPipeline getPipeline() throws Exception {
return pipeline;
}
}
This detects any read or write idle state on the channel after 5 seconds.
As you can see it is a little bit ICAP-specific but this doesn't matter for the question.
To react to an idle event I need the following listener:
channel.getCloseFuture().addListener(new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture future) throws Exception {
doSomething();
}
});

Netty Connection Retries

Retry Connection in Netty
I am building a client socket system. The requirements are:
First attemtp to connect to the remote server
When the first attempt fails keep on trying until the server is online.
I would like to know whether there is such feature in netty to do it or how best can I solve that.
Thank you very much
This is the code snippet I am struggling with:
protected void connect() throws Exception {
this.bootstrap = new ClientBootstrap(new NioClientSocketChannelFactory(
Executors.newCachedThreadPool(),
Executors.newCachedThreadPool()));
// Configure the event pipeline factory.
bootstrap.setPipelineFactory(new SmpPipelineFactory());
bootstrap.setOption("writeBufferHighWaterMark", 10 * 64 * 1024);
bootstrap.setOption("sendBufferSize", 1048576);
bootstrap.setOption("receiveBufferSize", 1048576);
bootstrap.setOption("tcpNoDelay", true);
bootstrap.setOption("keepAlive", true);
// Make a new connection.
final ChannelFuture connectFuture = bootstrap
.connect(new InetSocketAddress(config.getRemoteAddr(), config
.getRemotePort()));
channel = connectFuture.getChannel();
connectFuture.addListener(new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture future)
throws Exception {
if (connectFuture.isSuccess()) {
// Connection attempt succeeded:
// Begin to accept incoming traffic.
channel.setReadable(true);
} else {
// Close the connection if the connection attempt has
// failed.
channel.close();
logger.info("Unable to Connect to the Remote Socket server");
}
}
});
}

Assuming netty 3.x the simplest example would be:
// Configure the client.
ClientBootstrap bootstrap = new ClientBootstrap(
new NioClientSocketChannelFactory(
Executors.newCachedThreadPool(),
Executors.newCachedThreadPool()));
ChannelFuture future = null;
while (true)
{
future = bootstrap.connect(new InetSocketAddress("127.0.0.1", 80));
future.awaitUninterruptibly();
if (future.isSuccess())
{
break;
}
}
Obviously you'd want to have your own logic for the loop that set a max number of tries, etc. Netty 4.x has a slightly different bootstrap but the logic is the same. This is also synchronous, blocking, and ignores InterruptedException; in a real application you might register a ChannelFutureListener with the Future and be notified when the Future completes.
Add after OP edited question:
You have a ChannelFutureListener that is getting notified. If you want to then retry the connection you're going to have to either have that listener hold a reference to the bootstrap, or communicate back to your main thread that the connection attempt failed and have it retry the operation. If you have the listener do it (which is the simplest way) be aware that you need to limit the number of retries to prevent an infinite recursion - it's being executed in the context of the Netty worker thread. If you exhaust your retries, again, you'll need to communicate that back to your main thread; you could do that via a volatile variable, or the observer pattern could be used.
When dealing with async you really have to think concurrently. There's a number of ways to skin that particular cat.

Thank you Brian Roach. The connected variable is a volatile and can be accessed outside the code or further processing.
final InetSocketAddress sockAddr = new InetSocketAddress(
config.getRemoteAddr(), config.getRemotePort());
final ChannelFuture connectFuture = bootstrap
.connect(sockAddr);
channel = connectFuture.getChannel();
connectFuture.addListener(new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture future)
throws Exception {
if (future.isSuccess()) {
// Connection attempt succeeded:
// Begin to accept incoming traffic.
channel.setReadable(true);
connected = true;
} else {
// Close the connection if the connection attempt has
// failed.
channel.close();
if(!connected){
logger.debug("Attempt to connect within " + ((double)frequency/(double)1000) + " seconds");
try {
Thread.sleep(frequency);
} catch (InterruptedException e) {
logger.error(e.getMessage());
}
bootstrap.connect(sockAddr).addListener(this);
}
}
}
});