We Keep Coding

Event driven and asynchronous serial port communication simultaneously

I'm completely new to serial port communication and need some help grasping it.
I need to communicate with a control board. This board can sometimes send events that I need to react to, and I need to send events to the board and await a response.
We have established a protocol where each event is always 12 bytes and the first 2 bytes determine the event type.
I know that when I send a specific message, I need to await a message with specific signifying bytes. At the same time I want it to be possible to react to events that are sent from the board. For instance the board might say that it is overheating, and at the same time I'm asking it to perform some command and reply.
My question is, if I write to the port and block for a second while awaiting the expected response, how I do ensure I don't "steal" the data my listener expects? E.g. do a serial ports work like a stream, where once I've read I've advanced past the point where it can be re-read.
I've done some implementation of this using jSerialComm, hopefully this can shed some light on my question.
First a listener that is registered using the addDataListener method. I want this to trigger when an event is present on the port that starts with "T".
private static LockerSerialPort getLockerSerialPort(final DeviceClient client) {
return MySerialPort.create(COM_PORT)
.addListener(EventListener.newBuilder()
.addEventHandler(createLocalEventHandler())
.build());
}
private static EventHandler createLocalEventHandler() {
return new EventHandler() {
#Override
public void execute(final byte[] event) {
System.out.println(new String(event));
}
#Override
public byte[] getEventIdentifier() {
// I want this listener to be executed when events that start with T are sent to the port
return "T".getBytes();
}
#Override
public String getName() {
return "T handler";
}
};
}
Next, I want to be able to write to the port and immediately get the response because it is needed to know if the command was successful or not.
private byte[] waitForResponse(final byte[] bytes) throws LockerException {
write(bytes);
return blockingRead();
}
private void write(final byte[] bytes) throws LockerException {
try (var out = serialPort.getOutputStream()) {
out.write(bytes);
} catch (final IOException e) {
throw Exception.from(e, "Failed to write to serial port %s", getComPort());
}
}
public byte[] blockingRead() {
return blockingRead(DEFAULT_READ_TIMEOUT);
}
private byte[] blockingRead(final int readTimeout) {
serialPort.setComPortTimeouts(SerialPort.TIMEOUT_READ_SEMI_BLOCKING, readTimeout, 0);
try {
byte[] readBuffer = new byte[PACKET_SIZE];
final int bytesRead = serialPort.readBytes(readBuffer, readBuffer.length);
if (bytesRead != PACKET_SIZE) {
throw RuntimeException.from(null, "Expected %d bytes in packet, got %d", PACKET_SIZE, bytesRead);
}
return readBuffer;
} catch (final Exception e) {
throw RuntimeException.from(e, "Failed to read packet within specified time (%d ms)", readTimeout);
}
}
When I call waitForResponse("command"), how do I know my blocking read doesn't steal data from my listener?
Are these two patterns incompatible? How would one usually handle a scenario like this?

Netty HTTP2 Frame Forwarding/Proxing - pipeline config question

I'm trying to create a Netty (4.1) POC which can forward h2c (HTTP2 without TLS) frames onto a h2c server - i.e. essentially creating a Netty h2c proxy service. Wireshark shows Netty sending the frames out, and the h2c server replying (for example with the response header and data), although I'm then having a few issues receiving/processing the response HTTP frames within Netty itself.
As a starting point, I've adapted the multiplex.server example (io.netty.example.http2.helloworld.multiplex.server) so that in HelloWorldHttp2Handler, instead of responding with dummy messages, I connect to a remote node:
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
Channel remoteChannel = null;
// create or retrieve the remote channel (one to one mapping) associated with this incoming (client) channel
synchronized (lock) {
if (!ctx.channel().hasAttr(remoteChannelKey)) {
remoteChannel = this.connectToRemoteBlocking(ctx.channel());
ctx.channel().attr(remoteChannelKey).set(remoteChannel);
} else {
remoteChannel = ctx.channel().attr(remoteChannelKey).get();
}
}
if (msg instanceof Http2HeadersFrame) {
onHeadersRead(remoteChannel, (Http2HeadersFrame) msg);
} else if (msg instanceof Http2DataFrame) {
final Http2DataFrame data = (Http2DataFrame) msg;
onDataRead(remoteChannel, (Http2DataFrame) msg);
send(ctx.channel(), new DefaultHttp2WindowUpdateFrame(data.initialFlowControlledBytes()).stream(data.stream()));
} else {
super.channelRead(ctx, msg);
}
}
private void send(Channel remoteChannel, Http2Frame frame) {
remoteChannel.writeAndFlush(frame).addListener(new GenericFutureListener() {
#Override
public void operationComplete(Future future) throws Exception {
if (!future.isSuccess()) {
future.cause().printStackTrace();
}
}
});
}
/**
* If receive a frame with end-of-stream set, send a pre-canned response.
*/
private void onDataRead(Channel remoteChannel, Http2DataFrame data) throws Exception {
if (data.isEndStream()) {
send(remoteChannel, data);
} else {
// We do not send back the response to the remote-peer, so we need to release it.
data.release();
}
}
/**
* If receive a frame with end-of-stream set, send a pre-canned response.
*/
private void onHeadersRead(Channel remoteChannel, Http2HeadersFrame headers)
throws Exception {
if (headers.isEndStream()) {
send(remoteChannel, headers);
}
}
private Channel connectToRemoteBlocking(Channel clientChannel) {
try {
Bootstrap b = new Bootstrap();
b.group(new NioEventLoopGroup());
b.channel(NioSocketChannel.class);
b.option(ChannelOption.SO_KEEPALIVE, true);
b.remoteAddress("localhost", H2C_SERVER_PORT);
b.handler(new Http2ClientInitializer());
final Channel channel = b.connect().syncUninterruptibly().channel();
channel.config().setAutoRead(true);
channel.attr(clientChannelKey).set(clientChannel);
return channel;
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
When initializing the channel pipeline (in Http2ClientInitializer), if I do something like:
#Override
public void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast(Http2MultiplexCodecBuilder.forClient(new Http2OutboundClientHandler()).frameLogger(TESTLOGGER).build());
ch.pipeline().addLast(new UserEventLogger());
}
Then I can see the frames being forwarded correctly in Wireshark and the h2c server replies with the header and frame data, but Netty replies with a GOAWAY [INTERNAL_ERROR] due to:
14:23:09.324 [nioEventLoopGroup-3-1] WARN
i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was
fired, and it reached at the tail of the pipeline. It usually means
the last handler in the pipeline did not handle the exception.
java.lang.IllegalStateException: Stream object required for
identifier: 1 at
io.netty.handler.codec.http2.Http2FrameCodec$FrameListener.requireStream(Http2FrameCodec.java:587)
at
io.netty.handler.codec.http2.Http2FrameCodec$FrameListener.onHeadersRead(Http2FrameCodec.java:550)
at
io.netty.handler.codec.http2.Http2FrameCodec$FrameListener.onHeadersRead(Http2FrameCodec.java:543)...
If I instead try making it have the pipeline configuration from the http2 client example, e.g.:
#Override
public void initChannel(SocketChannel ch) throws Exception {
final Http2Connection connection = new DefaultHttp2Connection(false);
ch.pipeline().addLast(
new Http2ConnectionHandlerBuilder()
.connection(connection)
.frameLogger(TESTLOGGER)
.frameListener(new DelegatingDecompressorFrameListener(connection, new InboundHttp2ToHttpAdapterBuilder(connection)
.maxContentLength(maxContentLength)
.propagateSettings(true)
.build() ))
.build());
}
Then I instead get:
java.lang.UnsupportedOperationException: unsupported message type:
DefaultHttp2HeadersFrame (expected: ByteBuf, FileRegion) at
io.netty.channel.nio.AbstractNioByteChannel.filterOutboundMessage(AbstractNioByteChannel.java:283)
at
io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:882)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1365)
If I then add in a HTTP2 frame codec (Http2MultiplexCodec or Http2FrameCodec):
#Override
public void initChannel(SocketChannel ch) throws Exception {
final Http2Connection connection = new DefaultHttp2Connection(false);
ch.pipeline().addLast(
new Http2ConnectionHandlerBuilder()
.connection(connection)
.frameLogger(TESTLOGGER)
.frameListener(new DelegatingDecompressorFrameListener(connection, new InboundHttp2ToHttpAdapterBuilder(connection)
.maxContentLength(maxContentLength)
.propagateSettings(true)
.build() ))
.build());
ch.pipeline().addLast(Http2MultiplexCodecBuilder.forClient(new Http2OutboundClientHandler()).frameLogger(TESTLOGGER).build());
}
Then Netty sends two connection preface frames, resulting in the h2c server rejecting with GOAWAY [PROTOCOL_ERROR]:
So that is where I am having issues - i.e. configuring the remote channel pipeline such that it will send the Http2Frame objects without error, but also then receive/process them back within Netty when the response is received.
Does anyone have any ideas/suggestions please?

I ended up getting this working; the following Github issues contain some useful code/info:
Generating a Http2StreamChannel, from a Channel
A Http2Client with Http2MultiplexCode
I need to investigate a few caveats further, although the gist of the approach is that you need to wrap your channel in a Http2StreamChannel, meaning that my connectToRemoteBlocking() method ends up as:
private Http2StreamChannel connectToRemoteBlocking(Channel clientChannel) {
try {
Bootstrap b = new Bootstrap();
b.group(new NioEventLoopGroup()); // TODO reuse existing event loop
b.channel(NioSocketChannel.class);
b.option(ChannelOption.SO_KEEPALIVE, true);
b.remoteAddress("localhost", H2C_SERVER_PORT);
b.handler(new Http2ClientInitializer());
final Channel channel = b.connect().syncUninterruptibly().channel();
channel.config().setAutoRead(true);
channel.attr(clientChannelKey).set(clientChannel);
// TODO make more robust, see example at https://github.com/netty/netty/issues/8692
final Http2StreamChannelBootstrap bs = new Http2StreamChannelBootstrap(channel);
final Http2StreamChannel http2Stream = bs.open().syncUninterruptibly().get();
http2Stream.attr(clientChannelKey).set(clientChannel);
http2Stream.pipeline().addLast(new Http2OutboundClientHandler()); // will read: DefaultHttp2HeadersFrame, DefaultHttp2DataFrame
return http2Stream;
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
Then to prevent the "Stream object required for identifier: 1" error (which is essentially saying: 'This (client) HTTP2 request is new, so why do we have this specific stream?' - since we were implicitly reusing the stream object from the originally received 'server' request), we need to change to use the remote channel's stream when forwarding our data on:
private void onHeadersRead(Http2StreamChannel remoteChannel, Http2HeadersFrame headers) throws Exception {
if (headers.isEndStream()) {
headers.stream(remoteChannel.stream());
send(remoteChannel, headers);
}
}
Then the configured channel inbound handler (which I've called Http2OutboundClientHandler due to its usage) will receive the incoming HTTP2 frames in the normal way:
#Sharable
public class Http2OutboundClientHandler extends SimpleChannelInboundHandler<Http2Frame> {
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
super.exceptionCaught(ctx, cause);
cause.printStackTrace();
ctx.close();
}
#Override
public void channelRead0(ChannelHandlerContext ctx, Http2Frame msg) throws Exception {
System.out.println("Http2OutboundClientHandler Http2Frame Type: " + msg.getClass().toString());
}
}

Netty 4 - The pool returns a channel which is not yet ready to send the the actual message

I have created an inbound handler of type SimpleChannelInboundHandler and added to pipeline. My intention is every time a connection is established, I wanted to send an application message called session open message and make the connection ready to send the actual message. To achieve this, the above inbound handler
over rides channelActive() where session open message is sent, In response to that I would get a session open confirmation message. Only after that I should be able to send any number of actual business message. I am using FixedChannelPool and initialised as follows. This works well some time on startup. But if the remote host closes the connection, after that if a message is sent calling the below sendMessage(), the message is sent even before the session open message through channelActive() and its response is obtained. So the server ignores the message as the session is not open yet when the business message was sent.
What I am looking for is, the pool should return only those channel that has called channelActive() event which has already sent the session open message and it has got its session open confirmation message from the server. How to deal with this situation?
public class SessionHandler extends SimpleChannelInboundHandler<byte[]> {
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
super.channelActive(ctx);
if (ctx.channel().isWritable()) {
ctx.channel().writeAndFlush("open session message".getBytes()).;
}
}
}
// At the time of loading the applicaiton
public void init() {
final Bootstrap bootStrap = new Bootstrap();
bootStrap.group(group).channel(NioSocketChannel.class).remoteAddress(hostname, port);
fixedPool = new FixedChannelPool(bootStrap, getChannelHandler(), 5);
// This is done to intialise connection and the channelActive() from above handler is invoked to keep the session open on startup
for (int i = 0; i < config.getMaxConnections(); i++) {
fixedPool.acquire().addListener(new FutureListener<Channel>() {
#Override
public void operationComplete(Future<Channel> future) throws Exception {
if (future.isSuccess()) {
} else {
LOGGER.error(" Channel initialzation failed...>>", future.cause());
}
}
});
}
}
//To actually send the message following method is invoked by the application.
public void sendMessage(final String businessMessage) {
fixedPool.acquire().addListener(new FutureListener<Channel>() {
#Override
public void operationComplete(Future<Channel> future) throws Exception {
if (future.isSuccess()) {
Channel channel = future.get();
if (channel.isOpen() && channel.isActive() && channel.isWritable()) {
channel.writeAndFlush(businessMessage).addListener(new GenericFutureListener<ChannelFuture>() {
#Override
public void operationComplete(ChannelFuture future) throws Exception {
if (future.isSuccess()) {
// success msg
} else {
// failure msg
}
}
});
fixedPool.release(channel);
}
} else {
// Failure
}
}
});
}

If there is no specific reason that you need to use a FixedChannelPool then you can use another data structure (List/Map) to store the Channels. You can add a channel to the data structure after sending open session message and remove it in the channelInactive method.
If you need to perform bulk operations on channels you can use a ChannelGroup for the purpose.
If you still want you use the FixedChannelPool you may set an attribute in the channel on whether open message was sent:
ctx.channel().attr(OPEN_MESSAGE_SENT).set(true);
you can get the attribute as follows in your sendMessage function:
boolean sent = ctx.channel().attr(OPEN_MESSAGE_SENT).get();
and in the channelInactive you may set the same to false or remove it.
Note OPEN_MESSAGE_SENT is an AttributeKey:
public static final AttributeKey<Boolean> OPEN_MESSAGE_SENT = AttributeKey.valueOf("OPEN_MESSAGE_SENT");

I know this is a rather old question, but I stumbled across the similar issue, not quite the same, but my issue was the ChannelInitializer in the Bootstrap.handler was never called.
The solution was to add the pipeline handlers to the pool handler's channelCreated method.
Here is my pool definition code that works now:
pool = new FixedChannelPool(httpBootstrap, new ChannelPoolHandler() {
#Override
public void channelCreated(Channel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
pipeline.addLast(HTTP_CODEC, new HttpClientCodec());
pipeline.addLast(HTTP_HANDLER, new NettyHttpClientHandler());
}
#Override
public void channelAcquired(Channel ch) {
// NOOP
}
#Override
public void channelReleased(Channel ch) {
// NOOP
}
}, 10);
So in the getChannelHandler() method I assume you're creating a ChannelPoolHandler in its channelCreated method you could send your session message (ch.writeAndFlush("open session message".getBytes());) assuming you only need to send the session message once when a connection is created, else you if you need to send the session message every time you could add it to the channelAcquired method.

Netty ByteToMessageDecoder runs more times than it has to

I am trying to write a simple Client Server application using netty.
I followed this tutorial, and specifically the time server model along with the POJO model. My problem is with the ByteToMessageDecoder : it runs more times than it has to, meaning instead of stopping when it reads a null ByteBuf it reads one more time and for some reason, that I cannot understand, it finds the previous message that my client has sent! I am certain that the client only sends said message only once!
So the idea is this : a simple Client Server model where the Client sends a "DataPacket" with a "hello world" message in it and the server responds with a "DataPacket" with an "ACK". I am using the DataPacket type because in the future I want to pass more things in it, add a header and built something a bit more complicated...But for starters I need to see what am I doing wrong in this one...
The ERROR :
As you can see the Server starts normally, my Client (Transceiver) sends the message, the encoder activates and converts it from DataPacket to ByteBuf, the message is sent and received from the server, the Decoder from the Server activates and converts it from ByteBuf to DataPacket and then the Server handles it accordingly... It should sent the ACK and repeat the same backwards but this is were things go wrong and I cannot understand why.
I have read some posts here and already tried a LengthFieldBasedFrameDecoder, it did not work and I also want to see what is wrong with this one if possible and not use something else...
Code:
Encoder and Decoder class :
package org.client_server;
import java.util.List;
import io.netty.buffer.ByteBuf;
import io.netty.channel.ChannelHandlerContext;
import io.netty.handler.codec.ByteToMessageDecoder;
import io.netty.handler.codec.MessageToByteEncoder;
import io.netty.util.CharsetUtil;
public class EncoderDecoder {
public static class NettyEncoder extends MessageToByteEncoder<DataPacket> {
#Override
protected void encode(ChannelHandlerContext ctx, DataPacket msg, ByteBuf out)
throws Exception {
System.out.println("Encode: "+msg.getData());
out.writeBytes(msg.convertData());
}
}
public static class NettyDecoder extends ByteToMessageDecoder{
#Override
protected void decode(ChannelHandlerContext ctx, ByteBuf in,
List<Object> out) throws Exception {
if((in.readableBytes() < 4) ) {
return;
}
String msg = in.toString(CharsetUtil.UTF_8);
System.out.println("Decode:"+msg);
out.add(new DataPacket(msg));
}
}
}
Server handler :
class DataAvroHandler extends ChannelInboundHandlerAdapter {
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg)
throws Exception {
try {
DataPacket in = (DataPacket)msg;
System.out.println("[Server]: Message received..."+in.getData());
}finally {
ReferenceCountUtil.release(msg);
//ctx.close();
}
}
#Override
public void channelReadComplete(ChannelHandlerContext ctx)
throws Exception {
System.out.println("[Server]: Read Complete...");
DataPacket pkt = new DataPacket("ACK!");
//pkt.setData(Unpooled.copiedBuffer("ACK", CharsetUtil.UTF_8));
ctx.writeAndFlush(pkt);
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause)
throws Exception {
serverLog.warning("[Server]: Error..." + cause.toString());
ctx.close();
}
The Client handler :
class DataAvroHandlerCl extends ChannelInboundHandlerAdapter {
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
System.out.println("[Transceiver]: Channel Active!!!");
DataPacket pkt = new DataPacket("Hello World!");
ChannelFuture f = ctx.writeAndFlush(pkt);
//f.addListener(ChannelFutureListener.CLOSE);
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
try {
DataPacket in = (DataPacket)msg;
System.out.println("[Transceiver]: Message received..."+in.getData());
}finally {
ReferenceCountUtil.release(msg);
//ctx.close();
}
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
transLog.warning("[Transceiver] : Error..." + cause.getMessage());
ctx.close();
}
}
The Server and Client pipelines :
ch.pipeline().addLast("Decoder", new EncoderDecoder.NettyDecoder());
ch.pipeline().addLast("Encoder", new EncoderDecoder.NettyEncoder());
ch.pipeline().addLast("DataAvroHandler", new DataAvroHandler());

Your problem arises from using the toString() method of the ByteBuf in in your NettyDecoder.
Quoting from the javadoc (http://netty.io/4.0/api/io/netty/buffer/ByteBuf.html#toString%28java.nio.charset.Charset%29):
This method does not modify readerIndex or writerIndex of this buffer.
Now, the ByteToMessageDecoder doesn't know how many bytes you have actually decoded! It looks like you decoded 0 bytes, because the buffer's readerIndex was not modified and therefore you also get the error messages in your console.
You have to modify the readerIndex manually:
String msg = in.toString(CharsetUtil.UTF_8);
in.readerIndex(in.readerIndex() + in.readableBytes());
System.out.println("Decode:"+msg);

AsynchronousSocketChannel#read is never completing.

I'm experimenting with NIO2 and running into an issue.
Here's the code I'm using:
ByteBuffer buffer = ByteBuffer.allocate(DEFAULT_BUFFER_SIZE);
channel.read(buffer, null, new CompletionHandler<Integer, Object>() {
#Override
public void completed(Integer result, Object attachment) {
Packet packet = new Packet(buffer.getInt(), buffer);
PacketHandler handler = PacketHandler.forOpcode(packet.getOpcode());
if(!Objects.isNull(handler)) {
handler.handle(channel, packet);
} else {
System.out.println("Unexpected opcode received from client. Opcode: " + packet.getOpcode());
}
}
#Override
public void failed(Throwable exc, Object attachment) {
System.out.println("DEBUG A");
exc.printStackTrace();
}
});
The issue is that no-matter what I send the server, it never completes. For testing purposes I have a very flat-format login packet set up and I'm sending this data through the client:
ByteBuffer buffer = ByteBuffer.allocate(28);
buffer.putInt(1); //opcode
ByteBufferUtils.putString(buffer, "admin");
ByteBufferUtils.putString(buffer, "admin");
channel.write(buffer);
Even though the client writes the data, the server never finishes reading this. I've also made sure that (DEFAULT_BUFFER_SIZE) was equal to the sent buffer size to see if that was the issue, but there were still not any changes in functionality.
Whenever I disconnect the client (Currently using a thread to keep it alive, for absolutely no reason) I get the following print stack trace from #failed
java.io.IOException: The specified network name is no longer available.
at sun.nio.ch.Iocp.translateErrorToIOException(Iocp.java:309)
at sun.nio.ch.Iocp.access$700(Iocp.java:46)
at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:399)
at java.lang.Thread.run(Thread.java:745)

You aren't sending anything. You need to flip() the buffer before calling write(), and compact() it afterwards.