I'm trying to create a TCP proxy that forwards request to many other TCP endpoints using Netty/Java.
For example:
/--> SERVER A
Client A --> PROXY --
\--> SERVER B
If Client A sends a TCP command through the proxy, the proxy opens two TCP connections to Server A and Server B, and concurrently proxies the request sent by Client A to both of them.
If Client A subsequently sends another command, the proxy theoretically has previously cached the two connections in a pool, so without opening two new connections again, proxies the request to the two servers.
Regarding the response handling, I would like to have two options:
Show two responses one after the other to Client A.
Or completely ignore the response.
If a connection is lost or closed, the proxy should be able to automatically recreate it and add it back to the connection pool.
I've been looking at the Netty examples, and tried to use ChannelGroup to handle the connection pool, but without success. Also, in my code below, after sending the first request the proxy stops working. Any tips?
import io.netty.bootstrap.Bootstrap;
import io.netty.buffer.Unpooled;
import io.netty.channel.Channel;
import io.netty.channel.ChannelFuture;
import io.netty.channel.ChannelFutureListener;
import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.ChannelInboundHandlerAdapter;
import io.netty.channel.ChannelOption;
import io.netty.channel.group.ChannelGroup;
import io.netty.channel.group.DefaultChannelGroup;
import io.netty.util.concurrent.GlobalEventExecutor;
import java.util.LinkedList;
import java.util.List;
public class TcpProxyHandler extends ChannelInboundHandlerAdapter {
private static List<String> hosts = new LinkedList<>();
private static List<String> connected = new LinkedList<>();
static {
hosts.add("127.0.0.1:10000");
hosts.add("127.0.0.1:20000");
}
static final ChannelGroup channels = new DefaultChannelGroup(
GlobalEventExecutor.INSTANCE);
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
final Channel inboundChannel = ctx.channel();
for (String host : hosts) {
if (!connected.contains(host)) {
String address = host.split(":")[0];
int port = Integer.parseInt(host.split(":")[1]);
Channel outboundChannel = ConnectionPool.getConnection(address,
port);
if (outboundChannel == null) {
Bootstrap b = new Bootstrap();
b.group(inboundChannel.eventLoop())
.channel(ctx.channel().getClass())
.handler(new TcpProxyBackendHandler(inboundChannel))
.option(ChannelOption.AUTO_READ, false);
ChannelFuture f = b.connect(address, port);
outboundChannel = f.channel();
f.addListener(new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture future)
throws Exception {
if (future.isSuccess()) {
// connection complete start to read first data
inboundChannel.read();
} else {
// Close the connection if the connection
// attempt
// has failed.
inboundChannel.close();
}
}
});
channels.add(outboundChannel);
connected.add(host);
System.out.println("Connected to " + host);
}
}
}
}
#Override
public void channelRead(final ChannelHandlerContext ctx, Object msg)
throws Exception {
channels.flushAndWrite(msg);
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause)
throws Exception {
cause.printStackTrace();
ctx.close();
}
static void closeOnFlush(Channel ch) {
if (ch.isActive()) {
ch.writeAndFlush(Unpooled.EMPTY_BUFFER).addListener(
ChannelFutureListener.CLOSE);
}
}
static class TcpProxyBackendHandler extends ChannelInboundHandlerAdapter {
private final Channel inboundChannel;
public TcpProxyBackendHandler(Channel inboundChannel) {
this.inboundChannel = inboundChannel;
}
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
ctx.read();
ctx.write(Unpooled.EMPTY_BUFFER);
}
#Override
public void channelRead(final ChannelHandlerContext ctx, Object msg)
throws Exception {
inboundChannel.writeAndFlush(msg).addListener(
new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture future)
throws Exception {
if (future.isSuccess()) {
ctx.channel().read();
} else {
future.channel().close();
}
}
});
}
#Override
public void channelInactive(ChannelHandlerContext ctx) throws Exception {
TcpProxyHandler.closeOnFlush(inboundChannel);
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause)
throws Exception {
cause.printStackTrace();
TcpProxyHandler.closeOnFlush(ctx.channel());
}
}
}
You can try to call connect() and read() in another thread to give a chance to ChannelGrop's worker to do its work.
If you haven't tried yet, enable AUTO_READ again and remove the manual invokations to read(). You may initialize your server also with auto read set to false, you could try to change this too.
Related
I have following binding to handle UDP packets
private void doStartServer() {
final UDPPacketHandler udpPacketHandler = new UDPPacketHandler(messageDecodeHandler);
workerGroup = new NioEventLoopGroup(threadPoolSize);
try {
final Bootstrap bootstrap = new Bootstrap();
bootstrap
.group(workerGroup)
.handler(new LoggingHandler(nettyLevel))
.channel(NioDatagramChannel.class)
.option(ChannelOption.SO_BROADCAST, true)
.handler(udpPacketHandler);
bootstrap
.bind(serverIp, serverPort)
.sync()
.channel()
.closeFuture()
.await();
} finally {
stop();
}
}
and handler
#ChannelHandler.Sharable << note this
#Slf4j
#AllArgsConstructor
public class UDPPacketHandler extends SimpleChannelInboundHandler<DatagramPacket> {
private final MessageP54Handler messageP54Handler;
#Override
public void channelReadComplete(final ChannelHandlerContext ctx) {
ctx.flush();
}
#Override
public void exceptionCaught(final ChannelHandlerContext ctx, final Throwable cause) {
log.error("Exception in UDP handler", cause);
ctx.close();
}
}
At some point I get this exception java.net.SocketException: Network dropped connection on reset: no further information which is handled in exceptionCaught. This triggers ChannelHandlerContext to close. And at this point whole my server stops (executing on finally block from first snippet)
How to correctly handle exception so that I can handle new connections even after such exception occurs?
you shouldn't close the ChannelHandlerContext on an IOException when using a DatagramChannel. As DatagramChannel is "connection-less" the exception is specific to one "receive" or one "send" operation. So just log it (or whatever you want to do) and move on.
I'm trying to create a Netty (4.1) POC which can forward h2c (HTTP2 without TLS) frames onto a h2c server - i.e. essentially creating a Netty h2c proxy service. Wireshark shows Netty sending the frames out, and the h2c server replying (for example with the response header and data), although I'm then having a few issues receiving/processing the response HTTP frames within Netty itself.
As a starting point, I've adapted the multiplex.server example (io.netty.example.http2.helloworld.multiplex.server) so that in HelloWorldHttp2Handler, instead of responding with dummy messages, I connect to a remote node:
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
Channel remoteChannel = null;
// create or retrieve the remote channel (one to one mapping) associated with this incoming (client) channel
synchronized (lock) {
if (!ctx.channel().hasAttr(remoteChannelKey)) {
remoteChannel = this.connectToRemoteBlocking(ctx.channel());
ctx.channel().attr(remoteChannelKey).set(remoteChannel);
} else {
remoteChannel = ctx.channel().attr(remoteChannelKey).get();
}
}
if (msg instanceof Http2HeadersFrame) {
onHeadersRead(remoteChannel, (Http2HeadersFrame) msg);
} else if (msg instanceof Http2DataFrame) {
final Http2DataFrame data = (Http2DataFrame) msg;
onDataRead(remoteChannel, (Http2DataFrame) msg);
send(ctx.channel(), new DefaultHttp2WindowUpdateFrame(data.initialFlowControlledBytes()).stream(data.stream()));
} else {
super.channelRead(ctx, msg);
}
}
private void send(Channel remoteChannel, Http2Frame frame) {
remoteChannel.writeAndFlush(frame).addListener(new GenericFutureListener() {
#Override
public void operationComplete(Future future) throws Exception {
if (!future.isSuccess()) {
future.cause().printStackTrace();
}
}
});
}
/**
* If receive a frame with end-of-stream set, send a pre-canned response.
*/
private void onDataRead(Channel remoteChannel, Http2DataFrame data) throws Exception {
if (data.isEndStream()) {
send(remoteChannel, data);
} else {
// We do not send back the response to the remote-peer, so we need to release it.
data.release();
}
}
/**
* If receive a frame with end-of-stream set, send a pre-canned response.
*/
private void onHeadersRead(Channel remoteChannel, Http2HeadersFrame headers)
throws Exception {
if (headers.isEndStream()) {
send(remoteChannel, headers);
}
}
private Channel connectToRemoteBlocking(Channel clientChannel) {
try {
Bootstrap b = new Bootstrap();
b.group(new NioEventLoopGroup());
b.channel(NioSocketChannel.class);
b.option(ChannelOption.SO_KEEPALIVE, true);
b.remoteAddress("localhost", H2C_SERVER_PORT);
b.handler(new Http2ClientInitializer());
final Channel channel = b.connect().syncUninterruptibly().channel();
channel.config().setAutoRead(true);
channel.attr(clientChannelKey).set(clientChannel);
return channel;
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
When initializing the channel pipeline (in Http2ClientInitializer), if I do something like:
#Override
public void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast(Http2MultiplexCodecBuilder.forClient(new Http2OutboundClientHandler()).frameLogger(TESTLOGGER).build());
ch.pipeline().addLast(new UserEventLogger());
}
Then I can see the frames being forwarded correctly in Wireshark and the h2c server replies with the header and frame data, but Netty replies with a GOAWAY [INTERNAL_ERROR] due to:
14:23:09.324 [nioEventLoopGroup-3-1] WARN
i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was
fired, and it reached at the tail of the pipeline. It usually means
the last handler in the pipeline did not handle the exception.
java.lang.IllegalStateException: Stream object required for
identifier: 1 at
io.netty.handler.codec.http2.Http2FrameCodec$FrameListener.requireStream(Http2FrameCodec.java:587)
at
io.netty.handler.codec.http2.Http2FrameCodec$FrameListener.onHeadersRead(Http2FrameCodec.java:550)
at
io.netty.handler.codec.http2.Http2FrameCodec$FrameListener.onHeadersRead(Http2FrameCodec.java:543)...
If I instead try making it have the pipeline configuration from the http2 client example, e.g.:
#Override
public void initChannel(SocketChannel ch) throws Exception {
final Http2Connection connection = new DefaultHttp2Connection(false);
ch.pipeline().addLast(
new Http2ConnectionHandlerBuilder()
.connection(connection)
.frameLogger(TESTLOGGER)
.frameListener(new DelegatingDecompressorFrameListener(connection, new InboundHttp2ToHttpAdapterBuilder(connection)
.maxContentLength(maxContentLength)
.propagateSettings(true)
.build() ))
.build());
}
Then I instead get:
java.lang.UnsupportedOperationException: unsupported message type:
DefaultHttp2HeadersFrame (expected: ByteBuf, FileRegion) at
io.netty.channel.nio.AbstractNioByteChannel.filterOutboundMessage(AbstractNioByteChannel.java:283)
at
io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:882)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1365)
If I then add in a HTTP2 frame codec (Http2MultiplexCodec or Http2FrameCodec):
#Override
public void initChannel(SocketChannel ch) throws Exception {
final Http2Connection connection = new DefaultHttp2Connection(false);
ch.pipeline().addLast(
new Http2ConnectionHandlerBuilder()
.connection(connection)
.frameLogger(TESTLOGGER)
.frameListener(new DelegatingDecompressorFrameListener(connection, new InboundHttp2ToHttpAdapterBuilder(connection)
.maxContentLength(maxContentLength)
.propagateSettings(true)
.build() ))
.build());
ch.pipeline().addLast(Http2MultiplexCodecBuilder.forClient(new Http2OutboundClientHandler()).frameLogger(TESTLOGGER).build());
}
Then Netty sends two connection preface frames, resulting in the h2c server rejecting with GOAWAY [PROTOCOL_ERROR]:
So that is where I am having issues - i.e. configuring the remote channel pipeline such that it will send the Http2Frame objects without error, but also then receive/process them back within Netty when the response is received.
Does anyone have any ideas/suggestions please?
I ended up getting this working; the following Github issues contain some useful code/info:
Generating a Http2StreamChannel, from a Channel
A Http2Client with Http2MultiplexCode
I need to investigate a few caveats further, although the gist of the approach is that you need to wrap your channel in a Http2StreamChannel, meaning that my connectToRemoteBlocking() method ends up as:
private Http2StreamChannel connectToRemoteBlocking(Channel clientChannel) {
try {
Bootstrap b = new Bootstrap();
b.group(new NioEventLoopGroup()); // TODO reuse existing event loop
b.channel(NioSocketChannel.class);
b.option(ChannelOption.SO_KEEPALIVE, true);
b.remoteAddress("localhost", H2C_SERVER_PORT);
b.handler(new Http2ClientInitializer());
final Channel channel = b.connect().syncUninterruptibly().channel();
channel.config().setAutoRead(true);
channel.attr(clientChannelKey).set(clientChannel);
// TODO make more robust, see example at https://github.com/netty/netty/issues/8692
final Http2StreamChannelBootstrap bs = new Http2StreamChannelBootstrap(channel);
final Http2StreamChannel http2Stream = bs.open().syncUninterruptibly().get();
http2Stream.attr(clientChannelKey).set(clientChannel);
http2Stream.pipeline().addLast(new Http2OutboundClientHandler()); // will read: DefaultHttp2HeadersFrame, DefaultHttp2DataFrame
return http2Stream;
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
Then to prevent the "Stream object required for identifier: 1" error (which is essentially saying: 'This (client) HTTP2 request is new, so why do we have this specific stream?' - since we were implicitly reusing the stream object from the originally received 'server' request), we need to change to use the remote channel's stream when forwarding our data on:
private void onHeadersRead(Http2StreamChannel remoteChannel, Http2HeadersFrame headers) throws Exception {
if (headers.isEndStream()) {
headers.stream(remoteChannel.stream());
send(remoteChannel, headers);
}
}
Then the configured channel inbound handler (which I've called Http2OutboundClientHandler due to its usage) will receive the incoming HTTP2 frames in the normal way:
#Sharable
public class Http2OutboundClientHandler extends SimpleChannelInboundHandler<Http2Frame> {
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
super.exceptionCaught(ctx, cause);
cause.printStackTrace();
ctx.close();
}
#Override
public void channelRead0(ChannelHandlerContext ctx, Http2Frame msg) throws Exception {
System.out.println("Http2OutboundClientHandler Http2Frame Type: " + msg.getClass().toString());
}
}
I want to start both TCP echo server and client in one app, client after server.
What I do is:
Start a client in a ChannelFutureListener returned by server.bind().sync() like this:
public void runClientAndServer() {
server.run().addListener((ChannelFutureListener) future -> {
// client.run(); //(1) this doesn't work!
new Thread(()->client.run()).start(); //(2) this works!
});
}
and server.run() is like this:
public ChannelFuture run() {
ServerBootstrap b = new ServerBootstrap();
//doing channel config stuff
return b.bind(6666).sync();
}
and client.run() is like this:
public void run() {
Bootstrap b = new Bootstrap();
//do some config stuff
f = b.connect(host, port).sync(); //wait till connected to server
}
What happens is:
In the statement (2) that works just fine; While in the statement (1), the client sent message, that can be observed in Wireshark, and the server replies TCP ACK segment, but no channelRead() method in server side ChannelInboundHandlerAdapter is invoked, nor any attempt to write message to socket can be observed, like this capture:
wireshark capture
I guess there must be something wrong with Netty threads, but I just cannot figure out
I have prepared an example based on the newest netty version (4.1.16.Final) and the code you posted. This works fine without an extra thread maybe you did something wrong initializing your server or client.
private static final NioEventLoopGroup EVENT_LOOP_GROUP = new NioEventLoopGroup();
private static ChannelFuture runServer() throws Exception {
return new ServerBootstrap()
.group(EVENT_LOOP_GROUP)
.channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer<Channel>() {
#Override
protected void initChannel(Channel channel) throws Exception {
System.out.println("S - Channel connected: " + channel);
channel.pipeline().addLast(new SimpleChannelInboundHandler<ByteBuf>() {
#Override
protected void channelRead0(ChannelHandlerContext ctx, ByteBuf msg) throws Exception {
System.out.println("S - read: " + msg.toString(StandardCharsets.UTF_8));
}
});
}
})
.bind(6666).sync();
}
private static void runClient() throws Exception {
new Bootstrap()
.group(EVENT_LOOP_GROUP)
.channel(NioSocketChannel.class)
.handler(new ChannelInitializer<Channel>() {
#Override
protected void initChannel(Channel channel) throws Exception {
System.out.println("C - Initilized client");
channel.pipeline().addLast(new SimpleChannelInboundHandler<ByteBuf>() {
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
super.channelActive(ctx);
System.out.println("C - write: Hey this is a test message enjoy!");
ctx.writeAndFlush(Unpooled.copiedBuffer("Hey this is a test message enjoy!".getBytes(StandardCharsets.UTF_8)));
}
#Override
protected void channelRead0(ChannelHandlerContext ctx, ByteBuf msg) throws Exception { }
});
}
})
.connect("localhost", 6666).sync();
}
public static void main(String[] args) throws Exception {
runServer().addListener(future -> {
runClient();
});
}
That's what the output should look like:
C - Initilized client
C - write: Hey this is a test message enjoy!
S - Channel connected: [id: 0x1d676489, L:/127.0.0.1:6666 - R:/127.0.0.1:5079]
S - read: Hey this is a test message enjoy!
I also tried this example with a single threaded eventloopgroup which also worked fine but throw me an BlockingOperationException which did not affect the functionality of the program. If this code should work fine you should probably check your code and try to orientate your code on this example (Please don't create inline ChannelHandler for your real project like I did in this example).
I am trying to write a simple Client Server application using netty.
I followed this tutorial, and specifically the time server model along with the POJO model. My problem is with the ByteToMessageDecoder : it runs more times than it has to, meaning instead of stopping when it reads a null ByteBuf it reads one more time and for some reason, that I cannot understand, it finds the previous message that my client has sent! I am certain that the client only sends said message only once!
So the idea is this : a simple Client Server model where the Client sends a "DataPacket" with a "hello world" message in it and the server responds with a "DataPacket" with an "ACK". I am using the DataPacket type because in the future I want to pass more things in it, add a header and built something a bit more complicated...But for starters I need to see what am I doing wrong in this one...
The ERROR :
As you can see the Server starts normally, my Client (Transceiver) sends the message, the encoder activates and converts it from DataPacket to ByteBuf, the message is sent and received from the server, the Decoder from the Server activates and converts it from ByteBuf to DataPacket and then the Server handles it accordingly... It should sent the ACK and repeat the same backwards but this is were things go wrong and I cannot understand why.
I have read some posts here and already tried a LengthFieldBasedFrameDecoder, it did not work and I also want to see what is wrong with this one if possible and not use something else...
Code:
Encoder and Decoder class :
package org.client_server;
import java.util.List;
import io.netty.buffer.ByteBuf;
import io.netty.channel.ChannelHandlerContext;
import io.netty.handler.codec.ByteToMessageDecoder;
import io.netty.handler.codec.MessageToByteEncoder;
import io.netty.util.CharsetUtil;
public class EncoderDecoder {
public static class NettyEncoder extends MessageToByteEncoder<DataPacket> {
#Override
protected void encode(ChannelHandlerContext ctx, DataPacket msg, ByteBuf out)
throws Exception {
System.out.println("Encode: "+msg.getData());
out.writeBytes(msg.convertData());
}
}
public static class NettyDecoder extends ByteToMessageDecoder{
#Override
protected void decode(ChannelHandlerContext ctx, ByteBuf in,
List<Object> out) throws Exception {
if((in.readableBytes() < 4) ) {
return;
}
String msg = in.toString(CharsetUtil.UTF_8);
System.out.println("Decode:"+msg);
out.add(new DataPacket(msg));
}
}
}
Server handler :
class DataAvroHandler extends ChannelInboundHandlerAdapter {
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg)
throws Exception {
try {
DataPacket in = (DataPacket)msg;
System.out.println("[Server]: Message received..."+in.getData());
}finally {
ReferenceCountUtil.release(msg);
//ctx.close();
}
}
#Override
public void channelReadComplete(ChannelHandlerContext ctx)
throws Exception {
System.out.println("[Server]: Read Complete...");
DataPacket pkt = new DataPacket("ACK!");
//pkt.setData(Unpooled.copiedBuffer("ACK", CharsetUtil.UTF_8));
ctx.writeAndFlush(pkt);
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause)
throws Exception {
serverLog.warning("[Server]: Error..." + cause.toString());
ctx.close();
}
The Client handler :
class DataAvroHandlerCl extends ChannelInboundHandlerAdapter {
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
System.out.println("[Transceiver]: Channel Active!!!");
DataPacket pkt = new DataPacket("Hello World!");
ChannelFuture f = ctx.writeAndFlush(pkt);
//f.addListener(ChannelFutureListener.CLOSE);
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
try {
DataPacket in = (DataPacket)msg;
System.out.println("[Transceiver]: Message received..."+in.getData());
}finally {
ReferenceCountUtil.release(msg);
//ctx.close();
}
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
transLog.warning("[Transceiver] : Error..." + cause.getMessage());
ctx.close();
}
}
The Server and Client pipelines :
ch.pipeline().addLast("Decoder", new EncoderDecoder.NettyDecoder());
ch.pipeline().addLast("Encoder", new EncoderDecoder.NettyEncoder());
ch.pipeline().addLast("DataAvroHandler", new DataAvroHandler());
Your problem arises from using the toString() method of the ByteBuf in in your NettyDecoder.
Quoting from the javadoc (http://netty.io/4.0/api/io/netty/buffer/ByteBuf.html#toString%28java.nio.charset.Charset%29):
This method does not modify readerIndex or writerIndex of this buffer.
Now, the ByteToMessageDecoder doesn't know how many bytes you have actually decoded! It looks like you decoded 0 bytes, because the buffer's readerIndex was not modified and therefore you also get the error messages in your console.
You have to modify the readerIndex manually:
String msg = in.toString(CharsetUtil.UTF_8);
in.readerIndex(in.readerIndex() + in.readableBytes());
System.out.println("Decode:"+msg);
I'm getting java.io.IOException: Connection reset by peer when I try to reuse a client connection in Netty (this does not happen if I send one request, but happens every time if I send two requests, even from a single thread). My current approach involves the following implementing a simple ChannelPool whose code is below. Note that the key method obtains a free channel from the freeChannels member or creates a new channel if none are available. The method returnChannel() is the method responsible for freeing up a channel when we are done with the request. It is called inside the pipeline after we process the response (see messageReceived() method of ResponseHandler in the code below). Does anyone see what I'm doing wrong, and why I'm getting an exception?
Channel pool code (note use of freeChannels.pollFirst() to get a free channel that has been returned via a call to returnChannel()):
public class ChannelPool {
private final ClientBootstrap cb;
private Deque<Channel> freeChannels = new ArrayDeque<Channel>();
private static Map<Channel, Channel> proxyToClient = new ConcurrentHashMap<Channel, Channel>();
public ChannelPool(InetSocketAddress address, ChannelPipelineFactory pipelineFactory) {
ChannelFactory clientFactory =
new NioClientSocketChannelFactory(
Executors.newCachedThreadPool(),
Executors.newCachedThreadPool());
cb = new ClientBootstrap(clientFactory);
cb.setPipelineFactory(pipelineFactory);
}
private void writeToNewChannel(final Object writable, Channel clientChannel) {
ChannelFuture cf;
synchronized (cb) {
cf = cb.connect(new InetSocketAddress("localhost", 18080));
}
final Channel ch = cf.getChannel();
proxyToClient.put(ch, clientChannel);
cf.addListener(new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture arg0) throws Exception {
System.out.println("channel open, writing: " + ch);
ch.write(writable);
}
});
}
public void executeWrite(Object writable, Channel clientChannel) {
synchronized (freeChannels) {
while (!freeChannels.isEmpty()) {
Channel ch = freeChannels.pollFirst();
System.out.println("trying to reuse channel: " + ch + " " + ch.isOpen());
if (ch.isOpen()) {
proxyToClient.put(ch, clientChannel);
ch.write(writable).addListener(new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture cf) throws Exception {
System.out.println("write from reused channel complete, success? " + cf.isSuccess());
}
});
// EDIT: I needed a return here
}
}
}
writeToNewChannel(writable, clientChannel);
}
public void returnChannel(Channel ch) {
synchronized (freeChannels) {
freeChannels.addLast(ch);
}
}
public Channel getClientChannel(Channel proxyChannel) {
return proxyToClient.get(proxyChannel);
}
}
Netty pipeline code (Note that RequestHandler calls executeWrite() which uses either a new or an old channel, and ResponseHandler calls returnChannel() after the response is received and the content is set in the response to the client):
public class NettyExample {
private static ChannelPool pool;
public static void main(String[] args) throws Exception {
pool = new ChannelPool(
new InetSocketAddress("localhost", 18080),
new ChannelPipelineFactory() {
public ChannelPipeline getPipeline() {
return Channels.pipeline(
new HttpRequestEncoder(),
new HttpResponseDecoder(),
new ResponseHandler());
}
});
ChannelFactory factory =
new NioServerSocketChannelFactory(
Executors.newCachedThreadPool(),
Executors.newCachedThreadPool());
ServerBootstrap sb = new ServerBootstrap(factory);
sb.setPipelineFactory(new ChannelPipelineFactory() {
public ChannelPipeline getPipeline() {
return Channels.pipeline(
new HttpRequestDecoder(),
new HttpResponseEncoder(),
new RequestHandler());
}
});
sb.setOption("child.tcpNoDelay", true);
sb.setOption("child.keepAlive", true);
sb.bind(new InetSocketAddress(2080));
}
private static class ResponseHandler extends SimpleChannelHandler {
#Override
public void messageReceived(ChannelHandlerContext ctx, final MessageEvent e) {
final HttpResponse proxyResponse = (HttpResponse) e.getMessage();
final Channel proxyChannel = e.getChannel();
Channel clientChannel = pool.getClientChannel(proxyChannel);
HttpResponse clientResponse = new DefaultHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK);
clientResponse.setHeader(HttpHeaders.Names.CONTENT_TYPE, "text/html; charset=UTF-8");
HttpHeaders.setContentLength(clientResponse, proxyResponse.getContent().readableBytes());
clientResponse.setContent(proxyResponse.getContent());
pool.returnChannel(proxyChannel);
clientChannel.write(clientResponse);
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e) {
e.getCause().printStackTrace();
Channel ch = e.getChannel();
ch.close();
}
}
private static class RequestHandler extends SimpleChannelHandler {
#Override
public void messageReceived(ChannelHandlerContext ctx, final MessageEvent e) {
final HttpRequest request = (HttpRequest) e.getMessage();
pool.executeWrite(request, e.getChannel());
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e) {
e.getCause().printStackTrace();
Channel ch = e.getChannel();
ch.close();
}
}
}
EDIT: To give more detail, I've written a trace of what's happening on the proxy connection. Note that the following involves two serial requests performed by a synchronous apache commons client. The first request uses a new channel and completes fine, and the second request attempts to reuse the same channel, which is open and writable, but inexplicably fails (I've not been able to intercept any problem other than noticing the exception thrown from the worker thread). Evidently, the second request completes successfully when a retry is made. Many seconds after both requests complete, both connections finally close (i.e., even if the connection were closed by the peer, this is not reflected by any event I've intercepted):
channel open: [id: 0x6e6fbedf]
channel connect requested: [id: 0x6e6fbedf]
channel open, writing: [id: 0x6e6fbedf, /127.0.0.1:47031 => localhost/127.0.0.1:18080]
channel connected: [id: 0x6e6fbedf, /127.0.0.1:47031 => localhost/127.0.0.1:18080]
trying to reuse channel: [id: 0x6e6fbedf, /127.0.0.1:47031 => localhost/127.0.0.1:18080] true
channel open: [id: 0x3999abd1]
channel connect requested: [id: 0x3999abd1]
channel open, writing: [id: 0x3999abd1, /127.0.0.1:47032 => localhost/127.0.0.1:18080]
channel connected: [id: 0x3999abd1, /127.0.0.1:47032 => localhost/127.0.0.1:18080]
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
at sun.nio.ch.IOUtil.read(IOUtil.java:186)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:63)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:373)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:247)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Finally, figured this out. There were two issues causing a connection reset. First, I was not calling releaseConnection() from the apache commons HttpClient that was sending requests to the proxy (see the follow up question). Second, executeWrite was twice issuing the same call to the proxied server in the case the connection was reused. I needed to return after the first write rather than continuing with the while loop. The result of this double proxy call was that I was issuing duplicate responses to the original client, mangling the connection with the client.