We Keep Coding

Netty, reading message results with IllegalReferenceCountgException refCnt: 0

I am new to netty and I try to receive byte response from a server (non-netty). However, I belvie I missundersteand the reference count. While reading the response I am running into the IllegalReferenceCountException: refCnt:0
What I do is not that complicated. I set up the client, then I send a message and I am awaiting a response.
My ResponseHandler looks like that:
public class ResponseHandler extends DriverResponseHandler
{
private ByteBuf received;
#Override
public void handlerAdded(ChannelHandlerContext ctx) throws Exception
{
super.handlerAdded(ctx);
received = ctx.alloc().buffer(32);
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception
{
super.channelRead(ctx, msg);
received.writeBytes((ByteBuf) msg);
}
#Override
public void channelReadComplete(ChannelHandlerContext ctx) throws Exception
{
super.channelReadComplete(ctx);
if (received.writerIndex() == received.capacity() || received.readerIndex() == received.capacity()) {
received.clear();
received = ctx.alloc().buffer(32);
}
processResponse(received);
}
}
while reading the response in channelRead the exception occurres. Why does it happen. Could someone explain it to me?

Okay, I solved the problem. Everything I need to do is add ByteToMessageDedcoder, otherwise the incoming response will never be allocated so it won't be accessible.

How to write a http proxy using netty

I want to write a simple program using netty to proxy http request send by browser.
I think it can be divided into 3 steps
get request send by browser
send it to the website
receive data from website and send it back to the browser.
Question:
How to translate url into host and port when I using Bootstrap.connect(host, port);
When I using HttpServerResponseHandler.connect and ChannelHandlerContext.writeAndFlush(httpMessage); to send data to website, how can I get the response data from the website and send it back to the browser?
It's my first day studying netty, so please try to answer as easy as possible. Thank you very much.
public class Server {
public static void main(String[] args) throws InterruptedException {
final int port = 8888;
// copy from https://github.com/netty/netty/wiki/User-guide-for-4.x
EventLoopGroup bossGroup = new NioEventLoopGroup();
EventLoopGroup workerGroup = new NioEventLoopGroup();
try {
ServerBootstrap b = new ServerBootstrap();
b.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer<SocketChannel>() {
#Override
public void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast(new HttpRequestDecoder(), new HttpServerRequestHandler());
}
})
.option(ChannelOption.SO_BACKLOG, 128)
.childOption(ChannelOption.SO_KEEPALIVE, true);
// Bind and start to accept incoming connections.
ChannelFuture f = b.bind(port).sync();
// Wait until the server socket is closed.
// In this example, this does not happen, but you can do that to gracefully
// shut down your server.
f.channel().closeFuture().sync();
} finally {
workerGroup.shutdownGracefully();
bossGroup.shutdownGracefully();
}
}
}
public class HttpServerRequestHandler extends ChannelInboundHandlerAdapter {
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
// step 1 get data from browser
if (msg instanceof LastHttpContent) {
ctx.close();
return;
}
DefaultHttpRequest httpMessage = (DefaultHttpRequest) msg;
System.out.println("浏览器请求====================");
System.out.println(msg);
System.out.println();
doWork(ctx, httpMessage);
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
cause.printStackTrace();
ctx.close();
}
private void doWork(ChannelHandlerContext ctx, final DefaultHttpRequest msg) {
// step 2 send data to website
// translate url into host and port
String host = msg.uri();
int port = 80;
if (host.startsWith("https://")) {
host = host.replaceFirst("https://", "");
port = 443;
} else if (host.startsWith("http://")) {
host = host.replaceFirst("http://", "");
port = 80;
}
if (host.contains(":443")) {
host = host.replace(":443", "");
port = 443;
}
EventLoopGroup workerGroup = new NioEventLoopGroup();
try {
Bootstrap b = new Bootstrap();
b.group(workerGroup);
b.channel(NioSocketChannel.class);
//b.option(ChannelOption.AUTO_READ, true);
b.handler(new ChannelInitializer<SocketChannel>() {
#Override
public void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast(new HttpServerResponseHandler(msg), new HttpRequestEncoder());
}
});
// question 1
ChannelFuture f = b.connect(host, port).sync();
//ChannelFuture f = b.connect("www.baidu.com", 443).sync();
f.channel().closeFuture().sync();
} catch (Exception e) {
e.printStackTrace();
} finally {
workerGroup.shutdownGracefully();
}
}
}
public class HttpServerResponseHandler extends ChannelOutboundHandlerAdapter {
private Object httpMessage;
public HttpServerResponseHandler(Object o) {
this.httpMessage = o;
}
#Override
public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) {
System.out.println("网页请求结果=========================");
System.out.println(httpMessage);
System.out.println();
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) { // (4)
cause.printStackTrace();
ctx.close();
}
#Override
public void connect(ChannelHandlerContext ctx, SocketAddress remoteAddress,
SocketAddress localAddress, ChannelPromise promise) throws Exception {
System.out.println("connect !!!!!!!!!!!");
// question 2
ctx.writeAndFlush(httpMessage);
}
}

Coincidentally, I've also been working on a Netty proxy server for the purposes of learning. I've a fully working code that you can find on my GitHub, but I'll answer your questions here. Netty also has an official proxy server example here but unlike my code, they don't have unit tests.
(FYI, My code is in Kotlin).
Core Idea:
When creating a proxy server, you need a server to accept client requests, as well as a client for the remote than you are proxying. You created the server, but not the client. It is best to reuse the EventLoop created by the server rather than creating a new one for the client. Each event loop runs on a dedicated thread, so creating more would produce additional threads, necessitating context switching when exchanging data between the accepted Channel and the client Channel.
How to translate url into host and port
To keep things simple, I've used a HttpObjectAggregator that aggregates an HttpMessage and its following HttpContents into a single FullHttpRequest or FullHttpResponse (depending on if it used to handle requests or responses). Setting the URL is trivial: just call FullHttpRequest.setUri.
To get the host and port, call Channel.remoteAddress() on the client channel and cast the resulting SocketAddress to an InetSocketAddress, from which you can get the host and port. Don't forget to similarly reset the Host header if present.
how can I get the response data
After establishing a client channel (that you're missing), you need to make a request on that channel. The client channel has a handler with a reference to the original server channel. Once the handler receives a response, it writes it to the server channel.

Netty 4.x.x WebSocket handshake for streaming server

I have a Echo server example from Official Netty
Echo Server
How to add ability to connect and streaming to it from websocket?
here is my ServerHandler code:
public class ServerHandler extends ChannelInboundHandlerAdapter
{
#Override
public void channelRegistered(ChannelHandlerContext ctx) throws Exception {
super.channelRegistered(ctx);
// !!!!! Think here should be WebSocket Handshake?
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg)
{
System.out.println(msg);
ctx.write(msg);
}
#Override
public void channelReadComplete(ChannelHandlerContext ctx)
{
ctx.flush();
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause)
{
// Close the connection when an exception is raised.
cause.printStackTrace();
}
}
Right now Chrome connection says: WebSocket connection to 'ws://127.0.0.1:8080/' failed: Error during WebSocket handshake: Invalid status line

Netty servers do not automatically handle all protocols, so you would need to add support for WebSockets.
I find the best place to start is to examine the relevant examples in Netty's xref page. Scroll down the package list until you get to the io.netty.example packages. In that list you will find a package called io.netty.example.http.websocketx.server. There is a fairly simple and well laid out example on how to implement a websocket server, or just the handler.
Websocket servers are slightly more complicated than other servers in that they must start life as an HTTP server because the protocol specifies that websockets must be initiated by "upgrading" an HTTP connection, but as I said, the example referenced above makes this fairly clear.

So, I found solution! It is not compliant to native documentation of web-socket, but who cares it works as I expected!
public void channelRead(ChannelHandlerContext ctx, Object msg)
{
DefaultHttpRequest httpRequest = null;
if (msg instanceof DefaultHttpRequest)
{
httpRequest = (DefaultHttpRequest) msg;
// Handshake
WebSocketServerHandshakerFactory wsFactory = new WebSocketServerHandshakerFactory("ws://127.0.0.1:8080/", null, false);
final Channel channel = ctx.channel();
final WebSocketServerHandshaker handshaker = wsFactory.newHandshaker(httpRequest);
if (handshaker == null) {
} else {
ChannelFuture handshake = handshaker.handshake(channel, httpRequest);
}
}
}
Do not forget to add
p.addLast(new HttpRequestDecoder(4096, 8192, 8192, false));
p.addLast(new HttpResponseEncoder());
to your pipeline.

Netty write several messages

I have the same issue than here (Netty write several messages through one channel) and I can't solve it. Here you are some cases:
I can send several messages from client-side, and the server receives only if it doesn't respond.
If I send a message to the server and it treats (with handler), generates, and writeAndFlush the response, the channel will switch to closed and I can't send more messages from client-side.
If I send a message from client with write() method, the message doesn't arrive to server because flush is necessary. It doesn't close the channel.
It looks like that the problem is on the server-side when generates a response and writeAndFlush.
I'm developing an RTSP Client/Server and this protocol is based on HTTP/1.1 Following the Netty documentation for Snoop Http Server (http://netty.io/4.0/xref/io/netty/example/http/snoop/package-summary.html), if you use the keep-alive option, the messages only write and doesn't flush. In my use of case, I repeat, the messages don't arrive to the server.
Here you are some code:
Client-Side
// Bootstrap
public RtspClient(String host, String port) throws NumberFormatException, InterruptedException{
// Configure the client.
EventLoopGroup group = new NioEventLoopGroup();
try {
b = new Bootstrap();
b.group(group)
.channel(NioSocketChannel.class)
.handler(new RtspClientInitializer());
ch = b.connect(host, Integer.parseInt(port)).sync().channel();
} finally {
// Shut down executor threads to exit.
// group.shutdownGracefully();
}
}
// Channel Pipeline definition
class RtspClientInitializer extends ChannelInitializer<SocketChannel> {
#Override
public void initChannel(SocketChannel ch) {
ChannelPipeline p = ch.pipeline();
p.addLast("encoder", new RtspRequestEncoder());
p.addLast("decoder", new RtspResponseDecoder());
p.addLast("handler", new RtspClientHandler());
}
}
// The method which sends RTSP OPTIONS protocol method
public void sendOptions() throws InterruptedException{
// HTTP/RTSP request.
DefaultFullHttpRequest request = new DefaultFullHttpRequest(
RtspVersions.RTSP_1_0, RtspMethods.OPTIONS, "*");
request.headers().set(RtspHeaders.Names.CONNECTION, RtspHeaders.Values.KEEP_ALIVE);
request.headers().set(RtspHeaders.Names.CONTENT_LENGTH, request.content().readableBytes());
request.headers().set(RtspHeaders.Names.CSEQ, this.CSeq);
this.CSeq++;
// Write to channel
System.out.println("Request: ------------------------------------------------------");
System.out.println(request.toString());
System.out.println("---------------------------------------------------------------");
ch.writeAndFlush(request);
}
Server-Side
// Bootstrap
public void initRtspServer() throws InterruptedException{
EventLoopGroup bossGroup = new NioEventLoopGroup(1);
EventLoopGroup workerGroup = new NioEventLoopGroup();
try {
ServerBootstrap b = new ServerBootstrap();
b.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.class)
.handler(new LoggingHandler(LogLevel.INFO)).childHandler(new RtspServerInitializer());
ChannelFuture cf = b.bind(this.port);
System.err.println("RTSP Server run at this URL: " +
"rtsp" + "://127.0.0.1:" + this.port + '/');
cf.channel().closeFuture().sync();
} finally {
bossGroup.shutdownGracefully();
workerGroup.shutdownGracefully();
}
// Channel Pipeline definition
class RtspServerInitializer extends ChannelInitializer<SocketChannel>{
#Override
public void initChannel(SocketChannel ch) {
ChannelPipeline p = ch.pipeline();
p.addLast("decoder",new RtspRequestDecoder());
p.addLast("encoder",new RtspResponseEncoder());
p.addLast("handler",new RtspServerHandler());
}
}
// In this case I only handle the OPTIONS method on server-side
class RtspServerHandler extends SimpleChannelInboundHandler<HttpRequest> {
private HttpRequest request;
private FullHttpResponse response;
#Override
protected void channelRead0(ChannelHandlerContext ctx, HttpRequest request)
throws Exception {
this.response = new DefaultFullHttpResponse(RTSP_1_0,OK);
String sequence = request.headers().get(RtspHeaders.Names.CSEQ);
this.response.headers().set(RtspHeaders.Names.CSEQ, sequence);
this.response.headers().set(RtspHeaders.Names.PUBLIC, "OPTIONS,SETUP,TEARDOWN,PLAY");
ctx.writeAndFlush(response).isSuccess(); // I think server closes the channel here
}
}
If I do the following in the main class from client:
RtspClient client = new RtspClient("127.0.0.1","554");
client.sendOptions();
Thread.sleep(1000);
client.sendOptions();
the second sendOptions() do nothing.
With Netty 3.2 I didn't have this problem, It works properly with send() method.

How to use Netty to handle Http Keep-Alive connections

I'm trying to write a HTTP client that uses HTTP keep-alive connections. When I connection from the ClientBoostrap I get the channel. Can I reuse this for sending multiple HTTP requests? Is there any examples demonstrating the HTTP Keep Alive functionality?
Also I have another question. Now my client works without keep-alive connections. I'm calling the channel.close in the messageReceived method of the ClientHandler. But it seems the connections are not getting closed and after some time the sockets run out and I get a BindException. Any pointers will be really appreciated.
Thanks

As long as the Connection header is not set to CLOSE (and possible the HttpVersion is 1.1, though uncertain) by a line of code similar to this...
request.setHeader(HttpHeaders.Names.CONNECTION, HttpHeaders.Values.CLOSE);
...your channel should remain open for multiple request/response pairs.
Here is some example code that I whipped up today to test it. You can bounce any number of requests off of Google prior to the channel closing:
public class TestHttpClient {
static class HttpResponseReader extends SimpleChannelUpstreamHandler {
int remainingRequests = 2;
#Override
public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) throws Exception {
HttpResponse response = (HttpResponse) e.getMessage();
System.out.println("Beginning -------------------");
System.out.println(new String(response.getContent().slice(0, 50).array()));
System.out.println("End -------------------\n");
if(remainingRequests-- > 0)
sendRequest(ctx.getChannel());
else
ctx.getChannel().close();
}
}
public static void main(String[] args) {
ClientBootstrap bootstrap = new ClientBootstrap(new NioClientSocketChannelFactory());
bootstrap.setPipeline(Channels.pipeline(
new HttpClientCodec(),
new HttpResponseReader()));
// bootstrap.setOption("child.keepAlive", true); // no apparent effect
ChannelFuture future = bootstrap.connect(new InetSocketAddress("google.com", 80));
Channel channel = future.awaitUninterruptibly().getChannel();
channel.getCloseFuture().addListener(new ChannelFutureListener() {
public void operationComplete(ChannelFuture future) throws Exception {
// this winds up getting called immediately after the receipt of the first message by HttpResponseReader!
System.out.println("Channel closed");
}
});
sendRequest(channel);
while(true) {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
private static void sendRequest(Channel channel) {
// Prepare the HTTP request.
HttpRequest request = new DefaultHttpRequest(
HttpVersion.HTTP_1_1, HttpMethod.GET, "http://www.google.com");
request.setHeader(HttpHeaders.Names.HOST, "google.com");
request.setHeader(HttpHeaders.Names.ACCEPT_ENCODING, HttpHeaders.Values.GZIP);
channel.write(request);
}
}