We Keep Coding

How to write a http proxy using netty

I want to write a simple program using netty to proxy http request send by browser.
I think it can be divided into 3 steps
get request send by browser
send it to the website
receive data from website and send it back to the browser.
Question:
How to translate url into host and port when I using Bootstrap.connect(host, port);
When I using HttpServerResponseHandler.connect and ChannelHandlerContext.writeAndFlush(httpMessage); to send data to website, how can I get the response data from the website and send it back to the browser?
It's my first day studying netty, so please try to answer as easy as possible. Thank you very much.
public class Server {
public static void main(String[] args) throws InterruptedException {
final int port = 8888;
// copy from https://github.com/netty/netty/wiki/User-guide-for-4.x
EventLoopGroup bossGroup = new NioEventLoopGroup();
EventLoopGroup workerGroup = new NioEventLoopGroup();
try {
ServerBootstrap b = new ServerBootstrap();
b.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer<SocketChannel>() {
#Override
public void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast(new HttpRequestDecoder(), new HttpServerRequestHandler());
}
})
.option(ChannelOption.SO_BACKLOG, 128)
.childOption(ChannelOption.SO_KEEPALIVE, true);
// Bind and start to accept incoming connections.
ChannelFuture f = b.bind(port).sync();
// Wait until the server socket is closed.
// In this example, this does not happen, but you can do that to gracefully
// shut down your server.
f.channel().closeFuture().sync();
} finally {
workerGroup.shutdownGracefully();
bossGroup.shutdownGracefully();
}
}
}
public class HttpServerRequestHandler extends ChannelInboundHandlerAdapter {
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
// step 1 get data from browser
if (msg instanceof LastHttpContent) {
ctx.close();
return;
}
DefaultHttpRequest httpMessage = (DefaultHttpRequest) msg;
System.out.println("浏览器请求====================");
System.out.println(msg);
System.out.println();
doWork(ctx, httpMessage);
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
cause.printStackTrace();
ctx.close();
}
private void doWork(ChannelHandlerContext ctx, final DefaultHttpRequest msg) {
// step 2 send data to website
// translate url into host and port
String host = msg.uri();
int port = 80;
if (host.startsWith("https://")) {
host = host.replaceFirst("https://", "");
port = 443;
} else if (host.startsWith("http://")) {
host = host.replaceFirst("http://", "");
port = 80;
}
if (host.contains(":443")) {
host = host.replace(":443", "");
port = 443;
}
EventLoopGroup workerGroup = new NioEventLoopGroup();
try {
Bootstrap b = new Bootstrap();
b.group(workerGroup);
b.channel(NioSocketChannel.class);
//b.option(ChannelOption.AUTO_READ, true);
b.handler(new ChannelInitializer<SocketChannel>() {
#Override
public void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast(new HttpServerResponseHandler(msg), new HttpRequestEncoder());
}
});
// question 1
ChannelFuture f = b.connect(host, port).sync();
//ChannelFuture f = b.connect("www.baidu.com", 443).sync();
f.channel().closeFuture().sync();
} catch (Exception e) {
e.printStackTrace();
} finally {
workerGroup.shutdownGracefully();
}
}
}
public class HttpServerResponseHandler extends ChannelOutboundHandlerAdapter {
private Object httpMessage;
public HttpServerResponseHandler(Object o) {
this.httpMessage = o;
}
#Override
public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) {
System.out.println("网页请求结果=========================");
System.out.println(httpMessage);
System.out.println();
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) { // (4)
cause.printStackTrace();
ctx.close();
}
#Override
public void connect(ChannelHandlerContext ctx, SocketAddress remoteAddress,
SocketAddress localAddress, ChannelPromise promise) throws Exception {
System.out.println("connect !!!!!!!!!!!");
// question 2
ctx.writeAndFlush(httpMessage);
}
}

Coincidentally, I've also been working on a Netty proxy server for the purposes of learning. I've a fully working code that you can find on my GitHub, but I'll answer your questions here. Netty also has an official proxy server example here but unlike my code, they don't have unit tests.
(FYI, My code is in Kotlin).
Core Idea:
When creating a proxy server, you need a server to accept client requests, as well as a client for the remote than you are proxying. You created the server, but not the client. It is best to reuse the EventLoop created by the server rather than creating a new one for the client. Each event loop runs on a dedicated thread, so creating more would produce additional threads, necessitating context switching when exchanging data between the accepted Channel and the client Channel.
How to translate url into host and port
To keep things simple, I've used a HttpObjectAggregator that aggregates an HttpMessage and its following HttpContents into a single FullHttpRequest or FullHttpResponse (depending on if it used to handle requests or responses). Setting the URL is trivial: just call FullHttpRequest.setUri.
To get the host and port, call Channel.remoteAddress() on the client channel and cast the resulting SocketAddress to an InetSocketAddress, from which you can get the host and port. Don't forget to similarly reset the Host header if present.
how can I get the response data
After establishing a client channel (that you're missing), you need to make a request on that channel. The client channel has a handler with a reference to the original server channel. Once the handler receives a response, it writes it to the server channel.

okhttp3 websocket dynamic header

I'm trying to create a websocket and dynamically recalculate its header in every message sent. Is it possible?
I was trying to use an interceptor but is only called once.
public void run() {
// only open a websocket if there aren't websockets already open
if (this.webSocket == null || !this.openingWS) {
this.openingWS = true;
wsBuilder = new OkHttpClient.Builder();
OkHttpClient client = wsBuilder.addInterceptor(this)
.readTimeout(0, TimeUnit.MILLISECONDS)
.build();
Request request = new Request.Builder()
.url("wss://...")
.build();
client.newWebSocket(request, this);
// Trigger shutdown of the dispatcher's executor so this process can exit cleanly.
client.dispatcher().executorService().shutdown();
}
}
#Override public void onOpen(WebSocket webSocket, Response response) {
this.openingWS = false; // already open
this.webSocket = webSocket; // storing websocket for future usages
if (listener != null) listener.onWSOpen();
}
public void sendCommand(String cmd) {
System.out.println("SEND " + cmd);
if (webSocket != null) webSocket.send(cmd);
}
This same class is implementing the interceptor
public Response intercept(Chain chain) throws IOException {
Request originalRequest = chain.request();
if (!isSpecial()) return chain.proceed(originalRequest);
okhttp3.Request.Builder builder = originalRequest.newBuilder()
.addHeader("text", "...")
.addHeader("dfds", "...");
Request compressedRequest = builder.build();
return chain.proceed(compressedRequest);
}
The authentication code sent in the header will change every X seconds/minutes.
If it's not possible to change dynamically the header, what is the best way to approach this kind of connection?
Thank you for your help.

I think the headers are send only first time when you request the connection, later is depends on frames between the client and the server.
So if you want to inform the server that you had changed the header then send message with your new header. Or you can close the connection and start a new one with the new header.

Netty write several messages

I have the same issue than here (Netty write several messages through one channel) and I can't solve it. Here you are some cases:
I can send several messages from client-side, and the server receives only if it doesn't respond.
If I send a message to the server and it treats (with handler), generates, and writeAndFlush the response, the channel will switch to closed and I can't send more messages from client-side.
If I send a message from client with write() method, the message doesn't arrive to server because flush is necessary. It doesn't close the channel.
It looks like that the problem is on the server-side when generates a response and writeAndFlush.
I'm developing an RTSP Client/Server and this protocol is based on HTTP/1.1 Following the Netty documentation for Snoop Http Server (http://netty.io/4.0/xref/io/netty/example/http/snoop/package-summary.html), if you use the keep-alive option, the messages only write and doesn't flush. In my use of case, I repeat, the messages don't arrive to the server.
Here you are some code:
Client-Side
// Bootstrap
public RtspClient(String host, String port) throws NumberFormatException, InterruptedException{
// Configure the client.
EventLoopGroup group = new NioEventLoopGroup();
try {
b = new Bootstrap();
b.group(group)
.channel(NioSocketChannel.class)
.handler(new RtspClientInitializer());
ch = b.connect(host, Integer.parseInt(port)).sync().channel();
} finally {
// Shut down executor threads to exit.
// group.shutdownGracefully();
}
}
// Channel Pipeline definition
class RtspClientInitializer extends ChannelInitializer<SocketChannel> {
#Override
public void initChannel(SocketChannel ch) {
ChannelPipeline p = ch.pipeline();
p.addLast("encoder", new RtspRequestEncoder());
p.addLast("decoder", new RtspResponseDecoder());
p.addLast("handler", new RtspClientHandler());
}
}
// The method which sends RTSP OPTIONS protocol method
public void sendOptions() throws InterruptedException{
// HTTP/RTSP request.
DefaultFullHttpRequest request = new DefaultFullHttpRequest(
RtspVersions.RTSP_1_0, RtspMethods.OPTIONS, "*");
request.headers().set(RtspHeaders.Names.CONNECTION, RtspHeaders.Values.KEEP_ALIVE);
request.headers().set(RtspHeaders.Names.CONTENT_LENGTH, request.content().readableBytes());
request.headers().set(RtspHeaders.Names.CSEQ, this.CSeq);
this.CSeq++;
// Write to channel
System.out.println("Request: ------------------------------------------------------");
System.out.println(request.toString());
System.out.println("---------------------------------------------------------------");
ch.writeAndFlush(request);
}
Server-Side
// Bootstrap
public void initRtspServer() throws InterruptedException{
EventLoopGroup bossGroup = new NioEventLoopGroup(1);
EventLoopGroup workerGroup = new NioEventLoopGroup();
try {
ServerBootstrap b = new ServerBootstrap();
b.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.class)
.handler(new LoggingHandler(LogLevel.INFO)).childHandler(new RtspServerInitializer());
ChannelFuture cf = b.bind(this.port);
System.err.println("RTSP Server run at this URL: " +
"rtsp" + "://127.0.0.1:" + this.port + '/');
cf.channel().closeFuture().sync();
} finally {
bossGroup.shutdownGracefully();
workerGroup.shutdownGracefully();
}
// Channel Pipeline definition
class RtspServerInitializer extends ChannelInitializer<SocketChannel>{
#Override
public void initChannel(SocketChannel ch) {
ChannelPipeline p = ch.pipeline();
p.addLast("decoder",new RtspRequestDecoder());
p.addLast("encoder",new RtspResponseEncoder());
p.addLast("handler",new RtspServerHandler());
}
}
// In this case I only handle the OPTIONS method on server-side
class RtspServerHandler extends SimpleChannelInboundHandler<HttpRequest> {
private HttpRequest request;
private FullHttpResponse response;
#Override
protected void channelRead0(ChannelHandlerContext ctx, HttpRequest request)
throws Exception {
this.response = new DefaultFullHttpResponse(RTSP_1_0,OK);
String sequence = request.headers().get(RtspHeaders.Names.CSEQ);
this.response.headers().set(RtspHeaders.Names.CSEQ, sequence);
this.response.headers().set(RtspHeaders.Names.PUBLIC, "OPTIONS,SETUP,TEARDOWN,PLAY");
ctx.writeAndFlush(response).isSuccess(); // I think server closes the channel here
}
}
If I do the following in the main class from client:
RtspClient client = new RtspClient("127.0.0.1","554");
client.sendOptions();
Thread.sleep(1000);
client.sendOptions();
the second sendOptions() do nothing.
With Netty 3.2 I didn't have this problem, It works properly with send() method.

Server sending a greeting message with websocket and netty -> causing exception

I have a websocket server using Netty (4.0.17) that answers requests from a JavaScript client.
The communication works fine but I have an exception when I try to immediately send a greeting message when a client connects.
My code looks like that :
public class LiveOthelloWSHandler extends SimpleChannelInboundHandler<TextWebSocketFrame> {
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
super.channelActive(ctx);
ChannelFuture f = ctx.channel().writeAndFlush(new TextWebSocketFrame("(gameID=0)[LiveOthelloServer="+ VERSION_NUMBER + "]\n"));
}
// ...
#Override
protected void channelRead0(ChannelHandlerContext ctx, TextWebSocketFrame frame) throws Exception {
final String request = frame.text();
Channel thisChannel = ctx.channel();
// Do something with request
// Write back
thisChannel.writeAndFlush(new TextWebSocketFrame(response + "\n"));
}
}
The channelRead0() is ok, the client sends messages and the server answers back without any issue.
What doesn't work is the "greetings" part. I would like to send a welcoming message to the client (the string using VERSION_NUMBER in the ChannelActive() method) but I always get an exception :
java.lang.UnsupportedOperationException: unsupported message type: TextWebSocketFrame
I guess this is maybe because the channelActive() gets invoked as soon as the connection is established but before the websocket handshake is complete. How can I wait for the handshake to be finished and then send the greeting message (without the client having sent any request yet)?
For information, my initialization is:
#Override
public void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
pipeline.addLast(
new HttpRequestDecoder(),
new HttpObjectAggregator(65536),
new HttpResponseEncoder(),
new WebSocketServerProtocolHandler("/websocket"),
myLiveOthelloWSHandler);

Just RTFM...
http://netty.io/4.0/api/io/netty/handler/codec/http/websocketx/WebSocketServerProtocolHandler.html
Best way to detect handshake is to override ChannelInboundHandler.userEventTriggered...
So I just had to add:
#Override
public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception {
super.userEventTriggered(ctx, evt);
if (evt == WebSocketServerProtocolHandler.ServerHandshakeStateEvent.HANDSHAKE_COMPLETE) {
ChannelFuture f = ctx.channel().writeAndFlush(new TextWebSocketFrame("(gameID=0)[LiveOthelloServer="+ VERSION_NUMBER + "]\n"));
}
}

Simple way to use Netty to build an http proxy server?

I'm new to Netty, and am looking at using it to make a simple http proxy server that receives requests from a client, forwards the requests to another server, and then copies the response back to the response for the original request. One extra requirement is that I be able to support a timeout, so that if the proxied server takes too long to respond the proxy will respond by itself and close the connection to the proxied server.
I've already implemented such an application using Jetty, but with Jetty I need to use too many threads to keep inbound requests from getting blocked (this is a lightweight app that uses very little memory or cpu, but the latency of the proxied server is high enough that bursts in traffic cause either queueing in the proxy server, or require too many threads).
According to my understanding, I can use Netty to build a pipeline in which each stage performs a small amount of computation, then releases it's thread and waits until data is ready for the next stage in the pipeline to be executed.
My question is, is there a simple example of such an application? What I have so far is a simple modification of the server code for the basic Netty tutorial, but it lacks all support for a client. I saw the netty client tutorial, but am not sure how to mix code from the two to create a simple proxy app.
public static void main(String[] args) throws Exception {
ChannelFactory factory =
new NioServerSocketChannelFactory(
Executors.newCachedThreadPool(),
Executors.newCachedThreadPool());
ServerBootstrap bootstrap = new ServerBootstrap(factory);
bootstrap.setPipelineFactory(new ChannelPipelineFactory() {
public ChannelPipeline getPipeline() {
return Channels.pipeline(
new HttpRequestDecoder(),
new HttpResponseEncoder(),
/*
* Is there something I can put here to make a
* request to another server asynchronously and
* copy the result to the response inside
* MySimpleChannelHandler?
*/
new MySimpleChannelHandler()
);
}
});
bootstrap.setOption("child.tcpNoDelay", true);
bootstrap.setOption("child.keepAlive", true);
bootstrap.bind(new InetSocketAddress(8080));
}
private static class MySimpleChannelHandler extends SimpleChannelHandler {
#Override
public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) {
HttpRequest request = (HttpRequest) e.getMessage();
HttpResponse response = new DefaultHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK);
response.setContent(request.getContent());
Channel ch = e.getChannel();
ChannelFuture f = ch.write(response);
f.addListener(new ChannelFutureListener() {
public void operationComplete(ChannelFuture future) {
Channel ch = future.getChannel();
ch.close();
}
});
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e) {
e.getCause().printStackTrace();
Channel ch = e.getChannel();
ch.close();
}
}

you would have to look at LittleProxy to see how they did it as it is written on top of Netty.