I want to write a simple program using netty to proxy http request send by browser.
I think it can be divided into 3 steps
get request send by browser
send it to the website
receive data from website and send it back to the browser.
Question:
How to translate url into host and port when I using Bootstrap.connect(host, port);
When I using HttpServerResponseHandler.connect and ChannelHandlerContext.writeAndFlush(httpMessage); to send data to website, how can I get the response data from the website and send it back to the browser?
It's my first day studying netty, so please try to answer as easy as possible. Thank you very much.
public class Server {
public static void main(String[] args) throws InterruptedException {
final int port = 8888;
// copy from https://github.com/netty/netty/wiki/User-guide-for-4.x
EventLoopGroup bossGroup = new NioEventLoopGroup();
EventLoopGroup workerGroup = new NioEventLoopGroup();
try {
ServerBootstrap b = new ServerBootstrap();
b.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer<SocketChannel>() {
#Override
public void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast(new HttpRequestDecoder(), new HttpServerRequestHandler());
}
})
.option(ChannelOption.SO_BACKLOG, 128)
.childOption(ChannelOption.SO_KEEPALIVE, true);
// Bind and start to accept incoming connections.
ChannelFuture f = b.bind(port).sync();
// Wait until the server socket is closed.
// In this example, this does not happen, but you can do that to gracefully
// shut down your server.
f.channel().closeFuture().sync();
} finally {
workerGroup.shutdownGracefully();
bossGroup.shutdownGracefully();
}
}
}
public class HttpServerRequestHandler extends ChannelInboundHandlerAdapter {
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
// step 1 get data from browser
if (msg instanceof LastHttpContent) {
ctx.close();
return;
}
DefaultHttpRequest httpMessage = (DefaultHttpRequest) msg;
System.out.println("浏览器请求====================");
System.out.println(msg);
System.out.println();
doWork(ctx, httpMessage);
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
cause.printStackTrace();
ctx.close();
}
private void doWork(ChannelHandlerContext ctx, final DefaultHttpRequest msg) {
// step 2 send data to website
// translate url into host and port
String host = msg.uri();
int port = 80;
if (host.startsWith("https://")) {
host = host.replaceFirst("https://", "");
port = 443;
} else if (host.startsWith("http://")) {
host = host.replaceFirst("http://", "");
port = 80;
}
if (host.contains(":443")) {
host = host.replace(":443", "");
port = 443;
}
EventLoopGroup workerGroup = new NioEventLoopGroup();
try {
Bootstrap b = new Bootstrap();
b.group(workerGroup);
b.channel(NioSocketChannel.class);
//b.option(ChannelOption.AUTO_READ, true);
b.handler(new ChannelInitializer<SocketChannel>() {
#Override
public void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast(new HttpServerResponseHandler(msg), new HttpRequestEncoder());
}
});
// question 1
ChannelFuture f = b.connect(host, port).sync();
//ChannelFuture f = b.connect("www.baidu.com", 443).sync();
f.channel().closeFuture().sync();
} catch (Exception e) {
e.printStackTrace();
} finally {
workerGroup.shutdownGracefully();
}
}
}
public class HttpServerResponseHandler extends ChannelOutboundHandlerAdapter {
private Object httpMessage;
public HttpServerResponseHandler(Object o) {
this.httpMessage = o;
}
#Override
public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) {
System.out.println("网页请求结果=========================");
System.out.println(httpMessage);
System.out.println();
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) { // (4)
cause.printStackTrace();
ctx.close();
}
#Override
public void connect(ChannelHandlerContext ctx, SocketAddress remoteAddress,
SocketAddress localAddress, ChannelPromise promise) throws Exception {
System.out.println("connect !!!!!!!!!!!");
// question 2
ctx.writeAndFlush(httpMessage);
}
}
Coincidentally, I've also been working on a Netty proxy server for the purposes of learning. I've a fully working code that you can find on my GitHub, but I'll answer your questions here. Netty also has an official proxy server example here but unlike my code, they don't have unit tests.
(FYI, My code is in Kotlin).
Core Idea:
When creating a proxy server, you need a server to accept client requests, as well as a client for the remote than you are proxying. You created the server, but not the client. It is best to reuse the EventLoop created by the server rather than creating a new one for the client. Each event loop runs on a dedicated thread, so creating more would produce additional threads, necessitating context switching when exchanging data between the accepted Channel and the client Channel.
How to translate url into host and port
To keep things simple, I've used a HttpObjectAggregator that aggregates an HttpMessage and its following HttpContents into a single FullHttpRequest or FullHttpResponse (depending on if it used to handle requests or responses). Setting the URL is trivial: just call FullHttpRequest.setUri.
To get the host and port, call Channel.remoteAddress() on the client channel and cast the resulting SocketAddress to an InetSocketAddress, from which you can get the host and port. Don't forget to similarly reset the Host header if present.
how can I get the response data
After establishing a client channel (that you're missing), you need to make a request on that channel. The client channel has a handler with a reference to the original server channel. Once the handler receives a response, it writes it to the server channel.
Related
1. Backgroud
I found most of the http client examples using Netty seem to be following this code structure:
public void run() {
EventLoopGroup group = new NioEventLoopGroup();
try {
Bootstrap b = new Bootstrap();
b.group(group)
.channel(NioSocketChannel.class)
.handler(new HttpSnoopClientInitializer(sslCtx));
// Make the connection attempt.
Channel ch = b.connect(host, port).sync().channel();
// send something
ch.writeAndFlush(XXXX);
// Wait for the server to close the connection.
ch.closeFuture().sync();
} finally {
// Shut down executor threads to exit.
group.shutdownGracefully();
}
}
So if I understand it correctly, each time I send a request, I need to create a client, and call client.run() on it. That said, it seems I can only make one "fixed" request at a time.
2.My need
I need a long-standing client that can send multiple requests. More specifically, there will be another thread sending instruction to the client, and each time the client gets the instruction, it will send a request. Something like this:
Client client = new Client();
client.start();
client.sendRequest(request1);
client.sendRequest(request2);
...
client.shutDownGracefully(); // not sure if this shutdown is necessary or not
// because I need a long-standing client to wait for instructions to send requests
// in this sense it's kinda like a server.
3.What I've tried
I have tried something like this: from this link
public MyClient(String host, int port) {
System.out.println("Initializing client and connecting to server..");
EventLoopGroup workerGroup = new NioEventLoopGroup();
Bootstrap b = new Bootstrap();
b.group(workerGroup)
.channel(NioSocketChannel.class)
.option(ChannelOption.SO_KEEPALIVE, true)
.handler(new ChannelInitializer<SocketChannel>() {
#Override
protected void initChannel(SocketChannel channel) throws Exception {
channel.pipeline().addLast(new StringDecoder());
channel.pipeline().addLast(new StringEncoder());
channel.pipeline().addLast(new MyAppClientHandler());
}
});
channelFuture = b.connect(host, port);
}
public ResponseFuture send(final String msg) {
final ResponseFuture responseFuture = new ResponseFuture();
channelFuture.addListener(new GenericFutureListener<ChannelFuture>() {
#Override
public void operationComplete(ChannelFuture future)
throws Exception {
channelFuture.channel().pipeline().get(MyAppClientHandler.class).setResponseFuture(responseFuture);
channelFuture.channel().writeAndFlush(msg);
}
});
return responseFuture;
}
public void close() {
channelFuture.channel().close();
}
The problem is that it seems this code didn't call workerGroup.shutDownGracefully(), so I'm guessing this can have problems. Is there a way to split the "start client", "send request", "shutdown client" into separate methods? Thanks in advance!
The easiest solution will be to use a ChannelPool provided by netty: https://netty.io/news/2015/05/07/4-0-28-Final.html
It provides gracefulshutdown, maxconnections, etc out-of-the-box.
I have a Echo server example from Official Netty
Echo Server
How to add ability to connect and streaming to it from websocket?
here is my ServerHandler code:
public class ServerHandler extends ChannelInboundHandlerAdapter
{
#Override
public void channelRegistered(ChannelHandlerContext ctx) throws Exception {
super.channelRegistered(ctx);
// !!!!! Think here should be WebSocket Handshake?
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg)
{
System.out.println(msg);
ctx.write(msg);
}
#Override
public void channelReadComplete(ChannelHandlerContext ctx)
{
ctx.flush();
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause)
{
// Close the connection when an exception is raised.
cause.printStackTrace();
}
}
Right now Chrome connection says: WebSocket connection to 'ws://127.0.0.1:8080/' failed: Error during WebSocket handshake: Invalid status line
Netty servers do not automatically handle all protocols, so you would need to add support for WebSockets.
I find the best place to start is to examine the relevant examples in Netty's xref page. Scroll down the package list until you get to the io.netty.example packages. In that list you will find a package called io.netty.example.http.websocketx.server. There is a fairly simple and well laid out example on how to implement a websocket server, or just the handler.
Websocket servers are slightly more complicated than other servers in that they must start life as an HTTP server because the protocol specifies that websockets must be initiated by "upgrading" an HTTP connection, but as I said, the example referenced above makes this fairly clear.
So, I found solution! It is not compliant to native documentation of web-socket, but who cares it works as I expected!
public void channelRead(ChannelHandlerContext ctx, Object msg)
{
DefaultHttpRequest httpRequest = null;
if (msg instanceof DefaultHttpRequest)
{
httpRequest = (DefaultHttpRequest) msg;
// Handshake
WebSocketServerHandshakerFactory wsFactory = new WebSocketServerHandshakerFactory("ws://127.0.0.1:8080/", null, false);
final Channel channel = ctx.channel();
final WebSocketServerHandshaker handshaker = wsFactory.newHandshaker(httpRequest);
if (handshaker == null) {
} else {
ChannelFuture handshake = handshaker.handshake(channel, httpRequest);
}
}
}
Do not forget to add
p.addLast(new HttpRequestDecoder(4096, 8192, 8192, false));
p.addLast(new HttpResponseEncoder());
to your pipeline.
I have the same issue than here (Netty write several messages through one channel) and I can't solve it. Here you are some cases:
I can send several messages from client-side, and the server receives only if it doesn't respond.
If I send a message to the server and it treats (with handler), generates, and writeAndFlush the response, the channel will switch to closed and I can't send more messages from client-side.
If I send a message from client with write() method, the message doesn't arrive to server because flush is necessary. It doesn't close the channel.
It looks like that the problem is on the server-side when generates a response and writeAndFlush.
I'm developing an RTSP Client/Server and this protocol is based on HTTP/1.1 Following the Netty documentation for Snoop Http Server (http://netty.io/4.0/xref/io/netty/example/http/snoop/package-summary.html), if you use the keep-alive option, the messages only write and doesn't flush. In my use of case, I repeat, the messages don't arrive to the server.
Here you are some code:
Client-Side
// Bootstrap
public RtspClient(String host, String port) throws NumberFormatException, InterruptedException{
// Configure the client.
EventLoopGroup group = new NioEventLoopGroup();
try {
b = new Bootstrap();
b.group(group)
.channel(NioSocketChannel.class)
.handler(new RtspClientInitializer());
ch = b.connect(host, Integer.parseInt(port)).sync().channel();
} finally {
// Shut down executor threads to exit.
// group.shutdownGracefully();
}
}
// Channel Pipeline definition
class RtspClientInitializer extends ChannelInitializer<SocketChannel> {
#Override
public void initChannel(SocketChannel ch) {
ChannelPipeline p = ch.pipeline();
p.addLast("encoder", new RtspRequestEncoder());
p.addLast("decoder", new RtspResponseDecoder());
p.addLast("handler", new RtspClientHandler());
}
}
// The method which sends RTSP OPTIONS protocol method
public void sendOptions() throws InterruptedException{
// HTTP/RTSP request.
DefaultFullHttpRequest request = new DefaultFullHttpRequest(
RtspVersions.RTSP_1_0, RtspMethods.OPTIONS, "*");
request.headers().set(RtspHeaders.Names.CONNECTION, RtspHeaders.Values.KEEP_ALIVE);
request.headers().set(RtspHeaders.Names.CONTENT_LENGTH, request.content().readableBytes());
request.headers().set(RtspHeaders.Names.CSEQ, this.CSeq);
this.CSeq++;
// Write to channel
System.out.println("Request: ------------------------------------------------------");
System.out.println(request.toString());
System.out.println("---------------------------------------------------------------");
ch.writeAndFlush(request);
}
Server-Side
// Bootstrap
public void initRtspServer() throws InterruptedException{
EventLoopGroup bossGroup = new NioEventLoopGroup(1);
EventLoopGroup workerGroup = new NioEventLoopGroup();
try {
ServerBootstrap b = new ServerBootstrap();
b.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.class)
.handler(new LoggingHandler(LogLevel.INFO)).childHandler(new RtspServerInitializer());
ChannelFuture cf = b.bind(this.port);
System.err.println("RTSP Server run at this URL: " +
"rtsp" + "://127.0.0.1:" + this.port + '/');
cf.channel().closeFuture().sync();
} finally {
bossGroup.shutdownGracefully();
workerGroup.shutdownGracefully();
}
// Channel Pipeline definition
class RtspServerInitializer extends ChannelInitializer<SocketChannel>{
#Override
public void initChannel(SocketChannel ch) {
ChannelPipeline p = ch.pipeline();
p.addLast("decoder",new RtspRequestDecoder());
p.addLast("encoder",new RtspResponseEncoder());
p.addLast("handler",new RtspServerHandler());
}
}
// In this case I only handle the OPTIONS method on server-side
class RtspServerHandler extends SimpleChannelInboundHandler<HttpRequest> {
private HttpRequest request;
private FullHttpResponse response;
#Override
protected void channelRead0(ChannelHandlerContext ctx, HttpRequest request)
throws Exception {
this.response = new DefaultFullHttpResponse(RTSP_1_0,OK);
String sequence = request.headers().get(RtspHeaders.Names.CSEQ);
this.response.headers().set(RtspHeaders.Names.CSEQ, sequence);
this.response.headers().set(RtspHeaders.Names.PUBLIC, "OPTIONS,SETUP,TEARDOWN,PLAY");
ctx.writeAndFlush(response).isSuccess(); // I think server closes the channel here
}
}
If I do the following in the main class from client:
RtspClient client = new RtspClient("127.0.0.1","554");
client.sendOptions();
Thread.sleep(1000);
client.sendOptions();
the second sendOptions() do nothing.
With Netty 3.2 I didn't have this problem, It works properly with send() method.
I have a websocket server using Netty (4.0.17) that answers requests from a JavaScript client.
The communication works fine but I have an exception when I try to immediately send a greeting message when a client connects.
My code looks like that :
public class LiveOthelloWSHandler extends SimpleChannelInboundHandler<TextWebSocketFrame> {
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
super.channelActive(ctx);
ChannelFuture f = ctx.channel().writeAndFlush(new TextWebSocketFrame("(gameID=0)[LiveOthelloServer="+ VERSION_NUMBER + "]\n"));
}
// ...
#Override
protected void channelRead0(ChannelHandlerContext ctx, TextWebSocketFrame frame) throws Exception {
final String request = frame.text();
Channel thisChannel = ctx.channel();
// Do something with request
// Write back
thisChannel.writeAndFlush(new TextWebSocketFrame(response + "\n"));
}
}
The channelRead0() is ok, the client sends messages and the server answers back without any issue.
What doesn't work is the "greetings" part. I would like to send a welcoming message to the client (the string using VERSION_NUMBER in the ChannelActive() method) but I always get an exception :
java.lang.UnsupportedOperationException: unsupported message type: TextWebSocketFrame
I guess this is maybe because the channelActive() gets invoked as soon as the connection is established but before the websocket handshake is complete. How can I wait for the handshake to be finished and then send the greeting message (without the client having sent any request yet)?
For information, my initialization is:
#Override
public void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
pipeline.addLast(
new HttpRequestDecoder(),
new HttpObjectAggregator(65536),
new HttpResponseEncoder(),
new WebSocketServerProtocolHandler("/websocket"),
myLiveOthelloWSHandler);
Just RTFM...
http://netty.io/4.0/api/io/netty/handler/codec/http/websocketx/WebSocketServerProtocolHandler.html
Best way to detect handshake is to override ChannelInboundHandler.userEventTriggered...
So I just had to add:
#Override
public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception {
super.userEventTriggered(ctx, evt);
if (evt == WebSocketServerProtocolHandler.ServerHandshakeStateEvent.HANDSHAKE_COMPLETE) {
ChannelFuture f = ctx.channel().writeAndFlush(new TextWebSocketFrame("(gameID=0)[LiveOthelloServer="+ VERSION_NUMBER + "]\n"));
}
}
I'm trying to write a HTTP client that uses HTTP keep-alive connections. When I connection from the ClientBoostrap I get the channel. Can I reuse this for sending multiple HTTP requests? Is there any examples demonstrating the HTTP Keep Alive functionality?
Also I have another question. Now my client works without keep-alive connections. I'm calling the channel.close in the messageReceived method of the ClientHandler. But it seems the connections are not getting closed and after some time the sockets run out and I get a BindException. Any pointers will be really appreciated.
Thanks
As long as the Connection header is not set to CLOSE (and possible the HttpVersion is 1.1, though uncertain) by a line of code similar to this...
request.setHeader(HttpHeaders.Names.CONNECTION, HttpHeaders.Values.CLOSE);
...your channel should remain open for multiple request/response pairs.
Here is some example code that I whipped up today to test it. You can bounce any number of requests off of Google prior to the channel closing:
public class TestHttpClient {
static class HttpResponseReader extends SimpleChannelUpstreamHandler {
int remainingRequests = 2;
#Override
public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) throws Exception {
HttpResponse response = (HttpResponse) e.getMessage();
System.out.println("Beginning -------------------");
System.out.println(new String(response.getContent().slice(0, 50).array()));
System.out.println("End -------------------\n");
if(remainingRequests-- > 0)
sendRequest(ctx.getChannel());
else
ctx.getChannel().close();
}
}
public static void main(String[] args) {
ClientBootstrap bootstrap = new ClientBootstrap(new NioClientSocketChannelFactory());
bootstrap.setPipeline(Channels.pipeline(
new HttpClientCodec(),
new HttpResponseReader()));
// bootstrap.setOption("child.keepAlive", true); // no apparent effect
ChannelFuture future = bootstrap.connect(new InetSocketAddress("google.com", 80));
Channel channel = future.awaitUninterruptibly().getChannel();
channel.getCloseFuture().addListener(new ChannelFutureListener() {
public void operationComplete(ChannelFuture future) throws Exception {
// this winds up getting called immediately after the receipt of the first message by HttpResponseReader!
System.out.println("Channel closed");
}
});
sendRequest(channel);
while(true) {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
private static void sendRequest(Channel channel) {
// Prepare the HTTP request.
HttpRequest request = new DefaultHttpRequest(
HttpVersion.HTTP_1_1, HttpMethod.GET, "http://www.google.com");
request.setHeader(HttpHeaders.Names.HOST, "google.com");
request.setHeader(HttpHeaders.Names.ACCEPT_ENCODING, HttpHeaders.Values.GZIP);
channel.write(request);
}
}