We have a number of Jetty http(s) servers, all behind different firewalls. The http servers are at customer sites (not under our control). Opening ports in the firewalls at these sites is not an option. Right now, these servers only serve JSON documents in response to REST requests.
We have web clients that need to interact with a given http server based on URL parameter or header value.
This seems like a straightforward proxy server situation - except for the firewall.
The approach that I'm currently trying is this:
Have a centralized proxy server (also Jetty based) that listens for inbound registration requests from the remote http servers. The registration request will take the form of a Websocket connection, which will be kept alive as long at the remote HTTP server is available. On registration, the Proxy Server will capture the websocket connection and map it to a resource identifier.
The web client will connect the proxy server, and include the resource identifier in the URL or header.
The proxy server will determine the appropriate Websocket to use, then pass the request on to the HTTP server. So the request and response will travel over the Websocket. Once the response is received, it will be returned to the web client.
So this is all well and good in theory - what I'm trying to figure out is:
a) is there a better way to achieve this?
b) What's the best way to set up Jetty to do the proxying on the HTTP Server end of the pipe?
I suppose that I could use Jetty's HttpClient, but what I really want to do is just pull the HTTP bytes from the websocket and pipe them directly into the Jetty connector. It doesn't seem to make sense to parse everything out. I suppose that I could open a regular socket connection on localhost, grab the bytes from the websocket, and do it that way - but it seems silly to route through the OS like that (I'm already operating inside the HTTP Server's Jetty environment).
It sure seems like this is the sort of problem that may have already been solved... Maybe by using a custom jetty Connection that works on WebSockets instead of TCP/IP sockets?
Update: as I've been playing with this, it seems like another tricky problem is how to handle request/response behavior (and ideally support muxing over the websocket channel). One potential resource that I've found is the WAMP sub-protocol for websockets: http://wamp.ws/
In case anyone else is looking for an answer to this one - RESTEasy has a mocking framework that can be used to invoke the REST functionality without running through a full servlet container: http://docs.jboss.org/resteasy/docs/2.0.0.GA/userguide/html_single/index.html#RESTEasy_Server-side_Mock_Framework
This, combined with WAMP, appears to do what I'm looking for.
Related
We have a spring boot (with zuul) app using default embedded tomcat (I think). It has many clients implemented with different technologies and languages. And we have problem with too many port in TIME_WAIT: i.e. too many socket connections are opened/closed w.r.t the expected request behavior that should keep connections alive most of the time.
By retrieving the HttpRequest object in the deployed API, I can get information on the request header. This way I can track the http protocol used (http/1.1) and header parameter such as keep-alive (which, if present, is redundant with the use of http/1.1).
=> I would like to track opened and closed socket connections, but I don't see how?
Intermediate information would be better than nothing.
Note: I found some tutorial on a similar topic when using spring-websocket, but we don't.
I am using embedded jetty and spring for java to java communication over http. My problem is that my server application must handle plain TCP messages also on the same port.
Is there a way to detect if a TCP message arrived which cannot be handled by the servlet?
Thanks for the answers I add some more details:
I cannot modify the client. The reason for this is that the old version of the client uses pure java tcp socket and it turned out that the new server must be backward compatible with the old client.
Have to use the same port
Old client messages are short serialized text over simple socket. 1: open connection, 2: send text, 3: close connection
My server looks something like this: http://kielczewski.eu/2013/11/using-embedded-jetty-spring-mvc/
I do not need to parse the message. It is enough to detect that a message was arrived which is not using http and get the source host name.
You might want to take a look at how to add a custom ConnectionFactory to the ServerConnector of your HTTP port.
This ConnectionFactory concept is how the PROXY Protocol is supported within Jetty currently.
In your case, you might have something like ...
MyTcpConnectionFactory tcpConnectionFactory = new MyTcpConnectionFactory();
ServerConnector http = new ServerConnector(server);
http.addFirstConnectionFactory(tcpConnectionFactory);
server.addConnector(http);
In your case, you would override the newConnection(Connector connector, EndPoint endPoint) method and implement the check for your TCP flow, or the HTTP flow.
If its your flow, you handle the communications on that connection yourself and then throw an IOException when you are done indicating that you don't want jetty to process that connection as HTTP.
Otherwise you return that Connection object to Jetty to process as HTTP.
You are in for a wild ride here my friend. You need to realize that HTTP IS TCP ... its just the content being sent on the TCP socket that classifies it as HTTP or not. That being said, you can intercept the Connection with a filter ie
1) create a filter (google Java Application Server Filters and check the Jetty implementation) for ALL incoming connections
2) check for URI on the request, if it fails, then the request is not HTTP (might want to double check on the request testing logic here)
3) Redirect the request to the appropriate Servlet / Function based on serial socket / http request
On another note, why not use https (port 443) for http and port 80 for your socket requirments ?
I stand corrected. Filters wont work.
In that case, you will have to code a mini firewall. you have to scan all inputs for https headers and redirect accordingly. Can you at least provide some context on the plain TCP messages you want to receive? do you have any control over the sending code ? you do know you can upgrade a TCP/HTTP connection to a websocket (involves client and server) and it will work even better than plain TCP, same port connections, and comes built in Jetty so no custom boiler plates, just a websocket servlet
I have a problem where I have several servers sending HttpRequests (using round robin to decide which server to send to) to several servers that process the requests and return the response.
I would like to have a broker in the middle that examines the request and decides which server to forward it to but the responses can be very big so I would like the response to only be sent to the original requester and not be passed back through the broker. Kind of like a proxy but the way I understand a proxy is that all data is sent back through the proxy. Is this possible?
I'm working with legacy code and would rather not change the way the requests and responses are processed but only put something in the middle that can do some smarter routing of the requests.
All this is currently done using HttpServletRequest/Response and Servlets running on embedded Jetty web servers.
Thank you!
What you're after is that the broker component is using the client's IP address when connecting to the target server. That is called IP spoofing.
Are you sure that you want to implement this yourself? Intricacies of network implementation of such a solution are quite daunting. Consider using software that has this option builtin, such as HAProxy. See these blog posts.
I need something that can intercept HTTP requests, extract their information (content, destination,...), perform various analysing tasks, and finally determine if the request should be dropped or not. Legal requests must than be forwarded to the application.
Basically, same functionalities as an IDS. But mind, I am NOT looking for a packet sniffer/filter. I want something that operates on the HTTP level.
It should be implementable on linux and run on the same system as the application(s) to which the requests are headed.
As a bonus, https could be supported (unencrypted viewing of the request content)
Try mitmproxy.
mitmproxy is an SSL-capable man-in-the-middle proxy for HTTP. It provides a console interface that allows traffic flows to be inspected and edited on the fly.
mitmdump is the command-line version of mitmproxy, with the same functionality but without the user interface. Think tcpdump for HTTP.
Features
Intercept HTTP requests and responses and modify them on the fly.
Save complete HTTP conversations for later replay and analysis.
Replay the client-side of an HTTP conversations.
Replay HTTP responses of a previously recorded server.
Reverse proxy mode to forward traffic to a specified server.
Make scripted changes to HTTP traffic using Python.
SSL certificates for interception are generated on the fly.
Screenshot
Example
I setup an example Jekyll Bootstrap app which is listening on port 4000 on my localhost. To intercept it's traffic I'd do the following:
% mitmproxy --mode reverse:http://localhost:4000 -p 4001
Then connect to my mitmproxy on port 4001 from my web browser (http://localhost:4001), resulting in this in mitmproxy:
You can then select any of the GET results to see the header info associated to that GET:
Try using
Burp Proxy, sounds like what you need.
I use Wire Shark for this, if you provide all the server certs it wil even decypt HTTPS.
You should be able to use squid proxy for that (https://en.wikipedia.org/wiki/Squid_(software))
You should learn more about ICAP, then make an ICAP server of your HTTP filtering application.
I ended up using LittleProxy because it is java, fast and lightweight.
It is a originally forward proxy, so I had to adjust it for reverse proxy functionality by forwarding every request to the local host.
I did this simply by editing the HttpRequestHandler. I hardcoded the host and port address.
hostAndPort = "localhost:80";
Why not Apache HTTP Client http://hc.apache.org/httpclient-legacy/tutorial.html
This simple lib is useful.
i am here looking for resources or sample code for proxy server which can handle both http and https requests written in java. i searched google and found lot of data on how to handle http requests but not https.
A proxy cannot handle HTTPS and still provide end to end security. It is not possible using SSL/TLS which is what HTTPS is built on.
And a proxy that doesn't provide end-to-end security only has limited utility. So I'm not surprised you can't find an existing implementation.
I'm assuming that you want a normal HTTPS proxy here, that is a proxy that will not look into the request but merely relay all the traffic to the actual HTTPS server after the user-agent has used the HTTP CONNECT method. This is how HTTP proxy servers are normally used for HTTPS requests by browsers.
I haven't tried, but you could look at Jetty and its ConnectHandler.