I need something that can intercept HTTP requests, extract their information (content, destination,...), perform various analysing tasks, and finally determine if the request should be dropped or not. Legal requests must than be forwarded to the application.
Basically, same functionalities as an IDS. But mind, I am NOT looking for a packet sniffer/filter. I want something that operates on the HTTP level.
It should be implementable on linux and run on the same system as the application(s) to which the requests are headed.
As a bonus, https could be supported (unencrypted viewing of the request content)
Try mitmproxy.
mitmproxy is an SSL-capable man-in-the-middle proxy for HTTP. It provides a console interface that allows traffic flows to be inspected and edited on the fly.
mitmdump is the command-line version of mitmproxy, with the same functionality but without the user interface. Think tcpdump for HTTP.
Features
Intercept HTTP requests and responses and modify them on the fly.
Save complete HTTP conversations for later replay and analysis.
Replay the client-side of an HTTP conversations.
Replay HTTP responses of a previously recorded server.
Reverse proxy mode to forward traffic to a specified server.
Make scripted changes to HTTP traffic using Python.
SSL certificates for interception are generated on the fly.
Screenshot
Example
I setup an example Jekyll Bootstrap app which is listening on port 4000 on my localhost. To intercept it's traffic I'd do the following:
% mitmproxy --mode reverse:http://localhost:4000 -p 4001
Then connect to my mitmproxy on port 4001 from my web browser (http://localhost:4001), resulting in this in mitmproxy:
You can then select any of the GET results to see the header info associated to that GET:
Try using
Burp Proxy, sounds like what you need.
I use Wire Shark for this, if you provide all the server certs it wil even decypt HTTPS.
You should be able to use squid proxy for that (https://en.wikipedia.org/wiki/Squid_(software))
You should learn more about ICAP, then make an ICAP server of your HTTP filtering application.
I ended up using LittleProxy because it is java, fast and lightweight.
It is a originally forward proxy, so I had to adjust it for reverse proxy functionality by forwarding every request to the local host.
I did this simply by editing the HttpRequestHandler. I hardcoded the host and port address.
hostAndPort = "localhost:80";
Why not Apache HTTP Client http://hc.apache.org/httpclient-legacy/tutorial.html
This simple lib is useful.
Related
I have a REST API written in java and spring which makes a backend call to a SOAP service. The SOAP web service code are packaged in a 3rd party jar. So don't have access to change the code. But I can change the SOAP service endpoint by setting a env variable.
How can I capture the raw SOAP request sent by the REST API?
You can intercept the SOAP Request by using number of tools.
I would suggest either use TCPMonitor (https://ws.apache.org/tcpmon/tcpmontutorial.html) or wiresharks (https://www.wireshark.org/).
TCPMonitor acts like a proxy (So your endpoint should point to TCPMonitor), and it will relay the message back to your service.
Wireshark is a network protocol analyser. It captures all the TCP/UDP packets (which includes the HTTP as well). You just have to capture the packets. No modifications from server and client ends required. You can then filter the HTTP/HTTPS packets to look for your requests.
You can always use a network packet Analyser such as tcpdump or wireshark to capture packets sent specifically to your 'endpoint'.
ALTERNATIVELY
You can write a tiny proxy to capture requests on a local port and forward them to the actual endpoint and route response after printing it to sys-out. You, in that case, will specify the proxy as the endpoint to your program.
I am not sure of any existing proxy which does that, but I think haproxy will not.
If you're on Windows, you can use ProxyTrace, PocketSoap, and other PocketSoap.com tools.
I have a problem where I have several servers sending HttpRequests (using round robin to decide which server to send to) to several servers that process the requests and return the response.
I would like to have a broker in the middle that examines the request and decides which server to forward it to but the responses can be very big so I would like the response to only be sent to the original requester and not be passed back through the broker. Kind of like a proxy but the way I understand a proxy is that all data is sent back through the proxy. Is this possible?
I'm working with legacy code and would rather not change the way the requests and responses are processed but only put something in the middle that can do some smarter routing of the requests.
All this is currently done using HttpServletRequest/Response and Servlets running on embedded Jetty web servers.
Thank you!
What you're after is that the broker component is using the client's IP address when connecting to the target server. That is called IP spoofing.
Are you sure that you want to implement this yourself? Intricacies of network implementation of such a solution are quite daunting. Consider using software that has this option builtin, such as HAProxy. See these blog posts.
I use Fiddler a lot (especially the composer and reverse proxy features) in web debugging. I am able to successfully send traffic to my reverseproxy server ("myreverseproxy:8888"). This is basically using custom rules explained here, and it works with most cases.
However I tried to redirect traffic similar with one of the third party DLLs. These are native code, so can't decompile to see what's happening. Basically I don't have a source - but it connects to a host on a specified port - I wanted to observe the packets to take a closer look at headers, packets, authentication etc.
But in this particular case reverse proxy is not working. Probably the third party DLL is not using http stack.
My questions:
Is there anything I can do in this particular case to understand how this external native assembly is connecting to the server? or Fiddler can't be used as it probably can't intercept this traffic?
if that is the case, what other tools I can use to intercept the traffic?
Update
Here are the things I observed:
From client side, Fiddler is unable to capture packets (probably the api's Fiddler use at higher level)
Then I setup reverse proxy, to make sure I catch the traffic at 'rever proxy server' - however Fiddler still could catch the traffic. Thanks to Lex - looks like fiddler won't even show the packet if it can't decode payload as http (not too sure - but it's my conjecture from Lex's response and other posts)
Then I have installed Wireshark, captured the packets, and by default 'wireshark' showed the interactions between client and server as TLSv1 - so it confirms why fiddler couldn't catch
Just to confirm I applied HTTP decoder to see if 'default decoder' assigned by Wireshark is wrong - but, it couldn't decode details.
So, it confirms the native module is using TLSv1, not https and that explains why I can't use Fiddler.
If the protocol is known, you can utilize Wireshark to capture and analyze the packets.
If the protocol is private, you can only get raw bytes, which takes much more efforts to analyze.
Fiddler is designed for HTTP based protocols only, so it is not as general purpose as Wireshark.
We have a number of Jetty http(s) servers, all behind different firewalls. The http servers are at customer sites (not under our control). Opening ports in the firewalls at these sites is not an option. Right now, these servers only serve JSON documents in response to REST requests.
We have web clients that need to interact with a given http server based on URL parameter or header value.
This seems like a straightforward proxy server situation - except for the firewall.
The approach that I'm currently trying is this:
Have a centralized proxy server (also Jetty based) that listens for inbound registration requests from the remote http servers. The registration request will take the form of a Websocket connection, which will be kept alive as long at the remote HTTP server is available. On registration, the Proxy Server will capture the websocket connection and map it to a resource identifier.
The web client will connect the proxy server, and include the resource identifier in the URL or header.
The proxy server will determine the appropriate Websocket to use, then pass the request on to the HTTP server. So the request and response will travel over the Websocket. Once the response is received, it will be returned to the web client.
So this is all well and good in theory - what I'm trying to figure out is:
a) is there a better way to achieve this?
b) What's the best way to set up Jetty to do the proxying on the HTTP Server end of the pipe?
I suppose that I could use Jetty's HttpClient, but what I really want to do is just pull the HTTP bytes from the websocket and pipe them directly into the Jetty connector. It doesn't seem to make sense to parse everything out. I suppose that I could open a regular socket connection on localhost, grab the bytes from the websocket, and do it that way - but it seems silly to route through the OS like that (I'm already operating inside the HTTP Server's Jetty environment).
It sure seems like this is the sort of problem that may have already been solved... Maybe by using a custom jetty Connection that works on WebSockets instead of TCP/IP sockets?
Update: as I've been playing with this, it seems like another tricky problem is how to handle request/response behavior (and ideally support muxing over the websocket channel). One potential resource that I've found is the WAMP sub-protocol for websockets: http://wamp.ws/
In case anyone else is looking for an answer to this one - RESTEasy has a mocking framework that can be used to invoke the REST functionality without running through a full servlet container: http://docs.jboss.org/resteasy/docs/2.0.0.GA/userguide/html_single/index.html#RESTEasy_Server-side_Mock_Framework
This, combined with WAMP, appears to do what I'm looking for.
Environment: NetBeans 7.0.1, GlassFish 3.1
I want to sniff the communication between a WCF (.NET Framework 4) client and a Metro (2.1.1) web service, and then check the messages to be sure everything is how I want it to be. The web service uses Transport Security (SSL). I already know of Fiddler, but I did not have any success using it; only HTTPS browser traffic was visible. Is there a way to set up Fiddler to capture traffic between my services? Is there any other way?
UPDATE
I tried to start client and server on different machines and then use Fiddler, but no success. I tried Wireshark to capture traffic, but did not have any success running both on localhost. If i tried them on different machines, all I could see was TCP data exchange between the services.
On localhost + Wireshark, the packet counter next to the interfaces remained the same however I was calling the service lots of times.
UPDATE2
Tried to set up the proxy for NetBeans manually in the options, and programatically, but no success:
System.setProperty("http.proxyHost", "localhost");
System.setProperty("http.proxyPort", "8888");
System.setProperty("https.proxyHost", "localhost");
System.setProperty("https.proxyPort", "8888");
Tried to start GlassFish with these JVM options, but no :( :
<jvm-options>-Dhttp.proxyHost=localhost</jvm-options>
<jvm-options>-Dhttp.proxyPort=8888</jvm-options>
<jvm-options>-Dhttps.proxyHost=localhost</jvm-options>
<jvm-options>-Dhttps.proxyPort=8888</jvm-options>
As an alternative to Fiddler, if you control the WCF client, you could enable WCF Message Logging, and it will save all the unencrypted SOAP messages to a trace log. The logging can be enabled in the app.config file, so you don't even have to rebuild the app to enable or disable the logging.
you can setup metro to dump SOAP messages, info here. personally, i use charles proxy to watch soap exchanges. you configure the java proxy using the system properties in your "update2", works very well.
Use http://portswigger.net/burp/
It has a proxy. The proxy can be used for viewing the http traffic. It can also display SSL traffic by generating self signed certificate on the fly. You need to import the generated certificate into the java key store at the jax-ws client. Ensure that you have enabled "Support invisible proxy for non-proxy aware clients"
I often use commview to monitor traffic over the local loopback adapter. One of the few tools allowing traffic capture when both your client and service are on the same computer.
You can download a trail at http://www.tamos.com/products/commview/ and see if it works for you.
May be WebScarab is what you need. There are many manuals for sniffing ssl traffic, for instance that.
One approach I've tried successfully, is to make sure SSLv2 is used (as opposed to SSLv3), and then use Wireshark as described on the SSL page on the Wikishark Wiki, but better on Citrix support page "How to decrypt SSL and TLS traffic using Wireshark ". This works by giving Wireshark the private key of the server's SSL certificate, so that it can decrypt the conversation.
To force SSLv2 in your scenario, it seems sufficient to set -Dhttps.protocols=SSLv2Hello on your server-side JVM, but I googled that together. (See, e.g., the "Why do I get a javax.net.ssl.SSLException" question on the Java 1.4.2 Troubleshooting page, and the part on https.protocols in the JavaTM Secure Socket Extension (JSSE) Reference Guide.) I haven't done this part myself, and I can't seem to find clear documentation on this point.
(P.S. In my case, when I was decrypting .NET-to-.NET SSL traffic, I thought it was the switch back to SSLv2 which made the traffic readable by Wireshark. However, this blog post suggests that I was switching at the same time from a Diffie-Hellman cipher to a non-DH one.)