I have a console Java application. Some of HTTP requests return error codes, and I want to debug them. In browser I can watch POST requests in detail, but how can I manage to do it in Java application?
In your getHttpRequest (or whatever it is called) method print incoming request.
That would be the simplest solution.
More sophisticated - you can extend standard classes which work with requests and modify methods which get requests to print them first.
Try Wireshark or fiddler. The both sniff traffic in a similar manner as firefox and various browsers do now.
Related
I am trying to block certain websites using a web application. So, when a I type a url suppose "http://www.google.com" it should first check whether google is blocked by my application or not. If not open the website otherwise reject the browser request to open it. I am unable to find a way to capture all HTTP request from browser so that I can process it.
I Know proxies are the most suitable option but is there any alternative solution to this. After some searching I found a library - jpcap (a network packet capture library) and I was wondering if this could help me or not?
What you are trying to create is a proxy-server.
You have to configure the browser to go through the proxy, then you can deny websites, reroute them etc.
There are many proxies already there (open source and commercial) that offer what you want.
For example: Squid http://www.squid-cache.org/
See Wikipedia description of a proxy here: https://en.wikipedia.org/wiki/Proxy_server
Many firewall products offer the service of a transparent proxy, redirecting all http/https traffic going through the firewall into a proxy server. It seems, you have a direct connection but your packages are really filtered. Aka transparent proxy.
If your assignment does not allow this context, you need to check the assignment again, if you really got the scope of filtering right.
You cannot take over the browser's ip communication from a servlet or servlet filter. Using a (servlet) filter, you can only filter requests directed to your application. One step above, using an application server valve (Tomcat uses this term, others may use a different one), you can only filter requests directed at that server. One step above (or below) your application server is the physical server and the network it is running in.
If your client does not share the same network as your server, you can't even apply transparent proxy to it. Since browsers are running on the client computer, most clients in the world do not share the same network zone as the server.
It just does not work as you expect it.
what I want to do is to build a web application(proxy) that user use to request the webpage he want and
my application forward the request to the main server,
modify HTML code,
send to the client the modified one.
The question now is
How to keep my application between the client and main serevr
(for example when the user click any link inside the modified page-
ajax request - submit forms - and so on)
in another words
How to grantee that any request (after the first URL request) from the client sent to my proxy and any response come first to my proxy
The question is: Why do you need a proxy? Why do you want to build it - why not use already existing one like HAProxy ?
EDIT: sorry, I didn't read your whole post correctly. You can start with:
http://www.jtmelton.com/2007/11/27/a-simple-multi-threaded-java-http-proxy-server/
If the user is willing to, or can be forced1 to configure his clients (e.g. web browser) to use a web proxy, then your problem is already solved. Another way to do this (assuming that the user is cooperative) is to get them to install a trusted browser plugin that dynamically routes selected URLs through your proxy. But you can't do this using an untrusted webapp: the Browser sandbox won't (shouldn't) let you.
Doing it without the user's knowledge and consent requires some kind of interference at the network level. For example, a "smart" switch could recognizes TCP/IP packets on port 80 and deliberately route them to your proxy instead of the IP address that the client's browser specifies. This kind of thing is known as "deep packet inspection". It would be very difficult to implement yourself, and it requires significant compute power in your network switch if you are going to achieve high network rates through the switch.
The second problem is that making meaningful on-the-fly modifications to arbitrary HTML + Javascript responses is a really difficult problem.
The final problem is that this is only going to work with HTTP. HTTPS protects against "man in the middle" attacks ... such as this ... that monitor or interfere with the requests and responses. The best you could hope to do would be to capture the encrypted traffic between the client and the server.
1 - The normal way to force a user to do this is to implement a firewall that blocks all outgoing HTTP connections apart from those made via your proxy.
UPDATE
The problem now what should I change in the html code to enforce client to request any thing from my app --- for example for link href attribute may be www.aaaa.com?url=www.google.com but for ajax and form what I should do?
Like I said, it is a difficult task. You have to deal with the following problems:
Finding and updating absolute URLs in the HTML. (Not hard)
Finding and dealing with the base URL (if any). (Not hard)
Dealing with the URLs that you don't want to change; e.g. links to CSS, javascript (maybe), etc. (Harder ...)
Dealing with HTML that is syntactically invalid ... but not to the extent that the browser can't cope. (Hard)
Dealing with cross-site issues. (Uncertain ...)
Dealing with URLs in requests being made by javascript embedded in / called from the page. This is extremely difficult, given the myriad ways that javascript could assemble the URL.
Dealing with HTTPS. (Impossible to do securely; i.e. without the user not trusting the proxy to see private info such as passwords, credit card numbers, etc that are normally sent securely.)
and so on.
I'm trying to consume the external web service, and I'm able to do it using soapUI with xml.
And apparently I'm sending something else from my code, is there a way I can monitor all outgoing requests, inspect them and then forward them to the actual service?
Update:
The webservice is not on my machine, it sits somewhere on the internet.
Update II:
I downloaded wireshark, there are like milion things popping on my screen, I can't see what's what. Any better tools out there? I've used tcpMon before
I would recommend using the Fiddler tool, provided you're running Fiddler and the website on the same machine you should be able to see the requests being fired.
There is some extra setup required to capture the service calls, have a look at this Blog for more details.
The parameters should be available in the headers when you inspect the request.
Recently I used a Mac application called Spotflux. I think it's written in Java (because if you hover over its icon it literally says "java"...).
It's just a VPN app. However, to support itself, it can show you ads... while browsing. You can be browsing on chrome, and the page will load with a banner at the bottom.
Since it is a VPN app, it obviously can control what goes in and out of your machine, so I guess that it simply appends some html to any website response before passing it to your browser.
I'm not interested in making a VPN or anything like that. The real question is: how, using Java, can you intercept the html response from a website and append more html to it before it reaches your browser? Suppose that I want to make an app that literally puts a picture at the bottom of every site you visit.
This is, of course, a hypothetical answer - I don't really know how Spotflux works.
However, I'm guessing that as part of its VPN, it installs a proxy server. Proxy servers intercept HTTP requests and responses, for a variety of reasons - most corporate networks use proxy servers for caching, monitoring internet usage, and blocking access to NSFW content.
As a proxy server can see all HTTP traffic between your browser and the internet, it can modify that HTTP; often, a proxy server will inject an HTTP header, for instance; injecting an additional HTML tag for an image would be relatively easy.
Here's a sample implementation of a proxy server in Java.
There are many ways to do this. Probably the easiest would be to proxy HTTP requests through a web proxy like RabbIT (written in java). Then just extend the proxy to mess with the response being sent back to the browser.
In the case of Rabbit, this can either be done with custom code, or with a special Filter. See their FAQ.
WARNING: this is not as simple as you think. Adding an image to the bottom of every screen will be hard to do, depending on what kind of HTML is returned by the server. Depending on what CSS, javascript, etc that the remote site uses, you can't just put the same HTML markup in and expect it to act the same everywhere.
This problem relates to the Restlet framework and Java
When a client wants to discover the resources available on a server - they must send an HTTP request with OPTIONS as the request type. This is fine I guess for non human readable clients - i.e. in code rather than a browser.
The problem I see here is - browsers (human readable) using GET, will NOT be able to quickly discover the resources available to them and find out some extra help documentation etc - because they do not use OPTIONS as a request type.
Is there a way to make a browser send an OPTIONS/GET request so the server can fire back formatted XML to the client (as this is what happens in Restlet - i.e. the server response is to send all information back as XML), and display this in the browser?
Or have I got my thinking all wrong - i.e. the point of OPTIONS is that is meant to be used inside a client's code and not meant to be read via a browser.
Use the TunnelService (which by default is already enabled) and simply add the method=OPTIONS query parameter to your URL.
(The Restlet FAQ Q19 is a similar question.)
I think OPTIONS is not designed to be 'user-visible'.
How would you dispatch an OPTIONS request from the browser ? (note that the form element only allows GET and POST).
You could send it using XmlHttpRequest and then get back XML in your Javascript callback and render it appropriately. But I'm not convinced this is something that your user should really know about!