I am making a proxy application for a browser. It has to use only the standard libraries. So far, I've managed to create the server. When trying to access a web page from a client, i get the following information:
CONNECT gmail.com:443 HTTP/1.1
User-Agent: Mozilla/5.0 Firefox/49.0
Proxy-Connection: keep-alive
Connection: keep-alive
Host: gmail.com:443
My question is: what to use in order to handle the requests? How to handle a file download?
Once you get that CONNECT command, do what is asked: create the upstream connection, and return the appropriate success/failure response. If the upstream connection was successful, all you have to do now is copy bytes in both directions, simultaneously. The endpoints will take care of all SSL issues, uploads, downloads, etc. You have no further role to play.
The general behaviour of a proxy is as follows:
Receive request from browser
Make a request to the actual server, resolving all redirects if necessary
Get the response from server and passit on to client
I am not getting into complications of changing request/response headers, caching etc.
Now from the above, you are making a SSL connection to gmail.com refer.
The browser is actually sending correct request, in this case you need to implement the handshake and connect to gmail with HTTPS offloading SSL on your side and sending the response received to the browser through the negotiated SSL with the browser.
Suggestion is to use HTTP instead of HTTPS, if this is not a production grader system and try out the concept first
Related
I am writing an HTTPS proxy in play framework, yet every time I try to run it on a website that has ssl encryption I am getting this error in my server:
akka.actor.ActorSystemImpl(play-dev-mode) Illegal request, responding with status '400 Bad Request': CONNECT requests are not suppo
rted: Rejecting CONNECT request to '//any.website:443'
The proxy works well on HTTP, yet HTTPS is the problem. I tried changing my application.conf to have a port destined to work on HTTPS, but when I try to run it with the keystore i setup using the keytool command I am getting the following error in my browser:
Secure Connection Failed
The connection to the server was reset while the page was loading.
The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
Please contact the website owners to inform them of this problem.
Any workaround to allow HTTPS on a play framework proxy server?
I'm trying to download a file via FTP with a Java application.
The FTP url is accessible from this web page: http://professionnels.ign.fr/adminexpress.
More specifically, I'm trying to download this file.
From my home, I can download the file successfully with my java application, Firefox or Chrome.
From my work, I can do the same with Firefox and Chrome only. My application refuses to download anything.
NOTA: At work, the browsers and my application use the same HTTP proxy to access internet.
I'm using Apache Commons Net 3.6.
Here is a sample of the FTP exchanges of my application. I wasn't able to sniff those of Chrome or Firefox.
220 Bienvenue sur le site FTP de L INSTITUT NATIONAL DE L INFORMATION GEOGRAPHIQUE ET FORESTIERE
USER *******
331 Please specify the password.
PASS *******
230 Login successful.
TYPE I
200 Switching to Binary mode.
PASV
227 Entering Passive Mode (192,134,132,16,65,180).
RETR /ADMIN-EXPRESS-COG_2-0__SHP_WM__FRA_2019-05-20.7z.001
425 Failed to establish connection.
tl;dr
It turned out that the HTTP proxy at my work already handles all the FTP exchanges. This is why Firefox and Chrome could download the file. When they aren't behind an HTTP proxy, it seems they act as an FTP client by sending FTP commands directly.
A simple HTTP GET request to the HTTP proxy with the ftp url is enough to download the file.
Here is a sum up of solutions I found during my investigations:
Use passive mode (PASV command)
Check if there's an FTP proxy to use rather than an HTTP Proxy
Check the configuration of the FTP server (if you have access to it)
Check the configuration of the HTTP proxy (if you have access to it)
Precisely, the browsers perform a simple HTTP request as described below:
GET ftp://user:passw0rd#example.com/file.ext HTTP/1.1
Host: example.com
User-Agent: WebBrowser-UA/x.y.z
...
Then the HTTP proxy parses the FTP url and connects to the FTP server. The HTTP proxy returns the file content as a normal HTTP response.
HTTP/1.1 200 OK
Last-Modified: Tue, 21 May 2019 11:23:00 GMT
Content-Length: 115545060
Content-Type: octet/stream
Connection: Keep-Alive
Age: 22
Date: Thu, 27 Jun 2019 10:27:09 GMT
(file content here...)
However, in my case, the HTTP proxy allowed me to connect to the FTP server and exchange on the command FTP channel only. The data channel seemed to be blocked either in ACTIVE or PASSIVE mode.
During my investigations, I found many people hitting this very same problem. The solutions they found (when they found one...) didn't apply to me. Here is a sum up of the solutions expressed in all those questions:
Use passive mode (PASV command)
Check if there's an FTP proxy to use rather than an HTTP Proxy
Check if the HTTP proxy handles directly the FTP exchanges
Check the configuration of the FTP server (if you have access to it)
Check the configuration of the HTTP proxy (if you have access to it)
References:
Understanding FTP over HTTP
Connect to FTP server through http proxy
FTP connection through proxy with Java
Accessing FTP server behind a proxy via command prompt in Windows 7
[vsFTPd] 425 Failed to establish connection.
I am supporting another vendors legacy application.
This is a J2EE application that runs on Glassfish v3.1.2.2. It has a REST API implemented using JAX-RS. I have limited visibility to the application and source.
The symptoms are:
make an HTTP request to a REST API
application has its own auditing system, this shows a successful request
no errors in GF logs
GF access log notes the request
0 bytes are returned from the request to the caller
This happens for both remote calls as well as from calls made using curl on localhost.
If we make the same requests to a different port over HTTPS they succeed. We are reluctant to move the calls to that other port without knowing a root cause. These failed intermittently last night and now fail constantly today.
A packet capture of the request shows:
- TCP overhead/handshake
- A GET request
- A single ACK from the application back to the caller
- then nothing after that
What would cause Glassfish v3 to successfully handle and process an HTTP request but return no data?
Is there a mechanism in Glassfish v3 to flush or reset an HTTP listener and its associated thread pool?
Since this happens on a curl request on the same server to localhost I think I can rule out the network being the issue.
The ports being used communicate directly with Glassfish. There is no proxy (like Apache or Nginx) between the caller and the app server.
Are there logging or monitoring settings I should be enabling in Glassfish to observe what the HTTP listener is doing relative to the application and the network stack?
I have obfuscated some examples that show the symptoms:
Glassfish Access log:
"0:0:0:0:0:0:0:1" "NULL-AUTH-USER" "25/Oct/2018:11:21:02 -0500" "GET /api/obfuscated/by/me HTTP/1.1" 200 9002
Curl response for that same call:
* Trying OFBBFUSCATED
* Connected to hostname.local (OFBBFUSCATED) port 11080 (#0)
> GET /api/obfuscated/by/me HTTP/1.1
> Host: hostname.local:11080
> User-Agent: curl/7.43.0
> Accept: */*
> Authorization: Basic asdfdsfsdfdsfsdafsdafsdafw==
>
* Empty reply from server
* Connection #0 to host hostname.local left intact
UPDATE I changed a timeout setting for the HTTP network listener. I bumped it from 30 to 35 seconds because I was seeing a packet capture where the app was sending a FIN after 30 seconds. After making this change it started to work again.
It is not clear if this somehow flushed or reset something or if I had some kind of race condition.
The apparent root cause was high I/O on the system running these services. The applications normally used 50MB/sec, a new process drove that usage to 250MB/sec. Once the I/O problem was resolved all of the HTTP errors went away and haven't come back.
I'm trying to validate an SSL connection to an http server through a proxy server. In my case, I have 4 pieces of information which are all supplied by the user, and which I'd like to validate explicitly: target host, target port, proxy host, proxy port. I'd prefer to NOT make an actual HTTP request in order to do this validation, since that requires 2 more pieces of information: a request method, and a path (ie. "GET /"). I'd really like to be able to use the HttpClient library because it supports NTLM proxy auth.
I suppose what I want is to get the response of a CONNECT request sent to the proxy server, as all it requires are the 4 pieces of information I have (plus any proxy creds). However this seems to be an implicit request, the result of which is not available to the library client (unless it returns a 407 status code). Is there some way to trigger the CONNECT request explicitly?
You can use ProxyClient shipped with Apache HttpClient. It does precisely that.
I have been haunted for some time now trying to get my custom proxy to properly handle when the browser sends a CONNECT request. In order to keep it simple let me explain how I handle the process. Maybe at that point someone can help clarify what I'm doing wrong.
Create server with ServerSocketChannel on port 8080.
Bind that ServerSocketChannel to a Selector which essentially allows for non-blocking while the server waits for a request from port 8080.
As soon as I set my browser to port 8080 and send the request https://google.com it notifies the selector something sent to port 8080.
I get that request and see its a CONNECT so i immediately create a response "Connection Established" (request and response i send and receive are below)
Request from browser:
CONNECT google.com:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.18) Gecko/20110614 Firefox/3.6.18 GTB7.1
Proxy-Connection: keep-alive
Host: google.com
Response I send back to browser over my normal socket:
HTTP/1.1 200 Connection established\r\nProxy-connection: Keep-alive\r\n\r\n
Now I wait for something to be sent from browser I was assuming at this point a SSL request would be sent but nothing ever gets sent. I'm starting to think its because I have not established a SSL handshake with the browser so its not going to send a SSL message over that default created socket. Do you think I need to close that socket over port 8080 and establish a new SecureSocket on port 8080 right before I send the connection established response back to browser? This is my next step.I know that the browser needs to send me more data after the initial CONNECT. I don't have enough data with just the CONNECT to go to server yet. I'm thinking it than needs to send me another request something like the following in SSL:
GET /
Host: google.com
Once I get something like that then I can go establish my secure socket connection with the server and get back the response to send back to browser.
What you think on the right track? Its just that I get no additional message after i send connection established.
I get that request and see its a CONNECT so i immediately create a
response "Connection Established"
That's wrong for a start. You shouldn't send "Connection Established" until you have established the connection upstream. You're lying to your client!
I'm starting to think its because I have not established a SSL
handshake with the browser
Irrelevant. Once you have done so, the next thing you will get from the client is binary as far as you are concerned. All you should be doing from this point forwards is copying bytes.
Do you think I need to close that socket over port 8080 and establish
a new SecureSocket on port 8080
No.
Once I get something like that then I can go establish my secure
socket connection with the server
Wrong. Once you get some data from the client you should send it transparently over your existing plaintext connection with the upstream server. You don't have to engage in SSL yourself.
I don't have enough data with just the CONNECT to go to server yet.
Yes you do. You should form the upstream connection when you are told to do so, and tell the client that you have done so when you have actually done so, and not before.
What you think on the right track?
No.
Its just that I get no additional message after I send connection established.
That would suggest that you haven't actually sent anything yet. NIO code is tricky. What was the return code of the write() API?