I'm trying to download a file via FTP with a Java application.
The FTP url is accessible from this web page: http://professionnels.ign.fr/adminexpress.
More specifically, I'm trying to download this file.
From my home, I can download the file successfully with my java application, Firefox or Chrome.
From my work, I can do the same with Firefox and Chrome only. My application refuses to download anything.
NOTA: At work, the browsers and my application use the same HTTP proxy to access internet.
I'm using Apache Commons Net 3.6.
Here is a sample of the FTP exchanges of my application. I wasn't able to sniff those of Chrome or Firefox.
220 Bienvenue sur le site FTP de L INSTITUT NATIONAL DE L INFORMATION GEOGRAPHIQUE ET FORESTIERE
USER *******
331 Please specify the password.
PASS *******
230 Login successful.
TYPE I
200 Switching to Binary mode.
PASV
227 Entering Passive Mode (192,134,132,16,65,180).
RETR /ADMIN-EXPRESS-COG_2-0__SHP_WM__FRA_2019-05-20.7z.001
425 Failed to establish connection.
tl;dr
It turned out that the HTTP proxy at my work already handles all the FTP exchanges. This is why Firefox and Chrome could download the file. When they aren't behind an HTTP proxy, it seems they act as an FTP client by sending FTP commands directly.
A simple HTTP GET request to the HTTP proxy with the ftp url is enough to download the file.
Here is a sum up of solutions I found during my investigations:
Use passive mode (PASV command)
Check if there's an FTP proxy to use rather than an HTTP Proxy
Check the configuration of the FTP server (if you have access to it)
Check the configuration of the HTTP proxy (if you have access to it)
Precisely, the browsers perform a simple HTTP request as described below:
GET ftp://user:passw0rd#example.com/file.ext HTTP/1.1
Host: example.com
User-Agent: WebBrowser-UA/x.y.z
...
Then the HTTP proxy parses the FTP url and connects to the FTP server. The HTTP proxy returns the file content as a normal HTTP response.
HTTP/1.1 200 OK
Last-Modified: Tue, 21 May 2019 11:23:00 GMT
Content-Length: 115545060
Content-Type: octet/stream
Connection: Keep-Alive
Age: 22
Date: Thu, 27 Jun 2019 10:27:09 GMT
(file content here...)
However, in my case, the HTTP proxy allowed me to connect to the FTP server and exchange on the command FTP channel only. The data channel seemed to be blocked either in ACTIVE or PASSIVE mode.
During my investigations, I found many people hitting this very same problem. The solutions they found (when they found one...) didn't apply to me. Here is a sum up of the solutions expressed in all those questions:
Use passive mode (PASV command)
Check if there's an FTP proxy to use rather than an HTTP Proxy
Check if the HTTP proxy handles directly the FTP exchanges
Check the configuration of the FTP server (if you have access to it)
Check the configuration of the HTTP proxy (if you have access to it)
References:
Understanding FTP over HTTP
Connect to FTP server through http proxy
FTP connection through proxy with Java
Accessing FTP server behind a proxy via command prompt in Windows 7
[vsFTPd] 425 Failed to establish connection.
Related
I am supporting another vendors legacy application.
This is a J2EE application that runs on Glassfish v3.1.2.2. It has a REST API implemented using JAX-RS. I have limited visibility to the application and source.
The symptoms are:
make an HTTP request to a REST API
application has its own auditing system, this shows a successful request
no errors in GF logs
GF access log notes the request
0 bytes are returned from the request to the caller
This happens for both remote calls as well as from calls made using curl on localhost.
If we make the same requests to a different port over HTTPS they succeed. We are reluctant to move the calls to that other port without knowing a root cause. These failed intermittently last night and now fail constantly today.
A packet capture of the request shows:
- TCP overhead/handshake
- A GET request
- A single ACK from the application back to the caller
- then nothing after that
What would cause Glassfish v3 to successfully handle and process an HTTP request but return no data?
Is there a mechanism in Glassfish v3 to flush or reset an HTTP listener and its associated thread pool?
Since this happens on a curl request on the same server to localhost I think I can rule out the network being the issue.
The ports being used communicate directly with Glassfish. There is no proxy (like Apache or Nginx) between the caller and the app server.
Are there logging or monitoring settings I should be enabling in Glassfish to observe what the HTTP listener is doing relative to the application and the network stack?
I have obfuscated some examples that show the symptoms:
Glassfish Access log:
"0:0:0:0:0:0:0:1" "NULL-AUTH-USER" "25/Oct/2018:11:21:02 -0500" "GET /api/obfuscated/by/me HTTP/1.1" 200 9002
Curl response for that same call:
* Trying OFBBFUSCATED
* Connected to hostname.local (OFBBFUSCATED) port 11080 (#0)
> GET /api/obfuscated/by/me HTTP/1.1
> Host: hostname.local:11080
> User-Agent: curl/7.43.0
> Accept: */*
> Authorization: Basic asdfdsfsdfdsfsdafsdafsdafw==
>
* Empty reply from server
* Connection #0 to host hostname.local left intact
UPDATE I changed a timeout setting for the HTTP network listener. I bumped it from 30 to 35 seconds because I was seeing a packet capture where the app was sending a FIN after 30 seconds. After making this change it started to work again.
It is not clear if this somehow flushed or reset something or if I had some kind of race condition.
The apparent root cause was high I/O on the system running these services. The applications normally used 50MB/sec, a new process drove that usage to 250MB/sec. Once the I/O problem was resolved all of the HTTP errors went away and haven't come back.
I am using apache commons-net 3.6 library to connect FTPS server. FTPS server is behind NAT of thirdparty. and I can't change any settings on server side.
I can login to server, but can not list files. I've tried same code with some public FTP and FTPS servers, and result was successfull. Seems that they are not behind NAT. But filezilla can successfully connect and list files from my problematic server.
There is my code
ftps.connect(server, port);
System.out.println("Connected to " + server + ".");
reply = ftps.getReplyCode();
ftps.enterLocalPassiveMode();
if (!FTPReply.isPositiveCompletion(reply)) {
ftps.disconnect();
System.err.println("FTP server refused connection.");
System.exit(1);
}
if (!ftps.login(username, password)) {
ftps.logout();
}
// ftps.setEnabledSessionCreation(false);
ftps.feat();
ftps.execPBSZ(0);
ftps.execPROT("P");
ftps.setFileType(FTP.BINARY_FILE_TYPE);
FTPFile dirs[] = ftps.listDirectories();
And there is my ftps log:
220 FTP Server ready.
AUTH TLS
234 AUTH TLS successful
Connected to x.x.x.x
USER *******
331 Password required for azercell
PASS *******
230 User myuser logged in
FEAT
211-Features:
MDTM
MFMT
LANG bg-BG;en-US;fr-FR;it-IT;ja-JP;ko-KR;ru-RU;zh-CN;zh-TW
TVFS
UTF8
AUTH TLS
MFF modify;UNIX.group;UNIX.mode;
MLST modify*;perm*;size*;type*;unique*;UNIX.group*;UNIX.mode*;UNIX.owner*;
PBSZ
PROT
REST STREAM
SIZE
211 End
PBSZ 0
200 PBSZ 0 successful
PROT P
200 Protection set to Private
TYPE I
200 Type set to I
SYST
215 UNIX Type: L8
PASV
227 Entering Passive Mode (192,168,2,50,192,12).
[Replacing PASV mode reply address 192.168.2.50 with x.x.x.x]
LIST
150 Opening BINARY mode data connection for file list
425 Unable to build data connection: Operation not permitted
I'd read that prior to version 3.6 commons-net library prior couldnt handle behind NAT connections properly.
Can anyone help me? What is wrong with my code?
So my conclusion is problem was not related to NAT technology, apache-commons 3.6 does not handle all FTPS options properly. As I mentioned before we were integrating with 3rd party and had not option to change FTPS settings, at least we installed filezilla ftp server and were able to reproduce error. Fortunately I found solution at http://eng.wealthfront.com/2016/06/10/connecting-to-an-ftps-server-with-ssl-session-reuse-in-java-7-and-8/ by Luke Hansen. Great thanks him
I am making a proxy application for a browser. It has to use only the standard libraries. So far, I've managed to create the server. When trying to access a web page from a client, i get the following information:
CONNECT gmail.com:443 HTTP/1.1
User-Agent: Mozilla/5.0 Firefox/49.0
Proxy-Connection: keep-alive
Connection: keep-alive
Host: gmail.com:443
My question is: what to use in order to handle the requests? How to handle a file download?
Once you get that CONNECT command, do what is asked: create the upstream connection, and return the appropriate success/failure response. If the upstream connection was successful, all you have to do now is copy bytes in both directions, simultaneously. The endpoints will take care of all SSL issues, uploads, downloads, etc. You have no further role to play.
The general behaviour of a proxy is as follows:
Receive request from browser
Make a request to the actual server, resolving all redirects if necessary
Get the response from server and passit on to client
I am not getting into complications of changing request/response headers, caching etc.
Now from the above, you are making a SSL connection to gmail.com refer.
The browser is actually sending correct request, in this case you need to implement the handshake and connect to gmail with HTTPS offloading SSL on your side and sending the response received to the browser through the negotiated SSL with the browser.
Suggestion is to use HTTP instead of HTTPS, if this is not a production grader system and try out the concept first
I'm modifying some code that was previously working with an FTPS library I wrote myself. I've been asked to start using the Apache Commons Net library (FTPClient and FTPSClient mainly) and I'm running into problems doing a file listing. I've read other questions and it's not the enterLocalPassiveMode problem (Apache Commons Net FTPClient and listFiles()), as I'm using that after connecting, but before logging in. The same code works fine on a test server I set up (also using Apache FTP), but doesn't work on the server I need it for.
I've also tried using the "PBSZ 0" and "PROT P" commands, but they're not implemented on the remote system.
502 PBSZ Command not implemented.
502 PROT Command not implemented.
Code:
FTPSClient ftpsclient = new FTPSClient(true); // Implicit SSL
ftpsclient.addProtocolCommandListener(new PrintCommandListener(new PrintWriter(System.out), true));
ftpsclient.connect(HOST_ADDR, HOST_PORT); // Using port 990
ftpsclient.enterLocalPassiveMode();
ftpsclient.user(USERID);
ftpsclient.pass(PASSWORD);
ftpsclient.setFileType(FTP.BINARY_FILE_TYPE);
ftpsclient.changeWorkingDirectory(REMOTE_DIR);
ftpsclient.printWorkingDirectory();
FTPFile[] ftpfiles = ftpsclient.listFiles(); // This is where it breaks
I've tried specifying the directory explicitly and using the default:
FTPFile[] ftpfiles = ftpsclient.listFiles();
FTPFile[] ftpfiles = ftpsclient.listFiles(REMOTE_DIR);
... but both give the same result. This is the output of the debugging info:
220 FTPS (Version Thu Dec 10 17:23:00 2015) server ready.
USER ****
331 Password required for ****.
PASS ****
230 User **** logged in.
TYPE I
200 Type set to I
CWD outbound/directory
250 CWD Command successful.
PWD
257 "/usr/path/to/outbound/directory" is current directory.
SYST
215 UNIX
PASV
227 Entering Passive Mode (XX,XX,XX,XX,24,140) ***Edit: port 6284
LIST
150 Opening data connection for '/bin/ls'.
Then it times out after 30 seconds with this stack trace:
Stack Trace: org.apache.commons.net.ftp.FTPConnectionClosedException: Connection closed without indication.
at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:317)
at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:294)
at org.apache.commons.net.ftp.FTP.getReply(FTP.java:692)
at org.apache.commons.net.ftp.FTPClient.completePendingCommand(FTPClient.java:1813)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3308)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3271)
at org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:2930)
I've checked my firewall settings and that host is allowed to connect via port 990 and 6200-6300.
I've also read FTPClient.listFiles not working and Java application hanging during LIST command to FTP Server (Apache Commons FTPClient) and neither of these have helped with my problem.
EDIT: It looks as though FTPClient is not recognizing the data channel. I tried uploading a file instead of doing a listing, and it died after the "150 Opening data connection" message. I've confirmed that the ports assigned to the data connection from the PASV command are NOT blocked by our firewall.
Any ideas?
I fixed it. Turns out I was doing a directory listing with an invalid path. Instead of returning an error message, Apache FTP closed the connection. Not sure why. Anyway, it's working now.
I have a following scenario:
Oracle Web Service consumer procedure
Web Service gateway implemented using JAX-WS (runs in TomEE 1.7.2)
External Web Service
External web service is secured using HTTPS, but for testing purposes also has an unsecured version.
(1) and (2) interact over http
(2) and (3) interact over https
and for testing purposes (2) and (3) can be set to use http.
(1) consumes http version without any problem, but when we switch to secured channel, oracle (utl_http.get_response method) complains about http protocol error. SoapUI doesnt have any problems, reads soap response.
Here is the HTTP Response headers in SoapUI:
HTTP/1.1 200 OK
Connection: keep-alive
Date: Fri
Date: 27 Nov 2015 05:31:38 GMT
Server: nginx/1.2.6
Transfer-Encoding: chunked
X-Powered-By: Servlet/3.0 JSP/2.2 (GlassFish Server Open Source Edition 3.1.2.2 Java/Oracle Corporation/1.7)
Content-Type: text/xml;charset=utf-8
Transfer-Encoding: chunked
I am guessing the problem is in duplicate "Transfer-Encoding" header, as it might get appended to the first like "chucked, chunked" and oracle might not be smart enough :D to parse it.
Anyone has experienced this kind of a problem? Any suggestions?
Run in comments if you want more details to be able to help.
Thanks in advance!