We Keep Coding

425 Failed to establish connection

I'm trying to download a file via FTP with a Java application.
The FTP url is accessible from this web page: http://professionnels.ign.fr/adminexpress.
More specifically, I'm trying to download this file.
From my home, I can download the file successfully with my java application, Firefox or Chrome.
From my work, I can do the same with Firefox and Chrome only. My application refuses to download anything.
NOTA: At work, the browsers and my application use the same HTTP proxy to access internet.
I'm using Apache Commons Net 3.6.
Here is a sample of the FTP exchanges of my application. I wasn't able to sniff those of Chrome or Firefox.
220 Bienvenue sur le site FTP de L INSTITUT NATIONAL DE L INFORMATION GEOGRAPHIQUE ET FORESTIERE
USER *******
331 Please specify the password.
PASS *******
230 Login successful.
TYPE I
200 Switching to Binary mode.
PASV
227 Entering Passive Mode (192,134,132,16,65,180).
RETR /ADMIN-EXPRESS-COG_2-0__SHP_WM__FRA_2019-05-20.7z.001
425 Failed to establish connection.

tl;dr
It turned out that the HTTP proxy at my work already handles all the FTP exchanges. This is why Firefox and Chrome could download the file. When they aren't behind an HTTP proxy, it seems they act as an FTP client by sending FTP commands directly.
A simple HTTP GET request to the HTTP proxy with the ftp url is enough to download the file.
Here is a sum up of solutions I found during my investigations:
Use passive mode (PASV command)
Check if there's an FTP proxy to use rather than an HTTP Proxy
Check the configuration of the FTP server (if you have access to it)
Check the configuration of the HTTP proxy (if you have access to it)
Precisely, the browsers perform a simple HTTP request as described below:
GET ftp://user:passw0rd#example.com/file.ext HTTP/1.1
Host: example.com
User-Agent: WebBrowser-UA/x.y.z
...
Then the HTTP proxy parses the FTP url and connects to the FTP server. The HTTP proxy returns the file content as a normal HTTP response.
HTTP/1.1 200 OK
Last-Modified: Tue, 21 May 2019 11:23:00 GMT
Content-Length: 115545060
Content-Type: octet/stream
Connection: Keep-Alive
Age: 22
Date: Thu, 27 Jun 2019 10:27:09 GMT
(file content here...)
However, in my case, the HTTP proxy allowed me to connect to the FTP server and exchange on the command FTP channel only. The data channel seemed to be blocked either in ACTIVE or PASSIVE mode.
During my investigations, I found many people hitting this very same problem. The solutions they found (when they found one...) didn't apply to me. Here is a sum up of the solutions expressed in all those questions:
Use passive mode (PASV command)
Check if there's an FTP proxy to use rather than an HTTP Proxy
Check if the HTTP proxy handles directly the FTP exchanges
Check the configuration of the FTP server (if you have access to it)
Check the configuration of the HTTP proxy (if you have access to it)
References:
Understanding FTP over HTTP
Connect to FTP server through http proxy
FTP connection through proxy with Java
Accessing FTP server behind a proxy via command prompt in Windows 7
[vsFTPd] 425 Failed to establish connection.

java: getting Connection refused when trying to upload file to FTP

When trying to upload file to FTP with java program:
public void upload(String localFile,String remoteFile) throws Exception{
ftp = new FTPClient();
ftp.setControlKeepAliveTimeout(300);
ftp.connect(host,21);
ftp.enterLocalPassiveMode();
ftp.setUseEPSVwithIPv4(false);
ftp.login(user,password);
ftp.setFileType(FTPClient.BINARY_FILE_TYPE);
FileInputStream in = null;
in = new FileInputStream(localFile);
ftp.storeFile(remoteFile,in);
in.close();
ftp.disconnect();
}
I'm getting:
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:381)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:243)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:230)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:377)
at java.net.Socket.connect(Socket.java:539)
at org.apache.commons.net.SocketClient._connect(SocketClient.java:243)
at org.apache.commons.net.SocketClient.connect(SocketClient.java:202)
When I try to upload the same file with command line (from linux), I'm able to do it only when using EPSV:
llnx:~ ftp anonymous#9.20.1.116
Connected to 9.20.1.116.
220 Microsoft FTP Service
331 Anonymous access allowed, send identity (e-mail name) as password.
Password:
230 User logged in.
Remote system type is Windows_NT.
ftp> epsv
EPSV/EPRT on IPv4 off.
ftp> put /tmp/file1.xml /dir_1/file1.xml
local: /tmp/file1.xml remote: /dir_1/file1.xml
227 Entering Passive Mode (10,40,1,149,233,168).
125 Data connection already open; Transfer starting.
100% |*************************************| 117 KB 28.66 MB/s --:-- ETA
226 Transfer complete.
120032 bytes sent in 00:00 (7.96 MB/s)
So, Why does my java code getting Connection refused?
Maybe I'm not using the enterLocalPassiveMode() or setUseEPSVwithIPv4() method the right way?
*** I think the answer is how to run the EPSV command from Java program.
Thank you all.
Eithan.

This is purely a guess but java.net.ConnectException: Connection refused normally happens when there is nothing listening on the target host/port. You don't specify a port in the CLI example so maybe that is the problem. Try changing ftp.connect(host,21); to ftp.connect(host); to use the default. Also confirm the the hostnames are exactly the same.
This assumes that the error is on the call to connect(). You haven't provided a big enough stack trace to indicate either way.

Connection refused means that your TCP connection request has reached the remote server (or more correctly >>a<< remote server) but the server is not expecting / listening for an incoming connection. So it "refuses" it.
Here are the things to check:
Check that you have the correct remote hostname or IP address for the FTP server.
Check that you are using the correct port for the FTP server. Port 21 is the default, but it is possible that the server is on a non-standard port.
Check that the FTP server is actually running.
It is also possible that the problem is due to a firewall doing something deliberately confusing. But that is unlikely for a publicly routable FTP server.
Maybe I'm not using the enterLocalPassiveMode() or setUseEPSVwithIPv4() method the right way?
That can't be the problem. The stacktrace shows that your application failed while trying to establish the initial connection to the server. You haven't gotten to the point where the you can make those calls.

Getting "Unable to determine system type - response: 500 Unknow command" when listing files on ESP8266 with Apache Commons Net

I try to connect to FTP server in ESP8266. Connection is successful, but I can't get list of files on the server.
My code is:
FTPClient mFtpClient = new FTPClient();
mFtpClient.setConnectTimeout(10000);
mFtpClient.connect(InetAddress.getByName(ip));
status = mFtpClient.login(userName, pass);
Log.e("isFTPConnected", String.valueOf(status));
if (FTPReply.isPositiveCompletion(mFtpClient.getReplyCode())) {
mFtpClient.setFileType(FTP.BINARY_FILE_TYPE);
mFtpClient.enterLocalPassiveMode();
FTPFile[] mFileArray = mFtpClient.listFiles();
Log.e("Size", String.valueOf(mFileArray.length));
}
In logical I get the error :
java.io.IOException: Unable to determine system type - response: 500 Unknow command.
I use Apache Commons Net FTP library. So what is wrong in my code? From FileZilla Windows client, I can connect. May be the reason is that in ESP is SPIFF file system? Or another one reason?
Thanks for answers, and interest!

Your server does not support SYST command, that the FTPClient needs to decide how to parse a response of LIST command.
Solutions are:
If your server supports MLSD command, use mlistDir instead of listFiles.
Or use System.setProperty to set FTP_SYSTEM_TYPE_DEFAULT or FTP_SYSTEM_TYPE to suggest what directory listing format your server is using.

JAVA FTPSClient file listing issue

I am using apache commons-net 3.6 library to connect FTPS server. FTPS server is behind NAT of thirdparty. and I can't change any settings on server side.
I can login to server, but can not list files. I've tried same code with some public FTP and FTPS servers, and result was successfull. Seems that they are not behind NAT. But filezilla can successfully connect and list files from my problematic server.
There is my code
ftps.connect(server, port);
System.out.println("Connected to " + server + ".");
reply = ftps.getReplyCode();
ftps.enterLocalPassiveMode();
if (!FTPReply.isPositiveCompletion(reply)) {
ftps.disconnect();
System.err.println("FTP server refused connection.");
System.exit(1);
}
if (!ftps.login(username, password)) {
ftps.logout();
}
// ftps.setEnabledSessionCreation(false);
ftps.feat();
ftps.execPBSZ(0);
ftps.execPROT("P");
ftps.setFileType(FTP.BINARY_FILE_TYPE);
FTPFile dirs[] = ftps.listDirectories();
And there is my ftps log:
220 FTP Server ready.
AUTH TLS
234 AUTH TLS successful
Connected to x.x.x.x
USER *******
331 Password required for azercell
PASS *******
230 User myuser logged in
FEAT
211-Features:
MDTM
MFMT
LANG bg-BG;en-US;fr-FR;it-IT;ja-JP;ko-KR;ru-RU;zh-CN;zh-TW
TVFS
UTF8
AUTH TLS
MFF modify;UNIX.group;UNIX.mode;
MLST modify*;perm*;size*;type*;unique*;UNIX.group*;UNIX.mode*;UNIX.owner*;
PBSZ
PROT
REST STREAM
SIZE
211 End
PBSZ 0
200 PBSZ 0 successful
PROT P
200 Protection set to Private
TYPE I
200 Type set to I
SYST
215 UNIX Type: L8
PASV
227 Entering Passive Mode (192,168,2,50,192,12).
[Replacing PASV mode reply address 192.168.2.50 with x.x.x.x]
LIST
150 Opening BINARY mode data connection for file list
425 Unable to build data connection: Operation not permitted
I'd read that prior to version 3.6 commons-net library prior couldnt handle behind NAT connections properly.
Can anyone help me? What is wrong with my code?

So my conclusion is problem was not related to NAT technology, apache-commons 3.6 does not handle all FTPS options properly. As I mentioned before we were integrating with 3rd party and had not option to change FTPS settings, at least we installed filezilla ftp server and were able to reproduce error. Fortunately I found solution at http://eng.wealthfront.com/2016/06/10/connecting-to-an-ftps-server-with-ssl-session-reuse-in-java-7-and-8/ by Luke Hansen. Great thanks him

Manual FTP with commons.net.ftp

I am trying to use apache commons to send manual FTP commands as I have to send non-standard FTP commands to a specific server (that accepts them)
Before I try to send these non-standard commands I want to get FTP working manually with commons.net.ftp. Unfortunately I seem to be missing something.
This works fine (i.e. it retrieves the list of files)
FTPClient ftp = new FTPClient();
FTPClientConfig config = new FTPClientConfig();
ftp.configure(config);
ftp.connect("ftp.mozilla.org");
ftp.login("anonymous", "");
ftp.enterLocalPassiveMode();
FTPFile[] fileList = ftp.listFiles("/");
This doesn't
FTPClient ftp = new FTPClient();
FTPClientConfig config = new FTPClientConfig();
ftp.configure(config);
ftp.connect("ftp.mozilla.org");
ftp.login("anonymous", "");
ftp.sendCommand("PASV");
ftp.sendCommand("NLST");
I get the appropriate response for ftp.sendCommand("PASV"); but it times out on ftp.sendCommand("NLST"); finally giving me
425 Failed to establish connection.
I have tried to research the topic but most advice on this error is for people setting up servers (and it's usually a firewall problem).
Why does it work when net.ftp does it, but not when I send the commands manually?

Sending the PASV command manually is not enough, the client has to open the data connection to the port specified by the server in response to the PASV command. This is performed by calling the enterLocalPassiveMode() method. Since sending PASV manually doesn't initialize the data connection you get an error shortly after.
See http://slacksite.com/other/ftp.html#passive for more details on the FTP protocol.