We Keep Coding

After retrieving multiple files I get an error on the next file '425 can't open data connection for transfer of file'

I'm using apache commons-net FTPClient (ver 3.3) and below error was produced on one of my clients machine (I've tried reproducing the error on my dev machine without luck using testing folder with the same requests on the same server with the same login)
I have a process that check's the remote FTP server for new requests in form of an XML-files. After listing all of those files i proceed in loop to check if they're in XML format. If the file is in this format I do first change their name by changing format from *.xml to *.xmlProcessing, retrive them to a input stream, parse them to my object and create a request in my queue and finally change the name and move them to subfolder working as an archive.
After downloading random amount of files I get stuck while calling retrieveFileStream on the next file, without a timeout or IO Exception.
I've managed to get logs from FTP server and it just says it can't open a data connection
05.06.2019 12:45:45 - > RNFR /folder/file.xml
05.06.2019 12:45:45 - > 350 File exists, ready for destination name.
05.06.2019 12:45:45 - > RNTO /folder/file.xmlProcessing
05.06.2019 12:45:45 - > 250 file renamed successfully
05.06.2019 12:45:45 - > PORT *ports*
05.06.2019 12:45:45 - > 200 Port command successful
05.06.2019 12:45:45 - > RETR /folder/file.xmlProcessing
05.06.2019 12:45:45 - > 150 Opening data channel for file download from server of "/folder/file.xmlProcessing"
05.06.2019 12:45:45 - > 425 Can't open data connection for transfer of "/folder/file.xmlProcessing"
I've already tried diffrent FTP modes. Active local, remote, passive etc. (currently stuck with passive local mode).
I've tried the data timeout but it looks like while i finally got stuck on one of the files the method took more than 1 minute on that file despite me setting the timeout on 30s.
ftp = new FTPClient();
ftp.addProtocolCommandListener(new PrintCommandListener(new PrintWriter(System.out)));
ftp.connect(server, port);
int reply = ftp.getReplyCode();
if (!FTPReply.isPositiveCompletion(reply)) {
ftp.disconnect();
throw new IOException("Exception in connecting to FTP Server");
}
ftp.login(user, password);
ftp.enterLocalPassiveMode();
ftp.setKeepAlive(true);
ftp.setDataTimeout(30000);
Collection<String> listOfFiles = listFiles(FOLDER_PATH);
for(String filePath : listOfFiles){
if (filePath != null && filePath.endsWith(".xml")) {
ftp.rnfr(folder + filePath);
ftp.rnto(folder + filePath + "Processing");
InputStream fileInputStream = ftp.retrieveFileStream(folder + filePath + "Processing");
ftp.completePendingCommand();
//Parsing file to an instance of my object and creating request
ftp.rnfr(folder + filePath + "Processing");
ftp.rnto(archiveFolder + filePath);
if(fileInputStream != null){
fileInputStream.close();
}
}
}
Is there's a bigger likehood that this is fault of the FTP Server, Firewall issues or something else ?
I've runed the same code from my dev machine and it processed all files from test folder (there were around 400 of them) i don't know if im being unlucky for error not occuring on my local dev machine or is it actually something wrong with communication of my contractor with the remote server ?

Everytime you do a file transfer or a directory listing with FTP, the server (or client if using an active mode) assigns a random port number out of a configured range to that transfer. The port number is not released immediately, when the transfer completes. There's some cooldown interval. If you do too many file transfers in a short time interval, it can happen that the server runs out of the available ports – Because all ports end up in the cooldown state.
If you can, check the server configuration and configure a larger range of ports.
Or as a workaround, you can try to slow down the transfer rate.
For some background, see:
How many data channel ports do I need for an FTP?
Why does FTP passive mode require a port range as opposed to only one port?
Though this is just a guess, you should check the server's log, as it can show more details.
Another possibility is, that there's simply a limited number of transfers the server allows for a specific user or source address in some time interval.

Unable to write to OutputStream [java.io.IOException: Broken pipe]

I am trying to send HTTP request to Abstract Unix Socket within an Android Application, using LocalSocket but it gives java.io.IOException: Broken pipe. Status of isConnected() is true at the same time and I have also confirmed that the request I send is correct by forwarding the port using adb and making the same request using netcat.
LocalSocket receiver = new LocalSocket();
receiver.connect(new LocalSocketAddress("chrome_devtools_remote", LocalSocketAddress.Namespace.ABSTRACT));
InputStream input = receiver.getInputStream();
StringBuilder request = new StringBuilder().append("GET /json HTTP/1.0")
receiver.getOutputStream().write(request.toString().getBytes());
Unable to figure out what's going wrong. Help appreciated, Thanks in advance.

FTP to 1and1.com

The code below works to two other sites I've tried, but will not work with my domain hosted by 1and1. The return code is always 500 - Permanent Negative Completion reply.
I know I'm connecting because FTPReply.isPositiveCompletion(reply) returns true. I tried it with both reading file off phone storage and sending it from a byte array populated early in the code. This is the byte array. Both return 500. Both work on the other sites.
If I don't use enterLocalPassiveMode() the code stops executing on the storeFile call. No exception, no socket time-out. It just ends there and the async task will not call again in that session. The code does not do that on the other sites.
I've tried both ASCII and BINARY file types. Both return 500. 1and1 site says to use my domain and port 21. I can connect with CoreFTP and read and write using both of the accounts I've set up.
I also tired ftp4j and had the same response with all scenarios so went back to Apache because the code was already written with robust error trapping.
I've tried both mydomain.com and ftp.mydomian.com. 500 on both. I also tried the dot quad I can see in the CoreFTP window, but i get "cannot resolve host name" with the Apache Java code. Maybe not a static IP?
This is what CoreFTP does. It connects on port 21 and then goes in to passive mode and an ASCII data connection.
It's a long shot, but has anyone else ever FTPed to their 1and1 domain using Java in Android Studio?
Greg
Resolving mydomain.com...
Connect socket #5684 to xx.xx.xx.xxx, port 21...
220 Microsoft FTP Service
USER ftp79815757-0
331 Password required for ftp79815757-0.
PASS **********
230 User logged in.
SYST
215 Windows_NT
Keep alive off...
PWD
257 "/ftp79815757-0" is current directory.
PASV
227 Entering Passive Mode (xx,xxx,xx,xxx,xxx,xxx).
LIST
Connect socket #5700 to xx.xx.xx.xx, port 62894...
150 Opening ASCII mode data connection.
226 Transfer complete.
Transferred 51 bytes in 0.094 seconds
FTPClient mFtpClient = new FTPClient();
String ip = "my domain dot com";
String userName = "ftp79815757-0";
String pass = "password";
mFtpClient.connect(InetAddress.getByName(ip));
mFtpClient.login(userName, pass);
int reply = mFtpClient.getReplyCode();
if (FTPReply.isPositiveCompletion(reply)) {
mFtpClient.setFileType(FTP.BINARY_FILE_TYPE);
//one thread said this would do the trick
mFtpClient.enterLocalPassiveMode();
mFtpClient.enterRemotePassiveMode();
InputStream stream = new ByteArrayInputStream(imageData);
//I have two accounts. One points to images_in
/*if (!mFtpClient.changeWorkingDirectory("images_in")) {
Log.e("ChangeDir", String.valueOf(mFtpClient.getReplyCode()));
}*/
if (!mFtpClient.storeFile("remoteName.jpg", stream)) {
Log.e("FTPUpload", String.valueOf(mFtpClient.getReplyCode()));
}
stream.close();
mFtpClient.disconnect();
}

Finally got it. The main problem was that I was using an old version of the Apache library. The Jar I was using was commons-net-1.4.jar. Someone in another thread pointed me to commons-net-3.3.jar.
I commented out both mFtpClient.enterLocalPassiveMode() and mFtpClient.enterRemotePassiveMode(), and with some trail and error it worked with FTP.BINARY_FILE_TYPE and not ASCII_FILE_TYPE. ASCII got the file there, but it was garbage.

SSL Socket OutputStream Write Slow only on Windows

Server: Linux
Tested Clients: OS X, CentOS, Windows
Server/Client Programming Language: Java
Server-side
SSLServerSocketFactory sslserversocketfactory =
(SSLServerSocketFactory) SSLServerSocketFactory.getDefault();
SSLServerSocket sslserversocket =
(SSLServerSocket) sslserversocketfactory.createServerSocket(9999);
SSLSocket sslsocket = (SSLSocket) sslserversocket.accept();
InputStream inputstream = sslsocket.getInputStream();
InputStreamReader inputstreamreader = new InputStreamReader(inputstream);
BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
String string = null;
while ((string = bufferedreader.readLine()) != null) {
System.out.println(string);
System.out.flush();
}
Server Program executed with this option
-Djavax.net.ssl.keyStore=mySrvKeystore -Djavax.net.ssl.keyStorePassword=123456
Certificate: Created with keytool on Linux
keytool -genkey -keystore mySrvKeystore -keyalg RSA
Client Side
....
SocketAddress sa = new InetSocketAddress(ip, port);
....
SSLContext sslContext = SSLContext.getInstance("SSL");
....
sslContext.init(.........., new SecureRandom());
SSLSocketFactory socketFactory = sslContext.getSocketFactory();
this.clientSock = socketFactory.createSocket();
this.clientSock.connect(sa, this.ConnectTimeout);
this.clientSock.setSoTimeout(this.RecvTimeout);
this.clientSock.setReuseAddress(true);
....
public boolean writeTo(String procname, BufferedOutputStream out, byte[] data)
{....
out.write(data, 0, data.length);
out.flush();
....}
Results: Everything works great but only client program from windows, it gets delayed when write to outputstream.
SSL Socket connection made
Set Output/Input stream
Write data into output stream
flush
... for 4-5 secs, it got stuck here with no reason at this point ...
close
The delay happens on the same java client simple program executing from Windows 7, Windows XP.
Tested from 3 different windows machine.
Everywhere else works great.
So I tried to build SSL simple clients with C and PHP, works great from Windows.
That means only Java clients method is not working.
Does anyone have any idea or similar experience before?
I'v seen some posts suspecting on WINS/DNS, but it seems like it's not the case here.
Also the funny thing is, it's always about 4-5 seconds of delay from every windows machine.
Thanks a lot for your comments in advance.

I've experienced a similar issue. Using tcpdump, I found that the TCP connection from client to server (SYN, SYN-ACK, ACK) happens almost instantaneously, then a mysterious pause, then the client actually sends the data.
The tcpdump output indicates that, immediately after connecting, the client (running on Windows) sends a NetBIOS name request (presumably to find out the name of the server, perhaps so it can choose an appropriate certificate or something) and the server immediately replies with an ICMP response saying the NetBIOS UDP (137) port is unreachable. It looks like the windows client ignores that response. After 1.5 seconds the windows client sends another NetBIOS name request, and the server sends the same ICMP response. After another 1.5 seconds the windows client sends a third and final NetBIOS name request, and the server sends back the same ICMP response. The socket data is finally transmitted 1.5 seconds after that.
So it looks like windows tries three times to determine the NetBIOS name of the server, and waits 1.5 seconds each time. Eventually (after 4.5 seconds) it gives up and sends the data. I think this explains the 4-5 second pause you reported.
One workaround to eliminate the pause is to add a 'hosts' file entry for the server. (It doesn't really matter what hostname you use within that file. The mere presence of an ip address avoids the need for the NetBIOS name resolution.) There are probably other alternatives too (e.g. add the server name to WINS or DNS or something, depending on what your windows machine is using for name resolution).

You're complaining about writing and you're only showing the reading code. What does the writing code look like? If you're not using a buffered stream or writer in the stack to the SSLSocket you can get data size explosions of up to 44 times, which would affect performance severely. I did some extensive testing over the Internet a few years ago and the conclusion was that with properly written code, SSL isn't any worse than 3 times as slow as plaintext.

Sockets in Java...?

I need to build an application which can receive data from over a network and use this data to do some unrelevant things with.
Here's a piece of code to make clear what I'm doing.
On the server side:
static Socket client = null;
static ServerSocket ss = null;
if (ss != null) {
ss.close();
}
ss = new ServerSocket(5513);
isrunning = true;
System.out.println("Waiting for client...");
client = ss.accept();
System.out.println("Client accepted.");
BufferedReader in = new BufferedReader(new InputStreamReader(client.getInputStream()));
And the client side:
Socket client = null;
PrintWriter out = null;
try {
client = new Socket("hostname", 5513);
out = new PrintWriter(client.getOutputStream(), true);
}
Please note that this is just a piece of the code. There are no errors in the code.
After running the server-sided piece of code, it correctly waits for the client to connect.
Now here comes the problem. As soon as I try to connect from the client side, I'm getting a "connection refused"-error.
HOWEVER, I found something on the internet whoch told me to try telnetting from the client side. For example, let the server-sided IP be 192.168.1.1. So, after using this command:
telnet 192.168.1.1 5513
I actually get a connection with the server. The command will launch an empty screen, and everything I manually type in the command line will be sent to the server-side after pressing enter (checked with debugging).
So, I can manually connect to the server-side and send some data, but my code refuses to connect.
Anyone who knows what I am doing wrong?

Is this the code you're actually using?
client = new Socket("hostname", 5513);
Try changing it to:
client = new Socket("192.168.1.1", 5513);

client = new Socket("hostname", 5513);
Hostname needs to represent the IP Address you're connecting to. If you're trying to connect to yourself, it would be "localhost"
Also, the server is not listening for the client AT ALL TIMES, there must be a while loop so the server listens and accepts connections.
while (true) {
client = ss.accept();
out = new PrintWriter(client.getOutputStream(), true);
//You should probably assign it to a seperate thread to handle stuff for this client
}
And I should explain on why you're getting that particular error. When something says that the connection is refused, it usually means that the IP Address you want to connect to knows your sending a connection and is blocking it because it was not listening for that connection. Basically, when the server closed, you stopped listening for the client, so anything that came in on that port would be blocked. Of course, the other case could be that Java was blocked on your firewall and an exception should be made for it. Although this is rarely the case if what you're trying to accomplish is over a LAN.

You're not actually using "hostname" in your Socket object in the client are you?
It should the 192.168.1.1.

Are you on Windows? and If so have you added java.exe and javaw.exe to Firewall with inbound and outbound enabled? and have you added a rule for 5513 to your Firewall?
If yes Windows but no Firewall settings, that's your answer, open up your Firewall.