EDIT: Learned that Webmethods actually uses NLST, not LIST, if that matters
Our business uses the WebMethods integration server to handle most of our outbound communications, and its FTP functionality leaves something to be desired. We are having a problem that may be specific to WebMethods, but if anyone can point me in a direction of what kinds of things might cause this I'd appreciate it.
When polling two of our partners' FTP servers, we connect without issue but, when doing a NLST on a directory that is empty (no files and no subdirectories) it's timing out. The actual error is:
com.wm.net.ftpCException: [ISC.0064.9010] java.net.SocketTimeoutException: Accept timed out
It's being thrown during the invocation of the pub.client.ftp:ls service. I've logged in with a number of FTP clients without a problem to the same sites. I've used whatever the default FTP client is in windows, FileZilla and lftp. All without issue. The servers themselves aren't the same FTP server software from what I can tell. One is Microsoft FTP, the other I'm uncertain on but is definitely not Microsoft.
Any idea what could cause an FTP client to timeout when waiting for a NLST response on an empty directory? The visible responses from the FTP server appear to be the same, but is there a difference in how NLST responds for an empty directory that I'm unaware of?
This problem is consistent on these two servers. Everything functions fine on directories with files or subdirectories within it, but not when empty.
Any thoughts or directions would be appreciated.
Thanks!
Eric Sipple
I tried this in WebMethods IS Version 6.5 Updates WmPRT_6-5-1_SP1, IS_6-5_SP3.
It worked perfectly first time.
I turned on debugging on the FTP server (Debian's default ftpd). WebMethods' NLST honours the active/passive parameter passed to it.
There's nothing special about the NLST command, nor its correct behaviour with an empty directory -- if LIST works, then so should RETR, STOR and NLST. If NLST works with a non-empty directory, it should work with an empty one.
So my guess is that either:
Your version of WM has a bug mine doesn't
Your FTP server has a bug mine doesn't
There's a wacky protocol-aware firewall in your system that doesn't like FTP data sockets with no data in them.
Firewall vendors are a bit wayward when it comes to FTP...
When testing with other clients, make sure it's from the same machine on which WebMethods Integration Server is running.
Just for the record, here's what should happen for an active NLST
client opens a listening socket, and sends a PORT command with that socket's details
client sends NLST command
server connects to client's listening socket (this is the data socket)
server transmits listing over data socket (in this case, zero bytes)
server closes data socket
... and in passive mode
client sends PASV command
server opens a listening socket, and replies with PASV response containing its details
client connects to listening socket (this is the data socket)
client sends NLST command
server transmits listing over data socket (zero bytes again)
server closes data socket
I am not sure if it was the same problem but I had similar symptoms a while ago using another FTP client in Java (commons.net). The problem turned out to be caused by the active/passive mode of the connection.
I am sorry I can't give you more details, that's all I can remember... hope that help.
Guillermo Vasconcelos was correct in his answer. There are two FTP modes, Active and Passive. The default FTP mode is active. Active requires the server to connect back to the client on some TCP/IP port. This does not work with firewalls because chances are that this port would be blocked or if you are behind a Router with NAT, not mapped.
If you use Passive (PASV) mode instead, you should not get the hang.
I'm going to run some new tests with the settings to passive tomorrow when maintenance is done here, but I'm not sure that's the issue. We are able to get a directory listing if there are files or subdirectories in that directory. It only fails when the directory we're NLST-ing on is empty.
Would the active/passive difference only manifest for an empty directory, or is there another possibility?
FTP requires that both the specified port and the one above it be opened through the firewall. When I had problems with webMethods timing out, it was because the firewall did not have the return port open.
Howard
Related
I'm trying to send a file to a third party ftp server (hosted by Amazon it would appear) with a Camel FTP Producer, and am having an issue where I am getting Writing File failed with: File operation failed... Host attempting data connection x.x.x.x is not the same as server y.y.y.y which I've not seen before.
The producer is configured to be in passive mode, and according to the logs at TRACE level, this is enabled. (Although the error message sounds like it would relate more to an active mode issue)
The y.y.y.y IP address is one of those listed by nslookup for the target domain, so that bit makes sense. However, the x.x.x.x IP relates to a different Amazon hosting server, and so I presume some sort of hand-off or load-balancing has been performed, and the FTP client doesn't like that.
Is there some way of configuring Camel FTP to allow this (I'm presuming this is a security feature), or should passive mode allow this anyway?
I have no influence with the ftp server provider, so unfortunately I can't change anything but my client options.
Thanks for looking!
After some digging, and grepping the source code for Apache Commons FTP, the message in question is caused by a validation in the client which checks that the passive mode connection is the same as the initial server connection.
Given that this appears to be a load-balanced system, the passive mode data connection is a different IP from the target IP, and this fails the validation.
It can be fixed using Camel FTP by creating a specific instance of the FTPClient and setting remove verification off.
FTPClient ftp = new FTPClient();
ftp.setRemoteVerificationEnabled(false);
registry.put("FTPClient", ftp);
and then referencing this object in the URI for FTP
ftp://user#host:21/path?password=xxxx&passiveMode=true&tempPrefix=part.&ftpClient=#FTPClient
Clearly, by disabling this remote verification test you are making yourself more vulnerable to the FTP data being redirected or intercepted and your data being sent somewhere you didn't intend, but I guess if you're worried about that you wouldn't be using still be using unencrypted FTP in the first place.
I have a C# Winforms application that iteratively FTPs files from an FTP server, parses the file for information, then returns the information from each file in a loop. After so many FTP pulls into memory (I'm loading the text files into an array) Mcafee sees my pulls as a port scanning virus and disables to connection.
I thought delaying the thread (Thread.Sleep(int)) might trick my virus scanner from getting this error but the tradeoff is efficiency. Does anyone know the specs on how fast I can run and not get this port scan error? I'm not going outside the company firewall (both my laptop and the FTP server and within the firewall).
The reason for the warning is that for each transfer of a file a new connection between server and client is opened and in general the port is increased by one each time this happens. From the outside this can look like a port-scan to so called personal firewalls leading to this effect.
There are a couple of possible solutions:
It seems that you use the so called active mode for the data transfer, i.e. the FTP server is asked to open the data connection to your system. Switch to passive mode where the connections are established by your client, so the sympthom of an incoming port-scan shouldn't exist anymore keeping your personal firewall quiet.
Whitelist your application or the peers server in your personal firewall preventing it from blocking things.
Change the setting of your FTP-client (a java library you use in your program or why is there a Java-tag with this question?) to use the same port for the data transfer with active mode. Because it's always the same port, this should keep your personal firewall quiet as well.
I'm connecting and logging in to a server on port 21. What I want to do next is to send a command string, however I have been instructed to send this command through port 50.
How do I change the port to 50 whilst connected on port 21?
I have tried connecting to the server on port 50 intially, and the onnection is refused. I have tried sending the command whilst connected on port 21 using ftpClient.sendCommand and I get a reply code 500 Unknown command
You should go back to the one who told you this requirement and ask about specifics because - to be blunt - the whole thing doesn't make real sense:
FTP works with two connections, control and data. The control channel is the one you open on port 21 and there is only this one, you can't change it back to another port afterwards. So "send a command on port 50 but connect on port 21" is not a thing with FTP.
A data connection on a port < 1024 is not usual, so it's hard to believe that the one with that requirement meant that, either. For one that would require active FTP connections that everybody nowadays tries to avoid because they are a pain to configure. Opening a listener on a port < 1024 would require root-privileges on Unix-systems which is unlikely you will get just for doing FTP-transfers.
You also said that the command you actually try to send to the server is confidential. I find that hard to believe but if it really is that would mean that we're talking about a non-standard FTP-server working with its own set of FTP-commands, i.e. regular FTP-clients might not be able to do what you need at all.
So go back to the one who gave you this requirement and ask, what the heck he's talking about, ideally with an example how to do it, let's say using the FTP-client that comes with the Windows Command-Shell.
Typically when you connect, the server controls the port assignment. So when you issue a pasv command (passive mode), it typically sends you back an IP / Port to connect to for the data connection. I'm not aware of anything that will allow you to do what you want. The server can restrict the port range for these types of things. Here's a nice explanation that goes over this.
Active vs Passive FTP
You may be able to send a PORT command to connect to port 50 specifically, but it's really unusual to do something like this.
** Edit **
There's two things you can try, I've never used either, so YMMV.
In Active mode, the client gives the server a port to connect to.
Try setting the default port to 50 and turn on Active mode. Hopefully this is on your internal networks, because this would never get past security for a firewall request.
client.setDefaultPort(50);
client.enterLocalActiveMode(); // Apache FTPClient
For Passive mode, you can try to set the active port range.
client.setActivePortRange(50, 50)
client.enterLocalPassiveMode();
You should ask what mode the server is expecting.
From a FTP server perspective, if a client requests a file through RETR command, the server creates a data connection (socket) to the client through the specified port and starts the transfer by writing in the outputstream. The server is coded (JAVA) in such a way that after the write is complete in the socket, the outputstream is flushed and then the socket is closed. After this the code "226" is sent to the client in control channel.
Since the connection is over a very slow network, the 226 message reaches before the actual data transfer is complete. This is a tricky situation where the client code cannot be changed and the server has to make sure that the 226 is sent after client received the data.
I tried searching in the internet and got few inputs, but not sure which one is the standard.
1. to use setSoLinger() method to turn on SO_LINGER and to set timeout.
2. to introduce a delay after writing each byte in to socket (performance will be impacted for fast connections).
Is there any other options other than the above to solve the situation. Any idea about the standard followed for sending 226 in Linux/ Solaris/Windows FTP Servers.
I could see a similar thread in stackoverflow "When should 226 be sent from the FTP server?" , but could not find much info from that related to my question.
Help here is really appreciated...Thanks
Do not go with the delay for sure, the only thing I can think of is that you build a proxy layer that intercepts the acknowledge code, checks for the file, and reroutes the code to the application, sort of what telerik fiddler does as an application.
The same concept I used before with the JMS acknowledge mode when delivering messages to the server and I had to implement the same.
Wish you all luck my friend
When I run ActiveMQ by executing the batch file in its bin/ directory, I am able to go to its admin/management console by opening a web browser and going to http://localhost:8161/admin/.
This has me curious.
This is my local sandbox and I do not have any web server (httpd or otherweise) installed. So how is it that ActiveMQ is able to "register" a port on my machine, and listen to it exclusively?
If I try to go to http://localhost:8162/admin/ (notice the different port #), I get an unable to connect error.
Somewhere, somehow, AMQ is saying "map this URI (localhost:8161) to some root directory on this machine". As a programmer, I'm interested in how something like this works.
A Java process is able to use any port (>= 1024 on linux) as a web server or for any other purpose. You don't need a separate web server to do this
I suggest you read up on sockets: here. All a web server is, is a socket listener that handles the HTTP protocol. HTTP protocol is here.
Web servers often handle a lot of other things, but that is the basics. If you want a small program that also runs a web server I suggest not re-inventing the wheel. Try incorporating jetty into your server.
ActiveMQ starts an embedded Jetty server, which listens for HTTP connections on that port. You don't need any other server running. It's all done from Java. If you dig down deep enough, you'll find some variety of ServerSocket at the bottom of it all. You can learn all about sockets and listening on ports in the Java Tutorial.
At its simplest level, ActiveMQ is creating a ServerSocket instance within itself and listening for connections using this server socket. A socket is always bound to a port.
One: this port is greater than (or equal to) 1024, so it means a "non root" user can listen on it.
Second: you can bind to ports from dedicated addresses only. This means ActiveMQ may have only opened that port on 127.0.0.1 (localhost). Try and see if you can open that URL from your external interface's IP address: chances are you cannot.
If you are under a Unix system, you can check what program listens on which port by using netstat -ltpn.
The basic system call for binding to a port is listen(2).