We Keep Coding

Uploading a file to testcontainer FTP server fails with Connection refused after being connected

I'm working with FTPClient against an FTP server using Testcontainers.
A reproducible code sample is here:
import org.apache.commons.net.PrintCommandListener;
import org.apache.commons.net.ftp.FTP;
import org.apache.commons.net.ftp.FTPClient;
import org.apache.commons.net.ftp.FTPReply;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.images.builder.ImageFromDockerfile;
import org.testcontainers.junit.jupiter.Testcontainers;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.PrintWriter;
import static org.assertj.core.api.Assertions.assertThat;
#Testcontainers
class FtpUtilsTest {
private static final int PORT = 21;
private static final String USER = "user";
private static final String PASSWORD = "password";
private static final int FTP_TIMEOUT_IN_MILLISECONDS = 1000 * 60;
private static final GenericContainer ftp = new GenericContainer(
new ImageFromDockerfile()
.withDockerfileFromBuilder(builder ->
builder
.from("delfer/alpine-ftp-server:latest")
.build()
)
)
.withExposedPorts(PORT)
.withEnv("USERS", USER + "|" + PASSWORD);
#BeforeAll
public static void staticSetup() throws IOException {
ftp.start();
}
#AfterAll
static void afterAll() {
ftp.stop();
}
#Test
void test() throws IOException {
FTPClient ftpClient = new FTPClient();
ftpClient.setDataTimeout(FTP_TIMEOUT_IN_MILLISECONDS);
ftpClient.setConnectTimeout(FTP_TIMEOUT_IN_MILLISECONDS);
ftpClient.setDefaultTimeout(FTP_TIMEOUT_IN_MILLISECONDS);
// Log
ftpClient.addProtocolCommandListener(new PrintCommandListener(new PrintWriter(System.out), true));
// Connect
try {
ftpClient.connect("localhost", ftp.getMappedPort(PORT));
ftpClient.setSoTimeout(FTP_TIMEOUT_IN_MILLISECONDS);
int reply = ftpClient.getReplyCode();
if (!FTPReply.isPositiveCompletion(reply)) {
ftpClient.disconnect();
throw new AssertionError();
}
// Login
boolean loginSuccess = ftpClient.login(USER, PASSWORD);
if (!loginSuccess) {
throw new AssertionError();
}
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
ftpClient.enterLocalPassiveMode();
} catch (IOException e) {
throw new AssertionError(e);
}
String remoteFile = "fileonftp";
try (InputStream targetStream = new ByteArrayInputStream("Hello FTP".getBytes())) {
assertThat(ftpClient.isConnected()).isTrue();
ftpClient.storeFile(remoteFile, targetStream);
}
}
}
This prints:
220 Welcome Alpine ftp server https://hub.docker.com/r/delfer/alpine-ftp-server/
USER *******
331 Please specify the password.
PASS *******
230 Login successful.
TYPE I
200 Switching to Binary mode.
PASV
227 Entering Passive Mode (172,17,0,3,82,15).
[Replacing PASV mode reply address 172.17.0.3 with 127.0.0.1]
then fails with:
Connection refused (Connection refused)
java.net.ConnectException: Connection refused (Connection refused)
...
at java.base/java.net.Socket.connect(Socket.java:609)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:866)
at org.apache.commons.net.ftp.FTPClient._storeFile(FTPClient.java:1053)
at org.apache.commons.net.ftp.FTPClient.storeFile(FTPClient.java:3816)
at org.apache.commons.net.ftp.FTPClient.storeFile(FTPClient.java:3846)
What I don't understand is that it fails after successfully connecting and logging in, and returning true for isConnected.
Turns out that when removing the ftpClient.enterLocalPassiveMode(); it works, but I need it to work with the passive mode.
I guess the failure is related when switching to a different port for the passive call.
but when trying to add the ports to the withExposedPorts the container fails to start with:
Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for container port to open (localhost ports: [55600, 55601, 55602, 55603, 55604, 55605, 55606, 55607, 55608, 55609, 55598, 55599] should be listening)
Running against a docker (docker run -d -p 21:21 -p 21000-21010:21000-21010 -e USERS="user|password" delfer/alpine-ftp-server) works.
Local docker versions:
Docker version 20.10.11, build dea9396
Docker Desktop 4.3.1
Testcontainers - appears to behave the same both on 1.16.2 and 1.15.3
Link to testcontainers discussion

As you already figured out in the comments, the tricky part about FTP passive mode is that the server uses another port (not 21) for communication.
In the docker image you're using, it's a port from the 21000-21010 range by default. So you need to publish (expose) these additional container ports. In docker run command you used -p 21000-21010:21000-21010 for that.
However, Testcontainers library is designed to publish to random host ports to avoid the problem, when a desired fixed port (or a range of ports) is already occupied on the host side.
In case of FTP passive mode random ports on the host side cause problems, because afaik you can't instruct the ftp client to override the port, which FTP server returned for the passive mode. You'd need something like ftpClient.connect("localhost", ftp.getMappedPort(PORT)); but for passive mode ports as well.
Therefore the only solution I see here is to use a FixedHostPortContainer. Even though it's marked as deprecated and not recommended to use because of the mentioned issues with occupied ports, I think this is a valid use case for it here. FixedHostPortGenericContainer allows to publish fixed ports on the host side. Something like:
private static final int PASSIVE_MODE_PORT = 21000;
...
private static final FixedHostPortGenericContainer ftp = new FixedHostPortGenericContainer<>(
"delfer/alpine-ftp-server:latest")
.withFixedExposedPort(PASSIVE_MODE_PORT, PASSIVE_MODE_PORT)
.withExposedPorts(PORT)
.withEnv("USERS", USER + "|" + PASSWORD)
.withEnv("MIN_PORT", String.valueOf(PASSIVE_MODE_PORT))
.withEnv("MAX_PORT", String.valueOf(PASSIVE_MODE_PORT));
Keep in mind that this solution relies on the assumption that 21000 port is always free. If you're going to run this in the environment where it's not guaranteed, then you need to tweak it to find a free host port first. Like:
private static FixedHostPortGenericContainer ftp = new FixedHostPortGenericContainer<>(
"delfer/alpine-ftp-server:latest")
.withExposedPorts(PORT)
.withEnv("USERS", USER + "|" + PASSWORD);
#BeforeAll
public static void staticSetup() throws Exception {
Integer freePort = 0;
try (ServerSocket socket = new ServerSocket(0)) {
freePort = socket.getLocalPort();
}
ftp = (FixedHostPortGenericContainer)ftp.withFixedExposedPort(freePort, freePort)
.withEnv("MIN_PORT", String.valueOf(freePort))
.withEnv("MAX_PORT", String.valueOf(freePort));
ftp.start();
}

An answer similar to the accepted one but without using deprecated functionalities.
Note that we still have to use the fixed port 21000.
public static class FTPContainer extends GenericContainer<FTPContainer>
{
private static FTPContainer container;
private FTPContainer()
{
super(DockerImageName.parse("delfer/alpine-ftp-server:latest"));
}
#SuppressWarnings("resource")
public static FTPContainer getInstance()
{
if (container == null)
{
container = new FTPContainer().withEnv("USERS", "test|test|/home/").withExposedPorts(21)
.withCreateContainerCmdModifier(e -> e.getHostConfig()
.withPortBindings(new PortBinding(Ports.Binding.bindPort(21000), new ExposedPort(21000))))
.withEnv("MIN_PORT", "21000").withEnv("MAX_PORT", "21000");
}
return container;
}
#Override
public void start()
{
super.start();
}
#Override
public void stop()
{
// Handled when JVM stops
}
}
And then you can use your FTP server instance like
FTPContainer FTP = FTPContainer.getInstance();
FTP.start();

Java | Syslog Server | How to set up a publicly accessible Syslog server using java code

I want to set up a local Syslog sever using Java code.
I Have MAC machine and another virtual Linux machine.
I found the java code below and I ran it on my MAC machine and the Syslog server is established properly.
I sent the linux command below from my MAC and it worked, The syslog server got the message.
Now I want to send from another linux which is not in my MAC network the same syslog message and received it by My syslog server.
How can I do that? Should I change something on the Java code? maybe something that related to the config.setHost() method?
Should I add a route somehow to my MAC or to the other linux?
The linux command:
nc 10.10.1.1 9899 <<< "syslog message."
The Java Code:
import java.net.InetAddress;
import java.net.UnknownHostException;
import com.mprv.sysmoduleinfra.mgmt.syslog.TCPSyslogServerConfig;
import com.mprv.sysmoduleinfra.mgmt.syslog.UDPSyslogServerConfig;
import org.productivity.java.syslog4j.SyslogRuntimeException;
import org.productivity.java.syslog4j.server.SyslogServer;
import org.productivity.java.syslog4j.server.SyslogServerConfigIF;
/**
* Syslog server.
*
* #author Josef Cacek
*/
public class Server {
public static final int SYSLOG_PORT = 9899;
public static void main(String[] args) throws SyslogRuntimeException, UnknownHostException {
// Details for the properties -
// http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html
System.setProperty("jsse.enableSNIExtension", "false");
// just in case...
System.setProperty("sun.security.ssl.allowUnsafeRenegotiation", "true");
System.setProperty("sun.security.ssl.allowLegacyHelloMessages", "true");
// clear created server instances (TCP/UDP)
SyslogServer.shutdown();
String syslogProtocol = "tcp";
System.out.println("Simple syslog server (RFC-5424)");
System.out.println("Usage:");
System.out.println(" java -jar simple-syslog-server.jar [protocol]");
System.out.println();
System.out.println("Possible protocols: udp, tcp, tls");
System.out.println();
SyslogServerConfigIF config = getSyslogConfig(syslogProtocol);
if (config == null) {
System.err.println("Unsupported Syslog protocol: " + syslogProtocol);
System.exit(1);
}
config.setUseStructuredData(true);
// config.setHost(InetAddress.getByName(null).getHostAddress());
config.setHost("0.0.0.0");
config.setPort(SYSLOG_PORT);
System.out.println("Starting Simple Syslog Server");
System.out.println("Protocol: " + syslogProtocol);
System.out.println("Bind address: " + config.getHost());
System.out.println("Port: " + config.getPort());
// start syslog server
SyslogServer.createThreadedInstance(syslogProtocol, config);
}
private static SyslogServerConfigIF getSyslogConfig(String syslogProtocol) {
SyslogServerConfigIF config = null;
if ("udp".equals(syslogProtocol)) {
config = new UDPSyslogServerConfig();
} else if ("tcp".equals(syslogProtocol)) {
config = new TCPSyslogServerConfig();
}
return config;
}
}

what is 'proxy.mycompany1.local'

I just started working in Java networking protocols. I am trying to connect to the internet using my proxy server. When I see the post at 'https://www.tutorialspoint.com/javaexamples/net_poxy.htm', they set the http.proxyHost property to 'proxy.mycompany1.local'. I know I can set this to my proxy server IP, but I am curious to know why my program still works, even though I set it to some random string like "abcd".
A. What does 'proxy.mycompany1.local" stand for?
B. How come my program works, even though I set the http.proxyHost" to "abcd"?
Following is my working program:
import java.net.HttpURLConnection;
import java.net.InetSocketAddress;
import java.net.Proxy;
import java.net.ProxySelector;
import java.net.URI;
import java.net.URL;
public class TestProxy {
public static void main(String s[]) throws Exception {
try {
System.setProperty("http.proxyHost", "abcd");
System.setProperty("http.proxyPort", "8080");
URL u = new URL("http://www.google.com");
HttpURLConnection con = (HttpURLConnection) u.openConnection();
System.out.println(con.getResponseCode() + " : " + con.getResponseMessage());
} catch (Exception e) {
e.printStackTrace();
System.out.println(false);
}
Proxy proxy = (Proxy) ProxySelector.getDefault().select(new URI("http://www.google.com")).iterator().next();
System.out.println("proxy Type : " + proxy.type());
InetSocketAddress addr = (InetSocketAddress) proxy.address();
if (addr == null) {
System.out.println("No Proxy");
} else {
System.out.println("proxy hostname : " + addr.getHostName());
System.out.println("proxy port : " + addr.getPort());
}
}
}
This is the output:
200 : OK
proxy Type : HTTP
proxy hostname : abcd
proxy port : 8080

First of all, according System Properties tutorial.
Warning: Changing system properties is potentially dangerous and
should be done with discretion. Many system properties are not reread
after start-up and are there for informational purposes. Changing some
properties may have unexpected side-effects.
And my experience say you can get unpleasant issues on your system when you change *.proxyHost properties. So I highly wouldn't recommend you to change system properties for this task.
Much better use something like:
//Proxy instance, proxy ip = 127.0.0.1 with port 8080
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("127.0.0.1", 8080));
conn = new URL(urlString).openConnection(proxy);
and authorisation on proxy:
Authenticator authenticator = new Authenticator() {
#Override
public PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication("user",
"mypassword".toCharArray());
}
};
Authenticator.setDefault(authenticator);
Now return to main questions:
A. 'proxy.mycompany1.local" is just example. You can use any hostname
B. Class URL uses java.net.PlainSocketImpl class via Socket. It tries to resolve proxy hostname 'abcd', swallow error and go to google directly. Just try to play with this code:
import java.net.*;
import java.io.*;
public class RequestURI {
public static void main(String[] args) {
int port = 8181;
long startTime = System.currentTimeMillis();
try {
// System.getProperties().setProperty("http.proxyHost", "abcd");
// System.getProperties().setProperty("http.proxyPort", Integer.toString(port));
URL url = new URL("http://google.com");
HttpURLConnection uc = (HttpURLConnection) url.openConnection();
int resp = uc.getResponseCode();
if (resp != 200) {
throw new RuntimeException("Failed: Fragment is being passed as part of the RequestURI");
}
} catch (IOException e) {
e.printStackTrace();
}
System.out.println("Run time in ms ="+ (System.currentTimeMillis() - startTime));
}
}
You can see run time is bigger when you uncomment section with setProperty. Unsuccessful attempt to resolve hostname increases execution time.

First of all, proxy.mycompany1.local is just a host name, it is a sample, it is nothing special.
I tried your code in a non proxied network, and it worked as you described. I guess that url.openConnection() method ignores proxy settings, because if you manage your own proxy and use url.openConnection(proxy), then it fails with a java.net.UnknownHostException.
Here you are with a piece of code that fails:
SocketAddress addr = new InetSocketAddress("abcd", 8080);
Proxy proxy = new Proxy(Proxy.Type.HTTP, addr);
URL url = new URL("http://google.com");
URLConnection conn = url.openConnection(proxy);
InputStream in = conn.getInputStream();
in.close();
You can read more about Java Networking and Proxies.

FTPSClient retrievefile() hanging

I'm creating an apache FTPS client (because the remote server won't allow plain FTP). I can connect and delete files without problem, but when using retrieveFile() or retrieveFileStream(), it hangs.
For some reason, very small files do transfer (up to 5792 bytes), but anything else gives the following PrintCommandListener output:
run:
220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
220-You are user number 2 of 50 allowed.
220-Local time is now 19:42. Server port: 21.
220-This is a private system - No anonymous login
220-IPv6 connections are also welcome on this server.
220 You will be disconnected after 15 minutes of inactivity.
AUTH TLS
234 AUTH TLS OK.
USER
331 User OK. Password required
PASS
230 OK. Current restricted directory is /
TYPE A
200 TYPE is now ASCII
EPSV
229 Extended Passive mode OK (|||53360|)
RETR test.txt
150-Accepted data connection
150 7.3 kbytes to download
Here is the code:
try {
FTPSClient ftpClient = new FTPSClient("tls",false);
ftpClient.addProtocolCommandListener(new PrintCommandListener(new PrintWriter(System.out)));
ftpClient.connect(host, port);
int reply = ftpClient.getReplyCode();
if (FTPReply.isPositiveCompletion(reply)) {
ftpClient.enterLocalPassiveMode();
ftpClient.login(username, password);
ftpClient.enterLocalPassiveMode();
FileOutputStream outputStream = new FileOutputStream(tempfile);
ftpClient.setFileType(FTPClient.ASCII_FILE_TYPE);
ftpClient.retrieveFile("test.txt", outputStream);
outputStream.close();
ftpClient.logout();
ftpClient.disconnect();
}
} catch (IOException ioe) {
System.out.println("FTP client received network error");
}
Any ideas are greatly appreciated.

Typically the FTP command sequence for FTPS connections goes (per RFC 4217) AUTH TLS, PBSZ 0, then USER, PASS, etc. Thus you might try:
FTPSClient ftpClient = new FTPSClient("tls",false);
ftpClient.addProtocolCommandListener(new PrintCommandListener(new PrintWriter(System.out)));
ftpClient.connect(host, port);
int reply = ftpClient.getReplyCode();
if (FTPReply.isPositiveCompletion(reply)) {
ftpClient.execPBSZ(0);
reply = ftpClient.getReplyCode();
// Check for PBSZ error responses...
ftpClient.execPROT("P");
reply = ftpClient.getReplyCode();
// Check for PROT error responses...
ftpClient.enterLocalPassiveMode();
This explictly tells the server to not buffer the data connection (PBSZ 0), and to use TLS for protecting the data transfer (PROT P).
The fact that you are able to transfer some bytes indicates that the issue is not the usual complication with firewalls/routers/NAT, which is another common FTPS issue.
Hope this helps!

Even if PBSZ 0 and PROT P are called in the correct sequence, sometimes the server does require SSL session reuse which is not the case by default for the client.
For example, the following reply comes when trying to list a directory. As a result no content listing is returned, this way the client seeing as if the directory is empty:
LIST /
150 Here comes the directory listing.
522 SSL connection failed; session reuse required: see require_ssl_reuse option in sftpd.conf man page
To overcome that, custom initialization of the FTPSClient is needed by overriding _prepareDataSocket_() method.
The solution is explained in details here: https://eng.wealthfront.com/2016/06/10/connecting-to-an-ftps-server-with-ssl-session-reuse-in-java-7-and-8/
Working code sample taken from the above link:
import java.io.IOException;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.net.Socket;
import java.util.Locale;
import javax.net.ssl.SSLSession;
import javax.net.ssl.SSLSessionContext;
import javax.net.ssl.SSLSocket;
import org.apache.commons.net.ftp.FTPSClient;
import com.google.common.base.Throwables;
public class SSLSessionReuseFTPSClient extends FTPSClient {
// adapted from: https://trac.cyberduck.io/changeset/10760
#Override
protected void _prepareDataSocket_(final Socket socket) throws IOException {
if(socket instanceof SSLSocket) {
final SSLSession session = ((SSLSocket) _socket_).getSession();
final SSLSessionContext context = session.getSessionContext();
try {
final Field sessionHostPortCache = context.getClass().getDeclaredField("sessionHostPortCache");
sessionHostPortCache.setAccessible(true);
final Object cache = sessionHostPortCache.get(context);
final Method putMethod = cache.getClass().getDeclaredMethod("put",Object.class, Object.class);
putMethod.setAccessible(true);
final Method getHostMethod = socket.getClass().getDeclaredMethod("getHost");
getHostMethod.setAccessible(true);
Object host = getHostMethod.invoke(socket);
final String key = String.format("%s:%s", host, String.valueOf(socket.getPort())).toLowerCase(Locale.ROOT);
putMethod.invoke(cache, key, session);
} catch(Exception e) {
throw Throwables.propagate(e);
}
}
}
}

Hope someone finds my comment useful after several years.
In my case, I replaced retrieveFile with retrieveFileStream. It requires more code, but at least it works.

For me, I fixed the problem after upgrading Apache Commons Net to 3.8.0.
dependencies {
implementation 'commons-net:commons-net:3.8.0'
...
}

MySQL jdbc + SSL

I set system properties for a SSL-enabled MySQL client, which worked fine:
System.setProperty("javax.net.ssl.trustStore","truststore");
System.setProperty("javax.net.ssl.trustStorePassword","12345");
String url = "jdbc:mysql://abc.com:3306/test?" +
"user=abc&password=123" +
"&useUnicode=true" +
"&characterEncoding=utf8&useSSL=true"
A couple days ago I found the client couldn't connect to another web site in which a commercially signed SSL certificate is installed. Obviously the overriding keystores didn't work with regular https connections.
Then I decided to build my version of SocketFactory based on StandardSocketFactory.java in MySQL Connector/J source.
I added a method to create Socket objects in public Socket connect(String hostname, int portNumber, Properties props) method.
private Socket createSSLSocket(InetAddress address, int port) {
Socket socket;
try {
InputStream trustStream = new FileInputStream(new File("truststore"));
KeyStore trustStore = KeyStore.getInstance("JKS");
// load the stream to your store
trustStore.load(trustStream, trustPassword);
// initialize a trust manager factory with the trusted store
TrustManagerFactory trustFactory = TrustManagerFactory.getInstance("PKIX", "SunJSSE"); trustFactory.init(trustStore);
// get the trust managers from the factory
TrustManager[] trustManagers = trustFactory.getTrustManagers();
// initialize an ssl context to use these managers and set as default
SSLContext sslContext = SSLContext.getInstance("SSL");
sslContext.init(null, trustManagers, null);
if(address == null) {
socket = sslContext.getSocketFactory().createSocket();
} else {
socket = sslContext.getSocketFactory().createSocket(address, port);
}
} catch (Exception e) {
System.out.println(e.getLocalizedMessage());
return null;
}
return socket;
}
The url passed to jdbc driver is changed to:
String url = "jdbc:mysql://abc.com:3306/test?" +
"user=abc&password=123" +
"&useUnicode=true" +
"&characterEncoding=utf8&useSSL=true" +
"&socketFactory=" + MySocketFactory.class.getName();
The client did execute my version createSSLSocket() and return a Socket object. However, I got the following Exceptions after continuing the execution:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
javax.net.ssl.SSLException:
Unrecognized SSL message, plaintext connection?
I'm sure the MySQL was up and running, the address and port passed to createSSLSocket() were correct. Could anyone help? The client has to communicate to 2 sites at the same time: an HTTPS web server and a self-signed MySQL server.