We Keep Coding

Apache FTP Client running in Docker has problem with retrieving files contents from external server

I use Apache FTP Client to retrieve contents of files from FTP server. It runs on passive mode, retrieving with retrieveFileStream or retrieveFile and it works properly when I run it without Docker. There is a problem when I run this application from Docker container. It properly connects, log in, retrieving list of files. It has problem with retrieving content of file - it cannot retrieve full content of file, but only part of it. I thought that might be an buffer issue, I set values of all buffer settings to huge number and also tried to use buffer to read from stream (as was proposed on StackOverFlow when someone has problem with retrieving file), but without success. I also set the MTU settings of Docker to the value that my normal network uses, but also it did not help.
Here is some code:
FTPClient ftpClient = new FTPClient();
ftpClient.setControlKeepAliveTimeout(120);
ftpClient.setDataTimeout(120000);
ftpClient.setBufferSize(1024000);
ftpClient.setReceiveBufferSize(1024000);
ftpClient.setReceieveDataSocketBufferSize(1024000);
LOGGER.trace("Trying to connect to host {}.", connectionSettings.getHost());
ftpClient.connect(connectionSettings.getHost());
int replyCode = ftpClient.getReplyCode();
if (!FTPReply.isPositiveCompletion(replyCode)) {
ftpClient.disconnect();
LOGGER.error("Received negative reply code from host {}. Disconnected.", connectionSettings.getHost());
throw new FtpClientNotReadyException();
}
ftpClient.enterLocalPassiveMode();
LOGGER.trace("Received positive reply from host {}. Trying to log in.", connectionSettings.getHost());
try {
if (!ftpClient.login(connectionSettings.getUsername(), connectionSettings.getPassword())) {
LOGGER.warn("FTP Client login failed for user {}", connectionSettings.getUsername());
ftpClient.disconnect();
throw new FtpClientNotReadyException();
}
} catch (IOException e) {
ftpClient.disconnect();
throw e;
}
LOGGER.trace("Successfully logged in to host {} on account {}.", connectionSettings.getHost(),
connectionSettings.getUsername());
try {
if (Strings.isNotBlank(connectionSettings.getWorkingDirectory()) &&
!ftpClient.changeWorkingDirectory(connectionSettings.getWorkingDirectory())) {
LOGGER.warn("FTP Client could not change worked directory to {}.", connectionSettings.getWorkingDirectory());
throw new FtpClientNotReadyException();
}
} catch (IOException e) {
ftpClient.logout();
ftpClient.disconnect();
throw e;
}
InputStream retrievedFileContentInputStream = ftpClient.retrieveFileStream(fileName);
byte[] fileContentByteArray = retrievedFileContentInputStream.readAllBytes();
retrievedFileContentInputStream.close();
try {
checkFileSize(fileName, expectedFileSize, fileContentByteArray.length);
} catch (FileCorruptedException | FileBeingCurrentlyUploadedException e) {
throw e;
} finally {
if (!ftpClient.completePendingCommand()) {
LOGGER.trace("Complete pending command was not successful.");
}
LOGGER.trace("Completed pending command.");
}
These are solutions that I tried:
1) Buffer to read from InputStream:
InputStream ins = ftpClient.retrieveFileStream(fileName);
ByteArrayOutputStream swapStream = new ByteArrayOutputStream();
byte[] buff = new byte[2048];
int rc = 0;
int value =2048;
while ((rc = ins.read(buff, 0, value)) > 0) {
swapStream.write(buff, 0, rc);
}
byte[] fileByte = swapStream.toByteArray();
ftpClient.completePendingCommand();
2) Changing output stream to input stream
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
ftpClient.retrieveFile(fileName, outputStream);
outputStream.close();
byte[] fileContentByteArray = outputStream.toByteArray();
My typical log output when problem occurs:
Retrieved file size is lower than expected. Was: 228098. Expected: 236826.
Retrieved file size is lower than expected. Was: 1199. Expected: 1246.

Turns out the client on different platform than Windows received less bytes, because of different line endings (carriage return) that FTP server sent in ASCII MODE.

Speed up image download time

in my app I'm downloading images I get from an URL. The problem here is that it takes wayy too long. The images are compressed and only 9km big. I'm downloading 50 of them, so let's say 500kByte of images are being downloaded by the app. This should be pretty fast, right? Well, it isn't. I'm sometimes wating for 20 seconds until it's loaded. What is taking so long there? I have a pretty easy way to download the images. Here it is
int downloadingImages = getDownloadImageCount();
System.out.println(downloadingImages);
if(downloadingImages == 0){
for(int c = downloadingImages; c < 50; c++){
try{
bitmapArr[c] = getBitmapFromURL(imageURLArr[c]);
setDownloadImageCount(50);
publishProgress(c);
} catch (FileNotFoundException f1){
Log.v("FileNotFoundException", "Bitmap konnte nicht geladen werden");
} catch(NullPointerException e){
} catch (IllegalStateException ie){
}
setDownloadImageCount(50);
}
}
This is the getBitmapFromUrl function
public static Bitmap getBitmapFromURL(String src) throws FileNotFoundException {
try {
//Downloading the image
URL url = new URL(src);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoInput(true);
connection.connect();
try{
//Saving the image to a bitmap and return it
InputStream input = connection.getInputStream();
Bitmap myBitmap = BitmapFactory.decodeStream(input);
return myBitmap;
//Catch Exception if file not found
} catch(FileNotFoundException ffe){
Log.v("FileNotFoundException", "getBitmapFromURLMethod");
return null;
}
//Catch Exception if error at input and output
} catch (IOException e) {
e.printStackTrace();
return null;
}
}
I mean, what is taking so long there? How can I increase the speed. I will use cahing for images later, but still... I mean it's 500kbs and takes so long. Why is that?

Its completely natural that it takes a while. For each image you load from a URL a new HTTP-request is made (that is a request is sent to the server and it responds).
Thats 50 times the round-trip time over the network (not even including the time it may take the server to respond). You should cache the images locally once you have them retrieved.

The longest step in the process is probably sending a request to the server and waiting for it to respond, and you're repeating this step 50 times. Do you have the ability to modify the server? If so, consider adding an action that lets you download all 50 images at once (perhaps by putting them into a zip file, then unpacking them client-side).

Try to wrap your InputStream to BufferedInputStream
BufferedInputStream bs = new BufferedInputStream(input);
Bitmap myBitmap = BitmapFactory.decodeStream(input);

FTP Client issue

I'm having a tough time figuring something out. (I'm pretty new to all this.)
I wrote this java pgm to ftp a large file to a destination server.
Here's the code (codes been modified a bit for display):
public static void ftpUpload(String path, String upfileName, String dirName) throws Exception
{
FTPClient client = new FTPClient();
client.addProtocolCommandListener((ProtocolCommandListener) new PrintCommandListener(new PrintWriter(System.out)));
client.enterLocalPassiveMode();
FileInputStream fis = null;
int reply;
try {
client.connect(ftpserver);
client.login(ftpuserid, ftppasswd);
reply = client.getReplyCode();
if(FTPReply.isPositiveCompletion(reply)){
client.changeWorkingDirectory(ftpdirectoryName + "/" + dirName);
boolean mkDir = client.makeDirectory(getCurrentMMMYY().toLowerCase());
client.changeWorkingDirectory(getCurrentMMMYY().toLowerCase());
//Create an InputStream of the file to be uploaded
fis = new FileInputStream(path + upfileName);
//Store file to server
client.storeFile(upfileName, fis);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (fis != null) {
fis.close();
}
client.logout();
//client.disconnect();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Something weird is happening on files I'm sending...
One of my files on the origination server is 82575786 in size, and when I ftp this file it almost sends the entire file. It actually sends 82574867. (missing 919)
Another file on the origination server is 717885, and when I ftp this file it almost sends the entire file. It actually sends 717522. (missing 363)
I pulled the log to see if something crashed, but it didn't show anything wrong with the transfer. Here are the 2 log entries showing the transfer.
[08/09/11 20:21:13:618 EDT] 00000043 SystemOut O 221-You have transferred 717522 bytes in 1 files.
221-You have transferred 82574867 bytes in 1 files.
Anyone's help would greatly be appreciated.
Thanks
Dan.

Are you transferring in ASCII mode instead of binary? ASCII mode converts CR/LF to LF and vice-versa depending on server and client settings.
Are you using Apache's FTP client? It says the default is ASCII, you could try setting BINARY_FILE_TYPE with setFileType:
client.setFileType(FTPClient.BINARY_FILE_TYPE);

To upload a binary File you have to use the FTP.BINARY_FILE_TYPE but is not enough.
You are using only an INPUT stream, and you need to use an outputstream too
I hope that this example will help you:
FTPClient client = new FTPClient();
client.connect("192.168.30.20");
client.login("pwd", "pwd");
client.setFileType(FTP.BINARY_FILE_TYPE);
String path_base = "/myPath/";
InputStream fis = new FileInputStream("A.pdf");
OutputStream os = client.storeFileStream(path_base+ "B.pdf");
byte buf[] = new byte[8192];
int bytesRead = fis.read(buf);
while (bytesRead != -1) {
os.write(buf, 0, bytesRead);
bytesRead = fis.read(buf);}
fis.close();
os.close();
client.completePendingCommand();
client.logout();
client.disconnect();

Java/Android: Reading/writing a byte array over a socket

I have an Android application where I'm trying to send a picture to a server. I did this using Base64 encoding and it worked quite well, but it took too much memory (and time) to encode the picture before sending it.
I'm trying to strip the Android application down to where it just simply sends the byte array and doesn't fiddle around with any kind of encoding scheme so it'll save as much memory and CPU cycles as possible.
This is what I would like the Android code to look like:
public String sendPicture(byte[] picture, String address) {
try {
Socket clientSocket = new Socket(address, 8000);
OutputStream out = clientSocket.getOutputStream();
BufferedReader in = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
out.write(picture);
return in.readLine();
}
catch(IOException ioe) {
Log.v("test", ioe.getMessage());
}
return " ";
}
The server is written in Java. How do I write the server code so I can properly retrieve the exact same byte array? My goal is to save as many CPU cycles on the Android as possible.
So far, all the methods I've tried resulted in corrupt data or a thrown exception.
Any help will be appreciated.

Try something like this:
public byte[] getPicture(InputStream in) {
try {
ByteArrayOutputStream out = new ByteArrayOutputStream();
int data;
while ((data = in.read())>=0) {
out.write(data);
}
return out.toByteArray();
} catch(IOException ioe) {
//handle it
}
return new byte[]{};
}

Based on Robert's and Zaki's comment, here is the modified code that should perform better.
public byte[] getPicture(InputStream in) {
try {
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] data = new byte[1024];
int length = 0;
while ((length = in.read(data))!=-1) {
out.write(data,0,length);
}
return out.toByteArray();
} catch(IOException ioe) {
//handle it
}
return null;
}

If you want bi-directional communication, the server must know when you're ready - you should prepend a 4 byte length field to your sender side indicating the number of bytes to come.
On the server side you read the length and then stay listening until everything has arrived. Then you can reply your acknowledge string.
If it is enough to send only the picture, you can simply send the data and then close the connection. The server side is implemented as shown by #thejh.

getting bytes format in java of an image

i have a requirement, i want to take URL of image from user and then convert it to bytes format, and send it my server, i am using tomcat as web-app server and hibernate as ORM tool, i am already written the server side code for saving the incoming bytes into a table using BLOB, but my problem is that how can i convert the image into array of bytes, so that i can send the array to server to process further.
And adding to above, i can load the data, from table, and then can send the bytes back to client, but how to convert the bytes back to image.
Currently, i am using HTML at client side for web pages and Servlets for request and response.
Please help me.

If it's an image URL, then just read the URL straight into a byte array, like this:
public static byte[] getBytesFromURL(URL url) throws IOException {
InputStream in = null;
ByteArrayOutputStream out = null;
try {
in = url.openStream();
out = new ByteArrayOutputStream();
int len;
byte[] buf = new byte[1024 * 4];
while ((len = in.read(buf)) >= 0) {
out.write(buf, 0, len);
}
byte[] bytes = out.toByteArray();
return bytes;
} catch (IOException e) {
throw e;
} finally {
if (in != null) {
try {
in.close();
} catch (IOException e) {
}
}
if (out != null) {
try {
out.close();
} catch (IOException e) {
}
}
}
}

Check the ImageIO class in java. It has methods for both reading and writing images. For more information check here.

It seems that you dont need to do any special computation on your images. Just store them and send them back to a browser. If this is the case, you dont actually need to treat them as images, you can just handle them as java.io.File. Then you can store them as BLOB in your database.
To help you manage the upload you can use commons-fileupload. Or if you are using SpringMVC, have a look at the Multipart Resolver.