I'm using FTPClient for uploading files on ftp server. first, I have a TempFile object(no matters what is this object), that have a getBytes() method. this method returns List of bytes:
List<Byte> byteList = tempFile.getBytes();
Byte[] byteObjects= new Byte[byteList.size()];
byte[] byteArray = new byte[byteList.size()];
byteObjects = byteList.toArray(byteObjects);
byteArray = ArrayUtils.toPrimitive(byteObjects);
InputStream stream = new ByteArrayInputStream(byteArray);
ftpClient.storeFile(file.getName()+"."+file.getExtension(), stream);
After executing the last line of code, a file creates successfully with expected name, but when i want to open that file, i see "The image test.jpg can not be displayed, because it contains error" in the browser. What is the problem?
It sounds like you needs to set your ftp client to transmit as binary
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
As per the docs
If the current file type is ASCII, line separators in the file are transparently converted to the NETASCII format
Related
I need to download .png files from a FTP server in Java.
I have 3 different servers, each one contain a folder with exactly the same .png files.
On server 1 :
If I download my .png file of 4686 bytes stored on this server with FTPClient (apache.commons.net.ftp), I get a 4706 bytes .png file, and I can't open it.
If I download it with Total Commander, I get a 4686 bytes .png file, and I can open it.
On server 2 and 3 :
With FTPClient and Total Commander, I get in both case a 4686 bytes file and I can open it without problem.
My code :
FTPClient ftpClient = new FTPClient();
ftpClient.connect("...", PORT);
ftpClient.login("...", "...");
ftpClient.enterLocalPassiveMode();
FTPFile[] imageFiles = ftpClient.listFiles(distantPathForImages);
for (FTPFile imageFile : imageFiles) {
InputStream inputStream = ftpClient.retrieveFileStream(distantPathForImages + imageFile.getName());
OutputStream outputStream = new BufferedOutputStream(new FileOutputStream(new File(PATHDESTCSS + imageFile.getName())));
byte[] bytesArray = new byte[65536];
int bytesRead;
while ((bytesRead = inputStream.read(bytesArray)) != -1) {
outputStream.write(bytesArray, 0, bytesRead);
}
outputStream.close();
inputStream.close();
ftpClient.completePendingCommand();
}
Why does my file have these "extra bytes" only when I download it from the server 1, and how can I fix this?
FTPClient uses ascii mode by default.
You have to use binary mode to transfer binary files.
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
Your current code can by chance work on some servers even in the ascii mode, if the server is using Windows EOL sequence, hence no conversion takes place. And even then probably only, if the file by chance does not contain any lone #13.
One of your servers probably attempts to transmit the file as text, and your ftp client also thinks that it receives text.
Here is an excerpt from the javadoc:
If the current file type is ASCII, the returned InputStream
will convert line separators in the file to the local
representation.
If you are on windows, every line break will be replaced by 'linebreak + cr', wreaking havoc on all data structures in the png file.
The expected number of bytes for this scenario is: 4686 * (1 + 1 / 256) = 4704.3046875 , because on average, every 256-th byte in a png file should look like an ASCII line break, and will therefore result in an extra added byte. Your file ends up having 4706 bytes, which is pretty close.
Setting file type to FTP.BINARY_FILE_TYPE should fix this: https://commons.apache.org/proper/commons-net/apidocs/org/apache/commons/net/ftp/FTPClient.html#setFileType(int)
I have a jpeg, and on the end of it I wrote a zip file.
Inside this zip file is a single txt file called hidden.txt. I can change the extension to zip and read the file just fine on my laptop (debian) but when I try to read it using either a ZipInputStream or using ZipFile I get an error telling me it's not a zip file.
I tried separating the jpg part out first by reading the whole thing to a Bitmap then writing that to a byte[], however the byte[] encompassed more than just the image.
My method to combine the bitmap and the zipFile (a byte[])
private byte[] combineFiles(Bitmap drawn, byte[] zip) throws
IOException {
InputStream in;
ByteArrayOutputStream out = new ByteArrayOutputStream();
/*write the first file*/
byte[] img;
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
drawn.compress(Bitmap.CompressFormat.JPEG, 100, byteArrayOutputStream);
img = byteArrayOutputStream.toByteArray();
in = new ByteArrayInputStream(img);
IOUtils.copy(in, out);
in.close();
/*add the second (hidden) file*/
in = new ByteArrayInputStream(zip);
IOUtils.copy(in, out);
out.flush();
in.close();
return out.toByteArray();
}
So really I have two questions,
How do I separate the jpg and zip portions of the file?
How do I unzip hidden.txt (preferably into a byte[])
fairly certain I know this one, but what I am doing currently does not work, probably because I am doing #1 wrong
Ok, well here's how I would do this. Although it's very hacky.
The problem is that it's hard to tell the index of the boundary between the image data and the zip data. Assuming that you can write arbitrary data after the image data and still have a working image file, here is something you could try:
write out the image data.
write out a magical string like "BEGIN_ZIP"
write out the zip data.
Now, when you are trying to read things back in:
byte[] data = readAllTheBytes();
int index = searchFor("BEGIN_ZIP", data) + "BEGIN_ZIP".length();
// now you know that the zip data begins at index and goes to the end of the byte array
// so just use a regular zipinputstream to read in the zip data.
In JPEG file 0xFF, 0xD8 sequence of bytes indicates start of image and 0xFF, 0xD9 sequence of bytes indicates end of image JPEG Structure Wikipedia. So simply search for the latter sequence in file and you will be able to separate image and zip parts. Then use ZipInputStream to read (decompress) the data from zip file.
I am implementing a Direct Connect client. I am using the NMDC protocol. I can connect to a hub and other connected clients. I am trying to retrieve the file list from each client, I understand that in order to do that one must download the file files.xml.bz2 from the other client. The protocol to download a file is as follows:
-> $ADCGET file <filename> <params>|
<- $ADCSND file <fileName> <params>|
<- (*** binary data is now transfered from client B to client A ***)
I am trying to create a file named files.xml.bz2 using the binary data received. Here's my code:
//filesize is provided through the $ADCSND response from other client
byte[] data = new byte[filesize];
/*
Reading binary data from socket inputstream
*/
int read = 0;
for (int i=0; read<filesize;){
int available = in2.available();
int leftspace = filesize-read;
if (available>0){
in2.read(data, read, available>leftspace? leftspace:available);
++i;
}
read += (available>leftspace? leftspace:available)+1;
}
/*
writing the bytes to an actual file
*/
ByteArrayInputStream f = new ByteArrayInputStream(data);
FileOutputStream file = new FileOutputStream("files.xml.bz2");
file.write(data);
file.close();
The file is created, however, the contents (files.xml) are not readable. Opening it in firefox gives:
XML Parsing Error: not well-formed
Viewing the contents in the terminal only reads binary data. What am i doing wrong?
EDIT
I also tried Decompressing the file using the bz2 libray from Apache Ant.
ByteArrayInputStream f = new ByteArrayInputStream(data);
BZip2CompressorInputStream bzstream = new BZip2CompressorInputStream(f);
FileOutputStream xmlFile = new FileOutputStream("files.xml");
byte[] bytes = new byte[1024];
while((bzstream.read(bytes))!=-1){
xmlFile.write(bytes);
}
xmlFile.close();
bzstream.close();
I get an error, here's the stacktrace:
java.io.IOException: Stream is not in the BZip2 format
at org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.init(BZip2CompressorInputStream.java:240)
at org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.<init>(BZip2CompressorInputStream.java:132)
at org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.<init>(BZip2CompressorInputStream.java:109)
at control.Controller$1.run(Controller.java:196)
Usual, typical misuse of available(). All you need to copy a stream in Java is as follows:
while ((count = in.read(buffer)) >= 0)
{
out.write(buffer, 0, count);
}
Use this with any size buffer greater than zero, but preferably several kilobytes. You don't need a new buffer per iteration, and you don't need to know how much data is available to read without blocking, as you have to block, otherwise you're just smoking the CPU. But you do need to know how much data was actually read per iteration, and this is the first place where your code falls down.
The error java.io.IOException: Stream is not in the BZip2 format is generated by the constructor of class BZip2CompressorInputStream. I decided to scan the bytes, looking for the magic number to make sure that the file was bz2 format, it turns out that Java was right -- it wasnt in bz2 format.
Upon examining the source code of Jucy, I saw that the reason for this was a slight error in the command I sent to the other client, in essence, this error was caused a mistake in my protocol implementation. The solution was:
Replace:
$ADCGET file files.xml.bz2 0 -1 ZL1|
With:
$ADCGET file files.xml.bz2 0 -1|
ZL1 specifies compression of the files being sent (Not necessary).
I'm retrieving a file from a FTP Server. The file is encoded as UTF-8
ftpClient.connect(props.getFtpHost(), props.getFtpPort());
ftpClient.login(props.getUsername(), props.getPassword());
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
inputStream = ftpClient.retrieveFileStream(fileNameBuilder
.toString());
And then somewhere else I'm reading the input stream
bufferedReader = new BufferedReader(new InputStreamReader(
inputStream, "UTF-8"));
But the file is not getting read as UTF-8 Encoded!
I tried ftpClient.setAutodetectUTF8(true); but still doesn't work.
Any ideas?
EDIT:
For example a row in the original file is
...00248090041KENAN SARÐIN 00000000015.993FAC...
After downloading it through FTPClient, I parse it and load in a java object, one of the fields of the java object is name, which for this row is read as "KENAN SAR�IN"
I tried dumping to disk directly:
File file = new File("D:/testencoding/downloaded-file.txt");
FileOutputStream fop = new FileOutputStream(file);
ftpClient.retrieveFile(fileName, fop);
if (!file.exists()) {
file.createNewFile();
}
I compared the MD5 Checksums of the two files(FTP Server one and the and the one dumped to disk), and they're the same.
I would separate out the problems first: dump the file to disk, and compare it with the original. If it's the same as the original, the problem has nothing to do with UTF-8. The FTP code looks okay though, and if you're saying you want the raw binary data, I'd expect it not to mess with anything.
If the file is the same after transfer as before, then the problem has nothing to do with FTP. You say "the file is not getting read as UTF-8 Encoded" but it's not clear what you mean. How certain are you that it's UTF-8 text to start with? If you could edit your question with the binary data, how it's being read as text, and how you'd expect it to be read as text, that would really help.
Try to download the file content as bytes and not as characters using InputStream and OutputStream instead of InputStreamReader. This way you are sure that the file is not changed during transfer.
I'm trying to put a local file onto a remote host via XML-RPC Base64 encode/decode. This works perfectly fine for binary files, but when I try to send over the text file, all the line endings are removed. Why's this happening?
On the client side,
my $buf;
my $encoded = '';
while (read($FILE, $buf, 60 * 57)) {
$encoded .= encode_base64($buf);
}
To which it then sends over to my Redstone XML-RPC server, which takes it and writes it out:
// Create file
File file = new File(path);
file.createNewFile();
// Decode the encoded data sent over into bytes
byte[] bytes = Base64.decode(data.getBytes());
// Write them out to the file
FileOutputStream os = new FileOutputStream(file);
os.write(bytes);
os.flush();
os.close();
Try to set the $FILE in binary mode, you need to specify after the open command:
open my $FILE, '<', 'the_file_name.extension';
binmode $FILE;
# your code ...
The problem was that I was opening in Notepad, which wasn't recognizing CRLF endings.