I'm developing, in Java, an application that has to download from a server to client some very large files. So far I'm using the apache commons-net:
FileOutputStream out = new FileOutputStream(file);
client.retrieveFile(filename, out);
The connection commonly fails before the client finishes downloading the file. I need a way to resume the download of the file from the point where the connection failed, without downloading the whole file again, is it possible?
Things to know:
FileOutputStream has an append parameter, from doc;
#param append if true, then bytes will be written
to the end of the file rather than the beginning
FileClient has setRestartOffset which takes offset as parameter, from doc;
#param offset The offset into the remote file at which to start the
next file transfer. This must be a value greater than or
equal to zero.
We need to combine these two;
boolean downloadFile(String remoteFilePath, String localFilePath) {
try {
File localFile = new File(localFilePath);
if (localFile.exists()) {
// If file exist set append=true, set ofset localFile size and resume
OutputStream fos = new FileOutputStream(localFile, true);
ftp.setRestartOffset(localFile.length());
ftp.retrieveFile(remoteFilePath, fos);
} else {
// Create file with directories if necessary(safer) and start download
localFile.getParentFile().mkdirs();
localFile.createNewFile();
val fos = new FileOutputStream(localFile);
ftp.retrieveFile(remoteFilePath, fos);
}
} catch (Exception ex) {
System.out.println("Could not download file " + ex.getMessage());
return false;
}
}
Commons-net FTPClient supports restarting transfers from a specific offset. You'll have to keep track of what you've successfully retrieved, send the correct offset, and manage appending to the existing file. Assuming, of course, that the FTP server you're connecting to supports the REST (restart) command.
Related
I want to send some data from the database. Currently our SFTP is set up like so (using the apache commons vsf2 library).
try(StandardFileSystemManager manager = new StandardFileSystemManager())
File file = generateFileFromDatabase();
manager.init();
FileSystemOptions opts = new FileSystemOptions();
SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(
opts, "no");
SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(opts, true);
SftpFileSystemConfigBuilder.getInstance().setTimeout(opts, 10000);
// Create localfile object
FileObject localFile = manager.resolveFile(file.getAbsolutePath());
// Create remote file object
FileObject remoteFile = manager.resolveFile(sftpUri, opts);
// Copy local file to sftp server
remoteFile.copyFrom(localFile, Selectors.SELECT_SELF);
} catch (Exception ex) {
ex.printStackTrace();
}
However, generateFileFromDatabase() works by invoking the new keyword, as in
return new File();
I don't want this, because this saves a new file in the file system every time generateFileFromDatabase() is called. Is there a way to generate a File() without saving it to the file directory?
Just use
OutputStream os = remoteFile.getContent().getOutputStream();
and then write In a loop to it, just like you would with a local file.
This might be a problem for slow reading/large transfers with long open connections. In that case you might need to pre-produce chunks and write them in multiple appends.
I am trying to send Files in fragments using DatagramPackets in Java (part of an assignemt.) When I am trying to save the incoming File I get access denied error, but I believe that it is not a permissions issue.
Here is the brunt of it:
I let the user sending the file to choose it using FileChooser. And create a new Message object.
//....
File f = content.showFileChooser();
byte type = Byte.parseByte("4");
Message m;
try {
if (mode == 1){
m = new Message(f, content.getServerData().getFragmentSize(), (short) (sentMessages.size()+1), type);
serverThread.send(m);
}
//...
During Message creation the file gets split up into byte arrays, where the size of each array is predetermined by the user. The code is quite lengthy so I am not going to post the chopping process, but this is how I convert the File object into a big byte[] which then gets chopped up
Path p = Paths.get(file.getAbsolutePath());
this.rawData = Files.readAllBytes(p);
After the Message is created and chopped up into byte arrays I send them using DatagramPackets. The other side then uses those to create a new Message object. Once all fragments arrive rawData is extracted from the Message object again. The problem believe lies here:
Message m = receivedMessages.get(msgIndex-1);
byte[] fileData = m.getFile();
if (fileData != null){
System.out.println("All file fragments received.");
content.append("Received a file in" + m.getFragmentCount()+" fragments. Choose directory. " ,1);
//I BELIEVE THIS TO BE THE CRITICAL POINT
String filePath = content.chooseDirectory();
if (filePath == null)
return;
FileOutputStream fos;
try {
fos = new FileOutputStream(filePath);
fos.write(fileData);
fos.close();
} catch (IOException e) {
e.printStackTrace();
}
}
Once all fragments arrive I let the user select a directory using FileChooser with DIRECTORY_ONLY choice mode. As I understand, FileOutputStream requires a full path for the new File. Do I have to send the file name and extension separately or can it be extracted from the received File data?
You are writing directory path to filePath, then try to open that directory with FileOutputStream. No wonder that doesn't work, you have to specify the filename too.
String filename = "myfile.txt"; //Or receive and set this some other way
File outFile = new File(filePath, filename);
fos = new FileOutputStream(outFile);
I don't see you sending/receiving the filename anywhere, though. You'll need to either make it constant or transfer it along with file contents.
I am using commons compress to zip multiple files and send it the client from a Servlet.
The files could be a combination of any type of files(text, video, audio, archives, images etc). I take the inputStream of file and write to ServletOutputStream using IOUtils.copy(is, os).
The code usually works fine for any document combination but when there is a request to download files that contain more than 1 zip, I get java.io.IOException: Closed
As a result, the zip file created is corrupted even though the size of zip is summation of individual filesizes(I am not using compression).
I tried to locally create zip and use FileOutputStream instead of response.getOutputStream() in the constructor of ZipArchiveOutputStream and it succeeds.
So, it looks like the problem exists for ServletOutputStream.
Can anyone suggest any workaround.
Here is my code :
`try (ZipArchiveOutputStream zos = new ZipArchiveOutputStream( response.getOutputStream())) {
//get fileList
for(File file : files) {
addFileToZip(zos, file.getName(), new BufferedInputStream(new FileInputStream(file)));
}
zos.close()
}
`
public static void addFileToZip(ZipArchiveOutputStream zipOutputStream, String filename, InputStream inputStream) throws FileNotFoundException {
if(zipOutputStream != null && inputStream != null) {
try {
zipOutputStream.putArchiveEntry(new ZipArchiveEntry(filename));
IOUtils.copy(inputStream, zipOutputStream);
logger.debug("fileAddedToZip :" + filename);
} catch (IOException e) {
logger.error("Error in adding file :" + filename, e);
} finally {
try {
inputStream.close();
zipOutputStream.closeArchiveEntry(); //**Starts to fail here after 1st zip is added**
} catch (IOException e) {
logger.error("Error in closing zip entry :" + filename, e);
}
}
}
`
Here is the exception trace :
`
java.io.IOException: Closed
at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:627)
at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:577)
at org.apache.commons.compress.archivers.zip.ZipArchiveOutputStream.writeOut(ZipArchiveOutputStream.java:1287)
at org.apache.commons.compress.archivers.zip.ZipArchiveOutputStream.writeOut(ZipArchiveOutputStream.java:1272)
at org.apache.commons.compress.archivers.zip.ZipArchiveOutputStream.writeDataDescriptor(ZipArchiveOutputStream.java:997)
at org.apache.commons.compress.archivers.zip.ZipArchiveOutputStream.closeArchiveEntry(ZipArchiveOutputStream.java:461)
at xxx.yyy.zzz.util.ZipUtils.addFileToZip(ZipUtils.java:110)
line 110 is zipOutputStream.closeArchiveEntry(); //**Starts to fail here after 1st zip is added**
Thanks in advance.
The problem is that you use try-with-resources which automatically closes the stream you create in it, and yet you also close it manually, and therefore when the JVM tries to auto-close it is when you get java.io.IOException: Closed exception because it is already closed.
If you use try-with-resources, you don't need to close the streams you create in it. Remove your manual zos.close() statement:
try (ZipArchiveOutputStream zos =
new ZipArchiveOutputStream(response.getOutputStream())) {
//get fileList
for(File file : files) {
addFileToZip(zos, attachment.getFileName(), is);
}
} // Here zos will be closed automatically!
Also note that once zos is closed, it will also close the servlet's underlying OutputStream so you will not be able to add further entries. You have to add all before it is closed.
I'm developing my first application in vaadin. Now I'm trying to customize upload component. In summary I have to do the upload of an image.
Now my component is implemented in a standard way:
public OutputStream receiveUpload(String filename,String mimeType) {
// Create upload stream
FileOutputStream fos = null; // Stream to write to
try {
// Open the file for writing.
file = new File("/tmp/uploads/" + filename);
fos = new FileOutputStream(file);
} catch (final java.io.FileNotFoundException e) {
new Notification("Could not open file<br/>",e.getMessage(),Notification.Type.ERROR_MESSAGE).show(Page.getCurrent());
return null;
}
return fos; // Return the output stream to write to
}
I want to ask you, if i can do the upload of the document without use a temp file on server.
How can I do?
Of course,
you just need to provide a OutputStream for the upload component.
This could be a a ByteArrayOutputStream for example, so you have everything as a large bytearray.
Just be aware, when the user uploads a 10 GByte size file, you will also need that much memory on the server for that request
André
I need to save a file and download file in directory outside server context.
I am using Apache Tomacat
I am able to do this in directory present in webapps directory of application
If my directory structure is as follows,
--src
--WebContent
-- uploaddir
-- myfile.txt
Then I am able to download in by simply.
download
But, problem is when file is in some other directory say d:\\uploadedfile\\myfile.txt
then I wont be able to download it, as resource is not in server context as above.
I have file path to uuid mapping,
like,
d:\\uploadedfiles\\myfile.txt <-> some_uuid
then I want file should be downloaded, on click of following,
download
So, How to make file downloadable when it is outside the server context,
I heard about getResourceAsStream() method which would do this , But would any one help me on how to do this, probably with simple code snippet?
Try the below code which you can write in filedownloadservet. Fetch the file name from the request parameter and then read and write the file.
If you need to do some security checks then do that before processing the request.
File file = new File("/home/files", "file name which user wants to download");
response.setContentType(getServletContext().getMimeType(file.getName()));
response.setContentLength(file.length());
BufferedInputStream inputStream = null;
BufferedOutputStream outputStream = null;
try {
inputStream = new BufferedInputStream(new FileInputStream(file));
outputStream = new BufferedOutputStream(response.getOutputStream());
byte[] buf = new byte[2048];
int len;
while ((len = inputStream.read(buf)) > 0) {
outputStream.write(buf, 0, len);
}
} finally {
if (outputStream != null) {
try {
outputStream.close();
} catch (IOException e) {
//log it
}
}
// do the same for input stream also
}
here i found the answer,
response.setContentType("application/msword");
response.setHeader("Content-Disposition","attachment;filename=downloadname.doc");
File file=new File("d:\\test.doc");
InputStream is=new FileInputStream(file);
int read=0;
byte[] bytes = new byte[BYTES_DOWNLOAD];
OutputStream os = response.getOutputStream();
while((read = is.read(bytes))!= -1){
os.write(bytes, 0, read);
}
os.flush();
os.close();
Base path will not work that is for HTML and it works if the base path is also exposed by your web server which does not look like case here.
To download an arbitary file you need to open the file using a FileInputStream (and surround it by a buffered input stream), read a byte, then send that byte from your servlet to the client.
Then there are security concerns, so should google that (basically not give access to any file but only file that is to be shared, audit download etc as needed.
Again in your servlet set the mime type etc and then open a input stream and write the bytes to the output stream to client