Users of my web application have an option to start a process that generates a CSV file (populated by some data from a database) and uploads it to an FTP server (and another department will read the file from there). I'm just trying to figure out how to best implement this. I use commons net ftp functionality. It offers two ways to upload data to the FTP server:
storeFile(String remote, InputStream local)
storeFileStream(String remote)
It can take a while to generate all the CSV data so I think keeping a connection open the whole time (storeFileStream) would not be the best way. That's why I want to generate a temporary file, populate it and only then transfer it.
What is the best way to generate a temporary file in a webapp? Is it safe and recommended to use File.createTempFile?
As long as you don't create thousands of CSV files concurrently the upload-time doesn't matter from my point of view. Databases usually output the data row by row and if this is already the format you need for the CSV file I strongly recommend not to use temporary files at all - just do the conversion on-the-fly:
Create an InputStream implementation that reads the database data row by row, converts it to CSV and publish the data via it's read() methods.
BTW: You mentioned that the conversion is done by a web application and that it can take a long time - this can be problematic as the default web client has a timeout. Therefore the long lasting process should be better done by a background thread only triggered by the webapp interface.
It is ok to use createTempFile, new File(tmpDir, UUID.randomUUID().toString()) can do as well. Just do not use deleteOnExit(), it is a leak master. Make sure you delete the file on your right own.
Edit: since you WILL have the data in memory, do not store it anywhere; wrap a java.io.ByteArrayInputSteam and use the method w/ the InputStream. Much neater and better solution.
Related
I am trying to find a way to read just end of huge and dynamic log file (like 20-30 lines from end) via SFTP from server and to save the point until where I read, and if I need more lines, to read more from this point upper.
Everything I've tried takes too long time, I've tried to copy this file on machine and after this to read from end using ReversedLinesFileReader because this method need the File object, when via SFTP you will get only InputStream, takes a lot to download file.
Also tried to count lines and to read from n line but also takes too long and throws exception because sometime in this time file is modified. Another way I tried to connect via SSH and used tail -100 and get the desired result, but just for one time, because next time I will get also new logs, but I need to go upper. Is there a fast way to get the end of file and to save the point and to read more upper of this point later? Any idea?
You don't say what SFTP library you're using, but the most widely used Java SSH/SFTP library is JSch, so I'll assume you're using that.
The SFTP protocol has operations to perform random-access I/O on remote files. Unfortunately, the JSch SFTP client doesn't expose the full range of operations. However, it does have versions of the get operation (for getting a file from the remote server) which permit skipping over the first part of the remote file. You can use one of these operations to read for example the last 10 KB of a file.
Several of the JSch get operations return an InputStream. You can read the contents of the remote file from the input stream. If you want to access the remote file line-by-line, you can convert it to Reader using InputStreamReader.
So, a process might do the following:
Call stat() on the remote file to get its size.
Figure out where in the file you want to start reading from. You could keep track of where you stopped reading last time, or you could guess based on the amount of data you're willing to download and the expected size in bytes of these last 20-30 lines.
Call get() to start reading it.
Process data read from the InputStream returned by the get() call.
Best would be to have a kind of rotating log files, possibly with compression.
Hower rsync is a unidirectional synchronisation, that can transmit only the changed parts of a file: for a log the new end.
I am not sure whether it works sufficiently performant in your case, and ssh is a prerequisite.
I need to export a Table into a CSV File and serve it to download via my JSF/Icefaces Webapplication.
How can I do that? I have a Table with 20+ Columns and over 10 MIO rows.
At the moment, I use a Java Thread, loading all data into ram. Then I create a new File and iterate the Collection writing row for row into the file. If the Thread is done, the user can download the large file via Servlet.
But I dont want to write so many GB into ram. I cant secure, not to get a memory problem..
Is it possible that hibernate does it for me? Or does somebody has an other idea?
Im connected to a DB2 Datebase. The table I want to export is connected to a hibernate bean but it is also possible to write native sql.
Thank you for response!
Do you need the intermediate stage of a file ? Have you tried loading from the database and writing to your servlet output stream for each row ? That way you're acting simply as a pipe between the client and the db.
Simply set your content-disposition header appropriately and that will signal the client's browser to treat the incoming data as a CSV file itself.
I have also gone through with similar kind of problem , the way I solve the problem is that I initially write a CSV file on disc and fetch 25K batch records from DB ans save to the file, and iteratively repeat the process until all the data required by the report is not written on the file.
And then send the file URL to the client to download the file.
I have a Java client/server desktop application, where the communication between client and server is based on Sockets, and the messages exchanged between client and server are serialized objects (message objects, that incapsulate requests and responses).
Now I need to make the client able to upload a file from the local computer to the server, but I can't send the File through the buffer, since the Buffer is already used for exchanging the message objects.
Should i open another stream to send the file, or is there any better way to upload a file for my situation?
I need to make the client able to upload a file from the local computer to the server
- Open a Solely Dedicated Connection to the Server for File uploading.
- Use File Transfer Protocol to ease your work, and moreover its quite easy and reliable to use the Apache's common lib for File uploading and downloading....
See this link:
http://commons.apache.org/net/
You really only have two options:
Open another connection dedicated to the file upload and send it through that.
Make a message object representing bits of a file being uploaded, and send the file in chunks via these message objects.
The former seems simpler & cleaner to me, requiring less overhead and less complicated code.
You can keep your solution and pass the file content as an object, for example as a String - use Base64 encoding (or similar) of the content if it contains troublesome characters
I have a java webapp which needs to upload files via http and then store them on the server. The files need to be associated with specific records in an Oracle database, so there will be a table in the database storing the reference to the associated record, and a reference to the file, along with other descriptive data such as title etc. The options for this table appear to be:
store the file as a BLOB
store a BFILE reference to the file
store a String containing the path to the file
We would prefer to store the file outside of the database, so we will not store it as a BLOB. The DBAs have indicated that their preferred option is to store the reference as a BFILE. The oracle.sql.BFILE object provides access to an InputStream for reading the file, but no obvious way of writing to it.
What is the best way of writing the file data back to disk when the only reference to the storage directory is the Oracle directory alias?
We decided that simple java.io was the best way to write to the file, which means that the storage directory has to be available to the web application servers as a mount. Since the directory was available to the webapp anyway, we then decided that the BFILE was not required and just to store the filename in the database instead.
According to the Oracle JDBC Developers Guide - 14.6 Working with BFILEs:
BFILEs are read-only. You cannot insert data or otherwise write to a BFILE.
You cannot use JDBC to create a new BFILE. They are created only externally.
So it seems that you need a separate tool/method to upload the data to the Oracle storage directory, which probably makes BFILE a bad choice for your scenario.
"We would prefer to store the file outside of the database"
Why ? What is your backup/recovery scenarios for this. For example, in the event of a disk failure, how would you recover the file ? Are the files purely transitory, so you don't actually need to preserve them ?
Basically, the BFILE is a compromise between BLOB storage and simply storing a path in the database. You write to it in the same manner as you would a conventional file, then make it 'available' to the database for reading. If your web-app is running on a different physical server from the database, you'd need to have a storage location that is accessible to both boxes (and appropriate permissions for read/write).
This problem pertains to Java
By using RandomAccessFile I intend to be able to also modify the file without blanking it.
The FTP protocol only barely supports random access reads and writes.
That is to say, an FTP client can use the REST command to start reading or writing from a particular offset, but it will always truncate the file from that point.