Hie , I m generating report in csv format using solr , angularjs , Jax-rs and java. The input stream contain a csv response already because we have specified wt=csv while querying solr. Size of csv created from every input Stream might be 300mb .At java layer code is some thing like :
enter code here
InputStream is1;
InputStream is2;
// for is1 let file is csv1
// for is2 let file is csv2
// csvzip is the csv created form both two files
// now csvzip need to be downloaded through a popup
Creating big size file and zipfile in memory will not be a good approach surely.
Is there is any way to handle this?
Related
This is my first hands on using Java Spring boot in a project, as I have mostly used C# and I have a requirement of reading a file from a blob URL path and appending some string data(like a key) to the same file in the stream before my API downloads the file.
Here are the ways that I have tried to do it:
FileOutputStream/InputStream: This throws a FileNotfoundException as it is not able to resolve the blob path.
URLConnection: This got me somewhere and I was able to download the file successfully but when I tried to write/append some value to the file before I download, I failed.
the code I have been doing.
//EXTERNAL_FILE_PATH is the azure storage path ending with for e.g. *.txt
URL urlPath = new URL(EXTERNAL_FILE_PATH);
URLConnection connection = urlPath.openConnection();
connection.setDoOutput(true); //I am doing this as I need to append some data and the docs mention to set this flag to true.
OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());
out.write("I have added this");
out.close();
//this is where the issues exists as the error throws saying it cannot read data as the output is set to true and it can only write and no read operation is allowed. So, I get a 405, Method not allowed...
inputStream = connection.getInputStream();
I am not sure if the framework allows me to modify some file in the URL path and read it simultaneously and download the same.
Please help me in understanding if they is a better way possible here.
From logical point of view you are not appending data to the file from URL. You need to create new file, write some data and after that append content from file from URL. Algorithm could look like below:
Create new File on the disk, maybe in TMP folder.
Write some data to the file.
Download file from the URL and append it to file on the disk.
Some good articles from which you can start:
Download a File From an URL in Java
How to download and save a file from Internet using Java?
How to append text to an existing file in Java
How to write data with FileOutputStream without losing old data?
I have successfully implemented Apk Expansion Files for my project.
Problem:In my .obb i have a folder which has 100 xml files in it.Now the problem is i am using the below code to read the data directly from .obb files without extracting the data.
this is code given in the offical doc here http://developer.android.com/google/play/expansion-files.html under the topic Reading from a ZIP file
ZipResourceFile expansionFile = APKExpansionSupport.getAPKExpansionZipFile(MyActivity.this, 1, 0);
String pathToFileInsideZip = "main.1.com.my.expansionfiles.obb/data/" +filename;
InputStream fileStream = expansionFile.getInputStream(pathToFileInsideZip);
i have a for loop in that i am calling writing this code so that it will read all the xml one by one and make the data ready for me to display.
The above will read the data directly from .obb file, but the problem is its talking lot of time to extract the data?
Why so? i am doing any mistake here?
I believed that the pathToFileInsideZip would not be
main.1.com.my.expansionfiles.obb/data/[files].
I think it's just
"data/[files]"
I'm trying to generate a PDF document from an uploaded ".docx" file using JODConverter.
The call to the method that generates the PDF is something like this :
File inputFile = new File("document.doc");
File outputFile = new File("document.pdf");
// connect to an OpenOffice.org instance running on port 8100
OpenOfficeConnection connection = new SocketOpenOfficeConnection(8100);
connection.connect();
// convert
DocumentConverter converter = new OpenOfficeDocumentConverter(connection);
converter.convert(inputFile, outputFile);
// close the connection
connection.disconnect();
I'm using apache commons FileUpload to handle uploading the docx file, from which I can get an InputStream object. I'm aware that Java.io.File is just an abstract reference to a file in the system.
I want to avoid the disk write (saving the InputStream to disk) and the disk read (reading the saved file in JODConverter).
Is there any way I can get a File object refering to an input stream? just any other way to avoid disk IO will also do!
EDIT: I don't care if this will end up using a lot of system memory. The application is going to be hosted on a LAN with very little to zero number of parallel users.
File-based conversions are faster than stream-based ones (provided by StreamOpenOfficeDocumentConverter) but they require the OpenOffice.org service to be running locally and have the correct permissions to the files.
Try the doc to avoid disk writting:
convert(java.io.InputStream inputStream, DocumentFormat inputFormat, java.io.OutputStream outputStream, DocumentFormat outputFormat)
There is no way to do it and make the code solid. For one, the .convert() method only takes two Files as arguments.
So, this would mean you'd have to extend File, which is possible in theory, but very fragile, as you are required to delve into the library code, which can change at any time and make your extended class non functional.
(well, there is a way to avoid disk writes if you use a RAM-backed filesystem and read/write from that filesystem, of course)
Chances are that commons fileupload has written the upload to the filesystem anyhow.
Check if your FileItem is an instance of DiskFileItem. If this is the case the write implementation of DiskFileItem willl try to move the file to the file object you pass. You are not causing any extra disk io then since the write already happened.
I have an application that requires a user to upload a zipfile containing xml report file among other files.
What I want to do is, to verify it is a zip, then open and check if there is an xml file, and verify some few nodes which are required in that xml.
I want to do this before I save this zipfile to a disk/filesystem, and withought creating a temporary file. I will only save the file if it passes the validation.
I am using Spring multipart CommonsMultipartFile to manage uploads.
The application is using Java, jsp, tomcat
Thanks.
See my comment on the OP about the wisdom of buffering the entire file in memory.
One quick first check for a valid zip file would be to check the first 4 bytes for the appropriate "magic" bytes. a zip file should start with the first 4 bytes {(byte)0x50, (byte)0x4b, (byte)0x03, (byte)0x04}. the only way to really check it, however, is to attempt to unzip it.
If you want to check whether a file is a ZIP file, perhaps you can use getContentType() method of the URLConnection class? Something like this:
URL u = new URL(fileUrl);
URLConnection uc = u.openConnection();
String type = uc.getContentType();
But it would be much faster to detect the magic bytes which, for the ZIP format, are 50 4B.
We are running tomcat, and we are generating pdf files on the fly. I do not have the file size before hand, so I cannot direcly link to a file on the server. So I directly send the output.
response.setContentType("application/force-download");
OutputStream o = response.getOutputStream();
And then I directly output to this OutputStream.
The only problem is that the receiver does not get the filesize, so they do not know how long the download will take. Is there a way to tell the response how large the file is?
EDIT
I do know the filesize, I just cant tell the STREAM how big the file is.
The response object should have a setContentLength method:
// Assumes response is a ServletResponse
response.setContentLength(sizeHere);
Serialize the PDF byte stream to a file or in a byte array calculate its size, set the size and write it to the output stream.
I beleive you're ansering the qustion your self:
quote:
I do not have the file size before hand, so I directly send the output.
If you don't have the size you cant send it....
Why not generate the PDF file in to temp file system , or ram-base file system or memory-map file on the fly. then you can get the file size.
response.setContentType("application/force-download");
response.setContentLength(sizeHere);
OutputStream o = response.getOutputStream();