For some larger data we use HTTP streaming of objects in our Java-/Spring-Boot-Application. For that we need to bypass Spring MVC a bit like this:
#GetMapping("/report")
public void generateReport(HttpServletResponse response) throws TransformEntryException {
response.addHeader("Content-Disposition", "attachment; filename=report.json");
response.addHeader("Content-Type", "application/stream+json");
response.setCharacterEncoding("UTF-8");
OutputStream out = response.getOutputStream();
Long count = reportService.findReportData()
.map(entry -> transfromEntry(entry))
.map(entry -> om.writeValueAsBytes(entry))
.peek(entry -> out.write(entry))
.count();
LOGGER.info("Generated Report with {} entries.", count);
}
(...I know this code won't compile - just for illustration purposes...)
This works great so far - except if something goes wrong: let's say after streaming 12 entries successfully, the 13th entry will trigger an TransformEntryException during transfromEntry().
The stream will stop here. And the client gets indicated that his download finished successfully - while it was only part of the file.
We can log this server side and also attach some warning or even stacktrace to the downloaded file, but the client gets indicated that his download finished successfully - while it was only part of or even corrupt file.
I know that the HTTP status code gets sent with the header - which is already out. Is there any other way to indicate to the client a failed download?
("Client" most cases means some Webbrowser)
I'm an uploading a zipfile from a Java desktop application to an Httpserver (running Tomcat 7), Im using Apache httpClient 4.5.3 and I display a progress bar showing progress using this wrapper solution https://github.com/x2on/gradle-hockeyapp-plugin/blob/master/src/main/groovy/de/felixschulze/gradle/util/ProgressHttpEntityWrapper.groovy
So in my code Im updating progressbar every time the callback gets called
HttpEntity reqEntity = MultipartEntityBuilder.create()
.addPart("email", comment)
.addPart("bin", binaryFile)
.build();
ProgressHttpEntityWrapper.ProgressCallback progressCallback = new ProgressHttpEntityWrapper.ProgressCallback() {
#Override
public void progress(final float progress) {
SwingUtilities.invokeLater(
new Runnable()
{
public void run()
{
MainWindow.logger.severe("progress:"+progress);
Counters.getUploadSupport().set((int)progress);
SongKong.refreshProgress(CreateAndSendSupportFilesCounters.UPLOAD_SUPPORT_FILES);
}
}
);
}
};
httpPost.setEntity(new ProgressHttpEntityWrapper(reqEntity, progressCallback));
HttpResponse response = httpclient.execute(httpPost);
HttpEntity resEntity = response.getEntity();
MainWindow.logger.severe("HttpResponse:"+response.getStatusLine());
This reports files uploaded as a percentage, but there is a sizeable delay between it reporting 100% creation and actually receiving http status from server.
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.19408
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.40069
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.6073
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.81391
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.99768
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.99778
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.99789
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.999794
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.9999
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:100.0
07/07/2017 14.24.11:BST:CreateSupportFile:sendAsHttpPost:SEVERE: HttpResponse:HTTP/1.1 200 OK
07/07/2017 14.24.11:BST:CreateSupportFile:sendAsHttpPost:SEVERE: Unknown Request
Note is not due to my tomcat code doing much since I haven't yet implemented the tomcat code for this function so it just defaults to the "Unknown Request" code.
protected void doPost(javax.servlet.http.HttpServletRequest request,
javax.servlet.http.HttpServletResponse response)
throws javax.servlet.ServletException, java.io.IOException
{
String createMacUpdateLicense = request.getParameter(RequestParameter.CREATEMACUPDATELICENSE.getName());
if(createMacUpdateLicense!=null)
{
createMacUpdateLicense(response, createMacUpdateLicense);
}
else
{
response.setCharacterEncoding("UTF-8");
response.setContentType("text/plain; charset=UTF-8; charset=UTF-8");
response.getWriter().println("Unknown Request");
response.getWriter().close();
}
}
How can I more accurately report to the user when it will complete
Update
I have now fully implemented the serverside, this has increased the discrepancy
#Override
protected void doPost(javax.servlet.http.HttpServletRequest request, javax.servlet.http.HttpServletResponse response)
throws javax.servlet.ServletException, java.io.IOException
{
String uploadSupportFiles = request.getParameter(RequestParameter.UPLOADSUPPORTFILES.getName());
if(uploadSupportFiles!=null)
{
uploadSupportFiles(request, response, uploadSupportFiles);
}
else
{
response.setCharacterEncoding("UTF-8");
response.setContentType("text/plain; charset=UTF-8; charset=UTF-8");
response.getWriter().println("Unknown Request");
response.getWriter().close();
}
}
private void uploadSupportFiles(HttpServletRequest request, HttpServletResponse response, String email) throws IOException
{
Part filePart;
response.setCharacterEncoding("UTF-8");
response.setContentType("text/plain; charset=UTF-8; charset=UTF-8");
try
{
filePart = request.getPart("bin");
String fileName = getSubmittedFileName(filePart);
response.getWriter().println(email+":File:" + fileName);
//Okay now save the zip file somewhere and email notification
File uploads = new File("/home/jthink/songkongsupport");
File supportFile = new File(uploads, email+".zip");
int count =0;
while(supportFile.exists())
{
supportFile = new File(uploads, email+"("+count+").zip");
count++;
}
InputStream input;
input = filePart.getInputStream();
Files.copy(input, supportFile.toPath());
Email.sendAlert("SongKongSupportUploaded:" + supportFile.getName(), "SongKongSupportUploaded:" + supportFile.getName());
response.getWriter().close();
}
catch(ServletException se)
{
response.getWriter().println(email+":"+se.getMessage());
response.getWriter().close();
}
}
Assuming your server-side code just writes the uploaded file somewhere and responds something like "DONE" at the end, here is a rough timeline of what happens:
Bytes written to socket OutputStream
============================|
<--> Buffering |
Bytes sent by TCP stack |
============================
<------> Network latency|
Bytes received by Tomcat
============================
| (Tomcat waits for all data to finish uploading
| before handing it out as "parts" for your code)
| File written to local file on server
| =====
|
| Response "DONE" written by servlet to socket output
| ==
| <---> Network latency
| == Response "DONE" received by client
| |
| |
"100%" for entity wrapper ^ Actual 100% ^
Discrepancy
<----------------------->
"Twilight Zone" : part of discrepancy you cannot do much about.
(progress feedback impossible without using much lower level APIs)
<--------------------->
The scales are of course completely arbitrary, but it shows that there are several factors that can participate into the discrepancy.
Your server writes the file after receiving all bytes, but it does not make a big difference here.
So, the factors:
(client side) Buffering (possibly at several levels) between the Java I/O layer and the OS network stack
Network latency
(server-side) Buffering (possibly at several levels) between the OS network stack and the Java I/O layer
Time to write (or finish writing) zip file on disk
Time to print response (negligible)
Network latency
(client side) Time to read response (negligible)
So you could take that discrepancy into account and adjust the "upload complete" step to 90% of the total progress, and jump from 90 to 100 when you get the final response. From 0% to 90% the user would see "Uploading", with a nice progress bar moving, then you show "Processing...", perhaps with a throbber, and when done, jump to 100%.
That's what many other tools do. Even when I download a file with my browser, there is a small lag towards the end, the download seems stuck at "almost" 100% for a second (or more on my old computer) before the file is actually usable.
If the "twilight zone" time is much higher than the upload time as perceived by your progress wrapper, you might have a problem, and your question would thus be "where does this delay come from?" (for now I don't know). In that case, please provide complete timings (& make sure client & server machines have their clocks synchronized).
If you reaaaally need a more accurate/smooth progress report towards the end, you will need a much more involved setup. You will probably need to use more low-level APIs on the server side (e.g. not using #MultipartConfig etc), in order to have your server do something like writing to disk as data is received (which makes error handling much more difficult), print a dot to output and flush, for every 1% of the file that is written to disk (or any other kind of progress you want, provided it's actual progress on server-side). Your client side would then have the ability to read that response progressively, and get accurate progress report. You can avoid threading on client side, it's fine to do this sequentially:
POST data, report progress but scaled to 90% (ie if wrapper says 50%, you report 45%)
when done, start reading output from server, and report 91%, 95%, whatever, up until 100%.
Even with that I'm not sure it's possible to display progress info for all the steps (especially between 100% sent and first byte the server can possibly send), so maybe even that extremely complex setup would be useless (it could very well stall at 90% for a moment, then go 91/92/...99/100 in an instant).
So really at this point it's probably not worth it. If you really have a 17s step between last byte sent by client, and response received, something else is off. Initially I was assuming it was for humongous files, but since then you said your files were up to 50MB, so you might have something else to look at.
some of the server-side code might change depending on how the chunk data is represented, but the concept is roughly the same. Let's say you are uploading a 10MB file and you have your chunk size set to 1MB. You will send 10 requests to the server with 1MB of data each. The client is actually responsible for breaking all of this up. That is what you will do in Javascript. Then, each request is sent up via HttpRequest along with some other data about the file, chunk number and number of chunks. Again, I use the plupload plugin which handles this for me so some of the Request data may differ between implementations.
The method I am showing you is part of a Webservice which outputs JSON
data back to the client. Your javascript can then parse the JSON and
look for an error or success message and act appropriately. Depending
on your implementation, the data you send back might be different.
The javascript will ultimately handle the progress bar or percentage
or whatever, increasing it as it gets successful chunk uploads. My
implementation for my project lets plupload deal with all that, but
maybe that article I gave you will give you more control over the
client-side.
protected void Upload()
{
HttpPostedFile file = Request.Files[0];
String relativeFilePath = "uploads/";
try
{
if(file == null)
throw new Exception("Invalid Request.");
//plupload uses "chunk" to indicate which chunk number is being sent
int chunk = (int)Request.Form["chunk"];
//plupload uses "chunks" to indicate how many total chunks are being sent
int chunks = (int)Request.Form["chunks"];
//plupload uses "name" to indicate the original filename for the file being uploaded
String filename = Request.Form["name"];
relativeFilePath += filename;
//Create a File Stream to manage the uploaded chunk using the original filename
//Note that if chunk == 0, we are using FileMode.Create because it is the first chunk
//otherwise, we use FileMode.Append to add to the byte array that was previously saved
using (FileStream fs = new FileStream(Server.MapPath(relativeFilePath), chunk == 0 ? FileMode.Create : FileMode.Append))
{
//create the byte array based on the data uploaded and save it to the FileStream
var buffer = new byte[file.InputStream.Length];
file.InputStream.Read(buffer, 0, buffer.Length);
fs.Write(buffer, 0, buffer.Length);
}
if((chunks == 0) || ((chunks > 0)&&(chunk == (chunks - 1))))
{
//This is final cleanup. Either there is only 1 chunk because the file size
//is less than the chunk size or there are multiple chunks and this is the final one
//At this point the file is already saved and complete, but maybe the path is only
//temporary and you want to move it to a final location
//in my code I rename the file to a GUID so that there is never a duplicate file name
//but that is based on my application's needs
Response.Write("{\"success\":\"File Upload Complete.\"}");
}
else
Response.Write("{\"success\":\"Chunk "+chunk+" of "+chunks+" uploaded.\"}");
}
catch(Exception ex)
{
//write a JSON object to the page and HtmlEncode any quotation marks/HTML tags
Response.Write("{\"error\":\""+HttpContext.Current.Server.HtmlEncode(ex.Message)+"\"});
}
}
Im trying to wrap my head around Java Out/Inputstreams, closing and flushing. I have a situation where I want to create a file using Apache POI with data from a server. I would like the file to start downloading as soon as I retrieve the first record from the DB(Show the file at the bottom of the browser has started to download).
public void createExcelFile(final HttpServletResponse response,
final Long vendorId) {
try {
// setup responses types...
final XSSFWorkbook xssfWorkbook = new XSSFWorkbook();
final XSSFSheet sheet = xssfWorkbook.createSheet("sheets1");
// create file with data
writeExcelOutputData(sheet, xssfWorkbook);
xssfWorkbook.write(response.getOutputStream());
xssfWorkbook.close();
}
catch (final Exception e) {
LOGGER.error("Boom");
}
The above code will perform a file download no problem, but this could be a big file. I go off getting the data(around 20/30s) and then after that the download begins < no good...
Can I achive what I need or whats the best approach, thanks in advance
Cheers :)
Reasons could be as following:
maybe there is a read/write timeout with your http server, then if the process gets lengthy or becasue of low-bandwidth, so the connection will be closed by the server.
make sure the process(the excel work) gets completely done, maybe there would be an error/exception during work.
The solution of Jorge looks very promising. User need once request for a file, then server would do the work in background and then either user check the work process and download the file if ready, or server informs the user by email, web notification, etc...
Also you would keep the file in the server in a temp file for a while, and if the connection gets interrupted, server would respond the generated file partial(omit the bytes sent, like normal file download)
Keeping a connection alive to do a lengthy work is not very logical.
Again, if the file gets ready fast(really fast) for download/stream, and the download interrupts, if could be becasue of read/write timeout by server, or a very bad network.
In Servlet 3.1 it is possible to upload multiple files to servlet in one request (like a bulk upload), but what I want to achieve is to upload a big number of files to servlet as soon as each one is created without creating a new request on every file. In other words, there are 1000 files created on runtime, and I do not want 1st one to wait for the creation of the last to be uploaded. I want each file uploaded immediately after creation, so that servlet can check the upload timestamp for each file (and send back the result). The hard part for me is, I want neither to wait for creating all of them, nor to use separate requests for each file.
Is that possible? If yes, please give me some tips and directions/sources.
Yes, it is possible read multiple files in a single servlet request.
Here is how you could read the files in the servlet:
#WebServlet(urlPatterns = { "/UploadServlet" })
#MultipartConfig(location = "/uploads")
public class UploadServlet extends HttpServlet {
#Override
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
PrintWriter out = response.getWriter();
for (Part part : request.getParts()) {
String fileName = part.getSubmittedFileName();
out.println("... writing " + fileName");
part.write(fileName);
out.println("... uploaded to: /uploads/" + fileName);
}
}
}
Update:
Based on your updated question, you have ~1000 files that you want to "stream" to the server as they are created. It's not possible to add more parts to an existing post request, and I think it would be a bit excessive to try and upload all 1000 files in a single POST request anyway.
I recommend setting some arbitrary chunk size (say 20 files at a time for example) and batch them up and make a POST requests for every 20 files. This way, you are not waiting for all 1000 files to be created, and you are not doing everything over a single POST request. This will allow you to "stream" your files to the server in 20 file increments.
I know the title is a bit confusing. I will describe my task:
Now I'm writing a webpage, which allow me to download some data or do some other stuff. The front end is jsp and the back end is java. So every time when I do something on this page(e.g. sort the data or download the data), a request will be sent to java and after data processing the .jsp page will be refreshed and loaded again(of cause after sorting it should be refreshed). But I just found that if I download a data with the following code, the refreshing will be interrupted:
OutputStream out = response.getOutputStream(); // response is HttpServletResponse
doSomeOutput(out); // Just write something into OutputStream
out.flush(); // **PAY ATTENTION**. The refreshing is interrupted here!
out.close();
When I was debugging, the page was keeping loading until the third line. And then the page was not refreshed and the data was download.
I hope you can understand what I'm talking about:(
So now I have 2 questions: 1. How can I let the process keep working after out.flush();? 2. In my task I need the same behavior(the process is interrupted) in some other modules, but there's nothing to be download. So how can I interrupt the process besides out.flush();?
Thank you guys!!
1)
https://docs.oracle.com/javaee/5/api/javax/servlet/ServletResponse.html#getOutputStream%28%29
"Calling flush() on the ServletOutputStream commits the response"
as i understand this: active refreshing is stopped, as we got main part of data. But you would be able to send data even after commit. So you can keep working after flush.
Set headers are also sent with flush.
2) Pls explain more thoroughly. call "return", or close?