We Keep Coding

Prevent client timing out while a servlet generates a large download

I have a Java servlet that generates some arbitrary report file and returns it as a download to the user's browser. The file is written directly to the servlet's output stream, so if it is very large then it can successfully download in chunks. However, sometimes the resulting data is not large enough to get split into chunks, and either the client connection times out, or it runs successfully but the download doesn't appear in the browser's UI until it's 100% done.
This is what my servlet code looks like:
#Override
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
response.setContentType("application/pdf");
response.setHeader("Content-Disposition", "attachment; filename=\"" + report.getFileName(params) + "\"");
try(OutputStream outputStream = response.getOutputStream()) {
// Send the first response ASAP to suppress client timeouts
response.flushBuffer(); // This doesn't seem to change anything??
// This calls some arbitrary function that writes data directly into the given stream
generateFile(outputStream);
}
}
I ran a particularly bad test where the file generation took 110,826ms. Once it got to the end, the client had downloaded a 0 byte file - I assume this is the result of a timeout. I am expecting this specific result to be somewhere between 10 and 30 KB - smaller than the servlet's buffer. When I ran a different test, it generated a lot of data quickly (up to 80MB total), so the download appeared in my browser after the first chunk was filled up.
Is there a way to force a downloaded file to appear in the browser (and prevent a timeout from occurring) before any actual data has been generated? Am I on the right track with that flushBuffer() call?

Well, it looks like shrinking the size of my output buffer with response.setBufferSize(1000); allowed my stress test file to download successfully. I still don't know why response.flushBuffer() didn't seem to do anything, but at least as long as I generate data quickly enough to fill that buffer size before timing out, the download will complete.

Is it possible to write text into a web page after it has loaded?

Let me get straight to an example to explain further.
final var socket = new java.net.ServerSocket(1234);
for (;;)
{
try (final var client = socket.accept())
{
client.getOutputStream().write("HTTP/1.1 200 OK\r\n\r\n".concat(java.time.Instant.now().toString()).getBytes());
}
}
When I now open my browser of choice (Firefox cough) I'll receive the current time and date. The question now is how I can write to that socket at a later point in time.
hypothetical solution
Here's something I already tried, but doesn't work at all.
final var socket = new java.net.ServerSocket(1234);
for (;;)
{
try (final var client = socket.accept())
{
client.getOutputStream().write("HTTP/1.1 200 OK\r\n\r\n".concat(java.time.Instant.now().toString()).getBytes());
client.getOutputStream().flush();
Thread.sleep(1000L);
client.getOutputStream().write("And another paragraph.".getBytes());
}
}
The result is a web page loading for approximately a single second, printing out the following result (may vary due to different date and time on your end).
2019-01-19T18:19:15.607192500Z
And another paragraph.
Instead I would like the see something like that:
print out the current time and date.
wait a second without the content of the web page changing.
print out the next paragraph.
How would I go about implementing that?

Is it possible for the server to write text into a web page after it is loaded? Yes it definitely is, but these days I suspect it it is rarely done. I started web development in the 1990s and back then that was a pretty common technique. We used it to write live chat messages to browsers with no Javascript. These days Javascript is ubiquitous and powerful, so using client-side Javascript to update a page will be the best option in most cases.
That said, the technologies we used for writing server-side updates back then should still work now. I suspect the reason you don't see updates in your browser is because it doesn't know it should start displaying the page before everything is loaded. Using chunked transfer encoding, a 1990s technology still supported by modern browsers, should resolve that. It it allows the server to indicate when a 'chunk' of data is complete and browsers will generally process each chunk immediately rather than wait for all the chunks to arrive.
The easiest way to use chunked transfer encoding is to use an HTTP library like Apache HttpComponents, then wrap your output stream in the appropriate class:
final var socket = new java.net.ServerSocket(1234);
for (;;)
{
try (final var client = socket.accept())
{
var outputStream = new ChunkedOutputStream(client.getOutputStream());
outputStream.write("HTTP/1.1 200 OK\r\n\r\n".concat(java.time.Instant.now().toString()).getBytes());
outputStream.flush();
Thread.sleep(1000L);
outputStream.write("And another paragraph.".getBytes());
}
}

what is delay between HttpPost being sent to server and server responding

I'm an uploading a zipfile from a Java desktop application to an Httpserver (running Tomcat 7), Im using Apache httpClient 4.5.3 and I display a progress bar showing progress using this wrapper solution https://github.com/x2on/gradle-hockeyapp-plugin/blob/master/src/main/groovy/de/felixschulze/gradle/util/ProgressHttpEntityWrapper.groovy
So in my code Im updating progressbar every time the callback gets called
HttpEntity reqEntity = MultipartEntityBuilder.create()
.addPart("email", comment)
.addPart("bin", binaryFile)
.build();
ProgressHttpEntityWrapper.ProgressCallback progressCallback = new ProgressHttpEntityWrapper.ProgressCallback() {
#Override
public void progress(final float progress) {
SwingUtilities.invokeLater(
new Runnable()
{
public void run()
{
MainWindow.logger.severe("progress:"+progress);
Counters.getUploadSupport().set((int)progress);
SongKong.refreshProgress(CreateAndSendSupportFilesCounters.UPLOAD_SUPPORT_FILES);
}
}
);
}
};
httpPost.setEntity(new ProgressHttpEntityWrapper(reqEntity, progressCallback));
HttpResponse response = httpclient.execute(httpPost);
HttpEntity resEntity = response.getEntity();
MainWindow.logger.severe("HttpResponse:"+response.getStatusLine());
This reports files uploaded as a percentage, but there is a sizeable delay between it reporting 100% creation and actually receiving http status from server.
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.19408
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.40069
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.6073
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.81391
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.99768
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.99778
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.99789
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.999794
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:99.9999
07/07/2017 14.23.54:BST:CreateSupportFile$4$1:run:SEVERE: progress:100.0
07/07/2017 14.24.11:BST:CreateSupportFile:sendAsHttpPost:SEVERE: HttpResponse:HTTP/1.1 200 OK
07/07/2017 14.24.11:BST:CreateSupportFile:sendAsHttpPost:SEVERE: Unknown Request
Note is not due to my tomcat code doing much since I haven't yet implemented the tomcat code for this function so it just defaults to the "Unknown Request" code.
protected void doPost(javax.servlet.http.HttpServletRequest request,
javax.servlet.http.HttpServletResponse response)
throws javax.servlet.ServletException, java.io.IOException
{
String createMacUpdateLicense = request.getParameter(RequestParameter.CREATEMACUPDATELICENSE.getName());
if(createMacUpdateLicense!=null)
{
createMacUpdateLicense(response, createMacUpdateLicense);
}
else
{
response.setCharacterEncoding("UTF-8");
response.setContentType("text/plain; charset=UTF-8; charset=UTF-8");
response.getWriter().println("Unknown Request");
response.getWriter().close();
}
}
How can I more accurately report to the user when it will complete
Update
I have now fully implemented the serverside, this has increased the discrepancy
#Override
protected void doPost(javax.servlet.http.HttpServletRequest request, javax.servlet.http.HttpServletResponse response)
throws javax.servlet.ServletException, java.io.IOException
{
String uploadSupportFiles = request.getParameter(RequestParameter.UPLOADSUPPORTFILES.getName());
if(uploadSupportFiles!=null)
{
uploadSupportFiles(request, response, uploadSupportFiles);
}
else
{
response.setCharacterEncoding("UTF-8");
response.setContentType("text/plain; charset=UTF-8; charset=UTF-8");
response.getWriter().println("Unknown Request");
response.getWriter().close();
}
}
private void uploadSupportFiles(HttpServletRequest request, HttpServletResponse response, String email) throws IOException
{
Part filePart;
response.setCharacterEncoding("UTF-8");
response.setContentType("text/plain; charset=UTF-8; charset=UTF-8");
try
{
filePart = request.getPart("bin");
String fileName = getSubmittedFileName(filePart);
response.getWriter().println(email+":File:" + fileName);
//Okay now save the zip file somewhere and email notification
File uploads = new File("/home/jthink/songkongsupport");
File supportFile = new File(uploads, email+".zip");
int count =0;
while(supportFile.exists())
{
supportFile = new File(uploads, email+"("+count+").zip");
count++;
}
InputStream input;
input = filePart.getInputStream();
Files.copy(input, supportFile.toPath());
Email.sendAlert("SongKongSupportUploaded:" + supportFile.getName(), "SongKongSupportUploaded:" + supportFile.getName());
response.getWriter().close();
}
catch(ServletException se)
{
response.getWriter().println(email+":"+se.getMessage());
response.getWriter().close();
}
}

Assuming your server-side code just writes the uploaded file somewhere and responds something like "DONE" at the end, here is a rough timeline of what happens:
Bytes written to socket OutputStream
============================|
<--> Buffering |
Bytes sent by TCP stack |
============================
<------> Network latency|
Bytes received by Tomcat
============================
| (Tomcat waits for all data to finish uploading
| before handing it out as "parts" for your code)
| File written to local file on server
| =====
|
| Response "DONE" written by servlet to socket output
| ==
| <---> Network latency
| == Response "DONE" received by client
| |
| |
"100%" for entity wrapper ^ Actual 100% ^
Discrepancy
<----------------------->
"Twilight Zone" : part of discrepancy you cannot do much about.
(progress feedback impossible without using much lower level APIs)
<--------------------->
The scales are of course completely arbitrary, but it shows that there are several factors that can participate into the discrepancy.
Your server writes the file after receiving all bytes, but it does not make a big difference here.
So, the factors:
(client side) Buffering (possibly at several levels) between the Java I/O layer and the OS network stack
Network latency
(server-side) Buffering (possibly at several levels) between the OS network stack and the Java I/O layer
Time to write (or finish writing) zip file on disk
Time to print response (negligible)
Network latency
(client side) Time to read response (negligible)
So you could take that discrepancy into account and adjust the "upload complete" step to 90% of the total progress, and jump from 90 to 100 when you get the final response. From 0% to 90% the user would see "Uploading", with a nice progress bar moving, then you show "Processing...", perhaps with a throbber, and when done, jump to 100%.
That's what many other tools do. Even when I download a file with my browser, there is a small lag towards the end, the download seems stuck at "almost" 100% for a second (or more on my old computer) before the file is actually usable.
If the "twilight zone" time is much higher than the upload time as perceived by your progress wrapper, you might have a problem, and your question would thus be "where does this delay come from?" (for now I don't know). In that case, please provide complete timings (& make sure client & server machines have their clocks synchronized).
If you reaaaally need a more accurate/smooth progress report towards the end, you will need a much more involved setup. You will probably need to use more low-level APIs on the server side (e.g. not using #MultipartConfig etc), in order to have your server do something like writing to disk as data is received (which makes error handling much more difficult), print a dot to output and flush, for every 1% of the file that is written to disk (or any other kind of progress you want, provided it's actual progress on server-side). Your client side would then have the ability to read that response progressively, and get accurate progress report. You can avoid threading on client side, it's fine to do this sequentially:
POST data, report progress but scaled to 90% (ie if wrapper says 50%, you report 45%)
when done, start reading output from server, and report 91%, 95%, whatever, up until 100%.
Even with that I'm not sure it's possible to display progress info for all the steps (especially between 100% sent and first byte the server can possibly send), so maybe even that extremely complex setup would be useless (it could very well stall at 90% for a moment, then go 91/92/...99/100 in an instant).
So really at this point it's probably not worth it. If you really have a 17s step between last byte sent by client, and response received, something else is off. Initially I was assuming it was for humongous files, but since then you said your files were up to 50MB, so you might have something else to look at.

some of the server-side code might change depending on how the chunk data is represented, but the concept is roughly the same. Let's say you are uploading a 10MB file and you have your chunk size set to 1MB. You will send 10 requests to the server with 1MB of data each. The client is actually responsible for breaking all of this up. That is what you will do in Javascript. Then, each request is sent up via HttpRequest along with some other data about the file, chunk number and number of chunks. Again, I use the plupload plugin which handles this for me so some of the Request data may differ between implementations.
The method I am showing you is part of a Webservice which outputs JSON
data back to the client. Your javascript can then parse the JSON and
look for an error or success message and act appropriately. Depending
on your implementation, the data you send back might be different.
The javascript will ultimately handle the progress bar or percentage
or whatever, increasing it as it gets successful chunk uploads. My
implementation for my project lets plupload deal with all that, but
maybe that article I gave you will give you more control over the
client-side.
protected void Upload()
{
HttpPostedFile file = Request.Files[0];
String relativeFilePath = "uploads/";
try
{
if(file == null)
throw new Exception("Invalid Request.");
//plupload uses "chunk" to indicate which chunk number is being sent
int chunk = (int)Request.Form["chunk"];
//plupload uses "chunks" to indicate how many total chunks are being sent
int chunks = (int)Request.Form["chunks"];
//plupload uses "name" to indicate the original filename for the file being uploaded
String filename = Request.Form["name"];
relativeFilePath += filename;
//Create a File Stream to manage the uploaded chunk using the original filename
//Note that if chunk == 0, we are using FileMode.Create because it is the first chunk
//otherwise, we use FileMode.Append to add to the byte array that was previously saved
using (FileStream fs = new FileStream(Server.MapPath(relativeFilePath), chunk == 0 ? FileMode.Create : FileMode.Append))
{
//create the byte array based on the data uploaded and save it to the FileStream
var buffer = new byte[file.InputStream.Length];
file.InputStream.Read(buffer, 0, buffer.Length);
fs.Write(buffer, 0, buffer.Length);
}
if((chunks == 0) || ((chunks > 0)&&(chunk == (chunks - 1))))
{
//This is final cleanup. Either there is only 1 chunk because the file size
//is less than the chunk size or there are multiple chunks and this is the final one
//At this point the file is already saved and complete, but maybe the path is only
//temporary and you want to move it to a final location
//in my code I rename the file to a GUID so that there is never a duplicate file name
//but that is based on my application's needs
Response.Write("{\"success\":\"File Upload Complete.\"}");
}
else
Response.Write("{\"success\":\"Chunk "+chunk+" of "+chunks+" uploaded.\"}");
}
catch(Exception ex)
{
//write a JSON object to the page and HtmlEncode any quotation marks/HTML tags
Response.Write("{\"error\":\""+HttpContext.Current.Server.HtmlEncode(ex.Message)+"\"});
}
}

How to avoid EOFException for an empty HTTP response from HttpURLConnection?

I'm sending an HTTP request to a server which legitimately returns a blank response with HTTP response code = 200 (OK).
But this causes the marked line below to throw an EOFException:
InputStream responseStream = httpURLConnection.getInputStream();
final String contentEncoding = this.connection.getContentEncoding();
if ("gzip".equalsIgnoreCase(contentEncoding)) {
responseStream = new GZIPInputStream(responseStream); // throws EOFException
}
There may be an obvious way to prevent this but I can't find it. Should I perhaps be checking something like connection.getContentLength() > 0 or responseStream.available() > 0 before creating the GZIPInputStream? Neither of these seem quite right and I haven't come across anything similar in example code fragments...

Should I perhaps be checking something like connection.getContentLength() > 0
Yes.
or responseStream.available() > 0
Definitely not. available() == 0 isn't a valid test for EOF, and the Javadoc explicitly says so.

Shouldn't your server return HTTP code 204 (No Content), instead of 200? Except for a HEAD request that is. See Http Status Code definition
Apart from that, you could indeed just check to work around what looks like a not-all-too-compliant server implementation, or catch the exception and then determine the proper action depending on the type of request you sent.

What I understand from your question is that this exception is being thrown consistently everytime you get a 200 response from the server.
If responseStream.available() > 0 is not an option, it means that the stream actually contains something, but that stream seems to be incomplete or ending prematurely by any other means, since available states that there ARE readable bytes in the stream.
Update
Based on your comment (and after reading InputStream's doc again out of sheer curiosity) I also believe connection.getContentLength() > 0 is the way to go. If you're using Java 7, tough, connection.getContentLengthLong() is preferred over that because it returns a long directly.
Given what the docs say about InputStream#available :
The available method for class InputStream always returns 0.
And
Note that while some implementations of InputStream will return the total number of bytes in the stream, many will not. It is never correct to use the return value of this method to allocate a buffer intended to hold all data in this stream.
This discards that check as an option.

Unless I'm missing something, a response with Content-Encoding "gzip" by definition can not be empty.

HttpConnection truncating messages in Sony Ericsson W580

I've been having a problem while using HttpConnection with a Sony Ericsson W580.
The response to my http requests is application/octet-stream, and I'm sending a quite large array of bytes.
In this mobile phone however, it is consistently being cut down to 210 bytes...
I've tested the MIDP application in a large number of different mobile phones, using different mobile carriers and wi-fi, and no other mobile has shown this behavior.

Ok, I found the problem. Entirely my fault...
How I was reading the stream:
while(true){
int bytesRead = stream.read(tmpBuffer);
// if -1, EOF
if(bytesRead < 0)
break;
(...)
// WRONG LOGIC !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
// if we read the last chunk of data, and found EOF
if(bytesRead < tmpBufferArrayLength)
break;
// WRONG LOGIC !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
}
See the part between the "wrong logic !!!" comments? I was assuming that if read(tmpBuffer) could not fill the tmp buffer entirely, it was because EOF was being reached. The API does not guarantee this at all: it just states that EOF is signaled by a read(tmpBuffer) returning -1.
I didn't see this before because all the mobiles (and emulatores) I'd tested were being able to fill completely the buffer at every call.