I often need to generate content dynamically from a servlet / restlet or whatever, and don't know the length ahead of time. If the client is a browser, the progress bar doesn't work properly, because I haven't set the Content-Length header. Is there any way of setting an estimated content length, so the progress bar works "more or less"?
No, the Content-Length value must be the exact content length:
When a Content-Length is given in a message where a message-body is allowed, its field value MUST exactly match the number of OCTETs in the message-body. HTTP/1.1 user agents MUST notify the user when an invalid length is received and detected.
So you cannot send just an estimated Content-Length value to get a progress bar.
While I wouldn't recommend doing it you can set the content-length to an estimate of what the content is, and you will get a progress bar. As long as you can guarantee that your estimate is equal to or greater than the actual content this will generally work. If your actual content is greater than the estimate than the content will be truncated to the specified content-length.
I tested this in Firefox, IE, and Chrome without issue. The HTTP spec states that users agents MUST notify the user if the specified length doesn't match the actual length, but I have not observed this behavior with any browser I've tested.
I investigated this as an option but abandoned it due to the potential unforeseen conflicts for playing outside the bounds of the specification.
Not clear what you're asking. You can always set the Content-Length header yourself, though it needs to actually match the amount of data you're sending. The standard way to handle dynamic data where you don't know anything about the length ahead of time is to buffer the output, find the actual length, set the header, then dump the output. Not really answering what you appear to be asking, but I think what you're asking is impossible.
The only way to do this (as decribed in an RFC) is not to set the Content-Length header, ie. the response header does not contain a Content-Length line. In this case the browser does not know how long the body is, so the server "tells" the browser that all of the body has been sent by closing the connection.
I'm not sure whether the Java container will automatically close the connection in this case or if you can do it yourself via some kind of Filter.
To answer your question: I do not think that it is possible to give the browser an estimate.
If the length of your content is not known beforehand you can use the “chunked” content encoding (as per HTTP version 1.1). This will not solve your progress bar problem, though—and there’s just no way to make it work unless you know how much content you are going to send.
Related
I have written a java swing app that is sending SOAP requests based on this code here
Overall it is working great, however I have just started testing it when parsing Chinese characters in the soap:BODY and this causes an error where I get a 400 response from the Web Server:
s.AddParameter("xml", "班");
Using wireshark i eventually tracked it down to the Content-Length value that was constructed being incorrect when parsing these Chinese characters (and i am assuming any other multibyte(?) character).
I have proven this by overriding the content length generation by simply changing the code to this:
out.println("Content-Length: " + String.valueOf(postData.length()+2));
Obviously this is not a solution as it only proved my very isolated test case of sending a single character, but i believe the issue is that the postData.length() is calculated first and then on posting the data my 班 character is then converted to \347\217\255, throwing the content-length out and causing the request to fail.
So I am asking for advice on how to resolve this issue?
Is it possible for me to encode the value first, obtain the content length and suppress the encoding on the post? I am unsure what is actually encoding it; the PrintWriter i am assuming?
Regards.
I have a java program that searches a site for updates. It checks and than sleeps for a period of time set by user.
During some testing I set the sleep time to 0 secs, forgot about it and it over-requested (over-spammed?, don't know the terminology) the site. I wouldn't have noticed this if I didn't go to this site and manually checked for something.
The error I get is 429 : too many requests. I've searched the web and found out that the site should send me when is it ok to try again (retry-after header). The problem is that I either don't get this information or can't find it.
I've viewed the source via chrome and didn't find the information. Then I used getErrorStrem and didn't find the information.
In another question on stackoverflow it is said that
if everything is set up properly, you will also have received a
"Retry-after" header along with the 429 response
My knowledge of web technologies is rather limited. The program I wrote looked for patterns that I noticed viewing the source of page. My java knowledge is better, but not great.
So how do I get retry-after info? About 20 hours passed and when I tried again I got the same error.
As specified in the rfc 6585:
The 429 status code indicates that the user has sent too many
requests in a given amount of time ("rate limiting").
The response representations SHOULD include details explaining the
condition, and MAY include a Retry-After header indicating how long
to wait before making a new request.
This specific header is only optional. So if you don't see it in the headers that's because it isn't there.
Including the condition in the response representation is only a recommandation (and so not mandatory)
There is no rules regarding this limit: it depends on the server configuration.
For exact meaning of verbs commonly used in RFCs : you can read this
Jersey 2.4.1 gives us the ability to enable fixed length streaming. This is very useful when uploading large files. The new client property for enabling this is: HTTP_URL_CONNECTOR_FIX_LENGTH_STREAMING.
By default, when doing uploads, the whole entity content is buffered by the connector before the bytes are sent to their destination. This means that the client will likely run out of memory when uploading large files. Enabling fixed length streaming solves this problem.
Unfortunately this property is not honored when the content-length header is not specified (or is set to 0) in the request. My question is why? What problem are the Jersey runtimes trying to prevent by putting this restriction? Is the content length information necessary to stream the data?
Thanks,
Habib
Whether fixed length streaming is actived or not, the client should set the header anyway. With fixed length you know the size without the need of buffering the content but that only makes sense if you actually set the header. The server doesn't care if the client buffered the content to determine the length or not.
In HTTP, [the Content-Length field] SHOULD be sent whenever the message's length can be determined prior to being transferred, unless this is prohibited by the rules in section 4.4.
RFC 2616, section 14.13 Content-Length
Without setting the length header, the client could start streaming indefinitely, without a buffer. I guess this it what Jersey tries to prevent, because then the server wouldn't know when the content ends (exept some cases listed in
RFC 2616, section 4.4 Message Length).
I forward upload requests I receive from clients to an another endpoint. I do not control the presence of the content length header in the requests I receive, and therefore may not always have a content length header to send to the end point.
That said, I can see that we need to protect against the malicious case you mention above, although I initially thought this would be the backend's responsibility.
Thanks for the clarification.
I'm using play 2.1.0 and want to implement file upload with several parameters, i.e. multipart/form-data form has some small fields and file itself.
If I upload the file without using annotation
#BodyParser.Of(value = BodyParser.MultipartFormData.class, maxLength = MAX_FILE_SIZE_B)
and checking file size like uploadedFile.length > MAX_SIZE I can access request body and it's not null all the time.
If I'm using the annotation, when maxSizeExceeded ctx.request().body().asMultipartFormData() is null even my small parameters go first in the request sent by browser. Is it correct behaviour, is any way to get small parameters even file is too large?
Is it true that the first way is bad, because large files actually will be uploaded on the server?
The behavior is expected because, the header will contain the file size, and if the payload/file size has exceeded the max_size limit the server will not receive the file and the connection will be closed. So, you can't access any form fields. Instead try to add those fields as a part of request headers, if that helps.
There is no documentation that explains this, but that is how it is handled in http layer. The following code might explain a bit, when the payload exceeds the limit it wraps the object with body = null.
To answer your question, yes the second approach is good and helps your server from accepting large files unnecessarily.
First off my Java is beyond rusty and I've never done JSPs or servlets, but I'm trying to help someone else solve a problem.
A form rendered by JavaScript is posting back to a JSP.
Some of the fields in this form are over 100KB in size.
However when the form field is being retrieved on the JSP side the value of the field is being truncated to 100KB.
Now I know that there is a similar problem in ASP Request.Form which can be gotten around by using Request.BinaryRead.
Is there an equivalent in Java?
Or alternatively is there a setting in Websphere/Apache/IBM HTTP Server that gets around the same problem?
Since the posted request must be kept in-memory by the servlet container to provide the functionality required by the ServletRequest API, most servlet containers have a configurable size limit to prevent DoS attacks, since otherwise a small number of bogus clients could provoke the server to run out of memory.
It's a little bit strange if WebSphere is silently truncating the request instead of failing properly, but if this is the cause of your problem, you may find the configuration options here in the WebSphere documentation.
We have resolved the issue.
Nothing to do with web server settings as it turned out and nothing was being truncated in the post.
The form field prior to posting was being split into 102399 bytes sized chunks by JavaScript and each chunk was added to the form field as a value so it was ending up with an array of values.
Request.Form() appears to automatically concatenate these values to reproduce the single giant string but Java getParameter() does not.
Using getParameterValues() and rebuilding the string from the returned values however did the trick.
You can use getInputStream (raw bytes) or getReader (decoded character data) to read data from the request. Note how this interacts with reading the parameters. If you don't want to use a servlet, have a look at using a Filter to wrap the request.
I would expect WebSphere to reject the request rather than arbitrarily truncate data. I suspect a bug elsewhere.