I am trying to download a file from my Java application. But because UrlConnection uses HTTP 1.1 protocol i get a Tranfer Encoding: chunked response in which case i can not find out file size(content-length is not set). From what i could find HTTP version is hard coded in the class and there is no way to change it. Is it somehow possible to change the version back to one or tell the server not to use chunked encoding when sending a file?
Edit: I am not trying to retrive dynamic content my application is a download manager.
files i am downloading are static. Other downloaders i checked wget,igetter,curl use Http 1.0 and they get the size info from most servers. But my application and firefox issuing Http 1.1 always gets chunked encoding. I understand that content-length is not always present but i would like to get it most of time.
The Jakarta Commons HTTP Client contains a "preference architecture" that allows some fine grained control over the particulars of the HTTP connection. See http://hc.apache.org/httpclient-3.x/preference-api.html
It's very likely that the server can't specify a valid content-length, even if you specify HTTP/1.0. When content is dynamically produced, the server has to buffer it all to measure its total length. Not all servers are going to be able to fallback to this less efficient behavior.
If buffering the response is reasonable, why not do it in your client, where you have full control? This is safer than relying on the server.
Read the response without processing, just stuffing the data into a ByteArrayOutputStream. When you are done, measure the length of the resulting byte array. Then create a ByteArrayInputStream with it and process that stream in place of the stream you got from the URLConnection.
Related
When I say drop connection I mean actually closing the socket without reading any more bytes. I am trying to prevent a DoS attack where the attacker is trying to make a lot of HTTP requests that upload very very large files. For my purpose, I will consider any client trying to upload file larger than a pre-configured amount as attacker, and I would like to terminate the connection ASAP without reading even single more byte (I would like to send a HTTP 413 back to client if possible, but I don't care if the client receive the response or not, I just don't want any more byte from the client).
I have a app that runs within Jetty and from what I observed even if I throw exception, Jetty itself will still consume all the request body (and drop them) as part of the HTTP request life cycle.
So how do you guy do that? Through Jetty itself or through some kind of reverse proxy?
I know about maxFormContentSize and Apache File Upload, but they don't really do what I am looking for.
In a Servlet what happens If the developer messes up with the headers.
Let's say client's HTTP version is 1.0 and in
the request is not present a connection header,
inside my servlet I can do the following:
resp.setHeader("connection","keep-alive");
I also can set content-length to a specific value
and then send way less or more data
than the specified value
I'm sure exsist other situations that can cause similar "problems" as well.
How are these problems handled?
In general, it is unpredictable. In most cases the relevant specifications don't say what should happen. And even when they do, there is no guarantee that the respective server-side (or client-side) will do what is specified ... or advised by the spec.
Let's say client's HTTP version is 1.0 and in the request is not present a connection header, inside my servlet I can resp.setHeader("connection","keep-alive");
The client probably should just ignore the connection response.
I also can set content-length to a specific value and then send way less or more data than the specified value.
I recall seeing in the Servlet spec that the server-side framework is supposed to close the response automatically if the webapp writes more bytes than the content-length. But if the webapp writes less, there is nothing that the server-side can do. The client simply has to cope with this. The situation is indistinguishable from the case where the server crashes.
How are these problems handled?
It is generally implementation dependent.
This all handles on client side, and depends on browser, and vary from one to another.
But I suppose that described situation with connection header will be hadled very simple:
the browser with HTTP 1/.0 only is not supporting keep-alive connections, so will simply close connection afer each request.
Is this done at the code level, perhaps through JAX-WS handlers? Or is it done through some configuration at the app server?
I've read something about web compression in general, it appears that just as the message is about to head to the wire, compression is applied. Clients should be able to accept a GZIP MIME type for them to be able to decompress the message.
I'd like to find out who's supposed to apply that compression and how it's done.
It can be done either by code or by configuring the server to do it on the fly. How it's done with server configuration varies widely depending on the server. For Apache, the tool to use is mod_deflate. The instructions when using JBoss are here.
To do it in code, you need to:
compress the data with gzip
set the content-length header to the length in bytes of the compressed response
include the following header in the response:
Content-encoding: gzip
The request should include the header:
Accept-encoding: gzip
More info can be found in Wikipedia.
I'm doing a project on java download manager.i want to download a single file(which is in some website) with multiple connections(just like download Managers do,example-IDM).Is this possible in java ?.if yes please help me how can i implement that.if you people have any sample code then please post.Thank you in Advance..Have a Rocking Future.
Here are a couple of hints. No code though.
A multi-connection download manager relies on the support for the Accept-Ranges header in the HTTP 1.1 specification. Servers would use this header to indicate that they support sending of partial responses to the client.
HTTP clients use the Range header in the request to obtain partial responses. All partial responses will carry a Content-Range header.
A multi-connection download manager would make multiple connections to a server supporting this feature. Each connection would issue it's own range of headers to download. The responses would then be collated in the necessary order to obtain the desired file. The size of the ranges can be pre-calculated using an initial HTTP HEAD request, which returns the actual size of the file in the Content-Length response header; the task of downloading the file may now be split into suitable chunks.
I'd recommend reading about Segmented downloading, thinking of a way to implement it in Java and than asking concrete questions if you have any.
I am communicating to a Tomcat Server using a Java ME application on my mobile device.
I was wondering if I could compress my requests/responses using Gzip to reduce the number of bytes sent over the network.
Modern phones have so much CPU power and the network is relatively slow so compression makes perfect sense. It's quite easy to do also.
On the J2ME side, you do something like this (assuming you use HttpConnection),
hc.setRequestProperty("Accept-Encoding", "gzip, deflate");
if (hc.getResponseCode() == HttpConnection.HTTP_OK) {
InputStream in = hc.openInputStream();
if ("gzip".equals(hc.getEncoding()))
in = new GZIPInputStream(in);
...
We use GZIPInputStream from tinyline but I am sure there are others,
http://www.tinyline.com/utils/index.html
On the server side, it's all built-in. Just add following attributes to the Connector in server.xml on Tomcat,
<Connector
compression="on"
compressionMinSize="2048"
compressableMimeType="text/html,application/json"
... />
You can compress the content of an HTTP request or response, but not the headers. See section 3.6 of the HTTP 1.1 spec, and the later section that describes the Content-Encoding header.
EDIT: The flip side of this is that there is no guarantee that an HTTP server side will accept any particular compression format. And depending on the quality of the server-side implementation of HTTP, it might not even recognize that the request content has been compressed. So you don't want to do this unless you know that the server-side supports compressed request content.
Since you are using Tomcat, consider the possibility of putting an instance of Apache HTTP Server in front of the Tomcat server.
This can be done using the mod_jk module for Apache HTTP Server. Once you have done that, you can use mod_gzip/mod_deflate in Apache.
Of course, your client should have the ability to handle the compressed responses, for this to work. If you force your client to work with compressed responses, the client will end up displaying gibberish, since it would have been (usually) expecting plain text responses. You will find the definite indicator of the client's capability to handle compressed responses, in the client's Accept-Encoding headers.
This can be done programatically, using a servlet or a servlet filter that writes to a ZipOutputStream or GZipOutputStream, if you want to avoid the introduction of the Apache HTTP Server in the network. You will find some pointers on how to do this at the OReilly OnJava.com site.
On the server side, you can enable it as described here, but the mobile application will need a library that can decompress gzip, such as this one. Might be a bit of work to get it working though...