I'm doing a project on java download manager.i want to download a single file(which is in some website) with multiple connections(just like download Managers do,example-IDM).Is this possible in java ?.if yes please help me how can i implement that.if you people have any sample code then please post.Thank you in Advance..Have a Rocking Future.
Here are a couple of hints. No code though.
A multi-connection download manager relies on the support for the Accept-Ranges header in the HTTP 1.1 specification. Servers would use this header to indicate that they support sending of partial responses to the client.
HTTP clients use the Range header in the request to obtain partial responses. All partial responses will carry a Content-Range header.
A multi-connection download manager would make multiple connections to a server supporting this feature. Each connection would issue it's own range of headers to download. The responses would then be collated in the necessary order to obtain the desired file. The size of the ranges can be pre-calculated using an initial HTTP HEAD request, which returns the actual size of the file in the Content-Length response header; the task of downloading the file may now be split into suitable chunks.
I'd recommend reading about Segmented downloading, thinking of a way to implement it in Java and than asking concrete questions if you have any.
Related
I am trying to upload a file (and additional fields) using single REST POST request. This request has to be processed asynchronous and form has to be submitted using multipart-form content type (because beside file I am sending few other text fields with certain metadata).
I tried with Apache's org.apache.http.impl.nio.client.CloseableHttpAsyncClient but until now I didn't succeed (when request arrives on the server, fields from the form are not presented). When I do the same using org.apache.commons.httpclient.HttpClient request is processed without problems, so the issue is not on the server side.
Does anybody know how this async 'multipart-form' request could be made? If not possible with CloseableHttpAsyncClient, do you have some other library to suggest?
Thanks in advance...
I just want to give an update. I did some investigation and was able to fulfill my requirements with AsyncHttpClient (AHC)
It works very fast and is based on Netty Framework and NIO Java API.
I'm trying to find a good example of how HTTP ranges should be handled on the server side. I would like to be able to support multiple ranges. Could somebody please show me an example, or provide some useful links? I've read the RFC, but I'm still a bit confused regarding the multipart stuff.
I serve up files from a Jetty webserver which presently get downloaded via regular HTTP GET.
However, I am interested in a P2P model where users can download files via the webseeding. How would this be implemented in the context of a Jetty server with libtorrent?
Second, I dont want to "seed" ALL files on the Jetty webserver forever, instead I only want to be able to seed files "on demand".
For example rather than blindly seeding a torrent, I would like to have the file available for demand IF a request comes in for it (via GET or webseeding or whatever) - upon which it can be "seeded".
I want to seed or upload on demand because I have a multitude of files and do not know if I will be able to seed tens of thousands of files concurrently. Btw would anyone know what the upper limit is for number of files which can be seeded concurrently?
The relevant documentation about the libtorrent part is here: http://www.rasterbar.com/products/libtorrent/manual.html#http-seeding and the specs are http://bittorrent.org/beps/bep_0019.html and http://bittorrent.org/beps/bep_0017.html (both being supported by libtorrent, as "url seeds" and "http seeds").
IIRC, BEP19 (webseeds, or urlseeds) is rather straight-forward from the server POV, and you don't need to do anything special there - you just serve the files as you would do for a normal HTTP requests for that file (so, the second part of your question doesn't quite make sense here).
With BEP17, you rather use a unique http endpoint, and pass it get parameters to specify what the client wants (which for example allows for better throttling control and range selection) (eg: http://example.com/seed/?info_hash=X&piece=Y&ranges=Z).
This second approach is more flexible if you intend to have more (programmatic) control over what is downloaded, but obviously requires a lot more code to write to handle the requests though.
Again, from the server POV, this is not that different from regular HTTP transactions, and there is nothing special about "seeding" here. You just serve files (each with its own url, either directly, or via a handler).
A for the metadata part, with BEP19, you add a "url-list" extension (with the full url of your file: http://example.com/seeds/SOMEFILE.txt - watch out for multi-file torrents), whereas BEP17 uses the key "httpseeds" (with your endpoint, eg: http://example.com/seed/).
Depending on whether your Jetty also handle metadata generation or not, you might prefer BEP19 over BEP17, for your urls to be more predictable / the metadata generation to be simpler...
Hope that helps.
I am trying to download a file from my Java application. But because UrlConnection uses HTTP 1.1 protocol i get a Tranfer Encoding: chunked response in which case i can not find out file size(content-length is not set). From what i could find HTTP version is hard coded in the class and there is no way to change it. Is it somehow possible to change the version back to one or tell the server not to use chunked encoding when sending a file?
Edit: I am not trying to retrive dynamic content my application is a download manager.
files i am downloading are static. Other downloaders i checked wget,igetter,curl use Http 1.0 and they get the size info from most servers. But my application and firefox issuing Http 1.1 always gets chunked encoding. I understand that content-length is not always present but i would like to get it most of time.
The Jakarta Commons HTTP Client contains a "preference architecture" that allows some fine grained control over the particulars of the HTTP connection. See http://hc.apache.org/httpclient-3.x/preference-api.html
It's very likely that the server can't specify a valid content-length, even if you specify HTTP/1.0. When content is dynamically produced, the server has to buffer it all to measure its total length. Not all servers are going to be able to fallback to this less efficient behavior.
If buffering the response is reasonable, why not do it in your client, where you have full control? This is safer than relying on the server.
Read the response without processing, just stuffing the data into a ByteArrayOutputStream. When you are done, measure the length of the resulting byte array. Then create a ByteArrayInputStream with it and process that stream in place of the stream you got from the URLConnection.
This problem relates to the Restlet framework and Java
When a client wants to discover the resources available on a server - they must send an HTTP request with OPTIONS as the request type. This is fine I guess for non human readable clients - i.e. in code rather than a browser.
The problem I see here is - browsers (human readable) using GET, will NOT be able to quickly discover the resources available to them and find out some extra help documentation etc - because they do not use OPTIONS as a request type.
Is there a way to make a browser send an OPTIONS/GET request so the server can fire back formatted XML to the client (as this is what happens in Restlet - i.e. the server response is to send all information back as XML), and display this in the browser?
Or have I got my thinking all wrong - i.e. the point of OPTIONS is that is meant to be used inside a client's code and not meant to be read via a browser.
Use the TunnelService (which by default is already enabled) and simply add the method=OPTIONS query parameter to your URL.
(The Restlet FAQ Q19 is a similar question.)
I think OPTIONS is not designed to be 'user-visible'.
How would you dispatch an OPTIONS request from the browser ? (note that the form element only allows GET and POST).
You could send it using XmlHttpRequest and then get back XML in your Javascript callback and render it appropriately. But I'm not convinced this is something that your user should really know about!