Internet Explorer has a file download limit of 4GB (2 GB on IE6). Firefox does not have this problem (haven't tested safari yet)
(More info here: http://support.microsoft.com/kb/298618)
I am working on a site that will allow the user to download very large files (up to and exceeding 100GB)
What is the best way to do this without using FTP. The end user must be able to download the file from there browser using HTTP. I don't think Flash or Silverlight can save files to the client so as far as I know they won't cut it.
I'm guessing we will need an ActiveX or Java applet to pull this off. Something like the download manager that MSDN uses.
Does anyone know of a commercial (or free) component that will do that? We do not want the user to have to install a "browser wide" download manager (like GetRight), we want it to only work with downloading on our site.
Update: Here is some additional info to help clarify what I'm trying to do. Most of the files above the 4GB limit would be large HD video files (its for a video editing company). These will be downloaded by users across the internet, this isn't going to be people on a local network. We want the files to be available via HTTP (some users are going to be behind firewalls that aren't going to allow FTP, Bittorrent, etc.). The will be a library of files the end user could download, so we aren't talking about a one time large download. The will be download different large files on a semi-regular basis.
So far Vault that #Edmund-Tay suggested is the closest solution. The only problem is that it doesn't work for files larger than 4GB (it instantly fails before starting the download, they are probably using a 32bit integer somewhere which is exceeded/overflown by the content length of the file).
The best solution would be a java applet or ActiveX component, since the problem only exist in IE, that would work like the article #spoulson linked to. However, so far I haven't had any luck finding a solution that does anything like that (multipart downloads, resume, etc.).
It looks like we might have to write our own. Another option would be to write a .Net application (maybe ClickOnce) that is associated with an extension or mime type. Then the user would actually be downloading a small file from the web server that opens in the exe/ClickOnce app that tells the application what file to download. That is how the MSDN downloader works. The end user would then only have to download/install an EXE once. That would be better than downloading an exe every time they wanted to download a large file.
#levand:
My actual preference, as a user, in these situations is to download a lightweight .exe file that downloads the file for you.
That's a dealbreaker for many, many sites. Users either are or should be extremely reluctant to download .exe files from websites and run them willy-nilly. Even if they're not always that cautious, incautious behaviour is not something we should encourage as responsible developers.
If you're working on something along the lines of a company intranet, a .exe is potentially an okay solution, but for the public web? No way.
#TonyB:
What is the best way to do this without using FTP.
I'm sorry, but I have to ask why the requirement. Your question reads to me along the lines of "what's the best way to cook a steak without any meat or heat source?" FTP was designed for this sort of thing.
bittorrent?
There have been a few web-based versions already (bitlet, w3btorrent), and Azureus was built using java, so it's definitely possible.
Edit: #TonyB is it limited to port 80?
Please don't use ActiveX... I am so sick of sites that are only viewable in IE.
My actual preference, as a user, in these situations is to download a lightweight .exe file that downloads the file for you.
Can you split the files into pieces and then rejoin them after the download?
If you don't want to write java code in-house, there are commercial applet solutions available:
Vault
MyDownloder
Both of them have eval versions that you can download and test.
A few ideas:
Blizzard use a light-weight .exe BitTorrent wrapper for their patches. I'm not entirely sure how it is done, but it looks like a branded version of the official BitTorrent client.
Upload to Amazon S3, provide the torrent link of the file (all S3 files are automatically BitTorrent-enabled), plus the full HTTP download link as alternative. See S3 documentation
What about saying "We recommend that you install Free Download Manager to download this file. You will have the added benefit of being able to resume the file and accelerate the download."
Personally I never download anything using the built in browser download tool unless I have to (e.g. Gmail attachments)
#travis
Unfortunately It has to be over HTTP inside the users browser.
I'll update the question to be more clear about that.
#levand
The problem only exist in IE (it works in Firefox) so while ActiveX would only work on IE, IE is the only one we need the work around for.
#travis - interesting idea. Not sure if it will work for what I need but I'll keep it in mind. I'm hoping to find something to integrate with the existing site instead of having to go out to a third party. It would also require me to setup a bittorrent tracker which wouldn't be as easy as it sounds for this application because different users will have different access to different files.
#jjnguy
I'm looking for a java applet or ActiveX component that will do that for me. These are non-technical users so we really just want to have them click download and the full file ends up in the specified location
#ceejayoz
I totally agree but its part of the requirement for our client. There will be FTP access but each user will have the option of downloading via HTTP or FTP. There are some users that will be behind corporate firewalls that don't permit FTP
I have seen other sites do this in the past (MSDN, Adobe) so I was hoping there is something out there already instead of having to make one in house (and learning java and/or ActiveX)
I say click-once installed download manager, similar to msdn.
But becoming a CDN without a more optimized protocol for the job is no easy task. I can't imagine a business model that can be worthwhile enough to have such large file downloads as a core competency unless you are doing something like msdn. If you create a thick client, you at least get the chance to get some more face time with the users, for advertising or some other revenue model, since you will probably be paying in the hundreds of thousands of dollars to host such a service.
The problem with the applet approach mentioned is that unless you have the end user modify their java security properties your applet will not have permission to save to the hard drive.
It may be possible using Java Web Start (aka JNLP). I think that if it is a signed app it can get the extra permission to write to the hard drive. This is not too different from the download an exe approach. The problem with this is that the user has to have the correct version of Java installed and has to have the Java Web Start capability setup correctly.
I would recommend the exe approach as it will be the easiest for the non-technical users to use.
There are some users that will be behind corporate firewalls that don't permit FTP...
Are users with restrictive firewalls like that likely to be permitted to install and run a .exe file from your website?
Take a look at cURL. This article describes how to do a multi-part simultaneous download via HTTP. I've used cURL in the past to manage FTP downloads of files over 300GB.
Another tip: You can boost download times even more if you increase the size of the TCP Window on the client's NIC configuration. Set it as high as the OS allows and you should see up to 2x improvement depending on your physical network. This worked for me on Windows 2000 and 2003 when FTPing over a WAN. The down side is it may increase overhead for all other network traffic that wants only a few KB for a network packet, but is now forced to send/recv in 64KB packets. Your mileage may vary.
Edit: What exactly is this you're trying to accomplish? Who is the audience? I'm assumed for a bit that you're looking to do this over your own network; but you seem to imply the client side is someone on the internet. I think we need clearer requirements.
Create a folder of files to be downloaded on the server where the document service is running (either using Linux commands or using java to execute shell commands)
Write the file to be downloaded to this folder (using Linux command or Java shell command is OK). Considering the efficiency of program execution, WGet command is used here
Package the downloaded folder as a zip file (using shell command), configure nginx agent, return the access file path of nginx to the front end, and then download from the front end.
Related
On our company we have the solution inside web application where customer on web application can pre-define paths to files on disk (i.e: c:/invoices/summary.txt).
Afterwards by Java Applet we are reading files under defined paths to web application and modify the files on user disk.
Yes - here the problem appears - the Java Applets. The Chrome does not support it anymore and it is a legacy / unsecured solution. We are looking the replacement for this solution which allows us to modify the files on user disk and will support different Browsers / OS.
We considered the following solutions:
dedicated external apps for OS which will be upload the defined files to our system (like Dropbox) but the problem is that we have to develop application for various OS and support it - a lot of work
WebDav API and solution where customer download the script from web app and run it, script upload the files to server - the problem is that the script could be error prone
HTML 5 solutions - we considered it but we want to achieve the situation that user defines the path only once
Does anyone see other sensible solutions for this case worth to try?
You may consider using Microsoft Silverlight in place of Java applets. It has read-write access to local filesystem (only trusted ones). You can create silverlight app in Visual Studio which can run both as local application and as a web applet.
A simple code to write file in C# (Silverlight 4) would be
public void WriteFile(object sender, RoutedEventArgs event) {
string path = System.IO.Path.Combine(Environment.GetFolerPath(Environment.SpecialFolder.MyDocuments), "TextFile.txt");
/// or
string other_path = "C:/invoices/summary.txt";
StreamWriter writer;
if (!File.Exists(path))
writer = File.CreateText(path);
writer.Write("Some text to write");
writer.Close();
}
You can read more about creating trusted applications with silverlight.
I will try to answer my own question. Finally as a team we decided to use the WebDAV HTTP protocol extension. We wanted to avoid develop separate application for various systems (cost) and we needed solution which could be fast replaced in place of Java Applets.
After first test on QA I have to say that WebDAV solution works ok. Maybe the scripts for the client is not a best elegant solution but it works without bigger problems (after the client run the script he/she could observe the status on browser).
I could also suggest that anyone who will need separate application could consider the Dropbox API which allow to write/read files within Dropbox directory. The problem of maintenance separate application will be solved by Dropbox team :) Unfortunately we cannot do that due to our business domain but it also look as interesting way to deal with the problem.
I hope that answer will be helpful for someone in the future ;)
Some relevant background:
My application is a Java app compiled into a .exe via JSmooth. The anticipated user base would likely be a few hundred users, but could grow well beyond that, as it's a community specific application.
How it works:
2 .jar files, one that preforms initial checks, another with the meat of the application.
Ideally, the init jar displays the splash, checks the version in desktop.txt against server.txt, if they differ, it prompts the user to update.
What I need to figure out:
1) What is a cheap, scalable hosting service that I could use as the file host for updates?
2) How can I create an "updater" to actually preform the jar replacement? My current solution is simply writing an updater in Java, but I was hoping for something like the installers people are more familiar with.
All of the research I've done has resulted in lackluster results, as 99% of hosting searches result in site hosting results. I just need an update repository with reasonable security. i.e., decent DDoS resistance and not left wide open to the Internet.
Edit: formatting
Easy to do and very foolish cheap with Amazon S3 or Joyent Manta as both support time-limited signed URLs and headers (which can contain a SHA-1 of the file) to check to see if the update is needed before downloading
On startup your app would check the update URL to see if it has changed. If it has changed, download the JARs. Do this before the app loads classes from those JARs. Updating the updater itself will be trickier so consider that an update might need a new update URL to prevent expiry.
I would like to know if it's possible to edit a file from within the browser. I know where the file is and I know its format. I would like to some HTML5 and JavaScript but have little luck so don't mind using Flash or Java.
The file I would like to edit is the hosts file. C:\WINDOWS\system32\drivers\etc\hosts on Windows XP.
This is mostly for test run of a new breed of AdBlocking so would like to give my browser and the best chance of working. I have Google Chrome Dev opened as ROOT, running on Windows XP.
Java applet with the correct permissions can access the file probably but how easy is it for the user to setup those permissions is a question I can not answer.
Flash is not going to work 100% as it can not be loaded from the internet and access the hard drive at the same time, without going into details you can trust me on that one as the security model simply does not allow it and there is no actual API inside flash to read the file, one can only forward it through to a server(only upload, no download), nothing more (if running from the internet sandbox).
Javascript - not sure, i kind of feel like it can't for similar reasons as Flash but somebody should probably confirm.
The add-blockers usually work as addons to the browser so you should probably do the same as you should have access but then again, you probably won't need it in that case...
In any case, Flash definitely is not going to work.
Java applets can as long as they are signed ..
Follow these intructions to sign an applet.
P.S these instructions are for a linux setup. You might need to set up environment variables in windows.
Jar signing link
so al long as the users allow the applet to run the applet the applet can modify anything on the file sysem and even open connections to other computers.. and most importantly the applet can be hidden in a corner if that is what you wish
Editing local files from a browser, especially a system file, is something that all browser vendors actively try to PREVENT. This is because all the virus makers are actively trying to edit these too!
If you want an ad blocker editing the hosts file to do it is a terrible idea for a number of reasons. First of all it is only going to work on Windows and your ad blocking won't help for Mac and Linux users. Second it will require permissions escalation on Vista and Windows 7. Third it would have to be reliant on something like ActiveX or Java. Fourth the hosts file would eventually get large. Large hosts file slow down internet browsing because you have to do a full lookup every time.
If you want to block ads, block them at the browser level. I hope I have convinced you that modifying the hosts file for this is a bad idea.
I'm looking for a technology that can be used to upload large (~2GB) files from both Windows and Mac OSX machines. It should be possible to pause and resume uploads on request/error. The final artefact should be easily installed by a user with basic computer skills.
I'm thinking of using Adobe AIR as it's quite ubiquitous, so installing should be easy although I'm not sure that AIR would provide the power that we're after. Another option would be a Java app, which I believe would provide more choice in terms which upload protocols (HTTP/WebDAV/etc).
Does AIR have the power to handle such large file sizes and could it pause and resume? Would Java be better, as I could chunk the files more easily? Is there another solution that I've not thought of? Any input would be greatly appreciated!
Please don't suggest using a service like YouSendIt, DropBox, etc. I need a custom, branded solution that can be integrated with a CMS for the uploaded files.
Thanks!
Darren.
You can use the Apache FileUpload library which I've tested uploading about 4gigs to, and it didn't have any problems.
If you want more control I would advise an applet - more control and you still have the advantage of not having to install on the clients computer.
I've been thinking a lot lately about how cool it would be to have a website that allows a user to insert a disc into their drive and click a button to begin a ripping process. After the rip is complete, it would upload to a server and be hosted there. Ideally, the process would be seamless for both DVDs and CDs and work cross platform.
My question is for the developers here - what recommendations do you have on approaching this situation? Does it seem feasible? Is it something that would require a flex app, java? Can the user get away with doing this in their browser only, without having to download any software? I'd also like to make sure that if the upload disconnects during the uploading process, the user can log back in and resume the upload where it failed.
I found some software that kind of achieves what I'm looking to do, but it is downloaded and is windows only as it relies on some .dll files to complete the process. I can show you this software if it would help?
For the record, the DVDs and CDs uploaded would need to be legally owned and the uploader must have the copyright, so don't think I'm getting into any infringement or illegality issues.
Thanks for the insight!
Browsers are not the best way to upload ISOs of CD/DVD. Basically, it's like asking a user to rip the CD with UltraISO and "manually" upload the file in a Web Form.
The only way to achieve your result is definitely to create an executable application, and/or a Java/.NET applet (I don't remember if/how Mono supports .NET controls on Firefox). But the executable is the basic way. You must then use a portable framework like Java or .NET/Mono which work on Windows, Mac and Linux.
In order to avoid dependencies, you must also ship the package with all the ISO-ripping libraries needed. I don't think there is a single managed-code library to rip CDs, so you need some OS help. You program should differentiate between OSes: in Linux and Mac, use dd (lots of tutorials), in Windows use NeroAPI or an open source ripper.
Once the ISO is created on a temp folder, just open it as a stream and upload via socket. I suggest not to use HTTP but FTP instead, then!
A 1-click solution is feasible, but forget the browser