I need to retrieve from an Application Server (JBoss) a large file (gigabytes) and to avoid loading it in memory, I want to stream it through EJB.
Is it possible to take data out of an Application Server as a stream?
Create a HttpServlet, stream the file.
update
Be careful with your header. You cannot set the ContentLength-Header via setContentLength(), because it only accept int.
You wil have to set it with: setHeader("Content-Length", (long)length)
Maybe this will be helpful: Using ServletOutputStream to write very large files in a Java servlet without memory issues
There is a limit, but it depends on the client-side. If the client will hold the file in the memory, it will not work.
By EJB do you mean remote bean? These beans are typically based on RMI which in turn uses Java serialization. You cannot stream data using RMI.
However with servlets and HTTP this will be dead simple. Just open FileInputStream to your large file and copy it byte-by-byte to servlet output.
To remember:
Use input file buffering
At the very beginning set Content-Length header so that the client knows how much data is left
Related
I want to download a very large file using camel, but I don't want to hold the entire file in memory and THEN save it to file.
I want to stream the file in and save or write to a file in chunks.
Is this possible with Camel, and if so, how do I do this?
Note: Is it possible that the endpoint I am downloading the file does not support streaming/chunking? If yes, how can I verify this?
Camel's HTTP component uses Netty to make the request. Netty reads the entire response into memory, so there is no way to do what you are asking for.
You would need to implement your own endpoint for Camel that utilizes another HTTP library which has support for HTTP response streaming.
More documentation is available here :
https://cwiki.apache.org/confluence/display/CAMEL/Netty4+HTTP
You can 3 option to download the file i.e. using:
ftp://[username#]hostname[:port]/directoryname[?options]
sftp://[username#]hostname[:port]/directoryname[?options]
ftps://[username#]hostname[:port]/directoryname[?options]
There is a option of streamDownload in it.
For more check out http://camel.apache.org/ftp.html
I'm on a project that exposes ABAP functionality as a web service via Netweaver / Java (7.01 SP3 I think). We consume it on an .NET 4 UI tier. We are dealing with some large message structures (12MB serialized XML) that are taking too many seconds to shuttle between the various tiers.
We're tackling this performance on a number of fronts:
Disk, network, CPU and memory are fine and nowhere near saturated.
We're working to trial WCF Streaming mode
We may try gzip compression on the web service's server
And lastly, the point of this question: is there a way to enable binary serialization that's interoperable?
Assuming that you already tried everything to get the payload size down and to split it into smaller pieces (12 MB XML, seriously!), I'd say that depends on the kind of XML processing you need on the ABAP side. You could try to implement your own ICF HTTP handlers and go for some REST-style interface. This is especially interesting if you really want to transfer binary data (for example, some document you retrieve from an archive system) because you can then transfer the document via HTTP without the XML-binary-ugliness. Even if you have to use fill WSDL-y web services, you could try to refactor the binary parts out of that interface, just send some (GU)ID through the web service and have the client fetch the binary part from your custom ICF handler.
I'm trying to combine these articles: http://java.sun.com/developer/technicalArticles/RMI/rmi_corba/ and http://netbeans.org/kb/docs/javaee/entappclient.html to make simple client-server app using Glassfish, in which I could send a file from (local) client to a directory on the (local) server. This is something new for me and I feel a little overwhelmed at the moment. Any advice, please?
You're kind of in the wrong area. The things you're looking at are for support of RPC sessions. In theory you could send over an enormous byte array, but it is likely unwise to do so.
What would be preferable is to create a simple web app and push the file over HTTP.
Or you could try using a WS Web Service that's configured for MTOM -- it will handle large payloads as well. You can look here for an article of streaming MTOM messages. It's for WebLogic, but it's basically Sun JAX-WS so it should work on Glassfish out of the box.
An advantage of the Web Service is you can host it in an EJB, rather than having to deploy a separate WAR for this facility. What you want to watch out for is having the payload being all stored in RAM. For example, if you want to send a 10Gb file, the actual traffic is going to be the same, but done naively, you will end up holding all 10Gb in the heap on the client and/or the server, which obviously is not desirable.
In the end either way will work. The Web Service had the downside of having to dig in to the shadowy corners of the Web Service stack, where as with a generic Servlet and web app, it's more out in the open, however you will likely be diving in to the inner depths of HTTP to pull that off. For example, if you wanted to use Apache HTTP Client, you would need to create a custom RequestEntity to handle the streaming for you.
All possible, it's just less used and not the default, out of the box, 2 lines of code tutorial example.
I have a webapp with an architecture I'm not thrilled with. In particular, I have a servlet that handles a very large file upload (via commons-fileupload), then processes the file, passing it to a service/repository layer.
What has been suggested to me is that I simply have my servlet upload the file, and a service on the backend do the processing. I like the idea, but I have no idea to go about it. I do not know JMS.
Other details:
- App is a GWT app split into the recommended client/server/shared subpackages, using an MVP architecture.
- Currently, I am only running in GWT hosted mode, but am planning to move to Tomcat in the very near future.
I'm perfectly willing to learn whatever I need to in order to get this working (in fact, that's the point of writing the app). I'm not expecting anyone to write code for me, but can someone point me in the right direction to get started?
There are many options for this scenario, but the simplest may be just copying the uploaded file to a known location on the file system, and have a background daemon monitor the location and process when it finds it.
#Jason, there are many ways to solve your problem.
i) Have dump you file data into Database with column type BLOB. and have a DB polling thread(after a particular time period) polls table for newly inserted file .
ii) Have dump file into file system and have a file montioring process.
Benefit of i) over ii) is that DB is centralized and fast resource where as file systems are genrally slow and non-centalized in nature.
So basically servlet would dump either to DB or file system. Now about who will process that dumped file:- a) It could be either montioring process as discussed above or b) you can use JMS which is asynchronous in nature what it means servlet would put a trigger event in queue which will asynchronously trigger new processing thread.
Well don't introduce JMS in your system unnecessarily if you are ok with monitoring process.
This sounds interesting and familiar to me :). We do it in the similar way.
We have our four projects, all four projects includes file upload and file processing (Image/Video/PDF/Docs) etc. So we created a single project to handle all file processing, it is something like below:
All four projects and File processor use Amazon S3/Our File Storage for file storage so file storage is shared among all five projects.
We make request to File Processor providing details in XML via http request which include file-path on S3/Stoarge, aws authentication details, file conversion/processing parameters. File Processor does processing and puts processed files on S3/Storage, constructs XML with processed files details and sends XML via response.
We use Spring Frameowrk and Tomcat.
Since this is foremost a learning exercise, you need to pick an easy to use JMS provider. This discussion suggested FFMQ just one year ago.
Since you are starting with a simple processor, you can keep it simple and use a JMS Queue.
In the simplest form, each message send by the servlet has to correspond to a single job. You can either put the entire payload of the upload in the message, or just send a filename as reference to the content in the message. These are details you can refactor later.
On the processor side, if you are using Java EE, you can use a MessageBean. If you are not, then I would suggest a 3 JVM solution -- one each for Tomcat, the JMS server, and the message processor. This article includes the basics of a message consuming client.
I'm writing a Java desktop client which will send multiple files over the wire to a servlet using a post request. In the servlet I'm getting the input stream from the request to receive the files. The servlet will write the files to disk, one by one as they're read from the stream.
The implementation has a couple of requirements:
Only one HTTP request must be used to the server (so only a single stream)
The servlet must use a reasonable fixed amount of memory, no matter what the size of the files.
I had considered inserting markers into the stream so I know when one file ends and the next one begins. I'd then write some code to parse the stream in the servlet, and start writing the next file as appropriate.
Here's the thing... surely there's a library to do that. I've looked through apache commons and found nothing. Commons File Upload is interesting but since the upload comes from a Java app, not a browser it only solves the receiving end, not the sending.
Any ideas for a library which easily allows multiple file transfers across a single stream with fixed memory expectations even for very large files?
Thanks.
Just use HTTP multipart/form-data encoding on the POST request body. It's described in RFC-2388 and a standard way of uploading (multiple) files by HTTP.
You can do it with just java.net.URLConnection as described in this mini-tutorial, although it would generate lot of boilerplate code. A more convenienced approach would be using Apache Commons HttpClient.
In the servlet side you can then just use Apache Commons Fileupload to process the uploaded files the usual HTTP way (or when you're already on Servlet 3.0, the HttpServletRequest#getParts(), see also this answer for examples).