I'm attempting to use Panda with my GWT application. I can upload videos directly to my panda server using
POST MY_PANDA_SERVER/videos/MY_VIDEO_ID/upload
However I would like hide my panda server behind my J2EE (glassfish) server. I would like to achieve this:
Start upload to some servlet on my J2EE server
Authenticate user
POST the file to my panda server while still uploading to servlet
Ideally I would like to never store the file on the J2EE server, but just use it as a proxy to get to the panda server.
Commons FileUpload is nice, but not sufficient in your case. It will parse the entire body in memory before providing the file items (and streams). You're not interested in the individual items. You basically just want to stream the request body from the one to other side transparently without altering it or storing it in memory in any way. FileUpload would only parse the request body into some "useable" Java objects and HttpClient would only create the very same request body again based on those Java objects. Those Java objects consumes memory as well.
You don't need a library for this (or it must be Commons IO to replace the for loop with an oneliner using IOUtils#copy()). Just the basic Java NET and IO API's suffices. Here's a kickoff example:
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
URLConnection connection = new URL("http://your.url.to.panda").openConnection();
connection.setDoOutput(true); // POST.
connection.setRequestProperty("Content-Type", request.getHeader("Content-Type")); // This one is important! You may want to check other request headers and copy it as well.
// Set streaming mode, else HttpURLConnection will buffer everything.
int contentLength = request.getContentLength();
if (contentLength > -1) {
// Content length is known beforehand, so no buffering will be taken place.
((HttpURLConnection) connection).setFixedLengthStreamingMode(contentLength);
} else {
// Content length is unknown, so send in 1KB chunks (which will also be the internal buffer size).
((HttpURLConnection) connection).setChunkedStreamingMode(1024);
}
InputStream input = request.getInputStream();
OutputStream output = connection.getOutputStream();
byte[] buffer = new byte[1024]; // Uses only 1KB of memory!
for (int length = 0; (length = input.read(buffer)) > 0;) {
output.write(buffer, 0, length);
output.flush();
}
output.close();
connection.getInputStream(); // Important! It's lazily executed.
}
You can use apache commons file upload to receive the file. Then you can use http client to upload the file to your panda server with POST. With apache commons file upload you can process the file in memory so you don't have to store it.
Building upon Enrique's answer, I also recommend to use FileUpload and HttpClient. FileUpload can give you a stream of the uploaded file:
// Create a new file upload handler
ServletFileUpload upload = new ServletFileUpload();
// Parse the request
FileItemIterator iter = upload.getItemIterator(request);
while (iter.hasNext()) {
FileItemStream item = iter.next();
String name = item.getFieldName();
InputStream stream = item.openStream();
if (item.isFormField()) {
System.out.println("Form field " + name + " with value "
+ Streams.asString(stream) + " detected.");
} else {
System.out.println("File field " + name + " with file name "
+ item.getName() + " detected.");
// Process the input stream
...
}
}
You could then use HttpClient or HttpComponents to do the POST. You can find an example here.
The best solution is to use apache-camel servlet component:
http://camel.apache.org/servlet.html
Related
I wanted to convert blob images stored in database into jpeg files and then download them as zip file.
Here is the implementation so far.
public void downloadImages(HttpServletResponse response) throws IOException {
List<ScreenUserEntity> users = screenUserRepository.findAll()
.stream().filter( user -> user.getImage() != null && user.getImage().length > 0).collect(Collectors.toList());
String filename = "Image.jpeg";
response.setContentType("image/jpeg");
response.setHeader(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=\"" + filename + "\"");
byte[] decodedBytes = users.get(0).getImage();
ByteArrayInputStream bis = new ByteArrayInputStream(decodedBytes);
BufferedImage image = ImageIO.read(bis);
File outputFile = new File("output.png");
ImageIO.write(image, "png", outputFile);
}
Is there a way to add a file/ image / zip file into the response?
Thanks for your help/
List<ScreenUserEntity> users = screenUserRepository.findAll()
.stream()
.filter( user -> user.getImage() != null && user.getImage().length > 0)
.collect(Collectors.toList());
This loads them all into memory. I don't think you'd want to do that; a few users and your server is just going to keel over. Best case sccenario, it kills this response handler for eating excessive memory.
Is there a way to add a file/ image / zip file into the response?
Certainly! I'm a bit confused about the usage of raw HttpServletResponse; it complicates matters very slightly here: HTTP as a standard requires that you either [A] send the size of the content you send, before you send it, or [B] use chunked transfer encoding.
If you just grab an outputstream and start sending, then by default most servlet engines in java will either store that data into memory first (that's annoying; you have way too much to send for that), or to a temp file (that's unfortunate), and don't actually send any bytes over the wire until you're completely done. You don't want that. It depends on your implementation, see also this answer. It's something you may want to investigate, e.g. using the developer tools of your favourite browser.
At any rate, once you've got that sorted, this is how you can 'stream' a zip file (make the data on-the-fly, send it across in chunks, never require a ton of memory, and nevertheless avoid making temp files):
try (OutputStream raw = response.getOutputStream();
ZipOutputStream zip = new ZipOutputStream(raw)) {
for (ScreenUserEntity user : screenUserRepo.findAll()) {
byte[] img = user.getImage();
if (img == null || img.length == 0) continue;
zip.putNextEntry(new ZipEntry(user.getName() + ".jpg"));
zip.write(img);
}
}
I am working on a File upload/download functionality, in Java using Struts2 framework, where we are uploading to and downloading from a remote server path. All seems to work fine when I check the functionality at my local machine with a local path as the destined path from where i am downloading and to which am uploading the files of any format. The development environment has JBoss server.
But when I run the same over at the prod env, where the application is deployed in Weblogic server, files of .txt,.csv,.html (basically text format files) have my jsp source code appended to the file content.
Below is the code that I have used for downloading:
BufferedOutputStream bout=null;
FileInputStream inStream = null;
byte[] buffer = null;
try {
inStream = new FileInputStream(path+File.separator+filename);
buffer = new byte[8192];
String extension = "";
int pos = filename.lastIndexOf(".");
if (pos > 0)
extension = filename.substring(pos+1);
int bytesRead = 0, bytesBuffered = 0;
response.setContentType("application/octet-stream");
response.setHeader("content-disposition", "attachment; filename="+ filename);
bout = new BufferedOutputStream(response.getOutputStream());
while((bytesRead = fistrm.read(buffer)) > -1){
bout.write(buffer, 0, bytesRead);
bytesBuffered += bytesRead;
if(bytesBuffered > 1048576){
bytesBuffered = 0;
bout.flush();
}
}
} catch (IOException e) {
log.error(Logger.getStackTrace(e));
} finally {
if(bout!=null){
bout.flush();
bout.close();
}
if(inStream!=null)
inStream.close();
}
I have tried using different response content types with respect to the extension, but it was of no help.
Seems like the outputstream has the jsp source code in it even before writing from the inputstream.
Can anyone please suggest a solution and explain why is this happening ?
It is happening because you are writing directly in the outputstream, and then returning a struts result, that is your JSP. You are using an action as if it would be a servlet, which is not.
In Struts2, to achieve your goal, you need to use the Stream result type, as described in the following answers:
https://stackoverflow.com/a/16300376/1654265
https://stackoverflow.com/a/16900840/1654265
Otherwise, if you want to bypass the framework mechanisms and manually write to the outputStream by yourself (there are very rare cases in which it is useful, like downloading dynamically created ZIP), then you must return the NONE result.
Returning ActionSupport.NONE (or null) from an Action class method causes the results processing to be skipped. This is useful if the action fully handles the result processing such as writing directly to the HttpServletResponse OutputStream.
But I strongly suggest you to go with the Stream result, the standard way.
Does Apache Commons File Upload package provides a generic interface to stream parse multipart/form-data chunks via InputStream, appending Array<Byte>, or via any other generic streaming interface?
I know they have a streaming API but the example only shows you how to do that via ServletFileUpload, which I reckon must be specific to Servlet.
If not, are there any other alternative frameworks in JVM that lets you do exactly this? Sadly, the framework that I am using, Spray.io, doesn't seem to provide a way to do this.
bayou.io has a generic MultipartParser
You might need some adapters to work with it, since it has its own
Async
and ByteSource
interfaces.
The following example shows how to use the parser synchronously with InputStream
String msg = ""
//+ "preamble\r\n"
+"--boundary\r\n"
+"Head1: Value1\r\n"
+"Head2: Value2\r\n"
+"\r\n"
+"body.body.body.body."
+"\r\n--boundary\r\n"
+"Head1: Value1\r\n"
+"Head2: Value2\r\n"
+"\r\n"
+"body.body.body.body."
+"\r\n--boundary--"
+ "epilogue";
InputStream is = new ByteArrayInputStream(msg.getBytes("ISO-8859-1"));
ByteSource byteSource = new InputStream2ByteSource(is, 1024);
MultipartParser parser = new MultipartParser(byteSource, "boundary");
while(true)
{
try
{
MultipartPart part = parser.getNextPart().sync(); // async -> sync
System.out.println("== part ==");
System.out.println(part.headers());
ByteSource body = part.body();
InputStream stream = new ByteSource2InputStream(body, Duration.ofSeconds(1));
drain(stream);
}
catch (End end) // control exception from getNextPart()
{
System.out.println("== end of parts ==");
break;
}
}
I posted this question to the CXF list, without any luck. So here we go. I am trying to upload large files to a remote server (think of them virtual machine disks). So I have a restful service that accepts upload requests. The handler for the upload looks like:
#POST
#Consumes(MediaType.MULTIPART_FORM_DATA)
#Path("/doupload")
public Response receiveStream(MultipartBody multipart) {
List<Attachment> allAttachments = body.getAllAttachments();
Attachment att = null;
for (Attachment b : allAttachments) {
if (UPLOAD_FILE_DESCRIPTOR.equals(b.getContentId())) {
att = b;
}
}
Assert.notNull(att);
DataHandler dh = att.getDataHandler();
if (dh == null) {
throw new WebApplicationException(HTTP_BAD_REQUEST);
}
try {
InputStream is = dh.getInputStream();
byte[] buf = new byte[65536];
int n;
OutputStream os = getOutputStream();
while ((n = is.read(buf)) > 0) {
os.write(buf, 0, n);
}
ResponseBuilder rb = Response.status(HTTP_CREATED);
return rb.build();
} catch (IOException e) {
log.error("Got exception=", e);
throw new WebApplicationException(HTTP_INTERNAL_ERROR);
} catch (NoSuchAlgorithmException e) {
log.error("Got exception=", e);
throw new WebApplicationException(HTTP_INTERNAL_ERROR);
} finally {}
}
The client for this code is fairly simple:
public void sendLargeFile(String filename) {
WebClient wc = WebClient.create(targetUrl);
InputStream is = new FileInputStream(new File(filename));
Response r = wc.post(new Attachment(Constants.UPLOAD_FILE_DESCRIPTOR,
MediaType.APPLICATION_OCTET_STREAM, is));
}
The code works fine in terms of functionality. In terms of performance, I noticed that before my handler (receiveStream() method) gets the first byte out of the stream, the whole stream actually gets persisted into a temporary file (using a CachedOutputStream). Unfortunately, this is not acceptable for my purposes.
My handler simply passes the incoming bytes to a backend storage system (virtual machine disk repository), and waiting for the whole disk to be written to a cache only to be read again takes a lot of time, tying up a lot of resources, and reducing throughput.
There is a cost associated with writing the blocks and reading them again, since the app is running in the cloud, and the cloud provider charges per block read/written.
Since every byte is written to the local disk, my service VM must have enough disk space to accommodate the total sizes of all the streams being uploaded (i.e., if I have 10 uploads of 100GB each, I must have 1TB of disk just to cache the content). That again is extra money, as the size of the service VM grows dramatically, and the cloud provider charges for the provisioned disk size as well.
Given all of this, I am looking for a way to use the HTTP InputStream (or as close to it as possible) to read the attachment directly from there and handle it afterwards. I guess the question translates into one of:
- Is there a way to tell CXF not do caching
- OR - is there a way to pass CXF an output stream (one I write) to use, rather than using CachedOutputStream
I found a similar question here. The resolution says use CXF 2.2.3 or later, I am using 2.4.4 (and tried with 2.7.0) with no luck.
Thanks.
I think it's logically not possible (neither in CXF or anywhere else). You're calling getAllAttachements(), which means that the server should collect information about them from the HTTP input stream. It means that the entire stream has to go into memory for MIME parsing.
In your case you should work directly with the stream, and do the MIME parsing yourself:
public Response receiveStream(InputStream input) {
Now you have full control of the input and can consume it into memory byte-by-byte.
I ended up fixing the problem in an unelegant way, but it works, so I wanted to share my experience. Please do let me know if there are some "standard" or better ways.
Since I am writing the server side, I knew I was accessing all the attachments in the order they were sent, and process them as they are streamed in. So, to reflect that behavior of the handler method (receiveStream() method above), I created a new annotation on the server side called "#SequentialAttachmentProcessing" and annotatated my above method with it.
Also, wrote a subclass of Attachment, called SequentialAttachment that acts like a linked list. It has a skip() method that skips over the current attachment, and when an attachment ends, hasMore() method tells you whether there is another one.
Then I wrote a custom multipart/form-data provider which behaves as follows: If the target method is annotated as above, handle the attachment, otherwise call the default provider to do the handling. When it is handled by my provider, it always returns at most one attachment. Hence it could be misleading to a non-suspecting handling method. However, I think it is acceptable since the writer of the server must have annotated the method as "#SequentialAttachmentProcessing" and therefore must know what that entails.
As a result the implementation of the receiveStream() method is now something like:
#POST
#SequentialAttachmentProcessing
#Consumes(MediaType.MULTIPART_FORM_DATA)
#Path("/doupload")
public Response receiveStream(MultipartBody multipart) {
List<Attachment> allAttachments = body.getAllAttachments();
Assert.isTrue(allAttachments.size() <= 1);
if (allAttachment.size() > 0) {
Attachment head = allAttachments.get(0);
Assert.isTrue(head instanceof SequentialAttachment);
SequentialAttachment att = (SequentialAttachment) head;
while (att != null) {
DataHandler dh = att.getDataHandler();
InputStream is = dh.getInputStream();
byte[] buf = new byte[65536];
int n;
OutputStream os = getOutputStream();
while ((n = is.read(buf)) > 0) {
os.write(buf, 0, n);
}
if (att.hasMore()) {
att = att.next();
}
}
}
}
While this solved my immediate problem, I still believe there has to be a standard way of doing this. I hope this helps someone.
How to save the image on the servlet?
And how to send it back to a web page?
This servlet receives a request from JS with a file input (image).
I want to save the picture (maybe on a list -db not needed) and then servlet sends back a response with everything he received (together with the photo).
Is there any suggestion?
I tried unsuccessfully with this code:
response.setContentType("text/html; charset=UTF-8");
PrintWriter out = response.getWriter();
out.println("</HEAD>\n");
out.println("<BODY>\n" +
"<TABLE>\n" +
"<TR>\n" +
"<TH>--Asked Infos--" +
"<TH>--Your Input--");
/*for typical inputs from the form */
Enumeration paramNames = request.getParameterNames();
while( paramNames.hasMoreElements() )
{
String paramName = (String)paramNames.nextElement();
out.println("<TR><TD>" + paramName + "\n<TD>");
String[] paramValues = request.getParameterValues(paramName);
String paramValue = paramValues[0];
if ( paramValue.length() == 0 ){
/* .... store a default photo from servlet 'cause user didn't give file...*/
}
out.print(paramValue);
}
/* file input */
String fileName = request.getParameter("avatar");/*avatar is the file input name from JS*/
FileInputStream fis = new FileInputStream(new File("C:\\"+fileName));
BufferedInputStream bis = new BufferedInputStream(fis);
BufferedOutputStream output = new BufferedOutputStream(response.getOutputStream());
for (int data; (data = bis.read()) > -1;)
{
output.write(data);
}
out.println("</TABLE>\n</BODY></HTML>");
The way to upload a file is creating a form with <input type="file" name="avatar">. For the actual upload inside the servlet one should better use an apache or Spring upload servlet filter. For that code search in the internet - a good excercise.
A link here, for support in the newest JavaEE6 support (apache Geronimo server, but Glassfish should be the same). Before JEE6 one needed extra code, a servlet filter.