I run a simple backend app which allows to upload files. I use Jersey and run it in Jetty. The piece of my code looks like this:
#POST
#Consumes(MediaType.MULTIPART_FORM_DATA)
public Response uploadFile(#Context UriInfo uriInfo, FormDataMultiPart multipart, #Context HttpServletRequest httpRequest) {
FormDataBodyPart fileId = multipart.getField("fileId");
FormDataBodyPart fileSize = multipart.getField("fileSize");
FormDataBodyPart file = multipart.getField("file");
ContentDisposition cd = file.getContentDisposition();
String fileName = cd.getFileName();
long size = Long.valueOf(fileSize.getValue());
...
Upload works just fine, but I found that the method is called just when the whole stream is uploaded to the backend. So, for instance, if I send huge file (3Gigs to be uploaded) my POST request immediately appears on the backend, but the method above is invoked only when the whole 3 Gigs are uploaded through the network.
I would like to make some checks in the method and don't upload the file for some cases, so it doesn't need to pass the whole content to the backend and then send the error message back.
How can I avoid uploading the whole file content to the backend but make the method is invoked before I start to read from the data stream?
Eventually I worked it around with 2 sequential POSTs the client should make to upload such big files:
#POST
#Consumes(MediaType.MULTIPART_FORM_DATA)
public Response newUpload(#Context UriInfo uriInfo, FormDataMultiPart multipart,
#Context HttpServletRequest httpRequest) {
FormDataBodyPart fileSize = multipart.getField("fileSize");
long size = Long.valueOf(fileSize.getValue());
if (!checkSizeLimits(size)) {
return Response.status(Status.BAD_REQUEST).build();
}
// here comes some code which generates an unique id and its URI for the data
...
return Response.status(Status.CREATED).entity(new FileUploadInfo(uri.toString())).build();
}
#POST
#Path("/{fileId}")
#Consumes(MediaType.MULTIPART_FORM_DATA)
public Response uploadFile(#Context UriInfo uriInfo, FormDataMultiPart multipart,
#Context HttpServletRequest httpRequest, #PathParam("fileId") String fileId) {
FormDataBodyPart fileSize = multipart.getField("fileSize");
FormDataBodyPart file = multipart.getField("file");
//...
return Response.status(Status.CREATED).entity(new FileUploadInfo(uri.toString())).build();
}
It looks ugly, but better than to do the file size check after the data stream is completely loaded. So, client should make POST call to the newUpload() method first providing the file size. If the size is ok, the response contains URI for the second POST to stream the file data. The second method can do the file size check again (not provided), to be sure that the initial POST had the same file size field.
Related
can you please help on below points to understand Jersey flow.
Considering below example code
#POST
#Consumes(MediaType.MULTIPART_FORM_DATA)
#Produces({ MediaType.APPLICATION_JSON })
public CustomObject acceptFile(#FormDataParam("json") FormDataBodyPart json,
#FormDataParam("data") FormDataContentDisposition contentDisposition,
#FormDataParam("data") final InputStream input){
...
}
if I transfer file of 20 MB, control does not get into above method until all 20 MB data is transferred to server from client. this is my observation. am I correct?
Does Jersey creates temporary files or stores file contents in memory?
if Jersey creates temporary files where does these temporary files being created?
For completeness I tried to use HTML as client.
Is it possible to send two different Content-Type for POST method using post-man? Like, if the service is used for downloading excel file so in that case, #Consumes(MediaType.APPLICATION_JSON) is used for sending some user detail which is json structure and #Produces(MediaType.APPLICATION_OCTET_STREAM) is used for sending back the response as file.
NOTE: I don't want to use form-data, that is a constraint for this service.
On the client request, the Content-Type header is only for the type of data that is in the entity body. The Accept header is what you send for the type of data you want back.
On the server side, it is perfectly fine to do the following.
#POST
#Path("something")
#Consumes(MediaType.APPLICATION_JSON)
#Produces(MediaType.APPLICATION_OCTET_STREAM)
public Response post(Model model) throws Exception {
final InputStream in = new FileInputStream("path-to-file");
StreamingOutput entity = new StreamingOutput() {
#Override
public void write(OutputStream out) {
IOUtils.copy(in, out);
out.flush();
}
};
return Response.ok(entity)
.header(HttpHeaders.CONTENT_DISPOSITION,
"attachment;filename=somefile.xls")
.build();
}
The request to this endpoint should look like
POST /api/something HTTP 1.1
Content-Type: application/json;charset=utf-8
Accept: application/octet-stream
{"the":"json", "entity":"body"}
See also:
This post about purpose of #Produces and #Consumes and the role they play in content negotiation.
Aside
As an aside, consider using application/vnd.ms-excel as the Content-Type instead of application/octet-stream. This is already the standard MIME type for Microsoft Excel files1. When using StreamingOutput as the response entity type, you can make the #Produces anything you want, as there is no real conversion needed.
In Postman, when you use the "Send and download" feature, you will notice that when the Content-Type is application/octet-stream, it will suggest the file extension of .bin when saving the file. But if you have the Content-Type as application/vnd.ms-excel, then the suggested extension is .xls, which is what it should be.
1. Incomplete list of MIME types
I have a large file download that is served by a RestController on one server, that I need to stream through a RestController on another server. When calling the end server directly the result streams fine. However when using RestTemplate to call this server and then write the response to an OutputStream, the response is buffered on the front server until the whole file is ready, and then streamed. Is there a way I can write the file to an OutputStream as it comes in?
At the moment my code on the front server looks similar to this
#ResponseBody
public void downloadResults(HttpServletRequest request, HttpServletResponse response, #RequestParam("id") String jobId, OutputStream stream)
throws IOException
{
byte[] data = restTemplate.exchange("http://localhost/getFile", HttpMethod.POST, requestEntity, byte[].class, parameters).getBody();
stream.write(data);
}
I've set my RestTemplate to not buffer and I've verified that this is working by checking the Request type that is used, (SimpleStreamingClientHttpRequest).
The data all comes back correct, its just only written to the stream all at once, rather than as it comes in
RestTemplate is not meant for streaming the response body, as pointed out in this JIRA issue.
You can use restTemplate.execute. See https://www.baeldung.com/spring-resttemplate-download-large-file
I am looking to create a RESTful API for use with an Android and iOS app. So far I have been experimenting with using Jersey on the server and then the appropriate http libraries on the client side. At the moment I have been using multipart/related as the mimetype for the request with JSON forming the first part of the body then a jpeg image as the second.
So far I have had problems with making the request to the server, getting a 406 Not Acceptable from Jersey. I note that multipart/related is primarily used in sending emails. Is there actually a way that I can support mixed type content as an upload or have I entirely mis-understood the usage of multipart/related in this context?
You may want to look at this blog, for more information, but here is the important part to help you along:
http://www.mkyong.com/webservices/jax-rs/file-upload-example-in-jersey/
#POST
#Path("/upload")
#Consumes(MediaType.MULTIPART_FORM_DATA)
public Response uploadFile(
#FormDataParam("file") InputStream uploadedInputStream,
#FormDataParam("file") FormDataContentDisposition fileDetail) {
String uploadedFileLocation = "d://uploaded/" + fileDetail.getFileName();
// save it
writeToFile(uploadedInputStream, uploadedFileLocation);
String output = "File uploaded to : " + uploadedFileLocation;
return Response.status(200).entity(output).build();
}
I expect you want multipart/form-data instead, as this is part of the description of multipart/related:
The Multipart/Related media type is intended for compound objects
consisting of several inter-related body parts. For a
Multipart/Related object, proper display cannot be achieved by
individually displaying the constituent body parts. The content-type
of the Multipart/Related object is specified by the type parameter.
The "start" parameter, if given, points, via a content-ID, to the
body part that contains the object root. The default root is the
first body part within the Multipart/Related body.
For more on this mime type you can look at
https://www.rfc-editor.org/rfc/rfc2387
If you are wanting to submit image along with the json body, you can base64 encode the image and include the base64 string in the json. Then on the server side, you base64 decode the string and upload the image file to the blobstore. See the file upload example (at the bottom of the page) here https://developers.google.com/appengine/docs/java/blobstore/overview
Alternatively, you could do a separate upload to the blobstore and get the blobkey for the uploaded image. You can then include the blobkey in the json body that you post to the server.Using this approach you would need to get the uploadurl every time you need to do a new image upload.
I have this method signature in a jersery servlet. The servlet is being reached and the form data is present in the uploadedInputStream object, but the stream does not have the http artifacts removed from it. See below.
#POST
#Produces("text/plain")
#Consumes(MediaType.MULTIPART_FORM_DATA)
public String uploadFileIE(
#FormDataParam("qqfile") InputStream uploadedInputStream ){
}
When saved to a file, the input stream has these artifacts surrounding the byte data:
-----------------------------7dc1f42e3005a8
Content-Disposition: form-data; name="qqfile";filename="[filename]"
Content-Type: application/octet-stream
[bytes from data stream]
-----------------------------7dc1f42e3005a8--
Shouldn't these artifacts be removed already at this point? Is there any easy way to remove them without re-inventing the wheel?
With Jersey you need to 'consume' the extra header information with a FormDataContentDisposition object. Messy but necessary:
#POST
#Produces("text/plain")
#Consumes(MediaType.MULTIPART_FORM_DATA)
public String uploadFileIE(
#FormDataParam("qqfile") InputStream uploadedInputStream,
#FormDataParam("qqfile") FormDataContentDisposition fileDetail){
}