I have a large file download that is served by a RestController on one server, that I need to stream through a RestController on another server. When calling the end server directly the result streams fine. However when using RestTemplate to call this server and then write the response to an OutputStream, the response is buffered on the front server until the whole file is ready, and then streamed. Is there a way I can write the file to an OutputStream as it comes in?
At the moment my code on the front server looks similar to this
#ResponseBody
public void downloadResults(HttpServletRequest request, HttpServletResponse response, #RequestParam("id") String jobId, OutputStream stream)
throws IOException
{
byte[] data = restTemplate.exchange("http://localhost/getFile", HttpMethod.POST, requestEntity, byte[].class, parameters).getBody();
stream.write(data);
}
I've set my RestTemplate to not buffer and I've verified that this is working by checking the Request type that is used, (SimpleStreamingClientHttpRequest).
The data all comes back correct, its just only written to the stream all at once, rather than as it comes in
RestTemplate is not meant for streaming the response body, as pointed out in this JIRA issue.
You can use restTemplate.execute. See https://www.baeldung.com/spring-resttemplate-download-large-file
Related
I'm trying to create an application that download and uploads large files, so I don't want the file contents to be stored in memory.
On the mvc controller side I'm using an http message converter that converts to / from InputStream
#Override
public InputStream read(Class<? extends InputStream> clazz, HttpInputMessage inputMessage) throws IOException,
HttpMessageNotReadableException {
return inputMessage.getBody();
}
#Override
public void write(InputStream t, MediaType contentType, HttpOutputMessage outputMessage) throws IOException,
HttpMessageNotWritableException {
try {
IOUtils.copy(t, outputMessage.getBody());
} finally {
IOUtils.closeQuietly(t);
}
}
This works well on the server side.
On the client (RestTemplate) side I tried to use the same converter, but I got an exception that the stream has been closed (probably closed when the request was completed).
Client side code:
ResponseEntity<InputStream> res = rest.getForEntity(url, InputStream.class);
// res.getBody() is closed
I've also tried to copy the input stream into a buffer and create a new ByteArrayInputStream and return it to the RestTemplate client and it worked well, however it does require that the data will be read into memory which doesn't suite my demands.
My question is how to keep the stream open until I process it without having to read it all into memory / file?
Any idea will be appreciated.
Regards, Shay
As far as I am aware, RestTemplate's getForEntity() is not an appropriate way to get an InputStream. It's a convenience for converting to and from entity classes, so presumably that's where your problem lies.
Since you are used to HttpInputMessage, why don't you use HttpInputMessage.getBody() on the client side as well? It gets you a nice InputStream, which would be ready for passing straight to an OutputStream such as HttpServletResponse.getOutputStream().
Check how Spring MVC handles large files upload with org.springframework.web.multipart.commons.CommonsMultipartResolver. It has a 'maxInMemorySize' that can help control the memory requirements. See this thread for using a multipart resolver with the REST template Sending Multipart File as POST parameters with RestTemplate requests
I run a simple backend app which allows to upload files. I use Jersey and run it in Jetty. The piece of my code looks like this:
#POST
#Consumes(MediaType.MULTIPART_FORM_DATA)
public Response uploadFile(#Context UriInfo uriInfo, FormDataMultiPart multipart, #Context HttpServletRequest httpRequest) {
FormDataBodyPart fileId = multipart.getField("fileId");
FormDataBodyPart fileSize = multipart.getField("fileSize");
FormDataBodyPart file = multipart.getField("file");
ContentDisposition cd = file.getContentDisposition();
String fileName = cd.getFileName();
long size = Long.valueOf(fileSize.getValue());
...
Upload works just fine, but I found that the method is called just when the whole stream is uploaded to the backend. So, for instance, if I send huge file (3Gigs to be uploaded) my POST request immediately appears on the backend, but the method above is invoked only when the whole 3 Gigs are uploaded through the network.
I would like to make some checks in the method and don't upload the file for some cases, so it doesn't need to pass the whole content to the backend and then send the error message back.
How can I avoid uploading the whole file content to the backend but make the method is invoked before I start to read from the data stream?
Eventually I worked it around with 2 sequential POSTs the client should make to upload such big files:
#POST
#Consumes(MediaType.MULTIPART_FORM_DATA)
public Response newUpload(#Context UriInfo uriInfo, FormDataMultiPart multipart,
#Context HttpServletRequest httpRequest) {
FormDataBodyPart fileSize = multipart.getField("fileSize");
long size = Long.valueOf(fileSize.getValue());
if (!checkSizeLimits(size)) {
return Response.status(Status.BAD_REQUEST).build();
}
// here comes some code which generates an unique id and its URI for the data
...
return Response.status(Status.CREATED).entity(new FileUploadInfo(uri.toString())).build();
}
#POST
#Path("/{fileId}")
#Consumes(MediaType.MULTIPART_FORM_DATA)
public Response uploadFile(#Context UriInfo uriInfo, FormDataMultiPart multipart,
#Context HttpServletRequest httpRequest, #PathParam("fileId") String fileId) {
FormDataBodyPart fileSize = multipart.getField("fileSize");
FormDataBodyPart file = multipart.getField("file");
//...
return Response.status(Status.CREATED).entity(new FileUploadInfo(uri.toString())).build();
}
It looks ugly, but better than to do the file size check after the data stream is completely loaded. So, client should make POST call to the newUpload() method first providing the file size. If the size is ok, the response contains URI for the second POST to stream the file data. The second method can do the file size check again (not provided), to be sure that the initial POST had the same file size field.
I want to have two interceptors for every request I do on a webservice. One for the outgoing communication, and one for the response.
I am using ClientHttpRequestInterceptor which is working for the outgoing. I am setting it as follows:
//Rest template
RestTemplate tpl = api.getRestTemplate();
List<ClientHttpRequestInterceptor> interceptors = new ArrayList<ClientHttpRequestInterceptor>();
interceptors.add( new OutgoingRequestInterceptor() );
tpl.setInterceptors( interceptors );
However, I want something like this interceptor for the incoming (response). I checked Spring Framework sourcecode and I couldn't find anything for this.
Any tips?
Edit:
Maybe I am confused or something is wrong in my head. Im a bit ill today.
I've the following code in my interceptor class:
#Override
public ClientHttpResponse intercept( HttpRequest request, byte[] bytes, ClientHttpRequestExecution requestExecution ) throws IOException
{
SLog.d( "intercepted!!"+request.getURI()+". Bytes: "+bytes );
try
{
Thread.sleep( 5000 );
}
catch ( InterruptedException e )
{
e.printStackTrace();
}
ClientHttpResponse response = requestExecution.execute( request, bytes );
SLog.d( "Response Headers: " + response.getHeaders());
return response;
}
Question: Is this code working for outgoing, incoming, or both?
Because the log:request.getUri() is returning the destination URL.
Then, on the Response object, I get the headers sent by WebService.
So what I am sure of, is that response is actually the server response. But... How about getUri() thingy? Is it triggered before actually sending the request, or after?
Okay. After some tricky debugging, I got it.
Even though the interceptor class is called ClientHttpRequestInterceptor, it's intercepting both. Request from client, and respose from Server.
This interceptor class is something like a wrapper.
So...
This method is the wrapper of the whole request. From BEFORE request and after the request is done.
This part of the code is triggered BEFORE request is sent to webservice
This part of the code ACTUALLY CONTACTS WEBSERVICE, so it "pauses" there until it gets the response from the web service.
We return the response generated. Notice that if you use method response.getBody() which is an InputStream, you will consume it, so it will be null afterwards. I say that because you CAN'T directly log it. You've to mirror it first.
I have a servlet that accepts POST requests, and it works perfectly when I'm not trying to read parameters. The inputstream contains an XML file which parses just fine. However, when I try to read a parameter:
String account = request.getParameter("account");
It fails when reading the inputstream.
How can this possibly affect the inputstream?
I suppose you are using a HttpServletRequest and from the ServletRequest#getParameter() documentation:
If the parameter data was sent in the request body, such as occurs with an HTTP POST request, then reading the body directly via getInputStream() or getReader() can interfere with the execution of this method.
Are you using either getInputStream() or getReader() prior to your getParameter() ?
A related answer here might help you solve your problem: https://stackoverflow.com/a/17129256/1524381
I have a controller method that simply streams bytes for media (images, css, js, etc.) to the client. I first tried something like this:
#RequestMapping(value="/path/to/media/**", method=RequestMethod.GET)
#ResponseBody
public byte[] getMedia(HttpServletRequest request) throws IOException
{
//logic for getting path to media on server
return Files.readAllBytes(Paths.get(serverPathToMedia));
}
I originally tested this in Firefox, and it all seemed to work fine. However, I then tried it in Chrome, and then found that none of the images work. So, I then changed it to something like this:
#RequestMapping(value="/path/to/media/**", method=RequestMethod.GET)
public ResponseEntity<byte[]> getMedia(HttpServletRequest request) throws IOException
{
//logic for getting path to media on server
byte[] bytes = Files.readAllBytes(Paths.get(serverPathToMedia));
//logic for setting some header values like Content-Type and Content-Length
return new ResponseEntity<byte[]>(bytes, headers, HttpStatus.OK);
}
This gave the same results as before. I saw in the developer tools that my response headers were coming down as expected, but still no image bytes
Next I tried something like this:
#RequestMapping(value="/path/to/media/**", method=RequestMethod.GET)
public void getMedia(HttpServletRequest request, HttpServletResponse response) throws IOException
{
//logic for getting path to media on server
byte[] bytes = Files.readAllBytes(Paths.get(serverPathToMedia));
response.getOutputStream().write(bytes);
}
Without even setting any response headers, this works in Firefox and Chrome. Now, while I can just do it this last way since it works, this doesn't seem like the correct Spring MVC way. I want to know why the first two things I tried didn't work, as they seem more correct. Also, is there something I didn't try that would actually be the right way to do this?
Your last approach is pretty much the way to go about it. The only change that I can suggest is to not keep the entire content file to be streamed in memory, instead to stream out the content with buffering - IOUtils from Apache commons can do this for you.