I need to create a rest service in java which will in turn connect to another rest service for file download. For now, I just need to transfer the file from the other backend to client but in future some processing/transformations would be done.
For all the web services in my project, we are using spring rest (for providing as well as consuming the services).
My question is what would be the appropriate way of doing it considering that the files would be large and I don't want to run into OutOfMemory errors.
People in some other posts have suggested to use streams on both the ends but is that really possible? For this, do I need to write the file on disk first?
My current code for file download (consumer) -
public BackendResponse<byte[]> callBackendForFile(BackendRequest request) {
String body = null;
ResponseEntity<byte[]> responseEntity = null;
URI uri = createURI(request);
MultiValueMap<String, String> requestHeaders = getHeadersInfo(request.getHttpRequest());
if (HttpMethod.GET.equals(request.getMethod())) {
responseEntity = restTemplate.exchange(uri, request.getMethod(),
new HttpEntity<String>(body, requestHeaders), byte[].class);
} else {
LOG.error("Method:{} not supported yet", request.getMethod());
}
BackendResponse<byte[]> response = new BackendResponse<>();
response.setResponse(responseEntity);
return response;
}
My client code (provider):
#RequestMapping(value = "/file", method = RequestMethod.GET, produces = "application/xml")
#ResponseBody
public void downloadFileWithoutSpring(HttpMethod method, HttpServletRequest httpRequest,
HttpServletResponse httpResponse) {
BackendRequest request = new BackendRequest(method,
httpRequest.getRequestURI(), httpRequest.getQueryString(), httpRequest);
BackendResponse<byte[]> backendResponse = dutyplanService.getFile(request);
ResponseEntity<byte[]> response = backendResponse.getResponse();
httpResponse.addHeader("Content-Disposition", "attachment; filename=\"" + "attachment.zip" + "\"");
httpResponse.getOutputStream().write(response.getBody());
httpResponse.flushBuffer();
}
Note: The code above doesn't work somehow as the attachment downloaded is a corrupt file
I don't think you will need to create that file on server as long as you are having the bytearray content of it received from another server.
You can try changing value of produces annotation to the value application/zip (or application/octet-stream, depending on the target browser) instead of 'application/xml'
you can pass HttpServletResponse#getOutputStream() directly in restTemplate and write it without save file in server.
public void getFile(HttpServletResponse response) throws IOException {
restTemplate.execute(
"http://ip:port/temp.csv",
HttpMethod.GET,
null,
clientHttpResponse -> {
StreamUtils.copy(clientHttpResponse.getBody(), response.getOutputStream());
return null;
}
);
}
note that after call getFile(), you should close outputStream like this
response.getOutputStream().close()
Related
I am trying to download zip file from service A, where it calls service B for file. I need a solution to stream file across services
example when i call service A for file, it will call Service B. from here it should stream to Service A. from service A it'll stream to the caller.
The reason for stream between services is, i don't want to store the file in service A. i just want to pass to the caller without storing it.
And also let me know which option to use in service A. like ByteArrayResource or rest template response extractor etc..
Limitation Service B is not in my control so as of now i am accepting file as byte array
Here is the small simulation in single controller
#RestController
public class FileUploadController {
#Autowired
private RestTemplate restTemplate;
// Assume this is from Service A
#PostMapping(value = "/downloadresource",produces = MediaType.APPLICATION_OCTET_STREAM_VALUE)
public ResponseEntity<?> downloadByteResource() throws IOException{
ByteArrayResource responseObject;
HttpEntity httpEntity = new HttpEntity<>(new LinkedMultiValueMap<>());
responseObject= restTemplate.exchange("http://localhost:8080/test", HttpMethod.POST, httpEntity,
ByteArrayResource.class).getBody();
return ResponseEntity.ok().contentType(MediaType.APPLICATION_OCTET_STREAM)
.header(HttpHeaders.CONTENT_DISPOSITION,
"attachment; filename=test.zip")
.body(responseObject);
}
// Assume below from service B. which is not in my control
#PostMapping(value="/test")
public ResponseEntity<byte[]> test() throws IOException {
File f = new File("/Users/dummy/Downloads/test.zip");
byte[] b = Files.readAllBytes(f.toPath());
return ResponseEntity.ok().contentType(MediaType.APPLICATION_OCTET_STREAM)
.header(HttpHeaders.CONTENT_DISPOSITION,
"attachment; filename=" + f.getName())
.body(b);
}
}
I think i got some solution which is working with low memory as per visulaVM(to visualise heap memory).
Please let me know if any other better options.
#PostMapping(value = "/downloadextract",produces = MediaType.APPLICATION_OCTET_STREAM_VALUE)
public ResponseEntity<?> downloadExtract(HttpServletResponse response) throws IOException{
HttpEntity httpEntity = new HttpEntity<>(new LinkedMultiValueMap<>());
ResponseExtractor<Object> extractor = restClient -> {
StreamUtils.copy(restClient.getBody(), response.getOutputStream());
return null;
};
RequestCallback callback = req -> {
req.getHeaders().add("auth", "token");
};
response.setContentType(MediaType.APPLICATION_OCTET_STREAM_VALUE);
response.addHeader(HttpHeaders.CONTENT_DISPOSITION,
"attachment; filename=test.zip");
restTemplate.execute("http://localhost:8081/test", HttpMethod.POST, callback,
extractor);
return ResponseEntity.ok().build();
}
The UI for my webapp has the ability to either upload a file(csv), or send the data as json in request body. However either a file upload, or a json request would be present in the request and not both. I am creating a spring rest controller which combine file upload and also accepts the request json values as well.
With the below endpoint tested from postman, I am not getting exception:
org.apache.tomcat.util.http.fileupload.FileUploadException: the request was rejected because no multipart boundary was found
#RestController
public class MovieController {
private static final Logger LOGGER = LoggerFactory.getLogger(MovieController.class);
#PostMapping(value="/movies", consumes = {"multipart/form-data", "application/json"})
public void postMovies( #RequestPart String movieJson, #RequestPart(value = "moviesFile") MultipartFile movieFile ) {
// One of the below value should be present and other be null
LOGGER.info("Movies Json Body {}", movieJson);
LOGGER.info("Movies File Upload {}", movieFile);
}
}
Appreciate any help in getting this issue solved?
Note: I was able to build two separate endpoint for file upload and json request, but that won't suffice my requirement. Hence I'm looking for a solution to combine both
Try something like:
#RequestMapping(value = "/movies", method = RequestMethod.POST, consumes = { "multipart/form-data", "application/json" })
public void postMovies(
#RequestParam(value = "moviesFile", required = false) MultipartFile file,
UploadRequestBody request) {
In RequestBody you can add the parameters you want to send.
This will not send the data as JSON.
Edit:- I forgot to add the variable for the Multipart file and I mistakenly used the RequestBody which is reserved keyword in spring.
Hope it helps.
I would suggest to create two separate endpoints. This splits and isolates the different functionality and reduces the complexity of your code. In addition testing would be easier and provides better readability.
Your client actually has to know which variable to use. So just choose different endpoints for your request instead of using different variables for the same endpoint.
#PostMapping(value="/movies-file-upload", consumes = {"multipart/form-data"})
public void postMoviesFile(#RequestPart(value = "moviesFile") MultipartFile movieFile ) {
LOGGER.info("Movies File Upload {}", movieFile);
}
#PostMapping(value="/movies-upload", consumes = {"application/json"})
public void postMoviesJson( #RequestPart String movieJson) {
LOGGER.info("Movies Json Body {}", movieJson);
}
I have the below piece spring REST controller class.
#RestController
#RequestMapping("/global")
public class ProxyController extends BaseController{
#RequestMapping(value = "/**")
public ResponseEntity<String> proxy(HttpServletRequest request, HttpServletResponse response ) throws Exception {
try {
String restOfTheUrl = (String) request.getAttribute(
HandlerMapping.PATH_WITHIN_HANDLER_MAPPING_ATTRIBUTE);
URL uri = new URL("https://myrealserver" +
restOfTheUrl);
HttpHeaders headers = new HttpHeaders();
HttpEntity<String> httpEntity = new HttpEntity<>(headers);
RestTemplate restTemplate = new RestTemplate();
return resp;
} catch (Exception e) {
logger.error("Error ", e);
return new ResponseEntity<String>(HttpStatus.INTERNAL_SERVER_ERROR);
}
}
}
What I am trying to achieve here is to hide a server behind a proxy, which blindly forwards requests to the server.
This piece of code is invoked with url
https://myproxyserver/myapp1/end/point1
which in turn returns an html page with few clickable links. Now when the user clicks I am expecting the link to be invoked as
https://myproxyserver/myapp1/end/point2
Where as actually the endpoint invoked is
https://myproxyserver/end/point2
In the html page returned by the actual server, the path is end/point2 and has no mention of myapp1. So on click on those links my context changes to https://myproxyserver/end/point2 instead of https://myproxyserver/myapp1/end/point2
How do I ensure that the root context is always https://myproxyserver/myapp1 and not https://myproxyserver ?
You want to get your server context path. this is sample code.
like this :
public static String getServerNameAndContextPath(HttpServletRequest req) {
return "https://" + req.getServerName() + req.getContextPath();
}
Finally I resolved the problem by taking what D D suggested. I scanned through the whole response body, fortunately I had a pattern that I could use to scan and appended the context of the url to where ever required. That resolved the problem for me this problem.
I have implemented the below method to log response , but it throws an exception at IOUtils.toString(responseWrapper.getContentInputStream(), "UTF-8"); saying response has zero bytes . I am able to access headers and status. how can I get the response body?
#Override
public void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler,
ModelAndView modelAndView) throws Exception {
System.out.println("after Request");
ObjectMapper objectMapper = new ObjectMapper();
ContentCachingResponseWrapper responseWrapper = new ContentCachingResponseWrapper(response);
HttpStatus responseStatus = HttpStatus.valueOf(responseWrapper.getStatusCode());
System.out.println(responseWrapper.getStatusCode());
HttpHeaders responseHeaders = new HttpHeaders();
for (String headerName : responseWrapper.getHeaderNames()) {
System.out.println(headerName);
responseHeaders.add(headerName, responseWrapper.getHeader(headerName));
}
String responseBody = IOUtils.toString(responseWrapper.getContentInputStream(), "UTF-8");
System.out.println(responseBody);
log.info("Response body {} ",responseBody);
responseWrapper.copyBodyToResponse();
}
You never use response.getOutputStream() or response.getWriter() so no data is added to the response. That's why it is empty.
Your created entity
ResponseEntity<JsonNode> responseEntity = new ResponseEntity<>(responseJson,responseHeaders,responseStatus);
is never used and never sent to response. Try e.g.
responseWrapper.getWriter().write("test");
and check the body after.
How to read and copy the HTTP servlet response output stream content for logging
has your answer. The reason why getting your response output stream does not work is because the output stream gets written to and flushed regularly as you generate your output. This means that when you attempt to get it at the end, you just have an empty output. Alternatively, if you capture it at the beginning, that just means that your consumption of the outputstream denies the same consumption to your client. This is why you have to copy it as in the answer above.
My goal is to merge/minify all css files and return the result as String.
Here's my Spring test method :
#RequestMapping(value = "/stylesheet.css", method = RequestMethod.GET, produces = "text/css")
#ResponseBody
public void css(HttpServletResponse response) {
File path = new File(servletContext.getRealPath("/WEB-INF/includes/css/"));
File[] files = path.listFiles(...);
for (File file : files) {
InputStream is = new FileInputStream(file);
IOUtils.copy(is, response.getOutputStream());
response.flushBuffer();
is.close();
}
}
This is working with Chrome, Firefox and Safari but not with IE and Opera.
After some checks in the inspectors, the URL https://host/project/stylesheet.css is loading in each browsers. I can see the content but it does not seem to be recognized as text/css.
Also, even with produces = "text/css", I can not see the content-type http header in all browsers.
Error log in IE :
CSS ignored because of mime type incompatibility
Does anyone know how to correctly do this?
Working code :
#RequestMapping(value = "/stylesheet.css", method = RequestMethod.GET)
public ResponseEntity<Void> css(HttpServletResponse response) {
response.setContentType("text/css");
File path = new File(servletContext.getRealPath("/WEB-INF/includes/css/"));
File[] files = path.listFiles(...);
for (File file : files) {
InputStream is = new FileInputStream(file);
IOUtils.copy(is, response.getOutputStream());
IOUtils.closeQuietly(is);
}
response.flushBuffer();
return new ResponseEntity<Void>(HttpStatus.OK);
}
I suspect the problem is due to your usage of HttpServletResponse.flushBuffer().
As the API of HttpServletRequest states:
Forces any content in the buffer to be written to the client. A call
to this method automatically commits the response, meaning the status
code and headers will be written.
My assumption would be that Spring attempts to set the Content-Type header on the HttpServletResponse after the method on your controller has returned. However, because you have committed the response with your call to HttpServletResponse.flushBuffer(), it cannot do this.
I would try either:
Injecting the HttpServletResponse into your controller and setting the header yourself in code before calling HttpServletResponse.flushBuffer()
Removing your usage of HttpServletRequest.flushBuffer()
Since you're writing the content directly to the output stream, you don't need to use #ResponseBody. You just need to ensure that you set the Content-Type response header. Also, it'd be better to return a ResponseEntity (rather than void) to indicate to Spring that you're handling the response yourself.
#RequestMapping(value = "/stylesheet.css", method = RequestMethod.GET)
public ResponseEntity css(HttpServletResponse response) {
// Set the content-type
response.setHeader("Content-Type", "text/css");
File path = new File(servletContext.getRealPath("/WEB-INF/includes/css/"));
File[] files = path.listFiles(...);
for (File file : files) {
InputStream is = new FileInputStream(file);
IOUtils.copy(is, response.getOutputStream());
IOUtils.closeQuietly(is);
}
response.flushBuffer();
return new ResponseEntity(HttpStatus.OK)
}