We Keep Coding

Sending Inputstream in spring integration

I have a project where I want to sending a pdf file to an ftp server.
I am creating the file using pdfbox and changing it to an Inputstream and then I was to pass this input scream value to a remote FTP and save it as .pdf.
I have the below code but not sure how I can pass the data to the outbound adapter.
#Bean
public IntegrationFlow localToFtpFlow() {
return IntegrationFlows.from("toFtpChannel")
.handle(Ftp.outboundAdapter(sf())
.remoteDirectory("/ftp/forklift_checklist"))
.get();
}
#MessagingGateway
public interface MyGateway {
#Gateway(requestChannel = "toFtpChannel")
void sendToFtp(InputStream file);
}

Not sure what why is the question.
What you have so far is OK:
You call that sendToFtp() gateway's method with an InputStream for a local file.
The Ftp.outboundAdapter(sf() is based on the this.remoteFileTemplate.send(message, this.mode) operation which really supports an InputStream for a request payload:
else if (payload instanceof InputStream) {
return new StreamHolder((InputStream) payload, "InputStream payload");
}
So, share with us, please, what the problem are you observing with your configuration?
Perhaps you are looking into a fileName to give for that data while saving to FTP. Consider to have another gateway argument as a #Header(FileHeaders.FILENAME) String fileName. The RemoteFileTemplate relies on a DefaultFileNameGenerator which looks into that header by default.

Issues while File download

I'm trying to download a file from Angular UI, even after I got exception in backed code still I'm getting 200 ok as the response.
Here is the code I have :
public ResponseEntity<Object> downloadDocument(#PathVariable("docId") Long docId,
HttpServletResponse response) {
OutPutStream outputStream = null;
try {
outputStream = response.getOutputStream();
docService.downloadDocument(docId,outputStream);
return ResponseEntity.ok().contentType(MediaType.parseMediaType(MediaType.APPLICATION_OCTET_STREAM_VALUE))
.body("Success");
} catch(Exception e) {
return ResponseEntity.badRequest().body(e.getMessage());
} finally {
if (Objects.nonNull(outputStream)) {
IOUtils.closeQuietly(outputStream);
}
}
Can you please help me out what's wring here.

When you open the outputstream, the headers are sent. period.
There's no backtracking from that point on. You can't first open the outputstream and then later go: Oh, wait! No! nevermind! bad request!
Here's how it works - you pick a side and stick to it. You can either:
Handle it yourself; use response and the methods available there to set up your response; you can set headers, the return code and message, and you can obtain an outputstream for the response body, and send data that way. If you do this, you can't ALSO return a ResponseEntity!
Do NOT even add an HttpServletResponse parameter, and instead return a ResponseEntity object.
You're doing both, which is not allowed.
I'm frankly surprised; spring is a bit broken and ought to be throwing exceptions here, as it cannot possibly serve up what you're asking it to do here.
NB: Note that the type of an exception is usually more informative than the message (many exceptions don't even have a message).
Putting it all together:
public ResponseEntity<?> downloadDocument(#PathVariable("docId") Long docId) {
try {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
docService.downloadDocument(docId, baos);
return ResponseEntity.ok()
.contentType(MediaType.parseMediaType(MediaType.APPLICATION_OCTET_STREAM_VALUE))
.body(baos.toByteArray());
} catch(Exception e) {
return ResponseEntity.badRequest().body(e.getMessage());
}
}
'toString' defaults to printing its own type if there is no message, and its own type plus the message if there is, so you get e.g.:
NullPointerException
or
NullPointerException: parameter foo
which is what you want (versus a literal blank string in the former case, and just 'parameter foo' in the latter, which isn't particularly insightful either).
messages are generally intended not to necessarily make sense without the context of the type of the exception.
NB: This will cache the entire content of the downloaded document into the memory of the server before sending it onwards. If the document is large, this is a bad idea, but, if you want to 'stream' it, you have a pretty serious problem inherent in the HTTP protocol: Once you start sending, you've already sent the 'error status' (i.e. you already sent 200 OK), so if the document download process throws an exception halfway through, you cannot go back and send an error message. You need some sort of wire protocol where you send the body in chunks and have a 'type' code you send, so that the recipient can scan for these and knows 'this type means there's more data flowing', and 'this type means an error occured and now I can read the error description'. That all gets quite complicated. By caching it, you avoid this mess. But if that document can be very large you're going to have to deal with the above, there are no quick fixes.

How to decompress a Flux<DataBuffer> (and how to write one)?

I have a requirement to read and write compressed (GZIP) streams without intermediate storage. Currently, I'm using Spring RestTemplate to do the writing, and Apache HTTP client to do the reading (see my answer here for an explanation of why RestTemplate can't be used for reading large streams). The implementation is fairly straightforward, where I slap a GZIPInputStream on the response InputStream and move on.
Now, I'd like to switch to using Spring 5 WebClient (just because I'm not a fan of status quo). However, WebClient is reactive in nature and deals with Flux<Stuff>; I believe it's possible to get a Flux<DataBuffer>, where DataBuffer is an abstraction over ByteBuffer. Question is, how do I decompress it on the fly without having to store the full stream in memory (OutOfMemoryError, I'm looking at you), or writing to local disk? It's worth mentioning that WebClient uses Netty under the hood.
Also see Reactor Netty issue-251.
Also related to Spring integration issue-2300.
I'll admit to not knowing much about (de)compression, however, I did my research, but none of the material available online seemed particularly helpful.
compression on java nio direct buffers
Writing GZIP file with nio
Reading a GZIP file from a FileChannel (Java NIO)
(de)compressing files using NIO
Iterable gzip deflate/inflate in Java

public class HttpResponseHeadersHandler extends ChannelInboundHandlerAdapter {
private final HttpHeaders httpHeaders;
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
if (msg instanceof HttpResponse &&
!HttpStatus.resolve(((HttpResponse) msg).status().code()).is1xxInformational()) {
HttpHeaders headers = ((HttpResponse) msg).headers();
httpHeaders.forEach(e -> {
log.warn("Modifying {} from: {} to: {}.", e.getKey(), headers.get(e.getKey()), e.getValue());
headers.set(e.getKey(), e.getValue());
});
}
ctx.fireChannelRead(msg);
}
}
Then I create a ClientHttpConnector to use with WebClient and in afterNettyContextInit add the handler:
ctx.addHandlerLast(new ReadTimeoutHandler(readTimeoutMillis, TimeUnit.MILLISECONDS));
ctx.addHandlerLast(new Slf4JLoggingHandler());
if (forceDecompression) {
io.netty.handler.codec.http.HttpHeaders httpHeaders = new ReadOnlyHttpHeaders(
true,
CONTENT_ENCODING, GZIP,
CONTENT_TYPE, APPLICATION_JSON
);
HttpResponseHeadersHandler headersModifier = new HttpResponseHeadersHandler(httpHeaders);
ctx.addHandlerFirst(headersModifier);
}
ctx.addHandlerLast(new HttpContentDecompressor());
This, of course, would fail for responses that are not GZIP compressed, so I use this instance of WebClient for a particular use case only, where I know for sure that the response is compressed.
Writing is easy: Spring has a ResourceEncoder, so InputStream can simply be converted to InputStreamResource, and voila!

Noting this here as it confused me a bit - the API has changed a bit as of 5.1.
I have a similar setup to the accepted answer for the ChannelInboundHandler:
public class GzipJsonHeadersHandler extends ChannelInboundHandlerAdapter {
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
if (msg instanceof HttpResponse
&& !HttpStatus.resolve(((HttpResponse) msg).status().code()).is1xxInformational()) {
HttpHeaders headers = ((HttpResponse) msg).headers();
headers.clear();
headers.set(HttpHeaderNames.CONTENT_ENCODING, HttpHeaderValues.GZIP);
headers.set(HttpHeaderNames.CONTENT_TYPE, HttpHeaderValues.APPLICATION_JSON);
}
ctx.fireChannelRead(msg);
}
}
(The header values I needed are just hard-coded there for simplicity, otherwise it's identical.)
To register it however is different:
WebClient.builder()
.clientConnector(
new ReactorClientHttpConnector(
HttpClient.from(
TcpClient.create()
.doOnConnected(c -> {
c.addHandlerFirst(new HttpContentDecompressor());
c.addHandlerFirst(new HttpResponseHeadersHandler());
})
).compress(true)
)
)
.build();
It seems Netty now maintains a user list of handlers separate from (and after) the system list, and addHandlerFirst() only puts your handler at the front of the user list. It therefore requires an explicit call to HttpContentDecompressor to ensure it's definitely executed after your handler that inserts the correct headers.

Stream file download through RestTemplate

I have a large file download that is served by a RestController on one server, that I need to stream through a RestController on another server. When calling the end server directly the result streams fine. However when using RestTemplate to call this server and then write the response to an OutputStream, the response is buffered on the front server until the whole file is ready, and then streamed. Is there a way I can write the file to an OutputStream as it comes in?
At the moment my code on the front server looks similar to this
#ResponseBody
public void downloadResults(HttpServletRequest request, HttpServletResponse response, #RequestParam("id") String jobId, OutputStream stream)
throws IOException
{
byte[] data = restTemplate.exchange("http://localhost/getFile", HttpMethod.POST, requestEntity, byte[].class, parameters).getBody();
stream.write(data);
}
I've set my RestTemplate to not buffer and I've verified that this is working by checking the Request type that is used, (SimpleStreamingClientHttpRequest).
The data all comes back correct, its just only written to the stream all at once, rather than as it comes in

RestTemplate is not meant for streaming the response body, as pointed out in this JIRA issue.

You can use restTemplate.execute. See https://www.baeldung.com/spring-resttemplate-download-large-file

Large file transfer from HTTP server running Java Jersey Rest API

I am writing a web service using Java JDK 1.7 and Jersey Web service Framework. One of the things I need to provide is a way to allow authenticated clients to download certain large data files ( 1-3 GB). Ideally I would like this to be a pause and resume type downloadable option. I tried the jersey multi-part API and was able to get it to work on my client machine upto 400 MB but beyond that it ran into out-of memory issues. I am also worried that the server might fail when faced with simultaneous download requests. Any thoughts on how this can be done? Is Netty an option? Any pointers on how Netty can be integrated into a existing Jersey based web service? Are there other frame works available to help accomplish this? I do have to use java for the web service. Any pointers will be helpful.

If you are getting stuck on out-of-memory issues, you should check how you are handling the data you are downloading. If you are using Jersey's ClientResponse, make sure you are using getEntityInputStream() and not getEntity(). This way, you can stream the data, write it to file, and toss it aside, rather than letting it build up in the Java heap space.
I can't really speak about your simultaneous download concerns, but if you are using the web services framework, then it should be handled properly.
For both issues, more info on your specific implementation, especially code, will help you get a better response.

The server and the client must both support HTTP chunked encoding which allows one to stream data using HTTP. The code below should work with Jersey 2.11.
For downloading large files, try this on the server:
#GET
#Path("/files/{fileName}")
#Produces(MediaType.APPLICATION_OCTET_STREAM)
public StreamingOutput getFile(#PathParam("fileName") final String fileName) throws Exception {
//create instance of StreamingOutput here
return streamingOutput;
}
Try this for a client GET request using steams to download a file.
public String getFileReq(File outFile) throws IOException {
client = ClientBuilder.newClient(new ClientConfig());
client.property(ClientProperties.REQUEST_ENTITY_PROCESSING, "CHUNKED");
WebTarget target = client.target(URI)
OutputStream fileOutputStream = new FileOutputStream(outFile);
InputStream fileInputStream = target.request().get(InputStream.class);
writeFile(fileInputStream, fileOutputStream);
}
public static void writeFile(InputStream fileInputStream, OutputStream outputStream) throws IOException {
try {
byte[] buffer = new byte[1024];
int bytesRead;
while((bytesRead = fileInputStream.read(buffer)) !=-1) {
outputStream.write(buffer, 0, bytesRead);
}
fileInputStream.close();
outputStream.flush();
} catch (IOException e) {
e.printStackTrace();
} finally {
outputStream.close();
}