I have a project where I want to sending a pdf file to an ftp server.
I am creating the file using pdfbox and changing it to an Inputstream and then I was to pass this input scream value to a remote FTP and save it as .pdf.
I have the below code but not sure how I can pass the data to the outbound adapter.
#Bean
public IntegrationFlow localToFtpFlow() {
return IntegrationFlows.from("toFtpChannel")
.handle(Ftp.outboundAdapter(sf())
.remoteDirectory("/ftp/forklift_checklist"))
.get();
}
#MessagingGateway
public interface MyGateway {
#Gateway(requestChannel = "toFtpChannel")
void sendToFtp(InputStream file);
}
Not sure what why is the question.
What you have so far is OK:
You call that sendToFtp() gateway's method with an InputStream for a local file.
The Ftp.outboundAdapter(sf() is based on the this.remoteFileTemplate.send(message, this.mode) operation which really supports an InputStream for a request payload:
else if (payload instanceof InputStream) {
return new StreamHolder((InputStream) payload, "InputStream payload");
}
So, share with us, please, what the problem are you observing with your configuration?
Perhaps you are looking into a fileName to give for that data while saving to FTP. Consider to have another gateway argument as a #Header(FileHeaders.FILENAME) String fileName. The RemoteFileTemplate relies on a DefaultFileNameGenerator which looks into that header by default.
Related
I have a requirement to read and write compressed (GZIP) streams without intermediate storage. Currently, I'm using Spring RestTemplate to do the writing, and Apache HTTP client to do the reading (see my answer here for an explanation of why RestTemplate can't be used for reading large streams). The implementation is fairly straightforward, where I slap a GZIPInputStream on the response InputStream and move on.
Now, I'd like to switch to using Spring 5 WebClient (just because I'm not a fan of status quo). However, WebClient is reactive in nature and deals with Flux<Stuff>; I believe it's possible to get a Flux<DataBuffer>, where DataBuffer is an abstraction over ByteBuffer. Question is, how do I decompress it on the fly without having to store the full stream in memory (OutOfMemoryError, I'm looking at you), or writing to local disk? It's worth mentioning that WebClient uses Netty under the hood.
Also see Reactor Netty issue-251.
Also related to Spring integration issue-2300.
I'll admit to not knowing much about (de)compression, however, I did my research, but none of the material available online seemed particularly helpful.
compression on java nio direct buffers
Writing GZIP file with nio
Reading a GZIP file from a FileChannel (Java NIO)
(de)compressing files using NIO
Iterable gzip deflate/inflate in Java
public class HttpResponseHeadersHandler extends ChannelInboundHandlerAdapter {
private final HttpHeaders httpHeaders;
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
if (msg instanceof HttpResponse &&
!HttpStatus.resolve(((HttpResponse) msg).status().code()).is1xxInformational()) {
HttpHeaders headers = ((HttpResponse) msg).headers();
httpHeaders.forEach(e -> {
log.warn("Modifying {} from: {} to: {}.", e.getKey(), headers.get(e.getKey()), e.getValue());
headers.set(e.getKey(), e.getValue());
});
}
ctx.fireChannelRead(msg);
}
}
Then I create a ClientHttpConnector to use with WebClient and in afterNettyContextInit add the handler:
ctx.addHandlerLast(new ReadTimeoutHandler(readTimeoutMillis, TimeUnit.MILLISECONDS));
ctx.addHandlerLast(new Slf4JLoggingHandler());
if (forceDecompression) {
io.netty.handler.codec.http.HttpHeaders httpHeaders = new ReadOnlyHttpHeaders(
true,
CONTENT_ENCODING, GZIP,
CONTENT_TYPE, APPLICATION_JSON
);
HttpResponseHeadersHandler headersModifier = new HttpResponseHeadersHandler(httpHeaders);
ctx.addHandlerFirst(headersModifier);
}
ctx.addHandlerLast(new HttpContentDecompressor());
This, of course, would fail for responses that are not GZIP compressed, so I use this instance of WebClient for a particular use case only, where I know for sure that the response is compressed.
Writing is easy: Spring has a ResourceEncoder, so InputStream can simply be converted to InputStreamResource, and voila!
Noting this here as it confused me a bit - the API has changed a bit as of 5.1.
I have a similar setup to the accepted answer for the ChannelInboundHandler:
public class GzipJsonHeadersHandler extends ChannelInboundHandlerAdapter {
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
if (msg instanceof HttpResponse
&& !HttpStatus.resolve(((HttpResponse) msg).status().code()).is1xxInformational()) {
HttpHeaders headers = ((HttpResponse) msg).headers();
headers.clear();
headers.set(HttpHeaderNames.CONTENT_ENCODING, HttpHeaderValues.GZIP);
headers.set(HttpHeaderNames.CONTENT_TYPE, HttpHeaderValues.APPLICATION_JSON);
}
ctx.fireChannelRead(msg);
}
}
(The header values I needed are just hard-coded there for simplicity, otherwise it's identical.)
To register it however is different:
WebClient.builder()
.clientConnector(
new ReactorClientHttpConnector(
HttpClient.from(
TcpClient.create()
.doOnConnected(c -> {
c.addHandlerFirst(new HttpContentDecompressor());
c.addHandlerFirst(new HttpResponseHeadersHandler());
})
).compress(true)
)
)
.build();
It seems Netty now maintains a user list of handlers separate from (and after) the system list, and addHandlerFirst() only puts your handler at the front of the user list. It therefore requires an explicit call to HttpContentDecompressor to ensure it's definitely executed after your handler that inserts the correct headers.
I have a large file download that is served by a RestController on one server, that I need to stream through a RestController on another server. When calling the end server directly the result streams fine. However when using RestTemplate to call this server and then write the response to an OutputStream, the response is buffered on the front server until the whole file is ready, and then streamed. Is there a way I can write the file to an OutputStream as it comes in?
At the moment my code on the front server looks similar to this
#ResponseBody
public void downloadResults(HttpServletRequest request, HttpServletResponse response, #RequestParam("id") String jobId, OutputStream stream)
throws IOException
{
byte[] data = restTemplate.exchange("http://localhost/getFile", HttpMethod.POST, requestEntity, byte[].class, parameters).getBody();
stream.write(data);
}
I've set my RestTemplate to not buffer and I've verified that this is working by checking the Request type that is used, (SimpleStreamingClientHttpRequest).
The data all comes back correct, its just only written to the stream all at once, rather than as it comes in
RestTemplate is not meant for streaming the response body, as pointed out in this JIRA issue.
You can use restTemplate.execute. See https://www.baeldung.com/spring-resttemplate-download-large-file
I'm trying to create an application that download and uploads large files, so I don't want the file contents to be stored in memory.
On the mvc controller side I'm using an http message converter that converts to / from InputStream
#Override
public InputStream read(Class<? extends InputStream> clazz, HttpInputMessage inputMessage) throws IOException,
HttpMessageNotReadableException {
return inputMessage.getBody();
}
#Override
public void write(InputStream t, MediaType contentType, HttpOutputMessage outputMessage) throws IOException,
HttpMessageNotWritableException {
try {
IOUtils.copy(t, outputMessage.getBody());
} finally {
IOUtils.closeQuietly(t);
}
}
This works well on the server side.
On the client (RestTemplate) side I tried to use the same converter, but I got an exception that the stream has been closed (probably closed when the request was completed).
Client side code:
ResponseEntity<InputStream> res = rest.getForEntity(url, InputStream.class);
// res.getBody() is closed
I've also tried to copy the input stream into a buffer and create a new ByteArrayInputStream and return it to the RestTemplate client and it worked well, however it does require that the data will be read into memory which doesn't suite my demands.
My question is how to keep the stream open until I process it without having to read it all into memory / file?
Any idea will be appreciated.
Regards, Shay
As far as I am aware, RestTemplate's getForEntity() is not an appropriate way to get an InputStream. It's a convenience for converting to and from entity classes, so presumably that's where your problem lies.
Since you are used to HttpInputMessage, why don't you use HttpInputMessage.getBody() on the client side as well? It gets you a nice InputStream, which would be ready for passing straight to an OutputStream such as HttpServletResponse.getOutputStream().
Check how Spring MVC handles large files upload with org.springframework.web.multipart.commons.CommonsMultipartResolver. It has a 'maxInMemorySize' that can help control the memory requirements. See this thread for using a multipart resolver with the REST template Sending Multipart File as POST parameters with RestTemplate requests
My grails 2.2.4 app needs to support accepting files over HTTP from a third party application and sending the file, after making some tweaks to it, back out as a response to the third party application.
I want to convert the data sent by the third party application to a file using InputStream and then send the file back out using OutputStream
So I built this code:
API Classes
class ApiResponse {
ApiMeta meta
ApiObject apiObj
}
class ApiMeta {
int code
String errorType
List msgs = []
}
class ApiObject {
OutputStream os
}
//Object Marshaller
JSON.registerObjectMarshaller( ApiObject ) { ApiObject obj ->
//How can I send output stream as json?
}
Controller
//controller
def save() {
request.withFormat {
json {
ApiResponse resp
//How can I convert the JSON data in params to a file?
response.status = 201
resp = new ApiResponse(
meta: new ApiMeta(code: 201),
apiObj: new ApiObject(os: transfers))
render resp as JSON
}
multipartForm {
}
}
Question
How can I convert JSON payload sent by the third party service into a file?
Is it ok to put OutputStream in my ApiObject class so that I can send the JSON payload back to the service?
My grails 2.2.4 app needs to support accepting InputStream over HTTP
from a third party application and sending OutputStream as a response
back.
That does not really make sense. Third party apps can't really send an InputStream to your app and your app can't really send an OutputStream back. The third party app can send you data in the body of the request and you can read the body by retrieving an InputStream from the request, and the same sort of thing could happen when you put data in the response. At first read I thought maybe you were just wording things in a way that doesn't make sense but then when I saw your domain class, that suggests that maybe you really are confused about how this works.
class Request {
InputStream instream
OutputStream outstream
static constraints = {
instream nullable: false, blank: false
}
}
You can't do that. You cannot persist an InputStream or an OutputStream to the database.
EDIT:
If you have a controller like this:
class MyController {
def someAction(Widget w) {
// do whatever you need to do with the Widget
}
}
class Widget {
String name
String category
}
And you send a request to that controller action with a JSON body which looks like this...
{"name":"Robert","category":"Prog Rocker"}
Grails will automatically read the body of the request and do the corresponding binding. You would never have to directly interact with any input stream to make that happen. Is that the sort of thing you are looking for?
I have a GWT page where user enter data (start date, end date, etc.), then this data goes to the server via RPC call. On the server I want to generate Excel report with POI and let user save that file on their local machine.
This is my test code to stream file back to the client but for some reason I think it does not know how to stream file to the client when I'm using RPC:
public class ReportsServiceImpl extends RemoteServiceServlet implements ReportsService {
public String myMethod(String s) {
File f = new File("/excelTestFile.xls");
String filename = f.getName();
int length = 0;
try {
HttpServletResponse resp = getThreadLocalResponse();
ServletOutputStream op = resp.getOutputStream();
ServletContext context = getServletConfig().getServletContext();
resp.setContentType("application/octet-stream");
resp.setContentLength((int) f.length());
resp.setHeader("Content-Disposition", "attachment; filename*=\"utf-8''" + filename + "");
byte[] bbuf = new byte[1024];
DataInputStream in = new DataInputStream(new FileInputStream(f));
while ((in != null) && ((length = in.read(bbuf)) != -1)) {
op.write(bbuf, 0, length);
}
in.close();
op.flush();
op.close();
}
catch (Exception ex) {
ex.printStackTrace();
}
return "Server says: " + filename;
}
}
I've read somewhere on internet that you can't do file stream with RPC and I have to use Servlet for that. Is there any example of how to use Servlet and how to call that servlet from ReportsServiceImpl. Do I really need to make a servlet or it is possible to stream it back with my RPC?
You have to make a regular Servlet, you cannot stream binary data from ReportsServiceImpl. Also, there is no way to call the servlet from ReportsServiceImpl - your client code has to directly invoke the servlet.
On the client side, you'd have to create a normal anchor link with the parameters passed via the query string. Something like <a href="http://myserver.com/myservlet?parm1=value1&.."</a>.
On the server side, move your code to a standard Servlet, one that does NOT inherit from RemoteServiceServlet. Read the parameters from the request object, create the excel and send it back to the client. The browser will automatically popup the file download dialog box.
You can do that just using GWT RPC and Data URIs:
In your example, make your myMethod return the file content.
On the client side, format a Data URI with the file content received.
Use Window.open to open a file save dialog passing the formatted DataURI.
Take a look at this reference, to understand the Data URI usage:
Export to csv in jQuery
It's possible to get the binary data you want back through the RPC channel in a number of ways... uuencode, for instance. However, you would still have to get the browser to handle the file as a download.
And, based on your code, it appears that you are trying to trigger the standard browser mechanism for handling the given mime-type by modifying the response in the server so the browser will recognize it as a download... open a save dialog, for instance. To do that, you need to get the browser to make the request for you and you need the servlet there to handle the request. It can be done with rest urls, but ultimately you will need a serviet to do even that.
You need, in effect, to set a browser window URL to the URL that sends back the modified response object.
So this question (about streaming) is not really compatible with the code sample. One or the other (communication protocols or server-modified response object) approach has to be adjusted.
The easiest one to adjust is the communication method.