We Keep Coding

How to integrate a MessageHandler into the SFTP scenario based on SftpRemoteFileTemplate?

I have implemented a service of getting a file from, putting a file to, and removing a file from the SFTP server based on the SftpRemoteFileTemplate within Spring's Integration Package.
Here sftpGetPayload gets a file from the SFTP server and delivers its content.
This is my code so far:
public String sftpGetPayload(String sessionId,
String host, int port, String user, String password,
String remoteDir, String remoteFilename, boolean remoteRemove) {
LOG.info("sftpGetPayload sessionId={}", sessionId);
LOG.debug("sftpGetPayLoad host={}, port={}, user={}", host, port, user);
LOG.debug("sftpGetPayload remoteDir={}, remoteFilename={}, remoteRemove={}",
remoteDir, remoteFilename, remoteRemove);
final AtomicReference<String> refPayload = new AtomicReference<>();
SftpRemoteFileTemplate template = getSftpRemoteFileTemplate(host, port,
user, password, remoteDir, remoteFilename);
template.get(remoteDir + "/" + remoteFilename,
is -> refPayload.set(getAsString(is)));
LOG.info("sftpGetToFile {} read.", remoteDir + "/" + remoteFilename);
deleteRemoteFile(template, remoteDir, remoteFilename, remoteRemove);
return refPayload.get();
}
private SftpRemoteFileTemplate getSftpRemoteFileTemplate(String host, int port,
String user, String password, String remoteDir, String remoteFilename) {
SftpRemoteFileTemplate template =
new SftpRemoteFileTemplate(sftpSessionFactory(host, port, user, password));
template.setFileNameExpression(
new LiteralExpression(remoteDir + "/" + remoteFilename));
template.setRemoteDirectoryExpression(new LiteralExpression(remoteDir));
//template.afterPropertiesSet();
return template;
}
private void deleteRemoteFile(SftpRemoteFileTemplate template,
String remoteDir, String remoteFilename, boolean remoteRemove) {
LOG.debug("deleteRemoteFile remoteRemove={}", remoteRemove);
if (remoteRemove) {
template.remove(remoteDir + "/" + remoteFilename);
LOG.info("sftpGetToFile {} removed.", remoteDir + "/" + remoteFilename);
}
}
All those GET actions are active actions, meaning the file to get is considered to be already there. I would like to have a kind of a polling process, which calls my payload consuming method as soon as a file is received on the SFTP server.
I have found another implementation based on Spring beans, configured as Spring Integration Dsl, which declares a SftpSessionFactory, a SftpInboundFileSynchronizer, a SftpMessageSource, and a MessageHandler which polls a SFTP site for reception of a file and initiates a message handler automatically for further processing.
This code is as follows:
#Bean
public SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(myHost);
factory.setPort(myPort);
factory.setUser(myUser);
factory.setPassword(myPassword);
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<LsEntry>(factory);
}
#Bean
public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
SftpInboundFileSynchronizer fileSynchronizer = new SftpInboundFileSynchronizer(sftpSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setRemoteDirectory(myRemotePath);
fileSynchronizer.setFilter(new SftpSimplePatternFileListFilter(myFileFilter));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "sftpChannel", poller = #Poller(fixedDelay = "5000"))
public MessageSource<File> sftpMessageSource() {
SftpInboundFileSynchronizingMessageSource source = new SftpInboundFileSynchronizingMessageSource(
sftpInboundFileSynchronizer());
source.setLocalDirectory(myLocalDirectory);
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
return source;
}
#Bean
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
System.out.println(message.getPayload());
}
};
}
How can I include this #Poller/MessageHandler/#ServiceActivator concept into my implementation above? Or is there a way to implement this feature in the template-based implementation?
The scenario could be following:
I have a Spring Boot Application with several classes which represent tasks. Some of those tasks are called automatically via the Spring #Scheduled annotation and a CRON specification, others are not.
#Scheduled(cron = "${task.to.start.automatically.frequency}")
public void runAsTask() {
...
}
First task will start at ist #Sheduled specification and get a file from SFTP server and process it. It will do that with its own channel (host1, port1, user1, password1, remoteDir1, remoteFile1).
Second task will also be run by the scheduler and generate something to put to the SFTP server. It will do that with its own channel (host2, port2, user2, password2, remoteDir2, remoteFile2). Very likely will host2 = host1 and port2 = port1, but it is not a must.
Third task will aslo be run by the scheduler and generate something to put to the SFTP server. It will do that with the same channel as task1, but this task is a producer (not a consumer like task1) and will write another file than task1 (host1, port1, user1, password1, remoteDir3, remoteFile3).
Task four has no #Scheduled annotation because it should realize when the file, it has to process, is received from third party and hence available on its channel (host4, port4, user4, password4, remoteDir4, remoteFile4) to get its content to process it.
I have read the whole Integration stuff, but it is hard to transform for this use case, either from the XML configuration schemes to Java configuration with annotations and also by the reather static Spring bean approach to a merly dynamic approach at runtime.
I understood to use an IntegrationFlow to register the artefacts, an inbound adapter for task1, an outbound adapter for task2, an inbound adapter for task3 with the same (anywhere else registrated) session factory of task1, and - last but not least - an inbound adapter with poller feature for task4.
Or should all of them be gateways with its command feature? Or should I register SftpRemoteFileTemplate?
To define the channel I have:
public class TransferChannel {
private String host;
private int port;
private String user;
private String password;
/* getters, setters, hash, equals, and toString */
}
To have all SFTP settings together, I have:
public class TransferContext {
private boolean enabled;
private TransferChannel channel;
private String remoteDir;
private String remoteFilename;
private boolean remoteRemove;
private String remoteFilenameFilter;
private String localDir;
/* getters, setters, hash, equals, and toString */
}
As the heart of the SFTP processing each job will inject kind of a DynamicSftpAdapter:
#Scheduled(cron = "${task.to.start.automatically.frequency}")
public void runAsTask() {
#Autowired
DynamicSftpAdapter sftp;
...
sftp.connect("Task1", context);
File f = sftp.getFile("Task1", "remoteDir", "remoteFile");
/* process file content */
sftp.removeFile("Task1", "remoteDir", "remoteFile");
sftp.disconnect("Task1", context);
}
The DynamicSftpAdapter is not much more than a fragment yet:
#Component
public class DynamicSftpAdapter {
private static final Logger LOG = LoggerFactory.getLogger(DynamicTcpServer.class);
#Autowired
private IntegrationFlowContext flowContext;
#Autowired
private ApplicationContext appContext;
private final Map<TransferChannel, IntegrationFlowRegistration> registrations = new HashMap<>();
private final Map<String, TransferContext> sessions = new ConcurrentHashMap<>();
#Override
public void connect(String sessionId, TransferContext context) {
if (this.registrations.containsKey(context.getChannel())) {
LOG.debug("connect, channel exists for {}", sessionId);
}
else {
// register the required SFTP Outbound Adapter
TransferChannel channel = context.getChannel();
IntegrationFlow flow = f -> f.handle(Sftp.outboundAdapter(cashedSftpSessionFactory(
channel.getHost(), channel.getPort(),
channel.getUser(), channel.getPassword())));
this.registrations.put(channel, flowContext.registration(flow).register());
this.sessions.put(sessionId, context);
LOG.info("sftp session {} for {} started", sessionId, context);
}
}
private DefaultSftpSessionFactory sftpSessionFactory(String host, int port, String user, String password) {
LOG.debug("sftpSessionFactory");
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(host);
factory.setPort(port);
factory.setUser(user);
factory.setPassword(password);
factory.setAllowUnknownKeys(true);
return factory;
}
private CachingSessionFactory<LsEntry> cashedSftpSessionFactory(String host, int port, String user, String password) {
LOG.debug("cashedSftpSessionFactory");
CachingSessionFactory<LsEntry> cashedSessionFactory =
new CachingSessionFactory<LsEntry>(
sftpSessionFactory(host, port, user, password));
return cashedSessionFactory;
}
#Override
public void sftpGetFile(String sessionId, String remoteDir, String remoteFilename) {
TransferContext context = sessions.get(sessionId);
if (context == null)
throw new IllegalStateException("Session not established, sessionId " + sessionId);
IntegrationFlowRegistration register = registrations.get(context.getChannel());
if (register != null) {
try {
LOG.debug("sftpGetFile get file {}", remoteDir + "/" + remoteFilename);
register.getMessagingTemplate().send(
MessageBuilder.withPayload(msg)
.setHeader(...).build());
}
catch (Exception e) {
appContext.getBean(context, DefaultSftpSessionFactory.class)
.close();
}
}
}
#Override
public void disconnect(String sessionId, TransferContext context) {
IntegrationFlowRegistration registration = this.registrations.remove(context.getChannel());
if (registration != null) {
registration.destroy();
}
LOG.info("sftp session for {} finished", context);
}
}
I did not get how to initiate a SFTP command. I also did not get when using an OutboundGateway and having to specify the SFTP command (like GET) instantly, then would the whole SFTP handling be in one method, specifying the outbound gateway factory and getting an instance with get() and probably calling the message .get() in any way.
Obviously I need help.

First of all if you already use Spring Integration channel adapters, there is probably no reason to use that low-level API like RemoteFileTemplate directly.
Secondly there is a technical discrepancy: the SftpInboundFileSynchronizingMessageSource will produce a local file - a whole copy of the remote file. So, when we would come to your SftpRemoteFileTemplate logic downstream it would not work well since we would bring already just a local file (java.io.File), not an entity for remote file representation.
Even if your logic in the sftpGetPayload() doesn't look as complicated and custom as it would require such a separate method, it is better to have an SftpRemoteFileTemplate as a singleton and share it between different components when you work against the same SFTP server. It is just stateless straightforward Spring template pattern implementation.
If you still insist to use your method from the mentioned integration flow, you should consider to have a POJO method call for that #ServiceActivator(inputChannel = "sftpChannel"). See more in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/configuration.html#annotations.
You also may find an SFTP Outbound Gateway as useful component for your use-case. It has some common scenarios implementations: https://docs.spring.io/spring-integration/docs/current/reference/html/sftp.html#sftp-outbound-gateway

Download Attachments with Spring Integration and POP3

I have a Spring Boot project that is leveraging Spring Integration. My goal is to to poll a POP3 mail server regularly, and download any attachments associated with those messages. My relevant Spring Config looks like this:
#Configuration
public class MailIntegrationConfig {
#Value("${path.output.temp}")
private String outPath;
#Bean
public MessageChannel mailChannel() {
return new DirectChannel();
}
#Bean
#InboundChannelAdapter(value = "mailChannel", poller = #Poller(fixedDelay = "16000"))
public MessageSource<Object> fileReadingMessageSource() {
var receiver = new Pop3MailReceiver("pop3s://user:passwordexample.com/INBOX");
var mailProperties = new Properties();
mailProperties.setProperty("mail.pop3.port", "995");
mailProperties.put("mail.pop3.ssl.enable", true);
receiver.setShouldDeleteMessages(false);
receiver.setMaxFetchSize(10);
receiver.setJavaMailProperties(mailProperties);
// receiver.setHeaderMapper(new DefaultMailHeaderMapper());
var source = new MailReceivingMessageSource(receiver);
return source;
}
#Bean
#ServiceActivator(inputChannel = "mailChannel")
public MessageHandler popMessageHandler() {
return new MailReceivingMessageHandler(outPath);
}
}
My MailReceivingMessageHandler class (partial)
public class MailReceivingMessageHandler extends AbstractMessageHandler {
private String outDir;
public MailReceivingMessageHandler(String outDir) {
var outPath = new File(outDir);
if (!outPath.exists()) {
throw new IllegalArgumentException(String.format("%s does not exist.", outDir));
}
this.outDir = outDir;
}
#Override
protected void handleMessageInternal(org.springframework.messaging.Message<?> message) {
Object payload = message.getPayload();
if (!(payload instanceof Message)) {
throw new IllegalArgumentException(
"Unable to create MailMessage from payload type [" + message.getPayload().getClass().getName()
+ "], " + "expected MimeMessage, MailMessage, byte array or String.");
}
try {
var msg = (Message) payload;
System.out.println(String.format("Headers [%s] Subject [%s]. Content-Type [%s].", msg.getAllHeaders(),
msg.getSubject(), msg.getContentType()));
this.handleMessage(msg);
} catch (IOException | MessagingException e) {
e.printStackTrace();
}
}
private void handleMessage(Message msg) throws MessagingException, IOException {
var cType = msg.getContentType();
if (cType.contains(MediaType.TEXT_PLAIN_VALUE)) {
handleText((String) msg.getContent());
} else if (cType.contains(MediaType.MULTIPART_MIXED_VALUE)) {
handleMultipart((Multipart) msg.getContent());
}
}
// See
// https://stackoverflow.com/questions/1748183/download-attachments-using-java-mail
private void handleMultipart(Multipart msgContent) throws MessagingException, IOException {
var mCount = msgContent.getCount();
for (var i = 0; i < mCount; i++) {
this.processAttachments(msgContent.getBodyPart(i));
}
}
private void processAttachments(BodyPart part) throws IOException, MessagingException {
var content = part.getContent();
if (content instanceof InputStream || content instanceof String) {
if (Part.ATTACHMENT.equalsIgnoreCase(part.getDisposition()) || !part.getFileName().isBlank()) {
var fName = String.format("%s.%s", UUID.randomUUID().toString(),
FilenameUtils.getExtension(part.getFileName()));
FileUtils.copyInputStreamToFile(part.getInputStream(), new File(outDir + File.separator + fName));
}
if (content instanceof Multipart) {
Multipart multipart = (Multipart) content;
for (int i = 0; i < multipart.getCount(); i++) {
var bodyPart = multipart.getBodyPart(i);
processAttachments(bodyPart);
}
}
}
}
}
Whenever I run my code using the config above, I receive the following error:
javax.mail.MessagingException: No inputstream from datasource;
nested exception is:
java.lang.IllegalStateException: Folder is not Open
at javax.mail.internet.MimeMultipart.parse(MimeMultipart.java:576)
at javax.mail.internet.MimeMultipart.getCount(MimeMultipart.java:312)
at com.midamcorp.data.mail.MailReceivingMessageHandler.handleMultipart(MailReceivingMessageHandler.java:70)
at com.midamcorp.data.mail.MailReceivingMessageHandler.handleMessage(MailReceivingMessageHandler.java:58)
at com.midamcorp.data.mail.MailReceivingMessageHandler.handleMessageInternal(MailReceivingMessageHandler.java:44)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:62)
at org.springframework.integration.handler.ReplyProducingMessageHandlerWrapper.handleRequestMessage(ReplyProducingMessageHandlerWrapper.java:58)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:134)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:62)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:570)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:520)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
at org.springframework.integration.endpoint.SourcePollingChannelAdapter.handleMessage(SourcePollingChannelAdapter.java:196)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.messageReceived(AbstractPollingEndpoint.java:444)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:428)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.pollForMessage(AbstractPollingEndpoint.java:376)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$null$3(AbstractPollingEndpoint.java:323)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.lambda$execute$0(ErrorHandlingTaskExecutor.java:57)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:55)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$createPoller$4(AbstractPollingEndpoint.java:320)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Folder is not Open
at com.sun.mail.pop3.POP3Folder.checkOpen(POP3Folder.java:562)
at com.sun.mail.pop3.POP3Folder.getProtocol(POP3Folder.java:592)
at com.sun.mail.pop3.POP3Message.getRawStream(POP3Message.java:154)
at com.sun.mail.pop3.POP3Message.getContentStream(POP3Message.java:251)
at javax.mail.internet.MimePartDataSource.getInputStream(MimePartDataSource.java:78)
at javax.mail.internet.MimeMultipart.parse(MimeMultipart.java:570)
... 35 more
Obviously, the root cause is clear - the POP3 folder is closed. I have seen solutions that would likely be able to handle when just the Java mail classes are used, but none with Spring Integration. My question is how does one properly control when a folder is open or closed using Spring Integration Mail? I realize the Pop3MailReceiver class has a .setAutoCloseFolder() method. Based on the Spring Docs, I assume I need to set that, along something like the following to my handler:
Closeable closeableResource = StaticMessageHeaderAccessor.getCloseableResource(message);
if (closeableResource != null) {
closeableResource.close();
}
However, if I set autoCloseFolder to false, it does not appear as if the message even ever "hits" my handler, so unfortunately being able to close the resource does not even matter at this point. That is, when autoClose is set to false, the 'handleMessageInternal()' method in my handler class is never reached even though there are indeed message on the POP3 server. Instead I just get a bunch of logs like this:
2020-06-26 15:26:54.523 INFO 15348 --- [ scheduling-1] o.s.integration.mail.Pop3MailReceiver : attempting to receive mail from folder [INBOX]
What am I missing?
Thanks.

How to get the error message in Controller or Route after handling exception in errorHandler?

I had to customized Sftp Inbound default handler LoggingHandler and using my own CustomizedErrorHandler which extends ErrorHandler. But I can't return any message to my controller after handling exceptions.
I was researching couple of days and I found nothing to show my customized message to my UI using Controller. Below are some code snippet from my CustomizedErrorHandler, SftpInboundConfiguration.
SftpInboundConfiguration
public IntegrationFlow fileFlow() {
SftpInboundChannelAdapterSpec spec = Sftp
.inboundAdapter(getSftpSessionFactory())
.preserveTimestamp(true)
.remoteDirectory(getSourceLocation())
.autoCreateLocalDirectory(true)
.deleteRemoteFiles(false)
.localDirectory(new File(getDestinationLocation()));
return IntegrationFlows
.from(spec, e -> e.id(BEAN_ID)
.autoStartup(false)
.poller(sftpPoller())
)
.channel(sftpReceiverChannel())
.handle(sftpInboundMessageHandler())
.get();
}
... ... ...
public PollerMetadata sftpPoller() {
PollerMetadata pollerMetadata = new PollerMetadata();
List<Advice> adviceChain = new ArrayList<>();
pollerMetadata.setErrorHandler(customErrorMessageHandler());
pollerMetadata.setTrigger(new PeriodicTrigger(5000));
return pollerMetadata;
}
... ... ...
private CustomErrorMessageHandler customErrorMessageHandler() {
return new CustomErrorMessageHandler(
controlChannel(),
BEAN_ID
);
}
CustomErrorMessageHandler
public class CustomErrorMessageHandler implements ErrorHandler {
private final MessageChannel CONTROL_CHANNEL;
private final String BEAN_ID;
public CustomErrorMessageHandler(
MessageChannel controlChannel,
String beanID
) {
this.CONTROL_CHANNEL = controlChannel;
this.BEAN_ID = beanID;
}
public void handleError(#NotNull Throwable throwable) {
final Throwable rootCause = ExceptionUtils.getRootCause(throwable);
if (rootCause instanceof MessagingException) {
log.error("MessagingException : {} ", rootCause.getMessage());
} else if (rootCause instanceof SftpException) {
log.error("SftpException : {}", rootCause.getMessage());
} ... ... ...
else {
log.error("Unknown : Cause : {} , Error : {}",
rootCause, rootCause.getMessage());
}
log.info("Stopping SFTP Inbound");
boolean is_stopped = CONTROL_CHANNEL.send(
new GenericMessage<>("#" + BEAN_ID + ".stop()"));
if (is_stopped) {
log.info("SFTP Inbound Stopped.");
} else {
log.info("SFTP Inbound Stop Failed.");
}
}
}
Now I want to save some customized message from if-else statements and need to show it in UI. Is there any way to save the message and show it using Route or Controller ?

Don't customize the error handler, use poller.errorChannel("myErrorChannel") instead.
Then add an error channel flow
#Bean
IntegrationFlow errors() {
return IntegrationFLows.from("myErrorChannel")
.handle(...)
...
.get();
The message sent to the handler is an ErrorMessage with a MessagingException payload, with cause and failedMessage which was the message at the point of the failure and originalMessage which is the original message emitted by the adapter.
After handling the exception, you can simply call a method on your controller to tell it the state.

How to handle JmsChannelFactoryBean errors, is there a possibility of custom error channel usage?

I have following configuration for creation of two channels (by using the JmsChannelFactoryBean):
#Bean
public JmsChannelFactoryBean jmsChannel(ActiveMQConnectionFactory activeMQConnectionFactory) {
JmsChannelFactoryBean fb = new JmsChannelFactoryBean(true);
fb.setConnectionFactory(activeMQConnectionFactory);
fb.setDestinationName("something.queue");
fb.setErrorHandler(t -> log.error("something went wrong on jms channel", t));
return fb;
}
#Bean
public JmsChannelFactoryBean jmsChannelDLQ(ActiveMQConnectionFactory activeMQConnectionFactory) {
JmsChannelFactoryBean fb = new JmsChannelFactoryBean(true);
fb.setConnectionFactory(activeMQConnectionFactory);
fb.setDestinationName("something.queue.DLQ");
fb.setErrorHandler(t -> log.error("something went wrong on jms channel", t));
return fb;
}
The something.queue is configured to put the dead letter on something.queue.DLQ. Im using mostly Java DSL to configure the app, and if possible - would like to keep this.
Case is: the message is taken from jmsChannel put to sftp outbound gateway, if there is a problem on sending the file, the message is put back into the jmsChannel as not delivered. After some retries it is designed as poisonus, and put to something.queue.DLQ.
Is it possbile to have the info on error channel when that happens?
What is best practice to handle errors when using JMS backed message channels?
EDIT 2
The integration flow is defined as:
IntegrationFlows.from(filesToProcessChannel).handle(outboundGateway)
Where filesToProcessChannel is the JMS backed channel and outbound gateway is defined as:
#Bean
public SftpOutboundGateway outboundGateway(SftpRemoteFileTemplate sftpRemoteFileTemplate) {
SftpOutboundGateway gateway = new SftpOutboundGateway(sftpRemoteFileTemplate, AbstractRemoteFileOutboundGateway.Command.PUT.getCommand(), EXPRESSION_PAYLOAD);
ArrayList<Advice> adviceChain = new ArrayList<>();
adviceChain.add(errorHandlingAdvice());
gateway.setAdviceChain(adviceChain);
return gateway;
}
Im trying to grab exception using advice:
#Bean
public Advice errorHandlingAdvice() {
RequestHandlerRetryAdvice advice = new RequestHandlerRetryAdvice();
RetryTemplate retryTemplate = new RetryTemplate();
SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy();
retryPolicy.setMaxAttempts(1);
retryTemplate.setRetryPolicy(retryPolicy);
advice.setRetryTemplate(retryTemplate);
advice.setRecoveryCallback(new ErrorMessageSendingRecoverer(filesToProcessErrorChannel));
return advice;
}
Is this the right way?
EDIT 3
There is certanly something wrong with SFTPOutboundGateway and advices (or with me :/):
I used the folowing advice from the spring integration reference:
#Bean
public Advice expressionAdvice() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setSuccessChannelName("success.input");
advice.setOnSuccessExpressionString("payload + ' was successful'");
advice.setFailureChannelName("failure.input");
advice.setOnFailureExpressionString(
"payload + ' was bad, with reason: ' + #exception.cause.message");
advice.setTrapException(true);
return advice;
}
#Bean
public IntegrationFlow success() {
return f -> f.handle(System.out::println);
}
#Bean
public IntegrationFlow failure() {
return f -> f.handle(System.out::println);
}
And when I use :
return IntegrationFlows.from(filesToProcessChannel)
.handle((GenericHandler<File>) (payload, headers) -> {
if (payload.equals("x")) {
return null;
}
else {
throw new RuntimeException("some failure");
}
}, spec -> spec.advice(expressionAdvice()))
It gets called, and i get error message printed out (and that is expected), but when I try to use:
return IntegrationFlows.from(filesToProcessChannel)
.handle(outboundGateway, spec -> spec.advice(expressionAdvice()))
The advice is not called, and the error message is put back to JMS.
The app is using Spring Boot v2.0.0.RELEASE, Spring v5.0.4.RELEASE.
EDIT 4
I managed to resolve the advice issue using following configuration, still don't understand why the handler spec will not work:
#Bean
IntegrationFlow files(SftpOutboundGateway outboundGateway,
...
) {
return IntegrationFlows.from(filesToProcessChannel)
.handle(outboundGateway)
...
.log(LoggingHandler.Level.INFO)
.get();
}
#Bean
public SftpOutboundGateway outboundGateway(SftpRemoteFileTemplate sftpRemoteFileTemplate) {
SftpOutboundGateway gateway = new SftpOutboundGateway(sftpRemoteFileTemplate, AbstractRemoteFileOutboundGateway.Command.PUT.getCommand(), EXPRESSION_PAYLOAD);
ArrayList<Advice> adviceChain = new ArrayList<>();
adviceChain.add(expressionAdvice());
gateway.setAdviceChain(adviceChain);
return gateway;
}
#Bean
public ExpressionEvaluatingRequestHandlerAdvice expressionAdvice() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setSuccessChannelName("success.input");
advice.setOnSuccessExpressionString("payload + ' was successful'");
advice.setFailureChannelName("failure.input");
advice.setOnFailureExpressionString(
"payload + ' was bad, with reason: ' + #exception.cause.message");
advice.setTrapException(true);
return advice;
}
#Bean
public IntegrationFlow success() {
return f -> f.handle(System.out::println);
}
#Bean
public IntegrationFlow failure() {
return f -> f.handle(System.out::println);
}

Since the movement to the DLQ is performed by the broker, the application has no mechanism to log the situation - it is not even aware that it happened.
You would have to catch the exceptions yourself and publish the message the the DLQ yourself, after some number of attempts (JMSXDeliveryCount header), instead of using the broker policy.
EDIT
Add an Advice to the .handle() step.
.handle(outboundGateway, e -> e.advice(myAdvice))
Where myAdvice implements MethodInterceptor.
In the invoke method, after a failure, you can check the delivery count header and, if it exceeds your threshold, publish the message to the DLQ (e.g. send it to another channel that has a JMS outbound adapter subscribed) and log the error; if the threshold has not been exceeded, simply return the result of the invocation.proceed() (or rethrow the exception).
That way, you control publishing to the DLQ rather than having the broker do it. You can also add more information, such as the exception, to headers.
EDIT2
You need something like this
public class MyAdvice implements MethodInterceptor {
#Autowired
private MessageChannel toJms;
public Object invoke(MethodInvocation invocation) throws Throwable {
try {
return invocation.proceed();
}
catch Exception(e) {
Message<?> message = (Message<?>) invocation.getArguments()[0];
Integer redeliveries = messasge.getHeader("JMXRedeliveryCount", Integer.class);
if (redeliveries != null && redeliveries > 3) {
this.toJms.send(message); // maybe rebuild with additional headers about the error
}
else {
throw e;
}
}
}
}
(it should be close, but I haven't tested it). It assumes your broker populates that header.

What is proper way to advise failed transform w/ Spring Integration Java DSL

I have completed a "happy-path" (as below).
How I can advise a .transform call to have it invoke an error flow (via errorChannel) w/o interrupting the mainFlow?
Currently the mainFlow terminates on first failure occurrence in second .transform (when payload cannot be deserialized to type). My desired behavior is that I'd like to log and continue processing.
I've read about ExpressionEvaluatingRequestHandlerAdvice. Would I just add a second param to each .transform call like e -> e.advice(myAdviceBean) and declare such a bean with success and error channels? Assuming I'd need to break up my mainFlow to receive success from each transform.
On some commented direction I updated the original code sample. But I'm still having trouble taking this "all the way home".
2015-09-08 11:49:19,664 [pool-3-thread-1] org.springframework.integration.handler.ServiceActivatingHandler DEBUG handler 'ServiceActivator for [org.springframework.integration.dsl.support.BeanNameMessageProcessor#5f3839ad] (org.springframework.integration.handler.ServiceActivatingHandler#0)' produced no reply for request Message: ErrorMessage [payload=org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice$MessageHandlingExpressionEvaluatingAdviceException: Handler Failed; nested exception is org.springframework.integration.transformer.MessageTransformationException: failed to transform message; nested exception is com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "hasDoaCostPriceChanged" (class com.xxx.changehistory.jdbc.data.RatePlanLevelRestrictionLog), not marked as ignorable (18 known properties: "supplierUpdateDate", "fPLOSMaskArrival", "createDate", "endAllowed", "sellStateId", "ratePlanLevel", "ratePlanId", "startAllowed", "stayDate", "doaCostPriceChanged", "hotelId", "logActionTypeId" [truncated]])
at [Source: java.util.zip.GZIPInputStream#242017b8; line: 1, column: 32] (through reference chain: com.xxx.changehistory.jdbc.data.RatePlanLevelRestrictionLog["hasDoaCostPriceChanged"]), headers={id=c054d976-5750-827f-8894-51aba9655c77, timestamp=1441738159660}]
2015-09-08 11:49:19,664 [pool-3-thread-1] org.springframework.integration.channel.DirectChannel DEBUG postSend (sent=true) on channel 'errorChannel', message: ErrorMessage [payload=org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice$MessageHandlingExpressionEvaluatingAdviceException: Handler Failed; nested exception is org.springframework.integration.transformer.MessageTransformationException: failed to transform message; nested exception is com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "hasDoaCostPriceChanged" (class com.xxx.changehistory.jdbc.data.RatePlanLevelRestrictionLog), not marked as ignorable (18 known properties: "supplierUpdateDate", "fPLOSMaskArrival", "createDate", "endAllowed", "sellStateId", "ratePlanLevel", "ratePlanId", "startAllowed", "stayDate", "doaCostPriceChanged", "hotelId", "logActionTypeId" [truncated]])
at [Source: java.util.zip.GZIPInputStream#242017b8; line: 1, column: 32] (through reference chain: com.xxx.changehistory.jdbc.data.RatePlanLevelRestrictionLog["hasDoaCostPriceChanged"]), headers={id=c054d976-5750-827f-8894-51aba9655c77, timestamp=1441738159660}]
2015-09-08 11:49:19,664 [pool-3-thread-1] org.springframework.integration.channel.DirectChannel DEBUG preSend on channel 'mainFlow.channel#3', message: GenericMessage [payload=java.util.zip.GZIPInputStream#242017b8, headers={id=b80106f9-7f4c-1b92-6aca-6e73d3bf8792, timestamp=1441738159664}]
2015-09-08 11:49:19,664 [pool-3-thread-1] org.springframework.integration.aggregator.AggregatingMessageHandler DEBUG org.springframework.integration.aggregator.AggregatingMessageHandler#0 received message: GenericMessage [payload=java.util.zip.GZIPInputStream#242017b8, headers={id=b80106f9-7f4c-1b92-6aca-6e73d3bf8792, timestamp=1441738159664}]
2015-09-08 11:49:19,665 [pool-3-thread-1] org.springframework.integration.channel.DirectChannel DEBUG preSend on channel 'errorChannel', message: ErrorMessage [payload=org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.aggregator.AggregatingMessageHandler#0]; nested exception is java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?, headers={id=24e3a1c7-af6b-032c-6a29-b55031fba0d7, timestamp=1441738159665}]
2015-09-08 11:49:19,665 [pool-3-thread-1] org.springframework.integration.handler.ServiceActivatingHandler DEBUG ServiceActivator for [org.springframework.integration.dsl.support.BeanNameMessageProcessor#5f3839ad] (org.springframework.integration.handler.ServiceActivatingHandler#0) received message: ErrorMessage [payload=org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.aggregator.AggregatingMessageHandler#0]; nested exception is java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?, headers={id=24e3a1c7-af6b-032c-6a29-b55031fba0d7, timestamp=1441738159665}]
2015-09-08 11:49:19,665 [pool-3-thread-1] com.xxx.DataMigrationModule$ErrorService ERROR org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.aggregator.AggregatingMessageHandler#0]; nested exception is java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:84)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:101)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:97)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:287)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:245)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:95)
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:231)
at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:154)
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:102)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:105)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:101)
at org.springframework.integration.dispatcher.UnicastingDispatcher.access$000(UnicastingDispatcher.java:48)
at org.springframework.integration.dispatcher.UnicastingDispatcher$1.run(UnicastingDispatcher.java:92)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Null correlation not allowed. Maybe the CorrelationStrategy is failing?
at org.springframework.util.Assert.state(Assert.java:385)
at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.handleMessageInternal(AbstractCorrelatingMessageHandler.java:369)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
... 22 more
UPDATED (09-08-2015)
code sample
#Bean
public IntegrationFlow mainFlow() {
// #formatter:off
return IntegrationFlows
.from(
amazonS3InboundSynchronizationMessageSource(),
e -> e.poller(p -> p.trigger(this::nextExecutionTime))
)
.transform(unzipTransformer())
.split(f -> new FileSplitter())
.channel(MessageChannels.executor(Executors.newCachedThreadPool()))
.transform(Transformers.fromJson(persistentType()), , e -> e.advice(handlingAdvice()))
// #see http://docs.spring.io/spring-integration/reference/html/messaging-routing-chapter.html#agg-and-group-to
.aggregate(a ->
a.releaseStrategy(g -> g.size() == persistenceBatchSize)
.expireGroupsUponCompletion(true)
.sendPartialResultOnExpiry(true)
.groupTimeoutExpression("size() ge 2 ? 10000 : -1")
, null
)
.handle(jdbcRepositoryHandler())
// TODO add advised PollableChannel to deal with possible persistence issue and retry with partial batch
.get();
// #formatter:on
}
#Bean
public ErrorService errorService() {
return new ErrorService();
}
#Bean
public MessageChannel customErrorChannel() {
return MessageChannels.direct().get();
}
#Bean
public IntegrationFlow customErrorFlow() {
// #formatter:off
return IntegrationFlows
.from(customErrorChannel())
.handle("errorService", "handleError")
.get();
// #formatter:on
}
#Bean
ExpressionEvaluatingRequestHandlerAdvice handlingAdvice() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setOnFailureExpression("payload");
advice.setFailureChannel(customErrorChannel());
advice.setReturnFailureExpressionResult(true);
advice.setTrapException(true);
return advice;
}
protected class ErrorService implements ErrorHandler {
private final Logger log = LoggerFactory.getLogger(getClass());
#Override
public void handleError(Throwable t) {
stopEndpoints(t);
}
private void stopEndpoints(Throwable t) {
log.error(ExceptionUtils.getStackTrace(t));
}
}

Turns out I had things wrong in a few places, like:
I had to autowire a Jackson2 ObjectMapper (that I get from Sprint Boot auto-config) and construct an instance of JsonObjectMapper to be added as second arg in Transformers.fromJson; made for more lenient unmarshalling to persistent type (stops UnrecognizedPropertyException); and thus waived need for ExpressionEvaluatingRequestHandlerAdvice
Choosing the proper variant of .split method in IntegrationFlowDefinition in order to employ the FileSplitter, otherwise you don't get this splitter rather a DefaultMessageSplitter which pre-maturely terminates flow after first record read from InputStream
Moved transform, aggregate, handle to a its own pubsub channel employing an async task executor
Still not 100% of what I need, but it's much further along.
See what I ended up w/ below...
#Configuration
#EnableIntegration
#IntegrationComponentScan
public class DataMigrationModule {
private final Logger log = LoggerFactory.getLogger(getClass());
#Value("${cloud.aws.credentials.accessKey}")
private String accessKey;
#Value("${cloud.aws.credentials.secretKey}")
private String secretKey;
#Value("${cloud.aws.s3.bucket}")
private String bucket;
#Value("${cloud.aws.s3.max-objects-per-batch:1024}")
private int maxObjectsPerBatch;
#Value("${cloud.aws.s3.accept-subfolders:false}")
private String acceptSubFolders;
#Value("${cloud.aws.s3.remote-directory}")
private String remoteDirectory;
#Value("${cloud.aws.s3.local-directory-ref:java.io.tmpdir}")
private String localDirectoryRef;
#Value("${cloud.aws.s3.local-subdirectory:target/s3-dump}")
private String localSubdirectory;
#Value("${cloud.aws.s3.filename-wildcard:}")
private String fileNameWildcard;
#Value("${app.persistent-type:}")
private String persistentType;
#Value("${app.repository-type:}")
private String repositoryType;
#Value("${app.persistence-batch-size:2500}")
private int persistenceBatchSize;
#Value("${app.persistence-batch-release-timeout-in-milliseconds:5000}")
private int persistenceBatchReleaseTimeoutMillis;
#Autowired
private ListableBeanFactory beanFactory;
#Autowired
private ObjectMapper objectMapper;
private final AtomicBoolean invoked = new AtomicBoolean();
private Class<?> repositoryType() {
try {
return Class.forName(repositoryType);
} catch (ClassNotFoundException cnfe) {
log.error("Unknown repository implementation!", cnfe);
System.exit(0);
}
return null;
}
private Class<?> persistentType() {
try {
return Class.forName(persistentType);
} catch (ClassNotFoundException cnfe) {
log.error("Unsupported type!", cnfe);
System.exit(0);
}
return null;
}
public Date nextExecutionTime(TriggerContext triggerContext) {
return this.invoked.getAndSet(true) ? null : new Date();
}
#Bean
public FileToInputStreamTransformer unzipTransformer() {
FileToInputStreamTransformer transformer = new FileToInputStreamTransformer();
transformer.setDeleteFiles(true);
return transformer;
}
#Bean
public MessageSource<?> amazonS3InboundSynchronizationMessageSource() {
AWSCredentials credentials = new BasicAWSCredentials(this.accessKey, this.secretKey);
AmazonS3InboundSynchronizationMessageSource messageSource = new AmazonS3InboundSynchronizationMessageSource();
messageSource.setCredentials(credentials);
messageSource.setBucket(bucket);
messageSource.setMaxObjectsPerBatch(maxObjectsPerBatch);
messageSource.setAcceptSubFolders(Boolean.valueOf(acceptSubFolders));
messageSource.setRemoteDirectory(remoteDirectory);
if (!fileNameWildcard.isEmpty()) {
messageSource.setFileNameWildcard(fileNameWildcard);
}
String directory = System.getProperty(localDirectoryRef);
if (!localSubdirectory.startsWith("/")) {
localSubdirectory = "/" + localSubdirectory;
}
if (!localSubdirectory.endsWith("/")) {
localSubdirectory = localSubdirectory + "/";
}
directory = directory + localSubdirectory;
FileUtils.mkdir(directory);
messageSource.setDirectory(new LiteralExpression(directory));
return messageSource;
}
#Bean
public IntegrationFlow mainFlow() {
// #formatter:off
return IntegrationFlows
.from(
amazonS3InboundSynchronizationMessageSource(),
e -> e.poller(p -> p.trigger(this::nextExecutionTime))
)
.transform(unzipTransformer())
.split(new FileSplitter(), null)
.publishSubscribeChannel(new SimpleAsyncTaskExecutor(), p -> p.subscribe(persistenceSubFlow()))
.get();
// #formatter:on
}
#Bean
public IntegrationFlow persistenceSubFlow() {
JsonObjectMapper<?, ?> jsonObjectMapper = new Jackson2JsonObjectMapper(objectMapper);
ReleaseStrategy releaseStrategy = new TimeoutCountSequenceSizeReleaseStrategy(persistenceBatchSize,
persistenceBatchReleaseTimeoutMillis);
// #formatter:off
return f -> f
.transform(Transformers.fromJson(persistentType(), jsonObjectMapper))
// #see http://docs.spring.io/spring-integration/reference/html/messaging-routing-chapter.html#agg-and-group-to
.aggregate(
a -> a
.releaseStrategy(releaseStrategy)
.correlationStrategy(m -> m.getHeaders().get("id"))
.expireGroupsUponCompletion(true)
.sendPartialResultOnExpiry(true)
, null
)
.handle(jdbcRepositoryHandler());
// #formatter:on
}
#Bean
public JdbcRepositoryHandler jdbcRepositoryHandler() {
return new JdbcRepositoryHandler(repositoryType(), beanFactory);
}
protected class JdbcRepositoryHandler extends AbstractMessageHandler {
#SuppressWarnings("rawtypes")
private Insertable repository;
public JdbcRepositoryHandler(Class<?> repositoryClass, ListableBeanFactory beanFactory) {
repository = (Insertable<?>) beanFactory.getBean(repositoryClass);
}
#Override
protected void handleMessageInternal(Message<?> message) {
repository.insert((List<?>) message.getPayload());
}
}
protected class FileToInputStreamTransformer extends AbstractFilePayloadTransformer<InputStream> {
#Override
protected InputStream transformFile(File payload) throws Exception {
return new GZIPInputStream(new FileInputStream(payload));
}
}
}

Yes, you are correct. To advice the handle() method of Transformer's MessageHandler you should use exactly that e.advice method of the second parameter of .transform() EIP-method. And yes: you should define ExpressionEvaluatingRequestHandlerAdvice bean for your purpose.
You can reuse that Advice bean for different goals to handle successes and failures the same manner.
UPDATE
Although it isn't clear to me how you'd like to continue the flow with the wrong message, but you you can use onFailureExpression and returnFailureExpressionResult=true of the ExpressionEvaluatingRequestHandlerAdvice to return something after the unzipErrorChannel().
BTW the failureChannel logic doesn't work without onFailureExpression:
if (this.onFailureExpression != null) {
Object evalResult = this.evaluateFailureExpression(message, actualException);
if (this.returnFailureExpressionResult) {
return evalResult;
}
}