I want to download a file from ftp server periodically (only when the file has changed). Therefore I'd like to use Spring-Integration 4.0.
What is the annotation based configuration equivalent of the following setup with int-ftp?
<int-ftp:inbound-channel-adapter id="ftpInbound"
channel="ftpChannel"
session-factory="ftpClientFactory"
filename-pattern="*.txt"
auto-create-local-directory="true"
delete-remote-files="false"
remote-directory="/">
<int:poller fixed-rate="1000"/>
</int-ftp:inbound-channel-adapter>
I started with the following, but don't know how I can configure the channel with its attributes like session-factory, remote-directory etc.
#Configuration
#EnableIntegration
public class Application {
#Bean
public SessionFactory<FTPFile> sessionFactory() {
DefaultFtpSessionFactory ftp = new DefaultFtpSessionFactory();
ftp.setHost("ftp.test");
ftp.setPort(21);
ftp.setUsername("anonymous");
ftp.setPassword("anonymous");
return ftp;
}
#InboundChannelAdapter(value = "ftpChannel", poller = #Poller(fixedDelay = "1000", maxMessagesPerPoll = "1"))
public String connect() {
// return the ftp file
}
#ServiceActivator(inputChannel = "ftpChannel")
public void foo(String payload) {
System.out.println("paylod: " + payload);
}
}
The (S)FTP inbound adapters are on the more complex side; we're working in the DSL to make this easier but, currently, you need an #Bean FtpInboundFileSynchronizer wired up with appropriate properties and inject it into
Then
#Bean
#InboundChannelAdapter(value = "ftpChannel", poller = #Poller(fixedDelay = "1000", maxMessagesPerPoll = "1"))
public MessageSource receive() {
FtpInboundFileSynchronizingMessageSource messageSource = new FtpInboundFileSynchronizingMessageSource(synchronizer());
...
return messageSource;
}
Related
I have an Ftp Configuration file like
public class FtpConfig {
#Bean
public DefaultFtpSessionFactory ftpSessionFactory() {
DefaultFtpSessionFactory sessionFactory = new DefaultFtpSessionFactory();
sessionFactory.setHost("host");
sessionFactory.setPort(21);
sessionFactory.setUsername("ftpuser");
sessionFactory.setPassword("pass");
return sessionFactory;
}
#ServiceActivator(inputChannel = "fromFtpChannel")
#Bean
public FtpOutboundGateway getFiles() {
FtpOutboundGateway gateway = new FtpOutboundGateway(ftpSessionFactory(), "get", "payload");
gateway.setOptions(AbstractRemoteFileOutboundGateway.Option.STREAM.getOption());
gateway.setOutputChannelName("fileoutput");
return gateway;
}
}
And a Messaging Gateway
#MessagingGateway
public interface ReadFilesGateway {
#Gateway(requestChannel = "fromFtpChannel", replyChannel = "fileoutput")
InputStream readFiles(String directory);
}
The Messaging gateway is used to download files from the ftp server, to view the files as inputStream. A service FileService uses the gateway to read the files. How to write a possible testcase for the service, or the Messaging Gateway?
I have modified my RabbitMQ from a previous post (spring-rabbit JSON deserialization for ArrayList contents) to now use a DirectMessageListener with MessagePostProcessors to GZip and GUnzip the message payloads.
However, it doesn't appear to be working as the breakpoints are not activated, but also because my RabbitListeners are no longer receiving messages, whereas they did with a SimpleMessageFactoryListenerContainer.
Also, it appears the SimpleMessageListenerContainer(?) is still being used. On a side-note, I am autowiring the DirectMessageListenerContainer so I can dynamically set the queues I used.
spring-rabbit: 2.0.3.RELEASE.
spring-boot: 2.0.1.RELEASE.
RabbitMQ configuration:
#Configuration
#EnableRabbit
public class MessagingConfiguration implements ShutdownListener {
#Autowired
private RabbitListenerEndpointRegistry registry;
#Autowired
private DirectMessageListenerContainer container;
#Bean
public DirectMessageListenerContainer messageListenerContainer(final ConnectionFactory connectionFactory) {
final DirectMessageListenerContainer listenerContainer = new DirectMessageListenerContainer();
listenerContainer.setConnectionFactory(connectionFactory);
listenerContainer.setMessageConverter(jsonConverter()); // i.e.#RabbitListener to use Jackson2JsonMessageConverter
listenerContainer.setAutoStartup(false);
// container.setErrorHandler(errorHandler);
final MessageListenerAdapter messageListener = new MessageListenerAdapter(new Object() {
#SuppressWarnings("unused")
public String handleMessage(final String message) {
return message.toUpperCase();
}
});
messageListener.setBeforeSendReplyPostProcessors(new GZipPostProcessor());
listenerContainer.setMessageListener(messageListener);
listenerContainer.setAfterReceivePostProcessors(new GUnzipPostProcessor());
return listenerContainer;
}
#EventListener(ApplicationDatabaseReadyEvent.class)
public void onApplicationDatabaseReadyEvent() {
log.info("Starting all RabbitMQ Listeners..."); //$NON-NLS-1$
for (final MessageListenerContainer listenerContainer : registry.getListenerContainers()) {
listenerContainer.start();
}
log.info("Register is running: {}", registry.isRunning()); //$NON-NLS-1$
log.info("Started all RabbitMQ Listeners."); //$NON-NLS-1$
}
#Bean
public List<Declarable> bindings() {
final List<Declarable> declarations = new ArrayList<>();
final FanoutExchange exchange = new FanoutExchange("fx", true, false);
final Queue queue = QueueBuilder.durable("orders").build();
declarations.add(exchange);
declarations.add(queue);
declarations.add(BindingBuilder.bind(queue).to(exchange));
List<String> q = new ArrayList<>();
q.add(queue.getName());
container.addQueueNames(q.toArray(new String[queues.size()]));
return declarations;
}
#Bean
public Jackson2JsonMessageConverter jsonConverter() {
final Jackson2JsonMessageConverter converter = new Jackson2JsonMessageConverter();
converter.setClassMapper(classMapper());
return converter;
}
private static DefaultJackson2JavaTypeMapper classMapper() {
final DefaultJackson2JavaTypeMapper classMapper = new DefaultJackson2JavaTypeMapper();
classMapper.setTrustedPackages("*"); //$NON-NLS-1$ //TODO add trusted packages
return classMapper;
}
#ConditionalOnProperty(name = "consumer", havingValue = "true")
#Bean
public ConsumerListener listenerConsumer() {
return new ConsumerListener();
}
#ConditionalOnProperty(name = "producer", havingValue = "true")
#Bean
public ProducerListener listenerProducer() {
return new ProducerListener();
}
#Bean
public RabbitAdmin rabbitAdmin(final CachingConnectionFactory connectionFactory) {
return new RabbitAdmin(connectionFactory);
}
#Bean
public RabbitTemplate rabbitTemplate(final ConnectionFactory connectionFactory) {
final RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setMessageConverter(jsonConverter()); // convert all sent messages to JSON
rabbitTemplate.setReplyTimeout(TimeUnit.SECONDS.toMillis(3));
rabbitTemplate.setReceiveTimeout(TimeUnit.SECONDS.toMillis(3));
return rabbitTemplate;
}
#Override
public void shutdownCompleted(final ShutdownSignalException arg0) {
}
}
It doesn't work that way, you can't autowire containers for #RabbitListeners; they are not beans; they are created by the container factory and registered in the registry. Instead you have to retrieve them from the registry (by id).
However, since you have autoStartup set to false, it shouldn't be "stealing" messages from your #RabbitListener.
Generally, DEBUG logging should help.
I have the following configuration file for Spring Integration File:
#Configuration
#EnableIntegration
public class MyIntegrationConfiguration {
private static final String FILE_CHANNEL_PROCESSING = "processingfileChannel";
private static final String INTERVAL_PROCESSING = "5000";
private static final String FILE_PATTERN = "*.txt";
#Value("${import.path.source}")
private String sourceDir;
#Value("${import.path.output}")
private String outputDir;
#Bean
#InboundChannelAdapter(value = FILE_CHANNEL_PROCESSING, poller = #Poller(fixedDelay = INTERVAL_PROCESSING))
public MessageSource<File> sourceFiles() {
FileReadingMessageSource source = new FileReadingMessageSource();
source.setAutoCreateDirectory(true);
source.setDirectory(new File(sourceDir));
source.setFilter(new SimplePatternFileListFilter(FILE_PATTERN));
return source;
}
#Bean
#ServiceActivator(inputChannel = FILE_CHANNEL_PROCESSING)
public MessageHandler processedFiles() {
FileWritingMessageHandler handler = new FileWritingMessageHandler(new File(outputDir));
handler.setFileExistsMode(FileExistsMode.REPLACE);
handler.setDeleteSourceFiles(true);
handler.setExpectReply(true);
return handler;
}
#Bean
public IntegrationFlow processFileFlow() {
return IntegrationFlows
.from(FILE_CHANNEL_PROCESSING)
.transform(fileToStringTransformer())
.handle("fileProcessor", "processFile").get();
}
#Bean
public MessageChannel fileChannel() {
return new DirectChannel();
}
#Bean
public FileProcessor fileProcessor() {
return new FileProcessor();
}
#Bean
public FileToStringTransformer fileToStringTransformer() {
return new FileToStringTransformer();
}
}
For FileWritingMessageHandler from this documentation it says that if setExpectReply(true) is set:
Specify whether a reply Message is expected. If not, this handler will simply return null for a successful response or throw an Exception for a non-successful response.
My question is: where can I catch these exceptions or where can I retrieve this message/response?
The #ServiceActivator has an outputChannel attribute:
/**
* Specify the channel to which this service activator will send any replies.
* #return The channel name.
*/
String outputChannel() default "";
That's for successful replies.
Any exceptions (independently of the setExpectReply()) are just thrown to the caller. In your case the story is about an #InboundChannelAdapter. In this case the exception is caught and wrapped to the ErrorMessage to be sent to the errorChannel on the #Poller. It is a global errorChannel by default: https://docs.spring.io/spring-integration/docs/5.0.4.RELEASE/reference/html/configuration.html#namespace-errorhandler
However you have some other problem in your code. I see the second subscriber to the FILE_CHANNEL_PROCESSING. If it's not a PublishSubscribeChannel, you are going to have a round-robin distribution for messages sent to this channel. But that's already a different story. Just don't ask that question here, please!
I have successfully been able send file from one FTP Server(source) to another FTP server (target). I first send files from source to the local directory using the inbound adapter and then send files from the local directory to the target using the outbound adapter. So far this is working fine.
What I want to achieve is: to enrich the header of the message at the source with a hash code (which is generated using the file on source that is transferred) and then get that header at the target and match it with the hash code (which is generated using the file on the target)
Here is what I have tried so far:
Application.java
#SpringBootApplication
public class Application {
#Autowired
private Hashing hashing;
public static ConfigurableApplicationContext context;
public static void main(String[] args) {
context = new SpringApplicationBuilder(Application.class)
.web(false)
.run(args);
}
#Bean
#ServiceActivator(inputChannel = "ftpChannel")
public MessageHandler sourceHandler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
System.out.println("Reply channel isssss:"+message.getHeaders().getReplyChannel());
Object payload = message.getPayload();
System.out.println("Payload: " + payload);
File file = (File) payload;
// enrich header with hash code before sending to target FTP
Message<?> messageOut = MessageBuilder
.withPayload(message.getPayload())
.copyHeadersIfAbsent(message.getHeaders())
.setHeaderIfAbsent("hashCode", hashing.getHashCode(file)).build();
// send to target FTP
System.out.println("Trying to send " + file.getName() + " to target");
MyGateway gateway = context.getBean(MyGateway.class);
gateway.sendToFtp(messageOut);
}
};
}
}
FileTransferServiceConfig.java
#Configuration
#Component
public class FileTransferServiceConfig {
#Autowired
private ConfigurationService configurationService;
#Autowired
private Hashing hashing;
public static final String FILE_POLLING_DURATION = "5000";
#Bean
public SessionFactory<FTPFile> sourceFtpSessionFactory() {
DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
sf.setHost(configurationService.getSourceHostName());
sf.setPort(Integer.parseInt(configurationService.getSourcePort()));
sf.setUsername(configurationService.getSourceUsername());
sf.setPassword(configurationService.getSourcePassword());
return new CachingSessionFactory<>(sf);
}
#Bean
public SessionFactory<FTPFile> targetFtpSessionFactory() {
DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
sf.setHost(configurationService.getTargetHostName());
sf.setPort(Integer.parseInt(configurationService.getTargetPort()));
sf.setUsername(configurationService.getTargetUsername());
sf.setPassword(configurationService.getTargetPassword());
return new CachingSessionFactory<>(sf);
}
#MessagingGateway
public interface MyGateway {
#Gateway(requestChannel = "toFtpChannel")
void sendToFtp(Message message);
}
#Bean
public FtpInboundFileSynchronizer ftpInboundFileSynchronizer() {
FtpInboundFileSynchronizer fileSynchronizer = new FtpInboundFileSynchronizer(sourceFtpSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setRemoteDirectory(configurationService.getSourceDirectory());
fileSynchronizer.setFilter(new FtpSimplePatternFileListFilter(
configurationService.getFileMask()));
return fileSynchronizer;
}
#Bean
public AcceptOnceFileListFilter<File> acceptOnceFileListFilter() {
return new AcceptOnceFileListFilter<>();
}
#Bean
#InboundChannelAdapter(channel = "ftpChannel",
poller = #Poller(fixedDelay = FILE_POLLING_DURATION))
public MessageSource<File> ftpMessageSource() {
FtpInboundFileSynchronizingMessageSource source
= new FtpInboundFileSynchronizingMessageSource(ftpInboundFileSynchronizer());
source.setLocalDirectory(new File(configurationService.getLocalDirectory()));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(acceptOnceFileListFilter());
return source;
}
// makes sure transfer continues on connection reset
#Bean
public Advice expressionAdvice() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setTrapException(true);
advice.setOnFailureExpression("#acceptOnceFileListFilter.remove(payload)");
return advice;
}
#Bean
#ServiceActivator(inputChannel = "toFtpChannel")
public void listenOutboundMessage() {
// tried to subscribe to "toFtpChannel" but this was not triggered
System.out.println("Message received");
}
#Bean
#ServiceActivator(inputChannel = "ftpChannel", adviceChain = "expressionAdvice")
public MessageHandler targetHandler() {
FtpMessageHandler handler = new FtpMessageHandler(targetFtpSessionFactory());
handler.setRemoteDirectoryExpression(new LiteralExpression(
configurationService.getTargetDirectory()));
return handler;
}
}
Hashing.java
public interface Hashing {
public String getHashCode(File payload);
}
I have managed to enrich the message in sourceHandler(), built the message and sent it to the target but I cannot figure out how I can receive that message on the target so that I can get the header from the message?
Tell me if any more information is required. I would really appreciate your help.
You have two subscribers on ftpChannel - the target handler and your sourceHandler; they will get alternate messages unless ftpChannel is declared as a pubsub channel.
There should be no problems with your subscription to toFtpChannel.
Turn on DEBUG logging to see all the subscription activity when the application context starts.
EDIT
Remove the #Bean from the #ServiceActivator - such beans must be a MessageHandler.
#ServiceActivator(inputChannel = "toFtpChannel")
public void listenOutboundMessage(Message message) {
// tried to subscribe to "toFtpChannel" but this was not triggered
System.out.println("Message received:" + message);
}
works fine for me...
Payload: /tmp/foo/baz.txt
Trying to send baz.txt to target
Message received:GenericMessage [payload=/tmp/foo/baz.txt, headers={hashCode=foo, id=410eb9a2-fe8b-ea8a-015a-d5896387cf00, timestamp=1509115006278}]
Again; you must have only one subscriber on ftpChannel unless you make it a pubsub.
We are using the latest Spring Boot for a Spring app and using the latest Spring Integration for SFTP. I've been to the Spring Integration SFTP documentation site, and I took the Spring Boot Configuration as is:
#Bean
public SessionFactory<LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost("localhost");
factory.setPort(port);
factory.setUser("foo");
factory.setPassword("foo");
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<LsEntry>(factory);
}
#Bean
public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
SftpInboundFileSynchronizer fileSynchronizer = new SftpInboundFileSynchronizer(sftpSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setRemoteDirectory("/");
fileSynchronizer.setFilter(new SftpSimplePatternFileListFilter("*.xml"));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "sftpChannel")
public MessageSource<File> sftpMessageSource() {
SftpInboundFileSynchronizingMessageSource source =
new SftpInboundFileSynchronizingMessageSource(sftpInboundFileSynchronizer());
source.setLocalDirectory(new File("ftp-inbound"));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
return source;
}
#Bean
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
System.out.println(message.getPayload());
}
};
}
Let me be clear, after cutting and pasting, there are some unit tests that run. However, when loading the application context there was an error message because the Polling wasn't there.
When I googled that error, other posts on StackOverflow said I also had to add to remove this error message when loading the application context.
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata defaultPoller() {
PollerMetadata pollerMetadata = new PollerMetadata();
pollerMetadata.setTrigger(new PeriodicTrigger(60));
return pollerMetadata;
}
When I added this code, THEN at least my build would work and the tests would run because the application context was now being loaded correctly.
Now I am looking for a code sample on how to make this work and move files? The Spring Integration SFTP examples on GitHub are ok, but not great ... far from it.
The Basic Spring Integration Example shows how to read files from an SFTP Server, if the data is configured with an application-context.xml file. Where is the example where a Spring Boot configuration is used, and then the code to read from that server, and the code for the test?
I understand that regardless of whether you use a Java class for Spring Boot configuration or an application-context.xml file ... the working code should work the same for autowired SFTP channels and some inbound channel adapter.
So here is the code, I am trying to make work:
#Component
#Profile("sftpInputFetch")
public class SFTPInputFetcher implements InputFetcher
{
// The PollableChannel seems fine
#Autowired
PollableChannel sftpChannel;
#Autowired
SourcePollingChannelAdapter sftpChannelAdapter;
#Override
public Stream<String> fetchLatest() throws FileNotFoundException
{
Stream<String> stream = null;
sftpChannelAdapter.start();
Message<?> received = sftpChannel.receive();
File file = (File)received.getPayload();
// get Stream<String> from file
return stream;
}
Currently, "sftpChannelAdapter.start();" is the part I am having trouble with.
This implementation does not find the "SourcePollingChannelAdapter" class.
If this was defined in the classic XML application context with an "id" then this code autowires just fine. With a Spring Boot configuration, it doesn't look like you can define an "id" for a bean.
This just stems from my lack of knowledge on how to convert from using a traditional application-context XML file WITH annotations in the code, to using a complete Spring Boot application context configuration file.
Any help with this is much appreciated. Thanks!
I don't understand the question; you said
I had to add ... to make it work
and then
Now I am looking for a code sample on how to make this work?
What is not working?
You can also use
#InboundChannelAdapter(value = "sftpChannel", poller = #Poller(fixedDelay = "5000"))
instead of adding a default poller definition.
We will fix the docs for the missing poller config.
EDIT
I just copied the code into a new boot app (with the poller config) and it works as expected.
#SpringBootApplication
public class SftpJavaApplication {
public static void main(String[] args) {
new SpringApplicationBuilder(SftpJavaApplication.class).web(false).run(args);
}
#Bean
public SessionFactory<LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost("...");
factory.setPort(22);
factory.setUser("...");
factory.setPassword("...");
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<LsEntry>(factory);
}
#Bean
public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
SftpInboundFileSynchronizer fileSynchronizer = new SftpInboundFileSynchronizer(sftpSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setRemoteDirectory("foo");
fileSynchronizer.setFilter(new SftpSimplePatternFileListFilter("*.txt"));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "sftpChannel", poller = #Poller(fixedDelay = "5000"))
public MessageSource<File> sftpMessageSource() {
SftpInboundFileSynchronizingMessageSource source = new SftpInboundFileSynchronizingMessageSource(
sftpInboundFileSynchronizer());
source.setLocalDirectory(new File("ftp-inbound"));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
return source;
}
#Bean
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
System.out.println(message.getPayload());
}
};
}
}
Result:
16:57:59.697 [task-scheduler-1] WARN com.jcraft.jsch - Permanently added '10.0.0.3' (RSA) to the list of known hosts.
ftp-inbound/bar.txt
ftp-inbound/baz.txt