We are using the latest Spring Boot for a Spring app and using the latest Spring Integration for SFTP. I've been to the Spring Integration SFTP documentation site, and I took the Spring Boot Configuration as is:
#Bean
public SessionFactory<LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost("localhost");
factory.setPort(port);
factory.setUser("foo");
factory.setPassword("foo");
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<LsEntry>(factory);
}
#Bean
public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
SftpInboundFileSynchronizer fileSynchronizer = new SftpInboundFileSynchronizer(sftpSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setRemoteDirectory("/");
fileSynchronizer.setFilter(new SftpSimplePatternFileListFilter("*.xml"));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "sftpChannel")
public MessageSource<File> sftpMessageSource() {
SftpInboundFileSynchronizingMessageSource source =
new SftpInboundFileSynchronizingMessageSource(sftpInboundFileSynchronizer());
source.setLocalDirectory(new File("ftp-inbound"));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
return source;
}
#Bean
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
System.out.println(message.getPayload());
}
};
}
Let me be clear, after cutting and pasting, there are some unit tests that run. However, when loading the application context there was an error message because the Polling wasn't there.
When I googled that error, other posts on StackOverflow said I also had to add to remove this error message when loading the application context.
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata defaultPoller() {
PollerMetadata pollerMetadata = new PollerMetadata();
pollerMetadata.setTrigger(new PeriodicTrigger(60));
return pollerMetadata;
}
When I added this code, THEN at least my build would work and the tests would run because the application context was now being loaded correctly.
Now I am looking for a code sample on how to make this work and move files? The Spring Integration SFTP examples on GitHub are ok, but not great ... far from it.
The Basic Spring Integration Example shows how to read files from an SFTP Server, if the data is configured with an application-context.xml file. Where is the example where a Spring Boot configuration is used, and then the code to read from that server, and the code for the test?
I understand that regardless of whether you use a Java class for Spring Boot configuration or an application-context.xml file ... the working code should work the same for autowired SFTP channels and some inbound channel adapter.
So here is the code, I am trying to make work:
#Component
#Profile("sftpInputFetch")
public class SFTPInputFetcher implements InputFetcher
{
// The PollableChannel seems fine
#Autowired
PollableChannel sftpChannel;
#Autowired
SourcePollingChannelAdapter sftpChannelAdapter;
#Override
public Stream<String> fetchLatest() throws FileNotFoundException
{
Stream<String> stream = null;
sftpChannelAdapter.start();
Message<?> received = sftpChannel.receive();
File file = (File)received.getPayload();
// get Stream<String> from file
return stream;
}
Currently, "sftpChannelAdapter.start();" is the part I am having trouble with.
This implementation does not find the "SourcePollingChannelAdapter" class.
If this was defined in the classic XML application context with an "id" then this code autowires just fine. With a Spring Boot configuration, it doesn't look like you can define an "id" for a bean.
This just stems from my lack of knowledge on how to convert from using a traditional application-context XML file WITH annotations in the code, to using a complete Spring Boot application context configuration file.
Any help with this is much appreciated. Thanks!
I don't understand the question; you said
I had to add ... to make it work
and then
Now I am looking for a code sample on how to make this work?
What is not working?
You can also use
#InboundChannelAdapter(value = "sftpChannel", poller = #Poller(fixedDelay = "5000"))
instead of adding a default poller definition.
We will fix the docs for the missing poller config.
EDIT
I just copied the code into a new boot app (with the poller config) and it works as expected.
#SpringBootApplication
public class SftpJavaApplication {
public static void main(String[] args) {
new SpringApplicationBuilder(SftpJavaApplication.class).web(false).run(args);
}
#Bean
public SessionFactory<LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost("...");
factory.setPort(22);
factory.setUser("...");
factory.setPassword("...");
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<LsEntry>(factory);
}
#Bean
public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
SftpInboundFileSynchronizer fileSynchronizer = new SftpInboundFileSynchronizer(sftpSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setRemoteDirectory("foo");
fileSynchronizer.setFilter(new SftpSimplePatternFileListFilter("*.txt"));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "sftpChannel", poller = #Poller(fixedDelay = "5000"))
public MessageSource<File> sftpMessageSource() {
SftpInboundFileSynchronizingMessageSource source = new SftpInboundFileSynchronizingMessageSource(
sftpInboundFileSynchronizer());
source.setLocalDirectory(new File("ftp-inbound"));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
return source;
}
#Bean
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
System.out.println(message.getPayload());
}
};
}
}
Result:
16:57:59.697 [task-scheduler-1] WARN com.jcraft.jsch - Permanently added '10.0.0.3' (RSA) to the list of known hosts.
ftp-inbound/bar.txt
ftp-inbound/baz.txt
Related
My application is based on spring-boot 2.1.6 equipped with spring-batch (chunks approach) and spring-integration to handle the SFTP.
High level functionality is to fetch data from DB, generate a text file then send it through SFTP and this task run every 30 mins.
This application already running in production for some time, but if I see the logs there are error about ssh_msg_disconnect 11 idle connection. It will keep like that until I restart the app.
Below is my application code :
SftpConfig.java
#Configuration
public class SftpConfig {
#Autowired
ApplicationProperties applicationProperties;
#Bean
public SessionFactory<LsEntry> sftpSessionFactory() {
final DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(applicationProperties.getSftp().getHost());
factory.setUser(applicationProperties.getSftp().getUser());
factory.setPassword(applicationProperties.getSftp().getPass());
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<>(factory);
}
#Bean
#ServiceActivator(inputChannel = "toSftpChannel", adviceChain = "retryAdvice")
public MessageHandler handler() {
final SftpMessageHandler handler = new SftpMessageHandler(this.sftpSessionFactory());
handler.setRemoteDirectoryExpression(new LiteralExpression(applicationProperties.getSftp().getPath()));
handler.setFileNameGenerator((final Message<?> message) -> {
if (message.getPayload() instanceof File) {
return ((File) message.getPayload()).getName();
} else {
throw new IllegalArgumentException("File expected as payload.");
}
});
return handler;
}
#Bean
public RequestHandlerRetryAdvice retryAdvice() {
final RequestHandlerRetryAdvice advice = new RequestHandlerRetryAdvice();
final RetryTemplate retryTemplate = new RetryTemplate();
final SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy();
retryPolicy.setMaxAttempts(NumberConstants.FIVE);
retryTemplate.setRetryPolicy(retryPolicy);
advice.setRetryTemplate(retryTemplate);
return advice;
}
#MessagingGateway
public interface UploadGateway {
#Gateway(requestChannel = "toSftpChannel")
void upload(File file);
}
}
step for sending file to sftp
#Autowired
UploadGateway uploadGateway;
private boolean uploadToSharedFolderSuccess(final PaymentStatus paymentStatus, final String strLocalTmpPath) {
try {
final File fileLocalTmpFullPath = new File(strLocalTmpPath);
uploadGateway.upload(fileLocalTmpFullPath);
} catch (final Exception e) {
paymentStatus.setStatus(ProcessStatus.ERROR.toString());
paymentStatus.setRemark(StringUtil.appendIfNotEmpty(paymentStatus.getRemark(),
"Error during upload to shared folder - " + e.getMessage()));
}
return !StringUtils.equalsIgnoreCase(ProcessStatus.ERROR.toString(), paymentStatus.getStatus());
}
From the error, I know that seems like I opened too many connection. But I'm not sure how to check if the connection are closed every end of the spring-batch.
If you don't wrap the session factory in a CachingSessionFactory, the session will be closed after each use.
#Bean
public DefaultSftpSessionFactory sftpSessionFactory() {
final DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(applicationProperties.getSftp().getHost());
factory.setUser(applicationProperties.getSftp().getUser());
factory.setPassword(applicationProperties.getSftp().getPass());
factory.setAllowUnknownKeys(true);
return factory;
}
I have an application where I try to combine Spring MVC and Apache CFX(soap) web services. When I run just the app, everything seems fine, I see generated WSDL by this link(http://localhost:8080/services/customer?wsdl). But when I run tests, it throws WebServiceException: Could not send Message... Connection refused.
I've opened all ports for public, private and domain area through Windows Firewall Defender. Maybe I've missed something.
In a desperate attempt to investigate it, I've checked the link with this command (wsimport -keep -verbose http://localhost:8080/services/customer?wsdl). As a result, it gave this:
[ERROR] Server returned HTTP response code: 403 for URL: http://localhost:8080/services/customer?wsdl
Failed to read the WSDL document: http://localhost:8080/services/customer?wsdl, because 1) could not find the document; /2) the document could not be read; 3) the root element of the document is not <wsdl:definitions>.
[ERROR] Could not find wsdl:service in the provided WSDL(s):
At least one WSDL with at least one service definition needs to be provided.
Now I do not know which way to dig.
WebServiceDispatcherServletInitializer
public class WebServiceDispatcherServletInitializer implements WebApplicationInitializer {
#Override
public void onStartup(ServletContext servletContext) throws ServletException {
AnnotationConfigWebApplicationContext context = new AnnotationConfigWebApplicationContext();
context.register(WebServiceConfig.class);
servletContext.addListener(new ContextLoaderListener(context));
ServletRegistration.Dynamic dispatcher = servletContext.addServlet("dispatcher", new CXFServlet());
dispatcher.addMapping("/services/*");
}
}
WebServiceConfig
#Configuration
public class WebServiceConfig {
#Bean(name = Bus.DEFAULT_BUS_ID)
public SpringBus springBus() {
return new SpringBus();
}
#Bean
public Endpoint endpoint() {
EndpointImpl endpoint = new EndpointImpl(springBus(), new CustomerWebServiceImpl() );
endpoint.publish("http://localhost:8080/services/customer");
return endpoint;
}
}
ClientConfig
#Configuration
public class ClientConfig {
#Bean(name = "client")
public Object generateProxy() {
return proxyFactoryBean().create();
}
#Bean
public JaxWsProxyFactoryBean proxyFactoryBean() {
JaxWsProxyFactoryBean proxyFactory = new JaxWsProxyFactoryBean();
proxyFactory.setServiceClass(CustomerWebService.class);
proxyFactory.setAddress("http://localhost:8080/services/customer");
return proxyFactory;
}
}
CustomerWebServiceImplTest
#ActiveProfiles(profiles = "test")
#ContextConfiguration(classes = {
PersistenceConfig.class,
RootConfig.class,
WebServiceConfig.class,
ClientConfig.class
})
#WebAppConfiguration
public class CustomerWebServiceImplTest {
private ApplicationContext context = new AnnotationConfigApplicationContext(ClientConfig.class);
private CustomerWebService customerWsProxy = (CustomerWebService) context.getBean("client");
#Test
public void addCustomer() {
CustomerDto customer = new CustomerDto();
customer.setName("John");
assertEquals("Hello " + customer.getName(), customerWsProxy.addCustomer(customer));
}
}
Could you give a hint where the error might be?
UPD: I checked this setup on PC where I and my applications have full access rights and it still throws the Exception.
A solution was quite simple - just need to add #RunWith(SpringRunner.class). Because this annotation is run spring beans, not #WebAppConfiguration with #ContextConfiguration.
This is how it will look like
#RunWith(SpringRunner.class)
#ContextConfiguration(classes = {
RootConfig.class,
WebServiceConfig.class,
ClientConfig.class
})
public class CustomerWebServiceImplTest {
...
}
I am reading mails using spring mail inbound channel adapter once message is read i am performing some db operations in service activator of corresponding channel. My requirement is if any db operation fails adapter should read same message again.
Mail configuration :
#Bean
public DirectChannel inputChannel() {
return new DirectChannel();
}
#Bean
public IntegrationFlow pop3MailFlow() {
String url = "[url]";
return IntegrationFlows
.from(Mail.pop3InboundAdapter(url)
.javaMailProperties(p -> p.put("mail.pop3.socketFactory.class", "javax.net.ssl.SSLSocketFactory")),e -> e.autoStartup(true)
.poller(Pollers.fixedDelay(2000).transactionSynchronizationFactory(transactionSynchronizationFactory())))
.channel(inputChannel())
.handle(inboundEmailProcessor(),"messageProcess")
.get();
}
#Bean
public InboundEmailProcessor inboundEmailProcessor() {
return new InboundEmailProcessor();
}
#Bean
public TransactionSynchronizationFactory transactionSynchronizationFactory() {
TransactionSynchronizationFactory synchronizationFactory = new DefaultTransactionSynchronizationFactory(expressionEvaluatingTransactionSynchronizationProcessor());
return synchronizationFactory;
}
#Bean
public ExpressionEvaluatingTransactionSynchronizationProcessor expressionEvaluatingTransactionSynchronizationProcessor() {
ExpressionEvaluatingTransactionSynchronizationProcessor processor = new ExpressionEvaluatingTransactionSynchronizationProcessor();
ExpressionParser parser = new SpelExpressionParser();
processor.setAfterRollbackExpression(parser.parseExpression("new com.muraai.ex.config.Exp().process(payload)"));
return processor;
}
public class InboundEmailProcessor {
#Autowired
AttachmentsRepository attachmentsRepository;
#Transactional(rollbackFor = Exception.class)
public void messageProcess() {
// some db operations
// if it fails the same message should be read again
}
}
I thought this would work but its not working. Is there any way to achieve my requirement
public class Exp {
public void process(MimeMessage message) throws MessagingException {
message.setFlag(Flags.Flag.SEEN, false);
}
}
You need IMAP for that; with POP3, the server always marks them read.
You can add a spring-retry interceptor advice to the poller's advice chain and/or send the failed message to an error channel.
The retry advice can be configured for number of retries, back off policy etc.
I have successfully been able send file from one FTP Server(source) to another FTP server (target). I first send files from source to the local directory using the inbound adapter and then send files from the local directory to the target using the outbound adapter. So far this is working fine.
What I want to achieve is: to enrich the header of the message at the source with a hash code (which is generated using the file on source that is transferred) and then get that header at the target and match it with the hash code (which is generated using the file on the target)
Here is what I have tried so far:
Application.java
#SpringBootApplication
public class Application {
#Autowired
private Hashing hashing;
public static ConfigurableApplicationContext context;
public static void main(String[] args) {
context = new SpringApplicationBuilder(Application.class)
.web(false)
.run(args);
}
#Bean
#ServiceActivator(inputChannel = "ftpChannel")
public MessageHandler sourceHandler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
System.out.println("Reply channel isssss:"+message.getHeaders().getReplyChannel());
Object payload = message.getPayload();
System.out.println("Payload: " + payload);
File file = (File) payload;
// enrich header with hash code before sending to target FTP
Message<?> messageOut = MessageBuilder
.withPayload(message.getPayload())
.copyHeadersIfAbsent(message.getHeaders())
.setHeaderIfAbsent("hashCode", hashing.getHashCode(file)).build();
// send to target FTP
System.out.println("Trying to send " + file.getName() + " to target");
MyGateway gateway = context.getBean(MyGateway.class);
gateway.sendToFtp(messageOut);
}
};
}
}
FileTransferServiceConfig.java
#Configuration
#Component
public class FileTransferServiceConfig {
#Autowired
private ConfigurationService configurationService;
#Autowired
private Hashing hashing;
public static final String FILE_POLLING_DURATION = "5000";
#Bean
public SessionFactory<FTPFile> sourceFtpSessionFactory() {
DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
sf.setHost(configurationService.getSourceHostName());
sf.setPort(Integer.parseInt(configurationService.getSourcePort()));
sf.setUsername(configurationService.getSourceUsername());
sf.setPassword(configurationService.getSourcePassword());
return new CachingSessionFactory<>(sf);
}
#Bean
public SessionFactory<FTPFile> targetFtpSessionFactory() {
DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
sf.setHost(configurationService.getTargetHostName());
sf.setPort(Integer.parseInt(configurationService.getTargetPort()));
sf.setUsername(configurationService.getTargetUsername());
sf.setPassword(configurationService.getTargetPassword());
return new CachingSessionFactory<>(sf);
}
#MessagingGateway
public interface MyGateway {
#Gateway(requestChannel = "toFtpChannel")
void sendToFtp(Message message);
}
#Bean
public FtpInboundFileSynchronizer ftpInboundFileSynchronizer() {
FtpInboundFileSynchronizer fileSynchronizer = new FtpInboundFileSynchronizer(sourceFtpSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setRemoteDirectory(configurationService.getSourceDirectory());
fileSynchronizer.setFilter(new FtpSimplePatternFileListFilter(
configurationService.getFileMask()));
return fileSynchronizer;
}
#Bean
public AcceptOnceFileListFilter<File> acceptOnceFileListFilter() {
return new AcceptOnceFileListFilter<>();
}
#Bean
#InboundChannelAdapter(channel = "ftpChannel",
poller = #Poller(fixedDelay = FILE_POLLING_DURATION))
public MessageSource<File> ftpMessageSource() {
FtpInboundFileSynchronizingMessageSource source
= new FtpInboundFileSynchronizingMessageSource(ftpInboundFileSynchronizer());
source.setLocalDirectory(new File(configurationService.getLocalDirectory()));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(acceptOnceFileListFilter());
return source;
}
// makes sure transfer continues on connection reset
#Bean
public Advice expressionAdvice() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setTrapException(true);
advice.setOnFailureExpression("#acceptOnceFileListFilter.remove(payload)");
return advice;
}
#Bean
#ServiceActivator(inputChannel = "toFtpChannel")
public void listenOutboundMessage() {
// tried to subscribe to "toFtpChannel" but this was not triggered
System.out.println("Message received");
}
#Bean
#ServiceActivator(inputChannel = "ftpChannel", adviceChain = "expressionAdvice")
public MessageHandler targetHandler() {
FtpMessageHandler handler = new FtpMessageHandler(targetFtpSessionFactory());
handler.setRemoteDirectoryExpression(new LiteralExpression(
configurationService.getTargetDirectory()));
return handler;
}
}
Hashing.java
public interface Hashing {
public String getHashCode(File payload);
}
I have managed to enrich the message in sourceHandler(), built the message and sent it to the target but I cannot figure out how I can receive that message on the target so that I can get the header from the message?
Tell me if any more information is required. I would really appreciate your help.
You have two subscribers on ftpChannel - the target handler and your sourceHandler; they will get alternate messages unless ftpChannel is declared as a pubsub channel.
There should be no problems with your subscription to toFtpChannel.
Turn on DEBUG logging to see all the subscription activity when the application context starts.
EDIT
Remove the #Bean from the #ServiceActivator - such beans must be a MessageHandler.
#ServiceActivator(inputChannel = "toFtpChannel")
public void listenOutboundMessage(Message message) {
// tried to subscribe to "toFtpChannel" but this was not triggered
System.out.println("Message received:" + message);
}
works fine for me...
Payload: /tmp/foo/baz.txt
Trying to send baz.txt to target
Message received:GenericMessage [payload=/tmp/foo/baz.txt, headers={hashCode=foo, id=410eb9a2-fe8b-ea8a-015a-d5896387cf00, timestamp=1509115006278}]
Again; you must have only one subscriber on ftpChannel unless you make it a pubsub.
I want to download a file from ftp server periodically (only when the file has changed). Therefore I'd like to use Spring-Integration 4.0.
What is the annotation based configuration equivalent of the following setup with int-ftp?
<int-ftp:inbound-channel-adapter id="ftpInbound"
channel="ftpChannel"
session-factory="ftpClientFactory"
filename-pattern="*.txt"
auto-create-local-directory="true"
delete-remote-files="false"
remote-directory="/">
<int:poller fixed-rate="1000"/>
</int-ftp:inbound-channel-adapter>
I started with the following, but don't know how I can configure the channel with its attributes like session-factory, remote-directory etc.
#Configuration
#EnableIntegration
public class Application {
#Bean
public SessionFactory<FTPFile> sessionFactory() {
DefaultFtpSessionFactory ftp = new DefaultFtpSessionFactory();
ftp.setHost("ftp.test");
ftp.setPort(21);
ftp.setUsername("anonymous");
ftp.setPassword("anonymous");
return ftp;
}
#InboundChannelAdapter(value = "ftpChannel", poller = #Poller(fixedDelay = "1000", maxMessagesPerPoll = "1"))
public String connect() {
// return the ftp file
}
#ServiceActivator(inputChannel = "ftpChannel")
public void foo(String payload) {
System.out.println("paylod: " + payload);
}
}
The (S)FTP inbound adapters are on the more complex side; we're working in the DSL to make this easier but, currently, you need an #Bean FtpInboundFileSynchronizer wired up with appropriate properties and inject it into
Then
#Bean
#InboundChannelAdapter(value = "ftpChannel", poller = #Poller(fixedDelay = "1000", maxMessagesPerPoll = "1"))
public MessageSource receive() {
FtpInboundFileSynchronizingMessageSource messageSource = new FtpInboundFileSynchronizingMessageSource(synchronizer());
...
return messageSource;
}