I have an Ftp Configuration file like
public class FtpConfig {
#Bean
public DefaultFtpSessionFactory ftpSessionFactory() {
DefaultFtpSessionFactory sessionFactory = new DefaultFtpSessionFactory();
sessionFactory.setHost("host");
sessionFactory.setPort(21);
sessionFactory.setUsername("ftpuser");
sessionFactory.setPassword("pass");
return sessionFactory;
}
#ServiceActivator(inputChannel = "fromFtpChannel")
#Bean
public FtpOutboundGateway getFiles() {
FtpOutboundGateway gateway = new FtpOutboundGateway(ftpSessionFactory(), "get", "payload");
gateway.setOptions(AbstractRemoteFileOutboundGateway.Option.STREAM.getOption());
gateway.setOutputChannelName("fileoutput");
return gateway;
}
}
And a Messaging Gateway
#MessagingGateway
public interface ReadFilesGateway {
#Gateway(requestChannel = "fromFtpChannel", replyChannel = "fileoutput")
InputStream readFiles(String directory);
}
The Messaging gateway is used to download files from the ftp server, to view the files as inputStream. A service FileService uses the gateway to read the files. How to write a possible testcase for the service, or the Messaging Gateway?
Related
My application is based on spring-boot 2.1.6 equipped with spring-batch (chunks approach) and spring-integration to handle the SFTP.
High level functionality is to fetch data from DB, generate a text file then send it through SFTP and this task run every 30 mins.
This application already running in production for some time, but if I see the logs there are error about ssh_msg_disconnect 11 idle connection. It will keep like that until I restart the app.
Below is my application code :
SftpConfig.java
#Configuration
public class SftpConfig {
#Autowired
ApplicationProperties applicationProperties;
#Bean
public SessionFactory<LsEntry> sftpSessionFactory() {
final DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(applicationProperties.getSftp().getHost());
factory.setUser(applicationProperties.getSftp().getUser());
factory.setPassword(applicationProperties.getSftp().getPass());
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<>(factory);
}
#Bean
#ServiceActivator(inputChannel = "toSftpChannel", adviceChain = "retryAdvice")
public MessageHandler handler() {
final SftpMessageHandler handler = new SftpMessageHandler(this.sftpSessionFactory());
handler.setRemoteDirectoryExpression(new LiteralExpression(applicationProperties.getSftp().getPath()));
handler.setFileNameGenerator((final Message<?> message) -> {
if (message.getPayload() instanceof File) {
return ((File) message.getPayload()).getName();
} else {
throw new IllegalArgumentException("File expected as payload.");
}
});
return handler;
}
#Bean
public RequestHandlerRetryAdvice retryAdvice() {
final RequestHandlerRetryAdvice advice = new RequestHandlerRetryAdvice();
final RetryTemplate retryTemplate = new RetryTemplate();
final SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy();
retryPolicy.setMaxAttempts(NumberConstants.FIVE);
retryTemplate.setRetryPolicy(retryPolicy);
advice.setRetryTemplate(retryTemplate);
return advice;
}
#MessagingGateway
public interface UploadGateway {
#Gateway(requestChannel = "toSftpChannel")
void upload(File file);
}
}
step for sending file to sftp
#Autowired
UploadGateway uploadGateway;
private boolean uploadToSharedFolderSuccess(final PaymentStatus paymentStatus, final String strLocalTmpPath) {
try {
final File fileLocalTmpFullPath = new File(strLocalTmpPath);
uploadGateway.upload(fileLocalTmpFullPath);
} catch (final Exception e) {
paymentStatus.setStatus(ProcessStatus.ERROR.toString());
paymentStatus.setRemark(StringUtil.appendIfNotEmpty(paymentStatus.getRemark(),
"Error during upload to shared folder - " + e.getMessage()));
}
return !StringUtils.equalsIgnoreCase(ProcessStatus.ERROR.toString(), paymentStatus.getStatus());
}
From the error, I know that seems like I opened too many connection. But I'm not sure how to check if the connection are closed every end of the spring-batch.
If you don't wrap the session factory in a CachingSessionFactory, the session will be closed after each use.
#Bean
public DefaultSftpSessionFactory sftpSessionFactory() {
final DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(applicationProperties.getSftp().getHost());
factory.setUser(applicationProperties.getSftp().getUser());
factory.setPassword(applicationProperties.getSftp().getPass());
factory.setAllowUnknownKeys(true);
return factory;
}
I am reading mails using spring mail inbound channel adapter once message is read i am performing some db operations in service activator of corresponding channel. My requirement is if any db operation fails adapter should read same message again.
Mail configuration :
#Bean
public DirectChannel inputChannel() {
return new DirectChannel();
}
#Bean
public IntegrationFlow pop3MailFlow() {
String url = "[url]";
return IntegrationFlows
.from(Mail.pop3InboundAdapter(url)
.javaMailProperties(p -> p.put("mail.pop3.socketFactory.class", "javax.net.ssl.SSLSocketFactory")),e -> e.autoStartup(true)
.poller(Pollers.fixedDelay(2000).transactionSynchronizationFactory(transactionSynchronizationFactory())))
.channel(inputChannel())
.handle(inboundEmailProcessor(),"messageProcess")
.get();
}
#Bean
public InboundEmailProcessor inboundEmailProcessor() {
return new InboundEmailProcessor();
}
#Bean
public TransactionSynchronizationFactory transactionSynchronizationFactory() {
TransactionSynchronizationFactory synchronizationFactory = new DefaultTransactionSynchronizationFactory(expressionEvaluatingTransactionSynchronizationProcessor());
return synchronizationFactory;
}
#Bean
public ExpressionEvaluatingTransactionSynchronizationProcessor expressionEvaluatingTransactionSynchronizationProcessor() {
ExpressionEvaluatingTransactionSynchronizationProcessor processor = new ExpressionEvaluatingTransactionSynchronizationProcessor();
ExpressionParser parser = new SpelExpressionParser();
processor.setAfterRollbackExpression(parser.parseExpression("new com.muraai.ex.config.Exp().process(payload)"));
return processor;
}
public class InboundEmailProcessor {
#Autowired
AttachmentsRepository attachmentsRepository;
#Transactional(rollbackFor = Exception.class)
public void messageProcess() {
// some db operations
// if it fails the same message should be read again
}
}
I thought this would work but its not working. Is there any way to achieve my requirement
public class Exp {
public void process(MimeMessage message) throws MessagingException {
message.setFlag(Flags.Flag.SEEN, false);
}
}
You need IMAP for that; with POP3, the server always marks them read.
You can add a spring-retry interceptor advice to the poller's advice chain and/or send the failed message to an error channel.
The retry advice can be configured for number of retries, back off policy etc.
I have successfully been able send file from one FTP Server(source) to another FTP server (target). I first send files from source to the local directory using the inbound adapter and then send files from the local directory to the target using the outbound adapter. So far this is working fine.
What I want to achieve is: to enrich the header of the message at the source with a hash code (which is generated using the file on source that is transferred) and then get that header at the target and match it with the hash code (which is generated using the file on the target)
Here is what I have tried so far:
Application.java
#SpringBootApplication
public class Application {
#Autowired
private Hashing hashing;
public static ConfigurableApplicationContext context;
public static void main(String[] args) {
context = new SpringApplicationBuilder(Application.class)
.web(false)
.run(args);
}
#Bean
#ServiceActivator(inputChannel = "ftpChannel")
public MessageHandler sourceHandler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
System.out.println("Reply channel isssss:"+message.getHeaders().getReplyChannel());
Object payload = message.getPayload();
System.out.println("Payload: " + payload);
File file = (File) payload;
// enrich header with hash code before sending to target FTP
Message<?> messageOut = MessageBuilder
.withPayload(message.getPayload())
.copyHeadersIfAbsent(message.getHeaders())
.setHeaderIfAbsent("hashCode", hashing.getHashCode(file)).build();
// send to target FTP
System.out.println("Trying to send " + file.getName() + " to target");
MyGateway gateway = context.getBean(MyGateway.class);
gateway.sendToFtp(messageOut);
}
};
}
}
FileTransferServiceConfig.java
#Configuration
#Component
public class FileTransferServiceConfig {
#Autowired
private ConfigurationService configurationService;
#Autowired
private Hashing hashing;
public static final String FILE_POLLING_DURATION = "5000";
#Bean
public SessionFactory<FTPFile> sourceFtpSessionFactory() {
DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
sf.setHost(configurationService.getSourceHostName());
sf.setPort(Integer.parseInt(configurationService.getSourcePort()));
sf.setUsername(configurationService.getSourceUsername());
sf.setPassword(configurationService.getSourcePassword());
return new CachingSessionFactory<>(sf);
}
#Bean
public SessionFactory<FTPFile> targetFtpSessionFactory() {
DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
sf.setHost(configurationService.getTargetHostName());
sf.setPort(Integer.parseInt(configurationService.getTargetPort()));
sf.setUsername(configurationService.getTargetUsername());
sf.setPassword(configurationService.getTargetPassword());
return new CachingSessionFactory<>(sf);
}
#MessagingGateway
public interface MyGateway {
#Gateway(requestChannel = "toFtpChannel")
void sendToFtp(Message message);
}
#Bean
public FtpInboundFileSynchronizer ftpInboundFileSynchronizer() {
FtpInboundFileSynchronizer fileSynchronizer = new FtpInboundFileSynchronizer(sourceFtpSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setRemoteDirectory(configurationService.getSourceDirectory());
fileSynchronizer.setFilter(new FtpSimplePatternFileListFilter(
configurationService.getFileMask()));
return fileSynchronizer;
}
#Bean
public AcceptOnceFileListFilter<File> acceptOnceFileListFilter() {
return new AcceptOnceFileListFilter<>();
}
#Bean
#InboundChannelAdapter(channel = "ftpChannel",
poller = #Poller(fixedDelay = FILE_POLLING_DURATION))
public MessageSource<File> ftpMessageSource() {
FtpInboundFileSynchronizingMessageSource source
= new FtpInboundFileSynchronizingMessageSource(ftpInboundFileSynchronizer());
source.setLocalDirectory(new File(configurationService.getLocalDirectory()));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(acceptOnceFileListFilter());
return source;
}
// makes sure transfer continues on connection reset
#Bean
public Advice expressionAdvice() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setTrapException(true);
advice.setOnFailureExpression("#acceptOnceFileListFilter.remove(payload)");
return advice;
}
#Bean
#ServiceActivator(inputChannel = "toFtpChannel")
public void listenOutboundMessage() {
// tried to subscribe to "toFtpChannel" but this was not triggered
System.out.println("Message received");
}
#Bean
#ServiceActivator(inputChannel = "ftpChannel", adviceChain = "expressionAdvice")
public MessageHandler targetHandler() {
FtpMessageHandler handler = new FtpMessageHandler(targetFtpSessionFactory());
handler.setRemoteDirectoryExpression(new LiteralExpression(
configurationService.getTargetDirectory()));
return handler;
}
}
Hashing.java
public interface Hashing {
public String getHashCode(File payload);
}
I have managed to enrich the message in sourceHandler(), built the message and sent it to the target but I cannot figure out how I can receive that message on the target so that I can get the header from the message?
Tell me if any more information is required. I would really appreciate your help.
You have two subscribers on ftpChannel - the target handler and your sourceHandler; they will get alternate messages unless ftpChannel is declared as a pubsub channel.
There should be no problems with your subscription to toFtpChannel.
Turn on DEBUG logging to see all the subscription activity when the application context starts.
EDIT
Remove the #Bean from the #ServiceActivator - such beans must be a MessageHandler.
#ServiceActivator(inputChannel = "toFtpChannel")
public void listenOutboundMessage(Message message) {
// tried to subscribe to "toFtpChannel" but this was not triggered
System.out.println("Message received:" + message);
}
works fine for me...
Payload: /tmp/foo/baz.txt
Trying to send baz.txt to target
Message received:GenericMessage [payload=/tmp/foo/baz.txt, headers={hashCode=foo, id=410eb9a2-fe8b-ea8a-015a-d5896387cf00, timestamp=1509115006278}]
Again; you must have only one subscriber on ftpChannel unless you make it a pubsub.
We are using the latest Spring Boot for a Spring app and using the latest Spring Integration for SFTP. I've been to the Spring Integration SFTP documentation site, and I took the Spring Boot Configuration as is:
#Bean
public SessionFactory<LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost("localhost");
factory.setPort(port);
factory.setUser("foo");
factory.setPassword("foo");
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<LsEntry>(factory);
}
#Bean
public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
SftpInboundFileSynchronizer fileSynchronizer = new SftpInboundFileSynchronizer(sftpSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setRemoteDirectory("/");
fileSynchronizer.setFilter(new SftpSimplePatternFileListFilter("*.xml"));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "sftpChannel")
public MessageSource<File> sftpMessageSource() {
SftpInboundFileSynchronizingMessageSource source =
new SftpInboundFileSynchronizingMessageSource(sftpInboundFileSynchronizer());
source.setLocalDirectory(new File("ftp-inbound"));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
return source;
}
#Bean
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
System.out.println(message.getPayload());
}
};
}
Let me be clear, after cutting and pasting, there are some unit tests that run. However, when loading the application context there was an error message because the Polling wasn't there.
When I googled that error, other posts on StackOverflow said I also had to add to remove this error message when loading the application context.
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata defaultPoller() {
PollerMetadata pollerMetadata = new PollerMetadata();
pollerMetadata.setTrigger(new PeriodicTrigger(60));
return pollerMetadata;
}
When I added this code, THEN at least my build would work and the tests would run because the application context was now being loaded correctly.
Now I am looking for a code sample on how to make this work and move files? The Spring Integration SFTP examples on GitHub are ok, but not great ... far from it.
The Basic Spring Integration Example shows how to read files from an SFTP Server, if the data is configured with an application-context.xml file. Where is the example where a Spring Boot configuration is used, and then the code to read from that server, and the code for the test?
I understand that regardless of whether you use a Java class for Spring Boot configuration or an application-context.xml file ... the working code should work the same for autowired SFTP channels and some inbound channel adapter.
So here is the code, I am trying to make work:
#Component
#Profile("sftpInputFetch")
public class SFTPInputFetcher implements InputFetcher
{
// The PollableChannel seems fine
#Autowired
PollableChannel sftpChannel;
#Autowired
SourcePollingChannelAdapter sftpChannelAdapter;
#Override
public Stream<String> fetchLatest() throws FileNotFoundException
{
Stream<String> stream = null;
sftpChannelAdapter.start();
Message<?> received = sftpChannel.receive();
File file = (File)received.getPayload();
// get Stream<String> from file
return stream;
}
Currently, "sftpChannelAdapter.start();" is the part I am having trouble with.
This implementation does not find the "SourcePollingChannelAdapter" class.
If this was defined in the classic XML application context with an "id" then this code autowires just fine. With a Spring Boot configuration, it doesn't look like you can define an "id" for a bean.
This just stems from my lack of knowledge on how to convert from using a traditional application-context XML file WITH annotations in the code, to using a complete Spring Boot application context configuration file.
Any help with this is much appreciated. Thanks!
I don't understand the question; you said
I had to add ... to make it work
and then
Now I am looking for a code sample on how to make this work?
What is not working?
You can also use
#InboundChannelAdapter(value = "sftpChannel", poller = #Poller(fixedDelay = "5000"))
instead of adding a default poller definition.
We will fix the docs for the missing poller config.
EDIT
I just copied the code into a new boot app (with the poller config) and it works as expected.
#SpringBootApplication
public class SftpJavaApplication {
public static void main(String[] args) {
new SpringApplicationBuilder(SftpJavaApplication.class).web(false).run(args);
}
#Bean
public SessionFactory<LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost("...");
factory.setPort(22);
factory.setUser("...");
factory.setPassword("...");
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<LsEntry>(factory);
}
#Bean
public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
SftpInboundFileSynchronizer fileSynchronizer = new SftpInboundFileSynchronizer(sftpSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setRemoteDirectory("foo");
fileSynchronizer.setFilter(new SftpSimplePatternFileListFilter("*.txt"));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "sftpChannel", poller = #Poller(fixedDelay = "5000"))
public MessageSource<File> sftpMessageSource() {
SftpInboundFileSynchronizingMessageSource source = new SftpInboundFileSynchronizingMessageSource(
sftpInboundFileSynchronizer());
source.setLocalDirectory(new File("ftp-inbound"));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
return source;
}
#Bean
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
System.out.println(message.getPayload());
}
};
}
}
Result:
16:57:59.697 [task-scheduler-1] WARN com.jcraft.jsch - Permanently added '10.0.0.3' (RSA) to the list of known hosts.
ftp-inbound/bar.txt
ftp-inbound/baz.txt
i want to use activiti in spring mvc. my java config for activiti is below.
when i run project this exception no processes deployed with key throw. i put one-vacation-process.bpmn20.xml in resources folder. what is my problem? thanks for help.
#Configuration
public class ActivitiConfig {
#Bean
public ProcessEngine processEngine(ProcessEngineConfigurationImpl pec, ApplicationContext applicationContext) throws Exception {
ProcessEngineFactoryBean pe = new ProcessEngineFactoryBean();
pe.setProcessEngineConfiguration(pec);
pe.setApplicationContext(applicationContext);
return pe.getObject();
}
#Bean
public ProcessEngineConfigurationImpl getProcessEngineConfiguration(
DataSource dataSource,
PlatformTransactionManager transactionManager,
ApplicationContext context) {
SpringProcessEngineConfiguration pec = new SpringProcessEngineConfiguration();
pec.setDataSource(dataSource);
pec.setDatabaseSchemaUpdate("true");
pec.setJobExecutorActivate(true);
pec.setHistory("full");
pec.setMailServerPort(2025);
pec.setDatabaseType("mysql");
pec.setTransactionManager(transactionManager);
pec.setApplicationContext(context);
return pec;
}
#Bean
public RuntimeService getRuntimeService(ProcessEngine processEngine) {
return processEngine.getRuntimeService();
}
#Bean
public TaskService taskService(ProcessEngine processEngine) throws Exception {
return processEngine.getTaskService();
}
You need to deploy your process first.
There is API for different usecases, here I deploy a process where resourceName is the name of the process xml (e.g. one-vacation-process.bpmn20.xml) and content the actual file content as string.
RepositoryService repositoryService = processEngine.getRepositoryService();
DeploymentBuilder builder = repositoryService.createDeployment().addString(resourceName, content);
builder.enableDuplicateFiltering().deploy();
Have a look at org.activiti.engine.repository.DeploymentBuilder where there is API like:
DeploymentBuilder addInputStream(String resourceName, InputStream inputStream);
DeploymentBuilder addClasspathResource(String resource);
DeploymentBuilder addString(String resourceName, String text);
DeploymentBuilder addZipInputStream(ZipInputStream zipInputStream);
DeploymentBuilder addBpmnModel(String resourceName, BpmnModel bpmnModel);