I am using spring integration to connect and download files from ftp.
I have two filters , one by file name and another one to accept only one file using redis.
To the most part it works great however i notice two issues:
Some files are skipped and not downloaded at all
Some files are starting to be written but stop before it finished and left with the .writing temporary file extension - I suspect it occur when i restart my service or when the connection to ftp server is lost.
Below is my configuration for an sftp connection but i also have two more vendors one using ftp and the other ftps who have same problem.
<bean id="eeSftpClientFactory" class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="ftp.host.com"/>
<property name="port" value="22"/>
<property name="user" value="myUserName"/>
<property name="password" value="myPassword"/>
</bean>
<bean id="eeFilesFilter" class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
<list>
<bean class="org.springframework.integration.sftp.filters.SftpPersistentAcceptOnceFileListFilter">
<constructor-arg ref="redisMetadataStore"/>
<constructor-arg value=""/>
</bean>
<bean class="org.springframework.integration.sftp.filters.SftpSimplePatternFileListFilter">
<constructor-arg value="*.nm4"/>
</bean>
</list>
</constructor-arg>
</bean>
<int-sftp:inbound-channel-adapter id="eeChannelAdapter"
channel="eeFtpChannel"
session-factory="eeSftpClientFactory"
auto-startup="${ais.feeds.ee.enabled}"
auto-create-local-directory="true"
delete-remote-files="false"
remote-directory="/SAISData/"
filter="eeFilesFilter"
local-directory="${ais.feeds.base.path}/eeVendor">
<int:poller fixed-delay="500" max-messages-per-poll="-1"/>
</int-sftp:inbound-channel-adapter>
<int:channel id="eeFtpChannel">
<int:queue capacity="500"/>
</int:channel>
<int:service-activator id="eeFeedHandlerActivator"
input-channel="eeFtpChannel"
ref="eeFeedHandler"
method="execute">
<int:poller fixed-delay="500" max-messages-per-poll="-1"/>
</int:service-activator>
Your advice is greatly appriciated!
Found the cause for issue #2 -
The SftpPersistentAcceptOnceFileListFilter check if the file was already processed and adds it to the metadata store - if the process was stopped in the middle due to restart the file isn't rollback from the metadata store so when checking again after restart file already exists in the metadata store and therefore isn't re-downloaded.
Related
My project is using jms+ibmmq to send a message out. it's working properly, and I'm able to push a message to the outbound queue, and then receive a response from the inbound queue. Recently, a client asks if we can send out an alert email if unable to get a response within like 2 min. I checked the spring integration messaging related document, seems gatway can implement it? but still can't get an idea how to use it. can someone help me out?
my current xml configuration
<integration:service-activator id="serviceActivator1" input-channel="inputChannel"
ref="messageProcessService" method="callMsgProcessor" output-channel="requestChannel" />
<bean id="ibmConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory">
<bean class="com.ibm.mq.jms.MQQueueConnectionFactory">
<property name="transportType">
<util:constant static-field="com.ibm.msg.client.wmq.WMQConstants.WMQ_CM_CLIENT"/>
</property>
<property name="hostName" value="${hostName}"/>
<property name="queueManager" value="${queue-manager}"/>
<property name="channel" value="${channel}"/>
<property name="port" value="${port}"/>
</bean>
</property>
<property name="sessionCacheSize" value="5"/>
</bean>
<bean id="requestQueue" class="com.ibm.mq.jms.MQQueue">
<constructor-arg value="${request-queue}"/>
<property name="targetClient">
<util:constant static-field="com.ibm.msg.client.wmq.WMQConstants.WMQ_CLIENT_NONJMS_MQ"/>
</property>
</bean>
<jms:outbound-channel-adapter id="request.queue.adapter" connection-factory="ibmConnectionFactory"
destination="requestQueue" channel="requestChannel"/>
<integration:channel id="requestChannel"/>
<bean id="responseQueue" class="com.ibm.mq.jms.MQQueue">
<constructor-arg value="${response-queue}"/>
</bean>
<jms:message-driven-channel-adapter id="responseJMSAdapater"
destination="responseQueue"
channel="responseChannel"
connection-factory="ibmConnectionFactory"
error-channel="errorChannel"
acknowledge="transacted"
/>
<integration:channel id="responseChannel">
<integration:queue capacity="500"/>
<integration:interceptors>
<integration:wire-tap channel="logger"/>
</integration:interceptors>
</integration:channel>
if add a gateway, it looks like
<integration:gateway id="serviceAdaptedGateway" service-interface="com.mycompany.service.gatewayServiceInterface"
default-request-channel="requestChannel"
default-reply-channel="responseChannel"
default-reply-timeout="120000"/>
my question is
seems timeout not working, nothing happen even the waiting time exceed the timeout
although the gateway provide timeout property, where I can add my business logic like send email alert?
I'm running 4 instances of Spring Boot Integration based apps on 4 differents servers.
The process is :
Read XML files one by one in a shared folder.
Process the file (check structure, content...), transform the data and send email.
Write a report about this file in another shared folder.
Delete successfully processed file.
I'm looking for a non-blocking and safe solution to process theses files.
Use cases :
If an instance crashes while reading or processing a file (so without ending the integration chain) : another instance must process the file or the same instance must process the file after it restarts.
If an instance is processing a file, the others instances must not process the file.
I have built this Spring Integration XML configuration file (it includes JDBC metadatastore with a shared H2 database) :
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-file="http://www.springframework.org/schema/integration/file"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration
http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/file
http://www.springframework.org/schema/integration/file/spring-integration-file.xsd">
<int:poller default="true" fixed-rate="1000"/>
<int:channel id="inputFilesChannel">
<int:queue/>
</int:channel>
<!-- Input -->
<int-file:inbound-channel-adapter
id="inputFilesAdapter"
channel="inputFilesChannel"
directory="file:${input.files.path}"
ignore-hidden="true"
comparator="lastModifiedFileComparator"
filter="compositeFilter">
<int:poller fixed-rate="10000" max-messages-per-poll="1" task-executor="taskExecutor"/>
</int-file:inbound-channel-adapter>
<task:executor id="taskExecutor" pool-size="1"/>
<!-- Metadatastore -->
<bean id="jdbcDataSource" class="org.apache.commons.dbcp.BasicDataSource">
<property name="url" value="jdbc:h2:file:${database.path}/shared;AUTO_SERVER=TRUE;AUTO_RECONNECT=TRUE;MVCC=TRUE"/>
<property name="driverClassName" value="org.h2.Driver"/>
<property name="username" value="${database.username}"/>
<property name="password" value="${database.password}"/>
<property name="maxIdle" value="4"/>
</bean>
<bean id="jdbcMetadataStore" class="org.springframework.integration.jdbc.metadata.JdbcMetadataStore">
<constructor-arg ref="jdbcDataSource"/>
</bean>
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="jdbcDataSource"/>
</bean>
<bean id="compositeFilter" class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
<list>
<bean class="org.springframework.integration.file.filters.FileSystemPersistentAcceptOnceFileListFilter">
<constructor-arg index="0" ref="jdbcMetadataStore"/>
<constructor-arg index="1" value="files"/>
</bean>
</list>
</constructor-arg>
</bean>
<!-- Workflow -->
<int:chain input-channel="inputFilesChannel" output-channel="outputFilesChannel">
<int:service-activator ref="fileActivator" method="fileRead"/>
<int:service-activator ref="fileActivator" method="fileProcess"/>
<int:service-activator ref="fileActivator" method="fileAudit"/>
</int:chain>
<bean id="lastModifiedFileComparator" class="org.apache.commons.io.comparator.LastModifiedFileComparator"/>
<int-file:outbound-channel-adapter
id="outputFilesChannel"
directory="file:${output.files.path}"
filename-generator-expression ="payload.name">
<int-file:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="onSuccessExpressionString" value="headers[file_originalFile].delete()"/>
</bean>
</int-file:request-handler-advice-chain>
</int-file:outbound-channel-adapter>
</beans>
Problem :
With multiple files, when 1 file is successfully processed, the transaction commit the others existing files in the metadatastore (table INT_METADATA_STORE). So if the app is restarted, the other files will never be processed
(it works fine if the app crashes when the first file is being processed).
It seems it only apply for reading files, not for processing files in an integration chain ... How to manage rollback transaction on JVM crash file by file ?
Any help is very appreciated. It's going to make me crazy :(
Thanks !
Edits / Notes :
Inspired from https://github.com/caoimhindenais/spring-integration-files/blob/master/src/main/resources/context.xml
I have updated my configuration with the answer from Artem Bilan. And remove the transactional block in the poller block : I had conflict of transactions between instances (ugly table locks exceptions). Although the behaviour was the same.
I have unsuccessfully tested this configuration in the poller block (same behaviour) :
<int:advice-chain>
<tx:advice id="txAdvice" transaction-manager="transactionManager">
<tx:attributes>
<tx:method name="file*" timeout="30000" propagation="REQUIRED"/>
</tx:attributes>
</tx:advice>
</int:advice-chain>
Maybe a solution based on Idempotent Receiver Enterprise Integration Pattern could work. But I didn't manage to configure it... I don't find precise documentation.
You shouldn't use a PseudoTransactionManager, but DataSourceTransactionManager instead.
Since you use a JdbcMetadataStore, it is going to participate in the transaction and if downstream flow fails, the entry in the metadata store is going to be rolled back as well.
Ok. I found a working solution. Maybe not the cleanest one but it works :
Multi-instances on separate servers, sharing the same H2 database (network folder mount). I think it should work via remote TCP. MVCC has been activated on H2 (check its doc).
inbound-channel-adapter has scan-each-poll option activated to permit repolling files that could be previously ignored (if the process already begun by another instance). So, if another instance crashes, the file can be polled and processed again without restart for this very instance.
Option defaultAutoCommit is set to false on the DB.
I didn't use the FileSystemPersistentAcceptOnceFileListFilter because it was aggregating all read files in the metadatastore when one file get successfully processed. I didn't manage to use it in my context ...
I wrote my own conditions and actions in expressions through filter and transaction synchronization.
<!-- Input -->
<bean id="lastModifiedFileComparator" class="org.apache.commons.io.comparator.LastModifiedFileComparator"/>
<int-file:inbound-channel-adapter
id="inputAdapter"
channel="inputChannel"
directory="file:${input.files.path}"
comparator="lastModifiedFileComparator"
scan-each-poll="true">
<int:poller max-messages-per-poll="1" fixed-rate="5000">
<int:transactional transaction-manager="transactionManager" isolation="READ_COMMITTED" propagation="REQUIRED" timeout="60000" synchronization-factory="syncFactory"/>
</int:poller>
</int-file:inbound-channel-adapter>
<!-- Continue only if the concurrentmetadatastore doesn't contain the file. If if is not the case : insert it in the metadatastore -->
<int:filter input-channel="inputChannel" output-channel="processChannel" discard-channel="nullChannel" throw-exception-on-rejection="false" expression="#jdbcMetadataStore.putIfAbsent(headers[file_name], headers[timestamp]) == null"/>
<!-- Rollback by removing the file from the metadatastore -->
<int:transaction-synchronization-factory id="syncFactory">
<int:after-rollback expression="#jdbcMetadataStore.remove(headers[file_name])" />
</int:transaction-synchronization-factory>
<!-- Metadatastore configuration -->
<bean id="jdbcDataSource" class="org.apache.commons.dbcp.BasicDataSource">
<property name="url" value="jdbc:h2:file:${database.path}/shared;AUTO_SERVER=TRUE;AUTO_RECONNECT=TRUE;MVCC=TRUE"/>
<property name="driverClassName" value="org.h2.Driver"/>
<property name="username" value="${database.username}"/>
<property name="password" value="${database.password}"/>
<property name="maxIdle" value="4"/>
<property name="defaultAutoCommit" value="false"/>
</bean>
<bean id="jdbcMetadataStore" class="org.springframework.integration.jdbc.metadata.JdbcMetadataStore">
<constructor-arg ref="jdbcDataSource"/>
</bean>
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="jdbcDataSource"/>
</bean>
<!-- Workflow -->
<int:chain input-channel="processChannel" output-channel="outputChannel">
<int:service-activator ref="fileActivator" method="fileRead"/>
<int:service-activator ref="fileActivator" method="fileProcess"/>
<int:service-activator ref="fileActivator" method="fileAudit"/>
</int:chain>
<!-- Output -->
<int-file:outbound-channel-adapter
id="outputChannel"
directory="file:${output.files.path}"
filename-generator-expression ="payload.name">
<!-- Delete the source file -->
<int-file:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="onSuccessExpressionString" value="headers[file_originalFile].delete()"/>
</bean>
</int-file:request-handler-advice-chain>
</int-file:outbound-channel-adapter>
Any improvement or other solution is welcome.
I need to Query the data from mongoDB using spring integration, I can able Query the data from MongoDB, but the same data is returned more than once,
<bean id="mongoDBFactory"
class="org.springframework.data.mongodb.core.SimpleMongoDbFactory">
<constructor-arg name="mongo">
<bean class="com.mongodb.Mongo">
<constructor-arg name="host" value="localhost" />
<constructor-arg name="port" value="27017" />
</bean>
</constructor-arg>
<constructor-arg name="databaseName" value="test" />
</bean>
<int:channel id="controlChannel"/>
<int:control-bus input-channel="controlChannel"/>
<int-mongodb:inbound-channel-adapter
id="mongoInboundAdapter" channel="splittingChannel" auto-startup= "false"
query="{_id:1}"
collection-name="order"
mongodb-factory="mongoDBFactory">
<int:poller fixed-rate="10000" max-messages-per-poll="1000"/>
</int-mongodb:inbound-channel-adapter>
<int:splitter input-channel="splittingChannel" output-channel="logger"/>
<int:logging-channel-adapter id="logger" level="WARN"/>
I am using the control channel to start and stop,
please help me how can i stop the inbound-channel-adapter once the Query is completed.
Thanks in advance
I suggest you to use transaction-synchronization-factory to modify or remove documents instead of stopping Adapter. See Reference Manual for more info.
I have a int-sftp:inbound-channel-adapter which uses SftpPersistentAcceptOnceFileListFilter as part of a composite filter. Reading the documentation/ source code it should accept a file to be read again if the modified datetime has changed, but I cant get it to work, it only reads is the once. Im using redis as the store.
Any ideas what is wrong with the configuration, Im using spring integration 4.3.5
<int-sftp:inbound-channel-adapter id="sftpInboundAdapterCensus"
channel="sftpInboundCensus"
session-factory="sftpSessionFactory"
local-directory="${sftp.localdirectory}/census-local"
filter="censusCompositeFilter"
remote-file-separator="/"
remote-directory="${sftp.directory.census}">
<int:poller cron="${sftp.cron}" max-messages-per-poll="1" error-channel="pollerErrorChannel"/>
</int-sftp:inbound-channel-adapter>
<bean id="censusCompositeFilter"
class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
<list>
<bean class="org.springframework.integration.sftp.filters.SftpSimplePatternFileListFilter">
<constructor-arg value="*.xml" />
</bean>
<bean id="SftpPersistentAcceptOnceFileListFilter" class="org.springframework.integration.sftp.filters.SftpPersistentAcceptOnceFileListFilter">
<constructor-arg ref="metadataStore" />
<constructor-arg value="censusSampleLock_" />
</bean>
</list>
</constructor-arg>
</bean>
The SftpPersistentAcceptOnceFileListFilter only controls what we fetch from the server. You also need a FileSystemPersistentAcceptOnceFileListFilter in the local-filter (which determines which files that have been fetched end up being emitted as messages). The local filter is an AcceptOnceFileListFilter by default.
I have to poll an ftp location. For testing purpose I have created an ftp site on my machine using IIS manager. It listens at port 21 and is started.
The dependancies are proper for my project
This is the xml configuration for spring ftp
<bean id="ftpClientFactory"
class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="localhost"/>
<property name="port" value="21"/>
<property name="username" value="ICMAS"/>
<property name="password" value="kavita12"/>
<property name="clientMode" value="0"/>
<property name="fileType" value="2"/>
<property name="bufferSize" value="100000"/>
</bean>
<int-ftp:inbound-channel-adapter id="ftpInbound"
channel="ftpChannel"
session-factory="ftpClientFactory"
charset="UTF-8"
auto-create-local-directory="true"
delete-remote-files="true"
local-filter="compositeFilter"
remote-directory="c:\ftproot"
remote-file-separator="\"
preserve-timestamp="true"
local-directory="c:\data"
>
<int:poller fixed-rate="1000"/>
</int-ftp:inbound-channel-adapter>
<int:channel id="ftpChannel"/>
The filenamegenerator and the compositefilter are present in my code but havent pated their code here.
My problem is that the local-directory is getting polled instead of the remote-directory. I thought that the files are read from the remote-directory location the go to the filter and if successful will go to the filenamegenerator and be put in the local-directory location. What is wrong with this code???
Please correct me if I am doing something wrong.
Need help on this issue... Please put in your suggessions!!
Have resolved this issue.
firstly I needed the filter attribute rather than the local-filter as there is a difference in them.
Secondly and more importantly I have given the romote-directory location as the absolute path. That needs to be relative to the ftp directory mentioned while creating the ftp site.
Thanks. Hope this is useful to someone!!