I'm running 4 instances of Spring Boot Integration based apps on 4 differents servers.
The process is :
Read XML files one by one in a shared folder.
Process the file (check structure, content...), transform the data and send email.
Write a report about this file in another shared folder.
Delete successfully processed file.
I'm looking for a non-blocking and safe solution to process theses files.
Use cases :
If an instance crashes while reading or processing a file (so without ending the integration chain) : another instance must process the file or the same instance must process the file after it restarts.
If an instance is processing a file, the others instances must not process the file.
I have built this Spring Integration XML configuration file (it includes JDBC metadatastore with a shared H2 database) :
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-file="http://www.springframework.org/schema/integration/file"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration
http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/file
http://www.springframework.org/schema/integration/file/spring-integration-file.xsd">
<int:poller default="true" fixed-rate="1000"/>
<int:channel id="inputFilesChannel">
<int:queue/>
</int:channel>
<!-- Input -->
<int-file:inbound-channel-adapter
id="inputFilesAdapter"
channel="inputFilesChannel"
directory="file:${input.files.path}"
ignore-hidden="true"
comparator="lastModifiedFileComparator"
filter="compositeFilter">
<int:poller fixed-rate="10000" max-messages-per-poll="1" task-executor="taskExecutor"/>
</int-file:inbound-channel-adapter>
<task:executor id="taskExecutor" pool-size="1"/>
<!-- Metadatastore -->
<bean id="jdbcDataSource" class="org.apache.commons.dbcp.BasicDataSource">
<property name="url" value="jdbc:h2:file:${database.path}/shared;AUTO_SERVER=TRUE;AUTO_RECONNECT=TRUE;MVCC=TRUE"/>
<property name="driverClassName" value="org.h2.Driver"/>
<property name="username" value="${database.username}"/>
<property name="password" value="${database.password}"/>
<property name="maxIdle" value="4"/>
</bean>
<bean id="jdbcMetadataStore" class="org.springframework.integration.jdbc.metadata.JdbcMetadataStore">
<constructor-arg ref="jdbcDataSource"/>
</bean>
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="jdbcDataSource"/>
</bean>
<bean id="compositeFilter" class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
<list>
<bean class="org.springframework.integration.file.filters.FileSystemPersistentAcceptOnceFileListFilter">
<constructor-arg index="0" ref="jdbcMetadataStore"/>
<constructor-arg index="1" value="files"/>
</bean>
</list>
</constructor-arg>
</bean>
<!-- Workflow -->
<int:chain input-channel="inputFilesChannel" output-channel="outputFilesChannel">
<int:service-activator ref="fileActivator" method="fileRead"/>
<int:service-activator ref="fileActivator" method="fileProcess"/>
<int:service-activator ref="fileActivator" method="fileAudit"/>
</int:chain>
<bean id="lastModifiedFileComparator" class="org.apache.commons.io.comparator.LastModifiedFileComparator"/>
<int-file:outbound-channel-adapter
id="outputFilesChannel"
directory="file:${output.files.path}"
filename-generator-expression ="payload.name">
<int-file:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="onSuccessExpressionString" value="headers[file_originalFile].delete()"/>
</bean>
</int-file:request-handler-advice-chain>
</int-file:outbound-channel-adapter>
</beans>
Problem :
With multiple files, when 1 file is successfully processed, the transaction commit the others existing files in the metadatastore (table INT_METADATA_STORE). So if the app is restarted, the other files will never be processed
(it works fine if the app crashes when the first file is being processed).
It seems it only apply for reading files, not for processing files in an integration chain ... How to manage rollback transaction on JVM crash file by file ?
Any help is very appreciated. It's going to make me crazy :(
Thanks !
Edits / Notes :
Inspired from https://github.com/caoimhindenais/spring-integration-files/blob/master/src/main/resources/context.xml
I have updated my configuration with the answer from Artem Bilan. And remove the transactional block in the poller block : I had conflict of transactions between instances (ugly table locks exceptions). Although the behaviour was the same.
I have unsuccessfully tested this configuration in the poller block (same behaviour) :
<int:advice-chain>
<tx:advice id="txAdvice" transaction-manager="transactionManager">
<tx:attributes>
<tx:method name="file*" timeout="30000" propagation="REQUIRED"/>
</tx:attributes>
</tx:advice>
</int:advice-chain>
Maybe a solution based on Idempotent Receiver Enterprise Integration Pattern could work. But I didn't manage to configure it... I don't find precise documentation.
You shouldn't use a PseudoTransactionManager, but DataSourceTransactionManager instead.
Since you use a JdbcMetadataStore, it is going to participate in the transaction and if downstream flow fails, the entry in the metadata store is going to be rolled back as well.
Ok. I found a working solution. Maybe not the cleanest one but it works :
Multi-instances on separate servers, sharing the same H2 database (network folder mount). I think it should work via remote TCP. MVCC has been activated on H2 (check its doc).
inbound-channel-adapter has scan-each-poll option activated to permit repolling files that could be previously ignored (if the process already begun by another instance). So, if another instance crashes, the file can be polled and processed again without restart for this very instance.
Option defaultAutoCommit is set to false on the DB.
I didn't use the FileSystemPersistentAcceptOnceFileListFilter because it was aggregating all read files in the metadatastore when one file get successfully processed. I didn't manage to use it in my context ...
I wrote my own conditions and actions in expressions through filter and transaction synchronization.
<!-- Input -->
<bean id="lastModifiedFileComparator" class="org.apache.commons.io.comparator.LastModifiedFileComparator"/>
<int-file:inbound-channel-adapter
id="inputAdapter"
channel="inputChannel"
directory="file:${input.files.path}"
comparator="lastModifiedFileComparator"
scan-each-poll="true">
<int:poller max-messages-per-poll="1" fixed-rate="5000">
<int:transactional transaction-manager="transactionManager" isolation="READ_COMMITTED" propagation="REQUIRED" timeout="60000" synchronization-factory="syncFactory"/>
</int:poller>
</int-file:inbound-channel-adapter>
<!-- Continue only if the concurrentmetadatastore doesn't contain the file. If if is not the case : insert it in the metadatastore -->
<int:filter input-channel="inputChannel" output-channel="processChannel" discard-channel="nullChannel" throw-exception-on-rejection="false" expression="#jdbcMetadataStore.putIfAbsent(headers[file_name], headers[timestamp]) == null"/>
<!-- Rollback by removing the file from the metadatastore -->
<int:transaction-synchronization-factory id="syncFactory">
<int:after-rollback expression="#jdbcMetadataStore.remove(headers[file_name])" />
</int:transaction-synchronization-factory>
<!-- Metadatastore configuration -->
<bean id="jdbcDataSource" class="org.apache.commons.dbcp.BasicDataSource">
<property name="url" value="jdbc:h2:file:${database.path}/shared;AUTO_SERVER=TRUE;AUTO_RECONNECT=TRUE;MVCC=TRUE"/>
<property name="driverClassName" value="org.h2.Driver"/>
<property name="username" value="${database.username}"/>
<property name="password" value="${database.password}"/>
<property name="maxIdle" value="4"/>
<property name="defaultAutoCommit" value="false"/>
</bean>
<bean id="jdbcMetadataStore" class="org.springframework.integration.jdbc.metadata.JdbcMetadataStore">
<constructor-arg ref="jdbcDataSource"/>
</bean>
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="jdbcDataSource"/>
</bean>
<!-- Workflow -->
<int:chain input-channel="processChannel" output-channel="outputChannel">
<int:service-activator ref="fileActivator" method="fileRead"/>
<int:service-activator ref="fileActivator" method="fileProcess"/>
<int:service-activator ref="fileActivator" method="fileAudit"/>
</int:chain>
<!-- Output -->
<int-file:outbound-channel-adapter
id="outputChannel"
directory="file:${output.files.path}"
filename-generator-expression ="payload.name">
<!-- Delete the source file -->
<int-file:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="onSuccessExpressionString" value="headers[file_originalFile].delete()"/>
</bean>
</int-file:request-handler-advice-chain>
</int-file:outbound-channel-adapter>
Any improvement or other solution is welcome.
Related
I'm new in Spring. Recently I do try to make spring batch and spring integration work together. I want to have JobListener which will listen for message comes to specific channel and launch Spring Batch Job.
I found example on github(https://github.com/chrisjs/spring-batch-scaling/tree/master/message-job-launch) and I tried to configure some way copy Spring Batch and Spring Integration together and this looks like:
<!--Incomming channel OneToOne-->
<int:channel id="requests-channel"/>
<!--For multiple consumers OneToMany-->
<int:publish-subscribe-channel id="reply-channel"/>
<!--Channel for file adapter-->
<int:channel id="file-adapter-reply-channel"/>
<int:channel id="statuses">
<int:queue capacity="10"/>
</int:channel>
<int:channel id="jobLaunchReplyChannel"/>
<!--Intercept request-->
<int-http:inbound-gateway request-channel="requests-channel"
supported-methods="PUT"
path="/testData/setProfileDescription"
reply-timeout="30000"
reply-channel="reply-channel">
</int-http:inbound-gateway>
<!--Sending HTTP response back to user OR either 'no reply received within timeout'-->
<bean id="profileDescriptionActivator"
class="ru.tcsbank.service.integrations.activators.ProfileDescriptionActivator"/>
<int:service-activator ref="profileDescriptionActivator"
input-channel="requests-channel"
output-channel="reply-channel"
method="httpMessageActivator"/>
<!--Write profile description to file-->
<bean id="custom-file-name-generator"
class="ru.tcsbank.service.integrations.transformers_generators.ProfilesFileAdapterNameGenerator"/>
<file:outbound-channel-adapter channel="file-adapter-reply-channel"
directory="file:out"
filename-generator="custom-file-name-generator"/>
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" lazy-init="true" destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url" value="jdbc:mysql://localhost:3306/testdb"/>
<property name="username" value="test_user"/>
<property name="password" value="qwerty123"/>
</bean>
<bean id="stepScope" class="org.springframework.batch.core.scope.StepScope">
<property name="autoProxy" value="true"/>
</bean>
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource"/>
</bean>
<bean id="jobRepositoryInDB" class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean">
<property name="dataSource" ref="dataSource"/>
<property name="transactionManager" ref="transactionManager"/>
</bean>
<bean id="itemProcessor" class="ru.tcsbank.service.batch_processing.CustomItemProcessor"/>
<bean id="itemReader" class="ru.tcsbank.service.batch_processing.CustomReader" scope="step">
<property name="resource" value="classpath:fileOut/*.csv" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value=","/>
<property name="names" value="id,firstName,lastName"/>
</bean>
</property>
<property name="fieldSetMapper">
<bean class="ru.tcsbank.service.batch_processing.ProfileDescriptionLineMapper"/>
</property>
</bean>
</property>
</bean>
<bean id="itemWriter" class="ru.tcsbank.service.batch_processing.CustomWriter"/>
<batch:job id="helloWorldJob" job-repository="jobRepositoryInDB">
<batch:listeners>
<batch:listener ref="jobListener"/>
</batch:listeners>
<batch:step id="step1">
<batch:tasklet>
<batch:chunk reader="itemReader" writer="itemWriter" processor="itemProcessor" commit-interval="10"/>
</batch:tasklet>
</batch:step>
</batch:job>
<int:transformer input-channel="reply-channel" output-channel="file-adapter-reply-channel">
<bean class="ru.tcsbank.service.batch_processing.FileMessageToJobRequest">
<property name="job" ref="helloWorldJob"/>
<property name="fileParameterName" value="input.file.name"/>
</bean>
</int:transformer>
<bean id="jobListener" class="ru.tcsbank.service.batch_processing.CustomJobExecutionListener">
<constructor-arg index="0" ref="notificationSender"/>
</bean>
<batch-int:job-launching-gateway request-channel="reply-channel"
reply-channel="file-adapter-reply-channel"/>
<int:logging-channel-adapter channel="jobLaunchReplyChannel"/>
<int:channel id="notificationsChannel"/>
<int:gateway id="notificationSender"
service-interface="ru.tcsbank.service.batch_processing.NotificationSender"
default-request-channel="notificationsChannel"/>
I expect my helloWorldJob run when(as I understand correctly) my jobListener receives message from notificationsChannel. But it do not work(do not receives message from notificationsChannel) Beyond then it throws error like:
Dispatcher has no subscribers for channel
'application.notificationsChannel'.; nested exception is >org.springframework.integration.MessageDispatchingException: Dispatcher >has no subscribers, failedMessage=GenericMessage [payload=TEST. >Image processing job ran for: 0 minutes, 0 seconds.
It's hard to understand what you would like to achieve with all this custom code, but what I can say, that there is no subscribers for that notificationsChannel in your configuration. You indeed send messages to it via notificationSender gateway, but you don't provide any endpoint to consume that notificationsChannel.
In the sample you mention in the link we have something like this:
<int-jms:outbound-channel-adapter id="notifications" destination-name="notifications"
channel="notificationsChannel"/>
So, messages sent to the notificationsChannel are landed in the notifications queue on JMS broker. Your sample are leaking such a subscriber. Therefore I only can explain a reason of the exception, but definitely can't tell you what do to.
UPDATE
You may not use notificationSender in your solution. Looks like it just a result of the CustomJobExecutionListener. So, if you don't need to listen for job process, just remove that CustomJobExecutionListener and, therefore, this notificationSender declaration together with the notificationsChannel definition.
Everything else you are asking in comments is out of the scope of this SO question. Please, consider to raise those concerns in the separate SO thread.
I am using spring integration to connect and download files from ftp.
I have two filters , one by file name and another one to accept only one file using redis.
To the most part it works great however i notice two issues:
Some files are skipped and not downloaded at all
Some files are starting to be written but stop before it finished and left with the .writing temporary file extension - I suspect it occur when i restart my service or when the connection to ftp server is lost.
Below is my configuration for an sftp connection but i also have two more vendors one using ftp and the other ftps who have same problem.
<bean id="eeSftpClientFactory" class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="ftp.host.com"/>
<property name="port" value="22"/>
<property name="user" value="myUserName"/>
<property name="password" value="myPassword"/>
</bean>
<bean id="eeFilesFilter" class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
<list>
<bean class="org.springframework.integration.sftp.filters.SftpPersistentAcceptOnceFileListFilter">
<constructor-arg ref="redisMetadataStore"/>
<constructor-arg value=""/>
</bean>
<bean class="org.springframework.integration.sftp.filters.SftpSimplePatternFileListFilter">
<constructor-arg value="*.nm4"/>
</bean>
</list>
</constructor-arg>
</bean>
<int-sftp:inbound-channel-adapter id="eeChannelAdapter"
channel="eeFtpChannel"
session-factory="eeSftpClientFactory"
auto-startup="${ais.feeds.ee.enabled}"
auto-create-local-directory="true"
delete-remote-files="false"
remote-directory="/SAISData/"
filter="eeFilesFilter"
local-directory="${ais.feeds.base.path}/eeVendor">
<int:poller fixed-delay="500" max-messages-per-poll="-1"/>
</int-sftp:inbound-channel-adapter>
<int:channel id="eeFtpChannel">
<int:queue capacity="500"/>
</int:channel>
<int:service-activator id="eeFeedHandlerActivator"
input-channel="eeFtpChannel"
ref="eeFeedHandler"
method="execute">
<int:poller fixed-delay="500" max-messages-per-poll="-1"/>
</int:service-activator>
Your advice is greatly appriciated!
Found the cause for issue #2 -
The SftpPersistentAcceptOnceFileListFilter check if the file was already processed and adds it to the metadata store - if the process was stopped in the middle due to restart the file isn't rollback from the metadata store so when checking again after restart file already exists in the metadata store and therefore isn't re-downloaded.
I have a successfully running ActiveMQ 5.9.1 , Camel 2.11 and Tomcat 7.0.50 service layer application with a dependency on ActiveMQ to be started independently.
The reason Im using ActiveMQ is to have a shared datastore among 2 same load balanced instances for faster processing.
Here is what I want to do :
To be able to start ActiveMQ from pom.xml or worst case scenario from context.xml. So, lets say 2 instances are load balanced and they start their own ActiveMQ servers but they point to a single data store(directory) for queue information.
Please advise how can I have such a design to sustain optimum performance in a production environment.
I'm still on the hunt for any psuedo code that I can try , have not succeeded yet .
Code snippet from camelContext.xml
<broker id="broker" brokerName="myBroker" useShutdownHook="false" useJmx="true" persistent="true" dataDirectory="activemq-data"
xmlns="http://activemq.apache.org/schema/core">
<transportConnectors>
<transportConnector name="tcp" uri="tcp://localhost:61616"/>
</transportConnectors>
</broker>
<bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="tcp://myBroker?create=false&waitForStart=5000" />
</bean>
<bean id="pooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory"
init-method="start" destroy-method="stop">
<property name="maxConnections" value="8" />
<property name="connectionFactory" ref="jmsConnectionFactory" />
</bean>
<bean id="activeMQConfig"
class="org.apache.activemq.camel.component.ActiveMQConfiguration">
<property name="connectionFactory" ref="pooledConnectionFactory" />
<property name="concurrentConsumers" value="20" />
</bean>
<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="configuration" ref="activeMQConfig" />
<property name="transacted" value="true" />
<property name="cacheLevelName" value="CACHE_CONSUMER" />
</bean>
Please help .
I resolved the issue finally . In case somebody else is facing the same problem , I downgraded the ActivemQ version to 5.8.0 to resolve the issue.
I am using Spring and trying to setup a global transaction spanning over two MS SQL Server DBs. The app is running inside Tomcat 6.
I have these definitions:
<bean id="dataSource1" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
....
</bean>
<bean id="sessionFactory1"
class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
<property name="dataSource" ref="dataSource1"/>
....
</bean>
<bean id="hibernateTransactionManager1"
class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory">
<ref local="sessionFactory1"/>
</property>
</bean>
<bean id="dataSource2" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
....
</bean>
<bean id="sessionFactory2"
class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
<property name="dataSource" ref="dataSource2"/>
....
</bean>
<bean id="hibernateTransactionManager2"
class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory">
<ref local="sessionFactory2"/>
</property>
</bean>
Then also, each DAO is linked either to sessionFactory1 or to sessionFactory2.
<bean name="stateHibernateDao" class="com.project.dao.StateHibernateDao">
<property name="sessionFactory" ref="sessionFactory1"/>
</bean>
Also, I recently added these two.
<bean id="atomikosTransactionManager" class="com.atomikos.icatch.jta.UserTransactionManager" init-method="init" destroy-method="close">
<property name="forceShutdown" value="false" />
<property name="transactionTimeout" value="300" />
</bean>
<bean id="atomikosUserTransaction" class="com.atomikos.icatch.jta.UserTransactionImp">
<property name="transactionTimeout" value="300" />
</bean>
I am trying to programmatically manage the global transaction
(this is some old legacy code and I don't want to change it too
much so I prefer keeping this managed programmatically).
So now I have this UserTransaction ut (injected from Spring), so I call ut.begin(), do some DB/DAO operations to the two DBs through the DAOs, then I call ut.commit().
The thing is that even before the ut.commit() call, I can see the data is already committed to the DBs?!
I don't think Atomikos is aware of my two DBs, their data sources, session factories, etc. I don't think it starts any transactions on them. Looks like they are not enlisted at all in the global transaction.
To me it seems that each DB/DAO operation goes to the SQL Server on its own, so SQL Server creates an implicit transaction for just that DAO/DB operation, applies the operation and commits the implicit the transaction.
But 1) and 2) are just guesses of mine.
My questions:
Do I need to start the two DB transactions myself (but OK, this is what I am currently doing and I am trying to get rid of; that's why I am trying to use Atomikos to start with)?
How I can configure all this correctly so that when I call ut.begin() it begins a global transaction to the two DBs and when I call ut.commit() it commits it?
I haven't played with JTA recently so seems to me I am missing something quite basic here. What is it?
Edit 1
<bean id="globalTransactionManager" class="org.springframework.transaction.jta.JtaTransactionManager">
<property name="userTransaction" ref="atomikosUserTransaction"/>
<property name="transactionManager" ref="atomikosTransactionManager" />
<property name="allowCustomIsolationLevels" value="true" />
<property name="transactionSynchronization" value="2" />
</bean>
I have some legacy Spring MVC code mixed with gwt code in same artifact (built using maven) and I cannot make it run. It wants validation provider at runtime which i do not need (since I'm not using any JSR-303 validation annotations) and do not want in CP (it may conflict with some app containers this artifact will be deployed in)
How to force spring not to do any JSR-303 validations and get rid of runtime dependency on validation provider?
PS artifact has validation-api in CP since GWT is using it somehow
PPS
Seems like removing <mvc:annotation-driven/> from Spring config fixes this.
Binding and classic validations still works (I have <context:annotation-config/> enabled)
As you already discovered, <mvc:annotation-driven/> sets a lot of features including JSR-303. The equivalent is
<bean class="org.springframework.web.servlet.mvc.annotation.DefaultAnnotationHandlerMapping">
<property name="order" value="0" />
</bean>
<bean class="org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter">
<property name="webBindingInitializer">
<bean class="org.springframework.web.bind.support.ConfigurableWebBindingInitializer">
<property name="validator" ref="validator" />
</bean>
</property>
<property name="messageConverters">
<list>
<bean class="org.springframework.http.converter.ByteArrayHttpMessageConverter" />
<bean class="org.springframework.http.converter.StringHttpMessageConverter" />
<bean class="org.springframework.http.converter.FormHttpMessageConverter" />
<bean class="org.springframework.http.converter.xml.SourceHttpMessageConverter" />
</list>
</property>
</bean>
<bean id="validator"
class="org.springframework.validation.beanvalidation.LocalValidatorFactoryBean" />
<bean id="conversion-service"
class="org.springframework.format.support.FormattingConversionServiceFactoryBean" />
So you may substitute the tag onto this xml configuration and remove parts you don't need.