We Keep Coding

Spring integration; Copying files picked up by service before copy is completed

Application background
There are 2 services in application. Lets call them
1) service A
2) service B
Service A basically copies files that match a particular criteria from the source directory to the destination directory. Using spring integration here.
The destination directory for service A is the source directory of service B.
Service B is constantly polling the directory for files and processes them and then moves them to another subdirectory called "processed".
Problem:
The original problem was that while service A is copying files to the destination directory, service B picks up the half copied files and processes them.
Tried solution
Refer to the service B integration-context.xml below. I attached a composite filter to the inbound channel. I added a custom filter called LastCreatedFileListFilter to this composite filter. This filter is based on lines of LastModifiedFileListFilter provided by spring-integration-file and this basically discards any file whose age (by created time) is less than 30 seconds.
This filter works perfectly and does not pick a file until its 30 seconds old. But the problem now is that I am using prevent-duplicates="true". So what happens is the first time service B polls the folder and the age of the file is less than 30 seconds it filters out the file but after 30 seconds, the filter does not filter out the file which is correct behavior but by now the service has marked this as a duplicate and rejects it.
So, my problem here is that I do want to keep a prevent duplicate check and also not process a file until its fully copied.
I am not very familiar with spring-integration and the two approaches I am looking to realize is:
1) not mark files as duplicate in the above scenario until it is processed by the application? is this possible? is so, is this advised?
2) In service A if I could first create files with a temporary name and then rename them after the copy is complete? the thing here is that service B will not pick up the files until they start with a configured name. This way, I do not have to trust the age property which could not be 100% decisive as the source and destination are on a different network and copy times could be large (#MurphysLaw).
But the problem here is that, I am having trouble conceptualizing how to best achieve this solution with spring integration. Any guidance or suggestions?
Please refer to integration context for service A for current implementation. Let me know if any thing needs clarification.
Service A
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-file="http://www.springframework.org/schema/integration/file"
xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration
http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/file
http://www.springframework.org/schema/integration/file/spring-integration-file.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">
<context:property-placeholder location="classpath:application.properties"/>
<bean name="redisMetaDataStore" class="org.springframework.integration.redis.metadata.RedisMetadataStore">
<constructor-arg ref="redisConnectionFactory" />
</bean>
<bean id="redisConnectionFactory" class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory">
<property name="port" value="6379" />
</bean>
<int-file:inbound-channel-adapter id = "filesIn"
channel="fileChannel"
directory="file:${input.directory}"
filter="incomingCompositeFilter">
<int:poller id="fileInboudPoller" fixed-rate="${in.interval}" time-unit="SECONDS" />
</int-file:inbound-channel-adapter>
<bean id="incomingCompositeFilter" class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
<list>
<bean id="acceptOnceFilter" class="org.springframework.integration.file.filters.FileSystemPersistentAcceptOnceFileListFilter">
<constructor-arg ref="redisMetaDataStore"/>
<constructor-arg value="*"/>
</bean>
<bean id="notOlderThanDateFilter" class="com.fexco.bgeadmin.file.filter.NotOlderThanDateFilter">
<constructor-arg value="${file.lastModified.ignoreBeforeDate}"/>
</bean>
<bean id="documentConfigFilter" class="com.fexco.bgeadmin.file.filter.DocumentConfigFilter">
</bean>
</list>
</constructor-arg></bean>
<int:channel id="fileChannel"/>
<int-file:outbound-channel-adapter id="save-as-file"
auto-create-directory="true"
channel="fileChannel"
directory="file:${output.directory}"/>
</beans>
Service B
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-file="http://www.springframework.org/schema/integration/file"
xmlns:batch-int="http://www.springframework.org/schema/batch-integration"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration
http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/file
http://www.springframework.org/schema/integration/file/spring-integration-file.xsd
http://www.springframework.org/schema/batch-integration
http://www.springframework.org/schema/batch-integration/spring-batch-integration.xsd">
<int:channel id="inboundFileChannel"/>
<int:channel id="outboundJobRequestChannel"/>
<int:channel id="jobLaunchReplyChannel"/>
<int-file:inbound-channel-adapter id="filePoller"
channel="inboundFileChannel"
directory="${app.file.source}"
auto-create-directory="true"
prevent-duplicates="true"
filter="incomingCompositeFilter">
<int:poller fixed-rate="5000"/>
</int-file:inbound-channel-adapter>
<bean id="incomingCompositeFilter" class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
<list>
<bean id="fileNameFilter" class="org.springframework.integration.file.filters.RegexPatternFileListFilter">
<constructor-arg value=".*\.(xls|xlsx|csv)$" />
</bean>
<bean id="ageFilter" class="com.fexco.bgeadmin.integration.filter.LastCreatedFileListFilter">
<property name="age" value="30"/>
</bean>
</list>
</constructor-arg></bean>
<int:transformer input-channel="inboundFileChannel"
output-channel="outboundJobRequestChannel" method="toRequest">
<bean class="com.fexco.bgeadmin.integration.FileMessageToJobRequest"/>
</int:transformer>
<batch-int:job-launching-gateway request-channel="outboundJobRequestChannel"
reply-channel="jobLaunchReplyChannel"/>
<int:logging-channel-adapter channel="jobLaunchReplyChannel"/>
</beans>

The best solution for such problems is #2 - for A not to write the file to B "in-place". Using the last modified time is unreliable.
This is the standard procedure for the <int:file-outbound-channel-adapter/> - which uses a FileWritingMessageHandler underneath. It has a property temporaryFileSuffix, which is .writing by default. The file is renamed after being copied.

Spring integration - less boilerplate?

I have started using Spring-integration for listening on JMS-queues, and it's working satisfactory, but I have somewhat of an issue with the amount of xml-configuration required in order to set up a listener. Most of this is boilerplate, what changes is really just the jms-instance and the name of the queue to listen to, and the class and method to invoke upon receiving a message. Here is an example:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:jms="http://www.springframework.org/schema/integration/jms"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/jms http://www.springframework.org/schema/integration/jms/spring-integration-jms.xsd">
<bean id="jms.queue.myQueue" class="org.apache.activemq.command.ActiveMQQueue">
<constructor-arg value="${myApplication.jms.queue.myQueue}" />
</bean>
<bean id="jms.container.myJMSListender"
class="org.springframework.jms.listener.DefaultMessageListenerContainer"
destroy-method="destroy">
<property name="connectionFactory"
ref="jmsConnectionFactory" /> <!-- Defined elsewhere -->
<property name="destination"
ref="jms.queue.myQueue" />
<property name="sessionTransacted"
value="true" />
</bean>
<int:publish-subscribe-channel id="channel.myQueue" />
<jms:message-driven-channel-adapter id="channelAdapter.myQueue"
container="jms.container.myQueue"
channel="channel.myQueue"
acknowledge="transacted"/>
<int:service-activator id="serviceActivator.myQueue"
input-channel="channgel.myQueue"
ref="myQueueJMSListener"
method="handleMessage" />
</beans>
As you can see, a lot of config for something fairly simple, and with quite a lot of JMS-listeners this becomes tedious, both to read and write.
Is there a way to configure listening to a queue with Spring-integration that requires less boilerplate? I've looked into creating my own XML-tag, but I'm kinda hoping that there is a simpler solution.

You only need an external container if you need to configure properties that are not exposed on the namespace.
<jms:message-driven-channel-adapter id="channelAdapter.myQueue"
destination="jms.queue.myQueue"
connection-factory="jmsConnectionFactory"
channel="channel.myQueue"
acknowledge="transacted"/>

Found that Spring integration has annotation support.
Requires some setup, but this can be used for multiple listeners:
<amq:connectionFactory id="jmsConnectionFactory" brokerURL="${jms.host}">
<amq:redeliveryPolicy>
<amq:redeliveryPolicy maximumRedeliveries="0" />
</amq:redeliveryPolicy>
</amq:connectionFactory>
<bean id="cachingJmsConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="jmsConnectionFactory" />
</bean>
<jms:annotation-driven />
<bean id="jmsListenerContainerFactory" class="org.springframework.jms.config.DefaultJmsListenerContainerFactory">
<property name="connectionFactory" ref="cachingJmsConnectionFactory" />
</bean>
Then, listening to a certain queue is as simple as:
#JmsListener(destination = "${myQueueName}")
public void listen(#Payload final String payload) {

Error validating the file spring-ldap.xsd

i have this error in my bean configuration, but for another project works.. the xml is:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:ldap="http://www.springframework.org/schema/ldap"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/ldap http://www.springframework.org/schema/ldap/spring-ldap.xsd">
but in one project i have this error:
referenced file contains error http://www.springframework.org/schema/ldap/spring-ldap.xsd

UPDATE:
i modify my beam like this and work (i added conf. for pooling but is not important):
<!-- LDAP config -->
<bean id="contextSourceTarget" class="org.springframework.ldap.core.support.LdapContextSource">
<property name="url" value="${ldap.url}" />
<property name="base" value="${ldap.base}" />
<property name="userDn" value="${ldap.userDn}" />
<property name="password" value="${ldap.password}" />
<property name="pooled" value="false" />
</bean>
<bean id="contextSource"
class="org.springframework.ldap.pool.factory.PoolingContextSource">
<property name="contextSource" ref="contextSourceTarget" />
</bean>
<bean id="ldapTemplate" class="org.springframework.ldap.core.LdapTemplate">
<constructor-arg ref="contextSource" />
</bean>

I guess it's because Pivotal's site is displaying captcha before showing the actual XSD,
see Spring schemaLocation fails when there is no internet connection
It depends on whether you get the error in the IDE or in the runtime.
If it's a specialized XML editor you should refer to the documentation of your editor on how to override/specify schema location, if it's in the runtime likely you have to check your project dependencies(pom.xml if it's a maven project).

Spring uses its own mechanism to get the XSD in classpath library (runtime).
Uses following files
META-INF/spring.handlers (to parse tag xml to object by an Handler)
example of spring-core-3.2.2.jar
http://www.springframework.org/schema/c=org.springframework.beans.factory.xml.SimpleConstructorNamespaceHandler
http://www.springframework.org/schema/p=org.springframework.beans.factory.xml.SimplePropertyNamespaceHandler
http://www.springframework.org/schema/util=org.springframework.beans.factory.xml.UtilNamespaceHandler
META-INF/spring.schemas (to identify the correct xsd used)
example of spring-core-3.2.2.jar
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd=org/springframework/beans/factory/xml/spring-beans-2.0.xsd
http://www.springframework.org/schema/beans/spring-beans-2.5.xsd=org/springframework/beans/factory/xml/spring-beans-2.5.xsd
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd=org/springframework/beans/factory/xml/spring-beans-3.0.xsd
http://www.springframework.org/schema/beans/spring-beans-3.1.xsd=org/springframework/beans/factory/xml/spring-beans-3.1.xsd
http://www.springframework.org/schema/beans/spring-beans-3.2.xsd=org/springframework/beans/factory/xml/spring-beans-3.2.xsd
http://www.springframework.org/schema/beans/spring-beans.xsd=org/springframework/beans/factory/xml/spring-beans-3.2.xsd
http://www.springframework.org/schema/tool/spring-tool-2.0.xsd=org/springframework/beans/factory/xml/spring-tool-2.0.xsd
http://www.springframework.org/schema/tool/spring-tool-2.5.xsd=org/springframework/beans/factory/xml/spring-tool-2.5.xsd
http://www.springframework.org/schema/tool/spring-tool-3.0.xsd=org/springframework/beans/factory/xml/spring-tool-3.0.xsd
http://www.springframework.org/schema/tool/spring-tool-3.1.xsd=org/springframework/beans/factory/xml/spring-tool-3.1.xsd
http://www.springframework.org/schema/tool/spring-tool-3.2.xsd=org/springframework/beans/factory/xml/spring-tool-3.2.xsd
http://www.springframework.org/schema/tool/spring-tool.xsd=org/springframework/beans/factory/xml/spring-tool-3.2.xsd
http://www.springframework.org/schema/util/spring-util-2.0.xsd=org/springframework/beans/factory/xml/spring-util-2.0.xsd
http://www.springframework.org/schema/util/spring-util-2.5.xsd=org/springframework/beans/factory/xml/spring-util-2.5.xsd
http://www.springframework.org/schema/util/spring-util-3.0.xsd=org/springframework/beans/factory/xml/spring-util-3.0.xsd
http://www.springframework.org/schema/util/spring-util-3.1.xsd=org/springframework/beans/factory/xml/spring-util-3.1.xsd
http://www.springframework.org/schema/util/spring-util-3.2.xsd=org/springframework/beans/factory/xml/spring-util-3.2.xsd
http://www.springframework.org/schema/util/spring-util.xsd=org/springframework/beans/factory/xml/spring-util-3.2.xsd
The IDE want resolve the location of the XSD by schemaLocation, for this reason I suggest to remove XSD validation on your Eclipse.
Windows->Preferences->Validation-> "Suspend all validators"

Error occur when I start project that using DWR3.0 RC2 with Spring

I introduce DWR into my project these days. My project is using Spring 4, I was trying to integrate DWR with Spring by using annotation approch. Below are my code for integration work.
An error shown as below occured when I stratup my project on the Tomcat server.
the error:
java.lang.NoSuchMethodError: org.springframework.util.ClassUtils.forName(Ljava/lang/String;)Ljava/lang/Class;
at org.directwebremoting.spring.DwrAnnotationPostProcessor.getBeanDefinitionClass(DwrAnnotationPostProcessor.java:96)
at org.directwebremoting.spring.DwrAnnotationPostProcessor.postProcessBeanFactory(DwrAnnotationPostProcessor.java:52)
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPos
Processors(PostProcessorRegistrationDelegate.java:265)
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:177)
at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:609)
web.xml
<servlet>
<servlet-name>dwr-dispatcher</servlet-name>
<servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
<load-on-startup>2</load-on-startup>
</servlet>
dwr-dispatcher-servlet.xml
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:dwr="http://www.directwebremoting.org/schema/spring-dwr"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-4.0.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context-4.0.xsd
http://www.directwebremoting.org/schema/spring-dwr
http://www.directwebremoting.org/schema/spring-dwr-3.0.xsd">
<context:annotation-config />
<context:component-scan base-package="com.xdfaint.webapp.apps.*.action" />
<dwr:configuration />
<dwr:annotation-config id="dwrAnnotationConfig" />
<dwr:url-mapping />
<dwr:controller id="dwrController" debug="true" />
<bean class="org.springframework.web.servlet.mvc.annotation.DefaultAnnotationHandlerMapping">
<property name="order" value="1" />
</bean>
<bean class="org.springframework.web.servlet.handler.BeanNameUrlHandlerMapping">
<property name="order" value="2" />
</bean>
<bean class="org.springframework.web.servlet.view.InternalResourceViewResolver">
<property name="prefix">
<value>/core/jsp/</value>
</property>
<property name="suffix">
<value>.jsp</value>
</property>
</bean>
</beans>
I haven't found any solution yet. Then I try to use Spring 3.2 instead, the startup successed with no error any more. I am not sure if there are problems when Spring 4 meets DWR3. Can any body help? Thx~
DWR haven't updated for a long time, and I am not using it anymore. But it is still great helpful that worth to read source codes for studying, like concept of how rest api binding, object operating. About object operating, I found some useful functions like setObject(), was using massively when I did a frontend components encapsulation work. Both setObject function codes and using demo list below, with it I can init a filed in a any nested level of the object, even if there are number of parents object of that new field hasn't been declared yet.

In Spring 4 there is no method forName in ClassUtils that only accepts String as an argument. As you can see from the JavaDoc here that method was already deprecated in Spring 3
It seems that DWR has not yet upgraded their code base to support Spring 4. Seems like you'll have to wait for that to happen (although the project seems to be non-active so I wouldn't count on that happening any time soon).

MongoDB replication with Spring

I configured Spring with MongoDB on my local machine, without replication, and everything works fine.
I also have a replica set that works fine.
Now I tried to add the replica set, but the reads/writes still go to my local machine!
This is my configuration, host1-3 are virtual machines:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:aop="http://www.springframework.org/schema/aop"
xmlns:tx="http://www.springframework.org/schema/tx"
xmlns:mongo="http://www.springframework.org/schema/data/mongo"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/tx
http://www.springframework.org/schema/tx/spring-tx-2.0.xsd
http://www.springframework.org/schema/aop
http://www.springframework.org/schema/aop/spring-aop-2.0.xsd
http://www.springframework.org/schema/data/mongo
http://www.springframework.org/schema/data/mongo/spring-mongo-1.0.xsd">
<mongo:repositories base-package="my.repositories" />
<mongo:mongo replica-set="host1:27017,host2:27017,host3:27017" />
<mongo:db-factory dbname="my_db" />
<bean id="mongoTemplate" class="org.springframework.data.mongodb.core.MongoTemplate">
<constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
</bean>
Anyone has ideas/suggestions? Thanks!
Edit:
Ok so I found the problem, just needed to add mongo-ref to the factory. So the correct configuration is:
<mongo:mongo id="mongo" replica-set="host1:27017,host2:27017,host3:27017" />
<mongo:db-factory dbname="my_db" mongo-ref="mongo" />

Ok so I found the problem, just needed to add mongo-ref to the factory. So the correct configuration is:
<mongo:mongo id="mongo" replica-set="host1:27017,host2:27017,host3:27017">
<mongo:db-factory dbname="my_db" mongo-ref="mongo" />

Writes will always go to the primary. Reads go to the primary by default unless you change the Read Preference in your use of the Java driver.