We Keep Coding

Spring Integration lock busy, another session is active

I'm connecting to a POP3 inbox (it doesn't support IMAP, unfortunately) using Spring Integration. Here's my config:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/mail http://www.springframework.org/schema/integration/mail/spring-integration-mail.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-mail="http://www.springframework.org/schema/integration/mail"
xmlns:util="http://www.springframework.org/schema/util">
<int:channel id="mailInputChannel" />
<int:channel id="channel1" />
<int:channel id="channel2" />
<int:channel id="discardChannel" />
<!-- replace 'userid and 'password' with the real values -->
<int-mail:inbound-channel-adapter id="pop3ShouldDeleteTrue"
store-uri="pop3://userid:password#host.example.com/Inbox"
channel="mailInputChannel"
should-delete-messages="true"
auto-startup="true"
java-mail-properties="javaMailProperties">
<!-- Will poll every 1 seconds -->
<int:poller max-messages-per-poll="1000" fixed-delay="1000"/>
</int-mail:inbound-channel-adapter>
<int:recipient-list-router id="incomingMailRouter" input-channel="mailInputChannel">
<int:recipient channel="channel1" selector-expression="payload.subject matches '^.*channel1.*$'"/>
<int:recipient channel="channel2" selector-expression="payload.subject matches '^.*channel2.*$'"/>
</int:recipient-list-router>
<util:properties id="javaMailProperties">
<prop key="mail.pop3.socketFactory.fallback">false</prop>
<prop key="mail.debug">false</prop>
</util:properties>
</beans>
Basically I have an inbound-channel-adapter that reads in the messages and a simple recipient-list-router that directs it accordingly.
When I start up the application, it runs fine. However, after it processes the first email, it sometimes hangs. The debug output is:
19260 [task-scheduler-1] DEBUG org.springframework.integration.mail.Pop3MailReceiver - connecting to store [pop3://username:password#mailhost.example.com/Inbox]
DEBUG POP3: mail.pop3.apop.enable: false
DEBUG POP3: mail.pop3.disablecapa: false
DEBUG POP3: connecting to host "mailhost.example.com", port 110, isSSL false
+OK Qpopper (version 4.0.5) at cobweb starting.
CAPA
+OK Capability list follows
TOP
USER
LOGIN-DELAY 0
EXPIRE 0
UIDL
RESP-CODES
AUTH-RESP-CODE
X-MANGLE
X-MACRO
X-LOCALTIME Wed, 10 Aug 2016 11:50:57 -0700
IMPLEMENTATION Qpopper-version-4.0.5
.
DEBUG POP3: authentication command trace suppressed
DEBUG POP3: authentication command failed
QUIT
+OK Pop server at cobweb signing off.
30529 [task-scheduler-1] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory - Returning cached instance of singleton bean 'errorChannel'
30531 [task-scheduler-1] DEBUG org.springframework.integration.channel.PublishSubscribeChannel - preSend on channel 'errorChannel', message: ErrorMessage [payload=org.springframework.messaging.MessagingException: failure occurred while polling for mail; nested exception is javax.mail.AuthenticationFailedException: [IN-USE] /var/spool/maildrop/.program_name.pop lock busy! Is another session active? (11), headers={id=43817279-7717-f4d7-971a-3c283e2dbec3, timestamp=1470849586452}]
30531 [task-scheduler-1] DEBUG org.springframework.integration.handler.LoggingHandler - _org.springframework.integration.errorLogger.handler received message: ErrorMessage [payload=org.springframework.messaging.MessagingException: failure occurred while polling for mail; nested exception is javax.mail.AuthenticationFailedException: [IN-USE] /var/spool/maildrop/.program_name.pop lock busy! Is another session active? (11), headers={id=43817279-7717-f4d7-971a-3c283e2dbec3, timestamp=1470849586452}]
30534 [task-scheduler-1] ERROR org.springframework.integration.handler.LoggingHandler - org.springframework.messaging.MessagingException: failure occurred while polling for mail; nested exception is javax.mail.AuthenticationFailedException: [IN-USE] /var/spool/maildrop/.program_name.pop lock busy! Is another session active? (11)
at org.springframework.integration.mail.MailReceivingMessageSource.receive(MailReceivingMessageSource.java:131)
at org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:209)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:245)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:58)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:190)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:186)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:353)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:55)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:51)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:344)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: javax.mail.AuthenticationFailedException: [IN-USE] /var/spool/maildrop/.program_name.pop lock busy! Is another session active? (11)
at com.sun.mail.pop3.POP3Store.protocolConnect(POP3Store.java:207)
at javax.mail.Service.connect(Service.java:295)
at javax.mail.Service.connect(Service.java:176)
at javax.mail.Service.connect(Service.java:125)
at org.springframework.integration.mail.AbstractMailReceiver.connectStoreIfNecessary(AbstractMailReceiver.java:286)
at org.springframework.integration.mail.AbstractMailReceiver.openFolder(AbstractMailReceiver.java:297)
at org.springframework.integration.mail.AbstractMailReceiver.receive(AbstractMailReceiver.java:319)
at org.springframework.integration.mail.MailReceivingMessageSource.receive(MailReceivingMessageSource.java:112)
... 19 more
It stays in this loop for a (seemingly) random amount of time, sometimes up to 2 minutes, spitting out these exceptions every time it tries to connect. However, sometimes it connects just file right after processing a message. Processing a message is the only thing that causes it to fail.
Is Spring trying to hold on to that connection to that connection too long? Can I tell it to drop the connection right after it fetches the email?

Turns out it was a server-side issue. After switching to a production-grade mail server that supported IMAP, the issue was resolved.

Configuring Circuit breaker with outbound adapter for handling connection timed out issue

<int:service-activator input-channel="toKafka" ref="conditionalProducerService" method="producerCircuitBreaker">
<int:request-handler-advice-chain>
<ref bean="circuitBreakerAdvice1" />
</int:request-handler-advice-chain>
</int:service-activator>
<int:channel id="failedChannel2" />
<int-kafka:outbound-channel-adapter
id="kafkaOutboundChannelAdapter" kafka-producer-context-ref="producerContext" auto-startup="false" channel="toKafka" message-key="kafka_messageKey">
<int:poller fixed-delay="1000" error-channel="failedChannel2" />
</int-kafka:outbound-channel-adapter>
<int:chain input-channel="failedChannel2">
<int:transformer expression="'failed:' + payload.failedMessage.payload + ' with ' + payload.cause.message" />
<int-stream:stderr-channel-adapter append-newline="true"/>
</int:chain>
<bean id="circuitBreakerAdvice1" class="org.springframework.integration.handler.advice.RequestHandlerCircuitBreakerAdvice">
<property name="threshold" value="2" />
<property name="halfOpenAfter" value="12000" />
</bean>
public Message<?> producerCircuitBreaker(Message<?> payload) {
throw new RuntimeException("foo Pro");}
With the above configuration ,we are trying:
1.Expecting to get the failed message to propagate to the error-channel="failedChannel2" which is not happening.as I couldn't see the transformed output in the console.
2.CircuitBreaker is working for the ServiceActivator(for application related exception here as above) but how can we configure the CB for the failed case for outbound adapter. example: when connection timed out or the server is down suddenly /network connection problem/some environemnt issue before sending the message from SI channel to external(kafka ) server.Can we configure CB with outbound adapter for such situation.
As per the SI doc regarding Circuit Breaker Advice,found below.
"Typically, this Advice might be used for external services, where it might take some time to fail (such
as a timeout attempting to make a network connection)".
Please suggest on how to achieve this.Many thanks.
updated config:
<int:gateway default-request-channel="toKafka" error-channel="errorChannel"
default-reply-timeout="0" />
<int:service-activator input-channel="toKafka">
<bean class="com.XXX.ProducerMessageHandler" >
<constructor-arg ref="producerContext"/>
</bean>
<int:request-handler-advice-chain>
<ref bean="circuitBreakerAdvice" />
</int:request-handler-advice-chain>
<bean id="transformerService1" class="com.XXX.KafkaTransformerTest" />
<int:transformer input-channel="errorChannel"
order="1" ref="transformerService1" method="transformFailed">
</int:transformer>
public void transformFailed(Message<?> message) {
APPLOGGER.log("transformer message test" + message);
public class ProducerMessageHandler extends KafkaProducerMessageHandler{
public ProducerMessageHandler(KafkaProducerContext kafkaProducerContext) {
super(kafkaProducerContext);
// TODO Auto-generated constructor stub
}
#Override
public void handleMessageInternal(final Message<?> message) throws Exception {
//super.handleMessageInternal(message);
throw new RuntimeException("test foo");
}
log :
01-05#23:44:18,598 DEBUG org.springframework.integration.config.ServiceActivatorFactoryBean$1 - org.springframework.integration.config.ServiceActivatorFactoryBean$1#6a0ef4b6 received message: GenericMessage [payload=hello, headers={timestamp=1452017658598, id=e0591162-3b93-9bb6-0699-89b15b20e904}]
DEBUG: - com.XXX.ProducerMessageHandler#0 received message: GenericMessage [payload=hello, headers={timestamp=1452017658598, id=e0591162-3b93-9bb6-0699-89b15b20e904}]
got exception : org.springframework.messaging.MessageHandlingException: error occurred in message handler [com.XXX.ProducerMessageHandler#0]; nested exception is java.lang.RuntimeException: test foo
01-05#23:44:18,606 DEBUG org.springframework.integration.channel.PublishSubscribeChannel - preSend on channel 'toKafka', message: GenericMessage [payload=hello, headers={timestamp=1452017658605, id=61597941-b2f8-314d-141d-8f2c058dda4d}]
01-05#23:44:18,606 DEBUG org.springframework.integration.config.ServiceActivatorFactoryBean$1 - org.springframework.integration.config.ServiceActivatorFactoryBean$1#6a0ef4b6 received message: GenericMessage [payload=hello, headers={timestamp=1452017658605, id=61597941-b2f8-314d-141d-8f2c058dda4d}]
DEBUG: - com.XXX.ProducerMessageHandler#0 received message: GenericMessage [payload=hello, headers={timestamp=1452017658605, id=61597941-b2f8-314d-141d-8f2c058dda4d}]
got exception : org.springframework.messaging.MessageHandlingException: error occurred in message handler [com.XXX.ProducerMessageHandler#0]; nested exception is java.lang.RuntimeException: test foo
01-05#23:44:18,606 DEBUG org.springframework.integration.channel.PublishSubscribeChannel - preSend on channel 'toKafka', message: GenericMessage [payload=hello, headers={timestamp=1452017658606, id=119afbf1-6104-feb1-eb44-f646aa932277}]
01-05#23:44:18,606 DEBUG org.springframework.integration.config.ServiceActivatorFactoryBean$1 - org.springframework.integration.config.ServiceActivatorFactoryBean$1#6a0ef4b6 received message: GenericMessage [payload=hello, headers={timestamp=1452017658606, id=119afbf1-6104-feb1-eb44-f646aa932277}]
got exception : org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.config.ServiceActivatorFactoryBean$1#6a0ef4b6]; nested exception is org.springframework.integration.handler.advice.RequestHandlerCircuitBreakerAdvice$CircuitBreakerOpenException: Circuit Breaker is Open for org.springframework.integration.config.ServiceActivatorFactoryBean$1#6a0ef4b6
01-05#23:44:18,606 DEBUG org.springframework.integration.channel.PublishSubscribeChannel - preSend on channel 'toKafka', message: GenericMessage [payload=hello, headers={timestamp=1452017658606, id=8dafe2e0-8efe-c827-e745-1387e6045e7d}]
01-05#23:44:18,606 DEBUG org.springframework.integration.config.ServiceActivatorFactoryBean$1 - org.springframework.integration.config.ServiceActivatorFactoryBean$1#6a0ef4b6 received message: GenericMessage [payload=hello, headers={timestamp=1452017658606, id=8dafe2e0-8efe-c827-e745-1387e6045e7d}]
got exception : org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.config.ServiceActivatorFactoryBean$1#6a0ef4b6]; nested exception is org.springframework.integration.handler.advice.RequestHandlerCircuitBreakerAdvice$CircuitBreakerOpenException: Circuit Breaker is Open for org.springframework.integration.config.ServiceActivatorFactoryBean$1#6a0ef4b6

The advice only applies to the endpoint it is assigned to, not the downstream flow; unfortunately, the kafka schema doesn't permit it to be applied to the outbound channel adapter. I have created a JIRA issue for that.
A work-around would be to configure the KafkaProducerMessageHandler as a <bean/> and ref it from a <service-activator/>. Then you can apply your circuit breaker.
Another work-around would be to use an in-flow gateway...
<int:service-activator ... ref="gw">
<int:request-handler-advice-chain ...
</int:service-activator>
<int:gateway id="gw" default-request-channel="toKafka"
default-reply-timeout="0"
error-channel="..." ... />
I am not sure why you're not seeing a message in the error channel; usually, turning on DEBUG logging will help debug this kind of thing.
EDIT
I just tested with this and it works just fine...
<int:gateway default-request-channel="toKafka" error-channel="errorChannel"
default-reply-timeout="0" />
<int:service-activator input-channel="toKafka">
<bean class="com.example.Foo" />
<int:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.RequestHandlerCircuitBreakerAdvice">
<property name="threshold" value="2"/>
</bean>
</int:request-handler-advice-chain>
</int:service-activator>
EDIT2
If you are not using a gateway, you can handle it with a queue channel and a poller. This works fine for me too...
<int:channel id="toKafka">
<int:queue />
</int:channel>
<int:service-activator input-channel="toKafka">
<bean class="com.example.Foo" />
<int:poller error-channel="errorChannel" fixed-delay="1000" />
<int:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.RequestHandlerCircuitBreakerAdvice">
<property name="threshold" value="2"/>
<property name="halfOpenAfter" value="12000"/>
</bean>
</int:request-handler-advice-chain>
</int:service-activator>
Or, you can add a mid-flow gateway.

copied from chat for future reference :
Sam : Hi Gary
for(int i=0;i<4;i++){
try{
toKafka.send(MessageBuilder
.withPayload("hello").
build());
}catch(Exception e){
System.out.println("got exception : " + e); } }
this is how I am sending message
Gary : So you are sending directly to the channel - you should use a MessagingGateway instead.
Sam:Hi Gary.
thanks.it is working with Gateway.
configuring CB with KafkaProducerMessageHandler is fine but it covers for any failure comes under below method
public void handleMessageInternal(final Message message) throws Exception
but I want to cover the issue with network errors as well like invalid broker list/server down which it is not covering and i am getting the exception in the console like this:
log
12-24#16:46:46,250 DEBUGspringframework.integration.kafka.outbound.KafkaProducerMessageHandler - org.springframework.integration.kafka.outbound.KafkaProducerMessageHandler#0 received message: GenericMessage [payload=TestVo[data=sample message]], headers={replyChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#44286963, errorChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#44286963, kafka_topic=tried_in, kafka_partitionId=2, id=7b596368-0aee-ddaa-2168-dc403e22c38f, timestamp=1450955805294}]
12-24#16:55:12,630 WARN apache.kafka.common.network.Selector - Error in I/O with /1.2.0.3
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at org.apache.kafka.common.network.Selector.poll(Selector.java:238)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122)
at java.lang.Thread.run(Unknown Source)
want CB to get invoked in this case as well.
Gary : The connection exception should occur within the handleMessageInternal()
If the exception is not thrown, it's a bug. I'll take a look.
The Future is discarded in handleMessageInternal - I'll open a JIRA issue.
https://jira.spring.io/browse/INTEXT-218
Sam : ok.is it going to cover the case when kafka server is down for some reason?
Gary : yes; but you might want to reduce the timeout from the default (60s)

Transaction boundaries on Spring Integration with ActiveMQ and an a channel backed with an Message Store

I'am working on an application using spring-integration and ActiveMQ to integrate a main server and several "client" applications.
The main goal is to develop and small client with an embedded ActiveMQ broker, the embedded broker is started by this client application
in order to receive JMS messages from several third parties applications on the same site. The client application is connected to the server through an JMS connection with a
main installation of an ActiveMQ Server(this already exists).
I need to provide the client with an off-line capability, it means that if the client is disconnected it
should store the messages received from the third clients on the same site and send them when the server is online again.
All seems to work right when there is no transactionality involved, but I'am having problems about the transaction boundaries of the channels configured in spring-integration.
The main flow is :
Third Clients on same site -> Our Client on Site Application -> Reformat Message and add information on the messageTranslator -> Store on the remoteChannel -> Send to the Main Server
This is my on Site Client configuration:
The embedded ActiveMQ broker in the client app:
<bean id="broker1" class="org.apache.activemq.xbean.BrokerFactoryBean">
<property name="config" value="classpath:activemq/activemq.xml" />
<property name="start" value="true" />
</bean>
The connection Factories; remoteConnectionFactory is the connection to the main server and Queue definitions :
<!-- Local queue definitions -->
<bean id="localConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory">
<bean class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="vm://localhost"/>
</bean>
</property>
<property name="sessionCacheSize" value="10"/>
</bean>
<bean id="localEventQueue" class="org.apache.activemq.command.ActiveMQQueue" >
<constructor-arg value="event.client"/>
</bean>
<!-- Remote(Server) queue definitions -->
<bean id="remoteConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory">
<bean class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="tcp://192.168.0.125:62626"/>
</bean>
</property>
<property name="sessionCacheSize" value="10"/>
</bean>
<bean id="remoteQueue" class="org.apache.activemq.command.ActiveMQQueue">
<constructor-arg value="remote.event"/>
</bean>
Local Process, on same remote site:
<!-- local process -->
<int:channel id="transformChannel" >
<int:queue />
</int:channel>
<int-jms:message-driven-channel-adapter
id="jmsEventsFromClient"
connection-factory="localConnectionFactory"
destination="localEventQueue"
channel="transformChannel"
concurrent-consumers="4"
auto-startup="true"/>
<!-- Transforms the message from clients to a standard messages that server knows-->
<int:service-activator input-channel="transformChannel"
ref="messageTranslator"
output-channel="remoteChannel" >
<int:poller fixed-delay="500">
</int:poller>
</int:service-activator>
<int:transaction-synchronization-factory id="syncFactory">
<int:after-commit expression="#store.removeFromIdCache(headers.id.toString())" />
<int:after-rollback expression="#store.removeFromIdCache(headers.id.toString())"/>
</int:transaction-synchronization-factory>
<int:channel id="remoteChannel" >
<int:queue message-store="store" />
</int:channel>
<!-- local file database with HSQL store of the remotechannel -->
<bean id="store" class="org.springframework.integration.jdbc.store.JdbcChannelMessageStore">
<property name="dataSource" ref="dataSource"/>
<property name="channelMessageStoreQueryProvider">
<bean id="queryProvider" class="org.springframework.integration.jdbc.store.channel.HsqlChannelMessageStoreQueryProvider"/>
</property>
<property name="usingIdCache" value="true"/>
</bean>
Send to the remote Queue:
<!-- the message is send to the remote server -->
<int-jms:outbound-channel-adapter id="remoteJms"
connection-factory="remoteConnectionFactory"
destination="remoteQueue"
channel="remoteChannel"
session-transacted="true"
auto-startup="true">
<int:poller fixed-delay="500">
<int:transactional propagation="REQUIRED"
isolation="DEFAULT"
synchronization-factory="syncFactory"
transaction-manager="transactionManager"/>
</int:poller>
</int-jms:outbound-channel-adapter>
When there is not a JTA transaction manager configured and the remoteJms is not session-transacted all works just fine, even when I simulate an offline situation the messages are correctly stored on the store, and when the connection with the server comes alive again (remoteJms) the messages are send them correctly.
But when a JTA transaction manager is configured (atomikos) and the ActiveMQ main server goes down the messages are not been stored on the store. I thought as soon as the pooler in remoteJms receive the message it starts a new transaction, but this is not happening, it seams like the whole way from jmsEventsFromClient to the jmsOut looks like a single transaction and is failed because the remoteJms is down.
What I want is to have 2 transactions, the first should started when a message is receive on the eventQueue and finished when the message is leave it in the remoteChannel and the second should start when the message is take it from the remoteChannel and leave it on the remoteJms.
Thanks for your comments and help.
EDIT
This is happening only when the remote server is down, and the message is removed from the store, but that is what I don't want.
Add the log error.
2015-06-29 16:39:16.859 INFO 4982 --- [ask-scheduler-4] c.atomikos.jdbc.AbstractDataSourceBean : AtomikosDataSoureBean 'dataSource': getConnection ( null )...
2015-06-29 16:39:16.859 INFO 4982 --- [ask-scheduler-4] c.atomikos.jdbc.AbstractDataSourceBean : AtomikosDataSoureBean 'dataSource': init...
2015-06-29 16:39:16.859 INFO 4982 --- [ask-scheduler-4] c.a.icatch.imp.CompositeTransactionImp : addParticipant ( XAResourceTransaction: 7472616E73616374696F6E4D616E6167657230303038303030303430:7472616E73616374696F6E4D616E616765723830 ) for transaction transactionManager0008000040
2015-06-29 16:39:16.859 INFO 4982 --- [ask-scheduler-4] c.a.datasource.xa.XAResourceTransaction : XAResource.start ( 7472616E73616374696F6E4D616E6167657230303038303030303430:7472616E73616374696F6E4D616E616765723830 , XAResource.TMNOFLAGS ) on resource dataSource represented by XAResource instance org.hsqldb.jdbc.pool.JDBCXAResource#55f823a3
2015-06-29 16:39:16.859 INFO 4982 --- [ask-scheduler-4] c.a.icatch.imp.CompositeTransactionImp : registerSynchronization ( com.atomikos.jdbc.AtomikosConnectionProxy$JdbcRequeueSynchronization#c85d8b3 ) for transaction transactionManager0008000040
2015-06-29 16:39:16.859 INFO 4982 --- [ask-scheduler-4] c.atomikos.jdbc.AtomikosConnectionProxy : atomikos connection proxy for org.hsqldb.jdbc.pool.JDBCXAConnectionWrapper#731d1112: calling prepareStatement(SELECT COUNT(MESSAGE_ID) from INT_CHANNEL_MESSAGE where GROUP_KEY=? and REGION=?)...
hi message: una cadena
2015-06-29 16:39:17.820 INFO 4982 --- [ask-scheduler-9] c.atomikos.jdbc.AbstractDataSourceBean : AtomikosDataSoureBean 'dataSource': getConnection ( null )...
2015-06-29 16:39:17.820 INFO 4982 --- [ask-scheduler-9] c.atomikos.jdbc.AbstractDataSourceBean : AtomikosDataSoureBean 'dataSource': init...
2015-06-29 16:39:25.093 WARN 4982 --- [0.1:57663#61616] o.a.a.b.TransportConnection.Transport : Transport Connection to: tcp://192.168.0.125:57663 failed: java.io.EOFException
2015-06-29 16:39:27.145 WARN 4982 --- [ Atomikos:1] c.a.icatch.imp.ActiveStateHandler : Timeout/setRollbackOnly of ACTIVE coordinator !
2015-06-29 16:39:47.821 WARN 4982 --- [ask-scheduler-9] com.atomikos.jdbc.AtomikosSQLException : Connection pool exhausted - try increasing 'maxPoolSize' and/or 'borrowConnectionTimeout' on the DataSourceBean.
2015-06-29 16:39:47.822 INFO 4982 --- [ask-scheduler-4] c.a.icatch.imp.CompositeTransactionImp : registerSynchronization ( com.atomikos.jdbc.AtomikosConnectionProxy$JdbcRequeueSynchronization#c85d8b3 ) for transaction transactionManager0008000040
2015-06-29 16:39:47.822 INFO 4982 --- [ask-scheduler-4] c.atomikos.jdbc.AtomikosConnectionProxy : atomikos connection proxy for org.hsqldb.jdbc.pool.JDBCXAConnectionWrapper#731d1112: calling prepareStatement(SELECT COUNT(MESSAGE_ID) from INT_CHANNEL_MESSAGE where GROUP_KEY=? and REGION=?)...
2015-06-29 16:39:47.823 INFO 4982 --- [ask-scheduler-4] c.a.icatch.imp.CompositeTransactionImp : registerSynchronization ( com.atomikos.jdbc.AtomikosConnectionProxy$JdbcRequeueSynchronization#c85d8b3 ) for transaction transactionManager0008000040
2015-06-29 16:39:47.823 INFO 4982 --- [ask-scheduler-4] c.atomikos.jdbc.AtomikosConnectionProxy : atomikos connection proxy for org.hsqldb.jdbc.pool.JDBCXAConnectionWrapper#731d1112: calling prepareStatement(SELECT INT_CHANNEL_MESSAGE.MESSAGE_ID, INT_CHANNEL_MESSAGE.MESSAGE_BYTES from INT_CHANNEL_MESSAGE where INT_CHANNEL_MESSAGE.GROUP_KEY = ? and INT_CHANNEL_MESSAGE.REGION = ? order by CREATED_DATE, MESSAGE_SEQUENCE LIMIT 1)...
2015-06-29 16:39:47.823 INFO 4982 --- [ask-scheduler-4] c.atomikos.jdbc.AtomikosConnectionProxy : atomikos connection proxy for org.hsqldb.jdbc.pool.JDBCXAConnectionWrapper#731d1112: close()...
2015-06-29 16:39:47.823 INFO 4982 --- [ask-scheduler-4] c.a.datasource.xa.XAResourceTransaction : XAResource.end ( 7472616E73616374696F6E4D616E6167657230303038303030303430:7472616E73616374696F6E4D616E616765723830 , XAResource.TMSUCCESS ) on resource dataSource represented by XAResource instance org.hsqldb.jdbc.pool.JDBCXAResource#55f823a3
2015-06-29 16:39:47.823 INFO 4982 --- [ask-scheduler-4] c.a.icatch.imp.CompositeTransactionImp : commit() done (by application) of transaction transactionManager0008000040
2015-06-29 16:39:47.824 INFO 4982 --- [ask-scheduler-4] c.a.datasource.xa.XAResourceTransaction : XAResource.prepare ( 7472616E73616374696F6E4D616E6167657230303038303030303430:7472616E73616374696F6E4D616E616765723830 ) returning OK on resource dataSource represented by XAResource instance org.hsqldb.jdbc.pool.JDBCXAResource#55f823a3
2015-06-29 16:39:47.824 INFO 4982 --- [ask-scheduler-4] c.a.datasource.xa.XAResourceTransaction : XAResource.rollback ( 7472616E73616374696F6E4D616E6167657230303038303030303430:7472616E73616374696F6E4D616E616765723830 ) on resource dataSource represented by XAResource instance org.hsqldb.jdbc.pool.JDBCXAResource#55f823a3
2015-06-29 16:39:47.825 ERROR 4982 --- [ask-scheduler-4] o.s.integration.handler.LoggingHandler : org.springframework.transaction.UnexpectedRollbackException: JTA transaction unexpectedly rolled back (maybe due to a timeout); nested exception is javax.transaction.RollbackException: Prepare: NO vote
at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1024)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:757)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:726)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:521)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:291)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy85.call(Unknown Source)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:298)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:292)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.transaction.RollbackException: Prepare: NO vote
at com.atomikos.icatch.jta.TransactionImp.rethrowAsJtaRollbackException(TransactionImp.java:66)
at com.atomikos.icatch.jta.TransactionImp.commit(TransactionImp.java:206)
at com.atomikos.icatch.jta.TransactionManagerImp.commit(TransactionManagerImp.java:436)
at com.atomikos.icatch.jta.UserTransactionManager.commit(UserTransactionManager.java:177)
at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1021)
... 22 more
Caused by: com.atomikos.icatch.RollbackException: Prepare: NO vote
at com.atomikos.icatch.imp.ActiveStateHandler.prepare(ActiveStateHandler.java:231)
at com.atomikos.icatch.imp.CoordinatorImp.prepare(CoordinatorImp.java:681)
at com.atomikos.icatch.imp.CoordinatorImp.terminate(CoordinatorImp.java:970)
at com.atomikos.icatch.imp.CompositeTerminatorImp.commit(CompositeTerminatorImp.java:82)
at com.atomikos.icatch.imp.CompositeTransactionImp.commit(CompositeTransactionImp.java:336)
at com.atomikos.icatch.jta.TransactionImp.commit(TransactionImp.java:190)
... 25 more
2015-06-29 16:39:47.825 ERROR 4982 --- [ask-scheduler-9] o.s.integration.handler.LoggingHandler : org.springframework.messaging.MessageDeliveryException: failed to send Message to channel 'remoteChannel'; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.atomikos.jdbc.AtomikosSQLException: Connection pool exhausted - try increasing 'maxPoolSize' and/or 'borrowConnectionTimeout' on the DataSourceBean.
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:292)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:239)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:95)
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:248)
at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:171)
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:119)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:105)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
at org.springframework.integration.endpoint.PollingConsumer.handleMessage(PollingConsumer.java:74)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:219)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:55)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:149)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:146)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:298)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:292)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.atomikos.jdbc.AtomikosSQLException: Connection pool exhausted - try increasing 'maxPoolSize' and/or 'borrowConnectionTimeout' on the DataSourceBean.
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:630)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:909)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:970)
at org.springframework.integration.jdbc.store.JdbcChannelMessageStore.addMessageToGroup(JdbcChannelMessageStore.java:422)
at org.springframework.integration.store.MessageGroupQueue.doOffer(MessageGroupQueue.java:329)
at org.springframework.integration.store.MessageGroupQueue.put(MessageGroupQueue.java:274)
at org.springframework.integration.store.MessageGroupQueue.put(MessageGroupQueue.java:48)
at org.springframework.integration.channel.QueueChannel.doSend(QueueChannel.java:92)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:277)
... 28 more
Caused by: com.atomikos.jdbc.AtomikosSQLException: Connection pool exhausted - try increasing 'maxPoolSize' and/or 'borrowConnectionTimeout' on the DataSourceBean.
at com.atomikos.jdbc.AtomikosSQLException.throwAtomikosSQLException(AtomikosSQLException.java:46)
at com.atomikos.jdbc.AbstractDataSourceBean.throwAtomikosSQLException(AbstractDataSourceBean.java:90)
at com.atomikos.jdbc.AbstractDataSourceBean.throwAtomikosSQLException(AbstractDataSourceBean.java:85)
at com.atomikos.jdbc.AbstractDataSourceBean.getConnection(AbstractDataSourceBean.java:347)
at com.atomikos.jdbc.AbstractDataSourceBean.getConnection(AbstractDataSourceBean.java:394)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
... 37 more

I think I figured it out - try setting receive-timeout="0" on the poller that's pulling from the store-backed QueueChannel.

File Polling and Filtering Issues with SFTP Spring Integration

I am trying to pull remote files through SFTP Spring Integration and dump into my local drive, and kick-off a Spring Batch Job (Step-1 to read, process & write), followed by Step-2 where a Tasklet archives the local file downloaded to another location locally and deletes the downloaded file.
Assuming the files being downloaded is "data.service", "data.maintenance" and "data.transaction". I am only processing "data.service" but all 3 are archived and deleted.
I want the above steps to happen at a specific interval assuming that a new set of files will be overwritten at the remote location.
Bean def
<bean id="acceptAllFileListFilter" class="org.springframework.integration.file.filters.AcceptAllFileListFilter"/>
<bean id="acceptOnceFileListFilter" class="org.springframework.integration.file.filters.AcceptOnceFileListFilter"/>
conf
<int-sftp:inbound-channel-adapter id="sftpInbondAdapter"
channel="inboundFileChannel"
session-factory="sftpSessionFactory"
local-directory="file:${inbound.local.directory}"
remote-directory="${inbound.remote.directory}"
auto-create-local-directory="true"
delete-remote-files="false"
preserve-timestamp="true"
filename-pattern="*.*"
local-filter="acceptAllFileListFilter" >
<int:poller max-messages-per-poll="-1" fixed-rate="60000" />
</int-sftp:inbound-channel-adapter>
What is happening now is...
The above triggers this sequence and runs in an unending loop, without fetching the new files from remote. Details as below:
I start with an empty local folder, and remote server has all 3 files
[Start the program]
Spring Integration program downloads all 3 files to local location
Spring Batch Job Step-1 gets triggered Tasklet (Step-2) archives the
local files Tasklet (Step-2) deletes the local files Spring
Integration program downloads all 3 files to local location (why???)
program keeps running Step-1 to Step-5 in an unending loop
New file dropped on the remote server, same (1) to (5) no job triggered
Tried replacing as below, but the program just runs once, i.e., from (1) to (5) and nothing happens
local-filter="acceptOnceFileListFilter" >
Expected
Spring Integration program downloads all 3 files to local location
Spring Batch Job Step-1 gets triggered
Tasklet (Step-2) archives the local files
Tasklet (Step-2) deletes the local files
Spring Integration program waits for new files to be dropped in the remote location (polls every 60000 ms)
New file dropped on the remote server
Start with (1) through (6)
Per suggestion on a different post I have a unique job parameter as below:
#Transformer
public JobLaunchRequest toRequest(Message<File> message) {
JobParametersBuilder jobParametersBuilder = new JobParametersBuilder();
jobParametersBuilder.addString(fileParameterName, message.getPayload().getAbsolutePath());
jobParametersBuilder.addLong("currentTime", new Long(System.currentTimeMillis()));
return new JobLaunchRequest(job, jobParametersBuilder.toJobParameters());
}
Tried with different combinations of filter, none served my purpose. Any sample would be of great help, struggling with this since last week.
Here's my change per the suggestions:
Adding a composite filter as below, but what would be the argument that will be passed to SftpPersistentAcceptOnceFileListFilter? I could not see any samples in the internet.
<bean id="compositeFilter" class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
<list>
<bean class="org.springframework.integration.file.filters.FileSystemPersistentAcceptOnceFileListFilter"/>
<bean class="org.springframework.integration.sftp.filters.SftpPersistentAcceptOnceFileListFilter">
</bean>
</list>
</constructor-arg>
</bean>
Including in the adapter as below:
<int-sftp:inbound-channel-adapter id="sftpInbondAdapter"
channel="inboundFileChannel"
session-factory="sftpSessionFactory"
local-directory="file:${inbound.local.directory}"
remote-directory="${inbound.remote.directory}"
auto-create-local-directory="true"
delete-remote-files="false"
filter="compositeFilter">
<int:poller max-messages-per-poll="-1" fixed-rate="600000" />
</int-sftp:inbound-channel-adapter>
Update - Changes after adding persistent filter:
<int-sftp:inbound-channel-adapter id="sftpInbondAdapter"
channel="inboundFileChannel"
session-factory="sftpSessionFactory"
local-directory="file:${inbound.local.directory}"
remote-directory="file:${inbound.remote.directory}"
auto-create-local-directory="false"
delete-remote-files="false"
filter="compositeFilter"
>
<int:poller max-messages-per-poll="-1" fixed-rate="60000" />
</int-sftp:inbound-channel-adapter>
<bean id="compositeFilter" class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
<list>
<bean class="org.springframework.integration.file.filters.FileSystemPersistentAcceptOnceFileListFilter">
<constructor-arg name="store" ref="metadataStore"/>
<constructor-arg value="*.*"/>
</bean>
<bean class="org.springframework.integration.sftp.filters.SftpPersistentAcceptOnceFileListFilter">
<constructor-arg name="store" ref="metadataStore"/>
<constructor-arg value="*.*"/>
</bean>
</list>
</constructor-arg>
</bean>
<bean id="redisConnectionFactory" class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory">
<property name="port" value="7379"/>
</bean>
<bean name="metadataStore" class="org.springframework.integration.redis.metadata.RedisMetadataStore">
<constructor-arg name="connectionFactory" ref="redisConnectionFactory"/>
</bean>
This gives me the exception as below. I tried the suggestions in the Spring forum, but I don't have a connection issues.
2014-07-17 22:27:14,139 [task-scheduler-1] INFO com.jcraft.jsch -
Authentication succeeded (keyboard-interactive). 2014-07-17
22:27:14,192 [task-scheduler-1] INFO com.jcraft.jsch - Disconnecting
from localhost port 22 2014-07-17 22:27:14,195 [task-scheduler-1]
DEBUG
org.springframework.beans.factory.support.DefaultListableBeanFactory -
Returning cached instance of singleton bean 'errorChannel' 2014-07-17
22:27:14,197 [task-scheduler-1] DEBUG
org.springframework.integration.channel.PublishSubscribeChannel -
preSend on channel 'errorChannel', message: [Payload
MessagingException
content=org.springframework.messaging.MessagingException: Problem
occurred while synchronizing remote to local
directory][Headers={id=84cccecf-20ec-f67b-3f74-a938bf9abf3d,
timestamp=1405661234197}] 2014-07-17 22:27:14,197 [task-scheduler-1]
DEBUG org.springframework.integration.handler.LoggingHandler - (inner
bean)#22 received message: [Payload MessagingException
content=org.springframework.messaging.MessagingException: Problem
occurred while synchronizing remote to local
directory][Headers={id=84cccecf-20ec-f67b-3f74-a938bf9abf3d,
timestamp=1405661234197}] 2014-07-17 22:27:14,199 [task-scheduler-1]
ERROR org.springframework.integration.handler.LoggingHandler -
org.springframework.messaging.MessagingException: Problem occurred
while synchronizing remote to local directory at
org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:193)
at
org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.receive(AbstractInboundFileSynchronizingMessageSource.java:167)
at
org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:124)
at
org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:187)
at
org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:52)
at
org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:146)
at
org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:143)
at
org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:278)
at
org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
at
org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at
org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49)
at
org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:272)
at
org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at
org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695) Caused by:
org.springframework.messaging.MessagingException: Failed to execute on
session at
org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:311)
at
org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:167)
... 21 more Caused by: java.lang.ClassCastException:
com.jcraft.jsch.ChannelSftp$LsEntry cannot be cast to java.io.File at
org.springframework.integration.file.filters.FileSystemPersistentAcceptOnceFileListFilter.fileName(FileSystemPersistentAcceptOnceFileListFilter.java:28)
at
org.springframework.integration.file.filters.AbstractPersistentAcceptOnceFileListFilter.buildKey(AbstractPersistentAcceptOnceFileListFilter.java:88)
at
org.springframework.integration.file.filters.AbstractPersistentAcceptOnceFileListFilter.accept(AbstractPersistentAcceptOnceFileListFilter.java:48)
at
org.springframework.integration.file.filters.AbstractFileListFilter.filterFiles(AbstractFileListFilter.java:40)
at
org.springframework.integration.file.filters.CompositeFileListFilter.filterFiles(CompositeFileListFilter.java:97)
at
org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.filterFiles(AbstractInboundFileSynchronizer.java:157)
at
org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer$1.doInSession(AbstractInboundFileSynchronizer.java:173)
at
org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer$1.doInSession(AbstractInboundFileSynchronizer.java:167)
at
org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:302)
... 22 more
2014-07-17 22:27:14,199 [task-scheduler-1] DEBUG
org.springframework.integration.channel.PublishSubscribeChannel -
postSend (sent=true) on channel 'errorChannel', message: [Payload
MessagingException
content=org.springframework.messaging.MessagingException: Problem
occurred while synchronizing remote to local
directory][Headers={id=84cccecf-20ec-f67b-3f74-a938bf9abf3d,
timestamp=1405661234197}]
Update post removing FileSystemPersistentAcceptOnceFileListFilter filter leaving the 'SftpPersistentAcceptOnceFileListFilter'
2014-07-18 09:29:21,942 [task-scheduler-3] INFO com.jcraft.jsch -
Authentications that can continue:
publickey,keyboard-interactive,password 2014-07-18 09:29:21,942
[task-scheduler-3] INFO com.jcraft.jsch - Next authentication method:
publickey 2014-07-18 09:29:21,942 [task-scheduler-3] INFO
com.jcraft.jsch - Authentications that can continue:
keyboard-interactive,password 2014-07-18 09:29:21,942
[task-scheduler-3] INFO com.jcraft.jsch - Next authentication method:
keyboard-interactive 2014-07-18 09:29:22,022 [task-scheduler-3] INFO
com.jcraft.jsch - Authentication succeeded (keyboard-interactive).
2014-07-18 09:29:22,067 [task-scheduler-3] DEBUG
org.springframework.data.redis.core.RedisConnectionUtils - Opening
RedisConnection 2014-07-18 09:29:22,068 [task-scheduler-3] INFO
com.jcraft.jsch - Disconnecting from localhost port 22 2014-07-18
09:29:22,068 [task-scheduler-3] DEBUG
org.springframework.integration.channel.PublishSubscribeChannel -
preSend on channel 'errorChannel', message: [Payload
MessagingException
content=org.springframework.messaging.MessagingException: Problem
occurred while synchronizing remote to local
directory][Headers={id=55304e38-6f82-8350-e763-0f5bcb507788,
timestamp=1405700962068}] 2014-07-18 09:29:22,068 [Connect thread
localhost session] INFO com.jcraft.jsch - Caught an exception,
leaving main loop due to Socket closed 2014-07-18 09:29:22,068
[task-scheduler-3] DEBUG
org.springframework.integration.handler.LoggingHandler - (inner
bean)#22 received message: [Payload MessagingException
content=org.springframework.messaging.MessagingException: Problem
occurred while synchronizing remote to local
directory][Headers={id=55304e38-6f82-8350-e763-0f5bcb507788,
timestamp=1405700962068}] 2014-07-18 09:29:22,069 [task-scheduler-3]
ERROR org.springframework.integration.handler.LoggingHandler -
org.springframework.messaging.MessagingException: Problem occurred
while synchronizing remote to local directory at
org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:193)
at
org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.receive(AbstractInboundFileSynchronizingMessageSource.java:167)
at
org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:124)
at
org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:187)
at
org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:52)
at
org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:146)
at
org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:143)
at
org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:278)
at
org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
at
org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at
org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49)
at
org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:272)
at
org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at
org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695) Caused by:
org.springframework.messaging.MessagingException: Failed to execute on
session at
org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:311)
at
org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:167)
... 21 more Caused by:
org.springframework.data.redis.RedisConnectionFailureException: Cannot
get Jedis connection; nested exception is
redis.clients.jedis.exceptions.JedisConnectionException: Could not get
a resource from the pool at
org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:97)
at
org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:143)
at
org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:41)
at
org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:128)
at
org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:91)
at
org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:78)
at
org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:177)
at
org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:152)
at
org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:84)
at
org.springframework.data.redis.core.DefaultHashOperations.putIfAbsent(DefaultHashOperations.java:179)
at
org.springframework.data.redis.core.DefaultBoundHashOperations.putIfAbsent(DefaultBoundHashOperations.java:91)
at
org.springframework.data.redis.support.collections.RedisProperties.putIfAbsent(RedisProperties.java:228)
at
org.springframework.integration.redis.metadata.RedisMetadataStore.putIfAbsent(RedisMetadataStore.java:139)
at
org.springframework.integration.file.filters.AbstractPersistentAcceptOnceFileListFilter.accept(AbstractPersistentAcceptOnceFileListFilter.java:51)
at
org.springframework.integration.file.filters.AbstractFileListFilter.filterFiles(AbstractFileListFilter.java:40)
at
org.springframework.integration.file.filters.CompositeFileListFilter.filterFiles(CompositeFileListFilter.java:97)
at
org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.filterFiles(AbstractInboundFileSynchronizer.java:157)
at
org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer$1.doInSession(AbstractInboundFileSynchronizer.java:173)
at
org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer$1.doInSession(AbstractInboundFileSynchronizer.java:167)
at
org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:302)
... 22 more Caused by:
redis.clients.jedis.exceptions.JedisConnectionException: Could not get
a resource from the pool at
redis.clients.util.Pool.getResource(Pool.java:42) at
redis.clients.jedis.JedisPool.getResource(JedisPool.java:84) at
redis.clients.jedis.JedisPool.getResource(JedisPool.java:10) at
org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:90)
... 41 more Caused by:
redis.clients.jedis.exceptions.JedisConnectionException:
java.net.ConnectException: Connection refused at
redis.clients.jedis.Connection.connect(Connection.java:150) at
redis.clients.jedis.BinaryClient.connect(BinaryClient.java:71) at
redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1783) at
redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:65) at
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:819)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:429)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360)
at redis.clients.util.Pool.getResource(Pool.java:40) ... 44 more
Caused by: java.net.ConnectException: Connection refused at
java.net.PlainSocketImpl.socketConnect(Native Method) at
java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:382) at
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:241)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:228) at
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:431) at
java.net.Socket.connect(Socket.java:527) at
redis.clients.jedis.Connection.connect(Connection.java:144) ... 51
more
2014-07-18 09:29:22,069 [task-scheduler-3] DEBUG
org.springframework.integration.channel.PublishSubscribeChannel -
postSend (sent=true) on channel 'errorChannel', message: [Payload
MessagingException
content=org.springframework.messaging.MessagingException: Problem
occurred while synchronizing remote to local
directory][Headers={id=55304e38-6f82-8350-e763-0f5bcb507788,
timestamp=1405700962068}]

Gist Here
Hopefully self-explanatory; it just prints the file name on the console, then deletes it.

It could be the case that if the new files dropped on the server has same name(which are downloaded in the first iteration) then spring integration will not attempt to download the files again. "acceptOnceFileListFilter" seems like something that would restrict sftp integration component to download the same file again.
Update: Here is some sample code which polls a remote directory every 10 secs without introducing any filters(I am not sure why you need them, as you want to download the files regardless). Given that you dont want to change the file names, the only thing that you will have to make sure is that your downloaded local needs to deleted when the next file is dropped on the remote server. I verified it and worked fine for me.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-ftp="http://www.springframework.org/schema/integration/ftp"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/ftp http://www.springframework.org/schema/integration/ftp/spring-integration-ftp.xsd">
<int:channel id="incomingChannel"/>
<bean id="ftpClientFactory" class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="host"/>
<property name="username" value="username"/>
<property name="password" value="password"/>
</bean>
<!-- poll every 10 secs without worrying about file names-->
<int-ftp:inbound-channel-adapter id="triggerFtpInBound"
channel="incomingChannel"
auto-create-local-directory="true"
local-directory="C:\home\temp\ftp"
remote-directory="/export/home/rbaljinder/ftp-test"
filename-pattern="*.*"
session-factory="ftpClientFactory">
<int:poller cron="1/10 * * * * *" max-messages-per-poll="1"/>
</int-ftp:inbound-channel-adapter>
UPDATE: seems working now.Added local-filter="acceptAllFileListFilter".
<int-ftp:inbound-channel-adapter id="triggerFtpInBound"
channel="incomingChannel"
auto-create-local-directory="true"
local-directory="C:\home\temp\ftp"
remote-directory="/export/home/cwk2/ftp-test"
filename-pattern="*.*"
session-factory="ftpClientFactory"
local-filter="acceptAllFileListFilter">
<int:poller cron="1/10 * * * * *" max-messages-per-poll="1"/>
</int-ftp:inbound-channel-adapter>
<bean id="acceptAllFileListFilter"
class="org.springframework.integration.file.filters.AcceptAllFileListFilter"/>
<int:service-activator id="jobServiceActivator"
input-channel="incomingChannel"
ref="triggerJobLauncher"
method="launch"/>
#Component("triggerJobLauncher")
public static class TriggerJobLauncher {
#Autowired
JobLauncher jobLauncher;
public void launch(File file) throws Exception {
System.out.println("test:" + file);
}
}

How to stop tomcat from making failed attempts to connect to local memcached servers?

I have deployed my web app inside a tomcat container but due to a possible connection leak , the web app is constantly making failed attempts to connect to the local memcached server listening at port 11211 and 11212. I am using the spy-memcached client.
I have a ContextListener defined which basically shuts down all active memcached client connections.
However when I un-deploy my web app , it appears to me that tomcat is still trying to continue with failed attempts to connect to the memcached server, which it should not.I have checked the active tcp connections on memcached server using netstat but I could not find any entry.
I have also restarted the tomcat server but to no avail as such.
How should I restrict tomcat from making these connections?
2011-11-13 21:21:34.575 INFO net.spy.memcached.MemcachedConnection: Reconnecting due to failure to connect to {QA sa=localhost/127.0.0.1:11212, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:407)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:275)
at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:2030)
2011-11-13 21:21:34.576 WARN net.spy.memcached.MemcachedConnection: Closing, and reopening {QA sa=localhost/127.0.0.1:11212, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}, attempt 32.

I was facing the same problem.Setting daemon true works for me. I am using spymecached-2.8.4 I get the Memcached Client through net.spy.memcached.spring.MemcachedClientFactoryBean though Spring (spring - 3.1.1), here is my spring configuration that I use in my web applicaton :
<bean id="memcachedClient" class="net.spy.memcached.spring.MemcachedClientFactoryBean">
<property name="servers" value="localhost:11211"/>
<property name="protocol" value="BINARY"/>
<property name="transcoder">
<bean class="net.spy.memcached.transcoders.SerializingTranscoder">
<property name="compressionThreshold" value="1024"/>
</bean>
</property>
<property name="opTimeout" value="1000"/>
<property name="timeoutExceptionThreshold" value="1998"/>
<property name="hashAlg">
<value type="net.spy.memcached.DefaultHashAlgorithm">KETAMA_HASH</value>
</property>
<property name="locatorType" value="CONSISTENT"/>
<property name="failureMode" value="Redistribute"/>
<property name="useNagleAlgorithm" value="false"/>
<property name="daemon" value="true"/>
</bean>