We Keep Coding

Implementing retry logic in Ibm Websphere MQ with spring

I am working on below configuration for messaging using Spring and Webphere MQ.
I have a requirement of implementing retrying logic for a scenario, where I receive a message from the queue and put the message data on to a Elastic search server (search server is non transactional), if the search server is down I have to rollback the message on to the queue again and process the message after some interval of time(for example: 30 seconds) . this retry has to be done for 5 times. after 5 times the message has to be put on a Dead letter queue .we are using Tomcat as the server.
we are using spring integration jms:message-driven-channel-adapter for message receiving
How can I implement this behavior with spring and Websphere MQ?
I have crawled across many sites, and I could find support for Active MQ but not for IBM MQ.
<bean id="mqConnectionFactory" class="com.ibm.mq.jms.MQQueueConnectionFactory">
<property name="hostName">
<value>${queue_hostname}</value>
</property>
<property name="port">
<value>${queue_port}</value>
</property>
<property name="queueManager">
<value>${queue_manager}</value>
</property>
<property name="transportType">
<value>1</value>
</property>
</bean>
<!-- JMS Queue Connection Factory -->
<bean id="jmsQueueConnectionFactory"
class="org.springframework.jms.connection.SingleConnectionFactory102">
<property name="targetConnectionFactory">
<ref bean="mqConnectionFactory" />
</property>
<property name="pubSubDomain">
<value>false</value>
</property>
</bean>
<!-- JMS Destination Resolver -->
<bean id="jmsDestinationResolver"
class="org.springframework.jms.support.destination.DynamicDestinationResolver">
</bean>
<!-- JMS Queue Template -->
<bean id="jmsQueueTemplate" class="org.springframework.jms.core.JmsTemplate102">
<property name="connectionFactory">
<ref bean="jmsQueueConnectionFactory" />
</property>
<property name="destinationResolver">
<ref bean="jmsDestinationResolver" />
</property>
<property name="pubSubDomain">
<value>false</value>
</property>
<property name="receiveTimeout">
<value>20000</value>
</property>
</bean>

There is nothing in the JMS spec about delayed redeliveries. Some brokers have a custom mechanism/policy to implement it; you'll have to look at the broker documentation.
As has been said before, you can use the delivery count header to give up after some number of retries.
EDIT
In response to your comment below...
Only if you are using a version of MQ that supports JMS 2.0 - it looks like you are using a very old version requiring JmsTemplate102. 1.0.2 is ancient; 1.1 has been out for years; Spring has supported JMS 2.0 for a nearly 3 years. If you have a JMS 2.0 broker (and client library), set the deliveryDelay on a JmsTemplate bean. Then, configure the outbound channel adapter to use that template via the jms-template property.
In order to get visibility to the redelivery count, either pass the whole message into your service or, with a POJO method, configure it to get that header...
public MyReply process(#Payload MyObject foo,
#Header("JMSXRedeliveryCount") int redeliveryCOunt) {
...
}
Again, this is a JMS 2.0 feature (the header) although some brokers provide it in 1.1.

IBM MQ doesn't have the delay redelivery but one option would be to consider using the JMS2.0 concept of Delayed Delivery. This doesn't set the redelivered flag so won't engage any backout logic but it would implemented the timed behaviour.
For example when the message couldn't be processed it could be requeued by the application with a delay. This won't be then seen again for say 5 minutes. The application would probably have to mark the message with a counter. Both could be done under transacted session to achieve

spring integration - read file from FTP and process it. How can we achieve metastore and process it without polling

In spring integration - Im trying to read file from FTP and process it. How can we achieve metastore and process it without polling.
In the below configuration, in order to avoid reading the same file, if server restart happen ,I have introduced message-store in the ftpChannel.
Now, the processor of the file is service activator, which needs polling. How can i avoid polling in service activator and read the file from ftpChannel queue immediately. If i use int:dispatcher then, I couldnot use message-store.
How can we resolve this?
<int:channel id="ftpChannel">
<int:queue message-store="mongoDbMessageStore" />
<!-- <int:dispatcher task-executor="taskExecutor"/> -->
</int:channel>
<bean id="mongoDbMessageStore"
class="org.springframework.integration.mongodb.store.MongoDbMessageStore">
<constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
<constructor-arg name="collectionName" value="ftpInfo" />
</bean>
<int-ftp:inbound-channel-adapter id="ftpInbound"
channel="ftpChannel" session-factory="ftpClientFactory" charset="UTF-8"
auto-create-local-directory="true" delete-remote-files="false"
filename-pattern="*.gz" remote-directory="/myfilerepo/#{istDate.getISTDate()}"
remote-file-separator="/" local-filename-generator-expression="#this.toUpperCase()"
temporary-file-suffix=".writing" preserve-timestamp="true"
local-directory="/temp/spring/#{istDate.getISTDate()}">
<int:poller cron="0-5 0/5 * * * ?" max-messages-per-poll="-1"/>
</int-ftp:inbound-channel-adapter>
<int:service-activator id="jobServiceActivator"
input-channel="ftpChannel" ref="triggerJobLauncher" method="launch">
<int:poller fixed-delay="10" />
</int:service-activator>
<!-- job context -->
<bean id="jobRepository"
class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean">
<property name="transactionManager" ref="transactionManager" />
</bean>
<bean id="transactionManager"
class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />
<bean id="jobLauncher"
class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
<!-- job context -->

You do not need a message-store on the channel; you need to use an FtpPersistentAcceptOnceFileListFilter in the filter and/or a FileSystemPersistentAcceptOnceFileListFilter in the local-filter to avoid reprocessing files after a system restart.
They need a MetadataStore; if you want to use mongo you'll need to implement one; the framework currently doesn't have a mongo implementation.
EDIT:
As of version 4.2, the framework now has a mongo MetadataStore.

According to your concern to avoid reading the same file from the QueueChannel, I'd say that you worry that Spring Application always process the same message after application startup. But it isn't true. The message are removed from the queue (and, of course, from the MessageStore) when it is polled from there. If you MessageStore is transactional resource (e.g. JDBC), the record for messages is marked to remove until TX commit or rollback.
It isn't case for MongoDB though, but any message from the queue are polled only once, even if you have a cluster of such an application.
So, I don't understand why are you sure that your <service-activator> accepts the same file (as payload) because you restart the app.
From other side, if you want to access to the MessageGroup backed by your ftpChannel, you can do this:
mongoDbMessageStore.getMessageGroup("mongoDbMessageStore:ftpChannel");
From other side you always can purge QueueChannel manually, injection ftpChannel to some service as QueueChannelOperations.

Spring Integration Concurrent JMS adapter

I am using Spring Integration (3.0.6) and having problems using a message-driven-channel-adapter concurrently. If I send 2 jobs to the JMS queue and ensure the first one never finishes (kept in a loop) the second job is never taken from the queue.
If I remove the transaction-manager from the channel adapter I get the concurrency I expect. Does anybody see anything wrong with the way I am configuring this?
<int-jms:message-driven-channel-adapter
channel="jobsChannel"
connection-factory="jmsConnectionFactory"
pub-sub-domain="false"
destination-name="JOBS_QUEUE"
transaction-manager="jmsTxManager"
concurrent-consumers="5"
max-messages-per-task="1"
error-channel="errorChannel"
id="jobJmsReceiver" />
<amq:broker useJmx="false" persistent="true">
<amq:transportConnectors>
<amq:transportConnector uri="tcp://localhost:0" />
</amq:transportConnectors>
</amq:broker>
<bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="vm://localhost" />
</bean>
<bean id="jmsTxManager" class="org.springframework.jms.connection.JmsTransactionManager">
<property name="connectionFactory" ref="jmsConnectionFactory"></property>
</bean>

I'd have to see DEBUG logs, but you really don't need a transaction manager, set acknowledge="transacted" and the container will use local transactions on the session.
You typically only need an external transaction manager when synchronizing transactions (e.g. with a JDBC tm) or when using a JTA transaction manager in an app server.

Enforce socket timeout on ActiveMQ from Camel route?

So below I have Camel (via Spring DSL) successfully integrating my beans with ActiveMQ queues:
<!-- Note: this code is just a snippet; if you need to see more, please let me know! -->
<camelContext id="my-camel-context" xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="activemq-myinstance:queue:myqueue" />
<onException>
<exception>java.lang.Exception</exception>
<redeliveryPolicy maximumRedeliveries="2" />
<to uri="activemq-myinstance:queue_failures" />
</onException>
<to uri="bean:myBean?method=doCommand" />
</route>
</camelContext>
<bean id="jmsConnectionFactory-myqueue" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="${activemq.instance.url}" />
</bean>
<bean id="pooledConnectionFactory-myqueue" class="org.apache.activemq.pool.PooledConnectionFactory">
<property name="maxConnections" value="64" />
<property name="maximumActive" value="${max.active.consumers}" />
<property name="connectionFactory" ref="jmsConnectionFactory-myqueue" />
</bean>
<bean id="jmsConfig-myqueue" class="org.apache.camel.component.jms.JmsConfiguration">
<property name="connectionFactory" ref="pooledConnectionFactory-myqueue"/>
<property name="concurrentConsumers" value="${max.active.consumers}"/>
</bean>
<bean id="activemq-myqueue" class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="configuration" ref="jmsConfig-myqueue"/>
</bean>
I'd like to explicitly enforce a socket timeout (on Socket.read()) - between Camel and ActiveMQ - of 25 seconds. Thus, when Camel attempts to route a message to/from ActiveMQ, if ActiveMQ takes more than 25 seconds to complete that response, I want the thread to exit gracefully. Obviously, if it's possible to also set up some kind of failover (so that requests that timeout can get replayed at a future time) that is greatly preferred over just losing the message!
How can I accomplish this? Thanks in advance!
Update: if Camel/JMS/ActiveMQ doesn't support this out of the box, I don't mind writing my own "ThreadManager" that interrupts/stops threads after 25-seconds, but I'm not sure what interface/classes to implement/extend, and to subsequently wire into my Spring beans.

By default Camel activemq request times out after 20 seconds.
For send timeouts there is a sendTimeout property on org.apache.activemq.ActiveMQConnectionFactory.
Also check the requesttimeout option for JMSConfiguraiton.
For failover you can set the failover transport in the broker url.

just set the timeout property on your brokerURL
failover:(tcp\://localhost\:61616)?timeout=25000
this will propagate an error back to your producer so you can handle it instead of having it just blocking the thread forever...

Replies are not consumed from reply queue

I'm sending messages with InOut pattern to a JMS queue and set the replyTo to a predefined queue.
sometimes everything works fine and sometimes camel wont read replies from that queue and fail with TimedOutException.
sometimes I can see 2 or more consumers on the reply queue (using AMQ web console) but camel wont consume it.
When I see one consumer on the reply queue (using AMQ web console), everything works fine
and when I see many consumers on the reply queue (using AMQ web console), camel wont read replies from that queue and fail with TimedOutException.
Environment : Fuse ESB
Java Code
from("activemq:spirit.adapter.producer.shippingorder.commands")
.id("router-Shipping-Order-Commands")
.log(LoggingLevel.INFO, "Transport1", "router-Shipping-Order-Commands request: ${body}")
.to("activemq:transport.consumer.shippingorder.commands?replyTo=tra‌nsport.prod‌ucer.shippingorder.event&replyToType=Exclusive&requestTimeout=120000")‌​
.log(Logg‌​ingLevel.INFO, "Transport1", "router response: ${body}");
activemq configuration
<bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="tcp://10.10.40.94:61616"/>
</bean>
<bean id="pooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory"
init-method="start" destroy-method="stop">
<property name="maxConnections" value="1" />
<property name="connectionFactory" ref="jmsConnectionFactory" />
</bean>
<bean id="jmsConfig" class="org.apache.camel.component.jms.JmsConfiguration">
<property name="connectionFactory" ref="pooledConnectionFactory" />
</bean>
<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="configuration" ref="jmsConfig" />
</bean>
This configuration is written in the camel-context.xml file and in the same file there is another activemq configuration used for another route
could you please advise?

Can you post your activemq configuration you are using? For example make sure you configure the init and destroy methods on the pool so it can properly start|stop as documented here: http://camel.apache.org/activemq

Hi All-
The problem was caused by some bugs in camel-jms 2.9.x which was shipped in Fuse ESB 7.0.1 and solved in Fuse ESB 7.1.0