We Keep Coding

org.springframework.jms.IllegalStateException: The Session is closed

I am using Apache Camel and ActiveMQ in my spring boot application. Sometimes it starts throwing an error that session is closed. I have to restart the application to get rid of this error.
I have gone through some articles which suggest that application is exhausting the max session limit(default 500). Next time when this error will occur I will look into the activemq panel to see active sessions corresponding to a connection.
Here is my configuration file:
<camelContext id="camelContext" xmlns="http://camel.apache.org/schema/spring">
<contextScan/>
</camelContext>
<bean class="xy.acb.task.TaskServiceBean">
<property name="camelContext" ref="camelContext"/>
</bean>
<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent" destroy-method="shutdown">
<property name="autoStartup" value="true"/>
<property name="connectionFactory">
<bean class="org.apache.activemq.pool.PooledConnectionFactoryBean">
<property name="connectionFactory">
<bean class="org.apache.activemq.spring.ActiveMQConnectionFactory">
<property name="brokerURL"
value="${activemq.brokerURL:tcp://activemq:61616?daemon=true}"/>
</bean>
</property>
</bean>
</property>
<property name="acknowledgementModeName" value="CLIENT_ACKNOWLEDGE"/>
<property name="deliveryMode" value="1"/>
<property name="timeToLive" value="1260000"/>
</bean>
Below is the error trace:
The Session is closed; nested exception is javax.jms.IllegalStateException: The Session is closed
; nested exception is javax.jms.IllegalStateException: The Session is closed
at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:279)
at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:496)
at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.send(JmsConfiguration.java:228)
at org.apache.camel.component.jms.JmsProducer.doSend(JmsProducer.java:431)
at org.apache.camel.component.jms.JmsProducer.processInOnly(JmsProducer.java:385)
at org.apache.camel.component.jms.JmsProducer.process(JmsProducer.java:153)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:120)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:416)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:51)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:74)
at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:375)
at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:343)
at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:233)
at org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:343)
at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:168)
at org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:119)
at org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:105)

From Activemq 5.7, maximumActive=500(default), is replaced with maximumActiveSessionPerConnection property. Default connections is 8. so totally 500 * 8 = 4000 sessions are allowed. Check which Activemq version you are using and then set the right properties you need to make it work.

Rabbitmq : Failed message being reprocessed in an infinite loop

This is the rabbitmq configuration that I have :
<rabbit:admin connection-factory="rmqConnectionFactory"/>
<bean id="**retryAdvice**" class="org.springframework.amqp.rabbit.config.StatefulRetryOperationsInterceptorFactoryBean">
<property name="retryOperations" ref="retryTemplate"/>
</bean>
<bean id="**retryTemplate**" class="org.springframework.retry.support.RetryTemplate">
<property name="retryPolicy" ref="simpleRetryPolicy"/>
<property name="backOffPolicy">
<bean class="org.springframework.retry.backoff.FixedBackOffPolicy">
<property name="backOffPeriod" value="5000"/>
</bean>
</property>
<property name="retryContextCache" ref="retryContext"/>
</bean>
<bean id="**retryContext**" class="org.springframework.retry.policy.MapRetryContextCache"/>
<bean id="**simpleRetryPolicy**" class="org.springframework.retry.policy.SimpleRetryPolicy">
<property name="maxAttempts" value="3"/>
</bean>
<!-- Spring AMQP Template -->
<bean id="**rabbitTemplate**" class="org.springframework.amqp.rabbit.core.RabbitTemplate">
<property name="connectionFactory" ref="rmqConnectionFactory"/>
<property name="messageConverter" ref="stdJsonMessageConverter"/>
</bean>
<bean id="**stdJsonMessageConverter**" class="org.springframework.amqp.support.converter.JsonMessageConverter">
<property name="createMessageIds" value="true"/>
</bean>
And my queue is configured as follows :
<rabbit:queue name="${queue}" durable="true">
<rabbit:queue-arguments>
<entry key="x-ha-policy" value="all"/>
</rabbit:queue-arguments>
</rabbit:queue>
<rabbit:direct-exchange name="${exchange}">
<rabbit:bindings>
<rabbit:binding queue="${queue}" key="${routingKey}"/>
</rabbit:bindings>
</rabbit:direct-exchange>
When I published a message on the exchange for the first time, the listener failed with null ID exception. I purged the queue which had the bad message as well. Inspite of that, everytime I start my service, the failed message processing is retried and it fails continuously until there is a RetryCacheCapacityExceeded exception.
Has my failed message been cached somewhere? Is there a way to clear that?
Also, why do retries continue even though my retrytemplate suggest 3 reattempts at that interval of 5 secs?

When you use stateful retry, the retry state for each message id is kept in a cached (so we know when to stop).
If there's no ID, the message will fail (and keep being delivered) unless you add a MissingMessageIdAdvice to the advice chain (before the retry interceptor), which will allow 1 retry for messages with no id.

Why messages are stuck on solace server?

I'm using spring DMLC for my application with below settings, i'm facing strange behavior with DMLC that if I send 1000 messages on listener queue only ~1990 reaches to dmlc very quickly and ~10 get stuck on server, on further analysis i found that acknowledgements are not sent back for those 10 that's why i can see them on server, after few minutes acks is sent back but very slowly.
further on this i tried cacheConsumers=false in CachingConnectionFactory and everything becomes fine, however this makes frequent bind/unbind to mq server and creates huge consumer objects in jmv, does anyone have any solution how to solve this issue keeping cacheConsumers=true ?
<bean id="listenerContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="cachingjmsQueueConnectionFactory" />
<property name="destination" ref="queueDestination" />
<property name="messageListener" ref="queueDestination" />
<property name="concurrency" value="10-10" />
<property name="cacheLevel" value="1" />
<property name="transactionManager" ref="dbTransactionManager" />
<property name="sessionTransacted" value="true" />
</bean>
<bean id="cachingjmsQueueConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="jmsQueueConnectionFactory" />
<property name="reconnectOnException" value="true" />
<property name="cacheConsumers" value="true" />
<property name="cacheProducers" value="true" />
<property name="sessionCacheSize" value="1" />
</bean>

You can set cacheConsumer to false on the cachingConnectionFactory and also change the cacheLevel to level 3 (CACHE_CONSUMER) on the DefaultMessageListenerClass. This way, the consumer will be cached at the DMLC level and the issue with stuck messages should be resolved without seeing frequent binds/unbinds.
The cacheConsumer should be set to false and you should have the DefaultMessageListenerClasse control the caching because is preferable to have the listener container handle appropriate caching within it's lifecycle. The following note in the Spring documentation (http://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/jms/listener/DefaultMessageListenerContainer.html) discusses this:
Note: Don't use Spring's CachingConnectionFactory in combination with
dynamic scaling. Ideally, don't use it with a message listener
container at all, since it is generally preferable to let the listener
container itself handle appropriate caching within its lifecycle.
Also, stopping and restarting a listener container will only work with
an independent, locally cached Connection - not with an externally
cached one.

Enforce socket timeout on ActiveMQ from Camel route?

So below I have Camel (via Spring DSL) successfully integrating my beans with ActiveMQ queues:
<!-- Note: this code is just a snippet; if you need to see more, please let me know! -->
<camelContext id="my-camel-context" xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="activemq-myinstance:queue:myqueue" />
<onException>
<exception>java.lang.Exception</exception>
<redeliveryPolicy maximumRedeliveries="2" />
<to uri="activemq-myinstance:queue_failures" />
</onException>
<to uri="bean:myBean?method=doCommand" />
</route>
</camelContext>
<bean id="jmsConnectionFactory-myqueue" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="${activemq.instance.url}" />
</bean>
<bean id="pooledConnectionFactory-myqueue" class="org.apache.activemq.pool.PooledConnectionFactory">
<property name="maxConnections" value="64" />
<property name="maximumActive" value="${max.active.consumers}" />
<property name="connectionFactory" ref="jmsConnectionFactory-myqueue" />
</bean>
<bean id="jmsConfig-myqueue" class="org.apache.camel.component.jms.JmsConfiguration">
<property name="connectionFactory" ref="pooledConnectionFactory-myqueue"/>
<property name="concurrentConsumers" value="${max.active.consumers}"/>
</bean>
<bean id="activemq-myqueue" class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="configuration" ref="jmsConfig-myqueue"/>
</bean>
I'd like to explicitly enforce a socket timeout (on Socket.read()) - between Camel and ActiveMQ - of 25 seconds. Thus, when Camel attempts to route a message to/from ActiveMQ, if ActiveMQ takes more than 25 seconds to complete that response, I want the thread to exit gracefully. Obviously, if it's possible to also set up some kind of failover (so that requests that timeout can get replayed at a future time) that is greatly preferred over just losing the message!
How can I accomplish this? Thanks in advance!
Update: if Camel/JMS/ActiveMQ doesn't support this out of the box, I don't mind writing my own "ThreadManager" that interrupts/stops threads after 25-seconds, but I'm not sure what interface/classes to implement/extend, and to subsequently wire into my Spring beans.

By default Camel activemq request times out after 20 seconds.
For send timeouts there is a sendTimeout property on org.apache.activemq.ActiveMQConnectionFactory.
Also check the requesttimeout option for JMSConfiguraiton.
For failover you can set the failover transport in the broker url.

just set the timeout property on your brokerURL
failover:(tcp\://localhost\:61616)?timeout=25000
this will propagate an error back to your producer so you can handle it instead of having it just blocking the thread forever...

MDP JMS Transaction rolls back then reprocesses message in an endless loop

If I enable transaction management on my DefaultMessageListenerContainer by specifying sessionTransacted=true or transactionManager=jmsTransactionManager, whenever an exception occurs in the MDP, the transaction is rolled back and the message is placed back on the queue. That then causes the message to be processed again, the transaction to roll back again over and over again so that it creates an endless loop.
I guess my question is ... what am I missing here? Why would you want the message to go back on the queue if it just means it will be processed over and over again?
<!-- jms connection factory -->
<bean name="jmsConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:ConnectionFactory" />
</bean>
<!-- jms transaction manager -->
<bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
<property name="connectionFactory" ref="jmsConnectionFactory" />
</bean>
<!-- Destination for Inbound_Email_Q -->
<bean name="inboundEmailDestination" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="queue/inbound_Email_Queue" />
</bean>
<!-- JmsTemplate for Inbound_Email_Q -->
<bean name="jmsInboundEmailTemplate" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="jmsConnectionFactory" />
<property name="defaultDestination" ref="inboundEmailDestination" />
<property name="messageConverter" ref="xmlMessageConverter" />
</bean>
<!-- jms asynchronous listener -->
<bean id="emailMessageServiceMdp" class="org.site.wso.core.jms.EmailMessageServiceMdp" />
<bean class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="jmsConnectionFactory"/>
<!-- <property name="transactionManager" ref="jmsTransactionManager" /> -->
<!-- <property name="sessionTransacted" value="true"/> -->
<property name="destination" ref="inboundEmailDestination"/>
<property name="messageListener" ref="messageListener"/>
</bean>
<!-- jms message listener adapter -->
<bean id="messageListener" class="org.springframework.jms.listener.adapter.MessageListenerAdapter">
<constructor-arg>
<bean class="org.site.wso.core.jms.EmailMessageServiceMdp"/>
</constructor-arg>
<property name="messageConverter" ref="xmlMessageConverter"/>
</bean>
And here is my MDP:
public class EmailMessageServiceMdp implements MessageDelegate {
public void handleMessage(Object object) {
EmailMessageRequestVO requestVO = (EmailMessageRequestVO) object;
try {
//Service call that throw exception
} catch (Exception e) {
throw new ApplicationException(e);
}
}
}

The message redelivery is simply the default behaviour for your [configured] JMS implementation. It's up for endless debate what the relative usefulness of this is, but it would seem that rather than discard a message with some potentially unrecoverable data, some sort of retry is a sensible and conservative approach. For example, in your case, you appear to be converting a JMS message to an email message and dispatching to an SMTP server. If the SMTP gateway is down, you might want to hold onto the JMS messages and reprocess them when the gateway comes back up.
In general I would say your options for handling a message that failed processing are (depending on the JMS implementation):
Ditch the message.
Write the message to an error queue.
Redeliver the message after a delay of n seconds.
Redeliver the message n times and then write the message to an error queue.
Redeliver the message after a delay of n seconds x times and then write the message to an error queue.
If you prefer #1, then simply suppress the exception, commit the transaction and wave good bye to the message. For the rest, the JMS configuration (or the destination specific configuration) should handle those.
Additionally, if you want something more specific, you can interrogate the message's getJMSRedelivered() and/or the implementation specific message header property that indicates how many times the message has been redelivered (supported by most JMS implementations, but not standard) and dispose of the message accordingly.