I am currently testing a rather simple example concerning messaging transactions in connection with database transactions with spring amqp.
The use case is as follows:
message is received
a message is sent
database is updated
#Transactional
public void handleMessage(EventPayload event) {
MyEntity entity = new MyEntity();
entity.setName(event.getName());
rabbitTemplate.convertAndSend("myExchange", "payload.create", payload);
MyEntity savedEntity = entityRepository.save(entity);
}
The expected behavior in case of a failure during the database operation is that the received message is rolled back to the bus (DefaultRequeueRejected = false) and goes into a dead letter queue. Also the message sent should be rolled back.
I can achieve this with the following configuration:
#Bean
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory, MessageConverter messageConverter) {
RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setMessageConverter(messageConverter);
rabbitTemplate.setChannelTransacted(true);
return rabbitTemplate;
}
#Bean
SimpleMessageListenerContainer subscriberListenerContainer(ConnectionFactory connectionFactory,
MessageListenerAdapter listenerAdapter,
PlatformTransactionManager transactionManager) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(SUBSCRIBER_QUEUE_NAME);
container.setMessageListener(listenerAdapter);
container.setChannelTransacted(true);
container.setTransactionManager(transactionManager);
container.setDefaultRequeueRejected(false);
return container;
}
So this works fine - what I do not understand is that the observed behavior is exactly the same if I do not set the transaction manager on the SimpleMessageListenerContainer. So if I configure the following the bebavior does not change:
#Bean
SimpleMessageListenerContainer subscriberListenerContainer(ConnectionFactory connectionFactory,
MessageListenerAdapter listenerAdapter) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(SUBSCRIBER_QUEUE_NAME);
container.setMessageListener(listenerAdapter);
container.setDefaultRequeueRejected(false);
return container;
}
Can someone explain what is happening there? Why is the second case also working? What is different internally if the PlatformTransactionManager is registered on the SimpleMessageListenerContainer.
Assuming the transactionManager is your database tm, since your listener is #Transactional, there's not a lot of difference for these scenarios.
In the first case, the transaction is started by the container before the listener is invoked (actually before the message is retrieved from an internal queue so a transaction will start even if there's no message).
In the second case, the transaction is started by the transaction interceptor when we invoke the listener.
Consider the case where the listener is not transactional, but some downstream component is; let's say the listener invokes that component successfully, then does some more work before throwing an exception. In that case, the DB commit would be successful and the message rejected. This might not be the desired behavior, especially if messages are requeued. In cases like this, it's generally better to synchronize the rabbit transaction with the database transaction by injecting the database tm.
In your case, there is little chance of a failure between the db commit and the rabbit ack so this really doesn't apply in your case and you don't need a tm in the container.
Related
I am using the #JmsListener annotation from Spring JMS to consume messages from an IBM MQ queue, but when an exception occurs the database updates are not being rolled back. The JMS listener is only rolling back the MQ messages, i.e dequeuing the message and putting it back in the queue. How can I ensure that the database updates are also rolled back in this scenario?
Sample code:
#Service
#Transactional
#JMSListener(containerFactory="jmsListenerContainerFactory",destination="ibm.mq.request")
public class TestListener {
public void receive message(String message) {
1: // convert message to object apply business logic
2: // insert into order_table;
3: // convert object back to string and put it into response queue
4: jmsTemplate.convertAndSend("ibm.mq.response",message);
}
}
When the listener starts, picks a message from MQ, converts it to an Object, applies some logic, and persists in the database.
Issue: When an exception occurs at step 4, Listener picks up the same message (which indicates the message is still in the MQ) but the database insert did not roll back.
Can someone help me how to roll back database insert/updates?
JMS config file :
#Bean
public JmsTransactionManager jmsTransactionManager(MQConnectionFactory connectionFactory){
return new JmsTransactionManager(connectionFactory);
}
#Bean
public JmsTemplate jmsTemplate(MQConnectionFactory connectionFactory){
JmsTemplate jmsTemplate = new JmsTemplate (connectionFactory);
jmsTemplate.setSessionTransacted(true);
return jmsTemplate;
}
The JmsTransactionManager is only capable of managing transactions on the connections that are issued by the connection factory that it contains. In order to have synchronized transactions between JMS and your database, you will need to have a JTA transaction manager configured. This is best done by having the JTA transaction manager being the PlatformTransactionManager.
With synchronized transactions, you will have DUPS-OK functionality. The JMS message listener will start a transaction in a new transaction context. The JPA database persistence will then join the existing JTA transaction and there will be two transactions in the transaction context.
I am trying to get my head around the Spring CachingConnectionFactory.
ActiveMQ documentation recommends when using JmsTemplate that a Spring CachingConnectionFactory or an ActiveMQ PooledConnectionFactory is used as the connection factory implementation.
I understand this because using the normal ConnectionFactory a connection is created, a session started, and both are closed for EVERY call of the jmsTemplate.send() which is very wasteful.
So I am trying to implement a custom JmsTemplate bean with a CachingConnectionFactory for use where I may have many requests that are A) Persisted to DB B) Enqueued JMS.
#Configuration
public class JMSConfig {
#Autowired
private CachingConnectionFactory cachingConnectionFactory;
#Bean
#Qualifier("jmsTemplateCached")
public JmsTemplate jmsTemplateCachingConnectionFactory() {
cachingConnectionFactory.setSessionCacheSize(10);
JmsTemplate jmsTemplate = new JmsTemplate();
jmsTemplate.setDeliveryMode(DeliveryMode.PERSISTENT);
jmsTemplate.setSessionAcknowledgeMode(JmsProperties.AcknowledgeMode.CLIENT.getMode());
jmsTemplate.setSessionTransacted(true);
jmsTemplate.setDeliveryPersistent(true);
jmsTemplate.setConnectionFactory(cachingConnectionFactory);
return jmsTemplate;
}
}
My first question regards the Spring Docs For CachngConnecionFactory which say:
SingleConnectionFactory subclass that adds Session caching as well MessageProducer caching. This ConnectionFactory also switches the "reconnectOnException" property to "true" by default, allowing for automatic recovery of the underlying Connection.
By default, only one single Session will be cached, with further requested Sessions being created and disposed on demand. Consider raising the "sessionCacheSize" value in case of a high-concurrency environment.
But then in bold:
NOTE: This ConnectionFactory requires explicit closing of all Sessions obtained from its shared Connection. This is the usual recommendation for native JMS access code anyway. However, with this ConnectionFactory, its use is mandatory in order to actually allow for Session reuse.
Does this mean I only need to close sessions if I create a connection "manually" via the template or my CachingConnectionFactory bean? In other words like:
Connection connection = jmsTemplateCached.getConnectionFactory().createConnection();
Session sess = connection.createSession(true, JmsProperties.AcknowledgeMode.CLIENT.getMode());
MessageProducer producer = sess.createProducer(activeMQQueue);
try {
producer.send(activeMQQueue, new ActiveMQTextMessage());
sess.commit();
} catch (JMSException e) {
sess.rollback();
} finally {
sess.close();
}
If I use the template like below, should I close or not close the session?
#Autowired
public JmsTemplate jmsTemplateCached;
#Transactional
public InboundResponse peristAndEnqueueForProcessing(InboundForm inboundForm) throws IrresolvableException, JsonProcessingException, JMSException {
//Removed for clarity, an entity has been persisted and is then to be enqueued via JMS.
log.debug("Queue For Processing : {}", persistedRequest);
String serialisedMessage = objectMapper.writeValueAsString(persistedRequest);
ActiveMQTextMessage activeMQTextMessage = new ActiveMQTextMessage();
activeMQTextMessage.setText(serialisedMessage);
//Will throw JMS Exception on failure
Session sessionUsed = jmsTemplateCached.execute((session, messageProducer) -> {
messageProducer.send(activeMQQueue, activeMQTextMessage);
session.commit();
return session;
});
return response;
}
Secondly, if the above jmsTemplate.execute() throws an exception, what happens to the session? Will it rollback after x time?
The JmsTemplate reliably closes its resources after each operation (returning the session to the cache), including execute().
That comment is related to user code using sessions directly; the close operation is intercepted and used to return the session to the cache, instead of actually closing it. You MUST call close, otherwise the session will be orphaned.
Yes, the transaction will roll back (immediately) if its sessionTransacted is true.
You should NOT call commit - the template will do that when execute exits normally (if it is sessionTransacted).
We have a spring boot application which consumes messages from IBM MQ does some transformation and publishes the result to a Kafka topic. We use https://spring.io/projects/spring-kafka for this. I am aware that Kafka does not supports XA; however, in the documentation I found some inputs about using a ChainedKafkaTransactionManager to chain multiple transaction managers and synchronise the transactions. The same documentation also provides an example about how to synchronise Kafka and database while reading messages from Kafka and storing them in the database.
I follow the same example in my se case and chained the JmsTransactionManager with KafkaTransactionManager under the umbrella of a ChainedKafkaTransactionManager. The bean definitions follows below:
#Bean({"mqListenerContainerFactory"})
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(this.connectionFactory());
factory.setTransactionManager(this.jmsTransactionManager());
return factory;
}
#Bean
public JmsTransactionManager jmsTransactionManager() {
return new JmsTransactionManager(this.connectionFactory());
}
#Bean("chainedKafkaTransactionManager")
public ChainedKafkaTransactionManager<?, ?> chainedKafkaTransactionManager(
JmsTransactionManager jmsTransactionManager, KafkaTransactionManager kafkaTransactionManager) {
return new ChainedKafkaTransactionManager<>(kafkaTransactionManager, jmsTransactionManager);
}
#Transactional(transactionManager = "chainedKafkaTransactionManager", rollbackFor = Throwable.class)
#JmsListener(destination = "${myApp.sourceQueue}", containerFactory = "mqListenerContainerFactory")
public void receiveMessage(#Headers Map<String, Object> jmsHeaders, String message) {
// Processing the message here then publishing it to Kafka using KafkaTemplate
kafkaTemplate.send(sourceTopic,transformedMessage);
// Then throw an exception just to test the transaction behaviour
throw new RuntimeException("Not good Pal!");
}
When running the application what is happening is that he message keep getting rollbacked into the MQ Queue but messages keep growing in Kafka topic which means to me that kafkaTemplate interaction does not get rollbacked.
If I understand well according with the documentation this should not be the case. "If a transaction is active, any KafkaTemplate operations performed within the scope of the transaction use the transaction’s Producer."
In our application.yaml we configured the Kafka producer to use transactions by setting up spring.kafka.producer.transaction-id-prefix
The question is what I am missing here and how should I fix it.
Thank you in advance for your inputs.
Consumers can see uncommitted records by default; set the isolation.level consumer property to read_committed to avoid receiving records from rolled-back transactions.
I'm using JmsMessageSender inside WebServiceTemplate for Spring-WS communication over JMS. I have to work on topics, so I used setPubSubDomain method of JmsMessageSender's superclass, and the messages go correctly on target topic. However, the for handling response, a temporary queue is created, not topic. How can I setup spring beans to have temporary topic for the response, not queue?
To add one hint, there is a setReplyPubSubDomain method of AbstractMessageListenerContainer class, which looks exactly what I need, but I have never used this listener container and I'm not sure how could I wrap it into my beans.
My configuration below:
#Bean
public WebServiceTemplate webServiceTemplate() {
WebServiceTemplate webServiceTemplate = new WebServiceTemplate();
webServiceTemplate.setMessageFactory(messageFactory());
webServiceTemplate.setMessageSender(messageSender());
webServiceTemplate.setDefaultUri("jms:topicname.topicname.topicname?priority=3&deliveryMode=NON_PERSISTENT&messageType=TEXT_MESSAGE");
return webServiceTemplate;
}
#Bean
public JmsMessageSender messageSender() {
JmsMessageSender messageSender = new JmsMessageSender();
messageSender.setConnectionFactory(connectionFactory());
messageSender.setPubSubDomain(true);
return messageSender;
}
I am using Spring's #JmsListener (spring-jms-4.3.4.RELEASE.jar) for receiving messages from ActiveMQ using the below code:
#Component
public class TopicSubscriber {
#JmsListener(destination="xyz.topic1", subscription="xyz_topic_durable_subscription")
public void send(Product product) {
System.out.println(" reveived message ***"+product);
}
}
As per the Spring API's documentation (link given below), the above code should create a durable subscription with subscription name as xyz_topic_durable_subscription:
http://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/jms/annotation/JmsListener.html#subscription--
But, the issue is that the above code creates only Non-Durable subscription which I could find by monitoring the ActiveMQ using admin console (added screenshot below, look for 'xyz.topic1' Destination under 'Active Non-Durable Topic Subscribers' section).
Are there any changes to be made in the code to make the durable subscription ?
You need to configure the ListenerContainerFactory appropriately:
#Bean
public JmsListenerContainerFactory<?> myFactory(ConnectionFactory connectionFactory,
DefaultJmsListenerContainerFactoryConfigurer configurer) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setPubSubDomain(true);
factory.setSubscriptionDurable(true);
factory.setClientId("jmsDemo");
// This provides all boot's default to this factory, including the message converter
configurer.configure(factory, connectionFactory);
// You could still override some of Boot's default if necessary.
return factory;
}
There interesting part is here:
factory.setSubscriptionDurable(true);
factory.setClientId("jmsDemo");
Now when you enter the ActiveMQ WebConsole you should see this:
In the answer marked as correct above, the code:
factory.setPubSubDomain(true);
factory.setSubscriptionDurable(true);
factory.setClientId("jmsDemo");
must come after
configurer.configure(factory, connectionFactory);
or you will lose those settings.
You also need to configure the listener container factory to create a container for durable subscriptions.