We have a spring boot application which consumes messages from IBM MQ does some transformation and publishes the result to a Kafka topic. We use https://spring.io/projects/spring-kafka for this. I am aware that Kafka does not supports XA; however, in the documentation I found some inputs about using a ChainedKafkaTransactionManager to chain multiple transaction managers and synchronise the transactions. The same documentation also provides an example about how to synchronise Kafka and database while reading messages from Kafka and storing them in the database.
I follow the same example in my se case and chained the JmsTransactionManager with KafkaTransactionManager under the umbrella of a ChainedKafkaTransactionManager. The bean definitions follows below:
#Bean({"mqListenerContainerFactory"})
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(this.connectionFactory());
factory.setTransactionManager(this.jmsTransactionManager());
return factory;
}
#Bean
public JmsTransactionManager jmsTransactionManager() {
return new JmsTransactionManager(this.connectionFactory());
}
#Bean("chainedKafkaTransactionManager")
public ChainedKafkaTransactionManager<?, ?> chainedKafkaTransactionManager(
JmsTransactionManager jmsTransactionManager, KafkaTransactionManager kafkaTransactionManager) {
return new ChainedKafkaTransactionManager<>(kafkaTransactionManager, jmsTransactionManager);
}
#Transactional(transactionManager = "chainedKafkaTransactionManager", rollbackFor = Throwable.class)
#JmsListener(destination = "${myApp.sourceQueue}", containerFactory = "mqListenerContainerFactory")
public void receiveMessage(#Headers Map<String, Object> jmsHeaders, String message) {
// Processing the message here then publishing it to Kafka using KafkaTemplate
kafkaTemplate.send(sourceTopic,transformedMessage);
// Then throw an exception just to test the transaction behaviour
throw new RuntimeException("Not good Pal!");
}
When running the application what is happening is that he message keep getting rollbacked into the MQ Queue but messages keep growing in Kafka topic which means to me that kafkaTemplate interaction does not get rollbacked.
If I understand well according with the documentation this should not be the case. "If a transaction is active, any KafkaTemplate operations performed within the scope of the transaction use the transaction’s Producer."
In our application.yaml we configured the Kafka producer to use transactions by setting up spring.kafka.producer.transaction-id-prefix
The question is what I am missing here and how should I fix it.
Thank you in advance for your inputs.
Consumers can see uncommitted records by default; set the isolation.level consumer property to read_committed to avoid receiving records from rolled-back transactions.
Related
I am using the #JmsListener annotation from Spring JMS to consume messages from an IBM MQ queue, but when an exception occurs the database updates are not being rolled back. The JMS listener is only rolling back the MQ messages, i.e dequeuing the message and putting it back in the queue. How can I ensure that the database updates are also rolled back in this scenario?
Sample code:
#Service
#Transactional
#JMSListener(containerFactory="jmsListenerContainerFactory",destination="ibm.mq.request")
public class TestListener {
public void receive message(String message) {
1: // convert message to object apply business logic
2: // insert into order_table;
3: // convert object back to string and put it into response queue
4: jmsTemplate.convertAndSend("ibm.mq.response",message);
}
}
When the listener starts, picks a message from MQ, converts it to an Object, applies some logic, and persists in the database.
Issue: When an exception occurs at step 4, Listener picks up the same message (which indicates the message is still in the MQ) but the database insert did not roll back.
Can someone help me how to roll back database insert/updates?
JMS config file :
#Bean
public JmsTransactionManager jmsTransactionManager(MQConnectionFactory connectionFactory){
return new JmsTransactionManager(connectionFactory);
}
#Bean
public JmsTemplate jmsTemplate(MQConnectionFactory connectionFactory){
JmsTemplate jmsTemplate = new JmsTemplate (connectionFactory);
jmsTemplate.setSessionTransacted(true);
return jmsTemplate;
}
The JmsTransactionManager is only capable of managing transactions on the connections that are issued by the connection factory that it contains. In order to have synchronized transactions between JMS and your database, you will need to have a JTA transaction manager configured. This is best done by having the JTA transaction manager being the PlatformTransactionManager.
With synchronized transactions, you will have DUPS-OK functionality. The JMS message listener will start a transaction in a new transaction context. The JPA database persistence will then join the existing JTA transaction and there will be two transactions in the transaction context.
I am trying to get my head around the Spring CachingConnectionFactory.
ActiveMQ documentation recommends when using JmsTemplate that a Spring CachingConnectionFactory or an ActiveMQ PooledConnectionFactory is used as the connection factory implementation.
I understand this because using the normal ConnectionFactory a connection is created, a session started, and both are closed for EVERY call of the jmsTemplate.send() which is very wasteful.
So I am trying to implement a custom JmsTemplate bean with a CachingConnectionFactory for use where I may have many requests that are A) Persisted to DB B) Enqueued JMS.
#Configuration
public class JMSConfig {
#Autowired
private CachingConnectionFactory cachingConnectionFactory;
#Bean
#Qualifier("jmsTemplateCached")
public JmsTemplate jmsTemplateCachingConnectionFactory() {
cachingConnectionFactory.setSessionCacheSize(10);
JmsTemplate jmsTemplate = new JmsTemplate();
jmsTemplate.setDeliveryMode(DeliveryMode.PERSISTENT);
jmsTemplate.setSessionAcknowledgeMode(JmsProperties.AcknowledgeMode.CLIENT.getMode());
jmsTemplate.setSessionTransacted(true);
jmsTemplate.setDeliveryPersistent(true);
jmsTemplate.setConnectionFactory(cachingConnectionFactory);
return jmsTemplate;
}
}
My first question regards the Spring Docs For CachngConnecionFactory which say:
SingleConnectionFactory subclass that adds Session caching as well MessageProducer caching. This ConnectionFactory also switches the "reconnectOnException" property to "true" by default, allowing for automatic recovery of the underlying Connection.
By default, only one single Session will be cached, with further requested Sessions being created and disposed on demand. Consider raising the "sessionCacheSize" value in case of a high-concurrency environment.
But then in bold:
NOTE: This ConnectionFactory requires explicit closing of all Sessions obtained from its shared Connection. This is the usual recommendation for native JMS access code anyway. However, with this ConnectionFactory, its use is mandatory in order to actually allow for Session reuse.
Does this mean I only need to close sessions if I create a connection "manually" via the template or my CachingConnectionFactory bean? In other words like:
Connection connection = jmsTemplateCached.getConnectionFactory().createConnection();
Session sess = connection.createSession(true, JmsProperties.AcknowledgeMode.CLIENT.getMode());
MessageProducer producer = sess.createProducer(activeMQQueue);
try {
producer.send(activeMQQueue, new ActiveMQTextMessage());
sess.commit();
} catch (JMSException e) {
sess.rollback();
} finally {
sess.close();
}
If I use the template like below, should I close or not close the session?
#Autowired
public JmsTemplate jmsTemplateCached;
#Transactional
public InboundResponse peristAndEnqueueForProcessing(InboundForm inboundForm) throws IrresolvableException, JsonProcessingException, JMSException {
//Removed for clarity, an entity has been persisted and is then to be enqueued via JMS.
log.debug("Queue For Processing : {}", persistedRequest);
String serialisedMessage = objectMapper.writeValueAsString(persistedRequest);
ActiveMQTextMessage activeMQTextMessage = new ActiveMQTextMessage();
activeMQTextMessage.setText(serialisedMessage);
//Will throw JMS Exception on failure
Session sessionUsed = jmsTemplateCached.execute((session, messageProducer) -> {
messageProducer.send(activeMQQueue, activeMQTextMessage);
session.commit();
return session;
});
return response;
}
Secondly, if the above jmsTemplate.execute() throws an exception, what happens to the session? Will it rollback after x time?
The JmsTemplate reliably closes its resources after each operation (returning the session to the cache), including execute().
That comment is related to user code using sessions directly; the close operation is intercepted and used to return the session to the cache, instead of actually closing it. You MUST call close, otherwise the session will be orphaned.
Yes, the transaction will roll back (immediately) if its sessionTransacted is true.
You should NOT call commit - the template will do that when execute exits normally (if it is sessionTransacted).
I wanted to configure exclusive consumer for ActiveMQ with Spring boot
Configuring with java is easy
queue = new ActiveMQQueue("TEST.QUEUE?consumer.exclusive=true");
consumer = session.createConsumer(queue);
But with Spring boot, listener is configured as below.
#JmsListener(destination = "TEST.QUEUE", containerFactory = "myFactory")
public void receiveMessage(Object message) throws Exception {
......
}
Now, how to make this exclusive consumer? Does the below work?
#JmsListener(destination = "TEST.QUEUE?consumer.exclusive=true", containerFactory = "myFactory")
public void receiveMessage(Object message) throws Exception {
......
}
Yes, it's working this way.
Just set a breakpoint to the org.apache.activemq.command.ActiveMQQueue constructor and run your application in debug mode.
You will see that Spring Boot is calling
new ActiveMQQueue("TEST.QUEUE?consumer.exclusive=true") which corresponds to the official ActiveMQ documentation:
https://activemq.apache.org/exclusive-consumer
Moreavor you can go to the ActiveMQ admin and browse the active consumers of this queue: you will now see that the exclusive flag is set to true for your consumer.
We are planning to use Kafka for queueing in our application. I have some bit of experience in RabbitMQ and Spring.
With RabbitMQ and Spring, we used to manage queue creation while starting up the spring service.
With Kafka, I'm not sure what could be the best way to create the topics? Is there a way to manage the topics with Spring.
Or, should we write a separate script which helps in creating topics? Maintaining a separate script for creating topics seems a bit weird for me.
Any suggestions will be appreciated.
In spring it is possible to create topics during the start of the application using beans:
#Bean
public KafkaAdmin admin() {
Map<String, Object> configs = new HashMap<>();
configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,
StringUtils.arrayToCommaDelimitedString(kafkaEmbedded().getBrokerAddresses()));
return new KafkaAdmin(configs);
}
#Bean
public NewTopic topic1() {
return new NewTopic("foo", 10, (short) 2);
}
Alternatively you can write your own create topics by autowiring the AdminClient, so for instance reading the list from an input file or specify advanced properties such as partition numbers:
#Autowired
private KafkaAdmin admin;
//...your implementation
Also note that since Kafka 1.1.0 auto.create.topics.enable is enabled by default (see Broker configs).
For more information refer to the spring-kafka docs
To automatically create a Kafka topic in Spring Boot, only this is required:
#Bean
public NewTopic topic1() {
return new NewTopic("foo", 10, (short) 2);
//foo: topic name
//10: number of partitions
//2: replication factor
}
The Kafka Admin is being automatically created and configured by Spring Boot.
Version 2.3 of Spring Kafka introduced a TopicBuilder class, to make building topics fluent and more intuitive:
#Bean
public NewTopic topic(){
return TopicBuilder.name("foo")
.partitions(10)
.replicas(2)
.build();
}
I am currently testing a rather simple example concerning messaging transactions in connection with database transactions with spring amqp.
The use case is as follows:
message is received
a message is sent
database is updated
#Transactional
public void handleMessage(EventPayload event) {
MyEntity entity = new MyEntity();
entity.setName(event.getName());
rabbitTemplate.convertAndSend("myExchange", "payload.create", payload);
MyEntity savedEntity = entityRepository.save(entity);
}
The expected behavior in case of a failure during the database operation is that the received message is rolled back to the bus (DefaultRequeueRejected = false) and goes into a dead letter queue. Also the message sent should be rolled back.
I can achieve this with the following configuration:
#Bean
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory, MessageConverter messageConverter) {
RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setMessageConverter(messageConverter);
rabbitTemplate.setChannelTransacted(true);
return rabbitTemplate;
}
#Bean
SimpleMessageListenerContainer subscriberListenerContainer(ConnectionFactory connectionFactory,
MessageListenerAdapter listenerAdapter,
PlatformTransactionManager transactionManager) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(SUBSCRIBER_QUEUE_NAME);
container.setMessageListener(listenerAdapter);
container.setChannelTransacted(true);
container.setTransactionManager(transactionManager);
container.setDefaultRequeueRejected(false);
return container;
}
So this works fine - what I do not understand is that the observed behavior is exactly the same if I do not set the transaction manager on the SimpleMessageListenerContainer. So if I configure the following the bebavior does not change:
#Bean
SimpleMessageListenerContainer subscriberListenerContainer(ConnectionFactory connectionFactory,
MessageListenerAdapter listenerAdapter) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(SUBSCRIBER_QUEUE_NAME);
container.setMessageListener(listenerAdapter);
container.setDefaultRequeueRejected(false);
return container;
}
Can someone explain what is happening there? Why is the second case also working? What is different internally if the PlatformTransactionManager is registered on the SimpleMessageListenerContainer.
Assuming the transactionManager is your database tm, since your listener is #Transactional, there's not a lot of difference for these scenarios.
In the first case, the transaction is started by the container before the listener is invoked (actually before the message is retrieved from an internal queue so a transaction will start even if there's no message).
In the second case, the transaction is started by the transaction interceptor when we invoke the listener.
Consider the case where the listener is not transactional, but some downstream component is; let's say the listener invokes that component successfully, then does some more work before throwing an exception. In that case, the DB commit would be successful and the message rejected. This might not be the desired behavior, especially if messages are requeued. In cases like this, it's generally better to synchronize the rabbit transaction with the database transaction by injecting the database tm.
In your case, there is little chance of a failure between the db commit and the rabbit ack so this really doesn't apply in your case and you don't need a tm in the container.