I have JMS queue implementation in my project in which I am sending 100's of messages in one transaction but performing some DB operations before putting it in queue. i.e
//SuedoCode
#Transactional(propagation = Propagation.REQUIRES_NEW)
public void sendMsg(List orders )
{
for(Order order : orders)
{
order.setStatus("SENT");
sendToQueue(order);
}
}
But this transaction is still not committed and receiver picks up the orders before sender's transaction is committed. Now receiver process the messages and change the status again then commits but after that senders transaction commits and it overrides the status which should not happen.
Thus to solve this problem I created a new class (For spring proxy) which has method to change the status of order and this method is in REQUIRES_NEW transaction so the status has changed but If any error occur while sending the message to queue then again status needs to changed (because the previous transaction has already committed).
Please suggest me If this approach is correct or something better can be done.
Thanks in advance
The messages queued need to be part of the same transaction, so that everything gets committed at once.
Lacking that, another solution is to stored updated orders in a list and call sendToQueue outside this method as soon as your sure the transaction has been committed and database has been updated.
Related
I'm trying to understand how transaction works in Spring AMQP. Reading the docs: https://docs.spring.io/spring-amqp/reference/html/#transactions , I know the purpose for enabling transaction in publisher (Best Effort One Phase Commit pattern) but I have no idea why it could be necessary in MessageListener?
Let's take an example:
acknowledgeMode=AUTO
Consume message using #RabbitListener
Insert data into database
Publish message using rabbitTemplate
According to docs: https://docs.spring.io/spring-amqp/reference/html/#acknowledgeMode, if acknowledgeMode is set to AUTO then if any next operation fails, listener will fail too and message will be returned to queue.
Another question is whats the difference between local transaction and external transaction in that case (setting container.setTransactionManager(transactionManager()); or not)?
I would appreciate some clarification :)
Enable transactions in the listener so that any/all downstream RabbitTemplate operations participate in the same transaction.
If there is a failure, the container will rollback the transaction (removing the publishes), nack the message (or messages if batch size is greater than one) and then commit the nacks so the message(s) will be redelivered.
When using an external transaction manager (such as JDBC), the container will synchronize the AMQP transaction with the external transaction.
Downstream templates participate in the transaction regardless of whether it is local (AMQP only) or synchronized.
I am implementing a backend service with Spring Boot. This service receives a REST request and executes some database operations and finally updates the status of the record.
After that, I would like to start a new async process and execute another data manipulation on the same record this way:
#Service
public class ClassA {
#Autowired
private ClassB classB;
#Autowired
private MyEntityRepository repo;
#Transactional
public void doSomething(Long id) {
// executing the business logic
if (isOk()) {
repo.updateStatus(id, Status.VERIFIED)
}
// I need to commit this DB transaction and return.
// But after this transaction is committed, I need
// to start an async process that must work on the
// same record that was updated before.
classB.complete(id);
}
}
And this is my async method:
#Service
public class ClassB {
#Autowired
private MyEntityRepository repo;
#Async
#Transactional(propagation = Propagation.REQUIRES_NEW)
public void complete(Long id) {
Optional<MyEntity> myEntity = repo.findById(id);
if (myEntity.isPresent() && myEntity.get().getStatus == Status.VERIFIED) {
// execute 'business logic B'
}
}
}
The classA.doSomething() is called multiply times with the same id but business logic B must be executed only when the record status in the DB is VERIFIED.
The above solution works fine.
But my concern is the following: My test database is small and the classA.doSomething() method always finishes and closes its transaction BEFORE the classB.complete() starts to check the status of the same record in the DB. I see in the log that the SQLs are executed in the proper order:
* UPDATE STATUS FROM TABLE ... WHERE ID = 1 // doSomething()
* COMMIT
* SELECT * FROM TABLE WHERE ID = 1 // complete()
But is that 100% guaranteed that the 1st, classA.doSomething() method will always finish and commit the transaction before the 2nd classB.complete() async call check the status of the same record?
If the async method classB.complete() will be executed before classA.doSomething() finishes and execute its DB commit then I will break the business logic and the business logic B will be skipped (the new DB transaction will not see the updated status yet) and that will cause a big issue. Maybe this can happen if the database is huge and the commit takes longer than it takes in my small test DB.
Maybe I can operate with the DB transaction isolation levels described here but changing this can cause another issue in another part of the app.
What is the best way to implement this logic properly which guarantees the proper execution order with the async method?
It is NOT GUARANTEED that "the 1st, classA.doSomething() method will always finish and commit the transaction before the 2nd classB.complete() async call check the status of the same record".
Transactions are implemented as some kind of interceptors appropriate for the framework (this is true for CDI too). The method marked #Transactional is intercepted by the framework, so the transaction will not end before the closing } of the method. As a matter of fact, if the transaction was started by another method higher in the stack, it will end even later.
So, ClassB has plenty of time to run and see inconsistent state.
I would place the 1st part of doSomething in a separate REQUIRES_NEW transaction method (you may need to place it in a different class, depending on how you configured transaction interceptors; if you are using AOP, Spring may be able to intercept calls to methods of the same object, otherwise it relies on the injected proxy object to do the interception and calling a method through this will not activate the interceptor; again this is true for other frameworks as well, like CDI and EJB). The method doSomething calls the 1st part method, which finishes in a new transaction, then ClassB can continue asynchronously.
Now, in that case (as correctly pointed out in the comment), there is a chance that the 1st transaction succeeds and the 2nd fails. If this is the case, you will have to put logic in the system about how to compensate for this inconsistent state. Frameworks cannot deal with it because there is not one recipe, it is a per case "treatment". Some thoughts, in case they help: make sure that the state of the system after the 1st transaction clearly says that the second transaction should complete "shortly after". E.g. keep a "1st tx committed at" field; a scheduled task can check this timestamp and take action if it is too far in the past. JMS gives you all this - you get retries and a dead letter queue for the failed cases.
Instead of sending single message in a transaction:
jmsTemplate.convertAndSend(message);
How can I send multiple jms messages in a single transaction?
Is there an example I can loot at?
Start the transaction before calling the template
#Transactional
public void doSends() {
template.convertAndSend(...)
...
template.convertAndSend(...)
}
The transaction commits when the method exits. See the Spring documentation about transactions.
Or, use one the of the template's execute() methods and do the sends in the callback.
It´s more of a conceptual question: I currently have a working activemq queue which is consumed by a Java Spring application. Now I want the queue not to permanently delete the messages until the Java app tells it the message has been correctly saved in DB. After reading documentation I get I have to do it transactional and usa the commit() / rollback() methods. Correct me if I'm wrong here.
My problem comes with every example I find over the internet telling me to configure the app to work this or that way, but my nose tells me I should instead be setting up the queue itself to work the way I want. And I can't find the way to do it.
Otherwise, is the queue just working in different ways depending on how the consumer application is configured to work? What am I getting wrong?
Thanks in advance
The queue it self is not aware of any transactional system but you can pass the 1st parameter boolean to true to create a transactional session but i propose the INDIVIDUAL_ACKNOWLEDGE when creating a session because you can manage messages one by one. Can be set on spring jms DefaultMessageListenerContainer .
ActiveMQSession.INDIVIDUAL_ACKNOWLEDGE
And calling this method to ack a message, unless the method is not called the message is considered as dispatched but not ack.
ActiveMQTextMessage.acknowledge();
UPDATE:
ActiveMQSession.INDIVIDUAL_ACKNOWLEDGE can be used like this :
onMessage(ActiveMQTextMessage message)
try {
do some stuff in the database
jdbc.commit(); (unless auto-commit is enabled on the JDBC)
message.acknowledge();
}
catch (Exception e) {
}
There are 2 kinds of transaction support in ActiveMQ.
JMS transactions - the commit() / rollback() methods on a Session (which is like doing commit() / rollback() on a JDBC connection)
XA Transactions - where the XASession acts as an XAResource by communicating with the Message Broker, rather like a JDBC Connection takes place in an XA transaction by communicating with the database.
http://activemq.apache.org/how-do-transactions-work.html
Should I use XA transactions (two phase commit?)
A common use of JMS is to consume messages from a queue or topic, process them using a database or EJB, then acknowledge / commit the message.
If you are using more than one resource; e.g. reading a JMS message and writing to a database, you really should use XA - its purpose is to provide atomic transactions for multiple transactional resources. For example there is a small window from when you complete updating the database and your changes are committed up to the point at which you commit/acknowledge the message; if there is a network/hardware/process failure inside that window, the message will be redelivered and you may end up processing duplicates.
http://activemq.apache.org/should-i-use-xa.html
Code...
#Transactional
#JmsListener(destination = "QueueA")
public void process(String input) {
doSomethingWhichMayThrowException(input);
}
Consider following situation where...
Transaction is started (using Spring #Transactional annotation)
Persistent JMS message is read from QueueA (Queue use disk as message storage)
Disk is full and do not accept any write operations
Exception happens and transaction is rolled back
Is message lost?
If it's not then how message is read from queue under transaction (step 2)?
Is some kind of a queue browser used so message is read from queue but not consumed?
Is message lost?
No, the message is NOT lost as the transaction is rolledback.
If it's not then how message is read from queue under transaction
(step 2) ?
Once after the message listener's process()/onMessage() method completes and returns with a success or exception, then internally message acknowledgment (default is AUTO_ACKNOWLEDGE) happens (which is the last thing implicitly happens) to the JMS provider(IBMMQ, ActiveMQ, SonicMQ, etc..) which tells that the transaction is successful or not.
If the Transaction is successful, JMS provider deletes the message from the queue/topic.
If the Transaction is NOT successful, JMS provider preserve the message as is (until message TimetoLive expires).
Is some kind of a queue browser used so message is read from queue but
not consumed ?
You can think that it is like queue browser concept, but it is upto the implementation of the JMS provider how do they implement this internally. In order to achieve this, the message broker just reads the message content, but do not delete the actual message from the queue/topic until the acknowledgement is received from the message listener's process()/onMessage() method for the current transaction.