I am writing an application using activemq where I am using the redelivery policy to redeliver the messages. I am using the ActiveMQ's ExponentialBackOff concept.
My question is how does this ExponentialBackOff/setBackOffMultiplier work.
For example in my case I want to redeliver the message till the message expiration time, which is 15 minutes.I want to try to redeliver 10 times within 15 minutes.But ExponentialBackOff makes the message to redeliver beyond the 15 minutes expiry time of the message i.e. the message to be redelivered is still in the pending state even after the expiration time which is 15 minutes.
Why is this? I am kind of confused with this behavior. The redelivery policy I am using is as below.
RedeliveryPolicy queuePolicy = new RedeliveryPolicy();
queuePolicy.setInitialRedeliveryDelay(0);
queuePolicy.setBackOffMultiplier(3);
queuePolicy.setUseExponentialBackOff(true);
queuePolicy.setMaximumRedeliveries(10);
with this RedeliveryPolicy config, the RedeliveryPolicy will make attempts after each time waiting below :
after 1s
after 3s
9s
27s
81s
243s
729s
2187s
6561s
19683s
as you see like this attempts are executed after hours and in the meantime you see messages state is pending.
to prevent these long periods maybe you want to set the maximumRedeliveryDelay=300000L (5 minutes).
Note that
Once a message's redelivery attempts exceeds the maximumRedeliveries configured for the Redelivery Policy, a "Poison ack" is sent back to the broker letting him know that the message was considered a poison pill. The Broker then takes the message and sends it to a Dead Letter Queue so that it can be analyzed later on.
you need to adapt your RedeliveryPolicy because the message is pending as long as maximumRedeliveries is not exceeded.
http://activemq.apache.org/message-redelivery-and-dlq-handling.html
Related
I have configured ActiveMQ redelivery plugin as follows (with max 4 redeliveries)
<redeliveryPlugin fallbackToDeadLetter="true" sendToDlqIfMaxRetriesExceeded="true">
<redeliveryPolicyMap>
<redeliveryPolicyMap>
<defaultEntry>
<redeliveryPolicy initialRedeliveryDelay="5000" maximumRedeliveries="4" redeliveryDelay="10000"/>
</defaultEntry>
</redeliveryPolicyMap>
</redeliveryPolicyMap>
</redeliveryPlugin>
If a client fails to send ACK the message is redelivered. So far so good... However, the max redeliveries are completely ignored by the broker and it keeps redelivering the messages "infinitely" many times. Also the message is never moved to DLQ.
I also tried using:
?jms.redeliveryPolicy.maximumRedeliveries=4 on the connection URI (STOMP Connector), but also with no luck.
Any help is most appreciated!
For a STOMP client I would assume that the broker will not consider the message as being delivered unless the client either ACKs it or NACKs it otherwise it must assume that it never made it to a client and therefore treats it as always having a delivery count of zero. The broker redelivery plugin keys off the message's delivery count so if the message is treated as not having been delivered which in this case it likely is then it will take no action on the message.
Background
We have a data transfer solution with Azure Service Bus as the message broker. We are transferring data from x datasets through x queues - with x dedicated QueueClients as senders. Some senders publish messages at the rate of one message every two seconds, while others publish one every 15 minutes.
The application on the data source side (where senders are) is working just fine, giving us the desired throughput.
On the other side, we have an application with one QueueClient receiver per queue with the following configuration:
maxConcurrentCalls = 1
autoComplete = true (if receive mode = RECEIVEANDDELETE) and false (if receive mode = PEEKLOCK) - we have some receivers where, if they shut-down unexpectedly, would want to preserve the messages in the Service Bus Queue.
maxAutoRenewDuration = 3 minutes (lock duraition on all queues = 30 seconds)
an Executor service with a single thread
The MessageHandler registered with each of these receivers does the following:
public CompletableFuture<Void> onMessageAsync(final IMessage message) {
// deserialize the message body
final CustomObject customObject = (CustomObject)SerializationUtils.deserialize((byte[])message.getMessageBody().getBinaryData().get(0));
// process processDB1() and processDB2() asynchronously
final List<CompletableFuture<Boolean>> processFutures = new ArrayList<CompletableFuture<Boolean>>();
processFutures.add(processDB1(customObject)); // processDB1() returns Boolean
processFutures.add(processDB2(customObject)); // processDB2() returns Boolean
// join both the completablefutures to get the result Booleans
List<Boolean> results = CompletableFuture.allOf(processFutures.toArray(new CompletableFuture[processFutures.size()])).thenApply(future -> processFutures.stream()
.map(CompletableFuture<Boolean>::join).collect(Collectors.toList())
if (results.contains(false)) {
// dead-letter the message if results contains false
return getQueueClient().deadLetterAsync(message.getLockToken());
} else {
// complete the message otherwise
getQueueClient().completeAsync(message.getLockToken());
}
}
We tested with the following scenarios:
Scenario 1 - receive mode = RECEIVEANDDELETE, message publish rate: 30/ minute
Expected Behavior
The messages should be received continuosuly with a constant throughput (which need not necessarily be the throughput at source, where messages are published).
Actual behavior
We observe random, long periods of inactivity from the QueueClient - ranging from minutes to hours - there is no Outgoing Messages from the Service Bus namespace (observed on the Metrics charts) and there are no consumption logs for the same time periods!
Scenario 2 - receive mode = PEEKLOCK, message publish rate: 30/ minute
Expected Behavior
The messages should be received continuosuly with a constant throughput (which need not necessarily be the throughput at source, where messages are published).
Actual behavior
We keep seeing MessageLockLostException constantly after 20-30 minutes into the run of the application.
We tried doing the following -
we reduced the prefetch count (from 20 * processing rate - as mentioned in the Best Practices guide) to a bare minimum (to even 0 in one test cycle), to reduce the no. of messages that are locked for the client
increased the maxAutoRenewDuration to 5 minutes - our processDB1() and processDB2() do not take more than a second or two for almost 90% of the cases - so, I think the lock duration of 30 seconds and maxAutoRenewDuration are not issues here.
removed the blocking CompletableFuture.get() and made the processing synchronous.
None of these tweaks helped us fix the issue. What we observed is that the COMPLETE or RENEWMESSAGELOCK are throwing the MessageLockLostException.
We need help with finding answers for the following:
why is there a long period of inactivity of the QueueClient in scenario 1?
how do we know the MessageLockLostExceptions are thrown, because the locks have indeed expired? we suspect the locks cannot expire too soon, as our processing happens in a second or two. disabling prefetch also did not solve this for us.
Versions and Service Bus details
Java - openjdk-11-jre
Azure Service Bus namespace tier: Standard
Java SDK version - 3.4.0
For Scenario 1 :
If you have the duplicate detection history enabled, there is a possibility of this behavior happening as per the below explained scenario :
I had enabled for 30 seconds. I constantly hit Service bus with duplicate messages ( im my case messages with the same messageid from the client - 30 /per minute). I would be seeing a no activity outgoing for the window. Though the messages are received at the servicebus from the sending client, I was not be able to see them in outgoing messages. You could probably check whether you re encountering the duplicate messages which are filtered - inturn resulting inactivity in outgoing.
Also Note : You can't enable/disable duplicate detection after the queue is created. You can only do so at the time of creating the queue.
The issue was not with the QueueClient object per se. It was with the processes that we were triggering from within the MessageHandler: processDB1(customObject) and processDB2(customObject). since these processes were not optimized, the message consumption dropped and the locks gor expired (in peek-lock mode), as the handler was spending more time (in relation to the rate at which messages were published to the queues) in completing these opertations.
After optimizing the processes, the consumption and completion (in peek-lock mode) were just fine.
I'm using a Java 8 servlet as a Cloud Pub/Sub push endpoint.
On my push endpoint I have a long-running blocking operation, that sometimes runs for over a minute.
After the operation is done, I return a 200 response, acking the message.
If I return a 500 server error, the message is retried, which is expected.
Note that I create my subscription with a maximum allowed deadline ack period of 600 seconds.
What I have noticed is that if my long-running operation runs for over 30 seconds, the message is also retried. Seems like the HTTP connection that is used for push delivery does not live for over 30 seconds or something.
Is this intended? Is it configurable somehow? Thanks in advance.
For push subscriptions, Cloud Pub/Sub does not send a negative acknowledgment (sometimes known as a nack). If your webhook does not return a success code within the acknowledgment deadline, Cloud Pub/Sub retries delivery until the message expires after the subscription's message retention period. You can configure a default acknowledgment deadline for push subscriptions when you create the push subscription (select push subscription and set Acknowledgement deadline).
Note that, unlike for pull subscriptions, the deadline cannot be extended for individual messages. The deadline is effectively the amount of time the endpoint has to respond to the push request.
Yes. It is expected.
Pubsub guarantee at-least one time delivery until you acknowledge the message.
You can add delay in Push subscription by adding setting in Subsccription.
Go To subscription -> Edit ->Acknowledgement deadline ->
set values from 10 Seconds to 600 Seconds(10 Minutes).
Is it possible to rollback async processed message in ActiveMQ? I'm consuming next message while first one is still processing, so while I'm trying to rollback the first message on another (not activemq pool) thread, I'm getting above error. Eventually should I sednd message to DLQ manually?
Message error handling can work a couple ways:
Broker-side 'redelivery policy'. Where the client invokes a rollback n number (default is usually 6 retries) of times and the broker automatically moves the message to a Dead Letter Queue (DLQ)
Client-side. Application consumes the message and then produces to the DLQ.
Option #1 is good for unplanned/planned outages-- database down, etc. Where you want automatic retry. The re-delivery policy can also be configured when the client connects to the broker.
Option #2 is good for 'bad data' scenarios where you know the message will never be able to be processed. This is ideal, because you can move the message on the 1st consumption and not have to reject the message n number of times.
When you combine infinite retry with #1 and include #2 in your application flow, you can have a robust process flow of automatic retry, and move-bad-data-out-of-the-way-quickly. Best of breed =)
ActiveMQ Redelivery policy
I'm invoking:
GetResponse response = channel.basicGet("some.queue", false); // no auto-ack
....
channel.basicAck(deliveryTag, ...);
However, when I invoke basicGet, the messages in the queue stay in "Ready", rather than in "Unacknowledged". I want them to be in unacknowledged, so that I can either basic.ack them (thus discarding them from the queue), or basic.nack them
I'm doing the following to mimic Delaying the ack:
At consumption time
Get(consume) the message form the initial Queue.
Create a "PendingAck_123456" Queue.
123456 is a unique id of the message.
Set the following properties
x-message-ttl (to requeue after
timeout)
x-expires (to make sure the temp queue will be deleted)
x-dead-letter-exchange and x-deal-letter-routing-key to requeue to
the initial Queue upon TTL expiration.
Publish the message Pending ack to this "PendingAck_123456" Queue
Ack the message to delete it from the initial queue
At Acknowledge time
Calculate Queue Name from Message Id and Get from the "PendingAck_123456" Queue
Acknowledge it (no need to call .getBody() ).
That'll delete it from this pending queue, preventing the TTL to requeue it
Remarks
A Queue for only 1 message.. Is that an issue if there are a lot of such Queues ?
A requeued message will be sent at the queue input side.. not at the queue output (as would do a real ack).. There is an impact on the messages order.
Message is copied by the application to the Pending Queue.. This is an additional step that may have impacts on the overall performance.
To mimic a Nack/Reject, you you may want to Copy the message to the Initial Queue, and Ack it from the PendingAck queue. By default, the TTL would do it (later).
When doing ack immediately after the get it works fine. However, in my case, they were separated by a request. And spring's template closes the channel and connection on each execution. So there are three options:
keep one channel and connection open throughout the whole lifetime of the application
have some kind of conversation-scope (or worst-case: use the session) to store the same channel and reuse it.
use one channel per request, acknowledge receipt immediately, and store the messages in memory.
In the former two cases you can't do it with spring's RabbitTemplate