We Keep Coding

Multiple queues receiving same message from virtual topic creates a deadletter entry for one queue only

I'm am using Virtual Destinations to implement Publish Subscribe model in ActiveMQ 5.15.13.
I have a virtual topic VirtualTopic and there are two queues bound to it. Each queue has its own redelivery policy. Let's say Queue 1 will retry message 2 times in case there is an exception while processing the message and Queue 2 will retry message 3 times. Post retry message will be sent to deadletter queue. I'm also using Individual Dead letter Queue strategy so that each queue has it's own deadletter queue.
I've observed that when a message is sent to VirtualTopic, the message with same message id is delivered to both the queues. I'm facing an issue where if the consumers of both queues are not able to process the message successfully. The message destined for Queue 1 is moved to deadletter queue after retrying for 2 times. But there is no deadletter queue for Queue 2, though message in Queue 2 is retried for 3 times.
Is it the expected behavior?
Code:
public class ActiveMQRedelivery {
private final ActiveMQConnectionFactory factory;
public ActiveMQRedelivery(String brokerUrl) {
factory = new ActiveMQConnectionFactory(brokerUrl);
factory.setUserName("admin");
factory.setPassword("password");
factory.setAlwaysSyncSend(false);
}
public void publish(String topicAddress, String message) {
final String topicName = "VirtualTopic." + topicAddress;
try {
final Connection producerConnection = factory.createConnection();
producerConnection.start();
final Session producerSession = producerConnection.createSession(false, AUTO_ACKNOWLEDGE);
final MessageProducer producer = producerSession.createProducer(null);
final TextMessage textMessage = producerSession.createTextMessage(message);
final Topic topic = producerSession.createTopic(topicName);
producer.send(topic, textMessage, PERSISTENT, DEFAULT_PRIORITY, DEFAULT_TIME_TO_LIVE);
} catch (JMSException e) {
throw new RuntimeException("Message could not be published", e);
}
}
public void initializeConsumer(String queueName, String topicAddress, int numOfRetry) throws JMSException {
factory.getRedeliveryPolicyMap().put(new ActiveMQQueue("*." + queueName + ".>"),
getRedeliveryPolicy(numOfRetry));
Connection connection = factory.createConnection();
connection.start();
final Session consumerSession = connection.createSession(false, CLIENT_ACKNOWLEDGE);
final Queue queue = consumerSession.createQueue("Consumer." + queueName +
".VirtualTopic." + topicAddress);
final MessageConsumer consumer = consumerSession.createConsumer(queue);
consumer.setMessageListener(message -> {
try {
System.out.println("in listener --- " + ((ActiveMQDestination)message.getJMSDestination()).getPhysicalName());
consumerSession.recover();
} catch (JMSException e) {
e.printStackTrace();
}
});
}
private RedeliveryPolicy getRedeliveryPolicy(int numOfRetry) {
final RedeliveryPolicy redeliveryPolicy = new RedeliveryPolicy();
redeliveryPolicy.setInitialRedeliveryDelay(0);
redeliveryPolicy.setMaximumRedeliveries(numOfRetry);
redeliveryPolicy.setMaximumRedeliveryDelay(-1);
redeliveryPolicy.setRedeliveryDelay(0);
return redeliveryPolicy;
}
}
Test:
public class ActiveMQRedeliveryTest {
private static final String brokerUrl = "tcp://0.0.0.0:61616";
private ActiveMQRedelivery activeMQRedelivery;
#Before
public void setUp() throws Exception {
activeMQRedelivery = new ActiveMQRedelivery(brokerUrl);
}
#Test
public void testMessageRedeliveries() throws Exception {
String topicAddress = "testTopic";
activeMQRedelivery.initializeConsumer("queue1", topicAddress, 2);
activeMQRedelivery.initializeConsumer("queue2", topicAddress, 3);
activeMQRedelivery.publish(topicAddress, "TestMessage");
Thread.sleep(3000);
}
#After
public void tearDown() throws Exception {
}
}

I recently came across this problem. To fix this there are 2 attributes that needs to be added to individualDeadLetterStrategy as below
<deadLetterStrategy>
<individualDeadLetterStrategy destinationPerDurableSubscriber="true" enableAudit="false" queuePrefix="DLQ." useQueueForQueueMessages="true"/>
</deadLetterStrategy>
Explanation of attributes:
destinationPerDurableSubscriber - To enable a separate destination per durable subscriber.
enableAudit - The dead letter strategy has a message audit that is enabled by default. This prevents duplicate messages from being added to the configured DLQ. When the attribute is enabled, the same message that isn't delivered for multiple subscribers to a topic will only be placed on one of the subscriber DLQs when the destinationPerDurableSubscriber attribute is set to true i.e. say two consumers fail to acknowledge the same message for the topic, that message will only be placed on the DLQ for one consumer and not the other.

Consuming messages in batches in amazon sqs, alpine sqs spring boot

I had configured SQS listener to consume messages in List of Messages but I am only getting a single message at a time and getting error as cannot convert model.StudentData to the instance of java.util.ArrayList<com.amazonaws.services.sqs.model.Message>
my code is :-
#SqsListener(value = "${queueName}", deletionPolicy = SqsMessageDeletionPolicy.NEVER)
public void receiveMessage(final StudentData studentData,
#Header("SenderId") final String senderId, final Acknowledgment acknowledgment) {
// business logic
acknowledgment.acknowledge();
}
Any suggestion on how to configure sqs listener to consume multiple messages
any help will be appreciated

solution for the above issue is :-
final ExecutorService executorService = Executors.newSingleThreadExecutor();
executorService.execute(() -> {
while (true) {
final String queueUrl = amazonSqs.getQueueUrl("enter your queue name").getQueueUrl();
final var receiveMessageRequest = new ReceiveMessageRequest(queueUrl)
.withWaitTimeSeconds(20);
List<Message> messages = amazonSqs.receiveMessage(receiveMessageRequest).getMessages();
while (messages.size() > 0) {
for (final Message queueMessage : messages) {
try {
String message = queueMessage.getBody();
amazonSqs.deleteMessage(new DeleteMessageRequest(queueUrl, queueMessage
.getReceiptHandle()));
} catch (Exception e) {
log.error("Received message with errors " + e);
}
}
messages = amazonSqs.receiveMessage(new ReceiveMessageRequest(queueUrl)).getMessages();
}
}
});
executorService.shutdown();

The SQS listener annotation provides the most simple configuration, it will consume messages one by one. This limitation comes directly from spring's
QueueMessagingTemplate.
To consume batches you could use AmazonSQS client directly.
#Autowire AmazonSQSAsync amazonSqs;
...
String queueUrl = amazonSqs.getQueueUrl("queueName").getQueueUrl();
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest();
receiveMessageRequest.setQueueUrl(queueUrl);
receiveMessageRequest.setWaitTimeSeconds(10); // Listener for messages in the next 10 seconds
receiveMessageRequest.setMaxNumberOfMessages(1000); // If 10000 messages are read stop listening
ReceiveMessageResult receiveMessageResult = amazonSqs.receiveMessage(receiveMessageRequest);
receiveMessageResult.getMessages(); // batch of messages

RabbitHandler: How to catch "ListenerExecutionFailedException: Listener method 'no match' threw exception" correctly and proceed working

For an application I am doing some tests with Spring Boot and RabbitMQ.
I set up a very simple Sender - Receiver application:
Sender:
public class Tut1Sender
{
private final Gson gson = new Gson();
#Autowired
private RabbitTemplate template;
#Autowired
private Queue queue;
public static int count = 1;
#Scheduled(fixedDelay = 1000, initialDelay = 500)
public void send() throws InterruptedException
{
String message = "Hello World! "+" Nr. "+count;
MessageObject mo = new MessageObject(message);
String toJson = gson.toJson(mo);
this.template.convertAndSend(queue.getName(), toJson);
System.out.println(" [x] Sent '" + toJson + "'");
Thread.sleep(5);
count++;
}
}
This part works just fine and fill my queue with messages.
Here is my receiver:
#RabbitListener(queues = "hello")
public class Tut1Receiver
{
private final Gson gson = new Gson();
#RabbitHandler
public void receive(String in) throws InterruptedException
{
System.out.println("Received Raw: " + in);
MessageObject fromJson = gson.fromJson(in, MessageObject.class);
System.out.println("Received Message '" + fromJson + "'");
int nextInt = ThreadLocalRandom.current().nextInt(1000, 5000);
System.out.println("Sleep for " + nextInt + " ms");
Thread.sleep(nextInt);
}
}
Messages created by the Sender are handled correctly by the receiver. I get a nice output, the message is acknowledged and deleted from the queue.
Then I put a message directly into the queue by the Web-GUI of RabbitMQ.
The sender grabs this message. I can say this because the message created by me switched from status "Ready" to "Unacked" (as displayed in Web-GUI)
The sender gave me no output.
Then I configured the ContainerFactory:
#Profile("receiver")
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(
SimpleRabbitListenerContainerFactoryConfigurer configurer,
ConnectionFactory connectionFactory)
{
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
configurer.configure(factory, connectionFactory);
factory.setErrorHandler(e -> {
System.out.println("Error: "+e);
System.out.println("Raw: "+((ListenerExecutionFailedException) e).getFailedMessage().toString());
});
return factory;
}
Now I am getting the following error (in an endless loop)
Error: org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener method 'no match' threw exception
Raw: (Body:'[B#53452feb(byte[11])' MessageProperties [headers={content_type=text/plain, content_encoding=UTF-8}, contentLength=0, receivedDeliveryMode=PERSISTENT, priority=0, redelivered=true, receivedExchange=, receivedRoutingKey=hello, deliveryTag=1, consumerTag=NOT_SET, consumerQueue=hello])
How can I handle this error? The sender should just display the error, acknowledging the message and proceed with the next message.
What is the right way to handle faulty messages in general?

For broken message, consumers can reject or deliver the message. If you are sure the broken message can't be processed by any other consumers, you should tell the broker to discard the message or deliver it to a dead-letter-exchange.
From official document of spring amqp, I find:
Another alternative is to set the container's rejectRequeued property to false. This causes all failed messages to be discarded. When using RabbitMQ 2.8.x or higher, this also facilitates delivering the message to a Dead Letter Exchange.
Or, you can throw a AmqpRejectAndDontRequeueException; this prevents message requeuing, regardless of the setting of the rejectRequeued property.

How to discard Tibco EMS messages from a queue (consumer side)

I am consuming Tibco JMS (EMS) messages from a queue... I want to clear the queue each time the application runs. I can think of the below logic... I thought their might be a better way
public void clearMessages() throws JMSException{
Message msg = (Message) queueReceiver.receiveNoWait();
while(msg != null)
{
clearMessages();
}
return;
}

Option 1: you acknowledge each message individually; this approach, however, may take some time, if you have (many) thousands of messages enqueued:
public void clearMessages() throws JMSException{
Message message = null;
do {
message = consumer.receiveNoWait();
if (message != null) message.acknowledge();
}
while (message != null);
}
Option 2: using the TibjmsAdmin Object purging a JMS destination is done like this (click TIBCO EMS Admin Java API for JavaDoc):
public void clearMessages(String queueName) throws TibjmsAdminException, TibjmsAdminInvalidNameException{
TibjmsAdmin jmsAdmin = new TibjmsAdmin("tcp://localhost:7222", "admin", "admin");
jmsAdmin.purgeQueue(queueName);
// alternatively purge all queues:
// jmsAdmin.purgeQueues(">");
}
HTH,
Hendrik

Getting the response time for JMS requests using MessageListener interface

I am using the following block of code to post JMS messages to a queue, and get the response messages in a response queue. (The following code runs for 100 messages in batch of 20 per thread, five threads running concurrently)
for(int i=0;i<=20;i++)
{
msg=myMessages.get(i); // myMessages is an array of TextMessages
qsender = qsession.createSender((Queue)msg.getJMSDestination());
qreceiver=qsession.createReceiver((Queue)msg.getJMSDestination());
tempq = qsession.createTemporaryQueue();
responseConsumer = qsession.createConsumer(tempq);
msg.setJMSReplyTo(tempq);
responseConsumer.setMessageListener(new Listener());
msg.setJMSCorrelationID(msg.getJMSCorrelationID()+i);
qsender.send(msg);
}
The Listener implementation:
public class Listener
implements MessageListener
{
public void onMessage(Message msg)
{
TextMessage tm = (TextMessage) msg;
// to calculate the response time
}
}
The requirement is to get the response time each message takes and store it. How do I go about it? Thinking of setting the time/date in the properties for the message and then use the Correlation id to calculate the time in Listener.
Is there another way to go about it?

You could have a Map<String, Long> that maps your CorrelationID to time sent and then look them up from the listener. The process that is sending the responses will have to put the correct CorrelationID on the response message for this to work.
For this example assume timemap is a Map<String, Long> and that it is in scope for both the sender and response listener (How you want to accomplish that is up to you).
Your loop body from above, modified:
msg=myMessages.get(i); // myMessages is an array of TextMessages
qsender = qsession.createSender((Queue)msg.getJMSDestination());
qreceiver=qsession.createReceiver((Queue)msg.getJMSDestination());
tempq = qsession.createTemporaryQueue();
responseConsumer = qsession.createConsumer(tempq);
msg.setJMSReplyTo(tempq);
responseConsumer.setMessageListener(new Listener());
msg.setJMSCorrelationID(msg.getJMSCorrelationID()+i);
/* MODIFICATIONS */
synchronzied(timemap){
timemap.put(msg.getJMSCorrelationID(), System.currentTimeMillis());
} /* END MODIFICATIONS */
qsender.send(msg);
Your listener, modified:
public void onMessage(Message msg)
{
TextMessage tm = (TextMessage) msg;
long now = System.currentTimeMillis();
long responseTime = 0;
synchronized(timemap){
Long sent = timemap.get(msg.getJMSCorrelationID());
if(sent != null){
/* Store this value, this is the response time in milliseconds */
responseTime = now - sent;
}else{
/* Error condition. */
}
}
}