We Keep Coding

Multiple queues receiving same message from virtual topic creates a deadletter entry for one queue only

I'm am using Virtual Destinations to implement Publish Subscribe model in ActiveMQ 5.15.13.
I have a virtual topic VirtualTopic and there are two queues bound to it. Each queue has its own redelivery policy. Let's say Queue 1 will retry message 2 times in case there is an exception while processing the message and Queue 2 will retry message 3 times. Post retry message will be sent to deadletter queue. I'm also using Individual Dead letter Queue strategy so that each queue has it's own deadletter queue.
I've observed that when a message is sent to VirtualTopic, the message with same message id is delivered to both the queues. I'm facing an issue where if the consumers of both queues are not able to process the message successfully. The message destined for Queue 1 is moved to deadletter queue after retrying for 2 times. But there is no deadletter queue for Queue 2, though message in Queue 2 is retried for 3 times.
Is it the expected behavior?
Code:
public class ActiveMQRedelivery {
private final ActiveMQConnectionFactory factory;
public ActiveMQRedelivery(String brokerUrl) {
factory = new ActiveMQConnectionFactory(brokerUrl);
factory.setUserName("admin");
factory.setPassword("password");
factory.setAlwaysSyncSend(false);
}
public void publish(String topicAddress, String message) {
final String topicName = "VirtualTopic." + topicAddress;
try {
final Connection producerConnection = factory.createConnection();
producerConnection.start();
final Session producerSession = producerConnection.createSession(false, AUTO_ACKNOWLEDGE);
final MessageProducer producer = producerSession.createProducer(null);
final TextMessage textMessage = producerSession.createTextMessage(message);
final Topic topic = producerSession.createTopic(topicName);
producer.send(topic, textMessage, PERSISTENT, DEFAULT_PRIORITY, DEFAULT_TIME_TO_LIVE);
} catch (JMSException e) {
throw new RuntimeException("Message could not be published", e);
}
}
public void initializeConsumer(String queueName, String topicAddress, int numOfRetry) throws JMSException {
factory.getRedeliveryPolicyMap().put(new ActiveMQQueue("*." + queueName + ".>"),
getRedeliveryPolicy(numOfRetry));
Connection connection = factory.createConnection();
connection.start();
final Session consumerSession = connection.createSession(false, CLIENT_ACKNOWLEDGE);
final Queue queue = consumerSession.createQueue("Consumer." + queueName +
".VirtualTopic." + topicAddress);
final MessageConsumer consumer = consumerSession.createConsumer(queue);
consumer.setMessageListener(message -> {
try {
System.out.println("in listener --- " + ((ActiveMQDestination)message.getJMSDestination()).getPhysicalName());
consumerSession.recover();
} catch (JMSException e) {
e.printStackTrace();
}
});
}
private RedeliveryPolicy getRedeliveryPolicy(int numOfRetry) {
final RedeliveryPolicy redeliveryPolicy = new RedeliveryPolicy();
redeliveryPolicy.setInitialRedeliveryDelay(0);
redeliveryPolicy.setMaximumRedeliveries(numOfRetry);
redeliveryPolicy.setMaximumRedeliveryDelay(-1);
redeliveryPolicy.setRedeliveryDelay(0);
return redeliveryPolicy;
}
}
Test:
public class ActiveMQRedeliveryTest {
private static final String brokerUrl = "tcp://0.0.0.0:61616";
private ActiveMQRedelivery activeMQRedelivery;
#Before
public void setUp() throws Exception {
activeMQRedelivery = new ActiveMQRedelivery(brokerUrl);
}
#Test
public void testMessageRedeliveries() throws Exception {
String topicAddress = "testTopic";
activeMQRedelivery.initializeConsumer("queue1", topicAddress, 2);
activeMQRedelivery.initializeConsumer("queue2", topicAddress, 3);
activeMQRedelivery.publish(topicAddress, "TestMessage");
Thread.sleep(3000);
}
#After
public void tearDown() throws Exception {
}
}

I recently came across this problem. To fix this there are 2 attributes that needs to be added to individualDeadLetterStrategy as below
<deadLetterStrategy>
<individualDeadLetterStrategy destinationPerDurableSubscriber="true" enableAudit="false" queuePrefix="DLQ." useQueueForQueueMessages="true"/>
</deadLetterStrategy>
Explanation of attributes:
destinationPerDurableSubscriber - To enable a separate destination per durable subscriber.
enableAudit - The dead letter strategy has a message audit that is enabled by default. This prevents duplicate messages from being added to the configured DLQ. When the attribute is enabled, the same message that isn't delivered for multiple subscribers to a topic will only be placed on one of the subscriber DLQs when the destinationPerDurableSubscriber attribute is set to true i.e. say two consumers fail to acknowledge the same message for the topic, that message will only be placed on the DLQ for one consumer and not the other.

Spring message TTL does not work

I am using spring-amqp 1.5.2 and would like to publish message which can reside in the queue for at most 10 seconds:
I am setting the expiration to 10000 (10 seconds) for all my messages but the messages are still in the queue after 10 seconds. Here is my code snippet:
MessageProperties props = new MessageProperties();
props.setExpiration("10000");
Message message = new Message(event.toByteArray(), props);
this.rabbitTemplate.convertAndSend("my-exchange", "my-routing-key", message);

Use send() instead of convertAndSend().
Conversion is for sending some object that needs to be converted to a Message; you already have a Message.
EDIT
However, this was fixed in 1.1.0 JIRA so that should not be the problem.
This works fine for me...
#SpringBootApplication
public class So45824146Application {
public static void main(String[] args) {
SpringApplication.run(So45824146Application.class, args);
}
#Bean
public ApplicationRunner runner(RabbitTemplate template) {
return args -> {
MessageProperties props = new MessageProperties();
props.setExpiration("5000");
Message message = new Message("foo".getBytes(), props);
template.convertAndSend("foo", message);
};
}
}

How to start ActiveMQ when tomcat starts?

How do I configure my J2EE application so that I can invoke ActiveMQ service along with tomcat server? I am aware about embedded broker, here asking how to start the ActiveMQ whenever I start tomcat
Current Code (works fine) :
Now I want to remove main() method and use the code to run when tomcat runs.
public class JMSService {
public void produceJMS() throws NamingException, JMSException {
ConnectionFactory connFactory = new ActiveMQConnectionFactory(ActiveMQConnection.DEFAULT_BROKER_URL);
Connection conn = connFactory.createConnection();
conn.start();
Session session = conn.createSession(false,Session.AUTO_ACKNOWLEDGE);
Destination destination = session.createQueue("testQueue");
MessageProducer producer = session.createProducer(destination);
producer.setDeliveryMode(DeliveryMode.PERSISTENT);
TextMessage message = session.createTextMessage("Test Message ");
// send the message
producer.send(message);
System.out.println("sent: " + message);
}}
Here is my consumer :
public class JMSReceiver implements MessageListener,ExceptionListener {
public static void main(String args[]) throws Exception {
JMSReceiver re = new JMSReceiver();
re.receiveJMS();
}
public void receiveJMS() throws NamingException, JMSException {
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory(ActiveMQConnection.DEFAULT_BROKER_URL);
Connection connection = connectionFactory.createConnection();
connection.start();
Session session = connection.createSession(false,Session.AUTO_ACKNOWLEDGE);
// Getting the queue 'testQueue'
Destination destination = session.createQueue("testQueue");
MessageConsumer consumer = session.createConsumer(destination);
// set an asynchronous message listener
JMSReceiver asyncReceiver = new JMSReceiver();
consumer.setMessageListener(asyncReceiver);
connection.setExceptionListener(asyncReceiver);
}
#Override
public void onMessage(Message message) {
System.out.println("Received message : " +message);
}
}

What #Tim Bish said is correct. You either need to have a timer say for example receiver should listen for 1 hour- or make it available until program terminate. Either case you need to start your consumer program once:
Change your receiveJMS method as follows:
public void receiveJMS() throws NamingException, JMSException {
try{
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory(ActiveMQConnection.DEFAULT_BROKER_URL);
Connection connection = connectionFactory.createConnection();
connection.start(); // it's the start point
Session session = connection.createSession(false,Session.AUTO_ACKNOWLEDGE);
// Getting the queue 'testQueue'
Destination destination = session.createQueue("testQueue");
MessageConsumer consumer = session.createConsumer(destination);
// set an asynchronous message listener
// JMSReceiver asyncReceiver = new JMSReceiver();
//no need to create another object
consumer.setMessageListener(this);
connection.setExceptionListener(this);
// connection.close(); once this is closed consumer no longer active
Thread.sleep(60 *60 * 1000); // receive messages for 1 hour
}finally{
connection.close();// after 1 hour close it
}
}
The above program will listen upto 1 hour. If you want it as long as the program run, remove the finally block. But the recommended way is to close it somehow. since your application seems to be standalone ,you can check the java runtime shutdown hook, where you can specify how to release such resources while program terminates.
If your consumer is a web application you can close it in a ServletContextlistner.

You aren't giving the consumer application any time to actually receive a message, you create it, then you close it. You either need to use a timed receive call to do an sync receive of the message from the Queue or you need to add some sort of wait in the main method such as a CountDownLatch etc to allow the async onMessage call to trigger shutdown once processing of the message is complete.

Restart embedded broker in unit test : VMTransportServer already bound

I'm trying to write a test that simulates a "broker down" phase.
Therefore I want to
start a local broker
send message1
stop the broker
send message2 (which will of course not arrive)
start the broker again
send message3
According to http://activemq.apache.org/how-do-i-restart-embedded-broker.html it is recommended to init a new BrokerService to start the broker again.
So the code looks (almost) like this:
private BrokerService _broker;
private void startBroker() throws Exception {
_broker = new BrokerService();
_broker.addConnector("vm://localhost?broker.persistent=false");
_broker.start();
_broker.waitUntilStarted();
}
private void stopBroker() throws Exception {
_broker.stop();
_broker.waitUntilStopped();
}
#Test
public void publishMessagesWithServerBreakdownInBetween()
throws Exception
{
startBroker();
... send and receive message (works fine)
stopBroker();
... send message (fails of course)
startBroker(); // this fails with java.io.IOException: VMTransportServer already bound at: vm://localhost?broker.persistent=false
... send and receive message
}
The problem is already mentioned as comment in code:
The restart of the broker fails due to the error : java.io.IOException: VMTransportServer already bound at: vm://localhost?broker.persistent=false
I found a similar problem at ActiveMQ forum (http://activemq.2283324.n4.nabble.com/VMTransportServer-already-bound-td2364603.html), but in my case the hostname isn't null.
Another idea was to set 2 different broker names, but that also didn't help.
What am I doing wrong?

You want to control what the VM Transport does by telling it not to try and create a broker for you since you are adding it to an already created broker. The rest is pretty simply then:
public class AMQRestartTest {
private BrokerService broker;
private String connectorURI;
private ActiveMQConnectionFactory factory;
#Before
public void startBroker() throws Exception {
createBroker(true);
factory = new ActiveMQConnectionFactory("failover://" + connectorURI);
}
private void createBroker(boolean deleteAllMessages) throws Exception {
broker = new BrokerService();
TransportConnector connector = broker.addConnector("vm://localhost?create=false");
broker.setPersistent(false);
broker.start();
broker.waitUntilStarted();
connectorURI = connector.getConnectUri().toString();
}
#Test(timeout = 60_000)
public void test() throws Exception {
Connection connection = factory.createConnection();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Queue queue = session.createQueue("test");
MessageConsumer consumer = session.createConsumer(queue);
MessageProducer producer = session.createProducer(queue);
connection.start();
broker.stop();
broker.waitUntilStopped();
createBroker(false);
producer.send(session.createTextMessage("help!"));
Message received = consumer.receive();
assertNotNull(received);
assertTrue(received instanceof TextMessage);
}
}

Slow HornetQ Producer when Queue is persistent

I have tried with Persistent Queue in horntQ. I have made two separate examples (Producer, Consumer). My consumer is working well but the Producer is taking too much time to finish sending message. I have run both separately as well as together. What could be the problem?
my code is:
public class HornetProducer implements Runnable{
Context ic = null;
ConnectionFactory cf = null;
Connection connection = null;
Queue queue = null;
Session session = null;
MessageProducer publisher = null;
TextMessage message = null;
int messageSent=0;
public synchronized static Context getInitialContext()throws javax.naming.NamingException {
Properties p = new Properties( );
p.put(Context.INITIAL_CONTEXT_FACTORY,"org.jnp.interfaces.NamingContextFactory");
p.put(Context.URL_PKG_PREFIXES," org.jboss.naming:org.jnp.interfaces");
p.put(Context.PROVIDER_URL, "jnp://localhosts:1099");
return new javax.naming.InitialContext(p);
}
public HornetProducer()throws Exception{
ic = getInitialContext();
cf = (ConnectionFactory)ic.lookup("/ConnectionFactory");
queue = (Queue)ic.lookup("queue/testQueue2");
connection = cf.createConnection();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
publisher = session.createProducer(queue);
connection.start();
}
public void publish(){
try{
message = session.createTextMessage("Hello!");
System.out.println("StartDate: "+new Date());
for(int i=0;i<10000;i++){
messageSent++;
publisher.send(message);
}
System.out.println("EndDate: "+new Date());
}catch(Exception e){
System.out.println("Exception in Consume: "+ e.getMessage());
}
}
public void run(){
publish();
}
public static void main(String[] args) throws Exception{
new HornetProducer().publish();
}
}

You are sending these messages persistently, and non transactionally. What means, each message sent has to be completed individually.
That means for each message you send, you have to make a network round trip to the server, and wait it finish persistency before you can send another message.
If you had multiple producers on this situation, hornetq would batch both producers and you would save a lot of time. (i.e. the server will batch many write requests).
If you want to speed up the sending of a single producer, you should use transactions probably.
for example:
I - Change your session to transactioned:
session = connection.createSession(true, Session.SESSION_TRANSACTIONED);
II - commit every N messages:
for(int i=0;i<10000;i++){
messageSent++;
publisher.send(message);
if (messageSent % 1000 == 0) session.commit();
}
session.commit();
You could also disable sync on Persistent messages. (Sending them asynchronously).