I am using Rabbit MQ to replicate what Jenkins does.
The only issue I am facing is, lets say, when 10 messages are in queue. And there are some duplicate messages which are in unacknowledged state.
And I need to delete those messages from queue, how do I achieve this?
My rabbitmq configuration is as follows, where each queue only has one consumer. So if I have 10 messages, all will get processed through same consumer's thread.
Queue queue = new Queue(sfdcConnectionDetails.getGitRepoId() + "_" + sfdcConnectionDetails.getBranchConnectedTo(), true);
rabbitMqSenderConfig.amqpAdmin().declareQueue(queue);
rabbitMqSenderConfig.amqpAdmin().declareBinding(BindingBuilder.bind(queue).to(new DirectExchange(byRepositoryRepositoryId.getRepository().getRepositoryId())).withQueueName());
RabbitMqConsumer container = new RabbitMqConsumer();
container.setConnectionFactory(rabbitMqSenderConfig.connectionFactory());
container.setQueueNames(queue.getName());
container.setConcurrentConsumers(1);
container.setMessageListener(new MessageListenerAdapter(new ConsumerHandler(****, ***), new Jackson2JsonMessageConverter()));
container.startConsumers();
You can use any plugin (e.g this) for deduplicating messages on the rabbit side.
Use cache on your consumer for detecting if the same message was processing recently.
As already suggested by #ekiryuhin, One of the approach you could take is assign a request_id tag it to the payload before producing message to RabbitMQ & on your consumer's end cache the request_id. Look out for the request_id if already present ignore payload and delete it.
This request_id might work as deduplication-id for your payloads.
Related
We have a SpringBoot service implementation in which we are using delayed messaging with the below setup:
Initial queue (Queue 1) that gets the message has a TTL set, the queue also has a dead letter exchange mentioned with a specific dead letter routing key.
Another queue (Queue 2) is bound to the DLX of the previous queue with the routing key which is set as the dead letter routing key
A consumer listens to the messages on Queue 2.
The delayed messaging seems to work as expected but I am seeing an issue with messages getting redelivered in certain scenarios.
If I have a debug point in my consumer and keep the message just after reading it for some time then once the current message has been processed consumer gets another message which has the below properties:
Redelivered property as true.
Property deliveryAttempt as 1
Only the first message has an x-death header and redelivered messages do not seem to have it.
The attempt to deliver the message is done 3 times as many times as I pause the consumer using the debug point each time after reading each redelivered message.
My understanding was that the acknowledgment mode by default is AUTO so once the consumer has read the message then it would not be redelivered?
I have tried using maxAttempts=1 property but does not seem to help.
I am using the spring cloud stream to create the consumers and the queues.
I used to run into this issue when the message processing in the consumer failed (exception thrown). In this case, if you have DLQ configured, make sure to add the following configuration as well so the failed message will be routed to the DLQ not the original listening queue.
"
rabbit:
autoBindDlq: true
"
Otherwise if you don't set up the DLQ, configure "autoBindDlq" to "false".
TLDR; In the context of a topic exchange and queues created on the fly by the consumers, how to have a message redelivered / the producer notified when no consumer consumes the message?
I have the following components:
a main service, producing files. Each file has a certain category (e.g. pictures.profile, pictures.gallery)
a set of workers, consuming files and producing a textual output from them (e.g. the size of the file)
I currently have a single RabbitMQ topic exchange.
The producer sends messages to the exchange with routing_key = file_category.
Each consumer creates a queue and binds the exchange to this queue for a set of routing keys (e.g. pictures.* and videos.trending).
When a consumer has processed a file, it pushes the result in a processing_results queue.
Now - this works properly, but it still has a major issue. Currently, if the publisher sends a message with a routing key that no consumer is bound to, the message will be lost. This is because even if the queue created by the consumers is durable, it is destroyed as soon as the consumer disconnects since it is unique to this consumer.
Consumer code (python):
channel.exchange_declare(exchange=exchange_name, type='topic', durable=True)
result = channel.queue_declare(exclusive = True, durable=True)
queue_name = result.method.queue
topics = [ "pictures.*", "videos.trending" ]
for topic in topics:
channel.queue_bind(exchange=exchange_name, queue=queue_name, routing_key=topic)
channel.basic_consume(my_handler, queue=queue_name)
channel.start_consuming()
Loosing a message in this condition is not acceptable in my use case.
Attempted solution
However, "loosing" a message becomes acceptable if the producer is notified that no consumer received the message (in this case it can just resend it later). I figured out the mandatory field could help, since the specification of AMQP states:
This flag tells the server how to react if the message cannot be routed to a queue. If this flag is set, the server will return an unroutable message with a Return method.
This is indeed working - in the producer, I am able to register a ReturnListener :
rabbitMq.confirmSelect();
rabbitMq.addReturnListener( (int replyCode, String replyText, String exchange, String routingKey, AMQP.BasicProperties properties, byte[] body) -> {
log.info("A message was returned by the broker");
});
rabbitMq.basicPublish(exchangeName, "pictures.profile", true /* mandatory */, MessageProperties.PERSISTENT_TEXT_PLAIN, messageBytes);
This will as expected print A message was returned by the broker if a message is sent with a routing key no consumer is bound to.
Now, I also want to know when the message was correctly received by a consumer. So I tried registering a ConfirmListener as well:
rabbitMq.addConfirmListener(new ConfirmListener() {
void handleAck(long deliveryTag, boolean multiple) throws IOException {
log.info("ACK message {}, multiple = ", deliveryTag, multiple);
}
void handleNack(long deliveryTag, boolean multiple) throws IOException {
log.info("NACK message {}, multiple = ", deliveryTag, multiple);
}
});
The issue here is that the ACK is sent by the broker, not by the consumer itself. So when the producer sends a message with a routing key K:
If a consumer is bound to this routing key, the broker just sends an ACK
Otherwise, the broker sends a basic.return followed by a ACK
Cf the docs:
For unroutable messages, the broker will issue a confirm once the exchange verifies a message won't route to any queue (returns an empty list of queues). If the message is also published as mandatory, the basic.return is sent to the client before basic.ack. The same is true for negative acknowledgements (basic.nack).
So while my problem is theoretically solvable using this, it would make the logic of knowing if a message was correctly consumed very complicated (especially in the context of multi threading, persistence in a database, etc.):
send a message
on receive ACK:
if no basic.return was received for this message
the message was correctly consumed
else
the message wasn't correctly consumed
on receive basic.return
the message wasn't correctly consumed
Possible other solutions
Have a queue for each file category, i.e. the queues pictures_profile, pictures_gallery, etc. Not good since it removes a lot of flexibility for the consumers
Have a "response timeout" logic in the producer. The producer sends a message. It expects an "answer" for this message in the processing_results queue. A solution would be to resend the message if it hasn't been answered to after X seconds. I don't like it though, it would create some additional tricky logic in the producer.
Produce the messages with a TTL of 0, and have the producer listen on a dead-letter exchange. This is the official suggested solution to replace the 'immediate' flag removed in RabbitMQ 3.0 (see paragraph Removal of "immediate" flag). According to the docs of the dead letter exchanges, a dead letter exchange can only be configured per-queue. So it wouldn't work here
[edit] A last solution I see is to have every consumer create a durable queue that isn't destroyed when he disconnects, and have it listen on it. Example: consumer1 creates queue-consumer-1 that is bound to the message of myExchange having a routing key abcd. The issue I foresee is that it implies to find an unique identifier for every consumer application instance (e.g. hostname of the machine it runs on).
I would love to have some inputs on that - thanks!
Related to:
RabbitMQ: persistent message with Topic exchange (not applicable here since queues are created "on the fly")
Make sure the broker holds messages until at least one consumer gets it
RabbitMQ Topic Exchange with persisted queue
[edit] Solution
I ended up implementing something that uses a basic.return, as mentioned earlier. It is actually not so tricky to implement, you just have to make sure that your method producing the messages and the method handling the basic returns are synchronized (or have a shared lock if not in the same class), otherwise you can end up with interleaved execution flows that will mess up your business logic.
I believe that an alternate exchange would be the best fit for your use case for the part regarding the identification of not routed messages.
Whenever an exchange with a configured AE cannot route a message to any queue, it publishes the message to the specified AE instead.
Basically upon creation of the "main" exchange, you configure an alternate exchange for it.
For the referenced alternate exchange, I tend to go with a fanout, then create a queue (notroutedq) binded to it.
This means any message that is not published to at least one of the queues bound to your "main" exchange will end up in the notroutedq
Now regarding your statement:
because even if the queue created by the consumers is durable, it is destroyed as soon as the consumer disconnects since it is unique to this consumer.
Seems that you have configured your queues with auto-delete set to true.
If so, in case of disconnect, as you stated, the queue is destroyed and the messages still present on the queue are lost, case not covered by the alternate exchange configuration.
It's not clear from your use case description whether you'd expect in some cases for a message to end up in more than one queue, seemed more a case of one queue per type of processing expected (while keeping the grouping flexible). If indeed the queue split is related to type of processing, I do not see the benefit of setting the queue with auto-delete, expect maybe not having to do any cleanup maintenance when you want to change the bindings.
Assuming you can go with durable queues, then a dead letter exchange (would again go with fanout) with a binding to a dlq would cover the missing cases.
not routed covered by alternate exchange
correct processing already handled by your processing_result queue
problematic processing or too long to be processed covered by the dead letter exchange, in which case the additional headers added upon dead lettering the message can even help to identify the type of actions to take
I have several Java Clients sending messages to a queue using RabbitMQ default exchange.
I want to be able to analyse the messages inside of the queue (if they are not consumed obviously), identify the sender of the message and if a Client is sending too many messages then bind a special queue so that this client sends messages to that queue .
I am new with RabbitMQ so What I would like to know is :
Is it possible to append a client ID as a header in RabbitMQ?
How to create a special queue for clients with much workload?
Is it possible to analyse messages residing in a queue?
Any help would be much appreciated.
Is it possible to append a client ID as a header in RabbitMQ?
you can use the headers field on BasicProperties
Map<String, Object> headers = new HashMap<String, Object>();
headers.put("myclientid", 22222);
channel.basicPublish(exchangeName, routingKey,
new AMQP.BasicProperties.Builder()
.headers(headers)
.build()),
messageBodyBytes);
How to create a special queue for clients with much workload?
There are no "special" queues, you can add more consumers to the same queue, in case you have an high workload. See
https://www.rabbitmq.com/tutorials/tutorial-two-java.html
Is it possible to analyse messages residing in a queue?
You can use the API basic_get without ACK the message, in this way you can read a messages without remove them from the queue
I have a Java client which monitors RabbitMQ queue. I am able to get the count of messages currently in queue with this code
#Resource
RabbitAdmin rabbitAdmin;
..........
DeclareOk declareOk = rabbitAdmin.getRabbitTemplate().execute(new ChannelCallback<DeclareOk>() {
public DeclareOk doInRabbit(Channel channel) throws Exception {
return channel.queueDeclarePassive("test.pending");
}
});
return declareOk.getMessageCount();
I want to get some more additional details like -
Message body of currently enqueued items.
Total number of messages that was enqueued in the queue since the queue was created.
Is there any way to retrieve these data in Java client?
With AMQP protocol (including RabbitMQ implementation) you can't get such info with 100% guarantee.
The closest number to messages count is messages count returned with queue.declare-ok (AMQP.Queue.DeclareOk in java AMQP client library).
Whilst messages count you receive with queue.declare-ok may match exact messages number enqueues, you can't rely on it as it doesn't count messages which waiting acknowledges or published to queue during transaction but not committed yet.
It really depends what kind of precission do you need.
As to enqueued messages body, you may want to manually extract all messages in queue, view their body and put them back to queue. This is the only way to do what you want.
You can get some information about messages count with Management Plugin, RabbitMQ Management HTTP API and rabbitmqctl util (see list_queues, list_channels).
You can't get total published messages count since queue was created and I think nobody implement such stats while it useless (FYI, with messages flow in average 10k per second you will not even reach uint64 in a few thousand years).
AMQP.Queue.DeclareOk dok = channel.queueDeclare(QUEUE_NAME, true, false, false, queueArgs);
dok.getMessageCount();
To access queue details via http api,
http://public-domain-name:15672/api/queues/%2f/queue_name
To access queue details via command from localhost cli promt,
curl -i -u guest_uname:guest_password http://localhost:15672/api/queues/%2f/queue_name
Where,
%2f is default vhost "/"
I have a publisher that is pushing messages to a topic. I have multiple subscribers each doing a different task once they consume the message from the topic.
Now I want my system to scale to multiple instances of the same process running on different hosts/same host. e.g. I want to run multiple copies of my application A on different hosts so that if one instance of A is slow, then the other instances can pull in subsequent messages and make forward progress..
I found out that this is possible using virtual destinations. I followed the steps here -
http://activemq.apache.org/virtual-destinations.html
But how do i setup my multiple subscribers to the same topic with the same client id? when i try to do that, i get errors. when i try some other way, it doesn't work. can someone help?
Normally, I start a subscriber by doing the below steps -
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(ActiveMQConnection.DEFAULT_USER, ActiveMQConnection.DEFAULT_PASSWORD, ActiveMQConnection.DEFAULT_BROKER_URL;);
activeMQConnection = connectionFactory.createConnection();
activeMQConnection.setClientID("subscriber1");
activeMQConnection.setExceptionListener(exceptionListener);
activeMQSession = activeMQConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE);
activeMQTopic = activeMQSession.createTopic("myTopic");
activeConsumer = activeMQSession.createDurableSubscriber(activeMQTopic, "myTopic");
activeConsumer.setMessageListener(messageListener);
activeMQConnection.start();
when i try to create a 2nd subscriber and pass the topic name as "VirtualTopic.myTopic", nothing happens.
thanks
The virtual topics feature is very simple and quite powerful once you understand it.
When using virtual topics - there is no need for durable consumers. That is because for each client you will get an instance of regular queue created. If you have 5 clients (application A, B, C, D, E) you will get 5 queues created and populated with the copy of the messages every time message is sent to the virtual topic.
Actually it is a limitation of durable consumer - that only ONE connection is allowed per clientId. Being a regular queue, you can create as many consumers as you like and queue will guarantee that 1 message will be received only by 1 consumer. So if you have application A that takes 1 minute to process a message, you can create 5 instances of it listening to the same queue. When you will post 5 messages within 1 second, each of your application will receive its own message to process.
There are not well documented requirements which are not intuitive. To make virtual topic work you need
Use VirtualTopic. in your topic name, for example VirtualTopic.Orders (this prefix can be configured)
Use Consumer. in the name of the queue you. Like Consumer.ApplicationA.VirtualTopic.Orders where ApplicationA is actually your client id
Use regular subscribers not durable ones for the queue above.
Example:
string activeMqConsumerTopic = "Consumer.AmqTestConsumer.VirtualTopic.Orders";
IQueue queue = SessionUtil.GetQueue(session, activeMqConsumerTopic);
IMessageConsumer consumer = session.CreateConsumer(queue);
Queue is created for automatically whenever the first instance of consumer is subscribed to it. Since that moment all messages that are sent to topic are duplicated/copied into all associated queues.
Hope this helps.
Virtual Topics is the answer for you. However you have to define a naming standard for all virtual topic queues. Here is the answer for this:
Virtual Topics helps with following prospective:
1. Load Balancing of messages
2. Fast Failover of Subscriber
3. Re-using same connection Factory for different Producers and Consumers. (Durable Subscribers needs a Unique JMS Client Id and same cannot be reused for any other Producer or consumer)
here is the way to do it, below example creates prefix VTCON.*. So every queue with this prefix and Topic Name at the end will consumer the message.
<virtualDestinations>
<virtualTopic name="TEST.TP01" prefix="VTCON.*." selectorAware="false"/>
</virtualDestinations>
http://workingwithqueues.blogspot.com/2012/05/activemq-virtual-topics-or-virtual.html