Is there a best practice or guidance for sending persistent messages with asyncSend set to true.
We don't have transaction manager configured
We have ~40k-50k messages which are sent using jmsTemplate configured with
org.apache.activemq.pool.PooledConnectionFactory
We have a for loop which iterates over messages list and send them using
jmsTemplate.convertAndSend(destination, msg)
We see lot of message loss on frequent basis, when we turn off asyncSend we get the reliability but the producer performance drops by 95%
A bit of speculation as the question is not very detailed but anyway.
Depending on configuration, ActiveMQ might have memory limits on queues (might as well differ between persistent and non persistent messages). So when memory is up, your asyncSend calls will ignore warnings and continue to deliver messages to the "black hole" until memory is freed by the consumer.
There is no silver bullet to allow max performance and max reliability. Unfortunately.
However, I would try setting a producerWindowSize on the connection factory to allow some specified amount of data before a broker ack is received. Exact value is something you need to try out and depends on scenario as well as broker config/resources.
I solved this using a ProducerCallback
List<String> messageTexts = prepareListOfMessaeTexts();
ProducerCallback producerCallback = (session, producer) -> {
Topic destination = session.createTopic(myTopicName);
for (String messageText : myMessagmessageTextseBodies) {
producer.send(destination, session.createTextMessage(messageText));
}
return null;
};
jmsTemplate.execute(producerCallback);
Related
TLDR; In the context of a topic exchange and queues created on the fly by the consumers, how to have a message redelivered / the producer notified when no consumer consumes the message?
I have the following components:
a main service, producing files. Each file has a certain category (e.g. pictures.profile, pictures.gallery)
a set of workers, consuming files and producing a textual output from them (e.g. the size of the file)
I currently have a single RabbitMQ topic exchange.
The producer sends messages to the exchange with routing_key = file_category.
Each consumer creates a queue and binds the exchange to this queue for a set of routing keys (e.g. pictures.* and videos.trending).
When a consumer has processed a file, it pushes the result in a processing_results queue.
Now - this works properly, but it still has a major issue. Currently, if the publisher sends a message with a routing key that no consumer is bound to, the message will be lost. This is because even if the queue created by the consumers is durable, it is destroyed as soon as the consumer disconnects since it is unique to this consumer.
Consumer code (python):
channel.exchange_declare(exchange=exchange_name, type='topic', durable=True)
result = channel.queue_declare(exclusive = True, durable=True)
queue_name = result.method.queue
topics = [ "pictures.*", "videos.trending" ]
for topic in topics:
channel.queue_bind(exchange=exchange_name, queue=queue_name, routing_key=topic)
channel.basic_consume(my_handler, queue=queue_name)
channel.start_consuming()
Loosing a message in this condition is not acceptable in my use case.
Attempted solution
However, "loosing" a message becomes acceptable if the producer is notified that no consumer received the message (in this case it can just resend it later). I figured out the mandatory field could help, since the specification of AMQP states:
This flag tells the server how to react if the message cannot be routed to a queue. If this flag is set, the server will return an unroutable message with a Return method.
This is indeed working - in the producer, I am able to register a ReturnListener :
rabbitMq.confirmSelect();
rabbitMq.addReturnListener( (int replyCode, String replyText, String exchange, String routingKey, AMQP.BasicProperties properties, byte[] body) -> {
log.info("A message was returned by the broker");
});
rabbitMq.basicPublish(exchangeName, "pictures.profile", true /* mandatory */, MessageProperties.PERSISTENT_TEXT_PLAIN, messageBytes);
This will as expected print A message was returned by the broker if a message is sent with a routing key no consumer is bound to.
Now, I also want to know when the message was correctly received by a consumer. So I tried registering a ConfirmListener as well:
rabbitMq.addConfirmListener(new ConfirmListener() {
void handleAck(long deliveryTag, boolean multiple) throws IOException {
log.info("ACK message {}, multiple = ", deliveryTag, multiple);
}
void handleNack(long deliveryTag, boolean multiple) throws IOException {
log.info("NACK message {}, multiple = ", deliveryTag, multiple);
}
});
The issue here is that the ACK is sent by the broker, not by the consumer itself. So when the producer sends a message with a routing key K:
If a consumer is bound to this routing key, the broker just sends an ACK
Otherwise, the broker sends a basic.return followed by a ACK
Cf the docs:
For unroutable messages, the broker will issue a confirm once the exchange verifies a message won't route to any queue (returns an empty list of queues). If the message is also published as mandatory, the basic.return is sent to the client before basic.ack. The same is true for negative acknowledgements (basic.nack).
So while my problem is theoretically solvable using this, it would make the logic of knowing if a message was correctly consumed very complicated (especially in the context of multi threading, persistence in a database, etc.):
send a message
on receive ACK:
if no basic.return was received for this message
the message was correctly consumed
else
the message wasn't correctly consumed
on receive basic.return
the message wasn't correctly consumed
Possible other solutions
Have a queue for each file category, i.e. the queues pictures_profile, pictures_gallery, etc. Not good since it removes a lot of flexibility for the consumers
Have a "response timeout" logic in the producer. The producer sends a message. It expects an "answer" for this message in the processing_results queue. A solution would be to resend the message if it hasn't been answered to after X seconds. I don't like it though, it would create some additional tricky logic in the producer.
Produce the messages with a TTL of 0, and have the producer listen on a dead-letter exchange. This is the official suggested solution to replace the 'immediate' flag removed in RabbitMQ 3.0 (see paragraph Removal of "immediate" flag). According to the docs of the dead letter exchanges, a dead letter exchange can only be configured per-queue. So it wouldn't work here
[edit] A last solution I see is to have every consumer create a durable queue that isn't destroyed when he disconnects, and have it listen on it. Example: consumer1 creates queue-consumer-1 that is bound to the message of myExchange having a routing key abcd. The issue I foresee is that it implies to find an unique identifier for every consumer application instance (e.g. hostname of the machine it runs on).
I would love to have some inputs on that - thanks!
Related to:
RabbitMQ: persistent message with Topic exchange (not applicable here since queues are created "on the fly")
Make sure the broker holds messages until at least one consumer gets it
RabbitMQ Topic Exchange with persisted queue
[edit] Solution
I ended up implementing something that uses a basic.return, as mentioned earlier. It is actually not so tricky to implement, you just have to make sure that your method producing the messages and the method handling the basic returns are synchronized (or have a shared lock if not in the same class), otherwise you can end up with interleaved execution flows that will mess up your business logic.
I believe that an alternate exchange would be the best fit for your use case for the part regarding the identification of not routed messages.
Whenever an exchange with a configured AE cannot route a message to any queue, it publishes the message to the specified AE instead.
Basically upon creation of the "main" exchange, you configure an alternate exchange for it.
For the referenced alternate exchange, I tend to go with a fanout, then create a queue (notroutedq) binded to it.
This means any message that is not published to at least one of the queues bound to your "main" exchange will end up in the notroutedq
Now regarding your statement:
because even if the queue created by the consumers is durable, it is destroyed as soon as the consumer disconnects since it is unique to this consumer.
Seems that you have configured your queues with auto-delete set to true.
If so, in case of disconnect, as you stated, the queue is destroyed and the messages still present on the queue are lost, case not covered by the alternate exchange configuration.
It's not clear from your use case description whether you'd expect in some cases for a message to end up in more than one queue, seemed more a case of one queue per type of processing expected (while keeping the grouping flexible). If indeed the queue split is related to type of processing, I do not see the benefit of setting the queue with auto-delete, expect maybe not having to do any cleanup maintenance when you want to change the bindings.
Assuming you can go with durable queues, then a dead letter exchange (would again go with fanout) with a binding to a dlq would cover the missing cases.
not routed covered by alternate exchange
correct processing already handled by your processing_result queue
problematic processing or too long to be processed covered by the dead letter exchange, in which case the additional headers added upon dead lettering the message can even help to identify the type of actions to take
We have an application that synchronously reads 4MB messages from an IBM Websphere queue (version 7.5) using JMS. Under certain circumstances, I want to discard messages from the queue without reading them. I am trying to figure out if there is a way to do this programatically without reading the entire 4MB message, which takes several seconds (there could be hundreds of messages that need to be discarded). In the absense of a discard() method (or similar), here is what I have tried:
BytesMessage msg = (BytesMessage)queueReceiver.receiveNoWait();
bytesRead = msg.readBytes(msgBytes, 1024); // just read 1024 bytes
queueReceiver.close();
The above code is no quicker than retrieving the entire 4MB message from the queue (by reading into a larger buffer). This leads me to believe that the receiveNoWait() call is downloading the entire message into an internal buffer before the readBytes() call is made. The only other information I can provide is that the queue is set to "auto acknowledge" when the session is started:
queueSession = queueConnection.createQueueSession(false, QueueSession.AUTO_ACKNOWLEDGE);
If I were to change this to CLIENT_ACKNOWLEDGE and acknowlege the message using msg.acknowledge(), would that have the desired effect? Or is there something I am missing?
Thanks for any help,
Doug
There is no other way, a message must be consumed to take remove it from a queue.
Changing to CLIENT_ACKNOWLEDGE from AUTO_ACKNOWLEDGE will not make any difference as the acknowledge is way to tell the messaging provider to remove a message from queue. The AUTO_ACKNOWLEDGE option tells the JMS client to automatically send a confirmation to provider to remove a message whereas CLIENT_ACKNOWLEDGE is used by the application to explicitly tell the provider to remove message(s).
You could probably take a look at setting an expiry time on messages that you don't plan to consume. Messages with an expiry time set, will not be available for delivery after the expiry time is over. Read through JMSExpiration property of a message.
Given this some more thought and there is potentially one other way here; MQ has the concept of PCF messages - simply this is being able to send an administrative command as a message to a queue manager.
JMS can send these messages - so one open would be to send a CLEAR_QUEUE command when you know you don't want any more messages.
It's quite a broad approach - clearing the entire queue but it depends on what your criteria are for removing messages.
I can see the use case however for selectively removing messages - maybe worth raising a 'request for enhancement' RFE on the IBM developerWorks site?
As far as I know, JMS cannot read part of a message. You can only do that with C or Java (non-JMS).
MQGetMessageOptions gmo = new MQGetMessageOptions();
gmo.options = MQC.MQGMO_NO_WAIT + MQC.MQGMO_FAIL_IF_QUIESCING;
MQMessage getMsg = new MQMessage();
try
{
/* get the message with only 1 byte of message data */
_inQ.get(getMsg, gmo, 1);
}
catch (MQException e)
{
System.err.println(e.getLocalizedMessage() );
}
Our ActiveMQ consuming process runs out of memory and dies.
We have an ActiveMQ topic with one sender and two receivers. Superficially, all works fine---messages are sent and picked up by both receivers, but eventually we exhaust all memory. Heap dump shows 1.362 million instances each of LinkedList$Node, AtomicReference, ActiveMQObjectMessage, MessageId, and MessageDispatch. Meanwhile the client side message queue is empty or nearly empty throughout. I think the 1.362M may be on a list that tracks unacknowledged messages pending ack. The topic is specified to have AUTO_ACKNOWLEDGE set, so we're trying to ack, but possibly failing. (jmsSession = jmsConnection.createSession(false, Session.AUTO_ACKNOWLEDGE);)
The heap dump shows garbage that appears to be associated with the client's incoming message buffering exists in modest numbers (a few thousand). This seems consistent with the numbers of these objects that accumulate in a toy program that we set up to send and consume similar objects. They accumulate for a while, then get GC'd and the memory never grows significantly in the toy or in the failing program.
Is it correct to surmise that the five object types are associated with ACKS, and if so, what can cause the objects to remain on this structure despite being apparently fully consumed by both consumers? Is there some way we could be canceling the AUTO_ACKNOWLEDGE that we think we have set? BTW, one consumer is synchronous, using receive() and the other is asynchronous, using onMessage().
One possibly misleading symptom is that the ActiveMQ GUI shows the objects only being dequeued once, despite the presence of two Consumers. The toy shows two dequeues for each enqueue. However, the program itself says they are read the expected number of times.
// creating the async consumer.
connAmq = createActiveMqConnection();
connAmq.start();
session = connAmq.createSession(true, Session.AUTO_ACKNOWLEDGE);
Destination topic = session.createTopic(appProperties.getActiveMqTopicQuotesName());
MessageConsumer consumer = session.createConsumer(topic);
consumer.setMessageListener(this);
public void onMessage(Message message) {
...
try {
if (message instanceof ObjectMessage) {
ObjectMessage msg = (ObjectMessage)message;
if (msg instanceof Foo) {
Foo quote = (Foo)msg.getObject();
...
}
}
}
...
}
// creating the sync consumer
jmsConnection = mActiveMQConnectionFactory.createTopicConnection();
jmsConnection.start();
jmsSession = jmsConnection.createSession(false, Session.AUTO_ACKNOWLEDGE);
jmsDestination = jmsSession.createTopic(name);
jmsMessageConsumer = jmsSession.createConsumer(jmsDestination);
//the code for consuming looks like this for the synchronous consumer
while(true)
ObjectMessage m = (ObjectMessage) jmsMessageConsumer.receive();
if (m != null)
Process(m.getObject());
}
At least in the code snippet given you have created a session that is transacted for the async consumer but I see no commit call on the session. The transaction bits stay in memory on the broker and you will eventually exhaust the Broker's memory.
I have a Java client which monitors RabbitMQ queue. I am able to get the count of messages currently in queue with this code
#Resource
RabbitAdmin rabbitAdmin;
..........
DeclareOk declareOk = rabbitAdmin.getRabbitTemplate().execute(new ChannelCallback<DeclareOk>() {
public DeclareOk doInRabbit(Channel channel) throws Exception {
return channel.queueDeclarePassive("test.pending");
}
});
return declareOk.getMessageCount();
I want to get some more additional details like -
Message body of currently enqueued items.
Total number of messages that was enqueued in the queue since the queue was created.
Is there any way to retrieve these data in Java client?
With AMQP protocol (including RabbitMQ implementation) you can't get such info with 100% guarantee.
The closest number to messages count is messages count returned with queue.declare-ok (AMQP.Queue.DeclareOk in java AMQP client library).
Whilst messages count you receive with queue.declare-ok may match exact messages number enqueues, you can't rely on it as it doesn't count messages which waiting acknowledges or published to queue during transaction but not committed yet.
It really depends what kind of precission do you need.
As to enqueued messages body, you may want to manually extract all messages in queue, view their body and put them back to queue. This is the only way to do what you want.
You can get some information about messages count with Management Plugin, RabbitMQ Management HTTP API and rabbitmqctl util (see list_queues, list_channels).
You can't get total published messages count since queue was created and I think nobody implement such stats while it useless (FYI, with messages flow in average 10k per second you will not even reach uint64 in a few thousand years).
AMQP.Queue.DeclareOk dok = channel.queueDeclare(QUEUE_NAME, true, false, false, queueArgs);
dok.getMessageCount();
To access queue details via http api,
http://public-domain-name:15672/api/queues/%2f/queue_name
To access queue details via command from localhost cli promt,
curl -i -u guest_uname:guest_password http://localhost:15672/api/queues/%2f/queue_name
Where,
%2f is default vhost "/"
I've been evaluating ActiveMQ as a candidate message broker. I've written some test code to try and get an understanding of ActiveMQ's performance limitations.
I can produce a failure state in the broker by sending messages as fast as possible like this:
try {
while(true) {
byte[] payload = new byte[(int) (Math.random() * 16384)];
BytesMessage message = session.createBytesMessage();
message.writeBytes(payload);
producer.send(message);
} catch (JMSException ex) { ... }
I was surprised that the line
producer.send(message);
blocks when the broker enters a failed state. I was hoping that some exception would be thrown, so there would be some indication that the broker has failed.
I realize that my test code is spamming the broker, and I expect the broker to fail. However, I would prefer that the broker failed "loudly" as opposed to simply blocking.
Is this an unrealistic expectation?
Update:
Uri's answer references an ActiveMQ bug report that was filed in March. The bug description includes a proposal that sounds like what I'm looking for: "if the request on the transport had a timeout (this is to catch failure scenarios, so something that's not expected to reasonably happen), things would have errored out rather than building waiting threads."
However, after 8 months the bug is currently unassigned with a single vote. So I guess the question still stands, is this something ActiveMQ should (will?) implement?
You are testing the 'slow consumer' and producer flowcontrol issue all message brokers have to deal with. Do you wanna fail producers, block them or spool to disk?
Basically the out of the box default in ActiveMQ is to block producers. But you can configure message cursors to spool to disk.
BTW you've not said if you are using queues/topics or persistent/non-persistent; if you are using non persistent topics there are other strategies you can use for discarding messages etc.
Apprently there's a known issue, not sure if it's been fixed:
https://issues.apache.org/activemq/browse/AMQ-1625
Not sure about ActiveMQ config, but other JMS providers have various configuration options - so you maybe able to get ActiveMQ to do as you wish in that situation.
I know Fiorano has options to specify whether providers block or not in this situation.