we're using activemq as message queue of our Java stand-alone application. my problem is that based on the activemq web console, the queue has this certain number of messages enqueued and dequeued. however, based on sysout statements i added in the code, it seems that the application is consuming less than the number of messages displayed on the activemq web console. for example, on the activemq console, no. of messages enqueued and dequeued is around 1800. however, the number of messages dequeued as displayed on console (i increment a counter per message received) is only around 1700.
i really don't know where the approx. 100 messages went. so i'm thinking maybe i'll have some idea if i know how to make activemq log the message enqueued by the producer and dequeued by the consumer. is this possible? if yes, how can this be done?
enqueued == numbers of messages put into the queue since the last restart
dequeued == number of messages successfully processed by the consumers
the difference in the two numbers == number of messages in-flight, usually tracked by the "dispatched" counter. "in-flight" means sent to the consumer, but not yet ack'd.
Related
I have a consumer process which has to do heavy lifting, it is a process that takes more than 30 sec to complete. And we acknowledge the message when process is completed successfully. However, looks like queue is waiting for acknowledgement and as it does not receive the acknowledgement within time, it's putting the message back to queue and same message is consumed by other consumer instance. Are there any config that I can tweak?? I don't want to auto acknowledge the message as it's an important flow and autoscaling down of the cluster may cause message loss.
I am looking if there is any config that can help me with it or is my understanding incorrect? I don't want same message getting consumed by more than one consumer. We're using IBM MQ in this instance.
However, looks like queue is waiting for acknowledgement and as it
does not receive the acknowledgement within time, it's putting the
message back to queue and same message is consumed by other consumer
instance.
Neither the queue nor the queue manager by itself puts the message back to the queue. There is one exception to that rule and that is if the client application crashes. If the queue manager determines that the application has crashed then it will rollback the message to the queue.
Or are you saying that if the sending application does not receive an acknowledgement within a specified amount of time then it resends the same message? If that is the case, then tell the sending application to double or triple the wait time.
I have 2 applications A and B, trying to send messages from both to one queue.
Placed a while loop at both places which is sending message to queue.
if i start application A and start while loop it starts sending message to queue and consumer consumes message sent from A, now at same time if i start while loop from B application it doesn't publish messages to queue as consumer doesn't consumes any message sent from B.
So can someone clear the doubt if messages are being sent at same time from multiple producers to single queue or not.
PS- using IBM queue and using a single consumer.
Yes, we can have multiple producers for single queue.
Multiple producers can also publish messages at the same time.
I´m using activemq(5.14.5) with camel(2.13.4) because I still need java 6.
I have a queue and 15 consumers. The messages sent to them are request reply.
When I start the consumers, the messages are distributed one per consumer as soon as the messages arrive but, after some time, only one consumer receives the messages, the others stay idle and a lot of messages stay pending.
The consumers have this configuration:
concurrentConsumers=15&maxMessagesPerTask=1&destination.consumer.prefetchSize=0&transferException=true
The time spent to process each message can varies a lot because of our business rule so, I don´t know if activemq has some rule that manage slow consumers and redirect to only one that is more "efficient".
The behaviour that I was expecting is that all the messages that arrives, start to process until all the consumers are full, but it is not what is happening.
Anybody knows what is happening?
Following is an image about what is happening:
Your configuration has two eye-catching settings:
maxMessagesPerTask=1
If you did not intend to configure auto-scaling the threadpool, you should remove this setting completely. Is is by default unlimited and it sets how long to keep threads for processing (scaling up/down threadpool).
See also the Spring Docs about this setting
prefetchSize=0
Have you tried setting this to 1 so that every consumer just gets 1 message at a time?
The AMQ docs say about the prefetchSize:
Large prefetch values are recommended for high performance with high message volumes. However, for lower message volumes, where each message takes a long time to process, the prefetch should be set to 1. This ensures that a consumer is only processing one message at a time. Specifying a prefetch limit of zero, however, will cause the consumer to poll for messages, one at a time, instead of the message being pushed to the consumer.
A case where senders are sending messages to a Queue, for example message1 is sent by sender1 to a queue. Now a consumer named consumer1 connects to queue and reads the message message1.
There is another consumer named consumer2. But the message message1 is already consumed by consumer1 so it will not be available for consumer2.
When a next message arrives in queue, consumer2 might receive that message if it reads the queue before consumer1.
Does it mean that it all is a case whether one consumer reads the queue before the other in order to get the first message available from the queue?
This is the nature of a Queue in JMS, messages are sent to one consumer and once ack'd they are gone, the next consumer can get the next message and so on. This is often referred to as competing consumers or load balancing. The consumers can share the work as jobs or work items are enqueued which allows for higher throughput when the work associated with the items in the Queue can take significant time.
There are options depending on the messaging broker to make a consumer exclusive such that only that consumer can read messages from the queue while the other consumers sit and wait for the exclusive consumer to leave which makes them backups of a sort.
Other options are to use something like Apache Camel to route a given message to more than one queue, or to use AcitveMQ Virtual Topics to send messages to a Topic and have that message then enqueue onto specific consumer Queues.
The solution depends on the broker you are using and the problem you are trying to solve, none of which you've really made clear in the question.
I have a queue with 50 consumers and the prefetch count of around 100. All the consumer is from single JVM instance. So, when the application goes down, the messages were in READY state. The number of messages in ready state is equal to (prefetch_count * no. of consumers).
Now the question is, what will happen to the messages in READY state? Will it redelivered or will it get dead lettered?
They will be redelivered.
Messages will only be dead-lettered if the consumer specifically rejects (nacks) the message with requeue=false (and the queue is configured for dead-lettering).