I'm not sure if ActiveMQ is a right tool here...
I have a task queue and multiple consumers, so my idea was to use ActiveMQ to post tasks, which are then consumed by consumers.
But I need to be able to cancel the task, if it was not processed yet...
Is there an API for removing Message from Queue in ActiveMQ?
Destination destination = session.createQueue(TOPIC_NAME);
MessageProducer producer = session.createProducer(destination);
ObjectMessage message = session.createObjectMessage(jobData);
producer.send(message);
...
producer.cancel(message); (?)
The use-case is that, for any reason, performing the task is no longer needed, and the task is resource-consuming.
What about setting an expiry time on the message?
http://activemq.apache.org/how-do-i-set-the-message-expiration.html
If you want a message to be deleted if it has not been processed / consumed in a particular time frame, then message expiry seems the answer to me.
ActiveMQ exposes a JMX interface that allows for operations of this kind. The MBean that models a Queue (e.g., org.apache.activemq:type=broker,brokerName=amq,destinationType=Queue,destinationName=my_queue) exposes a method removeMessage (String id). There are also methods that remove messages that match a particular pattern.
So far as I know, this functionality is not exposed outside JMX.
But...
I have a nasty feeling that JMX operations that work on specific messages only work on messages that are paged into memory. By default that would usually be the 400 messages nearest the head of the queue. I know this is true for selector operations, although I'm not sure about JMX.
Some ActiveMQ message stores (e.g., the JDBC store) might also provide a way to get to the underlying message data and manipulate it. On a relational database this is usually safe to do, because messages that are 'in flight' in a JMS operation will be locked at the database level. However, this is a lot of hassle for what ought to be a simple job.
I wonder if JMS is really the right technology for this job? It isn't really intended for random access. Perhaps some sort of distributed data cache would work better (jgroups, Hazelcast,...)?
For those who are looking for a direct answer, there's a JMS API to control this behaviour:
Per JMS API docs:
setTimeToLive(long timeToLive)
Specifies the time to live of messages that are sent using this JMSProducer.
So you can set this value on the producer before sending:
...
producer.setTimeToLive(30000L);
producer.send();
With this particular setting, messages will be retained for 30 seconds before being deleted by the Message Broker
Related
I have below configuration for rabbitmq
prefetchCount:1
ack-mode:auto.
I have one exchange and one queue is attached to that exchange and one consumer is attached to that queue. As per my understanding below steps will be happening if queue has multiple messages.
Queue write data on a channel.
As ack-mode is auto,as soon as queue writes message on channel,message is removed from queue.
Message comes to consumer,consumer start performing on that data.
As Queue has got acknowledgement for previous message.Queue writes next data on Channel.
Now,my doubt is,Suppose consumer is not finished with previous data yet.What will happen with that next data queue has written in channel?
Also,suppose prefetchCount is 10 and I have just once consumer attached to queue,where these 10 messages will reside?
The scenario you have described is one that is mentioned in the documentation for RabbitMQ, and elaborated in this blog post. Specifically, if you set a sufficiently large prefetch count, and have a relatively small publish rate, your RabbitMQ server turns into a fancy network switch. When acknowledgement mode is set to automatic, prefetch limiting is effectively disabled, as there are never unacknowledged messages. With automatic acknowledgement, the message is acknowledged as soon as it is delivered. This is the same as having an arbitrarily large prefetch count.
With prefetch >1, the messages are stored within a buffer in the client library. The exact data structure will depend upon the client library used, but to my knowledge, all implementations store the messages in RAM. Further, with automatic acknowledgements, you have no way of knowing when a specific consumer actually read and processed a message.
So, there are a few takeaways here:
Prefetch limit is irrelevant with automatic acknowledgements, as there are never any unacknowledged messages, thus
Automatic acknowledgements don't make much sense when using a consumer
Sufficiently-large prefetch when auto-ack is off, or any use of autoack = on will result in the message broker not doing any queuing, and instead doing routing only.
Now, here's a little bit of expert opinion. I find the whole notion of a message broker that "pushes" messages out to be a little backwards, and for this very reason- it's difficult to configure properly, and it is unclear what the benefit is. A queue system is a natural fit for a pull-based system. The processor can ask the broker for the next message when it is done processing the current message. This approach will ensure that load is balanced naturally and the messages don't get lost when processors disconnect or get knocked out.
Therefore, my recommendation is to drop the use of consumers altogether and switch over to using basic.get.
I am trying to understand the best use of RabbitMQ to satisfy the following problem.
As context I'm not concerned with performance in this use case (my peak TPS for this flow is 2 TPS) but I am concerned about resilience.
I have RabbitMQ installed in a cluster and ignoring dead letter queues the basic flow is I have a service receive a request, creates a persistent message which it queues, in a transaction, to a durable queue (at this point I'm happy the request is secured to disk). I then have another process listening for a message, which it reads (not using auto ack), does a bunch of stuff, writes a new message to a different exchange queue in a transaction (again now happy this message is secured to disk). Assuming the transaction completes successfully it manually acks the message back to the original consumer.
At this point my only failure scenario is is I have a failure between the commit of the transaction to write to my second queue and the return of the ack. This will lead to a message being potentially processed twice. Is there anything else I can do to plug this gap or do I have to figure out a way of handling duplicate messages.
As a final bit of context the services are written in java so using the java client libs.
Paul Fitz.
First of all, I suggest you to look a this guide here which has a lot of valid information on your topic.
From the RabbitMQ guide:
At the Producer
When using confirms, producers recovering from a channel or connection
failure should retransmit any messages for which an acknowledgement
has not been received from the broker. There is a possibility of
message duplication here, because the broker might have sent a
confirmation that never reached the producer (due to network failures,
etc). Therefore consumer applications will need to perform
deduplication or handle incoming messages in an idempotent manner.
At the Consumer
In the event of network failure (or a node crashing), messages can be
duplicated, and consumers must be prepared to handle them. If
possible, the simplest way to handle this is to ensure that your
consumers handle messages in an idempotent way rather than explicitly
deal with deduplication.
So, the point is that is not possibile in any way at all to guarantee that this "failure" scenario of yours will not happen. You will always have to deal with network failure, disk failure, put something here failure etc.
What you have to do here is to lean on the messaging architecture and implement if possibile "idempotency" of your messages (which means that even if you process the message twice is not going to happen anything wrong, check this).
If you can't than you should provide some kind of "processed message" list (for example you can use a guid inside every message) and check this list every time you receive a message; you can simply discard them in this case.
To be more "theorical", this post from Brave New Geek is very interesting:
Within the context of a distributed system, you cannot have
exactly-once message delivery.
Hope it helps :)
I think plenty of (Spring in my case) applications using JMS may follow this workflow:
Database A ===> Producer ===> JMS Queue ===> Consumer ===> Database B
then reliability is a concern. Let's say if when a data record in Database A should always be marked as delivered, when the message contains the data record is truely consumed and persist the data in Database B. Then there are questions:
From my knowledge, currently JMS protocol does not define any functions to send acknowledgement from consumer to producer, but only to MOM, so the actual consumer-to-producer acknowledgement methods vary by JMS provider. So does it mean there is no way to develop a mechanism for such acknowledgement that can work for generally all JMS products(ActiveMQ, WebSphere MQ and Jboss MQ)?
Consider the scenario of a blackout, then does it make the messages in the queue just evaporate so need to resend? or different JMS products can pick up what is left, as the messages are Serialized, so that missing message can be only caused by transaction management or async/sync configuration but not because of application server is down?
JMS guarantees the delivery of the message by nature, if the message is posted, then it will delivered to a consumer if there is one, whatever happen, the MOM is designed to ensure this fact. Anyway, delivered does not necessary mean processed.
Reliability is ensured by various mechanism :
the first one is the persistence of message in the queue (the queue AND the message must be flagged as persistent, which is the default value) which ensure that message will not be lost in case of system interruption.
then you have the acknowledgement and the retry policy, message will be kept in the queue until consumer acknowledge it and in case of transacted session, will be redelivered until consumer effectively processed the message or max retry is reached. Failed message can then be redirected to a dead letter queue for analysis.
To ensure the coherency between the two datasources you have to use XA transaction at least on the producer side (you have at least 2 resources implied in the transaction database A and JMS queue) in order to guarantee that the message will not be posted to the queue if the commit in database A fails or the database will not be updated if the post to the queue fails. Message consumption should be transacted too to ensure redelivery in case of rollback.
The transaction boundaries will never include both consumer and producer because it conflicts with the asynchronous nature of the messaging system, you can't afford to lock the resources on the producer side until the consumer process the message because you have no guarantee on when it will happen.
NB : in the event that your database does not support XA (or to improve performance) and if you have only 2 resources implied in the transaction (database and JMS queue) you can have a look to Logging Last Resource Transaction Optimization
1) From my experience with queue managers (MQ Series, ActiveMQ and HornetQ) I never needed this kind of acknowledgement between producer/consumer. Also the environment that I used to deal with, the traffic was about 50/60 million per day of objects on several queues. And the queues are all persisted as well.
2) In my case, using the persistence mechanism on queue manager was totally sufficient to handle a blackout scenario. I used disk persistence on MQ Series and HornetQ.
However, sometimes to ack the amount of messages, we developed some mechanisms to compare Database A with Database B, to be sure that messages were consumed as well. I don't know if JMS architecture should provide this kind of mechanism, because such task could decrease the performance.
It's something - in my point of view - that you have to measure on your system architecture how important is to match this information, because it's not that easy to keep.
Regards.
If I understand your question, this seems like a case for JTA/XA transactions (as long as your DB/JMS vendors support them). Spring TX managers can help make the tx management (more) vendor agnostic.
FYI, I use Apache Camel for this type of flow which has pretty good error handling across producers/consumers.
Say I load messages in a queue from multiple nodes.
Then, one or many nodes are pulling messages from the queue.
Is it possible (or is this normal usage?) that the queue guarantees to not hand out a message to more than one server/node?
And does that server/node have to tell the queue it has completed the operation and the queue and delete the message?
A message queuing system that did not guarantee to hand out a given message to just one recipient would not be worth the using. Some message queue systems have transactional controls. In that case, if a message is collected by one receiver as part of a transaction, but the receiver does not then commit the transaction (and the message queue can identify that the original recipient is no longer available), then it would be reissued. However, the message would not be made available to two processes concurrently.
What messaging/queuing technology are you using ? AMQP can certainly guarantee this behaviour (amongst many others, including pub/sub models)
If you want this in Java - then a JMS compliant messaging system will do what you want - and most messaging systems have a JMS client. You can Use Spring's JmsTemplate for real ease of use too.
With JMS - a message from a Queue will only be consumed by one and only one client - and once it is consumed (acknowledged) - it will be removed from the messaging system. Also when you publish a message using JMS - if its persistent - it will be sent synchronously, and the send() method won't return until the message is stored on the broker's disk - this is important - if you don't want to run the risk of loosing messages in the event of failure.
I am getting the following error:
javax.jms.ResourceAllocationException: [C4073]: A JMS destination limit was reached. Too many Subscribers/Receivers for Queue
The setup was done in such a way that there is one provider and two consumers.
Is the reason this error is happening because it is a queue and it can only have one provider and one consumer?
Queues in JMS are point-to-point mechanisms, you're not supposed to use them if you have multiple consumers. You should use topics for that.
Post the full code if you'd like, so we can try and help more, since it's not really clear what your code is doing.
JMS basic concepts
http://java.sun.com/products/jms/tutorial/1_3_1-fcs/doc/basics.html
That being said, according to Sun, this is the info for this error:
C4073
Message A JMS destination limit was reached. Too many Subscribers/Receivers for {0} : {1}
{0} is replaced with “Queue” or “Topic” {1} is replaced with the destination name.
Cause The client runtime was unable to create a message consumer for the specified domain and destination due to a broker resource constraint.
You actually can use queues with multiple consumers. Topics may not be what you want. A topic will work as a publish/subscribe model, where each consumer will get a copy of the message; whereas with queues, the JMS provider should only dispatch the message to one consumer.
There might be a config setting on your JMS provider that is limiting you to one consumer per queue. I've only used ActiveMQ for JMS, but it allows you to have mutliple consumers on a single queue - it basically load balances the requests between the consumers.