Is there a message queue implementation that allows breaking up work into 'batches' by inserting 'message barriers' into the message stream? Let me clarify. No messages after a message barrier should be delivered to any consumers of the queue, until all messages before the barrier are consumed. Sort of like a synchronization point. I'd also prefer if all consumers received notification when they reached a barrier.
Anything like this out there?
I am not aware of existing, widely-available implementations, but if you'll allow me I'd propose a very simple, generic implementation using a proxy, where:
producers write to the proxy queue/topic
the proxy forwards to the original queue/topic until a barrier message is read by the proxy, at which point:
the proxy may notify topic subscribers of the barrier by forwarding the barrier message to the original topic, or
the proxy may notify queue subscribers of the barrier by:
periodically publishing barrier messages until the barrier has been cleared; this does not guarantee that all consumers will receive exactly one notification, although all will eventually clear the barrier (some may receive 0 notifications, others >1 notifications -- all depending on the type of scheduler used to distribute messages to consumers e.g. if non-roundrobin)
using a dedicated topic to notify each consumer exactly once per barrier
the proxy stops forwarding any messages from the proxy queue until the barrier has been cleared, that is, until the original queue has emptied and/or all consumers have acknowledged all queue/topic messages (if any) leading up to the barrier
the proxy resumes forwarding
UPDATE
Thanking Miklos for pointing out that under JMS the framework does not provide acknowledgements for asynchronous deliveries (what is referred to as "acknowledgements" in JMS are purely a consumer side concept and are not proxiable as-such.)
So, under JMS, the existing implementation (to be adapted for barriers) may already provide application-level acknowledgements via an "acknowledgement queue" (as opposed to the original queue -- which would be a "request queue".) The consumers would have to acknowledge execution of requests by sending acknowledgement messages to the proxy acknowledgement queue; the proxy would use the acknowledgement messages to track when the barrier has been cleared, after having also forwarded the acknowledgement messages to the producer.
If the existing implementation (to be adapted for barriers) does not already provide application-level acknowledgements via an "acknowledgement queue", then you could either:
have the proxy use the QueueBrowser, provided that:
you are dealing with queueus not events, that
you want to synchronize on delivery not acknowledgement of execution, and
it is OK to synchronize on first delivery, even if the request was actually aborted and has to be re-delivered (even after the barrier has been cleared.) I think Miklos already pointed this problem out IIRC.
otherwise, add an acknowledgment queue consumed by the proxy, and adapt the consumers to write acknowledgements to it (essentially the JMS scenario above, except it is not necessary for the proxy to forward acknowledgement messages to the producer unless your producer needs the functionality.)
You could achieve this using a topic for the 'Barrier Message' and a queue for the 'batched items' which are consumed with selective receivers.
Publishing the Barrier Message to a topic ensures that all consumers receive their own copy of the Barrier Message.
Each consumer will need two subscriptions:
To the Barrier Topic
A selective receiver against the batch queue, using selection criteria defined by the Barrier Message.
The Barrier Message will need to contain a batch key that must be applied to the queue consumers selection criteria.
e.g. batchId = n
or JMSMessageID < 100
or JMSTimestamp < xxx
Whenever a barrier message is received,
the current queue consumer must be closed
the queue selection criteria must be modified using the content of the Barrier Message
a new selective consumer must be started using the modified selection criteria
If you are going to use a custom batch key for the selection criteria such as 'batchId' above, then the assumption is that all message producers are capable of setting that JMS property or else a proxy will have to consume the messages set the property and republish to the queue where the selective consumers are listening.
For more info on selective receivers see these links:
http://java.sun.com/j2ee/1.4/docs/api/javax/jms/Message.html
http://java.sun.com/j2ee/sdk_1.3/techdocs/api/javax/jms/QueueSession.html#createReceiver(javax.jms.Queue,%20java.lang.String)
Related
A case where senders are sending messages to a Queue, for example message1 is sent by sender1 to a queue. Now a consumer named consumer1 connects to queue and reads the message message1.
There is another consumer named consumer2. But the message message1 is already consumed by consumer1 so it will not be available for consumer2.
When a next message arrives in queue, consumer2 might receive that message if it reads the queue before consumer1.
Does it mean that it all is a case whether one consumer reads the queue before the other in order to get the first message available from the queue?
This is the nature of a Queue in JMS, messages are sent to one consumer and once ack'd they are gone, the next consumer can get the next message and so on. This is often referred to as competing consumers or load balancing. The consumers can share the work as jobs or work items are enqueued which allows for higher throughput when the work associated with the items in the Queue can take significant time.
There are options depending on the messaging broker to make a consumer exclusive such that only that consumer can read messages from the queue while the other consumers sit and wait for the exclusive consumer to leave which makes them backups of a sort.
Other options are to use something like Apache Camel to route a given message to more than one queue, or to use AcitveMQ Virtual Topics to send messages to a Topic and have that message then enqueue onto specific consumer Queues.
The solution depends on the broker you are using and the problem you are trying to solve, none of which you've really made clear in the question.
Using RabbitMQ, I have two types of consumers: FileConsumer writes messages to file and MailConsumer mails messages. There may be multiple consumers of each type, say three running MailConsumers and one FileConsumer instance.
How can I do this:
Each published message should be handled by exactly one FileConsumer instance and one MailConsumer instance
Publishing a message should be done once, not one time for each queue (if possible)
If there are no consumers connected, messages should be queued until consumed, not dropped
What type of exchange etc should I use to get this behavior? I'd really like to see some example/pseudo-code to make this clear.
This should be easy to do, but I couldn't figure it out from the docs. It seems the fanout example should work, but I'm confused with these "anonymous queues" which seems like it will lead to sending same message into each consumer.
If you create queue without auto-delete flag, then queues will stay alive even after consumers disconnection.
Note, that if you declare queue as persistent, it will be present even after broker restart.
If you will publish then messages with delivery-mode=2 property set (that mean that message will be persistent), such messages will stay in persistent (this is important to make queue persistent) queues even after broker restart.
Using fanout exchange type is not mandatory. You can also use topic for better message routing handling if you need that.
UPD: step-by-step way to get what you show with schema.
Declare persistent exchange, say main, as exchange.declare(exchange-name=main, type=fanout, durable=true).
Delcare two queues, say, files and mails as queue.declare(queue-name=files, durable=true) and queue.declare(queue-name=mails, durable=true)
Bind both queues to exchange as queue.bind(queue-name=files, exchange-name=main) and queue.bind(queue-name=mails, exchange-name=main).
At this point you can publish messages to main exchange (see note about delivery-mode above) and consume with any consumer number from queues, from files with FileConsumer and from mails with MailConsumer. Without any consumers on queues messages will be queued and stay in queue until they consumed (or broker restart is they are not persistent).
My JMS consumer produces any number of messages on a JMS queue during the day. As soon as a
message arrives it goes to message listener. If in between I need some other message comes, it goes to another message listener does not wait for first one?
As per my understanding here I need to create two consumer(assume i want to process 2 message concurrently) each having its own session. Both consumer can use the same message listener. Right?
I am not sure if I can achieve it with single consumer but can I with multiple listeners?
something like this Single queue: concurrent message processing with multiple consumers
Per the JMS documentation #bgth cites, multiple MessageListeners in a single session will not provide concurrency:
"The session used to create the message consumer serializes the execution of all message listeners registered with the session. At any time, only one of the session’s message listeners is running"
For concurrency, you need multiple sessions and multiple consumers in separate threads. You can reuse the same MessageListener in this case, but it must be threadsafe.
I want to concurrently consume jms messages from multiple queues. All the messages should go to the DB after long running processing and I have no right to lose them.
Question: Is it possible to save messages for future acknowledgement and call oldMessage.acknowledge() when another message is being processed?
My first guess is that this is impossible since it is deep in the jms processing unit and I have to process message and acknowledgement within an onMessage(...) method.
Second guess is to split onMessage() concurrently and allow long running processing for many messages. But this is not a good option since I have to ensure that all messages are coming ordered!
2nd question: Is there any way to ensure the incoming order while concurrency processing?
1: JMS has a flag on Session that is *CLIENT_ACKNOWLEDGE* you can see it here. I never used it but seems to do what you want.
2:
2.1: You have N consumers for the same queue: You can explore the Exclusive Consumer that some implementations have support. (for AtiveMQ: here).
2.2 You have 1 consumer per queue but you want to order all messages from all queues.
You can use the concept of an ordered SlackBuffer.
You can explore another possibilities like: Redirect all messages to an output queue that maintains the order of messages and you will only consume messages from that single output queue. The order of messages and the redirection are accomplished by the MQ server. It is only a valid idea if you can control the MQ server.
I hope this can help
Say I load messages in a queue from multiple nodes.
Then, one or many nodes are pulling messages from the queue.
Is it possible (or is this normal usage?) that the queue guarantees to not hand out a message to more than one server/node?
And does that server/node have to tell the queue it has completed the operation and the queue and delete the message?
A message queuing system that did not guarantee to hand out a given message to just one recipient would not be worth the using. Some message queue systems have transactional controls. In that case, if a message is collected by one receiver as part of a transaction, but the receiver does not then commit the transaction (and the message queue can identify that the original recipient is no longer available), then it would be reissued. However, the message would not be made available to two processes concurrently.
What messaging/queuing technology are you using ? AMQP can certainly guarantee this behaviour (amongst many others, including pub/sub models)
If you want this in Java - then a JMS compliant messaging system will do what you want - and most messaging systems have a JMS client. You can Use Spring's JmsTemplate for real ease of use too.
With JMS - a message from a Queue will only be consumed by one and only one client - and once it is consumed (acknowledged) - it will be removed from the messaging system. Also when you publish a message using JMS - if its persistent - it will be sent synchronously, and the send() method won't return until the message is stored on the broker's disk - this is important - if you don't want to run the risk of loosing messages in the event of failure.