I understand that there are different ways (or permutations) to implementing a JMS Request-Reply mechanism, i.e. request queue and response queue, request topic and response topic, or a mix of either.
What I would like to know is, (1) what is the recommended (or most common) way and (2) how do the different permutations measure up?
Next, is it more correct to say
a. "Send a message to a queue" or b. "Send a message through a queue"?
Cheers!
Normally, use a queue. "Request" implies a recipient, not a notice to anyone who cares, so you probably want the behaviour of a queue.
Queues usually do better for one thing - or a limited number of peer things - receiving the message and processing it. They also tend to saner persistence models than topic, when it matters that the message actually get to someone who processes it. (eg: if dropping the message is a problem, you probably want a queue)
Topics are more broadcast oriented: say something, and anyone who cares will hear about it. Normally that goes hand-in-hand with "...and no direct response is expected" because the "zero or more listeners" model ... well, zero listeners is always a problem if you expect a response.
Topics can do persistence, but the rules are stranger, and seldom what you actually want.
Finally, I think most people say "to" a queue, because the queue and the thing(s) processing messages off it are distinct, but really, it doesn't matter much as log as you convey your meaning.
Also with a Queue you are able to have multiple subscribers process the messages so its kid of a built in loadbalancer. You cannot do this easily with a Topic.
Related
I want to publish a message on a EventBus channel and receive a response from everyone who's listening to that channel. The thing is, how to I know everyone has already responded if I don't know how many responses to expect?
I assume I need to know how many consumers there are in order to know that I already got all responses or not.
Is there any way to know how many consumers are currently "listening" to a Vert.x EventBus address?
EventBus works on a "best effort" basis, so even if you could track the number of subscribers, there is a chance that some of them would never return your message.
If you're still inclined to try it, there are a couple of ways I can thin of. None are really recommended, but I'll try to highlight the pros and cons anywya.
One is to get a reference to handlerMap inside EventBusImpl through reflection.
https://github.com/eclipse-vertx/vert.x/blob/master/src/main/java/io/vertx/core/eventbus/impl/EventBusImpl.java#L48
If you get it once, when your application starts, that should have much impact on the performance. Of course that will break at runtime if Vert.x team decides to as much as change the name of the field.
Another option is to use vertx.sharedData(), for example getLocalMap(). All consumers will add themselves to the map, and producer will check the map to figure out how many consumers are listening. The problem with that implementation:
Lots of logic to implement this
Unregistering consumers is hard (there's no guarantee that consumer will be able to unregister itself)
I need a solution for the following scenario which is similar to a queue:
I want to write messages to a queue continuously. My message is very big, containing a lot of data so I do want to make as few requests as possible.
So my queue will contain a lot of messages at some point.
My Consumer will read from the queue every 1 hour. (not whenever a new message is written) and it will read all the messages from the queue.
The problem is that I need a way to read ALL the messages from the queue using only one call (I also want the consumer to make as few requests to the queue as possible).
A close solution would be ActiveMQ but the problem is that you can only read one message at a time and I need to read them all in one request.
So my question is.. Would there be other ways of doing this more efficiently? The actual thing that I need is to persist in some way messages created continuously by some application and then consume them (also delete them) by the same application all at once, every 1 hour.
The reason I thought a queue would be fit is because as the messages are consumed they are also deleted but I need to consume them all at once.
I think there's some important things to keep in mind as you're searching for a solution:
In what way do you need to be "more efficient" (e.g. time, monetary cost, computing resources, etc.)?
It's incredibly hard to prove that there are, in fact, no other "more efficient" ways to solve a particular problem, as that would require one to test all possible solutions. What you really need to know is, given your specific use-case, what solution is good enough. This, of course, requires knowing specifically what kind of performance numbers you need and the constraints on acquiring those numbers (e.g. time, monetary cost, computing resources, etc.).
Modern message broker clients (e.g. those shipped with either ActiveMQ 5.x or ActiveMQ Artemis) don't make a network round-trip for every message they consume as that would be extremely inefficient. Rather, they fetch blocks of messages in configurable sizes (e.g. prefetchSize for ActiveMQ 5.x, and consumerWindowSize for ActiveMQ Artemis). Those messages are stored locally in a buffer of sorts and fed to the client application when the relevant API calls are made to receive a message.
Making "as few requests as possible" is rarely a way to increase performance. Modern message brokers scale well with concurrent consumers. Consuming all the messages with a single consumer drastically limits the message throughput as compared to spinning up multiple threads which each have their own consumer. Rather than limiting the number of consumer requests you should almost certainly be maximizing them until you reach a point of diminishing returns.
I am trying to understand the best use of RabbitMQ to satisfy the following problem.
As context I'm not concerned with performance in this use case (my peak TPS for this flow is 2 TPS) but I am concerned about resilience.
I have RabbitMQ installed in a cluster and ignoring dead letter queues the basic flow is I have a service receive a request, creates a persistent message which it queues, in a transaction, to a durable queue (at this point I'm happy the request is secured to disk). I then have another process listening for a message, which it reads (not using auto ack), does a bunch of stuff, writes a new message to a different exchange queue in a transaction (again now happy this message is secured to disk). Assuming the transaction completes successfully it manually acks the message back to the original consumer.
At this point my only failure scenario is is I have a failure between the commit of the transaction to write to my second queue and the return of the ack. This will lead to a message being potentially processed twice. Is there anything else I can do to plug this gap or do I have to figure out a way of handling duplicate messages.
As a final bit of context the services are written in java so using the java client libs.
Paul Fitz.
First of all, I suggest you to look a this guide here which has a lot of valid information on your topic.
From the RabbitMQ guide:
At the Producer
When using confirms, producers recovering from a channel or connection
failure should retransmit any messages for which an acknowledgement
has not been received from the broker. There is a possibility of
message duplication here, because the broker might have sent a
confirmation that never reached the producer (due to network failures,
etc). Therefore consumer applications will need to perform
deduplication or handle incoming messages in an idempotent manner.
At the Consumer
In the event of network failure (or a node crashing), messages can be
duplicated, and consumers must be prepared to handle them. If
possible, the simplest way to handle this is to ensure that your
consumers handle messages in an idempotent way rather than explicitly
deal with deduplication.
So, the point is that is not possibile in any way at all to guarantee that this "failure" scenario of yours will not happen. You will always have to deal with network failure, disk failure, put something here failure etc.
What you have to do here is to lean on the messaging architecture and implement if possibile "idempotency" of your messages (which means that even if you process the message twice is not going to happen anything wrong, check this).
If you can't than you should provide some kind of "processed message" list (for example you can use a guid inside every message) and check this list every time you receive a message; you can simply discard them in this case.
To be more "theorical", this post from Brave New Geek is very interesting:
Within the context of a distributed system, you cannot have
exactly-once message delivery.
Hope it helps :)
I am using Apache Camel with ActiveMQ and wanting to implement guaranteed message delivery.
I have been reading through the Camel in Action book as well as the Apache Camel Developer's Cookbook.
I am hoping someone here can advise me in my approach. I am not asking for code samples.
The way I envisioned the implementation is as follows:
1. Message is received on an endpoint
2. I inspect the message
3. I use the Wiretap pattern to drop it immediately on my "GuaranteedMessages" queue if the message asks for guaranteed delivery
4. I route the message to its proper destination
5. If no exceptions were encountered, I remove the message manually from the "GuaranteedMessages" queue
Sounds easy enough. However, I have been reading about the "Dead Letter Channel" pattern in Camel.
My understanding is using this pattern's implementation implies that instead of automatically dropping each (guaranteed) message to my "GuaranteedMessages" queue, I drop that approach and instead, I set the redelivery options (max attempts, exponential delay, redelivery delay, etc.). Then, I rely on Camel to try redelivering and simply drop it off in the dead letter channel delay if it never goes through.
Then, I would have a separate route that uses this dead letter queue as it's source. Again, it would be the same pattern. If it cannot succeed, send the message back to the dead letter queue.
Does this sound like a realistic implementation for a production system?
If instead, I decide to drop every incoming message (that needs to be guaranteed) on my own "GuaranteeMessage" queue, is it realistic to believe that I can later go and remove that specific message manually from the queue? My understanding is that I would have to manually browse the queue, iterate through any number of messages, and then consume that message manually. I am not sure how scalable such an architecture really is.
Presumably the final destination is not another ActiveMQ queue, it something that can through exception. Your idea of the wiretap is functionally the same as using the DLQ so you might as well use the Camel functionality, which works fine, to do as much work as possible.
However, two points. Firstly I would use an explicit queue to hold the messages that need retrying, rather than the DLQ, as there is only one DLQ per broker and you don't want anything else unexpected appearing on it.
Secondly if you are just going to take a message from the retry queue and resubmit it, why not just increase the retry count and delay in Camel exception handling? That way your retry queue just has messages that probably require some manual intervention. So when a message is on the retry queue, you manually check/fix whatever the underlying cause is and manually move the message to the input queue.
Say you have a JMS queue, and multiple consumers are watching the queue for messages. You want one of the consumers to get all of a particular type of message, so you decide to employ message selectors.
For example, you define a property to go in your JMS message header named, targetConsumer. Your message selector, which you apply to the consumer known as, A, is something like WHERE targetConsumer = 'CONSUMER_A'.
It's clear that consumer A will now just grab messages with the property set like it is in in the example. Will the other consumers have awareness of that, though? IOW, will another consumer, unconstrained by a message selector, grab the CONSUMER_A messages, if it looks at the queue before Consumer A? Do I need to apply message selectors like, WHERE targetConsumer <> 'CONSUMER_A' to the others?
I am RTFMing and gathering empirical data now, but was hoping someone might know off the top of their head.
When multiple consumers use the same queue, message selectors need to configured correctly across these consumers so that there is no conflict in determining the intended consumer.
In the case of message-driven-beans (a consumer of JMS messages), the selector can be specified in the ejb-jar.xml file thereby allowing for the configuration to be done at deployment time (instead of the opposing view of specifying the message selector during development).
Edit: In real life, this would make sense when different consumers are responsible for processing messages containing the same headers (often generated by the same producer) written onto the same queue. For instance, message selectors could be used in a trading application, to differentiate between buy and sell orders, when the producer is incapable of writing the JMS messages onto two separate buy and sell queues.
Yes, another consumer which is not using any message selector will get message intended for consumer A (or for that matter any message on top of the queue). Hence when sharing a queue, consumer applications must be disciplined and pick only those messages intended for them.
The 'first' JMS message consumer from a queue will pick up the message if the selector matches. What 'first' means is an implementation detail (could be round-robin, based on priority or network closeness). So when using selectors on queues you need to make sure that these selectors are 'non overlapping'.
More formally: no message must exist that matches 2 selectors on the same queue
This is yet another disadvantage of queues versus topics - in practice you should always consider using topics first. With a topic each matching consumer receives the message.