Is there a way to explicitly tell the broker to send a message to the queue's assigned dead letter queue?
I know we can configure a queue to automatically send messages to the DLQ after a certain number of re-delivery attempts. That makes perfect sense for transient errors such as database issues, network issues, etc. However in the case of a business rule error, it doesn't make sense to have that message attempt redelivery X number of times over X number of minutes before being sent to the dead letter queue when we know it's a business rule violate / malformed message, etc.
I was hoping there was a way that when we catch a business rule violation we can immediately send it to that queues dead letter queue. I know I could explicitly write the code to send it to that dead letter queue but we will have many (dozens) of configurable queues and their associated dead letter queues will be configured by our middleware team. I don't want to explicitly code the dead letter queue queue names or even configure them in my own properties. I'm hoping there is a simple way to tell the broker to immediately send the message to the dead letter queue and not attempt redelivery.
It seems like it should be something like message.deadLetter(). I feel like I must be missing something simple but I don't see any mechanism like that on the consumer, session or message.
There is no accommodation for the feature you're describing in the JMS specification. The JMS spec incorporates redelivery via things like the JMSRedelivered header and JMSXDeliveryCount property. However, it actually makes no mention of "dead-letter" destinations.
That said, such a feature might be provided by a particular JMS broker, but since you don't mention what JMS broker implementation you're using it's impossible to say whether your chosen broker implements such a feature. In any case, it would be configured and/or invoked via an implementation-specific mechanism that would be, by definition, non-portable between brokers and not available from the JMS API.
Related
Our app uses Spring Boot and JMS messages with Tibco. We have two production servers running and processing messages concurrently. Servers are listening to the same one queue. Each server has 10 concurrent listeners. I do not want the very same message gets processed by both servers at the same time. Nothing prevents our queue of having duplicate messages, like we can have two copies of the message A in the queue. If messages in the queue are: A, A, B, C, D, then if first A gets delivered to server1 and second A gets delivered to server2, and both servers process A at the same time, then they are chances of creating duplicate entities. I want to find a way to send all A messages to only one server. I can't use Message Selector b/c we have the same code base running on both servers. This is what I'm considering:
Based on the message, set properties in the headers. Once the message got delivered to the process() method, depending on which server is processing the message, either discard, simply return the message or process the message and acknowledge it. The problem with this solution is that since we need to dynamicacaly find out which server is processing the message, the server name needs to be hardcoded, meaning if the server moves, the code breaks!
Other solution - that might work - is the Destination field.
https://docs.spring.io/spring/docs/4.0.x/spring-framework-reference/html/jms.html
Destinations, like ConnectionFactories, are JMS administered objects
that can be stored and retrieved in JNDI. When configuring a Spring
application context you can use the JNDI factory class
JndiObjectFactoryBean / to perform dependency
injection on your object’s references to JMS destinations.
It's something I never done before. Is there anyway, to configure the Destination that it picks up the right server to route the message to? Meaning, if message1 is supposed to be delivered to server1, then it does not even gets delivered to server2 and remains in the queue until server1 consumes it?
What are other ways to implement this?
EDIT:
I still do not know what’s the best way to send certain messages to only one server for processing, however, accepted the response given to use database as validation, b/c this is what we consider to avoid creating duplicate entities when processing the data.
I think the idea of using the JMS Destination is a non-starter as there is nothing in the JMS specification which guarantees any kind of link between the destination and a broker. The destination is just an encapsulation for the provider-specific queue/topic name.
The bottom line here is that you either need to prevent the duplicate messages in the first place or have some way to coordinate the consumers to deal with the duplicates after they've been pulled off the queue. I think you could do either of these using an external system like a database, e.g.:
When producing the message check the database for an indication that the message was sent already. If no indication is found then write a record to the database (will need to use a primary key to prevent duplicates) and send the message. Otherwise don't send the message.
When consuming the message check the database for an indication that the message is being (or was) consumed already. If no indication is found then write a record to the database (will need to use a primary key to prevent duplicates) and process the message. Otherwise just acknowledge the message without processing it.
I suggest an alternative to "post DB sync".
Keep the servers and listeners as-is, and broadcast all+ the the processed messages on a topic. For servers just starting, you can use "durable subscribers" to not miss any messages.
If you broadcast each start and end of processing for messages A, B, C, etc AND consider adding a little pause (in milli), you should avoid collisions. It's the main risk of course.
It's not clear to me if you should validate for duplicate processing at the beginning or end of a message processing... it depends on your needs.
If this whole idea is not acceptable, DB validation might be the only option, but as stated in comments above, I fear for scaling.
I am trying to understand the best use of RabbitMQ to satisfy the following problem.
As context I'm not concerned with performance in this use case (my peak TPS for this flow is 2 TPS) but I am concerned about resilience.
I have RabbitMQ installed in a cluster and ignoring dead letter queues the basic flow is I have a service receive a request, creates a persistent message which it queues, in a transaction, to a durable queue (at this point I'm happy the request is secured to disk). I then have another process listening for a message, which it reads (not using auto ack), does a bunch of stuff, writes a new message to a different exchange queue in a transaction (again now happy this message is secured to disk). Assuming the transaction completes successfully it manually acks the message back to the original consumer.
At this point my only failure scenario is is I have a failure between the commit of the transaction to write to my second queue and the return of the ack. This will lead to a message being potentially processed twice. Is there anything else I can do to plug this gap or do I have to figure out a way of handling duplicate messages.
As a final bit of context the services are written in java so using the java client libs.
Paul Fitz.
First of all, I suggest you to look a this guide here which has a lot of valid information on your topic.
From the RabbitMQ guide:
At the Producer
When using confirms, producers recovering from a channel or connection
failure should retransmit any messages for which an acknowledgement
has not been received from the broker. There is a possibility of
message duplication here, because the broker might have sent a
confirmation that never reached the producer (due to network failures,
etc). Therefore consumer applications will need to perform
deduplication or handle incoming messages in an idempotent manner.
At the Consumer
In the event of network failure (or a node crashing), messages can be
duplicated, and consumers must be prepared to handle them. If
possible, the simplest way to handle this is to ensure that your
consumers handle messages in an idempotent way rather than explicitly
deal with deduplication.
So, the point is that is not possibile in any way at all to guarantee that this "failure" scenario of yours will not happen. You will always have to deal with network failure, disk failure, put something here failure etc.
What you have to do here is to lean on the messaging architecture and implement if possibile "idempotency" of your messages (which means that even if you process the message twice is not going to happen anything wrong, check this).
If you can't than you should provide some kind of "processed message" list (for example you can use a guid inside every message) and check this list every time you receive a message; you can simply discard them in this case.
To be more "theorical", this post from Brave New Geek is very interesting:
Within the context of a distributed system, you cannot have
exactly-once message delivery.
Hope it helps :)
I am using Apache Camel with ActiveMQ and wanting to implement guaranteed message delivery.
I have been reading through the Camel in Action book as well as the Apache Camel Developer's Cookbook.
I am hoping someone here can advise me in my approach. I am not asking for code samples.
The way I envisioned the implementation is as follows:
1. Message is received on an endpoint
2. I inspect the message
3. I use the Wiretap pattern to drop it immediately on my "GuaranteedMessages" queue if the message asks for guaranteed delivery
4. I route the message to its proper destination
5. If no exceptions were encountered, I remove the message manually from the "GuaranteedMessages" queue
Sounds easy enough. However, I have been reading about the "Dead Letter Channel" pattern in Camel.
My understanding is using this pattern's implementation implies that instead of automatically dropping each (guaranteed) message to my "GuaranteedMessages" queue, I drop that approach and instead, I set the redelivery options (max attempts, exponential delay, redelivery delay, etc.). Then, I rely on Camel to try redelivering and simply drop it off in the dead letter channel delay if it never goes through.
Then, I would have a separate route that uses this dead letter queue as it's source. Again, it would be the same pattern. If it cannot succeed, send the message back to the dead letter queue.
Does this sound like a realistic implementation for a production system?
If instead, I decide to drop every incoming message (that needs to be guaranteed) on my own "GuaranteeMessage" queue, is it realistic to believe that I can later go and remove that specific message manually from the queue? My understanding is that I would have to manually browse the queue, iterate through any number of messages, and then consume that message manually. I am not sure how scalable such an architecture really is.
Presumably the final destination is not another ActiveMQ queue, it something that can through exception. Your idea of the wiretap is functionally the same as using the DLQ so you might as well use the Camel functionality, which works fine, to do as much work as possible.
However, two points. Firstly I would use an explicit queue to hold the messages that need retrying, rather than the DLQ, as there is only one DLQ per broker and you don't want anything else unexpected appearing on it.
Secondly if you are just going to take a message from the retry queue and resubmit it, why not just increase the retry count and delay in Camel exception handling? That way your retry queue just has messages that probably require some manual intervention. So when a message is on the retry queue, you manually check/fix whatever the underlying cause is and manually move the message to the input queue.
I am trying to get the number of consumers of a particular Websphere MQ queue from Java? I need to know whether someone is going to consume the messages before placing them on the queue.
First, it is worth noting that the design proposed is a very, VERY bad design. The effect is to turn async messaging back into synchronous messaging. This couples message producers to consumers, introduces location and resolution dependencies, breaks clustering, defeats WMQ's load distribution and balancing, embeds network topology into the application, and makes the whole system brittle. Please do not blame WMQ for not working correctly after intentionally defeating all its best features except the actual queue/dequeue operations.
However, to answer your question more directly, use the getOpenInputCount method of the queue object to obtain the number of open input handles. Here's how:
MQQueue outQ = qMgr.accessQueue(qName,
openOptions,
null, // default q manager
null, // no dynamic q name
null); // no alternate user id
int inCount = outQ.getOpenInputCount();
Note that you can only inquire the input handles on a local queue. If the queue is hosted on a QMgr other than the one where the message sender is connected, this method will not work. Of course it is the normal case that the message sender and receiver would reside on different QMgrs. However since you do not mention much about the design, I'll assume for purposes of this answer that connections from the message producer and consumer attach to the same QMgr. If that's not the case, we need to have a discussion about PCF and even stronger warnings about the design.
I'm trying to move off of ActiveMQ but one feature we'd like to keep is the message group. By adding a session ID to the JMS header ActiveMQ will route all other messages on the queue with the same ID to the same consumer (our consumers may be on different machines) allowing the receiver to treat the group of messages as one unit of work.
My first thought was simply to put the session into CLIENT_ACKNOWLEDGE mode. My thinking was that if consumer A looked at the header and saw it wasn't an ID it was handling then it could just drop the message and consumer B would pick it up. I've hit several issues, including ActiveMQ's prefecting, and the more I read, the more it looks like that's not what that was designed for to begin with.
The one idea I can think of is to have a dispatch queue which would then route messages to each consumer's, for lack of a better word, sub-queue and manage matching the session IDs to the sub-queues ourselves.
Before I head down this path, which we're leery of since it'd add more complexity to the code then we'd like, is there anything I'm missing about CLIENT_ACKNOWLEDGE? Or something else entirely I should try first?
Is this what you are trying to do http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/5/html-single/HornetQ_User_Guide/index.html#message-grouping