I want to make my app resilient to connection issues that can happen when sending messages to rabbitmq. I want to get hold of all unsent messages, store them and send them latter, when rabbitmq becomes available.
Looking at the official documentation I didn't manage to figure out what is the difference between return callback and retry callback. I understand that retry callback is invoked when retry template exhausts configured policy but don't find this much useful since context doesn't contain message itself.
Based on "replyCode" that is passed in the "returnedMessage" method of ReturnCallback interface one can easily determine further behavior but haven't figured out when this callback is invoked.
One way to go is ConfirmCallback but there is an issue of having additional logic for keeping in sync CorrelationData and messages and statuses of those messages.
So...except ConfirmCallback interface is there any easier way to keep track of messages that are not successfully sent through RabbitMQ using Spring AMQP?
Returns are when the broker returns a message because it's undeliverable (no matching bindings on the exchange to which the message was published, and the mandatory bit is set).
Confirms are when the broker sends an ack back to the publisher, indicating that a message was successfully routed.
Retry is outside the broker realm so it is likely what you need for your use case.
Related
The system consists of 1 or more clients and a single server. Each client is added manually and provided with an identifier (client.id). Each client can send messages to the server. The server can send messages to each client. All messages can be divided into two groups: with an answer and without.
For example some signatures:
CompletableFuture<Response> call(Request requestToServer)
void execute(Data dataToSend)
where Response, Request, and Data are my POJO.
So, I need some sort of RMI for implementing message communication between the server and clients.
Requirements:
The server must be able to identify a client by its id client.id when processing a message, but the client, before sending that message, should not directly fill this identifier;
Messages should be POJO;
Ability to answer to a message with an exception;
Event-driven handlers (like #RabbitListener) - several handlers - spring bean per incoming message type, with or without return type. A handler should be resolved automatically, based on incoming message type;
Backed by RabbitMQ or ArtemisMQ;
Single service for sending messages from the server to clients: client id should be provided when sending a message. Example: void sendToClient(int clientId, Data dataToClient).
What I've tried to set up this method of communication:
Spring Integration
My own gateway with completable future - great. Also, can enrich message headers with client.id - great. But I didn't find an appropriate way to handle an incoming message and being able to answer it. Tried to publish an ApplicationEvent, but all event handlers have a void return type. So, my idea here is to get correlationId and send back message, providing that correlationId - that doesn't look like a clear solution.
RabbitListener/RabbitTemplate
Cons:
A lot of code to setup RabbitTemplate to send and receive messages;
Need to manually setup request and reply queues and bindings;
problem with resolving client.id inside #RabbitHandler.
AmqpProxyFactoryBean
The closest result to my needs, but several problems, that I cannot solve:
Resolve client.id on message handler;
Single handler per service interface method.
So, I need a lightweight solution to build up communication between services, backed by a message queue. It should be easy to add additional message type - declare the class, add the handler to the consumer and create an object of that class and send it from the producer.
But maybe I'm completely wrong, about services communication? Maybe I should not use message queue for that purpose?
Well using a message queue or message broker like RabbitMQ is a perfectly valid solution. But you have to think whether it is what you actually need.
A message broker allows you to decouple the producers and consumers of messages. It also allows you to introduce a level of fault tolerance. If the consumer is not available temporarily the message is not lost. However, it also means that if a reply is expected the producer of the message might incorrectly think that the other end has processed its request and keep waiting for a reply. Message brokers also often offer some kind of guarantee, like once-and-only-once, or at-least-once, and also policies how to handle undeliverable messages (like dead letter queues), time-to-live, and various routing capabilities (like consistent hash based routing). In your case, you will probably have to route by some header value carrying your client.id if your server-originated messages are to reach one client only.
If you don't care about any of these features and just want a communications fabric, maybe going for something a bit different might make more sense. For example Akka offers actor-based distributed communication.
If what is bothering you is just the cleanliness of the solution, or the amount of boilerplate, maybe having a look at other implementations might help. You might want to have a look at the Reactive Streams API implementation for RabbitMQ, which should be a bit more abstract.
Our app uses Spring Boot and JMS messages with Tibco. We have two production servers running and processing messages concurrently. Servers are listening to the same one queue. Each server has 10 concurrent listeners. I do not want the very same message gets processed by both servers at the same time. Nothing prevents our queue of having duplicate messages, like we can have two copies of the message A in the queue. If messages in the queue are: A, A, B, C, D, then if first A gets delivered to server1 and second A gets delivered to server2, and both servers process A at the same time, then they are chances of creating duplicate entities. I want to find a way to send all A messages to only one server. I can't use Message Selector b/c we have the same code base running on both servers. This is what I'm considering:
Based on the message, set properties in the headers. Once the message got delivered to the process() method, depending on which server is processing the message, either discard, simply return the message or process the message and acknowledge it. The problem with this solution is that since we need to dynamicacaly find out which server is processing the message, the server name needs to be hardcoded, meaning if the server moves, the code breaks!
Other solution - that might work - is the Destination field.
https://docs.spring.io/spring/docs/4.0.x/spring-framework-reference/html/jms.html
Destinations, like ConnectionFactories, are JMS administered objects
that can be stored and retrieved in JNDI. When configuring a Spring
application context you can use the JNDI factory class
JndiObjectFactoryBean / to perform dependency
injection on your object’s references to JMS destinations.
It's something I never done before. Is there anyway, to configure the Destination that it picks up the right server to route the message to? Meaning, if message1 is supposed to be delivered to server1, then it does not even gets delivered to server2 and remains in the queue until server1 consumes it?
What are other ways to implement this?
EDIT:
I still do not know what’s the best way to send certain messages to only one server for processing, however, accepted the response given to use database as validation, b/c this is what we consider to avoid creating duplicate entities when processing the data.
I think the idea of using the JMS Destination is a non-starter as there is nothing in the JMS specification which guarantees any kind of link between the destination and a broker. The destination is just an encapsulation for the provider-specific queue/topic name.
The bottom line here is that you either need to prevent the duplicate messages in the first place or have some way to coordinate the consumers to deal with the duplicates after they've been pulled off the queue. I think you could do either of these using an external system like a database, e.g.:
When producing the message check the database for an indication that the message was sent already. If no indication is found then write a record to the database (will need to use a primary key to prevent duplicates) and send the message. Otherwise don't send the message.
When consuming the message check the database for an indication that the message is being (or was) consumed already. If no indication is found then write a record to the database (will need to use a primary key to prevent duplicates) and process the message. Otherwise just acknowledge the message without processing it.
I suggest an alternative to "post DB sync".
Keep the servers and listeners as-is, and broadcast all+ the the processed messages on a topic. For servers just starting, you can use "durable subscribers" to not miss any messages.
If you broadcast each start and end of processing for messages A, B, C, etc AND consider adding a little pause (in milli), you should avoid collisions. It's the main risk of course.
It's not clear to me if you should validate for duplicate processing at the beginning or end of a message processing... it depends on your needs.
If this whole idea is not acceptable, DB validation might be the only option, but as stated in comments above, I fear for scaling.
I am trying to understand the best use of RabbitMQ to satisfy the following problem.
As context I'm not concerned with performance in this use case (my peak TPS for this flow is 2 TPS) but I am concerned about resilience.
I have RabbitMQ installed in a cluster and ignoring dead letter queues the basic flow is I have a service receive a request, creates a persistent message which it queues, in a transaction, to a durable queue (at this point I'm happy the request is secured to disk). I then have another process listening for a message, which it reads (not using auto ack), does a bunch of stuff, writes a new message to a different exchange queue in a transaction (again now happy this message is secured to disk). Assuming the transaction completes successfully it manually acks the message back to the original consumer.
At this point my only failure scenario is is I have a failure between the commit of the transaction to write to my second queue and the return of the ack. This will lead to a message being potentially processed twice. Is there anything else I can do to plug this gap or do I have to figure out a way of handling duplicate messages.
As a final bit of context the services are written in java so using the java client libs.
Paul Fitz.
First of all, I suggest you to look a this guide here which has a lot of valid information on your topic.
From the RabbitMQ guide:
At the Producer
When using confirms, producers recovering from a channel or connection
failure should retransmit any messages for which an acknowledgement
has not been received from the broker. There is a possibility of
message duplication here, because the broker might have sent a
confirmation that never reached the producer (due to network failures,
etc). Therefore consumer applications will need to perform
deduplication or handle incoming messages in an idempotent manner.
At the Consumer
In the event of network failure (or a node crashing), messages can be
duplicated, and consumers must be prepared to handle them. If
possible, the simplest way to handle this is to ensure that your
consumers handle messages in an idempotent way rather than explicitly
deal with deduplication.
So, the point is that is not possibile in any way at all to guarantee that this "failure" scenario of yours will not happen. You will always have to deal with network failure, disk failure, put something here failure etc.
What you have to do here is to lean on the messaging architecture and implement if possibile "idempotency" of your messages (which means that even if you process the message twice is not going to happen anything wrong, check this).
If you can't than you should provide some kind of "processed message" list (for example you can use a guid inside every message) and check this list every time you receive a message; you can simply discard them in this case.
To be more "theorical", this post from Brave New Geek is very interesting:
Within the context of a distributed system, you cannot have
exactly-once message delivery.
Hope it helps :)
On my JMS applications we use temporary queues on Producers to be able to receive replies back from Consumer applications.
I am facing exactly same issue on my end as mentioned in this thread: http://activemq.2283324.n4.nabble.com/jira-Created-AMQ-3336-Temporary-Destination-errors-on-H-A-failover-in-broker-network-with-Failover-tt-td3551034.html#a3612738
Whenever I restarted an arbitrary broker in my network, I was getting many errors like this in my Consumer application log while trying to send reply to a temporary queue:
javax.jms.InvalidDestinationException:
Cannot publish to a deleted Destination: temp-queue://ID:...
Then I saw Gary's response there suggesting to use
jms.watchTopicAdvisories=false
as a url param on the client brokerURL. I promptly changed my client broker URLs with this additional parameter. However now I am seeing errors like this when I restart my brokers in network for this failover testing:
javax.jms.JMSException:
The destination temp-queue:
//ID:client.host-65070-1308610734958-2:1:1 does not exist.
I am using ActiveMQ 5.5 version. And my client broker URL looks like this:
failover:(tcp://amq-host1:61616,tcp://amq-host2.tred.aol.com:61616,tcp://amq-host3:61616,tcp://amq-host4:61616)?jms.useAsyncSend=true&timeout=5000&jms.watchTopicAdvisories=false
Additionally here is my activemq config XML for one of the 4 brokers:
amq1.xml
Can someone here please look into this problem and suggest me what mistake I am making in this setup.
Update:
To clarify further on how I am doing request-response in my code:
I already use a per producer destination (i.e. temporary queue) and set this in reply-to header of every message.
I am already sending a per message unique correlation identifier in JMSCorrelationID header.
As far as I know even Camel and Spring are also using temporary queue for request-response mechanism. Only difference is that Spring JMS implementation creates and destroys temporary queue for every message whereas I create temporary queue for the lifetime of the producer. This temporary queue is destroyed when client (producer) app shutsdown or by the AMQ broker when it realizes there are no active producer attached with this temporary queue.
I am already setting a message expiry on each message on Producer side so that message is not held up in a queue for too long (60 sec).
There is a broker attribute, org.apache.activemq.broker.BrokerService#cacheTempDestinations that should help in the failover: case.
Set that to true in xml configuration, and a temp destination will not be removed immediately when a client disconnects.
A fast failover: reconnect will be able to producer and/or consume from the temp queue again.
There is a timer task based on timeBeforePurgeTempDestinations (default 5 seconds) that handles cache removal.
One caveat though, I don't see any tests in activemq-core that make use of that attribute so I can't give you any guarantee on this one.
Temporary queues are created on the broker to which the requestor (producer) in your request-reply scenario connects. They are created from a javax.jms.Session, so on that session disconnecting, either because of client disconnect or broker failure/failover, those queues are permanently gone. None of the other brokers will understand what is meant when one of your consumers attempts to reply to those queues; hence your exception.
This requires an architectural shift in mindset assuming that you want to deal with failover and persist all your messages. Here is a general way that you could attack the problem:
Your reply-to headers should refer to a queue specific to the requestor process: e.g. queue:response.<client id>. The client id might be a standard name if you have a limited number of clients, or a UUID if you have a large number of these.
The outbound message should set a correlation identifier (simply a sting that lets you associate a request with a response - requestors after all might make more than one request at the same time). This is set in the JMSCorrelationID header, and ought to be copied from the request to the response message.
The requestor needs to set up a listener on that queue that will return the message body to the requesting thread based on that correllation id. There is some multithreading code that needs to be written for this, as you'll need to manually manage something like a map of correlation ids to originating threads (via Futures perhaps).
This is a similar approach to that taken by Apache Camel for request-response over messaging.
One thing to be mindful of is that the queue will not go away when the client does, so you should set a time to live on the response message such that it gets deleted from the broker if it has not been consumed, otherwise you will get a backlog of unconsumed messages. You will also need to set up a dead letter queue strategy to automatically discard expired messages.
Say I load messages in a queue from multiple nodes.
Then, one or many nodes are pulling messages from the queue.
Is it possible (or is this normal usage?) that the queue guarantees to not hand out a message to more than one server/node?
And does that server/node have to tell the queue it has completed the operation and the queue and delete the message?
A message queuing system that did not guarantee to hand out a given message to just one recipient would not be worth the using. Some message queue systems have transactional controls. In that case, if a message is collected by one receiver as part of a transaction, but the receiver does not then commit the transaction (and the message queue can identify that the original recipient is no longer available), then it would be reissued. However, the message would not be made available to two processes concurrently.
What messaging/queuing technology are you using ? AMQP can certainly guarantee this behaviour (amongst many others, including pub/sub models)
If you want this in Java - then a JMS compliant messaging system will do what you want - and most messaging systems have a JMS client. You can Use Spring's JmsTemplate for real ease of use too.
With JMS - a message from a Queue will only be consumed by one and only one client - and once it is consumed (acknowledged) - it will be removed from the messaging system. Also when you publish a message using JMS - if its persistent - it will be sent synchronously, and the send() method won't return until the message is stored on the broker's disk - this is important - if you don't want to run the risk of loosing messages in the event of failure.