I am using RabbitMQ together with Spring's RabbitTemplate.
When sending messages to queues using the template send methods, I want the queue to automatically be created/declared if it is not already exists.
It is very important since according to our business logic queue names are generated on run-time and I cannot declare them in advance.
Previously we have used JmsTemplate and any call to send or receive automatically created the queue.
You can use a RabbitAdmin to automatically declare the exchange, queue, and binding. Check out this thread for more detail. This forum also bit related to your scenario. I have not tried spring with AMQP though, but I believe this would do it.
/**
* Required for executing adminstration functions against an AMQP Broker
*/
#Bean
public AmqpAdmin amqpAdmin() {
return new RabbitAdmin(connectionFactory());
}
Keep coding !
Yes, you can use a RabbitAdmin and admin.getQueueProperties() to see if the queue exists and admin.declareQueue(new Queue(...)) to add a queue. You should probably keep track of which one's you've already checked/created in order to avoid the overhead on every send.
You can also add exchanges and bind queues to them with the admin.
Related
we have a larger multi service java spring app that declares about 100 exchanges and queues in RabbitMQ on startup. Some are declared explicitly via Beans, but most of them are declared implicitly via #RabbitListener Annotations.
#Component
#RabbitListener(
bindings = #QueueBinding(key = {"example.routingkey"},
exchange = #Exchange(value = "example.exchange", type = ExchangeTypes.TOPIC),
value = #Queue(name = "example_queue", autoDelete = "true", exclusive = "true")))
public class ExampleListener{
#RabbitHandler
public void handleRequest(final ExampleRequest request) {
System.out.println("got request!");
}
There are quite a lot of these listeners in the whole application.
The services of the application sometimes talk to each other via RabbitMq, so take a example Publisher that publishes a message to the Example Exchange that the above ExampleListener is bound to.
If that publish happens too early in the application lifecycle (but AFTER all the Spring Lifecycle Events are through, so after ApplicationReadyEvent, ContextStartedEvent), the binding of the Example Queue to the Example Exchange has not yet happend and the very first publish and reply chain will fail. In other words, the above Example Listener would not print "got request".
We "fixed" this problem by simply waiting 3 seconds before we start sending any RabbitMq messages to give it time to declare all queues,exchanges and bindings but this seems like a very suboptimal solution.
Does anyone else have some advice on how to fix this problem? It is quite hard to recreate as I would guess that it only occurs with a large amount of queues/exchanges/bindings that RabbitMq can not create fast enough. Forcing Spring to synchronize this creation process and wait for a confirmation by RabbitMq would probably fix this but as I see it, there is no built in way to do this.
Are you using multiple connection factories?
Or are you setting usePublisherConnection on the RabbitTemplate? (which is recommended, especially for a complex application like yours).
Normally, a single connection is used and all users of it will block until the admin has declared all the elements (it is run as a connection listener).
If the template is using a different connection factory, it will not block because a different connection is used.
If that is the case, and you are using the CachingConnectionFactory, you can call createConnection().close() on the consumer connection factory during initialization, before sending any messages. That call will block until all the declarations are done.
What's the difference between SimpleMessageListenerContainer and DirectMessageListenerContainer in Spring AMQP? I checked both of their documentation pages, SimpleMessageListenerContainer has almost no explanation on inner workings, and DirectMessageListenerContainer has the following explanation:
The SimpleMessageListenerContainer is not so simple. Recent changes to the rabbitmq java client has facilitated a much simpler listener container that invokes the listener directly on the rabbit client consumer thread. There is no txSize property - each message is acked (or nacked) individually.
I don't really understand what these mean. It says listener container that invokes the listener directly on the rabbit client consumer thread. If so, then how does SimpleMessageListenerContainer do the invocation?
I wrote a small application and used DirectMessageListenerContainer and just to see the difference, I switched to SimpleMessageListenerContainer, but as far as I can see there was no difference on RabbitMQ side. From Java side the difference was in methods (SimpleMessageListenerContainer provides more) and logs (DirectMessageListenerContainer logged more stuff)
I would like to know the scenarios to use each one of those.
The SMLC has a dedicated thread for each consumer (concurrency) which polls an internal queue. When a new message arrives for a consumer on the client thread, it is put in the internal queue and the consumer thread picks it up and invokes the listener. This was required with early versions of the client to provide multi-threading. With the newer client that is not a problem so we can invoke the listener directly (hence the name).
There are a few other differences aside from txSize.
See Choosing a Container.
In the DirectMessageListenerContainer some of the logic is moved into the AMQP implementation as opposed to ListenerContainer as is SimpleMessageListenerContainer
This is what the Javadocs in SimpleMessageListenerContainer say for setTxSize() -
/**
* Tells the container how many messages to process in a single transaction (if the channel is transactional). For
* best results it should be less than or equal to {#link #setPrefetchCount(int) the prefetch count}. Also affects
* how often acks are sent when using {#link AcknowledgeMode#AUTO} - one ack per txSize. Default is 1.
* #param txSize the transaction size
*/
The client sends an ack every time txSize number of messages are processed. This is controlled in the method
private boolean doReceiveAndExecute(BlockingQueueConsumer consumer) throws Throwable { //NOSONAR
Channel channel = consumer.getChannel();
for (int i = 0; i < this.txSize; i++) {
logger.trace("Waiting for message from consumer.");
Message message = consumer.nextMessage(this.receiveTimeout);
.
.
In the newer implementations, each message is acked on the thread directly and based on the transactional model (Single or publisher confirms) the consumer sends Acknowledgments to Rabbit MQ
I've got a project where we are going to have hundreds (potentially thousands) of queues in rabbit and each of these queues will need to be consumed by a pool of consumers.
In rabbit (using spring-amqp), you have the rabbitlistener annotation which allows me to statically assign the queues this particular consumer(s) will handle.
My question is - with rabbit and spring, is there a clean way for me to grab a section of queues (lets say queues that start with a-c) and then also listen for any queues that are created while the consumer is running.
Example (at start):
ant-queue
apple-queue
cat-queue
While consumer is running:
Add bat-queue
Here is the (very simple) code I currently have:
#Component
public class MessageConsumer {
public MessageConsumer() {
// ideally grab a section of queues here, initialize a parameter and give to the rabbitlistener annotation
}
#RabbitListener(queues= {"ant-queue", "apple-queue", "cat-queue"})
public void processQueues(String messageAsJson) {
< how do I update the queues declared in rabbit listener above ? >
}
}
Edit:
I should add - I've gone through the spring amqp documentation I found online and I haven't found anything outside of statically (either hardcoded or via properties) declaring the queues
Inject (#Autowired or otherwise) the RabbitListenerEndpointRegistry.
Get a reference to the listener container (use the id attribute on the annotation to give it a known id) (registry.getListenerContainer(id)).
Cast the container to an AbstractMessageListenerContainer and call addQueues() or addQueueNames().
Note that is more efficient to use a DirectMessageListenerContainer when adding queues dynamically; with a SimpleMessageListenerContainer the consumer(s) are stopped and restarted. With the direct container, each queue gets its own consumer(s).
See Choosing a container.
Given a basic MessageListener implementation which consumes messages from a RabbitMQ queue, how can I send the message to different dead-letter-queues based on the type of exceptions that could be thrown while processing it?
The queue were the messages are originally published has the x-dead-letter-exchange and x-dead-letter-routing-key set on it, but this is not enough in my case.
In case it matters, my application is using Spring 4 and Spring Amqp.
As far as I understand RabbitMQ documentation and Spring AMQP, it is not possible to send a message to different DLQs based on conditions from inside the code. The reason I say this is that my understanding from debugging the code is that when a message has to be send to a DLQ, the code doesn't specify the exchange or the routing key and RabbitMQ uses the ones defined when the message was published.
So, the only solution I found is to implement something like this:
try {
try_to_do_useful_stuff(message);
} catch (BusinessException e) {
resend_the_message_to_business_dlq(message);
}
This way, if a business exception is thrown, then the message is manually send to the business DLQ. Of course, some details get lost, like originating queue, but this is not an issue if they're not used.
When a non-business exception is thrown then the standard path is followed: retry (if configured) and then route to the defined DLQ.
I'm using the following DefaultMessageListenerContainer to create a durable subscription to get messages even in downtimes which really works well.
#Bean
ConnectionFactory connectionFactory() {
SingleConnectionFactory connectionFactory = new SingleConnectionFactory(new ActiveMQConnectionFactory(
AMQ_BROKER_URL));
connectionFactory.setClientId(CLIENT_ID);
return connectionFactory;
}
#Bean
DefaultMessageListenerContainer container(final MessageListener messageListener,
final ConnectionFactory connectionFactory) {
return new DefaultMessageListenerContainer() {
{
setMessageListener(messageListener);
setConnectionFactory(connectionFactory);
setDestinationName(JMS_TARGET);
setPubSubDomain(true);
setSessionTransacted(true);
setSubscriptionDurable(true);
setDurableSubscriptionName(SUBSCRIPTION_ID);
setConcurrentConsumers(1);
}
};
}
The question is: What is the best way to remove the subscription when I don't need it anymore? Would it even be possible to temporarily remove the subscription (and miss some messages), but enable it later again?
The only way which worked so far, was to shutdown the DMLC and call unsubscribe afterwards.
dmlc.shutdown();
Connection connection = connectionFactory.createConnection();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
session.unsubscribe(SUBSCRIPTION_ID);
Is that a sensible solution? How could the subscription be reinitiated?
I've already seen this answer but I really don't know how to do this? Or would it be even better to subscribe and unsubscribe in a total different way?
Before I suggest my answer, allow me a word of caution: temporarily removing a durable subscription sounds a little bit like you don't really need a durable subscription. Durable subscriptions are for Consumers which need to receive messages that were sent while they are offline. If you need the messages only sometimes (say, when your consumer is connected), a non-durable topic is the item of choice.
Note that this only makes sense for Topics (1:n communication, pub/sub). Queues work slightly different as they are used for 1:1 communication (advanced techniques like load balancing set aside for now).
Durable topics incur a notable resource overhead on the message broker and removing and recreating subscriptions might lead to expensive initialization over and over again.
Now if you really want to (temporarily) unsubscribe from your durable topic (also works for "normal" topics), you'll have to extend DefaultMessageListenerContainer:
public class MyContainer extends DefaultMessageListenerContainer {
public Session retrieveSession() {
return getSession(); // this is protected, so we wrap it (could also make getSession public)
}
}
When you instantiate MyContainer (instead of DefaultMessageListenerContainer) in the container method, make sure to store the reference:
protected MyContainer listenerContainer;
...
listenerContainer = new MyContainer(...);
return listenerContainer;
You can then just call listenerContainer.retrieveSession().unsubscribe(...) to unsubscribe.
Please also note that you may only call this after all the session's consumers to that topic are closed and there are no messages in transit (direct quote from the documentation: "It is erroneous for a client to delete a durable subscription while there is an active (not closed) consumer for the subscription, or while a consumed message is part of a pending transaction or has not been acknowledged in the session."