I have a Spring Boot service that needs to listen to messages across multiple RMQ vhosts. So far I only need to consume messages, though in the short future I might need to publish messages to a third vhost. For this reason I've moved towards explicit configuration of the RMQ connection factory - one connection factory per vhost.
Looking at the documentation the PooledChannelConnectionFactory fits my needs. I do not need strict ordering of the messages, correlated publisher confirms, or caching connections to a single vhost. Everything I do with rabbit is take a message and update an entry in the database.
#Bean
PooledChannelConnectionFactory pcf() throws Exception {
ConnectionFactory rabbitConnectionFactory = new ConnectionFactory();
//Set the credentials
PooledChannelConnectionFactory pcf = new PooledChannelConnectionFactory(rabbitConnectionFactory);
pcf.setPoolConfigurer((pool, tx) -> {
if (tx) {
// configure the transactional pool
}
else {
// configure the non-transactional pool
}
});
return pcf;
}
What I require assistance with is understanding what the difference between the transactional and non-transactional pool is. My understanding of RMQ and AMQP is that everything is async unless you build RPC semantics ontop of it (reply queues and exchanges). Because of that how can this channel pool have transactional properties?
My current approach is to disable one of the configurations by setting min/max to 0, and set the other to a min/max of 1. I do not expect to have extreme volume through the service, and I expect to be horizontally scaling the application which will scale the capacity to consume messages. Is there anything else I should be considering?
The pools are independent; you won't get any transactional channels as long as you don't use a RabbitTemplate with channelTransacted set to true, so there is no need to worry about configuring the pool like that.
Transactions can be used, for example, to atomically send a series of messages (all sent or none sent if the transaction is rolled back). Useful if you are synchronizing with some other transaction, such as JDBC.
Related
I'm trying to understand how transaction works in Spring AMQP. Reading the docs: https://docs.spring.io/spring-amqp/reference/html/#transactions , I know the purpose for enabling transaction in publisher (Best Effort One Phase Commit pattern) but I have no idea why it could be necessary in MessageListener?
Let's take an example:
acknowledgeMode=AUTO
Consume message using #RabbitListener
Insert data into database
Publish message using rabbitTemplate
According to docs: https://docs.spring.io/spring-amqp/reference/html/#acknowledgeMode, if acknowledgeMode is set to AUTO then if any next operation fails, listener will fail too and message will be returned to queue.
Another question is whats the difference between local transaction and external transaction in that case (setting container.setTransactionManager(transactionManager()); or not)?
I would appreciate some clarification :)
Enable transactions in the listener so that any/all downstream RabbitTemplate operations participate in the same transaction.
If there is a failure, the container will rollback the transaction (removing the publishes), nack the message (or messages if batch size is greater than one) and then commit the nacks so the message(s) will be redelivered.
When using an external transaction manager (such as JDBC), the container will synchronize the AMQP transaction with the external transaction.
Downstream templates participate in the transaction regardless of whether it is local (AMQP only) or synchronized.
I've got an issue connecting to an ActiveMQ Artemis cluster (AMQ from Red Hat in fact) through Spring's DefaultJmsListenerContainerFactory.
DefaultMessageListenerContainer makes use of only one connection, regardless of the number of consumers you specify through the concurrency parameter. The problem is that, in the cluster, there are 3 brokers configured at the moment (and, as a dev, I shouldn't care about the topology of the cluster). Since here is only one connection consumers are only listening to one broker.
To solve the issue I disabled the cache (i.e. setCacheLevel(CACHE_NONE) in the factory).
It "solved" the problem because now I can see the connections distributing on all the nodes of the cluster but it's not a good solution because connections are perpetually dropped and recreated and that makes a lot of overhead at the broker side (it makes me think of a Christmas Tree :D).
Can you guys tell me what's the correct approach to handle this?
I trie using a JmsPoolConnectionFactory, but I didn't get any good results till now. I still have only one connection.
I'm using Spring Boot 2.7.4 with Artemis Starter.
You can find below a code snippet of the actual config.
(Side note, I don't use Spring autoconfig because i need to be able to switch between ActiveMQ Artemis and the old ActiveMQ "Classic" implementation).
#Bean
DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(connectionFactory());
factory.setDestinationResolver(destinationResolver());
factory.setSessionTransacted(true);
factory.setConcurrency(config.getConcurrency());
//Set this to allow load balancing of connections to all members of the cluster
factory.setCacheLevel(DefaultMessageListenerContainer.CACHE_NONE);
final ExponentialBackOff backOff = new ExponentialBackOff(
config.getRetry().getInitialInterval(), config.getRetry().getMultiplier());
backOff.setMaxInterval(config.getRetry().getMaxDuration());
factory.setBackOff(backOff);
return factory;
}
ConnectionFactory connectionFactory() {
return new ActiveMQJMSConnectionFactory(
config.getUrl(), config.getUser(), config.getPassword());
}
DestinationResolver destinationResolver() {
final ActiveMQQueue activeMQQueue = new ActiveMQQueue(config.getQueue());
return (session, destinationName, pubSubDomain) -> activeMQQueue;
}
#JmsListener(destination = "${slp.amq.queue}")
public void processLog(String log) {
final SecurityLog securityLog = SecurityLog.parse(log);
fileWriter.write(securityLog);
logsCountByApplicationId.increment(securityLog.getApplicationId());
if (elasticClient != null) {
elasticClient.write(securityLog);
}
}
The connection URL is:
(tcp://broker1:port,tcp://broker2:port,tcp://broker3:port)?useTopologyForLoadBalancing=true
The cluster can be configured so that any consumer on any node can consume messages sent to any node. Therefore, you shouldn't strictly need to "target all nodes" of the cluster with your consumer. Message redistribution and re-routing in the cluster should be transparent to your application. As you said, as a developer you shouldn't care about the topology of the cluster.
That said, the goal of clustering is to increase overall message throughput (i.e. performance) via horizontal scaling. Furthermore, every node in the cluster should ideally have sufficient producers and consumers so that messages aren't being redistributed or re-routed between cluster nodes as that's not optimal for performance. If you're in a situation where you have just a few consumers connected to your cluster then it's likely you don't actually need a cluster in the first place. A single ActiveMQ Artemis broker can handle millions of messages per second in certain use-cases.
we have a larger multi service java spring app that declares about 100 exchanges and queues in RabbitMQ on startup. Some are declared explicitly via Beans, but most of them are declared implicitly via #RabbitListener Annotations.
#Component
#RabbitListener(
bindings = #QueueBinding(key = {"example.routingkey"},
exchange = #Exchange(value = "example.exchange", type = ExchangeTypes.TOPIC),
value = #Queue(name = "example_queue", autoDelete = "true", exclusive = "true")))
public class ExampleListener{
#RabbitHandler
public void handleRequest(final ExampleRequest request) {
System.out.println("got request!");
}
There are quite a lot of these listeners in the whole application.
The services of the application sometimes talk to each other via RabbitMq, so take a example Publisher that publishes a message to the Example Exchange that the above ExampleListener is bound to.
If that publish happens too early in the application lifecycle (but AFTER all the Spring Lifecycle Events are through, so after ApplicationReadyEvent, ContextStartedEvent), the binding of the Example Queue to the Example Exchange has not yet happend and the very first publish and reply chain will fail. In other words, the above Example Listener would not print "got request".
We "fixed" this problem by simply waiting 3 seconds before we start sending any RabbitMq messages to give it time to declare all queues,exchanges and bindings but this seems like a very suboptimal solution.
Does anyone else have some advice on how to fix this problem? It is quite hard to recreate as I would guess that it only occurs with a large amount of queues/exchanges/bindings that RabbitMq can not create fast enough. Forcing Spring to synchronize this creation process and wait for a confirmation by RabbitMq would probably fix this but as I see it, there is no built in way to do this.
Are you using multiple connection factories?
Or are you setting usePublisherConnection on the RabbitTemplate? (which is recommended, especially for a complex application like yours).
Normally, a single connection is used and all users of it will block until the admin has declared all the elements (it is run as a connection listener).
If the template is using a different connection factory, it will not block because a different connection is used.
If that is the case, and you are using the CachingConnectionFactory, you can call createConnection().close() on the consumer connection factory during initialization, before sending any messages. That call will block until all the declarations are done.
It´s more of a conceptual question: I currently have a working activemq queue which is consumed by a Java Spring application. Now I want the queue not to permanently delete the messages until the Java app tells it the message has been correctly saved in DB. After reading documentation I get I have to do it transactional and usa the commit() / rollback() methods. Correct me if I'm wrong here.
My problem comes with every example I find over the internet telling me to configure the app to work this or that way, but my nose tells me I should instead be setting up the queue itself to work the way I want. And I can't find the way to do it.
Otherwise, is the queue just working in different ways depending on how the consumer application is configured to work? What am I getting wrong?
Thanks in advance
The queue it self is not aware of any transactional system but you can pass the 1st parameter boolean to true to create a transactional session but i propose the INDIVIDUAL_ACKNOWLEDGE when creating a session because you can manage messages one by one. Can be set on spring jms DefaultMessageListenerContainer .
ActiveMQSession.INDIVIDUAL_ACKNOWLEDGE
And calling this method to ack a message, unless the method is not called the message is considered as dispatched but not ack.
ActiveMQTextMessage.acknowledge();
UPDATE:
ActiveMQSession.INDIVIDUAL_ACKNOWLEDGE can be used like this :
onMessage(ActiveMQTextMessage message)
try {
do some stuff in the database
jdbc.commit(); (unless auto-commit is enabled on the JDBC)
message.acknowledge();
}
catch (Exception e) {
}
There are 2 kinds of transaction support in ActiveMQ.
JMS transactions - the commit() / rollback() methods on a Session (which is like doing commit() / rollback() on a JDBC connection)
XA Transactions - where the XASession acts as an XAResource by communicating with the Message Broker, rather like a JDBC Connection takes place in an XA transaction by communicating with the database.
http://activemq.apache.org/how-do-transactions-work.html
Should I use XA transactions (two phase commit?)
A common use of JMS is to consume messages from a queue or topic, process them using a database or EJB, then acknowledge / commit the message.
If you are using more than one resource; e.g. reading a JMS message and writing to a database, you really should use XA - its purpose is to provide atomic transactions for multiple transactional resources. For example there is a small window from when you complete updating the database and your changes are committed up to the point at which you commit/acknowledge the message; if there is a network/hardware/process failure inside that window, the message will be redelivered and you may end up processing duplicates.
http://activemq.apache.org/should-i-use-xa.html
I'm using the following DefaultMessageListenerContainer to create a durable subscription to get messages even in downtimes which really works well.
#Bean
ConnectionFactory connectionFactory() {
SingleConnectionFactory connectionFactory = new SingleConnectionFactory(new ActiveMQConnectionFactory(
AMQ_BROKER_URL));
connectionFactory.setClientId(CLIENT_ID);
return connectionFactory;
}
#Bean
DefaultMessageListenerContainer container(final MessageListener messageListener,
final ConnectionFactory connectionFactory) {
return new DefaultMessageListenerContainer() {
{
setMessageListener(messageListener);
setConnectionFactory(connectionFactory);
setDestinationName(JMS_TARGET);
setPubSubDomain(true);
setSessionTransacted(true);
setSubscriptionDurable(true);
setDurableSubscriptionName(SUBSCRIPTION_ID);
setConcurrentConsumers(1);
}
};
}
The question is: What is the best way to remove the subscription when I don't need it anymore? Would it even be possible to temporarily remove the subscription (and miss some messages), but enable it later again?
The only way which worked so far, was to shutdown the DMLC and call unsubscribe afterwards.
dmlc.shutdown();
Connection connection = connectionFactory.createConnection();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
session.unsubscribe(SUBSCRIPTION_ID);
Is that a sensible solution? How could the subscription be reinitiated?
I've already seen this answer but I really don't know how to do this? Or would it be even better to subscribe and unsubscribe in a total different way?
Before I suggest my answer, allow me a word of caution: temporarily removing a durable subscription sounds a little bit like you don't really need a durable subscription. Durable subscriptions are for Consumers which need to receive messages that were sent while they are offline. If you need the messages only sometimes (say, when your consumer is connected), a non-durable topic is the item of choice.
Note that this only makes sense for Topics (1:n communication, pub/sub). Queues work slightly different as they are used for 1:1 communication (advanced techniques like load balancing set aside for now).
Durable topics incur a notable resource overhead on the message broker and removing and recreating subscriptions might lead to expensive initialization over and over again.
Now if you really want to (temporarily) unsubscribe from your durable topic (also works for "normal" topics), you'll have to extend DefaultMessageListenerContainer:
public class MyContainer extends DefaultMessageListenerContainer {
public Session retrieveSession() {
return getSession(); // this is protected, so we wrap it (could also make getSession public)
}
}
When you instantiate MyContainer (instead of DefaultMessageListenerContainer) in the container method, make sure to store the reference:
protected MyContainer listenerContainer;
...
listenerContainer = new MyContainer(...);
return listenerContainer;
You can then just call listenerContainer.retrieveSession().unsubscribe(...) to unsubscribe.
Please also note that you may only call this after all the session's consumers to that topic are closed and there are no messages in transit (direct quote from the documentation: "It is erroneous for a client to delete a durable subscription while there is an active (not closed) consumer for the subscription, or while a consumed message is part of a pending transaction or has not been acknowledged in the session."