I'm creating application using Spring Boot with RabbitMQ.
I've created configuration for Rabbit like this:
#Configuration
public class RabbitConfiguration {
public static final String RESEND_DISPOSAL_QUEUE = "RESEND_DISPOSAL";
#Bean
public Queue resendDisposalQueue() {
return new Queue(RESEND_DISPOSAL_QUEUE, true);
}
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory (ConnectionFactory connectionFactoryr) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
return factory;
}
#Bean
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory){
return new RabbitTemplate(connectionFactory);
}
}
Also I've created listener for Rabbit messages like this:
#RabbitListener(queues = RESEND_DISPOSAL_QUEUE)
public void getResendDisposalPayload(String messageBody){
LOGGER.info("[getResendDisposalPayload] message = {}", messageBody);
// And there is some business logic
}
All works pretty good, but there is one problem.
When I got exception in method getResendDisposalPayload which listens RESEND_DISPOSAL_QUEUE queue (for example temporary problem with database) Rabbit starts resend last not processed message without any delay. It produces a big amount of log and for some reason uncomfortable for my system.
As I've read in this article https://www.baeldung.com/spring-amqp-exponential-backoff "While using a Dead Letter Queue is a standard way to deal with failed messages".
In order to use this pattern I've to create RetryOperationsInterceptor which defines count attempt to deliver message and delay between attempts.
For example:
#Bean
public RetryOperationsInterceptor retryInterceptor() {
return RetryInterceptorBuilder.stateless()
.backOffOptions(1000, 3.0, 10000)
.maxAttempts(3)
.recoverer(messageRecoverer)
.build();
}
It sounds very good but only one problem: I can't define infinity attempt amount in options maxAttempts.
After maxAttempts I have to save somewhere broken message and deal with it in the future. It demands some extra code.
The question is: Is there any way to configure Rabbit to infinity resend broken messages with some delay, say with one second delay?
Rabbit starts resend last not processed message without any delay
That's how redelivery works: it re-push the same message again and again, until you ack it manually or drop altogether. There is no delay in between redeliveries just because an new message is not pulled from the queue until something is done with this one.
I can't define infinity attempt amount in options maxAttempts
Have you tried an Integer.MAX_VALUE? Pretty decent number of attempts.
The other way is to use a Delayed Exchange: https://docs.spring.io/spring-amqp/docs/current/reference/html/#delayed-message-exchange.
You can configure that retry with a RepublishMessageRecoverer to publish into a your original queue back after some attempts are exhausted: https://docs.spring.io/spring-amqp/docs/current/reference/html/#async-listeners
We are develop a micro-service system that use ActiveMQ Artemis as the communication method between service. Since the requirement ask to be able to stop the listeners at runtime, we can not use #JmsListener provide by spring-artemis. After digging the internet and finding out that spring use MessageListenerContainer behind the scence, we come up with the idea of maintain a list of MessageListenerContainer our self.
#Bean(name = "commandJmsListenerContainerFactory")
public DefaultJmsListenerContainerFactory commandJmsListenerContainerFactory(
DefaultJmsListenerContainerFactoryConfigurer configurer) {
var factory = new DefaultJmsListenerContainerFactory();
configurer.configure(factory, connectionFactory);
factory.setPubSubDomain(false);
return factory;
}
// Use
private Map<String, DefaultMessageListenerContainer> commandQueue;
public void subscribeToCommandQueue(String queueName, CommandListener<?> command) {
commandQueue.computeIfAbsent(queueName, key -> {
var endPoint = new SimpleJmsListenerEndpoint();
endPoint.setDestination(queueName);
endPoint.setMessageListener(message -> {
try {
var body = message.getBody(String.class);
command.execute(commandMessageConverter.deserialize(body));
} catch (JMSException e) {
throw new RuntimeException("Error while process message for queue: " + queueName, e);
}
});
var container = commandJmsListenerContainerFactory.createListenerContainer(endPoint);
// https://stackoverflow.com/questions/44555106/defaultmessagelistenercontainer-not-reading-messages-from-ibm-mq
// for Every object of Spring classes that implement InitializingBean created manually, we need to call afterPropertiesSet to make the object "work"
container.afterPropertiesSet();
container.start();
return container;
});
}
public void start() {
commandQueue = new ConcurrentHashMap<>();
}
public void stop() {
commandQueue.values().forEach(DefaultMessageListenerContainer::destroy);
commandQueue.clear();
}
While testing, I notice that after we destroy all the listener by calling stop() , the queue and the address in the Artemis console are deleted too. It isn't the case for durable subscription.
#Bean(name = "eventJmsListenerContainerFactory")
public DefaultJmsListenerContainerFactory eventJmsListenerContainerFactory(
CachingConnectionFactory cachingConnectionFactory,
DefaultJmsListenerContainerFactoryConfigurer configurer) {
cachingConnectionFactory.setClientId(UUID.randomUUID().toString());
var factory = new DefaultJmsListenerContainerFactory();
configurer.configure(factory, cachingConnectionFactory);
factory.setPubSubDomain(true);
factory.setSubscriptionDurable(true);
return factory;
}
// usage is the same as the first block code, except we store multicast subscriptions in another map
private Map<String, DefaultMessageListenerContainer> eventTopic;
After running the unit tests and destroying all the listeners of two maps, only the test-event-topic address and its queues were kept, the test-command-queue was deleted. Why both the queues behave differently?
Also, what is the correct behavior? We afraid the auto deletion will remove messages that aren't sent in the queue yet. On the other hand, new queue under test-event-topic keep being created if we run the test again and again. I think it is because of the line cachingConnectionFactory.setClientId(UUID.randomUUID().toString()); . But for durable subscription, not setting clientId result in error.
The connection factory used in the app is an CachingConnectionFactory created by spring-artemis
By default the broker will auto-create addresses and queues as required when a message is sent or a consumer is created by the core JMS client. These resources will also be auto-deleted by default when they're no longer needed (i.e. when a queue has no consumers and messages or when an address no longer has any queues bound to it). This is controlled by these settings in broker.xml which are discussed in the documentation:
auto-create-queues
auto-delete-queues
auto-create-addresses
auto-delete-addresses
To be clear, auto-deletion should not cause any message loss by default as queues should only be deleted when they have 0 consumers and 0 messages. However, you can always set auto-deletion to false to be 100% safe.
Queues representing durable JMS topic subscriptions won't be deleted as they are meant to stay and gather messages while the consumer is offline. In other words, a durable topic subscription will remain if the client using the subscription is shutdown without first explicitly removing the subscription. That's the whole point of durable subscriptions - they are durable. Any client can use a durable topic subscription if it connects with the same client ID and uses the same subscription name. However, unless the durable subscription is a "shared" durable subscription then only one client at a time can be connected to it. Shared durable topic subscriptions were added in JMS 2.0.
I have a #Repository reading from a real-time data source. I am making the data available using Flux.create() { sink->sink.next() }
A #Service is doing the following;
#Autowired MyRepository myRepository;
#PostConstruct() public void startUp() {
ConnectableFlux<Object> cf = myRepository.flux.publish();
cf.subscribe(System.out::println);
cf.connect();
}
That works and prints the data, but I do not get "Netty started" in the logs and #Controllers do not respond. If I omit cf.connect(), Netty starts. So I assume that cf.connect() is blocking Netty.
Ideally, I want the subscription to auto-start. Is using connect() in #PostConstuct too early? Should I listen for a "Netty Started" event, then connect(), or is my subscription just plain wrong?
Edit: If connect is run within a deamon Thread, Netty does start and the subscription works.
Putting #EnableAsync on the Spring Boot's main Application class and #Async on the method above seemed to work.
Edit:
I found a better solution here Connectable Flux blocks on toIterator.forEach, while Flux does not. #1549. So my code now looks like this;
Flux<MyClass> flux = Flux.create(
sink -> {
while(condition) {
sink.next(nextValueFromDataSource);
}
sink.complete();
}
)
.publish()
.autoConnect(1);
Also because of autoConnect(1), there is no need for #Async.
In my spring boot app, I have the following websocket controller:
#Controller
public class QuoteController {
#Autowired
private RabbitTemplate rabbitTemplate;
#SubscribeMapping("/quote/{symbol}")
public void singleQuote(#DestinationVariable("symbol") String symbol, Principal user) {
System.out.println("Salam");
}
}
I would like the clients to be able to subscribe to a channel, only if there exists an endpoint like above for that channel. Apparently (based on the logs in the browser) clients can subscribe to any arbitrary channel.
(and btw, isn't this a bad practice to let clients to subscribe to any channel? Specially security wise. Malicious clients can subscribe to millions of channel and cause the server to slow down, or subscribe to channels that are for internal use only)
Note that I'd like multiple message listeners to handle successive messages from the topic concurrently. In addition I'd like each message listener to operate transactionally so that a processing failure in a given message listener would result in that listener's message remaining on the topic.
The spring DefaultMessageListenerContainer seems to support concurrency for JMS queues only.
Do I need to instantiate multiple DefaultMessageListenerContainers?
If time flows down the vertical axis:
ListenerA reads msg 1 ListenerB reads msg 2 ListenerC reads msg 3
ListenerA reads msg 4 ListenerB reads msg 5 ListenerC reads msg 6
ListenerA reads msg 7 ListenerB reads msg 8 ListenerC reads msg 9
ListenerA reads msg 10 ListenerB reads msg 11 ListenerC reads msg 12
...
UPDATE:
Thanks for your feedback #T.Rob and #skaffman.
What I ended up doing is creating multiple DefaultMessageListenerContainers with concurrency=1 and then putting logic in the message listener so that only one thread would process a given message id.
You don't want multiple DefaultMessageListenerContainer instances, no, but you do need to configure the DefaultMessageListenerContainer to be concurrent, using the concurrentConsumers property:
Specify the number of concurrent
consumers to create. Default is 1.
Specifying a higher value for this
setting will increase the standard
level of scheduled concurrent
consumers at runtime: This is
effectively the minimum number of
concurrent consumers which will be
scheduled at any given time. This is a
static setting; for dynamic scaling,
consider specifying the
"maxConcurrentConsumers" setting
instead.
Raising the number of concurrent
consumers is recommendable in order to
scale the consumption of messages
coming in from a queue. However, note
that any ordering guarantees are lost
once multiple consumers are
registered. In general, stick with 1
consumer for low-volume queues.
However, there's big warning at the bottom:
Do not raise the number of concurrent consumers for a topic.
This would lead to concurrent
consumption of the same message, which
is hardly ever desirable.
This is interesting, and makes sense when you think about it. The same would occur if you had multiple DefaultMessageListenerContainer instances.
I think perhaps you need to rethink your design, although I'm not sure what I'd suggest. Concurrent consumption of pub/sub messages seems like a perfectly reasonable thing to do, but how to avoid getting the same message delivered to all of your consumers at the same time?
At least in ActiveMQ what you want is totally supported, his name is VirtualTopic
The concept is:
You create a VirtualTopic (Simply creating a Topic using the prefix VirtualTopic. ) eg. VirtualTopic.Color
Create a consumer subscribing to this VirtualTopic matching this pattern Consumer.<clientName>.VirtualTopic.<topicName> eg. Consumer.client1.VirtualTopic.Color, doing it, Activemq will create a queue with that name and that queue will subscribe to VirtualTopic.Color then every message published to this Virtual Topic will be delivered to client1 queue, note that it works like rabbitmq exchanges.
You are done, now you can consume client1 queue like every queue, with many consumers, DLQ, customized redelivery policy, etc.
At this point I think you understood that you can create client2, client3 and how many subscribers you want, all of them will receive a copy of the message published to VirtualTopic.Color
Here the code
#Component
public class ColorReceiver {
private static final Logger LOGGER = LoggerFactory.getLogger(MailReceiver.class);
#Autowired
private JmsTemplate jmsTemplate;
// simply generating data to the topic
long id=0;
#Scheduled(fixedDelay = 500)
public void postMail() throws JMSException, IOException {
final Color colorName = new Color[]{Color.BLUE, Color.RED, Color.WHITE}[new Random().nextInt(3)];
final Color color = new Color(++id, colorName.getName());
final ActiveMQObjectMessage message = new ActiveMQObjectMessage();
message.setObject(color);
message.setProperty("color", color.getName());
LOGGER.info("status=color-post, color={}", color);
jmsTemplate.convertAndSend(new ActiveMQTopic("VirtualTopic.color"), message);
}
/**
* Listen all colors messages
*/
#JmsListener(
destination = "Consumer.client1.VirtualTopic.color", containerFactory = "colorContainer"
selector = "color <> 'RED'"
)
public void genericReceiveMessage(Color color) throws InterruptedException {
LOGGER.info("status=GEN-color-receiver, color={}", color);
}
/**
* Listen only red colors messages
*
* the destination ClientId have not necessary exists (it means that his name can be a fancy name), the unique requirement is that
* the containers clientId need to be different between each other
*/
#JmsListener(
// destination = "Consumer.redColorContainer.VirtualTopic.color",
destination = "Consumer.client1.VirtualTopic.color",
containerFactory = "redColorContainer", selector = "color='RED'"
)
public void receiveMessage(ObjectMessage message) throws InterruptedException, JMSException {
LOGGER.info("status=RED-color-receiver, color={}", message.getObject());
}
/**
* Listen all colors messages
*/
#JmsListener(
destination = "Consumer.client2.VirtualTopic.color", containerFactory = "colorContainer"
)
public void genericReceiveMessage2(Color color) throws InterruptedException {
LOGGER.info("status=GEN-color-receiver-2, color={}", color);
}
}
#SpringBootApplication
#EnableJms
#EnableScheduling
#Configuration
public class Config {
/**
* Each #JmsListener declaration need a different containerFactory because ActiveMQ requires different
* clientIds per consumer pool (as two #JmsListener above, or two application instances)
*
*/
#Bean
public JmsListenerContainerFactory<?> colorContainer(ActiveMQConnectionFactory connectionFactory,
DefaultJmsListenerContainerFactoryConfigurer configurer) {
final DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setConcurrency("1-5");
configurer.configure(factory, connectionFactory);
// container.setClientId("aId..."); lets spring generate a random ID
return factory;
}
#Bean
public JmsListenerContainerFactory<?> redColorContainer(ActiveMQConnectionFactory connectionFactory,
DefaultJmsListenerContainerFactoryConfigurer configurer) {
// necessary when post serializable objects (you can set it at application.properties)
connectionFactory.setTrustedPackages(Arrays.asList(Color.class.getPackage().getName()));
final DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setConcurrency("1-2");
configurer.configure(factory, connectionFactory);
return factory;
}
}
public class Color implements Serializable {
public static final Color WHITE = new Color("WHITE");
public static final Color BLUE = new Color("BLUE");
public static final Color RED = new Color("RED");
private String name;
private long id;
// CONSTRUCTORS, GETTERS AND SETTERS
}
Multiple Consumers Allowed on the Same Topic Subscription in JMS 2.0, while this was not the case with JMS 1.1. Please refer:
https://www.oracle.com/technetwork/articles/java/jms2messaging-1954190.html
This is one of those occasions where the differences in transport providers bubble up through the abstraction of JMS. JMS wants to provide a copy of the message for each subscriber on a topic. But the behavior that you want is really that of a queue. I suspect that there are other requirements driving this to a pub/sub solution which were not described - for example other things need to subscribe to the same topic independent of your app.
If I were to do this in WebSphere MQ the solution would be to create an administrative subscription which would result in a single copy of each message on the given topic to be placed onto a queue. Then your multiple subscribers could compete for messages on that queue. This way your app could have multiple threads among which the messages are distributed, and at the same time other subscribers independent of this application could dynamically (un)subscribe to the same topic.
Unfortunately, there's no generic JMS-portable way of doing this. You are dependent on the transport provider's implementation to a great degree. The only one of these I can speak to is WebSphere MQ but I'm sure other transports support this in one way or another and to varying degrees if you are creative.
Here's a possibility:
1) create only one DMLC configured with the bean and method to handle the incoming message. Set its concurrency to 1.
2) Configure a task executor with its #threads equal to the concurrency you desire. Create an object pool for objects which are actually supposed to process a message. Give a reference of task executor and object pool to the bean you configured in #1. Object pool is useful if the actual message processing bean is not thread-safe.
3) For an incoming message, the bean in DMLC creates a custom Runnable, points it to the message and the object pool, and gives it to task executor.
4) The run method of Runnable gets a bean from the object pool and calls its 'process' method with the message given.
#4 can be managed with a proxy and the object pool to make it easier.
I haven't tried this solution yet, but it seems to fit the bill. Note that this solution is not as robust as EJB MDB. Spring e.g. will not discard an object from the pool if it throws a RuntimeException.
Creating a custom task executor seemingly solved the issue for me, w/o duplicate processing:
#Configuration
class BeanConfig {
#Bean(destroyMethod = "shutdown")
public ThreadPoolTaskExecutor topicExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setAllowCoreThreadTimeOut(true);
executor.setKeepAliveSeconds(300);
executor.setCorePoolSize(4);
executor.setQueueCapacity(0);
executor.setThreadNamePrefix("TOPIC-");
return executor;
}
#Bean
JmsListenerContainerFactory<?> topicListenerFactory(ConnectionFactory connectionFactory, DefaultJmsListenerContainerFactoryConfigurer configurer, #Qualifier("topicExecutor") Executor topicExecutor) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setPubSubDomain(true);
configurer.configure(factory, connectionFactory);
factory.setPubSubDomain(true);
factory.setSessionTransacted(false);
factory.setSubscriptionDurable(false);
factory.setTaskExecutor(topicExecutor);
return factory;
}
}
class MyBean {
#JmsListener(destination = "MYTOPIC", containerFactory = "topicListenerFactory", concurrency = "1")
public void receiveTopicMessage(SomeTopicMessage message) {}
}
I've run into the same problem. I'm currently investigating RabbitMQ, which seems to offer a perfect solution in a design pattern they call "work queues." More info here: http://www.rabbitmq.com/tutorials/tutorial-two-java.html
If you're not totally tied to JMS you might look into this. There might also be a JMS to AMQP bridge, but that might start to look hacky.
I'm having some fun (read: difficulties) getting RabbitMQ installed and running on my Mac but think I'm close to having it working, I will post back if I'm able to solve this.
on server.xml configs:
so , in maxSessions you can identify the number of sessions you want.
Came across this question. My configuration is :
Create a bean with id="DefaultListenerContainer", add property name="concurrentConsumers" value="10" and property name="maxConcurrentConsumers" value ="50".
Works fine, so far. I printed the thread id and verified that multiple threads do get created and also reused.