We Keep Coding

Spring AMQP #RabbitListener is not ready to receive messages on #ApplicationReadyEvent. Queues/Bindings declared too slow?

we have a larger multi service java spring app that declares about 100 exchanges and queues in RabbitMQ on startup. Some are declared explicitly via Beans, but most of them are declared implicitly via #RabbitListener Annotations.
#Component
#RabbitListener(
bindings = #QueueBinding(key = {"example.routingkey"},
exchange = #Exchange(value = "example.exchange", type = ExchangeTypes.TOPIC),
value = #Queue(name = "example_queue", autoDelete = "true", exclusive = "true")))
public class ExampleListener{
#RabbitHandler
public void handleRequest(final ExampleRequest request) {
System.out.println("got request!");
}
There are quite a lot of these listeners in the whole application.
The services of the application sometimes talk to each other via RabbitMq, so take a example Publisher that publishes a message to the Example Exchange that the above ExampleListener is bound to.
If that publish happens too early in the application lifecycle (but AFTER all the Spring Lifecycle Events are through, so after ApplicationReadyEvent, ContextStartedEvent), the binding of the Example Queue to the Example Exchange has not yet happend and the very first publish and reply chain will fail. In other words, the above Example Listener would not print "got request".
We "fixed" this problem by simply waiting 3 seconds before we start sending any RabbitMq messages to give it time to declare all queues,exchanges and bindings but this seems like a very suboptimal solution.
Does anyone else have some advice on how to fix this problem? It is quite hard to recreate as I would guess that it only occurs with a large amount of queues/exchanges/bindings that RabbitMq can not create fast enough. Forcing Spring to synchronize this creation process and wait for a confirmation by RabbitMq would probably fix this but as I see it, there is no built in way to do this.

Are you using multiple connection factories?
Or are you setting usePublisherConnection on the RabbitTemplate? (which is recommended, especially for a complex application like yours).
Normally, a single connection is used and all users of it will block until the admin has declared all the elements (it is run as a connection listener).
If the template is using a different connection factory, it will not block because a different connection is used.
If that is the case, and you are using the CachingConnectionFactory, you can call createConnection().close() on the consumer connection factory during initialization, before sending any messages. That call will block until all the declarations are done.

#async vs message queue difference

I have a spring boot project, deploying in two servers and using nginx. One method in the project will do:
set some key-values in redis
insert something in db
After 1, I want to do 2 in async way.
One solution is to let doDB() be a springboot #async method:
Class A {
public void ***() {
doRedis() // 1.set some key-values in redis
doDB() // 2.insert something in db
}
}
Class B {
#async
doDB()
}
Another solution is to send message to MQ:
Class A {
public void ***() {
doRedis() // 1.set some key-values in redis
sendMessage()
}
}
Class B {
onMessage(){
doDB()
}
}
If Class A and B are both in the spring boot project, just deploying this project in two servers. I think using #async is enough, there is no need to use MQ to achieve the async way because there is no difference between server one to do Class B doDB() and server two to do Class B doDB(). If class B is in another project, then using MQ is good because it's decoupling for project one doing redis work and project two doing db work.
Is it right? Thanks!

Basically, you are right, if it is going to be in the same application within the same server, no need for MQ because async is already has a queue. But there are some key points you should be decided on even if in the same application
if you care about ordering message queue is more meaningful, you can use async in this case too but you have to configure the thread pool to use only one thread to process async
if you care about losing messages and if some bad things happen before processing messages, you should use an MQ that saves messages to the disk or somewhere to process the rest of the messages later on
if your application gets a lot of requests and you did not carefully set threads in the async thread pool, you could get overflow errors or other problems with using machine resources
Choose within capabilities within your application, do not over-engineer from day one, you spin up from what you have and what you already tested

Two instances consuming from topic

I have deployed two instances of an application. Both applications runs the same code and consumes from the same topic.
#KafkaListener( offsetReset = OffsetReset.EARLIEST, offsetStrategy = OffsetStrategy.DISABLED )
public class AppConsumer implements ConsumerRebalanceListener, KafkaConsumerAware {
#Topic("topic")
public void consumeAppInfo(#KafkaKey String name, #Body #Nullable String someString) {
...
}
}
I have a problem where only one of the applications consumes the message. The topic has only one partition, partition 0, which i believe is default.
I have tried to add group-id and threads to the KafkaListener. This seems to work sometimes and other time not.
#KafkaListener(groupId="myGroup", threads=10)
What is the simplest solution to getting both applications to consume the same message?

You could not do the group and just give each application a separate consumer id each consumer consumes all messages (unless they are also assigned to a group).
Groups are used for parallel processing of messages each consumer in a group get assigned to a partition for processing messages.
More info => difference between groupid and consumerid in Kafka consumer

In kafka, only one consumer within consumer group is assigned to each partition. If you want to consume the data from the same partition by different applications, you need to specify different consumer groups for each different application / instance.

Spring Kafka - Subscribe new topics during runtime

I'm using the annotation #KafkaListener to consume topics on my application. My issue is that if I create a new topic in kafka but my consumer is already running, it seems the consumer will not pick up the new topic, even if it matches with the topicPattern I'm using. Is there a way to "refresh" the subscribed topics periodically, so that new topics are picked up and rebalanced upon my running consumers?
I'm using Spring Kafka 1.2.2 with Kafka 0.10.2.0.
Regards

You can't dynamically add topics at runtime; you have to stop/start the container to start listening to new topics.
You can #Autowire the KafkaListenerEndpointRegistry and stop/start listeners by id.
You can also stop/start all listeners by calling stop()/start() on the registry itself.

Actually it is possible.
It worked for me with Kafka 1.1.1.
Under the hood Spring uses consumer.subscribe(topicPattern) and now it is totally depends on Kafka lib whether the message will be seen by consumer.
There is consumer config property called metadata.max.age.ms which is 5 mins by default. It basically controls how often client will go to broker for the updates, meaning new topics will not be seen by consumer for up to 5 minutes. You can decrease this value (e.g. 20 seconds) and should see KafkaListener started to pick messages from new topics quicker.

The following way works well for me.
ContainerProperties containerProps = new ContainerProperties("topic1", "topic2");
KafkaMessageListenerContainer<Integer, String> container = createContainer(containerProps);
containerProps.setMessageListener(new MessageListener<Integer, String>() {
#Override
public void onMessage(ConsumerRecord<Integer, String> message) {
logger.info("received: " + message);
}
});
container.setBeanName("testAuto");
container.start();
ref: http://docs.spring.io/spring-kafka/docs/1.0.0.RC1/reference/htmlsingle/
In practical application, I use a ConcurrentMessageListenerContainer instead of single-threaded KafkaMessageListenerContainer.

Which offers better performance: A single JMS topic with a switch statement, or individual topics?

I'm writing a Spring Boot application that will listen on one or more JMS topics. We've chosen ActiveMQ for our JMS implementation.
Our main concerns are performance and scalability. We're expecting potentially thousands of messages to arrive every second. We've come up with two alternatives:
Use a single topic and a single #JmsListener method. Inside the listener, inspect a JMS property to determine the correct strategy for handling the message.
#JmsListener(destination = "response", containerFactory = "myTopicFactory")
public void onMessage(Message message) throws IOException {
message.getProperty("message.type")
JmsHandler handler = messageStrategyMap.get(messageType);
handler.handle(message);
}
Use multiple topics and multiple #JmsListener methods. No switching logic is needed in this case.
#JmsListener(destination = "responseType1", containerFactory = "myTopicFactory")
public void onMessage1(Message message) throws IOException {
// Handle message type 1
}
#JmsListener(destination = "responseType2", containerFactory = "myTopicFactory")
public void onMessage2(Message message) throws IOException {
// Handle message type 2
}
An advantage of approach #1 is that there's no need to manage resources for multiple destinations; there's only only one destination. This also means that the collaborating systems only need to know about a single topic.
An advantage of approach #2 is that there's no need for any type of switching logic. Any message coming in on topic A has a payload formatted specifically for topic A. The concern with this approach is that resources for handling incoming messages might not be allocated efficiently between the different JMS listeners.
In addition, any advice on container factory configuration is appreciated.

It depends on message size and volume. With multiple listeners you are also getting multiple threads, so that may run faster up until the drop-off of having lots of topics (and therefore threads). I suggest setting up a quick test and benchmark on your representative system and you'll be armed with firm numbers to compare.