We Keep Coding

Kafka consumer in test only works with "auto.offset.reset"="earliest"

I'm struggling to understand my Kafka consumer behaviours in some integration tests.
I have a Spring boot service which uses a default, autowired KafkaTemplate<String, String> to produce messages to a topic. In my integration tests, I create a KafkaConsumer in each test:
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(
Map.of( ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, KAFKA_CONTAINER.getBootstrapServers(),
ConsumerConfig.GROUP_ID_CONFIG, "test-consumer-group-" + UUID.randomUUID(),
ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, UUID.randomUUID().toString() ),
new StringDeserializer(), new StringDeserializer() );
consumer.subscribe( topics );
return consumer;
with the intent of having a test flow that looks something like:
Create a new consumer for the topics we're testing
Perform action under test which sends messages to some topics
Poll the topics of interest and verify the messages are there
Close consumer
My expectation was that since the default behaviour of a new consumer is to have auto.offset.reset set to latest I would only get messages sent after I create the consumer, which looks fine in this case. However my consumer never receives any messages! I have to set the consumer to earliest - but this is problematic since I don't want messages created by other tests interfering.
The messages don't have any kind of unique identifier on them, which makes consuming the entire topic each time a tricky proposition in terms of test verifications.
I've tried various permutations of auto committing, polling before running the test but after subscribing, manual syncs but nothing seems to work - how can manage my test lifecycle as described above (or is it not possible)?
The kafka instance is managed using TestContainers in case that's relevant.

RabbitMQ's receiveAndConvert for clustered environment

I have this method implemented in a SpringBoot application
#Scheduled(fixedDelay = 5000)
public void pullMessage() {
MessageDTO message = null;
try {
message = rabbitTemplate.receiveAndConvert(properties.getQueueName(), new ParameterizedTypeReference<MessageDTO>() {});
// more code here...
}
every 5 seconds I'm pulling a message from RabbitMQ and processing something with it. The application is running on Kubernetes and right now I have to duplicate the POD. In this scenario, could the two pods pull the same message?

If the queue is the same for all the instances, then no: only one consumer takes a message from a queue. That's fundamental purpose of the queue pattern at all.
See AMQP docs for publish-subscribe patterns: https://www.rabbitmq.com/tutorials/tutorial-three-java.html

No only a single instance will process the message at one time, the whole purpose of having multiple consumers is not to have any downtime for the application!
Refer the official documentation of RabbitMQ for more clarification!
https://www.rabbitmq.com/tutorials/tutorial-one-java.html

Should i create NewTopics in each service spring kafka?

I'm using Kafka for sending messages between services. I use NewTopic bean for configuring number of partitions, for example:
#Bean
fun kafkaTopic(kafkaProperties: KafkaProperties): NewTopic = NewTopic(
kafkaProperties.topics.schedulerCalculationTopic.name,
kafkaProperties.topics.schedulerCalculationTopic.partitions,
1
)
My question is simple, should i add this bean into consumer service and producer service or only in one of them?

I would put it in the producer service and then consider the producer as 'owner' of those topics.
But it get a bit complicated if you have a scenario if you would have several producers to the same topic(s).

If you are not creating the topic on the fly, the best practice is to create topic before reading/writing to it.
Rationale is to prevent brokers to create topic whenever they receive metadata fetch request or consume request with the same topic name. Otherwise, if the consumer starts before the producer, you might end up wrong number of partition. (Broker will create your topic with default number of partitions setting.)

How to delete dynamically created consumer groups in spring boot app

I have multiple instances of my spring boot app consuming from a kafka topic. Since I want all instances to get data from all partitions of this topic, I assigned different consumers groups for each instances which would be created dynamically when starting this application.
#Configuration
#EnableKafka
public class KafkaStreamConfig {
#Bean("provisioningStreamsBuilderFactoryBean")
public StreamsBuilderFactoryBean myStreamsBuilderFactoryBean() {
String myCGName = "MY-CG-" + UUID.randomUUID().toString();
Properties streamsConfiguration = new Properties();
streamsConfiguration.put(APPLICATION_ID_CONFIG, myCGName); // setting consumer group name
// setting other props
StreamsBuilderFactoryBean streamsBuilderFactoryBean = new StreamsBuilderFactoryBean();
streamsBuilderFactoryBean.setStreamsConfiguration(streamsConfiguration);
return streamsBuilderFactoryBean;
}
}
So every time an instance restarts or a new instance is created, a new consumer group is created. And this's the consumer which reads from my topic.
#Component
public class MyConsumer {
#Autowired
private StreamsBuilder streamsBuilder;
#PostConstruct
public void consume() {
final KStream<String, GenericRecord> events = streamsBuilder.stream("my-topic");
events
.selectKey((key, record) -> record.get("id").toString())
.foreach((id, record) -> {
// some computations with the events consumed
});
}
}
Now because of these dynamically created consumer groups stay on, and since they're not used in my application once an instance restarts, these don't consume messages anymore and show a lot of lag and hence give rise to false alerts.
So I'd like to delete these consumer groups when the application shuts down with Kafka's AdminClient api. I was thinking of trying to delete it in a shutdown hook like in a method annotated with #PreDestroy inside MyConsumer class like this:
#PreDestroy
public void destroyMYCG() {
try (AdminClient admin = KafkaAdminClient.create(properties)) {
DeleteConsumerGroupsResult deleteConsumerGroupsResult = admin.deleteConsumerGroups(Collections.singletonList(provGroupName));
KafkaFuture<Void> future = deleteConsumerGroupsResult.all();
future.whenComplete((aVoid, throwable) -> {
System.out.println("EXCEPTION :: " + ExceptionUtils.getStackTrace(throwable));
});
}
System.out.println(getClass().getCanonicalName() + " :: DESTROYING :: " + provGroupName);
}
but I'm getting this exception if I tried that and consumer groups still shows up in the list of consumer groups:
org.apache.kafka.common.errors.TimeoutException: The AdminClient thread is not accepting new calls.
Can someone please help me with this?

Using UUID as the consumer goup name is terrible.You can definition a final str as consumer goup name for each spring boot app.

IMHO this is logical mistake to create consumer group with UUID. Logically if the same process restarts, it is the same app - the same consumer. You will solve your problem giving good consumer groups names related to what logically do the app.
I would delete consumer groups on the server side, having "GC" set on certain level of lag.
Again consumer group is not application id. It is not intended to be randomly created.
And honestly spoken I not sure what kind of problem do you solve doing this.
Because in fact by saying that consumer group is random, you say my code is doing random things and I have no clue what happens in message processing.
We have very complex Kafka message processing and always there is better or worse name for the process, but at least exist one, which is not random.

Spring 2.1.0.M4 rabbitmq declare queues and bind them to listeners in runtime

Let's say I have a declared listener:
Listener.java
#RabbitListener(id = "test listener 1")
public String test2(String req) {
return req + " result";
}
I'm trying to expose it via queue during runtime:
ListenerTest.java
Queue declaredQueue = new Queue("new.queue", false);
admin.declareQueue(declaredQueue);
SimpleMessageListenerContainer listener = (SimpleMessageListenerContainer)
registry.getListenerContainer("test listener 1");
listener.addQueues(declaredQueue);
And then I try message the newly declared queue :
String result = template.convertSendAndReceiveAsType("new.queue", "req", ParameterizedTypeReference.forType(String.class));
But it just times out and returns null.
When I inspect the listener in debugger I can't see any consumers bound to the new queue
You can find by rabbit config here and the rest of the source to test this here.
It's worth to note that this exact setup works in spring boot version 2.0.5.RELEASE so it might be a bug. I need to find a way to reinitialize the consumers.

Adding queues at runtime will cause the container to recycle its consumers (the equivalent of stopping and restarting the container). See https://github.com/spring-projects/spring-amqp/blob/master/spring-rabbit/src/main/java/org/springframework/amqp/rabbit/listener/SimpleMessageListenerContainer.java#L687 This is due to the way the consumer is designed; each consumer thread consumes from multiple queues.
Changing the consumer count does not restart all consumers; see https://github.com/spring-projects/spring-amqp/blob/master/spring-rabbit/src/main/java/org/springframework/amqp/rabbit/listener/SimpleMessageListenerContainer.java#L168
The new DirectMessageListenerContainer does not need to restart its consumers when queues are added (there is at least one consumer per queue).
However, it does not support dynamic concurrency scaling.

After the 2.1.0 release SimpleMessageListenerContainer no longer overrides AbstractMessageListenerContainer#addQueues which causes SimpleMessageListenerContainer#queuesChanged to not be invoked (likely a bug). However SimpleMessageListenerContainer#addQueueNames is overrided and working. So this either has to be changed to
SimpleMessageListenerContainer listener = (SimpleMessageListenerContainer)
registry.getListenerContainer("test listener 1");
listener.addQueueNames(declaredQueue.getName());
Or queuesChanged has to be called manually
SimpleMessageListenerContainer listener = (SimpleMessageListenerContainer)
registry.getListenerContainer("test listener 1");
listener.addQueues(declaredQueue);
Method method = SimpleMessageListenerContainer.class.getDeclaredMethod("queuesChanged");
method.setAccessible(true);
method.invoke(listener);