We Keep Coding

Which is best using separate kafka template / using same kafka template for different topic

My application needs to send different records to different topics. My application is using the same Kafka cluster. Since the application uses the same Kafka cluster, creating one producer factory is sufficient(Let me know if I need more).
In my mind, I have two options.
Using the same kafkaTemplate for both topics and calling the send method with the topic as below(Kindly assume I used spring default Kafka producer configurations). here we need to pass the topic for each call & we use the same Kafka template for multiple topics.
class ProducerService {
#Autowired
private KafkaTemplate<GenericRecord, GenericRecord> kafkaTemplate;
public void send(String topic, GenericRecord key, GenericRecord value) {
ListenableFuture<SendResult<GenericRecord, GenericRecord>> future = kafkaTemplate.send(topic, key, value);
}
}
Using different Kafka templates for different topics. I want to know Is this setup will increase the performance.
import org.apache.avro.generic.GenericRecord;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.core.ProducerFactory;
#Configuration
public class KafkaConfig {
#Value("kafka.topic.first")
private String firstTopic;
#Value("kafka.topic.second")
private String secondTopic;
#Bean(name = "firstKafkaTemplate")
public KafkaTemplate<GenericRecord, GenericRecord> firstKafkaTemplate(ProducerFactory<GenericRecord, GenericRecord> defaultKafkaProducerFactory) {
KafkaTemplate<GenericRecord, GenericRecord> kafkaTemplate = new KafkaTemplate<>(defaultKafkaProducerFactory);
kafkaTemplate.setDefaultTopic(firstTopic);
return kafkaTemplate;
}
#Bean(name = "secondKafkaTemplate")
public KafkaTemplate<GenericRecord, GenericRecord> secondKafkaTemplate(ProducerFactory<GenericRecord, GenericRecord> defaultKafkaProducerFactory) {
KafkaTemplate<GenericRecord, GenericRecord> kafkaTemplate = new KafkaTemplate<>(defaultKafkaProducerFactory);
kafkaTemplate.setDefaultTopic(secondTopic);
return kafkaTemplate;
}
}
class ProducerService {
#Autowired
#Qualifier("firstKafkaTemplate")
private KafkaTemplate<GenericRecord, GenericRecord> firstTopicTemplate;
#Autowired
#Qualifier("secondKafkaTemplate")
private KafkaTemplate<GenericRecord, GenericRecord> secondTopicTemplate;
public void send(String topic, GenericRecord key, GenericRecord value) {
ListenableFuture<SendResult<GenericRecord, GenericRecord>> future;
if ("first".equalsIgnoreCase(topic)) {
future = firstTopicTemplate.sendDefault(key, value);
} else if ("second".equalsIgnoreCase(topic)) {
future = secondTopicTemplate.sendDefault(key, value);
} else {
throw new RuntimeException("topic is not configured");
}
}
}
Internally, Kafka does batch processing and sends batches to Kafka by a separate thread.
Which way is a better way to send records to gain performance? or there is no difference in performance?

I answer my own question based on the throughput. When I processed the records, I got time-out issue.
Single Producer is efficient in most cases
If you are facing any timeout issues due to queueing records at a much faster rate than they can be sent. Then tweak the below parameters to get rid of the timeout issue. Please note that here I added dummy values. you have to test your application to get the desired values for the application.
spring.kafka.producer.properties.[linger.ms]=100
spring.kafka.producer.properties.[batch.size]=100000
spring.kafka.producer.properties.[request.timeout.ms]=30000
spring.kafka.producer.properties.[delivery.timeout.ms]=200000
request.timeout.ms
how long the producer will wait for a reply from the server when sending data will control by this parameter. If the timeout is reached without reply, the producer will either retry sending or respond with an error (either through exception or the send callback).
linger.ms
linger.ms controls the amount of time to wait for additional messages before sending the current batch. Kafka producer sends a batch of messages either when the current batch is full or when the linger.ms limit is reached. By default, the producer will send messages as soon as there is a sender thread available to send them, even if there’s just one message in the batch. By setting linger.ms higher than 0, we instruct the producer to wait a few milliseconds to add additional messages to the batch before sending it to the brokers. This increases latency but also increases throughput (because we send more messages at once, there is less overhead per message).
batch.size
When multiple records are sent to the same partition, the producer will batch them together. This parameter controls the amount of memory in bytes (not messages!) that will be used for each batch. When the batch is full, all the messages in the batch will be sent. However, this does not mean that the producer will wait for the batch to become full. The producer will send half-full batches and even batches with just a single message in them. Therefore, setting the batch size too large will not cause delays in sending messages; it will just use more memory for the batches. Setting the batch size too small will add some overhead because the producer will need to send messages more frequently.
delivery.timeout.ms
An upper bound on the time to report success or failure after a call to send() returns. This limits the total time that a record will be delayed prior to sending, the time to await acknowledgement from the broker (if expected), and the time allowed for retriable send failures. The producer may report a failure to send a record earlier than this config if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this config should be greater than or equal to the sum of request.timeout.ms and linger.ms.
If you are still facing the timeout issue, then you need more producers
Increase the producers by increasing threads for same kafka template
To do this, when you are creating the producer factory, you have to enable the below setProducerPerThread to True.
I have added one TaskExecutor to control the number of producers since the number of producers = number of threads
#Configuration
Public class Conf{
#Bean("kafkaTaskExecutor")
public TaskExecutor getKafkaAsyncExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(5);
executor.setMaxPoolSize(15);
executor.setWaitForTasksToCompleteOnShutdown(true);
executor.setThreadNamePrefix("Kafka-Async-");
return executor;
}
#Bean
public KafkaTemplate<GenericRecord, GenericRecord> kafkaTemplate(ProducerFactory<GenericRecord, GenericRecord> producerFactory) {
if (producerFactory instanceof DefaultKafkaProducerFactory<GenericRecord, GenericRecord> defaultFactory) {
defaultFactory.setProducerPerThread(true);
}
return new KafkaTemplate<>(producerFactory);
}
}
Don't change your Kafka code. Let it be the same. We are going to create a new layer to make it work.
class AsyncProducer{
#Autowired
private KafkaProducer producer;
#Value("${topic.name}")
private String topic;
#Autowired
#Qualifier("kafkaTaskExecutor")
private TaskExecutor taskExecutor;
public void sendAsync(GenericRecord key, GenericRecord value){
CompletableFuture.completeFuture(value).thenAcceptAsync( val-> producer.send(topic,key,value), taskExecutor);
}
}
With the above setup, 5 producers will start to send the record initially, when the load is going high it will be increased to 15 producers
Using multiple Kafka Templates
If you thought, you are still not achieving your throughput, then you can try to increase the number of templates. but actually, I didn't try this since I got the desired result with the second approach.

Rabbit MQ + Spring Boot: delay between resend broken messages

I'm creating application using Spring Boot with RabbitMQ.
I've created configuration for Rabbit like this:
#Configuration
public class RabbitConfiguration {
public static final String RESEND_DISPOSAL_QUEUE = "RESEND_DISPOSAL";
#Bean
public Queue resendDisposalQueue() {
return new Queue(RESEND_DISPOSAL_QUEUE, true);
}
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory (ConnectionFactory connectionFactoryr) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
return factory;
}
#Bean
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory){
return new RabbitTemplate(connectionFactory);
}
}
Also I've created listener for Rabbit messages like this:
#RabbitListener(queues = RESEND_DISPOSAL_QUEUE)
public void getResendDisposalPayload(String messageBody){
LOGGER.info("[getResendDisposalPayload] message = {}", messageBody);
// And there is some business logic
}
All works pretty good, but there is one problem.
When I got exception in method getResendDisposalPayload which listens RESEND_DISPOSAL_QUEUE queue (for example temporary problem with database) Rabbit starts resend last not processed message without any delay. It produces a big amount of log and for some reason uncomfortable for my system.
As I've read in this article https://www.baeldung.com/spring-amqp-exponential-backoff "While using a Dead Letter Queue is a standard way to deal with failed messages".
In order to use this pattern I've to create RetryOperationsInterceptor which defines count attempt to deliver message and delay between attempts.
For example:
#Bean
public RetryOperationsInterceptor retryInterceptor() {
return RetryInterceptorBuilder.stateless()
.backOffOptions(1000, 3.0, 10000)
.maxAttempts(3)
.recoverer(messageRecoverer)
.build();
}
It sounds very good but only one problem: I can't define infinity attempt amount in options maxAttempts.
After maxAttempts I have to save somewhere broken message and deal with it in the future. It demands some extra code.
The question is: Is there any way to configure Rabbit to infinity resend broken messages with some delay, say with one second delay?

Rabbit starts resend last not processed message without any delay
That's how redelivery works: it re-push the same message again and again, until you ack it manually or drop altogether. There is no delay in between redeliveries just because an new message is not pulled from the queue until something is done with this one.
I can't define infinity attempt amount in options maxAttempts
Have you tried an Integer.MAX_VALUE? Pretty decent number of attempts.
The other way is to use a Delayed Exchange: https://docs.spring.io/spring-amqp/docs/current/reference/html/#delayed-message-exchange.
You can configure that retry with a RepublishMessageRecoverer to publish into a your original queue back after some attempts are exhausted: https://docs.spring.io/spring-amqp/docs/current/reference/html/#async-listeners

How to delete dynamically created consumer groups in spring boot app

I have multiple instances of my spring boot app consuming from a kafka topic. Since I want all instances to get data from all partitions of this topic, I assigned different consumers groups for each instances which would be created dynamically when starting this application.
#Configuration
#EnableKafka
public class KafkaStreamConfig {
#Bean("provisioningStreamsBuilderFactoryBean")
public StreamsBuilderFactoryBean myStreamsBuilderFactoryBean() {
String myCGName = "MY-CG-" + UUID.randomUUID().toString();
Properties streamsConfiguration = new Properties();
streamsConfiguration.put(APPLICATION_ID_CONFIG, myCGName); // setting consumer group name
// setting other props
StreamsBuilderFactoryBean streamsBuilderFactoryBean = new StreamsBuilderFactoryBean();
streamsBuilderFactoryBean.setStreamsConfiguration(streamsConfiguration);
return streamsBuilderFactoryBean;
}
}
So every time an instance restarts or a new instance is created, a new consumer group is created. And this's the consumer which reads from my topic.
#Component
public class MyConsumer {
#Autowired
private StreamsBuilder streamsBuilder;
#PostConstruct
public void consume() {
final KStream<String, GenericRecord> events = streamsBuilder.stream("my-topic");
events
.selectKey((key, record) -> record.get("id").toString())
.foreach((id, record) -> {
// some computations with the events consumed
});
}
}
Now because of these dynamically created consumer groups stay on, and since they're not used in my application once an instance restarts, these don't consume messages anymore and show a lot of lag and hence give rise to false alerts.
So I'd like to delete these consumer groups when the application shuts down with Kafka's AdminClient api. I was thinking of trying to delete it in a shutdown hook like in a method annotated with #PreDestroy inside MyConsumer class like this:
#PreDestroy
public void destroyMYCG() {
try (AdminClient admin = KafkaAdminClient.create(properties)) {
DeleteConsumerGroupsResult deleteConsumerGroupsResult = admin.deleteConsumerGroups(Collections.singletonList(provGroupName));
KafkaFuture<Void> future = deleteConsumerGroupsResult.all();
future.whenComplete((aVoid, throwable) -> {
System.out.println("EXCEPTION :: " + ExceptionUtils.getStackTrace(throwable));
});
}
System.out.println(getClass().getCanonicalName() + " :: DESTROYING :: " + provGroupName);
}
but I'm getting this exception if I tried that and consumer groups still shows up in the list of consumer groups:
org.apache.kafka.common.errors.TimeoutException: The AdminClient thread is not accepting new calls.
Can someone please help me with this?

Using UUID as the consumer goup name is terrible.You can definition a final str as consumer goup name for each spring boot app.

IMHO this is logical mistake to create consumer group with UUID. Logically if the same process restarts, it is the same app - the same consumer. You will solve your problem giving good consumer groups names related to what logically do the app.
I would delete consumer groups on the server side, having "GC" set on certain level of lag.
Again consumer group is not application id. It is not intended to be randomly created.
And honestly spoken I not sure what kind of problem do you solve doing this.
Because in fact by saying that consumer group is random, you say my code is doing random things and I have no clue what happens in message processing.
We have very complex Kafka message processing and always there is better or worse name for the process, but at least exist one, which is not random.

How to make several threads do takes from RabbitMQ queue using Spring Boot?

Our application consumes data from several queues that are provided by RabbitMQ. To increase throughput we start several threads per queue that do blocking takes from those queues.
For a new service we want to use Spring Boot and again have several threads per queue that take data from those queues. Here is the canonical Spring Boot code for processing data that arrived from some queue:
#StreamListener(target = Processor.INPUT)
#SendTo(Processor.OUTPUT)
public Message<SomeData> process(Message<SomeData> message) {
SomeData result = service.process(message.getPayload());
return MessageBuilder
.withPayload(result)
.copyHeaders(message.getHeaders())
.build();
}
Question is now how to make Spring Boot spawn several threads to serve one queue instead of a single thread. Throughput is very critical for our application, hence the need for this.

Check the available properties, search for rabbitmq.
spring.rabbitmq.listener.simple.concurrency= # Minimum number of listener invoker threads
That looks promising

You can set the concurrent consumers for the queue when you configurate it.
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory
(MessageConverter contentTypeConverter,
SimpleRabbitListenerContainerFactoryConfigurer configurer) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
// the number of consumers is set as 5
factory.setConcurrentConsumers(5);
configurer.configure(factory, connectionFactory);
factory.setMessageConverter(contentTypeConverter);
return factory;
}

Amount parallel processing Simple Queue Service (SQS)

I am using Spring Cloud to consume Simple Queue Service (SQS). I have the following configurations for parallel processing:
#Bean
public SimpleAsyncTaskExecutor simpleAsyncTaskExecutor() {
SimpleAsyncTaskExecutor simpleAsyncTaskExecutor = new SimpleAsyncTaskExecutor();
simpleAsyncTaskExecutor.setConcurrencyLimit(50);
return simpleAsyncTaskExecutor;
}
#Bean
public SimpleMessageListenerContainerFactory simpleMessageListenerContainerFactory(
SimpleAsyncTaskExecutor simpleAsyncTaskExecutor) {
SimpleMessageListenerContainerFactory factory = new SimpleMessageListenerContainerFactory();
factory.setAutoStartup(true);
factory.setTaskExecutor(simpleAsyncTaskExecutor);
factory.setWaitTimeOut(20);
factory.setMaxNumberOfMessages(10);
return factory;
}
I need to process 50 messages in 50 threads (configuration in the bean SimpleAsyncTaskExecutor), but is processing only 10 messages in parallel (maxNumberOfMessages returned from SQS)
How can I process 50 messages instead 10?

I found the solution.
It's necessary to annotate the method with #Async, change deletionPolicy to NEVER, and delete the message when finalizing execution.
In this way, the queue consume will respect the configured number of threads. For example, if you have 50 threads, will make 5 requests in the SQS queue (10 messages per request), thus processing a total of 50 messages in parallel.
The code looks like this:
#Async
#SqsListener(value = "sqsName", deletionPolicy = SqsMessageDeletionPolicy.NEVER)
public void consume(String message, Acknowledgment acknowledgment) throws InterruptedException, ExecutionException {
//your code
acknowledgment.acknowledge().get(); //To delete message from queue
}

I wouldn't be into specific numbers (like 50 messages for 50 threads) too much. Try performance testing it instead (build something to push the expected number of messages in peak-hours to the queue, and let your service handle them, to see if it bottlenecks).
As per your actual question, you can't. AWS SQS simply doesnt support fetching more than 10 messages pr. request. see http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html for reference. (it's in the 1st paragraph).