We Keep Coding

Which is best using separate kafka template / using same kafka template for different topic

My application needs to send different records to different topics. My application is using the same Kafka cluster. Since the application uses the same Kafka cluster, creating one producer factory is sufficient(Let me know if I need more).
In my mind, I have two options.
Using the same kafkaTemplate for both topics and calling the send method with the topic as below(Kindly assume I used spring default Kafka producer configurations). here we need to pass the topic for each call & we use the same Kafka template for multiple topics.
class ProducerService {
#Autowired
private KafkaTemplate<GenericRecord, GenericRecord> kafkaTemplate;
public void send(String topic, GenericRecord key, GenericRecord value) {
ListenableFuture<SendResult<GenericRecord, GenericRecord>> future = kafkaTemplate.send(topic, key, value);
}
}
Using different Kafka templates for different topics. I want to know Is this setup will increase the performance.
import org.apache.avro.generic.GenericRecord;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.core.ProducerFactory;
#Configuration
public class KafkaConfig {
#Value("kafka.topic.first")
private String firstTopic;
#Value("kafka.topic.second")
private String secondTopic;
#Bean(name = "firstKafkaTemplate")
public KafkaTemplate<GenericRecord, GenericRecord> firstKafkaTemplate(ProducerFactory<GenericRecord, GenericRecord> defaultKafkaProducerFactory) {
KafkaTemplate<GenericRecord, GenericRecord> kafkaTemplate = new KafkaTemplate<>(defaultKafkaProducerFactory);
kafkaTemplate.setDefaultTopic(firstTopic);
return kafkaTemplate;
}
#Bean(name = "secondKafkaTemplate")
public KafkaTemplate<GenericRecord, GenericRecord> secondKafkaTemplate(ProducerFactory<GenericRecord, GenericRecord> defaultKafkaProducerFactory) {
KafkaTemplate<GenericRecord, GenericRecord> kafkaTemplate = new KafkaTemplate<>(defaultKafkaProducerFactory);
kafkaTemplate.setDefaultTopic(secondTopic);
return kafkaTemplate;
}
}
class ProducerService {
#Autowired
#Qualifier("firstKafkaTemplate")
private KafkaTemplate<GenericRecord, GenericRecord> firstTopicTemplate;
#Autowired
#Qualifier("secondKafkaTemplate")
private KafkaTemplate<GenericRecord, GenericRecord> secondTopicTemplate;
public void send(String topic, GenericRecord key, GenericRecord value) {
ListenableFuture<SendResult<GenericRecord, GenericRecord>> future;
if ("first".equalsIgnoreCase(topic)) {
future = firstTopicTemplate.sendDefault(key, value);
} else if ("second".equalsIgnoreCase(topic)) {
future = secondTopicTemplate.sendDefault(key, value);
} else {
throw new RuntimeException("topic is not configured");
}
}
}
Internally, Kafka does batch processing and sends batches to Kafka by a separate thread.
Which way is a better way to send records to gain performance? or there is no difference in performance?

I answer my own question based on the throughput. When I processed the records, I got time-out issue.
Single Producer is efficient in most cases
If you are facing any timeout issues due to queueing records at a much faster rate than they can be sent. Then tweak the below parameters to get rid of the timeout issue. Please note that here I added dummy values. you have to test your application to get the desired values for the application.
spring.kafka.producer.properties.[linger.ms]=100
spring.kafka.producer.properties.[batch.size]=100000
spring.kafka.producer.properties.[request.timeout.ms]=30000
spring.kafka.producer.properties.[delivery.timeout.ms]=200000
request.timeout.ms
how long the producer will wait for a reply from the server when sending data will control by this parameter. If the timeout is reached without reply, the producer will either retry sending or respond with an error (either through exception or the send callback).
linger.ms
linger.ms controls the amount of time to wait for additional messages before sending the current batch. Kafka producer sends a batch of messages either when the current batch is full or when the linger.ms limit is reached. By default, the producer will send messages as soon as there is a sender thread available to send them, even if there’s just one message in the batch. By setting linger.ms higher than 0, we instruct the producer to wait a few milliseconds to add additional messages to the batch before sending it to the brokers. This increases latency but also increases throughput (because we send more messages at once, there is less overhead per message).
batch.size
When multiple records are sent to the same partition, the producer will batch them together. This parameter controls the amount of memory in bytes (not messages!) that will be used for each batch. When the batch is full, all the messages in the batch will be sent. However, this does not mean that the producer will wait for the batch to become full. The producer will send half-full batches and even batches with just a single message in them. Therefore, setting the batch size too large will not cause delays in sending messages; it will just use more memory for the batches. Setting the batch size too small will add some overhead because the producer will need to send messages more frequently.
delivery.timeout.ms
An upper bound on the time to report success or failure after a call to send() returns. This limits the total time that a record will be delayed prior to sending, the time to await acknowledgement from the broker (if expected), and the time allowed for retriable send failures. The producer may report a failure to send a record earlier than this config if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this config should be greater than or equal to the sum of request.timeout.ms and linger.ms.
If you are still facing the timeout issue, then you need more producers
Increase the producers by increasing threads for same kafka template
To do this, when you are creating the producer factory, you have to enable the below setProducerPerThread to True.
I have added one TaskExecutor to control the number of producers since the number of producers = number of threads
#Configuration
Public class Conf{
#Bean("kafkaTaskExecutor")
public TaskExecutor getKafkaAsyncExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(5);
executor.setMaxPoolSize(15);
executor.setWaitForTasksToCompleteOnShutdown(true);
executor.setThreadNamePrefix("Kafka-Async-");
return executor;
}
#Bean
public KafkaTemplate<GenericRecord, GenericRecord> kafkaTemplate(ProducerFactory<GenericRecord, GenericRecord> producerFactory) {
if (producerFactory instanceof DefaultKafkaProducerFactory<GenericRecord, GenericRecord> defaultFactory) {
defaultFactory.setProducerPerThread(true);
}
return new KafkaTemplate<>(producerFactory);
}
}
Don't change your Kafka code. Let it be the same. We are going to create a new layer to make it work.
class AsyncProducer{
#Autowired
private KafkaProducer producer;
#Value("${topic.name}")
private String topic;
#Autowired
#Qualifier("kafkaTaskExecutor")
private TaskExecutor taskExecutor;
public void sendAsync(GenericRecord key, GenericRecord value){
CompletableFuture.completeFuture(value).thenAcceptAsync( val-> producer.send(topic,key,value), taskExecutor);
}
}
With the above setup, 5 producers will start to send the record initially, when the load is going high it will be increased to 15 producers
Using multiple Kafka Templates
If you thought, you are still not achieving your throughput, then you can try to increase the number of templates. but actually, I didn't try this since I got the desired result with the second approach.

Rabbit MQ + Spring Boot: delay between resend broken messages

I'm creating application using Spring Boot with RabbitMQ.
I've created configuration for Rabbit like this:
#Configuration
public class RabbitConfiguration {
public static final String RESEND_DISPOSAL_QUEUE = "RESEND_DISPOSAL";
#Bean
public Queue resendDisposalQueue() {
return new Queue(RESEND_DISPOSAL_QUEUE, true);
}
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory (ConnectionFactory connectionFactoryr) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
return factory;
}
#Bean
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory){
return new RabbitTemplate(connectionFactory);
}
}
Also I've created listener for Rabbit messages like this:
#RabbitListener(queues = RESEND_DISPOSAL_QUEUE)
public void getResendDisposalPayload(String messageBody){
LOGGER.info("[getResendDisposalPayload] message = {}", messageBody);
// And there is some business logic
}
All works pretty good, but there is one problem.
When I got exception in method getResendDisposalPayload which listens RESEND_DISPOSAL_QUEUE queue (for example temporary problem with database) Rabbit starts resend last not processed message without any delay. It produces a big amount of log and for some reason uncomfortable for my system.
As I've read in this article https://www.baeldung.com/spring-amqp-exponential-backoff "While using a Dead Letter Queue is a standard way to deal with failed messages".
In order to use this pattern I've to create RetryOperationsInterceptor which defines count attempt to deliver message and delay between attempts.
For example:
#Bean
public RetryOperationsInterceptor retryInterceptor() {
return RetryInterceptorBuilder.stateless()
.backOffOptions(1000, 3.0, 10000)
.maxAttempts(3)
.recoverer(messageRecoverer)
.build();
}
It sounds very good but only one problem: I can't define infinity attempt amount in options maxAttempts.
After maxAttempts I have to save somewhere broken message and deal with it in the future. It demands some extra code.
The question is: Is there any way to configure Rabbit to infinity resend broken messages with some delay, say with one second delay?

Rabbit starts resend last not processed message without any delay
That's how redelivery works: it re-push the same message again and again, until you ack it manually or drop altogether. There is no delay in between redeliveries just because an new message is not pulled from the queue until something is done with this one.
I can't define infinity attempt amount in options maxAttempts
Have you tried an Integer.MAX_VALUE? Pretty decent number of attempts.
The other way is to use a Delayed Exchange: https://docs.spring.io/spring-amqp/docs/current/reference/html/#delayed-message-exchange.
You can configure that retry with a RepublishMessageRecoverer to publish into a your original queue back after some attempts are exhausted: https://docs.spring.io/spring-amqp/docs/current/reference/html/#async-listeners

Why ActiveMQ Artemis auto delete address if all listeners are destroyed?

We are develop a micro-service system that use ActiveMQ Artemis as the communication method between service. Since the requirement ask to be able to stop the listeners at runtime, we can not use #JmsListener provide by spring-artemis. After digging the internet and finding out that spring use MessageListenerContainer behind the scence, we come up with the idea of maintain a list of MessageListenerContainer our self.
#Bean(name = "commandJmsListenerContainerFactory")
public DefaultJmsListenerContainerFactory commandJmsListenerContainerFactory(
DefaultJmsListenerContainerFactoryConfigurer configurer) {
var factory = new DefaultJmsListenerContainerFactory();
configurer.configure(factory, connectionFactory);
factory.setPubSubDomain(false);
return factory;
}
// Use
private Map<String, DefaultMessageListenerContainer> commandQueue;
public void subscribeToCommandQueue(String queueName, CommandListener<?> command) {
commandQueue.computeIfAbsent(queueName, key -> {
var endPoint = new SimpleJmsListenerEndpoint();
endPoint.setDestination(queueName);
endPoint.setMessageListener(message -> {
try {
var body = message.getBody(String.class);
command.execute(commandMessageConverter.deserialize(body));
} catch (JMSException e) {
throw new RuntimeException("Error while process message for queue: " + queueName, e);
}
});
var container = commandJmsListenerContainerFactory.createListenerContainer(endPoint);
// https://stackoverflow.com/questions/44555106/defaultmessagelistenercontainer-not-reading-messages-from-ibm-mq
// for Every object of Spring classes that implement InitializingBean created manually, we need to call afterPropertiesSet to make the object "work"
container.afterPropertiesSet();
container.start();
return container;
});
}
public void start() {
commandQueue = new ConcurrentHashMap<>();
}
public void stop() {
commandQueue.values().forEach(DefaultMessageListenerContainer::destroy);
commandQueue.clear();
}
While testing, I notice that after we destroy all the listener by calling stop() , the queue and the address in the Artemis console are deleted too. It isn't the case for durable subscription.
#Bean(name = "eventJmsListenerContainerFactory")
public DefaultJmsListenerContainerFactory eventJmsListenerContainerFactory(
CachingConnectionFactory cachingConnectionFactory,
DefaultJmsListenerContainerFactoryConfigurer configurer) {
cachingConnectionFactory.setClientId(UUID.randomUUID().toString());
var factory = new DefaultJmsListenerContainerFactory();
configurer.configure(factory, cachingConnectionFactory);
factory.setPubSubDomain(true);
factory.setSubscriptionDurable(true);
return factory;
}
// usage is the same as the first block code, except we store multicast subscriptions in another map
private Map<String, DefaultMessageListenerContainer> eventTopic;
After running the unit tests and destroying all the listeners of two maps, only the test-event-topic address and its queues were kept, the test-command-queue was deleted. Why both the queues behave differently?
Also, what is the correct behavior? We afraid the auto deletion will remove messages that aren't sent in the queue yet. On the other hand, new queue under test-event-topic keep being created if we run the test again and again. I think it is because of the line cachingConnectionFactory.setClientId(UUID.randomUUID().toString()); . But for durable subscription, not setting clientId result in error.
The connection factory used in the app is an CachingConnectionFactory created by spring-artemis

By default the broker will auto-create addresses and queues as required when a message is sent or a consumer is created by the core JMS client. These resources will also be auto-deleted by default when they're no longer needed (i.e. when a queue has no consumers and messages or when an address no longer has any queues bound to it). This is controlled by these settings in broker.xml which are discussed in the documentation:
auto-create-queues
auto-delete-queues
auto-create-addresses
auto-delete-addresses
To be clear, auto-deletion should not cause any message loss by default as queues should only be deleted when they have 0 consumers and 0 messages. However, you can always set auto-deletion to false to be 100% safe.
Queues representing durable JMS topic subscriptions won't be deleted as they are meant to stay and gather messages while the consumer is offline. In other words, a durable topic subscription will remain if the client using the subscription is shutdown without first explicitly removing the subscription. That's the whole point of durable subscriptions - they are durable. Any client can use a durable topic subscription if it connects with the same client ID and uses the same subscription name. However, unless the durable subscription is a "shared" durable subscription then only one client at a time can be connected to it. Shared durable topic subscriptions were added in JMS 2.0.

Deletion of kafka logs using retention time

I am working on windows platform, a tool in which i will produce messages to kafka from spring boot application very often(A function will produces message continuously). And i will consumes the messages from node.js application. So the application creates so many topics in a day, kafka logs is occupying full disk space within a week. So i tried with log.retention.hours function to delete the logs but i am getting Error while deleting segments, java.nio.file.FileSystemException: The process cannot access the file because it is being used by another process.
NOTE: I haven't get a solution to fix it, i don't know the reason why it is happening. I have two questions
1) Do i need to configure any thing in my application or do i need to send confirmation to kafka server from my spring boot application that i finished producing messages to the topic so that kafka will delete it.
2) how can i connect to kafka server from other machines?(i am hosting kafka, zookeeper in a machine and i am producing messages to a topic from the same machine. now i am trying to consumes messages from other machine but i couldn't connect to the kafka server)
Below are the configuration i am using in the spring boot application and i will produce messages to topics.
#Configuration
public class KafkaProducerConfig {
#Bean
public ProducerFactory<String, String> producerFactory() {
Map<String, Object> configProps = new HashMap<>();
configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
return new DefaultKafkaProducerFactory<>(configProps);
}
#Bean
public KafkaTemplate<String, String> kafkaTemplate() {
return new KafkaTemplate<>(producerFactory());
}
}

Spring JMS listener acknowledging even when exception

I am using JMS to send/receive messages to my SQS queue, however i am unable to redeliver the message when there is an exception even while using client_acknowledge. How to achieve this?
I tried a simple test,
#JmsListener(destination = "test-normalqueue")
public void receiveNormalQueue(String message)
{
try {
logger.info("message received in normal queue: " + message);
throw new NullPointerException();
} catch (Exception e) {
logger.error(LoggingUtil.getStackTrace(e));;
}
}
Even after exception message doesnt come back to queue.
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(getSQSConnectionFactory());
factory.setConcurrency("1-2");
factory.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
return factory;
}

You have to use transactions with the DMLC.
Use Session.AUTO_ACKNOWLEDGE and setSessionTransacted(true).
If the listener exits normally, the message is removed. If the listener throws an exception, the message will be rolled-back onto the queue.
You can also use client mode with transactions, but you have to acknowledge successful messages yourself.
You don't have to use transactions with a SimpleMessageListenerContainer but you still have to throw an exception to get the message requeued.

The Messages comes back to queue only if the listener stops and disconnects from the broker, the behavior you are describing is on the client side on the DefaultMessageListenerContainer which dispatch messages to your listener and manage exceptions and retries, the broker is not aware of those treatments and he only knows that these messages are dispatched to the client and is waiting acknowledgements.
It is depends on the SQS methods and capabilities if there a method like reset or restart on the implementation of MessageConsumer.
You can try recover() method of jms session but i think this will only restart delivery on the client side.
https://docs.oracle.com/javaee/7/api/javax/jms/Session.html#recover--
This is a not good practice but if you restart the connection or the DefaultMessageListenerContainer the messages not acknowledged comes back to broker and the delivery restarts.