We Keep Coding

Reconnect KafkaProducer

We are about to write an application to send kafka events
after startup, this application will run for hours/days/weeks
depending on the load we will send hunderds of events a minute but there also will be periods, where we will send no events for hours
Is it allowed to create a producer while startup and use the same producer all the time?
in other words: is it guaranteed that the kafka producer can reconnect after every exception (i.E. after network or broker problems, once the resources is available again) (case A) or are there exceptions where the application has to close the producer an create a new producer (case B)?
case a:
(shown simplified)
while (true) {
try {
ProducerRecord<String, String> record = new ProducerRecord<String, String>(topic, key, value);
producer.send(record).get(); // producer always can re-establish the connection in case of a prior exception
} catch (Exception ex) {
ex.printStackTrace();
}
}
case b:
(shown simplified)
while (true) {
try {
ProducerRecord<String, String> record = new ProducerRecord<String, String>(topic, key, value);
producer.send(record).get();
} catch (ExceptionAbc? ex) {
// application has to re-create the producer in some cases
producer = new KafkaProducer<String, String>(props);
} catch (Exception ex) {
ex.printStackTrace();
}
}
Thank you for your answer

The producer will refresh its state every metadata.max.age.ms, therefore try to establish connection to the broker and not close without exception during the lifetime of the application.
If the producer is closed, then the app itself should probably fatally crash and have a process manager restart the app rather than the app try to only restart the producer client.

thank you for your answer. will the kafkaproducer be closed automatically if the connection can't be reestablished? i can't find a hint in kafka documentation about this.
and if so, how can i know, if the producer is closed or not? the kafka producer and the producer interface has no "isClosed" method. in the source code i can see, that a illegalstate exception will be thrown, if the producer is used after the producer was closed, but this exception is not unique.

Reply timeout when using AsyncRabbitTemplate::sendAndReceive - RabbitMQ

I recently changed from using a standard Rabbit Template, in my Spring Boot application, to using an Async Rabbit Template. In the process, I switched from the standard send method to using the sendAndReceive method.
Making this change does not seem to affect the publishing of messages to RabbitMQ, however I do now see stack traces as follows when sending messages:
org.springframework.amqp.core.AmqpReplyTimeoutException: Reply timed out
at org.springframework.amqp.rabbit.AsyncRabbitTemplate$RabbitFuture$TimeoutTask.run(AsyncRabbitTemplate.java:762) [spring-rabbit-2.3.10.jar!/:2.3.10]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.3.9.jar!/:5.3.9]
I have tried modifying various settings including the reply and receive timeouts but all that changes is the time it takes to receive the above error. I have also tried setting useDirectReplyToContainer to true as well as setting useChannelForCorrelation to true.
I have managed to recreate the issue in a main method, included bellow, using a RabbitMQ broker running in docker.
public static void main(String[] args) {
com.rabbitmq.client.ConnectionFactory cf = new com.rabbitmq.client.ConnectionFactory();
cf.setHost("localhost");
cf.setPort(5672);
cf.setUsername("<my-username>");
cf.setPassword("<my-password>");
cf.setVirtualHost("<my-vhost>");
ConnectionFactory connectionFactory = new CachingConnectionFactory(cf);
RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setExchange("primary");
rabbitTemplate.setUseDirectReplyToContainer(true);
rabbitTemplate.setReceiveTimeout(10000);
rabbitTemplate.setReplyTimeout(10000);
rabbitTemplate.setUseChannelForCorrelation(true);
AsyncRabbitTemplate asyncRabbitTemplate = new AsyncRabbitTemplate(rabbitTemplate);
asyncRabbitTemplate.start();
System.out.printf("Async Rabbit Template Running? %b\n", asyncRabbitTemplate.isRunning());
MessageBuilderSupport<MessageProperties> props = MessagePropertiesBuilder.newInstance()
.setContentType(MessageProperties.CONTENT_TYPE_TEXT_PLAIN)
.setMessageId(UUID.randomUUID().toString())
.setHeader(PUBLISH_TIME_HEADER, Instant.now(Clock.systemUTC()).toEpochMilli())
.setDeliveryMode(MessageDeliveryMode.NON_PERSISTENT);
asyncRabbitTemplate.sendAndReceive(
"1.1.1.csv-routing-key",
new Message(
"a,test,csv".getBytes(StandardCharsets.UTF_8),
props.build()
)
).addCallback(new ListenableFutureCallback<>() {
#Override
public void onFailure(Throwable ex) {
System.out.printf("Error sending message:\n%s\n", ex.getLocalizedMessage());
}
#Override
public void onSuccess(Message result) {
System.out.println("Message successfully sent");
}
});
}
I am sure that I am just missing a configuration option but any help would be appricated.
Thanks. :)

asyncRabbitTemplate.sendAndReceive(..) will always expect a response from the consumer of the message, hence the timeout you are receiving.
To fire and forget use the standard RabbitTemplate.send(...) and catching any exceptions in a try/catch block:
try {
rabbitTemplate.send("1.1.1.csv-routing-key",
new Message(
"a,test,csv".getBytes(StandardCharsets.UTF_8),
props.build());
} catch (AmqpException ex) {
log.error("failed to send rabbit message, routing key = {}", routingKey, ex);
}

Set reply timeout to some bigger number and see the effect.
rabbitTemplate.setReplyTimeout(60000);
https://docs.spring.io/spring-amqp/reference/html/#reply-timeout

How to handle backpressure when using apache camel and Kafka?

I am trying to write an application that will integrate with Kafka using Camel. (Version - 3.4.2)
I have an approach borrowed from the answer to this question.
I have a route that listens for messages from a Kafka topic. The processing of this message is decoupled from consumption by using a simple executor. Each processing is submitted as a task to this executor. The ordering of the messages is not important and the only concerning factor is how quickly and efficiently the message can be processed. I have disabled the auto-commit and manually commit the messages once the tasks are submitted to the executor. The loss of the messages that are currently being processed (due to crash/shutdown) is okay but the ones in Kafka that have never been submitted for the processing should not be lost (due to committing of the offset). Now to the questions,
How can I efficiently handle the load? For e.g, there are 1000 messages but I can only parallelly process 100 at a time.
Right now the solution I have is to block the consumer polling thread and trying to continuously submit the job. But a suspension of polling would be a much better approach but I cannot find any way to achieve that in Camel.
Is there a better way (Camel way) to decouple processing from consumption and handle backpressure?
public static void main(String[] args) throws Exception {
String consumerId = System.getProperty("consumerId", "1");
ExecutorService executor = new ThreadPoolExecutor(100, 100, 0L, TimeUnit.MILLISECONDS,
new SynchronousQueue<>());
LOGGER.info("Consumer {} starting....", consumerId);
Main main = new Main();
main.init();
CamelContext context = main.getCamelContext();
ComponentsBuilderFactory.kafka().brokers("localhost:9092").metadataMaxAgeMs(120000).groupId("consumer")
.autoOffsetReset("earliest").autoCommitEnable(false).allowManualCommit(true).maxPollRecords(100)
.register(context, "kafka");
ConsumerBean bean = new ConsumerBean();
context.addRoutes(new RouteBuilder() {
#Override
public void configure() {
from("kafka:test").process(exchange -> {
LOGGER.info("Consumer {} - Exhange is {}", consumerId, exchange.getIn().getHeaders());
processTask(exchange);
commitOffset(exchange);
});
}
private void processTask(Exchange exchange) throws InterruptedException {
try {
executor.submit(() -> bean.execute(exchange.getIn().getBody(String.class)));
} catch (Exception e) {
LOGGER.error("Exception occured {}", e.getMessage());
Thread.sleep(1000);
processTask(exchange);
}
}
private void commitOffset(Exchange exchange) {
boolean lastOne = exchange.getIn().getHeader(KafkaConstants.LAST_RECORD_BEFORE_COMMIT, Boolean.class);
if (lastOne) {
KafkaManualCommit manual = exchange.getIn().getHeader(KafkaConstants.MANUAL_COMMIT,
KafkaManualCommit.class);
if (manual != null) {
LOGGER.info("manually committing the offset for batch");
manual.commitSync();
}
} else {
LOGGER.info("NOT time to commit the offset yet");
}
}
});
main.run();
}

You can use throttle EIP for this purpose.
from("your uri here")
.throttle(maxRequestCount)
.timePeriodMillis(inTimePeriodMs)
.to(yourProcessorUri)
.end()
Please take a look at the original documentation here.

unable to send single message to kafka topic

I am using kafka java client 0.11.0 and kafka server 2.11-0.10.2.0.
My code :
KafkaManager
public class KafkaManager {
// Single instance for producer per topic
private static Producer<String, String> karmaProducer = null;
/**
* Initialize Producer
*
* #throws Exception
*/
private static void initProducer() throws Exception {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, Constants.kafkaUrl);
props.put(ProducerConfig.RETRIES_CONFIG, Constants.retries);
//props.put(ProducerConfig.BATCH_SIZE_CONFIG, Constants.batchSize);
props.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, Constants.requestTimeout);
//props.put(ProducerConfig.LINGER_MS_CONFIG, Constants.linger);
//props.put(ProducerConfig.ACKS_CONFIG, Constants.acks);
//props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, Constants.bufferMemory);
//props.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, Constants.maxBlock);
props.put(ProducerConfig.CLIENT_ID_CONFIG, Constants.kafkaProducer);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
try {
karmaProducer = new org.apache.kafka.clients.producer.KafkaProducer<String, String>(props);
}
catch (Exception e) {
throw e;
}
}
/**
* get Producer based on topic
*
* #return
* #throws Exception
*/
public static Producer<String, String> getKarmaProducer(String topic) throws Exception {
switch (topic) {
case Constants.topicKarma :
if (karmaProducer == null) {
synchronized (KafkaProducer.class) {
if (karmaProducer == null) {
initProducer();
}
}
}
return karmaProducer;
default:
return null;
}
}
/**
* Flush and close kafka producer
*
* #throws Exception
*/
public static void closeKafkaInstance() throws Exception {
try {
karmaProducer.flush();
karmaProducer.close();
} catch (Exception e) {
throw e;
}
}
}
Kafka Producer
public class KafkaProducer {
public void sentToKafka(String topic, String data) {
Producer<String, String> producer = null;
try {
producer = KafkaManager.getKarmaProducer(topic);
ProducerRecord<String, String> producerRecord = new ProducerRecord<String, String>(topic, data);
producer.send(producerRecord);
} catch (Exception e) {
e.printStackTrace();
}
}
}
Main Class
public class App {
public static void main(String[] args) throws InterruptedException {
System.out.println("Hello World! I am producing to stream " + Constants.topicKarma);
String value = "google";
KafkaProducer kafkaProducer = new KafkaProducer();
for (int i = 1; i <= 1; i++) {
kafkaProducer.sentToKafka(Constants.topicKarma, value + i);
//Thread.sleep(100);
System.out.println("Send data to producer=" + value);
System.out.println("Send data to producer=" + value + i + " to tpoic=" + Constants.topicKarma);
}
}
}
What is my problem:
When my loop length if around 1000 (in class App), I am successfully able to send data to Kafka topic.
But when My loop length is 1 or less than 10, I am not able to send data to Kafka topic. Note I am not getting any error.
According to my finding, If I want to send a single message to Kafka topic, According to this program I get the successful message but never get a message on my topic.
But If I use Thread.sleep(10) (as you can see in my App class I have commented it), then I successfully send data on my topic.
Can you please explain why kafka showing this ambigous behabiour.

Each call to KafkaProducer.send() is returning a Future. You can use the last of those Futures to block the main Thread before exiting. Even easier, you can just call KafkaProducer.flush() after sending all your messages:
http://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html#flush()
Invoking this method makes all buffered records immediately available to send (even if linger.ms is greater than 0) and blocks on the completion of the requests associated with these records.

You are facing the problem because the producer executes sending in async way. When you send, the message is put inside an internal buffer in order to get a bigger batch and then send more messages with one shot.
This batching features is configured with batch.size and linger.ms, it means that messages are sent when the batch size reached that value or a linger time elapsed.
I have replied on something similar here : Cannot produce Message when Main Thread sleep less than 1000
Even you are saying "When my loop length if around 1000 (in class App), I am successfully able to send data to Kafka topic." ... but maybe you don't see all the sent messages because the latest batch isn't sent. With shorter loop the above conditions aren't reached in time so you shutdown the application before producer has enough time/batch size for sending.

Can you add Thread.sleep(100); just before exiting main?
If I understand correctly then everything works well if you sleep for a small amount of time. If that's the case, then it implies that your application is getting killed before the message is sent asynchronously.

JMS - Going from one to multiple consumers

I have a JMS client which is producing messages and sending over a JMS queue to its unique consumer.
What I want is more than one consumer getting those messages. The first thing that comes to my mind is converting the queue to a topic, so current and new consumers can subscribe and get the same message delivered to all of them.
This will obviously involve modifying the current clients code in both producer and consumer side of things.
I would like to also look at other options like creating a second queue, so that I don't have to modify the existing consumer. I believe there are advantages in this approach like (correct me if I am wrong) balancing the load between two different queues rather than one, which might have a positive impact on performance.
I would like to get advise on these options and cons / pros that you might see. Any feedback is highly appreciated.

You have a few options as you stated.
If you convert it to a topic to get the same effect you will need to make the consumers persistent consumers. One thing the queue offers is persistence if your consumer isn't alive. This will depend on the MQ system you are using.
If you want to stick with queues, you will create a queue for each consumer and a dispatcher that will listen on the original queue.
Producer -> Queue_Original <- Dispatcher -> Queue_Consumer_1 <- Consumer_1
-> Queue_Consumer_2 <- Consumer_2
-> Queue_Consumer_3 <- Consumer_3
Pros of Topics
Easier to dynamically add new consumers. All consumers will get new messages without any work.
You can create round-robin topics, so that Consumer_1 will get a message, then Consumer_2, then Consumer_3
Consumers can be pushed new messages, instead of having to query a queue making them reactive.
Cons of Topics
Messages are not persistent unless your Broker supports this configuration. If a consumer goes off line and comes back it is possible to have missed messages unless Persistent consumers are setup.
Difficult to allow Consumer_1 and Consumer_2 to receive a message but not Consumer_3. With a Dispatcher and Queues, the Dispatcher can not put a message in Consumer_3's queue.
Pros of Queues
Messages are persistent until a Consumer removes them
A dispatcher can filter which consumers get which messages by not placing messages into the respective consumers queues. This can be done with topics through filters though.
Cons of Queues
Additional Queues need to be created to support multiple consumers. In a dynamic environment this wouldn't be efficient.
When developing a Messaging System I prefer topics as it gives me the most power, but seeing as you are already using Queues it would require you to change how your system works to implement Topics instead.
Design and Implementation of Queue System with multiple consumers
Producer -> Queue_Original <- Dispatcher -> Queue_Consumer_1 <- Consumer_1
-> Queue_Consumer_2 <- Consumer_2
-> Queue_Consumer_3 <- Consumer_3
Source
Keep in mind there are other things you'll need to take care of such as problem exception handling, reconnection to the connection and queues if you lose your connection, etc. This is just designed to give you an idea of how to accomplish what I described.
In a real system I probably wouldn't exit out at the first exception. I would allow the system to continue operating the best it could and log errors. As it stands in this code if putting a message in a single consumers queue fails, the whole dispatcher will stop.
Dispatcher.java
/*
* To change this template, choose Tools | Templates
* and open the template in the editor.
*/
package stackoverflow_4615895;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.MessageProducer;
import javax.jms.Queue;
import javax.jms.QueueConnection;
import javax.jms.QueueConnectionFactory;
import javax.jms.QueueSession;
import javax.jms.Session;
public class Dispatcher {
private static long QUEUE_WAIT_TIME = 1000;
private boolean mStop = false;
private QueueConnectionFactory mFactory;
private String mSourceQueueName;
private String[] mConsumerQueueNames;
/**
* Create a dispatcher
* #param factory
* The QueueConnectionFactory in which new connections, session, and consumers
* will be created. This is needed to ensure the connection is associated
* with the correct thread.
* #param source
*
* #param consumerQueues
*/
public Dispatcher(
QueueConnectionFactory factory,
String sourceQueue,
String[] consumerQueues) {
mFactory = factory;
mSourceQueueName = sourceQueue;
mConsumerQueueNames = consumerQueues;
}
public void start() {
Thread thread = new Thread(new Runnable() {
public void run() {
Dispatcher.this.run();
}
});
thread.setName("Queue Dispatcher");
thread.start();
}
public void stop() {
mStop = true;
}
private void run() {
QueueConnection connection = null;
MessageProducer producer = null;
MessageConsumer consumer = null;
QueueSession session = null;
try {
// Setup connection and queues for receiving the messages
connection = mFactory.createQueueConnection();
session = connection.createQueueSession(false, Session.DUPS_OK_ACKNOWLEDGE);
Queue sourceQueue = session.createQueue(mSourceQueueName);
consumer = session.createConsumer(sourceQueue);
// Create a null producer allowing us to send messages
// to any queue.
producer = session.createProducer(null);
// Create the destination queues based on the consumer names we
// were given.
Queue[] destinationQueues = new Queue[mConsumerQueueNames.length];
for (int index = 0; index < mConsumerQueueNames.length; ++index) {
destinationQueues[index] = session.createQueue(mConsumerQueueNames[index]);
}
connection.start();
while (!mStop) {
// Only wait QUEUE_WAIT_TIME in order to give
// the dispatcher a chance to see if it should
// quit
Message m = consumer.receive(QUEUE_WAIT_TIME);
if (m == null) {
continue;
}
// Take the message we received and put
// it in each of the consumers destination
// queues for them to process
for (Queue q : destinationQueues) {
producer.send(q, m);
}
}
} catch (JMSException ex) {
// Do wonderful things here
} finally {
if (producer != null) {
try {
producer.close();
} catch (JMSException ex) {
}
}
if (consumer != null) {
try {
consumer.close();
} catch (JMSException ex) {
}
}
if (session != null) {
try {
session.close();
} catch (JMSException ex) {
}
}
if (connection != null) {
try {
connection.close();
} catch (JMSException ex) {
}
}
}
}
}
Main.java
QueueConnectionFactory factory = ...;
Dispatcher dispatcher =
new Dispatcher(
factory,
"Queue_Original",
new String[]{
"Consumer_Queue_1",
"Consumer_Queue_2",
"Consumer_Queue_3"});
dispatcher.start();

You may not have to modify the code; it depends on how you wrote it.
For example, if your code sends messages using MessageProducer rather than QueueSender, then it will work for topics as well as queues. Similarly if you used MessageConsumer rather than QueueReceiver.
Essentially, it is good practice in JMS applications to use non-specific interfaces to interact with the JMS system, such as MessageProducer, MessageConsumer, Destination, etc. If that's the case, it's a "mere" matter of configuration.