We Keep Coding

How to get rid of "Closing the Kafka producer with timeoutMillis = ..."

I am sending Apache Avro formatted messages to a Kafka broker instance via the following code:
ProducerRecord<String, byte[]> producerRecord = new ProducerRecord<>(kafkaTopic.getTopicName(), null, null,
avroConverter.getSchemaId().toString(), convertRecordToByteArray(kafkaRecordToSend));
String avroSchemaName = null;
// some of my AVRO schemas are unions, some are simple:
if (_avroSchema.getTypes().size() == 1) {
avroSchemaName = _avroSchema.getTypes().get(0).getName();
} else if (_avroSchema.getTypes().size() == 2) {
avroSchemaName = _avroSchema.getTypes().get(1).getName();
}
// some custom header items...
producerRecord.headers().add(MessageHeaders.MESSAGE_ID.getText(), messageID.getBytes());
producerRecord.headers().add(MessageHeaders.AVRO_SCHEMA_REGISTRY_SUBJECT.getText(),
avroSchemaName.getBytes());
producerRecord.headers().add(MessageHeaders.AVRO_SCHEMA_REGISTRY_SCHEMA_ID.getText(),
avroConverter.getSchemaId().toString().getBytes());
if (multiline) {
producerRecord.headers().add(MessageHeaders.AVRO_SCHEMA_MULTILINE_RECORD_NAME.getText(),
MULTILINE_RECORD_NAME.getBytes());
}
try {
Future<RecordMetadata> result = kafkaProducer.send(producerRecord);
RecordMetadata sendResult = result.get();
MessageLogger.logResourceBundleMessage(_messages, "JAPCTOAVROKAFKAPRODUCER:DEBUG0002",
sendResult.offset());
} catch (Exception e) {
MessageLogger.logError(e);
throw e;
}
The code works fine, the messages end up in Kafka and are processed to end up in an InfluxDB. The problem is that every send operation produces a lot of INFO messages (client ID number is an example):
[Producer clientId=producer-27902] Closing the Kafka producer with timeoutMillis = 10000 ms.
[Producer clientId=producer-27902] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
Kafka startTimeMs: ...
Kafka commitId: ...
[Producer clientId=producer-27902] Cluster ID:
which Spam our Graylog.
I use similar code to send String formatted messages. This code is executed without producing INFO messages...
ProducerRecord<String, String> recordToSend = new ProducerRecord<>(queueName, messageText);
recordToSend.headers().add("messageID", messageID.getBytes());
Future<RecordMetadata> result = _producerConnection.send(recordToSend);
I know that the INFO message are logged from class org.apache.kafka.clients.producer.KafkaProducer. I need to get rid of these messages, but I do not have access to the logging.mxl defining the logger properties for Graylog.
Is there a way to get rid of these messages via POM-entries or programatically?

The reason of the code behavior was a design flaw:
The code in the post above was placed in a method which was called for sending the message to Kafka. The KafkaProducer class was instantiated in that method and each time when the method was called. Surprisingly, KafkaProducer issues the Closing the KafkaProducer with timeoutMillis = not only at an explicit close() by the calling code, but as well when the strong reference to the instance is lost ( in my case when the code leaves the method), In the latter case, the timeoutMillis is set to 9223372036854775807 (largest long number).
To get rid of the many messages, I moved the KafkaProducer instantiation out of the method and made the instance variable a class attribute and I do not call an explicit close() after the send(...) any more.
Furthermore I changed the instance of my class instantiating KafkaProducer to a strong referencing class member.
By doing so, I get some messages by the KafkaProducer at instantiation, then there is silence.

Redis Streams one message per consumer with Java

I'am trying to implement a java application with redis streams where every consomer consumes exactly one message. Like a pipeline/queue where every consumer takes exactly one message, processes it and after finishing the consumer takes the next message which was not processed so far in the stream.
What works is that every message is consumed by exactly one consumer (with xreadgroup).
I started with this tutorial from redislabs
The code:
RedisClient redisClient = RedisClient.create("redis://pw#host:port");
StatefulRedisConnection<String, String> connection = redisClient.connect();
RedisCommands<String, String> syncCommands = connection.sync();
try {
syncCommands.xgroupCreate(XReadArgs.StreamOffset.from(STREAM_KEY, "0-0"), ID_READ_GROUP);
} catch (RedisBusyException redisBusyException) {
System.out.println(String.format("\t Group '%s' already exists", ID_READ_GROUP));
}
System.out.println("Waiting for new messages ");
while (true) {
List<StreamMessage<String, String>> messages = syncCommands.xreadgroup(
Consumer.from(ID_READ_GROUP, ID_WORKER), ReadArgs.StreamOffset.lastConsumed(STREAM_KEY));
if (!messages.isEmpty()) {
System.out.println(messages.size()); //
for (StreamMessage<String, String> message : messages) {
System.out.println(message.getId());
Thread.sleep(5000);
syncCommands.xack(STREAM_KEY, ID_READ_GROUP, message.getId());
}
}
}
My current problem is that a consumer takes more that one message from the queue and in some situations the other consumers are waiting and one consumer is processing 10 messages at once.
Thanks in advance!

Notice that XREADGROUP can get COUNT argument.
See the JavaDoc how to do it in Lettuce xreadgroup, by passing XReadArgs.

Kafka CommitFailedException when running 2 consumers separately with different topics

I am trying to run 2 consumers subscribed to 2 different topics. Both the consumer programs run properly when running one at a time, but when running them at the same time, one consumer always displays the exception:
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
I followed the suggestions and have set the max.pool.size to 2, and session.timeout.ms to 30000, heartbeat.interval.ms to 1000
Below is my consumer function, this function is same for both the files, only the topic name changes to Test2, and I am running these two functions in 2 different classes running both at the same time.
public void consume()
{
//Kafka consumer configuration settings
List<String> topicNames = new ArrayList<String>();
topicNames.add("Test1");
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test");
props.put("enable.auto.commit", "false");
props.put("session.timeout.ms", "30000");
props.put("heartbeat.interval.ms", "1000");
props.put("max.poll.records", "2");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);
consumer.subscribe(topicNames);
try {
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records) {
System.out.println("Record: "+record.value());
String responseString = "successfull";
if (responseString.equals("successfull")) {
consumer.commitSync();
}
}
}
}
catch (Exception e) {
LOG.error("Exception: ", e);
}
finally {
consumer.close();
}
}
Due to this error, the records are not getting committed in the Kafka topic.
How do I overcome this error?

In your case you need to assign different group IDs to consumer. You are making two consumer with same group ID (that is okay), but calling subscribe twice is not okay.
You are able to run one consumer at a time because you are calling subscribe only once.
If you need any further help let me know. Happy to help.

How to ensure messages reach kafka broker?

I have have a message producer on my local machine and a broker on remote host (aws).
After sending a message from the producer,
I wait and call the console consumer on the remote host and
see excessive logs.
Without the value from producer.
The producer flushes the data after calling the send method.
Everything is configured correctly.
How can I check to see that the broker received the message from the producer and to see if the producer received the answer?

The Send method asynchronously sends the message to the topic and
returns a Future of RecordMetadata.
java.util.concurrent.Future<RecordMetadata> send(ProducerRecord<K,V> record)
Asynchronously sends a record to a topic
After the flush call,
check to see that the Future has completed by calling the isDone method.
(for example, Future.isDone() == true)
Invoking this method makes all buffered records immediately available to send (even if linger.ms is greater than 0) and blocks on the completion of the requests associated with these records. The post-condition of flush() is that any previously sent record will have completed (e.g. Future.isDone() == true). A request is considered completed when it is successfully acknowledged according to the acks configuration you have specified or else it results in an error.
The RecordMetadata contains the offset and the partition
public int partition()
The partition the record was sent to
public long offset()
the offset of the record, or -1 if {hasOffset()} returns false.
Or you can also use Callback function to ensure messages was sent to topic or not
Fully non-blocking usage can make use of the Callback parameter to provide a callback that will be invoked when the request is complete.
here is clear example in docs
ProducerRecord<byte[],byte[]> record = new ProducerRecord<byte[],byte[]>("the-topic", key, value);
producer.send(myRecord,
new Callback() {
public void onCompletion(RecordMetadata metadata, Exception e) {
if(e != null) {
e.printStackTrace();
} else {
System.out.println("The offset of the record we just sent is: " + metadata.offset());
}
}
});

You can try get() API of send , which will return the Future of RecordMetadata
ProducerRecord<String, String> record =
new ProducerRecord<>("SampleTopic", "SampleKey", "SampleValue");
try {
producer.send(record).get();
} catch (Exception e) {
e.printStackTrace();
}

Use exactly-once-delivery and you won't need to worry about whether your message reached or not: https://www.baeldung.com/kafka-exactly-once, https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/

CommitFailedException Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member

I was using kafka 0.10.2 and now faced a CommitFailedException. like:
Commit cannot be completed since the group has already rebalanced and
assigned the partitions to another member. This means that the time
between subsequent calls to poll() was longer than the configured
max.poll.interval.ms, which typically implies that the poll loop is
spending too much time message processing. You can address this either
by increasing the session timeout or by reducing the maximum size of
batches returned in poll() with max.poll.records.
I have set max.poll.interval.ms to Integer.MAX_VALUE. so can anyone tell me why this still happens even I have set the value ?
Another question is:
I do as description to set session.timeout.ms to 60000 and it still happens. I try to reproduce by a simple code
public static void main(String[] args) throws InterruptedException {
Logger logger = Logger.getLogger(KafkaConsumer10.class);
logger.info("XX");
Properties props = new Properties();
props.put("bootstrap.servers", "kafka-broker:9098");
props.put("group.id", "test");
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "1000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("max.poll.interval.ms", "300000");
props.put("session.timeout.ms", "10000");
props.put("max.poll.records", "2");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("t1"));
while (true) {
Thread.sleep(11000);
ConsumerRecords<String, String> records = consumer.poll(100);
//Thread.sleep(11000);
Thread.sleep(11000);
for (ConsumerRecord<String, String> record : records)
System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
}
when I set session.timeout.ms to 10000, I try to sleep more than 10000 ms in my poll loop, but it seems work and no Exception out. so I'm confused about this. if heartbeat is triggered by consumer.poll and consumer.commit, seems heartbeat is out of session timeout in my code. why not throw CommitFailedException ?

session.timeout.ms set on the consumer should be less than the group.max.session.timeout.ms set on Kafka broker.
This resolved the issue for me.
Credit to github link Commit Failures

Hi For this you need to handle the rebalancing condition in your code and should process the ongoing message and commit it before rebalancing
Like :
private class HandleRebalance implements ConsumerRebalanceListener {
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
// Implement what you want to do once rebalancing is done.
}
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
// commit current method
}
}
and Use this syntax for subscribing the topic :
kafkaConsumer.subscribe(topicNameList , new HandleRebalance())
The advantage of doing this :
Messages will not repeat when the rebalancing is taking place.
No commit fail exception