We Keep Coding

Kafka consumer unable to read data on Windows

I am trying to write a simple Kafka consumer to read the data on a windows machine.
but my consumer is not able to read any data. There are more than 20 messages produced by the producer but none is getting consumed.
Below is my code for the consumer :
Properties properties= new Properties();
properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,"localhost:9092");
properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,StringDeserializer.class.getName());
properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG,"my_application");
properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
// properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest");
//create the consumer
KafkaConsumer<String, String> consumer= new KafkaConsumer<String, String>(properties);
consumer.subscribe(Collections.singleton("first_topic"));
ConsumerRecords<String, String> record=consumer.poll(Duration.ofMillis(100));
for(ConsumerRecord data: record){
System.out.println("Key: "+data.key()+" and value: "+data.value());
System.out.println("Topic: "+data.topic());
System.out.println("Partition: "+data.partition());
}
The same issue was happening when I was using console to consume the messages using the command :
kafka-console-consumer.bat --bootstrap-server 127.0.0.1:9092 --from-beginning --topic first_topic --group my-application
I had to use --partition 0 option explicitly to get the messages.
Is there a way I can fix this ?

Use any of the following property in the code-
`properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest")
It tells the consumer to read only the latest messages, that is, the messages which were published after the consumer started.
`properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
When a consumer starts to consume the messages from the partition, it will start to read from the beginning.
Use the following command
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
----group my-application it should be --group my-application
https://kafka.apache.org/quickstart
Please follow the below steps.
a. Get the partitions of a topic
./kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic test
Topic: test PartitionCount: 1 ReplicationFactor: 1 Configs: segment.bytes=1073741824
Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0
b. Get the offset of the topic.
/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic test --time -1
test:0:11
c. consume the messages of a partition
/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --partition 0
You can also specify an --offset parameter that indicates which offset to start from. If absent, the consumption starts at the end of the partition.
You can also use an GUI based tool to view data in each partition of a topic named kafka tool. http://www.kafkatool.com It’s a tool to manage our kafka cluster.
Java Code
TopicPartition partition0 = new TopicPartition(topic, 0);
TopicPartition partition1 = new TopicPartition(topic, 1);
consumer.assign(Arrays.asList(partition0, partition1));
You can get the detailed implementation using java at the following URL.
https://kafka.apache.org/10/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
https://www.logicbig.com/tutorials/misc/kafka/manually-assign-partition-to-a-consumer.html

The messages are not getting deleted from the file system when deleteRecords Kafka Admin Client Java API is invoked

I was trying to delete messages from my kafka topic using Java Admin Client API's delete Records method. Following are the steps that i have tried
1. I pushed 20000 records to my TEST-DELETE topic
2. Started a console consumer and consumed all the messages
3. Invoked my java program to delete all those 20k messages
4. Started another console consumer with a different group id. This consumer is not receiving any of the deleted messages
When I checked the file system, I could still see all those 20k records occupying the disk space. My intention is to delete those records forever from file system too.
My Topic configuration is given below along with server.properties settings
Topic:TEST-DELETE PartitionCount:4 ReplicationFactor:1 Configs:cleanup.policy=delete
Topic: TEST-DELETE Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: TEST-DELETE Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: TEST-DELETE Partition: 2 Leader: 0 Replicas: 0 Isr: 0
Topic: TEST-DELETE Partition: 3 Leader: 0 Replicas: 0 Isr: 0
log.retention.hours=24
log.retention.check.interval.ms=60000
log.cleaner.delete.retention.ms=60000
file.delete.delay.ms=60000
delete.retention.ms=60000
offsets.retention.minutes=5
offsets.retention.check.interval.ms=60000
log.cleaner.enable=true
log.cleanup.policy=compact,delete
My delete code is given below
public void deleteRecords(Map<String, Map<Integer, Long>> allTopicPartions) {
Map<TopicPartition, RecordsToDelete> recordsToDelete = new HashMap<>();
allTopicPartions.entrySet().forEach(topicDetails -> {
String topicName = topicDetails.getKey();
Map<Integer, Long> value = topicDetails.getValue();
value.entrySet().forEach(partitionDetails -> {
if (partitionDetails.getValue() != 0) {
recordsToDelete.put(new TopicPartition(topicName, partitionDetails.getKey()),
RecordsToDelete.beforeOffset(partitionDetails.getValue()));
}
});
});
DeleteRecordsResult deleteRecords = this.client.deleteRecords(recordsToDelete);
Map<TopicPartition, KafkaFuture<DeletedRecords>> lowWatermarks = deleteRecords.lowWatermarks();
lowWatermarks.entrySet().forEach(entry -> {
try {
logger.info(entry.getKey().topic() + " " + entry.getKey().partition() + " "
+ entry.getValue().get().lowWatermark());
} catch (Exception ex) {
}
});
}
The output of my java program is given below
2019-06-25 16:21:15 INFO MyKafkaAdminClient:247 - TEST-DELETE 1 5000
2019-06-25 16:21:15 INFO MyKafkaAdminClient:247 - TEST-DELETE 0 5000
2019-06-25 16:21:15 INFO MyKafkaAdminClient:247 - TEST-DELETE 3 5000
2019-06-25 16:21:15 INFO MyKafkaAdminClient:247 - TEST-DELETE 2 5000
My intention is to delete the consumed records from the file system as I am working with limited storage for my kafka broker.
I would like to get some help with my below doubts
I was in the impression that the delete Records will remove the messages from the file system too, but look like I got it wrong!!
How long those deleted records be present in the log directory?
Is there any specific configuration that i need to use in order to remove the records from the files system once the delete Records API is invoked?
Appreciate your help
Thanks

The recommended approach to handle this is to set retention.ms and related configuration values for the topics you're interested in. That way, you can define how long Kafka will store your data until it deletes it, making sure all your downstream consumers have had the chance to pull down the data before it's deleted from the Kafk cluster.
If, however, you still want to force Kafka to delete based on bytes, there's the log.retention.bytes and retention.bytes configuration values. The first one is a cluster-wide setting, the second one is the topic-specific setting, which by default takes whatever the first one is set to, but you can still override it per topic. The retention.bytes number is enforced per partition, so you should multiply it by the total number of topic partitions.
Be aware, however, that if you have a run-away producer that starts generating a lot of data suddenly, and you have it set to a hard byte limit, you might wipe out entire days worth of data in the cluster, and only be left with the last few minutes of data, maybe before even valid consumers can pull down the data from the cluster. This is why it's much better to set your kafka topics to have time-based retention, and not byte-based.
You can find the configuration properties and their explanation in the official Kafka docs: https://kafka.apache.org/documentation/

Apache Kafka consumer always lagging

I have the following Kafka cluster: 1 zk and 2 brokers, 1zk+1broker on one machine and 2broker on another.
I have the topic with repl-factor=2 partitions=2
I send around 4kmessages into this topic using my producer.
Also, I have a consumer that simply do nothing but consuming records:
try {
consumer.subscribe(topics);
while (true) {
ConsumerRecords<String, String> records = consumer.poll(0);
for (ConsumerRecord<String, String> record : records) {
}
}
} catch (WakeupException e) {
} finally {
consumer.close();
}
Consumer has AUTO_COMMIT_INTERVAL_MS_CONFIG set to 100ms;
So the question is why I always observe the lag between consumer and producer?
I use the following command to get the lag:
./bin/kafka-consumer-groups.sh --describe --group events-group-test --bootstrap-server kafka01:9092
I have cmd script that simply prints lag for me in a loop:
-bash-4.2$ while true; do date +%H:%M:%S;./bin/kafka-consumer-groups.sh --describe --group events-group-test --bootstrap-server kafka01:9092 | awk '$5 ~ /[0-9.]+/ { print "part:" $2" lag: "$5}'; sleep 20; done prints
11:31:04
Note: This will not show information about old Zookeeper-based consumers.
part:0 lag: 1207
part:1 lag: 1100
11:31:29
Note: This will not show information about old Zookeeper-based consumers.
part:0 lag: 5476
part:1 lag: 4692
11:31:53
Note: This will not show information about old Zookeeper-based consumers.
part:0 lag: 3389
part:1 lag: 1646
11:32:16
Note: This will not show information about old Zookeeper-based consumers.
part:0 lag: 1365
part:1 lag: 593
11:32:39
Note: This will not show information about old Zookeeper-based consumers.
part:0 lag: 4575
part:1 lag: 3488
11:33:03
Note: This will not show information about old Zookeeper-based consumers.
As you see it always shows the lag ~3-4k messages but my consumer literally does nothing with the records, so it should consume instantly and I expect the 0 lag.
Also, I've tried to decrease producer load to 30 messages per sec, but the consumer still shows the LAG (~30-50). Is it possible to reach 0 LAG?

The group coordinator is not available-Kafka

When I am write a topic to kafka,there is an error:Offset commit failed:
2016-10-29 14:52:56.387 INFO [nioEventLoopGroup-3-1][org.apache.kafka.common.utils.AppInfoParser$AppInfo:82] - Kafka version : 0.9.0.1
2016-10-29 14:52:56.387 INFO [nioEventLoopGroup-3-1][org.apache.kafka.common.utils.AppInfoParser$AppInfo:83] - Kafka commitId : 23c69d62a0cabf06
2016-10-29 14:52:56.409 ERROR [nioEventLoopGroup-3-1][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$DefaultOffsetCommitCallback:489] - Offset commit failed.
org.apache.kafka.common.errors.GroupCoordinatorNotAvailableException: The group coordinator is not available.
2016-10-29 14:52:56.519 WARN [kafka-producer-network-thread | producer-1][org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater:582] - Error while fetching metadata with correlation id 0 : {0085000=LEADER_NOT_AVAILABLE}
2016-10-29 14:52:56.612 WARN [pool-6-thread-1][org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater:582] - Error while fetching metadata with correlation id 1 : {0085000=LEADER_NOT_AVAILABLE}
When create a new topic using command,it is ok.
./kafka-topics.sh --zookeeper localhost:2181 --create --topic test1 --partitions 1 --replication-factor 1 --config max.message.bytes=64000 --config flush.messages=1
This is the producer code using Java：
public void create() {
Properties props = new Properties();
props.clear();
String producerServer = PropertyReadHelper.properties.getProperty("kafka.producer.bootstrap.servers");
String zookeeperConnect = PropertyReadHelper.properties.getProperty("kafka.producer.zookeeper.connect");
String metaBrokerList = PropertyReadHelper.properties.getProperty("kafka.metadata.broker.list");
props.put("bootstrap.servers", producerServer);
props.put("zookeeper.connect", zookeeperConnect);//声明ZooKeeper
props.put("metadata.broker.list", metaBrokerList);//声明kafka broker
props.put("acks", "all");
props.put("retries", 0);
props.put("batch.size", 1000);
props.put("linger.ms", 10000);
props.put("buffer.memory", 10000);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
producer = new KafkaProducer<String, String>(props);
}
Where is wrong？

I faced a similar issue. The problem was when you start your Kafka broker there is a property associated with it, "KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR". If you are working with a single node cluster make sure you set this property with the value '1'. As its default value is 3. This change resolved my problem. (you can check the value in Kafka.properties file)
Note: I was using base image of confluent kafka version 4.0.0 ( confluentinc/cp-kafka:4.0.0)

Looking at your logs the problem is that cluster probably don't have connection to node which is the only one know replica of given topic in zookeeper.
You can check it using given command:
kafka-topics.sh --describe --zookeeper localhost:2181 --topic test1
or using kafkacat:
kafkacat -L -b localhost:9092
Example result:
Metadata for all topics (from broker 1003: localhost:9092/1003):
1 brokers:
broker 1003 at localhost:9092
1 topics:
topic "topic1" with 1 partitions:
partition 0, leader -1, replicas: 1001, isrs: , Broker: Leader not available
If you have single node cluster then broker id(1001) should match leader of topic1 partition.
But as you can see the only one known replica of topic1 was 1001 - which is not available now, so there is no possibility to recreate topic on different node.
The source of the problem can be an automatic generation of broker id(if you don't have specified broker.id or it is set to -1).
Then on starting the broker(the same single broker) you probably receive broker id different that previously and different than was marked in zookeeper (this a reason why partition deletion can help - but it is not a production solution).
The solution may be setting broker.id value in node config to fixed value - according to documentation it should be done on produciton environment:
broker.id=1
If everything is alright you should receive sth like this:
Metadata for all topics (from broker 1: localhost:9092/1001):
1 brokers:
broker 1 at localhost:9092
1 topics:
topic "topic1" with 1 partitions:
partition 0, leader 1, replicas: 1, isrs: 1
Kafka Documentation:
https://kafka.apache.org/documentation/#prodconfig

Hi you have to keep your kafka replicas and replication factor for your code same.
for me i keep 3 as replicas and 3 as replication factor.

The solution for me was that I had to make sure KAFKA_ADVERTISED_HOST_NAME was the correct IP address of the server.

We had the same issue and replicas and replication factors both were 3. and the Partition count was 1 . I increased the partition count to 10 and it started working.

We faced same issue in production too. The code was working fine for long time suddenly we got this exception.
We analyzed that there is no issue in code. So we asked deployment team to restart the zookeeper. Restarting it solved the issue.

Kafka Consumer (Java) polls 0 messages

I am seeing this issue in my Kafka Java client where the consumer stop consuming after polling a few messages. Its not that the consumer hangs. Its unable to find messages in the topic partitions and polls 0 message. I have 4 partitions configured for the topic and 2 consumers for the consumer group.
Consumer Log:
Thread-5:2016-05-11 at 07:35:21.893 UTC INFO xxxxx.KafkaConsumerClient:71 pullFromQueue polled 0 messages from topic: test:[test-0, test-1] partition : []
Thread-5:2016-05-11 at 07:35:31.893 UTC INFO xxxxx.KafkaConsumerClient:71 pullFromQueue polled 0 messages from topic: test:[test-0, test-1] partition : []
Thread-5:2016-05-11 at 07:35:41.893 UTC INFO xxxxx.KafkaConsumerClient:71 pullFromQueue polled 0 messages from topic: test:[test-0, test-1] partition : []
Thread-5:2016-05-11 at 07:35:51.893 UTC INFO xxxxx.KafkaConsumerClient:71 pullFromQueue polled 0 messages from topic: test:[test-0, test-1] partition : []
Here, the log suggests that this consumer is connected to partitions 0 and 1 but is unable to consumer any message.
Consumer Offset:
Group Topic Pid Offset logSize Lag Owner
test-consumer test 0 1147335 1150034 2699 none
test-consumer test 1 1147471 1150033 2562 none
test-consumer test 2 1150035 1150035 0 none
test-consumer test 3 1150031 1150031 0 none
Here this shows that my topic has 2699 and 2562 messages pending on partitions 0 and 1 respectively