We Keep Coding

Kafka consumer unable to read data on Windows

I am trying to write a simple Kafka consumer to read the data on a windows machine.
but my consumer is not able to read any data. There are more than 20 messages produced by the producer but none is getting consumed.
Below is my code for the consumer :
Properties properties= new Properties();
properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,"localhost:9092");
properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,StringDeserializer.class.getName());
properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG,"my_application");
properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
// properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest");
//create the consumer
KafkaConsumer<String, String> consumer= new KafkaConsumer<String, String>(properties);
consumer.subscribe(Collections.singleton("first_topic"));
ConsumerRecords<String, String> record=consumer.poll(Duration.ofMillis(100));
for(ConsumerRecord data: record){
System.out.println("Key: "+data.key()+" and value: "+data.value());
System.out.println("Topic: "+data.topic());
System.out.println("Partition: "+data.partition());
}
The same issue was happening when I was using console to consume the messages using the command :
kafka-console-consumer.bat --bootstrap-server 127.0.0.1:9092 --from-beginning --topic first_topic --group my-application
I had to use --partition 0 option explicitly to get the messages.
Is there a way I can fix this ?

Use any of the following property in the code-
`properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest")
It tells the consumer to read only the latest messages, that is, the messages which were published after the consumer started.
`properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
When a consumer starts to consume the messages from the partition, it will start to read from the beginning.
Use the following command
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
----group my-application it should be --group my-application
https://kafka.apache.org/quickstart
Please follow the below steps.
a. Get the partitions of a topic
./kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic test
Topic: test PartitionCount: 1 ReplicationFactor: 1 Configs: segment.bytes=1073741824
Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0
b. Get the offset of the topic.
/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic test --time -1
test:0:11
c. consume the messages of a partition
/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --partition 0
You can also specify an --offset parameter that indicates which offset to start from. If absent, the consumption starts at the end of the partition.
You can also use an GUI based tool to view data in each partition of a topic named kafka tool. http://www.kafkatool.com It’s a tool to manage our kafka cluster.
Java Code
TopicPartition partition0 = new TopicPartition(topic, 0);
TopicPartition partition1 = new TopicPartition(topic, 1);
consumer.assign(Arrays.asList(partition0, partition1));
You can get the detailed implementation using java at the following URL.
https://kafka.apache.org/10/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
https://www.logicbig.com/tutorials/misc/kafka/manually-assign-partition-to-a-consumer.html

The group coordinator is not available-Kafka

When I am write a topic to kafka,there is an error:Offset commit failed:
2016-10-29 14:52:56.387 INFO [nioEventLoopGroup-3-1][org.apache.kafka.common.utils.AppInfoParser$AppInfo:82] - Kafka version : 0.9.0.1
2016-10-29 14:52:56.387 INFO [nioEventLoopGroup-3-1][org.apache.kafka.common.utils.AppInfoParser$AppInfo:83] - Kafka commitId : 23c69d62a0cabf06
2016-10-29 14:52:56.409 ERROR [nioEventLoopGroup-3-1][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$DefaultOffsetCommitCallback:489] - Offset commit failed.
org.apache.kafka.common.errors.GroupCoordinatorNotAvailableException: The group coordinator is not available.
2016-10-29 14:52:56.519 WARN [kafka-producer-network-thread | producer-1][org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater:582] - Error while fetching metadata with correlation id 0 : {0085000=LEADER_NOT_AVAILABLE}
2016-10-29 14:52:56.612 WARN [pool-6-thread-1][org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater:582] - Error while fetching metadata with correlation id 1 : {0085000=LEADER_NOT_AVAILABLE}
When create a new topic using command,it is ok.
./kafka-topics.sh --zookeeper localhost:2181 --create --topic test1 --partitions 1 --replication-factor 1 --config max.message.bytes=64000 --config flush.messages=1
This is the producer code using Java：
public void create() {
Properties props = new Properties();
props.clear();
String producerServer = PropertyReadHelper.properties.getProperty("kafka.producer.bootstrap.servers");
String zookeeperConnect = PropertyReadHelper.properties.getProperty("kafka.producer.zookeeper.connect");
String metaBrokerList = PropertyReadHelper.properties.getProperty("kafka.metadata.broker.list");
props.put("bootstrap.servers", producerServer);
props.put("zookeeper.connect", zookeeperConnect);//声明ZooKeeper
props.put("metadata.broker.list", metaBrokerList);//声明kafka broker
props.put("acks", "all");
props.put("retries", 0);
props.put("batch.size", 1000);
props.put("linger.ms", 10000);
props.put("buffer.memory", 10000);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
producer = new KafkaProducer<String, String>(props);
}
Where is wrong？

I faced a similar issue. The problem was when you start your Kafka broker there is a property associated with it, "KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR". If you are working with a single node cluster make sure you set this property with the value '1'. As its default value is 3. This change resolved my problem. (you can check the value in Kafka.properties file)
Note: I was using base image of confluent kafka version 4.0.0 ( confluentinc/cp-kafka:4.0.0)

Looking at your logs the problem is that cluster probably don't have connection to node which is the only one know replica of given topic in zookeeper.
You can check it using given command:
kafka-topics.sh --describe --zookeeper localhost:2181 --topic test1
or using kafkacat:
kafkacat -L -b localhost:9092
Example result:
Metadata for all topics (from broker 1003: localhost:9092/1003):
1 brokers:
broker 1003 at localhost:9092
1 topics:
topic "topic1" with 1 partitions:
partition 0, leader -1, replicas: 1001, isrs: , Broker: Leader not available
If you have single node cluster then broker id(1001) should match leader of topic1 partition.
But as you can see the only one known replica of topic1 was 1001 - which is not available now, so there is no possibility to recreate topic on different node.
The source of the problem can be an automatic generation of broker id(if you don't have specified broker.id or it is set to -1).
Then on starting the broker(the same single broker) you probably receive broker id different that previously and different than was marked in zookeeper (this a reason why partition deletion can help - but it is not a production solution).
The solution may be setting broker.id value in node config to fixed value - according to documentation it should be done on produciton environment:
broker.id=1
If everything is alright you should receive sth like this:
Metadata for all topics (from broker 1: localhost:9092/1001):
1 brokers:
broker 1 at localhost:9092
1 topics:
topic "topic1" with 1 partitions:
partition 0, leader 1, replicas: 1, isrs: 1
Kafka Documentation:
https://kafka.apache.org/documentation/#prodconfig

Hi you have to keep your kafka replicas and replication factor for your code same.
for me i keep 3 as replicas and 3 as replication factor.

The solution for me was that I had to make sure KAFKA_ADVERTISED_HOST_NAME was the correct IP address of the server.

We had the same issue and replicas and replication factors both were 3. and the Partition count was 1 . I increased the partition count to 10 and it started working.

We faced same issue in production too. The code was working fine for long time suddenly we got this exception.
We analyzed that there is no issue in code. So we asked deployment team to restart the zookeeper. Restarting it solved the issue.

Kafka consumer not consuming

Here is my code for building a kafka consumer from the java client.
def buildConsumer[Key, Value](
configuration: KafkaConfiguration, commitInterval: Long, groupId: Option[String] = None)(
implicit keyDeserializer: Deserializer[Key], valueDeserializer: Deserializer[Value]
): KafkaJavaConsumer[Key, Value] = {
val settingsMap: Map[String, Object] = Map(
"bootstrap.servers" -> s"${configuration.bootstrapHost}:${configuration.bootstrapPort}",
"group.id" -> groupId.getOrElse(s"${configuration.topic}-${UUID.randomUUID}"),
"enable.auto.commit" -> "true",
"auto.commit.interval.ms" -> commitInterval.toString,
"auto.offset.reset" -> "earliest"
) ++ configuration.additionalOptions.getOrElse(Map.empty[String, Object])
val consumer = new KafkaJavaConsumer[Key, Value](settingsMap.asJava, keyDeserializer, valueDeserializer)
consumer.subscribe(Seq(configuration.topic).asJava)
consumer
}
My kafka is running on port 6050 and I have tested it in the console to produce and consume from that specific port. Im wondering if my problem is related to my configuration above. I have also tested the code above with the EmbeddedKafka framework, the issue appears to be with an actual kafka server running.
EDIT:
I forgot to add that I have multiple consumers (with different group.id's) consuming from the same broker, not sure if this is the problem.

Make sure that,
No. of partitions in the topic >= No. of consumer instances in the
group
Otherwise, some of the consumer instances in the group won't be assigned any partitions.
To check the number of partitions, use the kafka-topics.sh command
> sh kafka-topics.sh --zookeeper localhost:2181 --topic test --describe
Topic:test PartitionCount:6 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: test Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: test Partition: 2 Leader: 0 Replicas: 0 Isr: 0
Topic: test Partition: 3 Leader: 0 Replicas: 0 Isr: 0
Topic: test Partition: 4 Leader: 0 Replicas: 0 Isr: 0
Topic: test Partition: 5 Leader: 0 Replicas: 0 Isr: 0

Im still not sure as to what the problem is, but by deleting the zookeeper data folder and all the kafka logs, the consumer/producer began to work as intended. I think this may have had to do with the problem of me deleting the log files to clear the topics without using the formal kafka admin tools for topic deletion.

Kafka: move partition from invalid broker

I have an invalid broker, which doesn't really exists. I started a server with invalid id by mistake, for a moment.
But one of the topics was assigned to this broker before:
Topic:raw-logs PartitionCount:3 ReplicationFactor:1 Configs:
Topic: raw-logs Partition: 0 Leader: -1 Replicas: 100 Isr:
Topic: raw-logs Partition: 1 Leader: none Replicas: 2 Isr:
Topic: raw-logs Partition: 2 Leader: none Replicas: 3 Isr:
I've already tried to reassign partitions with
{"topic":"raw-logs","partition":0,"replicas":[1]}
But it doesn't change anything. I guess Kafka waits until original Broker 100 will go up to copy data from it to Broker 1. Am I right?
But what if I don't don't want it? i'm ok to lose all data from this broker. Can I force Kafka to select new leader and rebalance topic? Manually, from command line

just guessing, but have you tried to use the bin/kafka-preferred-replica-election.sh --zookeeper zk_host:port/chroot method.

Kafka Consumer (Java) polls 0 messages

I am seeing this issue in my Kafka Java client where the consumer stop consuming after polling a few messages. Its not that the consumer hangs. Its unable to find messages in the topic partitions and polls 0 message. I have 4 partitions configured for the topic and 2 consumers for the consumer group.
Consumer Log:
Thread-5:2016-05-11 at 07:35:21.893 UTC INFO xxxxx.KafkaConsumerClient:71 pullFromQueue polled 0 messages from topic: test:[test-0, test-1] partition : []
Thread-5:2016-05-11 at 07:35:31.893 UTC INFO xxxxx.KafkaConsumerClient:71 pullFromQueue polled 0 messages from topic: test:[test-0, test-1] partition : []
Thread-5:2016-05-11 at 07:35:41.893 UTC INFO xxxxx.KafkaConsumerClient:71 pullFromQueue polled 0 messages from topic: test:[test-0, test-1] partition : []
Thread-5:2016-05-11 at 07:35:51.893 UTC INFO xxxxx.KafkaConsumerClient:71 pullFromQueue polled 0 messages from topic: test:[test-0, test-1] partition : []
Here, the log suggests that this consumer is connected to partitions 0 and 1 but is unable to consumer any message.
Consumer Offset:
Group Topic Pid Offset logSize Lag Owner
test-consumer test 0 1147335 1150034 2699 none
test-consumer test 1 1147471 1150033 2562 none
test-consumer test 2 1150035 1150035 0 none
test-consumer test 3 1150031 1150031 0 none
Here this shows that my topic has 2699 and 2562 messages pending on partitions 0 and 1 respectively