We Keep Coding

Kafka - Why is default serde for key and value required for groupBy and reduce?

I am developing a streaming application with Quarkus. My application looks as follows.
flatMap to change key and generate multiple messages from a single message.
join with a KTable using key from Step 1.
transform for a stateful operation.
flatMap change the key back to original key i.e., before Step 1.
groupBy with the key as in Step 4, which is actually that in, Step 1.
reduce to "merge" the records into a single message comprising a JSON array.
The net effect is to split an incoming message (with key as id1) into multiple messages (with different keys e.g., k1, k2, etc.). Enhance each of the message using join and transform. Then, change the key of each message back to id1. Finally, "merge" each of the enhanced message into a single message with key as id1.
I keep getting an error to set-up default key serde and value serde. While the default serde can be set in application.properties, I am not clear, why does this error even arise?
Note that, if I do not do Step 5 and Step 6, the application works successfully.
This is the Java exception I get.
2022-10-17 16:42:34,884 ERROR [org.apa.kaf.str.KafkaStreams] (app-alerts-6a7c4df8-7813-4d5d-9a86-d6f3db7c8ef0-StreamThread-1) stream-client [app-alerts-6a7c4df8-7813-4d5d-9a86-d6f3db7c8ef0] Encountered the following exception during processing and the registered exception handler opted to SHUTDOWN_CLIENT. The streams client is going to shut down now. : org.apache.kafka.streams.errors.StreamsException: org.apache.kafka.common.config.ConfigException: Please specify a key serde or set one through StreamsConfig#DEFAULT_KEY_SERDE_CLASS_CONFIG
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:627)
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:551)
Caused by: org.apache.kafka.common.config.ConfigException: Please specify a key serde or set one through StreamsConfig#DEFAULT_KEY_SERDE_CLASS_CONFIG
at org.apache.kafka.streams.StreamsConfig.defaultKeySerde(StreamsConfig.java:1587)
at org.apache.kafka.streams.processor.internals.AbstractProcessorContext.keySerde(AbstractProcessorContext.java:90)
at org.apache.kafka.streams.processor.internals.SerdeGetter.keySerde(SerdeGetter.java:47)
at org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.prepareSerde(WrappingNullableUtils.java:63)
at org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.prepareKeySerde(WrappingNullableUtils.java:90)
at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.initStoreSerde(MeteredKeyValueStore.java:195)
at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:144)
at org.apache.kafka.streams.processor.internals.ProcessorStateManager.registerStateStores(ProcessorStateManager.java:212)
at org.apache.kafka.streams.processor.internals.StateManagerUtil.registerStateStores(StateManagerUtil.java:97)
at org.apache.kafka.streams.processor.internals.StreamTask.initializeIfNeeded(StreamTask.java:231)
at org.apache.kafka.streams.processor.internals.TaskManager.tryToCompleteRestoration(TaskManager.java:454)
at org.apache.kafka.streams.processor.internals.StreamThread.initializeAndRestorePhase(StreamThread.java:865)
at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:747)
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:589)
... 1 more
These are StreamsConfig values:
acceptable.recovery.lag = 10000
application.id = machine-alerts
application.server =
bootstrap.servers = [kafka:9092]
buffered.records.per.partition = 1000
built.in.metrics.version = latest
cache.max.bytes.buffering = 10240
client.id =
commit.interval.ms = 1000
connections.max.idle.ms = 540000
default.deserialization.exception.handler = class org.apache.kafka.streams.errors.LogAndFailExceptionHandler
default.dsl.store = rocksDB
default.key.serde = null
default.list.key.serde.inner = null
default.list.key.serde.type = null
default.list.value.serde.inner = null
default.list.value.serde.type = null
default.production.exception.handler = class org.apache.kafka.streams.errors.DefaultProductionExceptionHandler
default.timestamp.extractor = class org.apache.kafka.streams.processor.FailOnInvalidTimestamp
default.value.serde = null
max.task.idle.ms = 0
max.warmup.replicas = 2
metadata.max.age.ms = 500
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = DEBUG
metrics.sample.window.ms = 30000
num.standby.replicas = 0
num.stream.threads = 1
poll.ms = 100
probing.rebalance.interval.ms = 600000
processing.guarantee = at_least_once
rack.aware.assignment.tags = []
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
repartition.purge.interval.ms = 30000
replication.factor = -1
request.timeout.ms = 40000
retries = 0
retry.backoff.ms = 100
rocksdb.config.setter = null
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
state.cleanup.delay.ms = 600000
state.dir = /tmp/kafka-streams
task.timeout.ms = 300000
topology.optimization = none
upgrade.from = null
window.size.ms = null
windowed.inner.class.serde = null
windowstore.changelog.additional.retention.ms = 86400000

I don't know how Quarkus works, However, the Serde error in groupBy statements is common when you start using Kafka Streams. The groupBy statement creates an internal and compacted topic where the data is going to be grouped by key, (Internally your stream application is going to send messages to that topic) for that reason you should specify the Serde for your GroupBy statement at the code level if the value or key are different types as default key and value Serde in your Stream properties.
KGroupedStream<String, User> groupedStream = stream.groupByKey(
Serialized.with(
Serdes.String(), /* key */
new CustomUserSerde()) /* value */
);
In the above example, I'm using the desired Serdes in the group statement, not at the property level.

Kafka Cluster sometimes returns no records during seek and poll

I'm experimenting with a Kafka clusters (3 nodes) and I was intending to run some tests around redundancy and availability (stopping nodes in cluster, etc) with a simple java app using the following kafka client dependency: -
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>2.8.0</version>
</dependency>
I'd configured a replication factor of 3 to ensure topics are replicated across all nodes and I'm using only 1 partition for the topic. I'm struggling to understand some behaviour I'm seeing on this sample code when specifically seeking to an offset (with one node offline):-
String topic = "test-topic";
TopicPartition partition = new TopicPartition(topic, 0);
List<TopicPartition> partitions = Collections.singletonList(partition);
while (true) {
Consumer<String, String> consumer = createConsumer();
consumer.assign(partitions);
consumer.seek(partition, 0);
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(2000));
if (records.isEmpty())
System.out.println("No Records Found");
else
System.out.println("Records Found: " + records.count());
consumer.close();
Thread.sleep(2000);
}
This code will on occasion return "No Records Found" when one of the nodes in the cluster is offline:-
No Records
Found Records Found: 1
No Records Found
Records Found: 1
Records Found: 1 Records Found: 1 No Records Found Records Found: 1
Records Found: 1 Records Found: 1 Records Found: 1 Records Found: 1
Records Found: 1 No Records Found Records Found: 1 Records Found: 1
Records Found: 1 Records Found: 1 Records Found: 1 No Records Found
You'll notice that I'm creating the consumer each time inside the while loop. This is to simulate different consumers coming in and connecting as each consumer has a different consumer group id. Moving the consumer creation outside of the while loop (and removing consumer.close()) gives mostly expected results, i.e. all logs show "Records Found: 1". However, "sometimes" the very first poll will return no records, with all remaining showing 1 record found:-
String topic = "test-topic";
TopicPartition partition = new TopicPartition(topic, 0);
List<TopicPartition> partitions = Collections.singletonList(partition);
Consumer<String, String> consumer = createConsumer();
while (true) {
consumer.assign(partitions);
consumer.seek(partition, 0);
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(2000));
if (records.isEmpty())
System.out.println("No Records Found");
else
System.out.println("Records Found: " + records.count());
Thread.sleep(2000);
}
createConsumer code is defined as follows for reference: -
public static Consumer<String, String> createConsumer() {
Properties config = new Properties();
config.put(ConsumerConfig.GROUP_ID_CONFIG, "test-consumer-" + UUID.randomUUID().toString());
config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "node-1:9092, node-2:9092, node-3:9092");
config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
config.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 500);
config.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
config.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest" );
config.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 10000);
Consumer<String, String> consumer = new KafkaConsumer<String, String>(config);
return consumer;
}
I'd like to understand this behaviour to be able to reliably run my availability tests.

I am also stuck with this problem, Finally solved it like this:
public ConsumerRecord<String, String> seekAndPoll(String topic, int partition, long offset) {
TopicPartition tp = new TopicPartition(topic, partition);
consumer.assign(Collections.singleton(tp));
System.out.println("assignment:" + consumer.assignment()); // 这里是有分配到分区的
// endOffset: the offset of the last successfully replicated message plus one
// if there has 5 messages, valid offsets are [0,1,2,3,4], endOffset is 4+1=5
Long endOffset = consumer.endOffsets(Collections.singleton(tp)).get(tp);
if (offset < 0 || offset >= endOffset) {
System.out.println("offset is illegal");
return null;
} else {
consumer.seek(tp, offset);
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100))
if(records.isEmpty()){
System.out.println("Not Found");
return null;
} else {
System.out.println("Found");
return records.iterator().next();
}
}
}

Why are all the partitions not assinged to the single KafkaConsumer that is running?

I am trying to read the last 3 records from the topic "input_topic".
I am using only a single consumer.
But it is consuming record from only one partition.
When I manually assigned other partitions, an error comes "You can only check the position for partitions assigned to this consumer."
But I am using a single consumer.
I am not able to understand the issue.
Please help me out if possible.
Properties properties = new Properties();
properties.setProperty(ConsumerConfig.CLIENT_ID_CONFIG,"4");
properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,"localhost:9092");
properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties);
String topic = "input_topic";
TopicPartition topicPartition = new TopicPartition(topic, 0);
TopicPartition topicPartition1 = new TopicPartition(topic, 1);
TopicPartition topicPartition2 = new TopicPartition(topic, 2);
List<TopicPartition> topics = Arrays.asList(topicPartition1,topicPartition,topicPartition2);
while (true) {
Thread.sleep(5000);
consumer.assign(topics);
consumer.seekToEnd(topics);
long current = consumer.position(topicPartition);
consumer.seek(topicPartition, current-3);
ConsumerRecords<String, String> records = consumer.poll(100);
System.out.println("-------------------------------------------> "+ records.count());
System.out.println("-------------------------------------------> "+ LocalDateTime.now());
for (ConsumerRecord<String, String> record : records) {
System.out.printf("offset = %d, key = %s, value = %s", record.offset(), record.key(), record.value());
System.out.println("_________________________" + record.partition());
}
}

My guess ... other than what Hatice said about assign which should be made outside of the loop just one time, I see this from your code.
You seek the position at the end of all topic partitions but then you seek on the offset for the last 3 records just for the topic partition 0.
At that point, the poll is able to consume just only those 3 records from topic partition 0 and not from other partitions because your position on them is at end (of course, it's true if you are not sending more messages to those partitions as well).

When kafka consumer poll return null records?

As show as below, my code is a high level consumer fetch a topic with 32 partitions in kafka server, I'm confused that why sometimes I get a empty return from consumer.poll().
I have tried to increase poll timeout, and then when I increase timeout to 1000, then each poll has return data while I set timeout to 10 or 0, then I see a lot of empty return.
Can anyone tell me how to set a correct timeout ?
def main(args: Array[String]): Unit = {
val props = new Properties()
props.put("bootstrap.servers", "kafka-01:9098")
props.put("group.id", "kch1")
props.put("enable.auto.commit", "true")
props.put("auto.commit.interval.ms", "1000")
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
//props.put("max.poll.records", "1000")
val consumers = new Array[KafkaConsumer[String, String]](16)
for(i <- 0 to 15) {
consumers(i) = new KafkaConsumer[String, String](props)
consumers(i).subscribe(util.Arrays.asList("veh321"))
}
var cnt = 0
var cacheIterator: Iterator[ConsumerRecord[String, String]] = null
for(i <- 0 to 15) {
new Thread(new Runnable {
override def run(): Unit = {
var finish = false
while(!finish) {
val start = System.currentTimeMillis()
cacheIterator = consumers(i).poll(100).iterator()
val end = System.currentTimeMillis() - start
if (end > 10 ) {
println(s"${Thread.currentThread().getId} + Duration is ${end}， ${cacheIterator.hasNext} ${cacheIterator.size}")
}
}
}
}).start()
}

Java consumer employs Linux's epoll as the underlying implementation scheme by invoking java.nio.channels.Selector.select(timeout). It is very likely to return nothing if you only give it 100 ms to try how many SelectionKeys are ready during that short time interval.
Besides, during the same 100 ms, consumer will do some other jobs including polling coordinator status, so the real time interval for record polling is obviously less than 100ms which makes harder to retrieve some really useful things.

NoSuchElementException kafka

I am new to kafka.I read documentation to get started and now I am trying to do hands on using embedded kafka mode.I tried a sample program for the same.
public static void main(String args[]) throws InterruptedException, IOException {
// setup Zookeeper
EmbeddedZookeeper zkServer = new EmbeddedZookeeper();
String zkConnect = ZKHOST + ":" + zkServer.port();
ZkClient zkClient = new ZkClient(zkConnect, 30000, 30000, ZKStringSerializer$.MODULE$);
ZkUtils zkUtils = ZkUtils.apply(zkClient, false);
// setup Broker
Properties brokerProps = new Properties();
brokerProps.setProperty("zookeeper.connect", zkConnect);
brokerProps.setProperty("broker.id", "0");
brokerProps.setProperty("log.dirs", Files.createTempDirectory("kafka-").toAbsolutePath().toString());
brokerProps.setProperty("listeners", "PLAINTEXT://" + BROKERHOST +":" + BROKERPORT);
KafkaConfig config = new KafkaConfig(brokerProps);
Time mock = new MockTime();
KafkaServer kafkaServer = TestUtils.createServer(config, mock);
// create topic
AdminUtils.createTopic(zkUtils, TOPIC, 1, 1, new Properties(), RackAwareMode.Disabled$.MODULE$);
// setup producer
Properties producerProps = new Properties();
producerProps.setProperty("bootstrap.servers", BROKERHOST + ":" + BROKERPORT);
producerProps.setProperty("key.serializer","org.apache.kafka.common.serialization.IntegerSerializer");
producerProps.setProperty("value.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer");
KafkaProducer<Integer, byte[]> producer = new KafkaProducer<Integer, byte[]>(producerProps);
List<PartitionInfo> partitionInfo = producer.partitionsFor("test");
System.out.println(partitionInfo);
// setup consumer
Properties consumerProps = new Properties();
consumerProps.setProperty("bootstrap.servers", BROKERHOST + ":" + BROKERPORT);
consumerProps.setProperty("group.id", "group0");
consumerProps.setProperty("client.id", "consumer0");
consumerProps.setProperty("key.deserializer","org.apache.kafka.common.serialization.IntegerDeserializer");
consumerProps.setProperty("value.deserializer", "org.apache.kafka.common.serialization.ByteArrayDeserializer");
consumerProps.put("auto.offset.reset", "earliest"); // to make sure the consumer starts from the beginning of the topic
KafkaConsumer<Integer, byte[]> consumer = new KafkaConsumer<>(consumerProps);
consumer.subscribe(Arrays.asList(TOPIC));
// send message
ProducerRecord<Integer, byte[]> data = new ProducerRecord<>(TOPIC, 42, "test-message".getBytes(StandardCharsets.UTF_8));
producer.send(data);
producer.close();
// starting consumer
ConsumerRecords<Integer, byte[]> records = consumer.poll(1000);
Iterator<ConsumerRecord<Integer, byte[]>> recordIterator = records.iterator();
ConsumerRecord<Integer, byte[]> record = recordIterator.next();
System.out.printf("offset = %d, key = %s, value = %s", record.offset(), record.key(), record.value());
kafkaServer.shutdown();
zkClient.close();
zkServer.shutdown();
}
}
but Iam not able fetch data for the topics.Iam getting the following exception while executing the programm
java.util.NoSuchElementException
at org.apache.kafka.common.utils.AbstractIterator.next(AbstractIterator.java:52)
at com.nuwaza.evlauation.embedded.kafka.EmbeddedKafka.main(EmbeddedKafka.java:105)
Can anyone guide me?
UPDATED-
WARN [main] (Logging.scala#warn:83) - No meta.properties file under dir C:\Users\bhavanak\AppData\Local\Temp\kafka-1238324273778000675\meta.properties
WARN [main] (Logging.scala#warn:83) - No meta.properties file under dir C:\Users\bhavanak\AppData\Local\Temp\kafka-1238324273778000675\meta.properties
WARN [kafka-producer-network-thread | producer-1] (NetworkClient.java#handleResponse:600) - Error while fetching metadata with correlation id 0 : {test=LEADER_NOT_AVAILABLE}
WARN [kafka-producer-network-thread | producer-1] (NetworkClient.java#handleResponse:600) - Error while fetching metadata with correlation id 1 : {test=LEADER_NOT_AVAILABLE}
WARN [kafka-producer-network-thread | producer-1] (NetworkClient.java#handleResponse:600) - Error while fetching metadata with correlation id 2 : {test=LEADER_NOT_AVAILABLE}
[Partition(topic = test, partition = 0, leader = 0, replicas = [0,], isr = [0,]]
ERROR [main] (NIOServerCnxnFactory.java#uncaughtException:44) - Thread Thread[main,5,main] died
java.util.NoSuchElementException
at org.apache.kafka.common.utils.AbstractIterator.next(AbstractIterator.java:52)
at com.nuwaza.evlauation.embedded.kafka.EmbeddedKafka.main(EmbeddedKafka.java:105)

Try to invoke producer.flush() before reading messages to ensure the produced messages indeed are persisted on disks.

This error signifies that your consumer is trying to read the message even before it gets persisted to kafka logs. Ideally you should run producer and consumer as separate process. I was facing the same issue but that was due to other reason being iterator.next() was invoked twice mistakenly. Just in case someone else facing same issue.