We Keep Coding

Not able to publish messages to Kafka from Lambda function

I am trying to publish some messages to kafka from a aws lambda function. When I'm trying to test this feature on my local, the messages are not getting published and the function is getting timed out. I was able to connect to the local kafka instance from the same lambda function using a consumer and list the topics. Is there anything I'm missing here?
String bootstrapServer = "host.docker.internal:9092";
String topic = "test-topic";
Properties properties = new Properties();
properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServer);
properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.setProperty(ProducerConfig.ACKS_CONFIG, "all");
properties.put(ProducerConfig.BATCH_SIZE_CONFIG, 16384);
Properties props = new Properties();
props.put("bootstrap.servers", bootstrapServer);
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> simpleConsumer = new KafkaConsumer<>(props);
// THIS IS WORKING
simpleConsumer.listTopics().forEach((t, v) -> logger.log(t + "\n"));
try (KafkaProducer<String, String> kafkaProducer = new KafkaProducer<>(properties)) {
ProducerRecord<String, String> record = new ProducerRecord<>(topic, value);
logger.log("Publishing '" + value + "' to kafka topic: " + topic + "\n");
Future<RecordMetadata> future = kafkaProducer.send(record);
logger.log("Waiting for flushing data\n");
kafkaProducer.flush();
logger.log("Waiting for the response\n");
RecordMetadata recordMetadata = future.get();
logger.log("Published record to kafka topic: " + recordMetadata.topic() + " partition: " + recordMetadata.partition() + " offset: " + recordMetadata.offset() + "\n");
I'm triggering the function in my local using sam cli as follows:
sam local invoke "MyLambdaFunction" -e events/event.json
This is the timeout message I get.
Invoking helloworld.App::handleRequest (java11)
Skip pulling image and use local one: public.ecr.aws/sam/emulation-java11:rapid-1.59.0-x86_64.
Mounting .aws-sam/build/MyLambdaFunction as /var/task:ro,delegated inside runtime container
START RequestId: c07609a4-7cab-4b58-97fc-cdddf20afd5c Version: $LATEST
Picked up JAVA_TOOL_OPTIONS: -XX:+TieredCompilation -XX:TieredStopAtLevel=1
test-topic
__consumer_offsets
Publishing 'abc' to kafka topic: test-topic
Function 'MyLambdaFunction' timed out after 20 seconds
No response from invoke container for MyLambdaFunction

Confluent Control Center - Consumer is not listed

I have the following code to connect to Kafka
Properties props = new Properties();
props.put("bootstrap.servers", "myconfluentkafkabroker:9092");
props.put("group.id","test");
props.put("enable.auto.commit","true");
props.put("auto.commit.interval.ms","1000");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "my_CG");
props.put("group.instance.id", "my_instance_CG_id");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put("key.deserializer", Class.forName("org.apache.kafka.common.serialization.StringDeserializer"));
props.put("value.deserializer", Class.forName("org.apache.kafka.common.serialization.StringDeserializer"));
KafkaConsumer<String,String> consumer = new KafkaConsumer<String,String>(props);
consumer.subscribe(Arrays.asList("MyTopic"));
try {
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records)
{
log.debug("topic = %s, partition = %d, offset = %d,"
customer = %s, country = %s\n",
record.topic(), record.partition(), record.offset(),
record.key(), record.value());
int updatedCount = 1;
if (custCountryMap.countainsKey(record.value())) {
updatedCount = custCountryMap.get(record.value()) + 1;
}
custCountryMap.put(record.value(), updatedCount)
JSONObject json = new JSONObject(custCountryMap);
System.out.println(json.toString(4));
}
}
} finally {
consumer.close();
}
Code didn't throw any errors but I still don't see the consumer listed
would this be an issue?
props.put("group.instance.id", "my_instance_CG_id");

You should verify the information that you see with the built-in tools that Kafka provides like kafka-consumer-groups.sh
You'll also need to actually poll messages and commit offsets, not just subscribe before you will see anything.
Otherwise, for that specific Control Center dashboard, it may require you to add the Monitoring Interceptors into your client

Kafka StickyAssignor breaking delivery to single consumer in the group

I have a topic with a single partition and two consumer processes that form a consumer group.
This way a message is always delivered to a single consumer. The StickyAssignor is used to prefer consumers that are already assigned to partition on rebalance.
I've been playing with this setup a bit and discovered that under certain circumstances, the messages are delivered to both consumers, which breaks the purpose of consumer group.
The scenario is following:
Start consumer C1
C1 begins receiving messages
Start consumer C2
C2 doesn't receive any messages - thanks to StickyAssignor strategy that prefers C1
Freeze C1 process - (using Java debugger - stopping all threads)
C2 takes over - begins receiving messages
Unfreeze C1 process
Now both C1 and C2 receive messages despite being in the same group
When using RangeAssignor/RoundRobinAssignor, this does not happen.
Am I missing something or is this a bug in Kafka?
Here's my consumer code:
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test");
props.put("client.id", consumerId);
props.put("enable.auto.commit", "false");
props.put("auto.commit.interval.ms", "1000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("partition.assignment.strategy", StickyAssignor.class.getName());
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singleton("my-events"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records)
System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
}

Inconsistent data output from Kafka consumer

I need to pull data from Kafka consumer to pass it on to my application. Below is the code that I have written to access the consumer:
public class ConsumerGroup {
public static void main(String[] args) throws Exception {
String topic = "kafka_topic";
String group = "0";
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", group);
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "1000");
props.put("session.timeout.ms", "30000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("auto.offset.reset", "earliest");
KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);
consumer.subscribe(Arrays.asList(topic));
System.out.println("Subscribed to topic: " + topic);
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records)
System.out.printf("offset = %d, key = %s, value = %s\n", record.offset(), record.key(), record.value());
}
}
}
When I run this code, sometimes the data is getting generated and sometimes no data is generated. Why this behavior is inconsistent? Is there any issue with my code?

Your code is Ok. You have autocommit option enabled, so after you read the records, they are automatically committed to Kafka. Every time when you run the code you start from the last processed offset, which is stored in __consumer_offsets topic. So you always read only the new records, which have arrived to Kafka after last run. To print the data constantly in the consumer app, you should put constantly new records into your topic.

Kafka 0.10: what's wrong with my consumer?

i'm trying to do some easy demo in kafka-0.10.0.0.
my producer is ok , but consumer maybe not correct, code as below.
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "group1");
props.put("enable.auto.commit", "false");
props.put("session.timeout.ms", "30000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("topictest2"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (TopicPartition partition : records.partitions()) {
List<ConsumerRecord<String, String>> partitionRecords = records.records(partition);
for (ConsumerRecord<String, String> record : partitionRecords) {
System.out.println("Thread = "+Thread.currentThread().getName()+" ");
System.out.printf("partition = %d, offset = %d, key = %s, value = %s",record.partition(), record.offset(), record.key(), record.value());
System.out.println("\n");
}
// consumer.commitSync();
long lastOffset = partitionRecords.get(partitionRecords.size() - 1).offset();
consumer.commitSync(Collections.singletonMap(partition, new OffsetAndMetadata(lastOffset + 1)));
}
}
but when i run this demo, No output! what is the problem in my code?

It looks valid.
I think the program is just waiting for new messages because
auto.offset.reset default is latest
If you have some messages in that topic and want to read them, try to add
props.put("auto.offset.reset", "earliest");
to start reading topic from beginning and reset your group.id to something unique to make sure it won't continue from saved offset or do not commit offset at all. Once its there for a group id the auto.offset.reset is skipped.
props.put("group.id", "group."+UUID.randomUUID().toString());