I am very new to Kafka.
I am trying to push data to kafka topic which is running in 172.11.22.12:9092 server. If I will run producer program in the same server (172.11.22.12) I'm able to push to the kafka topic and I can see in the consumer console also. If I am running in windows machine it is not pushing to the topic. Producer program is as follows:
public class SimpleKafkaProducer {
public static void main(String args[]) {
Properties props = new Properties();
props.put("bootstrap.servers", "172.11.22.12:9092");
props.put("acks", "all");
props.put("retries", 0);
props.put("batch.size", 16384);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
org.apache.kafka.clients.producer.Producer<String, String> producer = new org.apache.kafka.clients.producer.KafkaProducer<>(props);
for(int i = 1; i < 100; i++)
producer.send(new ProducerRecord<String, String>("test", Integer.toString(i), Integer.toString(i)));
producer.close();
}
}
Log is as follows
14:46:47.662 [main] DEBUG o.a.k.c.producer.internals.Metadata - Updated cluster metadata version 1 to Cluster(nodes = [Node(172.11.22.12, 9092)], partitions = [])
14:46:47.771 [kafka-producer-network-thread | producer-1] DEBUG o.a.k.c.producer.internals.Sender - Starting Kafka producer I/O thread.
14:46:47.771 [main] DEBUG o.a.k.clients.producer.KafkaProducer - Kafka producer started
14:46:47.773 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:47.773 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Init connection to node -1 for sending metadata request in the next iteration
14:46:47.773 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Initiating connection to node -1 at 172.11.22.12:9092.
14:46:47.776 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:47.876 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:47.977 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:48.077 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:48.177 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:48.277 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:48.378 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:48.478 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:48.578 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:48.678 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:48.778 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:48.780 [kafka-producer-network-thread | producer-1] WARN o.a.kafka.common.network.Selector - Error in I/O with /172.11.22.12
14:46:48.781 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Node -1 disconnected.
14:46:48.879 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:48.879 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Init connection to node -1 for sending metadata request in the next iteration
14:46:48.879 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Initiating connection to node -1 at 172.11.22.12:9092.
14:46:48.879 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:48.979 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:49.079 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:49.179 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:49.280 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:49.380 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:49.480 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:49.580 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:49.680 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:49.781 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:49.881 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node -1
14:46:49.884 [kafka-producer-network-thread | producer-1] WARN o.a.kafka.common.network.Selector - Error in I/O with /172.11.22.12
Help will be appreciated.
You need to flush the producer before closing it as follows:
public static void main(String args[]) {
Properties props = new Properties();
props.put("bootstrap.servers", "172.11.22.12:9092");
props.put("acks", "all");
props.put("retries", 0);
props.put("batch.size", 16384);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
org.apache.kafka.clients.producer.Producer<String, String> producer = new org.apache.kafka.clients.producer.KafkaProducer<>(props);
for(int i = 1; i < 100; i++)
producer.send(new ProducerRecord<String, String>("test", Integer.toString(i), Integer.toString(i)));
producer.flush();
producer.close();
}
Related
I've got a #KafkaListener method in my service which processes message and send it to another topic using KafkaTemplate and from time to time it completely stops working due to some reasons.
2022-10-04 16:53:18.218 ERROR 1 --- [pool-1-thread-2] o.s.k.support.LoggingProducerListener : Exception thrown when sending a message with key='null' and payload='{"type":"INFORMATION","messageId":"f39fabfd-e560-499b-9850-440ad811657b","phoneNumber":"+100000000...' to topic ss.fb.processing-notifications.send:
org.apache.kafka.common.errors.TimeoutException: Topic ss.fb.processing-notifications.send not present in metadata after 60000 ms.
2022-10-04 16:53:33.013 INFO 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Disconnecting from node -2 due to socket connection setup timeout. The timeout value is 29794 ms.
2022-10-04 16:53:33.014 WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Bootstrap broker prd-mqueue-srv2.obi.ru:9092 (id: -2 rack: null) disconnected
2022-10-04 16:53:41.005 INFO 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-nepcNotificationsGroup-1, groupId=nepcNotificationsGroup] Disconnecting from node -3 due to socket connection setup timeout. The timeout value is 27831 ms.
2022-10-04 16:53:41.005 WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-nepcNotificationsGroup-1, groupId=nepcNotificationsGroup] Bootstrap broker prd-mqueue-srv3.obi.ru:9092 (id: -3 rack: null) disconnected
There seems to be some network issues however after restarting the service everything works fine again. Anyway I wonder why eventually broker turns out to be disconnected? Is't producer supposed to infinitely try sending message to broker until it succedes?
You can wrap the KafkaTemplate call in a RetryTemplate or #Retryable method - see https://github.com/spring-projects/spring-retry - the RetryTemplate is already on the class path as a transitive dependency of spring-kafka.
This question already has answers here:
Connect to Kafka running in Docker
(5 answers)
Closed 9 months ago.
There is no problem everything is ok on the terminals; i can send message via producer and receive it on consumer. But am not able to get same result via KafkaProducer java.
kafka:
image: 'bitnami/kafka:latest'
ports:
- "9092:9092"
- '9093:9093'
environment:
- KAFKA_BROKER_ID=1
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
- KAFKA_CFG_LISTENERS=CLIENT://kafka:9092,EXTERNAL://localhost:9093
- KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092,EXTERNAL://localhost:9093
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT
depends_on:
- zookeeper
zookeeper:
image: 'bitnami/zookeeper:latest'
ports:
- "2181:2181"
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
networks:
microservicesnet:
driver: bridge
Java code for KafkaProducer
public Producer<String, String> setUpKafkaPropoerties() {
Properties properties = new Properties();
//Update the IP adress of Kafka server here//
properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9093");
properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
StringSerializer.class);
properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
StringSerializer.class);
properties.put("acks", "all");
properties.put("retries", 0);
properties.put("linger.ms", 0);
properties.put("partitioner.class",
"org.apache.kafka.clients.producer.internals.DefaultPartitioner");
properties.put("request.timeout.ms", 30000);
properties.put("timeout.ms", 30000);
properties.put("max.in.flight.requests.per.connection", 5);
properties.put("retry.backoff.ms", 5);
//Instantiate Producer Object
Producer<String, String> producer = new KafkaProducer<String, String>(properties);
return producer;
ERROR LOG:
2022-05-21 14:55:21.240 WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:9093) could not be established. Broker may not be available.
2022-05-21 14:55:21.240 WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Bootstrap broker localhost:9093 (id: -1 rack: null) disconnected
2022-05-21 14:55:22.254 WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:9093) could not be established. Broker may not be available.
Can you try removing those quotes and double quotes in the port forwarding part
In my docker-compose file I don't use any quotes for port forwarding
Can anyone let me know what causes the error
WARN org.apache.kafka.clients.producer.internals.Sender - [Producer clientId=KafkaExampleProducer1] Received invalid metadata error in produce request on partition my-example-topic1-1 due to org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to access log file on the disk.. Going to request metadata update now"
I am running a 3 broker Kafka setup on the windows environment, I have created 2 TOPICs with 5 partitions each, I am using 2 producer instances. One for each TOPIC and I am using a single Consumer instance to consume from both of these TOPICs. The setup worked fine for some time. Then one of the Producer and the Consumer stopped functioning, the following messages were printed on the Producer and Consumer Console,
my-example-topic1 & my-example-topic2 are the 2 TOPICs.
Producer Console:
74394 [kafka-producer-network-thread | KafkaExampleProducer1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=KafkaExampleProducer1] 1 partitions have leader brokers without a matching listener, including [my-example-topic1-1]
74494 [kafka-producer-network-thread | KafkaExampleProducer1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=KafkaExampleProducer1] 1 partitions have leader brokers without a matching listener, including [my-example-topic1-1]
74595 [kafka-producer-network-thread | KafkaExampleProducer1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=KafkaExampleProducer1] 1 partitions have leader brokers without a matching listener, including [my-example-topic1-1]
74697 [kafka-producer-network-thread | KafkaExampleProducer1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=KafkaExampleProducer1] 1 partitions have leader brokers without a matching listener, including [my-example-topic1-1]
74798 [kafka-producer-network-thread | KafkaExampleProducer1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=KafkaExampleProducer1] 1 partitions have leader brokers without a matching listener, including [my-example-topic1-1]
74900 [kafka-producer-network-thread | KafkaExampleProducer1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=KafkaExampleProducer1] 1 partitions have leader brokers without a matching listener, including [my-example-topic1-1]
75001 [kafka-producer-network-thread | KafkaExampleProducer1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=KafkaExampleProducer1] 1 partitions have leader brokers without a matching listener, including [my-example-topic1-1]
Consumer Console:
17533 [main] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-KafkaExampleConsumer1-1, groupId=KafkaExampleConsumer1] 1 partitions have leader brokers without a matching listener, including [my-example-topic1-1]
17636 [main] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-KafkaExampleConsumer1-1, groupId=KafkaExampleConsumer1] 1 partitions have leader brokers without a matching listener, including [my-example-topic1-1]
17738 [main] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-KafkaExampleConsumer1-1, groupId=KafkaExampleConsumer1] 1 partitions have leader brokers without a matching listener, including [my-example-topic1-1]
17840 [main] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-KafkaExampleConsumer1-1, groupId=KafkaExampleConsumer1] 1 partitions have leader brokers without a matching listener, including [my-example-topic1-1]
17943 [main] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-KafkaExampleConsumer1-1, groupId=KafkaExampleConsumer1] 1 partitions have leader brokers without a matching listener, including [my-example-topic1-1]
18044 [main] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-KafkaExampleConsumer1-1, groupId=KafkaExampleConsumer1] 1 partitions have leader brokers without a matching listener, including [my-example-topic1-1]
18147 [main] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-KafkaExampleConsumer1-1, groupId=KafkaExampleConsumer1] 1 partitions have leader brokers without a matching listener, including [my-example-topic1-1]
18248 [main] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-KafkaExampleConsumer1-1, groupId=KafkaExampleConsumer1] 1 partitions have leader brokers without a matching listener, including [my-example-topic1-1]
18350 [main] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-KafkaExampleConsumer1-1, groupId=KafkaExampleConsumer1] 1 partitions have leader brokers without a matching listener, including [my-example-topic1-1]
I had a simple demo to transfer data from one kafka cluster which did not use SASL to another kafka cluster which did use SASL/PLAIN. and the codes look like this:
Properties consumerProps = new Properties();
consumerProps.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.50.20:9092");
consumerProps.put(ConsumerConfig.GROUP_ID_CONFIG, GROUP);
consumerProps.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
consumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
consumerProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
Properties producerProps = new Properties();
producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.1.100:9092");
producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
producerProps.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_PLAINTEXT");
producerProps.put(SaslConfigs.SASL_MECHANISM, "PLAIN");
producerProps.put(SaslConfigs.SASL_JAAS_CONFIG, String.format(
PlainLoginModule.class.getName() + " required username=\"%s\" " + "password=\"%s\";",
"admin",
"admin"
));
//and some other producer properties
KafkaProducer producer = new KafkaProducer<>(props);
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records) {
ProducerRecord<String, String> record = new ProducerRecord<>("test", record.key()
,record.value());
producer.send(record);
}
}
It was just simply consuming the data and producing those data to another kafka cluster,but the thing is:
when I wrote another consumer client to consume those data from 192.168.1.100:9092
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.1.100:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, GROUP);
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_PLAINTEXT");
props.put(SaslConfigs.SASL_MECHANISM, "PLAIN");
String password = EncryptUtil.encryptPassword(USER_NAME, QUERY_KEY, QUERY_SECRET);
System.out.println(password);
props.put(SaslConfigs.SASL_JAAS_CONFIG, String.format(
PlainLoginModule.class.getName() + " required username=\"%s\" " + "password=\"%s\";",
"admin",
"admin"
));
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records) {
System.out.println(record.value());
}
}
and it had nothing to print but
9419 [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=snKiBQ0O, groupId=group-snKiBQ0O] Discovered group coordinator 192.168.1.100:9092 (id: 2147483647 rack: null)
18456 [main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=snKiBQ0O, groupId=group-snKiBQ0O] Revoking previously assigned partitions []
18456 [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=snKiBQ0O, groupId=group-snKiBQ0O] (Re-)joining group
18471 [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=snKiBQ0O, groupId=group-snKiBQ0O] Successfully joined group with generation 21
18471 [main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=snKiBQ0O, groupId=group-snKiBQ0O] Setting newly assigned partitions [flink-kafka-0]
27522 [main] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=snKiBQ0O, groupId=group-snKiBQ0O] Resetting offset for partition flink-kafka-0 to offset 0.
27537 [main] INFO org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=snKiBQ0O, groupId=group-snKiBQ0O] Node 0 sent an invalid full fetch response with extra=(flink-kafka-0, response=(
57556 [main] INFO org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=snKiBQ0O, groupId=group-snKiBQ0O] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 0: org.apache.kafka.common.errors.TimeoutException: Failed to send request after 30000 ms..
87563 [main] INFO org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=snKiBQ0O, groupId=group-snKiBQ0O] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 0: org.apache.kafka.common.errors.TimeoutException: Failed to send request after 30000 ms..
117585 [main] INFO org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=snKiBQ0O, groupId=group-snKiBQ0O] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 0: org.apache.kafka.common.errors.TimeoutException: Failed to send request after 30000 ms..
146158 [main] INFO org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=snKiBQ0O, groupId=group-snKiBQ0O] Node 0 sent an invalid full fetch response with extra=(flink-kafka-0, response=(
176263 [main] INFO org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=snKiBQ0O, groupId=group-snKiBQ0O] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 0: org.apache.kafka.common.errors.TimeoutException: Failed to send request after 30000 ms..
206333 [main] INFO org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=snKiBQ0O, groupId=group-snKiBQ0O] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 0: org.apache.kafka.common.errors.TimeoutException: Failed to send request after 30000 ms..
236418 [main] INFO org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=snKiBQ0O, groupId=group-snKiBQ0O] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 0: org.apache.kafka.common.errors.TimeoutException: Failed to send request after 30000 ms..
266492 [kafka-coordinator-heartbeat-thread | group-snKiBQ0O] INFO org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=snKiBQ0O, groupId=group-snKiBQ0O] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 0: org.apache.kafka.common.errors.TimeoutException: Failed to send request after 30000 ms..
296558 [main] INFO org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=snKiBQ0O, groupId=group-snKiBQ0O] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 0: org.apache.kafka.common.errors.TimeoutException: Failed to send request after 30000 ms..
317372 [kafka-coordinator-heartbeat-thread | group-snKiBQ0O] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=snKiBQ0O, groupId=group-snKiBQ0O] Group coordinator 192.168.1.100:9092 (id: 2147483647 rack: null) is unavailable or invalid, will attempt rediscovery
326571 [main] INFO org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=snKiBQ0O, groupId=group-snKiBQ0O] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 0: org.apache.kafka.common.errors.TimeoutException: Failed to send request after 30000 ms..
and then I was using this shell ./bin/kafka-consumer-groups.sh --describe --bootstrap-server 192.168.1.100:9092 --command-config config/client_plain.properties --group group-snKiBQ0O to check wether the data was sent correctly and the LOG-END-OFFSET has a value of 5912 but the CURRENT-OFFSET was always 0.
At last, I changed producer.send(record); to producer.send(record).get(); and the consumer client was successfully received the data. why is that? Why broker using SASL/PLAIN can not asynchronous sending data?Is there a good way to deal with it?
Thanks.
UPDATE: I deleted all kafka log and zookeeper data,and it can work fine.But still don't understand why this happend
I trying to send some data to kafka, but when i run my code i got
13:20:17.688 [kafka-producer-network-thread | producer-1]
DEBUG org.apache.kafka.clients.NetworkClient - Node -1 disconnected.
13:20:17.784 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - Initialize connection to node -1 for sending metadata request
13:20:17.784 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - Initiating connection to node -1 at kafkaAdress:2181.
13:20:18.781 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - Connection with kafkaAdress/addressId disconnected
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:54)
at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:72)
at org.apache.kafka.common.network.Selector.poll(Selector.java:274)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:256)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128)
at java.lang.Thread.run(Unknown Source)
Code:
String topic = "TST";
Properties props = new Properties();
props.put("bootstrap.servers", "kafkaAdress:2181");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
for(int i = 0; i < 100; i++)
producer.send(new ProducerRecord<String, String>(topic, "TestMessage"));
producer.close();
Does anyone know how to solve this?
I use kafka 0.9.1
Since Kafka 0.9 the producer API no longer uses Zookeeper.
The property bootstrap.servers should contain a list of brokers for establishing the initial connection to the Kafka cluster.
2181 is the zookeeper port. The default port for a broker is 9092.
You have some mistake on your configuration, bootstrap.server is the Kafka broker's address, not zookeeper. Producer always use the broker's address to publish messages.