I need to execute a Job at night that is going to get all the messages in a kafka queue and execute a process with them. I'm able to get the messages but the kafka stream is waiting for more messages and I'm not able to continue with my process. I have the following code:
...
private ConsumerConnector consumerConnector;
private final static String TOPIC = "test";
public MessageStreamConsumer() {
Properties properties = new Properties();
properties.put("zookeeper.connect", "localhost:2181");
properties.put("group.id", "test-group");
ConsumerConfig consumerConfig = new ConsumerConfig(properties);
consumerConnector = Consumer.createJavaConsumerConnector(consumerConfig);
}
public List<String> getMessages() {
Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
topicCountMap.put(TOPIC, new Integer(1));
Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumerConnector
.createMessageStreams(topicCountMap);
KafkaStream<byte[], byte[]> stream = consumerMap.get(TOPIC).get(0);
ConsumerIterator<byte[], byte[]> it = stream.iterator();
List<String> messages = new ArrayList<>();
while (it.hasNext())
messages.add(new String(it.next().message()));
return messages;
}
The code is able to get the messages but when it process the last message it stays in the line:
while (it.hasNext())
The question is, how can i get all the messages from the kafka, stop the stream and continue with my other tasks.
I hope you can help me
Thanks
It seems that kafka stream does not support to consume from beginning.
You could create a native kafka consumer and set auto.offset.reset to earliest, then it will consume message from beginning.
Something like this may work. Basically the idea is to use a Kafka Consumer and poll until you get some record and then stop when you get an empty batch.
package kafka.examples;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Collections;
import java.util.Date;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicBoolean;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
public class Consumer1 extends Thread
{
private final KafkaConsumer<Integer, String> consumer;
private final String topic;
private final DateFormat df;
private final String logTag;
private boolean noMoreData = false;
private boolean gotData = false;
private int messagesReceived = 0;
AtomicBoolean isRunning = new AtomicBoolean(true);
CountDownLatch shutdownLatch = new CountDownLatch(1);
public Consumer1(Properties props)
{
logTag = "Consumer1";
consumer = new KafkaConsumer<>(props);
this.topic = props.getProperty("topic");
this.df = new SimpleDateFormat("HH:mm:ss");
consumer.subscribe(Collections.singletonList(this.topic));
}
public void getMessages() {
System.out.println("Getting messages...");
while (noMoreData == false) {
//System.out.println(logTag + ": Doing work...");
ConsumerRecords<Integer, String> records = consumer.poll(1000);
Date now = Calendar.getInstance().getTime();
int recordsCount = records.count();
messagesReceived += recordsCount;
System.out.println("recordsCount: " + recordsCount);
if (recordsCount > 0) {
gotData = true;
}
if (gotData && recordsCount == 0) {
noMoreData = true;
}
for (ConsumerRecord<Integer, String> record : records) {
int kafkaKey = record.key();
String kafkaValue = record.value();
System.out.println(this.df.format(now) + " " + logTag + ":" +
" Received: {" + kafkaKey + ":" + kafkaValue + "}" +
", partition(" + record.partition() + ")" +
", offset(" + record.offset() + ")");
}
}
System.out.println("Received " + messagesReceived + " messages");
}
public void processMessages() {
System.out.println("Processing messages...");
}
public void run() {
getMessages();
processMessages();
}
}
I'm currently developing with Kafka 0.10.0.1 and found mixed information regarding the use of consumer property auto.offset.reset so I've done some experiments to figure out what actually happens.
Based on those, I now understand it this way: when you set property:
auto.offset.reset=earliest
this positions the consumer to EITHER the first available message in the partitions assigned (when no commits have been made on the paritions) OR it positions the consumer at the last committed partition offsets (notice that you always commit last read offset + 1 or else you'll be re-reading the last committed message on each restart of your consumer)
Alternatively you do not set auto.offset.reset which means default value of 'latest' will be used.
In that case you do not receive any old messages on connecting the consumer - only messages published to the topic after connecting the consumer will be received.
As a conclusion - if you want to ensure to receive all available messages for a certain topic and assigned partitions you'll have to call seekToBeginning().
It seems advised to call poll(0L) first to ensure your consumer gets partitions assigned (or implement your code in the ConsumerRebalanceListener!), then seek each of the assigned partitions to 'beginning':
kafkaConsumer.poll(0L);
kafkaConsumer.seekToBeginning(kafkaConsumer.assignment());
Related
A Tcp.OuttgoingConnection gathers data from an audio mixer and is send async to a sourceQueue, which processes the data.
After issuing a command there is no guarantee the next bit of data is the response. How can I feed back the response?
A 'dirty' way would be to have a static variable in which I put the data when processed with a Thread pause to wait for it but that is very inefficient. Is there an akka mechanism that can watch for a value to change and give a Future?
This is the current code:
public Q16SocketThread(ActorSystem system) {
Logger.debug("Loading Q16SocketThread.");
this.system = system;
final Flow<ByteString, ByteString, CompletionStage<Tcp.OutgoingConnection>> connection =
Tcp.get(system).outgoingConnection(ipAddress, port);
int bufferSize = 10;
final SourceQueueWithComplete<ByteBuffer> sourceQueue =
Source.<ByteBuffer>queue(bufferSize, OverflowStrategy.fail())
.map(input -> Hex.encodeHexString(input.array()))
.to(Sink.foreach(this::startProcessing))
.run(system);
final Flow<ByteString, ByteString, NotUsed> repl =
Flow.of(ByteString.class)
.map(ByteString::toByteBuffer)
.map(sourceQueue::offer)
.map(
text -> {
//Logger.debug("Server: " + Hex.encodeHexString(text.array()));
String hexCmd;
if (!nextCmd.isEmpty()) {
hexCmd = nextCmd.take();
} else {
hexCmd = "fe";
}
return ByteString.fromArray(Hex.decodeHex(hexCmd));
}).async();
CompletionStage<Tcp.OutgoingConnection> connectionCS = connection.join(repl).run(system);
}
#Override
public Receive createReceive() {
return receiveBuilder()
.match(String.class, message -> {
if (message.equalsIgnoreCase("start")) {
Logger.debug("Q16 thread started.");
nextCmd.put(sysExHeaderAllCall + "1201F7");
} else if (message.equalsIgnoreCase("stop")) {
Logger.debug("Stopping of data gathering");
nextCmd.put(sysExHeaderAllCall + "1200F7");
//self().tell(PoisonPill.getInstance(), ActorRef.noSender());
} else if (message.equalsIgnoreCase("version")){
Logger.debug("Requesting version.");
nextCmd.put(sysExHeaderAllCall + "1001F7");
}
}).build();
}
I understand by watching a variable as using the ask pattern and receive a message. In your case you want the message wraped in a Future. Is it what you mean?
If so this from the Akka docs (https://doc.akka.io/docs/akka/2.5/futures.html#use-with-actors) may help:
There are generally two ways of getting a reply from an AbstractActor: the first is by a sent message (actorRef.tell(msg, sender)), which only works if the original sender was an AbstractActor) and the second is through a Future.
Using the ActorRef’s ask method to send a message will return a Future. To wait for and retrieve the actual result the simplest method is:
import akka.dispatch.*;
import jdocs.AbstractJavaTest;
import scala.concurrent.ExecutionContext;
import scala.concurrent.Future;
import scala.concurrent.Await;
import scala.concurrent.Promise;
import akka.util.Timeout;
Timeout timeout = Timeout.create(Duration.ofSeconds(5));
Future<Object> future = Patterns.ask(actor, msg, timeout);
String result = (String) Await.result(future, timeout.duration());
Kafka consumer not receiving messages produced before the consumer gets started.
public class MyKafkaConsumer {
private final KafkaConsumer<String, String> consumer;
private final String TOPIC="javaapp";
private final String BOOTSTRAP_SERVERS="localhost:9092";
private int receivedCounter=0;
private ExecutorService executorService=Executors.newFixedThreadPool(1);
private BlockingQueue<ConsumerRecords<String, String>> queue=new LinkedBlockingQueue<>(500000);
private MyKafkaConsumer() {
final Properties props=new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, BOOTSTRAP_SERVERS);
props.put(ConsumerConfig.GROUP_ID_CONFIG, "KafkaGroup6");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
consumer=new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList(TOPIC));
}
public static void main(String[] args) throws InterruptedException {
MyKafkaConsumer perfKafkaConsumer=new MyKafkaConsumer();
perfKafkaConsumer.consumeMessage();
perfKafkaConsumer.runConsumer();
}
private void runConsumer() throws InterruptedException {
consumer.poll(Duration.ofMillis(1000));
while (true) {
final ConsumerRecords<String, String> consumerRecords=consumer.poll(Duration.ofMillis(10000));
if (!consumerRecords.isEmpty()) {
System.out.println("Adding result in queue " + queue.size());
queue.put(consumerRecords);
}
consumer.commitAsync();
}
}
private void consumeMessage() {
System.out.println("Consumer starts at " + Instant.now());
executorService.submit(() -> {
while (true) {
ConsumerRecords<String, String> poll=queue.take();
poll.forEach(record -> {
System.out.println("Received " + ++receivedCounter + " time " + Instant.now(Clock.systemUTC()));
});
}
});
}
}
ConsumerRecords are always empty
I checked the offset using Kafka tool
I have also tried with a different group name, it's not working. Same issue i.e. poll returns empty records
Although, if I start my consumer before than producer than its receiving the messages. (Kafka-client ver 2.4.1)
The auto.offset.reset consumer setting controls where a new consumer group will begin consuming from a topic. By default it is set to 'latest' which will set the consumer groups offset to the latest offset. You want to set this to 'earliest' if all consumer groups should start at the earliest offset in the topic.
I use following code to create one producer which produces around 2000 messages.
public class ProducerDemoWithCallback {
public static void main(String[] args) {
final Logger logger = LoggerFactory.getLogger(ProducerDemoWithCallback.class);
String bootstrapServers = "localhost:9092";
Properties properties = new Properties();
properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
// create the producer
KafkaProducer<String, String> producer = new KafkaProducer<String, String>(properties);
for (int i=0; i<2000; i++ ) {
// create a producer record
ProducerRecord<String, String> record =
new ProducerRecord<String, String>("TwitterProducer", "Hello World " + Integer.toString(i));
// send data - asynchronous
producer.send(record, new Callback() {
public void onCompletion(RecordMetadata recordMetadata, Exception e) {
// executes every time a record is successfully sent or an exception is thrown
if (e == null) {
// the record was successfully sent
logger .info("Received new metadata. \n" +
"Topic:" + recordMetadata.topic() + "\n" +
"Partition: " + recordMetadata.partition() + "\n" +
"Offset: " + recordMetadata.offset() + "\n" +
"Timestamp: " + recordMetadata.timestamp());
} else {
logger .error("Error while producing", e);
}
}
});
}
// flush data
producer.flush();
// flush and close producer
producer.close();
}
}
I want to count those messages and get int value.
I use this command and it works, but i am trying to get this count using code.
"bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic TwitterProducer --time -1"
and the result is
- TwitterProducer:0:2000
My code to do the same programmatically looks something like this, but I'm not sure if this is the correct way to get the count:
int valueCount = (int) recordMetadata.offset();
System.out.println("Offset value " + valueCount);
Can someone help me to get count of Kafka messages offset value using code.
You can have a look at implementation details of GetOffsetShell.
Here is a simplified code re-written in Java:
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.*;
import java.util.stream.Collectors;
public class GetOffsetCommand {
private static final Set<String> TopicNames = new HashSet<>();
static {
TopicNames.add("my-topic");
TopicNames.add("not-my-topic");
}
public static void main(String[] args) {
TopicNames.forEach(topicName -> {
final Map<TopicPartition, Long> offsets = getOffsets(topicName);
new ArrayList<>(offsets.entrySet()).forEach(System.out::println);
System.out.println(topicName + ":" + offsets.values().stream().reduce(0L, Long::sum));
});
}
private static Map<TopicPartition, Long> getOffsets(String topicName) {
final KafkaConsumer<String, String> consumer = makeKafkaConsumer();
final List<TopicPartition> partitions = listTopicPartitions(consumer, topicName);
return consumer.endOffsets(partitions);
}
private static KafkaConsumer<String, String> makeKafkaConsumer() {
final Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.GROUP_ID_CONFIG, "get-offset-command");
return new KafkaConsumer<>(props);
}
private static List<TopicPartition> listTopicPartitions(KafkaConsumer<String, String> consumer, String topicName) {
return consumer.listTopics().entrySet().stream()
.filter(t -> topicName.equals(t.getKey()))
.flatMap(t -> t.getValue().stream())
.map(p -> new TopicPartition(p.topic(), p.partition()))
.collect(Collectors.toList());
}
}
which produces the offset for each topic's partition and sum (total number of messages), like:
my-topic-0=184
my-topic-2=187
my-topic-4=189
my-topic-1=196
my-topic-3=243
my-topic:999
Why do you want to get that value? If you share more detail about the purpose, I can give you more good tip.
For your last question, it's not the correct way to get the count of messages with the offset value. If your topic has one partition and the producer is one, you can use it. You need to consider that the topic has several partitions.
If you want to get the number of messages from each producer, you can count it in the callback function that is onCompletion()
Or you can get the last offset using Consumer client like this:
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "your-brokers");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
Consumer<Long, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("topic_name");
Collection<TopicPartition> partitions = consumer.assignment();
consumer.seekToEnd(partitions);
for(TopicPartition tp: partitions) {
long offsetPosition = consumer.position(tp);
}
I have Java 8 application working with Apache Kafka 2.11-0.10.1.0. I need to use the seek feature to poll old messages from partitions. However I faced an exception of No current assignment for partition which is occurred every time I am trying to seekByOffset. Here's my class which is responsible for seeking topics to the specified timestamp:
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.OffsetAndTimestamp;
import org.apache.kafka.common.PartitionInfo;
import org.apache.kafka.common.TopicPartition;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.util.CollectionUtils;
import java.time.Instant;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.function.Function;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
/**
* The main purpose of this class is to move fetching point for each partition of the {#link KafkaConsumer}
* to some offset which is determined either by timestamp or by offset number.
*/
public class KafkaSeeker {
public static final long APP_STARTUP_TIME = Instant.now().toEpochMilli();
private final Logger LOGGER = LoggerFactory.getLogger(this.getClass());
private final KafkaConsumer<String, String> kafkaConsumer;
private ConsumerRecords<String, String> polledRecords;
public KafkaSeeker(KafkaConsumer<String, String> kafkaConsumer) {
this.kafkaConsumer = kafkaConsumer;
this.polledRecords = new ConsumerRecords<>(Collections.emptyMap());
}
/**
* For each assigned or subscribed topic {#link org.apache.kafka.clients.consumer.KafkaConsumer#seek(TopicPartition, long)}
* fetching pointer to the specified {#code timestamp}.
* If no messages were found in each partition for a topic,
* then {#link org.apache.kafka.clients.consumer.KafkaConsumer#seekToEnd(Collection)} will be called.
*
* Due to {#link KafkaConsumer#subscribe(Pattern)} and {#link KafkaConsumer#assign(Collection)} laziness
* method needs to execute dummy {#link KafkaConsumer#poll(long)} method. All {#link ConsumerRecords} which were
* polled from buffer are swallowed and produce warning logs.
*
* #param timestamp is used to find proper offset to seek to
* #param topics are used to seek only specific topics. If not specified or empty, all subscribed topics are used.
*/
public Map<TopicPartition, OffsetAndTimestamp> seek(long timestamp, Collection<String> topics) {
this.polledRecords = kafkaConsumer.poll(0);
Collection<TopicPartition> topicPartitions;
if (CollectionUtils.isEmpty(topics)) {
topicPartitions = kafkaConsumer.assignment();
} else {
topicPartitions = topics.stream()
.map(it -> {
List<Integer> partitions = kafkaConsumer.partitionsFor(it).stream()
.map(PartitionInfo::partition).collect(Collectors.toList());
return partitions.stream().map(partition -> new TopicPartition(it, partition));
})
.flatMap(it -> it)
.collect(Collectors.toList());
}
if (topicPartitions.isEmpty()) {
throw new IllegalStateException("Kafka consumer doesn't have any subscribed topics.");
}
Map<TopicPartition, Long> timestampsByTopicPartitions = topicPartitions.stream()
.collect(Collectors.toMap(Function.identity(), topicPartition -> timestamp));
Map<TopicPartition, Long> beginningOffsets = kafkaConsumer.beginningOffsets(topicPartitions);
Map<TopicPartition, OffsetAndTimestamp> offsets = kafkaConsumer.offsetsForTimes(timestampsByTopicPartitions);
for (Map.Entry<TopicPartition, OffsetAndTimestamp> entry : offsets.entrySet()) {
TopicPartition topicPartition = entry.getKey();
if (entry.getValue() != null) {
LOGGER.info("Kafka seek topic:partition [{}:{}] from [{} offset] to [{} offset].",
topicPartition.topic(),
topicPartition.partition(),
beginningOffsets.get(topicPartition),
entry.getValue());
kafkaConsumer.seek(topicPartition, entry.getValue().offset());
} else {
LOGGER.info("Kafka seek topic:partition [{}:{}] from [{} offset] to the end of partition.",
topicPartition.topic(),
topicPartition.partition());
kafkaConsumer.seekToEnd(Collections.singleton(topicPartition));
}
}
return offsets;
}
public ConsumerRecords<String, String> getPolledRecords() {
return polledRecords;
}
}
Before calling the method I have consumer subscribed to a single topic like this consumer.subscribe(singletonList(kafkaTopic));. When I get kafkaConsumer.assignment() it returns zero TopicPartitions assigned. But if I specify the topic and get its partitions then I have valid TopicPartitions, although they are failing on seek call with the error in the title. What is something I forgot?
The correct way to reliably seek and check current assignment is to wait for the onPartitionsAssigned() callback after subscribing. On a newly created (still not connected) consumer, calling poll() once does not guarantees it will immedaitely be connected and assigned partitions.
As a basic example, see the code below that subscribes to a topic, and in the assigned callback, seeks to the desired position. Finally you'll notice that the poll loop correctly only sees records from the seek location and not from the previous committed or reset offset.
public static final Map<TopicPartition, Long> offsets = Map.of(new TopicPartition("testtopic", 0), 5L);
public static void main(String args[]) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "test");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
try (Consumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(Collections.singletonList("testtopic"), new ConsumerRebalanceListener() {
#Override
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {}
#Override
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
System.out.println("Assigned " + partitions);
for (TopicPartition tp : partitions) {
OffsetAndMetadata oam = consumer.committed(tp);
if (oam != null) {
System.out.println("Current offset is " + oam.offset());
} else {
System.out.println("No committed offsets");
}
Long offset = offsets.get(tp);
if (offset != null) {
System.out.println("Seeking to " + offset);
consumer.seek(tp, offset);
}
}
}
});
for (int i = 0; i < 10; i++) {
System.out.println("Calling poll");
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100L));
for (ConsumerRecord<String, String> r : records) {
System.out.println("record from " + r.topic() + "-" + r.partition() + " at offset " + r.offset());
}
}
}
}
KafkaConsumer<String, byte[]> consumer = new KafkaConsumer<>(props);
// Get topic partitions
List<TopicPartition> partitions = consumer
.partitionsFor(topic)
.stream()
.map(partitionInfo ->
new TopicPartition(topic, partitionInfo.partition()))
.collect(Collectors.toList());
// Explicitly assign the partitions to our consumer
consumer.assign(partitions);
//seek, query offsets, or poll
Please note that this disables consumer group management and rebalancing operations. When possible use #Mickael Maison's approach.
I am trying to run Kafka producer found here https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example.
This is the Kafka Producer class:
package com.kafka.demo;
import java.util.*;
import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;
/* A Simple producer using API. This producer is creating log messages and sending to a topic called vulab123 on a node with name vulab-build-system Please make sure vulab-build-system is configured in the /etc/hosts file in your unix or linux environment */
public class KafkaProducer
{
public static void main(String[] args)
{
//long events = Long.parseLong(args[0]);
long events =100;
Random rnd = new Random();
Properties props = new Properties();
props.put("metadata.broker.list", "10.25.3.207:6667");
//props.put("producer.type", "sync");
props.put("zk.connect","10.25.3.207:2181");
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("partitioner.class", "com.kafka.demo.KafkaPartitioner");
props.put("request.required.acks", "1");
props.put("producer.type","async");
ProducerConfig config = new ProducerConfig(props);
Producer<String, String> producer = new Producer<String, String>(config);
for (long nEvents = 0; nEvents<events; nEvents++)
{
System.out.println("creating event "+nEvents);
long runtime = new Date().getTime();
String ip = "192.168.2."+ rnd.nextInt(255);
String msg = runtime + ",www.vulab.com," + ip;
KeyedMessage<String, String> data = new KeyedMessage<String, String>("vulab123", ip, msg);
producer.send(data);
//System.out.println(data);
}
producer.close();
}
}
This is the Kafka Partitioner class:
package com.kafka.demo;
import kafka.producer.Partitioner;
import kafka.utils.VerifiableProperties;
public class KafkaPartitioner implements Partitioner
{
public KafkaPartitioner (VerifiableProperties props)
{
}
public int partition(Object key, int a_numPartitions)
{
int partition = 0;
String stringKey = (String) key;
int offset = stringKey.lastIndexOf('.');
if (offset > 0)
{
partition = Integer.parseInt( stringKey.substring(offset+1)) % a_numPartitions;
}
return partition;
}
}
I get the following error:
Exception in thread "main" java.lang.IllegalArgumentException: wrong number of arguments
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at kafka.utils.Utils$.getObject(Utils.scala:567)
at kafka.javaapi.producer.Producer.<init>(Producer.scala:39)
at com.kafka.demo.KafkaProducer.main(KafkaProducer.java:34)
Any ideas or suggestions regarding how to solve this.
It is only intuition, but try to use:
KafkaProducer<String, String> producer = new KafkaProducer<String, String>(config);
instead of
Producer<String, String> producer = new Producer<String, String>(config);
Also make sure that you are using proper version of Apache Kafka library.