Pretext
I have:
A connected Kafka consumer
The consumer is part of a consumer group.
A rebalance listener attached to the consumer
Auto-commit is disabled
Additionally, I have a method that takes in two parameters, the consumer, and a rebalance-listener that tracks what partitions have been assigned to the consumer
void aggregateProcessing(ConsumerRecords<String, SomeClass> consumer, RebalanceListener listener)
public class RebalanceListener implements ConsumerRebalanceListener {
private Set<TopicPartition> assignedPartitions = new LinkedHashSet<>();
#Override
public void onPartitionsAssigned(final Collection<TopicPartition> partitions) {
// keep track of assigned partitions
}
}
Goal
This method is run on a timer, and its goal is to process records until there are none left to read or up until some max time in all of the partitions.
Since rebalancing could happen in the middle of consumption (after consumer.poll() has fired several times already), I would like to detect this, reset, and restart processing from the last committed offset for all assigned partitions (even if that had already been assigned).
Is there a way to reset the consumer's internal offset for each partition back to the latest committed offset for a list of assigned partitions?
Sidenote
I understand reprocessing for all partitions (instead of just for the ones that were changed) is less efficient than selectively expunging some of the processing, but will probably be significantly easier than tracking what data needs to be expunged when a partition is removed.
Thanks!
Yes You Can
You have to track your committed offset manually for this purpose.
Whenever the partitions are reinvoked from the consumer, you have to save the partitions and their committed offsets in DB.
When reassigned to the partitions to the consumer, you have to seek from a specific offset that is stored in your datastore.
Your rebalance listener will listen when these revoke and assign events happens.
A sample implementation of the rebalance listener
public class SaveOffsetsOnRebalance implements ConsumerRebalanceListener {
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
commitDBTransaction();
}
public void onPartitionsAssigned(Collection<TopicPartition> partitions){
for(TopicPartition partition: partitions) {
consumer.seek(partition, getOffsetFromDB(partition));
}
}
}
As Kafka documentation described in KafkaConsumer
Offsets and Consumer Position
Kafka maintains a numerical offset for
each record in a partition. This offset acts as a unique identifier of
a record within that partition, and also denotes the position of the
consumer in the partition. For example, a consumer which is at
position 5 has consumed records with offsets 0 through 4 and will next
receive the record with offset 5. There are actually two notions of
position relevant to the user of the consumer: The position of the
consumer gives the offset of the next record that will be given out.
It will be one larger than the highest offset the consumer has seen in
that partition. It automatically advances every time the consumer
receives messages in a call to poll(Duration).
The committed position is the last offset that has been stored
securely. Should the process fail and restart, this is the offset that
the consumer will recover to. The consumer can either automatically
commit offsets periodically; or it can choose to control this
committed position manually by calling one of the commit APIs (e.g.
commitSync and commitAsync).
So, When you need to start with latest committed offset and you have disabled enable.auto.commit, Then you can manually commit your proceesed message's offset.
can choose to control this
committed position manually by calling one of the commit APIs (e.g.
commitSync and commitAsync).
Then after restarting and rebalancing Kafka, Consumers will start to consume from last committed (processed) offset.
The above scenario is when you use the Kafka storage for consumer offsets. If you already have offsets you want to start consuming, Then you can control consumer's starting offset by Consumer.Seek() before start consuming.
Controlling The Consumer's Position
In most use cases the consumer will simply consume records from
beginning to end, periodically committing its position (either
automatically or manually). However Kafka allows the consumer to
manually control its position, moving forward or backwards in a
partition at will. This means a consumer can re-consume older records,
or skip to the most recent records without actually consuming the
intermediate records. There are several instances where manually
controlling the consumer's position can be useful.
One case is for time-sensitive record processing it may make sense for
a consumer that falls far enough behind to not attempt to catch up
processing all records, but rather just skip to the most recent
records.
Another use case is for a system that maintains local state as
described in the previous section. In such a system the consumer will
want to initialize its position on start-up to whatever is contained
in the local store. Likewise if the local state is destroyed (say
because the disk is lost) the state may be recreated on a new machine
by re-consuming all the data and recreating the state (assuming that
Kafka is retaining sufficient history).
Kafka allows specifying the position using seek(TopicPartition, long)
to specify the new position. Special methods for seeking to the
earliest and latest offset the server maintains are also available (
seekToBeginning(Collection) and seekToEnd(Collection) respectively).
Related
I'm doing manual commitSync to Kafka, and I notice in the DSL they use a simple Map and not MultiMap, so I guess then every time I invoke
consumer.commitSync(Map.of(topicPartition, new OffsetAndMetadata(record.offset())))
Is just for a single record in the partition.
Any chance to send two offsets of the same topicPartition in the same commitSync
It's a Map, so no, you cannot have multiple instances of the same topicPartition key.
The offset is a single number. If you were able to commit multiple, then your consumer (in the same group) would always have to start reading from the greatest of those values.
You can commit offsets for other TopicPartitions, however, in one commit call, or you can commit the same value to other consumer groups using a differently configured Consumer instance.
I have a consumer set up that manually commits offsets. Events are in the millions to low billions. I'm committing offsets if and only if processing was successful within the consumer batch being processed. However, we're noticing that even with commitSync being called successfully, we have hundreds of thousands of duplicates. We will commitSync and just repull the same exact data in the consumer on the next poll from the topic. Why would this happen?
#Ryan - Kindly ensure that you have set below property for Consumer
props.setProperty("enable.auto.commit", "false");
Even if above is not giving you the desired result due to huge load, kindly commit the current offset with below constructor so that you will not get the offset in next Poll.
public void commitSync(java.util.Map<TopicPartition,OffsetAndMetadata> offsets)
Follow API at
commitSync
We're using Storm with Kafka and ZooKeeper. We had a situation where we had to delete some topics and recreate them with different names. Our Kafka spouts stayed the same, aside from now reading from the new topic names. However now the spouts are using the offsets from the old topic partitions when trying to read from the new topics. So the tail position of my-topic-name partition 0 will be 500 but the offset will be something like 10000.
Is there a way to reset the offset position so it matches the tail of the topic?
There a multiple options (as Storm's KafkaSpout does not provide any API to define the starting offset).
If you want to consumer from the tail of the log you should delete old offsets
depending on you Kafka version
(pre 0.9) you can manipulate ZK (which is a little tricky)
(0.9+) or you try do delete the offset from the topic __consumer_offsets (which is also tricky and might delete other offset you want to preserve, too)
if no offsets are there, you can restart your spout with auto offset reset policy "latest" or "largest" (depending on you Kafka version)
as an alternative (which I would recommend), you can write a small client application that uses seek() to manipulate the offset in the way you need them and commit() the offsets. This client must use the same group ID as you KafkaSpout and must subscribe to the same topic(s). Furthermore, you need to make sure that this client application is running a single consumer group member so it get's all partitions assigned.
for this, you an either seek to the end of the log and commit
or you commit an invalid offset (like -1) and rely on auto offset reset configuration"latest" or "largest" (depending on you Kafka version)
For Kafka Streams, there is a "Application Reset Tool" that does a similar thing to manipulate committed offsets. If you want to get some details, you can read this blog post http://www.confluent.io/blog/data-reprocessing-with-kafka-streams-resetting-a-streams-application/
(disclaimer: I am the author of the post and it is about Kafka Streams -- nevertheless, the underlying offset manipulation ideas are the same)
I am attempting to use <KStream>.process() with a TimeWindows.of("name", 30000) to batch up some KTable values and send them on. It seems that 30 seconds exceeds the consumer timeout interval after which Kafka considers said consumer to be defunct and releases the partition.
I've tried upping the frequency of poll and commit interval to avoid this:
config.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, "5000");
config.put(StreamsConfig.POLL_MS_CONFIG, "5000");
Unfortunately these errors are still occurring:
(lots of these)
ERROR o.a.k.s.p.internals.RecordCollector - Error sending record to topic kafka_test1-write_aggregate2-changelog
org.apache.kafka.common.errors.TimeoutException: Batch containing 1 record(s) expired due to timeout while requesting metadata from brokers for kafka_test1-write_aggregate2-changelog-0
Followed by these:
INFO o.a.k.c.c.i.AbstractCoordinator - Marking the coordinator 12.34.56.7:9092 (id: 2147483547 rack: null) dead for group kafka_test1
WARN o.a.k.s.p.internals.StreamThread - Failed to commit StreamTask #0_0 in thread [StreamThread-1]:
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:578)
Clearly I need to be sending heartbeats back to the server more often. How?
My topology is:
KStreamBuilder kStreamBuilder = new KStreamBuilder();
KStream<String, String> lines = kStreamBuilder.stream(TOPIC);
KTable<Windowed<String>, String> kt = lines.aggregateByKey(
new DBAggregateInit(),
new DBAggregate(),
TimeWindows.of("write_aggregate2", 30000));
DBProcessorSupplier dbProcessorSupplier = new DBProcessorSupplier();
kt.toStream().process(dbProcessorSupplier);
KafkaStreams kafkaStreams = new KafkaStreams(kStreamBuilder, streamsConfig);
kafkaStreams.start();
The KTable is grouping values by key every 30 seconds. In Processor.init() I call context.schedule(30000).
DBProcessorSupplier provides an instance of DBProcessor. This is an implementation of AbstractProcessor where all the overrides have been provided. All they do is LOG so I know when each is being hit.
It's a pretty simple topology but it's clear I'm missing a step somewhere.
Edit:
I get that I can adjust this on the server side but Im hoping there is a client-side solution. I like the notion of partitions being made available pretty quickly when a client exits / dies.
Edit:
In an attempt to simplify the problem I removed the aggregation step from the graph. It's now just consumer->processor(). (If I send the consumer directly to .print() it works v quickly so I know it's ok). (Similarly If I output the aggregation (KTable) via .print() it seems ok too).
What I found was that the .process() - which should be calling .punctuate() every 30 seconds is actually blocking for variable lengths of time and outputting somewhat randomly (if at all).
Main program
Debug output
Processor Supplier
Processor
Further:
I set the debug level to 'debug' and reran. Im seeing lots of messages:
DEBUG o.a.k.s.p.internals.StreamTask - Start processing one record [ConsumerRecord <info>
but a breakpoint in the .punctuate() function isn't getting hit. So it's doing lots of work but not giving me a chance to use it.
A few clarifications:
StreamsConfig.COMMIT_INTERVAL_MS_CONFIG is a lower bound on the commit interval, ie, after a commit, the next commit happens not before this time passed. Basically, Kafka Stream tries to commit ASAP after this time passed, but there is no guarantee whatsoever how long it will actually take to do the next commit.
StreamsConfig.POLL_MS_CONFIG is used for the internal KafkaConsumer#poll() call, to specify the maximum blocking time of the poll() call.
Thus, both values are not helpful to heartbeat more often.
Kafka Streams follows a "depth-first" strategy when processing record. This means, that after a poll() for each record all operators of the topology are executed. Let's assume you have three consecutive maps, than all three maps will be called for the first record, before the next/second record will get processed.
Thus, the next poll() call will be made, after all record of the first poll() got fully processed. If you want to heartbeat more often, you need to make sure, that a single poll() call fetches less records, such that processing all records takes less time and the next poll() will be triggered earlier.
You can use configuration parameters for KafkaConsumer that you can specify via StreamsConfig to get this done (see https://kafka.apache.org/documentation.html#consumerconfigs):
streamConfig.put(ConsumerConfig.XXX, VALUE);
max.poll.records: if you decrease this value, less record will be polled
session.timeout.ms: if you increase this value, there is more time for processing data (adding this for completeness because it is actually a client setting and not a server/broker side configuration -- even if you are aware of this solution and do not like it :))
EDIT
As of Kafka 0.10.1 it is possible (and recommended) to prefix consumer and procuder configs within streams config. This avoids parameter conflicts as some parameter names are used for consumer and producer and cannot be distinguiesh otherwise (and would be applied to consumer and producer at the same time).
To prefix a parameter you can use StreamsConfig#consumerPrefix() or StreamsConfig#producerPrefix(), respectively. For example:
streamsConfig.put(StreamsConfig.consumerPrefix(ConsumerConfig.PARAMETER), VALUE);
One more thing to add: The scenario described in this question is a known issue and there is already KIP-62 that introduces a background thread for KafkaConsumer that send heartbeats, thus decoupling heartbeats from poll() calls. Kafka Streams will leverage this new feature in upcoming releases.
Using the new Kafka Java consumer api, I run a single consumer to consume messages. When all available messages are consumed, I kill it with kill -15.
Now I would like to reset the offsets to start. I would like to avoid to just use a different consumer group. What I tried is the following sequence of calls, using the same group as the consumer that just had finished reading the data.
assign(topicPartition);
OffsetAndMetadata om = new OffsetAndMetadata(0);
commitSync(Collections.singletonMap(topicPartition, 0));
I thought I had got this working in a test, but now I always just get:
ERROR internals.ConsumerCoordinator: Error UNKNOWN_MEMBER_ID occurred while committing offsets for group queue
Exception in thread "main" org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed due to group rebalance
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:552)
Is it in principle wrong to combine assign with commitSync, possibly because only subscribe and commitSync go together? The docs only say that assign does not go along with subscribe, but I thought this applies only in one consumer process. (In fact I was even hoping to run the offset-reset consumer while the other consumer is up, hoping that the other one might notice the offset change and start over again. But shutting it down first is fine too.)
Any ideas?
Found the problem. The approach described in my question works well, given we respect the following conditions:
There may be no other consumer running with the targeted group.id. Even if a consumer is subscribed only to other topics, this hinders committing topic offsets after calling assign() instead of subscribe().
After the last other consumer has stopped, it takes 30 seconds (I think it is group.max.session.timeout.ms) before the operation can succeed. The indicative log message from kafka is
Group X generation Y is dead and removed
Once this appears in the log, the sequence
assign(topicPartition);
OffsetAndMetadata om = new OffsetAndMetadata(0);
commitSync(Collections.singletonMap(topicPartition, 0));
can succeed.
Why even commit offsets in the first place?
Set enable.auto.commit to false in Properties and don't commit it at all if you just re-read all messages on restart.
To reset offset you can use for example these methods:
public void seek(TopicPartition partition, long offset)
public void seekToBeginning(TopicPartition... partitions)