We Keep Coding

Restart processing since last committed offset in Kafka

Pretext
I have:
A connected Kafka consumer
The consumer is part of a consumer group.
A rebalance listener attached to the consumer
Auto-commit is disabled
Additionally, I have a method that takes in two parameters, the consumer, and a rebalance-listener that tracks what partitions have been assigned to the consumer
void aggregateProcessing(ConsumerRecords<String, SomeClass> consumer, RebalanceListener listener)
public class RebalanceListener implements ConsumerRebalanceListener {
private Set<TopicPartition> assignedPartitions = new LinkedHashSet<>();
#Override
public void onPartitionsAssigned(final Collection<TopicPartition> partitions) {
// keep track of assigned partitions
}
}
Goal
This method is run on a timer, and its goal is to process records until there are none left to read or up until some max time in all of the partitions.
Since rebalancing could happen in the middle of consumption (after consumer.poll() has fired several times already), I would like to detect this, reset, and restart processing from the last committed offset for all assigned partitions (even if that had already been assigned).
Is there a way to reset the consumer's internal offset for each partition back to the latest committed offset for a list of assigned partitions?
Sidenote
I understand reprocessing for all partitions (instead of just for the ones that were changed) is less efficient than selectively expunging some of the processing, but will probably be significantly easier than tracking what data needs to be expunged when a partition is removed.
Thanks!

Yes You Can
You have to track your committed offset manually for this purpose.
Whenever the partitions are reinvoked from the consumer, you have to save the partitions and their committed offsets in DB.
When reassigned to the partitions to the consumer, you have to seek from a specific offset that is stored in your datastore.
Your rebalance listener will listen when these revoke and assign events happens.
A sample implementation of the rebalance listener
public class SaveOffsetsOnRebalance implements ConsumerRebalanceListener {
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
commitDBTransaction();
}
public void onPartitionsAssigned(Collection<TopicPartition> partitions){
for(TopicPartition partition: partitions) {
consumer.seek(partition, getOffsetFromDB(partition));
}
}
}

As Kafka documentation described in KafkaConsumer
Offsets and Consumer Position
Kafka maintains a numerical offset for
each record in a partition. This offset acts as a unique identifier of
a record within that partition, and also denotes the position of the
consumer in the partition. For example, a consumer which is at
position 5 has consumed records with offsets 0 through 4 and will next
receive the record with offset 5. There are actually two notions of
position relevant to the user of the consumer: The position of the
consumer gives the offset of the next record that will be given out.
It will be one larger than the highest offset the consumer has seen in
that partition. It automatically advances every time the consumer
receives messages in a call to poll(Duration).
The committed position is the last offset that has been stored
securely. Should the process fail and restart, this is the offset that
the consumer will recover to. The consumer can either automatically
commit offsets periodically; or it can choose to control this
committed position manually by calling one of the commit APIs (e.g.
commitSync and commitAsync).
So, When you need to start with latest committed offset and you have disabled enable.auto.commit, Then you can manually commit your proceesed message's offset.
can choose to control this
committed position manually by calling one of the commit APIs (e.g.
commitSync and commitAsync).
Then after restarting and rebalancing Kafka, Consumers will start to consume from last committed (processed) offset.
The above scenario is when you use the Kafka storage for consumer offsets. If you already have offsets you want to start consuming, Then you can control consumer's starting offset by Consumer.Seek() before start consuming.
Controlling The Consumer's Position
In most use cases the consumer will simply consume records from
beginning to end, periodically committing its position (either
automatically or manually). However Kafka allows the consumer to
manually control its position, moving forward or backwards in a
partition at will. This means a consumer can re-consume older records,
or skip to the most recent records without actually consuming the
intermediate records. There are several instances where manually
controlling the consumer's position can be useful.
One case is for time-sensitive record processing it may make sense for
a consumer that falls far enough behind to not attempt to catch up
processing all records, but rather just skip to the most recent
records.
Another use case is for a system that maintains local state as
described in the previous section. In such a system the consumer will
want to initialize its position on start-up to whatever is contained
in the local store. Likewise if the local state is destroyed (say
because the disk is lost) the state may be recreated on a new machine
by re-consuming all the data and recreating the state (assuming that
Kafka is retaining sufficient history).
Kafka allows specifying the position using seek(TopicPartition, long)
to specify the new position. Special methods for seeking to the
earliest and latest offset the server maintains are also available (
seekToBeginning(Collection) and seekToEnd(Collection) respectively).

Setting all partitions to their latest offset in Kafka but getting "No current assignment for partition" [duplicate]

I am using KafkaConsumer 0.10 Java api. I want to consume from a specific partition and specific offset. I looked up and found that there is a seek method but its throwing an exception. Anyone had a similar use case or solution ?
Code:
KafkaConsumer<String, byte[]> consumer = new KafkaConsumer<>(consumerProps);
consumer.seek(new TopicPartition("mytopic", 1), 4);
Exception
java.lang.IllegalStateException: No current assignment for partition mytopic-1
at org.apache.kafka.clients.consumer.internals.SubscriptionState.assignedState(SubscriptionState.java:251)
at org.apache.kafka.clients.consumer.internals.SubscriptionState.seek(SubscriptionState.java:276)
at org.apache.kafka.clients.consumer.KafkaConsumer.seek(KafkaConsumer.java:1135)
at xx.xxx.xxx.Test.main(Test.java:182)

Before you can seek() you first need to subscribe() to a topic or assign() partition of a topic to the consumer. Also keep in mind, that subscribe() and assign() are lazy -- thus, you also need to do a "dummy call" to poll() before you can use seek().
Note: as of Kafka 2.0, the new poll(Duration timeout) is async and it's not guaranteed that you have a complete assignment when poll returns. Thus, you might need to check your assignment before using seek() and also poll again to refresh the assignment. (Cf. KIP-266 for details)
If you use subscribe(), you use group management: thus, you can start multiple consumers using the same group.id and all partitions of the topic will be assigned evenly over all consumers within the group automatically (each partition will get assigned to a single consumer in the group).
If you want to read specific partitions, you need to use manual assignment via assign(). This allows you to do any assignment you want.
Btw: KafkaConsumer has a very long an detailed class JavaDoc including examples. It's worth to read it.

If you do not want to use poll() and retrieve map records, and change the offset itself.
Kafka version 0.11
Try this:
...
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("Test_topic1", "Test_topic2"));
List<TopicPartition> partitions =consumer.partitionsFor("Test_topic1").stream().map(part->{TopicPartition tp = new TopicPartition(part.topic(),part.partition()); return tp;}).collect(Collectors.toList());
Field coordinatorField = consumer.getClass().getDeclaredField("coordinator");
coordinatorField.setAccessible(true);
ConsumerCoordinator coordinator = (ConsumerCoordinator)coordinatorField.get(consumer);
coordinator.poll(new Date().getTime(), 1000);//Watch out for your local date and time settings
consumer.seekToBeginning(partitions); //or other seek
Poll for coordinator events. This ensures that the coordinator is known and that the consumer has joined the group (if it is using group management). This also handles periodic offset commits if they are enabled.

Please use consumer.assign with consumer.seek and not consumer.subscribe
After these changes, it will execute fine.

Emitting values from ktable and modifying it

I'm trying to solve the following problem with kafka.
There is a topic. let's call it src-topic. I receive records from this topic from time to time. I would like to store those values in a ktable and emit the values stored in the ktable every 10 seconds to dst-topic. When I emit a value from this ktable for the first time then I want to append 1 to the record I emit. Every subsequent time I would like to append 0 to the emitted record.
I'm looking for a correct and preferably idiomatic solution to this issue.
One of the solutions I see is to emit a record with 1 appended when I ingest from src-topic and then store in the ktable the record with 0 appended. Another thread will be reading from this ktable and emitting the records regularly. The problem with this approach is that it has a race condition.
Any advice will be appreciated.

There is no straight forward way to do this. Note, a KTable is a changelog stream (it might have a table state internally -- not all KTables do have a state --, but that's an implementation detail).
Thus, a KTable is a stream and you cannot flush a stream... And because the state (if there is any) is internal, you cannot flush the state either.
You can only access the state via Interactive Queries that also allow to do a range scan. However, this will not emit anything downstream but gives the data to the "non Streams part" of you application.
I think, you will need to use low-level Processor API to get the result you want.

Kafka KTable - shared aggregation across machines

Assume that I have a topic with numerous partitions. Im writing K/V data in there and want to aggregate said data in Tumbling Windows by keys.
Assume that I've launched as many worker instances as I have partitions and each worker instance is running on a separate machine.
How would I go about insuring that the resultant aggregations include all values for each key? IE I don't want each worker instance to have some subset of the values.
Is this something that a StateStore would be used for? Does Kafka manage this on its own or do I need to come up with a method?

How would I go about insuring that the resultant aggregations include all values for each key? IE I don't want each worker instance to have some subset of the values.
In general, Kafka Streams ensures that all values for the same key will be processed by the same (and only one) stream task, which also means only one application instance (what you described as "worker instance") will process the values for that key. Note that an app instance may run 1+ stream tasks, but these tasks are isolated.
This behavior is achieved through the partitioning of the data, and Kafka Streams ensures that a partition is always processed by the same and only one stream task. The logical link to keys/values is that, in Kafka and Kafka Streams, a key is always sent to the same partition (there is a gotcha here, but I'm not sure whether it makes sense to go into details for the scope of this question), hence one particular partition -- among possible many partitions -- contains all the values for the same key.
In some situations, such as when joining two streams A and B, you must ensure though that the aggregation will operate on the same key to ensure that data from both streams are co-located in the same stream task -- which, again, is all about ensuring that the relevant input stream partitions and thus matching the keys (from A and B, respectively) are made available in the same stream task. A typical method you'd use here is selectKey(). Once that is done, Kafka Streams ensures that, for joining the two streams A and B as well as for creating the joined output stream, all values for the same key will be processed by the same stream task and thus the same application instance.
Example:
Stream A has key userId with value { georegion }.
Stream B has key georegion with value { continent, description }.
Joining two streams only works (as of Kafka 0.10.0) when both streams use the same key. In this example, this means that you must re-key (and thus re-partition) stream A so that the resulting key is changed from userId to georegion. Otherwise, as of Kafka 0.10, you can't join A and B because data is not co-located in the stream task that is responsible for actually performing the join.
In this example, you could re-key/re-partition stream A via:
// Kafka 0.10.0.x (latest stable release as of Sep 2016)
A.map((userId, georegion) -> KeyValue.pair(georegion, userId)).through("rekeyed-topic")
// Upcoming versions of Kafka (not released yet)
A.map((userId, georegion) -> KeyValue.pair(georegion, userId))
The through() call is only required in Kafka 0.10.0 to actually trigger re-partitioning, and later versions of Kafka will do these automatically for you (this upcoming functionality is already completed and available in Kafka trunk).
Is this something that a StateStore would be used for? Does Kafka manage this on its own or do I need to come up with a method?
In general, no. The behavior above is achieved through partitioning, not through state stores.
Sometimes state stores are involved because of the operations you have defined for a stream, which might explain why you were asking this question. For example, a windowing operation will require state to be managed, and thus a state store will be created behind the scenes. But your actual question -- "insuring that the resultant aggregations include all values for each key" -- has nothing to do with state stores, it's about the partitioning behavior.

With worker instance, I assume you mean a Kafka Streams application instance, right? (Because there is no master/worker pattern in Kafka Streams -- it's a library and not a framework -- we do not use the term "worker".)
If you want to co-locate data per key, you need to partition the data by key. Thus, either your data is partitioned by key by your external producer when data gets written into a topic from the beginning on. Or you explicitly set a new key within Kafka Streams application (using for example selectKey() or map()) and re-distributed via a call to through().
(The explicit call to through() will not be necessary in future releases, ie, 0.10.1 and Kafka Streams will re-distribute records automatically if necessary.)
If messages/record should be partitioned, the key must not be null. You can also change the partitioning schema via producer configuration partitioner.class (see https://kafka.apache.org/documentation.html#producerconfigs).
Partitioning is completely independent from StateStores, even if StateStores are usually used on top of partitioned data.

Kafka KStreams - processing timeouts

I am attempting to use <KStream>.process() with a TimeWindows.of("name", 30000) to batch up some KTable values and send them on. It seems that 30 seconds exceeds the consumer timeout interval after which Kafka considers said consumer to be defunct and releases the partition.
I've tried upping the frequency of poll and commit interval to avoid this:
config.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, "5000");
config.put(StreamsConfig.POLL_MS_CONFIG, "5000");
Unfortunately these errors are still occurring:
(lots of these)
ERROR o.a.k.s.p.internals.RecordCollector - Error sending record to topic kafka_test1-write_aggregate2-changelog
org.apache.kafka.common.errors.TimeoutException: Batch containing 1 record(s) expired due to timeout while requesting metadata from brokers for kafka_test1-write_aggregate2-changelog-0
Followed by these:
INFO o.a.k.c.c.i.AbstractCoordinator - Marking the coordinator 12.34.56.7:9092 (id: 2147483547 rack: null) dead for group kafka_test1
WARN o.a.k.s.p.internals.StreamThread - Failed to commit StreamTask #0_0 in thread [StreamThread-1]:
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:578)
Clearly I need to be sending heartbeats back to the server more often. How?
My topology is:
KStreamBuilder kStreamBuilder = new KStreamBuilder();
KStream<String, String> lines = kStreamBuilder.stream(TOPIC);
KTable<Windowed<String>, String> kt = lines.aggregateByKey(
new DBAggregateInit(),
new DBAggregate(),
TimeWindows.of("write_aggregate2", 30000));
DBProcessorSupplier dbProcessorSupplier = new DBProcessorSupplier();
kt.toStream().process(dbProcessorSupplier);
KafkaStreams kafkaStreams = new KafkaStreams(kStreamBuilder, streamsConfig);
kafkaStreams.start();
The KTable is grouping values by key every 30 seconds. In Processor.init() I call context.schedule(30000).
DBProcessorSupplier provides an instance of DBProcessor. This is an implementation of AbstractProcessor where all the overrides have been provided. All they do is LOG so I know when each is being hit.
It's a pretty simple topology but it's clear I'm missing a step somewhere.
Edit:
I get that I can adjust this on the server side but Im hoping there is a client-side solution. I like the notion of partitions being made available pretty quickly when a client exits / dies.
Edit:
In an attempt to simplify the problem I removed the aggregation step from the graph. It's now just consumer->processor(). (If I send the consumer directly to .print() it works v quickly so I know it's ok). (Similarly If I output the aggregation (KTable) via .print() it seems ok too).
What I found was that the .process() - which should be calling .punctuate() every 30 seconds is actually blocking for variable lengths of time and outputting somewhat randomly (if at all).
Main program
Debug output
Processor Supplier
Processor
Further:
I set the debug level to 'debug' and reran. Im seeing lots of messages:
DEBUG o.a.k.s.p.internals.StreamTask - Start processing one record [ConsumerRecord <info>
but a breakpoint in the .punctuate() function isn't getting hit. So it's doing lots of work but not giving me a chance to use it.

A few clarifications:
StreamsConfig.COMMIT_INTERVAL_MS_CONFIG is a lower bound on the commit interval, ie, after a commit, the next commit happens not before this time passed. Basically, Kafka Stream tries to commit ASAP after this time passed, but there is no guarantee whatsoever how long it will actually take to do the next commit.
StreamsConfig.POLL_MS_CONFIG is used for the internal KafkaConsumer#poll() call, to specify the maximum blocking time of the poll() call.
Thus, both values are not helpful to heartbeat more often.
Kafka Streams follows a "depth-first" strategy when processing record. This means, that after a poll() for each record all operators of the topology are executed. Let's assume you have three consecutive maps, than all three maps will be called for the first record, before the next/second record will get processed.
Thus, the next poll() call will be made, after all record of the first poll() got fully processed. If you want to heartbeat more often, you need to make sure, that a single poll() call fetches less records, such that processing all records takes less time and the next poll() will be triggered earlier.
You can use configuration parameters for KafkaConsumer that you can specify via StreamsConfig to get this done (see https://kafka.apache.org/documentation.html#consumerconfigs):
streamConfig.put(ConsumerConfig.XXX, VALUE);
max.poll.records: if you decrease this value, less record will be polled
session.timeout.ms: if you increase this value, there is more time for processing data (adding this for completeness because it is actually a client setting and not a server/broker side configuration -- even if you are aware of this solution and do not like it :))
EDIT
As of Kafka 0.10.1 it is possible (and recommended) to prefix consumer and procuder configs within streams config. This avoids parameter conflicts as some parameter names are used for consumer and producer and cannot be distinguiesh otherwise (and would be applied to consumer and producer at the same time).
To prefix a parameter you can use StreamsConfig#consumerPrefix() or StreamsConfig#producerPrefix(), respectively. For example:
streamsConfig.put(StreamsConfig.consumerPrefix(ConsumerConfig.PARAMETER), VALUE);
One more thing to add: The scenario described in this question is a known issue and there is already KIP-62 that introduces a background thread for KafkaConsumer that send heartbeats, thus decoupling heartbeats from poll() calls. Kafka Streams will leverage this new feature in upcoming releases.