I've written a stream that takes in messages and sends out a table of the keys that have appeared. If something appears, it will show a count of 1. This is a simplified version of my production code in order to demonstrate the bug. In a live run, a message is sent out for each message received.
However, when I run it in a unit test using ProcessorTopologyTestDriver, I get a different behavior. If a key that has already been seen before is received, I get an extra message.
If I send messages with keys "key1", then "key2", then "key1", I get the following output.
key1 - 1
key2 - 1
key1 - 0
key1 - 1
For some reason, it decrements the value before adding it back in. This only happens when using ProcessorTopologyTestDriver. Is this expected? Is there a work around? Or is this a bug?
Here's my topology:
final StreamsBuilder builder = new StreamsBuilder();
KGroupedTable<String, String> groupedTable
= builder.table(applicationConfig.sourceTopic(), Consumed.with(Serdes.String(), Serdes.String()))
.groupBy((key, value) -> KeyValue.pair(key, value), Serialized.with(Serdes.String(), Serdes.String()));
KTable<String, Long> countTable = groupedTable.count();
KStream<String, Long> countTableAsStream = countTable.toStream();
countTableAsStream.to(applicationConfig.outputTopic(), Produced.with(Serdes.String(), Serdes.Long()));
Here's my unit test code:
TopologyWithGroupedTable top = new TopologyWithGroupedTable(appConfig, map);
Topology topology = top.get();
ProcessorTopologyTestDriver driver = new ProcessorTopologyTestDriver(config, topology);
driver.process(inputTopic, "key1", "theval", Serdes.String().serializer(), Serdes.String().serializer());
driver.process(inputTopic, "key2", "theval", Serdes.String().serializer(), Serdes.String().serializer());
driver.process(inputTopic, "key1", "theval", Serdes.String().serializer(), Serdes.String().serializer());
ProducerRecord<String, Long> outputRecord = driver.readOutput(outputTopic, keyDeserializer, valueDeserializer);
assertEquals("key1", outputRecord.key());
assertEquals(Long.valueOf(1L), outputRecord.value());
outputRecord = driver.readOutput(outputTopic, keyDeserializer, valueDeserializer);
assertEquals("key2", outputRecord.key());
assertEquals(Long.valueOf(1L), outputRecord.value());
outputRecord = driver.readOutput(outputTopic, keyDeserializer, valueDeserializer);
assertEquals("key1", outputRecord.key());
assertEquals(Long.valueOf(1L), outputRecord.value()); //this fails, I get 0. If I pull another message, it shows key1 with a count of 1
Here's a repo of the full code:
https://bitbucket.org/nsinha/testtopologywithgroupedtable/src/master/
Stream topology: https://bitbucket.org/nsinha/testtopologywithgroupedtable/src/master/src/main/java/com/nick/kstreams/TopologyWithGroupedTable.java
Test code: https://bitbucket.org/nsinha/testtopologywithgroupedtable/src/master/src/test/java/com/nick/kstreams/TopologyWithGroupedTableTests.java
It's not a bug, but behavior by design (c.f. explanation below).
The difference in behavior is due to KTable state store caching (cf. https://docs.confluent.io/current/streams/developer-guide/memory-mgmt.html). When you run the unit test, the cache is flushed after each record, while in your production run, this is not the case. If you disable caching in your production run, I assume that it behaves the same as in your unit test.
Side remark: ProcessorTopologyTestDriver is an internal class and not part of public API. Thus, there is no compatibility guarantee. You should use the official unit-test packages instead: https://docs.confluent.io/current/streams/developer-guide/test-streams.html
Why do you see two records:
In your code, you are using a KTable#groupBy() and in your specific use case, you don't change the key. However, in general, the key might be changed (depending on the value of the input KTable. Thus, if the input KTable is changed, the downstream aggregation needs to remove/subtract the old key-value pair from the aggregation result, and add the new key-value pair to the aggregation result—in general, the key of the old and new pair are different and thus, it's required to generate two records because the subtraction and addition could happen on different instances as different keys might be hashed differently. Does this make sense?
Thus, for each update of the input KTable, two updates two the result KTable on usually two different key-value pairs need to be computed. For you specific case, in which the key does not change, Kafka Stream does the same thing (there is no check/optimization for this case to "merge" both operations into one if the key is actually the same).
Related
I am currently performing my first steps on kafka streams and I have difficulties understanding how kafka application stores its state.
I would like to print the content of a kstream without having the offset updated, it feels like this is not something that I should want to do but I am struggling to understand why:
def rawPlanningStream(
builder: StreamsBuilder,
topicName: String
): KStream[String, Planning] =
builder.stream(topicName)(Consumed.`with`(Serdes.String, Planning.serde))
def printPlanning(
key: String,
value: Planning
) = {
val logger = LoggerFactory.getLogger("PlanningEventSyncLogger")
logger.warn(s"Planning: $key, $value")
}
def process(
builder: StreamsBuilder,
rawTopic: String
) = {
val raw_planning_stream = PlanningEventSync.rawPlanningStream(
builder,
rawTopic
)
raw_planning_stream.peek((k,v) => printPlanning(k,v))
//Here I would like to perform an operation on raw_planning_stream
//but offset is already "wrong" because of the peek done earlier
}
The first time I start process the content of the topic is printed as intended, if I start it again it no longer prints anything as the offset was updated.
My question is is it possible to perform 'non invasive' operations like a print in order to leave the offset as it is?
(note: I managed to use --reset-offsets --to-earliest from kafka-consumer-groups.sh on my group in order to manually reset the offset, but I would like to be able to programmatically perform operations without changing the offset for my consumer group)
If you cannot set enable.auto.commit=false, then the other option is to set application.id="<some random UUID>" so that every time you run the app, it'll create a new consumer group, starting from auto.offset.reset setting
I'm having a KStream<String,Event> which should be windowedBy and aggregated results in an out of memory:
java.lang.OutOfMemoryError: Java heap space
The KStream DSL is as follows:
TimeWindows timeWindows = TimeWindows.of(Duration.ofDays(1)).advanceBy(Duration.ofMillis(1));
Initializer<History> historyInitializer = History::new;
Aggregator<String, Event, History> historyAggregator = (key, value, aggregate) -> {
aggregate.key = value.uuid;
aggregate.addHistoryEventWindow(value);
return aggregate;
};
KTable<String, History> historyWindowed = eventStreamRaw
.filter((key, value) -> value != null)
.groupByKey(Grouped.with(Serdes.String(), this.eventSerde))
// segment our messages into 1-day windows
.windowedBy(timeWindows)
.aggregate(historyInitializer, historyAggregator, Named.as("name"), Materialized.with(Serdes.String(), this.historySerde))
.suppress(Suppressed.untilWindowCloses(BufferConfig.unbounded()))
.groupBy(
(key, value) -> new KeyValue<String, History>(
value.key + "|+|" + key.window().start() + "|+|" + key.window().end(), value),
Grouped.with(Serdes.String(), this.historySerde))
.aggregate(History::new, (key, value, aggValue) -> value, (key, value, aggValue) -> value,
Materialized.with(Serdes.String(), this.historySerde));
Reading some articles (for example Kafka Streams Window By & RocksDB Tuning) I noticed that I may have to configure the store "Materialized" with a retention of "1 day + 1 Milli".
But trying to add that doesn't work for me:
final Materialized<String, History, WindowStore<Bytes, byte[]>> store = Materialized.<String, History, WindowStore<Bytes, byte[]>>as("eventstore")
.withKeySerde(Serdes.String())
.withValueSerde(this.historySerde)
.withRetention(Duration.ofDays(1).plus(Duration.ofMillis(1)));
KTable<String, History> historyWindowed = eventStreamRaw
...
.aggregate(historyInitializer, historyAggregator, Named.as("name"), store)
The Java compile throw the following error:
The method
aggregate(Initializer<VR>, Aggregator<? super String,? super Event,VR>, Named, Materialized<String,VR,WindowStore<Bytes,byte[]>>)
in the type TimeWindowedKStream<String,Event> is not applicable for the arguments
(Initializer<History>, Aggregator<String,Event,History>, Named, Materialized<String,History,WindowStore<Bytes,byte[]>>)
To be honest, I don't get it. The parameters are correct; the VR type is 'History'.
So, do you know what I'm missing?
The idea of this windowedBy KTable is to have a state which holds all events for one "thing" for one day. Let's say a new alert is produced I want to attach all events of a "thing" for one day to the alert. I would then do a leftJoin from the KStream Alert to the KTable History. Would that the best way to add historical data to a Kafka event? Is there a way to just "look up" the last x days of the KStream Events? I've checked the KStream Alert-KStream Event leftJoin but that would produce an output for every new KStream Event. So, that would be from my point not practicable.
Many thanks for your help. I hope it's just a simple fix one. Highly appreciate!
looking at the following post Kafka Streams App - count and sum aggregate I've imported the wrong "Byte"-class. So, be sure to import the following class "org.apache.kafka.common.utils.Bytes".
But, maybe you have a better idea to enrich a Kafka message from one stream with historical data from another stream related by a (foreign) key.
Thanks guys.
I have a question about how to update JavaRDD values.
I have a JavaRDD<CostedEventMessage> with message objects containing information about to which partition of kafka topic it should be written to.
I'm trying to change the partitionId field of such objects using the following code:
rddToKafka = rddToKafka.map(event -> repartitionEvent(event, numPartitions));
where the repartitionEvent logic is:
costedEventMessage.setPartitionId(1);
return costedEventMessage;
But the modification does not happen.
Could you please advice why and how to correctly modify values in a JavaRDD?
Spark is lazy, so from the code you pasted above it's not clear if you actually performed any action on the JavaRDD (like collect or forEach) and how you came to the conclusion that data was not changed.
For example, if you assumed that by running the following code:
List<CostedEventMessage> messagesLst = ...;
JavaRDD<CostedEventMessage> rddToKafka = javaSparkContext.parallelize(messagesLst);
rddToKafka = rddToKafka.map(event -> repartitionEvent(event, numPartitions));
Each element in messagesLst would have partition set to 1, you are wrong.
That would hold true if you added for example:
messagesLst = rddToKafka.collect();
For more details refer to documentation
I have written spark streaming job which reads data from a s3.
The job has series of mapwithstate followed by maptopair calls, like below:
JavaDStream<String> cdrLines = ssc.textFileStream(cdrInputFile);
JavaDStream<CDR> cdrRecords = cdrLines.map(x -> cdrStreamParser.parse(x));
JavaDStream<CDR> cdrRecordsFiltered = cdrRecords
.filter(t -> t != null);
JavaPairDStream<String, CDR> sTripletStream = cdrRecordsFiltered
.mapToPair(s -> new Tuple2<String, CDR>(s
.gettNumber(), s));
JavaPairDStream<String, Tuple2<CDR, List<StatusCode>>> stateDstream1 = sTripletStream
.mapWithState(
StateSpec.function(hsMappingFunc).initialState(
tripletRDD)).mapToPair(s -> s);
JavaPairDStream<String,Tuple2<CDR,List<StatusCode>>> stateDstream2 = stateDstream1
.mapWithState(StateSpec.function(cfMappingFunc).initialState(cfHistoryRDD))
.mapToPair(s -> s);
JavaPairDStream<String, Tuple2<CDR, List<StatusCode>>> stateDstream3 = stateDstream2
.mapWithState(StateSpec.function(imeiMappingFunc).initialState(imeiRDD))
.mapToPair(s -> s);
I have spark.default.parallelism set to 6. I see first and last maptopair stages are fast enough. The second and third maptopair stages are very slow.
Each of these stages run through 6 tasks. In the second and third maptopair stages, 5 tasks run with 2s. But one task is taking very long time ~3-4min. the shuffle data that task is very high compared to other tasks, which causing bottleneck.
Is there a way we can distrubute the load among all tasks more uniformly?
This is use case for CDR processing. Each CDR event has these fields telno, imei, imsi, callforward, timestamp.
I maintain 3 kinds of info in spark state: 1. last know CDR event (record) for a given telephone number 2. callforward number list for each telephone 3. list of all known imei's.
Three mapwithstate function calls corresponds to below functionality:
step1 : As the CDR events comes in, i need to do some field comparisons with last known CDR event with same telephone number. I maintain latest event for a given telno in the spark state, so that i can do field comparisons as new CDR events comes in.
step2 : For a given telno., i want to check if the callforward number is known number or not. So i need to maintain history of telno. -> list of callforward numbers in the state.
step3 : I need to maintain list of all imei numbers came across, so far in the state, so that for each imei in the CDR event, we can say if its known or new imei.
I am currently storing marshalling libraries for different client versions in a HashMap.
The libs are loaded using the org.reflections API. For simplicity sake I'll just insert a few values here by hand. They are unordered by intent, because I have no influence on in which order the map is initialized on start-up by the reflections API.
The keys (ClientVersion) are enums.
HashMap<ClientVersion, IMarshalLib> MAP = new HashMap<>();
MAP.put(ClientVersion.V100, new MarshalLib100());
MAP.put(ClientVersion.V110, new MarshalLib110());
MAP.put(ClientVersion.V102, new MarshalLib102());
MAP.put(ClientVersion.V101, new MarshalLib101());
MAP.put(ClientVersion.V150, new MarshalLib150());
All and well so far, the problem now is, that there are client versions out there where the marshalling did not change since the previous version.
Let's say, we have a client version ClientVersion.V140. In this particular case I am looking for MarshalLib110, assigned to ClientVersion.V110.
How would I get the desired result (without iterating through all entries and grabbing "the next lower" value each time)?
Thanks in advance!
How would I get the desired result (without iterating through all entries and grabbing "the next lower" value each time)
There is nothing you can do about "iterating through all entries" part: since the map is unordered, there is no way of finding the next smaller item without iterating the entire set of keys.
However, there is something you can do about the "each time" part: if you make a copy of this map into a TreeMap, you would be able to look up the next smaller item by calling the floorEntry method.
Another alternative is to copy the keys into an array on the side, sort the array, and run a binary search each time that you need to look up the next smaller key. With the key in hand, you can look up the entry in your hash map.
I recommend you to use NavigableSet. Look at this example:
HashMap<Integer, String> map = new HashMap<>();
map.put(100, "MarshalLib100");
map.put(110, "MarshalLib110");
map.put(102, "MarshalLib102");
map.put(101, "MarshalLib101");
map.put(150, "MarshalLib150");
NavigableSet<Integer> set = new TreeSet<>(map.keySet());
Integer key = set.lower(150); // ^ -> 110
String val = map.get(key); // ^ -> MarshalLib110
// or
key = set.higher(110);// ^ -> 150
val = map.get(key); // ^ -> MarshalLib150
Update: Using TreeMap to find next lower key is not really correct. Example:
TreeMap<Integer, String> treeMap = new TreeMap<Integer, String>();
treeMap.put(100, "MarshalLib100");
treeMap.put(110, "MarshalLib110");
treeMap.put(102, "MarshalLib102");
treeMap.put(101, "MarshalLib101");
treeMap.put(150, "MarshalLib150");
System.out.println(treeMap.floorKey(102));
System.out.println(treeMap.floorEntry(102));
System.out.println(treeMap.ceilingKey(102));
System.out.println(treeMap.ceilingEntry(102));
Output:
102
102=MarshalLib102
102
102=MarshalLib102