We Keep Coding

Java Kafka stream processing tumbling windows

I am processing a stream of messages using Java KStream APIs. Currently I my code emits to get the output every 5 mins, but I want to get the out put at the top of the 5 min interval ( i.e 17:10, 17:15 etc)
Currently the interval depends on the time the program started. If the program starts at 17:08 the data gets collected at 17:13, 17:18, 17:23 etc intervals.
Is there a way that I can schedule so the data gets emitted at 5 min intervals which are multiples of 5?
class WindowedTransformerExample {
public static void main(String[] args) {
final StreamsBuilder builder = new StreamsBuilder();
final String stateStoreName = "stateStore";
final StoreBuilder<KeyValueStore<String, String>> keyValueStoreBuilder =
Stores.keyValueStoreBuilder(Stores.inMemoryKeyValueStore(stateStoreName),
Serdes.String(),
Serdes.String());
builder.addStateStore(keyValueStoreBuilder);
builder.<String, String>stream("topic").transform(new
WindowedTransformer(stateStoreName), stateStoreName)
.filter((k, v) -> k != null && v != null)
// Here's where you do something with records emitted after 10 minutes
.foreach((k, v)-> System.out.println());
}
static final class WindowedTransformer implements TransformerSupplier<String, String, KeyValue<String, String>> {
private final String storeName;
public WindowedTransformer(final String storeName) {
this.storeName = storeName;
}
#Override
public Transformer<String, String, KeyValue<String, String>> get() {
return new Transformer<String, String, KeyValue<String, String>>() {
private KeyValueStore<String, String> keyValueStore;
private ProcessorContext processorContext;
#Override
public void init(final ProcessorContext context) {
processorContext = context;
keyValueStore = (KeyValueStore<String, String>) context.getStateStore(storeName);
// could change this to PunctuationType.STREAM_TIME if needed
context.schedule(Duration.ofMinutes(5), PunctuationType.WALL_CLOCK_TIME, (ts) -> {
try(final KeyValueIterator<String, String> iterator = keyValueStore.all()) {
while (iterator.hasNext()) {
final KeyValue<String, String> keyValue = iterator.next();
processorContext.forward(keyValue.key, keyValue.value);
}
}
});
}
#Override
public KeyValue<String, String> transform(String key, String value) {
if (key != null) {
keyValueStore.put(key, value);
}
return null;
}
#Override
public void close() {
}
};
}
}
}

getting error on desirializing java object from Kafka

I started to learn Kafka, and now,
I'm on sending/receiving serialized/desirialised java class.
My question is about: what have I missed in my config, so I can't deserialize the object from Kafka
here is my class:
public class Foo {
private String item;
private int quantity;
private Double price;
public Foo(String item, int quantity, final double price) {
this.item = item;
this.quantity = quantity;
this.price = price;
}
public String getItem() { return item; }
public int getQuantity() { return quantity; }
public Double getPrice() { return price; }
public void setQuantity(int quantity) { this.quantity = quantity; }
public void setPrice(double price) { this.price = price; }
#Override
public String toString() {
return "item=" + item + ", quantity=" + quantity + ", price=" + price;
}
}
my Properties in main class:
producerPropsObject.put(ProducerConfig.CLIENT_ID_CONFIG,
AppConfigs.applicationProducerSerializedObject);
producerPropsObject.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
AppConfigs.bootstrapServers);
producerPropsObject.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
StringSerializer.class.getName());
producerPropsObject.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
FooSerializer.class.getName());
producerPropsObject.put("topic", AppConfigs.topicNameForSerializedObject);
consumerPropsObject.put(ConsumerConfig.GROUP_ID_CONFIG, AppConfigs.applicationProducerSerializedObject);
consumerPropsObject.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, AppConfigs.bootstrapServers);
consumerPropsObject.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
consumerPropsObject.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,FooDeserializer.class.getName());
consumerPropsObject.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 300000);
consumerPropsObject.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
consumerPropsObject.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
consumerPropsObject.put("topic", AppConfigs.topicNameForSerializedObject);
following are serializer/deserializer implementations:
public class FooSerializer implements org.apache.kafka.common.serialization.Serializer {
public void configure(Map map, boolean b) { }
public byte[] serialize(String s, Object o) {
try {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(o);
oos.close();
byte[] b = baos.toByteArray();
return b;
} catch (IOException e) { return new byte[0]; }
}
public void close() { }
}
public class FooDeserializer implements org.apache.kafka.common.serialization.Deserializer {
#Override
public void close() { }
#Override
public Foo deserialize(String arg0, byte[] arg1) {
//Option #1:
//ObjectMapper mapper = new ObjectMapper();
//Option #2:
JsonFactory factory = new JsonFactory();
factory.enable(JsonParser.Feature.ALLOW_SINGLE_QUOTES);
ObjectMapper mapper = new ObjectMapper(factory);
Foo fooObj = null;
try {
//Option #1:
//fooObj = mapper.readValue(arg1, Foo.class); // BREAKS HERE!!!
//Option #2:
fooObj = mapper.reader().forType(Foo.class).readValue(arg1); // BREAKS HERE!!!
}
catch (Exception e) { e.printStackTrace(); }
return fooObj;
}
}
and finally the way I'm trying to produce and consume my Foo from main:
seems, like it works fine, cause I see in kafka-topic my Key && Value later on
public void produceObjectToKafka(final Properties producerProps) {
final String[] ar = new String[]{"Matrix", "Naked Gun", "5th Element", "Die Hard", "Gone with a wind"};
KafkaProducer<String, byte[]> producer = new KafkaProducer<>(producerProps);
final Foo j = new Foo(ar[getAnInt(4)], getAnInt(10), getAnDouble());
producer.send(new ProducerRecord<>(producerProps.getProperty("topic"), j.getItem(), j.toString().getBytes()));
producer.flush();
producer.close();
}
however, while my Consumer is catching the output:
public void consumeFooFromKafka(final Properties consumerProps) {
final Consumer<String, Foo> myConsumer = new KafkaConsumer<>(consumerProps);
final Thread separateThread = new Thread(() -> {
try {
myConsumer.subscribe(Collections.singletonList(consumerProps.getProperty("topic")));
while (continueToRunFlag) {
final StringBuilder sb = new StringBuilder();
final ConsumerRecords<String, Foo> consumerRecords = myConsumer.poll(Duration.ofMillis(10));
if (consumerRecords.count() > 0) {
for (ConsumerRecord<String, Foo> cRec : consumerRecords) {
sb.append( cRec.key() ).append("<<").append(cRec.value().getItem() + ",").append(cRec.value().getQuantity() + ",").append(cRec.value().getPrice()).append("|");
}
}
if (sb.length() > 0) { System.out.println(sb.toString()); }
}
}
finally {
myConsumer.close();
}
});
separateThread.start();
}
=======================================
so, actually by running "consumeFooFromKafka" , when it trigger "FooDeserializer" ...... there, I always have same error(regardless of Option #1, or Option #2):
exception:
Method threw 'com.fasterxml.jackson.core.JsonParseException' exception.
detailedMessage:
Unexpected character ('¬' (code 172)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or
'false')
will be very appresiated for help.......
Thank you in advance,
Steve

If you want to deserialize from json, than u need to serialize it to json, use jackson in ur serializer also, and everything should be fine
public class FooSerializer implements org.apache.kafka.common.serialization.Serializer {
public void configure(Map map, boolean b) { }
public byte[] serialize(String s, Object o) {
try {
ObjectMapper om = new ObjectMapper();//objectmapper from jackson
byte[] b = om.writeValueAsString(o).getBytes();
return b;
} catch (IOException e) { return new byte[0]; }
}
public void close() { }
}

I don't know why you're using a bytearray outputstream, but trying to read JSON in the deserializer, but that explains the error. You could even test that without using Kafka at all by invoking the serialize/deserialize methods directly
In the link provided, the serializer uses objectMapper.writeValueAsString, which returns JSON text, and not the Java specific outputstream. If you wanted to consume and produce data between different programming languages (as is often the case in most companies), you'd want to avoid such specific serialization formats
Note: Confluent provides Avro, Protobuf, and JSON serializers for Kafka, so you shouldn't need to write your own if you want to use one of those formats

How to parse JSON without POJO

This is my json:
{
"taggedEntries": {
"user/f8cf24ef-4bd0-846f-a11a781ce81a/tag/TEST": [
"20HQtzLrqRe8xz8tybYf2aS087xS92Zi_1719877dbea:8e4:6eb16f2b"
],
"user/f8cf24ef-4bd0-846f-a11a781ce81a/tag/global.unsaved": [
"360ebRQH+hi4mCv/YibdkukUtv175h4JfU23PTw2O8M=_171888f776b:69cb:f8e58482",
"20HQtzL4prqRe8xz8tybYf2aS087xS92Zi+zuo=_171987c5e49:8ed:6eb16f2b",
"20HQtzL4rqRe8xz8tybYf2aS087xS92Zi+zuo=_171987d5d3d:8ee:6eb16f2b",
"20HQtzL4q9uNe8xz8tybYf2aS087xS92Zi+zuo=_1719854c09a:8bd:6eb16f2b"
],
"user/f8cf24ef-4bd0-846f-a11a781ce81a/tag/286f1f46-911c-4bc2-4b028b0d7ed0": [
"v1I7ZIsSoGZxr80NFebazQf2J2QviXCcdot3TOU=_1717e68bf58:fcd1:75b51987",
"360ebRQH+hibdkukUtv175h4JfU23PTw2O8M=_171888f776b:69cb:f8e58482",
"20HQtzL4q9uqRe8xz8tybYf2aS087xS92Zi+zuo=_171983b3399:8b7:6eb16f2b"
]
}
}
How can I parse it?
I would like to get the following structure
TaggedEntries<String, Array<String>>
where:
1 argument is tag name, 2 argument is ids tag
Tag name is a dynamic string. I can't get as static element of json.
Any idea?
Thanks for help :)

Parse to Map<String, Map<String, List<String>>> then call get("taggedEntries") to get the Map<String, List<String>> value you're looking for.

In your TaggedEntries class you should have HashMap>. I think your argument without POJO is hard or messy I think. You'll play with alot of JsonObject / JsonArray.

My solution:
val obj = JsonParser().parse(jsonStr).asJsonObject.getAsJsonObject("taggedEntries")
val entries = obj.entrySet() //will return members of your object
val data = mutableMapOf<String,ArrayList<String>>()
for ((key) in entries) {
val ids: JsonArray = obj.getAsJsonArray(key)
val listIdTags = arrayListOf<String>()
ids.forEach{
listIdTags.add(it.toString())
}
data.put(key,listIdTags)
}
// Print Data
data.forEach(){ key, value ->
println("Key: $key")
println("Value:")
value.forEach{
println( it)
}
}
Maybe there is a better solution but I haven't found it. If you have a solution I'd like to hear it.

When you need some special things, you may need some special handiwork to be done and this isn't a rocket science, really :) For example, using a tiny JSON parser
https://github.com/anatolygudkov/green-jelly:
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
public class ParseMyJson {
private static final String myJson = "{" +
"\"taggedEntries\": {" +
"\"user/f8cf24ef-4bd0-846f-a11a781ce81a/tag/TEST\": [" +
"\"20HQtzLrqRe8xz8tybYf2aS087xS92Zi_1719877dbea:8e4:6eb16f2b\"]," +
"\"user/f8cf24ef-4bd0-846f-a11a781ce81a/tag/global.unsaved\": [" +
"\"360ebRQH+hi4mCv/YibdkukUtv175h4JfU23PTw2O8M=_171888f776b:69cb:f8e58482\"," +
"\"20HQtzL4prqRe8xz8tybYf2aS087xS92Zi+zuo=_171987c5e49:8ed:6eb16f2b\"," +
"\"20HQtzL4rqRe8xz8tybYf2aS087xS92Zi+zuo=_171987d5d3d:8ee:6eb16f2b\"," +
"\"20HQtzL4q9uNe8xz8tybYf2aS087xS92Zi+zuo=_1719854c09a:8bd:6eb16f2b\"" +
"]," +
"\"user/f8cf24ef-4bd0-846f-a11a781ce81a/tag/286f1f46-911c-4bc2-4b028b0d7ed0\": [" +
"\"v1I7ZIsSoGZxr80NFebazQf2J2QviXCcdot3TOU=_1717e68bf58:fcd1:75b51987\"," +
"\"360ebRQH+hibdkukUtv175h4JfU23PTw2O8M=_171888f776b:69cb:f8e58482\"," +
"\"20HQtzL4q9uqRe8xz8tybYf2aS087xS92Zi+zuo=_171983b3399:8b7:6eb16f2b\"" +
"]" +
"}" +
"}";
public static void main(String[] args) {
final TaggedEntries<String, String> result = new TaggedEntries<>();
final JsonParser parser = new JsonParser();
parser.setListener(
new JsonParserListenerAdaptor() {
private String lastPropertyName = null;
private boolean inArray = false;
public boolean onObjectMember(final CharSequence name) {
lastPropertyName = name.toString();
return true;
}
#Override
public boolean onArrayStarted() {
inArray = true;
result.onTag(lastPropertyName);
return true;
}
#Override
public boolean onArrayEnded() {
inArray = false;
return true;
}
#Override
public boolean onStringValue(final CharSequence data) {
if (inArray) {
result.onValue(data.toString());
}
return true;
}
}
);
parser.parse(myJson);
parser.eoj();
System.out.println(result);
}
public static class TaggedEntries<T, V> {
private final Map<T, List<V>> allEntries = new HashMap();
private List<V> currentTagValues;
public TaggedEntries() {
}
public void onTag(final T tag) {
currentTagValues = new ArrayList<>();
allEntries.put(tag, currentTagValues);
}
public void onValue(final V value) {
if (currentTagValues == null) {
throw new IllegalStateException("onTag must be called before");
}
currentTagValues.add(value);
}
public List<V> tagValues(final T tag) {
return allEntries.get(tag);
}
public Set<T> tags() {
return allEntries.keySet();
}
#Override
public String toString() {
final StringBuilder result = new StringBuilder("TaggedEntries{\n");
allEntries.forEach((t, vs) -> {
result.append('\t').append(t).append('\n');
vs.forEach(v -> result.append("\t\t").append(v).append('\n'));
});
result.append('}');
return result.toString();
}
}
}

RxJava: Split Rx Flowable into multiple streams

I would like to perform some operations on stream, and then split stream into two streams, and then process them separately.
Example to show problem:
Flowable<SuccessfulObject> stream = Flowable.fromArray(
new SuccessfulObject(true, 0),
new SuccessfulObject(false, 1),
new SuccessfulObject(true, 2));
stream = stream.doOnEach(System.out::println);
Flowable<SuccessfulObject> successful = stream.filter(SuccessfulObject::isSuccess);
Flowable<SuccessfulObject> failed = stream.filter(SuccessfulObject::isFail);
successful.doOnEach(successfulObject -> {/*handle success*/}).subscribe();
failed.doOnEach(successfulObject -> {/*handle fail*/}).subscribe();
Class:
class SuccessfulObject {
private boolean success;
private int id;
public SuccessfulObject(boolean success, int id) {
this.success = success;
this.id = id;
}
public boolean isSuccess() {
return success;
}
public boolean isFail() {
return !success;
}
public void setSuccess(boolean success) {
this.success = success;
}
#Override
public String toString() {
return "SuccessfulObject{" +
"id=" + id +
'}';
}
}
But this code prints all elements twice whereas I would like to perform all operations before splitting only once.
Output:
OnNextNotification[SuccessfulObject{id=0}]
OnNextNotification[SuccessfulObject{id=1}]
OnNextNotification[SuccessfulObject{id=2}] OnCompleteNotification
OnNextNotification[SuccessfulObject{id=0}]
OnNextNotification[SuccessfulObject{id=1}]
OnNextNotification[SuccessfulObject{id=2}] OnCompleteNotification
How can I process the stream to receive this behaviour?

Use publish to share a subscription to the source:
Flowable<Integer> source = Flowable.range(1, 5);
ConnectableFlowable<Integer> cf = source.publish();
cf.filter(v -> v % 2 == 0).subscribe(v -> System.out.println("Even: " + v));
cf.filter(v -> v % 2 != 0).subscribe(v -> System.out.println("Odd: " + v));
cf.connect();

Generic tuple de-serialization in Jackson

It so happens that I need to support in Java JSON data coming from external data sources. There is one common pattern. It's an array containing fixed number of elements of certain different types. We call it tuple. Here is my example of de-serialization for 3-element tuple with particular expected types of elements using FasterXML Jackson:
public class TupleTest {
public static void main(String[] args) throws Exception {
String person = "{\"name\":\"qqq\",\"age\":35,\"address\":\"nowhere\",\"phone\":\"(555)555-5555\",\"email\":\"super#server.com\"}";
String jsonText = "[[" + person + ",[" + person + "," + person + "],{\"index1\":" + person + ",\"index2\":" + person + "}]]";
ObjectMapper om = new ObjectMapper().registerModule(new TupleModule());
List<FixedTuple3> data = om.readValue(jsonText, new TypeReference<List<FixedTuple3>>() {});
System.out.println("Deserialization result: " + data);
System.out.println("Serialization result: " + om.writeValueAsString(data));
}
}
class Person {
public String name;
public Integer age;
public String address;
public String phone;
public String email;
#Override
public String toString() {
return "Person{name=" + name + ", age=" + age + ", address=" + address
+ ", phone=" + phone + ", email=" + email + "}";
}
}
class FixedTuple3 {
public Person e1;
public List<Person> e2;
public Map<String, Person> e3;
#Override
public String toString() {
return "Tuple[" + e1 + ", " + e2 + ", " + e3 + "]";
}
}
class TupleModule extends SimpleModule {
public TupleModule() {
super(TupleModule.class.getSimpleName(), new Version(1, 0, 0, null, null, null));
setSerializers(new SimpleSerializers() {
#Override
public JsonSerializer<?> findSerializer(SerializationConfig config,
JavaType type, BeanDescription beanDesc) {
if (isTuple(type.getRawClass()))
return new TupleSerializer();
return super.findSerializer(config, type, beanDesc);
}
});
setDeserializers(new SimpleDeserializers() {
#Override
public JsonDeserializer<?> findBeanDeserializer(JavaType type,
DeserializationConfig config, BeanDescription beanDesc) throws JsonMappingException {
Class<?> rawClass = type.getRawClass();
if (isTuple(rawClass))
return new TupleDeserializer(rawClass);
return super.findBeanDeserializer(type, config, beanDesc);
}
});
}
private boolean isTuple(Class<?> rawClass) {
return rawClass.equals(FixedTuple3.class);
}
public static class TupleSerializer extends JsonSerializer<Object> {
public void serialize(Object value, JsonGenerator jgen, SerializerProvider provider) throws IOException, JsonProcessingException {
try {
jgen.writeStartArray();
for (int i = 0; i < 3; i++) {
Field f = value.getClass().getField("e" + (i + 1));
Object res = f.get(value);
jgen.getCodec().writeValue(jgen, res);
}
jgen.writeEndArray();
} catch (Exception ex) {
throw new IllegalStateException(ex);
}
}
}
public static class TupleDeserializer extends JsonDeserializer<Object> {
private Class<?> retClass;
public TupleDeserializer(Class<?> retClass) {
this.retClass = retClass;
}
public Object deserialize(JsonParser p, DeserializationContext ctx) throws IOException, JsonProcessingException {
try {
Object res = retClass.newInstance();
if (!p.isExpectedStartArrayToken()) {
throw new JsonMappingException("Tuple array is expected but found " + p.getCurrentToken());
}
JsonToken t = p.nextToken();
for (int i = 0; i < 3; i++) {
final Field f = res.getClass().getField("e" + (i + 1));
TypeReference<?> tr = new TypeReference<Object>() {
#Override
public Type getType() {
return f.getGenericType();
}
};
Object val = p.getCodec().readValue(p, tr);
f.set(res, val);
}
t = p.nextToken();
if (t != JsonToken.END_ARRAY)
throw new IOException("Unexpected ending token in tuple deserializer: " + t.name());
return res;
} catch (IOException ex) {
throw ex;
} catch (Exception ex) {
throw new IllegalStateException(ex);
}
}
}
}
But this approach means I have to make new class every time I face new type configuration in tuple of certain size. So I wonder if there is any way to define deserializer dealing with generic typing. So that it will be enough to have one tuple class per tuple size. For instance my generic tuple of size 3 could be defined like:
class Tuple3 <T1, T2, T3> {
public T1 e1;
public T2 e2;
public T3 e3;
#Override
public String toString() {
return "Tuple[" + e1 + ", " + e2 + ", " + e3 + "]";
}
}
And usage of it would look like:
List<Tuple3<Person, List<Person>, Map<String, Person>>> data =
om.readValue(jsonText,
new TypeReference<List<Tuple3<Person, List<Person>, Map<String, Person>>>>() {});
Is it something doable or not?

Ok. So... there may be a simpler way to do "tuple"-style. You can actually annotate POJOs to be serialized as arrays:
#JsonFormat(shape=JsonFormat.Shape.ARRAY)
#JsonPropertyOrder({ "name", "age" }) // or use "alphabetic"
public class POJO {
public String name;
public int age;
}
and if so, they'll get written as arrays, read from arrays.
But if you do what to handle custom generic types, you probably need to get type parameters resolved. This can be done using TypeFactory, method findTypeParameters(...). While this may seem superfluous, it is needed for general case if you sub-type (if not, JavaType actually has accessors for direct type parameters).

Yes, you must use Reflection to get ALL FIELDS, not to get the known fields by name.