We Keep Coding

MongoDB Morphia distinct

public Collection<OwnerDetail> getAllOwners() {
final MorphiaCursor<Listing> listings = datastore.find(Listing.class)
.iterator(new FindOptions()
.projection().include("ownerDetail"));
Map<String, OwnerDetail> map = new HashMap<>();
while (listings.hasNext()) {
final Listing next = listings.next();
if (next.getOwnerDetail() != null) {
map.put(next.getOwnerDetail().getEmail(), next.getOwnerDetail());
}
}
return map.values();
}
I want to remove java code that distinct values by email and handle it by mongodb morphia. How can I do that?

KafkaConsumer with Multithreading

Created below KafkaConsumer, that will take topicName, partitionNo, beginOffset and endOffset as parameters. But below logic I can execute for one partition at a time because KafkaConsumer is not thread safe. If I want to complete the all 20 partitions it is taking longer time. So how to implement the KafkaConsumer with multi threads so that I can search all partitions at the same time ?
"I have a topic with 20 partitions and has employee data. From UI search screen, I will pass employee number and birth Date, Now I want to search all these 20 partitions to find a particular employees data is there are not. If it is matches then I want put in a separate List and download as file."
public List<String> searchMessages(String topicName, int partitionNo, long beginOffset, long endOffset) {
List<String> filteredMessages = new ArrayList<>();
TopicPartition tp = new TopicPartition("topicName", partitionNo);
Properties clusterOneProps = kafkaConsumerConfig.getConsumerProperties();
KafkaConsumer<String, Object> consumer = new KafkaConsumer<>(clusterOneProps);
try {
consumer.subscribe(Collections.singletonList("topicName"), new ConsumerRebalanceListener() {
#Override
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
// TODO Auto-generated method stub
}
#Override
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
// TODO Auto-generated method stub
consumer.seek(tp, beginOffset);
}
});
Thread.sleep(100);
boolean flag = true;
System.out.println("search started......from offset is "+beginOffset);
while(flag) {
ConsumerRecords<String, Object> crs = consumer.poll(Duration.ofMillis(100L));
for (ConsumerRecord<String, Object> record : crs) {
// search criteria
if(record.value().toString().contains("01111") && record.value().toString().contains("2021-11-06")) {
System.out.println("founddddddddddddddddddddddddddddddddddddddd "+record.offset());
filteredMessages.add(record.value().toString());
}
if (record.offset() == endOffset) {
flag = false;
break;
}
}
}
System.out.println("doneeeeeeeeeeeeeeeee");
}catch(Exception e) {
e.printStackTrace();
}finally {
consumer.close();
}

You need to use the Kafka Parallel Consumer library. Check the library here, and this blog post.
It's possible to 'simulate' parallel consumption with the normal consumar (by having multiple consumers), but you have to hand roll a good amount of code. This blog post explains this approach, but I recommend using the parallel consumer.

Process and check event using kafka-streams during some period

I have a KStream eventsStream, which is get data from a topic "events".
There is two type of events, their keys:
1. {user_id = X, event_id = 1} {..value, include time_event...}
2. {user_id = X, event_id = 2} {..value, include time_event...}
I need to migrate events with event_id = 1 to a topic "results" if during 10 minutes there is not given an event with event_id = 2 by user.
For example,
1. First case: we get data {user_id = 100, event_id = 1} {.. time_event = xxxx ...} and no events during 10 minutes {user_id = 100, event_id = 2} {.. time_event = xxxx + 10 minutes...}, so we'll write it to results-topic
2. Second case: we get data {user_id = 100, event_id = 1} {.. time_event = xxxx ...} and an event during 10 minutes {user_id = 100, event_id = 2} {.. time_event = xxxx + 5 minutes...}, so we'll not write it to results-topic
How does it possible to realise in java code this behavior using kafka-streams?
My code:
public class ResultStream {
public static KafkaStreams newStream() {
Properties properties = Config.getProperties("ResultStream");
Serde<String> stringSerde = Serdes.String();
StreamsBuilder builder = new StreamsBuilder();
StoreBuilder<KeyValueStore<String, String>> store =
Stores.keyValueStoreBuilder(
Stores.inMemoryKeyValueStore("inmemory"),
stringSerde,
stringSerde
);
builder.addStateStore(store);
KStream<String, String> resourceEventStream = builder.stream(EVENTS.topicName(), Consumed.with(stringSerde, stringSerde));
resourceEventStream.print(Printed.toSysOut());
resourceEventStream.process(() -> new CashProcessor("inmemory"), "inmemory");
resourceEventStream.process(() -> new FilterProcessor("inmemory", resourceEventStream), "inmemory");
Topology topology = builder.build();
return new KafkaStreams(topology, properties);
}
}
public class FilterProcessor implements Processor {
private ProcessorContext context;
private String eventStoreName;
private KeyValueStore<String, String> eventStore;
private KStream<String, String> stream;
public FilterProcessor(String eventStoreName, KStream<String, String> stream) {
this.eventStoreName = eventStoreName;
this.stream = stream;
}
#Override
public void init(ProcessorContext processorContext) {
this.context = processorContext;
eventStore = (KeyValueStore) processorContext.getStateStore(eventStoreName);
}
#Override
public void process(Object key, Object value) {
this.context.schedule(Duration.ofMinutes(1), PunctuationType.WALL_CLOCK_TIME, timestamp -> {
System.out.println("Scheduler is working");
stream.filter((k, v) -> {
JsonObject events = new Gson().fromJson(k, JsonObject.class);
if (***condition***) {
return true;
}
return false;
}).to("results");
});
}
#Override
public void close() {
}
}
CashProcessor's role only to put events to local store, and delete record with event_id = 1 by user if there is given an event_id = 2 with the same user.
FilterProcess should filter events using local store every minute. But I can't invoke correctly this processing (as I do it in fact)...
I'm really need help.

Why do you pass KStream into your processor? That is not how the DSL works.
As you "connect" your processors via resourceEventStream.process() already, your FilterProcessor#process(key, value) method will be called for each record in the stream automatically -- however, a KStream#process() is a terminal operation and thus does not allow you to send any data downstream. Instead, you might want to use transform() (that is basically the same as process() plus an output KStream).
To actually forward data downstream in your punctuation, you should use context.forward() using the ProcessorContext that is provided via init() method.

Many-to-One records Kafka Streams

I would like to turn many records into one per message. I tried many things like custom reducing and aggregators, but they all still send one-to-one records back out. For example I would like to convert many strings into just one string. If my stream is messages with the same key, but different values, "the", "sky", "is", "blue", then I would like to outback back one concatenation of them in a new topic "the,sky,is,blue,". What I am instead getting is 4 messages "the,", "the, sky,", "the,sky, is,", "the,sky,is,blue,". When I send a second message to the kafka consumer, it will concatenate on the previous aggregation and I eventually receive this "the,sky,is,blue,the,sky,is,blue,"
I also tried using a custom storebuilder and changing a lot of the settings to see if that would do anything.
Map<String, String> changelogConfig = new HashMap<>();
changelogConfig.put("message.down.conversion.enable", "true");
changelogConfig.put("flush.messages", "0");
changelogConfig.put("flush.ms", "0");
StoreBuilder<KeyValueStore<String, String>> aggStoreSupplier = Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore("AggStore"),
Serdes.String(),
Serdes.String())
.withLoggingEnabled(changelogConfig);
KStream<String, String> results = source // single message get processed and eventually i get these string results I need to concatenate
.groupByKey() // this kgroupedstream has the N records, which was how many were sent in the message
.reduce(new Reducer<String>() {
#Override
public String apply(String aggValue, String value) {
return value + "," + aggValue;
}
}, Materialized.as("AggStore"))
.toStream();
results.to("results", Produced.with(Serdes.String(), Serdes.String()));
final Topology topology = builder.build(); // to describe topology
System.out.println(topology.describe()); // to print description
final KafkaStreams streams = new KafkaStreams(topology, props);
final CountDownLatch latch = new CountDownLatch(1);
// attach shutdown handler to catch control-c
Runtime.getRuntime().addShutdownHook(new Thread("streams-shutdown-hook") {
#Override
public void run() {
streams.close();
latch.countDown();
}
});
try {
streams.cleanUp();
streams.start();
latch.await();
} catch (Throwable e) {
System.exit(1);
}
System.exit(0);

Couchbase SDK 2 : bulk read operations , how to failover to replicas

We are in the process of refactoring a benchmark tool migrating from Couchbase Client 2 to new CouchBase SDK 2.
Previous version has following "bulk get" logic to retrive keys in bulk and if it fails reading from the master, there is a failover to read from the "replicas"
Legacy code :
List<Map.Entry<String, OperationFuture<CASValue<JsonNode>>>> futures = new java.util.ArrayList<>(keys.size());
for (String key : keys) {
futures.add(new AbstractMap.SimpleImmutableEntry<>(key, client.asyncGets(key, transcoder)));
}
Map<String, Long> casValues = new java.util.HashMap<>(keys.size(), 1f);
for (Map.Entry<String, OperationFuture<CASValue<JsonNode>>> e : futures) {
String key = e.getKey();
OperationFuture<CASValue<JsonNode>> future = e.getValue();
try {
CASValue<JsonNode> casVal = future.get();
if (checkStatus(future.getStatus(), errIfNotFound) == OK) {
result.put(key, JsonByteIterator.asMap(casVal.getValue()));
casValues.put(key, casVal.getCas());
} else {
return ERROR;
}
} catch (RuntimeException te) {
if (te.getCause() instanceof CheckedOperationTimeoutException) { ///READ FROM REPLICA
log.warn("Reading from Replica as reading from master has timed out.");
// This is a timeout operation on a read, let's try to read from slave
ReplicaGetFuture<JsonNode> futureReplica = client.asyncGetFromReplica(key, transcoder);
result.put(key, JsonByteIterator.asMap(futureReplica.get()));
} else {
throw te;
}
}
}
Using new Couchbase SDK2
According to the new Couchbase 2 SDK docs ,
http://docs.couchbase.com/developer/java-2.0/documents-bulk.html
I have following logic to retrieve in bulk.But I am not quite sure where to add the failover mechanism to read from "replicas" using
bucket.async().getFromReplica(key, ReplicaMode.ALL);
List<RawJsonDocument> rawDocs = idObs.flatMap((keys)->{
Observable<RawJsonDocument> rawJsonObs = bucket.async().get(key, RawJsonDocument.class);
return rawJsonObs;
}).toList()
.toBlocking()
.single();
How can I implement this "read from replica" failover mechanism with the new RxJava based CouchBase SDK ?

I think I found the anwer :
Observable<RawJsonDocument> rawDocs = idObs.flatMap((key)->{
System.out.println("key "+key);
Observable<RawJsonDocument> rawJsonObs = bucket.async().get(key, RawJsonDocument.class);
return rawJsonObs.onErrorResumeNext(new Func1<Throwable, Observable<RawJsonDocument>>() {
#Override
public Observable<RawJsonDocument> call(Throwable t1) {
if (t1.getCause() instanceof TimeoutException) { //we have a timeout
return bucket.async().getFromReplica(key, ReplicaMode.FIRST, RawJsonDocument.class).first();
}
throw OnErrorThrowable.from(t1);
}
});
});