I am trying to convert Flowable<List<TaskEntity>> to Flowable<List<Task>> but something is wrong.
To understand the problem I tried with converting a simpler list and it is working fine. When I try to apply the same logic to my actual problem, it is not working.
This logic is giving me expected output. [No.1 No.2 No.3]
Flowable.fromArray(Arrays.asList(1,2,3))
.flatMapIterable(ids->ids)
.map(s->"No. "+s)
.toList()
.toFlowable()
.subscribe(
t -> Log.d(TAG, "getAllActiveTasks: "+t)
);
This logic is not working . It prints Nothing
mTaskDao.getAllTasks(STATE_ACTIVE)
.flatMapIterable(task -> task)
.map(Task::create)
.toList()
.toFlowable()
.subscribe(
t -> Log.d(TAG, "getAllActiveTasks: "+t)
);
Edit 1
This is how Task.create() looks like.
public static Task create(TaskEntity eTask) {
Task task = new Task(eTask.getTaskId(), eTask.getTaskTitle(), eTask.getTaskStatus());
task.mTaskDescription = eTask.getTaskDescription();
task.mCreatedAt = eTask.getCreatedAt();
task.mTaskDeadline = eTask.getTaskDeadline();
return task;
}
Solution
As mentioned in the comments, toList() can only work if emitting source has finite number of items to emit. Since Flowable from Dao method contains an infinite stream of objects, toList() was not being used correctly by me.
Checkout this comment for the exact way to solve this problem.
https://stackoverflow.com/a/50318832/4989435
toList requires a finite source but getAllTasks is likely infinite, which is unfortunately quite typical from DAOs backed by Android databases. Change the getAllTasks to Single, use take(1), use timeout(), or use flatMap(Observable.fromIterable().map().toList()) instead of flatMapIterable.
I want to receive any updates made to tasks in db.
In this case, you need the latter option:
mTaskDao.getAllTasks(STATE_ACTIVE)
.flatMapSingle(task ->
Observable.fromIterable(task)
.map(Task::create)
.toList()
)
.subscribe(
t -> Log.d(TAG, "getAllActiveTasks: "+t)
);
You should use only map operator to convert TaskEntity to Task. I have created sample. You can check my solution which uses only map operator
Related
Say I want to call a webservice1 and then call webservice2 if the first was successful.
I can do the following (just indicative psuedo code) :-
Mono.just(reqObj)
.flatMap(r -> callServiceA())
.then(() -> callServiceB())
or
Mono.just(reqObj)
.flatMap(r -> callServiceA())
.flatMap(f -> callServiceB())
What is the difference between the two, when using the mono.just() for a single element?
flatMap should be used for non-blocking operations, or in short anything which returns back Mono, Flux.
map should be used when you want to do the transformation of an object /data in fixed time. The operations which are done synchronously.
For ex:
return Mono.just(Person("name", "age:12"))
.map { person ->
EnhancedPerson(person, "id-set", "savedInDb")
}.flatMap { person ->
reactiveMongoDb.save(person)
}
then should be used when you want to ignore element from previous Mono and want the stream to be finised
Here's a detailed explanation from #MuratOzkan
Copy pasting the TL DR answer:
If you care about the result of the previous computation, you can use map(), flatMap() or other map variant. Otherwise, if you just want the previous stream finished, use then().
In your example, looks like your service calls do not require the input of the upstream, then you could use this instead:
Mono.just(reqObj)
.then(() -> callServiceA())
.then(() -> callServiceB())
Sorry for some kind of theoretical question, but I'd like to find a way of quick reading someone else's functional code, building chain of methods use templates.
For example:
Case 1.
When I see use of .peek method or .wireTap from Spring Integration, I primarily expect logging, triggering monitoring or just transitional running external action, for instance:
.peek(params ->
log.info("creating cache configuration {} for key class \"{}\" and value class \"{}\"",
params.getName(), params.getKeyClass(), params.getValueClass()))
or
.peek(p ->
Try.run(() -> cacheService.cacheProfile(p))
.onFailure(ex ->
log.warn("Unable to cache profile: {}", ex.toString())))
or
.wireTap(sf -> sf.handle(msg -> {
monitoring.profileRequestsReceived();
log.trace("Client info request(s) received: {}", msg);
Case 2.
When I see use of .map method or .transform from Spring Integration, I understand that I'm up to get result of someFunction(input), for instance:
.map(e -> GenerateTokenRs.builder().token(e.getKey()).phoneNum(e.getValue()).build())
or
.transform(Message.class, msg -> {
ErrorResponse response = (ErrorResponse) msg.getPayload();
MessageBuilder builder = some tranforming;
return builder.build();
})
Current case.
But I don't have such a common view to .flatMap method.
Would you give me your opinion about this, please?
Add 1:
To Turamarth: I know the difference between .map and .flatMap methods. I actively use both .map, and .flatMap in my code.
But I ask community for theirs experience and coding templates.
It always helps to study the signature/javadoc of the streamish methods to understand them:
The flatMap() operation has the effect of applying a one-to-many transformation to the elements of the stream, and then flattening the resulting elements into a new stream.
So, typical code I expect, or wrote myself:
return someMap.values().stream().flatMap(Collection::stream)
The values of that map are sets, and I want to pull the entries of all these sets into a single stream for further processing here.
In other words: it is about "pulling out things", and getting them into a stream/collection for further processing.
I've found one more use template for .flatMap.
Let's have a look at the following code:
String s = valuesFromDb
.map(v -> v.get(k))
.getOrElse("0");
where Option<Map<String, String>> valuesFromDb = Option.of(.....).
If there's an entry k=null in the map, then we'll get null as a result of code above.
But we'd like to have "0" in this case as well.
So let's add .flatMap:
String s = valuesFromDb
.map(v -> v.get(k))
.flatMap(Option::of)
.getOrElse("0");
Regardless of having null as map's value we will get "0".
I'm pretty new to RX in general, and rxjava in particular, pardon mistakes.
This operation depends on a two async operations.
The first uses a filter function to attempt to get a single entity from a list returned by an async Observable.
The second is an async operation that communicates with a device and produces an Observable of status updates.
I want to take the Single that is created from the filter function, apply that to pairReader(...), and subscribe to its Observable for updates. I can get this to work as shown, but only if I include the take(1) commented, otherwise I get an exception because the chain tries to pull another value from the Single.
Observable<DeviceCredential> getCredentials() {
return deviceCredentialService()
.getCredentials()
.flatMapIterable(event -> event.getData());
}
Single<Organization> getOrgFromCreds(String orgid) {
return getCredentials()
// A device is logically constrained to only have a single cred per org
.map(DeviceCredential::getOrganization)
.filter(org -> org.getId().equals(orgid))
.take(1) // Without this I get an exception
.singleOrError();
}
Function<Organization, Observable<Reader.EnrollmentState>> pairReader(String name) {
return org -> readerService().pair(name, org);
}
getOrgFromCreds(orgid)
.flatMapObservable(pairReader(readerid))
.subscribe(state -> {
switch(state) {
case BEGUN:
LOG.d(TAG, "Pairing begun");
break;
case PAIRED:
LOG.d(TAG, "Pairing success");
callback.success();
break;
case NOTIFIED_SERVER:
LOG.d(TAG, "Pairing server notified");
break;
}},
error -> {
Crashlytics.logException(error);
callback.error(error.getLocalizedMessage());
});
If the source stream emits more than one item, singleOrError() is supposed to emit an error. Doc
For your case, use either first() or firstOrError() instead.
Single<Organization> getOrgFromCreds(String orgid) {
return getCredentials()
.map(DeviceCredential::getOrganization)
.filter(org -> org.getId().equals(orgid))
.firstOrError();
}
If I got you right, you need to make some action using previously retrieved async data. So, you could use .zip() operator.
Here is an example:
Observable.zip(
getOrgFromCreds().toObservable(),
getCredentials(),
(first, second) -> /*create output object here*/
)
.subscribe(
(n) -> /*do onNext*/,
(e) -> /*do onError*/
);
Note, that .zip() operator will wait for both emission from two streams, and then it will create outer emission using the function you provided in "create output object here".
If you don't want to wait for both items - you can use .combineLatest().
The problem here turned out to be that the API was designed in an odd way (and unfortunately has extremely poor documentation). I couldn't figure out why I was getting duplicates, and thought I was using flatMapIterable incorrectly.
What the deviceCredentialService.getCredentials() call actually creates is an observable that emits DataEvent objects which are simple wrappers over a list of results, and with a flag of where the results came from.
The API designer wanted to allow the user to use locally cached data to fill the UI immediately while a longer request to a REST API executes. The DataEvent.from property is an enum that flags the source, either from the local device cache or from the remote API call.
The way I solved this was to simply ignore the results coming from local cache and only emit results from the API:
Observable<DeviceCredential> getCredentials() {
return deviceCredentialService()
.getCredentials()
// Only get creds from network
.filter(e -> e.getFrom() == SyncedDataSourceObservableFactory.From.SOURCE)
.flatMapIterable(e -> e.getData());
}
Single<Organization> getOrgFromCreds(String orgid) {
return getCredentials()
// A device is logically constrained to only have a single cred per org
.map(DeviceCredential::getOrganization)
.filter(org -> org.getId().equals(orgid))
.singleOrError();
}
The plan then is to use memoization to cache entities in a way that gives the implementing app access to cache invalidation. Since the provided interface doesn't allow squelching the API call, there is no way to work only with cache if the app feels its is fresh.
I have an issue while processing a flux that is built from a Stream.generate construct.
The Java stream is fetching some data from a remote source, hence I implemented a custom supplier that has the data fetching logic embedded, and then used it to populate the Stream.
Stream.generate(new SearchSupplier(...))
My idea is to detect an empty list and use the Java9 feature of takeWhile ->
Stream.generate(new SearchSupplier(this, queryBody))
.takeWhile(either -> either.isRight() && either.get().nonEmpty())
(using Vavr's Either construct)
The repositoroy layer flux will then do:
return Flux.fromStream (
this.searchStream(...) //this is where the stream gets generated
)
.map(Either::get)
.flatMap(Flux::fromIterable);
The "service" layer is composed of some transformation steps on the flux, but the method signature is something like Flux<JsonObject> search(...).
Finally, the controller layer has a GetMapping:
#GetMapping(produces = "application/stream+json")
public Flux search(...) {
return searchService.search(...) //this is the Flux<JsonObject> parth
.subscriberContext(...) //stuff I need available during processing
.doOnComplete(() -> log.debug("DONE"));
}
My problem is that the Flux seems to never terminate.
Doing a call from Postman for example just shot the 'Loading...' part in the response section. When I terminate the process from my IDE the results are then flushed to postman and I see what I'm expecting. Also the doOnComplete lambda never gets called
What I noticed is that if I change the source of a Flux:
Flux.fromArray(...) //harcoded array of lists of jsons
the doOnComplete lambda is called and also the http connection closes, and results are displayed in postman.
Any idea of what might be the issue?
Thanks.
You could create the Flux directly using code that looks like this. Note that I'm adding some assumed methods which you would need to implement based on your how your SearchSupplier works:
Flux<SearchResultType> flux = Flux.generate(
() -> new SearchSupplier(this, queryBody),
(supplier, sink) -> {
SearchResultType current = supplier.next();
if (isNotLast(current)) {
sink.next(current);
} else {
sink.complete();
}
return supplier;
},
supplier -> anyCleanupOperations(supplier)
);
I'm trying to get to grips with Spark Streaming but I'm having difficulty. Despite reading the documentation and analysing the examples I wish to do something more than a word count on a text file/stream/Kafka queue which is the only thing we're allowed to understand from the docs.
I wish to listen to an incoming Kafka message stream, group messages by key and then process them. The code below is a simplified version of the process; get the stream of messages from Kafka, reduce by key to group messages by message key then to process them.
JavaPairDStream<String, byte[]> groupByKeyList = kafkaStream.reduceByKey((bytes, bytes2) -> bytes);
groupByKeyList.foreachRDD(rdd -> {
List<MyThing> myThingsList = new ArrayList<>();
MyCalculationCode myCalc = new MyCalculationCode();
rdd.foreachPartition(partition -> {
while (partition.hasNext()) {
Tuple2<String, byte[]> keyAndMessage = partition.next();
MyThing aSingleMyThing = MyThing.parseFrom(keyAndMessage._2); //parse from protobuffer format
myThingsList.add(aSingleMyThing);
}
});
List<MyResult> results = myCalc.doTheStuff(myThingsList);
//other code here to write results to file
});
When debugging I see that in the while (partition.hasNext()) the myThingsList has a different memory address than the declared List<MyThing> myThingsList in the outer forEachRDD.
When List<MyResult> results = myCalc.doTheStuff(myThingsList); is called there are no results because the myThingsList is a different instance of the List.
I'd like a solution to this problem but would prefer a reference to documentation to help me understand why this is not working (as anticipated) and how I can solve it for myself (I don't mean a link to the single page of Spark documentation but also section/paragraph or preferably still, a link to 'JavaDoc' that does not provide Scala examples with non-functional commented code).
The reason you're seeing different list addresses is because Spark doesn't execute foreachPartition locally on the driver, it has to serialize the function and send it over the Executor handling the processing of the partition. You have to remember that although working with the code feels like everything runs in a single location, the calculation is actually distributed.
The first problem I see with you code has to do with your reduceByKey which takes two byte arrays and returns the first, is that really what you want to do? That means you're effectively dropping parts of the data, perhaps you're looking for combineByKey which will allow you to return a JavaPairDStream<String, List<byte[]>.
Regarding parsing of your protobuf, looks to me like you don't want foreachRDD, you need an additional map to parse the data:
kafkaStream
.combineByKey(/* implement logic */)
.flatMap(x -> x._2)
.map(proto -> MyThing.parseFrom(proto))
.map(myThing -> myCalc.doStuff(myThing))
.foreachRDD(/* After all the processing, do stuff with result */)