We Keep Coding

Get Flux size when Flux is complete

I'm kinda stuck with a trivial task: whenever I query an external API with reactive spring WebClient or query reactive MongoDBRepository, I'd like to log how many entities got through my flux, eg. to log message like "Found n records in the database.". Eg:
return repository.findAll()
.doOnComplete { log.info("Found total n records!") } // how to get the n?
.filter { it.age > 10 }
.distinct { it.name }
TLDR: How to get a flux size (and perhaps it's contents) when it completes?

You can use ConnectableFlux. In your example:
var all = repository.findAll()
.filter { it.age > 10 }
.distinct { it.name }
.publish()
.autoConnect(2)
all.count()
.subscribe {c -> log.info("Found total {} records!", c)}
return all;

By calling the count(). It should emit a Mono when onComplete is observed.

Here was what I did,
AtomicInteger i = new AtomicInteger();
Flux<UserDetails> stringFlux =
Flux.using(() -> stringStream, Flux::fromStream,
Stream::close)
.doOnNext(s -> i.getAndIncrement())
.log()
.map(UserDetails::createUserDetails);
stringFlux
.subscribe(updateUserDetailsService::updateUserDetails);
log.info("number of records: {}", i);

How to return Completable from void methods

I've got the following piece of code and it works perfectly when my inner data structure of the writerObj is CopyOnWriteArrayList (concurrent one) and it crashes when I use ArrayList.
Here're my questions:
But there's only one thread in RxJava by default, isn't it?
Will the lines (between player { ... }) execute in sync way?
My code looks as follows:
.flatMapCompletable { player -> {
writerObj.write(player); // void write(Player player) adds player to inner data structure using ds.add()
return Completable.complete();
}
}

Depends on how the rest of your chain is coded.
Have a look at the following:
List<String> writerObj = new ArrayList<>();
Observable.range(0, 1000)
.map(i -> Observable.just("hello world"))
.flatMap(obs -> obs
.flatMapCompletable(elem -> {
writerObj.add(elem);
System.out.println(Thread.currentThread().getName() + " executing");
return Completable.complete();
})
.toObservable()
.subscribeOn(Schedulers.io())
)
.blockingSubscribe();
//Size of the list is not always 1000
System.out.println("The size of the list is : " + writerObj.size());
If you execute the code above, you might notice that the size of the List at the end is not always 1000. If you change the Implementation of List to CopyOnWriteArrayList, we get the desired result.
If you want the code in the flatMap to execute sequentially and by one thread at a time, change the flatMap to concatMap.
List<String> writerObj = new ArrayList<>();
Observable.range(0, 1000)
.map(i -> Observable.just("hello world"))
.concatMap(obs -> obs
.flatMapCompletable(elem -> {
writerObj.add(elem);
System.out.println(Thread.currentThread().getName() + " executing");
return Completable.complete();
})
.toObservable()
.subscribeOn(Schedulers.io())
)
.blockingSubscribe();
// Size is always 1000
System.out.println("The size of the list is : " + writerObj.size());
Hope it helps!

How to reorder Stream of CompletableFutures?

I deal with Streams of CompletableFutures. These take different times to complete. Those taking longer block stream processing while others might already have completed (and I know about Parallel Streams)
Therefore I would like to reorder items in a Stream (e.g. with a buffer) to move completed Futures ahead.
For example, this code blocks stream processing if one getUser call takes long
public static Boolean isValid(User user) { ... }
emails.stream()
// not using ::
// getUser() returns CompletableFuture<User>
.map( e -> getUser(e))
// this line blocks Stream processing
.filter( userF -> isValid( userF.get()) )
.map( f -> f.thenApply(User::getName))
and I would like to have something like
emails.stream()
.map( e -> getUser(e))
// this moves Futures into a bounded buffer
// and puts those finished first
// like CompletionService [1]
// and returns a Stream again
.collect(FutureReorderer.collector())
// this is not the same Stream but
// the one created by FutureReorderer.collector()
.filter( userF -> isValid( userF.get()) )
.map( f -> f.thenApply(User::getName))
[1] For example CompletionService https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ExecutorCompletionService.html returns completed tasks when calling take() and blocks otherwise. But CompletionService does not take futures, would one need to do cs.sumbit( () -> f.get() ) ?
How would I do that?
[Edit]
Changed example to include filter()
Added comment
Added CompletionService link

Having more context would definitely help in tailoring the answer - I have a feeling that problem is somewhere else and can be solved in an easier way.
But if your question is how to somehow keep completed futures at the beginning, there are few options:
Sorting the Stream using a custom Comparator:
.sorted(Comparator.comparing(f -> !f.isDone()))
Keep in mind that isDone returns true not only when a future completes successfully.
Storing futures in a PriorityQueue
PriorityQueue<CompletableFuture<String>> queue
= new PriorityQueue<>(Comparator.comparing(f -> !f.isDone()));
when polling elements, the queue will be returning elements according to their provided ordering.
Here it is in action:
PriorityQueue<CompletableFuture<String>> queue
= new PriorityQueue<>(Comparator.comparing(f -> !f.isDone()));
queue.add(CompletableFuture.supplyAsync(() -> {
try {
Thread.sleep(Integer.MAX_VALUE);
} catch (InterruptedException e) { }
return "42";
}));
queue.add(CompletableFuture.completedFuture("completed"));
queue.poll(); // "completed"
queue.poll(); // still going on
It's important to remember that if you do want to convert PriorityQueue to Stream, you can't do this simply using stream() - this will not preserve the priority order.
This is the right way to go:
Stream.generate(queue::poll).limit(queue.size())

I assume the requirements in OP is execute getUser concurrently and process the result Futures by completion order. Here is solution by ExecutorCompletionService:
final CompletionService<User> ecs = new ExecutorCompletionService<>(executor);
emails.stream().map(e -> ecs.submit(() -> getUser(e).get()))
.collect(Collectors.collectingAndThen(Collectors.toList(), fs -> fs.stream())) // collect the future list for concurrent execution
.map(f -> {
try {
return ecs.take().get();
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
}
})
.filter(u -> isValid(u)).map(User::getName)... //TODO;
Or:
final BlockingQueue<Future<User>> queue = new ArrayBlockingQueue<>(emails.size());
final CompletionService<User> ecs = new ExecutorCompletionService<>(executor, queue);
emails.stream().forEach(e -> ecs.submit(() -> getUser(e).get()));
IntStream.range(0, emails.size())
.mapToObj(i -> {
try {
return queue.poll().get();
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
}
})
.filter(u -> isValid(u)).map(User::getName);
It's simple but not straightforward.

Overhead in rxJava

I've been trying out rxJava in our code base, mostly looking to add concurrency boosting performance. However, there seems to be overhead/startup cost issues when I use rxJava. In the example below, in "doRx()" it takes ~130ms before getAllElements() is triggered, while in "doOld" it takes 0ms before getAllElements() is triggered. Any explanation to why I'm loosing 130ms initially in doRx()?
This is the logging I do, by using System.currentTimeMillis(). The () is elapsed time from init().
Existing implementation
(0) 2016-10-11T13:34:07.060: OldImpl: init()
(0) 2016-10-11T13:34:07.060: OldImpl: Call getAllElements()
(327) 2016-10-11T13:34:07.387: OldImpl: Received getAllElements()
RX implementation
(0) 2016-10-11T13:34:07.703: RxImpl: init()
(160) 2016-10-11T13:34:07.863: RxImpl: Call
getAllElements()
(392) 2016-10-11T13:34:08.095: RxImpl:
Received getAllElements()
The reasoning behind the code is that I first want to collect all elements, and then run them in parallel (under h2) since that is where we can save time as there are many backend invocations. I've used this blog as guidance for this setup.
public List<Element> doRx() {
List<Element> elements = new ArrayList<>();
Observable
.from(getAllElements())
.flatMap(
s -> Observable
.just(Element::new)
.subscribeOn(Schedulers.io())
.flatMap(
e -> {
List<Element> elements = new ArrayList<>();
for (SubElement se : e.getSubElements()) {
elements.add(se);
}
return Observable.from(elements);
}
)
)
.flatMap(
h1 -> Observable
.just(h1)
.subscribeOn(Schedulers.computation())
.flatMap(
h2 -> {
// Do additional things in parallell on all elements
return Observable
.just(h2);
}
)
)
.toBlocking()
.getIterator()
.forEachRemaining(myList::add);
return elements;
}
public List<Element> doOld() {
List<Element> elements = getAllElements();
for (Element e : elements) {
// Do stuff, same as under h2
}
return elements;
}

If I understand you code correctly, it's equivalent to the following:
public List<Element> doRx() {
return Observable
.from(getAllElements())
.flatMap(element -> Observable
.just(new Element(element))
.subscribeOn(Schedulers.io())
.flatMaplIterable(e -> e.getSubElements())
)
.observeOn(Schedulers.computation())
.doOnNext(element -> {
// Do additional things in parallell on all elements
})
.toList()
.toBlocking()
.single();
}
This has at minimum 2 context switches per element more than the sequential version. How are you doing your timings? X runs, ignore biggest and smallest numbers?

Reduce many async Observable<List> into one Observable<List>

So here is what I'm trying to do:
for each event i want to call some Service.getList() 10 times, then merge the 10 lists into one.
Now I tried this two ways, they both work in UTs but fail in real app (I'm guessing I am not doing correctly the reduce operation given async http calls in real app).
For both cases I cannot see the logs put after reduce(), neither onNext() nor onError(), so I am guessing the reduce() operation does not complete.
Try #1:
public Observable<List<Event>> getEventsForLocation(Location location) {
List<Observable<List<Event>>> obs = new ArrayList<>();
for (Venue v : location.getVenues()) {
obs.add(getEventsForVenue(v)); //does one http call, returns Observable<List<Event>>
}
return Observable.concat(Observable.from(obs))
.reduce((List<Event>) new ArrayList<Event>(), (events, events2) -> {
events.addAll(events2);
return events;
})
.doOnNext(events -> Log.d("reduce ", events.toString()))
.doOnError(throwable -> Log.e("reduce error", throwable.toString()));}
Try #2:
public Observable<List<Event>> getEventsForLocation(Location location) {
return Observable
.from(location.getVenues())
.flatMap(venue -> getEventsForVenue(venue)) //does one http call, returns Observable<List<Event>>
.reduce((List<Event>) new ArrayList<Event>(), (events, events2) -> {
events.addAll(events2);
return events;
})
.doOnNext(events -> Log.d("service", "total events " + events.toString()))
.doOnError(t -> Log.e("service", "total events error2 " + t.toString()));}
And the UT which passes for both approaches:
#Test
public void getEventsForLocation() {
Location loc = new Location("test", newArrayList(new Venue("v1", "url1"),new Venue("v2", "url2")));
when(httpGateway.downloadWebPage(Mockito.anyString())).thenReturn(
Observable.just(readResource("eventsForVenue1.html")),
Observable.just(readResource("eventsForVenue2.html"))
);
TestSubscriber<List<Event>> probe = new TestSubscriber<>();
service.getEventsForLocation(loc).subscribe(probe);
probe.assertNoErrors();
//assert the next event containts contents of all lists
List<Event> events = probe.getOnNextEvents().get(0);
//first list
Assert.assertEquals("Unexpected title", "event1", events.get(0).getName());
Assert.assertEquals("Unexpected artist", "artist1", events.get(0).getArtist());
//second list
Assert.assertEquals("Unexpected title", "event2", events.get(1).getName());
Assert.assertEquals("Unexpected artist", "artist2", events.get(1).getArtist());
}
UPDATE
Here is the more complete code, with the schedulers.
Observable
.just(loc)
.subscribeOn(AndroidSchedulers.mainThread())
.observeOn(Schedulers.io())
.flatMap(location -> service.getEventsForLocation(location))
.observeOn(AndroidSchedulers.mainThread())
.subscribe(getObserver();

If you don't care about the final order in the list, you can just use from + flatMap + flatMapIterable + toList:
Observable.from(location.getVenues())
.flatMap(venue -> getEventsForVenue(venue))
.flatMapIterable(list -> list)
.toList();
If order matters and you want to execute getEventsForVenue in "parallel", you can replace flatMap with concatMapEager:
Observable.from(location.getVenues())
.concatMapEager(venue -> getEventsForVenue(venue))
.concatMapIterable(list -> list)
.toList();

You can use Collect and it will do the trick. But in this scenarios where I´m merging severals items, normally I much rather use Scan. Which for every new item it will give you the last processed item. So you can append every item with the previous items.
Take a look the marble diagram in case is suitable for your case
http://reactivex.io/documentation/operators/scan.html
Try to start with this simple example and then try to apply it into your code
/**
* apply this function for every item against the previous emitted item from the source.
* Emitted:
0
1
3
6
10
15
*/
#Test
public void scanObservable() {
Integer[] numbers = {0, 1, 2, 3, 4, 5};
Observable.from(numbers)
.scan((lastItemEmitted, newItem) -> (lastItemEmitted + newItem))
.subscribe(System.out::println);
}

I finally found one solution, using zip() but i don't like it.
The problem should be able to be resolved with a combination of flatMap/reduce
public Observable<List<Event>> getEventsForLocation(Location location) {
List<Observable<List<Event>>> venues = new ArrayList<>();
for (Venue v : location.getVenues()) {
venues.add(Observable.just(v).flatMap(venue -> getEventsForVenue(venue)));
}
return Observable.zip(venues, new FuncN<List<Event>>() {
#Override
public List<Event> call(Object... args) {
List<Event> allEvents = new ArrayList<Event>();
for (Object o : args) {
List<Event> le = (List<Event>) o;
allEvents.addAll(le);
}
return allEvents;
}
});