I'm kinda stuck with a trivial task: whenever I query an external API with reactive spring WebClient or query reactive MongoDBRepository, I'd like to log how many entities got through my flux, eg. to log message like "Found n records in the database.". Eg:
return repository.findAll()
.doOnComplete { log.info("Found total n records!") } // how to get the n?
.filter { it.age > 10 }
.distinct { it.name }
TLDR: How to get a flux size (and perhaps it's contents) when it completes?
You can use ConnectableFlux. In your example:
var all = repository.findAll()
.filter { it.age > 10 }
.distinct { it.name }
.publish()
.autoConnect(2)
all.count()
.subscribe {c -> log.info("Found total {} records!", c)}
return all;
By calling the count(). It should emit a Mono when onComplete is observed.
Here was what I did,
AtomicInteger i = new AtomicInteger();
Flux<UserDetails> stringFlux =
Flux.using(() -> stringStream, Flux::fromStream,
Stream::close)
.doOnNext(s -> i.getAndIncrement())
.log()
.map(UserDetails::createUserDetails);
stringFlux
.subscribe(updateUserDetailsService::updateUserDetails);
log.info("number of records: {}", i);
Related
I want to use Context in my Flux pipe to bypass filtering.
Here's what I have:
public Flux<Bar> realtime(Flux<OHLCIntf> ohlcIntfFlux) {
return Flux.zip(
ohlcIntfFlux,
ohlcIntfFlux.skip(1),
Mono.subscriberContext().map(c -> c.getOrDefault("isRealtime", false))
)
.filter(l ->
l.getT3() ||
(!l.getT2().getEndTimeStr().equals(l.getT1().getEndTimeStr())))
.map(Tuple2::getT1)
.log()
.map(this::
}
which is input to this this:
public void setRealtime(Flux<Bar> input) {
Flux.zip(input, Mono.subscriberContext())
.doOnComplete(() -> {
...
})
.doOnNext(t -> {
...
})
.subscribe()
}
I can tell my code in ... is not failing, I can even access the Context map, but when the first iteration completes, I get:
onContextUpdate(Context1{reactor.onNextError.localStrategy=reactor.core.publisher.OnNextFailureStrategy$ResumeStrategy#35d5ac51})
and subscriber disconnects.
So my question is whether I am using it right and what can be an issue here?
EDIT:
I have tried to repeat() the Mono.subscriberContext() when I'm using value out of it:
return Flux.zip(
ohlcIntfFlux,
ohlcIntfFlux.skip(1),
Mono.subscriberContext()
.map(c -> c.getOrDefault("isRealtime", new AtomicBoolean())).repeat()
)
.filter(l ->
l.getT3().get() ||
(!l.getT2().getEndTime().isEqual(l.getT1().getEndTime())))
.map(Tuple2::getT1)
and set the AtomicBoolean to the context on the subscriber end and just change the value inside this variable reference, when I need the signal on the upstream, but it doesn't change at all:
input
.onErrorContinue((throwable, o) -> throwable.getMessage())
.doOnComplete(() -> {
System.out.println("Number of trades for the strategy: " + tradingRecord.getTradeCount());
// Analysis
System.out.println("Total profit for the strategy: " + new TotalProfitCriterion().calculate(timeSeries, tradingRecord));
})
.doOnNext(this::defaultRealtimeEvaluator)
.subscriberContext(Context.of("isRealtime", isRealtimeAtomic))
.subscribe();
at least with repeat the Flux doesn't disconnect but the value I'm getting out of it is not being updated. No other clues I have.
Spring-webflux: 2.1.3.RELEASE
this works:
input
.onErrorContinue((throwable, o) -> throwable.getMessage())
.doOnComplete(() -> { ... }
.flatMap(bar -> Mono.subscriberContext()
.map(c -> Tuples.of(bar, c)))
.doOnNext(this::defaultRealtimeEvaluator)
.subscriberContext(Context.of("isRealtime", new AtomicBoolean()))
.subscribe();
so the point is to set AtomicBoolean in my case as the cotnext and then extract this variable out of the context if you want to change it's value. the same on the upstream flux.
I need to link two RX Single responses - Retrofit to get an ArrayList with the two responses that return a List
I tried to process both answers with Map, Flatmap, but I have not achieved what I expected
final ArrayList <List<Consent>> listAllConsents = new ArrayList<>();
Single<List<Consent>> responseDspConsent = subscriptionCenterRemoteDataSource.getConsents(Globals.getAuthorizationTokenUser());
Single<List<Consent>> responseDspConsentByApp = subscriptionCenterRemoteDataSource.getConsentsByApp(Globals.getAuthorizationTokenUser());
responseDspConsentByApp.subscribeOn(Schedulers.newThread())
.observeOn(AndroidSchedulers.mainThread());
responseDspConsent.subscribeOn(Schedulers.newThread())
.observeOn(AndroidSchedulers.mainThread())
.flatMap(consentData -> {
List<Consent> consentList = consentData;
listAllConsents.add(consentList);
return responseDspConsentByApp.map(consentDataByApp -> {
List<Consent> consentListByApp = consentDataByApp;
listAllConsents.add(consentListByApp);
return listAllConsents;
});
})
.subscribe(consentData -> {
Log.v("Entramoss", "Valor: " + listAllConsents.get(0).get(0).getTitle());
paintAllConsents(listAllConsents);
});
I need to have all the objects of the two responses in the arrayList so I can paint them later.
You have 2 ways to do this.
1.You can use Observable.concat(Obs 1, Obs 2). The concat operator concatenates the observables and returns a single observable which first emits the items from the first observable then the second one. Source: http://reactivex.io/documentation/operators/concat.html
Single<List<Consent>> responseDspConsent = subscriptionCenterRemoteDataSource
.getConsents(Globals.getAuthorizationTokenUser())
.subscribeOn(Schedulers.newThread())
.observeOn(AndroidSchedulers.mainThread());
Single<List<Consent>> responseDspConsentByApp = subscriptionCenterRemoteDataSource
.getConsentsByApp(Globals.getAuthorizationTokenUser())
.subscribeOn(Schedulers.newThread())
.observeOn(AndroidSchedulers.mainThread());
Observable.concat(responseDspConsent.toObservable(),responseDspConsentByApp.toObservable())
.toList()
.doOnSuccess((list) -> {
paintAllConsents(list);
})
.subscribe();
2.You can use the .concatWith operator which does the same thing as the concat operator but now it concats an observable to another without creating a new observable.
Single<List<Consent>> responseDspConsent = subscriptionCenterRemoteDataSource
.getConsents(Globals.getAuthorizationTokenUser())
.subscribeOn(Schedulers.newThread())
.observeOn(AndroidSchedulers.mainThread());
Single<List<Consent>> responseDspConsentByApp = subscriptionCenterRemoteDataSource
.getConsentsByApp(Globals.getAuthorizationTokenUser())
.subscribeOn(Schedulers.newThread())
.observeOn(AndroidSchedulers.mainThread());
responseDspConsent.concatWith(responseDspConsentByApp)
.toList()
.doOnSuccess((list) -> {
paintAllConsents(list);
})
.subscribe();
if your order do meter i suggest you to do .concat as #ebasha response but if your order doesn't meter i suggest you to use .merge becase is much more faster than .concat because concat subscribe streams one by one and merge subscribe streams immediately
Single<List<Consent>> responseDspConsent = subscriptionCenterRemoteDataSource
.getConsents(Globals.getAuthorizationTokenUser())
.subscribeOn(Schedulers.newThread())
.observeOn(AndroidSchedulers.mainThread());
Single<List<Consent>> responseDspConsentByApp = subscriptionCenterRemoteDataSource
.getConsentsByApp(Globals.getAuthorizationTokenUser())
.subscribeOn(Schedulers.newThread())
.observeOn(AndroidSchedulers.mainThread());
Observable.merge(responseDspConsent.toObservable(),responseDspConsentByApp.toObservable())
.toList()
.doOnSuccess((list) -> {
paintAllConsents(list);
})
.subscribe();
I've got the following piece of code and it works perfectly when my inner data structure of the writerObj is CopyOnWriteArrayList (concurrent one) and it crashes when I use ArrayList.
Here're my questions:
But there's only one thread in RxJava by default, isn't it?
Will the lines (between player { ... }) execute in sync way?
My code looks as follows:
.flatMapCompletable { player -> {
writerObj.write(player); // void write(Player player) adds player to inner data structure using ds.add()
return Completable.complete();
}
}
Depends on how the rest of your chain is coded.
Have a look at the following:
List<String> writerObj = new ArrayList<>();
Observable.range(0, 1000)
.map(i -> Observable.just("hello world"))
.flatMap(obs -> obs
.flatMapCompletable(elem -> {
writerObj.add(elem);
System.out.println(Thread.currentThread().getName() + " executing");
return Completable.complete();
})
.toObservable()
.subscribeOn(Schedulers.io())
)
.blockingSubscribe();
//Size of the list is not always 1000
System.out.println("The size of the list is : " + writerObj.size());
If you execute the code above, you might notice that the size of the List at the end is not always 1000. If you change the Implementation of List to CopyOnWriteArrayList, we get the desired result.
If you want the code in the flatMap to execute sequentially and by one thread at a time, change the flatMap to concatMap.
List<String> writerObj = new ArrayList<>();
Observable.range(0, 1000)
.map(i -> Observable.just("hello world"))
.concatMap(obs -> obs
.flatMapCompletable(elem -> {
writerObj.add(elem);
System.out.println(Thread.currentThread().getName() + " executing");
return Completable.complete();
})
.toObservable()
.subscribeOn(Schedulers.io())
)
.blockingSubscribe();
// Size is always 1000
System.out.println("The size of the list is : " + writerObj.size());
Hope it helps!
I've been trying out rxJava in our code base, mostly looking to add concurrency boosting performance. However, there seems to be overhead/startup cost issues when I use rxJava. In the example below, in "doRx()" it takes ~130ms before getAllElements() is triggered, while in "doOld" it takes 0ms before getAllElements() is triggered. Any explanation to why I'm loosing 130ms initially in doRx()?
This is the logging I do, by using System.currentTimeMillis(). The () is elapsed time from init().
Existing implementation
(0) 2016-10-11T13:34:07.060: OldImpl: init()
(0) 2016-10-11T13:34:07.060: OldImpl: Call getAllElements()
(327) 2016-10-11T13:34:07.387: OldImpl: Received getAllElements()
RX implementation
(0) 2016-10-11T13:34:07.703: RxImpl: init()
(160) 2016-10-11T13:34:07.863: RxImpl: Call
getAllElements()
(392) 2016-10-11T13:34:08.095: RxImpl:
Received getAllElements()
The reasoning behind the code is that I first want to collect all elements, and then run them in parallel (under h2) since that is where we can save time as there are many backend invocations. I've used this blog as guidance for this setup.
public List<Element> doRx() {
List<Element> elements = new ArrayList<>();
Observable
.from(getAllElements())
.flatMap(
s -> Observable
.just(Element::new)
.subscribeOn(Schedulers.io())
.flatMap(
e -> {
List<Element> elements = new ArrayList<>();
for (SubElement se : e.getSubElements()) {
elements.add(se);
}
return Observable.from(elements);
}
)
)
.flatMap(
h1 -> Observable
.just(h1)
.subscribeOn(Schedulers.computation())
.flatMap(
h2 -> {
// Do additional things in parallell on all elements
return Observable
.just(h2);
}
)
)
.toBlocking()
.getIterator()
.forEachRemaining(myList::add);
return elements;
}
public List<Element> doOld() {
List<Element> elements = getAllElements();
for (Element e : elements) {
// Do stuff, same as under h2
}
return elements;
}
If I understand you code correctly, it's equivalent to the following:
public List<Element> doRx() {
return Observable
.from(getAllElements())
.flatMap(element -> Observable
.just(new Element(element))
.subscribeOn(Schedulers.io())
.flatMaplIterable(e -> e.getSubElements())
)
.observeOn(Schedulers.computation())
.doOnNext(element -> {
// Do additional things in parallell on all elements
})
.toList()
.toBlocking()
.single();
}
This has at minimum 2 context switches per element more than the sequential version. How are you doing your timings? X runs, ignore biggest and smallest numbers?
I have a list of users of unknown size. What i want is to query first 30 and update UI. Then i want to query all others by offset with step of 100 until i get last pack of users - should i use takeUntil here?) and when i get - i update UI by adding remaining users (combined with reduce() i belive).
This is my code:
final int INITIAL_OFFSET = 0;
final int INITIAL_LIMIT = 30;
// Loading first 30 users to immediately update UI (better UX)
getServerApi().getAllFriends(userId, "photo_50", INITIAL_OFFSET, INITIAL_LIMIT)
// Loading remaining users 100 by 100 and updating UI after all users been loaded
.flatMap(users -> {
AtomicInteger newOffset = new AtomicInteger(INITIAL_LIMIT);
return Observable.just(users)
.flatMap(users1 -> getServerApi().getAllFriends(userId, "photo_50", newOffset.get(), Config.DEFAULT_FRIEND_REQUEST_COUNT))
.subscribeOn(Schedulers.io())
.observeOn(Schedulers.io())
.collect(() -> new ArrayList<User>(), (b, s) -> {
b.addAll(s);
newOffset.set(newOffset.get() + Config.DEFAULT_FRIEND_REQUEST_COUNT);
})
.repeat()
.takeUntil(friends -> friends.size() == 0);
})
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(users -> getView().appendAllFriends(users),
throwable -> getView().setError(processFail(throwable, ServerApi.Action.GET_ALL_FRIENDS), false));
But seems i do something wrong because onNext is called each time the retrofit call is made.
Answering my own question. Adels answer is good, but i needed to have a single subscription (i'm using Nucleus MVP library) and i wanted to use collect() and takeUntil() instead of while loop (which requires blocking retrofit interface method).
Spent some hours and finally got it:
final int INITIAL_LIMIT = 30;
// Loading first 30 users to immediately update UI (better UX)
getServerApi().getAllFriends(userId, "photo_50", null, INITIAL_LIMIT)
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
// Updating UI 1st time or show error
.doOnNext(users -> getView().appendAllFriends(users))
.doOnError(throwable -> getView().setError(processFail(throwable, ServerApi.Action.GET_ALL_FRIENDS), false))
// Loading remaining users 100 by 100 and updating UI after all users been loaded
.flatMap(users -> {
AtomicInteger newOffset = new AtomicInteger(INITIAL_LIMIT);
ArrayList<User> remainingUsers = new ArrayList<>();
AtomicBoolean hasMore = new AtomicBoolean(true);
return Observable.just(users)
.flatMap(users1 -> getServerApi().getAllFriends(userId, "photo_50", newOffset.get(), Config.DEFAULT_FRIEND_REQUEST_COUNT))
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.collect(() -> remainingUsers, (b, s) -> {
// Needed for takeUntil
hasMore.set(b.addAll(s));
newOffset.set(newOffset.get() + Config.DEFAULT_FRIEND_REQUEST_COUNT);
})
.repeat()
.takeUntil(friends -> !hasMore.get())
// Grab all items emitted by collect()
.last()
// Updating UI last time
.doOnNext(users2 -> getView().appendAllFriends(users2));
})
.subscribe();
Maybe it will be useful for other people which are also using Nucleus.
// cache() will ensure that we load the first pack only once
Observable<Users> firstPack = firstPack().cache();
// this subscription is for updating the UI on the first batch
firstPack
.observeOn(AndroidSchedulers.mainThread())
.subscribe(x -> draw(x), e -> whoops(e));
// this subscription is for collecting all the stuff
// do whatever tricks you need to do with your backend API to get the full list of stuff
firstPack
.flatMap(fp -> rest(fp))
.observeOn(AndroidSchedulers.mainThread())
.subscribe(x -> allUsers(x), e -> whoops(e));
// I would do this in a simple while loop
Observable<List<User>> rest(List<User> firstPack) {
return Observable.create(sub -> {
final List<User> total = firstPack;
try {
while (!sub.isUnsubscribed()) {
final List<User> friends = api.getFriendsBlocking(total.size());
if (friends.isEmpty()) {
sub.onNext(total);
sub.onCompleted();
} else {
total.addAll(friends);
}
}
} catch(IOException e) {
sub.onError(e);
}
})
}