So here is what I'm trying to do:
for each event i want to call some Service.getList() 10 times, then merge the 10 lists into one.
Now I tried this two ways, they both work in UTs but fail in real app (I'm guessing I am not doing correctly the reduce operation given async http calls in real app).
For both cases I cannot see the logs put after reduce(), neither onNext() nor onError(), so I am guessing the reduce() operation does not complete.
Try #1:
public Observable<List<Event>> getEventsForLocation(Location location) {
List<Observable<List<Event>>> obs = new ArrayList<>();
for (Venue v : location.getVenues()) {
obs.add(getEventsForVenue(v)); //does one http call, returns Observable<List<Event>>
}
return Observable.concat(Observable.from(obs))
.reduce((List<Event>) new ArrayList<Event>(), (events, events2) -> {
events.addAll(events2);
return events;
})
.doOnNext(events -> Log.d("reduce ", events.toString()))
.doOnError(throwable -> Log.e("reduce error", throwable.toString()));}
Try #2:
public Observable<List<Event>> getEventsForLocation(Location location) {
return Observable
.from(location.getVenues())
.flatMap(venue -> getEventsForVenue(venue)) //does one http call, returns Observable<List<Event>>
.reduce((List<Event>) new ArrayList<Event>(), (events, events2) -> {
events.addAll(events2);
return events;
})
.doOnNext(events -> Log.d("service", "total events " + events.toString()))
.doOnError(t -> Log.e("service", "total events error2 " + t.toString()));}
And the UT which passes for both approaches:
#Test
public void getEventsForLocation() {
Location loc = new Location("test", newArrayList(new Venue("v1", "url1"),new Venue("v2", "url2")));
when(httpGateway.downloadWebPage(Mockito.anyString())).thenReturn(
Observable.just(readResource("eventsForVenue1.html")),
Observable.just(readResource("eventsForVenue2.html"))
);
TestSubscriber<List<Event>> probe = new TestSubscriber<>();
service.getEventsForLocation(loc).subscribe(probe);
probe.assertNoErrors();
//assert the next event containts contents of all lists
List<Event> events = probe.getOnNextEvents().get(0);
//first list
Assert.assertEquals("Unexpected title", "event1", events.get(0).getName());
Assert.assertEquals("Unexpected artist", "artist1", events.get(0).getArtist());
//second list
Assert.assertEquals("Unexpected title", "event2", events.get(1).getName());
Assert.assertEquals("Unexpected artist", "artist2", events.get(1).getArtist());
}
UPDATE
Here is the more complete code, with the schedulers.
Observable
.just(loc)
.subscribeOn(AndroidSchedulers.mainThread())
.observeOn(Schedulers.io())
.flatMap(location -> service.getEventsForLocation(location))
.observeOn(AndroidSchedulers.mainThread())
.subscribe(getObserver();
If you don't care about the final order in the list, you can just use from + flatMap + flatMapIterable + toList:
Observable.from(location.getVenues())
.flatMap(venue -> getEventsForVenue(venue))
.flatMapIterable(list -> list)
.toList();
If order matters and you want to execute getEventsForVenue in "parallel", you can replace flatMap with concatMapEager:
Observable.from(location.getVenues())
.concatMapEager(venue -> getEventsForVenue(venue))
.concatMapIterable(list -> list)
.toList();
You can use Collect and it will do the trick. But in this scenarios where I´m merging severals items, normally I much rather use Scan. Which for every new item it will give you the last processed item. So you can append every item with the previous items.
Take a look the marble diagram in case is suitable for your case
http://reactivex.io/documentation/operators/scan.html
Try to start with this simple example and then try to apply it into your code
/**
* apply this function for every item against the previous emitted item from the source.
* Emitted:
0
1
3
6
10
15
*/
#Test
public void scanObservable() {
Integer[] numbers = {0, 1, 2, 3, 4, 5};
Observable.from(numbers)
.scan((lastItemEmitted, newItem) -> (lastItemEmitted + newItem))
.subscribe(System.out::println);
}
I finally found one solution, using zip() but i don't like it.
The problem should be able to be resolved with a combination of flatMap/reduce
public Observable<List<Event>> getEventsForLocation(Location location) {
List<Observable<List<Event>>> venues = new ArrayList<>();
for (Venue v : location.getVenues()) {
venues.add(Observable.just(v).flatMap(venue -> getEventsForVenue(venue)));
}
return Observable.zip(venues, new FuncN<List<Event>>() {
#Override
public List<Event> call(Object... args) {
List<Event> allEvents = new ArrayList<Event>();
for (Object o : args) {
List<Event> le = (List<Event>) o;
allEvents.addAll(le);
}
return allEvents;
}
});
Related
I have a method that returns Mono<Output>:
interface Processor {
Mono<Output> process(Input input);
}
And I want to execute this processor method for a collection:
List<Input> inputs = // get inputs
Processor processor = // get processor
List<Mono<Output>> outputs = inputs.stream().map(supplier::supply).collect(toList());
But instead of a List<Mono<Output>> I want to get Mono<List<Output>> that will contain aggregated results.
I tried reduce, but the final result looks very clumsy:
Mono<List<Output>> result = inputs.stream().map(processor::process)
.reduce(Mono.just(new ArrayList<>()),
(monoListOfOutput, monoOfOutput) ->
monoListOfOutput.flatMap(list -> monoOfOutput.map(output -> {
list.add(output);
return list;
})),
(left, right) ->
left.flatMap(leftList -> right.map(rightList -> {
leftList.addAll(rightList);
return leftList;
})));
Can I achieve this with less code?
If you don't have to create stream for any reason, you could create Flux from your inputs, map it and collect list
Flux.fromIterable(inputs).flatMap(processor::process).collectList();
// first merge all the `Mono`s:
List<Mono<Output>> outputs = ...
Flux<Output> merged = Flux.empty();
for (Mono<Output> out : outputs) {
merged = merged.mergeWith(out);
}
// then collect them
return merged.collectList();
or (inspired by Alexander's answer)
Flux.fromIterable(outputs).flatMap(x -> x).collectList();
I've got the following piece of code and it works perfectly when my inner data structure of the writerObj is CopyOnWriteArrayList (concurrent one) and it crashes when I use ArrayList.
Here're my questions:
But there's only one thread in RxJava by default, isn't it?
Will the lines (between player { ... }) execute in sync way?
My code looks as follows:
.flatMapCompletable { player -> {
writerObj.write(player); // void write(Player player) adds player to inner data structure using ds.add()
return Completable.complete();
}
}
Depends on how the rest of your chain is coded.
Have a look at the following:
List<String> writerObj = new ArrayList<>();
Observable.range(0, 1000)
.map(i -> Observable.just("hello world"))
.flatMap(obs -> obs
.flatMapCompletable(elem -> {
writerObj.add(elem);
System.out.println(Thread.currentThread().getName() + " executing");
return Completable.complete();
})
.toObservable()
.subscribeOn(Schedulers.io())
)
.blockingSubscribe();
//Size of the list is not always 1000
System.out.println("The size of the list is : " + writerObj.size());
If you execute the code above, you might notice that the size of the List at the end is not always 1000. If you change the Implementation of List to CopyOnWriteArrayList, we get the desired result.
If you want the code in the flatMap to execute sequentially and by one thread at a time, change the flatMap to concatMap.
List<String> writerObj = new ArrayList<>();
Observable.range(0, 1000)
.map(i -> Observable.just("hello world"))
.concatMap(obs -> obs
.flatMapCompletable(elem -> {
writerObj.add(elem);
System.out.println(Thread.currentThread().getName() + " executing");
return Completable.complete();
})
.toObservable()
.subscribeOn(Schedulers.io())
)
.blockingSubscribe();
// Size is always 1000
System.out.println("The size of the list is : " + writerObj.size());
Hope it helps!
I have a list of users of unknown size. What i want is to query first 30 and update UI. Then i want to query all others by offset with step of 100 until i get last pack of users - should i use takeUntil here?) and when i get - i update UI by adding remaining users (combined with reduce() i belive).
This is my code:
final int INITIAL_OFFSET = 0;
final int INITIAL_LIMIT = 30;
// Loading first 30 users to immediately update UI (better UX)
getServerApi().getAllFriends(userId, "photo_50", INITIAL_OFFSET, INITIAL_LIMIT)
// Loading remaining users 100 by 100 and updating UI after all users been loaded
.flatMap(users -> {
AtomicInteger newOffset = new AtomicInteger(INITIAL_LIMIT);
return Observable.just(users)
.flatMap(users1 -> getServerApi().getAllFriends(userId, "photo_50", newOffset.get(), Config.DEFAULT_FRIEND_REQUEST_COUNT))
.subscribeOn(Schedulers.io())
.observeOn(Schedulers.io())
.collect(() -> new ArrayList<User>(), (b, s) -> {
b.addAll(s);
newOffset.set(newOffset.get() + Config.DEFAULT_FRIEND_REQUEST_COUNT);
})
.repeat()
.takeUntil(friends -> friends.size() == 0);
})
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(users -> getView().appendAllFriends(users),
throwable -> getView().setError(processFail(throwable, ServerApi.Action.GET_ALL_FRIENDS), false));
But seems i do something wrong because onNext is called each time the retrofit call is made.
Answering my own question. Adels answer is good, but i needed to have a single subscription (i'm using Nucleus MVP library) and i wanted to use collect() and takeUntil() instead of while loop (which requires blocking retrofit interface method).
Spent some hours and finally got it:
final int INITIAL_LIMIT = 30;
// Loading first 30 users to immediately update UI (better UX)
getServerApi().getAllFriends(userId, "photo_50", null, INITIAL_LIMIT)
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
// Updating UI 1st time or show error
.doOnNext(users -> getView().appendAllFriends(users))
.doOnError(throwable -> getView().setError(processFail(throwable, ServerApi.Action.GET_ALL_FRIENDS), false))
// Loading remaining users 100 by 100 and updating UI after all users been loaded
.flatMap(users -> {
AtomicInteger newOffset = new AtomicInteger(INITIAL_LIMIT);
ArrayList<User> remainingUsers = new ArrayList<>();
AtomicBoolean hasMore = new AtomicBoolean(true);
return Observable.just(users)
.flatMap(users1 -> getServerApi().getAllFriends(userId, "photo_50", newOffset.get(), Config.DEFAULT_FRIEND_REQUEST_COUNT))
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.collect(() -> remainingUsers, (b, s) -> {
// Needed for takeUntil
hasMore.set(b.addAll(s));
newOffset.set(newOffset.get() + Config.DEFAULT_FRIEND_REQUEST_COUNT);
})
.repeat()
.takeUntil(friends -> !hasMore.get())
// Grab all items emitted by collect()
.last()
// Updating UI last time
.doOnNext(users2 -> getView().appendAllFriends(users2));
})
.subscribe();
Maybe it will be useful for other people which are also using Nucleus.
// cache() will ensure that we load the first pack only once
Observable<Users> firstPack = firstPack().cache();
// this subscription is for updating the UI on the first batch
firstPack
.observeOn(AndroidSchedulers.mainThread())
.subscribe(x -> draw(x), e -> whoops(e));
// this subscription is for collecting all the stuff
// do whatever tricks you need to do with your backend API to get the full list of stuff
firstPack
.flatMap(fp -> rest(fp))
.observeOn(AndroidSchedulers.mainThread())
.subscribe(x -> allUsers(x), e -> whoops(e));
// I would do this in a simple while loop
Observable<List<User>> rest(List<User> firstPack) {
return Observable.create(sub -> {
final List<User> total = firstPack;
try {
while (!sub.isUnsubscribed()) {
final List<User> friends = api.getFriendsBlocking(total.size());
if (friends.isEmpty()) {
sub.onNext(total);
sub.onCompleted();
} else {
total.addAll(friends);
}
}
} catch(IOException e) {
sub.onError(e);
}
})
}
I'd like to know if there is a good way of reusing a common stream operation that varies in the end for different outputs.
The example bellow is exactly what I'm trying to compact into a one-step operation:
public static DepartmentInfo extractDepartmentInfo(BaselinePolicy resource) throws ResourceProcessorError {
Function<Exception, Exception> rpe = e -> new ResourceProcessorError(e.getMessage());
List<String> parents =
Objects.requireNonNull(
Exceptions.trying(
() -> Arrays.asList(Exceptions.dangerous(resource::getParentIds).expecting(CMException.class).throwing(rpe))
.stream()
.map(cId -> Exceptions.dangerous(cId, resource.getCMServer()::getPolicy).expecting(CMException.class).throwing(rpe))
.filter(policy -> PagePolicy.class.isAssignableFrom(policy.getClass()))
.map(PagePolicy.class::cast)
.filter(page -> Exceptions.dangerous(page,
p -> Boolean.valueOf(p.getComponentNotNull(ComponentConstants.POLOPOLY_CLIENT,
ComponentConstants.IS_HOME_DEPARTMENT,
Boolean.FALSE.toString())).booleanValue())
.expecting(CMException.class).throwing(rpe))
.map(page -> Exceptions.dangerous(page, p -> p.getExternalId().getExternalId()).expecting(CMException.class).throwing(rpe)), ResourceProcessorError.class)
.collect(Collectors.toList()));
String externalId = parents.get(parents.size()-1).toString();
List<String> list =
Objects.requireNonNull(
Exceptions.trying(
() -> Arrays.asList(Exceptions.dangerous(resource::getParentIds).expecting(CMException.class).throwing(rpe))
.stream()
.map(cId -> Exceptions.dangerous(cId, resource.getCMServer()::getPolicy).expecting(CMException.class).throwing(rpe))
.filter(policy -> PagePolicy.class.isAssignableFrom(policy.getClass()))
.map(PagePolicy.class::cast)
.map(page ->
Exceptions.dangerous(page,
p -> p.getChildPolicy(PATH_SEGMENT) != null &&
StringUtils.hasLength(SingleValued.class.cast(p.getChildPolicy(PATH_SEGMENT)).getValue())?
SingleValued.class.cast(p.getChildPolicy(PATH_SEGMENT)).getValue(): p.getName()).expecting(CMException.class).throwing(rpe))
.filter(val -> val != null && !val.isEmpty()), ResourceProcessorError.class)
.collect(Collectors.toList()));
if(list.size() > 3) {
list = list.subList(list.size() - 3, list.size()-1);
}
switch(list.size()) {
case 0: {
throw new ResourceProcessorError("br.com.oesp.XMLRender.error.noProduct");
}
case 1: {
return DepartmentInfo.withProduct(list.get(0), externalId);
}
case 2: {
return DepartmentInfo.withProduct(list.get(0), externalId).withDepartment(list.get(1));
}
default: {
return DepartmentInfo.withProduct(list.get(0), externalId).withDepartment(list.get(1)).withSubDepartment(list.get(2));
}
}
}
Notice that the first step is repeated for both:
List<String> parents =
Objects.requireNonNull(
Exceptions.trying(
() -> Arrays.asList(Exceptions.dangerous(resource::getParentIds).expecting(CMException.class).throwing(rpe))
.stream()
.map(cId -> Exceptions.dangerous(cId, resource.getCMServer()::getPolicy).expecting(CMException.class).throwing(rpe))
.filter(policy -> PagePolicy.class.isAssignableFrom(policy.getClass()))
.map(PagePolicy.class::cast)
It's not only a problem for reading but specially because I'm redoing a heavy operation twice, meanwhile in a more imperative way I'd do it once.
There are two things you're trying to do:
avoid the redundant work of creating the input array
avoid the redundant code of the map/filter/map
The first is easy:
List<Id> list = Arrays.asList(Exceptions.dangerous(resource::getParentIds)
.expecting(CMException.class)
.throwing(rpe));
Now you can pull streams from this source twice without rematerializing it.
The next bit is simply a Function from List to Stream:
Function<List<Id>, Stream<Something>> asStream =
list -> list.stream().map(...).filter(...).map(...);
Now, just start your stream with this:
asStream.apply(list).moreStuff().moreStuff()
I have the following code:
public void foo() {
Long[] gData = new Long[] { 1L, 2L };
rx.Observable.from(gData)
.concatMap(data -> {
rx.Observable<GmObject> depositObs1 = depositToUserBalance(data, 1);
rx.Observable<GmObject> depositObs2 = depositToUserBalance(data, 2);
return rx.Observable.zip(depositObs1, depositObs2, (depositObj1, depositObj2) -> {
depositObj1.putNumber("seat_index", data);
depositObj2.putNumber("seat_index", data);
return rx.Observable.merge(
rx.Observable.just(depositObj1),
rx.Observable.just(depositObj2));
})
})
.reduce(new ArrayList<Long>(), (payoutArr, payoutObj) -> {
int seatIndex = ((GmObject) payoutObj).getNumber("seat_index").intValue();
long payout = ((GmObject) payoutObj).getNumber("payout").longValue();
payoutArr.add(seatIndex, payout);
return payoutArr;
})
.subscribe(results -> {
System.out.println(results);
});
}
This code uses .zip to emits to observables, and then it adds a 'seat_index' property and calls .merge in order to use .reduce so eventually all results would be aggregated into an ArrayList.
There is a problem with this code: When .reduce processes its input it gets it as Observable and not as GmObject ...What function can 'extract' the GmObject from its Observable wrap?
Does it make sense to use rxJava this way? or there is a better technique?
Thanks!
the zip operator take as third argument a lambda. this lambda is a 2 args function which return an object that result of the composition of args. And not a Observable of the result of the composition (but, of course, the object can be an Observable, but it's not what you want in your case).
So after your zip call, you'll have an Observable<Observable<GmObject>> but you expect an Observable<GmObject>.
I don't think that the zip operator is the operator you're looking for.
public void foo() {
Long[] gData = new Long[] { 1L, 2L };
rx.Observable.from(gData)
.concatMap(data -> {
rx.Observable<GmObject> depositObs1 = depositToUserBalance(data, 1).doOnNext(obj -> obj.putNumber("seat_index", data));
rx.Observable<GmObject> depositObs2 = depositToUserBalance(data, 2).doOnNext(obj -> obj.putNumber("seat_index", data));
return rx.Observable.merge(depositObs1, depositObs2);
})
.reduce(new ArrayList<Long>(), (payoutArr, payoutObj) -> {
int seatIndex = ((GmObject) payoutObj).getNumber("seat_index").intValue();
long payout = ((GmObject) payoutObj).getNumber("payout").longValue();
payoutArr.add(seatIndex, payout);
return payoutArr;
})
.subscribe(results -> System.out.println(results));
}