We Keep Coding

How to get combined effects of whenComplete and thenCompose?

I am trying to come up with a CompletableFuture with the combined effects of whenComplete and thenCompose, specifically:
Returns a CompletionStage instead of just a result, similar to thenCompose.
Executes even when previous stage completes exceptionally, similar to whenComplete, and does not stop the exception from propagating.
This post is close to what I'm trying to achieve but I don't want to use handle which hides the exception. Thanks for any ideas.

I don't believe CompletionStage or CompletableFuture provides any single method for this. However, combining handle with thenCompose should do what you want, if I understand your requirements correctly.
A handle stage is executed whether the parent stage has completed normally or exceptionally and gives you access to the result or error, respectively. From this stage you could return another CompletionStage which would either be completed normally or exceptionally depending on what arguments the handle stage receives.
handle((T result, Throwable error) -> {
if (error != null) {
return CompletableFuture.<T>failedStage(error);
} else {
return processResult(result); // returns CompletionStage<T>
}
});
Now you have a CompletionStage<CompletionStage<T>>. Now we execute a flat map operation by invoking thenCompose:
thenCompose(Function.identity());
Which gives us a CompletionStage<T>. This CompletionStage<T> will be whatever instance was returned by handle. If that instance was a failed stage then the exception is still propagated; otherwise, the result is passed to whatever stage is dependent on the thenCompose stage and processing continues normally.
You can see this with the following example:
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionStage;
public class Main {
public static void main(String[] args) {
methodThatReturnsCompletionStage()
.handle((result, error) -> {
if (error != null) {
return CompletableFuture.<String>failedStage(error);
} else {
return processResult(result);
}
})
.thenCompose(future -> {
System.out.println("#thenCompose invoked");
return future; // Identity function
})
.thenApply(result -> {
System.out.println("#thenApply invoked");
return result; // Identity function (exists to show intermediary stage)
})
.whenComplete((result, error) -> {
System.out.println("#whenComplete invoked");
if (error != null) {
error.printStackTrace(System.out);
} else {
System.out.println(result);
};
});
}
private static CompletionStage<String> methodThatReturnsCompletionStage() {
return CompletableFuture.completedStage("Hello");
// return CompletableFuture.failedStage(new RuntimeException("OOPS"));
}
private static CompletionStage<String> processResult(String result) {
return CompletableFuture.completedFuture(result + ", World!");
}
}
This will result in each stage being invoked and an output of Hello, World!. But if you switch methodThatReturnsCompletionStage() to return the failed stage instead then thenApply is skipped (because the future has failed) and the exception is given to whenComplete (which, like handle, is invoked for both normal or exceptional completion).
Note: Everything above uses the CompletionStage interface directly but using CompletableFuture works just as well (and may be preferable).

Equivalent of VertX CompositeFuture in RxJava

The VertX example for when you need to query multiple asynchronous resources and use them all in a single operation is:
Future<HttpServer> httpServerFuture = Future.future();
httpServer.listen(httpServerFuture.completer());
Future<NetServer> netServerFuture = Future.future();
netServer.listen(netServerFuture.completer());
CompositeFuture.all(httpServerFuture, netServerFuture).setHandler(ar -> {
if (ar.succeeded()) {
// All servers started
} else {
// At least one server failed
}
});
We need to query two different databases and then use the results in business logic, but the flow is equivalent.
What's the VertX/RxJava equivalent?
Currently people are doing this by nesting a new .flatMap() call every time they need a new variable. I'm left feeling there must be a better way...
We don't actually need the queries to be concurrent but we need to cache both results and pass them to the business logic at the same time some how.

there are many ways to do this, but i've tried to pick an approach that tacks closely to your sample:
#Override
public void start(Future<Void> startFuture) throws Exception {
final HttpServer httpServer = vertx.createHttpServer();
final Completable initializeHttpServer = httpServer.rxListen().toCompletable();
final NetServer netServer = vertx.createNetServer();
final Completable initializeNetServer = netServer.rxListen().toCompletable();
initializeHttpServer.andThen(initializeNetServer)
.subscribe(
() -> { /* All servers started */ },
error -> { /* At least one server failed */ }
);
}
the rxListen() invocations are converted into Completable instances, which are then run serially upon subscription.
the subscriber's onComplete callback will be invoked when both servers are done binding to their respective ports, or...
the onError callback will be invoked if an exception occurs
(also, fwiw, "nesting" flatMap operations for something as trivial as this shouldn't be necessary. "chaining" such operations, however, would be idiomatic usage).
hope that helps!
--UPDATE--
having read the question more carefully, i now see that you were actually asking about how to handle the results of two discrete asynchronous operations.
an alternative to flatMap'ing your way to combining the results would be to use the zip operator, like so:
#Override
public void start(Future<Void> startFuture) throws Exception {
final Single<String> dbQuery1 = Single.fromCallable(() -> { return "db-query-result-1"; });
final Single<String> dbQuery2 = Single.fromCallable(() -> { return "db-query-result-2"; });
Single.zip(dbQuery1, dbQuery2, (result1, result2) -> {
// handle the results from both db queries
// (with Pair being a standard tuple-like class)
return new Pair(result1, result2);
})
.subscribe(
pair -> {
// handle the results
},
error -> {
// something went wrong
}
);
}
per the docs, zip allows you to specify a series of reactive types (Single, Observable, etc) along with a function to transform all the results at once, with the central idea being that it will not emit anything until all the sources have emitted once (or more, depending on the reactive type).

Can a ListenableFuture chain handle inner ExecutionException?

I am provided with an api (fnGrpc) that performs a gRPC call and returns a ListenableFuture that resolves to some value v (the implementation of which is fixed and unmodifiable).
I want to provide a helper function (fnHelper) that:
does some transformational processing on the gRPC result and itself returns a ListenableFuture that resolves to the transformed value t1.
handles failure of the gRPC call, and returns some other value t2 instead of having fnHelper's caller see an ExecutionException.
I can solve (1) by using Futures.transform():
package myHelper;
ListenableFuture<T> fnHelper() {
return Futures.transform(fnGrpc(), new Function<V, T>() {
#Override
public T apply(V v) {
T t1 = f(v);
return t1;
}
});
}
and the caller:
package myApp;
// ...
try {
T t = fnHelper().get();
} catch (ExecutionException | InterruptedException ex) {
// ...
}
How can I achieve (2) whilst still having fnHelper return a ListenableFuture and remain non-blocking?
I could have fnHelper itself create an additional thread within which I would call .get() on fnGrpc, but is there another way that avoids this additional thread?

I'm not an expert in Guava, but it seems you can do that using the same Futures utility class, in particular the method catchingAsync, where you can pass a function that returns a ListenableFuture with the fallback value (t2):
ListenableFuture<Integer> faultTolerantFuture = Futures.catchingAsync(originalFuture,
Exception.class, x -> immediateFuture(t2), executor);
You should then be able to chain this with the transform method, which does the transformation:
ListenableFuture<T> fnHelper() {
return Futures.catching(Futures.transform(fnGrpc(), new Function<V, T>() {
#Override
public T apply(V v) {
T t1 = f(v);
return t1;
}
}),
Exception.class, x -> immediateFuture(t2));
}
Note: In the last snippet, I used catching instead of catchingAsync to be consistent with the code in your question, and I didn't specify an executor. You probably need to use the methods with the Async suffix for non-blocking processing.

Creating stream of elements from collection

I am using Junit 5 Dynamic tests.
My intention is to create a stream of elements from the collection to pass it on to test in JUnit5.
However with this code, I am able to run only 1000 records. How do I make this work seamlessly non-blocking.
MongoCollection<Document> collection = mydatabase.getCollection("mycoll");
final List<Document> cache = Collections.synchronizedList(new ArrayList<Document>());
FindIterable<Document> f = collection.find().batchSize(1000);
f.batchCursor(new SingleResultCallback<AsyncBatchCursor<Document>>() {
#Override
public void onResult(AsyncBatchCursor<Document> t, Throwable thrwbl) {
t.next(new SingleResultCallback<List<Document>>() {
#Override
public void onResult(List<Document> t, Throwable thrwbl) {
if (thrwbl != null) {
th.set(thrwbl);
}
cache.addAll(t);
latch.countDown();;
}
});
}
});
latch.await();
return cache.stream().map(batch->process(batch));
Updated Code
#ParameterizedTest
#MethodSource("setUp")
void cacheTest(MyClazz myclass) throws Exception {
assertTrue(doTest(myclass));
}
public static MongoClient getMongoClient() {
// get client here
}
private static Stream<MyClazz> setUp() throws Exception {
MongoDatabase mydatabase = getMongoClient().getDatabase("test");
List<Throwable> failures = new ArrayList<>();
CountDownLatch latch = new CountDownLatch(1);
List<MyClazz> list = Collections.synchronizedList(new ArrayList<>());
mydatabase.getCollection("testcollection").find()
.toObservable().subscribe(
document -> {
list.add(process(document));
},
throwable -> {
failures.add(throwable);
},
() -> {
latch.countDown();
});
latch.await();
return list.stream();
}
public boolean doTest(MyClazz myclass) {
// processing goes here
}
public MyClazz process(Document doc) {
// doc gets converted to MyClazz
return MyClazz;
}
Even now, I see that all the data is loaded after which the unit testing happens.
I think this is because of latch.await(). However, if I remove that, there is a chance that no test cases are run as the db could possibly be loading collection.
My use case is : I have million records in mongo and am running sort of integration test case with them. It wouldn't be feasible to load all of them in memory and hence I am attempting the streaming solution.

I don't think I fully understand your use case but given that your question is tagged with java and mongo-asyc-driver this requirement is certainly achievable:
create a stream of elements from the collection to pass it on to test ... make this work seamlessly non-blocking
The following code:
Uses the MongoDB RxJava driver to query a collection
Creates a Rx Observable from that collection
Subscribes to that Observable
Records exceptions
Marks completion
CountDownLatch latch = new CountDownLatch(1);
List<Throwable> failures = new ArrayList<>();
collection.find()
.toObservable().subscribe(
// on next, this is invoked for each document returned by your find call
document -> {
// presumably you'll want to do something here to meet this requirement: "pass it on to test in JUnit5"
System.out.println(document);
},
/// on error
throwable -> {
failures.add(throwable);
},
// on completion
() -> {
latch.countDown();
});
// await the completion event
latch.await();
Notes:
This requires use of the MongoDB RxJava driver (i.e. classes in the com.mongodb.rx.client namespace ... the org.mongodb::mongodb-driver-rx Maven artifact)
In your question you are invoking collection.find().batchSize() which clearly indicates that you are not currently using the Rx driver (since batchSize cannot be a Rx friendly concept :)
The above code is verified with v1.4.0 of the MongoDB RxJava driver and v1.1.10 of io.reactive::rxjava
Update 1: based on the change to your question (which follows my original answer), you have asked: " I see that all the data is loaded after which the unit testing happens. I think this is because of latch.await()"? I think you are pop[ulating a list from the observable stream and only after the observable is exhausted do you start invoking doTest(). This approach involves (1) streaming results from MongoDB; (2) storing those results in-memory; (3) running doTest() for each stored result. If you really want to stream all-the-way then you should call doTest() from within your observable's subscription. For example:
mydatabase.getCollection("testcollection").find()
.toObservable().subscribe(
document -> {
doTest(process(document));
},
throwable -> {
failures.add(throwable);
},
() -> {
latch.countDown();
});
latch.await();
The above code will invoke doTest() as it receives each document from MongoDB and when the entire observable is exhausted the latch will be decremented and your code will complete.

RxJava: How to get all results AND errors from an Observable

I'm working on a project that involves Hystrix, and I decided to use RxJava. Now, forget Hystrix for the rest of this because I believe the main problem is with my complete screwing up of writing the Observable code correctly.
Need:
I need a way to return an observable that represents a number of observables, each running a user task. I want that Observable to be able to return all results from the tasks, even errors.
Problem:
Observable streams die on errors. If I have three tasks and the second task throws an exception, I never receive the third task even if it would have succeeded.
My Code:
public <T> Observable<T> observeManagedAsync(String groupName,List<EspTask<T>> tasks) {
return Observable
.from(tasks)
.flatMap(task -> {
try {
return new MyCommand(task.getTaskId(),groupName,task).toObservable().subscribeOn(this.schedulerFactory.get(groupName));
} catch(Exception ex) {
return Observable.error(ex);
}
});
}
Given that MyCommand is a class that extends HystrixObservableCommand, it returns an Observable and so shouldn't figure in on the problems I'm seeing.
Attempt 1:
Used Observable.flatMap as above
Good: Each Command is scheduled on it's own thread and the tasks run asynchronously.
Bad: On first Command exception, Observable completes having emitted previous successful results and emitting the Exception. Any in-flight Commands are ignored.
Attempt 2:
Used Observable.concatMapDelayError instead of flatMap
Bad: For some reason, tasks run synchronously. Why??
Good: I get all the successful results.
~Good: OnError gets a Composite exception with a list of the exceptions thrown.
Any help will be greatly appreciated and probably result in me being very embarrassed for not having thought of it myself.
Additional Code
This test succeeds with Observable.flatMap, but fails when using Observable.concatMapDelayError because the tasks do not run asynchronously:
java.lang.AssertionError: Execution time ran over the 350ms limit: 608
#Test
public void shouldRunManagedAsyncTasksConcurrently() throws Exception {
Observable<String> testObserver = executor.observeManagedAsync("asyncThreadPool",getTimedTasks());
TestSubscriber<String> testSubscriber = new TestSubscriber<>();
long startTime = System.currentTimeMillis();
testObserver.doOnError(throwable -> {
System.out.println("error: " + throwable.getMessage());
}).subscribe(testSubscriber);
System.out.println("Test execution time: "+(System.currentTimeMillis()-startTime));
testSubscriber.awaitTerminalEvent();
long execTime = (System.currentTimeMillis()-startTime);
System.out.println("Test execution time: "+execTime);
testSubscriber.assertCompleted();
System.out.println("Errors: "+testSubscriber.getOnErrorEvents());
System.out.println("Results: "+testSubscriber.getOnNextEvents());
testSubscriber.assertNoErrors();
assertTrue("Execution time ran under the 300ms limit: "+execTime,execTime>=300);
assertTrue("Execution time ran over the 350ms limit: "+execTime,execTime<=350);
testSubscriber.assertValueCount(3);
assertThat(testSubscriber.getOnNextEvents(),containsInAnyOrder("hello","wait","world"));
verify(this.mockSchedulerFactory, times(3)).get("asyncThreadPool");
}
Tasks for the above unit test:
protected List<EspTask<String>> getTimedTasks() {
EspTask longTask = new EspTask("helloTask") {
#Override
public Object doCall() throws Exception {
Thread.currentThread().sleep(100);
return "hello";
}
};
EspTask longerTask = new EspTask("waitTask") {
#Override
public Object doCall() throws Exception {
Thread.currentThread().sleep(150);
return "wait";
}
};
EspTask longestTask = new EspTask("worldTask") {
#Override
public Object doCall() throws Exception {
Thread.currentThread().sleep(300);
return "world";
}
};
return Arrays.asList(longTask, longerTask, longestTask);
}

You can use Observable.onErrorReturn(), and return special value (e.g. null), then filter non-special values downstream. Keep in mind that source observable will complete on error. Also depending on use case Observable.onErrorResumeNext()methods can be useful aswell. If you are interested in error notifications, use Observable.materialize(), this will convert items and onError(), onComplete() into Notifications, which then can be filtered by Notification.getKind()
Edit.
All operators mentioned above should be added right after .toObservable().subscribeOn(this.schedulerFactory.get(groupName)); assuming try/catch was absent.

You want to use mergeDelayError:
public <T> Observable<T> observeManagedAsync(String groupName,List<EspTask<T>> tasks) {
return Observable.mergeDelayError(Observable
.from(tasks)
.map(task -> {
try {
return new MyCommand(task.getTaskId(),groupName,task).toObservable().subscribeOn(this.schedulerFactory.get(groupName));
} catch(Exception ex) {
return Observable.error(ex);
}
}));
}
Note that your MyCommand constructor should not throw any exceptions; this allows your code to be written more concisely:
public <T> Observable<T> observeManagedAsync(String groupName,List<EspTask<T>> tasks) {
return from(tasks)
.map(task -> new MyCommand(task.getTaskId(), groupName, task)
.toObservable()
.subscribeOn(this.schedulerFactory.get(groupName)))
.compose(Observable::mergeDelayError);
}
Keep in mind that this will still invoke onError at most once; if you need explicit handling of all errors, use something like an Either<CommandResult, Throwable> as the return type (or handle the errors and return an empty observable).

Use .materialize() to allow all emissions and errors to come through as wrapped notifications then deal with them as you wish:
.flatMap(task -> {
try {
return new MyCommand(task.getTaskId(),groupName,task)
.toObservable()
.subscribeOn(this.schedulerFactory.get(groupName))
.materialize();
} catch(Exception ex) {
return Observable.error(ex).materialize();
}
});