We Keep Coding

How to sequentially chain Vertx CompositeFuture using RXJava?

I need to chain sequentially in order Vertx CompositeFutures in a RxJava style for dependent CompositeFuture, avoiding callback hell.
The use case:
Each CompositeFuture.any/all do some async operations that return futures, lets say myList1, myList2, myList3, but I must wait for CompositeFuture.any(myList1) to complete and return success before doing CompositeFuture.any(myList2), and the same from myList2 to myList3. Naturally, the CompositeFuture itself does the jobs async, but just for its set of operations, since the next set have to be done just after the first set goes well.
Doing it in a "callback-hell style" would be:
public static void myFunc(Vertx vertx, Handler<AsyncResult<CompositeFuture>> asyncResultHandler) {
CompositeFuture.any(myList1 < Future >)
.onComplete(ar1 -> {
if (!ar1.succeeded()) {
asyncResultHandler.handle(ar1);
} else {
CompositeFuture.any(myList2 < Future >)
.onComplete(ar2 -> {
if (!ar2.succeeded()) {
asyncResultHandler.handle(ar2);
} else {
CompositeFuture.all(myList3 < Future >)
.onComplete(ar3 -> {
asyncResultHandler.handle(ar3);
.... <ARROW OF CLOSING BRACKETS> ...
}
Now I tried somenthing like this:
public static void myFunc(Vertx vertx, Handler<AsyncResult<CompositeFuture>> asyncResultHandler) {
Single
.just(CompositeFuture.any(myList1 < Future >))
.flatMap(previousFuture -> rxComposeAny(previousFuture, myList2 < Future >))
.flatMap(previousFuture -> rxComposeAll(previousFuture, myList3 < Future >))
.subscribe(SingleHelper.toObserver(asyncResultHandler));
}
public static Single<CompositeFuture> rxComposeAny(CompositeFuture previousResult, List<Future> myList) {
if (previousResult.failed()) return Single.just(previousResult); // See explanation bellow
CompositeFuture compositeFuture = CompositeFuture.any(myList);
return Single.just(compositeFuture);
}
public static Single<CompositeFuture> rxComposeAll(CompositeFuture previousResult, List<Future> myList) {
if (previousResult.failed()) return Single.just(previousResult);
CompositeFuture compositeFuture = CompositeFuture.any(myList);
return Single.just(compositeFuture);
}
}
Much more compact and clear. But, I am not succeeding in passing the previous fails to the asyncResultHandler.
My idea was as follows: The flatMap passes the previous CompositeFuture result and I want to check if it failed. The next rxComposeAny/All first checks to see if previous failed, if so, just returns the failed CompositeFuture and so on until it hits the handler in the subscriber. If the previous passed the test, I`m ok to continue passing the current result till the last successful CompositeFuture hits the handler.
The problem is that the check
if (previousResult.failed()) return Single.just(previousResult); // See explanation bellow
doesn't work, and all the CompositeFutures are processed, but not tested for successful completion, just the last one ends up being passed to the asyncResultHandler which will test for overall failure (but in the case of my code, it ends up cheking just the last one)
I`m using Vertx 3.9.0 and RxJava 2 Vertx API.
Disclosure: I have experience in Vertx, but I'm totally new in RxJava. So I appreciate any answer, from technical solutions to conceptual explanations.
Thank you.
EDIT (after excellent response of #homerman):
I need to have the exact same behavior of the "callback hell style" of sequentially dependent CompositeFutures, ie, the next must be called after onComplete and test for completed with failure or success. The complexity comes from the fact that:
I have to use vertx CompositeAll/Any methods, not zip. Zip provides behaviour similar to CompositeAll, but not CompositeAny.
CompositeAll/Any return the completed future just inside onComplete method. If I check it before as showed above, since it is async, I will get unresolved futures.
CompositeAll/Any if failed will not throw error, but failed future inside onComplete, so I cannot use onError from rxJava.
For example, I tried the following change in the rxComposite function:
public static Single<CompositeFuture> rxLoadVerticlesAny(CompositeFuture previousResult, Vertx vertx, String deploymentName,
List<Class<? extends Verticle>> verticles, JsonObject config) {
previousResult.onComplete(event -> {
if (event.failed()) {
return Single.just(previousResult);
} else {
CompositeFuture compositeFuture = CompositeFuture.any(VertxDeployHelper.deploy(vertx, verticles, config));
return Single.just(compositeFuture);
}
}
);
}
But naturally it does not compile, since lambda is void. How can I reproduce this exact same behavior it rxJava in Vertx?

Just to clarify something...
Each CompositeFuture.any/all do some async operations that return
futures, lets say myList1, myList2, myList3, but I must wait for
CompositeFuture.any(myList1) to complete and return success before
doing CompositeFuture.any(myList2), and the same from myList2 to
myList3.
You've offered CompositeFuture.any() and CompositeFuture.all() as points of reference, but the behavior you describe is consistent with all(), which is to say the resulting composite will yield success only if all its constituents do.
For the purpose of my answer, I'm assuming all() is the behavior you expect.
In RxJava, an unexpected error triggered by an exception will result in termination of the stream with the underlying exception being delivered to the observer via the onError() callback.
As a small demo, assume the following setup:
final Single<String> a1 = Single.just("Batch-A-Operation-1");
final Single<String> a2 = Single.just("Batch-A-Operation-2");
final Single<String> a3 = Single.just("Batch-A-Operation-3");
final Single<String> b1 = Single.just("Batch-B-Operation-1");
final Single<String> b2 = Single.just("Batch-B-Operation-2");
final Single<String> b3 = Single.just("Batch-B-Operation-3");
final Single<String> c1 = Single.just("Batch-C-Operation-1");
final Single<String> c2 = Single.just("Batch-C-Operation-2");
final Single<String> c3 = Single.just("Batch-C-Operation-3");
Each Single represents a discrete operation to be performed, and they are logically named according to some logical grouping (ie they are meant to be executed together). For example, "Batch-A" corresponds to your "myList1", "Batch-B" to your "myList2", ...
Assume the following stream:
Single
.zip(a1, a2, a3, (s, s2, s3) -> {
return "A's completed successfully";
})
.flatMap((Function<String, SingleSource<String>>) s -> {
throw new RuntimeException("B's failed");
})
.flatMap((Function<String, SingleSource<String>>) s -> {
return Single.zip(c1, c2, c3, (one, two, three) -> "C's completed successfully");
})
.subscribe(
s -> System.out.println("## onSuccess(" + s + ")"),
t -> System.out.println("## onError(" + t.getMessage() + ")")
);
(If you're not familiar, the zip() operator can be used to combine the results of all the sources supplied as input to emit another/new source).
In this stream, because the processing of the B's ends up throwing an exception:
the stream is terminated during the execution of the B's
the exception is reported to the observer (ie the onError() handler is triggered)
the C's are never processed
If what you want, however, is to decide for yourself whether or not to execute each branch, one approach you could take is to pass the results from previous operations down the stream using some sort of state holder, like so:
class State {
final String value;
final Throwable error;
State(String value, Throwable error) {
this.value = value;
this.error = error;
}
}
The stream could then be modified to conditionally execute different batches, for example:
Single
.zip(a1, a2, a3, (s, s2, s3) -> {
try {
// Execute the A's here...
return new State("A's completed successfully", null);
} catch(Throwable t) {
return new State(null, t);
}
})
.flatMap((Function<State, SingleSource<State>>) s -> {
if(s.error != null) {
// If an error occurred upstream, skip this batch...
return Single.just(s);
} else {
try {
// ...otherwise, execute the B's
return Single.just(new State("B's completed successfully", null));
} catch(Throwable t) {
return Single.just(new State(null, t));
}
}
})
.flatMap((Function<State, SingleSource<State>>) s -> {
if(s.error != null) {
// If an error occurred upstream, skip this batch...
return Single.just(s);
} else {
try {
// ...otherwise, execute the C's
return Single.just(new State("C's completed successfully", null));
} catch(Throwable t) {
return Single.just(new State(null, t));
}
}
})
.subscribe(
s -> {
if(s.error != null) {
System.out.println("## onSuccess with error: " + s.error.getMessage());
} else {
System.out.println("## onSuccess without error: " + s.value);
}
},
t -> System.out.println("## onError(" + t.getMessage() + ")")
);

After some research in Vertx source code, I found a public method that the rx version of CompositeFuture uses to convert 'traditional' CompositeFuture to its rx version. The method is io.vertx.reactivex.core.CompositeFuture.newInstance. With this workaround, I could use my traditional method and then convert it to use in the rx chain. This was what I wanted, because it was problematic to change the existing traditional method.
Here is the code with comments:
rxGetConfig(vertx)
.flatMap(config -> {
return rxComposeAny(vertx, config)
.flatMap(r -> rxComposeAny(vertx, config))
.flatMap(r -> rxComposeAll(vertx, config));
})
.subscribe(
compositeFuture -> {
compositeFuture.onSuccess(event -> startPromise.complete());
},
error -> startPromise.fail(error));
public static Single<JsonObject> rxGetConfig(Vertx vertx) {
ConfigRetrieverOptions enrichConfigRetrieverOptions = getEnrichConfigRetrieverOptions();
// the reason we create new vertx is just to get an instance that is rx
// so this ConfigRetriever is from io.vertx.reactivex.config, instead of normal io.vertx.config
ConfigRetriever configRetriever = ConfigRetriever.create(io.vertx.reactivex.core.Vertx.newInstance(vertx), enrichConfigRetrieverOptions);
return configRetriever.rxGetConfig();
}
public static Single<io.vertx.reactivex.core.CompositeFuture> rxComposeAny(Vertx vertx, JsonObject config) {
// instead of adapted all the parameters of myMethodsThatReturnsFutures to be rx compliant,
// we create it 'normally' and the converts bellow to rx CompositeFuture
CompositeFuture compositeFuture = CompositeFuture.any(myMethodsThatReturnsFutures(config));
return io.vertx.reactivex.core.CompositeFuture
.newInstance(compositeFuture)
.rxOnComplete();
}

Execute code based on result of two parallel completable futures

I have a scenario where in I have to issue two REST calls that return a value each based on the current system state, and based on those two values, have to trigger a final clean up task asynchronously - the flow of control being more like a 'Y' scenario . I have looked through the CompletableFuture interface, and is unable to find a way to accomplish this in a non-blocking fashion
I have tried this, and cant seem to find a way to get it working
// Verify task status
CompletableFuture<AuditResult> checkOneFuture =
CompletableFuture.supplyAsync(() -> dummyService.fetchSystemState(var1, var2),
executorService);
CompletableFuture<AuditResult> checkTwoFuture =
CompletableFuture.supplyAsync(() -> dummyService.fetchSystemState(var1, var3),
executorService);
CompletableFuture<CompletableFuture<Boolean>> cleanUpFuture =
checkOneFuture.thenCombineAsync(checkTwoFuture, (check1, check2) -> {
if (check1.getSuccess() && check2.getSuccess()){
CompletableFuture<Boolean> cleanUpFutuer = CompletableFuture.supplyAsync(() -> cleanUp(check1.id), executorService);
return syncFuture;
} else {
return CompletableFuture.completedFuture(false);
}
}, executorService);
cleanUpFuture.join();
The cleanUpFuture is obviously syntactically not correct, and I am trying to figure ways to get this scenario working. Please help

As Slaw says in his comment, why not just return boolean?
CompletableFuture<Boolean> cleanUpFuture =
checkOneFuture.thenCombineAsync(checkTwoFuture, (check1, check2) -> {
if (check1.getSuccess() && check2.getSuccess()) {
return cleanUp(check1.id); // will be scheduled due to combineAsync
} else {
return false;
}
}, executorService);
Note: for a shorter version, you can do
(check1, check2) -> check1.getSuccess() && check2.getSuccess() && cleanUp(check1.id);

You can acheive this by a ForkJoinPool. Subdivizing your call in subtask by calling fork() and reassemble the whole with a join()
For this you have maybe to implements a RecursiveTask
EDIT : Using CompletableFuture
If your purpose is to run two async processings in parallel and trigger another on completion of the later then allOf() is the best method.
Here is an example :
public CompletableFuture<String> findSomeValue() {
return CompletableFuture.supplyAsync(() -> {
sleep(1);
return "Niraj";
});
}
#Test
public void completableFutureAllof() {
List<CompletableFuture<String>> list = new ArrayList<>();
IntStream.range(0, 5).forEach(num -> {
list.add(findSomeValue());
});
CompletableFuture<Void> allfuture = CompletableFuture.allOf(list.toArray(new CompletableFuture[list.size()]));//Created All of object
CompletableFuture<List<String>> allFutureList = allfuture.thenApply(val -> {
return list.stream().map(f -> f.join()).collect(Collectors.toList());
});
CompletableFuture<String> futureHavingAllValues = allFutureList.thenApply(fn -> {
System.out.println("I am here");
return fn.stream().collect(Collectors.joining());});
String concatenateString = futureHavingAllValues.join();
assertEquals("NirajNirajNirajNirajNiraj", concatenateString);
}
This is example and more explanations are provided in this article

RxJava: Why retryWhen/repeatWhen doesn't work?

I've been stuck with this for a day. Inspired in Dan Lew great post, I tried to make a simple testcase for repeatWhen() and retryWhen():
public class ObsTest {
private static final Logger LOG = LoggerFactory.getLogger(ObsTest.class);
#Test
public void test1() throws InterruptedException {
Observable<Integer> obs = rx.Observable.<Integer> create(observer -> {
LOG.info("onSubscribe");
Integer data = RandomUtils.nextInt(0, 1000);
if (data % 2 != 0) {
observer.onError(new RuntimeException("Odd number " + data));
} else {
observer.onNext(data);
}
observer.onCompleted();
}, BackpressureMode.BUFFER);
obs.repeatWhen(completed -> completed.delay(1, TimeUnit.MILLISECONDS))
.retryWhen(error -> error.delay(1, TimeUnit.MILLISECONDS))
.subscribe(i -> LOG.info("value={}", i), e -> LOG.info("Exception = {}", e.getMessage()));
}
My idea is this should run forever, emitting even numbers as "correct" results, and odd numbers as "errors".
Instead, this runs for one or two loops and then stops. And that is when the delay is 1 millisecond, for longer periods of time (ie. 1 second), it runs a single time, emitting just a single odd or even number.
I'm sure I'm doing something wrong, but I can't find what it is.

When you call delay which uses Schedulers.computation() by default you are introducing asynchrony. Once activity starts occurring in a background thread your test will finish and presumably your process is exited. You need to use a blockingSubscribe or put a longish Thread.sleep at the end.

As Dave Moten mentioned, delay uses Schedulers.computation() by default, but you can pass scheduler of your choice instead - for tests purposes you may use TestScheduler and "take control over time". Code below shows how can it be used - as you can see this subscription won't terminal for another 30 days, what is basically forever ;)
public class ObsTest {
#Test
public void test1() {
Observable<Integer> obs = rx.Observable.create(observer -> {
Integer data = RandomUtils.nextInt(0, 1000);
if (data % 2 != 0) {
observer.onError(new RuntimeException("Odd number " + data));
} else {
observer.onNext(data);
}
observer.onCompleted();
}, Emitter.BackpressureMode.BUFFER);
TestScheduler scheduler = Schedulers.<Integer>test();
AssertableSubscriber subscriber = obs.repeatWhen(completed -> completed.delay(1, TimeUnit.MILLISECONDS, scheduler))
.retryWhen(error -> error.delay(1, TimeUnit.MILLISECONDS, scheduler))
.subscribeOn(scheduler)
.test();
subscriber.assertNoValues();
scheduler.advanceTimeBy(30, TimeUnit.SECONDS);
subscriber.assertNoTerminalEvent();
scheduler.advanceTimeBy(30, TimeUnit.DAYS);
subscriber.assertNoTerminalEvent();
}
}

retryWhen operator never retries

I'm implementing a DB update approach with retrials.. Following the common pattern for retryWhen() operator as explained here: Using Rx Java retryWhen() ..
..But my retry logic never executes. I'm debugging it and can see the breakpoint hitting at place 3 shown below but it never goes back to retry logic at place 2. After place 3, its always going to place 4 which is the onComplete handler.
(Code is using Java 8 lambdas)
I've applied a workaround by removing the retryWhen() block altogether
and now invoking the updateWithRetrials() recursively from subscribe's > onError() block. That is working but I don't like that approach.
Please can anyone suggest what is incorrect when I use retryWhen() operator ?
private void updateWithRetrials(some input x)
{
AtomicBoolean retryingUpdate = new AtomicBoolean(false);
...
// 1- Start from here
Observable.<JsonDocument> just(x).map(x1 -> {
if (retryingUpdate.get())
{
//2. retry logic
}
//doing sth with x1 here
...
return <some observable>;
})
.retryWhen(attempts -> attempts.flatMap(n -> {
Throwable cause = n.getThrowable();
if (cause instanceof <errors of interest>)
{
// 3 - break-point hits here
// retry update in 1 sec again
retryingUpdate.set(true);
return Observable.timer(1, TimeUnit.SECONDS);
}
// fail in all other cases...
return Observable.error(n.getThrowable());
}))
.subscribe(
doc -> {
//.. update was successful
},
onError -> {
//for unhandled errors in retryWhen() block
},
{
// 4. onComplete block
Sysout("Update() call completed.");
}
); //subscribe ends here
}

Your problem is due to some performance optimisation with Observable.just().
This Operator, after emmiting the item, does not check if the subscribtion is not cancelled and sends onComplete on all cases.
Observable.retryWhen (and retry) resubscribes on Error, but terminates when source sends onComplete.
Thus, even if the retry operator resubscribes, it gets onComplete from previous subscription and stops.
You may see, that code below fails (as yours):
#Test
public void testJustAndRetry() throws Exception {
AtomicBoolean throwException = new AtomicBoolean(true);
int value = Observable.just(1).map(v->{
if( throwException.compareAndSet(true, false) ){
throw new RuntimeException();
}
return v;
}).retry(1).toBlocking().single();
}
But if you "don't forget" to check subscription, it Works!:
#Test
public void testCustomJust() throws Exception {
AtomicBoolean throwException = new AtomicBoolean(true);
int value = Observable.create((Subscriber<? super Integer> s) -> {
s.onNext(1);
if (!s.isUnsubscribed()) {
s.onCompleted();
}
}
).map(v -> {
if (throwException.compareAndSet(true, false)) {
throw new RuntimeException();
}
return v;
}).retry(1).toBlocking().single();
Assert.assertEquals(1, value);
}

I suppose the error occurs inside map because it cannot occur in just. This is not how retryWhen works.
Implement your observable using create and make sure no errors occur in map. If any error will be thrown in the create block the retryWhen will be called and the unit of work retried depending on your retry logic.
Observable.create(subscriber -> {
// code that may throw exceptions
}).map(item -> {
// code that will not throw any exceptions
}).retryWhen(...)
...

How do I return interim results from Callable?

I have Callable object executed using ExecutorService.
How to return interim results from this callable?
I know there is javax.swing.SwingWorker#publish(results) for Swing but I don't use Swing.

There are a couple of ways of doing this. You could do it with a callback or you could do it with a queue.
Here's an example of doing it with a callback:
public static interface Callback<T> {
public void on(T event);
}
Then, an implementation of the callback that does something with your in progress events:
final Callback<String> callback = new Callback<String>() {
public void on(String event) {
System.out.println(event);
}
};
Now you can use the callback in your pool:
Future<String> submit = pool.submit(new Callable<String>() {
public String call() throws Exception {
for(int i = 0; i < 10; i++) {
callback.on("process " + i);
}
return "done";
}
});

It is not clear what an "interim result" really is. The interfaces used in the concurrency package simply do not define this, but assume methods that resemble more or less pure functions.
Hence, instead this:
interim = compute something
finalresult = compute something else
do something like this:
interim = compute something
final1 = new Pair( interim, fork(new Future() { compute something else }) )
(Pseudocode, thought to convey the idea, not compileable code)
EDIT The idea is: instead of running a single monolithic block of computations (that happens to reach a state where some "interim results" are available) break it up so that the first task returns the former "interim" result and, at the same time, forks a second task that computes the final result. Of course, a handle to this task must be delivered to the caller so that it eventually can get the final result. Usually, this is done with the Future interface.

You can pass, let's say, an AtomicInteger to your class (the one that will be submitted by the executor) inside that class you increment it's value and from the calling thread you check it's value
Something like this:
public class LongComputation {
private AtomicInteger progress = new AtomicInteger(0);
public static void main(String[] args) throws InterruptedException,
ExecutionException {
AtomicInteger progress = new AtomicInteger(0);
LongComputation computation = new LongComputation(progress);
ExecutorService executor = Executors.newFixedThreadPool(2);
Future<Integer> result = executor.submit(() -> computation.compute());
executor.shutdown();
while (!result.isDone()) {
System.out.printf("Progress...%d%%%n", progress.intValue());
TimeUnit.MILLISECONDS.sleep(100);
}
System.out.printf("Result=%d%n", result.get());
}
public LongComputation(AtomicInteger progress) {
this.progress = progress;
}
public int compute() throws InterruptedException {
for (int i = 0; i < 100; i++) {
TimeUnit.MILLISECONDS.sleep(100);
progress.incrementAndGet();
}
return 1_000_000;
}
}

What you're looking for is java.util.concurrent.Future.
A Future represents the result of an asynchronous computation. Methods
are provided to check if the computation is complete, to wait for its
completion, and to retrieve the result of the computation. The result
can only be retrieved using method get when the computation has
completed, blocking if necessary until it is ready. Cancellation is
performed by the cancel method. Additional methods are provided to
determine if the task completed normally or was cancelled. Once a
computation has completed, the computation cannot be cancelled. If you
would like to use a Future for the sake of cancellability but not
provide a usable result, you can declare types of the form Future
and return null as a result of the underlying task.

You would have to roll your own API with something like Observer/Observerable if you want to publish intermediate results as a push. A simpler thing would be to just poll for current state through some self defined method.