We Keep Coding

How to sequentially chain Vertx CompositeFuture using RXJava?

I need to chain sequentially in order Vertx CompositeFutures in a RxJava style for dependent CompositeFuture, avoiding callback hell.
The use case:
Each CompositeFuture.any/all do some async operations that return futures, lets say myList1, myList2, myList3, but I must wait for CompositeFuture.any(myList1) to complete and return success before doing CompositeFuture.any(myList2), and the same from myList2 to myList3. Naturally, the CompositeFuture itself does the jobs async, but just for its set of operations, since the next set have to be done just after the first set goes well.
Doing it in a "callback-hell style" would be:
public static void myFunc(Vertx vertx, Handler<AsyncResult<CompositeFuture>> asyncResultHandler) {
CompositeFuture.any(myList1 < Future >)
.onComplete(ar1 -> {
if (!ar1.succeeded()) {
asyncResultHandler.handle(ar1);
} else {
CompositeFuture.any(myList2 < Future >)
.onComplete(ar2 -> {
if (!ar2.succeeded()) {
asyncResultHandler.handle(ar2);
} else {
CompositeFuture.all(myList3 < Future >)
.onComplete(ar3 -> {
asyncResultHandler.handle(ar3);
.... <ARROW OF CLOSING BRACKETS> ...
}
Now I tried somenthing like this:
public static void myFunc(Vertx vertx, Handler<AsyncResult<CompositeFuture>> asyncResultHandler) {
Single
.just(CompositeFuture.any(myList1 < Future >))
.flatMap(previousFuture -> rxComposeAny(previousFuture, myList2 < Future >))
.flatMap(previousFuture -> rxComposeAll(previousFuture, myList3 < Future >))
.subscribe(SingleHelper.toObserver(asyncResultHandler));
}
public static Single<CompositeFuture> rxComposeAny(CompositeFuture previousResult, List<Future> myList) {
if (previousResult.failed()) return Single.just(previousResult); // See explanation bellow
CompositeFuture compositeFuture = CompositeFuture.any(myList);
return Single.just(compositeFuture);
}
public static Single<CompositeFuture> rxComposeAll(CompositeFuture previousResult, List<Future> myList) {
if (previousResult.failed()) return Single.just(previousResult);
CompositeFuture compositeFuture = CompositeFuture.any(myList);
return Single.just(compositeFuture);
}
}
Much more compact and clear. But, I am not succeeding in passing the previous fails to the asyncResultHandler.
My idea was as follows: The flatMap passes the previous CompositeFuture result and I want to check if it failed. The next rxComposeAny/All first checks to see if previous failed, if so, just returns the failed CompositeFuture and so on until it hits the handler in the subscriber. If the previous passed the test, I`m ok to continue passing the current result till the last successful CompositeFuture hits the handler.
The problem is that the check
if (previousResult.failed()) return Single.just(previousResult); // See explanation bellow
doesn't work, and all the CompositeFutures are processed, but not tested for successful completion, just the last one ends up being passed to the asyncResultHandler which will test for overall failure (but in the case of my code, it ends up cheking just the last one)
I`m using Vertx 3.9.0 and RxJava 2 Vertx API.
Disclosure: I have experience in Vertx, but I'm totally new in RxJava. So I appreciate any answer, from technical solutions to conceptual explanations.
Thank you.
EDIT (after excellent response of #homerman):
I need to have the exact same behavior of the "callback hell style" of sequentially dependent CompositeFutures, ie, the next must be called after onComplete and test for completed with failure or success. The complexity comes from the fact that:
I have to use vertx CompositeAll/Any methods, not zip. Zip provides behaviour similar to CompositeAll, but not CompositeAny.
CompositeAll/Any return the completed future just inside onComplete method. If I check it before as showed above, since it is async, I will get unresolved futures.
CompositeAll/Any if failed will not throw error, but failed future inside onComplete, so I cannot use onError from rxJava.
For example, I tried the following change in the rxComposite function:
public static Single<CompositeFuture> rxLoadVerticlesAny(CompositeFuture previousResult, Vertx vertx, String deploymentName,
List<Class<? extends Verticle>> verticles, JsonObject config) {
previousResult.onComplete(event -> {
if (event.failed()) {
return Single.just(previousResult);
} else {
CompositeFuture compositeFuture = CompositeFuture.any(VertxDeployHelper.deploy(vertx, verticles, config));
return Single.just(compositeFuture);
}
}
);
}
But naturally it does not compile, since lambda is void. How can I reproduce this exact same behavior it rxJava in Vertx?

Just to clarify something...
Each CompositeFuture.any/all do some async operations that return
futures, lets say myList1, myList2, myList3, but I must wait for
CompositeFuture.any(myList1) to complete and return success before
doing CompositeFuture.any(myList2), and the same from myList2 to
myList3.
You've offered CompositeFuture.any() and CompositeFuture.all() as points of reference, but the behavior you describe is consistent with all(), which is to say the resulting composite will yield success only if all its constituents do.
For the purpose of my answer, I'm assuming all() is the behavior you expect.
In RxJava, an unexpected error triggered by an exception will result in termination of the stream with the underlying exception being delivered to the observer via the onError() callback.
As a small demo, assume the following setup:
final Single<String> a1 = Single.just("Batch-A-Operation-1");
final Single<String> a2 = Single.just("Batch-A-Operation-2");
final Single<String> a3 = Single.just("Batch-A-Operation-3");
final Single<String> b1 = Single.just("Batch-B-Operation-1");
final Single<String> b2 = Single.just("Batch-B-Operation-2");
final Single<String> b3 = Single.just("Batch-B-Operation-3");
final Single<String> c1 = Single.just("Batch-C-Operation-1");
final Single<String> c2 = Single.just("Batch-C-Operation-2");
final Single<String> c3 = Single.just("Batch-C-Operation-3");
Each Single represents a discrete operation to be performed, and they are logically named according to some logical grouping (ie they are meant to be executed together). For example, "Batch-A" corresponds to your "myList1", "Batch-B" to your "myList2", ...
Assume the following stream:
Single
.zip(a1, a2, a3, (s, s2, s3) -> {
return "A's completed successfully";
})
.flatMap((Function<String, SingleSource<String>>) s -> {
throw new RuntimeException("B's failed");
})
.flatMap((Function<String, SingleSource<String>>) s -> {
return Single.zip(c1, c2, c3, (one, two, three) -> "C's completed successfully");
})
.subscribe(
s -> System.out.println("## onSuccess(" + s + ")"),
t -> System.out.println("## onError(" + t.getMessage() + ")")
);
(If you're not familiar, the zip() operator can be used to combine the results of all the sources supplied as input to emit another/new source).
In this stream, because the processing of the B's ends up throwing an exception:
the stream is terminated during the execution of the B's
the exception is reported to the observer (ie the onError() handler is triggered)
the C's are never processed
If what you want, however, is to decide for yourself whether or not to execute each branch, one approach you could take is to pass the results from previous operations down the stream using some sort of state holder, like so:
class State {
final String value;
final Throwable error;
State(String value, Throwable error) {
this.value = value;
this.error = error;
}
}
The stream could then be modified to conditionally execute different batches, for example:
Single
.zip(a1, a2, a3, (s, s2, s3) -> {
try {
// Execute the A's here...
return new State("A's completed successfully", null);
} catch(Throwable t) {
return new State(null, t);
}
})
.flatMap((Function<State, SingleSource<State>>) s -> {
if(s.error != null) {
// If an error occurred upstream, skip this batch...
return Single.just(s);
} else {
try {
// ...otherwise, execute the B's
return Single.just(new State("B's completed successfully", null));
} catch(Throwable t) {
return Single.just(new State(null, t));
}
}
})
.flatMap((Function<State, SingleSource<State>>) s -> {
if(s.error != null) {
// If an error occurred upstream, skip this batch...
return Single.just(s);
} else {
try {
// ...otherwise, execute the C's
return Single.just(new State("C's completed successfully", null));
} catch(Throwable t) {
return Single.just(new State(null, t));
}
}
})
.subscribe(
s -> {
if(s.error != null) {
System.out.println("## onSuccess with error: " + s.error.getMessage());
} else {
System.out.println("## onSuccess without error: " + s.value);
}
},
t -> System.out.println("## onError(" + t.getMessage() + ")")
);

After some research in Vertx source code, I found a public method that the rx version of CompositeFuture uses to convert 'traditional' CompositeFuture to its rx version. The method is io.vertx.reactivex.core.CompositeFuture.newInstance. With this workaround, I could use my traditional method and then convert it to use in the rx chain. This was what I wanted, because it was problematic to change the existing traditional method.
Here is the code with comments:
rxGetConfig(vertx)
.flatMap(config -> {
return rxComposeAny(vertx, config)
.flatMap(r -> rxComposeAny(vertx, config))
.flatMap(r -> rxComposeAll(vertx, config));
})
.subscribe(
compositeFuture -> {
compositeFuture.onSuccess(event -> startPromise.complete());
},
error -> startPromise.fail(error));
public static Single<JsonObject> rxGetConfig(Vertx vertx) {
ConfigRetrieverOptions enrichConfigRetrieverOptions = getEnrichConfigRetrieverOptions();
// the reason we create new vertx is just to get an instance that is rx
// so this ConfigRetriever is from io.vertx.reactivex.config, instead of normal io.vertx.config
ConfigRetriever configRetriever = ConfigRetriever.create(io.vertx.reactivex.core.Vertx.newInstance(vertx), enrichConfigRetrieverOptions);
return configRetriever.rxGetConfig();
}
public static Single<io.vertx.reactivex.core.CompositeFuture> rxComposeAny(Vertx vertx, JsonObject config) {
// instead of adapted all the parameters of myMethodsThatReturnsFutures to be rx compliant,
// we create it 'normally' and the converts bellow to rx CompositeFuture
CompositeFuture compositeFuture = CompositeFuture.any(myMethodsThatReturnsFutures(config));
return io.vertx.reactivex.core.CompositeFuture
.newInstance(compositeFuture)
.rxOnComplete();
}

defer thenApplyAsync execution

I have following scenario.
CompletableFuture<T> result = CompletableFuture.supplyAsync(task, executor);
result.thenRun(() -> {
...
});
// ....
// after some more code, based on some condition I attach the thenApply() to result.
if ( x == 1) {
result.thenApplyAsync(t -> {
return null;
});
}
The question is what if the CompletableFuture thread finishes the execution before the main thread reaches the thenApplyAsync ? does the CompletableFuture result shall attach itself to thenApply. i.e should callback be declared at the time of defining CompletableFuture.supplyAsync() itself ?
Also what is the order of execution ? thenRun() is always executed at last (after thenApply()) ?
Is there any drawback to use this strategy?

You seem to be missing an important point. When you chain a dependent function, you are not altering the future you’re invoking the chaining method on.
Instead, each of these methods returns a new completion stage representing the dependent action.
Since you are attaching two dependent actions to result, which represent the task passed to supplyAsync, there is no relationship between these two actions. They may run in an arbitrary order and even at the same time in different threads.
Since you are not storing the future returned by thenApplyAsync anywhere, the result of its evaluation would be lost anyway. Assuming that your function returns a result of the same type as T, you could use
if(x == 1) {
result = result.thenApplyAsync(t -> {
return null;
});
}
to replace the potentially completed future with the new future that only gets completed when the result of the specified function has been evaluated. The runnable registered at the original future via thenRun still does not depend on this new future. Note that thenApplyAsync without an executor will always use the default executor, regardless of which executor was used to complete the other future.
If you want to ensure that the Runnable has been successfully executed before any other stage, you can use
CompletableFuture<T> result = CompletableFuture.supplyAsync(task, executor);
CompletableFuture<Void> thenRun = result.thenRun(() -> {
//...
});
result = result.thenCombine(thenRun, (t,v) -> t);
An alternative would be
result = result.whenComplete((value, throwable) -> {
//...
});
but here, the code will be always executed even in the exceptional case (which includes cancellation). You would have to check whether throwable is null, if you want to execute the code only in the successful case.
If you want to ensure that the runnable runs after both actions, the simplest strategy would be to chain it after the if statement, when the final completion stage is defined:
if(x == 1) {
result = result.thenApplyAsync(t -> {
return null;
});
}
result.thenRun(() -> {
//...
});
If that is not an option, you would need an incomplete future which you can complete on either result:
CompletableFuture<T> result = CompletableFuture.supplyAsync(task, executor);
//...
CompletableFuture<T> finalStage = new CompletableFuture<>();
finalStage.thenRun(() -> {
//...
});
// ...
if(x == 1) {
result = result.thenApplyAsync(t -> {
return null;
});
}
result.whenComplete((v,t) -> {
if(t != null) finalStage.completeExceptionally(t); else finalStage.complete(v);
});
The finalStage initially has no defined way of completion, but we can still chain dependent actions. Once we know the actual future, we can chain a handler which will complete our finalStage with whatever result we have.
As a final note, the methods without …Async, like thenRun, provide the least control over the evaluation thread. They may get executed in whatever thread completed the future, like one of executor’s threads in your example, but also directly in the thread calling thenRun, and even less intuitive, in your original example, the runnable may get executed during the unrelated thenApplyAsync invocation.

Explanation needed, applyAsync with nested allOf treats CompletionStage as finished

In my app I have 3 future calls, that are done in parallel and when a response for one of them is received, I have another 3 requests that all should finish before proceeding with code execution, precisely the DeferredResult from spring.
After a while, I realized that page is sometimes rendered before the latter 3 requests are done. Original source code (ommited logic for simplicity):
public DeferredResult<String> someControllerMethod() {
DeferredResult<String> result = new DeferredResult();
CompletableFuture.allOf(
future1(),
future2(),
future3()
)
.whenComplete((aVoid, throwable) -> result.setResult("something"));
return result;
}
public CompletableFuture<?> future3() {
return someService.asyncCall()
.thenApplyAsync(response -> {
....
return CompletableFuture.allOf(
future4(),
future5(),
future6()
);
}
);
}
With thenApplyAsync sometimes DeferredResult is completed before actual future, while changing to thenComposeAsync seems solve the issue. Could someone explain me why? Or it's a bug in my code somewhere and it should not behave this way?

thenApply[Async] accepts a function that evaluates to an arbitrary value. Once the value has been returned, the future will be completed with that value. When the function, like in your code, returns another future, this doesn’t add an additional meaning to it, the future will be the result value, whether completed or not, just like any other object.
In fact, your
public CompletableFuture<Void> future3() {
return someService.asyncCall()
.thenApplyAsync(response -> {
....
return CompletableFuture.allOf(
future4(),
future5(),
future6()
);
}
);
}
method does not even compile, as the result is CompletableFuture<CompletableFuture<Void>>, a future whose result value is another future. The only way not to spot the error, is to use a broader type, e.g. CompletableFuture<Object> or CompletableFuture<?>, as the return type of future3().
In contrast, thenCompose[Async] expects a function that evaluates to another future, to exactly the outcome you expect. That’s the fundamental different between “apply” and “compose”. If you keep the specific CompletableFuture<Void> return type for future3(), the compiler already guides you to use “compose”, as only that will be accepted.
public CompletableFuture<Void> future3() {
return someService.asyncCall()
.thenComposeAsync(response -> {
....
return CompletableFuture.allOf(
future4(),
future5(),
future6()
);
}
);
}

CompletionStage: Return a CompletionStage within exceptionally block [duplicate]

I don't see an obvious way to handle an exception with an asynchronous result.
For example, if I want to retry an async operation, I would expect something like this:
CompletionStage<String> cf = askPong("cause error").handleAsync((x, t) -> {
if (t != null) {
return askPong("Ping");
} else {
return x;
}
});
Where askPong asks an actor:
public CompletionStage<String> askPong(String message){
Future sFuture = ask(actorRef, message, 1000);
final CompletionStage<String> cs = toJava(sFuture);
return cs;
}
However handleAsync doesn't do what you think it does - it runs the callbacks on another thread asynchronously. Returning a CompletionStage here is not correct.
Jeopardy question of the day: thenApply is to thenCompose as exceptionally is to what?

Is this what you are looking for?
askPong("cause error")
.handle( (pong, ex) -> ex == null
? CompletableFuture.completedFuture(pong)
: askPong("Ping")
).thenCompose(x -> x);
Also, do not use the ...Async methods unless you intend for the body of the supplied function to be executed asynchronously. So when you do something like
.handleAsync((x, t) -> {
if (t != null) {
return askPong("Ping");
} else {
return x;
})
You are asking for the if-then-else to be run in a separate thread. Since askPong returns a CompletableFuture, there's probably no reason to run it asynchronously.

Jeopardy question of the day: thenApply is to thenCompose as exceptionally is to what?
I know this was initially java-8, but, since java-12, the answer would be exceptionallyCompose:
exceptionallyCompose[Async](Function<Throwable,? extends CompletionStage<T>> fn [, Executor executor])
Returns a new CompletionStage that, when this stage completes exceptionally, is composed using the results of the supplied function applied to this stage's exception.
As the JavaDoc indicates, the default implementation is:
return handle((r, ex) -> (ex == null)
? this
: fn.apply(ex))
.thenCompose(Function.identity());
That is, using handle() to call the fallback, and thenCompose() to unwrap the resulting nested CompletableFuture<CompletableFuture<T>> – i.e., what you would have done in previous versions of Java (like in Misha’s answer), except you would have to replace this with completedFuture(r).

After a lot of frustration in trying to figure out the proper way of doing Scala's recoverWith in Java 8, I ended up just writing my own. I still don't know if this is the best approach, but I created something like:
public RecoveryChainAsync<T> recoverWith(Function<Throwable,
CompletableFuture<T>> fn);
With repeated calls to recoverWith, I queue up the functions inside the recovery chain and implement the recovery flow myself with "handle". RecoveryChainAsync.getCompletableFuture() then returns a representative CompletableFuture for the entire chain. Hope this helps.

RxJava: Subscription's unsubscribe() method isn't invoked

In code bellow I need to release some resources on unsubscription (where it logs "release").
Observable first = Observable.create(new Observable.OnSubscribe<Object>() {
#Override
public void call(Subscriber<? super Object> subscriber) {
subscriber.add(Subscriptions.create(() -> {
log(“release”);
}));
}
}).doOnUnsubscribe(() -> log(“first”));
Observable second = Observable.create(…).doOnUnsubscribe(() -> log(“second”));
Observable result = first.mergeWith(second).doOnUnsubscribe(() -> log(“result”));
Subscription subscription = result.subscribe(…);
//…
subscription.unsubscribe();
But it logs only “result”. Looks like unsubscription is not propagated to merge’s child observables. So how to handle unsubscription inside of first observable’s Observable.OnSubscribe?

Most of the time, calling unsubscribe has only effect on a live sequence and may not propagate if certain sequences have completed: the operators may not keep their sources around so they can avoid memory leaks. The main idea would be that operators release any resources they manage on termination just before or just after they call their downstream's onError or onCompleted methods, but this is somewhat inconsistent with 1.x.
If you want to make sure resources are releases, look at the using operator which will release your resource upon termination or unsubscription:
Observable.using(
() -> "resource",
r -> Observable.just(r),
r -> System.out.println("Releasing " + r))
.subscribe(System.out::println);