I need to chain sequentially in order Vertx CompositeFutures in a RxJava style for dependent CompositeFuture, avoiding callback hell.
The use case:
Each CompositeFuture.any/all do some async operations that return futures, lets say myList1, myList2, myList3, but I must wait for CompositeFuture.any(myList1) to complete and return success before doing CompositeFuture.any(myList2), and the same from myList2 to myList3. Naturally, the CompositeFuture itself does the jobs async, but just for its set of operations, since the next set have to be done just after the first set goes well.
Doing it in a "callback-hell style" would be:
public static void myFunc(Vertx vertx, Handler<AsyncResult<CompositeFuture>> asyncResultHandler) {
CompositeFuture.any(myList1 < Future >)
.onComplete(ar1 -> {
if (!ar1.succeeded()) {
asyncResultHandler.handle(ar1);
} else {
CompositeFuture.any(myList2 < Future >)
.onComplete(ar2 -> {
if (!ar2.succeeded()) {
asyncResultHandler.handle(ar2);
} else {
CompositeFuture.all(myList3 < Future >)
.onComplete(ar3 -> {
asyncResultHandler.handle(ar3);
.... <ARROW OF CLOSING BRACKETS> ...
}
Now I tried somenthing like this:
public static void myFunc(Vertx vertx, Handler<AsyncResult<CompositeFuture>> asyncResultHandler) {
Single
.just(CompositeFuture.any(myList1 < Future >))
.flatMap(previousFuture -> rxComposeAny(previousFuture, myList2 < Future >))
.flatMap(previousFuture -> rxComposeAll(previousFuture, myList3 < Future >))
.subscribe(SingleHelper.toObserver(asyncResultHandler));
}
public static Single<CompositeFuture> rxComposeAny(CompositeFuture previousResult, List<Future> myList) {
if (previousResult.failed()) return Single.just(previousResult); // See explanation bellow
CompositeFuture compositeFuture = CompositeFuture.any(myList);
return Single.just(compositeFuture);
}
public static Single<CompositeFuture> rxComposeAll(CompositeFuture previousResult, List<Future> myList) {
if (previousResult.failed()) return Single.just(previousResult);
CompositeFuture compositeFuture = CompositeFuture.any(myList);
return Single.just(compositeFuture);
}
}
Much more compact and clear. But, I am not succeeding in passing the previous fails to the asyncResultHandler.
My idea was as follows: The flatMap passes the previous CompositeFuture result and I want to check if it failed. The next rxComposeAny/All first checks to see if previous failed, if so, just returns the failed CompositeFuture and so on until it hits the handler in the subscriber. If the previous passed the test, I`m ok to continue passing the current result till the last successful CompositeFuture hits the handler.
The problem is that the check
if (previousResult.failed()) return Single.just(previousResult); // See explanation bellow
doesn't work, and all the CompositeFutures are processed, but not tested for successful completion, just the last one ends up being passed to the asyncResultHandler which will test for overall failure (but in the case of my code, it ends up cheking just the last one)
I`m using Vertx 3.9.0 and RxJava 2 Vertx API.
Disclosure: I have experience in Vertx, but I'm totally new in RxJava. So I appreciate any answer, from technical solutions to conceptual explanations.
Thank you.
EDIT (after excellent response of #homerman):
I need to have the exact same behavior of the "callback hell style" of sequentially dependent CompositeFutures, ie, the next must be called after onComplete and test for completed with failure or success. The complexity comes from the fact that:
I have to use vertx CompositeAll/Any methods, not zip. Zip provides behaviour similar to CompositeAll, but not CompositeAny.
CompositeAll/Any return the completed future just inside onComplete method. If I check it before as showed above, since it is async, I will get unresolved futures.
CompositeAll/Any if failed will not throw error, but failed future inside onComplete, so I cannot use onError from rxJava.
For example, I tried the following change in the rxComposite function:
public static Single<CompositeFuture> rxLoadVerticlesAny(CompositeFuture previousResult, Vertx vertx, String deploymentName,
List<Class<? extends Verticle>> verticles, JsonObject config) {
previousResult.onComplete(event -> {
if (event.failed()) {
return Single.just(previousResult);
} else {
CompositeFuture compositeFuture = CompositeFuture.any(VertxDeployHelper.deploy(vertx, verticles, config));
return Single.just(compositeFuture);
}
}
);
}
But naturally it does not compile, since lambda is void. How can I reproduce this exact same behavior it rxJava in Vertx?
Just to clarify something...
Each CompositeFuture.any/all do some async operations that return
futures, lets say myList1, myList2, myList3, but I must wait for
CompositeFuture.any(myList1) to complete and return success before
doing CompositeFuture.any(myList2), and the same from myList2 to
myList3.
You've offered CompositeFuture.any() and CompositeFuture.all() as points of reference, but the behavior you describe is consistent with all(), which is to say the resulting composite will yield success only if all its constituents do.
For the purpose of my answer, I'm assuming all() is the behavior you expect.
In RxJava, an unexpected error triggered by an exception will result in termination of the stream with the underlying exception being delivered to the observer via the onError() callback.
As a small demo, assume the following setup:
final Single<String> a1 = Single.just("Batch-A-Operation-1");
final Single<String> a2 = Single.just("Batch-A-Operation-2");
final Single<String> a3 = Single.just("Batch-A-Operation-3");
final Single<String> b1 = Single.just("Batch-B-Operation-1");
final Single<String> b2 = Single.just("Batch-B-Operation-2");
final Single<String> b3 = Single.just("Batch-B-Operation-3");
final Single<String> c1 = Single.just("Batch-C-Operation-1");
final Single<String> c2 = Single.just("Batch-C-Operation-2");
final Single<String> c3 = Single.just("Batch-C-Operation-3");
Each Single represents a discrete operation to be performed, and they are logically named according to some logical grouping (ie they are meant to be executed together). For example, "Batch-A" corresponds to your "myList1", "Batch-B" to your "myList2", ...
Assume the following stream:
Single
.zip(a1, a2, a3, (s, s2, s3) -> {
return "A's completed successfully";
})
.flatMap((Function<String, SingleSource<String>>) s -> {
throw new RuntimeException("B's failed");
})
.flatMap((Function<String, SingleSource<String>>) s -> {
return Single.zip(c1, c2, c3, (one, two, three) -> "C's completed successfully");
})
.subscribe(
s -> System.out.println("## onSuccess(" + s + ")"),
t -> System.out.println("## onError(" + t.getMessage() + ")")
);
(If you're not familiar, the zip() operator can be used to combine the results of all the sources supplied as input to emit another/new source).
In this stream, because the processing of the B's ends up throwing an exception:
the stream is terminated during the execution of the B's
the exception is reported to the observer (ie the onError() handler is triggered)
the C's are never processed
If what you want, however, is to decide for yourself whether or not to execute each branch, one approach you could take is to pass the results from previous operations down the stream using some sort of state holder, like so:
class State {
final String value;
final Throwable error;
State(String value, Throwable error) {
this.value = value;
this.error = error;
}
}
The stream could then be modified to conditionally execute different batches, for example:
Single
.zip(a1, a2, a3, (s, s2, s3) -> {
try {
// Execute the A's here...
return new State("A's completed successfully", null);
} catch(Throwable t) {
return new State(null, t);
}
})
.flatMap((Function<State, SingleSource<State>>) s -> {
if(s.error != null) {
// If an error occurred upstream, skip this batch...
return Single.just(s);
} else {
try {
// ...otherwise, execute the B's
return Single.just(new State("B's completed successfully", null));
} catch(Throwable t) {
return Single.just(new State(null, t));
}
}
})
.flatMap((Function<State, SingleSource<State>>) s -> {
if(s.error != null) {
// If an error occurred upstream, skip this batch...
return Single.just(s);
} else {
try {
// ...otherwise, execute the C's
return Single.just(new State("C's completed successfully", null));
} catch(Throwable t) {
return Single.just(new State(null, t));
}
}
})
.subscribe(
s -> {
if(s.error != null) {
System.out.println("## onSuccess with error: " + s.error.getMessage());
} else {
System.out.println("## onSuccess without error: " + s.value);
}
},
t -> System.out.println("## onError(" + t.getMessage() + ")")
);
After some research in Vertx source code, I found a public method that the rx version of CompositeFuture uses to convert 'traditional' CompositeFuture to its rx version. The method is io.vertx.reactivex.core.CompositeFuture.newInstance. With this workaround, I could use my traditional method and then convert it to use in the rx chain. This was what I wanted, because it was problematic to change the existing traditional method.
Here is the code with comments:
rxGetConfig(vertx)
.flatMap(config -> {
return rxComposeAny(vertx, config)
.flatMap(r -> rxComposeAny(vertx, config))
.flatMap(r -> rxComposeAll(vertx, config));
})
.subscribe(
compositeFuture -> {
compositeFuture.onSuccess(event -> startPromise.complete());
},
error -> startPromise.fail(error));
public static Single<JsonObject> rxGetConfig(Vertx vertx) {
ConfigRetrieverOptions enrichConfigRetrieverOptions = getEnrichConfigRetrieverOptions();
// the reason we create new vertx is just to get an instance that is rx
// so this ConfigRetriever is from io.vertx.reactivex.config, instead of normal io.vertx.config
ConfigRetriever configRetriever = ConfigRetriever.create(io.vertx.reactivex.core.Vertx.newInstance(vertx), enrichConfigRetrieverOptions);
return configRetriever.rxGetConfig();
}
public static Single<io.vertx.reactivex.core.CompositeFuture> rxComposeAny(Vertx vertx, JsonObject config) {
// instead of adapted all the parameters of myMethodsThatReturnsFutures to be rx compliant,
// we create it 'normally' and the converts bellow to rx CompositeFuture
CompositeFuture compositeFuture = CompositeFuture.any(myMethodsThatReturnsFutures(config));
return io.vertx.reactivex.core.CompositeFuture
.newInstance(compositeFuture)
.rxOnComplete();
}
I have a scenario where in I have to issue two REST calls that return a value each based on the current system state, and based on those two values, have to trigger a final clean up task asynchronously - the flow of control being more like a 'Y' scenario . I have looked through the CompletableFuture interface, and is unable to find a way to accomplish this in a non-blocking fashion
I have tried this, and cant seem to find a way to get it working
// Verify task status
CompletableFuture<AuditResult> checkOneFuture =
CompletableFuture.supplyAsync(() -> dummyService.fetchSystemState(var1, var2),
executorService);
CompletableFuture<AuditResult> checkTwoFuture =
CompletableFuture.supplyAsync(() -> dummyService.fetchSystemState(var1, var3),
executorService);
CompletableFuture<CompletableFuture<Boolean>> cleanUpFuture =
checkOneFuture.thenCombineAsync(checkTwoFuture, (check1, check2) -> {
if (check1.getSuccess() && check2.getSuccess()){
CompletableFuture<Boolean> cleanUpFutuer = CompletableFuture.supplyAsync(() -> cleanUp(check1.id), executorService);
return syncFuture;
} else {
return CompletableFuture.completedFuture(false);
}
}, executorService);
cleanUpFuture.join();
The cleanUpFuture is obviously syntactically not correct, and I am trying to figure ways to get this scenario working. Please help
As Slaw says in his comment, why not just return boolean?
CompletableFuture<Boolean> cleanUpFuture =
checkOneFuture.thenCombineAsync(checkTwoFuture, (check1, check2) -> {
if (check1.getSuccess() && check2.getSuccess()) {
return cleanUp(check1.id); // will be scheduled due to combineAsync
} else {
return false;
}
}, executorService);
Note: for a shorter version, you can do
(check1, check2) -> check1.getSuccess() && check2.getSuccess() && cleanUp(check1.id);
You can acheive this by a ForkJoinPool. Subdivizing your call in subtask by calling fork() and reassemble the whole with a join()
For this you have maybe to implements a RecursiveTask
EDIT : Using CompletableFuture
If your purpose is to run two async processings in parallel and trigger another on completion of the later then allOf() is the best method.
Here is an example :
public CompletableFuture<String> findSomeValue() {
return CompletableFuture.supplyAsync(() -> {
sleep(1);
return "Niraj";
});
}
#Test
public void completableFutureAllof() {
List<CompletableFuture<String>> list = new ArrayList<>();
IntStream.range(0, 5).forEach(num -> {
list.add(findSomeValue());
});
CompletableFuture<Void> allfuture = CompletableFuture.allOf(list.toArray(new CompletableFuture[list.size()]));//Created All of object
CompletableFuture<List<String>> allFutureList = allfuture.thenApply(val -> {
return list.stream().map(f -> f.join()).collect(Collectors.toList());
});
CompletableFuture<String> futureHavingAllValues = allFutureList.thenApply(fn -> {
System.out.println("I am here");
return fn.stream().collect(Collectors.joining());});
String concatenateString = futureHavingAllValues.join();
assertEquals("NirajNirajNirajNirajNiraj", concatenateString);
}
This is example and more explanations are provided in this article
Given the following monos:
Mono<Void> mono1 = Mono.fromRunnable(() -> {
System.out.println("sleep1");
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
throw new IllegalStateException(e);
}
System.out.println("mono1");
});
Mono<Void> mono2 = Mono.fromRunnable(() -> {
System.out.println("mono2");
});
Mono<Void> mono3 = Mono.fromRunnable(() -> {
System.out.println("mono3");
});
Both:
mono1
.then(mono2)
.then(mono3)
.block();
And:
mono1
.and(mono2)
.and(mono3)
.block();
Have the same output:
sleep
mono1
mono2
mono3
What's the difference between Mono#then and Mono#and in this case?
From https://projectreactor.io/docs/core/release/reference/index.html#which-operator:
[If you] have a sequence but [you are] not interested in values and [you] want to switch to another Mono at the end, [use] Mono#then(mono).
[If you] want to combine publishers by coordinating their termination from 1 Mono and any source into a Mono, [use] Mono#and.
This doesn't help me finding a case where #and and #then would behave differently unfortunately.
Mono#and just "joins the termination signals from current mono and another source into the returned void mono". It always returns Mono<Void> and only lets you coordinate termination of two Monos.
Mono#then lets you chain two Monos together and the final result will be determined by the Mono passed as a parameter. In this sense, Mono#then is a more primitive version of Mono#flatMap, the only difference is that inside of Mono#flatMap you have access to the result of the previous Mono in a chain that you can transform into another Mono instance.
In addition to that, with Mono#then the operations will be executed sequentially, while with Mono#and there is no guarantee of ordering (at least from the documentation).
I'm trying to create a server call using RxJava2 library that will try to poll server for answer and if receives exception 3 times in a row to return that exception
I've set up a basic call that fetches the response from the server
final Observable<ResponseValue> responseValueObservable = Observable
.fromCallable((Callable) (c) -> return getDispatcher().performSubmit(submitValue);
}
});
return responseValueObservable
.retry(3)
.subscribeOn(Schedulers.io()
.onError((t) -> { log.error(t); Observable.timer(2, SECONDS);}
.retryUntil(() -> { return retryIsEnabled }
so getDispatcher().performSubmit(submitValue) returns either SubmitException or ResponseValue object.
I need the code to retry 3 times, pausing after each exception for 2 seconds and return either ResponseValue or the last SubmitException
So after reading Dan Lew's Blog from previous answer I was able to put together this piece of code which does exactly what I wanted to. retryWhen() on re-subscribes automatically after waiting 2 seconds. With the first successful reply from server it stops.
Observable.fromCallable((d::performSubmit))
.subscribeOn(Schedulers.io())
.doOnSubscribe(subscription -> System.out.println("Subscribing"))
.retryWhen(errors -> {
AtomicInteger counter = new AtomicInteger();
return errors
.takeWhile(e -> counter.incrementAndGet() < 3)
.flatMap(e -> {
System.out.println("delay retry by 2 second(s)");
return Observable.timer(2, TimeUnit.SECONDS);
});
}).blockingSubscribe(res -> result = Optional.of(res), throwable -> t = Optional.of((Exception) throwable));
Use the retryWhen() operator to customize the response to errors. From the excellent overview at Dan Lew's Blog:
responseValueObservable
.retryWhen( errorObservable -> errorObservable
.zipWith(Observable.range(1, 3), (n, i) -> i)
.flatMap(retryCount -> Observable.timer(2, TimeUnit.SECONDS)))
...
I'm having a very specific problem or misunderstanding with rxjava that someone hopefully can help with.
I'm running rxjava 2.1.5 and have the following code snippet:
public static void main(String[] args) {
final Observable<Object> observable = Observable.create(emitter -> {
// Code ...
});
observable.subscribeOn(Schedulers.io())
.retryWhen(error -> {
System.out.println("retryWhen");
return error.retry();
}).subscribe(next -> System.out.println("subscribeNext"),
error -> System.out.println("subscribeError"));
}
After executing this, the program prints:
retryWhen
Process finished with exit code 0
My question, and what I don't understand is: Why is retryWhen called instantly upon subscribing to an Observable? The observable does nothing.
What I want is retryWhen to be called when onError is called on the emitter. Am I misunderstanding how rx works?
Thanks!
Adding new snippet:
public static void main(String[] args) throws InterruptedException {
final Observable<Object> observable = Observable.create(emitter -> {
emitter.onNext("next");
emitter.onComplete();
});
final CountDownLatch latch = new CountDownLatch(1);
observable.subscribeOn(Schedulers.io())
.doOnError(error -> System.out.println("doOnError: " + error.getMessage()))
.retryWhen(error -> {
System.out.println("retryWhen: " + error.toString());
return error.retry();
}).subscribe(next -> System.out.println("subscribeNext"),
error -> System.out.println("subscribeError"),
() -> latch.countDown());
latch.await();
}
Emitter onNext and complete is called. DoOnError is never called. Output is:
retryWhen: io.reactivex.subjects.SerializedSubject#35fb3008
subscribeNext
Process finished with exit code 0
retryWhen calls the provided function when an Observer subscribes to it so you have a main sequence accompanied by a sequence that emits the Throwable the main sequence failed with. You should compose a logic onto the Observable you get in this Function so at the end, one Throwable will result in a value on the other end.
Observable.error(new IOException())
.retryWhen(e -> {
System.out.println("Setting up retryWhen");
int[] count = { 0 };
return e
.takeWhile(v -> ++count[0] < 3)
.doOnNext(v -> { System.out.println("Retrying"); });
})
.subscribe(System.out::println, Throwable::printStackTrace);
Since the e -> { } function body is executed for each individual subscriber, you can have a per subscriber state such as retry counter safely.
Using e -> e.retry() has no effect because the input error flow never gets its onError called.
One issue is, that you don't receive any more results because you'r creating a Thread using retryWhen() but your app seems to finish. To see that behaviour you may want to have a while loop to keep your app running.
That actually means that you need to add something like that to the end of your code:
while (true) {}
Another issue is that you dont emit any error in your sample. You need to emit at least one value to call onNext() else it wont repeat because it's waiting for it.
Here's a working example which a value, then it emits an error and repeat. you can use
.retryWhen(errors -> errors)
which is the same as
.retryWhen(errors -> errors.retry())
Working sample:
public static void main(String[] args) {
Observable
.create(e -> {
e.onNext("test");
e.onError(new Throwable("test"));
})
.retryWhen(errors -> errors.retry())
.subscribeOn(Schedulers.io())
.subscribe(
next -> System.out.println("subscribeNext"),
error -> System.out.println("subscribeError"),
() -> System.out.println("onCompleted")
);
while (true) {
}
}
The reason why you need to emit a result is, that Observable needs to emit a value, else it wait until it receives a new one.
This is because onError can only be called onec (in subscribe), but onNext emits 1..* values.
You can check this behaviour by using doOnError() which provides you the error everytime it retrys the Observable.
Observable
.create(e -> e.onError(new Exception("empty")))
.doOnError(e -> System.out.println("error received " + e))
.retryWhen(errors -> errors.retry())
.subscribeOn(Schedulers.io())
.subscribe(
nextOrSuccess -> System.out.println("nextOrSuccess " + nextOrSuccess),
error -> System.out.println("subscribeError")
);