The VertX example for when you need to query multiple asynchronous resources and use them all in a single operation is:
Future<HttpServer> httpServerFuture = Future.future();
httpServer.listen(httpServerFuture.completer());
Future<NetServer> netServerFuture = Future.future();
netServer.listen(netServerFuture.completer());
CompositeFuture.all(httpServerFuture, netServerFuture).setHandler(ar -> {
if (ar.succeeded()) {
// All servers started
} else {
// At least one server failed
}
});
We need to query two different databases and then use the results in business logic, but the flow is equivalent.
What's the VertX/RxJava equivalent?
Currently people are doing this by nesting a new .flatMap() call every time they need a new variable. I'm left feeling there must be a better way...
We don't actually need the queries to be concurrent but we need to cache both results and pass them to the business logic at the same time some how.
there are many ways to do this, but i've tried to pick an approach that tacks closely to your sample:
#Override
public void start(Future<Void> startFuture) throws Exception {
final HttpServer httpServer = vertx.createHttpServer();
final Completable initializeHttpServer = httpServer.rxListen().toCompletable();
final NetServer netServer = vertx.createNetServer();
final Completable initializeNetServer = netServer.rxListen().toCompletable();
initializeHttpServer.andThen(initializeNetServer)
.subscribe(
() -> { /* All servers started */ },
error -> { /* At least one server failed */ }
);
}
the rxListen() invocations are converted into Completable instances, which are then run serially upon subscription.
the subscriber's onComplete callback will be invoked when both servers are done binding to their respective ports, or...
the onError callback will be invoked if an exception occurs
(also, fwiw, "nesting" flatMap operations for something as trivial as this shouldn't be necessary. "chaining" such operations, however, would be idiomatic usage).
hope that helps!
--UPDATE--
having read the question more carefully, i now see that you were actually asking about how to handle the results of two discrete asynchronous operations.
an alternative to flatMap'ing your way to combining the results would be to use the zip operator, like so:
#Override
public void start(Future<Void> startFuture) throws Exception {
final Single<String> dbQuery1 = Single.fromCallable(() -> { return "db-query-result-1"; });
final Single<String> dbQuery2 = Single.fromCallable(() -> { return "db-query-result-2"; });
Single.zip(dbQuery1, dbQuery2, (result1, result2) -> {
// handle the results from both db queries
// (with Pair being a standard tuple-like class)
return new Pair(result1, result2);
})
.subscribe(
pair -> {
// handle the results
},
error -> {
// something went wrong
}
);
}
per the docs, zip allows you to specify a series of reactive types (Single, Observable, etc) along with a function to transform all the results at once, with the central idea being that it will not emit anything until all the sources have emitted once (or more, depending on the reactive type).
Related
I'm currently using an API which I unfortunately cannot change easily. This API has some methods in the style of this:
public void getOffers(Consumer<List<Offer>> offersConsumer) {
final Call<List<Offer>> offers = auctionService.getOffers();
handleGetOffers(offersConsumer, offers);
}
It's a web api using retrofit, and it enables me to process the response in a consumer, but I much rather want to work with CompletableFutures.
I'm using the data I receive from this endpoint to compose an interface in a game, and therefore compose an inventory, that basically acts as a frontend to the api. What I want to do, is to have my composing method to wait for the consumer to finish, and then provide the processed results. This is what I have so far, but I don't know how to do the step from the consumer to the CompletableFuture:
#Override
public CompletableFuture<Inventory> get(Player player) {
return CompletableFuture.supplyAsync(() -> {
auctionAPI.getOffers(offers -> {
//process the offers, then return the result of the processing, in form of an "Inventory"-Object.
}
});
});
}
I now need to return the result of the processing after all the Items have been received and then processed. How can I achieve this?
Something along the lines should work:
#Override
public CompletableFuture<Inventory> get(Player player) {
CompletableFuture<Inventory> result = new CompletableFuture<>();
CompletableFuture.supplyAsync(() -> {
auctionAPI.getOffers(offers -> {
//process the offers, then return the result of the processing, in form of an "Inventory"-Object.
result.complete(inventory);
}
});
return null;
});
return result;
}
I'm working on a small subsystem that integrates two simple components using RxJava 2.
These two components work in a simple client-server manner where the first component produces observable data opening a resource under the hood.
The resource is not exposed to the second component.
Moreover, it must be open as long as the observable is in use, however the observable object cannot determine when it should be closed.
Speaking in code, an example implementation is like this:
private Disposable disposable;
public void onCreate() {
final Maybe<Object> maybeResource = Maybe.defer(() -> {
System.out.println("open");
// here is the resource under the hood, it is encapsulated in the observable and never gets exposed
final Closeable resource = () -> { };
return Maybe.just(resource)
.doOnDispose(() -> {
// this "destructor" is never called, resulting in a resource leak
System.out.println("close");
resource.close();
})
// arbitrary data, does not represent the data I'm working with, but it hides the resource away
.map(closeable -> new Object());
});
disposable = maybeResource.subscribe(data -> System.out.println("process: " + data));
}
public void onUserWorflow() {
// ...
System.out.println("... ... ...");
// ...
}
public void onDestroy() {
disposable.dispose();
}
The output I'd anticipate to get is:
open
process: <...>
... ... ...
close <-- this is never produced
but the last line, close, is never produced as the doOnDispose method is not invoked and does not work as I might think it's supposed to.
Therefore the resource gets never released.
There is also Maybe.using that does a similar thing, but it does not allow to "span" across the "user workflow".
Is there an RxJava/RxJava 2 way that allows managing "closeable" resources closed on disposing a subscriber?
i guess you need to use Observable.create() instead of Maybe.
Something like that:
final Observable<Object> resourceObservable = Observable.create<Object> {(emitter ->
// do you staff
emitter.onNext(new Object()); //to make observable emit something
emitter.setCancellable (
System.out.println("close");
resource.close();
)
);
disposable = resourceObservable.subscribe(data -> System.out.println("process: " + data));
CompletableFuture executes a task on a separate thread ( uses a thread-pool ) and provides a callback function. Let's say I have an API call in a CompletableFuture. Is that an API call blocking? Would the thread be blocked till it does not get a response from the API? ( I know main thread/tomcat thread will be non-blocking, but what about the thread on which CompletableFuture task is executing? )
Mono is completely non-blocking, as far as I know.
Please shed some light on this and correct me if I am wrong.
CompletableFuture is Async. But is it non-blocking?
One which is true about CompletableFuture is that it is truly async, it allows you to run your task asynchronously from the caller thread and the API such as thenXXX allows you to process the result when it becomes available. On the other hand, CompletableFuture is not always non-blocking. For example, when you run the following code, it will be executed asynchronously on the default ForkJoinPool:
CompletableFuture.supplyAsync(() -> {
try {
Thread.sleep(1000);
}
catch (InterruptedException e) {
}
return 1;
});
It is clear that the Thread in ForkJoinPool that executes the task will be blocked eventually which means that we can't guarantee that the call will be non-blocking.
On the other hand, CompletableFuture exposes API which allows you to make it truly non-blocking.
For example, you can always do the following:
public CompletableFuture myNonBlockingHttpCall(Object someData) {
var uncompletedFuture = new CompletableFuture(); // creates uncompleted future
myAsyncHttpClient.execute(someData, (result, exception -> {
if(exception != null) {
uncompletedFuture.completeExceptionally(exception);
return;
}
uncompletedFuture.complete(result);
})
return uncompletedFuture;
}
As you can see, the API of CompletableFuture future provides you with the complete and completeExceptionally methods that complete your execution whenever it is needed without blocking any thread.
Mono vs CompletableFuture
In the previous section, we got an overview of CF behavior, but what is the central difference between CompletableFuture and Mono?
It worth to mention that we can do blocking Mono as well. No one prevents us from writing the following:
Mono.fromCallable(() -> {
try {
Thread.sleep(1000);
}
catch (InterruptedException e) {
}
return 1;
})
Of course, once we subscribe to the future, the caller thread will be blocked. But we can always work around that by providing an additional subscribeOn operator. Nevertheless, the broader API of Mono is not the key feature.
In order to understand the main difference between CompletableFuture and Mono, lets back to previously mentioned myNonBlockingHttpCall method implementation.
public CompletableFuture myUpperLevelBusinessLogic() {
var future = myNonBlockingHttpCall();
// ... some code
if (something) {
// oh we don't really need anything, let's just throw an exception
var errorFuture = new CompletableFuture();
errorFuture.completeExceptionally(new RuntimeException());
return errorFuture;
}
return future;
}
In the case of CompletableFuture, once the method is called, it will eagerly execute HTTP call to another service/resource. Even though we will not really need the result of the execution after verifying some pre/post conditions, it starts the execution, and additional CPU/DB-Connections/What-Ever-Machine-Resources will be allocated for this work.
In contrast, the Mono type is lazy by definition:
public Mono myNonBlockingHttpCallWithMono(Object someData) {
return Mono.create(sink -> {
myAsyncHttpClient.execute(someData, (result, exception -> {
if(exception != null) {
sink.error(exception);
return;
}
sink.success(result);
})
});
}
public Mono myUpperLevelBusinessLogic() {
var mono = myNonBlockingHttpCallWithMono();
// ... some code
if (something) {
// oh we don't really need anything, let's just throw an exception
return Mono.error(new RuntimeException());
}
return mono;
}
In this case, nothing will happen until the final mono is subscribed. Thus, only when Mono returned by the myNonBlockingHttpCallWithMono method, will be subscribed, the logic provided to Mono.create(Consumer) will be executed.
And we can go even further. We can make our execution much lazier. As you might know, Mono extends Publisher from the Reactive Streams specification. The screaming feature of Reactive Streams is backpressure support. Thus, using the Mono API we can do execution only when the data is really needed, and our subscriber is ready to consume them:
Mono.create(sink -> {
AtomicBoolean once = new AtomicBoolean();
sink.onRequest(__ -> {
if(!once.get() && once.compareAndSet(false, true) {
myAsyncHttpClient.execute(someData, (result, exception -> {
if(exception != null) {
sink.error(exception);
return;
}
sink.success(result);
});
}
});
});
In this example, we execute data only when subscriber called Subscription#request so by doing that it declared its readiness to receive data.
Summary
CompletableFuture is async and can be non-blocking
CompletableFuture is eager. You can't postpone the execution. But you can cancel them (which is better than nothing)
Mono is async/non-blocking and can easily execute any call on different Thread by composing the main Mono with different operators.
Mono is truly lazy and allows postponing execution startup by the subscriber presence and its readiness to consume data.
Building up on Oleh's answer, a possible lazy solution for CompletableFuture would be
public CompletableFuture myNonBlockingHttpCall(CompletableFuture<ExecutorService> dispatch, Object someData) {
var uncompletedFuture = new CompletableFuture(); // creates uncompleted future
dispatch.thenAccept(x -> x.submit(() -> {
myAsyncHttpClient.execute(someData, (result, exception -> {
if(exception != null) {
uncompletedFuture.completeExceptionally(exception);
return;
}
uncompletedFuture.complete(result);
})
}));
return uncompletedFuture;
}
Then, later on you simply do
dispatch.complete(executor);
That would make CompletableFuture equivalent to Mono, but without backpressure, I guess.
I am using Junit 5 Dynamic tests.
My intention is to create a stream of elements from the collection to pass it on to test in JUnit5.
However with this code, I am able to run only 1000 records. How do I make this work seamlessly non-blocking.
MongoCollection<Document> collection = mydatabase.getCollection("mycoll");
final List<Document> cache = Collections.synchronizedList(new ArrayList<Document>());
FindIterable<Document> f = collection.find().batchSize(1000);
f.batchCursor(new SingleResultCallback<AsyncBatchCursor<Document>>() {
#Override
public void onResult(AsyncBatchCursor<Document> t, Throwable thrwbl) {
t.next(new SingleResultCallback<List<Document>>() {
#Override
public void onResult(List<Document> t, Throwable thrwbl) {
if (thrwbl != null) {
th.set(thrwbl);
}
cache.addAll(t);
latch.countDown();;
}
});
}
});
latch.await();
return cache.stream().map(batch->process(batch));
Updated Code
#ParameterizedTest
#MethodSource("setUp")
void cacheTest(MyClazz myclass) throws Exception {
assertTrue(doTest(myclass));
}
public static MongoClient getMongoClient() {
// get client here
}
private static Stream<MyClazz> setUp() throws Exception {
MongoDatabase mydatabase = getMongoClient().getDatabase("test");
List<Throwable> failures = new ArrayList<>();
CountDownLatch latch = new CountDownLatch(1);
List<MyClazz> list = Collections.synchronizedList(new ArrayList<>());
mydatabase.getCollection("testcollection").find()
.toObservable().subscribe(
document -> {
list.add(process(document));
},
throwable -> {
failures.add(throwable);
},
() -> {
latch.countDown();
});
latch.await();
return list.stream();
}
public boolean doTest(MyClazz myclass) {
// processing goes here
}
public MyClazz process(Document doc) {
// doc gets converted to MyClazz
return MyClazz;
}
Even now, I see that all the data is loaded after which the unit testing happens.
I think this is because of latch.await(). However, if I remove that, there is a chance that no test cases are run as the db could possibly be loading collection.
My use case is : I have million records in mongo and am running sort of integration test case with them. It wouldn't be feasible to load all of them in memory and hence I am attempting the streaming solution.
I don't think I fully understand your use case but given that your question is tagged with java and mongo-asyc-driver this requirement is certainly achievable:
create a stream of elements from the collection to pass it on to test ... make this work seamlessly non-blocking
The following code:
Uses the MongoDB RxJava driver to query a collection
Creates a Rx Observable from that collection
Subscribes to that Observable
Records exceptions
Marks completion
CountDownLatch latch = new CountDownLatch(1);
List<Throwable> failures = new ArrayList<>();
collection.find()
.toObservable().subscribe(
// on next, this is invoked for each document returned by your find call
document -> {
// presumably you'll want to do something here to meet this requirement: "pass it on to test in JUnit5"
System.out.println(document);
},
/// on error
throwable -> {
failures.add(throwable);
},
// on completion
() -> {
latch.countDown();
});
// await the completion event
latch.await();
Notes:
This requires use of the MongoDB RxJava driver (i.e. classes in the com.mongodb.rx.client namespace ... the org.mongodb::mongodb-driver-rx Maven artifact)
In your question you are invoking collection.find().batchSize() which clearly indicates that you are not currently using the Rx driver (since batchSize cannot be a Rx friendly concept :)
The above code is verified with v1.4.0 of the MongoDB RxJava driver and v1.1.10 of io.reactive::rxjava
Update 1: based on the change to your question (which follows my original answer), you have asked: " I see that all the data is loaded after which the unit testing happens. I think this is because of latch.await()"? I think you are pop[ulating a list from the observable stream and only after the observable is exhausted do you start invoking doTest(). This approach involves (1) streaming results from MongoDB; (2) storing those results in-memory; (3) running doTest() for each stored result. If you really want to stream all-the-way then you should call doTest() from within your observable's subscription. For example:
mydatabase.getCollection("testcollection").find()
.toObservable().subscribe(
document -> {
doTest(process(document));
},
throwable -> {
failures.add(throwable);
},
() -> {
latch.countDown();
});
latch.await();
The above code will invoke doTest() as it receives each document from MongoDB and when the entire observable is exhausted the latch will be decremented and your code will complete.
I'm playing around with implementing my own observables or porting them from other languages for fun and profit.
The problem I've run into is that there's very little info on how to properly test observables or async code in general.
Consider the following test code:
// Create a stream of values emitted every 100 milliseconds
// `interval` uses Timer internally
final Stream<Number> stream =
Streams.interval(100).map(number -> number.intValue() * 10);
ArrayList<Number> expected = new ArrayList<>();
expected.add(0);
expected.add(10);
expected.add(20);
IObserver<Number> observer = new IObserver<Number>() {
public void next(Number x) {
assertEquals(x, expected.get(0));
expected.remove(0);
if(expected.size() == 0) {
stream.unsubscribe(this);
}
}
public void error(Exception e) {}
public void complete() {}
};
stream.subscribe(observer);
As soon as the stream is subscribed to, it emits the first value. onNext is called... And then the test exits successfully.
In JavaScript most test frameworks nowadays provide an optional Promise to the test case that you can call asynchronously on success/failure. Is anything similar available for Java?
Since the execution is asyncronious, you have to wait until is finish. You can just wait for some time in an old fashion way
your_code
wait(1000)
check results.
Or if you use Observables you can use TestSubscriber
In this example you can see how having an async operation we wait until the observer consume all items.
#Test
public void testObservableAsync() throws InterruptedException {
Subscription subscription = Observable.from(numbers)
.doOnNext(increaseTotalItemsEmitted())
.subscribeOn(Schedulers.newThread())
.subscribe(number -> System.out.println("Items emitted:" + total));
System.out.println("I finish before the observable finish. Items emitted:" + total);
new TestSubscriber((Observer) subscription)
.awaitTerminalEvent(100, TimeUnit.MILLISECONDS);
}
You can see more Asynchronous examples here https://github.com/politrons/reactive/blob/master/src/test/java/rx/observables/scheduler/ObservableAsynchronous.java