We Keep Coding

spring boot webflux query data in continuous flux

having the following code that produces a continuous flux of random strings :
#RestController
public class WebFluxController {
private final Random random = new Random();

 
#CrossOrigin
#GetMapping(value = "/documents")
public Flux getDocuments() {
 
return Flux.interval(Duration.ofSeconds(1))
 .map(x -> "document-" +
random.nextDouble());
}
}
...how can I replace the random with a query to the database that will return a field of the last record, something like :
#RestController
public class WebFluxController {
#Autowired
private ReactiveDocumentRepository reactiveDocumentRepository;
#CrossOrigin
 #GetMapping(value = "/documents")
public Flux getDocuments() {
 
return Flux.interval(Duration.ofSeconds(1))
 .map(x -> "document-" +
reactiveDocumentRepository.findLastDocument().map(d->d.getDescription);
}
}
}
...where
reactiveDocumentRepository.findLastDocument() returns a mono containing last document inserted in the db?
In other words, I want that query to be ran continuously over the database and publish last inserted record all the time

In reactive you need to build a flow using operators that will be evaluated when downstream (in your case webflux) subscribes to the flow. Result is not immediately available and you can't just concatenate it with a string. reactiveDocumentRepository.findLastDocument() is reactive and you need to use flatMap instead of map.
public Flux getDocuments() {
return Flux.interval(Duration.ofSeconds(1))
.flatMap(x ->
reactiveDocumentRepository.findLastDocument()
.map(d -> "document-" + d.getDescription)
);
}

How to keep an app reactive when needing to subscribe?

I have a fully reactive web app that aggregates the information from two other backend-services.
Incoming request -> sends request to service A and B -> aggregates responses -> response is emitted.
pseudocode:
public Mono<ResponseEntity<List<String>>> getValues() {
return Mono.zip(getValuesA(), getValuesB(),
(a, b) -> Stream.concat(a.stream(), b.stream()).collect(Collectors.toList()))
.map(result -> ResponseEntity.ok(result));
}
public Mono<String> getValuesA() {
return webClient.get()
.uri(uriA)
.retrieve()
.bodyToMono(new ParameterizedTypeReference<>() {});
}
// getValuesB same as A, but with uriB.
Because of the high request frequency, I want to bundle requests to the backend-services. I thought using Sinks would be the right way to go. A sink is returned as mono to every requesting party. After a threshold of 10 requests has been exceeded, the request will be handled and the response will be emitted to every sink.
public Mono<ResponseEntity<List<String>>> getValues() {
return Mono.zip(getValuesA(), getValuesB(),
(a, b) -> Stream.concat(a.stream(), b.stream()).collect(Collectors.toList()))
.map(result -> ResponseEntity.ok(result));
}
public Mono<String> getValuesA() {
Sink.One<List<String>> sink = Sinks.one();
queue.add(sink);
if(queue.size() > 10) {
webClient.get()
.uri(uriA)
.retrieve()
.bodyToMono(new ParameterizedTypeReference<>() {})
.subscribe(response -> {
for(Sink.One<List<String>> sinkItem : queue) {
sink.tryEmitValue(response);
}
});
}
return sink.asMono();
}
// getValuesB same as A, but with uriB.
The problem in this code is the 'subscribe' part. As soon as we're subscribing to the webclient's response, it will block the thread. This will only happen in 10% of the requests, but this is already too much for an endpoint that's being called very frequently. What can I do to 'unblock' this part. If using sinks wasn't the best choice, what could have been a better one?
PS. All pseudocode used is NOT production code. It may have many flaws and it is only meant to visualize the problem I'm facing at this moment.

Because of the high request frequency, I want to bundle requests to the backend-services. I thought using Sinks would be the right way to go.
You shouldn't need a sink to do this at all - assuming a Flux as input, you should be able to do this in 3 steps with a standard reactive chain:
Buffer the input with a length of 10, which transforms your Flux<Foo> into a Flux<List<Foo>> where each element is a list of size 10 (or lower than 10 if the flux completes with fewer than 10 remaining elements);
Flatmap to a zipped mono which contains the original list, the "A" web service response given the list, and the "B" web service response given the list;
Implement a method (let's call it expand()) which takes the original list of 10 items, the A service response, and the B service response, and then splits it out into a flux of multiple items. Flatmap to this method.
The end result would be a reactive chain that looked something like:
input.buffer(10)
.flatMap(list -> Mono.zip(Mono.just(list), getResponseFromA(list), getResponseFromB(list)))
.flatMap(response -> expand(response.getT1(), response.getT2(), response.getT3()))

How to parallelize database queries in Spring Flux?

I want to expose aggregated results from a mysql database with a Flux<JSONObject> stream in Spring.
#RestController
public class FluxController {
#GetMapping("/", produces = TEXT_EVENT_STREAM_VALUE)
public Flux<JSONObject> stream() {
return service.getJson();
}
}
#Service
public class DatabaseService {
public List<JSONObject> getJson() {
List<Long> refs = jdbc.queryForList(...);
MapSqlParameterSource params = new MapSqlParameterSource();
params.addValue("refs", refs);
//of course real world sql is much more complex
List<Long, Product> products = jdbc.query(SELECT * from products where ref IN (:refs), params);
List<Long, Item> items = jdbc.query(SELECT * from items where ref IN (:refs), params);
List<Long, Warehouse> warehouses = jdbc.query(SELECT * from warehouses where ref IN (:refs), params);
List<JSONObject> results = new ArrayList<>();
for (Long ref : refs) {
JSONObject json = new JSONObject();
json.put("ref", ref);
json.put("product", products.get(ref));
json.put("item", items.get(ref));
json.put("warehouse", warehouses.get(ref));
results.add(json);
}
return results;
}
Now I want to convert this to a flux, to expose it as an event stream. But how can I parallelize the db lookup and chain it together to a flux?
public Flux<JSONObject> getJsonFlux() {
//I need this as source
List<Long> refs = jdbc.queryForList(...);
return Flux.fromIterable(refs).map(refs -> {
//TODO how to aggregate the different database calls concurrently?
//and then expose each JSONObject one by one into the stream as soon as it is build?
};
}
Sidenote: I know this will still be blocking. But in my real application, I'm applying pagination and chunking, so each chunk will get exposed to the stream when ready.
Then main problem is that I don't know how to parallelize, and then aggregate/merge the results eg in the last flux step.

The idea is to firstly fetch complete list of refs, and then simultaneously fetch Products, Items, and Warehouses - I called this Tuple3 lookups. Then combine each ref with lookups and convert it to JSONObject one by one.
return Mono.fromCallable(jdbc::queryForList) //fetches refs
.subscribeOn(Schedulers.elastic())
.flatMapMany(refList -> { //flatMapMany allows to convert Mono to Flux in flatMap operation
Flux<Tuple3<Map<Long, Product>, Map<Long, Item>, Map<Long, Warehouse>>> lookups = Mono.zip(fetchProducts(refList), fetchItems(refList), fetchWarehouses(refList))
.cache().repeat(); //notice cache - it makes sure that Mono.zip is executed only once, not for each zipWith call
return Flux.fromIterable(refList)
.zipWith(lookups);
}
)
.map(t -> {
Long ref = t.getT1();
Tuple3<Map<Long, Product>, Map<Long, Item>, Map<Long, Warehouse>> lookups = t.getT2();
JSONObject json = new JSONObject();
json.put("ref", ref);
json.put("product", lookups.getT1().get(ref));
json.put("item", lookups.getT2().get(ref));
json.put("warehouse", lookups.getT3().get(ref));
return json;
});
Methods for each database call:
Mono<Map<Long, Product>> fetchProducts(List<Long> refs) {
return Mono.fromCallable(() -> jdbc.query(SELECT * from products where ref IN(:refs),params))
.subscribeOn(Schedulers.elastic());
}
Mono<Map<Long, Item>> fetchItems(List<Long> refs) {
return Mono.fromCallable(() -> jdbc.query(SELECT * from items where ref IN(:refs),params))
.subscribeOn(Schedulers.elastic());
}
Mono<Map<Long, Warehouse>> fetchWarehouses(List<Long> refs) {
return Mono.fromCallable(() -> jdbc.query(SELECT * from warehouses where ref IN(:refs),params))
.subscribeOn(Schedulers.elastic());
}
Why do I need subsribeOn?
I put it because of 2 reasons:
It allows to execute database query on the thread from dedicated
thread pool, which prevents blocking main thread:
https://projectreactor.io/docs/core/release/reference/#faq.wrap-blocking
It allows to truly parallelize Mono.zip. See this one, it's
regarding flatMap, but it's also applicable to zip:
When FlatMap will listen to multiple sources concurrently?
For completeness, the same is possible when using .flatMap() on the zip result. Though I'm not sure if .cache() is still necessary here.
.flatMapMany(refList -> {
Mono.zip(fetchProducts(refList), fetchItems(refList), fetchWarehouses(refList)).cache()
.flatMap(tuple -> Flux.fromIterable(refList).map(refId -> Tuples.of(refId, tuple)));
.map(tuple -> {
String refId = tuple.getT1();
Tuple lookups = tuple.getT2();
}
})

If I understand well you would like to execute queries by passing all refs as parameter.
It will not really be an event stream this way, since it will wait until all queries are finished and all json objects are in memory and just start streaming them after that.
public Flux<JSONObject> getJsonFlux()
{
return Mono.fromCallable(jdbc::queryForList)
.subscribeOn(Schedulers.elastic()) // elastic thread pool meant for blocking IO, you can use a custom one
.flatMap(this::queryEntities)
.map(this::createJsonObjects)
.flatMapMany(Flux::fromIterable);
}
private Mono<Tuple4<List<Long>, List<Product>, List<Item>, List<Warehouse>>> queryEntities(List<Long> refs)
{
Mono<List<Product>> products = Mono.fromCallable(() -> jdbc.queryProducts(refs)).subscribeOn(Schedulers.elastic());
Mono<List<Item>> items = Mono.fromCallable(() -> jdbc.queryItems(refs)).subscribeOn(Schedulers.elastic());
Mono<List<Warehouse>> warehouses = Mono.fromCallable(() -> jdbc.queryWarehouses(refs)).subscribeOn(Schedulers.elastic());
return Mono.zip(Mono.just(refs), products, items, warehouses); // query calls will be concurrent
}
private List<JSONObject> createJsonObjects(Tuple4<List<Long>, List<Product>, List<Item>, List<Warehouse>> tuple)
{
List<Long> refs = tuple.getT1();
List<Product> products = tuple.getT2();
List<Item> items = tuple.getT3();
List<Warehouse> warehouses = tuple.getT4();
List<JSONObject> jsonObjects = new ArrayList<>();
for (Long ref : refs)
{
JSONObject json = new JSONObject();
// build json object here
jsonObjects.add(json);
}
return jsonObjects;
}
The alternative way is to query entities for each ref separately. This way each JSONObject is queried individually and they can interleave in the stream. I'm not sure how the database handles that kind of load. That's something you should consider.
public Flux<JSONObject> getJsonFlux()
{
return Mono.fromCallable(jdbc::queryForList)
.flatMapMany(Flux::fromIterable)
.subscribeOn(Schedulers.elastic()) // elastic thread pool meant for blocking IO, you can use a custom one
.flatMap(this::queryEntities)
.map(this::createJsonObject);
}
private Mono<Tuple4<Long, Product, Item, Warehouse>> queryEntities(Long ref)
{
Mono<Product> product = Mono.fromCallable(() -> jdbc.queryProduct(ref)).subscribeOn(Schedulers.elastic());
Mono<Item> item = Mono.fromCallable(() -> jdbc.queryItem(ref)).subscribeOn(Schedulers.elastic());
Mono<Warehouse> warehouse = Mono.fromCallable(() -> jdbc.queryWarehouse(ref))
.subscribeOn(Schedulers.elastic());
return Mono.zip(Mono.just(ref), product, item, warehouse); // query calls will be concurrent
}
private JSONObject createJsonObject(Tuple4<Long, Product, Item, Warehouse> tuple)
{
Long ref = tuple.getT1();
Product product = tuple.getT2();
Item item = tuple.getT3();
Warehouse warehouse = tuple.getT4();
JSONObject json = new JSONObject();
// build json object here
return json;
}

Get Flux size when Flux is complete

I'm kinda stuck with a trivial task: whenever I query an external API with reactive spring WebClient or query reactive MongoDBRepository, I'd like to log how many entities got through my flux, eg. to log message like "Found n records in the database.". Eg:
return repository.findAll()
.doOnComplete { log.info("Found total n records!") } // how to get the n?
.filter { it.age > 10 }
.distinct { it.name }
TLDR: How to get a flux size (and perhaps it's contents) when it completes?

You can use ConnectableFlux. In your example:
var all = repository.findAll()
.filter { it.age > 10 }
.distinct { it.name }
.publish()
.autoConnect(2)
all.count()
.subscribe {c -> log.info("Found total {} records!", c)}
return all;

By calling the count(). It should emit a Mono when onComplete is observed.

Here was what I did,
AtomicInteger i = new AtomicInteger();
Flux<UserDetails> stringFlux =
Flux.using(() -> stringStream, Flux::fromStream,
Stream::close)
.doOnNext(s -> i.getAndIncrement())
.log()
.map(UserDetails::createUserDetails);
stringFlux
.subscribe(updateUserDetailsService::updateUserDetails);
log.info("number of records: {}", i);

RxJava2 combine multiple observables to make them return single result

How to combine multiple results emmited by observables into one result and emit it once?
I have a Retrofit service:
public interface MyService {
#GET("url")
Observable<UserPostsResult> getUserPosts(#Query("userId") int id);
#GET("url")
Observable<UserPostsResult> getUserPosts(#Query("userId") int id, #Query("page") int pageId);
}
And I have a model:
public class UserPostsResult {
#SerializedName("posts")
List<UserPost> mPosts;
#SerializedName("nextPage")
int mPageId;
}
Also I have ids List<Integer> friendsIds;
My goal is to have a method like this one:
public Observable<Feed> /*or Single<Feed>*/ getFeed(List<Integer> ids) {
...
}
It returns one Observable or Single that does the following:
Combines all getUserPosts(idFromList) to one observable
For each UserPostsResult must do:
if (userPostResult.mPageId > -1)
getUserPosts(currentUserId, userPostResult.mPageId);
And merge this result to the previous userPostResult
Return one single model as result of all operations.
Result class:
public class Feed {
List<UserPost> mAllPostsForEachUser;
}
EDIT (More details):
My client specifications was that I must take from social network user posts with no logging in, no token requesting. So I must parse HTML pages. That's why I have this complex structure.
EDIT (Partial solution)
public Single<List<Post>> getFeed(List<User> users) {
return Observable.fromIterable(users)
.flatMap(user-> mService.getUserPosts(user.getId())
.flatMap(Observable::fromIterable))
.toList()
.doOnSuccess(list -> Collections.sort(list, (o1, o2) ->
Long.compare(o1.getTimestamp(), o2.getTimestamp())
));
}
This solution doesn't include pages problem. Thats why it is only partial solution

There are a number of operators which transform things into other things. fromIterable() will emit each item in the iterable, and flatMap() will convert one type of observable into another type of observable and emit those results.
Observable.fromIterable( friendsIds )
.flatMap( id -> getUserPosts( id ) )
.flatMap( userPostResult -> userPostResult.mPageId
? getUserPosts(currentUserId, userPostResult.mPageId)
: Observable.empty() )
.toList()
.subscribe( posts -> mAllPostsForEachUser = posts);

If you need join two response in one you should use Single.zip
Single.zip(firsSingle.execute(inputParams), secondSingle.execute(inputPrams),
BiFunction<FirstResponse, SecondResponse, ResponseEmitted> { firstResponse, secondResponse ->
//here you put your code
return responseEmmitted
}
}).subscribe({ response -> },{ })