I want to merge 2 responses and return a Flux.
private Flux<Response<List<Company>, Error>> loopGet(List<Entity> registries, Boolean status) {
return Flux.fromIterable(registries)
.flatMap(this::sendGetRequest)
.mergeWith(Mono.just(fetch(status)));
}
This is what I am doing, is working but I would like the merge to wait before calling the Mono.just (fetch (status)).
I'll explain, sendGetRequest returns a Mono that makes an API call and from the result saves things to db. Subsequently the merge goes to call the db with the fetch method, but that data is not updated yet. If I then make the call again, I get the updated data.
You need concatWith and fromCallable to ensure that fetch is called lazily after the get requests are finished.
private Flux<Response<List<Company>, Error>> loopGet(List<Entity> registries, Boolean status) {
return Flux.fromIterable(registries)
.flatMap(this::sendGetRequest)
.concatWith(Mono.fromCallable(() -> fetch(status)));
}
Related
I am getting started with Spring Flux
When i try to make a call to the repo based on the repsonse of first call. The call to the repository is not made.
Looking up DocParts in a repo by docpartId and just building a new Docpart and writing over existing for now for test.
(Code for db handling in repo is removed for simplicity as the return values are the same as in this example)
The newly created docParts get flatMapped and should be upserted into the database but the upsert call is never made as I have debugged it and also cosnole logged from it.
First call to repo is made but second call is never made
IE it gets to
return docPartRepo.getDocPartById(docPartId)
.map(docPart-> DocPart.builder())
.build())
.flatMap(docPart -> docPartRepo.upsert(docPartId, docPart) // This upsert is never called
.doOnSuccess(xx -> LOGGER.info("Updated docRepo with docPart")));
Repo Code example
public Mono<DocPart> getDocPartById(Long docPartId) {
return Mono.just(DocPart.builder().
.build())
}
public Mono<Void> upsert(Long id, DocPart) {
System.out.println("In Upsert")
//Make db call
return Mono.empty();
}
If I test the code to just call upsert it is made OK but the doSuccess is not called
final Mono<DocPart> docPart = docPartRepo.getDocPartById(docPartId)
.map(docPart-> DocPart.builder())
.build())
final DocPart docPart = docPart.block();
docPartRepo.upsert(docPartId, docPart)
.doOnSuccess(xxx -> System.out.println("Updated db")); // Now the upsert is made but not doOnSuccess
Not sure if it is connected to calling the same repo nested or if anybody can give som inputto what might be hindering the call to upsert in flatmap not to be made
I created the below method to find an Analysis object, update the results field on it and then lastly save the result in the database but not wait for a return.
public void updateAnalysisWithResults(String uuidString, String results) {
findByUUID(uuidString).subscribe(analysis -> {
analysis.setResults(results);
computeSCARepository.save(analysis).subscribe();
});
}
This feels poorly written to subscribe within a subscribe.
Is this a bad practice?
Is there a better way to write this?
UPDATE:
entry point
#PatchMapping("compute/{uuid}/results")
public Mono<Void> patchAnalysisWithResults(#PathVariable String uuid, #RequestBody String results) {
return computeSCAService.updateAnalysisWithResults(uuid,results);
}
public Mono<Void> updateAnalysisWithResults(String uuidString, String results) {
// findByUUID(uuidString).subscribe(analysis -> {
// analysis.setResults(results);
// computeSCARepository.save(analysis).subscribe();
// });
return findByUUID(uuidString)
.doOnNext(analysis -> analysis.setResults(results))
.doOnNext(computeSCARepository::save)
.then();
}
Why it is not working is because you have misunderstood what doOnNext does.
Lets start from the beginning.
A Flux or Mono are producers, they produce items. Your application produces things to the calling client, hence it should always return either a Mono or a Flux. If you don't want to return anything you should return a Mono<Void>.
When the client subscribes to your application what reactor will do is call all operators in the opposite direction until it finds a producer. This is what is called the assembly phase. If all your operators don't chain together you are what i call breaking the reactive chain.
When you break the chain, the things broken from the chain wont be executed.
If we look at your example but in a more exploded version:
#Test
void brokenChainTest() {
updateAnalysisWithResults("12345", "Foo").subscribe();
}
public Mono<Void> updateAnalysisWithResults(String uuidString, String results) {
return findByUUID(uuidString)
.doOnNext(analysis -> analysis.setValue(results))
.doOnNext(this::save)
.then();
}
private Mono<Data> save(Data data) {
return Mono.fromCallable(() -> {
System.out.println("Will not print");
return data;
});
}
private Mono<Data> findByUUID(String uuidString) {
return Mono.just(new Data());
}
private static class Data {
private String value;
public void setValue(String value) {
this.value = value;
}
}
in the above example save is a callable function that will return a producer. But if we run the above function you will notice that the print will never be executed.
This has to do with the usage of doOnNext. If we read the docs for it it says:
Add behavior triggered when the Mono emits a data successfully.
The Consumer is executed first, then the onNext signal is propagated downstream.
doOnNext takes a Consumer that returns void. And if we look at doOnNext we see that the function description looks as follows:
public final Mono<T> doOnNext(Consumer<? super T> onNext)`
THis means that it takes in a consumer that is a T or extends a T and it returns a Mono<T>. So to keep a long explanation short, you can see that it consumes something but also returns the same something.
What this means is that this usually used for what is called side effects basically for something that is done on the side that does not hinder the current flow. One of those things could for instance logging. Logging is one of those things that would consume for instance a string and log it, while we want to keep the string flowing down our program. Or maybe we we want to increment a number on the side. Or modify some state somewhere. You can read all about side effects here.
you can of think of it visually this way:
_____ side effect (for instance logging)
/
___/______ main reactive flow
That's why your first doOnNext setter works, because you are modifying a state on the side, you are setting the value on your class hence modifying the state of your class to have a value.
The second statement on the other hand, the save, does not get executed. You see that function is actually returning something we need to take care of.
This is what it looks like:
save
_____
/ \ < Broken return
___/ ____ no main reactive flow
all we have to do is actually change one single line:
// From
.doOnNext(this::save)
// To
.flatMap(this::save)
flatMap takes whatever is in the Mono, and then we can use that to execute something and then return a "new" something.
So our flow (with flatMap) now looks like this:
setValue() save()
______ _____
/ / \
__/____________/ \______ return to client
So with the use of flatMap we are now saving and returning whatever was returned from that function triggering the rest of the chain.
If you then choose to ignore whatever is returned from the flatMap its completely correct to do as you have done to call then which will
Return a Mono which only replays complete and error signals from this
The general rule is, in a fully reactive application, you should never block.
And you generally don't subscribe unless your application is the final consumer. Which means if your application started the request, then you are the consumerof something else so you subscribe. If a webpage starts off the request, then they are the final consumer and they are subscribing.
If you are subscribing in your application that is producing data its like you are running a bakery and eating your baked breads at the same time.
don't do that, its bad for business :D
Subscribe inside a subscribe is not a good practise. You can use flatMap operator to solve this problem.
public void updateAnalysisWithResults(String uuidString, String results) {
findByUUID(uuidString).flatMap(analysis -> {
analysis.setResults(results);
return computeSCARepository.save(analysis);
}).subscribe();
}
So my application would call below expensive HTTP service multiple times (simultaneously by multiple threads with same as well as different Ids for every client request to my application).
Mono<Foo> response = myService.fetch(id);
I would like to cache the response (in-memory) for few hours, and then only on next client request make only one call to refresh the cache.
Approach 1:
Mono<Foo> cachedResponse = Mono.empty();
public Mono<Foo> get(String id){
return cachedResponse.switchIfEmpty(Mono.defer(()->
{
cachedResponse = myService.fetch(id).cache(Duration.ofHours(4));
return cachedResponse;
}));
}
is following approach OK? Specifically since multiple threads could call get method with same id. Also, when the cache is invalidated after 4 hours, would it make cachedResponse Mono empty for switchIfEmpty to work correctly?
Approach 2:
I could use some caching solution to store cache for few hours. e.g.
Foo getFromCacheSolution(String id);
and then,
public Mono<Foo> get(String id){
Foo cachedFoo = getFromCacheSolution(id);
if(cachedFoo != null){
return Mono.just(cachedFoo);
}
else{
return myService.fetch(id).doOnNext(value->storeToCacheSolution(id, value)); //line 7
}
}
The problem with this solution is that line 7 would be called multiple times resulting in multiple calls to expensive fetch service (for example if 3 threads enter into get method with id 123 and cachedFoo is null). Making method synchronized may not help as line 7 would complete instantaneously.
One work-around would be to store Mono in the cache solution instead of Foo (not sure if that's a good idea or not):
Mono<Foo> getFromCacheSolution(String id); //returns cached or empty Mono
and then,
public Mono<Foo> get(String id){
return getFromCacheSolution(id).switchIfEmpty(Mono.defer(()->
{
cachedResponse = myService.fetch(id).doOnNext(value->storeToCacheSolution(id, value));
return cachedResponse;
}));
}
Any recommendations or better alternatives?
Your question consists of two parts: about caching and about exclusive locking for calls with same parameters.
Caching.
Your second approach is good for in-memory cache. Alternatively you could use CacheMono from the reactor-extra
Mono<Foo> myFoo =
CacheMono.lookup(key -> Mono.justOrEmpty(myCache.getIfPresent(key))
.map(Signal::next), id)
.onCacheMissResume(() -> myService.fetch(id))
.andWriteWith((key, signal) -> Mono.fromRunnable(() ->
Optional.ofNullable(signal.get())
.ifPresent(value -> myCache.put(key, value))));
Exclusive locking for calls with same parameters.
Usually we should avoid any locking in the reactive world. But if you really need it, your lock should be nonblocking. I don't know any library, but you could find some ideas and links with examples in this question thread
Idea
I have a processing method which takes in a list of items and processes them asynchronously using external web service. The process steps also persist data while processing. At the end of whole process, I want to persist the whole process along with each processed results as well.
Problem
I convert each item in the list into CompletableFuture and run a processing task on them, and put them back into an array of futures. Now using its .ofAll method (in sequence method) to complete future when all the submitted tasks are completed and return another CompletableFuture which holds the result.
When I want to get that result, I call .whenComplete(..), and would want to set the returned result into my entity as data, and then persist to the database, however the repository save call just does nothing and continues threads just continue running, it's not going past the repository save call.
#Transactional
public void process(List<Item> items) {
List<Item> savedItems = itemRepository.save(items);
final Process process = createNewProcess();
final List<CompletableFuture<ProcessData>> futures = savedItems.stream()
.map(item -> CompletableFuture.supplyAsync(() -> doProcess(item, process), executor))
.collect(Collectors.toList());
sequence(futures).whenComplete((data, throwable) -> {
process.setData(data);
processRepository.save(process); // <-- transaction lost?
log.debug("Process DONE"); // <-- never reached
});
}
Sequence method
private static <T> CompletableFuture<List<T>> sequence(List<CompletableFuture<T>> futures) {
CompletableFuture<Void> allDoneFuture =
CompletableFuture.allOf(futures.toArray(new CompletableFuture[futures.size()]));
return allDoneFuture.thenApply(v ->
futures.stream().map(CompletableFuture::join).collect(Collectors.toList())
);
}
What is happening? Why is the persist call not passing. Is the thread that started the transaction not able to commit the transaction or where does it get lost? All the processed data returns fine and is all good. I've tried different transaction strategies, but how is it possible to control which thread is gonna finish the transaction, if it's the case?
Any advice?
The reason of your problem is, as said above, that the transaction ends
when the return of method process(..) is reached.
What you can do, is create the transaction manually, that gives you full
control over when it starts and ends.
Remove #Transactional
Autowire the TransactionManager then in process(..) :
TransactionDefinition txDef = new DefaultTransactionDefinition();
TransactionStatus txStatus = transactionManager.getTransaction(txDef);
try {
//do your stuff here like
doWhateverAsync().then(transactionManager.commit(txStatus);)
} catch (Exception e) {
transactionManager.rollback(txStatus);
throw e;
}
In case of Spring Boot Application , you need following configurations.
The main application method should be annotated with #EnableAsync.
#Async annotation should be on the top of method having #Transactional annotation. This is necessary to indicate processing will be taking place in child thread.
Is there a way to convert Mono objects to java Pojo?
I have a web client connecting to 3rd party REST service and instead of returning Mono I have to extract that object and interrogate it.
All the examples I have found return Mono<Pojo> but I have to get the Pojo itself. Currently, I am doing it by calling block() on Pojo but is there a better way to avoid block?
The issue with the block is that after few runs it starts throwing some error like block Terminated with error.
public MyPojo getPojo(){
return myWebClient.get()
.uri(generateUrl())
.headers(createHttpHeaders(headersMap))
.exchange()
.flatMap(evaluateResponseStatus())
.block();
}
private Function<ClientResponse, Mono<? extends MyPojo>> evaluateResponseStatus() {
return response -> {
if (response.statusCode() == HttpStatus.OK) {
return response.bodyToMono(MyPojo.class);
}
if (webClientUtils.isError(response.statusCode())) {
throw myHttpException(response);
// This invokes my exceptionAdvice
// but after few runs its ignored and 500 error is returned.
}
return Mono.empty();
};
}
It's not a good idea to block to operate on value in a reactive stream. Project Reactor offers you a selection of operators for you to handle the objects within a stream.
In your case, you can write getPojo() method like:
public Mono<MyPojo> getPojo() {
return myWebClient.get()
.uri(generateUrl())
.headers(createHttpHeaders(headersMap))
.retrieve()
.onStatus(status -> webClientUtils.isError(status),
response -> Mono.error(myHttpException(response))
.bodyToMono(MyPojo.class)
}
Note that using onStatus method, we replaced the whole evaluateResponseStatus method in your example.
You would use this method like the following:
// some method
...
getPojo()
.map(pojo -> /* do something with the pojo instance */)
...
I strongly advise you to look into Transforming an existing sequence in Project Reactor docs.
Since Webclient.block() is not recommended, the other way to retrieve the values from incoming httpresponse is to create a POJO in the calling application having the required fields. Then once the Mono is received, use Mono.subscribe(), within subscribe add a lambda function, with input say x, to retrieve the individual fields using the x.getters(). The values could be printed on console or assigned to a local var for further processing. This helps in two ways:-
Avoid the dreaded .block()
Keep the call Asynchronous when pulling large volumes of data.
This is one of many other ways to achieve the desired outcome.