*** I am sure there are better ways of doing this, but I am starting new on webflux, and still iearning***
I have a requirement where I need to make asynchronous calls to different web services fro one service( It is same end point with differet queryparameters each time, so calling it a different service). There are dynamic number of downstream calls
Service[A] ---> ServiceB[1],ServiceB[2],ServiceB[3],ServiceB[4], etc
Now, when I get the response from each instance of ServiceB[n] , I need to know 'this' response is for which request ( 1 or 2 or 3 or 4 , etc). Reason is tha I need to append thais at the end of request to send as response in specific order. They need to be appended in the same order as they were in the original request. ( so I need to identify which response to its request ). I am geiing a List that has the order in which they were requested.
I have the below code which works functionally. I.e. when I make one request to Service-A ( which makes 4 calls to 4 Servcie B calls)
BUT, When there is parallel requests to Service-A, I see the Flux.merge(asyncRequestList).collectList().block() call does not wait for ALL resposnes and still returns from the method. So I am missing some responses.
public List<ResultSet> invoke(List<ResultSet> requestPayload) {
String endpoint = "https://ourserver.hostname.com/our-service/api/v1/context?reference={servicereference}"
List<ResultSet> responsePayload = requestPayload;
ArrayList<Mono<String>> asyncRequestList = new ArrayList<>();
/*
"Request" is a Pojo with 4 fields
JsonNode request;
String reference;
JsonNode response;
int order;
*/
for (Requests req : requestPayload) {
asyncRequestList.add(
req.getOrder(),
webClient.post()
.uri(new UriTemplate(endpoint).expand(reference))
.body(BodyInserters.fromValue(req.getRequest()))
.retrieve()
.bodyToMono(String.class)
.timeout(Duration.ofSeconds(20)));
// register the callback
asyncRequestList.get(req.getOrder()).subscribe(s2 -> {
logger.info("Got response :: Request Id : {}, for Reference{}",
req.getRequestId(), req.getReference());
responsePayload.get(req.getOrder()).setResponse(mapper.mapFromString(s2));
});
}
//wait for all responses
Flux.merge(asyncRequestList).collectList().block();
logger.info("Got all responses. ");
return responsePayload;
}
Can anyone point me in right direction here ?
and what tools can I use to debug this ( any network monitors, etc)
Related
I'm trying to reactively fetch data from external API using two methods from some Service class.
I'm new to reactive and Spring in general, so it could be a very obvious mistake but I just can't find it
These are the two methods:
public Mono<SomeClass> get(int value) {
return webClient.get()
.uri("/" + value)
.retrieve()
.onRawStatus(HttpStatus.CONFLICT::equals, clientResponse -> {
return Mono.error(new SomeException1("Some message", clientResponse.rawStatusCode()));
})
.onRawStatus(HttpStatus.NOT_FOUND::equals, clientResponse -> {
return requestGeneration(value)
.flatMap(res -> Mono.error(new SomeException1("Some message", clientResponse.rawStatusCode())));
})
.bodyToMono(SomeClass.class)
.retryWhen(Retry.backoff(5, Duration.ofSeconds(8))
.filter(throwable -> throwable instanceof SomeException1));
}
private Mono<Void> requestGeneration(int value) {
return webClient.post()
.uri("/" + value)
.retrieve()
.onRawStatus(HttpStatus.BAD_REQUEST::equals, clientResponse -> {
return Mono.error(new SomeException2("Wrong value", clientResponse.rawStatusCode()));
})
.bodyToMono(Void.class);
}
Baasically what I'm trying to achieve is:
first GET from http://api.examplepage.com/{value}
if that API returns HTTP404 it means I need to first call POST to the same URL, because the data is not yet generated
the second function does the POST call and returns Mono<Void> because it is just HTTP200 or HTTP400 on bad generation seed (i don't need to process the response)
first function (GET call) could also return HTTP429 which means the data is generating right now, so I need to call again after some time period (5-300 seconds) and check if data has been generated already
then after some time it results in HTTP200 with generated data which I want to map to SomeClass and then return mapped data in controller below
#PostMapping("/follow/{value}")
public Mono<ResponseEntity<?>> someFunction(#PathVariable int value) {
return Mono.just(ResponseEntity.ok(service.get(value)));
}
all the code I posted is very simplified to the issues I'm struggling with and doesn't contain some things I think are not important in this question
and now the actual question:
it doesn't actually make the call? i really don't know what is happening
program doesn't enter onRawStatus, even if i change it to onStatus 2xx or whatever other httpstatus and log inside i see nothing as if it doesn't even enter the chains
when i manually call with postman it seems like the program calls have never been made because the GET call returns 404 (the program didn't request to generate data)
the controller only returns "scanAvailable": true, when i expect it to return mapped SomeClass json
// edit
i changed the code to be a full chain as suggested and it didn't solve the problem. all the status are still unreachable (code inside any onStatus nor onRawStatus never executes)
I have three WebClients that look something like this:
WebClient
public Mono<MyObject> getResponseOne() {
return webClient.get()
.uri(URI)
.header("header", header)
.bodyValue(body)
.retrieve()
.bodyToMono(MyObject.class);
}
Then I have a controller which calls multiple WebClients:
Controller
#ResponseBody
#PostMapping("/get")
public Mono<MyObject> processResponse() {
MyObject obj = getResponseOne().toFuture().get();
system.out.println("Got first response");
String str = getResponseTwo().toFuture().get();
system.out.println("Got second response");
//process and send with third WebClient
MyObject newObj = getResponseThree(obj, str).toFuture().get();
//process response from third WebClient and send to fourth WebClient
//return statement
}
When I call /get, the console only prints "Got first response" then it just stops there and anything below doesn't seem to be executing. I'm using Postman to send the request, so it keeps on waiting without getting any response.
This may or may not be relevant, but I'm using blocking calls because I need the both responses to be processed before sending it to a third WebClient, the response from the third WebClient will also go through additional processing before being returned as the response of processResponse().
Solution
I used Mono.zip() like Alex suggested:
#ResponseBody
#PostMapping("/get")
public Mono<MyObject> processResponse() {
//TupleN depends on the amount of Monos you want to process
Mono<Tuple2<MyObject,String>> output = Mono.zip(getResponseOne(),getResponseTwo());
return output.map(result ->{
// getT1() & getT2() is automatically generated by the tuple
MyObject obj = result.getT1();
String str = result.getT2();
getResponseThree(obj, str);
//process and return
});
}
More about Mono.zip()
In reactive API nothing happens until you subscribe. Your method returns Mono and you need to construct a flow combining publishers. There are multiple ways to combine depending on the required logic.
For example, if you need the result of the predecessor you could use flatMap to resolve Mono sequentially
return getResponseOne()
.flatMap(res -> getResponseTwo(res))
.flatMap(res -> getResponseThree(res));
In case calls are independent you could use then
return getResponseOne()
.then(getResponseTwo())
.then(getResponseThree());
You could also execute in parallel using Mono.when(getResponseOne(), getResponseTwo(), getResponseThree()) or Mono.zip(getResponseOne(), getResponseTwo(), getResponseThree()).
There are many other operators but the key here is to construct the flow and return Mono or Flux.
The following method will return a list of Strings with status OK.
#Async
#RequestMapping(path = "/device-data/search/asyncCompletable", method = RequestMethod.GET)
public CompletableFuture<ResponseEntity<?>> getValueAsyncUsingCompletableFuture() {
logger.info("** start getValueAsyncUsingCompletableFuture **");
// Get a future of type String
CompletableFuture<List<String>> futureList = CompletableFuture.supplyAsync(() -> processRequest());
return futureList.thenApply(dataList -> new ResponseEntity<List<String>>(dataList, HttpStatus.OK));
}
I would like to make this method more robust by adding the following features:
1) Since fetching the data will take some time ( processRequest())) than in the meanwhile I would like to return status ACCEPTED so there will be some notification for the user.
2) In case the list is empty then I want to return status NO_CONTENT.
How can I add these enhancements in the same method?
Thank you
Even though this is an #Async function, you're waiting for processRequest() to complete and thenApply() function takes the results and set HttpStatus.NO_CONTENT to ResponseEntity despite the results status.
You return that already processed result list, just like a synchronized invoke. I would rather let the client to decide if the List is empty or not, after an immediate response.
#Async
#RequestMapping(path = "/device-data/search/asyncCompletable", method = RequestMethod.GET)
public CompletableFuture<ResponseEntity<List<String>>> getValueAsyncUsingCompletableFuture() {
return CompletableFuture.supplyAsync(() -> ResponseEntity.accepted().body(processRequest()));
}
In the client code, once the get response via RestTemplate of WebClient wait for the response or continue the rest of the tasks.
CompletableFuture<ResponseEntity<List<String>>> response = ... // The response object, should be CompletableFuture<ResponseEntity<List<String>>> type hence returns from the upstream endpoint.
System.out.println(response.get().getBody()); // 'get()' returns a instance of 'ReponseEntity<List<String>>', that's why 'getBody()' invokes.
Well this CompletableFuture.get() is a blocking function and it would wait the current thread until the response arrives. You can continue without blocking for the response until particular response is required in the code.
I have a simple controller method that calls a spring data repository and returns a few objects.
#RequestMapping(value="/api/learnitemlists", method=RequestMethod.GET, produces=MediaType.APPLICATION_JSON_UTF8_VALUE)
public ResponseEntity<Iterable<LearnItemList>> getLearnItemLists(#RequestParam(value="fromLanguages") Optional<List<String>> fromLanguages,
#RequestParam(value="toLanguage") Optional<String> toLanguage,
#RequestParam("pagenumber") Integer pageNumber,
#RequestParam("pagesize") Integer pageSize) {
LOGGER.debug("start of learnItemLists call");
Page<LearnItemList> lists;
lists = learnItemListRepositoryCustom.findBasedOnLanguage(fromLanguages,toLanguage,new PageRequest(pageNumber,pageSize));
HttpHeaders headers = new HttpHeaders();
headers.add("X-total-count", Long.toString(lists.getTotalElements()));
headers.add("Access-Control-Expose-Headers", "X-total-count");
ResponseEntity<Iterable<LearnItemList>> result = new ResponseEntity<Iterable<LearnItemList>>(lists,headers,HttpStatus.OK);
LOGGER.debug("end of learnItemLists call");
return result;
}
I logged the beginning and the end of the method call:
22:06:11.914 - 22:06:12.541
So the actual retrieval of objects from the database is well under 1 second. However, the full request took about 2.68 when trying in a browser (integration tests show similar performance).
I can't help but think that something is off. Can serialization into JSON (I'm using Jackson) take this long? The whole JSON response is about 1 kb...
So is this normal (I doubt it), and if not, what steps should I take to find out the cause?
I have vertx server application where I am getting single client requests and from the server, I need to make two blocking calls. For instance, one call to back-end system A and another call to back-end system B. I am looking to make two concurrent calls to both the systems. I need to wait for the responses from both the calls and then merge two data from both the calls and then send the response back to client. I am unable to figure out how to do this in worker verticle.
Could anyone recommend what would be the best approach in vertx?
This sounds like a good use case for Promises. Give the module vertx-promises a try.
create a CompositeFuture from your launched Futures and handle it normally.
public Future<JsonArray> getEntitiesByIndFields(String keyspace, String entidad, String field1, String field2) {
Promise<JsonArray> p = Promise.promise();
// launch in parallel
Future<JsonArray> f1 = getEntitiesByIndField1(keyspace, entidad, field1);
Future<JsonArray> f2 = getEntitiesByIndField2(keyspace, entidad, field2);
CompositeFuture.all(f1, f2).setHandler(done ->
{
if (done.failed()) {
p.fail(done.cause());
return;
}
List<JsonArray> ja = done.result().list();
JsonArray finalarray = ja.get(0);
ja.get(1).forEach(jo ->
{ // add one by one, don't duplicate ids
long id = ((JsonObject) jo).getLong("id");
if (!containsKey(finalarray, id)) {
finalarray.add(jo);
}
});
;
p.complete(finalarray); // send union of founds
});
return p.future();
}