We Keep Coding

Spring Reactive. How wait for all monos to finish?

I have the following code where I call external APIs via webclient and return Mono.
I need to execute some logic when I receive data. And after all, requests are processed, execute one logic for all gathered data. I can collect all Monos and put them to flux and then execute some logic at the end. But I have serviceName filed which is accessible only in the loop, so I need to execute logic for mono in loop and here I'm stuck and don't know how to wait for all data to complete and do it in a reactive way.
#Scheduled(fixedDelay = 50000)
public void refreshSwaggerConfigurations() {
log.debug("Starting Service Definition Context refresh");
List<SwaggerServiceData> allServicesApi = new ArrayList<>();
swaggerProperties.getUrls().forEach((serviceName, serviceSwaggerUrl) -> {
log.debug("Attempting service definition refresh for Service : {} ", serviceName);
Mono<SwaggerServiceData> swaggerData = getSwaggerDefinitionForAPI(serviceName,
serviceSwaggerUrl);
swaggerData.subscribe(swaggerServiceData -> {
if (swaggerServiceData != null) {
allServicesApi.add(swaggerServiceData);
String content = getJSON(swaggerServiceData);
definitionContext.addServiceDefinition(serviceName, content);
} else {
log.error("Skipping service id : {} Error : Could not get Swagger definition from API ",
serviceName);
}
});
});
//I need to wait here for all monos to complete and after that proceed for All gathered data...
//Now it's empty And I know why, just don't know how to make it.
Optional<SwaggerServiceData> swaggerAllServicesData = getAllServicesApiSwagger(allServicesApi);
if (swaggerAllServicesData.isPresent()) {
String allApiContent = getJSON(swaggerAllServicesData.get());
definitionContext.addServiceDefinition("All", allApiContent);
}
}
private Mono<SwaggerServiceData> getSwaggerDefinitionForAPI(String serviceName, String url) {
log.debug("Accessing the SwaggerDefinition JSON for Service : {} : URL : {} ", serviceName,
url);
Mono<SwaggerServiceData> swaggerServiceDataMono = webClient.get()
.uri(url)
.exchangeToMono(clientResponse -> clientResponse.bodyToMono(SwaggerServiceData.class));
return swaggerServiceDataMono;
}

I would add a temporary class to group data and serivce name :
record SwaggerService(SwaggerServiceData swaggerServiceData, String serviceName) {
boolean hasData() {
return swaggerServiceData != null;
}
}
And then change your pipeline :
Flux.fromStream(swaggerProperties.getUrls().entrySet().stream())
.flatMap((e) -> {
Mono<SwaggerServiceData> swaggerDefinitionForAPI = getSwaggerDefinitionForAPI(e.getKey(),
e.getValue());
return swaggerDefinitionForAPI.map(swaggerServiceData -> new SwaggerService(swaggerServiceData, e.getKey()));
})
.filter(SwaggerService::hasData)
.map(swaggerService -> {
String content = getJSON(swaggerService.swaggerServiceData());
definitionContext.addServiceDefinition(swaggerService.serviceName(), content);
return swaggerService.swaggerServiceData();
})
// here we will collect all datas and they will be emmited as single Mono with list of SwaggerServiceData
.collectList()
.map(this::getAllServicesApiSwagger)
.filter(Optional::isPresent)
.map(Optional::get)
.subscribe(e -> {
String allApiContent = getJSON(e);
definitionContext.addServiceDefinition("All", allApiContent);
});
This does not deal with logging error when SwaggerServiceData is null but you can further change it if you want. Also I assume that DefinitionContext is thread safe.
Solution with error logging (using flatMap and Mono.empty()) :
Flux.fromStream(swaggerProperties.getUrls().entrySet().stream())
.flatMap((e) -> {
Mono<SwaggerServiceData> swaggerDefinitionForAPI = getSwaggerDefinitionForAPI(e.getKey(),
e.getValue());
return swaggerDefinitionForAPI
.flatMap(swaggerServiceData -> {
if(swaggerServiceData != null) {
return Mono.just(new SwaggerService(swaggerServiceData, e.getKey()));
} else {
log.error("Skipping service id : {} Error : Could not get Swagger definition from API ",
e.getKey());
return Mono.empty();
}
});
})
.map(swaggerService -> {
String content = getJSON(swaggerService.swaggerServiceData());
definitionContext.addServiceDefinition(swaggerService.serviceName(), content);
return swaggerService.swaggerServiceData();
}).collectList()
.map(this::getAllServicesApiSwagger)
.filter(Optional::isPresent)
.map(Optional::get)
.subscribe(e -> {
String allApiContent = getJSON(e);
definitionContext.addServiceDefinition("All", allApiContent);
});
You can also wrap those lambads into some meaningful methods to improve readibility.

How to merge multiple vertx web client responses

I am new to vertx and async programming.
I have 2 verticles communicating via an event bus as follows:
//API Verticle
public class SearchAPIVerticle extends AbstractVerticle {
public static final String GET_USEARCH_DOCS = "get.usearch.docs";
#Autowired
private Integer defaultPort;
private void sendSearchRequest(RoutingContext routingContext) {
final JsonObject requestMessage = routingContext.getBodyAsJson();
final EventBus eventBus = vertx.eventBus();
eventBus.request(GET_USEARCH_DOCS, requestMessage, reply -> {
if (reply.succeeded()) {
Logger.info("Search Result = " + reply.result().body());
routingContext.response()
.putHeader("content-type", "application/json")
.setStatusCode(200)
.end((String) reply.result().body());
} else {
Logger.info("Document Search Request cannot be processed");
routingContext.response()
.setStatusCode(500)
.end();
}
});
}
#Override
public void start() throws Exception {
Logger.info("Starting the Gateway service (Event Sender) verticle");
// Create a Router
Router router = Router.router(vertx);
//Added bodyhandler so we can process json messages via the event bus
router.route().handler(BodyHandler.create());
// Mount the handler for incoming requests
// Find documents
router.post("/api/search/docs/*").handler(this::sendSearchRequest);
// Create an HTTP Server using default options
HttpServer server = vertx.createHttpServer();
// Handle every request using the router
server.requestHandler(router)
//start listening on port 8083
.listen(config().getInteger("http.port", 8083)).onSuccess(msg -> {
Logger.info("*************** Search Gateway Server started on "
+ server.actualPort() + " *************");
});
}
#Override
public void stop(){
//house keeping
}
}
//Below is the target verticle should be making the multiple web client call and merging the responses
.
#Component
public class SolrCloudVerticle extends AbstractVerticle {
public static final String GET_USEARCH_DOCS = "get.usearch.docs";
#Autowired
private SearchRepository searchRepositoryService;
#Override
public void start() throws Exception {
Logger.info("Starting the Solr Cloud Search Service (Event Consumer) verticle");
super.start();
ConfigStoreOptions fileStore = new ConfigStoreOptions().setType("file")
.setConfig(new JsonObject().put("path", "conf/config.json"));
ConfigRetrieverOptions configRetrieverOptions = new ConfigRetrieverOptions()
.addStore(fileStore);
ConfigRetriever configRetriever = ConfigRetriever.create(vertx, configRetrieverOptions);
configRetriever.getConfig(ar -> {
if (ar.succeeded()) {
JsonObject configJson = ar.result();
EventBus eventBus = vertx.eventBus();
eventBus.<JsonObject>consumer(GET_USEARCH_DOCS).handler(getDocumentService(searchRepositoryService, configJson));
Logger.info("Completed search service event processing");
} else {
Logger.error("Failed to retrieve the config");
}
});
}
private Handler<Message<JsonObject>> getDocumentService(SearchRepository searchRepositoryService, JsonObject configJson) {
return requestMessage -> vertx.<String>executeBlocking(future -> {
try {
//I need to incorporate the logic here that adds futures to list and composes the compositefuture
/*
//Below is my logic to populate the future list
WebClient client = WebClient.create(vertx);
List<Future> futureList = new ArrayList<>();
for (Object collection : searchRepositoryService.findAllCollections(configJson).getJsonArray(SOLR_CLOUD_COLLECTION).getList()) {
Future<String> future1 = client.post(8983, "127.0.0.1", "/solr/" + collection + "/query")
.expect(ResponsePredicate.SC_OK)
.sendJsonObject(requestMessage.body())
.map(HttpResponse::bodyAsString).recover(error -> {
System.out.println(error.getMessage());
return Future.succeededFuture();
});
futureList.add(future1);
}
//Below is the CompositeFuture logic, but the logic and construct does not make sense to me. What goes as first and second argument of executeBlocking method
/*CompositeFuture.join(futureList)
.onSuccess(result -> {
result.list().forEach( x -> {
if(x != null){
requestMessage.reply(result.result());
}
}
);
})
.onFailure(error -> {
System.out.println("We should not fail");
})
*/
future.complete("DAO returns a Json String");
} catch (Exception e) {
future.fail(e);
}
}, result -> {
if (result.succeeded()) {
requestMessage.reply(result.result());
} else {
requestMessage.reply(result.cause()
.toString());
}
});
}
}
I was able to use the org.springframework.web.reactive.function.client.WebClient calls to compose my search result from multiple web client calls, as against using Future<io.vertx.ext.web.client.WebClient> with CompositeFuture.
I was trying to avoid mixing Springboot and Vertx, but unfortunately Vertx CompositeFuture did not work here:
//This method supplies the parameter for the future.complete(..) line in getDocumentService(SearchRepository,JsonObject)
private List<JsonObject> findByQueryParamsAndDataSources(SearchRepository searchRepositoryService,
JsonObject configJson,
JsonObject requestMessage)
throws SolrServerException, IOException {
List<JsonObject> searchResultList = new ArrayList<>();
for (Object collection : searchRepositoryService.findAllCollections(configJson).getJsonArray(SOLR_CLOUD_COLLECTION).getList()) {
searchResultList.add(new JsonObject(doSearchPerCollection(collection.toString(), requestMessage.toString())));
}
return aggregateMultiCollectionSearchResults(searchResultList);
}
public String doSearchPerCollection(String collection, String message) {
org.springframework.web.reactive.function.client.WebClient client =
org.springframework.web.reactive.function.client.WebClient.create();
return client.post()
.uri("http://127.0.0.1:8983/solr/" + collection + "/query")
.contentType(MediaType.APPLICATION_JSON)
.accept(MediaType.APPLICATION_JSON)
.body(BodyInserters.fromValue(message.toString()))
.retrieve()
.bodyToMono(String.class)
.block();
}
private List<JsonObject> aggregateMultiCollectionSearchResults(List<JsonObject> searchList){
//TODO: Search result aggregation
return searchList;
}
My use case is the second verticle should make multiple vertx web client calls and should combine the responses.
If an API call falls, I want to log the error and still continue processing and merging responses from other calls.
Please, any help on how my code above could be adaptable to handle the use case?
I am looking at vertx CompositeFuture, but no headway or useful example seen yet!

What you are looking for can done with Future coordination with a little bit of additional handling:
CompositeFuture.join(future1, future2, future3).onComplete(ar -> {
if (ar.succeeded()) {
// All succeeded
} else {
// All completed and at least one failed
}
});
The join composition waits until all futures are completed, either with a success or a failure.
CompositeFuture.join
takes several futures arguments (up to 6) and returns a future that is succeeded when all the futures are succeeded, and failed when all the futures are completed and at least one of them is failed
Using join you will wait for all Futures to complete, the issue is that if one of them fails you will not be able to obtain response from others as CompositeFuture will be failed. To avoid this you should add Future<T> recover(Function<Throwable, Future<T>> mapper) on each of your Futures in which you should log the error and pass an empty response so that the future does not fail.
Here is short example:
Future<String> response1 = client.post(8887, "localhost", "work").expect(ResponsePredicate.SC_OK).send()
.map(HttpResponse::bodyAsString).recover(error -> {
System.out.println(error.getMessage());
return Future.succeededFuture();
});
Future<String> response2 = client.post(8887, "localhost", "error").expect(ResponsePredicate.SC_OK).send()
map(HttpResponse::bodyAsString).recover(error -> {
System.out.println(error.getMessage());
return Future.succeededFuture();
});
CompositeFuture.join(response2, response1)
.onSuccess(result -> {
result.list().forEach(x -> {
if(x != null) {
System.out.println(x);
}
});
})
.onFailure(error -> {
System.out.println("We should not fail");
});
Edit 1:
Limit for CompositeFuture.join(Future...) is 6 Futures, in the case you need more you can use: CompositeFuture.join(Arrays.asList(future1, future2, future3)); where you can pass unlimited number of futures.

Change request body in ServerHttpRequestDecorator using AbstractGatewayFilterFactory - Webflux

Spring Boot Version: 2.5.1,
Spring Cloud Version: 2020.0.3
Hello guys !!!
I need your help ...
My question is that I can't modify the request body in spring gateway. Follow:
I have a MobileGatewayFilterFactory class that extends from AbstractGatewayFilterFactory where the apply method returns a custom filter: MobileGatewayFilter.
#Component
class MobileGatewayFilterFactory :
AbstractGatewayFilterFactory<MobileGatewayFilterFactory.Config>(Config::class.java),
Ordered {
override fun apply(config: Config): GatewayFilter {
logger.info { "Loading MobileGatewayFilter with config ${config.className}, ${config.execution}, ${config.custom}" }
return MobileGatewayFilter(config)
}
override fun getOrder(): Int {
return NettyWriteResponseFilter.WRITE_RESPONSE_FILTER_ORDER - 1
}
data class Config(
val className: String,
val execution: String,
val custom: String?
)
}
So, inside the MobileGatewayFilter class I implement the business rules to determine which filter is running: PRE or POST filter. This is done in the filter method of the MobileGatewayFilter class where there is a condition to determine the type of decoration being executed, using reflection. If it is a request, the ServerHttpRequestDecorator is executed and a ServerHttpResponseDecorator otherwise.
class MobileGatewayFilter(private val config: MobileGatewayFilterFactory.Config) : GatewayFilter, Ordered {
override fun filter(exchange: ServerWebExchange, chain: GatewayFilterChain): Mono<Void> {
return when (config.execution) {
"PRE" -> chain.filter(exchange.mutate().request(decoratorRequest(exchange)).build())
"POST" -> chain.filter(exchange.mutate().response(decoratorResponse(exchange)).build())
else -> chain.filter(exchange)
}
}
override fun getOrder(): Int {
return NettyWriteResponseFilter.WRITE_RESPONSE_FILTER_ORDER - 1
}
private fun decoratorResponse(exchange: ServerWebExchange): ServerHttpResponse {
val aClass = Class.forName(config.className)
val obj = aClass.getConstructor(ServerHttpResponse::class.java, MobileGatewayFilterFactory.Config::class.java)
return obj.newInstance(exchange.response, config) as ServerHttpResponseDecorator
}
private fun decoratorRequest(exchange: ServerWebExchange): ServerHttpRequest {
val aClass = Class.forName(config.className)
val obj = aClass.getConstructor(ServerHttpRequest::class.java, MobileGatewayFilterFactory.Config::class.java)
return obj.newInstance(exchange.request, config) as ServerHttpRequestDecorator
}
}
Furthermore, I have a CustomerDataBodyDecorator that extends the ServerHttpRequestDecorator and overrides the getBody method. The getBody method is where the request body must be modified.
class CustomerDataBodyDecorator(
private val exchange: ServerHttpRequest,
private val config: MobileGatewayFilterFactory.Config
) : ServerHttpRequestDecorator(exchange) {
override fun getBody(): Flux<DataBuffer> {
logger.info { "getBody chamado ..." }
val body: Flux<DataBuffer> = exchange.body
var requestData = ""
body.subscribe {
val content = ByteArray(it.readableByteCount())
it.read(content)
DataBufferUtils.release(it)
requestData = String(content, Charset.forName("UTF-8"))
logger.info { "Request: $requestData" }
}
val factory = DefaultDataBufferFactory()
val buffer = factory.wrap(requestData.toByteArray())
return Flux.just(buffer)
}
}
However, the above code doesn't work because the return is executed first with empty requestData and after subscribe method is executed. I know that in Webflux the subscribe method is necessary to indicate to the publisher the information consumption needs
application.yml
id: opengw-mobile-simulation
uri: ${custom.resources.opengw}
predicates:
- Path=/opengw/v1/mobile/simulation
filters:
- name: Mobile
args:
className: br.com.decorator.CustomerDataBodyDecorator
execution: PRE
custom: ${custom.resources.customer}
- RewritePath=/opengw/v1/(?<segment>/?.*), /$\{segment}
I read several topics here but I couldn't find a solution that worked.
How can I read and then modify the request body of the Flux object in this scenario?

Apache Camel Dynamic Routing Flow On Exception

I have successfully created few routes overriding configure() method of RouteBuilder. A for loop is used to generate routes on application startup, for eg:
Route1: from("direct:Route1").to("netty-http:http://localhost:8080/route1)
Route2: from("direct:Route2").to("netty-http:http://localhost:8081/route2)
Route3: from("direct:Route3").to("netty-http:http://localhost:8082/route3)
Route4: from("direct:Route4").to("netty-http:http://localhost:8083/route4)
Route5: from("direct:Route5").to("netty-http:http://localhost:8084/route5)
for (endpoint in endpoints.iterator()) {
from("direct:" + endpoint.getEndpointRouteName())
.process(getProcessor(endpoint.getEndpointProcessor(), endpoint.getEndpointRouteName(), objectMapper))
.setHeader(Exchange.HTTP_METHOD, simple(endpoint.getEndpointRequestMethod()))
.setHeader(Exchange.CONTENT_TYPE, constant(endpoint.getEndpointContentType()))
.to("netty-http:" + endpoint.getEndpointUrl())
}
private fun getProcessor(processorClassName: String, name: String, objectMapper: ObjectMapper): Processor {
var processorClass = Class.forName("com.demo.camelpoc.processors.$name.$processorClassName")
return processorClass.getDeclaredConstructor(ObjectMapper::class.java).newInstance(objectMapper) as Processor
}
And there is a source endpoint which starts the workflow. For example the default workflow generated in runtime:
// Workflow
from("netty-http:$sourceUrl").process {
it.setProperty("Workflow", workflowMap)
}
.dynamicRouter(bean(DynamicRouteBean::class.java, "route(*, *, ${startEndpoint})"))
where workflowMap (Used in DynamicRouteBean) is a map of endpoint strings like Key: "direct:Route1 " Value : "direct:Route2", Key: "direct:Route2 " Value : "direct:Route3"... etc
Requirement: Retry sending to the same endpoint in the workflow when exception is thrown in that particular route
For eg:
Lets say, an exception occurs at direct:Route2, I want to retry sending to direct:Route2.
Here is my DynamicRouteBean class.
class DynamicRouteBean {
fun route(
#Header(Exchange.SLIP_ENDPOINT) previousRoute: String?,
exchange: Exchange,
startEndpoint: String
): String? {
if(checkException(exchange)) {
return exchange.getProperty(Exchange.SLIP_ENDPOINT) as String
}
if (exchange.getProperty(Properties.STOP_ROUTE) != null && exchange.getProperty(Properties.STOP_ROUTE) == true) {
return null
}
val workflow: MutableMap<String, Pair<String, String?>> =
exchange.getProperty("Workflow") as MutableMap<String, Pair<String, String?>>
return when (previousRoute) {
null ->
startEndpoint
else -> {
val message = exchange.getIn(NettyHttpMessage::class.java)
// Signifies last endpoint and thus means end of the route
if (!workflow.containsKey(previousRoute)) {
return null
}
if (message?.getHeader(previousRoute.substring(9)) != null) {
val isSuccess = message.getHeader(previousRoute.substring(9)) == true
if (isSuccess) {
"${workflow[previousRoute]?.first}"
} else if (workflow[previousRoute]?.second != null) {
"${workflow[previousRoute]?.second}"
} else {
null
}
} else {
null
}
}
}
}
When I return current Exchange.SLIP_ENDPOINT property as String on exception, it doesn't call that endpoint again but exception message is returned back to the consumer.
Whereas it works in normal cases when there is no exception.
Suggestions would be very helpful on handling this scenario.

"Response has already been written" with Vertx

I am brand new to Vertx and trying to create my first HTTP Rest Server. However, I have been running into this issue. Here is the error I'm getting when I try to send a response.
Jan 07, 2017 3:54:36 AM io.vertx.ext.web.impl.RoutingContextImplBase
SEVERE: Unexpected exception in route
java.lang.IllegalStateException: Response has already been written
at io.vertx.core.http.impl.HttpServerResponseImpl.checkWritten(HttpServerResponseImpl.java:559)
at io.vertx.core.http.impl.HttpServerResponseImpl.putHeader(HttpServerResponseImpl.java:156)
at io.vertx.core.http.impl.HttpServerResponseImpl.putHeader(HttpServerResponseImpl.java:54)
at com.themolecule.utils.Response.sendResponse(Response.kt:25)
at com.themolecule.api.UserAPI.addUser(UserAPI.kt:52)
at TheMoleculeAPI$main$3.handle(TheMoleculeAPI.kt:50)
at TheMoleculeAPI$main$3.handle(TheMoleculeAPI.kt:19)
at io.vertx.ext.web.impl.RouteImpl.handleContext(RouteImpl.java:215)
at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:78)
at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:94)
at io.vertx.ext.web.handler.impl.AuthHandlerImpl.authorise(AuthHandlerImpl.java:86)
at
And this is how I have set up my routes.
// setup verx
val vertx = Vertx.vertx()
val router = Router.router(vertx)
val allowHeaders = HashSet<String>()
allowHeaders.add("x-requested-with")
allowHeaders.add("Access-Control-Allow-Origin")
allowHeaders.add("origin")
allowHeaders.add("Content-Type")
allowHeaders.add("accept")
val allowMethods = HashSet<HttpMethod>()
allowMethods.add(HttpMethod.GET)
allowMethods.add(HttpMethod.POST)
allowMethods.add(HttpMethod.DELETE)
allowMethods.add(HttpMethod.PATCH)
router.route().handler(CorsHandler.create("*")
.allowedHeaders(allowHeaders)
.allowedMethods(allowMethods))
router.route().handler(BodyHandler.create())
val config = JsonObject().put("keyStore", JsonObject()
.put("path", "keystore.jceks")
.put("type", "jceks")
.put("password", "password"))
// protect the API, login outside of JWT
router.route("/them/*").handler(JWTAuthHandler.create(jwt, "/them/login"))
//login routes
router.post("/them/login").handler { it -> loginAPI.login(it, jwt) }
//user routes
router.get("/them/user/getusers").handler { it -> userAPI.getUsers(it) }
router.post("/them/user/addUser").handler { it -> userAPI.addUser(it) }
This is the code that it seems to have the problem with.
fun sendResponse(responseCode: Int, responseMsg: String, context: RoutingContext) {
val userResponse = JsonObject().put("response", responseMsg)
context
.response()
.putHeader("content-type", "application/json")
.setStatusCode(responseCode)
.end(userResponse.encodePrettily())
}
Am I doing something wrong with the handlers? I tried to change the method for the response up a bunch of times, but nothing seems to work. This is written in Kotlin. If I need to add more context, just say the word!
edit: addUser method added
fun addUser(context: RoutingContext) {
val request = context.bodyAsJson
val newUser = NewUserRequestDTO(request.getString("userID").trim(), request.getString("password").trim())
newUser.userID.trim()
if (!newUser.userID.isNullOrEmpty() && !newUser.password.isNullOrEmpty()) {
//check user ID
if (userDAO.userIdInUse(newUser.userID)) {
Response.sendResponse(400, Response.ResponseMessage.ID_IN_USE.message, context)
return
}
//check password valid
val isValid: Boolean = SecurityUtils.checkPasswordCompliance(newUser.password)
if (!isValid) {
Response.sendResponse(400, Response.ResponseMessage.PASSWORD_IS_NOT_VALID.message, context)
return
}
val saltedPass = SecurityUtils.genSaltedPasswordAndSalt(newUser.password)
userDAO.addUser(newUser.userID, saltedPass.salt.toString(), saltedPass.pwdDigest.toString())
} else {
Response.sendResponse(401, Response.ResponseMessage.NO_USERNAME_OR_PASSWORD.message, context)
return
}
Response.sendResponse(200, Response.ResponseMessage.USER_CREATED_SUCCESSFULLY.message, context)
}