I have a client that returns objects wrapped in Uni. The client receives the objects one at a time, and I have a list of objects to send. I send the objects individually to the client and get Uni in response. I collect the responses into a list and get List<Uni>. How do I extract all the Uni's from this list, combine them into one Uni, along the way of the responses, converting them into other objects?
When i try Uni.combine().all().unis(List<Uni>) it says that "Not enough information to infer type variable O"
using mutiny api, tutorial
private fun makeHttpCall(dto: DtoRequest): Uni<DtoResponse> {
**mapping and calling http client**
}
fun someMethod(dtos: List<DtoRequest>): Uni<List<DtoNew>> {
val unis = dtos.map { makeHttpCall(it) }
return Uni.combine().all().unis(unis).combinedWith{ }
}
Related
I want to iterate through all pages of a given url and collect JSON objects. With this code I'm getting list of 10 objects:
List<EzamowieniaDto> ezam = WebClient
.create("https://ezamowienia.gov.pl/mo-board/api/v1/Board/Search?noticeType=ContractNotice&isTenderAmountBelowEU=true" +
"&publicationDateFrom=2022-03-16T00:00:00.000Z&orderType=Delivery&SortingColumnName=PublicationDate&SortingDirection=DESC" +
"&PageNumber=1")
.get()
.retrieve()
.bodyToMono(new ParameterizedTypeReference<List<EzamowieniaDto>>(){})
.block();
I've tried to just delete "PageNumber" from request, but it seems the pagination is hard-coded for this page.
(X-Pagination header from response: [{"TotalCount":88,"PageSize":10,"CurrentPage":1,"TotalPages":9,"HasNext":true,"HasPrevious":false}])
The question is: How can I iterate through number of pages mentioned in response header, and collect the whole data?
Here is the way you could handle paginaged requests with WebClient.
Create a method to retreive a single page of data. Typically you would use bodyToFlux(EzamowieniaDto.class) and return Flux<EzamowieniaDto> but because we need headers we have to use toEntityFlux(EzamowieniaDto.class) to wrap response in Mono<ResponseEntity.
Mono<ResponseEntity<Flux<EzamowieniaDto>>> getPage(String url, int pageNumber) {
return webClient.get()
.uri(url + "&PageNumber={pageNum}", pageNumber)
.retrieve()
.toEntityFlux(EzamowieniaDto.class);
}
Use expand to to fetch data until we reach the end
Flux<EzamowieniaDto> getData(String url) {
return getPage(url, 1)
.expand(response -> {
Pagination pagination = formJson(response.getHeaders().getFirst("X-Pagination"));
if (!pagination.hasNext()) {
// stop
return Mono.empty();
}
// fetch next page
getPage(url, pagination.getCurrentPage() + 1);
})
.flatMap(response -> response.getBody());
}
Firstly, do not use .block() method, because, shortly, it interrupts asynchronous stream and makes it synchronous, so there is no need, actually, in WebClient nowadays (here you can find some brief intro), to use it in such way. You can use, also, RestTemplate implementations like Retrofit. But in your case, to save the asynchronous pattern, you can use next solution:
List<EzamowieniaDto> ezam = WebClient
.create("https://ezamowienia.gov.pl/mo-board/api/v1/Board/Search?noticeType=ContractNotice&isTenderAmountBelowEU=true" +
"&publicationDateFrom=2022-03-16T00:00:00.000Z&orderType=Delivery&SortingColumnName=PublicationDate&SortingDirection=DESC" +
"&PageNumber=1")
.get()
.retrieve()
.bodyToFlux(EzamowieniaDto.class) // here you can just use Flux, it's like List from synchronous Java, simply
.map({body -> /*body = EzamowieniaDto , do any job with this object here*/})
...
Example
...
List<EzamowieniaDto> dtos = new ArrayList<>();
Flux<EzamowieniaDto> fluxDtos = WebClient
.create("http://some-url.com")
.get()
.retrieve()
.bodyToFlux(EzamowieniaDto.class)
.filter({body -> body.getId().equals(1L)}) // here just some filter for emitted elements
.subscribe({dto -> dtos.add(dto)}); // subscribe to end asynchronous flow , in simple words
System.out.println(dtos.get(0).getId().equals(1L); // some simple test or something like this, use dtos as you need.
Additionally
Using synchronous staff (Lists, Mono of List, etc.) mixed with asynchronous, you will always get synchronous behavior at some point of time, in the place in your code where it happens. Reactive programming implies that you use asynchronous programming (mostly, declarative programming) while the whole process from fetching asynchronously response to asynchronously writing to the database (Hibernate Reactive, for example).
Hope it helps somehow and I suggest to start learning reactive programming (Reactor or Spring WebFlux, for example), if you are not started yet to understand basics of asynchronous programming.
Best Regards, Anton.
I am trying to upload a file to S3 using this example:
https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/javav2/example_code/s3/src/main/java/com/example/s3/PutObject.java
But I have this error:
The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. (Service: S3, Status Code: 301, Request ID: 15F45F10B11A3267, Extended Request ID: 9TdN5/GMl9h6VGPq6ee4MttjlLFm+V5GjIaXIJLAO1YasdZCls0M4udxJso8n1P+qMr4iigP5g=)
It seems likely that this bucket was created in a different region. Please send your request to your region endpoint. The default region is as follow's
US Standard is us-east-1
The upper link have region 'Region.US_WEST_2'. First confirm your region then send your request accordingly.
What annoyed me is that if you create the S3 client once during initialization and re-use it (which is just the thing that makes sense), then you have to know ahead of time all the regions where you need to access buckets, and pre-init multiple S3 clients - there's no way that I can find (at least for the Java SDK) to "just send one request using a different endpoint".
Because pre-initing all the clients for all the regions that are possible (and they keep adding more all the time) is just insane, I wrote up this wrapper (it is using the async clients and exposes a Vert.x promises API, but it should be easy to adapt to anything else):
/**
* S3 client factory that helps resolve region-specific clients
* Licensed for any use under CC0
*/
public class S3AsyncClientFactory {
private ConcurrentHashMap<String, S3AsyncClient> cache =
new ConcurrentHashMap<>();
private S3AsyncClient global;
private S3AsyncClientFactory() {
// I call it "global" but it actually uses your "default region".
// There is a "global" region, but I couldn't be bothered to
// figure out how to create such a client.
global = S3AsyncClient.create();
}
public Future<Region> getBucketLocation(String bucket) {
return Future.fromCompletionStage(
global.getBucketLocation(b -> b.bucket(bucket)))
.map(loc -> loc.locationConstraintAsString())
// annoyingly, get location returns an empty string for
// the "standard region" and Region.of() can't handle that
.map(loc -> loc == null || loc.isEmpty() ?
Region.US_EAST_1 : Region.of(loc));
}
public Future<S3AsyncClient> forBucket(String bucket) {
return getBucketLocation(bucket).map(this::forRegion);
}
public S3AsyncClient global() {
return global;
}
public S3AsyncClient forRegion(Region region) {
return cache.computeIfAbsent(region.toString(),
// you may want to replace this part if you have custom
// init code or you need non-default credentials chain
r -> S3AsyncClient.builder().region(region).build());
}
public S3AsyncClient forRegion(String region) {
return forRegion(Region.of(region));
}
}
With that piece of code you just create and cache per-region S3 clients, as needed:
factory.forBucket(bucket).compose(c -> Future.fromCompletionStage(
c.getObject(b -> b.bucket(bucket).key(key), responseTransformer)))
.map(res -> ...
(please don't mind the ugly "different libraries have difference concurrency implementations" code 🤷)
You may also want to have a bucket-keyed client implementation cache (I have it in another layer where it makes more sense for my use case).
I am new to the Web Client reactive library.
Here is my problem :
It starts with a user submitting a request to post a packet of documents. They wait for a response.
A service consuming this request needs to run several tasks in parallel. Some of the sub-tasks within each task have to finish first ( 2 different Get requests ) before attempting the last sub-task which is the main Post request. Then I want to wait for the collection of all tasks 'Post sub-tasks' to finish (representing the packet), and then collect and reconcile the responses.
I need to reconcile at the end and make sure the entire parallel process succeeds (sending the packet) and then respond to a server (user) indicating that the process was successful or not.
My pseudo flow:
Create a set of documents to post to a server one at a time. A packet can contain up to 10 documents. (List of DocumentAndMetaData). Initially each document would contain some pre-filled known values like file path and document name.
For each document in a packet: (run in parallel)
I need to do file I/O and create a meta data object- call it getDocumentAndMetadata. To create a Metadata object I must do some
steps first within getDocumentAndMetadata:
Do a get Request to get Key A- call it getKeyA(requestA)
Do a get request to get Key B- call it getKeyB(requestB)
Merge Key A and Key B requests and use the responses from those requests to update the metadata object.
Then Read File to get a Byte array - call it getFile
Then pass the byte array (document) and metadata object to a function that:
Does a Http Post to a server sending the byte array and metadata object in the post request.
Accumulate the responses for each Post which are strings.
then block until all the documents are sent.
Finally evaluate all the string responses that are returned from the Post requests and make sure the number of responses match the number of documents posted to a server. Track any errors. If any Get or Post request fails, log the error.
I figured out how to do all these steps running block() on each sub-task 'Get request' and then block() on the main 'Post request', but I am afraid the performance will suffer using this approach.
I need help with how to generate the flow using Web-Client and reactive non blocking parallel processes.
Thanks for any help.
' I am afraid the performance will suffer using this approach.' - You are right. After all, the whole purpose of using WebFlux is to create a non-blocking application.
I have tried to mock most of the logic. I hope you can correlate the solution with your use-case.
#RestController
public class MyController {
#Autowired
private WebClient webClient;
#PostMapping(value = "/postPacketOfDocs")
public Mono<ResponseEntity<String>> upload(#RequestBody Flux<String> documentAndMetaDataList) {
return documentAndMetaDataList
.flatMap(documentAndMetaData -> {
//do File I/O
return getDocumentAndMetadata(documentAndMetaData);
})
.map(String::getBytes) //Read File to get a Byte array
.flatMap(fileBytes -> {
return webClient.post().uri("/send/byte/and/metadata")
.retrieve().bodyToMono(String.class);
})
.collectList()
.flatMap(allResponsesFromEachPOST -> {
//Do some validation
boolean allValidationsSuccessful = true;
if (allValidationsSuccessful) {
return Mono.just("Success");
} else {
return Mono.error(new RuntimeException()); //some custom exception which can be handled by #ExceptionHandler
}
})
.flatMap(msg -> Mono.just(ResponseEntity.ok().body(msg)));
}
private Mono<String> getDocumentAndMetadata(String documentAndMetaData) {
String metadata = "";//get metadata object from documentAndMetaData
Mono<String> keyAResponse = webClient.get().uri("/get/keyA").retrieve().bodyToMono(String.class);
Mono<String> keyBResponse = webClient.get().uri("/get/keyB").retrieve().bodyToMono(String.class);
return keyAResponse.concatWith(keyBResponse)
.collectList()
.flatMap(responses -> updateMetadata(responses, metadata));
}
private Mono<String> updateMetadata(List<String> responses, String metadata) {
String newMedataData = metadata + responses.get(0) + responses.get(1); //some update logic
return Mono.just(newMedataData);
}
}
I have a scenario where i need to get data from Soap Service API call for each company.
For each industry i have thousands companies.
To call API , i need to set/pass few parameters apart from companyid. These parameters like user name , password , url , port etc... more of configuration or property files ..
These wont change for each call i.e. each Actor.
I am trying to implement the same using Akka Actor system for company retrieval i.e. calling Soap Service API .
Me new to Akka , any help how to do this ?
Thanks and Regards.
Tried :
CompanyActor -- which expects a companyId so created a CompanyInput as below
CompanyInput{
String companyId,
DataAPIConnection conn;
}
I am preparing CompanyInput where I get connection object "conn" by passing all valid userName and pwd , url to Soap Service API call.
I set companyId also.
Then create CompanyActor . where
#Override
public void onReceive(Object msg) throws Throwable {
if(msg instanceof CompanyInput) {
CompanyInput input = (CompanyInput) msg;
logger.info("onRecive msg companyId : " + input.getCompanyId());
}
}
Hope the above works fine. Please suggest if any corrections needed.
But from this I need to return CompanyInfo object i.e. reading the fields from SoapAPI call and populating CompanyInfo VO.
How to return an object from an Actor call ? onReceive() returning void so I return a Future ?
In Akka, messages go in one direction and the receive function does not return a value. Instead, you can send a reply message back to the original sender:
https://doc.akka.io/docs/akka/current/actors.html#reply-to-messages
You can use the "ask pattern" to simplify sending a message that expects a reply. Calling ask returns a CompletionStage in Java (or a Future in Scala) that completes when the receiving actor replies to its sender:
https://doc.akka.io/docs/akka/current/actors.html#ask-send-and-receive-future
I need help on Spring Reactive where a rest call posts list of Json objects and spring boot server should send the processing events one by one.
Let me explain in brief with an example.
Let us take there are 20 products in the Front-end UI, user selects all the products to be processed.
Each product processing takes minimum 1 min in the server side. Whenever each product is processed, server should send json message structure as
event to the Front-end UI so that user will be able to see incremental progress of each product processing in the server.
In the UI , it should look like this.
Product 1 processed successfully
Product 2 processed successfully
Product 3 failed
like this.....
In the server side, the java code should be like this. Please suggest how to achieve using Spring Reactive.
public Flux<ProdModel> createAllCGs(List<Product> prodList) {
for(Product p : prodList) {
//Process here ...
}
//use Spring Reactor Flux
//return Flux type object in the form of Json structure event not as Text Stream event.
}
I know there are workarounds to achieve it using traditional polling mechanism or sending the product one by one.
My question is more on Spring Reactive side where the rest call sends a bunch of products to be processed one by one by providing corresponding
response in the json format to the UI side.
I do not know whether it is possible or not. If you think it is not possible using Spring Reactive, that is also fine for me so that I can communicate to my architect who has suggested this.
I struggled a bit to find out the answer,I an also new to Spring Reactive. I hope this answer will help to other.
I provide below the code snippet.
public Flux<ProdModel> createAllCGs(List<Product> prodList) {
return Flux.fromIterable(prodList)
.map(
prodModel -> {
System.out.println("Input Data VM ::: " + prodModel);
return getProdModel(reviewModel);
})
.delayElements(Duration.ofSeconds(3));
}
private getProdModel getProdModel(ProdModel prodModel) {
logger.debug("Time Now: {}", LocalDate.now());
ProdModel cgModel = new CGModel();
cgModel.setCgName("some Name");
cgModel.setMessage("some meaningful message");
cgModel.setTimestamp(LocalDateTime.now().toString());
return cgModel;
}
If you create a simple GET type rest end point and use the above method, you can see the output one by one after 3 seconds in the browser.