I have the following code where I call external APIs via webclient and return Mono.
I need to execute some logic when I receive data. And after all, requests are processed, execute one logic for all gathered data. I can collect all Monos and put them to flux and then execute some logic at the end. But I have serviceName filed which is accessible only in the loop, so I need to execute logic for mono in loop and here I'm stuck and don't know how to wait for all data to complete and do it in a reactive way.
#Scheduled(fixedDelay = 50000)
public void refreshSwaggerConfigurations() {
log.debug("Starting Service Definition Context refresh");
List<SwaggerServiceData> allServicesApi = new ArrayList<>();
swaggerProperties.getUrls().forEach((serviceName, serviceSwaggerUrl) -> {
log.debug("Attempting service definition refresh for Service : {} ", serviceName);
Mono<SwaggerServiceData> swaggerData = getSwaggerDefinitionForAPI(serviceName,
serviceSwaggerUrl);
swaggerData.subscribe(swaggerServiceData -> {
if (swaggerServiceData != null) {
allServicesApi.add(swaggerServiceData);
String content = getJSON(swaggerServiceData);
definitionContext.addServiceDefinition(serviceName, content);
} else {
log.error("Skipping service id : {} Error : Could not get Swagger definition from API ",
serviceName);
}
});
});
//I need to wait here for all monos to complete and after that proceed for All gathered data...
//Now it's empty And I know why, just don't know how to make it.
Optional<SwaggerServiceData> swaggerAllServicesData = getAllServicesApiSwagger(allServicesApi);
if (swaggerAllServicesData.isPresent()) {
String allApiContent = getJSON(swaggerAllServicesData.get());
definitionContext.addServiceDefinition("All", allApiContent);
}
}
private Mono<SwaggerServiceData> getSwaggerDefinitionForAPI(String serviceName, String url) {
log.debug("Accessing the SwaggerDefinition JSON for Service : {} : URL : {} ", serviceName,
url);
Mono<SwaggerServiceData> swaggerServiceDataMono = webClient.get()
.uri(url)
.exchangeToMono(clientResponse -> clientResponse.bodyToMono(SwaggerServiceData.class));
return swaggerServiceDataMono;
}
I would add a temporary class to group data and serivce name :
record SwaggerService(SwaggerServiceData swaggerServiceData, String serviceName) {
boolean hasData() {
return swaggerServiceData != null;
}
}
And then change your pipeline :
Flux.fromStream(swaggerProperties.getUrls().entrySet().stream())
.flatMap((e) -> {
Mono<SwaggerServiceData> swaggerDefinitionForAPI = getSwaggerDefinitionForAPI(e.getKey(),
e.getValue());
return swaggerDefinitionForAPI.map(swaggerServiceData -> new SwaggerService(swaggerServiceData, e.getKey()));
})
.filter(SwaggerService::hasData)
.map(swaggerService -> {
String content = getJSON(swaggerService.swaggerServiceData());
definitionContext.addServiceDefinition(swaggerService.serviceName(), content);
return swaggerService.swaggerServiceData();
})
// here we will collect all datas and they will be emmited as single Mono with list of SwaggerServiceData
.collectList()
.map(this::getAllServicesApiSwagger)
.filter(Optional::isPresent)
.map(Optional::get)
.subscribe(e -> {
String allApiContent = getJSON(e);
definitionContext.addServiceDefinition("All", allApiContent);
});
This does not deal with logging error when SwaggerServiceData is null but you can further change it if you want. Also I assume that DefinitionContext is thread safe.
Solution with error logging (using flatMap and Mono.empty()) :
Flux.fromStream(swaggerProperties.getUrls().entrySet().stream())
.flatMap((e) -> {
Mono<SwaggerServiceData> swaggerDefinitionForAPI = getSwaggerDefinitionForAPI(e.getKey(),
e.getValue());
return swaggerDefinitionForAPI
.flatMap(swaggerServiceData -> {
if(swaggerServiceData != null) {
return Mono.just(new SwaggerService(swaggerServiceData, e.getKey()));
} else {
log.error("Skipping service id : {} Error : Could not get Swagger definition from API ",
e.getKey());
return Mono.empty();
}
});
})
.map(swaggerService -> {
String content = getJSON(swaggerService.swaggerServiceData());
definitionContext.addServiceDefinition(swaggerService.serviceName(), content);
return swaggerService.swaggerServiceData();
}).collectList()
.map(this::getAllServicesApiSwagger)
.filter(Optional::isPresent)
.map(Optional::get)
.subscribe(e -> {
String allApiContent = getJSON(e);
definitionContext.addServiceDefinition("All", allApiContent);
});
You can also wrap those lambads into some meaningful methods to improve readibility.
I have defined completableFuture.supplyAsync() inside foreach loop, so each entry(each asynchronous task) adding a list and I need to get final list(after all asynchronous task adding list)from completableFuture.supplyAsync().How to achieve this?
Code snippet:
unporcessedList.forEach(entry -> {
CompletableFuture<List<ChangeLog>> cf =
CompletableFuture.supplyAsync((Supplier<List<ChangeLog>>) () -> {
mongoDBHelper.processInMongo(entry, getObject(entry, map),entryList);
return entryList;
}, executor);
});
Non blocking version
General example:
List<String> entries = new ArrayList<>(2);
entries.add("first");
entries.add("second");
List<CompletableFuture<String>> completableFutures = entries.stream()
.map((entry) -> {
return CompletableFuture.supplyAsync(() -> {
try {
Thread.sleep(new Random().nextInt(5000) + 500);
} catch (InterruptedException e) {
e.printStackTrace();
}
return entry.concat(String.valueOf(entry.length()));
}).thenApply((e) -> new StringBuilder(e).reverse().toString());
}
).collect(Collectors.toList());
CompletableFuture
.allOf(completableFutures.toArray(new CompletableFuture[completableFutures.size()]))
.thenApply((v) -> completableFutures.stream().map((cf) -> cf.join()))
.get()
.forEach(System.out::println);
Your case:
List<CompletableFuture<List<ChangeLog>>> completableFutures = unporcessedList.stream()
.map((entry) -> {
return CompletableFuture.supplyAsync((Supplier<List<ChangeLog>>) () -> {
mongoDBHelper.processInMongo(entry, getObject(entry, map), entryList);
return entryList;
}, executor);
}
).collect(Collectors.toList());
CompletableFuture
.allOf(completableFutures.toArray(new CompletableFuture[completableFutures.size()]))
.thenApply((v) -> completableFutures.stream().map((cf) -> cf.join()))
.get()
.forEach(System.out::println);
You can use the get() Method that will block your application until the future is completed. So use something like this:
// Block and get the result of the Future
Supplier<List<ChangeLog>> result = cf.get();
More examples are described here: https://www.callicoder.com/java-8-completablefuture-tutorial/
Hope this helps.
The below is my code snippet, and sometimes the Observable does not time out,
Observable<A> AObservable = Observable.fromCallable(() ->
//External Service Call
).timeout(800, TimeUnit.MILLISECONDS)
.subscribeOn(Schedulers.io())
.onErrorReturn(throwable -> {
LOGGER.warn(format("Server did not respond within %s ms for id=%s", 800, id));
return null;
});
Observable<B> BObservable = Observable.fromCallable(() ->
//External Service Call
).timeout(800, TimeUnit.MILLISECONDS)
.subscribeOn(Schedulers.io())
.onErrorReturn( throwable -> {
LOGGER.warn(format("Service did not respond within %s ms for id=%s", 800, Id));
return null;
});
// Build Default response
Observable<C> CObservable = Observable.fromCallable(() ->
// Build Default one
).subscribeOn(Schedulers.io());
return Observable.zip(AObservable, BObservable,CObservable,
(AResponse, BResponse, CResponse) -> {
// Handle response and combine them
}).toBlocking().first();
It appears to me that at times the if the service takes more than 800ms the timeout does not happen. Do I miss any attribute here please advise.
I have following code written in RXJava.
validateProduct(payload)
.map.(r -> {
if(r.getBoolean("valid")){
return createProduct(productPayload);
}else{
return null; // request.end() | end the chain here with some message as invalid product.
}
})
.map(r -> {
return linkCategories(catPayload);
})
.map(r -> {
return linkTags(tagPayload);
})
.doOnError(e -> log.error(e))
.subscribe(r -> {
JsonObject response = new JsonObject().put("status", true);
request.end(response);
}, e -> {
JsonObject response = new JsonObject().put("status", false);
request.end(response);
});
The first block has a condition check, this code is not working right now. Whats the best way to handle conditional chains in RX?
It looks like you are probably running into a null pointer exception. nulls are not acceptable in RxJava v2. Your first map is likely causing problems.
Generally when you need conditional logic in rxjava and may not be returning an object you have two options:
Return an object that signifies null (and possibly filter it out)
Use an operator like flatMap and add on an empty Observable
It looks like you are probably running into a null pointer exception. nulls are not acceptable in RxJavav2. Your first map is likely causing problems.
Option 1.
validateProduct(payload)
.map.(r -> {
if(r.getBoolean("valid")){
return createProduct(productPayload);
}else{
return createEmptyProduct(); // generate non null placeholder object
}
})
.filter(r->{
// check here via method call or instanceOf to filter out empty products
r instanceof ValidProduct
}).map(r -> {
return linkCategories(catPayload);
})
.map(r -> {
return linkTags(tagPayload);
})
.doOnError(e -> log.error(e))
.subscribe(r -> {
JsonObject response = new JsonObject().put("status", true);
request.end(response);
}, e -> {
JsonObject response = new JsonObject().put("status", false);
request.end(response);
});
Option 2
validateProduct(payload)
.flatMap(r -> {
if(r.getBoolean("valid")){
return createProduct(productPayload); // Assuming this returns an observable if not use Observable.just(createProduct(productPayload))
}else{
return Observable.empty(); // request.end() | end the chain here with some message as invalid product.
}
})
.map(r -> {
return linkCategories(catPayload);
})
.map(r -> {
return linkTags(tagPayload);
})
.doOnError(e -> log.error(e))
.subscribe(r -> {
JsonObject response = new JsonObject().put("status", true);
request.end(response);
}, e -> {
JsonObject response = new JsonObject().put("status", false);
request.end(response);
});
I am having a lot of trouble understanding the zip operator in RxJava for my android project.
Problem
I need to be able to send a network request to upload a video
Then i need to send a network request to upload a picture to go with it
finally i need to add a description and use the responses from the previous two requests to upload the location urls of the video and picture along with the description to my server.
I assumed that the zip operator would be perfect for this task as I understood we could take the response of two observables (video and picture requests) and use them for my final task.
But I cant seem to get this to occur how I envision it.
I am looking for someone to answer how this can be done conceptually with a bit of psuedo code.
Thank you
Zip operator strictly pairs emitted items from observables. It waits for both (or more) items to arrive then merges them. So yes this would be suitable for your needs.
I would use Func2 to chain the result from the first two observables.
Notice this approach would be simpler if you use Retrofit since its api interface may return an observable. Otherwise you would need to create your own observable.
// assuming each observable returns response in the form of String
Observable<String> movOb = Observable.create(...);
// if you use Retrofit
Observable<String> picOb = RetrofitApiManager.getService().uploadPic(...),
Observable.zip(movOb, picOb, new Func2<String, String, MyResult>() {
#Override
public MyResult call(String movieUploadResponse, String picUploadResponse) {
// analyze both responses, upload them to another server
// and return this method with a MyResult type
return myResult;
}
}
)
// continue chaining this observable with subscriber
// or use it for something else
A small example:
val observableOne = Observable.just("Hello", "World")
val observableTwo = Observable.just("Bye", "Friends")
val zipper = BiFunction<String, String, String> { first, second -> "$first - $second" }
Observable.zip(observableOne, observableTwo, zipper)
.subscribe { println(it) }
This will print:
Hello - Bye
World - Friends
In BiFunction<String, String, String> the first String the type of the first observable, the second String is the type of the second observable, the third String represents the type of the return of your zipper function.
I made a small example that calls two real endpoints using zip in this blog post
Here I have an example that I did using Zip in asynchronous way, just in case you´re curious
/**
* Since every observable into the zip is created to subscribeOn a diferent thread, it´s means all of them will run in parallel.
* By default Rx is not async, only if you explicitly use subscribeOn.
*/
#Test
public void testAsyncZip() {
scheduler = Schedulers.newThread();
scheduler1 = Schedulers.newThread();
scheduler2 = Schedulers.newThread();
long start = System.currentTimeMillis();
Observable.zip(obAsyncString(), obAsyncString1(), obAsyncString2(), (s, s2, s3) -> s.concat(s2)
.concat(s3))
.subscribe(result -> showResult("Async in:", start, result));
}
/**
* In this example the the three observables will be emitted sequentially and the three items will be passed to the pipeline
*/
#Test
public void testZip() {
long start = System.currentTimeMillis();
Observable.zip(obString(), obString1(), obString2(), (s, s2, s3) -> s.concat(s2)
.concat(s3))
.subscribe(result -> showResult("Sync in:", start, result));
}
public void showResult(String transactionType, long start, String result) {
System.out.println(result + " " +
transactionType + String.valueOf(System.currentTimeMillis() - start));
}
public Observable<String> obString() {
return Observable.just("")
.doOnNext(val -> {
System.out.println("Thread " + Thread.currentThread()
.getName());
})
.map(val -> "Hello");
}
public Observable<String> obString1() {
return Observable.just("")
.doOnNext(val -> {
System.out.println("Thread " + Thread.currentThread()
.getName());
})
.map(val -> " World");
}
public Observable<String> obString2() {
return Observable.just("")
.doOnNext(val -> {
System.out.println("Thread " + Thread.currentThread()
.getName());
})
.map(val -> "!");
}
public Observable<String> obAsyncString() {
return Observable.just("")
.observeOn(scheduler)
.doOnNext(val -> {
System.out.println("Thread " + Thread.currentThread()
.getName());
})
.map(val -> "Hello");
}
public Observable<String> obAsyncString1() {
return Observable.just("")
.observeOn(scheduler1)
.doOnNext(val -> {
System.out.println("Thread " + Thread.currentThread()
.getName());
})
.map(val -> " World");
}
public Observable<String> obAsyncString2() {
return Observable.just("")
.observeOn(scheduler2)
.doOnNext(val -> {
System.out.println("Thread " + Thread.currentThread()
.getName());
})
.map(val -> "!");
}
You can see more examples here https://github.com/politrons/reactive
zip operator allow you to compose a result from results of two different observable.
You 'll have to give am lambda that will create a result from datas emitted by each observable.
Observable<MovieResponse> movies = ...
Observable<PictureResponse> picture = ...
Observable<Response> response = movies.zipWith(picture, (movie, pic) -> {
return new Response("description", movie.getName(), pic.getUrl());
});
i have been searching for a simple answer on how to use the Zip operator, and what to do with the Observables i create to pass them to it, i was wondering if i should call subscribe() for every observable or not, non of these answers were simple to find, i had to figure it out by my self, so here is a simple example for using Zip operator on 2 Observables :
#Test
public void zipOperator() throws Exception {
List<Integer> indexes = Arrays.asList(0, 1, 2, 3, 4);
List<String> letters = Arrays.asList("a", "b", "c", "d", "e");
Observable<Integer> indexesObservable = Observable.fromIterable(indexes);
Observable<String> lettersObservable = Observable.fromIterable(letters);
Observable.zip(indexesObservable, lettersObservable, mergeEmittedItems())
.subscribe(printMergedItems());
}
#NonNull
private BiFunction<Integer, String, String> mergeEmittedItems() {
return new BiFunction<Integer, String, String>() {
#Override
public String apply(Integer index, String letter) throws Exception {
return "[" + index + "] " + letter;
}
};
}
#NonNull
private Consumer<String> printMergedItems() {
return new Consumer<String>() {
#Override
public void accept(String s) throws Exception {
System.out.println(s);
}
};
}
the printed result is :
[0] a
[1] b
[2] c
[3] d
[4] e
the final answers to the questions that where in my head were as follows
the Observables passed to the zip() method just need to be created only, they do not need to have any subscribers to them, only creating them is enough ... if you want any observable to run on a scheduler, you can specify this for that Observable ... i also tried the zip() operator on Observables where they should wait for there result, and the Consumable of the zip() was triggered only when both results where ready (which is the expected behavior)
This is my implementation using Single.zip and rxJava2
I tried to make it as easy to understand as possible
//
// API Client Interface
//
#GET(ServicesConstants.API_PREFIX + "questions/{id}/")
Single<Response<ResponseGeneric<List<ResponseQuestion>>>> getBaseQuestions(#Path("id") int personId);
#GET(ServicesConstants.API_PREFIX + "physician/{id}/")
Single<Response<ResponseGeneric<List<ResponsePhysician>>>> getPhysicianInfo(#Path("id") int personId);
//
// API middle layer - NOTE: I had feedback that the Single.create is not needed (but I haven't yet spent the time to improve it)
//
public Single<List<ResponsePhysician>> getPhysicianInfo(int personId) {
return Single.create(subscriber -> {
apiClient.getPhysicianInfo(appId)
.subscribeOn(Schedulers.io())
.observeOn(Schedulers.io())
.subscribe(response -> {
ResponseGeneric<List<ResponsePhysician>> responseBody = response.body();
if(responseBody != null && responseBody.statusCode == 1) {
if (!subscriber.isDisposed()) subscriber.onSuccess(responseBody.data);
} else if(response.body() != null && response.body().status != null ){
if (!subscriber.isDisposed()) subscriber.onError(new Throwable(response.body().status));
} else {
if (!subscriber.isDisposed()) subscriber.onError(new Throwable(response.message()));
}
}, throwable -> {
throwable.printStackTrace();
if(!subscriber.isDisposed()) subscriber.onError(throwable);
});
});
}
public Single<List<ResponseQuestion>> getHealthQuestions(int personId){
return Single.create(subscriber -> {
apiClient.getBaseQuestions(personId)
.subscribeOn(Schedulers.io())
.observeOn(Schedulers.io())
.subscribe(response -> {
ResponseGeneric<List<ResponseQuestion>> responseBody = response.body();
if(responseBody != null && responseBody.data != null) {
if (!subscriber.isDisposed()) subscriber.onSuccess(response.body().data);
} else if(response.body() != null && response.body().status != null ){
if (!subscriber.isDisposed()) subscriber.onError(new Throwable(response.body().status));
} else {
if (!subscriber.isDisposed()) subscriber.onError(new Throwable(response.message()));
}
}, throwable -> {
throwable.printStackTrace();
if(!subscriber.isDisposed()) subscriber.onError(throwable);
});
});
}
//please note that ResponseGeneric is just an outer wrapper of the returned data - common to all API's in this project
public class ResponseGeneric<T> {
#SerializedName("Status")
public String status;
#SerializedName("StatusCode")
public float statusCode;
#SerializedName("Data")
public T data;
}
//
// API end-use layer - this gets close to the UI so notice the oberver is set for main thread
//
private static class MergedResponse{// this is just a POJO to store all the responses in one object
public List<ResponseQuestion> listQuestions;
public List<ResponsePhysician> listPhysicians;
public MergedResponse(List<ResponseQuestion> listQuestions, List<ResponsePhysician> listPhysicians){
this.listQuestions = listQuestions;
this.listPhysicians = listPhysicians;
}
}
// example of Single.zip() - calls getHealthQuestions() and getPhysicianInfo() from API Middle Layer
private void downloadHealthQuestions(int personId) {
addRxSubscription(Single
.zip(getHealthQuestions(personId), getPhysicianInfo(personId), MergedResponse::new)
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(response -> {
if(response != null) {
Timber.i(" - total health questions downloaded %d", response.listQuestions.size());
Timber.i(" - physicians downloaded %d", response.listPhysicians.size());
if (response.listPhysicians != null && response.listPhysicians.size()>0) {
// do your stuff to process response data
}
if (response.listQuestions != null && response.listQuestions.size()>0) {
// do your stuff to process response data
}
} else {
// process error - show message
}
}, error -> {
// process error - show network error message
}));
}
You use the zip from rxjava with Java 8:
Observable<MovieResponse> movies = ...
Observable<PictureResponse> picture = ...
Observable<ZipResponse> response = Observable.zip(movies, picture, ZipResponse::new);
class ZipResponse {
private MovieResponse movieResponse;
private PictureResponse pictureResponse;
ZipResponse(MovieResponse movieResponse, PictureResponse pictureResponse) {
this.movieResponse = movieResponse;
this.pictureResponse = pictureResponse;
}
public MovieResponse getMovieResponse() {
return movieResponse;
}
public void setMovieResponse(MovieResponse movieResponse) {
this.movieResponse= movieResponse;
}
public PictureResponse getPictureResponse() {
return pictureResponse;
}
public void setPictureResponse(PictureResponse pictureResponse) {
this.pictureResponse= pictureResponse;
}
}
You can use .zipWith operator for Observable chains.
If uploadMovies() and uploadPictures() return Observable,
uploadMovies()
.zipWith(uploadPictures()) { m, p ->
"$m with $p were uploaded"
}
.subscribe { print(it) }