We Keep Coding

Spring Reactive. How wait for all monos to finish?

I have the following code where I call external APIs via webclient and return Mono.
I need to execute some logic when I receive data. And after all, requests are processed, execute one logic for all gathered data. I can collect all Monos and put them to flux and then execute some logic at the end. But I have serviceName filed which is accessible only in the loop, so I need to execute logic for mono in loop and here I'm stuck and don't know how to wait for all data to complete and do it in a reactive way.
#Scheduled(fixedDelay = 50000)
public void refreshSwaggerConfigurations() {
log.debug("Starting Service Definition Context refresh");
List<SwaggerServiceData> allServicesApi = new ArrayList<>();
swaggerProperties.getUrls().forEach((serviceName, serviceSwaggerUrl) -> {
log.debug("Attempting service definition refresh for Service : {} ", serviceName);
Mono<SwaggerServiceData> swaggerData = getSwaggerDefinitionForAPI(serviceName,
serviceSwaggerUrl);
swaggerData.subscribe(swaggerServiceData -> {
if (swaggerServiceData != null) {
allServicesApi.add(swaggerServiceData);
String content = getJSON(swaggerServiceData);
definitionContext.addServiceDefinition(serviceName, content);
} else {
log.error("Skipping service id : {} Error : Could not get Swagger definition from API ",
serviceName);
}
});
});
//I need to wait here for all monos to complete and after that proceed for All gathered data...
//Now it's empty And I know why, just don't know how to make it.
Optional<SwaggerServiceData> swaggerAllServicesData = getAllServicesApiSwagger(allServicesApi);
if (swaggerAllServicesData.isPresent()) {
String allApiContent = getJSON(swaggerAllServicesData.get());
definitionContext.addServiceDefinition("All", allApiContent);
}
}
private Mono<SwaggerServiceData> getSwaggerDefinitionForAPI(String serviceName, String url) {
log.debug("Accessing the SwaggerDefinition JSON for Service : {} : URL : {} ", serviceName,
url);
Mono<SwaggerServiceData> swaggerServiceDataMono = webClient.get()
.uri(url)
.exchangeToMono(clientResponse -> clientResponse.bodyToMono(SwaggerServiceData.class));
return swaggerServiceDataMono;
}

I would add a temporary class to group data and serivce name :
record SwaggerService(SwaggerServiceData swaggerServiceData, String serviceName) {
boolean hasData() {
return swaggerServiceData != null;
}
}
And then change your pipeline :
Flux.fromStream(swaggerProperties.getUrls().entrySet().stream())
.flatMap((e) -> {
Mono<SwaggerServiceData> swaggerDefinitionForAPI = getSwaggerDefinitionForAPI(e.getKey(),
e.getValue());
return swaggerDefinitionForAPI.map(swaggerServiceData -> new SwaggerService(swaggerServiceData, e.getKey()));
})
.filter(SwaggerService::hasData)
.map(swaggerService -> {
String content = getJSON(swaggerService.swaggerServiceData());
definitionContext.addServiceDefinition(swaggerService.serviceName(), content);
return swaggerService.swaggerServiceData();
})
// here we will collect all datas and they will be emmited as single Mono with list of SwaggerServiceData
.collectList()
.map(this::getAllServicesApiSwagger)
.filter(Optional::isPresent)
.map(Optional::get)
.subscribe(e -> {
String allApiContent = getJSON(e);
definitionContext.addServiceDefinition("All", allApiContent);
});
This does not deal with logging error when SwaggerServiceData is null but you can further change it if you want. Also I assume that DefinitionContext is thread safe.
Solution with error logging (using flatMap and Mono.empty()) :
Flux.fromStream(swaggerProperties.getUrls().entrySet().stream())
.flatMap((e) -> {
Mono<SwaggerServiceData> swaggerDefinitionForAPI = getSwaggerDefinitionForAPI(e.getKey(),
e.getValue());
return swaggerDefinitionForAPI
.flatMap(swaggerServiceData -> {
if(swaggerServiceData != null) {
return Mono.just(new SwaggerService(swaggerServiceData, e.getKey()));
} else {
log.error("Skipping service id : {} Error : Could not get Swagger definition from API ",
e.getKey());
return Mono.empty();
}
});
})
.map(swaggerService -> {
String content = getJSON(swaggerService.swaggerServiceData());
definitionContext.addServiceDefinition(swaggerService.serviceName(), content);
return swaggerService.swaggerServiceData();
}).collectList()
.map(this::getAllServicesApiSwagger)
.filter(Optional::isPresent)
.map(Optional::get)
.subscribe(e -> {
String allApiContent = getJSON(e);
definitionContext.addServiceDefinition("All", allApiContent);
});
You can also wrap those lambads into some meaningful methods to improve readibility.

How to merge multiple vertx web client responses

I am new to vertx and async programming.
I have 2 verticles communicating via an event bus as follows:
//API Verticle
public class SearchAPIVerticle extends AbstractVerticle {
public static final String GET_USEARCH_DOCS = "get.usearch.docs";
#Autowired
private Integer defaultPort;
private void sendSearchRequest(RoutingContext routingContext) {
final JsonObject requestMessage = routingContext.getBodyAsJson();
final EventBus eventBus = vertx.eventBus();
eventBus.request(GET_USEARCH_DOCS, requestMessage, reply -> {
if (reply.succeeded()) {
Logger.info("Search Result = " + reply.result().body());
routingContext.response()
.putHeader("content-type", "application/json")
.setStatusCode(200)
.end((String) reply.result().body());
} else {
Logger.info("Document Search Request cannot be processed");
routingContext.response()
.setStatusCode(500)
.end();
}
});
}
#Override
public void start() throws Exception {
Logger.info("Starting the Gateway service (Event Sender) verticle");
// Create a Router
Router router = Router.router(vertx);
//Added bodyhandler so we can process json messages via the event bus
router.route().handler(BodyHandler.create());
// Mount the handler for incoming requests
// Find documents
router.post("/api/search/docs/*").handler(this::sendSearchRequest);
// Create an HTTP Server using default options
HttpServer server = vertx.createHttpServer();
// Handle every request using the router
server.requestHandler(router)
//start listening on port 8083
.listen(config().getInteger("http.port", 8083)).onSuccess(msg -> {
Logger.info("*************** Search Gateway Server started on "
+ server.actualPort() + " *************");
});
}
#Override
public void stop(){
//house keeping
}
}
//Below is the target verticle should be making the multiple web client call and merging the responses
.
#Component
public class SolrCloudVerticle extends AbstractVerticle {
public static final String GET_USEARCH_DOCS = "get.usearch.docs";
#Autowired
private SearchRepository searchRepositoryService;
#Override
public void start() throws Exception {
Logger.info("Starting the Solr Cloud Search Service (Event Consumer) verticle");
super.start();
ConfigStoreOptions fileStore = new ConfigStoreOptions().setType("file")
.setConfig(new JsonObject().put("path", "conf/config.json"));
ConfigRetrieverOptions configRetrieverOptions = new ConfigRetrieverOptions()
.addStore(fileStore);
ConfigRetriever configRetriever = ConfigRetriever.create(vertx, configRetrieverOptions);
configRetriever.getConfig(ar -> {
if (ar.succeeded()) {
JsonObject configJson = ar.result();
EventBus eventBus = vertx.eventBus();
eventBus.<JsonObject>consumer(GET_USEARCH_DOCS).handler(getDocumentService(searchRepositoryService, configJson));
Logger.info("Completed search service event processing");
} else {
Logger.error("Failed to retrieve the config");
}
});
}
private Handler<Message<JsonObject>> getDocumentService(SearchRepository searchRepositoryService, JsonObject configJson) {
return requestMessage -> vertx.<String>executeBlocking(future -> {
try {
//I need to incorporate the logic here that adds futures to list and composes the compositefuture
/*
//Below is my logic to populate the future list
WebClient client = WebClient.create(vertx);
List<Future> futureList = new ArrayList<>();
for (Object collection : searchRepositoryService.findAllCollections(configJson).getJsonArray(SOLR_CLOUD_COLLECTION).getList()) {
Future<String> future1 = client.post(8983, "127.0.0.1", "/solr/" + collection + "/query")
.expect(ResponsePredicate.SC_OK)
.sendJsonObject(requestMessage.body())
.map(HttpResponse::bodyAsString).recover(error -> {
System.out.println(error.getMessage());
return Future.succeededFuture();
});
futureList.add(future1);
}
//Below is the CompositeFuture logic, but the logic and construct does not make sense to me. What goes as first and second argument of executeBlocking method
/*CompositeFuture.join(futureList)
.onSuccess(result -> {
result.list().forEach( x -> {
if(x != null){
requestMessage.reply(result.result());
}
}
);
})
.onFailure(error -> {
System.out.println("We should not fail");
})
*/
future.complete("DAO returns a Json String");
} catch (Exception e) {
future.fail(e);
}
}, result -> {
if (result.succeeded()) {
requestMessage.reply(result.result());
} else {
requestMessage.reply(result.cause()
.toString());
}
});
}
}
I was able to use the org.springframework.web.reactive.function.client.WebClient calls to compose my search result from multiple web client calls, as against using Future<io.vertx.ext.web.client.WebClient> with CompositeFuture.
I was trying to avoid mixing Springboot and Vertx, but unfortunately Vertx CompositeFuture did not work here:
//This method supplies the parameter for the future.complete(..) line in getDocumentService(SearchRepository,JsonObject)
private List<JsonObject> findByQueryParamsAndDataSources(SearchRepository searchRepositoryService,
JsonObject configJson,
JsonObject requestMessage)
throws SolrServerException, IOException {
List<JsonObject> searchResultList = new ArrayList<>();
for (Object collection : searchRepositoryService.findAllCollections(configJson).getJsonArray(SOLR_CLOUD_COLLECTION).getList()) {
searchResultList.add(new JsonObject(doSearchPerCollection(collection.toString(), requestMessage.toString())));
}
return aggregateMultiCollectionSearchResults(searchResultList);
}
public String doSearchPerCollection(String collection, String message) {
org.springframework.web.reactive.function.client.WebClient client =
org.springframework.web.reactive.function.client.WebClient.create();
return client.post()
.uri("http://127.0.0.1:8983/solr/" + collection + "/query")
.contentType(MediaType.APPLICATION_JSON)
.accept(MediaType.APPLICATION_JSON)
.body(BodyInserters.fromValue(message.toString()))
.retrieve()
.bodyToMono(String.class)
.block();
}
private List<JsonObject> aggregateMultiCollectionSearchResults(List<JsonObject> searchList){
//TODO: Search result aggregation
return searchList;
}
My use case is the second verticle should make multiple vertx web client calls and should combine the responses.
If an API call falls, I want to log the error and still continue processing and merging responses from other calls.
Please, any help on how my code above could be adaptable to handle the use case?
I am looking at vertx CompositeFuture, but no headway or useful example seen yet!

What you are looking for can done with Future coordination with a little bit of additional handling:
CompositeFuture.join(future1, future2, future3).onComplete(ar -> {
if (ar.succeeded()) {
// All succeeded
} else {
// All completed and at least one failed
}
});
The join composition waits until all futures are completed, either with a success or a failure.
CompositeFuture.join
takes several futures arguments (up to 6) and returns a future that is succeeded when all the futures are succeeded, and failed when all the futures are completed and at least one of them is failed
Using join you will wait for all Futures to complete, the issue is that if one of them fails you will not be able to obtain response from others as CompositeFuture will be failed. To avoid this you should add Future<T> recover(Function<Throwable, Future<T>> mapper) on each of your Futures in which you should log the error and pass an empty response so that the future does not fail.
Here is short example:
Future<String> response1 = client.post(8887, "localhost", "work").expect(ResponsePredicate.SC_OK).send()
.map(HttpResponse::bodyAsString).recover(error -> {
System.out.println(error.getMessage());
return Future.succeededFuture();
});
Future<String> response2 = client.post(8887, "localhost", "error").expect(ResponsePredicate.SC_OK).send()
map(HttpResponse::bodyAsString).recover(error -> {
System.out.println(error.getMessage());
return Future.succeededFuture();
});
CompositeFuture.join(response2, response1)
.onSuccess(result -> {
result.list().forEach(x -> {
if(x != null) {
System.out.println(x);
}
});
})
.onFailure(error -> {
System.out.println("We should not fail");
});
Edit 1:
Limit for CompositeFuture.join(Future...) is 6 Futures, in the case you need more you can use: CompositeFuture.join(Arrays.asList(future1, future2, future3)); where you can pass unlimited number of futures.

Short circuiting the chain of CompletionStage

I am using Java 8 and I have a chain of CompletionStage that I am trying to run.
I don't want to use join() or get(), I want to explicity complete the CompletionStage.
I am trying to run two database queries, the second has dependency on the result of the first query. I am starting a database transaction using session, running write query1, write query2 and only if both are successful I want to commit the transaction or else roll it back.
The transaction and session are part of Neo4j java API https://neo4j.com/docs/api/java-driver/current/org/neo4j/driver/async/AsyncSession.html#writeTransactionAsync-org.neo4j.driver.async.AsyncTransactionWork-
After running both queries success/failure I want to close the session(a standard database practice)
Here is psuedo code -
DB Session starts transaction
run Write Query1
run Write Query2
if both are successful
commit transaction
else
rollback transaction
close session
What I want to achieve is if query1/query2 fails then it should just rollback transaction and close session.
Query 1 can also throw a CustomException if the result from Query1 is incorrect(less than some threshold). In this case it should rollback transaction. I am rolling back transaction in the exceptionally block for each query.
The happy path works fine in the code below, but when I want to throw CustomException, the Query2 block is not called and even the Completable.allOf is never called.
CompletableFuture<String> firstFuture = new CompletableFuture();
CompletableFuture<String> secondFuture = new CompletableFuture();
CompletableFuture<String> lastFuture = new CompletableFuture();
//Lambda that executes transaction
TransactionWork<CompletionStage<String>> runTransactionWork = transaction -> {
//Write Query1
transaction.runAsync("DB WRITE QUERY1") //Running Write Query 1
.thenCompose(someFunctionThatReturnsCompletionStage)
.thenApply(val -> {
//throw CustomException if value less then threshold
if(val < threshold){
throw new CustomException("Incorrect value found");
}else{
//if value is correct then complete future
firstFuture.complete(val);
}
firstQuery.complete(val);
}).exceptionally(error -> {
//Since failure occured in Query1 want to roll back
transaction.rollbackAsync();
firstFuture.completeExceptionally(error);
throw new RuntimeException("There has been an error in first query " + error.getMessage());
});
//after the first write query is done then run the second write query
firstFuture.thenCompose(val -> transaction.runAsync("DB Write QUERY 2"))
.thenCompose(someFunctionThatReturnsCompletionStage)
.thenApply(val -> {
//if value is correct then complete
secondFuture.complete(val);
}
}).exceptionally(error -> {
//Incase of failure in Query2 want to roll back
transaction.rollbackAsync();
secondFuture.completeExceptionally(error);
throw new RuntimeException("There has been an error in second query " + error.getMessage());
});
//wait for both to complete and then complete the last future
CompletableFuture.allOf(firstFuture, secondFuture)
.handle((empty, ex) -> {
if(ex != null){
lastFuture.completeExceptionally(ex);
}else{
//commit the transaction
transaction.commitAsync();
lastFuture.complete("OK");
}
return lastFuture;
});
return lastFuture;
}
//Create a database session
Session session = driver.session();
//runTransactionWork is lambda that has access to transaction
session.writeTransactionAsync(runTransactionWork)
.handle((val, err) -> {
if(val != null){
session.closeAsync();
//send message to some broker about success
}else{
//fail logic
}
});
How can I achieve short circuiting the exception to ensure the transaction is rolled back and it directly goes to exception block on session.
These are my observations about the code blocks that are called based on different use cases, note these are based on debug points that I have placed in the code -
Happy path - firstFuture(success) -> secondFuture(success) -> LastFuture (success) -> session block success called (works fine)
First Future fail - firstFuture(failed due to exception) -> secondFuture(never called) -> LastFuture(never called) -> session block failure(never called)
Second Future fail - firstFuture(success) -> secondFuture(failed due to exception) -> LastFuture(never called) -> session block failure(never called)
I want #2 and #3 to work as well and the respective transaction should be rolled back and session should be closed.
My question is if is why does the exeption part from handle of allOf does not get called when one of the future completesExceptionally ?

When you throw that CustomException, firstFuture is not completed. As a matter of fact, nothing happens to it. Because it is not completed (successfully), this:
firstFuture.thenCompose...
will not be executed. The documentation of thenCompose says:
When this stage completes normally, the given function is invoked with this stage's result as the argument...
Since this is not the case, that code is obviously not going to be triggered. Because of that, nothing in turn happens to secondFuture either, so CompletableFuture::allOf has to do exactly zero. May be a simplified example will help:
public class CF {
public static void main(String[] args) {
CompletableFuture<Void> one = CompletableFuture.runAsync(CF::db1);
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
System.out.println(one.isCompletedExceptionally());
CompletableFuture<Void> two = one.thenRun(CF::db2);
System.out.println("first is done : " + FIRST_FUTURE.isDone());
System.out.println("second is done : " + SECOND_FUTURE.isDone());
CompletableFuture.allOf(FIRST_FUTURE, SECOND_FUTURE).thenRun(() -> {
System.out.println("allOf");
});
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
}
private static final boolean FAIL = true;
private static final CompletableFuture<String> FIRST_FUTURE = new CompletableFuture<>();
private static final CompletableFuture<String> SECOND_FUTURE = new CompletableFuture<>();
private static void db1() {
if(FAIL) {
throw new RuntimeException("failed one");
} else {
FIRST_FUTURE.complete("42");
}
}
private static void db2() {
System.out.println("Running");
SECOND_FUTURE.complete("42");
}
}
If you run this, you will notice that nothing gets printed...
Unfortunately I am not familiar with Neo4j, but you can most probably adjust this example to your needs:
public class CF {
public static void main(String[] args) {
CompletableFuture<Void> one = CompletableFuture.runAsync(CF::db1);
CompletableFuture<Void> terminal =
one.whenComplete((ok, th) -> {
if(th != null || FIRST_FUTURE.isCompletedExceptionally()) {
// no need to schedule the second one, need to rollback whatever the first one did
// transaction.rollbackAsync();
System.out.println("rollback because first one failed");
LAST_FUTURE.completeExceptionally(new RuntimeException("because first one failed"));
} else {
CompletableFuture<Void> two = CompletableFuture.runAsync(CF::db2);
two.whenComplete((ok2, th2) -> {
if(th2 != null || SECOND_FUTURE.isCompletedExceptionally()) {
System.out.println("rollback because second one failed");
// transaction.rollbackAsync();
LAST_FUTURE.completeExceptionally(new RuntimeException("because second one failed"));
} else {
LAST_FUTURE.complete("OK");
}
});
}
});
// simulate that someone will call this
terminal.join();
System.out.println(LAST_FUTURE.join());
}
private static final boolean FAIL_ONE = false;
private static final boolean FAIL_TWO = true;
private static final CompletableFuture<String> FIRST_FUTURE = new CompletableFuture<>();
private static final CompletableFuture<String> SECOND_FUTURE = new CompletableFuture<>();
private static final CompletableFuture<String> LAST_FUTURE = new CompletableFuture<>();
private static void db1() {
if(FAIL_ONE) {
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
RuntimeException ex = new RuntimeException("failed one");;
FIRST_FUTURE.completeExceptionally(ex);
} else {
FIRST_FUTURE.complete("42");
}
}
private static void db2() {
if(FAIL_TWO) {
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
RuntimeException ex = new RuntimeException("failed one");;
SECOND_FUTURE.completeExceptionally(ex);
} else {
SECOND_FUTURE.complete("42");
}
}
}

How to make Vertx MongoClient operation synchronous yet not blocking event loop in Java?

I am trying to save a new document to MongoDB using the Vertx MongoClient as follows:
MongoDBConnection.mongoClient.save("booking", query, res -> {
if(res.succeeded()) {
documentID = res.result();
System.out.println("MongoDB inserted successfully. + document ID is : " + documentID);
}
else {
System.out.println("MongoDB insertion failed.");
}
});
if(documentID != null) {
// MongoDB document insertion successful. Reply with a booking ID
String resMsg = "A confirmed booking has been successfully created with booking id as " + documentID +
". An email has also been triggered to the shared email id " + emailID;
documentID = null;
return new JsonObject().put("fulfillmentText", resMsg);
}
else {
// return intent response
documentID = null;
return new JsonObject().put("fulfillmentText",
"There is some issues while booking the shipment. Please start afreash.");
}
The above code successfully writes the query jsonObject to MongoDB collection booking. However, the function which contains this code always returns with There is some issues while booking the shipment. Please start afreash.
This is happening probably because the MongoClient save() handler "res" is asynchronous. But, I want to return conditional responses based on successful save() operation and on failed save operation.
How to achieve it in Vertx Java?

Your assumption is correct, you dont wait for the async response from the database. What you can do, is to wrap it in a Future like this:
public Future<JsonObject> save() {
Future<JsonObject> future = Future.future();
MongoDBConnection.mongoClient.save("booking", query, res -> {
if(res.succeeded()) {
documentID = res.result();
if(documentID != null) {
System.out.println("MongoDB inserted successfully. + document ID is : " + documentID);
String resMsg = "A confirmed booking has been successfully created with booking id as " + documentID +
". An email has also been triggered to the shared email id " + emailID;
future.complete(new JsonObject().put("fulfillmentText", resMsg));
}else{
future.complete(new JsonObject().put("fulfillmentText",
"There is some issues while booking the shipment. Please start afreash."))
}
} else {
System.out.println("MongoDB insertion failed.");
future.fail(res.cause());
}
});
return future;
}
Then i assume you have and endpoint that eventually calls this, eg:
router.route("/book").handler(this::addBooking);
... then you can call the save method and serve a different response based on the result
public void addBooking(RoutingContext ctx){
save().setHandler(h -> {
if(h.succeeded()){
ctx.response().end(h.result());
}else{
ctx.response().setStatusCode(500).end(h.cause());
}
})
}

You can use RxJava 2 and a reactive Mongo Client (io.vertx.reactivex.ext.mongo.MongoClient)
Here is a code snippet:
Deployer
public class Deployer extends AbstractVerticle {
private static final Logger logger = getLogger(Deployer.class);
#Override
public void start(Future<Void> startFuture) {
DeploymentOptions options = new DeploymentOptions().setConfig(config());
JsonObject mongoConfig = new JsonObject()
.put("connection_string",
String.format("mongodb://%s:%s#%s:%d/%s",
config().getString("mongodb.username"),
config().getString("mongodb.password"),
config().getString("mongodb.host"),
config().getInteger("mongodb.port"),
config().getString("mongodb.database.name")));
MongoClient client = MongoClient.createShared(vertx, mongoConfig);
RxHelper.deployVerticle(vertx, new BookingsStorage(client), options)
.subscribe(e -> {
logger.info("Successfully Deployed");
startFuture.complete();
}, error -> {
logger.error("Failed to Deployed", error);
startFuture.fail(error);
});
}
}
BookingsStorage
public class BookingsStorage extends AbstractVerticle {
private MongoClient mongoClient;
public BookingsStorage(MongoClient mongoClient) {
this.mongoClient = mongoClient;
}
#Override
public void start() {
var eventBus = vertx.eventBus();
eventBus.consumer("GET_ALL_BOOKINGS_ADDRESS", this::getAllBookings);
}
private void getAllBookings(Message msg) {
mongoClient.rxFindWithOptions("GET_ALL_BOOKINGS_COLLECTION", new JsonObject(), sortByDate())
.subscribe(bookings -> {
// do something with bookings
msg.reply(bookings);
},
error -> {
fail(msg, error);
}
);
}
private void fail(Message msg, Throwable error) {
msg.fail(500, "An unexpected error occurred: " + error.getMessage());
}
private FindOptions sortByDate() {
return new FindOptions().setSort(new JsonObject().put("date", 1));
}
}
HttpRouterVerticle
// inside a router handler:
vertx.eventBus().rxSend("GET_ALL_BOOKINGS_ADDRESS", new JsonObject())
.subscribe(bookings -> {
// do something with bookings
},
e -> {
// handle error
});

Rxjava Android how to use the Zip operator

I am having a lot of trouble understanding the zip operator in RxJava for my android project.
Problem
I need to be able to send a network request to upload a video
Then i need to send a network request to upload a picture to go with it
finally i need to add a description and use the responses from the previous two requests to upload the location urls of the video and picture along with the description to my server.
I assumed that the zip operator would be perfect for this task as I understood we could take the response of two observables (video and picture requests) and use them for my final task.
But I cant seem to get this to occur how I envision it.
I am looking for someone to answer how this can be done conceptually with a bit of psuedo code.
Thank you

Zip operator strictly pairs emitted items from observables. It waits for both (or more) items to arrive then merges them. So yes this would be suitable for your needs.
I would use Func2 to chain the result from the first two observables.
Notice this approach would be simpler if you use Retrofit since its api interface may return an observable. Otherwise you would need to create your own observable.
// assuming each observable returns response in the form of String
Observable<String> movOb = Observable.create(...);
// if you use Retrofit
Observable<String> picOb = RetrofitApiManager.getService().uploadPic(...),
Observable.zip(movOb, picOb, new Func2<String, String, MyResult>() {
#Override
public MyResult call(String movieUploadResponse, String picUploadResponse) {
// analyze both responses, upload them to another server
// and return this method with a MyResult type
return myResult;
}
}
)
// continue chaining this observable with subscriber
// or use it for something else

A small example:
val observableOne = Observable.just("Hello", "World")
val observableTwo = Observable.just("Bye", "Friends")
val zipper = BiFunction<String, String, String> { first, second -> "$first - $second" }
Observable.zip(observableOne, observableTwo, zipper)
.subscribe { println(it) }
This will print:
Hello - Bye
World - Friends
In BiFunction<String, String, String> the first String the type of the first observable, the second String is the type of the second observable, the third String represents the type of the return of your zipper function.
I made a small example that calls two real endpoints using zip in this blog post

Here I have an example that I did using Zip in asynchronous way, just in case you´re curious
/**
* Since every observable into the zip is created to subscribeOn a diferent thread, it´s means all of them will run in parallel.
* By default Rx is not async, only if you explicitly use subscribeOn.
*/
#Test
public void testAsyncZip() {
scheduler = Schedulers.newThread();
scheduler1 = Schedulers.newThread();
scheduler2 = Schedulers.newThread();
long start = System.currentTimeMillis();
Observable.zip(obAsyncString(), obAsyncString1(), obAsyncString2(), (s, s2, s3) -> s.concat(s2)
.concat(s3))
.subscribe(result -> showResult("Async in:", start, result));
}
/**
* In this example the the three observables will be emitted sequentially and the three items will be passed to the pipeline
*/
#Test
public void testZip() {
long start = System.currentTimeMillis();
Observable.zip(obString(), obString1(), obString2(), (s, s2, s3) -> s.concat(s2)
.concat(s3))
.subscribe(result -> showResult("Sync in:", start, result));
}
public void showResult(String transactionType, long start, String result) {
System.out.println(result + " " +
transactionType + String.valueOf(System.currentTimeMillis() - start));
}
public Observable<String> obString() {
return Observable.just("")
.doOnNext(val -> {
System.out.println("Thread " + Thread.currentThread()
.getName());
})
.map(val -> "Hello");
}
public Observable<String> obString1() {
return Observable.just("")
.doOnNext(val -> {
System.out.println("Thread " + Thread.currentThread()
.getName());
})
.map(val -> " World");
}
public Observable<String> obString2() {
return Observable.just("")
.doOnNext(val -> {
System.out.println("Thread " + Thread.currentThread()
.getName());
})
.map(val -> "!");
}
public Observable<String> obAsyncString() {
return Observable.just("")
.observeOn(scheduler)
.doOnNext(val -> {
System.out.println("Thread " + Thread.currentThread()
.getName());
})
.map(val -> "Hello");
}
public Observable<String> obAsyncString1() {
return Observable.just("")
.observeOn(scheduler1)
.doOnNext(val -> {
System.out.println("Thread " + Thread.currentThread()
.getName());
})
.map(val -> " World");
}
public Observable<String> obAsyncString2() {
return Observable.just("")
.observeOn(scheduler2)
.doOnNext(val -> {
System.out.println("Thread " + Thread.currentThread()
.getName());
})
.map(val -> "!");
}
You can see more examples here https://github.com/politrons/reactive

zip operator allow you to compose a result from results of two different observable.
You 'll have to give am lambda that will create a result from datas emitted by each observable.
Observable<MovieResponse> movies = ...
Observable<PictureResponse> picture = ...
Observable<Response> response = movies.zipWith(picture, (movie, pic) -> {
return new Response("description", movie.getName(), pic.getUrl());
});

i have been searching for a simple answer on how to use the Zip operator, and what to do with the Observables i create to pass them to it, i was wondering if i should call subscribe() for every observable or not, non of these answers were simple to find, i had to figure it out by my self, so here is a simple example for using Zip operator on 2 Observables :
#Test
public void zipOperator() throws Exception {
List<Integer> indexes = Arrays.asList(0, 1, 2, 3, 4);
List<String> letters = Arrays.asList("a", "b", "c", "d", "e");
Observable<Integer> indexesObservable = Observable.fromIterable(indexes);
Observable<String> lettersObservable = Observable.fromIterable(letters);
Observable.zip(indexesObservable, lettersObservable, mergeEmittedItems())
.subscribe(printMergedItems());
}
#NonNull
private BiFunction<Integer, String, String> mergeEmittedItems() {
return new BiFunction<Integer, String, String>() {
#Override
public String apply(Integer index, String letter) throws Exception {
return "[" + index + "] " + letter;
}
};
}
#NonNull
private Consumer<String> printMergedItems() {
return new Consumer<String>() {
#Override
public void accept(String s) throws Exception {
System.out.println(s);
}
};
}
the printed result is :
[0] a
[1] b
[2] c
[3] d
[4] e
the final answers to the questions that where in my head were as follows
the Observables passed to the zip() method just need to be created only, they do not need to have any subscribers to them, only creating them is enough ... if you want any observable to run on a scheduler, you can specify this for that Observable ... i also tried the zip() operator on Observables where they should wait for there result, and the Consumable of the zip() was triggered only when both results where ready (which is the expected behavior)

This is my implementation using Single.zip and rxJava2
I tried to make it as easy to understand as possible
//
// API Client Interface
//
#GET(ServicesConstants.API_PREFIX + "questions/{id}/")
Single<Response<ResponseGeneric<List<ResponseQuestion>>>> getBaseQuestions(#Path("id") int personId);
#GET(ServicesConstants.API_PREFIX + "physician/{id}/")
Single<Response<ResponseGeneric<List<ResponsePhysician>>>> getPhysicianInfo(#Path("id") int personId);
//
// API middle layer - NOTE: I had feedback that the Single.create is not needed (but I haven't yet spent the time to improve it)
//
public Single<List<ResponsePhysician>> getPhysicianInfo(int personId) {
return Single.create(subscriber -> {
apiClient.getPhysicianInfo(appId)
.subscribeOn(Schedulers.io())
.observeOn(Schedulers.io())
.subscribe(response -> {
ResponseGeneric<List<ResponsePhysician>> responseBody = response.body();
if(responseBody != null && responseBody.statusCode == 1) {
if (!subscriber.isDisposed()) subscriber.onSuccess(responseBody.data);
} else if(response.body() != null && response.body().status != null ){
if (!subscriber.isDisposed()) subscriber.onError(new Throwable(response.body().status));
} else {
if (!subscriber.isDisposed()) subscriber.onError(new Throwable(response.message()));
}
}, throwable -> {
throwable.printStackTrace();
if(!subscriber.isDisposed()) subscriber.onError(throwable);
});
});
}
public Single<List<ResponseQuestion>> getHealthQuestions(int personId){
return Single.create(subscriber -> {
apiClient.getBaseQuestions(personId)
.subscribeOn(Schedulers.io())
.observeOn(Schedulers.io())
.subscribe(response -> {
ResponseGeneric<List<ResponseQuestion>> responseBody = response.body();
if(responseBody != null && responseBody.data != null) {
if (!subscriber.isDisposed()) subscriber.onSuccess(response.body().data);
} else if(response.body() != null && response.body().status != null ){
if (!subscriber.isDisposed()) subscriber.onError(new Throwable(response.body().status));
} else {
if (!subscriber.isDisposed()) subscriber.onError(new Throwable(response.message()));
}
}, throwable -> {
throwable.printStackTrace();
if(!subscriber.isDisposed()) subscriber.onError(throwable);
});
});
}
//please note that ResponseGeneric is just an outer wrapper of the returned data - common to all API's in this project
public class ResponseGeneric<T> {
#SerializedName("Status")
public String status;
#SerializedName("StatusCode")
public float statusCode;
#SerializedName("Data")
public T data;
}
//
// API end-use layer - this gets close to the UI so notice the oberver is set for main thread
//
private static class MergedResponse{// this is just a POJO to store all the responses in one object
public List<ResponseQuestion> listQuestions;
public List<ResponsePhysician> listPhysicians;
public MergedResponse(List<ResponseQuestion> listQuestions, List<ResponsePhysician> listPhysicians){
this.listQuestions = listQuestions;
this.listPhysicians = listPhysicians;
}
}
// example of Single.zip() - calls getHealthQuestions() and getPhysicianInfo() from API Middle Layer
private void downloadHealthQuestions(int personId) {
addRxSubscription(Single
.zip(getHealthQuestions(personId), getPhysicianInfo(personId), MergedResponse::new)
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(response -> {
if(response != null) {
Timber.i(" - total health questions downloaded %d", response.listQuestions.size());
Timber.i(" - physicians downloaded %d", response.listPhysicians.size());
if (response.listPhysicians != null && response.listPhysicians.size()>0) {
// do your stuff to process response data
}
if (response.listQuestions != null && response.listQuestions.size()>0) {
// do your stuff to process response data
}
} else {
// process error - show message
}
}, error -> {
// process error - show network error message
}));
}

You use the zip from rxjava with Java 8:
Observable<MovieResponse> movies = ...
Observable<PictureResponse> picture = ...
Observable<ZipResponse> response = Observable.zip(movies, picture, ZipResponse::new);
class ZipResponse {
private MovieResponse movieResponse;
private PictureResponse pictureResponse;
ZipResponse(MovieResponse movieResponse, PictureResponse pictureResponse) {
this.movieResponse = movieResponse;
this.pictureResponse = pictureResponse;
}
public MovieResponse getMovieResponse() {
return movieResponse;
}
public void setMovieResponse(MovieResponse movieResponse) {
this.movieResponse= movieResponse;
}
public PictureResponse getPictureResponse() {
return pictureResponse;
}
public void setPictureResponse(PictureResponse pictureResponse) {
this.pictureResponse= pictureResponse;
}
}

You can use .zipWith operator for Observable chains.
If uploadMovies() and uploadPictures() return Observable,
uploadMovies()
.zipWith(uploadPictures()) { m, p ->
"$m with $p were uploaded"
}
.subscribe { print(it) }