We Keep Coding

How to wait for #Async annotated method to complete execution completely for all the elements of List<String> which has 130k element then execute next

I have used ThreadPoolTaskExecutor class to call my #Async annotated method as number of api calls are more then 130k+ so I am trying to achieve it through async api calls using executor framework, but once the list through which I am streaming and making async calls gets completed then next flow is getting executed, but here I want to wait until for all async calls gets completed. Which means I want to wait until I will get api response for all 130k+ call which has been made async while streaming the list
public void downloadData(Map.Entry<String, String> entry, String downloadPath,
Locale locale, ApiClient apiClient, Task task,
Set<Locale> downloadFailedLocales) {
String targetFileName = entry.getKey() + ".xml";
Path filePath = null;
try {
filePath = getTargetDestination(downloadPath, "2", entry.getKey(), targetFileName);
MultiValueMap<String, String> queryParameters = restelApiClient.fetchQueryParameters();
if (downloadPath != null && !downloadFileService.localFileExists(filePath)) {
fetchCountryAndHotelList(entry.getValue(), filePath, task, downloadFailedLocales, locale, queryParameters);
//After fetching hotelList proceed for fetching hotelInfo from hotelList xml Data
if (entry.getKey().equals(HotelConstants.HOTEL_LIST)) {
//fetching hotelCodes from downloaded xml of hotelList, to make API calls for hotelInfo
List<String> hotelInfoArray = getHotelCodeList(filePath);
AtomicInteger hotelCounter = new AtomicInteger();
String hotelInfoXml = apiClient.getApiClientSettings().getEndpoints()
.get(HotelConstants.HOTEL_INFO);
/*Fetching data from HotelInfo API Async but once it will stream the hotelinfo list then next flow of code execute and it won't wait all api calls to be made and get the response back. */
hotelInfoArray.stream().forEach(hotel -> {
StringBuilder fileName = new StringBuilder();
fileName.append(HotelConstants.HOTEL_INFO).append(hotelCounter.getAndIncrement()).append(".xml");
Path path = getTargetDestination(downloadPath, "2", HotelConstants.HOTEL_INFO,
fileName.toString());
StringBuilder hotelCode = new StringBuilder();
hotelCode.append("<codigo>").append(hotel).append("</codigo>");
String xml = String.format(hotelInfoXml).replace("<codigo></codigo>", hotelCode);
try {
hotelDataFetchThreadService.fetchHotelInfo(xml, path, task, downloadFailedLocales, locale, queryParameters);
} catch (DownloadFailedException e) {
log.info("Download failed for hotel code {} with exception {}", hotel, e);
downloadFileService.deleteIncompleteFiles(path);
}
});
}
} else {
log.info("file already exist skipping downloading again");
}
} catch (DownloadException e) {
downloadFileService.deleteIncompleteFiles(filePath);
log.info("Download failed for endpoint {} with exception {}", entry.getKey(), e);
} catch (DownloadFailedException e) {
throw new RuntimeException(e);
}
}
/*
This method make api call and write the xml response in local file in async way
*/
#Async("TestExecutor")
public void fetchHotelInfo(String xml, Path path, Task task, Set<Locale> downloadFailedLocales, Locale locale,
MultiValueMap<String, String> queryParameters) throws DownloadFailedException {
Flux<DataBuffer> bufferedData;
try {
// log.info("using thread {}", Thread.currentThread().getName());
bufferedData = apiClient.getWebClient()
.uri(uriBuilder -> uriBuilder
.queryParams(queryParameters)
.queryParam(HotelConstants.XML, xml.trim())
.build()
).retrieve()
.bodyToFlux(DataBuffer.class)
.retryWhen(Retry.fixedDelay(maxRetryAttempts, Duration.ofSeconds(maxRetryDelay))
.onRetryExhaustedThrow(
(RetryBackoffSpec retryBackoffSpec, Retry.RetrySignal retrySignal) -> {
throw new DownloadException(
"External Service failed to process after max retries");
}));
writeBufferDataToFile(bufferedData, path);
} catch (DownloadException e) {
downloadFileService.deleteIncompleteFiles(path);
downloadFailedLocales.add(locale);
if (locale.equals(task.getJob().getProvider().getDefaultLocale().getLocale())) {
throw new DownloadFailedException(
String.format("Network issue during download, Max retry reached: %s", e.getMessage()), e);
}
log.info("Download failed for with exception ", e);
}
}

Modify HashMap when CompletableFuture is finished

I have a multi-module system, where one module handles my database storage. This is the method which saves a document:
public CompletableFuture<?> runTransaction() {
return CompletableFuture.runAsync(() -> {
TransactionBody txnBody = (TransactionBody<String>) () -> {
MongoCollection<Document> collection = transaction.getDatabase().getCollection(transaction.getCollection().toString());
collection.insertOne(session, Document.parse(json));
return "Completed";
};
try {
session.withTransaction(txnBody);
} catch (MongoException ex) {
throw new UncheckedMongoException(ex);
}
});
}
the json instance is passed down in the object constructor. However, since this will be used by several modules, with each individual caching system, I'm trying to figure out how the caller can modify data structure, if this method completed without any errors.
For example
public void createClan(Transaction transaction, int id, int maxPlayers) {
MongoTransaction mongoTransaction = (MongoTransaction) transaction;
Clan clan = new Clan(id, maxPlayers);
String json = gson.toJson(clan);
TransactionExecutor executor = new MongoTransactionExecutor(mongoTransaction, json);
executor.runTransaction(); //Returns the completableFuture instance generated by the method. Modify hashmap here.
}
I've tried reading the docs, however it was a bit confusing, any help is appreciated!

As given in the comments, two options can be considered.
First option is to convert the async nature into sync nature using CompletableFuture#get. This way, the code execution is in a blocking context.
public void createClan(Transaction transaction, int id, int maxPlayers) {
MongoTransaction mongoTransaction = (MongoTransaction) transaction;
Clan clan = new Clan(id, maxPlayers);
String json = gson.toJson(clan);
TransactionExecutor executor = new MongoTransactionExecutor(mongoTransaction, json);
try {
Object obj = executor.runTransaction().get();
// HashMap update here
} catch(Exception e) {
//handle exceptions
}
}
Second option is to keep the async nature as is and chain using thenRun (there are many then options available). This way is more a non-blocking context.
public void createClan(Transaction transaction, int id, int maxPlayers) {
MongoTransaction mongoTransaction = (MongoTransaction) transaction;
final Clan clan = new Clan(id, maxPlayers);
String json = gson.toJson(clan);
TransactionExecutor executor = new MongoTransactionExecutor(mongoTransaction, json);
try {
executor.runTransaction().thenRun(() -> updateHashMap(clan));
} catch(Exception e) {
//handle exceptions
}
}

Short circuiting the chain of CompletionStage

I am using Java 8 and I have a chain of CompletionStage that I am trying to run.
I don't want to use join() or get(), I want to explicity complete the CompletionStage.
I am trying to run two database queries, the second has dependency on the result of the first query. I am starting a database transaction using session, running write query1, write query2 and only if both are successful I want to commit the transaction or else roll it back.
The transaction and session are part of Neo4j java API https://neo4j.com/docs/api/java-driver/current/org/neo4j/driver/async/AsyncSession.html#writeTransactionAsync-org.neo4j.driver.async.AsyncTransactionWork-
After running both queries success/failure I want to close the session(a standard database practice)
Here is psuedo code -
DB Session starts transaction
run Write Query1
run Write Query2
if both are successful
commit transaction
else
rollback transaction
close session
What I want to achieve is if query1/query2 fails then it should just rollback transaction and close session.
Query 1 can also throw a CustomException if the result from Query1 is incorrect(less than some threshold). In this case it should rollback transaction. I am rolling back transaction in the exceptionally block for each query.
The happy path works fine in the code below, but when I want to throw CustomException, the Query2 block is not called and even the Completable.allOf is never called.
CompletableFuture<String> firstFuture = new CompletableFuture();
CompletableFuture<String> secondFuture = new CompletableFuture();
CompletableFuture<String> lastFuture = new CompletableFuture();
//Lambda that executes transaction
TransactionWork<CompletionStage<String>> runTransactionWork = transaction -> {
//Write Query1
transaction.runAsync("DB WRITE QUERY1") //Running Write Query 1
.thenCompose(someFunctionThatReturnsCompletionStage)
.thenApply(val -> {
//throw CustomException if value less then threshold
if(val < threshold){
throw new CustomException("Incorrect value found");
}else{
//if value is correct then complete future
firstFuture.complete(val);
}
firstQuery.complete(val);
}).exceptionally(error -> {
//Since failure occured in Query1 want to roll back
transaction.rollbackAsync();
firstFuture.completeExceptionally(error);
throw new RuntimeException("There has been an error in first query " + error.getMessage());
});
//after the first write query is done then run the second write query
firstFuture.thenCompose(val -> transaction.runAsync("DB Write QUERY 2"))
.thenCompose(someFunctionThatReturnsCompletionStage)
.thenApply(val -> {
//if value is correct then complete
secondFuture.complete(val);
}
}).exceptionally(error -> {
//Incase of failure in Query2 want to roll back
transaction.rollbackAsync();
secondFuture.completeExceptionally(error);
throw new RuntimeException("There has been an error in second query " + error.getMessage());
});
//wait for both to complete and then complete the last future
CompletableFuture.allOf(firstFuture, secondFuture)
.handle((empty, ex) -> {
if(ex != null){
lastFuture.completeExceptionally(ex);
}else{
//commit the transaction
transaction.commitAsync();
lastFuture.complete("OK");
}
return lastFuture;
});
return lastFuture;
}
//Create a database session
Session session = driver.session();
//runTransactionWork is lambda that has access to transaction
session.writeTransactionAsync(runTransactionWork)
.handle((val, err) -> {
if(val != null){
session.closeAsync();
//send message to some broker about success
}else{
//fail logic
}
});
How can I achieve short circuiting the exception to ensure the transaction is rolled back and it directly goes to exception block on session.
These are my observations about the code blocks that are called based on different use cases, note these are based on debug points that I have placed in the code -
Happy path - firstFuture(success) -> secondFuture(success) -> LastFuture (success) -> session block success called (works fine)
First Future fail - firstFuture(failed due to exception) -> secondFuture(never called) -> LastFuture(never called) -> session block failure(never called)
Second Future fail - firstFuture(success) -> secondFuture(failed due to exception) -> LastFuture(never called) -> session block failure(never called)
I want #2 and #3 to work as well and the respective transaction should be rolled back and session should be closed.
My question is if is why does the exeption part from handle of allOf does not get called when one of the future completesExceptionally ?

When you throw that CustomException, firstFuture is not completed. As a matter of fact, nothing happens to it. Because it is not completed (successfully), this:
firstFuture.thenCompose...
will not be executed. The documentation of thenCompose says:
When this stage completes normally, the given function is invoked with this stage's result as the argument...
Since this is not the case, that code is obviously not going to be triggered. Because of that, nothing in turn happens to secondFuture either, so CompletableFuture::allOf has to do exactly zero. May be a simplified example will help:
public class CF {
public static void main(String[] args) {
CompletableFuture<Void> one = CompletableFuture.runAsync(CF::db1);
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
System.out.println(one.isCompletedExceptionally());
CompletableFuture<Void> two = one.thenRun(CF::db2);
System.out.println("first is done : " + FIRST_FUTURE.isDone());
System.out.println("second is done : " + SECOND_FUTURE.isDone());
CompletableFuture.allOf(FIRST_FUTURE, SECOND_FUTURE).thenRun(() -> {
System.out.println("allOf");
});
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
}
private static final boolean FAIL = true;
private static final CompletableFuture<String> FIRST_FUTURE = new CompletableFuture<>();
private static final CompletableFuture<String> SECOND_FUTURE = new CompletableFuture<>();
private static void db1() {
if(FAIL) {
throw new RuntimeException("failed one");
} else {
FIRST_FUTURE.complete("42");
}
}
private static void db2() {
System.out.println("Running");
SECOND_FUTURE.complete("42");
}
}
If you run this, you will notice that nothing gets printed...
Unfortunately I am not familiar with Neo4j, but you can most probably adjust this example to your needs:
public class CF {
public static void main(String[] args) {
CompletableFuture<Void> one = CompletableFuture.runAsync(CF::db1);
CompletableFuture<Void> terminal =
one.whenComplete((ok, th) -> {
if(th != null || FIRST_FUTURE.isCompletedExceptionally()) {
// no need to schedule the second one, need to rollback whatever the first one did
// transaction.rollbackAsync();
System.out.println("rollback because first one failed");
LAST_FUTURE.completeExceptionally(new RuntimeException("because first one failed"));
} else {
CompletableFuture<Void> two = CompletableFuture.runAsync(CF::db2);
two.whenComplete((ok2, th2) -> {
if(th2 != null || SECOND_FUTURE.isCompletedExceptionally()) {
System.out.println("rollback because second one failed");
// transaction.rollbackAsync();
LAST_FUTURE.completeExceptionally(new RuntimeException("because second one failed"));
} else {
LAST_FUTURE.complete("OK");
}
});
}
});
// simulate that someone will call this
terminal.join();
System.out.println(LAST_FUTURE.join());
}
private static final boolean FAIL_ONE = false;
private static final boolean FAIL_TWO = true;
private static final CompletableFuture<String> FIRST_FUTURE = new CompletableFuture<>();
private static final CompletableFuture<String> SECOND_FUTURE = new CompletableFuture<>();
private static final CompletableFuture<String> LAST_FUTURE = new CompletableFuture<>();
private static void db1() {
if(FAIL_ONE) {
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
RuntimeException ex = new RuntimeException("failed one");;
FIRST_FUTURE.completeExceptionally(ex);
} else {
FIRST_FUTURE.complete("42");
}
}
private static void db2() {
if(FAIL_TWO) {
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
RuntimeException ex = new RuntimeException("failed one");;
SECOND_FUTURE.completeExceptionally(ex);
} else {
SECOND_FUTURE.complete("42");
}
}
}

Unable to persist entity with managed executor in Quarkus

I am unable to persist an entity using ManagedExecutor in Quarkus.
managedExecutor.runAsync(() -> {
User user = User.findById(userId);
if (user == null) {
return;
}
BigDecimal amount;
try {
amount = new BigDecimal(split[1]);
} catch (NumberFormatException e) {
throw new BadRequestException("Invalid amount: " + split[1] + ", " + e.getMessage());
}
managedExecutor.runAsync(threadContext.contextualRunnable(new Runnable() {
#Override
#Transactional
#ActivateRequestContext
public void run() {
user.minimumAmount = amount;
user.persistAndFlush();
}
}));
sendMessage(userId, new NewMinimumAmountMessage(user.minimumAmount));
});
Right now there are no exceptions thrown and nothing after persistAndFlush() gets executed.
I have tried to put the persist in the initial runAsync but that also doesn't work.
The code here is in a websocket function which is annotated with #OnMessage, putting #Transactional here doesn't do anything.
Both the ManagedExecutor and the ThreadContext are injected.

I was able to fix it but I know it's not the proper way.
Response response = given()
.auth().oauth2(token)
.header("Bearer", token)
.header("Content-Type", "application/json")
.when().put("/api/user/amount/" + split[1])
.thenReturn();
I added a path in a resource and used the test function to call it.

Spring #Transactional and #Async

In my application, on the creation of a task, I need to make an API call to Google to create a google calendar event.
I decided to make that API call on a separate thread so that our client doesn't have to wait longer for the response.
#Override
#Transactional( rollbackFor = DataException.class )
public TaskResponseBean createTask( TaskCreationBean taskCreationBean, UserAccessDetails accessDetails )
throws DataException
{
String googleEventId = "";
try
{
TaskServiceUtil.validateInputBeforeCreatingTask(taskCreationBean, accessDetails);
MatterModel matterModel = matterService.giveMatterIfExistElseThrowException(taskCreationBean.getMatterId(),
owner);
//A task is unique for a user for a matter
taskCommons.throwExceptionIfTaskNameAlreadyExistForTheMatter(taskCreationBean.getTaskName().trim(), owner,
matterModel);
TaskModel savedTask = taskModelRepository.save(savableTask);
if( !NullEmptyUtils.isNull(savableTask.getDueDate()) )
{
final CreateEventBean createEventBean = getCreateEventBean(getEventParticipants(savedTask), savedTask);
calendarTrigerer.triggerEventCreation(createEventBean, savedTask.getId(), null,
GoogleCalendarTrigerer.EVENT_TYPE_CREATE);
}
// Keep track of the list of assignees of a task
if( taskCreationBean.getHaveAssignee() || taskCreationBean.getIsSelfAssigned() )
{
saveTaskAssignedHistory(savedTask, owner, savedTask.getAssignedTo(), false);
}
}
catch( DataException e )
{
LOGGER.error(GeneralConstants.ERROR, e);
if( !NullEmptyUtils.isNullOrEmpty(googleEventId) )
{
LOGGER.info("Deleting google event id {}", googleEventId);
googleCalendarService.deleteGoogleCalendarEvent(googleEventId);
}
throw e;
}
catch( Exception e )
{
LOGGER.error(GeneralConstants.ERROR, e);
if( !NullEmptyUtils.isNullOrEmpty(googleEventId) )
{
LOGGER.info("Deleting google event id {}", googleEventId);
googleCalendarService.deleteGoogleCalendarEvent(googleEventId);
}
throw new DataException(GeneralConstants.EXCEPTION, GeneralConstants.SOMETHING_WENT_WRONG,
HttpStatus.INTERNAL_SERVER_ERROR);
}
}
#Async
void triggerEventCreation( CreateEventBean createEventBean, Long taskId, String eventId, String eventType )
throws DataException
{
try
{
TaskModel taskModel = null;
if( !NullEmptyUtils.isNullOrEmpty(taskId) )
{
int retryCount = 0;
Optional<TaskModel> taskModelOptional = taskModelRepository.findByIdAndIsActiveTrue(taskId);
while( !taskModelOptional.isPresent() )
{
System.out.println("NOT PRESENT***********************************************");
taskModelOptional = taskModelRepository.findByIdAndIsActiveTrue(taskId);
if( retryCount++ > 50 )
{
throw new DataException(GeneralConstants.EXCEPTION, "Transaction is unable to commit",
HttpStatus.INTERNAL_SERVER_ERROR);
}
}
taskModel = taskModelOptional.get();
}
switch ( eventType )
{
case EVENT_TYPE_CREATE :
eventId = googleCalendarService.addGoogleCalendarEvent(createEventBean);
System.out.println("ADDED EVENT***********************************************" + eventId);
System.out.println("PRESENT***********************************************");
taskModel.setGoogleEventId(eventId);
taskModelRepository.save(taskModel);
break;
case EVENT_TYPE_DELETE :
NullEmptyUtils.throwExceptionIfInputIsNullOrEmpty(eventId);
googleCalendarService.deleteGoogleCalendarEvent(eventId);
taskModel.setGoogleEventId(null);
taskModel.setIsActive(false);
taskModelRepository.save(taskModel);
break;
case EVENT_TYPE_UPDATE :
NullEmptyUtils.throwExceptionIfInputIsNullOrEmpty(eventId);
NullEmptyUtils.throwExceptionIfInputIsNullOrEmpty(createEventBean);
taskModel.setGoogleEventId(
googleCalendarService.updateGoogleCalendarEvent(eventId, createEventBean));
taskModelRepository.save(taskModel);
break;
default :
throw new DataException(GeneralConstants.EXCEPTION, "Invalid eventType", HttpStatus.BAD_REQUEST);
}
}
catch( DataException e )
{
log.error(GeneralConstants.ERROR, e);
throw e;
}
catch( Exception e )
{
log.error(GeneralConstants.ERROR, e);
throw new DataException(GeneralConstants.EXCEPTION,
"Something went wrong while trigering create event action", HttpStatus.INTERNAL_SERVER_ERROR);
}
}
How I solved (make sure it works) while creating an event
In the separate thread, I will iterate and wait till the task gets created and then update it with the event id and save it.
But, a new problem arises, when updating the task, I already have its details in the database. In the update method, I will set updated values to TaskModel and do taskModelRepo.save() and in a separate thread I am calling google calendar update API and after successful API call, I have to update the corresponding TaskModel and save it.
The issue here is, sometimes when I fetch task by id after google API call is successful, I will get the TaskModel with non-updated values as the previous transaction is not committed yet.
So How to ensure the new thread runs only after the transaction of the method from which it is called is committed?

you can use the Google Guava Event Bus to solve this problem. It's a publish-subscribe model in which the producer is responsible for emitting the events, these events are then passed on to the event bus and are sent to all listeners that are subscribed to that event.
The listener, subscribes to an event and it is triggered when that event is posted from the producer, you could have a listener method run Synchronously or Asynchronously depending on the kind of event bus you use.
Here is the link : https://github.com/google/guava/wiki/EventBusExplained