We Keep Coding

CompletableFuture is Not waiting for child threads

I am trying to wait for processor.processFiles() to complete, the method returns void and it is an #Async method. The busy wait logic does not cause the process to wait for method to complete.
Can anybody please point out what am i missing?
try{
filesList.forEach(files -> {
List<CompletableFuture<Void>> completableFutures = new ArrayList<>();
files.forEach(file-> {
CompletableFuture<Void> completableFuture = CompletableFuture.runAsync(() ->
processor.processFiles());
completableFutures.add(completableFuture);
});
while(true) {
Thread.sleep(5000);
boolean isComplete = completableFutures.stream().allMatch(result -> result.isDone() == true);
if(isComplete){
break;
}
LOGGER.info("processing the file...");
}
});
}
catch(Exception e){
}
finally{
closeConnections();
}

I think you've overcomplicated things.
fileList.flatMap(List::stream).parallel().forEach(file -> processor.processFiles());
The forEach will run in parallel, and return when all of the files have been processed.
At the very least, don't use side effects to populate a List.
List<CompletableFuture<Void>> completableFutures = files.stream().map(
file -> CompletableFuture.runAsync(() -> processor.processFiles());
).collect( Collectors.toList());
I agree with the comment.
CompletableFuture<Void> all = CompletableFuture.allOf( completableFutures );
Then you can use get which will wait until the tasks are completed.
Another way to do this, that would skip the List + CompletableFuture.allOf and just return a single completable future.
CompletableFuture<Void> all = files.stream().map(
file -> CompletableFuture.runAsync(
() -> processor.processFiles()
)
).collect(
Collectors.reducing(
CompletableFuture.completedFuture(null), CompletableFuture::allOf
)
);
That will map file to a CompletableFuture then merge all of the resulting completable futures into a single completable future. You can call .get on it and it will return when everything is finished.

How to execute CompletableFuture function and get the result either which of them is done first?

i already have create 3 function that execute query to database, then i want to get each result and add into one List of object but the result is always empty, how can i do this properly? here is what i do :
i create 3 CompletableFuture function :
private CompletableFuture<List<TransSalesOrderOnlyResponseDto>> findPureUnAssignedSO() {
return CompletableFuture.supplyAsync(() -> iSalesOrderMapper.entityToSOOnlyDto(iTransSalesOrderQdslRepository.findUnAssignedSalesOrder()));
}
private CompletableFuture<List<TransSalesOrderOnlyResponseDto>> findSOHaveItemLeftOverOnly() {
return CompletableFuture.supplyAsync(() -> {
List<TransSalesOrder> transSalesOrders = iTransSalesOrderQdslRepository.findSOHaveLeftOverButDone();
return buildTransSalesOrdersResponseNew(transSalesOrders);
});
}
private CompletableFuture<List<TransSalesOrderOnlyResponseDto>> findSalesOrderWithBpsjInDeliveryOrder() {
return CompletableFuture.supplyAsync(() -> {
List<TransSalesOrder> transSalesOrders = iTransSalesOrderQdslRepository.findSalesOrderWithBpsjInDeliveryOrder();
return buildTransSalesOrdersBpsjOnlyResponseNew(transSalesOrders);
});
}
and then here is how i try to execute that 3 function :
ATTEMPT 1, using get() :
public List<TransSalesOrderOnlyResponseDto> findUnAssignedSO() {
CompletableFuture<List<TransSalesOrderOnlyResponseDto>> future = new CompletableFuture<>();
List<TransSalesOrderOnlyResponseDto> transSalesOrdersResponseNew = new ArrayList<>();
try {
transSalesOrdersResponseNew = findSOHaveItemLeftOverOnly().get();
transSalesOrdersResponseNew.addAll(findPureUnAssignedSO().get());
transSalesOrdersResponseNew.addAll(findSalesOrderWithBpsjInDeliveryOrder().get());
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
return transSalesOrdersResponseNew;
}
the result still empty
ATTEMPT 2 :
public List<TransSalesOrderOnlyResponseDto> findUnAssignedSO() {
List<TransSalesOrderOnlyResponseDto> transSalesOrdersResponseNew = new ArrayList<>();
CompletableFuture<List<TransSalesOrderOnlyResponseDto>> soHaveItemLeftOverOnly = findSOHaveItemLeftOverOnly();
CompletableFuture<List<TransSalesOrderOnlyResponseDto>> pureUnAssignedSO = findPureUnAssignedSO();
CompletableFuture<List<TransSalesOrderOnlyResponseDto>> salesOrderWithBpsjInDeliveryOrder = findSalesOrderWithBpsjInDeliveryOrder();
CompletableFuture.allOf(soHaveItemLeftOverOnly, pureUnAssignedSO, salesOrderWithBpsjInDeliveryOrder)
.thenRun(() -> {
transSalesOrdersResponseNew.addAll(soHaveItemLeftOverOnly.join());
transSalesOrdersResponseNew.addAll(pureUnAssignedSO.join());
transSalesOrdersResponseNew.addAll(salesOrderWithBpsjInDeliveryOrder.join());
});
}
return transSalesOrdersResponseNew;
}
the result is always empty if i do this, even i use .get() to block the result, how do i do completablefuture properly?

Both your attempts aren't working because you call an a completion stage without waiting for the result (I'm not sure about attempt number 1 though).
I don't know the signature of all the methods, but if somewhere you are returning a CompletionStage in the supplyAsync and you don't use thenCompose instead, the function will return ignoring the result of the CompletionStage
In Attempt number 2, it's easier to see where it's wrong. It's this part:
CompletableFuture.allOf(...).thenRun(() -> ...);
It doesn't matter what you do in the thenRun part. You don't wait anywhere for the result, so it will get to the return transSalesOrdersResponseNew; immediately, even if the function you have defined in the thenRun part hasn't finished yet.
Assuming that the methods findSOHaveItemLeftOverOnly, findPureUnAssignedSO and findSalesOrderWithBpsjInDeliveryOrder are correct (we cannot know that from the details you gave use), you could rewrite the code this way:
final List<TransSalesOrderOnlyResponseDto> transSalesOrdersResponseNew = ... ;
findSOHaveItemLeftOverOnly()
.thenAccept( transSalesOrdersResponseNew::addAll )
.thenCompose( v -> findPureUnAssignedSO() )
.thenAccept( transSalesOrdersResponseNew::addAll )
.thenCompose( v -> findSalesOrderWithBpsjInDeliveryOrder() )
.thenAccept( transSalesOrdersResponseNew::addAll )
.join();
return transSalesOrdersResponseNew;
Note that I'm using .thenCompose, this will use the result of the function as the next CompletionStage in the sequence. This way you won't lose result in the process.
You could also run the find methods in parallel (if the order doesn't matter) with CompletableFuture.allOf but in that case you need to make sure to use an implementation of List that's thread safe.
It would look like this:
final List<TransSalesOrderOnlyResponseDto> transSalesOrdersResponseNew = ... // Thread-safe list implementation;
CompletableFuture.allOf(
findSOHaveItemLeftOverOnly().thenAccept( transSalesOrdersResponseNew::addAll ),
findPureUnAssignedSO().thenAccept( transSalesOrdersResponseNew::addAll ),
findSalesOrderWithBpsjInDeliveryOrder().thenAccept( transSalesOrdersResponseNew::addAll )
).join();
return transSalesOrdersResponseNew;

CompleteableFuture Java 8 unusual behavior

I noticed some unusual behavior with CompleteableFutures in Java 8 with streaming.
String [] arr = new String[]{"abc", "def", "cde", "ghj"};
ExecutorService executorService = Executors.newFixedThreadPool(10);
List<String> lst =
Arrays.stream(arr)
.map(r ->
CompletableFuture.supplyAsync(() -> {
try {
Thread.sleep(5000);
return "e";
} catch (Exception e) {
e.printStackTrace();
}
return null;
}, executorService)
)
.map(CompletableFuture::join)
.collect(Collectors.toList());
This code above takes 4*5000 = 20 seconds to execute, so this means the futures are waiting on one another.
String [] arr = new String[]{"abc", "def", "cde", "ghj"};
ExecutorService executorService = Executors.newFixedThreadPool(10);
List<CompletableFuture<String>> lst =
Arrays.stream(arr)
.map(r ->
CompletableFuture.supplyAsync(() -> {
try {
Thread.sleep(5000);
return "d";
} catch (Exception e) {
e.printStackTrace();
}
return null;
}, executorService)
)
.collect(Collectors.toList());
List<String> s =
lst
.stream()
.map(CompletableFuture::join)
.collect(Collectors.toList());
System.out.println(s);
This code however, runs in 5 seconds meaning futures are running in parallel.
What I don't understand: In the second example I get a list of futures explicitly, then do a join, which takes 5 seconds, the first example I keep it streaming through and it seems to wait.
What's the reasoning behind this?

Streams don't necessarily do one stage, then the next. They can compose operations in any order they choose.
So for example,
Arrays.stream(array).map(e -> f(e)).map(e -> g(e)).collect(toList());
can end up being run the same way as
Arrays.stream(array).map(e -> g(f(e))).collect(toList());
...which would have the results you see: the futures are generated one at a time and immediately joined, instead of all being generated up front and then joined.
In point of fact, if you're not doing something async, it's usually more efficient to do it the second way. That way, the stream framework doesn't have to store all the results of f, then store all the results of g: it can only store the results of g(f(e)). The stream framework can't know you're doing async code, so it does the normal efficient thing.

I think the issue is with the second map function call in the original snippet. The map function is serial and hence calls the CF blocking function join for each one of the elements in the source array one after another.

How to reorder Stream of CompletableFutures?

I deal with Streams of CompletableFutures. These take different times to complete. Those taking longer block stream processing while others might already have completed (and I know about Parallel Streams)
Therefore I would like to reorder items in a Stream (e.g. with a buffer) to move completed Futures ahead.
For example, this code blocks stream processing if one getUser call takes long
public static Boolean isValid(User user) { ... }
emails.stream()
// not using ::
// getUser() returns CompletableFuture<User>
.map( e -> getUser(e))
// this line blocks Stream processing
.filter( userF -> isValid( userF.get()) )
.map( f -> f.thenApply(User::getName))
and I would like to have something like
emails.stream()
.map( e -> getUser(e))
// this moves Futures into a bounded buffer
// and puts those finished first
// like CompletionService [1]
// and returns a Stream again
.collect(FutureReorderer.collector())
// this is not the same Stream but
// the one created by FutureReorderer.collector()
.filter( userF -> isValid( userF.get()) )
.map( f -> f.thenApply(User::getName))
[1] For example CompletionService https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ExecutorCompletionService.html returns completed tasks when calling take() and blocks otherwise. But CompletionService does not take futures, would one need to do cs.sumbit( () -> f.get() ) ?
How would I do that?
[Edit]
Changed example to include filter()
Added comment
Added CompletionService link

Having more context would definitely help in tailoring the answer - I have a feeling that problem is somewhere else and can be solved in an easier way.
But if your question is how to somehow keep completed futures at the beginning, there are few options:
Sorting the Stream using a custom Comparator:
.sorted(Comparator.comparing(f -> !f.isDone()))
Keep in mind that isDone returns true not only when a future completes successfully.
Storing futures in a PriorityQueue
PriorityQueue<CompletableFuture<String>> queue
= new PriorityQueue<>(Comparator.comparing(f -> !f.isDone()));
when polling elements, the queue will be returning elements according to their provided ordering.
Here it is in action:
PriorityQueue<CompletableFuture<String>> queue
= new PriorityQueue<>(Comparator.comparing(f -> !f.isDone()));
queue.add(CompletableFuture.supplyAsync(() -> {
try {
Thread.sleep(Integer.MAX_VALUE);
} catch (InterruptedException e) { }
return "42";
}));
queue.add(CompletableFuture.completedFuture("completed"));
queue.poll(); // "completed"
queue.poll(); // still going on
It's important to remember that if you do want to convert PriorityQueue to Stream, you can't do this simply using stream() - this will not preserve the priority order.
This is the right way to go:
Stream.generate(queue::poll).limit(queue.size())

I assume the requirements in OP is execute getUser concurrently and process the result Futures by completion order. Here is solution by ExecutorCompletionService:
final CompletionService<User> ecs = new ExecutorCompletionService<>(executor);
emails.stream().map(e -> ecs.submit(() -> getUser(e).get()))
.collect(Collectors.collectingAndThen(Collectors.toList(), fs -> fs.stream())) // collect the future list for concurrent execution
.map(f -> {
try {
return ecs.take().get();
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
}
})
.filter(u -> isValid(u)).map(User::getName)... //TODO;
Or:
final BlockingQueue<Future<User>> queue = new ArrayBlockingQueue<>(emails.size());
final CompletionService<User> ecs = new ExecutorCompletionService<>(executor, queue);
emails.stream().forEach(e -> ecs.submit(() -> getUser(e).get()));
IntStream.range(0, emails.size())
.mapToObj(i -> {
try {
return queue.poll().get();
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
}
})
.filter(u -> isValid(u)).map(User::getName);
It's simple but not straightforward.

I want do something as future done order in CompletableFuture List

I have some service return CompletableFutures like this
Set<CompletableFuture<String>> futures = service.getSomething();
for (CompletableFuture<String> future : futures) {
System.out.println(future.get());
}
This code prints value iterate order. But I want fast result print first like using CompletionService.
Set<CompletableFuture<String>> futures = service.getSomething();
Set<CompletableFuture<String>> donefutures = new HashSet<>();
while (true) {
if (futures.equals(donefutures)) {
break;
}
futures
.stream()
.filter(f -> !donefutures.contains(f))
.filter(CompletableFuture::isDone)
.peek(donefutures::add)
.map(f -> {
try {
return f.get();
} catch (InterruptedException | ExecutionException e) {
return null;
}
})
.forEach(System.out::println);
Thread.sleep(100);
}
I tried this way. It's working. But I think really ugly. Is there better way?

You are working with CompletableFuture like with Future in a blocking way running an infinite loop. You have to specify callback function which will be invoked when your future completes.
So you can do something like this:
Set<CompletableFuture<String>> futures = service.getSomething();
futures.forEach(future -> future.whenComplete(
(result, throwable) -> System.out.println(result)
));
CompletableFuture
.allOf(futures.toArray(new CompletableFuture[0]))
.join();