I am trying to wait for processor.processFiles() to complete, the method returns void and it is an #Async method. The busy wait logic does not cause the process to wait for method to complete.
Can anybody please point out what am i missing?
try{
filesList.forEach(files -> {
List<CompletableFuture<Void>> completableFutures = new ArrayList<>();
files.forEach(file-> {
CompletableFuture<Void> completableFuture = CompletableFuture.runAsync(() ->
processor.processFiles());
completableFutures.add(completableFuture);
});
while(true) {
Thread.sleep(5000);
boolean isComplete = completableFutures.stream().allMatch(result -> result.isDone() == true);
if(isComplete){
break;
}
LOGGER.info("processing the file...");
}
});
}
catch(Exception e){
}
finally{
closeConnections();
}
I think you've overcomplicated things.
fileList.flatMap(List::stream).parallel().forEach(file -> processor.processFiles());
The forEach will run in parallel, and return when all of the files have been processed.
At the very least, don't use side effects to populate a List.
List<CompletableFuture<Void>> completableFutures = files.stream().map(
file -> CompletableFuture.runAsync(() -> processor.processFiles());
).collect( Collectors.toList());
I agree with the comment.
CompletableFuture<Void> all = CompletableFuture.allOf( completableFutures );
Then you can use get which will wait until the tasks are completed.
Another way to do this, that would skip the List + CompletableFuture.allOf and just return a single completable future.
CompletableFuture<Void> all = files.stream().map(
file -> CompletableFuture.runAsync(
() -> processor.processFiles()
)
).collect(
Collectors.reducing(
CompletableFuture.completedFuture(null), CompletableFuture::allOf
)
);
That will map file to a CompletableFuture then merge all of the resulting completable futures into a single completable future. You can call .get on it and it will return when everything is finished.
Hi I am bit new to async programming and CompletableFuture. I want to run a list of task in asynchronously (using CompletableFuture) and update the result to multiple objects.
Once all the tasks in the list updates the objects, I want my method to execute next lines synchronously.
I tried CompletableFuture.allOf() which Returns a new CompletableFuture that is completed when all of the given CompletableFutures are complete.
But in my case, I get it before all of the given CompletableFutures are complete. Please guide me if I am missing anything.
List<MyData> data = getFromAMethod();
OutputObject outputObject = new OutputObject();
List<AnotherValue> valueList1 = new ArrayList<>();
List<Value> valueList2 = new ArrayList<>();
for (MyData data : dataList) {
CompletableFuture<Void> requestCompletableFuture =
CompletableFuture.supplyAsync(
() -> executeMyMethod(data))
.exceptionally(throwable -> {
throw new SomeCustomException<>();
})
.thenAccept(
outputFromMyMethod -> {
if(outputObject.getSomeField()==null){
outputObject,setSomeField(outputFromMyMethod.getField);
}
valueList1.add(outputFromMyMethod.getValue1());
valueList2.add(outputFromMyMethod.getValue2());
});
completableFutures.add(requestCompletableFuture);
}
CompletableFuture<Void> allOfcompletableFuture =
CompletableFuture.allOf(completableFutures.toArray(new CompletableFuture[0]));
allOfcompletableFuture
.whenComplete(
(result, exception) -> {
if (exception != null) {
throw new SomeCustomException();
} else {
outputObject.setValueList1(valueList1);
outputObject.setValueList2(valueList2);
}
})
.join();
// next lines will be executed by main thread
taskOutput.setDisplayOutputs(ffxFeeDisplayOutputList);
In most of the runs valueList1 and valueList2 will not be getting updated with all the values, also I am not having any exceptions during these run.
I have also tried with get() method call after whenComplete still I am facing same issue.
I really appreciate any help you can provide.
I have a use case in which I want to filter out few elements in the list based on a Network call that I perform on the element. To accomplish this I am using streams, filter and Completable Future. The goal is to do async execution so that the operation becomes efficient. The pseudo code for this is mentioned below.
public List<Integer> afterFilteringList(List<Integer> initialList){
List<Integer> afterFilteringList =initialList.stream().filter(element -> {
boolean valid = true;
try{
valid = makeNetworkCallAndCheck().get();
} catch (Exception e) {
}
return valid;
}).collect(Collectors.toList());
return afterFilteringList;
}
public CompletableFuture<Boolean> makeNetworkCallAndCheck(Integer value){
return CompletableFuture.completedFuture(resultOfNetWorkCall(value);
}
The question I am having over here is, Am I doing this operation in an Async way itself?(As I am using 'get' function within the filter will it block the execution and make it sequential only) Or Is there a better way of doing this in Async way using Completable Future and Filters in Java 8.
When you call get immediately, you are indeed destroying the benefit of asynchronous execution. The solution is to collect all asynchronous jobs first, before joining.
public List<Integer> afterFilteringList(List<Integer> initialList){
Map<Integer,CompletableFuture<Boolean>> jobs = initialList.stream()
.collect(Collectors.toMap(Function.identity(), this::makeNetworkCallAndCheck));
return initialList.stream()
.filter(element -> jobs.get(element).join())
.collect(Collectors.toList());
}
public CompletableFuture<Boolean> makeNetworkCallAndCheck(Integer value){
return CompletableFuture.supplyAsync(() -> resultOfNetWorkCall(value));
}
Of course, the method makeNetworkCallAndCheck has to initiate a truly asynchronous operation as well. Calling a method synchronously and returning a completedFuture is not sufficient. I provided a simple exemplary asynchronous operation here, but for I/O operations, you likely want to provide your own Executor, tailored to the number of simultaneous connections you want to allow.
If you use get(), it will not be Async
get(): Waits if necessary for this future to complete, and then returns its result.
If you want to process all the request in Async. You can use CompletetableFuture.allOf()
public List<Integer> filterList(List<Integer> initialList){
List<Integer> filteredList = Collections.synchronizedList(new ArrayList());
AtomicInteger atomicInteger = new AtomicInteger(0);
CompletableFuture[] completableFutures = new CompletableFuture[initialList.size()];
initialList.forEach(x->{
completableFutures[atomicInteger.getAndIncrement()] = CompletableFuture
.runAsync(()->{
if(makeNetworkCallAndCheck(x)){
filteredList.add(x);
}
});
});
CompletableFuture.allOf(completableFutures).join();
return filteredList;
}
private Boolean makeNetworkCallAndCheck(Integer value){
// TODO: write the logic;
return true;
}
Collection.parallelStream() is an easy way to do the async stuff for a collection. You can modify your code as the following:
public List<Integer> afterFilteringList(List<Integer> initialList){
List<Integer> afterFilteringList =initialList
.parallelStream()
.filter(this::makeNetworkCallAndCheck)
.collect(Collectors.toList());
return afterFilteringList;
}
public Boolean makeNetworkCallAndCheck(Integer value){
return resultOfNetWorkCall(value);
}
You can customize your own executor by this way. And the result order is guaranteed according to this.
I have write the following code to verify my what I said.
public class DemoApplication {
public static void main(String[] args) throws ExecutionException, InterruptedException {
ForkJoinPool forkJoinPool = new ForkJoinPool(50);
final List<Integer> integers = new ArrayList<>();
for (int i = 0; i < 50; i++) {
integers.add(i);
}
long before = System.currentTimeMillis();
List<Integer> items = forkJoinPool.submit(() ->
integers
.parallelStream()
.filter(it -> {
try {
Thread.sleep(10000);
} catch (InterruptedException e) {
e.printStackTrace();
}
return true;
})
.collect(Collectors.toList()))
.get();
long after = System.currentTimeMillis();
System.out.println(after - before);
}
}
I create my own ForkJoinPool, and it takes me 10019 milliseconds to finish 50 jobs in parallel although each one costs 10000 milliseconds.
I deal with Streams of CompletableFutures. These take different times to complete. Those taking longer block stream processing while others might already have completed (and I know about Parallel Streams)
Therefore I would like to reorder items in a Stream (e.g. with a buffer) to move completed Futures ahead.
For example, this code blocks stream processing if one getUser call takes long
public static Boolean isValid(User user) { ... }
emails.stream()
// not using ::
// getUser() returns CompletableFuture<User>
.map( e -> getUser(e))
// this line blocks Stream processing
.filter( userF -> isValid( userF.get()) )
.map( f -> f.thenApply(User::getName))
and I would like to have something like
emails.stream()
.map( e -> getUser(e))
// this moves Futures into a bounded buffer
// and puts those finished first
// like CompletionService [1]
// and returns a Stream again
.collect(FutureReorderer.collector())
// this is not the same Stream but
// the one created by FutureReorderer.collector()
.filter( userF -> isValid( userF.get()) )
.map( f -> f.thenApply(User::getName))
[1] For example CompletionService https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ExecutorCompletionService.html returns completed tasks when calling take() and blocks otherwise. But CompletionService does not take futures, would one need to do cs.sumbit( () -> f.get() ) ?
How would I do that?
[Edit]
Changed example to include filter()
Added comment
Added CompletionService link
Having more context would definitely help in tailoring the answer - I have a feeling that problem is somewhere else and can be solved in an easier way.
But if your question is how to somehow keep completed futures at the beginning, there are few options:
Sorting the Stream using a custom Comparator:
.sorted(Comparator.comparing(f -> !f.isDone()))
Keep in mind that isDone returns true not only when a future completes successfully.
Storing futures in a PriorityQueue
PriorityQueue<CompletableFuture<String>> queue
= new PriorityQueue<>(Comparator.comparing(f -> !f.isDone()));
when polling elements, the queue will be returning elements according to their provided ordering.
Here it is in action:
PriorityQueue<CompletableFuture<String>> queue
= new PriorityQueue<>(Comparator.comparing(f -> !f.isDone()));
queue.add(CompletableFuture.supplyAsync(() -> {
try {
Thread.sleep(Integer.MAX_VALUE);
} catch (InterruptedException e) { }
return "42";
}));
queue.add(CompletableFuture.completedFuture("completed"));
queue.poll(); // "completed"
queue.poll(); // still going on
It's important to remember that if you do want to convert PriorityQueue to Stream, you can't do this simply using stream() - this will not preserve the priority order.
This is the right way to go:
Stream.generate(queue::poll).limit(queue.size())
I assume the requirements in OP is execute getUser concurrently and process the result Futures by completion order. Here is solution by ExecutorCompletionService:
final CompletionService<User> ecs = new ExecutorCompletionService<>(executor);
emails.stream().map(e -> ecs.submit(() -> getUser(e).get()))
.collect(Collectors.collectingAndThen(Collectors.toList(), fs -> fs.stream())) // collect the future list for concurrent execution
.map(f -> {
try {
return ecs.take().get();
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
}
})
.filter(u -> isValid(u)).map(User::getName)... //TODO;
Or:
final BlockingQueue<Future<User>> queue = new ArrayBlockingQueue<>(emails.size());
final CompletionService<User> ecs = new ExecutorCompletionService<>(executor, queue);
emails.stream().forEach(e -> ecs.submit(() -> getUser(e).get()));
IntStream.range(0, emails.size())
.mapToObj(i -> {
try {
return queue.poll().get();
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
}
})
.filter(u -> isValid(u)).map(User::getName);
It's simple but not straightforward.
I am trying to convert List<CompletableFuture<X>> to CompletableFuture<List<T>>. This is quite useful as when you have many asynchronous tasks and you need to get results of all of them.
If any of them fails then the final future fails. This is how I have implemented:
public static <T> CompletableFuture<List<T>> sequence2(List<CompletableFuture<T>> com, ExecutorService exec) {
if(com.isEmpty()){
throw new IllegalArgumentException();
}
Stream<? extends CompletableFuture<T>> stream = com.stream();
CompletableFuture<List<T>> init = CompletableFuture.completedFuture(new ArrayList<T>());
return stream.reduce(init, (ls, fut) -> ls.thenComposeAsync(x -> fut.thenApplyAsync(y -> {
x.add(y);
return x;
},exec),exec), (a, b) -> a.thenCombineAsync(b,(ls1,ls2)-> {
ls1.addAll(ls2);
return ls1;
},exec));
}
To run it:
ExecutorService executorService = Executors.newCachedThreadPool();
Stream<CompletableFuture<Integer>> que = IntStream.range(0,100000).boxed().map(x -> CompletableFuture.supplyAsync(() -> {
try {
Thread.sleep((long) (Math.random() * 10));
} catch (InterruptedException e) {
e.printStackTrace();
}
return x;
}, executorService));
CompletableFuture<List<Integer>> sequence = sequence2(que.collect(Collectors.toList()), executorService);
If any of them fails then it fails. It gives output as expected even if there are a million futures. The problem I have is: Say if there are more than 5000 futures and if any of them fails, I get a StackOverflowError:
Exception in thread "pool-1-thread-2611" java.lang.StackOverflowError
at
java.util.concurrent.CompletableFuture.internalComplete(CompletableFuture.java:210)
at
java.util.concurrent.CompletableFuture$ThenCompose.run(CompletableFuture.java:1487)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:193)
at
java.util.concurrent.CompletableFuture.internalComplete(CompletableFuture.java:210)
at
java.util.concurrent.CompletableFuture$ThenCompose.run(CompletableFuture.java:1487)
What am I doing it wrong?
Note: The above returned future fails right when any of the future fails. The accepted answer should also take this point.
Use CompletableFuture.allOf(...):
static<T> CompletableFuture<List<T>> sequence(List<CompletableFuture<T>> com) {
return CompletableFuture.allOf(com.toArray(new CompletableFuture<?>[0]))
.thenApply(v -> com.stream()
.map(CompletableFuture::join)
.collect(Collectors.toList())
);
}
A few comments on your implementation:
Your use of .thenComposeAsync, .thenApplyAsync and .thenCombineAsync is likely not doing what you expect. These ...Async methods run the function supplied to them in a separate thread. So, in your case, you are causing the addition of the new item to the list to run in the supplied executor. There is no need to stuff light-weight operations into a cached thread executor. Do not use thenXXXXAsync methods without a good reason.
Additionally, reduce should not be used to accumulate into mutable containers. Even though it might work correctly when the stream is sequential, it will fail if the stream were to be made parallel. To perform mutable reduction, use .collect instead.
If you want to complete the entire computation exceptionally immediately after the first failure, do the following in your sequence method:
CompletableFuture<List<T>> result = CompletableFuture.allOf(com.toArray(new CompletableFuture<?>[0]))
.thenApply(v -> com.stream()
.map(CompletableFuture::join)
.collect(Collectors.toList())
);
com.forEach(f -> f.whenComplete((t, ex) -> {
if (ex != null) {
result.completeExceptionally(ex);
}
}));
return result;
If, additionally, you want to cancel the remaining operations on first failure, add exec.shutdownNow(); right after result.completeExceptionally(ex);. This, of course, assumes that exec only exist for this one computation. If it doesn't, you'll have to loop over and cancel each remaining Future individually.
You can get Spotify's CompletableFutures library and use allAsList method. I think it's inspired from Guava's Futures.allAsList method.
public static <T> CompletableFuture<List<T>> allAsList(
List<? extends CompletionStage<? extends T>> stages) {
And here is a simple implementation if you don't want to use a library:
public <T> CompletableFuture<List<T>> allAsList(final List<CompletableFuture<T>> futures) {
return CompletableFuture.allOf(
futures.toArray(new CompletableFuture[futures.size()])
).thenApply(ignored ->
futures.stream().map(CompletableFuture::join).collect(Collectors.toList())
);
}
As Misha has pointed out, you are overusing …Async operations. Further, you are composing a complex chain of operations modelling a dependency which doesn’t reflect your program logic:
you create a job x which depends on the first and second job of your list
you create a job x+1 which depends on job x and the third job of your list
you create a job x+2 which depends on job x+1 and the 4th job of your list
…
you create a job x+5000 which depends on job x+4999 and the last job of your list
Then, canceling (explicitly or due to an exception) this recursively composed job might be performed recursively and might fail with a StackOverflowError. That’s implementation-dependent.
As already shown by Misha, there is a method, allOf which allows you to model your original intention, to define one job which depends on all jobs of your list.
However, it’s worth noting that even that isn’t necessary. Since you are using an unbounded thread pool executor, you can simply post an asynchronous job collecting the results into a list and you are done. Waiting for the completion is implied by asking for the result of each job anyway.
ExecutorService executorService = Executors.newCachedThreadPool();
List<CompletableFuture<Integer>> que = IntStream.range(0, 100000)
.mapToObj(x -> CompletableFuture.supplyAsync(() -> {
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos((long)(Math.random()*10)));
return x;
}, executorService)).collect(Collectors.toList());
CompletableFuture<List<Integer>> sequence = CompletableFuture.supplyAsync(
() -> que.stream().map(CompletableFuture::join).collect(Collectors.toList()),
executorService);
Using methods for composing dependent operations are important, when the number of threads is limited and the jobs may spawn additional asynchronous jobs, to avoid having waiting jobs stealing threads from jobs which have to complete first, but neither is the case here.
In this specific case one job simply iterating over this large number of prerequisite jobs and waiting if necessary may be more efficient than modelling this large number of dependencies and having each job to notify the dependent job about the completion.
To add upto the accepted answer by #Misha, it can be further expanded as a collector:
public static <T> Collector<CompletableFuture<T>, ?, CompletableFuture<List<T>>> sequenceCollector() {
return Collectors.collectingAndThen(Collectors.toList(), com -> sequence(com));
}
Now you can:
Stream<CompletableFuture<Integer>> stream = Stream.of(
CompletableFuture.completedFuture(1),
CompletableFuture.completedFuture(2),
CompletableFuture.completedFuture(3)
);
CompletableFuture<List<Integer>> ans = stream.collect(sequenceCollector());
An example sequence operation using thenCombine on CompletableFuture
public<T> CompletableFuture<List<T>> sequence(List<CompletableFuture<T>> com){
CompletableFuture<List<T>> identity = CompletableFuture.completedFuture(new ArrayList<T>());
BiFunction<CompletableFuture<List<T>>,CompletableFuture<T>,CompletableFuture<List<T>>> combineToList =
(acc,next) -> acc.thenCombine(next,(a,b) -> { a.add(b); return a;});
BinaryOperator<CompletableFuture<List<T>>> combineLists = (a,b)-> a.thenCombine(b,(l1,l2)-> { l1.addAll(l2); return l1;}) ;
return com.stream()
.reduce(identity,
combineToList,
combineLists);
}
}
If you don't mind using 3rd party libraries cyclops-react (I am the author) has a set of utility methods for CompletableFutures (and Optionals, Streams etc)
List<CompletableFuture<String>> listOfFutures;
CompletableFuture<ListX<String>> sequence =CompletableFutures.sequence(listOfFutures);
Disclaimer: This will not completely answer the initial question. It will lack the "fail all if one fails" part. However, I can't answer the actual, more generic question, because it was closed as a duplicate of this one: Java 8 CompletableFuture.allOf(...) with Collection or List. So I will answer here:
How to convert List<CompletableFuture<V>> to
CompletableFuture<List<V>> using Java 8's stream API?
Summary: Use the following:
private <V> CompletableFuture<List<V>> sequence(List<CompletableFuture<V>> listOfFutures) {
CompletableFuture<List<V>> identity = CompletableFuture.completedFuture(new ArrayList<>());
BiFunction<CompletableFuture<List<V>>, CompletableFuture<V>, CompletableFuture<List<V>>> accumulator = (futureList, futureValue) ->
futureValue.thenCombine(futureList, (value, list) -> {
List<V> newList = new ArrayList<>(list.size() + 1);
newList.addAll(list);
newList.add(value);
return newList;
});
BinaryOperator<CompletableFuture<List<V>>> combiner = (futureList1, futureList2) -> futureList1.thenCombine(futureList2, (list1, list2) -> {
List<V> newList = new ArrayList<>(list1.size() + list2.size());
newList.addAll(list1);
newList.addAll(list2);
return newList;
});
return listOfFutures.stream().reduce(identity, accumulator, combiner);
}
Example usage:
List<CompletableFuture<String>> listOfFutures = IntStream.range(0, numThreads)
.mapToObj(i -> loadData(i, executor)).collect(toList());
CompletableFuture<List<String>> futureList = sequence(listOfFutures);
Complete Example:
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.Executor;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ThreadLocalRandom;
import java.util.function.BiFunction;
import java.util.function.BinaryOperator;
import java.util.stream.IntStream;
import static java.util.stream.Collectors.toList;
public class ListOfFuturesToFutureOfList {
public static void main(String[] args) {
ListOfFuturesToFutureOfList test = new ListOfFuturesToFutureOfList();
test.load(10);
}
public void load(int numThreads) {
final ExecutorService executor = Executors.newFixedThreadPool(numThreads);
List<CompletableFuture<String>> listOfFutures = IntStream.range(0, numThreads)
.mapToObj(i -> loadData(i, executor)).collect(toList());
CompletableFuture<List<String>> futureList = sequence(listOfFutures);
System.out.println("Future complete before blocking? " + futureList.isDone());
// this will block until all futures are completed
List<String> data = futureList.join();
System.out.println("Loaded data: " + data);
System.out.println("Future complete after blocking? " + futureList.isDone());
executor.shutdown();
}
public CompletableFuture<String> loadData(int dataPoint, Executor executor) {
return CompletableFuture.supplyAsync(() -> {
ThreadLocalRandom rnd = ThreadLocalRandom.current();
System.out.println("Starting to load test data " + dataPoint);
try {
Thread.sleep(500 + rnd.nextInt(1500));
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Successfully loaded test data " + dataPoint);
return "data " + dataPoint;
}, executor);
}
private <V> CompletableFuture<List<V>> sequence(List<CompletableFuture<V>> listOfFutures) {
CompletableFuture<List<V>> identity = CompletableFuture.completedFuture(new ArrayList<>());
BiFunction<CompletableFuture<List<V>>, CompletableFuture<V>, CompletableFuture<List<V>>> accumulator = (futureList, futureValue) ->
futureValue.thenCombine(futureList, (value, list) -> {
List<V> newList = new ArrayList<>(list.size() + 1);
newList.addAll(list);
newList.add(value);
return newList;
});
BinaryOperator<CompletableFuture<List<V>>> combiner = (futureList1, futureList2) -> futureList1.thenCombine(futureList2, (list1, list2) -> {
List<V> newList = new ArrayList<>(list1.size() + list2.size());
newList.addAll(list1);
newList.addAll(list2);
return newList;
});
return listOfFutures.stream().reduce(identity, accumulator, combiner);
}
}
Your task could be done easily like following,
final List<CompletableFuture<Module> futures =...
CompletableFuture.allOf(futures.stream().toArray(CompletableFuture[]::new)).join();
In addition to Spotify Futures library you might try my code locate here: https://github.com/vsilaev/java-async-await/blob/master/net.tascalate.async.examples/src/main/java/net/tascalate/concurrent/CompletionStages.java (has a dependencies to other classes in same package)
It implements a logic to return "at least N out of M" CompletionStage-s with a policy how much errors it's allowed to tolerate. There are convinient methods for all/any cases, plus cancellation policy for the remaining futures, plus the code deals with CompletionStage-s (interface) rather than CompletableFuture (concrete class).
Javaslang has a very convenient Future API. It also allows to make a future of collection out of a collection of futures.
List<Future<String>> listOfFutures = ...
Future<Seq<String>> futureOfList = Future.sequence(listOfFutures);
See http://static.javadoc.io/io.javaslang/javaslang/2.0.5/javaslang/concurrent/Future.html#sequence-java.lang.Iterable-