Problem
Hi, I have a function where i going to return infinite stream of parallel (yes, it is much faster in that case) generated results. So obviously (or not) i used
Stream<Something> stream = Stream.generate(this::myGenerator).parallel()
It works, however ... it doesn't when i want to limit the result (everything is fine when the stream is sequential). I mean, it creates results when i make something like
stream.peek(System.out::println).limit(2).collect(Collectors.toList())
but even when peek output produces more than 10 elements, collect is still not finallized (generating is slow so those 10 can took even a minute)... and that is easy example. Actually, limiting those results is a future due the main expectation is to get only better than recent results until the user will kill the process (other case is to return first what i can make with throwing exception if nothing else will help [findFirst didn't, even when i had more elements on the console and no more results for about 30 sec]).
So, the question is...
how to copy with that? My idea was also to use RxJava, and there is another question - how to achieve similar result with that tool (or other).
Code sample
public Stream<Solution> generateSolutions() {
final Solution initialSolution = initialSolutionMaker.findSolution();
return Stream.concat(
Stream.of(initialSolution),
Stream.generate(continuousSolutionMaker::findSolution)
).parallel();
}
new Solver(instance).generateSolutions()
.map(Solution::getPurpose)
.peek(System.out::println)
.limit(5).collect(Collectors.toList());
Implementation of findSolution is not important.
It has some side effect like adding to solutions repo (singleton, sych etc..), but nothing more.
As explained in the already linked answer, the key point to an efficient parallel stream is to use a stream source already having an intrinsic size instead of using an unsized or even infinite stream and apply a limit on it. Injecting a size doesn’t work with the current implementation at all, while ensuring that a known size doesn’t get lost is much easier. Even if the exact size can’t be retained, like when applying a filter, the size still will be carried as an estimate size.
So instead of
Stream.generate(this::myGenerator).parallel()
.peek(System.out::println)
.limit(2)
.collect(Collectors.toList())
just use
IntStream.range(0, /* limit */ 2).unordered().parallel()
.mapToObj(unused -> this.myGenerator())
.peek(System.out::println)
.collect(Collectors.toList())
Or, closer to your sample code
public Stream<Solution> generateSolutions(int limit) {
final Solution initialSolution = initialSolutionMaker.findSolution();
return Stream.concat(
Stream.of(initialSolution),
IntStream.range(1, limit).unordered().parallel()
.mapToObj(unused -> continuousSolutionMaker.findSolution())
);
}
new Solver(instance).generateSolutions(5)
.map(Solution::getPurpose)
.peek(System.out::println)
.collect(Collectors.toList());
Unfortunately this is expected behavior. As I remember I've seen at least two topics on this matter, here is one of them.
The idea is that Stream.generate creates an unordered infinite stream and limit will not introduce the SIZED flag. Because of this when you spawn a parallel execution on that Stream, individual tasks have to sync their execution to see if they have reached that limit; by the time that sync happens there could be multiple elements already processed. For example this:
Stream.iterate(0, x -> x + 1)
.peek(System.out::println)
.parallel()
.limit(2)
.collect(Collectors.toList());
and this :
IntStream.of(1, 2, 3, 4)
.peek(System.out::println)
.parallel()
.limit(2)
.boxed()
.collect(Collectors.toList());
will always generate two elements in the List (Collectors.toList) and will always output two elements also (via peek).
On the other hand this:
Stream<Integer> stream = Stream.generate(new Random()::nextInt).parallel();
List<Integer> list = stream
.peek(x -> {
System.out.println("Before " + x);
})
.map(x -> {
System.out.println("Mapping x " + x);
return x;
})
.peek(x -> {
System.out.println("After " + x);
})
.limit(2)
.collect(Collectors.toList());
will generate two elements in the List, but it may process many more that later will be discarded by the limit. This is what you are actually seeing in your example.
The only sane way of going that (as far as I can tell) would be to create a custom Spliterator. I have not written many of them, but here is my attempt:
static class LimitingSpliterator<T> implements Spliterator<T> {
private int limit;
private final Supplier<T> generator;
private LimitingSpliterator(Supplier<T> generator, int limit) {
Preconditions.checkArgument(limit > 0);
this.limit = limit;
this.generator = Objects.requireNonNull(generator);
}
#Override
public boolean tryAdvance(Consumer<? super T> consumer) {
if (limit == 0) {
return false;
}
T nextElement = generator.get();
--limit;
consumer.accept(nextElement);
return true;
}
#Override
public LimitingSpliterator<T> trySplit() {
if (limit <= 1) {
return null;
}
int half = limit >> 1;
limit = limit - half;
return new LimitingSpliterator<>(generator, half);
}
#Override
public long estimateSize() {
return limit >> 1;
}
#Override
public int characteristics() {
return SIZED;
}
}
And the usage would be:
StreamSupport.stream(new LimitingSpliterator<>(new Random()::nextInt, 7), true)
.peek(System.out::println)
.collect(Collectors.toList());
I deal with Streams of CompletableFutures. These take different times to complete. Those taking longer block stream processing while others might already have completed (and I know about Parallel Streams)
Therefore I would like to reorder items in a Stream (e.g. with a buffer) to move completed Futures ahead.
For example, this code blocks stream processing if one getUser call takes long
public static Boolean isValid(User user) { ... }
emails.stream()
// not using ::
// getUser() returns CompletableFuture<User>
.map( e -> getUser(e))
// this line blocks Stream processing
.filter( userF -> isValid( userF.get()) )
.map( f -> f.thenApply(User::getName))
and I would like to have something like
emails.stream()
.map( e -> getUser(e))
// this moves Futures into a bounded buffer
// and puts those finished first
// like CompletionService [1]
// and returns a Stream again
.collect(FutureReorderer.collector())
// this is not the same Stream but
// the one created by FutureReorderer.collector()
.filter( userF -> isValid( userF.get()) )
.map( f -> f.thenApply(User::getName))
[1] For example CompletionService https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ExecutorCompletionService.html returns completed tasks when calling take() and blocks otherwise. But CompletionService does not take futures, would one need to do cs.sumbit( () -> f.get() ) ?
How would I do that?
[Edit]
Changed example to include filter()
Added comment
Added CompletionService link
Having more context would definitely help in tailoring the answer - I have a feeling that problem is somewhere else and can be solved in an easier way.
But if your question is how to somehow keep completed futures at the beginning, there are few options:
Sorting the Stream using a custom Comparator:
.sorted(Comparator.comparing(f -> !f.isDone()))
Keep in mind that isDone returns true not only when a future completes successfully.
Storing futures in a PriorityQueue
PriorityQueue<CompletableFuture<String>> queue
= new PriorityQueue<>(Comparator.comparing(f -> !f.isDone()));
when polling elements, the queue will be returning elements according to their provided ordering.
Here it is in action:
PriorityQueue<CompletableFuture<String>> queue
= new PriorityQueue<>(Comparator.comparing(f -> !f.isDone()));
queue.add(CompletableFuture.supplyAsync(() -> {
try {
Thread.sleep(Integer.MAX_VALUE);
} catch (InterruptedException e) { }
return "42";
}));
queue.add(CompletableFuture.completedFuture("completed"));
queue.poll(); // "completed"
queue.poll(); // still going on
It's important to remember that if you do want to convert PriorityQueue to Stream, you can't do this simply using stream() - this will not preserve the priority order.
This is the right way to go:
Stream.generate(queue::poll).limit(queue.size())
I assume the requirements in OP is execute getUser concurrently and process the result Futures by completion order. Here is solution by ExecutorCompletionService:
final CompletionService<User> ecs = new ExecutorCompletionService<>(executor);
emails.stream().map(e -> ecs.submit(() -> getUser(e).get()))
.collect(Collectors.collectingAndThen(Collectors.toList(), fs -> fs.stream())) // collect the future list for concurrent execution
.map(f -> {
try {
return ecs.take().get();
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
}
})
.filter(u -> isValid(u)).map(User::getName)... //TODO;
Or:
final BlockingQueue<Future<User>> queue = new ArrayBlockingQueue<>(emails.size());
final CompletionService<User> ecs = new ExecutorCompletionService<>(executor, queue);
emails.stream().forEach(e -> ecs.submit(() -> getUser(e).get()));
IntStream.range(0, emails.size())
.mapToObj(i -> {
try {
return queue.poll().get();
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
}
})
.filter(u -> isValid(u)).map(User::getName);
It's simple but not straightforward.
I'm in a bit of confusion right now, so I have a method that should return CompletableFuture<List<A>>
inside the method is:
CompletableFuture<List<String>> toReturn = asyncCall().thenApply(....)
.thenCompose(listOfStuff -> convertToList(listOfStuff.stream().map(
key -> asyncCall2(key)
.thenApply(optionalValue -> optionalValue.orElse(null))
).collect(Collectors.toList()));
and convertToList() simply joins futures to convert CompletableFuture<List<ComputableFuture<A>>> into CompletableFuture<List<A>>
Basically my intention is to filter null values that emerge from optionalValue.orElse(null) And it would be easy to do filter before collecting it all to list in the last line, but if I use it just before .collect it is working over CompletableFutures
I suspect there's a lot of restructuring I can do in my code.
EDIT:
private<T> CompletableFuture<List<T>> convertToList(List<CompletableFuture<T>> toConvert) {
return CompletableFuture.allOf(toConvert.toArray(new CompletableFuture[toConvert.size()]))
.thenApply(v -> toConvert.stream()
.map(CompletableFuture::join)
.collect(Collectors.toList())
);
}
The best way would probably be to change convertToList() so that it does not return a future of list, but of stream instead:
private <T> CompletableFuture<Stream<T>> convertToFutureOfStream(List<CompletableFuture<T>> toConvert) {
return CompletableFuture.allOf(toConvert.stream().toArray(CompletableFuture[]::new))
.thenApply(
v -> toConvert.stream()
.map(CompletableFuture::join)
);
}
This will be more reusable as the method will allow better chaining and will not force the caller to work with a list, while still allowing to easily get a list with a simple collect.
You can then simply filter that stream to remove empty optionals:
CompletableFuture<List<String>> toReturn = asyncCall()
.thenCompose(listOfStuff -> convertToFutureOfStream(
listOfStuff.stream()
.map(this::asyncCall2)
.collect(Collectors.toList())
)
.thenApply(stream ->
stream.filter(Optional::isPresent)
.map(Optional::get)
.collect(Collectors.toList())
)
);
You can even improve this a little further by changing convertToFutureOfStream() to take a stream as argument as well:
private <T> CompletableFuture<Stream<T>> convertToFutureOfStream(Stream<CompletableFuture<T>> stream) {
CompletableFuture<T>[] futures = stream.toArray(CompletableFuture[]::new);
return CompletableFuture.allOf(futures)
.thenApply(v -> Arrays.stream(futures).map(CompletableFuture::join));
}
(unfortunately this raises an unchecked assignment warning because of the array of generic types)
Which then gives
CompletableFuture<List<String>> toReturn = asyncCall()
.thenCompose(listOfStuff -> convertToFutureOfStream(
listOfStuff.stream().map(this::asyncCall2)
)
.thenApply(stream ->
stream.filter(Optional::isPresent)
.map(Optional::get)
.collect(Collectors.toList())
)
);
I have a list of users of unknown size. What i want is to query first 30 and update UI. Then i want to query all others by offset with step of 100 until i get last pack of users - should i use takeUntil here?) and when i get - i update UI by adding remaining users (combined with reduce() i belive).
This is my code:
final int INITIAL_OFFSET = 0;
final int INITIAL_LIMIT = 30;
// Loading first 30 users to immediately update UI (better UX)
getServerApi().getAllFriends(userId, "photo_50", INITIAL_OFFSET, INITIAL_LIMIT)
// Loading remaining users 100 by 100 and updating UI after all users been loaded
.flatMap(users -> {
AtomicInteger newOffset = new AtomicInteger(INITIAL_LIMIT);
return Observable.just(users)
.flatMap(users1 -> getServerApi().getAllFriends(userId, "photo_50", newOffset.get(), Config.DEFAULT_FRIEND_REQUEST_COUNT))
.subscribeOn(Schedulers.io())
.observeOn(Schedulers.io())
.collect(() -> new ArrayList<User>(), (b, s) -> {
b.addAll(s);
newOffset.set(newOffset.get() + Config.DEFAULT_FRIEND_REQUEST_COUNT);
})
.repeat()
.takeUntil(friends -> friends.size() == 0);
})
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(users -> getView().appendAllFriends(users),
throwable -> getView().setError(processFail(throwable, ServerApi.Action.GET_ALL_FRIENDS), false));
But seems i do something wrong because onNext is called each time the retrofit call is made.
Answering my own question. Adels answer is good, but i needed to have a single subscription (i'm using Nucleus MVP library) and i wanted to use collect() and takeUntil() instead of while loop (which requires blocking retrofit interface method).
Spent some hours and finally got it:
final int INITIAL_LIMIT = 30;
// Loading first 30 users to immediately update UI (better UX)
getServerApi().getAllFriends(userId, "photo_50", null, INITIAL_LIMIT)
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
// Updating UI 1st time or show error
.doOnNext(users -> getView().appendAllFriends(users))
.doOnError(throwable -> getView().setError(processFail(throwable, ServerApi.Action.GET_ALL_FRIENDS), false))
// Loading remaining users 100 by 100 and updating UI after all users been loaded
.flatMap(users -> {
AtomicInteger newOffset = new AtomicInteger(INITIAL_LIMIT);
ArrayList<User> remainingUsers = new ArrayList<>();
AtomicBoolean hasMore = new AtomicBoolean(true);
return Observable.just(users)
.flatMap(users1 -> getServerApi().getAllFriends(userId, "photo_50", newOffset.get(), Config.DEFAULT_FRIEND_REQUEST_COUNT))
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.collect(() -> remainingUsers, (b, s) -> {
// Needed for takeUntil
hasMore.set(b.addAll(s));
newOffset.set(newOffset.get() + Config.DEFAULT_FRIEND_REQUEST_COUNT);
})
.repeat()
.takeUntil(friends -> !hasMore.get())
// Grab all items emitted by collect()
.last()
// Updating UI last time
.doOnNext(users2 -> getView().appendAllFriends(users2));
})
.subscribe();
Maybe it will be useful for other people which are also using Nucleus.
// cache() will ensure that we load the first pack only once
Observable<Users> firstPack = firstPack().cache();
// this subscription is for updating the UI on the first batch
firstPack
.observeOn(AndroidSchedulers.mainThread())
.subscribe(x -> draw(x), e -> whoops(e));
// this subscription is for collecting all the stuff
// do whatever tricks you need to do with your backend API to get the full list of stuff
firstPack
.flatMap(fp -> rest(fp))
.observeOn(AndroidSchedulers.mainThread())
.subscribe(x -> allUsers(x), e -> whoops(e));
// I would do this in a simple while loop
Observable<List<User>> rest(List<User> firstPack) {
return Observable.create(sub -> {
final List<User> total = firstPack;
try {
while (!sub.isUnsubscribed()) {
final List<User> friends = api.getFriendsBlocking(total.size());
if (friends.isEmpty()) {
sub.onNext(total);
sub.onCompleted();
} else {
total.addAll(friends);
}
}
} catch(IOException e) {
sub.onError(e);
}
})
}
I am trying to convert List<CompletableFuture<X>> to CompletableFuture<List<T>>. This is quite useful as when you have many asynchronous tasks and you need to get results of all of them.
If any of them fails then the final future fails. This is how I have implemented:
public static <T> CompletableFuture<List<T>> sequence2(List<CompletableFuture<T>> com, ExecutorService exec) {
if(com.isEmpty()){
throw new IllegalArgumentException();
}
Stream<? extends CompletableFuture<T>> stream = com.stream();
CompletableFuture<List<T>> init = CompletableFuture.completedFuture(new ArrayList<T>());
return stream.reduce(init, (ls, fut) -> ls.thenComposeAsync(x -> fut.thenApplyAsync(y -> {
x.add(y);
return x;
},exec),exec), (a, b) -> a.thenCombineAsync(b,(ls1,ls2)-> {
ls1.addAll(ls2);
return ls1;
},exec));
}
To run it:
ExecutorService executorService = Executors.newCachedThreadPool();
Stream<CompletableFuture<Integer>> que = IntStream.range(0,100000).boxed().map(x -> CompletableFuture.supplyAsync(() -> {
try {
Thread.sleep((long) (Math.random() * 10));
} catch (InterruptedException e) {
e.printStackTrace();
}
return x;
}, executorService));
CompletableFuture<List<Integer>> sequence = sequence2(que.collect(Collectors.toList()), executorService);
If any of them fails then it fails. It gives output as expected even if there are a million futures. The problem I have is: Say if there are more than 5000 futures and if any of them fails, I get a StackOverflowError:
Exception in thread "pool-1-thread-2611" java.lang.StackOverflowError
at
java.util.concurrent.CompletableFuture.internalComplete(CompletableFuture.java:210)
at
java.util.concurrent.CompletableFuture$ThenCompose.run(CompletableFuture.java:1487)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:193)
at
java.util.concurrent.CompletableFuture.internalComplete(CompletableFuture.java:210)
at
java.util.concurrent.CompletableFuture$ThenCompose.run(CompletableFuture.java:1487)
What am I doing it wrong?
Note: The above returned future fails right when any of the future fails. The accepted answer should also take this point.
Use CompletableFuture.allOf(...):
static<T> CompletableFuture<List<T>> sequence(List<CompletableFuture<T>> com) {
return CompletableFuture.allOf(com.toArray(new CompletableFuture<?>[0]))
.thenApply(v -> com.stream()
.map(CompletableFuture::join)
.collect(Collectors.toList())
);
}
A few comments on your implementation:
Your use of .thenComposeAsync, .thenApplyAsync and .thenCombineAsync is likely not doing what you expect. These ...Async methods run the function supplied to them in a separate thread. So, in your case, you are causing the addition of the new item to the list to run in the supplied executor. There is no need to stuff light-weight operations into a cached thread executor. Do not use thenXXXXAsync methods without a good reason.
Additionally, reduce should not be used to accumulate into mutable containers. Even though it might work correctly when the stream is sequential, it will fail if the stream were to be made parallel. To perform mutable reduction, use .collect instead.
If you want to complete the entire computation exceptionally immediately after the first failure, do the following in your sequence method:
CompletableFuture<List<T>> result = CompletableFuture.allOf(com.toArray(new CompletableFuture<?>[0]))
.thenApply(v -> com.stream()
.map(CompletableFuture::join)
.collect(Collectors.toList())
);
com.forEach(f -> f.whenComplete((t, ex) -> {
if (ex != null) {
result.completeExceptionally(ex);
}
}));
return result;
If, additionally, you want to cancel the remaining operations on first failure, add exec.shutdownNow(); right after result.completeExceptionally(ex);. This, of course, assumes that exec only exist for this one computation. If it doesn't, you'll have to loop over and cancel each remaining Future individually.
You can get Spotify's CompletableFutures library and use allAsList method. I think it's inspired from Guava's Futures.allAsList method.
public static <T> CompletableFuture<List<T>> allAsList(
List<? extends CompletionStage<? extends T>> stages) {
And here is a simple implementation if you don't want to use a library:
public <T> CompletableFuture<List<T>> allAsList(final List<CompletableFuture<T>> futures) {
return CompletableFuture.allOf(
futures.toArray(new CompletableFuture[futures.size()])
).thenApply(ignored ->
futures.stream().map(CompletableFuture::join).collect(Collectors.toList())
);
}
As Misha has pointed out, you are overusing …Async operations. Further, you are composing a complex chain of operations modelling a dependency which doesn’t reflect your program logic:
you create a job x which depends on the first and second job of your list
you create a job x+1 which depends on job x and the third job of your list
you create a job x+2 which depends on job x+1 and the 4th job of your list
…
you create a job x+5000 which depends on job x+4999 and the last job of your list
Then, canceling (explicitly or due to an exception) this recursively composed job might be performed recursively and might fail with a StackOverflowError. That’s implementation-dependent.
As already shown by Misha, there is a method, allOf which allows you to model your original intention, to define one job which depends on all jobs of your list.
However, it’s worth noting that even that isn’t necessary. Since you are using an unbounded thread pool executor, you can simply post an asynchronous job collecting the results into a list and you are done. Waiting for the completion is implied by asking for the result of each job anyway.
ExecutorService executorService = Executors.newCachedThreadPool();
List<CompletableFuture<Integer>> que = IntStream.range(0, 100000)
.mapToObj(x -> CompletableFuture.supplyAsync(() -> {
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos((long)(Math.random()*10)));
return x;
}, executorService)).collect(Collectors.toList());
CompletableFuture<List<Integer>> sequence = CompletableFuture.supplyAsync(
() -> que.stream().map(CompletableFuture::join).collect(Collectors.toList()),
executorService);
Using methods for composing dependent operations are important, when the number of threads is limited and the jobs may spawn additional asynchronous jobs, to avoid having waiting jobs stealing threads from jobs which have to complete first, but neither is the case here.
In this specific case one job simply iterating over this large number of prerequisite jobs and waiting if necessary may be more efficient than modelling this large number of dependencies and having each job to notify the dependent job about the completion.
To add upto the accepted answer by #Misha, it can be further expanded as a collector:
public static <T> Collector<CompletableFuture<T>, ?, CompletableFuture<List<T>>> sequenceCollector() {
return Collectors.collectingAndThen(Collectors.toList(), com -> sequence(com));
}
Now you can:
Stream<CompletableFuture<Integer>> stream = Stream.of(
CompletableFuture.completedFuture(1),
CompletableFuture.completedFuture(2),
CompletableFuture.completedFuture(3)
);
CompletableFuture<List<Integer>> ans = stream.collect(sequenceCollector());
An example sequence operation using thenCombine on CompletableFuture
public<T> CompletableFuture<List<T>> sequence(List<CompletableFuture<T>> com){
CompletableFuture<List<T>> identity = CompletableFuture.completedFuture(new ArrayList<T>());
BiFunction<CompletableFuture<List<T>>,CompletableFuture<T>,CompletableFuture<List<T>>> combineToList =
(acc,next) -> acc.thenCombine(next,(a,b) -> { a.add(b); return a;});
BinaryOperator<CompletableFuture<List<T>>> combineLists = (a,b)-> a.thenCombine(b,(l1,l2)-> { l1.addAll(l2); return l1;}) ;
return com.stream()
.reduce(identity,
combineToList,
combineLists);
}
}
If you don't mind using 3rd party libraries cyclops-react (I am the author) has a set of utility methods for CompletableFutures (and Optionals, Streams etc)
List<CompletableFuture<String>> listOfFutures;
CompletableFuture<ListX<String>> sequence =CompletableFutures.sequence(listOfFutures);
Disclaimer: This will not completely answer the initial question. It will lack the "fail all if one fails" part. However, I can't answer the actual, more generic question, because it was closed as a duplicate of this one: Java 8 CompletableFuture.allOf(...) with Collection or List. So I will answer here:
How to convert List<CompletableFuture<V>> to
CompletableFuture<List<V>> using Java 8's stream API?
Summary: Use the following:
private <V> CompletableFuture<List<V>> sequence(List<CompletableFuture<V>> listOfFutures) {
CompletableFuture<List<V>> identity = CompletableFuture.completedFuture(new ArrayList<>());
BiFunction<CompletableFuture<List<V>>, CompletableFuture<V>, CompletableFuture<List<V>>> accumulator = (futureList, futureValue) ->
futureValue.thenCombine(futureList, (value, list) -> {
List<V> newList = new ArrayList<>(list.size() + 1);
newList.addAll(list);
newList.add(value);
return newList;
});
BinaryOperator<CompletableFuture<List<V>>> combiner = (futureList1, futureList2) -> futureList1.thenCombine(futureList2, (list1, list2) -> {
List<V> newList = new ArrayList<>(list1.size() + list2.size());
newList.addAll(list1);
newList.addAll(list2);
return newList;
});
return listOfFutures.stream().reduce(identity, accumulator, combiner);
}
Example usage:
List<CompletableFuture<String>> listOfFutures = IntStream.range(0, numThreads)
.mapToObj(i -> loadData(i, executor)).collect(toList());
CompletableFuture<List<String>> futureList = sequence(listOfFutures);
Complete Example:
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.Executor;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ThreadLocalRandom;
import java.util.function.BiFunction;
import java.util.function.BinaryOperator;
import java.util.stream.IntStream;
import static java.util.stream.Collectors.toList;
public class ListOfFuturesToFutureOfList {
public static void main(String[] args) {
ListOfFuturesToFutureOfList test = new ListOfFuturesToFutureOfList();
test.load(10);
}
public void load(int numThreads) {
final ExecutorService executor = Executors.newFixedThreadPool(numThreads);
List<CompletableFuture<String>> listOfFutures = IntStream.range(0, numThreads)
.mapToObj(i -> loadData(i, executor)).collect(toList());
CompletableFuture<List<String>> futureList = sequence(listOfFutures);
System.out.println("Future complete before blocking? " + futureList.isDone());
// this will block until all futures are completed
List<String> data = futureList.join();
System.out.println("Loaded data: " + data);
System.out.println("Future complete after blocking? " + futureList.isDone());
executor.shutdown();
}
public CompletableFuture<String> loadData(int dataPoint, Executor executor) {
return CompletableFuture.supplyAsync(() -> {
ThreadLocalRandom rnd = ThreadLocalRandom.current();
System.out.println("Starting to load test data " + dataPoint);
try {
Thread.sleep(500 + rnd.nextInt(1500));
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Successfully loaded test data " + dataPoint);
return "data " + dataPoint;
}, executor);
}
private <V> CompletableFuture<List<V>> sequence(List<CompletableFuture<V>> listOfFutures) {
CompletableFuture<List<V>> identity = CompletableFuture.completedFuture(new ArrayList<>());
BiFunction<CompletableFuture<List<V>>, CompletableFuture<V>, CompletableFuture<List<V>>> accumulator = (futureList, futureValue) ->
futureValue.thenCombine(futureList, (value, list) -> {
List<V> newList = new ArrayList<>(list.size() + 1);
newList.addAll(list);
newList.add(value);
return newList;
});
BinaryOperator<CompletableFuture<List<V>>> combiner = (futureList1, futureList2) -> futureList1.thenCombine(futureList2, (list1, list2) -> {
List<V> newList = new ArrayList<>(list1.size() + list2.size());
newList.addAll(list1);
newList.addAll(list2);
return newList;
});
return listOfFutures.stream().reduce(identity, accumulator, combiner);
}
}
Your task could be done easily like following,
final List<CompletableFuture<Module> futures =...
CompletableFuture.allOf(futures.stream().toArray(CompletableFuture[]::new)).join();
In addition to Spotify Futures library you might try my code locate here: https://github.com/vsilaev/java-async-await/blob/master/net.tascalate.async.examples/src/main/java/net/tascalate/concurrent/CompletionStages.java (has a dependencies to other classes in same package)
It implements a logic to return "at least N out of M" CompletionStage-s with a policy how much errors it's allowed to tolerate. There are convinient methods for all/any cases, plus cancellation policy for the remaining futures, plus the code deals with CompletionStage-s (interface) rather than CompletableFuture (concrete class).
Javaslang has a very convenient Future API. It also allows to make a future of collection out of a collection of futures.
List<Future<String>> listOfFutures = ...
Future<Seq<String>> futureOfList = Future.sequence(listOfFutures);
See http://static.javadoc.io/io.javaslang/javaslang/2.0.5/javaslang/concurrent/Future.html#sequence-java.lang.Iterable-