Dedicated ForkJoinPool uses common pool threads in JUnit test [duplicate] - java

This question already has an answer here:
Parallel stream from a HashSet doesn't run in parallel
(1 answer)
Closed 6 years ago.
So I know that if you use the parallelStream without a custom ForkJoinPool it will use the default ForkJoinPool which by default has one less threads as you have processors.
So, as stated here (and also in the other answer of that question) in order to have more parallelism, you have to:
submit the parallel stream execution to your own ForkJoinPool: yourFJP.submit(() -> stream.parallel().forEach(doSomething));
So, I did this:
import java.util.Set;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ForkJoinPool;
import java.util.stream.IntStream;
import com.google.common.collect.Sets;
public class Main {
public static void main(String[] args) throws InterruptedException, ExecutionException {
ForkJoinPool forkJoinPool = new ForkJoinPool(1000);
IntStream stream = IntStream.range(0, 999999);
final Set<String> thNames = Collections.synchronizedSet(new HashSet<String>());
forkJoinPool.submit(() -> {
stream.parallel().forEach(n -> {
System.out.println("Processing n: " + n);
try {
Thread.sleep(500);
thNames.add(Thread.currentThread().getName());
System.out.println("Size: " + thNames.size() + " activeCount: " + forkJoinPool.getActiveThreadCount());
} catch (Exception e) {
throw new RuntimeException(e);
}
});
}).get();
}
}
I made a Set of thread Names in order to see how many threads are being created, and also logged the number of active threads that the pool has and both numbers don't grow up more that 16, so that means that the parallelism here is not being more than 16 (why even 16?). If I do not use the forkJoinPool, I get 4 as parallelism, which is according to the number of processors I have.
Why does it give me 16 and not 1000?

Update
Originally this answer was an elaborate explanation claiming that the ForkJoinPool applies back-pressure and doesn't even reach the prescribed parallelism level, because there are always idle workers available to process the stream.
That's incorrect.
The actual answer is provided in the original question to which this was marked as duplicate - using a custom ForkJoinPool for stream processing is not officially supported, and when using forEach, the default pool parallelism is used to determine the stream spliterator behavior.
Here's an example how when tasks are manually submitted to a custom ForkJoinPool, the pool's active thread count easily reaches its parallelism level:
for (int i = 0; i < 1_000_000; ++i) {
forkJoinPool.submit(() -> {
try {
Thread.sleep(1);
thNames.add(Thread.currentThread().getName());
System.out.println("Size: " + thNames.size() + " activeCount: " + forkJoinPool.getActiveThreadCount() + " parallelism: " + forkJoinPool.getParallelism());
} catch (Exception e) {
throw new RuntimeException(e);
}
});
}
Thanks to Stuart Marks for pointing this out and to Sotirios Delimanolis for arguing that my original answer is wrong :)

It seems to me that when you submit a lambda to the FJP that lambda will use the common pool and not the FJP. Sotirios Delimanolis proved this with his comment, above. What you are submitting is a Task that runs in your FJP.
Try profiling this code to see what threads are actually being used.
You cannot name the threads within the FJP.

Related

Why does stream parallel() not use all available threads?

I tried to run 100 Sleep tasks in parallel using Java8(1.8.0_172) stream.parallel() submitted inside a custom ForkJoinPool with 100+ threads available. Each task would sleep for 1s. I expected the whole work would finish after ~1s, given the 100 sleeps could be done in parallel. However I observe a runtime of 7s.
#Test
public void testParallelStream() throws Exception {
final int REQUESTS = 100;
ForkJoinPool forkJoinPool = null;
try {
// new ForkJoinPool(256): same results for all tried values of REQUESTS
forkJoinPool = new ForkJoinPool(REQUESTS);
forkJoinPool.submit(() -> {
IntStream stream = IntStream.range(0, REQUESTS);
final List<String> result = stream.parallel().mapToObj(i -> {
try {
System.out.println("request " + i);
Thread.sleep(1000);
return Integer.toString(i);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}).collect(Collectors.toList());
// assertThat(result).hasSize(REQUESTS);
}).join();
} finally {
if (forkJoinPool != null) {
forkJoinPool.shutdown();
}
}
}
With output indicating ~16 stream elements are executed before a pause of 1s, then another ~16 and so on. So it seems even though the forkjoinpool was created with 100 threads, only ~16 get used.
This pattern emerges as soon as I use more than 23 threads:
1-23 threads: ~1s
24-35 threads: ~2s
36-48 threads: ~3s
...
System.out.println(Runtime.getRuntime().availableProcessors());
// Output: 4
Since the Stream implementation’s use of the Fork/Join pool is an implementation detail, the trick to force it to use a different Fork/Join pool is undocumented as well and seems to work by accident, i.e. there’s a hardcoded constant determining the actual parallelism, depending on the default pool’s parallelism. So using a different pool was not foreseen, originally.
However, it has been recognized that using a different pool with an inappropriate target parallelism is a bug, even if this trick is not documented, see JDK-8190974.
It has been fixed in Java 10 and backported to Java 8, update 222.
So a simple solution world be updating the Java version.
You may also change the default pool’s parallelism, e.g.
System.setProperty("java.util.concurrent.ForkJoinPool.common.parallelism", "100");
before doing any Fork/Join activity.
But this may have unintended effects on other parallel operations.
As you wrote it, you let the stream decide the parallelism of the executions.
There you have the effect that ArrayList.parallelStream tries to outsmart you by splitting the data up evenly, without taking the number of available threads into account. This is good for CPU-Bound operations, where it's not usefull to have more threads than CPU Cores, but is not made for processes that need to wait for IO.
Why not force-feed all your items sequentially to the ForkJoinPool, so it's forced to use all available threads?
IntStream stream = IntStream.range(0, REQUESTS);
List<ForkJoinTask<String>> results
= stream.mapToObj(i -> forkJoinPool.submit(() -> {
try {
System.out.println("request " + i);
Thread.sleep(1000);
return Integer.toString(i);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
})).collect(Collectors.toList());
results.forEach(ForkJoinTask::join);
This takes less than two seconds on my machine.

How to parallelize loops with Java 8's Fork/Join framework

How to parallelize loops with Java 8's Fork/Join framework. Accually I did not work with multiple threading . I read lots of question in SO .Now i am unable to implement the parallel processing of list in Java 8. Any one can help me ?
I have tried somthing like from this link.
routes.stream().parallel().forEach(this::doSomething);
Scenario like list based on routes list I need to devide the task and execute I need like insted of foreach loop I want parallel execution of based on array size.
My problem is when processing the updateSchedules service it is taking too much time. That is the reason I want to implement the threading concept here. scheduleService.updateSchedules(originId, destinationId,req.getJourneyDate());
for (Availabilities ar : routes) {
try {
log.info("Starting for bus" + ar);
Bus bus = new Bus();
// Get schedule list
BitlaSchedules schedule = scheduleRepo
.findByOriginIdAndDestinationIdAndScheduleIdAndTravelIdAndRouteId(originId,
destinationId, ar.getScheduleId(), ar.getTravelId(), ar.getRouteId());
if (schedule == null) {
scheduleService.updateSchedules(originId, destinationId,req.getJourneyDate());
schedule = scheduleRepo
.findByOriginIdAndDestinationIdAndScheduleIdAndTravelIdAndRouteId(originId,
destinationId, ar.getScheduleId(), ar.getTravelId(), ar.getRouteId());
}
} catch(Exception e) {
log.error(e.getMessage());
}
}
Probably the basic error is that you are trying to do it.
Don't! The fork/join framework is a very specific piece of engineering - which solves a very specific area:
- solving CPU intensive problems;
- that can be split without sharing resources (i.e. no synchronization or locking between ).
Your code seems to use an external service:
- if the service uses database of any kind, then your problem is not CPU intensive;
- even if not, then - since there is an obvious update, then there is a shared, mutable state that requires synchronization (especially since we seem to have multiple writers).
This means that you gain nothing by using the parallel stream.
Just use a standard executor with a thread pool and submit your items as tasks.
As #fdreger already said, it will only help you with CPU intensive tasks.
So before making any assumptions WHY something should be run parallel to gain performance, do yourself a favor and profile. Most of the time the bottleneck is IO related.
I will give you a very simple example how you could use parallel streams in java.
public class Test {
public static void main(String[] args) {
// some dummy data
List<Integer> list = new ArrayList<>();
for (int i = 0; i < 20; ++i) list.add(i);
// to simulate some CPU intensive work
Random random = new SecureRandom();
List<String> result = list.parallelStream().map(i -> {
// simulate work load
int millis = 0;
try {
millis = random.nextInt(1000);
Thread.sleep(millis);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
// return any desired result
return "Done something with " + i + " in thread " + Thread.currentThread().getName() + " took " + millis + "ms";
}).collect(Collectors.toList()); // collect joins - will return once all the workers are done
// print the result
result.forEach(System.out::println);
}
}

Why does the parallel stream not use all the threads of the ForkJoinPool? [duplicate]

This question already has an answer here:
Parallel stream from a HashSet doesn't run in parallel
(1 answer)
Closed 6 years ago.
So I know that if you use the parallelStream without a custom ForkJoinPool it will use the default ForkJoinPool which by default has one less threads as you have processors.
So, as stated here (and also in the other answer of that question) in order to have more parallelism, you have to:
submit the parallel stream execution to your own ForkJoinPool: yourFJP.submit(() -> stream.parallel().forEach(doSomething));
So, I did this:
import java.util.Set;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ForkJoinPool;
import java.util.stream.IntStream;
import com.google.common.collect.Sets;
public class Main {
public static void main(String[] args) throws InterruptedException, ExecutionException {
ForkJoinPool forkJoinPool = new ForkJoinPool(1000);
IntStream stream = IntStream.range(0, 999999);
final Set<String> thNames = Collections.synchronizedSet(new HashSet<String>());
forkJoinPool.submit(() -> {
stream.parallel().forEach(n -> {
System.out.println("Processing n: " + n);
try {
Thread.sleep(500);
thNames.add(Thread.currentThread().getName());
System.out.println("Size: " + thNames.size() + " activeCount: " + forkJoinPool.getActiveThreadCount());
} catch (Exception e) {
throw new RuntimeException(e);
}
});
}).get();
}
}
I made a Set of thread Names in order to see how many threads are being created, and also logged the number of active threads that the pool has and both numbers don't grow up more that 16, so that means that the parallelism here is not being more than 16 (why even 16?). If I do not use the forkJoinPool, I get 4 as parallelism, which is according to the number of processors I have.
Why does it give me 16 and not 1000?
Update
Originally this answer was an elaborate explanation claiming that the ForkJoinPool applies back-pressure and doesn't even reach the prescribed parallelism level, because there are always idle workers available to process the stream.
That's incorrect.
The actual answer is provided in the original question to which this was marked as duplicate - using a custom ForkJoinPool for stream processing is not officially supported, and when using forEach, the default pool parallelism is used to determine the stream spliterator behavior.
Here's an example how when tasks are manually submitted to a custom ForkJoinPool, the pool's active thread count easily reaches its parallelism level:
for (int i = 0; i < 1_000_000; ++i) {
forkJoinPool.submit(() -> {
try {
Thread.sleep(1);
thNames.add(Thread.currentThread().getName());
System.out.println("Size: " + thNames.size() + " activeCount: " + forkJoinPool.getActiveThreadCount() + " parallelism: " + forkJoinPool.getParallelism());
} catch (Exception e) {
throw new RuntimeException(e);
}
});
}
Thanks to Stuart Marks for pointing this out and to Sotirios Delimanolis for arguing that my original answer is wrong :)
It seems to me that when you submit a lambda to the FJP that lambda will use the common pool and not the FJP. Sotirios Delimanolis proved this with his comment, above. What you are submitting is a Task that runs in your FJP.
Try profiling this code to see what threads are actually being used.
You cannot name the threads within the FJP.

Java 8 Concurrency Simplest Canonical Form for Basic Task

I have two questions:
1. What is the simplest canonical form for running a Callable as a task in Java 8, capturing and processing the result?
2. In the example below, what is the best/simplest/clearest way to hold the main process open until all the tasks have completed?
Here's the example I have so far -- is this the best approach in Java 8 or is there something more basic?
import java.util.*;
import java.util.concurrent.*;
import java.util.function.*;
public class SimpleTask implements Supplier<String> {
private SplittableRandom rand = new SplittableRandom();
final int id;
SimpleTask(int id) { this.id = id; }
#Override
public String get() {
try {
TimeUnit.MILLISECONDS.sleep(rand.nextInt(50, 300));
} catch(InterruptedException e) {
System.err.println("Interrupted");
}
return "Completed " + id + " on " +
Thread.currentThread().getName();
}
public static void main(String[] args) throws Exception {
for(int i = 0; i < 10; i++)
CompletableFuture.supplyAsync(new SimpleTask(i))
.thenAccept(System.out::println);
System.in.read(); // Or else program ends too soon
}
}
Is there a simpler and clearer Java-8 way to do this? And how do I eliminate the System.in.read() in favor of a better approach?
The canonical way to wait for the completion of multiple CompletableFuture instance is to create a new one depending on all of them via CompletableFuture.allOf. You can use this new future to wait for its completion or schedule new follow-up actions just like with any other CompletableFuture:
CompletableFuture.allOf(
IntStream.range(0,10).mapToObj(SimpleTask::new)
.map(s -> CompletableFuture.supplyAsync(s).thenAccept(System.out::println))
.toArray(CompletableFuture<?>[]::new)
).join();
Of course, it always gets simpler if you forego assigning a unique id to each task. Since your first question was about Callable, I’ll demonstrate how you can easily submit multiple similar tasks as Callables via an ExecutorService:
ExecutorService pool = Executors.newCachedThreadPool();
pool.invokeAll(Collections.nCopies(10, () -> {
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(
ThreadLocalRandom.current().nextInt(50, 300)));
final String s = "Completed on "+Thread.currentThread().getName();
System.out.println(s);
return s;
}));
pool.shutdown();
The executor service returned by Executors.newCachedThreadPool() is unshared and won’t stay alive, even if you forget to invoke shutDown(), but it can take up to one minute before all threads are terminated then.
Since your first question literally was: “What is the simplest canonical form for running a Callable as a task in Java 8, capturing and processing the result?”, the answer might be that the simplest form still is invoking it’s call() method directly, e.g.
Callable<String> c = () -> {
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(
ThreadLocalRandom.current().nextInt(50, 300)));
return "Completed on "+Thread.currentThread().getName();
};
String result = c.call();
System.out.println(result);
There’s no simpler way…
Consider collecting the futures into a list. Then you can use join() on each future to await their completion in the current thread:
List<CompletableFuture<Void>> futures = IntStream.range(0,10)
.mapToObj(id -> supplyAsync(new SimpleTask(id)).thenAccept(System.out::println))
.collect(toList());
futures.forEach(CompletableFuture::join);

Custom thread pool in Java 8 parallel stream

Is it possible to specify a custom thread pool for Java 8 parallel stream? I can not find it anywhere.
Imagine that I have a server application and I would like to use parallel streams. But the application is large and multi-threaded so I want to compartmentalize it. I do not want a slow running task in one module of the applicationblock tasks from another module.
If I can not use different thread pools for different modules, it means I can not safely use parallel streams in most of real world situations.
Try the following example. There are some CPU intensive tasks executed in separate threads.
The tasks leverage parallel streams. The first task is broken, so each step takes 1 second (simulated by thread sleep). The issue is that other threads get stuck and wait for the broken task to finish. This is contrived example, but imagine a servlet app and someone submitting a long running task to the shared fork join pool.
public class ParallelTest {
public static void main(String[] args) throws InterruptedException {
ExecutorService es = Executors.newCachedThreadPool();
es.execute(() -> runTask(1000)); //incorrect task
es.execute(() -> runTask(0));
es.execute(() -> runTask(0));
es.execute(() -> runTask(0));
es.execute(() -> runTask(0));
es.execute(() -> runTask(0));
es.shutdown();
es.awaitTermination(60, TimeUnit.SECONDS);
}
private static void runTask(int delay) {
range(1, 1_000_000).parallel().filter(ParallelTest::isPrime).peek(i -> Utils.sleep(delay)).max()
.ifPresent(max -> System.out.println(Thread.currentThread() + " " + max));
}
public static boolean isPrime(long n) {
return n > 1 && rangeClosed(2, (long) sqrt(n)).noneMatch(divisor -> n % divisor == 0);
}
}
There actually is a trick how to execute a parallel operation in a specific fork-join pool. If you execute it as a task in a fork-join pool, it stays there and does not use the common one.
final int parallelism = 4;
ForkJoinPool forkJoinPool = null;
try {
forkJoinPool = new ForkJoinPool(parallelism);
final List<Integer> primes = forkJoinPool.submit(() ->
// Parallel task here, for example
IntStream.range(1, 1_000_000).parallel()
.filter(PrimesPrint::isPrime)
.boxed().collect(Collectors.toList())
).get();
System.out.println(primes);
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
} finally {
if (forkJoinPool != null) {
forkJoinPool.shutdown();
}
}
The trick is based on ForkJoinTask.fork which specifies: "Arranges to asynchronously execute this task in the pool the current task is running in, if applicable, or using the ForkJoinPool.commonPool() if not inForkJoinPool()"
The parallel streams use the default ForkJoinPool.commonPool which by default has one less threads as you have processors, as returned by Runtime.getRuntime().availableProcessors() (This means that parallel streams leave one processor for the calling thread).
For applications that require separate or custom pools, a ForkJoinPool may be constructed with a given target parallelism level; by default, equal to the number of available processors.
This also means if you have nested parallel streams or multiple parallel streams started concurrently, they will all share the same pool. Advantage: you will never use more than the default (number of available processors). Disadvantage: you may not get "all the processors" assigned to each parallel stream you initiate (if you happen to have more than one). (Apparently you can use a ManagedBlocker to circumvent that.)
To change the way parallel streams are executed, you can either
submit the parallel stream execution to your own ForkJoinPool: yourFJP.submit(() -> stream.parallel().forEach(soSomething)).get(); or
you can change the size of the common pool using system properties: System.setProperty("java.util.concurrent.ForkJoinPool.common.parallelism", "20") for a target parallelism of 20 threads.
Example of the latter on my machine which has 8 processors. If I run the following program:
long start = System.currentTimeMillis();
IntStream s = IntStream.range(0, 20);
//System.setProperty("java.util.concurrent.ForkJoinPool.common.parallelism", "20");
s.parallel().forEach(i -> {
try { Thread.sleep(100); } catch (Exception ignore) {}
System.out.print((System.currentTimeMillis() - start) + " ");
});
The output is:
215 216 216 216 216 216 216 216 315 316 316 316 316 316 316 316 415 416 416 416
So you can see that the parallel stream processes 8 items at a time, i.e. it uses 8 threads. However, if I uncomment the commented line, the output is:
215 215 215 215 215 216 216 216 216 216 216 216 216 216 216 216 216 216 216 216
This time, the parallel stream has used 20 threads and all 20 elements in the stream have been processed concurrently.
Alternatively to the trick of triggering the parallel computation inside your own forkJoinPool you can also pass that pool to the CompletableFuture.supplyAsync method like in:
ForkJoinPool forkJoinPool = new ForkJoinPool(2);
CompletableFuture<List<Integer>> primes = CompletableFuture.supplyAsync(() ->
//parallel task here, for example
range(1, 1_000_000).parallel().filter(PrimesPrint::isPrime).collect(toList()),
forkJoinPool
);
The original solution (setting the ForkJoinPool common parallelism property) no longer works. Looking at the links in the original answer, an update which breaks this has been back ported to Java 8. As mentioned in the linked threads, this solution was not guaranteed to work forever. Based on that, the solution is the forkjoinpool.submit with .get solution discussed in the accepted answer. I think the backport fixes the unreliability of this solution also.
ForkJoinPool fjpool = new ForkJoinPool(10);
System.out.println("stream.parallel");
IntStream range = IntStream.range(0, 20);
fjpool.submit(() -> range.parallel()
.forEach((int theInt) ->
{
try { Thread.sleep(100); } catch (Exception ignore) {}
System.out.println(Thread.currentThread().getName() + " -- " + theInt);
})).get();
System.out.println("list.parallelStream");
int [] array = IntStream.range(0, 20).toArray();
List<Integer> list = new ArrayList<>();
for (int theInt: array)
{
list.add(theInt);
}
fjpool.submit(() -> list.parallelStream()
.forEach((theInt) ->
{
try { Thread.sleep(100); } catch (Exception ignore) {}
System.out.println(Thread.currentThread().getName() + " -- " + theInt);
})).get();
We can change the default parallelism using the following property:
-Djava.util.concurrent.ForkJoinPool.common.parallelism=16
which can set up to use more parallelism.
To measure the actual number of used threads, you can check Thread.activeCount():
Runnable r = () -> IntStream
.range(-42, +42)
.parallel()
.map(i -> Thread.activeCount())
.max()
.ifPresent(System.out::println);
ForkJoinPool.commonPool().submit(r).join();
new ForkJoinPool(42).submit(r).join();
This can produce on a 4-core CPU an output like:
5 // common pool
23 // custom pool
Without .parallel() it gives:
3 // common pool
4 // custom pool
Until now, I used the solutions described in the answers of this question. Now, I came up with a little library called Parallel Stream Support for that:
ForkJoinPool pool = new ForkJoinPool(NR_OF_THREADS);
ParallelIntStreamSupport.range(1, 1_000_000, pool)
.filter(PrimesPrint::isPrime)
.collect(toList())
But as #PabloMatiasGomez pointed out in the comments, there are drawbacks regarding the splitting mechanism of parallel streams which depends heavily on the size of the common pool. See Parallel stream from a HashSet doesn't run in parallel .
I am using this solution only to have separate pools for different types of work but I can not set the size of the common pool to 1 even if I don't use it.
Note:
There appears to be a fix implemented in JDK 10 that ensures the Custom Thread Pool uses the expected number of threads.
Parallel stream execution within a custom ForkJoinPool should obey the parallelism
https://bugs.openjdk.java.net/browse/JDK-8190974
If you don't want to rely on implementation hacks, there's always a way to achieve the same by implementing custom collectors that will combine map and collect semantics... and you wouldn't be limited to ForkJoinPool:
list.stream()
.collect(parallel(i -> process(i), executor, 4))
.join()
Luckily, it's done already here and available on Maven Central:
http://github.com/pivovarit/parallel-collectors
Disclaimer: I wrote it and take responsibility for it.
Go to get abacus-common. Thread number can by specified for parallel stream. Here is the sample code:
LongStream.range(4, 1_000_000).parallel(threadNum)...
Disclosure: I'm the developer of abacus-common.
If you don't need a custom ThreadPool but you rather want to limit the number of concurrent tasks, you can use:
List<Path> paths = List.of("/path/file1.csv", "/path/file2.csv", "/path/file3.csv").stream().map(e -> Paths.get(e)).collect(toList());
List<List<Path>> partitions = Lists.partition(paths, 4); // Guava method
partitions.forEach(group -> group.parallelStream().forEach(csvFilePath -> {
// do your processing
}));
(Duplicate question asking for this is locked, so please bear me here)
Here is how I set the max thread count flag mentioned above programatically and a code sniped to verify that the parameter is honored
System.setProperty("java.util.concurrent.ForkJoinPool.common.parallelism", "2");
Set<String> threadNames = Stream.iterate(0, n -> n + 1)
.parallel()
.limit(100000)
.map(i -> Thread.currentThread().getName())
.collect(Collectors.toSet());
System.out.println(threadNames);
// Output -> [ForkJoinPool.commonPool-worker-1, Test worker, ForkJoinPool.commonPool-worker-3]
If you don't mind using a third-party library, with cyclops-react you can mix sequential and parallel Streams within the same pipeline and provide custom ForkJoinPools. For example
ReactiveSeq.range(1, 1_000_000)
.foldParallel(new ForkJoinPool(10),
s->s.filter(i->true)
.peek(i->System.out.println("Thread " + Thread.currentThread().getId()))
.max(Comparator.naturalOrder()));
Or if we wished to continue processing within a sequential Stream
ReactiveSeq.range(1, 1_000_000)
.parallel(new ForkJoinPool(10),
s->s.filter(i->true)
.peek(i->System.out.println("Thread " + Thread.currentThread().getId())))
.map(this::processSequentially)
.forEach(System.out::println);
[Disclosure I am the lead developer of cyclops-react]
I tried the custom ForkJoinPool as follows to adjust the pool size:
private static Set<String> ThreadNameSet = new HashSet<>();
private static Callable<Long> getSum() {
List<Long> aList = LongStream.rangeClosed(0, 10_000_000).boxed().collect(Collectors.toList());
return () -> aList.parallelStream()
.peek((i) -> {
String threadName = Thread.currentThread().getName();
ThreadNameSet.add(threadName);
})
.reduce(0L, Long::sum);
}
private static void testForkJoinPool() {
final int parallelism = 10;
ForkJoinPool forkJoinPool = null;
Long result = 0L;
try {
forkJoinPool = new ForkJoinPool(parallelism);
result = forkJoinPool.submit(getSum()).get(); //this makes it an overall blocking call
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
} finally {
if (forkJoinPool != null) {
forkJoinPool.shutdown(); //always remember to shutdown the pool
}
}
out.println(result);
out.println(ThreadNameSet);
}
Here is the output saying the pool is using more threads than the default 4.
50000005000000
[ForkJoinPool-1-worker-8, ForkJoinPool-1-worker-9, ForkJoinPool-1-worker-6, ForkJoinPool-1-worker-11, ForkJoinPool-1-worker-10, ForkJoinPool-1-worker-1, ForkJoinPool-1-worker-15, ForkJoinPool-1-worker-13, ForkJoinPool-1-worker-4, ForkJoinPool-1-worker-2]
But actually there is a weirdo, when I tried to achieve the same result using ThreadPoolExecutor as follows:
BlockingDeque blockingDeque = new LinkedBlockingDeque(1000);
ThreadPoolExecutor fixedSizePool = new ThreadPoolExecutor(10, 20, 60, TimeUnit.SECONDS, blockingDeque, new MyThreadFactory("my-thread"));
but I failed.
It will only start the parallelStream in a new thread and then everything else is just the same, which again proves that the parallelStream will use the ForkJoinPool to start its child threads.
I made utility method to run task in parallel with argument which defines max number of threads.
public static void runParallel(final int maxThreads, Runnable task) throws RuntimeException {
ForkJoinPool forkJoinPool = null;
try {
forkJoinPool = new ForkJoinPool(maxThreads);
forkJoinPool.submit(task).get();
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
} finally {
if (forkJoinPool != null) {
forkJoinPool.shutdown();
}
}
}
It creates ForkJoinPool with max number of allowed threads and it shuts it down after the task completes (or fails).
Usage is following:
final int maxThreads = 4;
runParallel(maxThreads, () ->
IntStream.range(1, 1_000_000).parallel()
.filter(PrimesPrint::isPrime)
.boxed().collect(Collectors.toList()));
The (currently) accepted answer is partly wrong. It is not sufficient to just submit() the parallel stream to the dedicated fork-join-pool. In this case, the stream will use that pool's threads and additionally the common fork-join-pool and even the calling thread to handle the workload of the stream, it seems up to the size of the common fork-join pool. The behaviour is a bit weird but definitely not what is required.
To actually restrict the work completely to the dedicated pool, you must encapsulate it into a CompletableFuture:
final int parallelism = 4;
ForkJoinPool forkJoinPool = null;
try {
forkJoinPool = new ForkJoinPool(parallelism);
final List<Integer> primes = CompletableFuture.supplyAsync(() ->
// Parallel task here, for example
IntStream.range(1, 1_000_000).parallel()
.filter(PrimesPrint::isPrime)
.boxed().collect(Collectors.toList()),
forkJoinPool) // <- passes dedicated fork-join pool as executor
.join(); // <- Wait for result from forkJoinPool
System.out.println(primes);
} finally {
if (forkJoinPool != null) {
forkJoinPool.shutdown();
}
}
This code stays with all operations in forkJoinPool on both Java 8u352 and Java 17.0.1.

Categories