Divide calculations among multiple threads - java

I've just started working with threads in java. I have a simple algorithm that does a lot of calculations. What I need to do is to divide those calculations among different threads. It looks like this:
while(...) {
....
doCalculations(rangeStart, rangeEnd);
}
And what I want to do is something like this:
while(...) {
...
// Notify N threads to start calculations in specific range
// Wait for them to finish calculating
// Check results
... Repeat
}
Calculating threads don't have to have a critical section or be synchronized between each other, because they don't change any shared variables.
What I can't figure out is how to order threads to start and wait them to finish.
thread[n].start() and thread[n].join() throws an exception.
Thank you!

I use an ExecutorService
private static final int procs = Runtime.getRuntime().availableProcessors();
private final ExecutorService es = new Executors.newFixedThreadPool(procs);
int tasks = ....
int blockSize = (tasks + procss -1) / procs;
List<Future<Results>> futures = new ArrayList<>();
for(int i = 0; i < procs; i++) {
int start = i * blockSize;
int end = Math.min(tasks, (i + 1) * blockSize);
futures.add(es.submit(new Task(start, end));
}
for(Future<Result> future: futures) {
Result result = future.get();
// check/accumulate result.
}

Use a CountDownLatch to start, and another CountDownLatch to finish:
CountDownLatch start = new CountDownLatch(1);
CountDownLatch finish = new CountDownLatch(NUMBER_OF_THREADS);
start.countDown();
finish.await();
And in each worker thread:
start.await();
// do the computation
finish.countDown();
And if you need to do that several times, then a CyclicBarrier is probably what you should use.

Learn MapReduce and Hadoop. I think the could be a better approach than rolling your own, at the cost of greater dependencies.

Related

Phaser using ThreadPool does not correctly arrive and wait. How can I fix this?

A repo with the basic idea of how I use the Phaser and the problem that I run into can be found here: https://github.com/hipy/phaser/tree/master/src
I've been working on making a Dijkstra algorithm more efficient with ThreadPools using a Phaser. I make a lot of iterations with a loop and for each iteration a Phaser is needed to wait for the threads in the ThreadPool to finish before continuing the current iteration.
I've run into a problem, the Phaser does not wait correctly. When I use ArriveAndDeregister() the Phaser enters a termination state after each thread is done. When I call Arrive() the amount of unarrived parties does not decrease and thus the iteration gets stuck.
All code below runs in an apply() method called once.
This code below creates the tasks for the threads to execute.
ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool((numberOfThreads));
Phaser phaser = new Phaser();
//Thread class
class ClosestNodeTask implements Runnable {
private int start;
private int end;
private Phaser phaser;
public ClosestNodeTask(int start, int end, Phaser phaser) {
this.start = start;
this.end = end;
this.phaser = phaser;
}
#Override
public void run() {
getNodeShortestDistanced(start, end, phaser); //method calls phaser.arrive() when done
}
}
for (int t = 0; t < numberOfThreads; t++) {
if (nodesModulo > 0 && numberOfThreads == (t + 1)) {
start = nodesPerThread * (t);
end = nodesPerThread * (t + 1) + nodesModulo;
tasks[t] = new ClosestNodeTask(start, end, phaser);
} else {
start = nodesPerThread * t;
end = nodesPerThread * (t + 1);
tasks[t] = new ClosestNodeTask(start, end, phaser);
}
}
The code below is executed for each iteration in a for loop. In this case there are 30.000 iterations:
phaser.register(); //register main thread
for(int t = 0; t < tasks.length; t++) {
phaser.register();
}
System.out.println("Phaser unarrived party size is now: " + phaser.getUnarrivedParties());
Skipping some code for the algorithm the following code is executed in the for loop, starting the threads and waiting for it to finish:
for(int t = 0; t < tasks.length; t++) {
executor.execute(tasks[t]);
}
phaser.arriveAndAwaitAdvance();
The output is as following:
Phasecount: 0
Phaser unarrived party size is now: 3
Task size: 2
Adding: 613 //Next closest node in a sub-group, result of work done in a thread
Adding: 2870
all tasks done
-----------------done-------------
Phasecount: 1
Phaser unarrived party size is now: 6
Task size: 2
Adding: 1
Adding: 2870
The first phase executes, doing the full iteration. The second phase gets stuck. The amount of unarrived parties is 6. 3 new parties and apparently 3 old ones that did not register as arrived even though I called phaser.arrive(). Also the ArriveAndAwaitAdvance() did not wait because in the next iteration there are 6 unarrived parties instead of 3.
I tried using arriveAndDeregister() but this results in a terminated phase(phasecount has a large negative value).
How could I solve this? I don't want to terminate a phase but I do want to register the parties as arrived each iteration.
Thanks!

Java factorial calculation with thread pool

I achieved to calculate factorial with two threads without the pool. I have two factorial classes which are named Factorial1, Factorial2 and extends Thread class. Let's consider I want to calculate the value of !160000. In Factorial1's run() method I do the multiplication in a for loop from i=2 to i=80000 and in Factorial2's from i=80001 to 160000. After that, i return both values and multiply them in the main method. When I compare the execution time it's much better (which is 5000 milliseconds) than the non-thread calculation's time (15000 milliseconds) even with two threads.
Now I want to write clean and better code because I saw the efficiency of threads at factorial calculation but when I use a thread pool to calculate the factorial value, the parallel calculation always takes more time than the non-thread calculation (nearly 16000). My code pieces look like:
for(int i=2; i<= Calculate; i++)
{
myPool.execute(new Multiplication(result, i));
}
run() method which is in Multiplication class:
public void run()
{
s1.Mltply(s2); // s1 and s2 are instances of my Number class
// their fields holds BigInteger values
}
Mltply() method which is in Number class:
public void Multiply(int number)
{
area.lock(); // result is going wrong without lock
Number temp = new Number(number);
value = value.multiply(temp.value); // value is a BigInteger
area.unlock();
}
In my opinion this lock may kills the all advantage of the thread usage because it seems like all that threads do is multiplication but nothing else. But without it, i can't even calculate the true result. Let's say i want to calculate !10, so thread1 calculates the 10*9*8*7*6 and thread2 calculate the 5*4*3*2*1. Is that the way I'm looking for? Is it even possible with thread pool? Of course execution time must be less than the normal calculation...
I appreciate all your help and suggestion.
EDIT: - My own solution to the problem -
public class MyMultiplication implements Runnable
{
public static BigInteger subResult1;
public static BigInteger subResult2;
int thread1StopsAt;
int thread2StopsAt;
long threadId;
static boolean idIsSet=false;
public MyMultiplication(BigInteger n1, int n2) // First Thread
{
MyMultiplication.subResult1 = n1;
this.thread1StopsAt = n2/2;
thread2StopsAt = n2;
}
public MyMultiplication(int n2,BigInteger n1) // Second Thread
{
MyMultiplication.subResult2 = n1;
this.thread2StopsAt = n2;
thread1StopsAt = n2/2;
}
#Override
public void run()
{
if(idIsSet==false)
{
threadId = Thread.currentThread().getId();
idIsSet=true;
}
if(Thread.currentThread().getId() == threadId)
{
for(int i=2; i<=thread1StopsAt; i++)
{
subResult1 = subResult1.multiply(BigInteger.valueOf(i));
}
}
else
{
for(int i=thread1StopsAt+1; i<= thread2StopsAt; i++)
{
subResult2 = subResult2.multiply(BigInteger.valueOf(i));
}
}
}
}
public class JavaApplication3
{
public static void main(String[] args) throws InterruptedException
{
int calculate=160000;
long start = System.nanoTime();
BigInteger num = BigInteger.valueOf(1);
for (int i = 2; i <= calculate; i++)
{
num = num.multiply(BigInteger.valueOf(i));
}
long end = System.nanoTime();
double time = (end-start)/1000000.0;
System.out.println("Without threads: \t" +
String.format("%.2f",time) + " miliseconds");
System.out.println("without threads Result: " + num);
BigInteger num1 = BigInteger.valueOf(1);
BigInteger num2 = BigInteger.valueOf(1);
ExecutorService myPool = Executors.newFixedThreadPool(2);
start = System.nanoTime();
myPool.execute(new MyMultiplication(num1,calculate));
Thread.sleep(100);
myPool.execute(new MyMultiplication(calculate,num2));
myPool.shutdown();
while(!myPool.isTerminated()) {} // waiting threads to end
end = System.nanoTime();
time = (end-start)/1000000.0;
System.out.println("With threads: \t" +String.format("%.2f",time)
+ " miliseconds");
BigInteger result =
MyMultiplication.subResult1.
multiply(MyMultiplication.subResult2);
System.out.println("With threads Result: " + result);
System.out.println(MyMultiplication.subResult1);
System.out.println(MyMultiplication.subResult2);
}
}
input : !160000
Execution time without threads : 15000 milliseconds
Execution time with 2 threads : 4500 milliseconds
Thanks for ideas and suggestions.
You may calculate !160000 concurrently without using a lock by splitting 160000 into disjunct junks as you explaint by splitting it into 2..80000 and 80001..160000.
But you may achieve this by using the Java Stream API:
IntStream.rangeClosed(1, 160000).parallel()
.mapToObj(val -> BigInteger.valueOf(val))
.reduce(BigInteger.ONE, BigInteger::multiply);
It does exactly what you try to do. It splits the whole range into junks, establishes a thread pool and computes the partial results. Afterwards it joins the partial results into a single result.
So why do you bother doing it by yourself? Just practicing clean coding?
On my real 4 core machine computation in a for loop took 8 times longer than using a parallel stream.
Threads have to run independent to run fast. Many dependencies like locks, synchronized parts of your code or some system calls leads to sleeping threads which are waiting to access some resources.
In your case you should minimize the time a thread is inside the lock. Maybe I am wrong, but it seems like you create a thread for each number. So for 1.000! you spawn 1.000 Threads. All of them trying to get the lock on area and are not able to calculate anything, because one thread has become the lock and all other threads have to wait until the lock is unlocked again. So the threads are only running in serial which is as fast as your non-threaded example plus the extra time for locking and unlocking, thread management and so on. Oh, and because of cpu's context switching it gets even worse.
Your first attempt to splitt the factorial in two threads is the better one. Each thread can calculate its own result and only when they are done the threads have to communicate with each other. So they are independent most of the time.
Now you have to generalize this solution. To reduce context switching of the cpu you only want as many threads as your cpu has cores (maybe a little bit less because of your OS). Every thread gets a rang of numbers and calculates their product. After this it locks the overall result and adds its own result to it.
This should improve the performance of your problem.
Update: You ask for additional advice:
You said you have two classes Factorial1 and Factorial2. Probably they have their ranges hard codes. You only need one class which takes the range as constructor arguments. This class implements Runnable so it has a run-Method which multiplies all values in that range.
In you main-method you can do something like that:
int n = 160_000;
int threads = 2;
ExecutorService executor = Executors.newFixedThreadPool(threads);
for (int i = 0; i < threads; i++) {
int start = i * (n/threads) + 1;
int end = (i + 1) * (n/threads) + 1;
executor.execute(new Factorial(start, end));
}
executor.shutdown();
executor.awaitTermination(1, TimeUnit.DAYS);
Now you have calculated the result of each thread but not the overall result. This can be solved by a BigInteger which is visible to the Factorial-class (like a static BigInteger reuslt; in the same main class.) and a lock, too. In the run-method of Factorial you can calculate the overall result by locking the lock and calculation the result:
Main.lock.lock();
Main.result = Main.result.multiply(value);
Main.lock.unlock();
Some additional advice for the future: This isn't really clean because Factorial needs to have information about your main class, so it has a dependency to it. But ExecutorService returns a Future<T>-Object which can be used to receive the result of the thread. Using this Future-Object you don't need to use locks. But this needs some extra work, so just try to get this running for now ;-)
In addition to my Java Stream API solution here another solution which uses a self-managed thread-pool as you demanded:
public static final int CHUNK_SIZE = 10000;
public static BigInteger fac(int max) {
ExecutorService executor = newCachedThreadPool();
try {
return rangeClosed(0, (max - 1) / CHUNK_SIZE)
.mapToObj(val -> executor.submit(() -> prod(leftBound(val), rightBound(val, max))))
.map(future -> valueOf(future))
.reduce(BigInteger.ONE, BigInteger::multiply);
} finally {
executor.shutdown();
}
}
private static int leftBound(int chunkNo) {
return chunkNo * CHUNK_SIZE + 1;
}
private static int rightBound(int chunkNo, int max) {
return Math.min((chunkNo + 1) * CHUNK_SIZE, max);
}
private static BigInteger valueOf(Future<BigInteger> future) {
try {
return future.get();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
private static BigInteger prod(int min, int max) {
BigInteger res = BigInteger.valueOf(min);
for (int val = min + 1; val <= max; val++) {
res = res.multiply(BigInteger.valueOf(val));
}
return res;
}

Set thread number limitation [duplicate]

I want to launch a lot of tasks to run on a database of +-42Mio records. I want to run this in batches of 5000 records/time (results in 850 tasks).
I also want to limit the number of threads (to 16) java starts to do this for me and I am using the current code to accomplish this task:
ExecutorService executorService = Executors.newFixedThreadPool(16);
for (int j = 1; j < 900 + 1; j++) {
int start = (j - 1) * 5000;
int stop = (j) * 5000- 1;
FetcherRunner runner = new FetcherRunner(routes, start, stop);
executorService.submit(runner);
Thread t = new Thread(runner);
threadsList.add(t);
t.start();
}
Is this the correct way to do this? Particularly as I have the impression that java just fires away all tasks ...(FetcherRunner implements runnable)
The first part using ExecutorService looks good:
...
FetcherRunner runner = new FetcherRunner(routes, start, stop);
executorService.submit(runner);
The part with Thread should not be there, I am assuming you have it there just to show how you had it before?
Update:
Yes, you don't require the code after executorService.submit(runner), that is going to end up spawning a huge number of threads. If your objective is to wait for all submitted tasks to complete after the loop, then you can get a reference to Future when submitting tasks and wait on the Future, something like this:
ExecutorService executorService = Executors.newFixedThreadPool(16);
List<Future<Result>> futures = ..;
for (int j = 1; j < 900+ 1; j++) {
int start = (j - 1) * 5000;
int stop = (j) * 5000- 1;
FetcherRunner runner = new FetcherRunner(routes, start, stop);
futures.add(executorService.submit(runner));
}
for (Future<Result> future:futures){
future.get(); //Do something with the results..
}
Is this the correct way of working?
The first part is correct. But you shouldn't be creating and starting new Thread objects. When you submit the Runnable, the ExecutorService puts it on its queue, and then runs it when a worker thread becomes available.
.... I use the threadlist to detect when all my threads are finished so I can continue processing results.
Well if you do what you are currently doing, you are running each task twice. Worse still, the swarm of manually created threads will all try to run in parallel.
A simple way to make sure that all of the tasks have completed is to call awaitTermination(...) on the ExecutorService. (An orderly shutdown of the executor service will have the same effect ... if you don't intend to use it again.)
The other approach is to create a Future for each FetcherRunner's results, and attempt to get the result after all of the tasks have been submitted. That has the advantage that you can start processing early results before later ones have been produced. (However, if you don't need to ... or can't ... do that, using Futures won't achieve anything.)
You don't need to the part after the call to submit. The code you have that creates a Thread will result in 900 threads being created! Yowza. The ExecutorService has a pool of 16 threads and you can run 16 jobs at once. Any jobs submitted when all 16 threads are busy will be queued. From the docs:
Creates a thread pool that reuses a fixed number of threads operating
off a shared
unbounded queue. At any point, at most nThreads threads will be active processing tasks.
If additional tasks are submitted when all threads are active, they will wait in the
queue until a thread is available. If any thread terminates due to a failure during
execution prior to shutdown, a new one will take its place if needed to execute
subsequent tasks. The threads in the pool will exist until it is explicitly shutdown.
So there is no need for yet another thread. If you need to be notified after a task has finished you can have it call out. Other options are to cache all of the Future's returned from submit, and upon each task being finished you can check to see if all Future's are done. After all Future's are finished you can dispatch another function to run. But it will run ON one of the threads in the ExecutorService.
Changed from your code:
ExecutorService executorService = Executors.newFixedThreadPool(16);
for (int j = 1; j < 900 + 1; j++) {
int start = (j - 1) * 5000;
int stop = (j) * 5000 - 1;
FetcherRunner runner = new FetcherRunner(routes, start, stop);
executorService.submit(runner);
}
The best way would be to use countdownlatch as follows
ExecutorService executorService = Executors.newFixedThreadPool(16);
CountdownLatch latch = new CountdownLatch(900);
FetcherRunner runner = new FetcherRunner(routes, start, stop, latch);
latch.await();
in the FetcherRunner under finally block use latch.countDown(); code after await() will be executed only when all the tasks are completed.

How to run a Thread for a user specified amount of time?

Am creating a program that is based on mixing and making perturbation in a population containing solutions Vector.
So I created a for loop that stops after a certain time given by the user.
Inside the loop, am going to call 5 procedures and I thought that if i put each procedure in a thread will make the program making more solutions in a same time than calling normal methods.
Here 5 created the 5 threads, but when i start them the don't want to stop even if i use the Thread.stop, Thread.suspend, Thread.interrupt or Thread.destroy
Here is my code and could u help me with your ideas ?
I have inserted a new variable :
public volatile boolean CrossOpb = true;`
Here is my code:
Thread CrossOp = new Thread(new Runnable() {
public void run() {
while(CrossOpb == true){
int rdmCross2=(int) (Math.random() * allPopulation.size()) ; // Crossover 1st vector
int rdmCross1=(int) (Math.random() * allPopulation.size()) ;
Vector muted = new Vector();
Vector copy = copi((Vector) allPopulation.get(rdmCross2));
Vector callp = copi((Vector) allPopulation.get(rdmCross1));
muted = crossover(callp, copy);
System.out.println("cross over Between two Randoms ----------->");
affiche_resultat(muted);
allPopulation.add(muted);
}
}
});
The loop :
CrossOp.setDaemon(true);
int loop = 1;
long StartTime = System.currentTimeMillis() / 1000;
for (int i = 0; i < loop; ++i) {
loop++;
if (timevalue < ((System.currentTimeMillis() / 1000) - StartTime)) {
loop = 0;
CrossOpb = false;
}
CrossOp.start();
}
I already answered to a similar question. In that case, it was C#, but the concept is the same.
You must not kill threads. Threads must exit on their own will.
Just put a volatile boolean variable somewhere, and set it to true/false, when you want your thread to terminate, then, in the thread, replace the while (true) with a while (myVariable == true/false).
Anyway, you say:
Inside the loop, am going to call 5 procedures ant i thought that if i put each procedure in a thread will make the program making more solutions in a same time than calling normal methods.
Well, that's generally false. If the procedures are data-dependent (each of them depends on the results of the previous one), putting them on threads will change nothing. It might be smarter to put iterations in a pipeline, so that you have 5 threads executing steps of successive iterations. I'm not sure if that's possible for genetic algorithms, and anyway you'll have to handle some special case (e.g. a mutation, that alters the population of partially computed iterations).
How to run a Thread for a specific amount of time:
Here is the basic approach is to keep calculate how long the Thread has run and exit and return the result, which in our case here is details on how long the Thread executed.
NOTE: you must use System.nanoTime() as System.currentTimeMillis() will just return the same thing every time you call it in the method.
I use a Random number to calculate different lifetimes for each of the Callables so that you can see that they don't execute exactly for the time specified but they are very very close, and the variance of the delta is pretty consistent, at least on my machine.
Here a Gist of the code below for easier access.
package com.stackoverflow.Q18818482;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.Random;
import java.util.concurrent.*;
public class Question18818482
{
public static Random RND;
static
{
RND = new Random();
}
public static void main(final String[] args)
{
try
{
final ExecutorService es = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
final List<Future<String>> results = new ArrayList<>(10);
for (int i = 0; i < 10; i++)
{
results.add(es.submit(new TimeSliceTask(RND.nextInt(10), TimeUnit.SECONDS)));
}
es.shutdown();
while(!results.isEmpty())
{
final Iterator<Future<String>> i = results.iterator();
while (i.hasNext())
{
final Future<String> f = i.next();
if (f.isDone())
{
System.out.println(f.get());
i.remove();
}
}
}
}
catch (InterruptedException e)
{
throw new RuntimeException(e);
}
catch (ExecutionException e)
{
throw new RuntimeException(e);
}
}
public static class TimeSliceTask implements Callable<String>
{
private final long timeToLive;
private final long duration;
public TimeSliceTask(final long timeToLive, final TimeUnit timeUnit)
{
this.timeToLive = System.nanoTime() + timeUnit.toNanos(timeToLive);
this.duration = timeUnit.toMillis(timeToLive);
}
#Override
public String call() throws Exception
{
while( timeToLive <= System.nanoTime() )
{
// simulate work here
Thread.sleep(500);
}
final long end = System.nanoTime();
return String.format("Finished Elapsed Time = %d, scheduled for %d", TimeUnit.NANOSECONDS.toMillis(timeToLive - end), this.duration );
}
}
}
Here is what one runs output looks like
NOTE: All times are in milliseconds
Finished Elapsed Time = 999, scheduled for 1000
Finished Elapsed Time = 2998, scheduled for 3000
Finished Elapsed Time = 5999, scheduled for 6000
Finished Elapsed Time = 1994, scheduled for 2000
Finished Elapsed Time = 8994, scheduled for 9000
Finished Elapsed Time = 6993, scheduled for 7000
Finished Elapsed Time = 6993, scheduled for 7000
Finished Elapsed Time = 5993, scheduled for 6000
Finished Elapsed Time = 5998, scheduled for 6000
After reading the whole last night about threads, i have discovered that the solution for my problem was not that hard.
The idea was to edit the condition of the stopping loop inside the thread so we control it by giving it a specific amount of time to run for it and here is my Example :
class ProcessorCordm extends Thread {
int runningtime;
public ProcessorCordm(int runningtime) {
this.runningtime = runningtime;
}
public void run() {
int loop = 1;
long StartTime = System.currentTimeMillis() / 1000;
for (int i = 0; i < loop; ++i) {
int rdmCross2 = (int) (Math.random() * allPopulation.size()); // Crossover 1st vector
int rdmCross1 = (int) (Math.random() * allPopulation.size());
Vector muted = new Vector();
Vector copy = copi((Vector) allPopulation.get(rdmCross2));
Vector callp = copi((Vector) allPopulation.get(rdmCross1));
muted = crossover(callp, copy);
System.out.println("cross over Between two Randoms ----------->");
affiche_resultat(muted);
addsolution(muted);
loop++;
if (timevalue < ((System.currentTimeMillis() / 1000) - StartTime)) {
loop = 0;
}
}
}
}
So if i want to run my Thread for 10 seconds i only need to :
ProcessorCoG CrossOpg = new ProcessorCoG(10);
And fo my case, I have to call many Threads simultaneously working for a specific TimeValue so i used the ExecutorServiceClass :
ProcessorCoG CrossOpg = new ProcessorCoG(timevalue);//extends Thread class
ProcessorCordm CrossOp = new ProcessorCordm(timevalue);//extends Thread class
ProcessorCordm CrossOp2 = new ProcessorCordm(timevalue);//extends Thread class
MutateGb MutGb = new MutateGb(timevalue);//extends Thread class
MutateRdm MutRdm = new MutateRdm(timevalue);//extends Thread class
MbsRdm MbsR = new MbsRdm(timevalue);//extends Thread class
ExecutorService executor = Executors.newFixedThreadPool(6);
executor.submit(MutGb);
executor.submit(MutRdm);
executor.submit(CrossOp);
executor.submit(CrossOp2);
executor.submit(CrossOpg);
executor.submit(MbsR);

Split Files in a directory uniformly across threads in JAVA

I have a variable list of files in a directory and I have different threads in Java to process them. The threads are variable depending upon the current processor
int numberOfThreads=Runtime.getRuntime().availableProcessors();
File[] inputFilesArr=currentDirectory.listFiles();
How do I split the files uniformly across threads? If I do simple math like
int filesPerThread=inputFilesArr.length/numberOfThreads
then I might end up missing some files if the inputFilesArr.length and numberOfThreads are not exactly divisible by each other. What is an efficient way of doing this so that the partition and load across all the threads are uniform?
Here is another take on this problem:
Use java's ThreaPoolExecutor. Here is an example.
It works on the principle of Thread Pool (you need not create threads every time you need but creates a specified number of threads at the start and uses the threads from the pool)
Idea is to treat the processing of each file in a directory as independent task, to be performed by each thread.
Now when you submit all tasks to the executor in loop (this makes sure that no files are left out).
Executor will actually add all of these tasks to a queue and the same time it will pick up threads from the Thread pool and assign them the task till all the threads are busy.
It waits till a thread becomes available. So configuring the threadpool size is vital here. Either you can have as many threads as number of files or lesser number than that.
Here I made an assumption that each file to be processed is independent of each other and its not required that a certain bunch of files to be processed by a single thread.
You can use round robin algorithm for most optimal distribution. Here is the pseudocode:
ProcessThread t[] = new ProcessThread[Number of Cores];
int i = 0;
foreach(File f in files)
{
t[i++ % t.length].queueForProcessing(f);
}
foreach(Thread tt in t)
{
tt.join();
}
The Producer Consumer pattern will solve this gracefully. Have one producer (the main thread) put all the files on a bound blocking queue (see BlockingQueue). Then have a number of worker threads take a file from the queue and process it.
The work (rather than the files) will be uniformly distributed over threads, since threads that are done processing one file, come ask for the next file to process. This avoids the possible problem that one thread gets assigned only large files to process, and other threads get only small files to process.
you can try to get the range (index of start and end in inputFilesArr) of files per thread:
if (inputFilesArr.length < numberOfThreads)
numberOfThreads = inputFilesArr.length;
int[][] filesRangePerThread = getFilesRangePerThread(inputFilesArr.length, numberOfThreads);
and
private static int[][] getFilesRangePerThread(int filesCount, int threadsCount)
{
int[][] filesRangePerThread = new int[threadsCount][2];
if (threadsCount > 1)
{
float odtRangeIncrementFactor = (float) filesCount / threadsCount;
float lastEndIndexSet = odtRangeIncrementFactor - 1;
int rangeStartIndex = 0;
int rangeEndIndex = Math.round(lastEndIndexSet);
filesRangePerThread[0] = new int[] { rangeStartIndex, rangeEndIndex };
for (int processCounter = 1; processCounter < threadsCount; processCounter++)
{
rangeStartIndex = rangeEndIndex + 1;
lastEndIndexSet += odtRangeIncrementFactor;
rangeEndIndex = Math.round(lastEndIndexSet);
filesRangePerThread[processCounter] = new int[] { rangeStartIndex, rangeEndIndex };
}
}
else
{
filesRangePerThread[0] = new int[] { 0, filesCount - 1 };
}
return filesRangePerThread;
}
If you are dealing with I/O even with one processor multiple threads can work in parallel, because while one thread is waiting on read(byte[]) processor can run another thread.
Anyway, this is my solution
int nThreads = 2;
File[] files = new File[9];
int filesPerThread = files.length / nThreads;
class Task extends Thread {
List<File> list = new ArrayList<>();
// implement run here
}
Task task = new Task();
List<Task> tasks = new ArrayList<>();
tasks.add(task);
for (int i = 0; i < files.length; i++) {
if (task.list.size() == filesPerThread && files.length - i >= filesPerThread) {
task = new Task();
tasks.add(task);
}
task.list.add(files[i]);
}
for(Task t : tasks) {
System.out.println(t.list.size());
}
prints 4 5
Note that it will create 3 threads if you have 3 files and 5 processors

Categories