We Keep Coding

How to manage ThreadPoolTaskExecutor to block some requests

We have advanced HR system which has a service calculates employee attendance and leaves it must follow the following if calculating for employee (X, Y, Z) it opens 3 threads and calculate in parallel but if a request to calculate data for employee X again before the previous calculation end it must postpone until previous thread calculating data for employee X finishes.
ScheduleWeekAttendanceBean scheduleWeekAttendanceBean = null;
ThreadPoolExecutor threadPoolExecutor = employeeThreadPoolExecutorMap.get(employmentBean.id);
if (threadPoolExecutor == null || threadPoolExecutor.isTerminating() || threadPoolExecutor.isTerminated()) {
threadPoolExecutor = new ThreadPoolExecutor(1, 1, 0L, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>(), new RejectedExecutionHandlerImpl());
employeeThreadPoolExecutorMap.put(employmentBean.id, threadPoolExecutor);
ThreadPoolTaskExecutorMonitorService threadPoolTaskExecutorMonitorService = new ThreadPoolTaskExecutorMonitorService(threadPoolExecutor, "#" + employmentBean.employeeId);
Thread thread = new Thread(threadPoolTaskExecutorMonitorService);
thread.start();
}
AttendanceBuilder attendanceBuilder = (AttendanceBuilder) AppContext.getBean("attendanceBuilder");
attendanceBuilder.initialize(employmentBean, selectedDate);
Future<ScheduleWeekAttendanceBean> future = threadPoolExecutor.submit(attendanceBuilder);
scheduleWeekAttendanceBean = future.get();
if (threadPoolExecutor.getActiveCount() == 0) {
employeeThreadPoolExecutorMap.remove(employmentBean.id);
threadPoolExecutor.shutdownNow();
}
return scheduleWeekAttendanceBean;
What happens here it process them one by one I need to implement this logic but only block if same employee exists in the map.

Creating separate ThreadPoolExecutor for each employee id is overhead. Each ThreadPoolExecutor contains a Thread which consumes a lot of memory, and this may lead to fatal OutOfMemoryError. So I suggest to use SerialExecutor instead, which is described in the documentation to the java.util.concurrent.Executor. SerialExecutor does not contain thread, instead, it uses an external Executor. You can create single Executor for all SerialExecutors, and tune its configuration (number of threads). Since SerialExecutor is small, you can keep all of them in the employeeThreadPoolExecutorMap permanently.
Another approach is to use Actors instead of Executors. An actor can be considered as a specialized Executor, designed to process specific tasks (messages). You can use Akka actors, or my Simple Actor.

ThreadPoolExecutor - dynamic configuration, pool size vs queue size

In a java project, in order to execute tasks in an asynchronous manner I'm using ThreadPoolExecutor. As expected while I'm creating my executor I'm setting the core and max pool sizes and a bounded (fixed-capacity) queue as well.
If my tasks get rejected (all threads are busy and the queue is full) I'm increasing dynamically the max-pool-size of my executor, this way I get more threads pulling tasks from my queue.
This is the standard approach I have been using so far but lately I encountered a different approach:
In this approach you set core and max pool sizes and an unbounded queue and you limit your queue as follows:
public class AsyncTaskexecutor {
private ThreadPoolExecutor threadPoolExecutor;
private int maxTasksInQueue;
private BlockingQueue queue;
public AsyncTaskexecutor(ThreadPoolExecutor threadPoolExecutor) {
this.queue = new LinkedBlockingQueue<>(); // creaing unbounded queue
this.threadPoolExecutor = createThreadPoolExecutor();
this.maxTasksInQueue = 100;
}
/**
* #return true if the task is either executed or pending in the queue for execution, else false (meaning rejected)
**/
public boolean executeAsync(Runnable task) {
if(this.queue.size() < maxTasksInQueue) {
threadPoolExecutor.execute(task);
return true;
} else {
return false; // rejected
}
}
private ThreadPoolExecutor createThreadPoolExecutor() {
ThreadFactory threadFactory = Executors.defaultThreadFactory();
ThreadPoolExecutor executor = new ThreadPoolExecutor(2, 4, 10, TimeUnit.SECONDS, this.queue, threadFactory, ThreadPoolExecutor.AbortPolicy);
return executor;
}
public void setQueueSize(int newSize) {
this.maxTasksInQueue = newSize;
}
}
So once tasks are rejected it is possible to call the setQueueSize method and dynamically increase the number of elements in the queue.
In the first approach we can play with max-pool-size which means we are bound to cpu resource while in the second approach we can play with the number of tasks in the queue which means we are bound to memory resource.
Are both approaches valid to handle bursts of tasks (avoid rejection as much as possible)?
Any other advantages/disadvantages I am missing here?

Create and add Runnable only when one/more of the worker Thread is available..?

I am executing millions of iteration and I want to parallelize this. Hence decided to add the task [each iteration] to the Thread Pool.
Now, if I add all the iteration to the Thread Pool, it might throw an OutOfMemoryError. I want to handle that gracefully, so is there any way to know about the availability of the worker Thread in the Thread Pool?
Once it's available, add the Runnable to the Worker Thread.
for(int i=0; i<10000000000; i++) {
executor.submit(new Task(i));
}
Each of those tasks merely take 1 sec to complete.

Why don't you set a limit to how many tasks can run concurrently. Like:
HashSet<Future> futures = new HashSet<>();
int concurrentTasks = 1000;
for (int ii=0; ii<100000000; ii++) {
while(concurrentTasks-- > 0 && ii<100000000) {
concurrentTasks.add(executor.submit(new Task(ii)));
}
Iterator<Future> it = concurrentTasks.iterator();
while(it.hasNext()) {
Future task = it.next();
if (task.isDone()) {
concurrentTasks++;
it.remove();
}
}
}

You'll want to use something like this:
ArrayBlockingQueue<Runnable> queue = new ArrayBlockingQueue<Runnable>(MAX_PENDING_TASKS);
Executor executor = new ThreadPoolExecutor(MIN_THREADS, MAX_THREADS, IDLE_TIMEOUT, TimeUnit.SECONDS, queue, new ThreadPoolExecutor.CallerRunsPolicy());
for(int i=0; i<10000000000; i++) {
executor.submit(new Task(i));
}
Basically you create a thread pool with min/max threads and an array backed queue. When you hit the limit of pending tasks, the "caller runs policy" kicks in and your main thread ends up running the next task (giving time for your other tasks to complete and open slots in the queue).
Since you've stated that your tasks are short lived, this seems like an optimal strategy.
The values for MAX_PENDING_TASKS and MIN_THREADS are something you can fiddle with to figure out what the optimal values are for your workload, but MAX_PENDING_TASKS should be at least twice MIN_THREADS and probably more like 10 to 100 times.

You should use java.lang.Runtime
The biggest memory issue is probably going to be your Object creation, not in adding them to your Executor, so that's where you should be calling Runtime.getRuntime().freeMemory().

Recursive concurrency

I have the following function, in pseudo-code:
Result calc(Data data) {
if (data.isFinal()) {
return new Result(data); // This is the actual lengthy calculation
} else {
List<Result> results = new ArrayList<Result>();
for (int i=0; i<data.numOfSubTasks(); ++i) {
results.add(calc(data.subTask(i));
}
return new Result(results); // merge all results in to a single result
}
}
I want to parallelize it, using a fixed number of threads.
My first attempt was:
ExecutorService executorService = Executors.newFixedThreadPool(numOfThreads);
Result calc(Data data) {
if (data.isFinal()) {
return new Result(data); // This is the actual lengthy calculation
} else {
List<Result> results = new ArrayList<Result>();
List<Callable<Void>> callables = new ArrayList<Callable<Void>>();
for (int i=0; i<data.numOfSubTasks(); ++i) {
callables.add(new Callable<Void>() {
public Void call() {
results.add(calc(data.subTask(i));
}
});
}
executorService.invokeAll(callables); // wait for all sub-tasks to complete
return new Result(results); // merge all results in to a single result
}
}
However, this quickly got stuck in a deadlock, because, while the top recursion level waits for all threads to finish, the inner levels also wait for threads to become available...
How can I efficiently parallelize my program without deadlocks?

Your problem is a general design problem when using ThreadPoolExecutor for tasks with dependencies.
I see two options:
1) Make sure to submit tasks in a bottom-up order, so that you never have a running task that depends on a task which didn't start yet.
2) Use the "direct handoff" strategy (See ThreadPoolExecutor documentation):
ThreadPoolExecutor executor = new ThreadPoolExecutor(poolSize, poolSize, 0, TimeUnit.SECONDS, new SynchronousQueue<Runnable>());
executor.setRejectedExecutionHandler(new CallerRunsPolicy());
The idea is using a synchronous queue so that tasks never wait in a real queue. The rejection handler takes care of tasks which don't have an available thread to run on. With this particular handler, the submitter thread runs the rejected tasks.
This executor configuration guarantees that tasks are never rejected, and that you never have deadlocks due to inter-task dependencies.

you should split your approach in two phases:
create all the tree down until data.isFinal() == true
recursively collect the results (only possible if the merging does not produce other operations/calls)
To do that, you can use [Futures][1] to make the results async. Means all results of calc will be of type Future[Result].
Immediately returning a Future will free the current thread and give space for the processing of others. With the collection of the Results (new Result(results)) you should wait for all results to be ready (ScatterGather-Pattern, you can use a semaphore to wait for all results). The collection itself will be walking a tree and checking (or waiting for the results to arrive) will happen in a single thread.
Overall you build a tree of Futures, that is used to collect the results and perform only the "expensive" operations in the threadpool.

How to implement a queue that can be processed by multiple threads?

I think I'm doing it wrong. I am creating threads that are suppose to crunch some data from a shared queue. My problem is the program is slow and a memory hog, I suspect that the queue may not be as shared as I hoped it would be. I suspect this because in my code I added a line that displayed the size of the queue and if I launch 2 threads then I get two outputs with completely different numbers and seem to increment on their own(I thought it could be the same number but maybe it was jumping from 100 to 2 and so on but after watching it shows 105 and 5 and goes at a different rate. If I have 4 threads then I see 4 different numbers).
Here's snippet of the relevant parts. I create a static class with the data I want in the queue at the top of the program
static class queue_class {
int number;
int[] data;
Context(int number, int[] data) {
this.number = number;
this.data = data;
}
}
Then I create the queue after sending some jobs to the callable..
static class process_threaded implements Callable<Void> {
// queue with contexts to process
private Queue<queue_class> queue;
process_threaded(queue_class request) {
queue = new ArrayDeque<queue_class>();
queue.add(request);
}
public Void call() {
while(!queue.isEmpty()) {
System.out.println("in contexts queue with a size of " + queue.size());
Context current = contexts.poll();
//get work and process it, if it work great then the solution goes elsewhere
//otherwise, depending on the data, its either discarded or parts of it is added back to queue
queue.add(new queue_class(k, data_list));
As you can see, there's 3 options for the data, get sent off if data is good, discard if its totally horrible or sent back to the queue. I think the queues are going when its getting sent back but I suspect because each thread is working on its own queue and not a shared one.
Is this guess correct and am I doing this wrong?

You are correct in your assessment that each thread is (probably) working with its own queue, since you are creating a queue in the constructor of your Callable. (It's actually very weird to have a Callable<Void> -- isn't that just a Runnable?)
There are other problems there, for example, the fact that you're working with a queue that isn't thread-safe, or the fact that your code won't compile as it is written.
The important question, though, is do you really need to explicitly create a queue in the first place? Why not have an ExecutorService to which you submit your Callables (or Runnables if you decide to make that switch): Pass a reference to the executor into your Callables, and they can add new Callables to the executor's queue of tasks to run. No need to reinvent the wheel.
For example:
static class process_threaded implements Runnable {
// Reference to an executor
private final ExecutorService exec;
// Reference to the job counter
private final AtomicInteger jobCounter;
// Request to process
private queue_class request;
process_threaded( ExecutorService exec, AtomicInteger counter, queue_class request) {
this.exec = exec;
this.jobCounter = counter;
this.jobCounter.incrementAndGet(); // Assuming that you will always
// submit the process_threaded to
// the executor if you create it.
this.request = request;
}
public run() {
//get work and process **request**, if it work great then the solution goes elsewhere
//otherwise, depending on the data, its either discarded or parts of are added back to the executor
exec.submit( new process_threaded( exec, new queue_class(k, data_list) ) );
// Can do some more work
// Always run before returning: counter update and notify the launcher
synchronized(jobCounter){
jobCounter.decrementAndGet();
jobCounter.notifyAll();
}
}
}
Edit:
To solve your problem of when to shut down the executor, I think the simplest solution is to have a job counter, and shutdown when it reaches 0. For thread-safety an AtomicInteger is probably the best choice. I added some code above to incorporate the change. Then your launching code would look something like this:
void theLauncher() {
AtomicInteger jobCounter = new AtomicInteger( 0 );
ExecutorService exec = Executors.newFixedThreadPool( Runtime.getRuntime().availableProcesses());
exec.submit( new process_threaded( exec, jobCounter, someProcessRequest ) );
// Can submit some other things here of course...
// Wait for jobs to complete:
for(;;jobCounter.get() > 0){
synchronized( jobCounter ){ // (I'm not sure if you have to have the synchronized block, but I think this is safer.
if( jobCounter.get() > 0 )
jobCounter.wait();
}
}
// Now you can shutdown:
exec.shutdown();
}

Don't reinvent the wheel! How about using ConcurrentLinkedQueue? From the javadocs:
An unbounded thread-safe queue based on linked nodes. This queue orders elements FIFO (first-in-first-out). The head of the queue is that element that has been on the queue the longest time. The tail of the queue is that element that has been on the queue the shortest time. New elements are inserted at the tail of the queue, and the queue retrieval operations obtain elements at the head of the queue. A ConcurrentLinkedQueue is an appropriate choice when many threads will share access to a common collection.