I'm trying to have a single thread loading records (say from a database). This thread feeds records into a thread pool that processes these individual tasks.
I was expecting this code to work, but it prints number until 60 and then stops.
ThreadPoolTaskExecutor accountLoaderTaskExecutor = new ThreadPoolTaskExecutor();
accountLoaderTaskExecutor.setCorePoolSize(1);
accountLoaderTaskExecutor.setMaxPoolSize(1);
accountLoaderTaskExecutor.initialize();
ThreadPoolTaskExecutor accountDeletionTaskExecutor = new ThreadPoolTaskExecutor();
accountDeletionTaskExecutor.setCorePoolSize(10);
accountDeletionTaskExecutor.setMaxPoolSize(10);
accountDeletionTaskExecutor.setQueueCapacity(50);
accountDeletionTaskExecutor.initialize();
accountLoaderTaskExecutor.submit(() -> {
List<Integer> customerAccountIds = getCustomerAccountIds(); // return 1000s integers
customerAccountIds.forEach(id -> {
accountDeletionTaskExecutor.submit(() -> {
try {
System.out.println(id);
Thread.sleep(500);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
});
});
});
Thread.currentThread().join();
I was expecting the accountLoaderTaskExecutor thread to block on accountDeletionTaskExecutor.submit but then continue as records are being processed until it exhausts all customerAccountIds.
If that comment "return 1000s integers" means that you have thousands of ids, then the code you have written will queue thousands of tasks to the threadpool and then proceed to Thread.currentThread().join() which does absolutely nothing, because joining the current thread to itself is meaningless: you can only join different threads.
Then, I presume you exit the application, and the default behavior is probably to terminate all threadpools on application exit. (I am not sure about that, I am speculating.)
The 60 tasks that you observe getting started probably manage to start while the remaining thousands of tasks are being queued.
To verify that this is what is happening, you can try replacing that call to join() with a Thread.sleep( 1000 ) and see if you observe more tasks being started.
If that is the case, then one approach to solve your problem might be to add proper graceful threadpool shut-down (of the kind that waits for all queued tasks to complete first.)
Related
The service I'm working on uses a Future to run multiple tasks in parallel; each task can take up to a minute to complete. However, it seems the external lib is buggy, since in some occasions (2% of the time) it doesn't return. In those cases I would like to give a 2-minute wait time, and if it hasn't returned, I would like to kill the future and re-schedule again later (it will succeed eventually).
How do I kill the Future?
private void run() {
ExecutorService queue = Executors.newFixedThreadPool(1);
Future<Integer> f = queue.submit(new MyTask());
Thread.sleep(500);
try {
Integer r = f.get(120, TimeUnit.SECONDS);
} catch (InterruptedException | ExecutionException | TimeoutException e) {
e.printStackTrace();
f.cancel(true);
}
// Bad future still running here and I need it dead.
}
private class MyTask implements Callable<Integer> {
private ExternalLibrary extlib = new ExternalLibrary();
#Override
public Integer call() throws Exception {
// step 1 - do a few things
// step 2 - process data
Integer val = this.extlib.doSomething(); // here's the problem!
// step 3 - do other things
return val;
}
}
I can see the external lib running and consuming CPU (for 24 hours)... doing nothing. It's a simple task that should never take more than 60 seconds to complete its work.
So far, I'm killing the whole JVM once a day to get rid of this issue, but I'm sure there must be a better way. I wonder how app servers (Tomcat, JBoss, Weblogic, etc.) do it with rogue processes.
Even if you could kill the future hanging in the buggy library, this does likely not solve your problem. The library might still have acquired some resource which will not be properly clean up. This might be memory allocations, open file handles or even monitors leaving some internal data structures in an inconsistent state. Eventually you will likely be back at the point where you have to restart your JVM.
There's basically two options: Fix or isolate it.
Fix: try to get the library fixed. If this is not possible,
isolate: isolate the library into a external service your application depends on. E.g. implement a REST API for calling the library and wrap everything up into a Docker image. Automate restarting of the Docker container as needed.
As others have mentioned, stopping a Future is cooperative, meaning, the thread running async must respond to cancellation from the waiting thread. If the async task isn't cooperative simply invoking shutdown or shutdownNow won't be enough as the underlying TPE will just interrupt the threads.
If you have no control over extlib, and extlib is not cooperative, I see two options
You can stop the thread currently running. This can cause issues if the thread being stopped currently is holding a lock or some other resource. It can lead to interesting bugs that could be hard to dissect.
This could take some more work, but you could run the async task as a separate process entirely. The TPE can still run the process and, on interruption, can destroy the process. This obviously has more interesting issues like how to load the process with required input.
If I understand your requirement correctly & based on your requirement (i.e. 1 thread), you can look for shutting down executorservice in 2 phases, code is available in java doc of executorservice:
try {
Integer r = f.get(120, TimeUnit.SECONDS);
} catch (InterruptedException | ExecutionException | TimeoutException e) {
e.printStackTrace();
//f.cancel(true); you can omit this call if you wish.
shutdownAndAwaitTermination(queue);
} ... //remaining method code
void shutdownAndAwaitTermination(ExecutorService pool) {
pool.shutdown(); // Disable new tasks from being submitted
try {
// Wait a while for existing tasks to terminate
if (!pool.awaitTermination(60, TimeUnit.SECONDS)) {
pool.shutdownNow(); // Cancel currently executing tasks
// Wait a while for tasks to respond to being cancelled
if (!pool.awaitTermination(60, TimeUnit.SECONDS))
System.err.println("Pool did not terminate");
}
} catch (InterruptedException ie) {
// (Re-)Cancel if current thread also interrupted
pool.shutdownNow();
// Preserve interrupt status
Thread.currentThread().interrupt();
}
}
Please read documentation about shutdown() , shutdownNow() how they behaves because it clearly mentions there is no 100% guarantee that tasks / executorservice will get stopped if its running.
Unfortunately if the external library is not co-operating to thread interrupts, there is nothing you can do to kill the Thread running the task managed by the ExecutorService.
An alternative that I can think of is to run the offending code as a separate process. Using ProcessBuilder and Process, your task can effectively control (or) even kill the offending process after a timeout (https://docs.oracle.com/javase/9/docs/api/java/lang/Process.html#destroyForcibly--).
Also see https://docs.oracle.com/javase/9/docs/api/java/lang/ProcessBuilder.html
#joe That is correct. Unless you have control over the thread and inside the thread you can't kill it.
this.extlib.doSomething();
if this line starts a thread then we need to get hold of that thread to kill it as we don't have reference to stop it.
In your code, the call:
this.extlib.doSomething()
must be synchronous, because if it is not, the code lost sense. With that assumption, you can try:
ExecutorService executor = Executors.newSingleThreadExecutor();
Future<Integer> future = executor.submit(new MyTask());
try {
future.get(120, TimeUnit.SECONDS);
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
} catch (TimeoutException e) {
future.cancel(true);
} finally {
executor.shutdownNow();
}
If this doesn't stop the doSomethig work is because this doSomething function is opening other threads to do the work. In that case, maybe you can check the threads that are running with:
Thread.getAllStackTraces()
And try to kill the right one...
Using #Scheduling to run method at #Scheduled(fixedRate = 10000) and set up #Scheduling threading by implementing SchedulingConfigurer
#Bean(destroyMethod="shutdown")
public Executor taskExecutor() {
return Executors.newScheduledThreadPool(10);
}
if I used Thread.sleep or Lock , no other thread is created by Executor unless Thread.sleep wake up or lock is cleared.
Can someone explain internal working if i have 10 pool size they 10 threads should be created at rate of 10000 millisec.
Basically such behavior comes from ScheduledExecutorService implementation which is used internally by spring. If you will run this code you will notice the same behavior:
public static void main(String[] args) throws Exception {
ScheduledExecutorService executor = Executors.newScheduledThreadPool(10);
executor.schedule(() -> {
System.out.println("Running task in thread " + Thread.currentThread().getId());
try {
Thread.sleep(Integer.MAX_VALUE);
} catch (InterruptedException e) {
System.out.println("interrupted while sleeping");
}
}, 1000, TimeUnit.MILLISECONDS);
Thread.sleep(10000);
executor.shutdownNow();
}
When you submit task to scheduled thread pool it is wrapped with RunnableScheduledFuture which is passed to delayedExecute method. This method adds the task to the tasks queue and starts new worker if current number of workers is less than corePoolSize. Worker tries to get a task from the queue and process it invoking run method. There is a dedicated DelayedWorkQueue implementation which returns tasks only if they are ready for execution. Here is how run method of RunnableScheduledFuture looks like:
/**
* Overrides FutureTask version so as to reset/requeue if periodic.
*/
public void run() {
boolean periodic = isPeriodic();
if (!canRunInCurrentRunState(periodic))
cancel(false);
else if (!periodic)
ScheduledFutureTask.super.run();
else if (ScheduledFutureTask.super.runAndReset()) {
setNextRunTime();
reExecutePeriodic(outerTask);
}
}
As you can see it invokes actual task logic in runAndReset, calculates the next running time and submits the same updated task to the queue again (reExecutePeriodic is almost the same as schedule). There is only a single periodic task for all executions which is resubmitted again and again with updated time after the previous execution is finished. So such thread pool runs only a single instance of each task type in any given moment and scales only for different type of tasks.
If you are interesting in how spring schedules tasks take a look at ScheduledTaskRegistrar class and particularly at scheduleFixedDelayTask method.
In the case that you use threadpool:
By default you going to have 10 threads in your pool(already initialized). The first time that #scheduled is executed, this function is going to execute in a thread from your pool (now remaining 9 threads), if the function yet finished and #scheduled is executed again, your function going to executed in other thread from you your pool, so now you have 8 thread remaining in your pool. (8 idle, 2 running threads)
If you aren`t use threadpool only one thread is used.
spring documentation:
If you do not provide a 'pool-size' attribute, the default thread pool
will only have a single thread. There are no other configuration
options for the scheduler.
https://docs.spring.io/spring/docs/4.3.x/spring-framework-reference/html/scheduling.html
So I'm trying to implement some kind of fork on a "toy language". It's like implementing my own compiler. So when the user does a fork, my program it's supposed to start/simulate some threading.
In my code, I prepare the callables and then I start the execution of the callables but when I use invokeAll my program doesn't terminate, it just doesn't do anything.
I ran the code through debug and it just stops the debugging when I get to invokeAll() but it doesn't terminate or throw error or anything ( I have the invoke inside a try-catch ). I tried with fixed threading pool too. It doesn't do anything
Some code:
// preparing the callables
java.util.List<Callable<MyClass>> callList = prgll.stream()
.map(p -> (Callable<MyClass>) () -> {
return p.oneStep(); //a method from my class
}).collect(Collectors.toList());
//start the execution of the callables
//it should return the list of new created threads
ExecutorService executor = Executors.newSingleThreadExecutor();
java.util.List<MyClass> fff;
try {
fff = executor.invokeAll(callList).stream() // here my program gets blocked but not all the time, only when I call use myFork class
.map(future-> {
try {
return future.get();
} catch (Exception e) {
e.printStackTrace();
throw new CustomException("Error in onestepforall" + e.getMessage());
}
}).filter(p->p != null).collect(Collectors.toList());
} catch (InterruptedException e) {
e.printStackTrace();
throw new CustomException("Error while trying executor" +e.getMessage());
}
Can I debug this to get deeper inside my code to see exactly why invokeAll stays on stand-by?
I also tried changing from a new SingleThread to a fixed pool but it still didn't do anything.
invokeAll() is a blocking method, i.e. it waits until all futures have completed.
.map the list of callables through the executor's submit() instead if you want async task submission.
If your question is why the tasks don't complete then you don't want to debug the thread submitting to the executor but the spawned thread(s) since the tasks are executed on a separate thread. Instead of using a breakpoint you can simply suspend the whole JVM with the attached debugger and then look at individual threads.
I found the problem. p.oneStep() was never completing for fork because there was an infinite loop inside the method. I got rid of it and everything else works fine.
I am using the Java ExecutorService framework to submit callable tasks for execution.
These tasks communicate with a web service and a web service timeout of 5 mins is applied.
However I've seen that in some cases the timeout is being ignored and thread 'hangs' on an API call - hence, I want to cancel all the tasks that take longer than say, 5 mins.
Currently, I have a list of futures and I iterate through them and call future.get until all tasks are complete. Now, I've seen that the future.get overloaded method takes a timeout and throws a timeout when the task doesnt complete in that window. So I thought of an approach where I do a future.get() with timeout and in case of TimeoutException I do a future.cancel(true) to make sure that this task is interrupted.
My main questions
1. Is the get with a timeout the best way to solve this issue?
2. Is there the possibility that I'm waiting with the get call on a task that hasnt yet been placed on the thread pool(isnt an active worker). In that case I may be terminating a thread that, when it starts may actually complete within the required time limit?
Any suggestions would be deeply appreciated.
Is the get with a timeout the best way to solve this issue?
This will not suffice. For instance, if your task is not designed to response to interruption, it will keep on running or be just blocked
Is there the possibility that I'm waiting with the get call on a task that hasnt yet been placed on the thread pool(isnt an active worker). In that case I may be terminating a thread that, when it starts may actually complete within the required time limit?
Yes, You might end up cancelling as task which is never scheduled to run if your thread-pool is not configured properly
Following code snippet could be one of the way you can make your task responsive to interruption when your task contains Non-interruptible Blocking. Also it does not cancel the task which are not scheduled to run. The idea here is to override interrupt method and close running tasks by say closing sockets, database connections etc. This code is not perfect and you need to make changes as per requirements, handle exceptions etc.
class LongRunningTask extends Thread {
private Socket socket;
private volatile AtomicBoolean atomicBoolean;
public LongRunningTask() {
atomicBoolean = new AtomicBoolean(false);
}
#Override
public void interrupt() {
try {
//clean up any resources, close connections etc.
socket.close();
} catch(Throwable e) {
} finally {
atomicBoolean.compareAndSet(true, false);
//set the interupt status of executing thread.
super.interrupt();
}
}
public boolean isRunning() {
return atomicBoolean.get();
}
#Override
public void run() {
atomicBoolean.compareAndSet(false, true);
//any long running task that might hang..for instance
try {
socket = new Socket("0.0.0.0", 5000);
socket.getInputStream().read();
} catch (UnknownHostException e) {
} catch (IOException e) {
} finally {
}
}
}
//your task caller thread
//map of futures and tasks
Map<Future, LongRunningTask> map = new HashMap<Future, LongRunningTask>();
ArrayList<Future> list = new ArrayList<Future>();
int noOfSubmittedTasks = 0;
for(int i = 0; i < 6; i++) {
LongRunningTask task = new LongRunningTask();
Future f = execService.submit(task);
map.put(f, task);
list.add(f);
noOfSubmittedTasks++;
}
while(noOfSubmittedTasks > 0) {
for(int i=0;i < list.size();i++) {
Future f = list.get(i);
LongRunningTask task = map.get(f);
if (task.isRunning()) {
/*
* This ensures that you process only those tasks which are run once
*/
try {
f.get(5, TimeUnit.MINUTES);
noOfSubmittedTasks--;
} catch (InterruptedException e) {
} catch (ExecutionException e) {
} catch (TimeoutException e) {
//this will call the overridden interrupt method
f.cancel(true);
noOfSubmittedTasks--;
}
}
}
}
execService.shutdown();
Is the get with a timeout the best way to solve this issue?
Yes it is perfectly fine to get(timeout) on a Future object, if the task that the future points to is already executed it will return immediately. If the task is yet to be executed or is being executed then it will wait until timeout and is a good practice.
Is there the possibility that I'm waiting with the get call on a task
that hasnt yet been placed on the thread pool(isnt an active worker)
You get Future object only when you place a task on the thread pool so it is not possible to call get() on a task without placing it on thread pool. Yes there is a possibility that the task has not yet been taken by a free worker.
The approach that you are talking about is ok. But most importantly before setting a threshold on the timeout you need to know what is the perfect value of thread pool size and timiout for your environment. Do a stress testing which will reveal whether the no of worker threads that you configured as part of Threadpool is fine or not. And this may even reduce the timeout value. So this test is most important i feel.
Timeout on get is perfectly fine but you should add to cancel the task if it throws TimeoutException. And if you do the above test properly and set your thread pool size and timeout value to ideal than you may not even need to cancel tasks externally (but you can have this as backup). And yes sometimes in canceling a task you may end up canceling a task which is not yet picked up by the Executor.
You can of course cancel a Task by using
task.cancel(true)
It is perfectly legal. But this will interrupt the thread if it is "RUNNING".
If the thread is waiting to acquire an intrinsic lock then the "interruption" request has no effect other than setting the thread's interrupted status. In this case you cannot do anything to stop it. For the interruption to happen, the thread should come out from the "blocked" state by acquiring the lock it was waiting for (which may take more than 5 mins). This is a limitation of using "intrinsic locking".
However you can use explicit lock classes to solve this problem. You can use "lockInterruptibly" method of the "Lock" interface to achieve this. "lockInterruptibly" will allow the thread to try to acquire a lock while remaining responsive to the interruption. Here is a small example to achieve that:
public void workWithExplicitLock()throws InterruptedException{
Lock lock = new ReentrantLock();
lock.lockInterruptibly()();
try {
// work with shared object state
} finally {
lock.unlock();
}
}
I'm using a Java executor to divide and process a given task in parallel. There could be any number of tasks. They are all queued at the beginning and the master controller object simply waits for all of them to finish.
The problem I'm running into is how to determine when all tasks have been completed. Since one large task is only every queued at a single time (i.e. all currently queued tasks all belong to the same master task) I can use the getCompletedTaskCount() method to compare the number of completed tasks to the number of the number of tasks originally queued.
However, this requires that I constantly poll the executor for the number of completed tasks and in my opinion isn't that great of a solution.
while (pool.getCompletedTaskCount() - start_count < num_tasks)
{
try
{
Thread.sleep(30);
}
catch (InterruptedException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
}
I thought about having a counter object which each task could increment, then if the count is equivalent to the number of expected tasks notify the master thread. Something like this (ignore the somewhat incomplete code like missing exception handlers and such):
Master thread:
counter_object.expected_count = num_tasks;
counter_object.count = 0;
queue_tasks();
synchronized(counter_object)
{
counter_object.wait();
}
// all tasks have finished
Worker tasks:
// ...do task
// task finished, update counter
synchronized(counter_object)
{
++counter_object.count;
if(counter_object.count == counter_object.expected_count)
{
// all tasks have finished, notify master thread
counter_object.notify();
}
}
This method also has the added benefit that I can use a single executor to run multiple master tasks since the counter object would be local to a given master.
Is there a better way to solve this problem? The number of tasks might very well be larger than the max number of threads the executor is allowed to create so I don't think a CyclicBarrier would work.
This sounds like a job of ExecutorService.invokeAll.
Collection<Callable> tasks = <get all sub tasks>;
executorService.invokeAll(tasks);
// Execution proceeds at the following line only once all "tasks" have been run
Alternatively (since you are likely dealing with Runnable instances and not Callable, you can use ExecutorService.submit(Runnable) and then wait for them to complete.
for (Runnable task:tasks) {
futures.add(executorService.submit(task));
}
for (Future<Void> result:futures) {
result.get();
}
Note: exception handling omitted