We Keep Coding

Shutting down a ThreadPoolExecutor when a worker thread has an Exception

What is the best way for a worker thread to signal that a graceful shutdown should be initiated?
I have a fixed size thread pool which works through a continuous set of tasks, each lasting no more than a few seconds. During normal operation this works well and chugs along with its workload.
The problem I am having is when an exception is thrown in one of the threads. If this happens I would like to bring the whole thing down and have been unable to get this working correctly.
Current approach
The naive approach that I have been using is to have a static method in the "Supervisor" class which shuts down the thread pool using the standard shutdown() and awaitTermination() approach. This is then called by any of the "Worker" classes if they encounter a problem. This was done rather than propagating the exception because execute() requires a Runnable and the run() method cannot throw exceptions.
Here is some pseudo code:
// Finds work to do and passes them on to workers
class Supervisor {
ThreadPoolExecutor exec;
static main() {
exec = new FixedThreadPool(...);
forever {
exec.execute(new Worker(next available task));
}
}
static stopThreadPool() {
exec.shutdown();
if(!exec.awaitTermination(timeout_value)) {
print "Timed out waiting on terminate"
}
}
}
class Worker {
run() {
try {
// Work goes here
} catch () {
Supervisor.stopThreadPool()
}
}
}
The effect that I am seeing is that the threads do pause for a while but then I see the timeout message and they all resume their processing. This pattern continues until I manually shut it down. If I put a call to stopThreadPool() in the main method after having broken out of the loop, the shutdown happens correctly as expected.
The approach is clearly wrong because it doesn't work, but it also feels like the design is not right.
To reiterate the question: What is the best way for a worker thread to signal that a graceful shutdown should be initiated?
Additional information
The questions I have looked at on SO have been of two types:
"How do I kill a thread in a thread pool?"
"How do I know all my threads are finished?"
That's not what I'm after. They also seem to exclusively talk about a finite set of tasks whereas I am dealing with a continuous feed.
I have read about an alternative approach using exec.submit() and Futures which puts the onus on the supervisor class to check that everything's ok but I don't know enough about it to know if it's a better design. The exception case is, well ... exceptional and so I wouldn't want to add work/complexity to the normal case unnecessarily.
(Minor side note: This is a work project and there are other people involved. I'm saying "I" in the question for simplicity.)

You are not that far from the correct solution, the problem is that you need to handle the interruption caused by the shutdown call properly. So your thread's run method should look like this:
run () {
try {
while (Thread.interrupted() == false) {
doSomeWork();
}
} catch (Exception e) {
myExecutor.shutdown();
}
}
Note that I explicitly used the shutdown() without awaitTermination() because otherwise the waiting thread is the one that keeps the Executor from properly terminating, because one thread is still waiting. Perfect single-thread deadlock. ;)
The check for interruption is by the way the hint on how to kill a thread gracefully: get the run method to end by either setting a running boolean to false or by interrupting, the thread will die a moment later.
To check if all of your threads have terminated (= are just about to end their run method), you can use a CountDownLatch for a simple case or the CyclicBarrier/Phaser class for more complex cases.

There are 2 problems here:
If you intend to just force a shutdown on any exception in a worker, then you do you use shutdown() and await counterparts. Just force it using shutdownNow and you should be good. shutdown does a graceful shutdown.
try to break your for loop when such a thing happens. The best way to do it is have a try catch in your for loop around the execute call. when an exception happens in a worker throw an unchecked exception and catch it in the for loop. Terminate the for loop and call your method to force shutdown on executor. This is a cleaner approach. Alternately you can also consider using consider a handler in your executor for doing this.

Cleanly Stopping a Thread

I've finally managed to implement Thread.interrupt() into my program instead of Thread.stop(). I am however not sure that I've done this well.
I have a class which extends Thread and declares several methods. EVERY method has been made to throw InterruptedException (Each method performs I/O intensive operations, some of which take several minutes to complete, I have therefore not used a thread-safe flag as the flag would not get checked until after the operation completed). I have also added the following code at several places within these methods to throw the exceptions:
if (this.isInterrupted()) throw new InterruptedException();
Within the run() method I execute all methods within a try/catch for InterruptedException. If caught, I execute Process.destroy() and BufferedReader.close() for my class variables.
This all works, and seems to work very well, however I have a couple of questions:
Is it correct to have more than 10 methods, all of which throw InterruptedException? Is there a better way to do this?
Is it correct to bloat the methods with checks for isInterrupted()?
At the end of the catch InterruptedException block, must I execute a 'return', or 'null' certain values to make the Thread available for GC? If I re-create the Thread it takes longer than usual to initialize.
Finally, are there any issues/enhancements related to what I've done?
Thanks in advance for your help!

Thread interruption in Java doesn't mean stopping the execution of that thread. It is not stop, it is interrupt. A thread can be interrupted when something fundamental and crucial changes, telling the thread that its execution context, its task or its enviroment changed in some significant way. A thread reaction to this message is implementation specific. It can be stop, it can be restart or any other action. A thread that doesn't handle interruptions cannot be interrupted, but its behaviour can still be altered, for example, by using a shared variable.
For example, imagine you have a number of threads, all searching through a part of a problem space for a solution. When one thread finds a solution, it can interrupt other threads, because their search for a solution is no longer relevant. A solution has already been found.
Or imagine one continuously working main thread and one network communication thread. Each time the network thread receives a messsage, it interrupts the working thread with the message. Based on what the message and the context is, the worker thread may decide what to do next. For example, if the message was "STOP", then it could stop all execution immediately. If the message was "RESET", it could start again from scratch or maybe not from scratch and reuse some previous work, based on the execution context.
Is it correct to have more than 10 methods, all of which throw
InterruptedException? Is there a better way to do this?
No, this is perfectly fine, as long as you know what you are doing. If you implement interruptions to just stop the threads, there is no need to throw InterruptedExceptions. A Thread's run() method is it's first, and the exception will not go any further the stack.
Is it correct to bloat the methods with checks for isInterrupted()?
Depending on the context. The checks would be usually added before some crucial code. Usually it is added as a first item in the loop block.
At the end of the catch InterruptedException block, must I execute a
'return', or 'null' certain values to make the Thread available for
GC? If I re-create the Thread it takes longer than usual to
initialize.
No. Once the Thread exists from the run() method, it's left at GC's mercy. Shared variables will not be GC'ed, as long as they are still referenced by other objects.

java ScheduledExecutorService runnable exception handling

I am realizing that if a exception are raised inside(or not, but should be related to) my runnable's run method, all my future tasks will not be run.
So my question is: How can I recover from such a exception (where to catch it)?
I have tried this:
ScheduledExecutorService Exception handling
If i do a while loop to catch the exception, the future tasks are still not executed. I also tried to schedule the catch, no help either.
I tried to put a huge try/catch to wrap all the code in run method but it seems to be not catching anything, and some exception are still not catches and causing all my future tasks to not run.

In the executor framework, you are giving control of running a job away from one main application thread to a thread pool thread. A thread submits the work through a schedule, or submit method is returned a Future object that allows it to get information through a get method. The get method will throw an executor exception whose cause is probably the exception that your code inside the runnable threw. If the main thread does not do that it will never see that exception, so it really depends on your application logic flow.
Another thing also to mention, if you try catch all, what do you mean by that if you are doing something similar to
try {
....
}
catch(Exception e) {
.... }
you are really not catching errors in your app (throwable is the father of Exception and Error) so you might have some static initializer error (an exception caught in a static block)
Again it all depends on how you want exception handling to happen you have full power,
Thank you

Java Exception handling mechanism

In Java,If I didn't catch the thrown Exception then the Thread execution stops there else if I catch the same then the Thread execution continues after the catch block.Why Java Exception handling designed in this way.
Thx

The purpose of an exception is to let the program understand that Something Weird Has Happened, and so doing what would ordinarily be next in the program is very likely to be wrong. A function you called couldn't give you a real answer, and you were relying on that answer, so it had to stop you.
There are two ways this can end: you handle the exception (via a catch block), or the entire program stops.
If your program doesn't know what to do when these things happen, it's best to do nothing. Let the Exception crash the thread, and then you can check the crash log later, and find out exactly what crashed the program and why. If your program can't handle an error, the Exception's "crash the thread" behavior lets you see what error was unhandled, so you can change the program to make it able to handle this kind of situation in the future, or prevent the situation from occuring.
Some errors might be pretty normal, though, and shouldn't stop the entire program- you need to have a way to recover from them. That's what a catch block is for: a chance to say "Hello Java, I know just what to do about this problem", and then do it. A Catch block lets you clean up the program and move on, if you can. Java assumes your Catch block deals with the problem and makes it go away. If there's a new problem, or for that matter the same one, you need to throw the caught exception again, or maybe a new one, so something else can try to fix the problem- even if that something else is you, as a programmer.
If exceptions always crashed the program, there would be no way to handle errors that are expected and can be dealt with. But if absolutely nothing is ready to handle the error, the program can't keep running, because now something's gone weird and it doesn't know what to do because you didn't program it to do anything.

In Java,If I didn't catch the thrown Exception then the Thread execution stops there else if I catch the same then the Thread execution continues after the catch block.
There is a third case. If the thread has an Thread.UncaughtExceptionHandler (registered by calling thread.setUncaughtExceptionHandler(handler)), then it will be notified in the event of an uncaught exception on the thread stack ... before the thread exits. (In fact, the behaviour is a bit more complicated than this; see the javadocs.)
Why Java Exception handling designed in this way.
Because the alternative is far worse ... in most cases.
Assuming that the thread has caused the run() method to terminate ('cos you didn't catch the exception), then the only thing you could do "not stop" would be have the thread infrastructure call run() again.
But the uncaught exception typically means something BAD has happened. Typical run() methods are not designed so them multiple times will do something sensible. And if the run() method just failed for some unknown reason (as far as the run() method is concerned), it is even less likely that calling it again will work.
Besides, in the few cases where it is sensible for your run() method to catch and resume from every exception, you can always code it to do that. And if you don't want to the thread to resume, you can implement an "uncaught exception handler" to do notify something else and (maybe) start higher level recovery.
The only thing that is maybe slightly questionable about the current design is that Thread termination due to an uncaught exception is often silent. But the cure for that is to implement a default uncaught exception handler that makes some noise ....

Who is calling the Java Thread interrupt() method if I'm not?

I've read and re-read Java Concurrency in Practice, I've read several threads here on the subject, I've read the IBM article Dealing with InterruptedException and yet there's something I'm simply not grasping which I think can be broken down into two questions:
If I'm never ever interrupting other threads myself, what can trigger an InterruptedException?
If I'm never ever interrupting other threads myself using interrupt() (say because I'm using other means to cancel my working threads, like poison pills and while (!cancelled) style loop [as both explained in JCIP]), what does an InterruptedException then mean? What am I supposed to do upon catching one? Shutdown my app?

The Thread interrupt mechanism is the preferred way to get a (cooperating) thread to respond a request to stop what it is doing. Any thread (including the thread itself I think) could call interrupt() on a Thread.
In practice, the normal use-cases for interrupt() involve some kind of framework or manager telling some worker thread to stop what they are doing. If the worker thread is "interrupt aware" it will notice that it has been interrupted via an exception, or by periodically checking its interrupted flag. On noticing that it has been interrupted, a well-behaved thread would abandon what it is doing and end itself.
Assuming the above use-case, your code is likely to be interrupted if it is run within a Java framework or from some worker thread. And when it is interrupted, your code should abandon what it is doing and cause itself to end by the most appropriate means. Depending on how your code was called, this might be done by returning or by throwing some appropriate exception. But it probably should not call System.exit(). (Your application does not necessarily know why it was interrupted, and it certainly does not know if there are other threads that need to be interrupted by the framework.)
On the other hand, if your code is not designed to run under the control of some framework, you could argue that the InterruptedException is an unexpected exception; i.e. a bug. In that case, you should treat the exception as you would other bugs; e.g. wrap it in an unchecked exception, and catch and log it at the same point you deal with other unexpected unchecked exceptions. (Alternatively, your application could simply ignore the interrupt and continue doing what it was doing.)
1) If I'm never ever interrupting other threads myself, what can trigger an InterruptedException?
One example is if your Runnable objects are executed using an ExecutorService and shutdownNow() is called on the service. And in theory, any 3rd-party thread pool or thread management framework could legitimately do something like this.
2) If I'm never ever interrupting other threads myself using interrupt() ... what does an InterruptedException then mean? What am I supposed to do upon catching one? Shutdown my app?
You need analyze the codebase to figure out what is making the interrupt() calls and why. Once you have figured that out, you can work out what >>your<< part of the app needs to do.
Until you know why InterruptedException is being thrown, I would advise treating it as a hard error; e.g. print a stacktrace to the log file and shut down the app. (Obviously, that's not always the right answer ... but the point is that this is "a bug", and it needs to be brought to the attention of the developer / maintainer.)
3) How do I find out who / what is calling interrupt()?
There is no good answer to this. The best I can suggest is to set a breakpoint on the Thread.interrupt() and look at the call stack.

If you decide to integrate your code with other libraries, they can call interrupt() on your code. e.g. if you decide in the future to execute your code within an ExecutorService, then that may force a shutdown via interrupt().
To put it briefly, I would consider not just where your code is running now, but in what context it may run in the future. e.g. are you going to put it in a library ? A container ? How will other people use it ? Are you going to reuse it ?

As others have pointed out, interrupting a thread (actually, interrupting a blocking call) is usually used for purposes of exiting cleanly or cancelling an ongoing activity.
However, you should not treat an InterruptedException alone as a "quit command". Instead, you should think of interrupts as a means to control the running status of threads, much in the same way as Object.notify() does. In the same way that you'd check the current state after waking up from a call to Object.wait() (you don't assume that the wakeup means your wait condition has been satisfied), after being nudged with an interrupt you should check why you were interrupted. There is usually a way to do this. For example, java.util.concurrent.FutureTask has an isCancelled() method.
Code sample:
public void run() {
....
try {
.... // Calls that may block.
} catch (InterruptedException e) {
if (!running) { // Add preferred synchronization here.
return; // Explicit flag says we should stop running.
}
// We were interrupted, but the flag says we're still running.
// It would be wrong to always exit here. The interrupt 'nudge'
// could mean something completely different. For example, it
// could be that the thread was blocking on a read from a particular
// file, and now we should read from a different file.
// Interrupt != quit (not necessarily).
}
....
}
public void stop() {
running = false; // Add preferred synchronization here.
myThread.interrupt();
}

The problem with the question is "I". "I" usually refers to a single instance of a class. I mean by that, that any particular piece of low-level code (class) should not rely upon the implementation of the entire system. Having said that you do have make some "architectural" decisions (like what platform to run on).
Possible unexpected interrupts coming from the JRE are canceled tasks in java.util.concurrent and shutting down applets.
Handling of thread interrupts is usually written incorrectly. Therefore, I suggest the architectural decision to avoid causing interrupts where possible. However, code handling interrupts should always be written correctly. Can't take interrupts out of the platform now.

You could learn this by creating your own thread class (extending java.lang.Thread) and overriding interrupt() method, in which you record the stacktrace into, say, a String field, and then transfer to super.interrupt().
public class MyThread extends Thread {
public volatile String interruptStacktrace; // Temporary field for debugging purpose.
#Override
public void interrupt() {
interruptStacktrace = dumpStack(); // You implement it somehow...
super.interrupt();
}
}

As already mentioned, another library can interrupt your threads. Even if the library doesn't have explicit access to the threads from your code, they can still get the list of threads that are running and interrupt them that way with the following method.

I think I understand why you are a bit confused about interruption. Please consider my answers in line:
If I'm never ever interrupting other threads myself, what can trigger an InterruptedException?
Firstly you may interrupt other threads; I know that in JCiP it is mentioned that you should never interrupt threads you do not own; however, this statement has to be properly understood. What it means is that your code which might be running in any arbitrary thread should not handle interruption because since it is not the owner of the thread it has no clue of its interruption policy. So you may request interruption on other threads, but let its owner take the course of interruption action; it has the interruption policy encapsulated within it, not your task code; at least be courteous to set the interruption flag!
There are many ways why there could be interruptions still, may be timeouts, JVM interrupts etc.
If I'm never ever interrupting other threads myself using interrupt() (say because I'm using other means to cancel my working threads, like poison pills and while (!cancelled) style loop [as both explained in JCIP]), what does an InterruptedException then mean? What am I supposed to do upon catching one? Shutdown my app?
You need to be very careful here; if you own the thread which threw InterruptedException (IE), then you know what to do upon catching it, say you may shutdown your app/service or you may replace this killed thread with a new one! However, if you do not own the thread then upon catching IE either rethrow it higher up the call stack or after doing something (may be logging), reset the interrupted status so that the code which owns this thread, when control reaches it, may learn that the thread was interrupted and hence take actions as it will since only it knows the interruption policy.
Hope this helped.

The InterruptedException says that a routine may be interrupted, but not necessarily that it will be.
If you don't expect the interrupt then you should treat it as you might any other unexpected exception. If it's in a critical section where an unexpected exception could have heinous consequences, it might be best to try and clean up resources and gracefully shutdown (because getting the interrupt signals that your well-engineered application that doesn't rely on interrupts is being used in a way it wasn't designed, and so there must be something wrong). Alternatively, if the code in question is something non-critical or trivial, you might want to ignore (or log) the interrupt and keep going.