I have a long running thread that does house keeping tasks. The failing of the thread is hard to detect given the system will not be immediately affected. I want to make sure this thread never dies. My understand is as long as the code catches everything (Throwable), it will never die. Here is the sample code:
while (true) {
try {
// house keeping logic
} catch (Throwable t) {
// do not do anything
}
}
Is my understanding correct? Is there a better way hosting long running background tasks? I know ScheduledExecutorService can schedule tasks periodically but if the task needs to keep checking some resource, it is probably more efficient to just put everything in a while loop.
You can't have guaranteed long-running thread. However, you can have re-spawning logic if something goes wrong with your thread.
What you can do is have a watcher that will check if the thread is running every x mins.
Please note, the watcher will fail if JVM is shut down or crashed. If you want to re-start the thread even if JVM shuts down, you need to have external monitoring. For this, you can use tools such as supervisord.
public class LongRunningThread extends Thread {
private volatile boolean IS_RUNNING = true;
#Override
public void run() {
while (IS_RUNNING) {
try {
// actions
this.houseKeep();
} catch (Exception e) {
}
}
}
private void houseKeep() {
// housekeeping logic here
}
}
class ThreadWatcher {
private Thread threadToBeWatched;
public ThreadWatcher(Thread threadToBeWatched) {
this.threadToBeWatched = threadToBeWatched;
}
#Scheduled(cron = "*/1 * * * *")
public void checkStatus() {
if (!threadToBeWatched.isAlive()) {
// Logic to restart the thread.
}
}
}
If the housekeeping logic includes some tasks that could be classified as periodic tasks ( e.g check the connectivity to some service once per 20 seconds ) - then you'd be better off with TimerTask
Some example could be of help.
If something (anything short of OOM) happens in the TimerTask - it won't affect the timer that is responsible for task scheduling.
A few points:
I would strongly recommend using ScheduledExecutorService or some other equivalent feature that may be provided by any tools or framework you are using (e.g. #Scheduled in Spring). There are lots of considerations in these situations that are not immediately obvious, and a well established and maintained library will deal with these.
Second, it is uncommon to poll for changes in this manner without some kind of delay. A typical scenario might require you to, for example, close connections with machines that haven't responded in 1 minute. For this sort of loop, you don't need to be constantly checking, as the above loop will do. At the very least, you should include a call to Thread.sleep() to prevent excessive resource use. (This is another issue that ScheduledExecutorService deals with nicely).
Finally, you should include some way to gracefully exit the loop when your application terminates. A simple boolean running = true; on the class would suffice, and you then change your loop to: while (running) {...}. When you want to exit, simply set running = false; and your loop will exit.
What is the best way for a worker thread to signal that a graceful shutdown should be initiated?
I have a fixed size thread pool which works through a continuous set of tasks, each lasting no more than a few seconds. During normal operation this works well and chugs along with its workload.
The problem I am having is when an exception is thrown in one of the threads. If this happens I would like to bring the whole thing down and have been unable to get this working correctly.
Current approach
The naive approach that I have been using is to have a static method in the "Supervisor" class which shuts down the thread pool using the standard shutdown() and awaitTermination() approach. This is then called by any of the "Worker" classes if they encounter a problem. This was done rather than propagating the exception because execute() requires a Runnable and the run() method cannot throw exceptions.
Here is some pseudo code:
// Finds work to do and passes them on to workers
class Supervisor {
ThreadPoolExecutor exec;
static main() {
exec = new FixedThreadPool(...);
forever {
exec.execute(new Worker(next available task));
}
}
static stopThreadPool() {
exec.shutdown();
if(!exec.awaitTermination(timeout_value)) {
print "Timed out waiting on terminate"
}
}
}
class Worker {
run() {
try {
// Work goes here
} catch () {
Supervisor.stopThreadPool()
}
}
}
The effect that I am seeing is that the threads do pause for a while but then I see the timeout message and they all resume their processing. This pattern continues until I manually shut it down. If I put a call to stopThreadPool() in the main method after having broken out of the loop, the shutdown happens correctly as expected.
The approach is clearly wrong because it doesn't work, but it also feels like the design is not right.
To reiterate the question: What is the best way for a worker thread to signal that a graceful shutdown should be initiated?
Additional information
The questions I have looked at on SO have been of two types:
"How do I kill a thread in a thread pool?"
"How do I know all my threads are finished?"
That's not what I'm after. They also seem to exclusively talk about a finite set of tasks whereas I am dealing with a continuous feed.
I have read about an alternative approach using exec.submit() and Futures which puts the onus on the supervisor class to check that everything's ok but I don't know enough about it to know if it's a better design. The exception case is, well ... exceptional and so I wouldn't want to add work/complexity to the normal case unnecessarily.
(Minor side note: This is a work project and there are other people involved. I'm saying "I" in the question for simplicity.)
You are not that far from the correct solution, the problem is that you need to handle the interruption caused by the shutdown call properly. So your thread's run method should look like this:
run () {
try {
while (Thread.interrupted() == false) {
doSomeWork();
}
} catch (Exception e) {
myExecutor.shutdown();
}
}
Note that I explicitly used the shutdown() without awaitTermination() because otherwise the waiting thread is the one that keeps the Executor from properly terminating, because one thread is still waiting. Perfect single-thread deadlock. ;)
The check for interruption is by the way the hint on how to kill a thread gracefully: get the run method to end by either setting a running boolean to false or by interrupting, the thread will die a moment later.
To check if all of your threads have terminated (= are just about to end their run method), you can use a CountDownLatch for a simple case or the CyclicBarrier/Phaser class for more complex cases.
There are 2 problems here:
If you intend to just force a shutdown on any exception in a worker, then you do you use shutdown() and await counterparts. Just force it using shutdownNow and you should be good. shutdown does a graceful shutdown.
try to break your for loop when such a thing happens. The best way to do it is have a try catch in your for loop around the execute call. when an exception happens in a worker throw an unchecked exception and catch it in the for loop. Terminate the for loop and call your method to force shutdown on executor. This is a cleaner approach. Alternately you can also consider using consider a handler in your executor for doing this.
Good day!
I got the problem about synchronizing threads in java. I am developing program which creates timers and allows to reset it, delete and stop. Just to learn how to using threads.
The problem is that code gives synchronizing only for some time... I can't understand my mistake. Maybe my way is wrong so i would like to know how to solve this issue.
I have next code:
public class StopWatch
{
//Create and start our timer
public synchronized void startClock( final int id )
{
//Creating new thread.
thisThread = new Thread()
{
#Override
public void run()
{
try
{
while( true )
{
System.out.printf( "Thread [%d] = %d\n", id, timerTime );
timerTime += DELAY; //Count 100 ms
Thread.sleep( DELAY );
}
}
catch( InterruptedException ex )
{
ex.printStackTrace();
}
}
};
thisThread.start();
}
…
//Starting value of timer
private long timerTime = 0;
//Number of ms to add and sleep
private static final int DELAY = 100;
private Thread thisThread;
}
I call this Class like:
StopWatch s = new StopWatch(1);
s.startClock();
StopWatch s2 = new StopWatch(2);
s2.startClock();
I think you may have misunderstood "synchronized".
It does not mean that the threads run in exactly synchronized time - rather that only one thread at a time is allowed to be executing the synchronized code block. In your case "synchronized" makes no difference, since you are calling the startClock method from the same thread....
In general, it is impossible in Java (and indeed most high level languages) to guarantee that two threads perform actions at exactly the same clock time even if you have multiple cores, since they are always vulnerable to being delayed by the OS scheduler or JVM garbage collection pauses etc.
Also, Thread.sleep(...) is unreliable as a timing mechanism, as the amount it sleeps for is only approximate. You're at the mercy of the thread scheduler.
Suggested solution:
use System.currentTimeMillis() if you want a thread-independent timing mechansim.
What do you mean it "only gives you synchronizing for some time?" The only thing you have synchronized here is the startClock method, which just means that two threads will not be within that method at the same time (and it doesn't look like you are doing that anyway). If you wanted to synchronize access to timerTime for example, you would need to put a synchronized block inside thread run method around the incrementing timerTime (or you could use an AtomicLong).
You should probably re-read the documentation for the "synchronize" keyword. I'm pretty sure in this case all it would do is keep the two calls of StartClock() from executing at the same time, which wouldn't happen given this code because they're called one after the other from one thread. Once the timer thread begins, there's nothing keeping them synchronized, if that's your goal.
Your first problem is that this is a time based only solution. This is bad because the program has no control over how long it takes to execute. Some operations take more time than others and each thread within your process doesn't execute at the same time. In general this won't synchronize anything unless you can guarantee everything else is the same . . .
Read about http://download.oracle.com/javase/6/docs/api/java/util/concurrent/Semaphore.html and you can also do
Thread.join(); to make the main loop wait for the execution of the child thread to finish before continuing execution.
I think you misunderstood what synchronized means. Synchronized is to ensure that multiple threads have limited access to a certain block of code so that you don't get conflicts between the two threads.
I think what you may be more interested in is a CyclicBarrier or a CountDownLatch. Both can be used to "synchronize" (overloaded use in this case) multiple threads so that they try to start doing things at the same time.
However, be aware that it's impossible to have multiple threads do things at exactly the same instant. You can only try to encourage to do them at about the same time. The rest is subject to OS scheduling on the cores in the system. And if you have a single core, they will never run at the same time.
I have a complex function (optimisation) that can potentially enter in a loop or just to take too much time, and the time allowed is set by the user.
Therefore I am trying to make to run the function in a separate thread, and to stop it if the maximum time is passed. I use a code similar to the one below, but it doesn't work, so
int timeMax = 2; //time in minutes
Thread Thread_Object = new Thread_Class(... args...);
try {
Thread_Object.start();
Thread_Object.join(timeMax*60*1000);
}
I think that I'm not using the function "join" properly, or it doesn't do what I have understood. Any idea?
Thanks!
Thanks for the answers, currently I have found a better idea here*. It works but it still uses the function "stop" that is deprecated. The new code is:
Thread Thread_Object = new Thread_Class(... args...);
try {
int timeMax = 1;
Thread_Object.start();
Thread.currentThread().sleep( timeMax * 1000 );
if ( Thread_Object.isAlive() ) {
Thread_Object.stop();
Thread_Object.join();
}
}
catch (InterruptedException e) {
}
not yet sure of the function of "join", I'll have to go to have a look at some book.
java scripting API - how to stop the evaluation
I suggest you use a Timer.
The join method will wait the current thread until the thread that is being joined on finishes. The join with milliseconds passed in as a parameter will wait for some amount of time, if the time elapses notify the waiting thread and return.
What you can do, is after the join completes interrupt the thread you joined on. Of course this requires your thread to be responsive to thread interruption.
Thread.join(milis) does not kill the thread. It just waits for the thread to end.
Java threading is cooperative: you can not stop or gracefully kill a thread without it's cooperation. One way to do it is to have an atomic flag (boolean field) that thread is checking and exiting if set.
Watchdog-Timers in Java are not a simple thing, since threading is cooperative. I remember that in one project we just used Thread.stop() although it is deprecated, but there was no elegant solution. We didn't face any issues using it, though.
A good example for a Java Watchdog implementation:
http://everything2.com/user/Pyrogenic/writeups/Watchdog+timer
This might be useful
http://tempus-fugit.googlecode.com/svn/site/documentation/concurrency.html#Scheduled_Interruption
exampl:
new Thread(new Runnable() {
public void run() {
while(condition) {
*code that must not be interrupted*
*some more code*
}
}
}).start();
SomeOtherThread.start();
YetAntherThread.start();
How can you ensure that code that must not be interrupted won't be interrupted?
You can't - at least not with normal Java, running on a normal, non-real-time operating system. Even if other threads don't interrupt yours, other processes might well do so. Basically you won't be able to guarantee that you get a CPU all to yourself until you're done. If you want this sort of guarantee you should use something like Java Real-Time System. I don't know enough about it to know whether that would definitely provide the facility you want though.
The best thing to do is avoid that requirement in the first place.
Assuming you're only concerned with application-level thread contention, and assuming you are willing to fuss with locks as suggested by others (which, IMHO, is a really bad idea), then you should use a ReadWriteLock and not simple object synchronization:
import java.java.util.concurrent.locks.*;
// create a fair read/write lock
final ReadWriteLock rwLock = new ReentrantReadWriteLock(true);
// the main thread grabs the write lock to exclude other threads
final Lock writeLock = rwLock.writeLock();
// All other threads hold the read lock whenever they do
// *anything* to make sure the writer is exclusive when
// it is running. NOTE: the other threads must also
// occasionally *drop* the lock so the writer has a chance
// to run!
final Lock readLock = rwLock.readLock();
new Thread(new Runnable() {
public void run() {
while(condition) {
writeLock.lock();
try {
*code that must not be interrupted*
} finally {
writeLock.unlock();
}
*some more code*
}
}
}).start();
new SomeOtherThread(readLock).start();
new YetAntherThread(readLock).start();
Actually, you can do this if you control the thread instance you are running on. Obviously, there are a ton of caveats on this (like hanging io operations), but essentially you can subclass Thread and override the interrupt() method. you can then put some sort of boolean in place such that when you flip a flag, interrupt() calls on your thread are either ignored or better yet stored for later.
You really need to leave more info.
You cannot stop other system processes from executing unless you run on a real-time OS. Is that what you mean?
You cannot stop garbage collection, etc unless you run a real-time java. Is that what you wanted?
The only thing left is: If you simply want all YOUR other java threads to not interrupt each other because they all tend to access some resource willy-nilly without control, you are doing it wrong. Design it correctly so that objects/data that NEED to be accessed in a synchronized manner are synchronized then don't worry about other threads interrupting you because your synchronized objects are safe.
Did I miss any possible cases?
Using the synchronized approach ( in the various forms posted here ) doesn't help at all.
That approach only helps to make sure that one thread executes the critical section at a time, but this is not what you want. You need to to prevent the thread from being interrupted.
The read/write lock seems to help, but makes no difference since no other thread is attempting to use the write lock.
It only makes the application a little slower because the JVM has to perform extra validations to execute the synchronized section ( used only by one thread , thus a waste of CPU )
Actually in the way you have it, the thread is not "really" being interrupted. But it seems like it does, because it has to yield CPU time to other threads. The way threads works is; the CPU gives to each thread a chance to run for a little while for very shorts periods of time. Even one when a single thread running, that thread is yielding CPU time with other threads of other applications ( Assuming a single processor machine to keep the discussion simple ).
That's probably the reason it seems to you like the thread is being paused/interrupted from time to time, because the system is letting each thread in the app run for a little while.
So, what can you do?
To increase the perception of no interruptions, one thing you can do is assign a higher priority to your thread and decrease it for the rest.
If all the threads have the same priority one possible schedule of threads 1,2,3 could be like this:
evenly distributed
1,2,3,1,2,3,1,2,3,1,2,3,1,2,3,1,2,3
While setting max for 1, and min for 2,3 it could be like this:
More cpu to thread 1
1,1,1,2,1,1,3,1,1,1,2,1,1,1,3,1,2,1,1,1
For a thread to be interrupted by another thread, it has to be in an interruptable state, achieved by calling, Object.wait, Thread.join, or Thread.sleep
Below some amusing code to experiment.
Code 1: Test how to change the priority of the threads. See the patterns on the ouput.
public class Test {
public static void main( String [] args ) throws InterruptedException {
Thread one = new Thread(){
public void run(){
while ( true ) {
System.out.println("eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee");
}
}
};
Thread two = new Thread(){
public void run(){
while ( true ) {
System.out.println(".............................................");
}
}
};
Thread three = new Thread(){
public void run(){
while ( true ) {
System.out.println("------------------------------------------");
}
}
};
// Try uncommenting this one by one and see the difference.
//one.setPriority( Thread.MAX_PRIORITY );
//two.setPriority( Thread.MIN_PRIORITY );
//three.setPriority( Thread.MIN_PRIORITY );
one.start();
two.start();
three.start();
// The code below makes no difference
// because "one" is not interruptable
Thread.sleep( 10000 ); // This is the "main" thread, letting the others thread run for aprox 10 secs.
one.interrupt(); // Nice try though.
}
}
Code 2. Sample of how can be a thread actually be interrupted ( while sleeping in this case )
public class X{
public static void main( String [] args ) throws InterruptedException {
Thread a = new Thread(){
public void run(){
int i = 1 ;
while ( true ){
if ( i++ % 100 == 0 ) try {
System.out.println("Sleeping...");
Thread.sleep(500);
} catch ( InterruptedException ie ) {
System.out.println( "I was interrpted from my sleep. We all shall die!! " );
System.exit(0);
}
System.out.print("E,");
}
}
};
a.start();
Thread.sleep( 3000 ); // Main thread letting run "a" for 3 secs.
a.interrupt(); // It will succeed only if the thread is in an interruptable state
}
}
Before a thread is interrupted, security manager's checkAccess() method is called.
Implement your own security manager, call System.setSecurityManager to install it and make sure it doesn't let any other thread interrupt you while it is in critical section.
Error processing is an example of a use case where it is very useful to stop threads from being interrupted. Say you have a large multi-threaded server and some external condition arises that causes errors to be detected on multiple worker threads simultaneously. Each worker thread generates a notification that an error occurred. Let's say further the desired response is to bring the server to a safe state that will allow it to restart after the error condition is cleared.
One way to implement this behavior is to have a state machine for the server that processes state changes in total order. Once an error notification arrives, you put it into the state machine and let the state machine process it in toto without interruption. This is where you want to avoid interruptions--you want the first notification to cause the error handler to run. Further notifications should not interrupt or restart it. This sounds easy but really isn't--suppose the state machine was putting the server online. You would want to interrupt that to let error processing run instead. So some things are interruptible but others are not.
If you interrupt the error processing thread it may blow the error handler out of the water during synchronized method processing, leaving objects in a potentially dirty state. This is the crux of the problem--thread interrupts go around the normal synchronization mechanism in Java.
This situation is rare in normal applications. However, when it does arise the result can be byzantine failures that are very difficult to anticipate let alone cure. The answer is to protect such critical sections from interrupts.
Java does not as far as I can tell give you a mechanism to stop a thread from being interrupted. Even if it did, you probably would not want to use it because the interrupt could easily occur in low-level libraries (e.g., TCP/IP socket processing) where the effect of turning off interrupts can be very unpredictable.
Instead, it seems as if the best way to handle this is to design your application in such a way that such interrupts do not occur. I am the author of a small state machine package called Tungsten FSM (https://code.google.com/p/tungsten-fsm). FSM implements a simple finite-state machine that ensures events are processed in total order. I'm currently working on a bug fix that addresses exactly the problem described here. FSM will offer one way to address this problem but there are many others. I suspect most of them involve some sort of state machine and/or event queue.
If you take the approach of preventing interruptions it of course creates another problem if non-interruptible threads become blocked for some reason. At that point you are simply stuck and have to restart the process. It does not seem all that different from a deadlock between Java threads, which is in fact one way non-interruptible threads can become blocked. There's really no free lunch on these types of issues in Java.
I have spent a lot of time looking at problems like this--they are very difficult to diagnose let alone solve. Java does not really handle this kind of concurrency problem very well at all. It would be great to hear about better approaches.
Just start your own sub-thread, and make sure that the interrupt calls never filter through to it.
new Thread(new Runnable() {
public void run() {
Thread t = new Thread() {
public void run() {
*code that must not be interrupted*
}
}
t.start(); //Nothing else holds a reference to t, so nothing call call interrupt() on it, except for your own code inside t, or malicious code that gets a list of every live thread and interrupts it.
while( t.isAlive() ) {
try {
t.join();
} catch( InterruptedException e ) {
//Nope, I'm busy.
}
}
*some more code*
}
}
}).start();
SomeOtherThread.start();
YetAntherThread.start();
I think you need to lock on an interrupt flag. What about something like this (not tested):
new Thread() {
boolean[] allowInterrupts = { true };
#Override
public void run() {
while(condition) {
allowInterrupts[0] = false;
*code that must not be interrupted*
allowInterrupts[0] = true;
*some more code*
}
}
#Override
public void interrupt() {
synchronized (allowInterrupts) {
if (allowInterrupts[0]) {
super.interrupt();
}
}
}
}.start();
SomeOtherThread.start();
YetAntherThread.start();
Best halfway solution would be to synchronize all threads on some common object so that no other threads are runnable while you're in the critical section.
Other than that I do not think it's possible. And I'm quite curious as to what kind of problem that requires this type of solution ?
A usual program does not randomly interrupt threads. So if you start a new Thread and you are not passing the reference to this Thread around, you can be quite sure that nothing will interrupt that Thread.
Keep the reference to the Thread private is sufficient in most scenarios. Everything else would be hacky.
Typically work queues like ExecutorService will interrupt their Thread's when asked to do so. In these cases you want to deal with interrupts.