I have a rather complicated shutdown in Java - there's a lot of clean up work to do. In particular I'm trying to figure out how to error handle from a shutdown hook thread. My code includes this currently:
try {
return shutdownPromise = doShutdown();
}
catch (Throwable exc) {
logger.error("An exception was thrown while shutting down the application.", exc);
System.exit(1);
return null;
}
When I originally wrote this I essentially thought, an error in shutdown should just go straight to exit. But exit is not so low level; it calls the shutdown hooks.
So I thought -
What does calling exit from a shutdown hook do?
What is the right way to error handle from a shutdown hook?
Firstly, the easy answer: your process deadlocks.
System.exit winds up calling Shutdown.exit, which ends with this code block:
synchronized (Shutdown.class) {
/* Synchronize on the class object, causing any other thread
* that attempts to initiate shutdown to stall indefinitely
*/
sequence();
halt(status);
}
The comment is completely accurate. But to understand this we need to see exactly what the the shutdown hook does.
Shutdown hooks are added via ApplicationShutdownHooks.add, as explained in this answer here. Note that the sequence method above eventually winds up on this line of code in Shutdown.runHooks:
if (hook != null) hook.run();
Firstly, note these hooks are not your application shutdown hooks. They are system shutdown hooks, which are different, and one of them is responsible for ensuring your application shutdown hooks run. Notice that is run, not start. In other words, it blocks. One of these shutdown hooks will be a system hook that concurrently starts all the application shutdown hooks (you can trace this code for yourself or in linked answer), then blocks until they all complete using a Thread.join call. One of these will be your application shutdown hook - which, if it calls System.exit, will be waiting to get the lock on Shutdown.class forever.
Next, the dumb answer: call Runtime.getRuntime().halt() instead. I say "the dumb answer" because this is probably the slightly lower level thing you were looking to do instead of exit. If System.exit represents a clean shutdown that failed, then halt is a hard shutdown that will always succeed immediately.
Next, the smarter answer: probably nothing. Just let your shutdown hook finish, for the following reasons. Your shutdown hook does not own the shutdown - in particular, there may be other shutdown hooks that need to do their work. Conversely, I suspect one of the responsibilities of this hook is to force a hard shutdown eventually. I would recommend creating that as a delayed task right away then letter this hook complete normally.
Related
As the ExecutorService document explains The shutdown() method will allow previously submitted tasks to execute before terminating. So in my initial understand, all submitted task will still be finally completed when we kill thread through ctrl+c. But it seemed the actually result is not.
The generation of the executorService as below:
CORE_POOL_SIZE = 1
return new ThreadPoolExecutor(CORE_POOL_SIZE, numConsumers,
KEEP_ALIVE_TIME, TimeUnit.SECONDS,
new LinkedBlockingQueue<>(),
consumerThreadFactory);
And in my task, I will log some information the task code sample like this:
log.info("before come into sleep.....")
try {
Thread.sleep(sleepTime);
} catch (InterruptedException e) {
log.warn(" exception during sleep.....{}", request.getExecutionId());
e.printStackTrace();
}
log.info("after come into sleep...")
I submit several task and then terminal thread through control+c, and the result is that only the log before come into sleep is print.
So I dont know whether my understand is correct that shutDown can ensure all submitted task be completed finally. I also google some similar question but find there are no explicit answer.
Control+c kills entire JVM
Pressing Control+c on the console terminates the entire Java process running your threads. See Wikipedia.
You are interrupting the JVM. This is not a graceful way to interact with your Java app.
So, no, you should not expect your executor service to continue running submitted tasks. You killed it.
The JVM is generally not designed to be manipulated in the console after launch. After launch, the only two ways I know of for interacting with the JVM is:
The user-interface of your app, GUI or console-based.
Java Management Extensions (JMX)
You said:
we kill thread through ctrl+c
No, ctrl+c does not specifically kill a thread. It kills your entire JVM.
awaitTermination
After calling ExecutorService#shutdown, if you want your code to block and wait until all submitted tasks are done, call ExecutorService#awaitTermination. Specify a certain amount of time to wait before throwing an exception if tasks are still running.
This waiting for termination is shown in the JavaDoc and has been discussed in many other Questions and Answers on Stack Overflow, some authored by me. So search to learn more.
I believe it is clear from the docs for ExecutorService (https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ExecutorService.html#shutdown--) that it doesn't wait for the tasks to complete. We have awaitTermination for this purpose.
This method does not wait for previously submitted tasks to complete execution. Use awaitTermination to do that.
A clear example of how to achieve this is available in Usage Example section of this documentation as well.
Why is awaitTermination important during graceful shutdown?
Using shutdown() is like informing your intention to stop all the threads managed by the ExecutorService. There is no blocking involved here. Your main thread shall resume execution and continue executing subsequent code after shutdown(). While handling graceful shutdown of a JVM (https://docs.oracle.com/javase/8/docs/technotes/guides/lang/hook-design.html), a separate thread will be called expecting it to complete its task and when it does complete, JVM shuts down. If you need to wait for the tasks to complete, block the shutdown hook thread using awaitTermination().
What happens when the JVM is terminated with System.exit(0) or ^C or anything of that kind? I read things like "the process is just blown away" and "every single thread is stopped", but I would like to know what happens exactly. I already know there is the shutdownHook that somehow still gets executed, but what happens before the shutdownHooks are invoked and does anything happen after all these threads have finished?
I would like to implement such a shutdownHook correctly and to do so I need to make the right assumptions about what might still be executed and what will not.
update:
some code:
class SomeObject {
private boolean stopped;
SomeObject() {
stopped = false;
Thread hook = new Thread() {
#Override
public void run() {
stopped = true;
}
};
hook.setPriority(Thread.MAX_PRIORITY);
Runtime.getRuntime().addShutdownHook(hook);
}
boolean map(Iterator<Object> it) {
while(it.hasNext() && !stopped) {
writeToOtherObject(it.next());
it.remove();
}
//have calculations finished?
return !it.hasNext();
}
}
The map function computes results that are gathered in some other object. This object should be stored in some file before everything is broken down (by normal-priority shutdownHooks too). Does the shutdownHook here make sense? As far I understand it, all threads are destroyed first and only then the shutdownHooks are run (concurrently, but I assume high-priority threads are run first...) and then objects are finalized. This makes the code above rather useless, because the intention of this shutdownHook would be to make sure no new loop is started when the shutdown has already started. Is my understanding correct and complete?
Let's begin from the different ways the shutdown sequence can be initiated:
The last non-daemon thread ends.
The JVM is interrupted (by using ctrlC or sending SIGINT).
The JVM is terminated (by sending SIGTERM)
One of the threads calls System.exit() or Runtime.exit().
When System.exit(int) is called, it calls Runtime.exit(). It checks with the security manager whether it is permitted to exit with the given status, and if so, calls Shutdown.exit().
If you interrupted the JVM or the system sent it the TERM signal, then by default, Shutdown.exit() is called directly without checking with the security manager.
The Shutdown class is an internal, package-private class in java.lang. It has, among others, an exit() and a halt() methods. Its exit() method does some stuff to prevent the hooks from being executed twice, and so on, but basically, what it does is
Run the System Hooks. The System Hooks are registered internally by JRE methods. They are ran sequentially, not in threads. The second system hook is what runs the application hooks that you have added. It starts each of them as a thread and then has a join for each of them at the end. Other system hooks may run before or after the application hooks.
If finalizers are supposed to be ran prior to halting, they are ran. This should generally not even happen, as the method has been deprecated. And if the exit is with a status other than zero, it ignores the runFinalizersOnExit anyway.
The JVM is halted.
Now, contrary to your supposition, it is at stage 3 that all the threads are stopped. The halt method is native, and I have not attempted to read the native code, but up to the moment it is called, the only code being ran is pure Java, and there is nothing that stops the threads anywhere in it. The documentation of Runtime.addShutdownHook says, in fact:
A shutdown hook is simply an initialized but unstarted thread. When the virtual machine begins its shutdown sequence it will start all registered shutdown hooks in some unspecified order and let them run concurrently. When all the hooks have finished it will then run all uninvoked finalizers if finalization-on-exit has been enabled. Finally, the virtual machine will halt. Note that daemon threads will continue to run during the shutdown sequence, as will non-daemon threads if shutdown was initiated by invoking the exit method.
(emphasis mine)
So you see, it is indeed part of the shutdown hook's job to tell threads that they should leave their loops and clean up.
Another misconception you have is about giving the thread a high priority. A high priority doesn't mean that the thread will run first, before all other hooks. It merely means that whenever the operating system has to make a decision which of the threads which are in "ready to run" state to give to a CPU to run, a high-priority thread will have a higher probability of "winning" - depending on the operating system's scheduling algorithm. In short, it may get a little more CPU access, but it will not - especially if you have more than one CPU core - necessarily start before other threads or complete before them.
One last thing - if you want to use a flag to tell a thread to stop working, that flag should be volatile.
Take a look at this article from DZone. It covers the JVM and its termination in sufficient detail, with a focus on the use of ShutdownHook.
From a high level, some of the important assumptions it covers include:
Shutdown Hooks may not be executed in some cases!
Once started, Shutdown Hooks can be forcibly stopped before completion.
You can have more than one Shutdown Hook, but their execution order is not guaranteed.
You cannot register / unregister Shutdown Hooks with in Shutdown Hooks
Once shutdown sequence starts, it can be stopped by Runtime.halt() only.
Using shutdown hooks require security permissions.
Exceptions thrown by the Shutdown Hooks are treated same as exceptions thrown by any other code segment.
How can I programatically force close the program upon exit.
I've added the shutdown hook which calls a System.exit(0) but it seems to have problems executing that. The javaw.exe process keeps running in memory even though the Jframe is closed and the shutdown hook was executed.
Additionally, when I manually close the batch file running the program, windows throws the Force Close message.
image: http://i.imgur.com/gy57OEV.png
Calling System.exit() in a shutdown hook is pointless. The JVM has already decided to shutdown before the hook is called. I imagine that calling System.exit() from a shutdown hook could be problematic ...
Presumably, the reason that your application is not closing is that your application has created other threads, and they are still alive.
One way to handle this is to have the JFrame close event (or the close button) trigger something that starts the JVM shutdown. Your f.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE) code is one way to do that. It would call System.exit ... and THAT would run any shutdown hooks as the JVM shuts down.
Another way is to mark the other threads as "daemon" threads, which means that they won't stop the JVM from deciding to shut down. (Normally, the JVM pulls the plug if there are no live non-daemon threads. For a single-threaded application, that means your "main" thread ... )
As a matter of interest, here's why a System.exit() call is problematic in a shutdown hook.
The javadoc says that System.exit() is equivalent to System.getRuntime().exit(), and the javadoc for the latter says this:
"If this method is invoked after the virtual machine has begun its shutdown sequence then if shutdown hooks are being run this method will block indefinitely."
So if you call System.exit() within a shutdown hook, that is sufficient to cause the hook to block indefinitely. And that will cause the shutdown sequence to stall.
System.exit() works properly in JOptionPane scenarios:
Object[] options={"YES","NO"};
int selection= JOptionPane.showOptionDialog(this, "Message input", "Title",
JOptionPane.YES_NO_OPTION, JOptionPane.WARNING_MESSAGE, null, options,
options[1]);
if(selection==JOptionPane.YES_OPTION){
reset();
}
else if(selection==JOptionPane.NO_OPTION){
System.exit(0);
}
but, as stated above f.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE); is still present in my main method
I have a call to System.exit(1) in my multi-threaded program. However from time to time instead of return code 1, the program exits with return code 0. I don't have any other calls System.exit() and I'm positive that the program doesn't exit cleanly. What can be the cause, and how can I avoid it?
Note that the error is intermittent and I cannot reproduce the same behavior in single threaded programs.
Modify your design to execute a more controlled shutdown.
There should be no expectation that calling System.exit() in an application with multiple threads would ever cause the program to exit cleanly.
Rather than calling System.exit() to leave the program, you should send shutdown messages to each moving component and use Thread.join() to recover any threads you created. Your application should be able to shut down all pieces nicely this way. The final command in the main thread should be to return your exit code. If you just call System.exit(), you're leaving all of these shut down details to the JVM, which is just going to take a heavy-handed approach and kill everything on the spot.
Have you used Runtime.getRuntime.addShutdownHook() at all? A call to System.exit() will invoke any shutdown hooks that may be installed and this could be changing the exit code.
The documentation for Runtime.halt(int) says the following about its argument:
If the exit (equivalently, System.exit) method has already been invoked then this status code will override the status code passed to that method.
So perhaps something is invoking Runtime.halt(int). In a shutdown hook or finalizer?
I think the only way this may happen if your JVM terminates before System.exit(1) actually executed. Do you think this may be possible in your system?
Either, the code with System.exit(1) is being executed in a daemon thread, and so when all other live (non-daemon) threads finish working JVM exits cleanly (or not cleanly, as you can still get 0 exit code if you program throws an exception!)
Alternatively, as #Erick Robertson suggested, maybe something is modifying the exit status from a hook or something, although I am not sure how this is possible.
Note: please disregard my previous comment. Calling System.exit(1) will terminate all currently running daemon/non-daemon threads all-together.
I have a Java program which is being started via ProcessBuilder from another Java program.
System.exit(0) is called from the child program, but for some of our users (on Windows) the java.exe process associated with the child doesn't terminate. The child program has no shutdown hooks, nor does it have a SecurityManager which might stop System.exit() from terminating the VM. I can't reproduce the problem myself on Linux or Windows Vista. So far, the only reports of the problem come from two Windows XP users and one Vista user, using two different JREs (1.6.0_15 and 1.6.0_18), but they're able to reproduce the problem every time.
Can anyone suggest reasons why the JVM would fail to terminate after System.exit(), and then only on some machines?
Edit 1: I got the user to install the JDK so we could get a thread dump from the offending VM. What the user told me is that the VM process disappears from VisualVM as soon as he clicks on the 'Quit' item in my menu---but, according to Windows Task Manager, the process hasn't terminated, and no matter how long the user waits (minutes, hours), it never terminates.
Edit 2: I have confirmed now that Process.waitFor() in the parent program never returns for at least one of the users having the problem. So, to summarize: The child VM seems to be dead (VisualVM doesn't even see it) but the parent still sees the process as live and so does Windows.
This can happen if your code (or a library you use) has a shutdown hook or a finalizer that doesn't finish cleanly.
A more vigorous (so should only be used in extreme cases!) way to force shutdown is by running:
Runtime.getRuntime().halt(0);
The parent process has one thread
dedicated to consuming each of the
child's STDOUT and STDERR (which
passes that output through to a log
file). So far as I can see, those are
working properly, since we're seeing
all the output we expect to see in the
log
i had a similar problem with my program not disappearing from task mgr when i was consuming the stdout/stderr. in my case, if I closed the stream that was listening before calling system.exit() then the javaw.exe hung around. strange, it wasn't writing to the stream...
the solution in my case was to simply flush the stream rather than close it before existing. of course, you could always flush and then redirect back to stdout and stderr before exit.
Check if there is a deadlock.
For example,
Runtime.getRuntime().addShutdownHook(new Thread(this::close));
System.exit(1);
and in the close()
public void close() {
// some code
System.exit(1);
}
The System.exit() actually calls the shutdown hook and finalizers. So, if your shutdown hook calls the exit() again internally, for some reason (for example, when the close() raises an exception for which you want to exit the program, then you will call exit() there also). Like this..
public void close() {
try {
// some code that raises exception which requires to exit the program
} catch(Exception exceptionWhichWhenOccurredShouldExitProgram) {
log.error(exceptionWhichWhenOccurredShouldExitProgram);
System.exit(1);
}
}
Though, it is a good practice to throw the exception, some may choose to log and exit.
Note, also, that Ctrl+C will also not work if there is a deadlock.
Since it also calls the shutdown hook.
Anyways, if it is the case, the problem can be solved by this workaround:
private static AtomicBoolean exitCalled=new AtomicBoolean();
private static void exit(int status) {
if(!exitCalled.get()) {
exitCalled.set(true);
System.exit(status);
}
}
Runtime.getRuntime().addShutdownHook(new Thread(MyClass::close));
exit(1);
}
private static void close() {
exit(1);
}
P.S: I feel that the above exit() version must actually be written
in the System.exit() method only (may be some PR for JDK?) Because,
there is practically no point (at least from what I see) in
entertaining a deadlock in System.exit()
Here are a couple of scenarios...
Per the definition of a Thread in http://java.sun.com/j2se/1.4.2/docs/api/java/lang/Thread.html
...
When a Java Virtual Machine starts up, there is usually a single non-daemon thread (which typically calls the method named main of some designated class). The Java Virtual Machine continues to execute threads until either of the following occurs:
1) The exit method of class Runtime has been called and the security manager has permitted the exit operation to take place.
2) All threads that are not daemon threads have died, either by returning from the call to the run method or by throwing an exception that propagates beyond the run method.
Another possibility is if the method runFinalizersOnExit has been called. as per the documentation in http://java.sun.com/j2se/1.4.2/docs/api/java/lang/System.html
Deprecated. This method is inherently unsafe. It may result in finalizers being called on live objects while other threads are concurrently manipulating those objects, resulting in erratic behavior or deadlock.
Enable or disable finalization on exit; doing so specifies that the finalizers of all objects that have finalizers that have not yet been automatically invoked are to be run before the Java runtime exits. By default, finalization on exit is disabled.
If there is a security manager, its checkExit method is first called with 0 as its argument to ensure the exit is allowed. This could result in a SecurityException.
Maybe a badly written finalizer? A shutdown hook was my first thought when I read the subject line. Speculation: would a thread that catches InterruptedException and keeps on running anyway hold up the exit process?
It seems to me that if the problem is reproducible, you should be able to attach to the JVM and get a thread list/stack trace that shows what is hung up.
Are you sure that the child is still really running and that it's not just an unreaped zombie process?
Does the parent process consumes the error- and outputstream from the child process?
If under some OS the childprocess print out some errors/warning on stdout/stderr and the parent process is not consuming the streams, the childprocess will block and not reach System.exit();
Hy i had the same problem, but the couse for me was i was using the remote Debugging(VmArgs: -Xdebug -Xrunjdwp:transport=dt_socket,address=%port%,server=y,suspend=y) when i disabled this the java.exe Process exited as expected.
I think all of the obvious causes have been provisionally covered; e.g. finalizers, shutdown hooks, not correctly draining standard output / standard error in the parent process. You now need more evidence to figure what is going on.
Suggestions:
Set up a Windows XP or Vista machine (or virtual), install the relevant JRE and your app, and try to reproduce the problem. Once you can reproduce the problem, either attach a debugger or send the relevant signal to get a thread dump to standard error.
If you cannot reproduce the problem as above, get one of your users to take a thread dump, and forward you the log file.
Another case not mentioned here is if the shutdown hooks hang on something so be careful in writing the shutdown hook code(and if a 3rd party library registered a shutdown hook that is hanging as well).
Dean