This is a follow up to my earlier question.
Tomcat 5.0.28 had a bug where the Servlet's destroy() method was not being invoked by the container on a shutdown. This is fixed in Tomcat 5.0.30, but if the Servlet's destroy() method had a System.exit(), it would result in the Tomcat windows service throwing the Error 1053 and refusing to shutdown gracefully (see above link for more details on this error)
Anybody has any idea on whether:
Calling System.exit() inside a Servlet's destroy() method to forcefully kill any non-daemon threads is a good idea?
Why does Tomcat 5.0.30 and (later versions including Tomcat 6.x.x) fail to shutdown properly if there's a System.exit() in the destroy() method of the Servlet.
Calling System.exit() inside a Servlet's destroy() method to forcefully kill any non-daemon threads is a good idea?
It is absolutely not a good idea - it is a horrible idea. The destroy() method is called when the servlet is taken out of service, which can happen for any number of reasons: the servlet/webapp has been stopped, the webapp is being undeployed, the webapp is being restarted etc.
System.exit() shuts down the entire JVM! Why would you want to forcibly shutdown the entire server simply because one servlet is being unloaded?
Why does Tomcat 5.0.30 and (later versions including Tomcat 6.x.x) fail to shutdown properly if there's a System.exit() in the destroy() method of the Servlet.
Probably to prevent such dangerous behavior like this.
You shouldn't write code that assumes that your code/application is the only thing running on the server.
You're asking two questions:
Question 1: Is calling System.exit() inside a Servlet's destroy() method to forcefully kill any non-daemon threads a good idea?
Calling System.exit() inside ANY servlet-related method is always 100% incorrect. Your code is not the only code running in the JVM - even if you are the only servlet running (the servlet container has resources it will need to cleanup when the JVM really exits.)
The correct way to handle this case is to clean up your threads in the destroy() method. This means starting them in a way that lets you gently stop them in a correct way. Here is an example (where MyThread is one of your threads, and extends ServletManagedThread):
public class MyServlet extends HttpServlet {
private List<ServletManagedThread> threads = new ArrayList<ServletManagedThread>();
// lots of irrelevant stuff left out for brevity
public void init() {
ServletManagedThread t = new MyThread();
threads.add(t);
t.start();
}
public void destroy() {
for(ServletManagedThread thread : threads) {
thread.stopExecuting();
}
}
}
public abstract class ServletManagedThread extends Thread {
private boolean keepGoing = true;
protected abstract void doSomeStuff();
protected abstract void probablySleepForABit();
protected abstract void cleanup();
public void stopExecuting() {
keepRunning = false;
}
public void run() {
while(keepGoing) {
doSomeStuff();
probablySleepForABit();
}
this.cleanup();
}
}
It's also worth noting that there are thread/concurrency libraries out there that can help with this - but if you really do have a handful of threads that are started at servlet initialization and should run until the servlet is destroyed, this is probably all you need.
Question 2: Why does Tomcat 5.0.30 and (later versions including Tomcat 6.x.x) fail to shutdown properly if there's a System.exit() in the destroy() method of the Servlet?
Without more analysis, it's hard to know for certain. Microsoft says that Error 1053 occurs when Windows asks a service to shutdown, but the request times out. That would make it seem like something happened internally to Tomcat that got it into a really bad state. I would certainly suspect that your call to System.exit() could be the culprit. Tomcat (specifically, Catalina) does register a shutdown hook with the VM (see org.apache.catalina.startup.Catalina.start(), at least in 5.0.30). That shutdown hook would get called by the JVM when you call System.exit(). The shutdown hook delegates to the running services, so each service could potentially be required to do alot of work.
If the shutdown hooks (triggered by your System.exit()) fail to execute (they deadlock or something like that,) then it is very easy to understand why the Error 1053 occurs, given the documentation of the Runtime.exit(int) method (which is called from System.exit()):
If this method is invoked after the
virtual machine has begun its shutdown
sequence then if shutdown hooks are
being run this method will block
indefinitely. If shutdown hooks have
already been run and on-exit
finalization has been enabled then
this method halts the virtual machine
with the given status code if the
status is nonzero; otherwise, it
blocks indefinitely.
This "indefinite blocking" behavior would definitely cause an Error 1053.
If you want a more complete answer than this, you can download the source and debug it yourself.
But, I would be willing to bet that if you properly handle your thread management issue (as outlined above,) your problems will go away.
In short, leave the System.exit() call to Tomcat - that's not your job.
Calling System.exit() inside a
Servlet's destroy() method to
forcefully kill any non-daemon threads
is a good idea?
Not a good idea. You will forcefully kill all threads, which might include part of Tomcat that is currently shutting down the system. This will cause Tomcat to un-gracefully shutdown. This can also prevent shutdown handlers from running which can lead to all sorts of problems.
Why does Tomcat 5.0.30 and (later
versions including Tomcat 6.x.x) fail
to shutdown properly if there's a
System.exit() in the destroy() method
of the Servlet.
A lot of code executes after a Servlet destory. The Context destroy and all of its other listeners for one... other servlets. Other applications. Tomcat itelf. By calling System.exit, you prevent all of that from running.
A better question is what are thse non-daemon threads, why are they running, and who starts them?
When writing thread shutdown code like Jared's, I normally make the "keepGoing" member and "stopExecuting()" method static so that all threads get the signal to go down with one shutdown call. Good idea or no?
Related
TL;DR: I have a Windows service written in Java, jarred, and installed with Procrun. I am starting it with W32Service.startService(). When does the service tell Windows it has started?
I'm working with windows services written in Java. I've been jarring and using Procrun to install them, and JNA to work with them (in particular, com.sun.jna.platform.win32.W32Service).
I would like to understand in the exact behavior of the W32Service object's waitForNonPendingState() method (which is the X to my Y: understand the exact behavior of startService()).
waitForNonPendingState() is actually very straightforward: it polls the status of the service until it's either in a non-pending state or a timeout occurs. How a service transitions to a non-pending state isn't so straightforward though.
Microsoft's Service State Transitions page says:
The initial state of a service is SERVICE_STOPPED. When the SCM starts the service, it sets the service state to SERVICE_START_PENDING and calls the service's ServiceMain function. The service then completes its initialization using one of the techniques described in Service ServiceMain Function. After the service completes its initialization and is ready to start receiving control requests, the service calls SetServiceStatus to report SERVICE_RUNNING...
But that doesn't really shed any light on how the service does this. The ServiceMain remarks also just say "The Service Control Manager (SCM) waits until the service reports a status of SERVICE_RUNNING."; that's pretty much as specific as I can find.
Which brings me to my question: How does a java Windows service "know" it has finished initializing?
In other words, if I have an installed service with a main() method:
public class SampleService {
public static void main(String[] args) {
if ("start".equals(args[0]))
new SampleService();
}
public SampleService() {
// do a whole bunch of stuff
}
}
and I call:
W32Service service = serviceManager.openService("SampleService",
Winsvc.SC_MANAGER_ALL_ACCESS);
service.startService();
At what point does my SampleService tell Windows it has initialized? Through experimentation, I can see that if there's a runtime exception during construction, the service's status is never SERVICE_RUNNING, so there's something in that process which sets that status. But some of my constructors wait on queues or enter spin loops and they do set the SERVICE_RUNNING status, so I can't tell where this status is set. W32Service's documentation is less than useless on this.
Your service is not really exposed as regular service it relies on Procrun as soon as the JVM is spawned as part of the process or separately the service state is set to running. In prunsrv.c which has the service container code for procrun you can check how serviceStart() is called and what happens on success.
I've a spring web app running in a Tomcat server. In this there is a piece of code,in one of the Spring beans, which waits for a database connection to become available. My scenario is that while waiting for a database connection, if the Tomcat is shutdown, I should stop waiting for a DB connection and break the loop.
private Connection prepareDBConnectionForBootStrapping() {
Connection connection = null;
while (connection == null && !Thread.currentThread().isInterrupted()) {
try {
connection = getConnection();
break;
} catch (MetadataServerException me) {
try {
if (!Thread.currentThread().isInterrupted()) {
Thread.sleep(TimeUnit.MINUTES.toMillis(1));
} else {
break;
}
} catch (InterruptedException ie) {
logger.error("Thread {} got interrupted while wating for the database to become available.",
Thread.currentThread().getName());
break;
}
}
}
return connection;
}
The above piece of code is executed by one of the Tomcat's thread and it's not getting interrupted when shutdown is invoked. I also tried to achieve my scenario by using spring-bean's destroy method, but to my surprise the destroy method was never called. My assumption is that Spring application context is not getting fully constructed - since, I've the above waiting loop in the Spring bean - and when shutdown is invoked corresponding context close is not getting invoked.
Any suggestions?
Tomcat defines a life-cycle for Web applications (well, it's a kind of common specification aspect and not just tomcat specific, but anyway...)
So there is a way to hook into this process and terminate the loop or whatever.
In spring its very easy, because if tomcat shuts down gracefully, tomcat attempts to "undeploy" the WAR before actually exiting the process. In servlet specification in order to do that, a special web listener can be defined and invoked (you can see javax.servlet.ServletContextListener API for more information)
Now spring actually implements such a listener and once invoked, it just closes the application context.
Once this happens, your #PreDestroy methods will be called automatically. This is already a spring stuff, tomcat has nothing to do with that.
So bottom line, specify #PreDestroy method on your bean that would set some flag or something and it would close the logic that attempts to close the connection.
Of course all the stated above doesn't really work if you just kill -9 <tomcat's pid> But in this case the whole jvm stops so the bean is irrelevant.
I have the following in my main method:
public static void main(String[] args) throws IOException {
Properties properties = getConfig();
Jedis jedis = configure(properties)
jedis.subscribe(queueHandler, "queue");
Runtime.getRuntime().addShutdownHook(new Thread() {
#Override
public void run() {
logger.debug("SHUTTING DOWN");
jedis.close();
}
});
}
I noticed that the code inside my shutdown hook is never run, why is that? How do I clean up resources that my main method has a hold of?
"Note that subscribe is a blocking operation ..." (from the AdvancedUsage wiki).
This in turn means that the shutdown-hook is only registered after you unsubscribe or shutdown Jedis,
and you probably only reach that part of the code when the JVM is already shutting down
(in which case the newly added shutdown hook is ignored: shutdown hooks have to be registered before the JVM shuts down).
"For more usage examples check the tests." (from the main Github page)
So let's look at the PublishSubscribeCommandsTest.
The test is just one subscribe operation but within the subscribe operation, the JedisPubSub un-subscribes itself when it receives a specific message thereby unblocking itself so that the test can finish.
Two things you can experiment with:
register the shutdown hook before calling jedis.subscribe
copy the code from the test and modify it so you can experiment with blocking and unblocking threads and shutting down Jedis from within a shutdown hook (to clean up used resources)
Regular exit of Main-Method differs a little from explicit calling System.exit(0);
so, try following:
public static void main(String[] args) throws IOException {
//all your code...
System.exit(0); //NOTE: non zero status will work different!
}
The shutdown hook will invoke if the jvm shuts down. Make sure that the program exits. If the jedis is running in the background (in a thread) even if the last statement is executed then the shutdown hook will not invoke. In such case, you have to stop that thread. The best way will be to try jedis.shutdown(); (after adding shutdown hook, here as the last statement) to shut it down.
I have a Servlet app which spawns worker threads in the init() method. These workers continually do transactional jobs i.e. it's important that each job, if started, is completed or else not started.
I hoped that I could use the destroy() method to stop these worker Threads in a graceful way, i.e. making sure they were finished with whatever last/current job before ending the destroy() method and terminating the Servlet.
The destroy() is triggered when a Chef scripts executes service tomcat7 stop
But from my logs/tests, the destroy() method on my Servlet does not block/complete. It is triggered, but then after a few moments, tomcat shuts down and I don't receive confirmation that the Threads stopped in a graceful way.
public void destroy() {
log.info("Signaling worker to stop current job/not begin another.");
this.worker.stop();
while(this.worker.isProcessingJob()) {
Thread.sleep(3 * 1000);
}
log.info("Worker successfully stopped");
log.info("destroy() complete");
super.destroy();
}
In many cases I don't see the last two logs, i.e. Tomcat goes down when my code is in while-loop waiting for this.worker to complete it's current job.
Is this expected behavior? Is there some way to tell tomcat to stop, but also allow my Servlet's destroy() method to block/complete?
I have a simple API that my clients use in a standalone application. Behind the scenes my API uses Ehcache for performance.
Everything works fine except that my client needs to invoke a shutdown() method in my API to invoke CacheManager.shutdown() without which Ehcache continues to run in the background even though the main thread is completed.
Is there a way I can avoid this extra shutdown() call for my client?
I tried using #PreDestroy Spring annotation to invoke this call, but it didn't work?
Here I am adding a sample client program.
public class ClientApp{
#Autowired
private IClientService service;
public static void main(String[] args){
try{
service.getClients();
...
} finally {
service.shutdown(); // to shutdown the cache background thread
}
}
}
In ClientServiceImpl.java, I have the following lines
public void shutdown(){
LOGGER.info("Shutting the cache down...");
ehcacheManager.shutdown();
}
Your example confirms the standalone application setup.
Ehcache should not prevent the JVM from shutting down when the main thread terminates.
If it does, you will need to add thread dumps to this issue so we can analyse further the issue and its cause.
Adding the following line does what I was looking for.
System.setProperty(CacheManager.ENABLE_SHUTDOWN_HOOK_PROPERTY, "true");