I created a webapplication that needs to do some cleanup on shutdown. This cleanup will take about a minute and its completely OK for it to do so.
When I deploy my webapp onto Tomcat 8 and then stop it, my ContextListener gets called and the cleanup begins. But it seems like Tomcat stops my thread the hard way and it won't complete anymore. At least on Tomcat 6 that wasn't an issue.
An ideas how to configure Tomcat 8 to stop from misbehaving?
Partial Answer:
I found out it has something to do with a performance optimization I did. I used startStopThreads="2" to start my applications in parallel, which works out well, but on shutdown this also seems to kill my threads.
If you have a task which is to be performed on shutdown, I would add this as shutdown hook. Most likely Tomcat 8 is called System.exit() which is a normal thing to do and this kills all user threads but start shutdown hooks.
A better solution is to never leave the system in a state where you really need this. i.e. you cannot assume an application will die gracefully.
if you are waiting for client to disconnect, I suggest you add a shutting down phase. During this phase you refuse new connections, move connections to another server or attempt to gracefully tell existing ones you are going away. After a short period or time out, you then shut down.
Related
I am following up on:
https://cloud.google.com/blog/topics/developers-practitioners/graceful-shutdowns-cloud-run-deep-dive
How to process SIGTERM signal gracefully in Java?
I have a CloudRun service which is doing some while cycles which currently seem not to end after the CloudRun revision is replaced. I guess the CloudRun manager is waiting for the revision to gracefully end, which unfortunately does not happen.
I tried adding:
Runtime.getRuntime().addShutdownHook()
And end the loops with this listener. Unfortunately there are two issues:
It is not possible to subscribe more shutdownHooks (each for one running loop) - getting errors of Hook already running. This is an implementation issue and I wonder if there is a different way to do this.
When I send the SIGTERM or SIGINT to locally running service, it ends immediately and the shutdownHook is never called. This is a logical issue and I am not sure if this is the way CloudRun ends the revisions - it seems not, otherwise the loops would be immediately ended like on my localhost.
We need to implement a graceful shutdown mechanism into our Servlet application.
EDIT: We want to make it simple as possible, which would be handling a kill signal sent via operating system's function. This would allow system admins to use built in shell utilities (kill or taskkill on Windows), otherwise they would have to install another utility just to "talk" with server.
This mechanism works in two phases:
upon shutdown request, deny certain critical activities
block until previously initiated critical actions are completed; these may take several hours
Phase #1 is implemented in our DAO layer.
Phase #2 is implemented in our ServletContextListener#contextDestroyed method
Our problem is that once contextDestroyed is called the Servlet container stops servicing further HTTP requests.
EDIT: contextDestroyed is called when someone is calling the operating system's kill function on server's process.
We would like to let the application alive during Phase #2, notifying the users that some activities are unavailable.
Use a filter to keep a list of all critical requests.
When the "prepare shutdown" request is received, the filter should start denying some requests.
Write a servlet that tells you how many critical jobs are still left in the queue.
In the shutdown tool, send the "prepare shutdown". The poll the servlet for the number of critical jobs. When this reaches 0, send the actual shutdown command.
To make this happen, create a service in the business layer which orchestrates this. Note that everything must happen before contextDestroyed() is being called! Your special application shutdown doesn't fit into the J2EE view of the world, so you have to manage it yourself.
The service should be able to tell interested parties when a shutdown is in progress, how many critical jobs are still running, etc. Servlets and filters can then use this service to deny requests or tell how many jobs are left.
When all jobs are done, deny all requests except access to the "shutdown info" servlet which should then tell that the app is now ready for death.
Write a tool which gives the administrators a nice UI to initiate shutdown of your app.
[EDIT] You may feel tempted to prevent the OS from shutting down your application. Don't do that.
What you should do is write a special tool to shut down your application using the two phase process that I described above. This should be the standard way to shutdown.
Yes, administrators will complain about it. On Unix, you can hide this tool by putting it into the init script, so no one will notice. There might be a similar solution on Windows.
Killing the server should always be possible to be able to stop it in case of (un)expected circumstances like: bugs in your shutdown code, emergency shutdown during power failure, bugs in your application code, or when Murphy happens.
My situation is like this:
Everytime before uploading war file to web-app folder, I stop Tomcat by calling sh shutdown.sh. It used to take about 30 seconds for a total shutdown. But now it doesn't work well anymore.
Actually, it did some work, because when I access the application from web-page it throws 503 error (Under Maintenance). But when I use ps aux | grep tomcat to check, the tomcat process is still there. And it will be there for around 5 - 10 mins.
I understand that it may need to take extra times to complete all the tasks, but it is way too slow (5 - 10 minutes), before it is stop totally. I don't understand why this happens, but there must be some reason. Maybe there's something to do with the code, or the new script of deployment we used recently. I just have almost no clue about where to check.
This is important to our team because we are using "auto-deployment", in which we use a script to auto-package war file, uploading and deploy on a specific time. If we started a new tomcat instance before the old one successfully shutdown, it will hang there for eternal, and cleaning up task by "kill -9" is daunting.
Is there anyone who has experimented this issue? Any clue would be appreciated.
Hoàng Long -
Thank you for the update.
1) The fact that you see your Quartz jobs running, and the error message, are both significant:
SEVERE: The web application [/project] appears to have started a
thread named [Resource Destroyer in BasicResourcePool.close()] but has
failed to stop it. This is very likely to create a memory leak.
2) One suggestion is configuration:
http://forum.springsource.org/showthread.php?17833-Spring-Quartz-Tomcat-no-shutdown
I had the same problem. I fixed it by adding
destroy-method="destroy" to the SchedulerFactoryBean definition.
This way spring closes down the scheduler when the application is
stopped.
3) Another suggestion is to add a shutdown listener:
http://forums.terracotta.org/forums/posts/list/15/4341.page
Using a context listener and introducing a timeout on shutdown solves
the issue for me. I just wait a second after shutting down:
public void contextDestroyed(ServletContextEvent sce) {
try {
factory.getScheduler().shutdown();
Thread.sleep(1000);
If this is something that mystically started to happen within the last few days, perhaps you're running into the Linux leap second bug? For more information, see
https://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-linux-server-crashes-during-a-leap-second
https://access.redhat.com/knowledge/articles/15145
http://pedroalves-bi.blogspot.fi/2012/07/java-leap-second-bug-how-to-fix-your.html
I am starting a java thread pool ThreadPoolExecutor (with a core pool size) inside a Weblogic server (WLS). To normally shutdown the pool i need to call shutdown() on this pool. But for reasons beyond my control I am not able to add any Application life cycle listener in this WLS , which means I dont have any shutdown hook to call this method.
So I am thinking to make the threads in pool as daemon threads , so that WLS JVM will not wait for these threads to finish when it is shutting down. Else JVM will not normally shutdown as core threads in pool are always running.
Is there any disadvantage of making threads in thread pool as daemon. I understand about daemon threads, when JVM exits, these threads will never be normally cleaned up and any thing running will not finish normally. This issue is only during shutdown stage.
Is there any other issue in making threads daemon in a thread pool ? (or a better way)
Given that you are in an application container, you may want to use Work Managers as #mprabhat suggests. But if you truly need threads that are under your control, you can still shut them down. WebLogic implements it's Application life cycle notification via shutdown hooks. And that mechanism is still available to you. You can use Runtime.addShutdownHook to register one. Don't forget that to do your work quickly. WebLogic has a timeout, so the shutdown sequence is basically:
Shutdown requested (either by System.exit() or a signal sent to the JVM process).
Your shutdown hooks and WebLogic's shutdown hooks execute.
After some timeout (5-10 sec I think) one of WebLogic's shutdown hooks calls System.halt().
The JVM terminates regardless of whether shutdown hooks have completed.
Please refer below link and I suggested Use java 7 thread concurrently
http://www.baptiste-wicht.com/2010/09/java-concurrency-part-7-executors-and-thread-pools/
enjoy
Searching memcached java in google, the first result is Using Memcached with Java.
The guy (who calls himself Just some Random Asshole in the Internet!) proposes a Singleton based on net.spy.memcached. It basically creates 20 threads and connections by creating 20 instances of MemcachedClient. For every request it chooses one at random.
However those threads and connections are never closed and they pile up every time I hot swap the application during development (with warnings from Tomcat 7).
SEVERE: The web application [/MyAppName] appears to have started a thread named
[...] but has failed to stop it. This is very likely to create a memory leak.
By looking at MemcachedClient JavaDoc, I see a method called shutdown with the only description being "Shut down immediately." Shut down what? The client? The server? I suppose is the client, since it's in MemcachedClient and I suppose that this method would close the connection and terminate the thread. EDIT: yes, it shuts down the client.
Question 1 How to force the execution of cleanup code in Tomcat 7, before the application is hot swapped?
Question 2 Is this approach of using memcached (with cleanup code), correct or is better I start over in a different way?
I think creating 20 memcache clients is silly - that's like creating 20 separate copies of your DB connection pool. The idea with that client is that it multiplexes a variety of requests with asynch IO.
http://code.google.com/p/spymemcached/wiki/Optimizations
As far as shutting it down, simply call:
yourClient.shutdown() to shutdown immediately, or
yourClient.shutdown(3, TimeUnit.SECONDS) for example, to allow some time for a more graceful shutdown.
That could be called from your Servlet's .destroy method, or a context listener for your whole WAR.
I don't know anything about memcached, but you could probably write a custom context listener and put some kind of shutdown hook in the context listener so that when the context shutdown you could loop through the items in your singleton and shut them down.
It turned out that it was a bug of Java AWS SDK and was not related to memcached. Version 1.2.2 of Java AWS SDK has this bug fixed.