Spring Boot - Embedded Tomcat - Shutdown Hook Hangs - java

When running embedded Tomcat, my app sometimes fails to shut down via the /shutdown endpoint. A thread dump reveals that every thread is either BLOCKED, or IN_NATIVE. Below is a thread dump.
https://gist.github.com/aglassman/9d6a5e1d121af871345a72e25db6319f
Does this look typical? Should I continue investigating if there are blocked threads causing shutdown to fail, or should I look elsewhere?

Related

Gracefully shutdown JBoss application

I am looking for an approach to gracefully handle shutdown of JBoss (Wildfly AS 8.2). This would mean that all current requests are served and the webapp stops receiving further requests. I found that this is possible via command line in version 9 of the application server -
./jboss-cli.sh --controller=remoting://<host>:<port>
--connect --command=:shutdown(timeout=t)
Using this JBoss gracefully handles all requests for t seconds and gracefully shuts down (this would require an upgrade from version 8 to 9).
One possible approach would be to handle this in the Java application by maintaining a count of active requests and waiting for this number to go to 0 till a timeout and then exitting, basically replicating the above mentioned functionality.
I need to shutdown the webapp/JBoss remotely, so we are looking for a JMX (Java Management Extension) based solution. Does JBoss expose any such operation to gracefully shut down possibly via JMX or any other technology?
PS- Ctrl-C or kill commands donot shutdown JBoss gracefully.
JBoss EAP 6/7 and above allows graceful shutdown via CLI and even:
The signals SIGHUP, SIGINT and SIGTERM all trigger a graceful shutdown of the JBoss 7/6 application server. Those are signals are respectively the commands kill -1 $PID, kill -2 $PID(or control+c), and kill $PID.
https://access.redhat.com/solutions/18601
You can use the Jboss cli remotely as the URL you pass in the controller param shows. You could even use the "REST" API to execute it.
The shutdown should be available from JMX.

Tomcat warning: The executor associated with thread pool [http-apr-8080] has not fully shutdown

I am running a web application using tomcat8 and jdk1.8 on a linux server.
When I shut down tomcat I get the following warning message:
WARNING [main] org.apache.tomcat.util.net.AbstractEndpoint.shutdownExecutor The executor associated with thread pool [http-apr-8080] has not fully shutdown. Some application threads may still be running
Note: I only get this warning when I shutdown tomcat, not when I undeploy my application.
How can I "fix" this warning so that the message won't appear.
I do a thread dump using Jstack while the application is running and I see several http-apr-8080-exec threads.
Thank you
We have to Stop any application before we shutdown the tomcat because the servlets might serve the requests during shutdown process.
Reference: https://www.safaribooksonline.com/library/view/tomcat-the-definitive/0596003188/ch01s02.html
Please check the section- Restarting Tomcat
It says that
"The Java Servlet Specification also dictates that, on shutdowns, servlet containers must wait for each servlet to finish serving all requests that are in progress or wait a container-specific timeout duration before taking servlets out of service. For Tomcat 4, that timeout duration is a maximum of a half-second per servlet. When a servlet misbehaves and takes too long to finish serving requests, it's up to Tomcat to figure out that the servlet has taken too long and to forcibly take it out of service so that Tomcat can shut down. This processing takes time, though, and slows Tomcat's own shutdown processing."
Best Practice:
When you Stop the application from tomcat, the servlets are out of service. Hence you will not face this issue.
Please follow the process of stopping the application first and shutdown the tomcat server.
This error was caused as a result of shutting down tomcat while the instance was still "attached" to the Elastic Load Balancer.
The ELB has a feature called Connection Draining which, by default, is set to 300 s.
You can disable connection draining or just remove the instance from the ELB before shutting down tomcat in order to not get the warning message.

Spring Retry is not allowing to stop the TC server Normally

I am using spring retry template, when there is retry process is running, I am not able to stop the server. If I am trying to stop the server, It is saying tc server is not responding, would you like to terminate the server.
I followed the following way, but unfortunately the same result.
How can I shutdown Spring task executor/scheduler pools before all other beans in the web app are destroyed?
Thanks in Advance
You need to call shutdownNow() to interrupt any waiting (interruptible) threads.

Tomcat8 kills my threads on shutdown

I created a webapplication that needs to do some cleanup on shutdown. This cleanup will take about a minute and its completely OK for it to do so.
When I deploy my webapp onto Tomcat 8 and then stop it, my ContextListener gets called and the cleanup begins. But it seems like Tomcat stops my thread the hard way and it won't complete anymore. At least on Tomcat 6 that wasn't an issue.
An ideas how to configure Tomcat 8 to stop from misbehaving?
Partial Answer:
I found out it has something to do with a performance optimization I did. I used startStopThreads="2" to start my applications in parallel, which works out well, but on shutdown this also seems to kill my threads.
If you have a task which is to be performed on shutdown, I would add this as shutdown hook. Most likely Tomcat 8 is called System.exit() which is a normal thing to do and this kills all user threads but start shutdown hooks.
A better solution is to never leave the system in a state where you really need this. i.e. you cannot assume an application will die gracefully.
if you are waiting for client to disconnect, I suggest you add a shutting down phase. During this phase you refuse new connections, move connections to another server or attempt to gracefully tell existing ones you are going away. After a short period or time out, you then shut down.

A web application appears to have started a thread named [22] but has failed to stop it. This is very likely to create a memory leak

I have a web application with Servlets in the back end deployed over tomcat. The application is simple java application.
I see this error frequently in the server logs:
SEVERE: A web application appears to have started a thread named
[22] but has failed to stop it. This is very likely
to create a memory leak.
Are there any potential reasons which might be causing it?
I'd use visualvm 1.3.2 and see what threads are being created. Be sure to add all the plug-ins.
If it's not being done by your code you won't have much control over it.
You also don't know if the message is a red herring or not. Load test your code over a period of time and measure what happens.
I faced similar situation recently Resolved in below steps
I took Thread dump. ( using Kill -QUIT pid )
Found the Runnable/Thread class form dump
Then i put a debug point in run method and started application in
debug mode.
Got the code which starts My Thread and I observed it was not
stopped while stoping application.
Introduced code to stop the thread in contextDestroyed method of
AppContextListener (This is My application class which extends
ServletContextListener ) as this method will be called when i stop
tomcat.
If you set Thread as Dameon Thread it is not going to help , you can visit explanation.
Tomcat waits for all the application's threads (user threads not daemon threads) to stop before it goes down, I guess that in your case this specific thread is a user thread and therefore tomcat generated this error.
I suggest you to change this thread to daemon (assuming this one is yours)

Categories