I am looking for an approach to gracefully handle shutdown of JBoss (Wildfly AS 8.2). This would mean that all current requests are served and the webapp stops receiving further requests. I found that this is possible via command line in version 9 of the application server -
./jboss-cli.sh --controller=remoting://<host>:<port>
--connect --command=:shutdown(timeout=t)
Using this JBoss gracefully handles all requests for t seconds and gracefully shuts down (this would require an upgrade from version 8 to 9).
One possible approach would be to handle this in the Java application by maintaining a count of active requests and waiting for this number to go to 0 till a timeout and then exitting, basically replicating the above mentioned functionality.
I need to shutdown the webapp/JBoss remotely, so we are looking for a JMX (Java Management Extension) based solution. Does JBoss expose any such operation to gracefully shut down possibly via JMX or any other technology?
PS- Ctrl-C or kill commands donot shutdown JBoss gracefully.
JBoss EAP 6/7 and above allows graceful shutdown via CLI and even:
The signals SIGHUP, SIGINT and SIGTERM all trigger a graceful shutdown of the JBoss 7/6 application server. Those are signals are respectively the commands kill -1 $PID, kill -2 $PID(or control+c), and kill $PID.
https://access.redhat.com/solutions/18601
You can use the Jboss cli remotely as the URL you pass in the controller param shows. You could even use the "REST" API to execute it.
The shutdown should be available from JMX.
Related
Gracefully shutting down a system ( using "shutdown" command ), terminates all the services registered under systemd in order and also send kill signal to all the running processes to give them a chance gracefully shut down.
Is there any specific order in which kill signal is sent to the processes which are not registered as service in systemd?
Is any any order between systemd services shut down and kill signal sent to other processes?
I've a java application process running on a VM and want that it's terminated only after a particular service registered under systemd has terminated. Is there any other way to achive this thing?
I would not count on non-service process order, because it might not exist or depend on OS flavors / versions.
How about creating our service which controls the process start/stop order and adding it to the system?
When running embedded Tomcat, my app sometimes fails to shut down via the /shutdown endpoint. A thread dump reveals that every thread is either BLOCKED, or IN_NATIVE. Below is a thread dump.
https://gist.github.com/aglassman/9d6a5e1d121af871345a72e25db6319f
Does this look typical? Should I continue investigating if there are blocked threads causing shutdown to fail, or should I look elsewhere?
I am running a web application using tomcat8 and jdk1.8 on a linux server.
When I shut down tomcat I get the following warning message:
WARNING [main] org.apache.tomcat.util.net.AbstractEndpoint.shutdownExecutor The executor associated with thread pool [http-apr-8080] has not fully shutdown. Some application threads may still be running
Note: I only get this warning when I shutdown tomcat, not when I undeploy my application.
How can I "fix" this warning so that the message won't appear.
I do a thread dump using Jstack while the application is running and I see several http-apr-8080-exec threads.
Thank you
We have to Stop any application before we shutdown the tomcat because the servlets might serve the requests during shutdown process.
Reference: https://www.safaribooksonline.com/library/view/tomcat-the-definitive/0596003188/ch01s02.html
Please check the section- Restarting Tomcat
It says that
"The Java Servlet Specification also dictates that, on shutdowns, servlet containers must wait for each servlet to finish serving all requests that are in progress or wait a container-specific timeout duration before taking servlets out of service. For Tomcat 4, that timeout duration is a maximum of a half-second per servlet. When a servlet misbehaves and takes too long to finish serving requests, it's up to Tomcat to figure out that the servlet has taken too long and to forcibly take it out of service so that Tomcat can shut down. This processing takes time, though, and slows Tomcat's own shutdown processing."
Best Practice:
When you Stop the application from tomcat, the servlets are out of service. Hence you will not face this issue.
Please follow the process of stopping the application first and shutdown the tomcat server.
This error was caused as a result of shutting down tomcat while the instance was still "attached" to the Elastic Load Balancer.
The ELB has a feature called Connection Draining which, by default, is set to 300 s.
You can disable connection draining or just remove the instance from the ELB before shutting down tomcat in order to not get the warning message.
I have a standalone java application based on spring and spring-rabbit libraries. I start it like this:
nohup java -jar myapp.jar &
But sometimes I have to restart the application for upgrading it. Now I use killall -9 java, but it's not the best way. How to stop it correctly and make sure that all requests which comes to rabbit listeners of this app at that period of time will not be partially processed and will just be rejected and go to other rabbit consumer?
First of all - do not use killall -9 - it sends a SIGKILL signal to the JVM which cannot be intercepted and will not allow for an orderly shutdown. Instead, use killall or killall -15 (15 is the default signal) that sends SIGTERM which is intercepted by the JVM and allows for an orderly shutdown.
Second of all - do not ACK the RabbitMQ message prematurely - only do so once the message has actually been processed. Until you ACK the message, RabbitMQ will keep it in the "unacked" state. If the consumer dies without ACKing, the message will be put back on the queue for another consumer to pick it up.
Depending on the framework you are using, you may need to register a shutdown hook to close your application in a clean way. For example, if you are using standalone Spring, you should call ConfigurableApplicationContext#registerShutdownHook on the ApplicationContext that you create to make sure all the beans (including RabbitMQ consumers) are closed correctly.
First of all, don't use autoack if you afraid of data loss. Set autoAck = false, then in your consumer, when handling a message, ack it or nack when you really finished message processing. So, you will not loss data if you somehow shutdown your java process (even if you shutdown your machine). RabbitMq will store it until your client will ack it.
read here https://www.rabbitmq.com/tutorials/tutorial-two-java.html
in 'message acknoledgement' part.
In order to properly run your java process, write your bash script, that will stop/start/restart your programm.
I have a service developed in DropWizard. I run the fat jar on an AWS server with the nohup command like this.
nohup java -jar myapp-0.0.1-SNAPSHOT.jar server my-configure.yml > outfile &
This works fine.
However, yesterday I tried to use the app that makes use of the service and it wasn't working. When I logged into the server I realized that the service was no longer running. It had been running for couple of months without any issues. In the logs (outfile) I just see the following:
INFO [2014-11-13 19:48:31,978] org.eclipse.jetty.server.Server: Graceful shutdown SocketConnector#0.0.0.0:9001
INFO [2014-11-13 19:48:31,987] org.eclipse.jetty.server.Server: Graceful shutdown InstrumentedBlockingChannelConnector#0.0.0.0:9000
INFO [2014-11-13 19:48:31,988] org.eclipse.jetty.server.Server: Graceful shutdown o.e.j.s.ServletContextHandler{/,null}
INFO [2014-11-13 19:48:31,988] org.eclipse.jetty.server.Server: Graceful shutdown o.e.j.s.ServletContextHandler{/,null}
Question
How can I troubleshoot why the service stopped running?...could AWS server have been restarted?
Is there a better way to do this that is not prone to errors like these
Is there a way I can be notified when the service stopped running ...rather than checking it every so often...
Edit
Just checked uptime on the AWS server and it says up for 7 days. Which coincides with when the service went down. I guess the solution is to have the service start automatically on server restart.
You can look at the solution used here to give yourself a notification when the program shuts down.
It won't know it was called with nohup though, that's something for the shell. But a java solution might work for you.
Based on your edit, you might want to create an init.d script to start (and stop) your process:
a template for that can be found here.