We need to implement a graceful shutdown mechanism into our Servlet application.
EDIT: We want to make it simple as possible, which would be handling a kill signal sent via operating system's function. This would allow system admins to use built in shell utilities (kill or taskkill on Windows), otherwise they would have to install another utility just to "talk" with server.
This mechanism works in two phases:
upon shutdown request, deny certain critical activities
block until previously initiated critical actions are completed; these may take several hours
Phase #1 is implemented in our DAO layer.
Phase #2 is implemented in our ServletContextListener#contextDestroyed method
Our problem is that once contextDestroyed is called the Servlet container stops servicing further HTTP requests.
EDIT: contextDestroyed is called when someone is calling the operating system's kill function on server's process.
We would like to let the application alive during Phase #2, notifying the users that some activities are unavailable.
Use a filter to keep a list of all critical requests.
When the "prepare shutdown" request is received, the filter should start denying some requests.
Write a servlet that tells you how many critical jobs are still left in the queue.
In the shutdown tool, send the "prepare shutdown". The poll the servlet for the number of critical jobs. When this reaches 0, send the actual shutdown command.
To make this happen, create a service in the business layer which orchestrates this. Note that everything must happen before contextDestroyed() is being called! Your special application shutdown doesn't fit into the J2EE view of the world, so you have to manage it yourself.
The service should be able to tell interested parties when a shutdown is in progress, how many critical jobs are still running, etc. Servlets and filters can then use this service to deny requests or tell how many jobs are left.
When all jobs are done, deny all requests except access to the "shutdown info" servlet which should then tell that the app is now ready for death.
Write a tool which gives the administrators a nice UI to initiate shutdown of your app.
[EDIT] You may feel tempted to prevent the OS from shutting down your application. Don't do that.
What you should do is write a special tool to shut down your application using the two phase process that I described above. This should be the standard way to shutdown.
Yes, administrators will complain about it. On Unix, you can hide this tool by putting it into the init script, so no one will notice. There might be a similar solution on Windows.
Killing the server should always be possible to be able to stop it in case of (un)expected circumstances like: bugs in your shutdown code, emergency shutdown during power failure, bugs in your application code, or when Murphy happens.
Related
I'm interested in different approaches to gracefully shutting down a Java command line program. Sending a kill signal is not an option.
I can think of a few different approaches.
Open a port and wait for a connection. When one is made, gracefully shutdown.
Watch for a file to be created, then shutdown.
Read some input from the terminal, such as "execute shutdown".
The third one is not ideal, since there is often program output pumped to the screen. The first one takes too much effort (I'm lazy). Do most programmers use the second option? If not, what else is possible/elegant/simple?
you can try something like this:
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() { /*
my shutdown code here
*/ }
});
edit:
the shutdown hook will not perform the shutting down of the app. instead, it gives the developer a way to perform any clean-up that he/she wishes at shutdown.
from the JavaDoc for Runtime (a good read if you are planning to use this method):
A shutdown hook is simply an initialized but unstarted thread. When the virtual machine begins its shutdown sequence it will start all registered shutdown hooks in some unspecified order and let them run concurrently. When all the hooks have finished it will then run all uninvoked finalizers if finalization-on-exit has been enabled. Finally, the virtual machine will halt. ...
you could try to use Runtime.getRuntime().addShutdownHook() that should satisfy your requisite. In this way you can register an hook to do cleanups, in order to perfom a gracefull shutdown.
EDIT
http://java.sun.com/j2se/1.4.2/docs/api/java/lang/Runtime.html#addShutdownHook(java.lang.Thread)
public void addShutdownHook(Thread hook)
Registers a new virtual-machine shutdown hook.
The Java virtual machine shuts down in response to two kinds of events:
The program exits normally, when the last non-daemon thread exits or when the exit (equivalently, System.exit) method is invoked, or
The virtual machine is terminated in response to a user interrupt, such as typing ^C, or a system-wide event, such as user logoff or system shutdown.
The benefit of the second option - checking for a file - over the first - listening on a port - is that you have some possibility of security.
You can set the permissions on the directory where the file is created so that only appropriate users can close the program. If you listen on a port any user can connect to it.
If you wanted to go with the socket version, it is very simple to implement. Here's the code:
ServerSocket server = new ServerSocket(8080);
System.out.println("Socket listening!");
server.accept();
System.out.println("Connection received!");
You could easily embed this code in a separate thread that you start with your program, and then have it modify global state to initiate shutdown.
The first two option is simple to implement. You could also use some JMX stuff (I don't know much about that). Tomcat uses the first approach and I applied 1 and 2 in two of my projects.
Consider having a JMX component. Then you can attach with JConsole either locally or over the network, and communicate with your component. Then the component can shut down the program properly.
With Java 6 u 10 or later, you can do the same with JVisualVM.
I would suggest to use the shutdown hook. It will allow your program do be controlled using standard OS tools. It also does not need any additional access to external resources (disk, ports, whatever).
Currently I have slow starting java service in systemd which takes about 60 seconds until it opens its HTTP port and serves other clients.
Another client service expects this service to be available (is a client of the this service), otherwise it dies after a certain retry. It also started with systemd. This is to be clear also a service. But uses the former like database.
Can I configure systemd to wait until the first service has made his socket available? (something like if the socket is actually listens , then the second client service should start).
Initialization Process Requires Forking
systemd waits for a daemon to initialize itself if the daemon forks. In your situation, that's pretty much the only way you have to do this.
The daemon offering the HTTP service must do all of its initialization in the main thread, once that initialization is done and the socket is listening for connections, it will fork(). The main process then exits. At that point systemd knows that your process was successfully initialized (exit 0) or not (exit 1).
Such a service receives the Type=... value of forking as follow:
[Service]
Type=forking
...
Note: If you are writing new code, consider not using fork. systemd already creates a new process for you so you do not have to fork. That was an old System V boot requirement for services.
"Requires" will make sure the process waits
The other services have to wait so they have to require the first to be started. Say your first service is called A, you would have a Requires like this:
[Unit]
...
Requires=A
...
Program with Patience in Mind
Of course, there is always another way which is for the other services to know to be patient. That means try to connect to the HTTP port, if it fails, sleep for a bit (in your case, 1 or 2 seconds would be just fine) then try again, until it works.
I have developed both methods and they both work very well.
Note: A powerful aspect to this method, if service A gets restarted, you'd get a new socket. This server can then auto-reconnect to the new socket when it detects that the old one goes down. This means you don't have to restart the other services when restarting service A. I like this method, but it's a bit more work to make sure it's all properly implemented.
Use the systemd Auto-Restart Feature?
Another way, maybe, would be to use the restart on failure. So if the child attempts to connect to that HTTP service and fails, it should fail, right? systemd can automatically restart your process over and over again until it succeeds. It's sucky, but if you have no control over the code of those daemons, it's probably the easiest way.
[Service]
...
Restart=on-failure
RestartSec=10
#SuccessExitStatus=3 7 # if success is not always just 0
...
This example waits 10 seconds after a failure before attempting to restart.
Hack (last resort, not recommended)
You could attempt a hack, although I do not ever recommend such things because something could happen that breaks such... in the services, change the files so that they have a sleep 60 then start the main process. For that, just write a script like so:
#!/bin/sh
sleep 60
"$#"
Then in the .service files, call that script as in:
ExecStart=/path/to/script /path/to/service args to service
This will run the script instead of directly your code. The script will first sleep for 60 seconds and then try to run your service. So if for some reason this time the HTTP service takes 90 seconds... it will still fail.
Still, this can be useful to know since that script could do all sorts of things, such as use the nc tool to probe the port before actually starting the service process. You could even write your own probing tool.
#!/bin/sh
while true
do
sleep 1
if probe
then
break
fi
done
"$#"
However, notice that such a loop is blocking until probe returns with exit code 0.
You have several options here.
Use a socket unit
The most elegant solution is to let systemd manage the socket for you. If you control the source code of the Java service, change it to use System.inheritedChannel() instead of allocating its own socket, and then use systemd units like this:
# example.socket
[Socket]
ListenStream=%t/example
[Install]
WantedBy=sockets.target
# example.service
[Service]
ExecStart=/usr/bin/java ...
StandardInput=socket
StandardOutput=socket
StandardError=journal
systemd will create the socket immediately (%t is the runtime directory, so in a system unit, the socket will be /run/example), and start the service as soon as the first connection attempt is made. (If you want the service to be started unconditionally, add an Install section to it as well, with WantedBy=multi-user.target.) When your client program connects to the socket, it will be queued by the kernel and block until the server is ready to accept connections on the socket. One additional benefit from this is that you can restart the service without any downtime on the socket – connection attempts will be queued until the restarted service is ready to accept connections again.
Make the service signal readiness to systemd
Alternatively, you can set up the service so that it signals to systemd when it is ready, and order the client after it. (Note that this requires After=example.service, not just Requires=example.service! Dependencies and ordering are orthogonal – without After=, both will be started in parallel.) There are two main service types that might make this possible:
Type=forking: systemd will consider the service to be ready as soon as the main program exits. Since you can’t fork in Java, I think you would have to write a small shell script which starts the server in the background and then waits until the socket is available (while ! test -S /run/example; do sleep 1s; done). Once the script exits, the service is considered ready.
Type=notify: systemd will wait for a message from the service before it is considered ready. Ideally, the message should be sent from the service PID itself: check if you can call the sd_notify function from libsystemd via JNI/JNA/whatever (specifically, sd_notify(0, "READY=1")). If that’s not possible, you can use the systemd-notify command-line tool (--ready option), but then you need to set NotifyAccess=all in the service unit (by default, only the main process may send notifications), and even then it likely will not work (systemd needs to process the message before systemd-notify exits, otherwise it will not be able to verify which cgroup the message came from).
I created a webapplication that needs to do some cleanup on shutdown. This cleanup will take about a minute and its completely OK for it to do so.
When I deploy my webapp onto Tomcat 8 and then stop it, my ContextListener gets called and the cleanup begins. But it seems like Tomcat stops my thread the hard way and it won't complete anymore. At least on Tomcat 6 that wasn't an issue.
An ideas how to configure Tomcat 8 to stop from misbehaving?
Partial Answer:
I found out it has something to do with a performance optimization I did. I used startStopThreads="2" to start my applications in parallel, which works out well, but on shutdown this also seems to kill my threads.
If you have a task which is to be performed on shutdown, I would add this as shutdown hook. Most likely Tomcat 8 is called System.exit() which is a normal thing to do and this kills all user threads but start shutdown hooks.
A better solution is to never leave the system in a state where you really need this. i.e. you cannot assume an application will die gracefully.
if you are waiting for client to disconnect, I suggest you add a shutting down phase. During this phase you refuse new connections, move connections to another server or attempt to gracefully tell existing ones you are going away. After a short period or time out, you then shut down.
I would like to inform all logged in users that the server will shutdown. This special interest would be nice in an ajaxfy application (RIA).
What are the possible solutions? What are the best practice solutions?
There were two possible end-scenarios:
Send a text $x to the server ergo to all users. ("The server will not be available for some minutes.")
Send a key $y to the server which will used to generate a (custom) text to all users. ("SERVER_SHUTDOWN")
Environment: Tomcat (6/7), Spring 3+
Messaging to users: with polling or pseudo-pushing via an async servlet.
Ideas
1. Context.destroy(): Implementing a custom ContextListener's destroy
I don't think it is a good solution to block within a "destroy()" -- blocking, because we should wait about 5-10 seconds to make sure that all logged in users receive a message.
2. JMX Beans
This would mean, that any server service operation (start, stop) have to invoke a special program which sends the message.
3. Any other messaging queues like AMQP or ActiveMQ
Like 2.
Unless the server shuts down regularly and the shutdown has a significant impact on users (for e.g. they will lose any unsubmitted work - think halfway through editing a big post on a page) then notifying of server shutdown won't really be of much benefit.
There are a couple of things you could do.
First, if the server is going to be shutdown due to planned maintenance then you could include a message on web pages like;
Server will be unavailable Monday 22nd Aug 9pm - 6am for planned
maintenance. Contact knalli#example.com for more information.
Second, before shutting down the server, redirect requests to a static holding page (just change your web server config). This holding page should have information on why the server is down and when it will be available again.
With both options, its also important to plan server downtime. It's normal to have maintenance windows outside of normal working hours. Alternatively, if you have more than one server you can cluster them. This allows you to take individual servers out of the cluster to perform maintenance without having any server downtime at all.
Searching memcached java in google, the first result is Using Memcached with Java.
The guy (who calls himself Just some Random Asshole in the Internet!) proposes a Singleton based on net.spy.memcached. It basically creates 20 threads and connections by creating 20 instances of MemcachedClient. For every request it chooses one at random.
However those threads and connections are never closed and they pile up every time I hot swap the application during development (with warnings from Tomcat 7).
SEVERE: The web application [/MyAppName] appears to have started a thread named
[...] but has failed to stop it. This is very likely to create a memory leak.
By looking at MemcachedClient JavaDoc, I see a method called shutdown with the only description being "Shut down immediately." Shut down what? The client? The server? I suppose is the client, since it's in MemcachedClient and I suppose that this method would close the connection and terminate the thread. EDIT: yes, it shuts down the client.
Question 1 How to force the execution of cleanup code in Tomcat 7, before the application is hot swapped?
Question 2 Is this approach of using memcached (with cleanup code), correct or is better I start over in a different way?
I think creating 20 memcache clients is silly - that's like creating 20 separate copies of your DB connection pool. The idea with that client is that it multiplexes a variety of requests with asynch IO.
http://code.google.com/p/spymemcached/wiki/Optimizations
As far as shutting it down, simply call:
yourClient.shutdown() to shutdown immediately, or
yourClient.shutdown(3, TimeUnit.SECONDS) for example, to allow some time for a more graceful shutdown.
That could be called from your Servlet's .destroy method, or a context listener for your whole WAR.
I don't know anything about memcached, but you could probably write a custom context listener and put some kind of shutdown hook in the context listener so that when the context shutdown you could loop through the items in your singleton and shut them down.
It turned out that it was a bug of Java AWS SDK and was not related to memcached. Version 1.2.2 of Java AWS SDK has this bug fixed.