We Keep Coding

Integrate a long running service into Spring boot framework

I have already implemented a kinesis stream consumer which will run forever and I want to integrate that into spring framework for monitoring and graceful shutdown. But I found I wasn't able to stop the consumer by the http shutdown request. More specifically, only the spring web app is stopped but not the consumer. Here's what I did:
I created a main class for spring as follows:
#SpringBootApplication
public class Application {
public static void main(String[] args) {
SpringApplication application = new SpringApplication(Application.class);
application.addListeners(new ApplicationPidFileWriter("./app.pid"));
application.run(args);
System.out.println("I'm Here!!");
}
}
And in the entrance of consumer class, I added #EventListener(ApplicationReadyEvent.class) to the startConsumer method
#EventListener(ApplicationReadyEvent.class)
public static void startConsumer() throws Exception {
init();
...
int exitCode = 0;
try {
worker.run(); // will run forever
} catch (Throwable t) {
System.err.println("Caught throwable while processing data.");
t.printStackTrace();
exitCode = 1;
}
System.exit(exitCode);
}
The consumer successfully started after mvn package && java -jar myJar, but when I use use the http shutdown to stop the program, only the spring app stops. The consumer was still running.
Any idea on how to stop the consumer? Or more generally how to integrate a long running process into spring framework? I've tried non-web choice, but that prevents me from using http requests to do monitoring.
Any suggestion will be appreciated!!!

One important thing is that it is not correct to block execution in EventListener. You should start a thread from the event listener method and that thread will do processing for you. So you need to invoke Worker.run in a separate thread. Alternatively you can mark your event listener as #Async.
Now the problem is to stop it correctly when spring boot application is stopped.
In order to be able to react to shutdown events in spring you can implement SmartLifecycle in your bean.
When stop is invoked you need to stop the Worker. This answer has some good options how to do that.
Make sure you invoke the Runnable passed to stop when worker shutdown is complete. For more details see SmartLifecycle.stop javadoc.

Why isn't my shutdownhook in my main method being called?

I have the following in my main method:
public static void main(String[] args) throws IOException {
Properties properties = getConfig();
Jedis jedis = configure(properties)
jedis.subscribe(queueHandler, "queue");
Runtime.getRuntime().addShutdownHook(new Thread() {
#Override
public void run() {
logger.debug("SHUTTING DOWN");
jedis.close();
}
});
}
I noticed that the code inside my shutdown hook is never run, why is that? How do I clean up resources that my main method has a hold of?

"Note that subscribe is a blocking operation ..." (from the AdvancedUsage wiki).
This in turn means that the shutdown-hook is only registered after you unsubscribe or shutdown Jedis,
and you probably only reach that part of the code when the JVM is already shutting down
(in which case the newly added shutdown hook is ignored: shutdown hooks have to be registered before the JVM shuts down).
"For more usage examples check the tests." (from the main Github page)
So let's look at the PublishSubscribeCommandsTest.
The test is just one subscribe operation but within the subscribe operation, the JedisPubSub un-subscribes itself when it receives a specific message thereby unblocking itself so that the test can finish.
Two things you can experiment with:
register the shutdown hook before calling jedis.subscribe
copy the code from the test and modify it so you can experiment with blocking and unblocking threads and shutting down Jedis from within a shutdown hook (to clean up used resources)

Regular exit of Main-Method differs a little from explicit calling System.exit(0);
so, try following:
public static void main(String[] args) throws IOException {
//all your code...
System.exit(0); //NOTE: non zero status will work different!
}

The shutdown hook will invoke if the jvm shuts down. Make sure that the program exits. If the jedis is running in the background (in a thread) even if the last statement is executed then the shutdown hook will not invoke. In such case, you have to stop that thread. The best way will be to try jedis.shutdown(); (after adding shutdown hook, here as the last statement) to shut it down.

Is it necessary to shutdown an ExecutorService for a webapplication running in a Tomcat container?

Is it necessary to shutdown an ExecutorService at some point if it runs in a Tomcat container for a Servlet. If yes, then where should I call the shutdown? I tried adding it after the submit() call but when I fire another request to the Servlet from a client browser, I get a RejectedExecutionException which is probably because I did a shutdown? I am trying to understand how it works in a Servlet within Tomcat and how I should use it.
I am doing the following in my webApplication (which seems to work fine without any shutdown):
// In configuration class
#Bean (name = "executorService")
public ExecutorService executorService() {
return Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors() + 1);
}
// In some other class
#Qualifier("executorService")
#Autowired
private ExecutorService executorService;
....
private void load() {
executorService.submit(new Runnable() {
#Override
public void run() {
doSomethingInTheBackground();
}
});
// If I enable this I will get a RejectedExecutionException
// for a next request.
// executorService.shutdown();
}

The idea behind the ExecutorService is to reuse threads. Creating threads is expensive and usually it is more efficient to create a thread once and then use that same thread multiple times. This is exactly what an ExecutorService does: it manages a pool of (possibly idle) threads and assigns work to them when you call its submit methods.
In a typical application you therefore do not want to shutdown the ExecutorService. You should however shut the ExecutorService down properly if your application is terminated. Since you are using Spring you don't have to worry about that:
By default, beans defined using Java config that have a public close or shutdown method are automatically enlisted with a destruction callback. [See the documentation.]
That means, if you close the ApplicationContext, Spring will automatically shutdown the ExecutorService for you.

wildfly have a ever running process run in the background

I'm making a html5/js game that will have online capabilities for my backend I've decided to use a wildfly server. The client will communicate with the server via web sockets.
I intended for my wildfly server to also be in charge of game logic decisions such as moving npc's. My plan was to have a #startup bean that would run a server game loop to handle this logic. The server loop would then talk to the serverEndPoint via HornetQ. My serverEndPoint and server loop look like this:
ServerEndPoint
#ServerEndpoint(value= "/game/{user-id}")
public class GameEndpoint {
#Inject
GameManager gameState;
GameWorld gameWorld;
Player p;
private Logger logger = Logger.getLogger(this.getClass().getName());
#OnOpen
public void onOpen(Session session){
//do stuff
}
#OnMessage
public void onMessage(Session session, String message){
//do stuff
}
#OnClose
public void onClose(CloseReason reason){
//do stuff
}
#OnError
public void error(Throwable t){
//do stuff
}
}
GameWorld
#Startup
#Singleton
public class GameWorld {
#Inject
GameManager gameState;
private Logger logger = Logger.getLogger(this.getClass().getName());
#PostConstruct
public void init(){
gameloop();
}
private void gameloop(){
while(true){
logger.info("This is a test!");
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
#PreDestroy
public void terminate(){
//do stuff
}
}
the issue with this is that the server loop freezes everything as it is a infinite loop(for instance if I try and access the html web page I get a 404). obviously this could be solved if the serverLoop was on its own seperate thread but after doing some research it seems threading in jboss is very difficult as its hard to know what dependencies to inject e.t.c
Can anyone shed some light on how I can solve this issue? any help on the matter would be amazing.

What you have encountered has to do with what Java EE is and what it not is: Java EE is optimized for handling many concurrent, short-lived requests, each of which (usually) handle a single transaction. The containers do that very well, particularly with stateless beans, but also with stateful beans (cluster replication etc). As such, Java EE might be well-suited to process the requests coming from your HTML5/JS clients and feed the requests to the messaging infrastructure. Java EE is not, however, designed for long running, thread-blocking background processes like yours.
FWIW: Another issue that you have not yet encountered is, even if you could get that one fixed: Next you'll encounter the transaction timeout on your #PostConstruct method.
I think you are better of with moving the game engine out of the Java EE stack. You already mentioned you plan to use HornetQ - then why not put the game engine in a simple standalone application that receives messages from HornetQ and feeds replies back into HornetQ.
Another option might be a dedicated Java game server engine, see, e.g., this question and its accepted answer on programmers.stackoverflow.com. (Update: it seems the "RedDwarf Server" project mentioned in that answer was discontinued 3 years ago).
If you absolutely want to use the Java EE environment, I suggest you use a TimerService instead. Note, however, that this also requires that your game loop calculation is quick and guaranteed to finish until the next timeout is scheduled, otherwise the container will skip the scheduled invocation (with a "still running" message or similar).
Finally, let me mention that if I were to start a new game server today, I would definitely take a look at Akka, Node.js or similar projects that support "reactive" programming.

Calling System.exit() in Servlet's destroy() method

This is a follow up to my earlier question.
Tomcat 5.0.28 had a bug where the Servlet's destroy() method was not being invoked by the container on a shutdown. This is fixed in Tomcat 5.0.30, but if the Servlet's destroy() method had a System.exit(), it would result in the Tomcat windows service throwing the Error 1053 and refusing to shutdown gracefully (see above link for more details on this error)
Anybody has any idea on whether:
Calling System.exit() inside a Servlet's destroy() method to forcefully kill any non-daemon threads is a good idea?
Why does Tomcat 5.0.30 and (later versions including Tomcat 6.x.x) fail to shutdown properly if there's a System.exit() in the destroy() method of the Servlet.

Calling System.exit() inside a Servlet's destroy() method to forcefully kill any non-daemon threads is a good idea?
It is absolutely not a good idea - it is a horrible idea. The destroy() method is called when the servlet is taken out of service, which can happen for any number of reasons: the servlet/webapp has been stopped, the webapp is being undeployed, the webapp is being restarted etc.
System.exit() shuts down the entire JVM! Why would you want to forcibly shutdown the entire server simply because one servlet is being unloaded?
Why does Tomcat 5.0.30 and (later versions including Tomcat 6.x.x) fail to shutdown properly if there's a System.exit() in the destroy() method of the Servlet.
Probably to prevent such dangerous behavior like this.
You shouldn't write code that assumes that your code/application is the only thing running on the server.

You're asking two questions:
Question 1: Is calling System.exit() inside a Servlet's destroy() method to forcefully kill any non-daemon threads a good idea?
Calling System.exit() inside ANY servlet-related method is always 100% incorrect. Your code is not the only code running in the JVM - even if you are the only servlet running (the servlet container has resources it will need to cleanup when the JVM really exits.)
The correct way to handle this case is to clean up your threads in the destroy() method. This means starting them in a way that lets you gently stop them in a correct way. Here is an example (where MyThread is one of your threads, and extends ServletManagedThread):
public class MyServlet extends HttpServlet {
private List<ServletManagedThread> threads = new ArrayList<ServletManagedThread>();
// lots of irrelevant stuff left out for brevity
public void init() {
ServletManagedThread t = new MyThread();
threads.add(t);
t.start();
}
public void destroy() {
for(ServletManagedThread thread : threads) {
thread.stopExecuting();
}
}
}
public abstract class ServletManagedThread extends Thread {
private boolean keepGoing = true;
protected abstract void doSomeStuff();
protected abstract void probablySleepForABit();
protected abstract void cleanup();
public void stopExecuting() {
keepRunning = false;
}
public void run() {
while(keepGoing) {
doSomeStuff();
probablySleepForABit();
}
this.cleanup();
}
}
It's also worth noting that there are thread/concurrency libraries out there that can help with this - but if you really do have a handful of threads that are started at servlet initialization and should run until the servlet is destroyed, this is probably all you need.
Question 2: Why does Tomcat 5.0.30 and (later versions including Tomcat 6.x.x) fail to shutdown properly if there's a System.exit() in the destroy() method of the Servlet?
Without more analysis, it's hard to know for certain. Microsoft says that Error 1053 occurs when Windows asks a service to shutdown, but the request times out. That would make it seem like something happened internally to Tomcat that got it into a really bad state. I would certainly suspect that your call to System.exit() could be the culprit. Tomcat (specifically, Catalina) does register a shutdown hook with the VM (see org.apache.catalina.startup.Catalina.start(), at least in 5.0.30). That shutdown hook would get called by the JVM when you call System.exit(). The shutdown hook delegates to the running services, so each service could potentially be required to do alot of work.
If the shutdown hooks (triggered by your System.exit()) fail to execute (they deadlock or something like that,) then it is very easy to understand why the Error 1053 occurs, given the documentation of the Runtime.exit(int) method (which is called from System.exit()):
If this method is invoked after the
virtual machine has begun its shutdown
sequence then if shutdown hooks are
being run this method will block
indefinitely. If shutdown hooks have
already been run and on-exit
finalization has been enabled then
this method halts the virtual machine
with the given status code if the
status is nonzero; otherwise, it
blocks indefinitely.
This "indefinite blocking" behavior would definitely cause an Error 1053.
If you want a more complete answer than this, you can download the source and debug it yourself.
But, I would be willing to bet that if you properly handle your thread management issue (as outlined above,) your problems will go away.
In short, leave the System.exit() call to Tomcat - that's not your job.

Calling System.exit() inside a
Servlet's destroy() method to
forcefully kill any non-daemon threads
is a good idea?
Not a good idea. You will forcefully kill all threads, which might include part of Tomcat that is currently shutting down the system. This will cause Tomcat to un-gracefully shutdown. This can also prevent shutdown handlers from running which can lead to all sorts of problems.
Why does Tomcat 5.0.30 and (later
versions including Tomcat 6.x.x) fail
to shutdown properly if there's a
System.exit() in the destroy() method
of the Servlet.
A lot of code executes after a Servlet destory. The Context destroy and all of its other listeners for one... other servlets. Other applications. Tomcat itelf. By calling System.exit, you prevent all of that from running.
A better question is what are thse non-daemon threads, why are they running, and who starts them?

When writing thread shutdown code like Jared's, I normally make the "keepGoing" member and "stopExecuting()" method static so that all threads get the signal to go down with one shutdown call. Good idea or no?