I am using spring retry template, when there is retry process is running, I am not able to stop the server. If I am trying to stop the server, It is saying tc server is not responding, would you like to terminate the server.
I followed the following way, but unfortunately the same result.
How can I shutdown Spring task executor/scheduler pools before all other beans in the web app are destroyed?
Thanks in Advance
You need to call shutdownNow() to interrupt any waiting (interruptible) threads.
Related
When the java app in the middle of the service process restarts, waits for the process and after finishing that continued or does abort the process.
Depends on which process we are talking about, in-flight http requests are killed, but this behaviour can be configured, see https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#web.graceful-shutdown.
When I send a stop signal(either with kill -SIGINT <pid> or System.exit(0) or environment.getApplicationContext().getServer().stop()) to the application, it waits for the shutdownGracePeriod (by default 30 sec or whatever I configure in .yml file) and also it does not accept new request. However, my requirement is to make the server wait for the ongoing request to complete before stopping. The ongoing request may take 30 sec or 30 minutes, it is unknown. Can somebody suggest me the way to achieve this?
Note: I've referred to the below links but could not achieve.
How to shutdown dropwizard application?
shutdownGracePeriod
We've used in-app healthcheck combined with some external load balancer service and prestop scripts. A healthcheck is turned off by the prestop script, then healthcheck says it is unhealthy so no new requests are sent by the load balancer (but existing ones are processed), only after draining period a stop signal is sent to the application.
Even this though has a specified time limit. I don't know how you would monitor requests that last an unknown amount of time.
I am running a web application using tomcat8 and jdk1.8 on a linux server.
When I shut down tomcat I get the following warning message:
WARNING [main] org.apache.tomcat.util.net.AbstractEndpoint.shutdownExecutor The executor associated with thread pool [http-apr-8080] has not fully shutdown. Some application threads may still be running
Note: I only get this warning when I shutdown tomcat, not when I undeploy my application.
How can I "fix" this warning so that the message won't appear.
I do a thread dump using Jstack while the application is running and I see several http-apr-8080-exec threads.
Thank you
We have to Stop any application before we shutdown the tomcat because the servlets might serve the requests during shutdown process.
Reference: https://www.safaribooksonline.com/library/view/tomcat-the-definitive/0596003188/ch01s02.html
Please check the section- Restarting Tomcat
It says that
"The Java Servlet Specification also dictates that, on shutdowns, servlet containers must wait for each servlet to finish serving all requests that are in progress or wait a container-specific timeout duration before taking servlets out of service. For Tomcat 4, that timeout duration is a maximum of a half-second per servlet. When a servlet misbehaves and takes too long to finish serving requests, it's up to Tomcat to figure out that the servlet has taken too long and to forcibly take it out of service so that Tomcat can shut down. This processing takes time, though, and slows Tomcat's own shutdown processing."
Best Practice:
When you Stop the application from tomcat, the servlets are out of service. Hence you will not face this issue.
Please follow the process of stopping the application first and shutdown the tomcat server.
This error was caused as a result of shutting down tomcat while the instance was still "attached" to the Elastic Load Balancer.
The ELB has a feature called Connection Draining which, by default, is set to 300 s.
You can disable connection draining or just remove the instance from the ELB before shutting down tomcat in order to not get the warning message.
On webapp start, it should connect to the socket and keep listening to events on the socket until the lifecycle of the webapp (forever, until of course tomcat is killed).
I have been searching for quite some time, but background tasks just return after their execution is done, but I want the thread to be alive to receive incoming data on the socket.
Any help ?
Do you use Spring? If so, you may want to take a look at Spring Integration project: http://docs.spring.io/spring-integration/reference/html/ip.html. I think it will be the best tool for your current problem.
Write the socket handler in ContextListener and then it'll be a part of the Main thread and won't die unless the application itself.
RabbitMQ RPC
I decided to use RabbitMQ RPC as described here.
My Setup
Incoming web requests (on Tomcat) will dispatch RPC requests over RabbitMQ to different services and assemble the results. I use one reply queue with one custom consumer that listens to all RPC responses and collects them with their correlation id in a simple hash map. Nothing fancy there.
This works great in a simple integration test on controller level.
Problem
When I try to do this in a web project deployed on Tomcat, Tomcat refuses to shut down. jstack and some debugging learned me a thread is spawn to listen for the RPC response and is blocking Tomcat from shutting down gracefully. I guess this is because the created thread is created on application level instead of request level and is not managed by Tomcat. When I set breakpoints in Servlet.destroy() or ServletContextListener.contextDestroyed(ServletContextEvent sce), they are not reached, so I see no way to manually clean things up.
Alternative
As an alternative, I could use a new reply queue (and simple QueueingConsumer) for each web request. I've tested this, it works and Tomcat shuts down as it should. But I'm wondering if this is the way to go.. Can a RabbitMQ cluster deal with thousands (or even millions) of short living queues/consumers? I can imagine queues aren't that big, but still.. constantly broadcasting to all cluster nodes.. the total memory footprint..
Question
So in short, is it wise do create a queue for each incoming web request or how should I setup RabbitMQ with one queue and consumer so Tomcat can shutdown gracefully?
I found a solution for my problem:
The Java client is creating his own threads. There is the possibility to add your own ExecutorService when creating a new connection. Doing so in the ServletContextListener.initialized() method, one can keep track of the ExecutorService and shut it down manually in the ServletContextListener.destroyed() method.
executorService.shutdown();
executorService.awaitTermination(20, TimeUnit.SECONDS);
I used Executors.newCachedThreadPool(); as the threads have many short executions, and they get cleaned up when being idle for more then 60s.
This is the link to the RabbitMQ Google group thread (thx to Michael Klishin for showing me the right direction)