We Keep Coding

Requests get dropped during springboot service pod rollout

I am using the Kubernetes cluster with docker. When I deploy the java services[springboot] some requests get dropped(for a couple of secs) with the following error.
exception=org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'controller': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!), stackTrace=[org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton() at line: 208]
I am already using livenessProbe & readinessProbe.
Java Version: 12
SpringBoot Version: 2.1.5.RELEASE
Hibernate Version: 2.4.3 with Postgres DB
As per my knowledge, it is happening due to the closing of the application context while executing some requests. Ideally, it should not be.
can anyone help here ?

the problem is not actually springboot, but rather the way Kubernetes stops pods.
at the moment when a pod from your old deployment/replicaset is being terminated (or rather actually set to state "terminating"), 2 things happen simultaneously:
A) pod is removed from service endpoints, so it does no longer receive new requests
B) pod container gets a SIGTERM, so apps can gracefully shutdown
so what you are seeing here is basically active requests that are being processed when the context gets shut down (as you already found out)
there are (at least) two solutions:
1 in kubernetes pod definition:
Kubernetes pods can be configured with a pre-stop hook that get executes a command in between A and B.
depending on your app, a simple "sleep" for a couple (milli)seconds should be sufficient, leaving the app enough time to finish the current requests before shutting down.
theres nice docu from google that goes more into detail:
https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace
2 in SpringBoot:
you can make Java wait for finishing up running tasks when receiving the shutdown interrupt.
this is (imho) nicely explained here:
https://www.baeldung.com/spring-boot-graceful-shutdown
Beware: kubernetes default graceful shutdown timeout is 30seconds, then the pod is forcefully removed. but as usual you can configure this timeout in terminationGracePeriodSeconds (also described in the google blog in (1)

How do I properly restart a Spring boot application?

I'm using the code from here to restart my Spring Boot application:
Thread restartThread = new Thread(() -> restartEndpoint.restart());
restartThread.setDaemon(false);
restartThread.start();
However, the restarted application fails to create beans, ultimately due to a RejectedExecutionException thrown when trying to schedule #Scheduled-annotated methods:
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'myBean' defined in URL [insert-path-here]: Initializetion of bean failed; nested exception is org.springframework.core.TaskRejectedException: Executor [java.util.concurrent.ScheduledThreadPoolExecutor#284a4a89[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 21508]] did not accept task: my.package.MyBean.scheduledMethod
The logs immediately after the doRestart() invocation indicate that the ExecutorService taskScheduleris being shut down, which would explain why this is happening. What I'm not entirely sure of is why the ExecutorService isn't recreated. Embarrassingly enough, it turns out that taskScheduler was a bean defined by my application and was not annotated #RefreshScope. Now the error I'm getting is
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'anotherBean' definen in URL[...]: Unsatisfied dependency expressed through constructor parameter 0 ... nested exception is java.lang.IllegalStateException: org.springframework.boot.servlet.context.AnnotationConfigServletWebServerApplicationContext#2a95076c has been closed already
Scrolling up a bit in the logs reveals that there's an ApplicationContextException thrown with the stacktrace containing doRestart:
org.springframework.ApplicationContextException: Unable to start web server; nested exception is java.lang.IllegalStateException: Filters cannot be added to context [/myapp] as the context has been initialized
at org.springframework.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:157)
...
at org.springframework.boot.SpringApplication.run(SpringApplication.java:316)
at org.springframework.cloud.context.restart.RestartEndpoint(RestartEndpoint.java:160)
at my.package.MyBean.lambda$doThingAndRestart$2 (MyBean.java: 118)
Googling this error seems to result in a single page that isn't relevant, although I do note that this application is running on tomcat as well.
I suspect that this is related to the application creating a wrong sort of application context, which is a AnnotationConfigServletWebServerApplicationContext, which doesn't extend AbstractRefreshableAplicationContext.
What do I need to do to get rid of this error?
Please bear with me when I only copy single lines from the stack trace. I'm on a two-computer set-up, so I can't copy-and paste and am no allowed to otherwise move the logs to a computer connected to the Internet.

It turns out that RestartEndpoint only works with Spring's internal Tomcat, not an external one. So, another solution is required. The ones I discovered are
Using Tomcat's WatchedResource to re-deploy the server when a configuration file is overwritten. This has the downside of needing to maintain the Tomcat configuration when maintaining the doThingAndRestart functionality.
Using Tomcat's Manager App. I noticed that the reload request blocks until the app is done reloading, which overflowed the heap space.
Using Spring's RefreshEndpoint. For whatever reason, this didn't refresh the datasources.

Is it possible for a spring bean to get notified about application context initialization errors?

I have a singleton bean in my spring context (call it 'beanX' ) that, when started spins off a few other threads and creates some state on the file system.
So I want to stop it in a clean way so that resources are freed when the context is shut down.
I have annotated it with #Bean(destroyMethod = "shutdown") and that works as expected without any problems.
Further information is that this bean loads very early in the context startup sequence (as it has many other beans that use it directly and indirectly) and there are many other beans that loads after it.
Now to the problem: it is fairly common that when we do development one of the other beans fail to start and makes the startup of the spring context to fail - but in these cases the beanX is not shut down properly.
This is perhaps ok in 'productionCode' as that usually means that the whole jvm stops and then resources are cleaned up - BUT when we run our 1000+ unit test suite (and some of the tests has an error causing the context to crash) this becomes a real problem because the test JVM is not stopped between tests, and a new spring context is created after each failing test - even though the test that failed has not properly cleaned up after itself as beanX is started but never stopped!
Today my laptop created 5 million file handles and 13k threads when I ran the test suite because of this.
SO, the only way we could make this work currently is to make beanX be a public static field (ClassX.beanX) that is lazily initialized by the spring context - and then in our test code manually call the ClassX.beanX.shutdown() if any context loading error occurs.
But I am curios if there is a better/different 'Spring' way to do this?

You can use event handling in Spring.
Event handling in the ApplicationContext is provided through the
ApplicationEvent class and ApplicationListener interface. If a bean
that implements the ApplicationListener interface is deployed into the
context, every time an ApplicationEvent gets published to the
ApplicationContext, that bean is notified. Essentially, this is the
standard Observer design pattern.

Is there a way to have spring run additional code at application startup failure?

We have a spring web-application running on a Tomcat server that we would like to have some additional code run ONLY if there is an issue with the webapp's startup.
However, any errors we receive will come from either Bean Creation Issues (which, hopefully we would catch before ever releasing) or a Flyway upgrade script issue. Both cases, the exceptions are caught within the spring core somewhere, and I'd like to run some additional code in those situations before the webapp failure finishes. I'm not 100% sure how to set this up though, considering where the exceptions are thrown.
I've been looking at Spring's Life Cycle configuration annotations, and while I have considered running the code in the #PreDestroy method we're providing, that means it would run our code every time we restarted the webapp, instead of just on startup failures. Is there a way to indicate the differences between a normal shutdown and a failed startup using the life cycle annotations?

You could register your own ServletContextListener instead of the Spring's one and delegate contextInitialized method invocation to the Spring implementation you use for loading Spring context (whichever one you use, for example org.springframework.web.context.ContextLoaderListener).
Then catch and handle the desired exceptions.

Continue to load webapp even if one spring bean initialization fails

So if spring initialization fails in a webapp then the webapp itself does not come up. To prevent this, I can probably not re-throw any exception from my code for that specific bean initialization and the webapp will continue to load, right?
Is there any other way to tell to spring not to fail the webapp itself on particular bean initialization failure?

Continue to load webapp even if one spring bean initialization fails
AFAIK, you can't do this.
I do multiple DNS lookups on start up. I do not want the webapp to fail if one of them fails.
In that case, you need to modify your bean(s) to handle the case where the DNS lookup fails, leaving the bean instance in a state where it is essentially inactive but harmless, or where it can retry the DNS lookup later on.
In short, you have to cope with this yourself.

Have attribute lazy-init="true" in that bean and every dependant bean. Link for more details.

why would you do this? If your spring context is not correct, something is seriously wrong and there will be issues. The correct way to deal with this is to fix the application context.