We Keep Coding

Requests get dropped during springboot service pod rollout

I am using the Kubernetes cluster with docker. When I deploy the java services[springboot] some requests get dropped(for a couple of secs) with the following error.
exception=org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'controller': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!), stackTrace=[org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton() at line: 208]
I am already using livenessProbe & readinessProbe.
Java Version: 12
SpringBoot Version: 2.1.5.RELEASE
Hibernate Version: 2.4.3 with Postgres DB
As per my knowledge, it is happening due to the closing of the application context while executing some requests. Ideally, it should not be.
can anyone help here ?

the problem is not actually springboot, but rather the way Kubernetes stops pods.
at the moment when a pod from your old deployment/replicaset is being terminated (or rather actually set to state "terminating"), 2 things happen simultaneously:
A) pod is removed from service endpoints, so it does no longer receive new requests
B) pod container gets a SIGTERM, so apps can gracefully shutdown
so what you are seeing here is basically active requests that are being processed when the context gets shut down (as you already found out)
there are (at least) two solutions:
1 in kubernetes pod definition:
Kubernetes pods can be configured with a pre-stop hook that get executes a command in between A and B.
depending on your app, a simple "sleep" for a couple (milli)seconds should be sufficient, leaving the app enough time to finish the current requests before shutting down.
theres nice docu from google that goes more into detail:
https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace
2 in SpringBoot:
you can make Java wait for finishing up running tasks when receiving the shutdown interrupt.
this is (imho) nicely explained here:
https://www.baeldung.com/spring-boot-graceful-shutdown
Beware: kubernetes default graceful shutdown timeout is 30seconds, then the pod is forcefully removed. but as usual you can configure this timeout in terminationGracePeriodSeconds (also described in the google blog in (1)

Difference between ManagedExecutorService and ManagedThreadFactory in Java EE

Our current legacy web-app creates threads in it not managed by Application Server Containers. I have to modify it with JavaEE standards for multi-threading.
My web-app works fine on Tomcat but fails on Websphere.
Error on Websphere :
... ... Caused by: javax.naming.ConfigurationException: A JNDI operation on a "java:" name cannot be completed because the server runtime is not able to associate the operation's thread with any J2EE application component. This condition can occur when the JNDI client using the "java:" name is not executed on the thread of a server application request. Make sure that a J2EE application does not execute JNDI operations on "java:" names within static code blocks or in threads created by that J2EE application. Such code does not necessarily run on the thread of a server application request and therefore is not supported by JNDI operations on "java:" names.
at com.ibm.ws.naming.java.javaURLContextImpl.throwExceptionIfDefaultJavaNS(javaURLContextImpl.java:534) ~[com.ibm.ws.runtime.jar:?]
at com.ibm.ws.naming.java.javaURLContextImpl.throwConfigurationExceptionWithDefaultJavaNS(javaURLContextImpl.java:564) ~[com.ibm.ws.runtime.jar:?]
at com.ibm.ws.naming.java.javaURLContextImpl.lookupExt(javaURLContextImpl.java:485) ~[com.ibm.ws.runtime.jar:?]
at com.ibm.ws.naming.java.javaURLContextRoot.lookupExt(javaURLContextRoot.java:485) ~[com.ibm.ws.runtime.jar:?]
In order to resolve this issue, I am referring Concurrency Utilities in Java EE. I found similar kind of description and example for ManagedExecutorService and ManagedThreadFactory.
ManagedExecutorService: A managed executor service is used by applications to execute submitted tasks asynchronously. Tasks are
executed on threads that are started and managed by the container. The
context of the container is propagated to the thread executing the
task.
ManagedThreadFactory: A managed thread factory is used by applications to create managed threads. The threads are started and
managed by the container. The context of the container is propagated
to the thread executing the task. This object can also be used to
provide custom factories for specific use cases (with custom Threads)
and, for example, set specific/proprietary properties to these
objects.
Which one is preferred in which condition and why ?

I have solved issue by using ManagedExecutorService.
ExecutorService framework indeed has more ways to deal threads while ManagedThreadFactory can only call newThread() method.
Websphere issue can be resolved by using ManagedExecutorService or ManagedThreadFactory. Both works. But for further thread processing, ManagedExecutorService turns out lot better.
Now, this solution causes same web-app to fail on Tomcat. JNDI naming exception. As per my R&D, container based concurrency is supported in TomEE server not in Tomcat so we have to use routing mechanism to switch between code as per underlying application server.

Is there a way to have spring run additional code at application startup failure?

We have a spring web-application running on a Tomcat server that we would like to have some additional code run ONLY if there is an issue with the webapp's startup.
However, any errors we receive will come from either Bean Creation Issues (which, hopefully we would catch before ever releasing) or a Flyway upgrade script issue. Both cases, the exceptions are caught within the spring core somewhere, and I'd like to run some additional code in those situations before the webapp failure finishes. I'm not 100% sure how to set this up though, considering where the exceptions are thrown.
I've been looking at Spring's Life Cycle configuration annotations, and while I have considered running the code in the #PreDestroy method we're providing, that means it would run our code every time we restarted the webapp, instead of just on startup failures. Is there a way to indicate the differences between a normal shutdown and a failed startup using the life cycle annotations?

You could register your own ServletContextListener instead of the Spring's one and delegate contextInitialized method invocation to the Spring implementation you use for loading Spring context (whichever one you use, for example org.springframework.web.context.ContextLoaderListener).
Then catch and handle the desired exceptions.

Running tests involves background theads performing db operations hungs when running inside a Junit Suite

Background
We are building a product using Spring / Hibernate (JPA based) / Restful API from Jersey. During project build phase , we are running Integration tests to make sure that all the Restful API's are behaving as expected.
In recent past, we have migrated to use two entity managers which is used for another sub-product merged with this. Things become started messing up from there onwards
We are having background threads which performs few operations asynchronously. Initially we faced issues for these background jobs to get hold of their respective Entity manager factory. But we somewhat solved the issue using OpenEntityManagerInViewFilter.
As part of integration tests run , we will be bringing up Jetty server and deploy the application there for testing.
Current problem Scenario
When running those tests involves the background processing individually, they are running fine.
As we had multiple tests in place , Starting and stopping Jetty server resulted in long duration of tests run.
So we have grouped all tests inside a test suite and configured this suite inside the surefire-plugin.
The tests which have worked individually started hanging under running inside a suite.
Test Suite setup is given below.
#BeforeClass
public static void setUp()
{
// Startup of jetty server happens here
}
#AfterClass
public static void tearDown()
{
// Stopping of jetty server happens here
}
In individual tests, we will complete the background process in #BeforeClass and performing the data cleanup activities in #Afterclass method.
I am litte weird about how the tests works fine as individual starts causing issues when running inside a test suite
Any configuration / ideas would be much appreciated.

My guess is that it is one of these things:
You are not mocking something correctly, and you are changing the state of the data you are using in tests
You are doing some operations on your database that the test is not able to "repair" (e.g. drop or rename table)
You are using some static variables somewhere and these are building you a state machine that you don't expect/want
If you can say how the tests are failing, that might help

The problem seems to happen due to connections are not returned from connection pool after being used. Once we started increasing the maximum.connection.pool.size the issue stops occurring.
Thanks for all your comments

Call method after JBoss deploy

I want to have some method run, only after my WAR has been deployed to JBoss.
The problem: Currently I am using #PostConstruct to load saved schedules from the DB. The problem is that I am creating instances of the Schedulers from this method, which in turn is starting the Quartz schedulers, which is stopping JBoss from completing the deploy.
If there are no schedules to load, my WAR deploys fine, but if there are schedules they are causing the deploy to fail, because JBoss is "waiting" for the schedules to actually complete.
Is there some way to delay the method call until after it is fully deployed?
Or alternatively, is it possible to make Async calls on the Server (from the Server code)?

Java EE specification heavily refrain any thread manipulation outside facility provided by the application server.
You shouldn't in any case use a container manged thread in an infinite loop; the container expect the thread to be returned. Thread creation can still be done without too much collateral damage (if you don't put several apps on the server as the container won't be able to manage the resources between all the applications) but the any container thread must be returned.
In the new Jboss 7 there is some Java EE scheduling facilities (#Scheduling and timer, with possible persistent timer). A quick search show some example how to run Quartz in JBoss 7: How to enable Quartz scheduling in Jboss AS 7.0?
In older JBoss more advanced integration exist (JCA integration is the only standard way to get finer thread management). Use google to find them.

Wouldn't the simple ServletContextListener solve the problem for you? Just implement whatever you need in contextInitialized method.

In JBoss 7, there is a management api.
Maybe you can use it to check if the server is started (with a singleton and a TimerService).
Sample code:
ModelControllerClient client = ModelControllerClient.Factory.create(InetAddress.getByName("localhost"), 9999);
ModelNode op = new ModelNode();
op.get(ClientConstants.OP).set("read-attribute");
op.get(ClientConstants.NAME).set("server-state");
ModelNode returnVal = client.execute(op);
if(StringUtils.equals(returnVal.get("result").toString(), "\"running\"")){
LOGGER.info("Server running, init start actions");
timer.cancel();
}else{
LOGGER.info("Server not running, wait");
}