I have Spring Boot app running in embedded tomcat. There are around 50 concurrent HTTP sessions and each of them is served by 5-7 concurrently running async backend calls (#Async). There is no specific threads configuration for Tomcat or Spring Boot.
I found that long running thread (does not not matter whether it is Tomcat or async call) seriously decreases performance of other. For example, if I generate report using CR JRC which takes 20-40 seconds, most of async threads look paralyzed.
How can I optimize the code and configuration to resolve the performance issue?
From your description, there could be several bottlenecks in your configuration. But one could be the number of threads available in your system. The best you could do from here is profile your application and check what threads are available, how they are used, and where do they block.
Furthermore, assuming the number of threads is the issue, then when you say
There is no specific threads configuration for Tomcat or Spring Boot.
if it means you are running on the default ThreadPoolExecutor, then you should check the documentation and default values on how to configure your thread pool, and scale accordingly.
The #Async annotation also allows you to specify which bean Executor to use.
// use default Executor
#Async
public void asyncMethodUsingDefaultExecutor() {}
// use of Executor with qualifier specificExecutorBeanQualifier
#Async("specificExecutorBeanQualifier")
public void asyncMethodUsingSpecificExecutor() {}
You could use this to have a separated Threadpool to handle long-running tasks and another one for the others.
Related
Context
We have a Spring boot application (an API used by an angular frontend).
It is running on a docker container.
It is using a single instance of a PostgreSQL database.
Our application had some load problems so we asked us to scale it.
We told us to run our API on several docker containers for that.
We have several questions / problems dealing with code synchronization over multiple docker instances executing our code.
Problem 1
We have some #Scheduled jobs integrated and deployed with our API code.
We don't want these scheduled jobs to be executed by all container instances, but only one.
I think we can simply handle this by disabling jobs on other containers through environment variables with the "-" value to disable the Spring scheduled cron.
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/scheduling/annotation/Scheduled.html#CRON_DISABLED
Does it sounds right?
Problem 2
The other problem is that we use Spring's #Lock annotation on some repository methods.
public interface IncrementRepository extends JpaRepository<IncrementEntity, UUID> {
#Lock(LockModeType.PESSIMISTIC_FORCE_INCREMENT)
Optional<IncrementEntity> findByAnnee(String pAnneeAA);
#Lock(LockModeType.PESSIMISTIC_WRITE)
IncrementEntity save(IncrementEntity pIncrementEntity);
}
This is critical for us to have a lock on that as we get / compute an increment used to act as a unique identifier of some of our data.
If I correctly understood this locking mechanism :
if a process execute this code, the Spring JPA #Transaction will acquire a lock on the IncrementEntity (lock the database table).
when another process tries do do the same thing before the first lock has been released by the first transaction, it should have a PessimisticLockException and the second transaction will rollback
this is managed by Spring at application level, NOT directly at database level (right??)
So what will happen if we're running our code on several containers ?
app running in container 1 sets a lock
app running in container 2 execute the same code and tries to set the same lock while the first one has not been released yet
each Spring application running in different containers will probably acquire the lock without problems as they don't share the same information?
Please tell me if I correctly understood how it works, and if we will effectively have a problem running such code on several docker containers.
I guess that solution would be to set a lock directly on the database table, as we have only one instance of it?
Is there a way to easily set / release the lock at database level using Spring JPA code ?
Or perhaps I misunderstood and setting a lock using Spring's #Lock annotation sets a real database lock ?
In that case, perhaps we don't have any problem at all, as the lock is correctly set on the database itself, shared by all containers instances??
Problem 3
To avoid having too much exceptions and reject some requests trying to acquire a lock at the same time, we also added a synchronized block around the above code.
String numIncrement;
synchronized (this.mutex) {
try {
numIncrement = this.incrementService.getIncrement(var);
} catch (Exception e) {
// rethrow custom technical exception
}
}
This way concurrent requests should be delayed and queued, which is better for our users experience.
I guess that we will also have problems here as docker instances doesn't share the same JVM, so synchronization can work only in the scope of the container itself... right?
Conclusion
For all these problems, please tell me if you have some solutions to workaround / adapt our code so it can be compatible with app scaling.
Following a set of tests I can confirm these points about my original question
Problem 1
We can disable a Spring CRON with the - value
#Scheduled(cron = "-")
Problem 2
The Spring's JPa #Lock annotation sets a lock on the database itself. It is not managed by Spring software.
So when duplicating containers, if the Spring app in the first container sets a lock, the database is locked and when the second app in another container tries to get data it has the PessimisticLockException.
Problem 3
Synchronized code using the synchronized JAVA keyword is obviously managed by JVM, so there is no code mutual exclusion between containers.
Our current legacy web-app creates threads in it not managed by Application Server Containers. I have to modify it with JavaEE standards for multi-threading.
My web-app works fine on Tomcat but fails on Websphere.
Error on Websphere :
... ... Caused by: javax.naming.ConfigurationException: A JNDI operation on a "java:" name cannot be completed because the server runtime is not able to associate the operation's thread with any J2EE application component. This condition can occur when the JNDI client using the "java:" name is not executed on the thread of a server application request. Make sure that a J2EE application does not execute JNDI operations on "java:" names within static code blocks or in threads created by that J2EE application. Such code does not necessarily run on the thread of a server application request and therefore is not supported by JNDI operations on "java:" names.
at com.ibm.ws.naming.java.javaURLContextImpl.throwExceptionIfDefaultJavaNS(javaURLContextImpl.java:534) ~[com.ibm.ws.runtime.jar:?]
at com.ibm.ws.naming.java.javaURLContextImpl.throwConfigurationExceptionWithDefaultJavaNS(javaURLContextImpl.java:564) ~[com.ibm.ws.runtime.jar:?]
at com.ibm.ws.naming.java.javaURLContextImpl.lookupExt(javaURLContextImpl.java:485) ~[com.ibm.ws.runtime.jar:?]
at com.ibm.ws.naming.java.javaURLContextRoot.lookupExt(javaURLContextRoot.java:485) ~[com.ibm.ws.runtime.jar:?]
In order to resolve this issue, I am referring Concurrency Utilities in Java EE. I found similar kind of description and example for ManagedExecutorService and ManagedThreadFactory.
ManagedExecutorService: A managed executor service is used by applications to execute submitted tasks asynchronously. Tasks are
executed on threads that are started and managed by the container. The
context of the container is propagated to the thread executing the
task.
ManagedThreadFactory: A managed thread factory is used by applications to create managed threads. The threads are started and
managed by the container. The context of the container is propagated
to the thread executing the task. This object can also be used to
provide custom factories for specific use cases (with custom Threads)
and, for example, set specific/proprietary properties to these
objects.
Which one is preferred in which condition and why ?
I have solved issue by using ManagedExecutorService.
ExecutorService framework indeed has more ways to deal threads while ManagedThreadFactory can only call newThread() method.
Websphere issue can be resolved by using ManagedExecutorService or ManagedThreadFactory. Both works. But for further thread processing, ManagedExecutorService turns out lot better.
Now, this solution causes same web-app to fail on Tomcat. JNDI naming exception. As per my R&D, container based concurrency is supported in TomEE server not in Tomcat so we have to use routing mechanism to switch between code as per underlying application server.
We run several spring batch jobs within tomcat in the same web application that serves up our UI. Lately we have been adding many more jobs and we are noticing that when we patch our app, several jobs may get stuck in a STARTING or STARTED status. Many of those jobs ensure that another job is not running before they start up, so this means after we patch the server, some of our jobs are broken until we manually run SQL to update the statuses of the jobs to ABANDONED or STOPPED.
I have read here that JobScope and StepScope jobs don't play nicely with shutting down.
That article suggests not using JobScope or StepScope but I can't help but think that this is a solved problem where people must be doing something when the application exits to prevent this problem.
Are there some best practices for handling this scenario? What are you doing in your applications?
We are using spring-batch version 3.0.3.RELEASE
I will provide you an idea on how to solve this scenario. Not necessarily a spring-batch solution.
Everytime I need to add jobs in an application I do as this:
Create a table to control the jobs (queue, priority, status, etc.)
Create a JobController class to manage all jobs
All jobs are defined by the status R-running, F-Finished, Q-Queue (you can add more as you need like aborted, cancelled, etc) (the jobs control these statuses)
The jobController must be loaded only once, you can define it as a spring bean for this
Add a boolean attribute to JobController to inform if you already checked the jobs when you instantiate it. Set it to false
Check if there are jobs with the R status which means that in the last stop of the server they were running so you update every job with this R status to Q and increase their priority so it will get executed first after a restart of the server. This check is inside the if for that boolean attribute, after the check set it to true.
That way every time you call the JobController for the first time and there are unfinished jobs from a server crash you will be able to set then all to a status where it can be executed again. And this check will happens only once since you will be checking that boolean attribute.
A thing that you should be aware of is caution with your jobs priority, if you manage it wrong you may run into a starvation problem.
You can easily adapt this solution to spring-batch.
Hope it helps.
In my application I have one cron job which connects to a FTP server and transfer files, a very simple functionality and it is configured using spring #Schedule annotation with cron expression as a parameter.
It was running fine for few months and then suddenly it stopped, got the connectException.
May be the FTP server was down or something happened which causes the cron thread to stop.
I looked (google) for the reasons but didnt get any ( Nothing much in the logs also - Just the exception name ).It may be a one time thing :)
my question is that can I put some check or watcher on the #Schedule cron job to know whether it is running or not ?
Sorry for my bad explanation/english
Thanks
my question is that can I put some check or watcher on the #Schedule
cron job to know whether it is running or not ?
Basically, you can't.
When you use #Scheduled, Spring uses a ScheduledAnnotationBeanPostProcessor to register the tasks you specify (annotated methods). It registers them with a ScheduledTaskRegistrar. The ScheduledAnnotationBeanPostProcessor is an ApplicationListener<ContextRefreshEvent>. When it receives the ContextRefreshEvent from the ApplicationContext, it schedules the tasks registered in the ScheduledTaskRegistrar.
During this step, these tasks are scheduled with a TaskScheduler which typically wraps a ScheduledExecutorService. If an exception is uncaught in a submitted task, then the task is removed from the ScheduledExecutorService queue.
The TaskScheduler class does not provide a public API to retrieve the scheduled tasks, ie. the ScheduledFuture objects. So you can't use it to find out if your tasks are running or not.
And you probably shouldn't. Develop your tasks, your #Scheduled methods, to be able to withstand an exception being thrown. Some times, obviously, that's not possible. With a network error, for example, you would probably have to restart your application. Without knowing anything else about your application, I would say more logging is your best bet.
I am currently working on a project involving lots of asynchronous tasks running independently. I have one spring configuration file.
<task:executor id="taskScheduler" pool-size="5-20">
<task:executor id="specificTaskScheduler" pool-size="5-50" queue-capacity="100">
<!-- integration beans and
several object pools, with a total number of 100 beans created
using CommonsPoolTargetSource -->
I specifically created two executors - one to be used for spring integration needs and custom executor in order for it to run only my tasks feeding it to integration beans with explicit reference. After that I supplied a long running task to be procesed. My EAR runs on WebLogic and I dumped stacktrasce of threads being run and was very disappointed to find out that most of fifty threads in my custom executor wait in a executor's queue for an object to be available from the pool. I did not want CommonsPoolTargetSource to use my executor as a platform for managing its sources. What can I do here? Maybe creating a separate spring file with CommonsTargetSource beans will solve it? Thank you for any ideas.
Thanks guys. Turned out that pool was not a problem, I just had to add more instances to it and slightly increase the pool size with queue capacity set to zero and rejection policy set to an execution of the call in the caller's thread. I have yet to test it under heavy load though.