I have been working on an old project where both Spring scheduler is enabled (#Scheduled actively being used) also some native JDK thread pool instances active too. In the project configuration xml I see below;
<task:scheduler id="taskScheduler" pool-size="${task-scheduler.pool-size}"/>
<task:executor id="taskExecutor" pool-size="${task-executor.pool-size}" queue-capacity="${task-executor.queue-capacity}"/>
<task:annotation-driven executor="taskExecutor" scheduler="taskScheduler"/>
And them some quartz implementation comes arise with its own job definition, trigger definition stuff where the Jobs defines their own ThreadPoolExecutors as below,
BlockingQueue<Runnable> workerTaskQueue = new ArrayBlockingQueue<Runnable>(poolSize*3);
threadPoolExecutor = new
ThreadPoolExecutor(poolSize,poolSize,1000L,TimeUnit.MILLISECONDS,workerTaskQueue);
then starts to submit tasks (Runnables) into the pool.
threadPoolExecutor.execute(new ImcpWorker(task, this, workerTaskUtil));
But what I see is that at some point the Spring task rejection exception thrown for these tasks. This is nonsense (unless Spring intercepts using AOP the thread pool executors, even if they are created natively). Because there is no spring managed executor.
2021-06-21 11:51:58,679 ERROR [pool-151-thread-81] LisJobHandler -
Exception occured: Executor
[java.util.concurrent.ThreadPoolExecutor#5532827b[Running, pool size =
1000, active threads = 1000, queued tasks = 100000, completed tasks =
135592411]] did not accept task:
org.springframework.aop.interceptor.AsyncExecutionInterceptor$1#5a237108
msisdn:5363443640 org.springframework.core.task.TaskRejectedException:
Executor [java.util.concurrent.ThreadPoolExecutor#5532827b[Running,
pool size = 1000, active threads = 1000, queued tasks = 100000,
completed tasks = 135592411]] did not accept task:
org.springframework.aop.interceptor.AsyncExecutionInterceptor$1#5a237108
So again the question, does spring scheduler and executors (if configured) intercepts ThreadPoolExecutors in an application ?
Well the issue was out of nowhere about my initial assumption. As I go deeper in spring debugging I see that the queue task submission comes from one of my other bean. It has async task registrations and each error in the app calls this async method to trigger some custom actions. So that once an endpoing failed really bad and can not recover, this issue occurs. Because the current design keeps calling the async method and each call occupies a slot in the executor's pool.
Related
I have a Spring Boot / Spring Integration application running that makes use of #Poller in Spring Integration and also #Scheduled on another method in a mostly-unrelated class. The #Poller is for polling an FTP server for new files. However I've found that it seems like the #Poller is somehow interfering with my #Scheduled method.
The #Poller has maxMessagesPerPoll = -1 so that it will process as many files as it can get. However, when I first start my application, there are over 100 files on the FTP server, so it's going to process them all. What I have found is that, if these files are being processed, then the #Scheduler stops triggering at all.
For example, if I set my #Scheduled to have a fixedDelay = 1 to trigger every millisecond and then start my application, the #Scheduled method will trigger a few times, until the #Poller triggers and begins processing messages, at which point my #Scheduled method completely stops triggering. I assumed that simply there was some task queue that was being filled by the #Poller so I simply needed to wait for all of the messages to be processed, but even after the #Poller is completely done and has processed all of the files, the #Scheduled method still does not trigger at all.
My thoughts are that maybe there is some task queue that is being filled by the #Poller, which is breaking my #Scheduled method, but if so, I still don't see any way that I can use a separate task queue for the different methods, or any other possible options for customizing or fixing this issue.
Does anyone have any idea what might be happening to my #Scheduled method, and how can I fix this?
#Poller:
#Bean
#InboundChannelAdapter(channel = "ftpChannel", poller = #Poller(cron = "0/5 * * ? * *", maxMessagesPerPoll = "-1"))
public MessageSource<InputStream> myMessageSource() {
//Build my message source
return messageSource;
}
#Scheduled:
#Scheduled(fixedDelay = 6000)
public void myScheduledMethod(){
//Do Stuff
}
They do use the same bean name for their scheduler taskScheduler.
It should only be a problem if you have 10 or more pollers (the default scheduler bean configured by Spring Integration has a pool size of 10 by default). A common mistake is having many queue channels (which hold on to scheduler threads for a second at a time, by default).
If you only have one poller, and not a lot of queue channels, I can't explain why you would get thread starvation.
You can increase the pool size - see Configuring the Task Scheduler.
Or you can use a different scheduler in the ScheduledAnnotationBeanPostProcessor.
As already pointed out, the problem is linked to task schedulers having the same name, although it may occur even if there are fewer than 10 pollers. Spring Boot auto-configuration provides scheduler with default pool size of 1 and registration of this scheduler may happen before the registration of taskScheduler, provided by Spring Integration.
Configuring task scheduler via Spring Integration properties doesn't help as this bean doesn't get registered at all. But providing own instance of TaskScheduler with adjusted pool size, changing pool size of auto-configured scheduler via spring.task.scheduling.pool.size property or excluding TaskSchedulingAutoConfiguration should solve the issue.
In our case, the Poller was used by inbound-channel-adapter to access mail from the IMAP server - but when it polls for an email with large attachments, it blocks the thread used by #Scheduled as it only uses a single thread for scheduling the task.
So we set the Spring property spring.task.scheduling.pool.size=2 - which now allows the #Scheduled method to run in a different thread even if the poller gets blocked (in a different thread) when trying to fetch mail from IMAP server
When using Spring webflux with Mono or Flux return type, the http connecting thread is parked/released while the connection waits for the response. Thus, the connection is not taking max-connections.
Question: how can I test/proof that the connection is really released while waiting for the response, and not blocking max-connections?
I already enabled DEBUG logging, but that did not show anything regarding this question.
#RestController
public class MyServlet {
#GetMapping("/")
public Mono<String>/Flux<String> test() {
return Mono.just(service.blocking());
}
}
#Service
public class SlowService {
public String blocking() {
TimeUnit.SECONDS.sleep(10);
return "OK";
}
}
Or is that incorrect at all, and I'd have to use:
Mono.fromCallable(() -> service.blocking()).subscribeOn(Schedulers.elastic());
But still, how can I see from the logs that the connection gets parked correctly?
To test, I'm using server.tomcat.max-threads=5. I'm trying to rewrite my blocking service so that those threads are not blocked during the sleep, and thus more than 5 connections can reach my service concurrently.
There are 2 thread pools, the forkjoin pool that will handle all the regular work. Then you have the Schedular pool that will handle scheduled tasks.
return Mono.just(service.blocking());
This will block one of the threads in the ForkJoinPool so less events can be handled by webflux hence slowing down your service.
Mono.fromCallable(() -> service.blocking()).subscribeOn(Schedulers.elastic());
This will assign the task and "offload" it to the scheduler pool of threads so another thread pool will handle this and not hog one of the ForkJoinPool threads.
How to test this? You need to load test your service, or as most people do, trust the framework to do what it is set out to do and trust that the spring team has tested their side of things.
I have used Spring Framework's Scheduled to schedule my job to run at every 5 mins using cron. But sometime my job waits infinitely for an external resource and I can't put timeout there. I can't use fixedDelay as previous process sometime goes in wait infinitely mode and I have to refresh data at every 5 mins.
So I was looking any option in Spring Framework's Scheduled to stop that process/thread after a fixed-time either it run successfully or not.
I have found below setting which initialized ThreadPoolExecutor with 120 seconds for keepAliveTime which I put in #Configuration class. Can anybody tell me will this work as I expected.
#Bean(destroyMethod="shutdown")
public Executor taskExecutor() {
int coreThreads = 8;
int maxThreads = 20;
final ThreadPoolExecutor threadPoolExecutor = new ThreadPoolExecutor(
coreThreads, maxThreads, 120L,
TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>()
);
threadPoolExecutor.allowCoreThreadTimeOut(true);
return threadPoolExecutor;
}
I'm not sure this will work as expected. Indeed the keepAlive is for IDLE thread and I don't know if your thread waiting for resources is in IDLE. Furthermore it's only when the number of threads is greater than the core so you can't really know when it happen unless you monitor the threadpool.
keepAliveTime - when the number of threads is greater than the core, this is the maximum time that excess idle threads will wait for new tasks before terminating.
What you can do is the following:
public class MyTask {
private final long timeout;
public MyTask(long timeout) {
this.timeout = timeout;
}
#Scheduled(cron = "")
public void cronTask() {
Future<Object> result = doSomething();
result.get(timeout, TimeUnit.MILLISECONDS);
}
#Async
Future<Object> doSomething() {
//what i should do
//get ressources etc...
}
}
Don't forget to add #EnableAsync
It's also possible to do the same without #Async by implementing a Callable.
Edit: Keep in mind that it will wait until timeout but the thread running the task won't be interrupted. You will need to call Future.cancel when TimeoutException occurs. And in the task check for isInterrupted() to stop the processing. If you are calling an api be sure that isInterrupted() is checked.
allowCoreThreadTimeOut and timeout setting doesn't help cause it just allow work thread to be ended after some time without work (See javadocs)
You say your job waits infinitely for an external resource. I'am sure it's because you (or some third-party library you using) use sockets with time out infinite-by-default.
Also keep in mind what jvm ignores Thread.interrupt() when it blocked on socket.connect/read.
So find out witch socket library used in your task (and how exactly it used) and change it's default timeout settings.
As example: there is RestTemplate widely used inside Spring (in rest client, in spring social, in spring security OAuth and so on). And there is ClientHttpRequestFactory implementation to create RestTemplate instances. By default, spring use SimpleClientHttpRequestFactory which use JDK sockets. And by default all it's timeouts are infinite.
So find out where exactly you freeze, read it's docs and configure it properly.
P.S. If you don't have enough time and "feeling lucky" try to run your app with setting jvm properties sun.net.client.defaultConnectTimeout and
sun.net.client.defaultReadTimeout to some reasonable values (See docs for more details)
The keepAliveTime is just for cleaning out worker threads that hasn't been needed for a while - it doesn't have any impact on the execution time of the tasks submitted to the executor.
If whatever is taking time respects interrupts you can start a new thread and join it with a timeout, interrupting it if it doesn't complete in time.
public class SomeService {
#Scheduled(fixedRate = 5 * 60 * 1000)
public void doSomething() throws InterruptedException {
Thread taskThread = new TaskThread();
taskThread.start();
taskThread.join(120 * 000);
if(taskThread.isAlive()) {
// We timed out
taskThread.interrupt();
}
}
private class TaskThread extends Thread {
public void run() {
// Do the actual work here
}
}
}
I have many I/O intensive jobs that are triggerred via Jersey webservice calls like this:
https://localhost/rest/execute/job/job1
I want to globally control the number of threads these jobs are using. How do I do this? I have thought of following two options, please suggest if I am in right direction or if there is a better solution.
Approach 1:
Create a wrapper class over ThreadPoolExecutor that provides threads to various services to submit runnables.
class GlobalThreadPool {
int corePoolSize = 1;
int maximumPoolSize = 4;
BlockingQueue<Runnable> q = BlockingQueue<Runnable>(10);
private ExecutorService pool = ThreadPoolExecutor(corePoolSize, maximumPoolSize, ..., q);
public Future<?> run (Runnable task) { pool.submit(task); }
}
Then use this class as ServletContextListener to start and shutdown with webservice.
(+) REST resource gets an instance of this class from ServletContext and submits the requested job.
(-)If a submitted task wants to use additional thread, it won't be able to do that as the run() method won't have access to service context.
Approach 2:
Create a Singleton of GlobalThreadPool. Then we Will not start it via web services listener. However, whenever a job needs it, it will instantiate the class and submit the runnable.
(+) any job has access to the pool irrespective of whether it has access to ServletContext
(-) shutting down the ExecutorService is not tied with webservice
Do you see any particular problem in any of these approaches that I might be missing? Is there any better (read standard) approach to do these things?
We have a service implementation in our Spring-based web application that increments some statistics counters in the db. Since we don't want to mess up response time for the user we defined them asynchronous using Spring's #Async:
public interface ReportingService {
#Async
Future<Void> incrementLoginCounter(Long userid);
#Async
Future<Void> incrementReadCounter(Long userid, Long productId);
}
And the spring task configuration like this:
<task:annotation-driven executor="taskExecutor" />
<task:executor id="taskExecutor" pool-size="10" />
Now, having the pool-size="10", we have concurrency issues when two threads try two create the same initial record that will contain the counter.
Is it a good idea here to set the pool-size="1" to avoid those conflicts? Does this have any side affects? We have quite a few places that fire async operations to update statistics.
The side-effects would depend on the speed at which tasks are added to the executor in comparison to how quickly a single thread can process them. If the number of tasks being added per second is greater than the number that a single thread can process in a second you run the risk of the queue increasing in size over time until you finally get an out of memory error.
Check out the executor section at this page Task Execution. They state that having an unbounded queue is not a good idea.
If you know that you can process tasks faster than they will be added then you are probably safe. If not, you should add a queue capacity and handle the input thread blocking if the queue reaches this size.
Looking at the two examples you posted, instead of a constant stream of #Async calls, consider updating a JVM local variable upon client requests, and then have a background thread write that to the database every now and then. Along the lines of (mind the semi-pseudo-code):
class DefaultReportingService implements ReportingService {
ConcurrentMap<Long, AtomicLong> numLogins;
public void incrementLoginCounterForUser(Long userId) {
numLogins.get(userId).incrementAndGet();
}
#Scheduled(..)
void saveLoginCountersToDb() {
for (Map.Entry<Long, AtomicLong> entry : numLogins.entrySet()) {
AtomicLong counter = entry.getValue();
Long toBeSummedWithTheValueInDb = counter.getAndSet(0L);
// ...
}
}
}