We Keep Coding

Threads are increasing but not decreasing when the task finished in Java

I have an application that I did thread configuration as follows,
corepoolsize = 10
maxpoolsize = 10
queuecapacity = 10
But when I do use Threads and functions that run on threads (asynchronous process)
I see it is increasing like the follows, But I don't see the threads are decreasing. Is that a indication of memory leak? Can anybody guide me?
Log prints like this,
2021-09-15 01:13:48.554 INFO 111 --- [Async-1]
2021-09-15 01:13:48.654 INFO 121 --- [Async-2]
2021-09-15 01:13:48.754 INFO 132 --- [Async-3]
2021-09-15 01:13:48.854 INFO 140 --- [Async-4]
2021-09-15 01:13:48.954 INFO 155 --- [Async-5]
2021-09-15 01:13:49.554 INFO 160 --- [Async-6]
But I don't see it is calling again [Async-1], [Async-2] like that. Is this a memory leak or a thread cache or after filling up corepoolsize then will it run [Async-1]?

Corepoolsize represents the worker thread that is always alive.
You can check ThreadPoolExecutor#runWorker, ThreadPoolExecutor#getTask method, these worker threads will always try to get tasks from the task queue and execute them. So if no new tasks are posted to the thread pool, then these worker threads will always live, usually blocked in the take method of workQueue, So you don't have to worry about the cpu idling.
Worker threads outside the number of corepoolsize will be recycled, still ThreadPoolExecutor#getTask method:
try {
Runnable r = timed ?
workQueue.poll(keepAliveTime, TimeUnit.NANOSECONDS) :
workQueue.take();
if (r != null)
return r;
timedOut = true;
} catch (InterruptedException retry) {
timedOut = false;
}
These threads to be recycled can survive at most keepAliveTime time.
The thread pool uses a Hash table to hold the thread's reference, so that the life cycle of the thread can be controlled by operations such as adding and removing references.(ThreadPoolExecutor#processWorkerExit method).

This is expected behaviour. The documentation for ThreadPoolExecutor explicitly states:
When a new task is submitted in method execute(Runnable), if fewer than corePoolSize threads are running, a new thread is created to handle the request, even if other worker threads are idle.

TaskRejectedException throw and thread pool is not full

I use spring thread pool to manage the threads in my project.
But there is some thing wrong when my code is running.
I get exceptions like:
org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPoolExecutor#4cfc01ab[Running, pool size = 200, active threads = 0, queued tasks = 40, completed tasks = 7990]] did not accept task: java.util.concurrent.FutureTask#6ba9fcd5
It throws TaskRejectedException when the pool "active threads" is zero.
I have read the documentation and the source code from Spring but have found nothing.
My TaskExecutor class:
#Bean
public ThreadPoolTaskExecutor taskExecutor() {
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setKeepAliveSeconds(200);
taskExecutor.setCorePoolSize(200);
taskExecutor.setMaxPoolSize(200);
taskExecutor.setQueueCapacity(40);
taskExecutor.setRejectedExecutionHandler(new
ThreadPoolExecutor.AbortPolicy());
return taskExecutor;
}
active threads sometime is 0 sometime is 10 , never 200 , weird.

In the exception message it has "queued tasks = 40". Since in your setup you have setQueueCapacity(40), executing a thread when queued tasks is already at 40 will throw that exception.
If you are submitting tasks very quickly I would try increasing the queue capacity. Threads do not immediately move from the queue to active, especially when the JVM is busy.
As far as active thread count, it is only an approximation. Java doc of ThreadPoolExecutor.getActiveThreadCount:
/**
* Returns the approximate number of threads that are actively
* executing tasks.
*
* #return the number of threads
*/
public int getActiveCount() {

Lettuce RedisCache throws java.util.concurrent.RejectedExecutionException Thread limit exceeded replacing blocked worker

I am using Spring Boot with Redis-Cache, with Lettuce default configuration, and receiving the following RejectedExecutionException after the server has been up for a few minutes:
org.springframework.data.redis.RedisSystemException: Unknown redis exception; nested exception is java.util.concurrent.RejectedExecutionException: Thread limit exceeded replacing blocked worker
at org.springframework.data.redis.FallbackExceptionTranslationStrategy.getFallback(FallbackExceptionTranslationStrategy.java:53)
at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:43)
at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:257)
at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.convertLettuceAccessException(LettuceStringCommands.java:718)
at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.get(LettuceStringCommands.java:63)
at org.springframework.data.redis.connection.DefaultedRedisConnection.get(DefaultedRedisConnection.java:210)
at org.springframework.data.redis.cache.DefaultRedisCacheWriter.lambda$get$1(DefaultRedisCacheWriter.java:109)
at org.springframework.data.redis.cache.DefaultRedisCacheWriter.execute(DefaultRedisCacheWriter.java:242)
at org.springframework.data.redis.cache.DefaultRedisCacheWriter.get(DefaultRedisCacheWriter.java:109)
at org.springframework.data.redis.cache.RedisCache.lookup(RedisCache.java:82)
at org.springframework.cache.support.AbstractValueAdaptingCache.get(AbstractValueAdaptingCache.java:58)
at org.springframework.cache.interceptor.AbstractCacheInvoker.doGet(AbstractCacheInvoker.java:73)
at org.springframework.cache.interceptor.CacheAspectSupport.findInCaches(CacheAspectSupport.java:525)
at org.springframework.cache.interceptor.CacheAspectSupport.findCachedItem(CacheAspectSupport.java:490)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:372)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:316)
at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:61)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
at my.controller.MyController.lambda$myFunc$0(MyController.java:60)
at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1386)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.util.concurrent.RejectedExecutionException: Thread limit exceeded replacing blocked worker
at java.util.concurrent.ForkJoinPool.tryCompensate(ForkJoinPool.java:2011)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3310)
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1775)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at io.lettuce.core.protocol.AsyncCommand.await(AsyncCommand.java:81)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:112)
at io.lettuce.core.FutureSyncInvocationHandler.handleInvocation(FutureSyncInvocationHandler.java:62)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy168.get(Unknown Source)
at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.get(LettuceStringCommands.java:61)
... 21 common frames omitted
It seems that Lettuce uses the common ForkJoinPool, that the DeferredResult uses as well, and all the requests and connections are choking the pool (please correct me if I'm wrong). What is the recommended approach? Should I move Lettuce to use a different pool? If so how? Please let me if there is any other configuration or other information that I can provide.

Lettuce uses Netty's EventLoop's as its threading infrastructure.
What here happens is that your task is executed on a ForkJoin pool. Lettuce uses CompletableFutures to return a handle for an async result processing and the synchronous API calls CompletableFuture.get(timeout, TimeUnit) to await command completion. Calling a blocking method on a ForkJoin pool involves the ManagedBlocker to potentially switch to a different thread that can proceed with work.
If too many threads await command completion, you eventually end up with RejectedExecutionException.
I suggest using a different execution pool for the lambda that you're invoking.

ThreaPool with LinkedBlockingQueue reject task when queue is empty

I created a thread pool with following code
ThreadPoolExecutor backgroupTaskPool = new ThreadPoolExecutor(100, 100, 10, TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(100));
backgroupTaskPool.allowCoreThreadTimeOut(true);
when submit tasks to this pool, it throws RejectedExecutionException, but the threads in the pool not reaches maximumPoolSize and the LinkedBlockingQueue is empty:
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask#15e9aebd rejected from java.util.concurrent.ThreadPoolExecutor#7774348b[Shutting down, pool size = 9, active threads = 9, queued tasks = 0, completed tasks = 105]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
Do you have any idea why could this happen?
THX.

Based on the exception message
Task java.util.concurrent.FutureTask#15e9aebd rejected from java.util.concurrent.ThreadPoolExecutor#7774348b[Shutting down, pool size = 9, active threads = 9, queued tasks = 0, completed tasks = 105]
You seem to be shutting down the ThreadPoolExecutor. You can't submit tasks after you've called shutdown.

Transaction is alternating Timeouts

I am using jboss 5.1.x, EJB3.0
I have MDB which listens to JMS queue. when the MDB taking a message, it dispatch a msg via TCP to some modem.
sometimes that Modem doesnt response when the server is waiting for an answer:
byte[] byteData = receive(is);
coz I cant set timeout on InputStream.
so thanks to the EJB container the transaction timeout(which is there by default) rolling back the operation and then a retry executed again.
this mechanism by default works fine for me, the problem is:
Sometimes the transaction never timed out, and after long time I get the following
msg in the console:
15:18:22,578 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.TransactionReaper_18] - TransactionReaper::check timeout for TX a6b2232:5f8:4d3591c6:76 in state RUN
15:18:22,578 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id a6b2232:5f8:4d3591c6:76 invoked while multiple threads active within it.
15:18:22,578 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action a6b2232:5f8:4d3591c6:76 aborting with 1 threads active!
15:18:22,578 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.TransactionReaper_7] - TransactionReaper::doCancellations worker Thread[Thread-10,5,jboss] successfully canceled TX a6b2232:5f8:4d3591c6:76
Any idea what's wrong? and why sometimes it work and sometimes it doesnt?
thanks,
ray.

JBossAS that uses the Arjuna's Transaction Manager. In EJB3 interceptor chain would begin to unroll and eventually hit the transaction manager interceptors whose job it is to abort the transaction.
For MDB's you can annote it with #ActivationConfigProperty(propertyName="transactionTimeout" value="1500")
For other beans you can have #TransactionTimeout(1500) at class level or method level.
When the transaction manager detects that the transaction has timed out and then aborts it from within an asynchronous thread (different from the thread running in method), but it never sends an interrupt to the currently running thread.
Therefore resulting in : invoked while multiple threads active within it ... aborting with 1 threads active!
Edit :
//---
ThreadGroup root = Thread.currentThread().getThreadGroup().getParent();
while (root.getParent() != null)
root = root.getParent();
findAllThread(root,0);
//---
public static findAllThread(ThreadGroup threadGroup, int level){
int actCount = threadGroup.activeCount();
Thread[] threads = new Thread[actCount*2];
actCount = threadGroup.enumerate(threads, false);
for (int i=0; i<actCount; i++) {
Thread thread = threads[i];
thread.interrupt();
}
int groupCount = threadGroup.activeGroupCount();
ThreadGroup[] groups = new ThreadGroup[numGroups*2];
groupCount = threadGroup.enumerate(groups, false);
for (int i=0; i<groupCount; i++)
findAllThread(groups[i], level+1);
//---
It will list other active threads also like Reference Handler, Finalizer, Signal Dispatcher etc.