We Keep Coding

"waiting on" vs "waiting to lock" vs "locked" in java thread dumps

Looking through a thread dump I see several BLOCKED threads but some have "waiting to lock " and some have "waiting on " while there is only one RUNNABLE thread that has presumably acquired the lock. The question I have is why are some "waiting to lock" and others "waiting on"? Also the thread that is "waiting on" appears to have the lock but is not the RUNNABLE thread that actually holds the lock as it says it locked the object at address lower in the stack trace, so why is it blocking if it has the lock (or does it not have the lock)?
Examples below:
The locked object is at 0x00000000c2ad4be8
waiting on
"ajp-bio-10032-exec-15" daemon prio=10 tid=0x00007f888c02d000 nid=0x74c8 in Object.wait() [0x00007f8821792000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c2ad4be8> (a java.lang.Object)
at java.lang.Object.wait(Object.java:503)
at com.tibco.tibjms.TibjmsxLinkTcp.disconnect(TibjmsxLinkTcp.java:1026)
waiting to lock
"ajp-bio-10032-exec-95" daemon prio=10 tid=0x00007f888c089800 nid=0x1e66 in Object.wait() [0x00007f881df4b000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c2ad4be8> (a java.lang.Object)
at java.lang.Object.wait(Object.java:503)
at com.tibco.tibjms.TibjmsxLinkTcp.disconnect(TibjmsxLinkTcp.java:1026)
- locked <0x00000000c2ad4be8> (a java.lang.Object)
at com.tibco.tibjms.TibjmsConnection._close(TibjmsConnection.java:2430)
and finally the RUNNABLE thread
"ajp-bio-10032-exec-142" daemon prio=10 tid=0x00007f888c0b7000 nid=0x3657 runnable [0x00007f88167d4000]
java.lang.Thread.State: RUNNABLE
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:713)
- locked <0x00000000f022c5e0> (a com.tibco.tibjms.TibjmsxLinkTcp$LinkReader)
at com.tibco.tibjms.TibjmsxLinkTcp.start(TibjmsxLinkTcp.java:968)
- locked <0x00000000c2ad4be8> (a java.lang.Object)

Why is the java.exe is consuming more CPU due to “org.xnio.nio” threads?

We are running WildFly 12 with JDK 1.8 version. We faced the portal down / slowness issue in our web application due to high CPU usage (90% above) of java.exe. Then, we have taken a thread dump using JStack and most of the high CPU threads are pointing to "org.xnio.nio"
Portal is running in HTTPS
CPU: 8 core
Total RAM: 24 GB
Allocated JVM memory: From 1 GB (min) to 5 GB (max)
Occupaid JVM memory when taken the thread dump: 4 GB
DB is in separate machine
Below are the thread dump results (High CPU threads):
"default I/O-1" #95 prio=5 os_prio=0 tid=0x00000000260bf800 nid=0x1988 runnable [0x000000002e49f000]
java.lang.Thread.State: RUNNABLE
at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:147)
at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
at io.undertow.conduits.ReadTimeoutStreamSourceConduit$2.readReady(ReadTimeoutStreamSourceConduit.java:89)
at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1145)
at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
"default I/O-13" #108 prio=5 os_prio=0 tid=0x0000000022186000 nid=0x2310 runnable [0x000000002f09f000]
java.lang.Thread.State: RUNNABLE
at java.lang.Object.notifyAll(Native Method)
at sun.nio.ch.WindowsSelectorImpl$StartLock.startThreads(WindowsSelectorImpl.java:189)
- locked <0x00000006618ae440> (a sun.nio.ch.WindowsSelectorImpl$StartLock)
at sun.nio.ch.WindowsSelectorImpl$StartLock.access$300(WindowsSelectorImpl.java:181)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:153)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000066508ea18> (a sun.nio.ch.Util$3)
- locked <0x000000066508ea08> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000665082b98> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:551)
"default I/O-13" #108 prio=5 os_prio=0 tid=0x0000000022186000 nid=0x2310 runnable [0x000000002f09e000]
java.lang.Thread.State: RUNNABLE
at java.util.AbstractCollection.toArray(AbstractCollection.java:183)
at sun.nio.ch.Util$3.toArray(Util.java:322)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:570)
- locked <0x000000066508ea18> (a sun.nio.ch.Util$3)
- locked <0x0000000665082b98> (a sun.nio.ch.WindowsSelectorImpl)
"default I/O-5" #100 prio=5 os_prio=0 tid=0x00000000260c0000 nid=0x858 runnable [0x000000002e89e000]
java.lang.Thread.State: RUNNABLE
at org.xnio.nio.WorkerThread.run(WorkerThread.java:480)
The issue is occuring (2 - 5) days once.
Thanks in advance

Apache Tomcat Threads in WAITING state while thread pool increases Class name is ImageLoadWorker

I got this type thread dump on tomcat, All thread are in wating state.
So application is slow down.
Please suggest me the solution for that.
I am using Tomcat 7 and java 7
"ImageLoadWorker(653)" prio=5 tid=0x2089 nid=0x829 in Object.wait() - stats: cpu=0 blk=-1 wait=-1
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on org.xhtmlrenderer.swing.ImageLoadQueue#4651e7d2
at java.lang.Object.wait(Object.java:503)
at org.xhtmlrenderer.swing.ImageLoadQueue.getTask(ImageLoadQueue.java:83)
at org.xhtmlrenderer.swing.ImageLoadWorker.run(ImageLoadWorker.java:53)
Locked synchronizers: count = 0

There is thread leakage in this class of flyingsaucer jar. I got answer for this problem on below URL.
https://technotailor.wordpress.com/2017/04/17/thread-leak-with-imageloadworker-in-flying-saucer-jar/

Blocking threads using javassist in web app

Could somebody help me and explain what exactly is going on, looking at the following thread dump. It is a web application running on Tomcat 7 and we experience that some requests do not get answered:
"ajp-bio-8012-exec-161" daemon prio=10 tid=0x00007fe170603000 nid=0x344f runnable [0x00007fe174fae000]
java.lang.Thread.State: RUNNABLE
at java.security.AccessController.doPrivileged(Native Method)
at java.io.FilePermission.init(FilePermission.java:209)
at java.io.FilePermission.<init>(FilePermission.java:285)
at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
at java.io.File.exists(File.java:808)
at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1080)
at sun.misc.URLClassPath$FileLoader.findResource(URLClassPath.java:1047)
at sun.misc.URLClassPath.findResource(URLClassPath.java:176)
at java.net.URLClassLoader$2.run(URLClassLoader.java:551)
at java.net.URLClassLoader$2.run(URLClassLoader.java:549)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findResource(URLClassLoader.java:548)
at java.lang.ClassLoader.getResource(ClassLoader.java:1147)
at java.lang.ClassLoader.getResource(ClassLoader.java:1142)
at org.apache.catalina.loader.WebappClassLoader.getResource(WebappClassLoader.java:1445)
at java.lang.Class.getResource(Class.java:2142)
at javassist.ClassClassPath.find(ClassClassPath.java:84)
at javassist.ClassPoolTail.find(ClassPoolTail.java:317)
at javassist.ClassPool.find(ClassPool.java:495)
at javassist.ClassPool.createCtClass(ClassPool.java:479)
at javassist.ClassPool.get0(ClassPool.java:445)
- locked <0x00000000db6cdb48> (a javassist.ClassPool)
at javassist.ClassPool.get(ClassPool.java:414)
at javassist.compiler.MemberResolver.lookupClass0(MemberResolver.java:425)
at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:389)
at javassist.compiler.MemberResolver.lookupClassByJvmName(MemberResolver.java:310)
at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:327)
at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:314)
at javassist.compiler.Javac.compileField(Javac.java:122)
at javassist.compiler.Javac.compile(Javac.java:91)
at javassist.CtField.make(CtField.java:163)
at com.mycompany.validation.util.BeanGenerator.addProperty(BeanGenerator.java:157)
...
at java.lang.Thread.run(Thread.java:745)
"ajp-bio-8012-exec-12" daemon prio=10 tid=0x0000000000c49800 nid=0x3d99 waiting for monitor entry [0x00007fe1756b7000]
java.lang.Thread.State: BLOCKED (on object monitor)
at javassist.ClassPool.get0(ClassPool.java:432)
- waiting to lock <0x00000000db6cdb48> (a javassist.ClassPool)
at javassist.ClassPool.get(ClassPool.java:414)
at javassist.compiler.MemberResolver.lookupClass0(MemberResolver.java:425)
at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:389)
at javassist.compiler.MemberResolver.lookupClassByJvmName(MemberResolver.java:310)
at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:327)
at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:314)
at javassist.compiler.Javac.compileField(Javac.java:122)
at javassist.compiler.Javac.compile(Javac.java:91)
at javassist.CtField.make(CtField.java:163)
at com.mycompany.validation.util.BeanGenerator.addProperty(BeanGenerator.java:157)
...
at java.lang.Thread.run(Thread.java:745)
"ajp-bio-8012-exec-5" daemon prio=10 tid=0x0000000001603800 nid=0x7c77 waiting for monitor entry [0x00007fe174aaa000]
java.lang.Thread.State: BLOCKED (on object monitor)
at javassist.ClassPool.makeClass(ClassPool.java:621)
- waiting to lock <0x00000000db6cdb48> (a javassist.ClassPool)
at javassist.ClassPool.makeClass(ClassPool.java:606)
at com.mycompany.validation.util.BeanGenerator.init(BeanGenerator.java:334)
...
at java.lang.Thread.run(Thread.java:745)
I am not a specialist in thread dumps and just would like to know how to get rid of the problem.
Has the first thread locked an object (ClassPool) that two other request threads are waiting for to unlock? Why is it locked? Do I have to synchronize and how?
Thanks for any hint!

You have a single ClassPool in your application which has the monitor id 0x00000000db6cdb48.
If you see the method signature of ClassPool.get0:
protected synchronized CtClass get0(String classname, boolean useCache)
So it is synchronized. I suppose you simply used the ClassPool.getDefault(). You have to know that this method returns a single instance, so all of your calls will be sychronized.
Create a new pool for each thread (with new ClassPool(ClassPool.getDefault()) for example) and it will be fine then.
Apart from that you might check your secrurity manager, why it takes so long.

Java process doesn't exit on SIGTERM under heavy load

In normal operation my application exits fine when sent a 'kill -s SIGTERM '.
However, under load sometimes the process does not exit.
I'm just wondering if it possible that http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6392332 is the reason for this, or if it could be something else?
Here are some parts of the stack trace of the process in question, showing the shutdown methods, any help is much appreciated.
Note, this is a Java process running on 64-bit RHEL 6.3.
2013-05-22 08:01:33
Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.21-b01 mixed mode):
...
"Thread-15" prio=10 tid=0x000000001994d000 nid=0x4d5a waiting on condition [0x00007f4da08a3000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000079fd9fce8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1468)
at org.jboss.netty.util.internal.ExecutorUtil.terminate(ExecutorUtil.java:109)
at org.jboss.netty.util.internal.ExecutorUtil.terminate(ExecutorUtil.java:49)
at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.releaseExternalResources(AbstractNioWorkerPool.java:77)
at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.releaseExternalResources(NioServerSocketChannelFactory.java:164)
at com.test.services.radius.server.RadiusServerImpl.stop(RadiusServerImpl.java:87)
at com.test.services.radius.ServiceProvider.unload(ServiceProvider.java:61)
at com.test.spf.ServiceProviderCacheImpl.clearCurrentCache(ServiceProviderCacheImpl.java:150)
- locked <0x00000006b8968038> (a com.test.spf.ServiceProviderCacheImpl)
at com.test.spf.ServiceProviderCacheImpl.unload(ServiceProviderCacheImpl.java:170)
at com.test.spf.SPAImpl.stop(SPAImpl.java:178)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:273)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:199)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:487)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:463)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:480)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:463)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:480)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:463)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:480)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:463)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:431)
- locked <0x00000006da966290> (a java.util.LinkedHashMap)
at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1048)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1022)
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:970)
- locked <0x000000073ad1a920> (a java.lang.Object)
at org.springframework.web.context.ContextLoader.closeWebApplicationContext(ContextLoader.java:384)
at org.springframework.web.context.ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:78)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4245)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4886)
- locked <0x00000006b8968118> (a org.apache.catalina.core.StandardContext)
at org.apache.catalina.core.ContainerBase.removeChild(ContainerBase.java:936)
at org.apache.catalina.startup.HostConfig.undeployApps(HostConfig.java:1359)
at org.apache.catalina.startup.HostConfig.stop(HostConfig.java:1330)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:326)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1098)
- locked <0x00000006b89682f8> (a org.apache.catalina.core.StandardHost)
at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1110)
- locked <0x000000068e63ff10> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.core.StandardEngine.stop(StandardEngine.java:468)
at org.apache.catalina.core.StandardService.stop(StandardService.java:604)
- locked <0x000000068e63ff10> (a org.apache.catalina.core.StandardEngine)
at org.apache.catalina.core.StandardServer.stop(StandardServer.java:788)
at org.apache.catalina.startup.Catalina.stop(Catalina.java:662)
at org.apache.catalina.startup.Catalina$CatalinaShutdownHook.run(Catalina.java:706)
"SIGTERM handler" daemon prio=10 tid=0x0000000025453000 nid=0x4d58 in Object.wait() [0x00007f4da0b2f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000072c10db20> (a org.apache.catalina.startup.Catalina$CatalinaShutdownHook)
at java.lang.Thread.join(Thread.java:1258)
- locked <0x000000072c10db20> (a org.apache.catalina.startup.Catalina$CatalinaShutdownHook)
at java.lang.Thread.join(Thread.java:1332)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
at java.lang.Shutdown.runHooks(Shutdown.java:123)
at java.lang.Shutdown.sequence(Shutdown.java:167)
at java.lang.Shutdown.exit(Shutdown.java:212)
- locked <0x0000000707855738> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:52)
at sun.misc.Signal$1.run(Signal.java:212)
at java.lang.Thread.run(Thread.java:722)

No Unix process will finish on SIGTERM till all threads are finished properly, thus returned from run method. High load could have its roots in deadlock - deadlocked threads usually never ends. Or endless loop in some thread.
Btw proper way of shutting down the Tomcat is to use bundled shutdown script. It could fail in some extraordinary situations, but then you can just SIGKILL it.
SIGTERM or SIGKILL, that's often the business question. Proper shutdown can easily take more than 15min, when complex application, swapped... a.s.o. So can you stand 15min outage or rather you'll kill it and start in next 2min?