I have a threaddump of an app which showed 3 threads like below.
===============
"http-443-11" daemon prio=10 tid=0x00000000473bc800 nid=0x3590 waiting on condition [0x0000000061818000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007612a3880> (a java.util.concurrent.Semaphore$NonfairSync)
"http-443-4" daemon prio=10 tid=0x00000000451f6000 nid=0x243a waiting on condition [0x0000000055354000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007612a3880> (a java.util.concurrent.Semaphore$NonfairSync)
"http-443-7" daemon prio=10 tid=0x000000004602e000 nid=0x2974 waiting on condition [0x000000005e6e7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007612a3880> (a java.util.concurrent.Semaphore$NonfairSync)
===============
What is the significance of the "waiting on condition []" ? What does the number in the [] signifiy ?
In the thread stack we can see that threads are daemon threads and are waiting for task to come. Since these threads are created on JVM startup , they dont get killed unless JVM exits or any non daemon thread i not running, thus they wait for tasks to come. Say Garbage collection thread is a daemoon thread which might not be running all the time, it could be in waiting state.
Related
I have take thread dumps of my application deployed in production, which uses logback. I am NOT expert in analysing thread dumps, however, I am required to do so. I am learning it, and have read some online articles.
The below is the real thread dumps:
"logback-8" #136 daemon prio=5 os_prio=0 tid=0x00007f3588001000 nid=0x13a waiting on condition [0x00007f35f8e7b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000068d740338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"logback-7" #135 daemon prio=5 os_prio=0 tid=0x00007f356c030800 nid=0x139 waiting on condition [0x00007f35f8d7a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000068d740338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"logback-6" #134 daemon prio=5 os_prio=0 tid=0x00007f357c018000 nid=0x138 waiting on condition [0x00007f3568ef0000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000068d740338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"logback-5" #133 daemon prio=5 os_prio=0 tid=0x00007f36b8003800 nid=0x137 waiting on condition [0x00007f35f9380000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000068d740338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
As I can see, most of these threads are in WAITING state.
But how do i know what exactly are they waiting for? Are these all threads trying to "wait" on "something"?
I did read some online materials , but when seeing a real thread dumps, it is not making any sense to me as how to take steps in understanding, as to what exactly these threads are trying to do, and at what thing are they WAITING, BLOCKED etc.
Can anyone please help me understand this?
As far as I have analyse thread dumps, When you are using such frameworks like Spring or other library like you have used here Logback. They create thread pool based on your configuration. For example, you can find such configuration in applicationcontext.xml or any of your java based configuration class. So, what it does is, it will create that much of threads at application startup or first initialisation call. If any task comes, then framework will pick one thread from pool and assign it. After completion of the task thread will come back in the pool and at that time the thread status would be like WAITING (parking) . This does not mean that it is block by any thread. It will not cause any performance headache if your server is capable enough to handle such pool. They are just sitting ideal because they do not have any task to do.
You can find more info on this and if you want to see visual way of analysis of your thread dump then you can use this. I found this site very helpful. Just upload your dump file.It will tell every thing that needed to be known by that thread dump.
Based on the above stack trace, these are the following inferences
There is a queue Q.
N number of jobs(Runnables) are added to the queue.
These jobs will be taken from the queue and executed by the threads
in the ThreadPoolExecutors(It can be cached/Dynamic).
Lets assume, we have a thread pool Executor with only two threads(Fixed Size) and you have submitted only one job to the queue.(Job1)
Thread1 takes the job1 from the queue and execute. Once the execution is over, It goes to the WAITING state because there are no elements in the queue. It has no work to do.
Presence of waiting threads on Stack trace doesn't imply they are harmful.
First thing you may look to do is to search for Runnable thread and the synchronizers that it has locked.
Runnable thread if hangs on for a long time there is a great chance for it to be performing IO or DB operations. Take subsequent thread dumps with Interval of 30 seconds or so and do a comparative study.
I have jstack where many thread are in state WAITING with description "parking to wait for " like:
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base#11.0.3/Native Method)
- parking to wait for <0x0000000307db96c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base#11.0.3/LockSupport.java:194)
What is this big hex number? Is it time? Is it identifier?
EDIT
I have dumped state of my Java application thereads with threads that work very long (few days) in the morning and in the afternoon. I see that "waiting on condition" is with the same big hex number, but other big hex number in "parking to wait for" is different:
In the morning:
"qtp792232038-1037-..." #1037 prio=5 os_prio=0 cpu=787.64ms elapsed=528768.56s tid=0x00007f164004a800 nid=0x1346
waiting on condition [0x00007f181fffd000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base#11.0.3/Native Method)
- parking to wait for <0x000000030a69c410> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base#11.0.3/LockSupport.java:194)
...
After few hours:
"qtp792232038-1037-..." #1037 prio=5 os_prio=0 cpu=787.64ms elapsed=546900.36s tid=0x00007f164004a800 nid=0x1346
waiting on condition [0x00007f181fffd000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base#11.0.3/Native Method)
- parking to wait for <0x0000000307db96c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base#11.0.3/LockSupport.java:194)
...
Yes, it is an internal identifier for the lock object.
You can use this to see which thread is waiting for which other thread.
Search the thread dump for this id, there should be another thread mentioned with a stack frame that is holding the lock with the same id.
If we will google something like 'java thread state' we will see approximately this diagram:
But if we will open jVisualVm we will see following:
Can you help to meatch these diagrams?
Sleeping state is just Thread.sleep()? Special case of the Running?
What the Park state?(I tried to google but I confused because I knew before only first diagram)
The diagram represents java.lang.Thread.State enum. The Javadoc is quite helpful to get an understanding of the mapping you seek.
The JVisualVM state represent the extra state description you would see in a thread dump, e.g.:
"Finalizer" daemon prio=8 tid=0x022f4000 nid=0xd14 in Object.wait() [0x044cf000]
java.lang.Thread.State: WAITING (on object monitor)
So you could decipher the state on your own, if you get a thread dump and compare the state from JVisualVM and the thread dump by a thread name.
Here is the mapping you want:
Running -> java.lang.Thread.State: RUNNABLE
Sleeping -> java.lang.Thread.State: TIMED_WAITING (sleeping)
Wait -> java.lang.Thread.State: WAITING TIMED_WAITING (on object monitor)
Park -> java.lang.Thread.State: WAITING TIMED_WAITING (parking)
Monitor -> java.lang.Thread.State: BLOCKED (on object monitor)
The Park state is a special case of WAITING or TIMED_WAITING. The difference from Wait is that Wait happens on an object monitor (i.e. Object.wait() within a synchronized block). The Park, on the other hand, removes a thread from scheduling via Unsafe.park without any need of holding a monitor (i.e. it doesn't need a synchronized block).
park is
sun.misc.Unsafe.park()
If I'm using more than one GenericKeyedObjectPool in an application with enabled asynchronous idle object eviction how many "idle object eviction" thread will run in the background?
Do multiple GenericKeyedObjectPools create only one eviction thread or do they create separate threads for every pool?
The current implementation (v1.6) uses a static timer, so in practice multiple pools use only one eviction thread. (Assuming that they are loaded into the same classloader.) You can check it with jstack, there is only one timer thread:
"Timer-0" daemon prio=10 tid=0x7bce5000 nid=0x1ca5 in Object.wait() [0x7b23d000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xa26c0fe8> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0xa26c0fe8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
Locked ownable synchronizers:
- None
My tomcat server is responding slowly frequently and dying out after some time. This is happening quite often once a week. I took a thread dump and it shows about 50% of the threads are in locked state. Class which is locking the thread is org.apache.tomcat.util.net.AprEndpoint
I think over the time number of thread locking is rising and finally taking up all threads in locked state.
Here is one statement from the thread dump which shows locked state
"http-8080-4" daemon prio=6 tid=0x000000006acca800 nid=0x9d8 in Object.wait() [0x000000007083f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000028ead838> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1511)
- locked <0x0000000028ead838> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1536)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None