We Keep Coding

Strange bug with wait statement in synchronized block

An excerpt from Worker.java file:
public class Worker extends Thread{
public void run(){
// Worker Thread periodically does its job.
Master.getInstance().decrementNumOfWorkingWorkers();
// This is the reporting part of the thread.
// Aimed to wait other threads finish their job.
synchronized (Master.getInstance().allFinished) {
while ( Master.getInstance().getNumOfWorkingWorkers() > 0) {
try {
Master.getInstance().allFinished.wait();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Main.printSync("Worker Thread-" + getPId() + " worked on");
}
}
}
This is from Master.Java:
import java.util.LinkedList;
import java.util.Timer;
import java.util.TimerTask;
public class Master extends Timer {
AllFinished allFinished;
int day;
public TimerTask task;
LinkedList<Worker> Workers;
private static Master instance = null;
int numOfWorkingWorkers = 0;
public class AllFinished
{
}
public class PeriodicIncrement extends TimerTask {
// Complete this class
public void run() {
Main.printSync("Day " + day + ":");
Main.printSync("Queue: " + TaskQueue.getInstance().ConvertToString());
day++;
for (int i = 0; i < Workers.size(); i++) {
synchronized (Workers.get(i)) {
Workers.get(i).notify();
}
}
if (0 == numOfWorkingWorkers) {
synchronized (allFinished) {
allFinished.notifyAll();
}
cancel(); // Terminate the timer thread
}
}
}
private Master(LinkedList<Worker> Workers) {
super();
this.task = new PeriodicIncrement();
day = 0;
allFinished = new AllFinished();
this.Workers = Workers;
numOfWorkingWorkers = this.Workers.size();
this.schedule(task, 100, 100);
}
}
For a test with 4 worker threads, everything was fine before I added the excerpted part in Worker.java. Then, to report each worker's action after all workers are done, I added that part. Algorithm is very simple. When a worker finishes its job, it checks if there is any job in a TaskQueue and ProductOwner. If there are not any, it breaks its loop and then decrement by 1 active worker threads counter in Master and then calls wait on AllFinished field of Master. run() method of PeriodicIncrement checks this counter and if it is 0 (meaning all workers finished their job), it calls notifyAll() on AllFinished .
Problem is, one sometimes two threads are entering that excerpted code block in Worker.java but remainings never enter so active worker threads counter never decremented to 0 and my program never finishes.
If I just comment out the excerpted part in Worker.java, except finishing and reporting randomly, everything is fine. What I mean is the excerpted part seems to be problematic.
Could you help me to find out?

That was a fun one to debug, after so much time not playing with low-level concurrency primitives. The trick to root-cause this was to use the jstack tool provided by the JDK.
╭───courtino ~
╰➤ sudo jstack -l 63978
2020-03-29 21:26:01
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.231-b11 mixed mode):
"DestroyJavaVM" #18 prio=5 os_prio=31 tid=0x00007ffa91491000 nid=0x1803 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Timer-0" #17 prio=5 os_prio=31 tid=0x00007ffa91f36800 nid=0x5903 waiting for monitor entry [0x0000700009aa4000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.amazon.adnumsmissionmanagerservice.homework.ScrumMaster$PeriodicIncrement.run(ScrumMaster.java:42)
- waiting to lock <0x000000076bc70db8> (a com.amazon.adnumsmissionmanagerservice.homework.Programmer)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
"Programmer-4" #15 prio=5 os_prio=31 tid=0x00007ffa92cc6800 nid=0x5603 in Object.wait() [0x000070000989e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.completeTasksUntilNoneAvailable(Programmer.java:230)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.work(Programmer.java:165)
- locked <0x000000076bc71a58> (a com.amazon.adnumsmissionmanagerservice.homework.Programmer)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.run(Programmer.java:241)
"Programmer-3" #14 prio=5 os_prio=31 tid=0x00007ffa923af800 nid=0x5503 in Object.wait() [0x000070000979b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.work(Programmer.java:176)
- locked <0x000000076bcc8ac0> (a com.amazon.adnumsmissionmanagerservice.homework.ScrumMaster$AllFinished)
- locked <0x000000076bc70db8> (a com.amazon.adnumsmissionmanagerservice.homework.Programmer)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.run(Programmer.java:241)
"Programmer-2" #13 prio=5 os_prio=31 tid=0x00007ffa92c25000 nid=0x3f03 in Object.wait() [0x0000700009698000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.work(Programmer.java:176)
- locked <0x000000076bcc8ac0> (a com.amazon.adnumsmissionmanagerservice.homework.ScrumMaster$AllFinished)
- locked <0x000000076bc5fca0> (a com.amazon.adnumsmissionmanagerservice.homework.Programmer)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.run(Programmer.java:241)
"Programmer-1" #12 prio=5 os_prio=31 tid=0x00007ffa91fab800 nid=0x4203 in Object.wait() [0x0000700009595000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.completeTasksUntilNoneAvailable(Programmer.java:230)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.work(Programmer.java:165)
- locked <0x000000076bc43c80> (a com.amazon.adnumsmissionmanagerservice.homework.Programmer)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.run(Programmer.java:241)
"Service Thread" #11 daemon prio=9 os_prio=31 tid=0x00007ffa91f30800 nid=0x4403 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #10 daemon prio=9 os_prio=31 tid=0x00007ffa9227c000 nid=0x3c03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #9 daemon prio=9 os_prio=31 tid=0x00007ffa9227b000 nid=0x4603 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #8 daemon prio=9 os_prio=31 tid=0x00007ffa92272800 nid=0x4803 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"JDWP Command Reader" #7 daemon prio=10 os_prio=31 tid=0x00007ffa9200f000 nid=0x3a03 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"JDWP Event Helper Thread" #6 daemon prio=10 os_prio=31 tid=0x00007ffa91019800 nid=0x4a03 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"JDWP Transport Listener: dt_socket" #5 daemon prio=10 os_prio=31 tid=0x00007ffa9181a000 nid=0x4b07 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007ffa9180d800 nid=0x3603 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007ffa91002000 nid=0x3003 in Object.wait() [0x0000700008b77000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076ab08ed8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x000000076ab08ed8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007ffa92006800 nid=0x2e03 in Object.wait() [0x0000700008a74000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076ab06c00> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x000000076ab06c00> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"VM Thread" os_prio=31 tid=0x00007ffa90843000 nid=0x2d03 runnable
"GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007ffa91001800 nid=0x2307 runnable
"GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007ffa91801800 nid=0x2a03 runnable
"GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007ffa91802000 nid=0x5303 runnable
"GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007ffa91802800 nid=0x5203 runnable
"VM Periodic Task Thread" os_prio=31 tid=0x00007ffa91357800 nid=0x3d03 waiting on condition
JNI global references: 2236
A few observations:
Thread Timer-0 (that's your periodic task) is in BLOCKED state, waiting for lock 0x000000076bc70db8, which is an instance of Programmer
there are 4 programmers:
2 of them are still doing some work and are holding one lock of type Programmer (they're actually holding a lock on themselves)
the other 2 programmers are done and are holding two locks: one on themselves, and one on AllFinished. Programmer-3 is an example of such thread.
Since the periodic task tries to acquire a lock on Programmer-3 before notifying it, it has to wait for Programmer-3 to release the lock on itself, which it can't do because it's waiting for all tasks to complete. Deadlock!
The reason why your programmers are holding a lock on themselves is this:
public synchronized void work()
This is going to put the entire work method into a synchronized block which monitor belongs to this. Since the Programmer class is stateless and it mostly does work that doesn't interact with other threads, you can actually synchronize a much smaller part of the work method. So you have two changes to make:
remove synchronized from the signature of work
synchronize the call to wait inside the work method:
synchronized (this) {
wait();
}
One lesson that it can give is that, when using synchronized block, you always want to synchronize as little code as possible. Anything that doesn't need to be synchronized should be outside of the block, in order to maximize parallelism (everything that happens in the block is sequential), potentially for requiring the lock less often (there might be conditions that allow you to skip the lock acquisition if you put it at the lowest level, so that will reduce the overhead of synchronization), and in some cases like this one, avoiding deadlocks.

What are threads (i.e. lightweight processes) named `{java}` created for?

I wrote a Java program that sleeps for a while:
package com.mycompany.app;
import java.lang.System;
import java.util.concurrent.TimeUnit;
public class Main {
public static void main(String[] args) {
System.out.println("the current process's pid is " + ProcessHandle.current().pid());
try {
TimeUnit.SECONDS.sleep(200);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Hello World!"); // Prints the string to the console.
}
}
I run the program with:
$ java -cp target com.mycompany.app.Main
the current process's pid is 10172
I inspect the processes that Ubuntu creates to run it:
$ pstree -pau -l -G -s 10172
systemd,1 splash
└─lxterminal,3194,t
└─bash,12150
└─java,10172 -cp target com.mycompany.app.Main
├─{java},10173
├─{java},10174
├─{java},10175
├─{java},10176
├─{java},10177
├─{java},10178
├─{java},10179
├─{java},10180
├─{java},10181
├─{java},10182
├─{java},10183
├─{java},10184
├─{java},10185
├─{java},10186
├─{java},10187
├─{java},10188
├─{java},10189
└─{java},10190
What are those threads (i.e. lightweight processes) named {java} created for?
Is it possible to find out what programs they run from shell using some commands?
Which processes (and LWPs) are running JVM?
Which processes (and LWPs) are running my Java program?

All of these threads belong to JVM.
Run jstack <pid> to get the thread list.
"main" #1 prio=5 os_prio=0 cpu=150.00ms elapsed=8.04s tid=0x00007f9f90011000 nid=0x107 waiting on condition [0x00007f9f99f9f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base#11.0.1/Native Method)
at java.lang.Thread.sleep(java.base#11.0.1/Thread.java:339)
at java.util.concurrent.TimeUnit.sleep(java.base#11.0.1/TimeUnit.java:446)
at com.mycompany.app.Main.main(Main.java:10)
"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=0.00ms elapsed=7.95s tid=0x00007f9f901f9000 nid=0x10e waiting on condition [0x00007f9f6c10f000]
java.lang.Thread.State: RUNNABLE
at java.lang.ref.Reference.waitForReferencePendingList(java.base#11.0.1/Native Method)
at java.lang.ref.Reference.processPendingReferences(java.base#11.0.1/Reference.java:241)
at java.lang.ref.Reference$ReferenceHandler.run(java.base#11.0.1/Reference.java:213)
"Finalizer" #3 daemon prio=8 os_prio=0 cpu=0.00ms elapsed=7.95s tid=0x00007f9f901fd800 nid=0x10f in Object.wait() [0x00007f9f65fef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base#11.0.1/Native Method)
- waiting on <0x0000000712108f80> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base#11.0.1/ReferenceQueue.java:155)
- waiting to re-lock in wait() <0x0000000712108f80> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base#11.0.1/ReferenceQueue.java:176)
at java.lang.ref.Finalizer$FinalizerThread.run(java.base#11.0.1/Finalizer.java:170)
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.00ms elapsed=7.93s tid=0x00007f9f90210000 nid=0x110 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 cpu=40.00ms elapsed=7.93s tid=0x00007f9f90212000 nid=0x111 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"C1 CompilerThread0" #7 daemon prio=9 os_prio=0 cpu=40.00ms elapsed=7.93s tid=0x00007f9f90214000 nid=0x112 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"Sweeper thread" #8 daemon prio=9 os_prio=0 cpu=10.00ms elapsed=7.93s tid=0x00007f9f90216000 nid=0x113 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Service Thread" #9 daemon prio=9 os_prio=0 cpu=0.00ms elapsed=7.90s tid=0x00007f9f902d3800 nid=0x114 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Common-Cleaner" #10 daemon prio=8 os_prio=0 cpu=0.00ms elapsed=7.89s tid=0x00007f9f902df800 nid=0x116 in Object.wait() [0x00007f9f656ef000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base#11.0.1/Native Method)
- waiting on <0x0000000712002df0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base#11.0.1/ReferenceQueue.java:155)
- waiting to re-lock in wait() <0x0000000712002df0> (a java.lang.ref.ReferenceQueue$Lock)
at jdk.internal.ref.CleanerImpl.run(java.base#11.0.1/CleanerImpl.java:148)
at java.lang.Thread.run(java.base#11.0.1/Thread.java:834)
at jdk.internal.misc.InnocuousThread.run(java.base#11.0.1/InnocuousThread.java:134)
"Attach Listener" #11 daemon prio=9 os_prio=0 cpu=0.00ms elapsed=0.21s tid=0x00007f9f44001000 nid=0x126 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"VM Thread" os_prio=0 cpu=0.00ms elapsed=7.95s tid=0x00007f9f901f1000 nid=0x10d runnable
"GC Thread#0" os_prio=0 cpu=0.00ms elapsed=8.01s tid=0x00007f9f90038800 nid=0x108 runnable
"G1 Main Marker" os_prio=0 cpu=0.00ms elapsed=8.00s tid=0x00007f9f90097800 nid=0x109 runnable
"G1 Conc#0" os_prio=0 cpu=0.00ms elapsed=8.00s tid=0x00007f9f90099800 nid=0x10a runnable
"G1 Refine#0" os_prio=0 cpu=0.00ms elapsed=8.00s tid=0x00007f9f9018d000 nid=0x10b runnable
"G1 Young RemSet Sampling" os_prio=0 cpu=0.00ms elapsed=8.00s tid=0x00007f9f9018f000 nid=0x10c runnable
"VM Periodic Task Thread" os_prio=0 cpu=0.00ms elapsed=7.90s tid=0x00007f9f902d6000 nid=0x115 waiting on condition
Here nid is the hexadecimal ID of a thread in the OS - you may match it to the output of pstree.
The first thread named main is the thread executing your code.
Reference Handler thread is responsible for adding Weak, Soft and Phantom references discovered by Garbage Collector into their registered ReferenceQueues.
Finalizer thread runs finalize method of the objects ready to be finalized.
Signal Dispatcher waits for specific OS signals and handles them. In particular, it makes thread dump on SIGQUIT, and also initiates VM shutdown process on SIGTERM, SIGINT and SIGHUP.
CompilerThreads perform JIT compilation of the bytecode.
Sweeper thread cleans up obsolete compiled methods.
Service Thread runs several background JVM tasks: detects low memory condition, cleans up StringTable and SymbolTable, sends deferred JVMTI events and GC notifications and so on.
Common-Cleaner runs cleaning actions of java.lang.ref.Cleaner instances.
Attach Listener thread supports Dynamic Attach mechanism. It listens for incoming Dynamic Attach connections and executes VM commands. For example, it is used by jstack, jmap and jcmd utilities.
VM Thread runs internal VM operations that require a safepoint. The examples of such operations are deoptimization, class redifinition, biased lock revocation, thread dump, heap inspection etc.
G1 threads are involved in Garbage Collection.
VM Periodic Task Thread is used to simulate timer interrupts.

Thread hung at sun.nio.ch.FileDispatcherImpl.size0(Native Method)

In my java application, I need to copy the content of a directory from one to another. But sometimes (very rare) the copyDirectory stuck forever and the code does not execute after that. Which result in high CPU.
I checked the jstack of my application multiple times and found that the same thread is in the runnable state for long. Below is the stack trace of the thread.
"pool-2-thread-3" #17 prio=5 os_prio=0 tid=0x00007fab5585c000 nid=0xa81 runnable [0x00007fab0af6f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.size0(Native Method)
at sun.nio.ch.FileDispatcherImpl.size(FileDispatcherImpl.java:84)
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:310)
- locked <0x00000000c5f59728> (a java.lang.Object)
at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:705)
at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1147)
at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1428)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1389)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1261)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1230)
I tried copying same manually with shell command but they copied successfully. Also, there is one more thread which is in running state for long with the following stack trace.
"pool-2-thread-52" #81581 prio=5 os_prio=0 tid=0x00007fab55951800 nid=0x5db runnable [0x00007faafb2f0000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileChannelImpl.position0(Native Method)
at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:288)
- locked <0x00000000c5ffde60> (a java.lang.Object)
at sun.nio.ch.FileChannelImpl.transferFromFileChannel(FileChannelImpl.java:651)
- locked <0x00000000c5ffde60> (a java.lang.Object)
at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:708)
at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1147)
at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1428)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1389)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1261)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1230)
I am not getting any clue why the thread stuck in that native call. Is there any environmental issue or related to the machine?

I just got the clue from the IO-385
Since I was using apache commons-io version 2.4. Which is having a bug where FileUtils.doCopyFile can potentially lead to infinite loop.
for(long count = 0L; pos < size; pos += output.transferFrom(input, pos, count))
{
count = size - pos > 31457280L ? 31457280L : size - pos;
}

Extremely simple code not working in HtmlUnit

I'm working with HtmlUnit 2.9 (the stable version that was released this month). Do you have any idea why the following code is not working?
public class Main {
public static void main(String[] args) {
WebClient webClient = new WebClient(BrowserVersion.FIREFOX_3_6);
webClient.setCssEnabled(true);
webClient.setCssErrorHandler(new SilentCssErrorHandler());
webClient.setThrowExceptionOnFailingStatusCode(false);
webClient.setThrowExceptionOnScriptError(false);
webClient.setRedirectEnabled(false);
webClient.setAppletEnabled(false);
webClient.setJavaScriptEnabled(false);
webClient.setPopupBlockerEnabled(true);
webClient.setTimeout(60000);
webClient.setPrintContentOnFailingStatusCode(false);
System.out.println("This is printed on screen");
try {
webClient.getPage("http://www.2cash.info/index.php");
} catch (Exception e) {
e.printStackTrace();
}
System.out.println("This is NEVER printed on screen");
}
}
I'm also adding the result of jstack. Notice I've marked a section that gets repeated constantly:
2011-08-26 03:15:45
Full thread dump Java HotSpot(TM) Server VM (20.1-b02 mixed mode):
"Attach Listener" daemon prio=10 tid=0x09520400 nid=0x5363 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"JS executor for com.gargoylesoftware.htmlunit.WebClient#a7c45e" daemon prio=10 tid=0x6feb7400 nid=0x5356 waiting on condition [0x6fcfe000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptExecutor.run(JavaScriptExecutor.java:166)
at java.lang.Thread.run(Thread.java:662)
"Low Memory Detector" daemon prio=10 tid=0x70204c00 nid=0x5352 runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x70202800 nid=0x5351 runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x70200800 nid=0x5350 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x09514c00 nid=0x534f runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x09503400 nid=0x534e in Object.wait() [0x70798000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x76af2ff0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x76af2ff0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x09501c00 nid=0x534d in Object.wait() [0x707e9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x7675cc58> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x7675cc58> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x09482400 nid=0x5349 runnable [0xb6c34000]
java.lang.Thread.State: RUNNABLE
at net.sourceforge.htmlunit.corejs.javascript.ScriptableObject.getSlot(ScriptableObject.java:2603)
at net.sourceforge.htmlunit.corejs.javascript.ScriptableObject.defineProperty(ScriptableObject.java:1699)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.configureConstantsPropertiesAndFunctions(JavaScriptEngine.java:350)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.configureClass(JavaScriptEngine.java:330)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.init(JavaScriptEngine.java:199)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.access$000(JavaScriptEngine.java:79)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$1.run(JavaScriptEngine.java:146)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:537)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:538)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.initialize(JavaScriptEngine.java:157)
at com.gargoylesoftware.htmlunit.WebClient.initialize(WebClient.java:1141)
at com.gargoylesoftware.htmlunit.WebWindowImpl.setEnclosedPage(WebWindowImpl.java:109)
at com.gargoylesoftware.htmlunit.html.FrameWindow.setEnclosedPage(FrameWindow.java:102)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parse(HTMLParser.java:200)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parseHtml(HTMLParser.java:179)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:221)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:106)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:433)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:311)
at com.gargoylesoftware.htmlunit.html.BaseFrame.<init>(BaseFrame.java:73)
at com.gargoylesoftware.htmlunit.html.HtmlInlineFrame.<init>(HtmlInlineFrame.java:46)
at com.gargoylesoftware.htmlunit.html.DefaultElementFactory.createElementNS(DefaultElementFactory.java:288)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.startElement(HTMLParser.java:506)
at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
at org.cyberneko.html.HTMLTagBalancer.callStartElement(HTMLTagBalancer.java:1136)
at org.cyberneko.html.HTMLTagBalancer.startElement(HTMLTagBalancer.java:742)
at org.cyberneko.html.filters.DefaultFilter.startElement(DefaultFilter.java:136)
at org.cyberneko.html.filters.NamespaceBinder.startElement(NamespaceBinder.java:278)
at org.cyberneko.html.HTMLScanner$ContentScanner.scanStartElement(HTMLScanner.java:2652)
at org.cyberneko.html.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2022)
at org.cyberneko.html.HTMLScanner.scanDocument(HTMLScanner.java:908)
at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:499)
at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:452)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.parse(HTMLParser.java:789)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parse(HTMLParser.java:225)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parseHtml(HTMLParser.java:179)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:221)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:106)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:433)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:311)
<THIS_SECTION_IS_PRINTED_AS_IF_IT_WERE_IN_A_LOOP>
at com.gargoylesoftware.htmlunit.html.BaseFrame.loadInnerPageIfPossible(BaseFrame.java:149)
at com.gargoylesoftware.htmlunit.html.BaseFrame.loadInnerPage(BaseFrame.java:99)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadFrames(HtmlPage.java:1760)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:194)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:440)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:311)
</THIS_SECTION_IS_PRINTED_AS_IF_IT_WERE_IN_A_LOOP>
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:311)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:373)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:358)
at main.Main.<init>(Main.java:42)
at main.Main.main(Main.java:23)
"VM Thread" prio=10 tid=0x094fe000 nid=0x534c runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x09489800 nid=0x534a runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x0948ac00 nid=0x534b runnable
"VM Periodic Task Thread" prio=10 tid=0x70207000 nid=0x5353 waiting on condition
JNI global references: 1234
I think there is some kind of loop regarding the automatic loading of frames. If that is the case, is there any way to disable that behaviour to break the loop?
Thanks in advance!

Well, although it is a horrible solution (workaround, actually...), I finally decided to disable the automatic loading of frames in HtmlUnit as adviced by one of the developers of HtmlUnit. This is what I did in detail:
Downloaded the HtmlUnit source
Downloaded maven from here
Commented the content (the body of the method, not the declaration) of the loadFrames() method of the HtmlPage class located in htmlunit-2.9/src/main/java/com/gargoylesoftware/htmlunit/html
Compiled this custom code skipping tests with: mvn -Dmaven.test.skip=true clean compile package
Got the new htmlunit-2.9.jar located in htmlunit-2.9/artifacts and replaced the current htmlunit-2.9.jar library file
This step might be the most delicate one as it will depend on each application. However, I'll show you the changes I needed to do to my application.
You know how my original code was (look at the question). That would download all frames and iframes from a page. I'm adding an example on how to get a page with frames just loading the frames you want:
try {
HtmlPage page = webClient.getPage("http://www.w3schools.com/HTML/tryit.asp?filename=tryhtml_noframes");
HtmlInlineFrame frame = page.getFirstByXPath("//iframe[#name='view']");
page = webClient.getPage(page.getFullyQualifiedUrl(frame.getSrcAttribute()));
System.out.println(page.asXml());
} catch (Exception e) {
e.printStackTrace();
}
After this library change, the content of the frame will be empty once the getPage() method finishes. Notice it won't be null, looks like it is just returning an empty frame. What we need to do is to download the content of the frames we are interested in manually, that's why I'm performing a getPage() again.
Well this is how I managed to selectively download frames and iframes with HtmlUnit. Any ideas on how to improve this will be appreciated. Anyway, I hope there will be added some way to disable the loading of the frames in HtmlUnit itself in the future, maybe adding a method such as getPage(URL url, boolean downloadFrames) or something.
Hope this helps someone out there!

When I open this site in my browser it does not ever finish loading the page. This might be the problem why HtmlUnit crashes, too. Tested with Chrome and FF.
Try loading a more simple site instead and you may know if this crash is site-depended.

Which Java thread is hogging the CPU?

Let's say your Java program is taking 100% CPU. It has 50 threads. You need to find which thread is guilty. I have not found a tool that can help. Currently I use the following very time consuming routine:
Run jstack <pid>, where pid is the process id of a Java process. The easy way to find it is to run another utility included in the JDK - jps. It is better to redirect jstack's output to a file.
Search for "runnable" threads. Skip those that wait on a socket (for some reason they are still marked runnable).
Repeat steps 1 and 2 a couple of times and see if you can locate a pattern.
Alternatively, you could attach to a Java process in Eclipse and try to suspend threads one by one, until you hit the one that hogs CPU. On a one-CPU machine, you might need to first reduce the Java process's priority to be able to move around. Even then, Eclipse often isn't able to attach to a running process due to a timeout.
I would have expected Sun's visualvm tool to do this.
Does anybody know of a better way?

Identifying which Java Thread is consuming most CPU in production server.
Most (if not all) productive systems doing anything important will use more than 1 java thread. And when something goes crazy and your cpu usage is on 100%, it is hard to identify which thread(s) is/are causing this. Or so I thought. Until someone smarter than me showed me how it can be done. And here I will show you how to do it and you too can amaze your family and friends with your geek skills.
A Test Application
In order to test this, we need a test application. So I will give you one. It consists of 3 classes:
A HeavyThread class that does something CPU intensive (computing MD5 hashes)
A LightThread class that does something not-so-cpu-intensive (counting and sleeping).
A StartThreads class to start 1 cpu intensive and several light threads.
Here is code for these classes:
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.UUID;
/**
* thread that does some heavy lifting
*
* #author srasul
*
*/
public class HeavyThread implements Runnable {
private long length;
public HeavyThread(long length) {
this.length = length;
new Thread(this).start();
}
#Override
public void run() {
while (true) {
String data = "";
// make some stuff up
for (int i = 0; i < length; i++) {
data += UUID.randomUUID().toString();
}
MessageDigest digest;
try {
digest = MessageDigest.getInstance("MD5");
} catch (NoSuchAlgorithmException e) {
throw new RuntimeException(e);
}
// hash the data
digest.update(data.getBytes());
}
}
}
import java.util.Random;
/**
* thread that does little work. just count & sleep
*
* #author srasul
*
*/
public class LightThread implements Runnable {
public LightThread() {
new Thread(this).start();
}
#Override
public void run() {
Long l = 0l;
while(true) {
l++;
try {
Thread.sleep(new Random().nextInt(10));
} catch (InterruptedException e) {
e.printStackTrace();
}
if(l == Long.MAX_VALUE) {
l = 0l;
}
}
}
}
/**
* start it all
*
* #author srasul
*
*/
public class StartThreads {
public static void main(String[] args) {
// lets start 1 heavy ...
new HeavyThread(1000);
// ... and 3 light threads
new LightThread();
new LightThread();
new LightThread();
}
}
Assuming that you have never seen this code, and all you have a PID of a runaway Java process that is running these classes and is consuming 100% CPU.
First let's start the StartThreads class.
$ ls
HeavyThread.java LightThread.java StartThreads.java
$ javac *
$ java StartThreads &
At this stage a Java process is running should be taking up 100 cpu. In my top I see:
In top press Shift-H which turns on Threads. The man page for top says:
-H : Threads toggle
Starts top with the last remembered 'H' state reversed. When
this toggle is On, all individual threads will be displayed.
Otherwise, top displays a summation of all threads in a
process.
And now in my top with Threads display turned ON i see:
And I have a java process with PID 28294. Lets get the stack dump of this process using jstack:
$ jstack 28924
2010-11-18 13:05:41
Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.0-b16 mixed mode):
"Attach Listener" daemon prio=10 tid=0x0000000040ecb000 nid=0x7150 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"DestroyJavaVM" prio=10 tid=0x00007f9a98027800 nid=0x70fd waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Thread-3" prio=10 tid=0x00007f9a98025800 nid=0x710d waiting on condition [0x00007f9a9d543000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at LightThread.run(LightThread.java:21)
at java.lang.Thread.run(Thread.java:619)
"Thread-2" prio=10 tid=0x00007f9a98023800 nid=0x710c waiting on condition [0x00007f9a9d644000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at LightThread.run(LightThread.java:21)
at java.lang.Thread.run(Thread.java:619)
"Thread-1" prio=10 tid=0x00007f9a98021800 nid=0x710b waiting on condition [0x00007f9a9d745000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at LightThread.run(LightThread.java:21)
at java.lang.Thread.run(Thread.java:619)
"Thread-0" prio=10 tid=0x00007f9a98020000 nid=0x710a runnable [0x00007f9a9d846000]
java.lang.Thread.State: RUNNABLE
at sun.security.provider.DigestBase.engineReset(DigestBase.java:139)
at sun.security.provider.DigestBase.engineUpdate(DigestBase.java:104)
at java.security.MessageDigest$Delegate.engineUpdate(MessageDigest.java:538)
at java.security.MessageDigest.update(MessageDigest.java:293)
at sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:197)
- locked <0x00007f9aa457e400> (a sun.security.provider.SecureRandom)
at sun.security.provider.NativePRNG$RandomIO.implNextBytes(NativePRNG.java:257)
- locked <0x00007f9aa457e708> (a java.lang.Object)
at sun.security.provider.NativePRNG$RandomIO.access$200(NativePRNG.java:108)
at sun.security.provider.NativePRNG.engineNextBytes(NativePRNG.java:97)
at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
- locked <0x00007f9aa4582fc8> (a java.security.SecureRandom)
at java.util.UUID.randomUUID(UUID.java:162)
at HeavyThread.run(HeavyThread.java:27)
at java.lang.Thread.run(Thread.java:619)
"Low Memory Detector" daemon prio=10 tid=0x00007f9a98006800 nid=0x7108 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x00007f9a98004000 nid=0x7107 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x00007f9a98001000 nid=0x7106 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x0000000040de4000 nid=0x7105 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x0000000040dc4800 nid=0x7104 in Object.wait() [0x00007f9a97ffe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f9aa45506b0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00007f9aa45506b0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x0000000040dbd000 nid=0x7103 in Object.wait() [0x00007f9a9de92000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f9aa4550318> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00007f9aa4550318> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x0000000040db8800 nid=0x7102 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x0000000040d6e800 nid=0x70fe runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000040d70800 nid=0x70ff runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000040d72000 nid=0x7100 runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000040d74000 nid=0x7101 runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f9a98011800 nid=0x7109 waiting on condition
JNI global references: 910
From my top I see that the PID of the top thread is 28938. And 28938 in hex is 0x710A. Notice that in the stack dump, each thread has an nid which is dispalyed in hex. And it just so happens that 0x710A is the id of the thread:
"Thread-0" prio=10 tid=0x00007f9a98020000 nid=0x710a runnable [0x00007f9a9d846000]
java.lang.Thread.State: RUNNABLE
at sun.security.provider.DigestBase.engineReset(DigestBase.java:139)
at sun.security.provider.DigestBase.engineUpdate(DigestBase.java:104)
at java.security.MessageDigest$Delegate.engineUpdate(MessageDigest.java:538)
at java.security.MessageDigest.update(MessageDigest.java:293)
at sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:197)
- locked <0x00007f9aa457e400> (a sun.security.provider.SecureRandom)
at sun.security.provider.NativePRNG$RandomIO.implNextBytes(NativePRNG.java:257)
- locked <0x00007f9aa457e708> (a java.lang.Object)
at sun.security.provider.NativePRNG$RandomIO.access$200(NativePRNG.java:108)
at sun.security.provider.NativePRNG.engineNextBytes(NativePRNG.java:97)
at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
- locked <0x00007f9aa4582fc8> (a java.security.SecureRandom)
at java.util.UUID.randomUUID(UUID.java:162)
at HeavyThread.run(HeavyThread.java:27)
at java.lang.Thread.run(Thread.java:619)
And so you can confirm that the thread which is running the HeavyThread class is consuming most CPU.
In read world situations, it will probably be a bunch of threads that consume some portion of CPU and these threads put together will lead to the Java process using 100% CPU.
Summary
Run top
Press Shift-H to enable Threads View
Get PID of the thread with highest CPU
Convert PID to HEX
Get stack dump of java process
Look for thread with the matching HEX PID.

jvmtop can show you the top consuming threads:
TID NAME STATE CPU TOTALCPU
25 http-8080-Processor13 RUNNABLE 4.55% 1.60%
128022 RMI TCP Connection(18)-10.101. RUNNABLE 1.82% 0.02%
36578 http-8080-Processor164 RUNNABLE 0.91% 2.35%
128026 JMX server connection timeout TIMED_WAITING 0.00% 0.00%

Try looking at the Hot Thread Detector plugin for visual VM -- it uses the ThreadMXBean API to take multiple CPU consumption samples to find the most active threads. It's based on a command-line equivalent from Bruce Chapman which might also be useful.

Just run up JVisualVM, connect to your app and and use the thread view. The one which remains continually active is your most likely culprit.

Have a look at the Top Threads plugin for JConsole.

I would recommend taking a look at Arthas tool open sourced by Alibaba.
It contains a bunch of useful commands that can help you debug your production code:
Dashboard: Overview of Your Java Process
SC: Search Class Loaded by Your JVM
Jad: Decompile Class Into Source Code
Watch: View Method Invocation Input and Results
Trace: Find the Bottleneck of Your Method Invocation
Monitor: View Method Invocation Statistics
Stack: View Call Stack of the Method
Tt: Time Tunnel of Method Invocations
Example of the dashboard:

If you're running under Windows, try Process Explorer. Bring up the properties dialog for your process, then select the Threads tab.

Take a thread dump. Wait for 10 seconds. Take another thread dump. Repeat one more time.
Inspect the thread dumps and see which threads are stuck at the same place, or processing the same request.
This is a manual way of doing it, but often useful.

Use ps -eL or top -H -p <pid>, or if you need to see and monitor in real time, run top (then shift H), to get the Light Weight Process ( LWP aka threads) associated with the java process.
root#xxx:/# ps -eL
PID LWP TTY TIME CMD
1 1 ? 00:00:00 java
1 7 ? 00:00:01 java
1 8 ? 00:07:52 java
1 9 ? 00:07:52 java
1 10 ? 00:07:51 java
1 11 ? 00:07:52 java
1 12 ? 00:07:52 java
1 13 ? 00:07:51 java
1 14 ? 00:07:51 java
1 15 ? 00:07:53 java
…
1 164 ? 00:00:02 java
1 166 ? 00:00:02 java
1 169 ? 00:00:02 java
Note LWP= Lightweight Process; In Linux, a thread is associated with a process so that it can be managed in the kernel; LWP shares files and other resources with the parent process.
Now let us see the threads that are taking most time
1 8 ? 00:07:52 java
1 9 ? 00:07:52 java
1 10 ? 00:07:51 java
1 11 ? 00:07:52 java
1 12 ? 00:07:52 java
1 13 ? 00:07:51 java
1 14 ? 00:07:51 java
1 15 ? 00:07:53 java
Jstack is a JDK utility to print Java Stack; It prints thread of the form.
Familiarize yourself with others cool JDK tools as well (jcmd jstat jhat jmap jstack etc — https://docs.oracle.com/javase/8/docs/technotes/tools/unix/)
jstack -l <process id>
The nid, Native thread id in the stack trace is the one that is connected to LWT in linux (https://gist.github.com/rednaxelafx/843622)
“GC task thread#0 (ParallelGC)” os_prio=0 tid=0x00007fc21801f000 nid=0x8 runnable
The nid is given in Hex; So we convert the thread id taking the most time
8,9,10,11,12,13,14,15 in DEC = 8,9,A, B,C,D,E,F in HEX.
(note that this particular stack was taken from Java in a Docker container, with a convenient process if of 1 )
Let us see the thread with this ids..
“GC task thread#0 (ParallelGC)” os_prio=0 tid=0x00007fc21801f000 nid=0x8 runnable
“GC task thread#1 (ParallelGC)” os_prio=0 tid=0x00007fc218020800 nid=0x9 runnable
“GC task thread#2 (ParallelGC)” os_prio=0 tid=0x00007fc218022800 nid=0xa runnable
“GC task thread#3 (ParallelGC)” os_prio=0 tid=0x00007fc218024000 nid=0xb runnable
“GC task thread#4 (ParallelGC)” os_prio=0 tid=0x00007fc218026000 nid=0xc runnable
“GC task thread#5 (ParallelGC)” os_prio=0 tid=0x00007fc218027800 nid=0xd runnable
“GC task thread#6 (ParallelGC)” os_prio=0 tid=0x00007fc218029800 nid=0xe runnable
“GC task thread#7 (ParallelGC)” os_prio=0 tid=0x00007fc21802b000 nid=0xf runnable
All GC related threads; No wonder it was taking lot of CPU time; But then is GC a problem here.
Use jstat (not jstack !) utility to have a quick check for GC.
jstat -gcutil <pid>

This is a kind of hacky way, but it seems like you could fire the application up in a debugger, and then suspend all the threads, and go through the code and find out which one isn't blocking on a lock or an I/O call in some kind of loop. Or is this like what you've already tried?

An option you could consider is querying your threads for the answer from within application. Via the ThreadMXBean you can query CPU usage of threads from within your Java application and query stack traces of the offending thread(s).
The ThreadMXBean option allows you to build this kind of monitoring into your live application. It has negligible impact and has the distinct advantage that you can make it do exactly what you want.

If you suspect VisualVM is a good tool, try it (because it does this) Find out the threads(s) only helps you in the general direction of why it is consuming so much CPU.
However, if its that obvious I would go straight to using a profiler to find out why you are consuming so much CPU.