We Keep Coding

Strange bug with wait statement in synchronized block

An excerpt from Worker.java file:
public class Worker extends Thread{
public void run(){
// Worker Thread periodically does its job.
Master.getInstance().decrementNumOfWorkingWorkers();
// This is the reporting part of the thread.
// Aimed to wait other threads finish their job.
synchronized (Master.getInstance().allFinished) {
while ( Master.getInstance().getNumOfWorkingWorkers() > 0) {
try {
Master.getInstance().allFinished.wait();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Main.printSync("Worker Thread-" + getPId() + " worked on");
}
}
}
This is from Master.Java:
import java.util.LinkedList;
import java.util.Timer;
import java.util.TimerTask;
public class Master extends Timer {
AllFinished allFinished;
int day;
public TimerTask task;
LinkedList<Worker> Workers;
private static Master instance = null;
int numOfWorkingWorkers = 0;
public class AllFinished
{
}
public class PeriodicIncrement extends TimerTask {
// Complete this class
public void run() {
Main.printSync("Day " + day + ":");
Main.printSync("Queue: " + TaskQueue.getInstance().ConvertToString());
day++;
for (int i = 0; i < Workers.size(); i++) {
synchronized (Workers.get(i)) {
Workers.get(i).notify();
}
}
if (0 == numOfWorkingWorkers) {
synchronized (allFinished) {
allFinished.notifyAll();
}
cancel(); // Terminate the timer thread
}
}
}
private Master(LinkedList<Worker> Workers) {
super();
this.task = new PeriodicIncrement();
day = 0;
allFinished = new AllFinished();
this.Workers = Workers;
numOfWorkingWorkers = this.Workers.size();
this.schedule(task, 100, 100);
}
}
For a test with 4 worker threads, everything was fine before I added the excerpted part in Worker.java. Then, to report each worker's action after all workers are done, I added that part. Algorithm is very simple. When a worker finishes its job, it checks if there is any job in a TaskQueue and ProductOwner. If there are not any, it breaks its loop and then decrement by 1 active worker threads counter in Master and then calls wait on AllFinished field of Master. run() method of PeriodicIncrement checks this counter and if it is 0 (meaning all workers finished their job), it calls notifyAll() on AllFinished .
Problem is, one sometimes two threads are entering that excerpted code block in Worker.java but remainings never enter so active worker threads counter never decremented to 0 and my program never finishes.
If I just comment out the excerpted part in Worker.java, except finishing and reporting randomly, everything is fine. What I mean is the excerpted part seems to be problematic.
Could you help me to find out?

That was a fun one to debug, after so much time not playing with low-level concurrency primitives. The trick to root-cause this was to use the jstack tool provided by the JDK.
╭───courtino ~
╰➤ sudo jstack -l 63978
2020-03-29 21:26:01
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.231-b11 mixed mode):
"DestroyJavaVM" #18 prio=5 os_prio=31 tid=0x00007ffa91491000 nid=0x1803 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Timer-0" #17 prio=5 os_prio=31 tid=0x00007ffa91f36800 nid=0x5903 waiting for monitor entry [0x0000700009aa4000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.amazon.adnumsmissionmanagerservice.homework.ScrumMaster$PeriodicIncrement.run(ScrumMaster.java:42)
- waiting to lock <0x000000076bc70db8> (a com.amazon.adnumsmissionmanagerservice.homework.Programmer)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
"Programmer-4" #15 prio=5 os_prio=31 tid=0x00007ffa92cc6800 nid=0x5603 in Object.wait() [0x000070000989e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.completeTasksUntilNoneAvailable(Programmer.java:230)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.work(Programmer.java:165)
- locked <0x000000076bc71a58> (a com.amazon.adnumsmissionmanagerservice.homework.Programmer)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.run(Programmer.java:241)
"Programmer-3" #14 prio=5 os_prio=31 tid=0x00007ffa923af800 nid=0x5503 in Object.wait() [0x000070000979b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.work(Programmer.java:176)
- locked <0x000000076bcc8ac0> (a com.amazon.adnumsmissionmanagerservice.homework.ScrumMaster$AllFinished)
- locked <0x000000076bc70db8> (a com.amazon.adnumsmissionmanagerservice.homework.Programmer)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.run(Programmer.java:241)
"Programmer-2" #13 prio=5 os_prio=31 tid=0x00007ffa92c25000 nid=0x3f03 in Object.wait() [0x0000700009698000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.work(Programmer.java:176)
- locked <0x000000076bcc8ac0> (a com.amazon.adnumsmissionmanagerservice.homework.ScrumMaster$AllFinished)
- locked <0x000000076bc5fca0> (a com.amazon.adnumsmissionmanagerservice.homework.Programmer)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.run(Programmer.java:241)
"Programmer-1" #12 prio=5 os_prio=31 tid=0x00007ffa91fab800 nid=0x4203 in Object.wait() [0x0000700009595000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.completeTasksUntilNoneAvailable(Programmer.java:230)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.work(Programmer.java:165)
- locked <0x000000076bc43c80> (a com.amazon.adnumsmissionmanagerservice.homework.Programmer)
at com.amazon.adnumsmissionmanagerservice.homework.Programmer.run(Programmer.java:241)
"Service Thread" #11 daemon prio=9 os_prio=31 tid=0x00007ffa91f30800 nid=0x4403 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #10 daemon prio=9 os_prio=31 tid=0x00007ffa9227c000 nid=0x3c03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #9 daemon prio=9 os_prio=31 tid=0x00007ffa9227b000 nid=0x4603 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #8 daemon prio=9 os_prio=31 tid=0x00007ffa92272800 nid=0x4803 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"JDWP Command Reader" #7 daemon prio=10 os_prio=31 tid=0x00007ffa9200f000 nid=0x3a03 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"JDWP Event Helper Thread" #6 daemon prio=10 os_prio=31 tid=0x00007ffa91019800 nid=0x4a03 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"JDWP Transport Listener: dt_socket" #5 daemon prio=10 os_prio=31 tid=0x00007ffa9181a000 nid=0x4b07 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007ffa9180d800 nid=0x3603 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007ffa91002000 nid=0x3003 in Object.wait() [0x0000700008b77000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076ab08ed8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x000000076ab08ed8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007ffa92006800 nid=0x2e03 in Object.wait() [0x0000700008a74000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076ab06c00> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x000000076ab06c00> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"VM Thread" os_prio=31 tid=0x00007ffa90843000 nid=0x2d03 runnable
"GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007ffa91001800 nid=0x2307 runnable
"GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007ffa91801800 nid=0x2a03 runnable
"GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007ffa91802000 nid=0x5303 runnable
"GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007ffa91802800 nid=0x5203 runnable
"VM Periodic Task Thread" os_prio=31 tid=0x00007ffa91357800 nid=0x3d03 waiting on condition
JNI global references: 2236
A few observations:
Thread Timer-0 (that's your periodic task) is in BLOCKED state, waiting for lock 0x000000076bc70db8, which is an instance of Programmer
there are 4 programmers:
2 of them are still doing some work and are holding one lock of type Programmer (they're actually holding a lock on themselves)
the other 2 programmers are done and are holding two locks: one on themselves, and one on AllFinished. Programmer-3 is an example of such thread.
Since the periodic task tries to acquire a lock on Programmer-3 before notifying it, it has to wait for Programmer-3 to release the lock on itself, which it can't do because it's waiting for all tasks to complete. Deadlock!
The reason why your programmers are holding a lock on themselves is this:
public synchronized void work()
This is going to put the entire work method into a synchronized block which monitor belongs to this. Since the Programmer class is stateless and it mostly does work that doesn't interact with other threads, you can actually synchronize a much smaller part of the work method. So you have two changes to make:
remove synchronized from the signature of work
synchronize the call to wait inside the work method:
synchronized (this) {
wait();
}
One lesson that it can give is that, when using synchronized block, you always want to synchronize as little code as possible. Anything that doesn't need to be synchronized should be outside of the block, in order to maximize parallelism (everything that happens in the block is sequential), potentially for requiring the lock less often (there might be conditions that allow you to skip the lock acquisition if you put it at the lowest level, so that will reduce the overhead of synchronization), and in some cases like this one, avoiding deadlocks.

What are threads (i.e. lightweight processes) named `{java}` created for?

I wrote a Java program that sleeps for a while:
package com.mycompany.app;
import java.lang.System;
import java.util.concurrent.TimeUnit;
public class Main {
public static void main(String[] args) {
System.out.println("the current process's pid is " + ProcessHandle.current().pid());
try {
TimeUnit.SECONDS.sleep(200);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Hello World!"); // Prints the string to the console.
}
}
I run the program with:
$ java -cp target com.mycompany.app.Main
the current process's pid is 10172
I inspect the processes that Ubuntu creates to run it:
$ pstree -pau -l -G -s 10172
systemd,1 splash
└─lxterminal,3194,t
└─bash,12150
└─java,10172 -cp target com.mycompany.app.Main
├─{java},10173
├─{java},10174
├─{java},10175
├─{java},10176
├─{java},10177
├─{java},10178
├─{java},10179
├─{java},10180
├─{java},10181
├─{java},10182
├─{java},10183
├─{java},10184
├─{java},10185
├─{java},10186
├─{java},10187
├─{java},10188
├─{java},10189
└─{java},10190
What are those threads (i.e. lightweight processes) named {java} created for?
Is it possible to find out what programs they run from shell using some commands?
Which processes (and LWPs) are running JVM?
Which processes (and LWPs) are running my Java program?

All of these threads belong to JVM.
Run jstack <pid> to get the thread list.
"main" #1 prio=5 os_prio=0 cpu=150.00ms elapsed=8.04s tid=0x00007f9f90011000 nid=0x107 waiting on condition [0x00007f9f99f9f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base#11.0.1/Native Method)
at java.lang.Thread.sleep(java.base#11.0.1/Thread.java:339)
at java.util.concurrent.TimeUnit.sleep(java.base#11.0.1/TimeUnit.java:446)
at com.mycompany.app.Main.main(Main.java:10)
"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=0.00ms elapsed=7.95s tid=0x00007f9f901f9000 nid=0x10e waiting on condition [0x00007f9f6c10f000]
java.lang.Thread.State: RUNNABLE
at java.lang.ref.Reference.waitForReferencePendingList(java.base#11.0.1/Native Method)
at java.lang.ref.Reference.processPendingReferences(java.base#11.0.1/Reference.java:241)
at java.lang.ref.Reference$ReferenceHandler.run(java.base#11.0.1/Reference.java:213)
"Finalizer" #3 daemon prio=8 os_prio=0 cpu=0.00ms elapsed=7.95s tid=0x00007f9f901fd800 nid=0x10f in Object.wait() [0x00007f9f65fef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base#11.0.1/Native Method)
- waiting on <0x0000000712108f80> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base#11.0.1/ReferenceQueue.java:155)
- waiting to re-lock in wait() <0x0000000712108f80> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base#11.0.1/ReferenceQueue.java:176)
at java.lang.ref.Finalizer$FinalizerThread.run(java.base#11.0.1/Finalizer.java:170)
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.00ms elapsed=7.93s tid=0x00007f9f90210000 nid=0x110 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 cpu=40.00ms elapsed=7.93s tid=0x00007f9f90212000 nid=0x111 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"C1 CompilerThread0" #7 daemon prio=9 os_prio=0 cpu=40.00ms elapsed=7.93s tid=0x00007f9f90214000 nid=0x112 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"Sweeper thread" #8 daemon prio=9 os_prio=0 cpu=10.00ms elapsed=7.93s tid=0x00007f9f90216000 nid=0x113 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Service Thread" #9 daemon prio=9 os_prio=0 cpu=0.00ms elapsed=7.90s tid=0x00007f9f902d3800 nid=0x114 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Common-Cleaner" #10 daemon prio=8 os_prio=0 cpu=0.00ms elapsed=7.89s tid=0x00007f9f902df800 nid=0x116 in Object.wait() [0x00007f9f656ef000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base#11.0.1/Native Method)
- waiting on <0x0000000712002df0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base#11.0.1/ReferenceQueue.java:155)
- waiting to re-lock in wait() <0x0000000712002df0> (a java.lang.ref.ReferenceQueue$Lock)
at jdk.internal.ref.CleanerImpl.run(java.base#11.0.1/CleanerImpl.java:148)
at java.lang.Thread.run(java.base#11.0.1/Thread.java:834)
at jdk.internal.misc.InnocuousThread.run(java.base#11.0.1/InnocuousThread.java:134)
"Attach Listener" #11 daemon prio=9 os_prio=0 cpu=0.00ms elapsed=0.21s tid=0x00007f9f44001000 nid=0x126 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"VM Thread" os_prio=0 cpu=0.00ms elapsed=7.95s tid=0x00007f9f901f1000 nid=0x10d runnable
"GC Thread#0" os_prio=0 cpu=0.00ms elapsed=8.01s tid=0x00007f9f90038800 nid=0x108 runnable
"G1 Main Marker" os_prio=0 cpu=0.00ms elapsed=8.00s tid=0x00007f9f90097800 nid=0x109 runnable
"G1 Conc#0" os_prio=0 cpu=0.00ms elapsed=8.00s tid=0x00007f9f90099800 nid=0x10a runnable
"G1 Refine#0" os_prio=0 cpu=0.00ms elapsed=8.00s tid=0x00007f9f9018d000 nid=0x10b runnable
"G1 Young RemSet Sampling" os_prio=0 cpu=0.00ms elapsed=8.00s tid=0x00007f9f9018f000 nid=0x10c runnable
"VM Periodic Task Thread" os_prio=0 cpu=0.00ms elapsed=7.90s tid=0x00007f9f902d6000 nid=0x115 waiting on condition
Here nid is the hexadecimal ID of a thread in the OS - you may match it to the output of pstree.
The first thread named main is the thread executing your code.
Reference Handler thread is responsible for adding Weak, Soft and Phantom references discovered by Garbage Collector into their registered ReferenceQueues.
Finalizer thread runs finalize method of the objects ready to be finalized.
Signal Dispatcher waits for specific OS signals and handles them. In particular, it makes thread dump on SIGQUIT, and also initiates VM shutdown process on SIGTERM, SIGINT and SIGHUP.
CompilerThreads perform JIT compilation of the bytecode.
Sweeper thread cleans up obsolete compiled methods.
Service Thread runs several background JVM tasks: detects low memory condition, cleans up StringTable and SymbolTable, sends deferred JVMTI events and GC notifications and so on.
Common-Cleaner runs cleaning actions of java.lang.ref.Cleaner instances.
Attach Listener thread supports Dynamic Attach mechanism. It listens for incoming Dynamic Attach connections and executes VM commands. For example, it is used by jstack, jmap and jcmd utilities.
VM Thread runs internal VM operations that require a safepoint. The examples of such operations are deoptimization, class redifinition, biased lock revocation, thread dump, heap inspection etc.
G1 threads are involved in Garbage Collection.
VM Periodic Task Thread is used to simulate timer interrupts.

why does trying to attach to myself crash my vm?

running the following code works fine on macos (sun jdk 1.8.0_51), but crashes every time with a thread dump on centOS (sun jdk 1.8.0 101) when it comes to attaching to itself, while succeeding to attach any other visible VM:
import com.sun.tools.attach.VirtualMachine;
import com.sun.tools.attach.VirtualMachineDescriptor;
public class Test {
public static void main(String[] args) throws Exception {
for(VirtualMachineDescriptor vmd : VirtualMachine.list()) {
VirtualMachine vm = VirtualMachine.attach(vmd);
}
}
}
the thread dump:
"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fbb780bf000 nid=0x28a0 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fbb780bc000 nid=0x289f runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fbb780ba000 nid=0x289e runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fbb780b7000 nid=0x289d runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fbb780b6000 nid=0x289c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fbb78082800 nid=0x289b in Object.wait() [0x00007fbb62eed000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000d6608ee0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000d6608ee0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fbb7807e000 nid=0x289a in Object.wait() [0x00007fbb62fee000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000d6606b50> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000d6606b50> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=0 tid=0x00007fbb78008000 nid=0x2894 waiting on condition [0x00007fbb7f307000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:100)
at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:78)
at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:250)
at Test.main(Test.java:10)
"VM Thread" os_prio=0 tid=0x00007fbb78076800 nid=0x2899 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fbb7801d800 nid=0x2895 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fbb7801f800 nid=0x2896 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fbb78021000 nid=0x2897 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fbb78023000 nid=0x2898 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007fbb780c2000 nid=0x28a1 waiting on condition
JNI global references: 19
Heap
PSYoungGen total 37888K, used 9835K [0x00000000d6600000, 0x00000000d9000000, 0x0000000100000000)
eden space 32768K, 30% used [0x00000000d6600000,0x00000000d6f9ad98,0x00000000d8600000)
from space 5120K, 0% used [0x00000000d8b00000,0x00000000d8b00000,0x00000000d9000000)
to space 5120K, 0% used [0x00000000d8600000,0x00000000d8600000,0x00000000d8b00000)
ParOldGen total 86016K, used 0K [0x0000000083200000, 0x0000000088600000, 0x00000000d6600000)
object space 86016K, 0% used [0x0000000083200000,0x0000000083200000,0x0000000088600000)
Metaspace used 3451K, capacity 4728K, committed 4864K, reserved 1056768K
class space used 382K, capacity 424K, committed 512K, reserved 1048576K
Exception in thread "main" com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:106)
at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:78)
at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:250)
at Test.main(Test.java:10)
can anyone explain?

how to interprete this thread dump from a hung Java Swing application?

I have the following thread dump from a hung java swing application. It hung after a button is clicked and the GUI changed to blank. Other threads in socket communication and task management are still working (from the log file I can tell). I have removed some non-relevant output.
The #13 AW-EventQueue-0 should send out a command through the socket but it seems failed there. The #20 and #21 are AW-EventQueue-0-SharedResourceRunner which is not the same as the #13? It seems there is no deadlock but the GUI is not responsive and became blank.
do you see any useful information about the cause of the hanging?
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.20-b23 mixed mode):
"DestroyJavaVM" #32 prio=5 os_prio=0 tid=0x00007f286c009800 nid=0xa41 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"TimerQueue" #22 daemon prio=5 os_prio=0 tid=0x00007f28002a8800 nid=0xa65 waiting on condition [0x00007f284c56f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000088a8f5c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
at javax.swing.TimerQueue.run(TimerQueue.java:171)
at java.lang.Thread.run(Thread.java:745)
"AWT-EventQueue-0-SharedResourceRunner" #21 daemon prio=6 os_prio=0 tid=0x00007f280021d000 nid=0xa64 in Object.wait() [0x00007f284d434000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000088aec748> (a jogamp.opengl.SharedResourceRunner)
at java.lang.Object.wait(Object.java:502)
at jogamp.opengl.SharedResourceRunner.run(SharedResourceRunner.java:276)
- locked <0x0000000088aec748> (a jogamp.opengl.SharedResourceRunner)
at java.lang.Thread.run(Thread.java:745)
"AWT-EventQueue-0-SharedResourceRunner" #20 daemon prio=6 os_prio=0 tid=0x00007f28001f3000 nid=0xa63 in Object.wait() [0x00007f284f7f5000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000088aed588> (a jogamp.opengl.SharedResourceRunner)
at java.lang.Object.wait(Object.java:502)
at jogamp.opengl.SharedResourceRunner.run(SharedResourceRunner.java:276)
- locked <0x0000000088aed588> (a jogamp.opengl.SharedResourceRunner)
at java.lang.Thread.run(Thread.java:745)
"AWT-EventQueue-0" #13 prio=6 os_prio=0 tid=0x00007f286c444800 nid=0xa59 in Object.wait() [0x00007f2858913000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000dc467018> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at com.mycp.common.task.BMBTaskBase.startTask(BMBTaskBase.java:551)
- locked <0x00000000dc467018> (a java.lang.Object)
at com.mycp.uiapp.workmgmt.WorkMgmtMgr.sendBegCmd(WorkMgmtMgr.java:334)
at com.mycp.uiapp.workmgmt.WorkMgmtPanelBase.prepareAndSendBegWork(WorkMgmtPanelBase.java:559)
at com.mycp.uiapp.workmmgmt.WorkMgmtPanel.prepareAndSendBegWork(WorkMgmtPanel.java:1479)
at com.mycp.uiapp.workmgmt.WorkMgmtPanelBase.btnPrepareClicked(WorkMgmtPanelBase.java:363)
at com.mycp.uiapp.workmgmt.WorkMgmtPanel.btnPrepareClicked(WorkMgmtPanel.java:1412)
at com.mycp.uiapp.workmgmt.WorkMgmtPanel.actionPerformed(WorkMgmtPanel.java:1336)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2346)
"AWT-Shutdown" #14 prio=5 os_prio=0 tid=0x00007f286c443000 nid=0xa58 in Object.wait() [0x00007f2858a17000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000088ae8c28> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at sun.awt.AWTAutoShutdown.run(AWTAutoShutdown.java:295)
- locked <0x0000000088ae8c28> (a java.lang.Object)
at java.lang.Thread.run(Thread.java:745)
"AWT-XAWT" #12 daemon prio=6 os_prio=0 tid=0x00007f286c384000 nid=0xa51 runnable [0x00007f285914f000]
java.lang.Thread.State: RUNNABLE
at sun.awt.X11.XToolkit.waitForEvents(Native Method)
at sun.awt.X11.XToolkit.run(XToolkit.java:559)
at sun.awt.X11.XToolkit.run(XToolkit.java:523)
at java.lang.Thread.run(Thread.java:745)
"Java2D Disposer" #10 daemon prio=10 os_prio=0 tid=0x00007f286c35e000 nid=0xa50 in Object.wait() [0x00007f2859250000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000087ab7ec0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
- locked <0x0000000087ab7ec0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
at sun.java2d.Disposer.run(Disposer.java:148)
at java.lang.Thread.run(Thread.java:745)
"Thread-0" #9 prio=5 os_prio=0 tid=0x00007f286c234800 nid=0xa4f waiting on condition [0x00007f2859af5000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.mycp.logging.BMBLogging$Task.run(BMBLogging.java:1072)
at java.lang.Thread.run(Thread.java:745)
"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f286c0cf800 nid=0xa4d runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f286c0b2000 nid=0xa4c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f286c0b0000 nid=0xa4b waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f286c0ad800 nid=0xa4a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f286c0ab000 nid=0xa49 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f286c07c000 nid=0xa48 in Object.wait() [0x00007f285a2dd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000087a7e6c8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
- locked <0x0000000087a7e6c8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f286c07a000 nid=0xa47 in Object.wait() [0x00007f285a3de000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000087a7e708> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x0000000087a7e708> (a java.lang.ref.Reference$Lock)
"VM Thread" os_prio=0 tid=0x00007f286c072800 nid=0xa46 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f286c01e800 nid=0xa42 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f286c020800 nid=0xa43 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f286c022000 nid=0xa44 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f286c024000 nid=0xa45 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f286c0d2000 nid=0xa4e waiting on condition
JNI global references: 485
Heap
PSYoungGen total 118272K, used 98176K [0x00000000d6e00000, 0x00000000de700000, 0x0000000100000000)
eden space 113152K, 82% used [0x00000000d6e00000,0x00000000dc8e00c8,0x00000000ddc80000)
from space 5120K, 100% used [0x00000000de180000,0x00000000de680000,0x00000000de680000)
to space 5120K, 0% used [0x00000000ddc80000,0x00000000ddc80000,0x00000000de180000)
ParOldGen total 159744K, used 76671K [0x0000000084a00000, 0x000000008e600000, 0x00000000d6e00000)
object space 159744K, 47% used [0x0000000084a00000,0x00000000894dfc50,0x000000008e600000)
Metaspace used 30027K, capacity 30212K, committed 30464K, reserved 1077248K
class space used 3528K, capacity 3582K, committed 3584K, reserved 1048576K

The thread dump shows stack traces from about 22 different threads. Many of them look like application threads (as opposed to JVM internal threads). Most of the application threads are waiting for something. Which of those threads should not be waiting?
I'd start by looking at thread 13: Looks like the Swing EDT, and it's waiting inside a call to a button's actionPerformed(...) handler. That can't be good.

I think I am late in the game. Anyways, from your logs we could see that a thread is in Parking Waiting state.
"TimerQueue" #22 daemon prio=5 os_prio=0 tid=0x00007f28002a8800 nid=0xa65 waiting on condition [0x00007f284c56f000]
java.lang.Thread.State: WAITING (parking)
We could see that this thread is expecting something from DelayQueue.
DelayQueue-An unbounded blocking queue of Delayed elements, in which an element can only be taken when its delay has expired.
So In the same TimerQueue, we have
at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
This take() function waits if an element with expired delay is available on this queue.
This could be the reason for application hanging issue as this thread is still waiting and doesn't shutdown. So, there are still threads alive. To resolve this you will need to kill these threads.
For this you could just use ExecutorServices.shutdown() method. OR Simply you could use System.exit().
I would recommend you to use System.exit().

Java thread lock on System.err()

I'm a little baffled.
I have two threads writing to System.err (and System.out), and eventually they get stuck in a thread lock. Are System.err and System.out really not thread-safe?
The structure of the program at that point is:
main thread having launched two Reader threads to do some processing and printout, called via a ThreadPoolExecutor.
one Reader thread ("s3gp-2") that is blocked on System.err.format(), from within a separate IOStats object, in a synchronized method called IOStats.add().
another Reader thread that is (correctly) blocked trying to call the same IOStats.add() method that is blocked on System.err.format().
Note that each Reader typically writes to System.err and System.out.
Should I have a separate singleton object that handles all output with synchronized methods? Any insight very appreciated, as usual.
Update
I realize that I should give a bit more context: that java program is invoked in parallel by all the n segments of a Greenplum DB (via a create external web table ... execute prgm). The locking behavior is extremely rare and usually all goes well. When the problem occurs, all the n processes are blocked. I am focusing on just one of them for the debugging.
Some more notes:
For a few minutes, I thought that the fact that I also use log4j.Logger might cause trouble. But, in fact, there is no Console appender used for that program so I'm less inclined to think that is the problem.
using lsof as well as ls -l /proc/$pid/fd, I observe that the stderr of the java process that is blocked in a pipe that is also read by postgres. As one commenter suggested, it might be that this pipe is not actually drained by postgres, and that blocks the System.err print out.
In fact here are the relevant lines:
% lsof | head -1
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
% lsof | grep $pid | grep ' 2w '
java 10047 gpadmin 2w FIFO 0,6 26854739 pipe
% lsof | grep $pid | grep ' 2w '
java 10047 gpadmin 2w FIFO 0,6 26854739 pipe
% node=26854739
% lsof | grep $node
postgres 10028 gpadmin 26r FIFO 0,6 26854739 pipe
sh 10046 gpadmin 2w FIFO 0,6 26854739 pipe
java 10047 gpadmin 2w FIFO 0,6 26854739 pipe
Details: The stack dump (jstack -l pid) gives me:
2012-04-17 12:23:02
Full thread dump Java HotSpot(TM) 64-Bit Server VM (21.0-b17 mixed mode):
"Attach Listener" daemon prio=10 tid=0x00000000061a3800 nid=0x75c5 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"s3gp-2" prio=10 tid=0x00002aaabc207000 nid=0x281f runnable [0x0000000041718000]
java.lang.Thread.State: RUNNABLE
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:318)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
- locked <0x00000000f601c4f8> (a java.io.BufferedOutputStream)
at java.io.PrintStream.write(PrintStream.java:482)
- locked <0x00000000f601c4d8> (a java.io.PrintStream)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
- locked <0x00000000f601c6f0> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
at java.io.PrintStream.write(PrintStream.java:527)
- locked <0x00000000f601c4d8> (a java.io.PrintStream)
at java.io.PrintStream.print(PrintStream.java:669)
at java.io.PrintStream.append(PrintStream.java:1065)
at java.io.PrintStream.append(PrintStream.java:57)
at java.util.Formatter$FixedString.print(Formatter.java:2563)
at java.util.Formatter.format(Formatter.java:2476)
at java.io.PrintStream.format(PrintStream.java:970)
- locked <0x00000000f601c4d8> (a java.io.PrintStream)
at com.foo.serv.util.Loader$IOStats.add(Loader.java:136)
- locked <0x00000000f6015870> (a com.foo.serv.util.Loader$IOStats)
at com.foo.serv.util.Loader$Reader.run(Loader.java:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Locked ownable synchronizers:
- <0x00000000f6024470> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"s3gp-1" prio=10 tid=0x00002aaabc1fa000 nid=0x281e waiting for monitor entry [0x00000000405d9000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.foo.serv.util.Loader$IOStats.add(Loader.java:129)
- waiting to lock <0x00000000f6015870> (a com.foo.serv.util.Loader$IOStats)
at com.foo.serv.util.Loader$Reader.run(Loader.java:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Locked ownable synchronizers:
- <0x00000000f601ea50> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"Service Thread" daemon prio=10 tid=0x00002aaab0054000 nid=0x27e7 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" daemon prio=10 tid=0x00002aaab0051800 nid=0x27e6 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" daemon prio=10 tid=0x00002aaab004e800 nid=0x27e4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" daemon prio=10 tid=0x00002aaab004c000 nid=0x27e3 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" daemon prio=10 tid=0x00002aaab0001000 nid=0x2762 in Object.wait() [0x00000000428cc000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f6015730> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000000f6015730> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
Locked ownable synchronizers:
- None
"Reference Handler" daemon prio=10 tid=0x0000000005edc800 nid=0x2760 in Object.wait() [0x00000000427cb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f603b9c0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00000000f603b9c0> (a java.lang.ref.Reference$Lock)
Locked ownable synchronizers:
- None
"main" prio=10 tid=0x0000000005e3f800 nid=0x2744 waiting on condition [0x0000000041abe000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f61df188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1433)
at com.foo.serv.util.Loader.execute(Loader.java:107)
at com.foo.serv.util.Loader.main(Loader.java:571)
Locked ownable synchronizers:
- None
"VM Thread" prio=10 tid=0x0000000005ed5000 nid=0x275e runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x0000000005e4a000 nid=0x2745 runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000005e4c000 nid=0x2746 runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000005e4e000 nid=0x2747 runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000005e4f800 nid=0x2748 runnable
"GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000005e51800 nid=0x2749 runnable
"GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000005e53800 nid=0x274a runnable
"GC task thread#6 (ParallelGC)" prio=10 tid=0x0000000005e55000 nid=0x274b runnable
"GC task thread#7 (ParallelGC)" prio=10 tid=0x0000000005e57000 nid=0x274c runnable
"GC task thread#8 (ParallelGC)" prio=10 tid=0x0000000005e59000 nid=0x274d runnable
"GC task thread#9 (ParallelGC)" prio=10 tid=0x0000000005e5b000 nid=0x274e runnable
"GC task thread#10 (ParallelGC)" prio=10 tid=0x0000000005e5c800 nid=0x274f runnable
"GC task thread#11 (ParallelGC)" prio=10 tid=0x0000000005e5e800 nid=0x2750 runnable
"GC task thread#12 (ParallelGC)" prio=10 tid=0x0000000005e60800 nid=0x2751 runnable
"VM Periodic Task Thread" prio=10 tid=0x00002aaab005e800 nid=0x27eb waiting on condition
JNI global references: 207