I have a pretty complex java program, which doesn't terminate.
The eclipse debugger is showing a thread that can be suspended, but has no stack trace.
It is called "Thread-2".
The jstack -l output for this thread is:
"Thread-2" #17 prio=5 os_prio=0 tid=0x00007f1268002800 nid=0x3342 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
I added a breakpoint to Thread.start(), but I cannot find a thread called "Thread-2".
The thread just appears after the two "AWT-Event-Queue" threads were created.
I don't create any threads manually in my program.
After the main thread and all other threads exit, and the JFrame is disposed, the following threads still exist:
Thread [AWT-EventQueue-0] (Running)
Thread [Thread-2] (Running)
Thread [DestroyJavaVM] (Running)
When suspending the VM, the following threads exist:
Daemon System Thread [Signal Dispatcher] (Suspended)
Daemon System Thread [Finalizer] (Suspended)
Daemon System Thread [Reference Handler] (Suspended)
Daemon System Thread [Java2D Disposer] (Suspended)
Daemon System Thread [AWT-XAWT] (Suspended)
Thread [AWT-EventQueue-0] (Suspended)
Thread [Thread-2] (Suspended)
Thread [DestroyJavaVM] (Suspended)
How can I get more information about this thread, or allow it to terminate?
EDIT 1:
According to the Dependency Hierarchy of the eclipse pom.xml view, I use the following third party libraries:
guava 17.0 [compile]
hamcrest-core 1.3 [test]
junit 4.11 [test]
log4j-api 2.0-beta9 [compile]
log4j-core 2.0-beta9 [compile]
EDIT 2:
Adding breakpoints to all constructors of the thread class, as suggested in https://stackoverflow.com/a/35128213/577485, I see that Thread-0 and Thread-1 are created by log4j, but not Thread-2. It still just appears as before and no breakpoint triggers when it is constructed.
EDIT 3:
Now it's getting creepy. Not even the stop() method works, when invoked on the thread. I added it to the code given in https://stackoverflow.com/a/35128149/577485.
At least System.exit(int) still works. But as said in a comment, I don't want to use that.
EDIT 4:
Info about my system:
I'm running the newest stable version of Ubuntu 15.10 Wily. I have the security, updates and backports repos enabled.
My JVM version is:
java version "1.7.0_91"
OpenJDK Runtime Environment (IcedTea 2.6.3) (7u91-2.6.3-0ubuntu0.15.10.1)
OpenJDK 64-Bit Server VM (build 24.91-b01, mixed mode)
EDIT 5:
I executed the program with Java version jre-8u71-linux-x64 directly downloaded from java.com, but the error persists. jstack -l shows the same strange thread. Note that the program was still built with the older java version. Edit: After compiling it with java8u72 from oracle.com, I get the same behaviour.
EDIT 6:
I iterated over all the threads fields, here's the output. I cannot get any hint out of those fields, the thread doesn't even have a target.
name: [C#6b67034
priority: 5
threadQ: null
eetop: 140274638530560
single_step: false
daemon: false
stillborn: false
target: null
group: java.lang.ThreadGroup[name=main,maxpri=10]
contextClassLoader: null
inheritedAccessControlContext: java.security.AccessControlContext#0
threadInitNumber: 3
threadLocals: null
inheritableThreadLocals: null
stackSize: 0
nativeParkEventPointer: 0
tid: 17
threadSeqNumber: 20
threadStatus: 5
parkBlocker: null
blocker: null
blockerLock: java.lang.Object#16267862
MIN_PRIORITY: 1
NORM_PRIORITY: 5
MAX_PRIORITY: 10
EMPTY_STACK_TRACE: [Ljava.lang.StackTraceElement;#453da22c
SUBCLASS_IMPLEMENTATION_PERMISSION: ("java.lang.RuntimePermission" "enableContextClassLoaderOverride")
uncaughtExceptionHandler: null
defaultUncaughtExceptionHandler: null
threadLocalRandomSeed: 0
threadLocalRandomProbe: 0
threadLocalRandomSecondarySeed: 0
EDIT 7:
Added a watchpoint to the name field of Thread. It is only accessed by my analytics code, and seems to be never written...
EDIT 8:
jstack -F -m throws an error for my program:
Attaching to process ID 10973, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.71-b15
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
at sun.tools.jstack.JStack.main(JStack.java:106)
Caused by: java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007ff68000c000 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:169)
at sun.jvm.hotspot.runtime.Threads.first(Threads.java:153)
at sun.jvm.hotspot.tools.PStack.initJFrameCache(PStack.java:200)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:71)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:58)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:66)
at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
... 6 more
Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x00007ff68000c000
at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:165)
... 16 more
The class name of the strange thread is java.lang.Thread.
I don't use any command line arguments for executing the program. Adding the -Dlog4j2.disable.jmx=true option gives the strange thread the name Thread-1.
I updated log4j to 2.5, and the strange thread now has name Thread-0, when the -Dlog4j2.disable.jmx=true option is given, and Thread-1, when it isn't.
EDIT 9:
Completely removed log4j, error persists. The thread is now called Thread-0.
Here's the project, if that helps.
I don't understand why the breakpoint on Thread.start() did not work, but you could also try intercepting the thread >>creation<< by setting a breakpoint on the Thread constructors, or on the (internal) Thread.init() method.
The fact that the thread has the name Thread-2 implies that it was created by one of the constructors that generates a default thread name. That suggests that it was not created by the JVM or standard Java class libraries. It also narrows down the constructors that could have been used to create it.
How can I get more information about this thread ...
I can't think of any way apart from setting breakpoints.
... or allow it to terminate?
If you can find where it is created, you should be able to use setDaemon(true) to mark it as a daemon thread. However, this needs to be done before the thread is started.
Another possibility would be to find the thread by traversing the ThreadGroup tree and then calling Thread.interrupt() on it. ( Thread.getAllStackTraces() is another way of tracking down the thread object. ) However, there is no guarantee that the thread will "respect" the interrupt and shut down.
Finally, you could just call System.exit(...).
UPDATE
I mentioned that the thread might not respect interrupt() and I'm not surprised that stop() doesn't work. (It is deprecated, and may not even be implemented on some platforms.)
However, if you have managed to implement code that actually finds the mystery thread, you could dig around to find either the Thread subclass, or the Runnable that it is instantiated with. If you can print out the fully qualified class name, that will give you a big clue as to where it comes from. (Assuming that you are still having no success with breakpoints, then you may need to use "nasty" reflection to extract the runnable from the thread's private target field.)
Not sure if this enough for you, but the following code will allow you to try to interrupt any Thread by its name:
//Set of current Threads
Set<Thread> setOfThread = Thread.getAllStackTraces().keySet();
//Iterate over set to find yours
for(Thread thread : setOfThread){
if (thread.getName().equals("Thread-2")) {
thread.interrupt();
break;
}
}
Also, take a look on this article from JavaSpecialists that tries to identify the creator of a Thread based on the fact that the constructor of Thread makes a call to the security manager. If we add a custom SecurityManager to our System, we may track the initiator of a Thread.
First off, you should change your _exit flag to volatile, since it's read from one thread (your main method), and written by another (JCF/Swing event handler), so it's possible your main thread isn't getting a "fresh" value. Specifically: the thread may be saving the field to a CPU register and not reloading it from memory as you're looping. 'volatile' will prevent that behavior:
private volatile boolean _exit;
However, based on your stack traces I don't think that's your problem, since we don't see your main method in there. But this should be done anyway, it's just good practice.
Assuming that doesn't fix it, I'm guessing your problem is that you have at least one other Window (besides AgentFrame) that isn't being disposed. The AWT thread won't stop until all Windows are disposed.
Put this at the end of your main method:
System.out.println(Arrays.toString(Window.getWindows()))
I'm guessing you're going to see more than just your DrawFrame there. If I were to guess, I'd say UISettingsFrame is in there, undisposed.
Related
I facing a problem with standard java http client. By some reason that I don't know yet it go into infinite loop and utilize all cpus I have. Below is an example stack trace and a method where the infinity loop is running.
"HttpClient-1-Worker-34" #232 daemon prio=5 os_prio=0 cpu=37704.71ms elapsed=231.22s tid=0x00007f1580279000 nid=0x31fc runnable [0x00007f1575059000]
java.lang.Thread.State: RUNNABLE
at jdk.internal.net.http.common.SSLFlowDelegate$Writer.processData(java.net.http#11.0.1/SSLFlowDelegate.java:758)
at jdk.internal.net.http.common.SSLFlowDelegate$Writer$WriterDownstreamPusher.run(java.net.http#11.0.1/SSLFlowDelegate.java:645)
at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(java.net.http#11.0.1/SequentialScheduler.java:147)
at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(java.net.http#11.0.1/SequentialScheduler.java:198)
at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(java.net.http#11.0.1/SequentialScheduler.java:271)
at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(java.net.http#11.0.1/SequentialScheduler.java:224)
at jdk.internal.net.http.common.SSLFlowDelegate$Writer.triggerWrite(java.net.http#11.0.1/SSLFlowDelegate.java:722)
at jdk.internal.net.http.common.SSLFlowDelegate.doHandshake(java.net.http#11.0.1/SSLFlowDelegate.java:1024)
at jdk.internal.net.http.common.SSLFlowDelegate.doClosure(java.net.http#11.0.1/SSLFlowDelegate.java:1094)
at jdk.internal.net.http.common.SSLFlowDelegate$Reader.unwrapBuffer(java.net.http#11.0.1/SSLFlowDelegate.java:500)
at jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData(java.net.http#11.0.1/SSLFlowDelegate.java:389)
- locked <0x0000000613d63450> (a java.lang.Object)
at jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run(java.net.http#11.0.1/SSLFlowDelegate.java:263)
at jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(java.net.http#11.0.1/SequentialScheduler.java:175)
- locked <0x0000000613d63488> (a java.lang.Object)
at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(java.net.http#11.0.1/SequentialScheduler.java:147)
at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(java.net.http#11.0.1/SequentialScheduler.java:198)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base#11.0.1/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base#11.0.1/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base#11.0.1/Thread.java:834)
I thought that root cause of the problem is this one. But after disabling TLSv1.3 the issue is not go away.
I checked if TLSv1.3 is disable in the following way using debug at start up of application:
Here is JVM parameters I pass in:
-Djdk.tls.disabledAlgorithms="TLSv1.3"
-Dhttps.protocols="TLSv1,TLSv1.1,TLSv1.2"
-Djdk.tls.client.protocols="TLSv1,TLSv1.1,TLSv1.2"
-Djdk.tls.server.protocols="TLSv1,TLSv1.1,TLSv1.2"
to be sure that TLSv1.3 is disabled.
Please, any ideas what can be a reason of the issue?
The case is that an application hangs infinitely from time to time.
Seems that the bug sits in the following snippet:
ForkJoinPool pool = new ForkJoinPool(1); // parallelism = 1
List<String> entries = ...;
pool.submit(() -> {
entries.stream().parallel().forEach(entry -> {
// An I/O op.
...
});
}).get();
Thread pool-4-thread-1 that executes the code freezes on get():
"pool-4-thread-1" #35 prio=5 os_prio=0 tid=0x00002b42e4013800 nid=0xb7d1 in Object.wait() [0x00002b427b72f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.concurrent.ForkJoinTask.externalInterruptibleAwaitDone(ForkJoinTask.java:367)
- locked <0x00000000e08b68b8> (a java.util.concurrent.ForkJoinTask$AdaptedRunnableAction)
at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1001)
...other app methods
One can assume that the task passed to submit() executes too long.
But surprisingly there is no ForkJoinPool-N-worker-N occurrences in the thread dump, so looks like the pool doesn't perform any computations!
How is that possible? If no tasks are executed by the pool, why pool-4-thread-1 thread waits inside get()?
P.S. I know that it's not recommended to execute I/O-related tasks in ForkJoinPool, but still interested in the root of the problem.
Update. When parallelism is set to value greater than 1, no problems are detected.
Set parallelism = N where N > 1 solved the problem.
Strange thing but seems that there is some bug in ForkJoinPool similar to what is stated here.
While executing script using JMeter in non gui mode i'm getting error message described below when Thread group count is 100 and ramp up period is 60 sec
Tidying up ...
The JVM should have exitted but did not.
The following non-daemon threads are still running (DestroyJavaVM is OK):
Thread[StandardJMeterEngine,5,main], stackTrace:sun.nio.fs.WindowsNativeDispatcher#CopyFileEx0
sun.nio.fs.WindowsNativeDispatcher#CopyFileEx
sun.nio.fs.WindowsFileCopy#copy
sun.nio.fs.WindowsFileSystemProvider#copy
java.nio.file.Files#copy
org.apache.jmeter.report.dashboard.TemplateVisitor#visitFile at line:126
org.apache.jmeter.report.dashboard.TemplateVisitor#visitFile at line:48
java.nio.file.Files#walkFileTree
java.nio.file.Files#walkFileTree
org.apache.jmeter.report.dashboard.HtmlTemplateExporter#export at line:487
org.apache.jmeter.report.dashboard.ReportGenerator#exportData at line:348
org.apache.jmeter.report.dashboard.ReportGenerator#generate at line:256
org.apache.jmeter.JMeter$ListenToTest#generateReport at line:1144
org.apache.jmeter.JMeter$ListenToTest#testEnded at line:1089
org.apache.jmeter.engine.StandardJMeterEngine#notifyTestListenersOfEnd at line:215
org.apache.jmeter.engine.StandardJMeterEngine#run at line:436
java.lang.Thread#run
Thread[DestroyJavaVM,5,main], stackTrace:
... end of run
How to resolve this error? what exactly this error means? How to handle this error?
This is a bug fixed in jmeter 3.1 (to be released soon):
https://bz.apache.org/bugzilla/show_bug.cgi?id=60053
Report generation is in progress while you have this message.
Wait for it to be generated and kill the jvm instance:
kill pid
Learning and getting a hands on java thread dump I created a thread dump of my IDEA intellij running process.
The problem is that I don't understand what it means.
Can you explain what this information means and how to read it ?
No need to list every single detail just the main fields and what each means and what it indicates as information.
Here is a small snippet
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode):
"JobScheduler FJ pool 0/4" #165 daemon prio=6 os_prio=0 tid=0x00007f9cb0001800 nid=0x1d2a waiting on condition [0x00007f9ca7e1e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e74c3c90> (a jsr166e.ForkJoinPool)
at jsr166e.ForkJoinPool.awaitWork(ForkJoinPool.java:1756)
at jsr166e.ForkJoinPool.scan(ForkJoinPool.java:1694)
at jsr166e.ForkJoinPool.runWorker(ForkJoinPool.java:1642)
at jsr166e.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:109)
and here is the whole file
<Name> <nature> <vm priority> <os priority><vm id> <os id> <status [adress]>
<thread state>
<current execution point>
- detail : wait until timeout or thread 0x00000000e74c3c90 notification
<stack>
As far as I understand you have a job scheduler worker thread which has been released to its thread pool (and is waiting for a new job)
See also How to analyze a java thread dump?
I've seen other references to this issue, such as here and here, although these reference different versions of Netty. Tried this using the latest in the 4.0 branch (4.0.29) and in the 5.0 alpha branch (5.0-Alpha3). Local (non-linux) jdk 1.8.040, fine. Remote (Linux) with java jdk 1.8.025-b17 get 100% cpu.
Linux kernel version 2.6.32.
Tried using EpollEventLoopGroup();
Tried calling
workerGroup = new NioEventLoopGroup();
workerGroup.rebuildSelectors();
Can anyone offer any suggestions? I've seen references to this bug w/different versions of Netty. Jdk bug? Netty bug? Process goes to 100% immediately on startup and stays there.
Update: Upgraded to java 1.8.045, same difference.
JStack output of all runnable threads (there's some rabbitmq stuff in there, only included for completeness - that's common to other applications, and is not the cause of the problem).
As we identified in the comments, the thread that consumed CPU is busy in the following stack:
"pool-9-thread-1" #49 prio=5 os_prio=0 tid=0x00007ffd508e8000 nid=0x3a0c runnable [0x00007ffd188b6000]
java.lang.Thread.State: RUNNABLE
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.poll(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
I have managed to reproduce a similar behavior by creating a ScheduledThreadPoolExecutor, configuring it to allow core threads to time out, and scheduling a lot of repeating tasks with a short delay. It yields a lot of CPU on my machine and the jstack output is similar (sometimes deeper into the poll method). This code reproduces it:
ScheduledThreadPoolExecutor executor = new ScheduledThreadPoolExecutor(1);
executor.setKeepAliveTime(1, TimeUnit.MINUTES);
executor.allowCoreThreadTimeOut(true);
for (long i = 0; i < 1000; i++) {
executor.scheduleAtFixedRate(new Runnable() {
#Override
public void run() {
}
}, 0, 1, TimeUnit.NANOSECONDS);
}
Now we just have to identify which code sets up a broken ScheduledThreadPoolExecutor. I searched through the RabbitMQ and Netty source code without finding anything obvoius. Could it be something you do in your own code?
Edit: As mentioned in the comments, the root cause was a ScheduledThreadPoolExecutor initialized with 0 which apparently can cause a CPU spin om some platforms. This was done in the OP's code.