First I'll summarize what I've found so far.
This answer suggests that changing the concurrencyLevel parameter of ConcurrentHashMap's constructor might help. I've tried that and my code still hanged.
Answers here suggest that it could be a runtime bug.
What I'm trying to do:
I have 10 worker threads running along with a main thread. The worker threads will have to process many arrays to find the index of the max element in the array (if there are multiple max values, the first occurrence will be used). Among these "many arrays," some of them can be duplicate, so I'm trying to avoid those full array scans to speed up the program.
The controller class contains a ConcurrentHashMap that maps the hash values of arrays to the corresponding max-element indices.
The worker threads will ask the controller class for the mapped index first before trying to calculate the index by doing full array scans. In the latter case, the newly calculated index will be put into the map.
The main thread does not access the hash map.
What happened:
My code will hang after 70,000 ~ 130,000 calls to getMaxIndex(). This count is obtained by putting a log string into getMaxIndex() so it might not be exactly accurate.
My CPU usage will gradually go up for ~6 seconds, and then it will go down to ~10% after peaked at ~100%. I have plenty of unused memory left. (Does this look like deadlock?)
If the code does not use map it works just fine (see getMaxIndex() version 2 below).
I've tried to add synchronized to getMaxIndex()'s signature and use the regular HashMap instead, that also did not work.
I've tried to use different initialCapacity values too (e.g. 50,000 & 100,000). Did not work.
Here's my code:
// in the controller class
int getMaxIndex(#NotNull double[] arr) {
int hash = Arrays.hashCode(arr);
if(maxIndices.containsKey(hash)) {
return maxIndices.get(hash);
} else {
int maxIndex =
IntStream.range(0, arr.length)
.reduce((a, b) -> arr[a] < arr[b] ? b : a)
.orElse(-1); // -1 to let program crash
maxIndices.put(hash, maxIndex);
return maxIndex;
}
}
The worker thread will call getMaxIndex() like this: return remaining[controller.getMaxIndex(arr)];, remaining is just another int array.
getMaxIndex() v2:
int getMaxIndex(#NotNull double[] arr) {
return IntStream.range(0, arr.length)
.reduce((a, b) -> arr[a] < arr[b] ? b : a)
.orElse(-1); // -1 to let program crash
}
JVM info in case it matters:
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
EDIT: stack dump; I used Phaser to synchronize the worker threads, so some of them appear to be waiting on the phaser, but pool-1-thread-2, pool-1-thread-10, pool-1-thread-11, and pool-1-thread-12 do not appear to be waiting on the phaser.
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.151-b12 mixed mode):
"Attach Listener" #23 daemon prio=9 os_prio=0 tid=0x00007f0c54001000 nid=0x4da2 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"pool-1-thread-13" #22 prio=5 os_prio=0 tid=0x00007f0c8c2cb800 nid=0x4d5e waiting on condition [0x00007f0c4eddd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e792f40> (a java.util.concurrent.Phaser$QNode)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.Phaser$QNode.block(Phaser.java:1140)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1067)
at java.util.concurrent.Phaser.arriveAndAwaitAdvance(Phaser.java:690)
at Ant.call(Ant.java:77)
at Ant.call(Ant.java:10)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-12" #21 prio=5 os_prio=0 tid=0x00007f0c8c2ca000 nid=0x4d5d waiting on condition [0x00007f0c4eede000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000775518738> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-11" #20 prio=5 os_prio=0 tid=0x00007f0c8c2c8000 nid=0x4d5c waiting on condition [0x00007f0c4efdf000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000775518738> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-10" #19 prio=5 os_prio=0 tid=0x00007f0c8c2c6000 nid=0x4d5b waiting on condition [0x00007f0c4f0e0000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000775518738> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-9" #18 prio=5 os_prio=0 tid=0x00007f0c8c2c4800 nid=0x4d5a waiting on condition [0x00007f0c4f1e1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e7c74f8> (a java.util.concurrent.Phaser$QNode)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.Phaser$QNode.block(Phaser.java:1140)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1067)
at java.util.concurrent.Phaser.arriveAndAwaitAdvance(Phaser.java:690)
at Ant.call(Ant.java:77)
at Ant.call(Ant.java:10)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-8" #17 prio=5 os_prio=0 tid=0x00007f0c8c2c2800 nid=0x4d59 waiting on condition [0x00007f0c4f2e2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e64fb78> (a java.util.concurrent.Phaser$QNode)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.Phaser$QNode.block(Phaser.java:1140)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1067)
at java.util.concurrent.Phaser.arriveAndAwaitAdvance(Phaser.java:690)
at Ant.call(Ant.java:77)
at Ant.call(Ant.java:10)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-7" #16 prio=5 os_prio=0 tid=0x00007f0c8c2c1000 nid=0x4d58 waiting on condition [0x00007f0c4f3e3000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e8b44c8> (a java.util.concurrent.Phaser$QNode)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.Phaser$QNode.block(Phaser.java:1140)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1067)
at java.util.concurrent.Phaser.arriveAndAwaitAdvance(Phaser.java:690)
at Ant.call(Ant.java:77)
at Ant.call(Ant.java:10)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-6" #15 prio=5 os_prio=0 tid=0x00007f0c8c2bf800 nid=0x4d57 waiting on condition [0x00007f0c4f4e4000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e5b4500> (a java.util.concurrent.Phaser$QNode)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.Phaser$QNode.block(Phaser.java:1140)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1067)
at java.util.concurrent.Phaser.arriveAndAwaitAdvance(Phaser.java:690)
at Ant.call(Ant.java:77)
at Ant.call(Ant.java:10)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-5" #14 prio=5 os_prio=0 tid=0x00007f0c8c2bd800 nid=0x4d56 waiting on condition [0x00007f0c4f5e5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e836958> (a java.util.concurrent.Phaser$QNode)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.Phaser$QNode.block(Phaser.java:1140)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1067)
at java.util.concurrent.Phaser.arriveAndAwaitAdvance(Phaser.java:690)
at Ant.call(Ant.java:77)
at Ant.call(Ant.java:10)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-4" #13 prio=5 os_prio=0 tid=0x00007f0c8c2bc000 nid=0x4d55 waiting on condition [0x00007f0c4f6e6000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e4f4cf0> (a java.util.concurrent.Phaser$QNode)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.Phaser$QNode.block(Phaser.java:1140)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1067)
at java.util.concurrent.Phaser.arriveAndAwaitAdvance(Phaser.java:690)
at Ant.call(Ant.java:77)
at Ant.call(Ant.java:10)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-3" #12 prio=5 os_prio=0 tid=0x00007f0c8c2ba000 nid=0x4d54 waiting on condition [0x00007f0c4f7e7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e40abb8> (a java.util.concurrent.Phaser$QNode)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.Phaser$QNode.block(Phaser.java:1140)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1067)
at java.util.concurrent.Phaser.arriveAndAwaitAdvance(Phaser.java:690)
at Ant.call(Ant.java:77)
at Ant.call(Ant.java:10)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-2" #11 prio=5 os_prio=0 tid=0x00007f0c8c2b8800 nid=0x4d53 waiting on condition [0x00007f0c4f8e8000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000775518738> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-1-thread-1" #10 prio=5 os_prio=0 tid=0x00007f0c8c2b5800 nid=0x4d52 waiting on condition [0x00007f0c4f9e9000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e486ab0> (a java.util.concurrent.Phaser$QNode)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.Phaser$QNode.block(Phaser.java:1140)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1067)
at java.util.concurrent.Phaser.arriveAndAwaitAdvance(Phaser.java:690)
at Ant.call(Ant.java:77)
at Ant.call(Ant.java:10)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f0c8c200800 nid=0x4d50 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f0c8c1fd800 nid=0x4d4f waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f0c8c1f8800 nid=0x4d4e waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f0c8c1f7800 nid=0x4d4d waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Monitor Ctrl-Break" #5 daemon prio=5 os_prio=0 tid=0x00007f0c8c1fb000 nid=0x4d4c runnable [0x00007f0c781b4000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
- locked <0x000000077550ecb0> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
- locked <0x000000077550ecb0> (a java.io.InputStreamReader)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:64)
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f0c8c181000 nid=0x4d49 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f0c8c14d800 nid=0x4d42 in Object.wait() [0x00007f0c78564000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000775500d08> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000000775500d08> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f0c8c149000 nid=0x4d41 in Object.wait() [0x00007f0c78665000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000775500d48> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x0000000775500d48> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=0 tid=0x00007f0c8c00c800 nid=0x4d35 waiting on condition [0x00007f0c91f77000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076dd5e268> (a java.util.concurrent.FutureTask)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at java.util.concurrent.AbstractExecutorService.invokeAll(AbstractExecutorService.java:244)
at ConcurrentACS.loop(ConcurrentACS.java:138)
at ConcurrentACS.compute(ConcurrentACS.java:165)
at ConcurrentACS.main(ConcurrentACS.java:192)
"VM Thread" os_prio=0 tid=0x00007f0c8c141800 nid=0x4d3f runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f0c8c022000 nid=0x4d37 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f0c8c024000 nid=0x4d38 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f0c8c025800 nid=0x4d39 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f0c8c027800 nid=0x4d3a runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f0c8c205800 nid=0x4d51 waiting on condition
JNI global references: 272
is it possible for ConcurrentHashMap to hang?
The short answer is no if by "hang" you mean some sort of program loop or deadlock. If you are implying that you have discovered a race condition (bug) in that code that would cause it to hang during normal JVM and system execution then I seriously doubt it.
I suspect that there is something else going on and just because you are using a CHM in the version that is hanging shouldn't imply that the class has a bug. I would use stack dumps or a profiler to show that the code is locked on a CHM line before I'd cast any blame that way.
Is it possible to be calling CHM at some large number of times per second so that the performance of your program suffers because of it? Sure. But it wouldn't hang in that it is stuck or deadlocked.
My CPU usage will gradually go up for ~6 seconds, and then it will go down to ~10% after peaked at ~100%. I have plenty of unused memory left. (Does this look like deadlock?)
Your now posted stack trace shows that no threads are locked in CHM code so it doesn't look to be the problem. The performance curve you are talking about seems to be happening because of the fork/join thread-pool that you are using initially starts X threads but then some of them finish their tasks and exit. This is to be expected. It has nothing to do with the CHM.
if(maxIndices.containsKey(hash)) {
return maxIndices.get(hash);
Just a quick comment. This code makes 2 calls to the CHM instead of something like:
Integer maxIndex = maxIndices.get(hash);
if (maxIndex != null) {
return maxIndex;
}
...
But that's just inefficient and wouldn't cause a bug. Also, it is important to recognize that race conditions in your code means that multiple threads might get a null for the index and calculate the index value. But also this is not a bug which would cause a "hang".
The first version isn't thread safe cause your check-than-act sequence isn't atomic. Try to use this implementation:
private final Map<Integer, Integer> maxIndices = new ConcurrentHashMap<>();
int getMaxIndex(final double[] arr) {
// make sure the content of the arr can't be modified concurrently
// otherwise create a copy of the array in this method
int hash = Arrays.hashCode(arr);
return maxIndices.computeIfAbsent(hash,
key -> IntStream.range(0, arr.length).reduce((a, b) -> arr[a] < arr[b] ? b : a).orElse(-1));
}
Related
My flink job, taskManager vm has too many threadPool, every pool contains one thread, thread names:"pool-{poolNumber}-thread-1".
thread dump:
// about 2200 threads like this
"pool-236-thread-1" #153 prio=5 os_prio=0 tid=0x00007f9e1c12d000 nid=0x245 waiting on condition [0x00007f9e40c66000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007a8082938> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
my debug:
I debug the job in my IDE, start the job.
there is one waiting thread names like 'pool-{poolNumber}-thread-1', which is "pool-5-thread-1".
I put a break on Thread.class's constructure.
the thread to create "pool-5-thread-1"'s stack is:
questions:
I don't know what to do next😢
Java spring boot application request goes in pending state as threads held in WAITING and TIMED_WAITING state.
Jstack logs:
"qtp886341817-1399" #1399 prio=5 os_prio=0 tid=0x00007f02142ae800 nid=0x22f904 waiting on condition [0x00007f01c3fa8000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000684588e00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.lang.Thread.run(Thread.java:748)
"threadPoolTaskExecutor-1" #114 prio=5 os_prio=0 tid=0x00007f02140b4800 nid=0x229d78 waiting on condition [0x00007f01c55b2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000684588e58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"qtp886341817-717" #717 prio=5 os_prio=0 tid=0x00007f021c102000 nid=0x22c546 in Object.wait() [0x00007f01ee774000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.eclipse.paho.client.mqttv3.internal.Token.waitUntilSent(Token.java:248)
- locked <0x0000000689516c80>
(a java.lang.Object) at org.eclipse.paho.client.mqttv3.MqttTopic.publish(MqttTopic.java:117)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.lang.Thread.run(Thread.java:748)
Details:
In this situation, the application is unable to serve API, the number of current threads went up to 250+, many threads go on deadlock state.
This spring application is hosted on AWS's t2.medium instance, Xms=1g, Xmx=2g, UseG1GC and we are using the jetty server.
This application generally servers a long thread wait APIs, it takes at least 12 to 60 seconds to respond to some of the APIs.
Questions:
Is there any way to find out how much threads can a spring application/JVM/jetty server can handle.
How can we tune this application to avoid such a situation (when application non-responsive)
How to restrict API's before this hung up situation.
Look here:
at org.eclipse.paho.client.mqttv3.internal.Token.waitUntilSent(Token.java:248)
- locked <0x0000000689516c80>
there is a lock, happened on timed out attempt to send message. Try to add here an asynchronous call.
I am writing a Java application to route a high number of concurrent messages. The application uses the Logback framework for logging and I am seeing a surprising behavior where the application hangs. In a stack trace, I can see that application threads are stuck in logging calls:
"New I/O client worker #1-1" #125 prio=5 os_prio=0 tid=0x00007f0524017000 nid=0x29f3 waiting on condition [0x00007f052ecea000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f089c4a7e88> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at java.util.concurrent.ArrayBlockingQueue.remainingCapacity(ArrayBlockingQueue.java:468)
at ch.qos.logback.core.AsyncAppenderBase.isQueueBelowDiscardingThreshold(AsyncAppenderBase.java:152)
at ch.qos.logback.core.AsyncAppenderBase.append(AsyncAppenderBase.java:144)
at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
at ch.qos.logback.classic.Logger.info(Logger.java:579)
at com.application.ClientListener$6.operationComplete(***.java:514)
- locked <0x00007f089c372b60> (a com.application.ClientListener)
at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:381)
at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:372)
at org.jboss.netty.channel.DefaultChannelFuture.setSuccess(DefaultChannelFuture.java:316)
at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:776)
at org.jboss.netty.channel.socket.nio.NioWorker.processRegisterTaskQueue(NioWorker.java:257)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:199)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00007f08a80fc118> (a java.util.concurrent.ThreadPoolExecutor$Worker)
It seems that the logging call is blocked trying to acquire an lock <0x00007f089c4a7e88>
inside a java.util.concurrent.ArrayBlockingQueue instance used in AsyncAppenderBase.
In the stack trace, I can see that the lock <0x00007f089c4a7e88> is held by another thread in a thread pool that is idle:
"dispatcher-3" #90 prio=5 os_prio=0 tid=0x00007f04d0004800 nid=0x29d2 waiting on condition [0x00007f0534ed3000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f089cbbaae8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00007f089c4a7e88> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
It looks like the internal lock of the ArrayBlockingQueue was held by that thread and subsequently
not released.
What is going on here? A race condition in java.util.concurrent.ArrayBlockingQueue? A bug in Logback?
I am using Java 8u40 and Logback 1.2.1.
You need to set asyncAppender neverBlock option to true for this to work.
I have the following thread dump from a hung java swing application. It hung after a button is clicked and the GUI changed to blank. Other threads in socket communication and task management are still working (from the log file I can tell). I have removed some non-relevant output.
The #13 AW-EventQueue-0 should send out a command through the socket but it seems failed there. The #20 and #21 are AW-EventQueue-0-SharedResourceRunner which is not the same as the #13? It seems there is no deadlock but the GUI is not responsive and became blank.
do you see any useful information about the cause of the hanging?
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.20-b23 mixed mode):
"DestroyJavaVM" #32 prio=5 os_prio=0 tid=0x00007f286c009800 nid=0xa41 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"TimerQueue" #22 daemon prio=5 os_prio=0 tid=0x00007f28002a8800 nid=0xa65 waiting on condition [0x00007f284c56f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000088a8f5c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
at javax.swing.TimerQueue.run(TimerQueue.java:171)
at java.lang.Thread.run(Thread.java:745)
"AWT-EventQueue-0-SharedResourceRunner" #21 daemon prio=6 os_prio=0 tid=0x00007f280021d000 nid=0xa64 in Object.wait() [0x00007f284d434000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000088aec748> (a jogamp.opengl.SharedResourceRunner)
at java.lang.Object.wait(Object.java:502)
at jogamp.opengl.SharedResourceRunner.run(SharedResourceRunner.java:276)
- locked <0x0000000088aec748> (a jogamp.opengl.SharedResourceRunner)
at java.lang.Thread.run(Thread.java:745)
"AWT-EventQueue-0-SharedResourceRunner" #20 daemon prio=6 os_prio=0 tid=0x00007f28001f3000 nid=0xa63 in Object.wait() [0x00007f284f7f5000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000088aed588> (a jogamp.opengl.SharedResourceRunner)
at java.lang.Object.wait(Object.java:502)
at jogamp.opengl.SharedResourceRunner.run(SharedResourceRunner.java:276)
- locked <0x0000000088aed588> (a jogamp.opengl.SharedResourceRunner)
at java.lang.Thread.run(Thread.java:745)
"AWT-EventQueue-0" #13 prio=6 os_prio=0 tid=0x00007f286c444800 nid=0xa59 in Object.wait() [0x00007f2858913000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000dc467018> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at com.mycp.common.task.BMBTaskBase.startTask(BMBTaskBase.java:551)
- locked <0x00000000dc467018> (a java.lang.Object)
at com.mycp.uiapp.workmgmt.WorkMgmtMgr.sendBegCmd(WorkMgmtMgr.java:334)
at com.mycp.uiapp.workmgmt.WorkMgmtPanelBase.prepareAndSendBegWork(WorkMgmtPanelBase.java:559)
at com.mycp.uiapp.workmmgmt.WorkMgmtPanel.prepareAndSendBegWork(WorkMgmtPanel.java:1479)
at com.mycp.uiapp.workmgmt.WorkMgmtPanelBase.btnPrepareClicked(WorkMgmtPanelBase.java:363)
at com.mycp.uiapp.workmgmt.WorkMgmtPanel.btnPrepareClicked(WorkMgmtPanel.java:1412)
at com.mycp.uiapp.workmgmt.WorkMgmtPanel.actionPerformed(WorkMgmtPanel.java:1336)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2346)
"AWT-Shutdown" #14 prio=5 os_prio=0 tid=0x00007f286c443000 nid=0xa58 in Object.wait() [0x00007f2858a17000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000088ae8c28> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at sun.awt.AWTAutoShutdown.run(AWTAutoShutdown.java:295)
- locked <0x0000000088ae8c28> (a java.lang.Object)
at java.lang.Thread.run(Thread.java:745)
"AWT-XAWT" #12 daemon prio=6 os_prio=0 tid=0x00007f286c384000 nid=0xa51 runnable [0x00007f285914f000]
java.lang.Thread.State: RUNNABLE
at sun.awt.X11.XToolkit.waitForEvents(Native Method)
at sun.awt.X11.XToolkit.run(XToolkit.java:559)
at sun.awt.X11.XToolkit.run(XToolkit.java:523)
at java.lang.Thread.run(Thread.java:745)
"Java2D Disposer" #10 daemon prio=10 os_prio=0 tid=0x00007f286c35e000 nid=0xa50 in Object.wait() [0x00007f2859250000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000087ab7ec0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
- locked <0x0000000087ab7ec0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
at sun.java2d.Disposer.run(Disposer.java:148)
at java.lang.Thread.run(Thread.java:745)
"Thread-0" #9 prio=5 os_prio=0 tid=0x00007f286c234800 nid=0xa4f waiting on condition [0x00007f2859af5000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.mycp.logging.BMBLogging$Task.run(BMBLogging.java:1072)
at java.lang.Thread.run(Thread.java:745)
"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f286c0cf800 nid=0xa4d runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f286c0b2000 nid=0xa4c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f286c0b0000 nid=0xa4b waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f286c0ad800 nid=0xa4a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f286c0ab000 nid=0xa49 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f286c07c000 nid=0xa48 in Object.wait() [0x00007f285a2dd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000087a7e6c8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
- locked <0x0000000087a7e6c8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f286c07a000 nid=0xa47 in Object.wait() [0x00007f285a3de000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000087a7e708> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x0000000087a7e708> (a java.lang.ref.Reference$Lock)
"VM Thread" os_prio=0 tid=0x00007f286c072800 nid=0xa46 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f286c01e800 nid=0xa42 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f286c020800 nid=0xa43 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f286c022000 nid=0xa44 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f286c024000 nid=0xa45 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f286c0d2000 nid=0xa4e waiting on condition
JNI global references: 485
Heap
PSYoungGen total 118272K, used 98176K [0x00000000d6e00000, 0x00000000de700000, 0x0000000100000000)
eden space 113152K, 82% used [0x00000000d6e00000,0x00000000dc8e00c8,0x00000000ddc80000)
from space 5120K, 100% used [0x00000000de180000,0x00000000de680000,0x00000000de680000)
to space 5120K, 0% used [0x00000000ddc80000,0x00000000ddc80000,0x00000000de180000)
ParOldGen total 159744K, used 76671K [0x0000000084a00000, 0x000000008e600000, 0x00000000d6e00000)
object space 159744K, 47% used [0x0000000084a00000,0x00000000894dfc50,0x000000008e600000)
Metaspace used 30027K, capacity 30212K, committed 30464K, reserved 1077248K
class space used 3528K, capacity 3582K, committed 3584K, reserved 1048576K
The thread dump shows stack traces from about 22 different threads. Many of them look like application threads (as opposed to JVM internal threads). Most of the application threads are waiting for something. Which of those threads should not be waiting?
I'd start by looking at thread 13: Looks like the Swing EDT, and it's waiting inside a call to a button's actionPerformed(...) handler. That can't be good.
I think I am late in the game. Anyways, from your logs we could see that a thread is in Parking Waiting state.
"TimerQueue" #22 daemon prio=5 os_prio=0 tid=0x00007f28002a8800 nid=0xa65 waiting on condition [0x00007f284c56f000]
java.lang.Thread.State: WAITING (parking)
We could see that this thread is expecting something from DelayQueue.
DelayQueue-An unbounded blocking queue of Delayed elements, in which an element can only be taken when its delay has expired.
So In the same TimerQueue, we have
at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
This take() function waits if an element with expired delay is available on this queue.
This could be the reason for application hanging issue as this thread is still waiting and doesn't shutdown. So, there are still threads alive. To resolve this you will need to kill these threads.
For this you could just use ExecutorServices.shutdown() method. OR Simply you could use System.exit().
I would recommend you to use System.exit().
Very rarely I get a deadlock while using wiki-java. Having a look at the full thread dump (acquired via kill -3 $JAVA-PID) suggests that the deadlock seems to be originating somewhere in the SSLSocketImpl. I'd prefer to avoid this deadlock in the first place (instead of doing some hacky recovery) but I am unsure how to find the cause and prevent it. Is there a way to set a timeout in the SSLSocketImpl or throw an exception in case of the deadlock? (It would be pretty straightforward to catch it in the main loop and redo the last call)
Full thread dump OpenJDK 64-Bit Server VM (24.51-b03 mixed mode):
"Service Thread" daemon prio=10 tid=0x00007f3cd816b000 nid=0x102c runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x00007f3cd8168800 nid=0x102b waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x00007f3cd8165800 nid=0x102a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007f3cd8163800 nid=0x1029 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00007f3cd8140800 nid=0x1028 waiting on condition [0x00007f3ccb9f7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007d77a9080> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:799)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:672)
at sun.security.ssl.SSLSocketImpl.sendAlert(SSLSocketImpl.java:2005)
at sun.security.ssl.SSLSocketImpl.warning(SSLSocketImpl.java:1832)
at sun.security.ssl.SSLSocketImpl.closeInternal(SSLSocketImpl.java:1600)
- locked <0x00000007d77a8d78> (a sun.security.ssl.SSLSocketImpl)
at sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:1538)
at sun.security.ssl.BaseSSLSocketImpl.finalize(BaseSSLSocketImpl.java:249)
at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101)
at java.lang.ref.Finalizer.access$100(Finalizer.java:32)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:190)
"Reference Handler" daemon prio=10 tid=0x00007f3cd813e800 nid=0x1027 in Object.wait() [0x00007f3ccbaf9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000078495a2c8> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x000000078495a2c8> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x00007f3cd8008000 nid=0x1021 waiting for monitor entry [0x00007f3cdfdb7000]
java.lang.Thread.State: BLOCKED (on object monitor)
at sun.security.ssl.SSLSocketImpl.getConnectionState(SSLSocketImpl.java:649)
- waiting to lock <0x00000007d77a8d78> (a sun.security.ssl.SSLSocketImpl)
at sun.security.ssl.SSLSocketImpl.isClosed(SSLSocketImpl.java:1446)
at java.net.Socket.getTcpNoDelay(Socket.java:965)
at sun.security.ssl.BaseSSLSocketImpl.getTcpNoDelay(BaseSSLSocketImpl.java:345)
at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:819)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:801)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
- locked <0x00000007d77a8d60> (a sun.security.ssl.AppOutputStream)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
- locked <0x00000007d77a8d48> (a java.io.BufferedOutputStream)
at java.io.PrintStream.flush(PrintStream.java:338)
- locked <0x00000007d77a8d28> (a java.io.PrintStream)
at sun.net.www.MessageHeader.print(MessageHeader.java:297)
- locked <0x00000007d6d057b0> (a sun.net.www.MessageHeader)
at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:599)
at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:610)
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:619)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1321)
- locked <0x00000007d6d05640> (a sun.net.www.protocol.https.DelegateHttpsURLConnection)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderFieldKey(HttpURLConnection.java:2731)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getHeaderFieldKey(HttpsURLConnectionImpl.java:307)
at shared.Wiki.grabCookies(Wiki.java:6907)
at shared.Wiki.fetch(Wiki.java:6462)
at shared.Wiki.getPageText(Wiki.java:1465)
at smallBots.Bot1.getText(Bot1.java:204)
at smallBots.Bot1.crawlCategory(Bot1.java:74)
at smallBots.Bot1.main(Bot1.java:49)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
"VM Thread" prio=10 tid=0x00007f3cd813a000 nid=0x1026 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f3cd801d800 nid=0x1022 runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f3cd801f800 nid=0x1023 runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f3cd8021800 nid=0x1024 runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f3cd8023000 nid=0x1025 runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f3cd8175800 nid=0x102d waiting on condition
JNI global references: 205
Found one Java-level deadlock:
=============================
"Finalizer":
waiting for ownable synchronizer 0x00000007d77a9080, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "main"
"main":
waiting to lock monitor 0x00007f3cac0015c8 (object 0x00000007d77a8d78, a sun.security.ssl.SSLSocketImpl),
which is held by "Finalizer"
Java stack information for the threads listed above:
===================================================
"Finalizer":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007d77a9080> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:799)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:672)
at sun.security.ssl.SSLSocketImpl.sendAlert(SSLSocketImpl.java:2005)
at sun.security.ssl.SSLSocketImpl.warning(SSLSocketImpl.java:1832)
at sun.security.ssl.SSLSocketImpl.closeInternal(SSLSocketImpl.java:1600)
- locked <0x00000007d77a8d78> (a sun.security.ssl.SSLSocketImpl)
at sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:1538)
at sun.security.ssl.BaseSSLSocketImpl.finalize(BaseSSLSocketImpl.java:249)
at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101)
at java.lang.ref.Finalizer.access$100(Finalizer.java:32)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:190)
"main":
at sun.security.ssl.SSLSocketImpl.getConnectionState(SSLSocketImpl.java:649)
- waiting to lock <0x00000007d77a8d78> (a sun.security.ssl.SSLSocketImpl)
at sun.security.ssl.SSLSocketImpl.isClosed(SSLSocketImpl.java:1446)
at java.net.Socket.getTcpNoDelay(Socket.java:965)
at sun.security.ssl.BaseSSLSocketImpl.getTcpNoDelay(BaseSSLSocketImpl.java:345)
at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:819)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:801)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
- locked <0x00000007d77a8d60> (a sun.security.ssl.AppOutputStream)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
- locked <0x00000007d77a8d48> (a java.io.BufferedOutputStream)
at java.io.PrintStream.flush(PrintStream.java:338)
- locked <0x00000007d77a8d28> (a java.io.PrintStream)
at sun.net.www.MessageHeader.print(MessageHeader.java:297)
- locked <0x00000007d6d057b0> (a sun.net.www.MessageHeader)
at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:599)
at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:610)
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:619)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1321)
- locked <0x00000007d6d05640> (a sun.net.www.protocol.https.DelegateHttpsURLConnection)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderFieldKey(HttpURLConnection.java:2731)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getHeaderFieldKey(HttpsURLConnectionImpl.java:307)
at shared.Wiki.grabCookies(Wiki.java:6907)
at shared.Wiki.fetch(Wiki.java:6462)
at shared.Wiki.getPageText(Wiki.java:1465)
at smallBots.Bot1.getText(Bot1.java:204)
at smallBots.Bot1.crawlCategory(Bot1.java:74)
at smallBots.Bot1.main(Bot1.java:49)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Found 1 deadlock.
Heap
PSYoungGen total 10752K, used 801K [0x00000007d6d00000, 0x00000007d7880000, 0x0000000800000000)
eden space 9728K, 1% used [0x00000007d6d00000,0x00000007d6d1fc20,0x00000007d7680000)
from space 1024K, 65% used [0x00000007d7780000,0x00000007d7828b40,0x00000007d7880000)
to space 1024K, 0% used [0x00000007d7680000,0x00000007d7680000,0x00000007d7780000)
ParOldGen total 93696K, used 69956K [0x0000000784800000, 0x000000078a380000, 0x00000007d6d00000)
object space 93696K, 74% used [0x0000000784800000,0x0000000788c51160,0x000000078a380000)
PSPermGen total 21504K, used 9537K [0x000000077a200000, 0x000000077b700000, 0x0000000784800000)
object space 21504K, 44% used [0x000000077a200000,0x000000077ab50720,0x000000077b700000)
The asker's answer: update to Java 8 to fix it.