We Keep Coding

java Finalizer thread blocked

Java's large number of threads are in a blocked state, waiting for a lock, but there is no thread to have this lock
blocked thread
[root#xxx xxx]# grep BLOCKED 11.txt -C5
at java.lang.ref.Reference.waitForReferencePendingList(java.base#12.0.2/Native Method)
at java.lang.ref.Reference.processPendingReferences(java.base#12.0.2/Reference.java:241)
at java.lang.ref.Reference$ReferenceHandler.run(java.base#12.0.2/Reference.java:213)
"Finalizer" #3 daemon prio=8 os_prio=0 cpu=273.67ms elapsed=1139.36s tid=0x00007fabf2c6a000 nid=0x11 waiting for monitor entry [0x00007fabb19fd000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.ref.Finalizer.runFinalizer(java.base#12.0.2/Finalizer.java:70)
- waiting to lock <0x000000048046f6f0> (a java.lang.Object)
at java.lang.ref.Finalizer$FinalizerThread.run(java.base#12.0.2/Finalizer.java:171)
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=1.42ms elapsed=1139.36s tid=0x00007fabf2c6b000 nid=0x12 runnable [0x0000000000000000]
--
at java.util.concurrent.LinkedBlockingQueue.take(java.base#12.0.2/LinkedBlockingQueue.java:433)
at com.alipay.remoting.ReconnectManager$HealConnectionRunner.run(ReconnectManager.java:144)
at java.lang.Thread.run(java.base#12.0.2/Thread.java:835)
"Init-TX-Disruptor-0" #26 prio=5 os_prio=0 cpu=4732.71ms elapsed=1122.35s tid=0x00007fabb40c5800 nid=0x5d waiting for monitor entry [0x00007fab939ec000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.ref.Finalizer.<init>(java.base#12.0.2/Finalizer.java:51)
- waiting to lock <0x000000048046f6f0> (a java.lang.Object)
at java.lang.ref.Finalizer.register(java.base#12.0.2/Finalizer.java:66)
at java.lang.Object.<init>(java.base#12.0.2/Object.java:50)
at org.rocksdb.AbstractNativeReference.<init>(AbstractNativeReference.java:25)
--
at xx.xxxx.xxx.xx.xxxx.xx.xx.onEvent(InitTxWorkHandler.java:24)
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143)
at java.lang.Thread.run(java.base#12.0.2/Thread.java:835)
"Init-TX-Disruptor-1" #27 prio=5 os_prio=0 cpu=4698.53ms elapsed=1122.35s tid=0x00007faba7b1a000 nid=0x5e waiting for monitor entry [0x00007fab9236b000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.ref.Finalizer.<init>(java.base#12.0.2/Finalizer.java:51)
- waiting to lock <0x000000048046f6f0> (a java.lang.Object)
at java.lang.ref.Finalizer.register(java.base#12.0.2/Finalizer.java:66)
at java.lang.Object.<init>(java.base#12.0.2/Object.java:50)
at org.rocksdb.AbstractNativeReference.<init>(AbstractNativeReference.java:25)
grep 0x000000048046f6f0 not found owns the monitor on 0x000000048046f6f0
grep 0x000000048046f6f0
The thread has been blocked for a long time, I want to know why

Using BufferedReader.lines() breaks in parallel

I'm writing some code that will be reading log lines and doing some processing in the background on that data. This processing would probably benifit from parallelization, such as what is offered by the Stream.parallel methods, and I was attempting to use this. This is the code I started out with that works perfectly.
public static void main(String[] args) {
try {
final Socket socket = new Socket(ADDRESS, PORT);
final BufferedReader reader = new BufferedReader(new InputStreamReader(socket.getInputStream()));
socket.getOutputStream().write(QUERY);
reader.lines().forEach(System.out::println);
} catch (IOException e) {
e.printStackTrace();
}
}
This code connects and prints out all of my data. I would very much like to restructure this code as follows:
public static void main(String[] args) {
try (Socket socket = new Socket(ADDRESS, PORT);
BufferedReader reader = new BufferedReader(new InputStreamReader(socket.getInputStream()))) {
socket.getOutputStream().write(QUERY);
reader.lines().forEach(System.out::println);
} catch (IOException e) {
e.printStackTrace();
}
}
But sadly this doesn't work. Even worse, going back to the original code, this doesn't even work:
public static void main(String[] args) {
try {
final Socket socket = new Socket(ADDRESS, PORT);
final BufferedReader reader = new BufferedReader(new InputStreamReader(socket.getInputStream()));
socket.getOutputStream().write(QUERY);
reader.lines().parallel().forEach(System.out::println);
} catch (IOException e) {
e.printStackTrace();
}
}
All that was added here was a .parallel call and this completely doesn't work. It just sits there and nothing is printed out.
I can live perfectly well and good without the 2nd version using the modified try(A a = new A()) {} as that doesn't look too good in this case. What I can't live without is figuring out why this .parallel call breaks everything.
I'm assuming the modified try statement is closing the streams as soon as I fall out of them (right after we start the forEach) so they are being killed and GC'd before operation. I can't for the life of me figure out what the hell is going on with the .parallel call.
As requested here is the output of jstack running on the .parellel() version of this code.
Full thread dump OpenJDK 64-Bit Server VM (25.112-b15 mixed mode):
"Attach Listener" #9 daemon prio=9 os_prio=0 tid=0x00007fd4f4001000 nid=0x4907 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fd5280be000 nid=0x48d2 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fd5280bb000 nid=0x48d1 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd5280b9800 nid=0x48d0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd5280b6800 nid=0x48cf waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd5280b5000 nid=0x48ce runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd528082000 nid=0x48cd in Object.wait() [0x00007fd515c6d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000ec008e98> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000ec008e98> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd52807d800 nid=0x48cc in Object.wait() [0x00007fd515d6e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000ec006b40> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000ec006b40> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=0 tid=0x00007fd528008000 nid=0x48c2 runnable [0x00007fd52fd9f000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
- locked <0x00000000ec086790> (a java.net.SocksSocketImpl)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at com.gravypod.Test.main(Test.java:48)
"VM Thread" os_prio=0 tid=0x00007fd528075800 nid=0x48ca runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd52801d800 nid=0x48c4 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fd52801f000 nid=0x48c5 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fd528021000 nid=0x48c6 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fd528022800 nid=0x48c7 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007fd5280c0800 nid=0x48d3 waiting on condition
JNI global references: 18
The Test.java:48 line is the Socket socket = new Socket line. This is the result of the fully-working non-parallel code (just using .lines()).
Full thread dump OpenJDK 64-Bit Server VM (25.112-b15 mixed mode):
"Attach Listener" #9 daemon prio=9 os_prio=0 tid=0x00007f9048001000 nid=0x4982 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f90800be800 nid=0x496f runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f90800bb000 nid=0x496e waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f90800b9800 nid=0x496d waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f90800b6800 nid=0x496c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f90800b5000 nid=0x496b runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f9080082000 nid=0x496a in Object.wait() [0x00007f907018d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000ec008e98> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000ec008e98> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f908007d800 nid=0x4969 in Object.wait() [0x00007f907028e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000ec006b40> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000ec006b40> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=0 tid=0x00007f9080008000 nid=0x4961 runnable [0x00007f90884c3000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
- locked <0x00000000ec08e890> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
- locked <0x00000000ec08e890> (a java.io.InputStreamReader)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at java.io.BufferedReader$1.hasNext(BufferedReader.java:571)
at java.util.Iterator.forEachRemaining(Iterator.java:115)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at com.gravypod.Test.main(Test.java:51)
"VM Thread" os_prio=0 tid=0x00007f9080075800 nid=0x4968 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f908001d800 nid=0x4963 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f908001f000 nid=0x4964 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f9080021000 nid=0x4965 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f9080022800 nid=0x4966 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f90800c1000 nid=0x4970 waiting on condition
JNI global references: 319
The line Test.java:51 is the reader.lines().forEach line.

I imagine that parallel() or the forEach() on a parallel stream waits to read all the input before parallelising the task. Because the server never closes the connection, it will wait forever.
Your task is not really parallelizable. Data comes sequentially over the wire, so reading it in parallel cannot work.

It seems, your application isn’t hanging, technically, but just waiting for a lot of input, before performing observable work. This is a combination of two implementation details. When you are starting a parallel stream operation, it will first try to split the workload until every CPU core has something to do, before actually starting to process elements. This combines with the Reader#lines() parallelizes badly due to nonconfigurable batch size issue.
Simply said, when a Stream has an unknown size, the implementation will try to buffer batches of sizes that are multiples of 1024, growing on each split. This great answer shows, how the splitting will happen for a stream of an unknown size with multiple cores, showing that multiples of 1024 elements will get buffered in the process. This can take a very long time, before the consumer passed to forEach is ever invoked.
Note that processing an infinite source via the non-short-circuiting forEach is outside the scope of the Stream API anyway. Assuming a timely side effect is an assumption about the processing order of the Stream, but there is no guaranty about it.
This answer guides you to a work-around. You can use something like
try(Socket socket = new Socket(ADDRESS, PORT);
BufferedReader reader = new BufferedReader(
new InputStreamReader(socket.getInputStream()))) {
socket.getOutputStream().write(QUERY);
Stream.generate(() -> {
try { return reader.readLine(); }
catch (IOException ex) { throw new UncheckedIOException(ex); }
}).parallel().forEach(System.out::println);
} catch(IOException|UncheckedIOException e) {
e.printStackTrace();
}
But, as said, this isn’t the intended use case of the Stream API…

why does trying to attach to myself crash my vm?

running the following code works fine on macos (sun jdk 1.8.0_51), but crashes every time with a thread dump on centOS (sun jdk 1.8.0 101) when it comes to attaching to itself, while succeeding to attach any other visible VM:
import com.sun.tools.attach.VirtualMachine;
import com.sun.tools.attach.VirtualMachineDescriptor;
public class Test {
public static void main(String[] args) throws Exception {
for(VirtualMachineDescriptor vmd : VirtualMachine.list()) {
VirtualMachine vm = VirtualMachine.attach(vmd);
}
}
}
the thread dump:
"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fbb780bf000 nid=0x28a0 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fbb780bc000 nid=0x289f runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fbb780ba000 nid=0x289e runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fbb780b7000 nid=0x289d runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fbb780b6000 nid=0x289c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fbb78082800 nid=0x289b in Object.wait() [0x00007fbb62eed000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000d6608ee0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000d6608ee0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fbb7807e000 nid=0x289a in Object.wait() [0x00007fbb62fee000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000d6606b50> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000d6606b50> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=0 tid=0x00007fbb78008000 nid=0x2894 waiting on condition [0x00007fbb7f307000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:100)
at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:78)
at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:250)
at Test.main(Test.java:10)
"VM Thread" os_prio=0 tid=0x00007fbb78076800 nid=0x2899 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fbb7801d800 nid=0x2895 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fbb7801f800 nid=0x2896 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fbb78021000 nid=0x2897 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fbb78023000 nid=0x2898 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007fbb780c2000 nid=0x28a1 waiting on condition
JNI global references: 19
Heap
PSYoungGen total 37888K, used 9835K [0x00000000d6600000, 0x00000000d9000000, 0x0000000100000000)
eden space 32768K, 30% used [0x00000000d6600000,0x00000000d6f9ad98,0x00000000d8600000)
from space 5120K, 0% used [0x00000000d8b00000,0x00000000d8b00000,0x00000000d9000000)
to space 5120K, 0% used [0x00000000d8600000,0x00000000d8600000,0x00000000d8b00000)
ParOldGen total 86016K, used 0K [0x0000000083200000, 0x0000000088600000, 0x00000000d6600000)
object space 86016K, 0% used [0x0000000083200000,0x0000000083200000,0x0000000088600000)
Metaspace used 3451K, capacity 4728K, committed 4864K, reserved 1056768K
class space used 382K, capacity 424K, committed 512K, reserved 1048576K
Exception in thread "main" com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:106)
at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:78)
at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:250)
at Test.main(Test.java:10)
can anyone explain?

How to handle a SSLSocketImpl Deadlock properly?

Very rarely I get a deadlock while using wiki-java. Having a look at the full thread dump (acquired via kill -3 $JAVA-PID) suggests that the deadlock seems to be originating somewhere in the SSLSocketImpl. I'd prefer to avoid this deadlock in the first place (instead of doing some hacky recovery) but I am unsure how to find the cause and prevent it. Is there a way to set a timeout in the SSLSocketImpl or throw an exception in case of the deadlock? (It would be pretty straightforward to catch it in the main loop and redo the last call)
Full thread dump OpenJDK 64-Bit Server VM (24.51-b03 mixed mode):
"Service Thread" daemon prio=10 tid=0x00007f3cd816b000 nid=0x102c runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x00007f3cd8168800 nid=0x102b waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x00007f3cd8165800 nid=0x102a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007f3cd8163800 nid=0x1029 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00007f3cd8140800 nid=0x1028 waiting on condition [0x00007f3ccb9f7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007d77a9080> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:799)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:672)
at sun.security.ssl.SSLSocketImpl.sendAlert(SSLSocketImpl.java:2005)
at sun.security.ssl.SSLSocketImpl.warning(SSLSocketImpl.java:1832)
at sun.security.ssl.SSLSocketImpl.closeInternal(SSLSocketImpl.java:1600)
- locked <0x00000007d77a8d78> (a sun.security.ssl.SSLSocketImpl)
at sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:1538)
at sun.security.ssl.BaseSSLSocketImpl.finalize(BaseSSLSocketImpl.java:249)
at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101)
at java.lang.ref.Finalizer.access$100(Finalizer.java:32)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:190)
"Reference Handler" daemon prio=10 tid=0x00007f3cd813e800 nid=0x1027 in Object.wait() [0x00007f3ccbaf9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000078495a2c8> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x000000078495a2c8> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x00007f3cd8008000 nid=0x1021 waiting for monitor entry [0x00007f3cdfdb7000]
java.lang.Thread.State: BLOCKED (on object monitor)
at sun.security.ssl.SSLSocketImpl.getConnectionState(SSLSocketImpl.java:649)
- waiting to lock <0x00000007d77a8d78> (a sun.security.ssl.SSLSocketImpl)
at sun.security.ssl.SSLSocketImpl.isClosed(SSLSocketImpl.java:1446)
at java.net.Socket.getTcpNoDelay(Socket.java:965)
at sun.security.ssl.BaseSSLSocketImpl.getTcpNoDelay(BaseSSLSocketImpl.java:345)
at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:819)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:801)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
- locked <0x00000007d77a8d60> (a sun.security.ssl.AppOutputStream)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
- locked <0x00000007d77a8d48> (a java.io.BufferedOutputStream)
at java.io.PrintStream.flush(PrintStream.java:338)
- locked <0x00000007d77a8d28> (a java.io.PrintStream)
at sun.net.www.MessageHeader.print(MessageHeader.java:297)
- locked <0x00000007d6d057b0> (a sun.net.www.MessageHeader)
at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:599)
at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:610)
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:619)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1321)
- locked <0x00000007d6d05640> (a sun.net.www.protocol.https.DelegateHttpsURLConnection)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderFieldKey(HttpURLConnection.java:2731)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getHeaderFieldKey(HttpsURLConnectionImpl.java:307)
at shared.Wiki.grabCookies(Wiki.java:6907)
at shared.Wiki.fetch(Wiki.java:6462)
at shared.Wiki.getPageText(Wiki.java:1465)
at smallBots.Bot1.getText(Bot1.java:204)
at smallBots.Bot1.crawlCategory(Bot1.java:74)
at smallBots.Bot1.main(Bot1.java:49)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
"VM Thread" prio=10 tid=0x00007f3cd813a000 nid=0x1026 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f3cd801d800 nid=0x1022 runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f3cd801f800 nid=0x1023 runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f3cd8021800 nid=0x1024 runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f3cd8023000 nid=0x1025 runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f3cd8175800 nid=0x102d waiting on condition
JNI global references: 205
Found one Java-level deadlock:
=============================
"Finalizer":
waiting for ownable synchronizer 0x00000007d77a9080, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "main"
"main":
waiting to lock monitor 0x00007f3cac0015c8 (object 0x00000007d77a8d78, a sun.security.ssl.SSLSocketImpl),
which is held by "Finalizer"
Java stack information for the threads listed above:
===================================================
"Finalizer":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007d77a9080> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:799)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:672)
at sun.security.ssl.SSLSocketImpl.sendAlert(SSLSocketImpl.java:2005)
at sun.security.ssl.SSLSocketImpl.warning(SSLSocketImpl.java:1832)
at sun.security.ssl.SSLSocketImpl.closeInternal(SSLSocketImpl.java:1600)
- locked <0x00000007d77a8d78> (a sun.security.ssl.SSLSocketImpl)
at sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:1538)
at sun.security.ssl.BaseSSLSocketImpl.finalize(BaseSSLSocketImpl.java:249)
at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101)
at java.lang.ref.Finalizer.access$100(Finalizer.java:32)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:190)
"main":
at sun.security.ssl.SSLSocketImpl.getConnectionState(SSLSocketImpl.java:649)
- waiting to lock <0x00000007d77a8d78> (a sun.security.ssl.SSLSocketImpl)
at sun.security.ssl.SSLSocketImpl.isClosed(SSLSocketImpl.java:1446)
at java.net.Socket.getTcpNoDelay(Socket.java:965)
at sun.security.ssl.BaseSSLSocketImpl.getTcpNoDelay(BaseSSLSocketImpl.java:345)
at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:819)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:801)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
- locked <0x00000007d77a8d60> (a sun.security.ssl.AppOutputStream)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
- locked <0x00000007d77a8d48> (a java.io.BufferedOutputStream)
at java.io.PrintStream.flush(PrintStream.java:338)
- locked <0x00000007d77a8d28> (a java.io.PrintStream)
at sun.net.www.MessageHeader.print(MessageHeader.java:297)
- locked <0x00000007d6d057b0> (a sun.net.www.MessageHeader)
at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:599)
at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:610)
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:619)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1321)
- locked <0x00000007d6d05640> (a sun.net.www.protocol.https.DelegateHttpsURLConnection)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderFieldKey(HttpURLConnection.java:2731)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getHeaderFieldKey(HttpsURLConnectionImpl.java:307)
at shared.Wiki.grabCookies(Wiki.java:6907)
at shared.Wiki.fetch(Wiki.java:6462)
at shared.Wiki.getPageText(Wiki.java:1465)
at smallBots.Bot1.getText(Bot1.java:204)
at smallBots.Bot1.crawlCategory(Bot1.java:74)
at smallBots.Bot1.main(Bot1.java:49)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Found 1 deadlock.
Heap
PSYoungGen total 10752K, used 801K [0x00000007d6d00000, 0x00000007d7880000, 0x0000000800000000)
eden space 9728K, 1% used [0x00000007d6d00000,0x00000007d6d1fc20,0x00000007d7680000)
from space 1024K, 65% used [0x00000007d7780000,0x00000007d7828b40,0x00000007d7880000)
to space 1024K, 0% used [0x00000007d7680000,0x00000007d7680000,0x00000007d7780000)
ParOldGen total 93696K, used 69956K [0x0000000784800000, 0x000000078a380000, 0x00000007d6d00000)
object space 93696K, 74% used [0x0000000784800000,0x0000000788c51160,0x000000078a380000)
PSPermGen total 21504K, used 9537K [0x000000077a200000, 0x000000077b700000, 0x0000000784800000)
object space 21504K, 44% used [0x000000077a200000,0x000000077ab50720,0x000000077b700000)

The asker's answer: update to Java 8 to fix it.

Google App Engine: Spinning While Scanning for JSPs During Deploy

Using 1.6.4.1 SDK, when I run appcfg.sh, I get to "5% Scanning for jsp files", and then
it spins forever (I've left it running overnight), consuming 100% of one of my CPUs. At first, I suspected that it was the same as the compiling JSPs issue, but I don't even get that far. It looks like some kind of bug in AbstractGlob and/or the javax.util.Regex. The project is here, in case you want to go duplicate it for yourself.
Anyone else have this experience? Any idea what might be causing it? Any way to circumvent it so that I can get my app uploaded?
Here's the stack traces that come from kill -QUIT:
1)
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.6-b01-415 mixed mode):
"Timer-0" daemon prio=5 tid=1029d0000 nid=0x10c304000 in Object.wait() [10c303000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7f45077b0> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <7f45077b0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Poller SunPKCS11-Darwin" daemon prio=1 tid=10b078000 nid=0x10c201000 waiting on condition [10c200000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:692)
at java.lang.Thread.run(Thread.java:680)
"Low Memory Detector" daemon prio=5 tid=10b00b800 nid=0x10bd01000 runnable [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=9 tid=10b00a800 nid=0x10ba04000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=9 tid=10b00a000 nid=0x10b901000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=9 tid=10b009000 nid=0x10a704000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=10b008000 nid=0x10a601000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=1028d2000 nid=0x10a0ef000 in Object.wait() [10a0ee000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7f44f0e08> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <7f44f0e08> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=1028d1000 nid=0x109fec000 in Object.wait() [109feb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7f4507570> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <7f4507570> (a java.lang.ref.Reference$Lock)
"main" prio=5 tid=102801000 nid=0x100501000 runnable [1004ff000]
java.lang.Thread.State: RUNNABLE
at java.lang.Character.codePointAt(Character.java:2335)
at java.util.regex.Pattern$CharProperty.match(Pattern.java:3344)
at java.util.regex.Pattern$Curly.match0(Pattern.java:3770)
at java.util.regex.Pattern$Curly.match(Pattern.java:3744)
at java.util.regex.Pattern$Curly.match0(Pattern.java:3782)
at java.util.regex.Pattern$Curly.match(Pattern.java:3744)
at java.util.regex.Pattern$Curly.match0(Pattern.java:3782)
at java.util.regex.Pattern$Curly.match(Pattern.java:3744)
at java.util.regex.Pattern$Curly.match0(Pattern.java:3782)
at java.util.regex.Pattern$Curly.match(Pattern.java:3744)
at java.util.regex.Pattern$Curly.match0(Pattern.java:3782)
at java.util.regex.Pattern$Curly.match(Pattern.java:3744)
at java.util.regex.Pattern$Slice.match(Pattern.java:3482)
at java.util.regex.Matcher.match(Matcher.java:1127)
at java.util.regex.Matcher.matches(Matcher.java:502)
at com.google.apphosting.utils.glob.AbstractGlob.matches(AbstractGlob.java:48)
at com.google.apphosting.utils.glob.AbstractGlob.matchesAll(AbstractGlob.java:56)
at com.google.apphosting.utils.glob.AbstractGlob.matchesAll(AbstractGlob.java:52)
at com.google.apphosting.utils.glob.GlobIntersector.reorderAndBlendChildren(GlobIntersector.java:95)
at com.google.apphosting.utils.glob.GlobIntersector.getIntersection(GlobIntersector.java:68)
at com.google.appengine.tools.admin.AppYamlTranslator$AbstractHandlerGenerator.getGlobPatterns(AppYamlTranslator.java:416)
at com.google.appengine.tools.admin.AppYamlTranslator$AbstractHandlerGenerator.size(AppYamlTranslator.java:390)
at com.google.appengine.tools.admin.AppYamlTranslator.translateWebXml(AppYamlTranslator.java:191)
at com.google.appengine.tools.admin.AppYamlTranslator.getYaml(AppYamlTranslator.java:83)
at com.google.appengine.tools.admin.Application.generateAppYaml(Application.java:897)
at com.google.appengine.tools.admin.Application.createStagingDirectory(Application.java:473)
at com.google.appengine.tools.admin.AppAdminImpl.doUpdate(AppAdminImpl.java:327)
at com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java:52)
at com.google.appengine.tools.admin.AppCfg$UpdateAction.execute(AppCfg.java:641)
at com.google.appengine.tools.admin.AppCfg.<init>(AppCfg.java:172)
at com.google.appengine.tools.admin.AppCfg.<init>(AppCfg.java:69)
at com.google.appengine.tools.admin.AppCfg.main(AppCfg.java:65)
"VM Thread" prio=9 tid=1028cc800 nid=0x109ee9000 runnable
"Gang worker#0 (Parallel GC Threads)" prio=9 tid=102802800 nid=0x102401000 runnable
"Gang worker#1 (Parallel GC Threads)" prio=9 tid=102803000 nid=0x102504000 runnable
"Gang worker#2 (Parallel GC Threads)" prio=9 tid=102803800 nid=0x102607000 runnable
"Gang worker#3 (Parallel GC Threads)" prio=9 tid=102804000 nid=0x10270a000 runnable
"Concurrent Mark-Sweep GC Thread" prio=9 tid=10287f800 nid=0x109bf0000 runnable
"VM Periodic Task Thread" prio=10 tid=10b01d000 nid=0x10be04000 waiting on condition
"Exception Catcher Thread" prio=10 tid=102801800 nid=0x1017f9000 runnable
JNI global references: 1671
Heap
par new generation total 19136K, used 13388K [7f3000000, 7f44c0000, 7f44c0000)
eden space 17024K, 77% used [7f3000000, 7f3cf67e8, 7f40a0000)
from space 2112K, 5% used [7f42b0000, 7f42cca00, 7f44c0000)
to space 2112K, 0% used [7f40a0000, 7f40a0000, 7f42b0000)
concurrent mark-sweep generation total 63872K, used 4234K [7f44c0000, 7f8320000, 7fae00000)
concurrent-mark-sweep perm gen total 21248K, used 14477K [7fae00000, 7fc2c0000, 800000000)
2)
^\2012-04-12 19:03:02
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.6-b01-415 mixed mode):
"Timer-0" daemon prio=5 tid=1029d0000 nid=0x10c304000 in Object.wait() [10c303000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7f45077b0> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
- locked <7f45077b0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Poller SunPKCS11-Darwin" daemon prio=1 tid=10b078000 nid=0x10c201000 waiting on condition [10c200000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:692)
at java.lang.Thread.run(Thread.java:680)
"Low Memory Detector" daemon prio=5 tid=10b00b800 nid=0x10bd01000 runnable [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=9 tid=10b00a800 nid=0x10ba04000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=9 tid=10b00a000 nid=0x10b901000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=9 tid=10b009000 nid=0x10a704000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=10b008000 nid=0x10a601000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=1028d2000 nid=0x10a0ef000 in Object.wait() [10a0ee000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7f44f0e08> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <7f44f0e08> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=1028d1000 nid=0x109fec000 in Object.wait() [109feb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7f4507570> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <7f4507570> (a java.lang.ref.Reference$Lock)
"main" prio=5 tid=102801000 nid=0x100501000 runnable [1004ff000]
java.lang.Thread.State: RUNNABLE
at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3366)
at java.util.regex.Pattern$Curly.match0(Pattern.java:3782)
at java.util.regex.Pattern$Curly.match(Pattern.java:3744)
at java.util.regex.Pattern$Curly.match0(Pattern.java:3782)
at java.util.regex.Pattern$Curly.match(Pattern.java:3744)
at java.util.regex.Pattern$Curly.match0(Pattern.java:3782)
at java.util.regex.Pattern$Curly.match(Pattern.java:3744)
at java.util.regex.Pattern$Curly.match0(Pattern.java:3782)
at java.util.regex.Pattern$Curly.match(Pattern.java:3744)
at java.util.regex.Pattern$Curly.match0(Pattern.java:3782)
at java.util.regex.Pattern$Curly.match(Pattern.java:3744)
at java.util.regex.Pattern$Curly.match0(Pattern.java:3782)
at java.util.regex.Pattern$Curly.match(Pattern.java:3744)
at java.util.regex.Pattern$Slice.match(Pattern.java:3482)
at java.util.regex.Matcher.match(Matcher.java:1127)
at java.util.regex.Matcher.matches(Matcher.java:502)
at com.google.apphosting.utils.glob.AbstractGlob.matches(AbstractGlob.java:48)
at com.google.apphosting.utils.glob.AbstractGlob.matchesAll(AbstractGlob.java:56)
at com.google.apphosting.utils.glob.AbstractGlob.matchesAll(AbstractGlob.java:52)
at com.google.apphosting.utils.glob.GlobIntersector.reorderAndBlendChildren(GlobIntersector.java:95)
at com.google.apphosting.utils.glob.GlobIntersector.getIntersection(GlobIntersector.java:68)
at com.google.appengine.tools.admin.AppYamlTranslator$AbstractHandlerGenerator.getGlobPatterns(AppYamlTranslator.java:416)
at com.google.appengine.tools.admin.AppYamlTranslator$AbstractHandlerGenerator.size(AppYamlTranslator.java:390)
at com.google.appengine.tools.admin.AppYamlTranslator.translateWebXml(AppYamlTranslator.java:191)
at com.google.appengine.tools.admin.AppYamlTranslator.getYaml(AppYamlTranslator.java:83)
at com.google.appengine.tools.admin.Application.generateAppYaml(Application.java:897)
at com.google.appengine.tools.admin.Application.createStagingDirectory(Application.java:473)
at com.google.appengine.tools.admin.AppAdminImpl.doUpdate(AppAdminImpl.java:327)
at com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java:52)
at com.google.appengine.tools.admin.AppCfg$UpdateAction.execute(AppCfg.java:641)
at com.google.appengine.tools.admin.AppCfg.<init>(AppCfg.java:172)
at com.google.appengine.tools.admin.AppCfg.<init>(AppCfg.java:69)
at com.google.appengine.tools.admin.AppCfg.main(AppCfg.java:65)
"VM Thread" prio=9 tid=1028cc800 nid=0x109ee9000 runnable
"Gang worker#0 (Parallel GC Threads)" prio=9 tid=102802800 nid=0x102401000 runnable
"Gang worker#1 (Parallel GC Threads)" prio=9 tid=102803000 nid=0x102504000 runnable
"Gang worker#2 (Parallel GC Threads)" prio=9 tid=102803800 nid=0x102607000 runnable
"Gang worker#3 (Parallel GC Threads)" prio=9 tid=102804000 nid=0x10270a000 runnable
"Concurrent Mark-Sweep GC Thread" prio=9 tid=10287f800 nid=0x109bf0000 runnable
"VM Periodic Task Thread" prio=10 tid=10b01d000 nid=0x10be04000 waiting on condition
"Exception Catcher Thread" prio=10 tid=102801800 nid=0x1017f9000 runnable
JNI global references: 1709
Heap
par new generation total 19136K, used 9948K [7f3000000, 7f44c0000, 7f44c0000)
eden space 17024K, 58% used [7f3000000, 7f39b6118, 7f40a0000)
from space 2112K, 0% used [7f40a0000, 7f40a1000, 7f42b0000)
to space 2112K, 0% used [7f42b0000, 7f42b0000, 7f44c0000)
concurrent mark-sweep generation total 63872K, used 4726K [7f44c0000, 7f8320000, 7fae00000)
concurrent-mark-sweep perm gen total 21248K, used 14489K [7fae00000, 7fc2c0000, 800000000)
3)
^\2012-04-12 19:03:50
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.6-b01-415 mixed mode):
"Timer-0" daemon prio=5 tid=1029d0000 nid=0x10c304000 in Object.wait() [10c303000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7f45077b0> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
- locked <7f45077b0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Poller SunPKCS11-Darwin" daemon prio=1 tid=10b078000 nid=0x10c201000 waiting on condition [10c200000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:692)
at java.lang.Thread.run(Thread.java:680)
"Low Memory Detector" daemon prio=5 tid=10b00b800 nid=0x10bd01000 runnable [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=9 tid=10b00a800 nid=0x10ba04000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=9 tid=10b00a000 nid=0x10b901000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=9 tid=10b009000 nid=0x10a704000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=10b008000 nid=0x10a601000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=1028d2000 nid=0x10a0ef000 in Object.wait() [10a0ee000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7f44f0e08> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <7f44f0e08> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=1028d1000 nid=0x109fec000 in Object.wait() [109feb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7f4507570> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <7f4507570> (a java.lang.ref.Reference$Lock)
"main" prio=5 tid=102801000 nid=0x100501000 runnable [1004ff000]
java.lang.Thread.State: RUNNABLE
at java.util.regex.Pattern$Curly.match0(Pattern.java:3789)
at java.util.regex.Pattern$Curly.match(Pattern.java:3744)
at java.util.regex.Pattern$Curly.match0(Pattern.java:3782)
at java.util.regex.Pattern$Curly.match(Pattern.java:3744)
at java.util.regex.Pattern$Curly.match0(Pattern.java:3782)
at java.util.regex.Pattern$Curly.match(Pattern.java:3744)
at java.util.regex.Pattern$Curly.match0(Pattern.java:3782)
at java.util.regex.Pattern$Curly.match(Pattern.java:3744)
at java.util.regex.Pattern$Curly.match0(Pattern.java:3782)
at java.util.regex.Pattern$Curly.match(Pattern.java:3744)
at java.util.regex.Pattern$Curly.match0(Pattern.java:3782)
at java.util.regex.Pattern$Curly.match(Pattern.java:3744)
at java.util.regex.Pattern$Slice.match(Pattern.java:3482)
at java.util.regex.Matcher.match(Matcher.java:1127)
at java.util.regex.Matcher.matches(Matcher.java:502)
at com.google.apphosting.utils.glob.AbstractGlob.matches(AbstractGlob.java:48)
at com.google.apphosting.utils.glob.AbstractGlob.matchesAll(AbstractGlob.java:56)
at com.google.apphosting.utils.glob.AbstractGlob.matchesAll(AbstractGlob.java:52)
at com.google.apphosting.utils.glob.GlobIntersector.reorderAndBlendChildren(GlobIntersector.java:95)
at com.google.apphosting.utils.glob.GlobIntersector.getIntersection(GlobIntersector.java:68)
at com.google.appengine.tools.admin.AppYamlTranslator$AbstractHandlerGenerator.getGlobPatterns(AppYamlTranslator.java:416)
at com.google.appengine.tools.admin.AppYamlTranslator$AbstractHandlerGenerator.size(AppYamlTranslator.java:390)
at com.google.appengine.tools.admin.AppYamlTranslator.translateWebXml(AppYamlTranslator.java:191)
at com.google.appengine.tools.admin.AppYamlTranslator.getYaml(AppYamlTranslator.java:83)
at com.google.appengine.tools.admin.Application.generateAppYaml(Application.java:897)
at com.google.appengine.tools.admin.Application.createStagingDirectory(Application.java:473)
at com.google.appengine.tools.admin.AppAdminImpl.doUpdate(AppAdminImpl.java:327)
at com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java:52)
at com.google.appengine.tools.admin.AppCfg$UpdateAction.execute(AppCfg.java:641)
at com.google.appengine.tools.admin.AppCfg.<init>(AppCfg.java:172)
at com.google.appengine.tools.admin.AppCfg.<init>(AppCfg.java:69)
at com.google.appengine.tools.admin.AppCfg.main(AppCfg.java:65)
"VM Thread" prio=9 tid=1028cc800 nid=0x109ee9000 runnable
"Gang worker#0 (Parallel GC Threads)" prio=9 tid=102802800 nid=0x102401000 runnable
"Gang worker#1 (Parallel GC Threads)" prio=9 tid=102803000 nid=0x102504000 runnable
"Gang worker#2 (Parallel GC Threads)" prio=9 tid=102803800 nid=0x102607000 runnable
"Gang worker#3 (Parallel GC Threads)" prio=9 tid=102804000 nid=0x10270a000 runnable
"Concurrent Mark-Sweep GC Thread" prio=9 tid=10287f800 nid=0x109bf0000 runnable
"VM Periodic Task Thread" prio=10 tid=10b01d000 nid=0x10be04000 waiting on condition
"Exception Catcher Thread" prio=10 tid=102801800 nid=0x1017f9000 runnable
JNI global references: 1709
Heap
par new generation total 19136K, used 9956K [7f3000000, 7f44c0000, 7f44c0000)
eden space 17024K, 58% used [7f3000000, 7f39b81b8, 7f40a0000)
from space 2112K, 0% used [7f40a0000, 7f40a1000, 7f42b0000)
to space 2112K, 0% used [7f42b0000, 7f42b0000, 7f44c0000)
concurrent mark-sweep generation total 63872K, used 4726K [7f44c0000, 7f8320000, 7fae00000)
concurrent-mark-sweep perm gen total 21248K, used 14489K [7fae00000, 7fc2c0000, 800000000)
[EDIT]
When I change my WebXml mapping for security-constraint from /** to /*, that solves the problem. My understanding is that this new security-constraint setting would match /foo but not /foo/bar: is that right? If so, then I might have a way to circumvent this bug. But it looks like a pretty nasty bug in AbstractGlob, where-ever one might find that code...

The Java GAE SDK has a version 1.6.4.1... It would seem like it addresses other problems, but it might be worth a try.
https://groups.google.com/d/msg/google-appengine/V4KHnq2f03Y/X24dzXVJyMQJ

When I change my WebXml mapping for security-constraint from /** to /*, that solves the problem. My understanding is that this new security-constraint setting would match /foo but not /foo/bar: is that right? If so, then I might have a way to circumvent this bug. But it looks like a pretty nasty bug in AbstractGlob, where-ever one might find that code...