Looking through a thread dump I see several BLOCKED threads but some have "waiting to lock " and some have "waiting on " while there is only one RUNNABLE thread that has presumably acquired the lock. The question I have is why are some "waiting to lock" and others "waiting on"? Also the thread that is "waiting on" appears to have the lock but is not the RUNNABLE thread that actually holds the lock as it says it locked the object at address lower in the stack trace, so why is it blocking if it has the lock (or does it not have the lock)?
Examples below:
The locked object is at 0x00000000c2ad4be8
waiting on
"ajp-bio-10032-exec-15" daemon prio=10 tid=0x00007f888c02d000 nid=0x74c8 in Object.wait() [0x00007f8821792000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c2ad4be8> (a java.lang.Object)
at java.lang.Object.wait(Object.java:503)
at com.tibco.tibjms.TibjmsxLinkTcp.disconnect(TibjmsxLinkTcp.java:1026)
waiting to lock
"ajp-bio-10032-exec-95" daemon prio=10 tid=0x00007f888c089800 nid=0x1e66 in Object.wait() [0x00007f881df4b000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c2ad4be8> (a java.lang.Object)
at java.lang.Object.wait(Object.java:503)
at com.tibco.tibjms.TibjmsxLinkTcp.disconnect(TibjmsxLinkTcp.java:1026)
- locked <0x00000000c2ad4be8> (a java.lang.Object)
at com.tibco.tibjms.TibjmsConnection._close(TibjmsConnection.java:2430)
and finally the RUNNABLE thread
"ajp-bio-10032-exec-142" daemon prio=10 tid=0x00007f888c0b7000 nid=0x3657 runnable [0x00007f88167d4000]
java.lang.Thread.State: RUNNABLE
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:713)
- locked <0x00000000f022c5e0> (a com.tibco.tibjms.TibjmsxLinkTcp$LinkReader)
at com.tibco.tibjms.TibjmsxLinkTcp.start(TibjmsxLinkTcp.java:968)
- locked <0x00000000c2ad4be8> (a java.lang.Object)
Found one Java-level deadlock:
"Thread-13":
waiting to lock monitor 0x000000000b297688 (object 0x00000000ef3c3528, a com.xxx.models.SCSToolScript),
which is held by "Thread-11"
"Thread-11":
waiting to lock monitor 0x000000000b297738 (object 0x00000000fd777fd0, a java.lang.Object),
which is held by "Thread-13"
Java stack information for the threads listed above:
"Thread-13":
at com.xxx.models.SCSToolScript.isStopped(SCSToolScript.java:116)
- waiting to lock <0x00000000ef3c3528> (a com.xxx.models.SCSToolScript)
at com.xxx.actions.SCSToolManager.isStopped(SCSToolManager.java:352)
- locked <0x00000000fd777fd0> (a java.lang.Object)
at com.xxx.actions.ExecuteScript.checkStopped(ExecuteScript.java:1620)
at com.xxx.actions.ExecuteScript.RCCommand(ExecuteScript.java:745)
at com.xxx.actions.ExecuteScript.access$800(ExecuteScript.java:37)
at com.xxx.actions.ExecuteScript$SCSThread.run(ExecuteScript.java:2112)
at java.lang.Thread.run(Thread.java:745)
"Thread-11":
at com.xxx.actions.SCSToolManager.isPaused(SCSToolManager.java:346)
- waiting to lock <0x00000000fd777fd0> (a java.lang.Object)
at com.xxx.actions.ExecuteScript.checkPaused(ExecuteScript.java:1616)
at com.xxx.actions.ExecuteScript.RCCommand(ExecuteScript.java:751)
- locked <0x00000000ef3c3528> (a com.xxx.models.SCSToolScript)
at com.xxx.actions.ExecuteScript.access$800(ExecuteScript.java:37)
at com.xxx.actions.ExecuteScript$SCSThread.run(ExecuteScript.java:2112)
at java.lang.Thread.run(Thread.java:745)
Found 1 deadlock.
I am launching a batch of traversal on an embedded Neo4J DB 2.0.3.
23M nodes
87M relations
16GB Heap
Tried different cache settings
Tried different thread settings ( From 20 to 5 threads )
The Job is running fine for some time (ex: 1h) and then the throughput slows down dramatically because the threads are spending most of their time waiting for locks.
It looks like the memory buffers (MappedPersistenceWindow) can't be shared by multiple threads, which sounds a bit weird.
Some samples from the thread dump :
"taskExecutor-6" - Thread t#17
java.lang.Thread.State: BLOCKED
at java.lang.Object.wait(Native Method)
- waiting on <3a67f9f7> (a org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow)
at java.lang.Object.wait(Object.java:503)
at org.neo4j.kernel.impl.nioneo.store.LockableWindow.lock(LockableWindow.java:96)
at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:197)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:430)
at org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getRecord(RelationshipStore.java:84)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction$3.load(NeoStoreTransaction.java:208)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction$3.load(NeoStoreTransaction.java:198)
at org.neo4j.kernel.impl.nioneo.xa.RecordChanges.getOrLoad(RecordChanges.java:63)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.relLoadLight(NeoStoreTransaction.java:1189)
at org.neo4j.kernel.impl.persistence.PersistenceManager.loadLightRelationship(PersistenceManager.java:109)
at org.neo4j.kernel.impl.core.NodeManager$2.loadById(NodeManager.java:114)
at org.neo4j.kernel.impl.core.NodeManager$2.loadById(NodeManager.java:110)
at org.neo4j.kernel.impl.cache.AutoLoadingCache.get(AutoLoadingCache.java:93)
at org.neo4j.kernel.impl.core.NodeManager.getRelationshipForProxy(NodeManager.java:544)
at org.neo4j.kernel.InternalAbstractGraphDatabase$6.lookupRelationship(InternalAbstractGraphDatabase.java:849)
at org.neo4j.kernel.impl.core.RelationshipProxy.getType(RelationshipProxy.java:141)
"taskExecutor-5" - Thread t#16
java.lang.Thread.State: RUNNABLE
at org.neo4j.kernel.impl.nioneo.store.LockableWindow.markAsInUse(LockableWindow.java:70)
- locked <6abe6e1> (a org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow)
at org.neo4j.kernel.impl.nioneo.store.BrickElement.getAndMarkWindow(BrickElement.java:94)
at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:147)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:430)
at org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getChainRecord(RelationshipStore.java:325)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.getMoreRelationships(NeoStoreTransaction.java:2331)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.getMoreRelationships(NeoStoreTransaction.java:1390)
at org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:94)
at org.neo4j.kernel.impl.core.RelationshipLoader.getMoreRelationships(RelationshipLoader.java:50)
at org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:779)
at org.neo4j.kernel.impl.core.NodeImpl.loadMoreRelationshipsFromNodeManager(NodeImpl.java:577)
at org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:540)
- locked <2b29ed5b> (a org.neo4j.kernel.impl.core.NodeImpl)
at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:98)
at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:36)
at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:55)
at org.neo4j.kernel.impl.core.NodeImpl.hasRelationship(NodeImpl.java:644)
at org.neo4j.kernel.impl.core.NodeProxy.hasRelationship(NodeProxy.java:183)
"taskExecutor-4" - Thread t#15
java.lang.Thread.State: BLOCKED
at java.lang.Object.wait(Native Method)
- waiting on <3a67f9f7> (a org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow)
at java.lang.Object.wait(Object.java:503)
at org.neo4j.kernel.impl.nioneo.store.LockableWindow.lock(LockableWindow.java:96)
at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:197)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:430)
at org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getChainRecord(RelationshipStore.java:325)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.getMoreRelationships(NeoStoreTransaction.java:2331)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.getMoreRelationships(NeoStoreTransaction.java:1390)
at org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:94)
at org.neo4j.kernel.impl.core.RelationshipLoader.getMoreRelationships(RelationshipLoader.java:50)
at org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:779)
at org.neo4j.kernel.impl.core.NodeImpl.loadMoreRelationshipsFromNodeManager(NodeImpl.java:577)
at org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:540)
- locked <41fe8c4f> (a org.neo4j.kernel.impl.core.NodeImpl)
at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:98)
at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:36)
at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:55)
at org.neo4j.kernel.impl.core.NodeImpl.hasRelationship(NodeImpl.java:644)
at org.neo4j.kernel.impl.core.NodeProxy.hasRelationship(NodeProxy.java:183)
"taskExecutor-3" - Thread t#14
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <3a67f9f7> (a org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow)
at java.lang.Object.wait(Object.java:503)
at org.neo4j.kernel.impl.nioneo.store.LockableWindow.lock(LockableWindow.java:96)
at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:197)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:430)
at org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getRecord(RelationshipStore.java:84)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction$3.load(NeoStoreTransaction.java:208)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction$3.load(NeoStoreTransaction.java:198)
at org.neo4j.kernel.impl.nioneo.xa.RecordChanges.getOrLoad(RecordChanges.java:63)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.relLoadLight(NeoStoreTransaction.java:1189)
at org.neo4j.kernel.impl.persistence.PersistenceManager.loadLightRelationship(PersistenceManager.java:109)
at org.neo4j.kernel.impl.core.NodeManager$2.loadById(NodeManager.java:114)
at org.neo4j.kernel.impl.core.NodeManager$2.loadById(NodeManager.java:110)
at org.neo4j.kernel.impl.cache.AutoLoadingCache.get(AutoLoadingCache.java:93)
at org.neo4j.kernel.impl.core.NodeManager.getRelationshipForProxy(NodeManager.java:544)
at org.neo4j.kernel.InternalAbstractGraphDatabase$6.lookupRelationship(InternalAbstractGraphDatabase.java:849)
at org.neo4j.kernel.impl.core.RelationshipProxy.getOtherNode(RelationshipProxy.java:108)
at org.neo4j.kernel.impl.traversal.TraversalBranchImpl.next(TraversalBranchImpl.java:145)
at org.neo4j.graphdb.traversal.PreorderDepthFirstSelector.next(PreorderDepthFirstSelector.java:49)
at org.neo4j.kernel.impl.traversal.MonoDirectionalTraverserIterator.fetchNextOrNull(MonoDirectionalTraverserIterator.java:68)
at org.neo4j.kernel.impl.traversal.MonoDirectionalTraverserIterator.fetchNextOrNull(MonoDirectionalTraverserIterator.java:35)
at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:55)
Any idea ?
I didn't notice it until today during some testing locally on my pc, that Tomcat had posted an error in the log file. I'm using Tomcat 6.0.29 and Java JDK 1.6.
Dec 1, 2010 12:36:57 pm org.apache.catalina.core.StandardContext reload
INFO: Reloading this Context has started
Dec 1, 2010 12:36:57 pm org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVER: The web application [/AutoSpyder] appears to have started a thread named
[AWT-Windows] but has failed to stop it. This is very likely to create a memory
leak.
What? I've never seen this before. So I checked my log file from yesterday, and sure enough, this error was there too. I don't quite understand what's causing this.
Can I assume that it would have to be one of my servlets that uses objects from the java.awt package? If so, how do I pinpoint the code that is causing this?
Edited to add Thread Dump
2010-12-01 14:28:18
Full thread dump Java HotSpot(TM) Client VM (17.1-b03 mixed mode, sharing):
"JMX server connection timeout 34" daemon prio=6 tid=0x03069400 nid=0x960 in Object.wait() [0x0461f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a [I)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
- locked (a [I)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"RMI Scheduler(0)" daemon prio=6 tid=0x03069000 nid=0xe88 waiting on condition [0x045cf000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"RMI TCP Connection(1)-192.168.0.102" daemon prio=6 tid=0x0308a400 nid=0xebc runnable [0x0457f000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
- locked (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
"RMI TCP Accept-0" daemon prio=6 tid=0x039e0c00 nid=0xc68 runnable [0x0452f000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"AWT-Windows" daemon prio=6 tid=0x02b65400 nid=0x7c4 runnable [0x042cf000]
java.lang.Thread.State: RUNNABLE
at sun.awt.windows.WToolkit.eventLoop(Native Method)
at sun.awt.windows.WToolkit.run(WToolkit.java:293)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"Java2D Disposer" daemon prio=10 tid=0x02fccc00 nid=0x93c in Object.wait() [0x039df000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at sun.java2d.Disposer.run(Disposer.java:127)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"http-8080-6" daemon prio=6 tid=0x03322800 nid=0xfec in Object.wait() [0x0395f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
- locked (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"http-8080-5" daemon prio=6 tid=0x02ba5c00 nid=0xdbc in Object.wait() [0x0390f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
- locked (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"http-8080-4" daemon prio=6 tid=0x02ff6400 nid=0xa1c in Object.wait() [0x038bf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
- locked (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"http-8080-3" daemon prio=6 tid=0x0317e400 nid=0x850 in Object.wait() [0x0386f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
- locked (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"http-8080-2" daemon prio=6 tid=0x03314800 nid=0xf9c in Object.wait() [0x0381f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
- locked (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"http-8080-1" daemon prio=6 tid=0x02efe800 nid=0x250 in Object.wait() [0x037cf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
- locked (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"TP-Monitor" daemon prio=6 tid=0x02eed800 nid=0xd64 in Object.wait() [0x0375f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:565)
- locked (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"TP-Processor4" daemon prio=6 tid=0x0318b000 nid=0x998 runnable [0x0370f000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:312)
at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:666)
at org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:877)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"TP-Processor3" daemon prio=6 tid=0x0308f800 nid=0x92c in Object.wait() [0x036bf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
- locked (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"TP-Processor2" daemon prio=6 tid=0x03192400 nid=0xfac in Object.wait() [0x0366f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
- locked (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"TP-Processor1" daemon prio=6 tid=0x03182400 nid=0x8d8 in Object.wait() [0x0361f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
- locked (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"http-8080-Acceptor-0" daemon prio=6 tid=0x03172400 nid=0xf04 runnable [0x035cf000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:352)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=6 tid=0x03163400 nid=0xbe8 waiting on condition [0x0357f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1579)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"GC Daemon" daemon prio=2 tid=0x0307bc00 nid=0x110 in Object.wait() [0x0349f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:100)
- locked (a sun.misc.GC$LatencyLock)
Locked ownable synchronizers:
- None
"Low Memory Detector" daemon prio=6 tid=0x02aecc00 nid=0x5b4 runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"CompilerThread0" daemon prio=10 tid=0x02ae7000 nid=0x798 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Attach Listener" daemon prio=10 tid=0x02ae5800 nid=0xddc waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" daemon prio=10 tid=0x02ae4000 nid=0xc00 runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" daemon prio=8 tid=0x02add400 nid=0x378 in Object.wait() [0x02caf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Locked ownable synchronizers:
- None
"Reference Handler" daemon prio=10 tid=0x02adbc00 nid=0x474 in Object.wait() [0x02c5f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked (a java.lang.ref.Reference$Lock)
Locked ownable synchronizers:
- None
"main" prio=6 tid=0x002b7000 nid=0x14c runnable [0x0090f000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
at org.apache.catalina.startup.Catalina.await(Catalina.java:662)
at org.apache.catalina.startup.Catalina.start(Catalina.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Locked ownable synchronizers:
- None
"VM Thread" prio=10 tid=0x02ada400 nid=0x210 runnable
"VM Periodic Task Thread" prio=10 tid=0x02aefc00 nid=0x79c waiting on condition
JNI global references: 1529
Try taking a thread dump to get a list of all the threads running on your server. From this, you may be able to pinpoint exactly which class started the thread based on the stack traces
Windows
on the console window press ctrl + break
Unix
in a command line enter: kill -3 <pid>
It's to do with Windows (i.e. MS Windows) and native calls to the underlying OS. These would normally be GUI-type window operations, but could be other OS-specific things, e.g Clipboard or Font related, for instance.
If there's nothing obvious that you can see in your code that might relate to why it's being triggered, though, I'd not worry about it - that warning's more for the general case where spinning off new threads in a web-app isn't something you should be explicitly doing without a full appreciation of the potential issues.
We have tomcat server and jsf. JSF client will request images from the server and cause AWT-Windows thread to get started. Appears to cause a memory leak since threads contextClassLoader is the WebappClassLoader.
Looks like calling Java2DResource can caused AWT-Windows thread to be created -
Daemon Thread [http-8080-1] (Suspended (entry into method in Thread))
Thread.(Runnable, String) line: 444 <-- thread name is AWT-Windows
WToolkit.() line: 244
NativeConstructorAccessorImpl.newInstance0(Constructor, Object[]) line: not available [native method]
NativeConstructorAccessorImpl.newInstance(Object[]) line: 39
DelegatingConstructorAccessorImpl.newInstance(Object[]) line: 27
Constructor.newInstance(Object...) line: 513
Class.newInstance0() line: 355
Class.newInstance() line: 308
Toolkit$2.run() line: 846
AccessController.doPrivileged(PrivilegedAction) line: not available [native method]
Toolkit.getDefaultToolkit() line: 826
D3DGraphicsDevice.() line: 47
SurfaceManagerFactory.createCachingManager(BufferedImage) line: 48
SurfaceManager.getManager(Image) line: 54
SurfaceData.getDestSurfaceData(Image) line: 123
Win32GraphicsEnvironment(SunGraphicsEnvironment).createGraphics(BufferedImage) line: 389
HeadlessGraphicsEnvironment.createGraphics(BufferedImage) line: 76
BufferedImage.createGraphics() line: 1137
TabStripeImage(Java2Dresource).getImage(ResourceContext) line: 115
TabStripeImage(Java2Dresource).send(ResourceContext) line: 89
ResourceLifecycle.sendResource(ResourceContext, InternetResource) line: 219
ResourceLifecycle.send(ResourceContext, InternetResource) line: 158
InternetResourceService.load(Object, Object) line: 335
LRUMapCache.load(Object, Object) line: 116
LRUMapCache.get(Object, Object) line: 87
InternetResourceService.serviceResource(String, HttpServletRequest, HttpServletResponse) line: 195
InternetResourceService.serviceResource(HttpServletRequest, HttpServletResponse) line: 141
Filter(BaseFilter).doFilter(ServletRequest, ServletResponse, FilterChain) line: 508
Ajax4jsfFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 56
SeamFilter$FilterChainImpl.doFilter(ServletRequest, ServletResponse) line: 69
LoggingFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 60
SeamFilter$FilterChainImpl.doFilter(ServletRequest, ServletResponse) line: 69
SeamFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 158
ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 235
ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 206
StandardWrapperValve.invoke(Request, Response) line: 233
StandardContextValve.invoke(Request, Response) line: 191
StandardHostValve.invoke(Request, Response) line: 127
ErrorReportValve.invoke(Request, Response) line: 102
AccessLogValve.invoke(Request, Response) line: 555
StandardEngineValve.invoke(Request, Response) line: 109
CoyoteAdapter.service(Request, Response) line: 298
Http11AprProcessor.process(long) line: 861
Http11AprProtocol$Http11ConnectionHandler.process(long) line: 579
AprEndpoint$Worker.run() line: 1584
Thread.run() line: 619 [local variables unavailable]
It looks like you are running JMX and I'm getting the impression that the Tomcat JMX server may be what's actually setting up the timer thread. In addition to the ones you mentioned in your parallel post on the JavaRanch, there's a java2D thread in the dump you have here.
I can't tell 100% because I don't know which threads were the parents for the orphan threads. If you could determine that, it would help.
Your Affine Transform is definitely pulling in graphics, however. It's a graphical function that will attempt to use your video card as a hardware accelerator. If you don't specify headless operation, it will tap into your windowing system to do so.
Inspired by this thread I found the following solution:
You only need to attach the AWT-Window-Thread to System-Classloader instead to Wabapp-Classloader. You can do it in a ServletContextListener like this:
public void contextInitialized(ServletContextEvent evt) {
Thread thread = Thread.currentThread();
ClassLoader ccl = thread.getContextClassLoader(); // PUSH
try {
thread.setContextClassLoader(ClassLoader.getSystemClassLoader());
Toolkit.getDefaultToolkit().createImage(new byte[]{});
} finally {
thread.setContextClassLoader(ccl); // POP
}
}
Are you running including some kind of graphical library like Jasper, JFreechart, LiquidOffice, StyleReport, ... that might interact with AWT ?
Try starting tomcat by providing the -Djava.awt.headless=true property to see that gets rid of the exception
I start a jetty server on linux using
nohup mvn jetty:run &
to run it in background and to stay when I quit from the console. Usually (mvn jetty:run) it starts in seconds, but in this case it takes about minute to start. What can be the cause of such a delay?
Stack trace
Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.0-b16 mixed mode):
"Attach Listener" daemon prio=10 tid=0x000000004bba8c00 nid=0x560a waiting on condition [0x0000000000000000..0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Timer-1" daemon prio=10 tid=0x00002aaac053d800 nid=0x55fb in Object.wait() [0x0000000042392000..0x0000000042392b90]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaaae554f40> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x00002aaaae554f40> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"StopJettyPluginMonitor" daemon prio=10 tid=0x00002aaac04b4c00 nid=0x55fa runnable [0x0000000042291000..0x0000000042291b10]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked <0x00002aaaae453860> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.mortbay.jetty.plugin.util.Monitor.run(Monitor.java:74)
"Timer-0" daemon prio=10 tid=0x000000004b5cf800 nid=0x55f9 in Object.wait() [0x0000000040c78000..0x0000000040c78a90]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaaaeea00b8> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x00002aaaaeea00b8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"1455093129#qtp-422797318-1 - Acceptor0 SelectChannelConnector#0.0.0.0:8080" prio=10 tid=0x000000004b6d9800 nid=0x55f7 runnable [0x000000004130c000..0x000000004130ca10]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00002aaaae44ac10> (a sun.nio.ch.Util$1)
- locked <0x00002aaaae44abf8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00002aaaae44a588> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:459)
at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:706)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
"91446054#qtp-422797318-0" prio=10 tid=0x000000004b621400 nid=0x55f6 in Object.wait() [0x000000004120b000..0x000000004120bd90]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab1171890> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:563)
- locked <0x00002aaab1171890> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
"pool-1-thread-5" prio=10 tid=0x00002aaac15d4c00 nid=0x55f5 in Object.wait() [0x0000000041a4f000..0x0000000041a4fd10]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaaafff4580> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
at java.lang.Object.wait(Object.java:485)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:316)
- locked <0x00002aaaafff4580> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:994)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1054)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-4" prio=10 tid=0x00002aaac0134000 nid=0x55f4 in Object.wait() [0x000000004194e000..0x000000004194ec90]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaaafff4580> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
at java.lang.Object.wait(Object.java:485)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:316)
- locked <0x00002aaaafff4580> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:994)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1054)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-3" prio=10 tid=0x000000004ba72800 nid=0x55f3 in Object.wait() [0x0000000042190000..0x0000000042190c10]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaaafff4580> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
at java.lang.Object.wait(Object.java:485)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:316)
- locked <0x00002aaaafff4580> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:994)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1054)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-2" prio=10 tid=0x000000004b8b6c00 nid=0x55f2 in Object.wait() [0x000000004208f000..0x000000004208fb90]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaaafff4580> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
at java.lang.Object.wait(Object.java:485)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:316)
- locked <0x00002aaaafff4580> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:994)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1054)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-1" prio=10 tid=0x000000004b8b6400 nid=0x55f1 in Object.wait() [0x0000000040415000..0x0000000040415b10]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaaafff4580> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
at java.lang.Object.wait(Object.java:485)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:316)
- locked <0x00002aaaafff4580> (a hidden.edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:994)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1054)
at hidden.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
at java.lang.Thread.run(Thread.java:619)
"Low Memory Detector" daemon prio=10 tid=0x000000004affd800 nid=0x55eb runnable [0x0000000000000000..0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x000000004affb800 nid=0x55ea waiting on condition [0x0000000000000000..0x0000000041d8b4d0]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x000000004aff7800 nid=0x55e9 waiting on condition [0x0000000000000000..0x0000000041c8a450]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x000000004aff5c00 nid=0x55e8 runnable [0x0000000000000000..0x0000000041b8a820]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x000000004afd2400 nid=0x55e7 in Object.wait() [0x000000004184d000..0x000000004184db10]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaaafdf0450> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x00002aaaafdf0450> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x000000004afd0800 nid=0x55e6 in Object.wait() [0x000000004174c000..0x000000004174ca90]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaaafdf0530> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00002aaaafdf0530> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x000000004af6e000 nid=0x55e4 in Object.wait() [0x000000004072f000..0x0000000040730ed0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab1196b60> (a org.mortbay.thread.QueuedThreadPool$Lock)
at java.lang.Object.wait(Object.java:485)
at org.mortbay.thread.QueuedThreadPool.join(QueuedThreadPool.java:298)
- locked <0x00002aaab1196b60> (a org.mortbay.thread.QueuedThreadPool$Lock)
at org.mortbay.jetty.plugin.Jetty6PluginServer.join(Jetty6PluginServer.java:206)
at org.mortbay.jetty.plugin.AbstractJettyMojo.startJetty(AbstractJettyMojo.java:461)
at org.mortbay.jetty.plugin.AbstractJettyMojo.execute(AbstractJettyMojo.java:383)
at org.mortbay.jetty.plugin.AbstractJettyRunMojo.execute(AbstractJettyRunMojo.java:210)
at org.mortbay.jetty.plugin.Jetty6RunMojo.execute(Jetty6RunMojo.java:184)
at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:483)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:678)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeStandaloneGoal(DefaultLifecycleExecutor.java:553)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(DefaultLifecycleExecutor.java:523)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:371)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:332)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:181)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:356)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:137)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:362)
at org.apache.maven.cli.compat.CompatibleMain.main(CompatibleMain.java:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)
at org.codehaus.classworlds.Launcher.main(Launcher.java:375)
"VM Thread" prio=10 tid=0x000000004afcb000 nid=0x55e5 runnable
"VM Periodic Task Thread" prio=10 tid=0x000000004b000000 nid=0x55ec waiting on condition
JNI global references: 861
I get so tired of watching people abuse nohup. Here's how real daemonization routines work:
Create a child process.
Launch daemon as a child of that child process.
Exit the first process.
A simple way to emulate this using sh is as follows.
sh -c "mvn jetty:run &"
Observe...
scott#optimusprime:~/code/leapfrog/lasso> sh -c "mvn jetty:run &"
scott#optimusprime:~/code/leapfrog/lasso> ps aux | grep jetty:run
scott 26165 45.5 5.8 813556 235764 pts/9 Sl 15:23 0:06 /usr/lib/jvm/java-6-sun//bin/java -Xmx512M -classpath /home/scott/opt/apache-maven-2.2.1//boot/classworlds-1.1.jar -Dclassworlds.conf=/home/scott/opt/apache-maven-2.2.1//bin/m2.conf -Dmaven.home=/home/scott/opt/apache-maven-2.2.1/ org.codehaus.classworlds.Launcher "jetty:run"
scott#optimusprime:~/code/leapfrog/lasso> exit
If I check my process list from a separate terminal, or later, the program is still running.
However I have to ask, why do this instead of just install your web application in a local application server, and start it? Hopefully you aren't managing your real environments using apache maven.
I would run it in the foreground, and trigger a thread dump (Ctrl+Break), and see what it's doing by inspecting each thread stack trace. Without any further information I can't offer any more advice.
I suggest you try using screen. Might be overkill for such a simple task, but you can wrap it in a shell script.
Using maven is very convenient to download, install and start an application. Among many other benefits, maven offers developers the ability to manage the version of Jetty. Using the advice provided here I think maven fits the bill as a production initiation mechanism.