I have Spring with Quartz jobs (clustered) running at periodic interval (1 minute). When server starts everything seems fine, but jobs don't get triggered after some time. Restart of the server makes the jobs run, but issue re-occurs after some time.
I suspected it to be a thread exhaustion issue and from thread dump I noticed that all my Quartz threads (10) are in TIMED_WAITING.
Config:
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 10
org.quartz.threadPool.threadPriority = 5
Thread dump:
quartzScheduler_Worker-10 - priority:10 - threadId:0x00007f8ae534d800 - nativeId:0x13c78 - state:TIMED_WAITING stackTrace:
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000066cd73220> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
- locked <0x000000066cd73220> (a java.lang.Object)
Using quartz 2.2.1 (i doubt if it could be version specific issue)
I verified from the logs that there are no DB connectivity issues.
Kindly help in diagnosing the problem. Is there a possibility that I have maxed out system resources (number of threads) ? But my jobs are synchronous and exist only when all its child threads have completed their task and I also have this annotation #DisallowConcurrentExecution
The root cause was we had too many miss fires in our quartz job. We have quartz kicks in every 1 minute and job doesn't really complete in say 1 min, so it's getting pilled up as miss fires and quartz tries to execute them first.
During this process there's an operation of update of miss fires which takes a lots of time which leads quartz to get stuck. This is evident from thread dump where in all our quartz threads are in TIMED_WAITING state as below
quartzScheduler_Worker-10 - priority:10 - threadId:0x00007f8ae534d800 - nativeId:0x13c78 - state:TIMED_WAITING
stackTrace:
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000066cd73220> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
- locked <0x000000066cd73220> (a java.lang.Object)
Refer : https://jira.terracotta.org/jira/si/jira.issueviews:issue-html/QTZ-357/QTZ-357.html
For our use case miss fires can be ignored and can be picked with next run. Hence I changed the Misfire instruction to ignore as below
<property name="misfireInstructionName" value="MISFIRE_INSTRUCTION_IGNORE_MISFIRE_POLICY" />
Related
from my application, many threads are trying to get a connection from oracle but I am seeing only one thread is able to get connection at a time. till this thread gets connection, other threads are blocked which is causing slowness in application. I need help determining why other threads are getting blocked. am i running out of connections than the available connections in pool?
Thread holding the lock.
org.apache.commons.dbcp2.managed.LocalXAConnectionFactory.createConnection(LocalXAConnectionFactory.java:68)
at org.apache.commons.dbcp2.managed.PoolableManagedConnectionFactory.makeObject(PoolableManagedConnectionFactory.java:64)
- locked <0x0000000641e88660> (a org.apache.commons.dbcp2.managed.PoolableManagedConnectionFactory)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.commons.dbcp2.managed.ManagedConnection.updateTransactionStatus(ManagedConnection.java:125)
at org.apache.commons.dbcp2.managed.ManagedConnection.<init>(ManagedConnection.java:59)
at org.apache.commons.dbcp2.managed.ManagedDataSource.getConnection(ManagedDataSource.java:81)
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413)
at com.manh.jdbc.ScopeDataSource.getConnection(ScopeDataSource.java:79)
Thread waiting for lock on PoolableManagedConnectionFactory
"pool-38-thread-5389" #267084 prio=5 os_prio=0 tid=0x00002b6b66541800 nid=0x18dcf waiting for monitor entry [0x00002b6bfd047000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.commons.dbcp2.managed.PoolableManagedConnectionFactory.makeObject(PoolableManagedConnectionFactory.java:64)
- waiting to lock <0x0000000641e88660> (a org.apache.commons.dbcp2.managed.PoolableManagedConnectionFactory)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.commons.dbcp2.managed.ManagedConnection.updateTransactionStatus(ManagedConnection.java:125)
at org.apache.commons.dbcp2.managed.ManagedConnection.<init>(ManagedConnection.java:59)
I'm trying to shutdown a Java application which has 1 #JmsListener running.
It doesn't matter how long I wait, but it will never stop.
It eternally hangs on the doShutdown of DefaultMessageListenerContainer.
The threadump is
"DefaultMessageListenerContainer-1#6759" prio=5 tid=0x51 nid=NA waiting
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Object.java:-1)
at java.lang.Thread.join(Thread.java:1252)
at java.lang.Thread.join(Thread.java:1326)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:107)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
at java.lang.Shutdown.runHooks(Shutdown.java:123)
at java.lang.Shutdown.sequence(Shutdown.java:167)
at java.lang.Shutdown.exit(Shutdown.java:212)
- locked <0x20a> (a java.lang.Class)
at java.lang.Runtime.exit(Runtime.java:109)
at java.lang.System.exit(System.java:971)
at my.package.MainApplication.stop(MainApplication.java:124)
I'm running Spring 4.3.19 and Spring Boot 1.5.16
I got this type thread dump on tomcat, All thread are in wating state.
So application is slow down.
Please suggest me the solution for that.
I am using Tomcat 7 and java 7
"ImageLoadWorker(653)" prio=5 tid=0x2089 nid=0x829 in Object.wait() - stats: cpu=0 blk=-1 wait=-1
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on org.xhtmlrenderer.swing.ImageLoadQueue#4651e7d2
at java.lang.Object.wait(Object.java:503)
at org.xhtmlrenderer.swing.ImageLoadQueue.getTask(ImageLoadQueue.java:83)
at org.xhtmlrenderer.swing.ImageLoadWorker.run(ImageLoadWorker.java:53)
Locked synchronizers: count = 0
There is thread leakage in this class of flyingsaucer jar. I got answer for this problem on below URL.
https://technotailor.wordpress.com/2017/04/17/thread-leak-with-imageloadworker-in-flying-saucer-jar/
I am launching a batch of traversal on an embedded Neo4J DB 2.0.3.
23M nodes
87M relations
16GB Heap
Tried different cache settings
Tried different thread settings ( From 20 to 5 threads )
The Job is running fine for some time (ex: 1h) and then the throughput slows down dramatically because the threads are spending most of their time waiting for locks.
It looks like the memory buffers (MappedPersistenceWindow) can't be shared by multiple threads, which sounds a bit weird.
Some samples from the thread dump :
"taskExecutor-6" - Thread t#17
java.lang.Thread.State: BLOCKED
at java.lang.Object.wait(Native Method)
- waiting on <3a67f9f7> (a org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow)
at java.lang.Object.wait(Object.java:503)
at org.neo4j.kernel.impl.nioneo.store.LockableWindow.lock(LockableWindow.java:96)
at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:197)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:430)
at org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getRecord(RelationshipStore.java:84)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction$3.load(NeoStoreTransaction.java:208)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction$3.load(NeoStoreTransaction.java:198)
at org.neo4j.kernel.impl.nioneo.xa.RecordChanges.getOrLoad(RecordChanges.java:63)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.relLoadLight(NeoStoreTransaction.java:1189)
at org.neo4j.kernel.impl.persistence.PersistenceManager.loadLightRelationship(PersistenceManager.java:109)
at org.neo4j.kernel.impl.core.NodeManager$2.loadById(NodeManager.java:114)
at org.neo4j.kernel.impl.core.NodeManager$2.loadById(NodeManager.java:110)
at org.neo4j.kernel.impl.cache.AutoLoadingCache.get(AutoLoadingCache.java:93)
at org.neo4j.kernel.impl.core.NodeManager.getRelationshipForProxy(NodeManager.java:544)
at org.neo4j.kernel.InternalAbstractGraphDatabase$6.lookupRelationship(InternalAbstractGraphDatabase.java:849)
at org.neo4j.kernel.impl.core.RelationshipProxy.getType(RelationshipProxy.java:141)
"taskExecutor-5" - Thread t#16
java.lang.Thread.State: RUNNABLE
at org.neo4j.kernel.impl.nioneo.store.LockableWindow.markAsInUse(LockableWindow.java:70)
- locked <6abe6e1> (a org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow)
at org.neo4j.kernel.impl.nioneo.store.BrickElement.getAndMarkWindow(BrickElement.java:94)
at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:147)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:430)
at org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getChainRecord(RelationshipStore.java:325)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.getMoreRelationships(NeoStoreTransaction.java:2331)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.getMoreRelationships(NeoStoreTransaction.java:1390)
at org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:94)
at org.neo4j.kernel.impl.core.RelationshipLoader.getMoreRelationships(RelationshipLoader.java:50)
at org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:779)
at org.neo4j.kernel.impl.core.NodeImpl.loadMoreRelationshipsFromNodeManager(NodeImpl.java:577)
at org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:540)
- locked <2b29ed5b> (a org.neo4j.kernel.impl.core.NodeImpl)
at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:98)
at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:36)
at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:55)
at org.neo4j.kernel.impl.core.NodeImpl.hasRelationship(NodeImpl.java:644)
at org.neo4j.kernel.impl.core.NodeProxy.hasRelationship(NodeProxy.java:183)
"taskExecutor-4" - Thread t#15
java.lang.Thread.State: BLOCKED
at java.lang.Object.wait(Native Method)
- waiting on <3a67f9f7> (a org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow)
at java.lang.Object.wait(Object.java:503)
at org.neo4j.kernel.impl.nioneo.store.LockableWindow.lock(LockableWindow.java:96)
at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:197)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:430)
at org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getChainRecord(RelationshipStore.java:325)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.getMoreRelationships(NeoStoreTransaction.java:2331)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.getMoreRelationships(NeoStoreTransaction.java:1390)
at org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:94)
at org.neo4j.kernel.impl.core.RelationshipLoader.getMoreRelationships(RelationshipLoader.java:50)
at org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:779)
at org.neo4j.kernel.impl.core.NodeImpl.loadMoreRelationshipsFromNodeManager(NodeImpl.java:577)
at org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:540)
- locked <41fe8c4f> (a org.neo4j.kernel.impl.core.NodeImpl)
at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:98)
at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:36)
at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:55)
at org.neo4j.kernel.impl.core.NodeImpl.hasRelationship(NodeImpl.java:644)
at org.neo4j.kernel.impl.core.NodeProxy.hasRelationship(NodeProxy.java:183)
"taskExecutor-3" - Thread t#14
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <3a67f9f7> (a org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow)
at java.lang.Object.wait(Object.java:503)
at org.neo4j.kernel.impl.nioneo.store.LockableWindow.lock(LockableWindow.java:96)
at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:197)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:430)
at org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getRecord(RelationshipStore.java:84)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction$3.load(NeoStoreTransaction.java:208)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction$3.load(NeoStoreTransaction.java:198)
at org.neo4j.kernel.impl.nioneo.xa.RecordChanges.getOrLoad(RecordChanges.java:63)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.relLoadLight(NeoStoreTransaction.java:1189)
at org.neo4j.kernel.impl.persistence.PersistenceManager.loadLightRelationship(PersistenceManager.java:109)
at org.neo4j.kernel.impl.core.NodeManager$2.loadById(NodeManager.java:114)
at org.neo4j.kernel.impl.core.NodeManager$2.loadById(NodeManager.java:110)
at org.neo4j.kernel.impl.cache.AutoLoadingCache.get(AutoLoadingCache.java:93)
at org.neo4j.kernel.impl.core.NodeManager.getRelationshipForProxy(NodeManager.java:544)
at org.neo4j.kernel.InternalAbstractGraphDatabase$6.lookupRelationship(InternalAbstractGraphDatabase.java:849)
at org.neo4j.kernel.impl.core.RelationshipProxy.getOtherNode(RelationshipProxy.java:108)
at org.neo4j.kernel.impl.traversal.TraversalBranchImpl.next(TraversalBranchImpl.java:145)
at org.neo4j.graphdb.traversal.PreorderDepthFirstSelector.next(PreorderDepthFirstSelector.java:49)
at org.neo4j.kernel.impl.traversal.MonoDirectionalTraverserIterator.fetchNextOrNull(MonoDirectionalTraverserIterator.java:68)
at org.neo4j.kernel.impl.traversal.MonoDirectionalTraverserIterator.fetchNextOrNull(MonoDirectionalTraverserIterator.java:35)
at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:55)
Any idea ?
We've been experiencing a strange deadlock during the startup of our java application. When I run jstack on the application to investigate, I see that the AWT-EventQueue is in Object.wait(), but the thread is still marked as RUNNABLE. I've included the relevent parts of the thread dump, and I'm hoping that someone can shed some light on this issue.
"AWT-EventQueue-0" prio=6 tid=0x5f0a2400 nid=0x19e4 in Object.wait() [0x6007e000]
java.lang.Thread.State: RUNNABLE
at com.ge.med.platinum.work.isu.ExamTransaction.getEAOTableLite(ExamTransaction.java:1514)
...
- locked <0x1fc87448> (a java.awt.Component$AWTTreeLock)
...
"Thread-63-Pool-9" prio=6 tid=0x5f1a2800 nid=0x1f54 waiting for monitor entry [0x61a9f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.awt.Component.setFont(Component.java:1777)
- waiting to lock <0x1fc87448> (a java.awt.Component$AWTTreeLock)
...
"Thread-289-Pool-3" prio=6 tid=0x60afe800 nid=0x12b8 waiting for monitor entry [0x623fe000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.awt.Component.setFont(Component.java:1777)
- waiting to lock <0x1fc87448> (a java.awt.Component$AWTTreeLock)
...
In addition, I've noticed this thread, which mentions that accessing a static variable may be involved. This is also the case in our application. The line in getEAOTableLite in question references a static method.
I'm not sure how I missed it, but if I had read down the stack trace a little better I would have found that the issue was that the static initialization of the EAOAlertManager class would eventaully make a call to the Component.setFont() method, which was be blocked by the AWT-EventQueue (it is illegal to call setFont() outside of the EventQueue). The EventQueue then ended up back in ExamTransaction.getEAOTableLite, which meant that it would reference the EAOAlertManager class again, causing it to wait for the class to finish loading. But the EAOAlertManager class was waiting on the EventQueue. That, my friends, is a deadlock.
"Thread-289-Pool-3" prio=6 tid=0x60afe800 nid=0x12b8 waiting for monitor entry [0x623fe000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.awt.Component.setFont(Component.java:1777)
- waiting to lock <0x1fc87448> (a java.awt.Component$AWTTreeLock)
at java.awt.Container.setFont(Container.java:1554)
at javax.swing.JComponent.setFont(JComponent.java:2723)
at javax.swing.LookAndFeel.installColorsAndFont(LookAndFeel.java:191)
at javax.swing.plaf.basic.BasicPanelUI.installDefaults(BasicPanelUI.java:49)
at javax.swing.plaf.basic.BasicPanelUI.installUI(BasicPanelUI.java:39)
at com.ge.med.ptk.laf.CuiPanelUI.installUI(CuiPanelUI.java:53)
at javax.swing.JComponent.setUI(JComponent.java:662)
at javax.swing.JPanel.setUI(JPanel.java:136)
at javax.swing.JPanel.updateUI(JPanel.java:109)
at javax.swing.JPanel.<init>(JPanel.java:69)
at javax.swing.JPanel.<init>(JPanel.java:92)
at javax.swing.JPanel.<init>(JPanel.java:100)
at javax.swing.JRootPane.createGlassPane(JRootPane.java:528)
at javax.swing.JRootPane.<init>(JRootPane.java:348)
at javax.swing.JDialog.createRootPane(JDialog.java:611)
at javax.swing.JDialog.dialogInit(JDialog.java:593)
at com.ge.med.plaf.wrapper.WJDialog.dialogInit(WJDialog.java:42)
at javax.swing.JDialog.<init>(JDialog.java:545)
at javax.swing.JDialog.<init>(JDialog.java:515)
at com.ge.med.plaf.wrapper.WJDialog.<init>(WJDialog.java:424)
at com.ge.med.platinum.gui.util.PlatinumDialog.<init>(PlatinumDialog.java:138)
at com.ge.med.platinum.gui.util.EAOAlertManager$EAOAlertDialog.<init>(EAOAlertManager.java:450)
at com.ge.med.platinum.gui.util.EAOAlertManager.<clinit>(EAOAlertManager.java:77)
at com.ge.med.platinum.work.isu.ExamTransaction.getEAOTableLite(ExamTransaction.java:1514)
This article seems to get directly to the point:
http://www.javaworld.com/javaworld/jw-04-1999/jw-04-toolbox.html
There are 5 pages. I don't know what it all means to your application, but it should help you.