We Keep Coding

Threads getting blocked while opening a connection using apache commons

from my application, many threads are trying to get a connection from oracle but I am seeing only one thread is able to get connection at a time. till this thread gets connection, other threads are blocked which is causing slowness in application. I need help determining why other threads are getting blocked. am i running out of connections than the available connections in pool?
Thread holding the lock.
org.apache.commons.dbcp2.managed.LocalXAConnectionFactory.createConnection(LocalXAConnectionFactory.java:68)
at org.apache.commons.dbcp2.managed.PoolableManagedConnectionFactory.makeObject(PoolableManagedConnectionFactory.java:64)
- locked <0x0000000641e88660> (a org.apache.commons.dbcp2.managed.PoolableManagedConnectionFactory)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.commons.dbcp2.managed.ManagedConnection.updateTransactionStatus(ManagedConnection.java:125)
at org.apache.commons.dbcp2.managed.ManagedConnection.<init>(ManagedConnection.java:59)
at org.apache.commons.dbcp2.managed.ManagedDataSource.getConnection(ManagedDataSource.java:81)
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413)
at com.manh.jdbc.ScopeDataSource.getConnection(ScopeDataSource.java:79)
Thread waiting for lock on PoolableManagedConnectionFactory
"pool-38-thread-5389" #267084 prio=5 os_prio=0 tid=0x00002b6b66541800 nid=0x18dcf waiting for monitor entry [0x00002b6bfd047000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.commons.dbcp2.managed.PoolableManagedConnectionFactory.makeObject(PoolableManagedConnectionFactory.java:64)
- waiting to lock <0x0000000641e88660> (a org.apache.commons.dbcp2.managed.PoolableManagedConnectionFactory)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.commons.dbcp2.managed.ManagedConnection.updateTransactionStatus(ManagedConnection.java:125)
at org.apache.commons.dbcp2.managed.ManagedConnection.<init>(ManagedConnection.java:59)

Apache Tomcat Threads in WAITING state while thread pool increases Class name is ImageLoadWorker

I got this type thread dump on tomcat, All thread are in wating state.
So application is slow down.
Please suggest me the solution for that.
I am using Tomcat 7 and java 7
"ImageLoadWorker(653)" prio=5 tid=0x2089 nid=0x829 in Object.wait() - stats: cpu=0 blk=-1 wait=-1
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on org.xhtmlrenderer.swing.ImageLoadQueue#4651e7d2
at java.lang.Object.wait(Object.java:503)
at org.xhtmlrenderer.swing.ImageLoadQueue.getTask(ImageLoadQueue.java:83)
at org.xhtmlrenderer.swing.ImageLoadWorker.run(ImageLoadWorker.java:53)
Locked synchronizers: count = 0

There is thread leakage in this class of flyingsaucer jar. I got answer for this problem on below URL.
https://technotailor.wordpress.com/2017/04/17/thread-leak-with-imageloadworker-in-flying-saucer-jar/

How To Get Thread NID(Native ID) in java

"remoting-jmx client-thread-4" daemon prio=6 tid=0x000000000f86b000 nid=0x20e0 >waiting on condition [0x000000000b82e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ebd8bcc0> (a >java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at >java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQue>ue.java:460)
at >java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.j>ava:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at >java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Here in this thread dump, we have thread Nid. This Nid how can i get it from a thread in java code.

Here is a gist which shows how to obtain the NID if you can include lombok.jarfrom project lombok and sa-jdi.jar from the JDK in your app.

Neo4j : Concurrency and memory mapped buffers

I am launching a batch of traversal on an embedded Neo4J DB 2.0.3.
23M nodes
87M relations
16GB Heap
Tried different cache settings
Tried different thread settings ( From 20 to 5 threads )
The Job is running fine for some time (ex: 1h) and then the throughput slows down dramatically because the threads are spending most of their time waiting for locks.
It looks like the memory buffers (MappedPersistenceWindow) can't be shared by multiple threads, which sounds a bit weird.
Some samples from the thread dump :
"taskExecutor-6" - Thread t#17
java.lang.Thread.State: BLOCKED
at java.lang.Object.wait(Native Method)
- waiting on <3a67f9f7> (a org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow)
at java.lang.Object.wait(Object.java:503)
at org.neo4j.kernel.impl.nioneo.store.LockableWindow.lock(LockableWindow.java:96)
at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:197)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:430)
at org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getRecord(RelationshipStore.java:84)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction$3.load(NeoStoreTransaction.java:208)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction$3.load(NeoStoreTransaction.java:198)
at org.neo4j.kernel.impl.nioneo.xa.RecordChanges.getOrLoad(RecordChanges.java:63)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.relLoadLight(NeoStoreTransaction.java:1189)
at org.neo4j.kernel.impl.persistence.PersistenceManager.loadLightRelationship(PersistenceManager.java:109)
at org.neo4j.kernel.impl.core.NodeManager$2.loadById(NodeManager.java:114)
at org.neo4j.kernel.impl.core.NodeManager$2.loadById(NodeManager.java:110)
at org.neo4j.kernel.impl.cache.AutoLoadingCache.get(AutoLoadingCache.java:93)
at org.neo4j.kernel.impl.core.NodeManager.getRelationshipForProxy(NodeManager.java:544)
at org.neo4j.kernel.InternalAbstractGraphDatabase$6.lookupRelationship(InternalAbstractGraphDatabase.java:849)
at org.neo4j.kernel.impl.core.RelationshipProxy.getType(RelationshipProxy.java:141)
"taskExecutor-5" - Thread t#16
java.lang.Thread.State: RUNNABLE
at org.neo4j.kernel.impl.nioneo.store.LockableWindow.markAsInUse(LockableWindow.java:70)
- locked <6abe6e1> (a org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow)
at org.neo4j.kernel.impl.nioneo.store.BrickElement.getAndMarkWindow(BrickElement.java:94)
at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:147)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:430)
at org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getChainRecord(RelationshipStore.java:325)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.getMoreRelationships(NeoStoreTransaction.java:2331)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.getMoreRelationships(NeoStoreTransaction.java:1390)
at org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:94)
at org.neo4j.kernel.impl.core.RelationshipLoader.getMoreRelationships(RelationshipLoader.java:50)
at org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:779)
at org.neo4j.kernel.impl.core.NodeImpl.loadMoreRelationshipsFromNodeManager(NodeImpl.java:577)
at org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:540)
- locked <2b29ed5b> (a org.neo4j.kernel.impl.core.NodeImpl)
at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:98)
at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:36)
at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:55)
at org.neo4j.kernel.impl.core.NodeImpl.hasRelationship(NodeImpl.java:644)
at org.neo4j.kernel.impl.core.NodeProxy.hasRelationship(NodeProxy.java:183)
"taskExecutor-4" - Thread t#15
java.lang.Thread.State: BLOCKED
at java.lang.Object.wait(Native Method)
- waiting on <3a67f9f7> (a org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow)
at java.lang.Object.wait(Object.java:503)
at org.neo4j.kernel.impl.nioneo.store.LockableWindow.lock(LockableWindow.java:96)
at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:197)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:430)
at org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getChainRecord(RelationshipStore.java:325)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.getMoreRelationships(NeoStoreTransaction.java:2331)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.getMoreRelationships(NeoStoreTransaction.java:1390)
at org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:94)
at org.neo4j.kernel.impl.core.RelationshipLoader.getMoreRelationships(RelationshipLoader.java:50)
at org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:779)
at org.neo4j.kernel.impl.core.NodeImpl.loadMoreRelationshipsFromNodeManager(NodeImpl.java:577)
at org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:540)
- locked <41fe8c4f> (a org.neo4j.kernel.impl.core.NodeImpl)
at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:98)
at org.neo4j.kernel.impl.core.RelationshipIterator.fetchNextOrNull(RelationshipIterator.java:36)
at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:55)
at org.neo4j.kernel.impl.core.NodeImpl.hasRelationship(NodeImpl.java:644)
at org.neo4j.kernel.impl.core.NodeProxy.hasRelationship(NodeProxy.java:183)
"taskExecutor-3" - Thread t#14
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <3a67f9f7> (a org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow)
at java.lang.Object.wait(Object.java:503)
at org.neo4j.kernel.impl.nioneo.store.LockableWindow.lock(LockableWindow.java:96)
at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:197)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:430)
at org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getRecord(RelationshipStore.java:84)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction$3.load(NeoStoreTransaction.java:208)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction$3.load(NeoStoreTransaction.java:198)
at org.neo4j.kernel.impl.nioneo.xa.RecordChanges.getOrLoad(RecordChanges.java:63)
at org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.relLoadLight(NeoStoreTransaction.java:1189)
at org.neo4j.kernel.impl.persistence.PersistenceManager.loadLightRelationship(PersistenceManager.java:109)
at org.neo4j.kernel.impl.core.NodeManager$2.loadById(NodeManager.java:114)
at org.neo4j.kernel.impl.core.NodeManager$2.loadById(NodeManager.java:110)
at org.neo4j.kernel.impl.cache.AutoLoadingCache.get(AutoLoadingCache.java:93)
at org.neo4j.kernel.impl.core.NodeManager.getRelationshipForProxy(NodeManager.java:544)
at org.neo4j.kernel.InternalAbstractGraphDatabase$6.lookupRelationship(InternalAbstractGraphDatabase.java:849)
at org.neo4j.kernel.impl.core.RelationshipProxy.getOtherNode(RelationshipProxy.java:108)
at org.neo4j.kernel.impl.traversal.TraversalBranchImpl.next(TraversalBranchImpl.java:145)
at org.neo4j.graphdb.traversal.PreorderDepthFirstSelector.next(PreorderDepthFirstSelector.java:49)
at org.neo4j.kernel.impl.traversal.MonoDirectionalTraverserIterator.fetchNextOrNull(MonoDirectionalTraverserIterator.java:68)
at org.neo4j.kernel.impl.traversal.MonoDirectionalTraverserIterator.fetchNextOrNull(MonoDirectionalTraverserIterator.java:35)
at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:55)
Any idea ?

Strange thread dump on deadlock

We've been experiencing a strange deadlock during the startup of our java application. When I run jstack on the application to investigate, I see that the AWT-EventQueue is in Object.wait(), but the thread is still marked as RUNNABLE. I've included the relevent parts of the thread dump, and I'm hoping that someone can shed some light on this issue.
"AWT-EventQueue-0" prio=6 tid=0x5f0a2400 nid=0x19e4 in Object.wait() [0x6007e000]
java.lang.Thread.State: RUNNABLE
at com.ge.med.platinum.work.isu.ExamTransaction.getEAOTableLite(ExamTransaction.java:1514)
...
- locked <0x1fc87448> (a java.awt.Component$AWTTreeLock)
...
"Thread-63-Pool-9" prio=6 tid=0x5f1a2800 nid=0x1f54 waiting for monitor entry [0x61a9f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.awt.Component.setFont(Component.java:1777)
- waiting to lock <0x1fc87448> (a java.awt.Component$AWTTreeLock)
...
"Thread-289-Pool-3" prio=6 tid=0x60afe800 nid=0x12b8 waiting for monitor entry [0x623fe000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.awt.Component.setFont(Component.java:1777)
- waiting to lock <0x1fc87448> (a java.awt.Component$AWTTreeLock)
...
In addition, I've noticed this thread, which mentions that accessing a static variable may be involved. This is also the case in our application. The line in getEAOTableLite in question references a static method.

I'm not sure how I missed it, but if I had read down the stack trace a little better I would have found that the issue was that the static initialization of the EAOAlertManager class would eventaully make a call to the Component.setFont() method, which was be blocked by the AWT-EventQueue (it is illegal to call setFont() outside of the EventQueue). The EventQueue then ended up back in ExamTransaction.getEAOTableLite, which meant that it would reference the EAOAlertManager class again, causing it to wait for the class to finish loading. But the EAOAlertManager class was waiting on the EventQueue. That, my friends, is a deadlock.
"Thread-289-Pool-3" prio=6 tid=0x60afe800 nid=0x12b8 waiting for monitor entry [0x623fe000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.awt.Component.setFont(Component.java:1777)
- waiting to lock <0x1fc87448> (a java.awt.Component$AWTTreeLock)
at java.awt.Container.setFont(Container.java:1554)
at javax.swing.JComponent.setFont(JComponent.java:2723)
at javax.swing.LookAndFeel.installColorsAndFont(LookAndFeel.java:191)
at javax.swing.plaf.basic.BasicPanelUI.installDefaults(BasicPanelUI.java:49)
at javax.swing.plaf.basic.BasicPanelUI.installUI(BasicPanelUI.java:39)
at com.ge.med.ptk.laf.CuiPanelUI.installUI(CuiPanelUI.java:53)
at javax.swing.JComponent.setUI(JComponent.java:662)
at javax.swing.JPanel.setUI(JPanel.java:136)
at javax.swing.JPanel.updateUI(JPanel.java:109)
at javax.swing.JPanel.<init>(JPanel.java:69)
at javax.swing.JPanel.<init>(JPanel.java:92)
at javax.swing.JPanel.<init>(JPanel.java:100)
at javax.swing.JRootPane.createGlassPane(JRootPane.java:528)
at javax.swing.JRootPane.<init>(JRootPane.java:348)
at javax.swing.JDialog.createRootPane(JDialog.java:611)
at javax.swing.JDialog.dialogInit(JDialog.java:593)
at com.ge.med.plaf.wrapper.WJDialog.dialogInit(WJDialog.java:42)
at javax.swing.JDialog.<init>(JDialog.java:545)
at javax.swing.JDialog.<init>(JDialog.java:515)
at com.ge.med.plaf.wrapper.WJDialog.<init>(WJDialog.java:424)
at com.ge.med.platinum.gui.util.PlatinumDialog.<init>(PlatinumDialog.java:138)
at com.ge.med.platinum.gui.util.EAOAlertManager$EAOAlertDialog.<init>(EAOAlertManager.java:450)
at com.ge.med.platinum.gui.util.EAOAlertManager.<clinit>(EAOAlertManager.java:77)
at com.ge.med.platinum.work.isu.ExamTransaction.getEAOTableLite(ExamTransaction.java:1514)

This article seems to get directly to the point:
http://www.javaworld.com/javaworld/jw-04-1999/jw-04-toolbox.html
There are 5 pages. I don't know what it all means to your application, but it should help you.