I am getting this error while I am trying to post some messages into a queue using JMSTemplate.
I am trying to push more than 10000 messages in queue using for loop and I get this error.
I have limit of 4096 threads on my system.
In the threadDumps I can see below thread mostly.
"SimpleAsyncTaskExecutor-36991" prio=10 tid=0x00007f857dc13000 nid=0x5a9 waiting on condition [0x00007f83f21e9000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000748860208> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
at org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1394)
at org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1925)
at org.apache.activemq.ActiveMQMessageProducer.<init>(ActiveMQMessageProducer.java:125)
at org.apache.activemq.ActiveMQSession.createProducer(ActiveMQSession.java:969)
at org.apache.activemq.jms.pool.PooledSession.getMessageProducer(PooledSession.java:395)
at org.apache.activemq.jms.pool.PooledSession.createProducer(PooledSession.java:359)
at org.springframework.jms.core.JmsTemplate.doCreateProducer(JmsTemplate.java:1044)
at org.springframework.jms.core.JmsTemplate.createProducer(JmsTemplate.java:1025)
at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:598)
at org.springframework.jms.core.JmsTemplate$3.doInJms(JmsTemplate.java:569)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:491)
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:566)
at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:655)
at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:646)
at com.jato.frameworks.commons.util.JATOJMSProducer.sendMessageToQueue(JATOJMSProducer.java:59)
What I suspect is the threads which are created to post a message in the queue are getting created more rapidly than the rate they are being killed.
Is there any way to solve this ?
Related
My flink job, taskManager vm has too many threadPool, every pool contains one thread, thread names:"pool-{poolNumber}-thread-1".
thread dump:
// about 2200 threads like this
"pool-236-thread-1" #153 prio=5 os_prio=0 tid=0x00007f9e1c12d000 nid=0x245 waiting on condition [0x00007f9e40c66000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007a8082938> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
my debug:
I debug the job in my IDE, start the job.
there is one waiting thread names like 'pool-{poolNumber}-thread-1', which is "pool-5-thread-1".
I put a break on Thread.class's constructure.
the thread to create "pool-5-thread-1"'s stack is:
questions:
I don't know what to do next😢
from my application, many threads are trying to get a connection from oracle but I am seeing only one thread is able to get connection at a time. till this thread gets connection, other threads are blocked which is causing slowness in application. I need help determining why other threads are getting blocked. am i running out of connections than the available connections in pool?
Thread holding the lock.
org.apache.commons.dbcp2.managed.LocalXAConnectionFactory.createConnection(LocalXAConnectionFactory.java:68)
at org.apache.commons.dbcp2.managed.PoolableManagedConnectionFactory.makeObject(PoolableManagedConnectionFactory.java:64)
- locked <0x0000000641e88660> (a org.apache.commons.dbcp2.managed.PoolableManagedConnectionFactory)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.commons.dbcp2.managed.ManagedConnection.updateTransactionStatus(ManagedConnection.java:125)
at org.apache.commons.dbcp2.managed.ManagedConnection.<init>(ManagedConnection.java:59)
at org.apache.commons.dbcp2.managed.ManagedDataSource.getConnection(ManagedDataSource.java:81)
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413)
at com.manh.jdbc.ScopeDataSource.getConnection(ScopeDataSource.java:79)
Thread waiting for lock on PoolableManagedConnectionFactory
"pool-38-thread-5389" #267084 prio=5 os_prio=0 tid=0x00002b6b66541800 nid=0x18dcf waiting for monitor entry [0x00002b6bfd047000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.commons.dbcp2.managed.PoolableManagedConnectionFactory.makeObject(PoolableManagedConnectionFactory.java:64)
- waiting to lock <0x0000000641e88660> (a org.apache.commons.dbcp2.managed.PoolableManagedConnectionFactory)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.commons.dbcp2.managed.ManagedConnection.updateTransactionStatus(ManagedConnection.java:125)
at org.apache.commons.dbcp2.managed.ManagedConnection.<init>(ManagedConnection.java:59)
I have an application which which is deployed on jetty server.
Jetty standalone server has default configuration.Application contains spring based websocket.
Clients connect to the server using web-socket connection.
Once connection is established client sends heartbeat messages for every 5 seconds and sends few requests like requesting specific Data.
When client opens connection without sending any heart beat messages or specific requests am able to open any number of connections(tried upto 10000)
On the other hand,when all the threads are opened and started sending heart beat messages and specific requests parallely, server is hanged and is no longer able to accept connections greater than 2000.
Jetty Server version: jetty-distribution-9.3.8.v20160314
But after 2000 connections only application is getting hanged
Using jstack took the thread dump and the following is the result:
"qtp1143839598-216" #216 prio=5 os_prio=0 tid=0x00007fcd88017000 nid=0x7365 waiting on condition [0x00007fcd344c3000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000850d7fb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
"qtp1143839598-215" #215 prio=5 os_prio=0 tid=0x00007fcd8c01c000 nid=0x7364 waiting on condition [0x00007fcd345c4000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000850d7fb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
"qtp1143839598-214" #214 prio=5 os_prio=0 tid=0x00007fcd805ea800 nid=0x735f waiting on condition [0x00007fcd346c5000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000850d7fb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
"qtp1143839598-213" #213 prio=5 os_prio=0 tid=0x00007fcd84135000 nid=0x7358 waiting on condition [0x00007fcd347c6000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000850d7fb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
"qtp1143839598-212" #212 prio=5 os_prio=0 tid=0x00007fcd78032000 nid=0x7354 waiting on condition [0x00007fcd348c7000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000850d7fb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
There are around 196 threads with similar to the above logs
We are expecting
500K live connections at any point of time 2. 2000 new connections per minute , 4000 re-connects (error and retry on client side)
Is there any way to configure jetty to handle the above requirement..and the above logs in jstack shows deadlock? or problem in application threads? Or is there any configuration to be made in spring websockets to achieve the above requirement. Sorry if language/framing of the question is not clear..
Could somebody help me and explain what exactly is going on, looking at the following thread dump. It is a web application running on Tomcat 7 and we experience that some requests do not get answered:
"ajp-bio-8012-exec-161" daemon prio=10 tid=0x00007fe170603000 nid=0x344f runnable [0x00007fe174fae000]
java.lang.Thread.State: RUNNABLE
at java.security.AccessController.doPrivileged(Native Method)
at java.io.FilePermission.init(FilePermission.java:209)
at java.io.FilePermission.<init>(FilePermission.java:285)
at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
at java.io.File.exists(File.java:808)
at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1080)
at sun.misc.URLClassPath$FileLoader.findResource(URLClassPath.java:1047)
at sun.misc.URLClassPath.findResource(URLClassPath.java:176)
at java.net.URLClassLoader$2.run(URLClassLoader.java:551)
at java.net.URLClassLoader$2.run(URLClassLoader.java:549)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findResource(URLClassLoader.java:548)
at java.lang.ClassLoader.getResource(ClassLoader.java:1147)
at java.lang.ClassLoader.getResource(ClassLoader.java:1142)
at org.apache.catalina.loader.WebappClassLoader.getResource(WebappClassLoader.java:1445)
at java.lang.Class.getResource(Class.java:2142)
at javassist.ClassClassPath.find(ClassClassPath.java:84)
at javassist.ClassPoolTail.find(ClassPoolTail.java:317)
at javassist.ClassPool.find(ClassPool.java:495)
at javassist.ClassPool.createCtClass(ClassPool.java:479)
at javassist.ClassPool.get0(ClassPool.java:445)
- locked <0x00000000db6cdb48> (a javassist.ClassPool)
at javassist.ClassPool.get(ClassPool.java:414)
at javassist.compiler.MemberResolver.lookupClass0(MemberResolver.java:425)
at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:389)
at javassist.compiler.MemberResolver.lookupClassByJvmName(MemberResolver.java:310)
at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:327)
at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:314)
at javassist.compiler.Javac.compileField(Javac.java:122)
at javassist.compiler.Javac.compile(Javac.java:91)
at javassist.CtField.make(CtField.java:163)
at com.mycompany.validation.util.BeanGenerator.addProperty(BeanGenerator.java:157)
...
at java.lang.Thread.run(Thread.java:745)
"ajp-bio-8012-exec-12" daemon prio=10 tid=0x0000000000c49800 nid=0x3d99 waiting for monitor entry [0x00007fe1756b7000]
java.lang.Thread.State: BLOCKED (on object monitor)
at javassist.ClassPool.get0(ClassPool.java:432)
- waiting to lock <0x00000000db6cdb48> (a javassist.ClassPool)
at javassist.ClassPool.get(ClassPool.java:414)
at javassist.compiler.MemberResolver.lookupClass0(MemberResolver.java:425)
at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:389)
at javassist.compiler.MemberResolver.lookupClassByJvmName(MemberResolver.java:310)
at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:327)
at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:314)
at javassist.compiler.Javac.compileField(Javac.java:122)
at javassist.compiler.Javac.compile(Javac.java:91)
at javassist.CtField.make(CtField.java:163)
at com.mycompany.validation.util.BeanGenerator.addProperty(BeanGenerator.java:157)
...
at java.lang.Thread.run(Thread.java:745)
"ajp-bio-8012-exec-5" daemon prio=10 tid=0x0000000001603800 nid=0x7c77 waiting for monitor entry [0x00007fe174aaa000]
java.lang.Thread.State: BLOCKED (on object monitor)
at javassist.ClassPool.makeClass(ClassPool.java:621)
- waiting to lock <0x00000000db6cdb48> (a javassist.ClassPool)
at javassist.ClassPool.makeClass(ClassPool.java:606)
at com.mycompany.validation.util.BeanGenerator.init(BeanGenerator.java:334)
...
at java.lang.Thread.run(Thread.java:745)
I am not a specialist in thread dumps and just would like to know how to get rid of the problem.
Has the first thread locked an object (ClassPool) that two other request threads are waiting for to unlock? Why is it locked? Do I have to synchronize and how?
Thanks for any hint!
You have a single ClassPool in your application which has the monitor id 0x00000000db6cdb48.
If you see the method signature of ClassPool.get0:
protected synchronized CtClass get0(String classname, boolean useCache)
So it is synchronized. I suppose you simply used the ClassPool.getDefault(). You have to know that this method returns a single instance, so all of your calls will be sychronized.
Create a new pool for each thread (with new ClassPool(ClassPool.getDefault()) for example) and it will be fine then.
Apart from that you might check your secrurity manager, why it takes so long.
"remoting-jmx client-thread-4" daemon prio=6 tid=0x000000000f86b000 nid=0x20e0 >waiting on condition [0x000000000b82e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ebd8bcc0> (a >java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at >java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQue>ue.java:460)
at >java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.j>ava:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at >java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Here in this thread dump, we have thread Nid. This Nid how can i get it from a thread in java code.
Here is a gist which shows how to obtain the NID if you can include lombok.jarfrom project lombok and sa-jdi.jar from the JDK in your app.