We Keep Coding

“java.lang.OutOfMemoryError : unable to create new native Thread”

I am getting this error while I am trying to post some messages into a queue using JMSTemplate.
I am trying to push more than 10000 messages in queue using for loop and I get this error.
I have limit of 4096 threads on my system.
In the threadDumps I can see below thread mostly.
"SimpleAsyncTaskExecutor-36991" prio=10 tid=0x00007f857dc13000 nid=0x5a9 waiting on condition [0x00007f83f21e9000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000748860208> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
at org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1394)
at org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1925)
at org.apache.activemq.ActiveMQMessageProducer.<init>(ActiveMQMessageProducer.java:125)
at org.apache.activemq.ActiveMQSession.createProducer(ActiveMQSession.java:969)
at org.apache.activemq.jms.pool.PooledSession.getMessageProducer(PooledSession.java:395)
at org.apache.activemq.jms.pool.PooledSession.createProducer(PooledSession.java:359)
at org.springframework.jms.core.JmsTemplate.doCreateProducer(JmsTemplate.java:1044)
at org.springframework.jms.core.JmsTemplate.createProducer(JmsTemplate.java:1025)
at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:598)
at org.springframework.jms.core.JmsTemplate$3.doInJms(JmsTemplate.java:569)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:491)
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:566)
at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:655)
at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:646)
at com.jato.frameworks.commons.util.JATOJMSProducer.sendMessageToQueue(JATOJMSProducer.java:59)
What I suspect is the threads which are created to post a message in the queue are getting created more rapidly than the rate they are being killed.
Is there any way to solve this ?

Apache Tomcat Threads in WAITING state while thread pool increases Class name is ImageLoadWorker

I got this type thread dump on tomcat, All thread are in wating state.
So application is slow down.
Please suggest me the solution for that.
I am using Tomcat 7 and java 7
"ImageLoadWorker(653)" prio=5 tid=0x2089 nid=0x829 in Object.wait() - stats: cpu=0 blk=-1 wait=-1
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on org.xhtmlrenderer.swing.ImageLoadQueue#4651e7d2
at java.lang.Object.wait(Object.java:503)
at org.xhtmlrenderer.swing.ImageLoadQueue.getTask(ImageLoadQueue.java:83)
at org.xhtmlrenderer.swing.ImageLoadWorker.run(ImageLoadWorker.java:53)
Locked synchronizers: count = 0

There is thread leakage in this class of flyingsaucer jar. I got answer for this problem on below URL.
https://technotailor.wordpress.com/2017/04/17/thread-leak-with-imageloadworker-in-flying-saucer-jar/

Jetty multiple acceptors blocked waiting to lock and object locked by other acceptor

I'm facing an issue where the application hangs and possibly is due to blocked Jetty threads, but i'm not 100% sure in how to interpret the thread dump, if is normal what is observed and how does the jetty acceptors lifecycle is.
This is a particular part of the thread dump i'm observing:
"qtp1291915867-82 Acceptor0 SelectChannelConnector#0.0.0.0:3581 STARTING" prio=5 tid=0x52 state=BLOCKED
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:281)
- waiting to lock java.lang.Object#71918ba2 [owned by "qtp1291915867-84 Acceptor2 SelectChannelConnector#0.0.0.0:3581 STARTING" tid=0x54]
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:97)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:833)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:598)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:533)
at java.lang.Thread.run(Thread.java:804)
"qtp1291915867-83 Acceptor1 SelectChannelConnector#0.0.0.0:3581 STARTING" prio=5 tid=0x53 state=BLOCKED
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:281)
- waiting to lock java.lang.Object#71918ba2 [owned by "qtp1291915867-84 Acceptor2 SelectChannelConnector#0.0.0.0:3581 STARTING" tid=0x54]
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:97)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:833)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:598)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:533)
at java.lang.Thread.run(Thread.java:804)
"qtp1291915867-84 Acceptor2 SelectChannelConnector#0.0.0.0:3581 STARTING" prio=5 tid=0x54 state=RUNNABLE (running in native)
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:305)
- locked java.lang.Object#71918ba2
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:97)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:833)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:598)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:533)
at java.lang.Thread.run(Thread.java:804)
"qtp1291915867-85 Acceptor3 SelectChannelConnector#0.0.0.0:3581 STARTING" prio=5 tid=0x55 state=BLOCKED
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:281)
- waiting to lock java.lang.Object#71918ba2 [owned by "qtp1291915867-84 Acceptor2 SelectChannelConnector#0.0.0.0:3581 STARTING" tid=0x54]
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:97)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:833)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:598)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:533)
at java.lang.Thread.run(Thread.java:804)
AFAIK, the amount of acceptors should be between 1 and # of CPU's, so i'm assuming this is running in a 4 Core CPU, the question is, from jetty perspective, is it normal that other acceptors be blocked waiting to other acceptors resource? if so, what this resource could be?
I'm not very fond on Jetty so i don't really know if this is normal and they are just waiting for they turn to attend incoming connections, if so, then the issue is in different place but i would like to discard the jetty part.
Btw, the jetty version is 7.5.4
Any ideas?
Thanks

Blocking threads using javassist in web app

Could somebody help me and explain what exactly is going on, looking at the following thread dump. It is a web application running on Tomcat 7 and we experience that some requests do not get answered:
"ajp-bio-8012-exec-161" daemon prio=10 tid=0x00007fe170603000 nid=0x344f runnable [0x00007fe174fae000]
java.lang.Thread.State: RUNNABLE
at java.security.AccessController.doPrivileged(Native Method)
at java.io.FilePermission.init(FilePermission.java:209)
at java.io.FilePermission.<init>(FilePermission.java:285)
at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
at java.io.File.exists(File.java:808)
at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1080)
at sun.misc.URLClassPath$FileLoader.findResource(URLClassPath.java:1047)
at sun.misc.URLClassPath.findResource(URLClassPath.java:176)
at java.net.URLClassLoader$2.run(URLClassLoader.java:551)
at java.net.URLClassLoader$2.run(URLClassLoader.java:549)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findResource(URLClassLoader.java:548)
at java.lang.ClassLoader.getResource(ClassLoader.java:1147)
at java.lang.ClassLoader.getResource(ClassLoader.java:1142)
at org.apache.catalina.loader.WebappClassLoader.getResource(WebappClassLoader.java:1445)
at java.lang.Class.getResource(Class.java:2142)
at javassist.ClassClassPath.find(ClassClassPath.java:84)
at javassist.ClassPoolTail.find(ClassPoolTail.java:317)
at javassist.ClassPool.find(ClassPool.java:495)
at javassist.ClassPool.createCtClass(ClassPool.java:479)
at javassist.ClassPool.get0(ClassPool.java:445)
- locked <0x00000000db6cdb48> (a javassist.ClassPool)
at javassist.ClassPool.get(ClassPool.java:414)
at javassist.compiler.MemberResolver.lookupClass0(MemberResolver.java:425)
at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:389)
at javassist.compiler.MemberResolver.lookupClassByJvmName(MemberResolver.java:310)
at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:327)
at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:314)
at javassist.compiler.Javac.compileField(Javac.java:122)
at javassist.compiler.Javac.compile(Javac.java:91)
at javassist.CtField.make(CtField.java:163)
at com.mycompany.validation.util.BeanGenerator.addProperty(BeanGenerator.java:157)
...
at java.lang.Thread.run(Thread.java:745)
"ajp-bio-8012-exec-12" daemon prio=10 tid=0x0000000000c49800 nid=0x3d99 waiting for monitor entry [0x00007fe1756b7000]
java.lang.Thread.State: BLOCKED (on object monitor)
at javassist.ClassPool.get0(ClassPool.java:432)
- waiting to lock <0x00000000db6cdb48> (a javassist.ClassPool)
at javassist.ClassPool.get(ClassPool.java:414)
at javassist.compiler.MemberResolver.lookupClass0(MemberResolver.java:425)
at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:389)
at javassist.compiler.MemberResolver.lookupClassByJvmName(MemberResolver.java:310)
at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:327)
at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:314)
at javassist.compiler.Javac.compileField(Javac.java:122)
at javassist.compiler.Javac.compile(Javac.java:91)
at javassist.CtField.make(CtField.java:163)
at com.mycompany.validation.util.BeanGenerator.addProperty(BeanGenerator.java:157)
...
at java.lang.Thread.run(Thread.java:745)
"ajp-bio-8012-exec-5" daemon prio=10 tid=0x0000000001603800 nid=0x7c77 waiting for monitor entry [0x00007fe174aaa000]
java.lang.Thread.State: BLOCKED (on object monitor)
at javassist.ClassPool.makeClass(ClassPool.java:621)
- waiting to lock <0x00000000db6cdb48> (a javassist.ClassPool)
at javassist.ClassPool.makeClass(ClassPool.java:606)
at com.mycompany.validation.util.BeanGenerator.init(BeanGenerator.java:334)
...
at java.lang.Thread.run(Thread.java:745)
I am not a specialist in thread dumps and just would like to know how to get rid of the problem.
Has the first thread locked an object (ClassPool) that two other request threads are waiting for to unlock? Why is it locked? Do I have to synchronize and how?
Thanks for any hint!

You have a single ClassPool in your application which has the monitor id 0x00000000db6cdb48.
If you see the method signature of ClassPool.get0:
protected synchronized CtClass get0(String classname, boolean useCache)
So it is synchronized. I suppose you simply used the ClassPool.getDefault(). You have to know that this method returns a single instance, so all of your calls will be sychronized.
Create a new pool for each thread (with new ClassPool(ClassPool.getDefault()) for example) and it will be fine then.
Apart from that you might check your secrurity manager, why it takes so long.

Strange thread dump on deadlock

We've been experiencing a strange deadlock during the startup of our java application. When I run jstack on the application to investigate, I see that the AWT-EventQueue is in Object.wait(), but the thread is still marked as RUNNABLE. I've included the relevent parts of the thread dump, and I'm hoping that someone can shed some light on this issue.
"AWT-EventQueue-0" prio=6 tid=0x5f0a2400 nid=0x19e4 in Object.wait() [0x6007e000]
java.lang.Thread.State: RUNNABLE
at com.ge.med.platinum.work.isu.ExamTransaction.getEAOTableLite(ExamTransaction.java:1514)
...
- locked <0x1fc87448> (a java.awt.Component$AWTTreeLock)
...
"Thread-63-Pool-9" prio=6 tid=0x5f1a2800 nid=0x1f54 waiting for monitor entry [0x61a9f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.awt.Component.setFont(Component.java:1777)
- waiting to lock <0x1fc87448> (a java.awt.Component$AWTTreeLock)
...
"Thread-289-Pool-3" prio=6 tid=0x60afe800 nid=0x12b8 waiting for monitor entry [0x623fe000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.awt.Component.setFont(Component.java:1777)
- waiting to lock <0x1fc87448> (a java.awt.Component$AWTTreeLock)
...
In addition, I've noticed this thread, which mentions that accessing a static variable may be involved. This is also the case in our application. The line in getEAOTableLite in question references a static method.

I'm not sure how I missed it, but if I had read down the stack trace a little better I would have found that the issue was that the static initialization of the EAOAlertManager class would eventaully make a call to the Component.setFont() method, which was be blocked by the AWT-EventQueue (it is illegal to call setFont() outside of the EventQueue). The EventQueue then ended up back in ExamTransaction.getEAOTableLite, which meant that it would reference the EAOAlertManager class again, causing it to wait for the class to finish loading. But the EAOAlertManager class was waiting on the EventQueue. That, my friends, is a deadlock.
"Thread-289-Pool-3" prio=6 tid=0x60afe800 nid=0x12b8 waiting for monitor entry [0x623fe000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.awt.Component.setFont(Component.java:1777)
- waiting to lock <0x1fc87448> (a java.awt.Component$AWTTreeLock)
at java.awt.Container.setFont(Container.java:1554)
at javax.swing.JComponent.setFont(JComponent.java:2723)
at javax.swing.LookAndFeel.installColorsAndFont(LookAndFeel.java:191)
at javax.swing.plaf.basic.BasicPanelUI.installDefaults(BasicPanelUI.java:49)
at javax.swing.plaf.basic.BasicPanelUI.installUI(BasicPanelUI.java:39)
at com.ge.med.ptk.laf.CuiPanelUI.installUI(CuiPanelUI.java:53)
at javax.swing.JComponent.setUI(JComponent.java:662)
at javax.swing.JPanel.setUI(JPanel.java:136)
at javax.swing.JPanel.updateUI(JPanel.java:109)
at javax.swing.JPanel.<init>(JPanel.java:69)
at javax.swing.JPanel.<init>(JPanel.java:92)
at javax.swing.JPanel.<init>(JPanel.java:100)
at javax.swing.JRootPane.createGlassPane(JRootPane.java:528)
at javax.swing.JRootPane.<init>(JRootPane.java:348)
at javax.swing.JDialog.createRootPane(JDialog.java:611)
at javax.swing.JDialog.dialogInit(JDialog.java:593)
at com.ge.med.plaf.wrapper.WJDialog.dialogInit(WJDialog.java:42)
at javax.swing.JDialog.<init>(JDialog.java:545)
at javax.swing.JDialog.<init>(JDialog.java:515)
at com.ge.med.plaf.wrapper.WJDialog.<init>(WJDialog.java:424)
at com.ge.med.platinum.gui.util.PlatinumDialog.<init>(PlatinumDialog.java:138)
at com.ge.med.platinum.gui.util.EAOAlertManager$EAOAlertDialog.<init>(EAOAlertManager.java:450)
at com.ge.med.platinum.gui.util.EAOAlertManager.<clinit>(EAOAlertManager.java:77)
at com.ge.med.platinum.work.isu.ExamTransaction.getEAOTableLite(ExamTransaction.java:1514)

This article seems to get directly to the point:
http://www.javaworld.com/javaworld/jw-04-1999/jw-04-toolbox.html
There are 5 pages. I don't know what it all means to your application, but it should help you.