We Keep Coding

Unable to create new native thread error - but very few threads are in use [duplicate]

This question already has answers here:
Java: Unable to create new native thread
(7 answers)
Closed 9 years ago.
We have an application that is widely deployed (several hundred workstations running it). At one site (and only one site - our product is widely deployed to many environments), we randomly get the following error:
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
Operating system is Windows 7 64 bit
We are running in a 32 bit JVM ( 1.7.0_45)
Using Windows Task Manager, I can see that the process has 39 native threads (not very many), so we don't have a thread leak in our app... There are no other processes consuming lots of threads (Explorer has 35, jvisualvm has 24, iexplore has 20, ... I don't have an exact count, but we are probably looking at maybe 300 threads for the user total).
I have attempted to attach JVisualVM, but it fails to connect to the process (probably b/c of thread exhaustion). But from the metrics I can obtain from JVisualVM, the number of Java threads is about 22 live and 11 daemon.
The heap is well behaved - heap is 500MB with 250MB actually used.
The process is launched with -Xmx512m
Our process is showing Memory usage (in Task Manager) of 597,744K.
The workstation has 8GB RAM, of which only 3.8-4.0GB are used (I know, a 32 bit process won't access all of that, but there's still plenty)
Used VMMap, and the stack is 49,920KB size with 2,284K committed.
The process shows 5358KB free, and the largest allocatable block in the free list is 1,024K in size.
I used Resource Monitor and it's showing the Commit (KB) to be 630428, working set (KB) is 676,996, Shareable (KB) is 79,252 and the Private (KB) is 597,744
I am at a complete loss as to what is going on here. I've read a ton of articles on this, and it sounds like on some Linux systems, there is a per-user thread limit that can cause problems (but this is not Linux, and the problems described in other articles usually talk about needing thousands of threads - definitely not our case here).
If our heap was really big, I could see that eating into space available for threads, but 500MB seems like a very reasonable and small heap (esp for a workstation with 8GB RAM).
So I've pretty much exhausted everything I know to do - does anyone have any additional pointers about what might be going on here?
EDIT 1:
I found this interesting article: Eclipse crashes with "Unable to create new native thread" - any ideas? (my settings and info inside)
They are suggesting that stack size could be the problem.
This article: where to find default XSS value for Sun/Oracle JVM? - gives a link to Oracle documentation saying that default stack size is 512KB. So if my app has about 40 threads, we are looking at 20 MB of stack. 500MB heap. This all seems to be well within normal bounds for a 32 bit Java process.
So that leaves me with two possibilities that I can think of:
Some transient condition is causing a huge number of threads to be created (but those threads are discarded before we have a chance to do diagnostics)
Memory segmentation is killing us for some reason. It is interesting that the largest allocatable block (per VMMap is 1MB) - that doesn't seem like very much... On another machine where things are working fine, the largest allocatable block is 470MB...
So, are there any pointers about how to check for memory segmentation?
EDIT 2:
Article linked to by #mikhael ( http://blog.egilh.com/2006/06/2811aspx.html ) gives some rough calculations for allowed # of threads on 32 bit JVM.
I'm going to assume:
OS process space limit: 2GB
Modern JVM requires 250MB (this is a big assumption - I just doubled what was in the linked article)
Stack size (default Oracle): 512KB
Heap: 512MB
PermGen: (can't remember exactly, but it was certainly less than 100MB, so let's just use that)
So I have a worst case scenario of: (2GB - .25GB - .5GB - .1GB)/.005GB = 230 threads
EDIT 3:
Info I should have included originally: The application runs fine for a good while (like 24 to 48 hours) before this problem happens. The application does continuous background processing, so has very little idle time. Not sure if that's important or not...
EDIT 4:
More info: Looking at VMMap from another failure, and I'm seeing native heap exhaustion.
The Heap size is 1.2GB, with only 59.8MB committed.
Either the Java runtime is the problem here, or maybe some issue with native resources not being released properly? Like maybe a memory mapped file that isn't getting released?
We do use memory mapped files, so I'll put my focus on those.
EDIT 4:
I think that I've tracked the problem down to an exception that happens as follows:
java.lang.OutOfMemoryError
at java.util.zip.Deflater.init(Native Method)
at java.util.zip.Deflater.<init>(Unknown Source)
at java.util.zip.Deflater.<init>(Unknown Source)
at java.util.zip.DeflaterOutputStream.<init>(Unknown Source)
at java.util.zip.DeflaterOutputStream.<init>(Unknown Source)
at ....
On some very small handful of streams (I have 4 examples now) we are deflating, the above happens. And when it happens, VMMap spikes the heap of the process (not the JVM heap, but the actual native heap) up to 2GB. Once that happens, everything falls apart. This is now very repeatable (running the same stream into the deflater results in the memory spiking)
So, are we maybe looking at a problem with the JRE's zip library? Seems crazy to think that would be it, but I'm really at a loss.
If I take the exact same stream and run it on a different system (even running the same JRE - 32 bit, Java 7u45), we don't get the problem. I have completely uninstalled the JRE and reinstalled it without any change in behavior.

Finally figured this out.
There were a couple of data streams that we processed (4 out of 10 million at this site) that wound up creating a ton of DeflaterOutputStream objects. A 3rd party library we were using was calling finish() on the stream instead of close(). The underlying Deflater finalizer was cleaning things up, so as long as the load wasn't too high, there were no problems. But past a tipping point, we started running into this:
http://jira.pentaho.com/browse/PRD-3211
which led us to this:
http://bugs.sun.com/view_bug.do?bug_id=4797189
Several hours after that happened, the system finally got itself into a corner that it couldn't get out of and was unable to create a native thread when we needed.
The fix was to get the 3rd party library to close the DeflaterOutputStream.
So definitely a native resource leak. If anyone else is ever hitting something like this, the VMMap tool was indispensable for eventually tracking down which data streams were causing the problem.

I suspect, though it is clearly difficult to prove, that you are running into a 32bit memory allocation problem.
Threads are allocated native memory, not heap memory which has to be contiguous, in order to run. Whilst I am sure that WOW64 allows 32 bit processes to operate in the region above 4gb I'm not so sure about allocating native memory for a new thread above the 4gb limit should the intervening space be used.
Hence your application and the heap are in lowish mem, other processes are taking the intervening 3.07gigs ( if memory serves )and then attempting to allocate a native memory block 4gb above the initial caller in order to create a new thread.
Could you confirm that this issue only occurs when memory use is around or above the 4gb mark?

Solr Indexing Time 32 bit vs 64 bit

I was running 32 bit Java (JDK, JRE & Tomcat) on my 64 bit Windows. For indexing I was not able to allocate more than 1.5GB Heap Space on my machine. Each time my tomcat process used to touch the upper bound (i.e. 1.5GB) very quickly so I thought of working on 64 bit Java/Tomcat. Now I dont see any substantial changes in the indexing running time (though I have increased the heap size now) , it is still taking the same time what it was taking while running on 32 bit software. However the only difference is now the memory consumed by Tomcat is extremely low i.e. now it is not touching that 1.5GB mark. What maybe the reason for this?

The fact that JVM was using X gigabytes of memory at a given time tells virtually nothing.
It could be that it was using those 1.5G for a split second and immediately jumped back to ground zero. Or maybe it was fluctuating around 1.5G all the time. Who knows?
One of the most reasonable ways to investigate this is to run your Tomcat with JMX enabled and connect with a JMX client, e.g. jconsole. You will be able to see memory consumption graphs which will tell you what is the typical memory usage.

Java memory usage on Linux

I'm running a handfull of Java Application servers that are all running the latest versions of Tomcat 6 and Sun's Java 6 on top of CentOS 5.5 Linux. Each server runs multiple instances of Tomcat.
I'm setting the -Xmx450m -XX:MaxPermSize=192m parameters to control how large the heap and permgen will grow. These settings apply to all the Tomcat instances across all of the Java Application servers, totaling about 70 Tomcat instances.
Here is a typical memory usage of one of those Tomcat instances as reported by Psi-probe
Eden = 13M
Survivor = 1.5M
Perm Gen = 122M
Code Cache = 19M
Old Gen = 390M
Total = 537M
CentOS however is reporting RAM usage for this particular process at 707M (according to RSS) which leaves 170M of RAM unaccounted for.
I am aware that the JVM itself and some of it's dependancy libraries must be loaded into memory so I decided to fire up pmap -d to find out their memory footprint.
According to my calculations that accounts for about 17M.
Next there is the Java thread stack, which is 320k per thread on the 32 bit JVM for Linux.
Again, I use Psi-probe to count the number of threads on that particular JVM and the total is 129 threads. So 129 + 320k = 42M
I've read that NIO uses memory outside of the heap, but we don't use NIO in our applications.
So here I've calculated everything that comes to (my) mind. And I've only accounted for 60M of the "missing" 170M.
What am I missing?

Try using the incremental garbage collector, using the -Xincgc command line option.
It's little more aggressive on the whole GC efforts, and has a special happy little anomaly: it actually hands back some of its unused memory to the OS, unlike the default and other GC choices !
This makes the JVM consume a lot less memory, which is especially good if you're running multiple JVM's on one machine. At the expense of some performance - but you might not notice it. The incgc is a little secret it seems, because noone ever brings it up... It's been there for eons (90's even).

Arnar, In JVM initialization process JVM will allocate a memory (mmap or malloc) of size specified by -Xmx and MaxPermSize,so anyways JVM will allocate 450+192=642m of heap space for application at the start of the JVM process. So java heap space for application is not 537 but its 642m.So now if you do the calculation it will give you your missing memory.Hope it helps.

Java allocates as much virtual memory as it might need up front, however the resident side will be how much you actually use. Note: Many of the libraries and threads have their own over heads and while you don't use direct memory, it doesn't mean none of the underlying system do. e.g. if you use NIO, it will use some direct memory even if you use heap ByteBuffers.
Lastly, 100 MB is worth about £8. It may be that its not worth spending too much time worrying about it.

Not a direct answer, but, have you also considered hosting multiple sites within the same Tomcat instance? This could save you some memory at the expense of some additional configuration.

Arnar, the JVM also mmap's all jar files in use, which will use NIO and will contribute to the RSS. I don't believe those are accounted for in any of your measurements above. Do you by chance have a significant number of large jar files? If so, the pages used for those could be your missing memory.

Why does the Sun JVM continue to consume ever more RSS memory even when the heap, etc sizes are stable?

Over the past year I've made huge improvements in my application's Java heap usage--a solid 66% reduction. In pursuit of that, I've been monitoring various metrics, such as Java heap size, cpu, Java non-heap, etc. via SNMP.
Recently, I've been monitoring how much real memory (RSS, resident set) by the JVM and am somewhat surprised. The real memory consumed by the JVM seems totally independent of my applications heap size, non-heap, eden space, thread count, etc.
Heap Size as measured by Java SNMP
Java Heap Used Graph http://lanai.dietpizza.ch/images/jvm-heap-used.png
Real Memory in KB. (E.g.: 1 MB of KB = 1 GB)
Java Heap Used Graph http://lanai.dietpizza.ch/images/jvm-rss.png
(The three dips in the heap graph correspond to application updates/restarts.)
This is a problem for me because all that extra memory the JVM is consuming is 'stealing' memory that could be used by the OS for file caching. In fact, once the RSS value reaches ~2.5-3GB, I start to see slower response times and higher CPU utilization from my application, mostly do to IO wait. As some point paging to the swap partition kicks in. This is all very undesirable.
So, my questions:
Why is this happening? What is going on "under the hood"?
What can I do to keep the JVM's real memory consumption in check?
The gory details:
RHEL4 64-bit (Linux - 2.6.9-78.0.5.ELsmp #1 SMP Wed Sep 24 ... 2008 x86_64 ... GNU/Linux)
Java 6 (build 1.6.0_07-b06)
Tomcat 6
Application (on-demand HTTP video streaming)
High I/O via java.nio FileChannels
Hundreds to low thousands of threads
Low database use
Spring, Hibernate
Relevant JVM parameters:
-Xms128m
-Xmx640m
-XX:+UseConcMarkSweepGC
-XX:+AlwaysActAsServerClassMachine
-XX:+CMSIncrementalMode
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintGCApplicationStoppedTime
-XX:+CMSLoopWarn
-XX:+HeapDumpOnOutOfMemoryError
How I measure RSS:
ps x -o command,rss | grep java | grep latest | cut -b 17-
This goes into a text file and is read into an RRD database my the monitoring system on regular intervals. Note that ps outputs Kilo Bytes.
The Problem & Solutions:
While in the end it was ATorras's answer that proved ultimately correct, it kdgregory who guided me to the correct diagnostics path with the use of pmap. (Go vote up both their answers!) Here is what was happening:
Things I know for sure:
My application records and displays data with JRobin 1.4, something I coded into my app over three years ago.
The busiest instance of the application currently creates
Over 1000 a few new JRobin database files (at about 1.3MB each) within an hour of starting up
~100+ each day after start-up
The app updates these JRobin data base objects once every 15s, if there is something to write.
In the default configuration JRobin:
uses a java.nio-based file access back-end. This back-end maps MappedByteBuffers to the files themselves.
once every five minutes a JRobin daemon thread calls MappedByteBuffer.force() on every JRobin underlying database MBB
pmap listed:
6500 mappings
5500 of which were 1.3MB JRobin database files, which works out to ~7.1GB
That last point was my "Eureka!" moment.
My corrective actions:
Consider updating to the latest JRobinLite 1.5.2 which is apparently better
Implement proper resource handling on JRobin databases. At the moment, once my application creates a database and then never dumps it after the database is no longer actively used.
Experiment with moving the MappedByteBuffer.force() to database update events, and not a periodic timer. Will the problem magically go away?
Immediately, change the JRobin back-end to the java.io implementation--a line line change. This will be slower, but it is possibly not an issue. Here is a graph showing the immediate impact of this change.
Java RSS memory used graph http://lanai.dietpizza.ch/images/stackoverflow-rss-problem-fixed.png
Questions that I may or may not have time to figure out:
What is going on inside the JVM with MappedByteBuffer.force()? If nothing has changed, does it still write the entire file? Part of the file? Does it load it first?
Is there a certain amount of the MBB always in RSS at all times? (RSS was roughly half the total allocated MBB sizes. Coincidence? I suspect not.)
If I move the MappedByteBuffer.force() to database update events, and not a periodic timer, will the problem magically go away?
Why was the RSS slope so regular? It does not correlate to any of the application load metrics.

Just an idea: NIO buffers are placed outside the JVM.
EDIT:
As per 2016 it's worth considering #Lari Hotari comment [ Why does the Sun JVM continue to consume ever more RSS memory even when the heap, etc sizes are stable? ] because back to 2009, RHEL4 had glibc < 2.10 (~2.3)
Regards.

RSS represents pages that are actively in use -- for Java, it's primarily the live objects in the heap, and the internal data structures in the JVM. There's not much that you can do to reduce its size except use fewer objects or do less processing.
In your case, I don't think it's an issue. The graph appears to show 3 meg consumed, not 3 gig as you write in the text. That's really small, and is unlikely to be causing paging.
So what else is happening in your system? Is it a situation where you have lots of Tomcat servers, each consuming 3M of RSS? You're throwing in a lot of GC flags, do they indicate the process is spending most of its time in GC? Do you have a database running on the same machine?
Edit in response to comments
Regarding the 3M RSS size - yeah, that seemed too low for a Tomcat process (I checked my box, and have one at 89M that hasn't been active for a while). However, I don't necessarily expect it to be > heap size, and I certainly don't expect it to be almost 5 times heap size (you use -Xmx640) -- it should at worst be heap size + some per-app constant.
Which causes me to suspect your numbers. So, rather than a graph over time, please run the following to get a snapshot (replace 7429 by whatever process ID you're using):
ps -p 7429 -o pcpu,cutime,cstime,cmin_flt,cmaj_flt,rss,size,vsize
(Edit by Stu so we can have formated results to the above request for ps info:)
[stu#server ~]$ ps -p 12720 -o pcpu,cutime,cstime,cmin_flt,cmaj_flt,rss,size,vsize
%CPU - - - - RSS SZ VSZ
28.8 - - - - 3262316 1333832 8725584
Edit to explain these numbers for posterity
RSS, as noted, is the resident set size: the pages in physical memory. SZ holds the number of pages writable by the process (the commit charge); the manpage describes this value as "very rough". VSZ holds the size of the virtual memory map for the process: writable pages plus shared pages.
Normally, VSZ is slightly > SZ, and very much > RSS. This output indicates a very unusual situation.
Elaboration on why the only solution is to reduce objects
RSS represents the number of pages resident in RAM -- the pages that are actively accessed. With Java, the garbage collector will periodically walk the entire object graph. If this object graph occupies most of the heap space, then the collector will touch every page in the heap, requiring all of those pages to become memory-resident. The GC is very good about compacting the heap after each major collection, so if you're running with a partial heap, there most of the pages should not need to be in RAM.
And some other options
I noticed that you mentioned having hundreds to low thousands of threads. The stacks for these threads will also add to the RSS, although it shouldn't be much. Assuming that the threads have a shallow call depth (typical for app-server handler threads), each should only consume a page or two of physical memory, even though there's a half-meg commit charge for each.

Why is this happening? What is going on "under the hood"?
JVM uses more memory than just the heap. For example Java methods, thread stacks and native handles are allocated in memory separate from the heap, as well as JVM internal data structures.
In your case, possible causes of troubles may be: NIO (already mentioned), JNI (already mentioned), excessive threads creation.
About JNI, you wrote that the application wasn't using JNI but... What type of JDBC driver are you using? Could it be a type 2, and leaking? It's very unlikely though as you said database usage was low.
About excessive threads creation, each thread gets its own stack which may be quite large. The stack size actually depends on the VM, OS and architecture e.g. for JRockit it's 256K on Linux x64, I didn't find the reference in Sun's documentation for Sun's VM. This impacts directly the thread memory (thread memory = thread stack size * number of threads). And if you create and destroy lots of thread, the memory is probably not reused.
What can I do to keep the JVM's real memory consumption in check?
To be honest, hundreds to low thousands of threads seems enormous to me. That said, if you really need that much threads, the thread stack size can be configured via the -Xss option. This may reduce the memory consumption. But I don't think this will solve the whole problem. I tend to think that there is a leak somewhere when I look at the real memory graph.

The current garbage collector in Java is well known for not releasing allocated memory, although the memory is not required anymore. It's quite strange however, that your RSS size increases to >3GB although your heap size is limited to 640MB. Are you using any native code in your application or are you having the native performance optimization pack for Tomcat enabled? In that case, you may of course have a native memory leak in your code or in Tomcat.
With Java 6u14, Sun introduced the new "Garbage-First" garbage collector, which is able to release memory back to the operating system if it's not required anymore. It's still categorized as experimental and not enabled by default, but if it is a feasible option for you, I would try to upgrade to the newest Java 6 release and enable the new garbage collector with the command line arguments "-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC". It might solve your problem.

Java memory consumption, "top" and HP-Ux

We ship Java applications that are run on Linux, AIX and HP-Ux (PA-RISC). We seem to struggle to get acceptable levels of performance on HP-Ux from applications that work just fine in the other two environments. This is true of both execution time and memory consumption.
Although I'm yet to find a definitive article on "why", I believe that measuring memory consumption using "top" is a crude approach due to things like the shared code giving misleading results. However, it's about all we have to go on with a customer site where memory consumption on HP-Ux has become an issue. It only became an issue this time when we moved from Java 1.4 to Java 1.5 (on HP-Ux 11.23 PA-RISC). By "an issue", I mean that the machine ceased to create new processes because we had exhausted all 16GB of physical memory.
By measuring "before" and "after" total "free memory" we are trying to gauge how much has been consumed by a Java application. I wrote a quick app that stores 10,000 random 64 bit strings in an ArrayList and tried this approach to measuring consumption on Linux and HP-Ux under Java 1.4 and Java 1.5.
The results:
HP Java 1.4 ~60MB
HP Java 1.5 ~150MB
Linux Java 1.4 ~24MB
Linux Java 1.5 ~16MB
Can anyone explain why these results might arise? Is this some idiosyncrasy of the way "top" measures free memory? Does Java 1.5 on HP really consume 2.5 times more memory than Java 1.4?
Thanks.

The JVMs might just have different default parameters. The heap will grow to the size that you have configured to let it. The default on the Sun VM is a certain percentage of the RAM in the machine - that's to say that Java will, by default, use more memory if you use a machine with more memory on it.
I'd be really surprised if the HP-UX VM hadn't had lots of tuning for this sort of thing by HP. I'd suggest you fiddle with the parameters on both - figure out what the smallest max heap size you can use without hurting performance or throughput.

I don't have a HP box right now to test my hypothesis. However, if I were you, I would use a profiler like JConsole(comes with JDK) OR yourkit to measure what is happening.
However, it appears that you started measuring after you saw something amiss; So, I'm NOT discounting that it's happening -- just pointing you at something I'd have done in the same situation.

First, it's not clear what did you measure by "10,000 random 64 bit strings" test. You supposed to start the application, measure it's bootstrap memory footprint, and then run your test. It could easily be that Java 1.5 acquires more heap right after start (due to heap manager settings, for instance).
Second, we do run Java apps under 1.4, 1.5 and 1.6 under HP-UX, and they don't demonstrate any special memory requirements. We have Itanium hardware, though.
Third, why do you use top? Why not just print Runtime.getRuntime().totalMemory()?
Fourth, by adding values to ArrayList you create memory fragmentation. ArrayList has to double it's internal storage now and then. Depending on GC settings and ArrayList.ensureCapacity() implementation the amount of non-collected memory may differ dramatically between 1.4 and 1.5.
Essentially, instead of figuring out the cause of problem you have run a random test that gives you no useful information. You should run a profiler on the application to figure out where the memory leaks.

You might also want to look at the problem you are trying to solve... I don't imagine there are many problems that eat 16GB of memory that aren't due for a good round of optimization.
Are you launching multiple VMs? Are you reading large datasets into memory, and not discarding them quickly enough? etc etc etc.