I always thought, that the memory of permsize of a JVM is filled with loading classes during starting up the JVM. Probably also with stuff like JNI during runtime ? But in general it should not growth during runtime "signifcantly".
Now I noticed, that since I load a lots of data (20GB) into the heapspace, which max is 32GB ( ArrayLists of Data ), then I get a 'OutOfMemoryError: PermGen space'.
Is there any correlation or just accidentally ?
I know howto increase the permsize. This is not the question.
With tomcat, I have set the following for increasing PermGen space.
set "JAVA_OPTS=-XX:MaxPermSize=256m"
You may like to do something like above.
I have set in MB(256m), I am not sure how to set for GB.
Hope helps.
The PermGen memory space is not part of the heap (sometimes this causes confusion). It's where some kind of objects are allocated, like
Class objects, Method objects, and the pool of strings objects. Unlike the name would indicate, this memory space is also collected (during
the FullGC), but often bring major headaches, as known
OutOfMemoryError.
Problems with bursting PermGen are difficult to diagnose precisely
because it is not the application objects . Most of the cases, the problem is connected to
an exaggerated amount of classes that are loaded into memory. A well known issue, was the use
of Eclipse with many plugins ( WTP ) with default JVM settings . Many classes were loaded in memory and ends with a burst of the permGEN.
Another problem of PermGen are the hot deploys in application servers. For several reasons, the server cannot release
the context classes at the destroy time . A new version of the application is then loaded,
but old the classes remains, increasing the PermGen.
That's why sometimes we need to restart the whole container because of the PermGen.
Related
I have recently discovered the following facts:
The first time I navigate through pages of my application, perm gem is growing up significantly. (That's normal)
My perm gen is growing up when I am navigating through pages of my application that I have already navigated through.
But this is only happening if I stop using my application for a few minutes. If I don't do that perm gen remains the same though I keep navigating.
The increase is not much but I think is not a normal behaviour.
I also noticed that perm gen never goes down or It goes down a little bit.
What can be the cause of that?
I am using Java6, Tomcat6, Hibernate, Spring 3.3.4 and JSF 1.2.
PermGen errors and leaks are tricky to investigate.
To deal with them, them, the first thing you can do is to check you statics. Your application can manage some statics that are holding references to objects that should have been garbage collected but are still marked as reachable because are referenced by the these statics.
For example, you are using Spring, and in Spring, beans are Singleton by default so it can be a source of PermGen problems.
To know more about the objects in memory you can use some profiling tools, such as VisualVM. It can help you to investigate how many instances of a given class is holding Tomcat, and it can give you a clue about where the leak can be.
To learn more about the PermGen and PermGen errors and issues you can look at The java.lang.OutOfMemoryError: PermGen Space error demystified
Yesterday I deployed my first Grails (2.3.6) app to a dev server and began monitoring it. I just got an automated monitor stating that CPU was pinned on this machine, and so I SSHed into it. I ran top and discovered that it was my Java app's PID that was pinning the server. I also noticed memory was at 40%. After a few seconds, the CPU stopped pinning, went down to a normal level, and memory went back down into the ~20% range. Classic major GC.
While it was collecting, I did a heap dump. After the GC, I then opened the dump in JVisualVM and saw that most of the memory was being allocated for an org.codehaus.groovy.runtime.metaclass.MetaMethodIndex.Entry class. There were almost 250,000 instances of these in total, eating up about 25 MB of memory.
I googled this class and took a look at it's ultra helpful Javadocs. So I still have no idea what this class does.
But googling it also brought up about a dozen or so related articles (some of them SO questions) involving this class and a PermGen/classloader leak with Grails/Groovy apps. And while it seems that my app did in fact clean up these 250K instance with a GC, it still is troubling that there were so many instances of it, and that the GC pinned CPU for over 5 minutes.
My questions:
What is this class and what is Groovy doing with it?
Can someone explain this answer to me? Why would -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled help this particular problem?
Why is this class particularly troublesome for the PermGen?
Groovy is a dynamic language, every method call is dispatched dynamically. To optimise that Groovy creates a MetaClass for every java.lang.Class in the MetaClassRegistry. These MetaClass instances are created on-demand and stored using Weak references.
The reason you see a lot of org.codehaus.groovy.runtime.metaclass.MetaMethodIndex.Entry is because Groovy is storing a map of classes and methods in memory so that they can be quickly dispatched by the runtime. Depending on the size of the application this can be as you have discovered thousands of classes as each class can have dozens sometimes hundreds of methods.
However, there is no "memory leak" in Groovy and Grails, what you are seeing is normal behaviour. Your application is running low on memory, probably because it hasn't been allocated enough memory, this in turn causes MetaClass instances to be garbage collected. Now say for example you have a loop:
for(str in strings) {
println str.toUpperCase()
}
In this case we are calling a method on the String class. If you are running low on memory what will happen is that for each iteration of the loop the MetaClass will be garbage collected and then recreated again for the next iteration. This can dramatically slow down an application and lead to the CPU being pinned as you have seen. This state is commonly referred to as "metaclass churn" and is a sign your application is running low on heap memory.
If Groovy was not garbage collecting these MetaClass instances then yes that would mean there is a memory leak in Groovy, but the fact that it is garbage collecting these classes is a sign that all is well, except for the fact that you have not allocated enough heap memory in the first place. That is not to say that there may be a memory leak in another part of the application that is eating up all the available memory and leaving not enough for Groovy to operate correctly.
As for the other answer you refer to, adding class unloading and PermGen tweaks won't actually do anything to resolve your memory issues unless you dynamically parsing classes at runtime. PermGen space is used by the JVM to store dynamically created classes. Groovy allows you to compile classes at runtime using GroovyClassLoader.parseClass or GroovyShell.evaluate. If you are continuously parsing classes then yes adding class unloading flags can help. See also this post:
Locating code that is filling PermGen with dead Groovy code
However, a typical Grails application does not dynamically compile classes at runtime and hence tweaking PermGen and class unloading settings won't actually achieve anything.
You should verify if you have allocated enough heap memory using the -Xmx flag and if not allocate more.
I know that in the JVM, the permgen area is used to store class definitions. In my Tomcat I see that the current memory usage of the permgen is near 100MB, it seems that it's just growing over time, even there's no one using the applications in Tomcat
My questions are:
Is it true that the permgen is never garbage collected, I mean the memory used there keeps growing and growing?
When the permgem gets garbage collected?
What does mean "CMSPermGenSweepingEnabled" and "CMSClassUnloadingEnabled"?
My max size of permgem is 256 and I don't want to have an OutMemoryException next week.
Please only accurate and documented answers.
I use Tomcat 7, Java 7, I use a lot the parallel deployment tecnique and I do undeploys, redeploys several times a week.
I never use the method intern() of Strings
actually it's not true that Permgen does never get garbage collected. It contains the classes that where loaded by the application, and gets collected when classloaders get garbage collected, typically in redeployment scenarios.
You can use these JVM flags to see when classes are loaded into and unloaded from the permgen:
-XX:+TraceClassLoading -XX:+TraceClassUnloading
To see which classes are getting loaded, use this flag:
-verbose:class
If the application is reflection intensive, that can be a cause too, have a look at this answer, try to use visualvm to take heap dumps an look for classes named lie sun.reflect.GeneratedMethodAccessor11.
For the garbage collection flags refer to this answer, but the best bet to fix the permgen leak is to see what classes are being created and why using some tooling/logs.
I constantly detect OOM in PermGen for my environment:
java 6
jboss-4.2.3
Not a big web-application
I know about String.intern() problem - but I don't have enough valuable usage of it.
Increasing of MaxPermGen size didn't take a force (from 128 Mb to 256 Mb).
What other reasons could invoke OOM for PermGen?
What scenario of investigation is the best in such situation (strategy, tools and etc.)?
Thanks for any help
See this note
Put JDBC driver in common/lib (as tomcat documentation says) and not in WEB-INF/lib
Don't put commons-logging into WEB-INF/lib since tomcat already bootstraps it
new class objects get placed into the PermGen and thus occupy an ever increasing amount of space. Regardless of how large you make the PermGen space, it will inevitably top out after enough deployments. What you need to do is take measures to flush the PermGen so that you can stabilize its size. There are two JVM flags which handle this cleaning:
-XX:+CMSPermGenSweepingEnabled
This setting includes the PermGen in a garbage collection run. By default, the PermGen space is never included in garbage collection (and thus grows without bounds).
-XX:+CMSClassUnloadingEnabled
This setting tells the PermGen garbage collection sweep to take action on class objects. By default, class objects get an exemption, even when the PermGen space is being visited during a garabage collection.
You typically get this error when redeploying an application while having a classloader leak, because it means all your classes are loaded again while the old versions stay around.
There are two solutions:
Restart the app server instead of redeploying the application - easy, but annoying
Investigate and fix the leak using a profiler. Unfortunately, classloader leaks can be very hard to pinpoint.
I know there are several memory types that Tomcat uses when running.
The only I have ever used - java heap. It can be controlled through JAVA_OPTS env property with something like '-Xmx128M -Xms64M'
I have found that there is also -XX:MaxPermSize, -XX:MaxNewSize and etc.
The reason I'm asking is that I'm trying to launch Tomcat5.5 on 200Mb RAM memory (it is VPS server). I have setup java heap size with '-Xmx128M -Xms64M', but it seems that right from startup it consumes more than that (if ever can start. Sometimes startup fails right off the bat with OutOfMemoryException), with no applications have been deployed
Noticable thing is that if I launch maven's tomcat plugin, it works just fine. Only separate tomcat fails with memory.
Thanks in advance for your ideas.
As you say, heap memory is just one of the JVM's memory pools, there are others.
Read this to get an idea of what they are, how to control them, and how to monitor them:
http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html
Heap and Non-heap Memory
The JVM manages two kinds of memory:
heap and non-heap memory, both created
when it starts.
Heap memory is the runtime data area
from which the JVM allocates memory
for all class instances and arrays.
The heap may be of a fixed or variable
size. The garbage collector is an
automatic memory management system
that reclaims heap memory for objects.
Non-heap memory includes a method area
shared among all threads and memory
required for the internal processing
or optimization for the JVM. It stores
per-class structures such as a runtime
constant pool, field and method data,
and the code for methods and
constructors. The method area is
logically part of the heap but,
depending on implementation, a JVM may
not garbage collect or compact it.
Like the heap, the method area may be
of fixed or variable size. The memory
for the method area does not need to
be contiguous.
In addition to the method area, a JVM
implementation may require memory for
internal processing or optimization
which also belongs to non-heap memory.
For example, the JIT compiler requires
memory for storing the native machine
code translated from the JVM code for
high performance.
Read here for some tips on setting the java heap size. It is quite strange that Tomcat is giving you OutOfMemoryExceptions even without any applications deployed. Perhaps there is something wrong with your configuration (what OS are you using, how do you start Tomcat?).