It looks like
MemoryError: PermGen space
java.lang.OutOfMemoryError: PermGen space
is a common problem. You can Increase the size of your perm space, but after 100 or 200 redeploys it will be full. Tracking ClassLoader memory leaks is nearly impossible.
What are your methods for Tomcat (or another simple servlet container - Jetty?) on production server? Is server restart after each deploy a solution?
Do you use one Tomcat for many applications ?
Maybe I should use many Jetty servers on different ports (or an embedded Jetty) and do undeploy/restart/deploy each time ?
I gave up on using the tomcat manager and now always shutdown tomcat to redeploy.
We run two tomcats on the same server and use apache webserver with mod_proxy_ajp so users can access both apps via the same port 80. This is nice also because the users see the apache Service Unavailable page when the tomcat is down.
You can try adding these Java options:
-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
This enables garbage collection in PermGen space (off by default) and allows the GC to unload classes. In addition you should use the -XX:PermSize=64m -XX:MaxPermSize=128m mentioned elsewhere to increase the amount of PermGen available.
Yes indeed, this is a problem. We're running three web apps on a Tomcat server: No. 1 uses a web application framework, Hibernate and many other JARs, no. 2 uses Hibernate and a few JARs and no. 3 is basically a very simple JSP application.
When we deploy no. 1, we always restart Tomcat. Otherwise a PermGen space error will soon bite us. No. 2 can sometimes be deployed without problem, but since it often changes when no. 1 does as well, a restart is scheduled anyway. No. 3 poses no problem at all and can be deployed as often as needed without problem.
So, yes, we usually restart Tomcat. But we're also looking forward to Tomcat 7, which is supposed to handle many memory / class loader problems that are burried into different third-party JARs and frameworks.
PermGen switches in HotSpot only delay the problem, and eventually you will get the OutOfMemoryError anyway.
We have had this problem a long time, and the only solution I've found so far is to use JRockit instead. It doesn't have a PermGen, so the problem just disappears. We are evaluating it on our test servers now, and we haven't had one PermGen issue since the switch. I also tried redeploying more than 20 times on my local machine with an app that gets this error on first redeploy, and everything chugged along beautifully.
JRockit is meant to be integrated into OpenJDK, so maybe this problem will go away for stock Java too in the future.
http://www.oracle.com/technetwork/middleware/jrockit/overview/index.html
And it's free, under the same license as HotSpot:
https://blogs.oracle.com/henrik/entry/jrockit_is_now_free_and
You should enable PermGen garbage collection. By default Hotspot VM does NOT collect PermGen garbage, which means all loaded class files remain in memory forever. Every new deployment loads a new set of class files which means you eventually run out of PermGen space.
Which version of Tomcat are you using? Tomcat 7 and 6.0.30 have many features to avoid these leaks, or at least warn you about their cause.
This presentation by Mark Thomas of SpringSource (and longtime Tomcat committer) on this subject is very interesting.
Just of reference, there is a new version of Plumbr tool, that can monitor and detect Permanent Generation leaks as well.
Related
I'm currently investigating some out of meta space issues we've been experiencing recently. One of the main culprits seems to be the loading of duplicate classes upon redeployment of a WAR. Trying it out locally, with just one of our WARS, by heap dumping after undeploying completely, I can see that the majority of the instances created by the application are still there (even after garbage collection).
From the heap dump, I can see that it seems to be the ManagedThreadFactoryImpl that is holding onto references.
Is there anything I can do/add to the application shutdown process so it cleans up after itself?
All our WARs are spring applications, most use scheduling/asynchronous elements.
We're using JDK8 with Wildfly 8.2
Seems like the classloaders are not unloading. Try Java Mission Control (JMC) and record the use case. This lets you go to a specific point in time in your recording and debug the issue. It gives the snapshot of classes loaded at a specific time with stacktrace, threaddumps and a lot of important things.
JMC is included in JDK. You can find more info here: https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr002.html#BABIBBDE
You dont have to go through the pain of taking heap dumps and then wait for a tool to analyze it.
Good morning, does anyone know how to solve this in jsf applications after a reload, stop, or undeploy?
The following web applications were stopped (reloaded, undeployed), but their
classes from previous runs are still loaded in memory, thus causing a memory
leak (use a profiler to confirm):
/aplicacaojsf
During development it's perfectly valid to do hot-deploy many times - it will result in an PermGen space error eventually (at least before using JDK 1.8) and you'll have to restart. The message is just a warning to the fact which happens to all containers when re-deploying, that parts of the previous classes are still in memory and will stay there until the end of days.
In production environment, it is good to restart the container - but even there it's not feasible all the times, since that may mean downtime for other applications as well. That's why commercial application servers like Oracle WebLogic do have a sophisticated admin console for jobs like that.
Is it possible to get this error on tomcat server in other way then redeploy wars or edit jsp files? We got this error on server where theoretically we don't do redeploys. Do you know best solution to monitor PermGen from linux console?
permGen means Permanent generation, this is the place where all your constant are stored like strings (in most cases before java 7) were stored on this permgen, One way to get rid of it is you simply increase the memory using
-XX:PermSize=512m
This is what I did, but from your scenario it feels like there is some kind of memory leak, I am not sure how to detect it, there are frameworks available for this and netbeans also provides application profiling support.
Here are some good links
http://www.eclipse.org/tptp/home/documents/tutorials/profilingtool/profilingexample_32.html
http://www.ej-technologies.com/products/jprofiler/overview.html
https://netbeans.org/kb/docs/java/profiler-intro.html
It doesn't have to be because of redeployments or more over a file edits.
It is more likely that your Java apps on the server are using up all allocated memory.
So yes, it is possible to get this error on tomcat server in other way then redeploy wars or edit jsp files.
When it comes to monitoring you might be interested in this API: http://docs.oracle.com/javase/7/docs/api/java/lang/management/MemoryMXBean.html
Or try looking for a monitoring software by typing tomcat monitor permgen in Google - lots of results are being returned.
There is a tool for remote monitoring of VM: http://visualvm.java.net/
I remember the default permgen on Tomcat being pretty low, if you have a decent sized application with a lot of third party dependencies this can cause loads of classes to reside in pergmen. You could legitimately need more pergmen space, try increasing it.
I am working in a Java/J2EE project with JBoss as an application server.
We build our war and do hot deployment in server using Jenkins.Sometimes, we get some Out of Memory error in JBoss.
I wonder if hot deployment is responsible for that. Also, would like to know if hot deployment has any pitfalls over normal manual start-stop deployment.
Can someone please provide some valuable inputs?
I agree with the answers about adjusting your heap/permgen space, although its hard to be specific without more information about how much memory is allowed, what you are using etc.
Also, would like to know if hot deployment has any pitfalls over normal manual start-stop deployment.
When you manually start and stop the service between deployments you can be a little sloppy about cleaning up your web app - and no one will ever know.
When you hot deploy, the previous instance of your servlet context is destroyed. In order to reduce the frequency of OutOfMemory exceptions, you want to make sure that when this occurs, you clean up after yourself. Even though there is nothing you can do about classloader PermGen memory problems, you don't want to compound the problem by introducing additional memory leaks.
For example - if your war file starts any worker threads, these need to be stopped. If you bind an object in JNDI, the object should be unbound. If there are any open files, database connects etc. these should be closed.
If you are using a web framework like Spring - much of this is already taken care of. Spring registers a ServletContextListener which automatically stops the container when the servlet context is destroyed. However you would still need to make sure that any beans which create resources during init will clean up those resources during destroy.
If you are doing a hand-crafted servlet, then you would want to register an implementation of ServletContextListener in your web.xml file, and in the implementation of contextDestroyed clean up any resources.
BTW - you should include the exact OutOfMemory exception in your answer. If it says something like java.lang.OutOfMemoryError: PermGen space, then it's probably an issue of class instances and there is not much you can do. If it is java.lang.OutOfMemoryError: Java heap space then perhaps it's memory in your application that is not being cleaned up
Hot deployment does not clear up the previously loaded Class instances in Perm Gen. It loads the Class instances afresh. A little google pointed me back to SO What makes hot deployment a "hard problem"?
You should increase Heap Space specifically Perm Space
-Xms<size> set initial Java heap size
-Xmx<size> set maximum Java heap size
-XX:MaxPermSize set maximum Permanent generation size
You can set it in JAVA_OPTS in your jboss run.sh or run.bat like:
-Xms256m -Xmx1024m -XX:MaxPermSize=512m
Recently I created a maven based web project and used tomcat as application server to debug ...
But tomcat is frequently dead (pergem error which means out of memery ) after run app from the Project context menu directly.
The worst is that It created many idle threads and they are all can not be killed by manually.
And tomcat status is also can not be detected by NetBeans, it can not be stop and restart.
I must restart my system to clean them.
My system is Fedora 12 x86...
Java version is SUN JDK 6 update 17.
NetBeans verison is 6.7.1.
I tried to create a bug about this, but the NetBeans developer rejected it ... I am very surprise this is a big stopper to use NetBeans to develop web app.
Today I used JBoss 5.1 in the latest NetBeans 6.8rc1 and also encountered the same problem...
I've tried adjusted the VM parameters and allocate more memory but no effects.
I've deployed several seam samples to JBoss 5.1 via seam ant build script , and there is no problem.
But I switch to use NetBeans to do this, It failed due to Out of memory.
What is wrong????
(I'm assuming that the idle threads you are talking about are application threads. If you are talking about Tomcat's worker threads, then they should not cause permgen problems when idle.)
I tried to create a bug about this, but the NetBeans developer rejected it ... I am very surprise ...
I'm very unsurprised. This is not really a NetBeans problem. It is not even really a Tomcat or JBoss problem.
The problem with idle / orphaned application threads is that you cannot safely kill threads in a running JVM. And without the ability to do this you cannot stop leakages of (in this case) permgen space. This is essentially a Java platform problem. It can only be solved when JVMs support the Isolate mechanisms defined by JSR 121.
But assuming that your problem is related to doing hot deployment, it is not just the idle application threads that you have to worry about. If there is any reachable instance of any class that you are redeploying, that will cause the original old class loader and all classes it loaded to remain reachable.
... this is a big stopper to use NetBeans to develop web app.
The solution is to restart your web container every now and then. This will clean out the idle threads and other cruft and release the leaked permgen space. You can ameliorate it by running with a larger heap and more permgen, but you cannot make it go away, especially if some of the orphaned threads are not idle ...
Oh yea, and don't be a big baby! This ain't a show stopper, its just a small inconvenience :-)