We are currently using Apache Tomcat 7.0.50 deployed with multiple web application instances and where we are trying to fix some of the Bugs in the catalina.out which appears severe. Two of those are as below.
org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/app_name] created a ThreadLocal with key of type [class name] (value [class name]) and a value of type [classname] (value [classname#4b188d84]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc SEVERE: The web application [/app_name] registered the JDBC driver [oracle.jdbc.driver.OracleDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
We have really seen many solutions to these errors above in various communities, blogs and FAQ sessions, but are trying to find
Why are these happening to be precise?
Are these considered severe and needs to be fixed , for sure?
How can we resolve , or work around these with some Apache Tomcat configurations? We have tried many in context.xml, web.xml, server.xml etc, but couldn't succeed.
In the worst case, if at any case these can be hidden from catalina.out, how can we do that?
Kindly help us here. Any help would be appreciated and accepted. Thanks in advance.
Note: For iten 2, I have found a solution as mentioned in Oracle driver memory leak - Tomcat but have not given it a try. Will that really work for Tomcat 7.0.50?
Regards,
Sam Mohan.
Both of these warnings happen for much the same reason. An object outside of the web application is holding onto a reference to an object that is within the application. These are code level problems that cannot be fixed via configuration.
How big a problem this turns out to be in practice really depends on 1) how often you hot undeploy/redeploy the application in question and 2) how big parts of the application that are being held on to are. I typically find that this problem stings me a lot more in development where I hot swap apps a lot, but much less so in production because I tend to restart tomcat as a deploy updates. Redeploy an app with this problem enough times, and the JVM will crash from running out of memory.
This particular problem cannot be resolved from a configuration change, unless you count scripting a server restart as a configuration solution. You will need to make code changes to break the object references that point into the web app. They can be fairly tricky to track down, but as the warnings highlight start by reviewing the use of all thread locals. A more complete review would include taking a heap dump of tomcat after unloading the application and then using a heap analyser to trace in memory references to the application that was unloaded (there should be none!).
Related
On tomcat/manager when I click on find leaks button I get the following message:
The following web applications were stopped (reloaded, undeployed), but their
classes from previous runs are still loaded in memory, thus causing a memory
leak (use a profiler to confirm):
This is actually causing big problems for me because when I need to re-deploy an app with some changes, the old version remains in the server and when I try to use it I get:
IllegalStateException: Illegal access: this web application instance has been stopped already
Even though in tomcat/manager the application shows as started.
How do I resolve this? I looked at here but did not solve the problem. I tried running jps command to get pid of JVM but it doesn't return the JVMs, I guess due to permissoin issue.
Is it possible to configure tomcat somehow so that when an application is undeployed, it shouldn't keep any classes of that application in the memory?
The question How to purge tomcat's cache when deploying a new .war file? Is there a config setting? is not solving my problem as I followed the steps given there:
undeploy the app
stop tomcat
delete the app from `work` directory of tomcat
clear browser cache
start tomcat
put the war file in `webapps`
wait a few moments
start the app
those steps didn't solve the problem
There is no way to configure tomcat to drop classes that are still referenced. An easy way (probably not the only one) to cause this is to start a thread from your webapplication and not terminate it when the app is undeployed: To tomcat, this thread is then unavailable, but for the JVM, the running thread is a valid root for classes that can't be garbage collected.
Tomcat does its best to detect and indicate such a condition, but really can't go against the JVM and garbage collector and discard perfectly valid objects that are still referenced.
The steps you give, including stopping and starting tomcat, seem impossible: Once you terminate the JVM, the only chance for a newly started JVM to reference the old classes is if it gets hold of them. But you also state that you're deleting the work directory (what about the temp directory?). You should also make sure that you're actually deleting the right content, not some other server's content in a similar directory. Other than that: It seems you're just missing an almost obvious place where the old classes are located. Once they're picked up on restart, the condition that I'm describing above might kick in. There's no way around implementing the webapp to behave nicely and not start random background threads without terminating them.
You might want to check if CATALINA_HOME and CATALINA_BASE are different and might give a hint about other applications deployed under the same name, and if your application gets started when you just delete the files you describe, then start (without new deployment)
I know Tomcat can reload the .war file when redeploy it, I don't need to kill the Tomcat process and restart it. I can remove the .war, wait for Tomcat to undeploy it, and copy the new .war to the web path. But, after many times of trivial update war without restart the Tomcat, is it possible that Tomcat will not release the memory effectively or cause some performance issues? Assume there is only one war application in one Tomcat instance.
The basic problem is that Java does currently not provide any kind of isolation between the parts of code running in a Java Virtual Machine (JVM), in the same way that an operating system does with processes. You can kill a process without affecting another process under Windows/Linux/etc. All you can do is ensuring that things can be garbage collected.
For Tomcat the way that WAR's are being handled - according to the various specifications - require that each war has its own classloader that is responsible for running that code. When the WAR is undeployed the final result should be that that class loader should be garbage collected.
Unfortunately the garbage collector can only handle objects that are completely unreferenced, and there is a large set of subtle bugs that can be present in WAR code that can prohibit this, and then every redeploy cause another classloader to be created and none destroyed so you have a memory leak. A lot of effort has gone into detecting and working around these type of bugs inside Tomcat itself, but it is close to impossible to do 100% right without JVM support.
The only cure besides fixing the WAR is to restart the JVM.
You can watch the memory usage with VisualVM even in production to see what happens over time with the Tomcat JVM.
Yes. It is far cleaner to stop Tomcat, deploy your new war, then restart Tomcat. One drawback is much of your application classes by default won't be loaded till a new request comes in to hit your application but its not a huge issue. Just means theres a few seconds of startup on the very first request to your new WAR. This is how we deploy wars in production.
Also allows us to setup health-checking in the logs if the new war prevents Tomcat from starting up correctly then we rollback the war knowing theres an issue, but thats a separate topic.
What About Down Time?
This might be out of scope of your question but when you want to prevent users from seeing any downtime you would run multiple instance of tomcat and deploy and restart one at a time.
I have a problem about Memory Leak in Tomcat server.
When I undeploy and redeploy the error message war appeared as below:
The following web applications were stopped (reloaded, undeployed), but their classes from previous runs are still loaded in memory, thus causing a memory leak (use a profiler to confirm): /myWebApplication
How can I fix it and deploy as normally
If anyone known that , please help me .
Thank you very much !
Make sure that you implement a ServletContextListener and clean up any resources (e.g. threads, timers, singletons, hibernate) that may still be active or referenced at the time that contextDestroyed() is called.
I have a web application which often takes an inordinate amount of time to deploy on Tomcat. My suspicion is that there's a database connection somewhere that's waiting until a timeout, but that's just a guess and I want to find out for sure what's causing the hold up so I can remedy the issue. Can anyone suggest a way I can go about doing this? Should I profile Tomcat when it's loading the WAR and look there for clues? If so is there a tutorial somewhere that's good for a beginner?
In case this matters my web application uses Spring and Hibernate. I've been told by a colleague that perhaps these are causing the slow down in that they are so huge that somewhere a class loader is choking on the sheer number of classes it needs to load.
Also I see this when I stop Tomcat or hot deploy the WAR to an already running Tomcat:
Jun 1, 2012 6:03:33 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/nacem-rest] registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Jun 1, 2012 6:03:34 PM org.apache.catalina.startup.HostConfig deployWAR
Perhaps this is part of the problem? Redeploying my web application almost always requires a restart of Tomcat as well, and I've always assumed that the above mentioned JDBC driver issue was to blame, but maybe it also has something to do with the sluggish start time as well?
You can use a profiler, but it is a bit of an overkill. Instead just make few thread dumps while the application loads. You can use jstack or jvisualvm, plenty of resources over the net describing how to do this. Basically you need a PID of your Tomcat (see: jps).
If you find it hard to analyse the thread dump - add it to your question. Typically it's pretty easy to find a bottleneck. Typical problems:
the application tries to fetch some resource from the Internet like XML schema
a lot of classes to scan on the CLASSPATH
excessive GC (enable verbose gc logging just in case)
not enough memory (swapping, see iostat/iotop)
the first I would look is how the CPU or RAM usage of tomcat perform during your application startup.
If you see a lot of CPU activity and RAM increase, then it's probably loading a lot of stuff, like a lot of classes, or performing some kind of huge pre-allocation or similar. In that case thread dumps can help you a lot, but also "lsof" (if your tomcat is running on an *nix environment) to see what files it is currently working on.
However, if you see it simply sitting there, then it is probably waiting on some kind of connection.
One cause may be a DB connection, as you supposed. The DB is usually fast to answer, and a failed attempt to connect to a DB is usually clearly logged somewhere, but still it could be.
Another, less known, cause may be some XML validation. It is not uncommon for a web application to load some XML data, and it is not uncommon to use a validating parser to load that XML. The validating parser needs a schema or DTD to validate, and the URL for the schema/DTD file is often included in the XML. So, some parsers will try to load the schema file from internet to validate the XML, and that could take a lot of time being an internet connection. Moreover, some parsers will fail silently and simply not validate the XML, probably after a rather long timeout. Sorry for being vague on "some parsers" etc.. but if XML loading is done by libraries used inside your webapp, they could use the JVM XML parser, any possible version of Xerces, any possible version for JDOM etc... and even worst different libraries may be using different parsers.
However, if it is a connection problem, it's easier to see it using "netstat -anlp | grep java" (in your tomcat is on a *nix environment, otherwise it should be "netstat -ano" on windows), and looking for what outgoing connection your tomcat is trying to make. There you can see DB connections and also outgoing (usually http) connections searching for schemes or other stuff.
If you are using Tomcat 7.0.x, it might be scanning the classpath to detect annotations.
If you have a lot of jars / classes, then this can be an issue.
As Simone says, taking and analysing a couple of thread dumps is a good way to start.
If you are not using a recent version of Tomcat, try upgrading & checking the conf/catalina.properties file for hints on how to reduce the number of jars that are scanned.
You should take a few thread dumps during deployment (the more the better), and analyze them. Look for blocks, waits etc. Thread Dump Analyzer might be of use.
This may be relate to this BUG of JDK
I readed this article, Tomcat7 starts too late on Ubuntu 14.04 x64 and resolved my problem.
Try to resolve it by Replacing securerandom.source=file:/dev/urandom with securerandom.source=file:/dev/./urandom in $JAVA_PATH/jre/lib/security/java.security
I found that I had a large delay when deploying a web application to Tomcat8. I had three database connection pools configured in my context.xml file within the WAR. Each connection pool had an initial count of 10 connections so Tomcat was creating 30 database connections at startup. I found that configuring each pool for 1 initial connection on my test system improved the deploy time from approximately 3 minutes to 22 seconds.
Append below in setenv.sh of your working tomcat7/bin directory:
JAVA_OPTS="-Djava.security.egd=file:/dev/./urandom"
Recently I created a maven based web project and used tomcat as application server to debug ...
But tomcat is frequently dead (pergem error which means out of memery ) after run app from the Project context menu directly.
The worst is that It created many idle threads and they are all can not be killed by manually.
And tomcat status is also can not be detected by NetBeans, it can not be stop and restart.
I must restart my system to clean them.
My system is Fedora 12 x86...
Java version is SUN JDK 6 update 17.
NetBeans verison is 6.7.1.
I tried to create a bug about this, but the NetBeans developer rejected it ... I am very surprise this is a big stopper to use NetBeans to develop web app.
Today I used JBoss 5.1 in the latest NetBeans 6.8rc1 and also encountered the same problem...
I've tried adjusted the VM parameters and allocate more memory but no effects.
I've deployed several seam samples to JBoss 5.1 via seam ant build script , and there is no problem.
But I switch to use NetBeans to do this, It failed due to Out of memory.
What is wrong????
(I'm assuming that the idle threads you are talking about are application threads. If you are talking about Tomcat's worker threads, then they should not cause permgen problems when idle.)
I tried to create a bug about this, but the NetBeans developer rejected it ... I am very surprise ...
I'm very unsurprised. This is not really a NetBeans problem. It is not even really a Tomcat or JBoss problem.
The problem with idle / orphaned application threads is that you cannot safely kill threads in a running JVM. And without the ability to do this you cannot stop leakages of (in this case) permgen space. This is essentially a Java platform problem. It can only be solved when JVMs support the Isolate mechanisms defined by JSR 121.
But assuming that your problem is related to doing hot deployment, it is not just the idle application threads that you have to worry about. If there is any reachable instance of any class that you are redeploying, that will cause the original old class loader and all classes it loaded to remain reachable.
... this is a big stopper to use NetBeans to develop web app.
The solution is to restart your web container every now and then. This will clean out the idle threads and other cruft and release the leaked permgen space. You can ameliorate it by running with a larger heap and more permgen, but you cannot make it go away, especially if some of the orphaned threads are not idle ...
Oh yea, and don't be a big baby! This ain't a show stopper, its just a small inconvenience :-)