I am having theses messages in my tomcat logs :
" org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
A web application registered the JBDC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered."
and
"org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: A web application appears to have started a thread named [pool-820-thread-1] but has failed to stop it. This is very likely to create a memory leak."
Actually I have a JDBC driver (.jar) in my java project that I always deploy on a tomcat server as a .war (meaning the driver is always in the war/libs directory).
After my searches, I found a good starting of answer here but unfortunately I cannot comment yet on stackoverflow to have more details on the accepted answer.
Here are my questions :
- Is the answer suggesting to remove completely the .jar from the war/libs directory ?
- If yes, where do I put it? Because I do not get how to get rid completely of the .jar and still be able to make my tests locally to the database.
Please advice on this.
Since Tomcat 6.0 , there has been a feature to detect classloader memory leaks. Read more here. The above messages by tomcat are purely for information purpose and tomcat already took sufficient measures to avoid classloader leak by de-registering the Driver.
To prevent it as you have rightly pointed out , you can either move the jar completely to tomcat's lib folder where it wont be affected by a application context reload. Or you can explicitly call a DriverManager.deregister(driver). (Read here)
To understand more about ClassLoader leaks (here)
To understand why its suggested to move it to tomcat lib from applications WEB_INF/lib you can read more here.
Edit in response to query in the comment
No its not recommended to have multiple jars as it can lead to classcastexception. Each class is identified by combination of class name and classloader. Look at this for a clearer understanding with an example.
The better approach according to me would be to write a servlet context listener and explicitly de-register your driver at context destroyed. This way you can keep you jdbc driver in your web-inf/lib and need not move it to tomcat/lib.
Even if you keep jars at multiple locations, according to the classloader hierarchy that tomcat follows which is different from what java delegation model (more info here) , tomcat will pick up the jar in your web-inf/lib first.
Related
I have WebApplication which is deployed in Tomcat 7.0.70. I simulated the following situation:
I created the heap dump.
Then I sent the Http request and in service's method I printed the current thread and its classLoader. And then I invoked Thread.currentThread.sleep(10000).
And at the same moment I clicked 'undeploy this application' in Tomcat's admin page.
I created new heap dump.
After some minutes I created new hep dump.
RESULTS
Thread dump
On the following screen you can see that after I clicked "redeploy", all threads (which were associated with this web application) were killed except the thread "http-apr-8081-exec-10". As I set Tomcat's attribute "renewThreadsWhenStoppingContext == true", so you can see that after some time this thread ("http-apr-8081-exec-10") was killed and new thread (http-apr-8081-exec-11) was created instead of it. So I didn't expect to have the old WCL after creation of heap dump 3, because there are not any old threads or objects.
Heapd dump 1
On the following two screens you can see that when the application was running there was only one WCL(its parameter "started" = true).
And the thread "http-apr-8081-exec-10" had the contextClassLoader = URLClassLoader ( because it was in the Tomcat's pool).
I'm speaking only about this thread because you will able to see that this thread will handle my future HTTP request.
Sending HTTP request
Now I send the HTTP request and in my code I get information about the current thread.You can see that my request is being handled by the thread "http-apr-8081-exec-10"
дек 23, 2016 9:28:16 AM c.c.c.f.s.r.ReportGenerationServiceImpl INFO: request has been handled in
thread = http-apr-8081-exec-10, its contextClassLoader = WebappClassLoader
context: /hdi
delegate: false
repositories:
/WEB-INF/classes/
----------> Parent Classloader: java.net.URLClassLoader#4162ca06
Then I click "Redeploy my web application" and I get the following message in console.
дек 23, 2016 9:28:27 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/hdi] appears to have started a thread named [http-apr-8081-exec-10] but has failed to stop it. This is very likely to create a memory leak.
Heapd dump 2
On the following screens you can see that there are two instances WebAppClassLoader. One of them( number #1) is old( its attribute "started" = false).
And the WCL #2 was created after redeploying application (its attribute "started" = true).
And the thread we review has contextClassLoader = "org.apache.catalina.loader.WebappClassLoader".
Why? I expected to see contextClassLoader = "java.net.URLClassLoader" (after all, when any thread finishes its work it is returned to the Tomcat's pool
and its attribute "contextClassLoader" is set to any base classloader).
Heapd dump 3
You can see that there isn't thread "http-apr-8081-exec-10", but there is thread "http-apr-8081-exec-11" and it has contextClassLoader = "WebappClassLoader"
(Why not URLClassLoader?).
In the end we have the following: there is thread "http-apr-8081-exec-11" which has the ref to the WebappClassLoader #1.
And obviosly when I make "Nearest GC Root" on the WCL #1 I will see the ref to the thread 11.
Questions.
How can I forcibly say to Tomcat to return old value contextClassLoader (URLClassLoader) after thread will finish its work?
How can I make sure Tomcat doesn't copy old value "contextClassLoader" during the thread renewal?
Maybe, do you know other way to resolve my problem?
Tomcat is usually not a good option on production environments. I was using Tomcat on a few production applications and I found that even if the heap size and other configurations are properly setup - and every time you reload your application, the memory consumption goes up and up. Until you don't restart the tomcat service, memory is not fully reclaimed. We did testing all such experiments like, clearing logs, redeploying all apps, regularly restarting tomcat once a month or a week during least busy hours. But at the end I have to say that we have shifted our production environments to Glassfish and WebSphere.
I hope you would already have gone through these pages:
Memory leak in a Java web application
Tomcat Fix Memory Leak?
https://developers.redhat.com/blog/2014/08/14/find-fix-memory-leaks-java-application/
http://www.tomcatexpert.com/blog/2010/04/06/tomcats-new-memory-leak-prevention-and-detection
If your web applications are not tightly coupled with Tomcat then you can think of using another web container. Now we use the Glassfish even on development machines and production and the day we make this decision, we saved a lot of our time. Though Glassfish and other such server take more time while they start as they are not as lightweight as the Tomcat is but after life is bit more easy.
From my experience with this problem, what was preventing tomcat to properly GC older class loaders was some ThreadLocals a couple of frameworks I was using were creating (and not properly handling).
Something similar to what is explained here: ThreadLocal & Memory Leak
I tried to properly finalize this ThreadLocals and my leak reduced A LOT. It was still leaking, but I could handle 10 times more redeploys than before.
I would definitely check your memory dumps to objects that could be connected somehow to ThreadLocals (they are very common, specially if you use something to control transactions or anything that is thread-isolated).
I hope it helps!
Memory leak in tomcat's redeploing is very old problem.
The only real way to solve it is restart tomcat instead of redeploy application. If you have several apps you need to run several tomcat's services on different ports and join it with nginx.
We have hundreds of Tomcat instances running in several environments (also production) and the only reasonable solution we've found to this issue is to stop and restart every Tomcat at a set time daily (in the nighttime).
We've tried many tricks, but this is the lasting solution for our uptime requirements.
Tomcat is usually not a good option on production environments. I was using Tomcat on a few production applications and I found that even if the heap size and other configurations are properly setup - and every time you reload your application, the memory consumption goes up and up. Until you don't restart the tomcat service, memory is not fully reclaimed. We did testing all such experiments like, clearing logs, redeploying all apps, regularly restarting tomcat once a month or a week during least busy hours. But at the end I have to say that we have shifted our production environments to Glassfish and WebSphere.
Check for ThreadLocal uses that prevent your ClassLoader to be garbage collected. Either remove references to your classes in ThreadLocal values or use https://github.com/codesinthedark/ImprovedThreadLocal instead of ThreadLocal
I have scoured the internet for over a day and still cannot seem to find a working solution. I am using MySQL Connector/J 5.1.35 and Tomcat JDBC Connection Pool with Tomcat 6.0.32 and I am not able to start/stop/reload an existing Web Application without the tomcat jdbc cleaner pool thread not being able to be stopped and causing memory leaks. I have tried everything that I could find on the net about it and I still have the same issues. If I stop the container, it wouldn't matter since once the container stops, the threads are dead. BUT, since I am not stopping the container, just merely stopping the application, BAM! - Memory Leak.
This is beyond frustrating and you would think that Apache would have a detailed solution for this, but they don't.
Anyone know how to resolve this once and for all?
Oh, and yes - I de-register the drivers and run AbandonedConnectionCleanupThread.shutdown() in a ServletContextListener and have tried both to deploy the drivers in $CATALINA_HOME/lib and WEB-INF/lib, but still have the same problem.
Update:
It looks like the connection pool may NOT be the issue. If I use DBCP, I do not get errors about memory leaks, but if I run "Find Leaks" on the manager page, Tomcat definitely discovers a leak in the application - just from starting it, running an operation that reads from the database, then stopping it. If I just start/stop it, then no issue reported. I have made sure that all connections/statements/resultSets are properly closed. If I switch back to Tomcat JDBC, I still have the same issue, it just gets reported (at least as far as I can tell) during stop/restart of the application as [Pool-Cleaner] Thread not being able to be stopped, and the Manager finding leaks when clicking the "Find Leaks" button.
Your JDBC drivers should be in $CATALINA_HOME/lib, and your database connection pools should be configured in the server.xml file using <Resource> tags. You context.xml file for the webapp should then link to the global data source with a <ResourceLink> tag.
This way connections are global and shared by all webapps connecting to the same database. Connections will not be released when a webapp is stopped or restarted, because they don't belong to the webapp, they belong to Tomcat.
No database connection memory leak, assuming all your webapp servlets/handlers clean up their resources correctly in finally blocks.
FYI: Deregistring a driver doesn't release any resources, such as open Connections, Statements, and ResultSets. It just prevents new connections from being created.
Ok, discovered that this is NOT AT ALL related to JDBC (face-palm). Turns out this is an RMI issue, so I am now going to re-post this question as a new question.
Thanks everyone for your time.
Let me give a little background to give context to this question. Over the course of time, the application I have been assigned to work on lost the ability to be built and deployed as a full application. By that I mean the previous developer compiled the code in his local IDE, and dropped single class files, as opposed to building out proper JARs and WARs to be deployed out to tomcat. So I have been tasked to clean up the project so it is a standard deployable app again. So to sum up the important part, the application exists in a working format on a windows tomcat environment, that hasn't had a clean deploy in a long time, my goal is to make the app buildable and deployable via a jenkins CI server to a tomcat instance running on a Linux server. Now on to the problem. I get the following toplink exception in one application module.
Local Exception Stack:
Exception [TOPLINK-6007] (OracleAS TopLink - 10g (9.0.4) (Build 031126)): oracle.toplink.exceptions.QueryException
Exception Description: Missing descriptor for [class edu.cornell.finsys.datamodel.AccountDTO].
Query: ReadObjectQuery(edu.cornell.finsys.datamodel.AccountDTO)
I have verified that the toplink mapping file has been loaded by tomcat, and the AccountDTO is mapped in the file. To double check that, I have moved the mapping file out, and I get a completly different error at load time. I know the file mapping is correct, as it worked correctly on the old server. I don't know what else could be causing the toplink exception.
For more information, I am on tomcat version 6.0.37, Java version 1.6.0_45, toplink version 9.0.4.031126
Any ideas?
This may be it; right from Metalink:
Cause:
This is an issue with loading the sessions.xml file, once when the application was first deployed using that application's classloader and then not reloading the session when the application was redeployed and a new XMLContext was created. The caused us to attempt to use a session that had been loaded with a different classloader than the current application.
Possible solution:
The issue occurs when TopLink is part of the main classpath (Eg: Applib directory rather than stored in the ear file). When this happens, the static SessionManager has trouble finding descriptors the second and subsequent times an application is deployed. Basically the descriptors are indexed by class. When the application is redeployed, the classes used for the indexing are unloaded and new versions of those classes appear. This causes "Descriptor not found" exceptions.
Potential Workarounds
Put the toplink.jar file in the ear.
Write some code to manually clean up when the toplink application is torn down.
For certain architectures they may be able to use the version sessionManager.getSession() that allows the session to be refreshed.
Workaround 3 is best and very feasible. The session has to be lazily
obtained from the SessionManager and held in an application code
variable. When the application is restarted its handle on the TopLink
session will be null. The session can then be obtained from the
SesssionManager using the getSession(XMLLoader xmlLoader, String
sessionName, ClassLoader objectClassLoader, boolean
shouldLoginSession, boolean shouldRefreshSession) API with
shouldRefreshSession set to true. This will ensure that the old
session (if any) is logged out and removed from the SessionManager
before a new one is created.
Did any of the workarounds help?
Is there a way to find out the java classes loaded in the server stack and replace the same with the latest version of the same without restarting the web or application server?
On Tomcat, there is an attribute called reloadable which support automatic reloading of changed classes/libraries. From Tomcat site:
reloadable
"Set to true if you want Catalina to monitor classes in /WEB-INF/classes/ and /WEB-INF/lib for changes, and automatically reload the web application if a change is detected. This feature is very useful during application development, but it requires significant runtime overhead and is not recommended for use on deployed production applications. That's why the default setting for this attribute is false. You can use the Manager web application, however, to trigger reloads of deployed applications on demand."
Sample usages is (Add following line in server.xml file):
<Context path="/webdev" docBase="/webdev" reloadable="true"></Context>
for more info, please refer http://tomcat.apache.org/tomcat-6.0-doc/config/context.html
I've made a web application using Java, Struts and running over Apache Server and Tomcat. It would be very useful to be able to restart the application from the web. I don't want to restart Tomcat, only this webapp, the same way as Tomcat Manager does it.
Do you know how can I do it? If not, any way to simulate that behaviour (reload config.properties, make Hibernate init process, etc)?
Thank you a lot.
I took a quick look at the source code for the Tomcat Manager. It looks like there's a class that's part of the Tomcat source called "Container Servlet". From the javadocs:
A ContainerServlet is a servlet
that has access to Catalina internal
functionality, and is loaded from the
Catalina class loader instead of the
web application class loader.
A ContainerServlet automatically gets passed a wrapper that can be used to get the Context and Deployer -- and the Deployer has helpful methods such as start(String contextPath) and stop(String contextPath) that will do what you want.
So, what I think you would need to do is write your own servlet that inherits from ContainerServlet, and configure Tomcat to load your servlet using the Catalina class loader (look at how the Manager is configured to see how). Note that this is probably not going to be an option for you in a hosted environment.
Then your servlet could have, say, a button you press to reload the application. I recommend putting password-protection of some kind in front of that. :)
Just hit the URLs
http://<username>:<password>#<hostname>:<port>/manager/stop?path=/<context path>
to stop and
http://<username>:<password>#<hostname>:<port>/manager/start?path=/<context path>
to start. That simulates you using the manager application.
Tomcat Manager offers an http interface to start/stop an application and other tasks. There are Ant tasks that you can use to easily access these operations.