I have scoured the internet for over a day and still cannot seem to find a working solution. I am using MySQL Connector/J 5.1.35 and Tomcat JDBC Connection Pool with Tomcat 6.0.32 and I am not able to start/stop/reload an existing Web Application without the tomcat jdbc cleaner pool thread not being able to be stopped and causing memory leaks. I have tried everything that I could find on the net about it and I still have the same issues. If I stop the container, it wouldn't matter since once the container stops, the threads are dead. BUT, since I am not stopping the container, just merely stopping the application, BAM! - Memory Leak.
This is beyond frustrating and you would think that Apache would have a detailed solution for this, but they don't.
Anyone know how to resolve this once and for all?
Oh, and yes - I de-register the drivers and run AbandonedConnectionCleanupThread.shutdown() in a ServletContextListener and have tried both to deploy the drivers in $CATALINA_HOME/lib and WEB-INF/lib, but still have the same problem.
Update:
It looks like the connection pool may NOT be the issue. If I use DBCP, I do not get errors about memory leaks, but if I run "Find Leaks" on the manager page, Tomcat definitely discovers a leak in the application - just from starting it, running an operation that reads from the database, then stopping it. If I just start/stop it, then no issue reported. I have made sure that all connections/statements/resultSets are properly closed. If I switch back to Tomcat JDBC, I still have the same issue, it just gets reported (at least as far as I can tell) during stop/restart of the application as [Pool-Cleaner] Thread not being able to be stopped, and the Manager finding leaks when clicking the "Find Leaks" button.
Your JDBC drivers should be in $CATALINA_HOME/lib, and your database connection pools should be configured in the server.xml file using <Resource> tags. You context.xml file for the webapp should then link to the global data source with a <ResourceLink> tag.
This way connections are global and shared by all webapps connecting to the same database. Connections will not be released when a webapp is stopped or restarted, because they don't belong to the webapp, they belong to Tomcat.
No database connection memory leak, assuming all your webapp servlets/handlers clean up their resources correctly in finally blocks.
FYI: Deregistring a driver doesn't release any resources, such as open Connections, Statements, and ResultSets. It just prevents new connections from being created.
Ok, discovered that this is NOT AT ALL related to JDBC (face-palm). Turns out this is an RMI issue, so I am now going to re-post this question as a new question.
Thanks everyone for your time.
Related
I've a Spring Boot Java application in production that uses ActiveJDBC to access a MariaDB database.
If at launch the application boots before the db server, of if the db server crashes and restarts, the apps doesn't re-estabilish the connection with the db.
ActiveJDBC is on version 1.4.13 and if possible I'd prefer not upgrading it, to avoid possible breakages. The db parameters are configured using the database.properties file and typically the usage pattern is:
try {
Base.open();
...
} finally {
Base.close();
}
Is there a way to circumvent this problem, without monitoring and relaunching the application? Maybe using connection pools? If this is the case, are there any docs or examples?
If you are using a direct JDBC connection in your database.properties file, a new connection will be open every time you execute Base.open(). This means that any old connection that is broken is not relevant anymore. If you use a JDNI pool such as:
production.jndi=java:comp/env/jdbc/acme
then you want to configure your containers' pool to ensure that every connection served from the pool is valid just before the pool serves the connection to your app. It is up to the implementation and documentation of your container/pool how to do that.
In any case, I do not think you are going to have issues.
I have WebApplication which is deployed in Tomcat 7.0.70. I simulated the following situation:
I created the heap dump.
Then I sent the Http request and in service's method I printed the current thread and its classLoader. And then I invoked Thread.currentThread.sleep(10000).
And at the same moment I clicked 'undeploy this application' in Tomcat's admin page.
I created new heap dump.
After some minutes I created new hep dump.
RESULTS
Thread dump
On the following screen you can see that after I clicked "redeploy", all threads (which were associated with this web application) were killed except the thread "http-apr-8081-exec-10". As I set Tomcat's attribute "renewThreadsWhenStoppingContext == true", so you can see that after some time this thread ("http-apr-8081-exec-10") was killed and new thread (http-apr-8081-exec-11) was created instead of it. So I didn't expect to have the old WCL after creation of heap dump 3, because there are not any old threads or objects.
Heapd dump 1
On the following two screens you can see that when the application was running there was only one WCL(its parameter "started" = true).
And the thread "http-apr-8081-exec-10" had the contextClassLoader = URLClassLoader ( because it was in the Tomcat's pool).
I'm speaking only about this thread because you will able to see that this thread will handle my future HTTP request.
Sending HTTP request
Now I send the HTTP request and in my code I get information about the current thread.You can see that my request is being handled by the thread "http-apr-8081-exec-10"
дек 23, 2016 9:28:16 AM c.c.c.f.s.r.ReportGenerationServiceImpl INFO: request has been handled in
thread = http-apr-8081-exec-10, its contextClassLoader = WebappClassLoader
context: /hdi
delegate: false
repositories:
/WEB-INF/classes/
----------> Parent Classloader: java.net.URLClassLoader#4162ca06
Then I click "Redeploy my web application" and I get the following message in console.
дек 23, 2016 9:28:27 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/hdi] appears to have started a thread named [http-apr-8081-exec-10] but has failed to stop it. This is very likely to create a memory leak.
Heapd dump 2
On the following screens you can see that there are two instances WebAppClassLoader. One of them( number #1) is old( its attribute "started" = false).
And the WCL #2 was created after redeploying application (its attribute "started" = true).
And the thread we review has contextClassLoader = "org.apache.catalina.loader.WebappClassLoader".
Why? I expected to see contextClassLoader = "java.net.URLClassLoader" (after all, when any thread finishes its work it is returned to the Tomcat's pool
and its attribute "contextClassLoader" is set to any base classloader).
Heapd dump 3
You can see that there isn't thread "http-apr-8081-exec-10", but there is thread "http-apr-8081-exec-11" and it has contextClassLoader = "WebappClassLoader"
(Why not URLClassLoader?).
In the end we have the following: there is thread "http-apr-8081-exec-11" which has the ref to the WebappClassLoader #1.
And obviosly when I make "Nearest GC Root" on the WCL #1 I will see the ref to the thread 11.
Questions.
How can I forcibly say to Tomcat to return old value contextClassLoader (URLClassLoader) after thread will finish its work?
How can I make sure Tomcat doesn't copy old value "contextClassLoader" during the thread renewal?
Maybe, do you know other way to resolve my problem?
Tomcat is usually not a good option on production environments. I was using Tomcat on a few production applications and I found that even if the heap size and other configurations are properly setup - and every time you reload your application, the memory consumption goes up and up. Until you don't restart the tomcat service, memory is not fully reclaimed. We did testing all such experiments like, clearing logs, redeploying all apps, regularly restarting tomcat once a month or a week during least busy hours. But at the end I have to say that we have shifted our production environments to Glassfish and WebSphere.
I hope you would already have gone through these pages:
Memory leak in a Java web application
Tomcat Fix Memory Leak?
https://developers.redhat.com/blog/2014/08/14/find-fix-memory-leaks-java-application/
http://www.tomcatexpert.com/blog/2010/04/06/tomcats-new-memory-leak-prevention-and-detection
If your web applications are not tightly coupled with Tomcat then you can think of using another web container. Now we use the Glassfish even on development machines and production and the day we make this decision, we saved a lot of our time. Though Glassfish and other such server take more time while they start as they are not as lightweight as the Tomcat is but after life is bit more easy.
From my experience with this problem, what was preventing tomcat to properly GC older class loaders was some ThreadLocals a couple of frameworks I was using were creating (and not properly handling).
Something similar to what is explained here: ThreadLocal & Memory Leak
I tried to properly finalize this ThreadLocals and my leak reduced A LOT. It was still leaking, but I could handle 10 times more redeploys than before.
I would definitely check your memory dumps to objects that could be connected somehow to ThreadLocals (they are very common, specially if you use something to control transactions or anything that is thread-isolated).
I hope it helps!
Memory leak in tomcat's redeploing is very old problem.
The only real way to solve it is restart tomcat instead of redeploy application. If you have several apps you need to run several tomcat's services on different ports and join it with nginx.
We have hundreds of Tomcat instances running in several environments (also production) and the only reasonable solution we've found to this issue is to stop and restart every Tomcat at a set time daily (in the nighttime).
We've tried many tricks, but this is the lasting solution for our uptime requirements.
Tomcat is usually not a good option on production environments. I was using Tomcat on a few production applications and I found that even if the heap size and other configurations are properly setup - and every time you reload your application, the memory consumption goes up and up. Until you don't restart the tomcat service, memory is not fully reclaimed. We did testing all such experiments like, clearing logs, redeploying all apps, regularly restarting tomcat once a month or a week during least busy hours. But at the end I have to say that we have shifted our production environments to Glassfish and WebSphere.
Check for ThreadLocal uses that prevent your ClassLoader to be garbage collected. Either remove references to your classes in ThreadLocal values or use https://github.com/codesinthedark/ImprovedThreadLocal instead of ThreadLocal
Similar to my question here Spring Tomcat C3P0PooledConnectionPoolManager creates a memory leak I wish to understand why Spring or C3P0 itself not cleaning up the threads it creates on shutdown?
I get the following logs in Tomcat
SEVERE: The web application [/ul-xtrain] appears to have started a thread named [C3P0PooledConnectionPoolManager[identityToken->2wpukr9b7ohfj11xtbfft|725548e3]-AdminTaskTimer] but has failed to stop it. This is very likely to create a memory leak.
I am not interested in solution as I already have working solution based on these threads
Hibernate4 + c3p0 + Derby - Memory Leak on Tomcat stop or web application reload
Tomcat Guice/JDBC Memory Leak
To prevent a memory leak, the JDBC Driver has been forcibly unregistered
I just wish to understand why Spring is not closing it. Thank you
I am having theses messages in my tomcat logs :
" org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
A web application registered the JBDC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered."
and
"org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: A web application appears to have started a thread named [pool-820-thread-1] but has failed to stop it. This is very likely to create a memory leak."
Actually I have a JDBC driver (.jar) in my java project that I always deploy on a tomcat server as a .war (meaning the driver is always in the war/libs directory).
After my searches, I found a good starting of answer here but unfortunately I cannot comment yet on stackoverflow to have more details on the accepted answer.
Here are my questions :
- Is the answer suggesting to remove completely the .jar from the war/libs directory ?
- If yes, where do I put it? Because I do not get how to get rid completely of the .jar and still be able to make my tests locally to the database.
Please advice on this.
Since Tomcat 6.0 , there has been a feature to detect classloader memory leaks. Read more here. The above messages by tomcat are purely for information purpose and tomcat already took sufficient measures to avoid classloader leak by de-registering the Driver.
To prevent it as you have rightly pointed out , you can either move the jar completely to tomcat's lib folder where it wont be affected by a application context reload. Or you can explicitly call a DriverManager.deregister(driver). (Read here)
To understand more about ClassLoader leaks (here)
To understand why its suggested to move it to tomcat lib from applications WEB_INF/lib you can read more here.
Edit in response to query in the comment
No its not recommended to have multiple jars as it can lead to classcastexception. Each class is identified by combination of class name and classloader. Look at this for a clearer understanding with an example.
The better approach according to me would be to write a servlet context listener and explicitly de-register your driver at context destroyed. This way you can keep you jdbc driver in your web-inf/lib and need not move it to tomcat/lib.
Even if you keep jars at multiple locations, according to the classloader hierarchy that tomcat follows which is different from what java delegation model (more info here) , tomcat will pick up the jar in your web-inf/lib first.
My application is trowing the following exception:
java.sql.SQLException: Connection is not associated with a managed connection.org.jboss.resource.adapter.jdbc.jdk6.WrappedConnectionJDK6#4fe91321
This is happening in production I'm not able to get this problem in development and for that I'm not able to solve it.
The root of the exception is code line dbConn.prepareStatement(sql);
From what I was able to find in the web, the cause for this can be:
Over jdbc connection, a jboss connection is wrapped but the wrapper is empty. The original jdbc connection inside is no longer there.
JDBC Connection is already closed and trying to work with close connection is the reason why I'm getting this exception.
The transaction manger detects transaction that is taking to long then the transaction timeout.
So if someone can point me what is the problem because I'm not able to get this problem in mine development environment.
Also what logs can I add that will help me detect the problem in production - I'm using Java, JBoss, Postgre.
I'have enable connection close monitoringa, and also to add is that the issue is not consistent
I just resolved the issue.
It's my jndi driver not compatible with the latest java frameworks.
I'm using Oracle db and ojdbc6, and i download a latest ojdbc6 from maven repository and replace the old same name jar file. It works like bird now.
If you are using an EJB, it's possible that your Stateless Session Bean Pool is not big enough. The connection can't find an associated thread.
Maybe you have an extended Transaction for a not Stateful Session Bean, wich is maybe not a good idea.