I run some tomcat application, use jndi connection pool.
In some time connection pool stops to give connections and application hangs.
Seems because some code receives connection and doesn't return it back to the pool.
How can I monitor - which code does it ?
More common - I want to see what all connections do at the moment.
I cannot change application. But I can adjust Tomcat, maybe add some interceptors.
Most connection pool implementations can be configured to detect connections that are not returned to the pool. E.g. for Tomcat's JDBC connection pool there are various configurations options for "abandoned connections" (connections for which the lease expired). If you search for "Abandoned" on this web-page, you'll find the options:
removeAbandoned
removeAbandonedTimeout
logAbandoned
suspectTimeout
As mentioned on the web-page, these settings will add a little overhead but at least your application will not hang. When testing your application, set a low value for removeAbandonedTimeout and a low value for maxActive so that you can catch unreturned connections early.
I never use the connection pool API itself, I always wrap it in a helper.
That way, I can do this in the helper:
private Exception created = (0 == 1) ? new Exception() : null;
When I run into problems like yours, I just change one character (0 -> 1) to have a stack trace of who created this instance in my debugger.
Related
I am using Jedis in a tomcat web app to connect to an Elascticache Redis node. The app is used by hundreds of users in day time. I am not sure of this is normal or not, but whenever I check the current connections count with cloudwatch metrics, I see the current connections increasing without falling down.
This is my Jedis pool configuration:
public static JedisPool getPool(){
if(pool == null){
JedisPoolConfig config = new JedisPoolConfig();
config.setMinIdle(5);
config.setMaxIdle(35);
config.setMaxTotal(1500);
config.setMaxWaitMillis(3000);
config.setTestOnBorrow(true);
config.setTestWhileIdle(true);
pool = new JedisPool(config, PropertiesManager.getInstance().getRedisServer());
}
return pool;
}
and this is how I always use the pool connections to execute redis commands:
Jedis jedis = JedisUtil.getPool().getResource();
try{
//Redis commands
}
catch(JedisException e){
e.printStackTrace();
throw e;
}finally{
if (jedis != null) JedisUtil.getPool().returnResource(jedis);
}
With this configuration, the count is currently over 200. Am I missing something that is supposed to discard or kill unsused connections ? I set maxIdle to 35 and I expected that the count falls down to 35 when the traffic is very low but this never happened.
we had the same problem. After investigating a little bit more further we came across with this (from redis official documentation - http://redis.io/topics/clients) :
By default recent versions of Redis don't close the connection with the client if the client is idle for many seconds: the connection will remain open forever.
By default, aws provides a timeout value of 0. Therefore, any connection that has been initialised with your redis instance will be kept by redis even if the connection initialised by your client is down.
Create a new cache parameter policy in aws with a timeout different of 0 and then you should be good
In the cache parameter group you can edit timeout. It defaults to 0 which leaves idle connection in redis. If you set it to 100 it will remove connections idle for 100 seconds.
You can check the pool size using JMX. Activating the idle evictor thread is a good idea. You can do so by setting the timeBetweenEvictionRunsMillis parameter on the JedisPoolConfig.
If you don't use transactions (EXEC) or blocking operations (BLPOP, BRPOP), you could stick to one connection if connection count is a concern for you. The lettuce client is thread-safe with one connection
I'm having problems with the connection pool in Tomcat after a database outage.
There's this heartbeat service servlet that won't come back up after a database outage.
I've already tried the standard answer, and beyond, specifically:
Adding to the Resource section in server.xml
validationQuery="select 1 from dual"
testOnBorrow="true"
removeAbandoned="true"
removeAbandonedTimeout="120"
logAbandoned="true"
Trying to close the connection with a check to validity:
if (connection != null && !connection.isValid(10)) {
connection.close();
}
(Resulted in java.sql.SQLException: Connection is closed)
Trying to abort the connection (not sure whether done right)
if (connection != null && !connection.isValid(10)) {
connection.abort();
}
(Resulted in java.lang.NoSuchMethodError: java.sql.Connection.abort(Ljava/util/concurrent/Executor;)V)
Tries 2) and 3) show that indeed the connection is invalid, and it knows it. The question is - how to destroy it?
Tomcat version: 7.0.29
The pool will destroy the invalid connection for you thanks to "testOnBorrow=true" setting (note that the "Abandoned" settings only apply to connections which are not returned to the pool).
So this is how it should go:
servlet borrows connection, does queries and returns connection to the pool.
servlet borrows connection, database crashes, servlet receives all kinds of horrible SQL errors but still returns connection to the pool.
database has restarted
on next heartbeat, servlet borrows connection. Tomcat's pool will "test on borrow", find that the old connection is broken, removes (and closes) connection, tries another, finds that it is also broken, etc, and THEN the pool decides to create a new connection, test it OK and hands it over to the servlet.
servlet receives new valid connection, does queries and returns connection to the pool.
I'm not sure how Tomcat's pool behaves when a "test on borrow" indicates a connection is broken (1): it might not create new connections right away or throw an error when it can't get a valid connection from the pool. But I expect the pool to effectively flush itself and re-populate with new (valid) connection.
(1) That is, if the "test on borrow" is actually done which this post indicates is not the case ...
If the pool does not flush itself, you can try to do this programmatically once you find connections are invalid. I have not tried this before, hopefully you can get it to work. Following method is described here:
Reach into the JNDI context, pull-out the DataSource object, cross your fingers, cast it to org.apache.tomcat.jdbc.pool.DataSource, and call the purge() method.
Alterntively, use JMX and call the purge method via the MBean.
If you experience hanging threads after a database crash, you might have to resort to a work around described in this answer.
If we want to dispose an ill java.sql.connection from Tomcat jdbc connection pool,
we may do this explicitly in the program.
Unwrap it into an org.apache.tomcat.jdbc.pool.PooledConnection, setDiscarded(true)
and close the JDBC connection finally, the ConnectionPool will remove the underlying connection once it has been returned.
(ConnectionPool.returnConnection(....))
PooledConnection pconn = conn.unwrap(PooledConnection.class);
pconn.setDiscarded(true);
conn.close();
I put Jsch into commons-pool (with spring pool support) with initial success
http://docs.spring.io/spring/docs/3.2.4.RELEASE/spring-framework-reference/htmlsingle/#aop-ts-pool
However:
Should we pool the channels within the Session instead of pooling the sessions? Each Jsch session creates one thread. Pooling Jsch sessions will create x threads. Pooling channels, there will really be only one Jsch thread.
(commons-pool) what happens if the Jsch session went stale? How to regenerate the session in the context of the commons-pool or using spring pool support? How to detect whether it goes stale?
Thanks
Figured out my own question. I will share my project in the next day or two.
Pooling channels are much more effective. There is really no need to create multiple sessions (if the session connects to the same sftp endpoint).
I implemented a JSch connection pool (pooling channels) with spring pool and commons-pool. I will post to the github in the next day or two. The most important question is, what if the connection went stale.
I found out that based on my implementation of 1 Session - multiple channels, and if the connection went stale, the pooled objects (in this case, the channel) will be stale. The pooled object should be invalidated and deleted from the pool. When the connection comes back up, and when new application thread "borrows" from the pool, new pool objects will be created.
To validate my observation, my not-so-automated test:
a) Create a set (say 10) of app threads checking out channel resource from the pool.
b) Have the thread to sleep 20 seconds
c) Create another set of app threads checking out channel resources from the pool.
At a), set breakpoint when i==7, break the connection by "iptable drop (linux) or pfctl -e; pfctl -f /etc/pf.conf (mac, google how to do!)". This first set of app threads will get exception because the channel is broken.
At b), restart the connection
At c), the 2nd set of app threads will be successfully completing the operation because the broken connection has been restored.
I'm new to c3op, and confused about the use of :
c3p0.idle_test_period
In this link : HowTo configure the C3P0 connection pool
idleTestPeriod : Must be set in hibernate.cfg.xml (or hibernate.properties), Hibernate default:
0, If this is a number greater than 0, c3p0 will test all idle, pooled but unchecked-out
connections, every this number of seconds.
What is the purpose of this kind of test (idel, pooled connections), and the relationship between c3p0.idle_test_period and c3p0.timeout?
The database server may close a connection on its side after a certain amount of time - causing some error in your application, because it'll attempt to send a query on a connection which is no longer available on the server side.
In order to avoid this you can let the pool periodically check a connection (Think of a ping) for it's validity. This is what idle_test_period is for.
timeout is the timespan after which the pool will remove a connection from the pool, because the connection wasn't checked out (used) for a while and the pool contains more connections than c3pO.min_size.
I think this setting is used in hibernate in order to validate pooled connection after every few seconds . Consider a scenario if on database side, password is changed. Hibernate already pooled connections on old password. So it is security breach having pool with wrong password.So when hibernate will validate after few seconds it . It will invalidate that pooled connection.
Does anybody know a way to reestablish/retry again hibernate connection. I mean for example: the remote DB is down and I start my application. Hibernate is not able to establish a connection. it fails. But the application is not closed. Is there a way to say hibernate to try one more time to establish a connecton?
Thanks in advance
You should really go for C3P0 connection pooling: http://www.mchange.com/projects/c3p0/index.html#hibernate-specific
There is a section in C3P0 documentation on that subject: http://www.mchange.com/projects/c3p0/index.html#configuring_recovery
First you have to properly configure c3p0, which in case of using hibernate must happen in c3p0.properties file.
In your c3p0.properties put these properties to retry to reconnect indefinitely every 3 seconds when database is down:
c3p0.acquireRetryAttempts = 0
c3p0.acquireRetryDelay = 3000
c3p0.breakAfterAcquireFailure = false
Also, to avoid broken connections lying in your pool indefinitely, use connection age management:
c3p0.maxConnectionAge = 6000
c3p0.maxIdleTime = 6000
c3p0.maxIdleTimeExcessConnections = 1800
c3p0.idleConnectionTestPeriod = 3600
These may be quite expensive, but helpful if above is not enough:
c3p0.testConnectionOnCheckout = true
c3p0.preferredTestQuery = SELECT 1;
You may also want to check for connection leaks which prevent recovery:
c3p0.debugUnreturnedConnectionStackTraces = true
And finally, make sure that C3P0 is hooked with hibernate correctly, enable debug logging for "com.mchange" package and see if C3P0 tells you anything about itself. It should state configuration properties which are loaded, so see if it's all there.
I hope this helps.
C3P0 is the internal connection-pool implementation for hibernate.
Add "hibernate.connection.provider_class = org.hibernate.connection.C3P0ConnectionProvider"
in hibernate properties file. Create a file c3p0.properties setting the parameters accordingly. This file & c3p0-x.jar must be in classpath.
c3p0.properties
c3p0.idleConnectionTestPeriod : If this is a number greater than 0, c3p0 will test all idle, pooled but unchecked-out connections, every this number of seconds.
c3p0.testConnectionOnCheckout : Use only if necessary. Expensive. If true, an operation will be performed at every connection checkout to verify that the connection is valid. Better choice: verify connections periodically using idleConnectionTestPeriod.
There are several other properties that can be configured in hibernate.properties & c3p0.properties.
May be, you try to call method .getCurrentSession() instead .openSession()?
If connection falls you must establish new one.
I hope this helps.