I am calling org.apache.commons.dbcp2.BasicDataSource#getConnection, then working with it, then calling Connection#close() when connection is not needed anymore.
Unfortunately, my pool grows infinitely and all connection are remain active.
How to put connection to idle, so pool could re-use it?
Related
I am creating an ETL application where I actually need a large number of concurrent connections running long slow queries. It is not uncommon to see the number of concurrent connections to be as large as 100 depending on the machine running the application.
Let's assume it takes about 2s to establish a connection with the database. If I don't use pooling and parallelize connection retrieval with 100 threads then all connection are still established in about 2s. However, while using HikariCP I've noticed that time to establish 100 connections at the start of the application when there is a spike in connection requests, it takes about 200s to establish all connections and often results in timeout.
This drives me to the conclusion that obtaining a new connection is a blocking call. Also it seems that hikariCP pool is lazy initialized and I assume that once it establishes all 100 connections it will try to keep the pool size at 100.
Is there a way to enable more concurrent behaviour of establishing connections in hikariCP? Could I at least force it to concurrently initialize (establish 100 connections) the pool?
One could say that the time to initially establish all connections is irelevant in the lifetime of the application, but I also want to have timeout set to 30seconds which will always result in timeout exception during initial spike demand.
i have a question about connection pool and oracle database. if i set the min size of the connection pool to 5, does that means even when the app is idle, oracle database will still keep 5 sessions active? thanks!
Yes, it should keep connections even if app is idle. Good idea is to check if connection on pool is still alive. Wrappers like HikariCP/Commons DBCP handle such cases.
//edit
Connection pool wont start 5 connections on application start but it will initialize new connection if it will be needed. And of course pool can be set to shutdown connection after finishing operation on it.
Connection pool behavior doc link
Until now, whenever I query the database I open a fresh connection to the database. How do I implement the property that once I open the connection I can reuse that?
With this done, please tell me if I could leak the resources.
Basically you need JDBC connection pool, typically implementing DataSource interface. Have a look at dbcp and c3p0. Chances are you container/server already provides an implementation of connection pooling.
When you use a connection pool every time you open a connection you are actually taking one from the pool (or opening if pool is empty). When closing the connection, it is actually returned to the pool. The leak can only occur if you forget the latter. (or forget closing ResultSet, Statement...)
You can (and should) reuse db connections. Connection pooling is one of the techniques for this. A thorough tutorial on connection pooling can be read here : http://java.sun.com/developer/onlineTraining/Programming/JDCBook/conpool.html
My application uses Tomcat JDBC connection pool, with MySQL DB.
Seems like a process that run during the night (anti virus scan?) cause the memory and CPU on the machine to increase, and as a result connections from the pool stuck on active until the connection pool can't response to any connection request.
I'm getting errors like:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after statement closed.
Timeout: Pool empty. Unable to fetch a connection in 10 seconds, none available[size:100; busy:97; idle:0; lastwait:10000]. (That's weird, where are the remaining 3?)
Looking at a chart I'm generating describing the active connection state, it is flat until at some point it start increasing until it reach the maximum and stays there.
My connection pool is configure to remove unclosed connections (setRemoveAbandoned = true).
Do you have any idea how can I solve this issue?
I think this is because your application not closing connections after use. Please check your code and make sure all connections are closing after use.
Does anyone have any information comparing performance characteristics of different ConnectionPool implementations?
Background: I have an application that runs db updates in background threads to a mysql instance on the same box. Using the Datasource com.mchange.v2.c3p0.ComboPooledDataSource would give us occasional SocketExceptions:
com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:
** BEGIN NESTED EXCEPTION **
java.net.SocketException
MESSAGE: Broken pipe
STACKTRACE:
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
Increasing the mysql connection timeout increased the frequency of these errors.
These errors have disappeared on switching to a different connection pool (com.mysql.jdbc.jdbc2.optional.MysqlConnectionPoolDataSource); however the performance may be worse and the memory profile is noticeably so (we get fewer, and much larger, GC's than the c3p0 pool).
Whatever connection pool you use, you need to assume that the connection could be randomly closed at any moment and make your application deal with it.
In the case with a long-standing DB connection on a "trusted" network, what often happens is that the OS applies a time limit to how long connections can be open, or periodically runs some "connection cleanup" code. But the cause doesn't matter too much -- it's just part of networking life that you should assume the connection can be "pulled from under your feet", and deal with this scenario accordingly.
So given that, I really can't see the point of a connection pool framework that doesn't allow you to handle this case programmatically.
(Incidentally, this is another of my cases where I'm glad I just write my own connection pool code; no black boxes mysteriously eating memory, and no having to fish around to find the "magic parameter"...)
You may want to have a look at some benchmark numbers up at http://jolbox.com - the site hosting BoneCP, a connection pool that is faster than both C3P0 and DBCP.
I had this error pop up with mysql & c3p0 as well - I tried various things and eventually made it go away. I can't remember, but what might have solved it was the autoReconnect flag a la
url="jdbc:mysql://localhost:3306/database?autoReconnect=true"
Have you tried Apache DBCP? I don't know about c3po but DBCP can handle idle connections in different ways:
It can remove idle connections from the pool
It can run a query on idle connections after a certain period of inactivity
It can also test if a connection is valid just before giving it to the application, by running a query on it; if it gets an exception, it discards that connection and tries with another one (or creates a new one if it can). Way more robust.
Broken pipe
That roughly means that the other side has aborted/timedout/closed the connection. Aren't you keeping connections that long open? Ensure that your code is properly closing all JDBC resources (Connection, Statement and ResultSet) in the finally block.
Increasing the mysql connection timeout increased the frequency of these errors.
Take care that this timeout doesn't exceed the DB's own timeout setting.