Java ComboPooledDataSource exceeds pool size and does not reuse - java

I have the following code that gets called by an other application (That I can not change) to read form a Database.
The method is called in a loop very often and DOSes the DB.
At The DB I can see that there are many connecetions opend ... increasing to some hundred ... and than the DB crashed due to the load.
// Called in a loop
private <T> T execute(String query, PostProcessor<T> postProc, PreProcessor preProcs) throws OperationFailedException {
try (Connection conn
= Objects.requireNonNull(dataSourceRef.get(), "No Connection").getConnection();
PreparedStatement smt = conn.prepareStatement(query)) {
preProc.process(smt);
return postProc.process(smt.executeQuery());
} catch (SQLException e) {
throw new OperationFailedException(e.getMessage(), e);
}
}
the date source gets initialised before ...
// class variable
// AtomicReference<PooledDataSource> dataSourceRef = new AtomicReference<PooledDataSource>();
// Init method
ComboPooledDataSource cpds = new ComboPooledDataSource();
cpds.setDriverClass(config.getConnectionDriver());
cpds.setJdbcUrl(config.getConnectionString());
cpds.setUser(config.getConnectionUser());
cpds.setPassword(config.getConnectionPassword());
// cpds.setMaxPoolSize(10);
dataSourceRef.getAndSet(cpds);
My question why is this happening.
I thought due to the pooling not for each query a new connections should be used.
As well by setting the max pool size this is not working.
As well I tried it with try-catch-finally construct and closing the stm and the conn after use.
(As I've read somewhere that finally might get called delayed under high load scenarios ... I thouht that might be the case)
But still why is the pool size exceeded?
How can I limit that and block the method untill a connection is avalible again before continuing?

During connection polling in c3p0 you have to consider some options. Those are given bellow application.property file:
db.driver: oracle.jdbc.driver.OracleDriver // for Oracle
db.username: YOUR_USER_NAME
db.password: YOUR_USER_PASSWORD
db.url: DATABASE_URL
minPoolSize:5 // number of minimum poolSize
maxPoolSize:100 // number of maximum poolSize
maxIdleTime:5 // In seconds. After that time it will realease the unused connection.
maxStatements:1000
maxStatementsPerConnection:100
maxIdleTimeExcessConnections:10000
Here, maxIdleTime is the main points. It defines how many seconds this will release unused connection. It is in second.
Another is minPoolSize . It defines how many connection it will hold during idle mode.
Another is maxPoolSize . It defines how many maximum connection it will hold during loaded mode.
Now, how you configure ComboPooledDataSource? Here is the code:
#Bean
public ComboPooledDataSource dataSource(){
ComboPooledDataSource dataSource = new ComboPooledDataSource();
try {
dataSource.setDriverClass(env.getProperty("db.driver"));
dataSource.setJdbcUrl(env.getProperty("db.url"));
dataSource.setUser(env.getProperty("db.username"));
dataSource.setPassword(env.getProperty("db.password"));
dataSource.setMinPoolSize(Integer.parseInt(env.getProperty("minPoolSize")));
dataSource.setMaxPoolSize(Integer.parseInt(env.getProperty("maxPoolSize")));
dataSource.setMaxIdleTime(Integer.parseInt(env.getProperty("maxIdleTime")));
dataSource.setMaxStatements(Integer.parseInt(env.getProperty("maxStatements")));
dataSource.setMaxStatementsPerConnection(Integer.parseInt(env.getProperty("maxStatementsPerConnection")));
dataSource.setMaxIdleTimeExcessConnections(10000);
} catch (PropertyVetoException e) {
e.printStackTrace();
}
return dataSource;
}
For details implementation, please check this thread . Here i added practical implementation
Edit (Getting connection)
You can get connection using bellow way:
Session session = entityManager.unwrap(Session.class);
session.doWork(connection -> doSomeStuffWith(connection));
How you get EntityManager?
#PersistenceContext
private EntityManager entityManager;
Hope this will help you.
Thanks :)

Related

create another connection in transaction mysql

I want know if it is possibile create two connection different connections in one transaction, So suppose to have this code( I do a create operation in table "employe" and I create a log in table "log"):
try {
InitialContext initialContext = new InitialContext();
DataSource datasource = (DataSource) initialContext.lookup("java:/comp/env/jdbc/postgres");
Connection connection_db= datasource.getConnection();
PreparateStatement p1 //Preparate statement to put the employe parameter
connessione_db.setAutoCommit(false);
connessione_db.setTransactionIsolation(Connection.TRANSACTION_SERIALIZABLE);
p1.execute();
//This create another connect
LOGStatic.createLog("CREATE EMPLOYE");
connessione_db.commit();
connessione_db.setAutoCommit(true);
}catch(....){
connection_db.rollback();
}
This is the method LOGStatic.createLog("CREATE EMPLOYE");
public static void creaLogException(String messaggio) {
Connection connection_db= null;
InitialContext initialContext;
try {
initialContext = new InitialContext();
DataSource datasource = (DataSource) initialContext.lookup("java:/comp/env/jdbc/postgres");
connection_db= datasource.getConnection();
// the code continues with the save operation
} catch (NamingException n) {
}
// actual jndi name is "jdbc/postgres"
catch (SQLException s) {
}
My question if is it possible this behavior and there aren't problem with commit and rollback operatio or it is not possible to have another connection?
Anyone can help
At least in JDBC, the transaction is tightly coupled with the Connection.
In a Java EE server, if you are writing session beans, then the transaction will be managed my the server. So, in that case, you could call several methods, and the transaction will follow the method calls.
In JDBC the simple solution is to not close the connection, but to pass it around to different methods as an input parameter.
But, be very careful, not closing connections is almost a sure shot way of getting to OutOfMemoryError.
(Instead of passing around the connection as input parameters to various methods, you could use ThreadLocal. But that is another source of memory leak, if you don't clean up ThreadLocal variables.)
For more information on how transaction-lifecycle and Connection are tied in JDBC, refer this: How to start a transaction in JDBC?
Note: Even in JavaEE nested transaction is not possible as nested transactions is not supported in JTA.
Passing the connection around:
public static void createEmployee(){
InitialContext initialContext = new InitialContext();
DataSource datasource = (DataSource) initialContext.lookup("java:/comp/env/jdbc/postgres");
Connection connection_db= datasource.getConnection();
try {
connessione_db.setAutoCommit(false);
connessione_db.setTransactionIsolation(Connection.TRANSACTION_SERIALIZABLE);
PreparateStatement p1 //Preparate statement to put the employe parameter
p1.execute();
//This create another connect
createLog("CREATE EMPLOYE", connessione_db);
connessione_db.commit();
//connessione_db.setAutoCommit(true); //No need
}catch(....){
try{ connection_db.rollback(); }catch(Exception e){ /*You can also check some flags to avoid exception*/ }
}finally{
try{ connection_db.close(); }catch(Exception e){ /*Safe to ignore*/ }
}
}
public static void createLog(String messaggio, Connection connection_db) {
try {
// the code continues with the save operation
} catch (SQLException s) {
}
}

Websphere 7.0.0.17 error with Pool Connection

Im having a problem with an application, I'm using EclipseLink 2.4.1 with oracle 11g, I'm getting a J2CA0045E error, because the pooling does not set free any connections pool from the jndi,
Have you have this problem?, apparently the connection does not get released and it reach the limit I've configured.
Any ideas, this is my Connection pool
Connection timeout: 180
Maximum connections: 10
Minimum connections: 1
Reap time: 60
Unused timeout: 180
Aged timeout: 120
Purge policy: Entire Pool
EDIT
I have a web service in the same service, and the behavior is the same, using same jndi, the connection increase and does not release any connection
initCtx = new InitialContext();
Context envCtx = (Context) initCtx.lookup("java:comp/env");
DataSource ds = (DataSource) envCtx.lookup("jdbc/abc");
conn = ds.getConnection();
Based on the code provided, it looks like you are storing the Connection reference at a class scope. You should avoid storing references to Connection objects that could span multiple requests.
Instead, store a reference to the DataSource.
To illustrate. This approach is preferred:
public class GoodService {
DataSource ds;
public GoodService() throws NamingException {
// Only the DataSource reference is stored, this is allowed
ds = new InitialContext().lookup("java:comp/env/jdbc/abc");
}
public void doSomething() throws Exception {
// Get the connection in a request.
Connection conn = ds.getConnection();
try {
// do something...
} finally {
// Always close the connection in the finally block
// so it gets returned to the pool no matter what
conn.close();
}
}
}
The following approach is considered bad practice, and is likely to cause the J2CA0045E error you are seeing:
public class BadService {
Connection conn; // BAD: don't store a Connection across requests!
public BadService() throws NamingException, SQLException {
DataSource ds = new InitialContext().lookup("java:comp/env/jdbc/abc");
// BAD: don't leave connections open indefinitely like this
conn = ds.getConnection();
}
public void doSomething() throws Exception {
// do something using conn...
}
}

H2 connection pool

I want to make a connection pool for my h2 database. But I think my pool opens new connection every time I call getConnection(). I guess there should be a fixed amount of reusable connections, but if I run this code :
Connection conn = DataSource.getInstance().getConnection();
Statement stmt = conn.createStatement();
ResultSet rs;
rs = stmt.executeQuery("SELECT * FROM NODE_USERS;");
while (rs.next()) {
System.out.println(rs.getString("login"));
}
try {
// wait a bit
Thread.sleep(20000);
} catch (InterruptedException e) {
e.printStackTrace();
}
stmt.close();
rs.close();
conn.close();
DataSource:
public class DataSource {
private static volatile DataSource datasource;
private BasicDataSource ds;
private DataSource() throws IOException, SQLException, PropertyVetoException {
ds = new BasicDataSource();
ds.setUsername("sa");
ds.setPassword("sa");
ds.setUrl("jdbc:h2:tcp://localhost/~/test");
ds.setMinIdle(5);
ds.setMaxActive(10);
ds.setMaxIdle(20);
ds.setMaxOpenPreparedStatements(180);
}
public static DataSource getInstance() throws IOException, SQLException, PropertyVetoException {
if (datasource == null) {
synchronized (DataSource.class) {
if (datasource == null) {
datasource = new DataSource();
}
}
datasource = new DataSource();
}
return datasource;
}
public Connection getConnection() throws SQLException {
return this.ds.getConnection();
}
}
and then execute select * from information_schema.sessions;, there will be two rows. What is wrong? Also I was trying H2 tutorial example, but I've got the same result.
You are using a connection pool, namely BasicDataSource. It will create a configured number of connections initially and then when getConnection() is called, it will either reused a free pooled connection or create a new one, up to a configured limit (or no limit if configured so). When the connection obtained is "closed" using Connection.close(), it is actually returned to the pool instead of being closed right away.
You basically have no control over how many open connctions there will be at a given time, apart of configuring the minimum and maximum allowed open connections. You observing two open connections therefore doesn't prove anything. If you want to see whether there is a limit on open connections, configure the BasicDataSource to use at most 2 connections using BasicDataSource.maxActive(), then try to obtain three connections at the same time. And for another test, with the same configuration, try obtaining two connections, returning them using Connection.close() and then obtaining another two connections.

Java JDBC efficiency: How long should a connection be maintained?

I'm still working on the same problem mention here. It seems to work fine especially after creating an AbstractModel class shown below:
public abstract class AbstractModel {
protected static Connection myConnection = SingletonConnection.instance().establishConnection();
protected static Statement stmt;
protected static ResultSet rs;
protected boolean loginCheck; // if userId and userLoginHistoryId are valid - true, else false
protected boolean userLoggedIn; // if user is already logged in - true, else false
public AbstractModel (int userId, Long userLoginHistoryId){
createConnection(); // establish connection
loginCheck = false;
userLoggedIn = false;
if (userId == 0 && userLoginHistoryId == 0){ // special case for login
loginCheck = true; // 0, 0, false, false
userLoggedIn = false; // set loginCheck to true, userLogged in to false
} else {
userLoggedIn = true;
try{
String query = "select \"user_login_session_check\"(" + userId + ", " + userLoginHistoryId + ");";
System.out.println("query: " + query);
stmt = myConnection.createStatement();
rs = stmt.executeQuery(query);
while (rs.next()){
loginCheck = rs.getBoolean(1);
}
} catch (SQLException e){
System.out.println("SQL Exception: ");
e.printStackTrace();
}
}
}
// close connection
public void closeConnection(){
try{
myConnection.close();
} catch (SQLException e){
System.out.println("SQL Exception: ");
e.printStackTrace();
}
}
// establish connection
public void createConnection(){
myConnection = SingletonConnection.instance().establishConnection();
}
// login session check
public boolean expiredLoginCheck (){
if (loginCheck == false && userLoggedIn == true){
closeConnection();
return false;
} else {
return true;
}
}
}
I've already posted the stored procedures and Singleton Pattern implementation in the link to the earlier question above.
I'm under the impression that I don't need to close the connection to the database after each single data transaction, as it would just slow the application. I'm looking at about 30 users for this system I'm building, so performance and usability is important.
Is it correct to prolong the connection for at least 3-4 data transactions? Eg. Validation checks to user inputs for some form, or, something similar to google's auto-suggest ... These are all separate stored function calls based on user input. Can I use one connection instance, instead of connecting and disconnecting after each data transaction? Which is more efficient?
If my assumptions are correct (more efficient to use one connection instance) then opening and closing of the connection should be handled in the controller, which is why I created the createConnection() and closeConnection() methods.
Thanks.
Your code should never depend on the fact, that your application is currently the only client to the database or that you have only 30 users. So you should handle database connections like files, sockets and all other kinds of scarce resources that you may run ouf of.
Thus you should always clean up after yourself. No matter what you do. Open connection, do your stuff (one or SQL statements) and close connection. Always!
In your code you create your connection and save it into a static variable - this connection will last as long as your AbstractModel class lives, probably forever - this is bad. As with all similar cases put you code inside try/finally to make sure the connection gets always closed.
I have seen application servers running ouf of connections because of web applications not closing connections. Or because they closed at logout and somebody said "we will never have more then that much users at the same time" but it just scaled a little to high.
Now as you have your code running and closing the connections properly add connection pooling, like zaske said. This will remedy the performance problem of opening/closing database connection, which truely is costly. On the logical layer (your application) you doesn't want to know when to open/close physical connection, the db layer (db pool) will handle it for you.
Then you can even go and set up a single connection for your whole session model, which is also supported by DBCP - this is no danger, since you can reconfigure the pool afterwards if you need without touching your client code.
Like Tomasz said, you should never ever depend on the fact that your application will be used by a small number of clients. The fact that the driver will timeout after a certain amount of time does not guarantee you that you will have enough available connections. Picture this: a lot of databases come pre-configured with a maximum number of connections set to (say) 15 and a timeout of (let's say) 10-15 minutes. If you have 30 clients and each does an operation, somewhere around half-way you'll be stuck short on connections.
You should handle connections, files, streams and other resources the following way:
public void doSomething()
{
Connection connection = null;
Statement stmt = null;
ResultSet rs = null;
final String sql = "SELECT ....");
try
{
connection = getConnection();
stmt = connection.createStatement();
rs = stmt.executeQuery(sql);
if (rs.next())
{
// Do something here...
}
}
catch (SQLException e)
{
e.printStackTrace();
}
finally
{
closeResultSet(rs);
closeStatement(stmt);
closeConnection(connection);
}
}
The try/catch/finally guarantees you that the connection will get closed no matter the outcome. If there is some sort of failure, the finally block will still close the connection, just like it would do, if things were okay.
Similarly, with file and streams you need to do the same thing. Initialize the respective object as null outside your try/catch/finally, then follow the approach above.
This misconception makes a lot of Java applications misbehave under Windows, where people don't close files (streams to files, etc) and these files become locked, forcing you to either kill the JVM, or even restart your machine.
You can also use a connection pool such as for example Apache's DBCP, but even then you should take care of closing your resources, despite the fact that internally, the different connection pool implementations do not necessarily close the connections.
You'are right that you don't need to close the connection after each call.
Bare in mind that that modern database implement internal connection pools, but your application still need to connect and retrieve a connection object, and this is what it does now.
You should consider using a database connection pool - there are various Java frameworks to provide you such a solution, and they will define (you will be able to configure of course) when a database connection pool is closed.
In general - you should ask yourself whether your database serves only your application, or does it serve other application as well - if it does not serve other application as well, you might be able to be more "greedy" and keep connections open for a longer time.
I would also recommend that your application will create on start a fixed number of connections (define it in your configuration with a value of "Minimum connections number") and you will let it grow if needed to a maximum connection numbers.
As I previously mentioned - the ideas are suggest are implemented already by all kinds of frameworks, for example - the DBCP project of Apache.
Here is the Singleton Pattern which I initialize the myConenction field in all my Models to:
public class DatabaseConnection {
private static final String uname = "*******";
private static final String pword = "*******";
private static final String url = "*******************************";
Connection connection;
// load jdbc driver
public DatabaseConnection(){
try{
Class.forName("org.postgresql.Driver");
establishConnection();
} catch (ClassNotFoundException ce) {
System.out.println("Could not load jdbc Driver: ");
ce.printStackTrace();
}
}
public Connection establishConnection() {
// TODO Auto-generated method stub
try{
connection = DriverManager.getConnection(url, uname, pword);
} catch (SQLException e){
System.out.println("Could not connect to database: ");
e.printStackTrace();
}
return connection;
}
}
public class SingletonConnection {
private static DatabaseConnection con;
public SingletonConnection(){}
public static DatabaseConnection instance(){
assert con == null;
con = new DatabaseConnection();
return con;
}
}
Of course each and every connection to the database from the app goes through a Model.

Java Threads and MySQL

I have a threaded chat server application which requires MySQL authencation.
Is the best way to have 1 class create the MySQL connection, keep that connection open and let every thread use that connection but use own Query handler?
Or is it better to have all threads make a seperate connection to MySQL to authencate?
Or is it better to let 1 class handle the queries AND connections?
We are looking at a chatserver that should be able to handle upto 10.000 connections/users.
I am now using c3p0, and I created this:
public static void main(String[] args) throws PropertyVetoException
{
ComboPooledDataSource pool = new ComboPooledDataSource();
pool.setDriverClass("com.mysql.jdbc.Driver");
pool.setJdbcUrl("jdbc:mysql://localhost:3306/db");
pool.setUser("root");
pool.setPassword("pw");
pool.setMaxPoolSize(100);
pool.setMinPoolSize(10);
Database database = new Database(pool);
try
{
ResultSet rs = database.query("SELECT * FROM `users`");
while (rs.next()) {
System.out.println(rs.getString("userid"));
System.out.println(rs.getString("username"));
}
}
catch(Exception ex)
{
System.out.println(ex.getMessage());
}
finally
{
database.close();
}
}
public class Database {
ComboPooledDataSource pool;
Connection conn;
ResultSet rs = null;
Statement st = null;
public Database (ComboPooledDataSource p_pool)
{
pool = p_pool;
}
public ResultSet query (String _query)
{
try {
conn = pool.getConnection();
st = conn.createStatement();
rs = st.executeQuery(_query);
} catch (SQLException e) {
e.printStackTrace();
} finally {
}
return rs;
}
public void close ()
{
try {
st.close();
conn.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
}
Would this be thread safe?
c3p0 connection pool is a robust solution. You can also check dbcp but c3p0 shows better performance, supports auto-reconnection and some other features.
Have you looked at connection pooling ? Check out (for example) Apache DBCP or C3P0.
Briefly, connection pooling means that a pool of authenticated connections are used, and free connections are passed to you on request. You can configure the number of connections as appropriate. When you close a connection, it's actually returned to the pool and made available for another client. It makes life relatively easy in your scenario, since the pool looks after the authentication and connection management.
You should not have just one connection. It's not a thread-safe class. The idea is to get a connection, use it, and close it in the narrowest scope possible.
Yes, you'll need a pool of them. Every Java EE app server will have a JNDI pooling mechanism for you. I wouldn't recommend one class for all queries, either. Your chat ap
Your chat app ought to have a few sensible objects in its domain model. I'd create data access objects for them as appropriate. Keep the queries related to a particular domain model object in its DAO.
is the info in this thread up-to-date? Googling brings up a lot of different things, as well as this - http://dev.mysql.com/tech-resources/articles/connection_pooling_with_connectorj.html

Categories