We Keep Coding

Apache dbcp connection pooling not working properly

I am trying to connect to mysql database using apache dbcp2 connection pooling following its tutorial from here:
https://git-wip-us.apache.org/repos/asf?p=commons-dbcp.git;a=blob;f=doc/PoolingDataSourceExample.java;h=2a12c74898930b9623223db1597b8a8052a6f1df;hb=HEAD
My Database connection class that return the connection looks something like this:
public class DbConnection {
private static interface Singleton {
final DbConnection INSTANCE = new DbConnection();
}
private final PoolingDataSource<PoolableConnection> dataSource;
private DbConnection() {
// A ConnectionFactory that the pool will use to create Connections.
DriverManagerConnectionFactory connectionFactory = new DriverManagerConnectionFactory(url, user, pass); //url,user,pass for db connnection
//implement the pooling functionality.
PoolableConnectionFactory poolableConnectionFactory = new PoolableConnectionFactory(connectionFactory, null);
GenericObjectPoolConfig config = new GenericObjectPoolConfig();
config.setMaxWaitMillis(500);
config.setMaxTotal(20);
config.setMaxIdle(5);
config.setMinIdle(5);
//PoolingDataSource expect an ObjectPool
ObjectPool<PoolableConnection> connectionPool = new GenericObjectPool<>(poolableConnectionFactory, config);
// Set the poolableConnectionFactory's pool property to the owning pool
poolableConnectionFactory.setPool(connectionPool);
this.dataSource = new PoolingDataSource<>(connectionPool);
}
public static Connection getCon() throws SQLException {
return Singleton.INSTANCE.dataSource.getConnection();
}
}
I am using apache jmeter to test the connection and return something from my mysql database. I created 100 users with Ramp-up periods(in seconds) being 2 second. I created a Http request and when I tried to see my response in view results tree I successfully get response for first 20 request. Later requests from (21 to 100) have blank response. I have gone through many of the issues involving:
java.sql.SQLException: Cannot get a connection, pool error Timeout
waiting for idle object
I access my code as:
try (PreparedStatement ps = DbConnection.getCon().prepareStatement("SELECT id FROM test WHERE id =?;")) {
ps.setString(1, id);
try (
ResultSet rs = ps.executeQuery()) {
return rs.next();
}
} catch (Exception ex) {
}

You aren't closing Connection object, you must declare such variable in try-with-resources block :
try (Connection conn = DbConnection.getCon();
PreparedStatement ps = conn.prepareStatement("SELECT id FROM test WHERE id =?;")) {
Also you need to close ResultSet, either call close method ResultSet in method or add it to try-with-resources block using a new method returning it

Should I open a JDBC connection for each rest call?

I am dealing with high traffic in my Spring Boot project and my goal is serving clients as much fast as possible. In this case, I have more than 500 requests per second. In each rest endpoint call, I should connect my schema and gather multiple information from multiple tables. To be able to do that, should I create new connection for each eendpoint call or create & close before each db query?
I wrote a JDBC connection class but I am not sure that it is a good way. Maybe you can give me some opinion.
JDBC Connection Class
#PropertySource({"classpath:application.properties"})
#Configuration
public class FraudJDBConfiguration {
private final Logger LOGGER = LogManager.getLogger(FraudJDBConfiguration.class);
private final Environment env;
#Autowired
public FraudJDBConfiguration(Environment env) {
this.env = env;
}
#Bean
public Connection getFraudConnection() {
// Step 1: Loading or
// registering Oracle JDBC driver class
String connectionClass = env.getProperty("fraud.db.driver-class-name");
try {
Class.forName(connectionClass);
} catch (ClassNotFoundException cnfex) {
LOGGER.error(cnfex.getMessage());
throw new RuntimeException("JDBC driver class'ı bulunamadı");
}
// Step 2: Opening database connection
try {
String environmentType = env.getProperty("environment");
if (environmentType == null) {
LOGGER.error("environment Tip Hatası (TEST - UAT - LIVE)");
throw new RuntimeException("environment Tip Hatası (TEST - UAT - LIVE)");
} else {
String connectionString = null;
String username = null;
String password = null;
switch (environmentType.toLowerCase()) {
case "dev":
connectionString = env.getProperty(/*someurl*/);
username = env.getProperty(/*someusername*/);
password = env.getProperty(/*somepassword*/);
break;
case "tst":
connectionString = env.getProperty(/*someurl*/);
username = env.getProperty(/*someusername*/);
password = env.getProperty(/*somepassword*/);
break;
case "liv":
connectionString = env.getProperty(/*someurl*/);
username = env.getProperty(/*someusername*/);
password = env.getProperty(/*somepassword*/);
break;
case "uat":
connectionString = env.getProperty(/*someurl*/);
username = env.getProperty(/*someusername*/);
password = env.getProperty(/*somepassword*/);
break;
}
// Step 2.A: Create and
// get connection using DriverManager class
if (connectionString == null) {
LOGGER.error("fraud şeması için connection string bulunamadı");
throw new RuntimeException("fraud şeması için connection string bulunamadı");
}
return DriverManager.getConnection(connectionString, username, password);
}
} catch (SQLException e) {
LOGGER.error(e.getMessage());
}
return null;
}
}
DAO
#Component
public interface FraudCommTransactionsDao {
Long count();
}
DAO IMPL
#Service
public class FraudCommTransactionsDaoImpl implements FraudCommTransactionsDao {
private final FraudJDBConfiguration fraudJDBConfiguration;
#Autowired
public FraudCommTransactionsDaoImpl(FraudJDBConfiguration fraudJDBConfiguration) {
this.fraudJDBConfiguration = fraudJDBConfiguration;
}
#Override
public Long count() {
try(Connection connection = fraudJDBConfiguration.getFraudConnection()) {
Statement stmt = connection.createStatement();
ResultSet rs = stmt.executeQuery(/*some query*/);
if (rs.next()) {
return rs.getLong("transaction_id");
} else {
return 0L;
}
} catch (SQLException ex) {
ex.printStackTrace();
}
return null;
}
}

No, establishing a new physical connection to a database server is costly. It involves multiple steps: user authorization, establishing session defaults, allocating memory on both client and server, etc. This overhead should not be added to every single request.
It's a common practice to create a connection pool to share the physical connections between application threads. This introduces a concept of logical connections e.g. a Connection object created with DriverManager.getConnection() is a physical connection while DataSource.getConnection() returns a logical connection which is a proxy.
There are multiple database connection pooling libraries for Java that you can use e.g. HikariCP. Don't write your own, this is not simple.

Get fast data and deliver to client could be possible using the simplest way of using application.properties file. You may use this to get database connection to your datasource.

Websphere 7.0.0.17 error with Pool Connection

Im having a problem with an application, I'm using EclipseLink 2.4.1 with oracle 11g, I'm getting a J2CA0045E error, because the pooling does not set free any connections pool from the jndi,
Have you have this problem?, apparently the connection does not get released and it reach the limit I've configured.
Any ideas, this is my Connection pool
Connection timeout: 180
Maximum connections: 10
Minimum connections: 1
Reap time: 60
Unused timeout: 180
Aged timeout: 120
Purge policy: Entire Pool
EDIT
I have a web service in the same service, and the behavior is the same, using same jndi, the connection increase and does not release any connection
initCtx = new InitialContext();
Context envCtx = (Context) initCtx.lookup("java:comp/env");
DataSource ds = (DataSource) envCtx.lookup("jdbc/abc");
conn = ds.getConnection();

Based on the code provided, it looks like you are storing the Connection reference at a class scope. You should avoid storing references to Connection objects that could span multiple requests.
Instead, store a reference to the DataSource.
To illustrate. This approach is preferred:
public class GoodService {
DataSource ds;
public GoodService() throws NamingException {
// Only the DataSource reference is stored, this is allowed
ds = new InitialContext().lookup("java:comp/env/jdbc/abc");
}
public void doSomething() throws Exception {
// Get the connection in a request.
Connection conn = ds.getConnection();
try {
// do something...
} finally {
// Always close the connection in the finally block
// so it gets returned to the pool no matter what
conn.close();
}
}
}
The following approach is considered bad practice, and is likely to cause the J2CA0045E error you are seeing:
public class BadService {
Connection conn; // BAD: don't store a Connection across requests!
public BadService() throws NamingException, SQLException {
DataSource ds = new InitialContext().lookup("java:comp/env/jdbc/abc");
// BAD: don't leave connections open indefinitely like this
conn = ds.getConnection();
}
public void doSomething() throws Exception {
// do something using conn...
}
}

H2 connection pool

I want to make a connection pool for my h2 database. But I think my pool opens new connection every time I call getConnection(). I guess there should be a fixed amount of reusable connections, but if I run this code :
Connection conn = DataSource.getInstance().getConnection();
Statement stmt = conn.createStatement();
ResultSet rs;
rs = stmt.executeQuery("SELECT * FROM NODE_USERS;");
while (rs.next()) {
System.out.println(rs.getString("login"));
}
try {
// wait a bit
Thread.sleep(20000);
} catch (InterruptedException e) {
e.printStackTrace();
}
stmt.close();
rs.close();
conn.close();
DataSource:
public class DataSource {
private static volatile DataSource datasource;
private BasicDataSource ds;
private DataSource() throws IOException, SQLException, PropertyVetoException {
ds = new BasicDataSource();
ds.setUsername("sa");
ds.setPassword("sa");
ds.setUrl("jdbc:h2:tcp://localhost/~/test");
ds.setMinIdle(5);
ds.setMaxActive(10);
ds.setMaxIdle(20);
ds.setMaxOpenPreparedStatements(180);
}
public static DataSource getInstance() throws IOException, SQLException, PropertyVetoException {
if (datasource == null) {
synchronized (DataSource.class) {
if (datasource == null) {
datasource = new DataSource();
}
}
datasource = new DataSource();
}
return datasource;
}
public Connection getConnection() throws SQLException {
return this.ds.getConnection();
}
}
and then execute select * from information_schema.sessions;, there will be two rows. What is wrong? Also I was trying H2 tutorial example, but I've got the same result.

You are using a connection pool, namely BasicDataSource. It will create a configured number of connections initially and then when getConnection() is called, it will either reused a free pooled connection or create a new one, up to a configured limit (or no limit if configured so). When the connection obtained is "closed" using Connection.close(), it is actually returned to the pool instead of being closed right away.
You basically have no control over how many open connctions there will be at a given time, apart of configuring the minimum and maximum allowed open connections. You observing two open connections therefore doesn't prove anything. If you want to see whether there is a limit on open connections, configure the BasicDataSource to use at most 2 connections using BasicDataSource.maxActive(), then try to obtain three connections at the same time. And for another test, with the same configuration, try obtaining two connections, returning them using Connection.close() and then obtaining another two connections.

Java - Code Coverage

I have a method in one of the classes in my code base that for the life of me, I cannot get into with my junit tests.
Basically this class is called when I request a database connection, if a stale connection is returned, a new connection is established
Here is the snippet of the mthod in my class (trimmed down for this purpose)
public class TCSOracleDataSourceWrapper extends OracleDataSource {
private static final int STALE_CONNECTION_EX_CODE = 17143;
private OracleConnectionCacheManager cacheManager;
private String cacheName;
/** Local log variable **/
private final Log logger = LogFactory.getLog(getClass());
/**
* Class constructor
* #throws SQLException
*/
public TCSOracleDataSourceWrapper() throws SQLException {
super();
}
private static final long serialVersionUID = 1L;
#Override
/**
* Get a connection but if the connection is stale then refresh all DB connections
*
*/
public final Connection getConnection() throws SQLException {
logger.debug("Retrieving a database connection from the pool");
Connection connection = null;
try{
connection = super.getConnection();
}
catch(SQLException e)
{
if(e.getErrorCode() == STALE_CONNECTION_EX_CODE)
{
logger.error("Stale Oracle connection found in the Connection Pool. Refreshing invalid DB connections.");
//refresh invalid connections
cacheManager.refreshCache(cacheName, OracleConnectionCacheManager.REFRESH_INVALID_CONNECTIONS);
//now try to get the connection again
connection = super.getConnection();
}
else
{
throw e;
}
}
return connection;
}}
Any idea how I can ensure my junit tests execute the if statement?
I am currently using EasyMock and Powermock but I cannot find a way to get into this if statment using these tools
All help is greatly appreciated
Thank you
Damien

You should refactor your class to become a proxy for another data source, rather than inherit from one. This way you can easily inject into it a mock data source instead of the real one.
import javax.sql.DataSource;
public class TCSOracleDataSourceWrapper implements DataSource {
...
private DataSource wrappedDataSource;
...
public TCSOracleDataSourceWrapper(DataSource ds) {
wrappedDataSource = ds;
}
...
public final Connection getConnection() throws SQLException {
...
Connection connection = null;
try{
connection = ds.getConnection();
}
catch(SQLException e)
{
...
}
return connection;
}
}

One idea springs to mind: use aggregation rather than inheritance. This problem and others like it would go away because you can then mock the aggregated object to have whatever behavior you want. I don't see another way of getting in there right off hand. In fact, the name TCSOracleDataSourceWrapper already indicates that it's wrapping a data source (aggregation), when it actually isn't.

One quick workaround is to factor out the super.getConnection() call to a new private / protected method. Once you make that change it would be easy to mock the getBaseConnection method using power mock. This is short term fix, like the other answers suggest it is better to use delegation instead of inheritance for the wrapper implementation.
Connection getBaseConnection() throws SQLException {
return super.getConnection();
}
public final Connection getConnection() throws SQLException {
logger.debug("Retrieving a database connection from the pool");
Connection connection = null;
try{
connection = getBaseConnection();
}
catch(SQLException e)
{
if(e.getErrorCode() == STALE_CONNECTION_EX_CODE)
{
logger.error("Stale Oracle connection found in the Connection Pool. Refreshing invalid DB connections.");
//refresh invalid connections
cacheManager.refreshCache(cacheName, OracleConnectionCacheManager.REFRESH_INVALID_CONNECTIONS);
//now try to get the connection again
connection = getBaseConnection();
}
else
{
throw e;
}
}
return connection;
}