We Keep Coding

Managing database connections in java

A full day of googling this problem has left me more confused than ever so I'm appealing to SO for help. I'm trying to use pooled connections to a mysql DB but I'm clearly misunderstanding something. Below are snippets of code from an application that scans a folder for new directories that represent "jobs"; when found, database objects are created for each folder found. I based the _insert() method on a pattern I found on SO. My understanding is that the connections are properly closed and returned to the connection pool. However, I noticed that, after adding 8 objects, the code would hang on getConnection(). I found somewhere that the default number of active connections was 8, so I added the debug line where I limit the number of active connections to 2. Sure enough, only two objects get added before the code hangs.
What's going on? What do I need to change to make these connections get freed and added back to the pool? I found one post that mentioned the PoolableConnection class but I'm confused by the documentation as well as by the fact that most other examples I've found don't seem to use it.
The Scanner class that creates Job objects in the database based on folders found in a particular directory on disk:
public class Scanner extends Thread {
public void run() {
syncJobs();
}
void syncJobs(List<String> folderNames) {
for (String folderName : folderNames) {
Job job = addJobToDB(folderName);
}
}
Job addJobToDB(String folderName ) {
Job job = new Job();
job.name = folderName;
job.save();
return job;
}
}
There's an abstract base class for all objects (each objects overrides _insert):
public abstract class DBObject {
private final int insert() {
return _insert();
}
public final void save() {
if (id == 0)
id = insert();
else
update();
}
}
And there's the actual Job object (with only the insert method shown):
public class Job extends DBObject {
public int _insert() {
String query = "insert into jobs (name) values (?)";
Connection conn = null;
PreparedStatement ps = null;
ResultSet rs = null;
int id = 0;
try {
conn = Database.getConnection();
ps = conn.prepareStatement(query, Statement.RETURN_GENERATED_KEYS);
ps.setInt(1, id);
ps.executeUpdate();
rs = ps.getGeneratedKeys();
rs.next();
id = rs.getInt(1);
} catch (Exception e) {
System.out.println(e.getMessage());
} finally {
DbUtils.closeQuietly(rs);
DbUtils.closeQuietly(ps);
DbUtils.closeQuietly(conn);
}
return id;
}
}
And, lastly, the Database object that provides connections:
import org.apache.commons.dbcp.BasicDataSource;
public final class Database {
private static final BasicDataSource dataSource = new BasicDataSource();
static {
dataSource.setUrl("jdbc:mysql://localhost:3306/dbName?useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC");
dataSource.setUsername("user");
dataSource.setPassword("****");
// This line added for debugging: sure enough, only 2 objects are created.
dataSource.setMaxActive(2);
}
public static Connection getConnection() throws SQLException {
return dataSource.getConnection();
}
}

Oracle JDBC UCP and Java

I'm wondering if anyone can shed some light on this topic, as I have been racking my brain for days and can't quite understand why this does not work. I have three classes
main, RetrieveDBVersion,GetOracleConnection I've been doing some testing with oracle JDBC, UCP and Java 1.7.
According to the Oracle documentation, If I use connection pooling the connection will be returned to the pool as soon as I close the connection, Invalidate it and set it to null See Here. So I decided to give it a whirl and see if it would perform just like the documentation says it should. In my Main application I have a simple loop which makes a connection 200 times by calling RetrieveDBVersion. RetrieveDBVersion is simply performing a query and returning the driver version. My loop works fine until I hit the magic number of 68 and then I receive an error which states
java.sql.SQLException: Exception occurred while getting connection:
oracle.ucp.UniversalConnectionPoolException:
Cannot get Connection from Datasource: java.sql.SQLException:
Listener refused the connection with the following error:
ORA-12516, TNS:listener could not find available handler with matching protocol stack
These are the detail of the 3 methods. These methods are not in a server environment. They are simply calling a local oracle express database and I'm running them from my desktop. Why would I keep getting this error? If I'm returning the connections back to the pool?
Main
import com.jam.DB.JDBCVersion;
import static java.lang.System.out;
public class MainApp {
public static void main(String[] args) {
String myMainJDBCVar;
try{
for(int i=1; i<200; i++ )
{
myMainJDBCVar= JDBCVersion.RetrieveDBVersion();
out.println(myMainJDBCVar + " " + i);
}
out.println("this is Done!");
}
catch (Exception e) {
System.out.println(e.getMessage());
}
}
}
RetrieveDBVersion
import java.sql.*;
import oracle.ucp.jdbc.ValidConnection;
public class JDBCVersion {
public static String DBVersion;
public static String RetrieveDBVersion()throws SQLException {
Connection conn = JDBCConnection.GetOracleConnection("test");
try {
DatabaseMetaData meta = conn.getMetaData();
//get driver info
System.out.println("JDBC driver version is " + meta.getDriverMajorVersion());
DBVersion = meta.getDriverVersion();
} catch (SQLException e) {
e.printStackTrace();
DBVersion = e.getMessage();
}
finally {
System.out.println("hit the finally clause");
((ValidConnection) conn).setInvalid();
conn.close();
conn=null;
}
return DBVersion;
}
GetOracleConnection
import oracle.ucp.jdbc.PoolDataSource;
import oracle.ucp.jdbc.PoolDataSourceFactory;
import java.sql.*;
public class JDBCConnection {
public static Connection GetOracleConnection(String Enviroment) throws SQLException{
PoolDataSource pds = PoolDataSourceFactory.getPoolDataSource();
Connection conn = null; //ora.defaultConnection();
try {
pds.setConnectionFactoryClassName("oracle.jdbc.pool.OracleDataSource");
pds.setURL("jdbc:oracle:thin:#//localhost:1521/xe");
pds.setUser("system");
//pds.setInitialPoolSize(5);
pds.setPassword("xxx");
pds.setMaxStatements(10);
conn = pds.getConnection();
return conn;
}
catch(Exception e){
e.printStackTrace();
}
return conn;
}

So after careful though and getting a little extra help from the Oracle forum. I finally understand why the above referenced code is giving the error message that I'm receiving. See Here For Response
Because I'm setting the data source everytime the loop goes around, I'm essentially creating more than one pool. The way to do this, is create one pool and than pull connections from that pool.
New code to replace the GetOracleConnection I created a singleton class for datasource and in code I simply retrieve the connection from the data source like such
Connection conn = Database.getInstance().GetPoolSource().getConnection();
package com.jam.DB;
import oracle.ucp.jdbc.PoolDataSource;
import oracle.ucp.jdbc.PoolDataSourceFactory;
public class Database {
private static Database dbIsntance;
private static PoolDataSource pds;
private Database() {
// private constructor //
}
public static Database getInstance() {
if (dbIsntance == null) {
dbIsntance = new Database();
}
return dbIsntance;
}
public PoolDataSource GetPoolSource() {
if (pds == null) {
pds = PoolDataSourceFactory.getPoolDataSource();
try {
pds.setConnectionFactoryClassName("oracle.jdbc.pool.OracleDataSource");
pds.setURL("jdbc:oracle:thin:#//localhost:1521/xe");
pds.setUser("system");
pds.setPassword("xxxx");
pds.setMaxStatements(15);
return pds;
} catch (Exception e) {
}
return pds;
}
return pds;
}
}

PreparedStatement execute() doesn't affect the database

I have issues when inserting data into a mysql database with jdbc.
when adding a new user to the database, I get a new connection, create a prepared statement and execute the query. However, no results are shown in the database.
For example, let's assume I manually add a new user in the database with MySql Query Browser.
I add a new user --> name = Stefano, pin = 1010
An auto-increment id is generated: id = 1.
Suppose I decide to add a new user programmaticaly:
I call the method addUser(String username, int pin) --> addUser("pippo", 7636);
No error occurs
I open MySql Query Browser and none user is added.
Finally I add a new user manually:
name = pluto , pin = 3434.
Now my table result in:
id
name
pin
1
stefano
1010
3
pluto
3434
id=2 is missing. So pippo has been added but I can't see it.
What's wrong?
Here my java code simplified:
package simpleexample;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.util.logging.Level;
import java.util.logging.Logger;
public class SimpleExample {
public static void addUser(String username, int pin) {
Connection conn = null;
PreparedStatement preparedStmt = null;
String insertSQL = "insert into users(name, pin) values (?, ?)";
try {
conn = DBConnectionPool.getConnection();
preparedStmt = conn.prepareStatement(insertSQL);
preparedStmt.setString(1, username);
preparedStmt.setInt(2, pin);
preparedStmt.execute();
} catch (SQLException ex) {
Logger.getLogger(SimpleExample.class.getName()).log(Level.SEVERE, null, ex);
} finally {
if (conn != null) {
DBConnectionPool.releaseConnection(conn);
}
}
}
public static void main(String[] args) {
addUser("pippo", 7636);
}
}
Here the class DBConnectionPool:
package simpleexample;
import java.util.*;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.sql.*;
/**
* This class simulates a db connection pool
*/
public class DBConnectionPool {
/*
* This code prepare the db connection pool. In particular, it creates the
* free connections queue and defines the db properties.
*/
static {
freeDbConnections = new ArrayList<Connection>();
try {
DBConnectionPool.loadDbProperties();
DBConnectionPool.loadDbDriver();
} catch (ClassNotFoundException e) {
System.out.println("DB DRIVER NOT FOUND!");
System.exit(1);
} catch (IOException e) {
System.out.println("DB CONNECTION POOL ERROR!");
System.exit(2);
}
}
/**
* The db properties (driver, url, login, and password)
*/
private static Properties dbProperties;
/**
* The free connection queue
*/
private static List<Connection> freeDbConnections;
/**
* Returns a free db connection accessing to the free db connection queue.
* If the queue is empty a new db connection will be created.
*
* #return A db connection
* #throws SQLException
*/
public static synchronized Connection getConnection() throws SQLException {
Connection connection;
if (!freeDbConnections.isEmpty()) {
// Extract a connection from the free db connection queue
connection = freeDbConnections.get(0);
DBConnectionPool.freeDbConnections.remove(0);
try {
// If the connection is not valid, a new connection will be
// analyzed
if (connection.isClosed()) {
connection = DBConnectionPool.getConnection();
}
} catch (SQLException e) {
connection = DBConnectionPool.getConnection();
}
} else // The free db connection queue is empty, so a new connection will
// be created
{
connection = DBConnectionPool.createDBConnection();
}
return connection;
}
/**
* Releases the connection represented by <code>pReleasedConnection</code>
* parameter
*
* #param pReleasedConnection The db connection to release
*/
public static synchronized void releaseConnection(
Connection pReleasedConnection) {
// Add the connection to the free db connection queue
DBConnectionPool.freeDbConnections.add(pReleasedConnection);
}
/**
* Creates a new db connection
*
* #return A db connection
* #throws SQLException
*/
private static Connection createDBConnection() throws SQLException {
Connection newConnection = null;
// Create a new db connection using the db properties
// newConnection = DriverManager.getConnection(
// "jdbc:mysql://localhost/resources", "root", "");
newConnection = DriverManager.getConnection(
DBConnectionPool.dbProperties.getProperty("url"),
DBConnectionPool.dbProperties.getProperty("username"),
DBConnectionPool.dbProperties.getProperty("password"));
newConnection.setAutoCommit(false);
return newConnection;
}
private static void loadDbDriver() throws ClassNotFoundException {
Class.forName(DBConnectionPool.dbProperties.getProperty("driver"));
}
/**
* Loads the db properties
*
* #throws IOException
*/
private static void loadDbProperties() throws IOException {
InputStream fileProperties = new FileInputStream("database.properties");
DBConnectionPool.dbProperties = new Properties();
DBConnectionPool.dbProperties.load(fileProperties);
}
}
Note: I have a file Database.properties in the project with
driver=org.gjt.mm.mysql.Driver
url=jdbc:mysql://localhost/name_db
username=root
password='password'
I already used the DBConnectionPool class in other projects and it's always worked fine. So I don't understand what's wrong. Maybe something about transactions?

You disable auto commit in your connection pool (seriously: don't roll your own, use an existing one, they will do a better job than this).
You never call commit anywhere in your code, therefor the result is never committed (and eventually when the connection really gets closed or otherwise lost, the change will get rolled back).
My advice:
Call commit() (or rollback()) when you have finished your unit of work.
Start using a real connection pool, like HikariCP, DBCP or c3p0

You need to use executeUpdate() not execute().
https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#executeUpdate(java.lang.String)
int executeUpdate(String sql)
throws SQLException
Executes the given SQL statement, which may be an INSERT, UPDATE, or
DELETE statement or an SQL statement that returns nothing, such as an
SQL DDL statement.
https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#execute(java.lang.String)
boolean execute(String sql)
throws SQLException
The execute method executes an SQL statement and indicates the form of
the first result. You must then use the methods getResultSet or
getUpdateCount to retrieve the result, and getMoreResults to move to
any subsequent result(s).

Can I use the same connection object when calling within Methods?

I have got a program this way:
public void MethodOne()
{
String sqlquery = "select * from vendor_items where category_id = 1 ";
PreparedStatement consildatedPst = connection.prepareStatement(sqlquery);
ResultSet consilatedReslset = consildatedpst.executeQuery();
while(consilatedReslset.next())
{
String name = consilatedReslset.getString("name");
if(name!=null)
{
MethodTwo();
}
}
}
public void MethodTwo(String name)
{
String sqlquery2 = "select ename from Vendor where name=?";
PreparedStatement otherPst = connection.prepareStatement(sqlquery2);
otherPst.setString(1,name);
}
This is the way connection is established (Later I will go for Connection Pooling).
public class DBConnection {
public static Connection getDBConnection() {
String sURL="jdbc:mysql://localhost:3306/oms";
String sUserName="root";
String sPwd="";
Connection conn = null;
try {
Class.forName("com.mysql.jdbc.Driver");
conn = DriverManager.getConnection(sURL, sUserName,sPwd);
return conn;
} catch (SQLException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
return conn;
}
}
My question is Can I use the same connection object when calling within Methods??

Yes, you can.
When you do:
connection.prepareStatement(sqlquery2);
It creates a new statement object using the same connection. So the ResultSets that you obtain from them will belong to different Statements and will be different and there will be NO PROBLEM for you.
In short: Different Statements manage different ResultSets. If you get 2 ResultSets from the same Statement when you get the second one the first one will be dropped but if you have 2 Statements you can manage 2 ResulSets without problem (while the connection is open, of course)

Only if you aren't using the connection in multiple threads or nesting your own methods. In other words, no. Use a new connection per method. To avoid overhead use a connection pool.

Java newbie needs help in database connection

I'm new to Java and even newer to java database connections. I've managed to create a database connection and query a table when I put it in the Main class. Now that I've moved it into a new class called Connection I am getting errors:
package lokate;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.ResultSet;
public class Connection {
private static Statement stmt = null;
private static ResultSet rs = null;
private static Connection con = null;
public Connection() throws SQLException {
try {
Class.forName("com.mysql.jdbc.Driver");
String connectionUrl = "jdbc:mysql://localhost:3306/Lokate?" +
"user=root&password=";
con = DriverManager.getConnection(connectionUrl);
stmt = con.createStatement();
retriveData("SELECT * FROM Users");
int rowsEffected = 0;
} catch (SQLException sqlEx) {
System.out.println("SQL Exception: "+ sqlEx.toString());
} catch (ClassNotFoundException classEx) {
System.out.println("Class Not Found Exception: "+ classEx.toString());
} catch (Exception Ex) {
System.out.println("Exception: "+ Ex.toString());
}
}
public static void retriveData(String SQL) throws Exception {
rs = stmt.executeQuery(SQL);
while (rs.next())
{
System.out.println(rs.getString("fname") + " : " + rs.getString("lname"));
}
}
}
I'm getting an error saying cannot find symbol. Symbol:method createStatement() and incomparable types for con = DriveManager.....
Can anyone help?
Also, is it best practice to put the connection in the class like this then call a new object every time I want to do something with the db?
Regards,
Billy

I'd say your code is an example of many worst practices. Let me count the ways:
Your Connection class is a poor abstraction that offers nothing over and above that of java.sql.Connection.
If you use your class, you'll never get to take advantage of connection pooling.
You hard wire your driver class, your connection URL, etc. You can't change it without editing and recompiling. A better solution would be to externalize such things.
Printing an error message in the catch blocks is far less information than supplying the entire stack trace.
Your code hurts my eyes. It doesn't follow the Sun Java coding standards.
Your retrieveData method is utterly worthless. What will you do with all those printed statements? Wouldn't it be better to load them into a data structure or object so the rest of your code might use that information?
It's rowsAffected - "affect" is the verb, "effect" is the noun. Another variable that's not doing any good.
You're on the wrong track. Rethink it.
I think you'll find this code more helpful.
package persistence;
import java.sql.*;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
public class DatabaseUtils
{
public static Connection createConnection(String driver, String url, String username, String password) throws ClassNotFoundException, SQLException
{
Class.forName(driver);
if ((username == null) || (password == null) || (username.trim().length() == 0) || (password.trim().length() == 0))
{
return DriverManager.getConnection(url);
}
else
{
return DriverManager.getConnection(url, username, password);
}
}
public static void close(Connection connection)
{
try
{
if (connection != null)
{
connection.close();
}
}
catch (SQLException e)
{
e.printStackTrace();
}
}
public static void close(Statement st)
{
try
{
if (st != null)
{
st.close();
}
}
catch (SQLException e)
{
e.printStackTrace();
}
}
public static void close(ResultSet rs)
{
try
{
if (rs != null)
{
rs.close();
}
}
catch (SQLException e)
{
e.printStackTrace();
}
}
public static void rollback(Connection connection)
{
try
{
if (connection != null)
{
connection.rollback();
}
}
catch (SQLException e)
{
e.printStackTrace();
}
}
public static List<Map<String, Object>> map(ResultSet rs) throws SQLException
{
List<Map<String, Object>> results = new ArrayList<Map<String, Object>>();
try
{
if (rs != null)
{
ResultSetMetaData meta = rs.getMetaData();
int numColumns = meta.getColumnCount();
while (rs.next())
{
Map<String, Object> row = new HashMap<String, Object>();
for (int i = 1; i <= numColumns; ++i)
{
String name = meta.getColumnName(i);
Object value = rs.getObject(i);
row.put(name, value);
}
results.add(row);
}
}
}
finally
{
close(rs);
}
return results;
}
}

Your problem is that DriverManager.getConnection returns a java.sql.Connection. Since your class is also called Connection you are getting a name clash between lokate.Connection and java.sql.Connection. You Will need to specify the fully qualified class name where ever you want to use java.sql.Connection, otherwise lokate.Connection is assumed.
Specify fully qualified class name like so:
java.sql.Connection con = null;
// ....
con = DriverManager.getConnection(connectionUrl);
Alternatively, rename your Connection class to something else and you will not get this naming conflict.

Connection is an existing type in the java.sql package, which is what DriverManager.getConnection returns. You have named your class as Connection too, so this is causing the confusion. The simplest way out would be to rename your class to something else and add an import java.sql.Connection; at the top.

Also, is it best practice to put the connection in the class like this then call a new object every time I want to do something with the db?
I think the best practice would be to use an existing solution to this problem, so that you can avoid re-inventing the wheel and focus on what makes your problem unique.
If you are writing a server application (it isn't clear whether you are), then I would also consider using a database connection pool. Creating new database connections on the fly is not efficient and doesn't scale well. You can read about database connection issues in this article I wrote a while back.