We Keep Coding

SQL connections dangling: Where am I not correctly closing up connections correctly?

I am building a basic java application to load some files into a mysql database. I am able to load the files up and populate my tables without any problems. However after speaking to someone who reviewed my code, I am apparently not correctly closing my connections and wasting resources. Where am I not closing up the connections? Have I done this incorrectly?
I am using the try-with-resources construct within my DbSinger class to execute prepared statements to my database, which should automatically close the connection so long as the AutoCloseable interface is implemented, which it is in the parent class of Db. The close() method however is never reached. The DbSinger is instantiated inside my main() and then runs it's single method populateSingers() with an ArrayList of Singer objects.
Connection Class
public class SQLConnection {
private static final String servername = "localhost";
private static final int port = 3306;
private static final String user = "ng_user";
private static final String pass = "ng";
private static final String db = "ng_music";
private static final String connectionString = "jdbc:mysql://" + servername + ":" + port + "/" + db;
public Connection provide() {
try {
Class.forName("com.mysql.cj.jdbc.Driver");
return DriverManager.getConnection(connectionString, user, pass);
}
catch (SQLException | ClassNotFoundException e) {
throw new SQLConnectionException(e);
}
}
public class SQLConnectionException extends RuntimeException {
SQLConnectionException(Exception e) {super(e);}
}
}
Abstract parent class
public abstract class Db implements AutoCloseable{
private Connection connection;
Db() {
SQLConnection sqlC = new SQLConnection();
this.connection = sqlC.provide();
}
#Override
public synchronized void close() throws SQLException {
if(connection != null) {
connection.close();
connection = null;
System.out.println("Connection closed");
}
}
Connection getConnection() {
return connection;
}
boolean checkIfPopulated(String query){
try {
PreparedStatement ps = getConnection().prepareStatement(query);
ResultSet rs = ps.executeQuery();
return !rs.next();
} catch (SQLException e) {
e.printStackTrace();
}
return true;
}
}
Concrete class to execute queries to database for singers table
public class DbSinger extends Db {
public DbSinger() {
super();
}
public void populateSingers(ArrayList<Singer> singers) {
String populateSingersQuery = "insert into ng_singers(name, dob, sex) values(?,?,?)";
if(!checkIfPopulated("select * from ng_singers")){
System.out.println("Singer Table is already populated");
return;
}
try (PreparedStatement ps = getConnection().prepareStatement(populateSingersQuery)) {
for (Singer s : singers) {
ps.setString(1, s.getName());
ps.setDate(2, java.sql.Date.valueOf(s.getDob()));
ps.setString(3, s.getSex());
ps.addBatch();
}
ps.executeBatch();
System.out.println("Singers added to table");
} catch (SQLException e) {
e.printStackTrace();
}
}
}
My code is able to execute is able to run fine and does what it needs to, but I want to understand why and where I am not closing connections, and to understand how I can resolve this. Or at least understand if I am approaching this wrong.

In your case, you need to instantiate DBSinger class in try-with-resources statement to close the underlying connection.
Instead of doing:
DbSinger dbSinger = new DbSinger();
You need to do:
try (DbSinger dbSinger = new DbSinger()) {
// Your other code
}
This way the close() method you are overriding in your Db class will be called automatically.
Also, close the preparedStatement you created in your checkIfPopulated method by:
try (PreparedStatement ps = getConnection().prepareStatement(query)) {
// Other codes
}

Your code is old way. And you do need close manually. However, with Java 8, you can use try with resource like below,
try (Connection conn = ds.getConnection();
Statement stmt = conn.createStatement()) {
try {
stmt.execute(dropsql);
} catch (Exception ignore) {} // ignore if table not dropped
stmt.execute(createsql);
stmt.execute(insertsql);
try (ResultSet rs = stmt.executeQuery(selectsql)) {
rs.next();
} catch (Exception e2) {
e2.printStackTrace();
return("failed");
}
} catch(Exception e) {
e.printStackTrace();
return("failed");
}

Managing database connections in java

A full day of googling this problem has left me more confused than ever so I'm appealing to SO for help. I'm trying to use pooled connections to a mysql DB but I'm clearly misunderstanding something. Below are snippets of code from an application that scans a folder for new directories that represent "jobs"; when found, database objects are created for each folder found. I based the _insert() method on a pattern I found on SO. My understanding is that the connections are properly closed and returned to the connection pool. However, I noticed that, after adding 8 objects, the code would hang on getConnection(). I found somewhere that the default number of active connections was 8, so I added the debug line where I limit the number of active connections to 2. Sure enough, only two objects get added before the code hangs.
What's going on? What do I need to change to make these connections get freed and added back to the pool? I found one post that mentioned the PoolableConnection class but I'm confused by the documentation as well as by the fact that most other examples I've found don't seem to use it.
The Scanner class that creates Job objects in the database based on folders found in a particular directory on disk:
public class Scanner extends Thread {
public void run() {
syncJobs();
}
void syncJobs(List<String> folderNames) {
for (String folderName : folderNames) {
Job job = addJobToDB(folderName);
}
}
Job addJobToDB(String folderName ) {
Job job = new Job();
job.name = folderName;
job.save();
return job;
}
}
There's an abstract base class for all objects (each objects overrides _insert):
public abstract class DBObject {
private final int insert() {
return _insert();
}
public final void save() {
if (id == 0)
id = insert();
else
update();
}
}
And there's the actual Job object (with only the insert method shown):
public class Job extends DBObject {
public int _insert() {
String query = "insert into jobs (name) values (?)";
Connection conn = null;
PreparedStatement ps = null;
ResultSet rs = null;
int id = 0;
try {
conn = Database.getConnection();
ps = conn.prepareStatement(query, Statement.RETURN_GENERATED_KEYS);
ps.setInt(1, id);
ps.executeUpdate();
rs = ps.getGeneratedKeys();
rs.next();
id = rs.getInt(1);
} catch (Exception e) {
System.out.println(e.getMessage());
} finally {
DbUtils.closeQuietly(rs);
DbUtils.closeQuietly(ps);
DbUtils.closeQuietly(conn);
}
return id;
}
}
And, lastly, the Database object that provides connections:
import org.apache.commons.dbcp.BasicDataSource;
public final class Database {
private static final BasicDataSource dataSource = new BasicDataSource();
static {
dataSource.setUrl("jdbc:mysql://localhost:3306/dbName?useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC");
dataSource.setUsername("user");
dataSource.setPassword("****");
// This line added for debugging: sure enough, only 2 objects are created.
dataSource.setMaxActive(2);
}
public static Connection getConnection() throws SQLException {
return dataSource.getConnection();
}
}

PreparedStatement execute() doesn't affect the database

I have issues when inserting data into a mysql database with jdbc.
when adding a new user to the database, I get a new connection, create a prepared statement and execute the query. However, no results are shown in the database.
For example, let's assume I manually add a new user in the database with MySql Query Browser.
I add a new user --> name = Stefano, pin = 1010
An auto-increment id is generated: id = 1.
Suppose I decide to add a new user programmaticaly:
I call the method addUser(String username, int pin) --> addUser("pippo", 7636);
No error occurs
I open MySql Query Browser and none user is added.
Finally I add a new user manually:
name = pluto , pin = 3434.
Now my table result in:
id
name
pin
1
stefano
1010
3
pluto
3434
id=2 is missing. So pippo has been added but I can't see it.
What's wrong?
Here my java code simplified:
package simpleexample;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.util.logging.Level;
import java.util.logging.Logger;
public class SimpleExample {
public static void addUser(String username, int pin) {
Connection conn = null;
PreparedStatement preparedStmt = null;
String insertSQL = "insert into users(name, pin) values (?, ?)";
try {
conn = DBConnectionPool.getConnection();
preparedStmt = conn.prepareStatement(insertSQL);
preparedStmt.setString(1, username);
preparedStmt.setInt(2, pin);
preparedStmt.execute();
} catch (SQLException ex) {
Logger.getLogger(SimpleExample.class.getName()).log(Level.SEVERE, null, ex);
} finally {
if (conn != null) {
DBConnectionPool.releaseConnection(conn);
}
}
}
public static void main(String[] args) {
addUser("pippo", 7636);
}
}
Here the class DBConnectionPool:
package simpleexample;
import java.util.*;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.sql.*;
/**
* This class simulates a db connection pool
*/
public class DBConnectionPool {
/*
* This code prepare the db connection pool. In particular, it creates the
* free connections queue and defines the db properties.
*/
static {
freeDbConnections = new ArrayList<Connection>();
try {
DBConnectionPool.loadDbProperties();
DBConnectionPool.loadDbDriver();
} catch (ClassNotFoundException e) {
System.out.println("DB DRIVER NOT FOUND!");
System.exit(1);
} catch (IOException e) {
System.out.println("DB CONNECTION POOL ERROR!");
System.exit(2);
}
}
/**
* The db properties (driver, url, login, and password)
*/
private static Properties dbProperties;
/**
* The free connection queue
*/
private static List<Connection> freeDbConnections;
/**
* Returns a free db connection accessing to the free db connection queue.
* If the queue is empty a new db connection will be created.
*
* #return A db connection
* #throws SQLException
*/
public static synchronized Connection getConnection() throws SQLException {
Connection connection;
if (!freeDbConnections.isEmpty()) {
// Extract a connection from the free db connection queue
connection = freeDbConnections.get(0);
DBConnectionPool.freeDbConnections.remove(0);
try {
// If the connection is not valid, a new connection will be
// analyzed
if (connection.isClosed()) {
connection = DBConnectionPool.getConnection();
}
} catch (SQLException e) {
connection = DBConnectionPool.getConnection();
}
} else // The free db connection queue is empty, so a new connection will
// be created
{
connection = DBConnectionPool.createDBConnection();
}
return connection;
}
/**
* Releases the connection represented by <code>pReleasedConnection</code>
* parameter
*
* #param pReleasedConnection The db connection to release
*/
public static synchronized void releaseConnection(
Connection pReleasedConnection) {
// Add the connection to the free db connection queue
DBConnectionPool.freeDbConnections.add(pReleasedConnection);
}
/**
* Creates a new db connection
*
* #return A db connection
* #throws SQLException
*/
private static Connection createDBConnection() throws SQLException {
Connection newConnection = null;
// Create a new db connection using the db properties
// newConnection = DriverManager.getConnection(
// "jdbc:mysql://localhost/resources", "root", "");
newConnection = DriverManager.getConnection(
DBConnectionPool.dbProperties.getProperty("url"),
DBConnectionPool.dbProperties.getProperty("username"),
DBConnectionPool.dbProperties.getProperty("password"));
newConnection.setAutoCommit(false);
return newConnection;
}
private static void loadDbDriver() throws ClassNotFoundException {
Class.forName(DBConnectionPool.dbProperties.getProperty("driver"));
}
/**
* Loads the db properties
*
* #throws IOException
*/
private static void loadDbProperties() throws IOException {
InputStream fileProperties = new FileInputStream("database.properties");
DBConnectionPool.dbProperties = new Properties();
DBConnectionPool.dbProperties.load(fileProperties);
}
}
Note: I have a file Database.properties in the project with
driver=org.gjt.mm.mysql.Driver
url=jdbc:mysql://localhost/name_db
username=root
password='password'
I already used the DBConnectionPool class in other projects and it's always worked fine. So I don't understand what's wrong. Maybe something about transactions?

You disable auto commit in your connection pool (seriously: don't roll your own, use an existing one, they will do a better job than this).
You never call commit anywhere in your code, therefor the result is never committed (and eventually when the connection really gets closed or otherwise lost, the change will get rolled back).
My advice:
Call commit() (or rollback()) when you have finished your unit of work.
Start using a real connection pool, like HikariCP, DBCP or c3p0

You need to use executeUpdate() not execute().
https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#executeUpdate(java.lang.String)
int executeUpdate(String sql)
throws SQLException
Executes the given SQL statement, which may be an INSERT, UPDATE, or
DELETE statement or an SQL statement that returns nothing, such as an
SQL DDL statement.
https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#execute(java.lang.String)
boolean execute(String sql)
throws SQLException
The execute method executes an SQL statement and indicates the form of
the first result. You must then use the methods getResultSet or
getUpdateCount to retrieve the result, and getMoreResults to move to
any subsequent result(s).

Can I use the same connection object when calling within Methods?

I have got a program this way:
public void MethodOne()
{
String sqlquery = "select * from vendor_items where category_id = 1 ";
PreparedStatement consildatedPst = connection.prepareStatement(sqlquery);
ResultSet consilatedReslset = consildatedpst.executeQuery();
while(consilatedReslset.next())
{
String name = consilatedReslset.getString("name");
if(name!=null)
{
MethodTwo();
}
}
}
public void MethodTwo(String name)
{
String sqlquery2 = "select ename from Vendor where name=?";
PreparedStatement otherPst = connection.prepareStatement(sqlquery2);
otherPst.setString(1,name);
}
This is the way connection is established (Later I will go for Connection Pooling).
public class DBConnection {
public static Connection getDBConnection() {
String sURL="jdbc:mysql://localhost:3306/oms";
String sUserName="root";
String sPwd="";
Connection conn = null;
try {
Class.forName("com.mysql.jdbc.Driver");
conn = DriverManager.getConnection(sURL, sUserName,sPwd);
return conn;
} catch (SQLException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
return conn;
}
}
My question is Can I use the same connection object when calling within Methods??

Yes, you can.
When you do:
connection.prepareStatement(sqlquery2);
It creates a new statement object using the same connection. So the ResultSets that you obtain from them will belong to different Statements and will be different and there will be NO PROBLEM for you.
In short: Different Statements manage different ResultSets. If you get 2 ResultSets from the same Statement when you get the second one the first one will be dropped but if you have 2 Statements you can manage 2 ResulSets without problem (while the connection is open, of course)

Only if you aren't using the connection in multiple threads or nesting your own methods. In other words, no. Use a new connection per method. To avoid overhead use a connection pool.

How to Update the dabtabase using for loop and getting the last sequence value

I am trying to do multi-threading here, now I have to update my database using DbHandler class
The program execution begins in a controller class which has a main method and a thread pool:
public class RunnableController {
// Main method
public static void main(String[] args) throws InterruptedException {
try {
RunnableController controller = new RunnableController();
controller.initializeDb();
controller.initialiseThreads();
System.out.println("Polling");
} catch (Exception e) {
e.printStackTrace();
}
}
private void initialUpdate()
{
DBhandler dbhandler = new DBhandler();
dbhandler.updateDb(getOutgoingQueue());
}
private void initialiseThreads() {
try {
threadExecutorRead = Executors.newFixedThreadPool(10);
PollingSynchronizer read = new PollingSynchronizer(incomingQueue, dbConncetion);
threadExecutorRead.submit(read);
} catch (Exception e){
e.printStackTrace();
}
}
}
My poller class which fetches new data and should do updating simulateously:
public class PollingSynchronizer implements Runnable {
public PollingSynchronizer(Collection<KamMessage> incomingQueue,
Connection dbConnection) {
super();
this.incomingQueue = incomingQueue;
this.dbConnection = dbConnection;
}
private int seqId;
public int getSeqId() {
return seqId;
}
public void setSeqId(int seqId) {
this.seqId = seqId;
}
// The method which runs Polling action and record the time at which it is done
public void run() {
int seqId = 0;
while (true) {
List<KamMessage> list = null;
try {
list = fullPoll(seqId);
if (!list.isEmpty()) {
seqId = list.get(0).getSequence();
incomingQueue.addAll(list);
this.outgoingQueue = incomingQueue;
System.out.println("waiting 3 seconds");
System.out.println("new incoming message");
Thread.sleep(3000);//at this wait I should execute run()
//when I debug my execution stops here and throws " Class not found Exception "
// its does not enters the message processor class
MessageProcessor processor = new MessageProcessor() {
//the run method which should fetch the message processor class.
final public void run() {
RunnableController.setOutgoingQueue(generate(outgoingQueue));
}
};
new Thread(processor).start();
}
} catch (Exception e1) {
e1.printStackTrace();
}
}
}
}
My message processor class:
public class MessageProcessor implements Runnable {
private Collection<KpiMessage> fetchedMessages;
private Connection dbConnection;
Statement st = null;
ResultSet rs = null;
PreparedStatement pstmt = null;
private Collection<KamMessage> outgoingQueue;
public Collection<KamMessage> MessageProcessor(Collection<KamMessage> outgoingQueue){
this.outgoingQueue = outgoingQueue;
this.dbConnection = dbConnection;
return outgoingQueue;
}
/**
* Method for updating new values into database in preference for dummy processing of message
* #param outgoingQueue
* #return
*/
#SuppressWarnings("javadoc")
public Collection<KamMessage> generate(Collection<KamMessage> outgoingQueue)
{
for (KamMessage pojoClass : outgoingQueue) {
KamMessage updatedValue = createKamMsg804(pojoClass);
System.out.print(" " + pojoClass.getSequence());
System.out.print(" " + pojoClass.getTableName());
System.out.print(" " + pojoClass.getAction());
System.out.print(" " + updatedValue.getKeyInfo1());
System.out.print(" " + updatedValue.getKeyInfo2());
System.out.println(" " + pojoClass.getEntryTime());
}
return outgoingQueue;
}
/**
*
* #param pojoClass
* #return msg
*/
public KamMessage createKamMsg804(KamMessage pojoClass)
{
if(pojoClass.getAction() == 804){
pojoClass.setKeyInfo1("ENTITYKEY9");
pojoClass.setKeyInfo2("STATUSKEY9");
}
return pojoClass;
}
private KamMessage convertRecordsetToPojo(ResultSet rs) throws SQLException {
KamMessage msg = new KamMessage();
int sequence = rs.getInt("SEQ");
msg.setSequence(sequence);
String tablename = rs.getString("TABLENAME");
msg.setTableName(tablename);
Timestamp entrytime = rs.getTimestamp("ENTRYTIME");
Date entryTime = new Date(entrytime.getTime());
msg.setEntryTime(entryTime);
Timestamp processingtime=rs.getTimestamp("PROCESSINGTIME");
if(processingtime!=null){
Date processingTime = new Date(processingtime.getTime());
msg.setProcessingTime(processingTime);
}
String keyInfo1 = rs.getString("KEYINFO1");
msg.setKeyInfo1(keyInfo1);
String keyInfo2 = rs.getString("KEYINFO2");
msg.setKeyInfo2(keyInfo2);
return msg;
}
#Override
public void run() {
// TODO Auto-generated method stub
}
}
This is my DBhandler Class, which should do updating in database
public class DBhandler {
Connection conn = null;
Statement st = null;
ResultSet rs = null;
PreparedStatement pstmt = null;
public DBhandler(){
super();
}
/**
* Method to initialize the database connection
* #return conn
* #throws Exception
*
*/
public Connection initializeDB() throws Exception {
System.out.println("JDBC connection");
DriverManager.registerDriver(new oracle.jdbc.driver.OracleDriver());
conn = DriverManager.getConnection("jdbc:oracle:thin:#VM-SALES-
MB:1521:SALESDB1","bdeuser", "edb"); // Connection for Database SALES-DB1
return conn;
}
//The method for updating Database
public void updateDb(Collection<KpiMessage> updatedQueue){
for(KpiMessage pojoClass : updatedQueue){
//**How the query should be used so that it gets last sequence vale and Updates into
Database**
String query = "UPDATE msg_new_to_bde Set KEYINFO1= ?, KEYINFO2 = ? WHERE SEQ = and
action = 804";
}
}
/**
* Method for Closing the connection
* #throws Exception
*
*/
public void closeDB() throws Exception {
st.close();
conn.close();
}
}
I just need to Update the database using update query in this class(DbHAndler) by calling the updatedQueue in the controller class.
My program flow - I have three classes: 1.Controller 2.PollerSynchro 3.Msgprocessor
I have database records, which are converted into POJO form and stored in a Collection. With these POJOs my classes try to do multiprocessing and updating in a single stretch.
Controller - has the thread pool, initiates poller class with poll method - done
Poller - should poll for new incoming messages and stores it in incoming queue - done
MsgProcessor - should look for new incoming messages and pass them from outgoing queue to incoming queue - also done
DbHandler- which should update in the database.
Problem:
Now my problem is
I have to implement this update while the poll thread sleeps for 3 sec -Done
In my code for the second void run() method in the Poller class, the outgoing queue is not passed and fed to the messageprocessor class for updating. My flow of execution only just loops back to first run method and am getting Class exception-Resolved
How to Update this in the database in Dbhanler class
Please help me to solve these problems.

The exception seems to come from this line (is this MessageProcessor.java line 38?)
return (KpiMsg804) fetchedMessages;
The fetchedMessages at this point seem to be an ArrayList.