A full day of googling this problem has left me more confused than ever so I'm appealing to SO for help. I'm trying to use pooled connections to a mysql DB but I'm clearly misunderstanding something. Below are snippets of code from an application that scans a folder for new directories that represent "jobs"; when found, database objects are created for each folder found. I based the _insert() method on a pattern I found on SO. My understanding is that the connections are properly closed and returned to the connection pool. However, I noticed that, after adding 8 objects, the code would hang on getConnection(). I found somewhere that the default number of active connections was 8, so I added the debug line where I limit the number of active connections to 2. Sure enough, only two objects get added before the code hangs.
What's going on? What do I need to change to make these connections get freed and added back to the pool? I found one post that mentioned the PoolableConnection class but I'm confused by the documentation as well as by the fact that most other examples I've found don't seem to use it.
The Scanner class that creates Job objects in the database based on folders found in a particular directory on disk:
public class Scanner extends Thread {
public void run() {
syncJobs();
}
void syncJobs(List<String> folderNames) {
for (String folderName : folderNames) {
Job job = addJobToDB(folderName);
}
}
Job addJobToDB(String folderName ) {
Job job = new Job();
job.name = folderName;
job.save();
return job;
}
}
There's an abstract base class for all objects (each objects overrides _insert):
public abstract class DBObject {
private final int insert() {
return _insert();
}
public final void save() {
if (id == 0)
id = insert();
else
update();
}
}
And there's the actual Job object (with only the insert method shown):
public class Job extends DBObject {
public int _insert() {
String query = "insert into jobs (name) values (?)";
Connection conn = null;
PreparedStatement ps = null;
ResultSet rs = null;
int id = 0;
try {
conn = Database.getConnection();
ps = conn.prepareStatement(query, Statement.RETURN_GENERATED_KEYS);
ps.setInt(1, id);
ps.executeUpdate();
rs = ps.getGeneratedKeys();
rs.next();
id = rs.getInt(1);
} catch (Exception e) {
System.out.println(e.getMessage());
} finally {
DbUtils.closeQuietly(rs);
DbUtils.closeQuietly(ps);
DbUtils.closeQuietly(conn);
}
return id;
}
}
And, lastly, the Database object that provides connections:
import org.apache.commons.dbcp.BasicDataSource;
public final class Database {
private static final BasicDataSource dataSource = new BasicDataSource();
static {
dataSource.setUrl("jdbc:mysql://localhost:3306/dbName?useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC");
dataSource.setUsername("user");
dataSource.setPassword("****");
// This line added for debugging: sure enough, only 2 objects are created.
dataSource.setMaxActive(2);
}
public static Connection getConnection() throws SQLException {
return dataSource.getConnection();
}
}
Related
I'm trying to wrap my head around C3P0 and database calls. I originally had a program to run on SQLite and now I'm trying to allow concurrency to test queries on MariaDB. There are a few items I'm not grasping with this. The original design for SQLite was to have a producer thread that put queries onto a Queue and a consumer thread that would take from the queue and issue the DB query to the database.
I'm wondering if this single thread will be able to issue concurrent requests or not (since it's only one thread).
Secondly I am having an issue with this apparently not returning connections, or so it seems as such as it stops after about 18 queries. There are still items in the queue but the program just stops and waits at the try for a new connection.
My main Database calling thread class:
public class DBRunnable extends DBExtend implements Runnable
{
/**
* Call the query builder instance
*/
protected QueryBuilder qb = QueryBuilder.getInstance();
/**
* Call the point type converter instance
*/
protected PointTypeConv pv = PointTypeConv.getInstance();
/**
* Singleton object
*/
private static DBRunnable db = null;
/**
* Constructor
*/
public DBRunnable()
{
}
/**
* Main thread functionality
*/
#Override
public void run()
{
try
{
while (true)
{
long startTime = 0;
QueryRequest msg = null;
try
{
// Pull any existing query requests off the queue, if not, wait for one.
msg = (QueryRequest) DBMigrationTool.dbProcQueue.take();
} catch (Exception e)
{
errorLog.error("Unable to fetch message from message processing queue.");
}
// Good practice to create a new result set instead of reusing
ResultSet rs = null;
Statement stmt = null;
// Fetch the query and the request out of the QueryRequest object
String query = msg.getQuery();
// Make sure the query request isn't empty, if it is, there is no point in sending it to the DB
try (Connection conn = cpds.getConnection())
{
// Execute the given query and fetch the result from it
stmt = conn.createStatement();
startTime = System.currentTimeMillis();
stmt.setQueryTimeout(1800);
System.out.println(query);
stmt.execute(query);
rs = stmt.getResultSet();
if (rs != null)
{
try
{
int count = 0;
while (rs.next())
{
count++;
}
System.out.println("Query Complete: " + (System.currentTimeMillis() - startTime) + "ms. Result count: " + count);
if (msg.getFlag() == 1)
{
DBMigrationTool.flag = 0;
}
} catch (Exception e)
{
errorLog.error("Failed to process database result set.");
}
}
conn.close();
} catch (SQLException e)
{
errorLog.error("Query Error: " + msg.getQuery());
errorLog.error("Failed to issue database command: " + e);
} finally
{
if (rs != null)
{
try
{
rs.close();
} catch (SQLException e)
{
errorLog.error("Failed to close JDBC result set.");
}
}
if (stmt != null)
{
try
{
stmt.close();
} catch (SQLException e)
{
errorLog.error("Failed to close JDBC statement.");
}
}
}
}
} finally
{
closeDB();
DBMigrationTool.dbProcHandle.cancel(true);
}
}
My interface DB class that contains connection information:
public class DBExtend
{
/**
* Standard timeout
*/
public static final int DB_TIMEOUT = 30;
/**
* Standard error logger for log4j2
*/
protected static Logger errorLog = LogManager.getLogger(DBExtend.class.getName());
/**
* Call to the query builder instance
*/
private static QueryBuilder qb = QueryBuilder.getInstance();
/**
* DB connection
*/
protected static ComboPooledDataSource cpds;
/**
* Constructor
*/
public DBExtend()
{
}
/**
* startDB is an initialization function used to open a database connection
*
* #param dbPath - System path to the database file
*/
public void startDB(String dbPath)
{
cpds = new ComboPooledDataSource();
cpds.setJdbcUrl("jdbc:sqlite:" + dbPath);
cpds.setMinPoolSize(1);
cpds.setTestConnectionOnCheckout(true);
cpds.setAcquireIncrement(5);
cpds.setMaxPoolSize(20);
errorLog.info("Connection to SQLite has been established.");
}
public void startMariaDB(String tableName)
{
cpds = new ComboPooledDataSource();
cpds.setJdbcUrl("jdbc:mariadb://localhost:3306/" + tableName);
cpds.setUser("root");
cpds.setPassword("joy");
cpds.setMinPoolSize(1);
cpds.setTestConnectionOnCheckout(true);
cpds.setAcquireIncrement(5);
cpds.setMaxPoolSize(20);
errorLog.info("Connection to MariaDB has been established.");
}
/**
* Close DB is to close a database instance
*/
public void closeDB()
{
try
{
cpds.close();
errorLog.info("Connection to SQLite has been closed.");
} catch (SQLException e)
{
errorLog.error(e.getMessage());
} finally
{
try
{
if (cpds.getConnection() != null)
{
cpds.getConnection().close();
}
if (cpds != null)
{
cpds.close();
}
} catch (SQLException ex)
{
errorLog.error(ex.getMessage());
}
}
}
}
A JDBC driver is required to be thread safe, and it abstracts away the implementation details. Note that although drivers are threadsafe, it is still not a good idea to use the same connection object concurrently from multiple threads.
As to your actual problem, you are using the data source from C3P0 entirely wrong. A data source backed by a connection pool gives users a connection from this pool using the getConnection() method. This connection is returned to the pool when you close that connection.
This means that you get a connection from the pool, do your work and then close it so it is returned to the pool for use by other parts of your application.
This means that the following code in DBRunnable is wrong:
if (cpds.getConnection().isValid(DB_TIMEOUT))
You get a connection from the pool and then immediately leak it (it isn't returned to the pool) as you hold no reference to it. Note that most connection pools (sometimes optionally) do connection validation before returning a connection, so it shouldn't be necessary to test it.
Similarly for your DBExtend class, this is wrong:
In selectMariaDB:
cpds.getConnection().setCatalog(DBName);
Here you get a connection from the pool, and never close it, meaning you have 'leaked' this connection. Setting the catalog has no effect, as this connection will not be reused. Setting the catalog in this case should be part of your connection pool configuration.
In closeDB:
cpds.getConnection().close();
This obtains a connection from the pool and immediately closes it (returning it to the pool). That has no practical purpose.
I'm wondering if anyone can shed some light on this topic, as I have been racking my brain for days and can't quite understand why this does not work. I have three classes
main, RetrieveDBVersion,GetOracleConnection I've been doing some testing with oracle JDBC, UCP and Java 1.7.
According to the Oracle documentation, If I use connection pooling the connection will be returned to the pool as soon as I close the connection, Invalidate it and set it to null See Here. So I decided to give it a whirl and see if it would perform just like the documentation says it should. In my Main application I have a simple loop which makes a connection 200 times by calling RetrieveDBVersion. RetrieveDBVersion is simply performing a query and returning the driver version. My loop works fine until I hit the magic number of 68 and then I receive an error which states
java.sql.SQLException: Exception occurred while getting connection:
oracle.ucp.UniversalConnectionPoolException:
Cannot get Connection from Datasource: java.sql.SQLException:
Listener refused the connection with the following error:
ORA-12516, TNS:listener could not find available handler with matching protocol stack
These are the detail of the 3 methods. These methods are not in a server environment. They are simply calling a local oracle express database and I'm running them from my desktop. Why would I keep getting this error? If I'm returning the connections back to the pool?
Main
import com.jam.DB.JDBCVersion;
import static java.lang.System.out;
public class MainApp {
public static void main(String[] args) {
String myMainJDBCVar;
try{
for(int i=1; i<200; i++ )
{
myMainJDBCVar= JDBCVersion.RetrieveDBVersion();
out.println(myMainJDBCVar + " " + i);
}
out.println("this is Done!");
}
catch (Exception e) {
System.out.println(e.getMessage());
}
}
}
RetrieveDBVersion
import java.sql.*;
import oracle.ucp.jdbc.ValidConnection;
public class JDBCVersion {
public static String DBVersion;
public static String RetrieveDBVersion()throws SQLException {
Connection conn = JDBCConnection.GetOracleConnection("test");
try {
DatabaseMetaData meta = conn.getMetaData();
//get driver info
System.out.println("JDBC driver version is " + meta.getDriverMajorVersion());
DBVersion = meta.getDriverVersion();
} catch (SQLException e) {
e.printStackTrace();
DBVersion = e.getMessage();
}
finally {
System.out.println("hit the finally clause");
((ValidConnection) conn).setInvalid();
conn.close();
conn=null;
}
return DBVersion;
}
GetOracleConnection
import oracle.ucp.jdbc.PoolDataSource;
import oracle.ucp.jdbc.PoolDataSourceFactory;
import java.sql.*;
public class JDBCConnection {
public static Connection GetOracleConnection(String Enviroment) throws SQLException{
PoolDataSource pds = PoolDataSourceFactory.getPoolDataSource();
Connection conn = null; //ora.defaultConnection();
try {
pds.setConnectionFactoryClassName("oracle.jdbc.pool.OracleDataSource");
pds.setURL("jdbc:oracle:thin:#//localhost:1521/xe");
pds.setUser("system");
//pds.setInitialPoolSize(5);
pds.setPassword("xxx");
pds.setMaxStatements(10);
conn = pds.getConnection();
return conn;
}
catch(Exception e){
e.printStackTrace();
}
return conn;
}
So after careful though and getting a little extra help from the Oracle forum. I finally understand why the above referenced code is giving the error message that I'm receiving. See Here For Response
Because I'm setting the data source everytime the loop goes around, I'm essentially creating more than one pool. The way to do this, is create one pool and than pull connections from that pool.
New code to replace the GetOracleConnection I created a singleton class for datasource and in code I simply retrieve the connection from the data source like such
Connection conn = Database.getInstance().GetPoolSource().getConnection();
package com.jam.DB;
import oracle.ucp.jdbc.PoolDataSource;
import oracle.ucp.jdbc.PoolDataSourceFactory;
public class Database {
private static Database dbIsntance;
private static PoolDataSource pds;
private Database() {
// private constructor //
}
public static Database getInstance() {
if (dbIsntance == null) {
dbIsntance = new Database();
}
return dbIsntance;
}
public PoolDataSource GetPoolSource() {
if (pds == null) {
pds = PoolDataSourceFactory.getPoolDataSource();
try {
pds.setConnectionFactoryClassName("oracle.jdbc.pool.OracleDataSource");
pds.setURL("jdbc:oracle:thin:#//localhost:1521/xe");
pds.setUser("system");
pds.setPassword("xxxx");
pds.setMaxStatements(15);
return pds;
} catch (Exception e) {
}
return pds;
}
return pds;
}
}
I have a java based web application in which am inserting a row in aws MySQL database.
The problem is that, after 1-2 hours, the code stops inserting the rows in the database and am not getting any sort of error in my log files.
The structure of the table is as below:
Now when am calling the servlet, am using this piece of code.
JSONObject result=t_s.ro(jc.getT_conn(), t,true);
t is the json and true/false ia a boolean value according to my case.
Now inside jc.getT_conn() am using this code:
public static Connection getT_conn() throws ClassNotFoundException, JSONException {
Connection c=null;
if(t_conn==null)
{
c=rds_conn();
}
else
{
c=t_conn;
}
return c;
}
Here t_conn is a global variable for that java file and rds_conn() returns me a new connection after creating it.
Now from t_s.ro class am calling a function which inserts the row into the database based on a condition, if that's satisfied.
Here is the code:
public static boolean dPOI(Connection conn,String d,String u,ArrayList<String> l,ArrayList<String> li) throws SQLException
{
long startTime=System.currentTimeMillis();
System.out.println("Time for sql start is : "+System.currentTimeMillis());
PreparedStatement stmt = null;
boolean action=false;
try {
String sql="INSERT INTO `ce`.`cse`(`twsD`,`twsID`,`twsi`)VALUES(?,?,?)";
stmt = conn.prepareStatement(sql);
stmt.setString(1, u);
stmt.setString(2, d);
stmt.setString(3, l.toString()+"~"+li.toString());
System.out.println(stmt.toString());
action = stmt.execute();
//conn.close();
} catch (SQLException e) {
// handle sql exception
System.out.println("SQL Exception");
e.printStackTrace();
}catch (Exception e) {
// TODO: handle exception for class.forName
System.out.println("Exception");
e.printStackTrace();
}
stmt.close();
long endTime=System.currentTimeMillis();
System.out.println("Time taken inside sql Query is : "+(endTime-startTime));
return action;
}
Below is the log file which am getting.
Time for sql start is : 1486393105661
com.mysql.jdbc.JDBC42PreparedStatement#59037dda: INSERT INTO `ce`.`cse`(`twsD`,`twsID`,`twsi`)VALUES('Bana','2fdb0c926765','[\'FOM\', \'MONEY CENTER KOLA - BAORE\']~[83.80, 272.20]')
Time taken inside sql Query is : 1
Now if you can see, I am not getting any SQL exception or any other kind of exception. And moreover, the time taken is always 1 (when it stops inserting) otherwise it's somewhere between 20-25.
Moreover, thee auto increment ID always gets used up, by that what I mean is if the last row was inserted at ID 1, the subsequent query which I insert through MySQL workbench has an ID somewhere around 40 i.e if we assume that 39 of the remaining rows didn't get inserted.
Taking Mark B's comment as a starting point, I decided to create a connection pool which will be providing the connections.
Below is the code which I used:
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import org.apache.commons.dbcp2.BasicDataSource;
public final class Database {
private static final String SQL_EXIST = "show tables;";
public static void main(String[] args) throws SQLException {
// TODO Auto-generated method stub
boolean exist = false;
try (
Connection connection = Database.getConnection();
PreparedStatement statement = connection.prepareStatement(SQL_EXIST);
)
{
try (ResultSet resultSet = statement.executeQuery()) {
exist = resultSet.next();
}
}
System.out.println("Value is : "+ exist);
}
private static final BasicDataSource dataSource = new BasicDataSource();
static {
dataSource.setDriverClassName("com.mysql.jdbc.Driver");
dataSource.setUrl("CONNECTION_STRING");
dataSource.setUsername("USERNAME");
dataSource.setPassword("PASSWORD");
dataSource.setMaxTotal(100);
}
private Database() {
//
}
public static Connection getConnection() throws SQLException {
return dataSource.getConnection();
}
}
FIrst function was just for testing purpose.
Now after creating this Database class, just call Database.getConnection() whenever you need to get the connection. The connection pool will take care of providing you with a valid connection.
Correct me if am wrong.
I am trying to run few queries using a multithreaded approach, however I think I am doing something wrong because my program takes about five minute to run a simple select statement like
SELECT * FROM TABLE WHERE ID = 123'
My implementation is below and I am using one connection object.
In my run method
public void run() {
runQuery(conn, query);
}
runQuery method
public void runQuery(Connection conn, String queryString){
Statement statement;
try {
statement = conn.createStatement();
ResultSet rs = statement.executeQuery(queryString);
while (rs.next()) {}
} catch (SQLException e) {
e.printStackTrace();
}
}
Finally in the main method, I start the threads using the snippet below.
MyThread bmthread = new MyThread(conn, query);
ArrayList<Thread> allThreads = new ArrayList<>();
double start = System.currentTimeMillis();
int numberOfThreads = 1;
for(int i=0; i<=numberOfThreads; i++){
Thread th = new Thread(bmthread);
th.setName("Thread "+i);
System.out.println("Starting Worker "+th.getName());
th.start();
allThreads.add(th);
}
for(Thread t : allThreads){
try {
t.join();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
double end = System.currentTimeMillis();
double total = end - start;
System.out.println("Time taken to run threads "+ total);
Update : I am now using separate connection for each thread.
ArrayList<Connection> sqlConn = new ArrayList<>();
for(int i =0; i<10; i++){
sqlConn.add(_ut.initiateConnection(windowsAuthURL, driver));
}
loop:
MyThread bmthread = new MyThread(sqlConn.get(i), query);
As rohivats and Asaph said, one connection must be used by one and only one thread, that said, consider using a database connection pool. Taking into account that c3p0, DBCP and similars are almost abandoned, I would use HikariCP which is really fast and reliable.
If you want something very simple you could implement a really simple connection pool using a thread safe collection (such as LinkedList), for example:
public class CutrePool{
String connString;
String user;
String pwd;
static final int INITIAL_CAPACITY = 50;
LinkedList<Connection> pool = new LinkedList<Connection>();
public String getConnString() {
return connString;
}
public String getPwd() {
return pwd;
}
public String getUser() {
return user;
}
public CutrePool(String connString, String user, String pwd) throws SQLException {
this.connString = connString;
for (int i = 0; i < INITIAL_CAPACITY; i++) {
pool.add(DriverManager.getConnection(connString, user, pwd));
}
this.user = user;
this.pwd = pwd;
}
public synchronized Connection getConnection() throws SQLException {
if (pool.isEmpty()) {
pool.add(DriverManager.getConnection(connString, user, pwd));
}
return pool.pop();
}
public synchronized void returnConnection(Connection connection) {
pool.push(connection);
}
}
As you can see getConnection and returnConnection methods are synchronized to be thread safe. Get a connection (conn = pool.getConnection();) and don't forget to return/free a connection after being used (pool.returnConnection(conn);)
Don't use the same connection object in all threads. Give each thread a dedicated database connection.
One Connection can only execute one query at a time. You need multiple connections available to execute database operations in parallel. Try using a DataSource with a connection pool, and make each thread request a connection from the pool.
I have got a program this way:
public void MethodOne()
{
String sqlquery = "select * from vendor_items where category_id = 1 ";
PreparedStatement consildatedPst = connection.prepareStatement(sqlquery);
ResultSet consilatedReslset = consildatedpst.executeQuery();
while(consilatedReslset.next())
{
String name = consilatedReslset.getString("name");
if(name!=null)
{
MethodTwo();
}
}
}
public void MethodTwo(String name)
{
String sqlquery2 = "select ename from Vendor where name=?";
PreparedStatement otherPst = connection.prepareStatement(sqlquery2);
otherPst.setString(1,name);
}
This is the way connection is established (Later I will go for Connection Pooling).
public class DBConnection {
public static Connection getDBConnection() {
String sURL="jdbc:mysql://localhost:3306/oms";
String sUserName="root";
String sPwd="";
Connection conn = null;
try {
Class.forName("com.mysql.jdbc.Driver");
conn = DriverManager.getConnection(sURL, sUserName,sPwd);
return conn;
} catch (SQLException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
return conn;
}
}
My question is Can I use the same connection object when calling within Methods??
Yes, you can.
When you do:
connection.prepareStatement(sqlquery2);
It creates a new statement object using the same connection. So the ResultSets that you obtain from them will belong to different Statements and will be different and there will be NO PROBLEM for you.
In short: Different Statements manage different ResultSets. If you get 2 ResultSets from the same Statement when you get the second one the first one will be dropped but if you have 2 Statements you can manage 2 ResulSets without problem (while the connection is open, of course)
Only if you aren't using the connection in multiple threads or nesting your own methods. In other words, no. Use a new connection per method. To avoid overhead use a connection pool.