Connection pooling in multithreaded Java application - java

I am working on an application that has about 15 threads running the entire time.We recently started using HikariCP for connection pooling.
These threads are restarted every 24 hours. When the threads are restarted, we explicitly close the Hikari datasource by calling dataSource.close() Until before we started to use Connection pooling, One connection object was passed around in the thread to all functions. Now, when the dataSource is closed and if the old connection object was already passed to a method, that returned an error that said dataSource has already been closed which makes sense.
To get around this issue, instead of passing around same connection object in a thread, we started creating them in methods in DBUtils class(Basically functions with queries)
This is how run method of a thread in our application looks like:
#Override
public void run() {
consumer.subscribe(this.topics);
while (!isStopped.get()) {
try {
for (ConsumerRecord<Integer, String> record : records) {
try{
/*some code*/
}catch(JsonProcessingException ex){
ex.printStackTrace();
}
}
DBUtils.Messages(LOGGER.getName(),entryExitList);
} catch (IOException exception) {
this.interrupt();
}
consumer.close();
}
Now, after starting to use HikariCP, instead of passing connection object to DBUtils.Messages, we get a connection from the pool in the method itself
i.e
public static final void Messages(String threadName, List<EntryExit> entryExitMessages) throws SQLException {
Connection connection = DBUtils.getConnection(threadName);
/*code*/
try{
connection.close();
}catch(SQLException se){}
}
This is what getConnection method of DBUtils looks like
public static synchronized Connection getConnection(String threadName) {
Connection connection = null;
try {
if (ds == null || ds.isClosed()) {
config.setJdbcUrl(getProperty("postgres.url"));
config.setUsername(getProperty("postgres.username"));
config.setPassword(getProperty("postgres.password"));
config.setDriverClassName(getProperty("postgres.driver"));
config.setMaximumPoolSize(getProperty("postgres.max-pool-size"));
config.setMetricRegistry(ApplicationUtils.getMetricRegistry());
config.setConnectionTimeout(getProperty("postgres.connection-timeout"));
config.setLeakDetectionThreshold(getProperty("postgres.leak-detection-threshold"));
config.setIdleTimeout(getProperty("postgres.idle-timeout"));
config.setMaxLifetime(getProperty("postgres.max-lifetime"));
config.setValidationTimeout(getProperty("postgres.validation-timeout"));
config.setMinimumIdle(getProperty("postgres.minimum-idle"));
config.setPoolName("PostgresConnectionPool");
ds = new HikariDataSource(config);
}
connection = ds.getConnection();
return connection;
} catch (Exception exception) {
exception.printStackTrace();
}
}
But since call to this method is inside while loop in the thread, PostgresConnectionPool.pool.Wait keeps increasing.
.What's the best way to deal with this?
Edit: PostgresConnection is the pool name . PoolPostgresConnectionPool.pool.Wait is coming from Dropwizard metrics :
https://github.com/brettwooldridge/HikariCP/wiki/Dropwizard-Metrics

Related

Does try-with resource on db connection pool close the connection each time?

I'm using HikariDataSource for managing the connection pool to my Postgres DB.
I'm using try with a resource for getting the connection from HikariDataSource and i want to understand the following:
Is the connection is really close each time?
If yes - There are no pros of working with prepared statements this way?
What is the best practice for using prepared statements with a connection pool?
Here is my connection code:
public <T> CompletableFuture<T> withConnection(FunctionThatThrowsChecedException<Connection, T> action) {
return CompletableFuture.supplyAsync(() -> {
try (Connection connection = ds.getConnection()) {
return action.apply(connection);
} catch (SQLException | IOException e) {
throw new RuntimeException("error while getting collection", e);
}
}, workerThreads);
}
Here some example of executing query with a prepared statement:
public CompletableFuture<Integer> delete(String batchId) {
return postgresProvider.withConnection(connection -> {
PreparedStatement ps = connection.prepareStatement(DELETE_QUERY);
ps.setString(1, batchId);
return ps.executeUpdate();
});
}
No, the connection is returned to the pool when close() is called. The connection pool's Connection wrapper overrides close() that way so the pool can work.
It's not closed.
Just like you normally do. Using a connection pool rarely requires anything special, and if you intend to tune the pool you need to measure performance and actually know what you're trying to do.

If using Hibernate with Database (and c3p0 dbpooling) is is still possible and safe to directly get connection bypassing Hibernate

Using Hibernate 4.3.11 with H2 1.4.199 and C3p0
If using Hibernate with Database is is still possible and safe to directly get connection from pool bypassing Hibernate?
Im trying to write an sql query, now I know I can use session.createSQLQuery() but this returns a Hibernate org.hibernate SQLQuery class rather than a java.sql.Connection and this is causing a problem for me trying to set the parameters, so I wanted to try using a Connection instead.
Try this.
import org.hibernate.engine.jdbc.connections.spi.ConnectionProvider;
sessionFactory.getSessionFactoryOptions().getServiceRegistry().getService(ConnectionProvider.class).getConnection();
I have not tested this and cannot do it at this time. I am only 50-50 on the ConnectionProvider, since I am not sure what the provider class for c3p0. May be org.hibernate.c3p0.internal.C3P0ConnectionProvider or you already know it.
EDIT 11-10-2019
The connections are returned from the implementations of org.hibernate.engine.jdbc.connections.spi.ConnectionProvider interface, as far as I can see.
org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl
org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl
Both these implementations are fetching a java.sql.Connection from Datasource or PooledConnections that are internally maintained at the time of service registry.
DatasourceConnectionProviderImpl has the following methods.
#Override
public Connection getConnection() throws SQLException {
if ( !available ) {
throw new HibernateException( "Provider is closed!" );
}
return useCredentials ? dataSource.getConnection( user, pass ) : dataSource.getConnection();
}
#Override
public void closeConnection(Connection connection) throws SQLException {
connection.close();
}
DriverManagerConnectionProviderImpl methods follow
#Override
public Connection getConnection() throws SQLException {
if ( !active ) {
throw new HibernateException( "Connection pool is no longer active" );
}
return pool.poll();
}
public Connection poll() throws SQLException {
Connection conn = availableConnections.poll();
if ( conn == null ) {
synchronized (allConnections) {
if(allConnections.size() < maxSize) {
addConnections( 1 );
return poll();
}
}
throw new HibernateException( "The internal connection pool has reached its maximum size and no connection is currently available!" );
}
conn.setAutoCommit( autoCommit );
return conn;
}
#Override
public void closeConnection(Connection conn) throws SQLException {
if (conn == null) {
return;
}
pool.add( conn );
}
And as you can see, they both have different ways of handling connections. I would not imagine major issues if you close the connection, since the first one returns new connection and the second one replenishes the pool if the connection is closed. However, if you are not going to close, know which implementation is getting invoked in your application and make sure you do not have a leak.
The other two implementations are for Hikari and UserSupplied mode.

create another connection in transaction mysql

I want know if it is possibile create two connection different connections in one transaction, So suppose to have this code( I do a create operation in table "employe" and I create a log in table "log"):
try {
InitialContext initialContext = new InitialContext();
DataSource datasource = (DataSource) initialContext.lookup("java:/comp/env/jdbc/postgres");
Connection connection_db= datasource.getConnection();
PreparateStatement p1 //Preparate statement to put the employe parameter
connessione_db.setAutoCommit(false);
connessione_db.setTransactionIsolation(Connection.TRANSACTION_SERIALIZABLE);
p1.execute();
//This create another connect
LOGStatic.createLog("CREATE EMPLOYE");
connessione_db.commit();
connessione_db.setAutoCommit(true);
}catch(....){
connection_db.rollback();
}
This is the method LOGStatic.createLog("CREATE EMPLOYE");
public static void creaLogException(String messaggio) {
Connection connection_db= null;
InitialContext initialContext;
try {
initialContext = new InitialContext();
DataSource datasource = (DataSource) initialContext.lookup("java:/comp/env/jdbc/postgres");
connection_db= datasource.getConnection();
// the code continues with the save operation
} catch (NamingException n) {
}
// actual jndi name is "jdbc/postgres"
catch (SQLException s) {
}
My question if is it possible this behavior and there aren't problem with commit and rollback operatio or it is not possible to have another connection?
Anyone can help
At least in JDBC, the transaction is tightly coupled with the Connection.
In a Java EE server, if you are writing session beans, then the transaction will be managed my the server. So, in that case, you could call several methods, and the transaction will follow the method calls.
In JDBC the simple solution is to not close the connection, but to pass it around to different methods as an input parameter.
But, be very careful, not closing connections is almost a sure shot way of getting to OutOfMemoryError.
(Instead of passing around the connection as input parameters to various methods, you could use ThreadLocal. But that is another source of memory leak, if you don't clean up ThreadLocal variables.)
For more information on how transaction-lifecycle and Connection are tied in JDBC, refer this: How to start a transaction in JDBC?
Note: Even in JavaEE nested transaction is not possible as nested transactions is not supported in JTA.
Passing the connection around:
public static void createEmployee(){
InitialContext initialContext = new InitialContext();
DataSource datasource = (DataSource) initialContext.lookup("java:/comp/env/jdbc/postgres");
Connection connection_db= datasource.getConnection();
try {
connessione_db.setAutoCommit(false);
connessione_db.setTransactionIsolation(Connection.TRANSACTION_SERIALIZABLE);
PreparateStatement p1 //Preparate statement to put the employe parameter
p1.execute();
//This create another connect
createLog("CREATE EMPLOYE", connessione_db);
connessione_db.commit();
//connessione_db.setAutoCommit(true); //No need
}catch(....){
try{ connection_db.rollback(); }catch(Exception e){ /*You can also check some flags to avoid exception*/ }
}finally{
try{ connection_db.close(); }catch(Exception e){ /*Safe to ignore*/ }
}
}
public static void createLog(String messaggio, Connection connection_db) {
try {
// the code continues with the save operation
} catch (SQLException s) {
}
}

The last packet sent successfully to the server was 79,547 milliseconds ago. is longer than the server configured value of 'wait_timeout'

I have a big problem and I do not know how to fix it:
I have a singleton instance of the database as follow:
public Connection getConnection() throws SQLException {
if (db_con == null)
db_con = createConnection();
return db_con;
}
And I have a code that as follow:
shortTextScoringComponent.scoreComponent( "RS",SelectDataBase.getBlogs(rightSarcastic));
shortTextScoringComponent.scoreComponent( "RNS",SelectDataBase.getBlogs(rightNonSarcasm));
shortTextScoringComponent.scoreComponent( "FNS",SelectDataBase.getBlogs(wrongNonSarcasm));
shortTextScoringComponent.scoreComponent( "FS",SelectDataBase.getBlogs(wrongSarcasm));
So as you can see I call the database 4 times and it is note worthy that between each call there is a long time of processing so after he second line is performed successfuly I mean this line:
SelectDataBase.getBlogs(rightNonSarcasm);
when it comes to the third line I get the following error:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 79,547,948 milliseconds ago. The last packet sent successfully to the server was 79,547,964 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
I searched a lot but there are many different answers which confuses me, do you have any idea what my exact problem is?
First of all as the exception says do add
autoReconnect=true
in your connectionString also add this as well
tcpKeepAlive=true
Secondly you can keep a polling thread to keep checking connection activeness
class PollingThread extends Thread
{
private java.sql.Connection connection;
PollingThread(int index, java.sql.Connection connection)
{
super();
this.connection = connection;
}
public void run()
{
Statement stmt = null;
while (!interrupted())
{
try
{
sleep(15 * 60 * 1000L);
stmt = connection.createStatement();
stmt.execute("do 1");
}
catch (InterruptedException e)
{
break;
}
catch (Exception sqle)
{
/* This thread should run even on DB error. If autoReconnect is true,
* the connection will reconnect when the DB comes back up. */
}
finally
{
if (stmt != null)
{
try
{
stmt.close();
} catch (Exception e)
{}
}
stmt = null;
}
}

Java Threads and MySQL

I have a threaded chat server application which requires MySQL authencation.
Is the best way to have 1 class create the MySQL connection, keep that connection open and let every thread use that connection but use own Query handler?
Or is it better to have all threads make a seperate connection to MySQL to authencate?
Or is it better to let 1 class handle the queries AND connections?
We are looking at a chatserver that should be able to handle upto 10.000 connections/users.
I am now using c3p0, and I created this:
public static void main(String[] args) throws PropertyVetoException
{
ComboPooledDataSource pool = new ComboPooledDataSource();
pool.setDriverClass("com.mysql.jdbc.Driver");
pool.setJdbcUrl("jdbc:mysql://localhost:3306/db");
pool.setUser("root");
pool.setPassword("pw");
pool.setMaxPoolSize(100);
pool.setMinPoolSize(10);
Database database = new Database(pool);
try
{
ResultSet rs = database.query("SELECT * FROM `users`");
while (rs.next()) {
System.out.println(rs.getString("userid"));
System.out.println(rs.getString("username"));
}
}
catch(Exception ex)
{
System.out.println(ex.getMessage());
}
finally
{
database.close();
}
}
public class Database {
ComboPooledDataSource pool;
Connection conn;
ResultSet rs = null;
Statement st = null;
public Database (ComboPooledDataSource p_pool)
{
pool = p_pool;
}
public ResultSet query (String _query)
{
try {
conn = pool.getConnection();
st = conn.createStatement();
rs = st.executeQuery(_query);
} catch (SQLException e) {
e.printStackTrace();
} finally {
}
return rs;
}
public void close ()
{
try {
st.close();
conn.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
}
Would this be thread safe?
c3p0 connection pool is a robust solution. You can also check dbcp but c3p0 shows better performance, supports auto-reconnection and some other features.
Have you looked at connection pooling ? Check out (for example) Apache DBCP or C3P0.
Briefly, connection pooling means that a pool of authenticated connections are used, and free connections are passed to you on request. You can configure the number of connections as appropriate. When you close a connection, it's actually returned to the pool and made available for another client. It makes life relatively easy in your scenario, since the pool looks after the authentication and connection management.
You should not have just one connection. It's not a thread-safe class. The idea is to get a connection, use it, and close it in the narrowest scope possible.
Yes, you'll need a pool of them. Every Java EE app server will have a JNDI pooling mechanism for you. I wouldn't recommend one class for all queries, either. Your chat ap
Your chat app ought to have a few sensible objects in its domain model. I'd create data access objects for them as appropriate. Keep the queries related to a particular domain model object in its DAO.
is the info in this thread up-to-date? Googling brings up a lot of different things, as well as this - http://dev.mysql.com/tech-resources/articles/connection_pooling_with_connectorj.html

Categories