We Keep Coding

Connection pooling in multithreaded Java application

I am working on an application that has about 15 threads running the entire time.We recently started using HikariCP for connection pooling.
These threads are restarted every 24 hours. When the threads are restarted, we explicitly close the Hikari datasource by calling dataSource.close() Until before we started to use Connection pooling, One connection object was passed around in the thread to all functions. Now, when the dataSource is closed and if the old connection object was already passed to a method, that returned an error that said dataSource has already been closed which makes sense.
To get around this issue, instead of passing around same connection object in a thread, we started creating them in methods in DBUtils class(Basically functions with queries)
This is how run method of a thread in our application looks like:
#Override
public void run() {
consumer.subscribe(this.topics);
while (!isStopped.get()) {
try {
for (ConsumerRecord<Integer, String> record : records) {
try{
/*some code*/
}catch(JsonProcessingException ex){
ex.printStackTrace();
}
}
DBUtils.Messages(LOGGER.getName(),entryExitList);
} catch (IOException exception) {
this.interrupt();
}
consumer.close();
}
Now, after starting to use HikariCP, instead of passing connection object to DBUtils.Messages, we get a connection from the pool in the method itself
i.e
public static final void Messages(String threadName, List<EntryExit> entryExitMessages) throws SQLException {
Connection connection = DBUtils.getConnection(threadName);
/*code*/
try{
connection.close();
}catch(SQLException se){}
}
This is what getConnection method of DBUtils looks like
public static synchronized Connection getConnection(String threadName) {
Connection connection = null;
try {
if (ds == null || ds.isClosed()) {
config.setJdbcUrl(getProperty("postgres.url"));
config.setUsername(getProperty("postgres.username"));
config.setPassword(getProperty("postgres.password"));
config.setDriverClassName(getProperty("postgres.driver"));
config.setMaximumPoolSize(getProperty("postgres.max-pool-size"));
config.setMetricRegistry(ApplicationUtils.getMetricRegistry());
config.setConnectionTimeout(getProperty("postgres.connection-timeout"));
config.setLeakDetectionThreshold(getProperty("postgres.leak-detection-threshold"));
config.setIdleTimeout(getProperty("postgres.idle-timeout"));
config.setMaxLifetime(getProperty("postgres.max-lifetime"));
config.setValidationTimeout(getProperty("postgres.validation-timeout"));
config.setMinimumIdle(getProperty("postgres.minimum-idle"));
config.setPoolName("PostgresConnectionPool");
ds = new HikariDataSource(config);
}
connection = ds.getConnection();
return connection;
} catch (Exception exception) {
exception.printStackTrace();
}
}
But since call to this method is inside while loop in the thread, PostgresConnectionPool.pool.Wait keeps increasing.
.What's the best way to deal with this?
Edit: PostgresConnection is the pool name . PoolPostgresConnectionPool.pool.Wait is coming from Dropwizard metrics :
https://github.com/brettwooldridge/HikariCP/wiki/Dropwizard-Metrics

Should I open a JDBC connection for each rest call?

I am dealing with high traffic in my Spring Boot project and my goal is serving clients as much fast as possible. In this case, I have more than 500 requests per second. In each rest endpoint call, I should connect my schema and gather multiple information from multiple tables. To be able to do that, should I create new connection for each eendpoint call or create & close before each db query?
I wrote a JDBC connection class but I am not sure that it is a good way. Maybe you can give me some opinion.
JDBC Connection Class
#PropertySource({"classpath:application.properties"})
#Configuration
public class FraudJDBConfiguration {
private final Logger LOGGER = LogManager.getLogger(FraudJDBConfiguration.class);
private final Environment env;
#Autowired
public FraudJDBConfiguration(Environment env) {
this.env = env;
}
#Bean
public Connection getFraudConnection() {
// Step 1: Loading or
// registering Oracle JDBC driver class
String connectionClass = env.getProperty("fraud.db.driver-class-name");
try {
Class.forName(connectionClass);
} catch (ClassNotFoundException cnfex) {
LOGGER.error(cnfex.getMessage());
throw new RuntimeException("JDBC driver class'ı bulunamadı");
}
// Step 2: Opening database connection
try {
String environmentType = env.getProperty("environment");
if (environmentType == null) {
LOGGER.error("environment Tip Hatası (TEST - UAT - LIVE)");
throw new RuntimeException("environment Tip Hatası (TEST - UAT - LIVE)");
} else {
String connectionString = null;
String username = null;
String password = null;
switch (environmentType.toLowerCase()) {
case "dev":
connectionString = env.getProperty(/*someurl*/);
username = env.getProperty(/*someusername*/);
password = env.getProperty(/*somepassword*/);
break;
case "tst":
connectionString = env.getProperty(/*someurl*/);
username = env.getProperty(/*someusername*/);
password = env.getProperty(/*somepassword*/);
break;
case "liv":
connectionString = env.getProperty(/*someurl*/);
username = env.getProperty(/*someusername*/);
password = env.getProperty(/*somepassword*/);
break;
case "uat":
connectionString = env.getProperty(/*someurl*/);
username = env.getProperty(/*someusername*/);
password = env.getProperty(/*somepassword*/);
break;
}
// Step 2.A: Create and
// get connection using DriverManager class
if (connectionString == null) {
LOGGER.error("fraud şeması için connection string bulunamadı");
throw new RuntimeException("fraud şeması için connection string bulunamadı");
}
return DriverManager.getConnection(connectionString, username, password);
}
} catch (SQLException e) {
LOGGER.error(e.getMessage());
}
return null;
}
}
DAO
#Component
public interface FraudCommTransactionsDao {
Long count();
}
DAO IMPL
#Service
public class FraudCommTransactionsDaoImpl implements FraudCommTransactionsDao {
private final FraudJDBConfiguration fraudJDBConfiguration;
#Autowired
public FraudCommTransactionsDaoImpl(FraudJDBConfiguration fraudJDBConfiguration) {
this.fraudJDBConfiguration = fraudJDBConfiguration;
}
#Override
public Long count() {
try(Connection connection = fraudJDBConfiguration.getFraudConnection()) {
Statement stmt = connection.createStatement();
ResultSet rs = stmt.executeQuery(/*some query*/);
if (rs.next()) {
return rs.getLong("transaction_id");
} else {
return 0L;
}
} catch (SQLException ex) {
ex.printStackTrace();
}
return null;
}
}

No, establishing a new physical connection to a database server is costly. It involves multiple steps: user authorization, establishing session defaults, allocating memory on both client and server, etc. This overhead should not be added to every single request.
It's a common practice to create a connection pool to share the physical connections between application threads. This introduces a concept of logical connections e.g. a Connection object created with DriverManager.getConnection() is a physical connection while DataSource.getConnection() returns a logical connection which is a proxy.
There are multiple database connection pooling libraries for Java that you can use e.g. HikariCP. Don't write your own, this is not simple.

Get fast data and deliver to client could be possible using the simplest way of using application.properties file. You may use this to get database connection to your datasource.

How to create plugin openfire for crud

I'm very new in openfire and first time using java, I got stuck when I trying to develop plugin for crud. Could you give me some sample to make crud plugin ability? Thanks for your help before...

You can start from this answer: Mapping Openfire Custom plugin with aSmack Client
and follow the official tutorial with first 3 points of the answer.
About CRUD:
Let's assume you want to audit all your messages as XML in your database, so you'll implement a PacketInterceptor just to keep an easy scenario.
Your class plugin will looks like:
public class MyCustomPlugin implements Plugin, PacketInterceptor {//foo}
in method initializePlugin you'll have an invokation like:
public void initializePlugin(PluginManager manager, File pluginDirectory)
{
InterceptorManager.getInstance().addInterceptor(this);
}
and in method interceptPacket something like that:
#Override
public void interceptPacket(Packet packet, Session session,
boolean incoming, boolean processed) throws PacketRejectedException {
if (!processed)
{
boolean done = doMyCRUDAction(packet);
}
if (!done)
{ //do something if error occourred}
}
now let's write on database:
private static final String AUDIT_CHAT =
"INSERT INTO MYTABLE(MESSAGEASXML) VALUES (?)";
private boolean doMyCRUDAction(Packet packet)
{
if ((packet instanceof Message))
{
Message message = (Message) packet.createCopy();
boolean isAudited = false;
Connection con = null;
PreparedStatement statement = null;
try {
con = DbConnectionManager.getConnection();
statement = con.prepareStatement(AUDIT_CHAT);
statement.setString(1, message.toString());
statement.executeQuery();
isAudited = true;
}
catch (SQLException e) {
Log.error(e.getMessage(), e);
}
catch (Exception ex)
{
Log.error(ex.getMessage(), ex);
}
finally {
DbConnectionManager.closeConnection(statement, con);
}
return isAudited;
}
}
please keep in mind this is a reduced snippet of a working code, so there can be some sintax to fix
If your CRUD must follow an explicit IQ request, you'll have to extends an IQHandler and create a custom IQ and send to the client in handleIQ(IQ packet) method. You can check in Openfire sourcecode about detailed and complex implementations.

Java JDBC efficiency: How long should a connection be maintained?

I'm still working on the same problem mention here. It seems to work fine especially after creating an AbstractModel class shown below:
public abstract class AbstractModel {
protected static Connection myConnection = SingletonConnection.instance().establishConnection();
protected static Statement stmt;
protected static ResultSet rs;
protected boolean loginCheck; // if userId and userLoginHistoryId are valid - true, else false
protected boolean userLoggedIn; // if user is already logged in - true, else false
public AbstractModel (int userId, Long userLoginHistoryId){
createConnection(); // establish connection
loginCheck = false;
userLoggedIn = false;
if (userId == 0 && userLoginHistoryId == 0){ // special case for login
loginCheck = true; // 0, 0, false, false
userLoggedIn = false; // set loginCheck to true, userLogged in to false
} else {
userLoggedIn = true;
try{
String query = "select \"user_login_session_check\"(" + userId + ", " + userLoginHistoryId + ");";
System.out.println("query: " + query);
stmt = myConnection.createStatement();
rs = stmt.executeQuery(query);
while (rs.next()){
loginCheck = rs.getBoolean(1);
}
} catch (SQLException e){
System.out.println("SQL Exception: ");
e.printStackTrace();
}
}
}
// close connection
public void closeConnection(){
try{
myConnection.close();
} catch (SQLException e){
System.out.println("SQL Exception: ");
e.printStackTrace();
}
}
// establish connection
public void createConnection(){
myConnection = SingletonConnection.instance().establishConnection();
}
// login session check
public boolean expiredLoginCheck (){
if (loginCheck == false && userLoggedIn == true){
closeConnection();
return false;
} else {
return true;
}
}
}
I've already posted the stored procedures and Singleton Pattern implementation in the link to the earlier question above.
I'm under the impression that I don't need to close the connection to the database after each single data transaction, as it would just slow the application. I'm looking at about 30 users for this system I'm building, so performance and usability is important.
Is it correct to prolong the connection for at least 3-4 data transactions? Eg. Validation checks to user inputs for some form, or, something similar to google's auto-suggest ... These are all separate stored function calls based on user input. Can I use one connection instance, instead of connecting and disconnecting after each data transaction? Which is more efficient?
If my assumptions are correct (more efficient to use one connection instance) then opening and closing of the connection should be handled in the controller, which is why I created the createConnection() and closeConnection() methods.
Thanks.

Your code should never depend on the fact, that your application is currently the only client to the database or that you have only 30 users. So you should handle database connections like files, sockets and all other kinds of scarce resources that you may run ouf of.
Thus you should always clean up after yourself. No matter what you do. Open connection, do your stuff (one or SQL statements) and close connection. Always!
In your code you create your connection and save it into a static variable - this connection will last as long as your AbstractModel class lives, probably forever - this is bad. As with all similar cases put you code inside try/finally to make sure the connection gets always closed.
I have seen application servers running ouf of connections because of web applications not closing connections. Or because they closed at logout and somebody said "we will never have more then that much users at the same time" but it just scaled a little to high.
Now as you have your code running and closing the connections properly add connection pooling, like zaske said. This will remedy the performance problem of opening/closing database connection, which truely is costly. On the logical layer (your application) you doesn't want to know when to open/close physical connection, the db layer (db pool) will handle it for you.
Then you can even go and set up a single connection for your whole session model, which is also supported by DBCP - this is no danger, since you can reconfigure the pool afterwards if you need without touching your client code.

Like Tomasz said, you should never ever depend on the fact that your application will be used by a small number of clients. The fact that the driver will timeout after a certain amount of time does not guarantee you that you will have enough available connections. Picture this: a lot of databases come pre-configured with a maximum number of connections set to (say) 15 and a timeout of (let's say) 10-15 minutes. If you have 30 clients and each does an operation, somewhere around half-way you'll be stuck short on connections.
You should handle connections, files, streams and other resources the following way:
public void doSomething()
{
Connection connection = null;
Statement stmt = null;
ResultSet rs = null;
final String sql = "SELECT ....");
try
{
connection = getConnection();
stmt = connection.createStatement();
rs = stmt.executeQuery(sql);
if (rs.next())
{
// Do something here...
}
}
catch (SQLException e)
{
e.printStackTrace();
}
finally
{
closeResultSet(rs);
closeStatement(stmt);
closeConnection(connection);
}
}
The try/catch/finally guarantees you that the connection will get closed no matter the outcome. If there is some sort of failure, the finally block will still close the connection, just like it would do, if things were okay.
Similarly, with file and streams you need to do the same thing. Initialize the respective object as null outside your try/catch/finally, then follow the approach above.
This misconception makes a lot of Java applications misbehave under Windows, where people don't close files (streams to files, etc) and these files become locked, forcing you to either kill the JVM, or even restart your machine.
You can also use a connection pool such as for example Apache's DBCP, but even then you should take care of closing your resources, despite the fact that internally, the different connection pool implementations do not necessarily close the connections.

You'are right that you don't need to close the connection after each call.
Bare in mind that that modern database implement internal connection pools, but your application still need to connect and retrieve a connection object, and this is what it does now.
You should consider using a database connection pool - there are various Java frameworks to provide you such a solution, and they will define (you will be able to configure of course) when a database connection pool is closed.
In general - you should ask yourself whether your database serves only your application, or does it serve other application as well - if it does not serve other application as well, you might be able to be more "greedy" and keep connections open for a longer time.
I would also recommend that your application will create on start a fixed number of connections (define it in your configuration with a value of "Minimum connections number") and you will let it grow if needed to a maximum connection numbers.
As I previously mentioned - the ideas are suggest are implemented already by all kinds of frameworks, for example - the DBCP project of Apache.

Here is the Singleton Pattern which I initialize the myConenction field in all my Models to:
public class DatabaseConnection {
private static final String uname = "*******";
private static final String pword = "*******";
private static final String url = "*******************************";
Connection connection;
// load jdbc driver
public DatabaseConnection(){
try{
Class.forName("org.postgresql.Driver");
establishConnection();
} catch (ClassNotFoundException ce) {
System.out.println("Could not load jdbc Driver: ");
ce.printStackTrace();
}
}
public Connection establishConnection() {
// TODO Auto-generated method stub
try{
connection = DriverManager.getConnection(url, uname, pword);
} catch (SQLException e){
System.out.println("Could not connect to database: ");
e.printStackTrace();
}
return connection;
}
}
public class SingletonConnection {
private static DatabaseConnection con;
public SingletonConnection(){}
public static DatabaseConnection instance(){
assert con == null;
con = new DatabaseConnection();
return con;
}
}
Of course each and every connection to the database from the app goes through a Model.

Java memory leak when running on Red Hat but no memory leak on Mac OS X

I have a java webobjects app which is showing memory leak problems when running on Red Hat but we had no such problems when it was running on Mac OS X. The JVMs are similar.
Mac OS X 10.6.5 using java 1.6.0_22 64 bit from Apple
Red Hat EL 5.0 using java 1.6.0_20 64 bit from Sun
I configured it to do a heap dump when it ran out of memory, and analysing this with the eclipse memory analyzer tool suggests that the problem is in a part of the code which creates a thread which sends an HTTP Request to a web service. The reason for creating the thread is to implement a timeout on the request because the web service is sometimes not available.
Does anyone have any ideas?
WOHTTPConnection connection = new WOHTTPConnection(host, port);
WORequest request = new WORequest(strMethod, strQuery, strHttpVersion, nsdHeader, content, null);
WebServiceRequester theRequester = new WebServiceRequester(connection, request);
Thread requestThread = new Thread(theRequester);
requestThread.start();
try {
requestThread.join(intTimeoutSend); //timeout in milliseconds = 10000
if ( requestThread.isAlive() ) {
requestThread.interrupt();
}
} catch(InterruptedException e) {
}
requestThread = null;
if(!theRequester.getTfSent()) {
return null;
}
WOResponse response = connection.readResponse();
...
class WebServiceRequester implements Runnable {
private WORequest theRequest;
private WOHTTPConnection theConnection;
private boolean tfSent = false;
public WebServiceRequester(WOHTTPConnection c, WORequest r) {
theConnection = c;
theRequest = r;
}
public void run() {
tfSent = theConnection.sendRequest(theRequest);
}
public boolean getTfSent() {
return tfSent;
}
}
EDIT: leaked class names as reported by eclipse memory analyzer tool:
1,296 instances of "java.lang.Thread", loaded by "<system class loader>" occupy 111,947,632 (43.21%) bytes.
1,292 instances of "er.extensions.eof.ERXEC", loaded by "java.net.URLClassLoader # 0x2aaab375b7c0" occupy 37,478,352 (14.46%) bytes.
1,280 instances of "er.extensions.appserver.ERXRequest", loaded by "java.net.URLClassLoader # 0x2aaab375b7c0" occupy 27,297,992 (10.54%) bytes.

I have heard that WOHTTPConnection is broken and shouldn't be used. WOHTTPConnection does not give you a reliable way of closing the connection. It is also unreliable in other ways.
The solution is to rewrite the code using Apache HttpComponents HttpClient

I think the problem is that Thread.interrupt does not actually stop your Thread. And the JVM will never clean up the thread if its running.
I would add a closeConnection method to your thread and try calling that instead of or in addition to your Thread.interrupt call. You might need to modify it a bit, but the idea is to explicitly stop the IO that is keeping the Thread running:
WOHTTPConnection connection = new WOHTTPConnection(host, port);
WORequest request = new WORequest(strMethod, strQuery, strHttpVersion, nsdHeader, content, null);
WebServiceRequester theRequester = new WebServiceRequester(connection, request);
Thread requestThread = new Thread(theRequester);
requestThread.start();
try {
requestThread.join(intTimeoutSend); //timeout in milliseconds = 10000
if ( requestThread.isAlive() ) {
requestThread.closeConnection();
requestThread.interrupt();
}
} catch(InterruptedException e) {
}
requestThread = null;
if(!theRequester.getTfSent()) {
return null;
}
WOResponse response = connection.readResponse();
...
class WebServiceRequester implements Runnable {
private WORequest theRequest;
private WOHTTPConnection theConnection;
private boolean tfSent = false;
public WebServiceRequester(WOHTTPConnection c, WORequest r) {
theConnection = c;
theRequest = r;
}
public void run() {
tfSent = theConnection.sendRequest(theRequest);
}
public boolean getTfSent() {
return tfSent;
}
public void closeConnection() {
this.theConnection.close();
}
}

Do you need to close the WOHTTPConnection handle? (I'm not familiar with that API ...).
FOLLOWUP
Checked into it, looks like the connection.readResponse() closes the connection so I don't need to do it manually.
#Rudiger - you are assuming that the call to connection.readResponse() always succeeds. What if the problem is that it is throwing an exception that is not getting reported. (The default behavior is to silently ignore errors thrown on child threads.)
I think you should close the connection handle in a finally block ... just in case.
Or better still, ditch WOHTTPConnection entirely.