In a javase database application, I process a lot of short-lived object (say accounting documents like bills etc.). Processing each object consists in opening a connection to a database and looking up for some data. Not all objects are looked up on the same database, but i select a specific database according to some object property, so i'll end up having several connections opened.
What i actually need is no more than one connection for each database.
So I've done something like this:
public MyPool {
Map<String, Connection> activeConnections = new TreeMap<String, Connection>();
public Connection getConnection(String database_name) throws SQLException {
if (activeConnections.containsKey(database_name)) {
return activeConnections.get(database_name);
}
//Retrive the configuration data from a configuration object
ConnectionConfig c = Configuration.getConnectionConfig(database_name);
Connection connection = DriverManager.getConnection(c.url, c.user, c.password);
return connection;
}
The questions are:
1) Since I see around a lot of pooling libraries, DBCP, c3p0 and others: what is the point of all those libraries, what do they add to a "basic" approach like this?
Tutorials like this doesn't help much in responding this question, since the basic solution exposed here fits perfectly their definition of connection pooling.
2) This is a piece of code that will be "exposed" to other developers, that in turn may develop procedures to retrieve data from databases with different structures, probably getting connections from this "pool object".
Is it correct, in the docs and in the code, referring to it as a "pool", or is it something different, so that calling "pool" would be misleading?
Your code isn't a connection pool implementation in the colloquial use of the term since each datasource only has a single physical connection. The concept behind object pooling (in this case, the object is a connection) is that some objects require overhead to configure. In the case of a connection pool, as you know, a database connection must be opened before you can talk to the database.
The difference here is that your code isn't thread safe for a concurrent environment like the popular connection pool implementations you've mentioned. Applications running in high concurrency circumstances like the web shouldn't need to absorb the overhead of establishing a connection on each request. Instead, a pool of open connections is maintained and when the request has finished working on the connection, it is returned to the pool for subsequent requests to make use of.
This is required because connections are stateful. You can't have multiple requests sharing the same connection at the same time and guarantee any sort of reasonable transaction semantics.
Use BoneCP and wrap the connection pool like this:
Do not try to create your own connection pool, that is what BoneCP or any number of other very good and well tested pools are for.
object ConnectionPool {
Class.forName("[ENTER DRIVER]")
private val connstring = [ENTER YOUR STRING]
private var cp : BoneCP = _
createConnectionPool() //upon init create the cp
/**
* Create a new connection pool
*/
def createConnectionPool() = {
if(cp == null) {
try {
val config = new BoneCPConfig()
config.setJdbcUrl(connstring)
config.setMaxConnectionsPerPartition(3)
config.setMinConnectionsPerPartition(1)
config.setPartitionCount(1)
cp = new BoneCP(config)
}
catch {
case e: SQLException => e.printStackTrace()
}
}
}
def getConnection () = { cp.getConnection }
Related
I am implementing a program that needs to establish connection to mysql database. I'm currently connecting independently for each class (like sign up, sign in, show users etc), using the following lines:
Class.forName("com.mysql.cj.jdbc.Driver");
con = DriverManager.getConnection("jdbc:mysql://localhost:3308/myfirstdb","root");
What I am thinking is to implement a class for the connection and call its default constructor each time. Does it give any advantages/disadvantages or doesn't matter. This is the class:
public class MyConnection {
private Connection con;
public MyConnection() {
//establishing connection
try {
Class.forName("com.mysql.cj.jdbc.Driver");
con = DriverManager.getConnection("jdbc:mysql://localhost:3308/myfirstdb","root","");
} catch (ClassNotFoundException | SQLException e) {
e.printStackTrace();
}
}
//for future use
public Connection getCon() {
return con;
}
}
With very rare exceptions, a single connection to the database for the entire program is the "correct" thing to do.
Connecting costs something. All your queries can use that one connection.
Sure, have the "constructor" do the "connect". And have the "destructor" do "disconnect".
The class should be a "singleton" (or equivalent) to avoid accidentally getting two connections.
Your code has public MyConnection(), which begs the user to create multiple connections. That should be private and called only once, such as this way:
public Connection getCon() {
if (con == NULL) { // or whatever the syntax is
MyConnection(); // Here's the _one_ connection
}
return con;
}
Then users call getCon() to get the one connection and use it to perform queries.
Beware: If you language facilitates multiple threads, do one of these:
Do database operations only from one thread, or
Have (at most) one connection per thread.
PS: I believe this advice applies to any OO language.
As a footnote, establishing two connections in an attempt to get extra performance is likely to be futile. Or, at least, not worth the effort.
Another thing to be aware of... HTTP is, mostly, stateless. So, if you have one web page to "sign up", that will come and go -- one connection with some number of SQLs, then it goes away. Another web page to "sign in" will involve another HTTP request, another connection, etc.
After that, the memory that the user is "signed in" need to be held somewhere:
In URL parameters -- subject to hacking
In a cookie -- reasonable for lightweight apps
Other -- You need security advice if working with sensitive (credit card, health care, etc) info and need to go from web page to web page.
This is a solved problem in Java, don't try to write this yourself. Use a javax.sql.DataSource implementation that is backed by a connection pool (for example Apache DBCP, HikariCP or c3p0).
You can then obtain a connection from the data source for a unit-of-work, and close the connection when you're done. Closing the connection will return the connection to the connection pool for re-use, eliminating much of the overhead of opening a connection.
I'm creating app for library management with Java and MySQL ( JDBC to connect with DB ) , and I have a problem , I checked a lot of topics, books, and websites but I didn't find good answer for me. Is it the good way to deal with connections ? I think that one connection for entire app is good option in this case. My idea is that in every function in every class when I need to use Connection object , these functions will need a connection parameter. In main class I'll call manager object 'Man' for example and to every constructor etc I'll pass Man.getMyConn() as this parameter and call Man.close() when Main frame will be closed . Is it bad idea ? Maybe I should use singleton pattern or connection pool ?
Sorry for my English , I'm still learning.
public class manager {
private Connection myConn;
public manager() throws Exception {
Properties props = new Properties();
props.load(new FileInputStream("app.properties"));
String user = props.getProperty("user");
String password = props.getProperty("password");
String dburl = props.getProperty("dburl");
myConn = DriverManager.getConnection(dburl, user, password);
System.out.println("DB connection successful to: " + dburl);
}
public Connection getMyConn() {
return myConn;
}
//close class etc.
}
Usually not. Further answer depends on type of the application. If you're making web application then you should definitely go with connection pool. If you're making e.g. desktop application (where only one user can access it at the time), then you can open and close connection upon each request.
I have working applications that do it your way. As #Branislav says, it's not adequate if you want to do multiple concurrent queries. There's also a danger that the connection to the database might be lost, and you would need to restart your application to get a new one, unless you write code to catch that and recreate the connection.
Using a singleton would be overcomplicated. Having a getConnection() method (as you have done) is very important as it means you can easily change your code to use a pool later if you find you need to.
My server app uses prepared statements in almost all cases, to prevent sql injection. Nevertheless a possibility is needed providing special users executing raw SELECT queries.
How can I more or less securely make sure the query does not modify the database? Is it possible to execute a query read only, or is there any other 'secure' way making sure noone tries any sql injection?
(Using sqlite3, so I cannot use any privileges)
Thanks a lot!
JDBC supports read-only connections by calling Connection.setReadOnly(true). However the javadoc says:
Puts this connection in read-only mode as a hint to the driver to enable database optimizations.
Some JDBC drivers will enforce the read-only request, others will use it for optimizations only, or simply ignore it. I don't know how sqlite3 implements it. You'll have to test that.
Otherwise, you could do a "simple" parse of the SQL statement, to ensure that it's a single valid SELECT statement.
I'm not aware of a general JBDC configuration which specifies readonly. But Sqlite does have special database open modes and this can be leveraged in your connection to your sqlite database. Eg.
Properties config = new Properties();
config.setProperty("open_mode", "1"); //1 == readonly
Connection conn = DriverManager.getConnection("jdbc:sqlite:sample.db", config);
Credit: https://stackoverflow.com/a/18092761/62344
FWIW All supported open modes can be seen here.
If you use some sort of factory class to create or return connections to the database, you can individually set connections to be read-only:
public Connection getReadOnlyConnection() {
// Alternatively this could come from a connection pool:
final Connection conn = DriverManager.getConnection("jdbc:sqlite:sample.db");
conn.setReadOnly(true);
return conn;
}
If you're using a connection pool, then you may also want to provide a method for getting writeable connections too:
public Connection getWriteableConnection() {
final Connection conn = getPooledConnection(); // I'm assuming this method exists!
conn.setReadOnly(false);
return conn;
}
You could also provide just a single getConnection(boolean readOnly) method and simply pass the parameter through to the setReadOnly(boolean) call. I prefer the separate methods personally, as it makes your intent much clearer.
Alternatively, some databases like Oracle provide a read only mode that can be enabled. SQLite doesn't provide one, but you can emulate it by simply setting the actual database files (including directories) to read only on the filesystem itself.
Another way of doing it is as follows (credit goes to deadlock for the below code):
public Connection getReadOnlyConnection() {
SQLiteConfig config = new SQLiteConfig();
config.setReadOnly(true);
Connection conn = DriverManager.getConnection("jdbc:sqlite:sample.db",
config.toProperties());
}
Can anybody provide examples or links on how to establish a JDBC connection pool?
From searching google I see many different ways of doing this and it is rather confusing.
Ultimately I need the code to return a java.sql.Connection object, but I am having trouble getting started..any suggestions welcome.
Update: Doesn't javax.sql or java.sql have pooled connection implementations? Why wouldn't it be best to use these?
If you need a standalone connection pool, my preference goes to C3P0 over DBCP (that I've mentioned in this previous answer), I just had too much problems with DBCP under heavy load. Using C3P0 is dead simple. From the documentation:
ComboPooledDataSource cpds = new ComboPooledDataSource();
cpds.setDriverClass( "org.postgresql.Driver" ); //loads the jdbc driver
cpds.setJdbcUrl( "jdbc:postgresql://localhost/testdb" );
cpds.setUser("swaldman");
cpds.setPassword("test-password");
// the settings below are optional -- c3p0 can work with defaults
cpds.setMinPoolSize(5);
cpds.setAcquireIncrement(5);
cpds.setMaxPoolSize(20);
// The DataSource cpds is now a fully configured and usable pooled DataSource
But if you are running inside an application server, I would recommend to use the built-in connection pool it provides. In that case, you'll need to configure it (refer to the documentation of your application server) and to retrieve a DataSource via JNDI:
DataSource ds = (DataSource) new InitialContext().lookup("jdbc/myDS");
HikariCP
It's modern, it's fast, it's simple. I use it for every new project.
I prefer it a lot over C3P0, don't know the other pools too well.
Usually if you need a connection pool you are writing an application that runs in some managed environment, that is you are running inside an application server. If this is the case be sure to check what connection pooling facilities your application server providesbefore trying any other options.
The out-of-the box solution will be the best integrated with the rest of the application servers facilities. If however you are not running inside an application server I would recommend the Apache Commons DBCP Component. It is widely used and provides all the basic pooling functionality most applications require.
Don't reinvent the wheel.
Try one of the readily available 3rd party components:
Apache DBCP - This one is
used internally by Tomcat, and by
yours truly.
c3p0
Apache DBCP comes with different example on how to setup a pooling javax.sql.DataSource. Here is one sample that can help you get started.
I would recommend using the commons-dbcp library. There are numerous examples listed on how to use it, here is the link to the move simple one. The usage is very simple:
BasicDataSource ds = new BasicDataSource();
ds.setDriverClassName("oracle.jdbc.driver.OracleDriver")
ds.setUsername("scott");
ds.setPassword("tiger");
ds.setUrl(connectURI);
...
Connection conn = ds.getConnection();
You only need to create the data source once, so make sure you read the documentation if you do not know how to do that. If you are not aware of how to properly write JDBC statements so you do not leak resources, you also might want to read this Wikipedia page.
In the app server we use where I work (Oracle Application Server 10g, as I recall), pooling is handled by the app server. We retrieve a javax.sql.DataSource using a JNDI lookup with a javax.sql.InitialContext.
it's done something like this
try {
context = new InitialContext();
jdbcURL = (DataSource) context.lookup("jdbc/CachedDS");
System.out.println("Obtained Cached Data Source ");
}
catch(NamingException e)
{
System.err.println("Error looking up Data Source from Factory: "+e.getMessage());
}
(We didn't write this code, it's copied from this documentation.)
Pool
Pooling Mechanism is the way of creating the Objects in advance. When a class is loaded.
It improves the application performance [By re using same object's to perform any action on Object-Data] & memory [allocating and de-allocating many objects creates a significant memory management overhead].
Object clean-up is not required as we are using same Object, reducing the Garbage collection load.
« Pooling [ Object pool, String Constant Pool, Thread Pool, Connection pool]
String Constant pool
String literal pool maintains only one copy of each distinct string value. which must be immutable.
When the intern method is invoked, it check object availability with same content in pool using equals method.
« If String-copy is available in the Pool then returns the reference.
« Otherwise, String object is added to the pool and returns the reference.
Example: String to verify Unique Object from pool.
public class StringPoolTest {
public static void main(String[] args) { // Integer.valueOf(), String.equals()
String eol = System.getProperty("line.separator"); //java7 System.lineSeparator();
String s1 = "Yash".intern();
System.out.format("Val:%s Hash:%s SYS:%s "+eol, s1, s1.hashCode(), System.identityHashCode(s1));
String s2 = "Yas"+"h".intern();
System.out.format("Val:%s Hash:%s SYS:%s "+eol, s2, s2.hashCode(), System.identityHashCode(s2));
String s3 = "Yas".intern()+"h".intern();
System.out.format("Val:%s Hash:%s SYS:%s "+eol, s3, s3.hashCode(), System.identityHashCode(s3));
String s4 = "Yas"+"h";
System.out.format("Val:%s Hash:%s SYS:%s "+eol, s4, s4.hashCode(), System.identityHashCode(s4));
}
}
Connection pool using Type-4 Driver using 3rd party libraries[ DBCP2, c3p0, Tomcat JDBC]
Type 4 - The Thin driver converts JDBC calls directly into the vendor-specific database protocol Ex[Oracle - Thick, MySQL - Quora]. wiki
In Connection pool mechanism, when the class is loaded it get's the physical JDBC connection objects and provides a wrapped physical connection object to user. PoolableConnection is a wrapper around the actual connection.
getConnection() pick one of the free wrapped-connection form the connection objectpool and returns it.
close() instead of closing it returns the wrapped-connection back to pool.
Example: Using ~ DBCP2 Connection Pool with Java 7[try-with-resources]
public class ConnectionPool {
static final BasicDataSource ds_dbcp2 = new BasicDataSource();
static final ComboPooledDataSource ds_c3p0 = new ComboPooledDataSource();
static final DataSource ds_JDBC = new DataSource();
static Properties prop = new Properties();
static {
try {
prop.load(ConnectionPool.class.getClassLoader().getResourceAsStream("connectionpool.properties"));
ds_dbcp2.setDriverClassName( prop.getProperty("DriverClass") );
ds_dbcp2.setUrl( prop.getProperty("URL") );
ds_dbcp2.setUsername( prop.getProperty("UserName") );
ds_dbcp2.setPassword( prop.getProperty("Password") );
ds_dbcp2.setInitialSize( 5 );
ds_c3p0.setDriverClass( prop.getProperty("DriverClass") );
ds_c3p0.setJdbcUrl( prop.getProperty("URL") );
ds_c3p0.setUser( prop.getProperty("UserName") );
ds_c3p0.setPassword( prop.getProperty("Password") );
ds_c3p0.setMinPoolSize(5);
ds_c3p0.setAcquireIncrement(5);
ds_c3p0.setMaxPoolSize(20);
PoolProperties pool = new PoolProperties();
pool.setUrl( prop.getProperty("URL") );
pool.setDriverClassName( prop.getProperty("DriverClass") );
pool.setUsername( prop.getProperty("UserName") );
pool.setPassword( prop.getProperty("Password") );
pool.setValidationQuery("SELECT 1");// SELECT 1(mysql) select 1 from dual(oracle)
pool.setInitialSize(5);
pool.setMaxActive(3);
ds_JDBC.setPoolProperties( pool );
} catch (IOException e) { e.printStackTrace();
} catch (PropertyVetoException e) { e.printStackTrace(); }
}
public static Connection getDBCP2Connection() throws SQLException {
return ds_dbcp2.getConnection();
}
public static Connection getc3p0Connection() throws SQLException {
return ds_c3p0.getConnection();
}
public static Connection getJDBCConnection() throws SQLException {
return ds_JDBC.getConnection();
}
}
public static boolean exists(String UserName, String Password ) throws SQLException {
boolean exist = false;
String SQL_EXIST = "SELECT * FROM users WHERE username=? AND password=?";
try ( Connection connection = ConnectionPool.getDBCP2Connection();
PreparedStatement pstmt = connection.prepareStatement(SQL_EXIST); ) {
pstmt.setString(1, UserName );
pstmt.setString(2, Password );
try (ResultSet resultSet = pstmt.executeQuery()) {
exist = resultSet.next(); // Note that you should not return a ResultSet here.
}
}
System.out.println("User : "+exist);
return exist;
}
jdbc:<DB>:<drivertype>:<HOST>:<TCP/IP PORT>:<dataBaseName>
jdbc:oracle:thin:#localhost:1521:myDBName
jdbc:mysql://localhost:3306/myDBName
connectionpool.properties
URL : jdbc:mysql://localhost:3306/myDBName
DriverClass : com.mysql.jdbc.Driver
UserName : root
Password :
Web Application: To avoid connection problem when all the connection's are closed[MySQL "wait_timeout" default 8 hours] in-order to reopen the connection with underlying DB.
You can do this to Test Every Connection by setting testOnBorrow = true and validationQuery= "SELECT 1" and donot use autoReconnect for MySQL server as it is deprecated. issue
===== ===== context.xml ===== =====
<?xml version="1.0" encoding="UTF-8"?>
<!-- The contents of this file will be loaded for a web application -->
<Context>
<Resource name="jdbc/MyAppDB" auth="Container"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
type="javax.sql.DataSource"
initialSize="5" minIdle="5" maxActive="15" maxIdle="10"
testWhileIdle="true"
timeBetweenEvictionRunsMillis="30000"
testOnBorrow="true"
validationQuery="SELECT 1"
validationInterval="30000"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/myDBName"
username="yash" password="777"
/>
</Context>
===== ===== web.xml ===== =====
<resource-ref>
<description>DB Connection</description>
<res-ref-name>jdbc/MyAppDB</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
</resource-ref>
===== ===== DBOperations ===== =====
servlet « init() {}
Normal call used by sevlet « static {}
static DataSource ds;
static {
try {
Context ctx=new InitialContext();
Context envContext = (Context)ctx.lookup("java:comp/env");
ds = (DataSource) envContext.lookup("jdbc/MyAppDB");
} catch (NamingException e) { e.printStackTrace(); }
}
See these also:
am-i-using-jdbc-connection-pooling
configuring-jdbc-pool-high-concurrency
In late 2017 Proxool, BoneCP, C3P0, DBCP are mostly defunct at this time. HikariCP (created in 2012) seems promising, blows the doors off anything else I know of.
http://www.baeldung.com/hikaricp
Proxool has a number of issues:
- Under heavy load can exceed max number of connections and not return below max
- Can manage to not return to min connections even after connections expire
- Can lock up the entire pool (and all server/client threads) if it has trouble connecting to the database during HouseKeeper thread (does not use .setQueryTimeout)
- HouseKeeper thread, while having connection pool lock for its process, requests the Prototyper thread to recreate connections (sweep) which can result in race condition/lockup. In these method calls the last parameter should always be sweep:false during the loop, only sweep:true below it.
- HouseKeeper only needs the single PrototypeController sweep at the end and has more [mentioned above]
- HouseKeeper thread checks for testing of connections before seeing what connections may be expired [some risk of testing expired connection that may be broken/terminated through other timeouts to DB in firewall, etc.]
- The project has unfinished code (properties that are defined but not acted upon)
- The Default max connection life if not defined is 4 hours (excessive)
- HouseKeeper thread runs every five seconds per pool (excessive)
You can modify the code and make these improvements. But as it was created in 2003, and updated in 2008, its lacking nearly 10 years of java improvements that solutions like hikaricp utilize.
As answered by others, you will probably be happy with Apache Dbcp or c3p0. Both are popular, and work fine.
Regarding your doubt
Doesn't javax.sql or java.sql have
pooled connection implementations? Why
wouldn't it be best to use these?
They don't provide implementations, rather interfaces and some support classes, only revelant to the programmers that implement third party libraries (pools or drivers). Normally you don't even look at that. Your code should deal with the connections from your pool just as they were "plain" connections, in a transparent way.
Vibur DBCP is another library for that purpose. Several examples showing how to configure it for use with Hibernate, Spring+Hibernate, or programatically, can be found on its website: http://www.vibur.org/
Also, see the disclaimer here.
Apache Commons has a library for that purpose: DBCP. Unless you have strange requirements around your pools, I'd use a library as it's bound to be trickier and more subtle than you would hope.
You should consider using UCP.
Universal Connection Pool (UCP) is a Java connection pool. It is a features rich connection pool and tightly integrated with Oracle's Real Application Clusters (RAC), ADG, DG databases.
Refer to this page for more details about UCP.
MiniConnectionPoolManager is a one-java-file implementation, if you're looking for an embeddable solution and are not too concerned about performances (though I haven't tested it in that regard).
It is multi-licensed EPL, LGPL and MPL.
Its documentation also gives alternatives worth checking (on top of DBCP and C3P0):
Proxool
BoneCP
Tomcat JDBC Connection Pool
I have create a getDBConnection method in my Java application. This returns a connection object, and hence I haven't closed this connection in this method itself.
Now, I am invoking this method from various methods in my application at regular intervals, and closing them inside a try - finally block. I thought this should free up the connection after use. However, I am seeing a large number of connections opened (about 50) in the MySQL Administrator's Server Connections tab.
//Defining a method to retrieve a database connection
// PropDemo is a properties class that retrieves Database related values from a file
public Connection getDBConnection() {
//Instantiating the Properties object
PropDemo prop = new PropDemo();
Connection con = null;
// Retrieving values from the parameters.properties file
String JdbcDriver = prop.getMessage("JdbcDriver");
String JdbcUrlPrefix = prop.getMessage("JdbcUrlPrefix");
String DBIP = prop.getMessage("DBIP");
String DBName = prop.getMessage("DBName");
String DBUser = prop.getMessage("DBUser");
String DBPassword = prop.getMessage("DBPassword");
try {
// Loading and instantiating the JDBC MySQL connector driver class
Class.forName(JdbcDriver).newInstance();
con = DriverManager.getConnection(JdbcUrlPrefix + DBIP + "/" + DBName, DBUser, DBPassword);
if (con.isClosed())
Logger.log("Connection cannot be established", "vm");
} catch (Exception e) {
Logger.log("Exception: " + e, "vm");
Logger.log(Logger.stack2string(e), "vm");
}
return con;
}
I am also closing the associated ResultSet and Statement Objects. What could be missing here?
I am planning to replace all the Statements with PreparedStatements for efficiency and security reasons. Will that help significantly? What else can be done?
EDIT:
This is just a core java application that is repeatedly quering for changes in some fields in a MySQL database through MySQL-JDBC connector. I am not using any framework like Spring or Hibernate.
Your code looks sane.
That's how you're creating a new connection.
Probably the error is where you close it.
You should close it in a finally block.
Some additional questions.
1) Are you sure those 50 conections come from this program ? Maybe there are some others comming from your same office. To confirm this you would need to stop the program, and look again in your connection monitor.
2) Does your application uses many connection simultaneously? Probably its a peak when you're using 50 at the same time.
If you can post the code where you close the connection. Chances are the problem is there.
Additionally I would suggest you to use a connection pool. You can build one your self or you can see the results from this page:
How many JDBC connections in Java?
Are you closing the connection object when you application closes as well?
Are you using your JDBC connection within a J2EE application server or with Hibernate?
Both of these tend to start out with a fairly high connection pool to begin with, so you would see a large number.
Check out the details on connection pooling.
You could take a Singleton approach to the problem and only create a new Connection object if the current one is null:
If (connectionObject != null){
return connectionObject;
}else {
//create new connection object
}
This will make sure that you only have one non-null connection at any time.