I have an java-spring web application, which will read, write and delete information from a user upload SQLite DB. I am using JDBCtemplate to set connection, query the DB and update the information.
I observed one behavior during my tests:
Every time,after users uploaded a new SQLite db file(it will has the same name, place at the same directory as the old DB file), if they do not reboot/restart tomcat, jdbcquery will report the db was corrupted exception.
To me this looked like the JDBCtemplate somehow cached the connection and is trying to resume the connection with the old db?
If so, do you know anyway to refresh the connection without rebooting the application?
final SingleConnectionDataSource dataSource = new singleConnectionDataSource();
try {
Class.forName("org.sqlite.JDBC");
} catch (ClassNotFoundException e) {
throw new applicationException(MessageTemplateNames.GENERAL_UNKNOWN_ERROR, e);
}
createDirectoryForDbIfNotExists();
dataSource.setUrl(String.format("%s%s", JDBC.PREFIX, getDbFileLocation()));
dataSource.setAutoCommit(true);
JDBCTemplate does not handle connection.It obtains the connection from the datasource set to it.
From the reference documentation for SingleConnectionDataSource
Implementation of SmartDataSource that wraps a single JDBC Connection
which is not closed after use. .....
This is primarily intended for testing. For example, it enables easy
testing outside an application server, for code that expects to work
on a DataSource. In contrast to DriverManagerDataSource, it reuses the
same Connection all the time, avoiding excessive creation of physical
Connections.
A DriverManagerDataSource will suite your requirement to get a new connection without reboot
Simple implementation of the standard JDBC DataSource interface,
configuring the plain old JDBC DriverManager via bean properties, and
returning a new Connection from every getConnection call.
Update
My answer might not solve your original problem , but will only answer the question for new connection without reboot. Please go through #Thilo's comment on the same.
To pretense, i'm new to web development. I have a java web application that is deployed through Jetty, and I have an issue that is really confusing me. In order to display data from my database in the web app I must establish a database connection, which I can do when I am unit testing my code, but when I call the same methods from .jsp pages to populate the web app I get a message telling me this:
java.sql.SQLException: No suitable driver found for jdbc:sqlserver://localhost:1433;integratedSecurity=true;database=TicketITBookingSystemDatabase;loginTimeout=30;
Here is the rest of the relevant code:
public void queryDatabase(String query){
ResultSet resultSet = null;
String connectionUrl = establishDatabaseConnection();
try (Connection connection = DriverManager.getConnection(connectionUrl);
Statement statement = connection.createStatement();) {
//Execute the sql statement passed as the query parameter.
resultSet = statement.executeQuery(query);
processResultSet(resultSet);
statement.close();
}
// Handle any errors that may have occurred.
catch (SQLException e) {
e.printStackTrace(); //why is this being hit but only on the web build
}
And the event that is being called:
public List<String[]> getAllEvents(){
queryDatabase("EXEC dbo.sp_GetAllEventDetails");
return events;
}
Thanks.
You have to add the driver of the database on the classpath of Jetty.
Check if it is present or not.
The problem is that either you did not add the driver jar to the classpath of Jetty, or - if you deployed it with the application - you need to load the driver explicitly with Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver"). JDBC automatic driver loading only works when the driver is on the initial classpath, not if it is on a context classpath.
However, it is generally inadvisable to use DriverManager directly from a web application. The use of a DataSource is usually better, especially if the data source implementation provides a connection pool.
I have found this method in a Java file and I am wondering what's happening here? Does this really work? I think this method does nothing because the getConnection()-method creates a new connection and then it gets closed. Am I correct?
public void closeAllConnections()
{
for(String db : this.dbList)
try {
DataSource ds = (DataSource) this.applicationContext.getBean(db+"DataSource");
ds.getConnection().close();
} catch (Exception e) {
//...
}
}
It depends how your DataSource is defined.
If it uses connection pooling it can reuse existing connection
for example see oracle explanation
Connection pooling in the JDBC 2.0 extension API is a framework for
caching database connections. This allows reuse of physical
connections and reduced overhead for your application.
Can anybody provide examples or links on how to establish a JDBC connection pool?
From searching google I see many different ways of doing this and it is rather confusing.
Ultimately I need the code to return a java.sql.Connection object, but I am having trouble getting started..any suggestions welcome.
Update: Doesn't javax.sql or java.sql have pooled connection implementations? Why wouldn't it be best to use these?
If you need a standalone connection pool, my preference goes to C3P0 over DBCP (that I've mentioned in this previous answer), I just had too much problems with DBCP under heavy load. Using C3P0 is dead simple. From the documentation:
ComboPooledDataSource cpds = new ComboPooledDataSource();
cpds.setDriverClass( "org.postgresql.Driver" ); //loads the jdbc driver
cpds.setJdbcUrl( "jdbc:postgresql://localhost/testdb" );
cpds.setUser("swaldman");
cpds.setPassword("test-password");
// the settings below are optional -- c3p0 can work with defaults
cpds.setMinPoolSize(5);
cpds.setAcquireIncrement(5);
cpds.setMaxPoolSize(20);
// The DataSource cpds is now a fully configured and usable pooled DataSource
But if you are running inside an application server, I would recommend to use the built-in connection pool it provides. In that case, you'll need to configure it (refer to the documentation of your application server) and to retrieve a DataSource via JNDI:
DataSource ds = (DataSource) new InitialContext().lookup("jdbc/myDS");
HikariCP
It's modern, it's fast, it's simple. I use it for every new project.
I prefer it a lot over C3P0, don't know the other pools too well.
Usually if you need a connection pool you are writing an application that runs in some managed environment, that is you are running inside an application server. If this is the case be sure to check what connection pooling facilities your application server providesbefore trying any other options.
The out-of-the box solution will be the best integrated with the rest of the application servers facilities. If however you are not running inside an application server I would recommend the Apache Commons DBCP Component. It is widely used and provides all the basic pooling functionality most applications require.
Don't reinvent the wheel.
Try one of the readily available 3rd party components:
Apache DBCP - This one is
used internally by Tomcat, and by
yours truly.
c3p0
Apache DBCP comes with different example on how to setup a pooling javax.sql.DataSource. Here is one sample that can help you get started.
I would recommend using the commons-dbcp library. There are numerous examples listed on how to use it, here is the link to the move simple one. The usage is very simple:
BasicDataSource ds = new BasicDataSource();
ds.setDriverClassName("oracle.jdbc.driver.OracleDriver")
ds.setUsername("scott");
ds.setPassword("tiger");
ds.setUrl(connectURI);
...
Connection conn = ds.getConnection();
You only need to create the data source once, so make sure you read the documentation if you do not know how to do that. If you are not aware of how to properly write JDBC statements so you do not leak resources, you also might want to read this Wikipedia page.
In the app server we use where I work (Oracle Application Server 10g, as I recall), pooling is handled by the app server. We retrieve a javax.sql.DataSource using a JNDI lookup with a javax.sql.InitialContext.
it's done something like this
try {
context = new InitialContext();
jdbcURL = (DataSource) context.lookup("jdbc/CachedDS");
System.out.println("Obtained Cached Data Source ");
}
catch(NamingException e)
{
System.err.println("Error looking up Data Source from Factory: "+e.getMessage());
}
(We didn't write this code, it's copied from this documentation.)
Pool
Pooling Mechanism is the way of creating the Objects in advance. When a class is loaded.
It improves the application performance [By re using same object's to perform any action on Object-Data] & memory [allocating and de-allocating many objects creates a significant memory management overhead].
Object clean-up is not required as we are using same Object, reducing the Garbage collection load.
« Pooling [ Object pool, String Constant Pool, Thread Pool, Connection pool]
String Constant pool
String literal pool maintains only one copy of each distinct string value. which must be immutable.
When the intern method is invoked, it check object availability with same content in pool using equals method.
« If String-copy is available in the Pool then returns the reference.
« Otherwise, String object is added to the pool and returns the reference.
Example: String to verify Unique Object from pool.
public class StringPoolTest {
public static void main(String[] args) { // Integer.valueOf(), String.equals()
String eol = System.getProperty("line.separator"); //java7 System.lineSeparator();
String s1 = "Yash".intern();
System.out.format("Val:%s Hash:%s SYS:%s "+eol, s1, s1.hashCode(), System.identityHashCode(s1));
String s2 = "Yas"+"h".intern();
System.out.format("Val:%s Hash:%s SYS:%s "+eol, s2, s2.hashCode(), System.identityHashCode(s2));
String s3 = "Yas".intern()+"h".intern();
System.out.format("Val:%s Hash:%s SYS:%s "+eol, s3, s3.hashCode(), System.identityHashCode(s3));
String s4 = "Yas"+"h";
System.out.format("Val:%s Hash:%s SYS:%s "+eol, s4, s4.hashCode(), System.identityHashCode(s4));
}
}
Connection pool using Type-4 Driver using 3rd party libraries[ DBCP2, c3p0, Tomcat JDBC]
Type 4 - The Thin driver converts JDBC calls directly into the vendor-specific database protocol Ex[Oracle - Thick, MySQL - Quora]. wiki
In Connection pool mechanism, when the class is loaded it get's the physical JDBC connection objects and provides a wrapped physical connection object to user. PoolableConnection is a wrapper around the actual connection.
getConnection() pick one of the free wrapped-connection form the connection objectpool and returns it.
close() instead of closing it returns the wrapped-connection back to pool.
Example: Using ~ DBCP2 Connection Pool with Java 7[try-with-resources]
public class ConnectionPool {
static final BasicDataSource ds_dbcp2 = new BasicDataSource();
static final ComboPooledDataSource ds_c3p0 = new ComboPooledDataSource();
static final DataSource ds_JDBC = new DataSource();
static Properties prop = new Properties();
static {
try {
prop.load(ConnectionPool.class.getClassLoader().getResourceAsStream("connectionpool.properties"));
ds_dbcp2.setDriverClassName( prop.getProperty("DriverClass") );
ds_dbcp2.setUrl( prop.getProperty("URL") );
ds_dbcp2.setUsername( prop.getProperty("UserName") );
ds_dbcp2.setPassword( prop.getProperty("Password") );
ds_dbcp2.setInitialSize( 5 );
ds_c3p0.setDriverClass( prop.getProperty("DriverClass") );
ds_c3p0.setJdbcUrl( prop.getProperty("URL") );
ds_c3p0.setUser( prop.getProperty("UserName") );
ds_c3p0.setPassword( prop.getProperty("Password") );
ds_c3p0.setMinPoolSize(5);
ds_c3p0.setAcquireIncrement(5);
ds_c3p0.setMaxPoolSize(20);
PoolProperties pool = new PoolProperties();
pool.setUrl( prop.getProperty("URL") );
pool.setDriverClassName( prop.getProperty("DriverClass") );
pool.setUsername( prop.getProperty("UserName") );
pool.setPassword( prop.getProperty("Password") );
pool.setValidationQuery("SELECT 1");// SELECT 1(mysql) select 1 from dual(oracle)
pool.setInitialSize(5);
pool.setMaxActive(3);
ds_JDBC.setPoolProperties( pool );
} catch (IOException e) { e.printStackTrace();
} catch (PropertyVetoException e) { e.printStackTrace(); }
}
public static Connection getDBCP2Connection() throws SQLException {
return ds_dbcp2.getConnection();
}
public static Connection getc3p0Connection() throws SQLException {
return ds_c3p0.getConnection();
}
public static Connection getJDBCConnection() throws SQLException {
return ds_JDBC.getConnection();
}
}
public static boolean exists(String UserName, String Password ) throws SQLException {
boolean exist = false;
String SQL_EXIST = "SELECT * FROM users WHERE username=? AND password=?";
try ( Connection connection = ConnectionPool.getDBCP2Connection();
PreparedStatement pstmt = connection.prepareStatement(SQL_EXIST); ) {
pstmt.setString(1, UserName );
pstmt.setString(2, Password );
try (ResultSet resultSet = pstmt.executeQuery()) {
exist = resultSet.next(); // Note that you should not return a ResultSet here.
}
}
System.out.println("User : "+exist);
return exist;
}
jdbc:<DB>:<drivertype>:<HOST>:<TCP/IP PORT>:<dataBaseName>
jdbc:oracle:thin:#localhost:1521:myDBName
jdbc:mysql://localhost:3306/myDBName
connectionpool.properties
URL : jdbc:mysql://localhost:3306/myDBName
DriverClass : com.mysql.jdbc.Driver
UserName : root
Password :
Web Application: To avoid connection problem when all the connection's are closed[MySQL "wait_timeout" default 8 hours] in-order to reopen the connection with underlying DB.
You can do this to Test Every Connection by setting testOnBorrow = true and validationQuery= "SELECT 1" and donot use autoReconnect for MySQL server as it is deprecated. issue
===== ===== context.xml ===== =====
<?xml version="1.0" encoding="UTF-8"?>
<!-- The contents of this file will be loaded for a web application -->
<Context>
<Resource name="jdbc/MyAppDB" auth="Container"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
type="javax.sql.DataSource"
initialSize="5" minIdle="5" maxActive="15" maxIdle="10"
testWhileIdle="true"
timeBetweenEvictionRunsMillis="30000"
testOnBorrow="true"
validationQuery="SELECT 1"
validationInterval="30000"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/myDBName"
username="yash" password="777"
/>
</Context>
===== ===== web.xml ===== =====
<resource-ref>
<description>DB Connection</description>
<res-ref-name>jdbc/MyAppDB</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
</resource-ref>
===== ===== DBOperations ===== =====
servlet « init() {}
Normal call used by sevlet « static {}
static DataSource ds;
static {
try {
Context ctx=new InitialContext();
Context envContext = (Context)ctx.lookup("java:comp/env");
ds = (DataSource) envContext.lookup("jdbc/MyAppDB");
} catch (NamingException e) { e.printStackTrace(); }
}
See these also:
am-i-using-jdbc-connection-pooling
configuring-jdbc-pool-high-concurrency
In late 2017 Proxool, BoneCP, C3P0, DBCP are mostly defunct at this time. HikariCP (created in 2012) seems promising, blows the doors off anything else I know of.
http://www.baeldung.com/hikaricp
Proxool has a number of issues:
- Under heavy load can exceed max number of connections and not return below max
- Can manage to not return to min connections even after connections expire
- Can lock up the entire pool (and all server/client threads) if it has trouble connecting to the database during HouseKeeper thread (does not use .setQueryTimeout)
- HouseKeeper thread, while having connection pool lock for its process, requests the Prototyper thread to recreate connections (sweep) which can result in race condition/lockup. In these method calls the last parameter should always be sweep:false during the loop, only sweep:true below it.
- HouseKeeper only needs the single PrototypeController sweep at the end and has more [mentioned above]
- HouseKeeper thread checks for testing of connections before seeing what connections may be expired [some risk of testing expired connection that may be broken/terminated through other timeouts to DB in firewall, etc.]
- The project has unfinished code (properties that are defined but not acted upon)
- The Default max connection life if not defined is 4 hours (excessive)
- HouseKeeper thread runs every five seconds per pool (excessive)
You can modify the code and make these improvements. But as it was created in 2003, and updated in 2008, its lacking nearly 10 years of java improvements that solutions like hikaricp utilize.
As answered by others, you will probably be happy with Apache Dbcp or c3p0. Both are popular, and work fine.
Regarding your doubt
Doesn't javax.sql or java.sql have
pooled connection implementations? Why
wouldn't it be best to use these?
They don't provide implementations, rather interfaces and some support classes, only revelant to the programmers that implement third party libraries (pools or drivers). Normally you don't even look at that. Your code should deal with the connections from your pool just as they were "plain" connections, in a transparent way.
Vibur DBCP is another library for that purpose. Several examples showing how to configure it for use with Hibernate, Spring+Hibernate, or programatically, can be found on its website: http://www.vibur.org/
Also, see the disclaimer here.
Apache Commons has a library for that purpose: DBCP. Unless you have strange requirements around your pools, I'd use a library as it's bound to be trickier and more subtle than you would hope.
You should consider using UCP.
Universal Connection Pool (UCP) is a Java connection pool. It is a features rich connection pool and tightly integrated with Oracle's Real Application Clusters (RAC), ADG, DG databases.
Refer to this page for more details about UCP.
MiniConnectionPoolManager is a one-java-file implementation, if you're looking for an embeddable solution and are not too concerned about performances (though I haven't tested it in that regard).
It is multi-licensed EPL, LGPL and MPL.
Its documentation also gives alternatives worth checking (on top of DBCP and C3P0):
Proxool
BoneCP
Tomcat JDBC Connection Pool
I recently wrote and deployed a Java web application to a server and I'm finding an unusual problem which didn't appear during development or testing.
When a user logs in after so long and goes to display data from the database, the page indicates that there are no records to see. But upon page refresh, the first x records are shown according to the pagination rules.
Checking the logs, I find:
ERROR|19 09 2009|09 28 54|http-8080-4|myDataSharer.database_access.Database_Metadata_DBA| - Error getting types of columns of tabular Dataset 12
com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:
** BEGIN NESTED EXCEPTION **
java.io.EOFException
STACKTRACE:
java.io.EOFException
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1956)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2368)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2867)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1616)
And so on for several hundred lines.
The application is currently set for about 100 users but is not yet in full use. It uses connection pooling between the Apache Tomcat servlets / jsps and a MySQL database with the following code example forming the general arrangement of a database operation, of which there are typically several per page:
// Gets a Dataset.
public static Dataset getDataset(int DatasetNo) {
ConnectionPool_DBA pool = ConnectionPool_DBA.getInstance();
Connection connection = pool.getConnection();
PreparedStatement ps = null;
ResultSet rs = null;
String query = ("SELECT * " +
"FROM Dataset " +
"WHERE DatasetNo = ?;");
try {
ps = connection.prepareStatement(query);
ps.setInt(1, DatasetNo);
rs = ps.executeQuery();
if (rs.next()) {
Dataset d = new Dataset();
d.setDatasetNo(rs.getInt("DatasetNo"));
d.setDatasetName(rs.getString("DatasetName"));
...
}
return d;
}
else {
return null;
}
}
catch(Exception ex) {
logger.error("Error getting Dataset " + DatasetNo + "\n", ex);
return null;
}
finally {
DatabaseUtils.closeResultSet(rs);
DatabaseUtils.closePreparedStatement(ps);
pool.freeConnection(connection);
}
}
Is anyone able to advise a way of correcting this problem?
I believe it is due to MySQL leaving connection poll connections open for up to eight hours but am not certain.
Thanks
Martin O'Shea.
Just to clarify one point made about my method of connection pooling, it isn't Oracle that I'm using in my application but a class of my own as follows:
package myDataSharer.database_access;
import java.sql.*;
import javax.sql.DataSource;
import javax.naming.InitialContext;
import org.apache.log4j.Logger;
public class ConnectionPool_DBA {
static Logger logger = Logger.getLogger(ConnectionPool_DBA.class.getName());
private static ConnectionPool_DBA pool = null;
private static DataSource dataSource = null;
public synchronized static ConnectionPool_DBA getInstance() {
if (pool == null) {
pool = new ConnectionPool_DBA();
}
return pool;
}
private ConnectionPool_DBA() {
try {
InitialContext ic = new InitialContext();
dataSource = (DataSource) ic.lookup("java:/comp/env/jdbc/myDataSharer");
}
catch(Exception ex) {
logger.error("Error getting a connection pool's datasource\n", ex);
}
}
public void freeConnection(Connection c) {
try {
c.close();
}
catch (Exception ex) {
logger.error("Error terminating a connection pool connection\n", ex);
}
}
public Connection getConnection() {
try {
return dataSource.getConnection();
}
catch (Exception ex) {
logger.error("Error getting a connection pool connection\n", ex);
return null;
}
}
}
I think the mention of Oracle is due to me using a similar name.
There are a few pointers on avoiding this situation, obtained from other sources, especially from the connection pool implementations of other drivers and from other application servers. Some of the information is already available in the Tomcat documentation on JNDI Data Sources.
Establish a cleanup/reaper schedule that will close connections in the pool, if they are inactive beyond a certain period. It is not good practice to leave a connection to the database open for 8 hours (the MySQL default). On most application servers, the inactive connection timeout value is configurable and is usually less than 15 minutes (i.e. connections cannot be left in the pool for more than 15 minutes unless they are being reused time and again). In Tomcat, when using a JNDI DataSource, use the removeAbandoned and removeAbandonedTimeout settings to do the same.
When a new connection is return from the pool to the application, ensure that it is tested first. For instance, most application servers that I know, can be configured so that connection to an Oracle database are tested with an execute of "SELECT 1 FROM dual". In Tomcat, use the validationQuery property to set the appropriate query for MySQL - I believe this is "SELECT 1" (without quotes). The reason why setting the value of the validationQuery property helps, is because if the query fails to execute, the connection is dropped from the pool, and new one is created in its place.
As far are the behavior of your application is concerned, the user is probably seeing the result of the pool returning a stale connection to the application for the first time. The second time around, the pool probably returns a different connection that can service the application's queries.
Tomcat JNDI Data Sources are based on Commons DBCP, so the configuration properties applicable to DBCP will apply to Tomcat as well.
I'd wonder why you're using ConnectionPool_DBA in your code instead of letting Tomcat handle the pooling and simply looking up the connection using JNDI.
Why are you using an Oracle connection pool with MySQL? When I do JNDI lookups and connection pooling, I prefer the Apache DBCP library. I find that it works very well.
I'd also ask if your DatabaseUtils methods throw any exceptions, because if either of the calls prior to your call to pool.freeConnection() throw one you'll never free up that connection.
I don't like your code much because a class that performs SQL operations should have its Connection instance passed into it, and should not have the dual responsibility of acquiring and using the Connection. A persistence class can't know if it's being used in a larger transaction. Better to have a separate service layer that acquires the Connection, manages the transaction, marshals the persistence classes, and cleans up when it's complete.
UPDATE:
Google turned up the Oracle class with the same name as yours. Now I really don't like your code, because you wrote something of your own when a better alternative was easily available. I'd ditch yours right away and redo this using DBCP and JNDI.
This error indicates server closes connection unexpectedly. This can occur in following 2 cases,
MySQL closes idle connection after certain time (default is 8 hours). When this occurs, no thread is responsible for closing the connection so it gets stale. This is most likely the cause if this error only happens after long idle.
If you don't completely read all the responses, the connection may get returned to the pool in busy state. Next time, a command is sent to MySQL and it closes connection for wrong state. If the error occurs quite frequent, this is probably the cause.
Meanwhile, setting up an eviction thread will help to alleviate the problem. Add something like this to the Data Source,
...
removeAbandoned="true"
removeAbandonedTimeout="120"
logAbandoned="true"
testOnBorrow="false"
testOnReturn="false"
timeBetweenEvictionRunsMillis="60000"
numTestsPerEvictionRun="5"
minEvictableIdleTimeMillis="30000"
testWhileIdle="true"
validationQuery="select now()"
Is there a router between the web server and the database that transparently closes idle TCP/IP connections?
If so, you must have your connection pool either discard unused-for-more-than-XX-minutes connections from the pool, or do some kind of ping every YY minutes on the connection to keep it active.
On the off chance you haven't found your answer I've been dealing with this for the last day. I am essentially doing the same thing you are except that I'm basing my pooling off of apache.commons.pool. Same exact error you are seeing EOF. Check your mysqld error log file which is most likely in your data directory. Look for mysqld crashing. mysqld_safe will restart your mysqld quickly if it crashes so it won't be apparent that this is the case unless you look in its logfile. /var/log is not help for this scenario.
Connections that were created before the crash will EOF after the crash.