We Keep Coding

is public Connection getInstance(String databasename); a form of connection pooling?

In a javase database application, I process a lot of short-lived object (say accounting documents like bills etc.). Processing each object consists in opening a connection to a database and looking up for some data. Not all objects are looked up on the same database, but i select a specific database according to some object property, so i'll end up having several connections opened.
What i actually need is no more than one connection for each database.
So I've done something like this:
public MyPool {
Map<String, Connection> activeConnections = new TreeMap<String, Connection>();
public Connection getConnection(String database_name) throws SQLException {
if (activeConnections.containsKey(database_name)) {
return activeConnections.get(database_name);
}
//Retrive the configuration data from a configuration object
ConnectionConfig c = Configuration.getConnectionConfig(database_name);
Connection connection = DriverManager.getConnection(c.url, c.user, c.password);
return connection;
}
The questions are:
1) Since I see around a lot of pooling libraries, DBCP, c3p0 and others: what is the point of all those libraries, what do they add to a "basic" approach like this?
Tutorials like this doesn't help much in responding this question, since the basic solution exposed here fits perfectly their definition of connection pooling.
2) This is a piece of code that will be "exposed" to other developers, that in turn may develop procedures to retrieve data from databases with different structures, probably getting connections from this "pool object".
Is it correct, in the docs and in the code, referring to it as a "pool", or is it something different, so that calling "pool" would be misleading?

Your code isn't a connection pool implementation in the colloquial use of the term since each datasource only has a single physical connection. The concept behind object pooling (in this case, the object is a connection) is that some objects require overhead to configure. In the case of a connection pool, as you know, a database connection must be opened before you can talk to the database.
The difference here is that your code isn't thread safe for a concurrent environment like the popular connection pool implementations you've mentioned. Applications running in high concurrency circumstances like the web shouldn't need to absorb the overhead of establishing a connection on each request. Instead, a pool of open connections is maintained and when the request has finished working on the connection, it is returned to the pool for subsequent requests to make use of.
This is required because connections are stateful. You can't have multiple requests sharing the same connection at the same time and guarantee any sort of reasonable transaction semantics.

Use BoneCP and wrap the connection pool like this:
Do not try to create your own connection pool, that is what BoneCP or any number of other very good and well tested pools are for.
object ConnectionPool {
Class.forName("[ENTER DRIVER]")
private val connstring = [ENTER YOUR STRING]
private var cp : BoneCP = _
createConnectionPool() //upon init create the cp
/**
* Create a new connection pool
*/
def createConnectionPool() = {
if(cp == null) {
try {
val config = new BoneCPConfig()
config.setJdbcUrl(connstring)
config.setMaxConnectionsPerPartition(3)
config.setMinConnectionsPerPartition(1)
config.setPartitionCount(1)
cp = new BoneCP(config)
}
catch {
case e: SQLException => e.printStackTrace()
}
}
}
def getConnection () = { cp.getConnection }

A persistent connection with JDBC to MySQL

I have an application that connects to MySQL using JDBC. There are cases where the JDBC connection lies idle for hours (maybe even days) and its loosing its connection to MySQL and then excepts when it tries to execute a query. What is the best solution for this?

Keeping the connection open for an undertemined time is a bad practice. The DB will force a close when it's been open for a too long time. You should write your JDBC code so that it always closes the connection (and statement and resultset) in the finally block of the very same try block where you've acquired them in order to prevent resource leaking like this.
However, acquiring the connection on every hiccup is indeed a pretty expensive task, so you'd like to use a connection pool. Decent connection pools will manage the opening, testing, reusing and closing the connections themselves. This does however not imply that you can change your JDBC code to never close them. You still need to close them since that would actually release the underlying connection back to the pool for future reuse.
There are several connection pools, like Apache DBCP which is singlethreaded and thus poor in performance, C3P0 which is multithreaded and performs better, and Tomcat JDBC for the case that you're using Tomcat and wouldn't like to use the builtin DBCP due to bad performance.
You can create connection pools programmatically, here's an example with C3P0:
ComboPooledDataSource dataSource = new ComboPooledDataSource();
dataSource.setDriverClass("com.mysql.jdbc.Driver");
dataSource.setJdbcUrl("jdbc:mysql://localhost:3306/dbname");
dataSource.setUser("username");
dataSource.setPassword("password");
Do it once during application's startup, then you can use it as follows:
Connection connection = null;
// ...
try {
connection = dataSource.getConnection();
// ...
} finally {
// ...
if (connection != null) try { connection.close(); } catch (SQLException ignore) {}
}
When you're running inside a JNDI-capable container like a servletcontainer (e.g. Tomcat), then you can also declare it as a java.sql.DataSource (Tomcat specific manual here). It will then use the servletcontainer-provided connection pooling facilities. You can then acquire the datasource as follows:
DataSource dataSource = (DataSource) new InitialContext().lookup("java:comp/env/jdbc/YourDataSourceName");

There are libraries, such as Apache's DBCP which can do connection pooling. A part of this is they can be setup to automatically test the connection when you go to use it (such as "SELECT NOW() FROM DUAL", or something else harmless) and automatically re-establish the connection transparently if necessary, allowing your application to pretend that the connection is everlasting.

Check here:
http://download.oracle.com/javase/tutorial/jdbc/basics/connecting.html
Basically, you should use DataSource and always do a getConnection() before using it. DataSource, unless there's something terribly wrong, will reconnect if necessary.

Best way to manage DB connections without JNDI

I have a website, in which currently I am getting 1000 page views. I am expecting it will go around 30k per day in future. Now the problem for me to manage the DB connections.
At present I am just connecting to DB directly from java program. I know it is worst design in the world. But for time being I have written like that.
I have plan to manage connection pooling using JNDI. But the problem is my hosting provider is not supporting JNDI.
Can anyone suggest me how to manage DB connections without jndi?

Connection pooling does not per se require the connections to be obtained by JNDI. You can also just setup and use a connection pool independently from JNDI. Let's assume that you'd like to use C3P0, which is one of the better connection pools, then you can find "raw" JNDI-less setup details in this tutorial.
Here's an extract of the tutorial:
ComboPooledDataSource cpds = new ComboPooledDataSource();
cpds.setDriverClass( "org.postgresql.Driver" ); //loads the jdbc driver
cpds.setJdbcUrl( "jdbc:postgresql://localhost/testdb" );
cpds.setUser("swaldman");
cpds.setPassword("test-password");
Create the datasource once during application's startup and store it somewhere in the context. The connection can then be acquired and used as follows:
Connection connection = null;
// ...
try {
connection = cpds.getConnection();
// ...
} finally {
// ...
if (connection != null) try { connection.close(); } catch (SQLException ignore) {}
}
Yes, closing in finally is still mandatory, else the connection pool won't be able to take the connection back in pool for future reuse and it'll run out of connections.

How should I manage connecting to a database using java servlets, JSP & Tomcat

I'm pretty new to Servlets and JSP and to using databases.
At the moment I have a class in the 'model' part of my web app which has lots of methods that I've written to perform database queries and updates. At the moment, in each of those methods I am creating a database connection, doing the SQL stuff, then closing the connection.
This works fine while I'm just making small apps for myself but I'm starting to realise that if lots of people were using my app concurrently then it would start to become apparent that creating database connections and closing them for each method call is a time costly process. So I need to change the way I do things.
In Head First Servlet & JSP by Basham, Sierra & Bates, they describe how it's possible to use a ServletContextListener implementation to create an object on the deployment of the web app that will be added as an attribute of the ServletContext. The authors don't go into it, but imply that people often add a database connection as an attribute of the ServletContext. I thought I would like to implement this for myself, but after reading this stackoverflow article on database connection management I'm not so sure.
However as I'm just starting with servlets and JSP, let alone the rest of J2EE, a lot of that article is beyond me.
The points that stand out for me from that article are:
Something could happen to break that database connection and if we are relying only on that connection then we would need to redeploy our app in order to restart a connection. is this correct?
We should reply on the container to manage the database connections for us. Great, but how is this acheived? How can I communicate with the container? (Please bear in mind that I've just started with Servlets and JSP).
In terms of Servlet design in general, I have one servlet class per request type and that normally only has one type of call to a database ie a specific update or query. Instead of having a class with all the methods for querying the database, is it a better design for me to have the methods within their respective servlets or would that contravene the Model-View-Controller pattern?
I can't imagine that I'll be having too many problems with too many users slowing down the user experience just yet :) but I'd like to start doing things right if possible.
Many thanks in advance for your comments
Joe

The following page on Tomcat's website describes how to connect Tomcat and mySQL in detail. You do not want to roll your own, there are too many DataSource pools already available that have been debugged and tried in production environments.
The main thing about using a pool is that a connection is not terminated when you call close, instead it is just returned to the pool. Therefore it is important to make sure that you close your resources in a try/finally block. Look here for a sample.

I would check out connection pooling and specifically frameworks like C3P0 or Apache Commons DBCP.
Both these packages will look after maintaining and managing a collection of database connections for you.
Typically connections are established in advance, and handed out to requesting threads as they require them. Connections can be validated prior to being handed out (and remade in advance of the client using them if the connection is broken).

The way to go in a web application is to have a connection pool manage your connections. This will allow your threads of execution to share the database connections, which is an important point as connecting to a database is usually a costly operation. Using a connection pool is usually just a configuration task as most containers support managing a connection pool.
From the viewpoint of your code, there are very few changes. Basically:
Connection pools are accessed through the DataSource interface, while non-pooled connections may be accessed through the old DriverManager.
To get the DataSource you will usually have to use JNDI, as this is the standard method for publishing connection pools in a J2EE application.
You want to close() Connection objects as soon as possible. This returns the connection to the pool wothout disconnecting from the DB, so that other threads can use it.
As always, you should call close() on every JDBC resource (connections, statements, resultsets) to avoid leaking. This is specially important in a server application because they are infrequently restarted so leaks accumulate over time and eventually will make your application malfunction.
This is some example code from http://download.oracle.com/javase/1.4.2/docs/guide/jdbc/getstart/datasource.html (NOTE: not exception-safe). As you can see, once you get the Connection reference there is nothing really special.
Context ctx = new InitialContext();
DataSource ds = (DataSource)ctx.lookup("jdbc/AcmeDB");
Connection con = ds.getConnection("genius", "abracadabra");
con.setAutoCommit(false);
PreparedStatement pstmt = con.prepareStatement(
"SELECT NAME, TITLE FROM PERSONNEL WHERE DEPT = ?");
pstmt.setString(1, "SALES");
ResultSet rs = pstmt.executeQuery();
System.out.println("Sales Department:");
while (rs.next()) {
String name = rs.getString("NAME");
String title = rs.getString("TITLE");
System.out.println(name + " ; ;" + title);
}
pstmt.setString(1, "CUST_SERVICE");
ResultSet rs = pstmt.executeQuery();
System.out.println("Customer Service Department:");
while (rs.next()) {
String name = rs.getString("NAME");
String title = rs.getString("TITLE");
System.out.println(name + " ; ;" + title);
}
rs.close();
pstmt.close();
con.close();

The authors don't go into it, but imply that people often add a database connection as an attribute of the ServletContext.
That's not the standard way to handle this. The traditional approach is to use a connection pool i.e. a pool of ready to use connections. Then, applications borrow connections from and return them to the pool when done.
There are several standalone connection pool implementations available (C3P0, Commons DBCP, Bone CP) that you can bundle in your application. But when using a Servlet or Java EE container, I would use the connection pool provided by the container. Then, obtain a DataSource (a handle on a connection pool) via JNDI from the application to get a JDBC connection from it (and close it to return it to the pool).
The way to configure a pool is obviously container specific so you need to refer to the documentation of your container. The good news is that Tomcat provides several examples showing how to do this, how to obtain a datasource via JNDI and how to write proper JDBC code (read until the bottom of the page).
References
Apache Tomcat User Guide
JDBC Data Sources
JNDI Datasource HOW-TO

Connection pooling with Java and MySQL in Tomcat web application

I recently wrote and deployed a Java web application to a server and I'm finding an unusual problem which didn't appear during development or testing.
When a user logs in after so long and goes to display data from the database, the page indicates that there are no records to see. But upon page refresh, the first x records are shown according to the pagination rules.
Checking the logs, I find:
ERROR|19 09 2009|09 28 54|http-8080-4|myDataSharer.database_access.Database_Metadata_DBA| - Error getting types of columns of tabular Dataset 12
com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:
** BEGIN NESTED EXCEPTION **
java.io.EOFException
STACKTRACE:
java.io.EOFException
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1956)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2368)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2867)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1616)
And so on for several hundred lines.
The application is currently set for about 100 users but is not yet in full use. It uses connection pooling between the Apache Tomcat servlets / jsps and a MySQL database with the following code example forming the general arrangement of a database operation, of which there are typically several per page:
// Gets a Dataset.
public static Dataset getDataset(int DatasetNo) {
ConnectionPool_DBA pool = ConnectionPool_DBA.getInstance();
Connection connection = pool.getConnection();
PreparedStatement ps = null;
ResultSet rs = null;
String query = ("SELECT * " +
"FROM Dataset " +
"WHERE DatasetNo = ?;");
try {
ps = connection.prepareStatement(query);
ps.setInt(1, DatasetNo);
rs = ps.executeQuery();
if (rs.next()) {
Dataset d = new Dataset();
d.setDatasetNo(rs.getInt("DatasetNo"));
d.setDatasetName(rs.getString("DatasetName"));
...
}
return d;
}
else {
return null;
}
}
catch(Exception ex) {
logger.error("Error getting Dataset " + DatasetNo + "\n", ex);
return null;
}
finally {
DatabaseUtils.closeResultSet(rs);
DatabaseUtils.closePreparedStatement(ps);
pool.freeConnection(connection);
}
}
Is anyone able to advise a way of correcting this problem?
I believe it is due to MySQL leaving connection poll connections open for up to eight hours but am not certain.
Thanks
Martin O'Shea.
Just to clarify one point made about my method of connection pooling, it isn't Oracle that I'm using in my application but a class of my own as follows:
package myDataSharer.database_access;
import java.sql.*;
import javax.sql.DataSource;
import javax.naming.InitialContext;
import org.apache.log4j.Logger;
public class ConnectionPool_DBA {
static Logger logger = Logger.getLogger(ConnectionPool_DBA.class.getName());
private static ConnectionPool_DBA pool = null;
private static DataSource dataSource = null;
public synchronized static ConnectionPool_DBA getInstance() {
if (pool == null) {
pool = new ConnectionPool_DBA();
}
return pool;
}
private ConnectionPool_DBA() {
try {
InitialContext ic = new InitialContext();
dataSource = (DataSource) ic.lookup("java:/comp/env/jdbc/myDataSharer");
}
catch(Exception ex) {
logger.error("Error getting a connection pool's datasource\n", ex);
}
}
public void freeConnection(Connection c) {
try {
c.close();
}
catch (Exception ex) {
logger.error("Error terminating a connection pool connection\n", ex);
}
}
public Connection getConnection() {
try {
return dataSource.getConnection();
}
catch (Exception ex) {
logger.error("Error getting a connection pool connection\n", ex);
return null;
}
}
}
I think the mention of Oracle is due to me using a similar name.

There are a few pointers on avoiding this situation, obtained from other sources, especially from the connection pool implementations of other drivers and from other application servers. Some of the information is already available in the Tomcat documentation on JNDI Data Sources.
Establish a cleanup/reaper schedule that will close connections in the pool, if they are inactive beyond a certain period. It is not good practice to leave a connection to the database open for 8 hours (the MySQL default). On most application servers, the inactive connection timeout value is configurable and is usually less than 15 minutes (i.e. connections cannot be left in the pool for more than 15 minutes unless they are being reused time and again). In Tomcat, when using a JNDI DataSource, use the removeAbandoned and removeAbandonedTimeout settings to do the same.
When a new connection is return from the pool to the application, ensure that it is tested first. For instance, most application servers that I know, can be configured so that connection to an Oracle database are tested with an execute of "SELECT 1 FROM dual". In Tomcat, use the validationQuery property to set the appropriate query for MySQL - I believe this is "SELECT 1" (without quotes). The reason why setting the value of the validationQuery property helps, is because if the query fails to execute, the connection is dropped from the pool, and new one is created in its place.
As far are the behavior of your application is concerned, the user is probably seeing the result of the pool returning a stale connection to the application for the first time. The second time around, the pool probably returns a different connection that can service the application's queries.
Tomcat JNDI Data Sources are based on Commons DBCP, so the configuration properties applicable to DBCP will apply to Tomcat as well.

I'd wonder why you're using ConnectionPool_DBA in your code instead of letting Tomcat handle the pooling and simply looking up the connection using JNDI.
Why are you using an Oracle connection pool with MySQL? When I do JNDI lookups and connection pooling, I prefer the Apache DBCP library. I find that it works very well.
I'd also ask if your DatabaseUtils methods throw any exceptions, because if either of the calls prior to your call to pool.freeConnection() throw one you'll never free up that connection.
I don't like your code much because a class that performs SQL operations should have its Connection instance passed into it, and should not have the dual responsibility of acquiring and using the Connection. A persistence class can't know if it's being used in a larger transaction. Better to have a separate service layer that acquires the Connection, manages the transaction, marshals the persistence classes, and cleans up when it's complete.
UPDATE:
Google turned up the Oracle class with the same name as yours. Now I really don't like your code, because you wrote something of your own when a better alternative was easily available. I'd ditch yours right away and redo this using DBCP and JNDI.

This error indicates server closes connection unexpectedly. This can occur in following 2 cases,
MySQL closes idle connection after certain time (default is 8 hours). When this occurs, no thread is responsible for closing the connection so it gets stale. This is most likely the cause if this error only happens after long idle.
If you don't completely read all the responses, the connection may get returned to the pool in busy state. Next time, a command is sent to MySQL and it closes connection for wrong state. If the error occurs quite frequent, this is probably the cause.
Meanwhile, setting up an eviction thread will help to alleviate the problem. Add something like this to the Data Source,
...
removeAbandoned="true"
removeAbandonedTimeout="120"
logAbandoned="true"
testOnBorrow="false"
testOnReturn="false"
timeBetweenEvictionRunsMillis="60000"
numTestsPerEvictionRun="5"
minEvictableIdleTimeMillis="30000"
testWhileIdle="true"
validationQuery="select now()"

Is there a router between the web server and the database that transparently closes idle TCP/IP connections?
If so, you must have your connection pool either discard unused-for-more-than-XX-minutes connections from the pool, or do some kind of ping every YY minutes on the connection to keep it active.

On the off chance you haven't found your answer I've been dealing with this for the last day. I am essentially doing the same thing you are except that I'm basing my pooling off of apache.commons.pool. Same exact error you are seeing EOF. Check your mysqld error log file which is most likely in your data directory. Look for mysqld crashing. mysqld_safe will restart your mysqld quickly if it crashes so it won't be apparent that this is the case unless you look in its logfile. /var/log is not help for this scenario.
Connections that were created before the crash will EOF after the crash.