We Keep Coding

How to improve performance of JDBC query execution?

I'm running a bot on Discord that receives a lot of requests to the MySQL database, and recently MySQL has started blocking threads, causing major delays in the program.
After dumping the thread, I've found that the problematic line resides within the PreparedStatement code from JDBC, but I'm really not sure what could be causing this issue.
The code block below is where the error occurs:
public List<HashMap<String, Object>> find(String haystack, Object... needles){
PreparedStatement prep = null;
List<HashMap<String, Object>> results = new ArrayList<>();
ResultSet rs = null;
try{
prep = connection.prepareStatement(haystack);
for(int i = 0; i < needles.length; i++){
prep.setObject(i+1, needles[i]);
}
rs = prep.executeQuery();
while(rs.next()){
HashMap<String, Object> result = new HashMap<>();
for(int i = 1; i < rs.getMetaData().getColumnCount() + 1; i++){
result.put(rs.getMetaData().getColumnName(i), rs.getObject(i));
}
results.add(result);
}
}catch(SQLException e){
System.out.println("MySQL > Unable to execute query: " + e.getMessage());
}finally{
try{
if(rs!=null)rs.close();
if(prep!=null)prep.close();
}catch(SQLException e){
System.out.println("(find) Error closing: " + e.getMessage());
}
}
return results;
}
with rs = prep.executeQuery(); being the problematic line of code.
Is there any way to stop MySQL from blocking threads?

I see that you are using only one connection throughout the application. You should create a pool of connections if you have a large number of request to handle which can be done using following approaches:
You can create connection pooling on the application side. You can use apache connection pool.
You can create connection pooling on the server end. Read this.
Best, use hibernate, there you have a property called hibernate.connection.pool_size.
If you are using JDBC prepared statement then use batch processing and avoid using Statement (which I see you are doing) because of the reason mentioned in this post.

how to check JDBC connection is live or closed Using try-with-resources?

Once try-with-resources statement execution completion after how to check JDBC connection is live or closed ?
Code :
try (Connection conn = ConnectionString.getConnection();
PreparedStatement psmt = conn.prepareStatement(SQL_Queries.GET_PUBLISHER_ID_QUERY);
ResultSet rs = psmt.executeQuery();)
{
psmt.setString(1, appId);
while (rs.next()) {
publisherId = rs.getString(Schema.PUBLISHER_PUBLISHID);
}
//System.out.println(conn != null ? "live" : "close");
}

The idea is that using the try-with-resource block, you are creating an inner scope for your connection. The try-with-resource closes the connection and the conn object scope ends, thus is ready for garbage collection. Read more about scope here.
To check this, run this code
String url, username, pass;
url = "your-connection-url";
username = "db-username";
pass = "db-password";
Connection outerConnection = null;
try (Connection innerConnection = DriverManager.getConnection(url, username, pass);
PreparedStatement psmt = innerConnection.prepareStatement("SELECT 1");
ResultSet rs = psmt.executeQuery();)
{
while (rs.next()) {
System.out.println(rs.getString(1));
}
outerConnection = innerConnection;
// after the try catch, innerConnection won't exist anymore
// (the scope ends, the compiler compains if you access it outside the block!)
} catch (SQLException e)
{
e.printStackTrace();
}
if (outerConnection != null) {
try {
System.out.println("Is closed? " + outerConnection.isClosed());
}
catch (SQLException e) {
e.printStackTrace();
}
}
else {
System.out.println("Is null");
}
In the console you'll get
1
Is closed? true

First of all, the Connection object scope is limited to the try block, it's not visible outside of it.
Secondly, you said it yourself, all resources in a try-with-resources statement, which must all implement AutoCloseable, are closed at the end of the statement. So even if your Connection was visible, it would be closed.

Try / Try-with-resources and Connection, Statement and ResultSet closing

I have recently having some discussions with my professor about how to handle the basic jdbc connection scheme. Suppose we want to execute two queries, this is what he proposes
public void doQueries() throws MyException{
Connection con = null;
try {
con = DriverManager.getConnection(dataSource);
PreparedStatement s1 = con.prepareStatement(updateSqlQuery);
PreparedStatement s2 = con.prepareStatement(selectSqlQuery);
// Set the parameters of the PreparedStatements and maybe do other things
s1.executeUpdate();
ResultSet rs = s2.executeQuery();
rs.close();
s2.close();
s1.close();
} catch (SQLException e) {
throw new MyException(e);
} finally {
try {
if (con != null) {
con.close();
}
} catch (SQLException e2) {
// Can't really do anything
}
}
}
I don't like this approach, and I have two questions about it:
1.A) I think that, if any exception is thrown where we do 'other things', or in the line rs.close() or s2.close() then s1 wouldn't have been closed when the method ends. Am I right about that?
1.B) The professor keeps asking me to explicitly close the ResultSet (even when the Statement documentation makes clear that it will close the ResultSet) She says that Sun recommends it. Is there any reason to do so?
Now this is what I think is the correct code for the same thing:
public void doQueries() throws MyException{
Connection con = null;
PreparedStatement s1 = null;
PreparedStatement s2 = null;
try {
con = DriverManager.getConnection(dataSource);
s1 = con.prepareStatement(updateSqlQuery);
s2 = con.prepareStatement(selectSqlQuery);
// Set the parameters of the PreparedStatements and maybe do other things
s1.executeUpdate();
ResultSet rs = s2.executeQuery();
} catch (SQLException e) {
throw new MyException(e);
} finally {
try {
if (s2 != null) {
s2.close();
}
} catch (SQLException e3) {
// Can't do nothing
}
try {
if (s1 != null) {
s1.close();
}
} catch (SQLException e3) {
// Can't do nothing
}
try {
if (con != null) {
con.close();
}
} catch (SQLException e2) {
// Can't do nothing
}
}
}
2.A) Is this code correct? (Is it guaranteed that all will be closed when the method ends?)
2.B) This is very large and verbose (and it gets worse if there are more Statements) Is there any shorter or more elegant way to do this without using try-with-resources?
Finally this is the code I like the most
public void doQueries() throws MyException{
try (Connection con = DriverManager.getConnection(dataSource);
PreparedStatement s1 = con.prepareStatement(updateSqlQuery);
PreparedStatement s2 = con.prepareStatement(selectSqlQuery))
{
// Set the parameters of the PreparedStatements and maybe do other things
s1.executeUpdate();
ResultSet rs = s2.executeQuery();
} catch (SQLException e) {
throw new MyException(e);
}
}
3) Is this code correct? I think my professor doesn't like this way because there is no explicit close of the ResultSet, but she has told me that she is fine with it as long as in the documentation it is clear that all is closed. Can you give any link to the official documentation with a similar example, or based in the documentation show that there is are no problems with this code?

tl;dr
In theory closing the statement closes the result set.
In practice, some faulty JDBC driver implementations failed to do so, notoriously. Thus the advice from your instructor that she learned from the School Of Hard Knocks.
Unless you are familiar with every implementation of every JDBC
driver that might be deployed for your app, use
try-with-resources to auto-close every level of your JDBC
work such as statements and result sets.
Use try-with-resources syntax
None of your code is fully using try-with-resources. In try-with-resources syntax, you declare and instantiate your Connection, PreparedStatement, and ResultSet in parentheses, before the braces. See Tutorial by Oracle.
While your ResultSet is not being explicitly closed in your last code example, it should be closed indirectly when its statement is closed. But as discussed below, it might not be closed because of faulty JDBC driver.
AutoCloseable
Any such objects implementing AutoCloseable will automatically have their close method invoked. So no need for those finally clauses.
How do you know which objects are auto-closable and which are not? Look at their class documentation to see if it declares AutoCloseable as a super-interface. Conversely, see the JavaDoc page for AutoCloseable for a list of all the bundled sub-interfaces and implementing classes (dozens actually).
For example, for SQL work, we see that Connection, Statement, PreparedStatement, ResultSet, and RowSet are all auto-closable but DataSource is not. This makes sense, as DataSource stores data about potential resources (database connections) but is not itself a resource. A DataSource is never “open” so no need to close.
See Oracle Tutorial, The try-with-resources Statement.
Code example
Your last code example is getting close to good, but should have wrapped ResultSet in a try-with-resources statement to get automatically closed.
To quote ResultSet JavaDoc:
A ResultSet object is automatically closed when the Statement object that generated it is closed, re-executed, or used to retrieve the next result from a sequence of multiple results.
As your teacher has been suggesting, there have been serious flaws in some JDBC drivers that failed to live up to the promise of the JDBC spec to close the ResultSet when its Statement or PreparedStatement is closed. So many programmers make a habit of closing each ResultSet object explicitly.
This extra duty is easier now with the try-with-resources syntax. In real work you’ll likely have a try-else around all your AutoCloseable objects such as ResultSet anyways. So my own opinion is: Why not make it a try-with-resources + else? Does not hurt, makes your code more self-documenting about your intentions, and it might help if your code ever encounters one of those faulty JDBC drivers. The only cost is a pair of parens, assuming you’d have a try-catch-else in place anyways.
As stated in the Oracle Tutorial, multiple AutoCloseable objects declared together will be closed in reverse order, just as we would want.
Tip: The try-with-resources syntax allows an optional semicolon on the last declared resource item. I include the semicolon as a habit because it reads well to my eye, is consistent, and facilitates cut-and-paste editing. I include it on your PreparedStatement s2 line.
public void doQueries() throws MyException{
// First try-with-resources.
try ( Connection con = DriverManager.getConnection( dataSource ) ;
PreparedStatement s1 = con.prepareStatement( updateSqlQuery ) ;
PreparedStatement s2 = con.prepareStatement( selectSqlQuery ) ;
) {
… Set parameters of PreparedStatements, etc.
s1.executeUpdate() ;
// Second try-with-resources, nested within first.
try (
ResultSet rs = s2.executeQuery() ;
) {
… process ResultSet
} catch ( SQLException e2 ) {
… handle exception related to ResultSet.
}
} catch ( SQLException e ) {
… handle exception related to Connection or PreparedStatements.
}
}
I suppose there is a more elegant syntax for this kind of work that might be invented in a future programming language. But for now, we have try-with-resources, and I do use it happily. While try-with-resources is not perfectly elegant, it is a big improvement over the older syntax.
By the way, Oracle recommends using a DataSource implementation for getting connections rather than the DriverManager approach seen in your code. Using DataSource throughout your code makes it easier to switch drivers or switch to a connection pool. See if your JDBC driver provides an implementation of DataSource.
Update: Java 9
Now in Java 9 you can initialize the resources before the try-with-resources. See this article. This flexibility may be useful in some scenarios.

The fun thing about JDBC code is that you're coding to a spec where it's not always clear how compliant your implementation is. There are a lot of different databases and drivers and some drivers are better-behaved than others. That tends to make people err on the side of caution, recommending things like closing everything explicitly. You could be ok with closing only the connection here. Closing the resultSet just to be on the safe side is hard to argue with. You don't indicate what database or driver you're using here, i wouldn't want to hardcode in assumptions about the driver that might not be valid for some implementation.
Closing things in sequence does leave you open to problems where an exception can get thrown and cause some of the closes to be skipped. You're right to be concerned about that.
Be aware this is a toy example. Most real code uses a connection pool, where calling the close method doesn't actually close the connection, instead it returns the connection to the pool. So resources may not get closed once you use a pool. If you want to change this code to use a connection pool then you'll have to go back and close the statements at least.
Also if you're objecting to the verbosity of this, the answer to that is to hide the code in reusable utilities that use strategies, resultSet mappers, prepared statement setters, etc. This has all been done before, of course; you'll be on the road to reinventing Spring JDBC.
Speaking of which: Spring JDBC closes everything explicitly (probably because it needs to work with as many drivers as possible, and doesn't want to cause problems due to some driver's not being well-behaved).

This is what I find to be the best solution for handling resources like JDBC. This method provides an immutable function, by leveraging final variables, and only declaring and assigning those variables if they are needed, it is very CPU efficient, and guarantees in all cases, that all resources that are assigned and opened are closed regardless of the state of exceptions. The technique you are using leaves gaps that can result in resource leaks if not carefully implemented to address all scenarios. This technique does not allow for a resource leak provided the pattern is always followed:
1) assign resource
2) try
3) use resource
4) finally close resource
public void doQueries() throws MyException {
try {
final Connection con = DriverManager.getConnection(dataSource);
try {
final PreparedStatement s1 = con.prepareStatement(updateSqlQuery);
try {
// Set the parameters of the PreparedStatements and maybe do other things
s1.executeUpdate();
} finally {
try { s1.close(); } catch (SQLException e) {}
}
final PreparedStatement s2 = con.prepareStatement(selectSqlQuery);
try {
// Set the parameters of the PreparedStatements and maybe do other things
final ResultSet rs = s2.executeQuery();
try {
// Do something with rs
} finally {
try { rs.close(); } catch (SQLException e) {}
}
} finally {
try { s2.close(); } catch (SQLException e) {}
}
} finally {
try { con.close(); } catch (SQLException e) {}
}
} catch (SQLException e) {
throw new MyException(e);
}
}
With Java 7, you can leverage the new try -with-resources to simplify this even more: The new try -with-resources follows the above logic flow, in that it will guarantee all resources include in the with resources block that are assigned get closed. any exception thrown in the with resources block will the thrown, but those assigned resources will still be closed. This code is much simplified and looks like this:
public void doQueries() throws MyException {
try (
final Connection con = DriverManager.getConnection(dataSource);
final PreparedStatement s1 = con.prepareStatement(updateSqlQuery);
final PreparedStatement s2 = con.prepareStatement(selectSqlQuery);
final ResultSet rs = s2.executeQuery()) {
s1.executeUpdate();
// Do something with rs
} catch (SQLException e) {
throw new MyException(e);
}
}
[EDIT]: moved rs assignment into the resources block to show the simplest implementation. In practice, this simple solution does not really work, as this is not efficient. A connection should be reused, since establishing the connection is a very costly operation. Additionally, this simple example does not assign query parameters to the prepared statement. Care should be taken to handle these scenarios, as the the resource block should only include assignment statements. To depict this, I have also added another example
public void doQueries() throws MyException {
final String updateSqlQuery = "select ##servername";
final String selecSqlQuery = "select * from mytable where col1 = ? and col2 > ?";
final Object[] queryParams = {"somevalue", 1};
try (final Connection con = DriverManager.getConnection(dataSource);
final PreparedStatement s1 = newPreparedStatement(con, updateSqlQuery);
final PreparedStatement s2 = newPreparedStatement(con, selectSqlQuery, queryParams);
final ResultSet rs = s2.executeQuery()) {
s1.executeUpdate();
while (!rs.next()) {
// do something with the db record.
}
} catch (SQLException e) {
throw new MyException(e);
}
}
private static PreparedStatement newPreparedStatement(Connection con, String sql, Object... args) throws SQLException
{
final PreparedStatement stmt = con.prepareStatement(sql);
for (int i = 0; i < args.length; i++)
stmt.setObject(i, args[i]);
return stmt;
}

This is indeed the primary motivation for try-with-resources. See the Java tutorials as reference. Your professor is out-of-date. If you want to deal with the result set issue you can always enclose it in another try-with-resources statement.

You could make a util class to handle closing of these resources. i.e
FYI i just ignored SQLExceptions from trying to close resources in the util class, but you could as verywell log or collect and throw them once you are done closing the resources in the collection depending on your needs
public class DBUtil {
public static void closeConnections(Connection ...connections){
if(connections != null ){
for(Connection conn : connections){
if(conn != null){
try {
conn.close();
} catch (SQLException ignored) {
//ignored
}
}
}
}
}
public static void closeResultSets(ResultSet ...resultSets){
if(resultSets != null ){
for(ResultSet rs: resultSets){
if(rs != null){
try {
rs.close();
} catch (SQLException ignored) {
//ignored
}
}
}
}
}
public static void closeStatements(Statement ...statements){
if(statements != null){
for(Statement statement : statements){
if(statement != null){
try {
statement.close();
} catch (SQLException ignored) {
//ignored
}
}
}
}
}
}
and then just call it from you method:
public void doQueries() throws MyException {
Connection con = null;
try {
con = DriverManager.getConnection(dataSource);
PreparedStatement s1 = null;
PreparedStatement s2 = null;
try {
s1 = con.prepareStatement(updateSqlQuery);
s2 = con.prepareStatement(selectSqlQuery);
// Set the parameters of the PreparedStatements and maybe do other things
s1.executeUpdate();
ResultSet rs = null;
try {
rs = s2.executeQuery();
} finally {
DBUtil.closeResultSets(rs);
}
} finally {
DBUtil.closeStatements(s2, s1);
}
} catch (SQLException e) {
throw new MyException(e);
} finally {
DBUtil.closeConnections(con);
}
}

I prefer to let Java auto-close. So I do something like this when I have to set values for ResultSet.
try (Connection conn = DB.getConn();
PreparedStatement ps = conn.prepareStatement("SELECT * FROM x WHERE y = ?")
) {
ps.setString(1, "yValue");
try(ResultSet rs = ps.executeQuery()) {
while(rs.next()) {
...
}
}
} catch (SQLException e) {
e.printStackTrace(e);
...
}

Is it Ok to Pass ResultSet?

In my situation, I am querying a database for a specific return (in this case registration information based on a username).
//Build SQL String and Query Database.
if(formValid){
try {
SQL = "SELECT * FROM users WHERE username=? AND email=?";
Collections.addAll(fields, username, email);
results = services.DataService.getData(SQL, fields);
if (!results.next()){
errMessages.add("User account not found.");
} else {
user = new User();
user.fillUser(results); //Is it ok to pass ResultSet Around?
}
} catch (SQLException e) {
e.printStackTrace();
} finally {
services.DataService.closeDataObjects(); //Does this close the ResultSet I passed to fillUser?
}
}
So once I query the database, if a result is found I create a new User object and populate it with the data I received from the database. I used to do all of this directly in the method that I was pulling the resultset into, but I realized I was doing a lot of redundant coding throughout my project so I moved it all into one central method that lives in the actual User bean.
public void fillUser(ResultSet data) throws SQLException{
setUserId(data.getInt("id"));
setFirstName(data.getString("first_name"));
setLastName(data.getString("last_name"));
setUsername(data.getString("username"));
setType(data.getString("type"));
setEmail(data.getString("email"));
}
I have done a few tests and from what I can determine, because I close the original resultset in the finally block of the query, the resultset that I pass into the fillUser method also gets closed. Or am I wrong and am I seriously leaking data? This is actually the second time I pass a resultset (so its two instances of one) because the block I use to query my database is
public static ResultSet getData(String SQL, ArrayList fields) throws SQLException {
try{
connection = Database.getConnection();
preparedStatement = connection.prepareStatement(SQL);
for(int i=0; i<fields.size(); i++){
Integer num = i + 1;
Object item = fields.get(i);
if(item instanceof String){
preparedStatement.setString(num, (String) item); //Array item is String.
} else if (item instanceof Integer){
preparedStatement.setInt(num, (Integer) item); //Array item is Integer.
}
}
resultSet = preparedStatement.executeQuery();
return resultSet;
}finally{
}
}
All of these code snippets live in separate classes and are reused multiple times throughout my project. Is it ok to pass a resultset around like this, or should I be attempting another method? My goal is to reduce the codes redundancy, but i'm not sure if i'm going about it in a legal manner.

Technically, it's OK to pass result sets, as long as you are not serializing and passing it to a different JVM, and your JDBC connection and statement are still open.
However, it's probably a better software engineer and programming practice to have DB access layer that returns you the result set in a Java encoded way (a list of User in your example). That way, your code would be cleaner and you won't have to worry if the ResultSet is already opened, or you have to scroll it to the top, you name it...

As everyone before me said its a bad idea to pass the result set. If you are using Connection pool library like c3p0 then you can safely user CachedRowSet and its implementation CachedRowSetImpl. Using this you can close the connection. It will only use connection when required. Here is snippet from the java doc:
A CachedRowSet object is a disconnected rowset, which means that it makes use of a connection to its data source only briefly. It connects to its data source while it is reading data to populate itself with rows and again while it is propagating changes back to its underlying data source. The rest of the time, a CachedRowSet object is disconnected, including while its data is being modified. Being disconnected makes a RowSet object much leaner and therefore much easier to pass to another component. For example, a disconnected RowSet object can be serialized and passed over the wire to a thin client such as a personal digital assistant (PDA).
Here is the code snippet for querying and returning ResultSet:
public ResultSet getContent(String queryStr) {
Connection conn = null;
Statement stmt = null;
ResultSet resultSet = null;
CachedRowSetImpl crs = null;
try {
Connection conn = dataSource.getConnection();
stmt = conn.createStatement();
resultSet = stmt.executeQuery(queryStr);
crs = new CachedRowSetImpl();
crs.populate(resultSet);
} catch (SQLException e) {
throw new IllegalStateException("Unable to execute query: " + queryStr, e);
}finally {
try {
if (resultSet != null) {
resultSet.close();
}
if (stmt != null) {
stmt.close();
}
if (conn != null) {
conn.close();
}
} catch (SQLException e) {
LOGGER.error("Ignored", e);
}
}
return crs;
}
Here is the snippet for creating data source using c3p0:
ComboPooledDataSource cpds = new ComboPooledDataSource();
try {
cpds.setDriverClass("<driver class>"); //loads the jdbc driver
} catch (PropertyVetoException e) {
e.printStackTrace();
return;
}
cpds.setJdbcUrl("jdbc:<url>");
cpds.setMinPoolSize(5);
cpds.setAcquireIncrement(5);
cpds.setMaxPoolSize(20);
javax.sql.DataSource dataSource = cpds;

JDBC connection with auto reconnect

I am using JDBC to connect to a database server.
The connection is over a wireless network and can be dodgy at times.
At the moment when the connection is lost I need to close and restart the application.
Does anyone have some examples of code where I could write some sort of wrapper to automatically reconnect and rerun the last query? This would save a lot of hassles.
I am just not sure how it should/could be implemented.
Maybe there is already something available?

Even if you use JDBC connection pool either application server provided or apache commons pooling, it is worthwhile to code a retry logic. Based on the configuration of your application server, the app server would purge all the pooled connections and recreate a fresh set of connections. Here is a sample:
Connection conn = null;
Statement stmt = null;
ResultSet rs = null;
//
// How many times do you want to retry the transaction
// (or at least _getting_ a connection)?
//
int retryCount = 5;
boolean transactionCompleted = false;
do {
try {
conn = getConnection(); // assume getting this from a
// javax.sql.DataSource, or the
// java.sql.DriverManager
retryCount = 0;
stmt = conn.createStatement();
String query = "Some sample SQL";
rs = stmt.executeQuery(query);
while (rs.next()) {
}
rs.close();
rs = null;
stmt.close();
stmt = null;
conn.close();
conn = null;
transactionCompleted = true;
} catch (SQLException sqlEx) {
//
// The two SQL states that are 'retry-able'
// for a communications error.
//
// Only retry if the error was due to a stale connection,
// communications problem
//
String sqlState = sqlEx.getSQLState();
if ("Substitute with Your DB documented sqlstate number for stale connection".equals(sqlState) ) {
retryCount--;
} else {
retryCount = 0;
}
} finally {
if (rs != null) {
try {
rs.close();
} catch (SQLException sqlEx) {
// log this
}
}
if (stmt != null) {
try {
stmt.close();
} catch (SQLException sqlEx) {
// log this
}
}
if (conn != null) {
try {
//
// If we got here, and conn is not null, the
// transaction should be rolled back, as not
// all work has been done
try {
conn.rollback();
} finally {
conn.close();
}
} catch (SQLException sqlEx) {
//
// If we got an exception here, something
// pretty serious is going on, so we better
// pass it up the stack, rather than just
// logging it. . .
throw sqlEx;
}
}
}
} while (!transactionCompleted && (retryCount > 0));

Let a connection pool handle this for you, many of them can validate a connection. So does DBCP which has a testOnBorrow parameter that forces a sanity check on every connection before it's used. The default value of this parameter is true, it just needs validationQuery to be set to a non-null string to have any effect. So set the validationQuery and there you go! Check out the documentation.

Check out Oracle's Universal Connection Pool (UCP) libraries. They are fully JDBC 4.0 compliant and implement the isValid() call to check if a connection is live. It's easy to do this check, if false reconnect, then run your query.
Oracle UCP Download Page
While I know you didn't ask about connection pools, you should probably be using one anyway so this will help you twofold.