We Keep Coding

How to improve performance of JDBC query execution?

I'm running a bot on Discord that receives a lot of requests to the MySQL database, and recently MySQL has started blocking threads, causing major delays in the program.
After dumping the thread, I've found that the problematic line resides within the PreparedStatement code from JDBC, but I'm really not sure what could be causing this issue.
The code block below is where the error occurs:
public List<HashMap<String, Object>> find(String haystack, Object... needles){
PreparedStatement prep = null;
List<HashMap<String, Object>> results = new ArrayList<>();
ResultSet rs = null;
try{
prep = connection.prepareStatement(haystack);
for(int i = 0; i < needles.length; i++){
prep.setObject(i+1, needles[i]);
}
rs = prep.executeQuery();
while(rs.next()){
HashMap<String, Object> result = new HashMap<>();
for(int i = 1; i < rs.getMetaData().getColumnCount() + 1; i++){
result.put(rs.getMetaData().getColumnName(i), rs.getObject(i));
}
results.add(result);
}
}catch(SQLException e){
System.out.println("MySQL > Unable to execute query: " + e.getMessage());
}finally{
try{
if(rs!=null)rs.close();
if(prep!=null)prep.close();
}catch(SQLException e){
System.out.println("(find) Error closing: " + e.getMessage());
}
}
return results;
}
with rs = prep.executeQuery(); being the problematic line of code.
Is there any way to stop MySQL from blocking threads?

I see that you are using only one connection throughout the application. You should create a pool of connections if you have a large number of request to handle which can be done using following approaches:
You can create connection pooling on the application side. You can use apache connection pool.
You can create connection pooling on the server end. Read this.
Best, use hibernate, there you have a property called hibernate.connection.pool_size.
If you are using JDBC prepared statement then use batch processing and avoid using Statement (which I see you are doing) because of the reason mentioned in this post.

Java SQL Server Application Stuck On statement.executeUpdate()

I have a rather annoying issue. In the piece of code below, I am trying to insert a new row to the "RevisionDispersion" table in my database. However, whenever I call stmt.executeUpdate() the program freezes and there ends up being no transaction to the database. No matter how long I wait; the database just won't be updated. Below is the code of interest:
private static final String INSERT_DISPERSION = "insert into RevisionDispersion("
+ Assignments.ID + ", "
+ Persons.EMAIL + ", "
+ Handins.ID + ")"
+ " values(?, ?, ?)";
public static void disperse(DataSource source, Assignment assignment) throws Exception
{
List<String> handins = assignment.getHandins();
//used to decide who checks which assignment
int maxRNG = Math.max(1, handins.size() / assignment.getPeerCount());
int rng = new Random().nextInt(maxRNG);
PreparedStatement stmt = null;
Connection con = null;
try{
//Get the connection, set it to TRANSACTION_SERIALIZABLE and set autocommit to false
con = source.getConnection();
configureConnection(con);
//Prepare the statement to insert the new dispersion
stmt = con.prepareStatement(INSERT_DISPERSION);
stmt.setString(1, assignment.getID());
//Iterate over all hand-ins and decide from which peer a peer receives feedback
for(int i = 0; i < handins.size(); i++)
{
HandIn handin = new HandIn(source.getConnection(), handins.get(i));
String student = handin.getEmail();
stmt.setString(2, student);
for(int j = 1; j <= assignment.getPeerCount(); j++)
{
HandIn otherHandin = new HandIn(source.getConnection(), handins.get(j * rng));
stmt.setString(3, otherHandin.getId());
stmt.executeUpdate();
}
}
con.commit();
}catch(Exception e){
throw e;
}finally{
closeQuietly(con, stmt);
}
}
//This method is originally in the DBAO class, but I put it here for you folks.
protected static void configureConnection(Connection connection) throws SQLException
{
connection.setAutoCommit(false);
connection.setTransactionIsolation(Connection.TRANSACTION_SERIALIZABLE);
}
This problem occurs in no other places in the application. Whenever I run the SQL statement in SQL Server Management Studio, with identical parameters, it does not get stuck and it inserts the new rows just fine. After deleting the rows and trying the same in the application, it gets stuck.
Can anyone point me in the right direction of what is going wrong? I've been trying for 3 hours straight now...
Stuff I already tried
-use stmt.addBatch() rather than executeUpdate() (did not make a difference. It would get stuck at executeBatch())
-Check if all connections are being closed properly; they are.
-Check if other statements/resultsets are still open that use RevisionDispersion table (there are none still open. Even if there were, should not make a difference I think?)
-Completely delete the database and set it back up

I solved the issue...
In a different piece of code I had the following:
private static final String GET_NOT_DISPERSED = "select * from Assignments where "
+ Assignments.CLOSE_DATE + "<=? and "
+ Assignments.PEER_START_DATE + ">=? and "
+ Assignments.ID + " not in(select " + Assignments.ID + " from RevisionDispersion)";
private void makeMailDispersion() throws Exception
{
DateTime currentDate = DateTime.getCurrentDateTime();
PreparedStatement assignmentsStmt = null;
ResultSet assignments = null;
Connection con = null;
try{
con = source.getConnection();
configureConnection(con);
assignmentsStmt = con.prepareStatement(GET_NOT_DISPERSED);
assignmentsStmt.setString(1, currentDate.toString());
assignmentsStmt.setString(2, currentDate.toString());
assignments = assignmentsStmt.executeQuery();
ArrayList<Assignment> requiresDispersion = new ArrayList<>();
assignments.close();
assignmentsStmt.close();
while(assignments.next())
{
Assignment assignment = new Assignment(source.getConnection(), assignments.getString(Assignments.ID));
AssignmentDisperser.disperse(source, assignment);
}
}catch(Exception e){
throw e;
}finally{
closeQuietly(con, assignmentsStmt, assignments);
}
}
In this piece of code, I closed the variables 'assignments' and 'assignmentsStmt'. I thought this would be sufficient to unlock the table after having used the GET_NOT_DISPERSED query. Apparently it was not: the table was still locked.
What I had to do in order to fix it: aside from calling assignments.close() and assignmentsStmt.close() I also had to call con.close(). That completely unlocked the table and allowed the code to run properly.

SQLException while running a query with JDBC

In my program I execute some SQL queries with JDBC. When I run the program for this certain query I get the following error:
SQLException: Exception in thread "main" java.sql.SQLException:at TransformData.main(TransformData.java:213)
Here is this part of code:
try
{
dbcon = DriverManager.getConnection(url,"username","password");
stmt = dbcon.createStatement();
stmt1 = dbcon.createStatement();
stmt13 = dbcon.createStatement();
stmt14 = dbcon.createStatement();
String sql1 = "SELECT DISTINCT payer_id FROM transactions ORDER BY payer_id";
rs1 = stmt1.executeQuery(sql1);
while (rs1.next())
{
Integer payer_id = rs1.getInt("payer_id");
payer_ids.add(payer_id);
}
rs1.close();
stmt1.close();
for(int i = 0; i < payer_ids.size(); i++)
{
String sql13 = "SELECT COUNT(*) AS counter, isCOrporate FROM transformed_table WHERE payer_id = "+payer_ids.get(i)+" ";
rs5 = stmt13.executeQuery(sql13);
while(rs5.next())
{
int counter = rs5.getInt("counter");
int isCorporate = rs5.getInt("isCorporate");
if ((counter - payer_ids.get(i).intValue() - isCorporate) < 1)
{
String sql14 = "DELETE FROM transformed_table WHERE payer_id = "+payer_ids.get(i)+" ";
stmt14.executeUpdate(sql14);
}
}
}
rs5.close();
stmt13.close();
stmt14.close();
dbcon.close();
}
catch(SQLException e)
{
System.out.print("SQLException: ");
throw new SQLException(errorMessages);
}
Line 213 is this line: throw new SQLException(errorMessages); in catch.
I am trying to find what might throw this exception. Can someone help?

There are lots of things wrong with this code, but here's the pertinent hint: Your catch block is wrong. Write it this way:
catch(SQLException e) {
e.printStackTrace();
}
Your way drains all the useful information out of the stack trace. You can't debug what else you've done wrong.
Make the change, rerun the code, and read the stack trace. It'll tell you what your real problem is.
So many things to correct:
Resources aren't closed properly. Those should be done in individual try/catch blocks in a finally block.
Badly written SQL. The DELETE could be done far more efficiently with a JOIN.
Poor decomposition. There are 2 or 3 methods hiding in here.
No transaction manager; no ACID.
No connection pooling; should be passed into this method rather than instantiated in scope.
Should use PreparedStatement and binding rather than concatenating String.
I'm not certain, but it looks like you have three SQL queries:
SELECT all the payer IDs
Get a COUNT of payer IDs
DELETE all the payer IDs in the list you got from SELECT.
Do I read that correctly? If yes, why don't you do it in one query, like this?
DELETE
FROM transformed_table
WHERE payer_id IN (SELECT DISTINCT payer_id FROM transactions)

Avoiding database locks on SQL Server 2012 when updating using Java

I have a web application which is based on SQL Server 2012, and I use Java to update data in the database. (Windows Server 2008, JSP, Tomcat7, Java7)
The relevant code is as follows:
public static synchronized int execute(String dsName, String packageAndFunction, List fields) {
// prepare insertStr
String executeStr = buildStatement(dsName, packageAndFunction, null, fields);
dbConn = DBConnection.getInstance();
Connection conn = dbConn.getConnection();
CallableStatement stmt = null;
int result = RESULT_FAILED;
try {
stmt = conn.prepareCall(executeStr);
// fill statement parameters (each ?)
fillStatement(stmt, fields);
stmt.execute();
result = stmt.getInt(fields.size());
} catch(SQLException e) {
Log.getInstance().write("Exception on executeGeneral (" + packageAndFunction + ") " + e.toString());
} finally {
try {
stmt.close();
dbConn.returnConnection(conn);
} catch(SQLException e) {
Log.getInstance().write("Exception on executeGeneral (" + packageAndFunction + ") " + e.toString());
}
}
return result;
}
About 90% of the time, the code works great. The rest of the time there is some kind of lock on the table which will disappear by itself in perhaps half an hour or so. The lock prevents even simple SELECT queries on the table from executing (in SQL Server Management Studio). In severe cases it has prevented the entire application from working.
I had an idea to use stmt.executeUpdate() instead of stmt.execute(), but I have tried to research this and I do not see any evidence that using stmt.execute() for updating causes locks.
Can anyone help?
Thanks!

It's difficult to diagnose with that code. The next time that it happens, pull up your activity monitor on the SQL server and see what sql command is holding the lock.

Need advice on fixing my JAVA query statement?

My JAVA script consists of 2 JAVA classes: RMS, queryRMS
In the RMS class I call the method in the queryRMS class
RMS Java Class (I left out the start execution part, below is just the method)
for (int i = 1; i <= itemCount; i++) {
GlobalVariables.numberRow = i;
JavaDatapool.settings();
String item = queryRPM.connectDB_Multi(configFile,"SELECT ITEM FROM ORDSKU WHERE ORDER_NO = '" + orderNo + "' ORDER BY ITEM ASC",i);
JavaDatapool.writeXLS("item",item,GlobalVariables.sheetXLS);
sleep(1);
}
queryRMS JAVA class
public static String connectDB_Multi(String configFile, String query, int i) throws FileNotFoundException, IOException, SQLException, ClassNotFoundException{
Properties p = new Properties();
p.load(new FileInputStream(configFile));
String serverName = (p.getProperty("RMS_DBServerName"));
String portNumber = (p.getProperty("RMS_PortNumber"));
String sid = (p.getProperty("RMS_SID"));
String url = "jdbc:oracle:thin:#//" + serverName + ":" + portNumber + "/" + sid;
String username = (p.getProperty("RMS_Username"));
String password = (p.getProperty("RMS_Password"));
// jdbc:oracle:thin:#//localhost:1521/orcl
Class.forName("oracle.jdbc.driver.OracleDriver");
Connection connection = DriverManager.getConnection(url,username,password);
String setr = null;
try {
Statement stmt = connection.createStatement();
try {ResultSet rset = stmt.executeQuery(query);
try {
while(rset.absolute(i))
setr = rset.getString(1);
return setr;
}
finally {
try { rset.close();
}
catch (Exception ignore) {}
}
}
finally {
try { stmt.close();
}
catch (Exception ignore) {}
}
}
finally {
try { connection.close();
}
catch (Exception ignore) {}
}
}
So what it does is call the connectDB_multi class and then returns the String where the next part is saving it inside an Excel worksheet.
The loop should return all rows, one at a time and then save it inside the Excel worksheet.
In the second time in loop the query is faulted, eventhough the query should return 1 column consisting of 2 rows.
the original contained the part while(rset.next()) instead of while(rset.absolute(i))
but next only return the first row everytime. so the script works when only one column and row is retrieved from the Database.

Your logic looks a bit messed up.
Look at the first loop you posted. You are, effectivly, executing:
SELECT ITEM FROM ORDSKU WHERE ORDER_NO = '" + orderNo + "' ORDER BY ITEM ASC
itemCount number of times. Each time you execute it, you are attempting to access the n:th row, n being loop counter. Do you see a problem there? How do you know that the query will return itemCount number of rows? Because if it doesn't, it will fail since you are attempting to access a row that doesn't exist.
What I suspect you WANT to do is something like this
Statement stmt = connection.createStatement();
ResultSet rset = stmt.executeQuery(query);
while(rset.next()) {
JavaDatapool.writeXLS("item",rset.getString(1),GlobalVariables.sheetXLS);
}
You should also seriously consider using some form of connection pooling to avoid having to re-open new connections all the time as that is a pretty time-consuming operation.

This code seems very inefficient, for each row you want to fetch from the database you read a property file, create a connection, select all matching rows, skip ahead to the row you want and return just that row. (Or at least I think that is what you are trying to do).
Your code
while(rset.absolute(i))
setr = rset.getString(1);
is probably an infinite loop as it will continue to go to the same row as long as it is ok to go to that row, so either that row does not exist (and the while exists) or the row does exist (and while continues forever).
You should probably restructure your program to only do one select and read all rows that you want and store them in your excel file. While doing this, you can debug to see if you actually are getting the data you expect.
Apart from the inefficient code of creating new connections and querying once for each row, how do you know how many rows you want?
I think in the end you want something like this
....
while(rset.next()) {
JavaDatapool.writeXLS("item",item,GlobalVariables.sheetXLS);
}
And what is the sleep(1) support to accomplish?

FYI: if you open and close statement too often as your logic or pap's solution, you can get the " java.sql.SQLException: ORA-01000: maximum open cursors exceeded" error message.
I suggest you to not do 'too much generalize'. I saw a lot of OOP programmers overdid generalization and that is painful. You should design by a goal and the goal should not be 'just alignment' nor 'code look beautiful', it has to have a purpose for designing.