I wrote a simple java program to use JDBC to run 20 queries to fetch data from a view
int connectionSize = 10;
ds.init(connectionSize, settings);
ExecutorService executor = Executors.newFixedThreadPool(10);
try {
stopwatch.start();
for (int i = 0; i < 20; i++) {
executor.execute(new Runnable() {
#Override
public void run() {
String sql = String.format("select * from viewA where userId in (%s)", randomUserIds(5));
PreparedStatement ps = null;
Connection conn = null;
try {
while ((conn = ds.getConnection()) == null) {
try {
Thread.sleep(10000);
} catch (InterruptedException e) {
logger.error(e.getMessage(), e);
}
}
logger.info("conn: " + conn.toString());
ps = conn.prepareStatement(sql);
ps.executeQuery();
ps.close();
} catch (Exception ex) {
ex.printStackTrace();
} finally {
if (conn != null) {
ds.returnConnection(conn);
}
}
}
});
}
executor.shutdown();
boolean finished = executor.awaitTermination(10, TimeUnit.MINUTES);
stopwatch.stop();
logger.log(Level.INFO, "Query Complete in " + stopwatch);
} catch (Exception ex) {
ex.printStackTrace();
}
e.g. select * from ViewA where userId in (random few user Ids)
I used a single connection, and inside a for loop executed the 20 queries in a sequential way
I set up 10 connections in a pool and running the 20 queries in 10 threads
I expected the second approach would use less time to finish the 20 queries, but after testing, the results show me these two approaches return similar time consumption.
I can confirm when I was running the second approach, it created 10 sessions in db.
Is the second approach supposed to give a better performance than the first one? What would be the problem to make the second performance same as the first one?
Related
I am trying to have a retry logic for getting JDBC connection in case I get SQL Exception with something like :
int counter = 0;
Connection conn = null;
while (null == conn) {
try {
conn = GetConnectionObject;
} catch (SQLException e) {
if (++counter > MAX_RETRY) {
//Log ERROR
break;
}
} finally {
if (null != conn) {
try {
DbUtils.close(conn);
} catch (SQLException e) {
logger.error("Exception while closing the connection object");
}
}
}
}
I cannot test this currently hence need some help.
This will work fine if I get exception and then I can log after retrying. But if we DO NOT get exception, it will come to the finally block and close the connection. Try-Catch-Finally are inside while loop.
So If I close my connection, flow if reach
while( null== conn)
Will my connection object become null after closing ?
Or If there is some other way around to implement retry part ?
No, it won't become null after closing. Use Connection.isClosed() instead of while( null== conn). Also, you should get //Do some task. out of this code since it's goal is to get a JDBC connection.
Here is the tested method for your problem. This method tries 3 times for the connection and when it will get the DB connection it will Print Success message and will run the query and display the result otherwise it will print the error message. Also if the connection is successful then it will close the connection after executing query in finally block.
public void retryDBConnection(){
int counter = 0;
Connection con = null;
while(counter < 3 && con == null){
try{
String str = "com.microsoft.sqlserver.jdbc.SQLServerDriver";
Class.forName(str).newInstance();
con = DriverManager.getConnection("jdbc:sqlserver://localhost:1433;Database=TestDB;", "sa", "sqldb");
System.out.println("DB Connection Successful!");
PreparedStatement prep = con.prepareStatement("select ID, User_Name from tblUser where ID = 9");
ResultSet rs = prep.executeQuery();
if(rs.next()){
System.out.println("User ID = " + rs.getString(1));
//name = rs.getString(2);
}
}
catch(SQLException e){
// System.out.println(e.getSQLState());
if(e.getErrorCode() == 0 || e.getErrorCode() == 4060)
counter++;
System.out.println("Attempt: " + counter +", Could not establish DB Connection!");
System.out.println("Error Code: " + e.getErrorCode());
}
catch(Exception e){
e.printStackTrace();
}finally{
if(con != null){
try {
con.close();
System.out.println("Connection closed...");
} catch (SQLException e) {
e.printStackTrace();
}
}
}
}
}
Here is the out put of the method.
Attempt: 1, Could not establish DB Connection! Error Code: 0 Attempt: 2, Could not establish DB Connection! Error Code: 4060 DB
Connection Successful! User ID = 9 Connection closed...
My colleague told me that in high concurrency mysql could not process update correctly, e.g.
update product set count = count - 1 where id = ? and count > 0;
maybe have count less than 0, I think he is wrong, so I wrote below code to prove this.
int nThreads = 140; //less than max_connections 151
ExecutorService pool = Executors.newFixedThreadPool(nThreads);
CountDownLatch startLatch = new CountDownLatch(nThreads);
CountDownLatch endLatch = new CountDownLatch(nThreads);
for (int i = 0; i < nThreads; i++) {
pool.submit(() -> {
startLatch.countDown();
try { startLatch.await(); } catch (Exception e1) { } //waiting for all task is submitted to guarantee concurrency
String sql = "update t set count = count-1 where id =1 and count>0";
try {
Connection connection = DriverManager.getConnection(url, user, password);
Statement stat = connection.createStatement();
stat.execute(sql);
endLatch.countDown();
} catch (Exception e) {
e.printStackTrace();
}
});
}
endLatch.await(); //waiting for all task is done
System.out.println("done");
System.exit(0);
I'd like to know my above code is could mock high concurrency correctly? and if could simplify above code by java8?
It is not right that mysql can't update data correctly.
MySql lock the record for the update until the transaction is terminated, so no other thread can try to update it if the previous transaction on the same record has not been finished.
OK, I know that Batch Processing allows to group related SQL statements into a batch and submit them with one call to the database. When you send several SQL statements to the database at once, you reduce the amount of communication overhead, thereby improving performance. In this particular situation (see code below) I don't think batching does it's sole purpose. Cause stmt.executeBatch() is called straight away after adding a batch(?) Wouldn't stmt.executeUpdate() do the same thing?
public void tableChanged(TableModelEvent e) {
int row = e.getFirstRow();
int col = e.getColumn();
model = (MyTableModel) e.getSource();
String stulpPav = model.getColumnName(col);
Object data = model.getValueAt(row, col);
Object studId = model.getValueAt(row, 0);
System.out.println("tableChanded works");
try {
new ImportData(stulpPav, data, studId);
bottomLabel.setText(textForLabel());
} catch (ClassNotFoundException e1) {
e1.printStackTrace();
} catch (SQLException e1) {
e1.printStackTrace();
}
}
public class ImportData {
public ImportData(String a, Object b, Object c)
throws ClassNotFoundException, SQLException {
Statement stmt = null;
try {
connection = TableWithBottomLine.getConnection();
String stulpPav = a;
String duom = b.toString();
String studId = c.toString();
System.out.println(duom);
connection.setAutoCommit(false);
stmt = connection.createStatement();
stmt.addBatch("update finance.fin set " + stulpPav + " = " + duom
+ " where ID = " + studId + ";");
stmt.executeBatch();
connection.commit();
} catch (BatchUpdateException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
} finally {
if (stmt != null)
stmt.close();
connection.setAutoCommit(true);
System.out.println("Data was imported to database");
}
}
}
In this case using batch has no advantage at all. It might even introduce additional overhead over a direct executeUpdate (but that is driver and database dependent).
However, don't assume that batching has advantages with all JDBC drivers. I haven't looked at the specifics of MySQL, but I know there are JDBC drivers where batching internally is a normal execute for each statement in the batch.
The code in your question however has a bigger problem: it is vulnerable to SQL injection.
I have faced a scenario, the task was to read a file which contains 3 Millions IP Address.
There is MySQL Table which contains Id,PrimaryIP, PrimaryIP can by multiple IP separated by #, more over that PrimaryIP can also contain CIDR IP.
So totally, there are 8000 records, each record with multiple IP and CIDR IP.
Now, my task was to read that file, check it against with database and write the matching IP,ID to a file.
Initially, when i run my program, my program failed because: java.lang.OutOfMemoryError: Java heap space, so i have increased it by 3GB, still it was failing, then later i split the file into 6 subfiles, as 0.5 Millions each.
To find CIDR IP List, i have used Apache SubnetUtils.
Below is my code :
public static void main(String[] args) {
String sqlQuery = "SELECT id,PrimaryIP from IPTable where PrimaryIP != '' limit 100000;";
Connection connection = null;
Statement statement = null;
File oFile = new File("output.txt");
System.out.println(new Date());
try{
List<String> fileData = FileUtils.readLines(new File("input.txt"));
System.out.println("File Data Size : "+fileData.size());
Class.forName("com.mysql.jdbc.Driver");
connection = DriverManager.getConnection("jdbc:mysql://localhost/db?user=root&password=pwd");
statement = connection.createStatement();
ResultSet resultSet = statement.executeQuery(sqlQuery);
System.out.println("Started with MySQL Querying");
Map<String, Integer> primaryIPIDMap = new HashMap<String, Integer>();
while (resultSet.next()) {
primaryIPIDMap.clear();
int recordID = resultSet.getInt(1);
if (resultSet.getString(2).contains("#")) {
String primaryIP[] = resultSet.getString(2).split("#");
for (int i = 0; i < primaryIP.length; i++) {
if (primaryIP[i].contains("/")) {
String allIP[] = getAllIP(primaryIP[i]);
for (int allIPi = 0; allIPi < allIP.length; allIPi++) {
primaryIPIDMap.put(allIP[allIPi].intern(), recordID);
}
} else {
primaryIPIDMap.put(primaryIP[i].intern(), recordID);
}
}
} else {
primaryIPIDMap.put(resultSet.getString(2).intern(), recordID);
}
Iterator entries = fileData.iterator();
while (entries.hasNext()) {
String t = (String) entries.next();
if (primaryIPIDMap.containsKey(t)) {
FileUtils.writeStringToFile(oFile, recordID + "," + t);
}
}
primaryIPIDMap.clear();
}
resultSet.close();
statement.close();
connection.close();
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
if (statement != null)
statement.close();
} catch (Exception se2) {
}
try {
if (connection != null)
connection.close();
} catch (Exception se) {
se.printStackTrace();
}
}
System.out.println("Finished");
System.out.println("End Time : "+new Date());
}
private static String[] getAllIP(String ip) {
return new SubnetUtils(ip).getInfo().getAllAddresses();
}
Can someone tell me the best practice to solve this.
Today it just 3 Millions, tomorrow it may be 5 Millions. I can't keep on creating subfile.
I fixed the problem using
Reading the input file line-by-line
I didn't change MySQL Table structure because it has a dependency in many places and table was not designed by me.
Here is my problem, I need to select data, but the data may be a lot so need to select the total results as well, so i need to call PreparedStatement twice for the same fields. I don't want to repeatedly write the same code twice, so I want put PreparedStatement into a different method. See ex:
public Order getOrders(){
Connection myCon = null;
PreparedStatement preparedStmt=null;
try{
myCon =getUnusedConnection();
String sql="select * from order where name=? ....... limit 0,3";
preparedStmt=myCon.prepareStatement(sql);
getOrderPreparedStatement(name,....);
ResultSet results=preparedStmt.executeQuery();
int rowCount=0;
while(results.next()){
......
rowCount++;
}
if(rowCount==3){
String sql2="select count(*) from Order where name=?....";
preparedStmt=myCon.prepareStatement(sql);
getOrderPreparedStatement(name,....);
ResultSet results2=preparedStmt.executeQuery();
if(results2){
int totalRow=....;
}
}
}
catch (SQLException ex) {
while (ex != null) {
System.out.println ("SQL Exception: " +
ex.getMessage ());
ex = ex.getNextException ();
}
}catch (java.lang.Exception e) {
System.out.println("***ERROR-->" + e.toString());
}
finally {
closeStatement(preparedStmt);
releaseConnection(myCon);
}
return null;
}
public void getOrderPreparedStatement(PreparedStatement preparedStmt, String name....)
{
try{
preparedStmt.setString(name);
... a lot of set field here....
}
catch (SQLException ex) {
while (ex != null) {
System.out.println ("SQL Exception: " +
ex.getMessage ());
ex = ex.getNextException ();
}
}catch (java.lang.Exception e) {
System.out.println("***ERROR-->" + e.toString());
}
finally {
closeStatement(preparedStmt); // do i need to close Statement in finally here
}
}
Is it a good practice to put Connection in 1 method & PreparedStatement in a seperated method. If it is ok to do that then "do i need to closeStatement in finally in getOrderPreparedStatement method?"
Or can u suggest a better solution?
Although it is definitely a good idea to move the code for repeated tasks into a method, you need to be very careful when deciding how much code to reuse.
Specifically, in your example you should not attempt to close the statement in the finally clause, because the statement that you prepare would be unusable outside the getOrderPreparedStatement.
One thing that you could do differently is dropping the exception-handling code from the method. This would keep the logic inside the method cleaner, so your readers would have easier time understanding your intentions. This would also reduce code duplication, because currently the code inside the catch (SQLException ex) block is identical.