I have application that reads from excel sheet , number of records more than 25000 records. I calculated time to insert records to database is
15 minutes,currently using MySQL which may change to db2 later on.
I insert all statement direct to MySQL, the time taken is 14 minutes.
Is it normal ? Are there any ways to increase performance? or code enhancement ?
/**
* insert records excel sheeet in tables
* #param dbConnection
* #throws Exception
*/
void insertRecords(Connection dbConnection,Sheet sheet,int sizeColumns ,String tableName) throws Exception {
PreparedStatement preparedStatement = null;
try {
Sheet datatypeSheet =sheet;
Iterator<Row> iterator = datatypeSheet.iterator();
StringBuilder sbInsert = new StringBuilder( 1024 );
//skip first row
iterator.next();
//iterator for rows excel sheet
while (iterator.hasNext()) {
sbInsert.setLength(0);
Row currentRow = iterator.next();
sbInsert.append("insert into "+tableName.trim().replaceAll(" ","_")+" values(");
int currentCellLenght=0;
//iterator for cell rows
for(int cn=0; cn<sizeColumns; cn++) {
Cell currentCell = currentRow.getCell(cn, MissingCellPolicy.CREATE_NULL_AS_BLANK);
currentCell.setCellType(Cell.CELL_TYPE_STRING);
String cellValue;
cellValue=currentCell.getStringCellValue();
sbInsert.append("'"+cellValue.replaceAll("\'", "")+"'");
currentCellLenght++;
if(currentCellLenght==sizeColumns) {
break;
}
//add insert rows
if(currentCellLenght!=sizeColumns) {
sbInsert.append(",");
}
}
sbInsert.append(")");
preparedStatement = dbConnection.prepareStatement(sbInsert.toString());
preparedStatement.execute();
}
} catch (EncryptedDocumentException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
throw new Exception(e.getMessage());
}finally {
if (preparedStatement != null) {
preparedStatement.close();
}
dbConnection.close();
}
}
When you naively hit an InnoDB table in MySQL with a series of insert statements, it automatically commits each statement before it takes the next one. That takes lots of extra time.
You can work around this by doing your inserts in multiple-row chunks.
One way is to chunk your inserts with transactions. At the beginning of your operation, do Connection.setAutoCommit(false);. Then, every few hundred rows do Connection.commit();. Don't forget to do a last Connection.commit(); after all your rows are processed. And, if you'll go on to use the same connection for other things, do Connection.setAutoCommit(true);.
Another way is to issue multi-row inserts. They look something like this.
INSERT INTO table VALUES
(val1, val2, val3, val4),
(val5, val6, val7, val8),
...
(val9, vala, valb, valc);
Each set of values in parentheses is a single row. You can fit ten or even fifty rows in each of these insert statements. This itself is a way of chunking your inserts, because each multirow insert uses just one transaction.
Another way to speed this up (probably an inferior way). Use a MyISAM table rather than InnoDB. MyISAM doesn't have transactions, so it doesn't have the overhead. But transactions are good when you use tables in production.
Chunking makes a big difference to bulk insertion performance problems like yours.
1st with JAVA the 2nd run is always faster because of loads and other initialization. Keep up the good work.
Code Review.
Your evaluating the same thing twice.
You could shave some time here with an else statement.
IRL your iterating for the sizeColumns no need to check for it. 1st if statement not needed.
IRL Do the first column then start the iterations now just put a comma before each value and close out the statement at the end. 2nd if statement is no longer needed.
if(currentCellLenght==sizeColumns) {
break;
}
//add insert rows
if(currentCellLenght!=sizeColumns) {
sbInsert.append(",");
}
Related
I'm trying to create a simple method that receives a ResultSet as a parameter and returns an int that contains the row count of the ResultSet. Is this a valid way of doing this or not so much?
int size = 0;
try {
while(rs.next()){
size++;
}
}
catch(Exception ex) {
System.out.println("------------------Tablerize.getRowCount-----------------");
System.out.println("Cannot get resultSet row count: " + ex);
System.out.println("--------------------------------------------------------");
}
I tried this:
int size = 0;
try {
resultSet.last();
size = resultSet.getRow();
resultSet.beforeFirst();
}
catch(Exception ex) {
return 0;
}
return size;
But I got an error saying
com.microsoft.sqlserver.jdbc.SQLServerException:
The requested operation is not supported on forward only result sets.
If you have access to the prepared statement that results in this resultset, you can use
connection.prepareStatement(sql,
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
This prepares your statement in a way that you can rewind the cursor. This is also documented in the ResultSet Javadoc
In general, however, forwarding and rewinding cursors may be quite inefficient for large result sets. Another option in SQL Server would be to calculate the total number of rows directly in your SQL statement:
SELECT my_table.*, count(*) over () total_rows
FROM my_table
WHERE ...
Statement s = cd.createStatement();
ResultSet r = s.executeQuery("SELECT COUNT(*) AS recordCount FROM FieldMaster");
r.next();
int count = r.getInt("recordCount");
r.close();
System.out.println("MyTable has " + count + " row(s).");
Sometimes JDBC does not support following method gives Error like `TYPE_FORWARD_ONLY' use this solution
Sqlite does not support in JDBC.
resultSet.last();
size = resultSet.getRow();
resultSet.beforeFirst();
So at that time use this solution.
your sql Statement creating code may be like
statement = connection.createStatement();
To solve "com.microsoft.sqlserver.jdbc.SQLServerException: The requested operation is not supported on forward only result sets" exception, change above code with
statement = connection.createStatement(
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
After above change you can use
int size = 0;
try {
resultSet.last();
size = resultSet.getRow();
resultSet.beforeFirst();
}
catch(Exception ex) {
return 0;
}
return size;
to get row count
I just made a getter method.
public int getNumberRows(){
try{
statement = connection.creatStatement();
resultset = statement.executeQuery("your query here");
if(resultset.last()){
return resultset.getRow();
} else {
return 0; //just cus I like to always do some kinda else statement.
}
} catch (Exception e){
System.out.println("Error getting row count");
e.printStackTrace();
}
return 0;
}
Do a SELECT COUNT(*) FROM ... query instead.
Most drivers support forward only resultset - so method like last, beforeFirst etc are not supported.
The first approach is suitable if you are also getting the data in the same loop - otherwise the resultSet has already been iterated and can not be used again.
In most cases the requirement is to get the number of rows a query would return without fetching the rows. Iterating through the result set to find the row count is almost same as processing the data. It is better to do another count(*) query instead.
If you have table and are storing the ID as primary and auto increment then this will work
Example code to get the total row count http://www.java2s.com/Tutorial/Java/0340__Database/GettheNumberofRowsinaDatabaseTable.htm
Below is code
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.Statement;
public class Main {
public static void main(String[] args) throws Exception {
Connection conn = getConnection();
Statement st = conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_UPDATABLE);
st.executeUpdate("create table survey (id int,name varchar(30));");
st.executeUpdate("insert into survey (id,name ) values (1,'nameValue')");
st.executeUpdate("insert into survey (id,name ) values (2,null)");
st.executeUpdate("insert into survey (id,name ) values (3,'Tom')");
st = conn.createStatement();
ResultSet rs = st.executeQuery("SELECT * FROM survey");
rs = st.executeQuery("SELECT COUNT(*) FROM survey");
// get the number of rows from the result set
rs.next();
int rowCount = rs.getInt(1);
System.out.println(rowCount);
rs.close();
st.close();
conn.close();
}
private static Connection getConnection() throws Exception {
Class.forName("org.hsqldb.jdbcDriver");
String url = "jdbc:hsqldb:mem:data/tutorial";
return DriverManager.getConnection(url, "sa", "");
}
}
Your function will return the size of a ResultSet, but its cursor will be set after last record, so without rewinding it by calling beforeFirst(), first() or previous() you won't be able to read its rows, and rewinding methods won't work with forward only ResultSet (you'll get the same exception you're getting in your second code fragment).
Others have already answered how to solve your problem, so I won't repeat what has already been said, but I will says this: you should probably figure out a way to solve your problems without knowing the result set count prior to reading through the results.
There are very few circumstances where the row count is actually needed prior to reading the result set, especially in a language like Java. The only case I think of where a row count would be necessary is when the row count is the only data you need(in which case a count query would be superior). Otherwise, you are better off using a wrapper object to represent your table data, and storing these objects in a dynamic container such as an ArrayList. Then, once the result set has been iterated over, you can get the array list count. For every solution that requires knowing the row count before reading the result set, you can probably think of a solution that does so without knowing the row count before reading without much effort. By thinking of solutions that bypass the need to know the row count before processing, you save the ResultSet the trouble of scrolling to the end of the result set, then back to the beginning (which can be a VERY expensive operation for large result sets).
Now of course I'm not saying there are never situations where you may need the row count before reading a result set. I'm just saying that in most circumstances, when people think they need the result set count prior to reading it, they probably don't, and it's worth taking 5 minutes to think about whether there is another way.
Just wanted to offer my 2 cents on the topic.
Following two options worked for me:
1) A function that returns the number of rows in your ResultSet.
private int resultSetCount(ResultSet resultSet) throws SQLException{
try{
int i = 0;
while (resultSet.next()) {
i++;
}
return i;
} catch (Exception e){
System.out.println("Error getting row count");
e.printStackTrace();
}
return 0;
}
2) Create a second SQL statement with the COUNT option.
The ResultSet has it's methods that move the Cursor back and forth depending on the option provided. By default, it's forward moving(TYPE_FORWARD_ONLY ResultSet type).
Unless CONSTANTS indicating Scrollability and Update of ResultSet properly, you might end up getting an error.
E.g. beforeLast()
This method has no effect if the result set contains no rows.
Throws Error if it's not TYPE_FORWARD_ONLY.
The best way to check if empty rows got fetched --- Just to insert new record after checking non-existence
if( rs.next() ) {
Do nothing
} else {
No records fetched!
}
See here
Here's some code that avoids getting the count to instantiate an array, but uses an ArrayList instead and just before returning converts the ArrayList to the needed array type.
Note that Supervisor class here implements ISupervisor interface, but in Java you can't cast from object[] (that ArrayList's plain toArray() method returns) to ISupervisor[] (as I think you are able to do in C#), so you have to iterate through all list items and populate the result array.
/**
* Get Supervisors for given program id
* #param connection
* #param programId
* #return ISupervisor[]
* #throws SQLException
*/
public static ISupervisor[] getSupervisors(Connection connection, String programId)
throws SQLException
{
ArrayList supervisors = new ArrayList();
PreparedStatement statement = connection.prepareStatement(SQL.GET_SUPERVISORS);
try {
statement.setString(SQL.GET_SUPERVISORS_PARAM_PROGRAMID, programId);
ResultSet resultSet = statement.executeQuery();
if (resultSet != null) {
while (resultSet.next()) {
Supervisor s = new Supervisor();
s.setId(resultSet.getInt(SQL.GET_SUPERVISORS_RESULT_ID));
s.setFirstName(resultSet.getString(SQL.GET_SUPERVISORS_RESULT_FIRSTNAME));
s.setLastName(resultSet.getString(SQL.GET_SUPERVISORS_RESULT_LASTNAME));
s.setAssignmentCount(resultSet.getInt(SQL.GET_SUPERVISORS_RESULT_ASSIGNMENT_COUNT));
s.setAssignment2Count(resultSet.getInt(SQL.GET_SUPERVISORS_RESULT_ASSIGNMENT2_COUNT));
supervisors.add(s);
}
resultSet.close();
}
} finally {
statement.close();
}
int count = supervisors.size();
ISupervisor[] result = new ISupervisor[count];
for (int i=0; i<count; i++)
result[i] = (ISupervisor)supervisors.get(i);
return result;
}
From http://docs.oracle.com/javase/1.4.2/docs/api/java/sql/ResultSetMetaData.html
ResultSet rs = stmt.executeQuery("SELECT a, b, c FROM TABLE2");
ResultSetMetaData rsmd = rs.getMetaData();
int numberOfColumns = rsmd.getColumnCount();
A ResultSet contains metadata which gives the number of rows.
I have one script which fetches around 25.000 different ID values and uses them to make some changes in other table. But the programmer created this code which searches ID (dialid in the code) through the table of 10 million records (line 3) and every query in loop is executing around 1 second. My idea is to fetch last 30 days of records with the SQL and to put it into an array and check only the array.
And my question is, how to do that in Java? Is it the in_array function? I'm solid in PHP, but beginner in Java code...
private Integer getDialId(int predictiveId) {
Integer dialid = null;
StringBuilder sql = new StringBuilder("SELECT dialid from dial where PREDICTIVE_DIALID=");
sql.append(predictiveId); //this predictiveId is calculated in other part of code
ResultSet rsDialId = null;
Statement s1 = null;
try {
s1 = oracle.getConn().createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_UPDATABLE, ResultSet.CLOSE_CURSORS_AT_COMMIT);
rsDialId = s1.executeQuery(String.valueOf(sql));
if (rsDialId.next()) {
dialid = rsDialId.getInt("dialid");
}
} catch (SQLException ex) {
Logger.getLogger(MediatelCdrSync.class.getName()).log(Level.SEVERE, null, ex);
} finally {
try {
if (s1 != null) {
s1.close();
}
if (rsDialId != null) {
rsDialId.close();
}
} catch (SQLException ex) {
Logger.getLogger(MediatelCdrSync.class.getName()).log(Level.SEVERE, null, ex);
}
}
System.out.println("DIALID = " + dialid);
return dialid;
}
Thnx
If you have a performance problem I'd start to see why the query takes one second per execution, if it's database time because the dial table does not have and index on PREDICTIVE_DIALID column you can do very little at the java level.
Anyway the jdbc code reveals some problems especially when used with an oracle database.
The biggest issue is that you are hardcoding your query parameter causing Oracle to re"hard parse" the query every time; the second (minor one) is that the resultset is scrollable and updatable while you need only to load the first row. If you want to make some little modification to your code you should change to somethig like this pseudo code:
PreparedStatement ps =connection.prepareStatement("SELECT dialid from dial where PREDICTIVE_DIALID=?");
for (int i=0;i<10;i++) {//your 25000 loop elements is this one
//this shoudl be the start of the body of your getDialId function that takes also a prepared statement
ps.setInt(1, i);
ResultSet rs=ps.executeQuery();
if (rs.next()) {
rs.getInt("dialid");
}
rs.close();
//your getDialId end here
}
ps.close();
With this minimal java solution you should note a performance increase, but you must chek the performance of the single query since if there is a missing index you cand very little at a java code.
Another solution, more complicated, is to to create a temporart table, fill it with all the 25000 predictiveId values and then issue a query that joins dial and you temporary table; so with one resultset(and one query) you can find all the dialid you need. A jdbc batch insert into the temp table speeds up insertion time noticeably.
If you are planning to fetch less record and store that result in some array then
I think it is better for you to limit your search by creating a view in Database with limited record's (say record for last 2 year's)
And Use that view in your select query
"SELECT dialid from dial_view WHERE PREDICTIVE_DIALID = "
Hope it will help :)
The code is getting data from a queue, adding it to a prepared statement as batch, executes the batch and then commits:
int counter = 0;
while (!this.dataQueue.isEmpty()
&& counter <= config.MAX_FLUSH_COUNT) {
DataRecord record = this.dataQueue.poll();
if (record.ip.length() > 40) {
record.ip = record.ip.substring(0, 40);
}
try {
this.dataFlush.setInt(1, record.uid);
this.dataFlush.setInt(2, record.fid);
this.dataFlush
.setTimestamp(3, new Timestamp(record.time));
this.dataFlush.setString(4, record.ip);
this.dataFlush.addBatch();
} catch (Exception ex) {
Log.logError("Error building record: " + ex.getMessage());
}
counter++;
}
if (counter > 0) {
try {
this.dataFlush.executeBatch();
this.dataFlush.getConnection().commit();
} catch (SQLException ex) {
Log.logError("Error executing flush: " + ex.getMessage());
}
}
This flush might contain thousands of entries (limited to 2000 per batch by MAX_FLUSH_COUNT) so a deadlock sometimes happens (flush is executed every 8 seconds and deadlocks occur around every 5 to 6 hours).
Edit: This is my Prepared statement:
Connection conn = DBBase.getConnection();
conn.setAutoCommit(false);
this.snatchFlush = conn.prepareStatement("INSERT INTO slog (uid, fid, time, ip) VALUES (?, ?, ?, ?) ON DUPLICATE KEY UPDATE fid=VALUES(fid), time=VALUES(time), ip=VALUES(ip)");
uid is the primary key of this table.
Database Engine is InnoDB, no foreign keys, no triggers.
The table might be read from or written to by the webserver as well as by this application at the same time which is very unlikely but possible.
My first change will be to split up the batch into multiple batches and limit their size to 200-500 datasets.
My problem now: When a deadlock occurs, the usual solution is to restart the transaction.
Does this mean I have to re-add the data to the dataFlush Prepared Statement, executeBatch and commit again? If yes, that would mean that I need to save the data polled from the queue because it would be lost otherwise.
Is there any better solution that would not require me to save the polled data? (I am assuming a simple executeBatch and commit in the catch-Block would not work, since executeBatch removes the batch data).
My JAVA script consists of 2 JAVA classes: RMS, queryRMS
In the RMS class I call the method in the queryRMS class
RMS Java Class (I left out the start execution part, below is just the method)
for (int i = 1; i <= itemCount; i++) {
GlobalVariables.numberRow = i;
JavaDatapool.settings();
String item = queryRPM.connectDB_Multi(configFile,"SELECT ITEM FROM ORDSKU WHERE ORDER_NO = '" + orderNo + "' ORDER BY ITEM ASC",i);
JavaDatapool.writeXLS("item",item,GlobalVariables.sheetXLS);
sleep(1);
}
queryRMS JAVA class
public static String connectDB_Multi(String configFile, String query, int i) throws FileNotFoundException, IOException, SQLException, ClassNotFoundException{
Properties p = new Properties();
p.load(new FileInputStream(configFile));
String serverName = (p.getProperty("RMS_DBServerName"));
String portNumber = (p.getProperty("RMS_PortNumber"));
String sid = (p.getProperty("RMS_SID"));
String url = "jdbc:oracle:thin:#//" + serverName + ":" + portNumber + "/" + sid;
String username = (p.getProperty("RMS_Username"));
String password = (p.getProperty("RMS_Password"));
// jdbc:oracle:thin:#//localhost:1521/orcl
Class.forName("oracle.jdbc.driver.OracleDriver");
Connection connection = DriverManager.getConnection(url,username,password);
String setr = null;
try {
Statement stmt = connection.createStatement();
try {ResultSet rset = stmt.executeQuery(query);
try {
while(rset.absolute(i))
setr = rset.getString(1);
return setr;
}
finally {
try { rset.close();
}
catch (Exception ignore) {}
}
}
finally {
try { stmt.close();
}
catch (Exception ignore) {}
}
}
finally {
try { connection.close();
}
catch (Exception ignore) {}
}
}
So what it does is call the connectDB_multi class and then returns the String where the next part is saving it inside an Excel worksheet.
The loop should return all rows, one at a time and then save it inside the Excel worksheet.
In the second time in loop the query is faulted, eventhough the query should return 1 column consisting of 2 rows.
the original contained the part while(rset.next()) instead of while(rset.absolute(i))
but next only return the first row everytime. so the script works when only one column and row is retrieved from the Database.
Your logic looks a bit messed up.
Look at the first loop you posted. You are, effectivly, executing:
SELECT ITEM FROM ORDSKU WHERE ORDER_NO = '" + orderNo + "' ORDER BY ITEM ASC
itemCount number of times. Each time you execute it, you are attempting to access the n:th row, n being loop counter. Do you see a problem there? How do you know that the query will return itemCount number of rows? Because if it doesn't, it will fail since you are attempting to access a row that doesn't exist.
What I suspect you WANT to do is something like this
Statement stmt = connection.createStatement();
ResultSet rset = stmt.executeQuery(query);
while(rset.next()) {
JavaDatapool.writeXLS("item",rset.getString(1),GlobalVariables.sheetXLS);
}
You should also seriously consider using some form of connection pooling to avoid having to re-open new connections all the time as that is a pretty time-consuming operation.
This code seems very inefficient, for each row you want to fetch from the database you read a property file, create a connection, select all matching rows, skip ahead to the row you want and return just that row. (Or at least I think that is what you are trying to do).
Your code
while(rset.absolute(i))
setr = rset.getString(1);
is probably an infinite loop as it will continue to go to the same row as long as it is ok to go to that row, so either that row does not exist (and the while exists) or the row does exist (and while continues forever).
You should probably restructure your program to only do one select and read all rows that you want and store them in your excel file. While doing this, you can debug to see if you actually are getting the data you expect.
Apart from the inefficient code of creating new connections and querying once for each row, how do you know how many rows you want?
I think in the end you want something like this
....
while(rset.next()) {
JavaDatapool.writeXLS("item",item,GlobalVariables.sheetXLS);
}
And what is the sleep(1) support to accomplish?
FYI: if you open and close statement too often as your logic or pap's solution, you can get the " java.sql.SQLException: ORA-01000: maximum open cursors exceeded" error message.
I suggest you to not do 'too much generalize'. I saw a lot of OOP programmers overdid generalization and that is painful. You should design by a goal and the goal should not be 'just alignment' nor 'code look beautiful', it has to have a purpose for designing.
I'm trying to create a simple method that receives a ResultSet as a parameter and returns an int that contains the row count of the ResultSet. Is this a valid way of doing this or not so much?
int size = 0;
try {
while(rs.next()){
size++;
}
}
catch(Exception ex) {
System.out.println("------------------Tablerize.getRowCount-----------------");
System.out.println("Cannot get resultSet row count: " + ex);
System.out.println("--------------------------------------------------------");
}
I tried this:
int size = 0;
try {
resultSet.last();
size = resultSet.getRow();
resultSet.beforeFirst();
}
catch(Exception ex) {
return 0;
}
return size;
But I got an error saying
com.microsoft.sqlserver.jdbc.SQLServerException:
The requested operation is not supported on forward only result sets.
If you have access to the prepared statement that results in this resultset, you can use
connection.prepareStatement(sql,
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
This prepares your statement in a way that you can rewind the cursor. This is also documented in the ResultSet Javadoc
In general, however, forwarding and rewinding cursors may be quite inefficient for large result sets. Another option in SQL Server would be to calculate the total number of rows directly in your SQL statement:
SELECT my_table.*, count(*) over () total_rows
FROM my_table
WHERE ...
Statement s = cd.createStatement();
ResultSet r = s.executeQuery("SELECT COUNT(*) AS recordCount FROM FieldMaster");
r.next();
int count = r.getInt("recordCount");
r.close();
System.out.println("MyTable has " + count + " row(s).");
Sometimes JDBC does not support following method gives Error like `TYPE_FORWARD_ONLY' use this solution
Sqlite does not support in JDBC.
resultSet.last();
size = resultSet.getRow();
resultSet.beforeFirst();
So at that time use this solution.
your sql Statement creating code may be like
statement = connection.createStatement();
To solve "com.microsoft.sqlserver.jdbc.SQLServerException: The requested operation is not supported on forward only result sets" exception, change above code with
statement = connection.createStatement(
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
After above change you can use
int size = 0;
try {
resultSet.last();
size = resultSet.getRow();
resultSet.beforeFirst();
}
catch(Exception ex) {
return 0;
}
return size;
to get row count
I just made a getter method.
public int getNumberRows(){
try{
statement = connection.creatStatement();
resultset = statement.executeQuery("your query here");
if(resultset.last()){
return resultset.getRow();
} else {
return 0; //just cus I like to always do some kinda else statement.
}
} catch (Exception e){
System.out.println("Error getting row count");
e.printStackTrace();
}
return 0;
}
Do a SELECT COUNT(*) FROM ... query instead.
Most drivers support forward only resultset - so method like last, beforeFirst etc are not supported.
The first approach is suitable if you are also getting the data in the same loop - otherwise the resultSet has already been iterated and can not be used again.
In most cases the requirement is to get the number of rows a query would return without fetching the rows. Iterating through the result set to find the row count is almost same as processing the data. It is better to do another count(*) query instead.
If you have table and are storing the ID as primary and auto increment then this will work
Example code to get the total row count http://www.java2s.com/Tutorial/Java/0340__Database/GettheNumberofRowsinaDatabaseTable.htm
Below is code
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.Statement;
public class Main {
public static void main(String[] args) throws Exception {
Connection conn = getConnection();
Statement st = conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_UPDATABLE);
st.executeUpdate("create table survey (id int,name varchar(30));");
st.executeUpdate("insert into survey (id,name ) values (1,'nameValue')");
st.executeUpdate("insert into survey (id,name ) values (2,null)");
st.executeUpdate("insert into survey (id,name ) values (3,'Tom')");
st = conn.createStatement();
ResultSet rs = st.executeQuery("SELECT * FROM survey");
rs = st.executeQuery("SELECT COUNT(*) FROM survey");
// get the number of rows from the result set
rs.next();
int rowCount = rs.getInt(1);
System.out.println(rowCount);
rs.close();
st.close();
conn.close();
}
private static Connection getConnection() throws Exception {
Class.forName("org.hsqldb.jdbcDriver");
String url = "jdbc:hsqldb:mem:data/tutorial";
return DriverManager.getConnection(url, "sa", "");
}
}
Your function will return the size of a ResultSet, but its cursor will be set after last record, so without rewinding it by calling beforeFirst(), first() or previous() you won't be able to read its rows, and rewinding methods won't work with forward only ResultSet (you'll get the same exception you're getting in your second code fragment).
Others have already answered how to solve your problem, so I won't repeat what has already been said, but I will says this: you should probably figure out a way to solve your problems without knowing the result set count prior to reading through the results.
There are very few circumstances where the row count is actually needed prior to reading the result set, especially in a language like Java. The only case I think of where a row count would be necessary is when the row count is the only data you need(in which case a count query would be superior). Otherwise, you are better off using a wrapper object to represent your table data, and storing these objects in a dynamic container such as an ArrayList. Then, once the result set has been iterated over, you can get the array list count. For every solution that requires knowing the row count before reading the result set, you can probably think of a solution that does so without knowing the row count before reading without much effort. By thinking of solutions that bypass the need to know the row count before processing, you save the ResultSet the trouble of scrolling to the end of the result set, then back to the beginning (which can be a VERY expensive operation for large result sets).
Now of course I'm not saying there are never situations where you may need the row count before reading a result set. I'm just saying that in most circumstances, when people think they need the result set count prior to reading it, they probably don't, and it's worth taking 5 minutes to think about whether there is another way.
Just wanted to offer my 2 cents on the topic.
Following two options worked for me:
1) A function that returns the number of rows in your ResultSet.
private int resultSetCount(ResultSet resultSet) throws SQLException{
try{
int i = 0;
while (resultSet.next()) {
i++;
}
return i;
} catch (Exception e){
System.out.println("Error getting row count");
e.printStackTrace();
}
return 0;
}
2) Create a second SQL statement with the COUNT option.
The ResultSet has it's methods that move the Cursor back and forth depending on the option provided. By default, it's forward moving(TYPE_FORWARD_ONLY ResultSet type).
Unless CONSTANTS indicating Scrollability and Update of ResultSet properly, you might end up getting an error.
E.g. beforeLast()
This method has no effect if the result set contains no rows.
Throws Error if it's not TYPE_FORWARD_ONLY.
The best way to check if empty rows got fetched --- Just to insert new record after checking non-existence
if( rs.next() ) {
Do nothing
} else {
No records fetched!
}
See here
Here's some code that avoids getting the count to instantiate an array, but uses an ArrayList instead and just before returning converts the ArrayList to the needed array type.
Note that Supervisor class here implements ISupervisor interface, but in Java you can't cast from object[] (that ArrayList's plain toArray() method returns) to ISupervisor[] (as I think you are able to do in C#), so you have to iterate through all list items and populate the result array.
/**
* Get Supervisors for given program id
* #param connection
* #param programId
* #return ISupervisor[]
* #throws SQLException
*/
public static ISupervisor[] getSupervisors(Connection connection, String programId)
throws SQLException
{
ArrayList supervisors = new ArrayList();
PreparedStatement statement = connection.prepareStatement(SQL.GET_SUPERVISORS);
try {
statement.setString(SQL.GET_SUPERVISORS_PARAM_PROGRAMID, programId);
ResultSet resultSet = statement.executeQuery();
if (resultSet != null) {
while (resultSet.next()) {
Supervisor s = new Supervisor();
s.setId(resultSet.getInt(SQL.GET_SUPERVISORS_RESULT_ID));
s.setFirstName(resultSet.getString(SQL.GET_SUPERVISORS_RESULT_FIRSTNAME));
s.setLastName(resultSet.getString(SQL.GET_SUPERVISORS_RESULT_LASTNAME));
s.setAssignmentCount(resultSet.getInt(SQL.GET_SUPERVISORS_RESULT_ASSIGNMENT_COUNT));
s.setAssignment2Count(resultSet.getInt(SQL.GET_SUPERVISORS_RESULT_ASSIGNMENT2_COUNT));
supervisors.add(s);
}
resultSet.close();
}
} finally {
statement.close();
}
int count = supervisors.size();
ISupervisor[] result = new ISupervisor[count];
for (int i=0; i<count; i++)
result[i] = (ISupervisor)supervisors.get(i);
return result;
}
From http://docs.oracle.com/javase/1.4.2/docs/api/java/sql/ResultSetMetaData.html
ResultSet rs = stmt.executeQuery("SELECT a, b, c FROM TABLE2");
ResultSetMetaData rsmd = rs.getMetaData();
int numberOfColumns = rsmd.getColumnCount();
A ResultSet contains metadata which gives the number of rows.