We Keep Coding

Pagination in Java JDBC

I have a table with millions of records in it. In order to make the system faster, I need to implement the pagination concept in my Java code. I need to fetch just 1000 records at a time and process them, then pick another 1000 records and do my processing and so on. I have already tried a few things and none of them is working. Some of the things I tried are listed below -
1) String query = "select * from TABLENAME" + " WHERE ROWNUM BETWEEN %d AND %d";
sql = String.format(query, firstrow, firstrow + rowcount);
In the above example, when the query is SELECT * from TABLENAME Where ROWNUM BETWEEN 0 and 10 it gives me a result but when the query is SELECT * from TABLENAME Where ROWNUM BETWEEN 10 and 20, it returns an empty result set. I even tried to run it in the DB, it return Empty result set (not sure why!!)
2) preparedStatement.setFetchSize(100); I have that in my Java code, but it still fetches all the records from the table. Adding this statement didnt affect my code in anyway.
Please help!

It sounds like you are not actually needed to paginate the results but to just process the results in batches. If this is the case then all you need to do is set the fetch size to 1000 using setFetchSize and iterate over the resultset as usual (using resultset.next()) and process the results as you iterate. There are many resources describing setFetchSize and what it does. Do some research:
What does Statement.setFetchSize(nSize) method really do in SQL Server JDBC driver?
How JDBC Statement.SetFetchsize exaclty works
What and when should I specify setFetchSize()?
For oracle pagination there are a lot of resources describing how to do this. Just do a web search. Here are a couple of resources that describe how to do it:
http://www.databasestar.com/limit-the-number-of-rows-in-oracle/
http://ocptechnology.com/how-to-use-row-limiting-clause/
Pagination is not very useful if you do not define a consistent ordering (ORDER BY clause) since you cannot rely on the order they are returned.
This answer explains why your BETWEEN statement is not working: https://stackoverflow.com/a/10318244/908961
From the answer if using oracle older than 12c you need to do a sub select to get your results. Something like:
SELECT c.*
FROM (SELECT c.*, ROWNUM as rnum
FROM (SELECT * FROM TABLENAME ORDER BY id) c) c
WHERE c.rnum BETWEEN %d AND %d
If you are using Oracle 12c or greater I would recommend using the newer OFFSET FETCH syntax instead of fiddling with rownum. See the first link above or
http://www.toadworld.com/platforms/oracle/b/weblog/archive/2016/01/23/oracle-12c-enhanced-syntax-for-row-limiting-a-k-a-top-n-queries
So your query would be something like
String query = "select * from TABLENAME OFFSET %d ROWS FETCH NEXT 1000 ONLY";
String.format(query, firstrow);
or using prepared statements
PreparedStatement statement = con.prepareStatement("select * from TABLENAME OFFSET ? ROWS FETCH NEXT 1000 ONLY");
statement.setInt(1, firstrow);
ResultSet rs = statement.executeQuery();
Alternately you can also use the limit keyword as described here http://docs.oracle.com/javadb/10.10.1.2/ref/rrefjdbclimit.html and your query would be something like
String query = "select * from TABLENAME { LIMIT 1000 OFFSET %d }";
String.format(query, firstrow);

The normal way to implement pagination in Oracle is to use an analytic windowing function, e.g. row_number together with an ORDER BY clause that defines the row ordering. The query with the analytic function is then wrapped into an inline view (or a "window"), from which you can query the row numbers you need. Here's an example that queries the first 1000 rows from my_table (ordering by column_to_sort_by):
select rs.* from
(select t.*,
row_number() over (order by column_to_sort_by) as row_num
from my_table t
) rs
where rs.row_num >= 1 and rs.row_num < 1001
order by rs.row_num
A JDBC implementation could then look like the following:
public void queryWithPagination() throws SQLException {
String query = "select rs.* from"
+ " (select t.*,"
+ " row_number() over (order by column_to_sort_by) as row_num"
+ " from my_table t"
+ " ) rs"
+ " where rs.row_num >= ? and rs.row_num < ?"
+ " order by rs.row_num";
final int pageSize = 1000;
int rowIndex = 1;
try (PreparedStatement ps = myConnection.prepareStatement(query)) {
do {
ps.setInt(1, rowIndex);
ps.setInt(2, rowIndex + pageSize);
rowIndex += pageSize;
} while (handleResultSet(ps, pageSize));
}
}
private boolean handleResultSet(PreparedStatement ps, final int pageSize)
throws SQLException {
int rows = 0;
try (ResultSet rs = ps.executeQuery()) {
while (rs.next()) {
/*
* handle rows here
*/
rows++;
}
}
return rows == pageSize;
}
Note that the table should remain unchanged while you're reading it so that the pagination works correctly across different query executions.
If there are so many rows in the table that you're running out of memory, you probably need to purge/serialize your list after some pages have been read.
EDIT:
If the ordering of rows doesn't matter to you at all, then -- as #bdrx mentions in his answer -- you don't probably need pagination, and the fastest solution would be to query the table without a WHERE condition in the SELECT. As suggested, you can adjust the fetch size of the statement to a larger value to improve throughput.

UCanAccess data exception: numeric value out of range

I have a table called [Elenco_Aziende] from which I extract all the record in a resultset. [Elenco_Aziende] is in a relation one to many with other two tables called [Elenco_Autisti] and [Elenco_Veicoli] via a field called [Partita_IVA_Azienda] that is also primary key in [Elenco_Aziende] table.
After extracting all records from [Elenco_Aziende] I perform a loop for each value of [Partita_IVA_Azienda] and then open a new result set to try to read values in related fields of [Elenco_Autisti] and [Elenco_Veicoli] tables and do for each of them some operation.
And here comes strange thing: As long as [Partita_IVA_Azienda] (which is defined as a Text field in the Access DB) values are all the same length everything is OK when I try to read in [Elenco_Autisti] and [Elenco_Veicoli] tables, but if some of [Partita_IVA_Azienda] has a different length then I got error:
net.ucanaccess.jdbc.UcanaccessSQLException: UCAExc:::3.0.1 data exception: numeric value out of range
More precisely here is the nested loop scenario:
Connection con = DriverManager.getConnection("jdbc:ucanaccess://"
+ filepath);
String qry = "SELECT * FROM Elenco_Aziende";
ResultSet rs = stmt.executeQuery(qry);
String cognometest = "";
String nometest ="";
while (rs.next()) {
String partitaiva = "Partita IVA: "
+ rs.getString("Partita_IVA_Azienda") + "\n\r";
String partitaivazienda = rs.getString("Partita_IVA_Azienda");
Statement stmtautisti = con.createStatement();
System.out.println("Sto per eseguire la query per partita iva azienda = " + partitaivazienda + "\n\r");
String qryautisti = "SELECT * FROM Elenco_Autisti WHERE Partita_IVA_Azienda="
+ partitaivazienda; /*!!!!! AND HERE WHEN I EXECUTE NEXT QUERY IS WHERE I GET THE EXCEPTION net.ucanaccess.jdbc.UcanaccessSQLException: UCAExc:::3.0.1 data exception: numeric value out of range more!!!!!*/
ResultSet rsautisti = stmtautisti.executeQuery(qryautisti);
while (rsautisti.next()) {
do something here
}
Statement stmtveicoli = con.createStatement();
String qryveicoli = "SELECT * FROM Elenco_Veicoli WHERE Partita_IVA_Azienda="
+ rs.getString("Partita_IVA_Azienda");
ResultSet rsveicoli = stmtveicoli.executeQuery(qryveicoli);
while (rsveicoli.next()) {
do something else here
}
that is as soon as I execute the query
String qryautisti = "SELECT * FROM Elenco_Autisti WHERE Partita_IVA_Azienda="+ partitaivazienda;
for a different length value of [Partita_IVA_Azienda] I get the problem.
I even tried to export the database in a comma separated value and reimporting it in a brand new one but it did not help. Furthermore, the problem seems to happen just for large number of records in tables [Elenco_Autisti] (138 records) and [Elenco_Veicoli] (287 records), while seems not to happen for small number of records. [Elenco_Aziende] is small (no more than 10 records).

According to the little of what I know about SQL, a WHERE with a text field should be written with the value in apostrophes:
String qryautisti = "SELECT * FROM Elenco_Autisti WHERE Partita_IVA_Azienda='"
+ partitaivazienda + "'";

Using a database API cursor with JDBC and SQLServer to select batch results

SOLVED (See answer below.)
I did not understand my problem within the proper context. The real issue was that my query was returning multiple ResultSet objects, and I had never come across that before. I have posted code below that solves the problem.
PROBLEM
I have an SQL Server database table with many thousand rows. My goal is to pull the data back from the source database and write it to a second database. Because of application memory constraints, I will not be able to pull the data back all at once. Also, because of this particular table's schema (over which I have no control) there is no good way for me to tick off the rows using some sort of ID column.
A gentleman over at the Database Administrators StackExchange helped me out by putting together something called a database API cursor, and basically wrote this complicated query that I only need to drop my statement into. When I run the query in SQL Management Studio (SSMS) it works great. I get all the data back, a thousand rows at a time.
Unfortunately, when I try to translate this into JDBC code, I get back the first thousand rows only.
QUESTION
Is it possible using JDBC to retrieve a database API cursor, pull the first set of rows from it, allow the cursor to advance, and then pull the subsequent sets one at a time? (In this case, a thousand rows at a time.)
SQL CODE
This gets complicated, so I'm going to break it up.
The actual query can be simple or complicated. It doesn't matter. I've tried several different queries during my experimentation and they all work. You just basically drop it into the the SQL code in the appropriate place. So, let's take this simple statement as our query:
SELECT MyColumn FROM MyTable;
The actual SQL database API cursor is far more complicated. I will print it out below. You can see the above query buried in it:
-- http://dba.stackexchange.com/a/82806
DECLARE #cur INTEGER
,
-- FAST_FORWARD | AUTO_FETCH | AUTO_CLOSE
#scrollopt INTEGER = 16 | 8192 | 16384
,
-- READ_ONLY, CHECK_ACCEPTED_OPTS, READ_ONLY_ACCEPTABLE
#ccopt INTEGER = 1 | 32768 | 65536
,#rowcount INTEGER = 1000
,#rc INTEGER;
-- Open the cursor and return the first 1,000 rows
EXECUTE #rc = sys.sp_cursoropen #cur OUTPUT
,'SELECT MyColumn FROM MyTable'
,#scrollopt OUTPUT
,#ccopt OUTPUT
,#rowcount OUTPUT;
IF #rc <> 16 -- FastForward cursor automatically closed
BEGIN
-- Name the cursor so we can use CURSOR_STATUS
EXECUTE sys.sp_cursoroption #cur
,2
,'MyCursorName';
-- Until the cursor auto-closes
WHILE CURSOR_STATUS('global', 'MyCursorName') = 1
BEGIN
EXECUTE sys.sp_cursorfetch #cur
,2
,0
,1000;
END;
END;
As I've said, the above creates a cursor in the database and asks the database to execute the statement, keep track (internally) of the data it's returning, and return the data a thousand rows at a time. It works great.
JDBC CODE
Here's where I'm having the problem. I have no compilation problems or run-time problems with my Java code. The problem I am having is that it returns only the first thousand rows. I don't understand how to utilize the database cursor properly. I have tried variations on the Java basics:
// Hoping to get all of the data, but I only get the first thousand.
ResultSet rs = stmt.executeQuery(fq.getQuery());
while (rs.next()) {
System.out.println(rs.getString("MyColumn"));
}
I'm not surprised by the results, but all of the variations I've tried produce the same results.
From my research it seems like the JDBC does something with database cursors when the database is Oracle, but you have to set the data type returned in the result set as an Oracle cursor object. I'm guessing there is something similar with SQL Server, but I have been unable to find anything yet.
Does anyone know of a way?
I'm including example Java code in full (as ugly as that gets).
// FancyQuery.java
import java.sql.*;
public class FancyQuery {
// Adapted from http://dba.stackexchange.com/a/82806
String query = "DECLARE #cur INTEGER\n"
+ " ,\n"
+ " -- FAST_FORWARD | AUTO_FETCH | AUTO_CLOSE\n"
+ " #scrollopt INTEGER = 16 | 8192 | 16384\n"
+ " ,\n"
+ " -- READ_ONLY, CHECK_ACCEPTED_OPTS, READ_ONLY_ACCEPTABLE\n"
+ " #ccopt INTEGER = 1 | 32768 | 65536\n"
+ " ,#rowcount INTEGER = 1000\n"
+ " ,#rc INTEGER;\n"
+ "\n"
+ "-- Open the cursor and return the first 1,000 rows\n"
+ "EXECUTE #rc = sys.sp_cursoropen #cur OUTPUT\n"
+ " ,'SELECT MyColumn FROM MyTable;'\n"
+ " ,#scrollopt OUTPUT\n"
+ " ,#ccopt OUTPUT\n"
+ " ,#rowcount OUTPUT;\n"
+ " \n"
+ "IF #rc <> 16 -- FastForward cursor automatically closed\n"
+ "BEGIN\n"
+ " -- Name the cursor so we can use CURSOR_STATUS\n"
+ " EXECUTE sys.sp_cursoroption #cur\n"
+ " ,2\n"
+ " ,'MyCursorName';\n"
+ "\n"
+ " -- Until the cursor auto-closes\n"
+ " WHILE CURSOR_STATUS('global', 'MyCursorName') = 1\n"
+ " BEGIN\n"
+ " EXECUTE sys.sp_cursorfetch #cur\n"
+ " ,2\n"
+ " ,0\n"
+ " ,1000;\n"
+ " END;\n"
+ "END;\n";
public String getQuery() {
return this.query;
}
public static void main(String[ ] args) throws Exception {
String dbUrl = "jdbc:sqlserver://tc-sqlserver:1433;database=MyBigDatabase";
String user = "mario";
String password = "p#ssw0rd";
String driver = "com.microsoft.sqlserver.jdbc.SQLServerDriver";
FancyQuery fq = new FancyQuery();
Class.forName(driver);
Connection conn = DriverManager.getConnection(dbUrl, user, password);
Statement stmt = conn.createStatement();
// We expect to get 1,000 rows at a time.
ResultSet rs = stmt.executeQuery(fq.getQuery());
while (rs.next()) {
System.out.println(rs.getString("MyColumn"));
}
// Alas, we've only gotten 1,000 rows, total.
rs.close();
stmt.close();
conn.close();
}
}

I figured it out.
stmt.execute(fq.getQuery());
ResultSet rs = null;
for (;;) {
rs = stmt.getResultSet();
while (rs.next()) {
System.out.println(rs.getString("MyColumn"));
}
if ((stmt.getMoreResults() == false) && (stmt.getUpdateCount() == -1)) {
break;
}
}
if (rs != null) {
rs.close();
}
After some additional googling, I found a bit of code posted back in 2004:
http://www.coderanch.com/t/300865/JDBC/databases/SQL-Server-JDBC-Registering-cursor
The gentleman who posted the snippet that I found helpful (Julian Kennedy) suggested: "Read the Javadoc for getUpdateCount() and getMoreResults() for a clear understanding." I was able to piece it together from that.
Basically, I don't think I understood my problem well enough at the outset in order to phrase it correctly. What it comes down to is that my query will be returning the data in multiple ResultSet instances. What I needed was a way to not merely iterate through each row in a ResultSet but, rather, iterate through the entire set of ResultSets. That's what the code above does.

If you want all records from the table, just do "Select * from table".
The only reason to retrieve in chunks is if there is some intermediate place for the data: e.g. if you are showing it on the screen, or storing it in memory.
If you are simply reading from one and inserting to another, just read everything from the first.You will not get any better performance by trying to retrieve in batches. If there is a difference, it will be negative. Frame your query in a way that brings back everything. The JDBC software will handle all the other breaking-up and reconstituting that you need.
However, you should batch the update/insert side of things.
The set-up would create two statements on the two connections:
Statement stmt = null;
ResultSet rs = null;
PreparedStatement insStmt = null;
stmt = conDb1.createStatement();
insStmt = conDb2.prepareStament("insert into tgt_db2_table (?,?,?,?,?......etc. ?,?) ");
rs = stmt.executeQuery("select * from src_db1_table");
Then, loop over the select as normal, but use batching on the target.
int batchedRecordCount = 0;
while (rs.next()) {
System.out.println(rs.getString("MyColumn"));
//Here you read values from the cursor and set them to the insStmt ...
String field1 = rs.getString(1);
String field2 = rs.getString(2);
int field3 = rs.getInt(3);
//--- etc.
insStmt.setString(1, field1);
insStmt.setString(2, field2);
insStmt.setInt(3, field3);
//----- etc. for all the fields
batchedRecordCount++;
insStmt.addBatch();
if (batchRecordCount > 1000) {
insStmt.executeBatch();
}
}
if (batchRecordCount > 0) {
//Finish of the final (partial) set of records
insStmt.executeBatch();
}
//Close resources...

ResultSet to ArrayList<String[]>

I want to create an array for each row.
while (result.next()) {
String[] result1 = {
"Dog ID: " + result.getLong("dogs.id"),
", Dog name: " + result.getString("dogs.first_name"),
", Owner name: "
+ result.getString("owners.first_name"),
", Owner phone: " + result.getString("owners.phone") };
resultList.add(result1);
My code write every one row in one array.
Can i get numbers of columns and put a limit?
while (resultset.next()) {
int i = 1;
while(i <= numberOfColumns) {
It's because i can't send entire table as a result from server to client.

You can query by column number result.getLong(columnIndex) but it doesn't make sense in your case withing a loop because you have columns of different types (unless complicating the code).
If you want to optimize the traffic from server to client the way to go is querying for just the columns you need.

If you want to limit the rows returned, it might be better to put the limiting criteria into the SQL query and only return the rows you want to include.

In order to get number of columns in your ResultSet you can use the following piece of code :
Statement stat = conn.createStatement();
ResultSet rs = stat.executeQuery(myQuery);
ResultSetMetaData metaData = rs.getMetaData();
int numOfColumns = metaData .getColumnCount();

Error: Before start of result set in Java

I know this would be a foolish question to ask but still i need to do this.
This is a basic program in java application where I want to use 3 queries simultaneously to print the table.
(I'm not using any Primary key in this case so please help me to resolve this without making my attributes as primary keys - I know this is not a good practice but for now i need to complete it.)
my code:
Connection con = null;
Statement stat1 = null, stat2 = null, stat3 = null;
ResultSet rs1, rs2, rs3;
stat1 = con.createStatement();
stat2 = con.createStatement();
stat3 = con.createStatement();
String str = "\nProduct\tC.P\tS.P.\tStock\tExpenditure\tSales";
info.setText(str);
String s1 = "SELECT type, cp, sp, stock FROM ts_items GROUP BY type ORDER BY type";
String s2 = "SELECT expenditure FROM ts_expenditure GROUP BY type ORDER BY type";
String s3 = "SELECT sales FROM ts_sales GROUP BY type ORDER BY type";
rs1 = stat1.executeQuery(s1);
rs2 = stat2.executeQuery(s2);
rs3 = stat3.executeQuery(s3);
String type;
int cp, sp, stock, expenditure, sales;
while( rs1.next() || rs2.next() || rs3.next() )
{
type = rs1.getString("type");
cp = rs1.getInt("cp");
sp = rs1.getInt("sp");
stock = rs1.getInt("stock");
expenditure = rs2.getInt("expenditure");
sales = rs3.getInt("sales");
info.append("\n" + type + "\t" + cp + "\t" + sp + "\t" + stock + "\t" + expenditure + "\t" + sales);
}
Output:
Runtime Exception: Before start of result set

This is the problem:
while( rs1.next() || rs2.next() || rs3.next() )
If rs1.next() returns true, rs2.next() and rs3.next() won't be called due to short-circuiting. So rs2 and rs3 will both be before the first row. And if rs1.next() returns false, then you couldn't read from that anyway...
I suspect you actually want:
while (rs1.next() && rs2.next() && rs3.next())
After all, you only want to keep going while all three result sets have more information, right?
It's not clear why you're not doing an appropriate join, to be honest. That would make a lot more sense to me... Then you wouldn't be trying to use multiple result sets on a single connection, and you wouldn't be relying on there being the exact same type values in all the different tables.

You do an OR so imagine only one ResultSet has a result.
What you end up with is trying to read from empty result sets.

Suppose rs1 has one result and rs3 has 3 results. Now as per your code it will fail for rs1.getString("type"); during second iteration.
Better to loop over each resultSet separately.

This is going to go badly wrong, in the event that there is a type value that's missing from one of your three tables. Your code just assumes you'll get all of the types from all of the tables. It may be the case for your current data set, but it means that your code is not at all robust.
I would seriously recommend having just one SQL statement, that has each of your three selects as subselects, then joins them all together. Your java can just iterate over the result from this one SQL statement.