We Keep Coding

Stream result of JDBC query with JDBCTemplate

I'm using spring java.
I need to return a Stream of objects from the database query (I use ObjectMapper to map them to JSON).
The query results can be quite large (over 500k objects) so I don't want to store them in memory.
I've done it with JpaRepository.
I'd like to know how it can be done with JdbcTemplate and if there would be an advantage to do so?
I.e ... can we optimize the throughput and memory usage using JdbcTemplate or perhaps even other libraries.
My goal is really at the end to find the optimal way to run a query and print all objects to an output stream in terms of memory/time/processing.

Yes, there will be an advantage with streams since it is a common abstraction for processing data without all of it being in memory. E.g. passing the stream to a HTTP response.
Spring 5.3
If you use Spring 5.3, there is a convenient method JdbcTemplate.queryForStream() that can be used like this:
String sql = "select * from table";
Stream<Person > stream = jdbcTemplate.queryForStream(sql, (resultSet, rowNum) -> {
return new Person(resultSet.getInt(1), resultSet.getString(2));
});
Previous Spring versions
Older versions of JDBCTemplate does not have functionality for streams directly. But you can make a stream by using the underlying database connection:
String sql = "select * from table";
Connection connection = jdbcTemplate.getDataSource().getConnection();
PreparedStatement statement = connection.prepareStatement(sql);
ResultSet resultSet = statement.executeQuery();
PersonMapper personMapper = new PersonMapper();
Spliterator<Person> spliterator =
Spliterators.spliteratorUnknownSize(
new Iterator<Person>() {
#Override public boolean hasNext() {
try {
return !resultSet.isAfterLast();
} catch (SQLException e) {
throw new RuntimeException(e);
}
}
#Override public Person next() {
try {
if (resultSet.isBeforeFirst()) {
resultSet.next();
}
Person result = new Person(resultSet.getInt(1), resultSet.getString(2));
resultSet.next();
return result;
} catch (SQLException e) {
throw new RuntimeException(e);
}
}
},
Spliterator.IMMUTABLE);
Runnable closer = () -> {
try {
resultSet.close();
statement.close();
connection.close();
} catch (SQLException e) {
throw new RuntimeException(e);
}
};
Stream<Person> = StreamSupport.stream(spliterator, false).onClose(closer);

Look at all the query() methods of JdbcTemplate.
The ones that has a return type of void obviously don't build up the full data set in memory. They actually all take a RowCallbackHandler:
An interface used by JdbcTemplate for processing rows of a ResultSet on a per-row basis.
The processRow() callback method can then add the data to the streaming JSON text, ensuring minimal memory use.
That of course assumes that the JDBC driver doesn't load the entire ResultSet into memory, but that's a different issue, so solution (if needed) entirely depends on the JDBC driver in use.

Java SE8 JDBC trouble [duplicate]

I'm trying to create a simple method that receives a ResultSet as a parameter and returns an int that contains the row count of the ResultSet. Is this a valid way of doing this or not so much?
int size = 0;
try {
while(rs.next()){
size++;
}
}
catch(Exception ex) {
System.out.println("------------------Tablerize.getRowCount-----------------");
System.out.println("Cannot get resultSet row count: " + ex);
System.out.println("--------------------------------------------------------");
}
I tried this:
int size = 0;
try {
resultSet.last();
size = resultSet.getRow();
resultSet.beforeFirst();
}
catch(Exception ex) {
return 0;
}
return size;
But I got an error saying
com.microsoft.sqlserver.jdbc.SQLServerException:
The requested operation is not supported on forward only result sets.

If you have access to the prepared statement that results in this resultset, you can use
connection.prepareStatement(sql,
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
This prepares your statement in a way that you can rewind the cursor. This is also documented in the ResultSet Javadoc
In general, however, forwarding and rewinding cursors may be quite inefficient for large result sets. Another option in SQL Server would be to calculate the total number of rows directly in your SQL statement:
SELECT my_table.*, count(*) over () total_rows
FROM my_table
WHERE ...

Statement s = cd.createStatement();
ResultSet r = s.executeQuery("SELECT COUNT(*) AS recordCount FROM FieldMaster");
r.next();
int count = r.getInt("recordCount");
r.close();
System.out.println("MyTable has " + count + " row(s).");
Sometimes JDBC does not support following method gives Error like `TYPE_FORWARD_ONLY' use this solution
Sqlite does not support in JDBC.
resultSet.last();
size = resultSet.getRow();
resultSet.beforeFirst();
So at that time use this solution.

your sql Statement creating code may be like
statement = connection.createStatement();
To solve "com.microsoft.sqlserver.jdbc.SQLServerException: The requested operation is not supported on forward only result sets" exception, change above code with
statement = connection.createStatement(
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
After above change you can use
int size = 0;
try {
resultSet.last();
size = resultSet.getRow();
resultSet.beforeFirst();
}
catch(Exception ex) {
return 0;
}
return size;
to get row count

I just made a getter method.
public int getNumberRows(){
try{
statement = connection.creatStatement();
resultset = statement.executeQuery("your query here");
if(resultset.last()){
return resultset.getRow();
} else {
return 0; //just cus I like to always do some kinda else statement.
}
} catch (Exception e){
System.out.println("Error getting row count");
e.printStackTrace();
}
return 0;
}

Do a SELECT COUNT(*) FROM ... query instead.

Most drivers support forward only resultset - so method like last, beforeFirst etc are not supported.
The first approach is suitable if you are also getting the data in the same loop - otherwise the resultSet has already been iterated and can not be used again.
In most cases the requirement is to get the number of rows a query would return without fetching the rows. Iterating through the result set to find the row count is almost same as processing the data. It is better to do another count(*) query instead.

If you have table and are storing the ID as primary and auto increment then this will work
Example code to get the total row count http://www.java2s.com/Tutorial/Java/0340__Database/GettheNumberofRowsinaDatabaseTable.htm
Below is code
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.Statement;
public class Main {
public static void main(String[] args) throws Exception {
Connection conn = getConnection();
Statement st = conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_UPDATABLE);
st.executeUpdate("create table survey (id int,name varchar(30));");
st.executeUpdate("insert into survey (id,name ) values (1,'nameValue')");
st.executeUpdate("insert into survey (id,name ) values (2,null)");
st.executeUpdate("insert into survey (id,name ) values (3,'Tom')");
st = conn.createStatement();
ResultSet rs = st.executeQuery("SELECT * FROM survey");
rs = st.executeQuery("SELECT COUNT(*) FROM survey");
// get the number of rows from the result set
rs.next();
int rowCount = rs.getInt(1);
System.out.println(rowCount);
rs.close();
st.close();
conn.close();
}
private static Connection getConnection() throws Exception {
Class.forName("org.hsqldb.jdbcDriver");
String url = "jdbc:hsqldb:mem:data/tutorial";
return DriverManager.getConnection(url, "sa", "");
}
}

Your function will return the size of a ResultSet, but its cursor will be set after last record, so without rewinding it by calling beforeFirst(), first() or previous() you won't be able to read its rows, and rewinding methods won't work with forward only ResultSet (you'll get the same exception you're getting in your second code fragment).

Others have already answered how to solve your problem, so I won't repeat what has already been said, but I will says this: you should probably figure out a way to solve your problems without knowing the result set count prior to reading through the results.
There are very few circumstances where the row count is actually needed prior to reading the result set, especially in a language like Java. The only case I think of where a row count would be necessary is when the row count is the only data you need(in which case a count query would be superior). Otherwise, you are better off using a wrapper object to represent your table data, and storing these objects in a dynamic container such as an ArrayList. Then, once the result set has been iterated over, you can get the array list count. For every solution that requires knowing the row count before reading the result set, you can probably think of a solution that does so without knowing the row count before reading without much effort. By thinking of solutions that bypass the need to know the row count before processing, you save the ResultSet the trouble of scrolling to the end of the result set, then back to the beginning (which can be a VERY expensive operation for large result sets).
Now of course I'm not saying there are never situations where you may need the row count before reading a result set. I'm just saying that in most circumstances, when people think they need the result set count prior to reading it, they probably don't, and it's worth taking 5 minutes to think about whether there is another way.
Just wanted to offer my 2 cents on the topic.

Following two options worked for me:
1) A function that returns the number of rows in your ResultSet.
private int resultSetCount(ResultSet resultSet) throws SQLException{
try{
int i = 0;
while (resultSet.next()) {
i++;
}
return i;
} catch (Exception e){
System.out.println("Error getting row count");
e.printStackTrace();
}
return 0;
}
2) Create a second SQL statement with the COUNT option.

The ResultSet has it's methods that move the Cursor back and forth depending on the option provided. By default, it's forward moving(TYPE_FORWARD_ONLY ResultSet type).
Unless CONSTANTS indicating Scrollability and Update of ResultSet properly, you might end up getting an error.
E.g. beforeLast()
This method has no effect if the result set contains no rows.
Throws Error if it's not TYPE_FORWARD_ONLY.
The best way to check if empty rows got fetched --- Just to insert new record after checking non-existence
if( rs.next() ) {
Do nothing
} else {
No records fetched!
}
See here

Here's some code that avoids getting the count to instantiate an array, but uses an ArrayList instead and just before returning converts the ArrayList to the needed array type.
Note that Supervisor class here implements ISupervisor interface, but in Java you can't cast from object[] (that ArrayList's plain toArray() method returns) to ISupervisor[] (as I think you are able to do in C#), so you have to iterate through all list items and populate the result array.
/**
* Get Supervisors for given program id
* #param connection
* #param programId
* #return ISupervisor[]
* #throws SQLException
*/
public static ISupervisor[] getSupervisors(Connection connection, String programId)
throws SQLException
{
ArrayList supervisors = new ArrayList();
PreparedStatement statement = connection.prepareStatement(SQL.GET_SUPERVISORS);
try {
statement.setString(SQL.GET_SUPERVISORS_PARAM_PROGRAMID, programId);
ResultSet resultSet = statement.executeQuery();
if (resultSet != null) {
while (resultSet.next()) {
Supervisor s = new Supervisor();
s.setId(resultSet.getInt(SQL.GET_SUPERVISORS_RESULT_ID));
s.setFirstName(resultSet.getString(SQL.GET_SUPERVISORS_RESULT_FIRSTNAME));
s.setLastName(resultSet.getString(SQL.GET_SUPERVISORS_RESULT_LASTNAME));
s.setAssignmentCount(resultSet.getInt(SQL.GET_SUPERVISORS_RESULT_ASSIGNMENT_COUNT));
s.setAssignment2Count(resultSet.getInt(SQL.GET_SUPERVISORS_RESULT_ASSIGNMENT2_COUNT));
supervisors.add(s);
}
resultSet.close();
}
} finally {
statement.close();
}
int count = supervisors.size();
ISupervisor[] result = new ISupervisor[count];
for (int i=0; i<count; i++)
result[i] = (ISupervisor)supervisors.get(i);
return result;
}

From http://docs.oracle.com/javase/1.4.2/docs/api/java/sql/ResultSetMetaData.html
ResultSet rs = stmt.executeQuery("SELECT a, b, c FROM TABLE2");
ResultSetMetaData rsmd = rs.getMetaData();
int numberOfColumns = rsmd.getColumnCount();
A ResultSet contains metadata which gives the number of rows.

How to improve performance of JDBC query execution?

I'm running a bot on Discord that receives a lot of requests to the MySQL database, and recently MySQL has started blocking threads, causing major delays in the program.
After dumping the thread, I've found that the problematic line resides within the PreparedStatement code from JDBC, but I'm really not sure what could be causing this issue.
The code block below is where the error occurs:
public List<HashMap<String, Object>> find(String haystack, Object... needles){
PreparedStatement prep = null;
List<HashMap<String, Object>> results = new ArrayList<>();
ResultSet rs = null;
try{
prep = connection.prepareStatement(haystack);
for(int i = 0; i < needles.length; i++){
prep.setObject(i+1, needles[i]);
}
rs = prep.executeQuery();
while(rs.next()){
HashMap<String, Object> result = new HashMap<>();
for(int i = 1; i < rs.getMetaData().getColumnCount() + 1; i++){
result.put(rs.getMetaData().getColumnName(i), rs.getObject(i));
}
results.add(result);
}
}catch(SQLException e){
System.out.println("MySQL > Unable to execute query: " + e.getMessage());
}finally{
try{
if(rs!=null)rs.close();
if(prep!=null)prep.close();
}catch(SQLException e){
System.out.println("(find) Error closing: " + e.getMessage());
}
}
return results;
}
with rs = prep.executeQuery(); being the problematic line of code.
Is there any way to stop MySQL from blocking threads?

I see that you are using only one connection throughout the application. You should create a pool of connections if you have a large number of request to handle which can be done using following approaches:
You can create connection pooling on the application side. You can use apache connection pool.
You can create connection pooling on the server end. Read this.
Best, use hibernate, there you have a property called hibernate.connection.pool_size.
If you are using JDBC prepared statement then use batch processing and avoid using Statement (which I see you are doing) because of the reason mentioned in this post.

DB MetaData change - how to create a dynamic sql?

The db tables that I am using are changing very often meaning new column can be add which will reflect my sql’s.
The solution that I was thinking is first “read” the meta data into some map and use it in order to retrieve the values something like this.
Read meta data:
public class Dynamic {
static public final Map<Integer, String> metadata = initMetaData();
HashMap<String, String> data = new HashMap<String, String>();
private static Map<Integer, String> initMetaData() {
Map<Integer, String> tmpMap = new HashMap<Integer, String>();
try {
Connection connection = DBConnection.getConnection();
try {
Statement stmt = connection.createStatement();
ResultSet result = stmt.executeQuery("SELECT * FROM TMP WHERE ROWNUM = 1");
for (int i = 1; i <= result.getMetaData().getColumnCount(); i++) {
tmpMap.put(new Integer(i), result.getMetaData().getColumnName(i));
}
} finally {
connection.close();
}
} catch (SQLException ex) {
…..
}
return Collections.unmodifiableMap(tmpMap);
}
public static String getColumnName(Integer index) {
return metadata.get(index);
}
And when running the sql:
public static void test()
try {
Connection connection = DBConnection.getConnection()
try {
Statement stmt = connection.createStatement();
ResultSet result = stmt.executeQuery("SELECT * FROM TMP where idx = 'R5'");
while (result.next()) {
Dynamic d = new Dynamic()
for (int i = 1; i <= Dynamic.metadata.size(); i++) {
d.setData(Dynamic.getColumnName(i),result.getString(Dynamic.getColumnName(i)));
}
}
In this approach I have two problems(that I notice):
1) I need to execute two loops
2) I don’t know which get function to use via resultset since the type can also change.
How can I overcome those problems ?
I would also appreciate to get some other suggestion maybe there is a simple why
Thanks

1) What is your alternative to the inner loop? How would you get the field values? Do you think there is a lot of overhead in looping over small number of integers?
2) You can get extra information about field data type from the same metadata where you get the field name and map the method names accordingly.
3) You should really create a map for multiple tables - table/fieldSeq/fieldType/methodName and maybe few extra details - you don't have to get them all dynamically all the time.

How to get row count using ResultSet in Java?

I'm trying to create a simple method that receives a ResultSet as a parameter and returns an int that contains the row count of the ResultSet. Is this a valid way of doing this or not so much?
int size = 0;
try {
while(rs.next()){
size++;
}
}
catch(Exception ex) {
System.out.println("------------------Tablerize.getRowCount-----------------");
System.out.println("Cannot get resultSet row count: " + ex);
System.out.println("--------------------------------------------------------");
}
I tried this:
int size = 0;
try {
resultSet.last();
size = resultSet.getRow();
resultSet.beforeFirst();
}
catch(Exception ex) {
return 0;
}
return size;
But I got an error saying
com.microsoft.sqlserver.jdbc.SQLServerException:
The requested operation is not supported on forward only result sets.

If you have access to the prepared statement that results in this resultset, you can use
connection.prepareStatement(sql,
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
This prepares your statement in a way that you can rewind the cursor. This is also documented in the ResultSet Javadoc
In general, however, forwarding and rewinding cursors may be quite inefficient for large result sets. Another option in SQL Server would be to calculate the total number of rows directly in your SQL statement:
SELECT my_table.*, count(*) over () total_rows
FROM my_table
WHERE ...

Statement s = cd.createStatement();
ResultSet r = s.executeQuery("SELECT COUNT(*) AS recordCount FROM FieldMaster");
r.next();
int count = r.getInt("recordCount");
r.close();
System.out.println("MyTable has " + count + " row(s).");
Sometimes JDBC does not support following method gives Error like `TYPE_FORWARD_ONLY' use this solution
Sqlite does not support in JDBC.
resultSet.last();
size = resultSet.getRow();
resultSet.beforeFirst();
So at that time use this solution.

your sql Statement creating code may be like
statement = connection.createStatement();
To solve "com.microsoft.sqlserver.jdbc.SQLServerException: The requested operation is not supported on forward only result sets" exception, change above code with
statement = connection.createStatement(
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
After above change you can use
int size = 0;
try {
resultSet.last();
size = resultSet.getRow();
resultSet.beforeFirst();
}
catch(Exception ex) {
return 0;
}
return size;
to get row count

I just made a getter method.
public int getNumberRows(){
try{
statement = connection.creatStatement();
resultset = statement.executeQuery("your query here");
if(resultset.last()){
return resultset.getRow();
} else {
return 0; //just cus I like to always do some kinda else statement.
}
} catch (Exception e){
System.out.println("Error getting row count");
e.printStackTrace();
}
return 0;
}

Do a SELECT COUNT(*) FROM ... query instead.

Most drivers support forward only resultset - so method like last, beforeFirst etc are not supported.
The first approach is suitable if you are also getting the data in the same loop - otherwise the resultSet has already been iterated and can not be used again.
In most cases the requirement is to get the number of rows a query would return without fetching the rows. Iterating through the result set to find the row count is almost same as processing the data. It is better to do another count(*) query instead.

If you have table and are storing the ID as primary and auto increment then this will work
Example code to get the total row count http://www.java2s.com/Tutorial/Java/0340__Database/GettheNumberofRowsinaDatabaseTable.htm
Below is code
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.Statement;
public class Main {
public static void main(String[] args) throws Exception {
Connection conn = getConnection();
Statement st = conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_UPDATABLE);
st.executeUpdate("create table survey (id int,name varchar(30));");
st.executeUpdate("insert into survey (id,name ) values (1,'nameValue')");
st.executeUpdate("insert into survey (id,name ) values (2,null)");
st.executeUpdate("insert into survey (id,name ) values (3,'Tom')");
st = conn.createStatement();
ResultSet rs = st.executeQuery("SELECT * FROM survey");
rs = st.executeQuery("SELECT COUNT(*) FROM survey");
// get the number of rows from the result set
rs.next();
int rowCount = rs.getInt(1);
System.out.println(rowCount);
rs.close();
st.close();
conn.close();
}
private static Connection getConnection() throws Exception {
Class.forName("org.hsqldb.jdbcDriver");
String url = "jdbc:hsqldb:mem:data/tutorial";
return DriverManager.getConnection(url, "sa", "");
}
}

Your function will return the size of a ResultSet, but its cursor will be set after last record, so without rewinding it by calling beforeFirst(), first() or previous() you won't be able to read its rows, and rewinding methods won't work with forward only ResultSet (you'll get the same exception you're getting in your second code fragment).

Others have already answered how to solve your problem, so I won't repeat what has already been said, but I will says this: you should probably figure out a way to solve your problems without knowing the result set count prior to reading through the results.
There are very few circumstances where the row count is actually needed prior to reading the result set, especially in a language like Java. The only case I think of where a row count would be necessary is when the row count is the only data you need(in which case a count query would be superior). Otherwise, you are better off using a wrapper object to represent your table data, and storing these objects in a dynamic container such as an ArrayList. Then, once the result set has been iterated over, you can get the array list count. For every solution that requires knowing the row count before reading the result set, you can probably think of a solution that does so without knowing the row count before reading without much effort. By thinking of solutions that bypass the need to know the row count before processing, you save the ResultSet the trouble of scrolling to the end of the result set, then back to the beginning (which can be a VERY expensive operation for large result sets).
Now of course I'm not saying there are never situations where you may need the row count before reading a result set. I'm just saying that in most circumstances, when people think they need the result set count prior to reading it, they probably don't, and it's worth taking 5 minutes to think about whether there is another way.
Just wanted to offer my 2 cents on the topic.

Following two options worked for me:
1) A function that returns the number of rows in your ResultSet.
private int resultSetCount(ResultSet resultSet) throws SQLException{
try{
int i = 0;
while (resultSet.next()) {
i++;
}
return i;
} catch (Exception e){
System.out.println("Error getting row count");
e.printStackTrace();
}
return 0;
}
2) Create a second SQL statement with the COUNT option.

The ResultSet has it's methods that move the Cursor back and forth depending on the option provided. By default, it's forward moving(TYPE_FORWARD_ONLY ResultSet type).
Unless CONSTANTS indicating Scrollability and Update of ResultSet properly, you might end up getting an error.
E.g. beforeLast()
This method has no effect if the result set contains no rows.
Throws Error if it's not TYPE_FORWARD_ONLY.
The best way to check if empty rows got fetched --- Just to insert new record after checking non-existence
if( rs.next() ) {
Do nothing
} else {
No records fetched!
}
See here

Here's some code that avoids getting the count to instantiate an array, but uses an ArrayList instead and just before returning converts the ArrayList to the needed array type.
Note that Supervisor class here implements ISupervisor interface, but in Java you can't cast from object[] (that ArrayList's plain toArray() method returns) to ISupervisor[] (as I think you are able to do in C#), so you have to iterate through all list items and populate the result array.
/**
* Get Supervisors for given program id
* #param connection
* #param programId
* #return ISupervisor[]
* #throws SQLException
*/
public static ISupervisor[] getSupervisors(Connection connection, String programId)
throws SQLException
{
ArrayList supervisors = new ArrayList();
PreparedStatement statement = connection.prepareStatement(SQL.GET_SUPERVISORS);
try {
statement.setString(SQL.GET_SUPERVISORS_PARAM_PROGRAMID, programId);
ResultSet resultSet = statement.executeQuery();
if (resultSet != null) {
while (resultSet.next()) {
Supervisor s = new Supervisor();
s.setId(resultSet.getInt(SQL.GET_SUPERVISORS_RESULT_ID));
s.setFirstName(resultSet.getString(SQL.GET_SUPERVISORS_RESULT_FIRSTNAME));
s.setLastName(resultSet.getString(SQL.GET_SUPERVISORS_RESULT_LASTNAME));
s.setAssignmentCount(resultSet.getInt(SQL.GET_SUPERVISORS_RESULT_ASSIGNMENT_COUNT));
s.setAssignment2Count(resultSet.getInt(SQL.GET_SUPERVISORS_RESULT_ASSIGNMENT2_COUNT));
supervisors.add(s);
}
resultSet.close();
}
} finally {
statement.close();
}
int count = supervisors.size();
ISupervisor[] result = new ISupervisor[count];
for (int i=0; i<count; i++)
result[i] = (ISupervisor)supervisors.get(i);
return result;
}

From http://docs.oracle.com/javase/1.4.2/docs/api/java/sql/ResultSetMetaData.html
ResultSet rs = stmt.executeQuery("SELECT a, b, c FROM TABLE2");
ResultSetMetaData rsmd = rs.getMetaData();
int numberOfColumns = rsmd.getColumnCount();
A ResultSet contains metadata which gives the number of rows.