I want to do the following:
Connect to a Postgres database and select the contents of a particular table 2 rows at a time
Take the 2 rows, convert them to CSV strings and send that string over the wire to a RESTful API endpoint
The problem I am having is that in order to process the ResultSet and convert it into a CSV String, I need to call rs.next() but in doing that, I am pulling the next 'batch' from the database. Here's the code I have so far:
public void load() throws Exception {
Connection conn = connectToDB();
conn.setAutoCommit(false);
Statement st = conn.createStatement();
st.setFetchSize(2);
ResultSet rs = st.executeQuery("SELECT * FROM friends");
String csv = resultSetToCSV(rs);
System.out.println("CSV output is: " + csv);
rs.close();
st.close();
}
private static String resultSetToCSV(ResultSet rs) throws Exception {
int colCount = rs.getMetaData().getColumnCount();
StringBuilder result = new StringBuilder();
while (rs.next()) {
for (int i = 0; i < colCount; i++) {
result.append("\"").append(rs.getString(i + 1)).append('"');
}
result.append(System.lineSeparator());
}
return result.toString();
}
How can I get this to work so that each "batch" of 2 rows is processed and converted to a CSV string? I want to do this because I am going to be running this on a very large table which won't fit into memory (I am just testing it out with a batch size of 2).
You can set the fetch size with ResultSet.setFetchSize(int) but generally you should not concern yourself with the entire table fitting in memory and let the driver do the job.
Related
I need to check in database exist ~2k records, I test id on db size 1.000.000 and it takes 46s. Its too long, because in future this db can have more than 500.000.000 records. Is any way to speed up searching from db ? I use JDBC in java, here's code :
public int search(List<String> toSearch) throws SQLException {
String query = "SELECT * FROM strings where string = ?";
StringBuilder sB = new StringBuilder(query);
for (int i=0; i<toSearch.size()-1; i++) {
sB.append("OR string=?");
}
System.out.println(toSearch.size());
PreparedStatement prep = con.prepareStatement(sB.toString());
int i=1;
for (String string : toSearch) {
prep.setString(i, string);
i++;
}
long data = System.currentTimeMillis();
ResultSet resultSet = prep.executeQuery();
long data2 = System.currentTimeMillis();
System.out.println((data2 - data) / 1000);
List<String> toReturn = new ArrayList<>();
while (resultSet.next()) {
toReturn.add(resultSet.getString("string"));
}
return toReturn.size();
}
Table name is strings, column string.
First of all, you need to have an index on string column.
CREATE INDEX indexName ON strings(string);
Do not construct queries with variable number of arguments for H2. Use
PreparedStatement ps = con.prepareStatement("SELECT * FROM strings WHERE string = ANY(?)");
ps.setObject(1, toSearch.toArray(new String[0]));
If you use some old unsupported version of H2, use its TABLE() function instead as described in documentation distributed with your version (Prepared Statements and IN(...) section in PDF).
With other DBMS you may need other tricks to avoid dynamic generation of SQL code, = ANY(?) will work in PostgreSQL and its forks and in H2, but not in other DBMS.
I'm trying to create a simple method that receives a ResultSet as a parameter and returns an int that contains the row count of the ResultSet. Is this a valid way of doing this or not so much?
int size = 0;
try {
while(rs.next()){
size++;
}
}
catch(Exception ex) {
System.out.println("------------------Tablerize.getRowCount-----------------");
System.out.println("Cannot get resultSet row count: " + ex);
System.out.println("--------------------------------------------------------");
}
I tried this:
int size = 0;
try {
resultSet.last();
size = resultSet.getRow();
resultSet.beforeFirst();
}
catch(Exception ex) {
return 0;
}
return size;
But I got an error saying
com.microsoft.sqlserver.jdbc.SQLServerException:
The requested operation is not supported on forward only result sets.
If you have access to the prepared statement that results in this resultset, you can use
connection.prepareStatement(sql,
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
This prepares your statement in a way that you can rewind the cursor. This is also documented in the ResultSet Javadoc
In general, however, forwarding and rewinding cursors may be quite inefficient for large result sets. Another option in SQL Server would be to calculate the total number of rows directly in your SQL statement:
SELECT my_table.*, count(*) over () total_rows
FROM my_table
WHERE ...
Statement s = cd.createStatement();
ResultSet r = s.executeQuery("SELECT COUNT(*) AS recordCount FROM FieldMaster");
r.next();
int count = r.getInt("recordCount");
r.close();
System.out.println("MyTable has " + count + " row(s).");
Sometimes JDBC does not support following method gives Error like `TYPE_FORWARD_ONLY' use this solution
Sqlite does not support in JDBC.
resultSet.last();
size = resultSet.getRow();
resultSet.beforeFirst();
So at that time use this solution.
your sql Statement creating code may be like
statement = connection.createStatement();
To solve "com.microsoft.sqlserver.jdbc.SQLServerException: The requested operation is not supported on forward only result sets" exception, change above code with
statement = connection.createStatement(
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
After above change you can use
int size = 0;
try {
resultSet.last();
size = resultSet.getRow();
resultSet.beforeFirst();
}
catch(Exception ex) {
return 0;
}
return size;
to get row count
I just made a getter method.
public int getNumberRows(){
try{
statement = connection.creatStatement();
resultset = statement.executeQuery("your query here");
if(resultset.last()){
return resultset.getRow();
} else {
return 0; //just cus I like to always do some kinda else statement.
}
} catch (Exception e){
System.out.println("Error getting row count");
e.printStackTrace();
}
return 0;
}
Do a SELECT COUNT(*) FROM ... query instead.
Most drivers support forward only resultset - so method like last, beforeFirst etc are not supported.
The first approach is suitable if you are also getting the data in the same loop - otherwise the resultSet has already been iterated and can not be used again.
In most cases the requirement is to get the number of rows a query would return without fetching the rows. Iterating through the result set to find the row count is almost same as processing the data. It is better to do another count(*) query instead.
If you have table and are storing the ID as primary and auto increment then this will work
Example code to get the total row count http://www.java2s.com/Tutorial/Java/0340__Database/GettheNumberofRowsinaDatabaseTable.htm
Below is code
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.Statement;
public class Main {
public static void main(String[] args) throws Exception {
Connection conn = getConnection();
Statement st = conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_UPDATABLE);
st.executeUpdate("create table survey (id int,name varchar(30));");
st.executeUpdate("insert into survey (id,name ) values (1,'nameValue')");
st.executeUpdate("insert into survey (id,name ) values (2,null)");
st.executeUpdate("insert into survey (id,name ) values (3,'Tom')");
st = conn.createStatement();
ResultSet rs = st.executeQuery("SELECT * FROM survey");
rs = st.executeQuery("SELECT COUNT(*) FROM survey");
// get the number of rows from the result set
rs.next();
int rowCount = rs.getInt(1);
System.out.println(rowCount);
rs.close();
st.close();
conn.close();
}
private static Connection getConnection() throws Exception {
Class.forName("org.hsqldb.jdbcDriver");
String url = "jdbc:hsqldb:mem:data/tutorial";
return DriverManager.getConnection(url, "sa", "");
}
}
Your function will return the size of a ResultSet, but its cursor will be set after last record, so without rewinding it by calling beforeFirst(), first() or previous() you won't be able to read its rows, and rewinding methods won't work with forward only ResultSet (you'll get the same exception you're getting in your second code fragment).
Others have already answered how to solve your problem, so I won't repeat what has already been said, but I will says this: you should probably figure out a way to solve your problems without knowing the result set count prior to reading through the results.
There are very few circumstances where the row count is actually needed prior to reading the result set, especially in a language like Java. The only case I think of where a row count would be necessary is when the row count is the only data you need(in which case a count query would be superior). Otherwise, you are better off using a wrapper object to represent your table data, and storing these objects in a dynamic container such as an ArrayList. Then, once the result set has been iterated over, you can get the array list count. For every solution that requires knowing the row count before reading the result set, you can probably think of a solution that does so without knowing the row count before reading without much effort. By thinking of solutions that bypass the need to know the row count before processing, you save the ResultSet the trouble of scrolling to the end of the result set, then back to the beginning (which can be a VERY expensive operation for large result sets).
Now of course I'm not saying there are never situations where you may need the row count before reading a result set. I'm just saying that in most circumstances, when people think they need the result set count prior to reading it, they probably don't, and it's worth taking 5 minutes to think about whether there is another way.
Just wanted to offer my 2 cents on the topic.
Following two options worked for me:
1) A function that returns the number of rows in your ResultSet.
private int resultSetCount(ResultSet resultSet) throws SQLException{
try{
int i = 0;
while (resultSet.next()) {
i++;
}
return i;
} catch (Exception e){
System.out.println("Error getting row count");
e.printStackTrace();
}
return 0;
}
2) Create a second SQL statement with the COUNT option.
The ResultSet has it's methods that move the Cursor back and forth depending on the option provided. By default, it's forward moving(TYPE_FORWARD_ONLY ResultSet type).
Unless CONSTANTS indicating Scrollability and Update of ResultSet properly, you might end up getting an error.
E.g. beforeLast()
This method has no effect if the result set contains no rows.
Throws Error if it's not TYPE_FORWARD_ONLY.
The best way to check if empty rows got fetched --- Just to insert new record after checking non-existence
if( rs.next() ) {
Do nothing
} else {
No records fetched!
}
See here
Here's some code that avoids getting the count to instantiate an array, but uses an ArrayList instead and just before returning converts the ArrayList to the needed array type.
Note that Supervisor class here implements ISupervisor interface, but in Java you can't cast from object[] (that ArrayList's plain toArray() method returns) to ISupervisor[] (as I think you are able to do in C#), so you have to iterate through all list items and populate the result array.
/**
* Get Supervisors for given program id
* #param connection
* #param programId
* #return ISupervisor[]
* #throws SQLException
*/
public static ISupervisor[] getSupervisors(Connection connection, String programId)
throws SQLException
{
ArrayList supervisors = new ArrayList();
PreparedStatement statement = connection.prepareStatement(SQL.GET_SUPERVISORS);
try {
statement.setString(SQL.GET_SUPERVISORS_PARAM_PROGRAMID, programId);
ResultSet resultSet = statement.executeQuery();
if (resultSet != null) {
while (resultSet.next()) {
Supervisor s = new Supervisor();
s.setId(resultSet.getInt(SQL.GET_SUPERVISORS_RESULT_ID));
s.setFirstName(resultSet.getString(SQL.GET_SUPERVISORS_RESULT_FIRSTNAME));
s.setLastName(resultSet.getString(SQL.GET_SUPERVISORS_RESULT_LASTNAME));
s.setAssignmentCount(resultSet.getInt(SQL.GET_SUPERVISORS_RESULT_ASSIGNMENT_COUNT));
s.setAssignment2Count(resultSet.getInt(SQL.GET_SUPERVISORS_RESULT_ASSIGNMENT2_COUNT));
supervisors.add(s);
}
resultSet.close();
}
} finally {
statement.close();
}
int count = supervisors.size();
ISupervisor[] result = new ISupervisor[count];
for (int i=0; i<count; i++)
result[i] = (ISupervisor)supervisors.get(i);
return result;
}
From http://docs.oracle.com/javase/1.4.2/docs/api/java/sql/ResultSetMetaData.html
ResultSet rs = stmt.executeQuery("SELECT a, b, c FROM TABLE2");
ResultSetMetaData rsmd = rs.getMetaData();
int numberOfColumns = rsmd.getColumnCount();
A ResultSet contains metadata which gives the number of rows.
I've looked through this forum and searched the web for a solution to my above problem but could not find something that points me in the right direction. Please forgive me if this is a duplication.
I'm working on a java project where my application interacts with an MS Access 2016 database. One of the functions of my program is to query the db for a specific record and display the data of that record in a gui. Here is my code to retrieve the data:
int i = 0;
String q = "select * from QueryData where id=123456";
try {
pstmnt=conn.prepareStatement(q);
Object obj ;
rs = pstmnt.executeQuery();
while (rs.next()) {
obj=rs.getObject(i+1);
data.add(obj); //where data is a List object i++;
}
} catch ....
Problem is I only get the first value in this record (1st column of record) and there are more data available in the record/row.
Could it be the rs.next() method that is doing this and if so, what should I use to get the next value in this specific record?
ResultSet#next() iterates over the rows in the result set (which, in this case, is just a single row). If you don't know the result set's structure upfront, you can dynamically deduce it from a ResultSetMetaData object:
int i=0;
String q="select * from QueryData where id=123456";
try (PreparedStatement pstmnt = conn.prepareStatement(q);
ResultSet rs = pstmnt.executeQuery()) {
ResultSetMetaData rsmd = rs.getMetaData();
// Assume it's just one row.
// If there's more than one, you need a while loop
if (rs.next()) {
for (int i = 0; i < rsmd.getColumnCount(); ++i) {
data.add(rs.getObject(i + 1));
}
}
}
I'm writing data from Java to an Access database on Windows 32 bit. When I write a record, I need to retrieve the row ID / primary key so that I can a) update the record easily if I want to and b) cross reference other data to that record.
When did something similar in C, I could make a updatable cursor which allowed me to write a new record and simultaneously retrieve the row ID. With Java, it looks as though I should be able to do this, but it throws an exception with the following code.
con = openAccessDatabase();
String selectString = "SELECT ID, RunCount FROM SpeedTable";
try {
PreparedStatement selectStatement = con.prepareStatement(selectString,
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_UPDATABLE);
ResultSet idResult = selectStatement.executeQuery();
int id;
for (int i = 0; i < nWrites; i++) {
idResult.moveToInsertRow();
idResult.updateObject(1, null); // this line makes no difference whatsoever !
idResult.updateInt(2, i);
idResult.insertRow(); // throws java.sql.SQLException: [Microsoft][ODBC Microsoft Access Driver]Error in row
id = idResult.getInt(1);
}
selectStatement.close();
} catch (SQLException e) {
e.printStackTrace();
}
The only thing I've been able to do is to write a new record and then run a different query to get the Row id back ...
String insertString = "INSERT INTO SpeedTable (RunCount) VALUES (?)";
String idString = "SELECT ID FROM SpeedTable ORDER BY ID DESC";
//
try {
ResultSet idResult = null;
PreparedStatement preparedStatement, idStatement;
preparedStatement = con.prepareStatement(insertString,
ResultSet.TYPE_FORWARD_ONLY,
ResultSet.CONCUR_READ_ONLY);
idStatement = con.prepareStatement(idString,
ResultSet.TYPE_FORWARD_ONLY,
ResultSet.CONCUR_READ_ONLY);
for (int i = 0; i < nWrites; i++) {
// write the data into the database
preparedStatement.setInt(1, i);
preparedStatement.execute();
// re-run the query to get the index back from the database.
idResult = idStatement.executeQuery();
idResult.next();
int lastIndex = idResult.getInt(1);
idResult.close();
}
This works but becomes impossibly slow when the table has more than a few 10's of 1000's of records in it. There is also a risk of returning the wrong ID if two parts of the program start writing at the same time (unlikely but not impossible).
I know that at least one suggestion will be to either not use Java or not use Access, but they are not options. It's also part of a free open source software package, so I'm reluctant to pay for anything. Writing my own C JNI interface which provides the basic functionality that I need for my application is even less appealing.
Not sure if this works for MS Access but you can try:
st.executeUpdate("INSERT INTO SpeedTable (RunCount) VALUES (1000)", Statement.RETURN_GENERATED_KEYS);
ResultSet rs = st.getGeneratedKeys();
rs.next();
long id = rs.getLong(1);
I'm trying to create a simple method that receives a ResultSet as a parameter and returns an int that contains the row count of the ResultSet. Is this a valid way of doing this or not so much?
int size = 0;
try {
while(rs.next()){
size++;
}
}
catch(Exception ex) {
System.out.println("------------------Tablerize.getRowCount-----------------");
System.out.println("Cannot get resultSet row count: " + ex);
System.out.println("--------------------------------------------------------");
}
I tried this:
int size = 0;
try {
resultSet.last();
size = resultSet.getRow();
resultSet.beforeFirst();
}
catch(Exception ex) {
return 0;
}
return size;
But I got an error saying
com.microsoft.sqlserver.jdbc.SQLServerException:
The requested operation is not supported on forward only result sets.
If you have access to the prepared statement that results in this resultset, you can use
connection.prepareStatement(sql,
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
This prepares your statement in a way that you can rewind the cursor. This is also documented in the ResultSet Javadoc
In general, however, forwarding and rewinding cursors may be quite inefficient for large result sets. Another option in SQL Server would be to calculate the total number of rows directly in your SQL statement:
SELECT my_table.*, count(*) over () total_rows
FROM my_table
WHERE ...
Statement s = cd.createStatement();
ResultSet r = s.executeQuery("SELECT COUNT(*) AS recordCount FROM FieldMaster");
r.next();
int count = r.getInt("recordCount");
r.close();
System.out.println("MyTable has " + count + " row(s).");
Sometimes JDBC does not support following method gives Error like `TYPE_FORWARD_ONLY' use this solution
Sqlite does not support in JDBC.
resultSet.last();
size = resultSet.getRow();
resultSet.beforeFirst();
So at that time use this solution.
your sql Statement creating code may be like
statement = connection.createStatement();
To solve "com.microsoft.sqlserver.jdbc.SQLServerException: The requested operation is not supported on forward only result sets" exception, change above code with
statement = connection.createStatement(
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
After above change you can use
int size = 0;
try {
resultSet.last();
size = resultSet.getRow();
resultSet.beforeFirst();
}
catch(Exception ex) {
return 0;
}
return size;
to get row count
I just made a getter method.
public int getNumberRows(){
try{
statement = connection.creatStatement();
resultset = statement.executeQuery("your query here");
if(resultset.last()){
return resultset.getRow();
} else {
return 0; //just cus I like to always do some kinda else statement.
}
} catch (Exception e){
System.out.println("Error getting row count");
e.printStackTrace();
}
return 0;
}
Do a SELECT COUNT(*) FROM ... query instead.
Most drivers support forward only resultset - so method like last, beforeFirst etc are not supported.
The first approach is suitable if you are also getting the data in the same loop - otherwise the resultSet has already been iterated and can not be used again.
In most cases the requirement is to get the number of rows a query would return without fetching the rows. Iterating through the result set to find the row count is almost same as processing the data. It is better to do another count(*) query instead.
If you have table and are storing the ID as primary and auto increment then this will work
Example code to get the total row count http://www.java2s.com/Tutorial/Java/0340__Database/GettheNumberofRowsinaDatabaseTable.htm
Below is code
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.Statement;
public class Main {
public static void main(String[] args) throws Exception {
Connection conn = getConnection();
Statement st = conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_UPDATABLE);
st.executeUpdate("create table survey (id int,name varchar(30));");
st.executeUpdate("insert into survey (id,name ) values (1,'nameValue')");
st.executeUpdate("insert into survey (id,name ) values (2,null)");
st.executeUpdate("insert into survey (id,name ) values (3,'Tom')");
st = conn.createStatement();
ResultSet rs = st.executeQuery("SELECT * FROM survey");
rs = st.executeQuery("SELECT COUNT(*) FROM survey");
// get the number of rows from the result set
rs.next();
int rowCount = rs.getInt(1);
System.out.println(rowCount);
rs.close();
st.close();
conn.close();
}
private static Connection getConnection() throws Exception {
Class.forName("org.hsqldb.jdbcDriver");
String url = "jdbc:hsqldb:mem:data/tutorial";
return DriverManager.getConnection(url, "sa", "");
}
}
Your function will return the size of a ResultSet, but its cursor will be set after last record, so without rewinding it by calling beforeFirst(), first() or previous() you won't be able to read its rows, and rewinding methods won't work with forward only ResultSet (you'll get the same exception you're getting in your second code fragment).
Others have already answered how to solve your problem, so I won't repeat what has already been said, but I will says this: you should probably figure out a way to solve your problems without knowing the result set count prior to reading through the results.
There are very few circumstances where the row count is actually needed prior to reading the result set, especially in a language like Java. The only case I think of where a row count would be necessary is when the row count is the only data you need(in which case a count query would be superior). Otherwise, you are better off using a wrapper object to represent your table data, and storing these objects in a dynamic container such as an ArrayList. Then, once the result set has been iterated over, you can get the array list count. For every solution that requires knowing the row count before reading the result set, you can probably think of a solution that does so without knowing the row count before reading without much effort. By thinking of solutions that bypass the need to know the row count before processing, you save the ResultSet the trouble of scrolling to the end of the result set, then back to the beginning (which can be a VERY expensive operation for large result sets).
Now of course I'm not saying there are never situations where you may need the row count before reading a result set. I'm just saying that in most circumstances, when people think they need the result set count prior to reading it, they probably don't, and it's worth taking 5 minutes to think about whether there is another way.
Just wanted to offer my 2 cents on the topic.
Following two options worked for me:
1) A function that returns the number of rows in your ResultSet.
private int resultSetCount(ResultSet resultSet) throws SQLException{
try{
int i = 0;
while (resultSet.next()) {
i++;
}
return i;
} catch (Exception e){
System.out.println("Error getting row count");
e.printStackTrace();
}
return 0;
}
2) Create a second SQL statement with the COUNT option.
The ResultSet has it's methods that move the Cursor back and forth depending on the option provided. By default, it's forward moving(TYPE_FORWARD_ONLY ResultSet type).
Unless CONSTANTS indicating Scrollability and Update of ResultSet properly, you might end up getting an error.
E.g. beforeLast()
This method has no effect if the result set contains no rows.
Throws Error if it's not TYPE_FORWARD_ONLY.
The best way to check if empty rows got fetched --- Just to insert new record after checking non-existence
if( rs.next() ) {
Do nothing
} else {
No records fetched!
}
See here
Here's some code that avoids getting the count to instantiate an array, but uses an ArrayList instead and just before returning converts the ArrayList to the needed array type.
Note that Supervisor class here implements ISupervisor interface, but in Java you can't cast from object[] (that ArrayList's plain toArray() method returns) to ISupervisor[] (as I think you are able to do in C#), so you have to iterate through all list items and populate the result array.
/**
* Get Supervisors for given program id
* #param connection
* #param programId
* #return ISupervisor[]
* #throws SQLException
*/
public static ISupervisor[] getSupervisors(Connection connection, String programId)
throws SQLException
{
ArrayList supervisors = new ArrayList();
PreparedStatement statement = connection.prepareStatement(SQL.GET_SUPERVISORS);
try {
statement.setString(SQL.GET_SUPERVISORS_PARAM_PROGRAMID, programId);
ResultSet resultSet = statement.executeQuery();
if (resultSet != null) {
while (resultSet.next()) {
Supervisor s = new Supervisor();
s.setId(resultSet.getInt(SQL.GET_SUPERVISORS_RESULT_ID));
s.setFirstName(resultSet.getString(SQL.GET_SUPERVISORS_RESULT_FIRSTNAME));
s.setLastName(resultSet.getString(SQL.GET_SUPERVISORS_RESULT_LASTNAME));
s.setAssignmentCount(resultSet.getInt(SQL.GET_SUPERVISORS_RESULT_ASSIGNMENT_COUNT));
s.setAssignment2Count(resultSet.getInt(SQL.GET_SUPERVISORS_RESULT_ASSIGNMENT2_COUNT));
supervisors.add(s);
}
resultSet.close();
}
} finally {
statement.close();
}
int count = supervisors.size();
ISupervisor[] result = new ISupervisor[count];
for (int i=0; i<count; i++)
result[i] = (ISupervisor)supervisors.get(i);
return result;
}
From http://docs.oracle.com/javase/1.4.2/docs/api/java/sql/ResultSetMetaData.html
ResultSet rs = stmt.executeQuery("SELECT a, b, c FROM TABLE2");
ResultSetMetaData rsmd = rs.getMetaData();
int numberOfColumns = rsmd.getColumnCount();
A ResultSet contains metadata which gives the number of rows.