Java sqlite performing thousands of queries - java

I have a java program where I perform thousands of queries on a SQLite database inside a loop. If the query shows up empty, then I insert the row. If the query has a result, I ignore. I generally perform these individual queries in batches of 1000, but in the end there will be some hundreds of thousands of total queries to finish this task.
Because I have thousands of lines that I'm checking with individual queries, this portion of the program runs very slowly.
Is there a more efficient way to perform this many queries?
Here is the loop that constantly pulls the original data from excel docs until all the information is read:
for(int i =0;i < batchSize;i++){
try {
String[] rowReader=(dataRows.get(i));
archiveID=rowReader[16];
DIVA = rowReader[41];
//Check if already in DB. If it is not, then adds to a batch
System.out.println("checking db");
if(!isInDB(conn, archiveID, DIVA)){
stmt.setString(1,archiveID);
stmt.setString(2,DIVA);
stmt.setString(3,docName);
stmt.addBatch();
}
}catch (IndexOutOfBoundsException ex){
endOfDoc = true;
}
//dump to database every batchSize
if(++count % batchSize == 0) {
//System.out.println("executing batch");
stmt.executeBatch();
conn.commit();
count=0;
}
}
Here is the actual query method:
//returns false if combo is not in All Records, returns true if there
public static boolean isInDB(Connection conn, String archiveID, String DIVA) throws SQLException {
Connection c = conn;
Statement stmt = null;
try {
Class.forName("org.sqlite.JDBC");
stmt = c.createStatement();
ResultSet rs = stmt.executeQuery( "SELECT * FROM AllRecords WHERE ArchiveID=\"" + archiveID +"\" AND DivaCat=\""+DIVA +"\"" );
if ( rs.next() ) {
return true;
}else{
System.out.println(archiveID+DIVA+" is not in DB");
rs.close();
stmt.close();
return false;
}
} catch ( Exception e ) {
System.err.println( e.getClass().getName() + ": " + e.getMessage() );
System.exit(0);
}
return false;
}
Thanks!

Without any index, finding the desired row(s) requires that the database goes through the entire table, for each query execution.
You can optimize the lookup in this particular query by indexing both lookup columns:
CREATE INDEX whatever ON AllRecords(ArchiveID, DivaCat);

Your isInDB method connect to database each time. You dont need it. You can do that with just one query too.
sqlQuery = "SELECT * FROM AllRecords WHERE "
for(int i =0;i < batchSize;i++){
...
if(i ==0)
sqlQuery + = "ArchiveID=\"" + archiveID +"\" AND DivaCat=\""+DIVA +"\"";
else
sqlQuery + = " OR ArchiveID=\"" + archiveID +"\" AND DivaCat=\""+DIVA +"\"";
after that execute sqlQuery query and check your each row with rs results.

Related

How to hold MySQL query result in an int variable if possible

I want to know whether a table exist or not before creating another one is there any way of holding result in a variable after execution of command, i am using this code but it keeps giving only true even if table doesn't exists.
public static boolean checkBefore(){
boolean r = false;
try{
query = "SELECT COUNT(*)FROM information_schema.tables WHERE table_schema = 'sms' AND table_name = 'auth';";
con = connectsms();
st = con.createStatement();
ResultSet rs = st.executeQuery(query);
r = rs.next();
}catch(SQLException e){
JOptionPane.showMessageDialog(errorMsg,"Exeption Fount: "+e,"Opps! Exception Found in checkBefore()",JOptionPane.ERROR_MESSAGE);
}
System.out.println(r);
return r;
}
Every JDBC guide will show you that after executing a query, you need to call next() to advance to the next/first row, then call getter methods to retrieve the column values of that row.
Queries with aggregating functions (COUNT, MIN, MAX, etc) without a GROUP BY clause will always return exactly one row, so for those kinds of queries, you don't need to check the return value from next(). For pretty much all other queries, you do.
When calling JDBC methods that return resources, you should use try-with-resources to make sure those resource are cleaned up correctly.
Query string does not need to end with a ; semi-colon.
All that means that your code should be:
public static boolean checkBefore() {
String sql = "SELECT COUNT(*)" +
" FROM information_schema.tables" +
" WHERE table_schema = 'sms'" +
" AND table_name = 'auth'";
try ( Connection con = connectsms();
Statement st = con.createStatement();
ResultSet rs = st.executeQuery(sql);
) {
rs.next(); // exactly one row returned, so next() always returns true here
int count = rs.getInt(1); // get value from first column
System.out.println("count = " + count);
return (count != 0);
} catch (SQLException e) {
JOptionPane.showMessageDialog(errorMsg, "Exeption Fount: " + e,
"Opps! Exception Found in checkBefore()",
JOptionPane.ERROR_MESSAGE);
return 0;
}
}
Change r = rs.next(); to rs.next(); and then add r = rs.getInt(1) > 0; and it will work.
This is the query that worked for me:
SELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME = 'databse_name';
and the code that i am using and is working correct:
public static boolean checkBefore(){
boolean result = false;
try{
query = "SELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME = 'sms'";
con = connectsms();
st = con.createStatement();
ResultSet rs = st.executeQuery(query);
result = rs.next();
System.out.println();
}catch(SQLException e){
JOptionPane.showMessageDialog(errorMsg,"Exeption Fount: "+e,"Opps! Exception Found in checkBefor()",JOptionPane.ERROR_MESSAGE);
}
try{
con.close();
}
catch(SQLException e){JOptionPane.showMessageDialog(errorMsg,"Exeption Fount: "+e,"unable to close connection",JOptionPane.ERROR_MESSAGE); }
System.out.println(result);
return result;
}

Can I use setMaxRows() with try-with-resouces?

I am attempting to write a method that selects 2 entries into an employee database and removes them (Based on a salary field), I am currently using a counter to accomplish this, however I tried using setMaxRows() so my result set would only have two entries, thus eliminating the need for the counter. I am using try-with-resources to create my statement and that seems to be causing an issue.
public void downSize(Connection con) {
String sql = "SELECT * FROM " + schemaName + "."+tableName+" WHERE EMPLOYEE_SALARY>200000";
try (
PreparedStatement statement = con.prepareStatement(sql, ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_UPDATABLE);
ResultSet rs = statement.executeQuery();
)
{
int counter = 0;
System.out.println("Now pruning workforce...");
while(rs.next() && counter<2) {
String name = rs.getString("EMPLOYEE_NAME");
rs.deleteRow();
counter++;
System.out.println(name+" was laid off.");
}
} catch(Exception e) {
e.printStackTrace();
System.out.print("Sql exception happened");
}
}

executeUpdate insert query returns 0 but works when inserted manually

I have multiple instances of preparedStatements, all of them works and all are
committed. Except this code block does not work:
public static int InsertBatchProcessingLogs(Connection con, Data dt) {
int batchID = -1;
PreparedStatement preparedStatement = null;
try {
System.out.println("INSERT INTO BATCH PROCESSING LOGS");
preparedStatement = con.prepareStatement("insert into batch_processing_logs (batchID, targetMSISDN,plancode,createddate) select ?,targetMSISDN,plancode,CURRENT_TIMESTAMP from batch_list_tbl where scheduleID = ? and batch_list_status = 1",1);
preparedStatement.setString(1, dt.getValue("batchID"));
preparedStatement.setString(2, dt.getValue("scheduleID"));
batchID = preparedStatement.executeUpdate();
// batchID = preparedStatement.executeUpdate("insert into batch_processing_logs (batchID, targetMSISDN,plancode,createddate) select "+dt.getValue("batchID")+",targetMSISDN,plancode,CURRENT_TIMESTAMP from batch_list_tbl where scheduleID = "+dt.getValue("batchID")+" "
// +" and batch_list_status = 1");
con.commit();
System.out.println("INSERT QUERY: " + batchID);
if (batchID > 0) {
System.out.println("INSERT BATCH LOGS SUCCESSFUL" +dt.getValue("batchID")+dt.getValue("scheduleID"));
} else {
System.out.println("FAILED EXECUTIONS OF INSERT BATCH LOGS");
System.out.println(preparedStatement);
}
} catch (Exception exprep) {
System.out.println(exprep);
// Log("InsertBatchProcessingLogs Exception:" + exprep.getMessage());
exprep.printStackTrace();
batchID = -1;
}
return batchID;
}
It always returns 0. As you can see, there's a print of the preparedStatement so when i try to query that manually, it works perfectly. I dont know why it does not work on executeUpdate() :( It also does not printStackTrace, no errors being shown. I tried a lot even the commented lines and it still does not work. Im not sure if im looking at the wrong place or not. I also tried setAutoCommit(false) then committing it after the executedUpdate. Also tried setAutoCommit(true) and removed the con.commit() code.

Queries returning multiple result sets

I have a MSSQL database and am running the following query:
select * from projects; select * from user
The above query returns two result sets at once, and I cannot fire both queries separately. How can I handle both the result set at once in a Java class?
Correct code to process multiple ResultSets returned by a JDBC statement:
PreparedStatement stmt = ...;
boolean isResultSet = stmt.execute();
int count = 0;
while(true) {
if(isResultSet) {
rs = stmt.getResultSet();
while(rs.next()) {
processEachRow(rs);
}
rs.close();
} else {
if(stmt.getUpdateCount() == -1) {
break;
}
log.info("Result {} is just a count: {}", count, stmt.getUpdateCount());
}
count ++;
isResultSet = stmt.getMoreResults();
}
Important bits:
getMoreResults() and execute() return false to indicate that the result of the statement is just a number and not a ResultSet.
You need to check stmt.getUpdateCount() == -1 to know if there are more results.
Make sure you either close the result sets or use stmt.getMoreResults(Statement.CLOSE_CURRENT_RESULT)
You can use Statement.execute(), getResultSet();
PreparedStatement stmt = ... prepare your statement result
boolean hasResults = stmt.execute();
while (hasResults) {
ResultSet rs = stmt.getResultSet();
... your code parsing the results ...
hasResults = stmt.getMoreResults();
}
Yes, You can. See this MSDN article
https://msdn.microsoft.com/en-us/library/ms378758(v=sql.110).aspx
public static void executeStatement(Connection con) {
try {
String SQL = "SELECT TOP 10 * FROM Person.Contact; " +
"SELECT TOP 20 * FROM Person.Contact";
Statement stmt = con.createStatement();
boolean results = stmt.execute(SQL);
int rsCount = 0;
//Loop through the available result sets.
do {
if(results) {
ResultSet rs = stmt.getResultSet();
rsCount++;
//Show data from the result set.
System.out.println("RESULT SET #" + rsCount);
while (rs.next()) {
System.out.println(rs.getString("LastName") + ", " + rs.getString("FirstName"));
}
rs.close();
}
System.out.println();
results = stmt.getMoreResults();
} while(results);
stmt.close();
}
catch (Exception e) {
e.printStackTrace();
}
}
I've tested that and it works fine.
Before use java, you need look at the RESULT SETS clause.
MSSQL has this feature that can help you with your java code, in a more practical way.
This example will exec two queries:
EXEC('SELECT id_person, name, age FROM dbo.PERSON; SELECT id_url, url FROM dbo.URL;')
WITH RESULT SETS
(
(
id_person BIGINT,
name VARCHAR(255),
age TINYINT
),
(
id_url BIGINT,
url VARCHAR(2000)
)
);
You can use stored procedures with RESULT SETS as well.
More about: https://technet.microsoft.com/en-us/library/ms188332(v=sql.110).aspx
public static void executeProcedure(Connection con) {
try {
CallableStatement stmt = con.prepareCall(...);
..... //Set call parameters, if you have IN,OUT, or IN/OUT parameters
boolean results = stmt.execute();
int rsCount = 0;
//Loop through the available result sets.
while (results) {
ResultSet rs = stmt.getResultSet();
//Retrieve data from the result set.
while (rs.next()) {
....// using rs.getxxx() method to retrieve data
}
rs.close();
//Check for next result set
results = stmt.getMoreResults();
}
stmt.close();
}
catch (Exception e) {
e.printStackTrace();
}
}
The UNION ALL query allows you to combine the result sets of 2 or more "select" queries. It returns all rows (even if the row exists in more than one of the "select" statements).
Each SQL statement within the UNION ALL query must have the same number of fields in the result sets with similar data types.........
select * from projects
UNION ALL
select * from user
The answer: it is NOT possible. The only way: Run them as separate queries.

ResultSet not populating with results

I am attempting to pull results from an Oracle database. I have written a query that is correct, and produces accurate results when issued manually in sqlplus. Furthermore, the code works as expected when when the query matches only one row (In other words, when the ResultSet has only one row, everything works). However, when more than one row match the query, the ResultSet returned by the Oracle JDBC is empty.
public Component[] getAllComponents(int typeId, int osId) throws SQLException
{
String query= "SELECT c.component_id, c.component_name, c.component_version, c.type_id, c.post_download_instructions, "
+ "o.os_id, o.os_name, o.description AS os_description, "
+ "i.file_location, i.release_date, i.patch_number, i.file_id, "
+ "i.description AS i_description "
+ "FROM components c, installation_files i, operating_systems o "
+ "WHERE c.type_id = ? "
+ "AND i.os_id = ? "
+ "AND c.component_id = i.component_id "
+ "AND i.os_id = o.os_id";
ResultSet results = null;
PreparedStatement stmt = null;
ArrayList<Component> found = new ArrayList<Component>();
try {
stmt = dbConn.prepareStatement(query); //dbConn is member variable
stmt.setInt(1, typeId);
stmt.setInt(2, osId);
results = stmt.executeQuery();
while(results.next()){
//Some logic
}
} finally {
if(results != null) results.close();
if(stmt != null) stmt.close();
dbConn.close();
}
//More Code
//etc. etc.
Inspecting the ResultSet shows that calling ResultSet.next() never produces true when the fetched results should contain more than one row. However, issuing the query manually does produce results, and when only one row is returned, everything works fine. Does anyone know what's going on? I'm using Oracle's ojdbc6.jar.
Thanks!
Before that query you can check if there really are some components with COUNT(*) instead of all fields. Then run your query only if COUNT(*) is one or more.

Categories