I have a JDBC ResultSet that gives me a TimeOut after only a few thousand rows are processed. I have a few million rows to process, so I'd like to tweak my program to avoid this, just not sure what needs to be tweaked.
Database table is indexed and returns data quickly using selection criteria, so I don't believe it is on the database side. I'm returned 14 columns mixed between address columns and ints. Not a lot of data.
I'm doing a connection.createStatement() and then building the SQL from there. The answer might be I should use a prepared statement.
Statement stmt = null;
ResultSet rs = null;
try {
stmt = conn.createStatement();
String jobNameFilter = (Cli.getJobName() != null) ? " AND [JobName] = '" + Cli.getJobName() + "'" : "";
String sortOrder = (Cli.isAscending()) ? "ASC" : "DESC";
String orderByClause = Cli.isRandom() ? " ORDER BY [Randomizer] " + sortOrder + ",[RecordID] " + sortOrder : " ORDER BY [RecordID] " + sortOrder;
String startingIdFilter = (Cli.getStartingId() != null) ? " AND [RecordId] > " + Cli.getStartingId() : "";
String driverQuery = "SELECT [RecordID], [Column1] AS [TrackingID], [Address]" + ", [Suite] AS [AptSuiteOther], [City], [Building2Key]"
+ ", [ST] AS [State], [ZIPCode]" + ", [BusinessName], [ContactLastName], [Suite]" + ", [Phone], [EmailAddress]"
+ " FROM [Project].[TestSet] WITH (READUNCOMMITTED)"
+ " INNER JOIN [Project].[State] sttable ON sttable.[ST] = UPPER([Project].[TestSet].[ST]) AND [TerritoryFlag] = 0" + " WHERE [BuildingKey] = 0 " + jobNameFilter
+ startingIdFilter + " AND (([FirstResponse] IS NULL AND ([Building2Key] IS NULL OR [Building2Key] = 0)) OR ([Building2Key] > 0 AND [SecondResponse] IS NULL)) " + orderByClause;
rs = stmt.executeQuery(driverQuery);
} catch (SQLException e1) {
logger.error("SQLException", e1);
}
try {
while (rs.next()) {
int recordId = rs.getInt("RecordID");
// Process data
numberProcessed++;
}
} catch (SQLException sqle) {
logger.error("SQLException", sqle);
}
I'm closing all the ResultSet, Connection and Statement in a finally statement at a different level also.
I'm not sure if I need to set the timeout to something higher, setFetchSize to something greater? Trap timeout and create ResultSet again?
Change logic to only pull one row at a time?
You'd have to profile your app to find out for sure, but I'm guessing that the "// Process data" part is the culprit. You're holding the connection open while process all of the rows.
I'd suggest that you read a batch of rows at a time, close the statement, and then process the batch. Then do a select for the next batch, rinse and repeat.
Selecting one row at a time would introduce a lot of overhead, so I wouldn't suggest doing that.
Also, make sure that you're using a connection pool, so that you don't actually have to build a new Connection each time. The pool will keep it open for you, and recycle it if it goes dead / times out.
Related
I’ve tested the following UPDATE-RETURNING statement in my PostgreSQL client, and it correctly updates and returns the updated rows.
However, in Java, I’m not able to retrieve the ResultSet. statement.execute() and statement.getMoreResults() always return false and statement.getResultSet() returns null, always.
Am I missing something here?
PreparedStatement statement = this.prepareStatement(
"WITH temp AS (" +
" SELECT id" +
" FROM mytable " +
" LIMIT 5 "
") " +
"UPDATE mytable " +
"SET updated = NOW() " +
"FROM temp " +
"WHERE temp.id = mytable.id " +
"RETURNING mytable.data"
);
boolean hasResult = statement.execute();
if (!hasResult) {
hasResult = statement.getMoreResults();
// hasResult is still false
// statement.getResultSet() still returns null
} else {
// statement.getUpdateCount() returns the correct count of updated rows
}
The return code from execute() indicates if there is a result available immediately or not.
getMoreResults() tells you if there is another result set after processing the first one that was indicated through the return value of execute().
So the following will work with your UPDATE statement:
boolean hasResult = statement.execute();
if (hasResult) {
ResultSet rs = statement.getResultSet();
... process the result ...
}
Note that you get a ResultSet even if the UPDATE didn't actually change any rows (e.g. because the WHERE clause didn't match anything). The result set will return false immediately when you call next().
You only need to check getMoreResults() if you expect more than one ResultSet. Then you would need a loop:
boolean hasResult = statement.execute();
while (hasResult) {
ResultSet rs = statement.getResultSet();
.... process the result ...
// check if there is another one
hasResult = statement.getMoreResults();
}
In theory you also need to check getUpdateCount() in the loop as well. that would be needed if the database returns a mixture of result sets and update counts. Not sure if Postgres actually supports that though.
Well, I wasn’t really able to solve this properly. However, I ended up using a workaround by wrapping the whole SQL request with a SELECT statement:
PreparedStatement statement = this.prepareStatement(
"WITH results AS (" +
"WITH temp AS (" +
" SELECT id" +
" FROM mytable " +
" LIMIT 5 "
") " +
"UPDATE mytable " +
"SET updated = NOW() " +
"FROM temp " +
"WHERE temp.id = mytable.id " +
"RETURNING mytable.data" +
") " +
"SELECT * FROM results"
);
ResultSet result = statement.executeQuery();
while (result.next()) {
// Work with result
}
I am working on Java GUI application which connects to SQL database on localhost (I use XAMPP). When I change some entry, for example Age, I click on "Save changes", it is saved and changes are done in SQL database, but when I click on ">" or "<" to view next or previous person and then go back to the person, where I did changes, every entry is without changes in its initial state. But when I close the application and reopen it, all the changes which I made are done. This is part of the code where is mistake, I think. Thank you.
private void jButtonSaveChangesActionPerformed(java.awt.event.ActionEvent evt) {
try {
Statement stmt = con.createStatement();
try {
String query1 = "UPDATE list1 SET " +
"name ='" + jTextFieldName.getText() + "', " +
"surname ='" + jTextFieldSurname.getText() + "', " +
"age ='" + jTextFieldAge.getText() + "' " +
"WHERE ID = " + jLabelActualID.getText();
stmt.executeUpdate(query1);
} catch (Exception e) {
System.err.println(e);
}
} catch (Exception e) {
System.err.println(e);
}
}
Picture of application:
You are not closing, which can be done more safe and automatically with try-with-resources.
This means a commit might not have happened yet. There is an autocommit setting too.
String query1 = "UPDATE list1 SET " +
"name = ?, " +
"surname = ?, " +
"age = ? " +
"WHERE ID = ?";
try (PreparedStatement stmt = con.prepareStatement(query1)) { // Closes stmt.
stmt.setString(1, jTextFieldName.getText());
stmt.setString(2, jTextFieldSurname.getText());
stmt.setInt(3, Integer.parseInt(jTextFieldAge.getText()));
stmt.setString(4, jLabelActualID.getText());
int updateCount = stmt.executeUpdate();
} catch (SQLException | NumberFormatException e) {
System.err.println(e);
}
The same may hold (or may not hold) for the SQL connection.
Also one should use a PreparedStatement for security (SQL injection) and type safeness / escaping of backslash, quote in strings. As you see it is even more readable.
Another case is a second application accessing the database: it can use its own cache, thereby be a bit outdated.
At the moment I am writing a small program which should check whether a file is in a database and if its SHA264 is correct. If it is not correct it should be updated in the database.
The function which I have written for updating are the following:
public static void updateSHA(String TABLE, String shaOLD, String shaNEW)
{
boolean success = false;
Connection con = null;
Statement statement = null;
ResultSet resultSet = null;
PreparedStatement prepStatement = null;
String update = "SET SQL_SAFE_UPDATES = 0" + "; \n" // if I remove this line resp. "'SET SQL_SAFE_UPDATES = 0' + '; \n' + " I get another error code, see below
+ "UPDATE " + dbName + "." + TABLE + " SET sha256 = '" + shaNEW + "' WHERE sha256 = '" + shaOLD + "'" + ";";
try
{
con = DBConnection();
con.prepareStatement(update);
statement = con.createStatement();
resultSet = statement.executeQuery(update);
con.commit();
System.out.println("Successfully updated " + shaOLD + " to " + shaNEW + " in " + dbName + "." + TABLE);
success = true;
}
catch (SQLException sqle)
{
System.out.println(update);
System.out.println("Error at updateSHA for " + shaOLD + ": " + "[" + sqle.getErrorCode() + "] " + sqle.getMessage());
}
finally
{
if (con != null)
{
try
{
con.close();
}
catch (SQLException sqle)
{
System.out.println("Error at updateSHA for " + shaOLD + " while closing con: " + "[" + sqle.getErrorCode() + "] " + sqle.getMessage());
}
}
if (statement != null)
{
try
{
statement.close();
}
catch (SQLException sqle)
{
System.out.println("Error at updateSHA for " + shaOLD + " while closing statement: " + "[" + sqle.getErrorCode() + "] " + sqle.getMessage());
}
}
if (!success)
{
System.exit(-1);
}
}
}
An example of an update query:
SET SQL_SAFE_UPDATES = 0;
UPDATE databaseName.tableName SET sha256 = '4e89f735c019ab1af439ec6aa23b85b66f2be1a1b15401b2471599d145cfda42' WHERE sha256 = '000c675c73567f4256c7de7ad7c43056d5f002e08c1a5c7b4ff77a4dd8e479a3';
The exact error is:
Error at updateSHA for 000c675c73567f4256c7de7ad7c43056d5f002e08c1a5c7b4ff77a4dd8e479a3: [1064] You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'UPDATE databaseName.tableName SET sha256 = '4e89f735c019ab1af439ec6' at line 2
I think it has something to do with the "new" sha which should replace the old one (because in the error it looks like the sha gets cut off or something like that).
If I copy and paste the above update query in MySQL Workbench I do not get any errors and everything works fine (the entry gets updated).
If I delete the line 'SET SQL_SAFE_UPDATES = 0" + "; \n' I get the following error code:
Error at updateSHA for 000c675c73567f4256c7de7ad7c43056d5f002e08c1a5c7b4ff77a4dd8e479a3: [0] Can not issue data manipulation statements with executeQuery().
An assumption is, that the second error occurs because of the MySQL safe update mode.
So my question: Why do I get the 1064 error reps. why does the new sha get cut?
You are trying to execute two statements in one statement execute. This is not allowed by JDBC. Technically the MySQL JDBC driver has a connection property to allow this, but it is disabled by default (as the behavior violates the JDBC spec). You need to split it into two queries. Note however that the default for SQL_SAFE_UPDATES already is 0.
See allowMultiQueries in Driver/Datasource Class Names, URL Syntax and Configuration Properties for Connector/J if you really need to execute two queries as one statement.
The second part of your problem (after removing the SET SQL_SAFE_UPDATES = 0;) is that you are trying to use executeQuery to execute a statement that does not produce a ResultSet. That is not allowed. You should use executeUpdate (or execute) instead.
This question already has answers here:
ResultSet exception - before start of result set
(6 answers)
Closed 5 years ago.
I get an error stating that I got an exception before start of a result set. I'm trying to get a value (score from the MySQL database) and add one to the Java rank based on the player score. This is to create a scoreboard.
So if the player's score is lower than the current score, it gets posted with rank 1. If it's higher, the program checks the score against the next entry in the MySQL database. I haven't yet implemented a feature to change all the current entries rank's to increment by 1.
Bottom Line: I'm creating a scoreboard using MySQL and Java. The Java program creates a score entry based on input, and then sends it off to the MySQL database.
System.out.println("Your score is: "+score*2+" (A lower score is better.)");
try {
// create a java mysql database connection
String myDriver = "com.mysql.jdbc.Driver";
String myUrl = "jdbc:mysql://4.30.110.246:3306/apesbridge2013";
String dbName = "apesbridge2013";
String tbName = period + "period";
Class.forName(myDriver);
Connection conn = DriverManager.getConnection(myUrl, "user", CENSORED);
next = conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,ResultSet.CONCUR_READ_ONLY);
ResultSet resultSet = next.executeQuery("SELECT * FROM " + tbName);
int cscore = resultSet.getInt("score");
for(int sscore = score; sscore > cscore;){
resultSet.next();
cscore = resultSet.getInt("score");
rank++;
}
stmt = conn.createStatement();
stmt.executeUpdate("insert into " + dbName + "." + tbName + " " + "values(" + rank + ", '" + name + "', " + score + ")");
stmt.close();
conn.close();
}
catch (Exception e)
{
System.err.println("Got an exception! ");
System.err.println(e.getMessage());
}
}
Put resultSet.next(); right below your executeQuery line.
As stated by #hd1, you need to call ResultSet.next() after the call to executeQuery:
while (resultSet.next()) {
...
Also, better to use PreparedStatement instead of java.sql.Statement and use parameter placeholders to protect against SQL Injection attacks:
There's a problem in your for loop; the exit condition should be when there are no more rows to fetch. Your query doesn't guarantee that the exit condition will ever be met, and you may attempt to fetch past the end of the resultset. (And even when your for loop does happen to be entered, and when if the for loop does happen to be exited, the rank value derived by that loop is non-deterministic, it's dependent on the order that rows are returned by the database.
I also don't see any call to resultSet.close() or next.close().
There's so many problems here, it's hard to know where to begin.
But firstly, it would be much more efficient to have the database return the rank to you, with a query:
"SELECT COUNT(1) AS rank FROM " + tbName + " WHERE score < " + score
rather than pulling back all the rows back, and comparing each score. That's just painful, and a whole lot of code that is just noise. That would allow you to focus on the code that DOES need to be there.
Once you get that working, you need to ensure that your statement is not vulnerable to SQL injection, and prepared statements with bind variables is really the way to go there.
And you really do need to ensure that calls are made to the close() methods on the resultset, prepared statements, and the connection. We typically want these in a finally block. Either use nested try/catch blocks, where the variables are immediately initialized, like this:
try {
Connection conn = DriverManager.getConnection(...
try {
stmt = conn.CreateStatement();
String query = "SELECT COUNT(1) AS `rank` FROM " + tbName + " WHERE `score` < " + score ;
try {
ResultSet rs = stmt.executeQuery(query);
while (rs.next()) {
rank = rs.getInt("rank");
}
} finally {
if (rs!=null) { rs.close() };
}
} finally {
if (stmt!=null) { stmt.close() };
}
} finally {
if (conn!=null) { conn.close() };
}
Or one big try/catch block can also be workable:
} finally {
if (resultSet!=null) { resultSet.close() };
if (next!=null) { next.close() };
if (conn!=null) { conn.close() };
)
The point is, the close methods really do need to be called.
I have a table A. I insert data into table A through a user interface. Table A has an ID(primary key), which is generated using a sequence, and 16 other columns. One of the column is called cntrct_no.
When I try to insert data into the table through UI, it works fine the first time. I check the table A and all the data are there.
But when I try to insert the same data again without changing anything, it looks like the data is getting added to the table and I do not get any errors. But when I check table A, the data inserted the second time is not there.
If I try to insert the same data directly thorough SQL developer, the data gets inserted into the table.
The weird thing is if I just change the value of the cntrct_no in the UI and leave rest of the data same, the data gets inserted.
Can anyone please explain to me what could possibly cause this?
Not sure if this helps: stmt.executeUpdate(); returns 0 when the data is not inserted and a 1 when it's inserted.
public void writeToAudit(String contractNo, String tripNo,
String tripEffDate,
String tripDiscDate, String transpModeId, String userId,
String transType, AplLeg[] legs) {
final Session session = HibernateUtil.getSession();
Connection con = null;
con = session.connection();
PreparedStatement stmt = null;
PreparedStatement stmtSelId = null;
ResultSet rs = null;
long nextId = -1;
int i=0;
try {
for(i=0;i<legs.length;i++) {
String sqlNextId = "SELECT rpt_audit_transportation_seq.NEXTVAL as seqval FROM DUAL";
stmtSelId = con.prepareStatement(sqlNextId, ResultSet.TYPE_SCROLL_SENSITIVE, ResultSet.CONCUR_UPDATABLE);
rs = stmtSelId.executeQuery();
rs.last();
final int rows = rs.getRow();
if (rows == 0){
nextId = -1;
}
rs.beforeFirst();
rs.next();
nextId = rs.getInt(1);
if(nextId==-1)
throw new SQLException("Cannot get next val from rpt_audit_transportation sequence.");
stmt = con.prepareStatement(WRITE_TO_AUDIT_DML);
stmt.setLong(1, nextId);
stmt.setString(2, userId.toUpperCase());
stmt.setString(3, transType);
stmt.setString(4, contractNo);
stmt.setString(5, tripNo);
stmt.setInt(6, Integer.parseInt(transpModeId));
stmt.setString(7, tripEffDate);
stmt.setString(8, tripDiscDate);
stmt.setLong(9, legs[i].getLegId().longValue());
int temp = stmt.executeUpdate();
con.commit();
}
stmt.close();
}
catch (Exception e) {
}
finally {
closeConnection(session, con, stmtSelId, rs);
}
}
THE SQL STATEMENT:
private static final String WRITE_TO_AUDIT_DML =
"INSERT INTO rpt_audit_transportation " +
"(audit_id, audit_date, audit_process, audit_userid, " +
"audit_trans_type, audit_route_no, audit_trip_no, " +
"audit_eff_dt, audit_disc_dt, audit_orig_facility_id, " +
"audit_dest_facility_id, audit_arvl_tm, audit_dprt_tm, " +
"audit_leg_seq_no, audit_freq_id, audit_trnsp_mode_id) " +
"(SELECT ?, " + // audit id
"SYSDATE, " +
"'TOPS_UI', " +
"?, " + // userId
"?, " +
"rte.cntrct_no, " +
"trp.trip_no, " +
"rte.cntrct_eff_dt, " +
"rte.cntrct_disc_dt, " +
"NVL(leg.orig_facility_id, trp.orig_fac_id), " +
"NVL(leg.dest_facility_id, trp.dest_fac_id), " +
"NVL(leg.arvl_tm, trp.arvl_tm), " +
"NVL(leg.dprt_tm, trp.dprt_tm), " +
"leg.leg_seq, " +
"trp.freq_id, " +
"rte.trnsp_mode_id " +
"FROM apl_contract rte, " +
"apl_trip trp, " +
"apl_leg leg " +
"WHERE rte.cntrct_no = ? " + // contract id
"AND trp.trip_no = ? " + // trip no
"AND rte.trnsp_mode_id = ? " + // transp mode id
"AND rte.cntrct_locked_ind = 'N' " +
"AND trp.trip_eff_dt = to_date(?,'MM/DD/YYYY') " + // trip eff date
"AND trp.trip_disc_dt = to_date(?,'MM/DD/YYYY') " + // trip disc date
"AND trp.cntrct_id = rte.cntrct_id " +
"AND leg.trip_id = trp.trip_id " +
"AND leg.leg_id = ?) ";
Looks like you're not inserting plain values, but a result of a select based on the parameters.
What you are using is an INSERT ... SELECT () clause, so if the SELECT part does not return any rows, the INSERT won't insert anything, and stmt.executeUpdate() will return 0. Find out why SELECT returns no rows.
This may be due some triggers saving stuff in other tables when you do the insert into rpt_audit_transportation, but it's just a guess.
The problem is that you have a catch that is swallowing your exceptions
catch (Exception e) {
}
That means that when the SQL statement throws an error, you're telling your code to catch the exception and ignore it. It is almost always an error to do that since, as you're discovering, it means that your code can fail to do what you expect with no way of letting you know that something failed. At a minimum, you would need to log the exception somewhere. In general, though, if you cannot fix whatever condition lead to the exception, you ought to re-raise the exception or simply not catch it in the first place.
My guess is that the second insert is violating some constraint defined on the table. But since your code is catching the exception and throwing it away, you're not being notified that there was a constraint violation nor is your code noting which constraint was violated.
When the cntrct_no is same you are getting an exception and you are supperessing that as told by #Justin Cave. This may be because you are having a unique constraint for that field and the DB throws an error and you are suppressing.
When cntrct_no is changed - obviously the constraint wont fail and for primary key since you are using the sequence it would have generated the next number and it happily gets inserted in the DB.
Don't ever suppress the exception. Do some thing in that block either rethrow as application specific exception or convert to error code and propagate that to the front end.