I am executing multiple SQL statements using single Connection and Statement object as two processes.
Pre-process (Preprocess.java)
Post-process (Postprocess.java)
My Process steps are as follows,
Creating a Connection object and setAutoCommit() as false.
Creating a Statement object.
During pre-process executing the required SQL statements using the created Statement object.
Then for post-process i pass the same Statement object and executing the required SQL statements.
Finally commit the transaction (connection.commit();) and close the Connection,Statement objects.
My Problem is:
Sometimes the post-process executed statements are not reflected in database. That is, if i insert or update any records in post-process,those records are not available in database.
But the post-process execution is working fine without any exception.
The Probability of this case is 5 out of 1/(Which means 4 times working as i expected but 1 time its problematic).
Can any one point out the problem?
I am using Jdk 1.7 and Postgresql 9.3 version.
Related
I came across an old piece of code which looks like below
Statement stmt = connection.createStatement();
stmt.addBatch(insertQuery);
stmt.addBatch(insertQuery);
stmt.addBatch(insertQuery);
stmt.addBatch(insertQuery);
//there is some data which needs to be deleted before inserting the new data.
stmt.execute(deleteQuery);
stmt.executeBatch();
Here we are batching up a few query and before executing the batch this code is executing some other delete query and then executing the batch.
Is it legal to do this?
Will the above code work as expected that it will first execute the delete query and then the batch update?
The JDBC specification (version 4.3) says:
The behavior of the methods executeQuery, executeUpdate, and
execute is implementation-defined when a statement’s batch is
non-empty.
In other words, the behaviour is not specified and depends on the driver implementation, this means it should not be relied on.
A quick (but not thorough) scan of the pgjdbc sources seems to indicate that the PostgreSQL driver indeed allows you to first add statements to the batch, execute a single statement, and then execute the batch.
But in the code shown, I'd suggest to simply first execute the delete query, and only then populate and execute the batch. That order would be a lot simpler to read for people unfamiliar with the code.
I have a stored procedure, with an argument(Parameter)
Let us call it x.
When I run,
select * from x('myParameter')
It completes in 4 minutes.
Now I try to execute the same stored prcedure in java as follows:-
PreparedStatement statement = connection.prepareStatement("Select * from x('myParameter')");
org.postgresql.PGStatement pgstmt = (org.postgresql.PGStatement)statement;
pgstmt.setPrepareThreshold(1);
boolean usingServerPrepare = pgstmt.isUseServerPrepare();
System.out.println(usingServerPrepare);
statement.execute();
This takes approximately 26 hours to complete.
I was initially trying without pgstmt.setPrepareThreshold(1);,
i.e. Default value for prepareThreshold.
The result was the same.
Then I tried the java program with "set enable_nestloop false" as the first line of the stored procedure and it completes in 4 minutes.
I am not able to understand why the query changes the execution plan, when I am explicitly forcing server side prepared statement.
I am using Postgres 9.4 and Java 8. I am also using PGBouncer for connection pooling.
My query here is - Why is JDBC influencing the execution plan if at all? Is there a way to force server side execution plan to my query triggered from java program?
Got the solution:-
There is a table used within the stored procedure say "tab1".
When we run stored procedure from PGAdmin/SQL window tab1 is already created/modified hours back.
In the automated running, using java, tab1 is created and loaded/modified just seconds before the execution of the stored procedure.
In the first case(Running stored procedure from PGAdmin/SQL window) auto-analyze on tab1 has finished by the time we run the stored procedure in PGAdmin, whereas in the second case(automated running using java) auto-analyze on tab1 is yet to start or is under process as the tab1 has just been modified .
This causes Postgres to choose a better plan at the time of running the query from PGAdmin, as compared to the time when we run using java.
Summary- Different plans were chosen because In the second case, auto-analyze did not complete by the time we ran the stored proc, resulting in a poor query plan selection.
I'm looking for information about prepared statements(ps) livecycle.
I'm using BasePoolableObjectFactory provided by apache to make connection pool. Then, using TSocket and TTransport, I'm making an object from Cassadra.Client class. This object has methods to execute cql3 queries. Also the object has method to prepare and execute cql3 query, i.e. we can make prepared statements using thrift.
Where are prepared statements stored? In database server or they exist as long as the object exist?
If they are stored in database server, how can I get then, without creating them again?
If they are stored in object, what is prepared then, when each time query is submitted to the server?
I'm asking, cos I want to know when it is worth making prepared statements. Because if I make a prepared statement, execute it once and close the connection, then there is no need to make prepared statements at all.
Prepared statements are cached per-connection. (You access them with the id that the server gives you back as the result of the prepare call.)
So no, there is no point in using them for one-shot queries. This is common across any prepared statement design.
I'm getting this BatchUpdateException from a stmt.executeBatch() statement:
BatchUpdateException: A resultset was created for update
The internet does not have any information on this exception message. What does it mean? The traceback doesn't contain anything useful other than that a stored procedure failed.
I'd interpret the message as meaning that an SQL statement that you added via addBatch() has produced a ResultSet, meaning that it's not your normal INSERT, UPDATE or DELETE statement.
Statements that should return results can't be executed in batches with JDBC.
The JDBC Tutorial (under the heading "Handling Batch Update Exceptions") confirms it:
You will get a BatchUpdateException when you call the method executeBatch if (1) one of the SQL statements you added to the batch produces a result set (usually a query) or (2) one of the SQL statements in the batch does not execute successfully for some other reason.
You seem to be running into case 1 here.
A batch-update are several insert/update/delete statements which are processed by the database together. This is usualy done for perfomance reasons. 1x 1000 inserts is much faster than 1000x 1 insert. A BatchUpdateException means 1 (or more) statements failed most often due to a constraint-violation.
You will have to look at the stored procedure to see what it has been doing. Mayby your dba can give you more information about what went wrong.
I dropped one column from the table.
When i tried to insert records to that table i was getting BatchUpdateException.
After running the below command the problem got solved
REORG TABLE TABLE_NAME
I have an application which processes a very large file and sends data to an oracle database (using Java 6, oracle 9).
In a loop, I use a PreparedStatement ps and create all SQL statements generated with ps.addBatch().
I have a situation where a BatchUpdateException bue is thrown somewhere during the ps.executeBatch(). At that point, the batch stops to be executed.
I'd like the batch execution to continue, so that I can then check on failed updates in a method processUpdateCounts(bue.getUpdateCounts()).
The javadoc about class BatchUpdateException says:
After a command in a batch update
fails to execute properly and a
BatchUpdateException is thrown, the
driver may or may not continue to
process the remaining commands in the
batch.
Is there a way to enforce continuation or do I need to alter my program so that it will execute the statement individually?
Just found this link:
JDBC Batch Update Problem
Apparently, it says there there is
NO WAY WITH ORACLE BATCH JDBC to proceed after first failure,
thus I am resorting to sending the inserts one by one.
Thank you
(sorry for not looking better to find the link above before).
there is a workaround that would allow you to use the batch feature. Instead of executing a simple INSERT statement, you can execute a PL/SQL block that will deal with the error appropriately:
BEGIN
INSERT INTO your_table VALUES (?,?,...?);
EXCEPTION
WHEN OTHERS THEN
/* deal with the error. For example, log the error id and error msg
so that you can list them after the batch */
INSERT INTO error_table VALUES (?, sqlerrm);
END
The performance should be on par with the batch insert (should be faster than individual execution of the statements). You could also call a stored procedure instead of a PL/SQL block.
Oracle itself can, see here: http://download.oracle.com/docs/cd/B19306_01/appdev.102/b14250/oci04sql.htm#sthref616
However, it doesn't seem that this functionality is exposed to JDBC, not even in the oracle specific classes.
Because of the rather useless JDBC error handling ("the driver may or may not continue"), I'm always setting a savepoint before the batch, and performing an rollback to that point on error. That's the only JDBC compliant way to establish a known state after an Oracle Batch Error--as far as I know.
Since the specification doesn't seem to mandate it (as clearly shown by the Javadoc), any "forced" continuation would have to be done on per-driver basis. A simple standard-compliant workaround would be to check the getUpdateCounts() returned array and "re-run" the batch for those statements which failed. You can make this approach a bit more sophisticated by putting in a logic for the number of retries.
Sure, this seems a bit messy (keeping track of the "batch" added and then checking the output) but would work across all databases and driver implementations. Just a thought...