I am doing batch inserts using postgres 9.3-1100-jdbc41 JDBC4 driver.
According to the JDBC specification, its up
to the application to disable autocommit and to commit or
rollback the transaction.
In my case, I am not using any transaction(i.e., auto commit is true) but still the inserts are rolled back if one of the insert in the batch fails.
According to JDBC specification "If one of the commands in a batch update fails to execute properly, this method throws a
BatchUpdateException, and a JDBC driver may or may not continue to process the remaining commands
in the batch.". Here it does not says that previously executed commands will be rolled back.
Is my understanding wrong? If not why driver is behaving in this way and if yes what is the right behavior according to specification.
As far as I can tell the spec essentially leaves this up to the driver; it doesn't specify whether or not already-processed work is committed if the batch fails.
PgJDBC executes the batch in a transaction, so if any part of the batch fails then it will all be aborted.
If you feel this behaviour to be incorrect, the first thing you need to do is write a test case demonstrating that other drivers consistently behave a different way to PgJDBC and submit it to the PgJDBC issue tracker. We do not have time to research behaviour of other drivers, so you need to write the test case and run it some other popular databases (MS SQL Server, Oracle, DB2, MySQL, etc) or arrange to have it run by others. If you show that PgJDBC's behaviour differs from how other drivers handle batches then it'll be worth thinking about adding an option to change the behaviour (and working on making it the default eventually).
Related
Earlier I was trying to get batch inserts working in Hibernate. I tried everything: For the config I set batch_size(50), order_inserts(true), order_updates(true), use_second_level_cache(false), use_query_cache(false). For the session I used setCacheMode(CacheMode.IGNORE) and setFlushMode(FlushMode.MANUAL). Still the MySQL query log showed that each insert was coming in separately.
The ONLY thing that worked was setting rewriteBatchedStatements=true in the JBDC connection string. This worries me, as my application is supposed to support any JBDC database and I'm trying to avoid DB specific optimizations.
Is the only reason hibernate can't actually use batch statements because the MySQL driver doesn't support them by default? What about other drivers, do I have to add options to the connection string so they can support batched inserts? If you need specific db's, think SQL server, sqlite, Postgres, etc
One reason it could not be working is that hibernate disables batching if you use the Identity id generation strategy.
Also MySQL doesn't support JDBC batch prepared statements the same way as other databases without turning on the rewrite option.
I don't see that it is a problem to turn this flag on though, if your are setting up your application for a different database you will have to change the settings such as dialect, driver name, etc. anyway and as this is part of the JDBC connect String then you are isolated from the configuration.
Basically I think you are doing the right thing.
As batch insert (or bulk insert) is part of the SQL standard, ORM frameworks like Hibernate support and implement it. Please see Chapter 13. Batch Processing and Hibernate / MySQL Bulk insert problem .
Basically, you need to set the JDBC batch size via the variable named hibernate.jdbc.batch_size to a reasonable size. Also don't forget to end the batch transaction with flush() and clear().
I have a service method on an api that can be called to check the health of my database connection.
The method is pulling the query string from a properties file (depends on DB vendor, using Sybase and HSQL for now, more in future), and executing it. Then the method lets the caller know if it succeeded or failed.
In addition to this, I was using the Query.setHint("javax.persistence.query.timeout") to set a timeout on the query:
javax.persistence.EntityManager entityManager;
...
Query heartbeatQuery = entityManager.createNativeQuery(heartbeatQueryString);
heartbeatQuery.setHint("javax.persistence.query.timeout", heartbeatTimeout);
heartbeatQuery.getResultList();
My problem is the timeout property is working against my Sybase DB, but not against my HSQL DB. It sounds like it depends on the vendor, so I don't know for sure when it will work.
Is there a better way to generically test the DB connection & include some kind of timeout parameter?
Well sadly no. JPA's query hints are not mandatory, i.e. it's up to the implementator (EclipseLink, Hibernate, etc) to enforce them or not. Moreover, even if the implementator does chose to recognize a certain query hint, if that hint's functionality is not supported by the database then it won't work (here some implementators are nice and tell you if a certain hint won't work agains the current db while others fail silently). In the case of HSQLDB there's no way to set the query timeout. You can only set a timeout for the login (i.e. how long should it wait for a successful login before failing), but not for the queries duration.
Things are not so grim however. On the one hand, even if you'd solve this, you'd still stumble over other issues with HSQLDB, as it does not support a lot of other nice functionalities that most dbs have. You should only use HSQLDB for basic integration/unit testing. For more involved testing, you can use the integrated MySQL Java library. You can find it here:
http://dev.mysql.com/doc/refman/5.0/en/connector-mxj.html
This is simply a packaged fully working Mysql server, which has a Java api for star and stop, works on most major OSs (win,lin, os x, etc). This way you can have your integration tests start a real Mysql server, and try your code there, where such stuff as a query timeout hint will work fine.
I have a transaction based application and it rolls the operation back on an error. However
in some cases, the rollback doesn't happen. ( Though its getting called from the application). Ours is a very complex application and there are chances that some code is directly committing the data. Is there a way to debug the commit to the database? (Either from java or from the database?). From java, we are not able to do this because, java.sql.Connection is an interface and the implementation is provided by sybase jconnect for which we don't have the source.
I am not sure this will help, but with this driver, this helps in tracing all the DML operations you perform on the database including commit & rollback. When you use p6spy
it logs every database hit into a log file, from where you can easily figure out where your application is performing a commit.
Apart from the above suggestion, I fell every database comes with some sort of monitoring tools, which with you can monitor which DML is fired within a span of time.
Is it possible to specify connection/query timeout for the Oracle database queries? Either on Oracle side or in Oracle's JDBC driver (10.2.0.4)? So, that Java client just got an error back after, let's say, 2 minutes instead of waiting until Oracle finishes executing the query?
If you are executing the query in the context of a transaction, the transaction timeout value of the JTA transaction monitor will be the determinant to query timeout. The configuration for this depends from one application server to another.
At an individual query level (in the absence of a JTA transaction monitor), the setQueryTimeout method can be used to set the timeout on the execution of a Statement/PreparedStatement/CallableStatement object.
Update
setQueryTimeout is not to be relied on, although it works (atleast from a J2SE client). It works via the JDBC driver performing a full round-trip to the Oracle database server. Then, it is upto the database to halt execution of the query. Don't rely on it for time critical applications.
Have a look at Oracle profiles. This allows you to specify several limits at the database level. One of them is a maximum CPU time per query.
If you have queries running for more than 2 minutes on a regular basis you might want to do some tuning of your queries first.
According to http://www.javamonamour.org/2012/09/oraclenetconnecttimeout.html
oracle.net.READ_TIMEOUT for jdbc versions < 10.1.0.5 oracle.jdbc.ReadTimeout for jdbc versions >=10.1.0.5
So if you are using a JDBC driver version 10.1.0.5 or higher, then oracle.jdbc.ReadTimeout is the correct property.
Setting oracle.jdbc.ReadTimeout helped to timeout the jdbc calls. I have used it in a production spring boot app by specifying datasource properties as below
spring.datasource.hikari.connection-timeout=1000
spring.datasource.hikari.dataSourceProperties=oracle.jdbc.ReadTimeout=2000
Note: Earlier this app was using tomcat connection pool and after setting the above properties for the same, timeouts were happening but the pool was not able to handle the closed connection objects efficiently. Therefore, I switched to hikariCP and got lot better results. I have made a video simulating the slow jdbc calls which compares the results with tomcat and hikari connection pools while using the ReadTimeout property.
I'm using an object/relational mapper to talk to the database. The library in my case is iBatis which is also used for transaction management.
However, I recently had a case where iBatis didn't actually start a transaction even though startTransaction() etc. was called as documented.
After some debugging I found out that there was a configuration mistake on my side.
One might blame iBatis but I would like to avoid such misconceptions in the future.
So here's the question:
How can I programmatically assert that the current database connection is running in a transaction?
The databases I'm using are Oracle, MySQL and H2 (for testing).
I'm not 100% sure if this is absolutely indicative of being in a tx, but Connection.getAutoCommit() tells you if the connection is in auto-commit mode, where auto-commit "on" means "no transaction".
There may well be cases where this assertion does not hold, but most JDBC-based frameworks will use that setting to control transactions.