I recently uses HikariCP. Before I use my own simple ConnectionPool to meet our needs. In our software sometimes we need to perform multiple database inserts where each inserts depends on some validation. More or less like the sample from this site: http://docs.oracle.com/javase/tutorial/jdbc/basics/transactions.html#commit_transactions
In my old way, when I was using my own conn pool, I will always set the connection object to setAutoCommit(false) before giving it to the requesting object so the database manager can rollback the data manually when something goes wrong. Like in the sample, if the try catches any exceptions, then it will call the rollback function. When I return the connection, I call the connection.commit() in the connection return and set autocommit back to true in the connection pool manager.
My question: does HikariCP still uses the same procedure for my needs? Meaning, set the autocommit to false (I read the manual, you have autocommit parameters for your config), and then we just manually rollback or commit the transaction then return to the pool? Or is there some automation done where we can just throw an exception and HikariCP will automatically call the rollback upon error or call commit upon connection return if I do not set the config param for Autocommit = false?
Thank you for any info.
Rendra
HikariCP auto-commit behavior is the same as without a pool. If autoCommit=false, you are responsible for commit/rollback in a try-finally. So, yes, you just commit/rollback and then return the connection to the pool.
The truth is that if autoCommit=false, and you run queries without committing, then HikariCP will automatically rollback on return to the pool. However, this is for safety and I discourage you from coding based on this behavior. Doing so will make your code less portable if you ever choose to switch pools.
Related
When working with a SQLite database in Java, suppose I set auto commit to false. Is it necessary when a SQLException occurs that I call rollback() method? Or can I simply ignore calling it and the transaction will automatically be rolled back (all the changes I made during the transaction will be undone automatically)?
Quick answer: The fact that you're asking means you're doing it wrong, probably. However, if you must know: Yes, you need to explicitly rollback.
What is happening under the hood
At the JDBC level (and if you're using JOOQ, JDBI, Hibernate, or something similar, that's a library built on top of JDBC usually), you have a Connection instance. You'd have gotten this via DriverManager.getConnection(...) - or a connection pooler got it for you, but something did.
That connection can be in the middle of a transaction (auto-commit mode merely means that the connection assumes you meant to write an additional commit() after every SQL statement you care to run in that connection's context, that's all auto-commit does, but, obviously, if that's on, you probably are in a 'clean' state, that is, the last command processed by that connection was either COMMIT or ROLLBACK).
If it is in the middle of a transaction and you close the connection, the ROLLBACK is implicit.
The connection has to make a choice, it can't keep existing, so, it commits or rolls back. The spec guarantees it doesn't just commit for funsies on you, so, therefore, it rolls back.
The question then boils down to your specific setup. This, specifically, is dangerous:
try (Connection con = ...) {
con.setAutoCommit(false);
try {
try (var s = con.createStatement()) {
s.execute("DROP TABLE foobar");
}
} catch (SQLException ignore) {
// ignoring an exception usually bad idea. But for sake of example..
}
// A second statement on the same connection...
try (var s = con.createStatement()) {
s.execute("DROP TABLE quux");
}
}
A JDBC driver is, as far as the spec is concerned, free to throw an SQLException along the lines of 'the connection is aborted; you must explicitly rollback first then you can use it again' on the second statement.
However, the above code is quite bad. You cannot use transaction isolation level SERIALIZABLE at all with this kind of code (once you get more than a handful of users, the app will crash and burn in a cavalcade of retry exceptions), and it is either doing something useless (re-using 1 connection for multiple transactions when you have a connection pooler in use), or is solving a problem badly (the problem of: Using a new connection for every transaction is pricey).
1 transaction, 1 connection
The only reason the above was dangerous is because we're doing two unrelated things (namely: 2 transactions) in a single try-block associated with a connection object. We're re-using the connection. This is a bad idea: connections have baggage associated with them: Properties that were set, and, yes, being in 'abort' state (where an explicit ROLLBACK is required before the connection is willing to execute any other SQL). By just closing the connection and getting a new one, you ditch all that baggage. This is the kind of baggage that results in bugs that unit tests are not going to catch easily, a.k.a. bugs that, if they ever trigger, cost a ton of money / eyeballs / goodwill / time to fix. Objectively you must prefer 99 easy-to-catch bugs if it avoids a single 100x-harder-to-catch bug, and this is one of those bugs that falls in the latter category.
Connections are pricey? What?
There's one problem with that: Just use a connection for a single transaction and then hand it back, which thus eliminates the need to rollback, as the connection will do that automatically if you close() it: Getting connections is quite resource-heavy.
So, folks tend to / should probably be using a connection pooler to avoid this cost. Don't write your own here either; use HikariCP or something like it. These tools pool connections for you: Instead of invoking DriverManager.getConnection, you ask HikariCP for one, and you hand your connection back to HikariCP when you're done with it. Hikari will take care of resetting it for you, which includes rolling back if the connection is halfway inside a transaction, and tackling any other per-connection settings, getting it back to known state.
The common DB interaction model is essentially this 'flow':
someDbAccessorObject.act(db -> {
// do a single transaction here
});
and that's it. This code, under the hood, does all sorts of things:
Uses a connection pooler.
Sets up the connection in the right fashion, which primarily involves setting auto-commit to false, and setting the right transaction isolation level.
will COMMIT at the end of the lambda block, if no exceptions occurred. Hands back the connection in either case, back to the pool.
Will catch SQLExceptions and analyse if they are retry exceptions. If yes, does nagle's algorithm or some other randomized exponential backoff and reruns the lambda block (that's what retry exceptions mean).
Takes care of having the code that 'gets' a connection (e.g. determines the right JDBC url to use) in a single place, so that a change in db config does not entail going on a global search/replace spree in your codebase.
In that model, it is somewhat rare that you run into your problem, because you end up in a '1 transaction? 1 connection!' model. Ordinarily that's pricey (creating connections is far more expensive that rolling back/committing as usual and then just continuing with a new transaction on the same connection object), but it boils down to the same thing once a pooler is being used.
In other words: Properly written DB code should not have your problem unless you're writing a connection pooler yourself, in which case the answer is definitely: roll back explicitly.
Trying to understand transactions... I'm using Spring's JdbcTemplate and #Transactional annotation in a method. Although the transaction is active during the method execution (as indicated by both the logs and TransactionSynchronizationManager.isActualTransactionActive()), autocommit also seems to be turned on (as shown by jdbcTemplate.getDataSource().getConnection().getAutoCommit())).
I don't understand two things
Isn't the idea of jdbc transactions turning off autocommit and manually commiting at the end of a code block? Isn't spring transaction manager supposed to be doing something like that under the hood?
If autocommit is really on: If one modification to the database succeeds, and a subsequent one fails, how can spring rollback the previous changes (which it does, despite the autocommit value)?
If the doBegin() method of the DataSourceTransactionManager is processed, the flag is handled.
https://docs.spring.io/spring/docs/2.5.x/javadoc-api/org/springframework/jdbc/datasource/DataSourceTransactionManager.html#doBegin(java.lang.Object,%20org.springframework.transaction.TransactionDefinition)
If you take a look at the sources there is:
if (con.getAutoCommit()) {
...
con.setAutoCommit(false);
}
I have a question about using prepared statements and a connection pool such as c3p0. I have a script running that interacts with my mysql database around once a second, ad infinitum. Each interaction executes a prepared statement. According to the documentation, a connection should be open and closed before and after a database interaction. My understanding is that the Connection object doesn't actually get destroyed, but added back to the pool. Since prepared statements are connection dependent, how do I use Prepared Statements without having to rebuild them every time that I get a connection from the pool - or do I just rebuild the statement after a connection is received by the pool and rely on the pool to do this efficiently via caching?
If your pool implements JDBC transparent Statement caching (as c3p0 does), you just use the ordinary JDBC PreparedStatement API and reuse of cached statements is handled for you.
Internally what happens is that when you call conn.prepareStatement(...) on some Connection, a lookup is performed on an internal hashtable using a key that includes the Connection's identity, the SQL text, and other characteristics of the requested prepared statement. If a suitable PreparedStatement is found, that is what is passed to the client. In none is, then the prepareStatement call gets passed to the Connection, and the returned PreparedStatement is cached for later reuse.
Statement caching itself has some overhead, and can be tricky to configure. Some newer Connection pools, most notably HikariCP, simply omit support, arguing that caching of PreparedStatements is better left to the DBMS. That, of course, is an empirical question, and will vary from DBMS to DBMS. If you do use Statement caching, the crucial point is that you need to allow for
[num_frequently_used_prepared_statments] * [num_connections]
distinct Statements to be cached. This is tricky to reason about, given the JDBC standard global maxStatements config property defines a global limit, even though PreparedStatements are scoped per-connection.
Much better, if you use c3p0, to set only the (nonstandard) maxStatementsPerConnection property. That should be set to at least the number of PreparedStatements frequently used by your application. You don't have to worry about how many Connections will be open, since maxStatementsPerConnection is scoped per Connection like the Statements themselves are.
I hope this helps!
please see the comments in spring DataSourceTransactionManager.java, function doBegin:
// Switch to manual commit if necessary. This is very expensive in some JDBC drivers,
// so we don't want to do it unnecessarily (for example if we've explicitly
// configured the connection pool to set it already).
if (con.getAutoCommit()) {
txObject.setMustRestoreAutoCommit(true);
if (logger.isDebugEnabled()) {
logger.debug("Switching JDBC Connection [" + con + "] to manual commit");
}
con.setAutoCommit(false);
}
In the project I'm working on, autocommit is not configured. So it is true by default. We are using Spring to manage transactions, and all SQLs are executed within #Transactional annotated functions. So Transactions are acturally manually committed. Everytime a transaction begin, the db connection is set autocommit to false, and after the transaction exit autocommit is set back to true. A typical workflow would be (at JDBC level)：
conn = dataSource.getConnection();
conn.setAutoCommit(false);
stmt = conn.createStatement();
stmt.executeQuery(...);
conn.commit()/ conn.rollback();
conn.setAutoCommit(true);
Is setting autocommit back and forth expensive? should we config datasource connection pool autocommit=false for performance reason? to skip the step 2 and step 6.
1) autocommit is totally dependent on the database, what it means is that, each and every statement through the connection will be executed in a separate transaction which is implicitly carried out. Unless and until, you want to use personal coding and avoid those locks being held by multiple statements which could lead to conflicts with other users, there is no need to set the autocommit to false.
2)From performance points of view,
a) if you have a lot of users and there is some conflict that is occuring because of holding of database locks,
then, there may be a need to check into the issues pertaining to it but as a general
rule, autocommit was introduced for simplification of stuff for beginners.
b) there may be instances where you need to rollback .
c) you want to commit the transaction manually based on a specific condition.
EDIT: I see you have edited the question, to answer you simply, autocommit=false will force you to write your own commit/rollback/etc, performance is totally dependent on the database, the number of locks held at a moment in real time!!
No. setting autocommit to false and true again will not increase the toll on the system.
NO, do not config datasource connection pool autocommit=false unless you are doing it for some specific reason and are an experienced person. From performance points of view, as i already decalred, it is dependent on the type of database and real time users accessing the database at an instance, for your project, 99.99 percent you wouldn't be needing to set it to false.
setting autocommit to true will just ensure that commit is called after each and every statement.
I also see that you are getting your connection from a datasource, in such cases, it is best to leave the connection with default settings so that the next time that connection is fetched from the pool, there wouldn't be any trouble with the workflow
Hope this helped!!
In bulk operations you can set it off in your session and set again on after bulk operation completion to gain performance.
SET autocommit=0;
your code here....
SET autocommit=1;
Update:
As #codemania explained very well that even there is option to disable autocommit as per your requirement but you should not do this. Even this is basic need of transaction to either successfully commit a set of instructions or rollback, if you do it disable then how you will achieve it.
This will be useful if you are doing some bulky task like data migration etc, for it you can disable autocommit to gain performance but only in that session.
You should set autocommit to true whenever you're issuing database transactions. A database trasaction is a logical unit of work, which usually consists of multiple database operations (usually multiple updates) and you want either all of them to succeed or all of them to fail. With autocommit=false, your changes will not be made persistant until you call commit() on the connection object, so this approach guarantees that all your updates will either succeed or fail (by calling rollback in case of an exception etc).
When autocommit is set to true (the default), you can, for instance, change one table but then on the second update (whether it be an update/insert or delete), an exception may occur and your second table doesn't get updated, thus leaving your database in an inconsistent state.
In conclusion, autocommit=true is OK when just reading data or when the database data model is simple and accessed by few users (so that concurrent access to the same regions of data is very rare and when some database inconsistencies can even be tolerated)
Is a connection only returned to the Connection pool in a JPA application if i call
entityManager.close();
?
Can the connection backing the entitymanger change during its lifecycle?
thanks in advance
mojoo
The JPA spec doesn't define such things and its up to the implementation to manage connections. When a transaction is active you'd be safe to assume the connection is the same until commit, for obvious reasons. Once the txn ends it may be handed back, or it may be held depending on implementation (and you don't mention yours)
This depends on the JPA implementation and configuration.
In EclipseLink by default a connection is only held for the duration of an active (dirty) transaction. i.e. from the first modification or lock, until the commit or rollback. For non-transactional queries a connection is acquired on demand and returned after the query execution. This allows for maximal usage of connection pooling. So, normally em.close() does nothing.
You can configure this using the "eclipselink.jdbc.exclusive-connection.mode" persistence unit property. "Always" will hold a connection for the life of the EntityManager.
You can also use different connection pools for transactions, versus non-transactional reads. This is useful with JTA, as you can use a non-JTA DataSource for reads.