We Keep Coding

Shared Transaction between different OracleDB Connections

After several days passed to investigate about the issue, I decided to submit this question because there is no sense apparently in what is happening.
The Case
My computer is configured with a local Oracle Express database.
I have a JAVA project with several JUnit Tests that extend a parent class (I know that it is not a "best practice") which opens an OJDBC Connection (using a static Hikari connection pool of 10 Connections) in the #Before method and rolled Back it in the #After.
public class BaseLocalRollbackableConnectorTest {
private static Logger logger = LoggerFactory.getLogger(BaseLocalRollbackableConnectorTest.class);
protected Connection connection;
#Before
public void setup() throws SQLException{
logger.debug("Getting connection and setting autocommit to FALSE");
connection = StaticConnectionPool.getPooledConnection();
}
#After
public void teardown() throws SQLException{
logger.debug("Rollback connection");
connection.rollback();
logger.debug("Close connection");
connection.close();
}
StacicConnectionPool
public class StaticConnectionPool {
private static HikariDataSource ds;
private static final Logger log = LoggerFactory.getLogger(StaticConnectionPool.class);
public static Connection getPooledConnection() throws SQLException {
if (ds == null) {
log.debug("Initializing ConnectionPool");
HikariConfig config = new HikariConfig();
config.setMaximumPoolSize(10);
config.setDataSourceClassName("oracle.jdbc.pool.OracleDataSource");
config.addDataSourceProperty("url", "jdbc:oracle:thin:#localhost:1521:XE");
config.addDataSourceProperty("user", "MyUser");
config.addDataSourceProperty("password", "MyPsw");
config.setAutoCommit(false);
ds = new HikariDataSource(config);
}
return ds.getConnection();
}
}
This project has hundreds tests (not in parallel) that use this connection (on localhost) to execute queries (insert/update and select) using Sql2o but transaction and clousure of connection is managed only externally (by the test above).
The database is completely empty to have ACID tests.
So the expected result is to insert something into DB, makes the assertions and then rollback. in this way the second test will not find any data added by previous test in order to maintain the isolation level.
The Problem
Running all tests together (sequentially), 90% of times they work properly. the 10% one or two tests, randomly, fail, because there is dirty data in the database (duplicated unique for example) by previous tests. looking the logs, rollbacks of previous tests were done properly. In fact, if I check the database, it is empty)
If I execute this tests in a server with higher performance but the same JDK, same Oracle DB XE, this failure ratio is increased to 50%.
This is very strange and I have no idea because the connections are different between tests and the rollback is called each time. The JDBC Isolation level is READ COMMITTED so even if we used the same connection, this should not create any problem even using the same connection.
So my question is:
Why it happen? do you have any idea? Is the JDBC rollback synchronous as I know or there could be some cases where it can go forward even though it is not fully completed?
These are my main DB params:
processes 100
sessions 172
transactions 189

I have run into the same problem 2-3 years ago (I have spent a lot of time to get this straight). The problem is that the #Before and #After is not always really sequential. [You could try this by starting the process in debug and place some breakpoints in the annotated methods.
Edit: I was not clear enough as Tonio pointed out. The order of #Before and #After is guarantied in terms of running before the test and afterwards it. The problem was in my case that sometimes the #Before and the #After was messed up.
Expected:
#Before -> test1() -> #After -> #Before -> #test2() -> #After
But sometimes I experienced the following order:
#Before -> test1() -> #Before -> #After -> #test2() -> #After
I am not sure thet it is a bug or not. At the time I dug into the depth of it and it seemed like some kind of (processor?) scheduling related magic.
The solution to that problem was in our case to run the tests on a single thread and call manually the init and cleanup processes... Something like this:
public class BaseLocalRollbackableConnectorTest {
private static Logger logger = LoggerFactory.getLogger(BaseLocalRollbackableConnectorTest.class);
protected Connection connection;
public void setup() throws SQLException{
logger.debug("Getting connection and setting autocommit to FALSE");
connection = StaticConnectionPool.getPooledConnection();
}
public void teardown() throws SQLException{
logger.debug("Rollback connection");
connection.rollback();
logger.debug("Close connection");
connection.close();
}
#Test
public void test() throws Exception{
try{
setup();
//test
}catch(Exception e){ //making sure that the teardown will run even if the test is failing
teardown();
throw e;
}
teardown();
}
}
I have not tested it but a much more elegant solution could be to syncronize the #Before and #After methods on the same object. Please update me if You have the chanse to give it a try. :)
I hope it will solve your problem too.

If your problem just needs to be "solved" (e.g. not "best practice") regardless of performance to just make the tests complete in order, try to set:
config.setMaximumPoolSize(1);
You might need to set a timeout higher since the tests in the test queue will wait for its turn and might timeout. I usually don't suggest solutions like this but your setup is suboptimal, it will lead to race conditions and data loss. However, good luck with the tests.

Try configure audit on all statements in Oracle. Then find sessions which live simultaneously. I think that there is the problem in tests. JDBC rollback is synchronous. Commit can be configured as commit nowait but I don't think you do it special in your tests.
Also pay attention on parallel dml. On one table in the same transaction you can't do parallel dml + any other dml without commit because you get Ora-12838.
Do you have autonoumous transaction? Business logic in tests can manually rollback them and during tests autonoumous transaction is like another session and it doesn't see any commits from parent session.

Not sure if this will fix it, but you could try:
public class BaseLocalRollbackableConnectorTest {
private static Logger logger = LoggerFactory.getLogger(BaseLocalRollbackableConnectorTest.class);
protected Connection connection;
private Savepoint savepoint;
#Before
public void setup() throws SQLException{
logger.debug("Getting connection and setting autocommit to FALSE");
connection = StaticConnectionPool.getPooledConnection();
savepoint = connection.setSavepoint();
}
#After
public void teardown() throws SQLException{
logger.debug("Rollback connection");
connection.rollback(savepoint);
logger.debug("Close connection");
connection.close();
while (!connection.isClosed()) {
try { Thread.sleep(500); } catch (InterruptedException ie) {}
}
}
Really there are two 'fixes' there - loop after the close to be sure the connection IS closed before returning to the pool. Second, create a savepoint before the test and restore it afterwards.

Like all other answers have pointed out, it's hard to say what goes wrong with the provided information. Further more, even if you manage to find the current issue by audit, it doesn't mean that your tests are free from data errors.
But here's an alternative: because you already have a blank database schema, you can export it to a SQL file. Then before each test:
Drop the schema
Re-create the schema again
Feed the sample data (if needed)
It would save lots of time debugging, make sure the database in its pristine state every time you run the tests. All of this can be done in a script.
Note: Oracle Enterprise has the flashback function to support your kind of operation. Also, if you can manage to use Hibernate and the likes, there's other in-memory databases (like HSQLDB) that you can utilize to both increase testing speed and maintain coherence in your data set.
EDIT: It seems implausible, but just in case: connection.rollback() only takes effect if you don't call commit()
before it.

After all confirmation from your answers that I am not mad with Rollbacks and transactions behavior in unit tests, i deeply checked all queries and all possible causes and fortunately (yes furtunately...even if I'm ashamed for that, I make my mind free) all works as expected (Transactions, Before, After, etc).
There are some queries that get the result of some complex views (and radically deep configured into the DAO layer) to identify the single row information.
This view is based on the MAX of a TIMESTAMP in order to identify latest of a particular event (in the real life the events coming after several months).
Doing the preparation of the database to proceed with the unit tests, these events are added sequentially by each test.
In some cases, when these insert queries under the same transaction are particular fast, more events related to the same object are added in the same Millisecond (The TIMESTAMP is added manually using a JODA DateTime) and the MAX of a date, returns two or more values.
For this reason it is explained the fact that on more performant computers/servers, this happened more frequently than the slower ones.
This view is used in more tests and depending by the test, the error is different and random (NULL value added as Primary Key, duplicated primary Key, etc) .
For Example: in the following INSERT SELECT query is evident this bug:
INSERT INTO TABLE1 (ID,COL1,COL2,COL3)
SELECT :myId, T.VAL1, T.VAL2, T.VAL3
FROM MyView v
JOIN Table2 t on t.ID = v.ID
WHERE ........
the parameter myId is added afterwards as Sql2o Parameter
MyView is
SELECT ID, MAX(MDATE) FROM TABLEV WHERE.... GROUP BY ...
When the view returns at least 2 results due to the same Max Date, it fails because the ID is fixed (generated by a sequence at beginning but stored using the parameter in a second time). This generates the PK constraint violated.
This is only one case but make me (and my colleagues) crazy due to this randomly behaviours...
Adding a sleep of 1 millisecond between those events insert, it is fixed. now we are working to find a different solution even though this case (an user that interact two times in the same millisecond) cannot happen in production system
but the important things is that no magic happens as usual!
Now you can insult me :)

You can do one thing increase the no. of connections in max pool size and rollback the operation in the same place where you committed the operation instead of using it in #after statement.
Hope it will work.

Avoid clients keeping reference of a connection while implementing a connection Pool

I have implemented a connection pool. All is good. Now If a client borrows a connection and even returns it to the pool but the client also keeps the reference of this connection with him. Now, if pool returns same connection to another client; this will lead to same connection being used by multiple people.
How can I avoid that ?

Do not return the underlying connection object, but another object which wraps it. Within that object (using some kind of private property) store the state of that object; is it still available for use, or has it been invalidated by being returned to the pool or some other condition like being timed out). Then you can intercept any method call that attempts to use it and check against its state. If it is no longer available for use, throw an exception.
The wrapped connection object will also need to be private, so that the client cannot access it directly.
You will have one wrapper per client, but two or more wrappers may share the underlying connection object. But because you are storing state per client, only one client can use the object at one time.
Edited to include an untested example - which now shows a big problem with my approach.
Assuming you are returning something which implements java.sql.Connection, you could return instances of the below class.
package same.package.as.your.pool; // so your pool has access to set isValidConnection
import java.sql.Connection;
class MyConnection implements Connection {
private Connection actualConnection;
private boolean isValidConnection = false;
MyConnection(Connection conn) {
// package acccess for pool class to create connection
actualConnection = conn;
isValidConnection = true;
}
public boolean getIsValidConnection() {
return isValidConnection;
}
void setIsValidConnection(boolean isValid) {
// pool class can call this to invalidate when returned to pool or timed out
isValidConnection = isValid;
}
// intercept java.sql.Connection methods, checking if connection is still valid first
// for example
PreparedStatement prepareStatement(String sql) {
if (! isValidConnection) {
// WHAT TO DO HERE?
}
return actualConnection.prepareStatement(sql);
}
// ... and the rest
First big problem is that - ideally you would throw an Exception from the methods like prepareStatement when the connection is no longer valid because it's been returned to the pool. But because you are constrained by the caught exceptions of the original interface (in this case, throws SQLException) you'd either need to throw an SQLException (yuk, it isn't really an SQLException) or an uncaught exception (yuk - client code would probably want to catch the case where the pooled connection is not longer valid) or something else :-)
Two other issues with the code above - package access to protect the methods meant to be only available to your pool code is not very robust. Maybe you could create the MyConnection code as some kind of inner class within your pool code. Finally, having to override all java.sql.Connection interface would be a pain.
}

Hibernate faster EntityManagerFactory creation

In my desktop application new databases get opened quite often. I use Hibernate/JPA as an ORM.
The problem is, creating the EntityManagerFactory is quite slow, taking about 5-6 Seconds on a fast machine. I know that the EntityManagerFactory is supposed to be heavyweight but this is just too slow for a desktop application where the user expects the new database to be opened quickly.
Can I turn off some EntityManagerFactory features to get an instance
faster? Or is it possible to create some of the EntityManagerFactory lazily to speed up cration?
Can I somehow create the EntityManagerFactory object before
knowing the database url? I would be happy to turn off all
validation for this to be possible.
By doing so, can I pool EntityManagerFactorys for later use?
Any other idea how to create the EntityManagerFactory faster?
Update with more Information and JProfiler profiling
The desktop application can open saved files. Our application document file format constists of 1 SQLite database + and some binary data in a ZIP file. When opening a document, the ZIP gets extracted and the db is opened with Hibernate. The databases all have the same schema, but different data obviously.
It seems that the first time I open a file it takes significantly longer than the following times.
I profiled the first and second run with JProfiler and compared the results.
1st Run:
create EMF: 4385ms
build EMF: 3090ms
EJB3Configuration configure: 900ms
EJB3Configuration <clinit>: 380ms
.
2nd Run:
create EMF: 1275ms
build EMF: 970ms
EJB3Configuration configure: 305ms
EJB3Configuration <clinit>: not visible, probably 0ms
.
In the Call tree comparison you can see that some methods are significantly faster (DatabaseManager. as starting point):
create EMF: -3120ms
Hibernate create EMF: -3110ms
EJB3Configuration configure: -595ms
EJB3Configuration <clinit>: -380ms
build EMF: -2120ms
buildSessionFactory: -1945ms
secondPassCompile: -425ms
buildSettings: -346ms
SessionFactoryImpl.<init>: -1040ms
The Hot spot comparison now has the interesting results:
.
ClassLoader.loadClass: -1686ms
XMLSchemaFactory.newSchema: -184ms
ClassFile.<init>: -109ms
I am not sure if it is the loading of Hibernate classes or my Entity classes.
A first improvement would be to create an EMF as soon as the application starts just to initialize all necessary classes (I have an empty db file as a prototype already shipped with my Application). #sharakan thank you for your answer, maybe a DeferredConnectionProvider would already be a solution for this problem.
I will try the DeferredConnectionProvider next! But we might be able to speed it up even further. Do you have any more suggestions?

You should be able to do this by implementing your own ConnectionProvider as a decorator around a real ConnectionProvider.
The key observation here is that the ConnectionProvider isn't used until an EntityManager is created (see comment in supportsAggressiveRelease() for a caveat to that). So you can create a DeferredConnectionProvider class, and use it to construct the EntityManagerFactory, but then wait for user input, and do the deferred initialization before actually creating any EntityManager instances. I'm written this as a wrapper around ConnectionPoolImpl, but you should be able to use any other implementation of ConnectionProvider as the base.
public class DeferredConnectionProvider implements ConnectionProvider {
private Properties configuredProps;
private ConnectionProviderImpl realConnectionProvider;
#Override
public void configure(Properties props) throws HibernateException {
configuredProps = props;
}
public void finalConfiguration(String jdbcUrl, String userName, String password) {
configuredProps.setProperty(Environment.URL, jdbcUrl);
configuredProps.setProperty(Environment.USER, userName);
configuredProps.setProperty(Environment.PASS, password);
realConnectionProvider = new ConnectionProviderImpl();
realConnectionProvider.configure(configuredProps);
}
private void assertConfigured() {
if (realConnectionProvider == null) {
throw new IllegalStateException("Not configured yet!");
}
}
#Override
public Connection getConnection() throws SQLException {
assertConfigured();
return realConnectionProvider.getConnection();
}
#Override
public void closeConnection(Connection conn) throws SQLException {
assertConfigured();
realConnectionProvider.closeConnection(conn);
}
#Override
public void close() throws HibernateException {
assertConfigured();
realConnectionProvider.close();
}
#Override
public boolean supportsAggressiveRelease() {
// This gets called during EntityManagerFactory construction, but it's
// just a flag so you should be able to either do this, or return
// true/false depending on the actual provider.
return new ConnectionProviderImpl().supportsAggressiveRelease();
}
}
a rough example of how to use it:
// Get an EntityManagerFactory with the following property set:
// properties.put(Environment.CONNECTION_PROVIDER, DeferredConnectionProvider.class.getName());
HibernateEntityManagerFactory factory = (HibernateEntityManagerFactory) entityManagerFactory;
// ...do user input of connection info...
SessionFactoryImpl sessionFactory = (SessionFactoryImpl) factory.getSessionFactory();
DeferredConnectionProvider connectionProvider = (DeferredConnectionProvider) sessionFactory.getSettings()
.getConnectionProvider();
connectionProvider.finalConfiguration(jdbcUrl, userName, password);
You could put the initial set up of the EntityManagerFactory on a separate thread or something, so that the user never has to wait for it. Then the only thing they'll wait for, after specifying the connection info, is the setting up of the connection pool, which should be fairly quick compared to parsing the object model.

Can I turn off some EntityManagerFactory features to get an instance faster?
Don't believe so. EMFs don't really have too many features, other than initializing a JDBC connection/pool.
Or is it possible to create some of the EntityManagerFactory lazily to
speed up cration?
Rather than creating the EMF lazily, when the user will notice the performance hit, I suggest you should head in the opposite direction - create the EMF proactively before the user actually needs it. Create it once, up-front, possibly in a separate thread during application initialisation (or at least as soon as you know about your database). Reuse it throughout the existence of your application/database.
Can I somehow create the EntityManagerFactory object before knowing the database url?
No - it creates a JDBC connection.
I think a better question is: why does your application dynamically discover database connection URLs? Are you saying your databases are created/made available on-the-fly and there's no way to anticipate in advance the connection parameters. That really is to be avoided.
By doing so, can I pool EntityManagerFactorys for later use?
No, you can't pool EMFs. It's the connections that you can pool.
Any other idea how to create the EntityManagerFactory faster?
I agree - 6 seconds is too slow for initialisation of EMFs.
I suspect it's more to do with your selected database technology than JPA/JDBC/JVM. My guess is that maybe your database is initialising itself as you connect. Are you using Access? What DB are you using?
Are you connecting to a database remotely located? Over a WAN? Is network speed/latency good?
Are the client PCs limited in performance?
EDIT: Added after comments
Implementing your own ConnectionProvider as a decorator around a real ConnectionProvider will not speed up the user's experience at all. The database instance still needs to be initialised, the EMF & EM created and the JDBC connection still needs to be subsequently established.
Options:
Share a common preloaded DB instance: seems not possible for your business scenario (although JSE technology supports this and also supports client-server design).
Change to a DB with a faster startup: Derby (a.k.a. Java DB) is included in modern JVMs and has a startup time of about 1.5 seconds (cold) and 0.7 seconds (warm - data pre-loaded).
In many (most?) scenarios, the fastest solution would be to load data directly into in-memory java objects using JAXB with STAX. Subsequently, use in-memory cached data (particularly using smart structures like maps, hashing and arraylists). Just as JPA can map POJO classes to database tables & columns, so JAXB can map POJO classes to XML schema & work with XML doc instances. If you have very complex queries using SQL set-based logic with multiple joins and strong use of DB indexes, this would be less desirable.
(2) would probably give the best improvement for limited effort.
Additionally:
- try to unzip the data files during deployment rather than during app usage.
- initialize the EMF in a startup thread that runs in parallel to the UI startup - try to start the DB initializing as one of the very first steps of the app (that means connecting to the actual instance using JDBC).

Best practice of managing database connections in a web app

I am developing a MongoDB app with Java but I think this question related to datastore connections for web apps in general.
I like to structure all web apps with four top-level packages which are called (which I think will be self explanatory):
Controller
Model
Dao
Util
Ideally I would like to have a class in the Dao package that handles all the connections details.
So far I have created a class that looks like this:
public class Dao {
public static Mongo mongo;
public static DB database;
public static DB getDB() throws UnknownHostException, MongoException{
mongo = new Mongo("localhost");
database = mongo.getDB("mydb");
return database;
}
public static void closeMongo(){
mongo.close();
}
}
I use it in my code with something like this
public static void someMethod(String someData){
try {
DB db = Dao.getDB();
DBCollection rColl = db.getCollection("mycollection");
// perform some database operations
Dao.closeMongo();
} catch (UnknownHostException e) { e.printStackTrace(); } catch (MongoException e) { e.printStackTrace();
}
}
This seems to work fine, but I'd be curious to know what people think is the "best" way to handle this issue, if there is such a thing.

The rule of thumb when connecting to relational database server is to have a pool. For example if you connect to an oracle database using a pool gives you some performance benefits both in terms of connection setup time and sql parsing time (if you are using bind variables). Other relational database may vary but my opinion is that a pool is a good pattern even for some other reason (eg. you may want to limit the maximum number of connections with your db user). You are using MongoDB so the first thing to check is how MongoDB handles connections, how expnsive is creating a connection,etc. I suggest to use/build a class that can implements a pool logic because it gives you the flexibility you may need in the future. Looking at your code it seems that you api
DB db=Dao.getDB();
should be paired with:
Dao.closeDB(DB db);
So you have a chance to really close the connection or to reuse it without affecting the Dao code. with these two methods can switch the way you manage connections without recoding the Dao objects

I would suggest you can write a java class to establish the connection with the database.
The arguments to the method should be the database name, password, host port and other necessary credentials.
You can always call the parametrized constructor everywhere where there is a need to establish database connectivity. This can be a model.

I got a 'nice' solution from this article. http://www.lennartkoopmann.net/post/722935345
Edit Since that link is dead, here's one from waybackmachine.org
http://web.archive.org/web/20120810083748/http://www.lennartkoopmann.net/post/722935345
Main Idea
What I found interesting was the use of a static synchronised method that returns an instance of the static class and its variables. Most professional devs probably find this obvious. I found this to be a useful pattern for managing the db connections.
Pooling
Mongo does automatic connection pooling so the key is to use just one connection to the datastore and let it handle its own pooling.

I think it is better if you call a method inside DAO to get data from database as well. If you do it in this way, say your database got changed. Then you have to edit many classes if you get data directly calling db queries. So if you separate db calling methods inside the DAO class itself and call that method to get data it is better.

How to keep SSH connection available?

I'm writing an application which uses sshj library for SSH connections. User opens Connect Dialog from menu, types login data and clicks Connect button to establish a connection. Then the user performs various operations from different panels and frames. My question is: where should I keep this connection to make it available to every panel and frame that needs it until the user clicks Disconnect button? I thought about static field in some custom class but I'm not convinced to do so. What are your ideas?

It sounds like you are interested in some type of pool.
You can do this in different ways. One way is to have a class that handles all the connections, which is a singleton, sort of. Then you simply ask the pool for a connection, for instance
SSHConnection con = ConnectionPool.getConnection(host, port);
You could also use a proxy for this. In a proxy, you actually get a placeholder instead of the actual connection. The proxy shares information with other instances, sort of like
class ConnectionProxy {
private static SSHConnection connection;
}
the connection variable is shared, and when you create a new ConnectionProxy, you will actually get the old connection, but it looks like you get a new connection.
Another option is to simply pass the instance around to all classes that needs it. This will allow you to keep track of who does what, but you will loose the flexibility of getting a connection from wherever you want.
Passing an instance around will help if you are debugging though, think of it like this. You are in a crowded room and you have a some money laying in a box that needs a key to open. If you hand that key to one person (the class that needs it) and leave the room and come back and the money is gone, you can blame that person. If instead you just leave the key on the table next to the box (the instance is global) and leave the room, come back and the money is gone, then good luck finding out who took it.
References:
*] Proxy pattern - WikiPedia
Proxy example
More specifically, it could look something like this:
public class ConnectionProxy implements Connectable {
// where Connectable is some shared interface between ConnectionProxy
// and SSHConnection. The proxy should "look like" a connection, it just
// hides the actual connection behind the scenes.
private static SSHConnection connection;
public ConnectionProxy() { }
public ConnectionProxy(String host, int port) {
connection = new SSHConnection(host, port);
}
public void connect(String host, int port) {
if (isConnected()) {
return;
}
connection = new SSHConnection(host, port);
}
public void reconnect() {
connection = new SSHConnection(connection.getHost(), connection.getPort());
}
public boolean isConnected() {
return connection != null && connection.isConnected();
}
}
You use the class by either instantiating it or connecting it.
class Program {
public void sendMessage() {
ConnectionProxy proxy = new ConnectionProxy();
if (!proxy.isConnected()) {
// I guess noone connected it
return;
}
proxy.sendBytes();
}
}
And in your connection dialog you instantiate or connect the ConnectionProxy. You could add support for several connections, i.e. different hosts and ports, just by making the variable connection a list instead, and checking the host and port for the right connection. You basically create a pool of connections, but to the client class it looks like it is creating the connection.

Doesn't matter where you put it, so long as you provide a getter method for retrieving it, and that getting method is public.
I think it's more a question of where it logically belongs. If it's a property of the application (as opposed to a window, frame, profile, whatever) put the getter in the main application class.

Design-wise, I'd recommend wrapping the operations you can perform over SSH in a separate class (RemoteCommands or some such), and injecting (setting as a property) an instance of this class everywhere you need to perform a remote command from.
If this seems like a lot of extra boilerplate code because every single frame and panel needs it, this shouldn't say "I need a global variable" to you. It should say "I should reduce the number of components that directly execute remote commands".