I have a web application which I've been slowly migrating from iBATIS 2 to JPA with Spring Data.
For the most part, things have been going well, with me just migrating the DAO for one domain object at a time. However, an issue that's been brought to my attention recently is that stale result lists are being show in some parts of the site.
For example, I have a "ticket" section, which shows a list of open tickets, and lets you view specific tickets on separate pages. When I create a new ticket, I can view that ticket on its specific page correctly. However, the open tickets list doesn't seem to show this new ticket until some time later.
Things I've tried to rule out:
I see this issue even on a system with MySQL's query cache disabled
I see this issue even when I set cacheModelsEnabled="false" in the iBATIS config.
I see this issue even when I completely remove the <cacheModel> element and cacheModel="x" attributes from my sqlMap file.
As soon as I restart the application, I see the up-to-date results.
When I execute the query iBATIS should be running here in a MySQL client, I do see the new ticket which is missing from iBATIS' results.
When I mocked up a simple ticket list using Spring MVC and Spring Data JPA, I do see the new ticket.
I've also tried to rule out some sort of weird transaction state with iBATIS, but it doesn't seem that any transaction is being used here at all.
What am I missing? Is there anything else I should be trying to figure this out? Or, should I just prioritize replacing the iBATIS layer completely with Spring Data JPA, which seems to be immune from this problem?
UPDATE
I've now gone through a lot of my recent changes with git bisect, and I've narrowed it down to a change that introduced Spring's org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.
So, it would seem that some transaction is living longer than it should. I'll add more logging to see if I can confirm this, and then look for a way to avoid using that filter.
If you are doing select, insert, select in the same SqlSession,
then the SqlSession cache is causing this issue. You will need to
clear the cache manually after the insert: sqlSession.clearCache().
So, it seems a combination of things ended up happening here:
Most of my code was not explicitly using transactions.
I had at some point changed to use Tomcat's JDBC Connection Pool, which does not reset autocommit by default when a connection is returned to the pool. I expect that my older DBCP-based stuff did this implicitly, though.
The introduction of OpenEntityManagerInViewFilter may have caused a SET autocommit=0 to be called at some point, with no corresponding SET autocommit=1 later, if nothing had changed.
By chance, or perhaps some design, the code that inserted a new record into the database and then immediately retrieved and showed it, seemed to get a different Connection than the code that showed my list of records.
The default MySQL transaction isolation level of REPEATABLE-READ meant that my listings were showing the old results.
The fix I've found, which seems to work in my testing so far, is to add these defaultAutoCommit and jdbcInterceptors attributes to my connection pool config:
<Resource name="jdbc/DB" auth="Container" type="javax.sql.DataSource"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
...
defaultAutoCommit="true" jdbcInterceptors="ConnectionState;StatementFinalizer" />
Related
I am trying to implement row level security so our application can enforce more stringent access control.
One of the technologies we are looking into is Oracle's Virtual Private Database, which allows row level security by basically augmenting all queries against specific tables with a where clause predicate. Since we are in a web environment, we need to set up a special context within Oracle, inside a single request's thread. We use connection pooling with a service account.
I started to look into Eclipse Link and Hibernate. Eclipse Link seems to have events that fit perfectly into this model.
This would involve us migrating from hibernate, which is not a problem, but we would then be bound to EL for these events.
Oracle seems to imply that they implement at the data source level in Web Logic product.
The context is set and cleared by the WebLogic data source code.
Question: Is it more appropriate to do this at the DataSource level with some series of events. What are the events or methods that I should pay the most attention too?
Added Question: How would I extend a connection pool to safely initialize an oracle context with some custom data? I am digging around in Apache, and it seems like extending BasicDataSource doesn't give me access to anything that would allow me to clean up the connection when Spring is done with it.
I need to set up a connection, and clean up a connection as the exit / enter the connection pool. I am hoping for an implementation that is so simple, no one can mess it up by breaking some delicate balance of products.
- Specifically we are currently using Apache Commons DBCP Basic Data Source
This would allow us to use various ways to connect to the database and still have our security enforced. But I don't see a great example or set of events to work with, and rolling my own security life cycle is never a good idea.
I eventually solved my problem by extending some of the Apache components.
First I extended org.apache.commons.pool.impl.GenericObjectPool and overrode both borrowObject() and returnObject(). I knew the type of the objects in the pool (java.sql.Connection) so I could safely cast and work with them.
Since for my case I was using Oracle VPD, I was able to set information in the Application context. I recommend you read about that in more depth. It is a little complicated and there are a lot of different options to hide or share data at various contexts level, and across RAC nodes. Start
In essence what I did was generate a nonce and use it to instantiate a session within oracle, and then set the access level of the user to a variable in that session, that the Oracle VPD policy would then read and use to do the row level filtering.
I instantiated and destroyed that information in my overridden borrowObject() and returnObject() The SQL I ran was something like this:
CallableStatement callStat =
conn.prepareCall("{call namespace.cust_ctx_pkg.set_session_id(" + Math.random() + ")}");
callStat.execute();
Note math.random() isn't a good nonce.
Next was to simply extend org.apache.commons.dbcp.BasicDataSource and set my object pool by overriding createConnectionPool(). Note that the way I did this disabled some functionality I did not need, so you may need to rewrite more or less than I did.
You can try any object level security mechanism for simplicity, like Spring Security ACL.
You will want to do this at the application layer. You will want a pre-commit hook and a post read hook.
The pre-commit hook is used to ensure that data from the client is being presented by a user authorized to modify that data. This prevents an unauthorized user from overwriting data that they shouldn't be able to access.
It's not intuitive, but the post read hook is used to keep the client from accessing data the user shouldn't be allowed to view. This happens post-view because this is being enforced at the application layer, not at the data layer. The application has no way to know if the caller is allowed to access the data until it's been retrieved from the data layer. In the post read hook you evaluate the credential on each row returned against the credential of the logged in user in order to determine whether or not access is allowed. If access is denied on any row then an exception would be raised and the data would not be returned to the client.
Application level security done in this way requires that you have a way to connect each row in a table to a permission/role required to access it and a way to evaluate a user's permissions on the server at runtime.
Hope that helps.
You will get better control by using one of the other Commons DBCP Datasources.
The Basic one is just that: basic :)
The ones in org.apache.commons.dbcp.datasources package gives you more fine-grained control.
I have a service method on an api that can be called to check the health of my database connection.
The method is pulling the query string from a properties file (depends on DB vendor, using Sybase and HSQL for now, more in future), and executing it. Then the method lets the caller know if it succeeded or failed.
In addition to this, I was using the Query.setHint("javax.persistence.query.timeout") to set a timeout on the query:
javax.persistence.EntityManager entityManager;
...
Query heartbeatQuery = entityManager.createNativeQuery(heartbeatQueryString);
heartbeatQuery.setHint("javax.persistence.query.timeout", heartbeatTimeout);
heartbeatQuery.getResultList();
My problem is the timeout property is working against my Sybase DB, but not against my HSQL DB. It sounds like it depends on the vendor, so I don't know for sure when it will work.
Is there a better way to generically test the DB connection & include some kind of timeout parameter?
Well sadly no. JPA's query hints are not mandatory, i.e. it's up to the implementator (EclipseLink, Hibernate, etc) to enforce them or not. Moreover, even if the implementator does chose to recognize a certain query hint, if that hint's functionality is not supported by the database then it won't work (here some implementators are nice and tell you if a certain hint won't work agains the current db while others fail silently). In the case of HSQLDB there's no way to set the query timeout. You can only set a timeout for the login (i.e. how long should it wait for a successful login before failing), but not for the queries duration.
Things are not so grim however. On the one hand, even if you'd solve this, you'd still stumble over other issues with HSQLDB, as it does not support a lot of other nice functionalities that most dbs have. You should only use HSQLDB for basic integration/unit testing. For more involved testing, you can use the integrated MySQL Java library. You can find it here:
http://dev.mysql.com/doc/refman/5.0/en/connector-mxj.html
This is simply a packaged fully working Mysql server, which has a Java api for star and stop, works on most major OSs (win,lin, os x, etc). This way you can have your integration tests start a real Mysql server, and try your code there, where such stuff as a query timeout hint will work fine.
I have a bunch of tests in a Hibernate/Spring application. Yesterday, I transitioned them from using the JUnit 3.8 base test class provided by Spring to the JUnit 4.4 one.
Everything works great, because now, my tests are wrapped in transactions, and data created/modified is automatically rolled back (instead of me writing code to delete newly-created entities).
The only problem is that I cannot peek into the database during test execution. If a test fails, I often add breakpoints near the end and peer into the MySQL database via SQL Yog to see what's going on. But now, I just see empty tables. (I mean in integration tests that simulate production very closely and actually touch the database.)
I tried setting the global isolation level to read uncommitted, but it didn't change the fact that I can't see the data. How can I configure Spring/Hibernate to allow me to view the data from another process?
I had the same issue, and found that setting the session isolation level while using YOG sometimes helped.
SET SESSION TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
This only uncovered another disturbing issue - while running tests, the Hibernate didn't actually run some of the actions unless I used HibernateTemplate.flush(); after every Hibernate operation.
As this very annoying, I finally set Hibernate so it would always flush queries, like this:
HibernateTemplate hibernateTemplate;
...
hibernateTemplate.setFlushMode(HibernateTemplate.FLUSH_ALWAYS);
I'm writing a web app that supports multiple users. Each user has their own database - using H2. all database schemas are the same.
I wish to use Spring + Hibernate for this application.
So I'm stuck at how to associate a user's database with that user - maybe associated it in the HTTPSession, and extend spring's AbstractRoutingDataSource? but wouldn't this effect Hibernate's cache? Another way is to have a SessionFactory with each datasource, even though every datasource's schema is the same... so I see that as a waste.
Anyways selecting the datasource needs to be dynamic - they can't be pre-configured in context files, as each new user will have its own database created. Is there any existing frameworks/solutions?
I don't know too much about Hibernate Shards, maybe that works?
I might be wrong about the (strict) need to have one SessionFactory per database, as suggested by some resources:
Dynamic DataSource Routing
I'll take some time to re-read everything tomorrow (I didn't get all the details to be honest) and to fully understand the implications of such a setup (although it seems clear that it will break the second-level cache). I'll come back on this later.
I'm writing a web app that supports multiple users. Each user has their own database - using H2. all database schemas are the same.
I wonder how this will scale... How many users do you have? How do you run H2, what mode?
So I'm stuck at how to associate a user's database with that user - maybe associated it in the HTTPSession, and extend spring's AbstractRoutingDataSource?
You'll have to build a SessionFactory per user and associate it to the logged user (in a Map, using the login as key) and then obtain a Session from a given SessionFactory. Binding the lifecycle of the SessionFactory to the HTTP session seems to be a good idea (to save some memory) but I am not sure Spring will be very helpful here. I might be wrong but a variation of the HibernateUtil class and a fully programmatic approach looks easier. I'm not sure you'll need multiple connections per user by the way.
but wouldn't this effect Hibernate's cache?
What cache?
Another way is to have a SessionFactory with each datasource, even though every datasource's schema is the same... so I see that as a waste.
Oh, it's a waste, but that's what you want to do (one database per user). And you don't have the choice (you need one SessionFactory per datadabase). Why do you need one database per user actually? Are you sure this is a wise decision? As already hinted, this means much troubles, won't scale well, adds complexity, etc. Why not using a single database and associating data to the user?
Anyways selecting the datasource needs to be dynamic - they can't be pre-configured in context files, as each new user will have its own database created. Is there any existing frameworks/solutions?
Not to my knowledge. Which is also why I think you'll have to do everything programatically.
I don't know too much about Hibernate Shards, maybe that works?
Given the dynamic needs of your application, I don't see how it could help.
This may help you:
Dynamic Datasource via Spring using HotSwappableTargetSource
Hibernate + Spring using multiple datasources?
Thanks to the help from the 2 people (Pascal and org.life.java)!
It is possible, but with some problems: e.g. the hibernate 2nd level cache/query cache.
This link supplied by Pascal is a very good resource:
http://www.jroller.com/kenwdelong/entry/horizontal_database_partitioning_with_spring.
My main motivation for giving each user a separate database is because the data is likely to grow rapidly, thus horizontal partitioning is required.
I have a scenario where the unit of work is defined as:
Update table T1 in database server S1
Update table T2 in database server S2
And I want the above unit of work to happen either completely or none at all (as the case with any database transaction). How can I do this? I searched extensively and found this post close to what I am expecting but this seems to be very specific to Hibernate.
I am using Spring, iBatis and Tomcat (6.x) as the container.
It really depends on how robust a solution you need. The minimal level of reliability on such a thing is XA transactions. To use that, you need a database and JDBC driver that supports it for starters, then you could configure Spring to use it (here is an outline).
If XA isn't robust enough for you (XA has failure scenarios, such as if something goes wrong in the second phase of commits, such as a hardware failure) then what you really need to do is put all the data in one database and then have a separate process propagate it. So the data may be inconsistent, but it is recoverable.
Edit: What I mean is that put the whole of the data into one database. Either the first database, or a different database for this purpose. This database would essentially become a queue from which the final data view is fed. The write to that database (assuming a decent database product) will be complete, or fail completely. Then, a separate thread would poll that database and distribute any missing data to the other databases. So if the process should fail, when that thread starts up again it will continue the distribution process. The data may not exist in every place you want it to right away, but nothing would get lost.
You want a distributed transaction manager. I like using Atomikos which can be run within a JVM.