I have a Java Spring Boot application reading and writing data to a local Oracle 19c database.
I have the following CommandLineRunner:
#Override
public void run(String... args) {
final EntityManager em = entityManagerFactory.createEntityManager();
final EntityTransaction transaction = em.getTransaction();
transaction.begin();
em.persist(customer());
//COMMENT1 em.flush();
/*COMMENT2
Query q = em.createQuery("from " + Customer.class.getName() + " c");
#SuppressWarnings("unchecked")
final Iterator<Object> iterator = (Iterator<Object>) q.getResultList().iterator();
while (iterator.hasNext()) {
Object o = iterator.next();
final Customer c = (Customer) o;
log.info(c.getName());
}
*/
transaction.rollback();
}
When I run this code, using a packet sniffer to monitor TCP traffic between the application and database, I see what I expect: nothing particularly interesting in the conversation, as the em.persist(customer()) will not be flushed.
When I include the code in COMMENT1, then I'm surprised to find that the conversation looks the same - there is nothing interesting after the connection handshake.
When I include the code in COMMENT2, however, then I get a more complete TCP conversation. Now, the captured packets show that the write operation was indeed flushed to the database, and I can also see evidence of the read operation to list all entities following it.
Why is it that the TCP conversation does not reflect the explicit flush() when only COMMENT1 is removed? Why do I need to include COMMENT2 to see an insert into customer... statement captured in the TCP connection?
A call to flush() synchronizes your changes in the persistence context with the database but it may not commit the transaction immediately. Consider it as an optimization to avoid unnecessary DB writes on each flush.
When you un-comment the 2nd block then you see the flush getting executed for sure. This happens because the EM ensures your select query gets all the results in the latest state from DB. It, therefore, commits the flushed changes (alongwith any other changes done via other transactions, if any).
em.persist(customer());
persist does not directly insert the object into the database:
it just registers it as new in the persistence context (transaction).
em.flush();
It flushes the changes to the database but doesn't commit the transaction.
A query gets fired if there is a change expected in database(insert/update/delete)
em.rollback or em.commit will actually rollback or commit the transaction
And all these scenario depends on the flush mode, and I think behaviour is
vendor dependent. Assuming Hibernate, most probably FlushMode is set to auto in
your application and so the desired result as in second scenario.
AUTO Mode : The Session is sometimes flushed before query execution in order to ensure
that queries never return stale state
https://docs.jboss.org/hibernate/orm/3.5/javadocs/org/hibernate/FlushMode.html
In spring boot I think you can set it as
spring.jpa.properties.org.hibernate.flushMode=AUTO
Related
I'm working on some legacy code and trying to understand what's going on. EJB 3 + Hibernate 3.2 (pray for my poor soul)
There are 2 EJBs (A and B):
EJB-A retrieves an Object1 and modifies some field data.
SessionFactory sf = PersistenceManager.getSessionFactory;
Transaction tx = sessionContext.getUserTransaction;
tx.begin;
Session s = sf.getCurrentSesssion();
ObjectA a = (ObjectA)s.get(ObjectA.class, id);
a.setField(value);
s.update(a);
// other stuff happens and then transaction is committed
tx.commit();
// starts a new transaction?
tx = sessionContext.getUserTransaction();
tx.begin();
// Re initializes object
ObjectA a = (ObjectA)s.get(ObjectA.class, id);
EJB-B b = getService(EJB-b) // sorry pseudo coding this
b.performOperation();
Inside EJB-B, it reads the piece of data set in EJB-A. What I'm seeing is that the data is not updated when EJB-B reads it. I'm a bit confused, I assume that committing the transaction would force a flush and the session to push the data to the database.
Can someone explain to me what could be going on? Is it possible the commit call is performing some async call and is taking longer to actually persist the data than it takes to get to the next EJB and retrieve the data? Is there any way to get around that? Would it be possible to call tx.wasCommitted in a loop or something to wait until proceeding?
Thank you in advance for any insight.
Hibernate doesn't flush the data instantaneously, it flushes when the internal cache reaches some limit or the transaction is closed.
You can use flush here, refer Correct use of flush() in JPA/Hibernate.
Background
We have 2 services written in Java - one handles database operations on different files (CRUD on database), the other handles long-running processing of those records (complicated background tasks). Simply we could say they are producer and consumer.
Supposed behavior is as follows:
Service 1 (uses the code bellow):
Store file into DB
If the file is of type 'C' put it into message queue for further processing
Service 2:
Receive the message from message queue
Load the file from the database (by ID)
Perform further processing
The code of Service 1 is as follows (I changed some names for corporate reasons)
private void persist() throws Exception {
try (SqlSession sqlSession = sessionFactory.openSession()) {
FileType fileType = FileType.fromFileName(filename);
FileEntity dto = new FileEntity(filename, currentTime(), null, user.getName(), count, data);
oracleFileStore.create(sqlSession, dto);
auditLog.logFileUploaded(user, filename, count);
sqlSession.commit();
if (fileType == FileType.C) {
mqClient.submit(new Record(dto.getId(), dto.getName(), user));
auditLog.logCFileDetected(user, filename);
}
}
}
Additional info
ActiveMQ 5.15 is used for message queue
Database is Oracle 12c
Database is handled by Mybatis 3.4.1
Problem
From time to time it happens, that Service 2 receives the message from MQ, tries to read the file from the database and surprisingly - file is not there. The incident is pretty rare but it happens. When we check the database, the file is there. It almost looks like the background processing of the file started before the file was put into database.
Questions
Is it possible that MQ call could be faster than the database commit? I created the file in DB, called commit and only after that I put the message into MQ. The MQ even contains the ID which is generated by database itself (sequence).
Does the connection needs to be closed to be sure the commit was performed? I always thought when I commit then it's in the database regardless if my transaction ended or not.
Can the problem be Mybatis? I've read some problems regarding Mybatis transactions/sessions but it doesn't seem similar to my problem
Update
I can provide some additional code although please understand that I cannot share everything for corporate reasons. If you don't see anything obvious in this, that's fine. Unfortunately I cannot continue in much more deeper analysis than this.
Also I basically wanted to confirm whether my understanding of SQL and Mybatis is correct and I can mark such response for correct as well.
SessionFactory.java (excerpt)
private SqlSessionFactory createLegacySessionFactory(DataSource dataSource) throws Exception
{
Configuration configuration = prepareConfiguration(dataSource);
return new SqlSessionFactoryBuilder().build(configuration);
}
//javax.sql.DataSource
private Configuration prepareConfiguration(DataSource dataSource)
{
//classes from package org.apache.ibatis
TransactionFactory transactionFactory = new JdbcTransactionFactory();
Environment environment = new Environment("development", transactionFactory, dataSource);
Configuration configuration = new Configuration(environment);
addSettings(configuration);
addTypeAliases(configuration);
addTypeHandlers(configuration);
configuration.addMapper(PermissionMapper.class);
addMapperXMLs(configuration); //just add all the XML mappers
return configuration;
}
public SqlSession openSession()
{
//Initialization of factory is above
return new ForceCommitSqlSession(factory.openSession());
}
ForceCommitSqlSession.java (excerpt)
/**
* ForceCommitSqlSession is wrapper around mybatis {#link SqlSession}.
* <p>
* Its purpose is to force commit/rollback during standard commit/rollback operations. The default implementation (according to javadoc)
* does
* not commit/rollback if there were no changes to the database - this can lead to problems, when operations are executed outside mybatis
* session (e.g. via {#link #getConnection()}).
*/
public class ForceCommitSqlSession implements SqlSession
{
private final SqlSession session;
/**
* Force the commit all the time (despite "generic contract")
*/
#Override
public void commit()
{
session.commit(true);
}
/**
* Force the roll back all the time (despite "generic contract")
*/
#Override
public void rollback()
{
session.rollback(true);
}
#Override
public int insert(String statement)
{
return session.insert(statement);
}
....
}
OracleFileStore.java (excerpt)
public int create(SqlSession session, FileEntity fileEntity) throws Exception
{
//the mybatis xml is simple insert SQL query
return session.insert(STATEMENT_CREATE, fileEntity);
}
Is it possible that MQ call could be faster than the database commit?
If database commit is done the changes are in the database. The creation of the task in the queue happens after that. The main thing here is that you need to check that commit does happen synchronously when you invoke commit on session. From the configuration you provided so far it seems ok, unless there's some mangling with the Connection itself. I can imagine that there is some wrapper over the native Connection for example. I would check in debugger that the commit call causes the call of the Connection.commit on the implementation from the oracle JDBC driver. It is even better to check the logs on the DB side.
Does the connection needs to be closed to be sure the commit was performed? I always thought when I commit then it's in the database regardless if my transaction ended or not.
You are correct. There is no need to close the connection that obeys JDBC specification (native JDCB connection does that). Of cause you can always create some wrapper that does not obey Connection API and does some magic (like delays commit until connection is closed).
Can the problem be Mybatis? I've read some problems regarding Mybatis transactions/sessions but it doesn't seem similar to my problem
I would say it is unlikely. You are using JdbcTransactionFactory which does commit to the database. You need to track what happens on commit to be sure.
Have you checked that the problem is not on the reader side? For example it may use long transaction with serialized isolation level, in this case it wouldn't be able to read changes in the database.
In postgres if the replication is used and replicas are used for read queries reader may see outdated data even if commit successfully completed on master. I'm not that familiar with oracle but it seems that if replication is used you may see the same issue:
A table snapshot is a transaction-consistent reflection of its master data as that data existed at a specific point in time. To keep a snapshot's data relatively current with the data of its master, Oracle must periodically refresh the snapshot
I would check the setup of the DB to know if this is the case. If replicatiin is usedyou need to change your approach to this.
I am trying to perform batch inserts with data that is currently being inserted to DB one statement per transaction. Transaction code statement looks similar to below. Currently, addHolding() method is being called for each quote that comes in from an external feed, and each of these quote updates happens about 150 times per second.
public class HoldingServiceImpl {
#Autowired
private HoldingDAO holdingDao;
#Transactional(propagation = Propagation.REQUIRES_NEW, rollbackFor = Exception.class)
public void addHolding(Quote quote) {
Holding holding = transformQuote(quote);
holdingDao.addHolding(holding);
}
}
And DAO is getting current session from Hibernate SessionFactory and calling save on object.
public class HoldingDAOImpl {
#Autowired
private SessionFactory sessionFactory;
public void addHolding(Holding holding) {
sessionFactory.getCurrentSession().save(holding);
}
}
I have looked at Hibernate batching documentation, but it is not clear from document how I would organize code for batch inserting in this case, since I don't have the full list of data at hand, but rather am waiting for it to stream.
Does merely setting Hibernate batching properties in properties file (e.g. hibernate.jdbc.batch_size=20) "magically" batch insert these? Or will I need to, say, capture each quote update in a synchronized list, and then insert list load and clear list when batch size limit reached?
Also, the whole purpose of implementing batching is to see if performance improves. If there is better way to handle inserts in this scenario, let me know.
Setting the property hibernate.jdbc.batch_size=20 is an indication for the hibernate to Flush the objects after 20. In your case hibernate automatically calls sessionfactory.flush() after 20 records saved.
When u call a sessionFactory.save(), the insert command is only fired to in-memory hibernate cache. Only once the Flush is called hibernate synchronizes these changes with the Database. Hence setting hibernate batch size is enough to do batch inserts. Fine tune the Batch size according to your needs.
Also make sure your transactions are handled properly. If you commit a transaction also forces hibernate to flush the session.
Why do we need to always Start a transaction in hibernate to save,insert,delete or update?
Is the auto commit feature by default false in hibernate?
like for this
public static void main(String[] args) {
System.out.println("creating empopbjects");
emp e1=new emp("a","x",1234);
emp e2=new emp("b","y",324);
emp e3=new emp("c","z",23345);
System.out.println("saving emp objects..");
Session s=myfactory.getsession();
s.save(e1);
s.save(e2);
s.save(e3);
s.close();
System.out.println("successfully saved");
}
This wont save anything whereas if i add Transaction then only it is added?why so?
public static void main(String[] args) {
System.out.println("creating empopbjects");
emp e1=new emp("a","x",1234);
emp e2=new emp("b","y",324);
emp e3=new emp("c","z",23345);
System.out.println("saving emp objects..");
Session s=myfactory.getsession();
Transaction t =s.beginTransaction();
s.save(e1);
s.save(e2);
s.save(e3);
t.commit();
s.close();
System.out.println("successfully saved");
}
Well defined answer of your question from one of the SO page is here.
Look at the following code, which
accesses the database without
transaction boundaries:
Session session = sessionFactory.openSession();
session.get(Item.class, 123l);
session.close();
By default, in a Java SE environment
with a JDBC configuration, this is
what happens if you execute this
snippet:
A new Session is opened. It doesn’t obtain a database connection at this
point.
The call to get() triggers an SQL SELECT. The Session now obtains a JDBC
Connection from the connection pool.
Hibernate, by default, immediately
turns off the autocommit mode on this
connection with setAutoCommit(false).
This effectively starts a JDBC
transaction!
The SELECT is executed inside this JDBC transaction. The Session is
closed, and the connection is returned
to the pool and released by Hibernate
— Hibernate calls close() on the JDBC
Connection. What happens to the
uncommitted transaction?
The answer to that question is, “It
depends!” The JDBC specification
doesn’t say anything about pending
transactions when close() is called on
a connection. What happens depends on
how the vendors implement the
specification. With Oracle JDBC
drivers, for example, the call to
close() commits the transaction! Most
other JDBC vendors take the sane route
and roll back any pending transaction
when the JDBC Connection object is
closed and the resource is returned to
the pool.
Obviously, this won’t be a problem for
the SELECT you’ve executed, but look
at this variation:
Session session = getSessionFactory().openSession();
Long generatedId = session.save(item);
session.close();
This code results in an INSERT
statement, executed inside a
transaction that is never committed or
rolled back. On Oracle, this piece of
code inserts data permanently; in
other databases, it may not. (This
situation is slightly more
complicated: The INSERT is executed
only if the identifier generator
requires it. For example, an
identifier value can be obtained from
a sequence without an INSERT. The
persistent entity is then queued until
flush-time insertion — which never
happens in this code. An identity
strategy requires an immediate INSERT
for the value to be generated.)
Bottom line: use explicit transaction demarcation.
Hibernate Session is a Unit of Work, and queries are only executed during flush time (which may occur prior to executing any query, or when your currently executing Transaction is committed).
Auto-commit is only meaningful in SQL consoles and it's undesirable in Enterprise applications. When using an ORM tool, you are managing Entity object state transitions rather than executing DML operations. It's only at the flush time, when state transitions are translated to DML operations.
So, while you can write JDBC statements in auto-commit, JPA doesn't allow you to do so.
I have a strange problem on one of my glassfish server. Have a look on this piece of code:
userTransaction.begin();
MyEntity entity = new MyEntity(12345);
//setting values..
entityManager.persist(entity);
MyEntity persistedEntity = entityManager.createQuery("SELECT p FROM MyEntity p WHERE p.idpk=12345").getSingleResult();
//...
userTransaction.commit(); //OK => the tuple is in the DB
Now there is a business problem and the transaction needs to be rollbacked.
userTransaction.begin();
MyEntity entity = new MyEntity(12345);
//setting values..
entityManager.persist(entity);
MyEntity persistedEntity = entityManager.createQuery("SELECT p FROM MyEntity p WHERE p.idpk=12345").getSingleResult();
//...
//Business problem => rollback
userTransaction.rollback(); //ERROR => the tuple 12345 is in the DB !
Even if the rollback seems to work (no exception raised or strange log output), the tuple has been committed in the database...
To search where the problem is, I tried the following code:
userTransaction.begin();
MyEntity entity = new MyEntity(12345);
//setting values..
entityManager.persist(entity);
//Business problem => rollback
userTransaction.rollback(); //OK => the tuple 12345 is NOT in the DB !
With this code (the entity is not retrieved), the tuple is not committed to the database, which is the correct behaviour. Let's go further:
userTransaction.begin();
MyEntity entity = new MyEntity(12345);
//setting values..
entityManager.persist(entity);
MyEntity persisted = entityManager.find(MyEntity.class, 12345);
//...
//Business problem => rollback
userTransaction.rollback(); //OK => the tuple 12345 is NOT in DB
In this last case, the result is still correct and there is no tuple committed to the database. It seems the EntityManager makes a magic [unwanted] commit when retrieving the entity with a query...
I also tried to make a query on another table and that doesn't cause the wrong commit (the rollback works). Moreover, I tried to reproduce the problem on my own server, and there is no problem: all the rollbacks work correctly. Hence, it should really be a server configuration problem.
For information, it is a glassfish v2.1.1 running on linux. The EntityManager is in FlushModeType.AUTO.
Does anyone have an idea ?
Thanks & best regards !
Found some discussion which seems to explain some of what you're seeing:
Via P-331 of EJB3 In Action Book:
By default, the database flush mode is set to AUTO. This means that
the Entity-Manager performs a flush operation automatically as needed.
In general, this occurs at the end of a transaction for
transaction-scoped EntityManagers and when the persistence context is
closed for application-managed or extendedscope EntityManagers. In
addition, if entities with pending changes are used in a query, the
persistence provider will flush changes to the database before
executing the query. If the flush mode is set to COMMIT, the
persistence provider will only synchronize with the database when the
transaction commits. However, you should be careful with this, as it
will be your responsibility to synchronize entity state with the
database before executing a query. If you don’t do this and an
EntityManager query returns stale entities from the database, the
application can wind up in an inconsistent state.
P-353 EJB3 In Action Book states:
If the Query is set to FlushModeType.COMMIT, the effect of updates
made to entities in the persistence context is not defined by the
specification, and the actual behavior is implementation specific.
Perhaps try toggling the flush mode on the query returned by
entityManager.createQuery("SELECT p FROM MyEntity p WHERE p.idpk=12345").getSingleResult();
or on the entity manager itself and see if that changes anything?
Some further discussion here:
http://www.coderanch.com/t/475041/ORM/databases/EntityManager-setFlushMode-COMMIT-Vs-Query
Why not do this:
MyEntity persisted = entityManager.persist(entity);
I finally found where was the problem: in the JDBC Connection Pool, there is a section "Transaction" where the option "Non Transactional Connections" was enabled... Simply removing this option made the whole thing work :)