We Keep Coding

Why is my data only flushed to the DB on read?

I have a Java Spring Boot application reading and writing data to a local Oracle 19c database.
I have the following CommandLineRunner:
#Override
public void run(String... args) {
final EntityManager em = entityManagerFactory.createEntityManager();
final EntityTransaction transaction = em.getTransaction();
transaction.begin();
em.persist(customer());
//COMMENT1 em.flush();
/*COMMENT2
Query q = em.createQuery("from " + Customer.class.getName() + " c");
#SuppressWarnings("unchecked")
final Iterator<Object> iterator = (Iterator<Object>) q.getResultList().iterator();
while (iterator.hasNext()) {
Object o = iterator.next();
final Customer c = (Customer) o;
log.info(c.getName());
}
*/
transaction.rollback();
}
When I run this code, using a packet sniffer to monitor TCP traffic between the application and database, I see what I expect: nothing particularly interesting in the conversation, as the em.persist(customer()) will not be flushed.
When I include the code in COMMENT1, then I'm surprised to find that the conversation looks the same - there is nothing interesting after the connection handshake.
When I include the code in COMMENT2, however, then I get a more complete TCP conversation. Now, the captured packets show that the write operation was indeed flushed to the database, and I can also see evidence of the read operation to list all entities following it.
Why is it that the TCP conversation does not reflect the explicit flush() when only COMMENT1 is removed? Why do I need to include COMMENT2 to see an insert into customer... statement captured in the TCP connection?

A call to flush() synchronizes your changes in the persistence context with the database but it may not commit the transaction immediately. Consider it as an optimization to avoid unnecessary DB writes on each flush.
When you un-comment the 2nd block then you see the flush getting executed for sure. This happens because the EM ensures your select query gets all the results in the latest state from DB. It, therefore, commits the flushed changes (alongwith any other changes done via other transactions, if any).

em.persist(customer());
persist does not directly insert the object into the database:
it just registers it as new in the persistence context (transaction).
em.flush();
It flushes the changes to the database but doesn't commit the transaction.
A query gets fired if there is a change expected in database(insert/update/delete)
em.rollback or em.commit will actually rollback or commit the transaction
And all these scenario depends on the flush mode, and I think behaviour is
vendor dependent. Assuming Hibernate, most probably FlushMode is set to auto in
your application and so the desired result as in second scenario.
AUTO Mode : The Session is sometimes flushed before query execution in order to ensure
that queries never return stale state
https://docs.jboss.org/hibernate/orm/3.5/javadocs/org/hibernate/FlushMode.html
In spring boot I think you can set it as
spring.jpa.properties.org.hibernate.flushMode=AUTO

JPA auto flush before any query

From JPA documentation I can see that the AUTO is the default flush mode, flush should happen before any query execution. I have tried this on spring boot jpa and I can see the flush won't happen on queries from different entities , is that expected ? even though different entity may have relation to it ( Department <--> Person here )
The flush should trigger before any query according to this article :
https://vladmihalcea.com/how-do-jpa-and-hibernate-define-the-auto-flush-mode/
// this triggers flush //
Person person = personRepository.findById(5L).get();
person.setName("hello test");
Person person1 = (Person) entityManager.createQuery("select person from Person
person where person.id=11").getSingleResult(); // flush before query
// this doesn't trigger flush even if the department has the person //
Person person = personRepository.findById(5L).get();
person.setName("hello test");
Department department= (Department) entityManager.createQuery("select
department from Department
department where department.id=1").getSingleResult();
Update:
I noticed the flush happens for JPQL queries on the same table only that has the DML , while for native sql queries it will always flush before any query if there is DML before. even though no flush happens the JPQL return the managed entity with modification not the one in DB. can anyone please explain if this follow JPA standard or not ?

As JPA is a specification this question is simple to answer. Check out the spec :-)
3.10.8 Queries and Flush Mode
The flush mode setting affects the result of a query as follows.
When queries are executed within a transaction, if FlushModeType.AUTO is set on the Query, TypedQuery, or StoredProcedureQuery object, or if the flush mode setting for the persistence context is AUTO (the default) and a flush mode setting has not been specified for the query object, the persistence provider is responsible for ensuring that all updates to the state of all entities in the persistence context which could potentially affect the result of the query are visible to the processing of the
query. The persistence provider implementation may achieve this by flushing those entities to the database or by some other means. If FlushModeType.COMMIT is set, the effect of updates made to entities in the persistence context upon queries is unspecified.
If the persistence context has not been joined to the current transaction, the persistence provider must
not flush to the database regardless of the flush mode setting.
package javax.persistence;
public enum FlushModeType {
COMMIT,
AUTO
}
If there is no transaction active, the persistence provider must not flush to the database
https://download.oracle.com/otn-pub/jcp/persistence-2_1-fr-eval-spec/JavaPersistence.pdf?AuthParam=1561799350_4cc62583442da694a6a033af82faf986
Then there is the Hibernate Doc:
6.1. AUTO flush
By default, Hibernate uses the AUTO flush mode which triggers a flush in the following circumstances:
prior to committing a Transaction
prior to executing a JPQL/HQL query that overlaps with the queued entity actions
before executing any native SQL query that has no registered synchronization
https://docs.jboss.org/hibernate/orm/5.4/userguide/html_single/Hibernate_User_Guide.html#flushing

How to use EntityManager to find an entity by id without hitting cache in one query?

I have a critical section of code where I need to read and lock an entity by id with pessimistic lock.
This section of code looks like this right now:
MyEntity entity = entityManager.find(MyEntity.class, key);
entityManager.refresh(entity, LockModeType.PESSIMISTIC_WRITE);
It works OK, but as I understand in case when there is no entity in the hibernate's cache, we will use 2 read transactions to a database. 1st transaction to find the entity by id and another transaction to refresh and lock the entity.
Is it possible to use only one transaction in such scenario?
I would imagine something like:
boolean skipCache = true;
MyEntity entity = entityManager.find(MyEntity.class, key,
LockModeType.PESSIMISTIC_WRITE, skipCache);
But there is no such parameter like skipCache. Is there another approach to read an entity by id directly from the database by using EntityManager?
UPDATE:
This query will hit the first level cache in case the entity exists in the cache. Thus, it may potentially return the outdated data and that is why isn't suitable for critical sections where any read should be blocked:
MyEntity entity = entityManager.find(MyEntity.class, key, LockModeType.PESSIMISTIC_WRITE);
The question is about skipping the cache and not about locking.

I've just found a method getReference in the EntityManager which gets an instance, whose state may be lazily fetched. As said in the documentation:
Get an instance, whose state may be lazily fetched. If the requested
instance does not exist in the database, the EntityNotFoundException
is thrown when the instance state is first accessed. (The persistence
provider runtime is permitted to throw the EntityNotFoundException
when getReference is called.) The application should not expect that
the instance state will be available upon detachment, unless it was
accessed by the application while the entity manager was open.
As a possible solution to find and lock an up to date entity by id in one query we can use the next code:
MyEntity entity = entityManager.getReference(MyEntity.class, key);
entityManager.refresh(entity, LockModeType.PESSIMISTIC_WRITE);
This query will create an entity (no database query) and then refresh and lock the entity.

Why not directly pass the requested lock along with the query itself?
MyEntity entity = entityManager.find(MyEntity.class, key, LockModeType.PESSIMISTIC_WRITE);
As far as I understand this is doing exactly what you wanted. (documentation)

You can also set entityManager property just before you use the find method to address not hitting the cache.
Specifying the Cache Mode
entityManager.setProperty("javax.persistence.cache.storeMode", CacheStoreMode.REFRESH);
MyEntity entity = entityManager.find(MyEntity.class, key);

JPA entities updated implicitly by new query

THe title can't definitely reflect my question, however I don't know how to express. I have a JPA entity (VatOperatorBalance which has a field saleBalance), lets say I retrieve the entity at the first time, and get a entity (VatOperatorBalance#3d6396f5), its saleBalance is 100.0. Now there are other operations which has modified the saleBalance to 200, now I query from database and get a new entity (VatOperatorBalance#10f8ed), sure the saleBalance of this entity is 200.0. However what make me confused it the saleBalance of the old entity (VatOperatorBalance#3d6396f5) is also 200.0.
All these queries and operations are in a single transaction, and the query isn't by EntityManager.find(java.lang.Class<T> entityClass, java.lang.Object primaryKey) which will return entity from cache.
Below is my code
#Rollback(true)
#Test
public void testSale_SingleBet_OK() throws Exception {
// prepare request ...
// query the VatOperatorBalance first
VatOperatorBalance oldBalance = this.getVatOperatorBalanceDao().findByOperator("OPERATOR-111");
//this.entityManager.detach(oldBalance);
logger.debug("------ oldBalance(" + oldBalance + ").");
// the operation which will modify the oldBalance
Context saleReqCtx = this.getDefaultContext(TransactionType.SELL_TICKET.getRequestType(),
clientTicket);
saleReqCtx.setGameTypeId(GameType.VAT.getType() + "");
Context saleRespCtx = doPost(this.mockRequest(saleReqCtx));
RaffleTicket respTicket = (RaffleTicket) saleRespCtx.getModel();
this.entityManager.flush();
this.entityManager.clear();
// assert vat sale balance
VatOperatorBalance newBalance = this.getVatOperatorBalanceDao().findByOperator("OPERATOR-111");
logger.debug("------ newBalance(" + newBalance + ").");
assertEquals(oldBalance.getSaleBalance().add(respTicket.getTotalAmount()).doubleValue(), newBalance
.getSaleBalance().doubleValue(), 0);
}
This testcase will fail, I don't understand why this will happen. JPA entity manager will update all entities of same entity type? The oldBalance entity and newBlance entity have same entityId, however they are different Java instance, what happened in JPA entity manager? If I detach the oldBalance entity from EntityManager, testcase will pass.
Note: my test is using Spring4.0.5 and JPA2.1
#piet.t since the entityManager would recognize it is the same entity by its primary key (feel free to try it). So all changes made to this entity through the same entityManager will all affect the same java instance
So in a entity manager, a given entity type with given primary key, there should be only one java instance or managed entity(if query from entity manager, no matter what query criteria, by id or not, the same java instance(managed entity) will be returned).
However in my test case, the entity 'oldBalance' will be updated by "the operation which will modify the oldBalance", and then the call of entityManager.clear() will detach all entities managed by this entity manager, that says 'oldBalance' is detached too.
And the 'newBalance' is managed entity then, that is why they have different java instance identifier. If 'oldBalance' is managed, for example by call entityManager.merge(), it will be the same instance of 'newBalance'.

I think most of your confusion does arise from the flush()-call in your code.
Calling flush will always store the changed value to the database - that's the whoe point of calling flush. When using transactions the changed value might still not be visible via other connections due to the databases transaction machanism but your entityManager will only see the changed value.
Without the clear-call your query - even though it is not using find - would still return the same instance that was previously created (VatOperatorBalance#3d6396f5) since the entityManager would recognize it is the same entity by its primary key (feel free to try it). So all changes made to this entity through the same entityManager will all affect the same java instance while modifications through another entity manager will most likely cause an exception because the entity was update from another transaction.
Some queries might cause an implicit flush, since the cached changes might influence the query-result, so all changes have to be written to the database before executing the query to get a correct result-set.
I hope that does help a bit.

GAE with JPA: Update entity

I have problems updating entities in Googles App Engine.
EntityManager em = ... // constructed like in the doc
MyEntity myE = new MyEntity();
myE.setType("1"); // String
em.persist(myE);em.refresh(myE);
myE.setType("2");
em.merge(myE);em.refresh(myE);
I expect a entity with type="2", but there is only one entity with type="1" :-(

That's the correct behaviour, let me explain (I assume that all your code runs in the same persistence context / transaction).
# This line sets the value in the in-memory object without changing the database
myE.setType("2");
# this line doesn't do anything here, as the entity is already managed in the current
# persistence context. The important thing to note is that merge() doesn't save the
# entity to the DB.
em.merge(myE);
# This reloads the entity from the DB discarding all the in-memory changes.
em.refresh(myE);

It's because merge creates a new instance of your entity, copies the state from the supplied entity, and makes the new copy managed. You can find more info on merge vs. persist here and a full discussion about it here

I was facing similar issue too. My issue is solved after my put the Reresh() after Commit().
It would be something like:
em.getTransaction().begin();
//Code to update the entity
em.persist(myE);
em.getTransaction().commit();
em.refresh(myE)
This will ensure the updated entity in JPA Cache gets refreshed with the updated data.
Hope this helps.