We Keep Coding

JPA Lazy loading is not working in Spring boot

I googled a lot and It is really bizarre that Spring Boot (latest version) may not have the lazy loading is not working. Below are pieces of my code:
My resource:
public ResponseEntity<Page<AirWaybill>> searchAirWaybill(CriteraDto criteriaDto, #PageableDefault(size = 10) Pageable pageable{
airWaybillService.searchAirWaybill(criteriaDto, pageable);
return ResponseEntity.ok().body(result);
}
My service:
#Service
#Transactional
public class AirWaybillService {
//Methods
public Page<AirWaybill> searchAirWaybill(AirWaybillCriteriaDto searchCriteria, Pageable pageable){
//Construct the specification
return airWaybillRepository.findAll(spec, pageable);
}
}
My Entity:
#Entity
#Table(name = "TRACKING_AIR_WAYBILL")
#JsonIdentityInfo(generator=ObjectIdGenerators.IntSequenceGenerator.class, property="#airWaybillId") //to fix Infinite recursion with LoadedAirWaybill class
public class AirWaybill{
//Some attributes
#NotNull
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "FK_TRACKING_CORPORATE_BRANCH_ID")
private CorporateBranch corporateBranch;
}
And when debugging, I still getting all lazy loaded attributed loaded. See image below.
One of my questions is could Jackson be involved in such behaviour?
Is there any way that I may have missed to activate the lazy loading?
EDIT
Another question, could the debugger be involved in ruining the lazy loading?
EDIT 2:
For specification build, I have :
public static Specification<AirWaybill> isBranchAirWayBill(long id){
return new Specification<AirWaybill>() {
#Override
public Predicate toPredicate(Root<AirWaybill> root, CriteriaQuery<?> query, CriteriaBuilder cb) {
return cb.equal(root.join("corporateBranch",JoinType.LEFT).get("id"),id);
}
};
}

Hibernate Session exists within method with #Transactional.
Passing entity outside Service class is a bad practise because session is being closed after leaving your search method. On the other hand your entity contains lazy initialised collections, which cannot be pulled once session is closed.
The good practise is to map entity onto transport object and return those transport objects from service (not raw entities).

SpringBoot by default has enabled:
spring.jpa.open-in-view = true
That means transaction is always open. Try to disable it.
more information here

Most likely you are debugging while still being inside the service, thus while the transaction is still active and lazy loading can be triggered (any method called on a lazy element triggered the fetch from the database).
The problem is that lazy loading cannot occur while being outside of the transaction. And Jackson is parsing your entity definitely outside the boundaries of one.
You either should fetch all the required dependencies when building your specification or try with the #Transactional on the resource level (but try that as of last resort).
Just so that you know, LAZY fetching strategy is only a hint.. not a mandatory action. Eager is mandatory:
The LAZY strategy is a hint to the persistence provider runtime that
data should be fetched lazily when it is first accessed. The
implementation is permitted to eagerly fetch data for which the LAZY
strategy hint has been specified.

When using a debugger, you are trying to access the value of your variables. So, at the moment you click that little arrow on your screen, the value of the variable in question is (lazily) loaded.

I suppose you are using Hibernate as JPA.
From specification:
The EAGER strategy is a requirement on the persistence provider runtime that data must be eagerly fetched. The LAZY strategy is a hint to the persistence provider runtime that data should be fetched lazily when it is first accessed. The implementation is permitted to eagerly fetch data for which the LAZY strategy hint has been specified. https://docs.jboss.org/hibernate/jpa/2.2/api/javax/persistence/FetchType.html
Hibernate ignores fetch type specially in OneToOne and ManyToOne relationships from non owning side.
There are few options how to force Hibernate use fetch type LAZY if you really need it.
The simplest one is to fake one-to-many relationship. This will work because lazy loading of collection is much easier then lazy loading of single nullable property but generally this solution is very inconvenient if you use complex JPQL/HQL queries.
The other one is to use build time bytecode instrumentation. For more details please read Hibernate documentation: 19.1.7. Using lazy property fetching. Remember that in this case you have to add #LazyToOne(LazyToOneOption.NO_PROXY) annotation to one-to-one relationship to make it lazy. Setting fetch to LAZY is not enough.
The last solution is to use runtime bytecode instrumentation but it will work only for those who use Hibernate as JPA provider in full-blown JEE environment (in such case setting "hibernate.ejb.use_class_enhancer" to true should do the trick: Entity Manager Configuration) or use Hibernate with Spring configured to do runtime weaving (this might be hard to achieve on some older application servers). In this case #LazyToOne(LazyToOneOption.NO_PROXY) annotation is also required.
For more informations look at this:
http://justonjava.blogspot.com/2010/09/lazy-one-to-one-and-one-to-many.html

Just a guess: you are forcing a fetch while building your specification.
I expect something like
static Specification<AirWaybill> buildSpec() {
return (root, query, criteriaBuilder) -> {
Join<AirWaybill, CorporateBranch> br = (Join) root.fetch("corporateBranch");
return criteriaBuilder.equal(br.get("addressType"), 1);
};
}
If this is the case, try changing root.fetch to root.join

The retrieved data already lazy but you are using debug mode its return value when click to watch a data from a debugger.

You can solve this problem with wit 2 steps with jackson-datatype-hibernate:
kotlin example
Add In build.gradle.kts:
implementation("com.fasterxml.jackson.datatype:jackson-datatype-hibernate5:$jacksonHibernate")
Create #Bean
#Bean
fun hibernate5Module(): Module = Hibernate5Module()
Notice that Module is com.fasterxml.jackson.databind.Module, not java.util.Module

Another consideration is while using Lombok, #Data/#Getter annotation causes to load lazy items without need. So be careful when using Lombok.
This was my case.

I think I might have a solution. You can give this a try. This worked for me after 4 hours of hit and trial -
User Entity :
class User {
#Id
String id;
#JsonManagedReference
#OneToMany(mappedBy = "user", fetch = FetchType.LAZY)
private List<Address> addressDetailVOList = new ArrayList<Address>();
}
Address entity :
class Address {
#JsonBackReference
#ManyToOne(fetch = FetchType.EAGER)
#JoinColumn(name = "userId")
private User user;
}
Your parent class will use #JsonManagedReference, and child class will use #JsonBackReference. With this, you can avoid the infinite loop of entity objects as response and stack overflow error.

I also faced the same issue with Spring data JPA. I added the below annotation & able to get the customer records for a given ORDER ID
Customer to Order : one to Many
Order to customer is lazy load.
Order.java
#ManyToOne(cascade = CascadeType.ALL,targetEntity = CustomerEntity.class,fetch = FetchType.LAZY)
#Fetch(FetchMode. JOIN)
#JoinColumn(name = "CUSTOMER_ID",referencedColumnName = "CUSTOMER_ID",insertable = false,updatable = false)
#LazyToOne(LazyToOneOption.PROXY)
Private CustomerEntity customer
Customer.java
#Entity
#TabLe(name = "CUSTOMER" ,
uniqueConstraints = #UniqueConstraint(columnNames= {"mobile"}))
public class CustomerEntity {
#GeneratedVaLue(strategy = GenerationType.IDENTITY)
#CoLumn(name = "customer_id" )
private Integer customerld;
private String name;
private String address;
private String city;
private String state;
private Integer zipCode;
private Integer mobileNumber;
#OneToMany(mappedBy = " customer" )
#Fetch(FetchMode.JOIN)
#LazyToOne(LazyToOneOption.PROXY)
private List<OrderEntity> orders;
}

N + 1 when ID is string (JpaRepository)

I have an entity with string id:
#Table
#Entity
public class Stock {
#Id
#Column(nullable = false, length = 64)
private String index;
#Column(nullable = false)
private Integer price;
}
And JpaRepository for it:
public interface StockRepository extends JpaRepository<Stock, String> {
}
When I call stockRepository::findAll, I have N + 1 problem:
logs are simplified
select s.index, s.price from stock s
select s.index, s.price from stock s where s.index = ?
The last line from the quote calls about 5K times (the size of the table). Also, when I update prices, I do next:
stockRepository.save(listOfStocksWithUpdatedPrices);
In logs I have N inserts.
I haven't seen similar behavior when id was numeric.
P.S. set id's type to numeric is not the best solution in my case.
UPDATE1:
I forgot to mention that there is also Trade class that has many-to-many relation with Stock:
#Table
#Entity
public class Trade {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private Integer id;
#Column
#Enumerated(EnumType.STRING)
private TradeType type;
#Column
#Enumerated(EnumType.STRING)
private TradeState state;
#MapKey(name = "index")
#ManyToMany(fetch = FetchType.EAGER)
#JoinTable(name = "trade_stock",
joinColumns = { #JoinColumn(name = "id", referencedColumnName = "id") },
inverseJoinColumns = { #JoinColumn(name = "stock_index", referencedColumnName = "index") })
private Map<String, Stock> stocks = new HashMap<>();
}
UPDATE2:
I added many-to-many relation for the Stock side:
#ManyToMany(cascade = CascadeType.ALL, mappedBy = "stocks") //lazy by default
Set<Trade> trades = new HashSet<>();
But now it left joins trades (but they're lazy), and all trade's collections (they are lazy too). However, generated Stock::toString method throws LazyInitializationException exception.

Related answer: JPA eager fetch does not join
You basically need to set #Fetch(FetchMode.JOIN), because fetch = FetchType.EAGER just specifies that the relationship will be loaded, not how.
Also what might help with your problem is
#BatchSize annotation, which specifies how many lazy collections will be loaded, when the first one is requested. For example, if you have 100 trades in memory (with stocks not initializes) #BatchSize(size=50) will make sure that only 2 queries will be used. Effectively changing n+1 to (n+1)/50.
https://docs.jboss.org/hibernate/orm/4.3/javadocs/org/hibernate/annotations/BatchSize.html
Regarding inserts, you may want to set
hibernate.jdbc.batch_size property and set order_inserts and order_updates to true as well.
https://vladmihalcea.com/how-to-batch-insert-and-update-statements-with-hibernate/

However, generated Stock::toString method throws
LazyInitializationException exception.
Okay, from this I am assuming you have generated toString() (and most likely equals() and hashcode() methods) using either Lombok or an IDE generator based on all fields of your class.
Do not override equals() hashcode() and toString() in this way in a JPA environment as it has the potential to (a) trigger the exception you have seen if toString() accesses a lazily loaded collection outside of a transaction and (b) trigger the loading of extremely large volumes of data when used within a transaction. Write a sensible to String that does not involve associations and implement equals() and hashcode() using (a) some business key if one is available, (b) the ID (being aware if possible issues with this approach or (c) do not override them at all.
So firstly, remove these generated methods and see if that improves things a bit.

With regards to the inserts, I do notice one thing that is often overlooked in JPA. I don't know what Database you use, but you have to be careful with
#GeneratedValue(strategy = GenerationType.AUTO)
For MySQL I think all JPA implementations map to an auto_incremented field, and once you know how JPA works, this has two implication.
Every insert will consist of two queries. First the insert and then a select query (LAST_INSERT_ID for MySQL) to get the generated primary key.
It also prevents any batch query optimization, because each query needs to be done in it's own insert.
If you insert a large number of objects, and you want good performance, I would recommend using table generated sequences, where you let JPA pre-allocate IDs in large chunks, this also allows the SQL driver do batch Insert into (...) VALUES(...) optimizations.
Another recommendation (not everyone agrees with me on this one). Personally I never use ManyToMany, I always decompose it into OneToMany and ManyToOne with the join table as a real entity. I like the added control it gives over cascading and fetch, and you avoid some of the ManyToMany traps that exist with bi-directional relations.

Spring Data JPA - concurrent Bulk inserts/updates

at the moment I develop a Spring Boot application which mainly pulls product review data from a message queue (~5 concurrent consumer) and stores them to a MySQL DB. Each review can be uniquely identified by its reviewIdentifier (String), which is the primary key and can belong to one or more product (e.g. products with different colors). Here is an excerpt of the data-model:
public class ProductPlacement implements Serializable{
private static final long serialVersionUID = 1L;
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
#Column(name = "product_placement_id")
private long id;
#ManyToMany(fetch = FetchType.LAZY, cascade = CascadeType.ALL, mappedBy="productPlacements")
private Set<CustomerReview> customerReviews;
}
public class CustomerReview implements Serializable{
private static final long serialVersionUID = 1L;
#Id
#Column(name = "customer_review_id")
private String reviewIdentifier;
#ManyToMany(fetch = FetchType.LAZY, cascade = CascadeType.ALL)
#JoinTable(
name = "tb_miner_review_to_product",
joinColumns = #JoinColumn(name = "customer_review_id"),
inverseJoinColumns = #JoinColumn(name = "product_placement_id")
)
private Set<ProductPlacement> productPlacements;
}
One message from the queue contains 1 - 15 reviews and a productPlacementId. Now I want an efficient method to persist the reviews for the product. There are basically two cases which need to be considered for each incomming review:
The review is not in the database -> insert review with reference to the product that is contained in the message
The review is already in the database -> just add the product reference to the Set productPlacements of the existing review.
Currently my method for persisting the reviews is not optimal. It looks as follows (uses Spring Data JpaRespoitories):
#Override
#Transactional
public void saveAllReviews(List<CustomerReview> customerReviews, long productPlacementId) {
ProductPlacement placement = productPlacementRepository.findOne(productPlacementId);
for(CustomerReview review: customerReviews){
CustomerReview cr = customerReviewRepository.findOne(review.getReviewIdentifier());
if (cr!=null){
cr.getProductPlacements().add(placement);
customerReviewRepository.saveAndFlush(cr);
}
else{
Set<ProductPlacement> productPlacements = new HashSet<>();
productPlacements.add(placement);
review.setProductPlacements(productPlacements);
cr = review;
customerReviewRepository.saveAndFlush(cr);
}
}
}
Questions:
I sometimes get constraintViolationExceptions because of violating the unique constraint on the "reviewIndentifier". This is obviously because I (concurrently) look if the review is already present and than insert or update it. How can I avoid that?
Is it better to use save() or saveAndFlush() in my case. I get ~50-80 reviews per secound. Will hibernate flush automatically if I just use save() or will it result in greatly increased memory usage?
Update to question 1: Would a simple #Lock on my Review-Repository prefent the unique-constraint exception?
#Lock(LockModeType.PESSIMISTIC_WRITE)
CustomerReview findByReviewIdentifier(String reviewIdentifier);
What happens when the findByReviewIdentifier returns null? Can hibernate lock the reviewIdentifier for a potential insert even if the method returns null?
Thank you!

From a performance point of view, I will consider evaluating the solution with the following changes.
Changing from bidirectional ManyToMany to bidirectional OneToMany
I had a same question on which one is more efficient from DML statements that gets executed. Quoting from Typical ManyToMany mapping versus two OneToMany.
The option one might be simpler from a configuration perspective, but it yields less efficient DML statements.
Use the second option because whenever the associations are controlled by #ManyToOne associations, the DML statements are always the most efficient ones.
Enable the batching of DML statements
Enabling the batching support would result in less number of round trips to the database to insert/update the same number of records.
Quoting from batch INSERT and UPDATE statements
hibernate.jdbc.batch_size = 50
hibernate.order_inserts = true
hibernate.order_updates = true
hibernate.jdbc.batch_versioned_data = true
Remove the number of saveAndFlush calls
The current code gets the ProductPlacement and for each review it does a saveAndFlush, which results in no batching of DML statements.
Instead I would consider loading the ProductPlacement entity and adding the List<CustomerReview> customerReviews to the Set<CustomerReview> customerReviews field of ProductPlacement entity and finally call the merge method once at the end, with these two changes:
Making ProductPlacement entity owner of the association i.e., by moving mappedBy attribute onto Set<ProductPlacement> productPlacements field of CustomerReview entity.
Making CustomerReview entity implement equals and hashCode method by using reviewIdentifier field in these method. I believe reviewIdentifier is unique and user assigned.
Finally, as you do performance tuning with these changes, baseline your performance with the current code. Then make the changes and compare if the changes are really resulting in the any significant performance improvement for your solution.

How does JPA cache refreshes entities externally modified?

I faced an issue earlier with JPA.
I have two apps : the main one, using Java/JPA (EclipseLink), and a second one, using PHP. The two apps have access to the same database.
Now, I'm accessing an "Expedition" object through Java, then calling the PHP app through a web-service (which is supposed to modify an attribute of this object in the shared database table "Expedition"), then accessing this attribute through the Java app.
Problem is, the object seems not to be modified in the Java app, even if it is modified in the database. I'm thinking about a cache problem.
The original code (simplified) :
System.out.println(expedition.getInfosexpedition()); // null
// Calling the web-service (modification of the "expedition" object in the database)
this.ec.eXtractor(expedition);
System.out.println(expedition.getInfosexpedition()); // Still null, should not be
Definitions of the "Expedition" and "Infosexpedition" classes :
Expedition :
#Entity
#Table(name = "expedition")
#XmlRootElement
public class Expedition implements Serializable {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Basic(optional = false)
#Column(name = "idExpedition")
private Integer idExpedition;
#OneToOne(cascade = CascadeType.ALL, mappedBy = "idExpedition")
#XmlTransient
private Infosexpedition infosexpedition;
Infosexpedition :
#Entity
#Table(name = "infosexpedition")
#XmlRootElement
public class Infosexpedition implements Serializable {
private static final long serialVersionUID = 1L;
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Basic(optional = false)
#Column(name = "idInfoExpedition")
private Integer idInfoExpedition;
#JoinColumn(name = "idExpedition", referencedColumnName = "idExpedition")
#OneToOne(optional = false)
#XmlTransient
private Expedition idExpedition;
I've been able to make the original code work by doing this :
System.out.println(expedition.getInfosexpedition()); // null
// Calling the web-service (modification of the "expedition" object in the database)
this.ec.eXtractor(expedition);
try
{
// Getting explicitly the "infosExpedition" item through a simple named request
Infosexpedition infos = this.ec.getFacade().getEm().createNamedQuery("Infosexpedition.findByIdExpedition", Infosexpedition.class)
.setParameter("idExpedition", expedition)
.setHint("eclipselink.refresh", "true")
.setHint("eclipselink.cache-usage", "DoNotCheckCache")
.setHint("eclipselink.read-only", "true") // This line did the trick
.getSingleResult();
expedition.setInfosexpedition(infos);
}
catch (NoResultException nre) {}
System.out.println(expedition.getInfosexpedition()); // Not null anymore, OK
I'm trying to understand what happens here, and why did I had to specify a "read-only" hint to make this work... Before that, I tried almost everything, from evictAll() calls to detach()/merge() calls, and nothing worked.
Can someone help me to understand how the different levels of cache worked here ? And why is my newly created line "read-only" ?
Thanks a lot.

The settings you are using are attempting to bypass the cache. ("eclipselink.read-only", "true") causes it to bypass the first level cache, while the ("eclipselink.cache-usage", "DoNotCheckCache") makes the query go to the database instead of pulling data from the second level cache. Finally ("eclipselink.refresh", "true") refreshes the data in the shared cache rather then return the prebuilt object. Your facade must be using the same EntityManager for both requests even though you have made changes to the objects between the requests. As mentioned in the comments, an EntityManager is meant to be used as a transaction would, so that you are isolated from changes made during your transactions. If this doesn't work for you, you should clear or release the entityManager after the first call, so that the calls after the web-service modifications can be picked up.
If applications outside this one are going to be making data changes frequently, you might want to look at disabling the shared cache as described here:
https://wiki.eclipse.org/EclipseLink/FAQ/How_to_disable_the_shared_cache%3F
And also implement optimistic locking to prevent either application from overwriting the other with stale data as described here:
https://wiki.eclipse.org/EclipseLink/UserGuide/JPA/Basic_JPA_Development/Mapping/Locking/Optimistic_Locking

What you call cache is the 1st level cache, id est the in memory projection of the database state at a time t.
This "cache" has the same lifecycle that the entity manager itself and generally won't be refreshed until you explicitely clear it (using myEntityManager.clear()) (you shouldn't) or force it to refreh a specific entity instance (using myEntityManager.refresh(myEntityInstance), this is the way you should go)
See Struggling to understand EntityManager proper use and Jpa entity lifecycle for a more detailed explanation

Does JPA support mapping to sql views?

I'm currently using Eclipselink, but I know now days most JPA implementations have been pretty standardized. Is there a native way to map a JPA entity to a view? I am not looking to insert/update, but the question is really how to handle the #Id annotation. Every entity in the JPA world must have an ID field, but many of the views I have created do not conform to this. Is there native support for this in the JPA or do I need to use hacks to get it to work? I've searched a lot and found very little information about doing this.

While using the #Id annotation with fields of directly supported types is not the only way to specify an entity's identity (see #IdClass with multiple #Id annotations or #EmbeddedId with #Embedded), the JPA specification requires a primary key for each entity.
That said, you don't need entities to use JPA with database views. As mapping to a view is no different from mapping to a table from an SQL perspective, you could still use native queries (createNativeQuery on EntityManager) to retrieve scalar values instead.

I've been looking into this myself, and I've found a hack that I'm not 100% certain works but that looks promising.
In my case, I have a FK column in the view that can effectively function as a PK -- any given instance of that foreign object can only occur once in the view. I defined two objects off of that one field: one is designated the ID and represents the raw value of the field, and the other is designated read-only and represents the object being referred to.
#Id
#Column(name = "foreignid", unique = true, nullable = false)
public Long getForeignId() {
...
#OneToOne
#JoinColumn(name = "foreignid", insertable=false, updatable=false)
public ForeignObject getForeignObject() {
...
Like I said, I'm not 100% sure on this one (and I'll just delete this answer if it turns out not to work), but it got my code past a particular crash point.
Dunno if it applies to your specific situation, though. And there's an excellent chance that after 11 months, you no longer care. :-) What the hell, that "Necromancer" badge doesn't just earn itself....

In my view I have a "unique" id, so I mapped it as the Entity id.
It works very well:
#Entity
#Table(name="table")
#NamedQuery(name="Table.findAll", query="SELECT n FROM Table n")
public class Table implements Serializable {
private static final long serialVersionUID = 1L;
#Id
#Column(name="column_a")
private int columnA;

JPA - 2.5.4
CREATE MATERIALIZED VIEW IF NOT EXISTS needed_article as select product_id, count(product_id) as count from product_article group by product_id;
CREATE MATERIALIZED VIEW IF NOT EXISTS available_article as select product_id, count(product_id) as count from article a inner join product_article p
on a.id = p.article_id and a.stock >= p.amount_of group by product_id;
CREATE UNIQUE INDEX productId_available_article ON available_article (product_Id);
CREATE UNIQUE INDEX productId_needed_article ON needed_article (product_Id);
Entity.java
#Entity
#Immutable // hibernate import
#Getter
#Setter
public class NeededArticle {
#Id
Integer productId;
Integer count;
}
Repository.java
#Repository
public interface AvailableProductRepository extends CrudRepository<AvailableArticle, Integer> {
#Query("select available.productId from AvailableArticle available, NeededArticle needed where available.productId = needed.productId and available.count = needed.count")
List<Integer> availableProduct();