I have a query with fetches an entity in pages. Each page contains 100 entities. For those 100 entities I need to fetch some nested entities which are fetched lazy.
What happens is that I have 1 SELECT executed for main entities and then 100 SELECTs for nested ones.
I am looking for a way to initialize those 100 nested ones in bulk, so that in total I have 1 SELECT for main entity and 1 for nested ones.
My structure is as follows, and I would prefer if it is not changed (we had a lot of problems with EAGER fetches)
public class MyMainEntity {
private NestedEntity nested;
#Override
#OneToOne(fetch = FetchType.LAZY, cascade = CascadeType.ALL, orphanRemoval = true)
#JoinColumn(foreignKey = #ForeignKey(name = "exFK_Nest_Enth"))
public NestedEntity getNested() {
...
}
Is there a way to do it with Hibernate.initialize()
You can use Hibernate Criteria query to fetch the MyMainEntity and use criteria.createAlias("nested", "ns", Criteria.Subselect). It will fire two query; one for fetching MyMainEntity and other for Nested with MyMainEntity query acting as subquery. And if you use Criteria.Join then it will fire only one query and fetching both MyMainActivity and Nested
By default, hibernate recent versions it follows JPA 2.0 spec:
Relationships with:
ToMany: Lazy
ToOne: Eager
but if you're using an older version of Hibernate, all relations comes by default with Lazy
In that case when you want to try to retrieve child relations use JOIN FETCH , for example:
Select m from MyMainEntity m JOIN FETCH m.nested;
Related
Entity{
String code;
String parentCode;
...
#ManyToOne
#JoinColumn(name="parentCode",referencedColumnName="code")
Entity parentEntity;
}
My entity class is like this. what i want to do is using findAll() to get an entity list with each entity get its own direct parent. But spring jpa will get parent's parent until the root , i need to avoid this.
Thank you!
Since a default fetch type for a #ManyToOne relation is an FetchType.EAGER I think you have just add a fetch type as LAZY explicitly:
Entity{
String code;
String parentCode;
...
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name="parentCode",referencedColumnName="code")
Entity parentEntity;
}
It's not about Spring JPA but JPA itself. When you add a relationship, following defaults apply unless specified otherwise.
#xxToOne - FetchType.EAGER
#xxToMany- FetchType.LAZY
So, in your example you have a #ManyToOne which has a default EAGER fetch and one join query is appended. If your parent has another #xxToOne it adds one more join and so on. It's good to know the boundaries of your entities and decide which type of FetchType is required.
Even if you add like this:
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name="parentCode",referencedColumnName="code")
Entity parentEntity;
.. if you parent has more relationships you might be getting everything loaded while fetching parent. Thus, all relationships need to be Lazy. It's a design choice based on entities.
But be aware about the ORM's N+1 problem : JPA Hibernate n+1 issue (Lazy & Eager Diff)
Ran into this issue today and I was not able to find any solution anywhere else. Actual entities in my application are much bigger, so here is a simplified version of the problem I am facing. I have an entity which has multiple child entities:
public class ParentEntity {
#OneToMany(fetch = FetchType.LAZY, mappedBy = "parentEntity", cascade = CascadeType.ALL)
#Filter(name = "childEntity_1_filter", condition="state = 'ACTIVE'")
private Set<ChildEntity_1> childEntity_1;
#OneToMany(fetch = FetchType.LAZY, mappedBy = "parentEntity", cascade = CascadeType.ALL)
#Filter(name = "childEntity_2_filter", condition = "entity_name not in ('x', 'y')")
private Set<ChildEntity_2> childEntity_2;
}
Now I am fetching the parent entity from the database using Hibernate Criteria.
So I specify both the joins and the filters from above, and then I specify the FetchMode for all these associations as FetchMode.JOIN.
This is where things don't work as expected. My records were not being returned properly and I debugged the SQL generated by hibernate. I saw something like:
# Note that this is hibernate generated sql
select
........
from
parent_entity pe,
childentity_1 ce1,
childentity_2 ce2
where
pe.id=ce1.id(+) and
ce1.state (+)= 'ACTIVE' and -- Notice the (+) here
pe.id=ce2.id(+) and
ce2.entity_name not in ('x', 'y'); -- Notice that (+) is missing here
As you can see above, the filter on the first child entity has the outer join properly applied to the query, while the "not in" filters do not have this join applied to them.
This is causing my query to not return records in some cases. Eg: If there are entities corresponding to the parent entity in the childentity_2 table, but those entities have entity_name = 'x'. This means that even though there are records after the join (since it is an outer join), all of them will be filtered out with the not in filter.
I tried changing the filter to use not equals condition:
#Filter(name = "childEntity_2_filter", condition = "entity_name != 'x' and ...")
In this case hibernate tried to add the (+), but failed spectacularly. It added the (+) between the ! and =. I tried various other combinations of the query like <>, but could not get anything to do what I wanted.
Just to be clear, I know how to fix the query. My question is how to tell hibernate to fix the query.
How do I fix the above #Filter annotation? My problem is that I have a lot of child entities which all have the equals #Filter and they all work fine, but hibernate is not generating the correct query when using not in #Filter.
I use hibernate 4.3.6, spring data jpa 1.9
I have 2 entity: DocumentOrder and OrderChain. OrderChain have ManyToOne DocumentOrders; DocumentOrder doesnt have link.
public class OrderChain {
#ManyToOne(fetch = FetchType.EAGER)
private DocumentOrder documentOrder;
}
When i do query "SELECT oc FROM OrderChain oc WHERE ..." i have many requests to database(I look in console).
I find similar topics:
preventing N+1 selects in JPA
FetchMode join makes no difference for ManyToMany relations in spring JPA repositories
JPA eager fetch does not join
but i dont know how solve it without adding join fetch in my Query.
edit: I try add #Fetch(FetchMode.JOIN) to documentOrder, but it doesnt make any effect.
Thanks
I understand that the N+1 problem is where one query is executed to fetch N records and N queries to fetch some relational records.
But how can it be avoided in Hibernate?
The problem
The N+1 query issue happens when you forget to fetch an association and then you need to access it.
For instance, let's assume we have the following JPA query:
List<PostComment> comments = entityManager.createQuery("""
select pc
from PostComment pc
where pc.review = :review
""", PostComment.class)
.setParameter("review", review)
.getResultList();
Now, if we iterate the PostComment entities and traverse the post association:
for(PostComment comment : comments) {
LOGGER.info("The post title is '{}'", comment.getPost().getTitle());
}
Hibernate will generate the following SQL statements:
SELECT pc.id AS id1_1_, pc.post_id AS post_id3_1_, pc.review AS review2_1_
FROM post_comment pc
WHERE pc.review = 'Excellent!'
INFO - Loaded 3 comments
SELECT pc.id AS id1_0_0_, pc.title AS title2_0_0_
FROM post pc
WHERE pc.id = 1
INFO - The post title is 'Post nr. 1'
SELECT pc.id AS id1_0_0_, pc.title AS title2_0_0_
FROM post pc
WHERE pc.id = 2
INFO - The post title is 'Post nr. 2'
SELECT pc.id AS id1_0_0_, pc.title AS title2_0_0_
FROM post pc
WHERE pc.id = 3
INFO - The post title is 'Post nr. 3'
That's how the N+1 query issue is generated.
Because the post association is not initialized when fetching the PostComment entities, Hibernate must fetch the Post entity with a secondary query, and for N PostComment entities, N more queries are going to be executed (hence the N+1 query problem).
The fix
The first thing you need to do to tackle this issue is to add [proper SQL logging and monitoring][1]. Without logging, you won't notice the N+1 query issue while developing a certain feature.
Second, to fix it, you can just JOIN FETCH the relationship causing this issue:
List<PostComment> comments = entityManager.createQuery("""
select pc
from PostComment pc
join fetch pc.post p
where pc.review = :review
""", PostComment.class)
.setParameter("review", review)
.getResultList();
If you need to fetch multiple child associations, it's better to fetch one collection in the initial query and the second one with a secondary SQL query.
How to automatically detect the N+1 query issue
This issue is better to be caught by integration tests.
You can use an automatic JUnit assert to validate the expected count of generated SQL statements. The db-util project already provides this functionality, and it's open-source and the dependency is available on Maven Central.
Suppose we have a class Manufacturer with a many-to-one relationship with Contact.
We solve this problem by making sure that the initial query fetches all the data needed to load the objects we need in their appropriately initialized state. One way of doing this is using an HQL fetch join. We use the HQL
"from Manufacturer manufacturer join fetch manufacturer.contact contact"
with the fetch statement. This results in an inner join:
select MANUFACTURER.id from manufacturer and contact ... from
MANUFACTURER inner join CONTACT on MANUFACTURER.CONTACT_ID=CONTACT.id
Using a Criteria query we can get the same result from
Criteria criteria = session.createCriteria(Manufacturer.class);
criteria.setFetchMode("contact", FetchMode.EAGER);
which creates the SQL :
select MANUFACTURER.id from MANUFACTURER left outer join CONTACT on
MANUFACTURER.CONTACT_ID=CONTACT.id where 1=1
in both cases, our query returns a list of Manufacturer objects with the contact initialized. Only one query needs to be run to return all the contact and manufacturer information required
for further information here is a link to the problem and the solution.
Native solution for 1 + N in Hibernate, is called:
20.1.5. Using batch fetching
Using batch fetching, Hibernate can load several uninitialized proxies if one proxy is accessed. Batch fetching is an optimization of the lazy select fetching strategy. There are two ways we can configure batch fetching: on the 1) class level and the 2) collection level...
Check these Q & A:
#BatchSize but many round trip in #ManyToOne case
Avoiding n+1 eager fetching of child collection element association
With annotations we can do it like this:
A class level:
#Entity
#BatchSize(size=25)
#Table(...
public class MyEntity implements java.io.Serializable {...
A collection level:
#OneToMany(fetch = FetchType.LAZY...)
#BatchSize(size=25)
public Set<MyEntity> getMyColl()
Lazy loading and batch fetching together represent optimization, which:
does not require any explicit fetching in our queries
will be applied on any amount of references which are (lazily) touched after the root entity is loaded (while explicit fetching effects only these named in query)
will solve issue 1 + N with collections (because only one collection could be fetched with root query) without need to farther processing To get DISTINCT root values (check: Criteria.DISTINCT_ROOT_ENTITY vs Projections.distinct)
You can even get it working without having to add the #BatchSize annotation everywhere, just set the property hibernate.default_batch_fetch_size to the desired value to enable batch fetching globally. See the Hibernate docs for details.
While you are at it, you will probably also want to change the BatchFetchStyle, because the default (LEGACY) is most likely not what you want. So a complete configuration for globally enabling batch fetching would look like this:
hibernate.batch_fetch_style=PADDED
hibernate.default_batch_fetch_size=25
Also, I'm suprised that one of the proposed solutions involves join-fetching. Join-fetching is rarely desirable because it causes more data to be transferred with every result row, even if the dependent entity has already been loaded into the L1 or L2 cache. Thus I would recommend to disable it completey by setting
hibernate.max_fetch_depth=0
This is a frequently asked question so I created the article Eliminate Spring Hibernate N+1 Queries to detail the solutions
To help you detect all the N+1 queries in your application and avoid adding more queries, I created the library spring-hibernate-query-utils that auto-detects the Hibernate N+1 queries.
Here is some code to explain how to add it to your application:
Add the library to your dependencies
<dependency>
<groupId>com.yannbriancon</groupId>
<artifactId>spring-hibernate-query-utils</artifactId>
<version>1.0.0</version>
</dependency>
Configure it in your application properties to return exceptions, default is error logs
hibernate.query.interceptor.error-level=EXCEPTION
If you are using Spring Data JPA to implement your repositories, you can specify lazy fetching in the JPA associations:
#Entity
#Table(name = "film", schema = "public")
public class Film implements Serializable {
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "language_id", nullable = false)
private Language language;
#OneToMany(cascade = CascadeType.ALL, fetch = FetchType.LAZY, mappedBy = "film")
private Set<FilmActor> filmActors;
...
}
#Entity
#Table(name = "film_actor", schema = "public")
public class FilmActor implements Serializable {
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "film_id", nullable = false, insertable = false, updatable = false)
private Film film;
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "actor_id", nullable = false, insertable = false, updatable = false)
private Actor actor;
...
}
#Entity
#Table(name = "actor", schema = "public")
public class Actor implements Serializable {
#OneToMany(cascade = CascadeType.ALL, fetch = FetchType.LAZY, mappedBy = "actor")
private Set<FilmActor> filmActors;
...
}
And add #EntityGraph to your Spring Data JPA-based repository:
#Repository
public interface FilmDao extends JpaRepository<Film, Integer> {
#EntityGraph(
type = EntityGraphType.FETCH,
attributePaths = {
"language",
"filmActors",
"filmActors.actor"
}
)
Page<Film> findAll(Pageable pageable);
...
}
My blog post at https://tech.asimio.net/2020/11/06/Preventing-N-plus-1-select-problem-using-Spring-Data-JPA-EntityGraph.html helps you preventing the N+1 select problem using Spring Data JPA and #EntityGraph.
Here are some snippet codes that would help you to fix the N+1 Problem.
One to Many Relationship with Manager and Client Entity.
Client JPA Repository -
public interface ClientDetailsRepository extends JpaRepository<ClientEntity, Long> {
#Query("FROM clientMaster c join fetch c.manager m where m.managerId= :managerId")
List<ClientEntity> findClientByManagerId(String managerId);
}
Manager Entity -
#Entity(name = "portfolioManager")
#Table(name = "portfolio_manager")
public class ManagerEntity implements Serializable {
// some fields
#OneToMany(fetch = FetchType.LAZY, mappedBy = "manager")
protected List<ClientEntity> clients = new ArrayList<>();
// Getter & Setter
}
Client Entity -
#Entity(name = "clientMaster")
#Table(name = "clientMaster")
public class ClientEntity implements Serializable {
// some fields
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "manager_id", insertable = false, updatable = false)
protected ManagerEntity manager;
// Getter & Setter
}
And finally, Generate output -
Hibernate: select cliententi0_.client_id as client_id1_0_0_, cliententi0_.manager_id as manager_id2_0_0_, managerent1_.manager_id as manager_id1_2_1_, cliententi0_.created_by as created_by7_0_0_, cliententi0_.created_date as created_date3_0_0_, cliententi0_.client_name as client_name4_0_0_, cliententi0_.sector_name as sector_name5_0_0_, cliententi0_.updated_by as updated_by8_0_0_, cliententi0_.updated_date as updated_date6_0_0_, managerent1_.manager_name as manager_name2_2_1_ from client_master cliententi0_, portfolio_manager managerent1_ where cliententi0_.manager_id=managerent1_.manager_id and managerent1_.manager_id=?```
I have these classes:
#Entity
public class Invoice implements Serializable {
#Id
#Basic(optional = false)
private Integer number;
private BigDecimal value;
//Getters and setters
}
#Entity
public class InvoiceItem implements Serializable {
#EmbeddedId
protected InvoiceItemPK invoiceItemPk;
#ManyToOne
#JoinColumn(name = "invoice_number", insertable = false, updatable = false)
private Invoice invoice;
//Getters and setters
}
When i run this query:
session.createQuery("select i from InvoiceItem i").list();
It executes one query to select the records from InvoiceItem, and if I have 10000 invoice items, it generates 10000 additional queries to select the Invoice from each InvoiceItem.
I think it would be a lot better if all the records could be fetched in a single sql. Actually, I find it weird why it is not the default behavior.
So, how can I do it?
The problem here is not related to Hibernate but to JPA.
Prior to JPA 1.0, Hibernate 3 used lazy loading for all associations.
However, the JPA 1.0 specification uses FetchType.LAZY only for collection associations:
#OneToMany,
#ManyToMany
#ElementCollection)
The #ManyToOne and #OneToOne associations use FetchType.EAGER by default, and that's very bad from a performance perspective.
The behavior described here is called the [N+1 query issue][5], and it happens because Hibernate needs to make sure that the #ManyToOne association is initialized prior to returning the result to the user.
Now, if you are using direct fetching via entityManager.find, Hibernate can use a LEFT JOIN to initialize the FetchTYpe.EAGER associations.
However, when executing a query that does not explicitly use a JOIN FETCH clause, Hibernate will not use a JOIN to fetch the FetchTYpe.EAGER associations, as it cannot alter the query that you already specified how to be constructed. So, it can only use secondary queries.
The fix is simple. Just use FetchType.LAZY for all associations:
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "invoice_number", insertable = false, updatable = false)
private Invoice invoice;
More, you should use the Hypersistence Utils to assert the number of statements executed by JPA and Hibernate.
Try with
session.createQuery("select i from InvoiceItem i join fetch i.invoice inv").list();
It should get all the data in a single SQL query by using joins.
Yes there is setting you need: #BatchSize(size=25). Check it here:
20.1.5. Using batch fetching
small cite:
Using batch fetching, Hibernate can load several uninitialized proxies if one proxy is accessed. Batch fetching is an optimization of the lazy select fetching strategy. There are two ways you can configure batch fetching: on the class level and the collection level.
Batch fetching for classes/entities is easier to understand. Consider the following example: at runtime you have 25 Cat instances loaded in a Session, and each Cat has a reference to its owner, a Person. The Person class is mapped with a proxy, lazy="true". If you now iterate through all cats and call getOwner() on each, Hibernate will, by default, execute 25 SELECT statements to retrieve the proxied owners. You can tune this behavior by specifying a batch-size in the mapping of Person:
<class name="Person" batch-size="10">...</class>
With this batch-size specified, Hibernate will now execute queries on demand when need to access the uninitialized proxy, as above, but the difference is that instead of querying the exactly proxy entity that being accessed, it will query more Person's owner at once, so, when accessing other person's owner, it may already been initialized by this batch fetch with only a few ( much less than 25) queries will be executed.
So, we can use that annotation on both:
collections/sets
classes/Entities
Check it also here:
#BatchSize but many round trip in #ManyToOne case
In this Method there are Multiple SQLs fired. This first one is fired for retrieving all the records in the Parent table. The remaining are fired for retrieving records for each Parent Record. The first query retrieves M records from database, in this case M Parent records. For each Parent a new query retrieves Child.