We Keep Coding

Why are lazy fields on related entities loaded

In my REST API project (Java 8, Spring Boot 2.3.1) I have a problem with some queries triggering massive query chains by loading lazy relations, even though the related objects are never accessed.
I have a UserEntity and a polymorphic CompanyEntity that are related with a ManyToMany relationship. I have an endpoint that returns all users and I include the IDs of the related companies in the JSON. I excpect a query to the user table and a query to the company table, however all related entities of one sub-entity of CompanyEntity are always loaded for each of those sub-entities resulting in large query chains.
Here are snippets of my classes:
User entity
#Entity(name = "USERS")
public class UserEntity {
#Id
#GeneratedValue
private UUID id;
#EqualsAndHashCode.Exclude
#Fetch(FetchMode.SUBSELECT)
#ManyToMany(fetch = FetchType.LAZY)
#JoinTable(
name = "users_company",
joinColumns = #JoinColumn(name = "USER_ID"),
inverseJoinColumns = #JoinColumn(name = "COMPANY_ID")
)
private Set<CompanyEntity> companies = new HashSet<>();
public List<UUID> getCompanyIds() {
return companies.stream()
.map(CompanyEntity::getId)
.collect(Collectors.toList());
}
}
Polymorphic company entity
#Entity(name = "COMPANY")
#Inheritance(strategy = InheritanceType.JOINED)
public abstract class CompanyEntity {
#Id
#GeneratedValue
private UUID id;
#Fetch(FetchMode.SUBSELECT)
#ManyToMany(mappedBy = "companies", fetch = FetchType.LAZY)
private Set<UserEntity> users = new HashSet<>();
}
Concrete company subclass that triggers the problem
#Entity(name = "CUSTOMER")
public class CustomerEntity extends CompanyEntity {
#NotNull
#OneToOne(cascade = {CascadeType.PERSIST, CascadeType.MERGE}, fetch = FetchType.LAZY)
private ContactPersonEntity contactPerson;
#Fetch(FetchMode.SUBSELECT)
#OneToMany(cascade = {CascadeType.PERSIST, CascadeType.MERGE}, fetch = FetchType.LAZY, mappedBy = "customer")
private Set<TransactionEntity> transactions = new HashSet<>();
public Set<UUID> getTransactionIds() {
return this.transactions.stream()
.map(TransactionEntity::getId)
.collect(Collectors.toSet());
}
}
In the REST controller I return the following mapping:
#GetMapping(value = "", produces = MediaType.APPLICATION_JSON_VALUE)
public List<UserReadModel> getUsers() {
return userRepository.findAll().stream()
.map(userEntity -> new UserReadModel(userEntity))
.collect(Collectors.toList());
}
Where the UserReadModel is a DTO:
#Data
public class UserReadModel {
private UUID id;
private List<UUID> companyIds;
}
Logging the database queries results in the following output:
// Expected
Hibernate: select userentity0_.id as id1_47_, ... from users userentity0_
Hibernate: select companies0_.user_id ... case when companyent1_1_.id is not null then 1 when companyent1_2_.id is not null then 2 when companyent1_.id is not null then 0 end as clazz_0_ from users_company companies0_ inner join company companyent1_ on companies0_.company_id=companyent1_.id left outer join customer companyent1_1_ on companyent1_.id=companyent1_1_.id left outer join external_editor companyent1_2_ on companyent1_.id=companyent1_2_.id where companies0_.user_id in (select userentity0_.id from users userentity0_)
// Unexpected as they are marked lazy and never accessed
Hibernate: select contactper0_.id ... from contact_person contactper0_ where contactper0_.id=?
Hibernate: select transactio0_.customer_id ... from transactions transactio0_ where transactio0_.customer_id=?
Hibernate: select contactper0_.id ... from contact_person contactper0_ where contactper0_.id=?
Hibernate: select transactio0_.customer_id ... from transactions transactio0_ where transactio0_.customer_id=?
...
I've read through loads of articles on entity mapping and lazy loading but I can't seem to find a reason why this behavior persists. Did anyone have this problem before?

You are accessing the collection, so Hibernate has to load the collection. Since you only need the ids and already have a DTO, I think this is a perfect use case for Blaze-Persistence Entity Views.
I created the library to allow easy mapping between JPA models and custom interface or abstract class defined models, something like Spring Data Projections on steroids. The idea is that you define your target structure(domain model) the way you like and map attributes(getters) via JPQL expressions to the entity model.
A DTO model for your use case could look like the following with Blaze-Persistence Entity-Views:
#EntityView(UserEntity.class)
public interface UserReadModel {
#IdMapping
UUID getId();
#Mapping("companies.id")
Set<UUID> getCompanyIds();
}
Querying is a matter of applying the entity view to a query, the simplest being just a query by id.
UserReadModel a = entityViewManager.find(entityManager, UserReadModel.class, id);
The Spring Data integration allows you to use it almost like Spring Data Projections: https://persistence.blazebit.com/documentation/entity-view/manual/en_US/index.html#spring-data-features
Page<UserReadModel> findAll(Pageable pageable);
The best part is, it will only fetch the state that is actually necessary! In your case, a query like the following will be generated:
select u.id, uc.company_id
from users u
left join users_company uc on uc.user_id = u.id
left join company c on c.id = uc.company_id
Depending on the Hibernate version, the join for the company might even be omitted.

I eventually figured out the solution and want to post it here, in case anyone stumbles upon this question. This was purely a mistake on my side and is not reproducible from the examples I posted.
I used lombok annotations to generate equals and hashcode methods on the customer entity (and all other entities for that matter) and forgot to annotate the contactPerson and transactions fields with #EqualsAndHashcode.Exclude. As the equals method was called somewhere along the execution, it triggered the lazy loading of those fields. Implementing equals and hashcode manually and using the guidelines from this article for that solved the problem.

Return ResultSet with intact joins in Hibernate / JPA

I have 2 entities: EntityA and EntityB. They are related with a One To Many relation.
public class EntityA {
#Identifier
#GeneratedValue(strategy = GenerationType.AUTO)
#Column(name="ID", updatable = false, nullable = false)
private long id;
#OneToMany(cascade = CascadeType.ALL, orphanRemoval = true)
#JoinColumn(name="ENTITY_A_ID", referencedColumnName="ID", nullable=true)
private List<EntityB> entityBs;
/* GETTERS SETTERS ... */
}
public class EntityB {
#Identifier
#GeneratedValue(strategy = GenerationType.AUTO)
#Column(name="ID", updatable = false, nullable = false)
private long id;
#Column(name="SOME_PROPERTY")
private String someProperty;
#ManyToOne
#JoinColumn(name="ENTITY_A_ID")
private EntityA entityA;
/* GETTERS SETTERS ... */
}
I have a query that joins EntityA with a LEFT JOIN to Entity B. And a 'ON' clause.
In normal SQL lingo this would be:
select * from EntityA eA left join EntityB eB
on (eA.ID = eB.ENTITY_A_ID and eB.SOME_PROPERTY = "blabla" )
where ...
So I'm having much needed information from my joined resultset. I only want records joined if they match certain properties. I need EntityA, allways, and an attached EntityB if EntityB matched the join clause.
The project is set up with Hibernate / JPA. I can't figure out how to retreive the information needed. At this moment I have:
public class EntityADAO {
public List<EntityA> findMethod() {
CriteriaBuilder builder = entityManager.getCriteriaBuilder();
CriteriaQuery<EntityA> query = builder.createQuery(EntityA.class);
Root<EntityA> entityARoot = query.from(EntityA.class);
Join<EntityA, EntityB> entityBJoin = entityARoot.join("entityB", JoinType.INNER);
entityBJoin.on(new Predicate [] {builder.equal(entityBJoin.get("someProperty"), "fixed_val_for_now"});
/* where clause left out for readability */
TypedQuery<EntityA> q = entityManager.createQuery(query);
return q.getResultList();
}
}
So here I am.. Stuck with my List of EntityAs. whenever I call getEntityBs() on a EntityA, I'm getting all of them.. And this makes sense.. But How can I retrieve the joined set?
I'm stuck with JPA and Hibernate, as this choice is not made by me.
Thanks in advance!

What you need here is a custom projection or DTO. Filtering the entity collection might cause a delete because entities always reflect the current DBMS state and are synchronized at the end of the transaction.
You can write a JPQL query, just like the SQL one, that does what you want.
SELECT a.id, b.id
FROM EntityA a
LEFT JOIN EntityB b ON a.id = b.entityA.id AND b.someProperty = 'blabla'
But this won't help you with the materialization of the results into rich objects. If an Object[] i.e. the tuples are good enough for your use case, then use this kind of query and be done, but if you want to map to rich objects, I can recommend that you take a look at Blaze-Persistence Entity-Views.
Blaze-Persitence is a query builder on top of JPA which supports many of the advanced DBMS features on top of the JPA model. I created Entity Views on top of it to allow easy mapping between JPA models and custom interface defined models, something like Spring Data Projections on steroids. The idea is that you define your target structure the way you like and map attributes(getters) via JPQL expressions to the entity model. Since the attribute name is used as default mapping, you mostly don't need explicit mappings as 80% of the use cases is to have DTOs that are a subset of the entity model.
A mapping for your model could look as simple as the following
#EntityView(EntityA.class)
public interface EntityAView {
long getId();
#Mapping("entityBs[someProperty = 'blabla']")
List<EntityBView> getEntityBs();
}
#EntityView(EntityB.class)
public interface EntityBView {
long getId();
}
Querying is a matter of applying the entity view to a query, the simplest being just a query by id.
EntityAView dto = entityViewManager.find(entityManager, EntityAView.class, id);
The Spring Data integration allows you to use it almost like Spring Data Projections: https://persistence.blazebit.com/documentation/entity-view/manual/en_US/index.html#spring-data-features

Hibernate #Where annotation not working with inheritance

I am using Hibernate 5.1.2
I have run into an unexpected problem that I can't seem to work around. Here's the summary of my data model:
dfip_project_version is my superclass table, and dfip_appln_proj_version is my subclass table. dfip_application contains a list of dfip_appln_proj_versions.
I have mapped this as follows:
#Table(name = "DFIP_PROJECT_VERSION")
#Entity
#Inheritance(strategy = InheritanceType.JOINED)
public abstract class AbstractProjectVersion {
#Id #GeneratedValue
#Column(name = "PROJECT_VERSION_OID")
Long oid;
#Column(name = "PROJ_VSN_EFF_FROM_DTM")
Timestamp effFromDtm;
#Column(name = "PROJ_VSN_EFF_TO_DTM")
Timestamp effToDtm;
#Column(name = "PROJECT_VERSION_TYPE")
#Type(type = "project_version_type")
ProjectVersionType projectVersionType;
}
#Table(name = "DFIP_APPLN_PROJ_VERSION")
#Entity
class ApplicationProjectVersion extends AbstractProjectVersion {
#OneToOne
#JoinColumn(name = "APPLICATION_OID", nullable = false)
Application application;
public ApplicationProjectVersion() {
projectVersionType = ProjectVersionType.APPLICATION;
}
}
#Table(name = "DFIP_APPLICATION")
#Entity
class Application {
#Id #GeneratedValue
#Column(name = "APPLICATION_OID")
Long oid;
#OneToMany(mappedBy="application", orphanRemoval = true, fetch = FetchType.EAGER)
#Fetch(FetchMode.SELECT)
#Where(clause = "PROJ_VSN_EFF_TO_DTM is null")
List<ApplicationProjectVersion> applicationVersions = [];
}
I am using the #Where annotation so that only the current ApplicationProjectVersion is retrieved with the Application.
The problem with this is that Hibernate assumes that the column I am referencing is in the dfip_appl_proj_version table, when it's actually on the super-class table (dfip_project_version).
Here's what I tried so far to work around this limitation:
Attempt 1
I tried putting the #Where annotation onto the AbstractProjectVersion super-class, like so:
#Table(name = "DFIP_PROJECT_VERSION")
#Entity
#Inheritance(strategy = InheritanceType.JOINED)
#Where(clause = "PROJ_VSN_EFF_TO_DTM is null")
public abstract class AbstractProjectVersion {
...etc...
}
This did nothing, as the WHERE clause does not seem to be noticed when retrieving the Application.
Attempt 2
I made the applicationVersions list on Application LAZY, and tried to map latestVersion manually like this:
#Table(name = "DFIP_APPLICATION")
#Entity
class Application {
#Id #GeneratedValue
#Column(name = "APPLICATION_OID")
Long oid;
#OneToMany(mappedBy="application", orphanRemoval = true, fetch = FetchType.LAZY)
#Fetch(FetchMode.SELECT)
List<ApplicationProjectVersion> applicationVersions = [];
#ManyToOne
#JoinColumnsOrFormulas([
#JoinColumnOrFormula(formula = #JoinFormula(value = "(APPLICATION_OID)", referencedColumnName="APPLICATION_OID")),
#JoinColumnOrFormula(formula = #JoinFormula(value = "(select apv.PROJECT_VERSION_OID from DFIP_PROJECT_VERSION pv, DFIP_APPLN_PROJ_VERSION apv where apv.PROJECT_VERSION_OID = pv.PROJECT_VERSION_OID and apv.APPLICATION_OID = APPLICATION_OID and pv.PROJ_VSN_EFF_TO_DTM is null)", referencedColumnName="PROJECT_VERSION_OID")),
])
ApplicationProjectVersion latestVersion;
}
This caused Hibernate to generate the following SQL (snippet):
from DFIP_APPLICATION this_
left outer join DFIP_APPLN_PROJ_VERSION applicatio2_
on (this_.APPLICATION_OID)=applicatio2_.APPLICATION_OID and
(select apv.PROJECT_VERSION_OID from DFIP_PROJECT_VERSION pv, DFIP_APPLN_PROJ_VERSION apv
where apv.PROJECT_VERSION_OID = pv.PROJECT_VERSION_OID and apv.APPLICATION_OID = this_.APPLICATION_OID
and pv.PROJ_VSN_EFF_TO_DTM is null)=applicatio2_.PROJECT_VERSION_OID
which resulted in ORA-01799: a column may not be outer-joined to a subquery.
If I can't specify a sub-query in my join formula, then I cannot join to the super-class manually...
Attempt 3
I noticed that usage of #JoinFormula makes Hibernate notice my #Where annotation on the super-class. So I tried the following:
#Table(name = "DFIP_PROJECT_VERSION")
#Entity
#Inheritance(strategy = InheritanceType.JOINED)
#Where(clause = "PROJ_VSN_EFF_TO_DTM is null")
public abstract class AbstractProjectVersion {
...etc...
}
#Table(name = "DFIP_APPLICATION")
#Entity
class Application {
#Id #GeneratedValue
#Column(name = "APPLICATION_OID")
Long oid;
#OneToMany(mappedBy="application", orphanRemoval = true, fetch = FetchType.LAZY)
#Fetch(FetchMode.SELECT)
List<ApplicationProjectVersion> applicationVersions = [];
#ManyToOne
#JoinFormula(value = "(APPLICATION_OID)", referencedColumnName="APPLICATION_OID")
ApplicationProjectVersion latestVersion;
}
This generated the following SQL (snippet):
from DFIP_APPLICATION this_
left outer join DFIP_APPLN_PROJ_VERSION applicatio2_
on (this_.APPLICATION_OID)=applicatio2_.APPLICATION_OID and ( applicatio2_1_.PROJ_VSN_EFF_TO_DTM is null)
left outer join DFIP_PROJECT_VERSION applicatio2_1_ on applicatio2_.PROJECT_VERSION_OID=applicatio2_1_.PROJECT_VERSION_OID
This is almost correct! Unfortunately it is not valid SQL, since applicatio2_1_ is used before it is declared on the next line :(.
Now I am out of ideas, so any help would be appreciated. Is there a way to specify a WHERE clause that will bring in only the current ProjectVersion, without getting rid of my inheritance structure?
Related Hibernate issue ticket

I have a solution to this problem. I must admit, it ended up being a little more cumbersome than what I hoped for, but it does work quite well. I waited a couple of months before posting, to make sure that there are no issues and so far, I have not experienced any problems.
My entities are still mapped exactly as described in the question, but instead of using the problematic #Where annotation, I had to use #Filter annotation instead:
public class Application {
#OneToMany(mappedBy="application", orphanRemoval = true, fetch = FetchType.EAGER)
#Cascade([SAVE_UPDATE, DELETE, MERGE])
#Fetch(FetchMode.SELECT)
// Normally we'd just use the #Where(clause = "PROJ_VSN_EFF_TO_DTM is null"), but that doesn't work with collections of
// entities that use inheritance, as we have here.
//
// Hibernate thinks that PROJ_VSN_EFF_TO_DTM is a column on DFIP_APPLN_PROJ_VERSION table, but it is actually on the "superclass"
// table (DFIP_PROJECT_VERSION).
//
// B/c of this, we have to do the same thing with a Filter, which is defined on AbstractProjectVersion.
// NOTE: This filter must be explicitly enabled, which is currently achieved by HibernateForceFiltersAspect
//
#Filter(name="currentProjectVersionOnly",
condition = "{pvAlias}.PROJ_VSN_EFF_TO_DTM is null",
deduceAliasInjectionPoints=false,
aliases=[ #SqlFragmentAlias(alias = "pvAlias", table = "DFIP_PROJECT_VERSION") ]
)
List<ApplicationProjectVersion> projectVersions = [];
}
Since we are using a Filter, we must also define it:
// NOTE: This filter needs to be explicitly turned on with session.enableFilter("currentProjectVersionOnly");
// This is currently achieved with HibernateForceFiltersAspect
#FilterDef(name="currentProjectVersionOnly")
#Table(name = "DFIP_PROJECT_VERSION")
#Inheritance(strategy = InheritanceType.JOINED)
public abstract class AbstractProjectVersion {
}
And of course, we must enable it, since Hibernate does not have a setting to automatically turn on all filters.
To do this I created a system-wide Aspect, whose job is to enable specified filters before every call to any DAO:
/**
* Enables provided Hibernate filters every time a Hibernate session is openned.
*
* Must be enabled and configured explicitly from Spring XML config (i.e. no auto-scan here)
*
* #author Val Blant
*/
#Aspect
public class HibernateForceFiltersAspect {
List<String> filtersToEnable = [];
#PostConstruct
public void checkConfig() throws Exception {
if ( filtersToEnable.isEmpty() ) {
throw new IllegalArgumentException("Missing required property 'filtersToEnable'");
}
}
/**
* This advice gets executed before all method calls into DAOs that extend from <code>HibernateDao</code>
*
* #param jp
*/
#Before("#target(org.springframework.stereotype.Repository) && execution(* ca.gc.agr.common.dao.hibernate.HibernateDao+.*(..))")
public void enableAllFilters(JoinPoint jp) {
Session session = ((HibernateDao)jp?.getTarget())?.getSession();
if ( session != null ) {
filtersToEnable.each { session.enableFilter(it) } // Enable all specified Hibernate filters
}
}
}
And the corresponding Spring configuration:
<!-- This aspect is used to force-enable specified Hibernate filters for all method calls on DAOs that extend HibernateDao -->
<bean class="ca.gc.agr.common.dao.hibernate.HibernateForceFiltersAspect">
<property name="filtersToEnable">
<list>
<value>currentProjectVersionOnly</value> <!-- Defined in AbstractProjectVersion -->
</list>
</property>
</bean>
And there you have it - polymorphic #Where clause :).

since you are looking for #Where with inheritance, I assume you are trying to apply some SQL logic globally, maybe hibernate interceptor or SQL inspector would be a better fit for this type of requirement

Hibernate ignoring discriminator column - always using 'dtype'

I have a strange situation in my SINGLE_TAB inheritance Hibernate config whereby the #DiscriminatorColumn seems to be ignored and the query is always defaulting back to the 'dtype' column. It's like the behaviour I would see when I had not included the annotation at all (the default column name being 'dtype').
Base entity:
#Entity
#Table(name = "post")
#Inheritance(strategy = InheritanceType.SINGLE_TABLE)
#DiscriminatorColumn(columnDefinition = "post_type", discriminatorType = DiscriminatorType.STRING)
public class Post {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private Integer id;
#Column(name = "title")
private String title;
#Column(name = "body")
private String body;
#NotNull
#Column(name = "post_type", insertable = false, updatable = false)
private String postType;
// other simple columns
// ommit getters/setters + hashcode etc
}
Subclass entity:
#Entity
#DiscriminatorValue(value = "EVENT")
public class Event extends Post {
// ommitted basic methods, no extra config
}
I also need access to the discriminator value itself within each object (the postType field). I still have the same behaviour even if I remove it so it doesn't seem to be the cause.
When I try to do a query on the subclass through a JPA repository:
public interface EventRepository extends JpaRepository<Event, Integer> {
List<Event> findAll();
}
Hibernate generates the query:
select post0_.id as id2_4_, post0_.bodyl as body_bod3_4_, post0_.title as title12_4_
from post post0_
where post0_.dtype='EVENT'
which of course generates an error as 'dtype' doesn't exist in the table.
The strange thing is that if I use #DiscriminatorFormula("post_type") on the Post entity instead, everything seems to work. It is however slower so I would prefer to use the #DiscriminatorColumn as it should fit my needs exactly.
I am using Hibernate 5.2.10-FINAL and Spring Data JPA 1.11.4 (or generally the latest of hopefully everything).
Any ideas on what could be causing this?

I'm think you have this problem because you specified wrong parameter of #DiscriminatorColumn anotation, you should use name instead of columnDefinition.

Is this an N+1 problem and how do I fix it?

I have a list of projects and a list of customers. A project can be for one customer and every customer can have many projects. So it's a simple 1:n relationship where the project is the owning side.
Simplified to the essential it is
#Entity
public class Project {
#Id
long id;
#ManyToOne(optional = true)
#JoinColumn(name = "customer", nullable = true, updatable = true)
Customer customer;
}
#Entity
public class Customer {
#Id
long id;
}
When I load a list of projects, I want to retrieve the customers efficiently at the same time. This is not the case. There is one single query for the projects and then for every distinct customer that is encountered a separate query is issued.
So say I have 100 projects that are assigned to 50 different customers. This would result in one query for the projects and 50 queries for the customers.
This quickly adds up and for large project/customer lists our application gets rather slow. Also this is just one example. All our entities with relationships are affected by this behavior.
I already tried #Fetch(FetchMode.JOIN) on the customers field as suggested here but it does nothing and FetchMode.SUBQUERY is not applicable according to Hibernate:
org.hibernate.AnnotationException: Use of FetchMode.SUBSELECT not allowed on ToOne associations
How can I fix this problem?

If you are using Spring Data JPA to implement your repositories, you can specify lazy fetching in the JPA entities:
#Entity
public class Project {
#Id
long id;
#ManyToOne(fetch = FetchType.LAZY, optional = true)
#JoinColumn(name = "customer", nullable = true, updatable = true)
Customer customer;
}
#Entity
public class Customer {
#Id
long id;
...
}
And add #EntityGraph to your Spring Data JPA-based repository:
#Repository
public interface ProjectDao extends JpaRepository<Project, Long> {
#EntityGraph(
type = EntityGraphType.FETCH,
attributePaths = {
"customer"
}
)
Optional<Project> findById(Long id);
...
}
My blog post at https://tech.asimio.net/2020/11/06/Preventing-N-plus-1-select-problem-using-Spring-Data-JPA-EntityGraph.html helps you preventing the N+1 select problem using Spring Data JPA and #EntityGraph.

Yes, it is a by-the-book example of the n+1 selects problem.
The approach I use in most cases is to make the association lazy and define a batch size.
Alternatively, you could use a JPQL query with [left] join fetch to initialize the association directly from the query result set:
select p from Project p left join fetch p.customer

Yes, it is a by-the-book example of the n+1 selects problem as #dragan-bozanovic said.
In Spring-Boot 2.1.3 #Fetch(FetchMode.JOIN) can be used to solve it:
#ManyToOne(optional = true)
#Fetch(FetchMode.JOIN)
#JoinColumn(name = "customer", nullable = true, updatable = true)
Customer customer;
Warning: If the relationship can be invalid, for example when marked with #NotFound(action = NotFoundAction.IGNORE), each invalid relationship will trigger another SELECT query.