We Keep Coding

Converting list of model entities with relationships to DTO takes time

I'm using Spring boot JPA to get list of objects (Using Java 8 now). Each object has relationships and I use the related objects also to transform to a dto list.
Let's say I have below model classes.
public class Product {
#EmbeddedId
private ProductId id;
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "userid", referencedColumnName = "USER_ID")
#MapsId("userId")
private User owner;
}
public class User {
#Id
#Column(name = "USER_ID")
private Long userId;
#OneToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "gp_code", referencedColumnName = "GP_CODE")
#JoinColumn(name = "userid", referencedColumnName = "USER_ID")
private UserGroup userGroup;
#OneToOne(fetch = FetchType.LAZY)
#JoinColumnsOrFormulas(value = {
#JoinColumnOrFormula(formula = #JoinFormula(value = "country_id", referencedColumnName = "COUNTRY_ID")),
#JoinColumnOrFormula(column = #JoinColumn(name = "region_code", referencedColumnName = "REGION_CODE")) })
private Country country;
}
I do query for List<Product> and using stream I'm converting it into a dto object. During which I call the related entity to get the data. I have the below code and works fine unless the list is too much. If I have 1000+ items in the list it takes around 30 seconds.
I believe because of lazy loading this is happening. What is the best way to optimize this?
One option is to do pagination, but I cannot do it. I need all results together.
Is there any way to parallelly execute this? I tried to call parellelStream() instead of stream(), but it's same result.
public List<ProductDTO> getProducts(String countryId) {
//List<Product> products = Query Result
List<ProductDTO> productsList = products.stream().filter(isOwnerFromCountryAndInAnyGroup(countryId))
.map(product -> getProductDTO(product)).collect(Collectors.toList());
}
private Predicate<? super Product> isOwnerFromCountryAndInAnyGroup(String countryId) {
return product -> {
User user = product.getOwner();
return null != user && null != user.getCountry()
&& user.getCountry().getCountryId().equals(countryId) && (null != user.getUserGroup());
};
}
private ProductDTO getProductDTO(Product product) {
ProductDTO productDTO = new ProductDTO();
productDTO.setProductNbr(product.getId().getProductNbr());
productDTO.setPrice(product.getPrice());
productDTO.setOwnerName(product.getOwner().getName());
return productDTO;
}
Edit
I missed to add the line productDTO.setOwnerName(product.getOwner().getName()); for the purpose of asking question here. With query or using filter I'm getting the correct number of results. And with lazy loading, query returns faster and then while calling getOwner() for each row, the process takes time (30 seconds).
And with FethType.EAGER, the query takes similar time(30 seconds) and then processes faster. Either way it is similar time.
To fasten the process, is there any way to execute the stream code block in parallel and collect all results together in list?

public List<ProductDTO> getProducts(String countryId) {
//List<Product> products = Query Result
List<ProductDTO> productsList = products.stream().filter(isOwnerFromCountryAndInAnyGroup(countryId))
.map(product -> getProductDTO(product)).collect(Collectors.toList());
}
From your use case here I am pretty confident that it is not the creation of DTO that takes time. It is that you retrieve a huge set from database (even the complete table of Products) and then you filter for a relation with a specific country just from java.
So Step1 optimization:
If you want to filter for products that are associated with a user from a specific country then this can go on JPA level and translated in optimal way in database. Then the allocation of resources (memory, cpu) would be much more optimal, instead of your java application trying to load a huge data set and filter it there.
#Query("SELECT p FROM Product p where p.owner IS NOT NULL AND p.owner.userGroup IS NOT NULL AND p.owner.country IS NOT NULL AND p.owner.country.id = :countryId")
List<Product> findProductRelatedWithUserFromCountry(#Param String countryId);
and remove the filtering from your method getProducts.
Step2 optimization:
In addition to the above, not only you can pass the java filtering in the database query by moving it to JPA layer but you can also optimize the query a bit more by defining in JPA that you want to load the associated Owner as well so that it doesn't hit later the database to retrieve it when you create the DTO. You can achieve this with join fetch, so your query should now become:
#Query("SELECT p FROM Product p JOIN FETCH p.owner own where p.owner IS NOT NULL AND own.userGroup IS NOT NULL AND own.country IS NOT NULL AND own.country.id = :countryId")
List<Product> findProductRelatedWithUserFromCountry(#Param String countryId);
Step3 optimization:
If we want to take it an extra step further it seems that most times using DTO projections would speed up the execution. This can happen as the query would define only specific information it needs to retrieve and convert into DTO instead of the complete entities.
So your query now would be:
#Query("SELECT new org.your.package.where.dto.is.ProductDTO(p.id.productNbr, p.price, own.name) FROM Product p JOIN FETCH p.owner own where p.owner IS NOT NULL AND own.userGroup IS NOT NULL AND own.country IS NOT NULL AND own.country.id = :countryId")
List<ProductDTO> findProductRelatedWithUserFromCountry(#Param String countryId);
Also remember to have the DTO constructor used in the JPA query available in your ProductDTO.class.

Why are lazy fields on related entities loaded

In my REST API project (Java 8, Spring Boot 2.3.1) I have a problem with some queries triggering massive query chains by loading lazy relations, even though the related objects are never accessed.
I have a UserEntity and a polymorphic CompanyEntity that are related with a ManyToMany relationship. I have an endpoint that returns all users and I include the IDs of the related companies in the JSON. I excpect a query to the user table and a query to the company table, however all related entities of one sub-entity of CompanyEntity are always loaded for each of those sub-entities resulting in large query chains.
Here are snippets of my classes:
User entity
#Entity(name = "USERS")
public class UserEntity {
#Id
#GeneratedValue
private UUID id;
#EqualsAndHashCode.Exclude
#Fetch(FetchMode.SUBSELECT)
#ManyToMany(fetch = FetchType.LAZY)
#JoinTable(
name = "users_company",
joinColumns = #JoinColumn(name = "USER_ID"),
inverseJoinColumns = #JoinColumn(name = "COMPANY_ID")
)
private Set<CompanyEntity> companies = new HashSet<>();
public List<UUID> getCompanyIds() {
return companies.stream()
.map(CompanyEntity::getId)
.collect(Collectors.toList());
}
}
Polymorphic company entity
#Entity(name = "COMPANY")
#Inheritance(strategy = InheritanceType.JOINED)
public abstract class CompanyEntity {
#Id
#GeneratedValue
private UUID id;
#Fetch(FetchMode.SUBSELECT)
#ManyToMany(mappedBy = "companies", fetch = FetchType.LAZY)
private Set<UserEntity> users = new HashSet<>();
}
Concrete company subclass that triggers the problem
#Entity(name = "CUSTOMER")
public class CustomerEntity extends CompanyEntity {
#NotNull
#OneToOne(cascade = {CascadeType.PERSIST, CascadeType.MERGE}, fetch = FetchType.LAZY)
private ContactPersonEntity contactPerson;
#Fetch(FetchMode.SUBSELECT)
#OneToMany(cascade = {CascadeType.PERSIST, CascadeType.MERGE}, fetch = FetchType.LAZY, mappedBy = "customer")
private Set<TransactionEntity> transactions = new HashSet<>();
public Set<UUID> getTransactionIds() {
return this.transactions.stream()
.map(TransactionEntity::getId)
.collect(Collectors.toSet());
}
}
In the REST controller I return the following mapping:
#GetMapping(value = "", produces = MediaType.APPLICATION_JSON_VALUE)
public List<UserReadModel> getUsers() {
return userRepository.findAll().stream()
.map(userEntity -> new UserReadModel(userEntity))
.collect(Collectors.toList());
}
Where the UserReadModel is a DTO:
#Data
public class UserReadModel {
private UUID id;
private List<UUID> companyIds;
}
Logging the database queries results in the following output:
// Expected
Hibernate: select userentity0_.id as id1_47_, ... from users userentity0_
Hibernate: select companies0_.user_id ... case when companyent1_1_.id is not null then 1 when companyent1_2_.id is not null then 2 when companyent1_.id is not null then 0 end as clazz_0_ from users_company companies0_ inner join company companyent1_ on companies0_.company_id=companyent1_.id left outer join customer companyent1_1_ on companyent1_.id=companyent1_1_.id left outer join external_editor companyent1_2_ on companyent1_.id=companyent1_2_.id where companies0_.user_id in (select userentity0_.id from users userentity0_)
// Unexpected as they are marked lazy and never accessed
Hibernate: select contactper0_.id ... from contact_person contactper0_ where contactper0_.id=?
Hibernate: select transactio0_.customer_id ... from transactions transactio0_ where transactio0_.customer_id=?
Hibernate: select contactper0_.id ... from contact_person contactper0_ where contactper0_.id=?
Hibernate: select transactio0_.customer_id ... from transactions transactio0_ where transactio0_.customer_id=?
...
I've read through loads of articles on entity mapping and lazy loading but I can't seem to find a reason why this behavior persists. Did anyone have this problem before?

You are accessing the collection, so Hibernate has to load the collection. Since you only need the ids and already have a DTO, I think this is a perfect use case for Blaze-Persistence Entity Views.
I created the library to allow easy mapping between JPA models and custom interface or abstract class defined models, something like Spring Data Projections on steroids. The idea is that you define your target structure(domain model) the way you like and map attributes(getters) via JPQL expressions to the entity model.
A DTO model for your use case could look like the following with Blaze-Persistence Entity-Views:
#EntityView(UserEntity.class)
public interface UserReadModel {
#IdMapping
UUID getId();
#Mapping("companies.id")
Set<UUID> getCompanyIds();
}
Querying is a matter of applying the entity view to a query, the simplest being just a query by id.
UserReadModel a = entityViewManager.find(entityManager, UserReadModel.class, id);
The Spring Data integration allows you to use it almost like Spring Data Projections: https://persistence.blazebit.com/documentation/entity-view/manual/en_US/index.html#spring-data-features
Page<UserReadModel> findAll(Pageable pageable);
The best part is, it will only fetch the state that is actually necessary! In your case, a query like the following will be generated:
select u.id, uc.company_id
from users u
left join users_company uc on uc.user_id = u.id
left join company c on c.id = uc.company_id
Depending on the Hibernate version, the join for the company might even be omitted.

I eventually figured out the solution and want to post it here, in case anyone stumbles upon this question. This was purely a mistake on my side and is not reproducible from the examples I posted.
I used lombok annotations to generate equals and hashcode methods on the customer entity (and all other entities for that matter) and forgot to annotate the contactPerson and transactions fields with #EqualsAndHashcode.Exclude. As the equals method was called somewhere along the execution, it triggered the lazy loading of those fields. Implementing equals and hashcode manually and using the guidelines from this article for that solved the problem.

Why does persisting entity twice avoid errors around cascading for entity with join columns

Why does this unit test fail if i do not perform the setup of the entity Role in two steps (two persists).
The error being:
java.lang.IllegalStateException: During synchronization a new object was found through a relationship that was not marked cascade PERSIST: io.osram.olt.extension.jpa.Role#16daa399.
private Role addRoleWithId(String roleId){
Role myRole = new Role();
myRole.setRoleId(roleId);
myRole.setRealmId("my");
myRole.setDescription("role-description-0");
myRole.setExternalCreator(true);
myRole.setName("role-name-0");
em.persist(myRole); //<--- Without this persisting the role fails with the error above.
//Setup joins:
myRole.setAContext(getApplications().get(0));
myRole.setAnotherContext(getTenants().get(0));
em.persist(myRole);
return myRole;
}
...
The Role Entity:
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "ANOTHER_CONTEXT_ID")
private AnotherContext anotherContext;
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "ACONTEXT_ID")
private AContext aContext;
...
public Role setAContext(AContext aContext) {
this.aContext = aContext;
if(aContext != null) {
aContext.addRole(this);
}
return this;
}
public Role setAnotherContext(AnotherContext anotherContext) {
this.anotherContext = anotherContext;
if(anotherContext != null){
anotherContext.addRole(this);
}
return this;
}
...
The AContext and AnotherContext both contain similar relations towards role:
#OneToMany
#JoinTable(
name="OLT_ROLES_ACONTEXT",
joinColumns = #JoinColumn( name="ACONTEXT_ID"),
inverseJoinColumns = #JoinColumn( name="ROLE_ID")
)
private Set<Role> roles = new HashSet<Role>();
It seems by creating the object in two steps I can avoid using cascading.

In your setAContext and setAnotherContext methods, you are trying to set the Role object which is not yet persisted.
So It's clear that it will not work without em.persist(myRole); before you set contexts since you have not specified CaseCadeType.PERSIST.

The default setting for cascading is cascade NONE , which causes the relationships in the persisted entity not to be persisted by default.
the corollary is that if you try to persist an entity without cascade.PERSIST to its relationship while the relationship is not managed , you will get the above exception.
An exception of the corollary is that if the entity you are persisting is the owner of the relation and the attribute in the relation is already in the database, yo will be able to persist it.
One small thing that I noticed in your mapping : It's a double unidirectional, one with a join column and the reverse with a join table, so is this intended?

Configure many-to-many relationships in Spring Data REST so deleting one object does not delete the other

I am using the latest Spring Data REST (with JPA and MySQL DB) project. I have two objects that I want to setup a relationship between. But when I delete one of the two, I only want to delete the relationship and the object that received the delete request (not both objects).
What I'm relating:
A task
A user
where a task can be related to ("owned by") multiple users and a user can be related to ("own") multiple tasks
Task Class:
#ManyToMany
#JoinTable(name = "task_user", joinColumns = #JoinColumn(name = "task_id", referencedColumnName = "id"), inverseJoinColumns = #JoinColumn(name = "user_id", referencedColumnName = "id"))
private Set<User> users;
User class:
#ManyToMany(mappedBy = "users")
private Set<Task> tasks;
The task_user table consists of two columns
task_id (the id of the task)
user_id (the id of the user)
When I delete a task (via DELETE to /task/{id}), it behaves correctly (the task is deleted from the task table and the relationship is deleted from the task_user table). The issue I have is that when I delete a user, it is deleted from the user table but the relationship still exists in the task_user table.
What I desire:
Deleting a task, deletes all corresponding entries in "task_user" and the entry in "task"; does NOT delete entries in "user"
Deleting a user, deletes all corresponding entries in "task_user" and the entry in "user"; does not delete entries in "task"
Is it possible to achieve that through configuration? Or do I need to use either foreign keys or custom logic, such as event handlers?

Try this:
public class User {
//...
#ManyToMany(cascade = {CascadeType.PERSIST, CascadeType.MERGE})
private Set<Task> tasks;
//...
}
public class Task {
//...
#ManyToMany(mappedBy = "tasks")
private Set<User> users;
//...
}
You chose a bidirectional variant of many-to-many so don't forget about 'helper methods' in User class. See documentation: associations many-to-many.
See my example and tests.

You need to change cascade type of foreign key user_id of table task_user to cascade delete. E.g using IntelliJ built-in database tool:

JPA nativeQuery returns cached resultList

I have following classes:
Company.class:
public class Company {
#JoinTable(name = "company_employee", joinColumns = #JoinColumn(name = "company_id") , inverseJoinColumns = #JoinColumn(name = "employee_id") )
#ManyToMany(fetch = FetchType.LAZY)
private Set<Employee> employees;
#Column(name = "score")
private BigDecimal score;
}
and Employee.class
public class Employee {
#ManyToMany(fetch = FetchType.EAGER, mappedBy="employees")
private Set<Company> companies;
}
The Score column of Company is always null in the db and never updated via dao, because there is other table containing score for each unique pair Company-Employee.
I need the value of Score, only for the case when I fetch Employee by id, so this case all Company instances in the Set should contain score, thus I will get Employee-Company score pairs where employee is fetched Employee.
I have following code to achieve that:
public Employee get(Long id) {
Employee emp = (Employee) dao.find(id);
List<Company> compList = compnanyService.getByEmpId(id);
Set<Company> compSet = new HashSet<Company>(compList);
emp.setCompanies(compSet);
return emp;
}
And Company Dao contains method:
public List<Company> getByEmpId(Long id) {
final Query query = this.entityManager.createNativeQuery("select company.comp_id, ...some other fields, score.score from company join score on company.company_id=score.company_id where score.employee_id=:employee_id",
Company.class);
query.setParameter("employee_id", id);
List<Company> comps = query.getResultList();
return comps;
}
The problem is that getByEmpId(id) gives a ResultList where company.score is null though executed in the db it is not null.
I suspected that there is some caching intervening, so I tried to remove some columns from the native query, and it should have invoked an exception with "no column found" (or alike) message while mapping, but this method still gives List<Company> with all fields on their places though Hibernate prints out my native query in the console with all changes I make.
What am I doing wrong here and how to achieve what I need? Thank you.

It might be associated with first level cache, which can be out of sync when using native SQL queries. From here:
If you bypass JPA and execute DML directly on the database, either
through native SQL queries, JDBC, or JPQL UPDATE or DELETE queries,
then the database can be out of synch with the 1st level cache. If you
had accessed objects before executing the DML, they will have the old
state and not include the changes. Depending on what you are doing
this may be ok, otherwise you may want to refresh the affected objects
from the database.
So you can try using refresh method from EntityManager.

So I ended up doing that:
Created view in db from the query:
CREATE VIEW companyscore AS select company.comp_id, score.emp_id ...some other fields, score.score from company join score on company.comp_id=score.comp_id;
Created corresponding entity CompanyScore with composite primary id as comp_id and emp_id and created view as table.
Changed Employee entity to:
public class Employee {
#OneToMany(fetch = FetchType.EAGER)
#JoinColumn(name = "emp_id")
private Set<CompanyScore> companies;
}
This way I not only have score field always consistent, but I can choose set of fields to show as the whole Company class is quite extensive and I don't need all the fields for this particular case.