We Keep Coding

How to paginate entities with OneToMany relationship with Fetch without warning firstResult/maxResults specified with collection fetch?

I would like to be able to create pagination for pulling all customers from the database (MYSQL), but I encountered a hibernate n+1 problem, which I then solved, but I encountered another problem: 2023-02-09 16:57:04.933 WARN 11660 --- [io-8080-exec-10] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory!
This problem I tried to solve with EntityGraph, but still nothing. Then I tried to use two Query, which collected the id and then used the IN clause, but this caused a huge sql query, which led to the generation of many "IN" which, with a huge dataset, can be problematic.
I am currently in a quandary and do not know how to solve this problem. I would like the figures to be fetched along with the customers, but I have no idea how to do it in such a way that the pagination works properly
I want to return CustomerDTO who have numberOfCreatedFigures attribute which is mapping from method in customer entity. This method is returning a size of customer figures.
I am using lombok for args/getters/setters. I've been trying to do everything, but nothing seems to fix the issue.
Config class with a mapper
#Bean
public ModelMapper modelMapper() {
ModelMapper modelMapper = new ModelMapper();
modelMapper.createTypeMap(Customer.class, CustomerDTO.class)
.addMappings(mapper -> mapper
.map(Customer::numberOfCreatedFigures, CustomerDTO::setNumberOfFigures));
return modelMapper;
}
Customer class
public class Customer implements UserDetails, Serializable {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
#NotBlank(message = "Your name cannot be blank")
private String name;
#NotBlank(message = "Your name cannot be blank")
private String surname;
#NotBlank(message = "Your login cannot be blank")
private String login;
#NotBlank(message = "Your password cannot be blank")
private String password;
#Enumerated(EnumType.STRING)
private Role role;
private Boolean locked = false;
private Boolean enabled = true;
#OneToMany(mappedBy = "createdBy",
cascade = {CascadeType.MERGE, CascadeType.PERSIST},
fetch = FetchType.LAZY,
orphanRemoval = true)
#ToString.Exclude
private Set<Figure> figures = new HashSet<>() ...;
Figure class
public abstract class Figure implements Serializable {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
#Column(updatable = false, insertable = false)
private String figureType;
#Version
private Integer version;
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "created_by_id")
#CreatedBy
#ToString.Exclude
private Customer createdBy;
#CreatedDate
private LocalDate createdAt;
#LastModifiedDate
private LocalDate lastModifiedAt;
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "last_modified_by_id")
#LastModifiedBy
#ToString.Exclude
private Customer lastModifiedBy;
private Integer numberOfModification = 0 ...;
CustomerDTO class
public class CustomerDTO {
private Long id;
private String name;
private String surname;
private String login;
private Integer numberOfFigures;
private Role role;}
Method from Customer Controller
#GetMapping
public ResponseEntity<Page<CustomerDTO>> listAll(#PageableDefault Pageable pageable) {
return new ResponseEntity<>(customerService.listAll(pageable)
.map(customer -> modelMapper
.map(customer, CustomerDTO.class)), HttpStatus.OK);
}

I think this is a perfect use case for Blaze-Persistence Entity Views.
I created the library to allow easy mapping between JPA models and custom interface or abstract class defined models, something like Spring Data Projections on steroids. The idea is that you define your target structure(domain model) the way you like and map attributes(getters) via JPQL expressions to the entity model.
A DTO model for your use case could look like the following with Blaze-Persistence Entity-Views:
#EntityView(Customer.class)
public interface CustomerDTO {
#IdMapping
Long getId();
String getName();
String getSurname();
String getLogin();
#Mapping("SIZE(figures)")
Integer getNumberOfFigures();
Role getRole();
}
Querying is a matter of applying the entity view to a query, the simplest being just a query by id.
CustomerDTO a = entityViewManager.find(entityManager, CustomerDTO.class, id);
The Spring Data integration allows you to use it almost like Spring Data Projections: https://persistence.blazebit.com/documentation/entity-view/manual/en_US/index.html#spring-data-features
Page<CustomerDTO> findAll(Pageable pageable);
The best part is, it will only fetch the state that is actually necessary!

You could load the customers with the figures relationship eagerly initialized.
For this case, an entity graph would be suitable. You'd need to create a new repository method like this:
#Repository
public interface CustomerRepository extends JpaRepository<Customer, Long> {
#EntityGraph(attributePaths = "figures")
List<Customer> findWithFiguresBy(Pageable pageable);
}
Then, you'd need call this repository method when searching instead of the one you are using now. With this approach, your figures relationship can remain lazily fetched (which is generally important as eager fetching is a code smell), but whenever you need to fetch customers with the figures eagerly loaded, you can use this method.
If you want to lear more about entity graphs, I recommend these articles:
JPA Entity Graph by Hibernate maintainer Vlad Mihalcea
JPA Entity Graph by Baeldung
Side note: if you had more than one association which needs to be loaded eagerly, you couldn't use an entity graph for that as it would result in a MultipleBagFetchException. Instead, you would load your parent entities as usual and then collect all ids into a list (say customerIds). Then, you'd need to load all child associations (say figures and otherFigures) by the customer id (JPQL example: select f from Figure f where f.customer.id in :customerIds) and place the figures in a Map<Long, List<Figure> (where the Long parameter is the customer id). Your mapper logic would then need to use the entities from the Maps for the DTOs instead of directly from the parent entity.

Spring Data/Hibernate Generates two queries instead of a JOIN

Context: I have two tables: Questionnaire and Question Section. A Questionnaire can have many Question Sections. Questionnaires and Question Sections both have Start and End Dates to determine if they are active records.
Here are my entities as written:
#Entity
#Data
public class Questionnaire {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private UUID id;
private String name;
private Date startDate;
private Date endDate;
private String description;
#OneToMany(cascade = CascadeType.All,
fetch = FetchType.LAZY,
mappedBy = "questionnaire")
#JsonManagedReference
private List<QuestionSection> questionSections = new ArrayList<QuestionSection>();
}
#Entity
#Data
public class QuestionSection {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private UUID id;
private String name;
private String description;
private int sectionLevel;
private Date startDate;
private Date endDate;
#ManyToOne(fetch = FetchType.LAZY, optional = false)
#JoinColumn(name = "QUESTIONNAIRE_ID", nullable = false)
#JsonBackReference
private Questionnaire questionnaire;
}
Here is my Spring Data Repository with a single declared method:
public interface QuestionnaireRepository extends JpaRepository<Questionnaire, UUID> {
Questionnaire findByNameAndEndDateIsNull(String name);
// Previous goal query, but worked all the way back to the above simple query
// Questionnaire findByIdAndQuestionSectionsEndDateIsNull(UUID id);
}
The above derived query generates two queries shown below:
-- For brevity
select questionnaire.id as id
questionnaire.description as description
questionnaire.end_date as end_date
questionnaire.start_date as start_date
from questionnaire
where questionnaire.name='Foo' and (questionnaire.end_date is null)
select questionsection.questionnaire_id as questionnaire id
...rest of fields here...
from question_section
where questionsection.questionnaire_id = id from above query
Then Spring Data or Hibernate is combining those two above queries into one data object representative of the questionnaire object and returning that.
My problem with this is that I would have expected One query to run with a Join between the two tables, not two and then combine the results in memory. I'm pretty experienced with Spring Data and ORMs in general and have not been able to find any documentation as to why this is happening. Honestly I wouldn't care except that my original intention was to query at the parent entity and 'filter' out children that have end dates (not active). This derived query (commented out above) exhibited the same behavior which ultimately resulted in the data set that was returned containing the end dated question sections.
I know there's 100 other ways I could solve this problem (which is fine) so this is more of an educational interest for me at this point if anyone has any insight into this behavior. I could be missing something really simple.

You should be able to do this using the Entity Graph feature introduced in JPA 2.1.
https://www.baeldung.com/jpa-entity-graph
Spring Data offers support for Entity Graphs via the #NamedEntityGraph and #EntityGraph annotations:
https://www.baeldung.com/spring-data-jpa-named-entity-graphs
So in your code:
Entity:
#Entity
#NamedEntityGraph(name = "Questionnaire.questionSections",
attributeNodes = #NamedAttributeNode("questionSections ")
)
public class Questionnaire{
//...
}
Repository:
public interface QuestionnaireRepository extends JpaRepository<Questionnaire, UUID> {
#NamedEntityGraph("Questionnaire.questionSections")
Questionnaire findByNameAndEndDateIsNull(String name);
}

public interface QuestionnaireRepository extends JpaRepository<Questionnaire, UUID> {
#EntityGraph(attributePaths = { "questionSections" })
Questionnaire findByNameAndEndDateIsNull(String name);
}

JPA Error: InvalidDataAccessApiUsageException: detached entity passed to persist

I have two entities mapped to one another using the oneToMany annotation. One entity is bookedBus and the second is drivers The drivers entity would already have a row inserted into that would later become a foreign reference (FK) to bookedBus entity(PK). Below are the two entities, setters and getter have been skipped for brevity.
First entity
#Entity
#Table(name = "bookedBuses")
public class BookedBuses implements Serializable {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
#ManyToOne(fetch = FetchType.LAZY, cascade = CascadeType.ALL)
#JoinColumn(name = "driver_id")
private Drivers driver;
}
Second entity
#Entity
public class Drivers implements Serializable {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
#OneToMany(fetch = FetchType.LAZY, cascade = CascadeType.ALL, orphanRemoval = true, mappedBy = "driver")
private List<BookedBuses> bookedBus;
}
Now When I try to save to the booked bus entity it throws the following exception
org.springframework.dao.InvalidDataAccessApiUsageException: detached entity passed to persist: com.bus.api.entity.Drivers; nested exception is org.hibernate.PersistentObjectException: detached entity passed to persist: com.bus.api.entity.Drivers
Below is how I tried saving to the bookedBus entity
BookedBuses bookedRecord = new BookedBuses();
bookedRecord.setBookedSeats(1);
bookedRecord.setBookedBusState(BookedBusState.LOADING);
bookedRecord.setBus(busService.getBusByPlateNumber(booking.getPlateNumber()));
bookedRecord.setRoute(booking.getRoute());
infoLogger.info("GETTING DRIVER ID ======= " + booking.getDriver().getId());
Drivers drivers = new Drivers(booking.getDriver().getId());
List<BookedBuses> d_bu = new ArrayList<>();
drivers.setBooked(d_bu);
drivers.addBooked(bookedRecord);
bookedRecord.setDriver(drivers);
bookedBusService.save(bookedRecord);
My BookBusService Save Method as requested
#Autowired
private BookedBusRepository bookedBusRepo;
public boolean save(BookedBuses bookedRecord) {
try {
bookedBusRepo.save(bookedRecord);
return true;
} catch (DataIntegrityViolationException ex) {
System.out.println(ex);
AppConfig.LOGGER.error(ex);
return false;
// Log error message
}
}

1st you have some mix up in naming: you have Driver & Drivers. Like this:
private Drivers driver;
Also selecting variable names like this:
BookedBuses bookedRecord = new BookedBuses();
will cause a lot of confusion. Do not mix plural & singular between types and preferably also do not introduce names that might not be easily associated like record. Also this:
private List<BookedBuses> bookedBus;
which should rather be like:
private List<BookedBus> bookedBuses;
(and would alsoi require change to your class name BookedBuses -> BookedBus)
Anyway the actual problem seems to lie here:
Drivers drivers = new Drivers(booking.getDriver().getId());
You need to fetch existing entity by id with a help of repository instead of creating a new one with id of existing. So something like:
Drivers drivers = driverRepo.findOne(booking.getDriver().getId()); // or findById(..)
It seems that you have a constructor (that you did not show) that enables to create a driver with id. That is not managed it is considered as detached. (You also have drivers.addBooked(bookedRecord); which you did not share but maybe it is trivial)
Note also some posts suggest to changeCascadeType.ALL to CascadeType.MERGE whether that works depends on your needs. Spring data is able to do some merging on save(..) based on entity id but not necessarily in this case.

This line
Drivers drivers = new Drivers(booking.getDriver().getId());
If you already have the driver ID available with you then there's no need to pull the driver ID again from the DB.
After removing the Cascade attribute from #OneToMany & #ManyToOne your code should work.
#Entity
#Table(name = "bookedBuses")
public class BookedBuses implements Serializable {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
`
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "driver_id")
private Drivers driver;
}
#Entity
public class Drivers implements Serializable {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
#OneToMany(fetch = FetchType.LAZY)
#JoinColumn(name = "driver_id")
private List<BookedBuses> bookedBus;
}

Spring data JPA: how to enable cascading delete without a reference to the child in the parent?

Maybe this is an overly simple question, but I am getting an exception when I try to delete a user entity.
The user entity:
#Entity
#Table(name = "users")
public class User
{
#Transient
private static final int SALT_LENGTH = 32;
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private int id;
#NotNull
private String firstName;
#NotNull
private String lastName;
#Column(unique = true, length = 254)
#NotNull
private String email;
// BCrypt outputs 60 character results.
#Column(length = 60)
private String hashedPassword;
#NotNull
private String salt;
private boolean enabled;
#CreationTimestamp
#Temporal(TemporalType.TIMESTAMP)
#Column(updatable = false)
private Date createdDate;
And I have an entity class which references a user with a foreign key. What I want to happen is that when the user is deleted, any PasswordResetToken objects that reference the user are also deleted. How can I do this?
#Entity
#Table(name = "password_reset_tokens")
public class PasswordResetToken
{
private static final int EXPIRATION_TIME = 1; // In minutes
private static final int RESET_CODE_LENGTH = 10;
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private int id;
private String token;
#OneToOne(targetEntity = User.class, fetch = FetchType.EAGER)
#JoinColumn(nullable = false, name = "userId")
private User user;
private Date expirationDate;
The exception I am getting boils down to Cannot delete or update a parent row: a foreign key constraint fails (`heroku_bc5bfe73a752182`.`password_reset_tokens`, CONSTRAINT `FKk3ndxg5xp6v7wd4gjyusp15gq` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`))
I'd like to avoid adding a reference to PasswordResetToken in the parent entity, becaue User shouldn't need to know anything about PasswordResetToken.

It is not possible on JPA level without creating a bidirectional relation. You need to specify cascade type in User class. User should be owner of the relation and it should provide the information on how to deal with related PasswordResetToken.
But if you cannot have a bidirectional relation I would recommend you to setup relation directly in schema generation SQL script.
If you create your schema via SQL script and not via JPA autogeneration (I believe all serious projects must follow this pattern) you can add ON DELETE CASCADE constraint there.
It will look somehow like this:
CREATE TABLE password_reset_tokens (
-- columns declaration here
user_id INT(11) NOT NULL,
CONSTRAINT FK_PASSWORD_RESET_TOKEN_USER_ID
FOREIGN KEY (user_id) REFERENCES users (id)
ON DELETE CASCADE
);
Here is the documentation on how to use DB migration tools with spring boot. And here is the information on how to generate schema script from hibernate (that will simplify the process of writing your own script).

Parent Entity:
#OneToOne
#JoinColumn(name = "id")
private PasswordResetToken passwordResetToken;
Child Entity:
#OneToOne(mappedBy = "PasswordResetToken", cascade = CascadeType.ALL, orphanRemoval = true)
private User user;
If you want the Password entity to be hidden from the client, you can write a custom responses and hide it. Or if you want to ignore it by using #JsonIgnore
If you don't want the reference in the Parent Entity (User), then you have to override the default method Delete() and write your logic to find and delete the PasswordResetToken first and then the User.

You can use Entity listener and Callback method #PreRemove to delete an associated 'Token' before the 'User'.
#EntityListeners(UserListener.class)
#Entity
public class User {
private String name;
}
#Component
public class UserListener {
private static TokenRepository tokenRepository;
#Autowired
public void setTokenRepository(TokenRepository tokenRepository) {
PersonListener.tokenRepository = tokenRepository;
}
#PreRemove
void preRemove(User user) {
tokenRepository.deleteByUser(user);
}
}
where deleteByPerson is very simple method of your 'Token' repository:
public interface TokenRepository extends JpaRepository<Token, Long> {
void deleteByUser(User user);
}
Pay attention on static declaration of tokenRepository - without this Spring could not inject TokenRepository because, as I can understand, UserListener is instantiated by Hybernate (see additional info here).
Also as we can read in the manual,
a callback method must not invoke EntityManager or Query methods!
But in my simple test all works OK.
Working example and test.

JPA. How to return null instead of LazyInitializationException

I have two tables with 'one to many' relationship. I use Jpa + Spring JpaRepository. Sometimes I have to get object from Database with internal object. Sometimes I dont't have to. Repositories always return object with internal objects.
I try to get 'Owner' from Database and I always get Set books; It's OK. But when I read fields of this internal Book , I get LazyInitializationException. How to get null instead of Exception?
#Entity
#Table(name = "owners")
#NamedEntityGraph(name = "Owner.books",
attributeNodes = #NamedAttributeNode("books"))
public class Owner implements Serializable {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(name = "owner_id", nullable = false, unique = true)
private Long id;
#Column(name = "owner_name", nullable = false)
private String name;
#OneToMany(fetch = FetchType.LAZY,mappedBy = "owner")
private Set<Book> books= new HashSet<>(0);
public Worker() {
}
}
#Entity
#Table(name = "books")
#NamedEntityGraph(name = "Book.owner",
attributeNodes = #NamedAttributeNode("owner"))
public class Book implements Serializable {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(name = "book_id", unique = true, nullable = false)
private Long id;
#Column(name = "book_name", nullable = false, unique = true)
private String name;
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "owner_id")
private Owner owner;
public Task() {
}
}
public interface BookRepository extends JpaRepository<Book,Long>{
#Query("select t from Book t")
#EntityGraph(value = "Book.owner", type = EntityGraph.EntityGraphType.LOAD)
List<Book> findAllWithOwner();
#Query("select t from Book t where t.id = :aLong")
#EntityGraph(value = "Book.owner", type = EntityGraph.EntityGraphType.LOAD)
Book findOneWithOwner(Long aLong);
}

You are getting LazyInitializationException because you are accessing the content of the books Set outside the context of a transaction, most likely because it's already closed. Example:
You get an Owner from the database with your DAO or Spring Data repository, in a method in your Service class:
public Owner getOwner(Integer id) {
Owner owner = ownerRepository.findOne(id);
// You try to access the Set here
return owner;
}
At this point you have an Owner object, with a books Set which is empty, and will only be populated when someone wants to access its contents. The books Set can only be populated if there is an open transaction. Unfortunately, the findOne method has opened and already closed the transaction, so there's no open transaction and you will get the infamous LazyInitializationException when you do something like owner.getBooks().size().
You have a couple of options:
Use #Transactional
As OndrejM said you need to wrap the code in a way that it all executes in the same transaction. And the easiest way to do it is using Spring's #Transactional annotation:
#Transactional
public Owner getOwner(Integer id) {
Owner owner = ownerRepository.findOne(id);
// You can access owner.getBooks() content here because the transaction is still open
return owner;
}
Use fetch = FetchType.EAGER
You have fetch = FecthType.LAZY in you #Column definition and that's why the Set is being loaded lazily (this is also the fetch type that JPA uses by default if none is specified). If you want the Set to be fully populated automatically right after you get the Owner object from the database you should define it like this:
#OneToMany(fetch = FetchType.EAGER, mappedBy = "owner")
private Set<Book> books= new HashSet<Book>();
If the Book entity is not very heavy and every Owner does not have a huge amount of books it's not a crime to bring all the books from that owner from the database. But you should also be aware that if you retrieve a list of Owner you are retrieving all the books from all those owners too, and that the Book entity might be loading other objects it depends on as well.

The purpose of LazyInitializationException is to to raise an error when the loaded entity has lost connection to the database but not yet loaded data which is now requested. By default, all collections inside an entity are loaded lazily, i.e. at the point when requested, usually by calling an operation on them (e.g. size() or isEmpty()).
You should wrap the code that calls the repository and then works with the entity in a single transaction, so that the entity does not loose connection to DB until the transaction is finished. If you do not do that, the repository will create a transaction on its own to load the data, and close the transaction right after. Returned entity is then without transaction and it is not possible to tell, if ots collections have some elements or not. Instead, LazyInitializationException is thrown.