We Keep Coding

In Hibernate, how to avoid the N+1 query problem and large results sets due to multiple joins

I am using Spring Boot and Hibernate.
Some complex logic, dictated by business, needs to use various nested fields, which traverse various DB relationships (again, some are NxN, Nx1, 1xN, 1x1).
I encountered the N+1 problem, and I solved it at first with HQL, but some queries need several joins and the result sets become unmanageable.
I started working on a custom utility that collects the ids of things that need to be fetched, fetches them all at once and uses the setters to then populate the fields on the starting objects. This utility works for ManyToOne relationships, but is still inefficient with ManyToMany relationships, because it falls back in the N+1 problem when I collect the ids (as it queries the join table once per object via the getter).
How can I solve this? Has this problem really not been solved yet? Am I missing some obvious settings that solves this automagically?
EDIT:
I made a toy example with some commentary: https://github.com/marcotama/n-1-queries-example

I had faced the same situation and I had 3 ways to solve it;
increase the fetchsize for the dependent attribute so that the queries are executed in batch
write a custom query for the purpose
define entity graph relations and map accordingly to attributes
I personally preferred the 3rd option as it was convenient to do that and was cleaner with spring data JPA.
you can refer to examples from the comments from the below answers:
Spring Data JPA And NamedEntityGraphs
What is the solution for the N+1 issue in JPA and Hibernate?

Write fetch logic on your own.
E.g You have author which has book, author_devices
You can join fetch author with books. Than you can separatly fetch author_devices using repository "where author_id IN (authorsList.stream().map(author.getId())". Than you should detach author and iterate author_devices and assign it to apropriate author devices list. I think it's only adequate solution for situations where you need to join-fetch more than 1 relation.

Is there better solution to map lots of fields entity into two tables?

I am getting data with 50 fields. How to map into two tables in jpa, in which I required only three fields and rest of fields I need to only pass to downstream?
This is for a new spring data jpa, running H2 db project. In the past, I’ve tried to map all 50 fields in one entity table. But I require only 3 essential fields from those 50 fields to process of last 5000 records from db.
#Entity
class SingleCustomer{
//ID field
// 3 fields only these fields I am using to process my data from db
//47 fields just to pass to another service without any process
}
To improve performance of my application which approach I should take?
Are there any other ways to improve application performance like split into two entities and join on one to one when needs to send another service.

As you said you can split all the fields into two entities and join on one-to-one relationShip. But you should notice that field with one-to-one relationShip is loaded eagerly by default, you should mark it as lazy-loaded to improve performance.

Using a separate entity for fields you don't need together with a #OneToOne association is one way of doing that, as #Lebecca pointed out.
However, if you'd rather keep all the properties in one entity, you could turn on Hibernate bytecode enhancement, which enables support for #Basic(fetch = LAZY). If you want, you should also be able to combine such an approach with using a #SecondaryTable to store the properties you don't usually need.
See here for more info, including info on how to fine-tune the lazy loading behavior with enhancement enabled.

How to speed up List access in JPA

In my mapper class, I always have to retrieve a list of departments from my company entity :
#OneToMany(mappedBy = "company", orphanRemoval = true, cascade = CascadeType.ALL)
public List<CompanyDepartments> getDepartments()
{
return departments; //java.util.List variable.
}
There are a couple thousand departments(about 2000+). I am trying to reduce the time taken to fetch company data and want to begin with the departments which are going to be fairly static in nature.
Option 1 : I can always have a private method in the mapper which populates a cache on the first load and return from the cache all the time.
But is there anything more simpler? I am probably not the first one to face this and wanted to know how I could approach this. Is there a known annotation that can be used. Like a #Singleton annotation on the variable in the entity ? (there is no such annotation obviously). What is the simplest way to make this list a singleton.
Just a typical spring mvc 3 application using spring data jpa for db interaction.

I suggest this link that solves the n +1 query problem ...
How can i resolve the N+1 Selects problem?
Moreover you could put a cache (lvl 2) on your search service,
if the data does not change during the life cycle of the application.
http://docs.spring.io/spring/docs/3.1.0.M1/spring-framework-reference/html/cache.html
Another approach is to add to indexes on the db.
I hope I've given you all the answers about your question.

What you're experiencing is part of the "object-relational impedance mismatch". One solution would be to extend your object model so that you can use a left join fetch to load multiple companies including their departments using only one SQL statement.
The problem using this technique is that you can populate only one list at a time. Although it's nice & easy to define many sub-lists in the world of objects, it's hard to load all these lists at the same time (and therefor efficiently) from a relational database. However it can be done on Oracle using object- or XMLType queries.
The straight-forward way to get such data out of a RDBMS is by writing custom queries that match exactly the task at hand and provide only the data that is actually needed. Therefor you'd need not only one company class, but many - one for each task. - you'd actually do inheritance instead of attribution
BTW, that is why ORM is still considered the Vietnam of Computer science - easy to get started, but hard to succeed with.

How to model relationship between different parent entities and one common child entity

I'm looking for some suggestions for best practices around modeling the relationship between various entities and their documents (binaries such as PDF, TIFF etc). The entities are standard JPA/Hibernate stored in a PostgreSQL database. The documents themselves will be stored in MongoDb database.
The plan is to create a child entity to represent the document, which contains the id to the binary data to retrieve it as needed. But what would the relationship be?
If I simply created one if these document entities for each parent entity then a simple one to many relationship would work, but that seems to redundant.
I could simply put a "type" column that indicates which entity the document belongs to, and then query the document table with a named query of "id = ? and type = ?". I guess that would work, but there is something about that I'm not crazy about either - just can't put my finger on it :) Maybe that's just fine.
Another option I have looked at (although I admit I have never used it before, and would need to study it a bit more) is to use a unidirectional one to many with join table. However, I don't think this will work either since there is no guarantee that there wouldn't be duplicate parent keys. I use a single sequence for all basic relation tables primary keys, which should guarantee it, but it still doesn't sound like a good idea.
Finally, I have considered whether I create an entity and then extend it for each parent entity, but I think that would have the same flaw - the theoretical existence of non-unique parent ids.
Before I make a final decision, I'd like to see what other suggestions the community might have to offer.
Thanks in advance for your ideas.

If I simply created one if these document entities for each parent entity then a simple one to many relationship would work, but that seems to redundant.
I'm a bit confused. If you create a document for each parent, isn't that one-to-one and not one-to-many? It sounds like you want a one-to-many relationship. In which case, you would only create a single document for all parent entities that reference it.

Hibernate & EntityManager with joins

In my hypothetical I have an annotated User model class. This User model also holds references to two sets:
A set of Pet objects (a Pet object is also an annotated model represented in the data layer)
A set of Food objects (a Pet object is also an annotated model represented in the data layer)
When I pull the User entity from the database (entityManager.find(User.class, id)) it will automatically fill all the User fields but it obviously wont fill the two sets.
Do I need to do entityManager.createQuery and just use a normal SQL join query then manually create the User object?
Thanks in advance

If you map your relations from User to Pet and Food using OneToMany you can chose whether to have the fields automatically collected or not.
See the API doc for javax.persistence OneToMany.
Depending on how you constructed the mapping (PK-FK or join tables etc), you may or may not get good performance with this. Having two OneToMany relations that are joined, means you may end up with a ridiculous amount of rows when you read up your user.

Mmm, No? That's probably not how you want to do it. I don't know why you say "it obviously won't fill the two sets." It's quite capable of filling in the sets for you, that's sort of the point behind using an ORM like hibernate in the first place. Your objects do what they look like they should in code and 'databasey' things are handled automatically as much as possible.
It is true that Hibernate will complain if you mark more than one collection as EAGER fetched, but it's not really clear you actually need either of them to be eager. Essentially once they are mapped, just accessing them causes the queries to be run to fill them in with data (assuming the Session is still open and so forth.) If you explain how you want it to work it would be easier to help with a solution.