We Keep Coding

In Hibernate, how to avoid the N+1 query problem and large results sets due to multiple joins

I am using Spring Boot and Hibernate.
Some complex logic, dictated by business, needs to use various nested fields, which traverse various DB relationships (again, some are NxN, Nx1, 1xN, 1x1).
I encountered the N+1 problem, and I solved it at first with HQL, but some queries need several joins and the result sets become unmanageable.
I started working on a custom utility that collects the ids of things that need to be fetched, fetches them all at once and uses the setters to then populate the fields on the starting objects. This utility works for ManyToOne relationships, but is still inefficient with ManyToMany relationships, because it falls back in the N+1 problem when I collect the ids (as it queries the join table once per object via the getter).
How can I solve this? Has this problem really not been solved yet? Am I missing some obvious settings that solves this automagically?
EDIT:
I made a toy example with some commentary: https://github.com/marcotama/n-1-queries-example

I had faced the same situation and I had 3 ways to solve it;
increase the fetchsize for the dependent attribute so that the queries are executed in batch
write a custom query for the purpose
define entity graph relations and map accordingly to attributes
I personally preferred the 3rd option as it was convenient to do that and was cleaner with spring data JPA.
you can refer to examples from the comments from the below answers:
Spring Data JPA And NamedEntityGraphs
What is the solution for the N+1 issue in JPA and Hibernate?

Write fetch logic on your own.
E.g You have author which has book, author_devices
You can join fetch author with books. Than you can separatly fetch author_devices using repository "where author_id IN (authorsList.stream().map(author.getId())". Than you should detach author and iterate author_devices and assign it to apropriate author devices list. I think it's only adequate solution for situations where you need to join-fetch more than 1 relation.

How to map custom data structures to bean entities with JPA/Hibernate?

We have a (possibly large) custom data structure implemented in Java (8+). It has a simple and optimal API for querying pieces of data. The logical structure is roughly similar to an RDMS (it has e. g. relations, columns, primary keys, and foreign keys), but there is no SQL driver.
The main goal is to access the data via ORM (mapping logical entities to JPA annotated beans). It would be nice if we could use JPQL. Hibernate is preferred but other alternatives are welcome too.
What is the simplest way to achieve this? Which are the key parts of such an implementation?
(P. S. Directly implementing SessionImplementor, EntityManagerImplementor etc. seems to be too complicated.)

You have two possibilities.
Implement a JDBC compliant driver for your system, so you can use a JPA implementation such as Hibernate "directly" (although you may need to create a custom dialect for your system).
Program directly against the JPA specification like ObjectDB does, which bypasses the need to go through SQL and JPA implementations completely.
The latter one is probably easier, but you'd still need to implement the full JPA API. If it's a custom in-house-only system, there's very little sense in doing either one.

One idea I thought up just now, that I feel may work is this:
Use an existing database implementation like H2 and use the JPA integration with that. H2 already has a JPA integration libraries, so it should be easy.
In this database, create a Java stored procedure or function and call it from your current application through JPA. See this H2 documentation on how to create a Java stored procedure or function. (You may want to explore the section "Using a Function as a Table" also.)
Define a protocol for the service methods and encapsulate it in a model class. An instance of this model class may be passed to the function/SP and responses retrieved.
Caveat: I have never done this myself but I think it will work.
Edit: Here is a diagram representing the thought. Though the diagram show H2 separately, it will most probably be in the same JVM as "Your Java/JEE application". However, since it is not necessary to use H2, I have shown it as as separate entity.

How to use prefix table names with jpa

I'm using java play to make a web application and I'm having some trouble persisting data using ebeans and jpa annotations.
I've searched a lot, and I'm a little confused, I think that this is something that should be common, I want to have tables named like this:
company 1_users
company 2_users
company N_users
but I want to have only one entity named user, can be possible to have this schema ?
I've searched about using one entity with multiple tables, about using table name prefixes and about using dynamic table name but I got nothing helpful.
Some ideas ??

As #GlennLane said in comment, creating separate tables isn't good idea, instead use some field as a company id/discriminator. You will save tons of nerves, and you won't need to update your code each time when new company joins the branch.
If you really want separate tables create separate model for each company, at least you won't mishmash that in code.

Creating tables dynamically in Hibernate environment

There's an enterprise application using Java + Hibernate + PostgreSQL. Hibernate is configured via annotations in the Java source code. So far the database schema is fixed, but I faced the problem that it needs to be dynamic:I can receive data from different locations and I have to store these in different tables. This means that I have to create tables run-time.
Fortunately, it seems that all of these data coming from the different institutes can have the same schema. But I still don't know how to do that using Hibernate. There are two main problems:
How to tell to Hibernate that many different tables have the same structure? For example the "Patient" class can be mapped to not just the "patient" table, but the "patient_mayo_clinic" table, "patient_northwestern" table, etc. I can feel that this causes ambiguity: how Hibernate knows which table to access when I do operations on the Patient class? It can be any (but only one) of the former listed tables.
How can I dynamically create tables with Hibernate and bind a class to them?
Response to suggestions:
Thanks for all of the suggestions. So far all of the answers discouraged the dynamic creation of tables. I'll mark Axel's answer, since it achieves certain goals, and it is a supported solution. More specifically it's called multi-tenancy. Sometimes it's important to know some important phrases which describes our problem (or part of our problem).
Here are some links about multi-tenancy:
Multi-tenancy in Hibernate
Hibernate Chapter 16. Multi-tenancy
Multi-tenancy Design
EclipseLink JPA multi-tenancy
In real world scenario multi-tenancy also involves the area of isolating the sets of data from each other (also in terms of access and authorization by different credentials) once they are shoved into one table.

You can't do this with Hibernate.
Why not extend your patient table with an institute column?
This way you'll be able to differentiate, without running into mapping issues.

I am afraid you can't do this easily in Hibernate. You would have to generate the Java source, compile it, add it to your classpath and load it dynamically with java.reflection package. If that works, which I doubt it, it will be an ugly solution (IMHO).
Have you consider using a schema less database i.e: NoSQL databases or RDF
databases. They are much more flexible in terms of what you can store in them , basically things are not tight up against a relational schema.

In most environments it is not a good idea to create tables dynamically simply because dbas will not give you the rights to create tables in production.
Axel's answer may be right for you. Also look into Inheritance Mapping for Hibernate.

I agree that its not advisable to create tables dynamically nevertheless it's doable.
Personally i would do as Axel Fontaine proposed but if dynamic tables is a must-have for you I would consider using Partitioning.
PostgreSQL allows you to create ona main table and few child tables (partitions), records are disjunctive between child tables, but every record from any child table is visible in parent table. This means that you can insert rows into any child table you want using just simple insert statement (its not cool but has the same level of complexity as composing and persisting an entity, so its acceptable in your case) and query database using HQL

Netbeans creating "JPA Controller classes from entity classes"

I want to achieve basic CRUD operations available of the db schema I already have for a JAVA program. To put it in another way, I have a DB schema I use with PHP and I just need them to be entities available in a JAVA application. I discovered I can use Netbeans and sucessfully created Entities from DB!
(Entities look like this: http://pastebin.com/f601b9218)
However when I try to create New > JPA controllers from entity classes in Netbeans I got empty JPA controller classes like:
package javaapplication3;
public class CustomerJpaController {
}
It is empty :) I was expecting CRUD functions inside the generated JPA controller classes as I read from examples and tutorials.
What could be the reason of empty JPA controller classes? Is there any other easy way for me to "just" match DB tables with JAVA classes for basic CRUD operation. (I wish there could be easy way to achieve active record pattern)
Thanks in advance.

Usually, I never like to go too much with the wizard thing. But it seems like it should generate the basic CRUD operations, though. I don't know exactly which tutorial you followed. Right now, I am looking at this. After reading this I am getting the same impression, by the way. But may be it just generates the empty classes, and nothing else. I am not sure never tried to do that.
However, coding it yourself would be quite simple, I believe. Especially when you already got the NamedQueries defined for your entities.