When trying to optimize MySQ slow queries generated by Hibernate 4.2 in a legacy project, I found out that the code below generates nearly 500 SQL queries (with many duplicates) :
class MyDAO {
public List<Message> findMessages() {
Session session = MyHibernateUtils.openSession();
String queryStr = "SELECT DISTINCT m FROM Message m "
+ " LEFT JOIN fetch m.types types "
+ " LEFT JOIN fetch m.mainType mainType "
+ " LEFT JOIN fetch m.place place "
+ " LEFT JOIN fetch m.building building "
+ " LEFT JOIN fetch m.city city "
+ " LEFT JOIN fetch m.kind kind "
+ " LEFT JOIN fetch m.domain domain "
+ " LEFT JOIN fetch m.action action "
+ " LEFT JOIN fetch m.customParameterA customParameterA "
+ " LEFT JOIN fetch m.customParameterB customParameterB "
+ " LEFT JOIN fetch m.scheduleEvents scheduleEvents "
+ " LEFT JOIN fetch m.comments comments "
+ " LEFT JOIN fetch m.messageLastActivities messageLastActivities "
+ " LEFT JOIN fetch m.customListA customListA "
+ " LEFT JOIN fetch m.childEvents childEvents "
+ " LEFT JOIN fetch m.parentEvent parentEvent "
+ " WHERE ...";
List<Message> messages;
try {
session.getTransaction().begin();
Query query = session.createQuery(queryStr);
query.setTimeout(10);
messages = query.list();
session.getTransaction().commit();
} catch (RuntimeException e) {
session.getTransaction().rollback();
throw e;
} finally {
session.close();
}
return messages;
}
}
How can I avoid having so many SQL queries ?
I don't know if it helps but there are many onyToMany and manyToMany relationships between the entities.
Thank for your help.
You should check the queries hibernate is generating, to see which table is accessed frequently.
You have to join fetch entities related by your related entities as well, See here:
Hibernate is doing multiple select requests instead one (using join fetch)
I personally prefer lazy loading with an annotated #BatchSize() to keep the lazy-query-count small. Just using a Batch-Size of 2 will cut your query count in half then.
Also have a look at the #Cache Annotation which can reduce your query count in a significant way. (Just thinking of all the almost static stuff like city/building/type/domain and the like)
Depending on your relationship design, default value of Fetch in #OneToMany and #ManyToMany is LAZY, that means for loading related record in child entity (when you call getter method) hibernate executes one more query to load that record (for example: select * from foo where id = ?) , so if loaded entity (main entity) contains many child entities such as ManyToMany or OneToMany you will see many queries in console.
To void these queries, you can set Fetch to EAGER but this is not recommended on optimization.
#Entity
public class MainEntity {
#ManyToMany(Fetch = FetchType.EAGER)
public List<Foo> foos;
}
Related
I am using hibernate for this select:
NativeQuery sqlQuery = session.createSQLQuery("select :id, min(a.time) " +
"from table1 a, (" +
" select parentid" +
" from (" +
" select LEVEL, parentid" +
" from table2 " +
" START WITH id = :id" +
" CONNECT BY NOCYCLE PRIOR parentid = id" +
" order by level desc)" +
" where rownum = 1" +
" ) b " +
"where a.id = b.parentid");
sqlQuery.setParameter("id", id);
List<Object[]> list = sqlQuery.list();
I need to use this for a lot of ids. Ideally I would pass a list of ids, run that in oracle and return the result. I cannot do that because of the START WITH clause.
This way I send a query to database for each id, which is really slow.
Is there any way to put a list of ids to the query, do this in loop and return back to the application with a list of results?
NativeQuery sqlQuery = session.createSQLQuery("...?...");
sqlQuery.setListParameter("ids", ids);
List<Object[]> list = sqlQuery.list();
Edit: I cannot use recursive with, because we use Oracle 10.
I don't know about the Oracle specific syntax, but with the SQL standard WITH RECURSIVE syntax, you can of course list multiple parameters. Just use the IN predicate e.g. for batches of 5 id IN (:id1, :id2, :id3, :id4, :id5) and then bind the values with setParameter("id1", ...). If you are interested, you can also formulate this with Blaze-Persistence on top of Hibernate by using the JPA model: https://persistence.blazebit.com/documentation/1.6/core/manual/en_US/#recursive-ctes
In your case this would look like the following:
#CTE
#Entity
public class ResultCte {
#Id
Long parentId;
Long rootId;
int level;
}
CriteriaBuilder<Tuple> cb = criteriaBuilderFactory.create(entityManager, Tuple.class);
cb.withRecursive(ResultCte.class)
.from(Entity2.class, "e2")
.where("e2.id").in(idList)
.bind("parentId").select("e2.parent.id")
.bind("rootId").select("e2.id")
.bind("level").select("1")
.unionAll()
.from(Entity2.class, "e2")
.from(ResultCte.class, "cte")
.where("e2.id").eqExpression("cte.parentId")
.bind("parentId").select("e2.parent.id")
.bind("rootId").select("cte.rootId")
.bind("level").select("cte.level + 1")
.end()
.from(Entity1.class "a")
.from(ResultCte.class, "cte")
.where("a.id").eqExpression("cte.parentId")
.select("cte.rootId")
.select("min(a.time)")
I'm getting a warning in the Server log "firstResult/maxResults specified with collection fetch; applying in memory!". However everything working fine. But I don't want this warning.
My code is
public employee find(int id) {
return (employee) getEntityManager().createQuery(QUERY).setParameter("id", id).getSingleResult();
}
My query is
QUERY = "from employee as emp left join fetch emp.salary left join fetch emp.department where emp.id = :id"
Although you are getting valid results, the SQL query fetches all data and it's not as efficient as it should.
So, you have two options.
Fixing the issue with two SQL queries that can fetch entities in read-write mode
The easiest way to fix this issue is to execute two queries:
. The first query will fetch the root entity identifiers matching the provided filtering criteria.
. The second query will use the previously extracted root entity identifiers to fetch the parent and the child entities.
This approach is very easy to implement and looks as follows:
List<Long> postIds = entityManager
.createQuery(
"select p.id " +
"from Post p " +
"where p.title like :titlePattern " +
"order by p.createdOn", Long.class)
.setParameter(
"titlePattern",
"High-Performance Java Persistence %"
)
.setMaxResults(5)
.getResultList();
List<Post> posts = entityManager
.createQuery(
"select distinct p " +
"from Post p " +
"left join fetch p.comments " +
"where p.id in (:postIds) " +
"order by p.createdOn", Post.class)
.setParameter("postIds", postIds)
.setHint(
"hibernate.query.passDistinctThrough",
false
)
.getResultList();
Fixing the issue with one SQL query that can only fetch entities in read-only mode
The second approach is to use SDENSE_RANK over the result set of parent and child entities that match our filtering criteria and restrict the output for the first N post entries only.
The SQL query can look as follows:
#NamedNativeQuery(
name = "PostWithCommentByRank",
query =
"SELECT * " +
"FROM ( " +
" SELECT *, dense_rank() OVER (ORDER BY \"p.created_on\", \"p.id\") rank " +
" FROM ( " +
" SELECT p.id AS \"p.id\", " +
" p.created_on AS \"p.created_on\", " +
" p.title AS \"p.title\", " +
" pc.id as \"pc.id\", " +
" pc.created_on AS \"pc.created_on\", " +
" pc.review AS \"pc.review\", " +
" pc.post_id AS \"pc.post_id\" " +
" FROM post p " +
" LEFT JOIN post_comment pc ON p.id = pc.post_id " +
" WHERE p.title LIKE :titlePattern " +
" ORDER BY p.created_on " +
" ) p_pc " +
") p_pc_r " +
"WHERE p_pc_r.rank <= :rank ",
resultSetMapping = "PostWithCommentByRankMapping"
)
#SqlResultSetMapping(
name = "PostWithCommentByRankMapping",
entities = {
#EntityResult(
entityClass = Post.class,
fields = {
#FieldResult(name = "id", column = "p.id"),
#FieldResult(name = "createdOn", column = "p.created_on"),
#FieldResult(name = "title", column = "p.title"),
}
),
#EntityResult(
entityClass = PostComment.class,
fields = {
#FieldResult(name = "id", column = "pc.id"),
#FieldResult(name = "createdOn", column = "pc.created_on"),
#FieldResult(name = "review", column = "pc.review"),
#FieldResult(name = "post", column = "pc.post_id"),
}
)
}
)
The #NamedNativeQuery fetches all Post entities matching the provided title along with their associated PostComment child entities. The DENSE_RANK Window Function is used to assign the rank for each Post and PostComment joined record so that we can later filter just the amount of Post records we are interested in fetching.
The SqlResultSetMapping provides the mapping between the SQL-level column aliases and the JPA entity properties that need to be populated.
Now, we can execute the PostWithCommentByRank #NamedNativeQuery like this:
List<Post> posts = entityManager
.createNamedQuery("PostWithCommentByRank")
.setParameter(
"titlePattern",
"High-Performance Java Persistence %"
)
.setParameter(
"rank",
5
)
.unwrap(NativeQuery.class)
.setResultTransformer(
new DistinctPostResultTransformer(entityManager)
)
.getResultList();
Now, by default, a native SQL query like the PostWithCommentByRank one would fetch the Post and the PostComment in the same JDBC row, so we will end up with an Object[] containing both entities.
However, we want to transform the tabular Object[] array into a tree of parent-child entities, and for this reason, we need to use the Hibernate ResultTransformer.
The DistinctPostResultTransformer looks as follows:
public class DistinctPostResultTransformer
extends BasicTransformerAdapter {
private final EntityManager entityManager;
public DistinctPostResultTransformer(
EntityManager entityManager) {
this.entityManager = entityManager;
}
#Override
public List transformList(
List list) {
Map<Serializable, Identifiable> identifiableMap =
new LinkedHashMap<>(list.size());
for (Object entityArray : list) {
if (Object[].class.isAssignableFrom(entityArray.getClass())) {
Post post = null;
PostComment comment = null;
Object[] tuples = (Object[]) entityArray;
for (Object tuple : tuples) {
if(tuple instanceof Identifiable) {
entityManager.detach(tuple);
if (tuple instanceof Post) {
post = (Post) tuple;
}
else if (tuple instanceof PostComment) {
comment = (PostComment) tuple;
}
else {
throw new UnsupportedOperationException(
"Tuple " + tuple.getClass() + " is not supported!"
);
}
}
}
if (post != null) {
if (!identifiableMap.containsKey(post.getId())) {
identifiableMap.put(post.getId(), post);
post.setComments(new ArrayList<>());
}
if (comment != null) {
post.addComment(comment);
}
}
}
}
return new ArrayList<>(identifiableMap.values());
}
}
The DistinctPostResultTransformer must detach the entities being fetched because we are overwriting the child collection and we don’t want that to be propagated as an entity state transition:
post.setComments(new ArrayList<>());
Reason for this warning is that when fetch join is used, order in result sets is defined only by ID of selected entity (and not by join fetched).
If this sorting in memory is causing problems, do not use firsResult/maxResults with JOIN FETCH.
To avoid this WARNING you have to change the call getSingleResult to
getResultList().get(0)
This warning tells you Hibernate is performing in memory java pagination. This can cause high JVM memory consumption.
Since a developer can miss this warning, I contributed to Hibernate by adding a flag allowing to throw an exception instead of logging the warning (https://hibernate.atlassian.net/browse/HHH-9965).
The flag is hibernate.query.fail_on_pagination_over_collection_fetch.
I recommend everyone to enable it.
The flag is defined in org.hibernate.cfg.AvailableSettings :
/**
* Raises an exception when in-memory pagination over collection fetch is about to be performed.
* Disabled by default. Set to true to enable.
*
* #since 5.2.13
*/
String FAIL_ON_PAGINATION_OVER_COLLECTION_FETCH = "hibernate.query.fail_on_pagination_over_collection_fetch";
the problem is you will get cartesian product doing JOIN. The offset will cut your recordset without looking if you are still on same root identity class
I guess the emp has many departments which is a One to Many relationship. Hibernate will fetch many rows for this query with fetched department records. So the order of result set can not be decided until it has really fetch the results to the memory. So the pagination will be done in memory.
If you do not want to fetch the departments with emp, but still want to do some query based on the department, you can achieve the result with out warning (without doing ordering in the memory). For that simply you have to remove the "fetch" clause. So something like as follows:
QUERY = "from employee as emp left join emp.salary sal left join emp.department dep where emp.id = :id and dep.name = 'testing' and sal.salary > 5000 "
As others pointed out, you should generally avoid using "JOIN FETCH" and firstResult/maxResults together.
If your query requires it, you can use .stream() to eliminate warning and avoid potential OOM exception.
try (Stream<ENTITY> stream = em.createQuery(QUERY).stream()) {
ENTITY first = stream.findFirst().orElse(null); // equivalents .getSingleResult()
}
// Stream returned is an IO stream that needs to be closed manually.
We are using the Hibernate #NamedQuery annotation on an Entity and are using join fetches to eagerly retrieve the data we want at query time. We have this working for the most part, but the FetchMode.SUBSELECT that we have on a relationship does not appear to be working. Instead, it appears that the query generated is still doing a single query with the join fetch, rather than the expected 2nd query.
Again, we're using Hibernate's #NamedQuery and not the JPA style.
The reason that we want to use the SUBSELECT for our case is that we have multiple #OneToMany relationships and we want to avoid massive perf issues with the Cartisean Product
Here's the setup we have:
#Entity
#NamedQueries(
#NamedQuery(
name = "StackOverflowNamedQuery",
query = "SELECT distinct f FROM Foo f "
+ "LEFT JOIN FETCH f.fooOneToMany bar " // This is a straight JOIN FETCH
+ "LEFT JOIN FETCH bar.barOneToMany " // WANT THIS TO DO A SUBSELECT
+ "LEFT JOIN FETCH f.manyToOne_boo "
+ "LEFT JOIN FETCH f.manyToOne_blah "
+ "WHERE f.id = :id "
+ "f.isRemoved = false"
)
)
Then over on Bar.getBarOneToMany(), here's the annotation setup we have for that OneToMany relationship
#OneToMany(fetch = FetchType.LAZY)
#JoinColumn(name = "entity_id", referencedColumnName = "id")
#Fetch(FetchMode.SUBSELECT)
public Set<BarOneToMany> getBarOneToMany() {
return barOneToMany;
}
So basically, we want to Fetch one of the OneToMany with a join fetch, and then we want the other one to be done by a SUBSELECT. Is this possible like this?
Thanks.
I'm getting this error and I can't find any reference to it:
org.hibernate.hql.internal.ast.QuerySyntaxException: expecting "all", found '(' near line 1, column 221 [select new EffectivePermissions(r.id, r.name, r.defaultValue, rc.value AS companyValue, ru.value AS userValue) from permissionsPackage.Entity.Permissions r left join permissionsPackage.Entity.CompanyPermissions rc fetch (r.id=rc.permissionId AND rc.companyId=2313 ) left join permissionsPackage.Entity.UserPermissions ru fetch (r.id=ru.permissionId AND ru.userId=1)]
at org.hibernate.hql.internal.ast.QuerySyntaxException.convert(QuerySyntaxException.java:54)
at org.hibernate.hql.internal.ast.QuerySyntaxException.convert(QuerySyntaxException.java:47)
at org.hibernate.hql.internal.ast.ErrorCounter.throwQueryException(ErrorCounter.java:79)
Below is the Query I'm using.
this.hql = "select new EffectivePermissions(r.id, r.name, r.defaultValue, rc.value AS companyValue, ru.value AS userValue) "
+ "from "
+ Permissions.class.getName()
+ " r "
+ "left join "
+ CompanyPermissions.class.getName()
+ " rc "
+ "fetch (r.id=rc.permissionId AND rc.companyId="
+ user.getCompany().getId()
+ " ) "
+ "left join "
+ UserPermissions.class.getName()
+ " ru "
+ "fetch (r.id=ru.permissionId AND ru.userId="
+ user.getId()
+ ")";
Query query = sessionFactory.getCurrentSession().createQuery(hql);
permissions = query.list();
I have a class called EffectivePermission with the properties in the constructor and I'd like to get a list based on it.
From the Hibernate Reference chapter 14.3. Associations and joins:
A "fetch" join allows associations or collections of values to be initialized along with their parent objects using a single select ... See Section 19.1, “Fetching strategies” for more information.
You'll also see there that it must be used after the joinkeyword as in from e1 left join fetch e2. It can also be used as in from entity fetch all properties (this is why Hibernate is expecting the all keyword).
Regarding your specific query, and as far as I know, with HQL you can not specify the join conditions. Hibernate will automatically perform the join using the conditions configured in the mapping. This is why it is necessary to have relationships mapped in order to use a HQL join.
And most importantly, remember that joins in HQL are quite different. You don't join two entities, you join an entity with one of its collection-valued properties (associations). Notice you can always do cartesian-like joins (pay attention to performance and always look at the generated queries and execution plans).
As a side note, it can be considered bad practice to append parameter values directly into the query (due to potential injection vulnerabilities). You should be using named parameters instead.
I have two tables - one containing Address and another containing Photographs. The only common field between them is the PersonID. These were mapped to two POJO Classes Address and Photo. I was able to fetch details in these tables by creating criteria and adding restrictions on the fields . How should we write a join on the two tables. Is it possible to get the result as two objects -Address and Photo.
I want to do a left join so that i can get records of persons without photos as well.
I have read that this is possible only using hql but Can this be done using criterias as well?
You can easily write HQL query that will return result as two objects using Theta Join (as Adrian noted). Here is an example:
String queryText = "select address, photo from Address address, Photo photo "
+ " where address.personID=photo.personId";
List<Object[]> rows = session.createQuery(queryText).list();
for (Object[] row: rows) {
System.out.println(" ------- ");
System.out.println("Address object: " + row[0]);
System.out.println("Photo object: " + row[1]);
}
As you can see query returns list of Object[] arrays that represents each fetched row. First element of this array will contain one obejct and second element - another.
EDIT:
In case of left join I think you need to use native SQL query (not HQL query). Here how you can do this:
String queryText = "select address.*, photo.* from ADDRESS address
left join PHOTO photo on (address.person_id=photo.person_id)";
List<Object[]> rows = sess.createSQLQuery(queryText)
.addEntity("address", Address.class)
.addEntity("photo", Photo.class)
.list();
This should work for your case.
Basically, you have two options:
Since Hibernate 5.1, you can use ad-hoc joins for unrelated entities.
Tuple postViewCount = entityManager.createQuery(
"select p as post, count(pv) as page_views " +
"from Post p " +
"left join PageView pv on p.slug = pv.slug " +
"where p.title = :title " +
"group by p", Tuple.class)
.setParameter("title", "High-Performance Java Persistence")
.getSingleResult();
Prior to Hibernate 5.1, you could only use theta-style joins. However, a theta-style join is equivalent to an equijoin, hence you can only emulate INNER JOINs not OUTER JOINs.
List<Tuple> postViewCount = entityManager.createQuery(
"select p as post, count(pv) as page_views " +
"from Post p, PageView pv " +
"where p.title = :title and " +
" ( p.slug = pv.slug ) " +
"group by p", Tuple.class)
.setParameter("title", "Presentations")
.getResultList();
Finally after 12 years the Hibernate team has implemented such a feature
From Hibernate docs:
The FROM clause can also contain explicit relationship joins using the join keyword. These joins can be either inner or left outer style joins.
List<Person> persons = entityManager.createQuery(
"select distinct pr " +
"from Person pr " +
"join pr.phones ph " +
"where ph.type = :phoneType", Person.class )
.setParameter( "phoneType", PhoneType.MOBILE )
.getResultList();
List<Person> persons = entityManager.createQuery(
"select distinct pr " +
"from Person pr " +
"left join pr.phones ph " +
"where ph is null " +
" or ph.type = :phoneType", Person.class )
.setParameter( "phoneType", PhoneType.LAND_LINE )
.getResultList();
Or you can use WITH and ON keywords. A remark on those
The important distinction is that in the generated SQL the conditions
of the WITH/ON clause are made part of the ON clause in the generated
SQL, as opposed to the other queries in this section where the
HQL/JPQL conditions are made part of the WHERE clause in the generated
SQL.
Example
List<Object[]> personsAndPhones = session.createQuery(
"select pr.name, ph.number " +
"from Person pr " +
"left join pr.phones ph with ph.type = :phoneType " )
.setParameter( "phoneType", PhoneType.LAND_LINE )
.list();
I am currently eager to try the new feature.
Joining two unrelated entities are possible in Hibernate 5.1.
Eg :
select objA from ObjectA objA
JOIN ObjectB objB on objB.variable = objA.variable
where objA.id = 1
It's best to have a class containing those classes you want to join to have them all together.
But if you are joining these tables just for some occasional purposes, you can use criteria and manually load data from each table and put them together. (and yes, you can have these tables' data separately if for Address and Photo there are two separate classes and tables)
What you are looking for is
HQL
Join on fields that you haven't modeled as relationships
Left Join
(During the time the question was first asked and this answer was given) Hibernate supports Theta Join which allows you to do 1 & 2. However, only inner join is available for theta join style.
Personally I would recommend you to model proper relationships, so you just need 1 & 3 which is well-supported in HQL.
(Another answer actually provided an update on new Hibernate feature that provides such feature, that you may simply refer to)