We are using the Hibernate #NamedQuery annotation on an Entity and are using join fetches to eagerly retrieve the data we want at query time. We have this working for the most part, but the FetchMode.SUBSELECT that we have on a relationship does not appear to be working. Instead, it appears that the query generated is still doing a single query with the join fetch, rather than the expected 2nd query.
Again, we're using Hibernate's #NamedQuery and not the JPA style.
The reason that we want to use the SUBSELECT for our case is that we have multiple #OneToMany relationships and we want to avoid massive perf issues with the Cartisean Product
Here's the setup we have:
#Entity
#NamedQueries(
#NamedQuery(
name = "StackOverflowNamedQuery",
query = "SELECT distinct f FROM Foo f "
+ "LEFT JOIN FETCH f.fooOneToMany bar " // This is a straight JOIN FETCH
+ "LEFT JOIN FETCH bar.barOneToMany " // WANT THIS TO DO A SUBSELECT
+ "LEFT JOIN FETCH f.manyToOne_boo "
+ "LEFT JOIN FETCH f.manyToOne_blah "
+ "WHERE f.id = :id "
+ "f.isRemoved = false"
)
)
Then over on Bar.getBarOneToMany(), here's the annotation setup we have for that OneToMany relationship
#OneToMany(fetch = FetchType.LAZY)
#JoinColumn(name = "entity_id", referencedColumnName = "id")
#Fetch(FetchMode.SUBSELECT)
public Set<BarOneToMany> getBarOneToMany() {
return barOneToMany;
}
So basically, we want to Fetch one of the OneToMany with a join fetch, and then we want the other one to be done by a SUBSELECT. Is this possible like this?
Thanks.
Related
I am running a real life scenario in our database and thus I have no room to alter its structure in case that this comes around as a suggestion. Here is the Structure in question:
EntityA {
#Id
.......
#OneToMany(mappedBy = "xxxx", fetch = FetchType.LAZY)
private Set<EntityB> entityB;
#Column(name = "partition", columnDefinition = "nchar", length = 3)
private String partitionKey;
}
EntityB {
#Id
..........
#Column(name = "partition")
private String partitionKey;
#ManyToOne(fetch = FetchType.LAZY) //EAGER is the default for the to-one
#JoinColumn(name = "bbbb")
private EntityA entityA;
#OneToMany(mappedBy = "EntityCPk.entityB", fetch = FetchType.LAZY)
private Set<EntityC> entityC;
#OneToOne(mappedBy = "cccc", cascade = CascadeType.ALL)
#PrimaryKeyJoinColumn
private EntityD entityD;
}
EntityC {
#EmbeddedId
private EntityCPk entityCPk;
#Embeddable
public static class EntityCPk implements Serializable {
#Column( name = "partition")
private String partitionKey;
#ManyToOne
#JoinColumn(name = "xxxx")
private EntityB entityB;
#ManyToOne
#JoinColumn(name = "xxxx")
private EntityE entityE;
}
}
EntityD {
#id
.........
#MapsId
#OneToOne
#JoinColumn(name = "zzzz", columnDefinition = "nchar")
#PrimaryKeyJoinColumn
private EntityB entityB;
}
EntityE {
#id
.........
#OneToMany(mappedBy = "yyyPk.entityE")
private Set<EntityC> entityC;
}
Now the requirement is to run a query in one go with joins and avoid 1+N scenarios. I assume as far as I have seen that the .LAZY or EAGER annotations are "overwritten" when using the Query annotation within the repository along with the FETCH option. So here is what I have achieved so much (entityXXX and entityYYY do not interfear with our case so I just mention them):
First Attempt with FetchType.LAZY in EntityB.entityC property:
"select a" +
"from EntityA a " +
"join FETCH a.entityB bs" +
"join FETCH bs.entityXXX as xxx " +
"join FETCH bs.entityYYY as yyy " +
"join FETCH bs.entityD latest " +
"where a.aProp in ( :props ) " +
"and xxx.id = 4 " +
"and yyy.id = 10" +
"and latest.entityB.aProp between :date and :date "
Results, as expected. I get 1 query BUT I get no collection returned in EntityB.entityC due to the lazy annotation and of course it is not present in the query. If I change the the EntityB.entityC to FetchType.EAGER then I get as expected 3 queries. One is the main and N per entityC in Set. So I guess the next step is to join entityC:
Second Attempt:
"select a " +
"from EntityA a " +
"join FETCH a.entityB bs" +
"join FETCH bs.entityXXX as xxx " +
"join FETCH bs.entityYYY as yyy " +
"join FETCH bs.entityD as latest " +
"join bs.entityC as cs " + //please note I am not using FETCH yet
"where a.aProp in ( :props ) " +
"and c.entityCPk.partitionKey = a.partitionKey " +
"and xxx.id = 4 " +
"and yyy.id = 10" +
"and latest.entityB.aProp between :date and :date "
The result is unexpected and I think it has been reported here as well. What I get now is multiples of a(s) all references to the same object equals to the sum of the amount of bs.entityC. So if for example a-1 -> has 1-bs -> has 17 cs and a2 -> has 1-bs -> has 67 cs then I end up with a result set of 84 a objects all the same! This is question one. Why is this happening?
Question 2 is that if I use the FETCH in my new join then I am still not getting my 1 query and now I am not getting exactly multiple instances of A but multiple instances of some kind of Wrappers with a handler property that has references to A makred as EntityA_$$_jvstbc0_4#.
Just to give some insight to the database structure, I am more than sure that this schema started as a many-to-many relationship with a lookup table being EntityB between EntityA and EntityC. I may try to tackle the issue using JoinTable on EntityC joining on partitionKey and id of EntityB while EntityA has the partitionkey and its Id to map on EntityB. However i am not very hopeful of this solution as EntityB has been contaminated with other columns over time which needs to be selected uppon and I am not sure how can I do this.
UPDATE 1: I can see that when join FETCH is used for cs it is augmenting the resultant SQL select with the columns that are necessary i.e. to populate the cs children. Running the query manually I am getting correctly the sum of children as rows. Which makes sense SQL wise but hibernate should have been able to aggregate the additional rows based on their properties. Right enough without the join FETCH I am getting only rows equals to the amount of a. So my second though is that somehow I need to instruct Hibernate to aggregate manually(?)
UPDATE 2: Change of strategy. Instead of starting following an SQL logic, we better have to answer to the following question: Which Class/Entity will give us the granularity we are looking for. In the previous examples we were starting from EntityA trying to limit its children to fit our expected results. However as it has been pointed, this is effectively a corruption of the objects. You cannot "limit" the children cause they all belong to the Entity and fetching a subset of them you run the risk of deleting(?) data. So the approach must be to get the children objects we are interested that point to the parent entities. That we don't alter the data. So here is a query that returns the correct amount of object. No distinct or inexplicable multiplicities:
"select c " +
"from EntityC c " +
"inner join c.EntityCPk.EntityB.EntityD latest " +
"join latest.EntityB.EntityXXX xxx " +
"join latest.EntityB.EntityYYY yyy " +
"join fetch c.EntityCPk.EntityB " +
"where latest.EntityB.EntityA.Id in ( :param ) " +
"and latest.EntityB.aField between :paramA and paramB "
So this seems to answer the issue of the multiplicity of the previous examples as every row is based on the "finer" child object that resolves its parent via the -ToOne relationship. Also there are no more dangerous aliases in the join fetch. There is only one more issue. It introduces a 1+N query problem for EntityB that I cannot get rid off.
When trying to optimize MySQ slow queries generated by Hibernate 4.2 in a legacy project, I found out that the code below generates nearly 500 SQL queries (with many duplicates) :
class MyDAO {
public List<Message> findMessages() {
Session session = MyHibernateUtils.openSession();
String queryStr = "SELECT DISTINCT m FROM Message m "
+ " LEFT JOIN fetch m.types types "
+ " LEFT JOIN fetch m.mainType mainType "
+ " LEFT JOIN fetch m.place place "
+ " LEFT JOIN fetch m.building building "
+ " LEFT JOIN fetch m.city city "
+ " LEFT JOIN fetch m.kind kind "
+ " LEFT JOIN fetch m.domain domain "
+ " LEFT JOIN fetch m.action action "
+ " LEFT JOIN fetch m.customParameterA customParameterA "
+ " LEFT JOIN fetch m.customParameterB customParameterB "
+ " LEFT JOIN fetch m.scheduleEvents scheduleEvents "
+ " LEFT JOIN fetch m.comments comments "
+ " LEFT JOIN fetch m.messageLastActivities messageLastActivities "
+ " LEFT JOIN fetch m.customListA customListA "
+ " LEFT JOIN fetch m.childEvents childEvents "
+ " LEFT JOIN fetch m.parentEvent parentEvent "
+ " WHERE ...";
List<Message> messages;
try {
session.getTransaction().begin();
Query query = session.createQuery(queryStr);
query.setTimeout(10);
messages = query.list();
session.getTransaction().commit();
} catch (RuntimeException e) {
session.getTransaction().rollback();
throw e;
} finally {
session.close();
}
return messages;
}
}
How can I avoid having so many SQL queries ?
I don't know if it helps but there are many onyToMany and manyToMany relationships between the entities.
Thank for your help.
You should check the queries hibernate is generating, to see which table is accessed frequently.
You have to join fetch entities related by your related entities as well, See here:
Hibernate is doing multiple select requests instead one (using join fetch)
I personally prefer lazy loading with an annotated #BatchSize() to keep the lazy-query-count small. Just using a Batch-Size of 2 will cut your query count in half then.
Also have a look at the #Cache Annotation which can reduce your query count in a significant way. (Just thinking of all the almost static stuff like city/building/type/domain and the like)
Depending on your relationship design, default value of Fetch in #OneToMany and #ManyToMany is LAZY, that means for loading related record in child entity (when you call getter method) hibernate executes one more query to load that record (for example: select * from foo where id = ?) , so if loaded entity (main entity) contains many child entities such as ManyToMany or OneToMany you will see many queries in console.
To void these queries, you can set Fetch to EAGER but this is not recommended on optimization.
#Entity
public class MainEntity {
#ManyToMany(Fetch = FetchType.EAGER)
public List<Foo> foos;
}
I have two entities in MySQL as below. The primary key of nnm_tran is a composite of id and source. The primary key of bargains is actually a foreign key link to the nnm_tran table
I'm trying to use JPA inheritance to represent these.
nnm_tran entity
#Entity
#Table(name = "nnm_tran")
#IdClass(CommonTransactionKey.class)
#Inheritance(strategy = InheritanceType.JOINED)
#DiscriminatorColumn(name = "bargain_flag", discriminatorType = DiscriminatorType.CHAR)
#DiscriminatorValue("N")
public class CommonTransaction {
#Id
#Column(name = "id", nullable = false)
private String transactionId;
#Column(name = "plan_number", nullable = false)
private String planNumber;
#Column(name = "tran_date")
private LocalDateTime transactionDatetime;
#Column(name = "bargain_flag")
private String bargainFlag;
...
}
bargains entity
#Data
#EqualsAndHashCode(callSuper = true)
#Entity
#Table(name = "bargains")
#DiscriminatorValue("B")
#PrimaryKeyJoinColumns({ #PrimaryKeyJoinColumn(name = "nnm_tran_id", referencedColumnName = "id"), #PrimaryKeyJoinColumn(name = "nnm_tran_source", referencedColumnName = "source") })
public class Bargain extends CommonTransaction implements Serializable {
#Column(name = "unit_price")
private BigDecimal unitPrice;
#Column(name = "client_price")
private BigDecimal clientPrice;
...
}
I think so far this is all hooked up correctly. My problem comes when I attach a spring-data repository with a custom query.
Repository
public interface CommonTransactionRepository extends CrudRepository<CommonTransaction, CommonTransactionKey> {
#Query("select t from CommonTransaction t left join IoPlan p ON t.planNumber = p.planNumber "
+ "where (p.planNumber is NULL or p.planNumber = '') "
+ "and t.transactionDatetime between ?1 and ?2 "
+ "and t.cancelled = false")
public Iterable<CommonTransaction> findOrphanedTransactionsByTranDate(LocalDateTime fromDate, LocalDateTime toDate);
...
}
When this gets proxied and the method is executed it generates the SQL statement
SELECT DISTINCT nnm_tran.bargain_flag FROM nnm_tran t1 LEFT OUTER JOIN io_plan t0 ON (t1.plan_number = t0.plan_number) WHERE ((((t0.plan_number IS NULL) OR (t0.plan_number = ?)) AND (t1.tran_date BETWEEN ? AND ?)) AND (t1.CANCELLED = ?))
The issue with this is that the nnm_tran table is aliased to t1 but the discriminator column is referencing the full table name nnm_tran.bargain_flag The result is a lovely
UnitOfWork(17171249)--Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column 'nnm_tran.bargain_flag' in 'field list'
Question here is, am I doing something wrong or is this a bug in spring-data and/or eclipselink?
Versions: spring-data 1.7.2, Eclipselink 2.5.2, MySQL 5.6.28
Using #manish's sample app as a starting point I started layering back on the complexity that was missing and quickly stumbled across the thing causing the rogue SQL. It was down to the join I had performed in the JPQL
NOTE: If you've come here from the future then ignore the remainder of this answer and instead use #Chris's comment instead.
Most of the time I don't need to look at or even think about the IoPlan table that can be seen in the #Query
#Query("select t from CommonTransaction t left join IoPlan p ON t.planNumber = p.planNumber "
+ "where (p.planNumber is NULL or p.planNumber = '') "
+ "and t.transactionDatetime between ?1 and ?2 "
+ "and t.cancelled = false")
and so this table is not a part of the CommonTransaction entity as a field. Even the result of this query doesn't really care because it's looking only as a one off for CommonTransaction with no associated join in the IoPlan table.
When I added the join back in to the sample app from #manish it all broke in the same way my app has in EclipseLink, but broke in a different way for Hibernate. Hibernate requires a field for you to join with, which if you ask me defeats the purpose of writing the join in the #Query. In fact in Hibernate you have to define the join purely in JPA so you might as well then use dot notation to access it in the JPQL.
Anyway, going along with this idea I tried adding a dummy field to hold an IoPlan in my CommonTransaction entity and it almost worked. It defaulted some of the join logic but it was closer
SELECT DISTINCT t1.bargain_flag FROM nnm_tran t1 LEFT OUTER JOIN io_plan t0 ON ((t0.ID = t1.IOPLAN_ID) AND (t1.plan_number = t0.plan_number)) WHERE ((((t0.plan_number IS NULL) OR (t0.plan_number = ?)) AND (t1.tran_date BETWEEN ? AND ?)) AND (t1.CANCELLED = ?))
In this case t1.IOPLAN_ID and t0.ID don't exist. So I ended up defining the entire join in my CommonTransaction entity
#OneToOne
#JoinColumn(insertable = false, updatable = false, name = "plan_number", referencedColumnName = "plan_number")
private IoPlan ioPlan;
and voila, it started working. It's not pretty and now I have a redundant join condition
LEFT OUTER JOIN io_plan t1
ON ((t1.plan_number = t0.plan_number) AND (t0.plan_number = t1.plan_number))
but I can fix that. It's still annoying that I have to define a field for it whatsoever, I don't actually want or need it there, not to mention that the result from this query is returning CommonTransaction entities that have no IoPlan so the field will be permanently null.
I have 3 classes, I am trying to get a list of all the events of an eventhost that a user is subscribed to. I am probably thinking way too complicated but I have very little experience with JPA/HQL.
User class
#ManyToMany
#JoinTable(name = "Subscriptions", joinColumns = #JoinColumn(name = "user_id", referencedColumnName = "id") , inverseJoinColumns = #JoinColumn(name = "event_host_id", referencedColumnName = "id") )
private List<EventHost> subscriptions;
EventHost class
#OneToMany(cascade = CascadeType.ALL, fetch = FetchType.LAZY)
#JoinColumn(name = "event_host_id", referencedColumnName = "id", updatable = true)
private List<Event> events;
I tried using this query, but it tells me that subscriptions is not mapped, which it is not since it's not a java class.
String hql = "SELECT o FROM Event WHERE event_host_id IN (SELECT a FROM EventHost WHERE id IN(SELECT b FROM User WHERE = + " + userid + "))";
I know injecting the userid like this is bad practice, I'm just doing it for testing purposes.
Please ask if you need something more, I would really like to understand how to write a query for this.
This question should really be HQL with two join tables, but I'll let you change it. Since its HQL, or JPA, it's database independent.
Anyway, any time you see a OneToMany or ManyToMany relationship you have a join table and so you should be thinking joins. It's always a good idea to look at the sql create table statements to see what's going on. In this case your user_subscriptions join table is:
create table user_subscriptions (user_id integer not null, subscriptions_id integer not null)
and your event_host_events join table is this:
create table event_host_events (event_host_id integer not null, events_id integer not null)
Nothing new there. When you're trying to get something new working that you don't intuitively understand, break it down into things you can do. For example, you can execute two queries, getting a Users subscriptions first, and then getting the Events for those subscriptions:
Query query = session.createQuery("select u.subscriptions from User u where name = :name");
query.setParameter("name", name);
List<EventHost> subscriptions = query.list();
List<Event> events = new ArrayList<Event>();
Query query2 = session.createQuery("select s.events from EventHost s where id = :id");
for (EventHost s: subscriptions ) {
query2.setParameter("id", s.getId());
events.addAll( query2.list());
}
Not elegant, but it works. Then, keeping join in mind, figure out how to make one statement out of the two of them.
Query query = session.createQuery("select s.events from User u join u.subscriptions s where u.name = :name)");
query.setParameter("name", name);
return query.list();
The join will use an inner join by default, so you're ok there. The JPA provider will auto-magically join your three Entity tables and two Join Tables for you:
select
event4_.id as id1_2_
from user user0_
inner join user_subscriptions subscripti1_ on user0_.id=subscripti1_.user_id
inner join event_host eventhost2_ on subscripti1_.subscriptions_id=eventhost2_.id
inner join event_host_events events3_ on eventhost2_.id=events3_.event_host_id
inner join event event4_ on events3_.events_id=event4_.id
where user0_.name=?
Aren't you glad you don't have to write that query?
I have a JPQL like this one:
select distinct d
from Department d
left join fetch d.employees
When I want to fetch one of the lazy property of my Department entity, the distinct is not working any more.
select distinct d, substring(d.htmlDescription, 1,400)
from Department d
left join fetch d.employees
The query returns as much Department as the number of employees in it.
The substring(d.htmlDescription) is important because the property is defined as a CLOB (type TEXT under postgresql):
#Column(columnDefinition = "TEXT")
#Basic(fetch = FetchType.LAZY)
String htmlBody;
The substring function is translated in sql thus limiting the amount of data transfered beetween the database and the web server.
As a workaround, I tried to break the query in two parts :
select d, substring(d.htmlDescription, 1,400)
from Department d where d in (
select distinct d1
from Department d1 left join fetch d1.employees
)
This doestn't work because the JOIN FETCH must not be used in the FROM clause of a subquery.
Finally I found a solution to my problem by :
modifying my mapping
cutting the request in 2 calls.
The htmlBody field is now in another entity. Thus the departement entity is lighter.
class Department{
...
#OneToOne (fetch = FetchType.LAZY,
cascade = {CascadeType.PERSIST, CascadeType.MERGE, CascadeType.REMOVE})
Content content = new Content();
...
}
class Content{
...
#Column(columnDefinition = "TEXT")
#Basic(fetch = FetchType.LAZY)
String htmlBody;
...
}
I can then use the following requests :
List<Department> deps = em.get().createQuery(
"select distinct d " +
"from Department d " +
"order by d.id desc ", Department.class)
.setFirstResult(first)
.setMaxResults(count)
.getResultList();
List<Object[]> tuple = em.get().createQuery(
"select d, substring(d.content.htmlBody, 1,400)" +
"from Department d " +
"left join fetch d.employees" +
"where d in (:deps) order by d.id desc")
.setParameter("deps", deps)
.getResultList();
... //Filter the duplicates due to the fetching
That way, I have 2 sql queries. The fetching of employees is done in the second query witch occurs on a small amount of datas. The substring is realized in SQL. Perfect!
Since I cannot make comments, I would like to point out few things that stick out to me as doubtfull.
What is the object returned with distinct d, substring(d.htmlDescription, 1,400)? Could you fetch that String with separate query, or get that substing using Java?
I would trust that that query can be rewritten into one without left join statement.
Maybe you could rewrite the query so you could put substring statement first and then distinct d?