We Keep Coding

How to Improve performance of iteration loops

I have a java method which finds all children (leaves) of a parent. It works ok.but there is performance issue. How can i make it work faster? When the children gets large it becomes problematic. Here is the method:
Private void getChildren(String orgId, List<String> children, List<String> organs, int j){
Query query= entityManager.createNativeQuery("select c.organization_id from organization c where c.parent_org_id="+orgId);
if(query.getResultList().size()>0)
organs.addAll((List<String>) query.getResultList());
else
children.add(orgId);
for(int i=j+1; i<organs.size();i++){
j=i;
query=entityManager.createNativeQuery("select c.organization_id from organization c where c.parent_org_id="+String.valueOf(organs.get(i)));
if(query.getResultList().size()>0)
organs.addAll((List<String>) query.getResultList());
else
children.add(String.valueOf(organs.get(i)));
}
}
I call this method like
getChildren("68",new ArrayList<>(),new ArrayList<>(),-1)
I have oracle as database so if it is better to acheive this via oracle query please note me.
By the way my spring version doesn't support streams!

You basically have a 1+N select problem (issue a single query for each id in your collection). Don't do this, instead use the IN clause and issue a single query!.
Another thing you shouldn't be doing is use concat for your query creation, never do this! What you should do is just return the result and use a query with an IN clause.
String query = "select c.organization_id from organization c where c.parent_org_id IN (:ids))
return em.createNativeQuery(query).setParameters("ids", organs);
or even better use a single query altogether. You can use a self join for that)
String query = "SELECT c1.organization_id " +
"FROM organization c1 LEFT OUTER JOIN organization c2 " +
"ON c1.parent_org_id=c2.organization_id " +
"WHERE c2.parent_org_id=?";
return em.createNativeQuery(query, String.class).setParameter(1, orgId).getResultList();
Judging from your solution I would strongly recommend studying SQL and JPA.

Try it in a big query
select c.organization_id from organization, (
select c.organization_id as sub1 from organization c where c.parent_org_id="+orgId +") as x_ where organization.parent_org_id = x_.sub1
The reason why it is slow, is that you query the database for every child. This is very inefficient, I assume even the load on the database will be smaller.

Answer provided by Deinum only gets the direct children. Whereas my java method returns all children including nested ones.
I have found a query for that which is much more optimized than my java method. It uses temporary tables:
with temp_tbl (id,parent_id) as
(
select organization_id,parent_org_id from organization where parent_org_id =?
union all
select c.organization_id ,c.parent_org_id from organization c join temp_tbl t on t.id=c.parent_org_id and t.parent_id !=t.id
)
Select to_char(g.id) from temp_tbl g
At the end i return to_char(g.id) to make it a string instead of big decimal. Because em.createNativeQuery(query, String.class) didn't turn it to string and threw error for me.
So just use em.createNativeQuery(query)

Spring Data JPA Specifications - Distinct + order by column in join

I have some specifications that I am combining with "and":
Specification.where(predicate1).and(predicate2).and(predicate3);
One of them has distinct set :
query.distinct(true);
Another one makes an order by on a column that is in a join.
query.orderBy(builder.desc(bJoin.get("orderbyColumn")));
This fails with a SQLGrammarException stating that order by column should be in distinct.
So basically we have entity A, the main entity, and some nested entity B, we select from A but want to order by B, and in generated sql it only selects columns from A. The only way I found to make it work (= making it select from B as well) is to replace the join by a fetch :
Fetch < A, B > bFetch = root.fetch("joinCol", JoinType.INNER);
Join < A, B > bJoin = (Join < A, B > ) bFetch;
that worked for some time, was testing locally in H2, but then after some time started getting another error :
org.hibernate.QueryException: query specified join fetching, but the
owner of the fetched association was not present in the select list
I solved it somehow in my local pointing to H2 by requiring some columns to not be null, but in real server using PostgreSQL, it's not working at all, getting that error for all cases when a fetch is present.
My question is : what is the right way to use distinct along with orderby on a nested entity that is not fetched? Is my solution with fetch ok and it just needs to be fixed (and if so how?) or I should go for another option entirely?
For the actual query I am using this method :
findAll(Specification<>, Pageable)
Isn't there a way to have distinct wrapping the whole query with order by (some sort of subquery?) and bypassing all this nightmare? Have it generate a query like this:
select distinct colA1, colA2, coAl3 from (select colA1, colA2, coAl3
from A inner join B b on ........ order by b.colB1)
Do I need to convert my specification to predicate manually and do something else with it to try to solve my issues (some kind of hybrid approach)?
Any pieces of advice will be greatly appreciated.

I encountered same error but actually it was not error :)
findAll(Specification<>, Pageable) this method throws 2 different queries.
First one is count query where you have to be careful.
Second is the rows query where you actually did it.
You can check the query type with code below
if (query.getResultType() != Long.class && query.getResultType() != long.class){
root.fetch("entity1");
}

DAO: diffrence between InMemory implementation and Database Implementation

I'm confusing with implementation of CRUD methods for DAODatabase (for Oracle 11 xe).
The problem is that the "U"-method (update) in case of storing in generally to a Map collection inserts a new element or renews it (key-value data like ID:AbstractBusinessObject) in a Map collection. And you don't care about it, when you write something like myHashMap.add(element). This method (update) is widely used in project's business logic.
Obviously, in case of using Oracle I must care about both inserting and renewing of existing elements. But I'm stucked to choose the way how to implement it:
There is no intrinsic function for so-called UPSERT in Oracle (at least in xe11g r2 version). However, I can emulate necessary function by SQL-query like this:
INSERT INTO mytable (id1, t1)
SELECT 11, 'x1' FROM DUAL
WHERE NOT EXISTS (SELECT id1 FROM mytble WHERE id1 = 11);
UPDATE mytable SET t1 = 'x1' WHERE id1 = 11;
(src:http://stackoverflow.com/a/21310345/2938167)
By using this kind of query (first - insert, second - update) I presume that the data mostly will be inserted not updated (at least it will be rather rare). (May it be not optimal for concurrency?).
Ok, it is possible. But at this point I'm confusing to decide:
-- should I write an SQL function (with approriate arguments of course) for this and call it via Java
-- or should I simply handle a serie of queries for preparedStatements and do them via .executeUpdate/.executeQuery? Should I handle the whole UPSERT SQL code for one preparedStatment or split it into several SQL-queries and prepared statements inside one method's body? (I'm using Tomcat's pool of connections and I pass a connection instance via static method getConnection() to each method implementation in DAODatabase) ?
Is there another possibility to solve the UPSERT quest?

The equivalent to your UPSERT statement would seem to be to use MERGE:
MERGE INTO mytable d
USING ( SELECT 11 AS id, 'x1' AS t1 FROM DUAL ) s
ON ( d.id = s.id )
WHEN NOT MATCHED THEN
INSERT ( d.id, d.t1 ) VALUES ( s.id, s.t1 )
WHEN MATCHED THEN
UPDATE SET d.t1 = s.t1;
You could also use (or wrap in a procedure):
DECLARE
p_id MYTABLE.ID%TYPE := 11;
p_t1 MYTABLE.T1%TYPE := 'x1';
BEGIN
UPDATE mytable
SET t1 = p_t1
WHERE id = p_id;
IF SQL%ROWCOUNT = 0 THEN
INSERT INTO mytable ( id, t1 ) VALUES ( p_id, p_t1 );
END IF;
END;
/
However, when you are handling a CRUD request - if you are doing a Create action then it should be represented by an INSERT (and if something already exists then you ought to throw the equivalent of the HTTP status code 400 Bad Request or 409 Conflict, as appropriate) and if you are doing an Update action it should be represented by an UPDATE (and if nothing is there to update then return the equivalent error to 404 Not Found.
So, while MERGE fits your description I don't think it is representative of a RESTful action as you ought to be separating the actions to their appropriate end-points rather than combining then into a joint action.

Union is not working in HQL [duplicate]

What alternatives do I have to implement a union query using hibernate? I know hibernate does not support union queries at the moment, right now the only way I see to make a union is to use a view table.
The other option is to use plain jdbc, but this way I would loose all my example/criteria queries goodies, as well as the hibernate mapping validation that hibernate performs against the tables/columns.

You could use id in (select id from ...) or id in (select id from ...)
e.g. instead of non-working
from Person p where p.name="Joe"
union
from Person p join p.children c where c.name="Joe"
you could do
from Person p
where p.id in (select p1.id from Person p1 where p1.name="Joe")
or p.id in (select p2.id from Person p2 join p2.children c where c.name="Joe");
At least using MySQL, you will run into performance problems with it later, though. It's sometimes easier to do a poor man's join on two queries instead:
// use set for uniqueness
Set<Person> people = new HashSet<Person>((List<Person>) query1.list());
people.addAll((List<Person>) query2.list());
return new ArrayList<Person>(people);
It's often better to do two simple queries than one complex one.
EDIT:
to give an example, here is the EXPLAIN output of the resulting MySQL query from the subselect solution:
mysql> explain
select p.* from PERSON p
where p.id in (select p1.id from PERSON p1 where p1.name = "Joe")
or p.id in (select p2.id from PERSON p2
join CHILDREN c on p2.id = c.parent where c.name="Joe") \G
*************************** 1. row ***************************
id: 1
select_type: PRIMARY
table: a
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 247554
Extra: Using where
*************************** 2. row ***************************
id: 3
select_type: DEPENDENT SUBQUERY
table: NULL
type: NULL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: NULL
Extra: Impossible WHERE noticed after reading const tables
*************************** 3. row ***************************
id: 2
select_type: DEPENDENT SUBQUERY
table: a1
type: unique_subquery
possible_keys: PRIMARY,name,sortname
key: PRIMARY
key_len: 4
ref: func
rows: 1
Extra: Using where
3 rows in set (0.00 sec)
Most importantly, 1. row doesn't use any index and considers 200k+ rows. Bad! Execution of this query took 0.7s wheres both subqueries are in the milliseconds.

Use VIEW. The same classes can be mapped to different tables/views using entity name, so you won't even have much of a duplication. Being there, done that, works OK.
Plain JDBC has another hidden problem: it's unaware of Hibernate session cache, so if something got cached till the end of the transaction and not flushed from Hibernate session, JDBC query won't find it. Could be very puzzling sometimes.

I have to agree with Vladimir. I too looked into using UNION in HQL and couldn't find a way around it. The odd thing was that I could find (in the Hibernate FAQ) that UNION is unsupported, bug reports pertaining to UNION marked 'fixed', newsgroups of people saying that the statements would be truncated at UNION, and other newsgroups of people reporting it works fine...
After a day of mucking with it, I ended up porting my HQL back to plain SQL, but doing it in a View in the database would be a good option. In my case, parts of the query were dynamically generated, so I had to build the SQL in the code instead.

I have a solution for one critical scenario (for which I struggled a lot )with union in HQL .
e.g. Instead of not working :-
select i , j from A a , (select i , j from B union select i , j from C) d where a.i = d.i
OR
select i , j from A a JOIN (select i , j from B union select i , j from C) d on a.i = d.i
YOU could do in Hibernate HQL ->
Query q1 =session.createQuery(select i , j from A a JOIN B b on a.i = b.i)
List l1 = q1.list();
Query q2 = session.createQuery(select i , j from A a JOIN C b on a.i = b.i)
List l2 = q2.list();
then u can add both list ->
l1.addAll(l2);

A view is a better approach but since hql typically returns a List or Set... you can do list_1.addAll(list_2). Totally sucks compared to a union but should work.

Perhaps I had a more straight-forward problem to solve. My 'for instance' was in JPA with Hibernate as the JPA provider.
I split the three selects (two in a second case) into multiple select and combined the collections returned myself, effectively replacing a 'union all'.

Hibernate 6 added support for UNION.
So, you can now use UNION in JPQL queries like this:
List<String> topics = entityManager.createQuery("""
select c.name as name
from Category c
union
select t.name as name
from Tag t
""", String.class)
.getResultList();
And you can also also use UNION ALL if there are no duplicates to be removed:
List<String> topics = entityManager.createQuery("""
select c.name as name
from Category c
union all
select t.name as name
from Tag t
""", String.class)
.getResultList();
Besides UNION, you can also use EXCEPT and INTERSECT.

I too have been through this pain - if the query is dynamically generated (e.g. Hibernate Criteria) then I couldn't find a practical way to do it.
The good news for me was that I was only investigating union to solve a performance problem when using an 'or' in an Oracle database.
The solution Patrick posted (combining the results programmatically using a set) while ugly (especially since I wanted to do results paging as well) was adequate for me.

Here is a special case, but might inspire you to create your own work around. The goal here is to count the total number of records from two different tables where records meet a particular criteria. I believe this technique will work for any case where you need to aggregate data from across multiple tables/sources.
I have some special intermediate classes setup, so the code which calls the named query is short and sweet, but you can use whatever method you normally use in conjunction with named queries to execute your query.
QueryParms parms=new QueryParms();
parms.put("PROCDATE",PROCDATE);
Long pixelAll = ((SourceCount)Fetch.row("PIXEL_ALL",parms,logger)).getCOUNT();
As you can see here, the named query begins to look an aweful lot like a union statement:
#Entity
#NamedQueries({
#NamedQuery(
name ="PIXEL_ALL",
query = "" +
" SELECT new SourceCount(" +
" (select count(a) from PIXEL_LOG_CURR1 a " +
" where to_char(a.TIMESTAMP, 'YYYYMMDD') = :PROCDATE " +
" )," +
" (select count(b) from PIXEL_LOG_CURR2 b" +
" where to_char(b.TIMESTAMP, 'YYYYMMDD') = :PROCDATE " +
" )" +
") from Dual1" +
""
)
})
public class SourceCount {
#Id
private Long COUNT;
public SourceCount(Long COUNT1, Long COUNT2) {
this.COUNT = COUNT1+COUNT2;
}
public Long getCOUNT() {
return COUNT;
}
public void setCOUNT(Long COUNT) {
this.COUNT = COUNT;
}
}
Part of the magic here is to create a dummy table and insert one record into it. In my case, I named it dual1 because my database is Oracle, but I don't think it matters what you call the dummy table.
#Entity
#Table(name="DUAL1")
public class Dual1 {
#Id
Long ID;
}
Don't forget to insert your dummy record:
SQL> insert into dual1 values (1);

As Patrick said, appending the LISTs from each SELECT would be a good idea but remember that it acts like UNION ALL. To avoid this side effect, just control if the object is already added in final collection or not. If no, then add it.
Something else that you should care about is that if you have any JOIN in each SELECT, the result would be a list of object array(List<Object[]>) so you have to iterate over it to only keep the object that you need.
Hope it works.

How to convert nested SQL to HQL

I am new to the Hibernate and HQL. I want to write an update query in HQL, whose SQL equivalent is as follows:
update patient set
`last_name` = "new_last",
`first_name` = "new_first"
where id = (select doctor_id from doctor
where clinic_id = 22 and city = 'abc_city');
doctor_id is PK for doctor and is FK and PK in patient. There is one-to-one mapping.
The corresponding Java classes are Patient (with fields lastName, firstName, doctorId) and Doctor (with fields doctorId).
Can anyone please tell what will be the HQL equivalent of the above SQL query?
Thanks a lot.

String update = "update Patient p set p.last_name = :new_last, p.first_name = :new_first where p.id = some (select doctor.id from Doctor doctor where doctor.clinic_id = 22 and city = 'abc_city')";
You can work out how to phrase hql queries if you check the specification. You can find a section about subqueries there.

I don't think you need HQL (I know, you ask that explicitly, but since you say you're new to Hibernate, let me offer a Hibernate-style alternative). I am not a favor of HQL, because you are still dealing with strings, which can become hard to maintain, just like SQL, and you loose type safety.
Instead, use Hibernate criteria queries and methods to query your data. Depending on your class mapping, you could do something like this:
List patients = session.CreateCriteria(typeof(Patient.class))
.createAlias("doctor", "dr")
.add(Restrictions.Eq("dr.clinic_id", 22))
.add(Restrictions.Eq("dr.city", "abc_city"))
.list();
// go through the patients and set the properties something like this:
for(Patient p : patients)
{
p.lastName = "new lastname";
p.firstName = "new firstname";
}
Some people argue that using CreateCriteria is difficult. It takes a little getting used to, true, but it has the advantage of type safety and complexities can easily be hidden behind generic classes. Google for "Hibernate java GetByProperty" and you see what I mean.

update Patient set last_name = :new_last , first_name = :new_first where patient.id = some(select doctor_id from Doctor as doctor where clinic_id = 22 and city = abc_city)

There is a significant difference between executing update with select and actually fetching the records to the client, updating them and posting them back:
UPDATE x SET a=:a WHERE b in (SELECT ...)
works in the database, no data is transferred to the client.
list=CreateCriteria().add(Restriction).list();
brings all the records to be updated to the client, updates them, then posts them back to the database, probably with one UPDATE per record.
Using UPDATE is much, much faster than using criteria (think thousands of times).

Since the question title can be interpreted generally as "How to use nested selects in hibernate", and the HQL syntax restricts nested selects only to be in the select- and the where-clause, I would like to add here the possibility to use native SQL as well. In Oracle - for instance - you may also use a nested select in the from-clause.
Following query with two nested inner selects cannot be expressed by HQL:
select ext, count(ext)
from (
select substr(s, nullif( instr(s,'.', -1) +1, 1) ) as ext
from (
select b.FILE_NAME as s from ATTACHMENT_B b
union select att.FILE_NAME as s from ATTACHEMENT_FOR_MAIL att
)
)
GROUP BY ext
order by ext;
(which counts, BTW, the occurences of each distinct file name extension in two different tables).
You can use such an sql string as native sql like this:
#Autowired
private SessionFactory sessionFactory;
String sql = ...
SQLQuery qry = sessionFactory.getCurrentSession().createSQLQuery(sql);
// provide an appropriate ResultTransformer
return qry.list();