We Keep Coding

How to create a dynamic WHERE clause for a query using an array list

I have a List<String> of categories and for each category, I want to add them to my WHERE clause by combining with AND operator like: SELECT question_id FROM question WHERE category = categ1 AND category = categ2 AND category = ...
Since the size of the categories list is changing, I cannot do something like this:
String sql = "SELECT question_id FROM question WHERE category = ? AND category = ?";
jdbcTemplate.query(sql, stringMapper, "categ1", "categ2");
How can I achieve what I want?

Either check if JDBC Template from Spring handle that for you using a syntax which could be something like (from the doc, I don't think it does)
SELECT question_id FROM question WHERE category in (?...)
Or write your own query with the problems that may arise:
List<Object> parameters = new ArrayList<>(categories.size());
StringBuilder sb = new StringBuilde("SELECT question_id FROM question WHERE 1=1");
if (!categories.isEmpty()) {
if (categories.size() == 1) {
sb.append(" and category = ?");
} else {
sb.append(" and category in ");
sb.append(categories.stream()
.map(ignored -> "?")
.collect(joining(", ", "(", ")")));
sb.append(")");
}
parameters.addAll(categories);
}
Object[] paramArray = parameters.toArray();
jdbcTemplate.query(sb.toString(), stringMapper, paramArray);
Notes:
some security/quality tool may report SQL issues because you are writing a dynamic SQL.
Oracle put a limit on 1000 elements per IN. You would have to partition categories per group of 1000 (or less).
I used a stream() in a more or less strange fashion in order to generate the "?". If you use commons-lang3, you can replace it by "(" + StringUtils.repeat("?", ", ", categories.size()) + ")" (the example in the javadoc was probably done with this kind of use).
if you only have category as single criteria, you may probably remove the 1=1 as well as the and.

I believe this may work for you:
// The SQL Query
String sql = "SELECT question_id FROM question";
// Create the WHERE clause based on the number of items in List...
StringBuilder whereClause = new StringBuilder(" WHERE ");
StringBuilder ps = new StringBuilder("");
for (int i = 0; i < categories.size(); i++) {
if (!ps.toString().isEmpty()) {
ps.append(" AND ");
}
ps.append("category = ?");
}
whereClause.append(ps.toString()).append(";");
//Append the WHERE clause string to the SQL query string
sql = sql + whereClause.toString();
//System.out.println(sql);
/* Convert the categories List to an Object[] Array so as to
pass in as varArgs to the jdbcTemplate.query() method. */
Object[] psArgs = categories.toArray(new Object[categories.size()]);
jdbcTemplate.query(sql, stringMapper, psArgs);

Avoiding "HHH000104: firstResult/maxResults specified with collection fetch; applying in memory!" using Spring Data [duplicate]

I'm getting a warning in the Server log "firstResult/maxResults specified with collection fetch; applying in memory!". However everything working fine. But I don't want this warning.
My code is
public employee find(int id) {
return (employee) getEntityManager().createQuery(QUERY).setParameter("id", id).getSingleResult();
}
My query is
QUERY = "from employee as emp left join fetch emp.salary left join fetch emp.department where emp.id = :id"

Although you are getting valid results, the SQL query fetches all data and it's not as efficient as it should.
So, you have two options.
Fixing the issue with two SQL queries that can fetch entities in read-write mode
The easiest way to fix this issue is to execute two queries:
. The first query will fetch the root entity identifiers matching the provided filtering criteria.
. The second query will use the previously extracted root entity identifiers to fetch the parent and the child entities.
This approach is very easy to implement and looks as follows:
List<Long> postIds = entityManager
.createQuery(
"select p.id " +
"from Post p " +
"where p.title like :titlePattern " +
"order by p.createdOn", Long.class)
.setParameter(
"titlePattern",
"High-Performance Java Persistence %"
)
.setMaxResults(5)
.getResultList();
List<Post> posts = entityManager
.createQuery(
"select distinct p " +
"from Post p " +
"left join fetch p.comments " +
"where p.id in (:postIds) " +
"order by p.createdOn", Post.class)
.setParameter("postIds", postIds)
.setHint(
"hibernate.query.passDistinctThrough",
false
)
.getResultList();
Fixing the issue with one SQL query that can only fetch entities in read-only mode
The second approach is to use SDENSE_RANK over the result set of parent and child entities that match our filtering criteria and restrict the output for the first N post entries only.
The SQL query can look as follows:
#NamedNativeQuery(
name = "PostWithCommentByRank",
query =
"SELECT * " +
"FROM ( " +
" SELECT *, dense_rank() OVER (ORDER BY \"p.created_on\", \"p.id\") rank " +
" FROM ( " +
" SELECT p.id AS \"p.id\", " +
" p.created_on AS \"p.created_on\", " +
" p.title AS \"p.title\", " +
" pc.id as \"pc.id\", " +
" pc.created_on AS \"pc.created_on\", " +
" pc.review AS \"pc.review\", " +
" pc.post_id AS \"pc.post_id\" " +
" FROM post p " +
" LEFT JOIN post_comment pc ON p.id = pc.post_id " +
" WHERE p.title LIKE :titlePattern " +
" ORDER BY p.created_on " +
" ) p_pc " +
") p_pc_r " +
"WHERE p_pc_r.rank <= :rank ",
resultSetMapping = "PostWithCommentByRankMapping"
)
#SqlResultSetMapping(
name = "PostWithCommentByRankMapping",
entities = {
#EntityResult(
entityClass = Post.class,
fields = {
#FieldResult(name = "id", column = "p.id"),
#FieldResult(name = "createdOn", column = "p.created_on"),
#FieldResult(name = "title", column = "p.title"),
}
),
#EntityResult(
entityClass = PostComment.class,
fields = {
#FieldResult(name = "id", column = "pc.id"),
#FieldResult(name = "createdOn", column = "pc.created_on"),
#FieldResult(name = "review", column = "pc.review"),
#FieldResult(name = "post", column = "pc.post_id"),
}
)
}
)
The #NamedNativeQuery fetches all Post entities matching the provided title along with their associated PostComment child entities. The DENSE_RANK Window Function is used to assign the rank for each Post and PostComment joined record so that we can later filter just the amount of Post records we are interested in fetching.
The SqlResultSetMapping provides the mapping between the SQL-level column aliases and the JPA entity properties that need to be populated.
Now, we can execute the PostWithCommentByRank #NamedNativeQuery like this:
List<Post> posts = entityManager
.createNamedQuery("PostWithCommentByRank")
.setParameter(
"titlePattern",
"High-Performance Java Persistence %"
)
.setParameter(
"rank",
5
)
.unwrap(NativeQuery.class)
.setResultTransformer(
new DistinctPostResultTransformer(entityManager)
)
.getResultList();
Now, by default, a native SQL query like the PostWithCommentByRank one would fetch the Post and the PostComment in the same JDBC row, so we will end up with an Object[] containing both entities.
However, we want to transform the tabular Object[] array into a tree of parent-child entities, and for this reason, we need to use the Hibernate ResultTransformer.
The DistinctPostResultTransformer looks as follows:
public class DistinctPostResultTransformer
extends BasicTransformerAdapter {
private final EntityManager entityManager;
public DistinctPostResultTransformer(
EntityManager entityManager) {
this.entityManager = entityManager;
}
#Override
public List transformList(
List list) {
Map<Serializable, Identifiable> identifiableMap =
new LinkedHashMap<>(list.size());
for (Object entityArray : list) {
if (Object[].class.isAssignableFrom(entityArray.getClass())) {
Post post = null;
PostComment comment = null;
Object[] tuples = (Object[]) entityArray;
for (Object tuple : tuples) {
if(tuple instanceof Identifiable) {
entityManager.detach(tuple);
if (tuple instanceof Post) {
post = (Post) tuple;
}
else if (tuple instanceof PostComment) {
comment = (PostComment) tuple;
}
else {
throw new UnsupportedOperationException(
"Tuple " + tuple.getClass() + " is not supported!"
);
}
}
}
if (post != null) {
if (!identifiableMap.containsKey(post.getId())) {
identifiableMap.put(post.getId(), post);
post.setComments(new ArrayList<>());
}
if (comment != null) {
post.addComment(comment);
}
}
}
}
return new ArrayList<>(identifiableMap.values());
}
}
The DistinctPostResultTransformer must detach the entities being fetched because we are overwriting the child collection and we don’t want that to be propagated as an entity state transition:
post.setComments(new ArrayList<>());

Reason for this warning is that when fetch join is used, order in result sets is defined only by ID of selected entity (and not by join fetched).
If this sorting in memory is causing problems, do not use firsResult/maxResults with JOIN FETCH.

To avoid this WARNING you have to change the call getSingleResult to
getResultList().get(0)

This warning tells you Hibernate is performing in memory java pagination. This can cause high JVM memory consumption.
Since a developer can miss this warning, I contributed to Hibernate by adding a flag allowing to throw an exception instead of logging the warning (https://hibernate.atlassian.net/browse/HHH-9965).
The flag is hibernate.query.fail_on_pagination_over_collection_fetch.
I recommend everyone to enable it.
The flag is defined in org.hibernate.cfg.AvailableSettings :
/**
* Raises an exception when in-memory pagination over collection fetch is about to be performed.
* Disabled by default. Set to true to enable.
*
* #since 5.2.13
*/
String FAIL_ON_PAGINATION_OVER_COLLECTION_FETCH = "hibernate.query.fail_on_pagination_over_collection_fetch";

the problem is you will get cartesian product doing JOIN. The offset will cut your recordset without looking if you are still on same root identity class

I guess the emp has many departments which is a One to Many relationship. Hibernate will fetch many rows for this query with fetched department records. So the order of result set can not be decided until it has really fetch the results to the memory. So the pagination will be done in memory.
If you do not want to fetch the departments with emp, but still want to do some query based on the department, you can achieve the result with out warning (without doing ordering in the memory). For that simply you have to remove the "fetch" clause. So something like as follows:
QUERY = "from employee as emp left join emp.salary sal left join emp.department dep where emp.id = :id and dep.name = 'testing' and sal.salary > 5000 "

As others pointed out, you should generally avoid using "JOIN FETCH" and firstResult/maxResults together.
If your query requires it, you can use .stream() to eliminate warning and avoid potential OOM exception.
try (Stream<ENTITY> stream = em.createQuery(QUERY).stream()) {
ENTITY first = stream.findFirst().orElse(null); // equivalents .getSingleResult()
}
// Stream returned is an IO stream that needs to be closed manually.

getting data from result set is too slow

Fetching data from PostgreSQL database with Result Set is too slow.
Here is my code.
for (int i = 0; i < qry_list.size(); i++) {
try {
resultSet = statement.executeQuery(qry_list.get(i));
resultSet.setFetchSize(0);
while (resultSet.next()) {
totalfilecrated = totalfilecrated
+ resultSet.getInt(columnname);
}
} catch (SQLException e) {
e.printStackTrace();
}
}
Here I try to fetch data inside a for loop.is it good?
Here is my query.
For getting ID of individual Organisations(org_unit_id).
"select org_unit_id from emd.emd_org_unit where org_unit_id
in(select org_unit_id from emd.emd_org_unit_detail where entity_type_id=1 and is_active=true) and
is_active=true order by org_unit_name_en";
Then i want to get the count of files with each org_unit_id
select count(*) as totalfilecreatedelectronic from fl_file ff
left join vw_employee_details_with_department epd on epd.post_detail_id=ff.file_opened_by_post_fk
where ff.file_nature = 'E' and ((ff.migration_date>='2011-01-01' and ff.migration_date<='2015-01-01') or
(ff.opening_date >='2011-01-01' and ff.opening_date <='2015-01-01')) and
epd.departmentid=org_unit_id";

Seeing how your second query already contains a reference to a column that's an org_unit_id, you might think joining emd_org_unit table in directly:
select org.org_unit_id, count(*) as totalfilecreatedelectronic
from fl_file ff
left join vw_employee_details_with_department epd on epd.post_detail_id=ff.file_opened_by_post_fk
-- join to active entries in emd_org_unit
inner join from emd.emd_org_unit org ON epd.departmentid=org.org_unit_id
AND org.is_active=true
where ff.file_nature = 'E'
and (
(ff.migration_date>='2011-01-01' and ff.migration_date<='2015-01-01') or
(ff.opening_date >='2011-01-01' and ff.opening_date <='2015-01-01'))
-- and now group by org_unit_id to get the counts
group by org_unit_id
If you'd create a SQLFiddle for this, things would get much clearer I guess.

How do I create my query with Hibernate Criteria

I have to create the following query with Criteria:
SELECT r.*,(
SELECT COUNT(*) FROM APP.P_FORM_QUESTION_ANSWER qa
WHERE qa.J_STATUS = 'CORRECT'
AND qa.J_FORM_ID = r.J_FORM_ID AND qa.J_AUTHOR_ID = r.J_AUTHOR_ID
AND qa.J_FORM_RESULT_ID = r.J_ROW_ID
) as correctCount
FROM APP.P_FORM_RESULT r
WHERE r.J_FORM_ID = '123456'
ORDER BY correctCount DESC;
I try to use DetachedCriteria for extra column, but I do not see how to represent :
"qa.J_FORM_ID = r.J_FORM_ID AND qa.J_AUTHOR_ID = r.J_AUTHOR_ID
AND qa.J_FORM_RESULT_ID = r.J_ROW_ID"
in the DetachedCriteria, and add my DetachedCriteria as a new column
Criteria criteria =
HibernateUtil.getSession().createCriteria(FormResult.class, "formResult");
criteria.add(Restrictions.eq("formResult.formId", formId));
DetachedCriteria correctCount =
DetachedCriteria.forClass(QuestionAnswer.class, "questionAnswer");
correctCount.add(
Restrictions.eq("questionAnswer.status",QuestionAnswerStatus.CORRECT));
// How to retrieve 'formResult' ?
correctCount.add(Restrictions.eq("questionAnswer.formId", "formResult.formId"));
// How to retrieve 'formResult' ?
correctCount.add(Restrictions.eq("questionAnswer.authorId", "formResult.authorId"));
// How to retrieve 'formResult' ?
correctCount.add(
Restrictions.eq("questionAnswer.formResult.rowId", "formResult.rowId"));
correctCount.setProjection(Projections.rowCount());
// How to add my DetachedCriteria as a new column
criteria.setProjection(Projections.alias(correctCount, "correctCount"));
The Lines with comments are the lines for which I cannot find the solution.

Finally, I created a different SQL query that gives me the same result:
SELECT r.J_ROW_ID, COUNT(DISTINCT qa.J_ROW_ID) as correctCount
FROM APP.P_FORM_RESULT r left join APP.P_FORM_QUESTION_ANSWER qa on qa.J_FORM_ID = r.J_FORM_ID AND qa.J_AUTHOR_ID = r.J_AUTHOR_ID AND qa.J_FORM_RESULT_ID = r.J_ROW_ID AND qa.J_STATUS = 'CORRECT'
WHERE r.J_FORM_ID = '123456'
GROUP BY r.J_ROW_ID
ORDER BY correctCount DESC;
And I used HQL because I cannot find how to do it in Criteria:
// HQL Query to retrieve the list of FormResult IDs
StringBuilder hql = new StringBuilder();
hql.append("SELECT r.id");
hql.append(" FROM QuestionAnswer qa RIGHT JOIN qa.formResult as r");
hql.append(" WHERE r.formId = :formId AND (qa.status = :status OR qa.status IS NULL)");
hql.append(" GROUP BY r.id");
hql.append(" ORDER BY COUNT(DISTINCT qa.id) DESC"));
Query query = HibernateUtil.getSession().createQuery(hql.toString());
query.setParameter("formId", formId);
query.setParameter("status", QuestionAnswerStatus.CORRECT);
// Retrieve the list of FormResult objects from the list of IDs previously retrieved
List<Long> result = query.list();
if (result != null && !result.isEmpty()) {
for (Long resultId : result) {
formResults.add(getData(FormResult.class, resultId));
}
}
The downside is that I have to retrieve each FormResult one by one from my list of IDs retrieved by my HQL query

Merging three queries that use 1:* relationships into one query

Lets say i have four tables i want to read from:
customer
customer_id, customer_name
1 Joe Bolggs
customer_orders
customer_id, order_no
----------------------------
1 1
1 2
1 3
customer_addresses
customer_id address
----------------------------
1 11 waterfall road
1 23 The broadway
customer_tel_no
customer_id number
----------------------------
1 523423423432
1 234342342343
The customer information shown above (for the customer with id=1) is to be stored in a Java object as shown below
public class Customer{
String customer_id;
String customerName;
ArrayList<String> customerOrders;
ArrayList<String> customerAddress;
ArrayList<String> customerTelephoneNumbers;
}
The only way i can think of to get the above information is by using three queries. The reason is that there is a 1:* relationship between the customer table and each of the other tables. To get the data i am doing something like this:
Customer customer = new Customer()
String customerSQL = "Select * from customer where customer_id = ?";
statement = getConnection().prepareStatement(contactsQuery);
statement.setString(1,1);
resultSet = statement.executeQuery();
while (resultSet.next()){
customer.customer_id = resultSet.get(1); //No getters/setters in this example
customer.customerName = resultSet.get(2);
}
String customerOrdersSQL = "Select * from customer_orders where customer_id = ?";
statement = getConnection().prepareStatement(customerOrdersSQL);
statement.setString(1,1);
resultSet = statement.executeQuery();
customer.customerOrders = new ArrayList();
while (resultSet.next()){
customer.customerOrders.add(resultSet.getString(2); // all the order numbers
}
String customerAddressesSQL = "Select * from customer_addresses where customer_id = ?";
statement = getConnection().prepareStatement(customerAddressesSQL );
statement.setString(1,1);
resultSet = statement.executeQuery();
customer.customerAddresses = new ArrayList();
while (resultSet.next()){
customer.customerAddresses.add(resultSet.getString(2); // all the addresses
}
String customerTelSQL = "Select * from customer_tel_no where customer_id = ?";
statement = getConnection().prepareStatement(customerTelSQL);
statement.setString(1,1);
resultSet = statement.executeQuery();
customer.customerTelephoneNumbers = new ArrayList();
while (resultSet.next()){
customer.customerTelephoneNumbers.add(resultSet.getString(2); // all the order numbers
}
The problem with the above is that i am making three calls to the database. Is there a way i can merge the above into a single query please?
I cant see how a join would work because for example, a join between customer and customer_orders will return a customer row for each row in customer_orders. Is there anyway i can merge the above three queries into one?

I would think that something like this would work:
SELECT c.customer_id, c.customer_name, o.order_no, a.address, t.number
FROM customer c LEFT JOIN customer_orders o ON c.customer_id = o.customer_id
LEFT JOIN customer_addresses a ON c.customer_id = a.customer_id
LEFT JOIN customer_tel_no t ON c.customer_id = t.customer_id
WHERE c.customer_id = ?
Then, in your code, after you execute the query:
while (resultSet.next())
{
customer.customerOrders.add(resultSet.getString(3));
customer.customerAddresses.add(resultSet.getString(4));
customer.customerTelephoneNumbers.add(resultSet.getString(5));
}
Of course, this does not take into account the fact that you will have null values along the way, so I'd advise checking for nulls to make sure that you aren't adding a lot of junk to your array lists. Still, that's probably a lot less costly than 3 separate queries.

Nothing prevents you from iterating and processing the joined result into your customer object. If your application is complex enough, you could look into ORM frameworks which would do that for you under the covers. If you are working with JavaEE, have a look at JPA.

use this query and reduce the number of call. And, in while loop process on data.
select customer.customer_id,customer.customer_name,order_no,address,number
from customer,customer_orders,customer_addresses,customer_tel_no
where customer.customer_id = customer_orders.customer_id
AND
customer.customer_id = customer_addresses.customer_id
AND
customer.customer_id = customer_tel_no.customer_id