I'm tearing my hair out over something that may very well be very simple,
but I just cant get it right.
My GroupBy clause is not being added to the SQL generated by EclipseLink.
Have tried many different orders and variations of the code below.
public List<Orders> findOrdersEntitiesBySearch(int maxResults, int firstResult, String column1, String column2, String key, boolean searchOrder) {
EntityManager em = getEntityManager();
try {
CriteriaBuilder cb = em.getCriteriaBuilder();
CriteriaQuery<Orders> cq = cb.createQuery(Orders.class);
Root<Orders> order = cq.from(Orders.class);
Join<Orders, Products> prod = order.join("productsCollection");
// Where like key
if (column1 != null && column2 != null) {
if (searchOrder) {
cq.where(cb.or(cb.like(cb.lower(order.get(column1).as(String.class)), "%" + key.toLowerCase() + "%"), cb.like(cb.lower(order.get(column2).as(String.class)), "%" + key.toLowerCase() + "%")));
} else {
cq.where(cb.or(cb.like(cb.lower(prod.get(column1).as(String.class)), "%" + key.toLowerCase() + "%"), cb.like(cb.lower(prod.get(column2).as(String.class)), "%" + key.toLowerCase() + "%")));
}
} else {
if (searchOrder) {
cq.where(cb.like(cb.lower(order.get(column1).as(String.class)), "%" + key.toLowerCase() + "%"));
} else {
cq.where(cb.like(cb.lower(prod.get(column1).as(String.class)), "%" + key.toLowerCase() + "%"));
}
}
// Order By
List<Order> orderByList = new ArrayList<Order>();
orderByList.add(cb.desc(order.get("ordDate")));
orderByList.add(cb.desc(order.get("pkOrdID")));
cq.orderBy(orderByList);
// Select
cq.select(order);
// Group by
//cq.groupBy(order.get("pkOrdID"));
//Expression<Integer> grouping = order.get("pkOrdID").as(Integer.class);
Expression<String> grouping = order.get("pkOrdID").as(String.class);
cq.groupBy(grouping);
Query q = em.createQuery(cq);
q.setMaxResults(maxResults);
q.setFirstResult(firstResult);
return q.getResultList();
} finally {
em.close();
}
}
The code compiles an runs fine, I get results but my GroupBy clause is not included.
As a nasty quickfix, I am running the list returned through a function to remove the duplicates until I can find the solution.
Thanks in advance for any assistance,
David
For clarity, re-written as regular JPQL query, you currently have something like this:
SELECT o
FROM Orders o JOIN o.productsCollection p
WHERE ...
GROUP BY o.pkOrdID...
There are two issues here. First, the group by is not correct, because you can't group by on a single column when a full object is selected - just as with standard SQL, all selected columns that are not aggregates must be listed in the group by. The second issue is that you don't need group by here at all. See below for your options:
Since you don't use any aggregate functions here, what you actually want is simply:
SELECT DISTINCT o
FROM Orders o JOIN o.productsCollection p
WHERE ...
Therefore, simply drop the group-by from your criteria API query, and use cq.distinct(true) instead.
If you really need group by with aggregate functions for a different query, instead of grouping on the primary key of a selected object, in JPA you group by the object itself. A simple JPQL example might be:
SELECT o, sum(p.quantity)
FROM Orders o JOIN o.productsCollection p
WHERE ...
GROUP BY o
In your query, this would be cq.groupBy(order).
Btw. I have no idea why eclipse link simply ignores your group by here instead of reporting an error. Which version are you using?
Related
Currently I am doing it like this:
List<Table1Entity> findAllMatchingEntities(Table1Entity table1Entity) {
String queryString = "SELECT table1.* FROM table1 "
+ "JOIN table2 t2 ON table1.id=t2.table1_id";
if (table1Entity.getName() != null) {
queryString +=" where name like ?";
}
Query query = em.createNativeQuery(queryString, Table1Entity.class);
if (table1Entity.getName() != null) {
query.setParameter(1, table1Entity.getName())
}
return query.getResultedList();
}
If I want to check more parameters in this join this will quickly turn into a lot of if statements and it would be really complicated to set parameters correctly.
I know I can check parameters with criteria Builder API like this:
if(table1Entity.getName() != null) {
table1EntitySpecification = (root, query, criteriaBuilder)
-> criteriaBuilder.like(
criteriaBuilder.lower(root
.get("name")),
("%" + table1Entity.getName() + "%")
.toLowerCase());;
}
and after that get them all with:
findAll(table1EntitySpecification) with findAll from simpleJPARepository. Now I can chain them together with .or or .and etc. and avoid setting the parameter and checking for null second time.
But how do I do join with criteria APi?
I know I can have in my #Repository something like this:
#Query(value = "SELECT table1.* FROM table1 JOIN table2 t2 ON table1.id=t2.table1_id", nativeQuery = true)
List<Table1Entity> findAllMatchingEntities(Table1Entity table1Entity);
But since name is optional (can be null) I can't just leave it in #Query.
What is the best solution here to avoid using native query and in case of having to check many parameters to avoid using if statements?
I don't know if I fully get your question, but regarding the possibility of nulls, and using the CRUD repository, you can always do a null check before like:
#Query(value = "SELECT table1.* FROM table1 JOIN table2 t2 ON table1.id=t2.table1_id WHERE table1.id is not null", nativeQuery = true)
List<Table1Entity> findAllMatchingEntities(Table1Entity table1Entity);
Depending on what you are trying to achieve, you can always compose the query with similar checks like (not related to your code):
#Query("SELECT c FROM Certificate c WHERE (:id is null or upper(c.id) = :id) "
+ "and (:name is null or upper(c.name) = :name)")
List<Table1> findStuff(#Param("id") String id,
#Param("name") String name);
I'm getting a warning in the Server log "firstResult/maxResults specified with collection fetch; applying in memory!". However everything working fine. But I don't want this warning.
My code is
public employee find(int id) {
return (employee) getEntityManager().createQuery(QUERY).setParameter("id", id).getSingleResult();
}
My query is
QUERY = "from employee as emp left join fetch emp.salary left join fetch emp.department where emp.id = :id"
Although you are getting valid results, the SQL query fetches all data and it's not as efficient as it should.
So, you have two options.
Fixing the issue with two SQL queries that can fetch entities in read-write mode
The easiest way to fix this issue is to execute two queries:
. The first query will fetch the root entity identifiers matching the provided filtering criteria.
. The second query will use the previously extracted root entity identifiers to fetch the parent and the child entities.
This approach is very easy to implement and looks as follows:
List<Long> postIds = entityManager
.createQuery(
"select p.id " +
"from Post p " +
"where p.title like :titlePattern " +
"order by p.createdOn", Long.class)
.setParameter(
"titlePattern",
"High-Performance Java Persistence %"
)
.setMaxResults(5)
.getResultList();
List<Post> posts = entityManager
.createQuery(
"select distinct p " +
"from Post p " +
"left join fetch p.comments " +
"where p.id in (:postIds) " +
"order by p.createdOn", Post.class)
.setParameter("postIds", postIds)
.setHint(
"hibernate.query.passDistinctThrough",
false
)
.getResultList();
Fixing the issue with one SQL query that can only fetch entities in read-only mode
The second approach is to use SDENSE_RANK over the result set of parent and child entities that match our filtering criteria and restrict the output for the first N post entries only.
The SQL query can look as follows:
#NamedNativeQuery(
name = "PostWithCommentByRank",
query =
"SELECT * " +
"FROM ( " +
" SELECT *, dense_rank() OVER (ORDER BY \"p.created_on\", \"p.id\") rank " +
" FROM ( " +
" SELECT p.id AS \"p.id\", " +
" p.created_on AS \"p.created_on\", " +
" p.title AS \"p.title\", " +
" pc.id as \"pc.id\", " +
" pc.created_on AS \"pc.created_on\", " +
" pc.review AS \"pc.review\", " +
" pc.post_id AS \"pc.post_id\" " +
" FROM post p " +
" LEFT JOIN post_comment pc ON p.id = pc.post_id " +
" WHERE p.title LIKE :titlePattern " +
" ORDER BY p.created_on " +
" ) p_pc " +
") p_pc_r " +
"WHERE p_pc_r.rank <= :rank ",
resultSetMapping = "PostWithCommentByRankMapping"
)
#SqlResultSetMapping(
name = "PostWithCommentByRankMapping",
entities = {
#EntityResult(
entityClass = Post.class,
fields = {
#FieldResult(name = "id", column = "p.id"),
#FieldResult(name = "createdOn", column = "p.created_on"),
#FieldResult(name = "title", column = "p.title"),
}
),
#EntityResult(
entityClass = PostComment.class,
fields = {
#FieldResult(name = "id", column = "pc.id"),
#FieldResult(name = "createdOn", column = "pc.created_on"),
#FieldResult(name = "review", column = "pc.review"),
#FieldResult(name = "post", column = "pc.post_id"),
}
)
}
)
The #NamedNativeQuery fetches all Post entities matching the provided title along with their associated PostComment child entities. The DENSE_RANK Window Function is used to assign the rank for each Post and PostComment joined record so that we can later filter just the amount of Post records we are interested in fetching.
The SqlResultSetMapping provides the mapping between the SQL-level column aliases and the JPA entity properties that need to be populated.
Now, we can execute the PostWithCommentByRank #NamedNativeQuery like this:
List<Post> posts = entityManager
.createNamedQuery("PostWithCommentByRank")
.setParameter(
"titlePattern",
"High-Performance Java Persistence %"
)
.setParameter(
"rank",
5
)
.unwrap(NativeQuery.class)
.setResultTransformer(
new DistinctPostResultTransformer(entityManager)
)
.getResultList();
Now, by default, a native SQL query like the PostWithCommentByRank one would fetch the Post and the PostComment in the same JDBC row, so we will end up with an Object[] containing both entities.
However, we want to transform the tabular Object[] array into a tree of parent-child entities, and for this reason, we need to use the Hibernate ResultTransformer.
The DistinctPostResultTransformer looks as follows:
public class DistinctPostResultTransformer
extends BasicTransformerAdapter {
private final EntityManager entityManager;
public DistinctPostResultTransformer(
EntityManager entityManager) {
this.entityManager = entityManager;
}
#Override
public List transformList(
List list) {
Map<Serializable, Identifiable> identifiableMap =
new LinkedHashMap<>(list.size());
for (Object entityArray : list) {
if (Object[].class.isAssignableFrom(entityArray.getClass())) {
Post post = null;
PostComment comment = null;
Object[] tuples = (Object[]) entityArray;
for (Object tuple : tuples) {
if(tuple instanceof Identifiable) {
entityManager.detach(tuple);
if (tuple instanceof Post) {
post = (Post) tuple;
}
else if (tuple instanceof PostComment) {
comment = (PostComment) tuple;
}
else {
throw new UnsupportedOperationException(
"Tuple " + tuple.getClass() + " is not supported!"
);
}
}
}
if (post != null) {
if (!identifiableMap.containsKey(post.getId())) {
identifiableMap.put(post.getId(), post);
post.setComments(new ArrayList<>());
}
if (comment != null) {
post.addComment(comment);
}
}
}
}
return new ArrayList<>(identifiableMap.values());
}
}
The DistinctPostResultTransformer must detach the entities being fetched because we are overwriting the child collection and we don’t want that to be propagated as an entity state transition:
post.setComments(new ArrayList<>());
Reason for this warning is that when fetch join is used, order in result sets is defined only by ID of selected entity (and not by join fetched).
If this sorting in memory is causing problems, do not use firsResult/maxResults with JOIN FETCH.
To avoid this WARNING you have to change the call getSingleResult to
getResultList().get(0)
This warning tells you Hibernate is performing in memory java pagination. This can cause high JVM memory consumption.
Since a developer can miss this warning, I contributed to Hibernate by adding a flag allowing to throw an exception instead of logging the warning (https://hibernate.atlassian.net/browse/HHH-9965).
The flag is hibernate.query.fail_on_pagination_over_collection_fetch.
I recommend everyone to enable it.
The flag is defined in org.hibernate.cfg.AvailableSettings :
/**
* Raises an exception when in-memory pagination over collection fetch is about to be performed.
* Disabled by default. Set to true to enable.
*
* #since 5.2.13
*/
String FAIL_ON_PAGINATION_OVER_COLLECTION_FETCH = "hibernate.query.fail_on_pagination_over_collection_fetch";
the problem is you will get cartesian product doing JOIN. The offset will cut your recordset without looking if you are still on same root identity class
I guess the emp has many departments which is a One to Many relationship. Hibernate will fetch many rows for this query with fetched department records. So the order of result set can not be decided until it has really fetch the results to the memory. So the pagination will be done in memory.
If you do not want to fetch the departments with emp, but still want to do some query based on the department, you can achieve the result with out warning (without doing ordering in the memory). For that simply you have to remove the "fetch" clause. So something like as follows:
QUERY = "from employee as emp left join emp.salary sal left join emp.department dep where emp.id = :id and dep.name = 'testing' and sal.salary > 5000 "
As others pointed out, you should generally avoid using "JOIN FETCH" and firstResult/maxResults together.
If your query requires it, you can use .stream() to eliminate warning and avoid potential OOM exception.
try (Stream<ENTITY> stream = em.createQuery(QUERY).stream()) {
ENTITY first = stream.findFirst().orElse(null); // equivalents .getSingleResult()
}
// Stream returned is an IO stream that needs to be closed manually.
My code snippet:
criteriaQuery.select(root);
Predicate ctfPredicate;
if (deptPattern.contains("%") || deptPattern.contains("_")) {
deptPattern = deptPattern.replaceAll("%", "^%").replaceAll("_", "^_");
}
System.out.println("case sensitive " +deptPattern);
ctfPredicate = criteriaBuilder.like((Expression)root.get("name"), "%" + deptPattern + "%", '^');
criteriaQuery.where(criteriaBuilder.and(ctfPredicate));
TypedQuery<Object> typedQuery = entitymanager.createQuery(criteriaQuery);
List<Object> resultlist = typedQuery.getResultList();
printResult(resultlist);
The resultList does not return anything, whereas db has 2 entries with dname Sales.
Query getting printed :
select department0_.deptno as deptno0_, department0_.loc as loc0_, department0_.dname as dname0_ from mydept department0_ where department0_.dname like ? escape ?
Database used is Oracle and JPA2.0 vendor is EclipseLink
The db had entries with name as Sales, without %.
And the input pattern was given as 'Sales%'. I changed it to 'Sales', since the code was adding '%' to the input.
And the result list returned both the entries.
I am developing an application using hibernate. When I try to create a Login page, The problem of Sql Injection arises.
I have the following code:
#Component
#Transactional(propagation = Propagation.SUPPORTS)
public class LoginInfoDAOImpl implements LoginInfoDAO{
#Autowired
private SessionFactory sessionFactory;
#Override
public LoginInfo getLoginInfo(String userName,String password){
List<LoginInfo> loginList = sessionFactory.getCurrentSession().createQuery("from LoginInfo where userName='"+userName+"' and password='"+password+"'").list();
if(loginList!=null )
return loginList.get(0);
else return null;
}
}
How will i prevent Sql Injection in this scenario ?The create table syntax of loginInfo table is as follows:
create table login_info
(user_name varchar(16) not null primary key,
pass_word varchar(16) not null);
Query q = sessionFactory.getCurrentSession().createQuery("from LoginInfo where userName = :name");
q.setParameter("name", userName);
List<LoginInfo> loginList = q.list();
You have other options too, see this nice article from mkyong.
You need to use named parameters to avoid sql injection. Also (nothing to do with sql injection but with security in general) do not return the first result but use getSingleResult so if there are more than one results for some reason, the query will fail with NonUniqueResultException and login will not be succesful
Query query= sessionFactory.getCurrentSession().createQuery("from LoginInfo where userName=:userName and password= :password");
query.setParameter("username", userName);
query.setParameter("password", password);
LoginInfo loginList = (LoginInfo)query.getSingleResult();
What is SQL Injection?
SQL Injection happens when a rogue attacker can manipulate the query
building process so that he can execute a different SQL statement than
what the application developer has originally intended
How to prevent the SQL injection attack
The solution is very simple and straight-forward. You just have to make sure that you always use bind parameters:
public PostComment getPostCommentByReview(String review) {
return doInJPA(entityManager -> {
return entityManager.createQuery("""
select p
from PostComment p
where p.review = :review
""", PostComment.class)
.setParameter("review", review)
.getSingleResult();
});
}
Now, if some is trying to hack this query:
getPostCommentByReview("1 AND 1 >= ALL ( SELECT 1 FROM pg_locks, pg_sleep(10) )");
the SQL Injection attack will be prevented:
Time:1, Query:["select postcommen0_.id as id1_1_, postcommen0_.post_id as post_id3_1_, postcommen0_.review as review2_1_ from post_comment postcommen0_ where postcommen0_.review=?"], Params:[(1 AND 1 >= ALL ( SELECT 1 FROM pg_locks, pg_sleep(10) ))]
JPQL Injection
SQL Injection can also happen when using JPQL or HQL queries, as demonstrated by the following example:
public List<Post> getPostsByTitle(String title) {
return doInJPA(entityManager -> {
return entityManager.createQuery(
"select p " +
"from Post p " +
"where" +
" p.title = '" + title + "'", Post.class)
.getResultList();
});
}
The JPQL query above does not use bind parameters, so it’s vulnerable to SQL injection.
Check out what happens when I execute this JPQL query like this:
List<Post> posts = getPostsByTitle(
"High-Performance Java Persistence' and " +
"FUNCTION('1 >= ALL ( SELECT 1 FROM pg_locks, pg_sleep(10) ) --',) is '"
);
Hibernate executes the following SQL query:
Time:10003, QuerySize:1, BatchSize:0, Query:["select p.id as id1_0_, p.title as title2_0_ from post p where p.title='High-Performance Java Persistence' and 1 >= ALL ( SELECT 1 FROM pg_locks, pg_sleep(10) ) --()=''"], Params:[()]
Dynamic queries
You should avoid queries that use String concatenation to build the query dynamically:
String hql = " select e.id as id,function('getActiveUser') as name from " + domainClass.getName() + " e ";
Query query=session.createQuery(hql);
return query.list();
If you want to use dynamic queries, you need to use Criteria API instead:
Class<Post> entityClass = Post.class;
CriteriaBuilder cb = entityManager.getCriteriaBuilder();
CriteriaQuery<Tuple> query = cb.createTupleQuery();
Root<?> root = query.from(entityClass);
query.select(
cb.tuple(
root.get("id"),
cb.function("now", Date.class)
)
);
return entityManager.createQuery(query).getResultList();
I would like to add here that is a peculiar SQL Injection that is possible with the use of Like queries in searches.
Let us say we have a query string as follows:
queryString = queryString + " and c.name like :name";
While setting the name parameter, most would generally use this.
query.setParameter("name", "%" + name + "%");
Now, as mentioned above traditional parameter like "1=1" cannot be injected because of the TypedQuery and Hibernate will handle it by default.
But there is peculiar SQL Injection possible here which is because of the LIKE Query Structure which is the use of underscores
The underscore wildcard is used to match exactly one character in
MySQL meaning, for example, select * from users where user like
'abc_de'; This will produce outputs as users that start with abc, end
with de and have exactly 1 character in between.
Now, if in our scenario, if we set
name="_" produces customers whose name is at least 1 letter
name="__" produces customers whose name is at least 2 letters
name="___" produces customers whose name is at least 3 letters
and so on.
Ideal fix:
To mitigate this, we need to escape all underscores with a prefix .
___ will become \_\_\_ (equivalent to 3 raw underscores)
Likewise, the vice-versa query will also result in an injection in which %'s need to be escaped.
We should always try to use stored Procedures in general to prevent SQLInjection.. If stored procedures are not possible; we should try for Prepared Statements.
I have the following criteria query:
String cat = "H";
Criteria criteria = currentSession().createCriteria(this.getPersistentClass()).
add(Restrictions.ne("category", cat)).
createAlias("employees", "emp").
createAlias("emp.company", "company");
Disjunction disjunction = Restrictions.disjunction();
for(Region r: regions){
disjunction.add(Restrictions.eq("company.region", r));
}
criteria.add(disjunction);
if(status != null) {
criteria.add(Restrictions.eq("status", status));
}
if (period != null) {
criteria.add(Restrictions.eq("period", period));
}
criteria.setProjection(Projections.groupProperty("id")) //this line was added to try to "fix" the error, but it still happened.
criteria.addOrder(Order.asc("id"));
I guess a query that explains my criteria query could be:
select n.* from NOMINATION n
join NOMINEE i on n.NOM_ID = i.NOM_ID
join EMPLOYEE e on e.EMP_ID = i.EMP_ID
join COMPANY c on c.COMPANY_CODE = e.COMPANY_CODE
where n.CATEGORY_CODE!='H' and (c.REGION_ID = ? or c.REGION_ID = ? or c.REGION_ID = ?) and n.STATUS_ID = ? and n.PERIOD_ID = ?
order by n.NOM_ID
What I am trying to do here, is pretty confusing but for the most part it works except when I add this specific line (though the query works fine):
criteria.addOrder(Order.asc("id"));
and then I get error:
java.sql.SQLException: Column "NOMINATION.NOM_ID" is invalid in the ORDER BY clause because it is not contained in either an aggregate function or the GROUP BY clause.
Which I suspect is something that has to do with SQL-SERVER. I am already grouping by id. So what am I doing wrong here, or should I just use HQL?
Your current query seems to be a simple Query which doesn't have any group function used or not a group by query. According to your current requirements you do not have to use this line.
criteria.setProjection(Projections.groupProperty("id")).addOrder(Order.asc("id"));
Or you have to modify your sql statements.