I have an hibernate query with paging. I want to see parameter values related to the paging. I use SQLServer2008Dialect so my query looks like:
WITH query AS (/* criteria query */ select
ROW_NUMBER() OVER (
order by
this_.event_id desc)
...
...
) SELECT
*
FROM
query
WHERE
__hibernate_row_nr__ BETWEEN ? AND ?
I set
hibernate.show_sql = true
hibernate.format_sql = true
hibernate.use_sql_comments = true
in hibernate config used by my application.
I enabled also logging query parameters in log4j by setting
org.hibernate.type.descriptor.sql.BasicBinder to TRACE level.
This works fine for fine for all other parameters but I can't see parameter values related to the row number limit. Is there any way to inspect which are the current limit parameter values?
In the past I've always used that configuration with hibernate, but recently I've switched to use BoneCP which allows to me log the exact query being executed to avoid having the same problem you are facing.
If you can use BoneCP, you could configure your data source to enable logging statements, and set a log4j logger for com.jolbox.bonecp to debug, and you are set.
Related
I am trying to use Pagination with EntityManager.createNativeQuery(). Below is the skeleton code that I am using:
var query = em.createNativeQuery("select distinct id from ... group by ... having ...");
List<BigDecimal> results = query
.setMaxResults(pageSize)
.setFirstResult(pageNumber * pageSize)
.getResultList();
When pageNumber is 0 (first page), I get the expected List of BigDecimals:
But as soon as pageNumber > 0 (example, second page), I get a List of Objects, and each object in this list seems to contain two BigDecimals, the first of which contains the value from the db, and the second BigDecimal seems to be the position of this row.
and obviously I get this exception
java.lang.ClassCastException: class [Ljava.lang.Object; cannot be cast to class java.math.BigDecimal
Can someone please explain this discrepancy, and how this can be fixed to always return a List of BigDecimals? Thank you.
Update-1 : I have created a sample project to reproduce this issue. I was able to reproduce this issue only with an Oracle database. With H2 database, it worked fine, and I consistently got a list of BigDecimals irrelevant of the page number.
Update-2 : I have also created a sample project with H2 where it works without this issue.
The problem that you are running into is that your OracleDialect adds a column to its selected ResultSet. It wraps the query that you are running as discussed in SternK's answer.
If you were using the Hibernate SessionFactory and the Session interfaces, then the function that you would be looking for would be the "addScalar" method. Unfortunately, there doesn't seem to be an implementation in pure JPA (see the question asked here: Does JPA have an equivalent to Hibernate SQLQuery.addScalar()?).
I would expect your current implementation to work just fine in DB2, H2, HSQL, Postgres, MySQL (and a few other DB engines). However, in Oracle, it adds a row-number column to the ResultSet which means that Hibernate gets 2 columns from the ResultSet. Hibernate does not implement any query parsing in this case, which means that it simply parses the ResultSet into your List. Since it gets 2 values, it converts them into an Object[] rather than a BigDecimal.
As a caveat, relying on the JDBC driver to provide the expected-data-type is a bit dangerous, since Hibernate will ask the JDBC driver which data-type it suggests. In this case, it suggests a BigDecimal, but under certain conditions and certain implementations would be allowed to return a Double or some other type.
You have a couple options then.
You can modify your oracle-dialect (as SternK) suggests. This will take advantage of an alternate oracle-paging implementation.
If you are not opposed to having hibnerate-specific aspects in your JPA implementation, then you can take advantage of additional hibernate functions that are not offered in the JPA standard. (See the following code...)
List<BigDecimal> results = entitymanager.createNativeQuery("select distinct id from ... group by ... having ...")
.unwrap(org.hibernate.query.NativeQuery.class)
.addScalar("id", BigDecimalType.INSTANCE)
.getResultList();
System.out.println(results);
This does have the advantage of explicitly telling hibnerate, that you are only interested in the "id" column of your ResultSet, and that hibernate needs to explicitly convert to the returned object to a BigDecimal, should the JDBC-driver decide that a different type would be more appropriate as a default.
The root cause of your problem in the way how the pagination implemented in your hibernate oracle dialect.
There are two cases:
When we have setFirstResult(0) the following sql will be generated:
-- setMaxResults(5).setFirstResult(0)
select * from (
select test_id from TST_MY_TEST -- this is your initial query
)
where rownum <= 5;
As you can see, this query returns exactly the same columns list as your initial query, and therefore you do not have problem with this case.
When we set setFirstResult in not 0 value the following sql will be generated:
-- setMaxResults(5).setFirstResult(2)
select * from (
select row_.*, rownum rownum_
from (
select test_id from TST_MY_TEST -- this is your initial query
) row_
where rownum <= 5
)
where rownum_ > 2
As you can see, this query returns the columns list with additional rownum_ column, and therefore you do have the problem with casting this result set to the BigDecimal.
Solution
If you use Oracle 12c R1 (12.1) or higher you can override this behavior in your dialect using new row limiting clause in this way:
import org.hibernate.dialect.Oracle12cDialect;
import org.hibernate.dialect.pagination.AbstractLimitHandler;
import org.hibernate.dialect.pagination.LimitHandler;
import org.hibernate.dialect.pagination.LimitHelper;
import org.hibernate.engine.spi.RowSelection;
public class MyOracleDialect extends Oracle12cDialect
{
private static final AbstractLimitHandler LIMIT_HANDLER = new AbstractLimitHandler() {
#Override
public String processSql(String sql, RowSelection selection) {
final boolean hasOffset = LimitHelper.hasFirstRow(selection);
final StringBuilder pagingSelect = new StringBuilder(sql.length() + 50);
pagingSelect.append(sql);
/*
see the documentation https://docs.oracle.com/database/121/SQLRF/statements_10002.htm#BABHFGAA
(Restrictions on the row_limiting_clause)
You cannot specify this clause with the for_update_clause.
*/
if (hasOffset) {
pagingSelect.append(" OFFSET ? ROWS");
}
pagingSelect.append(" FETCH NEXT ? ROWS ONLY");
return pagingSelect.toString();
}
#Override
public boolean supportsLimit() {
return true;
}
};
public MyOracleDialect()
{
}
#Override
public LimitHandler getLimitHandler() {
return LIMIT_HANDLER;
}
}
and then use it.
<property name="hibernate.dialect">com.me.MyOracleDialect</property>
For my test data set for the following query:
NativeQuery query = session.createNativeQuery(
"select test_id from TST_MY_TEST"
).setMaxResults(5).setFirstResult(2);
List<BigDecimal> results = query.getResultList();
I got:
Hibernate:
/* dynamic native SQL query */
select test_id from TST_MY_TEST
OFFSET ? ROWS FETCH NEXT ? ROWS ONLY
val = 3
val = 4
val = 5
val = 6
val = 7
P.S. See also HHH-12087
P.P.S I simplified my implementation of the AbstractLimitHandler by removing checking presents FOR UPDATE clause. I think we will not have nothing good in this case and with this checking.
For example for the following case:
NativeQuery query = session.createNativeQuery(
"select test_id from TST_MY_TEST FOR UPDATE OF test_id"
).setMaxResults(5).setFirstResult(2);
hibernate (with Oracle12cDialect) will generate the following sql:
/* dynamic native SQL query */
select * from (
select
row_.*,
rownum rownum_
from (
select test_id from TST_MY_TEST -- initial sql without FOR UPDATE clause
) row_
where rownum <= 5
)
where rownum_ > 2
FOR UPDATE OF test_id -- moved for_update_clause
As you can see, hibernate tries to fix query by moving FOR UPDATE to the end of the query. But anyway, we will get:
ORA-02014: cannot select FOR UPDATE from view with DISTINCT, GROUP BY, etc.
I've simulated your consult and everything works fine. I've used DataJpaTest to instance entityManager for me, h2 memory database and JUnit 5 to run the test. See below:
#Test
public void shouldGetListOfSalaryPaginated() {
// given
Person alex = new Person("alex");
alex.setSalary(BigDecimal.valueOf(3305.33));
Person john = new Person("john");
john.setSalary(BigDecimal.valueOf(33054.10));
Person ana = new Person("ana");
ana.setSalary(BigDecimal.valueOf(1223));
entityManager.persist(alex);
entityManager.persist(john);
entityManager.persist(ana);
entityManager.flush();
entityManager.clear();
// when
List<BigDecimal> found = entityManager.createNativeQuery("SELECT salary FROM person").setMaxResults(2).setFirstResult(2*1).getResultList();
// then
Assertions.assertEquals(found.size(), 1);
Assertions.assertEquals(found.get(0).longValue(), 1223L);
}
I suggest that you review your native query. It's preferable that you use Criteria API instead and let native queries for extreme cases like complex consults.
Update
After the author posted the project, I could reproduce the problem and it was related to the oracle dialect. For unknown reason the query which is running for the second call is: select * from ( select row_.*, rownum rownum_ from ( SELECT c.SHOP_ID FROM CUSTOMER c ) row_ where rownum <= ?) where rownum_ > ?, and that's why this is generating a bug, because it's querying 2 columns instead of only one. The undesired one is this rownum. For other dialects there is no such problem.
I suggest you try other oracle dialect version and whether none of them work, my final tip is try to do the pagination yourself.
After a lot of trails with different versions of different spring libraries, I was finally able to figure out the issue. In one of my attempts, the issue seems to have disappeared, as soon as I updated the spring-data-commons library from v2.1.5.RELEASE to v2.1.6.RELEASE. I looked up the changelog of this release, and this bug, which is related to this bug in spring-data-commons, is the root cause of this issue. I was able to fix the issue after upgrading the spring-data-commons library.
I am using Vaadin Flow together with Spring-data-jpa and Hibernate with DB2/400 dialect. I am trying to use paging queries as my dataset could be very large. I have Hibernate logging on so that I can see the statements that Hibernate is executing. It works for the first page as the Hibernate query just asks for the first 50 records. However for the second page it asks for 100 records and filters out the first 50 but the query does not return any results. A slightly simplified version of the Hibernate generated query is:
select * from (
select inner2_.*, rownumber() over(order by order of inner2_) as rownumber_ from (
select * from flxalll1 flxalll1x0_ where upper(flxalll1x0_.aoukey) like upper('%te%') fetch first 100 rows only
) as inner2_
) as inner1_ where rownumber_ > 50 order by rownumber_;
I have run this myself using the IBMi Run SQL Script tool and no results are returned. However, if I just do the inner two selects:
select inner2_.*, rownumber() over(order by order of inner2_) as rownumber_ from (
select * from flxalll1 flxalll1x0_ where upper(flxalll1x0_.aoukey) like upper('%te%') fetch first 100 rows only
) as inner2_ ;
I get the expects list of result, though of course all of them and not just the last 50. I have done some more experimentation and discovered that (unsurprisingly) this works:
select * from (
select * from flxalll1
);
in that it lists all the records, but this:
select * from (
select * from (
select * from flxalll1
)
);
produces no records.
Obviously there is no sense in that but I'm wondering if there is a problem with DB2/400 in that it won't do a select with two nested sub-selects, or something like that, and is that the reason why my original query does not return any records?
Solving the problem could be tricky but for now at least I am just trying to work out where the problem lies.
I have now solved my problem by not using the (IBMi) logical as mentioned in my comment above but by allowing spring data jpa to do the UNION. I created an abstract super class for my 10 tables and changed their Java class declarations to extend it.
I am indebted to this post and to the answer by Patrice Blanchardie:
Union tables with spring data jpa
TL;DR
I have a Spring Boot application that makes use of parquet files stored on the file system. To access them we are using Apache Drill.
Since I have multiple users that might access them, I've set up a connection pool in Spring.
When I'm using the connection pool, Drill somehow executes a "limit 0" query before executing my actual query, and this affect performances. The same "limit 0" query is NOT executed when I run my queries through a simple Statement obtained from direct Connection.
This seems to be related to the fact that Spring JdbcTemplate makes use of PreparedStatements instead of simple Statements.
Is there a way to get rid of those "limit 0" queries?
-- Details --
The connection pool in the Spring configuration class looks like this:
#Bean
#ConfigurationProperties(prefix = "datasource.parquet")
#Qualifier("parquetDataSource")
public DataSource parquetDataSource() {
return DataSourceBuilder.create().build();
}
And the corresponding properties in the development profile YML file are:
datasource:
parquet:
url: jdbc:drill:drillbit=localhost:31010
jdbcUrl: jdbc:drill:drillbit=localhost:31010
jndiName: jdbc/app_parquet
driverClassName: org.apache.drill.jdbc.Driver
maximumPoolSize: 5
initialSize: 1
maxIdle: 10
maxActive: 20
validation-query: SELECT 1 FROM sys.version
test-on-borrow: true
When I execute a query using the JdbcTemplate created with the mentioned Drill DataSource, 3 different queries might be executed:
the validation query SELECT 1 FROM sys.version;
a "limit 0" query that looks like SELECT * FROM (<my actual query>) LIMIT 0;
my actual query.
Here's the execution code (parquetJdbcTemplate is an instance of a class that extends org.springframework.jdbc.core.JdbcTemplate):
parquetJdbcTemplate.query(sqlQuery, namedParameters,
resultSet -> {
MyResultSet result = new MyResultSet();
while (resultSet.next()) {
// populate the "result" object
}
return result;
});
Here's a screenshot from the Profile page of my Drill monitor:
The bottom query is the "limit 0" one, then in the middle you have the validation query and on top (even if the query is not shown) the actual query that returns the data I want.
As you can see, the "limit 0" query takes more than 1/3 of the entire execution time to run. The validation query is fine, since the execution time is negligible and it's needed to check the connection.
The fact is, when I execute the same query using a Connection through the Drill driver (thus, with no pool), I only see my actual query in the UI monitor:
public void executeQuery(String myQuery) {
Class.forName("org.apache.drill.jdbc.Driver");
Driver.load();
Connection connection = DriverManager.getConnection("jdbc:drill:drillbit=localhost:31010");
Statement st = connection.createStatement();
ResultSet resultSet = st.executeQuery(myQuery);
while (resultSet.next()) {
// do stuff
}
}
As you can see, the total execution time improves by a lot (~14 seconds instead of ~26), just because the "limit 0" query is not executed.
As far as I know, those "limit 0" queries are executed to validate and get information about the underlying schema of the parquet files. Is there a way to disable them while using the connection pool? I ideally would like to still use PreparedStatements over simple Statements, but I could switch to simple Statements if needed, because I have full control over those queries (so, no SQL injection should be possible unless someone hacks the deployed artifacts).
You are right Drill executes limit 0 prior prepared statements to get information about schema. I don't think there is a way to disable such behavior. Though I can recommend to enable planner.enable_limit0_optimization option which is false by default, this may speed limit 0 query execution. Another way to speed limit 0 queries is to indicate schema explicitly using casts through the view usage or directly in queries.
Regarding not showing query, I think this was fixed in the latest Drill version.
Is there any way I can check, how is the query being framed or what values are being passed ? I want to check for this query :
String hql = "from Scheduled where stime <= current_time()"; // QUERY
List list = session.createQuery(hql).list();
I want to know what value of current_time() is being sent ?
current_time() is not replaced with a specific time in hibernate - it is passed as part of the SQL to the database and is evaluated there. Therefore, current_time() will be whatever the current time is on the database server at the time the statement is executed.
You can enable Hibernate logging and these two params should be of help to you:
org.hibernate.SQL - Log all SQL DML statements as they are
executed
org.hibernate.type - Log all JDBC parameters
If your underlying database is SQL Server, then another option is SQL Profiler.
Here are entries from log4j.properties which works for me to print hibernate queries and parameters.
log4j.logger.org.hibernate=DEBUG, stdout
log4j.logger.org.hibernate.SQL=DEBUG
log4j.logger.org.hibernate.type=TRACE,stdout
this is example of how sql is showed when using show_sql=true
Hibernate:
select
propertyse0_.entity_name as entity1_3_0_,
propertyse0_.entity_id as entity2_3_0_,
propertyse0_.entity_key as entity3_3_0_,
propertyse0_.key_type as key4_3_0_,
propertyse0_.boolean_val as boolean5_3_0_,
propertyse0_.double_val as double6_3_0_,
propertyse0_.string_val as string7_3_0_,
propertyse0_.long_val as long8_3_0_,
propertyse0_.int_val as int9_3_0_,
propertyse0_.date_val as date10_3_0_
from
OS_PROPERTYENTRY propertyse0_
where
propertyse0_.entity_name=?
and propertyse0_.entity_id=?
and propertyse0_.entity_key=?
possible to show value to gather with the sql rather than '?'
Set your logging leven to "TRACE".
In your log4j.properties (assuming your using Log4J):
log4j.logger.org.hibernate=TRACE
Will result in lots of logging tough...
You need to set up your logging framework to log this level of details. See here for the various loggers that Hibernate uses, and how to use them.
The particular one that you want is:
org.hibernate.type - Log all JDBC parameters
not directly. you can use log4jdbc to log all the data that is sent over jdbc. It has a logger that inlines the prepared statement values.