We Keep Coding

JPA Query yields different results on different developers' machines

When I run our Java program, I get a weird error that none of my team mates get, because a JPA-Query gives a different result on my machine, even if I and a fellow developer check out the exact same git commit, and the same DB content, build it and then debug it together.
What this JPA-Query does is, it basically checks if a certain entity already exists in the database. It explicitly excludes entries that have the same Id, so the entity doesn't get compared to itself. It appears that this doesn't work on my machine. Here's a dummy version of the code:
import com.mysema.query.BooleanBuilder;
import com.mysema.query.jpa.impl.JPAQuery;
public class ExampleClass {
#Autowired
protected ClassThatExtendsJpaRepository customerDAO;
public void checkUniqueness(Customer inputCustomer) throws NotUniqueException {
//Or-condition
BooleanBuilder condition = new BooleanBuilder();
Condition.or(QCustomer.customer.registerNr.eq(inputCustomer.getRegisterNr());
Condition.or(QCustomer.customer.registerNr.eq(inputCustomer.getRegisterNr());
JPAQuery query = new JPAQuery().from(Qcustomer.customer).where(condition);
//This should exclude customers of the same id, but it has no effect
query.where(Qcustomer.customer.id.ne(inputCustomer.getId());
Customer existingCustomer = customerDAO.findOne(query);
//Result: finds customer with same id! Should find nothing!
if(existingCustomer != null) {
throw new NotUniqueException();
}
}
}
As you can see, aside from the check that the existing entity doesn't have the same ID as the one we're trying to compare it to, there are also other conditions connected by an OR. For simplicity, I've used the same condition for both in this example. The way the code is written now, it will find the entity with the same ID in the database and throw the NotUniqueException even though it shouldn't. But remove one of the Condition.or, and it works. This leads me to suspect that the query is putting the brackets wrong.
According to the debugger, the query is
Select customer
from Customer customer
where (customer.registerNr = ?1 or customer.registerNr = ?1) and customer.id <> ?2
(And for the record: when I run this directly on my DB it works correctly, finding nothing.)
But I suspect it's actually running it as if the brackets were different:
Select customer
from Customer customer
where customer.registerNr = ?1 or (customer.registerNr = ?1 and customer.id <> ?2)
Either way, this still doesn't explain why this occurs, and why it's only on my machine.
We don't want to change the code, since this bug is only on my machine and doesn't stop me from working. So hopefully if I find the cause I can fix it some other way.
Versions:
Spring-data-jpa: 1.9.6.RELEASE
com.mysema.querydsl: 3.7.4
Hibernate: 4.2.21.Final
OJDBC: 19.17.0.0
Database: Oracle Database 21c Express Edition Release 21.0.0.0.0 - Production
Java: 8

The solution was: I downgraded Oracle to version 18, like all the other developers.
Yes, that's right, apparently the queries can give different results in Oracle 21.

EntityManager.createNativeQuery returning list of objects instead of list of BigDecimal when using Pagination

I am trying to use Pagination with EntityManager.createNativeQuery(). Below is the skeleton code that I am using:
var query = em.createNativeQuery("select distinct id from ... group by ... having ...");
List<BigDecimal> results = query
.setMaxResults(pageSize)
.setFirstResult(pageNumber * pageSize)
.getResultList();
When pageNumber is 0 (first page), I get the expected List of BigDecimals:
But as soon as pageNumber > 0 (example, second page), I get a List of Objects, and each object in this list seems to contain two BigDecimals, the first of which contains the value from the db, and the second BigDecimal seems to be the position of this row.
and obviously I get this exception
java.lang.ClassCastException: class [Ljava.lang.Object; cannot be cast to class java.math.BigDecimal
Can someone please explain this discrepancy, and how this can be fixed to always return a List of BigDecimals? Thank you.
Update-1 : I have created a sample project to reproduce this issue. I was able to reproduce this issue only with an Oracle database. With H2 database, it worked fine, and I consistently got a list of BigDecimals irrelevant of the page number.
Update-2 : I have also created a sample project with H2 where it works without this issue.

The problem that you are running into is that your OracleDialect adds a column to its selected ResultSet. It wraps the query that you are running as discussed in SternK's answer.
If you were using the Hibernate SessionFactory and the Session interfaces, then the function that you would be looking for would be the "addScalar" method. Unfortunately, there doesn't seem to be an implementation in pure JPA (see the question asked here: Does JPA have an equivalent to Hibernate SQLQuery.addScalar()?).
I would expect your current implementation to work just fine in DB2, H2, HSQL, Postgres, MySQL (and a few other DB engines). However, in Oracle, it adds a row-number column to the ResultSet which means that Hibernate gets 2 columns from the ResultSet. Hibernate does not implement any query parsing in this case, which means that it simply parses the ResultSet into your List. Since it gets 2 values, it converts them into an Object[] rather than a BigDecimal.
As a caveat, relying on the JDBC driver to provide the expected-data-type is a bit dangerous, since Hibernate will ask the JDBC driver which data-type it suggests. In this case, it suggests a BigDecimal, but under certain conditions and certain implementations would be allowed to return a Double or some other type.
You have a couple options then.
You can modify your oracle-dialect (as SternK) suggests. This will take advantage of an alternate oracle-paging implementation.
If you are not opposed to having hibnerate-specific aspects in your JPA implementation, then you can take advantage of additional hibernate functions that are not offered in the JPA standard. (See the following code...)
List<BigDecimal> results = entitymanager.createNativeQuery("select distinct id from ... group by ... having ...")
.unwrap(org.hibernate.query.NativeQuery.class)
.addScalar("id", BigDecimalType.INSTANCE)
.getResultList();
System.out.println(results);
This does have the advantage of explicitly telling hibnerate, that you are only interested in the "id" column of your ResultSet, and that hibernate needs to explicitly convert to the returned object to a BigDecimal, should the JDBC-driver decide that a different type would be more appropriate as a default.

The root cause of your problem in the way how the pagination implemented in your hibernate oracle dialect.
There are two cases:
When we have setFirstResult(0) the following sql will be generated:
-- setMaxResults(5).setFirstResult(0)
select * from (
select test_id from TST_MY_TEST -- this is your initial query
)
where rownum <= 5;
As you can see, this query returns exactly the same columns list as your initial query, and therefore you do not have problem with this case.
When we set setFirstResult in not 0 value the following sql will be generated:
-- setMaxResults(5).setFirstResult(2)
select * from (
select row_.*, rownum rownum_
from (
select test_id from TST_MY_TEST -- this is your initial query
) row_
where rownum <= 5
)
where rownum_ > 2
As you can see, this query returns the columns list with additional rownum_ column, and therefore you do have the problem with casting this result set to the BigDecimal.
Solution
If you use Oracle 12c R1 (12.1) or higher you can override this behavior in your dialect using new row limiting clause in this way:
import org.hibernate.dialect.Oracle12cDialect;
import org.hibernate.dialect.pagination.AbstractLimitHandler;
import org.hibernate.dialect.pagination.LimitHandler;
import org.hibernate.dialect.pagination.LimitHelper;
import org.hibernate.engine.spi.RowSelection;
public class MyOracleDialect extends Oracle12cDialect
{
private static final AbstractLimitHandler LIMIT_HANDLER = new AbstractLimitHandler() {
#Override
public String processSql(String sql, RowSelection selection) {
final boolean hasOffset = LimitHelper.hasFirstRow(selection);
final StringBuilder pagingSelect = new StringBuilder(sql.length() + 50);
pagingSelect.append(sql);
/*
see the documentation https://docs.oracle.com/database/121/SQLRF/statements_10002.htm#BABHFGAA
(Restrictions on the row_limiting_clause)
You cannot specify this clause with the for_update_clause.
*/
if (hasOffset) {
pagingSelect.append(" OFFSET ? ROWS");
}
pagingSelect.append(" FETCH NEXT ? ROWS ONLY");
return pagingSelect.toString();
}
#Override
public boolean supportsLimit() {
return true;
}
};
public MyOracleDialect()
{
}
#Override
public LimitHandler getLimitHandler() {
return LIMIT_HANDLER;
}
}
and then use it.
<property name="hibernate.dialect">com.me.MyOracleDialect</property>
For my test data set for the following query:
NativeQuery query = session.createNativeQuery(
"select test_id from TST_MY_TEST"
).setMaxResults(5).setFirstResult(2);
List<BigDecimal> results = query.getResultList();
I got:
Hibernate:
/* dynamic native SQL query */
select test_id from TST_MY_TEST
OFFSET ? ROWS FETCH NEXT ? ROWS ONLY
val = 3
val = 4
val = 5
val = 6
val = 7
P.S. See also HHH-12087
P.P.S I simplified my implementation of the AbstractLimitHandler by removing checking presents FOR UPDATE clause. I think we will not have nothing good in this case and with this checking.
For example for the following case:
NativeQuery query = session.createNativeQuery(
"select test_id from TST_MY_TEST FOR UPDATE OF test_id"
).setMaxResults(5).setFirstResult(2);
hibernate (with Oracle12cDialect) will generate the following sql:
/* dynamic native SQL query */
select * from (
select
row_.*,
rownum rownum_
from (
select test_id from TST_MY_TEST -- initial sql without FOR UPDATE clause
) row_
where rownum <= 5
)
where rownum_ > 2
FOR UPDATE OF test_id -- moved for_update_clause
As you can see, hibernate tries to fix query by moving FOR UPDATE to the end of the query. But anyway, we will get:
ORA-02014: cannot select FOR UPDATE from view with DISTINCT, GROUP BY, etc.

I've simulated your consult and everything works fine. I've used DataJpaTest to instance entityManager for me, h2 memory database and JUnit 5 to run the test. See below:
#Test
public void shouldGetListOfSalaryPaginated() {
// given
Person alex = new Person("alex");
alex.setSalary(BigDecimal.valueOf(3305.33));
Person john = new Person("john");
john.setSalary(BigDecimal.valueOf(33054.10));
Person ana = new Person("ana");
ana.setSalary(BigDecimal.valueOf(1223));
entityManager.persist(alex);
entityManager.persist(john);
entityManager.persist(ana);
entityManager.flush();
entityManager.clear();
// when
List<BigDecimal> found = entityManager.createNativeQuery("SELECT salary FROM person").setMaxResults(2).setFirstResult(2*1).getResultList();
// then
Assertions.assertEquals(found.size(), 1);
Assertions.assertEquals(found.get(0).longValue(), 1223L);
}
I suggest that you review your native query. It's preferable that you use Criteria API instead and let native queries for extreme cases like complex consults.
Update
After the author posted the project, I could reproduce the problem and it was related to the oracle dialect. For unknown reason the query which is running for the second call is: select * from ( select row_.*, rownum rownum_ from ( SELECT c.SHOP_ID FROM CUSTOMER c ) row_ where rownum <= ?) where rownum_ > ?, and that's why this is generating a bug, because it's querying 2 columns instead of only one. The undesired one is this rownum. For other dialects there is no such problem.
I suggest you try other oracle dialect version and whether none of them work, my final tip is try to do the pagination yourself.

After a lot of trails with different versions of different spring libraries, I was finally able to figure out the issue. In one of my attempts, the issue seems to have disappeared, as soon as I updated the spring-data-commons library from v2.1.5.RELEASE to v2.1.6.RELEASE. I looked up the changelog of this release, and this bug, which is related to this bug in spring-data-commons, is the root cause of this issue. I was able to fix the issue after upgrading the spring-data-commons library.

Spring JDBCTemplate Always Throws QueryTimeOutException

This is something that I've been scratching my head with - especially since it's infuriating to deal with.
Consider the following code:
String query = "UPDATE ORDERS SET VOLUME=?,CONTRACT_ID=?,PROJECT_ID=?,WORKSITE_ID=?,DROPZONE_ID=?,DESCRIPTION_ID=?,MANAGER_ID=?,DELIVERY_DATE=?,REVISION=REVISION+1) WHERE ID=?";
jdbcTemplate.update(query, orderEntity.getVolume(), orderEntity.getContractNo(), orderEntity.getProjectID(), orderEntity.getWorksiteID(), orderEntity.getDropzoneID(), orderEntity.getDescriptionID(), orderEntity.getManagerID(), orderEntity.getDeliveryDate(), id);
We can see that the SQL query is incorrect - and will therefore throw some SQL error but one might have missed that. Spring will (for me) throw a QueryTimeoutException in response to this. I'm sort of okay with that but it's not helpful.
Now let's try
String query = "INSERT INTO ORDERS(ID,REISION,CONTRACT_ID,PROJECT_ID,WORKSITE_ID,DROPZONE_ID,DESCRIPTION_ID,MANAGER_ID,VOLUME,DELIVERY_DATE) VALUES(?,?,?,?,?,?,?,?,?,?)";
jdbcTemplate.update(query, id, revision, etc);
Another spelling mistake that's easily missed - REVISION is misspelled as REISION) Spring throws another QueryTimeoutException again. This now means that if I get that exception I don't actually know what it is. Is it a syntax error? Is it a column spelling error? Is it the (much harder to notice) fact that the foreign key constraint not being followed?
While debugging, this is quite possibly the most infuriating thing ever - all I know is that my query failed to run. How can I get something useful? Is there something I've not added to my pom.xml file?
EDIT:
Here's a nicer example. I have a DESCRIPTIONS table, with an ID, REVISION and TEXT column. All of those are marked as not being nullable.
DescriptionEntity descriptionEntity = new DescriptionEntity("newDesc", 1, null);
String query = "INSERT INTO DESCRIPTIONS (ID,REVISION,TEXT) VALUES(?,?,?)";
jdbcTemplate.update(query, descriptionEntity.getID(), 1, descriptionEntity.getText());
That will also throw a query timeout exception, when running the query in mysql gives me ERROR 1048 (23000): Column 'TEXT' cannot be null
This is, to put it politely, a bit of a pain.

It's not a spelling mistake in the first example, as you left out the opening paren. I would say this isn't an issue with Spring or JDBC, but rather your DB is trying to process the SQL, waiting for more input or something, and never returning.
In the second one, I am not sure what you are talking about since I don't know the table design. I have to assume what you mean is ID is not unique? Again, I wouldn't blame Spring or JDBC, maybe the drive, most likely the database server.
Keep in mind, in a lot of cases, the way SQL is handled in the user Client UI is not the same as how it gets handled through JDBC. For instance, in SQL Server the SQL is set as a string, the passed in parameters set as variables, and it uses sp_executesql() to run it. I discovered that when I had a report that ran PERFECTLY fine through SQL Studio Manager client, but blew up when I ran it live because the query plan optimizer took a different path due to the differences in how the SQL was ran.

This is quite possibly the most stupid error I've ever come across: the issue was in how Maven resolved all the dependencies.
The requirement for Spring Security was placed before the JDBC requirement. That made Spring Security pull down org.springframework:spring-tx:jar:3.0.7.RELEASE:compile which satisfied the dependency for JDBC. Moving the JDBC requirement up meant JDBC pulled down org.springframework:spring-tx:jar:3.2.2.RELEASE:compile.

Hibernate: Issues with running update query

I see there are two ways to create update query in Hibernate. First you can go with the standard approach where we have hql like:
Query q = session.createQuery("update" + LogsBean.class.getName() + " LogsBean " + "set LogsBean.jobId= :jobId where LogsBean.jobId= :oldValue ");
q.setLong("jobId", jobId);
q.setLong("oldValue", 0);
return q.executeUpdate();
or we can go and run
getHibernateTemplate.saveorupdate(jobId);
Now am getting java.lang.IllegalArgumentException: node to traverse cannot be null! on running first query and am not sure hwo to provide condition in getHibernateTemplate example, i want to update jobIds in log table whose value matches 0 and so i want to run something like
Update logs set jobId = 23 where jobId = 0
Above is the simple sql query that I am trying to run but I want to run this via hibernate, tried couple ways but it is not working, any suggestions?
Update:
As noted by Jeff, issue was not having space after update and so that issue got resolved but still values are not updated, i have updated show_sql true for hibernate and checking what could be the cause of the issue, will be running query generated by hibernate to run again db and see if records are updated.

Just a few things that might help you to resolve this:
What does .executeUpdate() return, 0 (as it did not update any
rows)?
Does it throw a HibernateException that you are
silently catching or rethrowing?
Which FlushMode do you have configured?
Does the update get to the DB? You could switch on the query log for your DB server.

JPA (Hibernate) Native Query for Prepared Statement SLOW

Having strange performance issue using Hibernate 3.3.2GA behind JPA (and the rest of the Hibernate packages included in JBoss 5.)
I'm using Native Query, and assembling SQL into a prepared statement.
EntityManager em = getEntityManager(MY_DS);
final Query query = em.createNativeQuery(fullSql, entity.getClass());
The SQL has a lot of joins, but is actually very basic, with a single parameter. Like:
SELECT field1, field2, field3 FROM entity left join entity2 on... left join entity3 on
WHERE stringId like ?
and the query runs in under a second on MSSQL Studio.
If I add
query.setParameter(0, "ABC123%");
The query will pause for 9 seconds
2012-01-20 14:36:21 - TRACE: - AbstractBatcher.getPreparedStatement:(484) | preparing statement
2012-01-20 14:36:21 - TRACE: - StringType.nullSafeSet:(133) | binding 'ABC123%' to parameter: 1
2012-01-20 14:36:30 - DEBUG: - AbstractBatcher.logOpenResults:(382) | about to open ResultSet (open ResultSets: 0, globally: 0)
However, if I just replace the "?" with the value (making it not a Prepared Statement, but just a straight SQL query.
fullSql = fullSql.replace("?", "'ABC123%'");
the query will complete in less that a second.
I would really prefer to us a Prepared Statement (the input for the parameters is being extracted from user data) to prevent injection attacks.
Tracing down the slow point in the code, I arrived deep within the jtds-1.2.2 package. The offending line seems to be SharedSocket line 841 "getIn().readFully(hdrBuf);" Nothing really obvious there though...
private byte[] readPacket(byte buffer[])
throws IOException {
//
// Read rest of header
try {
getIn().readFully(hdrBuf);
} catch (EOFException e) {
throw new IOException("DB server closed connection.");
}
Arrived to through this stack...
at net.sourceforge.jtds.jdbc.SharedSocket.readPacket(SharedSocket.java:841)
at net.sourceforge.jtds.jdbc.SharedSocket.getNetPacket(SharedSocket.java:722)
at net.sourceforge.jtds.jdbc.ResponseStream.getPacket(ResponseStream.java:466)
at net.sourceforge.jtds.jdbc.ResponseStream.read(ResponseStream.java:103)
at net.sourceforge.jtds.jdbc.ResponseStream.peek(ResponseStream.java:88)
at net.sourceforge.jtds.jdbc.TdsCore.wait(TdsCore.java:3928)
at net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:1045)
at net.sourceforge.jtds.jdbc.TdsCore.microsoftPrepare(TdsCore.java:1178)
at net.sourceforge.jtds.jdbc.ConnectionJDBC2.prepareSQL(ConnectionJDBC2.java:657)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:776)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1808)
at org.hibernate.loader.Loader.doQuery(Loader.java:697)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.doList(Loader.java:2228)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125)
at org.hibernate.loader.Loader.list(Loader.java:2120)
at org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:312)
at org.hibernate.impl.SessionImpl.listCustomQuery(SessionImpl.java:1722)
at org.hibernate.impl.AbstractSessionImpl.list(AbstractSessionImpl.java:165)
at org.hibernate.impl.SQLQueryImpl.list(SQLQueryImpl.java:175)
at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:67)

I'll leave this question and answer out here in case anyone has the same issue in the future.
The issue is in the way the JTDS drivers send the parameter strings to MSSQL. Apparently Java will attempt to send the parameters Unicode by default, and MSSQL will translate it to Ascii. Why that takes 9 seconds, I do not know.
Lot's of references to this out there, but nothing that helped my till I was able to isolate that it was an issue with the driver to MSSQL connection.
This link was helpful:
[http://server.pramati.com/blog/2010/06/02/perfissues-jdbcdrivers-mssqlserver/]
This is the string using the Microsoft driver.
jdbc:sqlserver://localhost\SQLEXPRESS;
DatabaseName=TESTDB;
sendStringParametersAsUnicode=false
You just need to get the sendStringParametersAsUnicode=false passed to your driver URL setup and you are good.

Check the query plans that SQL server is producing. Prepared statements can be especially problematic.
Let me explain...
If you do this:
SELECT field1, field2, field3 FROM entity left join entity2 on... left join entity3 on
WHERE stringId like 'ABC123%';
and you have an index on "stringId" SQL server knows it can use it.
However if you do this:
SELECT field1, field2, field3 FROM entity left join entity2 on... left join entity3 on
WHERE stringId like ?;
SQL server doesn't know it can use the index when it creates the prepared statement (as you could fill in the parameter with '%ABC123' instead of 'ABC123%') and thus may choose a completely different query plan.

And another answer for people potentially using Oracle with a similar Unicode problem...
Check to make sure someone hasn't set the property oracle.jdbc.defaultNChar=true
This is sometimes done to resolve unicode problems but it means all columns are treated as nvarchars. If you have an index on a varchar column, it won't be used because oracle has to use a function to convert the character encoding.