We Keep Coding

Design approach for a SQL query builder with Java

In my webapp I need to create a query engine module where the user selects in the view what columns and filters he want and get the datas related to these queries.
So, I have to build SQL dynamic queries with this kind of format :
SELECT {columns} FROM MainTable FULL OUTER JOIN SecondTable ON ... FULL OUTER JOIN ThirdTable ON ... WHERE {filters}
The columns and filters are known at runtime. (read-only)
Actually, I have a big SQL Server view (Mapped entity in Java) which make 3 FULL OUTER JOIN of others SQL server views. And I build the query in the source code by parsing keywords as AND, OR, number, date, text, ...
Finally, I return a response in the front-end module a table with datas.
But, I'm facing performance issues (scalability) with this method and I'm looking for a more efficient way to do that.
Is it possible to split the SELECT query mentionned above in subqueries to improve performances ? Or there is better design approach for that ?
Here the request I actually use (names changed) with Hibernate :
SELECT * FROM BigView "+WHERE+" OPTION(ROBUST PLAN)
(BigView is aggregation of 4 views : RootTable, Table2, Table3, Table4)
Btw, I know "+WHERE+" is a bad practice but this is not the concern of this topic
I thought using this instead (delete BigView in SQL Server and the related entity) but it actually does same thing even if there is a little performance gain with the columns restriction :
SELECT ROW_NUMBER() OVER (ORDER BY id) AS rownum, tmp.* FROM ( SELECT '' AS id, "+columns+" "+
"FROM RootTable FULL OUTER JOIN "+
"Table2 ON RootTable.RT_ID = Table2.RT_N FULL OUTER JOIN "+
"Table3 ON RootTable.RT_ID = Table3.RT_ID FULL OUTER JOIN "+
"Table4 ON RootTable.RT_ID = Table4.RT_N "+
WHERE+" ) AS tmp
My previous deleted question was not enough focused so I hope this one is correct.
EDIT : I add this filter (WHERE clause) to show what can be requested for example :
RT_ID>'10' AND ( Table2_Topic LIKE '%test%' OR Table3_Date=CONVERT(datetime, '24/09/2020', 103) OR ( Table3_N is not null and Table2_ID<>0 ) )

EntityManager.createNativeQuery returning list of objects instead of list of BigDecimal when using Pagination

I am trying to use Pagination with EntityManager.createNativeQuery(). Below is the skeleton code that I am using:
var query = em.createNativeQuery("select distinct id from ... group by ... having ...");
List<BigDecimal> results = query
.setMaxResults(pageSize)
.setFirstResult(pageNumber * pageSize)
.getResultList();
When pageNumber is 0 (first page), I get the expected List of BigDecimals:
But as soon as pageNumber > 0 (example, second page), I get a List of Objects, and each object in this list seems to contain two BigDecimals, the first of which contains the value from the db, and the second BigDecimal seems to be the position of this row.
and obviously I get this exception
java.lang.ClassCastException: class [Ljava.lang.Object; cannot be cast to class java.math.BigDecimal
Can someone please explain this discrepancy, and how this can be fixed to always return a List of BigDecimals? Thank you.
Update-1 : I have created a sample project to reproduce this issue. I was able to reproduce this issue only with an Oracle database. With H2 database, it worked fine, and I consistently got a list of BigDecimals irrelevant of the page number.
Update-2 : I have also created a sample project with H2 where it works without this issue.

The problem that you are running into is that your OracleDialect adds a column to its selected ResultSet. It wraps the query that you are running as discussed in SternK's answer.
If you were using the Hibernate SessionFactory and the Session interfaces, then the function that you would be looking for would be the "addScalar" method. Unfortunately, there doesn't seem to be an implementation in pure JPA (see the question asked here: Does JPA have an equivalent to Hibernate SQLQuery.addScalar()?).
I would expect your current implementation to work just fine in DB2, H2, HSQL, Postgres, MySQL (and a few other DB engines). However, in Oracle, it adds a row-number column to the ResultSet which means that Hibernate gets 2 columns from the ResultSet. Hibernate does not implement any query parsing in this case, which means that it simply parses the ResultSet into your List. Since it gets 2 values, it converts them into an Object[] rather than a BigDecimal.
As a caveat, relying on the JDBC driver to provide the expected-data-type is a bit dangerous, since Hibernate will ask the JDBC driver which data-type it suggests. In this case, it suggests a BigDecimal, but under certain conditions and certain implementations would be allowed to return a Double or some other type.
You have a couple options then.
You can modify your oracle-dialect (as SternK) suggests. This will take advantage of an alternate oracle-paging implementation.
If you are not opposed to having hibnerate-specific aspects in your JPA implementation, then you can take advantage of additional hibernate functions that are not offered in the JPA standard. (See the following code...)
List<BigDecimal> results = entitymanager.createNativeQuery("select distinct id from ... group by ... having ...")
.unwrap(org.hibernate.query.NativeQuery.class)
.addScalar("id", BigDecimalType.INSTANCE)
.getResultList();
System.out.println(results);
This does have the advantage of explicitly telling hibnerate, that you are only interested in the "id" column of your ResultSet, and that hibernate needs to explicitly convert to the returned object to a BigDecimal, should the JDBC-driver decide that a different type would be more appropriate as a default.

The root cause of your problem in the way how the pagination implemented in your hibernate oracle dialect.
There are two cases:
When we have setFirstResult(0) the following sql will be generated:
-- setMaxResults(5).setFirstResult(0)
select * from (
select test_id from TST_MY_TEST -- this is your initial query
)
where rownum <= 5;
As you can see, this query returns exactly the same columns list as your initial query, and therefore you do not have problem with this case.
When we set setFirstResult in not 0 value the following sql will be generated:
-- setMaxResults(5).setFirstResult(2)
select * from (
select row_.*, rownum rownum_
from (
select test_id from TST_MY_TEST -- this is your initial query
) row_
where rownum <= 5
)
where rownum_ > 2
As you can see, this query returns the columns list with additional rownum_ column, and therefore you do have the problem with casting this result set to the BigDecimal.
Solution
If you use Oracle 12c R1 (12.1) or higher you can override this behavior in your dialect using new row limiting clause in this way:
import org.hibernate.dialect.Oracle12cDialect;
import org.hibernate.dialect.pagination.AbstractLimitHandler;
import org.hibernate.dialect.pagination.LimitHandler;
import org.hibernate.dialect.pagination.LimitHelper;
import org.hibernate.engine.spi.RowSelection;
public class MyOracleDialect extends Oracle12cDialect
{
private static final AbstractLimitHandler LIMIT_HANDLER = new AbstractLimitHandler() {
#Override
public String processSql(String sql, RowSelection selection) {
final boolean hasOffset = LimitHelper.hasFirstRow(selection);
final StringBuilder pagingSelect = new StringBuilder(sql.length() + 50);
pagingSelect.append(sql);
/*
see the documentation https://docs.oracle.com/database/121/SQLRF/statements_10002.htm#BABHFGAA
(Restrictions on the row_limiting_clause)
You cannot specify this clause with the for_update_clause.
*/
if (hasOffset) {
pagingSelect.append(" OFFSET ? ROWS");
}
pagingSelect.append(" FETCH NEXT ? ROWS ONLY");
return pagingSelect.toString();
}
#Override
public boolean supportsLimit() {
return true;
}
};
public MyOracleDialect()
{
}
#Override
public LimitHandler getLimitHandler() {
return LIMIT_HANDLER;
}
}
and then use it.
<property name="hibernate.dialect">com.me.MyOracleDialect</property>
For my test data set for the following query:
NativeQuery query = session.createNativeQuery(
"select test_id from TST_MY_TEST"
).setMaxResults(5).setFirstResult(2);
List<BigDecimal> results = query.getResultList();
I got:
Hibernate:
/* dynamic native SQL query */
select test_id from TST_MY_TEST
OFFSET ? ROWS FETCH NEXT ? ROWS ONLY
val = 3
val = 4
val = 5
val = 6
val = 7
P.S. See also HHH-12087
P.P.S I simplified my implementation of the AbstractLimitHandler by removing checking presents FOR UPDATE clause. I think we will not have nothing good in this case and with this checking.
For example for the following case:
NativeQuery query = session.createNativeQuery(
"select test_id from TST_MY_TEST FOR UPDATE OF test_id"
).setMaxResults(5).setFirstResult(2);
hibernate (with Oracle12cDialect) will generate the following sql:
/* dynamic native SQL query */
select * from (
select
row_.*,
rownum rownum_
from (
select test_id from TST_MY_TEST -- initial sql without FOR UPDATE clause
) row_
where rownum <= 5
)
where rownum_ > 2
FOR UPDATE OF test_id -- moved for_update_clause
As you can see, hibernate tries to fix query by moving FOR UPDATE to the end of the query. But anyway, we will get:
ORA-02014: cannot select FOR UPDATE from view with DISTINCT, GROUP BY, etc.

I've simulated your consult and everything works fine. I've used DataJpaTest to instance entityManager for me, h2 memory database and JUnit 5 to run the test. See below:
#Test
public void shouldGetListOfSalaryPaginated() {
// given
Person alex = new Person("alex");
alex.setSalary(BigDecimal.valueOf(3305.33));
Person john = new Person("john");
john.setSalary(BigDecimal.valueOf(33054.10));
Person ana = new Person("ana");
ana.setSalary(BigDecimal.valueOf(1223));
entityManager.persist(alex);
entityManager.persist(john);
entityManager.persist(ana);
entityManager.flush();
entityManager.clear();
// when
List<BigDecimal> found = entityManager.createNativeQuery("SELECT salary FROM person").setMaxResults(2).setFirstResult(2*1).getResultList();
// then
Assertions.assertEquals(found.size(), 1);
Assertions.assertEquals(found.get(0).longValue(), 1223L);
}
I suggest that you review your native query. It's preferable that you use Criteria API instead and let native queries for extreme cases like complex consults.
Update
After the author posted the project, I could reproduce the problem and it was related to the oracle dialect. For unknown reason the query which is running for the second call is: select * from ( select row_.*, rownum rownum_ from ( SELECT c.SHOP_ID FROM CUSTOMER c ) row_ where rownum <= ?) where rownum_ > ?, and that's why this is generating a bug, because it's querying 2 columns instead of only one. The undesired one is this rownum. For other dialects there is no such problem.
I suggest you try other oracle dialect version and whether none of them work, my final tip is try to do the pagination yourself.

After a lot of trails with different versions of different spring libraries, I was finally able to figure out the issue. In one of my attempts, the issue seems to have disappeared, as soon as I updated the spring-data-commons library from v2.1.5.RELEASE to v2.1.6.RELEASE. I looked up the changelog of this release, and this bug, which is related to this bug in spring-data-commons, is the root cause of this issue. I was able to fix the issue after upgrading the spring-data-commons library.

Checking if table exist or not

I am retrieving data from database using jdbc. In my code I am using 3-4 tables to get data. But sometimes if table is not present in database my code gives exception. How to handle this situation. I want my code to continue working for other tables even if one table is not present. Please help.
I have wrote a code like this
sql="select * from table"
now Result set and all.
If table is not present in database it give exception that no such table. I want to handle it. In this code I cannot take tables which are already present in advance . I want to check here itself if table is there or not.
Please do not mark it as a duplicate question. The link you shared doesnot give me required answer as in that question they are executing queries in database not through JDBC code

For Sybase ASE the easiest/quickest method would consist of querying the sysobjects table in the database where you expect the (user-defined) table to reside:
select 1 from sysobjects where name = 'table-name' and type = 'U'
if a record is returned => table exists
if no record is returned => table does not exist
How you use the (above) query is up to you ...
return a 0/1-row result set to your client
assign a value to a #variable
place in a if [not] exists(...) construct
use in a case statement
If you know for a fact that there won't be any other object types (eg, proc, trigger, view, UDF) in the database with the name in question then you could also use the object_id() function, eg:
select object_id('table-name')
if you receive a number => the object exists
if you receive a NULL => the object does not exist
While object_id() will obtain an object's id from the sysobjects table, it does not check for the object type, eg, the (above) query will return a number if there's a stored proc named 'table-name'.
As with the select/sysobjects query, how you use the function call in your code is up to you (eg, result set, populate #variable, if [not] exists() construct, case statement).
So, addressing the additional details provided in the comments ...
Assuming you're submitting a single batch that needs to determine table existence prior to running the desired query(s):
-- if table exists, run query(s); obviously if table does not exist then query(s) is not run
if exists(select 1 from sysobjects where name = 'table-name' and type = 'U')
begin
execute("select * from table-name")
end
execute() is required to keep the optimizer from generating an error that the table does not exist, ie, the query is not parsed/compiled unless the execute() is actually invoked
If your application can be written to use multiple batches, something like the following should also work:
# application specific code; I don't work with java but the gist of the operation would be ...
run-query-in-db("select 1 from sysobjects where name = 'table-name' and type = 'U'")
if-query-returns-a-row
then
run-query-in-db("select * from table-name")
fi

This is the way of checking if the table exists and drop it:
IF EXISTS (
SELECT 1
FROM sysobjects
WHERE name = 'a_table'
AND type = 'U'
)
DROP TABLE a_table
GO
And this is how to check if a table exists and create it.
IF NOT EXISTS (
SELECT 1
FROM sysobjects
WHERE name = 'a_table'
AND type = 'U'
)
EXECUTE("CREATE TABLE a_table (
col1 int not null,
col2 int null
)")
GO
(They are different because in table-drop a temporary table gets created, so if you try to create a new one you will get an exception that it already exists)

Before running the query which has some risk in table not existing, run the following sql query and check if the number of results is >= 1. if it is >= 1 then you are safe to execute the normal query. otherwise, do something to handle this situation.
SELECT count(*)
FROM information_schema.TABLES
WHERE (TABLE_SCHEMA = 'your_db_name') AND (TABLE_NAME = 'name_of_table')
I am no expert in Sybase but take a look at this,
exec sp_tables '%', '%', 'master', "'TABLE'"
Sybase Admin

Postgresql - ERROR : tuple concurrently updated

The following query is performed concurrently by two threads logged in with two different users:
WITH raw_stat AS (
SELECT
host(client_addr) as client_addr,
pid ,
usename
FROM
pg_stat_activity
WHERE
usename = current_user
)
INSERT INTO my_stat(id, client_addr, pid, usename)
SELECT
nextval('mystat_sequence'), t.client_addr, t.pid, t.usename
FROM (
SELECT
client_addr, pid, usename
FROM
raw_stat s
WHERE
NOT EXISTS (
SELECT
NULL
FROM
my_stat u
WHERE
current_date = u.creation
AND
s.pid = u.pid
AND
s.client_addr = u.client_addr
AND
s.usename = u.usename
)
) t;
From time to time, I get the following error:
tuple concurrently updated
I can't figure out what throw this error and why this error is thrown. Can you shed a light ?
Here is the sql definition of the table mystat.
mystats.sql
CREATE TABLE mystat
(
id bigint NOT NULL,
creation date NOT NULL DEFAULT current_date,
client_addr text NOT NULL,
pid integer NOT NULL,
usename name NOT NULL,
CONSTRAINT mystat_pkey PRIMARY KEY (id)
)
WITH (
OIDS=FALSE
);

This isn't really an answer - so much as maybe helping someone else who stumbles on this error.
In my case, I was trying to be fancy and encapsulate the creation of all my functions within one function.
Something like
CREATE OR REPLACE FUNCTION main_func()
BEGIN
CREATE OR REPLACE FUNCTION child_func1()
BEGIN
END
CREATE OR REPLACE FUNCTION child_func1()
BEGIN
END
main func stuff...
END
For whatever reason, I could call this function no problem from inside pgAdmin. And I could call it as much as I wanted from Java -> MyBatis.
However, as soon as I started calling the function from two different threads, I got the error from the OP: ERROR : tuple concurrently updated
The fix was, simply take those child functions out of the main function, and maintain them separately.
Looking back on it, it's a pretty bad idea to be creating functions as a result of calling a function. However, the idea was to 'encapsulate' all the functionality together.
Hope this helps someone.

If the pg hackers threads are anything to go by, the error kicks in when the same row is concurrently being updated by competing transactions. In your case it's likely due to the not exists() clause, which can potentially yield true and two competing inserts of the same tuple.
To work around it, you'd want to either use more robust locking (e.g. a predicate lock), serializable isolation level, or place the needed logic in an upsert statement (can be done using a function with an exception block).

From the docs(https://www.postgresql.org/docs/current/functions-sequence.html) from Postgres, Because sequences are non-transactional, changes made by setval are not undone if the transaction rolls back.
It means that you need to update provide thread safety by yourself using transaction so running the query inside transaction might fix your problem.

I manage to solve my problem by changing my query to this one:
INSERT INTO my_stat(id, client_addr, pid, usename)
SELECT
nextval('mystat_sequence'), client_addr, pid, usename
FROM (
SELECT
host(client_addr) as client_addr,
pid ,
usename
FROM
pg_stat_activity
WHERE
usename = current_user
) s
WHERE
NOT EXISTS (
SELECT
NULL
FROM
my_stat u
WHERE
current_date = u.creation
AND
s.pid = u.pid
AND
s.client_addr = u.client_addr
AND
s.usename = u.usename
);
I think something happened under the hood right from the Postgresql internals but I can't figure out what ...

MySQL database query and update

I was wondering if somebody knew a better way to do the following:
I need to query a database and return a value (in this case an int), then using this value, calculate the new value and update the database with this new value.
My current approach is using a method to get the current int value from the database, passing this value to another method to perform the calculations and then passing the new value to a third method to update the database.
So, the problem(?) with this is that it opens a new connection from the pool when getting the initial value from the db and then when updating it. Obviously it closes the connection at the end of the method but is there some easier / better way of doing this ? It seems a bit messy.

Try this:
SELECT fieldValue FROM table_name FOR UPDATE;
UPDATE table_name SET fieldToUpdate = fieldValue + 1;
See the UPDATE Syntax

You don't have to open a new connection for each query. Just open a connection at the start of your request, save its reference to a global variable that you can use in all your methods, and close it at the end of your request.

If you can do the calculations in SQL:
UPDATE TableToUpdate
SET ColumnB =
Calculations( ( SELECT ColumnA
FROM TableToSelect
WHERE (conditions for selecting)
)
)
WHERE (conditions for updating)

Depending on your requirements you could exploit multiple-table UPDATE:
UPDATE TableToUpdate U JOIN TableToSelect S ON ( -- join condition for selection value to process )
SET U.ColumnB = Calculations( S.ColumnC )
WHERE U.ColumnC = -- whatever selection condition