I'd like to optimize the current merge batch for about hundreds of thousands entities, using JDBC 7 and SQL Server 2014.
The prepared statement syntax is somewhat like this:
BEGIN
DECLARE #field1 type1 = ?;
DECLARE #field2 type2 = ?;
MERGE INTO dbo.target_table AS TARGET
USING (SELECT __field1 = #field1, __field2 = #field2) AS SOURCE
ON (TARGET.field1 = SOURCE.__field1)
WHEN MATCHED THEN UPDATE SET field1 = #field1, field2 = #field2
WHEN NOT MATCHED BY TARGET THEN INSERT (field1, field2) VALUES (#field1, #field2);
END
The program then binds POJOs values into the declared above variables.
After reading the documentation of SQL Server, it seems the batched statement will be executed separately on server side, furthermore each merge requires a full table search which could lead to long waiting time.
I wonder whether i can optimize the process by using DECLARE #my_table_var table(field1 type1, field2 type2), inserting data into this #my_table_var then performing merge like :
MERGE INTO dbo.target_table AS TARGET
USING #my_table_var AS SOURCE
ON (TARGET.field1 = #my_table_var.field1)
WHEN MATCHED THEN UPDATE SET TARGET.field1 = #my_table_var.field1, TARGET.field2 = #my_table_var.field2
WHEN NOT MATCHED BY TARGET THEN INSERT (field1, field2) VALUES (#my_table_var.field1, #my_table_var.field2);
Related
I'm using JDBI3 (and would like to use #SQLUpdate) and an Oracle DB.
I want to insert an item with 4 columns into the table if it does not exist, if it does exist I want to instead update 3 of the 4 values of the item. If it wasn't Oracle I would've used some ON DUPLICATE_KEY logic but that does not exist in Oracle. I read some things about using Merge but the Queries seemed really wonky for what I was trying to do. Any tips on what to look for?
Additional question: If it is Merge I should use (with some form of sub queries I assume), how does the query affect performance? I think this database is quite write heavy.
MERGE INTO device db USING (SELECT 'abc' AS col1, 'bcd' as col2, 'cde' as col3, 'def' as col4 FROM DUAL) input
on (db.col1 = input.col1 AND db.col2= input.col2)
WHEN MATCHED THEN UPDATE
SET db.col4 = input.col4
WHEN NOT MATCHED THEN INSERT
(db.col1, db.col2, db.col3, db.col4)
VALUES (input.col1, input.col2, input.col3, input.col4)
Merge it is. Performs well.
Dummy example based on your description:
merge into target_table a
using source_table b
on (a.id = b.id)
when matched then update set
a.name = b.name,
a.job = b.job,
a.sal = b.sal
when not matched then
insert (id, name, job, sal)
values (b.id, b.name, b.job, b.sal);
I generate the SQL template like this with jOOQ 3.11.11.
DSLContext context = new DefaultDSLContext(conf);
Query query = context.select()
.from("table1")
.where(DSL.field("report_date").eq(DSL.param("bizdate")))
.orderBy(DSL.param("sort"));
String sqlTemp = context.renderNamedParams(query);
SQL template:
select *
from table1
where report_date = :bizdate
order by :sort
The SQL template is stored and the params are decided at realtime query condition.
ResultQuery resultQuery = context.resultQuery(sqlTemp, DSL.param("bizdate", "20190801"), DSL.param("sort", "id desc"));
The realtime SQL:
select *
from table1
where report_date = '20190801'
order by 'id desc'
There is something wrong with the order by clause.
So. How to replace the order by param sort with "id desc" or "name asc" and eliminate the quotes?
DSL.param() creates a bind variable, which is generated as ? in SQL, or :bizdate if you choose to use named parameters, or '20190801' if you choose to inline the bind variables. More about bind variables can be seen here.
You cannot use DSL.param() to generate column references or keywords. A column expression (e.g. a reference) is described in the jOOQ expression tree by the Field type. Keywords are described by the Keyword type, but you probably do not want to go this low level. Instead you want to handle some of the logic in your query expression. For example:
String sortField = "id";
SortOrder sortOrder = SortOrder.ASC;
Query query = context.select()
.from("table1")
.where(DSL.field("report_date").eq(DSL.param("bizdate")))
.orderBy(DSL.field(sortField).sort(sortOrder));
The mistake you're making is to think that you can use a single SQL template for all sorts of different dynamic SQL queries, but what if you're dynamically adding another predicate? Or another join? Or another column? You'd have to build a different jOOQ expression tree anyway. Just like here. You could store two SQL strings (one for each sort order), and repeat that for each sort column.
But, instead of pre-generating a single SQL string, I recommend you extract a function that takes the input parameters and generates the query every time afresh, e.g.:
ResultQuery<?> query(String bizDate, Field<?> sortField, SortOrder sortOrder) {
return context.selectFrom("table1")
.where(field("report_date").eq(bizDate))
.orderBy(sortField.sort(sortOrder));
}
Here is some further reading about using jOOQ for dynamic SQL:
https://www.jooq.org/doc/latest/manual/sql-building/dynamic-sql
https://blog.jooq.org/2017/01/16/a-functional-programming-approach-to-dynamic-sql-with-jooq
I'm confusing with implementation of CRUD methods for DAODatabase (for Oracle 11 xe).
The problem is that the "U"-method (update) in case of storing in generally to a Map collection inserts a new element or renews it (key-value data like ID:AbstractBusinessObject) in a Map collection. And you don't care about it, when you write something like myHashMap.add(element). This method (update) is widely used in project's business logic.
Obviously, in case of using Oracle I must care about both inserting and renewing of existing elements. But I'm stucked to choose the way how to implement it:
There is no intrinsic function for so-called UPSERT in Oracle (at least in xe11g r2 version). However, I can emulate necessary function by SQL-query like this:
INSERT INTO mytable (id1, t1)
SELECT 11, 'x1' FROM DUAL
WHERE NOT EXISTS (SELECT id1 FROM mytble WHERE id1 = 11);
UPDATE mytable SET t1 = 'x1' WHERE id1 = 11;
(src:http://stackoverflow.com/a/21310345/2938167)
By using this kind of query (first - insert, second - update) I presume that the data mostly will be inserted not updated (at least it will be rather rare). (May it be not optimal for concurrency?).
Ok, it is possible. But at this point I'm confusing to decide:
-- should I write an SQL function (with approriate arguments of course) for this and call it via Java
-- or should I simply handle a serie of queries for preparedStatements and do them via .executeUpdate/.executeQuery? Should I handle the whole UPSERT SQL code for one preparedStatment or split it into several SQL-queries and prepared statements inside one method's body? (I'm using Tomcat's pool of connections and I pass a connection instance via static method getConnection() to each method implementation in DAODatabase) ?
Is there another possibility to solve the UPSERT quest?
The equivalent to your UPSERT statement would seem to be to use MERGE:
MERGE INTO mytable d
USING ( SELECT 11 AS id, 'x1' AS t1 FROM DUAL ) s
ON ( d.id = s.id )
WHEN NOT MATCHED THEN
INSERT ( d.id, d.t1 ) VALUES ( s.id, s.t1 )
WHEN MATCHED THEN
UPDATE SET d.t1 = s.t1;
You could also use (or wrap in a procedure):
DECLARE
p_id MYTABLE.ID%TYPE := 11;
p_t1 MYTABLE.T1%TYPE := 'x1';
BEGIN
UPDATE mytable
SET t1 = p_t1
WHERE id = p_id;
IF SQL%ROWCOUNT = 0 THEN
INSERT INTO mytable ( id, t1 ) VALUES ( p_id, p_t1 );
END IF;
END;
/
However, when you are handling a CRUD request - if you are doing a Create action then it should be represented by an INSERT (and if something already exists then you ought to throw the equivalent of the HTTP status code 400 Bad Request or 409 Conflict, as appropriate) and if you are doing an Update action it should be represented by an UPDATE (and if nothing is there to update then return the equivalent error to 404 Not Found.
So, while MERGE fits your description I don't think it is representative of a RESTful action as you ought to be separating the actions to their appropriate end-points rather than combining then into a joint action.
I'm using org.jooq.impl.Executor to create a query that I want to pass to JdbcOperations. I'm passing parameter values by using .where(Condition...).
After chaining methods for the executor, I get a Query object. The problem is when I call query.getSQL(), the returned query string contains parameters ?,?,? instead of my inserted values.
This is the code I'm using to build the SQL query.
Note that TableA has three foreign keys to TableB. I tried to use JOIN ON with OR to join TableA and TableB but the performance was too slow.
Query query = executor.select(fieldsToSelect)
.from("TableA")
.join("TableB").on("TableA.FirstForeignKey = TableB.TableBID")
.join("TableC")
.on("TableC.TableCID = TableB.TableCForeignKey")
.where(condition)
.union(executor.select(fieldsToSelect)
.from("TableA")
.join("TableB").on("TableA.SecondForeignKey = TableB.TableBID")
.join("TableC")
.on("TableC.TableCID = TableB.TableCForeignKey")
.where(condition)
.union(executor.select(fieldsToSelect)
.from("TableA")
.join("TableB").on("TableA.ThirdForeignKey = TableB.TableBID")
.join("TableC")
.on("TableC.TableCID = TableB.TableCForeignKey")
.where(condition)));
This is how I create Condition object for the executor:
MySQLFactory.fieldByName(Integer.class, TABLE_NAME_TABLEA, "TableAID")
.in(ArrayUtils.toObject(ids));
This is how I perform the query:
jdbcOperations.query(query.getSQL(),query.getBindValues().toArray(), myMapper);
How can I correctly map parameters with values and use the query with JdbcOperations?
By default, the Query.getSQL() method returns a query with bind variables (?). You'll then pass that SQL string to a PreparedStatement and bind the variables individually, extracting them first via Query.getBindValue()
An alternative version of the Query.getSQL(boolean) or Query.getSQL(ParamType) method will indicate to jOOQ that the bind variables should be inlined into the SQL string, in order to form a static SQL statement, not a prepared statement.
You can also tell jOOQ to generate all SQL statements as static statements by using StatementType.STATIC_STATEMENT on the Settings that you provide the jOOQ Configuration with.
This is all documented here:
http://www.jooq.org/doc/latest/manual/sql-building/bind-values/inlined-parameters/
In MySQL, if you specify ON DUPLICATE KEY UPDATE and a row is inserted that would cause a duplicate value in a UNIQUE index or PRIMARY KEY, an UPDATE of the old row is performed. For example, if column a is declared as UNIQUE and contains the value 1, the following two statements have identical effect:
INSERT INTO table (a,b,c) VALUES (1,2,3)
ON DUPLICATE KEY UPDATE c=c+1;
UPDATE table SET c=c+1 WHERE a=1;
I don't believe I've come across anything of the like in T-SQL. Does SQL Server offer anything comparable to MySQL's ON DUPLICATE KEY UPDATE?
I was surprised that none of the answers on this page contained an example of an actual query, so here you go:
A more complex example of inserting data and then handling duplicate
MERGE
INTO MyBigDB.dbo.METER_DATA WITH (HOLDLOCK) AS target
USING (SELECT
77748 AS rtu_id
,'12B096876' AS meter_id
,56112 AS meter_reading
,'20150602 00:20:11' AS time_local) AS source
(rtu_id, meter_id, meter_reading, time_local)
ON (target.rtu_id = source.rtu_id
AND target.time_local = source.time_local)
WHEN MATCHED
THEN UPDATE
SET meter_id = '12B096876'
,meter_reading = 56112
WHEN NOT MATCHED
THEN INSERT (rtu_id, meter_id, meter_reading, time_local)
VALUES (77748, '12B096876', 56112, '20150602 00:20:11');
There's no DUPLICATE KEY UPDATE equivalent, but MERGE and WHEN MATCHED might work for you
Inserting, Updating, and Deleting Data by Using MERGE
You can try the other way around. It does the same thing more or less.
UPDATE tablename
SET field1 = 'Test1',
field2 = 'Test2'
WHERE id = 1
IF ##ROWCOUNT = 0
INSERT INTO tablename
(id,
field1,
field2)
VALUES (1,
'Test1',
'Test2')
SQL Server 2008 has this feature, as part of TSQL.
See documentation on MERGE statement here - http://msdn.microsoft.com/en-us/library/bb510625.aspx
SQL server 2000 onwards has a concept of instead of triggers, which can accomplish the wanted functionality - although there will be a nasty trigger hiding behind the scenes.
Check the section "Insert or update?"
http://msdn.microsoft.com/en-us/library/aa224818(SQL.80).aspx