We Keep Coding

Hibernate+SQLServer / Batch insert only the new records

I'm trying, using Hibernate 4.3 and SQL-Server 2014, to perform batch insert into a table for only the entities that are not already stored.
I have created a simple table with a primary key definded to ignore duplicate keys
create table items
(
itemid uniqueidentifier not null,
itemname nvarchar(30) not null,
)
alter table items add constraint items_pk primary key ( itemid ) with ( ignore_dup_key = on );
Trying to perform a batch insert through the StatelessSession insert method, the batch insert may fail if one or more entities are already stored into the database table: Hibernate throws a StaleStateException:
org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1
at org.hibernate.jdbc.Expectations$BasicExpectation.checkBatched(Expectations.java:81)
at org.hibernate.jdbc.Expectations$BasicExpectation.verifyOutcome(Expectations.java:73)
at org.hibernate.engine.jdbc.batch.internal.NonBatchingBatch.addToBatch(NonBatchingBatch.java:63)
at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3124)
at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3581)
at org.hibernate.internal.StatelessSessionImpl.insert(StatelessSessionImpl.java:144)
at org.hibernate.internal.StatelessSessionImpl.insert(StatelessSessionImpl.java:123)
at it.test.testingestion.HibernateStatelessSessionPersisterImpl.persistData(HibernateStatelessSessionPersisterImpl.java:18)
at it.test.testingestion.App.main(App.java:76)
When the batch statements are completed Hibernate performs a check over the returned rows count that is different than the expected due to the ignore duplicate keys.
With JDBC, performing a batch insert using a prepared statement, the entities that are already stored into the destination table are skipped but the new entities are saved correctly.
How is it possible to configure Hibernate to perform batch insert ignoring the existing data or wihout performing a check of the affected rows?
Thanks a lot
Update #1
As workaround, to force the number of rows affected even if a duplicated insert occurs, I have created the following Hibernate Interceptor:
public class CustomInterceptor extends EmptyInterceptor {
private static final long serialVersionUID = -8022068618978801796L;
private String getTemporaryTableName() {
String currentThreadName = Thread.currentThread().getName();
return "##" + currentThreadName.replaceAll("[^A-Za-z0-9\\_]", "");
}
private void createTemporaryTable(Connection connection) {
String tempTableName = this.getTemporaryTableName();
String commandText = String.format("if (object_id('tempdb.dbo.%s') is null) begin create table [%s] ( dummyfield int ); insert into %s ( dummyfield ) values ( 0 ) end ", tempTableName, tempTableName, tempTableName);
try (PreparedStatement statement = connection.prepareStatement(commandText)) {
statement.execute();
connection.commit();
} catch (SQLException e) {
throw new RuntimeException(String.format("An error has been occurred trying to create the temporary table %s", tempTableName), e);
}
}
public CustomInterceptor(Connection connection) {
this.createTemporaryTable(connection);
}
#Override
public String onPrepareStatement(String sql) {
int ps = sql.toLowerCase().indexOf("insert into ");
if (ps == 0) {
String tableName = this.getTemporaryTableName();
return sql + "; if (##rowcount = 0) update [" + tableName + "] set dummyfield = 1";
}
return super.onPrepareStatement(sql);
}
}
The interceptor, when a new instance is created, creates a new temporary table inserting a new record.
When an insert statement is intercepted, an update of the record saved into the instanced temporary table is performed if no rows has been affected by the insert statement: this tricks Hibernate about the returned rows event if a duplicate entity is inserted and no StatelessSessionImpl exception is thrown.
Obviously the downside of the trick is the cost to perform the extra update for each row that is not inseted into the table.
Does anyone knows a better way, that does not affect the insert performances, to insert entities into a table thats ignore the duplicate entries using Hibernate?
Thanks

For better performance, I would prefer using JDBCBatchUpdate
Approach 1:
As you are filtering the new records, the number of records will not be a constraint. So you could specify an association mapping in the entity layer and could perform a Hibernate batch insert or JDBC Batch update.
Approach 2:
Using Native SQl Query
Session session = sessionFactory.openSession();
Transaction tx = session.beginTransaction();
//get Connction from Session
session.doWork(new Work() {
#Override
public void execute(Connection conn) throws SQLException {
PreparedStatement pstmt = null;
try{
String sqlInsert = "insert into tbl(name) values (?) ";
pstmt = conn.prepareStatement(sqlInsert );
int i=0;
for(String name : list){
pstmt .setString(1, name);
pstmt .addBatch();
//20 : JDBC batch size
if ( i % 20 == 0 ) {
pstmt .executeBatch();
}
i++;
}
pstmt .executeBatch();
}
finally{
pstmt .close();
}
}
});
tx.commit();
session.close();

Get the inserted id of the record before or after execute the insert statement in sql

I have this method in my DAO class to insert record to a table called idea this is my method:
public long addIdea(AddIdeaDto addIdeaDto, int userId) {
Connection connection = null;
PreparedStatement preparedStatement = null;
try {
connection = getConnection();
preparedStatement = connection.prepareStatement(
"INSERT INTO IDEA ( IDEA.I_ID,IDEA.I_NO,IDEA.I_APPROVER_NAME_CODE, IDEA.I_TITLE,IDEA.I_DESCRIPITION, IDEA.I_CREATED_DATE,IDEA.I_STATUS_CODE, "
+ "IDEA.I_IS_CODE, IDEA.I_CONTRIBUTION_CODE, IDEA.I_POSITIVE_IMPACT, IDEA.I_SECOND_MEMBER_ID,IDEA.I_THIRD_MEMBER_ID,IDEA.I_FOURTH_MEMBER_ID,"
+ "IDEA.I_FIFTH_MEMBER_ID, IDEA.I_POINTS,IDEA.I_CREATED_USER_ID)"
+ " VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)");
preparedStatement.executeQuery("SELECT IDEA_SEQ.nextval FROM DUAL");
// Set parameters
preparedStatement.setObject(1, Types.NUMERIC);
preparedStatement.setObject(2, Types.NUMERIC);
preparedStatement.setObject(3, addIdeaDto.getApproverNameCode());
preparedStatement.setString(4, addIdeaDto.getTitle());
preparedStatement.setString(5, addIdeaDto.getDescription());
preparedStatement.setDate(6, addIdeaDto.getCreatedDate() == null ? null
: new java.sql.Date(addIdeaDto.getCreatedDate().getTime()));
preparedStatement.setObject(7, addIdeaDto.getStatusCode());
preparedStatement.setObject(8, addIdeaDto.getIsNewCode());
preparedStatement.setObject(9, addIdeaDto.getContributionCode());
preparedStatement.setString(10, addIdeaDto.getPositiveImpact());
preparedStatement.setObject(11, addIdeaDto.getSecondMemberName());
preparedStatement.setObject(12, addIdeaDto.getThirdMemberName());
preparedStatement.setObject(13, addIdeaDto.getFourthMemberName());
preparedStatement.setObject(14, addIdeaDto.getFifthMemberName());
preparedStatement.setObject(15, addIdeaDto.getPoints());
preparedStatement.setInt(16, userId);
preparedStatement.executeQuery();
return addIdeaDto.getIdeaId();
} catch (Exception e) {
throw new RuntimeException(e);
} finally {
try {
preparedStatement.close();
} catch (SQLException e) {
throw new RuntimeException(e);
}
}
}
actually what I want is after or before the insert statement I want to get the id (IDEA_SEQ.nextval) and save it in a value in order to use it as an input to insert in anther table.
For example, I insert this record : id = 1 , no = 1, approver code = 2, title = 'test'.............
I want this value id = 1 to use it in order to insert in table A, A_id = 33, IDEA.I_ID = 1, A_name ='testing'
how i can achieve it in properer way?
I update the code based on the comments that i receive but I did not achieve it

Usually ID that need to be reuse can be handle using a previous and separate SQL query
previousPreparedStatement = connection.prepareStatement(
"select IDEA_SEQ.nextval as nextval from dual");
Result saved as a int or String parameter according to column (number or varchar) which is passed to the existing insert statement:
(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)");
Notice also an answer from DBA forum
you won't be able to use plain SQL to overcome this limitation: you will need some PL/SQL

A better way to handle this is the RETURNING INTO clause, which uses a single, atomic statement:
INSERT INTO mytable (id, col1, col2)
VALUES ( seq_id.nextval, c1, c2 )
RETURNING id INTO myval;

You can use PreparedStatement.getGeneratedKeys() to obtain the generated value. There is no need to use a separate statement:
You also can't prefix column names with the table name in list of columns of an INSERT statement.
String insert =
"INSERT INTO IDEA ( I_ID,I_NO,I_APPROVER_NAME_CODE, I_TITLE,I_DESCRIPITION, I_CREATED_DATE,I_STATUS_CODE, "
+ "I_IS_CODE, I_CONTRIBUTION_CODE, I_POSITIVE_IMPACT, I_SECOND_MEMBER_ID,I_THIRD_MEMBER_ID,I_FOURTH_MEMBER_ID,"
+ "I_FIFTH_MEMBER_ID, I_POINTS,I_CREATED_USER_ID)"
+ " VALUES (idea_seq.nextval,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)";
preparedStatement = connection.prepareStatement(insertSql, new String[] {"I_ID"});
preparedStatement.setInt(1, ???); // don't know where the value for I_NO comes from
preparedStatement.setString(2, addIdeaDto.getApproverNameCode());
preparedStatement.setString(3, addIdeaDto.getTitle());
... other parameters
preparedStatement.executeUpdate();
ResultSet rs = preparedStatement.getGeneratedKeys();
long newId = -1;
if (rs.next()) {
newId = rs.getLong("I_ID");
}
... use the NewId ...
The parameter new String[] {"I_ID"} for the prepareStatement() call tells the JDBC driver to return the generated value for that column. That value can be retrieved through getGeneratedKeys() which returns a ResultSet that contains one row for each inserted row (so exactly one in this case). The ID value can then be extracted from the ResultSet using the the usual getLong() (or getInt()) methods.

How can i insert data using procedure sql? [duplicate]

In my app I need to do a lot of INSERTS. Its a Java app and I am using plain JDBC to execute the queries. The DB being Oracle. I have enabled batching though, so it saves me network latencies to execute queries. But the queries execute serially as separate INSERTs:
insert into some_table (col1, col2) values (val1, val2)
insert into some_table (col1, col2) values (val3, val4)
insert into some_table (col1, col2) values (val5, val6)
I was wondering if the following form of INSERT might be more efficient:
insert into some_table (col1, col2) values (val1, val2), (val3, val4), (val5, val6)
i.e. collapsing multiple INSERTs into one.
Any other tips for making batch INSERTs faster?

This is a mix of the two previous answers:
PreparedStatement ps = c.prepareStatement("INSERT INTO employees VALUES (?, ?)");
ps.setString(1, "John");
ps.setString(2,"Doe");
ps.addBatch();
ps.clearParameters();
ps.setString(1, "Dave");
ps.setString(2,"Smith");
ps.addBatch();
ps.clearParameters();
int[] results = ps.executeBatch();

Though the question asks inserting efficiently to Oracle using JDBC, I'm currently playing with DB2 (On IBM mainframe), conceptually inserting would be similar so thought it might be helpful to see my metrics between
inserting one record at a time
inserting a batch of records (very efficient)
Here go the metrics
1) Inserting one record at a time
public void writeWithCompileQuery(int records) {
PreparedStatement statement;
try {
Connection connection = getDatabaseConnection();
connection.setAutoCommit(true);
String compiledQuery = "INSERT INTO TESTDB.EMPLOYEE(EMPNO, EMPNM, DEPT, RANK, USERNAME)" +
" VALUES" + "(?, ?, ?, ?, ?)";
statement = connection.prepareStatement(compiledQuery);
long start = System.currentTimeMillis();
for(int index = 1; index < records; index++) {
statement.setInt(1, index);
statement.setString(2, "emp number-"+index);
statement.setInt(3, index);
statement.setInt(4, index);
statement.setString(5, "username");
long startInternal = System.currentTimeMillis();
statement.executeUpdate();
System.out.println("each transaction time taken = " + (System.currentTimeMillis() - startInternal) + " ms");
}
long end = System.currentTimeMillis();
System.out.println("total time taken = " + (end - start) + " ms");
System.out.println("avg total time taken = " + (end - start)/ records + " ms");
statement.close();
connection.close();
} catch (SQLException ex) {
System.err.println("SQLException information");
while (ex != null) {
System.err.println("Error msg: " + ex.getMessage());
ex = ex.getNextException();
}
}
}
The metrics for 100 transactions :
each transaction time taken = 123 ms
each transaction time taken = 53 ms
each transaction time taken = 48 ms
each transaction time taken = 48 ms
each transaction time taken = 49 ms
each transaction time taken = 49 ms
...
..
.
each transaction time taken = 49 ms
each transaction time taken = 49 ms
total time taken = 4935 ms
avg total time taken = 49 ms
The first transaction is taking around 120-150ms which is for the query parse and then execution, the subsequent transactions are only taking around 50ms. (Which is still high, but my database is on a different server(I need to troubleshoot the network))
2) With insertion in a batch (efficient one) - achieved by preparedStatement.executeBatch()
public int[] writeInABatchWithCompiledQuery(int records) {
PreparedStatement preparedStatement;
try {
Connection connection = getDatabaseConnection();
connection.setAutoCommit(true);
String compiledQuery = "INSERT INTO TESTDB.EMPLOYEE(EMPNO, EMPNM, DEPT, RANK, USERNAME)" +
" VALUES" + "(?, ?, ?, ?, ?)";
preparedStatement = connection.prepareStatement(compiledQuery);
for(int index = 1; index <= records; index++) {
preparedStatement.setInt(1, index);
preparedStatement.setString(2, "empo number-"+index);
preparedStatement.setInt(3, index+100);
preparedStatement.setInt(4, index+200);
preparedStatement.setString(5, "usernames");
preparedStatement.addBatch();
}
long start = System.currentTimeMillis();
int[] inserted = preparedStatement.executeBatch();
long end = System.currentTimeMillis();
System.out.println("total time taken to insert the batch = " + (end - start) + " ms");
System.out.println("total time taken = " + (end - start)/records + " s");
preparedStatement.close();
connection.close();
return inserted;
} catch (SQLException ex) {
System.err.println("SQLException information");
while (ex != null) {
System.err.println("Error msg: " + ex.getMessage());
ex = ex.getNextException();
}
throw new RuntimeException("Error");
}
}
The metrics for a batch of 100 transactions is
total time taken to insert the batch = 127 ms
and for 1000 transactions
total time taken to insert the batch = 341 ms
So, making 100 transactions in ~5000ms (with one trxn at a time) is decreased to ~150ms (with a batch of 100 records).
NOTE - Ignore my network which is super slow, but the metrics values would be relative.

The Statement gives you the following option:
Statement stmt = con.createStatement();
stmt.addBatch("INSERT INTO employees VALUES (1000, 'Joe Jones')");
stmt.addBatch("INSERT INTO departments VALUES (260, 'Shoe')");
stmt.addBatch("INSERT INTO emp_dept VALUES (1000, 260)");
// submit a batch of update commands for execution
int[] updateCounts = stmt.executeBatch();

You'll have to benchmark, obviously, but over JDBC issuing multiple inserts will be much faster if you use a PreparedStatement rather than a Statement.

You can use this rewriteBatchedStatements parameter to make the batch insert even faster.
you can read here about the param: MySQL and JDBC with rewriteBatchedStatements=true

SQLite: The above answers are all correct. For SQLite, it is a little bit different. Nothing really helps, even to put it in a batch is (sometimes) not improving performance. In that case, try to disable auto-commit and commit by hand after you are done (Warning! When multiple connections write at the same time, you can clash with these operations)
// connect(), yourList and compiledQuery you have to implement/define beforehand
try (Connection conn = connect()) {
conn.setAutoCommit(false);
preparedStatement pstmt = conn.prepareStatement(compiledQuery);
for(Object o : yourList){
pstmt.setString(o.toString());
pstmt.executeUpdate();
pstmt.getGeneratedKeys(); //if you need the generated keys
}
pstmt.close();
conn.commit();
}

How about using the INSERT ALL statement ?
INSERT ALL
INTO table_name VALUES ()
INTO table_name VALUES ()
...
SELECT Statement;
I remember that the last select statement is mandatory in order to make this request succeed. Don't remember why though.
You might consider using PreparedStatement instead as well. lots of advantages !
Farid

You can use addBatch and executeBatch for batch insert in java See the Example : Batch Insert In Java

In my code I have no direct access to the 'preparedStatement' so I cannot use batch, I just pass it the query and a list of parameters. The trick however is to create a variable length insert statement, and a LinkedList of parameters. The effect is the same as the top example, with variable parameter input length.See below (error checking omitted).
Assuming 'myTable' has 3 updatable fields: f1, f2 and f3
String []args={"A","B","C", "X","Y","Z" }; // etc, input list of triplets
final String QUERY="INSERT INTO [myTable] (f1,f2,f3) values ";
LinkedList params=new LinkedList();
String comma="";
StringBuilder q=QUERY;
for(int nl=0; nl< args.length; nl+=3 ) { // args is a list of triplets values
params.add(args[nl]);
params.add(args[nl+1]);
params.add(args[nl+2]);
q.append(comma+"(?,?,?)");
comma=",";
}
int nr=insertIntoDB(q, params);
in my DBInterface class I have:
int insertIntoDB(String query, LinkedList <String>params) {
preparedUPDStmt = connectionSQL.prepareStatement(query);
int n=1;
for(String x:params) {
preparedUPDStmt.setString(n++, x);
}
int updates=preparedUPDStmt.executeUpdate();
return updates;
}

if you use jdbcTemplate then:
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.BatchPreparedStatementSetter;
public int[] batchInsert(List<Book> books) {
return this.jdbcTemplate.batchUpdate(
"insert into books (name, price) values(?,?)",
new BatchPreparedStatementSetter() {
public void setValues(PreparedStatement ps, int i) throws SQLException {
ps.setString(1, books.get(i).getName());
ps.setBigDecimal(2, books.get(i).getPrice());
}
public int getBatchSize() {
return books.size();
}
});
}
or with more advanced configuration
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.ParameterizedPreparedStatementSetter;
public int[][] batchInsert(List<Book> books, int batchSize) {
int[][] updateCounts = jdbcTemplate.batchUpdate(
"insert into books (name, price) values(?,?)",
books,
batchSize,
new ParameterizedPreparedStatementSetter<Book>() {
public void setValues(PreparedStatement ps, Book argument)
throws SQLException {
ps.setString(1, argument.getName());
ps.setBigDecimal(2, argument.getPrice());
}
});
return updateCounts;
}
link to source

Using PreparedStatements will be MUCH slower than Statements if you have low iterations. To gain a performance benefit from using a PrepareStatement over a statement, you need to be using it in a loop where iterations are at least 50 or higher.

Retrieve a range of records from SQL query Oracle

I have this Java code which is used for JSF pagination:
public List<ActiveSessionObj> list(int firstRow, int rowCount, String sortField, boolean sortAscending) throws Exception {
String SQL_LIST_BY_ORDER_AND_LIMIT = "SELECT * FROM ACTIVESESSIONSLOG ORDER BY ? ? LIMIT ?, ?";
if (ds == null) {
throw new SQLException();
}
String sortDirection = sortAscending ? "ASC" : "DESC";
String sql = String.format(SQL_LIST_BY_ORDER_AND_LIMIT, sortField, sortDirection);
Connection conn = ds.getConnection();
PreparedStatement preparedStatement = null;
ResultSet resultSet = null;
List<ActiveSessionObj> dataList = new ArrayList<ActiveSessionObj>();
try {
conn.setAutoCommit(false);
boolean committed = false;
preparedStatement = conn.prepareStatement(sql);
preparedStatement.setString(1, sortField);
preparedStatement.setString(2, sortDirection);
preparedStatement.setInt(3, firstRow);
preparedStatement.setInt(4, rowCount);
resultSet = preparedStatement.executeQuery();
/* take the result from the SQL query and insert it into Array List collection */
dataList = ActiveSessionsArrayList(resultSet);
} catch (SQLException e) {
throw new Exception(e);
} finally {
conn.close();
}
return dataList;
}
I use this SQL statement to generate ArrayList:
SELECT * FROM ACTIVESESSIONSLOG ORDER BY ? ? LIMIT ?, ?
Can this SQL query be used for Oracle? Or this is MySQL specific?
Best wishes

Case1: If you want first `twenty Records` only then
select * from (
select rn,a.*
from Activesessionlogs a
order by ??)
where rn <=20
order by rn
Case2:If you want the record between `5 to 10` then
select * from (
select rownum rn,e.* from Activesessionlogs e order by ??)
where rn >=5 and rn<=10
order by rn
Eg:
Lets find an example below
select * from (
select rownum rn,e.* from emp e order by hiredate)
where rn >=5 and rn<=10
order by rn asc
Note :Rownum> or rownum>= will not work in the same query .A very good blog explaining this Oracle Rownum

LIMIT is MySQL specific. However in Oracle you can use rownum like this:
SELECT *
FROM (SELECT columnA, columnB, rownum as my_rownum
FROM ACTIVESESSIONSLOG
ORDER BY ? ?)
WHERE my_rownum <= ?
AND my_rownum >= ?

Unfortunately LIMIT is not a valid statement in oracle, it is used in MySql. However oracle has a variable ROWNUM which you can use to limit the number of rows returned:
select * from mytable where rownum <= 100 and rownum > 50
returns the second 50 records from your query.

Getting index of inserted rows from a MySQL database

I'm using Java (jdbc) to interact with a MySQL database. I have table with a primary index which is AUTO INCREMENT. When I insert a row, I need to get the index it just received. How do I do that?

From: http://dev.mysql.com/doc/refman/5.0/en/connector-j-usagenotes-basic.html#connector-j-usagenotes-last-insert-id
stmt.executeUpdate(
"INSERT INTO autoIncTutorial (dataField) "
+ "values ('Can I Get the Auto Increment Field?')",
Statement.RETURN_GENERATED_KEYS);
//
// Example of using Statement.getGeneratedKeys()
// to retrieve the value of an auto-increment
// value
//
int autoIncKeyFromApi = -1;
rs = stmt.getGeneratedKeys();
if (rs.next()) {
autoIncKeyFromApi = rs.getInt(1);
} else {
// throw an exception from here
}
rs.close();
rs = null;

Thanks to John Boker for his excellent response.
If you wish to use a PreparedStatement, you can still use RETURN_GENERATED_KEYS, but you have to apply the commands differently:
PreparedStatement ps = mysql.prepareStatement(
"INSERT INTO myTable (colA, colB, colC) VALUES (?, ?, ?)",
Statement.RETURN_GENERATED_KEYS );
ps.setString(1, "My text");
ps.setTimestamp(2, new java.sql.Timestamp(new java.util.Date().getTime()););
ps.setInt(3, 5150);
ps.executeUpdate();
ResultSet results = ps.getGeneratedKeys();
results.next(); // Assume just one auto-generated key; otherwise, use a while loop here
System.out.println(results.getInt(1)); // there should only be 1 column in your results: the value of the auto-generated key
Add the RETURN_GENERATED_KEYS param in the prepareStatement()
function.
Get results not from statement.executeUpdate() but from
statement.getGeneratedKeys().

Alternatively, using Spring JDBC it would look like:
Map<String, Object> map = new HashMap<String, Object>();
map.put("column1", "test");
map.put("column2", Boolean.TRUE);
SimpleJdbcInsert insert = new SimpleJdbcInsert(template).withTableName("table").usingGeneratedKeyColumns("id");
int id = insert.executeAndReturnKey(map).intValue();