We Keep Coding

Avoiding MySQL Deadlocks in a multithreaded Spring app

The scenario is simple.
I have a somehow large MySQL db containing two tables:
-- Table 1
id (primary key) | some other columns without constraints
-----------------+--------------------------------------
1 | foo
2 | bar
3 | foobar
... | ...
-- Table 2
id_src | id_trg | some other columns without constraints
-------+--------+---------------------------------------
1 | 2 | ...
1 | 3 | ...
2 | 1 | ...
2 | 3 | ...
2 | 5 | ...
...
On table1 only id is a primary key. This table contains about 12M entries.
On table2 id_src and id_trg are both primary keys and both have foreign key constraints on table1's id and they also have the option DELETE ON CASCADE enabled. This table contains about 110M entries.
Ok, now what I'm doing is only to create a list of ids that I want to remove from table 1 and then I'm executing a simple DELETE FROM table1 WHERE id IN (<the list of ids>);
The latter process is as you may have guessed would delete the corresponding id from table2 as well. So far so good, but the problem is that when I run this on a multi-threaded env and I get many Deadlocks!
A few notes:
There is no other process running at the same time nor will be (for the time being)
I want this to be fast! I have about 24 threads (if this does make any difference in the answer)
I have already tried almost all of transaction isolation levels (except the TRANSACTION_NONE) Java sql connection transaction isolation
Ordering/sorting the id's I think would not help!
I have already tried SELECT ... FOR UPDATE, but a simple DELETE would take up to 30secs! (so there is no use of using it) :
DELETE FROM table1
WHERE id IN (
SELECT id FROM (
SELECT * FROM table1
WHERE id='some_id'
FOR UPDATE) AS x);
How can I fix this?
I would appreciate any help and thanks in advance :)
Edit:
Using InnoDB engine
On a single thread this process would take a dozen hours even maybe a whole day, but I'm aiming for a few hours!
I'm already using a connection pool manager: java.util.concurrent
For explanation on double nested SELECTs please refer to MySQL can’t specify target table for update in FROM clause
The list that is to be deleted from DB, may contain a couple of million entries in total which is divided into chunks of 200
The FOR UPDATE clause is that I've heard that it locks a single row instead of locking the whole table
The app uses Spring's batchUpdate(String sqlQuery) method, thus the transactions are managed automatically
All ids have index enabled and the ids are unique 50 chars max!
DELETE ON CASCADE on id_src and id_trg (each separately) would mean that every delete on table1 id=x would lead to deletes on table2 id_src=x and id_trg=x
Some code as requested:
public void write(List data){
try{
Arraylist idsToDelete = getIdsToDelete();
String query = "DELETE FROM table1 WHERE id IN ("+ idsToDelete + " )";
mysqlJdbcTemplate.getJdbcTemplate().batchUpdate(query);
} catch (Exception e) {
LOG.error(e);
}
}
and myJdbcTemplate is just an abstract class that extends JdbcDaoSupport.

First of all your first simple delete query in which you are passing ids, should not create problem if you are passing ids till a limit like 1000 (total no of rows in child table also should be near about but not to many like 10,000 etc.), but if you are passing like 50,000 or more then it can create locking issue.
To avoid deadlock, you can follow below approach to take care this issue (assuming bulk deletion will not be part of production system)-
Step1: Fetch all ids by select query and keep in cursor.
Step2: now delete these ids stored in cursor in a stored procedure one by one.
Note: To check why deletion is acquiring locks we have to check several things like how many ids you are passing, what is transaction level set at DB level, what is your Mysql configuration setting in my.cnf etc...

It may be dangereous to delete many (> 10000) parent records each having child records deleted by cascade, because the most records you delete in a single time, the most chances of lock conflict leading to deadlock or rollback.
If it is acceptable (meaning you can make a direct JDBC connection to the database) you should (no threading involved here) :
compute the list of ids to delete
delete them by batches (between 10 and 100 a priori) committing every 100 or 1000 records
As the heavier job should be on database part, I hardly doubt that threading will help here. If you want to try it, I would recommend :
one single thread (with a dedicated database connection) computing the list of ids to delete and alimenting a synchronized queue with them
a small number of threads (4 maybe 8), each with its own database connection that :
use a prepared DELETE FROM table1 WHERE id = ? in batches
take ids from the queue and prepare the batches
send a batch to the database every 10 or 100 records
do a commit every 10 or 100 batches
I cannot imagine that the whole process could take more than several minutes.
After some other readings, it looks like I was used to old systems and that my numbers are really conservative.

Ok here's what I did, it might not actually avoid having Deadlocks but was my only option at time being.
This solution is actually a way of handling MySQL Deadlocks using Spring.
Catch and retry Deadlocks:
public void write(List data){
try{
Arraylist idsToDelete = getIdsToDelete();
String query = "DELETE FROM table1 WHERE id IN ("+ idsToDelete + " )";
try {
mysqlJdbcTemplate.getJdbcTemplate().batchUpdate(query);
} catch (org.springframework.dao.DeadlockLoserDataAccessException e) {
LOG.info("Caught DEADLOCK : " + e);
retryDeadlock(query); // Retry them!
}
} catch (Exception e) {
LOG.error(e);
}
}
public void retryDeadlock(final String[] sqlQuery) {
RetryTemplate template = new RetryTemplate();
TimeoutRetryPolicy policy = new TimeoutRetryPolicy();
policy.setTimeout(30000L);
template.setRetryPolicy(policy);
try {
template.execute(new RetryCallback<int[]>() {
public int[] doWithRetry(RetryContext context) {
LOG.info("Retrying DEADLOCK " + context);
return mysqlJdbcTemplate.getJdbcTemplate().batchUpdate(sqlQuery);
}
});
} catch (Exception e1) {
e1.printStackTrace();
}
}

Another solution could be to use Spring's multiple step mechanism.
So that the DELETE queries are split into 3 and thus by starting the first step by deleting the blocking column and other steps delete the two other columns respectively.
Step1: Delete id_trg from child table;
Step2: Delete id_src from child table;
Step3: Delete id from parent table;
Of course the last two steps could be merged into 1, but in that case two distinct ItemsWriters would be needed!

hibernate hbm: automatic (unwanted) update found

In my project I use hibernate hbm and spring,i run an sql query to update a single column,
Query sql = getSession().createSQLQuery("update HISTORIQUE_DETAIL_APPELS set token_otp = '"+historiqueDetailAppelsVO.getCodeOtp()+"' where id = '"+historiqueDetailAppelsVO.getId()+"'");
try {
sql.executeUpdate();
} catch (Exception e) {
e.printStackTrace();
}
I found that another query is executed and update the table in data base,
Hibernate: update HISTORIQUE_DETAIL_APPELS set token_otp = '14d3fc' where id = '150017'
Hibernate: update HISTORIQUE_DETAIL_APPELS set cod_cent=?, adresse_ip=?, id_conseiller=?, type_piece=?, num_piece_ident=?, msisdn=?, mois1_detail=?, mois2_detail=?, mois3_detail=?, date_demande=?, no_ticket_caisse=?, date_ticket=?, cod_user=?, dat_maj=?, flag_imp_data=?, date_imp_data=?, token_otp=?, send_mail=?, client_mail=?, date_debut=?, date_fin=? where id=?
where does the origin of the second update ?

I faced the same issue before, and after some research i found :
When user update any record by using direct query based operation, it is directly updated to database.
But, if the same record(previous copy) is already present in current session(that is previously read by user in current session) then, there is a difference occurs between database record(that is updated by query based operation) and current session record, due to this, hibernate again runs update query to update session record during either flushing of session or on transaction completion.
To avoid the second execution executed by hibernate either during flushing of session or on transaction completion.
I wish it will help you.
Thanks

I solve this probleme by adding : dynamic-update="true" in hbm.xml file
I found the solution here:
"The dynamic-update attribute tells Hibernate whether to include
unmodified properties in the SQL UPDATE statement."

JDBC and Concurrency issues

I want some advice on some concurrency issues regarding jdbc, i basically need to update a value and then retrieve that value using a update then a select, I'm assuming by turning auto commit off no other transaction can access this table, hence other transactions won't be able to perform update and select queries until this has been committed.
Below is some example code. Do you think this will work and does any one else have a better solution to implementing this?
int newVal=-1;
con.setAutoCommit(false);
PreparedStatement statement = con.prepareStatement("UPDATE atable SET val=val+1 WHERE id=?");
statement.setInt(1, id);
int result = statement.executeUpdate();
if (result != 1) {
throw new SQLException("Nothing updated");
} else {
statement = con.prepareStatement("SELECT val FROM atable WHERE id=?");
statement.setInt(1, id);
ResultSet resultSet = statement.executeQuery();
if (resultSet.next()) {
newVal = resultSet.getInt("val");
}
}
statement.close();
con.commit();
con.setAutoCommit(true);
Thanks.

Assuming you use some form of data source, you may configure there if you want transactionality and the isolation level. But to be explicit:
try(Connection con = ds.getConnection()){
con.setTransactionIsolation(Connection.TRANSACTION_SERIALIZABLE);
con.setAutoCommit(false);
//...
} catch(SQLException sqle) {
throw new MyModelException(e)
}
Now, you could trigger pesimistic locking by updating a version (or timestamp) field in your table. This will trigger a lock in the database (most likely at the record level):
try(PreparedStatement pStm = con.prepareStatement("update atable set version=version+1")){
pStm.executeUpdate();
}
At this point, if another user is trying to update the same record simultaneously, this connection will either wait or timeout, so you must be ready for both things. The record will not be unlocked until your transaction ends (commit or rollback).
Then, you can safely select and update whatever you want and be sure that nobody else is touching your record as you process your data. If anybody else tries they will be put on wait until you finish (or they will timeout depending on connection configuration).
Alternatively you could use optimistic locking. In this case you read your record, do modifications to it, but in the update you make sure nobody else has changed it since you read it by checking that the version/timestamp field is the same as the one you orginally read. In this case you must be prepared to retry a transaction (or abort it alltogether) if you realize you have stale/outdated data.
i.e. update atable set afield=? where id=? and version=1
If the number of rows affected is 0, then you know that is probable that the record was updated between your read and your update and the record is no longer in version 1.

Setting autocommit=false on your connection will not prevent other connections/threads from changing the row in the database! It will only disable automatic commits after each JDBC operation on that specific connection.
You will need to lock the row, eg. with select ... for update to prevent other transactions against the row, and also you will need to do your selects and updates within a single transaction.
Cheers,

What's the fastest way to check if a row exists in DB using Hibernate & spring?

I need to check if a row exists in a database in a very fast way.
Let's say I've got the primary key.
I found this code snippet in Hibernate's FAQ website:
Integer count = (Integer) session.createQuery("select count(*) from ....").uniqueResult();
I just started using spring, so I have HibernateTemplate object injected into my class.
How do I translate this snippet to work with HibernateTemplate.
Does anyone knows a better/faster way than this ?
Thanks.

Long count = hibernateTemplate.execute(new HibernateCallback<Long>() {
#Override
public Long doInHibernate(Session session) {
return (Long) session.createQuery("select count(someEntity.id) from SomeEntiuty someEntity ...").uniqueResult();
}
});
Hibernate used Integer for count queries before, but now uses Long. Also, note that even if not deprecated, Spring recommends not to use HibernateTemplate anymore and use the Hibernate API directly (using sessionFactory.getCurrentSession()).

Fastest way of checking primary key exist or not in database.
public void exist(Long id) {
Session session = sessionFactory.getCurrentSession();
String queryString = "select 1 from Employee e where e.id= :id";
Query query = session.createQuery(queryString);
query.setParameter("id", 1l);
Integer result = (Integer) query.uniqueResult();
System.out.println(result);
}

Again this also depends on a lot on what engine that you are using MyISAM vs innodb.
select count(col1) from table; will return the number of rows where the column is not null.
select count(*) from table; will return the number of rows.
Depending upon the database that you are using , a select count(*) will be more expensive than reading it from meta data table or system level tables that keep track of the row count.
Just my 2 cents.
Depending upon various other factors like indexes and other information / joins / access privileges this may be faster
SELECT table_rows FROM `information_schema`.`TABLES` where table_schema = 'database_schema_name' and table_name = 'table_name';

I think it's better to get an specific representative field of the first row found (using the PK or at least another indexed field), than counting all of the possible records that would match your search criteria.
If you're using Spring it will throw EmptyResultDataAccessException if no record was found.

JPA and MySQL transaction isolation level

I have a native query that does a batch insert into a MySQL database:
String sql = "insert into t1 (a, b) select x, y from t2 where x = 'foo'";
EntityTransaction tx = entityManager.getTransaction();
try {
tx.begin();
int rowCount = entityManager.createNativeQuery(sql).executeUpdate();
tx.commit();
return rowCount;
}
catch(Exception ex) {
tx.rollback();
log.error(...);
}
This query causes a deadlock: while it reads from t2 with insert .. select, another process tries to insert a row into t2.
I don't care about the consistency of reads from t2 when doing an insert .. select and want to set the transaction isolation level to READ_UNCOMMITTED.
How do I go about setting it in JPA?
Update
So I ended up creating a regular SQL connection for this case as it seemed to me the simplest option. Thanks everyone!

You need to set it at the connection level, get the session from the entitymanager and do this:
org.hibernate.Session session = (Session)entityManager.getDelegate();
Connection connection = session.connection();
connection.setTransactionIsolation(Connection.READ_UNCOMMITTED);

In JPA you don't. JDO is the only standard that supports setting txn isolation. Obviously going for particular implementations methods can allow it, but then you become non-portable

Since you are using BMT, you can do the following using a datasource to get the connection.
and set the iso. level.
DataSource source = (javax.sql.DataSource) jndiCntxt.lookup("java:comp/env/jdbc/myds");
Connection con = source.getConnection( );
con.setTransactionIsolation(Connection.TRANSACTION_READ_UNCOMMITTED);