BatchUpdate with ACID properties in Spring + mysql - java

Currently our application uses Spring, SimpleJdbcTemplate & MySQL.
DataSource used is org.apache.commons.dbcp.BasicDataSource with url properties "rewriteBatchedStatements=true".
During the batch insert with SimpleJdbcTemplate.batchUpdate(List<Object[]>), there will be duplicate records(based on the primary key of the table we are inesrting) in the input batch.
Under the duplicate record scenario is it possible to
1) Insert all the non-duplicate records and get the response about the number of successful inserts?
OR
2) Completely rollback the batchInsert, no record should be inserted?
We are able to partially achieve the first requirement using the "INSERT IGNORE" of mysql. But the SimpleJdbcTemplate.batchUpdate() returns every record as updated.(Not able to capture only the inserted record count ignoring the duplicates)
And to achieve the second requirement we have to turn off the "rewriteBatchedStatements". But this parameter has been fine tuned after performance test. So we can't set this to "false".
Is the possible to achieve one of two above cases, within the constraints of the components we are using as mentioned in the first line?
I am new to spring and jdbc world, so a detailed explanation will help us better.
Thanks

1) Insert all the non-duplicate records and get the response about the number of successful inserts?
=> Well you can, insert all non.duplicate records, and get the response. batchUpdate() returns int[] i.e. array of integers which represents the numbers of rows affected by each update in the batch (inserted/updated).
2) Completely rollback the batchInsert no record should be inserted?
=> As batchInsert will be within single transaction, either all records will be inserted or no record is inserted. If transaction gets committed all records will be inserted. If any exception occurs, transaction will be rolled back automatically if you are using spring transaction management (either using #Transactional annotation or spring aop based tx advice). Here make sure, you set BasicDataSource.defaultAutoCommit = false.

Related

hibernate auditions generating same rev id in for multiple insert in one table

I am using hibernate auditions with hibernate version 3.5. while it is working fine when I try to insert single record in one table in one transation but the problem is when a "BATCH" runs
and it inserts multiple records in one table in single transation then single "rev id" is being generated for audit tables which causes "integrityconstraintsviolation".
As it is the normal behavior of hibernate that all the insert queries are fired at the end of tansaction(When it flushes) but at this time only one query is being fired for "rev id" generation.
select hibernate_sequence.nextval from dual;
Kindly tell me whether is it a bug in audit or i am missing something ??
Thanks in advance
A revid always span modifications in multiple tables!
Yes, the inserts are called at the end of the transaction, but: If you use optimistic transaction isolation, a transaction can read the uncommitted state of another transactions that are currently active but not yet committed.
If you become a integrityconstraintsviolation the column rev_id is unfortunatelly configured "unique". This is a wrong database schema! Correct the schema by removing the Uniqueness of the column rev_id!
The rev_id-Column in the global-hibernate-relationtable must be unique because it is a Primary-Key! (the global-hibernate-relationtable usually contains also a timestamp).

How to force commit Spring - hibernate transaction safely

We are using spring and hibernate for an web application:
The application has a shopping cart where user can place items in it. in order to hold the items to be viewed between different login's the item values in the shopping cart are stored in tables. when submitting the shopping cart the items will be saved into different table were we need to generate the order number.
When we insert the values into the table to get the order number, we use to get the max order number and add +1 to it. we are using spring transaction manager and hibernate, in the code flow we get the order number and update the hibernate object to hold the order num value. when i debug, i noticed that only when the complete transaction is issued the order number entity bean is being inserted.
Issue here is when we two request is being submitted to the server at the same time, the same order number is being used, and only one request data is getting inserted. could not insert the other request value which is again a unique one.
The order num in the table is a unique one.
i noticed when debugging the persistant layer is not getting inserted into the database even after issuing session flush
session.flush()
its just updating the memory and inserting the data to db only at the end of the spring transaction . i tried explicitly issuing a commit to transaction
session.getTransaction().commit();
this inserted the values into the database immediately, but on further code flow displayed message that could not start transaction.
Any help is highly appreciated.
Added:
Oracle database i used.
There is a sequence number which is unique for that table and also the order number maps to it.
follow these steps :- ,
1) Create a service method with propagation REQUIRES_NEW in different service class .
2)Move your code (whatever code you want to flush in to db ) in this new method .
3)Call this method from existing api (Because of proxy in spring, we have to call this new service method from different class otherwise REQUIRES_NEW will not work which make sure your flushing data ).
I would set the order number with a trigger which will run in the same transaction with the shopping cart insert one.
After you save the shopping cart, to see the updated order count, you'll have to call:
session.refresh(cart);
The count shouldn't be managed by Hibernate (insertable/updatable = false or #Transient).
Your first problem is that of serial access around the number generation when multiple thread are executing the same logic. If you could use Oracle sequences this would have been automatically taken care of at the database level as the sequences
are guranteed to return unique values any number of times they are called. However since this needs to be now managed at server side, you would need to
use synchronization mechanism around your number generation logic ( select max and increment by one) across the transaction boundary. You can make the Service
method synchronized ( your service class would be singleton and Spring managed) and declare the transaction boundary around it. However please note that this would be have performance implications and is usually bad for
scalability.
Another option could be variation of this - store the id to be allocated in a seperate table with one column "currentVal" and use pessimistic lock
for getting the next number. This way, the main table would not have any big lock. This way a lock would be held for the sequence generator code for the time the main entity creation transaction is complete. The main idea behind these techniques is to serialize
access to the sequence generator and hold the lock till the main entity transaction commits. Also delay the number generator as late as possible.
The solution suggested by #Vlad is an good one if using triggers is fine in your design.
Regarding your question around the flush behaviour, the SQL is sent to the database at flush call, however the data is not committed until the transaction is committed declaratively or a manual commit is called. The transaction can however see the data it purposes to change but not other transactions depending upon the isolation nature of transaction.

Records are not getting updated with ResultSet

I have some JDBC stuff defined over here.
I am basically, grabbing records from one table(test.selectiontable) located at 11.11.1.111 and inserting into 22.22.2.222(test.insertiontable). It's a thread based process, so the thread will keep on looking & transferring
the records until all records are transferred.
and
Since I don't want to insert duplicate records into another table, I have a field in test.selectiontable, called DTSStatusType_ti which has initial value of 1 in the table. Hence,I am updating it after transferring the records
to test.insertiontable as follows:
Line # 112 SelectQueueRS.updateInt( "DTSStatusType_ti", 3 );
Line # 113 SelectQueueRS.updateString( "Queued_DialerIP_vch", 22.22.2.222 );
The initial value of Queued_DialerIP_vch in test.selectiontable is 11.11.1.111.
Although the records are getting updated using above two lines of code in my code but I don't think it's an efficient way to update. Could anyone please suggest some efficient way to update the records and 100% make sure that these records are not duplicated ever. Please feel free to suggest any changes in my code. Thanks
Instead of updating the ResultSet of the original query, you can:
Start a transaction with connMain.setAutoCommit(false)
If others can modify the data, you should use SELECT FOR UPDATE which locks the selected rows
Collect the primary key ids from the query result
Do the insert to the remote datasource
Update the original table with one update query, by the collected ids (WHERE id IN (...))
Commit the transaction with connMain.commit()
3 and 4 can be done in the same cycle.
By the way you can also start a transaction on the remote datasource and do a batch insert.
You should also be aware of doing transactions in try-catch blocks and closing your resources. This may help: http://docs.oracle.com/javase/tutorial/jdbc/basics/transactions.html

TX-row lock contention : Inserting Duplicate values for Unique Key

We are getting a TX-row lock contention error while trying to insert data
It happens while running a job which processes an xml with almost 10000 records, inserting the data into a table
We are having a unique key constraint on one of the columns in the table, and in the request we are getting duplicate values. This is causing the locks and thereby the job is taking more time.
We are using hibernate and spring. The DAO method we use is hibernate template's 'save' annotated with Spring Transaction Manager's #Transactional
Any suggestions please?
It's not clear whether you're getting locking problems or errors.
"TX-row lock contention" is an event indicating that two sessions are trying to insert the same value into a primary or unique constraint column set -- an error is not raised until the first one commits, then the second one gets the error. So you definitely have multiple sessions inserting rows. If you just had one session then you'd receive the error immediately, with no "TX-row lock contention" event being raised.
Suggestions:
Insert into a temporary table without the constraint, then load to the real table using logic that eliminates the duplicates
Eliminate the duplicates as part of the read of the XML.
Use Oracle's error logging syntax -- example here. http://docs.oracle.com/cd/B28359_01/server.111/b28286/statements_9014.htm#SQLRF55004

Hibernate Relationship Mapping/Speed up batch inserts

I have 5 MySQL InnoDB tables: Test,InputInvoice,InputLine,OutputInvoice,OutputLine and each is mapped and functioning in Hibernate. I have played with using StatelessSession/Session, and JDBC batch size. I have removed any generator classes to let MySQL handle the id generation- but it is still performing quite slow.
Each of those tables is represented in a java class, and mapped in hibernate accordingly. Currently when it comes time to write the data out, I loop through the objects and do a session.save(Object) or session.insert(Object) if I'm using StatelessSession. I also do a flush and clear (when using Session) when my line count reaches the max jdbc batch size (50).
Would it be faster if I had these in a 'parent' class that held the objects and did a session.save(master) instead of each one?
If I had them in a master/container class, how would I map that in hibernate to reflect the relationship? The container class wouldn't actually be a table of it's own, but a relationship all based on two indexes run_id (int) and line (int).
Another direction would be: How do I get Hibernate to do a multi-row insert?
The ID generation strategy is critical for batch insertion in Hibernate. In particular, IDENTITY generation will usually not work (note that AUTO typically maps to IDENTITY as well). This is because during batch insert Hibernate has a flag called "requiresImmediateIdAccess" that says whether or not generated IDs are immediately required or not; if so, batch processing is disabled.
You can easily spot this in the DEBUG-level logs when it says "executing identity-insert immediately" - this means it has skipped batch processing because it was told that generated IDs are required immediately after insertion.
Generation strategies that typically do work are TABLE and SEQUENCE, because Hibernate can pre-generate the IDs, thereby allowing for batch insertion.
A quick way to spot whether your batch insertion works is to activate DEBUG-level logs as BatchingBatcher will explicitly tell you the batch size it's executing ("Executing batch size: " + batchSize ).
Additionally, the following properties are important for achieving batch insertion. I daren't say they're required as I'm not enough of a Hibernate-expert to do so - perhaps it's just my particular configuration - but in my experience they were needed nonetheless:
hibernate.order_inserts = true
hibernate.order_updates = true
These properties are pretty poorly documented, but I believe what they did was enable for the SQL INSERT and UPDATE statements to be properly grouped for batch execution; I think this might be the multi-row inserts you're after. Don't shoot me if I'm wrong on this, I'm recalling from memory.
I'll also go ahead and assume that you set the following property; if not, this should serve as a reminder:
hibernate.jdbc.batch_size = xx
Where xx is your desired batch size, naturally.
The final solution for me was to use voetsjoeba's response as a jumping off point.
My hibernate config uses the following options:
hibernate.order_inserts = true
hibernate.order_updates = true
I changed from using Session to
StatelessSession
Re-ordered the
Java code to process all the elements
in a batch a table at a time. So all
of table x, then table y, etc.
Removed the <generator> from each
class. Java now creates it and
assigns it to the object
Created logic that allowed me to determine if just
an id was being set and not write
'empty' lines to the database
Finally, I turned on dynamic-insert
for my classes in their hibernate
definitions like so: <class name="com.my.class" table="MY_TABLE" dynamic-insert="true">

Categories