I try load 50000 rows in table with 200 columns. I do executeBatch() every 1000 rows. And I get lock for this table. The same code works for MS SQL and Oracle but with postgresql I get this issue. When I decrease executeBatch number from 1000 to 75 all works correctly.
Is there any param in config file witch responding for batch buffer size?
Same issue http://www.postgresql.org/message-id/c44a1bc0-dcb1-4b57-8106-e50f9303b7d1#79g2000hsk.googlegroups.com
When I execute insert statements in batch for tables with a large number of columns occurs hanging, when call statement.executeBatch().
It is specific for postgresql jdbc driver.
To avoid this issue we should increase buffer size params(SO_SNDBUF, SO_RCVBUF) for socket.
For Windows we have to create such params in register:
[HKEY_LOCAL_MACHINE \SYSTEM \CurrentControlSet \Services \Afd \Parameters]
DefaultReceiveWindow(type=DWORD) = 1640960(Decimal)
DefaultSendWindow(type=DWORD) = 1640960(Decimal)
This number(1640960) I get from internet as general recommendation!
And it works for me.
Generally you need to look for the following things.
Are you able to actually get a lock for the table?
Do you have other java code locks that you are waiting for?
In general the first place to check is the pg_stat_activity system view which will show you the query being executed and whether it is active, idle, waiting, etc. Then if it is waiting (i.e. waiting is t), then you will want to check the pg_locks view and see what else may have a lock on anything in the relation.
If waiting is not true, then you are better off checking your java code for client-side locks but since this works for MySQL and Oracle, I assume this is less of an issue.
There is one other thing to be aware of. I am not sure about MySQL and Oracle here but PostgreSQL limits you to one query at a time per connection. You might have some locking there?
Related
I am working on a Java web application that uses Weblogic to connect to an Informix database. In the application we have multiple threads creating records in a table.
It happens pretty often that it fails and the following error is thrown:
java.sql.SQLException: Could not do a physical-order read to fetch next row....
Caused by: java.sql.SQLException: ISAM error: record is locked.
I am assuming that both threads are trying to insert or update when the record is locked.
I did some research and found that there is an option to set the database that instead of throwing an error, it should wait for the lock to be released.
SET LOCK MODE TO WAIT;
SET LOCK MODE TO WAIT 17;
I don't think that there is an option in JDBC to use this setting. How do I go about using this setting in my java web app?
You can always just send that SQL straight up, using createStatement(), and then send that exact SQL.
The more 'normal' / modern approach to this problem is a combination of MVCC, the transaction level 'SERIALIZABLE', retry, and random backoff.
I have no idea if Informix is anywhere near that advanced, though. Modern DBs such as Postgres are (mysql does not count as modern for the purposes of MVCC/serializable/retry/backoff, and transactional safety).
Doing MVCC/Serializable/Retry/Backoff in raw JDBC is very complicated; use a library such as JDBI or JOOQ.
MVCC: A mechanism whereby transactions are shallow clones of the underlying data. 2 separate transactions can both read and write to the same records in the same table without getting in each other's way. Things aren't 'saved' until you commit the transaction.
SERIALIZABLE: A transaction level (also called isolationlevel), settable with jdbcDbObj.setTransactionIsolation(Connection.TRANSACTION_SERIALIZABLE); - the safest level. If you know how version control systems work: You're asking the database to aggressively rebase everything so that the entire chain of commits is ordered into a single long line of events: Each transaction acts as if it was done after the previous transaction was completed. The simplest way to implement this level is to globally lock all the things. This is, of course, very detrimental to multithread performance. In practice, good DB engines (such as postgres) are smarter than that: Multiple threads can simultaneously run transactions without just being frozen and waiting for locks; the DB engine instead checks if the things that the transaction did (not just writing, also reading) is conflict-free with simultaneous transactions. If yes, it's all allowed. If not, all but one simultaneous transaction throw a retry exception. This is the only level that lets you do this sequence of events safely:
Fetch the balance of isaace's bank account.
Fetch the balance of rzwitserloot's bank account.
subtract €10,- from isaace's number, failing if the balance is insufficient.
add €10,- to rzwitserloot's number.
Write isaace's new balance to the db.
Write rzwitserloot's new balance to the db.
commit the transaction.
Any level less than SERIALIZABLE will silently fail the job; if multiple threads do the above simultaneously, no SQLExceptions occur but the sum of the balance of isaace and rzwitserloot will change over time (money is lost or created – in between steps 1 & 2 vs. step 5/6/7, another thread sets new balances, but these new balances are lost due to the update in 5/6/7). With serializable, that cannot happen.
RETRY: The way smart DBs solve the problem is by failing (with a 'retry' error) all but one transaction, by checking if all SELECTs done by the entire transaction are not affected by any transactions that been committed to the db after this transaction was opened. If the answer is yes (some selects would have gone differently), the transaction fails. The point of this error is to tell the code that ran the transaction to just.. start from the top and do it again. Most likely this time there won't be a conflict and it will work. The assumption is that conflicts CAN occur but usually do not occur, so it is better to assume 'fair weather' (no locks, just do your stuff), check afterwards, and try again in the exotic scenario that it conflicted, vs. trying to lock rows and tables. Note that for example ethernet works the same way (assume fair weather, recover errors afterwards).
BACKOFF: One problem with retry is that computers are too consistent: If 2 threads get in the way of each other, they can both fail, both try again, just to fail again, forever. The solution is that the threads twiddle their thumbs for a random amount of time, to guarantee that at some point, one of the two conflicting retriers 'wins'.
In other words, if you want to do it 'right' (see the bank account example), but also relatively 'fast' (not globally locking), get a DB that can do this, and use JDBI or JOOQ; otherwise, you'd have to write code to run all DB stuff in a lambda block, catch the SQLException, check the SqlState to see if it is indicating that you should retry (sqlstate codes are DB-engine specific), and if yes, rerun that lambda, after waiting an exponentially increasing amount of time that also includes a random factor. That's fairly complicated, which is why I strongly advise you rely on JOOQ or JDBI to take care of this for you.
If you aren't ready for that level of DB usage, just make a statement and send "SET LOCK MDOE TO WAIT 17;" as SQL statement straight up, at the start of opening any connection. If you're using a connection pool there is usually a place you can configure SQL statements to be run on connection start.
The Informix JDBC driver does allow you to automatically set the lock wait mode when you connect to the server.
Simply pass via the DataSource or connection URL the following parameter
IFX_LOCK_MODE_WAIT=17
The values for JDBC are
(-1) Wait forever
(0) not wait (default)
(> 0) wait this many seconds
See https://www.ibm.com/support/knowledgecenter/SSGU8G_14.1.0/com.ibm.jdbc.doc/ids_jdbc_040.htm
Connection conn = DriverManager.getConnection ( "jdbc:Informix-sqli://cleo:1550:
IFXHOST=cleo;PORTNO=1550;user=rdtest;password=my_passwd;IFX_LOCK_MODE_WAIT=17";);
We are using Oracle 11G and JDK1.8 combination.
In our application we are using XAConnection, XAResource for DB transaction.
ie) distributed transactions.
On few occasions we need to kill our Java process to stop the application.
After killing, if we restart our application then we are getting the below exception while doing DB transaction.
java.sql.SQLException: ORA-02049: timeout: distributed transaction
waiting for lock
After this for few hours we are unable to use our application till the lock releases.
Can someone provide me some solution so that we can continue working instead of waiting for the lock to release.
I have tried the below option:
a) Fetched the SID and killed the session using alter command.After this also table lock is not released.
I am dealing with very small amount of data.
I followed one topic similar with that with tips about what to do with distributed connections.
Oracle connections remains open until you end your local session or until the number of database links for your session exceeds the value of OPEN_LINKS. To reduce the network overhead associated with keeping a database link open, then use this clause to close the link explicitly if you do not plan to use it again in your session.
I believe that, by closing your connections and sessions after DDL execution, this issue should not happens.
Other possibility is given on this question:
One possible way might be to increase the INIT.ORA parameter for distributed_lock_timeout to a larger value. This would then give you a longer time to observe the v$lock table as the locks would last for longer.
To achieve automation of this, you can either
- Run an SQL job every 5-10 seconds that logs the values of v$lock or the query that sandos has given above into a table and then analyze it to see which session was causing the lock.
- Run a STATSPACK or an AWR Report. The sessions that got locked should show up with high elapsed time and hence can be identified.
v$session has 3 more columns blocking_instance, blocking_session, blocking_session_statusthat can be added to the query above to give a picture of what is getting locked.
I hope I helped you, my friend.
I create multiple connections and make batch inserts into myTable simultaneously (multi threading)
insertString = "INSERT INTO ... + values + ") ";
insertTable.addBatch(insertString);
insertTable.executeBatch();
insertTable.clearBatch();
Sometime it works fine, however in other cases it hangs. I understand this is because I am inserting into the same table so it gets locked .
How can I write an INSERT statement such that it does not lock the table?
Are there any special transaction start . . commands that can prevent the table from being locked ? In addition, I am curious why it works fine sometimes ?
PS: the maximum number of connections that I used was 1024 (worked perfect sometimes)
Thanks
1024 session is totally insane. Your DBA should block your user for that.
You probably get blocked sessions when you load multiple rows with the same PI.
Single-row INSERTs are the slowest way to load data. A single session with a batch size of a few 1000 will outperform dozens of single-row sessions (but then only use one session).
Or switch to JDBC FastLoad if the target table is empty.
Check
http://developer.teradata.com/connectivity/articles/speed-up-your-jdbcodbc-applications
Is this a staging table?
I'm doing a sql query in some java code, but the execution blocked at a call to ResultSet.next(). At the same time, I found I'm visiting the queried table using IBM DB2 Control Center. When I closed Control Center, the call to ResultSet.next() is resumed. So I suspect my manipulation on Control Center locked the queried table. Is this possible? How can I confirm this(i.e. how can I know if a certain table is locked in DB2)?
Yes, it can be blocked depending, on the queries. It does not matter that it is the control center, it can be any other kind of connection. The fact is that in a connection, you put a lock on a table or a row, and the other connection tries to access that table or row, however the second one stays in 'Lock Wait' because it should wait for the lock release from the first connection.
Let's suppose you create a table from the Control Center, but you does not execute a commit. When you issue a select from the Java application, the table is with a lock wait on it. You should issue a rollback or commit in the control center in order to release the lock.
In order to see the locks, there are many tools, however, db2top, option u is a good one for me when analyzing real time situations.
Finally, why you use control center? it is outdated. Try IBM Data Studio.
While AngocA is right in general, it must be said that what you experienced may hint to some configuration problems, as two read-only transactions shouldn't lock out each other under normal circumstances.
DB2 normaly uses row locking. However, if the number of locked rows gets too big for the so called lock list, the db2 server may perform a so called "lock escalation" and replace millions of row locks for some table with a single exclusive lock on the table.
The behaviour also depends on the so called isolation mode used, but this is a topic not easily explained in a SO post; you need to read some documentation to understand it.
The size of the lock list can be configured with db paramater LOCKLIST. The diagnostic log (look for file db2diag.log) tells you whether lock escalations have taken place.
I am executing a series of sql statements using a JDBC connection on a DB2 server. On the last execute() of the simple sql: DELETE FROM MYTABLE, the thread gets hung for a long period of time even if the table somply contains a single record.
The application server I am using is WAS. I wonder if this is an issue specific to WAS and DB2 combination as the same code works on other environments.
Does anybody have any idea what is going on here?
Have you issue the command directly from the CLP? It could be other problem such as:
Transaction log problem: There are a lot of rows to delete, and this takes time. Also, the transaction logs have reached the limit, and the database does not do a rollback but waits for empty log freed by other transactions.
Lock problem (concurrency): some of the rows your are trying to delete have locks in other transactions, and the applications has to wait to release them (lock wait)
Also, try to do frequent commits.
Deleting rows in a database can be a terrible work: don't forget the database server will log all the data of the table in case of a ROLLBACK. Then I assume the problem is coming from the database especially if the table has many rows.
Have you tried to run manually all the SQL requests yourself in an interactive environment?