I am considering creating a HBase table when my application starts up and leaving it open as long as my application is running. My application may run indefinitely.
What happens if I never close the HBase table?
Is there a maximum time the connection can be open/idle before it need to be reinitialized?
How is the connection closed if the system crashed?
I have HBase The Definitive Guide but I have not found the information I am looking for in there. If there are any online references for this then please provide them.
This was extracted from "HBase in Action" page 25:
"Closing the table when you’re finished with it allows the underlying
connection resources to be returned to the pool."
This blog post is about timeouts in HBase. Generally speaking, there is a lot of them:
ZK session timeout (zookeeper.session.timeout)
RPC timeout (hbase.rpc.timeout)
RecoverableZookeeper retry count and retry wait (zookeeper.recovery.retry, zookeeper.recovery.retry.intervalmill)
Client retry count and wait (hbase.client.retries.number, hbase.client.pause)
You may try to raise them a bit and set a really high value for retry count. This can make your sessions be alive for a very long period of time.
When the system of HBase client crashed, the connection is closed by timeout.
Related
We are using Oracle 11G and JDK1.8 combination.
In our application we are using XAConnection, XAResource for DB transaction.
ie) distributed transactions.
On few occasions we need to kill our Java process to stop the application.
After killing, if we restart our application then we are getting the below exception while doing DB transaction.
java.sql.SQLException: ORA-02049: timeout: distributed transaction
waiting for lock
After this for few hours we are unable to use our application till the lock releases.
Can someone provide me some solution so that we can continue working instead of waiting for the lock to release.
I have tried the below option:
a) Fetched the SID and killed the session using alter command.After this also table lock is not released.
I am dealing with very small amount of data.
I followed one topic similar with that with tips about what to do with distributed connections.
Oracle connections remains open until you end your local session or until the number of database links for your session exceeds the value of OPEN_LINKS. To reduce the network overhead associated with keeping a database link open, then use this clause to close the link explicitly if you do not plan to use it again in your session.
I believe that, by closing your connections and sessions after DDL execution, this issue should not happens.
Other possibility is given on this question:
One possible way might be to increase the INIT.ORA parameter for distributed_lock_timeout to a larger value. This would then give you a longer time to observe the v$lock table as the locks would last for longer.
To achieve automation of this, you can either
- Run an SQL job every 5-10 seconds that logs the values of v$lock or the query that sandos has given above into a table and then analyze it to see which session was causing the lock.
- Run a STATSPACK or an AWR Report. The sessions that got locked should show up with high elapsed time and hence can be identified.
v$session has 3 more columns blocking_instance, blocking_session, blocking_session_statusthat can be added to the query above to give a picture of what is getting locked.
I hope I helped you, my friend.
Good time.
Some days ago our DB team has detected that there are sessions with an 'ACTIVE' status for clients that are not active any more. An investigation showed that there are two main sources for such issues:
remote SQL Developer connections (actually, this case is not very interesting),
abnormal tomcat (where our application was running) shutdowns (like 'kill -9')
It is strange for me that all that sessions are in the 'ACTIVE' status. Could somebody, please, clarify how could this be (maybe there is something with underlying processes that wait on the corresponding sockets or ... As far as everything is ok upon a proper tomcat shutdown, it seems like a root cause is with transactions...)?
Would it help if we set the 'IDLE_TIME' (for all connections) and 'EXPIRE_TIME' (for all our RAC instances)?
Am I right that the following scenarios should take place (with the above parameters set up):
When a client connects it's session is marked as 'ACTIVE'
Without respect to the 'ACTIVE' status, there is a 'ping' process, originated by the 'EXPIRE_TIME' parameter that pings a client.
If a ping process fails for the EXPIRE_TIME time period, even though a session is 'ACTIVE', the session is being killed by oracle.
If a client responds on pings, but does not do any processing, after an IDLE_TIME time period it's session becomes 'INACTIVE' and (if the 'IDLE_TIME' parameter is set) after some time - 'SNIPED'. After that a 'SMON' process starts a housekeeping activity for this session (and others with the 'SNIPED' status).
UPDATE:
It seems that the only way to work with such a situation is to configure Oracle instances.
There are results of my investigation:
https://community.oracle.com/thread/873226?start=0&tstart=0
For Dead Connection Detection is used server side sqlnet.ora file
parameter SQLNET.EXPIRE_TIME= <# of minutes>
The other option would be implement idle_time in profile setting. And then with some job kill SNIPED sessions (when idle_time will be
reached, session will become from INACTIVE to SNIPED).
If I open up a connection and go off to lunch, an IDLE_TIME limit will
cause my session to be terminated after 15 minutes of inactivity. An
EXPIRE_TIME of 15 minutes will merely have Oracle send a packet to my
client application to verify that the client has not failed. If the
client is up, it will answer the ping, and my session will stay around
indefinitely. EXPIRE_TIME only causes sessions to be killed if the
client application fails to respond to the ping, implying that the
client process has failed or that the client system has failed.
IDLE_TIME will kill sessions that don't have activity for a period of
time, but that generally doesn't work well with applications that
maintain a connection pool, since the assumption there is that the
connection pool will have a fair number of connections that are idle
for periods of the day and since applications that use connection
pools tend to react poorly to connections in the pool getting killed.
https://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:2233109200346833212
TCP/IP doesn't interrupt things by default, by design. When a
connection goes away, the client and/or server do not immediately get
notified. So, if your client connects to the database and does nothing
and you unplug your client (blue screen it, kill it, pull out the network cable, crash the computer, whatever) the odds are the session
will stay in the database. The server will not know that it will never
receive a message from you. We have dead client detection for that if
it becomes an issue:
http://download.oracle.com/docs/cd/B12037_01/network.101/b10776/sqlnet.htm#sthref476
As for the active session, that is really easy. You open a connection,
you submit a request over this connection like "lock table T". Table t
is locked by your transaction in your session. You then submit a block
of code like:
begin loop dbms_lock.sleep(5); end loop; end; /
your session will be active for as long as that code is running - the
client process is blocked on a socket read waiting for the server to
send back a result - a response (which of course will never come). The
server isn't touching the network at all right now - it is active with
your code. So, if your client 'dies' right now - the block of code
will continue to run, and run, and run - and since it never commits -
it'll just hang in there and run and of course any locks you have will
remain in place.
http://www.databaseskill.com/4267817/
http://www.dba-oracle.com/t_connect_time_idle_expire_timeout.htm
The sqlnet.expire_time parameter is used to set a time interval, in
minutes, to determine how often a probe should be sent verifying that
client/server connections are active. If you need to ensure that
connections are not left open indefinitely (or up to the time set by
operating system-specific parameters), you should set a value that is
greater than 0. This protects the system from connections left open
due to an abnormal client termination.
https://asktom.oracle.com/pls/apex/f?p=100:11:0::NO::P11_QUESTION_ID:453256655431
If the session is waiting for a resource locks or latches, and if this
wait time exceeds idle_time setting in th profile, does the session
gets sniped, even if the session is in the middle of a transaction and
waiting for lock etc.
If so, will there be any entries in the alert log.
Followup
if waiting for a lock, you are active -- not idle.
These page get my attention, six month before I had an Oracle Support
for a Data Guard issue, so one of the Oracle guys, notice that I use
the Idle_Time and he told me that this parameter dont work very well
because Oracle dont release the resource of sessions that were marked
as snipped, until the next time the user try to use it (waiting to
tell your session was killed, to clear the session resources)
Followup
... after an investigation... the "session" is there, that
will not go away until the client acknowledges it, but the
"transaction" is gone.
Tom, I've altered a profile to have IDLE_TIME=240(4 hours) and made
sure my resource_limit parameter is set to TRUE. When I query
v$session I see some "snipped" sessions, but also "inactive" ones that
have been idle for more than a day. All those users have this profile
assigned to them. If the user session was connected before idle_time
was set, would those sessions be affected by this change or not? I've
made a change quite some time ago. Is there anything else I should
have done?
Followup
If the user session was connected before idle_time was set,
they are "grandfathered" in -- they will not be sniped. it only
affects new sessions.
http://agstamy.blogspot.ru/2011/03/profile-and-resource-limit.html
additional stuff and recommendations: https://rebby.com/blog.php?detail=32
We have checked the parameters, listed in the above investigation and everything worked correctly!
What is the standard approach in Java to test whether a MySQL connection has been timed out by the server / keep the conneciton alive? (com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 73.596.145 milliseconds ago ...)
Some articles recommended periodic pinging to keep the connection alive. I could start a separated thread which does this, ok. However, is there some easy test that I could perform and reconnect only if necessary? Or is periodic pinging the best-performance way to go here?
boolean java.sql.Connection.isClosed() states "This method is guaranteed to return true only when it is called after the method Connection.close has been called" and I guess that this will not do the job for me.
I could internally keep track of seconds since last request, compare that to select ##wait_timeout; and reconnect if too much time has passed.
So, what is the standard approach to know if the connection has timed out / keep the connection alive?
Thanks a lot in advance!
I have encountered the same problem.
A solution that must be tested is to call the "isValid(int timeout)" method from the java.sql.Connection
Or set autoreconnect in mysql: http://dev.mysql.com/doc/refman/5.0/en/auto-reconnect.html
Or change the timeout in mysql (default = 8 hours): http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#sysvar_wait_timeout
EDIT: but I think that the best pratice is to open the connection during a transaction and then close it directly after that.
I need some help analyzing a problem I am encountering during the last weeks.
Every here and then my application hangs. Basically it is based on postgres but for some very speedy interactions we transfer critical data into mongodb which is kept synced. This works out - its pretty fast and we do not habe problems keeping it synced.
We use java 1.6 and spring 3.2. I instantiate one Mongo-class and #Autowire it in about 15 business logic classes.
Now my problem is: After about 2 days I get exceptions:
com.mongodb.DBPortPool$ConnectionWaitTimeOut: Connection wait timeout after 120000 ms
at com.mongodb.DBPortPool.get(DBPortPool.java:186)
at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:327)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:212)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:189)
at com.mongodb.DBApiLayer$Result._advance(DBApiLayer.java:452)
at com.mongodb.DBApiLayer$Result.hasNext(DBApiLayer.java:418)
at com.mongodb.DBCursor._hasNext(DBCursor.java:503)
at com.mongodb.DBCursor.hasNext(DBCursor.java:523)
I suppose, the pool is running out of connections.
Now my question:
Do I always have to call db.requestDone()?
According to this page:
http://www.mongodb.org/display/DOCS/Java+Driver+Concurrency - first paragraph, I read:
The Mongo object maintains an internal pool of connections to the
database (default pool size of 10). For every request to the DB (find,
insert, etc) the java thread will obtain a connection from the pool,
execute the operation, and release the connection.
To me this sounded like the connection is being passed back whenever the thread is closed, which should have happened after the request has finished.
We do not have any long-living threads in our app accessing mongodb.
Thanks for any hint!
Jan
We ran into this problem when we failed to close our cursors after reading. After adding a cursor.close()
after all our cursor operations, the problem went away
We are facing an unusual problem in our application, in the last one month our application reached an unrecoverable state, It was recovered post application restart.
Background : Our application makes a DB query to fetch some information and this Database is hosted on a separate node.
Problematic case : When the thread dump was analyzed we see all the threads are in runnable state fetching the data from the database, but it didn't finished even after 20 minutes.
Post the application restart as expected all threads recovered. And the CPU usage was also normal.
Below is the thread dump
ThreadPool:2:47" prio=3 tid=0x0000000007334000 nid=0x5f runnable
[0xfffffd7fe9f54000] java.lang.Thread.State: RUNNABLE at
oracle.jdbc.driver.T2CStatement.t2cParseExecuteDescribe(Native Method)
at
oracle.jdbc.driver.T2CPreparedStatement.executeForDescribe(T2CPreparedStatement.java:518)
at
oracle.jdbc.driver.T2CPreparedStatement.executeForRows(T2CPreparedStatement.java:764)
at ora
All threads in the same state.
Questions:
what could be the reason for this state?
how to recover under this case ?
It's probably waiting for network data from the database server. Java threads waiting (blocked) on I/O are described by the JVM as being in the state RUNNABLE even though from the program's point of view they're blocked.
As others mentioned already, that native methods are always in runnable, as the JVM doesn't know/care about them.
The Oracle drivers on the client side have no socket timeout by default. This means if you have network issues, the client's low level socket may "stuck" there for ever, resulting in a maxxed out connection pool. You could also check the network trafic towards the Oracle server to see if it even transmits data or not.
When using the thin client, you can set oracle.jdbc.ReadTimeout, but I don't know how to do that for the thick (oci) client you use, I'm not familiar with it.
What to do? Research how can you specify read timeout for the thick ojdbc driver, and watch for exceptions related to the connection timeout, that will clearly signal network issues. If you can change the source, you can wrap the calls and retry the session when you catch timeout-related SQLExceptions.
To quickly address the issue, terminate the connection on the Oracle server manually.
Worth checking the session contention, maybe a query blocks these sessions. If you find one, you'll see which database object causes the problem.
Does your code manually handle transaction? If then, maybe some of the code didn't commit() after changing data. Or maybe someone ran data modification query directly through PLSQL or something and didn't commit, and that leads all reading operation to be hung.
When you experienced that "hung" and DB has recovered from the status, did you check the data if some of them were rolled back? Asking this since you said "It was recovered post application restart.". It's happening when JDBC driver changed stuff but didn't commit, and timeout happened... DB operation will be rolled back. ( can be different based on the configuration though )
Native methods remain always in RUNNABLE state (ok, unless you change the state from the native method, itself, but this doesn't count).
The method can be blocked on IO, any other event waiting or just long cpu intense task... or endless loop.
You can make your own pick.
how to recover under this case ?
drop the connection from oracle.
Is the system or JVM getting hanged?
If configurable and if possible, reduce the number of threads/ parallel connections.
The thread simply waste CPU cycles when waiting for IO.
Yes your CPU is unfortunately kept busy by the threads who are awaiting a response from DB.