We Keep Coding

My application is stucked when DB is down

I'm using Hibernate and Tomcat JDBC connection pool to use my MySQL DB.
When, from any reason, the DB is down, my application got stuck.
For example, I have REST resources (with Jersey), they are not getting any requests.
I also using quartz for schedule tasks, they aren't running as well.
When I start my DB again, everything goes back to normal.
I don't even know where to start looking, anyone has an idea?
Thanks

What must be happening is your application must be receiving request but there must be some exception while establishing database connection .see the logs.
try some flow where your are not doing any database operation. It must work fine.

When the application has hung, get a thread dump of the JVM, this will tell you the state of each thread and, rather than guessing as to the cause, you'll have concrete evidence.
Having said that, and going with the guess work approach, a lot comes down to how your connection pool is configured and the action your code takes when it receives the SQLException. If the application is totally hung, then my first port of call would be to find out if the db accessing threads are in a WAIT state, or even deadlocked. The thread dump will help you to determine if that is the case.
see kill -3 to get java thread dump for how to take a thread dump

JDBC requests to Oracle 11g failing to be commited although apparently succeding

We have an older web-based application (Java with Spring 2.5.4 framework) running on a GlassFish 3.1 (build 43) server. This application was recently (a few weeks ago) re-directed to use an Oracle 11g (11.2.0.3.0) database and ojdbc6.jar/orai18n.jar (up from Oracle 10g 10.2.0.3.0 and ojdbc14.jar) -- using a JDBC Thin connection. The application is using org.apache.commons.dbcp.BasicDataSource version 1.2.2 for connections and the database requests are handled either through Spring jdbcTemplate (via the JdbcDaoSupport abstract class) or Spring's PlatformTransactionManager.
This morning we noticed that application users were able to enter information, modify it and later to retrieve and print that data through the application, but that there were no committed updates for the last 24 hours. This application currently has only a few users each day and they are apparently sharing the same connection which has been kept open by the connection pool during the last day and so their uncommitted updates were visible through the application, but not through other connections to the database. When the connection was closed, the uncommitted updates were lost.
Examining the server logs showed no errors from the time of the last committed changes to the database through the times of printed reports the next morning. In addition, even if some of the changes had been (somehow) made with the JDBC connection being set to Auto-Commit false, there were specific commits made for some of those updates that were part of a transaction which, as part of a try/catch block should have either executed one of the "transactionManager.commit(transactionStatus);" or "transactionManager.rollback(transactionStatus);" calls that must have been processed without error. It looks as though the commit was returning successfully, but no commit actually occurred.
Restarting the GlassFish domain and the application restored the normal operation with the various updates being committed as they are entered.
My question is has anyone seen or heard about anything like this occurring and, if so, what could have caused it?
Thank you for any ideas here -- we are at a loss.
Some new information:
Examination of our Oracle 11g Server showed that near the time that we believe that the commits appeared to stop, there were four operations blocked on some other operation that we were not able to fully resolve, but was probably an update.
Examination of the Glassfish Server logs showed that the appearance of the worker threads changed following this estimated start time and fewer threads were appearing in the log until only one thread continued to be used for several hours.
The problem occurred again about one week later and was caught after about 1/2 hour. At this time, there were two worker threads in operation.

The problem occurred due to a combination of two things. The first was a method that setup a Spring Transaction, but had an exit that bypassed both the TransactionManager.commit() and the TransactionManager.rollback() (as well as the several SQL requests making up the transaction). Although this was admittedly incorrect coding, in the past, this transaction was closed and therefore had no effect on subsequent usage.
The solution was to insure that the transaction was not started if there was nothing to be done; or, in general double check to make sure that all transactions, once started, are completed.
I am not certain of the exact how or why this problem began presenting itself, so the following is partly conjectured. Apparently, upgrading to Oracle 11g and/or switching to the ojdbc6.jar driver altered the earlier behavior of the incorrect code so that the transaction was not terminated and the connection auto-commit was left false. (It could also be due to some other change that we have not identified since the special case above happens rarely – but does happen.) The corresponding JDBC connection appears to be bound to a specific GlassFish worker thread (I will call this a 'bad' thread in the following as opposed to the normally acting 'good' threads). Whenever this 'bad' thread is used to handle an application request (for this particular application), changes are uncommitted and selects return dirty data. As time goes on, when a change is requested on a 'good' thread and JDBC connection that already has an uncommitted change made on the 'bad' thread, than the new request hangs and the worker thread also hangs. Eventually all but the 'bad' worker thread are hung and everything seems to work correctly from the application viewpoint, but nothing is ever committed.
Again, the solution was to correct the bad code.

spring batch remains in EXECUTING

I created a job which uses reader of type
org.springframework.batch.item.database.HibernateCursorItemReader to execute a query.
The problem is database connection in this case is hitting connection limit (I have a oracle error ORA-12519, TNS:no appropriate service handler found) and, surprisingly, I noticed exit_code=EXECUTING and status=STARTED on BATCH_STEP_EXECUTION table.
If I run again the job it will respond "A job execution for this job is already running" and if I issue -restart on this task, it complains with message "No failed or stopped execution found for job".
How does spring batch manages these fatal failure situations? Do I have to remove these execution information manually or is there a reset option?
Thank you for any help

the current release of Spring Batch (2.2.0) doesn't appear to have an out of the box solution for this situation. as discussed in this question, 'manual' intervention in the database may be required. alternatively, if this is a particular job that is hanging (that is, you know the job name), you can do the following as well;
use the JobExplorer.findRunningJobExecutions(jobName)
go through the list of executions and 'fail' them (JobExecution.upgradeStatus(BatchStatus.FAILED))
save the change using JobRepository.update(jobExecution)

Just an FYI why this problem of connection limit occurs when using a CursorItemReader (JDBCCursorItemReader or HibernateCursorItemReader)
The cursorItemReader opens a separate connection even if there is already a connection opened for the transaction (Reader -> Processors -> Writer). So, each step execution needs two connections even if it is in a single transaction and hitting the same db. This causes the connection bottleneck and so the number of db connections should be double the number of threads configured in thread pool to execute the steps in parallel.
This can also be resolve if you provide a separate connection to your CursorReader.
JdbcPagingItemReader is another implementation of ItemReader which uses the same connection opened for the transaction.

An SQLException was provoked, java.lang.InterruptedException, am I running out of db connections?

So we run a Hibernate, Spring, Spring Webflow stack. From what I've read so far it might also be important to know we use c3p0-0.9.1.2.
Over the last couple of days we've noticed the server suddenly stop. Users cannot log into the website, nothing appears to happen, the browser simply sits loading the page forever. The server logs also simply halt.
When we notice this we shutdown the tomcat instance and all of a sudden quite a few of the following errors get logged;
13:05:57.492 [TP-Processor7] WARN o.h.util.JDBCExceptionReporter - SQL Error: 0, SQLState: null
13:05:57.492 [TP-Processor7] ERROR o.h.util.JDBCExceptionReporter - An SQLException was provoked by the following failure: java.lang.InterruptedException
Any ideas what these mean? Google hasn't been too helpful. Are we leaking db connections somewhere and the pool cannot gain a new session?
We have just put in a couple new Spring Webflow flows and are experiencing a slightly increased amount of website traffic but we haven't seen this behaviour before.

I suspect those InterruptExceptions come from the actual shutdown of those threads by the container, and simply indicate that those threads are existant when Tomcat shuts down.
Instead, I would grab a thread dump from Tomcat when it next freezes. I would also get a DBA to tell you what's happening in the database. From the above I'm guessing you're hung on a database resource, but a thread dump and analysis from a DBA will certainly point you in the right direction.
Here's a Thread Dump JSP as an alternative means of generating thread dumps.

javax.persistence.PersistenceException: Transaction failed to flush

I got the JPA exception
"javax.persistence.PersistenceException:
Transaction failed to flush"
Then I deleted my local datastore(datastore-indexes-auto.xml and local_db.bin) from my system. Recreated all the data again and after that, the exception was gone. I want to know what did just happened ?
The following is the stacktrace
[RPC Fault faultString="org.springframework.orm.jpa.JpaSystemException : Transaction failed to flush; nested exception is javax.persistence.PersistenceException: Transaction failed to flush" faultCode="Server.Processing" faultDetail="null"]
at mx.rpc::AbstractInvoker/http://www.adobe.com/2006/flex/mx/internal::faultHandler()[C:\autobuild\3.5.0\frameworks\projects\rpc\src\mx\rpc\AbstractInvoker.as:290]
at mx.rpc::Responder/fault()[C:\autobuild\3.5.0\frameworks\projects\rpc\src\mx\rpc\Responder.as:58]
at mx.rpc::AsyncRequest/fault()[C:\autobuild\3.5.0\frameworks\projects\rpc\src\mx\rpc\AsyncRequest.as:103]
at NetConnectionMessageResponder/statusHandler()[C:\autobuild\3.5.0\frameworks\projects\rpc\src\mx\messaging\channels\NetConnectionChannel.as:581]
at mx.messaging::MessageResponder/status()[C:\autobuild\3.5.0\frameworks\projects\rpc\src\mx\messaging\MessageResponder.as:222]

I don't know google-app-engine, but I assume you have some limited space in DB there? Maybe you just run out of space?

I believe it is due to this problem with the time it takes for AppEngine to start up, thereby causing timeout errors.
http://googleappengine.blogspot.com/2009/12/request-performance-in-java.html
If you've been following the App Engine Java runtime group, you may
have noticed some discussions about performance of the Java runtime.
Many of you have complained about hard-to-predict
DeadlineExceededExceptions, or unexpectedly slow requests that use a
high amount of CPU. These issues often have the same root cause: App
Engine is preparing a new instance of your code to respond an incoming
request.
It was reported by Grails http://jira.grails.org/browse/GPAPPENGINE-67
There is an open issue Google has not fixed yet, even after several years.
https://code.google.com/p/googleappengine/issues/detail?id=7706
As a Java project becomes more complicated and requires loading more
classes & jars at startup, instance startup time degrades to the point
where instances blow the 60s user-facing request deadline.
You MIGHT be able to work around this by keeping an idle instance resident in memory so it doesn't have to spin up.
https://developers.google.com/appengine/docs/adminconsole/performancesettings#scheduler
https://appengine.google.com/settings