We Keep Coding

Spring batch long running tasklet database timeout

Given I have this Spring Batch configuration for my workflow job and I am using Sql Server database for my spring batch tables:
public class MyConfiguration extends AbstractConfiguration {
#Bean
#Qualifier("pollStep")
public Step pollStep() {
return stepBuilderFactory.get("pollStep")
.tasklet(filePollingTasklet())
.listener(promoteContextListener())
.build();
}
#Bean
#StepScope
private Tasklet filePollingTasklet() {
return ((stepContribution, chunkContext) -> getStatus(stepContribution, chunkContext));
}
private RepeatStatus getStatus(StepContribution stepContribution, ChunkContext chunkContext) {
//some code
Map<String, Boolean> result = poller.pollForFile(myContext, sourceInfo);
return RepeatStatus.FINISHED;
}
}
My application polls for a file on remote server. After 100 mins when it can't find a file the poller.pollForFile() throws a runtime exception and my step status is UNKNOWN and the application exits with exceptions:
c.m.s.j.SQLServerException: Connection reset at
c.m.s.j.SQLServerConnection.terminate(SQLServerConnection.java:1667) at
c.m.s.j.SQLServerConnection.terminate(SQLServerConnection.java:1654) at
c.m.s.j.TDSChannel.write(IOBuffer.java:1805) at c.m.s.jdbc.TDSWriter.flush(IOBuffer.java:3581) at
c.m.s.jdbc.TDSWriter.writePacket(IOBuffer.java:3482) at
c.m.s.jdbc.TDSWriter.endMessage(IOBuffer.java:3062) at
c.m.s.j.TDSCommand.startResponse(IOBuffer.java:6120) at
c.m.s.j.TDSCommand.startResponse(IOBuffer.java:6106) at
c.m.s.j.SQLServerConnection$1ConnectionCommand.doExecute(SQLServerConnection.java:1756) at
c.m.s.j.TDSCommand.execute(IOBuffer.java:5696) at
c.m.s.j.SQLServerConnection.executeCommand(SQLServerConnection.java:1715) at
c.m.s.j.SQLServerConnection.connectionCommand(SQLServerConnection.java:1761) at
c.m.s.j.SQLServerConnection.rollback(SQLServerConnection.java:1964) at
c.z.h.p.ProxyConnection.rollback(ProxyConnection.java:375) at
c.z.h.p.HikariProxyConnection.rollback(HikariProxyConnection.java) at
o.h.r.j.i.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:116) ... 50 common frames omitted Wrapped by: u003c#7f0e356au003e o.h.TransactionException: Unable to rollback against JDBC Connection at ...
I think the sql server db connection is timed out and closed and spring batch is unable to perform rollback and db updates. Ideally, I want status to be FAILED which it is when I run locally with H2 but on this instance what strategy or techniques can I use to overcome this issue? The exit message doesnt have the error from exception thrown by pollForFile(), instead it is org.springframework.transaction.TransactionSystemException: Could not roll back JPA transaction; nested exception is org.hibernate.TransactionException: Unable to rollback against JDBC Connectionat
Is there a way to fix this issue? What if I were to move from tasklet to chunk-oriented and perform the poll logic in read() method of ItemReader ?

Your thinking is correct. When the commit fails, Spring Batch is unable to correctly update the step status which ends in UNKNOWN instead of FAILED. There is an open issue for that here: https://github.com/spring-projects/spring-batch/issues/1826. While your exception is different, the problem is the same. I had an attempt to fix that here: https://github.com/spring-projects/spring-batch/pull/591 but I decided to discard it (you can find more details about the reasons in that PR).
To work around the issue, you need to make sure any (runtime) exception is handled in the tasklet (or in item writer in case of a chunk-oriented step). In your case, you can increase the timeout of your transaction and catch runtime exception in the tasklet (which you can wrap in a meaningful exception that you re-throw from the tasklet to make it fail).
EDIT: add example of increasing transaction timeout
#Bean
#Qualifier("pollStep")
public Step pollStep() {
DefaultTransactionAttribute attribute = new DefaultTransactionAttribute();
attribute.setTimeout(60 * 100);
// set other transaction attributes
return stepBuilderFactory.get("pollStep")
.tasklet(filePollingTasklet())
.transactionAttribute(attribute)
.listener(promoteContextListener())
.build();
}

weblogic.transaction.internal.AppSetRollbackOnlyException: setRollbackOnly called on transaction

I have the bellow case, Where I call doSomeTask() of BeanA but if doSomeTask() fails I want to persist ErrorInfo into another table as well calling the saveError(ErrorInfo) of BeanA. Both of them has #TransactionAttribute(REQUIRES_NEW).
class BeanA {
#TransactionAttribute(REQUIRES_NEW)
public void doSomeTask(){
if(someCondition){
throw new SomeException();
}
// do task
}
#TransactionAttribute(REQUIRES_NEW)
public void saveError(ErrorInfo error) {
// save error info if doSomeTask fails
}
}
class BeanB {
BeanA beanA;
void performTask(){
try{
beanA.doSomeTask();
}catch(Exception e){
ErrorInfo error = getErrorInfo(e)
beanA.saveError(error);
}
}
}
But when doSomeTask() throws Exception saveError() doesn't work and throws Exception
Caused by: weblogic.transaction.internal.AppSetRollbackOnlyException: setRollbackOnly called on transaction
What am I doing wrong and how to fix this error? Thanks in advance for any help.

Sorry, for the late answer. The problem was resolved.
The actual error was hidden. In my case, the actual error was just a JSR 303 validation error of ErrorInfo instance while persisting. Had to add
Dweblogic.transaction.allowOverrideSetRollbackReason=true
on <domain_home>/bin/setDomainEnv.sh to find out the actual error and fix it. Thanks to this answer https://stackoverflow.com/a/38584687/1563286

I have debugged the similar issue not so long ago. In my case the issue was the following:
there was a top level transaction open when the REQUIRES_NEW method is called
After exception in nested transaction commit of the top level one failed to commit as "marked as rollback only"
It turned out that through a new transaction is started the connection holder is shared on TransactionManager level. When exception is thrown inside nested transaction the connection itself is marked as rollback only. So later this is causing the issue.
I was able to resolve the issue by using the savepoints (available since JDBC 3.0). Usually savepoints are disabled in many environments/ORM by default and using them requires additional configuration.
Hope this is of some help.

Java/Postgres: Deadlock with Flyway clean

I have a scheduled task working (including writing!) on my PostgreSQL database:
#Transactional
#Scheduled(fixedDelay = SCHEDULE_DELAY_IN_SECONDS * MILLISECONDS_PER_SECOND)
public synchronized void doStuff() {
// snip
}
And I do have a service to Flyway.clean() my database:
#Service
public class MyService {
#Transactional
public void deleteDatabase() {
flyway.clean();
flyway.migrate();
// snip
}
}
Now every once in a while when deleting the database a deadlock occurs:
26-Jan-2017 17:15:57.174 SEVERE [https-jsse-nio-8443-exec-4] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [dispatcher] in context with path [/foo] threw exception [Request processing failed; nested exception is org.flywaydb.core.api.FlywayException: Unable to drop "public"."bar"] with root cause
org.postgresql.util.PSQLException: ERROR: deadlock detected
Detail: Process 122 waits for AccessExclusiveLock on relation 69572 of database 16388; blocked by process 97.
Process 97 waits for AccessShareLock on relation 69575 of database 16388; blocked by process 122.
Hint: See server log for query details.
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2284)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2003)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:200)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:424)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:161)
at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:155)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:198)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:198)
at org.flywaydb.core.internal.dbsupport.JdbcTemplate.execute(JdbcTemplate.java:219)
at org.flywaydb.core.internal.dbsupport.postgresql.PostgreSQLTable.doDrop(PostgreSQLTable.java:43)
at org.flywaydb.core.internal.dbsupport.SchemaObject.drop(SchemaObject.java:80)
at org.flywaydb.core.internal.dbsupport.postgresql.PostgreSQLSchema.doClean(PostgreSQLSchema.java:84)
at org.flywaydb.core.internal.dbsupport.Schema.clean(Schema.java:148)
at org.flywaydb.core.internal.command.DbClean$4.doInTransaction(DbClean.java:182)
at org.flywaydb.core.internal.command.DbClean$4.doInTransaction(DbClean.java:180)
at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:72)
at org.flywaydb.core.internal.command.DbClean.cleanSchema(DbClean.java:180)
at org.flywaydb.core.internal.command.DbClean.clean(DbClean.java:130)
at org.flywaydb.core.Flyway$3.execute(Flyway.java:1017)
at org.flywaydb.core.Flyway$3.execute(Flyway.java:1013)
at org.flywaydb.core.Flyway.execute(Flyway.java:1361)
at org.flywaydb.core.Flyway.clean(Flyway.java:1013)
at com.acme.MyService.deleteDatabase(MyService.java:54)
// ...
And I suspect the scheduler to be this other process. Now what? I'd say Flyway - as the more invasive operator, plus the one throwing the error - has to wait for exclusive access, but how do I achieve this in the most efficient way? Especially as both methods are #Transactional already... Will
#Transactional(isolation = SERIALIZABLE)
help? This isolation level seems to be the most restrictive... Or is it just the missing synchronized of deleteDatabase()? This dead lock is kind of hard to reproduce, so any hint is appreciated.

Transactional error while commiting

I am trying to solve this error in Payara41 server Java EE 7, this sample works on WildFly-9 Java 7 EE and on Glassfish-3.1 Java EE 6 (without #Transactional and #TransactionalManagement)
#Stateful
#Transactional //default TxType.REQUIRED
#TransactionManagement(TransactionManagementType.BEAN)
public class ImprovementDaoImpl extends AbstractBaseDaoClass implements ImprovementDao {
/*^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^*/
#PersistenceContext(unitName = "pu", type = PersistenceContextType.EXTENDED)
private EntityManager em;
#Resource
private UserTransaction tx;
...
}
Here's some stacktrace, what apperas after executing tx.flush();:
javax.transaction.TransactionalException: Managed bean with Transactional annotation and TxType of REQUIRED encountered exception during commit javax.transaction.RollbackException: Transaction marked for rollback.
(...)
Caused by: javax.transaction.RollbackException: Transaction marked for rollback.
So far I've tried to use interceptor and #TransactionAttribute, but none helped...
Thanks for any advice/help! :)

I know this is old, but hopefully this helps someone out there...
Question 21363423: Throwing an application exception causes TransactionalException says
You are throwing an exception from a method whose invocation will be
intercepted at runtime and additional logic wrapped around it:
transaction management;
exception handling.
Your exception cannot transparently jump over that logic, and the
specification (probably) says a TransactionalException will be thrown,
wrapping your original exception...
Question 18888572: How do you find out what Exception caused the CDI Transaction Rollback?
Shows how to use a CDI Interceptor to catch the exceptions. I can't tell from the limited info from the OP what his/her specific issue is, but when I received this exception I has to review the WebLogic server log and found the entry where it told me a unique contraint was violated.. Time to add some interceptors...

Client's Transaction Aborted when accessing two databases

I have a propertiesbean which gives me SqlSessions.
This bean is annotated with all this jazz # the class level
#Singleton
#LocalBean
#Startup
#TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED)
i also have this
public SqlSessionFactory getSqlSessionFactory(ConnTypes connType) {...}
which returns the SqlSession you want.
I have two databases. One is a mysql instance and the other one isn't (lets call it db2).
When I run the project locally everything is fine. Both databases are accessed with no problem.
When I run the project on our test server it starts to throw a Client Transaction aborted error. I've done a fair amount of research on this and it seems like people get these exceptions when there is a problem with a database query or some database access. The entire transaction is then marked as rollback and this exception is thrown (at least that's what i've read)
It looks to me like it throws an XA-Resource error right before the first time it throws the client's transaction aborted error. I know that you can get those when your bean tries to access another session. I've seen this error before when running locally and trying to maintain connections to both databases in one method.
Could it be that my singleton properties bean, which accesses both databases, gets into a weird transactional state trying to return a mysql session in one thread and a db2 session in the other?