We Keep Coding

Spring batch long running tasklet database timeout

Given I have this Spring Batch configuration for my workflow job and I am using Sql Server database for my spring batch tables:
public class MyConfiguration extends AbstractConfiguration {
#Bean
#Qualifier("pollStep")
public Step pollStep() {
return stepBuilderFactory.get("pollStep")
.tasklet(filePollingTasklet())
.listener(promoteContextListener())
.build();
}
#Bean
#StepScope
private Tasklet filePollingTasklet() {
return ((stepContribution, chunkContext) -> getStatus(stepContribution, chunkContext));
}
private RepeatStatus getStatus(StepContribution stepContribution, ChunkContext chunkContext) {
//some code
Map<String, Boolean> result = poller.pollForFile(myContext, sourceInfo);
return RepeatStatus.FINISHED;
}
}
My application polls for a file on remote server. After 100 mins when it can't find a file the poller.pollForFile() throws a runtime exception and my step status is UNKNOWN and the application exits with exceptions:
c.m.s.j.SQLServerException: Connection reset at
c.m.s.j.SQLServerConnection.terminate(SQLServerConnection.java:1667) at
c.m.s.j.SQLServerConnection.terminate(SQLServerConnection.java:1654) at
c.m.s.j.TDSChannel.write(IOBuffer.java:1805) at c.m.s.jdbc.TDSWriter.flush(IOBuffer.java:3581) at
c.m.s.jdbc.TDSWriter.writePacket(IOBuffer.java:3482) at
c.m.s.jdbc.TDSWriter.endMessage(IOBuffer.java:3062) at
c.m.s.j.TDSCommand.startResponse(IOBuffer.java:6120) at
c.m.s.j.TDSCommand.startResponse(IOBuffer.java:6106) at
c.m.s.j.SQLServerConnection$1ConnectionCommand.doExecute(SQLServerConnection.java:1756) at
c.m.s.j.TDSCommand.execute(IOBuffer.java:5696) at
c.m.s.j.SQLServerConnection.executeCommand(SQLServerConnection.java:1715) at
c.m.s.j.SQLServerConnection.connectionCommand(SQLServerConnection.java:1761) at
c.m.s.j.SQLServerConnection.rollback(SQLServerConnection.java:1964) at
c.z.h.p.ProxyConnection.rollback(ProxyConnection.java:375) at
c.z.h.p.HikariProxyConnection.rollback(HikariProxyConnection.java) at
o.h.r.j.i.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:116) ... 50 common frames omitted Wrapped by: u003c#7f0e356au003e o.h.TransactionException: Unable to rollback against JDBC Connection at ...
I think the sql server db connection is timed out and closed and spring batch is unable to perform rollback and db updates. Ideally, I want status to be FAILED which it is when I run locally with H2 but on this instance what strategy or techniques can I use to overcome this issue? The exit message doesnt have the error from exception thrown by pollForFile(), instead it is org.springframework.transaction.TransactionSystemException: Could not roll back JPA transaction; nested exception is org.hibernate.TransactionException: Unable to rollback against JDBC Connectionat
Is there a way to fix this issue? What if I were to move from tasklet to chunk-oriented and perform the poll logic in read() method of ItemReader ?

Your thinking is correct. When the commit fails, Spring Batch is unable to correctly update the step status which ends in UNKNOWN instead of FAILED. There is an open issue for that here: https://github.com/spring-projects/spring-batch/issues/1826. While your exception is different, the problem is the same. I had an attempt to fix that here: https://github.com/spring-projects/spring-batch/pull/591 but I decided to discard it (you can find more details about the reasons in that PR).
To work around the issue, you need to make sure any (runtime) exception is handled in the tasklet (or in item writer in case of a chunk-oriented step). In your case, you can increase the timeout of your transaction and catch runtime exception in the tasklet (which you can wrap in a meaningful exception that you re-throw from the tasklet to make it fail).
EDIT: add example of increasing transaction timeout
#Bean
#Qualifier("pollStep")
public Step pollStep() {
DefaultTransactionAttribute attribute = new DefaultTransactionAttribute();
attribute.setTimeout(60 * 100);
// set other transaction attributes
return stepBuilderFactory.get("pollStep")
.tasklet(filePollingTasklet())
.transactionAttribute(attribute)
.listener(promoteContextListener())
.build();
}

Hikari Connection Pool - Slow , Block , Connection is not available : SpringBoot

Problem:
I have a springboot application which has hikari configured (auto). I'm getting error
Connection is not available, request timed out after 30113ms
when I just do an insert operation in database and flow is like Controller > Service > Repository > save(entity) also not using #Transactional in repository, but the result is the same if I use it.
While load test 50request/1sec to this service sequentially getting success for 20-30 requests? remaining failed with below exception.
2019-03-28 20:58:29.507 ERROR 90260 --- [http-nio-8080-exec-234] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection] with root cause
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30113ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:697) ~[HikariCP-3.3.1.jar:na]
I am doing kind of load testing as triggering 50req/1sec and half and half getting success and failure. Enabled leak detection also but no trace in log.
Am I overdoing the configurations for this test or should I need to tune the pool connections? or it supports only that much?
Also hikari getconnection after 2nd request and subsequent requests takes almost increased 5+ seconds (blocks) why? Its not parallel why? Please help me or guide me on how much I need to tune to accept like 200 request per 1 min.
application.yml
spring:
application:
name: demo
datasource:
hikari:
connection-timeout: 20000
minimum-idle: 5
maximum-pool-size: 50
idle-timeout: 300000
max-lifetime: 1200000
auto-commit: true
driver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriver
jdbc-url:: jdbc:sqlserver://ip:port;databaseName=sample
username: username
leak-detection-threshold: 30000
BootApplication.java
#SpringBootApplication
public class Sample{
public static void main(String[] args) {
SpringApplication.run(Sample.class, args);
}
#Bean
#ConfigurationProperties(prefix = "spring.datasource.hikari")
public DataSource dataSource() {
HikariDataSource dataSource=new HikariDataSource();
//configuring pass from vault
return dataSource;
}
}
SampleService.java
#Service
public class SampleService implements SampleService {
#Autowired
private SampleRepository sampleRepository;
#Override
public List<String> getAll() {
return (List<String>) sampleRepository.findAll();
}
#Override
public String saveOrUpdate(Sample obj) {
return sampleRepository.save(obj);
}
}

How rollback transaction after timeout in spring boot application in same way as on weblogic

So in my weblogic application we are you using some jtaWeblogicTransactionManager. There is some default timeout which can be override in annotation #Transactional(timeout = 60). I created some infinity loop to read data from db which correctly timeout:
29 Apr 2018 20:44:55,458 WARN [[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'] org.springframework.jdbc.support.SQLErrorCodesFactory : Error while extracting database name - falli
ng back to empty error codes
org.springframework.jdbc.support.MetaDataAccessException: Error while extracting DatabaseMetaData; nested exception is java.sql.SQLException: Unexpected exception while enlisting XAConnection java.sql.SQLExceptio
n: Transaction rolled back: Transaction timed out after 240 seconds
BEA1-2C705D7476A3E21D0AB1
at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1760)
at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1645)
at weblogic.jdbc.wrapper.JTAConnection.getXAConn(JTAConnection.java:232)
at weblogic.jdbc.wrapper.JTAConnection.checkConnection(JTAConnection.java:94)
at weblogic.jdbc.wrapper.JTAConnection.checkConnection(JTAConnection.java:77)
at weblogic.jdbc.wrapper.Connection.preInvocationHandler(Connection.java:107)
at weblogic.jdbc.wrapper.Connection.getMetaData(Connection.java:560)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:331)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:366)
at org.springframework.jdbc.support.SQLErrorCodesFactory.getErrorCodes(SQLErrorCodesFactory.java:212)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.setDataSource(SQLErrorCodeSQLExceptionTranslator.java:134)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.<init>(SQLErrorCodeSQLExceptionTranslator.java:97)
at org.springframework.jdbc.support.JdbcAccessor.getExceptionTranslator(JdbcAccessor.java:99)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:655)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:690)
now I would like to make same behavior in my spring boot application so I tried this:
#EnableTransactionManagement
.
.
.
#Bean(name = "ds1")
#ConfigurationProperties(prefix = "datasource.ds1")
public DataSource logDataSource() {
AtomikosDataSourceBean ds = new AtomikosDataSourceBean();
return ds;
}
#Bean(name = "ds2")
#ConfigurationProperties(prefix = "datasource.ds2")
public DataSource refDataSource() {
AtomikosDataSourceBean ds = new AtomikosDataSourceBean();
return ds;
}
tm:
#Bean(name = "userTransaction")
public UserTransaction userTransaction() throws Throwable {
UserTransactionImp userTransactionImp = new UserTransactionImp();
userTransactionImp.setTransactionTimeout(120);
return userTransactionImp;
}
#Bean(name = "atomikosTransactionManager", initMethod = "init", destroyMethod = "close")
public TransactionManager atomikosTransactionManager() throws Throwable {
UserTransactionManager userTransactionManager = new UserTransactionManager();
userTransactionManager.setForceShutdown(false);
userTransactionManager.setTransactionTimeout(120);
return userTransactionManager;
}
#Bean(name = "transactionManager")
#DependsOn({ "userTransaction", "atomikosTransactionManager" })
public JtaTransactionManager transactionManager() throws Throwable {
UserTransaction userTransaction = userTransaction();
TransactionManager atomikosTransactionManager = atomikosTransactionManager();
return new JtaTransactionManager(userTransaction, atomikosTransactionManager);
}
and application.properties:
datasource.ref.xa-data-source-class-name=oracle.jdbc.xa.client.OracleXADataSource
datasource.ref.unique-resource-name=ref
datasource.ref.xa-properties.URL=jdbc:oracle:thin:#...
datasource.ref.xa-properties.user=...
#datasource.ref.xa-properties.databaseName=...
datasource.ref.password=301d24ae7d0d69614734a499df85f1e2
datasource.ref.test-query=SELECT 1 FROM DUAL
datasource.ref.max-pool-size=5
datasource.log.xa-data-source-class-name=oracle.jdbc.xa.client.OracleXADataSource
datasource.log.unique-resource-name=log
datasource.log.xa-properties.URL=jdbc:oracle:thin:#...
datasource.log.xa-properties.user=...
#datasource.log.xa-properties.databaseName=...
datasource.log.password=e58605c2a0b840b7c6d5b20b3692c5db
datasource.log.test-query=SELECT 1 FROM DUAL
datasource.log.max-pool-size=5
spring.jta.atomikos.properties.log-base-dir=target/transaction-logs/
spring.jta.enabled=true
spring.jta.atomikos.properties.service=com.atomikos.icatch.standalone.UserTransactionServiceFactory
spring.jta.atomikos.properties.max-timeout=600000
spring.jta.atomikos.properties.default-jta-timeout=10000
spring.transaction.default-timeout=900
but with no success. My infinity loop never ends (I wait about 15 minutes and then I stop my app). The only time when I saw rollback was when I tried Thread.sleep and after sleep this transaction timeout with rollback but this is not what I want to. So is there some way how to interrupt process after timeout(use timeout in annotation or use default) in same way how in my weblogic application ?
UPDATE
I tested it like this:
public class MyService {
public void customMethod(){
customDao.readSomething();
}
}
public class CustomDao {
#Transactional(timeout = 120)
public void readSomething()
while(true){
//read data from db. app on weblogic throw timeout, spring boot app in docker did nothing and after 15 I give it up and kill it
}
}
}
UPDATE2
When I turn on atomikos debug I can see there is warning during init and some atomikos timer:
2018-05-03 14:00:54.833 [main] WARN c.a.r.xa.XaResourceRecoveryManager - Error while retrieving xids from resource - will retry later...
javax.transaction.xa.XAException: null
at oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:730)
at com.atomikos.datasource.xa.RecoveryScan.recoverXids(RecoveryScan.java:32)
at com.atomikos.recovery.xa.XaResourceRecoveryManager.retrievePreparedXidsFromXaResource(XaResourceRecoveryManager.java:158)
at com.atomikos.recovery.xa.XaResourceRecoveryManager.recover(XaResourceRecoveryManager.java:67)
at com.atomikos.datasource.xa.XATransactionalResource.recover(XATransactionalResource.java:449)
at com.atomikos.datasource.xa.XATransactionalResource.setRecoveryService(XATransactionalResource.java:416)
at com.atomikos.icatch.config.Configuration.notifyAfterInit(Configuration.java:466)
at com.atomikos.icatch.config.Configuration.init(Configuration.java:450)
at com.atomikos.icatch.config.UserTransactionServiceImp.initialize(UserTransactionServiceImp.java:105)
at com.atomikos.icatch.config.UserTransactionServiceImp.init(UserTransactionServiceImp.java:219)
at com.atomikos.icatch.jta.UserTransactionImp.checkSetup(UserTransactionImp.java:59)
at com.atomikos.icatch.jta.UserTransactionImp.setTransactionTimeout(UserTransactionImp.java:127)
maybe this is the reason. How I can fix this ? I am using oracle 12 with ojdbc8 driver
UPDATE 3
after fix UPDATE2 to grant user permission to db I can see in log warning:
2018-05-03 15:16:30.207 [Atomikos:4] WARN c.a.icatch.imp.ActiveStateHandler - Transaction 127.0.1.1.tm152535336001600001 has timed out and will rollback.
problem is that app is still reading data from db after this timeout. Why it is not rollbacked ?
UPDATE 4
so I found in ActiveStateHandler when timeout occurs there is code:
...
setState ( TxState.ACTIVE );
...
and AtomikosConnectionProxy is checking timeout this way
if ( ct.getState().equals(TxState.ACTIVE) ) ct.registerSynchronization(new JdbcRequeueSynchronization( this , ct ));
else AtomikosSQLException.throwAtomikosSQLException("The transaction has timed out - try increasing the timeout if needed");
so why timeout is set state which not cause exception in AtomikosConnectionProxy ?
UPDATE 5
so I found that property
com.atomikos.icatch.threaded_2pc
will solve my problem and now it starts rollback how I want. But I still dont understand why I should set this to true because now I am testing it on some task which should run in single thread

set com.atomikos.icatch.threaded_2pc=true in jta.properties fixed my problem. Idk why this default value was change to false in web application.
* #param single_threaded_2pc (!com.atomikos.icatch.threaded_2pc)
* If true then commit is done in the same thread as the one that
* started the tx.

XA transactions are horribly complicated and you really want to have a very good reason for using them (ie it's literally impossible to add some business process that removes the need for XA), because you are going to get into trouble out in the wild...
That said, My guess is that it's about timeout discrepancies between XA phases.
With XA there are 2 timeouts - a timeout for the 1st phase, known as the Voting phase (which is typically the one set by the #Transactional annotation, but this depends on the JTA provider) and another timeout for the 2nd phase, known as the commit phase, which is typically a lot longer, because the Transaction Manager has already got the agreement from all parties that the commit is ready to go, and therefore provide greater leeway for things like transient network failures and so on.
My guess is that the WebLogic JTA is simply behaving differently to Atomikos with how it's handling the 2nd phase notifications back from the participants, until atomikos is changed to use the multithreaded ack.
If you application is just you and the database, then you can probably get away without an XA Transaction Manager. I'd expect this would behave the way you want for timeouts.
Good Luck!

Spring Boot- Throw an Error and Stop application starting up on some criteria

I am working on an application where in a bean's post construct method i am adding some logic to create an object. Now , what I want to do is, if there is an exception and the creation of the object has some error, then do not let the application start up. Yes, I can see an exception being thrown on the console when it starts up, if there is any issue with the construction of an object, but I would like something better than that to inform me that construction of an object has failed, and what better criteria than application failing to start up.
Any help is much appreciated.
Thank you in advance.

You can look for FailureAnalyzer for this type of requirements where additional information will given in case application start failes. If any exception raised during application start, all the FailureAnalyzer classes will be invoked in a sequence. If any of the FailureAnalyzer class returning FailureAnalysis object then the exception won't be propagated to further FailureAnalysis classes.
Please make sure you register your FailureAnalysis class in resource/META-INF/spring.factories file.
#Component
public class SomeObject {
#PostConstruct
public void init() throws Exception {
throw new Exception("SomeObject init threw exception");
}
}
public class ObjConstructionFailureAnalyzer extends
AbstractFailureAnalyzer<BeanCreationException> {
#Override
protected FailureAnalysis analyze(Throwable rootFailure,
BeanCreationException cause) {
System.out.println("\n===>ObjConstructionFailureAnalyzer::analyze()\n");
String desciption = "Object creation failed, [Reason]: " +
cause.getMessage();
String action = "Please handle exceptions in your init methods";
return new FailureAnalysis(desciption, action, cause);
}
}
In spring.factories file
org.springframework.boot.diagnostics.FailureAnalyzer=examples.stackoverflow.ObjConstructionFailureAnalyzer
Exception stacktrace
===>ObjConstructionFailureAnalyzer::analyze()
2018-02-21 10:16:59.552 ERROR 9500 --- [ main]
o.s.b.d.LoggingFailureAnalysisReporter :
*************************** APPLICATION FAILED TO START
Description:
Object creation failed, [Reason]: Error creating bean with name
'someObject': Invocation of init method failed; nested exception is
java.lang.Exception: SomeObject init threw exception
Action:
Please handle exceptions in your init methods
You can additionally visit here for code sample.

Spring UnknownServiceException

I have scheduled task in separate thread that access #Transactional service.
In this task I try to access method like this
List<Obj> objects = new ArrayList<Obj>(objService.getObjWithStatus(Status.PROCESSING));
and sometimes get the exception
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.service.UnknownServiceException: Unknown service requested [org.hibernate.engine.jdbc.connections.spi.ConnectionProvider]
at org.springframework.orm.hibernate4.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:544)
Caused by: org.hibernate.service.UnknownServiceException: Unknown service requested [org.hibernate.engine.jdbc.connections.spi.ConnectionProvider]
I tried to synchronize access to the service over all threads, but exception still occures randomly. Is there any way to prevent this kind of error?

It seems that spring doesn't close scheduled tasks on exit, and they are keep running without context. Adding
#PreDestroy
private void destroyListener() {
synchronized (this) {
scheduledTask.cancel(false);
}
}
seems to fix this issue