Hikari Connection Pool - Slow , Block , Connection is not available : SpringBoot - java

Problem:
I have a springboot application which has hikari configured (auto). I'm getting error
Connection is not available, request timed out after 30113ms
when I just do an insert operation in database and flow is like Controller > Service > Repository > save(entity) also not using #Transactional in repository, but the result is the same if I use it.
While load test 50request/1sec to this service sequentially getting success for 20-30 requests? remaining failed with below exception.
2019-03-28 20:58:29.507 ERROR 90260 --- [http-nio-8080-exec-234] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection] with root cause
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30113ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:697) ~[HikariCP-3.3.1.jar:na]
I am doing kind of load testing as triggering 50req/1sec and half and half getting success and failure. Enabled leak detection also but no trace in log.
Am I overdoing the configurations for this test or should I need to tune the pool connections? or it supports only that much?
Also hikari getconnection after 2nd request and subsequent requests takes almost increased 5+ seconds (blocks) why? Its not parallel why? Please help me or guide me on how much I need to tune to accept like 200 request per 1 min.
application.yml
spring:
application:
name: demo
datasource:
hikari:
connection-timeout: 20000
minimum-idle: 5
maximum-pool-size: 50
idle-timeout: 300000
max-lifetime: 1200000
auto-commit: true
driver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriver
jdbc-url:: jdbc:sqlserver://ip:port;databaseName=sample
username: username
leak-detection-threshold: 30000
BootApplication.java
#SpringBootApplication
public class Sample{
public static void main(String[] args) {
SpringApplication.run(Sample.class, args);
}
#Bean
#ConfigurationProperties(prefix = "spring.datasource.hikari")
public DataSource dataSource() {
HikariDataSource dataSource=new HikariDataSource();
//configuring pass from vault
return dataSource;
}
}
SampleService.java
#Service
public class SampleService implements SampleService {
#Autowired
private SampleRepository sampleRepository;
#Override
public List<String> getAll() {
return (List<String>) sampleRepository.findAll();
}
#Override
public String saveOrUpdate(Sample obj) {
return sampleRepository.save(obj);
}
}

Related

How to prevent CommunicationsException?

I am currently working with an app which using two different DB(different instance).
DB A is totally under A's project, however DB B is under the other project.(I am managing these via gcloud app engine.)
What is my problem :
DB B always disconnected if no request more than few hours with below error message.
{"timestamp":1555464776769,"status":500,"error":"Internal Server Error","exception":"org.springframework.transaction.CannotCreateTransactionException","message":"Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: com.mysql.cj.jdbc.exceptions.CommunicationsException: The last packet successfully received from the server was 43,738,243 milliseconds ago. The last packet sent successfully to the server was 43,738,243 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.","path":"/client/getAllCompany"}
To resolve this issue, I tried.
1) add 'autoReconnect=true' at application.properties
api.datasource.url = jdbc:mysql://google/projectB?cloudSqlInstance=projectB:australia-southeast1:projectB&socketFactory=com.google.cloud.sql.mysql.SocketFactory&useSSL=false&autoReconnect=true
2) add below config at application.properties.
spring.datasource.tomcat.test-while-idle=true
spring.datasource.tomcat.time-between-eviction-runs-millis=3600000
spring.datasource.tomcat.min-evictable-idle-time-millis=7200000
spring.datasource.tomcat.test-on-borrow=true
spring.datasource.tomcat.validation-query=SELECT 1
(My project doesn't have web.xml file)
If i re-deploy this project, i can access data from DB B as well.
How can I config to prevent killed the connectivity with DB B?
Wish to listen advice. Thank you in advanced.
HibernateConfig Code for DB B
#Bean(name = "apiDataSource")
#ConfigurationProperties(prefix = "api.datasource")
public DataSource dataSource() {
return DataSourceBuilder.create().build();
}
#Bean(name = "apiEntityManagerFactory")
public LocalContainerEntityManagerFactoryBean apiEntityManagerFactory(
EntityManagerFactoryBuilder builder, #Qualifier("apiDataSource") DataSource dataSource
) {
return builder.dataSource(dataSource).packages("com.workspez.api.entity").persistenceUnit("api").build();
}
#Bean(name = "apiTransactionManager")
public PlatformTransactionManager apiTransactionManager(
#Qualifier("apiEntityManagerFactory") EntityManagerFactory apiEntityManagerFactory
) {
return new JpaTransactionManager(apiEntityManagerFactory);
}
#Bean(name = "apiJdbc")
public NamedParameterJdbcTemplate apiJdbcTemplate() {
return new NamedParameterJdbcTemplate(dataSource());
}

Spring boot 2 connection to rabbitmq via Apache Camel

I have problem with connection to rabbitmq via Apache Camel on Spring Boot 2.
I did following steps:
My dependencies:
implementation "org.apache.camel:camel-spring-boot-starter:${camelVersion}"
implementation "org.apache.camel:camel-jackson-starter:${camelVersion}"
implementation "org.apache.camel:camel-core:${camelVersion}"
implementation "org.apache.camel:camel-rabbitmq-starter:${camelVersion}"
implementation "org.springframework.boot:spring-boot-starter-amqp"
Application.yaml
spring:
rabbitmq:
dynamic: true
host: 192.168.1.1
port: 5672
username: X
password: Y
And I have following route:
#Component
public class BasicRoute extends RouteBuilder {
#Override
public void configure() throws Exception {
from("direct:loggerQueue")
.id("loggerQueue")
.to("rabbitmq://TEST-QUEUE.exchange?queue=TEST-QUEUE.queue&autoDelete=false&connectionFactory=#rabbitConnectionFactory")
.end();
}
}
Finnaly I have still following issue:
2019-03-06 12:46:05.766 WARN 19464 --- [ restartedMain] o.a.c.c.rabbitmq.RabbitMQProducer : Failed to create connection. It will attempt to connect again when publishing a message.
java.net.ConnectException: Connection refused: connect
Connection seems ok, I tested it. Something is bad with rabbitConnectionFactory.
I don't know what I have bad.
The problem appears to be that RabbitMQComponent is expecting to find a connection factory of type com.rabbitmq.client.ConnectionFactory.
However, the springboot auto-configure is creating a connection factory of type org.springframework.amqp.rabbit.connection.CachingConnectionFactory.
So, whenever the RabbitMQComponent attempts to find the appropriate connection factory, because it is looking for the specific type, and because it does not subclass the rabbitmq ConnectionFactory, it returns a null value, and fails to use the appropriate host name and configuration parameters specified in your application.yml.
You should also see the following in your log if you have debug level set:
2019-12-15 17:58:53.631 DEBUG 48710 --- [ main] o.a.c.c.rabbitmq.RabbitMQComponent : Creating RabbitMQEndpoint with host null:0 and exchangeName: asterix
2019-12-15 17:58:55.927 DEBUG 48710 --- [ main] o.a.c.c.rabbitmq.RabbitMQComponent : Creating RabbitMQEndpoint with host null:0 and exchangeName: asterix-sink
EDIT:
The CachingConnectionFactory is configured with the required Rabbit connection factory as part of the autoconfiguration. However, you need to provide a link to the correct factory.
Therefore, you need to add a #Bean to disambiguate.
#Configuration
#RequiredArgsConstructor
public class CamelConfig {
private final CachingConnectionFactory rabbitConnectionFactory;
#Bean
com.rabbitmq.client.ConnectionFactory rabbitSourceConnectionFactory() {
return rabbitConnectionFactory.getRabbitConnectionFactory();
}
}
and in your endpoint configuration:
rabbitmq:asterix?connectionFactory=#rabbitSourceConnectionFactory
Note that the # is optional, as it gets stripped out within the code when it is trying to find the rabbit connection factory bean.
In your application.yml, configure the connection parameters (the url is no longer included in the endpoint URI).
spring:
rabbitmq:
host: localhost
port: 5672
username: guest
password: guest

How rollback transaction after timeout in spring boot application in same way as on weblogic

So in my weblogic application we are you using some jtaWeblogicTransactionManager. There is some default timeout which can be override in annotation #Transactional(timeout = 60). I created some infinity loop to read data from db which correctly timeout:
29 Apr 2018 20:44:55,458 WARN [[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'] org.springframework.jdbc.support.SQLErrorCodesFactory : Error while extracting database name - falli
ng back to empty error codes
org.springframework.jdbc.support.MetaDataAccessException: Error while extracting DatabaseMetaData; nested exception is java.sql.SQLException: Unexpected exception while enlisting XAConnection java.sql.SQLExceptio
n: Transaction rolled back: Transaction timed out after 240 seconds
BEA1-2C705D7476A3E21D0AB1
at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1760)
at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1645)
at weblogic.jdbc.wrapper.JTAConnection.getXAConn(JTAConnection.java:232)
at weblogic.jdbc.wrapper.JTAConnection.checkConnection(JTAConnection.java:94)
at weblogic.jdbc.wrapper.JTAConnection.checkConnection(JTAConnection.java:77)
at weblogic.jdbc.wrapper.Connection.preInvocationHandler(Connection.java:107)
at weblogic.jdbc.wrapper.Connection.getMetaData(Connection.java:560)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:331)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:366)
at org.springframework.jdbc.support.SQLErrorCodesFactory.getErrorCodes(SQLErrorCodesFactory.java:212)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.setDataSource(SQLErrorCodeSQLExceptionTranslator.java:134)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.<init>(SQLErrorCodeSQLExceptionTranslator.java:97)
at org.springframework.jdbc.support.JdbcAccessor.getExceptionTranslator(JdbcAccessor.java:99)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:655)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:690)
now I would like to make same behavior in my spring boot application so I tried this:
#EnableTransactionManagement
.
.
.
#Bean(name = "ds1")
#ConfigurationProperties(prefix = "datasource.ds1")
public DataSource logDataSource() {
AtomikosDataSourceBean ds = new AtomikosDataSourceBean();
return ds;
}
#Bean(name = "ds2")
#ConfigurationProperties(prefix = "datasource.ds2")
public DataSource refDataSource() {
AtomikosDataSourceBean ds = new AtomikosDataSourceBean();
return ds;
}
tm:
#Bean(name = "userTransaction")
public UserTransaction userTransaction() throws Throwable {
UserTransactionImp userTransactionImp = new UserTransactionImp();
userTransactionImp.setTransactionTimeout(120);
return userTransactionImp;
}
#Bean(name = "atomikosTransactionManager", initMethod = "init", destroyMethod = "close")
public TransactionManager atomikosTransactionManager() throws Throwable {
UserTransactionManager userTransactionManager = new UserTransactionManager();
userTransactionManager.setForceShutdown(false);
userTransactionManager.setTransactionTimeout(120);
return userTransactionManager;
}
#Bean(name = "transactionManager")
#DependsOn({ "userTransaction", "atomikosTransactionManager" })
public JtaTransactionManager transactionManager() throws Throwable {
UserTransaction userTransaction = userTransaction();
TransactionManager atomikosTransactionManager = atomikosTransactionManager();
return new JtaTransactionManager(userTransaction, atomikosTransactionManager);
}
and application.properties:
datasource.ref.xa-data-source-class-name=oracle.jdbc.xa.client.OracleXADataSource
datasource.ref.unique-resource-name=ref
datasource.ref.xa-properties.URL=jdbc:oracle:thin:#...
datasource.ref.xa-properties.user=...
#datasource.ref.xa-properties.databaseName=...
datasource.ref.password=301d24ae7d0d69614734a499df85f1e2
datasource.ref.test-query=SELECT 1 FROM DUAL
datasource.ref.max-pool-size=5
datasource.log.xa-data-source-class-name=oracle.jdbc.xa.client.OracleXADataSource
datasource.log.unique-resource-name=log
datasource.log.xa-properties.URL=jdbc:oracle:thin:#...
datasource.log.xa-properties.user=...
#datasource.log.xa-properties.databaseName=...
datasource.log.password=e58605c2a0b840b7c6d5b20b3692c5db
datasource.log.test-query=SELECT 1 FROM DUAL
datasource.log.max-pool-size=5
spring.jta.atomikos.properties.log-base-dir=target/transaction-logs/
spring.jta.enabled=true
spring.jta.atomikos.properties.service=com.atomikos.icatch.standalone.UserTransactionServiceFactory
spring.jta.atomikos.properties.max-timeout=600000
spring.jta.atomikos.properties.default-jta-timeout=10000
spring.transaction.default-timeout=900
but with no success. My infinity loop never ends (I wait about 15 minutes and then I stop my app). The only time when I saw rollback was when I tried Thread.sleep and after sleep this transaction timeout with rollback but this is not what I want to. So is there some way how to interrupt process after timeout(use timeout in annotation or use default) in same way how in my weblogic application ?
UPDATE
I tested it like this:
public class MyService {
public void customMethod(){
customDao.readSomething();
}
}
public class CustomDao {
#Transactional(timeout = 120)
public void readSomething()
while(true){
//read data from db. app on weblogic throw timeout, spring boot app in docker did nothing and after 15 I give it up and kill it
}
}
}
UPDATE2
When I turn on atomikos debug I can see there is warning during init and some atomikos timer:
2018-05-03 14:00:54.833 [main] WARN c.a.r.xa.XaResourceRecoveryManager - Error while retrieving xids from resource - will retry later...
javax.transaction.xa.XAException: null
at oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:730)
at com.atomikos.datasource.xa.RecoveryScan.recoverXids(RecoveryScan.java:32)
at com.atomikos.recovery.xa.XaResourceRecoveryManager.retrievePreparedXidsFromXaResource(XaResourceRecoveryManager.java:158)
at com.atomikos.recovery.xa.XaResourceRecoveryManager.recover(XaResourceRecoveryManager.java:67)
at com.atomikos.datasource.xa.XATransactionalResource.recover(XATransactionalResource.java:449)
at com.atomikos.datasource.xa.XATransactionalResource.setRecoveryService(XATransactionalResource.java:416)
at com.atomikos.icatch.config.Configuration.notifyAfterInit(Configuration.java:466)
at com.atomikos.icatch.config.Configuration.init(Configuration.java:450)
at com.atomikos.icatch.config.UserTransactionServiceImp.initialize(UserTransactionServiceImp.java:105)
at com.atomikos.icatch.config.UserTransactionServiceImp.init(UserTransactionServiceImp.java:219)
at com.atomikos.icatch.jta.UserTransactionImp.checkSetup(UserTransactionImp.java:59)
at com.atomikos.icatch.jta.UserTransactionImp.setTransactionTimeout(UserTransactionImp.java:127)
maybe this is the reason. How I can fix this ? I am using oracle 12 with ojdbc8 driver
UPDATE 3
after fix UPDATE2 to grant user permission to db I can see in log warning:
2018-05-03 15:16:30.207 [Atomikos:4] WARN c.a.icatch.imp.ActiveStateHandler - Transaction 127.0.1.1.tm152535336001600001 has timed out and will rollback.
problem is that app is still reading data from db after this timeout. Why it is not rollbacked ?
UPDATE 4
so I found in ActiveStateHandler when timeout occurs there is code:
...
setState ( TxState.ACTIVE );
...
and AtomikosConnectionProxy is checking timeout this way
if ( ct.getState().equals(TxState.ACTIVE) ) ct.registerSynchronization(new JdbcRequeueSynchronization( this , ct ));
else AtomikosSQLException.throwAtomikosSQLException("The transaction has timed out - try increasing the timeout if needed");
so why timeout is set state which not cause exception in AtomikosConnectionProxy ?
UPDATE 5
so I found that property
com.atomikos.icatch.threaded_2pc
will solve my problem and now it starts rollback how I want. But I still dont understand why I should set this to true because now I am testing it on some task which should run in single thread
set com.atomikos.icatch.threaded_2pc=true in jta.properties fixed my problem. Idk why this default value was change to false in web application.
* #param single_threaded_2pc (!com.atomikos.icatch.threaded_2pc)
* If true then commit is done in the same thread as the one that
* started the tx.
XA transactions are horribly complicated and you really want to have a very good reason for using them (ie it's literally impossible to add some business process that removes the need for XA), because you are going to get into trouble out in the wild...
That said, My guess is that it's about timeout discrepancies between XA phases.
With XA there are 2 timeouts - a timeout for the 1st phase, known as the Voting phase (which is typically the one set by the #Transactional annotation, but this depends on the JTA provider) and another timeout for the 2nd phase, known as the commit phase, which is typically a lot longer, because the Transaction Manager has already got the agreement from all parties that the commit is ready to go, and therefore provide greater leeway for things like transient network failures and so on.
My guess is that the WebLogic JTA is simply behaving differently to Atomikos with how it's handling the 2nd phase notifications back from the participants, until atomikos is changed to use the multithreaded ack.
If you application is just you and the database, then you can probably get away without an XA Transaction Manager. I'd expect this would behave the way you want for timeouts.
Good Luck!

#Transactional annotation not working as expected

I need some help trying to debug why the transaction management of my spring boot app is not working.
The basic idea is that I have 2 tables I would like to write something in. When anything goes wrong in one of the 2 tables, the transaction should be rolled back and nothing should be written to the database.
Here is a simplified version of the code:
#Transactional
public void archiveTask(String taskId) {
OffloadedRun run = new OffloadedRun();
run.setStartDateTime(LocalDateTime.now());
calculationRunRepository.save(run);
List<SingleContractCalculationResults> activeResults = contractCalculationResultAccessService.get(taskId);
for (SingleContractCalculationResults result : example) {
for (Map.Entry<String, ContractResults> entry : result.getResultsPerScenario().entrySet()) {
String scenario = entry.getKey();
ContractResults results = entry.getValue();
OffloadedCalculationResult offloadedCalculationResult = new OffloadedCalculationResult();
// offloadedCalculationResult.setOffloadedRun(run);
offloadedCalculationResult.setContractId(result.getContractId());
calculationResultRepository.save(offloadedCalculationResult);
}
}
}
The classes that I execute the save methods on are Spring Data JPA repositories that are defined like this:
public interface CalculationRunRepository extends JpaRepository<OffloadedRun, String> {
}
the line I commented out is a mandatory column. I do this to enforce a ConstraintViolationException to test what happens on an exception when saving something in the second table.
What happens is that the first entity is saved successfully, which should not have happened. I'm trying to figure out why this is.
My spring boot application is configured with #EnableTransactionManagement to enable the #Transactional annotations in my own services (like this one).
I changed the logging level for org.springframework.transaction.interceptor to TRACE to see what's going on:
o.s.t.i.TransactionInterceptor : Getting transaction for [be.sodemo.calculator.offloading.TaskArchiverImpl.archiveTask]
o.s.t.i.TransactionInterceptor : Getting transaction for [org.springframework.data.jpa.repository.support.SimpleJpaRepository.save]
o.s.t.i.TransactionInterceptor : Completing transaction for [org.springframework.data.jpa.repository.support.SimpleJpaRepository.save]
o.s.t.i.TransactionInterceptor : Getting transaction for [org.springframework.data.jpa.repository.support.SimpleJpaRepository.save]
o.s.t.i.TransactionInterceptor : Completing transaction for [org.springframework.data.jpa.repository.support.SimpleJpaRepository.save]
o.s.t.i.TransactionInterceptor : Getting transaction for [org.springframework.data.jpa.repository.support.SimpleJpaRepository.save]
o.s.t.i.TransactionInterceptor : Completing transaction for [org.springframework.data.jpa.repository.support.SimpleJpaRepository.save]
o.s.t.i.TransactionInterceptor : Getting transaction for [org.springframework.data.jpa.repository.support.SimpleJpaRepository.save]
o.s.t.i.TransactionInterceptor : Completing transaction for [org.springframework.data.jpa.repository.support.SimpleJpaRepository.save]
o.s.t.i.TransactionInterceptor : Getting transaction for [org.springframework.data.jpa.repository.support.SimpleJpaRepository.save]
o.s.t.i.TransactionInterceptor : Completing transaction for [org.springframework.data.jpa.repository.support.SimpleJpaRepository.save]
o.s.t.i.TransactionInterceptor : Completing transaction for [be.sodemo.calculator.offloading.TaskArchiverImpl.archiveTask]
o.h.e.j.s.SqlExceptionHelper : SQL Error: 1048, SQLState: 23000
o.h.e.j.s.SqlExceptionHelper : Column 'run_id' cannot be null
o.h.e.j.b.i.AbstractBatchImpl : HHH000010: On release of batch it still contained JDBC statements
o.a.c.c.C.[.[.[.[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.dao.DataIntegrityViolationException: could not execute statement; SQL [n/a]; constraint [null]; nested exception is org.hibernate.exception.ConstraintViolationException: could not execute statement] with root cause
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Column 'run_id' cannot be null
at sun.reflect.GeneratedConstructorAccessor2599.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java) ~[?:1.8.0_102]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_102]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) ~[mysql-connector-java-5.1.40.jar:5.1.40]
After this
I'm not sure what the logging would look like if transaction management works properly but it looks like it's completing every transaction.
Does anyone have an idea what I can try next to see what's going wrong?
Edit: Until now I have been using a MySQL database. I noticed that when I tested this exact same code on a H2 database, it seems like the rollback works as intended. The only difference I see with that is that it throws another vendor-specific exception.
I've tried explicitly setting the rollbackFor attribute on the #Transactional annotation like so:
#Transactional(rollbackFor = {Exception.class, MySQLIntegrityConstraintViolationException.class})
But even that didn't cause a rollback.
Edit:
These are my spring boot settings related to JPA/Hibernate:
spring:
jpa:
hibernate:
ddl-auto: none
dialect: org.hibernate.dialect.MySQL5Dialect
database: mysql
properties:
hibernate:
order_inserts: true
jdbc:
batch_size: 50
datasource:
url: jdbc:mysql://localhost/local-test-db
driver-class-name: com.mysql.jdbc.Driver
I am not sure why you are using String for your id type in repository <OffloadedRun, String>. Your OffloadedRun domain has id with String type? It should match with type of your id field in OffloadedRun domain. Make sure for the case. To confirm,
Could you please post your OffloadedRun domain code also?
You have to use org.springframework.orm.hibernate4.HibernateTransactionManager.
You might be using org.springframework.orm.jpa.JpaTransactionManager.
Please verify whether you are using single dataSource or multiple dataSources in your application.
If you are using single dataSource then #Transactional will pick that single dataSource by default else it will pick any one from multiple dataSources and this kind of untraceable issue occurs.
Please have a look at
#Transaction annotation with different data sources
I have resolved the issue using
/* This code is present in #Configuration class */
#Bean(name = "postgresDataSource")
DataSource postgresDataSource(){
// DataSource configuration code
}
#Bean(name = "postgresTransactionManager")
public PlatformTransactionManager transactionManager(#Qualifier("postgresDataSource") DataSource dataSource) {
return new DataSourceTransactionManager(dataSource);
}
/* This code is present in #Service class */
#Override
#Transactional("postgresTransactionManager")
public void save(Map queueMsg) {
// Transaction specific code
}

Java/Postgres: Deadlock with Flyway clean

I have a scheduled task working (including writing!) on my PostgreSQL database:
#Transactional
#Scheduled(fixedDelay = SCHEDULE_DELAY_IN_SECONDS * MILLISECONDS_PER_SECOND)
public synchronized void doStuff() {
// snip
}
And I do have a service to Flyway.clean() my database:
#Service
public class MyService {
#Transactional
public void deleteDatabase() {
flyway.clean();
flyway.migrate();
// snip
}
}
Now every once in a while when deleting the database a deadlock occurs:
26-Jan-2017 17:15:57.174 SEVERE [https-jsse-nio-8443-exec-4] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [dispatcher] in context with path [/foo] threw exception [Request processing failed; nested exception is org.flywaydb.core.api.FlywayException: Unable to drop "public"."bar"] with root cause
org.postgresql.util.PSQLException: ERROR: deadlock detected
Detail: Process 122 waits for AccessExclusiveLock on relation 69572 of database 16388; blocked by process 97.
Process 97 waits for AccessShareLock on relation 69575 of database 16388; blocked by process 122.
Hint: See server log for query details.
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2284)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2003)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:200)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:424)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:161)
at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:155)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:198)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:198)
at org.flywaydb.core.internal.dbsupport.JdbcTemplate.execute(JdbcTemplate.java:219)
at org.flywaydb.core.internal.dbsupport.postgresql.PostgreSQLTable.doDrop(PostgreSQLTable.java:43)
at org.flywaydb.core.internal.dbsupport.SchemaObject.drop(SchemaObject.java:80)
at org.flywaydb.core.internal.dbsupport.postgresql.PostgreSQLSchema.doClean(PostgreSQLSchema.java:84)
at org.flywaydb.core.internal.dbsupport.Schema.clean(Schema.java:148)
at org.flywaydb.core.internal.command.DbClean$4.doInTransaction(DbClean.java:182)
at org.flywaydb.core.internal.command.DbClean$4.doInTransaction(DbClean.java:180)
at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:72)
at org.flywaydb.core.internal.command.DbClean.cleanSchema(DbClean.java:180)
at org.flywaydb.core.internal.command.DbClean.clean(DbClean.java:130)
at org.flywaydb.core.Flyway$3.execute(Flyway.java:1017)
at org.flywaydb.core.Flyway$3.execute(Flyway.java:1013)
at org.flywaydb.core.Flyway.execute(Flyway.java:1361)
at org.flywaydb.core.Flyway.clean(Flyway.java:1013)
at com.acme.MyService.deleteDatabase(MyService.java:54)
// ...
And I suspect the scheduler to be this other process. Now what? I'd say Flyway - as the more invasive operator, plus the one throwing the error - has to wait for exclusive access, but how do I achieve this in the most efficient way? Especially as both methods are #Transactional already... Will
#Transactional(isolation = SERIALIZABLE)
help? This isolation level seems to be the most restrictive... Or is it just the missing synchronized of deleteDatabase()? This dead lock is kind of hard to reproduce, so any hint is appreciated.

Categories