We Keep Coding

Spring Boot how to configure list of Hikari datasources dynamically

If I only have one datasource and I want to configure it with Hikari then in application.yml I can do something like this:
spring:
datasource:
app:
platform: POSTGRESQL
url: jdbc:postgresql://localhost:5432/mydb
username: mydbuser
password: mydbpass
hikari:
maximum-pool-size: 15
leak-detection-threshold: 15000
max-lifetime: 900000
data-source-properties: stringtype=unspecified
pool-name: defaultHikariPool
And in code I do this:
#Bean
#Primary
#ConfigurationProperties("spring.datasource.app")
fun appDataSourceProperties(): DataSourceProperties {
return DataSourceProperties()
}
#Bean
#ConfigurationProperties(prefix = "spring.datasource.app.hikari")
fun appDataSource(): HikariDataSource {
return appDataSourceProperties()
.initializeDataSourceBuilder()
.type(HikariDataSource::class.java)
.build()
}
This works very well. But let's say I have N datasources where size of N can vary depending on deployment. I created a block like this in the applicaiton.yml:
my-custom-parties:
parties:
- my-custom-party-id: 1
datasource:
platform: POSTGRESQL
url: jdbc:postgresql://localhost:5432/my-db-1
username: user1
password: pass1
hikari:
pool-name: db1HikariPool
maximum-pool-size: 3
leak-detection-threshold: 5000
max-lifetime: 900000
data-source-properties: stringtype=unspecified
- my-custom-party-id: 2
datasource:
platform: POSTGRESQL
url: jdbc:postgresql://localhost:5432/my-db-2
username: user2
password: pass2
hikari:
pool-name: db2HikariPool
maximum-pool-size: 10
leak-detection-threshold: 20000
max-lifetime: 900000
data-source-properties: stringtype=unspecified
And I created a configuration class like this:
#Component
#ConfigurationProperties(prefix = "my-custom-parties")
class MyCustomParties {
private var parties: List<MyCustomParty> = listOf()
fun setParties(parties: List<MyCustomParty>) {
this.parties = parties
}
fun getParties() = parties
}
class MyCustomParty {
lateinit var myCustomPartyId: String
private var datasource: DataSourceProperties? = null
fun setDatasource(datasource: DataSourceProperties) {
this.datasource = datasource
}
fun getDatasource() = datasource?.initializeDataSourceBuilder()?.type(HikariDataSource::class.java)?.build()
}
This intializes a hikari datasource, but it doesn't pick up the configuration properties from the file.
I can't hardcode #Bean's for every datasource because the number of datasources is different in different deployments.
So how can I wire this together correctly?

Quartz Spring Boot There is no DataSource named 'dataSource'

I have the following quartz.properties:
org.quartz.jobStore.class=org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.dataSource=dataSource
org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
org.quartz.jobStore.tablePrefix=qrtz_
org.quartz.threadPool.threadCount=1
org.quartz.scheduler.skipUpdateCheck=true
org.quartz.plugin.triggerHistory.class=org.quartz.plugins.history.LoggingTriggerHistoryPlugin
Also, I added QuartzConfiguration:
#Configuration
#EnableScheduling
public class QuartzConfiguration {
public static final String CONTEXT_KEY = "applicationContext";
#Autowired
private DataSource dataSource;
#Bean
public SchedulerFactoryBean schedulerFactoryBean() {
SchedulerFactoryBean scheduler = new SchedulerFactoryBean();
scheduler.setApplicationContextSchedulerContextKey("applicationContext");
scheduler.setConfigLocation(new ClassPathResource("quartz.properties"));
scheduler.setDataSource(dataSource);
scheduler.setWaitForJobsToCompleteOnShutdown(true);
return scheduler;
}
}
In the application.properties I have defined:
#PostgreSQL
spring.datasource.url=${postgresql.datasource.url}
spring.datasource.username=${postgresql.datasource.username}
spring.datasource.password=${postgresql.datasource.password}
Right now, during start up the application fails with the following exception:
Caused by: org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'dataSource': java.sql.SQLException: There is no DataSource named 'dataSource' [See nested exception: java.sql.SQLException: There is no DataSource named 'dataSource']
at org.quartz.impl.jdbcjobstore.JobStoreSupport.getConnection(JobStoreSupport.java:783)
at org.quartz.impl.jdbcjobstore.JobStoreTX.getNonManagedTXConnection(JobStoreTX.java:71)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3861)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverJobs(JobStoreSupport.java:839)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.schedulerStarted(JobStoreSupport.java:695)
... 45 more
Caused by: java.sql.SQLException: There is no DataSource named 'dataSource'
at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:104)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.getConnection(JobStoreSupport.java:780)
... 49 more
What am I doing wrong and how to provide a correct DataSource to the org.quartz.jobStore.dataSource ?

Two options:
create a configuration bean that specifically creates you DataSource.
#Bean
public DataSource getDataSource()
{
DataSourceBuilder dataSourceBuilder = DataSourceBuilder.create();
dataSourceBuilder.driverClassName("org.h2.Driver");
dataSourceBuilder.url("jdbc:h2:file:C:/temp/test");
dataSourceBuilder.username("sa");
dataSourceBuilder.password("");
return dataSourceBuilder.build();
}
You can alternatively use JNDI, use the spring.datasource.jndi-name
see also

Spring boot + Qurartz + Oracle problem with JobStoreTX

I´m creating an application with Spring boot + Quartz + Oracle and i would like to save the scheduling in the database (persistent, in case the server crashes). With RAMJobStore works fine, but when I try to use JobStoreTX it doesn't work, it always uses RAMJobStore, where colud be the problem?. I'm surely making a lot of mistakes, but it's my first application with spring boot + Quartz, can you give me an idea?
The events will be created dynamically, receiving the information in the controller.
application.yaml (In addition to Quartz, the application connects to the database to query tables, but the application and Quartz will use the same database)
hibernate:
globally_quoted_identifiers: true
show_sql: true
logging:
level:
org:
hibernate:
SQL: ${hibernate.logging}
pattern:
console: '%d{yyyy-MM-dd HH:mm:ss} %-5level %logger{36} - %msg%n'
server:
port: ${port}
spring:
activemq:
broker-url: ${activemq-url}
password: ${activemq-password}
user: ${activemq-user}
datasource:
driver-class-name: ${driverClassName}
password: ${ddbb-password}
jdbcUrl: ${ddbb-url}
username: ${ddbb-user}
jpa:
properties:
hibernate:
dialect: org.hibernate.dialect.Oracle12cDialect
quartz:
job-store-type: jdbc
jdbc:
initialize-schema: never
properties:
org:
quartz:
scheduler:
instanceId: AUTO
jobStore:
useProperties: true
isClustered: false
clusterCheckinInterval: 5000
class: org.quartz.impl.jdbcjobstore.JobStoreTX
driverDelegateClass: org.quartz.impl.jdbcjobstore.oracle.OracleDelegate
dataSource: quartzDataSource
dataSource:
quartzDataSource:
driver: oracle.jdbc.driver.OracleDriver
URL: ${ddbb-url}
user: ${ddbb-user}
password: ${ddbb-password}
Class SchedulerConfiguration
#Configuration
public class SchedulerConfiguration {
#Bean
public SchedulerFactoryBean schedulerFactory(ApplicationContext applicationContext) {
SchedulerFactoryBean schedulerFactoryBean = new SchedulerFactoryBean();
schedulerFactoryBean.setJobFactory(new AutoWiringSpringBeanJobFactory());
return schedulerFactoryBean;
}
#Bean
public Scheduler scheduler(ApplicationContext applicationContext) throws SchedulerException {
Scheduler scheduler = schedulerFactory(applicationContext).getScheduler();
scheduler.start();
return scheduler;
}
#Bean
#QuartzDataSource
#ConfigurationProperties(prefix = "spring.datasource")
public DataSource quartzDataSource() {
return DataSourceBuilder.create().build();
}
}
class AutoWiringSpringBeanJobFactor
public class AutoWiringSpringBeanJobFactory extends SpringBeanJobFactory implements
ApplicationContextAware{
private transient AutowireCapableBeanFactory beanFactory;
#Override
public void setApplicationContext(final ApplicationContext context) {
beanFactory = context.getAutowireCapableBeanFactory();
}
#Override
protected Object createJobInstance(final TriggerFiredBundle bundle) throws Exception {
final Object job = super.createJobInstance(bundle);
beanFactory.autowireBean(job);
return job;
}
}
Job class
#Component
public class CampaignJob implements Job{
#Override
public void execute(JobExecutionContext jobExecutionContext) throws JobExecutionException {
System.out.println("Hi, the job works");
}
}
The class where the scheduler it´s created
public class ManagerServiceImpl implements ManagerService {
#Autowired
private ProducerQueue producer;
#Autowired
private ManagementDatabase managementDatabase;
#Autowired
private Scheduler scheduler;
#Override
public String processCampaign(ScheduleCampaign scheduleCampaign) {
try {
ZonedDateTime dateTime = ZonedDateTime.of(scheduleCampaign.getDateTime(), scheduleCampaign.getTimeZone());
JobDetail jobDetail = buildJobDetail(scheduleCampaign);
Trigger trigger = buildJobTrigger(jobDetail, dateTime);
scheduler.scheduleJob(jobDetail, trigger);
} catch (SchedulerException e) {
System.out.println("There was an error creating the scheduler: "+e);
}
return "Scheduler created";
}
private JobDetail buildJobDetail(ScheduleCampaign scheduleCampaign) {
JobDataMap jobDataMap = new JobDataMap();
System.out.println("Function: buildJobDetail - campaign value: "+scheduleCampaign.getCampaign());
jobDataMap.put("campaign", scheduleCampaign.getCampaign());
return JobBuilder.newJob(CampaignJob.class)
.withIdentity(UUID.randomUUID().toString(), "campaign-jobs")
.requestRecovery(true)
.storeDurably(true)
.withDescription("campaign job planned")
.usingJobData(jobDataMap)
.storeDurably()
.build();
}
private Trigger buildJobTrigger(JobDetail jobDetail, ZonedDateTime startAt) {
return TriggerBuilder.newTrigger()
.forJob(jobDetail)
.withIdentity(jobDetail.getKey().getName(), "campaign-triggers")
.withDescription("campaign job Trigger")
.startAt(Date.from(startAt.toInstant()))
.withSchedule(SimpleScheduleBuilder.simpleSchedule().withMisfireHandlingInstructionFireNow())
.build();
}
}
Logs
2020-12-28 16:16:08 INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
2020-12-28 16:16:09 INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Start completed.
2020-12-28 16:16:09 INFO o.h.jpa.internal.util.LogHelper - HHH000204: Processing PersistenceUnitInfo [name: default]
2020-12-28 16:16:10 INFO org.hibernate.Version - HHH000412: Hibernate ORM core version 5.4.23.Final
2020-12-28 16:16:10 INFO org.quartz.impl.StdSchedulerFactory - Using default implementation for ThreadExecutor
2020-12-28 16:16:10 INFO o.quartz.core.SchedulerSignalerImpl - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
2020-12-28 16:16:10 INFO org.quartz.core.QuartzScheduler - Quartz Scheduler v.2.3.2 created.
2020-12-28 16:16:10 INFO org.quartz.simpl.RAMJobStore - RAMJobStore initialized.
2020-12-28 16:16:10 INFO org.quartz.core.QuartzScheduler - Scheduler meta-data: Quartz Scheduler (v2.3.2) 'schedulerFactory' with instanceId 'NON_CLUSTERED'
Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.

Try to autowire your app default dataSource, if you are pointing the same DB for quartz jobs as well.
The below configuration worked for me with PostgreSQL
#Autowired
DataSource dataSource;
#Autowired
JobFactory jobFactory;
#Bean
public JobFactory jobFactory(ApplicationContext applicationContext) {
AutoWiringSpringBeanJobFactory jobFactory = new AutoWiringSpringBeanJobFactory();
jobFactory.setApplicationContext(applicationContext);
return jobFactory;
}
#Bean
public SchedulerFactoryBean schedulerFactoryBean() throws IOException {
SchedulerFactoryBean factory = new SchedulerFactoryBean();
factory.setOverwriteExistingJobs(true);
factory.setAutoStartup(true);
factory.setDataSource(dataSource);
factory.setJobFactory(jobFactory);
factory.setQuartzProperties(quartzProperties());
return factory;
}
#Bean
public Properties quartzProperties() throws IOException {
PropertiesFactoryBean propertiesFactoryBean = new PropertiesFactoryBean();
propertiesFactoryBean.setLocation(new ClassPathResource("/quartz.properties"));
propertiesFactoryBean.afterPropertiesSet();
return propertiesFactoryBean.getObject();
}

How set initial size connection to postgresql in application.yml

Always when i connect to my database, I see 10 idle connection. How can I set this in application.yml.
I use spring boot 1.5.6.RELEASE.
It's not working:
spring:
datasource:
maxActive: 5
maxIdle: 5
minIdle: 5
initialSize: 5
When I created #Bean it's working, but I need solution in application.yml
#Configuration
public class DBConfig {
#Value("${dbconfig.driver-class-name}")
private String driverClassName;
#Value("${dbconfig.url}")
private String url;
#Value("${dbconfig.username}")
private String username;
#Value("${dbconfig.password}")
private String password;
#Bean
public DataSource dataSource() throws SQLException {
BasicDataSource ds = new BasicDataSource();
ds.setDriverClassName(driverClassName);
ds.setUrl(url);
ds.setUsername(username);
ds.setPassword(password);
return ds;
}

I think you're missing to say these are properties. I think the following will work.
spring:
dataSource:
properties:
maxActive: 5
maxIdle: 5
minIdle: 5
initialSize: 5
Note: If you're using tomcat-jdbc, you have to define it explicitly like,
spring:
dataSource:
tomcat:
max-active: 5
max-idle: 5
min-idle: 5
initial-size: 5

How to use HikariCP in Spring Boot with two datasources in conjunction with Flyway

I want to use HikariCP as JDBC connection pool in my Spring boot application. I have two datasources (MySQL database as the primary database and accessing those data through Hibernate and additionally an Oracle database for reading some other data through JDBCTemplate).
I set the MySQL datasource as primary bean:
#Bean
#Primary
#ConfigurationProperties("spring.datasource")
public DataSourceProperties mySQLDataSourceProperties() {
return new DataSourceProperties();
}
#Bean
#Primary
#ConfigurationProperties("spring.datasource")
public DataSource mySQLDataSource() {
return mySQLDataSourceProperties().initializeDataSourceBuilder().build();
}
#Bean
#ConfigurationProperties("oracle.datasource")
public DataSourceProperties oracleDataSourceProperties() {
return new DataSourceProperties();
}
#Bean(name = "oracleDatabase")
#ConfigurationProperties("oracle.datasource")
public DataSource oracleDataSource() {
return oracleDataSourceProperties().initializeDataSourceBuilder().build();
}
#Bean
public JdbcTemplate oracleJdbcTemplate(#Qualifier("oracleDatabase") DataSource oracleDb) {
return new JdbcTemplate(oracleDb);
}
and I put the following configurations in my application.properties :
spring.datasource.type=com.zaxxer.hikari.HikariDataSource
spring.datasource.hikari.minimum-idle=7
spring.datasource.hikari.pool-name=Test-1
spring.datasource.hikari.data-source-properties.prepStmtCacheSize=250
spring.datasource.hikari.data-source-properties.prepStmtCacheSqlLimit=2048
spring.datasource.hikari.data-source-properties.cachePrepStmts=true
spring.datasource.hikari.data-source-properties.useServerPrepStmts=true
Unforuntately, these HikariCP configurations are not being read :
HikariConfig - dataSourceJNDI..................none
HikariConfig - dataSourceProperties............{password=<masked>}
HikariConfig - driverClassName................."com.mysql.jdbc.Driver"
HikariConfig - healthCheckProperties...........{}
HikariConfig - healthCheckRegistry.............none
HikariConfig - idleTimeout.....................600000
HikariConfig - initializationFailFast..........true
HikariConfig - initializationFailTimeout.......1
HikariConfig - isolateInternalQueries..........false
HikariConfig - jdbc4ConnectionTest.............false
HikariConfig - jdbcUrl........................."jdbc:mysql://localhost:3306/testDB"
HikariConfig - leakDetectionThreshold..........0
HikariConfig - maxLifetime.....................1800000
HikariConfig - maximumPoolSize.................10
HikariConfig - metricRegistry..................none
HikariConfig - metricsTrackerFactory...........none
HikariConfig - minimumIdle.....................10
HikariConfig - password........................<masked>
HikariConfig - poolName........................"HikariPool-1"
Creating the HikariCP beans and deactivating the DataSource autoconfiguration and removing "spring.datasource" :
#EnableAutoConfiguration(exclude = {DataSourceAutoConfiguration.class})
#SpringBootApplication
#ComponentScan
public class SpringApplication {
#Bean
#Primary
#ConfigurationProperties(prefix = "spring.datasource.hikari")
public HikariConfig hikariConfig() {
return new HikariConfig();
}
#Bean
public DataSource dataSource() {
return new HikariDataSource(hikariConfig());
}
solves my problem :
HikariConfig - dataSourceJNDI..................none
HikariConfig - dataSourceProperties............{password=<masked>, prepStmtCacheSqlLimit=2048, cachePrepStmts=true, useServerPrepStmts=true, prepStmtCacheSize=250}
HikariConfig - driverClassName................."com.mysql.jdbc.Driver"
HikariConfig - healthCheckProperties...........{}
HikariConfig - healthCheckRegistry.............none
HikariConfig - idleTimeout.....................600000
HikariConfig - initializationFailFast..........true
HikariConfig - initializationFailTimeout.......1
HikariConfig - isolateInternalQueries..........false
HikariConfig - jdbc4ConnectionTest.............false
HikariConfig - jdbcUrl........................."jdbc:mysql://localhost:3306/testDB?autoReconnect=true"
HikariConfig - leakDetectionThreshold..........0
HikariConfig - maxLifetime.....................1800000
HikariConfig - poolName........................"Test-1"
But then the Flyway showing some weird warnings which were not shown before and I have to create the database Schema manually before running the Spring application, that is : the create schema does not work anymore.
[WARN ] JdbcTemplate - DB: Can't create database 'test'; database exists (SQL State: HY000 - Error Code: 1007)
[WARN ] JdbcTemplate - DB: Unknown table 'testSchema.tenant' (SQL State: 42S02 - Error Code: 1051)
[WARN ] JdbcTemplate - DB: Unknown table 'testSchema.user' (SQL State: 42S02 - Error Code: 1051)
My Flyway SQL scripts are plain DDL scripts :
CREATE SCHEMA IF NOT EXISTS `testSchema` DEFAULT CHARACTER SET utf8 ;
DROP TABLE IF EXISTS `testSchema`.`tenant`;
CREATE TABLE `testSchema`.`tenant` (
`id` int NOT NULL AUTO_INCREMENT,
I think that disabling the Auto-Datasource configuration is not the best solution since Flyway stops creating the schema and showing warnings. Is there any other way to solve this ?

Declaring your own DataSource will already have implicity disabled Spring Boot's auto-configuration of a data source. In other words this won't be having any effect:
#EnableAutoConfiguration(exclude = {DataSourceAutoConfiguration.class})
I think the problem lies in the fact that you aren't binding Hikari-specific configuration to your MySQL DataSource. You need to do something like this:
#Bean
#Primary
#ConfigurationProperties("spring.datasource.hikari")
public DataSource mySQLDataSource() {
return mySQLDataSourceProperties().initializeDataSourceBuilder().build();
}
This will mean that your mySQLDataSourceProperties are configured with general-purpose data source configuration. They then create a HikariDataSource which is further configured with the Hikari-specific configuration.

Thank you Andy for your fast and valuable answer ! You set me on the right track. After fiddling around, I found this configuration is working for me :
#Bean
#Primary
#ConfigurationProperties("spring.datasource")
//#ConfigurationProperties("spring.datasource.hikari") can also be used, no difference
public DataSourceProperties mySQLDataSourceProperties() {
return new DataSourceProperties();
}
#Bean
#Primary
#ConfigurationProperties("spring.datasource.hikari")
public DataSource mySQLDataSource() {
return mySQLDataSourceProperties().initializeDataSourceBuilder().build();
}
#Bean
#ConfigurationProperties(prefix = "spring.datasource.hikari")
public HikariConfig hikariConfig() {
return new HikariConfig();
}
#Bean
public DataSource dataSource() {
return new HikariDataSource(hikariConfig());
}
and I had to add these settings in the application.properties:
# this is absolutely mandatory otherwise BeanInstantiationException in mySQLDataSource !
spring.datasource.url=${JDBC_CONNECTION_STRING}
spring.datasource.hikari.jdbc-url=${JDBC_CONNECTION_STRING}
spring.datasource.hikari.username=user
spring.datasource.hikari.password=pass

I used the following approach
first.datasource.jdbc-url=jdbc-url
first.datasource.username=username
first.datasource.password=password
.
.
.
.
=================== In Java Configuration File ==================
#Primary
#Bean(name = "firstDataSource")
#ConfigurationProperties(prefix = "first.datasource")
public DataSource dataSource() {
return DataSourceBuilder.create().build();
}
#Primary
#Bean(name = "firstEntityManagerFactory")
public LocalContainerEntityManagerFactoryBean barEntityManagerFactory(EntityManagerFactoryBuilder builder,
#Qualifier("firstDataSource") DataSource dataSource) {
Map<String, String> props = new HashMap<String, String>();
props.put("spring.jpa.database-platform", "org.hibernate.dialect.Oracle12cDialect");
.
.
.
return builder.dataSource(dataSource).packages("com.first.entity").persistenceUnit("firstDB")
.properties(props)
.build();
}
#Primary
#Bean(name = "firstTransactionManager")
public PlatformTransactionManager firstTransactionManager(
#Qualifier("firstEntityManagerFactory") EntityManagerFactory firstEntityManagerFactory) {
return new JpaTransactionManager(firstEntityManagerFactory);
}
second.datasource.jdbc-url=jdbc-url
second.datasource.username=username
second.datasource.password=password
.
.
.
.
=================== In Java Configuration File ==================
#Bean(name = "secondDataSource")
#ConfigurationProperties(prefix = "second.datasource")
public DataSource dataSource() {
return DataSourceBuilder.create().build();
}
#Bean(name = "secondEntityManagerFactory")
public LocalContainerEntityManagerFactoryBean barEntityManagerFactory(EntityManagerFactoryBuilder builder,
#Qualifier("secondDataSource") DataSource dataSource) {
Map<String, String> props = new HashMap<String, String>();
props.put("spring.jpa.database-platform", "org.hibernate.dialect.Oracle12cDialect");
.
.
.
return builder.dataSource(dataSource).packages("com.second.entity").persistenceUnit("secondDB")
.properties(props)
.build();
}
#Bean(name = "secondTransactionManager")
public PlatformTransactionManager secondTransactionManager(
#Qualifier("secondEntityManagerFactory") EntityManagerFactory secondEntityManagerFactory) {
return new JpaTransactionManager(secondEntityManagerFactory);
}