I have a Spring Boot app that uses Hibernate and Spring Batch. I am using PostgreSQL for my backend database.
My project has 2 different data sources configured: one for Hibernate and one for Spring Batch. They are both in the same database but in different schemas.
My spring batch connection string is the following:
spring.batch.datasource.url=jdbc:postgresql://localhost:5432/mytestapp?currentSchema=springbatch
I have the Datasource configured in the following way:
#Bean("b_ds_prop")
#ConfigurationProperties("spring.batch.datasource")
public DataSourceProperties dataSourceProperties() {
return new DataSourceProperties();
}
#Bean("b_ds")
#DependsOn({"b_ds_prop"})
public DataSource batchFrameworkDatasource(DataSourceProperties dataSourceProperties) {
System.out.println("start print");
// dataSourceProperties.getSchema().stream().forEach(System.out::println);
// System.out.println("datasrcprop is null : " + (dataSourceProperties==null));
// System.out.println("datasrcprop.schema is null : " + (dataSourceProperties.getSchema()==null));
System.out.println("end print");
// dataSourceProperties.setSchema(Arrays.asList("spring_batch"));
return dataSourceProperties.initializeDataSourceBuilder().build();
}
#Bean
#DependsOn({"b_ds"})
public BatchConfigurer defaultBatchConfigurer(#Qualifier("b_ds") DataSource dataSource) {
return new DefaultBatchConfigurer(dataSource);
}
The currentSchema=springbatch setting is not respected, no matter what I do. All my Spring Batch tables keep ending up in public in Postgres.
The schema named springbatch is already present in my database : mytestapp.
I have tried currentSchema=springbatch, public, I have even tried spring.batch.table-prefix=springbatch..
I have tried everything, but still I cannot understand why my batch tables keep ending up in the public schema.
Note I have another connection string in my project like:
spring.hib.datasource.url=jdbc:postgresql://localhost:5432/mytestapp
I have also tried programmatically checking what is the value of schema in DataSourceProperties.
System.out.println("datasrcprop.schema is null : " + (dataSourceProperties.getSchema()==null));
This evaluates to true.
How can I set the schema to be different for Spring Batch ?
By default, Spring boot automatically creates the spring batch metadata tables at starting. It executes the default script which is located in org.springframework.batch.core package. In your situtation, this script is schema-postgresql.sql. This script creates tables in the postgre default schema so public if nothing is specified (and as we don't know which datasource spring boot uses to create spring batch metadata tables, we don't know what is the default schema).
When executing creation script, spring boot doesn't take into account spring.batch.table-prefix property which is only used to say spring batch where metadata tables are.
If you want to control this spring boot feature, you should :
modify the creation script (add the springbatch schema) and give spring boot the path for the modified script (spring.batch.schema property)
OR
create by yourself spring batch metadata tables before starting application and disable spring boot auto creation (spring.batch.initialize-schema=never property)
Good luck
Have you tried this configration:
spring.datasource.url=jdbc:postgresql://localhost:5432/mytestapp
#spring batch properties
spring.batch.job.enabled=false
spring.batch.initializer.enabled=false
spring.batch.initialize-schema=never
spring.batch.table-prefix=springbatch.
Related
I have standalone application. It’s on java, spring-boot, postgres and it has liquibase.
I need to deploy my app and liquibase should create all tables, etc. But it should do it into custom schema not in public. All service tables of liquibase (databasechangelog and databasechangeloglock) should be in custom schema too. How can I create my schema in DB before liquibase start to work? I must do it inside my app when it’s deploying, in config or some like. Without any manual intervention into the DB.
application.properties:
spring.datasource.jndi-name=java:/PostgresDS
spring.jpa.properties.hibernate.default_schema=my_schema
spring.jpa.show-sql = false
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.PostgreSQLDialect
spring.datasource.continue-on-error=true
spring.datasource.sql-script-encoding=UTF-8
liquibase.change-log = classpath:liquibase/changelog-master.yaml
liquibase.default-schema = my_schema
UPD:
When liquibase start, it's create two tables databasechangelogs and one more table. After that, liquibase start working. But I want liquibase in liquibase.default-schema = my_schema, but it's not exist when liquibase start to work and it an error: exception is liquibase.exception.LockException: liquibase.exception.DatabaseException: ERROR: schema "my_schema" does not exist
I want liquibase work in custom schema, not in public:
liquibase.default-schema = my_schema
but before liquibase can do it, the schema must be created. Liquibase can't do this because it not started yet and for start it needs schema.
Vicious circle.
I found a solution with my application.properties.
org.springframework.jdbc.datasource.init.ScriptUtils worked before liquibase.
Logs:
14:11:14,760 INFO [org.springframework.jdbc.datasource.init.ScriptUtils]
(ServerService Thread Pool -- 300) Executing SQL script from URL
[vfs:/content/app.war/WEB-INF/classes/schema.sql]
14:11:14,761 INFO [org.springframework.jdbc.datasource.init.ScriptUtils]
(ServerService Thread Pool -- 300) Executed SQL script from URL
[vfs:/content/app.war/WEB-INF/classes/schema.sql] in 1 ms.
14:11:14,912 ERROR [stderr] (ServerService Thread Pool -- 300) INFO 9/27/18
2:11 PM: liquibase: Successfully acquired change log lock
14:11:15,292 ERROR [stderr] (ServerService Thread Pool -- 300) INFO 9/27/18
2:11 PM: liquibase: Reading from my_schema.databasechangelog
14:11:15,320 ERROR [stderr] (ServerService Thread Pool -- 300) INFO 9/27/18
2:11 PM: liquibase: Successfully released change log lock
I just put schema.sql with CREATE SCHEMA IF NOT EXISTS my_schema; into resources dir and all working properly.
Thanks all for help.
Update: It's work for Spring boot 1.X. If you are use Spring Boot 2, you should enable schema.sql in properties file, with spring.datasource.initialization-mode=always.
More info in Spring Boot - Loading Initial Data
Update 2: In Spring Boot 2.5.2 (maybe in earlier versions too) this solution is not working now, as #peterh has wrote in comment. Sad but true. The last version I was try this solution and it's work was Spring Boot 2.0.9 In docs Spring Boot says that it was redesigned from Spring Boot 2.5.x
Update 3: Some information why they kill this feature -> https://github.com/spring-projects/spring-boot/issues/22741
You can use Spring Boot Pre-Liquibase module for this. It is exactly what it is meant for. It executes some SQL prior to executing Liquibase itself. Pre-Liquibase sets itself up in the Spring Boot AutoConfigure chain so that it is guaranteed to always execute before Liquibase.
Step 1
Add the following Starter to your project:
<dependency>
<groupId>net.lbruun.springboot</groupId>
<artifactId>preliquibase-spring-boot-starter</artifactId>
<version> ---latest-version--- </version>
</dependency>
Step 2
Add a SQL file to src/main/resources/preliquibase with a name of postgresql.sql and content like this:
CREATE SCHEMA IF NOT EXISTS ${spring.liquibase.default-schema};
The ${} syntax denotes a placeholder variable. Pre-Liquibase will resolve it from the properties in your Spring Environment.
Step 3
Set application properties like this:
spring.liquibase.default-schema=${my.db.schemaname}
spring.jpa.properties.hibernate.default_schema=${my.db.schemaname}
Now - in this example - the only thing left to decide is where the my.db.schemaname value comes from. That is your choice. The example project advocates that it should come from an OS environment variable, in particular if your are deploying to a cloud.
Final words
WARNING: Pre-Liquibase is possibly way too flexible in that it allows to execute any SQL code. Don't be tempted to put stuff in Pre-Liquibase files which rightfully belong in an Liquibase ChangeSet. Honestly, the only usage I can think of for Pre-Liquibase is to set up a database "home" (meaning a schema or a catalog) where Liquibase db objects can live so that instances of the same application can be separated by schema or catalog while residing on the same database server.
(Disclosure: I'm the author of Pre-Liquibase module)
To solve this, we need to run a SQL statement that creates the schema during Spring Boot initialization at the point when DataSource bean had been already initialized so DB connections can be easily obtained but before Liquibase runs.
By default, Spring Boot runs Liquibase by creating an InitializingBean named SpringLiquibase. This happens in LiquibaseAutoConfiguration.
Knowing this, we can use AbstractDependsOnBeanFactoryPostProcessor to configure SpringLiquibase to depend on our custom schema creating bean (SchemaInitBean in the example below) which depends on DataSource. This arranges the correct execution order.
My application.properties:
db.schema=my_schema
spring.jpa.hibernate.ddl-auto=validate
spring.jpa.open-in-view=false
spring.jpa.properties.hibernate.jdbc.lob.non_contextual_creation=true
spring.jpa.properties.hibernate.default_schema=${db.schema}
spring.datasource.url=jdbc:postgresql://localhost:5432/postgres
spring.datasource.username=postgres
spring.datasource.password=postgres
spring.liquibase.enabled=true
spring.liquibase.change-log=classpath:db/changelog/db.changelog-master.xml
spring.liquibase.defaultSchema=${db.schema}
Add the #Configuration class below to the project, for example put it in a package processed by component scan.
#Slf4j
#Configuration
#ConditionalOnClass({ SpringLiquibase.class, DatabaseChange.class })
#ConditionalOnProperty(prefix = "spring.liquibase", name = "enabled", matchIfMissing = true)
#AutoConfigureAfter({ DataSourceAutoConfiguration.class, HibernateJpaAutoConfiguration.class })
#Import({SchemaInit.SpringLiquibaseDependsOnPostProcessor.class})
public class SchemaInit {
#Component
#ConditionalOnProperty(prefix = "spring.liquibase", name = "enabled", matchIfMissing = true)
public static class SchemaInitBean implements InitializingBean {
private final DataSource dataSource;
private final String schemaName;
#Autowired
public SchemaInitBean(DataSource dataSource, #Value("${db.schema}") String schemaName) {
this.dataSource = dataSource;
this.schemaName = schemaName;
}
#Override
public void afterPropertiesSet() {
try (Connection conn = dataSource.getConnection();
Statement statement = conn.createStatement()) {
log.info("Going to create DB schema '{}' if not exists.", schemaName);
statement.execute("create schema if not exists " + schemaName);
} catch (SQLException e) {
throw new RuntimeException("Failed to create schema '" + schemaName + "'", e);
}
}
}
#ConditionalOnBean(SchemaInitBean.class)
static class SpringLiquibaseDependsOnPostProcessor extends AbstractDependsOnBeanFactoryPostProcessor {
SpringLiquibaseDependsOnPostProcessor() {
// Configure the 3rd party SpringLiquibase bean to depend on our SchemaInitBean
super(SpringLiquibase.class, SchemaInitBean.class);
}
}
}
This solution does not require external libraries like Spring Boot Pre-Liquibase and not affected by limitations on data.sql / schema.sql support. My main motivation for finding this solution was a requirement I had that schema name must be a configurable property.
Putting everything in one class and using plain JDBC is for brevity.
Simple solution based on Pavel D. answer.
It also can be used without liquibase
#Slf4j
#Component
public class SchemaConfig implements BeanPostProcessor {
#Value("${db.schema}")
private String schemaName;
#Override
public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException {
if (!StringUtils.isEmpty(schemaName) && bean instanceof DataSource) {
DataSource dataSource = (DataSource) bean;
try (Connection conn = dataSource.getConnection();
Statement statement = conn.createStatement()) {
log.info("Going to create DB schema '{}' if not exists.", schemaName);
statement.execute("create schema if not exists " + schemaName);
} catch (SQLException e) {
throw new RuntimeException("Failed to create schema '" + schemaName + "'", e);
}
}
return bean;
}
}
I have ONE spring boot (1.5.4.RELEASE) project using java 8 deployed on AWS HPC. This project architect scope works for Spring Web Application(Website), Rest API Services(Mobile Developer) & Account Administration for Company.
So there is 3 different respective Database like (2-SQL Server & 1-MySQL).
Here on stack-overflow, I'm posting my question for find a best way to implementation this Spring-Boot Project by help of talented stack-overflow users.
Here is my configure properties files.
application.properties
#For Public Website
spring.datasource.clone.web.driver = com.microsoft.sqlserver.jdbc.SQLServerDriver
spring.datasource.clone.web.url = jdbc:sqlserver://127.0.0.01\\dbo:1433;databaseName=PROD_WEB;
# Username and password
spring.datasource.web.username = web
spring.datasource.web.password = ED5FLW64ZU976Q36
#For Rest API
spring.datasource.clone.url = jdbc:mysql://localhost:3306/PROD_REST;
# Username and password
spring.datasource.clone.username = rest
spring.datasource.clone.password = Firewall77#
#For Account Administration for Company Users
spring.datasource.admin.url = jdbc:sqlserver://127.0.0.01\\dbo:1433;databaseName=PROD_ADMIN;
# Username and password
spring.datasource.admin.username = admin
spring.datasource.admin.password = Firewall77#
# Backup & Cron Policy
...
I would greatly appreciate for some very good suggestion to implement it. your knowledge on this subject would help me, Thanks.
You need to implement two different beans, one for each datasource and make them take the corresponding configuration properties respectively:
First bean will be responsible for the first datasource configuration, and should be daclared as primary datasource with #Primary, so it can be setup as the main datasource for the project.
The second bean will configure the second datasource.
This is how they should be implemented in Spring:
#Bean
#Primary
#ConfigurationProperties(prefix="spring.datasource.web")
public DataSource primaryDataSource() {
return DataSourceBuilder.create().build();
}
#Bean
#ConfigurationProperties(prefix="spring.datasource.rest")
public DataSource secondaryDataSource() {
return DataSourceBuilder.create().build();
}
Here's how should be your application.properties configured, to take these two beans configuration into account:
#For Public Website
spring.datasource.web.driver = com.microsoft.sqlserver.jdbc.SQLServerDriver
spring.datasource.web.url = jdbc:sqlserver://127.0.0.01\\dbo:1433;databaseName= PROD_WEB;
# Username and password
spring.datasource.web.username = web
spring.datasource.web.password = ED5FLW64ZU976Q36
#For Rest API
spring.datasource.rest.driver = com.microsoft.sqlserver.jdbc.SQLServerDriver
spring.datasource.rest.url = jdbc:mysql://localhost:3306/PROD_REST;
# Username and password
spring.datasource.rest.username = rest
spring.datasource.rest.password = Firewall77#
Note:
I changed the configuration properties here so they can be differentiated between web and rest datasources:
Properties starting with spring.datasource.web will be dedicated to configure the first datasource.
Properties starting with spring.datasource.rest will be dedicated to configure the second datasource.
try to use DataSource configuration specified at Configuration for each data sources for more help check this Using multiple datasources with Spring Boot and Spring Data
I am working in spring hibernate project where I am connecting to multiple database. These database details are loaded in a properties file and is imported into hibernate.xml file. Each key was mapped as below.
dbName= dbHost
Mapping as below :
<bean id="dbId" parent="abstractDataSource">
<property name="url" value="${dbName}" />
</bean>
But now i need to include a condition such that the property key to be mapped to multiple values like below
E.g. dbName= statusFlag,dbHost,dbBackupHostName
I need to modify the hibernate configuration file such that using the keyName, corresponding values should be retrieved and splitted with comma separated.
Then I need to check for the statusFlag. If it is true, then i should set the dbValue to 'dbHost', else if the flag is false, it should be set to 'dbBackupHostName'.
Please help me how to implement this condition check in hibernate configuration file.
Also please let me know if this approach is fine to proceed or am i making it too difficult to implement?
Waiting for some valueable answers..
~~~Suriya
To use complex logic in bean wiring and creation in spring you can use Java Configuration
It should work for Spring 3. Sample pseudo code is below.
#Configuration
public class MyCustomConfig {
#Value
String statusFlag;
#Value
String dbHost;
#Value
String dbBackupHostName;
#Bean
public BasicDataSource datasource () {
BasicDataSource datasource = new BasicDataSource();
//Do some custom processing to create datasource
...
return datasource;//return after creation
}
}
Use PropertyPlaceholderConfigurer to load properties as #PropertySource is not available in 3.0.5.
If you can change the spring version to 3.1 which shouldn't make much difference then you can use #Profile along with #PropertySource.
If the logic is not too complex and you can separate set of properties that have to be active at a time. (Local db, Dev db, Prod db, Custom etc.) Then you can try using #Profile.
#Profile annotated bean is created only if that profile is active. A profile can be activated by setting spring.profiles.active.
So to activate the Dev profile we can set in the properties files
spring.profiles.active=dev
//activating 2 profiles
spring.profiles.active=dev,mvc
Below is the sample pseudo code.
#Configuration
#PropertySource("bootstrap.properties")//set spring.profiles.active in it
public class MyCustomConfig {
#Profile("profile1")
public BasicDataSource datasource1 () {
//config using ${profile1.dbName} etc
...
}
#Profile("profile2")
public BasicDataSource datasource2 () {
//config using ${profile2.dbName} etc
...
}
#Profile("profile3")
public BasicDataSource datasource3 () {
//config using ${profile3.dbName} etc
...
}
}
Our application allows our customer to have multiple databases, all running on one instance of the database server.
For example, databases might be dbcommon, dbLive, dbStaging, dbUAT, dbDev, dbSandbox. The common database and the Production database always exists, but the others are optional (and there is no limit). In dbcommon there is a table that tells us all the databases....so that's where I would need to start. The tables in common are different from the others, and the others all have the same schema (subscriber data)
Using Hibernate, how can I dynamically create/use a connection to either Live or Staging (or any of the others)? I am using Spring if that helps.
I have come across answers that suggest creating different connections in configuration, but because the number of subscriber databases can vary (per install, not while the app is running), this isn't an option for me.
As I discovered after posting this question, and as the user manish suggested, Hibernate's Multi Tenancy support (using the Database MultiTenancyStrategy) works for me. I had to piece together a solution using various resources (listed below).
http://www.ticnfae.co.uk/blog/2014/07/16/hibernate-multi-tenancy-with-spring/
Setting up a MultiTenantConnectionProvider using Hibernate 4.2 and Spring 3.1.1
Multi-Tenancy with Spring + Hibernate: "SessionFactory configured for multi-tenancy, but no tenant identifier specified"
I'm still looking for a way to be able to reference the common (shared) database at the same time as tenant databases...and will try to add that to this answer when complete.
The simplest way I can see to do this is to manage everything via profiles in Spring.
I accomplished this by using application.yml. I'm also using a Hikari connection pool, but that doesn't effect the configuration too much.
Here is an example of a application.yml with 3 profiles listed, and I've defined two of them as an example.
spring:
profiles:
include: dev,test,production
active: dev
---
spring:
profiles: dev
oms:
omsDataSource:
driverClassName: com.informix.jdbc.IfxDriver
jdbcUrl: jdbc:informix-sqli://devdb:9000/hol:INFORMIXSERVER=m_tcp_1;client_deve=en_US.8859-1;db_deve=en_US.8859-1;LOBCACHE=-1
password: oms
username: oms
connectionTestQuery: select count(*) from systables
maximumPoolSize: 5
---
spring:
profiles: test
oms:
omsDataSource:
driverClassName: com.informix.jdbc.IfxDriver
jdbcUrl: jdbc:informix-sqli://testdb:9000/hol:INFORMIXSERVER=m_tcp_1;client_deve=en_US.8859-1;db_deve=en_US.8859-1;LOBCACHE=-1
password: oms
username: oms
connectionTestQuery: select count(*) from systables
maximumPoolSize: 5
In my DB config class, I set the JPA repositories, and tell it what entityManager to use. I also setup the configuration properties to pull frmo the application.yml . This means it will swap out the details based on the profile the app is using on launch.
#EnableJpaRepositories(entityManagerFactoryRef = "entityManagerFactoryOms",
transactionManagerRef = "transactionManagerOms",
basePackages= "persistence.oms")
#Configuration
#ConfigurationProperties(prefix = "oms.omsDataSource")
public class omsDbConfig extends HikariConfig {
//This will automatically fill in the required fields from the application.yml.
#Bean
public HikariDataSource orcaDataSource() throws SQLException {
return new HikariDataSource(this);
}
//I use that datasource to define my entityMangerFactory
#Bean(name = "entityManagerFactoryOms")
public LocalContainerEntityManagerFactoryBean entityManagerFactoryOrca() throws SQLException {
JpaVendorAdapter adapter = new HibernateJpaVendorAdapter();
Properties props = new Properties();
props.setProperty("hibernate.dialect","org.hibernate.dialect.InformixDialect");
LocalContainerEntityManagerFactoryBean emfb =
new LocalContainerEntityManagerFactoryBean();
emfb.setDataSource(orcaDataSource());
emfb.setPackagesToScan("persistence.oms");
emfb.setJpaProperties(props);
emfb.setJpaVendorAdapter(adapter);
return emfb;
}
}
The entities and repositories are defined normally, there's nothing special there. The DB will switch connection based on whatever profile I tell it to run.
I just switch out the active profile in the application.yml to whichever one I need.
Safety note: Define a production profile, don't have production as a default profile.
I trying to integrate Flyway for migrations in a Spring Boot project with Hibernate and Spring JPA. I'm getting the following Exception:
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flyway' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Invocation of init method failed; nested exception is org.flywaydb.core.api.FlywayException: Found non-empty schema "PUBLIC" without metadata table! Use init() or set initOnMigrate to true to initialize the metadata table.
My pom.xml is looking like this:
<dependency>
<groupId>org.flywaydb</groupId>
<artifactId>flyway-core</artifactId>
<version>3.2</version>
</dependency>
I'm using Hibernate and a config java file for postgres (dev stage) and h2 (local). The signatures are looking like this:
#Bean(initMethod = "migrate")
public Flyway flyway() {
Flyway fly = new Flyway();
fly.clean();
fly.init();
//flyway.setInitOnMigrate(true);
fly.setSchemas("SBA_DIALOG");
//flyway.setLocations("filesystem:src/main/resources/db/migration");
fly.setDataSource(this.dataSource());
fly.migrate();
return fly;
}
#Bean(name = "sbaEntityManagerFactory") #DependsOn("flyway")
public LocalContainerEntityManagerFactoryBean entityManagerFactory() {
...
I can't find anything about my problem described in this question.
Can anybody help?
Spring-Boot is capable on it's own to do this.
Just add flyway as dependency to your project and spring-boot will pick it up.
Flyway migration will start when the service starts up.
If you already have some tables in the database add:
spring.flyway.baselineOnMigrate = true
in your property file to keep flyway calm when it discovers that some tables already exist. ;-)
Flyway should pick up your datasource. If you need for example another user or something like that for flyway, you can set these properties:
spring.flyway.url: jdbc:postgresql://${db.host}/${db.name}
spring.flyway.user: MYUSER
spring.flyway.password: MYPWD
(Of course add your values! You can use SPEL to reference other properties)
Update
One word of caution: If you use a clustered database you may encounter problems that multiple instances that are started at the same time try to perform the updates at the same time. This is a problem when the table locks don't work, which happened to me using a clustered mariaDB.
For any one who want to solve it in java code you can use :
fly.setBaselineOnMigrate(true);
Edit(22-09-2020)
another solution also is :
spring:
flyway:
baselineOnMigrate: true
validateOnMigrate: false