We Keep Coding

Define an arbitrary number of datasources in external Spring Boot config

I have a Spring Boot web service that serves different types of data, where each type resides in its own database. As new types are added, I don't want to have to configure the datasource for each type in the code. This is what I've got so far:
Data types and their database connections are defined in application.yml:
datatypes:
someType:
url: jdbc:postgresql://mydatabase.com:5432/some_type
username: user1
password: pass1
otherType:
url: jdbc:postgresql://mydatabase.com:5432/other_type
username: user2
password: pass2
A configuration class reads the properties and creates DataSources and JdbcTemplates:
#Configuration
#EnableConfigurationProperties
public class JdbcConfig {
#Bean
#ConfigurationProperties(prefix = "datatypes")
public Map<String, DataSourceProperties> databaseConfig() {
return new HashMap<>();
}
#Bean
public Map<String, NamedParameterJdbcTemplate> jdbcTemplateMap() {
Map<String, DataSourceProperties> databaseConfig = databaseConfig();
Map<String, DataSource> dataSourceMap = databaseConfig().entrySet().stream()
.collect(Collectors.toMap(Map.Entry::getKey, entry ->
entry.getValue().initializeDataSourceBuilder().build()));
return dataSourceMap.entrySet().stream()
.collect(Collectors.toMap(Map.Entry::getKey, entry ->
new NamedParameterJdbcTemplate(entry.getValue())));
}
}
And finally a repository that fetches data based on its type:
#Repository
public class MyRepository {
#Autowired
private Map<String, NamedParameterJdbcTemplate> jdbcTemplateMap;
public Item getItem(String dataType, String id) {
String sql = "select * from item where id = :id";
return jdbcTemplateMap.get(dataType)
.queryForObject(sql, Map.of("id", id), new ItemRowMapper());
}
}
When Spring tries to autowire jdbcTemplateMap in the repository it can't find any jdbc template beans, so the auto configuration kicks in but fails as the properties for the datasource are not where it expects them in the yml. This can be fixed by disabling the auto configuration: #SpringBootApplication(exclude = DataSourceAutoConfiguration.class)
This setup almost works. However, since the DataSource instances are not registered in the application context, I miss out on some other auto configuration magic, like the actuator health check for instance. I tried registering them myself, by adding this to JdbcConfig and calling registerDataSource() from the jdbcTemplateMap() bean method:
#Autowired
private ApplicationContext applicationContext;
private void registerDataSource(String beanName, DataSource dataSource) {
ConfigurableApplicationContext context =
(ConfigurableApplicationContext) applicationContext;
ConfigurableListableBeanFactory beanFactory = context.getBeanFactory();
beanFactory.registerSingleton(beanName, dataSource);
}
With this in place I should be able to enable the datasouce auto configuration again, but it runs before jdbcTemplateMap() has a chance to run. Actuator won't pick them up either. Can this be fixed?

Spring boot jdbc data repository by multi data source

please help me.
I use multi data source in my project
data source properties:
spring.datasource.url=jdbc:sqlserver://localhost:1433;databaseName=db
spring.datasource.username=xxxxx
spring.datasource.password=xxxxx
spring.datasource.driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver
spring.datasource2.url=jdbc:mysql://localhost:3306/db2
spring.datasource2.username=xxxx
spring.datasource2.password=xxx
spring.datasource2.driver-class-name=com.mysql.cj.jdbc.Driver
config class:
#Configuration
#EnableJdbcRepositories(
jdbcOperationsRef = "mysqlNamedParameterJdbcOperations",
basePackages = "com.example.demo.mysqlModels"
)
public class Config extends AbstractJdbcConfiguration {
#Bean("mysqlDataSource")
#ConfigurationProperties(prefix = "spring.datasource2")
public DataSource mysqlDataSource() {
return DataSourceBuilder.create()
.build();
}
#Bean(name = "mysqlNamedParameterJdbcOperations")
NamedParameterJdbcOperations mysqlNamedParameterJdbcOperations(#Qualifier("mysqlDataSource") DataSource mysqlDataSource) {
return new NamedParameterJdbcTemplate(mysqlDataSource);
}}
#Configuration
#EnableJdbcRepositories(
jdbcOperationsRef = "mssqlNamedParameterJdbcOperations",
basePackages = "com.example.demo.mssqlModels"
)
public class Config2 extends AbstractJdbcConfiguration {
#Bean("mssqlDataSource")
#ConfigurationProperties(prefix = "spring.datasource")
public DataSource mssqlDataSource() {
return DataSourceBuilder.create()
.build();
}
#Bean(name = "mssqlNamedParameterJdbcOperations")
NamedParameterJdbcOperations mssqlNamedParameterJdbcOperations(#Qualifier("mssqlDataSource") DataSource mssqlDataSource) {
return new NamedParameterJdbcTemplate(mssqlDataSource);
}}
repository in com.example.demo.mssqlModels:
public interface MssqlRepository extends PagingAndSortingRepository<MyEntity, Integer> {}
repository in com.example.demo.mysqlModels:
public interface MysqlRepository extends PagingAndSortingRepository<MyEntity, Integer> {}
my service:
#Slf4j
#Service
public class MyService {
#Autowired
private final MssqlRepository mssqlRepository;
#Autowired
private final MysqlRepository mysqlRepository;
#PostConstruct
public void init() {
log.info("mssql result {}", mssqlRepository.findAll());
log.info("mysql result {}", mysqlRepository.findAll());
}}
but result is same and both repositories read data from mysql datasource
thanks

You might be interested in looking at my question I raised recently here regarding 2 data sources but each applied to a different repository.
In your configuration classes you should also create two TransactionManagers with unique names.
In each repository annotate it with #Transactional(transactionManager = 'transaction manager name') replacing the transaction name with the appropriate name
You'll probably need to override the default methods such as saveAll() with the same annotation as in (2).
However as per my question I found that an incorrect data source is sometimes used (I've since found that having the Postgres classes as the primary resolved my problem but I don't know why this worked)

Spring Data Cassandra config change on the fly

I am trying to do a similar thing with my application. I am using following versions of Spring boot and Cassandra:
spring-data-cassandra - 2.0.8.RELEASE
spring-boot-starter-parent - 2.0.4.RELEASE
I need to change some properties(mostly hostnames) of Cassandra on the fly and want it to make a new connection with the application. For config change we have internal Cloud Config Change Management and it runs fine on changes and listens to it.
This is my class :
#Configuration
#Order(Ordered.HIGHEST_PRECEDENCE)
#RefreshScope
#EnableCassandraRepositories(basePackages = {"com.*.*.*.dao.repo"})
public class AppConfig {
private static final Logger LOGGER = LoggerFactory.getLogger(AppConfig.class);
#Value("${application['cassandraPort']}")
private String cassandraPort;
#Value("${application['cassandraEndpoint']}")
private String cassandraEndpoint;
#Value("${application['keyspaceName']}")
private String keyspaceName;
#Value("${application['cassandraConsistency']}")
private String cassandraConsistency;
#Value("${application['cassandraUserName']}")
private String cassandraUserName;
#Autowired
private AppConfig appConfig;
public AppConfig() {
System.out.println("AppConfig Constructor");
}
public String getCassandraPort() {
return cassandraPort;
}
public void setCassandraPort(String cassandraPort) {
this.cassandraPort = cassandraPort;
}
public String getCassandraEndpoint() {
return cassandraEndpoint;
}
public void setCassandraEndpoint(String cassandraEndpoint) {
this.cassandraEndpoint = cassandraEndpoint;
}
public String getKeyspaceName() {
return keyspaceName;
}
public void setKeyspaceName(String keyspaceName) {
this.keyspaceName = keyspaceName;
}
public String getCassandraConsistency() {
return cassandraConsistency;
}
public void setCassandraConsistency(String cassandraConsistency) {
this.cassandraConsistency = cassandraConsistency;
}
public String getCassandraUserName() {
return cassandraUserName;
}
public void setCassandraUserName(String cassandraUserName) {
this.cassandraUserName = cassandraUserName;
}
#Bean
// #RefreshScope
public CassandraConverter converter() {
return new MappingCassandraConverter(this.mappingContext());
}
#Bean
// #RefreshScope
public CassandraMappingContext mappingContext() {
return new CassandraMappingContext();
}
#Bean
//#RefreshScope
public CassandraSessionFactoryBean session() {
CassandraSessionFactoryBean session = new CassandraSessionFactoryBean();
session.setCluster(this.cluster().getObject());
session.setKeyspaceName(appConfig.getKeyspaceName());
session.setConverter(this.converter());
session.setSchemaAction(SchemaAction.NONE);
return session;
}
#Bean
//#RefreshScope
public CassandraClusterFactoryBean cluster() {
CassandraClusterFactoryBean cluster = new CassandraClusterFactoryBean();
cluster.setContactPoints(appConfig.getCassandraEndpoint());
cluster.setPort(Integer.valueOf(appConfig.getCassandraPort()));
cluster.setUsername(appConfig.getCassandraUserName());
cluster.setPassword("password");
cluster.setQueryOptions(new QueryOptions().setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM));
return cluster;
}
}
However, when I try to use #RefreshScope with that Configuration class, the application fails to start. This is what it shows in console :
***************************
APPLICATION FAILED TO START
***************************
Description:
Parameter 2 of constructor in org.springframework.boot.autoconfigure.data.cassandra.CassandraDataAutoConfiguration required a bean of type 'com.datastax.driver.core.Cluster' that could not be found.
- Bean method 'cassandraCluster' not loaded because auto-configuration 'CassandraAutoConfiguration' was excluded
Action:
Consider revisiting the entries above or defining a bean of type 'com.datastax.driver.core.Cluster' in your configuration.
Is there some guidelines on using #RefreshScope with Cassandra Bean? If anyone has done that earlier can you share the same?

You're mixing a couple of things here.
The config carries properties and bean definitions.
#RefreshScope on AppConfig causes some interference with Spring Boot's auto-configuration and the declared beans aren't used (that's why you see Parameter 2 of constructor…).
To clean up, we will reuse what Spring Boot provides as much as possible, and only declare what's really needed.
Follow these steps to solve the issue (based on your code above):
Create a #ConfigurationProperties bean that encapsulates your properties, or better, reuse CassandraProperties.
Re-enable CassandraAutoConfiguration and remove your own MappingContext and CassandraConverter beans, keep only Cluster and Session bean definitions
Declare Cluster and Session beans as needed and make them use #RefreshScope. Your #Configuration class should look like:
Example Configuration:
#Configuration
public class MyConfig {
#Bean(destroyMethod = "close")
#RefreshScope
public Cluster cassandraCluster(CassandraProperties properties) {
Cluster.Builder builder = Cluster.builder().addContactPoints(properties.getContactPoints().toArray(new String[0]))
.withoutJMXReporting();
return builder.build();
}
#Bean(destroyMethod = "close")
#RefreshScope
public Session cassandraSession(CassandraProperties properties, Cluster cluster) {
return cluster.connect(properties.getKeyspaceName());
}
}

How to make the junit tests use the embedded mongoDB in a springboot application?

I am learning springboot and have created a simple springboot application. I want it to use the embedded mongoDB when it runs the unit tests and the external mongoDB for the rest of the application. However it uses the external mongoDB for the unit tests instead of the embedded one.
I have following two dependencies in my POM.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
<dependency>
<groupId>de.flapdoodle.embed</groupId>
<artifactId>de.flapdoodle.embed.mongo</artifactId>
<scope>test</scope>
</dependency>
my properties file has the following:
# MongoDB properties
mongo.db.name=person_testDB
mongo.db.url=localhost
#external Mongo url
spring.data.mongodb.uri=mongodb://localhost:27017/personDB
I have a config file(MongoDBConfig.java) which includes embedded MongoDB configurations:
#EnableMongoRepositories
public class MongoDBConfig {
#Value("${mongo.db.url}")
private String MONGO_DB_URL;
#Value("${mongo.db.name}")
private String MONGO_DB_NAME;
#Bean
public MongoTemplate mongoTemplate() {
MongoClient mongoClient = new MongoClient(MONGO_DB_URL);
MongoTemplate mongoTemplate = new MongoTemplate(mongoClient, MONGO_DB_NAME);
return mongoTemplate;
}
}
Following is my PersonService.java class:
#Service
public class PersonService {
private static final Logger logger = LoggerFactory.getLogger(PersonService.class);
#Autowired
MongoTemplate mongoTemplate;
public void savePerson(Person person) {
String name = person.getName();
String collectionName = name.substring(0, 1);
mongoTemplate.save(person, collectionName);
}
}
My unit test for the PersonsService class is as follows:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(classes = {MongoDBConfig.class})
#SpringBootTest(classes = PersonService.class)
#DataMongoTest
public class PersonServiceTest {
#Autowired
PersonService personService;
MongodForTestsFactory factory;
MongoClient mongo;
#Before
public void setup() throws Exception {
factory = MongodForTestsFactory.with(Version.Main.PRODUCTION);
mongo = factory.newMongo();
}
#After
public void teardown() throws Exception {
if (factory != null)
factory.shutdown();
}
#Test
public void testSave(){
Person person = new Person("Bob Smith " , 25);
personService.savePerson(person);
}
}
It creates the collection name and the document name correctly in the external MongoDB which is not what I want. How can I restrict the unitTests to an embedded Mongo?

An alternative would be to run entire spring boot application in test. In this case your spring boot application will be discovered automatically and embedded mongoDB will be downloaded and started by Spring Boot
#RunWith(SpringRunner.class)
#SpringBootTest
public class YourSpringBootApplicationTests {
08:12:14.676 INFO EmbeddedMongo:42 - note: noprealloc may hurt
performance in many applications 08:12:14.694 INFO EmbeddedMongo:42 -
2017-12-31T08:12:14.693+0200 I CONTROL [initandlisten] MongoDB
starting : pid=2246 port=52299 08:12:22.005 INFO connection:71 -
Opened connection [connectionId{localValue:2, serverValue:2}] to
localhost:52299
In case of your example you might modify the code in order to start embedded Mongo on different port:
add test/resoures/test.properties file in order to override properties from application.properties
mongo.db.name=person_testDB
mongo.db.url=localhost
mongo.db.port=12345
modify MongoDBConfig: add MONGO_DB_PORT field
#EnableMongoRepositories
public class MongoDBConfig {
#Value("${mongo.db.url}")
private String MONGO_DB_URL;
#Value(("${mongo.db.port:27017}"))
private int MONGO_DB_PORT;
#Value("${mongo.db.name}")
private String MONGO_DB_NAME;
#Bean
public MongoTemplate mongoTemplate() {
MongoClient mongoClient = new MongoClient(MONGO_DB_URL, MONGO_DB_PORT);
MongoTemplate mongoTemplate = new MongoTemplate(mongoClient, MONGO_DB_NAME);
return mongoTemplate;
}
}
modify test class:
remove #DataMongoTest annotation. This annotation forces starting embedded mongoDB instance
static MongodExecutable mongodExecutable;
#BeforeClass
public static void setup() throws Exception {
MongodStarter starter = MongodStarter.getDefaultInstance();
String bindIp = "localhost";
int port = 12345;
IMongodConfig mongodConfig = new MongodConfigBuilder()
.version(Version.Main.PRODUCTION)
.net(new Net(bindIp, port, Network.localhostIsIPv6()))
.build();
mongodExecutable = null;
try {
mongodExecutable = starter.prepare(mongodConfig);
mongodExecutable.start();
} catch (Exception e){
// log exception here
if (mongodExecutable != null)
mongodExecutable.stop();
}
}
#AfterClass
public static void teardown() throws Exception {
if (mongodExecutable != null)
mongodExecutable.stop();
}
One more way is to use MongoRepository and init embedded Mongo as part of test #Configuration class: it's outlined here: How do you configure Embedded MongDB for integration testing in a Spring Boot application?

Why isn't my spring boot (mongo) bean being created / used?

I'm trying to use SpringBoot to talk to a Mongo database.
It is working using spring-boot-starter-data-mongodb and auto configuring a default bean which does allow my MongoRepository classes to talk to the DB ok.
However, I want to override the defaults. I could use application.properties but I need to be able to pass the connection parameters as options on the command line as the application starts up.
I've tried changing the port to break it, I've added debug to the Mongo config and it seems whatever I do the default spring config is being used regardless. It's as if the #Configuration annotation is ignored.
I've tried various flavours of configuring the main application class (specifying conf location, adding #Configuration to main class, with and without #SpringBootApplication ...), but here is where I am at the moment....
package somepackage
#EnableAutoConfiguration
#ComponentScan
public class MyApplication {
public static void main(String[] args) {
ApplicationContext ctx = SpringApplication.run(MyApplication.class, args);
....
}
package somepackage.conf; // should be picked up by ComponentScan, no?
#Configuration
public class MongoConf {
#Bean
public MongoClientFactoryBean mongo() throws Exception {
MongoClientFactoryBean mongo = new MongoClientFactoryBean();
/*
setting to silly values to try to prove it is trying to create connections using this bean - expected to see errors because can't create connection... */
mongo.setHost("flibble");
mongo.setPort(345);
return mongo;
}
}

You should actually use built in Spring Boot MongoDb Starter features and related auto configuration through application properties. Custom host, port, passwords etc. can and should be set via dedicated Spring Boot MongoDB Properties:
spring.data.mongodb.authentication-database= # Authentication database name.
spring.data.mongodb.database=test # Database name.
spring.data.mongodb.field-naming-strategy= # Fully qualified name of the FieldNamingStrategy to use.
spring.data.mongodb.grid-fs-database= # GridFS database name.
spring.data.mongodb.host=localhost # Mongo server host.
spring.data.mongodb.password= # Login password of the mongo server.
spring.data.mongodb.port=27017 # Mongo server port.
spring.data.mongodb.repositories.enabled=true # Enable Mongo repositories.
spring.data.mongodb.uri=mongodb://localhost/test # Mongo database URI. When set, host and port are ignored.
spring.data.mongodb.username= # Login user of the mongo server.
And link to the full list of supported properties is here.

In addition to RafalG's suggestion about MongoProperties, I combined that with the ApplicationArguments class and now I'm getting somewhere....
#Bean
#Primary
public MongoProperties mongoProperties(ApplicationArguments args) {
MongoProperties props = new MongoProperties();
String[] mongoHostAndPort = args.getSourceArgs()[3].split(":");
props.setHost(mongoHostAndPort[0]);
props.setPort(Integer.parseInt(mongoHostAndPort[1]));
return props;
}
#Bean
public MongoClientFactoryBean mongo() {
return new MongoClientFactoryBean();
}
Of course there's lots of error handling to add (nulls, non-ints etc) but hopefully if may help someone else.

#Configuration
#EnableAutoConfiguration(exclude = { EmbeddedMongoAutoConfiguration.class })
#Profile("!testing")
public class TestMongoConfig extends AbstractMongoConfiguration {
private static final MongodStarter starter = MongodStarter.getDefaultInstance();
private MongodExecutable _mongodExe;
private MongodProcess _mongod;
private MongoClient _mongo;
#Value("${spring.data.mongodb.host}")
private String host;
#Value("${spring.data.mongodb.port}")
private Integer port;
#Override
protected String getDatabaseName() {
return "test";
}
#Bean
public Mongo mongo() throws Exception {
_mongodExe = starter.prepare(new MongodConfigBuilder()
.version(Version.Main.PRODUCTION)
.net(new Net(port, Network.localhostIsIPv6()))
.build());
_mongod = _mongodExe.start();
return new MongoClient(host, port);
}
#Override
public String getMappingBasePackage() {
return "com.test.domain";
}