We Keep Coding

Configuring a Spring Cloud Data Flow Task with its own Database

I have a task application with its own database that I'd like to run in Spring Cloud Data Flow.
My problem is that SCDF overwrites the datasource configuration in the task with the datasource configuration for SCDF. (Both databases are Oracle DBs.)
My task should write to a different database (but I also want to know its status in SCDF database).
How is it possible to configure my task to connect to its own database as well as to SCDF's database?

I found the solution.
I defined both data sources in a configuration class (one for JPA and one for SCDF) following this as an example: https://www.baeldung.com/spring-data-jpa-multiple-databases
However this wasn't enough because the Data Flow Server accepts only one data source by default. To overcome this, one needs to extend the DefaultTaskConfigurer and set the Data Flow Server's data source in the constructor.
#Component
public class GeneratorTaskConfigurer extends DefaultTaskConfigurer {
public GeneratorTaskConfigurer(#Qualifier("dataflowDataSource") DataSource dataSource) {
super(dataSource);
}
}

You can have one config class with SCDF datasource code like this
#Configuration
#Profile("cloud")
public class MySqlConfiguration {
#Bean
public Cloud cloud() {
return new CloudFactory().getCloud();
}
#Bean
#Primary
public DataSource dataSource() {
return cloud().getSingletonServiceConnector(DataSource.class, null);
}
#Bean
#Primary
public PlatformTransactionManager getTransactionManager() {
return new DataSourceTransactionManager(dataSource());
}
#Bean
public JobRepository jobRepositoryFactoryBean() throws Exception{
JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean();
factory.setDataSource(dataSource());
factory.setTransactionManager(getTransactionManager());
factory.afterPropertiesSet();
return factory.getObject();
}
#Bean
#Primary
public DefaultTaskConfigurer defaultTaskConfigurer() {
return new DefaultTaskConfigurer(dataSource());
}
}
And then have your other datasource configuration in a separate class for the database you want to write to.
Make sure you mark the SCDF one #Primary, otherwise you get multiple datasource error.
Hope this helps.

Spring-Boot: How to restrict the visibility of Beans

I have two custom PlatformTransactionManager beans injected into the Spring framework with specific names as follows:
#Bean(name = "ubldbTransactionManager")
protected PlatformTransactionManager transactionManager(
#Qualifier("ubldbEntityManagerFactory") EntityManagerFactory entityManagerFactory) {
return new JpaTransactionManager(entityManagerFactory);
}
#Bean(name = "bpdbTransactionManager")
public PlatformTransactionManager bpdbTransactionManager(
#Qualifier("bpdbEntityManagerFactory") EntityManagerFactory entityManagerFactory) {
return new JpaTransactionManager(entityManagerFactory);
}
A 3rd-party library has a #Autowired protected PlatformTransactionManager transactionManager; dependency. So, the 3rd party library is not supposed to use none of the two TransactionManagers. However, as you see there is no Qualifier for the dependency injection in the external library and I get an error as follows:
Field transactionManager in org.camunda.bpm.spring.boot.starter.configuration.impl.DefaultDatasourceConfiguration required a single bean, but 2 were found:
- bpdbTransactionManager: defined by method 'bpdbTransactionManager' in class path resource [eu/nimble/service/bp/config/BusinessProcessDBConfig.class]
- ubldbTransactionManager: defined by method 'transactionManager' in class path resource [eu/nimble/service/bp/config/UBLDBConfig.class]
So, how can I restrict the visibility of the two Beans so that they would not be accessible by the 3rd-party library?

DefaultDatasourceConfiguration is provided to use default Spring beans e.g. DataSource named dataSource and PlatformTransactionManager named transcationManager. It's there to glue Camunda into a Spring Boot application which be default has a single data source.
Since you have created your own PlatformTransactionManager beans this disabled Spring Boot's default transaction manager bean named transcationManager (as per TransactionAutoConfiguration Spring Boot auto-configuration logic).
You most likely need to define one more transactionManager (and potentially dataSource) for Camunda's process engine, which requires it's own schema. Make sure to use the right bean name as below:
#Bean
public PlatformTransactionManager transactionManager() {
...
}
Starting from Spring 4 the bean name is the default qualifier when auto-wiring so the new transaction manager will be wired into DefaultDatasourceConfiguration as it matches the field name in the class.
Alternatively don't use DefaultDatasourceConfiguration and roll out your own configuration if Spring Boot defaults are not working for you.

Use #Qualifier annotation
The #Qualifier annotation is used to resolve the autowiring conflict, when there are multiple beans of same type.
#Bean
#Qualifier("ubldbTransactionManager")
protected PlatformTransactionManager transactionManager
and
#Bean
#Qualifier("bpdbTransactionManager")
public PlatformTransactionManager bpdbTransactionManager

Spring Boot Datasource in unit tests

I have a simple Spring Boot web app, that reads from a database and return a JSON response. I have the following test configuration:
#RunWith(SpringRunner.class)
#SpringBootTest(classes=MyApplication.class, properties={"spring.config.name=myapp"})
#AutoConfigureMockMvc
public class ControllerTests {
#Autowired
private MockMvc mvc;
#MockBean
private ProductRepository productRepo;
#MockBean
private MonitorRepository monitorRepo;
#Before
public void setupMock() {
Mockito.when(productRepo.findProducts(anyString(), anyString()))
.thenReturn(Arrays.asList(dummyProduct()));
}
#Test
public void expectBadRequestWhenNoParamters() throws Exception {
mvc.perform(get("/products"))
.andExpect(status().is(400))
.andExpect(jsonPath("$.advice.status", is("ERROR")));
}
//other tests
}
I have a DataSource bean that is configured in the main configuration of the application. When I run the tests Spring tries to load the context and fails, because the datasource is taken from JNDI. In general I want to avoid creating a datasource for this tests, because I have the repositories mocked.
Is it possible to skip the creation of datasource when running the unit tests?
In memory database for testing is not an option, because my database creation script has a specific structure and cannot be easily executed from classpath:schema.sql
Edit
The datasource is defined in MyApplication.class
#Bean
DataSource dataSource(DatabaseProeprties databaseProps) throws NamingException {
DataSource dataSource = null;
JndiTemplate jndi = new JndiTemplate();
setJndiEnvironment(databaseProps, jndi);
try {
dataSource = jndi.lookup(databaseProps.getName(), DataSource.class);
} catch (NamingException e) {
logger.error("Exception loading JNDI datasource", e);
throw e;
}
return dataSource;
}

Since you are loading configuration class MyApplication.class datasource bean will be created, Try moving datasource in another bean which is not used in a test, make sure all classes loaded for tests are not dependant on datasource. Or In your tests create a config class marked with #TestConfiguration and include it in SpringBootTest(classes=TestConfig.class) mocking data source there like
#Bean
public DataSource dataSource() {
return Mockito.mock(DataSource.class);
}
But this may fail since method call to this mocked datasouce for connection will return null, In that case, you'll have to create an in-memory datasource and then mock jdbcTemplate and rest of dependencies.

Try adding your datasource as a #MockBean too:
#MockBean
private DataSource dataSource
That way Spring will do the replacing logic for you, having the advantage that your production code bean creation won't even be executed (no JNDI lookup).

Change Spring config connection in runtime

I need to change my connection with the Database in runtime. For Example, if parameter requested is BD1, connect on database 1, if BD2, connect in database 2, etc.
I am using spring boot. What is the best way for this.
I have this #Configuration, but not know to say my repository how to use.
#Configuration
public class DataSourceConfiguration {
#Bean(name = "ccteste")
#ConfigurationProperties("spring.ciclocairu.teste.datasource")
#Primary
public DataSource ciclocairuTeste() {
return DataSourceBuilder.create().build();
}
#Bean(name = "ccprod")
#ConfigurationProperties("spring.ciclocairu.prod.datasource")
public DataSource ciclocairuProd() {
return DataSourceBuilder.create().build();
}
#Bean(name = "tmccteste")
#Autowired
#Primary
DataSourceTransactionManager transactionManagerCicloCairuTeste(#Qualifier("ccteste") DataSource datasource) {
DataSourceTransactionManager txm = new DataSourceTransactionManager(datasource);
return txm;
}
#Bean(name = "tmccprod")
#Autowired
#Primary
DataSourceTransactionManager transactionManagerCicloCairuProd(#Qualifier("ccprod") DataSource datasource) {
DataSourceTransactionManager txm = new DataSourceTransactionManager(datasource);
return txm;
}
}

Looks like you are looking for some data source routing. Spring has AbstractRoutingDataSource for run-time detection what data source should be used.
Abstract DataSource implementation that routes getConnection() calls
to one of various target DataSources based on a lookup key.
Also you can set default datasource by setDefaultTargetDataSource method.
It works in such way: you put data sources that you need in a map in AbstractRoutingDataSource at the bean configuration stage, and when you need to use a specific data source you put the key for this source into the context that linked to the router. This ds-key is linked to the current thread.
Here are examples : Dynamic DataSource Routing with Spring and Spring DataSource Routing

Might take a look at my answer at Manage transactions with multiple datasource, entity managers for same application code
or my blog post: Multi-tenant applications using Spring Boot, JPA, Hibernate and Postgres

#EnableTransactionManagement annotation with 2 transaction managers

I am using #Configuration annotation for configuration of spring instead of xml file. I am configuring 2 datasources with different session factory and different transaction managers. I am stuck with a problem here for #EnableTransactionManagement annotation. I read in its documentation that,
#EnableTransactionManagement is more flexible; it will fall back to a
by-type lookup for any PlatformTransactionManager bean in the
container. Thus the name can be "txManager", "transactionManager", or
"tm": it simply does not matter.
This means whatever name I give to method, it will always search for the method which returns PlatformTransactionManager object while I have 2 transactionmanagers. Now the problem is, when I test this class, it gives me error:
org.springframework.beans.factory.NoSuchBeanDefinitionException: No unique bean of type [org.springframework.transaction.PlatformTransactionManager] is defined: expected single bean but found 2
I even tried to have 2 different Configuration classes but in vain. In xml configuration, this was not the case. I registered my both transaction managers with two <tx:annotation-driven transaction-manager="" /> tag and it worked fine. But not able to do same here with annotations.
What should I do if I want to configure 2 datasources with 2 different transaction managers in Spring annotated configuration class?

In your configuration class, use #EnableTransactionManagement annotation.
Define a transaction manager in this class as:
#Bean(name="txName")
public HibernateTransactionManager txName() throws IOException{
HibernateTransactionManager txName= new HibernateTransactionManager();
txName.setSessionFactory(...);
txName.setDataSource(...);
return txName;
}
There on, in your class/method that executes transactional job(s), annotate as follows:
#Transactional("txName")
or
#Transactional(value = "txName")
This is how you would tie a name qualified transaction manager to wherever you need it. You can now have as many transaction managers as you want and use it accordingly wherever you need.

Just in case anyone runs into this problem, I found a solution:
#Configuration
#EnableTransactionManagement
#DependsOn("myTxManager")
#ImportResource("classpath:applicationContext.xml")
public class AppConfig implements TransactionManagementConfigurer {
#Autowired
private PlatformTransactionManager myTxManager;
...
#Override
public PlatformTransactionManager annotationDrivenTransactionManager() {
return this.myTxManager;
}
In this way, you can use a specific txManager defined in an xml configuration.
In case you want to define the txManager used on service-level, you shall remove the #EnableTransactionManagement annotation from the #Configuration class and specify the txManager in the #Transactional annotations, e.g.
#Service
#Transactional(value="myTxManager", readOnly = true)
public class MyServiceImpl implements MyService { ... }

From the java doc
For those that wish to establish a more direct relationship between
#EnableTransactionManagement and the exact transaction manager bean to be used, the
TransactionManagementConfigurer callback interface may be implemented - notice the
implements clause and the #Override-annotated method below:
Your #Configuration class needs to implement TransactionManagementConfigurer interface - implement the annotationDrivenTransactionManager which will return the reference to the transactionManager that should be used.

I am not sure why you are using two TransactionManagers . You could consider using the same TransactionManager for multiple datasource via the AbstractRoutingDataSource . Please refer
http://blog.springsource.org/2007/01/23/dynamic-datasource-routing/
for a sample on its usage.

I have to use JPA and Reactive Mongo in one project. What works at last was:
create a #Configuraition class to explicitly create a JPA transaction manager, like here:
private Environment env;
#Bean
#Primary
public LocalContainerEntityManagerFactoryBean dbEntityManager() {
LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
em.setDataSource(dbDatasource());
em.setPackagesToScan(new String[]{"projectone.mysql"});
em.setPersistenceUnitName("dbEntityManager");
HibernateJpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
em.setJpaVendorAdapter(vendorAdapter);
HashMap<String, Object> properties = new HashMap<>();
properties.put("hibernate.dialect",env.getProperty("hibernate.dialect"));
properties.put("hibernate.show-sql",env.getProperty("jdbc.show-sql"));
em.setJpaPropertyMap(properties);
return em;
}
#Primary
#Bean
public DataSource dbDatasource() {
DriverManagerDataSource dataSource
= new DriverManagerDataSource();
dataSource.setDriverClassName(
env.getProperty("spring.datasource.driverClassName"));
dataSource.setUrl(env.getProperty("spring.datasource.url"));
dataSource.setUsername(env.getProperty("spring.datasource.username"));
dataSource.setPassword(env.getProperty("spring.datasource.password"));
return dataSource;
}
#Primary
#Bean
public PlatformTransactionManager jpaTransactionManager() {
JpaTransactionManager transactionManager
= new JpaTransactionManager();
transactionManager.setEntityManagerFactory(
dbEntityManager().getObject());
return transactionManager;
}
}
Notice that the bean name jpaTransactionManager, which would be the txManager name used in JPA #Transactional.
create MongoConfiguration to explicitly create a Mongo transaction manager(a lot of beans to define)
in #Transactional, call them with name. The default one transactionManger will not work. You have to distinguish, like jpaTransactionManager and reactiveMongoTransactionManger.
#Transactional(value="jpaTransactionManager")
public void xxx() {
...
}
Note that JPA transaction methods cannot Reactor types as return value(Mono/Flux). Spring will force methods returning Mono/Flux to use ReactiveTransactionManager, it will cause confusion.

Some of the other answers imply that using two transaction managers is in some way wrong; however, Spring's XML configuration allows for using multiple transaction managers as stated in the online documentation (below). Unfortunately, there does not seem to be a way to make the #EnableTransactionManagement annotation work in a similar manner. As a result, I simply use an #ImportResource annotation to load an XML file that includes the <tx:annotation-driven/> line. This allows you to get a Java configuration for most things but still make use of #Transactional with an optional Transaction Manager qualifier.
http://docs.spring.io/spring/docs/3.1.x/spring-framework-reference/html/transaction.html
Most Spring applications only need a single transaction manager, but there may be situations where you want multiple independent transaction managers in a single application. The value attribute of the #Transactional annotation can be used to optionally specify the identity of the PlatformTransactionManager to be used. This can either be the bean name or the qualifier value of the transaction manager bean. For example, using the qualifier notation, the following Java code

Try to use chained TransactionalManager
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.transaction.ChainedTransactionManager;
import org.springframework.transaction.PlatformTransactionManager;
#Configuration
public class ChainedDBConfig {
#Bean("chainedTransactionManager")
public PlatformTransactionManager transactionManager(
#Qualifier("database1TransactionManager") final PlatformTransactionManager db1PlatformTransactionManager,
#Qualifier("database2TransactionManager") final PlatformTransactionManager db2PlatformTransactionManager) {
return new ChainedTransactionManager(db1PlatformTransactionManager, db2PlatformTransactionManager);
}
}
And place the following annotation on your service class:
#Transactional(transactionManager = "chainedTransactionManager")
public class AggregateMessagesJobIntegrationTest {
...
}
You can also use it inside the integration tests:
#RunWith(SpringRunner.class)
#Transactional(transactionManager = "chainedRawAndAggregatedTransactionManager")
#Rollback
public class ExampleIntegrationTest extends AbstractIntegrationTest {
....
}
and it will do a rollback for both DB transaction managers.