The problem rears its ugly head when the JdbcJobInstanceDao attempts to call the FIND_JOBS_WITH_KEY query:
SELECT JOB_INSTANCE_ID, JOB_NAME from %PREFIX%JOB_INSTANCE where JOB_NAME = ? and JOB_KEY = ?
the %PREFIX% token is replaced with the value of application.properties key spring.batch.table-prefix which defaults to "BATCH_".
The application properties are definitely loading from the files as my small test demonstrates:
#ActiveProfiles("test") // to load `application-test.properties`
#RunWith(SpringRunner.class)
// we don't need a web context as we are playing with only server side classes
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.NONE, classes = {TestDatabaseConfig.class,
MyBatchProperties.class, SpringBatchTestConfig.class})
#ComponentScan(basePackageClasses = {MyBatchConfig.class})
// MyBatchConfig has #EnableBatchProcessing and all job configurations.
public class BatchTest {
#Autowired
private JobLauncherTestUtils jobLauncherTestUtils;
#Autowired
private ApplicationContext applicationContext;
#Before
public void setup() {
Environment environment = applicationContext.getEnvironment();
System.out.println(environment.getProperty("spring.batch.table-prefix"));
// above prints MY_SCEHMA_USER.BATCH_ as expected
}
#Test
public void checkJobRuns() {
try {
jobLauncherTestUtils.launchJob();
} catch (Exception e) {
e.printStackTrace(); // <-- fails here with the query returning "table not found" because the prefix was not configured correctly.
}
}
}
application-test.properties:
spring.batch.table-prefix=MY_SCHEMA_USER.BATCH_
I've been working with custom configuration for job runs for a long time but the JobLauncherTestUtils doesn't seem to honour these configuration properties.
I need the different table prefix as the batch database tables are owned by a different schema to the connected database user. (i.e. MY_APP_USER trying to access MY_SCHEMA_USER.BATCH_JOB_INSTANCE). unqualified references to tables try (and fail) to resolve the batch tables against MY_APP_USER instead of MY_SCHEMA_USER.
I've tried to create a JobRepositoryFactoryBean bean and annotate it with #ConfigurationProperties("spring.batch"). However - along with this not working anyway - I don't see why I should configure these this way rather than with properties.
How to I get the Batch related beans properly configured with application properties in junit tests using JobLauncherTestUtils?
Fixed
The problem was because i was creating my own BatchConfigurer bean inside of the MyBatchConfig class.
The BatchConfigurer registers the properties for configuration with each component of the Batch framework (jobRepository, daos, etc...)
This meant that the properties were not being populated through the #Component annotated DefaultBatchConfigurer class. As to why I decided to place this piece of code in here even I - the author - Am not sure. I guess i need to not code after minimal sleep.
Thanks for indulging my stupidity!
Related
How can I bootstrap my Spring Boot 2 integration tests so that across all of them I can have one set of configurations which pre-seeds the test database with some test data that can be used across all integration tests?
Supposing you are using h2 test database.
My src/test/resources/application.properties file has:
spring.jpa.hibernate.ddl-auto=create-drop
You'll need a configuration file with the following structure. (This is a configuration example located inside the folder src/test/java):
#Profile("test")
#Configuration
public class H2Config {
#Autowired
private DataSource datasource;
#PostConstruct
public void loadSQL() throws Exception {
ScriptUtils.executeSqlScript(datasource.getConnection(), new ClassPathResource("/sql/load_database.sql"));
}
}
The file 'load_database.sql': (the full path is /src/test/resources/sql/load_database.sql)
CREATE OR REPLACE TABLE OPER_DISPN(
ID NUMBER NOT NULL,
DT_VCTO_OPER DATE NOT NULL
);
INSERT INTO OPER_DISPN(ID,DT_VCTO_OPER) VALUES (1,TO_DATE('2018/09/21', 'yyyy/mm/dd'));
If you are using mapped entities(with #Entity) ( with create-drop) you won't need the 'CREATE TABLE' part for that.
And now, all your integration tests have the script data inserted
Edit: ( My test structure) I've created at github my example application. Please see the test structure and run the tests:
TNotMappedRepository.testLoadDataFind()
PersonRepository.testLoadDataFind()
Github: https://github.com/thiagochagas/insert-data-tests
I have an application that uses Spring Batch to define a preset number of jobs, which are currently all defined in the XML.
We add more jobs over time which requires updating the XML, however these jobs are always based on the same parent and can easily be predetermined using a simple SQL query.
So I've been trying to switch to use some combination of XML configuration and Java-based configuration but am quickly getting confused.
Even though we have many jobs, each job definition falls into essentially one of two categories. All of the jobs inherit from one or the other parent job and are effectively identical, besides having different names. The job name is used in the process to select different data from the database.
I've come up with some code much like the following but have run into problems getting it to work.
Full disclaimer that I'm also not entirely sure I'm going about this in the right way. More on that in a second; first, the code:
#Configuration
#EnableBatchProcessing
public class DynamicJobConfigurer extends DefaultBatchConfigurer implements InitializingBean {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private JobRegistry jobRegistry;
#Autowired
private DataSource dataSource;
#Autowired
private CustomJobDefinitionService customJobDefinitionService;
private Flow injectedFlow1;
private Flow injectedFlow2;
public void setupJobs() throws DuplicateJobException {
List<JobDefinition> jobDefinitions = customJobDefinitionService.getAllJobDefinitions();
for (JobDefinition jobDefinition : jobDefinitions) {
Job job = null;
if (jobDefinition.getType() == 1) {
job = jobBuilderFactory.get(jobDefinition.getName())
.start(injectedFlow1).build()
.build();
} else if (jobDefinition.getType() == 2) {
job = jobBuilderFactory.get(jobDefinition.getName())
.start(injectedFlow2).build()
.build();
}
if (job != null) {
jobRegistry.register(new ReferenceJobFactory(job));
}
}
}
#Override
public void afterPropertiesSet() throws Exception {
setupJobs();
}
public void setInjectedFlow1(Flow injectedFlow1) {
this.injectedFlow1 = injectedFlow1;
}
public void setInjectedFlow2(Flow injectedFlow2) {
this.injectedFlow2 = injectedFlow2;
}
}
I have the flows that get injected defined in the XML, much like this:
<batch:flow id="injectedFlow1">
<batch:step id="InjectedFlow1.Step1" next="InjectedFlow1.Step2">
<batch:flow parent="InjectedFlow.Step1" />
</batch:step>
<batch:step id="InjectedFlow1.Step2">
<batch:flow parent="InjectedFlow.Step2" />
</batch:step>
</batch:flow>
So as you can see, I'm effectively kicking off the setupJobs() method (which is intended to dynamically create these job definitions) from the afterPropertiesSet() method of InitializingBean. I'm not sure that's right. It is running, but I'm not sure if there's a different entry point that's better intended for this purpose. Also I'm not sure what the point of the #Configuration annotation is to be honest.
The problem I'm currently running into is as soon as I call register() from JobRegistry, it throws the following IllegalStateException:
To use the default BatchConfigurer the context must contain no more than one DataSource, found 2.
Note: my project actually has two data sources defined. The first is the default dataSource bean which connects to the database that Spring Batch uses. The second data source is an external database, and this second one contains all the information I need to define my list of jobs. But the main one does use the default name "dataSource" so I'm not quite sure how else I can tell it to use that one.
First of all - I don't recommend using a combination of XML as well as Java Configuration. Use only one, preferably Java one as its not much of an effort to convert XML config to Java config. (Unless you have some very good reasons to do it - that you haven't explained)
I haven't used Spring Batch alone as I have always used it with Spring Boot and I have a project where I have defined multiple jobs and it always worked well for similar code that you have shown.
For your issue, there are some answers on SO like this OR this which are basically trying to say that you need to write your own BatchConfigurer and not rely on default one.
Now coming to solution using Spring Boot
With Spring Boot, You should try segregate job definitions and job executions.
You should first try to just define jobs and initialize Spring context without enabling jobs (spring.batch.job.enabled=false)
In your Spring Boot main method, when you start app with something like - SpringApplication.run(Application.class, args); ...you will get ApplicationContext ctx
Now you can get your relevant beans from this context & launch specif jobs by getting names from property or command line etc & using JobLauncher.run(...) method.
You can refer my this answer if willing to order job executions. You can also write job schedulers using Java.
Point being, you separate your job building / bean configurations & job execution concerns.
Challenge
Keeping multiple jobs in a single project can be challenging when you try to have different settings for each job as application.properties file is environment specific and not job specific i.e. spring boot properties will apply to all jobs.
In my particular case, the solution was to actually eliminate the #Configuration
and #EnableBatchProcessing annotations from my class above. Something about these caused it to try and use the DefaultBatchConfigurer which fails when you have more than one data source defined (even if you've identified them clearly with "dataSource" as the primary and some other name for the secondary).
The #Configuration class in particular wasn't necessary because all it really does is lets your class get auto-instantiated without having to define it as a bean in the app context. But since I was doing that anyway this one was superfluous.
One of the downsides of removing #EnableBatchProcessing was that I could no longer auto-wire the JobBuilderFactory bean. So instead I just had to do to create it:
JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean();
factory.setDataSource(dataSource);
factory.setTransactionManager(transactionManager);
factory.afterPropertiesSet();
jobRepository = factory.getObject();
jobBuilderFactory = new JobBuilderFactory(jobRepository);
Then it seems I was on the right track already by using jobRegistry.register(...) to define my jobs. So essentially once I removed those annotations above everything started working. I'm going to mark Sabir's answer as the correct one however because it helped me out.
I have an application where I use Spring (annotations, not xml), and I need to load the beans in my unit tests. I have the AppConfig class from my code which I want to use, but with a different datasource (one I define in the test folder). This is because I want to use an in memory DB in my tests, and not the real DB.
Here's how I try to run the AppConfig class:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(classes = {App.class, AppConfig.class})
public class DAOManagerTest {
//All code goes here
#AutoWired
UserDAO userDAO;
#Test
public void testGet() {
List<User> test = userDAO.selectAll();
for (User u: test) {
u.toString();
}
}
}
This doesn't entirely work, as it fails on creating a bean inside the UserDAO class. I guess I need some tutorial / guide on how to deal with spring in unit tests. Should I define new beans in my test folder, or is it possible to user the spring class from my code? Also, is it possible to define a seperate datasource for the tests?
Yes. For example, if you define some beans in DAOManagerTest, using #Primary if necessary, and add DAOManagerTest.class to #ContextConfiguration.
There are so many other ways of arranging it though, like using profiles or mocks, etc.
I am building a spring 4 + Hibernate5 application. I wonder whether is there any difference in defining the data base connection properties like url ,username etc via DataSource object and via hibernate properties like "hibernate.connection.url", "hibernate.connection.username" etc. Offcourse ultimately the datasource object will be tied to the session factory. Just want to make sure to do the things in right way.
I want to define a separate datesource object via dataSource property, so that I can use the AbstractTransactionalTestNGSpringContextTests for test cases. This classs is always expecting a data source object. I want to use the #Rollback feature and this feature is working with AbstractTransactionalTestNGSpringContextTests. AbstractTestNGSpringContextTests is not supporting the roll back feature but still persisting working perfectly.
Need inputs to implements in the right way.
Adding example code to provide more information.
#ContextConfiguration(locations = { "classpath:spring/fpda_persistence_config.xml" })
#Rollback
#Transactional
public class BankTransactionDAOTest extends AbstractTestNGSpringContextTests {
#Autowired
private BankTransactionDAO bankTransactionDao;
#Test
public void createBankTransactionTest(){
BankTransaction bt = new BankTransaction();
bt.setAuthoritativeTableId(new BigDecimal(1234));
bt.setBankTransactionTypeCode(new Character('C'));
bt.setInstanceId(new BigDecimal(1234));
bt.setRowCreatedEpochTime(new BigDecimal(1234));
bt.setTransactionId(new BigDecimal(1234));
bt.setTransactionKey(new BigDecimal(System.currentTimeMillis()));
bankTransactionDao.createBankTransaction(bt);
}
}
here to make transaction roll back happen sucessfully, I came to know that we should extend AbstractTransactionalTestNGSpringContextTests instead of AbstractTestNGSpringContextTests. Then I should have declared the datasource property instead of defining all properties in hibernate properties.
so overall is it a right approach to declare some properties in data source and some properties in hibernate?. will it make any difference.
Thanks in Advance.
In our project we are using class-level annotation for test classes
#TestPropertySource(locations = "classpath:application-test.properties")
I have datasource defined in my application.properties as Oracle database and that's ok, the controller & repository work fine, I can persist entities and fetch database records.
I have integration tests written. Before I connected my app with database I created some objects and persisted them in #PostConstruct method - and that was ok. But now, when I connected everything with database, my tests try to fetch records from the table which is pretty large.
I think that's due to my application.properties file in which defined datasource is Oracle database.
spring.datasource.driverClassName=oracle.jdbc.driver.OracleDriver
spring.datasource.url=jdbc:oracle:thin:#blabla
spring.datasource.username=blabla
spring.datasource.password=blabla
spring.jpa.hibernate.ddl-auto=none
spring.jpa.generate-ddl=false
I want to carry on tests using some in-memory HSQL, not real database. But I still want my controller and repository to use the real one. How can I do it? Is there any possibility to add second datasource in application.properties? Or maybe an easier solution?
In Spring 4 you have #Profile annotation so you can use its advantage.
Create another like application-test.properties file with it's own properties.
Create configuration for your tests:
#Configuration
#Profile("test")
public class TestConfiguration {
....
}
Then annotate your class with #ActiveProfiles annotation:
#RunWith(SpringJUnit4ClassRunner.class)
#SpringApplicationConfiguration(classes = TestConfiguration.class)
#ActiveProfiles("test")
public class TestRepository {
...
}
There are many ways to achieve this, one way is to use Spring profiles. The test profile would use a different properties file than the production one.
Spring has org.springframework.mock.jndi.SimpleNamingContextBuilder package that allows you to bind the dataSource programatically.
Sample Test class that I use:
public class MockJNDIContext {
#BeforeClass
public static void setUpClass() throws Exception {
SimpleNamingContextBuilder builder = SimpleNamingContextBuilder.emptyActivatedContextBuilder();
OracleConnectionPoolDataSource ds = new OracleConnectionPoolDataSource();//Use your own Connection Pool DS here as you require
ds.setURL("jdbc:oracle:thin:#10.xxx.xxx.xx:9999:SID");
ds.setUser("USER");
ds.setPassword("PASSWORD");
//Bind it to the JNDI you need
builder.bind("java:comp/env/jdbc/MY/APP", ds);
}
#Test
public void test(){
System.out.println("JNDI Initialized");
}
}
Suite Class:
#RunWith(Suite.class)
#Suite.SuiteClasses({
MockJNDIContext.class,
InitializeTestConfig.class
})
public class ServiceSuite{
}
May be this helps? If you are trying to load from one more application.props, use one more class (InitializeTestConfig.class) that initializes and passes the DS args and add to suite as mentioned above and try?