We Keep Coding

SpringBatch Sharing Large Amounts of Data Between Steps

I have a need to share relatively large amounts of data between job steps for a spring batch implementation. I am aware of StepExecutionContext and JobExecutionContext as mechanisms for this. However, I read that since these must be limited in size (less than 2500 characters). That is too small for my needs. In my novice one-step Spring Batch implementation, my single step job is as below:
#Configuration
#EnableBatchProcessing
public class BatchConfig {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
private static final String GET_DATA =
" SELECT " +
"stuffA, " +
"stuffB, " +
"FROM STUFF_TABLE " +
"ORDER BY stuffA ASC";
#Bean
public ItemReader<StuffDto> databaseCursorItemReader(DataSource dataSource) {
return new JdbcCursorItemReaderBuilder<StuffDto>()
.name("cursorItemReader")
.dataSource(dataSource)
.sql(GET_DATA)
.rowMapper(new BeanPropertyRowMapper<>(StuffDto.class))
.build();
}
#Bean
ItemProcessor<StuffDto, StuffDto> databaseXmlItemProcessor() {
return new QueryLoggingProcessor();
}
#Bean
public ItemWriter<StuffDto> databaseCursorItemWriter() {
return new LoggingItemWriter();
}
#Bean
public Step databaseCursorStep(#Qualifier("databaseCursorItemReader") ItemReader<StuffDto> reader,
#Qualifier("databaseCursorItemWriter") ItemWriter<StuffDto> writer,
StepBuilderFactory stepBuilderFactory) {
return stepBuilderFactory.get("databaseCursorStep")
.<StuffDto, StuffDto>chunk(1)
.reader(reader)
.writer(writer)
.build();
}
#Bean
public Job databaseCursorJob(#Qualifier("databaseCursorStep") Step exampleJobStep,
JobBuilderFactory jobBuilderFactory) {
return jobBuilderFactory.get("databaseCursorJob")
.incrementer(new RunIdIncrementer())
.flow(exampleJobStep)
.end()
.build();
}
}
This works fine in the sense that I can successfully read from the database and write in the writer step to a loggingitemwriter like this:
public class LoggingItemWriter implements ItemWriter<StuffDto> {
private static final Logger LOGGER = LoggerFactory.getLogger(LoggingItemWriter.class);
#Override
public void write(List<? extends StuffDto> list) throws Exception {
LOGGER.info("Writing stuff: {}", list);
}
}
However, I need to be able to make available that StuffDto (or equivalent) and it's data to a second step that would be performing some processing against it rather than just logging it.
I would be grateful for any ideas on how that could be accomplished if you assume that the step and job contexts are too limited. Thanks.

If you do not want to write the data in the database or filesystem, one way to achieve the same is like below:
Create your own job context bean in your config class having the required properties and annotated it with #JobScope.
Implement org.springframework.batch.core.step.tasklet interface to your reader, processor and writer classes. If you want more control over steps you can also implement org.springframework.batch.core.StepExecutionListener with it.
Get your own context object using #Autowire and use the setter-getter method of it to store and retrieve the data.
Sample Code:
Config.java
#Autowired
private Processor processor;
#Autowired
private Reader reader;
#Autowired
private Writer writer;
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Bean
#JobScope
public JobContext getJobContexts() {
JobContext newJobContext = new JobContext();
return newJobContext;
}
#Bean
public Step reader() {
return stepBuilderFactory.get("reader")
.tasklet(reader)
.build();
}
#Bean
public Step processor() {
return stepBuilderFactory.get("processor")
.tasklet(processor)
.build();
}
#Bean
public Step writer() {
return stepBuilderFactory.get("writer")
.tasklet(writer)
.build();
}
public Job testJob() {
return jobBuilderFactory.get("testJob")
.start(reader())
.next(processor())
.next(writer())
.build();
}
//Below will start the job
#Scheduled(fixedRate = 1000)
public void starJob(){
Map<String, JobParameter> confMap = new HashMap<>();
confMap.put("time", new JobParameter(new Date()));
JobParameters jobParameters = new JobParameters(confMap);
monitorJobLauncher.run(testJob(), jobParameters);
}
JobContext.java
private List<StuffDto> dataToProcess = new ArrayList<>();
private List<StuffDto> dataToWrite = new ArrayList<>();
//getter
SampleReader.java
#Component
public class SampleReader implements Tasklet,StepExecutionListener{
#Autowired
private JobContext context;
#Override
public void beforeStep(StepExecution stepExecution) {
//logic that you need to perform before the execution of this step.
}
#Override
public void afterStep(StepExecution stepExecution) {
//logic that you need to perform after the execution of this step.
}
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext){
// Whatever code is here will get executed for reader.
// Fetch StuffDto object from database and add it to jobContext
//dataToProcess list.
return RepeatStatus.FINISHED;
}
}
SampleProcessor.java
#Component
public class SampleProcessor implements Tasklet{
#Autowired
private JobContext context;
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext){
// Whatever code is here will get executed for processor.
// context.getDataToProcessList();
// apply business logic and set the data to write.
return RepeatStatus.FINISHED;
}
Same ways for the writer class.
Note: Please note here that here you need to write database-related boilerplate code on your own. But this way you can have more control over your logic and nothing to worry about context size limit. All the data will be in memory so as soon as operation done those will be garbage collected. I hope you get the idea of what I willing to convey.
To read more about Tasklet vs Chunk read this.

Spring boot batch scheduler run once

I started to learn Spring Boot Batch in version 2.1.4
I want to run my job in scheduler and this job runs only once. I mean ItemProcessor and ItemWriter run only once. ItemReader runs every time. anyone have an idea what I did wrong. In the future, I want to change scheduler to Java WatchService and pass filePath to the job but now parameter for filePath is like a string in the function parameter. This is my code:
This is my reader:
#Component
public class UserReaderImpl {
#StepScope
public ItemReader<UserCsvStructure> read(String filepath) {
FlatFileItemReader<UserCsvStructure> reader = new FlatFileItemReader();
reader.setLinesToSkip(1);
reader.setResource(new FileSystemResource(filepath));
reader.setLineMapper(new DefaultLineMapper<UserCsvStructure>() {
{
setLineTokenizer(new DelimitedLineTokenizer() {
{
setNames(new String[]{"firstName","lastName","email"});
}
});
setFieldSetMapper(new BeanWrapperFieldSetMapper<UserCsvStructure>() {
{
setTargetType(UserCsvStructure.class);
}
});
}
});
return reader;
}
}
This in my ItemProcessor
#StepScope
#Component
public class UserProcessorImpl implements ItemProcessor<UserCsvStructure, User> {
#Override
public User process(UserCsvStructure userCsvStructure) throws Exception {
return User.builder()
.email(userCsvStructure.getEmail())
.firstName(userCsvStructure.getFirstName())
.lastName(userCsvStructure.getLastName())
.build();
}
}
This is my ItemWriter
#Component
#StepScope
public class UserWriterImpl implements ItemWriter<User>{
#Autowired
private UserRepository userRepository;
#Override
public void write(List<? extends User> list) throws Exception {
System.out.println(list);
userRepository.saveAll(list);
}
}
And this is my configuration
#Component
public class UserBatchCsvConfig {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
private UserReaderImpl userReader;
#Autowired
private UserWriterImpl userWriter;
#Autowired
private UserProcessorImpl userProcessor;
public Job csvFileToDatabaseJob(UserJobCompletionNotificationListener listener, String fileName) {
return jobBuilderFactory.get("userCsvProcess")
.incrementer(new RunIdIncrementer())
.listener(listener)
.flow(csvFileToDatabaseStep(fileName))
.end()
.build();
}
private Step csvFileToDatabaseStep(String fileName) {
return stepBuilderFactory.get("userCsvProcess")
.<UserCsvStructure, User>chunk(1)
.reader(userReader.read(fileName))
.processor(userProcessor)
.writer(userWriter)
.build();
}
}
Last class is my scheduler:
#Component
public class UserCsvProcessor {
#Autowired
private JobLauncher jobLauncher;
#Autowired
private UserBatchCsvConfig job;
#Autowired
private UserJobCompletionNotificationListener userJobCompletionNotificationListener;
#Scheduled(fixedDelay = 10000)
public void runJob() throws Exception {
jobLauncher.run(job.csvFileToDatabaseJob(userJobCompletionNotificationListener, "C:\\Users\\Anik\\Desktop\\angular\\test.csv"), new JobParameters());
}
}

I know what should I add in my code
In UserCsvProcessor class I need to change my scheduled function to:
#Scheduled(fixedDelay = 10000)
public void runJob() throws Exception {
JobParameters params = new JobParametersBuilder()
.addString("JobID", String.valueOf(System.currentTimeMillis()))
.toJobParameters();
jobLauncher.run(job.csvFileToDatabaseJob(userJobCompletionNotificationListener, "C:\\Users\\Anik\\Desktop\\angular\\test.csv"), params);
}
If someone has other idea or better idea just add an answer

With the configuration you have in #Scheduled annotation you are indicating to be executed every 10 seconds. So, when your first execution is completed it will wait 10 seconds and then execute it again and so on.
#Scheduled(fixedDelay = 10000)
If you want to execute it once (I guess it is once a day) you can use cron expression in your #Scheduled annotation. Check the example below where the cron expression indicates that the method should be executed every day at 10:15 a.m.
#Scheduled(cron = "0 15 10 * * *")
If you want to run it once a month/year you can handle the cron expression to do that.
Additionally, you can read that expression from the configuration file using something like the following:
#Scheduled(cron = "${cron.expression}")

Spring Quartz Triggering Job programmatically

I have Sprint Boot - Java 8 application that has a quartz job that I configure upon start up and set a schedule. The job runs automatically per the schedule as you would expect with a quartz job. However, now I want to be able to allow the user to manually trigger these jobs with a click of a button on the front end WITHOUT messing up the normal scheduling of that job. Here are all my relevant files.
application.yml
quartz:
fooCron: 0 0 1 * * ?
fooGroup: foo-quartz-group
QuartzConfig.java
#Configuration
#ConfigurationProperties(prefix = "quartz")
public class QuartzConfig {
private String fooCron;
private String fooGroup;
#Autowired
private ApplicationContext applicationContext;
#Autowired
private PlatformTransactionManager transactionManager;
#Autowired
private DataSource dataSource;
#Bean
public SchedulerFactoryBean quartzScheduler() {
AutowiringSpringBeanJobFactory jobFactory = new AutowiringSpringBeanJobFactory();
jobFactory.setApplicationContext(applicationContext);
Trigger[] triggers = {fooTrigger().getObject()};
SchedulerFactoryBean quartzScheduler = new SchedulerFactoryBean();
quartzScheduler.setJobFactory(jobFactory);
quartzScheduler.setTransactionManager(transactionManager);
quartzScheduler.setDataSource(dataSource);
quartzScheduler.setOverwriteExistingJobs(true);
quartzScheduler.setSchedulerName("foo-scheduler");
quartzScheduler.setQuartzProperties(quartzProperties());
quartzScheduler.setTriggers(triggers);
return quartzScheduler;
}
#Bean
public CronTriggerFactoryBean fooTrigger() {
CronTriggerFactoryBean cronTriggerFactoryBean = new CronTriggerFactoryBean();
cronTriggerFactoryBean.setJobDetail(fooJob().getObject());
cronTriggerFactoryBean.setCronExpression(fooCron);
cronTriggerFactoryBean.setGroup(fooGroup);
return cronTriggerFactoryBean;
}
#Bean
public JobDetailFactoryBean fooJob() {
JobDetailFactoryBean jobDetailFactoryBean = new JobDetailFactoryBean();
jobDetailFactoryBean.setJobClass(FooJob.class);
jobDetailFactoryBean.setGroup(fooGroup);
jobDetailFactoryBean.setDurability(true);
return jobDetailFactoryBean;
}
#Bean
public Properties quartzProperties() {
PropertiesFactoryBean propertiesFactoryBean = new PropertiesFactoryBean();
propertiesFactoryBean.setLocation(new ClassPathResource("/quartz/quartz.properties"));
Properties properties = null;
try {
propertiesFactoryBean.afterPropertiesSet();
properties = propertiesFactoryBean.getObject();
} catch (IOException e) {
}
return properties;
}
//setters
}
FooJob.java
#Service
public class FooJob implements Job {
private final FooRepository fooRepo; //This is a repository class annotated with #Repository.
public FooJob(FooRepository fooRepo) {
this.fooRepo = fooRepo;
}
#Override
public void execute(final JobExecutionContext context) throws JobExecutionException {
//do stuff
}
}
Now this runs just fun on a timed scheduled. Per the cron configuration in the yml file, 0 0 1 * * ?, the job executes everyday at 1am. Great! But now I want to execute this manually. So I build a controller to receive manual trigger requests from the UI.
QuartzController.java
#RestController
#RequestMapping("/quartz")
public class QuartzController {
private SchedulerFactoryBean schedulerFactoryBean;
private Scheduler scheduler;
public DevopsController(final SchedulerFactoryBean quartzScheduler) {
this.schedulerFactoryBean = quartzScheduler;
scheduler = schedulerFactoryBean.getScheduler();
}
#PostMapping("/execute")
public ResponseEntity executeJob() {
HttpStatus status = OK;
try {
TriggerKey triggerKey = new TriggerKey("fooTrigger", "foo-quartz-group");
Trigger trigger = scheduler.getTrigger(triggerKey);
ScheduleBuilder scheduleBuilder = SimpleScheduleBuilder.simpleSchedule().withIntervalInSeconds(1).withRepeatCount(0);
JobDetail jobDetail = scheduler.getJobDetail(trigger.getJobKey());
Trigger newTrigger = TriggerBuilder.newTrigger()
.forJob(jobDetail)
.startNow()
.withIdentity(triggerKey)
.withSchedule(scheduleBuilder)
.startAt(Date.from(LocalDate.now().atStartOfDay().atZone(ZoneId.systemDefault()).toInstant()))
.build();
//I have tried all 3 of the following lines
scheduler.scheduleJob(jobDetail, new HashSet<>(Arrays.asList(trigger)), true);
//scheduler.addJob(jobDetail, true);
//scheduler.rescheduleJob(triggerKey, newTrigger);
} catch (SchedulerException e) {
status = BAD_REQUEST;
}
return new ResponseEntity<>(status);
}
}
But everytime I run the application and hit the controller's scheduleJob method I get the following error in the console:
org.quartz.SchedulerException: Job instantiation failed
at org.springframework.scheduling.quartz.AdaptableJobFactory.newJob(AdaptableJobFactory.java:45)
at org.quartz.core.JobRunShell.initialize(JobRunShell.java:127)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:375)
Caused by: java.lang.InstantiationException: com.test.jobs.FooJob
at java.lang.Class.newInstance(Class.java:427)
at org.springframework.scheduling.quartz.AdaptableJobFactory.createJobInstance(AdaptableJobFactory.java:58)
at org.springframework.scheduling.quartz.SpringBeanJobFactory.createJobInstance(SpringBeanJobFactory.java:74)
at com.test.config.AutowiringSpringBeanJobFactory.createJobInstance(AutowiringSpringBeanJobFactory.java:27)
at org.springframework.scheduling.quartz.AdaptableJobFactory.newJob(AdaptableJobFactory.java:41)
... 2 common frames omitted
Caused by: java.lang.NoSuchMethodException: com.test.jobs.FooJob.<init>()
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.newInstance(Class.java:412)
... 6 common frames omitted
What exactly am I doing wrong? How do I get this job to run automatically as per schedule but also execute upon a manual request?
I am using sprint boot 1.5.9.RELEASE and quartz 2.2.1

As #scary-wombat said, you have to add a no parameter constructor in FooJob. The problem with your approach is you won't get a FooRepository in that way.
public FooJob() {
}
You have 2 options
1) If FooRepository has a #Respository annotation, you can add a #Autowired annotation to your constructor.
#Autowired
public FooJob(FooRepository fooRepo) {
this.fooRepo = fooRepo;
}
Thanks to you have a #Repository annotation Spring will notice that it can create an instance of the required object to create an instance of the #service.
2) You can add a configuration class.
#Configuration
public class Config {}
And you have to create there an instance of your service (FooJob).
The first option looks better in my humble opinion.
Let me know if it works!

How to trigger a scheduled Spring Batch Job?

I want to be able to start my job with a REST controller, then when the job is started, it should run on a scheduled basis, until i stop it again with REST.
So this is my Controller:
#RestController
public class LauncherController {
#Autowired
JobLauncher jobLauncher;
#Autowired
Job job;
#RequestMapping("/launch")
public String launch() throws Exception {
...
jobLauncher.run(job, jobParameters);
}
This is some part of the Batch conf:
#Configuration
#EnableBatchProcessing
#EnableScheduling
public class BatchConfiguration {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Scheduled(cron = "0/5 * * * * ?")
#Bean
public Job job() {
return jobBuilderFactory.get("job")
.incrementer(new RunIdIncrementer())
.flow(step1())
.end()
.build();
}
#Bean
public Step step1() {
return stepBuilderFactory.get("step1")
.<Person, Person> chunk(10)
.reader(reader())
.processor(processor())
.writer(writer())
.build();
}
I have also set the property spring.batch.job.enabled=false, since i do not want the jobs run as soon as the Spring Boot App starts.
Now i can call my Rest api lauch, and the job runs, but only once. Scheduler does not work. And I could not figure it our where exactly i should define my #Scheduled Annotation..

I would approach it in a way, that scheduled job runs always, but it does something only when the flag is set to true:
#Component
class ScheduledJob {
private final AtomicBoolean enabled = new AtomicBoolean(false);
#Scheduled(fixedRate = 1000)
void execute() {
if (enabled.get()) {
// run spring batch here.
}
}
void toggle() {
enabled.set(!enabled.get());
}
}
and a controller:
#RestController
class HelloController {
private final ScheduledJob scheduledJob;
// constructor
#GetMapping("/launch")
void toggle() {
scheduledJob.toggle();
}
}

In first you are defining the job:
#Bean
#Qualifier("fancyScheduledJob")
public Job job() {
return jobBuilderFactory.get("job")
.incrementer(new RunIdIncrementer())
.flow(step1())
.end()
.build();
}
In second you are initiating the execution of this job:
#Autowired
#Qualifier(value = "fancyScheduledJob")
private Job job;
#Autowired
private JobLauncher jobLauncher;
#Scheduled(cron = "0/5 * * * * ?")
public void launch() throws JobParametersInvalidException, JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException, JobInstanceAlreadyExistsException, NoSuchJobException {
jobLauncher.run(job, JobParametersBuilder()
.addLong("launchTime", System.currentTimeMillis())
.toJobParameters())
}
Also note that the "launchTime" paramter is introduced: by default spring batch is preventing launching the job with same parameter values.
While your schedule is quite tight - every 5 seconds you should be aware of concurrency. Or if you want to be assured that at each and every moment only 1 instance of the job is executed you can configure custom single threaded job launcher:
#Bean(name = "fancyJobExecutorPool")
public TaskExecutor singleThreadedJobExecutorPool() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(1);
executor.setMaxPoolSize(1);
executor.setQueueCapacity(100500);
executor.setThreadNamePrefix("fancy-job-batch-");
return executor;
}
#Bean(name = "fancyJobLauncher")
public JobLauncher singleThreadedJobLauncher(JobRepository jobRepository)
{
SimpleJobLauncher sjl = new SimpleJobLauncher();
sjl.setJobRepository(jobRepository);
sjl.setTaskExecutor(singleThreadedJobExecutorPool());
return sjl;
}
And use this single threaded job launcher during launch time.
#Autowired
#Qualifier("fancyJobLauncher")
private JobLauncher jobLauncher;
With this your job instances will be executed one by one (but this doesn't limits parallel execution of steps inside of your job).

In this solution you will be able to schedule and unschedule pre defined jobs using http requests. In this example we will create a daily, weekly and an oneTime Job. The application is using Quartz.
<!--Quartz Scheduler -->
<dependency>
<groupId>org.quartz-scheduler</groupId>
<artifactId>quartz</artifactId>
<version>2.2.3</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-tx</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context-support</artifactId>
</dependency>
First we have to create an AutowiringSpringBeanJobFactory class extends SpringBeanJobFactory.
Subclass of {#link AdaptableJobFactory} that also supports
Spring-style * dependency injection on bean properties. This is
essentially the direct * equivalent of Spring's {#link QuartzJobBean}
in the shape of a Quartz * {#link org.quartz.spi.JobFactory}. * *
Applies scheduler context, job data map and trigger data map
entries * as bean property values. If no matching bean property is
found, the entry * is by default simply ignored. This is analogous to
QuartzJobBean's behavior.
public final class AutowiringSpringBeanJobFactory extends SpringBeanJobFactory implements ApplicationContextAware {
private transient AutowireCapableBeanFactory beanFactory;
#Override
public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
beanFactory = applicationContext.getAutowireCapableBeanFactory();
}
#Override
protected Object createJobInstance(final TriggerFiredBundle bundle) throws Exception {
final Object job = super.createJobInstance(bundle);
beanFactory.autowireBean(job);
return job;
}
}
The second part is to configure the quartz configuration. In this config we need to create a
SchedulerFactoryBean where we set global config and the application context,
JobDetailFactoryBean where we set our job, the jobGroup and the class,
CronTriggerFactoryBean where we set the cron expression.
QuartzConfig.class
#Configuration
public class QuartzConfig {
#Autowired
ApplicationContext context;
#Bean
public SchedulerFactoryBean quartzScheduler(){
SchedulerFactoryBean quartzScheduler = new SchedulerFactoryBean();
quartzScheduler.setOverwriteExistingJobs(true);
quartzScheduler.setSchedulerName("job-scheduler");
AutowiringSpringBeanJobFactory jobFactory = new AutowiringSpringBeanJobFactory();
jobFactory.setApplicationContext(context);
quartzScheduler.setJobFactory(jobFactory);
return quartzScheduler;
}
#Bean
#Scope(value = "prototype")
public JobDetailFactoryBean getJobBean(String jobName, String jobGroup, Class<?> clazz){
JobDetailFactoryBean bean = new JobDetailFactoryBean();
bean.setJobClass(clazz);
bean.setGroup(jobGroup);
bean.setName(jobName);
return bean;
}
#Bean
#Scope(value = "prototype")
public CronTriggerFactoryBean getCronTriggerBean(String cronExpression, String triggerGroup){
CronTriggerFactoryBean bean = new CronTriggerFactoryBean();
bean.setCronExpression(cronExpression);
bean.setGroup(triggerGroup);
return bean;
}
}
So, after the config is done we are now able to create our jobs where the business logic will be placed. For that we have to create a class which implements Job.
#Component
public class DailyJob implements Job{
#Override
public void execute(JobExecutionContext context) throws JobExecutionException {
System.out.println("Daily Job runs!");
}
}
The DailyJob class is now ready to get scheduled. We want to schedule this job from outside via a http request. In this example we have a controller where we can send the jobname and the cron expression to schedule the dailyJob.
#Controller
public class JobController {
#Autowired
private Scheduler scheduler;
#Autowired
private ApplicationContext context;;
#ResponseBody
#RequestMapping(value = "/job/create/daily", method = RequestMethod.POST)
public ResponseEntity<JobModel> dailyJob(#RequestBody JobModel jobModel) throws SchedulerException {
JobDetail jobDetail = context.getBean(
JobDetail.class, jobModel.getName(), "MyDailyJob", DailyJob.class);
Trigger cronTrigger = context.getBean(
Trigger.class, jobModel.getCronExpression(), "MyDailyJob");
scheduler.scheduleJob(jobDetail, cronTrigger);
return new ResponseEntity<JobModel>(jobModel, HttpStatus.CREATED);
}
}
What we see here is that we will send a post request with a JobModel as #RequestBody. JobModel is a simple Pojo with two attributes name and cronExpression both Strings.
In this method we have to create the bean instances which we have configured previously in our config class. First create JobDetail with Quartz JobDetail.class, the name of your job, the name of the group and the Class which should be scheduled (in this case DailyJob.class). After that we have to create the Trigger with Quartz Trigger.class, the cronExpression and the group name.
After both beans are created we need to schedule the job now. So we have autowired Quartz Scheduler to schedule the job. After that the job is enabled and ready to do its job.
So let's test the stuff. Start the application and send a post request to /job/create/daily:
{"name":"Job 1", "cronExpression":"0 * * * * ?"}
Here we say that the job should run every minute (just to see that everything works). In your console you should see every minute Daily Job runs!.
And here are some additional things you can do. For example get a list of the scheduled jobs:
#ResponseBody
#RequestMapping("job/list")
public List<String> jobList() throws SchedulerException {
return scheduler.getJobGroupNames();
}
To delete a job you can create endpoints, too. For example:
#ResponseBody
#RequestMapping(value = "job/delete/daily", method = RequestMethod.POST)
public ResponseEntity<Boolean> deleteJob(#RequestBody JobModel jobModel) throws SchedulerException {
JobKey jobKey = new JobKey(jobModel.getName(), "MyDailyJob");
return new ResponseEntity<Boolean>(scheduler.deleteJob(jobKey), HttpStatus.OK);
}
You are free to create many different endpoints to get informations about currently running jobs, how often jobs were running, reschedule jobs and so on. Important is just, that your jobname and the jobgroup( in our case "MyDailyJob") are reusable. Those information are needed to create the jobKey.
P.S.: Just to show the other mappings for the other jobs:
#ResponseBody
#RequestMapping(value = "/job/create/weekly", method = RequestMethod.POST)
public ResponseEntity<JobModel> weeklyJob(#RequestBody JobModel jobModel) throws SchedulerException {
JobDetail jobDetail = context.getBean(JobDetail.class, jobModel.getName(), JobGroup.WEEKLY_GROUP.name(),
WeeklyJob.class);
Trigger cronTrigger = context.getBean(Trigger.class, jobModel.getCronExpression(),
JobGroup.WEEKLY_GROUP.name());
scheduler.scheduleJob(jobDetail, cronTrigger);
return new ResponseEntity<JobModel>(jobModel, HttpStatus.CREATED);
}
#ResponseBody
#RequestMapping(value = "/job/create/oneTime", method = RequestMethod.POST)
public ResponseEntity<JobModel> oneTimeJob(#RequestBody JobModel jobModel) throws SchedulerException {
JobDetail jobDetail = context.getBean(JobDetail.class, jobModel.getName(), JobGroup.ONE_TIME_GROUP.name(),
OneTimeJob.class);
Trigger cronTrigger = context.getBean(Trigger.class, jobModel.getCronExpression(),
JobGroup.ONE_TIME_GROUP.name());
scheduler.scheduleJob(jobDetail, cronTrigger);
return new ResponseEntity<JobModel>(jobModel, HttpStatus.CREATED);
}
The full application is on github

#Scheduled is defined on a method and not on a Bean. So create a new Class which will be a Bean
public class BatchConfiguration {
...
#Bean
public Job job() {
return new Job();
}
new Class:
public class Job {
#Scheduled(cron = "0/5 * * * * ?")
public Job job() {
return jobBuilderFactory.get("job")
.incrementer(new RunIdIncrementer())
.flow(step1())
.end()
.build();
}

Spring Batch WriterNotOpenException

I have created a simple single-step Spring batch job that reads items from a DB, processes them and writes the result to a csv.
During runtime I end up with a
org.springframework.batch.item.WriterNotOpenException: Writer must be open before it can be written to
The relevant code:
#Configuration
#EnableBatchProcessing
#EnableAutoConfiguration
public class CleanEmailJob {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
public DataSource dataSource;
#Bean
public ResourcelessTransactionManager transactionManager() {
return new ResourcelessTransactionManager();
}
#Bean
public MapJobRepositoryFactoryBean mapJobRepositoryFactory(ResourcelessTransactionManager txManager)
throws Exception {
MapJobRepositoryFactoryBean factory = new MapJobRepositoryFactoryBean(txManager);
factory.afterPropertiesSet();
return factory;
}
#Bean
public JobRepository jobRepository(MapJobRepositoryFactoryBean factory) throws Exception {
return factory.getObject();
}
#Bean
public JobExplorer jobExplorer(MapJobRepositoryFactoryBean factory) {
return new SimpleJobExplorer(factory.getJobInstanceDao(), factory.getJobExecutionDao(),
factory.getStepExecutionDao(), factory.getExecutionContextDao());
}
#Bean
public SimpleJobLauncher jobLauncher(JobRepository jobRepository) {
SimpleJobLauncher launcher = new SimpleJobLauncher();
launcher.setJobRepository(jobRepository);
return launcher;
}
#Bean
public Job cleanEmailAddressesJob() throws Exception {
return jobBuilderFactory.get("cleanEmailAddresses")
.incrementer(new RunIdIncrementer())
.start(processEmailAddresses())
.build();
}
#Bean
public Step processEmailAddresses() throws UnexpectedInputException, ParseException, Exception {
return stepBuilderFactory.get("processAffiliates")
.<AffiliateEmailAddress, VerifiedAffiliateEmailAddress> chunk(10)
.reader(reader())
.processor(processor())
.writer(report())
.build();
}
#Bean
public ItemWriter<VerifiedAffiliateEmailAddress> report(){
FlatFileItemWriter<VerifiedAffiliateEmailAddress> reportWriter = new FlatFileItemWriter<VerifiedAffiliateEmailAddress>();
reportWriter.setResource(new ClassPathResource("report.csv"));
DelimitedLineAggregator<VerifiedAffiliateEmailAddress> delLineAgg = new DelimitedLineAggregator<VerifiedAffiliateEmailAddress>();
delLineAgg.setDelimiter(",");
BeanWrapperFieldExtractor<VerifiedAffiliateEmailAddress> fieldExtractor = new BeanWrapperFieldExtractor<VerifiedAffiliateEmailAddress>();
fieldExtractor.setNames(new String[] {"uniekNr", "reason"});
delLineAgg.setFieldExtractor(fieldExtractor);
reportWriter.setLineAggregator(delLineAgg);
reportWriter.setShouldDeleteIfExists(true);
return reportWriter;
}
As described in the documentation I would expect the lifecycle events(open, close) are automatically taken care of since I am in a single threaded and single writer job?

To elaborate on the comment left, Spring Batch will register any ItemStream implementations automatically when it finds them so that they will be automatically opened when the step begins. When using java config, Spring only knows what the return type is. Since you are returning ItemReader, we don't know that your implementation also implements ItemStream. When using java config, I usually recommend returning the implementation if it's known (instead of the interface). That allows Spring to introspect it fully. So in this example, returning FlatFileItemReader instead of ItemReader will fix the issue.

Finally it turned out to be the Spring Dev Tools interfering with my batch application. The report.csv is in my classpath, when writing to the report file, Spring Dev Tools detects a change in my classpath and triggers a reload of the application causing the report.csv resource being closed...
spring.devtools.restart.enabled=false
in my application.properties made it work again