We Keep Coding

Spring Boot Batch Automatically run instead of sending a request. How can I fix it?

I have an issue regarding running batch process through a request instead of run it automatically when the application runs.
Normally, when I start the application, batch process automatically is handled and it allows to save all values into database. I don't want to do that.
After I want to make a request defined in controller, batch process will starts. It is what I really want to do.
How can I do that?
Here is the project link : Link
Here is the BatchConfiguration class shown below.
#Configuration // Informs Spring that this class contains configurations
#EnableBatchProcessing // Enables batch processing for the application
#RequiredArgsConstructor
public class BatchConfiguration {
private final JobBuilderFactory jobBuilderFactory;
private final StepBuilderFactory stepBuilderFactory;
private final UserRepository userRepository;
#Bean
public FlatFileItemReader<UserInput> reader() {
FlatFileItemReader<UserInput> itemReader = new FlatFileItemReader<>();
itemReader.setResource(new FileSystemResource("src/main/resources/MOCK_DATA.csv"));
itemReader.setName("csvReader");
itemReader.setLinesToSkip(1);
itemReader.setLineMapper(lineMapper());
return itemReader;
}
private LineMapper<UserInput> lineMapper() {
DefaultLineMapper<UserInput> lineMapper = new DefaultLineMapper<>();
DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
lineTokenizer.setDelimiter(",");
lineTokenizer.setStrict(false);
lineTokenizer.setNames(
"personId","firstName","lastName","email","gender","birthday","country"
);
BeanWrapperFieldSetMapper<UserInput> fieldSetMapper = new BeanWrapperFieldSetMapper<>();
fieldSetMapper.setTargetType(UserInput.class);
lineMapper.setLineTokenizer(lineTokenizer);
lineMapper.setFieldSetMapper(fieldSetMapper);
return lineMapper;
}
#Bean
public UserProcessor processor() {
return new UserProcessor();
}
#Bean
public RepositoryItemWriter<User> writer() {
RepositoryItemWriter<User> writer = new RepositoryItemWriter<>();
writer.setRepository(userRepository);
writer.setMethodName("save");
return writer;
}
#Bean
public Step step1() {
return stepBuilderFactory.get("csv-step").<UserInput, User>chunk(10)
.reader(reader())
.processor(processor())
.writer(writer())
.listener(stepExecutionListener())
.taskExecutor(taskExecutor())
.build();
}
#Bean
public Job runJob() {
return jobBuilderFactory.get("importuserjob")
.listener(jobExecutionListener())
.flow(step1()).end().build();
}
#Bean
public TaskExecutor taskExecutor() {
SimpleAsyncTaskExecutor asyncTaskExecutor = new SimpleAsyncTaskExecutor();
asyncTaskExecutor.setConcurrencyLimit(10);
return asyncTaskExecutor;
}
#Bean
public UserJobExecutionNotificationListener stepExecutionListener() {
return new UserJobExecutionNotificationListener(userRepository);
}
#Bean
public UserStepCompleteNotificationListener jobExecutionListener() {
return new UserStepCompleteNotificationListener();
}
}
Here is the controller class shown below.
public class BatchController {
private final JobLauncher jobLauncher;
private final Job job;
#PostMapping("/importuserjob")
public ResponseEntity<String> importCsvToDBJob() {
log.info("BatchController | importCsvToDBJob is called");
JobParameters jobParameters = new JobParametersBuilder()
.addLong("startAt", System.currentTimeMillis()).toJobParameters();
try {
jobLauncher.run(job, jobParameters);
} catch (JobExecutionAlreadyRunningException | JobRestartException |
JobInstanceAlreadyCompleteException | JobParametersInvalidException e) {
log.info("BatchController | importCsvToDBJob | error : " + e.getMessage());
e.printStackTrace();
}
return new ResponseEntity<>("Batch Process started!!", HttpStatus.OK);
}
}

To stop the job running automatically set the following config flag
spring:
batch:
job:
enabled: false
you can always set the value at launch if you are running a jar
java -Dspring.batch.job.enabled=false -jar myapp.jar
Not sure if it's just a cut and paste issue, but you are missing a constructor in your controller which is required to set final variables, joblauncher and job. Also I assume your controller has #Controller & #Slf4j annotations

how to connect 2 mongodb databases in spring mvc

Hi I have this config file to connect to my remote mongodb service now I need to connect 1 more db simultaneously but I don't know how can anyone help?
#Configuration
public class MongoConfig{
#Bean
public MongoDbFactory mongoDbFactory(){
MongoClientURI uri = new MongoClientURI("mongodb://user:pass#localhost:27017/?authSource=admin");
return new SimpleMongoDbFactory(new MongoClient(uri), "DataBuild");
}
#Bean
public MongoOperations mongoOperations(){
return new MongoTemplate(mongoDbFactory());
}
#Bean
MongoTransactionManager transactionManager(MongoDbFactory dbFactory) {
return new MongoTransactionManager(dbFactory);
}
}

Take a look in this tutorial. Here is the #Configuration class they use:
#Configuration
public class MultipleMongoConfig {
#Primary
#Bean(name = "newdb1Properties")
#ConfigurationProperties(prefix = "spring.data.mongodb.newdb1")
public MongoProperties getNewDb1Props() throws Exception {
return new MongoProperties();
}
#Bean(name = "newdb2Properties")
#ConfigurationProperties(prefix = "spring.data.mongodb.newdb2")
public MongoProperties getNewDb2Props() throws Exception {
return new MongoProperties();
}
#Primary
#Bean(name = "newdb1MongoTemplate")
public MongoTemplate newdb1MongoTemplate() throws Exception {
return new MongoTemplate(newdb1MongoDatabaseFactory(getNewDb1Props()));
}
#Bean(name ="newdb2MongoTemplate")
public MongoTemplate newdb2MongoTemplate() throws Exception {
return new MongoTemplate(newdb2MongoDatabaseFactory(getNewDb2Props()));
}
#Primary
#Bean
public MongoDatabaseFactory newdb1MongoDatabaseFactory(MongoProperties mongo) throws Exception {
return new SimpleMongoClientDatabaseFactory(
mongo.getUri()
);
}
#Bean
public MongoDatabaseFactory newdb2MongoDatabaseFactory(MongoProperties mongo) throws Exception {
return new SimpleMongoClientDatabaseFactory(
mongo.getUri()
);
}
}

Springbatch is unable to detect database type for mysql

I am using springboot 2.6.6 with mysql and quite new to springbatch. I was getting an error "spring context is forming a cycle" when I kept the datasource in the same config file where readers and writers are kept in SampleJob.java. Suggested solution was to put the datasource in another class so I placed the datasource within the same class as main().
Now I am getting this issue:
Error
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.autoconfigure.batch.BatchDataSourceScriptDatabaseInitializer]: Factory method 'batchDataSourceInitializer' threw exception; nested exception is java.lang.IllegalStateException: Unable to detect database type
Caused by: java.lang.IllegalStateException: Unable to detect database type
SampleJob.java
#Configuration
public class SampleJob {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Bean
public Job chunkJob() {
return jobBuilderFactory.get("Chunk Job")
.incrementer(new RunIdIncrementer())
.start(firstChunkStep())
.build();
}
private Step firstChunkStep() {
return stepBuilderFactory.get("First Chunk Step")
.<StudentJdbc, StudentJdbc>chunk(3)
.reader(jdbcItemReader())
.writer(flatFileItemWriter(null))
.build();
}
#Bean
#StepScope
public JdbcCursorItemReader<StudentJdbc> jdbcItemReader(){
JdbcCursorItemReader<StudentJdbc> jdbcReader = new JdbcCursorItemReader<StudentJdbc>();
jdbcReader.setSql("select id, first_name as firstName, last_name as lastName, email from students");
jdbcReader.setDataSource(universitydatasource());
jdbcReader.setRowMapper(new BeanPropertyRowMapper<StudentJdbc>() {
{
setMappedClass(StudentJdbc.class);
}
});
return jdbcReader;
}
#StepScope
#Bean
public FlatFileItemWriter<StudentJdbc> flatFileItemWriter(
#Value("#{jobParameters['outputFile']}") FileSystemResource fileSystemResource
){
FlatFileItemWriter<StudentJdbc> flatFileItemWriter = new FlatFileItemWriter<StudentJdbc>();
flatFileItemWriter.setResource(fileSystemResource);
flatFileItemWriter.setResource(fileSystemResource);
flatFileItemWriter.setHeaderCallback(new FlatFileHeaderCallback() {
#Override
public void writeHeader(Writer writer) throws IOException {
writer.write("Id, First Name, Last Name, Email");
}
});
flatFileItemWriter.setLineAggregator(new DelimitedLineAggregator(){
{
setFieldExtractor(new BeanWrapperFieldExtractor<StudentResponse>() {
{
setNames(new String[] {"id","firstName","lastName","email"});
}
});
}
});
flatFileItemWriter.setFooterCallback(new FlatFileFooterCallback() {
#Override
public void writeFooter(Writer writer) throws IOException {
writer.write("Created # "+new Date());
}
});
return flatFileItemWriter;
}
}
Main class file
#SpringBootApplication
#EnableBatchProcessing
public class ChunksApplication {
public static void main(String[] args) {
SpringApplication.run(ChunksApplication.class, args);
}
#Bean
#Primary
#ConfigurationProperties(prefix="spring.datasource")
public DataSource datasource() {
return DataSourceBuilder.create().build();
}
#Bean
#ConfigurationProperties(prefix="spring.universitydatasource")
public DataSource universitydatasource() {
return DataSourceBuilder.create().build();
}
}
application.properties
spring.datasource.url=jdbc:mysql://localhost:3306/udemy-springbatch-chunks
spring.datasource.username=____
spring.datasource.password=____
spring.datasource.platform=mysql
spring.datasource.driverClassName=com.mysql.cj.jdbc.Driver
#alternate datasource for db input for reader
spring.universitydatasource.url=jdbc:mysql://localhost:3306/university?createDatabaseIfNotExist=true
spring.universitydatasource.username=____
spring.universitydatasource.password=____
spring.universitydatasource.platform=mysql
spring.universitydatasource.driverClassName=com.mysql.cj.jdbc.Driver

Found the solution. It seems that #ConfigurationProperties weren't picking up the url, username and password from application.properties so this worked in the main class file itself:
#Bean
#Primary
public DataSource datasource() {
DataSourceBuilder dataSourceBuilder = DataSourceBuilder.create();
dataSourceBuilder.url("jdbc:mysql://localhost:3306/udemy-springbatch-chunks");
dataSourceBuilder.username("____");
dataSourceBuilder.password("____");
return dataSourceBuilder.build();
}
#Bean("universityDatasource")
public DataSource universitydatasource() {
DataSourceBuilder dataSourceBuilder = DataSourceBuilder.create();
dataSourceBuilder.url("jdbc:mysql://localhost:3306/university?createDatabaseIfNotExist=true");
dataSourceBuilder.username("____");
dataSourceBuilder.password("____");
return dataSourceBuilder.build();
}

It works for me
spring.batch.schema=classpath:org/springframework/batch/core/schema-mysql.sql

My solution was this, you have to add the jdbcUrl
Properties file example:
spring.datasource.url=jdbc:mysql://localhost:3306/udemy-springbatch-chunks
spring.datasource.jdbcUrl=${spring.datasource.url}
spring.datasource.username=____
spring.datasource.password=____
spring.datasource.platform=mysql
spring.datasource.driverClassName=com.mysql.cj.jdbc.Driver
#alternate datasource for db input for reader
spring.universitydatasource.url=jdbc:mysql://localhost:3306/university
spring.universitydatasource.jdbcUrl=${spring.universitydatasource.url}
spring.universitydatasource.username=____
spring.universitydatasource.password=____
spring.universitydatasource.platform=mysql
spring.universitydatasource.driverClassName=com.mysql.cj.jdbc.Driver
Regards.

Spring batch job runs automatically

I'm using a spring batch to read a CSV file and write it to the DB, using the controller trigger. On starting the application, before I hit from the browser url, I see the print statements from my reader, on the startup. Although it doesn't print it for my processor or writer, which are in separate classes which I have autowired. Is it because the reader is a bean?
I see the print statements from my FlatFileItemReader in the log on the application startup. But the print statements for my processor and writer only show up in the console when I hit the controller url.
I've tried adding spring.batch.job.enabled=false in the application.properties file, but it doesnt stop the execution of the reader bean. How can I prevent auto execution of the reader bean in the SpringBatchConfig class:
SpringBatchConfig class:
#Configuration
#EnableBatchProcessing
public class SpringBatchConfig {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private DataSource dataSource;
#Autowired
private DBWriter writer1;
#Autowired
private Processor processor1;
//Step 1 - CSV to DB
#Bean
public FlatFileItemReader<User> itemReader() {
FlatFileItemReader<User> flatFileItemReader = new FlatFileItemReader<>();
flatFileItemReader.setResource(new FileSystemResource("src/main/resources/users.csv"));
flatFileItemReader.setName("CSV-Reader");
flatFileItemReader.setLinesToSkip(1);
flatFileItemReader.setLineMapper(lineMapper());
System.out.println("inside file reader 1 !!!!!");
return flatFileItemReader;
}
#Bean
public LineMapper<User> lineMapper() {
DefaultLineMapper<User> defaultLineMapper = new DefaultLineMapper<>();
DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
lineTokenizer.setDelimiter(",");
lineTokenizer.setStrict(false);
lineTokenizer.setNames(new String[]{"id", "name", "dept", "salary"});
BeanWrapperFieldSetMapper<User> fieldSetMapper = new BeanWrapperFieldSetMapper<>();
fieldSetMapper.setTargetType(User.class);
defaultLineMapper.setLineTokenizer(lineTokenizer);
defaultLineMapper.setFieldSetMapper(fieldSetMapper);
return defaultLineMapper;
}
#Bean
public Step step1() throws Exception{ // Step 1 - Read CSV and Write to DB
return stepBuilderFactory.get("step1")
.<User,User>chunk(100)
.reader(itemReader())
.processor(processor1)
.writer(writer1)
.build();
}
#Bean
public Job job() throws Exception{
return this.jobBuilderFactory.get("BATCH JOB")
.incrementer(new RunIdIncrementer())
.start(step1())
.build();
}
DBWriter class:
#Component
public class DBWriter implements ItemWriter<User> {
#Autowired
private UserRepository userRepository;
#Override
public void write(List<? extends User> users) throws Exception {
System.out.println("Inside DB Writer");
System.out.println("Data Saved for Users: " + users);
userRepository.save(users);
}
}
Processor class:
#Component
public class Processor implements ItemProcessor<User, User> {
private static final Map<String, String> DEPT_NAMES =
new HashMap<>();
public Processor() {
DEPT_NAMES.put("001", "Technology");
DEPT_NAMES.put("002", "Operations");
DEPT_NAMES.put("003", "Accounts");
}
#Override
public User process(User user) throws Exception {
String deptCode = user.getDept();
String dept = DEPT_NAMES.get(deptCode);
user.setDept(dept);
user.setTime(new Date());
System.out.println(String.format("Converted from [%s] to [%s]", deptCode, dept));
return user;
}
}
Controller Class:
#RestController
#RequestMapping("/load")
public class LoadController {
#Autowired
JobLauncher jobLauncher;
#Autowired
Job job;
#GetMapping("/users")
public BatchStatus load() throws JobParametersInvalidException, JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException {
Map<String, JobParameter> maps = new HashMap<>();
maps.put("time", new JobParameter(System.currentTimeMillis()));
JobParameters parameters = new JobParameters(maps);
JobExecution jobExecution = jobLauncher.run(job, parameters);
System.out.println("JobExecution: " + jobExecution.getStatus());
System.out.println("Batch is Running...");
while (jobExecution.isRunning()) {
System.out.println("...");
}
return jobExecution.getStatus();
}
}

The spring.batch.job.enabled=false property is used to prevent running jobs at application startup.
The method that creates the reader will be still be called at configuration time, so it's normal that you see the print statement. But that does not mean the reader was called inside a running job.

Spring batch Jobparamaters with file name

I want to create project with spring batch rest controller and dynamic input filename.
My code : Rest Controller
#RestController
public class FileNameController {
#Autowired
JobLauncher jobLauncher;
#Autowired
Job job;
#RequestMapping("/launchjob")
public String handle(#RequestParam("fileName") String fileName) throws Exception {
Logger logger = LoggerFactory.getLogger(this.getClass());
try {
JobParameters jobParameters = new JobParametersBuilder()
.addString("input.file.name", fileName)
.addLong("time", System.currentTimeMillis())
.toJobParameters();
jobLauncher.run(job, jobParameters);
} catch (Exception e) {
logger.info(e.getMessage());
}
return "Done";
}
}
The Job config:
#Configuration
#EnableBatchProcessing
public class JobConfig {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
Filetasklet ft=new Filetasklet();
Logger log = LoggerFactory.getLogger(this.getClass().getName());
private String pathFile = urlCoffreFort + "\\" + ft.getFileName();
// => Configuration of Job
#Bean
public Job job() throws IOException {
return jobBuilderFactory.get("job")
.incrementer(new RunIdIncrementer())
.flow(step1())
.end()
.build();
}
//###### Steps
// => Step cecStep1
#Bean
public Step step1() throws IOException {
return stepBuilderFactory.get("fileDecrypt")
.<Person, String>chunk(100)
.reader(reader1())
.processor(processor1FileDecrypt())
.writer(writer1())
.faultTolerant()
.skip(Exception.class)
.skipLimit(100)
.build();
}
// ####### readers
// => reader1()
#Bean
public FlatFileItemReader<Person> reader1() throws IOException{
return new FlatFileItemReaderBuilder<CSCivique>().name("personItemReader")
.resource(new ClassPathResource(pathFile))
.delimited()
.delimiter(";")
.names(new String[] { "id", "nomNaissance", "prenom" })
.targetType(CSCivique.class)
.build();
}
// ######Processors
#Bean
public PersonItemProcessor1FileDecrypt processor1FileDecrypt() {
return new PersonItemProcessor1FileDecrypt();
}
// ######Writers
#Bean
public FlatFileItemWriter<String> writer1() {
return new FlatFileItemWriterBuilder<String>().name("greetingItemWriter")
.resource(new FileSystemResource("sav/greetings.csv"))
.lineAggregator(new PassThroughLineAggregator<>()).build();
}
}
When I write the url :
http://localhost:8080/launchjob?fileName=djecc5cpt.csv
The console print :
PreparedStatementCallback; bad SQL grammar [SELECT JOB_INSTANCE_ID, JOB_NAME from BATCH_JOB_INSTANCE where JOB_NAME =? and JOB_KEY =?]; nested exception is org.postgresql.util.PSQLException: ERROR: the relation "batch_job_instance" does not exist
   Position: 39

I do not have a request, normally the framework will create its tables
Spring Batch won't make the decision to create tables on your behalf in your production database. You need to make that decision and do it manually upfront. Otherwise, if you use Spring Boot, you can tell Spring Boot to do it for your by setting spring.batch.initialize-schema=always.
Please see https://stackoverflow.com/a/51891852/5019386 for a similar question/answer.