We Keep Coding

Loop Spring Batch

I have a simple job with only one step, but in some way the Batch loops from reader to processor and then to reader again. I can't understand why.
This is the structure:
The reader makes a double select on the same database. The first select needs to search in the first table some records in some state and the second select needs to match those results, get some records from the second table and send them to processor that call an api for every record.
I need to stop the batch running at this point, so after the processor. But I have some problems with this.
Example of my batch:
#Configuration
#EnableBatchProcessing
#EnableScheduling
public class LoadIdemOperationJob {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
public JobLauncher jobLauncher;
#Autowired
public JobRegistry jobRegistry;
#Scheduled(cron = "* */3 * * * *")
public void perform() throws Exception {
JobParameters jobParameters = new JobParametersBuilder()
.addString("JobID", String.valueOf(System.currentTimeMillis()))
.toJobParameters();
jobLauncher.run(jobRegistry.getJob("firstJob"), jobParameters);
}
#Bean
public Job firstJob(Step firstStep) {
return jobBuilderFactory.get("firstJob")
.start(firstStep)
.build();
}
#Bean
public Step firstStep(MyReader reader,
MyProcessor processor) {
return stepBuilderFactory.get("firstStep")
.<List<String>, List<String>>chunk(1)
.reader(reader)
.processor(processor)
.writer(new NoOpItemWriter())
.build();
}
#Bean
#StepScope
public MyReader reader(#Value("${hours}") String hours) {
return new MyReader(hours);
}
#Bean
public MyProcessor processor() {
return new MyProcessor();
}
public static class NoOpItemWriter implements ItemWriter<Object> {
#Override
public void write(#NonNull List<?> items) {
}
}
#Bean
public JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor() {
JobRegistryBeanPostProcessor postProcessor = new JobRegistryBeanPostProcessor();
postProcessor.setJobRegistry(jobRegistry);
return postProcessor;
}
#Bean
public RequestContextListener requestContextListener() {
return new RequestContextListener();
}
}
Example of Reader:
public class MyReader implements ItemReader<List<String>> {
public String hours;
private List<String> results;
#Autowired
private JdbcTemplate jdbcTemplate;
public MyReader(String hours) {
this.hours = hours;
}
#Override
public List<String> read() throws Exception {
results = this.jdbcTemplate.queryForList(// 1^ query, String.class);
if (results.isEmpty()) {
return null;
}
List<String> results = this.jdbcTemplate.queryForList(// 2^ query, String.class);
if (results.isEmpty()) {
return null;
}
return results;
}
}
And Processor:
public class MyProcessor implements ItemProcessor<List<String>, List<String>> {
#Override
public List<String> process(#NonNull List<String> results) throws Exception {
results.forEach(result -> // calling service);
return null;
}
}
Thanks for help!

What you are seeing is the implementation of the chunk-oriented processing model of Spring Batch, where items are read and processed in sequence one by one, and written in chunks.
That said, the design and configuration of your chunk-oriented step is not ideal: the reader returns a List of Strings (so an item in your case is the List itself, not an element from the list), the processor loops over the elements of each List (while it is not intended to do so), and finally there is no item writer (this is a sign that either you don't need a chunk-oriented step, or the step is not well designed).
I can recommend to modify your step design as follows:
The reader should return a single item and not a List. For example, by using the iterator of results and make the reader return iterator.next().
Remove the processor and move its code in the item writer. In fact, the item processor is optional in a chunk-oriented step
Create an item writer with the code of the item processor. Posting results to a REST endpoint is in fact a kind of write operation, so an item writer is definitely better suited than an item processor in this case.
With that design, you should see your chunk-oriented step reading and writing all items from your list without the impression that the job is "looping". This is actually the implementation of the pattern described above.

Spring boot mongodb transaction rollback

Here is the behavior I wish to achieve:
#Transactional(rollbackFor = NullPointerException.class)
#PostMapping(consumes = {"application/json"})
public Employee createEmployee(#Valid #RequestBody Employee employee) {
Optional<Department> departmentOptional = departmentRepository.findById(employee.getDeptId());
EmployeeAggregate employeeAggregate = employeeAggregateRepository.findAll().get(0);
employeeAggregate.setTotalEmployeeCount(employeeAggregate.getTotalEmployeeCount()+1);
employeeAggregateRepository.save(employeeAggregate); <--This line should be rolled back when exception is thrown.
if(departmentOptional.isPresent()) {
Department department = departmentOptional.get();
return employeeRepository.save(employee);
}
else{
throw new NullPointerException();
}
}
There are two documents, Employees, and EmployeeAggregate. I want to be able to rollback the employee count stored in EmployeeAggregate if I am unable to create an employee in Employees document.
Here is my mongo config file:
#Configuration
public class MongoConfig {
#Bean
#Autowired
#ConditionalOnExpression("'${mongo.transactions}'=='enabled'")
MongoTransactionManager mongoTransactionManager(MongoDbFactory dbFactory) {
return new MongoTransactionManager(dbFactory);
}
public #Bean MongoClient mongoClient() {
return new MongoClient("localhost");
}
public #Bean
MongoTemplate mongoTemplate() {
return new MongoTemplate(mongoClient(), "EmployeeDatabase");
}
}
Currently, no rollback takes place if Employee creation fails. The count gets updated by one irrespective of successful employee creation or not. An explanation of multidocument transactions in spring boot would be appreciated, since the documentation is a bit confusing.

Transaction partially committing or rolling back

I am facing some issue with transactions configured in my code.
Below is the code with transactions which writes data to DB.
Writer.java
class Writer {
#Inject
private SomeDAO someDAO;
#Transactional(propagation = Propagation.REQUIRES_NEW)
public void write(){
this.batchWrite();
}
private void batchWrite () {
try {
someDAO.writeToTable1(true);
} catch(Exception ex) {
someDAO.writeToTable1(false);
}
someDAO.writeToTable2();
}
}
SomeDAO.java
class SomeDAO {
#Inject
private JdbcTemplate JdbcTemplate;
public void writeToTable1(boolean flag) {
// Writes data to table 1 using jdbcTemplate
jdbcTemplate.update();
}
pulic void writeToTable2() {
// Writes data to table 2 using jdbcTemplate
jdbcTemplate.update();
}
}
Here data is getting stored into table 1 properly but sometimes, table 2 is getting skipped.
I am not sure how this is happening as both the tables have been written within same transaction.
Either transaction is partially committing the data or partially rolling back.
I have doubt that in the SomeDAO class I am injecting JdbcTemplate object which is creating new connection instead of using existing connection of transaction.
Can anyone please help me here?

Try binding a Transaction Manager bean having your jdbcTemplate inside #Transactional:
//Create a bean
#Bean
public PlatformTransactionManager txnManager() throws Exception {
return new DataSourceTransactionManager(jdbcTemplate().getDataSource());
}
And then use this transaction manager in #Transactional("txnManager").

JdbcTemplate insert success but no rows in database

This question was asked a couple of years ago, but the answer didn't work for me. I have added the suggested annotations to the config and to the dao. I am sure that Template is actually connecting to the DB because I was getting appropriate errors when I had a column too small. The update call is doing a single row insert and it is returning 1 with no exceptions. Yet, when I check the database there is no data in it.
Any help would be appreciated.
Config:
#Configuration
#EnableTransactionManagement
#ComponentScan("com.XXX.query.repository")
public class SqlConfig {
#Autowired
private Environment env;
#Bean
public DataSource getDatasource() {
DriverManagerDataSource datasource = new DriverManagerDataSource();
datasource.setDriverClassName(env.getProperty("sql-datasource.driverClassName"));
datasource.setUrl(env.getProperty("sql-datasource.url"));
datasource.setUsername(env.getProperty("sql-datasource.username"));
datasource.setPassword(env.getProperty("sql-datasource.password"));
return datasource;
}
#Bean
public JdbcTemplate jdbcTemplate() {
return new JdbcTemplate(getDatasource());
}
DAO:
#Repository
public class SqlRepositoryImpl implements SqlRepository {
#Autowired
private JdbcTemplate template;
<snip>
#Transactional
#Override
public void addOrder(String foo, String bar, String bat, String cat) {
int i;
try {
// we may already have this data
System.out.println("here");
i = template.update(
"insert into someTable "
+ "(A, B, C, D)"
+ " values (?,?,?,?)",
foo, bar, bat, cat);
} catch (DataAccessException ex) {
checkForDupeKey(ex);
}
<snip>

There was a stupid mistake in the area that threw an exception and unwound the transaction. For some reason that exception never bubbled to the surface so I wasn't aware of it until I stepped through everything one line at a time. My apologies for wasting people's time.

Spring Batch multiple insert for a one read

I've a Spring Batch process that read Report objects from a CSV and insert Analytic objects into a MySQL DB correctly, but the logical has changed for a more than one Analytics insert for each Report readed.
I'm new in Spring Batch and the actually process was very difficult for me, and I don't know how to do this change.
I haven't XML configuration, all is with annotations. Report and Analytics classes have a getter and a setter for two fields, adId and value. The new logic has seven values for an adId and I need to insert seven rows into table.
I hide, delete or supress some code that not contribute for the question.
Here is my BatchConfiguration.java:
#Configuration
#EnableBatchProcessingpublic
class BatchConfiguration {
#Autowired
private transient JobBuilderFactory jobBuilderFactory;
#Autowired
private transient StepBuilderFactory stepBuilderFactory;
#Autowired
private transient DataSource dataSource;
public FlatFileItemReader<Report> reader() {
// The reader from the CSV works fine.
}
#Bean
public JdbcBatchItemWriter<Analytic> writer() {
final JdbcBatchItemWriter<Analytic> writer = new JdbcBatchItemWriter<Analytic>();
writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<Analytic>());
writer.setSql("INSERT INTO TABLE (ad_id, value) VALUES (:adId, :value)");
writer.setDataSource(dataSource);
return writer;
}
#Bean
public AnalyticItemProcessor processor() {
return new AnalyticItemProcessor();
}
#Bean
public Step step() {
return stepBuilderFactory.get("step1").<Report, Analytic> chunk(10000).reader(reader()).processor(processor()).writer(writer()).build();
}
#Bean
public Job process() {
final JobBuilder jobBuilder = jobBuilderFactory.get("process");
return jobBuilder.start(step()).build();
}
}
Then the AnalyticItemProcessor.java
public class AnalyticItemProcessor implements ItemProcessor<Report, Analytic> {
#Override
public Analytic process(final Report report) {
// Creates a new Analytic call BeanUtils.copyProperties(report, analytic) and returns analytic.
}
}
And the Process:
#SpringBootApplication
public class Process {
public static void main(String[] args) throws Exception {
SpringApplication.run(Process.class, args);
}
}
How can I do this change? Maybe with ItemPreparedStatementSetter or ItemSqlParameterSourceProvider? Thanks.

If I'm understanding your question correctly, you can use the CompositeItemWriter to wrap multiple JdbcBatchItemWriter instances (one per insert you need to accomplish). That would allow you to insert multiple rows per item. Otherwise, you'd need to write your own ItemWriter implementation.