I need to insert a record in my database so another system can read that record. After that, I will write (with my ItemWriter) the response I recieve from that system in a CSV file.
My problem is that the system can't read the record since Spring Batch is transactional. How can I disable that property?
Disable spring batch transaction:
User .transactionAttribute(attributes()) method from AbstractTaskletStepBuilder with a parameter DefaultTransactionAttribute.
Build DefaultTransactionAttribute object with transaction propagation.
Example:
#Bean
public Step springBatchStep() {
return this.stepBuilderFactory.get("springBatchStep")
...
.reader()
.processor()
.writer()
.transactionAttribute(attributesForTransaction())
...
.build();
}
private DefaultTransactionAttribute attributesForTransaction() {
DefaultTransactionAttribute attribute = new DefaultTransactionAttribute();
attribute.setPropagationBehavior(Propagation.NOT_SUPPORTED.value());
attribute.setIsolationLevel(Isolation.DEFAULT.value());
return attribute;
}
You do not want to disable transaction enforcement, that can open you up to a serious data integrity problem. Even a 1% error issue can easily cause cause ten's of thousands, or even tens of millions of incomplete or bad records. Especially if say the one of the databases you are interacting with, or file systems you are writing to become unavailable. Which over time WILL happen. It would also break the job retry features.
A better option would be to break the process up into multiple steps, or jobs so the transaction boundaries fit your process. So one of them write out to this other database, and then the next step or job does the reading and writing to the file.
Spring Batch is highly opinionated Framework. You can split Step or do your job inside REQUIRES_NEW transaction. Or choose alternative less restrictive Framework.
Related
Good day,
I have a Spring batch that using different step, the Job is something like follow:
#Bean
public Job myJob() throws Exception {
return jobBuilderFactory.get("MyJob").repository(batchConfiguration.jobRepository())
.start(step1()).next(step2()).build();
}
In my step1(),it has its own reader, processor, and writer, in this writer, I will update table A.
And then in my step2(), it also has its own reader, processor, and writer. And this reader is read from table A, and by logic, it need to depends on the data update in table A.
However, when I run this batch job, I found that my step2() reader is actually selecting same data as step1(), anyway I can make the step1() writer to commit first, then my step2() reader read the updated data?
If you're using Spring Data, you can annotate your step1 as #Transactional(propagation = Propagation.REQUIRES_NEW), so the operation will always open a new transaction, execute and commit when the method finish. Another way if you are using Spring Data in recent version you can call saveAndFlush().
For hibernate/jdbc legacy you can simply open a connection do the first step and commit before initiate the step2 (session and EntityManager has flush method).
For more info:
https://www.baeldung.com/spring-transactional-propagation-isolation
What does EntityManager.flush do and why do I need to use it?
https://www.tutorialspoint.com/jdbc/commit-rollback.htm
In one of the steps of my Spring Batch job, I'm trying to configure it so that when ObjectOptimisticLockingFailureException happens, the step can be retried and hopefully the retry will work.
#Bean
public Step myStep(StaxEventItemReader<Response> staxEventResponseReader,
ItemWriter<Response> itemWriter,
ItemProcessor<? super Response, ? extends Response> responseProcessor) {
return stepBuilderFactory.get("myStep")
.<Response, Response>chunk(1)
.reader(staxEventResponseReader)
.processor(responseProcessor)
.writer(itemWriter)
//.faultTolerant().retryLimit(3).retry(Exception.class)
.build();
}
The logic of writer for the step is pretty simple: it tries to read a row from the database, and once it finds the row, it updates it. I was able to reproduce the ObjectOptimisticLockingFailureException by setting a breakpoint right after the find method, manually bump the version column for the row in database and commit it, then resume.
However, after uncommenting the retry definition in my step, no retries were attempted. After some debugging, it seems that the Spring retry logic is inside the chunk's transaction; but since the ObjectOptimisticLockingFailureException is not thrown by my code in the writer, but by Spring's chunk transaction committing logic, no retries were attempted at all.
Chunk Transaction Begin
Begin Retry loop in FaultTolerantChunkProcessor.write()
Writer logic in my Step
End Retry loop
Chunk Transaction Commit - Throws ObjectOptimisticLockingFailureException
When I tried to explicitly throw ObjectOptimisticLockingFailureException in my writer, the retry logic worked perfectly as expected. My questions are:
How to make the retry logic work if the exception is not thrown from my writer code in the step, but by the time the chunk transaction is committed by Spring Batch?
Another weird behavior is, when I manually cause the ObjectOptimisticLockingFailureException by bumping the version column in database, with the retry definition commented in the step, the final status of the step is FAILED which is expected. But with the retry definition uncommented, the final status of the step is COMPLETE. Why is that?
How to make the retry logic work if the exception is not thrown from my writer code in the step, but by the time the chunk transaction is committed by Spring Batch?
There is an open issue for that here: https://github.com/spring-projects/spring-batch/issues/1826. The workaround is to (try to anticipate and) throw any exception that might happen at the commit time in the writer. This is what you tried already and confirmed that works when you say When I tried to explicitly throw ObjectOptimisticLockingFailureException in my writer, the retry logic worked perfectly as expected.
Another weird behavior is, when I manually cause the ObjectOptimisticLockingFailureException by bumping the version column in database, with the retry definition commented in the step, the final status of the step is FAILED which is expected. But with the retry definition uncommented, the final status of the step is COMPLETE. Why is that?
This is related to the previous issue, but caused by a different one: https://github.com/spring-projects/spring-batch/issues/1189. That said, it is ok to play with the version field during a debugging session to understand how things work, but I would not recommend changing the version column in your code. Spring Batch relies on this column heavily in its optimistic locking strategy, and it is not expected to change values of this column in user code, otherwise unexpected behaviour might happen.
I have two data managers for two very unrelated tables. I want to perform an operation where I update both tables using the two data managers. However, if one of the transactions fails, I want to rollback both transactions. This can happen when say, first transaction succeeds , but second one fails. I want both the transactions to be rolled back.
I am using spring hibernateTemplate in my data managers due to legacy reasons.
This can be handled in Spring using #Transactional annotation. There are two ways to handle, one is Declarative transaction using annotation and another is programmatic transaction handling where you have to write a bit code for this.
If you are using legacy code, you can use programmatic transaction, I provide below snippet.
transactionTemplate.execute(new TransactionCallbackWithoutResult() {
protected void doInTransactionWithoutResult(TransactionStatus status) {
try {
updateOperation1();
updateOperation2();
} catch (SomeBusinessExeption ex) {
status.setRollbackOnly();
}
}
});
You can refer below the Spring documentation for programmatic transaction.
https://docs.spring.io/spring/docs/4.2.x/spring-framework-reference/html/transaction.html#transaction-programmatic
For complete example about it, you can refer below the links.
https://www.tutorialspoint.com/spring/programmatic_management.htm
https://www.baeldung.com/spring-programmatic-transaction-management
So I currently have a spring batch process that has a composite skip policy implemented for a few custom exception types. So the issue that I am now running into is the fact that I don't always just want to skip when I get an exception.
For some database related exceptions I would like to retry a few times and then if it still fails move on and skip the record. Unfortunately I don't see a way to do this.
I tried implementing my own RetryPolicy but the only option for canRetry is true or false (rather than false I would like to throw my skippable exception).
So am I missing something here or is this not really functionality that spring batch has?
Thanks
From a StepBuilderFactory, you can do that:
stepBuilder.reader(reader).writer(writer).faultTolerant().retryPolicy(retryPolicy).skipPolicy(skipPolicy).
And yes it is working. I had the same issue and after test, I see that my items are retried following my RetryPolicy and then they are skipped following my SkipPolicy.
I'm using Hibernate and Spring (in mode OpenSessionInView) in a webapp.
I've introduced Quartz for scheduling Job for checks.
To be able to inject beans into the Job I make use of the following approach: https://gist.github.com/jelies/5085593
A job in my app is like:
#Service
public class SampleJob implements Job {
#Autowired
private SampleBusiness sampleBusiness;
#Override
public void execute(JobExecutionContext jobExecutionContext) throws JobExecutionException {
for(Student student : sampleBusiness.getStudents()) {
System.out.println(student.getName());
}
}
}
When the webapp starts, and at the first execution of the Job the result is:
Tom
Bruce
Steven
The result is ok.
After that I change in the webapp the name of "Tom" in "Tommy".
On the next execution of the Job the result is:
Tom
Bruce
Steven
Why Hibernate session holds the old data?
I think I have to attach to the Job an Interceptor like I do for my other beans, but I'm not able to attach it to the Job.
Thanks
The possible reason for the above mentioned situation might be the cache oh the hibernate.
Hibernate uses two levels of caching, level 1 caching and level 2 caching.
The level 1 caching caches the data that is read frequently, but in case data is not present in hibernate cache it then looks for the same in the level 2 cache.
If the data is present in the cache level2, then it returns the data, if the data is not present here also it then actually hits the database to retrieve the data.
You should try to clear the cache data before hitting the same method.
Although whenever you update a record from the cached data, this data is re-cached. but in case you are trying to update data from other thread or session, it might not get reflected in the first thread, as the data is already present in the level 1 cache of this session.
I hope this drives you to the right direction.