So I currently have a spring batch process that has a composite skip policy implemented for a few custom exception types. So the issue that I am now running into is the fact that I don't always just want to skip when I get an exception.
For some database related exceptions I would like to retry a few times and then if it still fails move on and skip the record. Unfortunately I don't see a way to do this.
I tried implementing my own RetryPolicy but the only option for canRetry is true or false (rather than false I would like to throw my skippable exception).
So am I missing something here or is this not really functionality that spring batch has?
Thanks
From a StepBuilderFactory, you can do that:
stepBuilder.reader(reader).writer(writer).faultTolerant().retryPolicy(retryPolicy).skipPolicy(skipPolicy).
And yes it is working. I had the same issue and after test, I see that my items are retried following my RetryPolicy and then they are skipped following my SkipPolicy.
Related
I'm not adding a retry against EJB exception in listener but somehow I m thinking that retry is happening. How can I be sure if retry is happening or not. Note: I'm using spring.retry.RetryTemplate
I am using MANUAL_IMMEDIATE ack mode, Spring-kafka 1.3.9 (Can't change to Java 8), and committing offsets when processing is complete in Listener code. I am using Custom deserializer and its working fine unless I encounter Deserialization exception. Then onwards kafka gets stuck. I have handled this in by Deserializer, like instead of throwing the exception(When deserialization exception happens) I get a new instance of deserialized object and set the original message (which caused the deserialization exception) in a field(exceptionData) and let it pass to the Listener code(annotated with #KafkaListener).
In the Listener code I check if the field exceptionData is NULL or not and accordingly decide 'Good Message' or 'Bad Message' , Because I need to handle 'Bad Messages' too.
Now I want to know is there any way to handle all these from container level. Where I stuck is I don't get the Consumer reference in this version of Spring-kafka 1.3.9. I need to commit offset on error, any error.
No; it is not possible to get the consumer with that version. The ConsumerAwareErrorHandler was added in 2.0.
In one of the steps of my Spring Batch job, I'm trying to configure it so that when ObjectOptimisticLockingFailureException happens, the step can be retried and hopefully the retry will work.
#Bean
public Step myStep(StaxEventItemReader<Response> staxEventResponseReader,
ItemWriter<Response> itemWriter,
ItemProcessor<? super Response, ? extends Response> responseProcessor) {
return stepBuilderFactory.get("myStep")
.<Response, Response>chunk(1)
.reader(staxEventResponseReader)
.processor(responseProcessor)
.writer(itemWriter)
//.faultTolerant().retryLimit(3).retry(Exception.class)
.build();
}
The logic of writer for the step is pretty simple: it tries to read a row from the database, and once it finds the row, it updates it. I was able to reproduce the ObjectOptimisticLockingFailureException by setting a breakpoint right after the find method, manually bump the version column for the row in database and commit it, then resume.
However, after uncommenting the retry definition in my step, no retries were attempted. After some debugging, it seems that the Spring retry logic is inside the chunk's transaction; but since the ObjectOptimisticLockingFailureException is not thrown by my code in the writer, but by Spring's chunk transaction committing logic, no retries were attempted at all.
Chunk Transaction Begin
Begin Retry loop in FaultTolerantChunkProcessor.write()
Writer logic in my Step
End Retry loop
Chunk Transaction Commit - Throws ObjectOptimisticLockingFailureException
When I tried to explicitly throw ObjectOptimisticLockingFailureException in my writer, the retry logic worked perfectly as expected. My questions are:
How to make the retry logic work if the exception is not thrown from my writer code in the step, but by the time the chunk transaction is committed by Spring Batch?
Another weird behavior is, when I manually cause the ObjectOptimisticLockingFailureException by bumping the version column in database, with the retry definition commented in the step, the final status of the step is FAILED which is expected. But with the retry definition uncommented, the final status of the step is COMPLETE. Why is that?
How to make the retry logic work if the exception is not thrown from my writer code in the step, but by the time the chunk transaction is committed by Spring Batch?
There is an open issue for that here: https://github.com/spring-projects/spring-batch/issues/1826. The workaround is to (try to anticipate and) throw any exception that might happen at the commit time in the writer. This is what you tried already and confirmed that works when you say When I tried to explicitly throw ObjectOptimisticLockingFailureException in my writer, the retry logic worked perfectly as expected.
Another weird behavior is, when I manually cause the ObjectOptimisticLockingFailureException by bumping the version column in database, with the retry definition commented in the step, the final status of the step is FAILED which is expected. But with the retry definition uncommented, the final status of the step is COMPLETE. Why is that?
This is related to the previous issue, but caused by a different one: https://github.com/spring-projects/spring-batch/issues/1189. That said, it is ok to play with the version field during a debugging session to understand how things work, but I would not recommend changing the version column in your code. Spring Batch relies on this column heavily in its optimistic locking strategy, and it is not expected to change values of this column in user code, otherwise unexpected behaviour might happen.
I need to insert a record in my database so another system can read that record. After that, I will write (with my ItemWriter) the response I recieve from that system in a CSV file.
My problem is that the system can't read the record since Spring Batch is transactional. How can I disable that property?
Disable spring batch transaction:
User .transactionAttribute(attributes()) method from AbstractTaskletStepBuilder with a parameter DefaultTransactionAttribute.
Build DefaultTransactionAttribute object with transaction propagation.
Example:
#Bean
public Step springBatchStep() {
return this.stepBuilderFactory.get("springBatchStep")
...
.reader()
.processor()
.writer()
.transactionAttribute(attributesForTransaction())
...
.build();
}
private DefaultTransactionAttribute attributesForTransaction() {
DefaultTransactionAttribute attribute = new DefaultTransactionAttribute();
attribute.setPropagationBehavior(Propagation.NOT_SUPPORTED.value());
attribute.setIsolationLevel(Isolation.DEFAULT.value());
return attribute;
}
You do not want to disable transaction enforcement, that can open you up to a serious data integrity problem. Even a 1% error issue can easily cause cause ten's of thousands, or even tens of millions of incomplete or bad records. Especially if say the one of the databases you are interacting with, or file systems you are writing to become unavailable. Which over time WILL happen. It would also break the job retry features.
A better option would be to break the process up into multiple steps, or jobs so the transaction boundaries fit your process. So one of them write out to this other database, and then the next step or job does the reading and writing to the file.
Spring Batch is highly opinionated Framework. You can split Step or do your job inside REQUIRES_NEW transaction. Or choose alternative less restrictive Framework.
We do use Spring Boot Health Checks in our application. On one of the checked applications it looks like the DB cannot answer in a timely manner. We are using the DataSourceHealthIndicator, and this is answering after several seconds with an exception which is fine, but takes different timeframes to come back.
Could we set a timeout on this HealthIndicator (and most probably on others as well), so that the HealthIndicator reports back after say 2s in the latest with an error, if no connection to the DB can be established?
Or could we ask the Datasource if there are still any open connections available?
I know, that we should fix the problem with these connections and we are already working on this one, but a HealthCheck for something like this would be nice as well.
You could disable the default db health indicator in your application.properties
management.health.db.enabled=false
and implement custom HealthIndicator that has desired behavior, as described here.
In your custom HealthIndicator implemetation you could use a different JdbcTemplatethat will have desired timeout value of 2 seconds, something like this:
JdbcTemplate jdbcTemplate = new JdbcTemplate(datasource);
jdbcTemplate.setQueryTimeout(2);
jdbcTemplate.execute(...);
If execute call throws an exception, your indicator should return Health.down(), otherwise Health.up() should be returned.