Basically the title. I am using a JDBC item reader and JDBC item writer and I'm changing a particular status through the processor by using an API, if the API fails to change the status, I want to use exponential backoff to retry this at a later instant. I'm not able to figure out how to implement this
You have two options:
1. Handle the retry operation manually in your item processor
There are basically two ways to do that, either programmatically or in a declarative way.
1.1 Programmatic approach
You first define your retry template with a backoff policy as needed:
#Bean
public RetryTemplate retryTemplate() {
// configure backoff policy
ExponentialBackOffPolicy exponentialBackOffPolicy = new ExponentialBackOffPolicy();
exponentialBackOffPolicy.setInitialInterval(1000);
exponentialBackOffPolicy.setMultiplier(2.0);
exponentialBackOffPolicy.setMaxInterval(10000);
// configure retry policy
SimpleRetryPolicy simpleRetryPolicy = new SimpleRetryPolicy();
simpleRetryPolicy.setMaxAttempts(5);
// configure retry template
RetryTemplate retryTemplate = new RetryTemplate();
retryTemplate.setBackOffPolicy(exponentialBackOffPolicy);
retryTemplate.setRetryPolicy(simpleRetryPolicy);
return retryTemplate;
}
Then use that retry template in your item processor:
import org.springframework.batch.item.ItemProcessor;
import org.springframework.retry.RetryCallback;
import org.springframework.retry.RetryContext;
import org.springframework.retry.support.RetryTemplate;
public class MyRetryableItemProcessor implements ItemProcessor {
RetryTemplate retryTemplate;
public MyRetryableItemProcessor(RetryTemplate retryTemplate) {
this.retryTemplate = retryTemplate;
}
#Override
public Object process(Object item) throws Exception {
return retryTemplate.execute(new RetryCallback<Object, Exception>() {
#Override
public Object doWithRetry(RetryContext retryContext) throws Exception {
// API call
return item;
}
});
}
}
1.2 Declarative approach using annotations
Here is an example:
import org.springframework.batch.item.ItemProcessor;
import org.springframework.retry.RetryCallback;
import org.springframework.retry.RetryContext;
import org.springframework.retry.annotation.Backoff;
import org.springframework.retry.annotation.Retryable;
import org.springframework.retry.support.RetryTemplate;
import org.springframework.stereotype.Component;
#Component
public class MyAnnotationBasedRetryableItemProcessor implements ItemProcessor {
#Override
#Retryable(backoff = #Backoff(delay = 1000L, maxDelay = 10000, multiplier = 2.0))
public Object process(Object item) throws Exception {
// Do API call
return item;
}
}
2. Let Spring Batch handle the retry for you by using a fault-tolerant step
In this case, you can set a custom RetryPolicy in your fault-tolerant step:
#Bean
public Step step(StepBuilderFactory stepBuilderFactory) {
// configure backoff policy
ExponentialBackOffPolicy exponentialBackOffPolicy = new ExponentialBackOffPolicy();
exponentialBackOffPolicy.setInitialInterval(1000);
exponentialBackOffPolicy.setMultiplier(2.0);
exponentialBackOffPolicy.setMaxInterval(10000);
// configure retry policy
SimpleRetryPolicy simpleRetryPolicy = new SimpleRetryPolicy();
simpleRetryPolicy.setMaxAttempts(5);
return stepBuilderFactory.get("myStep")
.<Integer, Integer>chunk(5)
.reader(itemReader())
.processor(itemProcessor())
.writer(itemWriter())
.faultTolerant()
.retryPolicy(simpleRetryPolicy)
.build();
}
Note that in this case, whenever your processor throws an exception for an item, the entire chunk is retried item by item (and each item will be re-processed in its own transaction).
The examples above use spring-retry since you mentioned you have a preference for that. But the same ideas can be applied with any other fault-tolerance library.
Related
I’ve a SQS queue which has max receives value of 3 & default visibility timeout of 30 seconds.
Currently I’m listening to its messages using annotation #SqsListener, which works fine.
Now I want to implement exponential backoff for retries on this queue.
The only pointer I got in this direction in AWS documentation is to use ClientConfiguration.
But I'm not able to find any exmample on how to use it.
I thought SimpleMessageListenerContainer might provide a setter to apply backoff strategy, but it just provides to add a timer.
If there any examples to guide how to add backoff in spring in SQS listener, that will be great.
Exponential backoff can be achieved in a Spring SQS listener by using a custom error handler in combination with the SimpleMessageListenerContainer.
#EnableScheduling
#Configuration
public class ExponentialBackoffSqsListener {
private QueueMessagingTemplate queueMessagingTemplate;
private String queueUrl;
#Autowired
public ExponentialBackoffSqsListener(AmazonSQSAsync amazonSqs, String queueUrl) {
this.queueMessagingTemplate = new QueueMessagingTemplate(amazonSqs);
this.queueUrl = queueUrl;
}
#SqsListener(value = "${queue.name}")
public void receiveMessage(String message) {
// Your business logic goes here
}
#Bean
public SimpleMessageListenerContainer messageListenerContainer() {
SimpleMessageListenerContainer listenerContainer = new SimpleMessageListenerContainer();
listenerContainer.setAmazonSqs(this.queueMessagingTemplate.getAmazonSqs());
listenerContainer.setErrorHandler(t -> {
if (RetryUtils.isRetryableServiceException(t)) {
RetryPolicy retryPolicy = RetryUtils.getDefaultRetryPolicy();
int backoffTime = retryPolicy.getBackoffStrategy().computeBackoffInMilliseconds(retryPolicy.getRetryCondition().getRetryCount());
// Schedule a retry for the failed message after the backoff time
scheduleRetry(backoffTime, message);
}
});
listenerContainer.setQueueUrl(this.queueUrl);
return listenerContainer;
}
private void scheduleRetry(int backoffTime, String message) {
// Schedule a retry using the #Scheduled annotation
new ScheduledThreadPoolExecutor(1).schedule(() -> {
this.queueMessagingTemplate.convertAndSend(this.queueUrl, message);
}, backoffTime, TimeUnit.MILLISECONDS);
}
}
I'm using Caffeine cache for Java Spring Boot com.github.ben-manes.caffeine:caffeine:3.1.1. This is my cache configuration class:
import com.github.benmanes.caffeine.cache.Caffeine;
import org.springframework.cache.CacheManager;
import org.springframework.cache.caffeine.CaffeineCacheManager;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.util.ArrayList;
import java.util.List;
#Configuration
public class CacheConfig {
#Bean
public Caffeine<Object, Object> caffeineConfig() {
return Caffeine.newBuilder()
.maximumSize(10000)
.recordStats()
.removalListener((key, value, removalCause) -> {
// TODO: log
});
}
#Bean
public CacheManager cacheManager(Caffeine<Object, Object> caffeine) {
CaffeineCacheManager caffeineCacheManager = new CaffeineCacheManager();
caffeineCacheManager.setCaffeine(caffeine);
List<String> list = new ArrayList<>();
list.add("AsyncTest.getStr");
list.add("AsyncTest.getStr2");
list.add("AsyncTest.getStr3");
list.add("AsyncTest.getStr4");
list.add("AsyncTest.getStr5");
caffeineCacheManager.setCacheNames(list);
return caffeineCacheManager;
}
}
I also have test methods with Cacheable annotation like this:
#Cacheable("AsyncTest.getStr")
public String getStr(int i) {
return "abc";
}
and finally I have a test controller which populates cache:
for (int i = 0; i < 1000; i++) {
asyncTest.getStr(i);
}
However, when I check Spring actuator metrics I only see one cache-related metric: size
"myapp.cache.size": 250.0,
"myapp.cache.size.tags": "{name=AsyncTest.getStr, cache=AsyncTest.getStr, cacheManager=cacheManager}",
However, why don't I see other metrics like cache.gets, cache.puts etc. as should be provided by Micrometer?
The metrics you are seeing are the common CacheManger metrics. To get Caffeine specific ones, you'll need to monitor those caches as they are created.
I've not run this code, but subclassing the CaffeineCacheManager and hooking into the createNativeCaffeineCache should do the trick:
public CacheManager cacheManager(Caffeine<Object, Object> caffeine, MeterRegistry meterRegistry) {
CaffeineCacheManager caffeineCacheManager = new CaffeineCacheManager(){
protected com.github.benmanes.caffeine.cache.Cache<Object, Object> createNativeCaffeineCache(String name) {
return CaffeineCacheMetrics.monitor(meterRegistry, super.createNativeCaffeineCache(name), name);
}
};
caffeineCacheManager.setCaffeine(caffeine);
}
There is likely a better way. But this should work.
I am new to spring-retry. Basically, for retrying calls to REST APIs, I have integrated spring-retry into my spring-boot application. To do this, I have made following changes:
Added spring-retry to pom.xml.
Added following configuration:
#Configuration
#EnableRetry
public class RetryConfiguration {
}
Finally added #Retryable annotation to the class (this class is not a Spring Bean) method that I would like to be retried for various exceptions as follows:
public class OAuth1RestClient extends OAuthRestClient {
#Override
#Retryable(maxAttempts = 3, value = {
Exception.class},
backoff = #Backoff(delay = 100, multiplier = 3))
public Response executeRequest(OAuthRequest request)
throws InterruptedException, ExecutionException, IOException {
System.out.println("Inside Oauth1 client");
return myService.execute(request);
}
Now, the executeRequest method is not retrying. I am not able to understand if I am missing anything here.
Could anyone please help? Thanks.
If your class is not Spring managed (e.g. #Component/#Bean) the
annotation processor for #Retryable won't pick it up.
You can always manually define a retryTemplate and wrap calls with it:
RetryTemplate.builder()
.maxAttempts(2)
.exponentialBackoff(100, 10, 1000)
.retryOn(RestClientException.class)
.traversingCauses()
.build();
and then
retryTemplate.execute(context -> myService.execute(request));
If you want to retry on multiple exception, this can happen via custom RetryPolicy
Map<Class(? extends Throwable), Boolean> exceptionsMap = new HashMap<>();
exceptionsMap.put(InternalServerError.class, true);
exceptionsMap.put(RestClientException.class, true);
SimpleRetryPolicy policy = new SimpleRetryPolicy(5, exceptionsMap, true);
RetryTemplate.builder()
.customPolicy(policy)
.exponentialBackoff(100, 10, 1000)
.build();
FYI: RetryTemplate is blocking and you might want to explore a non-blocking async retry approach like async-retry. - and the retryOn() supports a list of exceptions.
I have a RabbitMQ triggered Spring Batch application.
I need to nack the message on exception. However, Spring Batch doesn't throw Exceptions outside the run method. Thus, I need to nack the message in the SkipListener. I have managed to get the message tag as a JobParameter, but I am not able to send Channel as a JobParameter, since it is not Serializable.
Is there any way how to pass the Channel to the batch process or any other way of error handling ?
This is an example of what I have right now.
MessageConsumer.java
import com.project.common.dto.DataImportStartDto;
import com.rabbitmq.client.Channel;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.amqp.support.AmqpHeaders;
import org.springframework.messaging.handler.annotation.Header;
import org.springframework.stereotype.Service;
#Service
public class MessageConsumer {
private final DataImportService dataImportService;
public MessageConsumer(DataImportService dataImportService) {
this.dataImportService = dataImportService;
}
#RabbitListener(queues = "etl_queue", concurrency = "1", ackMode = "MANUAL")
public void receiveDataImportMessage(DataImportStartDto dataImportStartDto, Channel channel, #Header(AmqpHeaders.DELIVERY_TAG) long tag) throws Exception {
dataImportService.startImport(dataImportStartDto, tag);
channel.basicAck(tag, false);
}
DataImportService .java
import org.springframework.batch.core.*;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.stereotype.Service;
public class DataImportServiceImpl implements DataImportService {
private final JobLauncher jobLauncher;
private final Job dataImportJob;
#Service
public DataImportServiceImpl(JobLauncher jobLauncher, #Qualifier("dataImport") Job dataImportJob) {
this.jobLauncher = jobLauncher;
this.dataImportJob= dataImportJob;
}
#Override
public void startImport(DataImportStartDto dto, long tag) {
jobLauncher.run(dataImportJob, buildJobParameters(dto, tag));
}
}
private JobParameters buildJobParameters(DataImportStartDto dto, long tag) {
return new JobParametersBuilder()
.addString("unique", String.valueOf(UUID.randomUUID())) // needed if there are jobs with the same parameters running in parallel
.addLong("tag", tag)
.toJobParameters();
}
BatchExceptionHandler.java
import org.springframework.batch.core.SkipListener;
import org.springframework.batch.core.configuration.annotation.StepScope;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
#StepScope
#Component
public class BatchExceptionHandler implements SkipListener<IntervalDataWrapper<String>, IntervalDataWrapper<String>> {
#Value("#{jobParameters['tag']}")
private long tag; // this value is here on onSkipInRead execution
#Override
public void onSkipInRead(Throwable t) {
System.out.println(t);
// channel.basicNack(tag, false, true); // this is what I need to do here
}
#Override
public void onSkipInWrite(IntervalDataWrapper<String> item, Throwable t) {
System.out.println(t);
}
#Override
public void onSkipInProcess(IntervalDataWrapper<String> item, Throwable t) {
System.out.println(t);
}
}
IntervalDataWrapper is my custom data wrapper with batch configuration.
I believe you should not be doing that nack in the skip listener. Those calls are not at the same level of abstraction IMO. What I understand from your implementation is that you want to nack when there is an error on read. Your current setup is now:
#RabbitListener(queues = "etl_queue", concurrency = "1", ackMode = "MANUAL")
public void receiveDataImportMessage(DataImportStartDto dataImportStartDto, Channel channel, #Header(AmqpHeaders.DELIVERY_TAG) long tag) throws Exception {
// run a job
dataImportService.startImport(dataImportStartDto, tag); // send nack = true from within the job
// when the job is finished, send nack = false
channel.basicAck(tag, false);
}
So as you can see, one acknowledgment is sent from within the job and the other one is sent outside the job. What I would do is if there is a skippable exception, I would set the exit status of the job to something like FINISHED_WITH_SKIPS (or even fail it if needed) and then check the status of the job to know which type of acknowledgement should be sent to the channel:
#RabbitListener(queues = "etl_queue", concurrency = "1", ackMode = "MANUAL")
public void receiveDataImportMessage(DataImportStartDto dataImportStartDto, Channel channel, #Header(AmqpHeaders.DELIVERY_TAG) long tag) throws Exception {
JobExecution jobExecution = dataImportService.startImport(dataImportStartDto, tag);
if (jobExecution.getExitStatus().getExitCode().equals("FINISHED_WITH_SKIPS") {
channel.basicNack(tag, false, true);
} else {
channel.basicAck(tag, false);
};
}
This approach does not require the channel to be injected in the skip listener. Moreover, this approach keeps the batch processing logic separate from the messaging logic which is easier to test, deploy and think about.
Spring's #Retryable annotation will retry three times (default) and fallback to the #Recovery method. #CircuitBreaker however, will retry once and fall back when the state is closed.
I want to combine these two: when the circuit breaker state is closed, will retry three times before falling back (to deal with transient errors), if the state is open, will directly fall back.
Any elegant way to do this? A possible approach is to implement the retry logic inside the function, but I feel that it wouldn't be the best solution.
The #CircuitBreaker already implements #Retry as a stateful = true, that's how he knows how many calls failed.
I think the best approach here, would be use a RetryTemplate inside your method:
#CircuitBreaker(maxAttempts = 2, openTimeout = 5000l, resetTimeout = 10000l)
void call() {
retryTemplate.execute(new RetryCallback<Void, RuntimeException>() {
#Override
public Void doWithRetry(RetryContext context) {
myService.templateRetryService();
}
});
}
Declaring the RetryTemplate:
#Configuration
public class AppConfig {
#Bean
public RetryTemplate retryTemplate() {
RetryTemplate retryTemplate = new RetryTemplate();
FixedBackOffPolicy fixedBackOffPolicy = new FixedBackOffPolicy();
fixedBackOffPolicy.setBackOffPeriod(2000l);
retryTemplate.setBackOffPolicy(fixedBackOffPolicy);
SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy();
retryPolicy.setMaxAttempts(2);
retryTemplate.setRetryPolicy(retryPolicy);
return retryTemplate;
}
}
Enabling Spring Retry in the project:
#Configuration
#EnableRetry
public class AppConfig { ... }