I want to implement the circuit breaker design pattern using Spring-Retry. The main problem that I am facing is opening the circuit for all requests. It still keeps retrying if I make a new request from the browser.
I have a RetryTemplate with a CircuitBreakerRetryPolicy defined as follows:
#Configuration
public class MyApplicationConfig {
#Bean
public RetryTemplate retryTemplate(RetryPolicy cbRetry) {
RetryTemplate retryTemplate = new RetryTemplate();
FixedBackOffPolicy fixedBackOffPolicy = new FixedBackOffPolicy();
fixedBackOffPolicy.setBackOffPeriod(1000);
retryTemplate.setBackOffPolicy(fixedBackOffPolicy);
retryTemplate.setRetryPolicy(circuitBreakerRetryPolicy);
return retryTemplate;
}
#Bean("cbRetry")
public RetryPolicy circuitBreaker() {
CircuitBreakerRetryPolicy circuitBreakerRetryPolicy = new CircuitBreakerRetryPolicy();
circuitBreakerRetryPolicy.setResetTimeout(20000);
circuitBreakerRetryPolicy.setOpenTimeout(2000);
return circuitBreakerRetryPolicy;
}
}
Then I use RestTemplate to make a call to a non-existing url, just to make the call fail and I count the retries. The circuit is supposed to open in 2 seconds, but if I try another call right after the first one fails, it keeps retrying again.
How do I make the circuit stay open for a given openTimeOut period for all incoming requests?
I experimented with RetryState by providing it to the retryTemplate call, but that doesn't help either.
Thank you!
spring-retry state is thread-bound.
You would need to write a custom policy to maintain global state.
Related
I am new to spring-retry. Basically, for retrying calls to REST APIs, I have integrated spring-retry into my spring-boot application. To do this, I have made following changes:
Added spring-retry to pom.xml.
Added following configuration:
#Configuration
#EnableRetry
public class RetryConfiguration {
}
Finally added #Retryable annotation to the class (this class is not a Spring Bean) method that I would like to be retried for various exceptions as follows:
public class OAuth1RestClient extends OAuthRestClient {
#Override
#Retryable(maxAttempts = 3, value = {
Exception.class},
backoff = #Backoff(delay = 100, multiplier = 3))
public Response executeRequest(OAuthRequest request)
throws InterruptedException, ExecutionException, IOException {
System.out.println("Inside Oauth1 client");
return myService.execute(request);
}
Now, the executeRequest method is not retrying. I am not able to understand if I am missing anything here.
Could anyone please help? Thanks.
If your class is not Spring managed (e.g. #Component/#Bean) the
annotation processor for #Retryable won't pick it up.
You can always manually define a retryTemplate and wrap calls with it:
RetryTemplate.builder()
.maxAttempts(2)
.exponentialBackoff(100, 10, 1000)
.retryOn(RestClientException.class)
.traversingCauses()
.build();
and then
retryTemplate.execute(context -> myService.execute(request));
If you want to retry on multiple exception, this can happen via custom RetryPolicy
Map<Class(? extends Throwable), Boolean> exceptionsMap = new HashMap<>();
exceptionsMap.put(InternalServerError.class, true);
exceptionsMap.put(RestClientException.class, true);
SimpleRetryPolicy policy = new SimpleRetryPolicy(5, exceptionsMap, true);
RetryTemplate.builder()
.customPolicy(policy)
.exponentialBackoff(100, 10, 1000)
.build();
FYI: RetryTemplate is blocking and you might want to explore a non-blocking async retry approach like async-retry. - and the retryOn() supports a list of exceptions.
I have spring integration flow that gets triggered once a every day, that pulls all parties from database and sends each party to an executorChannel.
The next flow would pull data for each party and then process them parallelly by sending in to a different executor channel.
Challenge i'm facing is how do i know when this entire process ends. Any ideas on how to acheve this .
Here's my pseudo code of executor channels and integration flows.
#Bean
public IntegrationFlow fileListener() {
return IntegrationFlows.from(Files.inboundAdapter(new
File("pathtofile"))).channel("mychannel").get();
}
#Bean
public IntegrationFlow flowOne() throws ParserConfigurationException {
return IntegrationFlows.from("mychannel").handle("serviceHandlerOne",
"handle").nullChannel();
}
#Bean
public IntegrationFlow parallelFlowOne() throws ParserConfigurationException {
return IntegrationFlows.from("executorChannelOne").handle("parallelServiceHandlerOne",
"handle").nullChannel();
}
#Bean
public IntegrationFlow parallelFlowTwo() throws ParserConfigurationException {
return IntegrationFlows.from("executorChannelTwo").handle("parallelServiceHandlerTwo",
"handle").nullChannel();
}
#Bean
public MessageChannel executorChannelOne() {
return new ExecutorChannel(
Executors.newFixedThreadPool(10));
}
#Bean
public MessageChannel executorChannelTwo;() {
return new ExecutorChannel(
Executors.newFixedThreadPool(10));
}
#Component
#Scope("prototype")
public class ServiceHandlerOne{
#Autowired
MessageChannel executorChannelOne;
#ServiceActivator
public Message<?> handle(Message<?> message) {
List<?> rowDatas = repository.findAll("parties");
rowDatas.stream().forEach(data -> {
Message<?> message = MessageBuilder.withPayload(data).build();
executorChannelOne.send(message);
});
return message;
}
}
#Component
#Scope("prototype")
public class ParallelServiceHandlerOne{
#Autowired
MessageChannel executorChannelTwo;;
#ServiceActivator
public Message<?> handle(Message<?> message) {
List<?> rowDatas = repository.findAll("party");
rowDatas.stream().forEach(data -> {
Message<?> message = MessageBuilder.withPayload(data).build();
executorChannelTwo;.send(message);
});
return message;
}
}
First of all no reason to make your services as #Scope("prototype"): I don't see any state holding in your services, so they are stateless, therefore can simply be as singleton. Second: since you make your flows ending with the nullChannel(), therefore point in returning anything from your service methods. Therefore just void and flow is going to end over there naturally.
Another observation: you use executorChannelOne.send(message) directly in the code of your service method. The same would be simply achieved if you just return that new message from your service method and have that executorChannelOne as the next .channel() in your flow definition after that handle("parallelServiceHandlerOne", "handle").
Since it looks like you do that in the loop, you might consider to add a .split() in between: the handler return your List<?> rowDatas and splitter will take care for iterating over that data and replies each item to that executorChannelOne.
Now about your original question.
There is really no easy to say that your executors are not busy any more. They might not be at the moment of request just because the message for task has not reached an executor channel yet.
Typically we recommend to use some async synchronizer for your data. The aggregator is a good way to correlate several messages in-the-flight. This way the aggregator collects a group and does not emit reply until that group is completed.
The splitter I've mentioned above adds a sequence details headers by default, so subsequent aggregator can track a message group easily.
Since you have layers in your flow, it looks like you would need a several aggregators: two for your executor channels after splitting, and one top level for the file. Those two would reply to the top-level for the final, per-file grouping.
You also may think about making those parties and party calls in parallel using a PublishSubscribeChannel, which also can be configured with a applySequence=true. This info then will be used by the top-level aggregator for info per file.
See more in docs:
https://docs.spring.io/spring-integration/docs/current/reference/html/core.html#channel-implementations-publishsubscribechannel
https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#splitter
https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#aggregator
I am using resilience4j with Spring Boot 2.x
What is the impact of using Retry and circuit breaker modules on memory and cpu?
Also what is the memory impact if I have 2000 events/s incoming each payload around 10Mb and I have kept the wait duration of retry as 15 seconds with exponential backoff multiplier as 2?
I have 8Gb of application memory
The best way is to monitor your application with a profile like VisualVM. Then you can know where is the bottleneck.
One thing that I know that matters is where you create your circuitbreaker instance. In the end there is a collector that remove the instances not used. But in your case it seems to be a good idea to not place the creation of the circuit breaker on the get method like here
#Service
public static class DemoControllerService {
private RestTemplate rest;
private CircuitBreakerFactory cbFactory;
public DemoControllerService(RestTemplate rest, CircuitBreakerFactory cbFactory) {
this.rest = rest;
this.cbFactory = cbFactory;
}
public String slow() {
return cbFactory.create("slow").run(() -> rest.getForObject("/slow", String.class), throwable -> "fallback");
}
}
But create the circuitbreaker on the constructor.
#Service
public class DemoControllerService {
private RestTemplate restTemplate = new RestTemplate();
private CircuitBreakerFactory circuitBreakerFactory;
private CircuitBreaker circuitBreaker;
#Autowired
public DemoControllerService(CircuitBreakerFactory circuitBreakerFactory) {
this.circuitBreakerFactory = circuitBreakerFactory;
this.circuitBreaker = circuitBreakerFactory.create("circuitbreaker");
}
There are also discussions to place one circuitbreaker per host. Other thing that you can do is to remove the instance of the Registry by yourself and not wait for the circuitbreaker component rome it in the future.
registry.remove(circuitBreakerName);
Here is a discussion also about to clean up the Registry memory.
Spring's #Retryable annotation will retry three times (default) and fallback to the #Recovery method. #CircuitBreaker however, will retry once and fall back when the state is closed.
I want to combine these two: when the circuit breaker state is closed, will retry three times before falling back (to deal with transient errors), if the state is open, will directly fall back.
Any elegant way to do this? A possible approach is to implement the retry logic inside the function, but I feel that it wouldn't be the best solution.
The #CircuitBreaker already implements #Retry as a stateful = true, that's how he knows how many calls failed.
I think the best approach here, would be use a RetryTemplate inside your method:
#CircuitBreaker(maxAttempts = 2, openTimeout = 5000l, resetTimeout = 10000l)
void call() {
retryTemplate.execute(new RetryCallback<Void, RuntimeException>() {
#Override
public Void doWithRetry(RetryContext context) {
myService.templateRetryService();
}
});
}
Declaring the RetryTemplate:
#Configuration
public class AppConfig {
#Bean
public RetryTemplate retryTemplate() {
RetryTemplate retryTemplate = new RetryTemplate();
FixedBackOffPolicy fixedBackOffPolicy = new FixedBackOffPolicy();
fixedBackOffPolicy.setBackOffPeriod(2000l);
retryTemplate.setBackOffPolicy(fixedBackOffPolicy);
SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy();
retryPolicy.setMaxAttempts(2);
retryTemplate.setRetryPolicy(retryPolicy);
return retryTemplate;
}
}
Enabling Spring Retry in the project:
#Configuration
#EnableRetry
public class AppConfig { ... }
I need to implement with the help of spring-intagration libraries a message pipeline. At the beginning, as I see now, it needs to contain several elements:
a. Messaging gateway,
#MessagingGateway(name = "entryGateway", defaultRequestChannel = "requestChannel")
public interface MessageGateway {
public boolean processMessage(Message<?> message);
}
which is called when I want to start the pipeline:
messageGateway.processMessage(message);
b. Channel for transmitting the messages:
#Bean
public MessageChannel requestChannel() {
return new DirectChannel();
}
c.Router which decides then where flow the messages
#MessageEndpoint
#Component
public class MessageTypeRouter {
Logger log = Logger.getLogger(MessageTypeRouter.class);
#Router(inputChannel="requestChannel")
public String processMessageByPayload(Message<?> message){...}
There can be many incoming messages in a small period of time, so I wanted to realize a channel (b) as QueueChannel:
#Bean
public MessageChannel requestChannel() {
return new QueueChannel();
}
On the other hand I would like the router to start as soon as a message comes through gateway and the other messages to wait in the queue. But in this case I received an error, which said that I should have used a poller.
May be you could give me a piece of advice, how I can realize my scheme. Thank you in advance.
As we know with XML config we must declare a <poller> component and mark it with default="true". That allows any PollingConsumer endpoint to pick up the default Poller from the context.
With Java config we must declare a #Bean for similar purpose:
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata defaultPoller() {
PollerMetadata pollerMetadata = new PollerMetadata();
pollerMetadata.setTrigger(new PeriodicTrigger(10));
return pollerMetadata;
}
Where the PollerMetadata.DEFAULT_POLLER is specific constant to define the default poller. Although the same name is used from XML config in case of default="true".
From other side #Router annotation has poller attribute to specify something similar like we do with nested <poller> in XML.