We Keep Coding

Spring AMQP #RabbitListener is not ready to receive messages on #ApplicationReadyEvent. Queues/Bindings declared too slow?

we have a larger multi service java spring app that declares about 100 exchanges and queues in RabbitMQ on startup. Some are declared explicitly via Beans, but most of them are declared implicitly via #RabbitListener Annotations.
#Component
#RabbitListener(
bindings = #QueueBinding(key = {"example.routingkey"},
exchange = #Exchange(value = "example.exchange", type = ExchangeTypes.TOPIC),
value = #Queue(name = "example_queue", autoDelete = "true", exclusive = "true")))
public class ExampleListener{
#RabbitHandler
public void handleRequest(final ExampleRequest request) {
System.out.println("got request!");
}
There are quite a lot of these listeners in the whole application.
The services of the application sometimes talk to each other via RabbitMq, so take a example Publisher that publishes a message to the Example Exchange that the above ExampleListener is bound to.
If that publish happens too early in the application lifecycle (but AFTER all the Spring Lifecycle Events are through, so after ApplicationReadyEvent, ContextStartedEvent), the binding of the Example Queue to the Example Exchange has not yet happend and the very first publish and reply chain will fail. In other words, the above Example Listener would not print "got request".
We "fixed" this problem by simply waiting 3 seconds before we start sending any RabbitMq messages to give it time to declare all queues,exchanges and bindings but this seems like a very suboptimal solution.
Does anyone else have some advice on how to fix this problem? It is quite hard to recreate as I would guess that it only occurs with a large amount of queues/exchanges/bindings that RabbitMq can not create fast enough. Forcing Spring to synchronize this creation process and wait for a confirmation by RabbitMq would probably fix this but as I see it, there is no built in way to do this.

Are you using multiple connection factories?
Or are you setting usePublisherConnection on the RabbitTemplate? (which is recommended, especially for a complex application like yours).
Normally, a single connection is used and all users of it will block until the admin has declared all the elements (it is run as a connection listener).
If the template is using a different connection factory, it will not block because a different connection is used.
If that is the case, and you are using the CachingConnectionFactory, you can call createConnection().close() on the consumer connection factory during initialization, before sending any messages. That call will block until all the declarations are done.

CircuitBreaker immediate fallback

I followed https://resilience4j.readme.io/docs/getting-started-3 for Resilience4J documentation.
I have a problem, my circuitbreaker is immediately connecting to the fallback( on the first invoke) when the primary-backend is not available.
#Bean
public CircuitBreakerConfig circuitBreakerConfig() {
return CircuitBreakerConfig.custom()
.slidingWindowType(CircuitBreakerConfig.SlidingWindowType.COUNT_BASED)
.slidingWindowSize(5)
.minimumNumberOfCalls(5)
.failureRateThreshold(4)
.build();
}
I also tried moving the config to application.yml, but still the same behavior.
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
#CircuitBreaker(name = "backendA", fallbackMethod = "fallbackA")
public String backendA() {
return restTemplate.getForObject("http://localhost:9999/backendA", String.class);
}
public String fallbackA(Exception e) {
return restTemplate.getForObject("http://localhost:8080/partner", String.class);
}
Can someone help?
//UPDATE
More details below.
As shown in the code above, my app is making a call to /backendA, behind a circuit breaker. It is expected that as per config, if backendA is not available, the should fail 5 times and 6th call onwards fallback to /partner API. Or in other words, circuit should open after 5 calls.
This is how I tested.
I started the application. Both backendA and fallbackA are available. After a few calls, I killed backendA. the next call to backendA is falling back to /partner while I expect the next 5 calls to backendA fails without a fallback. Is my expectation correct?

After some research, I came to know that fallback on intermittent failures is an expected behaviour, though I could not find it documented anywhere. So even if the circuit is CLOSED, fallback will be invoked if the method throws a matching exception.
Note:
If you are not happy with the immediate fallback, wrap it with a RETRY config.

#CircuitBreaker- you should expect the result that if it's status is open that you will immediately receive a fallback and it no longer calls your desired service.
If you expect the result that when you turn off your called backend service and you want 5 call until you get fallback you need use #Retry, if the call fails, then it will try to call your service again.
If you use multiple notations, you can read more about the order here

Why does the RetryScheduler in Axon Framework not retry after a NoHandlerForCommandException?

so I have a Saga and the Saga sends a command to a different microservice on a specific event. I wanted to configure the commandGateway with a RetryScheduler, so that it retries to send the command in case that the other microservice is down. The RetryScheduler will only perform retries if the exception is a RuntimeException, which the NoHandlerForCommandException that is thrown when the other service if offline definately is.
If i dont set the maxRetryCount then the error message is o.a.c.gateway.IntervalRetryScheduler : Processing of Command [XXXCommand] resulted in an exception 1 times. Giving up permanently
If I do set the attribute the error message is o.a.c.gateway.IntervalRetryScheduler : Processing of Command [XXXCommand] resulted in an exception and will not be retried
If the other microservice is running, then the command is handled correctly, no problems.
Does anybody have an idea what could be the issue?
This is my configuration for the commandGateway with a RetryScheduler:
#Bean
public CommandGateway commandGateway(){
Configurer configurer = DefaultConfigurer.defaultConfiguration();
CommandBus commandBus = configurer.buildConfiguration().commandBus();
ScheduledExecutorService scheduledExecutorService = Executors.newScheduledThreadPool(1);
RetryScheduler rs = IntervalRetryScheduler.builder().retryExecutor(scheduledExecutorService).maxRetryCount(100).retryInterval(1000).build();
CommandGateway commandGateway = DefaultCommandGateway.builder().commandBus(commandBus).retryScheduler(rs).build();
return commandGateway;
}

To resolve the problem at hand, you could provide your own implementation of the IntervalRetryScheduler, which overrides the IntervalRetryScheduler#isExplicitlyNonTransient(Throwable) method to also take into account retrying the NoHandlerForCommandException.
Note though that the IntervalRetryScheduler is intentionally only retrying on exceptions which are of type AxonNonTransientException, is those typically signal recoverable exceptions. The NoHandlerForCommandException means that the CommandBus implementation being used has no clue whom to give the command too, something which in general suggests an thing which cant be retried.
It however seems you have a scenario where it does make sense. Thus, like I point out at the start, overriding the isExplicitlyNonTransient(Throwable) method to exclude NoHandlerForCommandException would be the way to go for you I think.
Hope this helps you out!

How to handle exceptions during spring-boot start-up?

This isn't about how to handle exceptions in Spring MVC or anything. I specifically need to handle an exception that can happen while spring is starting, i.e. before the whole application context is even initialised.
For a bit of background, the application in question is an IoT node that allows remote access to electronic equipment. It has a little h2 database built in to persist some data. That data is nice to have at some moments, but not really essential for the application to work.
It so happens that the device the application is running on can get its power cut every once in a while, and if that happens while there was a write operation to the database going on, the file is corrupt and a JdbcSQLException will be thrown when the application tries to boot again.
Since the data is not really essential, the easiest way to make the application work again is to just delete the database and let h2 recreate it. But in order to do that, I have to catch the exception so I can react to it. The application does not have to continue starting, it will be booted up again by systemd. I really just need to identify the exception and delete the file, that's it.
There is one obvious way to do it, which is to put SpringApplication.run in a try-catch block. But it's also really ugly, because I get the exception I'm looking for nested inside a gazillion spring exceptions that were caused by h2 failing to start.
It was also suggested that I catch the exception in the bean that instantiates the database, but unfortunately there is no bean instantiating it. The DB serves as a Quartz job-store and as such is fully managed by spring. Its entire presence in the code are the following entries in the properties file:
spring.quartz.job-store-type=jdbc
spring.quartz.properties.org.quartz.jobStore.misfireThreshold=900000
spring.datasource.name=h2
spring.datasource.url=jdbc:h2:file:${config.folder}controller
spring.datasource.driverClassName=org.h2.Driver
My question is, is there a way to register some kind of exception handler, or other means, to handle the exception directly when it happens, when I can identify it much more easily?

Depends how you've declared the bean. What's wrong with simply wrapping the bean like this?
#Configuration
class Conf {
#Bean
public DB foo() throws JdbcSQLException
{
try
{
return new DB();
}
catch(JdbcSQLException e)
{
deleteDatabase();
throw JdbcSQLException;
}
}
public static void deleteDatabase()
{
//...
}
}

Spring declarative transactions and manually scheduling threads

I'm having a strange issue.
In a class I have:
private final ScheduledExecutorService executor
= Executors.newSingleThreadScheduledExecutor();
public MyClass(final MyService service) {
executor.scheduleAtFixedRate(new Runnable() {
#Override
public void run() {
service.foo();
}
}, 0, 30, TimeUnit.SECONDS);
}
MyService is a spring bean that has #Transactional on its foo method. MyClass is instantiated only once (effectively singleton in the application)
After the first invocation of service.foo() (which works fine), on subsequent requests to the application I am randomly getting:
java.lang.IllegalStateException: Already value [SessionImpl(PersistenceContext[entityKeys=[],collectionKeys=[]];ActionQueue[insertions=[] updates=[] deletions=[] collectionCreations=[] collectionRemovals=[] collectionUpdates=[]])] for key [org.hibernate.impl.SessionFactoryImpl#2cd91000] bound to thread [http-bio-8080-exec-10]
A few observations:
when the exception is thrown, the session stored in the TransactionSynchronizationManager is closed
the transaction synchronization manager resource map for the manually scheduled thread is empty
the exception occurs in http-bio-8080-exec threads, but the manually scheduled one is a pool- thread - so there is no 'thread polution'
MyClass is instantiated on startup, in a thread named "Thread-5", i.e. it is not in any way related to the http-bio threads.
If I comment the invocation to service.foo(), or get rid of the #Transactioanl annotation, everything works (except, of course, that data is not inserted in the db)
Any clues what the issue might be?
(Note: I prefer not to use #Scheduled - I don't want MyClass to be a spring bean, and the runnable has to operate on some of its internal state before invoking the service)
Update: After a while I'm able to reproduce it even without the scheduling stuff. So probably a general spring problem with the latest snapshot I'm using.

I assume that exception comes from an invocation of the TransactionInterceptor or the like (some Spring infrastructure bean), or are you using the TransactionSynchronizationManager from your own code somewhere? It appears to me that something is binding sessions to a thread being managed by your container (is that Tomcat 7?) and failing to unbind them before they're returned to the container's thread pool. Thus when the same thread is used for another transactional request later, Spring can't bind the new Session to it because the old one wasn't cleaned up.
I don't actually see anything to make me think it's directly related to your custom scheduling with MyClass. Are you sure it's not just a coincidence that you didn't see the exception when you remove the service.foo() call?
If you could catch one of those threads in a debugger when it's being returned to the pool with a Session still bound to it, you might be able to backtrack to what it was used for. An omniscient debugger would theoretically be perfect for this, though I've never used one myself: ODB and TOD are the two I know of.
Edit: An easier way to find the offending threads: add a Filter (servlet filter, that is) to your app that runs "around" everything else. After chain.doFilter(), as the last act of handling a request before it leaves your application, check the value of TransactionSynchronizationManager.getResourceMap(). It should be an empty map when you're done handling a request. When you find one that isn't, that's where you need to backtrack from to see what happened.