I've got a route which is defined in Java in a route builder as follows:
from("jms:topic:trigger?maxConcurrentConsumers=1")
.autoStartup(true)
.beanRef("someProcessorBeen");
This has worked well for a few years.
However, under certain circumstances I want to be able to throw a specific 'retry' exception in the java processor bean, so that the same exchange is re-sent to the Bean.
I've tried adding &transacted=true to the JMS topic reference URI in the 'from' part, but I just get an ERROR log message from the DefaultErrorHandler which says "Failed delivery for ......" when I throw my retry Exception.
I've also tried adding:
onException(MyRetryException.class).maximumRedeliveries(10);
Before the from route definition, but the route doesn't seem to get created, as the route does not process any topic messages.
Note that this is all running within an OSGi environment, with ActiveMQ handling the JMS stuff.
UPDATE: OK, I found that adding the following in the route builder gives me what I want:
RedeliveryPolicyDefinition redeliveryPolicy = new RedeliveryPolicyDefinition();
// Set redelivery policy so it retries every 5 seconds for 10 minutes,
// then log an error when the retries have been exhausted
redeliveryPolicy.maximumRedeliveries(120)
.redeliveryDelay(5000)
.logExhausted(true)
.retriesExhaustedLogLevel(LoggingLevel.ERROR);
onException(MyRetryException.class)
.setRedeliveryPolicy(redeliveryPolicy);
So the question now is: Is this the correct way to achieve this?
Related
I try to use Camel to deliver files from one folder to a rest call and Im trying to achieve that on Error it's tried to redeliver twice and then moved to an error folder if the second redelivery fails as well. My code in the RouteBuilder's configure method looks like this:
errorHandler(deadLetterChannel("file:///home/camelerror").useOriginalMessage());
from("file:///home/camelefiles")
.onException(RetryableException.class)
.log("RetryableException handled")
.maximumRedeliveries(2)
.end()
.routeId(port.id())
.throwException(new RetryableException());
I get the "RetryableException handled" logs so I guess the exception is handled correctly but it redelivers the message an infinite number of times.
What am I doing wrong and how can I achieve that the message is only redelivered twice and then the deadLetterChannel is used?
Is it possible to rollback async processed message in ActiveMQ? I'm consuming next message while first one is still processing, so while I'm trying to rollback the first message on another (not activemq pool) thread, I'm getting above error. Eventually should I sednd message to DLQ manually?
Message error handling can work a couple ways:
Broker-side 'redelivery policy'. Where the client invokes a rollback n number (default is usually 6 retries) of times and the broker automatically moves the message to a Dead Letter Queue (DLQ)
Client-side. Application consumes the message and then produces to the DLQ.
Option #1 is good for unplanned/planned outages-- database down, etc. Where you want automatic retry. The re-delivery policy can also be configured when the client connects to the broker.
Option #2 is good for 'bad data' scenarios where you know the message will never be able to be processed. This is ideal, because you can move the message on the 1st consumption and not have to reject the message n number of times.
When you combine infinite retry with #1 and include #2 in your application flow, you can have a robust process flow of automatic retry, and move-bad-data-out-of-the-way-quickly. Best of breed =)
ActiveMQ Redelivery policy
We had a requirement to retry certain number of times with delay for camel SFTP Producer in case of errors. By using maximumReconnectAttempts along with reconnectDelay, camel attempts to retry for all type of errors but want to retry only for certain type of errors like socketException or connectionexception or jschExceptions and avoid retry for authentication exceptions. Here is the sample code we are using for this. How can we configure to retry only certain type of errors and not all?
from("file:///test/dummy/moveFailed=error&antInclude=*.txt&initialDelay=60000").routeId("test")
.log(LoggingLevel.DEBUG, " Message sending to Destination")
.setHeader(Exchange.FILE_NAME, simple("Test${date:now:yyyyMMdd}.CSV"))
.to("sftp://username#host/destinationpassword=password123&reconnectDelay=3000&maximumReconnectAttempts=5")
.log(LoggingLevel.INFO,"event=${in.header.event} || File successfully transmitted to Destination")
.end();
if you want to control the behavior per Exception you can do as follow:
onException(MyRetryableServiceException.class)
.useOriginalMessage()
.redeliveryDelay(1000)
.backOffMultiplier(1.5)
.maximumRedeliveries(3)
.retryAttemptedLogLevel(LoggingLevel.WARN)
.useExponentialBackOff();
When a MyRetryableServiceException is thrown the message will be redelivered as per maximumRedeliveries.
You can define multiple onException or wrap the ones you want to retry in a single Exception...
This has precedence over the default error handler, see exception-clause and camel error handling
Everything else will goes to the default error handled that will retry all kind of exception (prior camel 2.x, no retry from version >= 2.0). Unless you override it.
So, one possible solution is to define the OnException to skip retry for authentication errors and leave the default error handler to retry the others if you are using camel < 2.0
You can disable the default error handler with noErrorHandler(); or customize it as well, for example:
errorHandler(
deadLetterChannel("seda:errors")
.logNewException(true)
.log("Unable to process ${body}")
.maximumRedeliveries(1)
.retryAttemptedLogLevel(LoggingLevel.WARN)
.useExponentialBackOff()
);
There is an issue I've noticed when configuring InOut capable routes in Camel with the camel-rabbitmq extension.
When I set the main queue configuration to autoAck=false the same configuration is then replicated also for the temporary reply queue (it even uses the same prefetch(5) settings, easy to see in the RabbitMQ console). This causes the messages in the temp queue to sit there indefinitely until a server restart.
Virtual host Name Features State Ready Unacked Total incoming deliver / get ack
/test amq.gen-Hkdx9mckIfMc6JhDI6d-JA AD Excl idle 2 5 7 0.00/s 0.00/s 0.00/s
/test amq.gen-eUU7BRI3Ooo4F8Me7HrPnA AD Excl idle 2 5 7 0.00/s 0.00/s 0.00/s
Even though in the logs I can clearly see that the reply message is received just that the ack doesn't appear to be sent to RabbitMQ to clear our the message from the temp queue. And I've checked in the console that both temp queues have consumers on them so I would have expected Camel to send the ack.
o.a.c.c.r.RabbitMQMessagePublisher - Sending message to exchange: emailfeedbackExchange with CorrelationId = Camel-ID-VMS-1534332570964-0-11
o.a.c.c.r.r.ReplyManagerSupport - Received reply message with correlationID [Camel-ID-VMS-1534332570964-0-11]
The question is, how can I prevent this scenario while still keeping my autoAck=false and InOut capable route?
Probably I should mention here that there are no errors or the like, the flow works as expected and email processing works perfectly, the only issue is the stale messages on the temp queue.
Our Camel version is 2.20.2
This is the relevant Gradle config for all the Camel components we have:
compile ("org.apache.camel:camel-spring-boot-starter:${camelVersion}")
compile ("org.apache.camel:camel-rabbitmq:${camelVersion}")
compile ("org.apache.camel:camel-amqp:${camelVersion}")
The queue and route configurations:
restentrypointroute:
restEndpoint: /app
postEndpoint: /email
outputEmailEndpoint: rabbitmq://vms:5672/emailExchange?connectionFactory=rabbitConnectionFactory&autoDelete=false&queue=emailrouteQueue&exchangeType=topic&autoAck=false&bridgeEndpoint=true&concurrentConsumers=3&threadPoolSize=3&channelPoolMaxSize=3&prefetchCount=5&prefetchEnabled=true
emailroutebuilder:
serviceName: emailroutebuilder
inputEndpoint: rabbitmq://vms:5672/emailExchange?connectionFactory=rabbitConnectionFactory&autoDelete=false&queue=emailrouteQueue&exchangeType=topic&autoAck=false&bridgeEndpoint=true&concurrentConsumers=3&threadPoolSize=3&channelPoolMaxSize=3&prefetchCount=5&prefetchEnabled=true
emailProcessor: bean:emailProcessor
maximumRedeliveries: 5
redeliveryDelay: 30000
Here is the relevant bit from the RestRouteBuilder implementation:
#Override
public void configure() throws Exception {
restConfiguration().component("restlet").bindingMode(RestBindingMode.json);
rest(restEndpoint).post(postEndpoint)
.type(MyRequest.class)
.route()
.startupOrder(Integer.MAX_VALUE - 2)
.process(this::process)
.choice()
.when(header(DELIVERYSTATUS_HEADER)
.isEqualTo(Status.GENERATED)).to(outputEmailEndpoint)
.when(header(DELIVERYSTATUS_HEADER)
.isEqualTo(Status.COMPLETED)).to(outputEmailEndpoint, outputArchiveEndpoint).end()
.endRest();
The process() method adds the DELIVERYSTATUS_HEADER header to the Camel exchange and validates the payload.
The EmailRouteBuilder looks like this:
public void configure() throws Exception {
super.configure();
from("direct:" + getServiceName())
.to(emailProcessor)
.process(ex -> {
ex.setOut(ex.getIn());
});
}
Where the super.configure() call configures exception handling and dead-lettering, startup order, retry counts, max re-deliveries etc. It's quite a bit of code there but if you think something in there might be the cause of this issue I'll post it up. Also, if you need me to add any other configuration please let me know.
From the above it's kind of clear why we need an InOut route with autoAck=false as losing emails is bad from a business standpoint and the REST client would need a response based on how the EmailProcessor got on. Just how to get rid of the stale messages in the temp queues?
EDIT
Actually the route only works fine until the prefetch count is exhausted, after that it starts throwing exceptions and the REST client is getting HTTP 500 responses back.
org.apache.camel.ExchangeTimedOutException: The OUT message was not received within: 20000 millis due reply message with correlationID: Camel-ID-VMSYS119-1534407032085-0-284 not received on destination: amq.gen-eUU7BRI3Ooo4F8Me7HrPnA.
As per the comments, this turned out to be a bug in the camel-rabbitmq component and a fix has been now applied to the master branch.
Jira ticket here: https://issues.apache.org/jira/browse/CAMEL-12746
The fix will be available in versions 2.21.3, 2.22.1, 2.23.0 and above.
Edit:
Including the code change in the answer.
TemporaryQueueReplyManager line 139 - always start the consumer of temprary queues with auto acknowledge mode of true.
Changing this:
private void start() throws IOException {
tag = channel.basicConsume(getReplyTo(), endpoint.isAutoAck(), this);
}
To this:
private void start() throws IOException {
tag = channel.basicConsume(getReplyTo(), true, this);
}
I have set up an errorHandler in a Camel route that will retry a message several times before sending the message to a dead letter channel (an activemq queue in this case). What I would like is to see an ERROR log when the message failed to be retried the max number of times and was then sent to the dead letter queue.
Looking at the docs for error handling and dead letter channels, it seems that there are 2 options available on the RedeliveryPolicy: retriesAttemptedLogLevel and retriesExhaustedLogLevel. Supposedly by default the retriesExhaustedLogLevel is already set at LoggingLevel.ERROR, but it does not appear to actually log anything when it has expended all retries and routes the message to the dead letter channel.
Here is my errorHandler definition via Java DSL.
.errorHandler(this.deadLetterChannel(MY_ACTIVE_MQ_DEAD_LETTER)
.useOriginalMessage()
.maximumRedeliveries(3)
.useExponentialBackOff()
.retriesExhaustedLogLevel(LoggingLevel.ERROR)
.retryAttemptedLogLevel(LoggingLevel.WARN))
I have explicitly set the level to ERROR now and it still does not appear to log out anything (to any logging level). On the other hand, retryAttemptedLogLevel is working just fine and will log to the appropriate LoggingLevel (ie, I could set retryAttemptedLogLevel to LoggingLevel.ERROR and see the retries as ERROR logs). However I only want a single ERROR log in the event of exhaustion, instead of an ERROR log for each retry when a subsequent retry could potentially succeed.
Maybe I am missing something, but it seems that the retriesExhaustedLogLevel does not do anything...or does not log anything if the ErrorHandler is configured as a DeadLetterChannel. Is there a configuration that I am still needing, or does this feature of RedeliveryPolicy not execute for this specific ErrorHandlerFactory?
I could also set up a route to send my exhausted messages that simply logs and routes to my dead letter channel, but I would prefer to try and use what is already built into the ErrorHandler if possible.
Updated the ErrorHandler's DeadLetterChannel to be a direct endpoint. Left the 2 logLevel configs the same. I got the 3 retry attempted WARN logs, but no ERROR log telling me the retries were exhausted. I did, however, set up a small route listening to the direct dead letter endpoint that logs, and that is working.
Not a direct solution to my desire to have the ERROR log work for the exhaustion, but is an acceptable workaround for now.
Please try with this code:
.errorHandler(deadLetterChannel("kafka:sample-dead-topic")
.maximumRedeliveries(4).redeliveryDelay(60000)
.retriesExhaustedLogLevel(LoggingLevel.WARN)
.retryAttemptedLogLevel( LoggingLevel.WARN)
.retriesExhaustedLogLevel(LoggingLevel.ERROR)
.logHandled(true)
.allowRedeliveryWhileStopping(true)
.logRetryStackTrace(true)
.logExhausted(true)
.logStackTrace(true)
.logExhaustedMessageBody(true)
)
retry is configured for 1 minute interval.
Camel application logged the errors for evry retry with the detailed information.