We Keep Coding

Apache Camel - Seda endpoint, recipientList, BlockWhenFull=true and Queue full IllegalStateException

I have a code that retrieves messages from a RabbitMQ queue, aggregates them and then distributes the aggregates to another route that will dispatch them to differents routes via the recipientList component.
The problem is that the latter throws the "Error executing reactive work due to Queue full" exception despite the addition of the blockWhenFull=true property on the producer side.
from("direct:rabbitmq-ids-aggregate")
.aggregate(constant(true), new UpdatesAggregationStrategy())
.completionInterval("{{updates.aggregation.completionInterval}}")
.completionSize("{{updates.aggregation.completionSize}}")
.setHeader(CORRELATION_HEADER, simple("${exchangeId}"))
.to("seda:dispatch?blockWhenFull=true");
from("seda:dispatch")
.recipientList(simple("{{routes.hr-data}},{{routes.comments-data}},{{routes.legacy-hr-data}},{{routes.ranking-data}}"))
.end();
2023-01-10 19:46:35,944 WARN o.a.c.i.e.DefaultReactiveExecutor [Camel (integration-core) thread #5 - seda://dispatch] Error executing reactive work due to Queue full. This exception is ignored.
java.lang.IllegalStateException: Queue full
at java.util.AbstractQueue.add(AbstractQueue.java:98) ~[?:?]
at org.apache.camel.component.seda.SedaProducer.addToQueue(SedaProducer.java:251) ~[camel-seda-3.15.0.jar:3.15.0]
at org.apache.camel.component.seda.SedaProducer.process(SedaProducer.java:149) ~[camel-seda-3.15.0.jar:3.15.0]
at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:471) ~[camel-core-processor-3.15.0.jar:3.15.0]
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:187) [camel-base-engine-3.15.0.jar:3.15.0]
at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:64) [camel-base-engine-3.15.0.jar:3.15.0]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:184) [camel-core-processor-3.15.0.jar:3.15.0]
at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:399) [camel-base-engine-3.15.0.jar:3.15.0]
at org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:269) [camel-seda-3.15.0.jar:3.15.0]
at org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsumer.java:187) [camel-seda-3.15.0.jar:3.15.0]
at org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:130) [camel-seda-3.15.0.jar:3.15.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]

The blockWhenFull=true property on the producer side is not working because it is being set on the wrong endpoint (direct:rabbitmq-ids-aggregate) instead of seda:dispatch. To fix this issue, you should add the blockWhenFull=true property on the seda:dispatch endpoint like this:
from("seda:dispatch?blockWhenFull=true")
.recipientList(simple("{{routes.hr-data}},{{routes.comments-data}},
{{routes.legacy-hr-data}},{{routes.ranking-data}}")).end();
This will tell Camel to block and wait for the seda:dispatch queue to have space available before adding new messages to it, which should prevent the "Error executing reactive work due to Queue full" exception from being thrown.

Webflux, reactor connection reset by peer unhandled

So basically I am simulating a 'Connection reset by peer' locally using Toxiproxy or Wiremock (same behaviour for both of them) and I get the exception below.
The tricky part is that it will bypass any defined exception handlers and there's no way to revert the flow if there's a failure.
I tried configuring the WebClient as mentioned here: https://github.com/reactor/reactor-netty/issues/388 by either removing the connection pool or simply setting the HttpClient keepAlive property to false.
I defined a global exception handling mechanism by extending ErrorWebExceptionHandler but it doesn't go through.
Do you have any suggestions on how to manage this one? Or a proper configuration which I could try for either TcpClient / HttpClient or WebClient? What am I missing?
There must be a way to handle this properly..
[26/04/22 20:02:28] lvl=ERROR [ioEventLoopGroup-4-3] r.n.r.PooledConnectionProvider - [id: 0xaf559a88, L:0.0.0.0/0.0.0.0:56755 ! R:localhost/127.0.0.1:8989] Pooled connection observed an error
java.io.IOException: Connection reset by peer
at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:233)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:358)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:247)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1140)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

What does Retry mean in context of Java Couchbase SDK?

I am using Java couchbase sdk in my application. While setting up the DefaultCouchbaseEnvironment, I came across the property RetryStrategy. Now I am using the default configuration for which the retry strategy is BestEffortRetryStrategy. According to documentation
BestEffortRetryStrategy will retry the operation until it either succeeds or the maximum request lifetime is reached
By default the maximum request lifetime is 75 seconds.
Now what i what i want to understand here is what does retry mean here. Does retry mean retrying the request whenever an exception occurs or does it mean it will retry to allocate this request to some node to process the request in case it can't and it will keep retrying for 75 seconds?
I am looking at my application logs for different exceptions to understand this and I could see that TemporaryFailureException wasn't retried and i could also see that in some instances RequestCancelledException was being thrown after 75 seconds. Is it fair to assume that couchbase retries a request to assign it to node to process it and not actually retries on any exception once it makes it to the node that will process this request?
StackTrace for TemporaryFailureException-
stackTrace: com.couchbase.client.java.error.TemporaryFailureException: null
at com.couchbase.client.java.bucket.api.Mutate$2$1.call(Mutate.java:246)
at com.couchbase.client.java.bucket.api.Mutate$2$1.call(Mutate.java:220)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69)
at rx.observers.Subscribers$5.onNext(Subscribers.java:235)
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onNext(OnSubscribeDoOnEach.java:101)
at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
at rx.Subscriber.setProducer(Subscriber.java:205)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
at rx.Subscriber.setProducer(Subscriber.java:205)
at rx.Subscriber.setProducer(Subscriber.java:205)
at rx.subjects.AsyncSubject.onCompleted(AsyncSubject.java:103)
at com.couchbase.client.core.endpoint.AbstractGenericHandler.completeResponse(AbstractGenericHandler.java:508)
at com.couchbase.client.core.endpoint.AbstractGenericHandler.access$000(AbstractGenericHandler.java:86)
at com.couchbase.client.core.endpoint.AbstractGenericHandler$1.call(AbstractGenericHandler.java:526)
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java)
at java.lang.Thread.run(Thread.java:748)
Caused by: rx.exceptions.OnErrorThrowable$OnNextValue: OnError while emitting onNext value: com.couchbase.client.core.message.kv.UpsertResponse.class
at rx.exceptions.OnErrorThrowable.addValueAsLastCause(OnErrorThrowable.java:118)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:73)
... 21 common frames omitted```

BestEffortRetryStrategy should retry until the the request is cancelled by the timeout.
FailFastRetryStrategy should not retry. It should fail immediately.
If you have a TemporaryFailureException and have BestEffortRetryStrategy, that should have been retried. If you had one that was not retried can you share the stacktrace?
Mike

How to make the distinction between different exceptions in Netty's onChannelFailure callback

I am writing an HTTP client with Netty 4.1.12.Final and I have unit tests simulating the crash of the HTTP server in order to be able to handle it.
I noticed that, when it happens, the exceptionCaught callback method of my inbound handler is called with:
java.io.IOException: Une connexion existante a dû être fermée par l’hôte distant
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(Unknown Source)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
at sun.nio.ch.IOUtil.read(Unknown Source)
at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1100)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:372)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
Where the english equivalent of the exception message is quite probably:
java.io.IOException: An existing connection was forcibly closed by the remote host
Since this callback method is also called when an exception is thrown from my channelRead0 method of my inbound handler, I am asking a few questions:
Should I always consider an IOException "received" in the exceptionCaught callback as an indication that there is no point in continuing using the channel?
Since channelRead0 is declared to throw Exception, should I catch all IOException inside it in order to be sure that, when "receiving" an IOException in the exceptionCaught callback, it is related to the Channel?
Is there a way to know if an exception "received" in the exceptionCaught callback is related to I/O operations or to handlers operations?
Thank you for any hint!

1) If we are talking about a TCP connection then yes every IOException will result in having the connection closed automatically by netty as there is no way to recover.
2) I think I don't understand the question completely as each exception passed through the exceptionCaught(...) method is related the the channel which can be obtained by ctx.channel()
3) No there is no way in general. That said if its a TCP connection and its an IOException and its triggered by the actual transport we will close the connection.

Apache ConnectionClosedException: Connection closed,HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:344)

we have three applications A,B and C deployed on same tomcat server.There is an HTTP call that happens between A to B(REST CALL) and Another http call from B to C(REST CALL)
We were using synchronous HTTP call initially,recently we changed our code to Asynchronous call.
We have used HttpAsyncClients for (Application A to call Application B)
and for (Application B to call Application C)..
we are getting intermittent org.apache.http.ConnectionClosedException:
Some time we are getting this exception in Logs of application A(Async Client to B) and some time in Application B(Async Client to C)
CloseableHttpAsyncClient client = HttpAsyncClients.custom()
.setMaxConnPerRoute(100))
.setMaxConnTotal(config.getInt(150)).build();
and for Every request we gave timeout of
final RequestConfig params = RequestConfig.custom().setConnectTimeout(300000) // 5min
.setSocketTimeout(300000).setConnectionRequestTimeout(300000).build();
Please find the stack trace below:
org.apache.http.ConnectionClosedException: Connection closed
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:344) [httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261) [httpcore-nio-4.4.5.jar:na]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) [httpasyncclient-4.1.2.jar:na]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) [httpasyncclient-4.1.2.jar:na]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) [httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) [httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) [httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) [httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) [httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) [httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) [httpcore-nio-4.4.5.jar:4.4.5]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
Please Note this question is not duplicate of
Intermittent ConnectionClosedException with httpasynclient stacj trace is completely different..And event this issue happens intermittent.