I have a camel route which calls itself until a certain condition is met. Basically the idea is to implement the retry of route. When the application is deployed I am getting stackoverflow error when retries happen for a long period.
[Camel (camel-1) thread #1 - Multicast] ERROR com.application.RouteName.lambda$configure$0 - Exception occurred during execution on the exchange: Exchange[ID-batchrater-310822922-1-383133832-34058-1530798326741-0-6]
org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-batchrater-310822922-1-383133832-34058-1530798326741-0-6]
at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1779)
at org.apache.camel.impl.DefaultExchange.setException(DefaultExchange.java:351)
.
.
.
.
Caused by: java.lang.StackOverflowError: null
at org.springframework.beans.factory.support.AbstractBeanFactory.transformedBeanName(AbstractBeanFactory.java:1117)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:239)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084)
at org.apache.camel.spring.spi.ApplicationContextRegistry.lookupByNameAndType(ApplicationContextRegistry.java:47)
at org.apache.camel.impl.PropertyPlaceholderDelegateRegistry.lookupByNameAndType(PropertyPlaceholderDelegateRegistry.java:63)
at org.apache.camel.component.bean.BeanInfo.createParameterMappingStrategy(BeanInfo.java:177)
at org.apache.camel.component.bean.BeanInfo.<init>(BeanInfo.java:99)
I believe the stackoverflow error is due to the recursive call of route and I changed the route structure and now redelivery is handled by the retryDelivery mechanisms available in camel onException() . And my number of retries can be infinite until the condition is met.
I need to know will there be any chance of stackOverFlow again with this approach.
No this is the right approach to handle error handling redeliveries with the onException and other error handling features. Using the loop EIP leads to longer stack-frames and should not be used for looping very long. So you did the right fix.
Related
I'm trying to use Reactor Netty TcpClient in reactive way to interact with hosts, that may be unreachable. Here is an example of a channel initialization logic:
ConnectionProvider connectionProvider = ConnectionProvider.fixed("fixed", 50);
TcpClient.create(connectionProvider)
.host(host).port(port)
.wiretap(true)
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 50)
.doOnConnect(x -> log.trace("Connect to {}:{}", host, port))
.doOnConnected(conn -> log.trace("Connected {}", conn.channel()))
.connect()
.subscribe(this::utilizeConnection);
the output, that i receiving :
2019-09-04 08:23:13.612 TRACE 71988 --- [ioEventLoop-4-3] c.c.pcb.poc.network.tcp.NettyTcpSender : Connect to 192.168.88.210:2000
2019-09-04 08:23:13.684 WARN 71988 --- [actor-tcp-nio-4] io.netty.util.concurrent.DefaultPromise : An exception was thrown by reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.operationComplete()
reactor.core.Exceptions$ErrorCallbackNotImplemented: io.netty.channel.ConnectTimeoutException: connection timed out: /192.168.88.210:2000
Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: /192.168.88.210:2000
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:267) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) ~[netty-common-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127) ~[netty-common-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:405) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Assembly trace from producer [reactor.core.publisher.MonoCreate] :
reactor.core.publisher.Mono.create(Mono.java:183)
reactor.netty.resources.PooledConnectionProvider.acquire(PooledConnectionProvider.java:130)
Error has been observed by the following operator(s):
|_ Mono.create ⇢ reactor.netty.resources.PooledConnectionProvider.acquire(PooledConnectionProvider.java:130)
|_ Mono.doOnSubscribe ⇢ reactor.netty.tcp.TcpClientDoOn.connect(TcpClientDoOn.java:58)
The 'inbound' and 'outbound' are having a dedicated method to handle their errors, but they works on top of a Connection instance that won't be created if you got the 'connection timeout'.
I tried:
The exception, that i receiving is wrapped in 'ErrorCallbackNotImplemented'. But I wasn't able to find any way to implement any 'ErrorCallback'
The log contains a warning message from 'io.netty.util.concurrent.DefaultPromise' . but I wasn't able to find a way how to make own Promise to handle it in a right way.
No any configurations i've found that may somehow intercept connection timeouts.
workaround. The blocked approach to create a connection ( .block() instead of .subscribe()) will allow me to catch any Connection creating exceptions within plain try-catch block, but i'll lose the benefits of reactive approach with such workaround.
Do somebody may suggest me at least something to help me to find a right way to handle a 'io.netty.channel.ConnectTimeoutException'?
Do not forget to implement your error callback
Usually reactor.core.Exceptions$ErrorCallbackNotImplemented happens when there is subscription over labmda based .subscribe method (same for Mono and Flux).
If you are going to look at the sources here and here, you will find the place where reactor.core.Exceptions$ErrorCallbackNotImplemented is thrown!
Action Points
In order to handle the original io.netty.channel.ConnectTimeoutException I would recommend looking at Handling Errors section of the original Project Reactor documentation
(I didn't find proper answer from the existing questions that's why I posted this)
I have an application which processes quite big number of data. I am getting the below errors / exceptions and the process gets terminated. But increasing the ram size resolving the issue. (but we can't do it for some restriction)
2019-02-11 14:02:59,662 ERROR [net.xxx.RuleHandler] (Thread-185232 (HornetQ-client-global-threads-1521150484)) failed to get rules from db: org.hibernate.HibernateException: Transaction was rolled back in a different thread!
at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorTrackingImpl.processAnyDelayedAfterCompletion(SynchronizationCallbackCoordinatorTrackingImpl.java:105) [hibernate-core-4.2.27.Final-redhat-1.jar:4.2.27.Final-redhat-1]
2019-02-11 14:02:59,693 ERROR [net.xxx.ejb.SearchReqMDB] (Thread-185232 (HornetQ-client-global-threads-1521150484)) Something failed while search indexing: net.xxx.xxx.JMPException: javax.ejb.EJBTransactionRolledbackException: Transaction rolled back
How to resolve this issue? Do I need to increase the timeout value for ejb call? and if yes then where I need to change it.
Using JBOSS 6 EAP , EJB & JPA
Here is an overview of a crash on fabric.io
java.lang.IllegalStateException - Fatal exception thrown on Scheduler.Worker thread
rx.exceptions.OnErrorFailedException - an exception originating somewhere in the depth of rx and ending in SafeSubscriber#_onError.
rx.exceptions.CompositeException - 2 exceptions occured
Casual chain - that's the interesting part
Caused by rx.exceptions.CompositeException$CompositeExceptionCausalChain: Chain of Causes for CompositeException In Order Received =>
at com.crashlytics.android.core.TrimmedThrowableData.<init>(TrimmedThrowableData.java:19)
at com.crashlytics.android.core.TrimmedThrowableData.<init>(TrimmedThrowableData.java:20)
at com.crashlytics.android.core.TrimmedThrowableData.<init>(TrimmedThrowableData.java:20)
at com.crashlytics.android.core.CrashlyticsController.writeSessionEvent(CrashlyticsController.java:1090)
at com.crashlytics.android.core.CrashlyticsController.writeFatal(CrashlyticsController.java:852)
at com.crashlytics.android.core.CrashlyticsController.access$400(CrashlyticsController.java:59)
at com.crashlytics.android.core.CrashlyticsController$6.call(CrashlyticsController.java:292)
at com.crashlytics.android.core.CrashlyticsController$6.call(CrashlyticsController.java:285)
at java.util.concurrent.FutureTask.run(FutureTask.java:237)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
at io.fabric.sdk.android.services.common.ExecutorUtils$1$1.onRun(ExecutorUtils.java:75)
at io.fabric.sdk.android.services.common.BackgroundPriorityRunnable.run(BackgroundPriorityRunnable.java:30)
at java.lang.Thread.run(Thread.java:776)
retrofit.RetrofitError - 504 Gateway Timeout - probably supposed to be handled in one of the on onError
The version of Rx is 1.3.4. At first I thought that Rx missed the actual exception that occurred in one of the onError methods, but I found this issue https://github.com/ReactiveX/RxJava/issues/3679
and it seems to be resolved, by looking at source code of SafeSubscriber
https://github.com/ReactiveX/RxJava/blob/1.x/src/main/java/rx/observers/SafeSubscriber.java
it seems like rx should report the exception from onError correctly.
So, there is a CompositeException containing a retrofit exception, which is supposed to happen from time to time, and a stacktrace with references to crashlytics code with no references to any app code. I wonder what I should make out of this.
I have got the below exception , I suspect heap memory is full so GC exception was thrown . Kindly explain if any other perspective for the below application solution
2017:06:07 21:18:36.275 [loginputtaskexecutor-7] ERROR o.s.i.handler.LoggingHandler - org.springframework.messaging.MessageHandlingException: nested exception is java.lang.IllegalStateException: Cannot process message
at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:96)
at org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:89)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:109)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:148)
at org.springframework.integration.dispatcher.UnicastingDispatcher.access$000(UnicastingDispatcher.java:53)
at org.springframework.integration.dispatcher.UnicastingDispatcher$3.run(UnicastingDispatcher.java:129)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Cannot process message
at org.springframework.integration.util.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:333)
at org.springframework.integration.util.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:155)
at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:93)
... 11 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
**Application flow in detail :**
Spring integration application build to listen message from ActiveMQ , after consuming message from ActiveMQ it will be handed over to input channel (Executor channel) which has subscriber as service activator . In Service Activator message is converted to json then stored to Cassandra . # transaction was mentioned on the Service activator method .
With the above solution , I thought of breaking Message transaction flow by implementing executor channel , after message consumed it will be handed over to executor channel and the transaction ends . after then threads in executor channel would take care of performing parallel write to cassandra.
Is there any better way to write as fast as possible for large volume of data to casandra using java spring integration
If the data sink can't keep up, add a limit to the queue size in the TaskExecutor and use a CallerRunsPolicy or CallerBlocksPolicy when the queue is full.
That will naturally throttle the workload at the rate the sink can deal with.
I am having an Issue where when the hystrix circuit breaker trips, it does not close ever again. I have turned logging to debug and I do not see it trying to allow a test request through in which case it seems to me It will never close since its only supposed to close when a test execution goes through successfully indicating that the offending service is now healthy. According to the documentation the Circuit break configuration defaults should be working but I can not seem to tell why the test request is never allowed through.
2016-02-18 09:00:38,782 noodle-soup-service application-akka.actor.default-dispatcher-7 ERROR akka.actor.OneForOneStrategy - CallServiceCommand short-circuited and fallback failed.
com.netflix.hystrix.exception.HystrixRuntimeException: CallServiceCommand short-circuited and fallback failed.
at com.netflix.hystrix.AbstractCommand$16.call(AbstractCommand.java:816) ~[com.netflix.hystrix.hystrix-core-1.4.23.jar:1.4.23]
at com.netflix.hystrix.AbstractCommand$16.call(AbstractCommand.java:790) ~[com.netflix.hystrix.hystrix-core-1.4.23.jar:1.4.23]
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$1.onError(OperatorOnErrorResumeNextViaFunction.java:99) ~[io.reactivex.rxjava-1.1.0.jar:1.1.0]
at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71) ~[io.reactivex.rxjava-1.1.0.jar:1.1.0]
...
Caused by: java.lang.RuntimeException: Hystrix circuit short-circuited and is OPEN
at com.netflix.hystrix.AbstractCommand$1.call(AbstractCommand.java:414) ~[com.netflix.hystrix.hystrix-core-1.4.23.jar:1.4.23]
... 38 common frames omitted
In the Akka error handling strategy, I needed to schedule a new test call so that eventually when it succeeds, then Hystrix can close the Circuit.