We Keep Coding

Received SubsegmentNotFoundException when try to use AWS X-Ray with java spring #Scheduled

I am implementing AWS X-Ray in my Java Spring project, and generally speaking, it is working correctly.
Log requests and I can see the tracking on the AWS console.
However, in a specific case, it is generating an exception when starting the application.
I debugged the code and the problem is in a task scheduling class, which is annotated with #EnableScheduling.
If there is any method in that class with the #Scheduled annotation, the application returns the following error:
2020-12-17 22:50:51.427 ERROR 24379 --- [pool-2-thread-1] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task.
com.amazonaws.xray.exceptions.SubsegmentNotFoundException: Failed to end subsegment: subsegment cannot be found.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_265]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_265]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_265]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_265]
at com.amazonaws.xray.strategy.RuntimeErrorContextMissingStrategy.contextMissing(RuntimeErrorContextMissingStrategy.java:32) ~[aws-xray-recorder-sdk-core-2.8.0.jar:na]
at com.amazonaws.xray.contexts.ThreadLocalSegmentContext.endSubsegment(ThreadLocalSegmentContext.java:93) ~[aws-xray-recorder-sdk-core-2.8.0.jar:na]
at com.amazonaws.xray.AWSXRayRecorder.endSubsegment(AWSXRayRecorder.java:627) ~[aws-xray-recorder-sdk-core-2.8.0.jar:na]
at com.amazonaws.xray.AWSXRay.endSubsegment(AWSXRay.java:125) ~[aws-xray-recorder-sdk-core-2.8.0.jar:na]
at com.amazonaws.xray.spring.aop.BaseAbstractXRayInterceptor.processXRayTrace(BaseAbstractXRayInterceptor.java:58) ~[aws-xray-recorder-sdk-spring-2.8.0.jar:na]
at com.amazonaws.xray.spring.aop.BaseAbstractXRayInterceptor.traceAroundMethods(BaseAbstractXRayInterceptor.java:43) ~[aws-xray-recorder-sdk-spring-2.8.0.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_265]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_265]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_265]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_265]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627) ~[spring-aop-4.3.19.RELEASE.jar:4.3.19.RELEASE]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616) ~[spring-aop-4.3.19.RELEASE.jar:4.3.19.RELEASE]
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) ~[spring-aop-4.3.19.RELEASE.jar:4.3.19.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168) ~[spring-aop-4.3.19.RELEASE.jar:4.3.19.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.3.19.RELEASE.jar:4.3.19.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.19.RELEASE.jar:4.3.19.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671) ~[spring-aop-4.3.19.RELEASE.jar:4.3.19.RELEASE]
at com.domain.common.ToolBox$$EnhancerBySpringCGLIB$$a2712230.convertStackTraceToString(<generated>) ~[classes/:na]
at com.domain.event.ScheduledEventHandler.runsEvery1Minute(ScheduledEventHandler.java:111) ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_265]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_265]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_265]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_265]
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65) ~[spring-context-4.3.19.RELEASE.jar:4.3.19.RELEASE]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-4.3.19.RELEASE.jar:4.3.19.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_265]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_265]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_265]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_265]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_265]
I tried to use custom subsegments, as: Generating custom subsegments with the X-Ray SDK for Java and nothing.
The code would look something like this:
#XRayEnabled
#Component
#EnableScheduling
public class ScheduledEventHandler {
...
#Scheduled(fixedDelay = 1*60*1000)
protected void runsEvery1Minute() {
...
}
...
}
It seems to me that the #XRayEnabled annotation has some incompatibility with the #Scheduled annotation or there is a specific configuration for this case.
Can anyone help me with this?
Thanks!

I have faced with the same issue.
You can move scheduled method to the different class which is not annotated with #XRayEnabled.
Then you can just add
AWSXRay.beginSegment("youSegmentName");
...
call class annotated with #XRayEnabled
...
AWSXRay.endSegment();

Jenkins Declarative pipeline issue- unable to resolve class edu.umd.cs.findbugs.annotations.SuppressFBWarnings

I tried to run below Declarative Pipeline
pipeline {
agent any
stages {
stage('Hello') {
steps {
echo 'Hello World'
}
}
}
}
I got the below error
hudson.remoting.ProxyException: org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
jar:file:/home/durgaprasad/.jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy: 29: unable to resolve class edu.umd.cs.findbugs.annotations.SuppressFBWarnings
# line 29, column 1.
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings
^
below is the Exception stack trace
1 error
at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:958)
at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:554)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:250)
at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:766)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:170)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelStepLoader.getValue(ModelStepLoader.java:60)
at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:160)
at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:158)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:162)
at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
at WorkflowScript.run(WorkflowScript:1)
at ___cps.transform___(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
at com.cloudbees.groovy.cps.Next.step(Next.java:83)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:405)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:317)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:281)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Note:
I am using the following versions
Jenkins ver. 2.163 , Pipeline 2.6 , pipeline: Declarative plugin 1.4.0 .
according to the documentation this declarative pipeline should work with Pipline 2.5.
https://jenkins.io/doc/book/pipeline/jenkinsfile/
Please help me to resolve this error

java.lang.NoClassDefFoundError: org/apache/qpid/proton/engine/Collector in AmqpProviderFactory.createProvider

I'm getting the below exception thrown at AmqpProviderFactory.createProvider in qpid-jms-client-0.40.0.jar
java.lang.NoClassDefFoundError: org/apache/qpid/proton/engine/Collector
at org.apache.qpid.jms.provider.amqp.AmqpProviderFactory.createProvider(AmqpProviderFactory.java:69) ~[qpid-jms-client-0.40.0.jar:?]
at org.apache.qpid.jms.provider.amqp.AmqpProviderFactory.createProvider(AmqpProviderFactory.java:31) ~[qpid-jms-client-0.40.0.jar:?]
at org.apache.qpid.jms.provider.ProviderFactory.create(ProviderFactory.java:103) ~[qpid-jms-client-0.40.0.jar:?]
at org.apache.qpid.jms.provider.ProviderFactory.create(ProviderFactory.java:82) ~[qpid-jms-client-0.40.0.jar:?]
at org.apache.qpid.jms.JmsConnectionFactory.createProvider(JmsConnectionFactory.java:332) ~[qpid-jms-client-0.40.0.jar:?]
at org.apache.qpid.jms.JmsConnectionFactory.createConnection(JmsConnectionFactory.java:203) ~[qpid-jms-client-0.40.0.jar:?]
at org.apache.qpid.jms.JmsConnectionFactory.createConnection(JmsConnectionFactory.java:194) ~[qpid-jms-client-0.40.0.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_131]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_131]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) ~[spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at com.sun.proxy.$Proxy715.createConnection(Unknown Source) ~[?:?]
at org.springframework.jms.connection.SingleConnectionFactory.doCreateConnection(SingleConnectionFactory.java:365) ~[spring-jms-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.jms.connection.SingleConnectionFactory.initConnection(SingleConnectionFactory.java:305) ~[spring-jms-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.jms.connection.SingleConnectionFactory.getConnection(SingleConnectionFactory.java:283) ~[spring-jms-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.jms.connection.SingleConnectionFactory.createConnection(SingleConnectionFactory.java:224) ~[spring-jms-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:180) ~[spring-jms-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer.refreshConnectionUntilSuccessful(DefaultMessageListenerContainer.java:928) ~[spring-jms-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer.recoverAfterListenerSetupFailure(DefaultMessageListenerContainer.java:899) ~[spring-jms-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1077) ~[spring-jms-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Some of the dependencies I have in the classpath are
qpid-jms-client:0.40.0
proton-j:0.31.0
azure-eventhubs:2.2.0
Is there any other dependency that I should add to this to make it work?

The proton event collector class is a pretty basic bit of the proton-j library that has been in there for years so my guess is that you don't have the libraries on your classpath that you think you do. The failure indicates it cannot be found and the only reason for that should be that the library simply isn't there.

Unable to start flight recorder via JMC 6 within eclipse oxygen (4.7)

I have an error I don't understand. If I try to to use JFR using the JMC eclipse plugin it fails with the exception
com.oracle.jmc.rjmx.services.jfr.FlightRecorderException: Could not start the recording!
at com.oracle.jmc.rjmx.services.jfr.internal.FlightRecorderServiceV1.start(FlightRecorderServiceV1.java:149)
at com.oracle.jmc.flightrecorder.controlpanel.ui.jobs.StartRecordingJob.run(StartRecordingJob.java:76)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:56)
Caused by: java.io.IOException: javax.management.openmbean.OpenDataException: recording.jfr (Access denied)
at com.oracle.jmc.rjmx.services.jfr.internal.FlightRecorderCommunicationHelperV1.invokeOperation(FlightRecorderCommunicationHelperV1.java:62)
at com.oracle.jmc.rjmx.services.jfr.internal.FlightRecorderServiceV1.start(FlightRecorderServiceV1.java:137)
... 2 more
Caused by: javax.management.MBeanException: javax.management.openmbean.OpenDataException: recording.jfr (Access denied)
at com.sun.jmx.mbeanserver.MBeanIntrospector.unwrapInvocationTargetException(MBeanIntrospector.java:313)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:239)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at javax.management.StandardMBean.invoke(StandardMBean.java:405)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:361)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:283)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:260)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020)
at com.oracle.jmc.rjmx.internal.MCMBeanServerConnection.invoke(MCMBeanServerConnection.java:224)
at com.oracle.jmc.rjmx.ConnectionToolkit.invokeOperation(ConnectionToolkit.java:152)
at com.oracle.jmc.rjmx.services.jfr.internal.FlightRecorderCommunicationHelperV1.invokeJfrOperation(FlightRecorderCommunicationHelperV1.java:69)
at com.oracle.jmc.rjmx.services.jfr.internal.FlightRecorderCommunicationHelperV1.invokeOperation(FlightRecorderCommunicationHelperV1.java:56)
... 3 more
Caused by: javax.management.openmbean.OpenDataException: recording.jfr (Access denied)
at oracle.jrockit.jfr.FlightRecorder.setRecordingOptions(FlightRecorder.java:221)
at oracle.jrockit.jfr.FlightRecorder.setRecordingOptions(FlightRecorder.java:230)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at javax.management.StandardMBean.invoke(StandardMBean.java:405)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:361)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: recording.jfr (Access denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at java.io.FileWriter.<init>(FileWriter.java:90)
at oracle.jrockit.jfr.Recording.setDestination(Recording.java:464)
at oracle.jrockit.jfr.Recording.setOptions(Recording.java:481)
at oracle.jrockit.jfr.FlightRecorder.setRecordingOptions(FlightRecorder.java:219)
... 39 more
but if I use the JMC and JFR provided with the jdk (8_162) itself it works like charme. It is also the same jdk I use for eclipse. The exception idicates that there is some kind of file access problem, so I tried different locations but nothing worked so far.
I also tried to use ...jmxremote.access.file with the roles explained in the documentation. But still within eclipse it didn't work.
Note: parameter I currently use
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=3614
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
All other questions I found so far didn't help me any further so other ideas?
Thanks
Christian

When you start a recording, the JVM writes a placeholder file where the recording will be written when it is finished.
FileWriter fw = new FileWriter("recording.jfr");
fw.close();
The OS is denying the user, the process owner of the Java application you are connecting to, permission to write to that directory. If no path is given, i.e. "recording.jfr", the file will typically end up in the directory where the process was started.

Spring Batch JobExecutionException - ConcurrencyFailureException

I am unable to reproduce this error in my DEV environment, but it is happening randomly in our staging and QA environments. It has not happened in PROD yet, but that is probably just luck of the draw. Since I can not reproduce the error, I am at a loss for how to troubleshoot it. My best guess from reading the error dumps is that the Spring Job Transaction table is not properly handling the multithreaded nature of the app.
The code takes data from many sources, does some transforms, then creates data files to send to other sources. There is no need to save state between runs, on error we run a cleanup script, and start the process fresh. It is a single "job" but has many steps, and the steps use FlowBuilder to run steps in parallel.
A rerun of the process always clears it. Yesterday I ran the process over and over in my DEV environment and could not get it to happen a single time.
Here is the bean in question:
#Bean
public DataSource dataSource() {
EmbeddedDatabaseBuilder builder = new EmbeddedDatabaseBuilder();
EmbeddedDatabase db = builder
.setType(EmbeddedDatabaseType.HSQL)
.build();
return db;
}
Here is the error:
2017-06-13 07:16:18.543 ERROR 403 --- [ main] o.s.batch.core.job.AbstractJob : Encountered fatal error executing job
org.springframework.batch.core.JobExecutionException: Flow execution ended unexpectedly
at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:140) ~[spring-batch-core-3.0.6.RELEASE.jar!/:3.0.6.RELEASE]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:306) ~[spring-batch-core-3.0.6.RELEASE.jar!/:3.0.6.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135) [spring-batch-core-3.0.6.RELEASE.jar!/:3.0.6.RELEASE]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:128) [spring-batch-core-3.0.6.RELEASE.jar!/:3.0.6.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127) [spring-batch-core-3.0.6.RELEASE.jar!/:3.0.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at com.sun.proxy.$Proxy34.run(Unknown Source) [na:na]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:215) [spring-boot-autoconfigure-1.3.1.RELEASE.jar!/:1.3.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:232) [spring-boot-autoconfigure-1.3.1.RELEASE.jar!/:1.3.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:124) [spring-boot-autoconfigure-1.3.1.RELEASE.jar!/:1.3.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:118) [spring-boot-autoconfigure-1.3.1.RELEASE.jar!/:1.3.1.RELEASE]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:804) [spring-boot-1.3.1.RELEASE.jar!/:1.3.1.RELEASE]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:788) [spring-boot-1.3.1.RELEASE.jar!/:1.3.1.RELEASE]
at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:775) [spring-boot-1.3.1.RELEASE.jar!/:1.3.1.RELEASE]
at org.springframework.boot.SpringApplication.doRun(SpringApplication.java:366) [spring-boot-1.3.1.RELEASE.jar!/:1.3.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:305) [spring-boot-1.3.1.RELEASE.jar!/:1.3.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1124) [spring-boot-1.3.1.RELEASE.jar!/:1.3.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1113) [spring-boot-1.3.1.RELEASE.jar!/:1.3.1.RELEASE]
at com.dcsg.batch.Main.main(Main.java:146) [UniversalFeedGenerator.jar!/:VERSION 1.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:53) [UniversalFeedGenerator.jar!/:VERSION 1.1]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Caused by: org.springframework.batch.core.job.flow.FlowExecutionException: Ended flow=caliaFlow at state=caliaFlow.Calia Writer with exception
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:178) ~[spring-batch-core-3.0.6.RELEASE.jar!/:3.0.6.RELEASE]
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) ~[spring-batch-core-3.0.6.RELEASE.jar!/:3.0.6.RELEASE]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:93) ~[spring-batch-core-3.0.6.RELEASE.jar!/:3.0.6.RELEASE]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:90) ~[spring-batch-core-3.0.6.RELEASE.jar!/:3.0.6.RELEASE]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_121]
... 1 common frames omitted
Caused by: org.springframework.dao.ConcurrencyFailureException: PreparedStatementCallback; SQL [INSERT INTO BATCH_STEP_EXECUTION_CONTEXT (SHORT_CONTEXT, SERIALIZED_CONTEXT, STEP_EXECUTION_ID) VALUES(?, ?, ?)]; transaction rollback: serialization failure; nested exception is java.sql.SQLTransactionRollbackException: transaction rollback: serialization failure
at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:71) ~[spring-jdbc-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) ~[spring-jdbc-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) ~[spring-jdbc-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:645) ~[spring-jdbc-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:866) ~[spring-jdbc-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:927) ~[spring-jdbc-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.persistSerializedContext(JdbcExecutionContextDao.java:233) ~[spring-batch-core-3.0.6.RELEASE.jar!/:3.0.6.RELEASE]
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.saveExecutionContext(JdbcExecutionContextDao.java:187) ~[spring-batch-core-3.0.6.RELEASE.jar!/:3.0.6.RELEASE]
at org.springframework.batch.core.repository.support.SimpleJobRepository.add(SimpleJobRepository.java:171) ~[spring-batch-core-3.0.6.RELEASE.jar!/:3.0.6.RELEASE]
at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281) ~[spring-tx-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at com.sun.proxy.$Proxy32.add(Unknown Source) ~[na:na]
at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127) [spring-batch-core-3.0.6.RELEASE.jar!/:3.0.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at com.sun.proxy.$Proxy32.add(Unknown Source) ~[na:na]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:144) ~[spring-batch-core-3.0.6.RELEASE.jar!/:3.0.6.RELEASE]
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64) ~[spring-batch-core-3.0.6.RELEASE.jar!/:3.0.6.RELEASE]
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67) ~[spring-batch-core-3.0.6.RELEASE.jar!/:3.0.6.RELEASE]
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) ~[spring-batch-core-3.0.6.RELEASE.jar!/:3.0.6.RELEASE]
... 5 common frames omitted
Caused by: java.sql.SQLTransactionRollbackException: transaction rollback: serialization failure
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source) ~[hsqldb-2.3.3.jar!/:2.3.3]
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source) ~[hsqldb-2.3.3.jar!/:2.3.3]
at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source) ~[hsqldb-2.3.3.jar!/:2.3.3]
at org.hsqldb.jdbc.JDBCPreparedStatement.executeUpdate(Unknown Source) ~[hsqldb-2.3.3.jar!/:2.3.3]
at org.springframework.jdbc.core.JdbcTemplate$2.doInPreparedStatement(JdbcTemplate.java:873) ~[spring-jdbc-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate$2.doInPreparedStatement(JdbcTemplate.java:866) ~[spring-jdbc-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:629) ~[spring-jdbc-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
... 36 common frames omitted
Caused by: org.hsqldb.HsqlException: transaction rollback: serialization failure
at org.hsqldb.error.Error.error(Unknown Source) ~[hsqldb-2.3.3.jar!/:2.3.3]
at org.hsqldb.error.Error.error(Unknown Source) ~[hsqldb-2.3.3.jar!/:2.3.3]
at org.hsqldb.Session.executeCompiledStatement(Unknown Source) ~[hsqldb-2.3.3.jar!/:2.3.3]
at org.hsqldb.Session.execute(Unknown Source) ~[hsqldb-2.3.3.jar!/:2.3.3]
... 41 common frames omitted

Changing the data source to this
  #Bean
  public BasicDataSource dataSource() {
    BasicDataSource dataSource = new BasicDataSource();
    dataSource.setDriverClassName("org.hsqldb.jdbcDriver");
    dataSource.setUrl("jdbc:hsqldb:mem:testdb;sql.enforce_strict_size=true;hsqldb.tx=mvcc");
    dataSource.setUsername("sa");
    dataSource.setPassword("");
    return dataSource;
  }
Fixed it. The key change is using "tx=mvcc" which the Embedded Datasource does not support. That enables "Multiversion concurrency control"
This question explains what MVCC is and why to use it in the answers:
Database: What is Multiversion Concurrency Control (MVCC) and who supports it?