I am using EKS for the deployment of my Spring Boot App.
At startup, it crashes a couple of times, gets restarted by EKS, and finally starts serving requests.
Here are the logs: -
2022-04-20 13:08:51.836 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.retry.annotation.RetryConfiguration' of type [org.springframework.retry.annotation.RetryConfiguration$$EnhancerBySpringCGLIB$$e8ec2216] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2022-04-20 13:08:51.848 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$f428cee] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2022-04-20 13:09:42.455 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application run failed
java.lang.IllegalStateException: Logback configuration error detected:
ERROR in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy#1248276879 - Unexpected exception while waiting for compression job to finish java.lang.InterruptedException
ERROR in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy#828088650 - Unexpected exception while waiting for compression job to finish java.lang.InterruptedException
ERROR in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy#1248276879 - Timeout while waiting for clean-up job to finish java.util.concurrent.TimeoutException
ERROR in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy#828088650 - Timeout while waiting for clean-up job to finish java.util.concurrent.TimeoutException
at org.springframework.boot.logging.logback.LogbackLoggingSystem.loadConfiguration(LogbackLoggingSystem.java:169)
at org.springframework.boot.logging.AbstractLoggingSystem.initializeWithConventions(AbstractLoggingSystem.java:82)
at org.springframework.boot.logging.AbstractLoggingSystem.initialize(AbstractLoggingSystem.java:60)
at org.springframework.boot.logging.logback.LogbackLoggingSystem.initialize(LogbackLoggingSystem.java:117)
at org.springframework.boot.context.logging.LoggingApplicationListener.initializeSystem(LoggingApplicationListener.java:290)
at org.springframework.boot.context.logging.LoggingApplicationListener.initialize(LoggingApplicationListener.java:263)
at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEnvironmentPreparedEvent(LoggingApplicationListener.java:226)
at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEvent(LoggingApplicationListener.java:199)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127)
at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:75)
at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:54)
at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:347)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:306)
at com.dt.Application.main(Application.java:20)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
First two lines are warnings as per my research so far. they shouldn't cause the app to fail (let me know if I am wrong).
Application run failed and 4 Logback errors were printed: -
In Logback's TimeBasedRollingPolicy, there's a start() and stop() method.
When the stop() method is called, two async jobs are run. Compression job and Clean up job. The compression job threw InterruptedException and the clean-up job threw TimeoutException.
I have a few questions: -
I am not sure whether these Logback errors caused the app to crash or vice-versa
What invoked the stop() method of TimeBasedRollingPolicy
What interrupted the Compression job?
If you have any idea why this is happening, please let me know.
Thanks :)
It says "Logback configuration error detected". I would check your log configuration file to see if you have a typo in it.
I am updating the newrelic version from 2.x.x to 5.x.x in a project. I followed steps as mentioned in the newrelic docs (I updated the dependency version in pom file, updated the agent jar file and the config file).
But now on running project passing the required javaagent:
java -server -javaagent:/Users/amit.upadhyay/abcproject/abcproject-app/newrelic-stage/newrelic.jar -jar abcproject-app/target/abcproject-app-*.*.*-SNAPSHOT.jar /Users/amit.upadhyay/Downloads/abcproject.yml
I get java.lang.AbstractMethodError. Below is the error log:
ERROR StatusLogger Caught java.lang.AbstractMethodError setting feature http://xml.org/sax/features/external-general-entities to false on DocumentBuilderFactory org.apache.xerces.jaxp.DocumentBuilderFactoryImpl#e320068: java.lang.AbstractMethodError: javax.xml.parsers.DocumentBuilderFactory.setFeature(Ljava/lang/String;Z)V java.lang.AbstractMethodError: javax.xml.parsers.DocumentBuilderFactory.setFeature(Ljava/lang/String;Z)V at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.xml.XmlConfiguration.setFeature(XmlConfiguration.java:212) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.xml.XmlConfiguration.disableDtdProcessing(XmlConfiguration.java:205) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.xml.XmlConfiguration.newDocumentBuilder(XmlConfiguration.java:194) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.xml.XmlConfiguration.<init>(XmlConfiguration.java:92) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.xml.XmlConfigurationFactory.getConfiguration(XmlConfigurationFactory.java:46) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:420) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:347) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:261) at com.newrelic.agent.deps.org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:616) at com.newrelic.agent.deps.org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:637) at com.newrelic.agent.deps.org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:231) at com.newrelic.agent.deps.org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:243) at com.newrelic.agent.deps.org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45) at com.newrelic.agent.deps.org.apache.logging.log4j.LogManager.getContext(LogManager.java:174) at com.newrelic.agent.deps.org.apache.logging.log4j.LogManager.getLogger(LogManager.java:648) at com.newrelic.agent.logging.Log4jLogger.<init>(Log4jLogger.java:68) at com.newrelic.agent.logging.Log4jLogger.create(Log4jLogger.java:415) at com.newrelic.agent.logging.Log4jLogManager.createRootLogger(Log4jLogManager.java:38) at com.newrelic.agent.logging.Log4jLogManager.initializeRootLogger(Log4jLogManager.java:98) at com.newrelic.agent.logging.Log4jLogManager.<init>(Log4jLogManager.java:34) at com.newrelic.agent.logging.Log4jLogManager.create(Log4jLogManager.java:254) at com.newrelic.agent.logging.AgentLogManager.createAgentLogManager(AgentLogManager.java:19) at com.newrelic.agent.logging.AgentLogManager.<clinit>(AgentLogManager.java:9) at com.newrelic.agent.Agent.<clinit>(Agent.java:59) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.newrelic.bootstrap.BootstrapAgent.startAgent(BootstrapAgent.java:179) at com.newrelic.bootstrap.BootstrapAgent.premain(BootstrapAgent.java:81) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386) at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401) ERROR StatusLogger Caught java.lang.AbstractMethodError setting feature http://xml.org/sax/features/external-parameter-entities to false on DocumentBuilderFactory org.apache.xerces.jaxp.DocumentBuilderFactoryImpl#e320068: java.lang.AbstractMethodError: javax.xml.parsers.DocumentBuilderFactory.setFeature(Ljava/lang/String;Z)V java.lang.AbstractMethodError: javax.xml.parsers.DocumentBuilderFactory.setFeature(Ljava/lang/String;Z)V at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.xml.XmlConfiguration.setFeature(XmlConfiguration.java:212) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.xml.XmlConfiguration.disableDtdProcessing(XmlConfiguration.java:206) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.xml.XmlConfiguration.newDocumentBuilder(XmlConfiguration.java:194) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.xml.XmlConfiguration.<init>(XmlConfiguration.java:92) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.xml.XmlConfigurationFactory.getConfiguration(XmlConfigurationFactory.java:46) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:420) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:347) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:261) at com.newrelic.agent.deps.org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:616) at com.newrelic.agent.deps.org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:637) at com.newrelic.agent.deps.org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:231) at com.newrelic.agent.deps.org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:243) at com.newrelic.agent.deps.org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45) at com.newrelic.agent.deps.org.apache.logging.log4j.LogManager.getContext(LogManager.java:174) at com.newrelic.agent.deps.org.apache.logging.log4j.LogManager.getLogger(LogManager.java:648) at com.newrelic.agent.logging.Log4jLogger.<init>(Log4jLogger.java:68) at com.newrelic.agent.logging.Log4jLogger.create(Log4jLogger.java:415) at com.newrelic.agent.logging.Log4jLogManager.createRootLogger(Log4jLogManager.java:38) at com.newrelic.agent.logging.Log4jLogManager.initializeRootLogger(Log4jLogManager.java:98) at com.newrelic.agent.logging.Log4jLogManager.<init>(Log4jLogManager.java:34) at com.newrelic.agent.logging.Log4jLogManager.create(Log4jLogManager.java:254) at com.newrelic.agent.logging.AgentLogManager.createAgentLogManager(AgentLogManager.java:19) at com.newrelic.agent.logging.AgentLogManager.<clinit>(AgentLogManager.java:9) at com.newrelic.agent.Agent.<clinit>(Agent.java:59) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.newrelic.bootstrap.BootstrapAgent.startAgent(BootstrapAgent.java:179) at com.newrelic.bootstrap.BootstrapAgent.premain(BootstrapAgent.java:81) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386) at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401) ERROR StatusLogger Caught java.lang.AbstractMethodError setting feature http://apache.org/xml/features/nonvalidating/load-external-dtd to false on DocumentBuilderFactory org.apache.xerces.jaxp.DocumentBuilderFactoryImpl#e320068: java.lang.AbstractMethodError: javax.xml.parsers.DocumentBuilderFactory.setFeature(Ljava/lang/String;Z)V java.lang.AbstractMethodError: javax.xml.parsers.DocumentBuilderFactory.setFeature(Ljava/lang/String;Z)V at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.xml.XmlConfiguration.setFeature(XmlConfiguration.java:212) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.xml.XmlConfiguration.disableDtdProcessing(XmlConfiguration.java:207) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.xml.XmlConfiguration.newDocumentBuilder(XmlConfiguration.java:194) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.xml.XmlConfiguration.<init>(XmlConfiguration.java:92) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.xml.XmlConfigurationFactory.getConfiguration(XmlConfigurationFactory.java:46) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:420) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:347) at com.newrelic.agent.deps.org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:261) at com.newrelic.agent.deps.org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:616) at com.newrelic.agent.deps.org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:637) at com.newrelic.agent.deps.org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:231) at com.newrelic.agent.deps.org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:243) at com.newrelic.agent.deps.org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45) at com.newrelic.agent.deps.org.apache.logging.log4j.LogManager.getContext(LogManager.java:174) at com.newrelic.agent.deps.org.apache.logging.log4j.LogManager.getLogger(LogManager.java:648) at com.newrelic.agent.logging.Log4jLogger.<init>(Log4jLogger.java:68) at com.newrelic.agent.logging.Log4jLogger.create(Log4jLogger.java:415) at com.newrelic.agent.logging.Log4jLogManager.createRootLogger(Log4jLogManager.java:38) at com.newrelic.agent.logging.Log4jLogManager.initializeRootLogger(Log4jLogManager.java:98) at com.newrelic.agent.logging.Log4jLogManager.<init>(Log4jLogManager.java:34) at com.newrelic.agent.logging.Log4jLogManager.create(Log4jLogManager.java:254) at com.newrelic.agent.logging.AgentLogManager.createAgentLogManager(AgentLogManager.java:19) at com.newrelic.agent.logging.AgentLogManager.<clinit>(AgentLogManager.java:9) at com.newrelic.agent.Agent.<clinit>(Agent.java:59) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.newrelic.bootstrap.BootstrapAgent.startAgent(BootstrapAgent.java:179) at com.newrelic.bootstrap.BootstrapAgent.premain(BootstrapAgent.java:81) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386) at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401) Mar 16, 2020 01:40:38 +0530 [8698 1] com.newrelic INFO: Configuration file not found. The agent will attempt to read required values from environment variables. Mar 16, 2020 01:40:38 +0530 [8698 1] com.newrelic INFO: Using default collector host: collector.newrelic.com Mar 16, 2020 01:40:38 +0530 [8698 1] com.newrelic ERROR: Unable to start the New Relic Agent. Your application will continue to run but it will not be monitored. com.newrelic.agent.config.ConfigurationException: The agent requires an application name. Check the app_name setting in newrelic.yml at com.newrelic.agent.config.ConfigServiceFactory.validateConfig(ConfigServiceFactory.java:64) at com.newrelic.agent.config.ConfigServiceFactory.createConfigService(ConfigServiceFactory.java:27)
Since I am encountering AbstractMethodError at runtime i.e. somewhere I am using an old interface that is updated in the new versions of newrelic java agent 5.x.x.
The error logs suggested that there is a conflict between xerces version being used in my project and what is there in the updated newrelic javaagent. On googling I found these two SO links first and second with similar kinda issue. According to answers given, I added
<dependency>
<groupId>xerces</groupId>
<artifactId>xercesImpl</artifactId>
<version>2.11.0</version>
</dependency>
<dependency>
<groupId>xml-apis</groupId>
<artifactId>xml-apis</artifactId>
<version>1.4.01</version>
</dependency>
to my pom file and this AbstractMethodError error was gone.
But then on running again I started getting this exception: com.newrelic.agent.config.ConfigurationException
The logs for this exception is:
Mar 16, 2020 02:37:18 +0530 [9598 1] com.newrelic INFO: Configuration file not found. The agent will attempt to read required values from environment variables.
Mar 16, 2020 02:37:18 +0530 [9598 1] com.newrelic INFO: Using default collector host: collector.newrelic.com
Mar 16, 2020 02:37:18 +0530 [9598 1] com.newrelic ERROR: Unable to start the New Relic Agent. Your application will continue to run but it will not be monitored.
com.newrelic.agent.config.ConfigurationException: The agent requires an application name. Check the app_name setting in newrelic.yml
at com.newrelic.agent.config.ConfigServiceFactory.validateConfig(ConfigServiceFactory.java:64) ~
at com.newrelic.agent.config.ConfigServiceFactory.createConfigService(ConfigServiceFactory.java:27) ~
at com.newrelic.agent.service.ServiceManagerImpl.<init>(ServiceManagerImpl.java:126) ~
at com.newrelic.agent.Agent.premain(Agent.java:279)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at com.newrelic.bootstrap.BootstrapAgent.startAgent(BootstrapAgent.java:179)
at com.newrelic.bootstrap.BootstrapAgent.premain(BootstrapAgent.java:81)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386) [?:1.8.0_222]
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401) [?:1.8.0_222]
Now, I don't understand why is this throwing ConfigurationException (or why is it not able to find the configuration file). The same java command (with exactly the same path of agent jar and configuration file) when used with an older version of newrelic agent(2.x.x or 3.x.x) runs perfectly.
I wanted to understand what is going wrong/missing here?
I was able to update to the latest newrelic version in my project. It seems that they have updated the xerces version in their java agent where version is greater than 5.0.0. Along with the fix of updating xerces version, I needed to pass the newrelic config file explicitly as system properties to resolve the ConfigurationException.
Overall summary:
- update these libraries to
<dependency>
<groupId>xerces</groupId>
<artifactId>xercesImpl</artifactId>
<version>2.11.0</version>
</dependency>
<dependency>
<groupId>xml-apis</groupId>
<artifactId>xml-apis</artifactId>
<version>1.4.01</version>
</dependency>
And pass the newrelic config file as system property:
-Dnewrelic.config.file=./${NEW_RELIC_DIR_NAME}/newrelic.yml
I'm currently running an application within Jboss. My application need to consume JMS message from an ActiveMQ (my ActiveMQ is a module of my Jboss, I followed this procedure : https://developer.jboss.org/wiki/IntegrationOfJBossAS7WithActiveMQ).
As you can see on that link, I use a MessageDrivenBean in order to consume messages on my queue :
#MessageDriven(activationConfig = {
#ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
#ActivationConfigProperty(propertyName = "destination", propertyValue = "Mongo-DB")})
public class MongoConsumer implements MessageListener {
#Override
public void onMessage(Message message) {
TextMessage textMessage = (TextMessage) message;
try {
json = textMessage.getText();
collectionId = JsonUtils.extract(json, "_collectionId");
uuid = JsonUtils.extract(json, "_uuid");
queriesMongoDB.save(collectionId, json);
LOGGER.info("Insert in mongo : {}", uuid);
} catch (TechnicalException e) {
LOGGER.error("Something went wrong while calling Mongo : {}", e);
this.rollbackMdb(json);
} catch (JMSException e) {
LOGGER.error("Something went wrong with the Mongo Consumer", e);
}
}
}
My session is currently started in Auto-Acknowledge mode :
connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
All of this is working really fine when I'm using it with few messages. But when I benchmark my application, and sending a lot of messages, sometimes, the MDB is consuming twice the same message, and saving it in MongoDB, twice again, of course.
The problem is, it's only happening when I have a lot of messages (like 200k), I got like ~10 duplicates.
It seems that a thread is taking the message, in order to process it, and in the meantime, an other thread is doing the exact same thing.
I also changed the type of my JMS session, in CLIENT_acknowledge mode, and adding :
message.acknowledge();
At the beggining of my method, but that didn't help.
PS : Sorry for my bad english.
EDIT :
I just reproduce the bug and read the server.log and I got this type of error for a duplicate :
16:41:35,376 WARN [org.apache.activemq.TransactionContext] (default-threads - 39) commit of: XID:[131077,globalId=0:ffff0a48263f:-669b4574:57e296f4:23fc95,branchId=0:f
fff0a48263f:-669b4574:57e296f4:23fc96] failed with: javax.jms.JMSException: Transaction 'XID:[131077,globalId=0:ffff0a48263f:-669b4574:57e296f4:23fc95,branchId=0:ffff0a
48263f:-669b4574:57e296f4:23fc96]' has not been started. xaErrorCode:-4: javax.jms.JMSException: Transaction 'XID:[131077,globalId=0:ffff0a48263f:-669b4574:57e296f4:23f
c95,branchId=0:ffff0a48263f:-669b4574:57e296f4:23fc96]' has not been started. xaErrorCode:-4
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54) [activemq-client-5.10.0.jar:5.10.0]
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1420) [activemq-client-5.10.0.jar:5.10.0]
at org.apache.activemq.TransactionContext.syncSendPacketWithInterruptionHandling(TransactionContext.java:761) [activemq-client-5.10.0.jar:5.10.0]
at org.apache.activemq.TransactionContext.commit(TransactionContext.java:562) [activemq-client-5.10.0.jar:5.10.0]
at org.apache.activemq.ra.LocalAndXATransaction.commit(LocalAndXATransaction.java:92)
at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:682)
at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2278)
at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1479)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:98)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1189)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.afterDelivery(MessageEndpointInvocationHandler.java:72) [jboss-as-ejb3.jar:7.5.7.Final-redhat-3]
at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source) [:1.8.0_66]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_66]
at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_66]
at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.handle(AbstractInvocationHandler.java:60) [jboss-as-ejb3.jar:7.5.7.Final-redhat-3]
at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.doInvoke(MessageEndpointInvocationHandler.java:136) [jboss-as-ejb3.jar:7.5.7.Final-redhat-3]
at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:73) [jboss-as-ejb3.jar:7.5.7.Final-redhat-3]
at com.sun.proxy.$Proxy94.afterDelivery(Unknown Source)
at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:128)
at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:69)
at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:225)
at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:1016) [activemq-client-5.10.0.jar:5.10.0]
at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:169)
at org.jboss.jca.core.workmanager.WorkWrapper.run(WorkWrapper.java:215)
at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)
at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:808)
at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)
at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:849)
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_66]
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: javax.transaction.xa.XAException: Transaction 'XID:[131077,globalId=0:ffff0a48263f:-669b4574:57e296f4:23fc95,branchId=0:ffff0a48263f:-669b4574:57e296f4:23fc96]' has not been started. xaErrorCode:-4
at org.apache.activemq.transaction.XATransaction.newXAException(XATransaction.java:174)
at org.apache.activemq.broker.TransactionBroker.getTransaction(TransactionBroker.java:368)
at org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:252)
at org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:117)
at org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:498)
at org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100) [activemq-client-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:334)
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)
at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) [activemq-client-5.10.0.jar:5.10.0]
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) [activemq-client-5.10.0.jar:5.10.0]
at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:248)
at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133) [activemq-client-5.10.0.jar:5.10.0]
at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48) [activemq-client-5.10.0.jar:5.10.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_66]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_66]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_66]
16:41:35,410 WARN [com.arjuna.ats.jta] (default-threads - 39) ARJUNA016039: onePhaseCommit on < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a48263f:-669b4574:57e296f4:23fc95, node_name=1, branch_uid=0:ffff0a48263f:-669b4574:57e296f4:23fc96, subordinatenodename=null, eis_name=unknown eis name > ([org.apache.activemq.ra.LocalAndXATransaction#7e5f0986,TransactionContext{transactionId=null,connection=ActiveMQConnection {id=ID:tsfla902v-34440-1474467574382-7:1,clientId=ID:tsfla902v-34440-1474467574382-6:1,started=true}}]) failed with exception XAException.XAER_NOTA: javax.transaction.xa.XAException: Transaction 'XID:[131077,globalId=0:ffff0a48263f:-669b4574:57e296f4:23fc95,branchId=0:ffff0a48263f:-669b4574:57e296f4:23fc96]' has not been started. xaErrorCode:-4
at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:786) [activemq-client-5.10.0.jar:5.10.0]
at org.apache.activemq.TransactionContext.commit(TransactionContext.java:595) [activemq-client-5.10.0.jar:5.10.0]
at org.apache.activemq.ra.LocalAndXATransaction.commit(LocalAndXATransaction.java:92)
at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:682)
at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2278)
at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1479)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:98)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1189)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.afterDelivery(MessageEndpointInvocationHandler.java:72) [jboss-as-ejb3.jar:7.5.7.Final-redhat-3]
at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source) [:1.8.0_66]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_66]
at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_66]
at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.handle(AbstractInvocationHandler.java:60) [jboss-as-ejb3.jar:7.5.7.Final-redhat-3]
at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.doInvoke(MessageEndpointInvocationHandler.java:136) [jboss-as-ejb3.jar:7.5.7.Final-redhat-3]
at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:73) [jboss-as-ejb3.jar:7.5.7.Final-redhat-3]
at com.sun.proxy.$Proxy94.afterDelivery(Unknown Source)
at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:128)
at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:69)
at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:225)
at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:1016) [activemq-client-5.10.0.jar:5.10.0]
at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:169)
at org.jboss.jca.core.workmanager.WorkWrapper.run(WorkWrapper.java:215)
at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)
at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:808)
at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)
at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:849)
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_66]
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: javax.transaction.xa.XAException: Transaction 'XID:[131077,globalId=0:ffff0a48263f:-669b4574:57e296f4:23fc95,branchId=0:ffff0a48263f:-669b4574:57e296f4:23fc96]' has not been started. xaErrorCode:-4
at org.apache.activemq.transaction.XATransaction.newXAException(XATransaction.java:174)
at org.apache.activemq.broker.TransactionBroker.getTransaction(TransactionBroker.java:368)
at org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:252)
at org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:117)
at org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:498)
at org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100) [activemq-client-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:334)
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)
at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) [activemq-client-5.10.0.jar:5.10.0]
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) [activemq-client-5.10.0.jar:5.10.0]
at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:248)
at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133) [activemq-client-5.10.0.jar:5.10.0]
at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48) [activemq-client-5.10.0.jar:5.10.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_66]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_66]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_66]
16:41:35,418 WARN [com.arjuna.ats.arjuna] (default-threads - 39) ARJUNA012084: One-phase commit of action 0:ffff0a48263f:-669b4574:57e296f4:23fc95 received heuristic decision: TwoPhaseOutcome.HEURISTIC_HAZARD
EDIT 2 :
It seems that somebody had the same problem here : https://issues.apache.org/jira/browse/AMQ-5953
The version of my ActiveMQ module is 5.11, I will try to install the 5.13 version, see if it's work. I keep you guys updated.
I finally found the solution ! As I explain it in an edit of my original post, the problem seems to be related to an XATransaction being destroyed while my code was processing the event. The transaction was recreated in the queue, and my code was processing it after.
I was getting this stack in my server.log :
16:41:35,376 WARN [org.apache.activemq.TransactionContext] (default-threads - 39) commit of: XID:[131077,globalId=0:ffff0a48263f:-669b4574:57e296f4:23fc95,branchId=0:f
fff0a48263f:-669b4574:57e296f4:23fc96] failed with: javax.jms.JMSException: Transaction 'XID:[131077,globalId=0:ffff0a48263f:-669b4574:57e296f4:23fc95,branchId=0:ffff0a
48263f:-669b4574:57e296f4:23fc96]' has not been started. xaErrorCode:-4: javax.jms.JMSException: Transaction 'XID:[131077,globalId=0:ffff0a48263f:-669b4574:57e296f4:23f
c95,branchId=0:ffff0a48263f:-669b4574:57e296f4:23fc96]' has not been started. xaErrorCode:-4
Then, I found this : https://issues.apache.org/jira/browse/AMQ-5953. My ActiveMQ module inside my JBoss was in version 5.11, I managed to install 5.14. Since now, I don't have any duplicates problem anymore.
I'm trying to boot spring start with after the Linux login more'm failing, can anyone tell me what am I doing wrong?
I've got two problems ..
I can run the application just by invoking the sudo user ,if it does not happen the following error
java.lang.IllegalStateException: Tomcat connector in failed state
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.start(TomcatEmbeddedServletContainer.java:157)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.startEmbeddedServletContainer(EmbeddedWebApplicationContext.java:288)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:140)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:483)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:117)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:689)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:321)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:969)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:958)
at br.com.nextinfo.MainApplication.main(MainApplication.java:28)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:53)
at java.lang.Thread.run(Thread.java:745)
2015-11-16 19:59:22.368 INFO 3400 --- [ main] ationConfigEmbeddedWebApplicationContext : Closing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#52f6002c: startup date [Mon Nov 16 19:59:13 BRST 2015]; root of context hierarchy
2015-11-16 19:59:22.369 INFO 3400 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Unregistering JMX-exposed beans on shutdown
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:53)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Tomcat connector in failed state
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.start(TomcatEmbeddedServletContainer.java:157)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.startEmbeddedServletContainer(EmbeddedWebApplicationContext.java:288)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:140)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:483)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:117)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:689)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:321)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:969)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:958)
at br.com.nextinfo.MainApplication.main(MainApplication.java:28)
... 6 more
To explain to me why this error occurs ..
How can I make application to run after the User login on linux, noting that the login occurs automatically without the sudo permission, is a jar should run with the permissions 755 right?
thank you advanced.
Should be straightforward.I believe you are using port 80 and not 8080:
Ports below 1024 are privileged ports and hence require sudo user
Refer Doc :
http://www.w3.org/Daemon/User/Installation/PrivilegedPorts.html
Either pass server port via Run VM arguments that is like -Dserver.port=8080 or search for
application.properties and specify server.port=8080 in it.
The above should work just fine.
Hope this helps.
Good day. I have some difficulties using the Hazelcast in Glassfish OSGi environment.
I have an OSGi bundle, which provides some EJBs and intended to be deployed on glassfish. This bundle uses Hazelcast and imports some hazelcast packages. I don't want to embed hazelcast jar so I tried to deploy it in Glassfish prior to deploying the subj. bundle.
$ asadmin deploy --type=osgi minimal-json-0.9.1.jar
$ asadmin deploy --type=osgi hazelcast-3.3.3.jar
Hazelcast deployed successfully. But when I deploy my bundle, and when one of the EJBs from this bundle tries to create a HazelcastInstance, I get the following exception:
org.glassfish.osgijavaeebase.DeploymentException: Deployment of xxx.xxx.platon-cache [398] failed because of following reason: Failed while deploying bundle xxx.xxx.platon-cache [398] : java.lang.RuntimeException: Failed to deploy bundle [ xxx.xxx.platon-cache [398] ], root cause: Exception while loading the app
at org.glassfish.osgijavaeebase.AbstractOSGiDeployer.deploy(AbstractOSGiDeployer.java:127)
at org.glassfish.osgijavaeebase.OSGiContainer.deploy(OSGiContainer.java:154)
at org.glassfish.osgijavaeebase.JavaEEExtender.deploy(JavaEEExtender.java:109)
at org.glassfish.osgijavaeebase.JavaEEExtender.access$200(JavaEEExtender.java:61)
at org.glassfish.osgijavaeebase.JavaEEExtender$HybridBundleTrackerCustomizer$1.call(JavaEEExtender.java:153)
at org.glassfish.osgijavaeebase.JavaEEExtender$HybridBundleTrackerCustomizer$1.call(JavaEEExtender.java:150)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Failed to deploy bundle [ xxx.xxx.platon-cache [398] ], root cause: Exception while loading the app
at org.glassfish.osgijavaeebase.OSGiDeploymentRequest.deploy(OSGiDeploymentRequest.java:198)
at org.glassfish.osgijavaeebase.OSGiDeploymentRequest.execute(OSGiDeploymentRequest.java:120)
at org.glassfish.osgijavaeebase.AbstractOSGiDeployer.deploy(AbstractOSGiDeployer.java:123)
... 9 more
Caused by: javax.ejb.EJBException: javax.ejb.CreateException: Initialization failed for Singleton HazelcastService
at com.sun.ejb.containers.AbstractSingletonContainer$SingletonContextFactory.create(AbstractSingletonContainer.java:656)
at com.sun.ejb.containers.AbstractSingletonContainer.instantiateSingletonInstance(AbstractSingletonContainer.java:396)
at org.glassfish.ejb.startup.SingletonLifeCycleManager.initializeSingleton(SingletonLifeCycleManager.java:219)
at org.glassfish.ejb.startup.SingletonLifeCycleManager.initializeSingleton(SingletonLifeCycleManager.java:180)
at org.glassfish.ejb.startup.SingletonLifeCycleManager.doStartup(SingletonLifeCycleManager.java:158)
at org.glassfish.ejb.startup.EjbApplication.start(EjbApplication.java:166)
at org.glassfish.internal.data.EngineRef.start(EngineRef.java:122)
at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:291)
at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:352)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:497)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:219)
at org.glassfish.osgijavaeebase.OSGiDeploymentRequest.deploy(OSGiDeploymentRequest.java:185)
... 11 more
Caused by: javax.ejb.CreateException: Initialization failed for Singleton HazelcastService
at com.sun.ejb.containers.AbstractSingletonContainer.createSingletonEJB(AbstractSingletonContainer.java:483)
at com.sun.ejb.containers.AbstractSingletonContainer.access$000(AbstractSingletonContainer.java:81)
at com.sun.ejb.containers.AbstractSingletonContainer$SingletonContextFactory.create(AbstractSingletonContainer.java:654)
... 22 more
Caused by: java.lang.IllegalArgumentException: PortableFactory[-17] is already registered! com.hazelcast.concurrent.atomiclong.client.AtomicLongPortableHook$1#682d1719 -> com.hazelcast.concurrent.atomiclong.client.AtomicLongPortableHook$1#68479fbd
at com.hazelcast.nio.serialization.PortableHookLoader.register(PortableHookLoader.java:84)
at com.hazelcast.nio.serialization.PortableHookLoader.load(PortableHookLoader.java:51)
at com.hazelcast.nio.serialization.PortableHookLoader.<init>(PortableHookLoader.java:41)
at com.hazelcast.nio.serialization.SerializationServiceImpl.<init>(SerializationServiceImpl.java:119)
at com.hazelcast.nio.serialization.SerializationServiceBuilder.build(SerializationServiceBuilder.java:175)
at com.hazelcast.instance.Node.createSerializationService(Node.java:195)
at com.hazelcast.instance.Node.<init>(Node.java:142)
at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:120)
at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:153)
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:136)
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:112)
at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:58)
at xxx.xxx.platon.cache.HazelcastService.init(HazelcastService.java:18)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.ejb.containers.interceptors.BeanCallbackInterceptor.intercept(InterceptorManager.java:1035)
at com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:72)
at com.sun.ejb.containers.interceptors.CallbackInvocationContext.proceed(CallbackInvocationContext.java:205)
at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.ejb.containers.interceptors.CallbackInterceptor.intercept(InterceptorManager.java:986)
at com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:72)
at com.sun.ejb.containers.interceptors.CallbackInvocationContext.proceed(CallbackInvocationContext.java:205)
at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.init(SystemInterceptorProxy.java:125)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.ejb.containers.interceptors.CallbackInterceptor.intercept(InterceptorManager.java:986)
at com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:72)
at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:412)
at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:375)
at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:1949)
at com.sun.ejb.containers.AbstractSingletonContainer.createSingletonEJB(AbstractSingletonContainer.java:475)
... 24 more
]]
The hazelcast instance is created in the only place, so there shouldn't be any conflicts with some other hazelcast cluster. Everything works correctly in non-osgi environment.
The hazelcast config is the simpliest one:
<?xml version="1.0" encoding="UTF-8"?>
<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.1.xsd" xmlns="http://www.hazelcast.com/schema/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<group>
<name>dev</name>
<password>dev-pass</password>
</group>
<network>
<port auto-increment="true" port-count="100">5701</port>
<outbound-ports>
<ports>0</ports>
</outbound-ports>
<join>
<multicast enabled="false">
<multicast-group>224.2.2.3</multicast-group>
<multicast-port>54327</multicast-port>
</multicast>
<tcp-ip enabled="true">
<interface>127.0.0.1</interface>
</tcp-ip>
</join>
</network>
</hazelcast>
What is the reason of this exception?
it smells very much like this issue: https://github.com/hazelcast/hazelcast/issues/3922.
Can you please try it with the latest Hazelcast 3.3.4-SNAPSHOT? It should be fixed there.
EDIT: This workaround is not needed anymore; Latest releases of both 3.3 and 3.4 branches include this fix.