Log4J: FATAL exception on shutdown hook, can't figure out why - java

I just integrated log4j into my application, and I'm setting up AspectJ to use log4j and ultimately transfer all logging out of my main code. So, in short, I'm setting up aspects that execute log entries, and then running my program for short periods of time to test them.
Frequently, during these short runs, I'm getting the following exception:
2015-09-28 15:21:48,222 Thread-4 FATAL Unable to register shutdown hook because JVM is shutting down. java.lang.IllegalStateException: Cannot add new shutdown hook as this is not started. Current state: STOPPED
at org.apache.logging.log4j.core.util.DefaultShutdownCallbackRegistry.addShutdownCallback(DefaultShutdownCallbackRegistry.java:113)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.addShutdownCallback(Log4jContextFactory.java:271)
at org.apache.logging.log4j.core.LoggerContext.setUpShutdownHook(LoggerContext.java:240)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:201)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:233)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:162)
at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:507)
at org.jboss.logging.Log4j2Logger.<init>(Log4j2Logger.java:36)
at org.jboss.logging.Log4j2LoggerProvider.getLogger(Log4j2LoggerProvider.java:30)
at org.jboss.logging.Log4j2LoggerProvider.getLogger(Log4j2LoggerProvider.java:26)
at org.jboss.logging.Logger.getLogger(Logger.java:2465)
at org.jboss.logging.Logger$1.run(Logger.java:2565)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
at org.hibernate.internal.CoreLogging.messageLogger(CoreLogging.java:28)
at org.hibernate.internal.CoreLogging.messageLogger(CoreLogging.java:24)
at org.hibernate.event.internal.ReattachVisitor.<clinit>(ReattachVisitor.java:27)
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.performUpdate(DefaultSaveOrUpdateEventListener.java:293)
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.entityIsDetached(DefaultSaveOrUpdateEventListener.java:227)
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.performSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:92)
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:73)
at org.hibernate.internal.SessionImpl.fireSaveOrUpdate(SessionImpl.java:647)
at org.hibernate.internal.SessionImpl.saveOrUpdate(SessionImpl.java:639)
at org.hibernate.internal.SessionImpl.saveOrUpdate(SessionImpl.java:634)
at io.craigmiller160.stockmarket.controller.HibernatePortfolioDAO.savePortfolio(HibernatePortfolioDAO.java:225)
at io.craigmiller160.stockmarket.controller.StockMarketController.savePortfolio(StockMarketController.java:1441)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:353)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:276)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:925)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:932)
at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:997)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:973)
at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:893)
Occasionally, but not always, this exception is preceded by:
2015-09-28 15:21:48,152 Thread-4 ERROR An exception occurred processing Appender asyncAppender java.lang.IllegalStateException: AsyncAppender asyncAppender is not active
at org.apache.logging.log4j.core.appender.AsyncAppender.append(AsyncAppender.java:136)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:148)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:121)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:112)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:80)
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:390)
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:378)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:362)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:352)
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:59)
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:138)
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:999)
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:954)
at org.apache.logging.log4j.jcl.Log4jLog.debug(Log4jLog.java:81)
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:142)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:155)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:120)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:382)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:87)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:112)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:230)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:237)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:213)
at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52)
at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1443)
at io.craigmiller160.stockmarket.controller.HibernatePortfolioDAO.savePortfolio(HibernatePortfolioDAO.java:224)
at io.craigmiller160.stockmarket.controller.StockMarketController.savePortfolio(StockMarketController.java:1441)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:353)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:276)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:925)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:932)
at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:997)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:973)
at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:893)
Let me know if you need to see anything else to determine the cause of this. It doesn't always happen, but it's happening quite frequently when the program is run for only 10-20 seconds. I just don't get why this is happening.

Log4j2 try to use shutdown hook to close the logging service properly.
Shutdown hooks are executed in parallel and that's why the exception is not systematic.
There is a bug report regarding this problem.
You can disable shutdown hook in your configuration file :
<configuration shutdownHook="disable" ...>

Also see this link on log4j2's page:
https://logging.apache.org/log4j/2.x/manual/webapp.html
If you include this dependency, it will not try to add a shutdown hook.
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-web</artifactId>
<version>${log4j.version}</version>
</dependency>

As already mentioned here log4j-web is the jar for this problem. It worked without any problem for jetty-9.2.1.v20140609.
libraryDependencies += "org.apache.logging.log4j" % "log4j-web" % "2.5"
If you check the log4j-web (upto v2.9), it has log4j.shutdownHookEnabled=false
But with Servlet 3.0 and Tomcat 7.0.73, had to remove log4j from jarsToSkip key of conf/catalina.properties.
Strange things might happen. As for me, I am deploying two .wars, both using Non blocking Logger, worked for one service but not for the other. Tomcat 7/Websphere 8 restart cleaned it up though.
Reason for need to clean up logging resource is described in Using Log4j 2 in Web Applications -
when the Servlet Container shuts down or the web app is undeployed,
It's important for logging resources to be properly cleaned up
(database connections closed, files closed, etc.).
Because of the nature of ClassLoaders within web apps, Log4j resources cannot be
cleaned up through normal means. Log4j must be "started" when the web
app deploys and "shut down" when the web app undeploys.
Please read this official doc - Using Log4j 2 in Web Containers, its helpful.

This is a known issue. Log4j will now use the SimpleLogger during shutdown. Please see https://issues.apache.org/jira/browse/LOG4J2-1222.

Related

NoClassDefFoundError during Tomcat graceful shutdown

We are getting NoClassDefFoundErrors during Tomcat graceful shutdown process (i.e. using catalina.sh stop command). We are using Tomcat version 6.0.44.
Note that this error can not be reproduced consistently. Another question suggests class loader related problems during execution of JVM shutdown hook. But in this case, the error is raised during the execution of the ServletFilter::destroy(...) method.
Could you please provide any insight?
java.lang.NoClassDefFoundError: com/google/common/cache/RemovalCause
at com.google.common.cache.LocalCache$Segment.remove(LocalCache.java:3147)
at com.google.common.cache.LocalCache.remove(LocalCache.java:4236)
at org.apache.curator.framework.imps.NamespaceWatcherMap.remove(NamespaceWatcherMap.java:71)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.clearWatcherReferences(CuratorFrameworkImpl.java:189)
at org.apache.curator.framework.recipes.cache.PathChildrenCache.close(PathChildrenCache.java:376)
at org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.stopThreads(ZKDelegationTokenSecretManager.java:456)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.destroy(DelegationTokenManager.java:152)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.destroy(DelegationTokenAuthenticationHandler.java:153)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.destroy(AuthenticationFilter.java:401)
at org.apache.solr.servlet.SolrHadoopAuthenticationFilter.destroy(SolrHadoopAuthenticationFilter.java:177)
at org.apache.catalina.core.ApplicationFilterConfig.release(ApplicationFilterConfig.java:360)
at org.apache.catalina.core.StandardContext.filterStop(StandardContext.java:4111)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4880)
at org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java:1279)
at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1385)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:307)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1392)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1656)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1665)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1645)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.google.common.cache.RemovalCause
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)

java.lang.ClassNotFoundException: ch.qos.logback.classic.spi.ThrowableProxy?

while starting my weblogic(having my app war file containing logback-classic-1.0.1.jar),i am getting below exception.Any suggestion ?
<Sep 9, 2015 9:27:13 AM UTC> <Warning> <Common> <BEA-000632> <Resource Pool "JDBC Data Source-0" shutting down, ignoring 3 resources still in use by applications..>
Exception in thread "Thread-12" java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy
at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:125)
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:468)
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:424)
at ch.qos.logback.classic.Logger.log(Logger.java:824)
at org.apache.commons.logging.impl.SLF4JLocationAwareLog.error(SLF4JLocationAwareLog.java:225)
at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:415)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:114)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.cache.interceptor.CacheInterceptor$1.invoke(CacheInterceptor.java:58)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:213)
at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:66)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622)
................
at java.lang.Thread.run(Thread.java:701)
Caused by: java.lang.ClassNotFoundException: ch.qos.logback.classic.spi.ThrowableProxy
at weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:297)
at weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java:270)
at weblogic.utils.classloaders.ChangeAwareClassLoader.findClass(ChangeAwareClassLoader.java:64)
at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
at weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:179)
at weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:52)
... 18 more
Exception in thread "Thread-18" java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy
at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:125)
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:468)
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:424)
at ch.qos.logback.classic.Logger.log(Logger.java:824)
at org.apache.commons.logging.impl.SLF4JLocationAwareLog.error(SLF4JLocationAwareLog.java:225)
at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:415)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:114)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.cache.interceptor.CacheInterceptor$1.invoke(CacheInterceptor.java:58)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:213)
at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:66)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622)
.......................
at java.lang.Thread.run(Thread.java:701)
Caused by: java.lang.ClassNotFoundException: ch.qos.logback.classic.spi.ThrowableProxy
at weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:297)
at weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java:270)
at weblogic.utils.classloaders.ChangeAwareClassLoader.findClass(ChangeAwareClassLoader.java:64)
at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
at weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:179)
at weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:52)
... 18 more
This error happens when spring applications jar are rebuilt, removed, or updated during execution.
Ref: https://github.com/spring-projects/spring-boot/issues/4968
I have been getting this same error when breaking out (ctrl-c) from Dropwizard server, I finally learned why I get it.
My Dropwizard is build into one fat jar, that I build with Gradle, and when running server on my local computer I run it directly from build/libs/...fat.jar
So this exception occurs when I have changed source code, built app-server again, and Gradle has overwritten the same jar I am currently running with "java -jar ...", so it's not that weird that classes that haven't been used before that session will not load ok :)
This type of exception occurs when your classpath does not contain this class. If the jar's class is in your classpath, you pay attention to what classpath application you using at runtime. Indeed, often, classpath can be overwritten during runtime, or simply you can use a different one (for example, in a startup script).
For us, appears this meant "you are running out of file handles, increase your open file handle limit." (too many sockets were being created, using them all up). Credit to the original answer here.
Other things I've seen: make sure your spring boot is at least 1.3.8
Also make sure the jars underneath aren't being overwritten/changed/unstable file system. FWIW.
I got this runtime error in my spring boot app using Spring Boot version 2.3.5.RELEASE. Resolved by adding following dependencies in my build.gradle file.
compile group: 'ch.qos.logback', name: 'logback-classic', version: '1.2.3'
compile group: 'ch.qos.logback', name: 'logback-core', version: '1.2.3'
Two options I see(not enough information yet):
1. It happens right upon loading, I bet for classpath, see answers above.
2. It happens after some time, when app is runnig(my case), read next paragraphs.
When I got class not found errors and also have the same error like you right now, quite offten is issued with no space left on device aka full disk, full memory.
My environment, is spring boot, logback is included automagicaly, in maven have repackage, so wverithing I need is included.
This error occures after my app is runnig for 20 minutes or so, is has thousands of threads. If you do not limit memory for your virtual maschine, it can get very high even if it is not necessary.
Once(2 days ago) I spend 3 hours looking around claspath stuf, ... issue was full disk
If you exactly ensure that logback-class and logback-core already are in your classpath. See below.
In your Logback.xml, you need add debug="true"
<configuration debug="true">
I encountered the problem 'java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy' in my tomcat.
However, after adding debug = 'true', anything is ok.
The class that is missing definitely exists in the JAR you have, I checked logback-classic-1.0.1.jar:
2012-03-07 07:34:18 ..... 4729 2018 ch\qos\logback\classic\spi\ThrowableProxy.class
Make sure your classpath is set up correctly. By the way: if possible update logback because this version is 3 years old.
I solved this problem by doing "mvn install" instead of "mvn clean install".

Java Util Logging redirection to Log4j2 not working for Spring Boot applications

I'm trying to redirect all logging frameworks to Log4j2 in my Spring Boot application. It works fine for Java Commons Logging, SLF4J, and Log4j 1.x.
Unfortunately the Java Util Logging (JUL) redirection does not work because java.util.logging.Logger is used by the Plexus Launcher at a time when the application classpath has not been fully assembled.
The Log4j2 documentation demands to set the VM parameter -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager or to call the System.setProperty("java.util.logging.manager", "org.apache.logging.log4j.jul.LogManager") method to make the redirection work. Neither of them work:
System.setProperty() in my application code comes far too late. Some startup code already did some calls to LogManager or Logger so that the Redirection-LogManager cann be installed anymore.
The VM parameter also does not work as the application classpath is not available at the point in time when the
The call to System.setProperty() gets ignored because the Java Util Logging framework is already initialized. With the VM parameter the following exception occurs:
$ mvn spring-boot:run -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
Could not load Logmanager "org.apache.logging.log4j.jul.LogManager"
java.lang.ClassNotFoundException: org.apache.logging.log4j.jul.LogManager
at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50)
at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass(ClassRealm.java:271)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:247)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:239)
at java.util.logging.LogManager$1.run(LogManager.java:195)
at java.util.logging.LogManager$1.run(LogManager.java:181)
at java.security.AccessController.doPrivileged(Native Method)
at java.util.logging.LogManager.<clinit>(LogManager.java:181)
at java.util.logging.Logger.demandLogger(Logger.java:448)
at java.util.logging.Logger.getLogger(Logger.java:502)
at com.google.inject.internal.util.Stopwatch.<clinit>(Stopwatch.java:27)
at com.google.inject.internal.InternalInjectorCreator.<init>(InternalInjectorCreator.java:61)
at com.google.inject.Guice.createInjector(Guice.java:96)
at com.google.inject.Guice.createInjector(Guice.java:73)
at com.google.inject.Guice.createInjector(Guice.java:62)
at org.codehaus.plexus.DefaultPlexusContainer.addPlexusInjector(DefaultPlexusContainer.java:481)
at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:206)
at org.apache.maven.cli.MavenCli.container(MavenCli.java:542)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:279)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:197)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Digging into the code shows that the initialization is caused by the Guice library, which uses JUL internally. In this case it is just a call to com.google.inject.internal.util.Stopwatch.<clinit>(Stopwatch.java:27).
So the question is, does somone have an idea on how to resolve this?
Update 2015-08-23:
The suggestion from below to add the configuration property <fork>true</fork> works for running the application via the Spring boot Maven plugin mvn spring-boot:run.
Unfortunately it does not work when running the application via the command line java -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -jar myapp.jar. In this case the following exception occurs:
Could not load Logmanager "org.apache.logging.log4j.jul.LogManager"
java.lang.ClassNotFoundException: org.apache.logging.log4j.jul.LogManager
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.util.logging.LogManager$1.run(LogManager.java:195)
at java.util.logging.LogManager$1.run(LogManager.java:181)
at java.security.AccessController.doPrivileged(Native Method)
at java.util.logging.LogManager.<clinit>(LogManager.java:181)
at java.util.logging.Logger.demandLogger(Logger.java:448)
at java.util.logging.Logger.getLogger(Logger.java:502)
at org.springframework.boot.loader.Launcher.<init>(Launcher.java:43)
at org.springframework.boot.loader.ExecutableArchiveLauncher.<init>(ExecutableArchiveLauncher.java:48)
at org.springframework.boot.loader.ExecutableArchiveLauncher.<init>(ExecutableArchiveLauncher.java:45)
at org.springframework.boot.loader.JarLauncher.<init>(JarLauncher.java:30)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:45)
So, unfortunately the Spring Boot Launcher is also using Java Util Logging and by that preventing JUL redirection with Log4j2.
I looked at the source-code of the class org.springframework.boot.loader.Launcher and its child classes. The issue is caused by the declaration of the static logger variable in the Launcher class. This logger seems not to be used by any of the Launcher classes so it could easily be removed.
The question is whether the dependency to JUL could be removed completely to fix the JUL redirection issue? Maybe by using stderr for Launcher diagnostics, instead of JUL?
Your current approach means that both Maven and your application will try to use the custom log manager. Maven attempting to use it is causing the exception that you are seeing.
You could configure Spring Boot's Maven plugin in your pom.xml to fork a separate JVM for your application:
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<fork>true</fork>
</configuration>
</plugin>
And then use System.setProperty in your application's main method. Forking a separate JVM will prevent Maven's own logging from messing things up before your configuration takes effect.

UnsatisfiedLinkError when running Jetty9's setUID feature

I just tried to run Jetty 9 as non root users, using setuid feature without success for binding low port numbers.
I enabled the module setuid in the start.ini and added -Djava.library.path=/opt/jetty/lib/setuid
But I have the following stack trace when starting Jetty:
2015-06-09 16:27:27.211:WARN:oejx.XmlConfiguration:main: Config error
at | |
java.lang.reflect.InvocationTargetException in
file:/opt/jetty/etc/jetty-setuid.xml
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:321)
at org.eclipse.jetty.start.Main.start(Main.java:817)
at org.eclipse.jetty.start.Main.main(Main.java:112) Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.set(XmlConfiguration.java:479)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:411)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.newObj(XmlConfiguration.java:815)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.itemValue(XmlConfiguration.java:1125)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.value(XmlConfiguration.java:1030)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.call(XmlConfiguration.java:721)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:417)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:354)
at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:262)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1243)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1174)
... 7 more Caused by: java.lang.UnsatisfiedLinkError: org.eclipse.jetty.setuid.SetUID.getpwnam(Ljava/lang/String;)Lorg/eclipse/jetty/setuid/Passwd;
at org.eclipse.jetty.setuid.SetUID.getpwnam(Native Method)
at org.eclipse.jetty.setuid.SetUIDListener.setUsername(SetUIDListener.java:53)
... 23 more
Those are the only references to this error:
dev.eclipse.org/mhonarc/lists/jetty-users/msg01657.html
groups.google.com/forum/#!topic/dropwizard-user/aap2B_U_QPo
But they are either stopped or do not show the solution.
The source code I found for the setuid package is:
https://github.com/jetty-project/codehaus-jetty-project/blob/master/jetty-setuid/modules/java/src/main/java/org/mortbay/setuid/SetUID.java
Even though I cannot be sure this is the version I am using, given this one is from a org.mortbay package, while the one Jetty 9 uses is an Eclipse one.
I tried setting -Djetty.libsetuid.path in the java args (first try-catch block), either adding the path to the $PATH variable or setting -Djava.library.path (second try-catch block) or copying the so to /lib and, finally, leaving it as it is (third try-catch block). I got the same exception stack in all the cases.
I cannot be sure if Jetty is either not finding the so file or not being able to load it, given that, if I remove all the references to the path (the cases I described latter), I still get the same error message.
I use Java7 to run Jetty.
Edit
I added the following snippet to one of my webapps as as matter of test, given that if the webapp succeed, I would know the problem is not in finding the shared object:
SetUID.setgid(1002);
SetUID.setuid(1002);
Passwd pw = SetUID.getpwuid(1002);
System.setProperty("user.name", pw.getPwName());
System.setProperty("user.home", pw.getPwDir());
I have the same UnsatisfiedLinkError as a result.
Edit2
I tried, instead, the following:
System.load(SetUID.__FILENAME);
SetUID.setgid(1002);
Which got me the following error message:
javax.servlet.ServletException: java.lang.UnsatisfiedLinkError: Native
Library /lib/libsetuid.so already loaded in another classloader
I may conclude, therefore, that the library has been loaded, at least.
Exploring the libsetuid file (nm -D /path/to/the/so), I realized that the so I was using was outdated and was not compatible with Jetty 9, the functions were named as mortbay:
root#root:~/# nm -D libsetuid.so --size-sort | less
0000000000000010 T Java_org_mortbay_setuid_SetUID_setgid
0000000000000010 T Java_org_mortbay_setuid_SetUID_setuid
0000000000000015 T Java_org_mortbay_setuid_SetUID_setumask
0000000000000018 T throwNewJavaSecurityException 0000000000000058 T
throwNewJavaException 000000000000008e T
Java_org_mortbay_setuid_SetUID_setrlimitnofiles 00000000000000a7 T
getJavaMethodId 00000000000000f5 T
Java_org_mortbay_setuid_SetUID_getrlimitnofiles 0000000000000102 T
getJavaFieldInt 0000000000000111 T setJavaFieldInt 0000000000000111 T
setJavaFieldLong 0000000000000121 T setJavaFieldString
00000000000001ba T Java_org_mortbay_setuid_SetUID_getpwuid
00000000000001e5 T Java_org_mortbay_setuid_SetUID_getpwnam
000000000000027e T Java_org_mortbay_setuid_SetUID_getgrgid
000000000000029e T Java_org_mortbay_setuid_SetUID_getgrnam
Now, I am using the shared object that already comes in the jetty download, not the one linked in the wiki.
In addition to that, looking through the code of the setuid module, I decided using the property -Djetty.libsetuid.path in my java args for indicating the absolute path of the lib, e.g., -Djetty.libsetuid.path=/opt/jetty/lib/setuid.so

Why would Java classloading fail on Linux, but succeed on Windows?

I've got a Java web application (using Spring), deployed with Jetty. If I try to run it on a Windows machine everything works as expected, but if I try to run the same code on my Linux machine, it fails like this:
[normal startup output]
11:16:39.657 INFO [main] org.mortbay.jetty.servlet.ServletHandler$Context.log>(ServletHandler.java:1145) >16> Set web app root system property: 'webapp.root' = [/path/to/working/dir]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mortbay.start.Main.invokeMain(Main.java:151)
at org.mortbay.start.Main.start(Main.java:476)
at org.mortbay.start.Main.main(Main.java:94)
Caused by: java.lang.ExceptionInInitializerError
at org.springframework.web.util.Log4jWebConfigurer.initLogging(Log4jWebConfigurer.java:129)
at org.springframework.web.util.Log4jConfigListener.contextInitialized(Log4jConfigListener.java:51)
at org.mortbay.jetty.servlet.WebApplicationContext.doStart(WebApplicationContext.java:495)
at org.mortbay.util.Container.start(Container.java:72)
at org.mortbay.http.HttpServer.doStart(HttpServer.java:708)
at org.mortbay.util.Container.start(Container.java:72)
at org.mortbay.jetty.Server.main(Server.java:460)
... 7 more
Caused by: org.apache.commons.logging.LogConfigurationException: org.apache.commons.logging.LogConfigurationException: No suitable Log constructor [Ljava.lang.Class;#15311bd for org.apache.commons.logging.impl.Log4JLogger (Caused by java.lang.NoClassDefFoundError: org/apache/log4j/Category) (Caused by org.apache.commons.logging.LogConfigurationException: No suitable Log constructor [Ljava.lang.Class;#15311bd for org.apache.commons.logging.impl.Log4JLogger (Caused by java.lang.NoClassDefFoundError: org/apache/log4j/Category))
at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:543)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
at org.springframework.util.SystemPropertyUtils.(SystemPropertyUtils.java:42)
... 14 more
Caused by: org.apache.commons.logging.LogConfigurationException: No suitable Log constructor [Ljava.lang.Class;#15311bd for org.apache.commons.logging.impl.Log4JLogger (Caused by java.lang.NoClassDefFoundError: org/apache/log4j/Category)
at org.apache.commons.logging.impl.LogFactoryImpl.getLogConstructor(LogFactoryImpl.java:413)
at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
... 18 more
Caused by: java.lang.NoClassDefFoundError: org/apache/log4j/Category
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
at java.lang.Class.getConstructor0(Class.java:2699)
at java.lang.Class.getConstructor(Class.java:1657)
at org.apache.commons.logging.impl.LogFactoryImpl.getLogConstructor(LogFactoryImpl.java:410)
... 19 more
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Category
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
... 24 more
[shutdown output]
I've run the app with java -verbose:class, and according to that output, org.apache.log4j.Category is loaded from the log4j JAR in my /WEB-INF/lib, just before the first exception is thrown.
Now, the Java versions on the two machines are slightly different. Both the machines have Sun's java, the Linux machine has 1.6.0_10, while the Windows machine has 1.6.0_08, or maybe 07 or 06, I can't remember the exact number right now, and don't have the machine at hand. But even though the minor versions of the Javas are slightly different, the code shouldn't break like this. Does anyone understand what's wrong here?
You must understand that a classloader can't see everything; they can only see what a parent classloader has loaded or what they have loaded themselves. So if you have two classloaders, say one for Jetty and another for your webapp, your webapp can see log4j (since the JAR is the WEB-INF/lib) but Jetty's classloader can't.
If you manage to make a class available to Jetty (for example something in the DB layer) which uses log4j but which ends up running in the context (and classloader) of Jetty, you will get an error.
To debug this, set a breakpoint in org.springframework.web.util.Log4jWebConfigurer.initLogging(). If you can, copy the source of this class into your project (don't forget to delete it afterwards) and add this line:
ClassLoader cl = Thread.currentThread().getContextClassLoader();
Have a look at the cl object in your debugger. That should give you some information who created it. My guess is that this is the classloader from Jetty.
[EDIT] Note that you get in a different mess if you have log4j in both classloaders: In this case, you will have two classes with the same name which create objects which are not assignment compatible! So make sure there is only a single instance of this jar or that instances of log4j will never be passed between the two contexts (which is usually not possible).
This seems like a classic classloader problem. It could be due to another web app being loaded first which also uses log4j but a version that is different to the one used by the app you are testing. The classloader uses the first version of the class it finds. The server class loading policy can usually be changed in the config files. Sorry I am a bit rusty on this but maybe it can point you in the correct direction.
Make sure there are no other installed apps on the web server,
Make sure the log4j being loaded is the correct version,
Make sure you don't have a log4j lurking somewhere in the classpath of the server.
HTH
You're using the same WAR on both machines? Have you checked if the WAR files are identical (no transfer errors occured)?
Some random things to consider:
(1) Check if there are any other versions of log4j floating around on the linux instance, outside of the web-app directories?
(2) Is apache commons logging being used at all? You might want to consider SLF4J instead?
(3) Did the JAR/WAR become corrupt in some way - was it FTP'ed in ASCII or Binary?
(4) Print out the classloader hierarchy in each case, just to see if there are any discrepancies?
Even though the original problem was solved for the asker, I'll point out that a common source of problems when running the same code on Windows vs Linux (or Unix) is case-sensitivity issues. Windows ignores case while Linux or Unix is case-sensitive. This has bitten me more than once.
So if you specify a jar or directory on the classpath, but it isn't the right case then it will fail on Linux but succeed on Windows. This can also be the source of FileNotFoundExceptions.
Had the same problem and found an easy solution/workaround:
In Eclipse in Preferences > Java > Installed JREs, select the JRE > Edit and Add External JARs... and browse to your log4j.jar.
The other workaround is to add log4j.jar to every launch definition in Classpath tab.

Categories