I'm running a new Jeninks Master instance on Windows (windows server 2012).
The Jenkins master has failed twice in 5 days. the Jenkins instance has about 30 nodes and is not too busy.
I noticed the memory consumption of Jenkins was quite low (400mb) with the default java memory settings in jenkins.xml
I modified the jenkins.xml to -Xrs -Xmx1536m and the Java process is not using 515mb of memory.
I tried to set Xmx to a higher value but the Jenkins service would not start.
After the 2nd failure today, I got the same error message below repeated.
The windows server has 8 cores and 16GB memory. Average CPU is <5% and average memory usage is 2GB. Jenkins version is 1.609.3
Any help pointers appreciated....
Sep 21, 2015 12:00:48 PM org.eclipse.jetty.util.log.JavaUtilLog warn
WARNING:
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
at winstone.BoundedExecutorService.scheduleNext(BoundedExecutorService.java:74)
at winstone.BoundedExecutorService.execute(BoundedExecutorService.java:64)
at org.eclipse.jetty.util.thread.ExecutorThreadPool.dispatch(ExecutorThreadPool.java:126)
at org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.dispatch(SelectChannelConnector.java:300)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.dispatch(SelectChannelEndPoint.java:236)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.schedule(SelectChannelEndPoint.java:195)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Found a solution to this. I installed the JDK and used VisualVM to monitor the memory usage for JVM. This showed 32 bit memory boundaries. After a little digging the java.exe that Jenins points to by default in Jenkins_Home/jre/bin is a 32bit version of java. I modified the Jenkins.xml to point to the already installed 64 Jre. This worked fine and was able to modify the Xmx valve to 4G. Jenkins is now running happly using <2gb of memory for the JVM process.
Related
I am getting this java.lang.OutOfMemoryError: PermGen space Exception. I am using Eclipse Juno and Tomcat 7.0. This Exception occurs at least once in every 10-15 Minutes in a console. How to resolve it?
Feb 25, 2016 10:23:23 AM org.springframework.beans.factory.support.DefaultListableBeanFactory destroyBean
SEVERE: Destroy method on bean with name 'mvcUrlPathHelper' threw an exception
java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(Unknown Source)
at java.security.SecureClassLoader.defineClass(Unknown Source)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2895)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1173)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1681)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(Unknown Source)
at java.security.SecureClassLoader.defineClass(Unknown Source)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2895)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1173)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1681)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
at org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor.postProcessBeforeDestruction(PersistenceAnnotationBeanPostProcessor.java:375)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:242)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:925)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:932)
at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:997)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:547)
Feb 25, 2016 10:23:24 AM org.springframework.beans.factory.support.DefaultListableBeanFactory destroyBean
SEVERE: Destroy method on bean with name 'mvcPathMatcher' threw an exception
java.lang.OutOfMemoryError: PermGen space
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: PermGen space
What is java.lang.OutOfMemoryError: PermGen space?
Ans:
The java.lang.OutOfMemoryError: PermGen space message indicates that the Permanent Generation’s area in memory is exhausted.
Any Java applications is allowed to use a limited amount of memory. The exact amount of memory your particular application can use is specified during application startup.
Java memory is separated into different regions which can be seen in the following image:
What will be the solution for this error PermGen space?
Ans:
For Heap,
export JVM_ARGS="-Xms1024m -Xmx1024m"
For Permgen,
JVM_ARGS="-XX:PermSize=512M -XX:MaxPermSize=512m"
You can give also additional section
-XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC XX:+CMSClassUnloadingEnabled
You can also give more additional section :)
-XX:PermSize=512m -XX:MaxPermSize=512m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:SurvivorRatio=2 -XX:MaxTenuringThreshold=128 -XX:TargetSurvivorRatio=90 -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycleMin=0 -XX:CMSIncrementalDutyCycle=10 -XX:+UseTLAB -XX:+CMSParallelRemarkEnabled -XX:+CMSClassUnloadingEnabled
Metaspace: A new memory space is born
The JDK 8 HotSpot JVM is now using native memory for the representation of class metadata and is called Metaspace; similar to the Oracle JRockit and IBM JVM's.
The good news is that it means no more java.lang.OutOfMemoryError: PermGen space problems and no need for you to tune and monitor this memory space anymore.
Related Link:
https://plumbr.eu/outofmemoryerror/java-heap-space
https://plumbr.eu/outofmemoryerror/permgen-space
http://stas-blogspot.blogspot.in/2011/07/most-complete-list-of-xx-options-for.html#UseConcMarkSweepGC
https://www.javacodegeeks.com/2013/12/decoding-java-lang-outofmemoryerror-permgen-space.html
http://www.javavillage.in/reasons-outofmemory-permespace.php
Dealing with "java.lang.OutOfMemoryError: PermGen space" error
Java 8 Links
What is the use of MetaSpace in Java 8?
https://blogs.oracle.com/poonam/entry/about_g1_garbage_collector_permanent
https://plumbr.eu/outofmemoryerror/metaspace
https://dzone.com/articles/java-8-permgen-metaspace
It is easy to configure Java for more PermGenSpace but I would recommend to migrate to Java 1.8 runtime instead. Install Java 8 SDK and add two lines like
-vm
D:/jdk8/jre/bin/javaw.exe
to eclipse.ini. This problem is fixed starting from Java 1.8 and does not longer require any hacks.
Instead making changes in eclipse.ini, its better to install Java 8.
I Installed Java 8 and now I am not getting java.lang.OutOfMemoryError: PermGen space Exception.
I am running an APP using Jruby and Puma. We have an issue with a potential memory leak and I would like to generate a heap dump so that I can debug. My local environment is running Linux Mint 17.
I have run the following:
jmap -histo:live 6450
and this resulted in :
6450: Unable to open socket file: target process not responding or HotSpot VM not loaded The -F option can be used when the target process is not responding
I've also tried this:
jmap -dump:live,format=b,file=heap.bin 6450
result
6450: Unable to open socket file: target process not responding or HotSpot VM not loaded
As the propmt suggested I tried running with the -F flag:
jmap -F -dump:live,format=b,file=heap.bin 6450
Attaching to process ID 6450, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.45-b02
Dumping heap to heap.bin ...
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at sun.tools.jmap.JMap.runTool(JMap.java:201)
at sun.tools.jmap.JMap.main(JMap.java:130)
Caused by: sun.jvm.hotspot.utilities.AssertionFailure: can not get class data for java/lang/UNIXProcess$Platform$$Lambda$10x0000000100393428
at sun.jvm.hotspot.utilities.Assert.that(Assert.java:32)
at sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeInstance(HeapHprofBinWriter.java:803)
at sun.jvm.hotspot.utilities.AbstractHeapGraphWriter$1.doObj(AbstractHeapGraphWriter.java:95)
at sun.jvm.hotspot.oops.ObjectHeap.iterateLiveRegions(ObjectHeap.java:353)
at sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:171)
at sun.jvm.hotspot.utilities.AbstractHeapGraphWriter.write(AbstractHeapGraphWriter.java:51)
at sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:433)
at sun.jvm.hotspot.tools.HeapDumper.run(HeapDumper.java:62)
at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:83)
... 6 more`
Does anyone have any suggestions as to how to get around these errors and generate a heap dump ? I have tried using jvisualvm and Eclipse Memory Analyzer to generate a heap dump as well, and I run into the same issue.
This looks like a JVM bug in jdk 1.8_45 see this link for more info
https://bugs.openjdk.java.net/browse/JDK-8065318
Having a problem running AMTU on a windows 7 Pro Machine.
It runs fine when it opens but it crashes after running a few days.
Here is the Crash Report.
2013-09-14 09:48:19,482 [pool-3-thread-3] com.amazon.merchants.services.RetrieverService - Unable to retrieve unacknowledged report information - MWS Request ID unknown
2013-09-14 09:48:19,482 [pool-3-thread-3] com.amazon.merchants.services.RetrieverService - MWS responded with an error: Internal Error
com.amazonaws.mws.MarketplaceWebServiceException: Internal Error
at com.amazonaws.mws.MarketplaceWebServiceClient.processErrors(MarketplaceWebServiceClient.java:2342)
at com.amazonaws.mws.MarketplaceWebServiceClient.invoke(MarketplaceWebServiceClient.java:2231)
at com.amazonaws.mws.MarketplaceWebServiceClient.invoke(MarketplaceWebServiceClient.java:2011)
at com.amazonaws.mws.MarketplaceWebServiceClient.getReportList(MarketplaceWebServiceClient.java:1206)
at com.amazon.merchants.services.RetrieverService.processReportCheck(RetrieverService.java:94)
at com.amazon.merchants.services.RetrieverService.run(RetrieverService.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
I also cannot start the Windows Service called AMTU. Get the following Error:
The AMTU service on local computer has started and then stopped
I am suspecting its a Java version issue.
In the AMTU documentation it says "requires Java version 1.6.0 or later (listed as JDK/JRE - 6)" and links to http://www.oracle.com/technetwork/java/archive-139210.html. The link has many download links, but nothing listed "JDK/JRE - 6".
AMTU documentation: https://d28hcfptedr5ia.cloudfront.net/ug/AMTU_2.2_UserGuide.pdf
Any Ideas?
Thanks in advance.
The way I read this crash report, AMTU got an unexpected result from the MWS servers:
MWS responded with an error: Internal Error
It seems the error handling in AMTU is not good enough to gracefully handle a HTTP status 500, and there is not a lot you can do about this. You could put a HTTP proxy in between and catch a HTTP 500 on the line before it gets to the AMTU, replacing it with something that AMTU handles more gracefully (e.g. an empty XML result or a time out). It would be preferrable though that Amazon doesn't send those errors back, after all, a 500 is an error on their side of things.
I seriously doubt the Java version has anything to do with this, but just in case: You can still download JRE6 even though it is end-of-life:
Java 6 downloads
Please note that the true name of JRE6 is "Java SE runtime environment" and JDK6 is called "Java SE development kit". At the time of writing, the most current version is "update 45". You may need to uninstall Java 7 to make sure the AMTU actually uses the version you want it to. Please also note that since AMTU is a 32bit process, you will need the 32bit JRE even if it is run on a 64bit windows.
This question already has answers here:
How to clear PermGen space Error in tomcat
(11 answers)
Closed 9 years ago.
I installed tomcat 7 to upgraded my JIRA projeect version from 5.0 to 6. After I place the project folder in tomcat's webapps. I run this localhost:8080/jira
after long time running, it throws some error message. Please help us to fix this problem thanks in advance
java.lang.RuntimeException: PermGen space
at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:54)
at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$2.run(AsynchronousAbleEventDispatcher.java:66)
at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1.execute(AsynchronousAbleEventDispatcher.java:32)
at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:60)
at com.atlassian.event.internal.EventPublisherImpl.invokeListeners(EventPublisherImpl.java:160)
at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:79)
at com.atlassian.plugin.event.impl.DefaultPluginEventManager.broadcast(DefaultPluginEventManager.java:84)
at com.atlassian.plugin.manager.DefaultPluginManager.addPlugins(DefaultPluginManager.java:768)
at com.atlassian.plugin.manager.DefaultPluginManager.init(DefaultPluginManager.java:200)
at com.atlassian.jira.plugin.JiraPluginManager.start(JiraPluginManager.java:63)
at com.atlassian.jira.ComponentManager$PluginSystem.start(ComponentManager.java:635)
at com.atlassian.jira.ComponentManager.startJIRA(ComponentManager.java:214)
at com.atlassian.jira.ComponentManager.quickStart(ComponentManager.java:208)
at com.atlassian.jira.ComponentManager.start(ComponentManager.java:193)
at com.atlassian.jira.upgrade.PluginSystemLauncher.start(PluginSystemLauncher.java:23)
at com.atlassian.jira.startup.DefaultJiraLauncher$3.run(DefaultJiraLauncher.java:107)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:323)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:211)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:100)
at com.atlassian.jira.startup.DefaultJiraLauncher.access$100(DefaultJiraLauncher.java:27)
at com.atlassian.jira.startup.DefaultJiraLauncher$1.run(DefaultJiraLauncher.java:66)
at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:33)
at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:61)
at com.atlassian.jira.startup.LauncherContextListener.contextInitialized(LauncherContextListener.java:54)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4797)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5291)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:657)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1637)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: PermGen space
More helpful solution is to increase values in JAVA_OPTS variable.
Add next line to your Catalina.bat/Catalina.sh file in bin directory
For Windows (Catalina.bat)
set JAVA_OPTS="-Xms1024m -Xmx10246m -XX:NewSize=256m -XX:MaxNewSize=356m -XX:PermSize=256m -XX:MaxPermSize=356m"
For Unix (Catalina.sh)
export JAVA_OPTS="-Xms1024m -Xmx1024m -XX:NewSize=256m -XX:MaxNewSize=356m -XX:PermSize=256m -XX:MaxPermSize=356m"
More solutions you can find in this article
Sometimes after re-deployments the Garbage Collector can't destroy some objects on Tomcat and run out of space.
There are many reasons for these case:
If you use some CRUD-Objects - check if there are all closed after using.
If your app used other libs - sometimes the Objects from these libs cant also be destroyed, like MysqlConnector, Hibernate/C3p0 or other - try to put these libs on the tomcat/lib folder
Check the Tomcat with the "Java VisualVM" and check the permGen after re-deployment.
The default value for the permGen Space are often also very low you can increase them with the following javaVM parameter
-XX:PermSize=64M -XX:MaxPermSize=256m
You can follow these tutorial
http://www.mkyong.com/tomcat/tomcat-javalangoutofmemoryerror-permgen-space/
UPDATE
other solution maybe found here : what to do with tomcat PermGen space
add -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC to the javaVM options
Tweak the value of the permgen using -XX:PermSize={value} and -XX:MaxPermSize={value} via the JAVA_OPTS variable where 'value' is a string like 256m.
I need to run a Java application, which we are trying to port to Java 6, on an NT box.
I manage to run java 5 on it (although not officially supported), but when I try to run java 6 I get the following error:
Exception in thread "main" java.lang.UnsatisfiedLinkError: C:\Program Files\Java\jre1.6.0_05\bin\awt.dll: The specified procedure could not be found
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(Unknown Source)
at java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.lang.Runtime.loadLibrary0(Unknown Source)
at java.lang.System.loadLibrary(Unknown Source)
at sun.security.action.LoadLibraryAction.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.awt.NativeLibLoader.loadLibraries(Unknown Source)
at sun.awt.DebugHelper.<clinit>(Unknown Source)
at java.awt.EventQueue.<clinit>(Unknown Source)
at javax.swing.SwingUtilities.invokeLater(Unknown Source)
at ui.sequencer.test.WindowTest.main(WindowTest.java:136)
Anybody has any idea how to solve this?
This persists even when I move the java executables to another directory with no spaces in its name.
p.s.
I know, I should upgrade, but it's not up to me or my company - this is a very very bug huge gigantic company we work with, and they intend to keep NT for another 5 years.
OK, Thanks for all the viewers and to #Roel Spiker and #Partyzant for their answers.
It can't be done. Not unless you install windows2000 on the NT box. This is because awt.dll fr J6SE uses new methods in User32.dll, which is part of the windows OS (linked to kernel.dll et al). Use the dll dependency walker and see for yourself.
Another possible solution is to alter OpenJDK slightly to use other methods available in windows NT.
Java SE 6 requires at least Windows 2000.
I you are not using a GUI, for instance AWT, Swing or SWT, you could try starting you application in headless mode. See http://java.sun.com/developer/technicalArticles/J2SE/Desktop/headless/ for more information. To start java in headless mode, use java -Djava.awt.headless=true
It would take care of the UnsatisfiedLinkError. I don't know if that's the only obstacle though.