Our web application keep on crashing with error message as "GC overhead limit exceeded" after 30 minutes from start time.
I generated heap dump at that moment and loaded into eclipse MAT. It mentioned that TDSPacket as the problem suspect. As this is the library we are using to connect to SQL server, I am unable to trouble shoot from here.
Any suggestion please. Here are the JAVA_OPTIONS used.
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\Apps\Logs\tomcat\memory -Xms2g -Xmx2g -XX:+UseG1GC -XX:+DisableExplicitGC -Xloggc:gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=50 -XX:GCLogFileSize=2M exit -Xms2048m -Xmx2048m
In my java program I use java nio to communicate with another process, and my code doesn't allocate any direct ByteBuffer.
My code use an open source lib to communicate with another process with unix domain socket: https://github.com/jnr/jnr-unixsocket
and I use some jvm paramter to limit the memory size:
-Djdk.nio.maxCachedBufferSize=262144 -Xms80m -Xmx80m -Xmn40m -Xss256k -XX:NativeMemoryTracking=detail -XX:+UnlockDiagnosticVMOptions -XX:+PrintNMTStatistics -XX:MetaspaceSize=40m -XX:MaxMetaspaceSize=40m -XX:MaxDirectMemorySize=10m -XX:SurvivorRatio=2 -XX:TargetSurvivorRatio=90 -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=8 -XX:MaxTenuringThreshold=15 -XX:+PrintTenuringDistribution -XX:+UseParNewGC -XX:CMSFullGCsBeforeCompaction=5 -XX:+UseCMSCompactAtFullCollection -XX:+CMSPermGenSweepingEnabled -XX:CMSInitiatingOccupancyFraction=50 -XX:+CMSClassUnloadingEnabled -XX:+DisableExplicitGC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps
how after several hours, the size of java process grows to more than 400MB
and I use jcmd to check the memory of the process
the total is about 200MB, so why the physical memory consumed by the java prcocess grow to a size much bigger than jcmd output?
I use java 8.
here is the jvm settings for Jboss AS 7 / EAP 6
java version "1.6.0_35"
Java(TM) SE Runtime Environment (build 1.6.0_35-b10)
Java HotSpot(TM) 64-Bit Server VM (build 20.10-b01, mixed mode)
VM Arguments: -XX:+UseCompressedOops -Dprogram.name=standalone.bat
-XX:-TieredCompilation -XX:+PrintGCDetails -Xloggc:E:\serverLog\jvm.log
-Xms1303M -Xmx1303M -XX:MaxPermSize=256M
-Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000
-Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true
-Djboss.modules.system.pkgs=org.jboss.byteman
-Djboss.server.default.config=standalone.xml -Dorg.jboss.boot.log.file=E:\JAVA
\JBOSS\EAP-6.0.0.GA\jboss-eap-6.0\standalone\log\boot.log
-Dlogging.configuration=file:E:\JAVA\JBOSS\EAP-6.0.0.GA\jboss-eap-6.0
\standalone/configuration/logging.properties
I made several heavy-loading pages refresh every 30s, then I found in gc log, there are gradually frequent full garbage collection occurs, each full GC release part of the old generation but it getting smaller and smaller and finally just get overhead, here is the jvm log
I wonder whether this indicates a memory leak or a jvm tune up matter and how to get the full gc recover most of the memory each time?
UPDATE
Thanks everyone for the guideline, after retrieve heap dump and analyze with Eclipse MAT, it looks all the leaking coming from org.jboss.as.web.deployment.WebInjectionContainer
here are screenshots of the results
800+m memory leak
UPDATE 2
I don't know if it is the same issue but I tried to apply the same changes from another thread, I can see the application use less memory but the leak is still there. the full GC only recover small amount of tenured generation hence(well young generate leaked anyway) so after several full gc the server got its overhead...
I am running Tomcat 6 on Linux and set Xms and Xmx in setenv.sh. After starting the Tomcat server I don't see the memory usage increased by at least specified in Xms.
Before startup free -m is showing 1500 as free memory.
In my startup.sh I can see memory settings.
CATALINA_OPTS: -Xms512m -Xmx1024m
After server startup free -m is showing 1140 as free memory. Why free memory is not reduced by at least 512m?
jinfo 14064 is showing VM Flags: -Xms512m -Xmx1024m
jmap -heap is also showing memory settings are applied
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 1073741824 (1024.0MB)
Is there anything I am missing?
free -m command
http://www.linuxtoday.com/upload/understanding-free-command-in-linuxunix-130530185004.html
This command will show Actual free RAM.
RAM memory and Virtual memory are different.
Actual initial heap size and max heap size can be allocation by heap
could be identified with the following command.
add the following jvm argument -verbose:gc
and restart the server. Once the server is restarted and after initialization the verbose logs shows the following
// attribute name="maxHeapSize" value="0x40000000"
//attribute name="initialHeapSize" value="0x20000000"
Converting the value into the MB it will show the Xms and Xmx allocation.
Above shown is the snippet of the following command
java -Xms512M -Xmx1024M -verbose:gc -version
A JVM crashed and created a JVM core pid file.
I am inexperienced with JMV core files so I could use help with following.
The error which I am getting is:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?
#
# Internal Error (allocation.cpp:117), pid=20119, tid=797133728
# Error: ChunkPool::allocate
#
# JRE version: 6.0_21-b06
# Java VM: Java HotSpot(TM) Server VM (17.0-b16 mixed mode linux-x86 )
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
And Heap memeory statistics is,
Heap
PSYoungGen total 248832K, used 123509K [0x89850000, 0x9efa0000, 0xb42f0000)
eden space 238656K, 47% used [0x89850000,0x90701918,0x98160000)
from space 10176K, 99% used [0x98ab0000,0x9949bea0,0x994a0000)
to space 56448K, 0% used [0x9b880000,0x9b880000,0x9efa0000)
PSOldGen total 699072K, used 404738K [0x342f0000, 0x5eda0000, 0x89850000)
object space 699072K, 57% used [0x342f0000,0x4ce30870,0x5eda0000)
PSPermGen total 29056K, used 28878K [0x302f0000, 0x31f50000, 0x342f0000)
object space 29056K, 99% used [0x302f0000,0x31f23be8,0x31f50000)
JVM arguments,
VM Arguments:
jvm_args: -Xms1024M -Xmx2048M -verbose:gc -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintCommandLineFlags -XX:+HeapDumpOnOutOfMemoryError
--------------- S Y S T E M ---------------
OS:Red Hat Enterprise Linux AS release 4 (Nahant Update 6)
uname:Linux 2.6.9-67.ELsmp #1 SMP Wed Nov 7 13:58:04 EST 2007 i686
libc:glibc 2.3.4 NPTL 2.3.4
rlimit: STACK 10240k, CORE infinity, NPROC 274431, NOFILE 4096, AS infinity
load average:1.32 1.50 1.52
CPU:total 4 (2 cores per cpu, 1 threads per core) family 15 model 65 stepping 3, cmov, cx8, fxsr, mmx, sse, sse2, sse3, mmxext, 3dnow, 3dnowext
Memory: 4k page, physical 16631944k(115380k free), swap 18940592k(18614440k free)
vm_info: Java HotSpot(TM) Server VM (17.0-b16) for linux-x86 JRE (1.6.0_21-b06), built on Jun 22 2010 01:04:46 by "java_re" with gcc 3.2.1-7a (J2SE release)
time: Sat Dec 24 11:09:25 2011
elapsed time: 84994 seconds
Based on the above details from the core file,
1) With ~18GB of swap free, any idea why the error is "Out of swap space?"? It should not be the case right. Only physical memory is very low, which is just ~115 MB of free space.
2) Out of 16 GB of physical memory, only 2 GB is allocated to JVM. But as per the stats, almost 16 GB is completely used and only 115 MB is free. So, other process also would have occupied the memory. Should I check in this direction?
3) Ideally JVM will create and handle java objects and also it has to create its own native library objects. Which memory will be used for JVM's own native objects. will it be allocated within the specified heap limit or it will be allocated completely outside the heap?
It will be really helpful if can you answer the above question for my understanding and analysis.
you have only 2gb allocated to your application so if the memory goes above it, you get an OOM Exception
jvm_args: -Xms1024M -Xmx2048M --> This is important
I couldn't find proper links to get you more information, but physical memory is exhausted in your case, 99% used (object space), which is causing error. You may need to increase heap memory allocation by max heap memory parameter -Xmx.
I have seen this error when you have run out of virtual memory. Since you have a 32-bit JVM and you are close to the limit with a 2 GB heap. You can use up the rest of the available address space with shared library, thread stacks and direct memory and get an Out of swap space error.
The simplest solution is to use a 64-bit JVM which avoids this issue entirely. I would also use Java 6 update 30 as it has a number of fixes and performance improvements.
BTW you can just write the follow which is the same as -Xms1024M -Xmx2048M
-ms1g -mx2g
The 64-bit JVM will still use 32-bit references in the latest JVMs so you shouldn't see a significant increase in memory consumption.