My java application is frequently used in fullgc. I use jmap -dump:file=xxx to get a heap file. After using the mat tool for analysis, I can't find the problem, but I find that there are many 'unreachable_ Objects' is about 1g. I want to know which code generates so many garbage objects, but mat can't see the details. It can only see the byte[] type. How do I find references to these bytes[]?
Here is my JVM configuration
-server -Xmx6g -Xms6g -XX:NewRatio=2 -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:MaxTenuringThreshold=6 -XX:+ParallelRefProcEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:+HeapDumpOnOutOfMemoryError -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/export/Logs/gc.log
Environment
The mat picture is as follows
unreachable_objects
Object histogram
Please help check why my GC logs failed to generate, or is there anything wrong with it.
My java start parameter is
-Xms5G -Xmx5G -Xss1024K
-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:G1HeapRegionSize=4m
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:-PrintCommandLineFlags
-XX:+PrintAdaptiveSizePolicy -XX:+PrintTenuringDistribution -XX:+PrintReferenceGC
-Xloggc:/data/gclog/gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=100M
I've set a rotation to generate JVM G1 GC logs. It's my first time use the G1 GC. But it always create a single log "gc.0.current" with very few contents, "gc" is the name of the log. The content is as below.
Java HotSpot(TM) 64-Bit Server VM (25.73-b02) for linux-amd64 JRE (1.8.0_73-b02), built on Jan 29 2016 17:39:45 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
Memory: 4k page, physical 8056924k(5058272k free), swap 4194300k(4194300k free)
CommandLine flags: -XX:G1HeapRegionSize=4194304 -XX:GCLogFileSize=104857600 -XX:InitialHeapSize=5368709120 -XX:MaxGCPauseMillis=200 -XX:MaxHeapSize=5368709120 -XX:NumberOfGCLogFiles=5 -XX:+PrintAdaptiveSizePolicy -XX:-PrintCommandLineFlags -XX:+PrintGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintReferenceGC -XX:+PrintTenuringDistribution -XX:ThreadStackSize=1024 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:+UseGCLogFileRotation
0.012: [G1Ergonomics (Heap Sizing) expand the heap, requested expansion amount: 5368709120 bytes, attempted expansion amount: 5368709120 bytes]
2019-08-21T23:23:26.706-0500: 0.212: Application time: 0.1403261 seconds
2019-08-21T23:23:26.706-0500: 0.212: Total time for which application threads were stopped: 0.0001008 seconds, Stopping threads took: 0.0000202 seconds
Heap
garbage-first heap total 5242880K, used 8192K [0x0000000680000000, 0x0000000680402800, 0x00000007c0000000)
region size 4096K, 3 young (12288K), 0 survivors (0K)
Metaspace used 7309K, capacity 7452K, committed 7552K, reserved 1056768K
class space used 853K, capacity 888K, committed 896K, reserved 1048576K
2019-08-21T23:23:26.775-0500: 0.281: Application time: 0.0686930 seconds
No other logs created. Do you know why is that? I've run a lot of tasks on the server, suppose it should generate some GC logs.
Can you see anything abnormal with the above log content?
Is there any other way to view the GC activities on the Linux server ?
Thanks,
Heap layout printed at the bottom of GC log indicates that your Java process has terminated.
Heap layout is printed in two situations: before/after GC, if -XX:+PrintHeapAtGC is specified, and before JVM exits. In your case it is obviously the latter.
JVM exited almost immediately after start (it was running only for 281 milliseconds). Most likely, it either failed to start an application (check the VM output), or it did something very simple, i.e. just launched a new subprocess.
Note that a new JVM process overwrites the previosly written log file.
I tested all parameters on simple CRUD service and has not problem. But when I test on Hello world application, the printed log was the same as yours because GC not triggered. When I triggered GC manually with System.gc()on Hello World application, printed GC logs.
In summary, if GC does not take any action, log does not print.
Can be a problem with -XX:+UseGCLogFileRotation. I encountered the same issue before and this works for me:
-Xloggc:/data/gclog/gc
instead of
-Xloggc:/data/gclog/gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=100M
Here is my notes when tuning G1GC on my production, hope you can find some useful info: How Garbage First Garbage Collector (G1GC) affected the performance of our back-end.
Also, there is an advice to Avoid -XX:+UseGCLogFileRotation. One GC log file would be easier to be imported to analysing tools like GC Viewer.
Ran into this, the weird thing is, I can't find any pages about it. Searching for "PrintHeapAtGC not recognized" on google results in nothing helpful. In fact, it results in thread talking about how people are using it in JVM 11 with no problem.
I've printed the flags I use one per line, perhaps it's because of some combination I don't know is even a combination?
How do I even check that?
Can someone have a look at this and tell me what's wrong?
-XX:+ExitOnOutOfMemoryError
-XX:NewRatio=3
-XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90
-XX:MaxTenuringThreshold=8
-XX:+UseParallelGC
-XX:ParallelGCThreads=4
-XX:+CMSScavengeBeforeRemark
-XX:PretenureSizeThreshold=64m
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
-XX:CMSMaxAbortablePrecleanTime=6000
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-XX:-OmitStackTraceInFastThrow -verbose:gc
-XX:+PrintHeapAtGC
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime -Xloggc:/tmp/gc.log
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=9
-XX:GCLogFileSize=20M
-XX:+PrintFlagsFinal
-XX:+PrintFlagsWithComments -version
There is no more -XX:+PrintHeapAtGC and some other Print options since JDK 9.
The -XX logging flags were replaced with Unified JVM Logging mechanism.
New equivalent for PrintHeapAtGC is -Xlog:gc+heap=debug
Our web application keep on crashing with error message as "GC overhead limit exceeded" after 30 minutes from start time.
I generated heap dump at that moment and loaded into eclipse MAT. It mentioned that TDSPacket as the problem suspect. As this is the library we are using to connect to SQL server, I am unable to trouble shoot from here.
Any suggestion please. Here are the JAVA_OPTIONS used.
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\Apps\Logs\tomcat\memory -Xms2g -Xmx2g -XX:+UseG1GC -XX:+DisableExplicitGC -Xloggc:gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=50 -XX:GCLogFileSize=2M exit -Xms2048m -Xmx2048m
I'm wrote simple jRuby scriptlets support in my tomcat 7 java application. If I invoke page, that using jRuby first time, it loads about 3-7 seconds. Next time it loads faster. But, after some time jRuby unloading from memory, and after request I'm again wait 3-7 seconds.
Does exists any methods to always keep jRuby in memory withouth unloading?
PS. Current tomcat run options:
-Xmx2048M -Xms2048M -XX:ParallelGCThreads=8 -Xincgc -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSIncrementalPacing -XX:+AggressiveOpts -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -XX:MaxGCPauseMillis=500 -XX:SurvivorRatio=16 -XX:TargetSurvivorRatio=90 -XX:+UseAdaptiveGCBoundary -XX:-UseGCOverheadLimit -Xnoclassgc -XX:UseSSE=3 -XX:PermSize=512m -XX:LargePageSizeInBytes=4m
I'm using Oracle java 6u26.