We Keep Coding

[ZGC]Why the memory used is much larger than the dump file?

Here is some information:
Linux version 3.18.6-2.el7.centos.x86_64、openjdk-11.0.2
using ZGC and -Xms16384M -Xmx16384M
top command shows : 17.013t VIRT, 0.016t RES, 0.015t SHR, 35.2 %CPU, 13.0%MEM
heap dump size is 83M(using jcmd command)
VisualVM Summary shows heap size is 55M
In most GC situations:Garbage Collection (Proactive) 1808M(11%)->166M(1%)
I have searched and read following answers, but still can't solve my doubts.
The Java ZGC garbage collector USES a lot of memory
ZGC maxheap size exceed physical memory
Why is my Java heap dump size much smaller than used memory?

I'm not sure what exactly the point 6 is, it would be more helpful if you showed a few lines from your garbage collector log file. I also don't use VisualVM, so can't comment on that.
So this answer is mostly a guess.
The memory used by the JVM is divided into different regions, most importantly you have a heap and a stack. But still you will see that the heap will be bigger than what you configured with -Xmx because there is an area called Metaspace (or Permanent Generation in older JVMs) that holds static information like classes or for example String literals from your source code. And the Metaspace is on top of the "normal" heap. My guess is that some tools count it towards heap and some don't. And that might be where you get the different numbers from.

JPype / Java - Initialize with, or get, remaining heap space

We have software written in Python, which uses JPype to call Java, which performs various resource heavy calculations / report building. We originally assigned 800mb of heap space when starting the JVM. The java side is fully multithreaded and will work with whatever resources are available to it.
jvmArgs = ["-Djava.class.path=" + classpath, "-Xmx800M"]
jpype.startJVM(u"java\\jre8\\bin\\client\\jvm.dll", *jvmArgs)
This worked well until we tested on Windows XP for our legacy clients. The new machines are Win 7 64-bit with 4GB of RAM, whereas the old ones are Win XP 32-bit with only 2 GB of ram.
The issue is that JPype causes our application to ungracefully and silently crash if we allocate too much memory. A try catch doesn't even get triggered on the statement above.
I'm wondering if there's a way to use java from command line to determine how much memory we can allocate on a computer. We can check if it's 32-bit or 64-bit which helps, but we need to make sure they aren't running other programs taking up heap space on the JVM. If they are, our application will crash.
Reader's Digest: We'd like to allocate 500mb of heap space when initializing the JVM, but can't be sure of how much space is currently being used. If we allocate too much, the entire application silently crashes.
We use the following
JPype: 0.5.4.2
Python: 2.7
Java: 1.8 or 1.7 (64-bit or 32-bit)
Thanks.

The memory consumed by JVM consists of 2 main areas:
Heap memory
Non heap memory - MetaSpace, native method stacks, pc register, direct byte buffers, sockets, jni allocated memory, thread stacks and more
While the maximum size that will be used for the heap memory is known and configurable, the size of the non heap memory cannot be fully controlled.
The size of the native memory used by the JVM will be effected by the number of threads you use, the amount of classes being loaded and the use of buffers (use of I/O).
You can limit the size of the metapsace by setting the MaxMetaspaceSize (-XX:MaxMetaspaceSize). You can control the amount of memory used for thread stacks by limiting the number of threads and setting the thread stack size (-Xss).
Assuming you do not have native memory leaks, the amount of classes being loaded is stable (no excessive use of dynamic proxies and bytecode generation) and the amount of threads being used is known - you can speculate how much memory will be required for your application to run by monitoring the overall memory used by the JVM over a period of time. When you do it, make sure the entire heap is being allocated when the JVM starts.

Appropriate sizing of heap and old gen for JVM for data heavy application

I am running a server application using JVM sunjava-1.6.0_21.
My application is data heavy and acts as a cache server. So it stores a lot of long living data that we do not expect to get GC throughout the application is running.
I am setting following JVM parameters -Xmx16384M and -Xms16384M.
After the required data has been loaded, following is the memory usage of application
Total heap space is :13969522688
Maximum heap space is :15271002112
Free heap space is :3031718040
Long term (old gen) heap storage: Used=10426MB Max=10922MB
Used/Max=95%
Old gen usage - I have confirmed that is due to actual data and is not expected to get free.
My question is that by default JVM sizing of heap space ( it is allocating 10922MB old gen), that is leaving very little free space in old gen section.
Can less free space in old gen impact the application?
If yes, how should I handle this? Should I experiment with JVM tuning parameters like newratio and try to increase space available for old gen or any other way I should tune the application.

Can less free space in old gen impact the application?
If your Tenured Gen gets full a major collection will happened and this type of collection is costly.
You can use the options : -verbose:gc and -XX:+PrintGCDetails to know if Full GC happend too often. If it's the case so yes it can impact the performance of your application.
If yes, how should I handle this? Should I experiment with JVM tuning parameters like newratio and try to increase space available for old gen or any other way I should tune the application.
You can give a try to NewRatio but keep in mind that if your eden is too short your tenured gen will probably get filled faster.
To conclude, you should use a monitoring tool to have a better idea of the VM options you have to use. It will show you easily how your generations are filled during the app execution, it's easier to read and understand than gc logs ;)

If you know that the length of life of your objects is that long play with the parameters that set the size of the regions in relation to each other.
You can set ratios in young generation and old generation (eden and ternured spaces) as well as both survivors.
The target is to minimize the full garbage collection by allowing the minor garbage collection to free all memory.
You prevent the garbage collection to free objects by keeping them reachable in your application. I mean that you should only care for those objects being deleted by minor garbage collections.
Enable the parameters
-verbose:gc -Xloggc:/opt/tomcat/logs/gc.out -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
Then with a tool GCViewer you can see time spent in gc and the number (size) of objects removed. Among some useful metrics.

Jmap heap dump, does it include young generation?

Quick question
Does the jmap heap dump include only the old generation, or also the young generation ?
Long explanation
I have 2 heap dump (jmap -heap:format=b 9999) :
one of my server with no load (no HTTP request)
one while working 50% CPU, a high load (benchmarking)
Now, the first dump shows a heap size bigger than the second (which I thought was weird).
Could that be because the Young generation (at high load) is changing often because the Garbage collector is running often (yes, the JVM is almost full) ? Old generation is full at 99%, I've noticed the young generation space usage vary a lot.
So that would mean that I made the second dump right after the GC did his job, this is why its size is smaller. Am I right ?
Additionnal informations :
Java args :
-XX:+UseParallelGC -XX:+AggressiveHeap
-Xms2048m -Xmx4096m -XX:NewSize=64m
-XX:PermSize=64m -XX:MaxPermSize=512m

Quick Answer
Both - The Heap is made up of the Young and Old Generation. So when you take a heap dump, the contents contains both. The Stats of the heap dump should be seperated. Try removing the binary portion of your command and look at it in plain text. You will see a summary of your configuration, and then a break down of each generation. On the flip side, a -histo would just show all objects on the heap with no distinction
Long Answer
It could be that a garbage collection had just finished for the 2nd process. Or the opposite, the first process may not have had a full collection in a while and was sitting at higher memory. Was this application/server just restarted when you took the capture? If you look at an idle process using a tool like jvisualvm, you will see the memory allocation graphs move up and down even though your process isn't doing any work. This is just the JVM doing its own thing.
Typically your Full GC should kick off well before it reaches a 99% mark in the Old Gen. The JVM will decide when to run the full GC. Your Young Gen will fluctuate alot as this is where objects are created/removed the fastest. There will be many partial GC's done to clean out the young gen before a full GC get run. The difference between the two will mean a pause in your JVM activity. Seeing a partial GC will not hurt your application. But the pause times of a full GC will stop your application while it runs. So you will want to minimize those the best you can.
If you are looking for memory leaks or just profiling to see how your application(s) gc is working, I would recommend using the startup flags to print the garbage collection stats.
-XX:+PrintGCDetails -verbose:gc -Xloggc:/log/path/gc.log
run your program for a while and then load the captured log in a tool to help visualize the results. I personally use the Garbage Collection and Memory Visualizer offered in the IBM Support Assistant Workbench. It will provide you with a summary of the captured garbage collection stats as well as a dynamic graph which you can use to see how the memory in your application has been behaving. This will not give you what objects were on your heap.
At problem times in your application, if you can live with a pause, I would modify your jmap command to do the following
jmap -dump:format=b,file=/file/location/dump.hprof <pid>
Using a Tool like MAT, you will be able to see all of the objects, leak suspects, and various other stats about the generations, references of the heap.
Edit For Tuning discussion
Based on your start-up parameters there are a few things you can try
Set your -Xms equal to the same value
as your -Xmx. By doing this the JVM
doesn't have to spend time allocating
more heap. If you expect your application to take 4gb, then give it all right away
Depending on the number of processors
on the system you are running this
application you can set the flag for
-XX:ParallelGCThreads=##.
I havent tried this one but the
documentation shows a parameter for
-XX:+UseParallelOldGC which shows some promise to reduce time of old GC
collection.
Try changing your new generation size to 1/4 of the Heap. (1024 instead of 64) This could be forcing too much to the old generation. The default size is around 30%, you have your application configured to use around 2% for young gen. Though you have no max size configured, I worry that too much data would be moved to old gen because the new gen is too small to handle all of the requests. Thus you have to perform more full (paused) GC's in order to clean up the memory.
I do believe overall if you are allocating this much memory so fast, then it could be a generation problem where the memory is allocated to the Old generation prematurely. If a tweak to your new gen size doesn't work, then you need to either add more heap via the -Xmx configuration, or take a step back and find exactly what is holding onto the memory. The MAT tool which we discussed earlier can show you the references holding onto the objects in memory. I would recommend trying bullet points 1 & 4 first. This will be trial and error for you to get the right values

Does the Sun JVM slow down when more memory is allocated via -Xmx?

Does the Sun JVM slow down when more memory is available and used via -Xmx? (Assumption: The machine has enough physical memory so that virtual memory swapping is not a problem.)
I ask because my production servers are to receive a memory upgrade. I'd like to bump up the -Xmx value to something decadent. The idea is to prevent any heap space exhaustion failures due to my own programming errors that occur from time to time. Rare events, but they could be avoided with my rapidly evolving webapp if I had an obscene -Xmx value, like 2048mb or higher. The application is heavily monitored, so unusual spikes in JVM memory consumption would be noticed and any flaws fixed.
Possible important details:
Java 6 (runnign in 64-bit mode)
4-core Xeon
RHEL4 64-bit
Spring, Hibernate
High disk and network IO
EDIT: I tried to avoid posting the configuration of my JVM, but clearly that makes the question ridiculously open ended. So, here we go with relevant configuration parameters:
-Xms256m
-Xmx1024m
-XX:+UseConcMarkSweepGC
-XX:+AlwaysActAsServerClassMachine
-XX:MaxGCPauseMillis=1000
-XX:MaxGCMinorPauseMillis=1000
-XX:+PrintGCTimeStamps
-XX:+HeapDumpOnOutOfMemoryError

By adding more memory, it will take longer for the heap to fill up. Consequently, it will reduce the frequency of garbage collections. However, depending on how mortal your objects are, you may find that how long it takes to do any single GC increases.
The primary factor for how long a GC takes is how many live objects there are. Thus, if virtually all of your objects die young and once you get established, none of them escape the young heap, you may not notice much of a change in how long it takes to do a GC. However, whenever you have to cycle the tenured heap, you may find everything halting for an unreasonable amount of time since most of these objects will still be around. Tune the sizes accordingly.

If you just throw more memory at the problem, you will have better throughput in your application, but your responsiveness can go down if you're not on a multi core system using the CMS garbage collector. This is because fewer GCs will occur, but they will have more work to do. The upside is that you will get more memory freed up with your GCs, so allocation will continue to be very cheap, hence the higher througput.
You seem to be confusing -Xmx and -Xms, by the way. -Xms just sets the initial heap size, whereas -Xmx is your max heap size.

More memory usually gives you better performance in garbage collected environments, at least as long as this does not lead to virtual memory usage / swapping.
The GC only tracks references, not memory per se. In the end, the VM will allocate the same number of (mostly short-lived, temporary) objects, but the garbage collector needs to be invoked less often - the total amount of garbage collector work will therefore not be more - even less, since this can also help with caching mechanisms which use weak references.
I'm not sure if there is still a server and a client VM for 64 bit (there is for 32 bit), so you may want to investigate that also.

According to my experience, it does not slow down BUT the JVM tries to cut back to Xms all the time and try to stay at the lower boundary or close to. So if you can effort it, bump Xms as well. Sun is recommending both at the same size. Add some -XX:NewSize=512m (just a made up number) to avoid the costly pile up of old data in the old generation with leads to longer/heavier GCs on the way. We are running our web app with 700 MB NewSize because most data is short-lived.
So, bottom line: I do not expect a slow down, but put your more of memory to work. Set a larger new size area and set Xms to Xmx to lower the stress on the GC, because it does not need to try to cut back to Xms limits...

It typically will not help your peformance/throughput,if you increase -Xmx.
Theoretically there could be longer "stop the world" phases but in practice with the CMS that's not a real problem.
Of course you should not set -Xmx to some insane value like 300Gbyte unless you really need it :)