Why does my garbage collector(G1 GC) do not release memory altough he could?
The Used Heap is going up over time, the step fall at the end is because I forced the GC to release memory through jcmd <pid> GC.run
Why is that? Is that normal behaviour?
I started the GC with these parameter -Xms64m -Xmx256m -XX:G1ReservePercent=50
That graph looks like normal behavior.
G1 GC is a generational collector. That means that the heap is divided into (in this case) 2 generations, and the new generation is collected more frequently than the old one. Objects that survive a number of new generation collections get promoted to the old generation. The new generation is relatively small and is GC'd frequently. The old generation is larger, but it is only GC'd when it is starting to get full.
So ... the graph shows a series of "fine" sawtooths that correspond to the new generation collections. The line trends upwards as survivor objects get promoted to the old generation. And the old generation collection is not running ... because the JVM hasn't figured that the old generation is full enough to warrant collection.
Then, you called jcmd <pid> GC.run. That triggered an old generation collection, and you got back about 10MB of memory.
Then the new collection pattern resumed.
Q: Is this normal?
A: Yes.
Q: Is there a problem here?
A: There is no evidence of a problem. Certainly not with the GC itself. It is possible that your application has a memory leak, but there is no clear evidence one way or the other.
Q: What if you hadn't called jcmd <pid> GC.run?
A: At some point, the JVM will decide that the old generation is full enough to warrant starting an old generation collection. There is no need to force it. In fact, it is generally a bad idea to force it.
Q: Is this a flaw with GC? Why doesn't the GC release memory when >I< think it is free?
A: No it isn't a flaw. This is by design.
There is a trade-off between the (supposed) benefit of reclaiming memory early versus the cost of running the garbage collector. As a general rule, GC algorithms are most efficient when the ratio of collectable garbage to non-collectable objects is high; i.e. when the heap is getting close (but not too close) to full.
Note that even if you do force the GC to run, the JVM will be reluctant to give memory back to the OS for other processes to use.
It is totally common and its one of the flaws of Garbagecollection.
You can tell the Garbage collector that he should start cleaning your memory but it doesnt mean he will do it.
System.gc();
The official Oracle documentation states that:
Calling the gc method suggests that the Java Virtual Machine expend effort toward recycling unused objects in order to make the memory they currently occupy available for quick reuse
Related
In reading about ZGC I notice that it brags about being a 'single-generation garbage collector,' but I rarely see any details on exactly what this term means.
Normal Generational GC
I'm familiar with Eden, the survivor space, the nursery, hospice care, metaspace, permgen, zombified objects, Noah's Ark and the old age home, so I don't need an explanation of how concurrent mark sweep (CMS GC) or the garbage first (G1 GC) algorithms work. My understanding is those are both multi-generational, and I'm good with that.
Personally, I always liked being able to go through Java Mission Control and see how many generations of GC cycles an object has lived through. That was always helpful in troubleshooting a memory leak or GC problem.
Single vs Multi-Generational GC
So what exactly is a 'single-generation' garbage collector, and how does that differ from how objects are currently tracked through multiple garbage collection cycles in CMS and G1?
tl;dr
ZGC has much to brag about, but being single-generation is not among that.
Adding multi-generational is planned as future improvement.
Details
Some garbage collector implementations distinguish between new objects and old objects, the “young generation” versus “old generation”. The goal is to make it cheaper for the programmer to create short-lived objects as they will be more quickly and efficiently disposed of to free up memory.
ZGC does not “brag” about being single-generational. Quite the opposite, the team has indicated their desire to add generational features in future versions of ZGC. See a discussion in this section of a talk by Per Liden of Oracle (wait a few seconds for new slide “Generational ZGC”): https://youtu.be/88E86quLmQA
For more info on generations, see this Question: Java heap terminology: young, old and permanent generations?
You do know the details, but do you know why generational garbage collectors were invented? Specifically, why G1GC is generational?
The idea at the time was that scanning the entire heap was expensive, were expensive meant STW. G1 as a matter of fact started its life a non-generational (or single-generation as you call it), but the size of remembered sets and their time to go over them, triggered it to become generational. The fact that there are "generations" + the fact that G1 scans all the young regions in a single cycle (or better said if it "committed" to scan some X number of young regions at the beginning of some cycle - it must scan them all), means that you do not need for remembered sets to have connections between them in the young space (also card table is a lot smaller).
As to you direct question, ZGC (as well as Shenandoah) do not "track" how many cycles an Object has survived, they do not need to do that, as they have no generations, so no need to move Objects somewhere in the "old" to be scanned "later". They scan the entire heap on each cycle. They do this concurrently (in a very interesting way), so there is simply no need, at the moment, for them to be generational.
I am a newbie to Java. I understand what garbage collection and paging, swapping do in isolation. I wanted to understand how they relate to each other. Does GC cause paging or swapping? I know that the primary job of GC is to reclaim memory. But does it reclaim by paging or swapping if needed or is it not relevant to GC and is done by OS?
To understand the relation, note that Java performs generational garbage collection. There is a young and and old generation of objects allocated on the heap. From the JVM's point of view, it will not care about swapping but use the heap size it was configured with. However, the heap size will of course dictate the swapping behaviour of the OS that manages the JVM process.
In the young generation's collection, only rather new objects are collected. These objects should not have been swapped out by the OS due to their recent allocation. Of course, if you chose a size bigger than your RAM for your young generation, swapping will be required even for collecting the young generation what will slow down your garbage collector.
In the tenured generation, the performance of garbage collection will firstly depend on the strategy for collection. Consider a naive algorithm that performs a full garbage collection. This algorithm will have to check the entire application's object graph what requires access to the entire JVM heap. Obviously, the entire heap should fit into the RAM. Otherwise, a lot of swapping will be required for garbage collection what will result in a bad performance. In reality, the collector will not check the entire object graph. However, it remains a good practise to choose a heap size that fits into your RAM in order to avoid excessive swapping when for example configuring a Java application's production server.
Actually, I am a newbie in Java world too. I come here because I get confused with this just like you. However, if you thinking about the mechanism of swapping, you may find the difference between swap and garbage collection. swapping is a behaviour of operator system. and garbage collection is a behaviour of Java jvm which is a process/thread of the system.
So, swapping and garbage collection is different level thing. If memory is not enough, swapping will happen to figure out memory that unused temporary and then swapping out.
Above all, swapping may happen before full GC. and at the same time, garbage collection i.e. full gc. also may happen without swapping. A simple example is that: we config the JVM with a small heap size . bug we allocate a large buffer space, at this situation, full GC may happen and also OOM may happen.
I have three questions regarding garbage collection
I am trying to study the garbage collection in my application and I can notice that a full GC has occurred. By studying the GC logs I could find that old gen has not even used half the memory allocated to it. Then why would a full GC happen. Is there some other algorithm the JVM uses that releases the memory even when old gen is not completely utilized?
What can be called as a good GC trend. I mean if the full GC occurs at every 10- 15 mins can I call the application to be in a good state. I want to know how an ideal GC should be for an application. I know it depends considerably on the application, but there should be something to call ideal.
I have not set the NewSize or Newratio property. The default NewRatio in the machine seems to be 2. But I can see that my young gen is using only 1/4 th the heap size and the rest is used by tenured gen. How is this possible? All I have defined is the Xmx and permsize.
A major collection can happen for several reasons, in most cases you can see the cause by using jstat -gccause.
Few of the reasons are
-System.gc() if called from your app or any other code that you use and relies on this call.
-When the old space occupancy fractions has been reached
-When a PermGen collection takes place
-Depending on the collector you are using CMSIncrementalMode seems to be causing major collections before the limit of the old generation.
Most likely System.gc() is the cause of your unexpected major collections, try to use the flag -XX:+DisableExcplicitGC and see if you still get them.
--
There is no trend that can describe all usages. This should be based on your needs. Does the way your GC works now affect the performance of your app/service. Do you get long stop-the-world pauses that decrease your throughput ? What do you want to achieve? And the most important what is the garbage you are generating ? Try to analyze a heap dump and see if you can somehow reduce the numbers before you go and optimize the collector.
--
It depends on the flags you are using the version of the JVM your OS etc etc... In general GC ergonomics and more specifically the option -XX:+UseAdaptiveSizePolicy will be responsible of the sizings of your generations.
I have developed a J2ME web browser application, it is working fine. I am testing its memory consumption. It seems to me that it has a memory leak, because the green curve that represents the consumed memory of the memory monitor (of the wireless toolkit) reaches the maximum allocated memory (which is 687768 bytes) every 7 requests done by the browser, (i.e. when the end user navigates in the web browser from one page to other for 7 pages) after that the garbage collector runs and frees the allocated memory.
My question is:
is it a memory leak when the garbage collector runs automatically every 7 page navigation?
Do I need to run the garbage collector (System.gc()) manually one time per request to prevent the maximum allocated memory to be reached?
Please guide me, thanks
To determine if it is a memory leak, you would need to observe it more.
From your description, i.e. that once the maximum memory is reached, the GC kicks in and is able to free memory for your application to run, it does not sound like there is a leak.
Also you should not call GC yourself since
it is only an indication
could potentially affect the underlying algorithm affecting its performance.
You should instead focus on why your application needs so much memory in such a short period.
My question is: is it a memory leak when the garbage collector runs automatically every 7 page navigation?
Not necessarily. It could also be that:
your heap is too small for the size of problem you are trying to solve, or
your application is generating (collectable) garbage at a high rate.
In fact, given the numbers you have presented, I'm inclined to think that this is primarily a heap size issue. If the interval between GC runs decreased over time, then THAT would be evidence that pointed to a memory leak, but if the rate stays steady on average, then it would suggest that the rate of memory usage and reclamation are in balance; i.e. no leak.
Do I need to run the garbage collector (System.gc()) manually one time per request to prevent the maximum allocated memory to be reached?
No. No. No.
Calling System.gc() won't cure a memory leak. If it is a real memory leak, then calling System.gc() will not reclaim the leaked memory. In fact, all you will do is make your application RUN A LOT SLOWER ... assuming that the JVM doesn't ignore the call entirely.
Direct and indirect evidence that the default behaviour of HotSpot JVMs is to honour System.gc() calls:
"For example, the default setting for the DisableExplicitGC option causes JVM to honor Explicit garbage collection requests." - http://pic.dhe.ibm.com/infocenter/wasinfo/v7r0/topic/com.ibm.websphere.express.doc/info/exp/ae/rprf_hotspot_parms.html
"When JMX is enabled in this way, some JVMs (such as Sun's) that do distributed garbage collection will periodically invoke System.gc, causing a Full GC." - http://static.springsource.com/projects/tc-server/2.0/getting-started/html/ch11s07.html
"It is best to disable explicit GC by using the flag -XX:+DisableExplicitGC." - http://docs.oracle.com/cd/E19396-01/819-0084/pt_tuningjava.html
And from the Java 7 source code:
./openjdk/hotspot/src/share/vm/runtime/globals.hpp
product(bool, DisableExplicitGC, false, \
"Tells whether calling System.gc() does a full GC") \
where the false is the default value for the option. (And note that this is in the OS / M/C independent part of the code tree.)
I wrote a library that makes a good effort to force the GC. As mentioned before, System.gc() is asynchronous and won't do anything by itself. You may want to use this library to profile your application and find the spots where too much garbage is being produced. You can read more about it in this article where I describe the GC problem in detail.
That is (semi) normal behavior. Available (unreferenced) storage is not collected until the size of the heap reaches some threshold, triggering a collection cycle.
You can reduce the frequency of GC cycles by being a bit more "heap aware". Eg, a common error in many programs is to parse a string by using substring to not only parse off the left-most word, but also shorten the remaining string by substringing to the right. Creating a new String for the word is not easily avoided, but one can easily avoid repeatedly substringing the "tail" of the original string.
Running System.GC will accomplish nothing -- on most platforms it's a no-op, since it's so commonly abused.
Note that (outside of brain-dead Android) you can't have a true "memory leak" in Java (unless there's a serious JVM bug). What's commonly referred to as a "leak" in Java is the failure to remove all references to objects that will never be used again. Eg, you might keep putting data into a chain and never clear pointers to the stuff on the far end of the chain that is no longer going to be used. The resulting symptom is that the MINIMUM heap used (ie, the size immediately after GC runs) keeps rising each cycle.
Adding to the other excellent answers:
Looks like you are confusing memory leak with garbage collection.
Memory leak is when unused memory cannot be garbage collected because it still has references somewhere (although they're not used for anything).
Garbage collection is when a piece of software (the garbage collector) frees unreferenced memory automatically.
You should not call the garbage collector manually because that would affect its performance.
When I run a java program with the starting heap size of 3G (set by -Xms3072m VM argument), JVM doesn't start with that size. It start with 400m or so and then keeps on acquiring more memory as required.
This is a serious problem for me. I know JVM is going to need the said amount after some time. And when JVM increases is its memory as per the need, it slows down. During the time when JVM acquires more memory, considerable amount of time is spent in garbage collection. And I suppose memory acquisition is an expensive task.
How do I ensure that JVM actually respects the start heap size parameter?
Update: This application creates lots of objects, most of which die quickly. Some resulting objects are required to stay in memory (which get transferred out of young heap.) During this operation, all these objects need to be in memory. After the operation, I can see that all the objects in young heap are claimed successfully. So there are no memory leaks.
The same operation runs smoothly when the heap size reaches 3G. That clearly indicates the extra time required is spent in acquiring memory.
This Sun JDK 5.
If I am not mistaken, Java tries to get the reservation for the memory from the OS. So if you ask for 3 GB as Xms, Java will ask the OS, if this is available but not start with all the memory right away... it might even reserve it (not allocate it). But these are details.
Normally, the JVM runs up to the Xms size before it starts serious old generation garbage collection. Young generation GC runs all the time. Normally GC is only noticeable when old gen GC is running and the VM is in between Xms and Xmx or, in case you set it to the same value, hit roughly Xmx.
If you need a lot of memory for short lived objects, increase that memory area by setting the young area to... let's say 1 GB -XX:NewSize=1g because it is costly to move the "trash" from the young "buckets" into the old gen. Because in case it has not turned into real trash yet, the JVM checks for garbage, does not find any, copies it between the survivor spaces, and finally moves into the old gen. So try to suppress the check for the garbage in the young gen, when you know that you do not have any and postpone this somehow...
Give it a try!
I believe your problem is not coming from where you think.
It looks like what's costing you the most are the GC cycles, and not the allocation of heap size. If you are indeed creating and deleting lots of objects.
You should be focusing your effort on profiling, to find out exactly what is costing you so much, and work on refactoring that.
My hunch - object creation and deletion, and GC cycles.
In any case, -Xms should be setting minimum heap size (check this with your JVM if it is not Sun). Double-check to see exactly why you think it's not the case.
i have used sun's vm and started with minimum set to 14 gigs and it does start off with that.
maybe u should try setting both the xms and xmx values to the same amt, ie try this-
-Xms3072m -Xmx3072m
Why do you think the heap allocation is not right? Taking any operating system tool that shows only 400m does not mean it isn't allocated.
I don't get really what you are after. Is the 400m and above already a problem or is your program supposed to need that much? If you really have the need to deal with that much memory and it seems you need a lot of objects than you can do several things:
If the memory consumption doesn't match your gut feeling it is the right amount than you probably are leaking memory. That would explain why it "slows down" over time. Maybe you missed to remove objects from one structure so they don't get garbage collected and are slowing lookups and such down.
Your memory settings are maybe the trouble in itself. Garbage collection is not run per se. It is only called if there is some threshold reached. If you give it a big heap setting and your operating system has plenty of memory the garbage collection runs not often.
The characteristics you mentioned would be a scenario where a lot of objects are created and shortly after they would be deleted again. Otherwise the garbage collection wouldn't be a problem (some sort of generational gc). That means you have only "young" objects. Consider using an object pool if you are needing objects only a short period of time. That would eliminate the garbage collection at all.
If you know there are good times in your code for running gc you can consider running it manually to be able to see if it changes anything. This is what you would need
Runtime r = Runtime.getRuntime();
r.gc();
This is just for debugging purposes. The gc is doing a great job most of the time so there shouldn't be the need to invoke the gc on your own.