I have three questions regarding garbage collection
I am trying to study the garbage collection in my application and I can notice that a full GC has occurred. By studying the GC logs I could find that old gen has not even used half the memory allocated to it. Then why would a full GC happen. Is there some other algorithm the JVM uses that releases the memory even when old gen is not completely utilized?
What can be called as a good GC trend. I mean if the full GC occurs at every 10- 15 mins can I call the application to be in a good state. I want to know how an ideal GC should be for an application. I know it depends considerably on the application, but there should be something to call ideal.
I have not set the NewSize or Newratio property. The default NewRatio in the machine seems to be 2. But I can see that my young gen is using only 1/4 th the heap size and the rest is used by tenured gen. How is this possible? All I have defined is the Xmx and permsize.
A major collection can happen for several reasons, in most cases you can see the cause by using jstat -gccause.
Few of the reasons are
-System.gc() if called from your app or any other code that you use and relies on this call.
-When the old space occupancy fractions has been reached
-When a PermGen collection takes place
-Depending on the collector you are using CMSIncrementalMode seems to be causing major collections before the limit of the old generation.
Most likely System.gc() is the cause of your unexpected major collections, try to use the flag -XX:+DisableExcplicitGC and see if you still get them.
--
There is no trend that can describe all usages. This should be based on your needs. Does the way your GC works now affect the performance of your app/service. Do you get long stop-the-world pauses that decrease your throughput ? What do you want to achieve? And the most important what is the garbage you are generating ? Try to analyze a heap dump and see if you can somehow reduce the numbers before you go and optimize the collector.
--
It depends on the flags you are using the version of the JVM your OS etc etc... In general GC ergonomics and more specifically the option -XX:+UseAdaptiveSizePolicy will be responsible of the sizings of your generations.
Related
Why does my garbage collector(G1 GC) do not release memory altough he could?
The Used Heap is going up over time, the step fall at the end is because I forced the GC to release memory through jcmd <pid> GC.run
Why is that? Is that normal behaviour?
I started the GC with these parameter -Xms64m -Xmx256m -XX:G1ReservePercent=50
That graph looks like normal behavior.
G1 GC is a generational collector. That means that the heap is divided into (in this case) 2 generations, and the new generation is collected more frequently than the old one. Objects that survive a number of new generation collections get promoted to the old generation. The new generation is relatively small and is GC'd frequently. The old generation is larger, but it is only GC'd when it is starting to get full.
So ... the graph shows a series of "fine" sawtooths that correspond to the new generation collections. The line trends upwards as survivor objects get promoted to the old generation. And the old generation collection is not running ... because the JVM hasn't figured that the old generation is full enough to warrant collection.
Then, you called jcmd <pid> GC.run. That triggered an old generation collection, and you got back about 10MB of memory.
Then the new collection pattern resumed.
Q: Is this normal?
A: Yes.
Q: Is there a problem here?
A: There is no evidence of a problem. Certainly not with the GC itself. It is possible that your application has a memory leak, but there is no clear evidence one way or the other.
Q: What if you hadn't called jcmd <pid> GC.run?
A: At some point, the JVM will decide that the old generation is full enough to warrant starting an old generation collection. There is no need to force it. In fact, it is generally a bad idea to force it.
Q: Is this a flaw with GC? Why doesn't the GC release memory when >I< think it is free?
A: No it isn't a flaw. This is by design.
There is a trade-off between the (supposed) benefit of reclaiming memory early versus the cost of running the garbage collector. As a general rule, GC algorithms are most efficient when the ratio of collectable garbage to non-collectable objects is high; i.e. when the heap is getting close (but not too close) to full.
Note that even if you do force the GC to run, the JVM will be reluctant to give memory back to the OS for other processes to use.
It is totally common and its one of the flaws of Garbagecollection.
You can tell the Garbage collector that he should start cleaning your memory but it doesnt mean he will do it.
System.gc();
The official Oracle documentation states that:
Calling the gc method suggests that the Java Virtual Machine expend effort toward recycling unused objects in order to make the memory they currently occupy available for quick reuse
I've a AKKA-HTTP based service which is written in scala. This service works as a proxy for an API call. It creates a host connection pool for calling API using
https://doc.akka.io/docs/akka-http/current/client-side/host-level.html
The service is integrated with NewRelic and has the attached snapshots
I would like to understand the reasons for this kind of zig-zag patterns even when there is no traffic on the service and the connections in the host-pool gets terminated because of idle-timeout.
Moreover, I would also like to know Does the FULL GC will only occur after it reached a threshold say 7GB? or it can also occur at some other time when there is no traffic?
The service has XmX of 8GB. Moreover, there are also multiple dispatchers(fork-join-executor) which performs multiple tasks.
First, your graphs show a very healthy application. This "chainsaw" pattern is overall seen as a very good thing, without much to worry about.
When exactly a Full GC is going to happen is a bit hard to predict (I would use the word impossible, too). When your "live" objects have nowhere to move (because there is simply no space for that), a Full GC may be triggered. There are certain thresholds of when a concurrent phase (marking) is going to be initiated, but if that results in a Full GC or not is decided later.
Considering that G1 also re-sizes regions (makes them less/more) based on heuristics, and the fact that it can also shrink or grow your heap (up to -Xmx), the exact conditions when a Full GC might happen is not easy to predict (I guess some GC experts that know the exact internal details might be able to do that). Also, G1GC can do partial collections: when it collects young regions + some of the old regions (not all), still making it far better than a Full GC time-wise.
Unfortunately, your point about no traffic is correct. When there is very limited traffic, you might not get a Full GC, but immediately as traffic comes in, such a thing might happen. Old regions might slowly build up during your "limited traffic" and as soon as you have a spike - surprise. There are ways to trigger a Full GC on demand, and though I have heard of such applications that do this - I have not worked with one in practice.
In general with a GC that's not reference-counting, you'll see that zig-zag pattern because memory is only reclaimed when a GC runs.
G1 normally only collects areas of the heap where it expects to find a lot of garbage relative to live objects ("garbage collection" is a bit of a misnomer: it actually involves collecting the live objects and (in the case of a relocating garbage collector like G1) moving the live objects to a different area of the heap, which allows the area it collected in to then be declared ready for new allocations; therefore the fewer live objects it needs to handle, the less work it needs to do relative to the memory freed up).
At a high-level, G1 works by defining an Eden (a young generation) where newly created objects where newly created objects are allocated and it divides Eden into multiple regions with each thread being mapped to a region. When a region fills up, only that region is collected, with the survivors being moved into an older generation (this is simplifying). This continues until the survivor generation is full, at which point the survivor and eden generations are collected, with the surviving survivors being promoted to the old generation, and when the old generation fills up, you have a full GC.
So there isn't necessarily a fixed threshold where a full GC will get triggered, but in general the more heap gets used up, the more likely it becomes that a full GC will run. Beyond that, garbage collectors on the JVM tend to be more or less autonomous: most will ignore System.gc and/or other attempts to trigger a GC.
Conceivably with G1, if you allocated a multi-GiB array at startup, threw away the reference, and then after every period of idleness reallocated an array of the same size as the one you allocated at startup and then threw away the reference, you'd have a decent chance of triggering a full GC. This is because that array is big enough to bypass eden and go straight to the old generation where it will consume heap until the next full GC. Eventually there won't be enough contiguous free space in the old generation to allocate these arrays, and that will trigger a full GC. The only complications to this approach are that:
You'll eventually have to outsmart the JIT optimizer, which will see that you're allocating this array and throwing it away and decide that it doesn't actually have to allocate the array
If you have a long enough busy time that a full GC ran since the last allocate-and-throw-away, there's no guarantee that the allocation of the large array will succeed after a full GC, which will cause an OOM.
I know many things about what should be perceived from gc.logs like
you should check how frequently "Full GC" runs, if it is running frequently then it is sign of problem
you should also check how much memory "Full GC" is able to reclaim while finishes, if it is not much then again it is sign of problem as it would force "Full GC" to run again
you should revisit your heap space allocated for java process if "Full GC" runs frequently.
These are some points on which I am working on, I would be interested to know what else should be taken care, when I look at gc logs.
FYI, I have already gone through following threads....
What does "GC--" in gc.log mean?
What does "GC--" mean in a java garbage collection log?
How to analyse and monitor gc.log garbage collector log files from the JVM
Is gc.log writing asynchronous? safe to put gc.log on NFS mount?
First you need to know what wrong can GC do to your program. Depending on the type of collectors that you use for tenured and old gen contents of GC logs may vary. But all in all the baseline inference that we need to derive from gc logs is mostly concentrated to the following:
How long are the minor collections taking?
How long are the major collections taking?
What is the frequency of minor collections?
What is the frequency of major collections?
How much does a minor collection reclaim?
How much does a major collection reclaim?
Combinations of the above
Most Program have a very frequent minor collections that claim about 90-95% of heap and pass the rest to Survivor spaces. Subsequent collections clean up survivors by about 80% again and in essence just 2% to 4% of you actual minor collection makes it to old gen and tis cycles keeps on going no matter which Collector you use.
Now the pain areas are when you have hundreds of small sized minor collections per application request or thread and when added up they make a sizable time mostly in double digit seconds. Since in modern collectors minor pass and sweep are not stop the world cases so somethings this is bearable. With Old gen the problems come when collectors run but don't reclaim anything major. e.g: normally a collector runs when the old gen is about 80-85% full. This may be a stop the world episode since new data cannot be saved on heap unless the heap has more space which is probably the case here. So app threads are paused to let GC threads cleanup the space first. but once the collector finishes the heap fill ratio doesn't come down much as it should. A good sizing should reduce your heap by more than 40% in a single go. If it doesn't that means you need more heap to save your long lived objects.
So in essence GC analysis is not a 'do it based of a set of predefined steps' things. Its more of a hti and trial analysis. It more of an experiment were you set the initial sizes and settings and then note or monitor the GC activity and record findings. Then after say 8-10 runs you compare notes and see what works for your app and what doesn't. Its really an interesting hard work to do.
I am currently running an application which requires a maximum heap size of 16GB.
Currently I use the following flags to handle garbage collection.
-XX\:+UseParNewGC, -XX\:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=50, -XX\:+DisableExplicitGC, -XX\:+PrintGCDateStamps, -XX\:+PrintGCDetails, -Xloggc\:/home/user/logs/gc.log
However, I have noticed that during some garbage collections, the application locks up for a few seconds and then carries on - This is completely unacceptable as it's a game server.
An exert from my garbage collection logs can be found here.
Any advice on what I should change in order to reduce these long pauses would be greatly appreciated.
Any advice on what I should change in order to reduce these long pauses would be greatly appreciated.
The chances are that the CMS GC cannot keep up with the amount of garbage your system is generating. But the work that the GC has to perform is actually more closely related to the amount of NON-garbage that your system is retaining.
So ...
Try to reduce the actual memory usage of your application; e.g. by not caching so much stuff, or reducing the size of your "world".
Try to reduce the rate at which your application generates garbage.
Upgrade to a machine with more cores so that there are more cores available to run the parallel GC threads when necessary.
To Mysticial:
Yes in hindsight, it might have been better to implement the server in C++. However, we don't know anything about "the game". If it involves a complicated world model with complicated heterogeneous data structures, then implementing it in C++ could mean that that you replace the "GC pause" problem with the problem that the server crashes all the time due to problems with the way it manages its data structures.
Looking at your logs, I don't see any long pauses. But young GC is very frequent. Promotion rate is very low though (most garbage cleared by young GC as it should). At same time your old space utilization is low.
BTW are we talking about minecraft server?
To reduce frequency of young GC you should increase its size. I would suggest start with -XX:NewSize=8G -XX:MaxNewSize=8G
For such large young space, you should also reduce survivor space size -XX:SurvivorRatio=512
GC tuning is a path of trial and errors, so you may need some more iterations and tweaking.
You can find couple of useful articles at mu blog
HotSpot JVM GC options cheatsheet
Understanding young GC pauses in HotSpot JVM
I'm not an expert on Java garbage collection, but it looks like you're doing the right thing by using the concurrent collector (the UseConcMarkSweepGC flag), assuming the server has multiple processors. Follow the suggestions for troubleshooting at http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#cms. If you already have, let us know what happened when you tried them.
Which version of java are you using?http://docs.oracle.com/javase/7/docs/technotes/guides/vm/G1.html
For better try to minimize the use of instance variables in a class.It would be better to perform on local variables than instance varibles .It helps in gaining the performance and safe from synchronization problem.In the end of operation before exit of program always reset the used variables if you are using instance variables and set again when it is required. It helps more in enhancing performance.Besides in the version of java a good garbage collection policy is implemented.It would be better to move to new version if that is fleasible.
Also you can monitor the garbage collector pause time via VisualVm and you can get more idea when it is performing more garbage collection.
When I run a java program with the starting heap size of 3G (set by -Xms3072m VM argument), JVM doesn't start with that size. It start with 400m or so and then keeps on acquiring more memory as required.
This is a serious problem for me. I know JVM is going to need the said amount after some time. And when JVM increases is its memory as per the need, it slows down. During the time when JVM acquires more memory, considerable amount of time is spent in garbage collection. And I suppose memory acquisition is an expensive task.
How do I ensure that JVM actually respects the start heap size parameter?
Update: This application creates lots of objects, most of which die quickly. Some resulting objects are required to stay in memory (which get transferred out of young heap.) During this operation, all these objects need to be in memory. After the operation, I can see that all the objects in young heap are claimed successfully. So there are no memory leaks.
The same operation runs smoothly when the heap size reaches 3G. That clearly indicates the extra time required is spent in acquiring memory.
This Sun JDK 5.
If I am not mistaken, Java tries to get the reservation for the memory from the OS. So if you ask for 3 GB as Xms, Java will ask the OS, if this is available but not start with all the memory right away... it might even reserve it (not allocate it). But these are details.
Normally, the JVM runs up to the Xms size before it starts serious old generation garbage collection. Young generation GC runs all the time. Normally GC is only noticeable when old gen GC is running and the VM is in between Xms and Xmx or, in case you set it to the same value, hit roughly Xmx.
If you need a lot of memory for short lived objects, increase that memory area by setting the young area to... let's say 1 GB -XX:NewSize=1g because it is costly to move the "trash" from the young "buckets" into the old gen. Because in case it has not turned into real trash yet, the JVM checks for garbage, does not find any, copies it between the survivor spaces, and finally moves into the old gen. So try to suppress the check for the garbage in the young gen, when you know that you do not have any and postpone this somehow...
Give it a try!
I believe your problem is not coming from where you think.
It looks like what's costing you the most are the GC cycles, and not the allocation of heap size. If you are indeed creating and deleting lots of objects.
You should be focusing your effort on profiling, to find out exactly what is costing you so much, and work on refactoring that.
My hunch - object creation and deletion, and GC cycles.
In any case, -Xms should be setting minimum heap size (check this with your JVM if it is not Sun). Double-check to see exactly why you think it's not the case.
i have used sun's vm and started with minimum set to 14 gigs and it does start off with that.
maybe u should try setting both the xms and xmx values to the same amt, ie try this-
-Xms3072m -Xmx3072m
Why do you think the heap allocation is not right? Taking any operating system tool that shows only 400m does not mean it isn't allocated.
I don't get really what you are after. Is the 400m and above already a problem or is your program supposed to need that much? If you really have the need to deal with that much memory and it seems you need a lot of objects than you can do several things:
If the memory consumption doesn't match your gut feeling it is the right amount than you probably are leaking memory. That would explain why it "slows down" over time. Maybe you missed to remove objects from one structure so they don't get garbage collected and are slowing lookups and such down.
Your memory settings are maybe the trouble in itself. Garbage collection is not run per se. It is only called if there is some threshold reached. If you give it a big heap setting and your operating system has plenty of memory the garbage collection runs not often.
The characteristics you mentioned would be a scenario where a lot of objects are created and shortly after they would be deleted again. Otherwise the garbage collection wouldn't be a problem (some sort of generational gc). That means you have only "young" objects. Consider using an object pool if you are needing objects only a short period of time. That would eliminate the garbage collection at all.
If you know there are good times in your code for running gc you can consider running it manually to be able to see if it changes anything. This is what you would need
Runtime r = Runtime.getRuntime();
r.gc();
This is just for debugging purposes. The gc is doing a great job most of the time so there shouldn't be the need to invoke the gc on your own.