I am exploring a tree of stuff in Java, but I am memory limited. I handle this by looking at occupancy stats, and when I'm over 80% (say) stop allocating new bits of the tree to avoid the OutOfMemory. And just compute using what I've got so far using the 20% headroom.
Then every now and again I decide to move down the tree. This should free up 90% of the tree I've allocated because I forget the old root and move to one of it's children. And it does free that memory, but only if I call System.gc(), which is evil and stops my world. But if I don't call gc() my limiting code stops me adding to my tree.
What I want to do is call out to the G1 or CMS collectors that I could really do with the old generation getting a clean, and carrying on computing and allocate new bits of the tree as the collector does it's job.
Any ideas how that can be accomplished? Or, equally helpful, how I could avoid this artificial 80% limit, that is the root of my problem.
Setting -XX:+ExplicitGCInvokesConcurrent should trigger a concurrent cycle if you invoke System.gc()
Alternatively you could use -XX:CMSInitiatingOccupancyFraction=50 -XX:+UseCMSInitiatingOccupancyOnly for CMS which will initiate concurrent cycles once the heap reaches that threshold, since you're aiming for 80%+ occupancy that would basically continuously burn CPU time (possibly limited by GCTimeRatio, i'm not sure how those goals interact) to clean the old generation and thus free up memory swiftly.
The equivalent parameter for G1GC (InitiatingHeapOccupancyPercent) already defaults to 45%, so it should already be running mixed collection cycles if you have reached that point. You should inspect GC logs to verify.
Or, equally helpful, how I could avoid this artificial 80% limit, that is the root of my problem.
overprovision memory, RAM is often cheaper than developer hours and GCs need some breathing room to work efficiently anyway
consider using soft references to hold onto data that's useful but can be discarded by the GC if needed. They are not free though, so see if the extra costs are worth it.
Related
I've a AKKA-HTTP based service which is written in scala. This service works as a proxy for an API call. It creates a host connection pool for calling API using
https://doc.akka.io/docs/akka-http/current/client-side/host-level.html
The service is integrated with NewRelic and has the attached snapshots
I would like to understand the reasons for this kind of zig-zag patterns even when there is no traffic on the service and the connections in the host-pool gets terminated because of idle-timeout.
Moreover, I would also like to know Does the FULL GC will only occur after it reached a threshold say 7GB? or it can also occur at some other time when there is no traffic?
The service has XmX of 8GB. Moreover, there are also multiple dispatchers(fork-join-executor) which performs multiple tasks.
First, your graphs show a very healthy application. This "chainsaw" pattern is overall seen as a very good thing, without much to worry about.
When exactly a Full GC is going to happen is a bit hard to predict (I would use the word impossible, too). When your "live" objects have nowhere to move (because there is simply no space for that), a Full GC may be triggered. There are certain thresholds of when a concurrent phase (marking) is going to be initiated, but if that results in a Full GC or not is decided later.
Considering that G1 also re-sizes regions (makes them less/more) based on heuristics, and the fact that it can also shrink or grow your heap (up to -Xmx), the exact conditions when a Full GC might happen is not easy to predict (I guess some GC experts that know the exact internal details might be able to do that). Also, G1GC can do partial collections: when it collects young regions + some of the old regions (not all), still making it far better than a Full GC time-wise.
Unfortunately, your point about no traffic is correct. When there is very limited traffic, you might not get a Full GC, but immediately as traffic comes in, such a thing might happen. Old regions might slowly build up during your "limited traffic" and as soon as you have a spike - surprise. There are ways to trigger a Full GC on demand, and though I have heard of such applications that do this - I have not worked with one in practice.
In general with a GC that's not reference-counting, you'll see that zig-zag pattern because memory is only reclaimed when a GC runs.
G1 normally only collects areas of the heap where it expects to find a lot of garbage relative to live objects ("garbage collection" is a bit of a misnomer: it actually involves collecting the live objects and (in the case of a relocating garbage collector like G1) moving the live objects to a different area of the heap, which allows the area it collected in to then be declared ready for new allocations; therefore the fewer live objects it needs to handle, the less work it needs to do relative to the memory freed up).
At a high-level, G1 works by defining an Eden (a young generation) where newly created objects where newly created objects are allocated and it divides Eden into multiple regions with each thread being mapped to a region. When a region fills up, only that region is collected, with the survivors being moved into an older generation (this is simplifying). This continues until the survivor generation is full, at which point the survivor and eden generations are collected, with the surviving survivors being promoted to the old generation, and when the old generation fills up, you have a full GC.
So there isn't necessarily a fixed threshold where a full GC will get triggered, but in general the more heap gets used up, the more likely it becomes that a full GC will run. Beyond that, garbage collectors on the JVM tend to be more or less autonomous: most will ignore System.gc and/or other attempts to trigger a GC.
Conceivably with G1, if you allocated a multi-GiB array at startup, threw away the reference, and then after every period of idleness reallocated an array of the same size as the one you allocated at startup and then threw away the reference, you'd have a decent chance of triggering a full GC. This is because that array is big enough to bypass eden and go straight to the old generation where it will consume heap until the next full GC. Eventually there won't be enough contiguous free space in the old generation to allocate these arrays, and that will trigger a full GC. The only complications to this approach are that:
You'll eventually have to outsmart the JIT optimizer, which will see that you're allocating this array and throwing it away and decide that it doesn't actually have to allocate the array
If you have a long enough busy time that a full GC ran since the last allocate-and-throw-away, there's no guarantee that the allocation of the large array will succeed after a full GC, which will cause an OOM.
I have a repeating process that:
gets some data from the database
builds some objects in memory, adding to a Collection
writes the data from the Collection to a file
All of the objects/Collections go out of scope or are set to null after each iteration. (The Collection is reused for each iteration.)
Using Java VisualVM, I see a graph that looks like this, which seems very odd considering that it's a repeating process. Yes, the data coming back from the database is different, but it's generally the same amount.
Why does the heap size decrease at first?
Why does the used heap get so close to the heap size in the middle?
(the ~30-second blip at 1:43 was just when VisualVM froze momentarily)
I'm not as big of an expert on GC as some are, but the general idea is that when you've started the program you've given it the initial heap size, max heap size and other relevant parameters and then it's go time.
However the GC has plenty of intelligence and different algorithms that are optimized for different kinds of tasks. A naive implementation would just keep the heap size static and then collect the garbage when it's full. That's known as a "stop the world" collection, because the collector needs to stop everything so it can perform a little (or big) clean up.
Modern GC doesn't just cause long pauses into running programs because it needs to clean up, so there's always a little clean up going on as seen from the sawtooth. But when you start a program the GC has no idea what the program is going to do and how it will use memory. Therefore it has to observe what's happening, analyze memory usage and then decide what amounts of memory it needs to keep available for immediate use, whether it needs to grow the current heap size or if it can decrease the current heap size.
Depending on the behaviour of your program and the GC algorithm being used you can see a lot of different patterns. As long as you're not experiencing linear growth that ends up in an OutOfMemoryError, you should be relatively safe. Unless of course you want to optimize what's happening to increase throughput, responsiveness etc., but that's a more advanced subject and is more relevant when you've gotten your code working the way you want it.
I know many things about what should be perceived from gc.logs like
you should check how frequently "Full GC" runs, if it is running frequently then it is sign of problem
you should also check how much memory "Full GC" is able to reclaim while finishes, if it is not much then again it is sign of problem as it would force "Full GC" to run again
you should revisit your heap space allocated for java process if "Full GC" runs frequently.
These are some points on which I am working on, I would be interested to know what else should be taken care, when I look at gc logs.
FYI, I have already gone through following threads....
What does "GC--" in gc.log mean?
What does "GC--" mean in a java garbage collection log?
How to analyse and monitor gc.log garbage collector log files from the JVM
Is gc.log writing asynchronous? safe to put gc.log on NFS mount?
First you need to know what wrong can GC do to your program. Depending on the type of collectors that you use for tenured and old gen contents of GC logs may vary. But all in all the baseline inference that we need to derive from gc logs is mostly concentrated to the following:
How long are the minor collections taking?
How long are the major collections taking?
What is the frequency of minor collections?
What is the frequency of major collections?
How much does a minor collection reclaim?
How much does a major collection reclaim?
Combinations of the above
Most Program have a very frequent minor collections that claim about 90-95% of heap and pass the rest to Survivor spaces. Subsequent collections clean up survivors by about 80% again and in essence just 2% to 4% of you actual minor collection makes it to old gen and tis cycles keeps on going no matter which Collector you use.
Now the pain areas are when you have hundreds of small sized minor collections per application request or thread and when added up they make a sizable time mostly in double digit seconds. Since in modern collectors minor pass and sweep are not stop the world cases so somethings this is bearable. With Old gen the problems come when collectors run but don't reclaim anything major. e.g: normally a collector runs when the old gen is about 80-85% full. This may be a stop the world episode since new data cannot be saved on heap unless the heap has more space which is probably the case here. So app threads are paused to let GC threads cleanup the space first. but once the collector finishes the heap fill ratio doesn't come down much as it should. A good sizing should reduce your heap by more than 40% in a single go. If it doesn't that means you need more heap to save your long lived objects.
So in essence GC analysis is not a 'do it based of a set of predefined steps' things. Its more of a hti and trial analysis. It more of an experiment were you set the initial sizes and settings and then note or monitor the GC activity and record findings. Then after say 8-10 runs you compare notes and see what works for your app and what doesn't. Its really an interesting hard work to do.
I have three questions regarding garbage collection
I am trying to study the garbage collection in my application and I can notice that a full GC has occurred. By studying the GC logs I could find that old gen has not even used half the memory allocated to it. Then why would a full GC happen. Is there some other algorithm the JVM uses that releases the memory even when old gen is not completely utilized?
What can be called as a good GC trend. I mean if the full GC occurs at every 10- 15 mins can I call the application to be in a good state. I want to know how an ideal GC should be for an application. I know it depends considerably on the application, but there should be something to call ideal.
I have not set the NewSize or Newratio property. The default NewRatio in the machine seems to be 2. But I can see that my young gen is using only 1/4 th the heap size and the rest is used by tenured gen. How is this possible? All I have defined is the Xmx and permsize.
A major collection can happen for several reasons, in most cases you can see the cause by using jstat -gccause.
Few of the reasons are
-System.gc() if called from your app or any other code that you use and relies on this call.
-When the old space occupancy fractions has been reached
-When a PermGen collection takes place
-Depending on the collector you are using CMSIncrementalMode seems to be causing major collections before the limit of the old generation.
Most likely System.gc() is the cause of your unexpected major collections, try to use the flag -XX:+DisableExcplicitGC and see if you still get them.
--
There is no trend that can describe all usages. This should be based on your needs. Does the way your GC works now affect the performance of your app/service. Do you get long stop-the-world pauses that decrease your throughput ? What do you want to achieve? And the most important what is the garbage you are generating ? Try to analyze a heap dump and see if you can somehow reduce the numbers before you go and optimize the collector.
--
It depends on the flags you are using the version of the JVM your OS etc etc... In general GC ergonomics and more specifically the option -XX:+UseAdaptiveSizePolicy will be responsible of the sizings of your generations.
I have a Java client which consumes a large amount of data from a server. If the client does not keep up with the data stream at a fast enough rate, the server disconnects the socket connection. My client gets disconnected a few times per day. I ran jconsole to see the memory usage, and the heap space graph looks like a fairly well defined sawtooth pattern, oscillating between about 0.5GB and 1.8GB (2GB of heap space is allocated). But every time I get disconnected is during a full GC (but not on every full GC). I see the full GC takes a bit over 1 second on average. Depending on the time of day, full GC happens as often as every 5 minutes when busy, or up to 30 minutes can go by in between full GCs during the slow periods.
I suspect if I can reduce the full GC time, the client will be able to better keep up with the incoming data, but I do not have much experience with GC tuning. Does anyone have some insight on if this might be a good idea, and how to do it? Or is there an alternative idea which may work as well?
** UPDATE **
I used -XX:+UseConcMarkSweepGC and it improved, but I still got disconnected during the very busy moments. So I increased the heap allocation to 3GB to help weather through the busy moments and it seems to be chugging along pretty well now, but it's only been 1 day without a disconnection. Maybe if I get some time I will go through and try to reduce the amount of garbage created which I'm confident will help as well. Thanks for all the suggestions.
Full GC could take very long to complete, and is not that easy to tune.
One way to (easily) tune it is to increase the heap space - generally speaking, double the heap space can double the interval between two GCs, but will double the time consumed by a GC. If the program you are running has very clear usage patterns, maybe you can consider increase the heap space to make the interval so large that you can guarantee to have some idle time to try to make the system perform a GC. On the other hand, following this logic, if the heap is small a full garbage collection will finish in a instant, but that seems like inviting more troubles than helping.
Also, -XX:+UseConcMarkSweepGC might help since it will try to perform the GC operations concurrently (not stopping your program; see here).
Here's a very nice talk by Til Gene (CTO of Azul systems, maker of high performance JVM, and published several GC algos), about GC in JVM in general.
It is not easy to tune away the Full GC. A much better approach is to produce less garbage. Producing less garbage reduces pressure on the collection to pass objects into the tenured space where they are more expensive to collect.
I suggest you use a memory profiler to
reduce the amount of garbage produced. In many applications this can be reduce by a factor of 2 - 10x relatively easily.
reduce the size of the objects you are creating e.g. use primitive and smaller datatypes like double instead of BigDecimal.
recycle mutable object instead of discarding them.
retain less data on the client if you can.
By reducing the amount of garbage you create, objects are more likely to die in the eden, or survivor spaces meaning you have far less Full collections, which can be shorter as well.
Don't take it for granted you have to live with lots of collections, in extreme cases you can avoid it almost completely http://vanillajava.blogspot.ro/2011/06/how-to-avoid-garbage-collection.html
Take out calls to Runtime.getRuntime().gc() - When garbage collection is triggered manually it either does nothing or it does a full stop-the-world garbage collection. You want incremental GC to happen.
Have you tried using the server jvm from a jdk install? It changes a bunch of the default configuration settings (including garbage collection) and is easy to try - just add -server to your java command.
java -server
What is all the garbage that gets created? Can you generate less of it? Where possible, try to use the valueOf methods. By using less memory you'll save yourself time in gc AND in memory allocation.