I've a AKKA-HTTP based service which is written in scala. This service works as a proxy for an API call. It creates a host connection pool for calling API using
https://doc.akka.io/docs/akka-http/current/client-side/host-level.html
The service is integrated with NewRelic and has the attached snapshots
I would like to understand the reasons for this kind of zig-zag patterns even when there is no traffic on the service and the connections in the host-pool gets terminated because of idle-timeout.
Moreover, I would also like to know Does the FULL GC will only occur after it reached a threshold say 7GB? or it can also occur at some other time when there is no traffic?
The service has XmX of 8GB. Moreover, there are also multiple dispatchers(fork-join-executor) which performs multiple tasks.
First, your graphs show a very healthy application. This "chainsaw" pattern is overall seen as a very good thing, without much to worry about.
When exactly a Full GC is going to happen is a bit hard to predict (I would use the word impossible, too). When your "live" objects have nowhere to move (because there is simply no space for that), a Full GC may be triggered. There are certain thresholds of when a concurrent phase (marking) is going to be initiated, but if that results in a Full GC or not is decided later.
Considering that G1 also re-sizes regions (makes them less/more) based on heuristics, and the fact that it can also shrink or grow your heap (up to -Xmx), the exact conditions when a Full GC might happen is not easy to predict (I guess some GC experts that know the exact internal details might be able to do that). Also, G1GC can do partial collections: when it collects young regions + some of the old regions (not all), still making it far better than a Full GC time-wise.
Unfortunately, your point about no traffic is correct. When there is very limited traffic, you might not get a Full GC, but immediately as traffic comes in, such a thing might happen. Old regions might slowly build up during your "limited traffic" and as soon as you have a spike - surprise. There are ways to trigger a Full GC on demand, and though I have heard of such applications that do this - I have not worked with one in practice.
In general with a GC that's not reference-counting, you'll see that zig-zag pattern because memory is only reclaimed when a GC runs.
G1 normally only collects areas of the heap where it expects to find a lot of garbage relative to live objects ("garbage collection" is a bit of a misnomer: it actually involves collecting the live objects and (in the case of a relocating garbage collector like G1) moving the live objects to a different area of the heap, which allows the area it collected in to then be declared ready for new allocations; therefore the fewer live objects it needs to handle, the less work it needs to do relative to the memory freed up).
At a high-level, G1 works by defining an Eden (a young generation) where newly created objects where newly created objects are allocated and it divides Eden into multiple regions with each thread being mapped to a region. When a region fills up, only that region is collected, with the survivors being moved into an older generation (this is simplifying). This continues until the survivor generation is full, at which point the survivor and eden generations are collected, with the surviving survivors being promoted to the old generation, and when the old generation fills up, you have a full GC.
So there isn't necessarily a fixed threshold where a full GC will get triggered, but in general the more heap gets used up, the more likely it becomes that a full GC will run. Beyond that, garbage collectors on the JVM tend to be more or less autonomous: most will ignore System.gc and/or other attempts to trigger a GC.
Conceivably with G1, if you allocated a multi-GiB array at startup, threw away the reference, and then after every period of idleness reallocated an array of the same size as the one you allocated at startup and then threw away the reference, you'd have a decent chance of triggering a full GC. This is because that array is big enough to bypass eden and go straight to the old generation where it will consume heap until the next full GC. Eventually there won't be enough contiguous free space in the old generation to allocate these arrays, and that will trigger a full GC. The only complications to this approach are that:
You'll eventually have to outsmart the JIT optimizer, which will see that you're allocating this array and throwing it away and decide that it doesn't actually have to allocate the array
If you have a long enough busy time that a full GC ran since the last allocate-and-throw-away, there's no guarantee that the allocation of the large array will succeed after a full GC, which will cause an OOM.
Related
At least in old GCs, it holds true. (I know there are new ones like ZGC and Shenandoah that aim to eliminate that)
As far as I know GC keeps tracks of living objects, so shouldn't the GC times be mostly affected by the number of objects (living/needs to be cleared)?
EDIT:
I meant grows in terms of capacity, meaning bigger heap but same utilization of it by the application
Didn't you answer your own question?
As far as I know GC keeps tracks of living objects, so shouldn't the GC times be mostly affected by the number of objects (living/needs to be cleared)?
The more the heap grows, the more live objects it has, the slower the GC (I'm sure there are exceptions to this rule, in particular for minor collections, but that's the rough idea). The number of objects to be cleared is irrelevant, what matters most is the total number of live objects. Now if your heap is growing because you're storing long-lived objects, it might be ok as long as you don't keep on adding more and more of them. Eventually, long-lived objects will move towards the survivor space and will only be subject to major collections and not minor ones. As long as the minor GC always achieve sufficient memory freeing from the young generation, major GC won't be triggered on all objects (which includes long-lived ones).
I have also observed a different behaviour with G1. We had a low-latency application (40ms p99), so we attempted to configure G1 to make very short pauses (can't remember how much, maybe 5ms or so). What happened is that G1 was more or less meeting the 5ms target, but it had to run extremely frequently because 5ms was not enough to cope with all dead objects we had in our heap. Therefore, it's not exactly true to say individual garbage collection runs are going to get slower with increased heap size, however the average time spent in garbage collection in a given period of time is most likely going to increase.
There are many different algorithms that can be used to implement garbage collection. Not all of them exhibit the behaviour you mention.
In the case of your question, you are referring to algorithms that use a form of mark-sweep. If we take the HostSpot JVM as an example, the old generation can be collected using the CMS collector. This uses a marking phase, where all objects that are accessible from application code are marked. Initially, a root set of directly accessible objects (object references on the stack, registers, etc.) is created. Each object in this set has the mark-bit set in its header to indicate it is still in use. All references from these objects are recursively followed and ulitmately every accessible object has the mark-bit set. How long this takes is proportional to the number of live objects, not the size of the heap.
The sweeping phase then has to sweep throught the entire heap, looking for objects with the mark-bit set and determining the gaps between them so that they can be added to free lists. These are used to allocate space for objects being promoted from the young generation. Since the whole heap must be swept, the time this takes is proportional to the size of the heap, regardless of how much live data is in the heap.
In the case of G1, the algorithm is similar but each generation of the heap is divided into regions so that space can be reclaimed in a more efficent way.
I am exploring a tree of stuff in Java, but I am memory limited. I handle this by looking at occupancy stats, and when I'm over 80% (say) stop allocating new bits of the tree to avoid the OutOfMemory. And just compute using what I've got so far using the 20% headroom.
Then every now and again I decide to move down the tree. This should free up 90% of the tree I've allocated because I forget the old root and move to one of it's children. And it does free that memory, but only if I call System.gc(), which is evil and stops my world. But if I don't call gc() my limiting code stops me adding to my tree.
What I want to do is call out to the G1 or CMS collectors that I could really do with the old generation getting a clean, and carrying on computing and allocate new bits of the tree as the collector does it's job.
Any ideas how that can be accomplished? Or, equally helpful, how I could avoid this artificial 80% limit, that is the root of my problem.
Setting -XX:+ExplicitGCInvokesConcurrent should trigger a concurrent cycle if you invoke System.gc()
Alternatively you could use -XX:CMSInitiatingOccupancyFraction=50 -XX:+UseCMSInitiatingOccupancyOnly for CMS which will initiate concurrent cycles once the heap reaches that threshold, since you're aiming for 80%+ occupancy that would basically continuously burn CPU time (possibly limited by GCTimeRatio, i'm not sure how those goals interact) to clean the old generation and thus free up memory swiftly.
The equivalent parameter for G1GC (InitiatingHeapOccupancyPercent) already defaults to 45%, so it should already be running mixed collection cycles if you have reached that point. You should inspect GC logs to verify.
Or, equally helpful, how I could avoid this artificial 80% limit, that is the root of my problem.
overprovision memory, RAM is often cheaper than developer hours and GCs need some breathing room to work efficiently anyway
consider using soft references to hold onto data that's useful but can be discarded by the GC if needed. They are not free though, so see if the extra costs are worth it.
So, the jest of it is, a version of an application at my company is having some memory issues lately, and I'm not fully sure the best way to fix it that isn't just "Allocate more memory", so I wanted to get some guidance.
For the application, It looks like the eden heap is getting full pretty quickly when it has a concurrent users, so objects that won't be alive very long end up in the old heap. After running for a while, the old heap simply gets fulls, and never seems to automatically clean up, but manually running the garbage collection in VisualVM will clear it out (So I assume this means the old heap is full of dead objects)
Is there any setting suggested I could add so garbage collection gets run on the old heap once it gets to a certain threshold? And is there any pitfalls from changing the old/edin ratio from the stock 2:1 to 1:1? For the application, the majority of objects created are what I would consider short lived (From milliseconds to a few minutes)
It looks like the eden heap is getting full pretty quickly when it has a concurrent users, so objects that won't be alive very long end up in the old heap.
This is called "premature promotion"
After running for a while, the old heap simply gets fulls,
When it fills, the GC triggers a major or even a full collection.
never seems to automatically clean up
In which case, it is either used or it is not completely full. It might appear to be almost full, but the GC will be performed when it is actually full.
but manually running the garbage collection in VisualVM will clear it out
So the old gen wasn't almost but not actually full.
I could add so garbage collection gets run on the old heap once it gets to a certain threshold?
You can run System.gc() but this means more work for you application and slow it down. You don't want to be doing this.
If you use the CMS collector you can change the threshold at which it kicks in but unless you need low latency you might be better off leaving your settings as they are.
And is there any pitfalls from changing the old/edin ratio from the stock 2:1 to 1:1?
You reduce the old gen, you you may half the number of GCs you perform and double the amount of time an object can live and not end up in the old gen.
I work in the low latency space and usually set the young space to 24 GB and the old gen to 2 GB. I also use a lot of off heap data so I don't need much old gen. This is not an average use case, but it can work depending on your requirements.
If you are using < 32 GB, just adding a few more GB may be the simplest answer. Also you can use something like -Xmn4g -Xms6g to set the young space and maximum heap not worry about ratios.
For the application, the majority of objects created are what I would consider short lived (From milliseconds to a few minutes)
In that case, ideally you want your eden space large enough so you have a minor collection every few minutes. This way most of your objects will die in the eden space, and not be copied around.
Note: in extreme cases it is possible to have an application produce less than one GB per hour of garbage and run all day with a 24 GB Eden space without even a minor collection.
I have problem with growing memory consuming on Tomcat.
Just after start nothing happens,but if some user login, after this memory usage start growing in Edem. PermGen does not grow, but anyway, it anormal.
My analyze shows that thread RMI TCP Connection produces lot of Object[] char[] and String[] objects. I can not understand what's wrong and where to dig. Who starts this thread, is is postgres connections and what is this?
This is normal, and is NOT a memory leak. Objects are created and destroyed constantly by the threads used to manage the application. You see the memory increasing because the JVM garbage collector is not eagerly reclaiming unused memory. This happens periodically (based on previous statistics) or when memory is running low. If it were a real memory leak, you would not see the Eden memory usage drop down to almost zero after a collection. A memory leak is shown as the lowest point (right after a GC) increasing over time.
You are observing that you are observing:
The JVM gathers statistical information about itself and sends it to you. This consumes memory and uses the RMI transfer facilities.
What is RMI TCP Accept, Attach Listener, and Signal Dispatcher in Visual VM?
I also don't see a problem with what that image shows. Eden is basically always growing slowly since there is always a bit work that consumes memory.
Once Eden gets collected (~200MB worth towards the end) you can see that most of the memory is completely free and very little (~8MB) ends in the survivor spaces since there are probably still references to these objects. But they don't seem to leave survivor since OldGen is not growing, also the Histogram at the bottom shows that typical survivor objects make it to level 2 and are gone then.
This all looks pretty normal to me.
I have three questions regarding garbage collection
I am trying to study the garbage collection in my application and I can notice that a full GC has occurred. By studying the GC logs I could find that old gen has not even used half the memory allocated to it. Then why would a full GC happen. Is there some other algorithm the JVM uses that releases the memory even when old gen is not completely utilized?
What can be called as a good GC trend. I mean if the full GC occurs at every 10- 15 mins can I call the application to be in a good state. I want to know how an ideal GC should be for an application. I know it depends considerably on the application, but there should be something to call ideal.
I have not set the NewSize or Newratio property. The default NewRatio in the machine seems to be 2. But I can see that my young gen is using only 1/4 th the heap size and the rest is used by tenured gen. How is this possible? All I have defined is the Xmx and permsize.
A major collection can happen for several reasons, in most cases you can see the cause by using jstat -gccause.
Few of the reasons are
-System.gc() if called from your app or any other code that you use and relies on this call.
-When the old space occupancy fractions has been reached
-When a PermGen collection takes place
-Depending on the collector you are using CMSIncrementalMode seems to be causing major collections before the limit of the old generation.
Most likely System.gc() is the cause of your unexpected major collections, try to use the flag -XX:+DisableExcplicitGC and see if you still get them.
--
There is no trend that can describe all usages. This should be based on your needs. Does the way your GC works now affect the performance of your app/service. Do you get long stop-the-world pauses that decrease your throughput ? What do you want to achieve? And the most important what is the garbage you are generating ? Try to analyze a heap dump and see if you can somehow reduce the numbers before you go and optimize the collector.
--
It depends on the flags you are using the version of the JVM your OS etc etc... In general GC ergonomics and more specifically the option -XX:+UseAdaptiveSizePolicy will be responsible of the sizings of your generations.