I have install glowroot (java application monitoring) to my JVM. When my application idles, I get this kind graph formation of memory heap usage. The pattern seems almost uniform. Could someone please explain and point me to whatever blog post on why does the graph looks like that? I am curious.
The large-scale sawtooth pattern represents the memory utilization between GC cycles. The application is allocating objects steadily (the upsloping line) until the heap gets full enough for the VM to decide to run the GC (the point). Then the GC reclaims a large amount of garbage (the steep drop) and the process starts again.
The short spikes up and down are harder to understand. It is possible that the upward spikes represent anomalous "large" allocations (of short life-time objects) that are triggering a young generation cycle. The downward spikes might represent cached objects being freed in response to "memory pressure".
If you want a better understanding of the spikes, you need to look at the GC log messages, and try to correlate them with the graphs.
It looks like that because (at the least) you're observing it. If your application did absolutely nothing, and there were no threads doing any allocations, you'd get a horizontal line for the heap.
However since you're observing it, there are things being done in the JVM to get that data back to you. That's why you get the ubiquitous sawtooth pattern seen in many, many profiling questions. A few examples below.
java memory leak, visualvm showing wrong data
visualvm monitors memory usage
This is the java garbage collection at work. Each drop is the garbage collector freeing memory by removing unused references. The reason why it grows again is simply because your application keeps creating new references.
You can also have a look at this post for a more in-depth explaination:
Why a sawtooth shaped graph?
Related
I've a AKKA-HTTP based service which is written in scala. This service works as a proxy for an API call. It creates a host connection pool for calling API using
https://doc.akka.io/docs/akka-http/current/client-side/host-level.html
The service is integrated with NewRelic and has the attached snapshots
I would like to understand the reasons for this kind of zig-zag patterns even when there is no traffic on the service and the connections in the host-pool gets terminated because of idle-timeout.
Moreover, I would also like to know Does the FULL GC will only occur after it reached a threshold say 7GB? or it can also occur at some other time when there is no traffic?
The service has XmX of 8GB. Moreover, there are also multiple dispatchers(fork-join-executor) which performs multiple tasks.
First, your graphs show a very healthy application. This "chainsaw" pattern is overall seen as a very good thing, without much to worry about.
When exactly a Full GC is going to happen is a bit hard to predict (I would use the word impossible, too). When your "live" objects have nowhere to move (because there is simply no space for that), a Full GC may be triggered. There are certain thresholds of when a concurrent phase (marking) is going to be initiated, but if that results in a Full GC or not is decided later.
Considering that G1 also re-sizes regions (makes them less/more) based on heuristics, and the fact that it can also shrink or grow your heap (up to -Xmx), the exact conditions when a Full GC might happen is not easy to predict (I guess some GC experts that know the exact internal details might be able to do that). Also, G1GC can do partial collections: when it collects young regions + some of the old regions (not all), still making it far better than a Full GC time-wise.
Unfortunately, your point about no traffic is correct. When there is very limited traffic, you might not get a Full GC, but immediately as traffic comes in, such a thing might happen. Old regions might slowly build up during your "limited traffic" and as soon as you have a spike - surprise. There are ways to trigger a Full GC on demand, and though I have heard of such applications that do this - I have not worked with one in practice.
In general with a GC that's not reference-counting, you'll see that zig-zag pattern because memory is only reclaimed when a GC runs.
G1 normally only collects areas of the heap where it expects to find a lot of garbage relative to live objects ("garbage collection" is a bit of a misnomer: it actually involves collecting the live objects and (in the case of a relocating garbage collector like G1) moving the live objects to a different area of the heap, which allows the area it collected in to then be declared ready for new allocations; therefore the fewer live objects it needs to handle, the less work it needs to do relative to the memory freed up).
At a high-level, G1 works by defining an Eden (a young generation) where newly created objects where newly created objects are allocated and it divides Eden into multiple regions with each thread being mapped to a region. When a region fills up, only that region is collected, with the survivors being moved into an older generation (this is simplifying). This continues until the survivor generation is full, at which point the survivor and eden generations are collected, with the surviving survivors being promoted to the old generation, and when the old generation fills up, you have a full GC.
So there isn't necessarily a fixed threshold where a full GC will get triggered, but in general the more heap gets used up, the more likely it becomes that a full GC will run. Beyond that, garbage collectors on the JVM tend to be more or less autonomous: most will ignore System.gc and/or other attempts to trigger a GC.
Conceivably with G1, if you allocated a multi-GiB array at startup, threw away the reference, and then after every period of idleness reallocated an array of the same size as the one you allocated at startup and then threw away the reference, you'd have a decent chance of triggering a full GC. This is because that array is big enough to bypass eden and go straight to the old generation where it will consume heap until the next full GC. Eventually there won't be enough contiguous free space in the old generation to allocate these arrays, and that will trigger a full GC. The only complications to this approach are that:
You'll eventually have to outsmart the JIT optimizer, which will see that you're allocating this array and throwing it away and decide that it doesn't actually have to allocate the array
If you have a long enough busy time that a full GC ran since the last allocate-and-throw-away, there's no guarantee that the allocation of the large array will succeed after a full GC, which will cause an OOM.
When viewing my remote application in JVisualVM over JMX, I see a saw-tooth of memory usage while idle:
Taking a heap dump and analysing it with JVisualVM, I see a large chunk of memory is in a few big int[] arrays which have no references and by comparing heap dumps I can see that it seems to be these that are taking the memory and being reclaimed by a GC periodically.
I am curious to track these down since it piqued my interest that my own code never knowingly allocates any int[] arrays.
I do use a lot of libs like netty so the culprit could be elsewhere. I do have other servers with much the same mix of frameworks but don't see this sawtooth there.
How can I discover who is allocating them?
Take a heapdump and find out what objects are holding them. Once you know what objects are holding the arrays you should have an easy time idea figuring out what is allocating them.
It doesn't answer your question, but my question is:
Why do you care?
You've told the jvm garbage collector (GC) it can use up to 1GB of memory. Java is using less than 250M.
The GC tries to be smart about when it garbage collects and also how hard it works at garbage collection. In your graph, there is no demand for memory. The jvm isn't anywhere near that 1GB limit you set. I see no reason the GC should try very hard at all. Not sure why you would care either.
Its a good thing for the garbage collector to be lazy. The less the GC works, the more resources there are available for your application.
Have you tried triggering GC via the JVisualVM "Perform GC" button? That button should trigger a "stop the world" garbage collection operation. Try it when the graph is in the middle of one of those saw tooth ramp ups - I predict that the usage will drop to the base of the saw tooth or below. If it does, that proves that the memory saw tooth is just garbage accumulation and GC is doing the right thing.
Here is an screenshot of memory usage for a java swing application I use:
Notice the sawtooth pattern.
You said you are worried about int[]. When I start the memory profiler and have it profile everything I can see the allocations of int[]
Basically all allocations come from an ObjectOutputStream$HandleTable.growEntries method. It looks like the thread the allocations were made on was spun up to handle a network message.
I suspect its caused by jmx itself. Possibly by rmi (do you use rmi?). Or the debugger (do you have a debugger connected?).
I just thought I'd add to this question that the sawtooth pattern is very much normal and has nothing necessarily to do with your int[] arrays. It happens because new allocations happen in the Eden-gen, and an ephemeral collection only is triggered once it has filled up, leaving the old-gen be. So as long as your program does any allocations at all, the Eden gen will fill up and then empty repeatedly. Especially, then, when you have a regular amount of allocations per unit of time, you'll see a very regular sawtooth pattern.
There are tons of articles on the web detailing how Hotspot's GC works, so there's no need for me to expand on that here. If you don't know at all how ephemeral collection works, you may want to check out Wikipedia's article on the subject (see the "Generational GC" section; "generational" and "ephemeral" are synonymous in this context).
As for the int[] arrays, however, they are a bit mysterious. I'm seeing those as well, and there's another question here on SO on the subject of them without any real answer. It's not actually normal for objects with no references to show up in a heap dump, because a heap dump normally only contains live objects (because Hotspot always performs a stop-the-world collection before actually dumping the heap). My personal guess is that they are allocated as part of some kind of internal JVM data-structure (and therefore only have references from the C++ part of Hotspot rather than from the Java heap), but that's really just a pure guess.
I have a Java client which consumes a large amount of data from a server. If the client does not keep up with the data stream at a fast enough rate, the server disconnects the socket connection. My client gets disconnected a few times per day. I ran jconsole to see the memory usage, and the heap space graph looks like a fairly well defined sawtooth pattern, oscillating between about 0.5GB and 1.8GB (2GB of heap space is allocated). But every time I get disconnected is during a full GC (but not on every full GC). I see the full GC takes a bit over 1 second on average. Depending on the time of day, full GC happens as often as every 5 minutes when busy, or up to 30 minutes can go by in between full GCs during the slow periods.
I suspect if I can reduce the full GC time, the client will be able to better keep up with the incoming data, but I do not have much experience with GC tuning. Does anyone have some insight on if this might be a good idea, and how to do it? Or is there an alternative idea which may work as well?
** UPDATE **
I used -XX:+UseConcMarkSweepGC and it improved, but I still got disconnected during the very busy moments. So I increased the heap allocation to 3GB to help weather through the busy moments and it seems to be chugging along pretty well now, but it's only been 1 day without a disconnection. Maybe if I get some time I will go through and try to reduce the amount of garbage created which I'm confident will help as well. Thanks for all the suggestions.
Full GC could take very long to complete, and is not that easy to tune.
One way to (easily) tune it is to increase the heap space - generally speaking, double the heap space can double the interval between two GCs, but will double the time consumed by a GC. If the program you are running has very clear usage patterns, maybe you can consider increase the heap space to make the interval so large that you can guarantee to have some idle time to try to make the system perform a GC. On the other hand, following this logic, if the heap is small a full garbage collection will finish in a instant, but that seems like inviting more troubles than helping.
Also, -XX:+UseConcMarkSweepGC might help since it will try to perform the GC operations concurrently (not stopping your program; see here).
Here's a very nice talk by Til Gene (CTO of Azul systems, maker of high performance JVM, and published several GC algos), about GC in JVM in general.
It is not easy to tune away the Full GC. A much better approach is to produce less garbage. Producing less garbage reduces pressure on the collection to pass objects into the tenured space where they are more expensive to collect.
I suggest you use a memory profiler to
reduce the amount of garbage produced. In many applications this can be reduce by a factor of 2 - 10x relatively easily.
reduce the size of the objects you are creating e.g. use primitive and smaller datatypes like double instead of BigDecimal.
recycle mutable object instead of discarding them.
retain less data on the client if you can.
By reducing the amount of garbage you create, objects are more likely to die in the eden, or survivor spaces meaning you have far less Full collections, which can be shorter as well.
Don't take it for granted you have to live with lots of collections, in extreme cases you can avoid it almost completely http://vanillajava.blogspot.ro/2011/06/how-to-avoid-garbage-collection.html
Take out calls to Runtime.getRuntime().gc() - When garbage collection is triggered manually it either does nothing or it does a full stop-the-world garbage collection. You want incremental GC to happen.
Have you tried using the server jvm from a jdk install? It changes a bunch of the default configuration settings (including garbage collection) and is easy to try - just add -server to your java command.
java -server
What is all the garbage that gets created? Can you generate less of it? Where possible, try to use the valueOf methods. By using less memory you'll save yourself time in gc AND in memory allocation.
I am currently running an application which requires a maximum heap size of 16GB.
Currently I use the following flags to handle garbage collection.
-XX\:+UseParNewGC, -XX\:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=50, -XX\:+DisableExplicitGC, -XX\:+PrintGCDateStamps, -XX\:+PrintGCDetails, -Xloggc\:/home/user/logs/gc.log
However, I have noticed that during some garbage collections, the application locks up for a few seconds and then carries on - This is completely unacceptable as it's a game server.
An exert from my garbage collection logs can be found here.
Any advice on what I should change in order to reduce these long pauses would be greatly appreciated.
Any advice on what I should change in order to reduce these long pauses would be greatly appreciated.
The chances are that the CMS GC cannot keep up with the amount of garbage your system is generating. But the work that the GC has to perform is actually more closely related to the amount of NON-garbage that your system is retaining.
So ...
Try to reduce the actual memory usage of your application; e.g. by not caching so much stuff, or reducing the size of your "world".
Try to reduce the rate at which your application generates garbage.
Upgrade to a machine with more cores so that there are more cores available to run the parallel GC threads when necessary.
To Mysticial:
Yes in hindsight, it might have been better to implement the server in C++. However, we don't know anything about "the game". If it involves a complicated world model with complicated heterogeneous data structures, then implementing it in C++ could mean that that you replace the "GC pause" problem with the problem that the server crashes all the time due to problems with the way it manages its data structures.
Looking at your logs, I don't see any long pauses. But young GC is very frequent. Promotion rate is very low though (most garbage cleared by young GC as it should). At same time your old space utilization is low.
BTW are we talking about minecraft server?
To reduce frequency of young GC you should increase its size. I would suggest start with -XX:NewSize=8G -XX:MaxNewSize=8G
For such large young space, you should also reduce survivor space size -XX:SurvivorRatio=512
GC tuning is a path of trial and errors, so you may need some more iterations and tweaking.
You can find couple of useful articles at mu blog
HotSpot JVM GC options cheatsheet
Understanding young GC pauses in HotSpot JVM
I'm not an expert on Java garbage collection, but it looks like you're doing the right thing by using the concurrent collector (the UseConcMarkSweepGC flag), assuming the server has multiple processors. Follow the suggestions for troubleshooting at http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#cms. If you already have, let us know what happened when you tried them.
Which version of java are you using?http://docs.oracle.com/javase/7/docs/technotes/guides/vm/G1.html
For better try to minimize the use of instance variables in a class.It would be better to perform on local variables than instance varibles .It helps in gaining the performance and safe from synchronization problem.In the end of operation before exit of program always reset the used variables if you are using instance variables and set again when it is required. It helps more in enhancing performance.Besides in the version of java a good garbage collection policy is implemented.It would be better to move to new version if that is fleasible.
Also you can monitor the garbage collector pause time via VisualVm and you can get more idea when it is performing more garbage collection.
This question is a follow up to my previous Java GC question: Java Garbage Collector clarification
This question is also referring to the same article.
I'm a little confused on why the stop and copy method for defragmenting object heap allocation is so commonly used. Yes it defragments the heap however it seems like there is tons of overhead because basically you cut the total amount of heap size in half. Also you need to copy ALL the live objects when one half has run out of space.
Other than defragmentation is there any other fundamental reason why 'stop and copy' is better than say 'mark and sweep'?
Actually, fragmentation is fundamental, and the ability of some GC to defeat it is a considerable asset.
The stop-and-copy algorithm used to be popular in GC implementations because:
it is simple to implement;
it automatically defragments memory;
its running time is proportional to the amount of live objects, which makes it asymptotically very efficient.
More modern GC, including those used in Java, use much more complex strategies because they want to make short pauses (rather than making total GC time low, they prefer never to stop the application for a long time, because pauses are bad for interactivity), to interact more cleanly with caches and virtual memory, and to benefit from systems with multiple CPU.
The Jones and Lins book is a must-read for whoever wants to understand garbage collection.
A great tutorial on the garbage collector is Tuning Garbage Collection (unfortunately the new oracle website has messed its formatting up quiet a lot).
Your question is handled in chapter V. This basically explains which types of strategies you can use in the Java garbage collector and which are default. Most desktop applications will be interested in a stop that is as small as possible, because this is what the user might notice.
Note that your question is not about defragmentation. Both will eventually compress the memory space.