I have an application which reads XML-responses from a server.
This is working nice, until I try to read ~200.000 XML-responses. When I reach that magic number, the handling time reduces with a factor 10.
When I let it run, at some point the JVM would say that the GC is taking 90% of CPU-time. So I first tried to optimize my code - using fields instead of local variables, using intern on my strings (Since I have a lot of copies) and so on.
This helped a bit, but it still went slow after approx 100k XML-files. I then tried using Visual VM to see what was going on, and what I saw was:
Up until 18:02, everything works fine. Then suddenly the garbage collector is going bananas, and stealing CPU-time, which then in turn stabilizes memory consumption. I would understand this, if we we're hitting maximum memory of the heap, but I've set max heap size at 8 gb.
There is nothing different happening at that point, it's basically a giant loop doing the same thing over and over.
What is happening and what can I do in this situation?
Your heap size is insufficient for your workflow. You may have memory leak, or it just specific of your application.
Normal pattern for parallel GC algorithm (which you have enabled)
Young GC
Young GC
...
Full GC
Though, once old space is full (~5.6 GiB for your setup), pattern would switch to
Full GC
Full GC
Full GC
...
Full GC is order of magnitude longer, so application would stay in GC pause (with high CPU consumption) almost all time. VisialVM incorrectly charts GC CPU usage, in reality blue spikes are as high as orange line on CPU chart.
If memory usage grows due to memory leak, you should address that.
If it is application design specific, you need increase old space by
either increasing total heap size
or reducing young space (-Xmn=SIZE option) to save more memory for old space
Related
the Heap size of my java application is continuously growing till it reaches the Max Heap sizeof 1G. Why is that so?
I start my application with those parameters:
java -Xmx1G -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC -XX:ShenandoahUncommitDelay=5000 -XX:ShenandoahGuaranteedGCInterval=10000 -Dlog4j.configurationFile=./log4j2.xml -jar applicaiton.jar
Edit:
When I restart my application you see there isn't a gap between "Heap size" and "Used Heap" but this gap is getting bigger and bigger, can I somehow limit that gap?
I am not particularly familiar with the way that the new Shenandoah GC normally behaves. However, there is nothing particularly alarming (to me) with that graph.
According to https://shipilev.net/talks/devoxx-Nov2017-shenandoah.pdf, the modus operandi (MO) of a this GC with regards to memory utilization is a bit different to some other collectors.
"We shall take all the memory when we need it, but we shall also give it back when we don’t".
If there is significant load on the allocator (i.e. lots of objects being allocated) Shenandoah will aggressively expand the heap. This is based on the observation that a low-pause GC is most efficient (and most likely to keep up!) if it has plenty of space to work in.
But the flipside is that if your system is idle, the GC will give memory back to the OS more freely than many other GCs.
This seems to fit the memory graph in your question.
The other thing to note is that the heap size (orange) is nowhere near your max-heap limit. If you get close to that limit, the GC will stop growing the heap.
Finally, note that you can apparently encourage Shenandoah to give back uncommitted memory faster by using a smaller value for the -XX:ShenandoahUncommitDelay=<millis> option. However, it is recommended to NOT make it too small because that is liable to slow down the allocator.
(Source: https://www.javacodegeeks.com/2017/11/minimize-java-memory-usage-right-garbage-collector.html)
Our java backend system is running with tomcat and jdk1.8, max jvm size is 24g, system latency becomes long, and cpu load is high, after we analyze GC logs, we found GC pause is long, attached are some snapshots in GCViewer.
How can we improve the GC performance?
Without knowledge of your application, here are some general thoughts:
Your application seems to have a high amount of long-lived objects (only 9gb of the tenured generation get collected, 15gb remain), which cause your problems: While allocation speed determines the frequency of GC pauses it is the amount of living objects (that cannot be freed by GC) which determines the length of the GC pauses.
One possible cause could be that your application acts as an in-memory database to some extent. That is, the objects are not needed all the time, but are not written to a database to retain fast in-memory access to them. In that case, you should store them off-heap (in memory not managed by the GC) or use an in-memory database, e.g. Redis. This way, they will be accessible with low latency but won't be considered by the GC because they won't be reclaimed anyway.
Another possible solution might be to tune your GC settings: You could switch to G1, increase the size of your young generation or increase the promotion threshold; or a combination of those. Increasing the young generation and reducing promotion might lead to much less growth of the tenured generation and thus to less frequent major collections. Switching to the G1 collector would give you the benefit that this collector is able to only consider some survivor regions and can leave regions with immortal objects alone.
Long time ago I have written guide for that topic. It is still very relevant.
Most important thing for CMS garbage collector in sizing. It seems like both your young and old space are undersized.
Old space should be size of your live set (~ heap size after full GC) multiplied by certain factor. Factor is to be found empirically, 1.5 is a good starting point.
Bigger young space is better as long as you are fine with young GC pauses. Young size is also very empiric per application.
-XX:InitiatingHeapOccupancyPercent=n and -XX:+UseCMSInitiatingOccupancyOnly are recommended to ensure concurrent cycle timely start. In some situations -XX:CMSTriggerInterval=p is also useful.
Hello I am having a case of 150GB heap memory program using In Memory Data grid. I have some crazy requirement from the operational department to use a single machine. Now we all know what happens in if the parallel garbage collector is used over 150GB probably it will be tens of minutes of garbage collection if the FULL GC is invoked.
My hope was that with Java 9 is coming Shenandoah low pause GC. Unfortunately from what I see it is not listed for delivery in Java 9. Does anyone knows anything about that ?
Never the less, I am wondering how G1 GC will perform for this amount of Heap memory.
And one last question. Since I have non interactive batch application that is supposed to complete in 2 hours lets say. The main goal here is to ensure that the Full GC never kicks in. If I ensure that there is plenty of memory lets say if the maximum heap that can be reached is 150 and I allocate it 250GB may I say with good confidence that the Full GC will never kick in or ? Usually full GC is triggered if the new generation + the old generation touches the maximum heap. Can it be triggered in a different way ?
There is a duplicate request made I will try to explain here why this question is not a duplicate. First we are talking about 150GB Heap which adds completely different dimension to the question. Second I dont use RMI as it is in the question mentioned, third I am asking question about G1 garbage collector in between the lines.Also once we go beyond the 32GB heap barrier we are entering the 64 bit address space you can not convince me that a question in regards of <32GB Heap is the same as a question with heap >32GB Not to mentioned that things have changed a bit since Java 7 for instance PermSpace does not exist.
The rule of thumb for a compacting GC is that it should be able to process 1 GB of live objects per core per second.
Example on an Haswell i7 (4 cores/8 threads) and 20GB heap with the parallel collector:
[24.757s][info][gc,heap ] GC(109) PSYoungGen: 129280K->0K(917504K)
[24.757s][info][gc,heap ] GC(109) ParOldGen: 19471666K->7812244K(19922944K)
[24.757s][info][gc ] GC(109) Pause Full (Ergonomics) 19141M->7629M(20352M) (23.791s, 24.757s) 966.174ms
[24.757s][info][gc,cpu ] GC(109) User=6.41s Sys=0.02s Real=0.97s
The live set after compacting is 7.6GB. It takes 6.4 seconds worth of cpu-time, due to parallelism this translates to <1s pause time.
In principle the parallel collector should be able to handle a 150GB heap with full GC times < ~2 minutes on a multi-core system, even when most of the heap consists of live objects.
Of course this is just a rule of thumb. Some things that can affect it negatively:
paging
thermal CPU throttling
workloads consisting of very large, reference-heavy objects
non-local memory traffic in NUMA configurations
other processes competing for CPU time
heavy use of weak/soft references
In some cases tuning may be necessary to achieve this throughput.
If the Parallel collector does not work despite all that then CMS and G1 can be viable alternatives but only if there is enough spare heap capacity and CPU cores available to the JVM. They need significant breathing room to do their concurrent work without risking a full GC.
It is correct I said no interactive, but still I have a strict license agreements. I need to be finished with the whole processing in an hour. So I can no afford 30 minutes stop the world event.
Basically, you don't really need low pause times in the sense that CMS, G1, Shenandoah or Zing aim for (they aim for <100ms or even <10ms even on large heaps).
All you need is that STW pauses are not so catastrophically bad that they eat a significant portion of your compute time.
This should be feasible with most of the available collectors, ignoring the serial one.
In practice there are some pathological edge cases where they may fall down, but to get to that point you need setup a system with your actual workload and do some test runs. If you experience some real problems, then you can ask a question with more details.
So, the jest of it is, a version of an application at my company is having some memory issues lately, and I'm not fully sure the best way to fix it that isn't just "Allocate more memory", so I wanted to get some guidance.
For the application, It looks like the eden heap is getting full pretty quickly when it has a concurrent users, so objects that won't be alive very long end up in the old heap. After running for a while, the old heap simply gets fulls, and never seems to automatically clean up, but manually running the garbage collection in VisualVM will clear it out (So I assume this means the old heap is full of dead objects)
Is there any setting suggested I could add so garbage collection gets run on the old heap once it gets to a certain threshold? And is there any pitfalls from changing the old/edin ratio from the stock 2:1 to 1:1? For the application, the majority of objects created are what I would consider short lived (From milliseconds to a few minutes)
It looks like the eden heap is getting full pretty quickly when it has a concurrent users, so objects that won't be alive very long end up in the old heap.
This is called "premature promotion"
After running for a while, the old heap simply gets fulls,
When it fills, the GC triggers a major or even a full collection.
never seems to automatically clean up
In which case, it is either used or it is not completely full. It might appear to be almost full, but the GC will be performed when it is actually full.
but manually running the garbage collection in VisualVM will clear it out
So the old gen wasn't almost but not actually full.
I could add so garbage collection gets run on the old heap once it gets to a certain threshold?
You can run System.gc() but this means more work for you application and slow it down. You don't want to be doing this.
If you use the CMS collector you can change the threshold at which it kicks in but unless you need low latency you might be better off leaving your settings as they are.
And is there any pitfalls from changing the old/edin ratio from the stock 2:1 to 1:1?
You reduce the old gen, you you may half the number of GCs you perform and double the amount of time an object can live and not end up in the old gen.
I work in the low latency space and usually set the young space to 24 GB and the old gen to 2 GB. I also use a lot of off heap data so I don't need much old gen. This is not an average use case, but it can work depending on your requirements.
If you are using < 32 GB, just adding a few more GB may be the simplest answer. Also you can use something like -Xmn4g -Xms6g to set the young space and maximum heap not worry about ratios.
For the application, the majority of objects created are what I would consider short lived (From milliseconds to a few minutes)
In that case, ideally you want your eden space large enough so you have a minor collection every few minutes. This way most of your objects will die in the eden space, and not be copied around.
Note: in extreme cases it is possible to have an application produce less than one GB per hour of garbage and run all day with a 24 GB Eden space without even a minor collection.
When I run a java program with the starting heap size of 3G (set by -Xms3072m VM argument), JVM doesn't start with that size. It start with 400m or so and then keeps on acquiring more memory as required.
This is a serious problem for me. I know JVM is going to need the said amount after some time. And when JVM increases is its memory as per the need, it slows down. During the time when JVM acquires more memory, considerable amount of time is spent in garbage collection. And I suppose memory acquisition is an expensive task.
How do I ensure that JVM actually respects the start heap size parameter?
Update: This application creates lots of objects, most of which die quickly. Some resulting objects are required to stay in memory (which get transferred out of young heap.) During this operation, all these objects need to be in memory. After the operation, I can see that all the objects in young heap are claimed successfully. So there are no memory leaks.
The same operation runs smoothly when the heap size reaches 3G. That clearly indicates the extra time required is spent in acquiring memory.
This Sun JDK 5.
If I am not mistaken, Java tries to get the reservation for the memory from the OS. So if you ask for 3 GB as Xms, Java will ask the OS, if this is available but not start with all the memory right away... it might even reserve it (not allocate it). But these are details.
Normally, the JVM runs up to the Xms size before it starts serious old generation garbage collection. Young generation GC runs all the time. Normally GC is only noticeable when old gen GC is running and the VM is in between Xms and Xmx or, in case you set it to the same value, hit roughly Xmx.
If you need a lot of memory for short lived objects, increase that memory area by setting the young area to... let's say 1 GB -XX:NewSize=1g because it is costly to move the "trash" from the young "buckets" into the old gen. Because in case it has not turned into real trash yet, the JVM checks for garbage, does not find any, copies it between the survivor spaces, and finally moves into the old gen. So try to suppress the check for the garbage in the young gen, when you know that you do not have any and postpone this somehow...
Give it a try!
I believe your problem is not coming from where you think.
It looks like what's costing you the most are the GC cycles, and not the allocation of heap size. If you are indeed creating and deleting lots of objects.
You should be focusing your effort on profiling, to find out exactly what is costing you so much, and work on refactoring that.
My hunch - object creation and deletion, and GC cycles.
In any case, -Xms should be setting minimum heap size (check this with your JVM if it is not Sun). Double-check to see exactly why you think it's not the case.
i have used sun's vm and started with minimum set to 14 gigs and it does start off with that.
maybe u should try setting both the xms and xmx values to the same amt, ie try this-
-Xms3072m -Xmx3072m
Why do you think the heap allocation is not right? Taking any operating system tool that shows only 400m does not mean it isn't allocated.
I don't get really what you are after. Is the 400m and above already a problem or is your program supposed to need that much? If you really have the need to deal with that much memory and it seems you need a lot of objects than you can do several things:
If the memory consumption doesn't match your gut feeling it is the right amount than you probably are leaking memory. That would explain why it "slows down" over time. Maybe you missed to remove objects from one structure so they don't get garbage collected and are slowing lookups and such down.
Your memory settings are maybe the trouble in itself. Garbage collection is not run per se. It is only called if there is some threshold reached. If you give it a big heap setting and your operating system has plenty of memory the garbage collection runs not often.
The characteristics you mentioned would be a scenario where a lot of objects are created and shortly after they would be deleted again. Otherwise the garbage collection wouldn't be a problem (some sort of generational gc). That means you have only "young" objects. Consider using an object pool if you are needing objects only a short period of time. That would eliminate the garbage collection at all.
If you know there are good times in your code for running gc you can consider running it manually to be able to see if it changes anything. This is what you would need
Runtime r = Runtime.getRuntime();
r.gc();
This is just for debugging purposes. The gc is doing a great job most of the time so there shouldn't be the need to invoke the gc on your own.