We Keep Coding

JVM garbage collection suddenly takes a lot of CPU

I have an application which reads XML-responses from a server.
This is working nice, until I try to read ~200.000 XML-responses. When I reach that magic number, the handling time reduces with a factor 10.
When I let it run, at some point the JVM would say that the GC is taking 90% of CPU-time. So I first tried to optimize my code - using fields instead of local variables, using intern on my strings (Since I have a lot of copies) and so on.
This helped a bit, but it still went slow after approx 100k XML-files. I then tried using Visual VM to see what was going on, and what I saw was:
Up until 18:02, everything works fine. Then suddenly the garbage collector is going bananas, and stealing CPU-time, which then in turn stabilizes memory consumption. I would understand this, if we we're hitting maximum memory of the heap, but I've set max heap size at 8 gb.
There is nothing different happening at that point, it's basically a giant loop doing the same thing over and over.
What is happening and what can I do in this situation?

Your heap size is insufficient for your workflow. You may have memory leak, or it just specific of your application.
Normal pattern for parallel GC algorithm (which you have enabled)
Young GC
Young GC
...
Full GC
Though, once old space is full (~5.6 GiB for your setup), pattern would switch to
Full GC
Full GC
Full GC
...
Full GC is order of magnitude longer, so application would stay in GC pause (with high CPU consumption) almost all time. VisialVM incorrectly charts GC CPU usage, in reality blue spikes are as high as orange line on CPU chart.
If memory usage grows due to memory leak, you should address that.
If it is application design specific, you need increase old space by
either increasing total heap size
or reducing young space (-Xmn=SIZE option) to save more memory for old space

Need advice on turning Java Garbage Collection

So, the jest of it is, a version of an application at my company is having some memory issues lately, and I'm not fully sure the best way to fix it that isn't just "Allocate more memory", so I wanted to get some guidance.
For the application, It looks like the eden heap is getting full pretty quickly when it has a concurrent users, so objects that won't be alive very long end up in the old heap. After running for a while, the old heap simply gets fulls, and never seems to automatically clean up, but manually running the garbage collection in VisualVM will clear it out (So I assume this means the old heap is full of dead objects)
Is there any setting suggested I could add so garbage collection gets run on the old heap once it gets to a certain threshold? And is there any pitfalls from changing the old/edin ratio from the stock 2:1 to 1:1? For the application, the majority of objects created are what I would consider short lived (From milliseconds to a few minutes)

It looks like the eden heap is getting full pretty quickly when it has a concurrent users, so objects that won't be alive very long end up in the old heap.
This is called "premature promotion"
After running for a while, the old heap simply gets fulls,
When it fills, the GC triggers a major or even a full collection.
never seems to automatically clean up
In which case, it is either used or it is not completely full. It might appear to be almost full, but the GC will be performed when it is actually full.
but manually running the garbage collection in VisualVM will clear it out
So the old gen wasn't almost but not actually full.
I could add so garbage collection gets run on the old heap once it gets to a certain threshold?
You can run System.gc() but this means more work for you application and slow it down. You don't want to be doing this.
If you use the CMS collector you can change the threshold at which it kicks in but unless you need low latency you might be better off leaving your settings as they are.
And is there any pitfalls from changing the old/edin ratio from the stock 2:1 to 1:1?
You reduce the old gen, you you may half the number of GCs you perform and double the amount of time an object can live and not end up in the old gen.
I work in the low latency space and usually set the young space to 24 GB and the old gen to 2 GB. I also use a lot of off heap data so I don't need much old gen. This is not an average use case, but it can work depending on your requirements.
If you are using < 32 GB, just adding a few more GB may be the simplest answer. Also you can use something like -Xmn4g -Xms6g to set the young space and maximum heap not worry about ratios.
For the application, the majority of objects created are what I would consider short lived (From milliseconds to a few minutes)
In that case, ideally you want your eden space large enough so you have a minor collection every few minutes. This way most of your objects will die in the eden space, and not be copied around.
Note: in extreme cases it is possible to have an application produce less than one GB per hour of garbage and run all day with a 24 GB Eden space without even a minor collection.

Threshold on old space in ParallelOldGC Garbage Collector

I am using ParallelOldGC as garbage collector in my application.The max heap size is set as 2 GB and working on Java hot spot 6, 27 update.
My application create so many long life objects due to which old space got full. So as per the parallel GC algorithm , Full GC trigger when the old space is almost full.As the heap size is 2 GB so to clean the old space the collector take more than 100 seconds which is not acceptable.
I am thinking to set threshold on old space say 30 so that when old space completed with 30% then Full GC will be called , as this solution will increase the FullGC count but it will decrease the application pause time.
I have observed that CMS contain such facility with XX:CMSInitiatingOccupancyFraction but due to some drawback of CMS cant switch to CMS.So is there any facility to set in ParallelOldGC
Thanks in advance.

As the heap size is 2 GB so to clean the old space the collector take more than 100 seconds which is not acceptable.
Since what you're actually trying to solve is long pause times you can set a pause time goal via -XX:MaxGCPauseMillis= and the collector will try to meet that.
It may fail for various reasons to meet it, e.g. because it's not allowed to burn enough CPU time on collecting (via GCTimeRatio) or simply because that you have so many old, live objects that the goal cannot be met (use G1 or CMS in those cases).

What is the recommended max level of heap memory usage? Recommended garbage collection time?

I recently found that one of my services was taking a large amount of time in garbage collection because the max heap size was too small. The service was there before I started, so I was unaware of the low heap size. I would like to set an alarm to warn me if it goes back up past a certain point, but I also do not want to give it more resources than it actually needs. What do you think is a reasonable level to be alarmed about for garbage collection and percent heap used?
I was thinking that an alarm on average heap usage would be ~85 percent, and 100 ms of gc / 5 min.
I understand this is based on requirements and hardware, but I am really looking for some benchmark or standard upon which to make my decision.

Alex Lockwood's answer says this:
The recommended "max level" of heap memory usage and GC time is as little as possible.
That is misleading. I'd actually advise the reverse. It is a bad idea to try to squeeze the heap size, because that will lead to your application running the GC more frequently, and spending less time (on average) doing useful work.
The problem is basically this. The classical (non-concurrent) GC gets run when the JVM runs out of space to allocate objects. It then traverses the non-garbage objects, copying them to a different "space". The processor time to run a GC cycle is most strongly dependent on the amount of non-garbage ... but the useful work it achieves (the amount of space it releases) is proportional to heapsize - nongarbage. So when you squeeze the heapsize, you decrease the amount of useful work that the GC does ... for the same processor time expenditure.
The original question says this:
I was thinking that an alarm on average heap usage would be ~85 percent, and 100 ms of gc / 5 min.
It is probably not useful to set a monitor / alarm on an absolute level of GC CPU usage. GC time will depend on server activity as well as GC efficiency. You don't want the GC alarm to go off every time your server gets busy.
An average heap usage of 85% is reasonable level to be alarming at, though once again setting an alarm at a fixed level might give too many false alarms.
An alternative to this is to use the JVM options to set a "percentage time spent in GC" threshold, and combine that with the "kill JVM on OutOfMemoryException" option, and put a auto-restart loop in the server's launch scripts. Then monitor for restarts.

How to ensure JVM starts with value of Xms

When I run a java program with the starting heap size of 3G (set by -Xms3072m VM argument), JVM doesn't start with that size. It start with 400m or so and then keeps on acquiring more memory as required.
This is a serious problem for me. I know JVM is going to need the said amount after some time. And when JVM increases is its memory as per the need, it slows down. During the time when JVM acquires more memory, considerable amount of time is spent in garbage collection. And I suppose memory acquisition is an expensive task.
How do I ensure that JVM actually respects the start heap size parameter?
Update: This application creates lots of objects, most of which die quickly. Some resulting objects are required to stay in memory (which get transferred out of young heap.) During this operation, all these objects need to be in memory. After the operation, I can see that all the objects in young heap are claimed successfully. So there are no memory leaks.
The same operation runs smoothly when the heap size reaches 3G. That clearly indicates the extra time required is spent in acquiring memory.
This Sun JDK 5.

If I am not mistaken, Java tries to get the reservation for the memory from the OS. So if you ask for 3 GB as Xms, Java will ask the OS, if this is available but not start with all the memory right away... it might even reserve it (not allocate it). But these are details.
Normally, the JVM runs up to the Xms size before it starts serious old generation garbage collection. Young generation GC runs all the time. Normally GC is only noticeable when old gen GC is running and the VM is in between Xms and Xmx or, in case you set it to the same value, hit roughly Xmx.
If you need a lot of memory for short lived objects, increase that memory area by setting the young area to... let's say 1 GB -XX:NewSize=1g because it is costly to move the "trash" from the young "buckets" into the old gen. Because in case it has not turned into real trash yet, the JVM checks for garbage, does not find any, copies it between the survivor spaces, and finally moves into the old gen. So try to suppress the check for the garbage in the young gen, when you know that you do not have any and postpone this somehow...
Give it a try!

I believe your problem is not coming from where you think.
It looks like what's costing you the most are the GC cycles, and not the allocation of heap size. If you are indeed creating and deleting lots of objects.
You should be focusing your effort on profiling, to find out exactly what is costing you so much, and work on refactoring that.
My hunch - object creation and deletion, and GC cycles.
In any case, -Xms should be setting minimum heap size (check this with your JVM if it is not Sun). Double-check to see exactly why you think it's not the case.

i have used sun's vm and started with minimum set to 14 gigs and it does start off with that.
maybe u should try setting both the xms and xmx values to the same amt, ie try this-
-Xms3072m -Xmx3072m

Why do you think the heap allocation is not right? Taking any operating system tool that shows only 400m does not mean it isn't allocated.
I don't get really what you are after. Is the 400m and above already a problem or is your program supposed to need that much? If you really have the need to deal with that much memory and it seems you need a lot of objects than you can do several things:
If the memory consumption doesn't match your gut feeling it is the right amount than you probably are leaking memory. That would explain why it "slows down" over time. Maybe you missed to remove objects from one structure so they don't get garbage collected and are slowing lookups and such down.
Your memory settings are maybe the trouble in itself. Garbage collection is not run per se. It is only called if there is some threshold reached. If you give it a big heap setting and your operating system has plenty of memory the garbage collection runs not often.
The characteristics you mentioned would be a scenario where a lot of objects are created and shortly after they would be deleted again. Otherwise the garbage collection wouldn't be a problem (some sort of generational gc). That means you have only "young" objects. Consider using an object pool if you are needing objects only a short period of time. That would eliminate the garbage collection at all.
If you know there are good times in your code for running gc you can consider running it manually to be able to see if it changes anything. This is what you would need
Runtime r = Runtime.getRuntime();
r.gc();
This is just for debugging purposes. The gc is doing a great job most of the time so there shouldn't be the need to invoke the gc on your own.