We Keep Coding

Why could GC time increase in Kafka brokers with constant number of records, producers and consumers?

I use Kafka 2.1.0.
We have a Kafka cluster with 5 brokers (r5.xlarge machines). We often observe that the GC timings increase too much without any change in the rate of incoming messages severely impacting the performance of the cluster. Now, I don't understand what could be causing much sudden increase in GC time.
I have tried a few things with little improvement but I don't really understand the reason behind them.
export KAFKA_HEAP_OPTS="-Xmx10G -Xms1G"
export KAFKA_JVM_PERFORMANCE_OPTS="-XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80"
I would like to understand the most important parameters when tuning GC in a Kafka broker.
Seeing the configuration above, where am I going wrong? What can be done to rectify this?
All the producers and consumers are working fine, and the rate of incoming messages remains fairly constant. Till now, we have not been able to figure out any pattern behind the sudden increase in GC times, it seems random.
UPDATE
After some further analysis, It turns out there was indeed some increase in the amount of data per sec. One of the topics had increased message input from around 10 KBps to 200 KBps. But I believed that Kafka could easily handle this much of data.
Is there something I am missing??
Grafana Snapshot

I would start by looking to see if the problem is something else than a GC tuning issue. Here are a couple of possibilities:
A hard memory leak will cause GC times to increase. The work done by a GC is dominated by tracing and copying of reachable objects. If you have a leak, then more and more objects will be (incorrectly) reachable.
A cache that that is keeping too many objects reachable will also increase GC times.
Excessive use of Reference types, finalizers, etc may increase GC times.
I would enable GC logging, and look for patterns in memory and space utilization reported by the GC. If you suspect a memory leak because memory utilization is trending higher in the long term, go to the next step and use a memory profile to track down the leak.
Either way, it is important to understand what is causing the problem before trying to fix it.
After some further analysis, it turns out there was indeed some increase in the amount of data per sec. One of the topics had increased message input from around 10 KBps to 200 KBps. But I believed that Kafka could easily handle this much of data.
It most likely can. However, a 20x increase in throughput will inevitably lead to more objects being created and discarded ... and the GC will need to run more often to deal with this.
How come just 200 Kbps of data divided among 5 brokers was able to break GC.
What makes you think that you have "broken" the GC? 15% time in GC doesn't mean it is broken.
Now, I can imagine that the GC may have difficulty meeting your 20ms max pause time goal, and may be triggering occasional full GCs as a result. Your pause time goal is "ambitious", especially if the heap may grow to 10GB. I would suggest reducing the heap size, increasing the pause time goal, and/or increasing the number of physical cores available to the JVM(s).
By breaking I mean an increased delay in committing offsets and other producer and consumer offsets.
So ... you are just concerned that a 20 x increase in load has resulted in the GC using up to 15% of available CPU. Well that's NOT broken. That is (IMO) expected. The garbage collector is not magic. It needs to use CPU time to do its work. The more work it has to do, the more CPU it needs to use to do it. If your application's workload involves a lot of object allocation, then the GC has to deal with that.
In addition to the tuning ideas above, I suspect that you should set the G1HeapRegionSize size a lot smaller. According to "Garbage First Garbage Collector Tuning" by Monica Beckwith, the default is to have 2048 regions based on the minimum heap size. But your setting will give 1G / 16M == 64 initial regions.
Finally, if your overall goal is to reduce the CPU utilization of the GC, then you should be using the Throughput GC, not G1GC. This will minimize GC overheads. The downside is that GC pause minimization is no longer a goal, so occasional lengthy pauses are to be expected.
And if you plan to stay with G1GC, it is advisable to use the latest version of Java; i.e. Java 11. (See "G1 Garbage Collector is mature in Java 9, finally")

Kafka 2.1 uses G1GC by default, so I guess you can omit that argument. I'm assuming you're not using JDK 11. Compared to previous versions, JDK 11 brings significant improvement to G1GC. Instead of running a single-threaded full GC cycle, it can now achieve parallel processing. Even though that shouldn't improve the best case scenarios by a big margin, but worst case scenarios should see significant improvement. If possible, please share your results after migrating to JDK 11.
Note: I doubt that's the root cause, but let's see.

Need advice on turning Java Garbage Collection

So, the jest of it is, a version of an application at my company is having some memory issues lately, and I'm not fully sure the best way to fix it that isn't just "Allocate more memory", so I wanted to get some guidance.
For the application, It looks like the eden heap is getting full pretty quickly when it has a concurrent users, so objects that won't be alive very long end up in the old heap. After running for a while, the old heap simply gets fulls, and never seems to automatically clean up, but manually running the garbage collection in VisualVM will clear it out (So I assume this means the old heap is full of dead objects)
Is there any setting suggested I could add so garbage collection gets run on the old heap once it gets to a certain threshold? And is there any pitfalls from changing the old/edin ratio from the stock 2:1 to 1:1? For the application, the majority of objects created are what I would consider short lived (From milliseconds to a few minutes)

It looks like the eden heap is getting full pretty quickly when it has a concurrent users, so objects that won't be alive very long end up in the old heap.
This is called "premature promotion"
After running for a while, the old heap simply gets fulls,
When it fills, the GC triggers a major or even a full collection.
never seems to automatically clean up
In which case, it is either used or it is not completely full. It might appear to be almost full, but the GC will be performed when it is actually full.
but manually running the garbage collection in VisualVM will clear it out
So the old gen wasn't almost but not actually full.
I could add so garbage collection gets run on the old heap once it gets to a certain threshold?
You can run System.gc() but this means more work for you application and slow it down. You don't want to be doing this.
If you use the CMS collector you can change the threshold at which it kicks in but unless you need low latency you might be better off leaving your settings as they are.
And is there any pitfalls from changing the old/edin ratio from the stock 2:1 to 1:1?
You reduce the old gen, you you may half the number of GCs you perform and double the amount of time an object can live and not end up in the old gen.
I work in the low latency space and usually set the young space to 24 GB and the old gen to 2 GB. I also use a lot of off heap data so I don't need much old gen. This is not an average use case, but it can work depending on your requirements.
If you are using < 32 GB, just adding a few more GB may be the simplest answer. Also you can use something like -Xmn4g -Xms6g to set the young space and maximum heap not worry about ratios.
For the application, the majority of objects created are what I would consider short lived (From milliseconds to a few minutes)
In that case, ideally you want your eden space large enough so you have a minor collection every few minutes. This way most of your objects will die in the eden space, and not be copied around.
Note: in extreme cases it is possible to have an application produce less than one GB per hour of garbage and run all day with a 24 GB Eden space without even a minor collection.

java full gc taking too long

I have a Java client which consumes a large amount of data from a server. If the client does not keep up with the data stream at a fast enough rate, the server disconnects the socket connection. My client gets disconnected a few times per day. I ran jconsole to see the memory usage, and the heap space graph looks like a fairly well defined sawtooth pattern, oscillating between about 0.5GB and 1.8GB (2GB of heap space is allocated). But every time I get disconnected is during a full GC (but not on every full GC). I see the full GC takes a bit over 1 second on average. Depending on the time of day, full GC happens as often as every 5 minutes when busy, or up to 30 minutes can go by in between full GCs during the slow periods.
I suspect if I can reduce the full GC time, the client will be able to better keep up with the incoming data, but I do not have much experience with GC tuning. Does anyone have some insight on if this might be a good idea, and how to do it? Or is there an alternative idea which may work as well?
** UPDATE **
I used -XX:+UseConcMarkSweepGC and it improved, but I still got disconnected during the very busy moments. So I increased the heap allocation to 3GB to help weather through the busy moments and it seems to be chugging along pretty well now, but it's only been 1 day without a disconnection. Maybe if I get some time I will go through and try to reduce the amount of garbage created which I'm confident will help as well. Thanks for all the suggestions.

Full GC could take very long to complete, and is not that easy to tune.
One way to (easily) tune it is to increase the heap space - generally speaking, double the heap space can double the interval between two GCs, but will double the time consumed by a GC. If the program you are running has very clear usage patterns, maybe you can consider increase the heap space to make the interval so large that you can guarantee to have some idle time to try to make the system perform a GC. On the other hand, following this logic, if the heap is small a full garbage collection will finish in a instant, but that seems like inviting more troubles than helping.
Also, -XX:+UseConcMarkSweepGC might help since it will try to perform the GC operations concurrently (not stopping your program; see here).
Here's a very nice talk by Til Gene (CTO of Azul systems, maker of high performance JVM, and published several GC algos), about GC in JVM in general.

It is not easy to tune away the Full GC. A much better approach is to produce less garbage. Producing less garbage reduces pressure on the collection to pass objects into the tenured space where they are more expensive to collect.
I suggest you use a memory profiler to
reduce the amount of garbage produced. In many applications this can be reduce by a factor of 2 - 10x relatively easily.
reduce the size of the objects you are creating e.g. use primitive and smaller datatypes like double instead of BigDecimal.
recycle mutable object instead of discarding them.
retain less data on the client if you can.
By reducing the amount of garbage you create, objects are more likely to die in the eden, or survivor spaces meaning you have far less Full collections, which can be shorter as well.
Don't take it for granted you have to live with lots of collections, in extreme cases you can avoid it almost completely http://vanillajava.blogspot.ro/2011/06/how-to-avoid-garbage-collection.html

Take out calls to Runtime.getRuntime().gc() - When garbage collection is triggered manually it either does nothing or it does a full stop-the-world garbage collection. You want incremental GC to happen.
Have you tried using the server jvm from a jdk install? It changes a bunch of the default configuration settings (including garbage collection) and is easy to try - just add -server to your java command.
java -server
What is all the garbage that gets created? Can you generate less of it? Where possible, try to use the valueOf methods. By using less memory you'll save yourself time in gc AND in memory allocation.

Ways to reduce memory churn

Background
I have a Spring batch program that reads a file (example file I am working with is ~ 4 GB in size), does a small amount of processing on the file, and then writes it off to an Oracle database.
My program uses 1 thread to read the file, and 12 worker threads to do the processing and database pushing.
I am churning lots and lots and lots of young gen memory, which is causing my program to go slower than I think it should.
Setup
JDK 1.6.18
Spring batch 2.1.x
4 Core Machine w 16 GB ram
-Xmx12G
-Xms12G
-NewRatio=1
-XX:+UseParallelGC
-XX:+UseParallelOldGC
Problem
With these JVM params, I get somewhere around 5.x GB of memory for Tenured Generation, and around 5.X GB of memory for Young Generation.
In the course of processing this one file, my Tenured Generation is fine. It grows to a max of maybe 3 GB, and I never need to do a single full GC.
However, the Young Generation hits it's max many times. It goes up to 5 GB range, and then a parallel minor GC happens and clears Young Gen down to 500MB used. Minor GCs are good and better than a full GC, but it still slows down my program a lot (I am pretty sure the app still freezes when a young gen collection occurs, because I see the database activity die off). I am spending well over 5% of my program time frozen for minor GCs, and this seems excessive. I would say over the course of processing this 4 GB file, I churn through 50-60GB of young gen memory.
I don't see any obvious flaws in my program. I am trying to obey the general OO principles and write clean Java code. I am trying not to create objects for no reason. I am using thread pools, and whenever possible passing objects along instead of creating new objects. I am going to start profiling the application, but I was wondering if anyone had some good general rules of thumb or anti patterns to avoid that lead to excessive memory churn? Is 50-60GB of memory churn to process a 4GB file the best I can do? Do I have to revert to JDk 1.2 tricks like Object Pooling? (although Brian Goetz give a presentation that included why object pooling is stupid, and we don't need to do it anymore. I trust him a lot more than I trust myself .. :) )

I have a feeling that you are spending time and effort trying to optimize something that you should not bother with.
I am spending well over 5% of my program time frozen for minor GCs, and this seems excessive.
Flip that around. You are spending just under 95% of your program time doing useful work. Or put it another way, even if you managed to optimize the GC to run in ZERO time, the best you can get is something over 5% improvement.
If your application has hard timing requirements that are impacted by the pause times, you could consider using a low-pause collector. (Be aware that reducing pause times increases the overall GC overheads ...) However for a batch job, the GC pause times should not be relevant.
What probably matters most is the wall clock time for the overall batch job. And the (roughly) 95% of the time spent doing application specific stuff is where you are likely to get more pay-off for your profiling / targeted optimization efforts. For example, have you looked at batching the updates that you send to the database?
So.. 90% of my total memory is in char[] in "oracle.sql.converter.toOracleStringWithReplacement"
That would tend to indicate that most of your memory usage occurs in the Oracle JDBC drivers while preparing stuff to be sent to the database. There's very little you about that. I'd chalk it up as an unavoidable overhead.

It would be really usefull if you clarify your terms "young" and "tentured" generation because Java 6 has a slightly different GC-Model: Eden, S0+S1, Old, Perm
Have you experimented with the different garbage collection algorithms? How has "UseConcMarkSweepGC" or "UseParNewGC" performed.
And don't forget simply increasing the available space is NOT the solution, because a gc run will take much longer, decrease the size to normal values ;)
Are you sure you have no memory-leaks? In a consumer-producer-pattern - you describe - rarely seldom data should be in the Old Gen because those jobs are proccessed really fast and then "thrown away", or is your work queue filling up?
You should defintely observe your program with a memory analyzer.

I think a session with a memory profiler will shed a lot of light on the subject. This gives a nice overview how many objects are created and this is somtimes revealing.
I am always amazed how many strings are generated.
For domain objects crossreferencing them is also revealing. If you see suddenly 3 times more objects from a derived object than from the source then there something going on there.
Netbeans has a nice one built it. I used JProfiler in the past. I think if you bang long enough on eclipse you can get the same info from the PPTP tools.

You need to profile your application to see what is happening exactly. And I would also try first to use the ergonomics feature of the JVM, as recommended:
2. Ergonomics
A feature referred to here as
ergonomics was introduced in J2SE 5.0.
The goal of ergonomics is to provide
good performance with little or no
tuning of command line options by
selecting the
garbage collector,
heap size,
and runtime compiler
at JVM startup, instead of using fixed
defaults. This selection assumes that
the class of the machine on which the
application is run is a hint as to the
characteristics of the application
(i.e., large applications run on large
machines). In addition to these
selections is a simplified way of
tuning garbage collection. With the
parallel collector the user can
specify goals for a maximum pause time
and a desired throughput for an
application. This is in contrast to
specifying the size of the heap that
is needed for good performance. This
is intended to particularly improve
the performance of large applications
that use large heaps. The more general
ergonomics is described in the
document entitled “Ergonomics in the
5.0 Java Virtual Machine”. It is recommended that the ergonomics as
presented in this latter document be
tried before using the more detailed
controls explained in this document.
Included in this document are the
ergonomics features provided as part
of the adaptive size policy for the
parallel collector. This includes the
options to specify goals for the
performance of garbage collection and
additional options to fine tune that
performance.
See the more detailed section about Ergonomics in the Java SE 6 HotSpot[tm] Virtual Machine Garbage Collection Tuning guide.

In my opinion, the young generation should not be equally big as the old generation, so that the small garbage collections stay fast.
Do you have many objects that represent the same value? If you do, merge these duplicate objects using a simple HashMap:
public class MemorySavingUtils {
ConcurrentHashMap<String, String> knownStrings = new ConcurrentHashMap<String, String>();
public String unique(String s) {
return knownStrings.putIfAbsent(s, s);
}
public void clear() {
knownStrings.clear();
}
}
With the Sun Hotspot compiler, the native String.intern() is really slow for large numbers of Strings, that's why I suggest to build your own String interner.
Using this method, strings from the old generation are reused and strings from the new generation can be garbage collected quickly.

Read a line from a file, store as a string and put in a list. When the list has 1000 of these strings, put it in a queue to be read by worker threads. Have said worker thread make a domain object, peel a bunch of values off the string to set the fields (int, long, java.util.Date, or String), and pass the domain object along to a default spring batch jdbc writer
if that's your program, why not set a smaller memory size, like 256MB?

I'm guessing with a memory limit that high you must be reading the file entirely into memory before doing the processing. Could you consider using a java.io.RandomAccessFile instead?

Java 6 Excessive Memory Usage

Does Java 6 consume more memory than you expect for largish applications?
I have an application I have been developing for years, which has, until now taken about 30-40 MB in my particular test configuration; now with Java 6u10 and 11 it is taking several hundred while active. It bounces around a lot, anywhere between 50M and 200M, and when it idles, it does GC and drop the memory right down. In addition it generates millions of page faults. All of this is observed via Windows Task Manager.
So, I ran it up under my profiler (jProfiler) and using jVisualVM, and both of them indicate the usual moderate heap and perm-gen usages of around 30M combined, even when fully active doing my load-test cycle.
So I am mystified! And it not just requesting more memory from the Windows Virtual Memory pool - this is showing up as 200M "Mem Usage".
CLARIFICATION: I want to be perfectly clear on this - observed over an 18 hour period with Java VisualVM the class heap and perm gen heap have been perfectly stable. The allocated volatile heap (eden and tenured) sits unmoved at 16MB (which it reaches in the first few minutes), and the use of this memory fluctuates in a perfect pattern of growing evenly from 8MB to 16MB, at which point GC kicks in an drops it back to 8MB. Over this 18 hour period, the system was under constant maximum load since I was running a stress test. This behavior is perfectly and consistently reproducible, seen over numerous runs. The only anomaly is that while this is going on the memory taken from Windows, observed via Task Manager, fluctuates all over the place from 64MB up to 900+MB.
UPDATE 2008-12-18: I have run the program with -Xms16M -Xmx16M without any apparent adverse affect - performance is fine, total run time is about the same. But memory use in a short run still peaked at about 180M.
Update 2009-01-21: It seems the answer may be in the number of threads - see my answer below.
EDIT: And I mean millions of page faults literally - in the region of 30M+.
EDIT: I have a 4G machine, so the 200M is not significant in that regard.

In response to a discussion in the comments to Ran's answer, here's a test case that proves that the JVM will release memory back to the OS under certain circumstances:
public class FreeTest
{
public static void main(String[] args) throws Exception
{
byte[][] blob = new byte[60][1024*1024];
for(int i=0; i<blob.length; i++)
{
Thread.sleep(500);
System.out.println("freeing block "+i);
blob[i] = null;
System.gc();
}
}
}
I see the JVM process' size decrease when the count reaches around 40, on both Java 1.4 and Java 6 JVMs (from Sun).
You can even tune the exact behaviour with the -XX:MaxHeapFreeRatio and -XX:MinHeapFreeRatio options -- some of the options on that page may also help with answering the original question.

I don't know about the page faults. but about the huge memory allocated for Java:
Sun's JVM only allocates memory, never deallocates it (until JVM death) deallocates memory only after a specific ratio between internal memory needs and allocated memory drops beneath a (tunable) value. The JVM starts with the amount specified in -Xms and can be extended up to the amount specified in -Xmx. I'm not sure what the defaults are. Whenever the JVM needs more memory (new objects / primitives / arrays) it allocates an entire chunk from the OS. However, when the need subsides (a momentary need, see 2 as well) it doesn't deallocates the memory back the the OS immediately, but keeps it to itself until that ratio has been reached. I was once told that JRockit behaves better, but I can't verify it.
Sun's JVM runs a full GC based on several triggers. One of them is the amount of available memory - when it falls down too much the JVM tries to perform a full GC to free some more. So, when more memory is allocated from the OS (momentary need) the chance for a full GC is lowered. This means that while you may see 30Mb of "live" objects, there might be a lot more "dead" objects (not reachable), just waiting for a GC to happen. I know yourkit has a great view called "dead objects" where you may see these "left-overs".
In "-server" mode, Sun's JVM runs GC in parallel mode (as opposed the older serial "stop the world" GC). This means that while there may be garbage to collect, it might not be collected immediately because of other threads taking all available CPU time. It will be collected before reaching out of memory (well, kinda. see http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html), if more memory can be allocated from the OS, it might be before the GC runs.
Combined, a large initial memory configuration and short bursts creating a lot of short-lived objects might create a scenario as described.
edit: changed "never deallcoates" to "only after ratio reached".

Excessive thread creation explains your problem perfectly:
Each Thread gets its own stack, which is separate from heap memory and therefore not registered by profilers
The default thread stack size is quite large, IIRC 256KB (at least it was for Java 1.3)
Tread stack memory is probably not reused, so if you create and destroy lots of threads, you'll get lots of page faults
If you ever really need to have hundreds of threads aound, the thread stack size can be configured via the -Xss command line parameter.

Garbage collection is a rather arcane science. As the state of the art develops, un-tuned behaviour will change in response.
Java 6 has different default GC behaviour and different "ergonomics" to earlier JVM versions. If you tell it that it can use more memory (either explicitly on the command line, or implicitly by failing to specify anything more explicit), it will use more memory if it believes that this is likely to improve performance.
In this case, Java 6 appears to believe that reserving the extra space which the heap could grow into will give it better performance - presumably because it believes that this will cause more objects to die in Eden space, and limit the number of objects promoted to the tenured generation space. And from the specifications of your hardware, the JVM doesn't think that this extra reserved heap space will cause any problems. Note that many (though not all) of the assumptions the JVM makes in reaching its conclusion are based on "typical" applications, rather than your specific application. It also makes assumptions based on your hardware and OS profile.
If the JVM has made the wrong assumptions, you can influence its behaviour through the command line, though it is easy to get things wrong...
Information about performance changes in java 6 can be found here.
There is a discussion about memory management and performance implications in the Memory Management White Paper.

Over the last few weeks I had cause to investigate and correct a problem with a thread pooling object (a pre-Java 6 multi-threaded execution pool), where is was launching far more threads than required. In the jobs in question there could be up to 200 unnecessary threads. And the threads were continually dying and new ones replacing them.
Having corrected that problem, I thought to run a test again, and now it seems the memory consumption is stable (though 20 or so MB higher than with older JVMs).
So my conclusion is that the spikes in memory were related to the number of threads running (several hundred). Unfortunately I don't have time to experiment.
If someone would like to experiment and answer this with their conclusions, I will accept that answer; otherwise I will accept this one (after the 2 day waiting period).
Also, the page fault rate is way down (by a factor of 10).
Also, the fixes to the thread pool corrected some contention issues.

Lots of memory allocated outside Java's heap after upgrading to Java 6u10? Can only be one thing:
Java6 u10 Release Notes: "New Direct3D Accelerated Rendering Pipeline (...) Enabled by Default"
Sun enabled Direct 3D accelerations by default in Java 6u10. This option creates lots of (temporary?) native memory buffers, which are allocated outside the Java Heap. Add the following vm argument to disable it again:
-Dsun.java2d.d3d=false
Note that this will NOT disable 2D hardware acceleration, just some features that can make use of 3D hardware acceleration. You will see that your Java heap usage will increase by up to 7MB, but that's a good trade-off because you'll save ~100MB(+) of this temporary volatile memory.
I did a fair amount of testing within 2 Swing desktop application, on two platforms:
a high-end Intel-i7 with nVidia GTX 260 graphics card,
a 3-year laptop with Intel graphics.
On both hardware platforms the option made practically zero subjective difference. (Tests included: scrolling tables, zooming graphical flowsheets, charts, etc.). On the few tests where something was subtly different, disabling d3d counter-intuitively increased performance. I suspect that memory management/bandwidth problems counteracted whatever benefits the d3d accelerated functions were supposed to achieve. (Your mileage may vary!)
If you need to do some performance tuning, here's an excellent reference (e.g. "Troubleshooting Java 2D")

Are you using the ConcMarkSweep collector? It can increase the amount of memory required for your application due to increased memory fragmentation, and "floating garbage" - objects that become unreachable only after the collector has examined them, and therefore are not collected until the next pass.