I have a long-running (multiple days) application with objects that I expect to stay around for varying lengths of time before they can be garbage collected. Let's say there's four categories:
Very short-lived (<1s)
Alive for the duration of user attention (1s < 18h)
Daily data (~24h)
'Eternal' (very few, life of the application)
To help with tuning, I'd like to find a way of checking what actual data is getting in to the tenured generation, using the Java 6 Hotspot VM. Using jmap to generate HPROF files doesn't seem to include generational information. Is there another way of getting this information?
No, there is no simple way to get generation information for individual object. In fact if you ask for "live" objects, this will trigger a Full GC and place all objects into the old generation (so now you know where all the objects are, but not where they were)
Objects which survive a full GC are likely to be in old generation so if your system does a full GC every 5 minutes, anything which lasts longer than that is much the same.
The best thing you can do is to minimise discarded objects (use a memory profiler to help) This will improve GC performance and descrease how often they occur. In extreme examples you can use off heap memory which is difficult to work with but uses next to no heap. esp useful if you have many GB of data.
Related
Is there any condition when application will never perform Garbage Collection ? Theoretically is it possible to have such application design ?
Yes, there is. Please read about memory leaks in Java. An example is described in Effective Java Item 6: Eliminate obsolete object references
Garbage collection happens on objects which are not referenced anymore in your application.
With Java 11, there is a way to never purposely perform garbage collection, by running your JVM with the newly introduced Epsilon GC, a garbage collector which handles memory allocation but never releases the allocated memory.
There is at least one product in the market that implements high frequency trading using Java and jvm technology.
Obviously, an application that needs to react in microseconds can't afford a garbage collector to kick in and halt the system for arbitrary periods of time.
In this case, the solution was to write the whole application to never create objects that turn into garbage. For example, all input data is kept in fixed byte arrays (that are allocated once at start time) which are then used as buffers for all kinds of processing.
Unless I am mistaken, you can listen to more details on the software engineering radio podcast. I think it should be this episode: http://www.se-radio.net/2016/04/se-radio-episode-255-monica-beckwith-on-java-garbage-collection/
Is there any condition when application will never perform Garbage Collection ?
You can prevent the GC from running by having a Thread which doesn't reach a safe point.
Unless you use a concurrent collector, the GC will only be performed when a memory region, e.g. when the Eden or Tenure spaces fill.
If you make these large enough, and your garbage rate low enough, the GC won't run for long enough that you can either perform a GC overnight, in a maintenance window or restart the process.
Theoretically is it possible to have such application design?
I have worked on applications which GC less than once per day (and some of them are restarted every day)
For example, say you produce 300KB of garbage per second, or 1 GB per hour, with a 24 GB Eden size you can run for a whole day without a collection.
In reality, if you move most of your data off-heap e.g. Chronicle Map or Queue, you might find a 4 GB, can run for a day or even a week with a minor collection.
When viewing my remote application in JVisualVM over JMX, I see a saw-tooth of memory usage while idle:
Taking a heap dump and analysing it with JVisualVM, I see a large chunk of memory is in a few big int[] arrays which have no references and by comparing heap dumps I can see that it seems to be these that are taking the memory and being reclaimed by a GC periodically.
I am curious to track these down since it piqued my interest that my own code never knowingly allocates any int[] arrays.
I do use a lot of libs like netty so the culprit could be elsewhere. I do have other servers with much the same mix of frameworks but don't see this sawtooth there.
How can I discover who is allocating them?
Take a heapdump and find out what objects are holding them. Once you know what objects are holding the arrays you should have an easy time idea figuring out what is allocating them.
It doesn't answer your question, but my question is:
Why do you care?
You've told the jvm garbage collector (GC) it can use up to 1GB of memory. Java is using less than 250M.
The GC tries to be smart about when it garbage collects and also how hard it works at garbage collection. In your graph, there is no demand for memory. The jvm isn't anywhere near that 1GB limit you set. I see no reason the GC should try very hard at all. Not sure why you would care either.
Its a good thing for the garbage collector to be lazy. The less the GC works, the more resources there are available for your application.
Have you tried triggering GC via the JVisualVM "Perform GC" button? That button should trigger a "stop the world" garbage collection operation. Try it when the graph is in the middle of one of those saw tooth ramp ups - I predict that the usage will drop to the base of the saw tooth or below. If it does, that proves that the memory saw tooth is just garbage accumulation and GC is doing the right thing.
Here is an screenshot of memory usage for a java swing application I use:
Notice the sawtooth pattern.
You said you are worried about int[]. When I start the memory profiler and have it profile everything I can see the allocations of int[]
Basically all allocations come from an ObjectOutputStream$HandleTable.growEntries method. It looks like the thread the allocations were made on was spun up to handle a network message.
I suspect its caused by jmx itself. Possibly by rmi (do you use rmi?). Or the debugger (do you have a debugger connected?).
I just thought I'd add to this question that the sawtooth pattern is very much normal and has nothing necessarily to do with your int[] arrays. It happens because new allocations happen in the Eden-gen, and an ephemeral collection only is triggered once it has filled up, leaving the old-gen be. So as long as your program does any allocations at all, the Eden gen will fill up and then empty repeatedly. Especially, then, when you have a regular amount of allocations per unit of time, you'll see a very regular sawtooth pattern.
There are tons of articles on the web detailing how Hotspot's GC works, so there's no need for me to expand on that here. If you don't know at all how ephemeral collection works, you may want to check out Wikipedia's article on the subject (see the "Generational GC" section; "generational" and "ephemeral" are synonymous in this context).
As for the int[] arrays, however, they are a bit mysterious. I'm seeing those as well, and there's another question here on SO on the subject of them without any real answer. It's not actually normal for objects with no references to show up in a heap dump, because a heap dump normally only contains live objects (because Hotspot always performs a stop-the-world collection before actually dumping the heap). My personal guess is that they are allocated as part of some kind of internal JVM data-structure (and therefore only have references from the C++ part of Hotspot rather than from the Java heap), but that's really just a pure guess.
I have a Java client which consumes a large amount of data from a server. If the client does not keep up with the data stream at a fast enough rate, the server disconnects the socket connection. My client gets disconnected a few times per day. I ran jconsole to see the memory usage, and the heap space graph looks like a fairly well defined sawtooth pattern, oscillating between about 0.5GB and 1.8GB (2GB of heap space is allocated). But every time I get disconnected is during a full GC (but not on every full GC). I see the full GC takes a bit over 1 second on average. Depending on the time of day, full GC happens as often as every 5 minutes when busy, or up to 30 minutes can go by in between full GCs during the slow periods.
I suspect if I can reduce the full GC time, the client will be able to better keep up with the incoming data, but I do not have much experience with GC tuning. Does anyone have some insight on if this might be a good idea, and how to do it? Or is there an alternative idea which may work as well?
** UPDATE **
I used -XX:+UseConcMarkSweepGC and it improved, but I still got disconnected during the very busy moments. So I increased the heap allocation to 3GB to help weather through the busy moments and it seems to be chugging along pretty well now, but it's only been 1 day without a disconnection. Maybe if I get some time I will go through and try to reduce the amount of garbage created which I'm confident will help as well. Thanks for all the suggestions.
Full GC could take very long to complete, and is not that easy to tune.
One way to (easily) tune it is to increase the heap space - generally speaking, double the heap space can double the interval between two GCs, but will double the time consumed by a GC. If the program you are running has very clear usage patterns, maybe you can consider increase the heap space to make the interval so large that you can guarantee to have some idle time to try to make the system perform a GC. On the other hand, following this logic, if the heap is small a full garbage collection will finish in a instant, but that seems like inviting more troubles than helping.
Also, -XX:+UseConcMarkSweepGC might help since it will try to perform the GC operations concurrently (not stopping your program; see here).
Here's a very nice talk by Til Gene (CTO of Azul systems, maker of high performance JVM, and published several GC algos), about GC in JVM in general.
It is not easy to tune away the Full GC. A much better approach is to produce less garbage. Producing less garbage reduces pressure on the collection to pass objects into the tenured space where they are more expensive to collect.
I suggest you use a memory profiler to
reduce the amount of garbage produced. In many applications this can be reduce by a factor of 2 - 10x relatively easily.
reduce the size of the objects you are creating e.g. use primitive and smaller datatypes like double instead of BigDecimal.
recycle mutable object instead of discarding them.
retain less data on the client if you can.
By reducing the amount of garbage you create, objects are more likely to die in the eden, or survivor spaces meaning you have far less Full collections, which can be shorter as well.
Don't take it for granted you have to live with lots of collections, in extreme cases you can avoid it almost completely http://vanillajava.blogspot.ro/2011/06/how-to-avoid-garbage-collection.html
Take out calls to Runtime.getRuntime().gc() - When garbage collection is triggered manually it either does nothing or it does a full stop-the-world garbage collection. You want incremental GC to happen.
Have you tried using the server jvm from a jdk install? It changes a bunch of the default configuration settings (including garbage collection) and is easy to try - just add -server to your java command.
java -server
What is all the garbage that gets created? Can you generate less of it? Where possible, try to use the valueOf methods. By using less memory you'll save yourself time in gc AND in memory allocation.
This question is a follow up to my previous Java GC question: Java Garbage Collector clarification
This question is also referring to the same article.
I'm a little confused on why the stop and copy method for defragmenting object heap allocation is so commonly used. Yes it defragments the heap however it seems like there is tons of overhead because basically you cut the total amount of heap size in half. Also you need to copy ALL the live objects when one half has run out of space.
Other than defragmentation is there any other fundamental reason why 'stop and copy' is better than say 'mark and sweep'?
Actually, fragmentation is fundamental, and the ability of some GC to defeat it is a considerable asset.
The stop-and-copy algorithm used to be popular in GC implementations because:
it is simple to implement;
it automatically defragments memory;
its running time is proportional to the amount of live objects, which makes it asymptotically very efficient.
More modern GC, including those used in Java, use much more complex strategies because they want to make short pauses (rather than making total GC time low, they prefer never to stop the application for a long time, because pauses are bad for interactivity), to interact more cleanly with caches and virtual memory, and to benefit from systems with multiple CPU.
The Jones and Lins book is a must-read for whoever wants to understand garbage collection.
A great tutorial on the garbage collector is Tuning Garbage Collection (unfortunately the new oracle website has messed its formatting up quiet a lot).
Your question is handled in chapter V. This basically explains which types of strategies you can use in the Java garbage collector and which are default. Most desktop applications will be interested in a stop that is as small as possible, because this is what the user might notice.
Note that your question is not about defragmentation. Both will eventually compress the memory space.
Background
I have a Spring batch program that reads a file (example file I am working with is ~ 4 GB in size), does a small amount of processing on the file, and then writes it off to an Oracle database.
My program uses 1 thread to read the file, and 12 worker threads to do the processing and database pushing.
I am churning lots and lots and lots of young gen memory, which is causing my program to go slower than I think it should.
Setup
JDK 1.6.18
Spring batch 2.1.x
4 Core Machine w 16 GB ram
-Xmx12G
-Xms12G
-NewRatio=1
-XX:+UseParallelGC
-XX:+UseParallelOldGC
Problem
With these JVM params, I get somewhere around 5.x GB of memory for Tenured Generation, and around 5.X GB of memory for Young Generation.
In the course of processing this one file, my Tenured Generation is fine. It grows to a max of maybe 3 GB, and I never need to do a single full GC.
However, the Young Generation hits it's max many times. It goes up to 5 GB range, and then a parallel minor GC happens and clears Young Gen down to 500MB used. Minor GCs are good and better than a full GC, but it still slows down my program a lot (I am pretty sure the app still freezes when a young gen collection occurs, because I see the database activity die off). I am spending well over 5% of my program time frozen for minor GCs, and this seems excessive. I would say over the course of processing this 4 GB file, I churn through 50-60GB of young gen memory.
I don't see any obvious flaws in my program. I am trying to obey the general OO principles and write clean Java code. I am trying not to create objects for no reason. I am using thread pools, and whenever possible passing objects along instead of creating new objects. I am going to start profiling the application, but I was wondering if anyone had some good general rules of thumb or anti patterns to avoid that lead to excessive memory churn? Is 50-60GB of memory churn to process a 4GB file the best I can do? Do I have to revert to JDk 1.2 tricks like Object Pooling? (although Brian Goetz give a presentation that included why object pooling is stupid, and we don't need to do it anymore. I trust him a lot more than I trust myself .. :) )
I have a feeling that you are spending time and effort trying to optimize something that you should not bother with.
I am spending well over 5% of my program time frozen for minor GCs, and this seems excessive.
Flip that around. You are spending just under 95% of your program time doing useful work. Or put it another way, even if you managed to optimize the GC to run in ZERO time, the best you can get is something over 5% improvement.
If your application has hard timing requirements that are impacted by the pause times, you could consider using a low-pause collector. (Be aware that reducing pause times increases the overall GC overheads ...) However for a batch job, the GC pause times should not be relevant.
What probably matters most is the wall clock time for the overall batch job. And the (roughly) 95% of the time spent doing application specific stuff is where you are likely to get more pay-off for your profiling / targeted optimization efforts. For example, have you looked at batching the updates that you send to the database?
So.. 90% of my total memory is in char[] in "oracle.sql.converter.toOracleStringWithReplacement"
That would tend to indicate that most of your memory usage occurs in the Oracle JDBC drivers while preparing stuff to be sent to the database. There's very little you about that. I'd chalk it up as an unavoidable overhead.
It would be really usefull if you clarify your terms "young" and "tentured" generation because Java 6 has a slightly different GC-Model: Eden, S0+S1, Old, Perm
Have you experimented with the different garbage collection algorithms? How has "UseConcMarkSweepGC" or "UseParNewGC" performed.
And don't forget simply increasing the available space is NOT the solution, because a gc run will take much longer, decrease the size to normal values ;)
Are you sure you have no memory-leaks? In a consumer-producer-pattern - you describe - rarely seldom data should be in the Old Gen because those jobs are proccessed really fast and then "thrown away", or is your work queue filling up?
You should defintely observe your program with a memory analyzer.
I think a session with a memory profiler will shed a lot of light on the subject. This gives a nice overview how many objects are created and this is somtimes revealing.
I am always amazed how many strings are generated.
For domain objects crossreferencing them is also revealing. If you see suddenly 3 times more objects from a derived object than from the source then there something going on there.
Netbeans has a nice one built it. I used JProfiler in the past. I think if you bang long enough on eclipse you can get the same info from the PPTP tools.
You need to profile your application to see what is happening exactly. And I would also try first to use the ergonomics feature of the JVM, as recommended:
2. Ergonomics
A feature referred to here as
ergonomics was introduced in J2SE 5.0.
The goal of ergonomics is to provide
good performance with little or no
tuning of command line options by
selecting the
garbage collector,
heap size,
and runtime compiler
at JVM startup, instead of using fixed
defaults. This selection assumes that
the class of the machine on which the
application is run is a hint as to the
characteristics of the application
(i.e., large applications run on large
machines). In addition to these
selections is a simplified way of
tuning garbage collection. With the
parallel collector the user can
specify goals for a maximum pause time
and a desired throughput for an
application. This is in contrast to
specifying the size of the heap that
is needed for good performance. This
is intended to particularly improve
the performance of large applications
that use large heaps. The more general
ergonomics is described in the
document entitled “Ergonomics in the
5.0 Java Virtual Machine”. It is recommended that the ergonomics as
presented in this latter document be
tried before using the more detailed
controls explained in this document.
Included in this document are the
ergonomics features provided as part
of the adaptive size policy for the
parallel collector. This includes the
options to specify goals for the
performance of garbage collection and
additional options to fine tune that
performance.
See the more detailed section about Ergonomics in the Java SE 6 HotSpot[tm] Virtual Machine Garbage Collection Tuning guide.
In my opinion, the young generation should not be equally big as the old generation, so that the small garbage collections stay fast.
Do you have many objects that represent the same value? If you do, merge these duplicate objects using a simple HashMap:
public class MemorySavingUtils {
ConcurrentHashMap<String, String> knownStrings = new ConcurrentHashMap<String, String>();
public String unique(String s) {
return knownStrings.putIfAbsent(s, s);
}
public void clear() {
knownStrings.clear();
}
}
With the Sun Hotspot compiler, the native String.intern() is really slow for large numbers of Strings, that's why I suggest to build your own String interner.
Using this method, strings from the old generation are reused and strings from the new generation can be garbage collected quickly.
Read a line from a file, store as a string and put in a list. When the list has 1000 of these strings, put it in a queue to be read by worker threads. Have said worker thread make a domain object, peel a bunch of values off the string to set the fields (int, long, java.util.Date, or String), and pass the domain object along to a default spring batch jdbc writer
if that's your program, why not set a smaller memory size, like 256MB?
I'm guessing with a memory limit that high you must be reading the file entirely into memory before doing the processing. Could you consider using a java.io.RandomAccessFile instead?