We have a Java web server that often decides to do garbage-collection while it is running a service. We would like to tell it to do garbage-collection in the idle time, while no service is running. How can we do this?
You would need to be able to find out when the web container is idle, and that is likely to depend on the web container that you are using.
But I think that this is a bad idea. The way to force the GC to run is to call System.gc(). If that does anything (!) it will typically trigger a major garbage collection, and will likely take a long time (depending on the GC algorithm). Furthermore, the manually triggered collection will happen whether you need to run the GC or not1. And any request that arrives when the GC is running will be blocked.
In general, it is better to let the JVM decide when to run the GC. If you do this, the GC will run when it is efficient to do so, and will mostly run fast young space collections.
If you are concerned with request delays / jitter caused by long GC pauses, a better strategy is to tune the GC to minimize the pause times. (But beware: the low-pause collectors have greater overheads compared to the throughput collectors. This means that if your system is already heavily loaded a lot of the time, then this is liable to increase average response times.)
Other things to consider include:
Tuning your application to reduce its rate of generating garbage.
Tuning your application to reduce its heap working-set. For example, you might reduce in-memory caching by the application.
Tuning the web container. For example, check that you don't have too many worker threads.
1 - The best time to run the GC is when there is a lot of collectable garbage. Unfortunately, it is difficult for application code to know when that is. The JVM (on the other hand) has ways to keep track of how much free space there is, and when it is a good time to collect. The answer is not always when the heap is full.
Related
We are designing new software system architecture. and I am working by project manager.
but there is something on the issue within our team.
Our architect says "System memory should be kept as small as possible because it takes a long time when Full GC occurs. (JVM)
I am not sure of that opinion.
When setting up system memory, what level of Full GC(Garbage Collection) time should be reviewed?
How long will it take if Full GC occurs in a 16GB memory environment?
You might be worrying (your "architects") about something that might not be a problem for your throughput to begin with. Until java-9, the default collector is ParallelGC and there are dozens and dozens of applications that have not changed it and are happy with the pause times (and that collector pauses the world every time). So the only real answer is : measure. Enable GC logs and look into it.
On the other hand, if you choose a concurrent collector (you should start with G1), having enough breathing room for it in the heap is crucial. It is a lot more important for Shenandoan and ZGC, since they do everything concurrently. Every time GC initiates a concurrent phase, it works via so-called "barriers", which are basically interceptors for the objects in the heap. These structures used by these barriers require storage. If you will narrow this storage, GC is not going to be happy.
In rather simple words - the more "free" space in the heap, the better your GC will perform.
When setting up system memory, what level of Full GC(Garbage Collection) time should be reviewed?
This is not the correct question. If you are happy with your target response times, this does not matter. If you are not - you start analyzing gc logs and understand what is causing your delays (this is not going to be trivial, though).
How long will it take if Full GC occurs in a 16GB memory environment?
It depends on the collector and on the java version and on the type of Objects to be collected, there is no easy answer. Shenandah and ZGC - this is irrelevant since they do not care on the size of the heap to be scanned. For G1 it is going to be in the range of a "few seconds" most probably. But if you have WeakReferences and finalizers and a java version that is known to handle this not so good, the times to collect is going to be big.
How long will it take if Full GC occurs in a 16GB memory environment?
On a small heaps like that the ballpark figure is around 10 sec I guess.
But it's not what you should consider.
When setting up system memory, what level of Full GC(Garbage Collection) time should be reviewed?
All of the times. Whenever full gc occurs it should be reviewed if your application is latency-critical. This is what you should consider.
Full GC is a failure.
Failure on a multiple levels.
to address memory size available for application
to address GC type you use
to address the types of workloads
to address graceful degradation under the load
and the list goes on
Concurrent GC implicitly relies on a simple fact that it can collect faster then application allocates.
When allocation pressure becomes overwhelming GC has two options: slowdown allocations or stop them altogether.
And when it stops, you know, the hell breaks loose, futures time out, clusters brake apart and engineers fear and loathe large heaps for rest of their lives...
It's a common scenario for applications that evolve for years with increasing complexity and loads and lack of overhaul to accommodate to changing world.
It doesn't have to be this way though.
When you build new application from ground up you can design in with performance and latency in mind, with scalability and graceful degradation instead heap size and GC times.
You can split workloads that are not latency-critical but memory-heavy to different VM and run it under good 'ol ParallelGC, and it will outperform any concurrent GC in both throughput and memory overhead.
You can run latency-critical tasks under modern state-of-the-art GC like Shenandoah and have sub-second collection pauses on heaps of several TB if you don't mind some-30% memory overhead and considerable amount of CPU overhead.
Let the application and requirements dictate you heap size, not engineers.
I need help tuning one of our Microservices.
we are running a Spring based Microservice (Spring Integration, Spring Data JPA) on a jetty server in an OpenJDK8 Container. We are also using Mesosphere as our Container Orchestrating platform.
The application consumes messages from IBM MQ, does some processing and then stores the processed output in an Oracle DB.
We noticed that at some point on the 2nd of May that the queue processing stopped from our application. Our MQ team could still see that there were open connections against the queue, but the application was just not reading anymore. It did not die totally, as the healthCheck Api that DCOS hits still shows as healthy.
We use AppD for performance monitoring and what we could see is that on the same date there was a garbage collection done and from there the application never picked up messages from the queue. The graph above shows the amount of time spent doing GC on the different dates.
As part of the Java Opts we use to run the application we state
-Xmx1024m
The Mesosphere reservation for each of that Microservice is as shown below
Can someone please point me in the right direction to configure the right settings for Garbage Collection for my application.
Also, if you think that the GC is just a symptom, thanks for sharing your views on potential flaws I should be looking for.
Cheers
Kris
You should check up your code.
A GC operation will trigger a STW(Stop The World) operation which will block all the thread created in your code. But STW dosen't affect the code run state.
But gc will affect your code logic if you use such as System.currentTimeMillis to control you code run logic.
A gc operation will also effect the non-strong reference, if you're use WeakReference, SoftReference, WeakHashMap, after a full gc, these component may change their behavir.
A full gc operation is done,and freed memory dosen't allow your code to allocate new Object,your code will throw a 'OutOfMembryException' which will interrupt your code execution.
I think the things you should do now is:
First, check up the 'GC Cause', to determine if the full gc happend in System.gc() call or Allocate failed.
Then, if GC Cause is System.gc(), your should check up the non-strong reference used in your code.
Finally, if GC cause is Allocate failed, you should check up your log to determine weather there happend a OutOfMembryException in you code, if happend, you should allocate more memory to avoid OutOfMembryException.
As a suggestion, You SHOULD NOT keep your mq message in your microservice application memory. Mostlly, the source of gc problem is bad practice in your code.
I don't think that garbage collection is at fault here, or that you should be attempting to fix this by tweaking GC parameters.
I think it is one of two things:
A coincidence. A correlation (for a single data point) that doesn't imply causation.
Something about garbage collection, or the event that triggered the garbage collection has caused something to break in your application.
For the latter, there are any number of possibilities. But one that springs to mind is that something (e.g. a request) caused an application thread to allocate a really large object. That triggered a full GC in an attempt to find space. The GC failed; i.e. there still wasn't enough space after the GC did its best. That then turned into an OOME which killed the thread.
If the (hypothetical) thread that was killed by the OOME was critical to the operation application, AND the rest of the application didn't "notice" it had died, then the application as a whole would break.
One clue to look for would be an OOME logged when the thread died. But it is also possible (if the application is not written / configured appropriately) for the OOME not to appear in the logs.
Regarding the ApppD chart? Is that time in seconds? How many Full GCs do you have? Perhaps you should enable the log for the garbage collector.
Thanks for your contribution guys. We will be attempting to increase the CPU allocation from 0.5 CPU to 1.25 CPU, and execute another round of NFT tests.
We tried running the command below
jmap -dump:format=b,file=$FILENAME.bin $PID
to get a heap dump, but the utility is not present on the default OpenJDK8 container.
I have just seen your comments about CPU
increase the CPU allocation from 0.5 CPU to 1.25 CPU
Please, keep in mind that in order to execute the parallel GC you need at least two cores. I think with your configuration you are using serial collector and there is no reason to use a serial garbage collector nowadays when you can leverage the use of multiple cores. Have you consider trying at least two cores? I often use four as a minimum number for my application servers on production and performance.
You can see more information here:
On a machine with N hardware threads where N is greater than 8, the parallel collector uses a fixed fraction of N as the number of garbage collector threads. The fraction is approximately 5/8 for large values of N. At values of N below 8, the number used is N. On selected platforms, the fraction drops to 5/16. The specific number of garbage collector threads can be adjusted with a command-line option (which is described later). On a host with one processor, the parallel collector will likely not perform as well as the serial collector because of the overhead required for parallel execution (for example, synchronization). However, when running applications with medium-sized to large-sized heaps, it generally outperforms the serial collector by a modest amount on machines with two processors, and usually performs significantly better than the serial collector when more than two processors are available.
Source: https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/parallel.html
Raúl
We have linux servers running about 200 microservices in java each, using c-groups for isolation for cpu and memory isolation. One thing we have discovered is that java tends to require a full core of CPU to perform (major) garbage collection efficiently.
However obviously with 200 apps running and only 24 CPUs if they all decided to GC at the same time they would be limited by the c-groups. Since typical application CPU usage is relatively small (say about 15% of 1 cpu peak), it would be nice to find a way to ensure they don't all GC at the same time.
I'm looking into how we can schedule GCs so that each microserevice does not GC at the same time so that we can still run over 200 apps per host, but was wondering if anybody had some suggestions or experience on this topic before trying to re invent the wheel.
I found that there are command line methods that we can use, as well as using MBeans to actually trigger the GC, but read that it is not advised to do so as this will mess up the non-deterministic procedure java uses for GC.
Something that I'm thinking about is using performance metrics to monitor cpu, memory, and traffic to try and predict a GC, then if multiple are about to GC, perhaps we could trigger them one at a time, however this might be impractical or also bad idea.
We are running java 7, and 8.
You can't schedule a GC, since it depends on the allocation rate - i.e. it depends on the load and application logic. Some garbage collectors try to control major GC duration, total time consumed by GC, but not the rate. There is no guarantee as well that an externally triggered GC (e.g. via MBean) will actually run, and if it will run it might be ran later then it was triggered.
As other guys pointed, it is a very rare possibility (you can calculate it by gathering an average period in seconds of a major GC from all of your apps and bulding a histogram) to face it under the "normal" load. Under the "heavy" load you'll likely face CPU shortage much earlier than a probable simultaneous GC from an increased allocation rate will happen - because you would need to have a "lot" (depending on the size of your objects) of "long"-living objects to pollute the old generation to trigger a major GC.
We monitor our production JVMs and have monitoring triggers that (ideally) send warnings, when the JVM runs low on heap space. However, coming up with an effective detection algorithm is quite difficult, since it is the nature of garbage collection, that the application regularly has no available memory, before the GC kicks in.
There are many ways to work around this, I can think of. E.g. monitor the available space, send a warning when it becomes too low, but delay it and only trigger, when it is persistent for more than a minute. So, what works for you in practice?
Particular interesting:
How to detect a critical memory/heap problem, that needs immediate reaction?
How to detect a heap problem, that needs a precaution action?
What approaches work universally? E.g. without the need to adapt the triggers to certain JVM tuning parameters or vice versa, or, force a GC in certain time intervals.
Is there any best practice that is used widely?
I have found a very effective measure of JVM memory health to be the percentage of time the JVM spends in garbage collection. A healthy, well-tuned JVM will use very little (< 1% or so) of its CPU time collecting garbage. An unhealthy JVM will be "wasting" much of its time keeping the heap clean, and the percent of CPU used on collection will climb exponentially in a JVM experiencing a memory leak or with a max heap setting that is too low (as more CPU is used keeping the heap clean, less is used doing "real work"...assuming the inbound request rate doesn't slow down, it's easy to fall off a cliff where you become CPU bound and can't get enough work done quickly enough long before you actually get a java.lang.OutOfMemoryError).
It's worth noting that this is really the condition you want to guard against, too. You don't actually care if the JVM uses all of its heap, so long as it can efficiently reclaim memory without getting in the way of the "real work" it needs to do. (In fact, if you're never hitting the max heap size, you may want to consider shrinking your heap.)
This statistic is provided by many modern JVMs (certainly Oracle's and IBMs, at least).
Another somewhat effective measure can be the time between full GCs. The more often you are having to perform a full GC, the more time you're spending in GC.
I want to implement a heartbeat mechanism in which the child process will periodically ping the parent to tell parent it is still alive. I implement the ping task with threadPool's scheduleWithFixedDelay method in this way:
pingService.scheduleWithFixedDelay(new PingGroomServer(umbilical,
taskId), 0, pingPeriod, TimeUnit.MILLISECONDS);
umbilical is the RPC client to parent process?
Can scheduleWithFixedDelay be scheduled with the fixed delay? Will the ping thread be stopped during Stop-The-World GC?
Actually, I still miss the heartbeat even after I wait for 6 * pingPeriod ms.
So there are several choices for a more deterministic runtime (Gc pauses, thread sheduling etc) in your application
Go predicable, go realtime. One way is to heading for a realtime Java implementation based on RTSJ or other realtime Java implementations like:
http://fiji-systems.com/
http://www.aicas.com/
http://www-03.ibm.com/linux/realtime.html
Go low latency, go no stop-the-world GC. One other way is to use Java implementations with no stop-the-world Garbage collection pauses, like the Zing JVM from Azul systems which uses a low latency concurrent collector.
http://www.azulsystems.com/solutions/low-latency/overview
Since the above choices may is a rather big step for a small application there are things you can do with a "classic" Java implementation.
So, if you want short GC pauses with a Oracle/OpenJDK JVM there are some rule of thumbs:
Hardware - This is the most important. You must have a SMP system with a fast multi-threaded CPU and fast memory modules. Avoid swapping. So the faster hardware you have, the faster the Garbage collector will perform.
Use a multithreaded garbage collector. If you are using the Oracle JVM go for G1 or parallelGC (aka throughput collector)
Use a small heap, the larger heap the more space must be handled by the Garbage collector.
Tune memory (heap) ergonomics. Fine tune your memory ergonomics, let the objects be collected in the young generation (and not be promoted to the old generation) to avoid Full garbage collections.
Another answer to a similar question:
Why will a full gc on small heap takes 5 seconds?
For more information about GC ergonomics:
http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html
Yes, stop-the-world is exactly what it says it is - every thread that accesses objects on the heap that is not specifically involved with the garbage collector is stopped.
If you want a heartbeat in java, best to farm out to a separate VM running just that code so the VM pauses are not so long. Or better still, not rely upon millisecond timing - you can't assume that level of scheduling amongst processes in desktop OSs.