I have a server java component which has a huge memory demand at startup which mellows down gradually. So as an example at startup the memory requirement might shoot unto 4g; which after the initial surge is over will go down to 2g. I have configured the component to start with memory of 5g and the component starts well; the used memory surges upto 4g and then comes down close to 2g. The memory consumed by the heap at this point still hovers around 4g and I want to bring this down (essentially keep the free memory down to few hundred mb's rather than 2g. I tried using the MinFreeHeapRatio and MaxFreeHeapRatio by lowering them down from the default values but this resulted in garbage collection not being triggered after the initial run during the initial spike and the used memory stayed at a higher than usual level. Any pointers would greatly help.
First, I ask why you are worried about freeing up 2 GB of ram on a server? 2GB of ram is about $100 or less. If this is on a desktop I guess I can understand the worry.
If you really do have a good reason to think about it, this may be tied to the garbage collection algorithm you are using. Some algorithms will release unused memory back to the OS, some will not. There are some graphs and such related to this at http://www.stefankrause.net/wp/?p=14 . You might want to try the G1 collector, as it seems to release memory back to the OS easily.
Edit from comments
What if they all choose to max their load at once? Are you ok with some of them paging memory to disk and slowing the server to a crawl? I would run them on a server with enough memory to run ALL applications at max heap, + another 4-6GB for the OS / caching. Servers with 32 or 64 GB are pretty common, and you can get more.
You have to remember that the JVM reserves virtual memory on startup and never gives it back to the OS (until the program exits) The most you can hope for is that unused memory is swapped out (this is not a good plan) If you don't want the application to use more than certain amount of memory, you have to restrict it to that amount.
If you have a number of components which use a spike of memory, you should consider re-writing them to use less memory on startup or use multiple components in the same JVM so the extra memory is less significant.
Related
We have a few containers running java processes with docker. One thing we've been noticing is a huge amount of memory that is taken up just by running a simple spring-boot app without even including our own code (just to try and get some kind of memory profile independent of any issues we might introduce).
What I saw was the memory consumed by docker/the JVM was hovering around 2.5. We did have a decent amount of extra deps included in it (camel, hibernate, some spring-boot deps) but that wasn't what really threw me off. What I saw was that despite docker saying it consumed 2.5GB of memory for the app, running jconsole against it read that it was consuming up to 1GB (down to ~200MB after a GC and slowly climbing). The memory footprint on docker remained where it was after the GC as well (2.5GB).
Furthermore, when I dumped the heap to see what kinds of object are taking up that space, it looks like the heap was only 33MB large after I loaded the .hprof file into MAT. None of this makes much sense to me. Currently, I'm looking at the non-heap space in jconsole reported at 115MB while the heap space is at 331MB.
I've already read a ton (on SO and other sites) about the JVM memory regions and some things specifically reporting that the heap dumps might be smaller but none of them were this far off that I could tell and beyond that, many of the suggested things to watch for were that the GC is run whenever a heap dump is taken and that MAT has a setting to show or hide unreachable objects. All of this was taken into account before posting here and now I just feel like something else is at play that I can't capture myself and I haven't found online.
I fully expect that the numbers might be a little off but it seems extreme that they're off by a factor of 10 in the best case scenario and off by nearly a factor of 100 when looking at the docker-reported memory usage.
Does anyone know what I might be missing here?
EDIT: This is also an app running with Java 8, not yet running with Java 11. It's on the JIRA board to do but not yet planned for.
EDIT2: Adding screenshots. Spike in the JConsole screen shot is from running GC.
JConsole gives you the amount of committed memory: 3311616 KiB ~= 3GiB
This is how much memory your java process consumes, as seen by the OS.
It is unrelated to how much heap is currently in use to hold Java objects, also reported by JConsole as 130237 kbyte ~= 130 MiB.
This is also unrelated to how many Objects are actually alive: By default MAT will remove unreachable Objects when you load the heap dump. You can enable the option by going to Preferences -> Memory Analyzer -> Keep Unreachable Objects (See the MAT documentation). So if you have a lot of short lived objects, the difference can be quite massive.
I see that it also reports a Max Heap of about 9GiB. It means that you have set Xmx parameter to a large value.
Hotspot GC's are not very good at reclaiming unused memory. They tend to use all the space available to them (the Max heap size, set by Xmx) and then never decommit the heap, effectively keeping it reserved for the Java process instead of releasing it to the OS.
If you want to minimize the memory footprint of your process from the OS perspective, I recommend that you set a lower Xmx, maybe -Xmx1g, so as to not allow Java to grow too much (of course, Xmx will also need to be high enough to accomodate for your application workload!).
If really want an adaptative heap, you can also switch to G1 (-XX:+UseG1GC) and a more recent Java, as the hotspot team has delivered some improvements recently.
Dave
OS monitoring tools will show to you the amount of memory that is allocated by a process. So this:
mean that your java process have 2.664G of memory allocated (java heap + meta space)
JConsole shows to you the memory that your code is "consuming" (ignoring the meta space)
I see 2 possible explanations:
You have set -Xms with a huge value
You have a lot of static
code (or other content) loaded on your meta space.
I have an DFS algorithm java console application, that runs faster when more memory is provided. Just a DFS algorithm application, with neither I/O nor other outer-JVM resource usage. It consumes only CPU and memory. The application can run with an 1GB memory, but run much more faster with 2 GB memory. More memory provided, faster the application can run. I haven't touch the speed limit as 12GB of memory provided. So I must use all remain memory of a server to speed it up. And the application need not parallel, one request only at one time.
And I need to install the application on different server with different memory size.
Is there a way to let JVM use the all remain memory of the server?
-XX:MaxRAMFraction=1
MaxRAMFraction is not EVERY server good, some server will result in start JVM failure as location memory failure, some works good.
Use an wrapper application get system remain memory, and minus some memory usage other than Xmx, then start the real application with same Xms and Xms. The method will also result in JVM memory allocation error. Because the code below returns much more than memory we can use, not just a minus of Xss256m or some more non-heap JVM memory.
com.sun.management.OperatingSystemMXBean mbean = (com.sun.management.OperatingSystemMXBean)
ManagementFactory.getOperatingSystemMXBean();
long size = mbean.getFreePhysicalMemorySize();
So is there a good way to let JVM use all remain memory of a server?
For large regions of memory I use off heap and this reduces overhead on the GC, one of the benefitis is that is can be any size at runtime and even larger than main memory if you do it carefully. You can use direct ByteBuffers but I use a library I wrote which extends the ByteBuffer functionality (>> 2 GB and thread safe) Chronicle Bytes The largest any one uses this is ~100 TB of virtual memory mapped to disk.
We have two data structures on top of Chronicle Bytes, a key-value store Chronicle Map and a queue/journal Chronicle Queue. This can make storing data off heap easier with a higher level interface.
The way the heap works, it has to reserve the maximum heap size on start up as a single continuous block of virtual memory. In particular, the GC assumes random access to this memory on a clean up which means if you are have slightly over utilised your memory, possibly because a process started after yours and some of the heap is swapped out you will see a dramatic fall in performance for your whole machine. Windows tends to start swapping your GUI meaning you can't get back control without a power cycle. Linux isn't as bad, but you will want to kill your process at this point. This makes tuning it size to use all memory very hard if the usage of your machine changes.
By using virtual memory by comparison, the GC doesn't touch it so unused portions have little impact. You can have areas of virtual memory many times main memory but only your current working set matters, and this is a size entirely in your control at runtime. Note: on Linux you can have virtual memory sizes 1000x your free disk space, but use with care, if you run out by touching too many pages your program will crash.
I've got a strange problem in my Clojure app.
I'm using http-kit to write a websocket based chat application.
Client's are rendered using React as a single page app, the first thing they do when they navigate to the home page (after signing in) is create a websocket to receive things like real-time updates and any chat messages. You can see the site here: www.csgoteamfinder.com
The problem I have is after some indeterminate amount of time, it might be 30 minutes after a restart or even 48 hours, the JVM running the chat server suddenly starts consuming all the CPU. When I inspect it with NR (New Relic) I can see that all that time is being used by the garbage collector -- at this stage I have no idea what it's doing.
I've take a number of screenshots where you can see the effect.
You can see a number of spikes, those spikes correspond to large increases in CPU usage because of the garbage collector. To free up CPU I usually have to restart the JVM, I have been relying on receiving a CPU alert from NR in my slack account to make sure I jump on these quickly....but I really need to get to the root of the problem.
My initial thought was that I was possibly holding onto the socket reference when the client closed it at their end, but this is not the case. I've been looking at socket count periodically and it is fairly stable.
Any ideas of where to start?
Kind regards, Jason.
It's hard to imagine what could have caused such an issue. But at first what I would do is taking a heap dump at the time of crash. This can be enabled with -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<path_to_your_heap_dump> JVM args. As a general practice don't increase heap size more the size of physical memory available on your server machine. In some rare cases JVM is unable to dump heap space because process is doomed; in such cases you can use gcore(if you're on Linux, not sure about Windows).
Once you grab the heap dump, analyse it with mat, I have debugged such applications and this worked perfectly to pin down any memory related issues. Mat allows you to dissect the heap dump in depth so you're sure to find the cause of your memory issue if it is not the case that you have allocated very small heap space.
If your program is spending a lot of CPU time in garbage collection, that means that your heap is getting full. Usually this means one of two things:
You need to allocate more heap to your program (via -Xmx).
Your program is leaking memory.
Try the former first. Allocate an insane amount of memory to your program (16GB or more, in your case, based on the graphs I'm looking at). See if you still have the same symptoms.
If the symptoms go away, then your program just needed more memory. Otherwise, you have a memory leak. In this case, you need to do some memory profiling. In the JVM, the way this is usually done is to use jmap to generate a heap dump, then use a heap dump analyser (such as jhat or VisualVM) to look at it.
(Fair disclosure: I'm the creator of a jhat fork called fasthat.)
Most likely your tenure space is filling up triggering a full collection. At this time the GC uses all the CPUS for sometime seconds at time.
To diagnose why this is happening you need to look at your rate of promotion (how much data is moving from young generation to tenured space)
I would look at increasing the young generation size to decrease rate of promotion. You could also look at using CMS as this has shorter pause times (though it uses more CPU)
Things to try in order:
Reduce the heap size
Count the number of objects of each class, and see if the numbers makes sense
Do you have big byte[] that lives past generation 1?
Change or tune GC algorithm
Use high-availability, i.e. more than one JVM
Switch to Erlang
You have triggered a global GC. The GC time grows faster-than-linear depending on the amount of memory, so actually reducing the heap space will trigger the global GC more often and make it faster.
You can also experiment with changing GC algorithm. We had a system where the global GC went down from 200s (happened 1-2 times per 24 hours) to 12s. Yes, the system was at a complete stand still for 3 minutes, no the users were not happy :-) You could try -XX:+UseConcMarkSweepGC
http://www.fasterj.com/articles/oraclecollectors1.shtml
You will always have stops like this for JVM and similar; it is more about how often you will get it, and how fast the global GC will be. You should make a heap dump and get the count of the different objects of each class. Most likely, you will see that you have millions of one of them, somehow, you are keeping a pointer to them unnecessary in a ever growing cache or sessions or similar.
http://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks001.html#CIHCAEIH
You can also start using a high-availability solution with at least 2 nodes, so that when one node is busy doing GC, the other node will have to handle the total load for a time. Hopefully, you will not get the global GC on both systems at the same time.
Big binary objects like byte[] and similar is a real problem. Do you have those?
At some time, these needs to be compacted by the global GC, and this is a slow operation. Many of the data-processing JVM based solution actually avoid to store all data as plain POJOs on the heap, and implement heaps themselves in order to overcome this problem.
Another solution is to switch from JVM to Erlang. Erlang is near real time, and they got by not having the concept of a global GC of the whole heap. Erlang has many small heaps. You can read a little about it at
https://hamidreza-s.github.io/erlang%20garbage%20collection%20memory%20layout%20soft%20realtime/2015/08/24/erlang-garbage-collection-details-and-why-it-matters.html
Erlang is slower than JVM, since it copies data, but the performance is much more predictable. It is difficult to have both. I have a websocket Erlang based solution, and it really works well.
So you run into a problem that is expected and normal for JVM, Microsoft CLR and similar. It will get worse and more common during the next couple of years when heap sizes grows.
I have a question on my mind. Let's assume that I have two parameters passed to JVM:
-Xms256mb -Xmx1024mb
At the beginning of the program 256MB is allocated. Next, some objects are created and JVM process tries to allocate more memory. Let's say that JVM needs to allocate 800MB. Xmx attribute allows that but the memory which is currently available on the system (let's say Linux/Windows) is 600MB. Is it possible that OutOfMemoryError will be thrown? Or maybe swap mechanism will play a role?
My second question is related to the quality of GC algorithms. Let's say that I have jdk1.5u7 and jdk1.5u22. Is it possible that in the latter JVM the memory leaks vanish and OutOfMemoryError does not occur? Can the quality of GC be better in the latest version?
The quality of the GC (barring a buggy GC) does not affect memory leaks, as memory leaks are an artifact of the application -- GC can't collect what isn't actual garbage.
If a JVM needs more memory, it will take it from the system. If the system can swap, it will swap (like any other process). If the system can not swap, your JVM will fail with a system error, not an OOM exception, because the system can not satisfy the request and and this point its effectively fatal.
As a rule, you NEVER want to have an active JVM partially swapped out. GC event will crush you as the system thrashes cycling pages through the virtual memory system. It's one thing to have a idle background JVM swapped out as a whole, but if you machine as 1G of RAM and your main process wants 1.5GB, then you have a major problem.
The JVM like room to breathe. I've seen JVMs in a GC death spiral when they didn't have enough memory, even though they didn't have memory leaks. They simply didn't have enough working set. Adding another chunk of heap transformed that JVM from awful to happy sawtooth GC graphs.
Give a JVM the memory it needs, you and it will be much happier.
"Memory" and "RAM" aren't the same thing. Memory includes virtual memory (swap), so you can allocate a total of free RAM+ free swap before you get the OutOfMemoryError.
Allocation depends on the used OS.
If you allocate too much memory, maybe you could end up having loaded portions into swap, which is slow.
If the your program runs fater os slower depends on how VM handle the memory.
I would not specify a heap that's not so big to make sure it don't occupy all the memory preventing the slows from VM.
Concerning your first question:
Actually if the machine can not allocate the 1024 MB that you asked as max heap size it will not even start the JVM.
I know this because I noticed it often trying to open eclipse with large heap size and the OS could not allocate the larger heap space the JVM failed to load. You could also try it out yourself to confirm. So the rest of the details are irrelevant to you. If course if your program uses too much swap (same as in all languages) then the performance will be horrible.
Concerning your second question:
the memory leaks vanish
Not possible as they are bugs you will have to fix
and OutOfMemoryError does not occur? Can the quality of GC be better
in the latest version?
This could happen, if for example some different algorithm in GC is used and it manages to kick-in before you seeing the exception. But if you have a memory leak then it would probable mask it or you would see it intermittent.
Also various JVMs have different GCs you can configure
Update:
I have to admit (after see #Orochi note) that I noticed the behavior on max heap on Windows. I can not say for sure that this applies to linux as well. But you could try it yourself.
Update 2:
As an answer to comments of #DennisCheung
From IBM(my emphasis):
The table shows both the maximum Java heap possible and a recommended limit for the maximum Java heap size setting ......It is important to have more physical memory than is required by all of the processes on the machine combined to prevent paging or swapping. Paging reduces the performance of the system and affects the performance of the Java memory management system.
After readings How many characters can a Java String have? I started wonder wonder:
Why is there are max heap setting in current JVMs? Why not just request more memory from the operating system when heap memory runs out and the garbage collector was unable to free needed memory? Does anybody know the rationale behind it?
I believe that it helps sandbox the Java programs ie stops them taking all the memory on the physical machine. Also Memory Leaks can still happen in Java even with garbage collection, and they can be even more subtle than in C/C++ at times.
If you really need a bigger memory allowance for your Java software you can tweek the max VM size in the config files.
Because you don't want the JVM taking over every possible resource on your O/S (well in some cases you do, but in that case set the Max Heap size to be the max your JVM, O/S combo can handle).
The JVM is a virtual machine. When you create a virtual machine, you limit its resources, since typically you want more than one virtual machine on an actual machine.
Because too much heap memory can actually be detrimental - you could have a situation where a relatively small application uses all of a large heap so that when GC does kick in it brings your app to a halt while it reclaims oodles of memory.
A smaller heap would allow gc to run more often, but not cripple your program each time.
Because physical memory is limited too. You cannot request more and more. Actually you can and OS will allocate the virtual memory even if the physical RAM is unavailable. The memory will be allocated on disk that may cause serious performance problems. In worse case the whole computer got stuck and you cannot do anything with it. It is better to fail earlier, i.e. better that JVM crashes than the physical host got stuck.
Two things:
you don't want all your resources to be consumed
a too high heap size will cause problems with garbage collector, because your program might pause for a few minutes unexpectedly because the GC must do it's job
Problem 2. is large enough for situations where hundreds of GBs are allocated for big Java processes and even alternative solutions like Terracotta's bigmemory have been developed.
Maybe you don't want your whole memory to be used by only one application.
Notice, that memory may not be released if there is still some free memory, it may mean, that if you have application perfectly running with Xmx200m and run it on no max heap limit, it could take whole memory.