We Keep Coding

How to set max non-heap memory in a Java 8 (Spring Boot) application?

I have 20 Spring Boot (2.3) embedded Tomcat applications running on a Linux machine with 8GB. All applications are Java 1.8 apps. The machine was running out of memory and Linux started killing some of my app processes as a result.
Using Linux top and Spring Boot admin, I noticed that the max memory heap was set to 2GB:
java -XX:+PrintFlagsFinal -version | grep HeapSize
As a result, each of the 20 apps are trying to get 2GB of heap size (1/4th of physical mem). Using Spring Boot admin I could see only ~128 MB is being used. So I reduced the max heap size to 512 via java -Xmx512m ... Now, Spring Boot admin shows:
1.33 GB is allocated to non-heap space but only 121 MB is being used. Why is so much being allocated to non-heap space? How can I reduce?
Update
According to top each Java process is taking around 2.4GB (VIRT):
KiB Mem : 8177060 total, 347920 free, 7127736 used, 701404 buff/cache
KiB Swap: 1128444 total, 1119032 free, 9412 used. 848848 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2547 admin 20 0 2.418g 0.372g 0.012g S 0.0 4.8 27:14.43 java
.
.
.
Update 2
I ran jcmd 7505 VM.native_memory for one of the processes and it reported:
7505:
Native Memory Tracking:
Total: reserved=1438547KB, committed=296227KB
- Java Heap (reserved=524288KB, committed=123808KB)
(mmap: reserved=524288KB, committed=123808KB)
- Class (reserved=596663KB, committed=83423KB)
(classes #15363)
(malloc=2743KB #21177)
(mmap: reserved=593920KB, committed=80680KB)
- Thread (reserved=33210KB, committed=33210KB)
(thread #32)
(stack: reserved=31868KB, committed=31868KB)
(malloc=102KB #157)
(arena=1240KB #62)
- Code (reserved=254424KB, committed=27120KB)
(malloc=4824KB #8265)
(mmap: reserved=249600KB, committed=22296KB)
- GC (reserved=1742KB, committed=446KB)
(malloc=30KB #305)
(mmap: reserved=1712KB, committed=416KB)
- Compiler (reserved=1315KB, committed=1315KB)
(malloc=60KB #277)
(arena=1255KB #9)
- Internal (reserved=2695KB, committed=2695KB)
(malloc=2663KB #19903)
(mmap: reserved=32KB, committed=32KB)
- Symbol (reserved=20245KB, committed=20245KB)
(malloc=16817KB #167011)
(arena=3428KB #1)
- Native Memory Tracking (reserved=3407KB, committed=3407KB)
(malloc=9KB #110)
(tracking overhead=3398KB)
- Arena Chunk (reserved=558KB, committed=558KB)
(malloc=558KB)

First of all - no, 1.33GB is not allocated. On the screenshot you have 127MB of nonheap memory allocated. The 1.33GB is the max limit.
I see your metaspace is about 80MB which should not pose a problem. The rest of the memory can be composed by a lot of things. Compressed classes, code cache, native buffers etc...
To get the detailed view of what is eating up the offheap memory, you can query the MBean java.lang:type=MemoryPool,name=*, for example via VisualVM with an MBean plugin.
However, your apps may simply be eating too much native memory. For example many I/O buffers from Netty may be the culprit (used up by the java.nio.DirectByteBuffer). If that's the culprit, you can for example limit the caching of the DirectByteBuffers with the flag -Djdk.nio.maxCachedBufferSize, or place a limit with -XX:MaxDirectMemorySize.
For a definitive answer of what exactly is eating your RAM, you'd have to create a heap dump and analyze it.
So to answer your question "Why is so much being allocated to non-heap space? How can I reduce?" There's not a lot allocated to non-heap space. Most of it is native buffers for I/O, and JVM internals. There is no universal switch or flag to limit all the different caches and pools at once.
Now to adress the elephant in the room. I think your real issue stems from simply having very little RAM. You've said you are running 20 instances of JVM limited to 512MB of heap space on 8GB machine. That is unsustainable. 20 x 512MB = 10GB of heap, which is more than you can accommodate with 8GB of total RAM. And that is before you even count in the off-heap/native memory. You need to either provide more HW resources, decrease the JVM count or further decrease the heap/metaspace and other limits (which I strongly advise not to).

In addition to what has already been stated, here's a very good article about the Metaspace in the JVM which by defaults reserves about 1GB (though it may not actually use that much). So that's another thing you can tune using the flag -XX:MaxMetaspaceSize if you have many small apps and want to decrease the amount of memory used/reserved.

Accounting for Java memory consumption

We are running a Java spring boot application on AWS. The platform we use is Tomcat 8.5 with Java 8 running on 64bit Amazon Linux/3.3.6. The machines are 4GB machines. We run this Java application with JVM args -XMX and -XMS as 1536m. The problem we are facing is that these instances quite frequently goes in to warning state due to 90%+ memory usage. Now we are trying to account for memory usage process by process.
To start with we just ran the top command on these machines. Here is the part of output.
top - 11:38:13 up 4:39, 0 users, load average: 0.90, 0.84, 0.90Tasks: 101 total, 1 running,
73 sleeping, 0 stopped, 0 zombieCpu(s): 31.8%us, 3.7%sy, 5.6%ni, 57.2%id, 0.3%wa, 0.0%hi, 1.5%si, 0.0%st
Mem: 3824468k total, 3717908k used, 106560k free, 57460k buffersSwap: 0k total, 0k used, 0k free, 300068k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2973 tomcat 20 0 5426m 2.2g 0 S 37.1 60.6 173:54.98 java
As you can see Java is taking 2.2GB of memory. We have given XMX as 1.5GB. Although, we are aware that by using XMX we are just restricting heap, we wanted to analyse where exactly this extra 0.7GB is going towards. Towards that end, we decided to use NewRelic. And here is the graph on non heap memory usage.
The total memory non heap memory usage we could see comes around ~200MB. So with this 200MB and 1.5GB heap memory, we expect the total memory to be consumed by Java to be 1.7GB. This 1.7GB figure is also confirmed from NewRelic graphs as below:
As I mentioned earlier, the top command is telling us the Java is taking 2.2GB of memory. However we could only account for 1.7GB using NewRelic. How can we reconcile this extra 0.5GB of memory?

There's more than you see on the NewRelic's non-heap memory usage graph.
E.g. there are also Thread stacks which can occupy up to 1MB per thread.
There's a JVM feature called Native Memory Tracking that you can use to track some of the non-heap memory usage.
There can still be native allocations that aren't tracked at all.
I suggest you look at these excellent resources from #apangin:
Java using much more memory than heap size (or size correctly Docker memory limit)
Memory footprint of a Java process by Andrei Pangin: https://www.youtube.com/watch?v=c755fFv1Rnk&list=PLRsbF2sD7JVqPgMvdC-bARnJ9bALLIM3Q&index=6

Difference between Resident Set Size (RSS) and Java total committed memory (NMT) for a JVM running in Docker container

Scenario:
I have a JVM running in a docker container. I did some memory analysis using two tools: 1) top 2) Java Native Memory Tracking. The numbers look confusing and I am trying to find whats causing the differences.
Question:
The RSS is reported as 1272MB for the Java process and the Total Java Memory is reported as 790.55 MB. How can I explain where did the rest of the memory 1272 - 790.55 = 481.44 MB go?
Why I want to keep this issue open even after looking at this question on SO:
I did see the answer and the explanation makes sense. However, after getting output from Java NMT and pmap -x , I am still not able to concretely map which java memory addresses are actually resident and physically mapped. I need some concrete explanation (with detailed steps) to find whats causing this difference between RSS and Java Total committed memory.
Top Output
Java NMT
Docker memory stats
Graphs
I have a docker container running for most than 48 hours. Now, when I see a graph which contains:
Total memory given to the docker container = 2 GB
Java Max Heap = 1 GB
Total committed (JVM) = always less than 800 MB
Heap Used (JVM) = always less than 200 MB
Non Heap Used (JVM) = always less than 100 MB.
RSS = around 1.1 GB.
So, whats eating the memory between 1.1 GB (RSS) and 800 MB (Java Total committed memory)?

You have some clue in "
Analyzing java memory usage in a Docker container" from Mikhail Krestjaninoff:
(And to be clear, in May 2019, three years later, the situation does improves with openJDK 8u212 )
Resident Set Size is the amount of physical memory currently allocated and used by a process (without swapped out pages). It includes the code, data and shared libraries (which are counted in every process which uses them)
Why does docker stats info differ from the ps data?
Answer for the first question is very simple - Docker has a bug (or a feature - depends on your mood): it includes file caches into the total memory usage info. So, we can just avoid this metric and use ps info about RSS.
Well, ok - but why is RSS higher than Xmx?
Theoretically, in case of a java application
RSS = Heap size + MetaSpace + OffHeap size
where OffHeap consists of thread stacks, direct buffers, mapped files (libraries and jars) and JVM code itse
Since JDK 1.8.40 we have Native Memory Tracker!
As you can see, I’ve already added -XX:NativeMemoryTracking=summary property to the JVM, so we can just invoke it from the command line:
docker exec my-app jcmd 1 VM.native_memory summary
(This is what the OP did)
Don’t worry about the “Unknown” section - seems that NMT is an immature tool and can’t deal with CMS GC (this section disappears when you use an another GC).
Keep in mind, that NMT displays “committed” memory, not "resident" (which you get through the ps command). In other words, a memory page can be committed without considering as a resident (until it directly accessed).
That means that NMT results for non-heap areas (heap is always preinitialized) might be bigger than RSS values.
(that is where "Why does a JVM report more committed memory than the linux process resident set size?" comes in)
As a result, despite the fact that we set the jvm heap limit to 256m, our application consumes 367M. The “other” 164M are mostly used for storing class metadata, compiled code, threads and GC data.
First three points are often constants for an application, so the only thing which increases with the heap size is GC data.
This dependency is linear, but the “k” coefficient (y = kx + b) is much less then 1.
More generally, this seems to be followed by issue 15020 which reports a similar issue since docker 1.7
I'm running a simple Scala (JVM) application which loads a lot of data into and out of memory.
I set the JVM to 8G heap (-Xmx8G). I have a machine with 132G memory, and it can't handle more than 7-8 containers because they grow well past the 8G limit I imposed on the JVM.
(docker stat was reported as misleading before, as it apparently includes file caches into the total memory usage info)
docker stat shows that each container itself is using much more memory than the JVM is supposed to be using. For instance:
CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O
dave-1 3.55% 10.61 GB/135.3 GB 7.85% 7.132 MB/959.9 MB
perf-1 3.63% 16.51 GB/135.3 GB 12.21% 30.71 MB/5.115 GB
It almost seems that the JVM is asking the OS for memory, which is allocated within the container, and the JVM is freeing memory as its GC runs, but the container doesn't release the memory back to the main OS. So... memory leak.

Disclaimer: I am not an expert
I had a production incident recently when under heavy load, pods had a big jump in RSS and Kubernetes killed the pods. There was no OOM error exception, but Linux stopped the process in the most hardcore way.
There was a big gap between RSS and total reserved space by JVM. Heap memory, native memory, threads, everything looked ok, however RSS was big.
It was found out that it is due to the fact how malloc works internally. There are big gaps in the memory where malloc takes chunks of memory from. If there are a lot of cores on your machine, malloc tries to adapt and give every core each own space to take free memory from to avoid resource contention. Setting up export MALLOC_ARENA_MAX=2 solved the issue. You can find more about this situation here:
Growing resident memory usage (RSS) of Java Process
https://devcenter.heroku.com/articles/tuning-glibc-memory-behavior
https://www.gnu.org/software/libc/manual/html_node/Malloc-Tunable-Parameters.html
https://github.com/jeffgriffith/native-jvm-leaks
P.S. I don't know why there was a jump in RSS memory. Pods are built on Spring Boot + Kafka.

Tomcat process killed by Linux kernel after running out of swap space; don't get any JVM OutOfMemory error

I was performing load testing against a tomcat server. The server has 10G physical memory and 2G swap space. The heap size (xms and xmx) was set to 3G before, and the server just worked fine. Since I still saw a lot free memory left and the performance was not good, I increased heap size to 7G and ran the load testing again. This time I observed physical memory was eaten up very quickly, and the system started consuming swap space. Later, tomcat crashed after running out of swap space. I included -XX:+HeapDumpOnOutOfMemoryError when starting tomcat, but I didn't get any heap dump. When I checked /var/log/messages, I saw kernel: Out of memory: Kill process 2259 (java) score 634 or sacrifice child.
To provide more info, here's what I saw from Linux top command when heap size set to 3G and 7G
xms&xmx = 3G (which worked fine):
Before starting tomcat:
Mem: 10129972k total, 1135388k used, 8994584k free, 19832k buffers
Swap: 2097144k total, 0k used, 2097144k free, 56008k cached
After starting tomcat:
Mem: 10129972k total, 3468208k used, 6661764k free, 21528k buffers
Swap: 2097144k total, 0k used, 2097144k free, 143428k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2257 tomcat 20 0 5991m 1.9g 19m S 352.9 19.2 3:09.64 java
After starting load for 10 min:
Mem: 10129972k total, 6354756k used, 3775216k free, 21960k buffers
Swap: 2097144k total, 0k used, 2097144k free, 144016k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2257 tomcat 20 0 6549m 3.3g 10m S 332.1 34.6 16:46.87 java
xms&xmx = 7G (which caused tomcat crash):
Before starting tomcat:
Mem: 10129972k total, 1270348k used, 8859624k free, 98504k buffers
Swap: 2097144k total, 0k used, 2097144k free, 74656k cached
After starting tomcat:
Mem: 10129972k total, 6415932k used, 3714040k free, 98816k buffers
Swap: 2097144k total, 0k used, 2097144k free, 144008k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2310 tomcat 20 0 9.9g 3.5g 10m S 0.3 36.1 3:01.66 java
After starting load for 10 min (right before tomcat was killed):
Mem: 10129972k total, 9960256k used, 169716k free, 164k buffers
Swap: 2097144k total, 2095056k used, 2088k free, 3284k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2310 tomcat 20 0 10.4g 5.3g 776 S 9.8 54.6 14:42.56 java
Java and JVM Version:
Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
Tomcat Version:
6.0.36
Linux Server:
Red Hat Enterprise Linux Server release 6.4 (Santiago)
So my questions are:
Why would this issue happen? When JVM runs out of memory why is there no OutOfMemoryError thrown? And why does it go straight to using swap?
Why top RES shows that java is using 5.3G memory, there's much more memory consumed?
I have been investigating and searching for a while, still cannot find the root cause for this issue. Thanks a lot!

Why would this issue happen? When JVM runs out of memory why is there no OutOfMemoryException thrown?
It is not the JVM that has run out of memory. It is the Host Operating System that has run out of memory-related resources, and is taking drastic action. The OS has no way of knowing that the process (in this case the JVM) is capable of shutting down in an orderly fashion when told "No" in response to a request for more memory. It HAS to hard-kill something or else there is a serious risk of the entire OS hanging.
Anyway, the reason you are not seeing OOMEs is that this is not an OOME situation. In reality, the JVM has already been given too much memory by the OS, and there is no way to take it back. That's the problem the OS has to deal with by hard-killing processes.
And why does it go straight to using swap?
It uses swap because the total virtual memory demand of the entire system won't fit in physical memory. This is NORMAL behaviour for a UNIX / Linux operating system.
Why top RES shows that java is using 5.3G memory, there's much more memory consumed
The RES numbers can be a little misleading. What they refer to is the amount of physical memory that the process is currently using ... excluding stuff that is shared or shareable with other processes. The VIRT number is more relevant to your problem. It says your JVM is using 10.4g of virtual ... which is more than the available physical memory on your system.
As the other answer says, it is concerning that it concerns you that you don't get an OOME. Even if you did get one, it would be unwise to do anything with it. An OOME is liable to do collateral damage to your application / container that is hard to detect and harder to recover from. That's why OOME is an Error not an Exception.
Recommendations:
Don't try to use significantly more virtual memory than you have physical memory, especially with Java. When a JVM is running a full garbage collection, it will touch most of its VM pages, multiple times in random order. If you have over-allocated your memory significantly this is liable to cause thrashing which kills performance for the entire system.
Do increase your system's swap space. (But that might not help ...)
Don't try to recover from OOMEs.

You probably have other processes on the same computer that also use memory. It looks like your java process reaches around 5.3GB before the machine is desperately out of RAM and swap. (Other processes are then probably using 12GB-5.3GB = 6.7GB) So your linux kernel sacrifices your java process to keep other processes running. The java memory limit is never reached so you're not getting an OutOfMemoryException.
Consider all the processes you need running on the entire machine, and adjust your Xmx setting accordingly (enough to leave room for all the other processes). Perhaps 5gb?
In any case, counting of OutOfMemoryExceptions being delivered is a pretty bad code smell. If I recall correctly, getting even a single OutOfMemoryException can leave the JVM in an "all-bets-are-off" state and should probably be restarted to not become unstable.

One night I left the system alone, the next morning it was fixed by itself! without any change. without even restarting.

Java high memory usage

I have a problem with a Java app. Yesterday, when i deployed it to have a test run, we noticed that our machine started swapping, even though this is not a real monster app, if you know what i mean.
Anyway, i checked the results of top and saw that it eats around 100mb of memory (RES in top) I tried to profile memory and check if there is a memory leak, but i couldn't find one. There was an unclosed PreparedStatement, which i fixed, but it didn't mean much.
I tried setting the min and max heap size (some said that min heap size is not required), and it didn't make any difference.
This is how i run it now:
#!/bin/sh
$JAVA_HOME/bin/java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9025 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -XX:MaxPermSize=40m -Xmx32M -cp ./jarName.jar uk.co.app.App app.properties
Here is the result of top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16703 root 20 0 316m 109m 6048 S 0.0 20.8 0:14.80 java
The thing i don't understand that i configure max PermSize and max Heap size, which add up to 72mb. Which is enough, the app runs well. Why is it eating 109mb of memory still and what is eating it up? It is a 37mb difference, which is quite high ratio. (34%).
I don't think this is a memory leak, because max heap size is set and there is no out of memory error, or anything.
One intertesting thing may be that i made a heap dump with VisualVM and then checked it with EclipseMAT and it said that there is a possible leak in a classloader.
This is what it says:
The classloader/component "sun.misc.Launcher$AppClassLoader #
0x87efa40" occupies 9,807,664 (64.90%) bytes. The memory is
accumulated in one instance of "short[][]" loaded by "".Keywords sun.misc.Launcher$AppClassLoader # 0x87efa40
I cannot make much of this, but may be useful.
Thanks for your help in advance.
EDIT
I found this one, maybe there is nothing i can do...
Tomcat memory consumption is more than heap + permgen space

Java's memory includes
the heap space.
the perm gen space
thread stack areas.
shared libraries, including that of the JVM (will be shared)
the direct memory size.
the memory mapped file size (will be shared)
There is likely to be others which are for internal use.
Given that 37 MB of PC memory is worth about 20 cents, I wouldn't worry about it too much. ;)

Did you try using JConsole to profile the application http://docs.oracle.com/javase/1.5.0/docs/guide/management/jconsole.html
Otherwise you can also use JProfiler trial version to profile the application
http://www.ej-technologies.com/products/jprofiler/overview.html?gclid=CKbk1p-Ym7ACFQ176wodgzy4YA
However first step to check high memory usage should be to check if you are using collection of objects in your application like array,map,set,list etc. If yes then check if they keep the references to objects (even though of not used) with them ?