We Keep Coding

Kubernetes and JVM memory settings

In a Kubernetes cluster with numerous microservices, one of them is used exclusively for a Java Virtual Machine (JVM) that runs a Java 1.8 data processing application.
Up to recently, jobs running in that JVM pod consumed less than 1 GB of RAM, so the pod has been setup with 4 GB of maximum memory, without any explicit heap size settings for the JVM.
Some new data now require about 2.5 GB for the entire pod, including the JVM (as reported by the kubernetes top command, after launching with an increased memory limit of 8 GB), but the pod crashes soon after starting with a limit of 4 GB.
Using a head size range like -Xms256m -Xmx3072m with a limit of 4 GB does not solve the problem. In fact, now the pod does not even start.
Is there any way to parameterize the JVM for accommodating the 2.5 GB needed, without increasing the 4 GB maximum memory for the pod?

The default "max heap" if you don't specify -Xmx is 1/4 (25%) of the host RAM.
JDK 10 improved support for containers in that it uses container's RAM limits instead of underlying host. As pointed by #David Maze this has been backported to JDK 8.
Assuming you have a sufficiently recent version of JDK 8, you can use -XX:MaxRAMPercentage to modify the default percentage of total RAM used for Max heap. So instead of specifying -Xmx you can tell, e.g. -XX:MaxRAMPercentage=75.0. See also https://medium.com/adorsys/usecontainersupport-to-the-rescue-e77d6cfea712
Here's an example using alpine JDK docker image: https://hub.docker.com/_/openjdk (see section "Make JVM respect CPU and RAM limits" in particular).
# this is running on the host with 2 GB RAM
docker run --mount type=bind,source="$(pwd)",target=/pwd -it openjdk:8
# running with MaxRAMPercentage=50 => half of the available RAM is used as "max heap"
root#c9b0b4d9e85b:/# java -XX:+PrintFlagsFinal -XX:MaxRAMPercentage=50.0 -version | grep -i maxheap
uintx MaxHeapFreeRatio = 100 {manageable}
uintx MaxHeapSize := 1044381696 {product}
openjdk version "1.8.0_265"
OpenJDK Runtime Environment (build 1.8.0_265-b01)
OpenJDK 64-Bit Server VM (build 25.265-b01, mixed mode)
# running without MaxRAMPercentage => default 25% of RAM is used
root#c9b0b4d9e85b:/# java -XX:+PrintFlagsFinal -version | grep -i maxheap
uintx MaxHeapFreeRatio = 100 {manageable}
uintx MaxHeapSize := 522190848 {product}
openjdk version "1.8.0_265"

In my K8s setup, I am using consul to manage the pod configuration. Here is a command to override the jvm setting on the fly. It is a pretty much project specific but it might give you a hint if you are using consul for configuration.
kubectl -n <namespace> exec -it consul-server -- bash -c "export CONSUL_HTTP_ADDR=https://localhost:8500 && /opt/../home/bin/bootstrap-config --token-file /opt/../config/etc/SecurityCertificateFramework/tokens/consul/default/management.token kv write config/processFlow/jvm/java_option_xmx -Xmx8192m"

OpenJDK 1.8.0_242, MaxRAMFraction setting not reflecting

I am running a Springboot application in the alpine-OpenJDK image and facing OutOfMemory issues. Max heap is being capped at 256MB. I tried updating the MaxRAMFraction setting to 1 but did not see it getting reflected in the Java_process. I have an option to increase the container memory limit to 3000m but would prefer to use Cgroup memory with MaxRamfraction=1. Any thoughts?
Java-Version
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (IcedTea 3.15.0) (Alpine 8.242.08-r0)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
bash-5.0$ java -XX:+PrintFlagsFinal -version | grep -Ei "maxheapsize|MaxRAMFraction"
uintx DefaultMaxRAMFraction = 4 {product}
uintx MaxHeapSize := 262144000 {product}
uintx MaxRAMFraction = 4 {product}
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (IcedTea 3.15.0) (Alpine 8.242.08-r0)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
Container Resource limits
ports:
- containerPort: 8080
name: 8080tcp02
protocol: TCP
resources:
limits:
cpu: 350m
memory: 1000Mi
requests:
cpu: 50m
memory: 1000Mi
securityContext:
capabilities: {}
Container JAVA_OPTS screenshot

Max heap is being capped at 256MB.
You mean via -m in docker? If such, this is not the java heap you are specifying, but the total memory.
I tried updating the MaxRAMFraction setting to 1
MaxRAMFraction is deprecated and un-used, forget about it.
UseCGroupMemoryLimitForHeap
is deprecated and will be removed. Use UseContainerSupport that was ported to java-8 also.
MaxRAM=2g
Do you know what this actually does? It sets the value for the "physical" RAM that the JVM is supposed to think you have.
I assume that you did not set -Xms and -Xmx on purpose here? Since you do not know how much memory the container will have? If such, we are in the same shoes. We do know that the min we are going to get is 1g, but I have no idea of the max, as such I prefer not to set -Xms and -Xmx explicitly.
Instead, we do:
-XX:InitialRAMPercentage=70
-XX:MaxRAMPercentage=70
-XX:+UseContainerSupport
-XX:InitialHeapSize=0
And that's it. What this does?
InitialRAMPercentage is used to calculate the initial heap size, BUT only when InitialHeapSize/Xms are missing. MaxRAMPercentage is used to calculate the maximum heap. Do not forget that a java process needs more than just heap, it needs native structures also; that is why that 70 (%).

What is effective Xmx value on my redhat machine running tomcat

I am running tomcat on RHEL 7 machine with 1GB RAM. I have setup tomcat and java both to have Xmx=1G and below statements support that,
[root#ip-172-31-28-199 bin]# java -XX:+PrintFlagsFinal -version | grep
HeapSize Picked up _JAVA_OPTIONS: -Xmx1g
uintx ErgoHeapSizeLimit = 0 {product}
uintx HeapSizePerGCThread = 87241520 {product}
uintx InitialHeapSize := 16777216 {product}
uintx LargePageHeapSizeThreshold = 134217728 {product}
uintx MaxHeapSize := 1073741824 {product} openjdk version "1.8.0_161"
and
tomcat 2799 1 1 02:21 ? 00:00:07 /usr/bin/java
-Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Xmx1024M -Dignore.endorsed.dirs= -classpath /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar
-Dcatalina.base=/opt/tomcat -Dcatalina.home=/opt/tomcat -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start
But when I get exception, I get following message,
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 244043776 bytes for committing reserved memory.
I know java can never claim 1GB memory as that is the total memory of the machine. but why I am getting error with this size mentioned?

Try adding -Xms1g too, so it initially allocates all the memory, and you'll find that it cannot even start Tomcat.
If you want to squeeze as much memory into Tomcat as possible (not recommended), slowly reduce both numbers (same value for mx and ms) until Tomcat starts.
That is the absolute maximum you can give Tomcat, but you shouldn't do that. Java may still need more as it runs, and the OS will need more occasionally, so you should give Tomcat less than that absolute maximum.
Now that you've found the number, you can leave -Xms undefined again, if you want to.

There is an interesting post here that suggests disabling OOPS.
Are you running a physical server or a VM ?
I agree a 1G server is under-sized, you should run Xmx=512M and allow some swappiness (vm.swappiness = 60 is the default, which should be OK for a small Tomcat)

When i was allocate new memory in heap in J meter ,Memory Leak was happen

My question is how should i release memory from the heap?
"My c drive had more than 40 GB space but now it showing less than 3 GB"
C:\Windows\system32>java -XX:+PrintFlagsFinal -version | findstr /i "HeapSize PermSize ThreadStackSize"
intx CompilerThreadStackSize = 0
{pd product}
uintx ErgoHeapSizeLimit = 0
{product}
uintx HeapSizePerGCThread = 87241520
{product}
uintx InitialHeapSize := 62914560
{product}
uintx LargePageHeapSizeThreshold = 134217728
{product}
uintx MaxHeapSize := 987758592
{product}
intx ThreadStackSize = 0
{pd product}
intx VMThreadStackSize = 0
{pd product}
java version "1.8.0_77"
Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
C:\Windows\system32>
java -XX:+PrintFlagsFinal -Xms512m -Xmx1024m -Xss512k -XX:PermSize=64m -XX:MaxPermSize=128m -version | grep -iE 'HeapSize|PermSize|ThreadStackSize'
when after allocating memory in heap i was run program using jmeter each and every time its allocate space in c drive..#Peter Lawrey

When your disk space fills up you need to find the files you don't need and clean them up.
If you have produced a heap dump I suggest you look for a large file in the directory where your program ran.
BTW When a program exits, all the resource it used are freed up except any files it leaves behind.

Use TreeSize Free tools and open it with Administrator and search java_.hprof files fie from there and go to directory and delete those file from there!!!

Increasing Heap Size on Linux Machines

I work on Ubuntu desktop machine and I'd like to increase heap size for Java. The RAM is 16GB and the current Max Heap Size is 3GB
I checked this post post Increasing Tomcat Heap Size
Not much found about Ubuntu, so I tried this command:
java -Xmx10000m -X2000m -XshowSettings:all
and the result is:
Min Heap Size: 1.95G
Max Heap Size: 9.77G
then sudo gedit /etc/tomcat7/default
and changed this lino to:
JAVA_OPTS="-Djava.awt.headless=true -Xmx10000m -XX:MaxPermSize=2000m"
but then I restarted the machine an checked the max size using:
java -XshowSettings:all
and this shows:
Max Heap Size (Estimated): 3.80GB
I wanted to take advantage of the high RAM i got (16 GB). Is there anything else i can do?

Changing Tomcat config wont effect all JVM instances to get theses settings.
This is not how it works, the setting will be used only to launch JVMs used by Tomcat, not started in the shell.
Look here for permanently changing the heap size.

You can use the following code snippet :
java -XX:+PrintFlagsFinal -Xms512m -Xmx1024m -Xss512k -XX:PermSize=64m -XX:MaxPermSize=128m
-version | grep -iE 'HeapSize|PermSize|ThreadStackSize'
In my pc I am getting following output :
uintx InitialHeapSize := 536870912 {product}
uintx MaxHeapSize := 1073741824 {product}
uintx PermSize := 67108864 {pd product}
uintx MaxPermSize := 134217728 {pd product}
intx ThreadStackSize := 512 {pd product}