We Keep Coding

How to interpret jeprof output

Recently I came across some java related memory leaks (continuously decreasing server-free memory and finally getting RAM warning which we have set up using nagios) and I did an investigation and found that the memory leak is not related to the heap ara. But still tomcat process's memory consumption keeps growing.
server memory graph - 7 days
Did a heap memory analysis and nothing found in there ( if I run jcmd <pid> GC.run heap memory usage drops to around 200MB from 2.8GB). heap memory graph - 7 days
Checked metaspace area and other memory areas related to the JVM as per the discussion on this video and post.
https://www.youtube.com/watch?t=2483&v=c755fFv1Rnk&feature=youtu.be
https://github.com/jeffgriffith/native-jvm-leaks/blob/master/README.md
Finally, I added jemalloc to profile native memory allocation, and here is some of the output that I got.
ouptput 1
ouptput 2
But I couldn't interpret this output and I'm not sure whether this output is correct or not.
And also I have a doubt regarding whether that jeprof is working with oracle JDK.
Could you please help me on this?
Additional info:
server memory: 4GB
Xmx: 3072M (recently we changed to this and earlier it was 2048M. but the memory behavior is similar on both occasions)
Xms: 3072M (recently we changed to this and earlier it was 2048M. but the memory behavior is similar on both occasions)
javac -version: jdk1.8.0_72
java version:
"1.8.0_72"
Java(TM) SE Runtime Environment (build 1.8.0_72-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.72-b15, mixed mode)
jemelloc configs:
jemelloc version: https://github.com/jemalloc/jemalloc/releases/download/5.2.1/jemalloc-5.2.1.tar.bz2
export LD_PRELOAD=/usr/local/lib/libjemalloc.so
export MALLOC_CONF=prof:true,lg_prof_interval:31,lg_prof_sample:17,prof_prefix:/opt/jemalloc/jeprof-output/jeprof
My application is running on a tomcat server in an ec2 instance (only one application running on that server).

Installed Elasticsearch 5.5 and failed to start the service

Installed elasticsearch v5.5 in centos and ran the following command to initiate the service.
sudo service elasticsearch start
Getting following error while running the above command.
Starting elasticsearch: OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000085330000, 2060255232, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 2060255232 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid15359.log
Suggest me how to fix this.

Elasticsearch starts with 2 GB of RAM as default in 5.X versions.
Assuming that you are using virtual machine, it seems like your VM has less free memory than 2GB. Try giving your VM more memory or change your Elasticsearch JVM settings in /etc/elasticsearch/jvm.options (for example set -Xms512m -Xmx512m).

Jenkins: Out of memory issue

Suddenly I have started getting following error from integration test cases. Using Java 8 so I added MAVEN_OPTS = -Xmx512m. But it did not work. What am I missing here and how can I fix it? Between it works fine on local machine.
SUREFIRE-859: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c9800000, 54001664, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 54001664 bytes for committing reserved memory.
# An error report file with more information is saved as:

Looking at the error message, it looks like Java was not able to allocate enough memory, i.e. it's not Java's heap limit that's in the way but rather no more memory available to be given to Java by OS. Check that the machine is not running out of memory.

'Cannot allocate memory' (errno=12)' errors during runtime of Java application

While running a distributed application, I get a lot of these errors on the server as well as on the worker nodes:
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f4f8c800000, 549453824, 2097152, 0) failed; error='Cannot allocate memory' (errno=12)
Most of the time the process continues and finishes as expected but sometimes the process also fails.
I am calling my application with java -Xms512M -Xmx50G -cp myjar.jar myclass.Main
The nodes have 128 GBs of RAM where about 120 GBs are free.
I'm using the Oracle JVM:
$ java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
What do these messages mean and how can I get rid of them?

As Platypus suggested in the comments to my question, I downgraded Java to version 1.7.0_41. Unfortunately the problem persisted.
I went even farther back to version 1.7.0_25 and apparently this solved the error. I tried it many times and the error message didn't occur a single more time.

JVM error when try to allocate more than 128M Xms, without specifying Xmx

I am seeing an JVM issue when I am running my application and I simply to below java commands:
C:\Users\optitest>I:\j2sdk\bin\java -version
java version "1.6.0_17"
Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode)
C:\Users\optitest>I:\j2sdk\bin\java -Xms4g -version
Error occurred during initialization of VM
Incompatible minimum and maximum heap sizes specified
Even Xms is set to 128M does not work:
C:\Users\optitest>I:\j2sdk\bin\java -Xms128m -version
Error occurred during initialization of VM
Incompatible minimum and maximum heap sizes specified
Works only when Xms is set to 64M or less:
C:\Users\optitest>I:\j2sdk\bin\java -Xms64m -version
java version "1.6.0_17"
Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode)
The interesting thing is if I specify Xmx, then it works well.
C:\Users\optitest>I:\j2sdk\bin\java -Xms4g -Xmx4g-version
java version "1.6.0_17"
Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode)
C:\Users\optitest>I:\j2sdk\bin\java -Xms4g -Xmx8g-version
java version "1.6.0_17"
Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode)
More interesting thing is: All above commands run well on another machine with same OS (Windows Server 2008 R2 Enterprise SP1) & same jdk version. Physical Memory is 16GB.
Any idea?

Any idea?
Well the obvious conclusion is that if you want to use -Xms to specify the initial heap size, and you want to set the size to a value that is larger than the default maximum heap size, you need to specify a maximum heap size.
The reason that you are getting different results on different machines is that the JVM computes the default heap size in different ways, depending on the version of Java and on the execution platform. In some cases it is a constant. In others, it depends on the amount of physical memory on the system.
Just set the maximum heap size explicitly and you won't have this problem.
If you want to find out what the default heap size is for a given machine, run this command:
java -XX:+PrintFlagsFinal -version

The heap size of your machine depends on lot more than how much RAM you got!
Maximum heap size for 32 bit or 64 bit JVM looks easy to determine by
looking at addressable memory space like 2^32 (4GB) for 32 bit JVM and
2^64 for 64 bit JVM.
You can not really set 4GB as maximum heap size for 32 bit JVM using
-Xmx JVM heap options. You will get could not create the Java virtual machine Invalid maximum heap size: -Xmx error.
You can look here for a well explained document about the heap size.
Another important thing is that, You can only postpone the OutofMemory Exception by in creasing the Heap size. Unless your clean up your memory you will get the exception one time or another Use the applications like Visual VM to understand what's going on in the background. I suggest you try to Optimise code, for increasing performance.

I have this same issue. I'm still debugging it right now, but it appears that it might have something to do with the default MaxHeapSize being set to TOTALRAM/4 or (16GB/4 = 4GB = 2^32):
(uint32) 2^32 = 0
I get the following output from -XX:PrintFlagsFinal:
uintx MaxHeapSize := 0 {product}
And also the output of -XX:+PrintCommandLineFlags confirms the 4G value:
-XX:InitialHeapSize=268428160 -XX:MaxHeapSize=4294850560 -XX:+PrintCommandLineFlags -XX:+UseCompressedOops -XX:-UseLargePagesIndividualAllocation -XX:+UseParallelGC