We Keep Coding

How to interpret jeprof output

Recently I came across some java related memory leaks (continuously decreasing server-free memory and finally getting RAM warning which we have set up using nagios) and I did an investigation and found that the memory leak is not related to the heap ara. But still tomcat process's memory consumption keeps growing.
server memory graph - 7 days
Did a heap memory analysis and nothing found in there ( if I run jcmd <pid> GC.run heap memory usage drops to around 200MB from 2.8GB). heap memory graph - 7 days
Checked metaspace area and other memory areas related to the JVM as per the discussion on this video and post.
https://www.youtube.com/watch?t=2483&v=c755fFv1Rnk&feature=youtu.be
https://github.com/jeffgriffith/native-jvm-leaks/blob/master/README.md
Finally, I added jemalloc to profile native memory allocation, and here is some of the output that I got.
ouptput 1
ouptput 2
But I couldn't interpret this output and I'm not sure whether this output is correct or not.
And also I have a doubt regarding whether that jeprof is working with oracle JDK.
Could you please help me on this?
Additional info:
server memory: 4GB
Xmx: 3072M (recently we changed to this and earlier it was 2048M. but the memory behavior is similar on both occasions)
Xms: 3072M (recently we changed to this and earlier it was 2048M. but the memory behavior is similar on both occasions)
javac -version: jdk1.8.0_72
java version:
"1.8.0_72"
Java(TM) SE Runtime Environment (build 1.8.0_72-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.72-b15, mixed mode)
jemelloc configs:
jemelloc version: https://github.com/jemalloc/jemalloc/releases/download/5.2.1/jemalloc-5.2.1.tar.bz2
export LD_PRELOAD=/usr/local/lib/libjemalloc.so
export MALLOC_CONF=prof:true,lg_prof_interval:31,lg_prof_sample:17,prof_prefix:/opt/jemalloc/jeprof-output/jeprof
My application is running on a tomcat server in an ec2 instance (only one application running on that server).

Maximum heap size using for Java process in Windows 10 64 bit running 64 bit JVM

What is maximum Heap size for Java process running on Windows 10 64 bits, with 64 bits JVM? My machine has 8 GB of RAM. And I am running Java 8.
I trying to run BFS on huge graph for experimental purposes. While running BFS I am monitoring Heap size being used in Java Visual VM. According to Visual VM heap utilization is always less than 2000 MB regardless of providing following JVM parameters
-Xms2048m
-Xmx3072m
-XX:ReservedCodeCacheSize=240m
-XX:+UseConcMarkSweepGC
-XX:SoftRefLRUPolicyMSPerMB=50
-ea
-Dsun.io.useCanonCaches=false
-Djava.net.preferIPv4Stack=true
-XX:+HeapDumpOnOutOfMemoryError
-XX:-OmitStackTraceInFastThrow
I did some research over internet but could not find any specific answer related to the system specification I am using. Can a java process use more than 2 GB on Windows 10 64 bit and 64 bit JVM? As Guidelines for Java Heap sizing the limit for Windows XP/2008/7 is 2 GB.

On a 64-bit machine, with 64-bit JVM you can work with multi gigabyte heaps (dozens and dozens of GBs). I'm not sure it's limited in any way except by the available memory (and the theoretical address space of a 64-bit pointer).
Of course if you're working with a huge heap, the GC has a lot more work to do and you may find that you need to scale horizontally instead of vertically, to maintain a good performance.
If VisualVM isn't showing you using more than 2GB (the initial heap size given with -Xms), then it probably just doesn't need more than that. You've given the permission to use up to 3GB (-Xmx), but the JVM won't allocate more memory just for the fun of it.

Maximum Heap can be allocated for 32bit JVM is 2^32 = 4G, Again 4gb will be devided into 1+ GB for VM to use for runtime classes. It varies windows it is ~2GB and linux it is ~3GB.
As you are using 64bit machine maximum heap available is 2^64 it will be big enough for you to run BFS easily.
You can monitor the available memory using vm flags "-XX+PrintFlagsFinal | grep -iE HeapSize" will tell you the maximum available heap size that can be used. Configure slightly less than that and start using...

There is no definite size you could specify for 64 bit architecture but simple test helps you find what is the maximum contiguous space available or could be allocated for a process. This could be tested as follow by using simple command.
Try as below
java -Xmx -version
If the above command gives result then your system could be allowed to have Xmx to that level, If it fails then you can't specify that value.
Few test from system.
I tested the value with 20G.40g,100G,160G,300G all these gave java -version output but tried with 1600G that throws the error.
Output of the test
C:\Users\mpalanis>java -Xmx300G -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
C:\Users\mpalanis>java -Xmx1600G -version
Error occurred during initialization of VM
Unable to allocate 52431424KB bitmaps for parallel garbage collection for the requested 1677805568KB heap.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Hope this explanation helps.

If you are using IntelliJ Idea as an IDE you can do this directly from it,
From the main menu, select Help | Change Memory Settings
Set the necessary amount of memory that you want to allocate and click Save and Restart.
This changes the value of the -Xmx option used by the JVM and restarts IntelliJ IDEA with the new setting.

Setting Java Heap Size does not work with 256GB RAM 64-bit machine

I have been trying to find the answer to this but I still could not.
I have a 64-bit machine with 256 GB RAM.
I am trying to execute a Java program which links to MySQL. And it needs a quiet big heap size because when I used VM argument -Xmx1024m after few minutes this pops up:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
But, when I tried to set the program VM argument by -Xmx2048m or bigger, it does not work and said:
Error occurred during initialization of VM
Could not reserve enough space for 2097152KB object heap
I read that setting -Xmx2048 is not a problem for 64-bit machine but I really do not know why it's not working on my machine.
java -version output:
wmic OS get FreePhysicalMemory /Value output:
FreePhysicalMemory=251663664
wmic computersystem get TotalPhysicalMemory output:
TotalPhysicalMemory
274811621376
wmic os get osarchitecture output
OSArchitecture
64-bit
I could not execute systeminfo|find "Memory" as it says wrong syntax. Im not sure why either.

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
This message implies the java heap does not have enough space to do further allocation.
It seems you have set to 1G of Xmx which is insufficient for your application
else the application might leak memory which occupies space in java heap.
Error occurred during initialization of VM
Could not reserve enough space for 2097152KB object heap
During VM initialization. JVM does various init right from Heap to JIT modules.
Java heap are continiuos space while doing doing initialization if JVM could not find the
contigous memory space for the requested Xmx then above error message will be thrown.
These are virtual memory allocation so please do not confuse with physical memory values
Please run the command
java -verbose:init -Xmx2048 -version
this command will tell the steps undergoes by JVM while doing initialization and during what step it got failed.

Consider running your app as an administrator - just launch the PowerShell console via right click -> Run as Administrator.
Windows may deny granting a lot of memory to a single process that is running non-elevated.
Also - are you using Windows Server or a workstation version of Windows? It is generally recommended to use Windows Server for such "big" services.

Glassfish PermGen Not Collecting

Today after many days of running without issue, my glassfish application started throwing OOM: PermGen messages. After a restart it worked for about an hour and then failed again for the same reason. When I attach jconsole or visual vm to the instance, I notice that PermGen allocation grows and never is collected. If I force a GC, the PermGen memory is collected correctly and it goes back to the same level repeatedly. If I leave the GC alone, it never occurs and PermGen repeatedly grows to the max and then crashes. Why would this happen, and why would this happen out of the blue?
java version "1.7.0_51"
OpenJDK Runtime Environment (IcedTea 2.4.4) (7u51-2.4.4-0ubuntu0.12.04.2)
OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)

Why doesn't it look like -Xms and -Xmx are doing anything to Java's memory usage in Windows?

I am running this version of Java:
java version "1.7.0_07"
Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
I am running this on a Windows 2008 R2 64-bit server on Amazon EC2, in an m1.large instance that has 7.5GB of memory.
When I start my Java app, I am using this command line:
java -Xms6G -Xmx6G -server -jar start.jar
My intent is to have Java reserve 6GB for it's heap, so that when my application runs, it will be able to load its entire dataset into memory.
However, when I start the app, I only see 1.3GB of memory used in Task Manager?

The issue is you're looking at the default memory column shown in Task Manager, which is "Memory - Private Working Set." This doesn't reflect what is actually reserved for use.
In Task Manager, go to the View menu and choose Select Columns. Add the "Memory - Commit Size" column. You should see this column reflects the reserved heap size from Java. In my tests, it shows around 6.6GB committed for a commandline of -Xms6G.
Here is Microsoft's page that explains what each column means.
"Memory - Private Working Set: Subset of working set that specifically describes the amount of memory a process is using that cannot be shared by other processes."
"Memory - Commit Size: Amount of virtual memory that is reserved for use by a process."