We Keep Coding

How to interpret jeprof output

Recently I came across some java related memory leaks (continuously decreasing server-free memory and finally getting RAM warning which we have set up using nagios) and I did an investigation and found that the memory leak is not related to the heap ara. But still tomcat process's memory consumption keeps growing.
server memory graph - 7 days
Did a heap memory analysis and nothing found in there ( if I run jcmd <pid> GC.run heap memory usage drops to around 200MB from 2.8GB). heap memory graph - 7 days
Checked metaspace area and other memory areas related to the JVM as per the discussion on this video and post.
https://www.youtube.com/watch?t=2483&v=c755fFv1Rnk&feature=youtu.be
https://github.com/jeffgriffith/native-jvm-leaks/blob/master/README.md
Finally, I added jemalloc to profile native memory allocation, and here is some of the output that I got.
ouptput 1
ouptput 2
But I couldn't interpret this output and I'm not sure whether this output is correct or not.
And also I have a doubt regarding whether that jeprof is working with oracle JDK.
Could you please help me on this?
Additional info:
server memory: 4GB
Xmx: 3072M (recently we changed to this and earlier it was 2048M. but the memory behavior is similar on both occasions)
Xms: 3072M (recently we changed to this and earlier it was 2048M. but the memory behavior is similar on both occasions)
javac -version: jdk1.8.0_72
java version:
"1.8.0_72"
Java(TM) SE Runtime Environment (build 1.8.0_72-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.72-b15, mixed mode)
jemelloc configs:
jemelloc version: https://github.com/jemalloc/jemalloc/releases/download/5.2.1/jemalloc-5.2.1.tar.bz2
export LD_PRELOAD=/usr/local/lib/libjemalloc.so
export MALLOC_CONF=prof:true,lg_prof_interval:31,lg_prof_sample:17,prof_prefix:/opt/jemalloc/jeprof-output/jeprof
My application is running on a tomcat server in an ec2 instance (only one application running on that server).

Maximum heap size using for Java process in Windows 10 64 bit running 64 bit JVM

What is maximum Heap size for Java process running on Windows 10 64 bits, with 64 bits JVM? My machine has 8 GB of RAM. And I am running Java 8.
I trying to run BFS on huge graph for experimental purposes. While running BFS I am monitoring Heap size being used in Java Visual VM. According to Visual VM heap utilization is always less than 2000 MB regardless of providing following JVM parameters
-Xms2048m
-Xmx3072m
-XX:ReservedCodeCacheSize=240m
-XX:+UseConcMarkSweepGC
-XX:SoftRefLRUPolicyMSPerMB=50
-ea
-Dsun.io.useCanonCaches=false
-Djava.net.preferIPv4Stack=true
-XX:+HeapDumpOnOutOfMemoryError
-XX:-OmitStackTraceInFastThrow
I did some research over internet but could not find any specific answer related to the system specification I am using. Can a java process use more than 2 GB on Windows 10 64 bit and 64 bit JVM? As Guidelines for Java Heap sizing the limit for Windows XP/2008/7 is 2 GB.

On a 64-bit machine, with 64-bit JVM you can work with multi gigabyte heaps (dozens and dozens of GBs). I'm not sure it's limited in any way except by the available memory (and the theoretical address space of a 64-bit pointer).
Of course if you're working with a huge heap, the GC has a lot more work to do and you may find that you need to scale horizontally instead of vertically, to maintain a good performance.
If VisualVM isn't showing you using more than 2GB (the initial heap size given with -Xms), then it probably just doesn't need more than that. You've given the permission to use up to 3GB (-Xmx), but the JVM won't allocate more memory just for the fun of it.

Maximum Heap can be allocated for 32bit JVM is 2^32 = 4G, Again 4gb will be devided into 1+ GB for VM to use for runtime classes. It varies windows it is ~2GB and linux it is ~3GB.
As you are using 64bit machine maximum heap available is 2^64 it will be big enough for you to run BFS easily.
You can monitor the available memory using vm flags "-XX+PrintFlagsFinal | grep -iE HeapSize" will tell you the maximum available heap size that can be used. Configure slightly less than that and start using...

There is no definite size you could specify for 64 bit architecture but simple test helps you find what is the maximum contiguous space available or could be allocated for a process. This could be tested as follow by using simple command.
Try as below
java -Xmx -version
If the above command gives result then your system could be allowed to have Xmx to that level, If it fails then you can't specify that value.
Few test from system.
I tested the value with 20G.40g,100G,160G,300G all these gave java -version output but tried with 1600G that throws the error.
Output of the test
C:\Users\mpalanis>java -Xmx300G -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
C:\Users\mpalanis>java -Xmx1600G -version
Error occurred during initialization of VM
Unable to allocate 52431424KB bitmaps for parallel garbage collection for the requested 1677805568KB heap.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Hope this explanation helps.

If you are using IntelliJ Idea as an IDE you can do this directly from it,
From the main menu, select Help | Change Memory Settings
Set the necessary amount of memory that you want to allocate and click Save and Restart.
This changes the value of the -Xmx option used by the JVM and restarts IntelliJ IDEA with the new setting.

Setting Java Heap Size does not work with 256GB RAM 64-bit machine

I have been trying to find the answer to this but I still could not.
I have a 64-bit machine with 256 GB RAM.
I am trying to execute a Java program which links to MySQL. And it needs a quiet big heap size because when I used VM argument -Xmx1024m after few minutes this pops up:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
But, when I tried to set the program VM argument by -Xmx2048m or bigger, it does not work and said:
Error occurred during initialization of VM
Could not reserve enough space for 2097152KB object heap
I read that setting -Xmx2048 is not a problem for 64-bit machine but I really do not know why it's not working on my machine.
java -version output:
wmic OS get FreePhysicalMemory /Value output:
FreePhysicalMemory=251663664
wmic computersystem get TotalPhysicalMemory output:
TotalPhysicalMemory
274811621376
wmic os get osarchitecture output
OSArchitecture
64-bit
I could not execute systeminfo|find "Memory" as it says wrong syntax. Im not sure why either.

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
This message implies the java heap does not have enough space to do further allocation.
It seems you have set to 1G of Xmx which is insufficient for your application
else the application might leak memory which occupies space in java heap.
Error occurred during initialization of VM
Could not reserve enough space for 2097152KB object heap
During VM initialization. JVM does various init right from Heap to JIT modules.
Java heap are continiuos space while doing doing initialization if JVM could not find the
contigous memory space for the requested Xmx then above error message will be thrown.
These are virtual memory allocation so please do not confuse with physical memory values
Please run the command
java -verbose:init -Xmx2048 -version
this command will tell the steps undergoes by JVM while doing initialization and during what step it got failed.

Consider running your app as an administrator - just launch the PowerShell console via right click -> Run as Administrator.
Windows may deny granting a lot of memory to a single process that is running non-elevated.
Also - are you using Windows Server or a workstation version of Windows? It is generally recommended to use Windows Server for such "big" services.

What in Java is using 400M in virtual memory and how do I lower that usage?

Simple program:
public class SleepTest {
public static void main(String[] args) throws InterruptedException {
Thread.sleep(60 * 1000);
}
}
Then
$ javac SleepTest.java
$ java -cp . SleepTest
For OpenJDK 1.6.0_20 this uses 600M of virtual memory on my machine! That is, "top" shows "VIRT" 600M and RES 10m. (I am on Ubuntu 10.04, 32-bit or 64-bit).
For Sun's Java 1.6.0_22 it uses 400M of virtual memory.
What is using all that virtual memory, and how do I lower that usage?
Full "java -version":
OpenJDK:
java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.4) (6b20-1.9.4-0ubuntu1~10.04.1)
OpenJDK Client VM (build 19.0-b09, mixed mode, sharing)
Sun:
$ /usr/lib/jvm/java-6-sun-1.6.0.22/jre/bin/jav -version
java version "1.6.0_22"
Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
Java HotSpot(TM) Client VM (build 17.1-b03, mixed mode, sharing)
Edit:
Compiling with javac from either package doesn't seem to help.
Adding some code to print used memory is as follows:
private static String megabyteString(long bytes) {
return String.format("%.1f", ((float)bytes) / 1024 / 1024);
}
private static void printUsedMemory() {
Runtime run = Runtime.getRuntime();
long free = run.freeMemory();
long total = run.totalMemory();
long max = run.maxMemory();
long used = total - free;
System.out.println("Memory: used " + megabyteString(used) + "M"
+ " free " + megabyteString(free) + "M"
+ " total " + megabyteString(total) + "M"
+ " max " + megabyteString(max) + "M");
}
shows
Sun:
Memory: used 0.3M free 15.2M total 15.5M max 247.5M
OpenJDK:
Memory: used 0.2M free 15.3M total 15.5M max 494.9M
even with -Xmx5m, so it must have a minimum? I've read about defaults before (depends on jvm, virtual machine, a common strategy by default one quarter of physical memory), but is that causing the large virtual memory use and can I not decrease it?
Edit #2:
Adding -Xmx changes things:
$ java -Xmx5m -cp . SleepTest
OpenJDK:
$ java -Xmx5m SleepTest
Memory: used 0.2M free 4.7M total 4.9M max 5.8M
uses "only" 150M of virtual memory for either JVM.
Edit #3:
nos, bestsss, Mikaveli, and maybe others pointed out that virtual memory does not use swap. nos claims the OOM killer is smart enough to go by real memory usage. If those things are true, then I guess I don't care about virtual memory usage. RES (resident size) is small, so I'm good.
Edit #4:
Not sure which answer to accept. Either of these, if it shows up as an answer: "Don't worry about it because virtual memory is cheap" or some explanation of why Java reserves at least 150M in virtual memory no matter what -Xmx or -Xms I give it, even though real memory usage is tiny.
Edit #5:
This is a dup. I voted to close.

"RES" is resident set size - essentially the physical memory you're using for that process.
From your example, top states that it's using 10 MiB - that's roughly what I'd expect for a -Xmx setting of 5m (the total physical memory used often seems to be double, from my experience of Java on *nix systems.
Are you actually getting any memory issues or are you just concerned about the misleading output from "top"?
Also, the *nix virtual memory includes of the available memory space - physical and swap. If the process isn't using any "swap", then it is only using physical "resident" memory.
Stack Overflow answer to why JVM uses more memory than just heap setting.

$ java -cp . SleepTest -Xmx5m
wow! the -Xmx5m is an argument you can find at public static void main(String[] a). It is parameter to the java program NOT to the VM
move it 1st
$ java -Xmx5m -cp . SleepTest

Try using -Xms to specify minimum memory?

The JVM takes some defaults for heap space (that is all that memory usage) in count. This defaults are coming from the machine caracteristics (RAM, cores, etc) and the JVM implementation (Sun/Oracle, OpenJDK).
I think -Xmx5m is too low and the JVM ignores this parameter silently.
Here is a good reference with lots of links to read:
Java Memory explained (SUN JVM)

Maximum amount of memory per Java process on Windows?

What is the maximum heap size that you can allocate on 32-bit Windows for a Java process using -Xmx?
I'm asking because I want to use the ETOPO1 data in OpenMap and the raw binary float file is about 910 MB.

There's nothing better than an empirical experiment to answer your question.
I've wrote a Java program and run it while specifying the XMX flag (also used XMS=XMX to force the JVM pre-allocate all of the memory).
To further protect against JVM optimizations, I've actively allocate X number of 10MB objects.
I run a number of test on a number of JVMs increasing the XMX value together with increasing the number of MB allocated, on a different 32bit operating systems using both Sun and IBM JVMs, here's a summary of the results:
OS:Windows XP SP2, JVM: Sun 1.6.0_02, Max heap size: 1470 MB
OS: Windows XP SP2, JVM: IBM 1.5, Max heap size: 1810 MB
OS: Windows Server 2003 SE, JVM: IBM 1.5, Max heap size: 1850 MB
OS: Linux 2.6, JVM: IBM 1.5, Max heap size: 2750 MB
Here's the detailed run attempts together with the allocation class helper source code:
WinXP SP2, SUN JVM:
C:>java -version
java version "1.6.0_02"
Java(TM) SE Runtime Environment (build 1.6.0_02-b06)
Java HotSpot(TM) Client VM (build 1.6.0_02-b06, mixed mode)
java -Xms1470m -Xmx1470m Class1 142
...
about to create object 141
object 141 created
C:>java -Xms1480m -Xmx1480m Class1 145
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
WinXP SP2, IBM JVM
C:>c:\ibm\jdk\bin\java.exe -version
java version "1.5.0"
Java(TM) 2 Runtime Environment, Standard Edition (build pwi32devifx-20070323 (if
ix 117674: SR4 + 116644 + 114941 + 116110 + 114881))
IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Windows XP x86-32 j9vmwi3223ifx-2007
0323 (JIT enabled)
J9VM - 20070322_12058_lHdSMR
JIT - 20070109_1805ifx3_r8
GC - WASIFIX_2007)
JCL - 20070131
c:\ibm\jdk\bin\java.exe -Xms1810m -Xmx1810m Class1 178
...
about to create object 177
object 177 created
C:>c:\ibm\jdk\bin\java.exe -Xms1820m -Xmx1820m Class1 179
JVMJ9VM015W Initialization error for library j9gc23(2): Failed to instantiate he
ap. 1820M requested
Could not create the Java virtual machine.
Win2003 SE, IBM JVM
C:>"C:\IBM\java" -Xms1850m -Xmx1850m Class1
sleeping for 5 seconds.
Done.
C:>"C:\IBM\java" -Xms1880m -Xmx1880m
Class1
JVMJ9VM015W Initialization error for library j9gc23(2): Failed to instantiate he
ap. 1880M requested
Could not create the Java virtual machine.
Linux 2.6, IBM JVM
[root#myMachine ~]# /opt/ibm/java2-i386-50/bin/java -version
java version "1.5.0"
Java(TM) 2 Runtime Environment, Standard Edition (build pxi32dev-20060511 (SR2))
IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux x86-32 j9vmxi3223-20060504 (JIT enabled)
J9VM - 20060501_06428_lHdSMR
JIT - 20060428_1800_r8
GC - 20060501_AA)
JCL - 20060511a
/opt/ibm/java2-i386-50/bin/java -Xms2750m -Xmx2750m Class1 270
[root#myMachine ~]# /opt/ibm/java2-i386-50/bin/java -Xms2800m -Xmx2800m Class1 270
JVMJ9VM015W Initialization error for library j9gc23(2): Failed to instantiate heap. 2800M requested
Could not create the Java virtual machine.
Here's the code:
import java.util.StringTokenizer;
public class Class1 {
public Class1() {}
private class BigObject {
byte _myArr[];
public BigObject() {
_myArr = new byte[10000000];
}
}
public static void main(String[] args) {
(new Class1()).perform(Integer.parseInt(args[0]));
}
public void perform(int numOfObjects) {
System.out.println("creating 10 MB arrays.");
BigObject arr[] = new BigObject[numOfObjects];
for (int i=0;i <numOfObjects; i++) {
System.out.println("about to create object "+i);
arr[i] = new BigObject();
System.out.println("object "+i+" created");
}
System.out.println("sleeping for 5 seconds.");
try {
Thread.sleep(5000);
}catch (Exception e) {e.printStackTrace();}
System.out.println("Done.");
}
}

For a large file I suggest you use a memory mapped file. This doesn't use heap space (or very little) so maximum heap size shouldn't be a problem in this case.

We have recently ported from Windows to Linux (because of VM size issues).
I have heard of lots of numbers thrown around in the past for Windows VM size (1200, 1400, 1600, 1800). On our Windows Servers (2003), in our environment, with our applications, ... I have never successfully used more than 1280MB. Beyond that our application started exhibiting GC and OOM issues.
Everytime I got a new VM version I tried changing the number and it never varied.
You have a 900MB file now, what if the file increases to 1300MB? What will you do?
You have a number of options
Port to Linux/Solaris. This just needs hardware/software and what is often a simple porting exercise.
Use 64bit Windows. This may not be free of GC issues though - I have heard of different tales with 64bit vms.
Redesign the app to process the file differently, Can you split the file logically in some way, can you read the file in chunks and process it differently etc?
Other people using OpenMap must have encountered this issue. Can you tap into their knowledge and not re-invent any wheels?

As noted in the question mentioned in the comment, there is a practical limit, circa 1200 MB.
However the situation you're describing has more depth to it than sheer memory size.
When you read a 910 MB binary data and build a network objects off of it (as opposed to just maintaining the data as an array of bytes), you end up consuming much more memory than 910 MB. A reasonable estimate would be that the in-memory representation will consume twice as much memory - that's because (1) each object contains an additional pointer (to the class of the object); and (2) there's a lot bookkeeping data. For instance if you use a HashMap to manage your objects then in addition to each object you also allocate a Map.Entry object which can easily consume 16 or 20 bytes (implementation dependent).
On the other hand, there's still hope: do you really need to maintain all 910 MB in memory? Can't you just build something that reads the data in a lazy manner? Combined with WeakReferences I think you can pull this off.

On 32-bit Windows, by default, every application can use up to 2 GB virtual address space. I guess this makes -Xmx2048M. However, if you have more RAM installed, you can increase the virtual address space up to 3 GB by using boot time parameters.
In boot.ini, you can create a new boot options like this:
[boot loader]<br>
timeout=5<br>
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS<br>
[operating systems]<br>
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional - magyar" /noexecute=optin /fastdetect /usepmtimer<br>
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional - magyar 3GB" /noexecute=optin /fastdetect /usepmtimer /3GB /USERVA=2800<br>
Here by adjusting the /USERVA=2800 parameter, you can tune your machine. But be aware that some configurations don't like high values in this parameter - expect crashes.