We Keep Coding

How to interpret jeprof output

Recently I came across some java related memory leaks (continuously decreasing server-free memory and finally getting RAM warning which we have set up using nagios) and I did an investigation and found that the memory leak is not related to the heap ara. But still tomcat process's memory consumption keeps growing.
server memory graph - 7 days
Did a heap memory analysis and nothing found in there ( if I run jcmd <pid> GC.run heap memory usage drops to around 200MB from 2.8GB). heap memory graph - 7 days
Checked metaspace area and other memory areas related to the JVM as per the discussion on this video and post.
https://www.youtube.com/watch?t=2483&v=c755fFv1Rnk&feature=youtu.be
https://github.com/jeffgriffith/native-jvm-leaks/blob/master/README.md
Finally, I added jemalloc to profile native memory allocation, and here is some of the output that I got.
ouptput 1
ouptput 2
But I couldn't interpret this output and I'm not sure whether this output is correct or not.
And also I have a doubt regarding whether that jeprof is working with oracle JDK.
Could you please help me on this?
Additional info:
server memory: 4GB
Xmx: 3072M (recently we changed to this and earlier it was 2048M. but the memory behavior is similar on both occasions)
Xms: 3072M (recently we changed to this and earlier it was 2048M. but the memory behavior is similar on both occasions)
javac -version: jdk1.8.0_72
java version:
"1.8.0_72"
Java(TM) SE Runtime Environment (build 1.8.0_72-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.72-b15, mixed mode)
jemelloc configs:
jemelloc version: https://github.com/jemalloc/jemalloc/releases/download/5.2.1/jemalloc-5.2.1.tar.bz2
export LD_PRELOAD=/usr/local/lib/libjemalloc.so
export MALLOC_CONF=prof:true,lg_prof_interval:31,lg_prof_sample:17,prof_prefix:/opt/jemalloc/jeprof-output/jeprof
My application is running on a tomcat server in an ec2 instance (only one application running on that server).

OpenJ9 tomcat won't start with high -Xmx heap option

I have a Spring app running in a Tomcat 9.0.6 on Linux 64. Because it needs a lot of memory, I would like to try the OpenJ9 JVM which is supposedly more efficient in that regard (current heap limit with Hotspot: -Xmx128G).
I installed the 64-bit adoptopenjdk-8-jdk-openj9:
/usr/lib/jvm/adoptopenjdk-8-jdk-openj9/bin/java -version
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (build 1.8.0_212-b04)
Eclipse OpenJ9 VM (build openj9-0.14.2, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20190521_315 (JIT enabled, AOT enable
OpenJ9 - 4b1df46fe
OMR - b56045d2
JCL - a8c217d402 based on jdk8u212-b04)
Starting the tomcat causes the following error:
This JVM package only includes the '-Xcompressedrefs' configuration. Please run the VM without specifying the '-Xnocompressedrefs' option or by specifying the '-Xcompressedrefs' option.
After I set this option I get the following error:
JVMJ9GC028E Option too large: '-Xmx'
JVMJ9VM015W Initialization error for library j9gc29(2): Failed to initialize
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Documentation isn't that clear, but I found this:
https://www.ibm.com/support/knowledgecenter/SSYKE2_8.0.0/com.ibm.java.vm.80.doc/docs/mm_gc_compressed_refs.html
Compressed references are used by default on a 64-bit IBM SDK when the value of -Xmx, which sets the maximum Java heap size, is in the correct range. Start of changes for service refresh 2 fix pack 10On AIX®, Linux and Windows systems, the default range is 0 - 57 GB. For larger heap sizes, you can try to use compressed references by explicitly setting -Xcompressedrefs.End of changes for service refresh 2 fix pack 10 However, larger heap sizes might result in an out of memory condition at run time because the VM requires some memory at low addresses. You might be able to resolve an out of memory condition in low addresses by using the -Xmcrs option.
So basically, at least this build of the JDK only supports compressedrefs, and in order to use that, I must set it manually since my Xmx is above the range where it is enabled by default, but that fails because my OS already allocated to much of <4GB memory ranges, but some is needed to use compressedrefs. Since I can never guarantee that that won't be the case, is there any way I can use OpenJ9 without compressedrefs? And will that even yield the benefits in terms of memory consumption? Or is there any way I can use compressedrefs with very high Xmx settings?
I also tried setting this option, but it didn't help: https://www.ibm.com/support/knowledgecenter/SSYKE2_8.0.0/openj9/xmcrs/index.html?view=embed
How do I find the correct size for it? 1G and 64m failed. Even if I find the correct setting, how would this value guarantee that the OS hasn't already allocated all the lower memory addresses?

The limit to use the compressed refs JVM is 57G and you can't run it if the -Xnocompressedrefs option is specified.
The 57G division is documented here: https://www.eclipse.org/openj9/docs/xcompressedrefs/
The -Xnocompressedrefs problem is mentioned in the release notes: https://github.com/eclipse/openj9/blob/master/doc/release-notes/0.15/0.15.md
With a reference to: https://github.com/eclipse/openj9/issues/479
Creating a single JVM that supports both is covered by: https://github.com/eclipse/openj9/issues/643
https://github.com/eclipse/openj9/pull/7505
(With thanks to the help from the Eclipse OpenJ9 slack community, especially to Peter Shipton)

I found this build which allows noncompressedrefs and thus solves my issues: https://adoptopenjdk.net/releases.html?variant=openjdk8&jvmVariant=openj9#linuxxl

Why doesn't it look like -Xms and -Xmx are doing anything to Java's memory usage in Windows?

I am running this version of Java:
java version "1.7.0_07"
Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
I am running this on a Windows 2008 R2 64-bit server on Amazon EC2, in an m1.large instance that has 7.5GB of memory.
When I start my Java app, I am using this command line:
java -Xms6G -Xmx6G -server -jar start.jar
My intent is to have Java reserve 6GB for it's heap, so that when my application runs, it will be able to load its entire dataset into memory.
However, when I start the app, I only see 1.3GB of memory used in Task Manager?

The issue is you're looking at the default memory column shown in Task Manager, which is "Memory - Private Working Set." This doesn't reflect what is actually reserved for use.
In Task Manager, go to the View menu and choose Select Columns. Add the "Memory - Commit Size" column. You should see this column reflects the reserved heap size from Java. In my tests, it shows around 6.6GB committed for a commandline of -Xms6G.
Here is Microsoft's page that explains what each column means.
"Memory - Private Working Set: Subset of working set that specifically describes the amount of memory a process is using that cannot be shared by other processes."
"Memory - Commit Size: Amount of virtual memory that is reserved for use by a process."

HtmlUnit is throwing Out Of Memory and maybe leaking memory

I use Selenium with HtmlUnitDriver with javascript enabled and I get Out Of Memory errors (I use Java). I just browse the same page. I am only using a single GET command. Which is the solution to overcome the situation?

I've had a similar issue. It ended up being an issue with auto-loading of frames... a feature that can't be disabled.
Take a look at this: Extremely simple code not working in HtmlUnit
It might be of help.
Update
Current version of HtmlUnit is 2.10. I started using HtmlUnit from version 2.8 and each new version ended up eating more memory. I got to a point in which fetching 5 pages with javascript enabled resulted in a process of 2GB.
There are many ways to improve this situation from a javascript point of view. However, when you can't modify the javascript (eg: if you are crawling a site) your hands are tied. Disabling javascript is, of course, the best way to go. However, this might result in fetched pages being different from the expected ones.
I did manage to overcome this situation, though. After many tests, I noticed that it might not be an issue with HtmlUnit (which I thought was the guilty one from the beginning). It seemed to be the JVM. Changing from Sun's JVM to OpenJDK did the trick and now the process instead of eating 2GB of memory only requires 200MB. I'm adding version information.
Sun's (Oracle) 32-bit JVM:
$java -version
java version "1.6.0.26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) Server VM (build 20.1-b02, mixed mode)
OpenJDK 32-bit JVM:
$java -version
java version "1.6.0_18"
OpenJDK Runtime Environment (IcedTea6 1.8.13) (6b18-1.8.13-0+squeeze2)
OpenJDK Server VM (build 14.0-b16, mixed mode)
Operative system:
$ uname -a
Linux vostro1015 2.6.32-5-686-bigmem #1 SMP Sun May 6 04:39:05 UTC 2012 i686 GNU/Linux
Please, share your experience with this.

Give more memory to the JVM by adding this to the java command line that starts the JVM in which Selenium is running:
-Xmx512m
This example give a maximum of 512 Mb to the JVM.
It depends on where you're running Selenium from. If maven, you can add it to the MAVEN_OPTS environment variable, if Eclipse, you'll need to edit the run configuration for the test class, etc.

Related to HtmlUnit:
Do not forget to call webClient.closeAllWindows();. I always put it in a finally-block around the area I use the webclient. This way it is sure that all javascript is stopped and all resources are released.
Aslo useful is setting for the webClient:
webClient.setJavaScriptTimeout(JAVASCRIPT_TIMOUT);
webClient.setTimeout(WEB_TIMEOUT);
webClient.setCssEnabled(false); // for most pages you do not need css to be enabled
webClient.setThrowExceptionOnScriptError(false); // I never want Exceptions because of javascript
JAVASCRIPT_TIMOUT should be not too high long running javascript may be a reason for memory problems.
WEB_TIMEOUT think about how long you want to wait maximal.

How to get remote debugging work for Java on Solaris

I am trying to get remote debugging working with Java on Solaris OS. Following is what I have tried-
I have a Java class called TestP which has the main method.
When I try
java -classpath . TestP
the program works fine. But when I try adding the debug parameters to the JVM-
java -Xdebug -Xrunjdwp:transport=dt_socket,address=8001,server=y,suspend=n -classpath . TestP
It fails giving this error-
UTF ERROR ["../../../src/solaris/npt/utf_md.c":49]: Failed to complete iconv_open() setup
Can anyone please help me on figuring out why this error is coming up?? The above works fine on my Linux box.
Java version on Solaris:
Java(TM) SE Runtime Environment (build 1.6.0_15-b03)
Java HotSpot(TM) Client VM (build 14.1-b02, mixed mode)
Java version on Linux:
java version "1.6.0_0"
OpenJDK Runtime Environment (IcedTea6 1.5) (fedora-20.b16.fc10-i386)
OpenJDK Server VM (build 14.0-b15, mixed mode)

Hmm... This seems to be a known Solaris (not specific to Solaris 10) issue with Java 6, not a Java issue (see this thread).
Someone has successfully applied a workaround (see this blog post) from a Sun guy, Jeff Moguillansky, but I wouldn't recommend it and rather consider searching sunsolve for a patch as indicated on Sun's forums.
Look at this one: http://sunsolve.sun.com/search/document.do?assetkey=1-1-6586755-1 (you'll need a Sun Online Account with a valid Support Contract or Software Subscription).

Using truss I found out that the process was looking for /usr/lib/iconv/geniconvtbl/binarytables/UTF-8%646.bt and 646%UTF-8.bt, so I just copied ISO8859-1%ISO646.bt to UTF-8%646.bt and ISO646%ISO8859-1.bt to 646%UTF-8.bt (yes, it is "646", not "ISO646"!)
This is of course a very ugly workaround and I have no idea if it has any negative effects on the JVM, but at least it starts the JVM without aborting. (I did this on OpenSolaris 2009.06, btw)

If Pascal Thivent is right, then you may want to try running OpenSolaris (either on a blank machine or in a vm) and see if the problem is also there. If not, then consider using that version for now if possible.