We Keep Coding

How can I analyse large size heap dump around of 35-40 GB

I have to analyse java heap dump of size 35-40GB, which can't be loaded on local machine except of remote servers of large memory.
I found Tool for analyzing large Java heap dumps as the best link till now. But after configuring all the things and properly executing all the command lines, I was not able to get any report file.
My ParseHeapDump.sh file looks as
#!/bin/sh
#
# This script parses a heap dump.
#
# Usage: ParseHeapDump.sh <path/to/dump.hprof> [report]*
#
# The leak report has the id org.eclipse.mat.api:suspects
# The top component report has the id org.eclipse.mat.api:top_components
#
./MemoryAnalyzer -consolelog -application org.eclipse.mat.api.parse "$#" -vmargs -Xms8g -Xmx10g -XX:-UseGCOverheadLimit
and MemoryAnalyzer.ini file looks as
-startup
plugins/org.eclipse.equinox.launcher_1.5.0.v20180512-1130.jar
--launcher.library
plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.700.v20180518-1200
java -Xmx8g -Xms10g -jar plugins/org.eclipse.equinox.launcher_1.5.0.v20180512-1130.jar -consoleLog -consolelog -application org.eclipse.mat.api.parse "$#"
-vmargs
-Xms8g
-Xmx10g
Please tell me If I'm doing any mistake in configuration or suggest me any other tool available in the market.

Processing large heap dump is a challenge. Both VisualVM and Eclipse Memory Analyzer required too much memory to process heap dumps in order of few dozen of GiB.
Commercial profilers show better result (YourKit in particular) though I not sure of their practical limit.
To routinely process 100+ GiB, I came up you with headless solution heaplib, which based on code base from VisualVM (Netbeans actually).
Heaplib is neigther graphical, nor interactive. It is oriented to automated reporting.
Tool allows you to write code for heap analysis in OQL/JavaScript (or Java if you wish), though capabilities are limited to accommodate memory requirements. Processing of 100GiB could take hour, but for non interactive workflow it is acceptable.

why there is difference in Heap dump size generated by jmap and jcmd?

I am trying to take heap dump using below 2 commands
jcmd $pid GC.heap_dump /tmp/filename.dump
jmap -dump:format=b,file=/tmp/filename.dump $pid
jcmd produces file size of ~300M and jmap produces file size of ~1.4G. why these are different sizes, do we have any additional information in jmap ? am I missing some arguments in jcmd ?
JDK is 1.8.0_162
Xms and Xmx is 4G

By Default (When no [options] are provided],
JMAP took an all-objects dump and JCMD took only live-objects dump.
Using JMAP command:
While using this command you don't need to specify anything as it is by default produce the heap dump of all the objects. If you need live objects alone, you can pass '-dump:live' option in JMAP.
Using JCMD command:
While using this command you have to pass -all option. Otherwise, it will request a full GC and generates only live objects dump.
JCMD - without any options of object state - By default it dumps only the live objects.
JMAP - without any options of object state - By default it dumps all the objects.
For more information refer here

Get error "windbg error:OpenDumpFile failed" when open the core dump with serviceability agent

I can successfully attach to Java live process by pid using serviceability agent ., however failed to open the core dump with it, and got error "windbg error:OpenDumpFile failed" when open the core dump with serviceability agent .
BTW,I capture the dump with the following command , and it would hung if I omit the -F.
jmap.exe -F -dump:format=b,file=c:\temp\HeapDump.hprof pid
The environment:
win 7 64, JDK 7.0(hotspot)

A heap dump is not a core dump.
Heap dumps generated by jmap can be opened in the tools like VisualVM, Eclipse Memory Analyzer, YourKit Java Profiler etc.
Core dumps (or Minidumps in Windows terminology) are written by the OS or by the debuggers like WinDbg. jmap can also be used to extract a heap dump from a minidump.

Have a try to use jstack like this in the command line:
jstack 10776 e:\dump.txt
Replace the number 10776 with your own java process id.

How to analyse the heap dump using jmap in java

I am creating heap dump using below command:
jmap -dump:file=DumpFile.txt <process-id>
I have opened the generated file - DumpFile.txt but it is not in readable format.
So please let me know how to analyze the data in the generated file.

You should use jmap -heap:format=b <process-id> without any paths. So it creates a *.bin file which you can open with jvisualvm.exe (same path as jmap). It's a great tool to open such dump files.

You can use jhat (Java Heap Analysis Tool) to read the generated file:
jhat [ options ] <heap-dump-file>
The jhat command parses a java heap dump file and launches a webserver. jhat enables you to browse heap dumps using your favorite webbrowser.
Note that you should have a hprof binary format output to be able to parse it with jhat. You can use format=b option to generate the dump in this format.
-dump:format=b,file=<filename>

Very late to answer this, but worth to take a quick look at. Just 2 minutes needed to understand in detail.
First create this java program
import java.util.ArrayList;
import java.util.List;
public class GarbageCollectionAnalysisExample{
public static void main(String[] args) {
List<String> l = new ArrayList<String>();
for (int i = 0; i < 100000000; i++) {
l = new ArrayList<String>(); //Memory leak
System.out.println(l);
}
System.out.println("Done");
}
}
Use jps to find the vmid (virtual machine id i.e. JVM id)
Go to CMD and type below commands >
C:\>jps
18588 Jps
17252 GarbageCollectionAnalysisExample
16048
2084 Main
17252 is the vmid which we need.
Now we will learn how to use jmap and jhat
Use jmap - to generate heap dump
From java docs about jmap
“jmap prints shared object memory maps or heap memory details of a given process or core file or a remote debug server”
Use following command to generate heap dump >
C:\>jmap -dump:file=E:\heapDump.jmap 17252
Dumping heap to E:\heapDump.jmap ...
Heap dump file created
Where 17252 is the vmid (picked from above).
Heap dump will be generated in E:\heapDump.jmap
Now use Jhat
Jhat is used for analyzing the garbage collection dump in java -
C:\>jhat E:\heapDump.jmap
Reading from E:\heapDump.jmap...
Dump file created Mon Nov 07 23:59:19 IST 2016
Snapshot read, resolving...
Resolving 241865 objects...
Chasing references, expect 48 dots................................................
Eliminating duplicate references................................................
Snapshot resolved.
Started HTTP server on port 7000
Server is ready.
By default, it will start http server on port 7000.
Then we will go to http://localhost:7000/
Courtesy : JMAP, How to monitor and analyze the garbage collection in 10 ways

If you use Eclipse as your IDE I would recommend the excellent eclipse plugin memory analyzer
Another option is to use JVisualVM, it can read (and create) heap dumps as well, and is shipped with every JDK. You can find it in the bin directory of your JDK.

VisualVm does not come with Apple JDK. You can use VisualVM Mac Application bundle(dmg) as a separate application, to compensate for that.

MAT, jprofiler,jhat are possible options. since jhat comes with jdk, you can easily launch it to do some basic analysis. check this out

If you just run jmap -histo:live or jmap -histo, it outputs the contents on the console!

How to check the configured Xmx value for a running java application

I'm creating an NSIS script, where the Xmx value for the java application being installed can be set during the installation process. I'm not sure if this parameter is being set correctly. Is there a way to check the configured Xmx value when the application is running?

In my case, jmap is the best solution I could find:
jmap -heap <pid>
The above command shows full heap configuration + current usage.
The jmap command is included inside the jdk in the bin directory.

Cheap and dirty (not sure on reliability):
Runtime.getRuntime().maxMemory();
Have also used the following with success:
MemoryMXBean memoryBean = ManagementFactory.getMemoryMXBean();
memoryBean.getHeapMemoryUsage().getMax();

jps is a good solution, just run
jps # shows pids
jps -v <pid> # shows params
remember to run it as the user that launched the process though, or it will not work properly.

Doron Gold gave the correct answer for Java 8 and below. For Java 9 and above you can get the same info by using
jhsdb jmap --heap --pid <pid>

I'm a big fan of kill -3 < pid > , which will give you details on the current memory and garbage collections along with stacks for all threads.

The following worked for me for JVM version 11.0.16 (I had installed OpenJDK 11)
sudo jhsdb jinfo --flags --pid <pid>
It may work for you without using "sudo".
I had to do this after sudo jhsdb jmap --heap --pid <pid> was giving me the following error message:
Exception in thread "main" sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address