I want to collect heap dump on JVM crash
So i wrote a simple code
public class Test {
private String name;
public Test(String name) {
this.name = name;
}
public void execute() {
Map<String,String> randomData = new HashMap<String,String>();
for(int i=0;i<1000000000;i++) {
randomData.put("Key:" + i,"Value:" + i);
}
}
public void addData() {
}
public static void main(String args[]) {
String myName = "Aniket";
Test tStart = new Test(myName);
tStart.execute();
}
}
and I am running it as follows
[aniket#localhost Desktop]$ java -cp . -Xms2m -Xmx2m Test
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at Test.execute(Test.java:15)
at Test.main(Test.java:25)
I got OutOfMemoryError which I wanted but there is no heap dump in the working directory(like hs_err_pidXXXX.log which I expected). What am I missing? How do I get a heap dump?
Update :
I tried -XX:ErrorFile=. still no use. If above is not the way to get the heap dump(Crash JVM) how can I crash my JVM to get those logs?
You are confusing an exception or error being thrown as a JVM crash.
A JVM crash occurs due to an internal error in the JVM, you cannot trigger this by writing a normal Java program (or should not unless you find a bug)
What you are doing is triggering an Error which means the program continues to run until all the non daemon threads exit.
The simplest tool to examine the heap is VisualVM which comes with the JDK. If you want to trigger a heap dump on an OutOfMemoryError you can use -XX:+HeapDumpOnOutOfMemoryError
Use Jmap
jmap [options] pid
pid is the process id of application
When you see the below
Exception in thread "main" java.lang.OutOfMemoryError
It means your error or exception is handled by the exception handler. This is not a crash.
Eclipse has an awesome Heap Analyzer
Also, you can use jps to get the PID, and then jmap for the heap itself.
In case, you want to crash the JVM, your best guess would be native code.
Find the process id for which you want to take the heap dump
ps -ef | grep java
Once you get PID by running above command run below command to generate heap dump.
jmap -dump:format=b,file=<fileName> <java PID>
You can pass below JVM arguments to your application:
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=
This argument will automatically trigger heap dump in the specified 'file-path' when your application experiences OutOfMemoryError. There are 7 different options to take heap dumps from your application:
jmap
-XX:+HeapDumpOnOutOfMemoryError
jcmd
JVisualVM
JMX
Programmatic Approach
Administrative consoles
Details about each option can be found in this article. Once you have captured heap dump, you may use tools like Eclipse Memory Analysis tool, HeapHero to analyze the captured heap dumps.
Related
I currently hava a running Java application, which has a bug. I don't know how to fully reproduce it and it didn't happen for weeks until now. When it occurs one times, I can reproduce it over and over again easily until I restart the application. The bug causes a StackOverflowError because of a recursion and I don't know how this happens. The printed stacktrace of the StackOverflowError isn't helpful because it contains only the repeating part, but not the more insteresting initial part, because the JVM has a limit for stacktrace entries. The -XX:MaxJavaStackTraceDepth=... can be used to set this limit as explained here. The problem is that I think I have to restart my application in order to add this flag. But if I do so, I won't be able to reproduce the bug anymore. Is there any solution how I can get the full stacktrace or set this flag without restarting the application?
I know at least two solutions.
Create HotSpot Serviceability Agent tool to find the address of MaxJavaStackTraceDepth variable in memory, and then update the memory of the process using OS-specific mechanism.
Attach a JVM TI agent that intercepts StackOverflowErrors and prints a stack trace right from the agent.
Here is the code for the first solution (as it is presumably shorter):
import sun.jvm.hotspot.debugger.Address;
import sun.jvm.hotspot.runtime.VM;
import sun.jvm.hotspot.tools.Tool;
import java.io.IOException;
import java.io.RandomAccessFile;
public class ChangeVMFlag extends Tool {
private static String pid;
#Override
public void run() {
Address addr = VM.getVM().getCommandLineFlag("MaxJavaStackTraceDepth").getAddress();
long addrValue = VM.getVM().getDebugger().getAddressValue(addr);
try (RandomAccessFile raf = new RandomAccessFile("/proc/" + pid + "/mem", "rw")) {
raf.seek(addrValue);
raf.writeInt(Integer.reverseBytes(1_000_000));
} catch (IOException e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
pid = args[0];
new ChangeVMFlag().execute(new String[]{pid});
}
}
This tool changes the value of MaxJavaStackTraceDepth in the target process to 1 million.
Note: it uses Linux-specific /proc API to write into the target process' memory. Other OSes have different interfaces.
How to run
On JDK 8
java -cp .:$JAVA_HOME/lib/sa-jdi.jar ChangeVMFlag <pid>
On JDK 9+
java --add-modules=jdk.hotspot.agent \
--add-exports jdk.hotspot.agent/sun.jvm.hotspot.tools=ALL-UNNAMED \
--add-exports jdk.hotspot.agent/sun.jvm.hotspot.runtime=ALL-UNNAMED \
--add-exports jdk.hotspot.agent/sun.jvm.hotspot.debugger=ALL-UNNAMED \
ChangeVMFlag <pid>
I'm using Lucene v4.10.4. I have pretty big index, it could be over few GBs. So I get OutOfMemoryError on initializing IndexSearcher:
try (Directory dir = FSDirectory.open(new File(indexPath))) {
//Out of Memory here!
IndexSearcher searcher = new IndexSearcher(DirectoryReader.open(indexDir));
How to tell Lucene's DirectoryReader to not load into memory more than 256 MB at once?
Log
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.fst.BytesStore.<init>(BytesStore.java:68)
at org.apache.lucene.util.fst.FST.<init>(FST.java:386)
at org.apache.lucene.util.fst.FST.<init>(FST.java:321)
at org.apache.lucene.codecs.blocktree.FieldReader.<init>(FieldReader.java:85)
at org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.<init>(BlockTreeTermsReader.java:192)
at org.apache.lucene.codecs.lucene41.Lucene41PostingsFormat.fieldsProducer(Lucene41PostingsFormat.java:441)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:197)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:254)
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:120)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:108)
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:62)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:923)
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:53)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:67)
First you should check the current heap size of your JVM.
java -XX:+PrintFlagsFinal -version | grep MaxHeapSize
If this number is not reasonable for your use case, you should increase it when running your program with -Xmx option of java command. A sample command to assign 8GB of heap memory would look like:
java -Xmx8g -jar your_jar_file
Hope this helps.
Please help in this, i want to run a shell script where it should take jave heap dump every 30 seconds using kill -3 command. Thanks in advance.
Have you tried such a simple shell script?
while true
do
jmap -dump:file=/tmp/java-`date +%s`.hprof PID_OF_JVM
sleep 30
done
This will create one file pear each snapshot. For thread dump you can use similar script:
while true
do
jstack PID_OF_JVM > stack-`date +%s`.txt
sleep 30
done
I guess you can use kill -3 instead of jstack.
you can do thread dumping from java application using code like this
public static String getDumpFor(Thread thread) {
StringBuilder sb = new StringBuilder();
if (thread.isAlive()) {
StackTraceElement[] stackTrace = thread.getStackTrace();
sb.append(thread.toString()).append("\n")
.append(String.format(" State - %s,", thread.getState()))
.append(String.format(" Is daemon = %s,", thread.isDaemon()));
for (StackTraceElement s : stackTrace)
sb.append("\tat ").append(s.getClassName()).append(".").append(s.getMethodName()).append("(").append(s.getFileName()).append(":").append(s.getLineNumber()).append(")")
.append("\n");
}
return sb.toString();
}
public static void dumpActiveThreads() {
Map<Thread, StackTraceElement[]> stackTraces = Thread.getAllStackTraces();
Set<Thread> keySet = stackTraces.keySet();
System.out.println("\nThread dump begin:");
for (Thread thread : keySet)
dumpActiveThread(thread);
System.out.println("\nThread dump end.");
}
and then schedule task like this
final ScheduledFuture<?> scheduledFuture = scheduledExecutorService.scheduleWithFixedDelay(
new Runnable() {dumpActiveThreads()},
0,
30, TimeUnit.SECONDS);
I have not used kill -3 command but i have used jmap command provided by sun sdk
You can write a script and then in script run below command.
${JAVA_HOME}/bin/jmap -dump:file=/home/MyDump.hprof PID
3 will give only the thread dump but not the heap dump.Thread dump means you can only check the stack traces for each thread in the JVM.But you are lookin g for the heap dump on linux then need to use the below commands.
jmap -dump:file=myheap.bin {pid of which you are looking to take heap dump}. The output "myheap.bin" is not human readable,to read the file you can use the MAT tool.
MAT download link: http://www.eclipse.org/mat/
I tried to read a text file which have only one line. File size is over 50Mb. When I tried to read it using the following code it gives
java.lang.outofmemory exception jvm
heap size insufficient
I change the java heap memory to 1GB . But still it gives the same exception.
set JAVA_OPTS=-Xms1024m -Xmx1024m
I use foollowing code pragment to read the file.
BufferedReader Filein1=new
BufferedReader(new FileReader( new
File( "C:\ABC\MsgStream.txt" )));
s=Filein1.readLine();
Can some one please tell me how to overcome this problem. Thanks in advance.
The JAVA_OPTS environment variable is only respected by certain applications (for example the wrapper scripts that are typically used to launch Tomcat). The java command doesn't pay any attention to it.
You need to put the options on the java command line ... before the classname; e.g.
java -Xms1024m -Xmx1024m ... some.pkg.MainClass ...
(A 1Gb heap should be more than adequate for buffering a 50Mb file.)
Are JAVA_OPTS actually being run in the output? You may need to actually put them on the command line you're running, or include $JAVA_OPTS on it.
It's _JAVA_OPTIONS not JAVA_OPTS
If you have a class like
public class Test {
public static void main(String args[]) {
System.out.println("value : " + System.getProperty("foo"));
}
}
then you should get
> set _JAVA_OPTIONS=-Dfoo=bar
> java Test
> value : bar
does java 6 generate thread dump in addition to heap dump (java_pid14941.hprof)
this is what happened to one of my applications.
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to java_pid14941.hprof ...
I did find ava_pid14941.hprof in working directory, but didn't find any file which contains thread dump. I need to know what all the threads were doing when I got this OutOfMemory error.
Is there any configuration option which will generate thread dump in addition to heap dump on out of memory exception?
If you're in a Linux/Unix environment you can do this:
-XX:OnOutOfMemoryError="kill -3 %p"
This way you don't have to have your application generate periodic thread dumps and you'll get a snapshot when it actually chokes.
With %p, you don't need to pass the PID, JVM will automatically pick the correct process id as mentioned here.
How to generate thread dump java on
out of memory error?
Your question can be simplified into:
how to generate a thread dump
and:
how to catch an out of memory error (don't pay attention to naysayer here, they're missing the bigger picture, see my comment)
So it's actually quite easy, you could do it like this:
install a default uncaught exception handler
upon catching an uncaught exception, check if you have an OutOfMemoryError
if you have an OutOfMemoryError, generate yourself a full thread dump and either ask the user to send it to you by email or offer to send it automatically
Bonus: it works fine on 1.5 too :)
Thread.setDefaultUncaughtExceptionHandler( new Thread.UncaughtExceptionHandler() {
public void uncaughtException( final Thread t, final Throwable e ) {
...
}
You may want to look into this:
e.getMessage();
and this:
Thread.getAllStackTraces();
I'm doing this all the time in an app that is shipped on hundreds of different 1.5 and 1.6 JVM (on different OSes).
It's possible to trigger a thread dump when OnOutOfMemoryError is triggered using jstack. e.g:-
jstack -F pid > /var/tmp/<identifier>.dump
I don't think there is anything in java that would provide you with on-exit thread dumps. I tackle this when necessary by having a cronjob that does periodic kill -3 pid. Yes, it does clutter the logs a bit, but the footprint is still negligible.
And if you are suffering from OOM, it might be be beneficial to see how the situation evolved thread-wise.
Based on the accepted answer I created utility class. This one you can define as a Spring bean and you're all set with extended logging.
import java.util.Iterator;
import java.util.Map;
import javax.annotation.PostConstruct;
import org.apache.commons.lang3.exception.ExceptionUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class UncaughtExceptionLogger {
private final static Logger logger = LoggerFactory.getLogger(UncaughtExceptionLogger.class);
#PostConstruct
private void init() {
Thread.setDefaultUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler() {
public void uncaughtException(final Thread t, final Throwable e) {
String msg = ExceptionUtils.getRootCauseMessage(e);
logger.error(String.format("Uncaght exception handler captured expcetion '%s'", msg), e);
if (msg.contains("unable to create new native thread")) {
String dump = captureThreadDump();
logger.error(String.format(
"OutOfMemoryError has been captured for threads limit. Thread dump: \n %s", dump), e);
}
if (ExceptionUtils.getRootCause(e) instanceof OutOfMemoryError) {
String dump = captureThreadDump();
logger.error(String.format("OutOfMemoryError has been captured. Thread dump: \n %s", dump), e);
}
}
});
}
public static String captureThreadDump() {
/**
* http://stackoverflow.com/questions/2787976/how-to-generate-thread-
* dump-java-on-out-of-memory-error
* http://henryranch.net/software/capturing-a-thread-dump-in-java/
*/
Map<Thread, StackTraceElement[]> allThreads = Thread.getAllStackTraces();
Iterator<Thread> iterator = allThreads.keySet().iterator();
StringBuffer stringBuffer = new StringBuffer();
while (iterator.hasNext()) {
Thread key = (Thread) iterator.next();
StackTraceElement[] trace = (StackTraceElement[]) allThreads.get(key);
stringBuffer.append(key + "\r\n");
for (int i = 0; i < trace.length; i++) {
stringBuffer.append(" " + trace[i] + "\r\n");
}
stringBuffer.append("");
}
return stringBuffer.toString();
}
}
-XX:OnOutOfMemoryError="kill -3 %p"
The JVM argument to take thread dump sadly don't work.
Child process cannot end SIGQUIT to parent.
Oracle has -XX:CrashOnOutOfMemoryError but this is available on Java 8.
Assuming that you have a JDK (not a JRE) installed on your target server, you can rely on the command jcmd to generate your thread dump, this way it will work whatever the OS of your server.
The JVM option defining the command to execute on OutOfMemoryError is then:
-XX:OnOutOfMemoryError="jcmd %p Thread.print"
Where %p designates the current process id
Moreover, as we could want to have the thread dump into a file, it is possible to have it into the JVM log file by adding the JVM options -XX:+UnlockDiagnosticVMOptions -XX:+LogVMOutput -XX:LogFile=jvm.log, this way the heap dump will be available in the file jvm.log located in the working directory of the JVM along with other information intended for diagnosing the JVM.
The full list of JVM options to add is then:
-XX:OnOutOfMemoryError="jcmd %p Thread.print" -XX:+UnlockDiagnosticVMOptions
-XX:+LogVMOutput -XX:LogFile=jvm.log
Be aware that the file is always created even when no OOME happens, so on a server, if you want to avoid having the previous JVM file being replaced at the next startup, you should consider adding the process id in the name of the log file like for example jvm_pid%p.log but don't forget to remove the files regularly.
If the JVM option -XX:LogFile is not set, by default the name of the file is of type hotspot_pid%p.log where %p designates the current process id.
In practice, the only JVM option that is needed is -XX:+HeapDumpOnOutOfMemoryError as it allows to generate an hprof file when an OnOutOfMemoryError occurs which already contains a thread dump.
See below how to get a thread dump from an hprof file using VisualVM: