I am having real problems getting to the bottom of the java hotspot errors we are experiencing. The errors are seemingly random and the problematic frame varies. An example error is shown below:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f65ce3c3a84, pid=26082, tid=0x00007f65bc2ab700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_101-b13) (build 1.8.0_101-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.101-b13 mixed mode linux-amd64 )
# Problematic frame:
# V [libjvm.so+0x5c3a84] G1ParScanThreadState::copy_to_survivor_space(InCSetState, oopDesc*, markOopDesc*)+0x174
#
# Core dump written. Default location: /opt/dap/domains/tc0419/prod1/tomcat/bin/core or core.26082 (max size 1 kB). To ensure a full core dump, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
The java application is a relatively simple, it is just a war file running in a tomcat on a server.
Please let me know if any additional information would be helpful to access, I'm a bit new to this. The current thinking is that this is a hardware issue but any further information would be much appreciated.
Thanks,
Adam
The problematic frame "G1ParScanThreadState::copy_to_survivor_space(InCSetState, oopDesc*, markOopDesc*)+0x174" represents it is an gc crash. It is bug in JVM. There are couple of issues reported before
1. https://bugs.openjdk.java.net/browse/JDK-8170451
2. https://bugs.openjdk.java.net/browse/JDK-8143310
As we don't have way to reproduce or they occured once in a while. As you can see both the issues are closed either as Incomplete or cannot reproduce.
If you have a strong reproduce kindly file a bug through http://bugreport.java.com/submit_intro.do along with steps/test case to reproduce the issue.
It will be addressed... :)
You are using JDK 8 update 101, Kindly upgrade to latest build JDK8 update 121 from here - http://www.oracle.com/technetwork/java/javase/downloads/index.html
Related
Sometimes my code crashes leaving me with an error file but no core dump because the latter is disabled.
Now it is suggested to set ulimit -c unlimited to allow core dumps.
If I would be running the code from console, it would be no issue to set ulimit before starting the Java Application. But the error seems to be much more frequent when debugged in Eclipse as when running as a standalone. (Actually as a standalone it crashed this way only once in hundreds of running hours, in debugging it crashed 4 times in the last two months (and some times before)).
Is there a way to tell Eclipse to set ulimit before launching a Debug-Session in Eclipse ?
And will a Core-Dump help me to find what causes the crash ?
For completeness, I am working on macOS and my error files all start like this (only pidchanges between them):
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x00007fffa08c144e, pid=617, tid=0x0000000000000307
#
# JRE version: Java(TM) SE Runtime Environment (8.0_181-b13) (build 1.8.0_181-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# C [AppKit+0x3a544e] -[NSApplication _crashOnException:]+0x6d
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
I suspect (but have nothing to prove it except the "moment when it crashes") that the crashes are related with the handling of BufferedImage within the Java code, but then again, the Error-Log tells that the error happened outside the JVM.
We are seeing the JVM getting crashed at times with segfault. The only error we see in logs is as below.
Anyone can suggest something by looking at the below error trace.
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fef7f1d3eb0, pid=42623, tid=0x00007feea62c8700
#
# JRE version: OpenJDK Runtime Environment (8.0_222-b10) (build 1.8.0_222-b10)
# Java VM: OpenJDK 64-Bit Server VM (25.222-b10 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J 62683 C2 org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeObject0(Ljava/lang/Object;)V (331 bytes) # 0x00007fef7f1d3eb0 [0x00007fef7f1d3e00+0xb0]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /tmp/hsperfdata_pvappuser/hs_err_pid42623.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
While trying to understand the reason for this crash Oracle JVM docs https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/crashes001.html ,this looks to be the case of 5.1.2 Crash in Compiled Code as the problematic frame is java frame(has a "J")
Though could not get much further from it, we also not sure when it comes, the only probale pattern is it comes when JVM is running for 5-6 days so usually on Friday.
We are using openjdk-8 ("1.8.0_232") distribution provided by RedHat running on RHEL 6.10.
Looking forward to get any leading point in tracing this error.
The current stack frame has writeObject0 as the last called method. There is a naming convention that native method's names end with 0. Therefore check whether that method is indeed native.
If it is, it is probably written in C, an ancient unsafe language whose programs tend to crash in an uncontrolled way. This often leads to SIGSEGV.
In this case, that method is written in Java though.
As you were told in the error message, read hs_err_pid42623.log for further details. In that file you will find the registers and a few machine instructions around the code that crashed.
One of our clients have contact me complaining that he's getting the following JRE crash while using our Java Agent.
According to the error (below) the crash is on the native code since the problematic frame is categorized as 'C'.
I've did some googling and it seems like there are some open bugs which are quite similar around this issue in this while using java agents. See the following links:
https://bugs.openjdk.java.net/browse/JDK-8094079
https://bugs.openjdk.java.net/browse/JDK-8041920
The issue is that the customer is reluctant to upgrade the JDK since he mentions that he has other java agents which are running without any issue.
Any suggestions on how to solve this issue?
For the completeness, please see the error that he had sent:
cat /opt/somecompany/apps/some-product-platform/some-product-name/hs_err_pid6697.log | grep sealights
7fe19d9b5000-7fe19d9d7000 r--s 00401000 ca:01 3539943 /opt/somecompany/apps/some-product-platform/some-product-name/sealights/sl-test-listener.jar
jvm_args: -javaagent:/opt/somecompany/apps/some-product-platform/hawtio/jolokia-jvm.jar=config=/opt/somecompany/apps/some-product-platform/some-product-name/conf/jolokia-agent.properties -javaagent:/opt/somecompany/apps/some-product-platform/some-product-name/agent/newrelic.jar -DNEWS_product_HOME=/opt/somecompany/apps/some-product-platform/some-product-name -Dsl.environmentName=Functional Tests DEV-INT -Dsl.customerId=myCustomer -Dsl.appName=ABB-product-name -Dsl.server=https://my-server.com -Dsl.includes=com.somecompany.* -javaagent:/opt/somecompany/apps/some-product-platform/some-product-name/sealights/sl-test-listener.jar -Dlog.dir=/opt/somecompany/apps/some-product-platform/logs -Dlog.threshold=debug
java_class_path (initial): some-product-name.jar:/opt/somecompany/apps/some-product-platform/hawtio/jolokia-jvm.jar:/opt/somecompany/apps/some-product-platform/some-product-name/agent/newrelic.jar:/opt/somecompany/apps/some-product-platform/some-product-name/sealights/sl-test-listener.jar
JVM crash message:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000000000000055, pid=6697, tid=140604865455872
#
# JRE version: Java(TM) SE Runtime Environment (8.0_20-b26) (build 1.8.0_20-b26)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.20-b23 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C 0x0000000000000055
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /opt/somecompany/apps/some-product-platform/some-product-name/hs_err_pid6697.log
Compiled method (c1) 565518 13823 1 sun.invoke.util.ValueConversions::identity (2 bytes)
total in heap [0x00007fe1a76857d0,0x00007fe1a7685a48] = 632
relocation [0x00007fe1a76858f8,0x00007fe1a7685918] = 32
main code [0x00007fe1a7685920,0x00007fe1a7685980] = 96
stub code [0x00007fe1a7685980,0x00007fe1a7685a10] = 144
metadata [0x00007fe1a7685a10,0x00007fe1a7685a18] = 8
scopes data [0x00007fe1a7685a18,0x00007fe1a7685a20] = 8
scopes pcs [0x00007fe1a7685a20,0x00007fe1a7685a40] = 32
dependencies [0x00007fe1a7685a40,0x00007fe1a7685a48] = 8
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
#
After asking for further details in the chat discussion, I assume that this error is related to the use of the ASM AdviceAdapter in combination with other agents in the same application. If the other agent expected a specific structure of a class, using the adapter might reorder variable indices of the class in question what the other agent does not anticipate. As the VM skips the verification of built-in classes, this results in a hard crash.
The solution would be the change the instrumentation to not use the AdviceAdapter.
I have working with Android Studio for quite a long time. Today, I have upgraded my android studio's some of the build tools as well as upgraded to a latest version (1.5 or so), I can't create any more new project. I am getting a core dump error. I am from Ubuntu 14.04, I am running it from my terminal.
Here is the Error terminal is giving me
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f1520266ca8, pid=4094, tid=139728681764608
#
# JRE version: Java(TM) SE Runtime Environment (7.0_80-b15) (build 1.7.0_80-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.80-b11 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V [libjvm.so+0x436ca8] FreeListSpace_DCTOC::walk_mem_region_with_cl_par(MemRegion, HeapWord*, HeapWord*, FilteringClosure*)+0x268
#
# Core dump written. Default location: /home/monkey/Code/core or core.4094
#
# An error report file with more information is saved as:
# /home/monkey/java_error_in_STUDIO_4094.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
Aborted (core dumped)
I did a little googling and found out that it might happen if my Java runs out of memory. So, I went up and gave Java unlimited memory (it has basic 382MB or something memory). I have been in this error for whole day long.
Thank you for your input.
Just try to downgrade libglib2.0-0 from synaptic. and don't forget to lock the downgrade version, so it will not update again in the future.
downgrade:
package -> force version
lock:
package -> lock version
This is the entire error message
http://pastebin.com/bDgye0rt
The error log is too big that I can't attach it here. I am not extremely familiar with how jvm works in the background and what registers it makes use of. I am hoping someone can look at this error log and explain to me what it means.
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007ffd18cbcafe, pid=29906, tid=140725158119168
#
# JRE version: 6.0_27-b27
# Java VM: OpenJDK 64-Bit Server VM (20.0-b12 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea6 1.12.6
# Distribution: Debian GNU/Linux 7.1 (wheezy), package 6b27-1.12.6-1~deb7u1
# Problematic frame:
# C [libresolv.so.2+0x7afe] __libc_res_nquery+0x19e
#
# If you would like to submit a bug report, please include
# instructions how to reproduce the bug and visit:
# http://icedtea.classpath.org/bugzilla
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
I can't understand why and how it is created. It keeps happening all the time.
PS:
I have the error log of another 3 crashes that happened in my machine which I can share if you think its necessary.
EDIT
I have around 8 logs and out of the 8 logs it is made clear that the last call before the program crashes is
java.net.Inet6AddressImpl.lookupAllHostAddr
I do not believe the input is that much related because there is a static list of inputs that are parsed serially, and the error occurs at different time frames (Sometimes 1 hour after the program has been running and some other times 6 hours, it appears to be random).
It is an issue related to glibc and IPv6.
It would be safe to add -Djava.net.preferIPv4Stack=true to ask JVM use IPv4 first to avoid this issue.