Getting a JVM Crash error. Possible Memory error - java

So I am getting the most random JVM crashes while my Web Application is in production. My environment is Linux Redhat enterprise edition 7, Jboss AS 7, Java 8, JSF 2.2
This crash seems to happen when my user persists some data to the database. Here is the JVM log:
Here is a link to my full log it exceeded the maximum character amount to post on this site.
https://drive.google.com/file/d/0B_JrHFckYWh5d2ktRGphUTJlRFk/view?usp=sharing
I have no idea what is relevant, this has been happening a lot recently I have a hunch it's because of too little memory allocated somewhere, I appreciate the help,
Sam

Crash seems to be Kernel related issue, Please find the complete blog details here https://blogs.oracle.com/poonam/entry/analysis_of_strange_hotspot_crashes
Form your logs it is not clear which kernel version you are using, logs are missing System information. Please upgrade your kernel version. FInd the similar issue here - https://bugs.openjdk.java.net/browse/JDK-8161645

Crash occurs somewhere in code cache while trying to run this:
J 12713 C2 com.sun.crypto.provider.GCTR.update([BII[BI)I (158 bytes).
I found similar bug in openjdk's jira and it says that -XX:-UseAESIntrinsics can help you.
Also you can exclude this method from JIT compilation -XX:CompileCommand=exclude,com/sun/crypto/provider/GCTR.update

Related

Analyzing crash dump (NULL_CLASS_PTR_READ)

My application is crashing from time to time. Looking at windows crash dump, the following seems interesting:
ExceptionAddress: 000000006abc0608 (jvm!JVM_ResolveClass+0x000000000001d6b8)
ExceptionCode: c0000005 (Access violation)
DEFAULT_BUCKET_ID: NULL_CLASS_PTR_READ
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
FAILURE_BUCKET_ID: NULL_CLASS_PTR_READ_c0000005_jvm.dll!JVM_ResolveClass
BUCKET_ID: X64_APPLICATION_FAULT_NULL_CLASS_PTR_READ_jvm!JVM_ResolveClass+1d6b8
Can you please assist with how should I analyze it? How can I tell if it's something in my application code or in the JVM code?
Thanks
Your Java program should not be able to crash the JVM (I assume you're not using JNI or similar).
I would upgrade your JVM and try a newer version.
Seems to be a JVM issue. I would recommend trying your application on another computer just to be sure. Make sure you're using the latest version of Java as well.

Debugging the "safepoint" error - need theoretical OR practical to debugging JVM crashes?

We have an amazingly elusive jvm crash occuring on an Ubuntu server that runs on AWS.
Our JVM crashes while crawling a few web pages.
The crash occurs at line 308 of the "safepoint" cpp module. At the stage where a gauranteeArmed==0 statement occurs.
Our sysadmin has advised that , at the time of crashing, there are a massive amount of threads created by the JVM.
We have not reproduced this bug in other Linux or OSX boxes.
We use the Ning library to crawl a few
Web pages.
Related Posts
How do I investigate the cause of a JVM crash?
JBoss / HotSpot JVM crashing
In each of these posts a "safepoint" related crash which comes from "nowhere" was observed. Most interestingly, the first above post actually exhibits a JVM crash during network related events.
The cryptic nature of this bug leads me to believe that there is a bug related to thread creation and scheduling which is specific to our current version of Ubuntu with respect to the way java invokes some of its concurrency features, or some underlying library incompatibility that is highly idiosyncratic to our particular situation.
My Question(s)
My main question here is - what is the best method for debugging a JVM stack trace involving these "safepoints", and where can I get started learning about dealing with such errors ? There have been other questions along this line, but I have not seen a generic answer .
Secondary, any insight into aws, java, networking, and how Ubuntu might behave differently in the cloud would be useful here.
Try using the very latest JVM (6u32 or 7u4) and see if it's still reproducible. If you are on an older version, there's at least a decent chance it's already been fixed in the latest.

Why does the Java VM update 25 crash with internal error

Since Java update 25 the VM crashes occasionally with internal error. With previous versions <25 it was working fine. According to the release notes, the hotspot compiler was modified in update 25. Does it produce defect code that causes the crash? It does not crash if the JIT compiler is turned off with -Xint. I filed a bug here http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7054478 .
How do I interpret the log file to find out where the crash occurs. I am not able to identify the lines in my Java code where it happenes.
I searched the Bug Database for the string "Unexpected result from topLevelExceptionFilter", and there were three other hits. They all seem to be about unexpected exceptions in native code propagating back across the JNI boundary.
Is that clue relevant to your application?
The reason is that an internal assertion failed - the JVM was not in the state it expected to be. This is a good thing because it avoids propagating errors, but a bad thing because it doesn't tell you how to get around it.
If simple tricks like -client or -server doesn't help, then consider a different JVM.
IBM has a Windows JVM, but it is a bit tricky to get. The easiest for now would be a development package http://www.ibm.com/developerworks/java/jdk/eclipse/index.html
Oracle also have JRockit. http://www.oracle.com/technetwork/middleware/jrockit/index.html
This will allow you to work while Oracle has a look at your bug. It is low priority though, so it make take a while.

Possible causes of Java VM EXCEPTION_ACCESS_VIOLATION?

When a Java VM crashes with an EXCEPTION_ACCESS_VIOLATION and produces an hs_err_pidXXX.log file, what does that indicate? The error itself is basically a null pointer exception. Is it always caused by a bug in the JVM, or are there other causes like malfunctioning hardware or software conflicts?
Edit: there is a native component, this is an SWT application on win32.
Most of the times this is a bug in the VM.
But it can be caused by any native code (e.g. JNI calls).
The hs_err_pidXXX.log file should contain some information about where the problem happened.
You can also check the "Heap" section inside the file. Many of the VM bugs are caused by the garbage collection (expecially in older VMs). This section should show you if the garbage was running at the time of the crash. Also this section shows, if some sections of the heap are filled (the percentage numbers).
The VM is also much more likely to crash in a low memory situation than otherwise.
Answer found!
I had the same error and noticed that others who provided the contents of the pid log file were running 64 bit Windows. Just like me. At the end log file, it included the PATH statement. There I could see C:\Windows\SysWOW64 was incorrectly listed ahead of: %SystemRoot%\system32. Once I corrected it, the exception disappeared.
First thing you should do is upgrade your JVM to the latest you can.
Can you repeat the issue? Or does it seem to happen randomly? We recently had a problem where our JVM was crashing all over the place, at random times. Turns out it was a hardware problem. We put the drives in a new server and it completely went away.
Bottom line, the JVM should never crash, as the poster above mentioned if your not doing any JNI then my gut is that you have a hardware problem.
The cause of the problem will be documented in the hs_err* file, if you know what to look for. Take a look, and if it still isn't clear, consider posting the first 5 or 10 lines of the stack trace and other pertinent info (don't post the whole thing, there's tons of info in there that won't help - but you have to figure out which 1% is important :-) )
Are you using a Browser widget and executing javascript in the Browser widget? If so, then there are bugs in some versions of SWT that causes the JVM to crash in native code, in various Windows libraries.
Two examples (that I opened) are bug 217306 and bug 127960. These two bug reports are not the only bug reports of the JVM crashing in SWT, however.
If you aren't using the Browser widget then these suggestions won't help you. In that case, you can search for a list of SWT bugs causing a JVM crash. If none of those are your issue, then I highly recommend that you open a bug report with SWT.
I have the same problem with a JNLP application that I have been using for a long time and is pretty reliable. The problem started immediately after I upgraded from Windows 7 to Windows 10. According to my investigation, it is most likely a bug in Win 10.
The following is not a solution, but an ugly workaround. In jre/bin directory, there is javaws.exe. If I right-clicked /Properties/Compatibility and ticked Run this program as an administrator, the JNLP app started to work.
Please, be aware that this approach could cause security issues and use it only if you have no other option and 100% know what you are doing.

What can I do if a Java VM crashes repeatedly?

What is the best practice to solve a Java VM crash if the follow conditions are true:
No own or third party native code. 100% pure java
The same program run on many other system without any problems.
PS: With VM crash I means that the VM write a dump file like hs_err_pid1234.log and terminate.
Read the hs_err_pid1234.log file (or whatever the error log file name is). There are usually clues in there. The next step depends on what you discover in the log.
Yes, it could be a bug in the specific version of the JVM implementation you are using, but I have also seen problems caused by memory fragmentation in the operating system. Windows, for example, is prone to pin dlls at inappropriate locations, and fail to allocate a contiguous block of memory when the JVM asks for it as a result. Other out opf memory problems can also manifest themselves through crash dumps of this type.
Update or replace your JVM. If you currently have the newest version, then try an older one, or if you don't have the latest version, try updating to it. Maybe its a known issue in your particular version?
Assuming the JVM version across machines is the same:
Figure out what is different about the machine where the JVM is crashing. Same OS and OS version? We have problems with JVMs crashing on a particular version of Red Hat for example. And we have also found some older Red Hat versions unable to cope with extra memory properly, resulting in running out of swap space. (Our solution was to upgrade RedHat).
Also, is the program doing exactly the same thing across machines? Is it accessing a shared filesystem? Is the file system mounted similarly on your machines (SMB/NFS etc)? Something must be different.
The log file should give you some idea of where the crash occurred (malloc for example).
Take a look at the stacktraces in the dump file, as it should tell you what was going on when the crash occurred.
As well as digging into the hs_err dump file, I'd also submit it to Sun or whomever made your JVM (I believe there are instructions in how to do so at the top of the file?). It can't hurt.
32bit? 64bit? Amount of ram in client machine? processor? os? See if there is any connection between the systems. A connection may lead to a clue. If all else fails, consider using different major/minor versions of the JVM. Also, if the problem JUST started can you get to a time (via version control) where the program didn't crash? Look through the hs_err log, you may get an idea of what caused the crash. It could be a version of some other client library the JVM uses. Lastly, run the program in debug/profile and maybe you'll see some symptons before the crash (assuming you can duplicate it)

Categories