We Keep Coding

Why when processing JVMTI EVENT_EXCEPTION the JVM sometimes crashes or starts slowly?

I decided using JVMTI to register all exceptions occured in JVM and build Histogram: ClassName_method_ExceptionName:count
Agent attached like this: taskset -c 10,12 java -agentpath:./jni-agent.so ${JVM_OPTION} -cp module.jar org.springframework.boot.loader.PropertiesLauncher &
I using this code for JNI agent:
enter code here
#include <stdio.h>
#include <jvmti.h>
#include <jni.h>
#include <map>
#include <string>
#include <cstring>
#include "utils.h"
long excCount;
using namespace std;
map<string, long> mapExceptions;
map<string, long> mapContendedThreadStack;
void mapAddException(map<string,long> &inMap, string key){
map<string, long> :: iterator it = inMap.find(key);
if(it == inMap.end()){
inMap.insert(pair<string, long>(key,1));
return;
}
inMap.find(key)->second ++;
return;
}
..
..
void JNICALL ExceptionCallback(jvmtiEnv* jvmti, JNIEnv *env, jthread thread, jmethodID method, jlocation location,
jobject exception, jmethodID catch_method, jlocation catch_location) {
excCount ++;
char *class_name;
jclass exception_class = env->GetObjectClass(exception);
jvmti->GetClassSignature(exception_class, &class_name, NULL);
char* method_name;
jvmti->GetMethodName(method, &method_name, NULL, NULL);
jclass holder;
jvmti->GetMethodDeclaringClass(method, &holder);
char* holder_name;
jvmti->GetClassSignature(holder, &holder_name, NULL);
if(strstr(holder_name, "java/lang") != NULL
|| strstr(holder_name, "java/net") != NULL
|| strstr(holder_name, "sun/reflect") != NULL
|| strstr(holder_name, "org/springframework/boot/loader/jar/JarURLConnection") != NULL
|| strstr(holder_name, "okhttp3/internal/connection/RealConnection") != NULL
|| strstr(holder_name, "okio/AsyncTimeout") != NULL ){
jvmti->Deallocate((unsigned char*) method_name);
jvmti->Deallocate((unsigned char*) holder_name);
jvmti->Deallocate((unsigned char*) class_name);
return;
}
string trimres = trimClassName(convertToString(holder_name, strlen(holder_name)));
char buftotal[1024];
snprintf(buftotal, sizeof(buftotal) - 1, "%s#%s()_%s", trimres.c_str(), method_name, trimClassName(class_name).c_str());
replacechar(buftotal, '/', '.');
//mapAddException(mapExceptions, buftotal); <-- HERE
jvmti->Deallocate((unsigned char*) method_name);
jvmti->Deallocate((unsigned char*) holder_name);
jvmti->Deallocate((unsigned char*) class_name);
}
extern "C" JNIEXPORT jint JNICALL Agent_OnLoad(JavaVM *vm, char *options, void *reserved) {
jvmtiCapabilities capabilities;
jvmtiEventCallbacks callbacks;
jvmtiEnv *jvmti;
vm->GetEnv((void **) &jvmti, JVMTI_VERSION_1_0);
printf("\nNative agent loaded by -agent.....\n");
capabilities = {0};
capabilities.can_generate_exception_events = 1;
jvmti->AddCapabilities(&capabilities);
callbacks = {0};
callbacks.Exception = ExceptionCallback;
jvmti->SetEventCallbacks(&callbacks, sizeof(callbacks));
jvmti->SetEventNotificationMode(JVMTI_ENABLE, JVMTI_EVENT_EXCEPTION, NULL);
return 0;
}
With commented line mapAddException(...) the JVM started much longer than without the JNI-agent.
Is this Ok ?
I forgot to say - this is Spring Boot application with "Hello World" all ok :)
But when I uncommented mapAddException(...) JVM sometimes crashed. Not at all, sometimes(~ 2-4 of 50).
This is cut from CrashDump:
..
# SIGSEGV (0xb) at pc=0x00007f32781bf0b4, pid=47559, tid=0x00007f31e29e1700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_231-b11) (build 1.8.0_231-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.231-b11 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libstdc++.so.6+0x750b4]
..
#
--------------- T H R E A D ---------------
Current thread (0x00007f3275d82000): JavaThread "tomcat-thread-16" daemon [_thread_in_native, id=47682, stack(0x00007f31e28e1000,0x00007f31e29e2000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000010
Registers:
RAX=0x0000000000000000, RBX=0x00007f3190009340, RCX=0x0000000000000001, RDX=0x00007f3274ba0eb0
RSP=0x00007f31e29dc7c8, RBP=0x00007f327414fea0, RSI=0x00007f32786541d0, RDI=0x00007f327414fea0
R8 =0x0000000000000002, R9 =0x0000000000000030, R10=0x000000000000000c, R11=0x00007f327a316f40
R12=0x00007f31a00504f0, R13=0x00007f32786541c8, R14=0x00007f32786541d0, R15=0x00007f3190009340
RIP=0x00007f32781bf0b4, EFLAGS=0x0000000000010283, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e
....
The Current thread are different every time at CrashDump.

The crash reason is simple: your code is not thread safe.
Exception callback is invoked on the same thread that throws an exception. If multiple threads throw an exception concurrently, the callback is also invoked concurrently. std::map is not thread safe: concurrent access may cause a crash inside libstdc++.
The reason of slowness is more interesting.
How to analyze performance issues related to interaction between JVM and native? With async-profiler, of course. When profiling application startup, it's convenient to start profiler as an agent.
java -agentpath:/path/to/libasyncProfiler.so=start,event=cpu,file=out.svg -agentpath:./your-agent.so -cp ...
When I compared profiles with and without your agent, I noticed one major difference. With the agent, there was a noticeable [deoptimization] block on the left side:
Why does deoptimization happen? async-profiler will help again: let's profile Deoptimization::deoptimize event now. It's not really an event, this is just a function name in HotSpot JVM (async-profiler can trace invocations of native functions).
java -agentpath:/path/to/libasyncProfiler.so=start,event=Deoptimization::deoptimize,file=deopt.svg ...
There was more than 8000 deoptimization events! Here is just a small part of the graph:
Most deoptimizations originate from exception_handler_for_pc_helper. If we look at the source code of this function, we'll see an interesting comment:
if (JvmtiExport::can_post_on_exceptions()) {
// To ensure correct notification of exception catches and throws
// we have to deoptimize here.
<...>
// We don't really want to deoptimize the nmethod itself since we
// can actually continue in the exception handler ourselves but I
// don't see an easy way to have the desired effect.
Deoptimization::deoptimize_frame(thread, caller_frame.id());
What this means is - when JVM TI Exception notifications are on, HotSpot forces deoptimization every time an exception happens in the compiled code! This can surely hit performance if exceptions are frequent.
How to analyze exceptions without JVM TI overhead then?
At this point you may already guess the answer - with async-profiler again :) Besides CPU and native function profiling, it can also intercept any Java method call.
To find all places where exceptions and errors are created, we can just intercept Throwable constructors. To do so, specify java.lang.Throwable.<init> as async-profiler event. This time it's handy to produce a reversed call tree (add reversed,file=tree.html options), so that throwables will be grouped by the type and the construction site.
java -agentpath:/path/to/libasyncProfiler.so=start,event=java.lang.Throwable.\<init\>,reverse,file=tree.html ...
This will produce an interactive HTML tree, where you'll find all exceptions with counters and full stack traces:
Ouch! A simple Spring Boot application throws 4700 exceptions: most of them are ClassNotFoundException (90%) and NoSuchMethodException (7%).
What is important, async-profiler uses bytecode instrumentation for method tracing, so it does not suffer from the performance problem of JVM TI exception callback.
Bonus
When profiling your agent for the first time, besides the deoptimization issue, I also found a decent number of stack traces where CPU time is spent in your ExceptionCallback calling GetMethodDeclaringClass and GetMethodName:
This is actually a performance bug in JDK 8, which I reported 3 years ago: JDK-8185348. At last, it has been fixed in JDK 8u281. However, there was no such problem in JDK 9+.
In short, since all JVM TI Method functions are slow, it is a good idea to cache the result of these functions, so that the agent will not call JVM TI function for the same method more than once.

JNINativeInterface_ * is null

I am developing a JNI application. However JNINativeInterface_ * inside the struct struct JNIEnv_ is null, hence causing any call for JNI (example : env->NewStringUTF(...)) functions to throw a segmentation fault error. JNIEnv_ struct looks like this :
struct JNIEnv_ {
const struct JNINativeInterface_ *functions;
.
.
.
I honestly dont know how to fix this, as I think java is supposed to fill this when you make a call to System.loadLibrary(...). I appreciate any help.

Two possibilities:
1) Your C++ code is called from Java as native function:
For example:
JNIEXPORT void JNICALL Java_MyJNative_doSomething(JNIEnv *env, jobject jo)
{
std::cout << "doSomething() : ";
}
The JNIEnv* pointer is given to you by the Java environment.
2) Your C++ code calls the Java code via JNI:
In this case you have to setup a JavaVM* and JNIEnv* pointer, following the JNI invocation logic. See for example this article.
A special case to be mentionned, if you have already invoked JNIEnv* but you work with multiple threads on C++ side. Then every thread has to attach to the JNIEnv* using:
JNIEnv *env; // Pointer to native interface
jint rc = jvm->AttachCurrentThread ((void **)&env, nullptr); // Attach without addutiobal parameters
// where jvm is a valid JavaVM*

Generate Java Heap Dump on uncaught Exception

I try to generate a Heap Dump when a uncaught exception is fired. I tried using jmap, but because the process is finished when the exception happens this is not possible.
Using a UncaughtExceptionHandler is no option either, because I only have the binaries of the programs that is executed.
Can anyone help me?
EDIT: It is important that the technique is available through a command line or similar, because I need to automated this. Using a GUI is no option

This can be achieved with JVMTI agent that will listen to VMDeath event and then use JMM interface to initiate Heap Dump.
Here is a sample source code of such JVMTI agent:
#include <jvmti.h>
#include <string.h>
#include <stdio.h>
#include "jmm.h"
JNIEXPORT void* JNICALL JVM_GetManagement(jint version);
void JNICALL VMDeath(jvmtiEnv* jvmti, JNIEnv* jni) {
JmmInterface* jmm = (JmmInterface*) JVM_GetManagement(JMM_VERSION_1_0);
if (jmm == NULL) {
printf("Sorry, JMM is not supported\n");
} else {
jstring path = (*jni)->NewStringUTF(jni, "dump.hprof");
jmm->DumpHeap0(jni, path, JNI_TRUE);
printf("Heap dumped\n");
}
}
JNIEXPORT jint JNICALL Agent_OnLoad(JavaVM* vm, char* options, void* reserved) {
jvmtiEnv* jvmti;
(*vm)->GetEnv(vm, (void**)&jvmti, JVMTI_VERSION_1_0);
jvmtiEventCallbacks callbacks;
memset(&callbacks, 0, sizeof(callbacks));
callbacks.VMDeath = VMDeath;
(*jvmti)->SetEventCallbacks(jvmti, &callbacks, sizeof(callbacks));
(*jvmti)->SetEventNotificationMode(jvmti, JVMTI_ENABLE, JVMTI_EVENT_VM_DEATH, NULL);
return 0;
}
After you've compiled it into the shared library (libdump.so) run Java with -agentpath option:
java -agentpath:/path/to/libdump.so MainClass
If you wish to handle uncaught exceptions instead of waiting for VMDeath, you may use similar technique to install callback for Exception event. Look here for an example.

Try to put your processing into a deamon thread. this way you can access it with memory analysis tools. JVisualVM is a JDK tool you can find it in your JAVA_HOME\bin.
There are also another way, called dump analyser. You run your application with these JVM parameters :
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath="your_path"
But this is used only if you have an OutOfMemoryError. Try to find if it's possible to generate the dump for any performance.
Another good eclipse based tool is MemoryAnalyzer

I would like to suggest Java Visual VM. It can connect dynamically. I found it useful. You might want to give a try.

Calling java code from stand-alone native code

In android OS I want to call an user defined java class API from a standalone code.
i.e. If there is a class "HelloWorldActivity" which has "getint" API . I would like to call this from a native app "nativecaller"
I found post related to this however I was not clear how the implementation was done.
https://groups.google.com/forum/#!topic/android-ndk/_JidHzVWHM8
So here is the code snippet:
#include <jni.h>
#include <cutils/log.h>
#include <stdlib.h>
int main(){
JavaVM *jvm;
JNIEnv *env;
JavaVMInitArgs vm_args;
JavaVMOption options[1];
options[0].optionString = "-Djava.class.path=/data/";
vm_args.version = JNI_VERSION_1_6;
vm_args.options = options;
vm_args.nOptions = 1;
vm_args.ignoreUnrecognized = JNI_FALSE;
/* Create the Java VM */
int res = JNI_CreateJavaVM(&jvm, &env, &vm_args);
if(!res){
/* invoke the Main.test method using the JNI */
jclass cls = env->FindClass("com/abc/mypackage/HelloWorld"); //it is not able to find the class
if(!cls)LOGE("\n\n\nclass not found!!!");
else{
jmethodID mid = env->GetMethodID(cls, "getint", "(V)I");
env->CallStaticVoidMethod(cls, mid,10);
}
/* We are done. */
jvm->DestroyJavaVM();
}
else
LOGE("\n\n\n\n CreateJAVAVM failed!!");
}
FindClass is returning null.
1.Is it possible to access class inside an activity (an apk)
2.What should -Djava.class.path point to ?
Any input is appreciated!

Dalvik provides a command called dalvikvm, which isn't too far removed from what you're trying to do. It's just a command-line wrapper for libdvm.so (try adb shell dalvikvm -help). You can see the source code here.
Try a quick test: instead of looking up your application class, look up something that you know will be there (say, java/lang/String). That will tell you if the VM is able to do anything at all.
On a device, BOOTCLASSPATH will already be configured in your environment (adb shell printenv BOOTCLASSPATH), but CLASSPATH will not. Set the CLASSPATH environment variable to a colon-separated list of .jar or .apk files, not a list of directories.
You will need to run as root, so that your command-line application has permission to create an entry in /data/dalvik-cache for your APK. (If such an entry already exists, you may not need to be root.)
If something doesn't work, check the logcat output for details.

How do I get an error message when failing to load a JVM via JNI?

I would like to retrieve an error message that explains why the jvm failed to load. From the examples provided here:
http://java.sun.com/docs/books/jni/html/invoke.html
I extracted this example:
/* Create the Java VM */
res = JNI_CreateJavaVM(&jvm, (void**)&env, &vm_args);
if (res < 0) {
// retrieve verbose error here?
fprintf(stderr, "Can't create Java VM\n");
exit(1);
}
In my specific case I'm providing invalid arguments in the vm_args and would expect to see what I get on the command line: "Unrecognized option: -foo=bar"
On further testing it looks like the jvm is putting the message I want to stdout or stderr. I believe I would need to capture stdout and stderr to get the error I'm looking for (unless of course there is a simpler way). I'm coding in C++ so if someone can show a way to capture the error into a stringstream that would be ideal.
Thanks,
Randy

I was able to get what I needed by using the "vfprintf" option described here:
https://web.archive.org/web/20111229234347/http://java.sun.com/products/jdk/faq/jnifaq-old.html
although I used the jdk1.2 options. This code snippet summarizes my solution:
static string jniErrors;
static jint JNICALL my_vfprintf(FILE *fp, const char *format, va_list args)
{
char buf[1024];
vsnprintf(buf, sizeof(buf), format, args);
jniErrors += buf;
return 0;
}
...
JavaVMOption options[1];
options[0].optionString = "vfprintf";
options[0].extraInfo = my_vfprintf;
JavaVMInitArgs vm_args;
memset(&vm_args, 0, sizeof(vm_args));
vm_args.nOptions = 1;
vm_args.options = options;
vm_args.version = JNI_VERSION_1_4;
vm_args.ignoreUnrecognized = JNI_FALSE;
JNIEnv env;
JavaVM jvm;
jint res = JNI_CreateJavaVM(&jvm, (void**)&env, &vm_args);
if (res != JNI_OK)
setError(jniErrors);
jniErrors.clear();
Also interesting was that I could not for the life of me capture stdout or stderr correctly with any of the freopen or dup2 tricks. I could load my own dll and redirect correctly but not the jvm. Anyway, it's better this way because I wanted the errors in memory rather than in a file.

When I wrote this code I would also get the error via stdout/stderr.
The best way to redirect stdout/stderr in your process is by using freopen. Here is a StackOverflow question specifically about that subject.
However, once that call is passed, you will then have a JNIEnv, and all further error checking can and should be done by calling JNIEnv::ExceptionOccurred(), which will may return a Throwable object that you can then interrogate with your JNI code.

After getting a hold of stdout and stderr, which you'll need anyway, add -Xcheck:jni to your jvm command line to get extra jni-related warnings from the jvm.