We Keep Coding

Why when processing JVMTI EVENT_EXCEPTION the JVM sometimes crashes or starts slowly?

I decided using JVMTI to register all exceptions occured in JVM and build Histogram: ClassName_method_ExceptionName:count
Agent attached like this: taskset -c 10,12 java -agentpath:./jni-agent.so ${JVM_OPTION} -cp module.jar org.springframework.boot.loader.PropertiesLauncher &
I using this code for JNI agent:
enter code here
#include <stdio.h>
#include <jvmti.h>
#include <jni.h>
#include <map>
#include <string>
#include <cstring>
#include "utils.h"
long excCount;
using namespace std;
map<string, long> mapExceptions;
map<string, long> mapContendedThreadStack;
void mapAddException(map<string,long> &inMap, string key){
map<string, long> :: iterator it = inMap.find(key);
if(it == inMap.end()){
inMap.insert(pair<string, long>(key,1));
return;
}
inMap.find(key)->second ++;
return;
}
..
..
void JNICALL ExceptionCallback(jvmtiEnv* jvmti, JNIEnv *env, jthread thread, jmethodID method, jlocation location,
jobject exception, jmethodID catch_method, jlocation catch_location) {
excCount ++;
char *class_name;
jclass exception_class = env->GetObjectClass(exception);
jvmti->GetClassSignature(exception_class, &class_name, NULL);
char* method_name;
jvmti->GetMethodName(method, &method_name, NULL, NULL);
jclass holder;
jvmti->GetMethodDeclaringClass(method, &holder);
char* holder_name;
jvmti->GetClassSignature(holder, &holder_name, NULL);
if(strstr(holder_name, "java/lang") != NULL
|| strstr(holder_name, "java/net") != NULL
|| strstr(holder_name, "sun/reflect") != NULL
|| strstr(holder_name, "org/springframework/boot/loader/jar/JarURLConnection") != NULL
|| strstr(holder_name, "okhttp3/internal/connection/RealConnection") != NULL
|| strstr(holder_name, "okio/AsyncTimeout") != NULL ){
jvmti->Deallocate((unsigned char*) method_name);
jvmti->Deallocate((unsigned char*) holder_name);
jvmti->Deallocate((unsigned char*) class_name);
return;
}
string trimres = trimClassName(convertToString(holder_name, strlen(holder_name)));
char buftotal[1024];
snprintf(buftotal, sizeof(buftotal) - 1, "%s#%s()_%s", trimres.c_str(), method_name, trimClassName(class_name).c_str());
replacechar(buftotal, '/', '.');
//mapAddException(mapExceptions, buftotal); <-- HERE
jvmti->Deallocate((unsigned char*) method_name);
jvmti->Deallocate((unsigned char*) holder_name);
jvmti->Deallocate((unsigned char*) class_name);
}
extern "C" JNIEXPORT jint JNICALL Agent_OnLoad(JavaVM *vm, char *options, void *reserved) {
jvmtiCapabilities capabilities;
jvmtiEventCallbacks callbacks;
jvmtiEnv *jvmti;
vm->GetEnv((void **) &jvmti, JVMTI_VERSION_1_0);
printf("\nNative agent loaded by -agent.....\n");
capabilities = {0};
capabilities.can_generate_exception_events = 1;
jvmti->AddCapabilities(&capabilities);
callbacks = {0};
callbacks.Exception = ExceptionCallback;
jvmti->SetEventCallbacks(&callbacks, sizeof(callbacks));
jvmti->SetEventNotificationMode(JVMTI_ENABLE, JVMTI_EVENT_EXCEPTION, NULL);
return 0;
}
With commented line mapAddException(...) the JVM started much longer than without the JNI-agent.
Is this Ok ?
I forgot to say - this is Spring Boot application with "Hello World" all ok :)
But when I uncommented mapAddException(...) JVM sometimes crashed. Not at all, sometimes(~ 2-4 of 50).
This is cut from CrashDump:
..
# SIGSEGV (0xb) at pc=0x00007f32781bf0b4, pid=47559, tid=0x00007f31e29e1700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_231-b11) (build 1.8.0_231-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.231-b11 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libstdc++.so.6+0x750b4]
..
#
--------------- T H R E A D ---------------
Current thread (0x00007f3275d82000): JavaThread "tomcat-thread-16" daemon [_thread_in_native, id=47682, stack(0x00007f31e28e1000,0x00007f31e29e2000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000010
Registers:
RAX=0x0000000000000000, RBX=0x00007f3190009340, RCX=0x0000000000000001, RDX=0x00007f3274ba0eb0
RSP=0x00007f31e29dc7c8, RBP=0x00007f327414fea0, RSI=0x00007f32786541d0, RDI=0x00007f327414fea0
R8 =0x0000000000000002, R9 =0x0000000000000030, R10=0x000000000000000c, R11=0x00007f327a316f40
R12=0x00007f31a00504f0, R13=0x00007f32786541c8, R14=0x00007f32786541d0, R15=0x00007f3190009340
RIP=0x00007f32781bf0b4, EFLAGS=0x0000000000010283, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e
....
The Current thread are different every time at CrashDump.

The crash reason is simple: your code is not thread safe.
Exception callback is invoked on the same thread that throws an exception. If multiple threads throw an exception concurrently, the callback is also invoked concurrently. std::map is not thread safe: concurrent access may cause a crash inside libstdc++.
The reason of slowness is more interesting.
How to analyze performance issues related to interaction between JVM and native? With async-profiler, of course. When profiling application startup, it's convenient to start profiler as an agent.
java -agentpath:/path/to/libasyncProfiler.so=start,event=cpu,file=out.svg -agentpath:./your-agent.so -cp ...
When I compared profiles with and without your agent, I noticed one major difference. With the agent, there was a noticeable [deoptimization] block on the left side:
Why does deoptimization happen? async-profiler will help again: let's profile Deoptimization::deoptimize event now. It's not really an event, this is just a function name in HotSpot JVM (async-profiler can trace invocations of native functions).
java -agentpath:/path/to/libasyncProfiler.so=start,event=Deoptimization::deoptimize,file=deopt.svg ...
There was more than 8000 deoptimization events! Here is just a small part of the graph:
Most deoptimizations originate from exception_handler_for_pc_helper. If we look at the source code of this function, we'll see an interesting comment:
if (JvmtiExport::can_post_on_exceptions()) {
// To ensure correct notification of exception catches and throws
// we have to deoptimize here.
<...>
// We don't really want to deoptimize the nmethod itself since we
// can actually continue in the exception handler ourselves but I
// don't see an easy way to have the desired effect.
Deoptimization::deoptimize_frame(thread, caller_frame.id());
What this means is - when JVM TI Exception notifications are on, HotSpot forces deoptimization every time an exception happens in the compiled code! This can surely hit performance if exceptions are frequent.
How to analyze exceptions without JVM TI overhead then?
At this point you may already guess the answer - with async-profiler again :) Besides CPU and native function profiling, it can also intercept any Java method call.
To find all places where exceptions and errors are created, we can just intercept Throwable constructors. To do so, specify java.lang.Throwable.<init> as async-profiler event. This time it's handy to produce a reversed call tree (add reversed,file=tree.html options), so that throwables will be grouped by the type and the construction site.
java -agentpath:/path/to/libasyncProfiler.so=start,event=java.lang.Throwable.\<init\>,reverse,file=tree.html ...
This will produce an interactive HTML tree, where you'll find all exceptions with counters and full stack traces:
Ouch! A simple Spring Boot application throws 4700 exceptions: most of them are ClassNotFoundException (90%) and NoSuchMethodException (7%).
What is important, async-profiler uses bytecode instrumentation for method tracing, so it does not suffer from the performance problem of JVM TI exception callback.
Bonus
When profiling your agent for the first time, besides the deoptimization issue, I also found a decent number of stack traces where CPU time is spent in your ExceptionCallback calling GetMethodDeclaringClass and GetMethodName:
This is actually a performance bug in JDK 8, which I reported 3 years ago: JDK-8185348. At last, it has been fixed in JDK 8u281. However, there was no such problem in JDK 9+.
In short, since all JVM TI Method functions are slow, it is a good idea to cache the result of these functions, so that the agent will not call JVM TI function for the same method more than once.

How to set classpath of a running JVM in C++(JNI)?

In C++ (JNI), I get an already running JVM using the following JNI function:
JNI_GetCreatedJavaVMs(&jvm,1,&vmnb);
Then I attach the current thread using the following code:
jvm->AttachCurrentThread((void**)&env, NULL);
Question: How can I set classpath in that case? Thanks.
Note: Creating a new JVM and passing the classpath in vm_args to the new JVM is not an option for me.

As it is possible to append the classpath inside Java, so I found an alternative way to set classpath inside C++ through Java. As I'm already running JVM, I use the append classpath method (void addPath(String path), that is posted in this answer), inside the Java program that is already running in the JVM. I access the addPath java method from C++ using JNI calls to append the classpath. The classpath passed to addPath method from C++ should not include "-Djava.class.path" and it should be just complete path to the .jar file, i.e. "C:\\folder\\abc.jar". So sequence is: 1) get already running JVM, attach current thread to the JVM, get JNI environment pointer and then call addPath java function (of another running thread) from C++ using JNI. I can now access the classes of new classpath (.jar file) successfully from C++.

I ran into this issue, and I did not have the option to call back into the JVM, so I implemented the whole add_path busniess on the JNI side.
void add_path(JNIEnv* env, const std::string& path)
{
const std::string urlPath = "file:/" + path;
jclass classLoaderCls = env->FindClass("java/lang/ClassLoader");
jmethodID getSystemClassLoaderMethod = env->GetStaticMethodID(classLoaderCls, "getSystemClassLoader", "()Ljava/lang/ClassLoader;");
jobject classLoaderInstance = env->CallStaticObjectMethod(classLoaderCls, getSystemClassLoaderMethod);
jclass urlClassLoaderCls = env->FindClass("java/net/URLClassLoader");
jmethodID addUrlMethod = env->GetMethodID(urlClassLoaderCls, "addURL", "(Ljava/net/URL;)V");
jclass urlCls = env->FindClass("java/net/URL");
jmethodID urlConstructor = env->GetMethodID(urlCls, "<init>", "(Ljava/lang/String;)V");
jobject urlInstance = env->NewObject(urlCls, urlConstructor, env->NewStringUTF(urlPath.c_str()));
env->CallVoidMethod(classLoaderInstance, addUrlMethod, urlInstance);
std::cout << "Added " << urlPath << " to the classpath." << std::endl;
}

Integrating log4j with SWIG/JNI?

I'm working on a Java program which has a C++ module. I'd like for my C++'s stdout/stderr to be sent to a slf4j/log4j logger.
Some possibilities:
Have my C++ module capture stdout/stderr in a string which is returned and sent to a logger. A downside is the C++ module might be expensive and this would be asynchronous.
Have my Java software create/manage a temp file to which the C++ module can write. Upon completion, my software would read and then delete the file and send the data to slf4j.
is there some way to get an ostream from log4j I could pass through my swig interface?
it looks like I could create a Director in swig to pass a string to Java from C++ to forward to the logger.
How have other people solved this problem?

There are three problems you need to solve to get C++ (called via SWIG generated wrappers) logging to log4j:
How we capture the C++ output (i.e. hook the iostream objects).
How we insert this hooking into the generated wrapper.
How we actually call log4j with the stuff we've captured.
In answering this question I wrote the following header file to demonstrate how my solution works:
#include <iostream>
void test1() {
std::cout << "OUT: " << "test1\n";
std::cerr << "ERR: " << "test1\n";
}
struct HelloWorld {
static void test2() {
std::cout << "OUT: " << "test2\n";
std::cerr << "ERR: " << "test2\n";
}
void test3() const {
std::cout << "OUT: " << "test3\n";
std::cerr << "ERR: " << "test3\n";
}
};
At the end of this I wanted to see both std::cout and std::cerr going to log4j in the correct order. I've answered that question before, in this instance to keep it simple and portable I started out using rdbuf() to swap the internal buffer used by std::cout and std::cerr to one that I'd actually created inside a std::stringstream, something like:
std::stringstream out; // Capture into this
// Save state so we can restore it
auto old_buf = std::cout.rdbuf();
// Swap buffer on cout
std::cout.rdbuf(out.rdbuf());
// Do the real call to C++ here
// ...
// Reset things
std::cout.rdbuf(old_buf);
// Walk through what we captured in out
Of course this won't capture output from libc functions (printf() etc.) or system calls (write() etc.) but it will get all your standard C++ output.
So that's problem #1 crossed off our list. For problem #2 SWIG has the %exception directive that matches what we want to do quite nicely, it gives us a chance to execute C++ code before and after a call into a wrapped function gets dispatched. In the above example all we need to do is use the special variable $action to get the substitution to happen where the comment "do the real call to C++ here".
For problem #3 we need to make some Java calls happen. I started out thinking that JNI wouldn't be too bad, perhaps a little verbose. Basically all we want to do is duplicate the following Java code (from the log4j docs):
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
public class HelloWorld {
private static final Logger logger = LogManager.getLogger("HelloWorld");
public static void main(String[] args) {
logger.info("Hello, World!");
}
}
but inside JNI rather than Java and passing the right string into the getLogger call.
So putting this all together into a SWIG interface we get:
%module test
%{
#include "test.hh"
#include <sstream>
#include <cassert>
static const char *defaultLogname="$module"; // Use if we're not in a class
%}
// Exception handling for all wrapped calls
%exception {
// Hook output into this:
std::stringstream out;
// Save old states
auto old_outbuf = std::cout.rdbuf();
auto old_errbuf = std::cerr.rdbuf();
// Do actual buffer switch
std::cout.rdbuf(out.rdbuf());
std::cerr.rdbuf(out.rdbuf());
try {
$action
}
catch (...) {
// TODO: use RAII instead of poor finally substitute?
std::cout.rdbuf(old_outbuf);
std::cerr.rdbuf(old_errbuf);
throw;
}
// Restore state
std::cout.rdbuf(old_outbuf);
std::cerr.rdbuf(old_errbuf);
// JNI calls to find mid and instance for Logger.error(String) for the right name
static const std::string class_name = "$parentclassname";
// prepare static call to org.apache.logging.log4j.LogManager.getLogger(String)
// start with class lookup:
jclass logmanagercls = JCALL1(FindClass, jenv, "org/apache/logging/log4j/LogManager");
assert(logmanagercls);
// find method ID for right overload of getLogger
jmethodID getloggermid = JCALL3(GetStaticMethodID, jenv, logmanagercls, "getLogger", "(Ljava/lang/String;)Lorg/apache/logging/log4j/Logger;");
assert(getloggermid);
// Prep name strign to pass into getLogger
jstring logname = JCALL1(NewStringUTF, jenv, (class_name.size() ? class_name.c_str(): defaultLogname));
// Actually get the Logger instance for us to use
jobject logger = JCALL3(CallStaticObjectMethod, jenv, logmanagercls, getloggermid, logname);
assert(logger);
// Lookup .error() method ID on logger, we need the jclass to start
jclass loggercls = JCALL1(GetObjectClass, jenv, logger);
assert(loggercls);
// and the method ID of the right overload
jmethodID errormid = JCALL3(GetMethodID, jenv, loggercls, "error", "(Ljava/lang/String;)V");
assert(errormid);
// Loop over all the lines we got from C++:
std::string msg;
while(std::getline(out, msg)) {
// Pass string into Java logger
jstring jmsg = JCALL1(NewStringUTF, jenv, msg.c_str());
JCALL3(CallVoidMethod, jenv, logger, errormid, jmsg);
}
}
// And of course actually wrap our test header
%include "test.hh"
I added some Java to prove this works:
public class run {
public static void main(String[] argv) {
System.loadLibrary("test");
test.test1();
HelloWorld.test2();
HelloWorld h1 = new HelloWorld();
h1.test3();
}
}
Compiled and ran with log4j 2.6 jars in current directory:
swig3.0 -c++ -java -Wall test.i
javac *.java
g++ -std=c++1y -Wall -Wextra -shared -o libtest.so test_wrap.cxx -I/usr/lib/jvm/default-java/include/ -I/usr/lib/jvm/default-java/include/linux -fPIC
LD_LIBRARY_PATH=. java -classpath log4j-api-2.6.jar:log4j-core-2.6.jar:. run
When runs gives:
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
22:31:08.383 [main] ERROR test - OUT: test1
22:31:08.384 [main] ERROR test - ERR: test1
22:31:08.386 [main] ERROR HelloWorld - OUT: test2
22:31:08.386 [main] ERROR HelloWorld - ERR: test2
22:31:08.386 [main] ERROR HelloWorld - OUT: test3
22:31:08.386 [main] ERROR HelloWorld - ERR: test3
Discussion points:
Is the JNI verbose? Definitely yes, but is that bad enough to go down another road? For me not (although the directors idea is viable it's not as simple as it sounds due to complexities with producing interfaces)
The timestamps on your log events will be "wrong" here because all the log information that gets captured will be pushed out after the underlying C++ function call returns, not during its execution
The cout/cerr messages get mixed and logged at the same level. I did this because of the previous point, if they weren't intermixed like this getting the relative ordering would be impossible without more work
If your C++ methods produce large quantities of output the buffer will grow rather large
If you've got a large C++ project I'd hope you had a better logging framework in use than just iostreams. If that's the case you're probably better off implementing its output interface(s) with an adaptor that passes directly to Java with timestamps, log level etc. info retained. (Or vice versa)
Even sticking with iostream the Boost iostreams library makes it easier1 to write something that would actually hook the output writes at the point they happen rather than the current hooking technique (but introduce bigger dependencies so it's a trade off). You could also do it without boost, but the basic_streambuf class is fairly unwieldy.
1 I can probably illustrate the boost version in this answer if you're interested.

JNINativeInterface_ * is null

I am developing a JNI application. However JNINativeInterface_ * inside the struct struct JNIEnv_ is null, hence causing any call for JNI (example : env->NewStringUTF(...)) functions to throw a segmentation fault error. JNIEnv_ struct looks like this :
struct JNIEnv_ {
const struct JNINativeInterface_ *functions;
.
.
.
I honestly dont know how to fix this, as I think java is supposed to fill this when you make a call to System.loadLibrary(...). I appreciate any help.

Two possibilities:
1) Your C++ code is called from Java as native function:
For example:
JNIEXPORT void JNICALL Java_MyJNative_doSomething(JNIEnv *env, jobject jo)
{
std::cout << "doSomething() : ";
}
The JNIEnv* pointer is given to you by the Java environment.
2) Your C++ code calls the Java code via JNI:
In this case you have to setup a JavaVM* and JNIEnv* pointer, following the JNI invocation logic. See for example this article.
A special case to be mentionned, if you have already invoked JNIEnv* but you work with multiple threads on C++ side. Then every thread has to attach to the JNIEnv* using:
JNIEnv *env; // Pointer to native interface
jint rc = jvm->AttachCurrentThread ((void **)&env, nullptr); // Attach without addutiobal parameters
// where jvm is a valid JavaVM*

JNI %1 is not a valid Win32 application

I'm running Netbeans on 64-bit Windows 8, with JDK 1.7_25 (64-bit), following the instructions for the Beginning JNI with NetBeans (https://netbeans.org/kb/docs/cnd/beginning-jni-linux.html)
The instructions are for linux, but the principle is the same for Windows I believe (generating a .dll file instead of .so, using win32 includes in the JDK, etc)
I have Cygwin64 installed as well as Cygwin32. Using Cygwin64, I'm able to generate a 64-bit DLL from my C/C++ Dynamic Library project. However, when I call System.load("path/to/JNITest.dll"), I get:
Exception in thread "main" java.lang.UnsatisfiedLinkError: C:\Users\Andrew\Documents\NetBeansProjects\JNITestLib\dist\JNITest.dll: %1 is not a valid Win32 application
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1957)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1882)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1843)
at java.lang.Runtime.load0(Runtime.java:795)
at java.lang.System.load(System.java:1061)
at jnitest.JNITest.main(JNITest.java:8)
Java Result: 1
From what I gather, this is most often the case when loading a 64-bit application on a 32-bit virtual machine, but my netbeans.conf is pointing to a 64-bit JVM.
Additionally, when I use the 32-bit version of Cygwin to compile things and run, I get
Can't load IA 32-bit .dll on a AMD 64-bit platform
I'm pretty sure I'm correctly generating the DLL file, it's just a simple HelloWorld printf to follow the JNI tutorial. I'm very new to JNI and C, so I'm not really sure where to start debugging. The best I've done is tried both 32 and 64-bit DLLs, and I've made sure my C compiler (Cygwin) is 64-bit, and my JVM is too.
I'd appreciate any insight!
Edit: Here are the included files
=== Java (JNITest.java) ===
package jnitest;
public class JNITest {
public static void main(String[] args) {
System.out.println("JVM: " + System.getProperty("sun.arch.data.model"));
System.load("C:\\Users\\Andrew\\Documents\\NetBeansProjects\\JNITestLib\\dist\\JNITest.dll");
new JNITest().doHello();
}
public native void doHello();
}
=== Generated javah header (jnitest_JNITest.h) ===
/* DO NOT EDIT THIS FILE - it is machine generated */
#include <jni.h>
/* Header for class jnitest_JNITest */
#ifndef _Included_jnitest_JNITest
#define _Included_jnitest_JNITest
#ifdef __cplusplus
extern "C" {
#endif
/*
* Class: jnitest_JNITest
* Method: doHello
* Signature: ()V
*/
JNIEXPORT void JNICALL Java_jnitest_JNITest_doHello
(JNIEnv *, jobject);
#ifdef __cplusplus
}
#endif
#endif
=== C (JNITest.c) ===
#include <jni.h>
#include "jnitest_JNITest.h"
JNIEXPORT void JNICALL Java_jnidemojava_Main_nativePrint
(JNIEnv *env, jobject obj)
{
printf("\nHello World from C\n");
}
Edit:
The problem seems to be with the DLL, since I can load other 64-bit DLLs just fine. I thought that Cygwin might be the problem, so I changed my compiler to MinGW-w64. It compiled fine, and the library loads, but now I get a new exception:
Exception in thread "main" java.lang.UnsatisfiedLinkError: jnitest.JNITest.doHello()V
at jnitest.JNITest.doHello(Native Method)
at jnitest.JNITest.main(JNITest.java:10)
Java Result: 1
Some more digging found that the error is thrown when ClassLoader reads libs.size() here:
// Invoked in the VM class linking code.
static long findNative(ClassLoader loader, String name) {
Vector<NativeLibrary> libs =
loader != null ? loader.nativeLibraries : systemNativeLibraries;
synchronized (libs) {
int size = libs.size();
for (int i = 0; i < size; i++) {
NativeLibrary lib = libs.elementAt(i);
long entry = lib.find(name);
if (entry != 0)
return entry;
}
}
return 0;
}
Edit: ANSWERS!
Finally figured it out.
Firstly, something was wrong with Cygwin64. Using a different 64-bit C compiler got rid of the not a valid win32 application error.
Secondly, my JNITest.c file's method signature was incorrect. It should have been:
Java_jnitest_JNITest_doHello
Instead of
Java_jnitest_Main_doHello
After changing that, it works!
(though I can't answer my own question for another 6 hours... so dum de dum)

Finally figured it out.
Firstly, something was wrong with Cygwin64. Using a different 64-bit C compiler got rid of the not a valid win32 application error.
Secondly, my JNITest.c file's method signature was incorrect. It should have been:
Java_jnitest_JNITest_doHello
Instead of
Java_jnitest_Main_doHello
After changing that, it works!