We Keep Coding

Why when processing JVMTI EVENT_EXCEPTION the JVM sometimes crashes or starts slowly?

I decided using JVMTI to register all exceptions occured in JVM and build Histogram: ClassName_method_ExceptionName:count
Agent attached like this: taskset -c 10,12 java -agentpath:./jni-agent.so ${JVM_OPTION} -cp module.jar org.springframework.boot.loader.PropertiesLauncher &
I using this code for JNI agent:
enter code here
#include <stdio.h>
#include <jvmti.h>
#include <jni.h>
#include <map>
#include <string>
#include <cstring>
#include "utils.h"
long excCount;
using namespace std;
map<string, long> mapExceptions;
map<string, long> mapContendedThreadStack;
void mapAddException(map<string,long> &inMap, string key){
map<string, long> :: iterator it = inMap.find(key);
if(it == inMap.end()){
inMap.insert(pair<string, long>(key,1));
return;
}
inMap.find(key)->second ++;
return;
}
..
..
void JNICALL ExceptionCallback(jvmtiEnv* jvmti, JNIEnv *env, jthread thread, jmethodID method, jlocation location,
jobject exception, jmethodID catch_method, jlocation catch_location) {
excCount ++;
char *class_name;
jclass exception_class = env->GetObjectClass(exception);
jvmti->GetClassSignature(exception_class, &class_name, NULL);
char* method_name;
jvmti->GetMethodName(method, &method_name, NULL, NULL);
jclass holder;
jvmti->GetMethodDeclaringClass(method, &holder);
char* holder_name;
jvmti->GetClassSignature(holder, &holder_name, NULL);
if(strstr(holder_name, "java/lang") != NULL
|| strstr(holder_name, "java/net") != NULL
|| strstr(holder_name, "sun/reflect") != NULL
|| strstr(holder_name, "org/springframework/boot/loader/jar/JarURLConnection") != NULL
|| strstr(holder_name, "okhttp3/internal/connection/RealConnection") != NULL
|| strstr(holder_name, "okio/AsyncTimeout") != NULL ){
jvmti->Deallocate((unsigned char*) method_name);
jvmti->Deallocate((unsigned char*) holder_name);
jvmti->Deallocate((unsigned char*) class_name);
return;
}
string trimres = trimClassName(convertToString(holder_name, strlen(holder_name)));
char buftotal[1024];
snprintf(buftotal, sizeof(buftotal) - 1, "%s#%s()_%s", trimres.c_str(), method_name, trimClassName(class_name).c_str());
replacechar(buftotal, '/', '.');
//mapAddException(mapExceptions, buftotal); <-- HERE
jvmti->Deallocate((unsigned char*) method_name);
jvmti->Deallocate((unsigned char*) holder_name);
jvmti->Deallocate((unsigned char*) class_name);
}
extern "C" JNIEXPORT jint JNICALL Agent_OnLoad(JavaVM *vm, char *options, void *reserved) {
jvmtiCapabilities capabilities;
jvmtiEventCallbacks callbacks;
jvmtiEnv *jvmti;
vm->GetEnv((void **) &jvmti, JVMTI_VERSION_1_0);
printf("\nNative agent loaded by -agent.....\n");
capabilities = {0};
capabilities.can_generate_exception_events = 1;
jvmti->AddCapabilities(&capabilities);
callbacks = {0};
callbacks.Exception = ExceptionCallback;
jvmti->SetEventCallbacks(&callbacks, sizeof(callbacks));
jvmti->SetEventNotificationMode(JVMTI_ENABLE, JVMTI_EVENT_EXCEPTION, NULL);
return 0;
}
With commented line mapAddException(...) the JVM started much longer than without the JNI-agent.
Is this Ok ?
I forgot to say - this is Spring Boot application with "Hello World" all ok :)
But when I uncommented mapAddException(...) JVM sometimes crashed. Not at all, sometimes(~ 2-4 of 50).
This is cut from CrashDump:
..
# SIGSEGV (0xb) at pc=0x00007f32781bf0b4, pid=47559, tid=0x00007f31e29e1700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_231-b11) (build 1.8.0_231-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.231-b11 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libstdc++.so.6+0x750b4]
..
#
--------------- T H R E A D ---------------
Current thread (0x00007f3275d82000): JavaThread "tomcat-thread-16" daemon [_thread_in_native, id=47682, stack(0x00007f31e28e1000,0x00007f31e29e2000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000010
Registers:
RAX=0x0000000000000000, RBX=0x00007f3190009340, RCX=0x0000000000000001, RDX=0x00007f3274ba0eb0
RSP=0x00007f31e29dc7c8, RBP=0x00007f327414fea0, RSI=0x00007f32786541d0, RDI=0x00007f327414fea0
R8 =0x0000000000000002, R9 =0x0000000000000030, R10=0x000000000000000c, R11=0x00007f327a316f40
R12=0x00007f31a00504f0, R13=0x00007f32786541c8, R14=0x00007f32786541d0, R15=0x00007f3190009340
RIP=0x00007f32781bf0b4, EFLAGS=0x0000000000010283, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e
....
The Current thread are different every time at CrashDump.

The crash reason is simple: your code is not thread safe.
Exception callback is invoked on the same thread that throws an exception. If multiple threads throw an exception concurrently, the callback is also invoked concurrently. std::map is not thread safe: concurrent access may cause a crash inside libstdc++.
The reason of slowness is more interesting.
How to analyze performance issues related to interaction between JVM and native? With async-profiler, of course. When profiling application startup, it's convenient to start profiler as an agent.
java -agentpath:/path/to/libasyncProfiler.so=start,event=cpu,file=out.svg -agentpath:./your-agent.so -cp ...
When I compared profiles with and without your agent, I noticed one major difference. With the agent, there was a noticeable [deoptimization] block on the left side:
Why does deoptimization happen? async-profiler will help again: let's profile Deoptimization::deoptimize event now. It's not really an event, this is just a function name in HotSpot JVM (async-profiler can trace invocations of native functions).
java -agentpath:/path/to/libasyncProfiler.so=start,event=Deoptimization::deoptimize,file=deopt.svg ...
There was more than 8000 deoptimization events! Here is just a small part of the graph:
Most deoptimizations originate from exception_handler_for_pc_helper. If we look at the source code of this function, we'll see an interesting comment:
if (JvmtiExport::can_post_on_exceptions()) {
// To ensure correct notification of exception catches and throws
// we have to deoptimize here.
<...>
// We don't really want to deoptimize the nmethod itself since we
// can actually continue in the exception handler ourselves but I
// don't see an easy way to have the desired effect.
Deoptimization::deoptimize_frame(thread, caller_frame.id());
What this means is - when JVM TI Exception notifications are on, HotSpot forces deoptimization every time an exception happens in the compiled code! This can surely hit performance if exceptions are frequent.
How to analyze exceptions without JVM TI overhead then?
At this point you may already guess the answer - with async-profiler again :) Besides CPU and native function profiling, it can also intercept any Java method call.
To find all places where exceptions and errors are created, we can just intercept Throwable constructors. To do so, specify java.lang.Throwable.<init> as async-profiler event. This time it's handy to produce a reversed call tree (add reversed,file=tree.html options), so that throwables will be grouped by the type and the construction site.
java -agentpath:/path/to/libasyncProfiler.so=start,event=java.lang.Throwable.\<init\>,reverse,file=tree.html ...
This will produce an interactive HTML tree, where you'll find all exceptions with counters and full stack traces:
Ouch! A simple Spring Boot application throws 4700 exceptions: most of them are ClassNotFoundException (90%) and NoSuchMethodException (7%).
What is important, async-profiler uses bytecode instrumentation for method tracing, so it does not suffer from the performance problem of JVM TI exception callback.
Bonus
When profiling your agent for the first time, besides the deoptimization issue, I also found a decent number of stack traces where CPU time is spent in your ExceptionCallback calling GetMethodDeclaringClass and GetMethodName:
This is actually a performance bug in JDK 8, which I reported 3 years ago: JDK-8185348. At last, it has been fixed in JDK 8u281. However, there was no such problem in JDK 9+.
In short, since all JVM TI Method functions are slow, it is a good idea to cache the result of these functions, so that the agent will not call JVM TI function for the same method more than once.

Java Exceptions counter on JVM HotSpot

I am wondering is it possible to log every exception which occurs on JVM level without changing application code? By every exception I mean caught and uncaught exception... I would like to analyze those logs later and group them by exception type (class) and simply count exceptions by type. I am using HotSpot ;)
Maybe there is smarter why of doing it? For example by any free profiler (YourKit has it but it is not free)? I think that JRockit has exception counter in management console, but don't see anything similar for HotSpot.

I believe there are free tools to do it, but even making your own tool is easy. JVMTI will help.
Here is a simple JVMTI agent I made to trace all exceptions:
#include <jni.h>
#include <jvmti.h>
#include <string.h>
#include <stdio.h>
void JNICALL ExceptionCallback(jvmtiEnv* jvmti, JNIEnv* env, jthread thread,
jmethodID method, jlocation location, jobject exception,
jmethodID catch_method, jlocation catch_location) {
char* class_name;
jclass exception_class = (*env)->GetObjectClass(env, exception);
(*jvmti)->GetClassSignature(jvmti, exception_class, &class_name, NULL);
printf("Exception: %s\n", class_name);
}
JNIEXPORT jint JNICALL Agent_OnLoad(JavaVM* vm, char* options, void* reserved) {
jvmtiEnv* jvmti;
jvmtiEventCallbacks callbacks;
jvmtiCapabilities capabilities;
(*vm)->GetEnv(vm, (void**)&jvmti, JVMTI_VERSION_1_0);
memset(&capabilities, 0, sizeof(capabilities));
capabilities.can_generate_exception_events = 1;
(*jvmti)->AddCapabilities(jvmti, &capabilities);
memset(&callbacks, 0, sizeof(callbacks));
callbacks.Exception = ExceptionCallback;
(*jvmti)->SetEventCallbacks(jvmti, &callbacks, sizeof(callbacks));
(*jvmti)->SetEventNotificationMode(jvmti, JVMTI_ENABLE, JVMTI_EVENT_EXCEPTION, NULL);
return 0;
}
To use it, make a shared library (.so) from the given source code, and run Java with -agentpath option:
java -agentpath:libextrace.so MyApplication
This will log all exception class names on stdout. ExceptionCallback also receives a thread, a method and a location where the exception occured, so you can extend the callback to print much more details.

JNI %1 is not a valid Win32 application

I'm running Netbeans on 64-bit Windows 8, with JDK 1.7_25 (64-bit), following the instructions for the Beginning JNI with NetBeans (https://netbeans.org/kb/docs/cnd/beginning-jni-linux.html)
The instructions are for linux, but the principle is the same for Windows I believe (generating a .dll file instead of .so, using win32 includes in the JDK, etc)
I have Cygwin64 installed as well as Cygwin32. Using Cygwin64, I'm able to generate a 64-bit DLL from my C/C++ Dynamic Library project. However, when I call System.load("path/to/JNITest.dll"), I get:
Exception in thread "main" java.lang.UnsatisfiedLinkError: C:\Users\Andrew\Documents\NetBeansProjects\JNITestLib\dist\JNITest.dll: %1 is not a valid Win32 application
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1957)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1882)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1843)
at java.lang.Runtime.load0(Runtime.java:795)
at java.lang.System.load(System.java:1061)
at jnitest.JNITest.main(JNITest.java:8)
Java Result: 1
From what I gather, this is most often the case when loading a 64-bit application on a 32-bit virtual machine, but my netbeans.conf is pointing to a 64-bit JVM.
Additionally, when I use the 32-bit version of Cygwin to compile things and run, I get
Can't load IA 32-bit .dll on a AMD 64-bit platform
I'm pretty sure I'm correctly generating the DLL file, it's just a simple HelloWorld printf to follow the JNI tutorial. I'm very new to JNI and C, so I'm not really sure where to start debugging. The best I've done is tried both 32 and 64-bit DLLs, and I've made sure my C compiler (Cygwin) is 64-bit, and my JVM is too.
I'd appreciate any insight!
Edit: Here are the included files
=== Java (JNITest.java) ===
package jnitest;
public class JNITest {
public static void main(String[] args) {
System.out.println("JVM: " + System.getProperty("sun.arch.data.model"));
System.load("C:\\Users\\Andrew\\Documents\\NetBeansProjects\\JNITestLib\\dist\\JNITest.dll");
new JNITest().doHello();
}
public native void doHello();
}
=== Generated javah header (jnitest_JNITest.h) ===
/* DO NOT EDIT THIS FILE - it is machine generated */
#include <jni.h>
/* Header for class jnitest_JNITest */
#ifndef _Included_jnitest_JNITest
#define _Included_jnitest_JNITest
#ifdef __cplusplus
extern "C" {
#endif
/*
* Class: jnitest_JNITest
* Method: doHello
* Signature: ()V
*/
JNIEXPORT void JNICALL Java_jnitest_JNITest_doHello
(JNIEnv *, jobject);
#ifdef __cplusplus
}
#endif
#endif
=== C (JNITest.c) ===
#include <jni.h>
#include "jnitest_JNITest.h"
JNIEXPORT void JNICALL Java_jnidemojava_Main_nativePrint
(JNIEnv *env, jobject obj)
{
printf("\nHello World from C\n");
}
Edit:
The problem seems to be with the DLL, since I can load other 64-bit DLLs just fine. I thought that Cygwin might be the problem, so I changed my compiler to MinGW-w64. It compiled fine, and the library loads, but now I get a new exception:
Exception in thread "main" java.lang.UnsatisfiedLinkError: jnitest.JNITest.doHello()V
at jnitest.JNITest.doHello(Native Method)
at jnitest.JNITest.main(JNITest.java:10)
Java Result: 1
Some more digging found that the error is thrown when ClassLoader reads libs.size() here:
// Invoked in the VM class linking code.
static long findNative(ClassLoader loader, String name) {
Vector<NativeLibrary> libs =
loader != null ? loader.nativeLibraries : systemNativeLibraries;
synchronized (libs) {
int size = libs.size();
for (int i = 0; i < size; i++) {
NativeLibrary lib = libs.elementAt(i);
long entry = lib.find(name);
if (entry != 0)
return entry;
}
}
return 0;
}
Edit: ANSWERS!
Finally figured it out.
Firstly, something was wrong with Cygwin64. Using a different 64-bit C compiler got rid of the not a valid win32 application error.
Secondly, my JNITest.c file's method signature was incorrect. It should have been:
Java_jnitest_JNITest_doHello
Instead of
Java_jnitest_Main_doHello
After changing that, it works!
(though I can't answer my own question for another 6 hours... so dum de dum)

Finally figured it out.
Firstly, something was wrong with Cygwin64. Using a different 64-bit C compiler got rid of the not a valid win32 application error.
Secondly, my JNITest.c file's method signature was incorrect. It should have been:
Java_jnitest_JNITest_doHello
Instead of
Java_jnitest_Main_doHello
After changing that, it works!

Java JNIEnv Segmentation Fault

I'm in the process of doing some Java<->.NET interop code using a native library between them, so far things are going fairly well.
However for some reason I cannot run any methods under JNIEnv.
System::String^ JNIStringToNet(JNIEnv * env, jstring js)
{
const char *buf = env->GetStringUTFChars(js, 0); // segfault
Now I can pass variables back and forth and do all other kinds of communication, so I'm figuring I haven't initialized this correctly or something.
I'm loading it like this:
this.Lib = (LibHandler)Native.loadLibrary(this.Name, LibHandler.class);
(I prefer Native.loadLibrary because it seems to allow me to do more with it easier, such as class sharing between multiple libraries and unhooking and rehooking it from the JVM on the fly).
Edit:
Seriously any method:
std::cout << "Getting version:" << std::endl;
std::cout << env->GetVersion() << std::endl;
Getting version:
(segfault)
Any ideas on wht JNIEnv would segfault for every method? This should be set up by the JVM, correct?
Edit 2:
This is a Java application that is calling a C++ library that will interface with a .NET library (so it's a CLR compiled C++ library, if that makes any difference), to limit any external factors I'm not even calling the .NET DLL but just converting strings that come in back out (or well... trying).
So for example from Java:
this.Lib = (LibHandler)Native.loadLibrary(this.Name, LibHandler.class);
this.Lib.myCPPMethod(); // Segmentation fault during this call, JVM crashes.
Curious if it was the CLR that was causing it: Disabled clr compilation and stripped everything out that was CLR related, still does it.
Edit 3:
Got it to dump:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000000000000, pid=1596, tid=7248
#
# JRE version: 6.0_23-b05
# Java VM: Java HotSpot(TM) 64-Bit Server VM (19.0-b09 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C 0x0000000000000000
So yeah, looks like the JVM isn't giving me memory access for some reason.
Edit 4:
Actual call:
JNIEXPORT jstring JNICALL Query(JNIEnv * env, jobject jobj, jstring start, jstring end)
{
std::cout << "Getting version:" << std::endl;
jint j = env->GetVersion();
return jstring("test");
}
Edit 5:
Works with System.loadLibrary:
JNIEXPORT void JNICALL Java_LibTest_T(JNIEnv *env, jobject jobj)
{
std::cout << "Getting version:" << std::endl;
jint j = env->GetVersion();
std::cout << j << std::endl;
}
Output:
java -Djava.library.path="(dir)\lib\64" EntryPoint
Getting version:
65542
Ack! I mean some headway, but I can't unload libraries from the JVM that are loaded in System.loadLibrary can I?
I basically need to be able to unhook these libraries from the JVM and swap them out, on top of that they all need to "share" a single class and be able to be bound to the class at runtime... which is kinda why I went with Native.loadLibrary.
Currently I do this:
Load the DLL:
this.Lib = (LibHandler)Native.loadLibrary(this.Name, LibHandler.class);
Unhook it:
this.Lib = null;
Runtime.getRuntime().gc(); // Force the JVM to drop it immediately.
Class I load them all into:
public interface LibHandler extends Library{
void T();
}
Any way to work similarly with System.loadLibrary?
Edit 6:
Feel free to call me dumb, I'm using JNA, NOT JNI, which is completely different and a huge source of my problems.... is there a way to do this with JNI? Or can I get JNIEnv to register with JNA somehow? I'm guessing I can drop JNI from the C++ library and straight up use wstrings?
I'll get back with this tomorrow.

Well I feel bad.
Native.loadLibrary == JNA.
System.loadLibrary == JNI.
The purpose of JNA is to not require any real knowledge of the JVM environment, so you can run native libraries as is, so instead of jstring you can use char*...

Fork and drop privileges with Java

I'm writing a server program in Java that will allow users to submit jobs using DRMAA. Although the main server process runs as root, all it does is authenticate the user, then start another Java program which runs as that user and actually does the work in order to comply with the principle of minimising privileges. Initially, I was doing this with Runtime.exec() and sudo (example below) which works fine until the process is dæmonised, at which point sudo gets upset because it doesn't have a terminal.
String[] command = {"sudo", "-i", "-u", username, java, theOtherJavaProgram};
Runtime.getRuntime().exec(command, null, getHomeDirectory(username));
What's the best way to do this fork and drop privileges pattern in Java when running as a daemon? Is there a way? Am I going to have to break out the C and learn how to create JVMs with JNI?

It's probably easier to just use JNI to drop privileges.
Here's one I knocked up earlier:
UID.java
public class UID {
public static native int setuid(int uid);
static {
System.loadLibrary("uid");
}
}
unix_uid.c
#include <sys/types.h>
#include <unistd.h>
#include <jni.h>
#include "UID.h"
JNIEXPORT jint JNICALL
Java_UID_setuid(JNIEnv * jnienv, jclass j, jint uid)
{
return((jint)setuid((uid_t)uid));
}
UID.h is machine generated from UID.class using javah.

You could use su(1) instead of sudo(8). su(1) is much less involved, and probably won't want the terminal itself. (Of course, if your PAM configuration requires terminal input for su(1), then this might not work well either.)

If you only want to start a non-root process as root, then su will be sufficient. It will not ask for a password when going from root to another user, so it should not need a terminal.