We Keep Coding

Getting an UnsatisfiedLinkError (undefined symbol) in Java while loading JNI dependencies even after successfully loading the required object file

I'm using Google OR-tools library (v6.4) for a project (though my question is not specific to this library). This consists of one jar, which has a few native dependencies (a bunch of ".so"/".dylib" object files, depending on the OS). This build for my project is being made on Ubuntu 14.04
The problem I'm facing: On trying to load a specific object file at runtime (using System.load()), I'm getting an UnsatisfiedLinkError with the message as "undefined symbol" (I've added the stacktrace below). However, I am loading the object file defining this symbol just before this, so I'm not sure why this error is being thrown.
I'm loading the dependencies in the following way: The object files are being packed into the jar created by Maven during build, and are being extracted and loaded (using System.load()) at runtime. The method for that is as follows:
public class EnvironmentUtils {
public static void loadResourceFromJar(String prefix, String suffix) {
String tempFilesDirectory = System.getProperty("java.io.tmpdir");
File tempFile = null;
try {
tempFile = new File(tempFilesDirectory + "/" + prefix + suffix);
tempFile.deleteOnExit();
try (final InputStream inputStream = EnvironmentUtils.class.getClassLoader().
getResourceAsStream(prefix+suffix)) {
if (inputStream == null) {
throw new RuntimeException(prefix + suffix + " was not found inside JAR.");
} else {
Files.copy(inputStream, tempFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
}
}
System.load(tempFile.getAbsolutePath());
} catch (Exception e) {
//Log top 10 lines of stack trace
}
}
}
This method is being called inside a static block for all dependencies:
public class DummyClass {
static {
String sharedLibraryExtension = EnvironmentUtils.getSharedLibraryExtension(); //.so for linux, .dylib for Mac
String jniLibraryExtension = EnvironmentUtils.getJniLibraryExtension(); //.so for linux, .jnilib for Mac
EnvironmentUtils.loadResourceFromJar("libfap", sharedLibraryExtension);
EnvironmentUtils.loadResourceFromJar("libcvrptw_lib", sharedLibraryExtension);
EnvironmentUtils.loadResourceFromJar("libortools", sharedLibraryExtension);
EnvironmentUtils.loadResourceFromJar("libdimacs", sharedLibraryExtension);
EnvironmentUtils.loadResourceFromJar("libjniortools", jniLibraryExtension);
}
}
On running System.load() for libdimacs.so, an UnsatisfiedLinkError is thrown. Stacktrace:
java.lang.UnsatisfiedLinkError: /tmp/libdimacs.so: /tmp/libdimacs.so: undefined symbol: _ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at com.(PROJECT_NAME).utils.EnvironmentUtils.loadResourceFromJar(EnvironmentUtils.java:78)
at com.(PROJECT_NAME).DummyClass.<clinit>(DummyClass.java:28)
However, this symbol "_ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_" is present in libortools.so, which is being loaded before libdimacs. I verified this by running the following command:
objdump -t (LIBRARY_PATH)/libortools.so | grep _ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_
This gave me the following output:
0000000000ce12cc gw F .text 00000091 _ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_
So it would seem that the symbol should have been defined at the time of the System.load() call, unless there was some issue in loading the containing object file. To check if the object file had been loaded correctly, I used the approach detailed in this solution. Apart from the class detailed in that answer, I added the following lines after System.load() call in EnvironmentUtils.loadResourceFromJar() to print the most recently loaded library name:
public class EnvironmentUtils {
public static void loadResourceFromJar(String prefix, String suffix) {
...
System.load(tempFile.getAbsolutePath());
final String[] libraries = ClassScope.getLoadedLibraries(ClassLoader.getSystemClassLoader());
System.out.println(libraries[libraries.length - 1]);
}
}
The output (till just before the UnsatisfiedLinkError) is as follows:
/tmp/libfap.so
/tmp/libcvrptw_lib.so
/tmp/libortools.so
So libortools.so seems to be loading correctly, which means the symbol should be loaded in memory. The exact same code is working perfectly with the corresponding Mac (".dylib") dependencies (Built on MacOS Sierra 10.12.5). Would appreciate any advice on resolving this. Thank you.

I'm apologize that the java artifact may be broken currently...
you can use c++filt to demangle the symbol ;)
c++filt _ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_
google::FlagRegisterer::FlagRegisterer<bool>(char const*, char const*, char const*, bool*, bool*)
In fact gflag has recently change its namespace from google:: to gflags:: and glog or protobobuf? try to find the correct one and I guess it failed...
note: Still not completely sure whose is the bad guy who use the google:: namespace since libortools merge all its static dependencies but I guess now you understand the bug...
note2: I have a patch in mizux/shared branch https://github.com/google/or-tools/commit/805bc0600f4b5645114da704a0eb04a0b1058e28#diff-e8590fe6fb5044985c8bf8c9e73c0d88R114
warning: this branch is currently broken and not ready yet. I'm trying ,for unix, to move from static to dynamic dependencies, so I need to fix all rpath, transitives deps etc... and in the process I also had to fix this issue (that I didn't reproduced while using static dependencies)
If too long to finish (we should create a release 6.7.2 or 6.8 (i.e. new artifact) by the end of May 2018) which maybe only contains this fix and not my branch...

Java JNI GDAL native library error with ClassLoader when redeploying as web application

I'm using GDAL native library (C++ and it is installed in /usr/lib/java/gdal). I found a trick short time ago, to allow Tomcat can load the web application and this library (cannot use System.load() or System.loadLibrary() as all will return error)
Caused by: java.lang.UnsatisfiedLinkError: org.gdal.osr.osrJNI.new_SpatialReference__SWIG_1()J
Then I need to use a trick to add the library path to JVM when application starts:
final Field usrPathsField = ClassLoader.class.getDeclaredField("usr_paths");
usrPathsField.setAccessible(true);
// get array of paths
final String[] paths = (String[]) usrPathsField.get(null);
// check if the path to add is already present
for (String path : paths) {
if (path.equals(pathToAdd)) {
return;
}
}
//add the new path
final String[] newPaths = Arrays.copyOf(paths, paths.length + 1);
newPaths[newPaths.length - 1] = pathToAdd;
usrPathsField.set(null, newPaths);
This works well when the Tomcat starts with application, however, if I redeploy the application, it will return error:
Caused by: java.lang.UnsatisfiedLinkError: Native Library /usr/lib/java/gdal/libgdaljni.so already loaded in another classloader
I could not find any solution in StackOverflow, so I ask here if anyone can give some information. I also cannot change or add library path to environment variable or Tomcat folder, all should be done in Java code only.

So to avoid to add library to Tomcat/lib folder, I copy all the GDAL native folder to a temp directory with time stamp (e.g: /tmp/gdal_native/date.time), then I use the code above normally, except when it checks for the previous path, it will override with the new one.
String tmpTargetNativeFolderPath = "/tmp/gdal_native" + "/" + current date time
int i = 0;
// check if the path to add is already present
for (String path : paths) {
String pathFolder = StringUtils.substringBeforeLast(path, "/");
if (pathFolder.equals("/tmp/gdal_native")) {
// Override the old path with the new one
paths[i] = tmpTargetNativeFolderPath;
usrPathsField.set(null, paths);
return;
}
i++;
}
Then Classloader will load the library from another folder when the web application is redeployed without the error and the usrPathsField only contains one folder path to /tmp/gdal_native/timestamp.

getting error : java.lang.UnsatisfiedLinkError: Native Library C:\opencv\build\java\x64\opencv_java300.dll

I am running a servlet program to read an image using opencv,
getting error :
java.lang.UnsatisfiedLinkError: Native Library C:\opencv\build\java\x64\opencv_java300.dll already loaded in another classloader . When restarting the IDE it works fine.
I loaded System.loadLibrary ( Core.NATIVE_LIBRARY_NAME ) ; in servlet only ones.
Can anybody suggest a solution for how to unload it. And also anybody know how to read an image from browser using opencv in java.?

It is because the library is not in the system path, it needs to first added to the system path, then load. First extract the OpenCV to C drive something like this c:\opencv\... then use this code below to during initializing, it will automatically load the OpenCV lib in windows environment.
public static void loadOpenCV_Lib() throws Exception {
String model = System.getProperty("sun.arch.data.model");
String libraryPath = "C:/opencv/build/java/x86/";
if(model.equals("64")) {
libraryPath = "C:/opencv/build/java/x64/";
}
System.setProperty("java.library.path", libraryPath);
Field sysPath = ClassLoader.class.getDeclaredField("sys_paths");
sysPath.setAccessible(true);
sysPath.set(null, null);
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
}
And also it will automatically detect the system model and load the lib according to the system model.

JAVA NoClassDefFoundError on external jar library

I'm writing an Android app in JAVA using this library for formatting phone numbers.
I am receiving the following exception:
java.lang.NoClassDefFoundError: Failed resolution of: Lcom/google/i18n/phonenumbers/PhoneNumberUtil;
Here:
public class InvocationTargetException extends ReflectiveOperationException {
...
public InvocationTargetException(Throwable exception) {
super(null, exception);
target = exception;
}
...
}
When executing the command:
PhoneNumberUtil phoneUtil = PhoneNumberUtil.getInstance();
I have imported the com.google.i18n.phonenumbers lib, but I cant figure out what is the L in the exception description.

L is added for internal JVM objectype representation, but it is actually looking for com/google/i18n/phonenumbers/PhoneNumberUtil class.
It seems the library is missing in your Runtime classpath.
Field Descriptions Documentation.

Did you add the jar file to your class path? Are you using Android Studio? In that case you can add the entry to the app/build.gradle file.

fannj library doesn't work

I'm trying to run project which uses fannj library, but I'm getting error:
Exception in thread "main" java.lang.UnsatisfiedLinkError: Error looking up function 'fann_create_standard_array':
at com.sun.jna.Function.<init>(Function.java:179)
at com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:347)
at com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:327)
at com.sun.jna.Native.register(Native.java:1355)
at com.sun.jna.Native.register(Native.java:1032)
at com.googlecode.fannj.Fann.<clinit>(Fann.java:46)
at javaapplication9.JavaApplication9.main(JavaApplication9.java:14)
Java Result: 1
This is what I did:
I put fannfloat.dll to C:\Windows\System32
I added fannj-0.3.jar to project
I added newest jna.jar to project
here is code:
public static void main(String[] args) {
System.setProperty("jna.library.path", "C:\\Windows\\System32");
System.loadLibrary("fannfloat");
Fann fann=new Fann("D:\\SunSpots.net");
fann.close();
}
SunSpots.net is file from example package. fannfloat.dll: you can get from here.

The "#8" at the end of _fann_create_standard_array indicates that the library is using the stdcall calling convention, so your library interface needs to implement that interface (StdCallLibrary) and it will automatically get the function name mapper applied that converts your simple java name to the decorated stdcall one.
This is covered in the JNA documentation.

It was the first time I had to work with FANN and it took me some time to make it work.
Downloaded Fann 2.2.0. Extract (in my case "C:/FANN-2.2.0-Source") and check the path of the fannfloat.dll file. This is the library that we will use later.
Download fannj-0.6.jar from http://code.google.com/p/fannj/downloads/list.
The dll is compiled for 32 bit environment. So, make sure you have a 32 bit Java installed (even in 64 bit Windows).
I suppose you already have the .net file with your ANN. Write something like this in Java
public class FannTest {
public static void main(String[] args) {
System.setProperty("jna.library.path", "C:/FANN-2.2.0-Source/bin");
Fann fann = new Fann("C:/MySunSpots.net" );
float[] inputs = new float[]{0.686470295f, 0.749375936f, 0.555167249f, 0.816774838f, 0.767848228f, 0.60908637f};
float[] outputs = fann.run( inputs );
fann.close();
for (float f : outputs) {
System.out.print(f + ",");
}
}
}