We Keep Coding

Getting an UnsatisfiedLinkError (undefined symbol) in Java while loading JNI dependencies even after successfully loading the required object file

I'm using Google OR-tools library (v6.4) for a project (though my question is not specific to this library). This consists of one jar, which has a few native dependencies (a bunch of ".so"/".dylib" object files, depending on the OS). This build for my project is being made on Ubuntu 14.04
The problem I'm facing: On trying to load a specific object file at runtime (using System.load()), I'm getting an UnsatisfiedLinkError with the message as "undefined symbol" (I've added the stacktrace below). However, I am loading the object file defining this symbol just before this, so I'm not sure why this error is being thrown.
I'm loading the dependencies in the following way: The object files are being packed into the jar created by Maven during build, and are being extracted and loaded (using System.load()) at runtime. The method for that is as follows:
public class EnvironmentUtils {
public static void loadResourceFromJar(String prefix, String suffix) {
String tempFilesDirectory = System.getProperty("java.io.tmpdir");
File tempFile = null;
try {
tempFile = new File(tempFilesDirectory + "/" + prefix + suffix);
tempFile.deleteOnExit();
try (final InputStream inputStream = EnvironmentUtils.class.getClassLoader().
getResourceAsStream(prefix+suffix)) {
if (inputStream == null) {
throw new RuntimeException(prefix + suffix + " was not found inside JAR.");
} else {
Files.copy(inputStream, tempFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
}
}
System.load(tempFile.getAbsolutePath());
} catch (Exception e) {
//Log top 10 lines of stack trace
}
}
}
This method is being called inside a static block for all dependencies:
public class DummyClass {
static {
String sharedLibraryExtension = EnvironmentUtils.getSharedLibraryExtension(); //.so for linux, .dylib for Mac
String jniLibraryExtension = EnvironmentUtils.getJniLibraryExtension(); //.so for linux, .jnilib for Mac
EnvironmentUtils.loadResourceFromJar("libfap", sharedLibraryExtension);
EnvironmentUtils.loadResourceFromJar("libcvrptw_lib", sharedLibraryExtension);
EnvironmentUtils.loadResourceFromJar("libortools", sharedLibraryExtension);
EnvironmentUtils.loadResourceFromJar("libdimacs", sharedLibraryExtension);
EnvironmentUtils.loadResourceFromJar("libjniortools", jniLibraryExtension);
}
}
On running System.load() for libdimacs.so, an UnsatisfiedLinkError is thrown. Stacktrace:
java.lang.UnsatisfiedLinkError: /tmp/libdimacs.so: /tmp/libdimacs.so: undefined symbol: _ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at com.(PROJECT_NAME).utils.EnvironmentUtils.loadResourceFromJar(EnvironmentUtils.java:78)
at com.(PROJECT_NAME).DummyClass.<clinit>(DummyClass.java:28)
However, this symbol "_ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_" is present in libortools.so, which is being loaded before libdimacs. I verified this by running the following command:
objdump -t (LIBRARY_PATH)/libortools.so | grep _ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_
This gave me the following output:
0000000000ce12cc gw F .text 00000091 _ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_
So it would seem that the symbol should have been defined at the time of the System.load() call, unless there was some issue in loading the containing object file. To check if the object file had been loaded correctly, I used the approach detailed in this solution. Apart from the class detailed in that answer, I added the following lines after System.load() call in EnvironmentUtils.loadResourceFromJar() to print the most recently loaded library name:
public class EnvironmentUtils {
public static void loadResourceFromJar(String prefix, String suffix) {
...
System.load(tempFile.getAbsolutePath());
final String[] libraries = ClassScope.getLoadedLibraries(ClassLoader.getSystemClassLoader());
System.out.println(libraries[libraries.length - 1]);
}
}
The output (till just before the UnsatisfiedLinkError) is as follows:
/tmp/libfap.so
/tmp/libcvrptw_lib.so
/tmp/libortools.so
So libortools.so seems to be loading correctly, which means the symbol should be loaded in memory. The exact same code is working perfectly with the corresponding Mac (".dylib") dependencies (Built on MacOS Sierra 10.12.5). Would appreciate any advice on resolving this. Thank you.

I'm apologize that the java artifact may be broken currently...
you can use c++filt to demangle the symbol ;)
c++filt _ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_
google::FlagRegisterer::FlagRegisterer<bool>(char const*, char const*, char const*, bool*, bool*)
In fact gflag has recently change its namespace from google:: to gflags:: and glog or protobobuf? try to find the correct one and I guess it failed...
note: Still not completely sure whose is the bad guy who use the google:: namespace since libortools merge all its static dependencies but I guess now you understand the bug...
note2: I have a patch in mizux/shared branch https://github.com/google/or-tools/commit/805bc0600f4b5645114da704a0eb04a0b1058e28#diff-e8590fe6fb5044985c8bf8c9e73c0d88R114
warning: this branch is currently broken and not ready yet. I'm trying ,for unix, to move from static to dynamic dependencies, so I need to fix all rpath, transitives deps etc... and in the process I also had to fix this issue (that I didn't reproduced while using static dependencies)
If too long to finish (we should create a release 6.7.2 or 6.8 (i.e. new artifact) by the end of May 2018) which maybe only contains this fix and not my branch...

Manually loading native libraries to circumvent a restrictive environment

I'm maintaining a Java Swing application that requires a connection to an instance of Microsoft SQL Server. For various reasons, I opted to replace the native SQL Server driver being used with jTDS (the aforementioned Microsoft drivers were not working at the time and have apparently failed in the field as well). When I try to run the executable .jar outside of the IDE, I run into issues because I'm missing the appropriate ntlmauth.dll dependency.
Before proceeding, it's important to note that this application is being developed and used in an extremely restrictive (Windows-only) environment:
I cannot install any software that requires Windows UAC authentication
My users cannot install or run any software that requires UAC authentication
This currently means I cannot write files to System32 or JAVA_HOME, and cannot use any sort of ProcessBuilder tomfoolery to start another JVM with whatever command line arguments I need
I cannot use executable wrappers/installers that would only require the UAC permission for the first time installation/setup
The solution I'm trying is a combination of this one and this one to check it--essentially packaging the .dll inside of the .jar, then extracting it and loading it if necessary--as most of the other solutions I've found have been incompatible with the above restrictions; however, I'm running into an issue where even after the native library is ostensibly "loaded," I get an exception saying it isn't.
My pre-startup code:
private static final String LIB_BIN = "/lib-bin/";
private static final String JTDS_AUTH = "ntlmauth";
// load required JTDS binaries
static {
logger.info("Attempting to load library {}.dll", JTDS_AUTH);
try {
System.loadLibrary(JTDS_AUTH);
} catch (UnsatisfiedLinkError e) {
loadFromJar();
}
try {
// do some quick checks to make sure that went ok
NativeLibraries nl = new NativeLibraries();
logger.debug("Loaded libraries: {}", nl.getLoadedLibraries().toString());
} catch (NoSuchFieldException ex) {
logger.info("Native library checker load failed", ex);
}
}
/**
* When packaged into JAR extracts DLLs, places these into
*/
private static void loadFromJar() {
// we need to put DLL in temp dir
String path = ***;
loadLib(path, JTDS_AUTH);
}
/**
* Puts library to temp dir and loads to memory
*/
private static void loadLib(String path, String name) {
name = name + ".dll";
try {
// have to use a stream
InputStream in = net.sourceforge.jtds.jdbc.JtdsConnection.class.getResourceAsStream(LIB_BIN + name);
// always write to different location
File fileOut = new File(System.getProperty("java.io.tmpdir") + "/" + path + LIB_BIN + name);
logger.info("Writing dll to: " + fileOut.getAbsolutePath());
OutputStream out = FileUtils.openOutputStream(fileOut);
IOUtils.copy(in, out);
in.close();
out.close();
System.load(fileOut.toString());
} catch (Exception e) {
logger.error("Exception with native library loader", e);
JOptionPane.showMessageDialog(null, "Exception loading native libraries: " + e.getLocalizedMessage(), "Exception", JOptionPane.ERROR_MESSAGE);
}
}
As you can see, I basically copied the solution from the first link verbatim, with a few minor modifications just to try and get the application running. I also copied the class from the second link and named it NativeLibraries, the invocation of that method is fairly irrelevant but it shows up in the logs.
Anyway here are the relevant bits of the log output on starting up the application:
2015-07-20 12:32:33 INFO - Attempting to load library ntlmauth.dll
2015-07-20 12:32:33 INFO - Writing dll to: C:\Users\***\lib-bin\ntlmauth.dll
2015-07-20 12:32:33 DEBUG - Loaded libraries: [C:\Program Files\Java\jre1.8.0_45\bin\zip.dll, C:\Program Files\Java\jre1.8.0_45\bin\prism_d3d.dll, C:\Program Files\Java\jre1.8.0_45\bin\prism_sw.dll, C:\Program Files\Java\jre1.8.0_45\bin\msvcr100.dll, C:\Program Files\Java\jre1.8.0_45\bin\glass.dll, C:\Program Files\Java\jre1.8.0_45\bin\net.dll, C:\Users\***\lib-bin\ntlmauth.dll]
2015-07-20 12:32:33 INFO - Application startup
***
2015-07-20 12:32:36 ERROR - Database exception
java.sql.SQLException: I/O Error: SSO Failed: Native SSPI library not loaded. Check the java.library.path system property.
at net.sourceforge.jtds.jdbc.TdsCore.login(TdsCore.java:654) ~[jtds-1.3.1.jar:1.3.1]
at net.sourceforge.jtds.jdbc.JtdsConnection.<init>(JtdsConnection.java:371) ~[jtds-1.3.1.jar:1.3.1]
at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:184) ~[jtds-1.3.1.jar:1.3.1]
at java.sql.DriverManager.getConnection(Unknown Source) ~[na:1.8.0_45]
at java.sql.DriverManager.getConnection(Unknown Source) ~[na:1.8.0_45]
One can see that the library was, indeed, "loaded," from the third line in the log (it's the last entry, if you don't feel like scrolling). However, I simply used the class that I felt like was probably using the native libraries (I also tried the TdsCore class to no avail), as the example that showed how to do this was just using a random class from the package the library was needed in.
Is there something I'm missing here? I'm not very experienced with the JNI or the inner workings of ClassLoaders, so I might just be loading it wrong. Any advice or suggestions would be greatly appreciated!

Welp I figured out a workaround: I ended up using JarClassLoader. This basically entailed copying all my dependencies, both Java and native, into a "libraries" folder within my main .jar, and disabling .jar signing in the IDE. The application is then run by a new class that simply creates a new JarClassLoader object and running the "invokeMain" method--an example is on the website. The whole thing took about three minutes, after several days of banging my head against a wall.
Hope this helps someone someday!

Load jar dynamically at runtime?

My current java project is using methods and variables from another project (same package). Right now the other project's jar has to be in the classpath to work correctly. My problem here is that the name of the jar can and will change because of increasing versions, and because you cannot use wildcards in the manifest classpath, it's impossible to add it to the classpath. So currently the only option of starting my application is using the -cp argument from the command line, manually adding the other jar my project depends on.
To improve this, I wanted to load the jar dynamically and read about using the ClassLoader. I read a lot of examples for it, however I still don't understand how to use it in my case.
What I want is it to load a jar file, lets say, myDependency-2.4.1-SNAPSHOT.jar, but it should be able to just search for a jar file starting with myDependency- because as I already said the version number can change at anytime. Then I should just be able to use it's methods and variables in my Code just like I do now (like ClassInMyDependency.exampleMethod()).
Can anyone help me with this, as I've been searching the web for a few hours now and still don't get how to use the ClassLoader to do what I just explained.
Many thanks in advance

(Applies to Java version 8 and earlier).
Indeed this is occasionally necessary. This is how I do this in production. It uses reflection to circumvent the encapsulation of addURL in the system class loader.
/*
* Adds the supplied Java Archive library to java.class.path. This is benign
* if the library is already loaded.
*/
public static synchronized void loadLibrary(java.io.File jar) throws MyException
{
try {
/*We are using reflection here to circumvent encapsulation; addURL is not public*/
java.net.URLClassLoader loader = (java.net.URLClassLoader)ClassLoader.getSystemClassLoader();
java.net.URL url = jar.toURI().toURL();
/*Disallow if already loaded*/
for (java.net.URL it : java.util.Arrays.asList(loader.getURLs())){
if (it.equals(url)){
return;
}
}
java.lang.reflect.Method method = java.net.URLClassLoader.class.getDeclaredMethod("addURL", new Class[]{java.net.URL.class});
method.setAccessible(true); /*promote the method to public access*/
method.invoke(loader, new Object[]{url});
} catch (final java.lang.NoSuchMethodException |
java.lang.IllegalAccessException |
java.net.MalformedURLException |
java.lang.reflect.InvocationTargetException e){
throw new MyException(e);
}
}

I needed to load a jar file at runtime for both java 8 and java 9+. Here is the method to do it (using Spring Boot 1.5.2 if it may relate).
public static synchronized void loadLibrary(java.io.File jar) {
try {
java.net.URL url = jar.toURI().toURL();
java.lang.reflect.Method method = java.net.URLClassLoader.class.getDeclaredMethod("addURL", new Class[]{java.net.URL.class});
method.setAccessible(true); /*promote the method to public access*/
method.invoke(Thread.currentThread().getContextClassLoader(), new Object[]{url});
} catch (Exception ex) {
throw new RuntimeException("Cannot load library from jar file '" + jar.getAbsolutePath() + "'. Reason: " + ex.getMessage());
}
}

load class from another class within same jar in java web start fails

I've a javafx application which is run by web start. In my fx application, I try to load the classes using ClassLoader as in below code. The parameter passed is a package name like "com.example.project.abcd"
public final static List<Class<?>> find(final String scannedPackage)
{
final ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
final String scannedPath = scannedPackage.replace(DOT, SLASH);
final Enumeration<URL> resources;
try {
resources = classLoader.getResources(scannedPath);
} catch (IOException e) {
throw new IllegalArgumentException(String.format(BAD_PACKAGE_ERROR, scannedPath, scannedPackage), e);
}
final List<Class<?>> classes = new LinkedList<Class<?>>();
while (resources.hasMoreElements()) {
final File file = new File(resources.nextElement().getFile());
classes.addAll(find(file, scannedPackage));
}
return classes;
}
Now I'm not able to get all the classes present inside "com.example.project.abcd" package when I run it thru java web start but through IDE it is working fine.
I'm using JDK 7, JavaFX 2.
As per http://docs.oracle.com/javase/7/docs/technotes/guides/javaws/developersguide/faq.html#s211 Thread.currentThread().getContextClassLoader() should work fine but it is not!.
Tried searching on net/googling but in vain. Checked http://lopica.sourceforge.net/faq.html#customcl as well and tried using URLClassLoader as suggested. But that didn't work as well (Though did not know what should be passed to the parameter 'urls')
Any help is much apreciated.

I think this works in IDE because your BIN/classes directory is used to get all the files.
In Webstart-Mode, all your classes are inside JARs.

Java Trouble compiling maxmind geoip LookupService class

I wonder if some one can please help as I am struggling to compile maxmind.geoip.LookupService.java
I have downloaded geoip-api-1.2.10.jar for inclusion in WEB-INF\lib and I have referenced it in my classes path, but it just won't compile.
I have compiled the following successfully so I'm a bit at a loss:
com.maxmind.geoip.Country
com.maxmind.geoip.DatabaseInfo
com.maxmind.geoip.Location
com.maxmind.geoip.Region
com.maxmind.geoip.timeZone
Can't seem to find a full set of compiled java classes for com.maxmind.geoip, any help would be much appreciated :-)

I resolved this by downloading the latest java files from http://dev.maxmind.com/geoip/legacy/downloadable/ unpacked the folder and then opened a command prompt and typed the following:
cd source/com/maxmind/geoip/
javac *.java
I'm using jdk1.6.0_34 and all classes compiled with no errors.
I copied the com.maxmind.geoip folder to \WEB-INF\classes and downloaded geoip-api-1.2.10.jar and placed that in the WEB-INF\lib folder.
Finally I download GeoIP.dat from http://dev.maxmind.com/geoip/legacy/geolite/ and placed it in a new folder called GeoIP under webapps so that all my applications can use it.
The following code is to obtain the country code from a users IP Address:
import com.maxmind.geoip.*;
import java.io.IOException;
class CountryLookupTest {
public static void main(String[] args) {
try {
String sep = System.getProperty("file.separator");
String dir = "C:/Program Files/Apache Software Foundation/Tomcat 7.0/GeoIP";
String dbfile = dir + sep + "GeoIP.dat";
LookupService cl = new LookupService(dbfile,LookupService.GEOIP_MEMORY_CACHE);
System.out.println(cl.getCountry("151.38.39.114").getCode());
System.out.println(cl.getCountry("151.38.39.114").getName());
System.out.println(cl.getCountry("12.25.205.51").getName());
System.out.println(cl.getCountry("64.81.104.131").getName());
System.out.println(cl.getCountry("200.21.225.82").getName());
cl.close();
}
catch (IOException e) {
System.out.println("IO Exception");
}
}
}
Hope this proves useful to others.

According to the MaxMind dev site, the API is available on the Maven Central Repository. You shouldn't need to compile anything unless you downloaded the source package.

You have to download a Jar file called geoIP-api from this link to maven repository,In case you haven't downloaded the other Jar files from go this geoIP2 also don't forget to download the .DAT file from geoIP.dat. Then add the files to your project class path from project properties and then libraries finally add Jar in netbeans.
Now use this code:
public String IpGeoLocation(String IP) {
try {
String dbfile = "C:\\Users\\User Name \\Documents\\NetBeansProjects\\IP Tools\\resources/GeoIP.dat";
String location = "";
LookupService cl = new LookupService(dbfile, LookupService.GEOIP_MEMORY_CACHE);
location = cl.getCountry(IP).getName() + " " + cl.getCountry(IP).getCode();
cl.close();
return location;
} catch (Exception e) {
return "Error";
}
}
I was able to find the country and country code only !!