We Keep Coding

Java JNI GDAL native library error with ClassLoader when redeploying as web application

I'm using GDAL native library (C++ and it is installed in /usr/lib/java/gdal). I found a trick short time ago, to allow Tomcat can load the web application and this library (cannot use System.load() or System.loadLibrary() as all will return error)
Caused by: java.lang.UnsatisfiedLinkError: org.gdal.osr.osrJNI.new_SpatialReference__SWIG_1()J
Then I need to use a trick to add the library path to JVM when application starts:
final Field usrPathsField = ClassLoader.class.getDeclaredField("usr_paths");
usrPathsField.setAccessible(true);
// get array of paths
final String[] paths = (String[]) usrPathsField.get(null);
// check if the path to add is already present
for (String path : paths) {
if (path.equals(pathToAdd)) {
return;
}
}
//add the new path
final String[] newPaths = Arrays.copyOf(paths, paths.length + 1);
newPaths[newPaths.length - 1] = pathToAdd;
usrPathsField.set(null, newPaths);
This works well when the Tomcat starts with application, however, if I redeploy the application, it will return error:
Caused by: java.lang.UnsatisfiedLinkError: Native Library /usr/lib/java/gdal/libgdaljni.so already loaded in another classloader
I could not find any solution in StackOverflow, so I ask here if anyone can give some information. I also cannot change or add library path to environment variable or Tomcat folder, all should be done in Java code only.

So to avoid to add library to Tomcat/lib folder, I copy all the GDAL native folder to a temp directory with time stamp (e.g: /tmp/gdal_native/date.time), then I use the code above normally, except when it checks for the previous path, it will override with the new one.
String tmpTargetNativeFolderPath = "/tmp/gdal_native" + "/" + current date time
int i = 0;
// check if the path to add is already present
for (String path : paths) {
String pathFolder = StringUtils.substringBeforeLast(path, "/");
if (pathFolder.equals("/tmp/gdal_native")) {
// Override the old path with the new one
paths[i] = tmpTargetNativeFolderPath;
usrPathsField.set(null, paths);
return;
}
i++;
}
Then Classloader will load the library from another folder when the web application is redeployed without the error and the usrPathsField only contains one folder path to /tmp/gdal_native/timestamp.

java.lang.ClassNotFoundException when running program on spark cluster

I have a spark scala program which loads a jar I wrote in java. From that jar a static function is called, which tried to read a serialized object from a file (Pattern.class), but throws a java.lang.ClassNotFoundException.
Running the spark program locally works, but on the cluster workers it doesn't. It's especially weird because before I try to read from the file, I instantiate a Pattern object and there are no problems.
I am sure that the Pattern objects I wrote in the file are the same as the Pattern objects I am trying to read.
I've checked the jar in the slave machine and the Pattern class is there.
Does anyone have any idea what the problem might be ? I can add more detail if it's needed.
This is the Pattern class
public class Pattern implements Serializable {
private static final long serialVersionUID = 588249593084959064L;
public static enum RelationPatternType {NONE, LEFT, RIGHT, BOTH};
RelationPatternType type;
String entity;
String pattern;
List<Token> tokens;
Relation relation = null;
public Pattern(RelationPatternType type, String entity, List<Token> tokens, Relation relation) {
this.type = type;
this.entity = entity;
this.tokens = tokens;
this.relation = relation;
if (this.tokens != null)
this.pattern = StringUtils.join(" ", this.tokens.toString());
}
}
I am reading the file from S3 the following way:
AmazonS3 s3Client = new AmazonS3Client(credentials);
S3Object confidentPatternsObject = s3Client.getObject(new GetObjectRequest("xxx","confidentPatterns"));
objectData = confidentPatternsObject.getObjectContent();
ois = new ObjectInputStream(objectData);
confidentPatterns = (Map<Pattern, Tuple2<Integer, Integer>>) ois.readObject();
LE: I checked the classpath at runtime and the path to the jar was not there. I added it for the executors but I still have the same problem. I don't think that was it, as I have the Pattern class inside the jar that is calling the readObject function.

Would suggest this adding this kind method to find out the classpath resources before call, to make sure that everything is fine from caller's point of view
public static void printClassPathResources() {
final ClassLoader cl = ClassLoader.getSystemClassLoader();
final URL[] urls = ((URLClassLoader) cl).getURLs();
LOG.info("Print All Class path resources under currently running class");
for (final URL url : urls) {
LOG.info(url.getFile());
}
}
This is sample configuration spark 1.5
--conf "spark.driver.extraLibrayPath=$HADOOP_HOME/*:$HBASE_HOME/*:$HADOOP_HOME/lib/*:$HBASE_HOME/lib/htrace-core-3.1.0-incubating.jar:$HDFS_PATH/*:$SOLR_HOME/*:$SOLR_HOME/lib/*" \
--conf "spark.executor.extraLibraryPath=$HADOOP_HOME/*" \
--conf "spark.executor.extraClassPath=$(echo /your directory of jars/*.jar | tr ' ' ',')
As described by this Trouble shooting guide :Class Not Found: Classpath Issues
Another common issue is seeing class not defined when compiling Spark programs this is a slightly confusing topic because spark is actually running several JVM’s when it executes your process and the path must be correct for each of them. Usually this comes down to correctly passing around dependencies to the executors. Make sure that when running you include a fat Jar containing all of your dependencies, (I recommend using sbt assembly) in the SparkConf object used to make your Spark Context. You should end up writing a line like this in your spark application:
val conf = new SparkConf().setAppName(appName).setJars(Seq(System.getProperty("user.dir") + "/target/scala-2.10/sparktest.jar"))
This should fix the vast majority of class not found problems. Another option is to place your dependencies on the default classpath on all of the worker nodes in the cluster. This way you won’t have to pass around a large jar.
The only other major issue with class not found issues stems from different versions of the libraries in use. For example if you don’t use identical versions of the common libraries in your application and in the spark server you will end up with classpath issues. This can occur when you compile against one version of a library (like Spark 1.1.0) and then attempt to run against a cluster with a different or out of date version (like Spark 0.9.2). Make sure that you are matching your library versions to whatever is being loaded onto executor classpaths. A common example of this would be compiling against an alpha build of the Spark Cassandra Connector then attempting to run using classpath references to an older version.

getting error : java.lang.UnsatisfiedLinkError: Native Library C:\opencv\build\java\x64\opencv_java300.dll

I am running a servlet program to read an image using opencv,
getting error :
java.lang.UnsatisfiedLinkError: Native Library C:\opencv\build\java\x64\opencv_java300.dll already loaded in another classloader . When restarting the IDE it works fine.
I loaded System.loadLibrary ( Core.NATIVE_LIBRARY_NAME ) ; in servlet only ones.
Can anybody suggest a solution for how to unload it. And also anybody know how to read an image from browser using opencv in java.?

It is because the library is not in the system path, it needs to first added to the system path, then load. First extract the OpenCV to C drive something like this c:\opencv\... then use this code below to during initializing, it will automatically load the OpenCV lib in windows environment.
public static void loadOpenCV_Lib() throws Exception {
String model = System.getProperty("sun.arch.data.model");
String libraryPath = "C:/opencv/build/java/x86/";
if(model.equals("64")) {
libraryPath = "C:/opencv/build/java/x64/";
}
System.setProperty("java.library.path", libraryPath);
Field sysPath = ClassLoader.class.getDeclaredField("sys_paths");
sysPath.setAccessible(true);
sysPath.set(null, null);
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
}
And also it will automatically detect the system model and load the lib according to the system model.

Load jar dynamically at runtime?

My current java project is using methods and variables from another project (same package). Right now the other project's jar has to be in the classpath to work correctly. My problem here is that the name of the jar can and will change because of increasing versions, and because you cannot use wildcards in the manifest classpath, it's impossible to add it to the classpath. So currently the only option of starting my application is using the -cp argument from the command line, manually adding the other jar my project depends on.
To improve this, I wanted to load the jar dynamically and read about using the ClassLoader. I read a lot of examples for it, however I still don't understand how to use it in my case.
What I want is it to load a jar file, lets say, myDependency-2.4.1-SNAPSHOT.jar, but it should be able to just search for a jar file starting with myDependency- because as I already said the version number can change at anytime. Then I should just be able to use it's methods and variables in my Code just like I do now (like ClassInMyDependency.exampleMethod()).
Can anyone help me with this, as I've been searching the web for a few hours now and still don't get how to use the ClassLoader to do what I just explained.
Many thanks in advance

(Applies to Java version 8 and earlier).
Indeed this is occasionally necessary. This is how I do this in production. It uses reflection to circumvent the encapsulation of addURL in the system class loader.
/*
* Adds the supplied Java Archive library to java.class.path. This is benign
* if the library is already loaded.
*/
public static synchronized void loadLibrary(java.io.File jar) throws MyException
{
try {
/*We are using reflection here to circumvent encapsulation; addURL is not public*/
java.net.URLClassLoader loader = (java.net.URLClassLoader)ClassLoader.getSystemClassLoader();
java.net.URL url = jar.toURI().toURL();
/*Disallow if already loaded*/
for (java.net.URL it : java.util.Arrays.asList(loader.getURLs())){
if (it.equals(url)){
return;
}
}
java.lang.reflect.Method method = java.net.URLClassLoader.class.getDeclaredMethod("addURL", new Class[]{java.net.URL.class});
method.setAccessible(true); /*promote the method to public access*/
method.invoke(loader, new Object[]{url});
} catch (final java.lang.NoSuchMethodException |
java.lang.IllegalAccessException |
java.net.MalformedURLException |
java.lang.reflect.InvocationTargetException e){
throw new MyException(e);
}
}

I needed to load a jar file at runtime for both java 8 and java 9+. Here is the method to do it (using Spring Boot 1.5.2 if it may relate).
public static synchronized void loadLibrary(java.io.File jar) {
try {
java.net.URL url = jar.toURI().toURL();
java.lang.reflect.Method method = java.net.URLClassLoader.class.getDeclaredMethod("addURL", new Class[]{java.net.URL.class});
method.setAccessible(true); /*promote the method to public access*/
method.invoke(Thread.currentThread().getContextClassLoader(), new Object[]{url});
} catch (Exception ex) {
throw new RuntimeException("Cannot load library from jar file '" + jar.getAbsolutePath() + "'. Reason: " + ex.getMessage());
}
}

Building a ServiceLoader file with gradle: howto?

I am starting to switch from a well-known Java build system to Gradle to build all my projects, and after barely two hours into it I have already been able to publish a new version of one of my projects without a problem -- a breeze.
But now I encounter a difficulty. In short, I need to replicate the functionality of this Maven plugin which generates the necessary files for a ServiceLoader-enabled service.
In short: given a base class foo.bar.MyClass, it generates a file named META-INF/services/foo.bar.MyClass whose content is a set of classes in the current project which implement that interface/extend that base class. Such a file would look like:
com.mycompany.MyClassImpl
org.othercompany.MyClassImpl
In order to do this, it uses I don't know what as a classloader, loads the Class objects for com.myCompany.MyClassImpl or whatever and checks whether this class implements the wanted interface.
I am trying to do the same in Gradle. Hours of googling led me to this plugin, but after discussing with its author a little, it appears this plugin is able to merge such files, not create them. So, I have to do that myself...
And I am a real beginner both with Gradle and Groovy, which does not help! Here is my current code, link to the full build.gradle here; output (which I managed to get somehow; doesn't work from a clean dir) shown below (and please bear with me... I do Java, and I am final happy; Groovy is totally new to me):
/*
* TEST CODE
*/
final int CLASS_SUFFIX = ".class".length();
final URLClassLoader classLoader = this.class.classLoader;
// Where the classes are: OK
final File classesDir = sourceSets.main.output.classesDir;
final String basePath = classesDir.getCanonicalPath();
// Add them to the classloader: OK
classLoader.addURL(classesDir.toURI().toURL())
// Recurse over each file
classesDir.eachFileRecurse {
// You "return" from a closure, you do not "continue"...
if (!isPotentialClass(it))
return;
// Transform into a class name
final String path = it.getAbsolutePath();
final String name = path.substring(basePath.length() + 1);
final String className = name.substring(0, name.length() - CLASS_SUFFIX)
.replace('/', '.');
// Try and load it
try {
classLoader.loadClass(className);
println(className);
} catch (NoClassDefFoundError ignored) {
println("failed to load " + className + ": " + ignored);
}
}
boolean isPotentialClass(final File file)
{
return file.isFile() && file.name.endsWith(".class")
}
The output:
com.github.fge.msgsimple.InternalBundle
failed to load com.github.fge.msgsimple.bundle.MessageBundle: java.lang.NoClassDefFoundError: com/github/fge/Frozen
failed to load com.github.fge.msgsimple.bundle.MessageBundleBuilder: java.lang.NoClassDefFoundError: com/github/fge/Thawed
com.github.fge.msgsimple.bundle.PropertiesBundle$1
com.github.fge.msgsimple.bundle.PropertiesBundle
com.github.fge.msgsimple.provider.MessageSourceProvider
com.github.fge.msgsimple.provider.LoadingMessageSourceProvider$1
com.github.fge.msgsimple.provider.LoadingMessageSourceProvider$2
com.github.fge.msgsimple.provider.LoadingMessageSourceProvider$3
com.github.fge.msgsimple.provider.LoadingMessageSourceProvider$Builder
com.github.fge.msgsimple.provider.LoadingMessageSourceProvider
com.github.fge.msgsimple.provider.MessageSourceLoader
com.github.fge.msgsimple.provider.StaticMessageSourceProvider$Builder
com.github.fge.msgsimple.provider.StaticMessageSourceProvider$1
com.github.fge.msgsimple.provider.StaticMessageSourceProvider
com.github.fge.msgsimple.source.MessageSource
com.github.fge.msgsimple.source.MapMessageSource$Builder
com.github.fge.msgsimple.source.MapMessageSource$1
com.github.fge.msgsimple.source.MapMessageSource
com.github.fge.msgsimple.source.PropertiesMessageSource
com.github.fge.msgsimple.locale.LocaleUtils
com.github.fge.msgsimple.serviceloader.MessageBundleFactory
com.github.fge.msgsimple.serviceloader.MessageBundleProvider
:compileJava UP-TO-DATE
The problem is in the two first lines: Frozen and Thawed are in a different project, which is in the compile classpath but not in the classpath I managed to grab so far... As such, these classes cannot even load.
How do I modify that code so as to have the full compile classpath availabe? Is my first question. Second question: how do I plug that code, when it works, into the build process?

Here are some hints:
Create a new URLClassLoader, rather than reusing an existing one.
Initialize the class loader with sourceSets.main.compileClasspath (which is an Iterable<File>) rather than classesDir.
Turn the code into a Gradle task class. For more information, see "Writing a simple task class" in the Gradle User Guide.
Ideally, you'd use a library like ASM to analyze the code, rather than using a class loader. To avoid the case where you cannot load a class because it internally references a class that's not on the compile class path, you may want to initialize the class loader with sourceSets.main.runtimeClasspath instead.