We Keep Coding

Jython import of Java package created by Swig fails to link

I am working on application that has components written in several languages. I am trying to get functionality that works fine in Java working in Jython. There is some native/C++ functionality that Java access via the JNI and is wrapped by SWIG.
Whenever I try to import all of the classes in the project I get errors that PROJECTJNI cannot be linked. Here is my minimum case to produce:
import sys
sys.path.append('PROJECT.jar')
from com.whatever.project import *
Here is the error message when this is executed:
$ jython Bootstrap.py
"my" variable $jythonHome masks earlier declaration in same scope at /usr/bin/jython line 15.
Traceback (most recent call last):
File "Bootstrap.py", line 9, in <module>
from com.whatever.project import *
java.lang.UnsatisfiedLinkError: com.whatever.project.PROJECTJNI.swig_module_init()V
at com.whatever.project.PROJECTJNI.swig_module_init(Native Method)
at com.whatever.project.PROJECTJNI.<clinit>(PROJECTJNI.java:974)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at org.python.core.Py.loadAndInitClass(Py.java:909)
at org.python.core.Py.findClassInternal(Py.java:844)
at org.python.core.Py.findClass(Py.java:869)
at org.python.core.packagecache.PackageManager.basicDoDir(PackageManager.java:107)
at org.python.core.packagecache.SysPackageManager.doDir(SysPackageManager.java:138)
at org.python.core.PyJavaPackage.fillDir(PyJavaPackage.java:123)
at org.python.core.imp.importAll(imp.java:1051)
at org.python.core.imp.importAll(imp.java:1039)
at org.python.pycode._pyx0.f$0(Bootstrap.py:9)
at org.python.pycode._pyx0.call_function(Bootstrap.py)
at org.python.core.PyTableCode.call(PyTableCode.java:165)
at org.python.core.PyCode.call(PyCode.java:18)
at org.python.core.Py.runCode(Py.java:1275)
at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:235)
at org.python.util.jython.run(jython.java:247)
at org.python.util.jython.main(jython.java:129)
java.lang.UnsatisfiedLinkError: java.lang.UnsatisfiedLinkError: com.whatever.project.PROJECTJNI.swig_module_init()V
The line 15 thing shows up on any time we invoke jython, we had been ignoring it.
We can get the classes from the Java Project to work by simply loading them one at a time:
com.whatever.project import Class1
com.whatever.project import Class2
...
com.whatever.project import Class50
This is highly impractical because even for short python script we might need a dozen classes. Many of them are exceptions that we are catching which have unique types. So anything that robustly handles errors might need a huge number of classes.
As per the jython documentation I should be able to hide PROJECTJNI so it is not loaded by doing something like this, but I found the docs less than perfectly clear. Here is what I attempted:
import com.whatever.project
__all__ = dir(com.whatever.project)
__all__.remove('PROJECTJNI')
from com.whatever.project import *
But this fails with errors and still clearly trying to load PROJECTJNI.
I also tried to fix the native executable so it could be linked against correctl. I learned another group using JRuby had no issue including everything, so I decided to check the source and the binary. I found void swig_module_init() in the file Project_wrap.cpp that Swig created. It was hidden behind a macro, but it was there, and objdump confirms:
$objdump libPROJECT.so -t |grep PROJECTJNI |grep init
000000000051a900 l O .data 00000000000009d0 _ZZ59Java_com_whatever_project_PROJECTJNI_swig_1module_1initE7methods
0000000000263f66 g F .text 00000000000000d1 Java_com_whatever_project_PROJECTJNI_swig_1module_1init
Am I doing something wrong with any of my troubleshooting steps? Is the a bug in Jython? Is there a simple Python workaround to make it skip loading PROJECTJNI?
Anything that lets me skip linking this or makes this link correctly will be accepted.

On one of the Java classes, the BinaryLoader, there was a method called void load_binary() that is called in a static section of that class:
public class BinaryLoader
{
static
{ load_binaries(); }
public static void load_binaries()
{
// Deep inside here is a call to actually load the shared library. Using
load_shared_library_from_jar("PROJECT");
}
... // more details here
}
Our Java code using the native API called this method as the class was loaded. Our JRuby code called this when the class was 'touched', just using the Constant's name, the name of the class, invoked the static section. To clarify here is a sample from our docs and live Ruby scripts that does this:
# Tells Jruby to load the Java Interopability layer.
require 'java'
# Loads the PROJECT Java tools.
require 'PROJECT.jar'
# Shorten the PROJECT tool names from their obnoxiously long Java name.
module PROJECT
include_package "com.whatever.project"
end
# Let Ruby know the BinaryLoader exists and it will have PROJECT load all the system binaries.
PROJECT::BinaryLoader
Apparently requiring the JarFile is enough to make all the Native symbols and Java classes available in Ruby and the binary isn't loaded until the last line which is just an expressions that resolves to the name of the class and force the static section to run. The actual method call would have looked like: PROJECT::BinaryLoader.load_binaries
In Jython the static section was never invoked even when calling other static methods on the BinaryLoader class. I forced it to call the method manually by importing the minimum of Java and Native symbols to call the method:
# Tells Jython to load the python system Interopability tools.
import sys
# When searching for python symbols, this Java jar should be search also.
sys.path.append('PROJECT.jar')
# Load just enough the have the JVM Know how to load the native PROJECT libraries
from com.whatever.project import PROJECT
from com.whatever.project import BinaryLoader
# Load any native binaries that are required
BinaryLoader.load_binaries()
# After this line is run in any given script then PROJECT can be used.
from com.whatever.project import *
Clearly the native binaries were simply not loaded as had been thought. Anything that loaded these would have been an acceptable solution.
This works around Jython not invoking the static section, by doing the work it should have as early as possible. This seems like a bug in Jython, but it might be that guarantees on static load order are not strongly enforced in Java. Either way because this incompatibility we will likely be removing the static section to prevent future Java ad Ruby Devs from adding something that could fail to load in Python.

UnsatisfiedLinkError for JNI library

I have the following Java class that invokes a native library on Linux (/usr/local/lib/libCAPJni.so):
public class MyClass {
private native float runCAP(String name, int[] data);
private static final String LD_LIBRARY_PATH = "/usr/local/lib";
static {
System.setProperty("java.library.path", LD_LIBRARY_PATH);
System.loadLibrary("CAPJni");
}
...
}
The native library libCAPJni.so is located in /usr/local/lib. I also set this lib path in my Eclipse's Build Path -> Native library location. However when I launched my application in Eclipse, I got the following error:
java.lang.UnsatisfiedLinkError: no CAPJni in java.library.path
This seems to be caused by that the native lib is not found in /usr/local/lib directory. But if I directly run the following in command line:
java -Djava.library.path=/usr/local/lib MyClass
It runs fine without any problem. Why can't my Tomcat web application find the native library?

Specifying the native library path in Eclipse should definitely work.
However, it does NOT work to specify the library path at runtime with setting the system property. The library path must be known and set when booting the jvm.
I haven't worked with tomcat yet but as you mentioned it, make sure that tomcat starts with this parameter as well which might be your problem here.
EDIT:
As stated here it might actually be possible to change the library path later on. But as this is a hack and messes with the classloader it is most likely not a good idea to use that in the tomcat environment.

Best way to override MATLAB's default static javaclasspath

MATLAB is configured to search its static java class path before searching the user-modifiable dynamic path. Unfortunately, the static path contains quite a number of very old public libraries, so if you are trying to use a new version you may end up loading the wrong implementation and get errors.
For instance, the static path contains an old copy of the google-collections.jar, which has long been supplanted by Google's guava library and which has some of the same class names (e.g. com.google.common.base.Objects). As a result, if you invoke a Guava method that uses a newer method of one of such a class, you will end up getting surprising NoSuchMethodErrors because the google-collections jar is found first.
As of R2012b, MATLAB lets you specify additional jars to add to the static path by putting a javaclasspath.txt file in your preferences folder, but that adds jars to the end of the path, and doesn't let you override jars that are built into MATLAB.
So what is the best way around this?

I got an official response from Mathworks:
As of MATLAB R2013a (also in R2012b), classes can be added to the front of the static Java class path by including the following line in javaclasspath.txt:
<before>
Any directory that is after this line in javaclasspath.txt will be added to the front of the static Java class path. This is an undocumented use of javaclasspath.txt as of R2013a.
But overall in MATLAB, the ability to add classes to the front of the static Java classpath is not available through javaclasspath.txt in MATLAB 8.0 (R2012b).
MATLAB searches for classpath.txt in the following order:
In the startup directory. As of MATLAB 8.0 (R2012b) a warning will be shown if the file is found there and it will be ignored.
In the first directory on the MATLABPATH environment variable. (This environment variable is used in the bin/matlab shell script on Linux and in general is not used by the end-user).
In the toolbox/local directory.
Although the MATLABPATH environment variable of point 2 is normally not used by end-users we can use it in a workaround to allow reading a custom classpath.txt outside of the toolbox/local directory.
On Windows:
You will need to create the MATLABPATH environment variable. The first directory on it should be your directory with the custom classpath.txt AND you will also need to add the toolbox\local directory as second option. So from a cmd prompt you could do:
set MATLABPATH=c:\Users\user\Documents\myMATLABClasspath;c:\Program Files\MATLAB\R2012b
\toolbox\local
matlab.exe

One hack that appears to work is to add the jar to the top of the classpath.txt file that can be found in your MATLAB installations toolbox/local folder. Unfortunately, this is automatically generated and may get rewritten at some unspecified time, such as when you install new toolboxes, so this approach would require you to have some way to notice when this happens and reapply the hack.

If you're distributing a jar that's intended to be used with matlab, it may be better to use proguard as described at http://code.google.com/p/guava-libraries/wiki/UsingProGuardWithGuava.
If you specify that all of your classes and their (public) fields and methods are to be preserved and include guava as a program jar (not a library), then it will rename all of guava's methods and update your compiled bytecode to reference the new names.
It seems a bit hackish, but depending on the audience, it may be significantly easier than teaching your users about static vs. dynamic classpath, and it won't break any matlab code that depends on the old behavior.

Instead of obfuscating the package as suggested by #user2443532, I have found it easier to "shade" the conflicting package instead of obfuscating it - unless you actually need obfuscation. One easy way to do this is to build your package using Maven and use the maven-shade-plugin. Internal calls are modified automatically, so you don't need to modify any of the Java code.
Direct calls from Matlab will need to be modified - for example, calls to com.opensource.Class become shaded.com.opensource.Class.
For more info on shading, see What is the maven-shade-plugin used for, and why would you want to relocate Java packages?

Requiring Java Classes in LuaJ

I'm attempting to make a game library in Java that uses Lua for the scripts. The real issue appears when I try to require a Java class (that is inside of a jar), and whenever I try to do so, I get an error much like the one below:
Exception in thread "main" org.luaj.vm2.LuaError: #/C:/xampp/htdocs/LevelDesigner/Projects/Lua Test/bin/levels/Test.lua:2 module
'resources.GameLevel' not found: resources.GameLevel
no field package.preload['resources.GameLevel']
How can I require a Java class that is within a jar? Right now it seems that, with Lua, I can only require .lua files, and not .class files. This is obviously problematic as Java files are compiled down to class files...And that is what I need to require.

The answer to this question is to use luajava.bindClass as opposed to require in all of your Lua scripts.

Loading DLL in Java - Eclipse - JNI

I am trying to load a dll in java using the following code
System.loadLibrary("mydll");
The project is placed in D:\development\project\ and i have placed the dll on D:. I then gave following VM argument in eclipse configuration
-Djava.library.path=D:/
But when i run i get UnsatisifiedLinkerError. After googling a bit, I used
System.load("D:\mydll.dll");
but again getting the same problem, could someone can help?

Where you specify the DLL filename in the library path, omit that. Additionally, your System.loadLibrary call should just be 'mydll'. I can tell you (from experience) that if you put the DLL in the root of your project in Eclipse (i.e., D:\Eclipse Workspace\Proj), it should work. Any further linker errors could be from dependency problems with finding other DLLs. The exception is the same. Use something like Dependency Walker (http://www.dependencywalker.com/) to see if your DLL relies on anything else not on the system library path.
Edit: UnsatisfiedLinkError: Thrown if the Java Virtual Machine cannot find an appropriate native-language definition of a method declared native -- it seems like you are using a JNI function which does not exist.

One problem you have is:
System.load("D:\mydll.dll");
should be
System.load("D:\\mydll.dll");
or
System.load("D:/mydll.dll");
I have had more success with System.load, but loadlibrary is better designed for multiplatform.
It figures out the extension for you.

Check out how to properly set up the native dependencies here. Additionally, make sure you use the correct JVM: in my case, the DLL was not found because it was a 32 bit DLL, but I used the x64 JVM!

Using System.loadLibrary("mydll") works fine, you can also use that one. If you used javah and you think with your DLL everything is fine, there are two possibilies:
The JVM does not find your DLL: In this case, your java library path is not correct (which I doubt) and you should probably set it to . and place your DLL in the current working dir.
The JVM does not find a DLL your DLL depends on: If you have any dependent libraries in your DLL, they are NOT searched by the JVM, but by Windows itself. And Windows does not know the java.library.path, so it will look in the system PATH variable for those. If you have the possibility, you can set the system PATH variable to the location of your DLLs before starting the JVM and everything will be fine. Or you can load all your DLLs using the JVM like this
System.loadLibrary("dll_1");
System.loadLibrary("dll_2");
System.loadLibrary("dll_3");
where dll_3.dll depends on dll_2.dll, which depends on dll_1.dll.
Hope that helps.

System.loadLibrary loads the DLL from the JVM path (JDK bin path).
If you want to load an explicit file with a path, use System.load()
See also: Difference between System.load() and System.loadLibrary in Java
public class MyClass
{
static
{
System.load("MyJNI.dll");
}
}

Put your Almafa.dll into the C:/Java/jre7/lib or /bin sorry, I can`t remember exactly. After you have done no more configuration needed, just say
static{
System.LoadLibrary("Almafa");
}
in the class, where you want to load it. It is works only in Java project, in Android like project you need to use JNI. I had posted now the result of 3 days no sleeping :)

I got my error resolved by using the following:
static {
try {
System.loadLibrary("myDLL");
} catch (Exception e) {
e.printStackTrace();
}
}
Instead of using System.load("myDLL.dll")

#alee- You can just copy and the paste the dll files in system32 folder of your windows and try to call the library through the System.loadLibrary("mydll")... i guess it may work...

Give the library path in your project as native library location,seems to be solved.