I am working on application that has components written in several languages. I am trying to get functionality that works fine in Java working in Jython. There is some native/C++ functionality that Java access via the JNI and is wrapped by SWIG.
Whenever I try to import all of the classes in the project I get errors that PROJECTJNI cannot be linked. Here is my minimum case to produce:
import sys
sys.path.append('PROJECT.jar')
from com.whatever.project import *
Here is the error message when this is executed:
$ jython Bootstrap.py
"my" variable $jythonHome masks earlier declaration in same scope at /usr/bin/jython line 15.
Traceback (most recent call last):
File "Bootstrap.py", line 9, in <module>
from com.whatever.project import *
java.lang.UnsatisfiedLinkError: com.whatever.project.PROJECTJNI.swig_module_init()V
at com.whatever.project.PROJECTJNI.swig_module_init(Native Method)
at com.whatever.project.PROJECTJNI.<clinit>(PROJECTJNI.java:974)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at org.python.core.Py.loadAndInitClass(Py.java:909)
at org.python.core.Py.findClassInternal(Py.java:844)
at org.python.core.Py.findClass(Py.java:869)
at org.python.core.packagecache.PackageManager.basicDoDir(PackageManager.java:107)
at org.python.core.packagecache.SysPackageManager.doDir(SysPackageManager.java:138)
at org.python.core.PyJavaPackage.fillDir(PyJavaPackage.java:123)
at org.python.core.imp.importAll(imp.java:1051)
at org.python.core.imp.importAll(imp.java:1039)
at org.python.pycode._pyx0.f$0(Bootstrap.py:9)
at org.python.pycode._pyx0.call_function(Bootstrap.py)
at org.python.core.PyTableCode.call(PyTableCode.java:165)
at org.python.core.PyCode.call(PyCode.java:18)
at org.python.core.Py.runCode(Py.java:1275)
at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:235)
at org.python.util.jython.run(jython.java:247)
at org.python.util.jython.main(jython.java:129)
java.lang.UnsatisfiedLinkError: java.lang.UnsatisfiedLinkError: com.whatever.project.PROJECTJNI.swig_module_init()V
The line 15 thing shows up on any time we invoke jython, we had been ignoring it.
We can get the classes from the Java Project to work by simply loading them one at a time:
com.whatever.project import Class1
com.whatever.project import Class2
...
com.whatever.project import Class50
This is highly impractical because even for short python script we might need a dozen classes. Many of them are exceptions that we are catching which have unique types. So anything that robustly handles errors might need a huge number of classes.
As per the jython documentation I should be able to hide PROJECTJNI so it is not loaded by doing something like this, but I found the docs less than perfectly clear. Here is what I attempted:
import com.whatever.project
__all__ = dir(com.whatever.project)
__all__.remove('PROJECTJNI')
from com.whatever.project import *
But this fails with errors and still clearly trying to load PROJECTJNI.
I also tried to fix the native executable so it could be linked against correctl. I learned another group using JRuby had no issue including everything, so I decided to check the source and the binary. I found void swig_module_init() in the file Project_wrap.cpp that Swig created. It was hidden behind a macro, but it was there, and objdump confirms:
$objdump libPROJECT.so -t |grep PROJECTJNI |grep init
000000000051a900 l O .data 00000000000009d0 _ZZ59Java_com_whatever_project_PROJECTJNI_swig_1module_1initE7methods
0000000000263f66 g F .text 00000000000000d1 Java_com_whatever_project_PROJECTJNI_swig_1module_1init
Am I doing something wrong with any of my troubleshooting steps? Is the a bug in Jython? Is there a simple Python workaround to make it skip loading PROJECTJNI?
Anything that lets me skip linking this or makes this link correctly will be accepted.
On one of the Java classes, the BinaryLoader, there was a method called void load_binary() that is called in a static section of that class:
public class BinaryLoader
{
static
{ load_binaries(); }
public static void load_binaries()
{
// Deep inside here is a call to actually load the shared library. Using
load_shared_library_from_jar("PROJECT");
}
... // more details here
}
Our Java code using the native API called this method as the class was loaded. Our JRuby code called this when the class was 'touched', just using the Constant's name, the name of the class, invoked the static section. To clarify here is a sample from our docs and live Ruby scripts that does this:
# Tells Jruby to load the Java Interopability layer.
require 'java'
# Loads the PROJECT Java tools.
require 'PROJECT.jar'
# Shorten the PROJECT tool names from their obnoxiously long Java name.
module PROJECT
include_package "com.whatever.project"
end
# Let Ruby know the BinaryLoader exists and it will have PROJECT load all the system binaries.
PROJECT::BinaryLoader
Apparently requiring the JarFile is enough to make all the Native symbols and Java classes available in Ruby and the binary isn't loaded until the last line which is just an expressions that resolves to the name of the class and force the static section to run. The actual method call would have looked like: PROJECT::BinaryLoader.load_binaries
In Jython the static section was never invoked even when calling other static methods on the BinaryLoader class. I forced it to call the method manually by importing the minimum of Java and Native symbols to call the method:
# Tells Jython to load the python system Interopability tools.
import sys
# When searching for python symbols, this Java jar should be search also.
sys.path.append('PROJECT.jar')
# Load just enough the have the JVM Know how to load the native PROJECT libraries
from com.whatever.project import PROJECT
from com.whatever.project import BinaryLoader
# Load any native binaries that are required
BinaryLoader.load_binaries()
# After this line is run in any given script then PROJECT can be used.
from com.whatever.project import *
Clearly the native binaries were simply not loaded as had been thought. Anything that loaded these would have been an acceptable solution.
This works around Jython not invoking the static section, by doing the work it should have as early as possible. This seems like a bug in Jython, but it might be that guarantees on static load order are not strongly enforced in Java. Either way because this incompatibility we will likely be removing the static section to prevent future Java ad Ruby Devs from adding something that could fail to load in Python.
Related
I would like to use jython for basic web scraping task rather than learning java. To learn the basics I'm using an example from http://blog.databigbang.com/web-scraping-ajax-and-javascript-sites/ I've been unsuccessfully trying to run the gartner.py code from Windows cmd. Could anyone suggest a resolution to why both
jython -J-classpath "path\to\the\jars\*" path\to\gartner.py
and
jython path\to\gartner.py
keep on throwing out
Traceback (most recent call last):
File "path\to\gartner.py", line 1, in <module>
import com.gargoylesoftware.htmlunit.WebClient as WebClient
ImportError: No module named gargoylesoftware
given I've got environment variables set up for jython path\to\jython\bin, for java path\to\Java\jdk-14.0.1\bin and for the htmlunit-2.40.0 I've added path\to\jars\htmlunit-2.40.0\lib to the CLASSPATH.
I understand that jython should pick up the specified package in jython -J-classpath "path\to\the\jars\*" path\to\gartner.py but it does not find it. Also, I understand that in the case of jython path\to\gartner.py the defined CLASSPATH variable is available to Java pointing at htmlunit-2.40.0 (as mentioned above) whilst jython serves only as a translator from python to java. So - in my understanding - java shouild have all parameters available to import the desired module. Please, could anyone confirm?
I appreciate this subject has been somewhat discussed but there is no clear resolution available. What could I be missing?
The error looks very clearly like you're missing a Java dependency. The jython issue with this specific library has already been discussed in a different thread: instantiating a webclient object in jython giving strange results
I'm trying to open a jar file and execute it's main function, but jpype is throwing an error that doesn't make sense to me. Here is my code:
jpype.startJVM(jpype.getDefaultJVMPath(), '-Djava.class.path="%s"' % jar)
CommandLine = jpype.JPackage('phylonet').coalescent.CommandLine
CommandLine.main(['-i', input_file, '-o', output_file])
jpype.shutdownJVM()
I get this error:
TypeError: Package phylonet.coalescent.CommandLine.main is not Callable
I've provided the absolute path to the jar file, and I've gotten the main function from META-INF/MANIFEST.MF:
cat tmp/META-INF/MANIFEST.MF | grep Main-Class
Main-Class: phylonet.coalescent.CommandLine
The jar file I'm trying to open is called astral, from here: https://github.com/smirarab/ASTRAL
Calling it like this works as expected:
java -Djava.class.path="./astral.jar"
So why not when I call it with jpype?
First of all, I have tested your code on my own jarfile. Indeed, I was presented with such error:
TypeError: Package clip.frontend.Start.main is not Callable
Then, after reading the docs carefully, I've used another method.
import jpype
# I've used other set of parameters to JVM, and modified a bit your classpath setting.
jpype.startJVM(jpype.getDefaultJVMPath(), "-ea", "-Djava.class.path=clip.jar")
# Second difference, I decided to use JClass because it was more clear for me.
# Parameter array was kept empty.
jpype.JClass("clip.frontend.Start").main([])
jpype.shutdownJVM()
And the output was correct:
% python2 main.py
2 2
+>+[<[>>+>+<<<-]>>[<<+>>-]>[[-]>>>>>>+<<<<<<<<<[-]>[-]>>>>>>>>[<<<<<<<<+>+>>>>>>>-]
<<<<<<<[>>>>>>>+<<<<<<<-]>>>>>>>[-]<<<<<<]<<<[>>+>+<<<-]>>[<<+>>-]>[[-]>>>>>>++
[<<<<<+>>>>>>>>>>>>+<<<<<<<-]<<<<<[>>>>>+<<<<<-]>>>>>>>>>>>>>[>>]+<<[<<]>[>[>>]
<+<[<<]>-]<<<<<<<[-]++[<<<<<+>>>>>>>>>>>>+<<<<<<<-]<<<<<[>>>>>+<<<<<-]>>>>>>>>>>>>>
[>>]+<<[<<]>[>[>>]<+<[<<]>-]<<<<<<<[-]#JVM has been shutdown
Now, I decided to translate my solution to match your problem:
import jpype
jpype.startJVM(jpype.getDefaultJVMPath(), "-ea", "-Djava.class.path=astral.jar")
jpype.JClass("phylonet.coalescent.CommandLine").main([])
jpype.shutdownJVM()
And the code works correctly. More important than the actual solution is the fact, why doesn't your code work. You used wrong set of parameters and specified the classpath in the other way.
Replacing JClass with JPackage, the code still works.
import jpype
jpype.startJVM(jpype.getDefaultJVMPath(), "-ea", "-Djava.class.path=astral.jar")
jpype.JPackage('phylonet').coalescent.CommandLine.main([])
jpype.shutdownJVM()
As the way you extract classes from classpath is correct, the only possible cause is specifying invalid parameter set. After removing -ea the code still works, so mistake you made lies in this fragment of code.
'-Djava.class.path="%s"' % jar
And in fact, I've used this in opposition to my answer, and bam, the code yields this:
TypeError: Package phylonet.coalescent.CommandLine.main is not Callable
This means, the parameter contained following:
-Djava.class.path="astral.jar"
instead of following
-Djava.class.path=astral.jar
The quotes were misplaced and raised the error in result.
This was a classic issue with JPype. If the jar can't be loaded then JPackage will return another JPackage which is not callable. Common causes of a failure to load include
The JVM loaded does not support the version of the jar (Check that getDefaultJVMPath() is not some old version)
A jar dependency is missing.
The JVM could not find the Jar as the specified path.
The previous solution was to use java.lang.Class.forName which would print the diagnostics on the jar loading. Version 0.7.0 which is currently in available as a release candidate has addressed this.
Also it is recommended that you use jpype.imports or JClass rather than JPackage when importing a class. It is much safer as it will report a more meaningful error. For example:
import jpype
import jpype.imports
jpype.startJVM()
jpype.imports.registerDomain('phylonet') # This is required as phylonet is not a tld
from phylonet.coalescent import CommandLine
You can mark a package as being conforming (Classes start upper, packages are lower) to force an error.
This post is related to this one : How to put properly a libgdx application inside swing application? but the question is different.
In this post I explained that I have two JARs : WorldEditor.jar and GameEngine.jar, and I load GameEngine.jar at runtime from WorldEditor.jar.
My problem is with one of the libraries included in GameEngine.jar, namely JRuby.
When I run java -jar GameEngine.jar everything is fine, but when I launche java -jar worldEditor.jar, the instance of JRuby ScriptManager I use returns null when I call ``getEngineByName`. I just can't point ou what is the problem.
By tracing the list of ScriptManagerFactories, I saw that in the good case I have [JRuby, Rhino], and in the bad one I have only [Rhino].
Would someone have an idea of what's going on ?
I don't have much experience with Java's ScriptEngine, but I ran into this issue while answering another question here. I think your problem boils down to classpath order issues.
Using this code:
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;
public class Script {
public static void main(String args[]) throws Exception {
ScriptEngineManager manager = new ScriptEngineManager();
ScriptEngine engine = manager.getEngineByName("jruby");
System.out.println(engine);
}
}
If I run it two different ways, I get two different results:
$ java -cp .:jruby.jar Script
null
$ java -cp jruby.jar:. Script
org.jruby.embed.jsr223.JRubyEngine#30c01f1c
Looking into it a bit, there is a special file in the jar that registers the various scripting containers:
The ScriptEngineManager uses the service provider mechanism described
in the Jar File Specification to obtain instances of all
ScriptEngineFactories available in the current ClassLoader.
My guess is that the JVM doesn't need to load the JRuby jar in the first case, so it hasn't registered the scripting engine. It would only load that jar when it cannot find some class. This means you might be able to force it to work by using some JRuby object before you ever call into the script.
I'm working on a Java/JRuby project which needs to be able to be able to interact with GAMS. I know we can use the Java API, but I would really like to be able to access it using JRuby if possible, since we're hoping to eventual add a DSL and some other complexity I'm not really excited about having to implement in pure Java.
Following the official Java API documentation for GAMS, I have downloaded and setup everything necessary to run GAMS from the command line, but I can't figure out how to include the GAMS directory in LD_LIBRARY_PATH and still run JRuby irb. When I run
export LD_LIBRARY_PATH=/home/wikk/Downloads/gams24.0_linux_x64_64_sfx
Then try to run irb with JRuby, I get
jruby: /home/wikk/Downloads/gams24.0_linux_x64_64_sfx/libstdc++.so.6: version 'GLIBCXX_3.4.15' not found (required by jruby)
I think this is what the documentation is asking me to do to run a Java program that calls the API, is there maybe some way to set LD_LIBRARY_PATH in irb, but before importing all the Java class files? I can do this successfully if I don't set LD_LIBRARY_PATH, but then GAMS tells me it can't find the main program when I try to create a new GAMSWorkspace object:
irb(main):002:0> ws = GAMSWorkspace.new
Java::ComGamsApi::GAMSException: could not find a GAMS system directory from
your environment variable, please set up properly before running a program!
from com.gams.api.GAMSWorkspace.verifySystemDirectory(GAMSWorkspace.java:335)
Am I doing this wrong? or does the API require some Java feature that isn't implemented in JRuby?
Finally came back to this problem, got it working through some trial and error. I also needed to run jruby with the -J-Djava.library.path=[GAMSDIR]/apifiles/Java/api flag, and add [GAMSDIR]/apifiles/Java/api/GAMSJavaAPI.jar to the classpath.
Once this is all in place, you can run gams models from ruby scripts:
import com.gams.api.GAMSWorkspace
import com.gams.api.GAMSJob
import com.gams.api.GAMSVariable
import com.gams.api.GAMSVariableRecord
import com.gams.api.GAMSWorkspace
ws = GAMSWorkspace.new
j1 = ws.addJobFromGamsLib('trnsport')
j1.run
j1.out_db.get_variable('x').each_entry do |rec|
puts "x(#{rec.get_keys[0]}, #{rec.get_keys[1]}): level = #{rec.get_level}, marginal = #{rec.get_marginal}"
end
I am writing here because it is the only thread related to the GAMS Java API problem.
In Eclipse, you have to go to "Run Configurations" and add two things:
1. (As already said) add a "-Djava.library.path=[GAMSDIR]\apifiles\Java\api\" to VM arguments
2. Go to Environment and SET explicitly a PATH variable to [GAMSDIR]. For some reason seeting path through windows is not working
I'm attempting to make a game library in Java that uses Lua for the scripts. The real issue appears when I try to require a Java class (that is inside of a jar), and whenever I try to do so, I get an error much like the one below:
Exception in thread "main" org.luaj.vm2.LuaError: #/C:/xampp/htdocs/LevelDesigner/Projects/Lua Test/bin/levels/Test.lua:2 module
'resources.GameLevel' not found: resources.GameLevel
no field package.preload['resources.GameLevel']
How can I require a Java class that is within a jar? Right now it seems that, with Lua, I can only require .lua files, and not .class files. This is obviously problematic as Java files are compiled down to class files...And that is what I need to require.
The answer to this question is to use luajava.bindClass as opposed to require in all of your Lua scripts.