I would like to use jython for basic web scraping task rather than learning java. To learn the basics I'm using an example from http://blog.databigbang.com/web-scraping-ajax-and-javascript-sites/ I've been unsuccessfully trying to run the gartner.py code from Windows cmd. Could anyone suggest a resolution to why both
jython -J-classpath "path\to\the\jars\*" path\to\gartner.py
and
jython path\to\gartner.py
keep on throwing out
Traceback (most recent call last):
File "path\to\gartner.py", line 1, in <module>
import com.gargoylesoftware.htmlunit.WebClient as WebClient
ImportError: No module named gargoylesoftware
given I've got environment variables set up for jython path\to\jython\bin, for java path\to\Java\jdk-14.0.1\bin and for the htmlunit-2.40.0 I've added path\to\jars\htmlunit-2.40.0\lib to the CLASSPATH.
I understand that jython should pick up the specified package in jython -J-classpath "path\to\the\jars\*" path\to\gartner.py but it does not find it. Also, I understand that in the case of jython path\to\gartner.py the defined CLASSPATH variable is available to Java pointing at htmlunit-2.40.0 (as mentioned above) whilst jython serves only as a translator from python to java. So - in my understanding - java shouild have all parameters available to import the desired module. Please, could anyone confirm?
I appreciate this subject has been somewhat discussed but there is no clear resolution available. What could I be missing?
The error looks very clearly like you're missing a Java dependency. The jython issue with this specific library has already been discussed in a different thread: instantiating a webclient object in jython giving strange results
Related
I am working on application that has components written in several languages. I am trying to get functionality that works fine in Java working in Jython. There is some native/C++ functionality that Java access via the JNI and is wrapped by SWIG.
Whenever I try to import all of the classes in the project I get errors that PROJECTJNI cannot be linked. Here is my minimum case to produce:
import sys
sys.path.append('PROJECT.jar')
from com.whatever.project import *
Here is the error message when this is executed:
$ jython Bootstrap.py
"my" variable $jythonHome masks earlier declaration in same scope at /usr/bin/jython line 15.
Traceback (most recent call last):
File "Bootstrap.py", line 9, in <module>
from com.whatever.project import *
java.lang.UnsatisfiedLinkError: com.whatever.project.PROJECTJNI.swig_module_init()V
at com.whatever.project.PROJECTJNI.swig_module_init(Native Method)
at com.whatever.project.PROJECTJNI.<clinit>(PROJECTJNI.java:974)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at org.python.core.Py.loadAndInitClass(Py.java:909)
at org.python.core.Py.findClassInternal(Py.java:844)
at org.python.core.Py.findClass(Py.java:869)
at org.python.core.packagecache.PackageManager.basicDoDir(PackageManager.java:107)
at org.python.core.packagecache.SysPackageManager.doDir(SysPackageManager.java:138)
at org.python.core.PyJavaPackage.fillDir(PyJavaPackage.java:123)
at org.python.core.imp.importAll(imp.java:1051)
at org.python.core.imp.importAll(imp.java:1039)
at org.python.pycode._pyx0.f$0(Bootstrap.py:9)
at org.python.pycode._pyx0.call_function(Bootstrap.py)
at org.python.core.PyTableCode.call(PyTableCode.java:165)
at org.python.core.PyCode.call(PyCode.java:18)
at org.python.core.Py.runCode(Py.java:1275)
at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:235)
at org.python.util.jython.run(jython.java:247)
at org.python.util.jython.main(jython.java:129)
java.lang.UnsatisfiedLinkError: java.lang.UnsatisfiedLinkError: com.whatever.project.PROJECTJNI.swig_module_init()V
The line 15 thing shows up on any time we invoke jython, we had been ignoring it.
We can get the classes from the Java Project to work by simply loading them one at a time:
com.whatever.project import Class1
com.whatever.project import Class2
...
com.whatever.project import Class50
This is highly impractical because even for short python script we might need a dozen classes. Many of them are exceptions that we are catching which have unique types. So anything that robustly handles errors might need a huge number of classes.
As per the jython documentation I should be able to hide PROJECTJNI so it is not loaded by doing something like this, but I found the docs less than perfectly clear. Here is what I attempted:
import com.whatever.project
__all__ = dir(com.whatever.project)
__all__.remove('PROJECTJNI')
from com.whatever.project import *
But this fails with errors and still clearly trying to load PROJECTJNI.
I also tried to fix the native executable so it could be linked against correctl. I learned another group using JRuby had no issue including everything, so I decided to check the source and the binary. I found void swig_module_init() in the file Project_wrap.cpp that Swig created. It was hidden behind a macro, but it was there, and objdump confirms:
$objdump libPROJECT.so -t |grep PROJECTJNI |grep init
000000000051a900 l O .data 00000000000009d0 _ZZ59Java_com_whatever_project_PROJECTJNI_swig_1module_1initE7methods
0000000000263f66 g F .text 00000000000000d1 Java_com_whatever_project_PROJECTJNI_swig_1module_1init
Am I doing something wrong with any of my troubleshooting steps? Is the a bug in Jython? Is there a simple Python workaround to make it skip loading PROJECTJNI?
Anything that lets me skip linking this or makes this link correctly will be accepted.
On one of the Java classes, the BinaryLoader, there was a method called void load_binary() that is called in a static section of that class:
public class BinaryLoader
{
static
{ load_binaries(); }
public static void load_binaries()
{
// Deep inside here is a call to actually load the shared library. Using
load_shared_library_from_jar("PROJECT");
}
... // more details here
}
Our Java code using the native API called this method as the class was loaded. Our JRuby code called this when the class was 'touched', just using the Constant's name, the name of the class, invoked the static section. To clarify here is a sample from our docs and live Ruby scripts that does this:
# Tells Jruby to load the Java Interopability layer.
require 'java'
# Loads the PROJECT Java tools.
require 'PROJECT.jar'
# Shorten the PROJECT tool names from their obnoxiously long Java name.
module PROJECT
include_package "com.whatever.project"
end
# Let Ruby know the BinaryLoader exists and it will have PROJECT load all the system binaries.
PROJECT::BinaryLoader
Apparently requiring the JarFile is enough to make all the Native symbols and Java classes available in Ruby and the binary isn't loaded until the last line which is just an expressions that resolves to the name of the class and force the static section to run. The actual method call would have looked like: PROJECT::BinaryLoader.load_binaries
In Jython the static section was never invoked even when calling other static methods on the BinaryLoader class. I forced it to call the method manually by importing the minimum of Java and Native symbols to call the method:
# Tells Jython to load the python system Interopability tools.
import sys
# When searching for python symbols, this Java jar should be search also.
sys.path.append('PROJECT.jar')
# Load just enough the have the JVM Know how to load the native PROJECT libraries
from com.whatever.project import PROJECT
from com.whatever.project import BinaryLoader
# Load any native binaries that are required
BinaryLoader.load_binaries()
# After this line is run in any given script then PROJECT can be used.
from com.whatever.project import *
Clearly the native binaries were simply not loaded as had been thought. Anything that loaded these would have been an acceptable solution.
This works around Jython not invoking the static section, by doing the work it should have as early as possible. This seems like a bug in Jython, but it might be that guarantees on static load order are not strongly enforced in Java. Either way because this incompatibility we will likely be removing the static section to prevent future Java ad Ruby Devs from adding something that could fail to load in Python.
I had to include some bittorrent java library to my Android project. My workspace: Android Studio 1.0.2 (osx) and jdk8. I've connected its maven-repository (ttorrent:1.4) with Gradle and after starting using main classes and features i've got an error:
java.lang.NoSuchMethodError: No static method encodeHex([BZ)[C in class Lorg/apache/commons/codec/binary/Hex; or its super classes (declaration of 'org.apache.commons.codec.binary.Hex' appears in /system/framework/ext.jar).
I went to library's code and find out that it's using org.apache.commons.codec from where ttorrent is importing encodeHex and calling it. Looks like binaryHex method is gone! Or it never been. But I went to commons.codec's code and found binaryHex in its place and with arguments that I was looking for. How come? Why? My Android Studio found it. But java runtime not.
In fact, the decision was more difficult than I thought. Let's start with the fact that I came across an article by Dieser Beitrag'a, from which it is clear that not one I had similar problems. The whole thing turned out that within the Android operating system already has some libraries that have a higher priority use, rather than loaded with dependencies along with the application. Among them there and my org.apache.commons.codec.
Yes, such things.
To solve the problem in two ways, either you need to pump source code library and using some tool to rename the project (i.e. org.apache.commons.codec to org.apache.commons.codec.android), collected it to a .jar file, include .jar in a project and at code use imports of the necessary classes only "our" library, or just get the required class to your project and do not pull a megabytes of unneeded code. However, I did just that.
Thanks for help!
I'm struggling trying to get remote actors setup in Scala. I'm running Scala 2.10.2 and Akka 2.2.1.
I compile using [I've shortened the paths on the classpath arg for clarity sake]:
$ scalac -classpath "akka-2.2.1/lib:akka-2.2.1/lib/scala-library.jar:akka-2.2.1/lib/akka:akka-2.2.1/lib/akka/scala-reflect-2.10.1.jar:akka-2.2.1/lib/akka/config-1.0.2.jar:akka-2.2.1/lib/akka/akka-remote_2.10-2.2.1.jar:akka-2.2.1/lib/akka/akka-kernel_2.10-2.2.1.jar:akka-2.2.1/lib/akka/akka-actor_2.10-2.2.1.jar:." [file.scala]
I've continuously added new libraries trying to debug this - I'm pretty sure all I really need to include is akka-remote, but the others shouldn't hurt.
No issues compiling.
I attempt to run like this:
$ scala -classpath "[same as above]" [application]
And I receive a NSM exception:
java.lang.NoSuchMethodException: akka.remote.RemoteActorRefProvider.<init>(java.lang.String, akka.actor.ActorSystem$Settings, akka.event.EventStream, akka.actor.Scheduler, akka.actor.DynamicAccess)
at java.lang.Class.getConstructor0(Class.java:2810)
at java.lang.Class.getDeclaredConstructor(Class.java:2053)
...
Looking into the source code, it appears that Akka 2.2.X's flavor of this constructor takes 4 arguments (the Scheduler is removed). But in Akka < 2.2.X, the constructor takes 5 args.
Thus, I'm thinking my classpath isn't setup quite right. At compile-time, Scala must be finding the <2.2.X flavor. I don't even know where it would be finding it, since I only have Akka 2.2.1 installed.
Any suggestions!? Thanks! (Please don't say to use SBT).
The problem here is that the Scala distribution contains akka-actor 2.1.0 and helpfully puts that in the boot class path for you. We very strongly recommend using a dependency manager like sbt or maven when building anything which goes beyond the most trivial projects.
As noted in another answer, the problem is that scala puts a different version of Akka into the bootclasspath.
To more directly answer your question (as you said you don't want to use sbt): you can execute your program with java instead of scala. You just have to put the appropriate Scala jars into the classpath.
Here is a spark-dev message about the problem. The important part is: "the workaround is to use java to launch the application instead of scala. All you need to do is to include the right Scala jars (scala-library and scala-compiler) in the classpath."
I'm working on a Java/JRuby project which needs to be able to be able to interact with GAMS. I know we can use the Java API, but I would really like to be able to access it using JRuby if possible, since we're hoping to eventual add a DSL and some other complexity I'm not really excited about having to implement in pure Java.
Following the official Java API documentation for GAMS, I have downloaded and setup everything necessary to run GAMS from the command line, but I can't figure out how to include the GAMS directory in LD_LIBRARY_PATH and still run JRuby irb. When I run
export LD_LIBRARY_PATH=/home/wikk/Downloads/gams24.0_linux_x64_64_sfx
Then try to run irb with JRuby, I get
jruby: /home/wikk/Downloads/gams24.0_linux_x64_64_sfx/libstdc++.so.6: version 'GLIBCXX_3.4.15' not found (required by jruby)
I think this is what the documentation is asking me to do to run a Java program that calls the API, is there maybe some way to set LD_LIBRARY_PATH in irb, but before importing all the Java class files? I can do this successfully if I don't set LD_LIBRARY_PATH, but then GAMS tells me it can't find the main program when I try to create a new GAMSWorkspace object:
irb(main):002:0> ws = GAMSWorkspace.new
Java::ComGamsApi::GAMSException: could not find a GAMS system directory from
your environment variable, please set up properly before running a program!
from com.gams.api.GAMSWorkspace.verifySystemDirectory(GAMSWorkspace.java:335)
Am I doing this wrong? or does the API require some Java feature that isn't implemented in JRuby?
Finally came back to this problem, got it working through some trial and error. I also needed to run jruby with the -J-Djava.library.path=[GAMSDIR]/apifiles/Java/api flag, and add [GAMSDIR]/apifiles/Java/api/GAMSJavaAPI.jar to the classpath.
Once this is all in place, you can run gams models from ruby scripts:
import com.gams.api.GAMSWorkspace
import com.gams.api.GAMSJob
import com.gams.api.GAMSVariable
import com.gams.api.GAMSVariableRecord
import com.gams.api.GAMSWorkspace
ws = GAMSWorkspace.new
j1 = ws.addJobFromGamsLib('trnsport')
j1.run
j1.out_db.get_variable('x').each_entry do |rec|
puts "x(#{rec.get_keys[0]}, #{rec.get_keys[1]}): level = #{rec.get_level}, marginal = #{rec.get_marginal}"
end
I am writing here because it is the only thread related to the GAMS Java API problem.
In Eclipse, you have to go to "Run Configurations" and add two things:
1. (As already said) add a "-Djava.library.path=[GAMSDIR]\apifiles\Java\api\" to VM arguments
2. Go to Environment and SET explicitly a PATH variable to [GAMSDIR]. For some reason seeting path through windows is not working
I am trying to access some of my clojure functions in my Java code (I am using Eclipse and CounterClockWise). Firstly, I was not sure how to do it? But then I found some answer here.
I am using clojure.lang.RT package for calling .clj files in my Java code. While naming namespaces, .clj files proper conventions has been followed like ns test.clojure.core is in test.clojure package where-in core is .clj file. Also upon following some information, I have added java-source-path (i.e. java-source-path src/test/java) and source-path (i.e. source-path src/test/clojure) in my project.clj.
But, when I run my Java file following error is given - a) Could not locate Clojure resource on classpath b) Attempting to call unbound fn.
Please if anyone in community could help me (assuming I am absolute beginner) through detailed procedure or tutorial I would be grateful, as only information I could get (for using clojure.lang.RT) is very little.
Looking forward to get a complete answer to end my frustration.
I guess you are using RT.loadResourceScript which is causing the proble.
You can try out something like this:
//Load the namespace
RT.var("clojure.core","eval").invoke(RT.var("clojure.core","read-string").invoke("(use 'test.clojure.core)"));
//Find a function in namespace
IFn fn = (IFn)RT.var("test.clojure.core","hello");
//Call that function
fn.invoke();