I want to access Java library within Ruby, in example Kafka already give jar for every operation, what things I need to do if I want to use it from Ruby?
Like maybe I just need to run shell command to run the Jar within Ruby, or do I need to port the library in Ruby? If it comes down to porting the library, how to do that too?
Thank you in advance
PS: The Java, Ruby, or Kafka are just examples. What I need to know is the big picture how to porting a library. Of course if you add some code example too I'll be more than happy :)
I agree with Aetherus that the fastest and most convenient way is to use JRuby. However I believe there other options than communicating with external Java processes. What to choose probably depends on what code you want to call. I see at least two other options.
Wrap the Java code you want to call in a main program and call it on the command line. This will be slow since Java needs to start and that takes forever but may be a fast way forward in some cases.
Call the Java code from C code that you compile with your Ruby. This will still need to load the JVM but you could probably make it happen only once. I found an article outlining how to get around doing it with JNI.
Both these paths will probably cause you a lot of pain but if it is important to stay on MRI it may be worth the trip. Have fun!
With JRuby, you can import the jar file, then include the classes you need in that jar:
require 'java'
require '/path/to/your.jar'
include_class 'com.really.long.ClassName'
But with Ruby implementations other than JRuby, you have no choice but to communicate with external java processes (through socket, IPC, kill, ...).
Related
I have a Java app that needs to integrate with a 3rd party library. The library is written in Python, and I don't have any say over that. I'm trying to figure out the best way to integrate with it. I'm trying out JEPP (Java Embedded Python) - has anyone used that before? My other thought is to use JNI to communicate with the C bindings for Python.
Any thoughts on the best way to do this would be appreciated. Thanks.
Why not use Jython? The only downside I can immediately think of is if your library uses CPython native extensions.
EDIT: If you can use Jython now but think you may have problems with a later version of the library, I suggest you try to isolate the library from your app (e.g. some sort of adapter interface). Go with the simplest thing that works for the moment, then consider JNI/CPython/etc if and when you ever need to. There's little to be gained by going the (painful) JNI route unless you really have to.
Frankly most ways to somehow run Python directly from within JVM don't work. They are either not-quite-compatible (new release of your third party library can use python 2.6 features and will not work with Jython 2.5) or hacky (it will break with cryptic JVM stacktrace not really leading to solution).
My preferred way to integrate the two would use RPC. XML RPC is not a bad choice here, if you have moderate amounts of data. It is pretty well supported — Python has it in its standard library. Java libraries are also easy to find. Now depending on your setup either Java or Python part would be a server accepting connection from other language.
A less popular but worth considering alternative way to do RPCs is Google protobuffers, which have 2/3 of support for nice rpc. You just need to provide your transport layer. Not that much work and the convenience of writing is reasonable.
Another option is to write a C wrapper around that pieces of Python functionality that you need to expose to Java and use it via JVM native plugins. You can ease the pain by going with SWIG SWIG.
Essentially in your case it works like that:
Create a SWIG interface for all method calls from Java to C++.
Create C/C++ code that will receive your calls and internally call python interpreter with right params.
Convert response you get from python and send it via swig back to your Java code.
This solution is fairly complex, a bit of an overkill in most cases. Still it is worth doing if you (for some reason) cannot afford RPCs. RPC still would be my preferred choice, though.
Many years later, just to add an option which is more popular these days...
If you need CPython functionality, py4j is a good option. py4j has seen seen frequent updates in 2016 2017 2018 2019 2020 and has gained some popularity, because it is used e.g. by Apache Spark to achieve CPython interoperability.
The best solutions, is to use Python programs throw REST API. You define your services and call them. You perhaps need to learn some new modules. But you will be more flexible for futures changes.
Here a small list of use full modules for this purpose:
Python modules
Flask
Flask-SQLAlchemy
Flask-Restful
SQlite3
Jsonify
Java modules (for calling rest api)
Jersey or Apache CXF
You will need a small Learning curve, but later you will get more productivity and modularity and even elasticity...
My other thought is to use JNI to communicate with the C bindings for Python.
I like very much JNA:
JNA provides Java programs easy access to native shared libraries (DLLs on Windows) without writing anything but Java code—no JNI or native code is required. This functionality is comparable to Windows' Platform/Invoke and Python's ctypes. Access is dynamic at runtime without code generation.
My 0.02$ :)
You could use a messaging service like ActiveMQ. It has both Python and Java support. This way, you can leave the complicated JNI or C bindings as they are and deal solely with what I consider a simple interface. Moreover, when the library gets updated, you don't need to change much, if anything.
Have you considered running Jython on the Java VM?
I've investigated a similar setup with JNI. Maybe this will help if haven't seen it yet:
http://wiki.cacr.caltech.edu/danse/index.php/Communication_between_Java_and_Python
http://jpe.sourceforge.net/
These are some of the tools which make it easier to bridge the gap between Python and Java:
1.Jython
Python implemented in Java
2.JPype
Allows Python to run java commands
3.Jepp
Java embedded Python
4.JCC
a C++ code generator for calling Java from C++/Python
5.Javabridge
a package for running and interacting with the JVM from CPython
6.py4j
Allows Python to run java commands.
7.voc
Part of BeeWare suite. Converts python code to Java bytecode.
8.p2j
Converts Python code to Java. No longer developed.
If you can get your Python code to work in Jython, then you should be able to use that to call it from Java:
http://jython.sourceforge.net/cgi-bin/faqw.py?req=show&file=faq06.001.htp
I also think that run command line in the Java wouldn't by bad practice (stackoverflow question here).
Potentially share the data through some database.
I like the way to connect two apps via the bash pipe, but I have not practice in this, so I'm wondering how difficult is to write logic to handle this on both python/java sides.
Or other productive way could be to use the Remote Procedure Call (RPC) which supports procedural programming. Using RPC you can invokes methods in shared environments. As an example you can call a function in a remote machine from the local computer using RPC. We can define RPC as a communication type in distributed systems.
(mentioned above by Marcin)
Or, very naive way would to be to communicate by the common file. But for sake of simplicity and speed, my vote is to use the shared database x rest API x socket communication.
Also I like the XML RPC as Marcin wrote.
I would like to recommend to avoid any complication to run Python under JVM or C++ binding. Better to use today trends which are obviously web technologies.
As a shared database the MongoDB may be good solution or even better the Redis as in memory database.
Use pipes to communicate with python(a python script calls python library) via subprocesses, the whole process is like java<-> Pipes <-> py.
If JNI, you should be familiar with python's bindings(not recommended) and compiled files like *.so to work with java.
The whole process is like py -> c -> .so/.pyd -> JNI -> jar.
Here is a good practice of stdio.
https://github.com/JULIELab/java-stdio-ipc
I am wondering if it is possible to call Python functions from Java code using Jython, or is it only for calling Java code from Python?
Jython: Python for the Java Platform - http://www.jython.org/index.html
You can easily call python functions from Java code with Jython. That is as long as your python code itself runs under jython, i.e. doesn't use some c-extensions that aren't supported.
If that works for you, it's certainly the simplest solution you can get. Otherwise you can use org.python.util.PythonInterpreter from the new Java6 interpreter support.
A simple example from the top of my head - but should work I hope: (no error checking done for brevity)
PythonInterpreter interpreter = new PythonInterpreter();
interpreter.exec("import sys\nsys.path.append('pathToModules if they are not there by default')\nimport yourModule");
// execute a function that takes a string and returns a string
PyObject someFunc = interpreter.get("funcName");
PyObject result = someFunc.__call__(new PyString("Test!"));
String realResult = (String) result.__tojava__(String.class);
As of 2021, Jython does not support Python 3.x
I think there are some important things to consider first with how strong you wish to have the linking between java and python.
Firstly Do you only want to call functions or do you actually want python code to change the data in your java objects? This is very important. If you only want to call some python code with or without arguments, then that is not very difficult. If your arguments are primitives it makes it even more easy. However if you want to have java class implement member functions in python, which change the data of the java object, then this is not so easy or straight forward.
Secondly are we talking cpython or will jython do? I would say cpython is where its at! I would advocate this is why python is so kool! Having such high abstractions however access to c,c++ when needed. Imagine if you could have that in java. This question is not even worth asking if jython is ok because then it is easy anyway.
So I have played with the following methods, and listed them from easy to difficult:
Java to Jython
Advantages: Trivially easy. Have actual references to java objects
Disadvantages: No CPython, Extremely Slow!
Jython from java is so easy, and if this is really enough then great. However it is very slow and no cpython! Is life worth living without cpython I don't think so! You can easily have python code implementing your member functions for you java objects.
Java to Jython to CPython via Pyro
Pyro is the remote object module for python. You have some object on a cpython interpreter, and you can send it objects which are transferred via serialization and it can also return objects via this method. Note that if you send a serialized python object from jython and then call some functions which change the data in its members, then you will not see those changes in java. You just need to remember to send back the data which you want from pyro. This I believe is the easiest way to get to cpython! You do not need any jni or jna or swig or .... You don't need to know any c, or c++. kool huh?
Advantages: Access to cpython, not as difficult as following methods
Disadvantages: Cannot change the member data of java objects directly from python. Is somewhat indirect, (jython is middle man).
Java to C/C++ via JNI/JNA/SWIG to Python via Embedded interpreter (maybe using BOOST Libraries?)
OMG this method is not for the faint of heart. And I can tell you it has taken me very long to achieve this in with a decent method. Main reason you would want to do this is so that you can run cpython code which as full rein over you java object. There are major major things to consider before deciding to try and bread java (which is like a chimp) with python (which is like a horse). Firstly if you crash the interpreter that's lights out for you program! And don't get me started on concurrency issues! In addition, there is allot allot of boiler, I believe I have found the best configuration to minimize this boiler but still it is allot! So how to go about this:
Consider that C++ is your middle man, your objects are actually c++ objects! Good that you know that now. Just write your object as if your program as in cpp not java, with the data you want to access from both worlds. Then you can use the wrapper generator called swig (http://www.swig.org/Doc1.3/Java.html) to make this accessible to java and compile a dll which you call System.load(dll name here) in java. Get this working first, then move on to the hard part!
To get to python you need to embed an interpreter. Firstly I suggest doing some hello interpreter programs or this tutorial Embedding python in C/C. Once you have that working, its time to make the horse and the monkey dance! You can send you c++ object to python via [boost][3] . I know I have not given you the fish, merely told you where to find the fish. Some pointers to note for this when compiling.
When you compile boost you will need to compile a shared library. And you need to include and link to the stuff you need from jdk, ie jawt.lib, jvm.lib, (you will also need the client jvm.dll in your path when launching the application) As well as the python27.lib or whatever and the boost_python-vc100-mt-1_55.lib.
Then include Python/include, jdk/include, boost and only use shared libraries (dlls) otherwise boost has a teary. And yeah full on I know. There are so many ways in which this can go sour. So make sure you get each thing done block by block. Then put them together.
It's not smart to have python code inside java. Wrap your python code with flask or another web framework to make it a microservice. This makes your java program able to call this microservice (e.g. via REST).
This approach is simple and it will save you tons of issues. And the codes are loosely coupled so they are scalable.
Updated on Mar 24th 2020:
According to #stx's comment, the above approach is not suitable for massive data transfer between client and server.
Here is another approach I recommended:
Connecting Python and Java with Rust(C/C++ also ok).
https://medium.com/#shmulikamar/https-medium-com-shmulikamar-connecting-python-and-java-with-rust-11c256a1dfb0
Several of the answers mention that you can use JNI or JNA to access cpython but I would not recommend starting from scratch because there are already open source libraries for accessing cpython from java. For example:
JEP
JPY
GraalVM is a good choice. I've done Java+Javascript combination with GraalVM for microservice design (Java with Javascript reflection). They recently added support for python, I'd give it a try especially with how big its community has grown over the years.
UPDATE June 2021
https://www.graalvm.org/reference-manual/python/ says
GraalVM provides a Python 3.8 compliant runtime. A primary goal of the GraalVM Python runtime is to support SciPy and its constituent libraries, as well as to work with other data science and machine learning libraries from the rich Python ecosystem. At this point, the Python runtime is made available for experimentation and curious end-users.
Here a library that lets you write your python scripts once and decide which integration method (Jython, CPython/PyPy via Jep and Py4j) to use at runtime:
https://github.com/subes/invesdwin-context-python
Since each method has its own benefits/drawbacks as explained in the link.
(3 modules: jep, Py4J, jython )
It depends on what do you mean by python functions? if they were written in cpython you can not directly call them you will have to use JNI, but if they were written in Jython you can easily call them from java, as jython ultimately generates java byte code.
Now when I say written in cpython or jython it doesn't make much sense because python is python and most code will run on both implementations unless you are using specific libraries which relies on cpython or java.
see here how to use Python interpreter in Java.
Depending on your requirements, options like XML-RPC could be useful, which can be used to remotely call functions virtually in any language supporting the protocol.
Jython has some limitations:
There are a number of differences. First, Jython programs cannot use CPython
extension modules written in C. These modules usually have files with the
extension .so, .pyd or .dll. If you want to use such a module, you should look
for an equivalent written in pure Python or Java. Although it is technically
feasible to support such extensions - IronPython does so - there are no plans
to do so in Jython.
Distributing my Python scripts as JAR files with Jython?
you can simply call python scripts (or bash or Perl scripts) from Java using Runtime or ProcessBuilder and pass output back to Java:
Running a bash shell script in java
Running Command Line in Java
java runtime.getruntime() getting output from executing a command line program
You can call any language from java using Java Native Interface
This gives a pretty good overview over the current options. Some of which are named in other answers. Jython is not usable until they decide to not implement Python 3. Many of the other projects are coming from the python side and want to access java. But there are a few options still, to name something which has not been named yet: gRPC
I have similar requirement, I think best solution is thrift for me(also a rpc solution), just run test passed successfully right now, and can use thrift-generator to gen thrift file from java interface, then gen python files and java client files from the thrift file
I am an iOS developer and I found a great library that handles all kinds of astronomy related calculations for me. The problem is that it's written in Java. Although I have enough experience with languages close to Java, I can't run Java on iOS.
I'd like to port it, but being that I've never ported anything before. Like I said, I don't think language will be the issue. It's a fairly simple library, involving mostly Date objects and math.
I'm trying to figure out the best way to do go about porting. Do I start with the core methods/functions of each class, or can I just go line by line in each file until I have translated everything?
Just start wherever you please. Generally, you'll want to start near the core, and work your way out. Just remember these three steps:
Make it work
Make it right
Make it fast/clean
So don't worry about getting it looking pretty or anything right off the hop, just get it working how it needs to, test it. Then look at where you can refactor if possible, etc.
Since this seem to be a library of functions, it would not depend on any platform specific functions.
Porting this should really be straightforward. You can port function by function, maybe start with functions that are important for your app.
The really tricky part (given it's a math lib) will be assuring that it works correctly: you should have the same set of unit tests as original lib. The best would be to create a java wrapper classes that call your lib via JNI so that you could run original unit tests on your lib. Pure Objective-C lib (no platform bindings) should run on both OS X and iOS, so you can run tests on OS X. Do something like this:
Look at original Java lib, and create functionally similar API in Objective-C: Same classes, same method names.
Take the original java classes and replace all content of methods with JNI calls to your native library.
Implement functionality in your lib.
Run java unit tests (which now calls your native lib) to make sure your lib works correctly.
For easier JNI development you can use JNA or any other JNI wrapper listed here.
I'm trying to make an java application to manage Linux servers from a web interface.
It is a bad idea to perform each task by calling bash shell ?
Are there any other options other than those to use C, Perl or another language?
It's not necessarily a bad idea to use bash to do the actual work. It would help if you gave us more of an idea what exactly the web interface was changing.
Java in particular does not provide much system-specific controls, because it was designed as a cross-platform language, so putting specific platform tools in would go against it's purpose.
You could certainly do it that way. Ideally you'd open up a port and accept specially crafted, specific actions which perform only the intended actions (an interface server) through a socket library.
I should think that the disadvantage(s) of calling Bash scripts for your commands is all related to error handling and return values. Each of your Bash scripts will need to return sensible, useful information to the Java app in the case of failures (or even successes). And you'll probably want a common interface for that such that each Bash script, no matter its function, returns the same types so that the Java can interpret it easily.
So, in that sense, making the changes from your Java program reduces the complexity of handing the information back and forth. On the other hand, if Bash is easier for you, you may find it more fast and flexible.
Opening up a single bash shell and sending commands (and properly parsing results) shouldn't be bad. It does tie your program to a single OS and even a single shell, but that's the nature of what you are trying to do.
What I wouldn't do is open a shell for each command and close it after each result. It seems to me that would cause unnecessary thrashing when many commands were executed in a row.
Security concerns jump at me. If you have a form like "type command" then someone clever could exploit some things. For example "configure network" is fine but "configure network; install rootkit" can be typed because a semi-colon allows commands to be chained.
All in all, this is not tight integration. If this is a personal project, go for it. It's a good learning project to turn a procedural script into a java program. If this is a serious effort to recreate the many various webapp admin tools there are, I'd seriously suggest skipping this. The VPanel/CPanel things I've seen I hate. There used to be a php based linux admin thing that looked ok but I just find them easily to learn, easy to outgrow because the net and community is full of command line knowledge.
If you are trying to automate a large deployment, look at the ruby-based puppet. It looks really neat.
I have a socket server were I perform different operations (using ifconfig by calling shell for example) and I plan to integrate an client in a JSF application.Because I'm not experienced ant I intend to make it my graduation project, but I'm not so sure if calling bash from java to configure a linux box is a good solution.
Java does have some Linux-configuring classes (well, at least, OpenJDK does). Check OperatingSystemMXBean or something like that.
You're beter to write your own server configuration utility in the language you prefer, sanitize it, make it secure and then call it via bash from your java app.
There are two questions here:
Should you try to do the configuration work in Java, or should you externalize it?
Assuming that you have externalized it, should you use bash scripts.
In this case, the answer to question 1) depends. This kind of thing can be difficult to implement in pure Java. This leaves two choices; externalize the task using a Process, or try to do it in a native code library via JNI or JPA. The latter approach is complicated and gives you JVM crashes if you make a mistake, so I would rule that out.
On the other hand, if you can find a good standard or 3rd party Java API that does what you need (without infesting your JVM with flakey JNI, etc), you should use that.
The answer to 2) is that bash scripts will work as well as any other scripting language. I think that using scripts gives you a bit more flexibility. For example, if you need to do things to compensate for differences in the different flavours of Linux, UNIX or maybe even Windows (!) you can put this into the externalized scripts. (A corollary is that the scripts need to be configurable, so don't embed them in your source code!)
Another alternative might be to run the commands (e.g. ifconfig) directly, using a fully qualified command name and supplying the arguments as an array of strings, etc. But unless your app is going to run the external command 100s of times a minute, it is probably not worth the (Java) coding effort and the loss of flexibility / configurability.
A lot of the response would depend on why you're doing this and why other more obvious solutions aren't possible. Knowing why you would choose to roll your own as opposed to installing Webmin would be good, or why you're choosing to use Web UI at all as opposed to VNC to control the box. There are some obvious responses to the later (firewall issues for instance), but there's no immediately obvious reason for the former. Without knowing more about the requirements, answering questions about implementation details like bash scripts versus perl or C is meaningless.
I'm just about to make my first trip into the world of JNI (Java Native Interface) to provide file system change notifications from platform specific C/C++ code to Java. That is unless someone suggest some brilliant library for doing this that I've missed.
Being new to JNI I've managed to find much documentation on the interface side of it and library generation side of it, but I've not found much on building the native library.
I've got an existing build based on ant for the pre-existing Java source, so I'm trying to work out if I should get ant to call make to create the library or if it's best to get make to call ant after creating the library?
Neither option jumps out as being terribly nice, but both seem better than trying to get ant to call a compiler to compile the code and generate the library directly.
I strongly dislike make because of its implicit ruleset and treatment of whitespace. Personally I would use cpp tasks (http://ant-contrib.sourceforge.net/cpptasks/index.html) to do my C compilation. They are not as flexible as make but they are also far less complex and it will mean you don't have to burden your developers with learning make.
As a simpler alternative to JNI, try JNA: https://jna.dev.java.net/, may solve this hassle for you and be simpler (assuming it can do what you want).
I'd skip JNI entirely, and use an external program which writes notifications on standard-output. Java can then simply read from the programs output stream and generate whatever event is necessary. JNI is way too much work if all you want is to send simple notifications.
Also, on Linux you can simply start "inotifywait" (with some suitable parameters, see "man inotifywait").
I'm working on something similar right now. Be aware that using swig from swig.org is often easier as it generates the stubs to the native library for you.
The short answer to your question is that the ant file should run the make file after the java library has already been built, as the native library depends on the swig generated header, which is generated from the java class files.
If you are super familiar with ant, and don't want to learn a new system, then http://ant-contrib.sourceforge.net/cpptasks/index.html, also linked by another poster, will let you build c++ in ant.
You could also try the terp C++ tasks at Codemesh. They are not free but they offer a high level of abstraction coupled with the ability to discover/specify the C++ compiler and the ability to iterate over more than one compiler/processor architecture/compiler configuration for multiplatform builds.