We Keep Coding

How to use python in a java project [duplicate]

I have a Java app that needs to integrate with a 3rd party library. The library is written in Python, and I don't have any say over that. I'm trying to figure out the best way to integrate with it. I'm trying out JEPP (Java Embedded Python) - has anyone used that before? My other thought is to use JNI to communicate with the C bindings for Python.
Any thoughts on the best way to do this would be appreciated. Thanks.

Why not use Jython? The only downside I can immediately think of is if your library uses CPython native extensions.
EDIT: If you can use Jython now but think you may have problems with a later version of the library, I suggest you try to isolate the library from your app (e.g. some sort of adapter interface). Go with the simplest thing that works for the moment, then consider JNI/CPython/etc if and when you ever need to. There's little to be gained by going the (painful) JNI route unless you really have to.

Frankly most ways to somehow run Python directly from within JVM don't work. They are either not-quite-compatible (new release of your third party library can use python 2.6 features and will not work with Jython 2.5) or hacky (it will break with cryptic JVM stacktrace not really leading to solution).
My preferred way to integrate the two would use RPC. XML RPC is not a bad choice here, if you have moderate amounts of data. It is pretty well supported — Python has it in its standard library. Java libraries are also easy to find. Now depending on your setup either Java or Python part would be a server accepting connection from other language.
A less popular but worth considering alternative way to do RPCs is Google protobuffers, which have 2/3 of support for nice rpc. You just need to provide your transport layer. Not that much work and the convenience of writing is reasonable.
Another option is to write a C wrapper around that pieces of Python functionality that you need to expose to Java and use it via JVM native plugins. You can ease the pain by going with SWIG SWIG.
Essentially in your case it works like that:
Create a SWIG interface for all method calls from Java to C++.
Create C/C++ code that will receive your calls and internally call python interpreter with right params.
Convert response you get from python and send it via swig back to your Java code.
This solution is fairly complex, a bit of an overkill in most cases. Still it is worth doing if you (for some reason) cannot afford RPCs. RPC still would be my preferred choice, though.

Many years later, just to add an option which is more popular these days...
If you need CPython functionality, py4j is a good option. py4j has seen seen frequent updates in 2016 2017 2018 2019 2020 and has gained some popularity, because it is used e.g. by Apache Spark to achieve CPython interoperability.

The best solutions, is to use Python programs throw REST API. You define your services and call them. You perhaps need to learn some new modules. But you will be more flexible for futures changes.
Here a small list of use full modules for this purpose:
Python modules
Flask
Flask-SQLAlchemy
Flask-Restful
SQlite3
Jsonify
Java modules (for calling rest api)
Jersey or Apache CXF
You will need a small Learning curve, but later you will get more productivity and modularity and even elasticity...

My other thought is to use JNI to communicate with the C bindings for Python.
I like very much JNA:
JNA provides Java programs easy access to native shared libraries (DLLs on Windows) without writing anything but Java code—no JNI or native code is required. This functionality is comparable to Windows' Platform/Invoke and Python's ctypes. Access is dynamic at runtime without code generation.
My 0.02$ :)

You could use a messaging service like ActiveMQ. It has both Python and Java support. This way, you can leave the complicated JNI or C bindings as they are and deal solely with what I consider a simple interface. Moreover, when the library gets updated, you don't need to change much, if anything.

Have you considered running Jython on the Java VM?

I've investigated a similar setup with JNI. Maybe this will help if haven't seen it yet:
http://wiki.cacr.caltech.edu/danse/index.php/Communication_between_Java_and_Python
http://jpe.sourceforge.net/

These are some of the tools which make it easier to bridge the gap between Python and Java:
1.Jython
Python implemented in Java
2.JPype
Allows Python to run java commands
3.Jepp
Java embedded Python
4.JCC
a C++ code generator for calling Java from C++/Python
5.Javabridge
a package for running and interacting with the JVM from CPython
6.py4j
Allows Python to run java commands.
7.voc
Part of BeeWare suite. Converts python code to Java bytecode.
8.p2j
Converts Python code to Java. No longer developed.

If you can get your Python code to work in Jython, then you should be able to use that to call it from Java:
http://jython.sourceforge.net/cgi-bin/faqw.py?req=show&file=faq06.001.htp

I also think that run command line in the Java wouldn't by bad practice (stackoverflow question here).
Potentially share the data through some database.
I like the way to connect two apps via the bash pipe, but I have not practice in this, so I'm wondering how difficult is to write logic to handle this on both python/java sides.
Or other productive way could be to use the Remote Procedure Call (RPC) which supports procedural programming. Using RPC you can invokes methods in shared environments. As an example you can call a function in a remote machine from the local computer using RPC. We can define RPC as a communication type in distributed systems.
(mentioned above by Marcin)
Or, very naive way would to be to communicate by the common file. But for sake of simplicity and speed, my vote is to use the shared database x rest API x socket communication.
Also I like the XML RPC as Marcin wrote.
I would like to recommend to avoid any complication to run Python under JVM or C++ binding. Better to use today trends which are obviously web technologies.
As a shared database the MongoDB may be good solution or even better the Redis as in memory database.

Use pipes to communicate with python(a python script calls python library) via subprocesses, the whole process is like java<-> Pipes <-> py.
If JNI, you should be familiar with python's bindings(not recommended) and compiled files like *.so to work with java.
The whole process is like py -> c -> .so/.pyd -> JNI -> jar.
Here is a good practice of stdio.
https://github.com/JULIELab/java-stdio-ipc

Python egg that makes Java calls to a running Java application

I am making a proposal for a Python adapter to the Oracle NoSQL Database. The Oracle NoSQL Database runs as a stand alone java application, and at least in a Java program, you interface with it by telling your program the hostname and port to connect to, and some configuration settings. Then you make java calls off of the "kvstore" object that contains that configuration.
I'd like to make a Python library that essentially exposes Python versions of the java methods Oracle NoSQL has, and converts those to Java to speak with the running Oracle NoSQL application, but I'm not sure what technologies would be best to be able to do that.
Does anyone know what technology I would want to use? I'd rather not use Jython (so the application could run in a standard Python environment) or JNI (as it seems to have some nasty caveats.)
EDIT: The only potentially technology I've found so far is: Jpype
Would it work for me?
Also, here are the ideal requirements the library would have. I would consider using Jython or JNI if one of them really did best match these requirements.
Performance. The main benefits of Oracle NoSQL are performance and scalability, so that would be the most important component for the adapter.
Easy to implement for the Python users. In order for the library to actually be used by Python programmers, it would have to relatively easy for them to use in a natural sort of way.
Reliability. It would need to be possible for it to be trustworthy and bug free, while working on the platforms you naturally expect Python to work on. (This is what made me concerned about JNI. It sounds like it is platform dependent for its implementation, and can be error-prone.)
Development speed. The last point of importance is that it be relatively fast to develop. The team of developers would enjoy learning Python or C, but we know Java better than any other programming langauge right now.

I've tried to answer each point in order, with it's own related notes.
Performance: My opinion is that JNI would be the winner here, but I could be wrong, because Jython could be JITed as well.
Easy to Implement: I am taking this as you mean easy to consume the library? That would depend entirely on how you build the API, 1 to 1 method calls, object handles, etc.
Reliability: Jython is the clear winner here, because there is no room for error when you merely instantiate POJOs/ directly access the API right in the client code.
Development Speed: JNI can be very tedious. You are essentially learning CPython modules, C, Python extensions, JNI and then referencing an existing API built in Java.
All in all, if you can make the jump, I think you'd get more benefit in the short term embedding Jython, mostly because there you can directly manipulate the API. I have personally embedded IronPython in a .NET codebase with good success. Yes you lose native speed, but the tradeoffs are hard to justify with the amount of C coding needed for a working JNI bridge. That said, you may well find projects like the one you listed (Jpype) that can do much of the legwork for you.
I would be asking what features of a native CPython runtime I need that you lose when going to Jython. Is your existing codebase in CPython heavily reliant on CPython native features?
Anyway, there's my attempt at an answer.

Calling Python in Java?

I am wondering if it is possible to call Python functions from Java code using Jython, or is it only for calling Java code from Python?

Jython: Python for the Java Platform - http://www.jython.org/index.html
You can easily call python functions from Java code with Jython. That is as long as your python code itself runs under jython, i.e. doesn't use some c-extensions that aren't supported.
If that works for you, it's certainly the simplest solution you can get. Otherwise you can use org.python.util.PythonInterpreter from the new Java6 interpreter support.
A simple example from the top of my head - but should work I hope: (no error checking done for brevity)
PythonInterpreter interpreter = new PythonInterpreter();
interpreter.exec("import sys\nsys.path.append('pathToModules if they are not there by default')\nimport yourModule");
// execute a function that takes a string and returns a string
PyObject someFunc = interpreter.get("funcName");
PyObject result = someFunc.__call__(new PyString("Test!"));
String realResult = (String) result.__tojava__(String.class);
As of 2021, Jython does not support Python 3.x

I think there are some important things to consider first with how strong you wish to have the linking between java and python.
Firstly Do you only want to call functions or do you actually want python code to change the data in your java objects? This is very important. If you only want to call some python code with or without arguments, then that is not very difficult. If your arguments are primitives it makes it even more easy. However if you want to have java class implement member functions in python, which change the data of the java object, then this is not so easy or straight forward.
Secondly are we talking cpython or will jython do? I would say cpython is where its at! I would advocate this is why python is so kool! Having such high abstractions however access to c,c++ when needed. Imagine if you could have that in java. This question is not even worth asking if jython is ok because then it is easy anyway.
So I have played with the following methods, and listed them from easy to difficult:
Java to Jython
Advantages: Trivially easy. Have actual references to java objects
Disadvantages: No CPython, Extremely Slow!
Jython from java is so easy, and if this is really enough then great. However it is very slow and no cpython! Is life worth living without cpython I don't think so! You can easily have python code implementing your member functions for you java objects.
Java to Jython to CPython via Pyro
Pyro is the remote object module for python. You have some object on a cpython interpreter, and you can send it objects which are transferred via serialization and it can also return objects via this method. Note that if you send a serialized python object from jython and then call some functions which change the data in its members, then you will not see those changes in java. You just need to remember to send back the data which you want from pyro. This I believe is the easiest way to get to cpython! You do not need any jni or jna or swig or .... You don't need to know any c, or c++. kool huh?
Advantages: Access to cpython, not as difficult as following methods
Disadvantages: Cannot change the member data of java objects directly from python. Is somewhat indirect, (jython is middle man).
Java to C/C++ via JNI/JNA/SWIG to Python via Embedded interpreter (maybe using BOOST Libraries?)
OMG this method is not for the faint of heart. And I can tell you it has taken me very long to achieve this in with a decent method. Main reason you would want to do this is so that you can run cpython code which as full rein over you java object. There are major major things to consider before deciding to try and bread java (which is like a chimp) with python (which is like a horse). Firstly if you crash the interpreter that's lights out for you program! And don't get me started on concurrency issues! In addition, there is allot allot of boiler, I believe I have found the best configuration to minimize this boiler but still it is allot! So how to go about this:
Consider that C++ is your middle man, your objects are actually c++ objects! Good that you know that now. Just write your object as if your program as in cpp not java, with the data you want to access from both worlds. Then you can use the wrapper generator called swig (http://www.swig.org/Doc1.3/Java.html) to make this accessible to java and compile a dll which you call System.load(dll name here) in java. Get this working first, then move on to the hard part!
To get to python you need to embed an interpreter. Firstly I suggest doing some hello interpreter programs or this tutorial Embedding python in C/C. Once you have that working, its time to make the horse and the monkey dance! You can send you c++ object to python via [boost][3] . I know I have not given you the fish, merely told you where to find the fish. Some pointers to note for this when compiling.
When you compile boost you will need to compile a shared library. And you need to include and link to the stuff you need from jdk, ie jawt.lib, jvm.lib, (you will also need the client jvm.dll in your path when launching the application) As well as the python27.lib or whatever and the boost_python-vc100-mt-1_55.lib.
Then include Python/include, jdk/include, boost and only use shared libraries (dlls) otherwise boost has a teary. And yeah full on I know. There are so many ways in which this can go sour. So make sure you get each thing done block by block. Then put them together.

It's not smart to have python code inside java. Wrap your python code with flask or another web framework to make it a microservice. This makes your java program able to call this microservice (e.g. via REST).
This approach is simple and it will save you tons of issues. And the codes are loosely coupled so they are scalable.
Updated on Mar 24th 2020:
According to #stx's comment, the above approach is not suitable for massive data transfer between client and server.
Here is another approach I recommended:
Connecting Python and Java with Rust(C/C++ also ok).
https://medium.com/#shmulikamar/https-medium-com-shmulikamar-connecting-python-and-java-with-rust-11c256a1dfb0

Several of the answers mention that you can use JNI or JNA to access cpython but I would not recommend starting from scratch because there are already open source libraries for accessing cpython from java. For example:
JEP
JPY

GraalVM is a good choice. I've done Java+Javascript combination with GraalVM for microservice design (Java with Javascript reflection). They recently added support for python, I'd give it a try especially with how big its community has grown over the years.
UPDATE June 2021
https://www.graalvm.org/reference-manual/python/ says
GraalVM provides a Python 3.8 compliant runtime. A primary goal of the GraalVM Python runtime is to support SciPy and its constituent libraries, as well as to work with other data science and machine learning libraries from the rich Python ecosystem. At this point, the Python runtime is made available for experimentation and curious end-users.

Here a library that lets you write your python scripts once and decide which integration method (Jython, CPython/PyPy via Jep and Py4j) to use at runtime:
https://github.com/subes/invesdwin-context-python
Since each method has its own benefits/drawbacks as explained in the link.
(3 modules: jep, Py4J, jython )

It depends on what do you mean by python functions? if they were written in cpython you can not directly call them you will have to use JNI, but if they were written in Jython you can easily call them from java, as jython ultimately generates java byte code.
Now when I say written in cpython or jython it doesn't make much sense because python is python and most code will run on both implementations unless you are using specific libraries which relies on cpython or java.
see here how to use Python interpreter in Java.

Depending on your requirements, options like XML-RPC could be useful, which can be used to remotely call functions virtually in any language supporting the protocol.

Jython has some limitations:
There are a number of differences. First, Jython programs cannot use CPython
extension modules written in C. These modules usually have files with the
extension .so, .pyd or .dll. If you want to use such a module, you should look
for an equivalent written in pure Python or Java. Although it is technically
feasible to support such extensions - IronPython does so - there are no plans
to do so in Jython.
Distributing my Python scripts as JAR files with Jython?
you can simply call python scripts (or bash or Perl scripts) from Java using Runtime or ProcessBuilder and pass output back to Java:
Running a bash shell script in java
Running Command Line in Java
java runtime.getruntime() getting output from executing a command line program

You can call any language from java using Java Native Interface

This gives a pretty good overview over the current options. Some of which are named in other answers. Jython is not usable until they decide to not implement Python 3. Many of the other projects are coming from the python side and want to access java. But there are a few options still, to name something which has not been named yet: gRPC

I have similar requirement, I think best solution is thrift for me(also a rpc solution), just run test passed successfully right now, and can use thrift-generator to gen thrift file from java interface, then gen python files and java client files from the thrift file

What kind of things can be done with Java but not Python?

I would to pick up a new programming language - Java, having been using Python for some time. But it seems most things that can be done with Java can be done with Python. So I would like to know
What kind of things can be done with Java but not Python?
mobile programming (Android).
POSIX Threads Programming.
Conversely, What kind of things can be done with Python but not Java if any?
clarification:
I hope to get an answer from a practical point of view but not a theoretical point of view and it should be about the current status, not future. So theoretically all programming languages can perform any task, practically each is limited in some way.

Both languages are Turing complete, both have vast libraries, and both support extensions written in C so that you can access low level code if needed. The main difference is where they are currently supported. Java in general has wider support than Python.
Your example of Android is one place where Java is the standard choice, although Python also has some support in the form of Android Scripting Environment. Java is already installed on most home computers. You can write Java applets and expect them to work in most browsers.
One thing you can't easily do in Java is quickly write short scripts that perform useful tasks. Python is more suitable for scripting than Java (although there are of course other alternatives too).

I guess using Jython, you can do anything with Python that you can do in Java.
Conversely, Python has the PyPy compiler, which is pretty cool - a virtual machine with multiple backeds (Java Runtime, LLVM, .net, and Python IIRC), multiple garbage collectors, multiple implementations (Stackless), etc. I know Java has a big choice of virtual machines, but the growth of PyPy is amazing, due to it being written in RPython - a fairly productive language.
Also, can a Java do this, in 1 file and less that 20 lines, with no library imports? Obviously both languages have libraries that can do this, but I'm just talking about the flexibility of the languages.
class Logger(object): # boilerplate code
def log(self,level,msg,*args,**kwargs): # *args, **kwargs = flexible arguments
self._log(level,msg,*args,**kwargs) # call with flexible argments
def _log(self,level,msg,*args,**kwargs):
# override me at runtime :)
# I think Java people call this Dependency Runtime Injection
if level>1:
print msg,args,kwargs
logger = Logger() # boilerplate code
def logged(level): # what pattern do you call this?
def logged_decorator(function): # and this?
def func(*args,**kwars):
name = func.__name__ # look ma, reflective metaprogramming!
logger.log(level,name,*args,**kwargs)
return func(*args,**kwargs)
return func # boilerplate code
return logged_decorator # boilerplate code
Example use:
#logged
def my_func(arg1,arg2):
# your code here
pass

You would surely love reading the comparisons made below between these 2 languages.
Check them :
Java is Dead ! Long live Python
Python-Java : A side-by-side comparison
Python is NOT java

What Python Can do Java Can't -
Nothing.
What Java Can do but Python Can't -
Java is a multithreading boss. So if you are trying to write a web server, where multiple requests come at the same time, java can just spawn multiple threads and CPU swaps them. That's why almost all big companies (Expedia, LinkedIn, Goldman, Amazon, Netflix, CITY, JPMC, VISA almost all) uses the JVM Web server for their main application.
CPython has something called GIL, which prevents itself from using OS-level thread efficiently. How Python Application servers (Gunicorn, Django ..) work then? Well, they fork new Processes instead of threads. Threads are lightweight, so forking new processes instead of threads still work till a threshold, but not a very scalable solution.
Sheer Execution speed - When you take var a = 1. and then do a = 10.07, you just stored float value in a variable that previously-stored an integer. When variables are assigned a new value then internally, Python creates a new object to store the value. This pointer is then assigned to the variable, this is called dynamic binding. Dynamic Binding(Python) is slower than Static Binding(Java) - as it requires object creation. And Java or C/C++ primitive types are an order of magnitude faster because of static binding.
Space used (RAM usage) - The default space python uses to store a variable is huge.
>>> import math, sys
>>> a=0
>>> sys.getsizeof(a)
24
But in Java, you can take a Byte size variable, which will take just a byte. So nobody writes an application in Python if asked to write memory-efficient software(Think about you are asked to create a DataBase like Cassandra, maybe design a compute engine like Spark).
Packaging - In Java, you can create something like a Jar. Which can run on any machine where JVM is installed. and that JAR contains all the dependencies. In python you can't just ship something like a JAR, you will have to write a script to install dependencies in every machine you want to run your code on.
Think about if Android was a Python application, before installing an App from the play store, you had to install the dependencies of that app separately.
Performance . Performance . Performance -
Java’s efficiency largely comes from its Just-In-Time (JIT) compiler and support for concurrency. The JIT compiler is a part of the Java Runtime Environment. It improves performance of Java programs by compiling bytecodes into native machine code “just in time” to run. Java Virtual Machine (JVM) calls the compiled code directly. Since the code is not interpreted, compiling does not require processor time and memory usage. Theoretically, this can make a Java program as fast as a native application.
While Java programs are compiled directly, Python is interpreted which slows down Python programs during runtime. Determining the variable type which occurs during runtime increases the workload of the interpreter. Also, remembering the object type of objects retrieved from container objects contributes to memory usage.
We can go on and on ...
People use python because python is easy to learn, easy to use. A lot of libraries for ML / DataScience etc . that saves you coding. But if you are asked to write a performant, scalable, durable, long term application, People who understand comp sci, will always choose Java or C/C++

CPython has a lot of libraries with bindings to native libraries--not so Java.

Are there tools to integrate Java and C++?

thanks for reading this question.
I am doing this homework which need a GUI as frond end to integrate with back end code which written in C++.
I wanna to write this front end GUI in java as its cross-platform feature and strong graphic components.
Is there any good way I can integrate java and C++ well?
Thank you

Swig works very well. It's a means to bind C/C++ to a huge variety of languages. I have experience of using this to talk to C++ with very little grief. Here's the manual page on using Swig and Java together. The tutorial gets you going very quickly, with many examples including Java.
I would however investigate splitting your application into a client/server architecture, to separate the C++ backend from the Java front end. You'll avoid the C++/Java development and integration pain = although you'll have to implement some communication protocol between the front and back end depending on requirements (e.g. basic sockets / webservice / HTTP+REST or possibly CORBA - which comes natively to Java and is designed for cross-language communication).

Assuming you are the back end component is on the same machine you could use an interface layer as described by others
JNI
JNA
Swig
QTJambi
These all require you c++ backend to be available in a dll and usually provide Java proxies for C functions and sometimes c++ classes. There is a learning curve to all of these and some work to enable the Proxy.
Another approach would be to use a c++ process and communicate with this using either
command line
stdin/stdout
If you want to support communication across a network
sockets
CORBA
WebServices
Thrift
These also have a learning curve and some set up costs
Of these the command-line or stdin/stdout is probably the fastest to get working with the minimal amount of effort and knowledge. However it does not scale well to large interfaces as you must encode the input and output of each message as text
For the command-line approach you execute the c++ process using command line switches for the options, the results are either read from the processes standard out or its exit code.
For stdin/stdout you start the process each request is sent to stdin of the process and the results are read from stdout.

Hava a look at JNI (Java Native Interface). Sun has an online book on JNI.

How about Thrift?
Thrift is a software framework for scalable cross-language services development. It combines a software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml.

If you are not writing the C++ backend library yourself, but just want to use a third-party library, the better alternative would be to use JNA.
The main benefit of using JNA over JNI in this case is that the bridging code is all written in Java (rather than in the native language, C++ in your case). This means you wouldn't need to complicate your build process by building C++ JNI interfaces, all your interface work would be written in the language of the main project.
If however, you are writing the C++ backend yourself, then any of the other options already provided would be equally applicable.