We Keep Coding

Py4J has bigger overhead than Jython and JPype

After searching for an option to run Java code from Django application(python), I found out that Py4J is the best option for me. I tried Jython, JPype and Python subprocess and each of them have certain limitations:
Jython. My app runs in python.
JPype is buggy. You can start JVM just once after that it fails to start again.
Python subprocess. Cannot pass Java object between Python and Java, because of regular console call.
On Py4J web site is written:
In terms of performance, Py4J has a bigger overhead than both of the previous solutions (Jython and JPype) because it relies on sockets, but if performance is critical to your application, accessing Java objects from Python programs might not be the best idea.
In my application performance is critical, because I'm working with Machine learning framework Mahout. My question is: Will Mahout also run slower because of Py4J gateway server or this overhead just mean that invoking Java methods from Python functions is slower (in latter case performance of Mahout will not be a problem and I can use Py4J).

JPype issue that #HIP_HOP mentioned with JVM getting detached from new threads can be overcome with the following hack (add it before the first call to Java objects in the new thread which does not have JVM yet):
# ensure that current thread is attached to JVM
# (essential to prevent JVM / entire container crashes
# due to "JPJavaEnv::FindClass" errors)
if not jpype.isThreadAttachedToJVM():
jpype.attachThreadToJVM()

PySpark uses Py4J quite successfully. If all the heavylifting is done on Spark (or Mahout in your case) itself, and you just want to return result back to "driver"/Python code, then Py4J might work for you very well as well.
Py4j has slightly bigger overhead for huge results (that's not necessarily the case for Spark workloads, as you only return summaries /aggregates for the dataframes). There is an improvement discussion for py4j to switch to binary serialization to remove that overhead for higher badnwidth requirements too: https://github.com/bartdag/py4j/issues/159

My solutions
java thread/process <-> Pipes <-> py subprocess
Use pipes by java's ProcessBuilder to call py with args "-u" to transfer data via pipes.
Here is a good practice.
https://github.com/JULIELab/java-stdio-ipc
Here is my stupid research result about "java <-> py"
[Jython] Java implement of python.
[Jpype] JPype is designed to allow the user to exercise Java as fluidly as possible from within Python. We can break this down into a few specific design goals.
Unlike Jython, JPype does not achieve this by re-implementing Python, but instead by interfacing both virtual machines at the native level. This shared memory based approach achieves good computing performance while providing the access to the entirety of CPython and Java libraries.
[Runtime] The Runtime class in java (old method).
[Process] Java ProcessBuilder class gives more structure to the arguments.
[Pipes] Named pipes could be the answer for you. Use subprocess. Popen to start the Java process and establish pipes to communicate with it.
Try mkfifo() implementation in python.
https://jj09.net/interprocess-communication-python-java/
-> java<-> Pipes <-> py https://github.com/JULIELab/java-stdio-ipc
[Protobuf] This is the opensource solution Google uses to do IPC between Java and Python. For serializing and deserializing data efficiently in a language-neutral, platform-neutral, extensible way, take a look at Protocol Buffers.
[Socket] CS-arch throgh socket
Server(Python) - Client(Java) communication using sockets
https://jj09.net/interprocess-communication-python-java/
Send File From Python Server to Java Client
[procbridge]
A super-lightweight IPC (Inter-Process Communication) protocol over TCP socket. https://github.com/gongzhang/procbridge
https://github.com/gongzhang/procbridge-python
https://github.com/gongzhang/procbridge-java
[hessian binary web service protocol] using python client and java server.
[Jython]
Jython is a reimplementation of Python in Java. As a result it has much lower costs to share data structures between Java and Python and potentially much higher level of integration. Noted downsides of Jython are that it has lagged well behind the state of the art in Python; it has a limited selection of modules that can be used; and the Python object thrashing is not particularly well fit in Java virtual machine leading to some known performance issues.
[Py4J]
Py4J uses a remote tunnel to operate the JVM. This has the advantage that the remote JVM does not share the same memory space and multiple JVMs can be controlled. It provides a fairly general API, but the overall integration to Python is as one would expect when operating a remote channel operating more like an RPC front-end. It seems well documented and capable. Although I haven’t done benchmarking, a remote access JVM will have a transfer penalty when moving data.
[Jep]
Jep stands for Java embedded Python. It is a mirror image of JPype. Rather that focusing on accessing Java from within Python, this project is geared towards allowing Java to access Python as a sub-interpreter. The syntax for accessing Java resources from within the embedded Python is quite similar to support for imports. Notable downsides are that although Python supports multiple interpreters many Python modules do not, thus some of the advantages of the use of Python may be hard to realize. In addition, the documentation is a bit underwhelming thus it is difficult to see how capable it is from the limited examples.
[PyJnius]
PyJnius is another Python to Java only bridge. Syntax is somewhat similar to JPype in that classes can be loaded in and then have mostly Java native syntax. Like JPype, it provides an ability to customize Java classes so that they appear more like native classes. PyJnius seems to be focused on Android. It is written using Cython .pxi files for speed. It does not include a method to represent primitive arrays, thus Python list must be converted whenever an array needs to be passed as an argument or a return. This seems pretty prohibitive for scientific code. PyJnius appears is still in active development.
[Javabridge]
Javabridge is direct low level JNI control from Python. The integration level is quite low on this, but it does serve the purpose of providing the JNI API to Python rather than attempting to wrap Java in a Python skin. The downside being of course you would really have to know a lot of JNI to make effective use of it.
[jpy]
This is the most similar package to JPype in terms of project goals. They have achieved more capabilities in terms of a Java from Python than JPype which does not support any reverse capabilities. It is currently unclear if this project is still active as the most recent release is dated 2014. The integration level with Python is fairly low currently though what they do provide is a similar API to JPype.
[JCC]
JCC is a C++ code generator that produces a C++ object interface wrapping a Java library via Java’s Native Interface (JNI). JCC also generates C++ wrappers that conform to Python’s C type system making the instances of Java classes directly available to a Python interpreter. This may be handy if your goal is not to make use of all of Java but rather have a specific library exposed to Python.
[VOC] https://beeware.org/project/projects/bridges/voc/_
A transpiler that converts Python bytecode into Java bytecode part of the BeeWare project. This may be useful if getting a smallish piece of Python code hooked into Java. It currently list itself as early development. This is more in the reverse direction as its goals are making Python code available in Java rather providing interaction between the two.
[p2j]
This lists itself as “A (restricted) python to java source translator”. Appears to try to convert Python code into Java. Has not been actively maintained since 2013. Like VOC this is primilarly for code translation rather that bridging.
[GraalVM]
Source: https://github.com/oracle/graal

I don't know Mahout. But think about that: At least with JPype and Py4J you will have performance impact when converting types from Java to Python and vice versa. Try to minimize calls between the languages. Maybe it's an alternative for you to code a thin wrapper in Java that condenses many Javacalls to one python2java call.

Because the performance is also a question about your usage screnario (how often you call the script and how large is the data that is moved) and because the different solutions have their own specific benefits/drawbacks, I have created an API to switch between different implementations without you having to change your python script: https://github.com/subes/invesdwin-context-python
Thus testing what works best or just being flexible about what to deploy to is really easy.

Calling Java methods and classes from Perl script

I have a quite huge Java class that has several imported packages and libraries (related to natural language processing). I want to call some specific methods of my Java classes and get results back using a Perl script. How should I do this? I guess the Inline::Java is not suitable for my purpose since the Java code is quite large to be incorporated at one place.

Why is Inline::Java not suitable? You don't need to provide access to all the bits of your Java API. You can write short methods that call into your Java stuff. A small Java adapter layer can show up in your Perl code to give you the access you need. Have you tried it yet?
How big is this Java code base? I've been on projects that easily integrated big Java SDKs (although I wasn't that one doing that part).

I had ran into similar situation lately. The best solution which worked for me was to use apache thrift service and expose the required methods through it. These method were then in turn consumed by the client written in perl.

Calling Python in Java?

I am wondering if it is possible to call Python functions from Java code using Jython, or is it only for calling Java code from Python?

Jython: Python for the Java Platform - http://www.jython.org/index.html
You can easily call python functions from Java code with Jython. That is as long as your python code itself runs under jython, i.e. doesn't use some c-extensions that aren't supported.
If that works for you, it's certainly the simplest solution you can get. Otherwise you can use org.python.util.PythonInterpreter from the new Java6 interpreter support.
A simple example from the top of my head - but should work I hope: (no error checking done for brevity)
PythonInterpreter interpreter = new PythonInterpreter();
interpreter.exec("import sys\nsys.path.append('pathToModules if they are not there by default')\nimport yourModule");
// execute a function that takes a string and returns a string
PyObject someFunc = interpreter.get("funcName");
PyObject result = someFunc.__call__(new PyString("Test!"));
String realResult = (String) result.__tojava__(String.class);
As of 2021, Jython does not support Python 3.x

I think there are some important things to consider first with how strong you wish to have the linking between java and python.
Firstly Do you only want to call functions or do you actually want python code to change the data in your java objects? This is very important. If you only want to call some python code with or without arguments, then that is not very difficult. If your arguments are primitives it makes it even more easy. However if you want to have java class implement member functions in python, which change the data of the java object, then this is not so easy or straight forward.
Secondly are we talking cpython or will jython do? I would say cpython is where its at! I would advocate this is why python is so kool! Having such high abstractions however access to c,c++ when needed. Imagine if you could have that in java. This question is not even worth asking if jython is ok because then it is easy anyway.
So I have played with the following methods, and listed them from easy to difficult:
Java to Jython
Advantages: Trivially easy. Have actual references to java objects
Disadvantages: No CPython, Extremely Slow!
Jython from java is so easy, and if this is really enough then great. However it is very slow and no cpython! Is life worth living without cpython I don't think so! You can easily have python code implementing your member functions for you java objects.
Java to Jython to CPython via Pyro
Pyro is the remote object module for python. You have some object on a cpython interpreter, and you can send it objects which are transferred via serialization and it can also return objects via this method. Note that if you send a serialized python object from jython and then call some functions which change the data in its members, then you will not see those changes in java. You just need to remember to send back the data which you want from pyro. This I believe is the easiest way to get to cpython! You do not need any jni or jna or swig or .... You don't need to know any c, or c++. kool huh?
Advantages: Access to cpython, not as difficult as following methods
Disadvantages: Cannot change the member data of java objects directly from python. Is somewhat indirect, (jython is middle man).
Java to C/C++ via JNI/JNA/SWIG to Python via Embedded interpreter (maybe using BOOST Libraries?)
OMG this method is not for the faint of heart. And I can tell you it has taken me very long to achieve this in with a decent method. Main reason you would want to do this is so that you can run cpython code which as full rein over you java object. There are major major things to consider before deciding to try and bread java (which is like a chimp) with python (which is like a horse). Firstly if you crash the interpreter that's lights out for you program! And don't get me started on concurrency issues! In addition, there is allot allot of boiler, I believe I have found the best configuration to minimize this boiler but still it is allot! So how to go about this:
Consider that C++ is your middle man, your objects are actually c++ objects! Good that you know that now. Just write your object as if your program as in cpp not java, with the data you want to access from both worlds. Then you can use the wrapper generator called swig (http://www.swig.org/Doc1.3/Java.html) to make this accessible to java and compile a dll which you call System.load(dll name here) in java. Get this working first, then move on to the hard part!
To get to python you need to embed an interpreter. Firstly I suggest doing some hello interpreter programs or this tutorial Embedding python in C/C. Once you have that working, its time to make the horse and the monkey dance! You can send you c++ object to python via [boost][3] . I know I have not given you the fish, merely told you where to find the fish. Some pointers to note for this when compiling.
When you compile boost you will need to compile a shared library. And you need to include and link to the stuff you need from jdk, ie jawt.lib, jvm.lib, (you will also need the client jvm.dll in your path when launching the application) As well as the python27.lib or whatever and the boost_python-vc100-mt-1_55.lib.
Then include Python/include, jdk/include, boost and only use shared libraries (dlls) otherwise boost has a teary. And yeah full on I know. There are so many ways in which this can go sour. So make sure you get each thing done block by block. Then put them together.

It's not smart to have python code inside java. Wrap your python code with flask or another web framework to make it a microservice. This makes your java program able to call this microservice (e.g. via REST).
This approach is simple and it will save you tons of issues. And the codes are loosely coupled so they are scalable.
Updated on Mar 24th 2020:
According to #stx's comment, the above approach is not suitable for massive data transfer between client and server.
Here is another approach I recommended:
Connecting Python and Java with Rust(C/C++ also ok).
https://medium.com/#shmulikamar/https-medium-com-shmulikamar-connecting-python-and-java-with-rust-11c256a1dfb0

Several of the answers mention that you can use JNI or JNA to access cpython but I would not recommend starting from scratch because there are already open source libraries for accessing cpython from java. For example:
JEP
JPY

GraalVM is a good choice. I've done Java+Javascript combination with GraalVM for microservice design (Java with Javascript reflection). They recently added support for python, I'd give it a try especially with how big its community has grown over the years.
UPDATE June 2021
https://www.graalvm.org/reference-manual/python/ says
GraalVM provides a Python 3.8 compliant runtime. A primary goal of the GraalVM Python runtime is to support SciPy and its constituent libraries, as well as to work with other data science and machine learning libraries from the rich Python ecosystem. At this point, the Python runtime is made available for experimentation and curious end-users.

Here a library that lets you write your python scripts once and decide which integration method (Jython, CPython/PyPy via Jep and Py4j) to use at runtime:
https://github.com/subes/invesdwin-context-python
Since each method has its own benefits/drawbacks as explained in the link.
(3 modules: jep, Py4J, jython )

It depends on what do you mean by python functions? if they were written in cpython you can not directly call them you will have to use JNI, but if they were written in Jython you can easily call them from java, as jython ultimately generates java byte code.
Now when I say written in cpython or jython it doesn't make much sense because python is python and most code will run on both implementations unless you are using specific libraries which relies on cpython or java.
see here how to use Python interpreter in Java.

Depending on your requirements, options like XML-RPC could be useful, which can be used to remotely call functions virtually in any language supporting the protocol.

Jython has some limitations:
There are a number of differences. First, Jython programs cannot use CPython
extension modules written in C. These modules usually have files with the
extension .so, .pyd or .dll. If you want to use such a module, you should look
for an equivalent written in pure Python or Java. Although it is technically
feasible to support such extensions - IronPython does so - there are no plans
to do so in Jython.
Distributing my Python scripts as JAR files with Jython?
you can simply call python scripts (or bash or Perl scripts) from Java using Runtime or ProcessBuilder and pass output back to Java:
Running a bash shell script in java
Running Command Line in Java
java runtime.getruntime() getting output from executing a command line program

You can call any language from java using Java Native Interface

This gives a pretty good overview over the current options. Some of which are named in other answers. Jython is not usable until they decide to not implement Python 3. Many of the other projects are coming from the python side and want to access java. But there are a few options still, to name something which has not been named yet: gRPC

I have similar requirement, I think best solution is thrift for me(also a rpc solution), just run test passed successfully right now, and can use thrift-generator to gen thrift file from java interface, then gen python files and java client files from the thrift file

Are there tools to integrate Java and C++?

thanks for reading this question.
I am doing this homework which need a GUI as frond end to integrate with back end code which written in C++.
I wanna to write this front end GUI in java as its cross-platform feature and strong graphic components.
Is there any good way I can integrate java and C++ well?
Thank you

Swig works very well. It's a means to bind C/C++ to a huge variety of languages. I have experience of using this to talk to C++ with very little grief. Here's the manual page on using Swig and Java together. The tutorial gets you going very quickly, with many examples including Java.
I would however investigate splitting your application into a client/server architecture, to separate the C++ backend from the Java front end. You'll avoid the C++/Java development and integration pain = although you'll have to implement some communication protocol between the front and back end depending on requirements (e.g. basic sockets / webservice / HTTP+REST or possibly CORBA - which comes natively to Java and is designed for cross-language communication).

Assuming you are the back end component is on the same machine you could use an interface layer as described by others
JNI
JNA
Swig
QTJambi
These all require you c++ backend to be available in a dll and usually provide Java proxies for C functions and sometimes c++ classes. There is a learning curve to all of these and some work to enable the Proxy.
Another approach would be to use a c++ process and communicate with this using either
command line
stdin/stdout
If you want to support communication across a network
sockets
CORBA
WebServices
Thrift
These also have a learning curve and some set up costs
Of these the command-line or stdin/stdout is probably the fastest to get working with the minimal amount of effort and knowledge. However it does not scale well to large interfaces as you must encode the input and output of each message as text
For the command-line approach you execute the c++ process using command line switches for the options, the results are either read from the processes standard out or its exit code.
For stdin/stdout you start the process each request is sent to stdin of the process and the results are read from stdout.

Hava a look at JNI (Java Native Interface). Sun has an online book on JNI.

How about Thrift?
Thrift is a software framework for scalable cross-language services development. It combines a software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml.

If you are not writing the C++ backend library yourself, but just want to use a third-party library, the better alternative would be to use JNA.
The main benefit of using JNA over JNI in this case is that the bridging code is all written in Java (rather than in the native language, C++ in your case). This means you wouldn't need to complicate your build process by building C++ JNI interfaces, all your interface work would be written in the language of the main project.
If however, you are writing the C++ backend yourself, then any of the other options already provided would be equally applicable.

Interfacing AutoCAD with JAVA

Any idea on interfacing with AutoCAD through a JAVA program. I am looking for a solution that can directly interface with an AutoCAD session (even start one), the way it works with the .NET extensions of AutoCAD. Or any way to work with ObjectARX through JAVA, definitely not writing a complete JNI wrapper over it.
Added: We are looking for something in open-source. In case there is none, we are ready to create one in open-source if someone can lend a helping hand explaining how it can be done. We have taken a look at JNA and JaWin. JNA clubbed with JNI might help but is too ugly. JaWin on the other hand has not been updated in the past 4 years, so sort of skeptical using it.

The only thing I can think of is to use a Java <-> .Net bridge like JNBridge. I've used this in other scenarios and it works fine. Never done any work with Java and AutoCad though so there might be other cheaper solutions.

If you are trying to interact with the AutoCAD application you will have a tough time.
If you want to interact with the dwg files themselves there is the Open Design Alliance which has libraries that allow working with dwg files without AutoCAD.

You can try to use the JavaBeans ActiveX bridge and the COM Automation to open AutoCAD and manipulate it.
That said, JavaBeans ActiveX bridge hasn't evolved in a while (I used it back in 2003) and Autodesk is seriously investing in .Net for everything related to extensions and automation in AutoCAD.

We are working on a similar project. What we are doing is writing the integration code in C#, and the business logic for our project in Java.
We are utilizing a C++ bridge that utilizes JNI to take Java calls down to C++, and then translates them back up to C#, and vice versa. Each function that needs to go from C# to Java has to be implemented in the bridge code. This is fairly involved, and there are some issues with getting the unmanaged C++ code to work with the C#, as well as the standard overhead of translating Jstrings into C# Strings and the like. The result is relatively fast, after we did some optimizations to insure that we aren't starting and stopping the JVM for each call into the Java layer.
We previously used COM objects which were called by the AutoCAD plugin, so that would be another approach.