I am prototyping an interface to our application to allow other people to use python, our application is written in java. I would like to pass some of our data from the java app to the python code but I am unsure how to pass an object to python. I have done a simple java->python function call using simple parameters using Jython and found it very useful for what I am trying to do. Given the class below, how can I then use it in Python/Jython as an input to a function/class:
public class TestObject
{
private double[] values;
private int length;
private int anotherVariable;
//getters, setters
}
One solution. You could use some sort of message system, queue, or broker of some sort to serialize/deserialize, or pass messages between python and java. Then create some sort workers/producer/consumers to put work on the queues to be processed in python, or java.
Also consider checking out for inspiration: https://www.py4j.org/
py4j is used heavily by/for pyspark and hadoop type stuff.
To answer your question more immediately.
Example using json-simple.:
import org.apache.commons.io.FileUtils;
import org.json.simple.JSONObject;
//import org.json.simple.JSONObject;
public class TestObject
{
private double[] values;
private int length;
private int anotherVariable;
private boolean someBool;
private String someString;
//getters, setters
public String toJSON() {
JSONObject obj=new JSONObject();
obj.put("values",new Double(this.values));
obj.put("length",new Integer(this.length));
obj.put("bool_val",new Boolean(this.SomeBool));
obj.put("string_key",this.someString);
StringWriter out = new StringWriter();
obj.writeJSONString(out);
return out.toString();
}
public void writeObject(){
Writer writer = new BufferedWriter(
new OutputStreamWriter(
new FileOutputStream("anObject.json"), "utf-8")
)
)
writer.write(this.toJSON());
}
public static void setObject(){
values = 100.134;
length = 12;
anotherVariable = 15;
someString = "spam";
}
}
And in python:
class DoStuffWithObject(object):
def __init__(self,obj):
self.obj = obj
self.changeObj()
self.writeObj()
def changeObj(self):
self.obj['values'] = 100.134;
self.obj['length'] = 12;
self.obj['anotherVariable'] = 15;
self.obj['someString'] = "spam";
def writeObj(self):
''' write back to file '''
with open('anObject.json', 'w') as f:
json.dump(self.obj, f)
def someOtherMethod(self, s):
''' do something else '''
print('hello {}'.format(s))
import json
with open('anObject.json','r') as f:
obj = json.loads(f.read())
# print out obj['values'] obj['someBool'] ...
for key in obj:
print(key, obj[key])
aThing = DoStuffWithObject(obj)
aThing.someOtherMethod('there')
And then in java read back the object. There are solutions that exist implementing this idea (JSON-RPC, XML-RPC, and variants). Depending, you may may also want to consider using something like ( http://docs.mongodb.org/ecosystem/drivers/java/ ) the benefit being that mongo does json.
See:
https://spring.io/guides/gs/messaging-reactor/
http://spring.io/guides/gs/messaging-rabbitmq/
http://spring.io/guides/gs/scheduling-tasks/
Celery like Java projects
Jedis
RabbitMQ
ZeroMQ
A more comprehensive list of queues:
http://queues.io/
Resources referenced:
http://www.oracle.com/technetwork/articles/java/json-1973242.html
How do I create a file and write to it in Java?
https://code.google.com/p/json-simple/wiki/EncodingExamples
Agree with the answer below. I think that the bottom line is that "Python and Java are separate interpreter-environments." You therefore shouldn't expect to transfer "an object" from one to the other. You shouldn't expect to "call methods." But it is reasonable to pass data from one to another, by serializing and de-serializing it through some intermediate data format (e.g. JSON) as you would do with any other program.
In some environments, such as Microsoft Windows, it's possible that a technology like OLE (dot-Net) might be usable to allow environments to be linked-together "actively," where the various systems implement and provide OLE-objects. But I don't have any personal experience with whether, nor how, this might be done.
Therefore, the safest thing to do is to treat them as "records," and to use serialization techniques on both sides. (Or, if you got very adventurous, run (say) Java in a child-thread.) An "adventurous" design could get out-of-hand very quickly, with little return on investment.
You need to make the python file to exe using py2exe , Refer the link : https://www.youtube.com/watch?v=kyoGfnLm4LA. Then use the program in java and pass arguements:
Please refer this link it will be having the details:
Calling fortran90 exe program from java is not executing
Related
In my end product, I provide the ability to extend the application code at runtime using small Groovy scripts that are edited via a form and whose code are persisted in the SQL database.
The scheme that these "custom code" snippets follow is generally returning a value based on input parameters. For example, during the invoicing of a service, the rating system might use a published schedule of predetermined rates, or values defined in a contract in the application, through custom groovy code, if an "overridden" value is returned, then it should be used.
In the logic that would determine the "override" value of the rate, I've incorporated something like these groovy code snippets that return a value, or if they return null, then the default value is used. E.g.
class GroovyRunner {
static final GroovyClassLoader classLoader = new GroovyClassLoader()
static final String GROOVY_CODE = MyDatabase().loadCustomCode()
static final String GROOVY_CLASS = MyDatabase().loadCustomClassName()
static final String TEMPDIR = System.getProperty("java.io.tmpdir")
double getOverrideRate(Object inParameters) {
def file = new File(TEMPDIR+GROOVY_CLASS+".groovy")
BufferedWriter bw = new BufferedWriter(new FileWriter(file))
bw.write(GROOVY_CODE)
bw.close()
Class gvy = classLoader.parseClass(file)
GroovyObject obj = (GroovyObject) gvy.getDeclaredConstructor().newInstance()
return Double.valueOf(obj.invokeMethod("getRate",inParameters)
}
}
And then, in the user-created custom groovy code:
class RateInterceptor {
def getRate(Object inParameters) {
def businessEntity = (SomeClass) inParameters
return businessEntity.getDiscount() == .5 ? .5 : null
}
}
The problem with this is that these "custom code" bits in GROOVY_CODE above, are pulled from a database during runtime, and contain an intricate groovy class. Since this method will be called numerous times in succession, it is impractical to create a new File object each time it is run.
Whether I use GroovyScriptEngine, or the GroovyClassLoader, these both involve the need of a java.io.File object. This makes the code execute extremely slowly, as the File will have to be created after the custom groovy code is retrieved from the database. Is there any way to run groovy code that can return a value without creating a temporary file to execute it?
The straight solution for your case would be using GroovyClassLoader.parseClass(String text)
http://docs.groovy-lang.org/latest/html/api/groovy/lang/GroovyClassLoader.html#parseClass(java.lang.String)
The class caching should not be a problem because you are creating each time a new GroovyClassLoader
However think about using groovy scripts instead of classes
your rate interceptor code could be like this:
def businessEntity = (SomeClass) context
return businessEntity.getDiscount() == .5 ? .5 : null
or even like this:
context.getDiscount() == .5 ? .5 : null
in script you could declare functions, inner classes, etc
so, if you need the following script will work also:
class RateInterceptor {
def getRate(SomeClass businessEntity) {
return businessEntity.getDiscount() == .5 ? .5 : null
}
}
return new RateInterceptor().getRate(context)
The java code to execute those kind of scripts:
import groovy.lang.*;
...
GroovyShell gs = new GroovyShell();
Script script = gs.parse(GROOVY_CODE);
// bind variables
Binding binding = new Binding();
binding.setVariable("context", inParams);
script.setBinding(binding);
// run script
Object ret = script.run();
Note that parsing of groovy code (class or script) is a heavy operation. And if you need to speedup your code think about caching of parsed class into some in-memory cache or even into a map
Map<String, Class<groovy.lang.Script>>
Straight-forward would be also:
GroovyShell groovyShell = new GroovyShell()
Closure groovy = { String name, String code ->
String script = "{ Map params -> $code }"
groovyShell.evaluate( script, name ) as Closure
}
def closure = groovy( 'SomeName', 'params.someVal.toFloat() * 2' )
def res = closure someVal:21
assert 42.0f == res
My use case:
I am trying to serve the models trained by python within our existing JVM service with libtensorflow_jni.
Now I am able to load the model by using SavedModelBundle.load(). But I find hard to feed the request into the model. As my user request is not simply a scalar matrix, but a map of features, like:
{'gender':1, 'age': 20, 'country': 100, other features ...}
By searching around the tensor flow tutorials, I see that Example protocol buffers may fit here as it basically holds a list of features. But I am not sure how to convert it into a Java Tensor object.
If I create a Tensor directly with serialized Example object, TensorFlow runtime seems not happy with datatype. For example, it I do following,
Tensor inputTensor = Tensor.create(example.toByteArray());
s.runner().feed(inputTensorName, inputTensor).fetch(outputTensorName).run().get(0);
I will get an IllegalArgumentException:
java.lang.IllegalArgumentException: Expected serialized to be a vector, got shape: []
Could you guys shed some light how I can move forward from here in case you happens to know or have same use cases?
Thanks!
Looking at your error message, it appears that the problem is that your model is expecting a vector of string tensors (most likely corresponding to a batch of serialized Example protocol buffer messages, probably from tf.parse_example) but you're feeding it a scalar string tensor.
Unfortunately, till issue #8531 is resolved, the Java API doesn't have a way to create a Tensor of strings except for scalars. Once that issue is resolved, things will be easier.
In the mean time, you could work around this by constructing a TensorFlow "model" to convert your scalar string into a vector of size 1 :). That could be done with something like this:
// A TensorFlow "model" that reshapes a string scalar into a vector.
// Should be much prettier once https://github.com/tensorflow/tensorflow/issues/7149
// is resolved.
private static class Reshaper implements AutoCloseable {
Reshaper() {
this.graph = new Graph();
this.session = new Session(graph);
this.in =
this.graph.opBuilder("Placeholder", "in")
.setAttr("dtype", DataType.STRING)
.build()
.output(0);
try (Tensor shape = Tensor.create(new int[] {1})) {
Output vectorShape =
this.graph.opBuilder("Const", "vector_shape")
.setAttr("dtype", shape.dataType())
.setAttr("value", shape)
.build()
.output(0);
this.out =
this.graph.opBuilder("Reshape", "out").addInput(in).addInput(vectorShape).build().output(0);
}
}
#Override
public void close() {
this.session.close();
this.graph.close();
}
public Tensor vector(Tensor input) {
return this.session.runner().feed(this.in, input).fetch(this.out).run().get(0);
}
private final Graph graph;
private final Session session;
private final Output in;
private final Output out;
}
With the above, you can convert your example proto tensor to a vector and feed it into the model you're interested in with something like this:
Tensor inputTensor = null;
try (Tensor scalar = Tensor.create(example.toByteArray())) {
inputTensor = reshaper.vector(scalar);
}
s.runner().feed(inputTensorName, inputTensor).fetch(outputTensorName).run().get(0);
For full details, see this example on github
Hope that helps!
I have a CloudQueueMessage and I want to insert "Hello World" - so I write:
CloudQueueMessage message = new CloudQueueMessage("Hello World");
it’s perfect.
My problem is that I want to put an object in place of "Hello World" like this:
CloudQueueMessage message = new CloudQueueMessage(new MyObject(parameter1, parameter2, ...));
But the compiler will not let me because it tells me:
The constructor CloudQueueMessage (MyObject) is undefined.
How can I achieve what I want?
CloudQueueMessage may be constructed with either binary data or a string, not an object to be serialized (though you may serialize it to a string yourself, then store it as a message). That's why you're getting an error: there's no constructor that takes a MyObject as its parameter.
For more details on the creation of a CloudQueueMessage, see this msdn page.
You may check out this question.
It outlines a technique for having "Strongly typed Queues" using C#. However the approach is really trivial, and I am sure you can achieve a similar behavior in Java. As long as you can serialize your object to a binary array.
This is a question that might help you, if you need to. And this also.
Here is the code that I utilize for serializing objects to strings and back for when dealing with Azure Queues. Just make sure that your objects are small, so that they fit within the 64k requirement
protected CloudQueueMessage CreateCloudQueueMessage(TQueueableEntity message)
{
using (var stringWriter = new StringWriter())
{
var xmlSerializer = new XmlSerializer(typeof(TQueueableEntity));
xmlSerializer.Serialize(stringWriter, message);
return new CloudQueueMessage(stringWriter.ToString());
}
}
protected TQueueableEntity CreateObjectFromMessage(CloudQueueMessage message)
{
var xmlSerializer = new XmlSerializer(typeof(TQueueableEntity));
//Deserialize message
using (var reader = new StringReader(message.AsString))
{
var result = (TQueueableEntity)xmlSerializer.Deserialize(reader);
reader.Close();
return result;
}
}
I have a problem to solve using FSTs.
Basically, I'll make a morphological parser, and in this moment i have to work with large transducers. The performance is The Big issue here.
Recently, i worked in c++ in other projects where the performance matters, but now, i'am considering java, because the java's benefits and because java is getting better.
I studied some comparisons between java and c++, but I cannot decide what language i should use for this specific problem because it depends on lib in use.
I can´t find much information about java's libs, so, my question is: Are there any open source java libs in which the performance is good, like The RWTH FSA Toolkit that i read in an article that is the fastest c++ lib?
Thanks all.
What are the "benefits" of Java, for your purposes? What specific problem does that platform solve that you need? What is the performance constraint you must consider? Were the "comparisons" fair, because Java is actually extremely difficult to benchmark. So is C++, but you can at least get some algorithmic boundary guarantees from STL.
I suggest you look at OpenFst and the AT&T finite-state transducer tools. There are others out there, but I think your worry about Java puts the cart before the horse-- focus on what solves your problem well.
Good luck!
http://jautomata.sourceforge.net/ and http://www.cs.duke.edu/csed/jflap/ are based Java finite state machine libraries, although I don't have experience using them so I cannot comment on the efficiency.
I'm one of the developers of the morfologik-stemming library. It's pure Java and its performance is very good, both when you build the automaton and when you use it. We use it for morphological analysis in LanguageTool.
The problem here is the minimum size of your objects in Java. In C++, without virtual methods and run time type identification, your objects weight exactly their content. And the time your automata take to manipulate memory has a big impact on performance.
I think that should be the main reason for choosing C++ over Java.
OpenFST is a C++ finite state transducer framework that is really comprehensive. Some people from CMU ported it to Java for use in their natural language processing.
A blog post series describing it.
The code is located on svn.
Update:
I ported it to java here
Lucene has a excellent implementation of FST, which is easy to use and high performance, making query engines like Elasticsearch, Solr deliver very fast sub-second term based query.Let me take an example:
import com.google.common.base.Preconditions;
import org.apache.lucene.store.ByteArrayDataInput;
import org.apache.lucene.store.DataInput;
import org.apache.lucene.store.GrowableByteArrayDataOutput;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.IntsRefBuilder;
import org.apache.lucene.util.fst.Builder;
import org.apache.lucene.util.fst.FST;
import org.apache.lucene.util.fst.PositiveIntOutputs;
import org.apache.lucene.util.fst.Util;
import java.io.IOException;
public class T {
private final String inputValues[] = {"cat", "dog", "dogs"};
private final long outputValues[] = {5, 7, 12};
// https://lucene.apache.org/core/8_4_0/core/org/apache/lucene/util/fst/package-summary.html
public static void main(String[] args) throws IOException {
T t = new T();
FST<Long> fst = t.buildFSTInMemory();
System.out.println(String.format("memory used for fst is %d bytes", fst.ramBytesUsed()));
t.searchFST(fst);
byte[] bytes = t.serialize(fst);
System.out.println(String.format("length of serialized fst is %d bytes", bytes.length));
fst = t.deserialize(bytes);
t.searchFST(fst);
}
private FST<Long> buildFSTInMemory() throws IOException {
// Input values (keys). These must be provided to Builder in Unicode sorted order! Use Collections.sort() to sort inputValues first.
PositiveIntOutputs outputs = PositiveIntOutputs.getSingleton();
Builder<Long> builder = new Builder<Long>(FST.INPUT_TYPE.BYTE1, outputs);
BytesRef scratchBytes = new BytesRef();
IntsRefBuilder scratchInts = new IntsRefBuilder();
for (int i = 0; i < inputValues.length; i++) {
// scratchBytes.copyChars(inputValues[i]);
scratchBytes.bytes = inputValues[i].getBytes();
scratchBytes.offset = 0;
scratchBytes.length = inputValues[i].length();
builder.add(Util.toIntsRef(scratchBytes, scratchInts), outputValues[i]);
}
FST<Long> fst = builder.finish();
return fst;
}
private FST<Long> deserialize(byte[] bytes) throws IOException {
DataInput in = new ByteArrayDataInput(bytes);
PositiveIntOutputs outputs = PositiveIntOutputs.getSingleton();
FST<Long> fst = new FST<Long>(in, outputs);
return fst;
}
private byte[] serialize(FST<Long> fst) throws IOException {
final int capicity = 32;
GrowableByteArrayDataOutput out = new GrowableByteArrayDataOutput(capicity);
fst.save(out);
return out.getBytes();
}
private void searchFST(FST<Long> fst) throws IOException {
for (int i = 0; i < inputValues.length; i++) {
Long value = Util.get(fst, new BytesRef(inputValues[i]));
Preconditions.checkState(value == outputValues[i], "fatal error");
}
}
}
This is a second part to my question here.
I now have a process but I want to know how to get the output from the process?
String filename = matlab.getfileName();
Process p = Runtime.getRuntime().exec("java -cp mediaProperty.java " + filename);
My mediaProperty.java:
public class mediaProperty {
public static Object main(String[] args) {
Object[] mediaProp = null;
java.util.List lstMedia = new ArrayList();
Media media = null;
try {
media = new Media();
lstMedia.add(args);
mediaProp = media.media(3, lstMedia);
} catch (Exception p) {
System.out.println("Exception: " + p.toString());
} finally {
MWArray.disposeArray(mediaProp);
if (media != null) {
media.dispose();
}
}
return mediaProp;
}
}
The mediaProperty.java will return an Object. Inside this is actually String array. How do I get the array? And is the way I'm calling exec() correct?
use public static void main (not Object as return type)
Serialize the object using ObjectOutputStream (all necessary examples are in the javadoc)
The only thing different from the example is the construction - construct it like
ObjectOutputStream oos = new ObjectOutputStream(System.out);
in the program calling exec(), get the output with process.getOutputStream()
Read in an ObjectInputStream based on the already retreived OutputStream (check this)
Deserialize the object (see the javadoc of ObjectInputStream)
Now, this is a weird way to do it, but as I don't know exactly what you are trying to achieve, it sounds reasonable.
You could do System.setOut(new PrintStream(p.getOutputStream())) if you'd like to have the process print its results directly to standard output. Of course, this will override the old standard output. But you could also do other things with the process's output stream, like have a thread that reads from it.
A problem with your code is that the main function of a class must be of type void, and will return nothing. You will not be able to pass Java objects between processes, as they are running in different JVMs. If you must do this you could serialize the object to disk, but I imagine you don't even need to run this in a separate process.
mediaProp is a local variable in your main() method. It's not accessible from the outside.
You'll have to redesign your mediaProperty class a bit.
First, you should use:
"java -cp . mediaProperty " + filename
for calling the java process. The "-cp ." defines the classpath and I have made the assumption that the java file is compiled and the generated class file is at the same path as the executing process.
Then, you need to print the result at the standard output and not just return it. Finally, read this article for reading the output.
Tip 1: Rename the class to MediaProperty
Tip 2: Why you don't call the MediaProperty class directly from your code? Is it necessary to start a new process?
There are a few gotcha's.
In exec you assume that java is on the path, and the filename should be fully qualified or you should know that the current working dir of the java process is OK.
main() should return void (nothing). If you want to pass the results out of your program use something like:
for (Object o : mediaProp) {
System.out.println(o);
}
and parse it again on the input stream (the calling software).
Better yet, include the MediaProperty class in the java path and call main(...) directly in stead of calling a separate java process.