We Keep Coding

Using trained TensorFlow model in Java

I have trained a TensorFlow model in Python and would like to use it in Java code. Training the model is done via something like this code:
def input_fn():
features = {'a': tf.constant([[1],[2]]),
'b': tf.constant([[3],[4]]) }
labels = tf.constant([0, 1])
return features, labels
feature_a = tf.contrib.layers.sparse_column_with_integerized_feature("a", bucket_size=10)
feature_b = tf.contrib.layers.sparse_column_with_integerized_feature("b", bucket_size=10)
feature_columns = [feature_a, feature_b]
model = tf.contrib.learn.LinearClassifier(feature_columns=feature_columns)
model.fit(input_fn=input_fn, steps=10)
Now I want to save this model to use it in Java. It seems that export_savedmodel is the new/preferred way of saving, so I tried:
feature_spec = tf.contrib.layers.create_feature_spec_for_parsing(feature_columns)
serving_input_fn = input_fn_utils.build_parsing_serving_input_fn(feature_spec)
model.export_savedmodel('export', serving_input_fn, as_text=True)
This results in a saved model, which can be loaded from Java with
model = SavedModelBundle.load(dir, "serve");
model.session().runner()
.feed("input_example_tensor", input)
.fetch("linear/binary_logistic_head/predictions/probabilities")
.run();
There is now a problem though: the input_example_tensor should be a Tensor containing Strings/byte[]s, but this is not supported in Java yet (see: Tensor.java#88 "throw new UnsupportedOperationException"). As far as I understand it, the reason that it wants a String is that build_parsing_serving_input_fn wants to parse serialized Example protocol buffers.
Maybe a different serving_input_fn would do better. input_fn_utils.build_default_serving_input_fn looks promising, but I didn't get that to work.
If I call it like:
features_dict = {'a':feature_a, 'b':feature_b}
serving_input_fn = input_fn_utils.build_default_serving_input_fn(features)
I get "AttributeError: '_SparseColumnIntegerized' object has no attribute 'get_shape'"
If I call it like:
features = {'a': tf.constant([[1],[2]]),
'b': tf.constant([[3],[4]]) }
serving_input_fn = input_fn_utils.build_default_serving_input_fn(features)
I get "ValueError: 'Const:0' is not a valid scope name".
What is the proper way to use input_fn_utils.build_default_serving_input_fn? I can't find any example that uses it.

How do you use stream operations to calculate an average of values within a list, omitting some?

I have a method that returns the average of a property over a number of model objects:
List<Activity> activities = ...;
double effortSum = 0;
double effortCount = 0;
activities.stream().forEach(a -> {
double effort = a.getEffort();
if (effort != Activity.NULL) {
effortCount++; < Compilation error, local variable
effortSum += effort; < Compilation error, local variable
}
});
But, the above attempt doesn't compile, as noted. The only solution that's coming to me is using an AtomicReference to a Double object, but that seems crufty, and adds a large amount of confusion to what should be a simple operation. (Or adding Guava and gaining AtomicDouble, but the same conclusion is reached.)
Is there a "best practice" strategy for modifying local variables using the new Java 8 loops?
Relevant code for Activity:
public class Activity {
public static final double NULL = Double.MIN_VALUE;
private double effort = NULL;
public void setEffort(double effort) { this.effort = effort; }
public double getEffort() { return this.effort; }
...
}

Is there a "best practice" strategy for modifying local variables using the new Java 8 loops?
Yes: don't. You can modify their properties -- though it's still a bad idea -- but you cannot modify them themselves; you can only refer to variables from inside a lambda if they are final or could be final. (AtomicDouble is indeed one solution, another is a double[1] that just serves as a holder.)
The correct way of implementing the "average" operation here is
activities.stream()
.mapToDouble(Activity::getEffort)
.filter(effort -> effort != Activity.NULL)
.average()
.getAsDouble();

In your case, there is a solution that is more functional - just compute the summary statistics from the stream from where you can grab the number of elements filtered and their sum:
DoubleSummaryStatistics stats =
activities.stream()
.mapToDouble(Activity::getEffort)
.filter(e -> e != Activity.NULL)
.summaryStatistics();
long effortCount = stats.getCount();
double effortSum = stats.getSum();
Is there a "best practice" strategy for modifying local variables
using the new Java 8 loops?
Don't try do to that. I think the main issues is that people try to translate their code using the new Java 8 features in an imperative way (like in your question - and then you have troubles!).
Try to see first if you can provide a solution which is functional (which is what the Stream API aim for, I believe).

Passing java object to python

I am prototyping an interface to our application to allow other people to use python, our application is written in java. I would like to pass some of our data from the java app to the python code but I am unsure how to pass an object to python. I have done a simple java->python function call using simple parameters using Jython and found it very useful for what I am trying to do. Given the class below, how can I then use it in Python/Jython as an input to a function/class:
public class TestObject
{
private double[] values;
private int length;
private int anotherVariable;
//getters, setters
}

One solution. You could use some sort of message system, queue, or broker of some sort to serialize/deserialize, or pass messages between python and java. Then create some sort workers/producer/consumers to put work on the queues to be processed in python, or java.
Also consider checking out for inspiration: https://www.py4j.org/
py4j is used heavily by/for pyspark and hadoop type stuff.
To answer your question more immediately.
Example using json-simple.:
import org.apache.commons.io.FileUtils;
import org.json.simple.JSONObject;
//import org.json.simple.JSONObject;
public class TestObject
{
private double[] values;
private int length;
private int anotherVariable;
private boolean someBool;
private String someString;
//getters, setters
public String toJSON() {
JSONObject obj=new JSONObject();
obj.put("values",new Double(this.values));
obj.put("length",new Integer(this.length));
obj.put("bool_val",new Boolean(this.SomeBool));
obj.put("string_key",this.someString);
StringWriter out = new StringWriter();
obj.writeJSONString(out);
return out.toString();
}
public void writeObject(){
Writer writer = new BufferedWriter(
new OutputStreamWriter(
new FileOutputStream("anObject.json"), "utf-8")
)
)
writer.write(this.toJSON());
}
public static void setObject(){
values = 100.134;
length = 12;
anotherVariable = 15;
someString = "spam";
}
}
And in python:
class DoStuffWithObject(object):
def __init__(self,obj):
self.obj = obj
self.changeObj()
self.writeObj()
def changeObj(self):
self.obj['values'] = 100.134;
self.obj['length'] = 12;
self.obj['anotherVariable'] = 15;
self.obj['someString'] = "spam";
def writeObj(self):
''' write back to file '''
with open('anObject.json', 'w') as f:
json.dump(self.obj, f)
def someOtherMethod(self, s):
''' do something else '''
print('hello {}'.format(s))
import json
with open('anObject.json','r') as f:
obj = json.loads(f.read())
# print out obj['values'] obj['someBool'] ...
for key in obj:
print(key, obj[key])
aThing = DoStuffWithObject(obj)
aThing.someOtherMethod('there')
And then in java read back the object. There are solutions that exist implementing this idea (JSON-RPC, XML-RPC, and variants). Depending, you may may also want to consider using something like ( http://docs.mongodb.org/ecosystem/drivers/java/ ) the benefit being that mongo does json.
See:
https://spring.io/guides/gs/messaging-reactor/
http://spring.io/guides/gs/messaging-rabbitmq/
http://spring.io/guides/gs/scheduling-tasks/
Celery like Java projects
Jedis
RabbitMQ
ZeroMQ
A more comprehensive list of queues:
http://queues.io/
Resources referenced:
http://www.oracle.com/technetwork/articles/java/json-1973242.html
How do I create a file and write to it in Java?
https://code.google.com/p/json-simple/wiki/EncodingExamples

Agree with the answer below. I think that the bottom line is that "Python and Java are separate interpreter-environments." You therefore shouldn't expect to transfer "an object" from one to the other. You shouldn't expect to "call methods." But it is reasonable to pass data from one to another, by serializing and de-serializing it through some intermediate data format (e.g. JSON) as you would do with any other program.
In some environments, such as Microsoft Windows, it's possible that a technology like OLE (dot-Net) might be usable to allow environments to be linked-together "actively," where the various systems implement and provide OLE-objects. But I don't have any personal experience with whether, nor how, this might be done.
Therefore, the safest thing to do is to treat them as "records," and to use serialization techniques on both sides. (Or, if you got very adventurous, run (say) Java in a child-thread.) An "adventurous" design could get out-of-hand very quickly, with little return on investment.

You need to make the python file to exe using py2exe , Refer the link : https://www.youtube.com/watch?v=kyoGfnLm4LA. Then use the program in java and pass arguements:
Please refer this link it will be having the details:
Calling fortran90 exe program from java is not executing

WEKA - Classifying New Data from Java - IDF Transform

We are trying to implement a WEKA classifier from inside a Java program. So far so good, everything works well however when building the classifier from the training set in Weka GUI we used the StringToWordVector IDF transform to help improve classification accuracy.
How, from within Java for new instances do I calculate the IDF transform to set for each token value in the new instance before passing the instance to the classifier?
The basic code looks like this:
Instances ins = vectorize(msg);
Instances unlabeled = new Instances(train,1);
Instance inst = new Instance(unlabeled.numAttributes());
String tmp = "";
for(int i=0; i < ins.numAttributes(); i++) {
tmp = ins.attribute(i).name();
if(unlabeled.attribute(tmp)!=null)
inst.setValue(unlabeled.attribute(tmp), 1.0); //TODO: Need to figure out the IDF transformed value to put here NOT 1!!
}
unlabeled.add(inst);
unlabeled.setClassIndex(classIdx);
.....cl.distributionForInstance(unlabeled.instance(i));
So how do I go about coding this so that I put the correct value in the new instance I want to classify?
Just to be clear the line inst.setValue(unlabeled.attribute(tmp), 1.0); needs to be changed from 1.0 to the IDF transformed number...

You need to use FilteredClassifier for this purpose. The code snippet is :
StringToWordVector strWVector = new StringToWordVector();
filteredClassifier fcls = new FilteredClassifier();
fcls.setFilter(strWVector);
fcls.setClassifier(new SMO());
fcls.buildClassifier(yourdata)
//rest of your code
This is much easier as you can pass your instances all at once.FilteredClassifier takes care of all other details. The code is not tested but it will get you started.
Edit : You can do in the following way too. This is code snippet from weka tutorial See http://weka.wikispaces.com/Use+WEKA+in+your+Java+code#Filter-Filtering%20on-the-fly Batch Mode for details
Instances train = ... // from somewhere
Instances test = ... // from somewhere
Standardize filter = new Standardize();
filter.setInputFormat(train); // initializing the filter once with training set
Instances newTrain = Filter.useFilter(train, filter); // configures the Filter based on train instances and returns filtered instances
Instances newTest = Filter.useFilter(test, filter); // create new test se
HTH

Java SWT interop with COM - putting a float[] into a Variant?

In my Java SWT application I'm hosting an 3rd party ActiveX control. I'm using OleClientSite to do this.
// Ah, this works. :-)
OleAutomation comObject = new OleAutomation(...);
There are 2 easy little functions I want to call from Java. Here are the COM function definitions:
[id(5)]
void easyFoo([in] int blah);
[id(20)]
void problemFoo([in] VARIANT floatArray);
Easy, right? Here's my pretend code:
// Ah, this works. :-)
OleAutomation comObject = new OleAutomation("Some3rdPartyControlHere");
// Call easyFoo(42). This works. :-)
int easyFooId = 5;
comObject.invoke(easyFooId, new Variant[] { new Variant(42) });
// Call problemFoo(new float[] { 4.2, 7.0 }). This doesn't work. :-(
int problemFooId = 20;
comObject.invoke(problemFooId, [ACK! What goes here?]);
The problem is on the last line: how do I pass a float array to the 3rd party COM object? HELP!

You need to pass a float array. In COM terms, that mean s a VARIANT with vt set to VT_R4|VT_ARRAY. An array of variants may not work as the document does not say it can accept an array of variants (VT_VARIANT |VT_ARRAY). In java you should be able to use float[] as the parameter type. If not you can always call the Windows API to construct a safe array of desired type.

I suspect there is no constructor that takes a float[] because VARIANTs don't have a float array member.
I think what you need to do to make this work is pack up your floats into a SAFEARRAY (ick; and I have no idea how to create one in Java).
Alternatively, you may try serializing your array to raw bits and use the BYTE* member of the VARIANT struct, and pass an int that has the count of bytes so you can accurately de-serialize on the other side (and I assume this is all in the same process and thread, otherwise it gets harder).
[id(20)]
void problemFoo([in] VARIANT bytes /* VT_BYREF|VT_UI1 */, [in] VARIANT byteCount /* VT_UI4 */);

What's wrong with creating an array of Variant and filling it with your float array values?
Variant[] problemFooArgs = new Variant[myFloats.length];
for( int i=0; i<myFloats.length; i++)
{
problemFooArgs[i] = new Variant(myFloats[i]);
}
If it really needs only one argument (an array of float), you could try one level of indirection:
Variant[] problemFooArgs = new Variant[1];
Variant[] myFooArgs = new Variant[1];
for( int i=0; i<myFloats.length; i++)
{
myFooArgs [i] = new Variant(myFloats[i]);
}
problemFooArgs[0] = myFooArgs;
If the simple approach does not work and you do need a SAFEARRAY, you could try and create one after the example "Reading and writing to a SAFEARRAY", using the constants of org.eclipse.swt.internal.win32.OS. But it seems for char[] only.
Other source of inspiration for creating the relevant SAFEARRAY:
class SafeArray of project com4j (and its associated class, like Variant)