I'm currently working on accessing HBase using python3. The way I'm doing is using py4j to call JAVA APIs that I'm writing to access HBase.
I've a question related to creating a Put object which takes a qualifier and value.
I want to pass a dictionary to a JAVA class which expects a hashmap. Is it possible through py4j.
I don't want to call Put for every column qualifier iteratively. I want to pass the dict to py4j and it should be received as HashMap on JAVA side.
Could you please some hints/pointers to how can this be done...
There are two ways to do what you want:
You can create a java.util.HashMap() and use it as a dict on the python side. This is good if you pass the dictionary a lot, but you do not modify it often on the python side. This is also good if the dictionary is modified on the java side and you want to see the modifications on the python side.
Py4J can automatically convert a python dict to a HashMap when calling a Java method. Note that the dictionary will be copied and that any change performed on the Java side won't be reflected on the Python side.
The easiest solution would be #1 I believe:
>>> m = gateway.jvm.java.util.HashMap()
>>> m["a"] = 0
>>> m.put("b",1)
>>> m
{u'a': 0, u'b': 1}
>>> u"b" in m
True
>>> del(m["a"])
>>> m
{u'b': 1}
>>> m["c"] = 2
Related
I am using a Machine learning and statistical based classifier called as EVM (Extreme value machine) using link https://pypi.org/project/EVM/. I train the model in python and save it as pickle file with the below code.
import EVM
import pickle
mevm = EVM.MultipleEVM(tailsize=300, cover_threshold = 0.7, distance_multiplier = 0.7, distance_function = scipy.spatial.distance.euclidean)
mevm.train([trainX[i] for i in list(trainX.keys())])
#Save the trained model
with open('model.pkl','wb') as f:
pickle.dump(mevm,f)
I could easily load the same model in Python using same Pickle module. When I save, all the weights gets saved inside the pickle file and I could do the inference as below,
import pickle
import EVM
with open('model.pkl','rb') as b:
mevm = pickle.load(b)
print("Probability of given input", mevm.max_probabilities([embeds[i]]))
But I could not read the same pickle file in Android/Java (I want to use this saved model in my android application). When I check the EVM class, there is a C file and a C++ file which is connected using a '.pyx' file. All these 3 files are read using a python EVM (i.e. import EVM) class. Please refer to the below link for the files
https://pypi.org/project/EVM/#files
My questions are
How can I read this saved model in Android/Java ?
Since this is linked with C,C++ and python, will it be possible to read in Java ?
Any other better method to save this, so that I can use it Andriod/Java ?
Any leads are appreciated.
Thanks in Advance :)
According to python documentation here
The pickle module implements binary protocols for serializing and de-serializing a Python object structure. “Pickling” is the process whereby a Python object hierarchy is converted into a byte stream, and “unpickling” is the inverse operation, whereby a byte stream (from a binary file or bytes-like object) is converted back into an object hierarchy. Pickling (and unpickling) is alternatively known as “serialization”, “marshalling,” 1 or “flattening”; however, to avoid confusion, the terms used here are “pickling” and “unpickling”
This means that the given byte stream is only syntactically valid in Python only unless you want to write an object transpiler.
You are approaching your problem in the wrong way. ML model is just a set of coefficients/parameters that you have optimized to get a close approximation of your target variable. You should be able to take that set of coefficients/parameters and use it in any language.
In Python 2.x, I could pass custom function to sorted and .sort functions
>>> x=['kar','htar','har','ar']
>>>
>>> sorted(x)
['ar', 'har', 'htar', 'kar']
>>>
>>> sorted(x,cmp=customsort)
['kar', 'htar', 'har', 'ar']
Because, in My language, consonents are comes with this order
"k","kh",....,"ht",..."h",...,"a"
But In Python 3.x, looks like I could not pass cmp keyword
>>> sorted(x,cmp=customsort)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'cmp' is an invalid keyword argument for this function
Is there any alternatives or should I write my own sorted function too?
Note: I simplified by using "k", "kh", etc. Actual characters are Unicodes and even more complicated, sometimes there is vowels comes before and after consonents, I've done custom comparison function, So that part is ok. Only the problem is I could not pass my custom comparison function to sorted or .sort
Use the key keyword and functools.cmp_to_key to transform your comparison function:
sorted(x, key=functools.cmp_to_key(customsort))
Use the key argument (and follow the recipe on how to convert your old cmp function to a key function).
functools has a function cmp_to_key mentioned at docs.python.org/3.6/library/functools.html#functools.cmp_to_key
A complete python3 cmp_to_key lambda example:
from functools import cmp_to_key
nums = [28, 50, 17, 12, 121]
nums.sort(key=cmp_to_key(lambda x, y: 1 if str(x)+str(y) < str(y)+str(x) else -1))
compare to common object sorting:
class NumStr:
def __init__(self, v):
self.v = v
def __lt__(self, other):
return self.v + other.v < other.v + self.v
A = [NumStr("12"), NumStr("121")]
A.sort()
print(A[0].v, A[1].v)
A = [obj.v for obj in A]
print(A)
Instead of a customsort(), you need a function that translates each word into something that Python already knows how to sort. For example, you could translate each word into a list of numbers where each number represents where each letter occurs in your alphabet. Something like this:
my_alphabet = ['a', 'b', 'c']
def custom_key(word):
numbers = []
for letter in word:
numbers.append(my_alphabet.index(letter))
return numbers
x=['cbaba', 'ababa', 'bbaa']
x.sort(key=custom_key)
Since your language includes multi-character letters, your custom_key function will obviously need to be more complicated. That should give you the general idea though.
I don't know if this will help, but you may check out the locale module. It looks like you can set the locale to your language and use locale.strcoll to compare strings using your language's sorting rules.
Use the key argument instead. It takes a function that takes the value being processed and returns a single value giving the key to use to sort by.
sorted(x, key=somekeyfunc)
I'm trying to use the Apache XML-RPC Java client into Matlab, in combination with a python SimpleXMLRPCserver.
From the python server, I try to return a dictionary that is linking strings with lists, juts like the following:
return {'node15': [12,58748], 'node34': [28,45784]}
Then when calling the method from the maltab client, I end up with what matlab calls a hashmap:
{'node15' = [Ljava.lang.Object;#6f02ae95, node34 = [Ljava.lang.Object;#1913f123}
I didn't find a way to extract my list into arrays, my next step was to use values() to extract them, but it only returns a "HashMap$Values" list containing this:
[ [Ljava.lang.Object;#6f02ae95, [Ljava.lang.Object;#1913f12]
It looks like matlab failed to translate the java structure into something it understands.
Any idea what to do with this ?
You need to convert it to array first, then you can use cell to cast to Matlab native types:
cell(myHashMap.values().toArray());
I want to use JRuby to run some ruby scripts, get a result and continue using Java/Scala. Unfortunately, I don't know how to convert an object from a RubyHash to a Java or Scala Map.
Is there a simple way to do this?
Any suggestions on how to do it?
class org.jruby.RubyHash implements java.util.Map, so, it already is a Java Map, albeit a non-generic "raw" type.
Unfortunately, JRuby's Hash class only inherits the generic to_java method from Kernel, unlike Array which has its own version.
This works:
[1, 2, 3].to_java(:int)
This doesn't, although it looks like it should:
{ 'one' => 1, 'two' => 2 }.to_java(:string, :int)
[It might be worthwhile filing this as a feature request.]
org.jruby.RubyHash implements java.util.Map in a way that it attempts to convert Ruby objects to Java with "normal" Map operations.
e.g. if you have a hash such as: { 'a' => 1, 'b' => 2 } which (internally) means you have RubyString keys mapped to RubyFixnum values in a hash instance, on the Java side hash.get("a") will work fine and return 1 as a java.lang.Long
I would suggest to change your approach slightly: this would involve only changing the way you initialize objects in ruby, the rest should be transparent.
Instead of using pure Ruby hash objects (o=Hash.new or o={}), please use instead java.util.HashMap like in this snippet:
require 'java'
o = java.util.HashMap.new # you could also use o = Java::JavaUtil::HashMap.new
From then on, o would be a java object used as a hash by Ruby code, and can still be accessed using java methods.
Another mode complete snippet:
>> require 'java'
=> true
>> o = Java::JavaUtil::HashMap.new
=> {}
>> o[1] = 2
=> 2
>> o["cm"] = true
=> true
>> p o.keys # ruby methods/interface
[1, "cm"]
=> nil
>> p o.keySet().to_a # java methods
[1, "cm"]
=> nil
I would like to embed Clojure code into Java. This site was helpful in the basics of setting this up, but the only arg it ever passes is of type String. I have tried using ints as well, and those also work.
My question is whether there is some formatted way to pass in structured data to Clojure. In particular, I have a list of points I would like to pass to Clojure and turn into a vector that would look something like this:
[[1 2] [3 4] [5 6]]
What is the easiest way to go about doing this? Is there preprocessing I can do on Java's end, or should I do postprocessing on Clojure's end, or is there something in Clojure that will handle this? I suspect it's passing in a String of numbers and the length of each tuple to Clojure, and letting it process the String into a vector. However, this aspect of Clojure doesn't have many examples, and I'm curious if I'm missing something obvious.
EDIT: Please look at mikera's answer is you're interested in passing in Java Objects. Please look at my answer below if you just want to format your data ahead of time into a Clojure format for a set/map/etc.
It depends a bit on what format your data is in to start with, but you may find it easiest just to pass the Java object that represents the data directly and read it using Clojure's Java interoperability features.
e.g. you could pass a Java array of Points directly and do something like:
(let [point (aget some-array index)]
(do-stuff-with-point point))
For those cases when you want to pass in a simple data structure that you've already formatted to look like Clojure in Java, you can pass in that arg as a string. So for the example I gave in my question, I would pass in
"[[1 2] [3 4] [5 6]]"
as my arg. When you've called Clojure using invoke(arg), you can make the first step of your function to be a call to readString on your arg:
(defn foo [d]
(def data (read-string d)))
The above will produce a vector when the example String is passed in.