We Keep Coding

How to consume a H2OModel computed and saved in Python API

I've been reading the H2O documentation for a while, and I haven't found a clear example of how to load model trained and saved using the Python API. I was following the next example.
import h2o
from h2o.estimators.naive_bayes import H2ONaiveBayesEstimator
model = H2ONaiveBayesEstimator()
h2o_df = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/airlines/allyears2k_headers.zip")
model.train(y = "IsDepDelayed", x = ["Year", "Origin"],
training_frame = h2o_df,
family = "binomial",
lambda_search = True,
max_active_predictors = 10)
h2o.save_model(model, path=models)
But if you check the official documentation it states that you have to download the model as a POJO from the flow UI. Is it the only way? or, may I achieve the same result via python? Just for information, I show the doc's example below. I need some guidance.
import java.io.*;
import hex.genmodel.easy.RowData;
import hex.genmodel.easy.EasyPredictModelWrapper;
import hex.genmodel.easy.prediction.*;
public class main {
private static String modelClassName = "gbm_pojo_test";
public static void main(String[] args) throws Exception {
hex.genmodel.GenModel rawModel;
rawModel = (hex.genmodel.GenModel) Class.forName(modelClassName).newInstance();
EasyPredictModelWrapper model = new EasyPredictModelWrapper(rawModel);
//
// By default, unknown categorical levels throw PredictUnknownCategoricalLevelException.
// Optionally configure the wrapper to treat unknown categorical levels as N/A instead:
//
// EasyPredictModelWrapper model = new EasyPredictModelWrapper(
// new EasyPredictModelWrapper.Config()
// .setModel(rawModel)
// .setConvertUnknownCategoricalLevelsToNa(true));
RowData row = new RowData();
row.put("Year", "1987");
row.put("Month", "10");
row.put("DayofMonth", "14");
row.put("DayOfWeek", "3");
row.put("CRSDepTime", "730");
row.put("UniqueCarrier", "PS");
row.put("Origin", "SAN");
row.put("Dest", "SFO");
BinomialModelPrediction p = model.predictBinomial(row);
System.out.println("Label (aka prediction) is flight departure delayed: " + p.label);
System.out.print("Class probabilities: ");
for (int i = 0; i < p.classProbabilities.length; i++) {
if (i > 0) {
System.out.print(",");
}
System.out.print(p.classProbabilities[i]);
}
System.out.println("");
}
}

h2o.save_model will save the binary model to the provided file system, however, looking at the Java application above it seems you want to use model into a Java based scoring application.
Because of that you should be using h2o.download_pojo API to save the model to local file system along with genmodel jar file. The API is documented as below:
download_pojo(model, path=u'', get_jar=True)
Download the POJO for this model to the directory specified by the path; if the path is "", then dump to screen.
:param model: the model whose scoring POJO should be retrieved.
:param path: an absolute path to the directory where POJO should be saved.
:param get_jar: retrieve the h2o-genmodel.jar also.
Once you have download POJO, you can use the above sample application to perform the scoring and make sure the POJO class name and the "modelClassName" are same along with model type.

Using ELKI with Mongodb

Using test cases I was able to see how ELKI can be used directly from Java but now I want to read my data from MongoDB and then use ELKI to cluster geographic (long, lat) data.
I can only cluster data from a CSV file using ELKI. Is it possible to connect de.lmu.ifi.dbs.elki.database.Database with MongoDB? I can see from the java debugger that there is a databaseconnection field in de.lmu.ifi.dbs.elki.database.Database.
I query MongoDB creating POJO for each row and now I want to cluster these objects using ELKI.
It is possible to read data from MongoDB and write it in a CSV file then use ELKI to read that CSV file but I would like to know if there is a simpler solution.
---------FINDINGS_1:
From ELKI - Use List<String> of objects to populate the Database I found that I need to implement de.lmu.ifi.dbs.elki.datasource.DatabaseConnection and specifically override the loadData() method which returns an instance of MultiObjectsBundle.
So I think I should wrap a list of POJO with MultiObjectsBundle. Now i'm looking at the MultiObjectsBundle and it looks like the data should be held in columns. Why columns datatype is List> shouldnt it be List? just a list of items you want to cluster?
I'm a little confused. How is ELKI going to know that it should look at the long and lat for POJO? Where do I tell ELKI to do this? Using de.lmu.ifi.dbs.elki.data.type.SimpleTypeInformation?
---------FINDINGS_2:
I have tried to use ArrayAdapterDatabaseConnection and I have tried implementing DatabaseConnection. Sorry I need thing in very simple terms for me to understand.
This is my code for clustering:
int minPts=3;
double eps=0.08;
double[][] data1 = {{-0.197574246, 51.49960695}, {-0.084605692, 51.52128377}, {-0.120973687, 51.53005939}, {-0.156876, 51.49313},
{-0.144228881, 51.51811784}, {-0.1680743, 51.53430039}, {-0.170134484,51.52834133}, { -0.096440751, 51.5073853},
{-0.092754157, 51.50597426}, {-0.122502346, 51.52395143}, {-0.136039674, 51.51991453}, {-0.123616824, 51.52994371},
{-0.127854211, 51.51772703}, {-0.125979294, 51.52635795}, {-0.109006325, 51.5216612}, {-0.12221963, 51.51477076}, {-0.131161087, 51.52505093} };
// ArrayAdapterDatabaseConnection dbcon = new ArrayAdapterDatabaseConnection(data1);
DatabaseConnection dbcon = new MyDBConnection();
ListParameterization params = new ListParameterization();
params.addParameter(de.lmu.ifi.dbs.elki.algorithm.clustering.DBSCAN.Parameterizer.MINPTS_ID, minPts);
params.addParameter(de.lmu.ifi.dbs.elki.algorithm.clustering.DBSCAN.Parameterizer.EPSILON_ID, eps);
params.addParameter(DBSCAN.DISTANCE_FUNCTION_ID, EuclideanDistanceFunction.class);
params.addParameter(AbstractDatabase.Parameterizer.DATABASE_CONNECTION_ID, dbcon);
params.addParameter(AbstractDatabase.Parameterizer.INDEX_ID,
RStarTreeFactory.class);
params.addParameter(RStarTreeFactory.Parameterizer.BULK_SPLIT_ID,
SortTileRecursiveBulkSplit.class);
params.addParameter(AbstractPageFileFactory.Parameterizer.PAGE_SIZE_ID, 1000);
Database db = ClassGenericsUtil.parameterizeOrAbort(StaticArrayDatabase.class, params);
db.initialize();
GeneralizedDBSCAN dbscan = ClassGenericsUtil.parameterizeOrAbort(GeneralizedDBSCAN.class, params);
Relation<DoubleVector> rel = db.getRelation(TypeUtil.DOUBLE_VECTOR_FIELD);
Relation<ExternalID> relID = db.getRelation(TypeUtil.EXTERNALID);
DBIDRange ids = (DBIDRange) rel.getDBIDs();
Clustering<Model> result = dbscan.run(db);
int i =0;
for(Cluster<Model> clu : result.getAllClusters()) {
System.out.println("#" + i + ": " + clu.getNameAutomatic());
System.out.println("Size: " + clu.size());
System.out.print("Objects: ");
for(DBIDIter it = clu.getIDs().iter(); it.valid(); it.advance()) {
DoubleVector v = rel.get(it);
ExternalID exID = relID.get(it);
System.out.print("DoubleVec: ["+v+"]");
System.out.print("ExID: ["+exID+"]");
final int offset = ids.getOffset(it);
System.out.print(" " + offset);
}
System.out.println();
++i;
}
The ArrayAdapterDatabaseConnection produces two clusters, I just had to play around with the value of epsilon, when I set epsilon=0.008 dbscan started creating clusters. When i set epsilon=0.04 all the items were in 1 cluster.
I have also tried to implement DatabaseConnection:
#Override
public MultipleObjectsBundle loadData() {
MultipleObjectsBundle bundle = new MultipleObjectsBundle();
List<Station> stations = getStations();
List<DoubleVector> vecs = new ArrayList<DoubleVector>();
List<ExternalID> ids = new ArrayList<ExternalID>();
for (Station s : stations){
String strID = Integer.toString(s.getId());
ExternalID i = new ExternalID(strID);
ids.add(i);
double[] st = {s.getLongitude(), s.getLatitude()};
DoubleVector dv = new DoubleVector(st);
vecs.add(dv);
}
SimpleTypeInformation<DoubleVector> type = new VectorFieldTypeInformation<>(DoubleVector.FACTORY, 2, 2, DoubleVector.FACTORY.getDefaultSerializer());
bundle.appendColumn(type, vecs);
bundle.appendColumn(TypeUtil.EXTERNALID, ids);
return bundle;
}
These long/lat are associated with an ID and I need to link them back to this ID to the values. Is the only way to go that using the ID offset (in the code above)? I have tried to add ExternalID column but I don't know how to retrieve the ExternalID for a particular NumberVector?
Also after seeing Using ELKI's Distance Function I tried to use Elki's longLatDistance but it doesn't work and I could not find any examples to implement it.

The interface for data sources is called DatabaseConnection.
JavaDoc of DatabaseConnection
You can implement a MongoDB-based interface to get the data.
It is not complicated interface, it has a single method.

Hazelcast not able to read data into my own class object using IMap.get, where data inserted using Memcached text protocol

I am inserting record into Hazelcast from C Application using Memcached Client Library API's, where record is as follows:
typedef struct _activeClient
{
char ID[25];
int IP;
char aMethod[16];
}activeClient;
Now I am trying reading same record using Hazelcast Java Native API's. Here is my Java program.
IMap < String, MemcacheEntry > mapInst = client.getMap("hz_memcache_ABC_MAP");
System.out.println("Map Size:" + mapInst.size());
String key = new String("70826892122991");
MemcacheEntry tmpValRec = pvrMapIst.get(key);
System.out.println("Key:" + key + "ID:" + tmpValRec.getValue());
Here tmpValRec.getValue() printing record content in single String format. But, I want to retrive each member value from tmpValRec to my own java class object. Here is the class
class ActiveClients
{
String ueID;
int Ip;
String aMethod;
ActiveClients()
{
ueID = "";
Ip = 0;
aMethod = "";
}
}
Pointing me to an example would be great help.

I guess the only option is to parse the string to deserialize your object. I know this is a pain, but I don't see a better alternative. Unless of course you store a blob as a value in memcached where the blob is the serialized content of the class.

Serializing RowMutation

Inside the Cassandra source trunk on Github (https://github.com/apache/cassandra), there's an example of writing data in examples/client_only/src/ClientOnlyExample.java:
private static void testWriting() throws Exception
{
// do some writing.
for (int i = 0; i < 100; i++)
{
RowMutation change = new RowMutation(KEYSPACE, ByteBufferUtil.bytes(("key" + i)));
ColumnPath cp = new ColumnPath(COLUMN_FAMILY).setColumn(("colb").getBytes());
change.add(new QueryPath(cp), ByteBufferUtil.bytes(("value" + i)), 0);
// don't call change.apply(). The reason is that is makes a static call into Table, which will perform
// local storage initialization, which creates local directories.
// change.apply();
StorageProxy.mutate(Arrays.asList(change), ConsistencyLevel.ONE);
System.out.println("wrote key" + i);
}
System.out.println("Done writing.");
}
I'm looking to serialize the data into a readable format (JSON) where the writes seem to happen, org.apache.cassandra.service.StorageProxy, inside of the method performWrite:
public static IWriteResponseHandler performWrite(IMutation mutation,
ConsistencyLevel consistency_level,
String localDataCenter,
WritePerformer performer)
throws UnavailableException, IOException
{
...
The IMutation parameter appears to be what I want, as RowMutation implements that class. I can get the table (keyspace), and the column families out, but can't seem to get the column name/values. If I'm inside of the mentioned method, how do I get that information from IMutation mutation?
// keyspace
String table = mutation.getTable();
// TODO won't work with batch?
UUID cfId = mutation.getColumnFamilyIds().iterator().next();
// column family name cfMetadata.cfName
CFMetaData cfMetadata = Schema.instance.getCFMetaData(cfId);
// row key
RowMutation data = new RowMutation(table, mutation.key());
String row = ByteBufferUtil.bytesToHex(data.key());
// column name/values ??
// data. ....

I looked in the row mutation source code where there was on same package a RowMutationSerializer which has method
serialize(RowMutation, DataOutputStream, int)
Could this serve for something?

Java ArrayList into Name value pair

In a java class, am using an arraylist say reports containing list of all the reports which have reportid, reportname, reporttype etc which i want to add into NameValuePair and send a Http postmethod call to a particular url.
I want to add the arraylists - reportname into name value pair(org.apache.commons.httpclient.NameValuePair) and then use the http client post method to submit the name value pair data to a particular url.
Here is my name value pair
if (validateRequest()) {
NameValuePair[] data = {
new NameValuePair("first_name", firstName),
new NameValuePair("last_name", lastName),
new NameValuePair("email", mvrUser.getEmail()),
new NameValuePair("organization", mvrUser.getOrganization()),
new NameValuePair("phone", mvrUser.getPhone()),
new NameValuePair("website", mvrUser.getWebsite()),
new NameValuePair("city", mvrUser.getCity()),
new NameValuePair("state", mvrUser.getState()),
new NameValuePair("country", mvrUser.getCountry()),
new NameValuePair(**"report(s)", reports**.)
};
please suggest me how to add the reports arraylist reportname into reports field of NameValuePair.
--
thanks
# adarsh
can I use with generics something like this?
reportname = "";
for (GSReport report : reports) {
reportname = reportname + report.getReportName();
reportname += ",";
}
and then add in namevalue pair as
new NameValuePair("report(s)", reportname)

for name value pair use map like things... eg. Hastable(it is synchronized) , u can use other
implementation of Map which are not synchronized.

I suggest to serialize your reports ArrayList into a JSON formatted String.
new NameValuePair("reports", reportsAsJson)
You can build your reportsAsJson variable using any of the JSON serialization libraries (like the one at http://json.org/java/). It will have approximatively this format :
reportsAsJson = "[{reportid:'1',reportname:'First Report',reporttype:'Type 1'}, {reportid:'2',reportname:'Seond Report',reporttype:'Type 2'}]";

Well, you cannot do that. NameValuePair takes in String arguments in the constructor. It makes sense as it is used for HTTP POST.
What you can do is come up with a way to serialize the Report object into String and send this string as a string parameter. One way of doing this maybe is to delimit the parameters of the Report class.
reports=reportName1|reportName2|reportName3
Assuming reports is your ArrayList,
String reportsStr = "";
for(int i = 0; i < reports.size(); i++) {
reportStr += reportName;
if(i != reports.size() - 1) {
reportsStr += "|";
}
}
NameValuePair nvPair = new NameValuePair("reports", reportsStr);