We Keep Coding

How to invoke model from TensorFlow Java?

The following python code passes ["hello", "world"] into the universal sentence encoder and returns an array of floats denoting their encoded representation.
import tensorflow as tf
import tensorflow_hub as hub
module = hub.KerasLayer("https://tfhub.dev/google/universal-sentence-encoder/4")
model = tf.keras.Sequential(module)
print("model: ", model(["hello", "world"]))
This code works but I'd now like to do the same thing using the Java API. I've successfully loaded the module, but I am unable to pass inputs into the model and extract the output. Here is what I've got so far:
import org.tensorflow.Graph;
import org.tensorflow.SavedModelBundle;
import org.tensorflow.Session;
import org.tensorflow.Tensor;
import org.tensorflow.Tensors;
import org.tensorflow.framework.ConfigProto;
import org.tensorflow.framework.GPUOptions;
import org.tensorflow.framework.GraphDef;
import org.tensorflow.framework.MetaGraphDef;
import org.tensorflow.framework.NodeDef;
import org.tensorflow.util.SaverDef;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
public final class NaiveBayesClassifier
{
public static void main(String[] args)
{
new NaiveBayesClassifier().run();
}
protected SavedModelBundle loadModule(Path source, String... tags) throws IOException
{
return SavedModelBundle.load(source.toAbsolutePath().normalize().toString(), tags);
}
public void run()
{
try (SavedModelBundle module = loadModule(Paths.get("universal-sentence-encoder"), "serve"))
{
Graph graph = module.graph();
try (Session session = new Session(graph, ConfigProto.newBuilder().
setGpuOptions(GPUOptions.newBuilder().setAllowGrowth(true)).
setAllowSoftPlacement(true).
build().toByteArray()))
{
Tensor<String> input = Tensors.create(new byte[][]
{
"hello".getBytes(StandardCharsets.UTF_8),
"world".getBytes(StandardCharsets.UTF_8)
});
List<Tensor<?>> result = session.runner().feed("serving_default_inputs", input).
addTarget("???").run();
}
}
catch (IOException e)
{
e.printStackTrace();
}
}
}
I used https://stackoverflow.com/a/51952478/14731 to scan the model for possible input/output nodes. I believe the input node is "serving_default_inputs" but I can't figure out the output node. More importantly, I don't have to specify any of these values when invoking the code in python through Keras so is there a way to do the same using the Java API?
UPDATE: Thanks to roywei I can now that confirm the input node is serving_default_input and output node is StatefulPartitionedCall_1 but when I plug these names into the aforementioned code I get:
2020-05-22 22:13:52.266287: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at lookup_table_op.cc:809 : Failed precondition: Table not initialized.
Exception in thread "main" java.lang.IllegalStateException: [_Derived_]{{function_node __inference_pruned_6741}} {{function_node __inference_pruned_6741}} Error while reading resource variable EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_25 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_25/class tensorflow::Var does not exist.
[[{{node EncoderDNN/DNN/ResidualHidden_0/dense/kernel/ConcatPartitions/concat/ReadVariableOp_25}}]]
[[StatefulPartitionedCall_1/StatefulPartitionedCall]]
at libtensorflow#1.15.0/org.tensorflow.Session.run(Native Method)
at libtensorflow#1.15.0/org.tensorflow.Session.access$100(Session.java:48)
at libtensorflow#1.15.0/org.tensorflow.Session$Runner.runHelper(Session.java:326)
at libtensorflow#1.15.0/org.tensorflow.Session$Runner.run(Session.java:276)
Meaning, I still cannot invoke the model. What am I missing?

I figured it out after roywei pointed me in the right direction.
I needed to use SavedModuleBundle.session() instead of constructing my own instance. This is because the loader initializes the graph variables.
Instead of passing a ConfigProto to the Session constructor, I passed it into the SavedModelBundle loader instead.
I needed to use fetch() instead of addTarget() to retrieve the output tensor.
Here is the working code:
public final class NaiveBayesClassifier
{
public static void main(String[] args)
{
new NaiveBayesClassifier().run();
}
public void run()
{
try (SavedModelBundle module = loadModule(Paths.get("universal-sentence-encoder"), "serve"))
{
try (Tensor<String> input = Tensors.create(new byte[][]
{
"hello".getBytes(StandardCharsets.UTF_8),
"world".getBytes(StandardCharsets.UTF_8)
}))
{
MetaGraphDef metadata = MetaGraphDef.parseFrom(module.metaGraphDef());
Map<String, Shape> nameToInput = getInputToShape(metadata);
String firstInput = nameToInput.keySet().iterator().next();
Map<String, Shape> nameToOutput = getOutputToShape(metadata);
String firstOutput = nameToOutput.keySet().iterator().next();
System.out.println("input: " + firstInput);
System.out.println("output: " + firstOutput);
System.out.println();
List<Tensor<?>> result = module.session().runner().feed(firstInput, input).
fetch(firstOutput).run();
for (Tensor<?> tensor : result)
{
{
float[][] array = new float[tensor.numDimensions()][tensor.numElements() /
tensor.numDimensions()];
tensor.copyTo(array);
System.out.println(Arrays.deepToString(array));
}
}
}
}
catch (IOException e)
{
e.printStackTrace();
}
}
/**
* Loads a graph from a file.
*
* #param source the directory containing to load from
* #param tags the model variant(s) to load
* #return the graph
* #throws NullPointerException if any of the arguments are null
* #throws IOException if an error occurs while reading the file
*/
protected SavedModelBundle loadModule(Path source, String... tags) throws IOException
{
// https://stackoverflow.com/a/43526228/14731
try
{
return SavedModelBundle.loader(source.toAbsolutePath().normalize().toString()).
withTags(tags).
withConfigProto(ConfigProto.newBuilder().
setGpuOptions(GPUOptions.newBuilder().setAllowGrowth(true)).
setAllowSoftPlacement(true).
build().toByteArray()).
load();
}
catch (TensorFlowException e)
{
throw new IOException(e);
}
}
/**
* #param metadata the graph metadata
* #return the first signature, or null
*/
private SignatureDef getFirstSignature(MetaGraphDef metadata)
{
Map<String, SignatureDef> nameToSignature = metadata.getSignatureDefMap();
if (nameToSignature.isEmpty())
return null;
return nameToSignature.get(nameToSignature.keySet().iterator().next());
}
/**
* #param metadata the graph metadata
* #return the output signature
*/
private SignatureDef getServingSignature(MetaGraphDef metadata)
{
return metadata.getSignatureDefOrDefault("serving_default", getFirstSignature(metadata));
}
/**
* #param metadata the graph metadata
* #return a map from an output name to its shape
*/
protected Map<String, Shape> getOutputToShape(MetaGraphDef metadata)
{
Map<String, Shape> result = new HashMap<>();
SignatureDef servingDefault = getServingSignature(metadata);
for (Map.Entry<String, TensorInfo> entry : servingDefault.getOutputsMap().entrySet())
{
TensorShapeProto shapeProto = entry.getValue().getTensorShape();
List<Dim> dimensions = shapeProto.getDimList();
long firstDimension = dimensions.get(0).getSize();
long[] remainingDimensions = dimensions.stream().skip(1).mapToLong(Dim::getSize).toArray();
Shape shape = Shape.make(firstDimension, remainingDimensions);
result.put(entry.getValue().getName(), shape);
}
return result;
}
/**
* #param metadata the graph metadata
* #return a map from an input name to its shape
*/
protected Map<String, Shape> getInputToShape(MetaGraphDef metadata)
{
Map<String, Shape> result = new HashMap<>();
SignatureDef servingDefault = getServingSignature(metadata);
for (Map.Entry<String, TensorInfo> entry : servingDefault.getInputsMap().entrySet())
{
TensorShapeProto shapeProto = entry.getValue().getTensorShape();
List<Dim> dimensions = shapeProto.getDimList();
long firstDimension = dimensions.get(0).getSize();
long[] remainingDimensions = dimensions.stream().skip(1).mapToLong(Dim::getSize).toArray();
Shape shape = Shape.make(firstDimension, remainingDimensions);
result.put(entry.getValue().getName(), shape);
}
return result;
}
}

There are two ways to get the names:
1) Using Java:
You can read the input and output names from the org.tensorflow.proto.framework.MetaGraphDef stored in saved model bundle.
Here is an example on how to extract the information:
https://github.com/awslabs/djl/blob/master/tensorflow/tensorflow-engine/src/main/java/ai/djl/tensorflow/engine/TfSymbolBlock.java#L149
2) Using python:
load the saved model in tensorflow python and print the names
loaded = tf.saved_model.load("path/to/model/")
print(list(loaded.signatures.keys()))
infer = loaded.signatures["serving_default"]
print(infer.structured_outputs)
I recommend to take a look at Deep Java Library, it automatically handle the input, output names.
It supports TensorFlow 2.1.0 and allows you to load Keras models as well as TF Hub Saved Model. Take a look at the documentation here and here
Feel free to open an issue if you have problem loading your model.

You can load TF model with Deep Java Library
System.setProperty("ai.djl.repository.zoo.location", "https://storage.googleapis.com/tfhub-modules/google/universal-sentence-encoder/1.tar.gz?artifact_id=encoder");
Criteria.Builder<NDList, NDList> builder =
Criteria.builder()
.setTypes(NDList.class, NDList.class)
.optArtifactId("ai.djl.localmodelzoo:encoder")
.build();
ZooModel<NDList, NDList> model = ModelZoo.loadModel(criteria);
See https://github.com/awslabs/djl/blob/master/docs/load_model.md#load-model-from-a-url for detail

I need to do the same, but seems still lots of missing pieces RE DJL usage. E.g., what to do after this?:
ZooModel<NDList, NDList> model = ModelZoo.loadModel(criteria);
I finally found an example in the DJL source code. The key take-away is to not use NDList for the input/output at all:
Criteria<String[], float[][]> criteria =
Criteria.builder()
.optApplication(Application.NLP.TEXT_EMBEDDING)
.setTypes(String[].class, float[][].class)
.optModelUrls(modelUrl)
.build();
try (ZooModel<String[], float[][]> model = ModelZoo.loadModel(criteria);
Predictor<String[], float[][]> predictor = model.newPredictor()) {
return predictor.predict(inputs.toArray(new String[0]));
}
See https://github.com/awslabs/djl/blob/master/examples/src/main/java/ai/djl/examples/inference/UniversalSentenceEncoder.java for the complete example.

Issues trying to execute Form Class in Netbeans

Good evening.
I have a problem using NetBeans v 11.1 for Windows.
I try to open two Frame Forms when the user select an item from a Combo Box.
The forms I want open have a string param in their constructors.
private void jComboBox1ActionPerformed(java.awt.event.ActionEvent evt) {
try {
// TODO add your handling code here:
Visualizza visualizza = new Visualizza(this.auth);
Inserisci inserisci = new Inserisci(this.auth);
var getItem = jComboBox1.getSelectedItem();
switch(getItem.toString()){
case ("Visualizza Iscritti nel Sistema"):
visualizza.setVisible(true);
break;
case ("Inserisci Atleti nel Sistema"):
inserisci.setVisible(true);
break;
default:
break;
}
} catch (IOException ex) {
Logger.getLogger(Select.class.getName()).log(Level.SEVERE, null, ex);
}
}
The constructors code is:
import java.io.IOException;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.swing.ButtonGroup;
import org.json.*;
public class Visualizza extends javax.swing.JFrame {
//Attributes declaration
...
public Visualizza(String auth) throws IOException {
initComponents();
btnRicerca.setVisible(false);
txtField.setVisible(false);
this.auth = auth;
Player[] player;
Requests r = new Requests("https://www.kingofthecage.it/API/getAllPlayers.php", auth);
r.sendGet();
if (r.res.equals("Errore") || r.res.equals("[]"))
{
List.removeAll();
List.add(errorMessage);
}
else
{
JSONArray arr = new JSONArray(r.res);
player = new Player[arr.length()];
String[] list = new String[arr.length()];
for (int i = 0; i < arr.length(); i++)
{
String id = arr.getJSONObject(i).getString("ID");
String name = arr.getJSONObject(i).getString("NOME");
String surname = arr.getJSONObject(i).getString("COGNOME");
String date = arr.getJSONObject(i).getString("DATA_NASCITA");
String birthplace = arr.getJSONObject(i).getString("LUOGO_NASCITA");
String residence = arr.getJSONObject(i).getString("RESIDENZA");
String cf = arr.getJSONObject(i).getString("CODICE_FISCALE");
String mail = arr.getJSONObject(i).getString("MAIL");
String mobile = arr.getJSONObject(i).getString("CELLULARE");
String team = arr.getJSONObject(i).getString("NOME_SQUADRA");
player[i] = new Player(id, name, surname, date, birthplace, residence, cf, mobile, mail, team);
List.add(list[i] = player[i].getPlayerString());
}
}
I import the library org.json to parse and interact with a response from a server, called with an HTTP Request and in the for cycle I returned params in a class called Player.
If I run the project from NetBeans works perfectly, the issue come when I build the project and I try to execute the .jar file saved in "./dist" folder. Substantially when I select a value from the ComboBox nothing happens but, as i've said, if I try to run the same code in NetBeans IDE it works.
I specify that I never change any setting from NetBeans IDE.
I hope you can solve my problem!

Get a part of a webpage using JSOUP

I am trying to programmatically search for a word meaning in google & save its meaning in a file in my computer. I have successfully called the page & get the response in Document (org.jsoup.nodes.Document). Now I do not know how to get only the word meaning from this Document. Please find the screenshot where I have indicated the part of data that I need.
The response html is so big that I can't understand from which element I will get my desired data. Please help. Here is what I have done so far:
public class Search {
private static Pattern patternDomainName;
private Matcher matcher;
private static final String DOMAIN_NAME_PATTERN
= "([a-zA-Z0-9]([a-zA-Z0-9\\-]{0,61}[a-zA-Z0-9])?\\.)+[a-zA-Z]{2,6}";
static {
patternDomainName = Pattern.compile(DOMAIN_NAME_PATTERN);
}
public static void main(String[] args) {
Search obj = new Search();
Set<String> result = obj.getDataFromGoogle("debug%20meaning");
for(String temp : result){
System.out.println(temp);
}
System.out.println(result.size());
}
public String getDomainName(String url){
String domainName = "";
matcher = patternDomainName.matcher(url);
if (matcher.find()) {
domainName = matcher.group(0).toLowerCase().trim();
}
return domainName;
}
private Set<String> getDataFromGoogle(String query) {
Set<String> result = new HashSet<String>();
String request = "https://www.google.com/search?q=" + query + "&num=20";
System.out.println("Sending request..." + request);
try {
// need http protocol, set this as a Google bot agent :)
Document doc = Jsoup
.connect(request)
.userAgent(
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)")
.timeout(5000).get();
/**********Here comes my data fetching logic*****************
* Dont know where to fing my desired data in such a big html
*/
/*
String sc = doc.html().replaceAll("\\n", "");
System.out.println(doc.html());
*/
} catch (IOException e) {
e.printStackTrace();
}
return result;
}
}

Google Dictionary API is deprecated!
But instead scraping through google search URI,which is what you are doing currently, you can do the same thing using this http://google-dictionary.so8848.com/ service which preferably more easy to scrape data from, with what you are doing currently.

Android - getting Google Places Photos and API calls

I wrote the method below in Spring to obtain a Google Places Photo this morning. The method is still buggy - 10 points for someone who can fix up the code - but it shows the gist of what I want to do:
#RequestMapping(method=RequestMethod.GET, value="/placedetails")
public BufferedImage PlaceDetails(#PathVariable String placeid) {
ArrayList<String> placePhotos = new ArrayList<>();
OkHttpClient client = new OkHttpClient();
Request request = new Request.Builder()
.url("https://maps.googleapis.com/maps/api/place/details/json?placeid="+placeid+"&key="+serverKey)
.build();
try {
//calling the GoogleAPI to get the PlaceDetails so that I can extract the photo_reference
Response response = client.newCall(request).execute();
//parsing the response with Jackson so that I can get the photo_reference
ObjectMapper m = new ObjectMapper();
JsonNode rootNode = m.readTree(response.body().string());
JsonNode resultNode = rootNode.get("result");
final JsonNode photoArrayNode = resultNode.get("photos");
if (photoArrayNode.isArray()) {
for (JsonNode photo: photoArrayNode) {
placePhotos.add(photo.get("photo_reference").textValue());
}
}
//calling the GoogleAPI again so that I can get the photoUrl
String photoUrl = String.format("https://maps.googleapis.com/maps/api/place/photo?maxwidth=%s&photoreference=%s&key=%s",
400,
placePhotos.get(0),
serverKey);
System.out.println(photoUrl);
//getting the actual photo
Request photoRequest = new Request.Builder().url(photoUrl).build();
Response photoResponse = client.newCall(photoRequest).execute();
if (!photoResponse.isSuccessful()) throw new IOException("Unexpected code " + response);
//returning the photo
return ImageIO.read(photoResponse.body().byteStream());
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return null;
}
I think to get an android app to display a Google Places picture, one would have to do the following:
Obtain the PlaceID first in Android. In my case, I obtained my PlaceID through an AutoCompleteTextView on my android app: (https://developers.google.com/places/android/autocomplete) (Call 1)
Then I call my method below. I call the Google Places API to get the Place Details (Call 2) and then once the details returns, I parse out the photo_reference using Jackson and call the Google Places API again to get the photo returned as a bitmap etc. (Call 3).
I'm making 3 calls to Google Places to return a Photo. When compared to the quota of 1000 calls a day, that is quite a significant amount of calls for getting 1 Photo.
Is there no other less way to get Photos without making so many calls?
I looked at this thread: How to get a picture of a place from google maps or places API
The person suggested that one uses panaramio instead which seems to be a really good option in the beginning but when I tested it out by typing in the example in my browser: http://www.panoramio.com/map/get_panoramas.php?set=public&from=0&to=20&minx=-33.868&miny=151.193&maxx=-33.864&maxy=151.197&size=medium&mapfilter=true, no photos were returned in the .php file.
I'm not sure if panaramio API still works?

Hi your problem is here
if (photoArrayNode.isArray()) {
for (JsonNode photo: photoArrayNode) {
placePhotos.add(photo.get("photo_reference").textValue());
}
Which should be
if (photoArrayNode.isArray()) {
for (JsonNode photo: photoArrayNode) {
placePhotos.add(photo.get("photo_reference").getString());
}
The photo_reference is a String value within the photo array element
Also, the below is unnecessary work:
//calling the GoogleAPI again so that I can get the photoUrl
String photoUrl = String.format("https://maps.googleapis.com/maps/api/place/photo?maxwidth=%s&photoreference=%s&key=%s",
There is no need to format the url string. The snippet below is part of the example I recommended below which answers your question specifically.
package in.wptrafficanalyzer.locationnearbyplacesphotos;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
import android.util.Log;
public class PlaceJSONParser {
/** Receives a JSONObject and returns a list */
public Place[] parse(JSONObject jObject){
JSONArray jPlaces = null;
try {
/** Retrieves all the elements in the 'places' array */
jPlaces = jObject.getJSONArray("results");
} catch (JSONException e) {
e.printStackTrace();
}
/** Invoking getPlaces with the array of json object
* where each json object represent a place
*/
return getPlaces(jPlaces);
}
private Place[] getPlaces(JSONArray jPlaces){
int placesCount = jPlaces.length();
Place[] places = new Place[placesCount];
/** Taking each place, parses and adds to list object */
for(int i=0; i<placesCount;i++){
try {
/** Call getPlace with place JSON object to parse the place */
places[i] = getPlace((JSONObject)jPlaces.get(i));
} catch (JSONException e) {
e.printStackTrace();
}
}
return places;
}
/** Parsing the Place JSON object */
private Place getPlace(JSONObject jPlace){
Place place = new Place();
try {
// Extracting Place name, if available
if(!jPlace.isNull("name")){
place.mPlaceName = jPlace.getString("name");
}
// Extracting Place Vicinity, if available
if(!jPlace.isNull("vicinity")){
place.mVicinity = jPlace.getString("vicinity");
}
if(!jPlace.isNull("photos")){
JSONArray photos = jPlace.getJSONArray("photos");
place.mPhotos = new Photo[photos.length()];
for(int i=0;i<photos.length();i++){
place.mPhotos[i] = new Photo();
place.mPhotos[i].mWidth = ((JSONObject)photos.get(i)).getInt("width");
place.mPhotos[i].mHeight = ((JSONObject)photos.get(i)).getInt("height");
place.mPhotos[i].mPhotoReference = ((JSONObject)photos.get(i)).getString("photo_reference");
JSONArray attributions = ((JSONObject)photos.get(i)).getJSONArray("html_attributions");
place.mPhotos[i].mAttributions = new Attribution[attributions.length()];
for(int j=0;j<attributions.length();j++){
place.mPhotos[i].mAttributions[j] = new Attribution();
place.mPhotos[i].mAttributions[j].mHtmlAttribution = attributions.getString(j);
}
}
}
place.mLat = jPlace.getJSONObject("geometry").getJSONObject("location").getString("lat");
place.mLng = jPlace.getJSONObject("geometry").getJSONObject("location").getString("lng");
} catch (JSONException e) {
e.printStackTrace();
Log.d("EXCEPTION", e.toString());
}
return place;
}
}
For a complete example please see: the source code is available for download.
http://wptrafficanalyzer.in/blog/showing-nearby-places-with-photos-at-any-location-in-google-maps-android-api-v2/

How to implement auto suggest using Lucene's new AnalyzingInfixSuggester API?

I am a greenhand on Lucene, and I want to implement auto suggest, just like google, when I input a character like 'G', it would give me a list, you can try your self.
I have searched on the whole net.
Nobody has done this , and it gives us some new tools in package suggest
But i need an example to tell me how to do that
Is there anyone can help ?

I'll give you a pretty complete example that shows you how to use AnalyzingInfixSuggester. In this example we'll pretend that we're Amazon, and we want to autocomplete a product search field. We'll take advantage of features of the Lucene suggestion system to implement the following:
Ranked results: We will suggest the most popular matching products first.
Region-restricted results: We will only suggest products that we sell in the customer's country.
Product photos: We will store product photo URLs in the suggestion index so we can display them in the search results, without having to do an additional database lookup.
First I'll define a simple class to hold information about a product in Product.java:
import java.util.Set;
class Product implements java.io.Serializable
{
String name;
String image;
String[] regions;
int numberSold;
public Product(String name, String image, String[] regions,
int numberSold) {
this.name = name;
this.image = image;
this.regions = regions;
this.numberSold = numberSold;
}
}
To index records in with the AnalyzingInfixSuggester's build method you need to pass it an object that implements the org.apache.lucene.search.suggest.InputIterator interface. An InputIterator gives access to the key, contexts, payload and weight for each record.
The key is the text you actually want to search on and autocomplete against. In our example, it will be the name of the product.
The contexts are a set of additional, arbitrary data that you can use to filter records against. In our example, the contexts are the set of ISO codes for the countries we will ship a particular product to.
The payload is additional arbitrary data you want to store in the index for the record. In this example, we will actually serialize each Product instance and store the resulting bytes as the payload. Then when we later do lookups, we can deserialize the payload and access information in the product instance like the image URL.
The weight is used to order suggestion results; results with a higher weight are returned first. We'll use the number of sales for a given product as its weight.
Here's the contents of ProductIterator.java:
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.ObjectOutputStream;
import java.io.UnsupportedEncodingException;
import java.util.Comparator;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Set;
import org.apache.lucene.search.suggest.InputIterator;
import org.apache.lucene.util.BytesRef;
class ProductIterator implements InputIterator
{
private Iterator<Product> productIterator;
private Product currentProduct;
ProductIterator(Iterator<Product> productIterator) {
this.productIterator = productIterator;
}
public boolean hasContexts() {
return true;
}
public boolean hasPayloads() {
return true;
}
public Comparator<BytesRef> getComparator() {
return null;
}
// This method needs to return the key for the record; this is the
// text we'll be autocompleting against.
public BytesRef next() {
if (productIterator.hasNext()) {
currentProduct = productIterator.next();
try {
return new BytesRef(currentProduct.name.getBytes("UTF8"));
} catch (UnsupportedEncodingException e) {
throw new Error("Couldn't convert to UTF-8");
}
} else {
return null;
}
}
// This method returns the payload for the record, which is
// additional data that can be associated with a record and
// returned when we do suggestion lookups. In this example the
// payload is a serialized Java object representing our product.
public BytesRef payload() {
try {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream out = new ObjectOutputStream(bos);
out.writeObject(currentProduct);
out.close();
return new BytesRef(bos.toByteArray());
} catch (IOException e) {
throw new Error("Well that's unfortunate.");
}
}
// This method returns the contexts for the record, which we can
// use to restrict suggestions. In this example we use the
// regions in which a product is sold.
public Set<BytesRef> contexts() {
try {
Set<BytesRef> regions = new HashSet();
for (String region : currentProduct.regions) {
regions.add(new BytesRef(region.getBytes("UTF8")));
}
return regions;
} catch (UnsupportedEncodingException e) {
throw new Error("Couldn't convert to UTF-8");
}
}
// This method helps us order our suggestions. In this example we
// use the number of products of this type that we've sold.
public long weight() {
return currentProduct.numberSold;
}
}
In our driver program, we will do the following things:
Create an index directory in RAM.
Create a StandardTokenizer.
Create an AnalyzingInfixSuggester using the RAM directory and tokenizer.
Index a number of products using ProductIterator.
Print the results of some sample lookups.
Here's the driver program, SuggestProducts.java:
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.UnsupportedEncodingException;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester;
import org.apache.lucene.search.suggest.Lookup;
import org.apache.lucene.store.RAMDirectory;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.Version;
public class SuggestProducts
{
// Get suggestions given a prefix and a region.
private static void lookup(AnalyzingInfixSuggester suggester, String name,
String region) {
try {
List<Lookup.LookupResult> results;
HashSet<BytesRef> contexts = new HashSet<BytesRef>();
contexts.add(new BytesRef(region.getBytes("UTF8")));
// Do the actual lookup. We ask for the top 2 results.
results = suggester.lookup(name, contexts, 2, true, false);
System.out.println("-- \"" + name + "\" (" + region + "):");
for (Lookup.LookupResult result : results) {
System.out.println(result.key);
Product p = getProduct(result);
if (p != null) {
System.out.println(" image: " + p.image);
System.out.println(" # sold: " + p.numberSold);
}
}
} catch (IOException e) {
System.err.println("Error");
}
}
// Deserialize a Product from a LookupResult payload.
private static Product getProduct(Lookup.LookupResult result)
{
try {
BytesRef payload = result.payload;
if (payload != null) {
ByteArrayInputStream bis = new ByteArrayInputStream(payload.bytes);
ObjectInputStream in = new ObjectInputStream(bis);
Product p = (Product) in.readObject();
return p;
} else {
return null;
}
} catch (IOException|ClassNotFoundException e) {
throw new Error("Could not decode payload :(");
}
}
public static void main(String[] args) {
try {
RAMDirectory index_dir = new RAMDirectory();
StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_48);
AnalyzingInfixSuggester suggester = new AnalyzingInfixSuggester(
Version.LUCENE_48, index_dir, analyzer);
// Create our list of products.
ArrayList<Product> products = new ArrayList<Product>();
products.add(
new Product(
"Electric Guitar",
"http://images.example/electric-guitar.jpg",
new String[]{"US", "CA"},
100));
products.add(
new Product(
"Electric Train",
"http://images.example/train.jpg",
new String[]{"US", "CA"},
100));
products.add(
new Product(
"Acoustic Guitar",
"http://images.example/acoustic-guitar.jpg",
new String[]{"US", "ZA"},
80));
products.add(
new Product(
"Guarana Soda",
"http://images.example/soda.jpg",
new String[]{"ZA", "IE"},
130));
// Index the products with the suggester.
suggester.build(new ProductIterator(products.iterator()));
// Do some example lookups.
lookup(suggester, "Gu", "US");
lookup(suggester, "Gu", "ZA");
lookup(suggester, "Gui", "CA");
lookup(suggester, "Electric guit", "US");
} catch (IOException e) {
System.err.println("Error!");
}
}
}
And here is the output from the driver program:
-- "Gu" (US):
Electric Guitar
image: http://images.example/electric-guitar.jpg
# sold: 100
Acoustic Guitar
image: http://images.example/acoustic-guitar.jpg
# sold: 80
-- "Gu" (ZA):
Guarana Soda
image: http://images.example/soda.jpg
# sold: 130
Acoustic Guitar
image: http://images.example/acoustic-guitar.jpg
# sold: 80
-- "Gui" (CA):
Electric Guitar
image: http://images.example/electric-guitar.jpg
# sold: 100
-- "Electric guit" (US):
Electric Guitar
image: http://images.example/electric-guitar.jpg
# sold: 100
Appendix
There's a way to avoid writing a full InputIterator that you might find easier. You can write a stub InputIterator that returns null from its next, payload and contexts methods. Pass an instance of it to AnalyzingInfixSuggester's build method:
suggester.build(new ProductIterator(new ArrayList<Product>().iterator()));
Then for each item you want to index, call the AnalyzingInfixSuggester add method:
suggester.add(text, contexts, weight, payload)
After you've indexed everything, call refresh:
suggester.refresh();
If you're indexing large amounts of data, it's possible to significantly speedup indexing using this method with multiple threads: Call build, then use multiple threads to add items, then finally call refresh.
[Edited 2015-04-23 to demonstrate deserializing info from the LookupResult payload.]