I am a greenhand on Lucene, and I want to implement auto suggest, just like google, when I input a character like 'G', it would give me a list, you can try your self.
I have searched on the whole net.
Nobody has done this , and it gives us some new tools in package suggest
But i need an example to tell me how to do that
Is there anyone can help ?
I'll give you a pretty complete example that shows you how to use AnalyzingInfixSuggester. In this example we'll pretend that we're Amazon, and we want to autocomplete a product search field. We'll take advantage of features of the Lucene suggestion system to implement the following:
Ranked results: We will suggest the most popular matching products first.
Region-restricted results: We will only suggest products that we sell in the customer's country.
Product photos: We will store product photo URLs in the suggestion index so we can display them in the search results, without having to do an additional database lookup.
First I'll define a simple class to hold information about a product in Product.java:
import java.util.Set;
class Product implements java.io.Serializable
{
String name;
String image;
String[] regions;
int numberSold;
public Product(String name, String image, String[] regions,
int numberSold) {
this.name = name;
this.image = image;
this.regions = regions;
this.numberSold = numberSold;
}
}
To index records in with the AnalyzingInfixSuggester's build method you need to pass it an object that implements the org.apache.lucene.search.suggest.InputIterator interface. An InputIterator gives access to the key, contexts, payload and weight for each record.
The key is the text you actually want to search on and autocomplete against. In our example, it will be the name of the product.
The contexts are a set of additional, arbitrary data that you can use to filter records against. In our example, the contexts are the set of ISO codes for the countries we will ship a particular product to.
The payload is additional arbitrary data you want to store in the index for the record. In this example, we will actually serialize each Product instance and store the resulting bytes as the payload. Then when we later do lookups, we can deserialize the payload and access information in the product instance like the image URL.
The weight is used to order suggestion results; results with a higher weight are returned first. We'll use the number of sales for a given product as its weight.
Here's the contents of ProductIterator.java:
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.ObjectOutputStream;
import java.io.UnsupportedEncodingException;
import java.util.Comparator;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Set;
import org.apache.lucene.search.suggest.InputIterator;
import org.apache.lucene.util.BytesRef;
class ProductIterator implements InputIterator
{
private Iterator<Product> productIterator;
private Product currentProduct;
ProductIterator(Iterator<Product> productIterator) {
this.productIterator = productIterator;
}
public boolean hasContexts() {
return true;
}
public boolean hasPayloads() {
return true;
}
public Comparator<BytesRef> getComparator() {
return null;
}
// This method needs to return the key for the record; this is the
// text we'll be autocompleting against.
public BytesRef next() {
if (productIterator.hasNext()) {
currentProduct = productIterator.next();
try {
return new BytesRef(currentProduct.name.getBytes("UTF8"));
} catch (UnsupportedEncodingException e) {
throw new Error("Couldn't convert to UTF-8");
}
} else {
return null;
}
}
// This method returns the payload for the record, which is
// additional data that can be associated with a record and
// returned when we do suggestion lookups. In this example the
// payload is a serialized Java object representing our product.
public BytesRef payload() {
try {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream out = new ObjectOutputStream(bos);
out.writeObject(currentProduct);
out.close();
return new BytesRef(bos.toByteArray());
} catch (IOException e) {
throw new Error("Well that's unfortunate.");
}
}
// This method returns the contexts for the record, which we can
// use to restrict suggestions. In this example we use the
// regions in which a product is sold.
public Set<BytesRef> contexts() {
try {
Set<BytesRef> regions = new HashSet();
for (String region : currentProduct.regions) {
regions.add(new BytesRef(region.getBytes("UTF8")));
}
return regions;
} catch (UnsupportedEncodingException e) {
throw new Error("Couldn't convert to UTF-8");
}
}
// This method helps us order our suggestions. In this example we
// use the number of products of this type that we've sold.
public long weight() {
return currentProduct.numberSold;
}
}
In our driver program, we will do the following things:
Create an index directory in RAM.
Create a StandardTokenizer.
Create an AnalyzingInfixSuggester using the RAM directory and tokenizer.
Index a number of products using ProductIterator.
Print the results of some sample lookups.
Here's the driver program, SuggestProducts.java:
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.UnsupportedEncodingException;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester;
import org.apache.lucene.search.suggest.Lookup;
import org.apache.lucene.store.RAMDirectory;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.Version;
public class SuggestProducts
{
// Get suggestions given a prefix and a region.
private static void lookup(AnalyzingInfixSuggester suggester, String name,
String region) {
try {
List<Lookup.LookupResult> results;
HashSet<BytesRef> contexts = new HashSet<BytesRef>();
contexts.add(new BytesRef(region.getBytes("UTF8")));
// Do the actual lookup. We ask for the top 2 results.
results = suggester.lookup(name, contexts, 2, true, false);
System.out.println("-- \"" + name + "\" (" + region + "):");
for (Lookup.LookupResult result : results) {
System.out.println(result.key);
Product p = getProduct(result);
if (p != null) {
System.out.println(" image: " + p.image);
System.out.println(" # sold: " + p.numberSold);
}
}
} catch (IOException e) {
System.err.println("Error");
}
}
// Deserialize a Product from a LookupResult payload.
private static Product getProduct(Lookup.LookupResult result)
{
try {
BytesRef payload = result.payload;
if (payload != null) {
ByteArrayInputStream bis = new ByteArrayInputStream(payload.bytes);
ObjectInputStream in = new ObjectInputStream(bis);
Product p = (Product) in.readObject();
return p;
} else {
return null;
}
} catch (IOException|ClassNotFoundException e) {
throw new Error("Could not decode payload :(");
}
}
public static void main(String[] args) {
try {
RAMDirectory index_dir = new RAMDirectory();
StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_48);
AnalyzingInfixSuggester suggester = new AnalyzingInfixSuggester(
Version.LUCENE_48, index_dir, analyzer);
// Create our list of products.
ArrayList<Product> products = new ArrayList<Product>();
products.add(
new Product(
"Electric Guitar",
"http://images.example/electric-guitar.jpg",
new String[]{"US", "CA"},
100));
products.add(
new Product(
"Electric Train",
"http://images.example/train.jpg",
new String[]{"US", "CA"},
100));
products.add(
new Product(
"Acoustic Guitar",
"http://images.example/acoustic-guitar.jpg",
new String[]{"US", "ZA"},
80));
products.add(
new Product(
"Guarana Soda",
"http://images.example/soda.jpg",
new String[]{"ZA", "IE"},
130));
// Index the products with the suggester.
suggester.build(new ProductIterator(products.iterator()));
// Do some example lookups.
lookup(suggester, "Gu", "US");
lookup(suggester, "Gu", "ZA");
lookup(suggester, "Gui", "CA");
lookup(suggester, "Electric guit", "US");
} catch (IOException e) {
System.err.println("Error!");
}
}
}
And here is the output from the driver program:
-- "Gu" (US):
Electric Guitar
image: http://images.example/electric-guitar.jpg
# sold: 100
Acoustic Guitar
image: http://images.example/acoustic-guitar.jpg
# sold: 80
-- "Gu" (ZA):
Guarana Soda
image: http://images.example/soda.jpg
# sold: 130
Acoustic Guitar
image: http://images.example/acoustic-guitar.jpg
# sold: 80
-- "Gui" (CA):
Electric Guitar
image: http://images.example/electric-guitar.jpg
# sold: 100
-- "Electric guit" (US):
Electric Guitar
image: http://images.example/electric-guitar.jpg
# sold: 100
Appendix
There's a way to avoid writing a full InputIterator that you might find easier. You can write a stub InputIterator that returns null from its next, payload and contexts methods. Pass an instance of it to AnalyzingInfixSuggester's build method:
suggester.build(new ProductIterator(new ArrayList<Product>().iterator()));
Then for each item you want to index, call the AnalyzingInfixSuggester add method:
suggester.add(text, contexts, weight, payload)
After you've indexed everything, call refresh:
suggester.refresh();
If you're indexing large amounts of data, it's possible to significantly speedup indexing using this method with multiple threads: Call build, then use multiple threads to add items, then finally call refresh.
[Edited 2015-04-23 to demonstrate deserializing info from the LookupResult payload.]
Related
I am attempting to query a table that has a partition key and sort key (however the partition key and sort key are 1:1 and I want to query only using the partition key [in which only one item would be returned]).
QueryRequest query = new QueryRequest()
.withTableName(TABLE_NAME)
.withKeyConditionExpression("testId = :" + "1234567890");
QueryResult result = client.query(query);
This is the code I tried but it did not work (testId is the partition key name and 1234567890 is the partition key value in String form); do y'all know of a method I could use to query by only using the partition key keeping in mind that only one item will be returned since the partition key and sort key are 1:1? Thank you so much in advance. [This is my first Stack Overflow post - my apologies if I worded things poorly, I'm happy to answer any questions about my wording]
FYI: this is the error statement I got when trying to use the code above:
errorMessage": "Invalid KeyConditionExpression: An expression attribute value used in expression is not defined
You should really update to use AWS SDK For Java V2 (Using AWS SDK for V1 is not best practice). Using AWS SDK for Java v2 is best practice for using Amazon DynamoDB API.
To learn more about the AWS V2 Java API, read the Developer Guide here.
Developer guide - AWS SDK for Java 2.x
Now I will answer this question with V2. The solution that worked for me was create a secondary index named year-index. This uses just my partition key named year (and does not use the sort key).
I can successfully query using this index, as shown here that uses the AWS Management Console.
Now only movies with the year 2014 are returned. That is how you query when your table has a composite key made up of a partition key and sort key and you only want to query on partition key.
By the way - you said you have a secondary index. A table can have more then 1 secondary index
Code that you need for V2 to query a secondary Index
I will show you how to use V2 to search for secondary index using three ways.
First way - Use the V2 Enhanced Client
Once you create the secondary index, you can use it to query. As mentioned, I created a secondary index named year-index. I can use this secondary index to query data by using the DynamoDB Enhanced Client.
Because, I am querying the Movies table, I have to create a Class named Movies like this. Notice the use of the #DynamoDbSecondaryPartitionKey annotation.
package com.example.dynamodb;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbBean;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbPartitionKey;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbSecondaryPartitionKey;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbSortKey;
#DynamoDbBean
public class Movies {
private String title;
private int year;
private String info;
#DynamoDbSecondaryPartitionKey(indexNames = { "year-index" })
#DynamoDbPartitionKey
public int getYear() {
return this.year;
}
public void setYear(int year) {
this.year = year;
}
#DynamoDbSortKey
public String getTitle() {
return this.title;
}
public void setTitle(String title) {
this.title = title;
}
public String getInfo() {
return this.info;
}
public void setInfo(String info) {
this.info = info;
}
}
Finally, here is the V2 code that lets you query using the secondary index.
package com.example.dynamodb;
// snippet-start:[dynamodb.java2.get_item_index.import]
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.core.pagination.sync.SdkIterable;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbIndex;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable;
import software.amazon.awssdk.enhanced.dynamodb.Key;
import software.amazon.awssdk.enhanced.dynamodb.TableSchema;
import software.amazon.awssdk.enhanced.dynamodb.model.Page;
import software.amazon.awssdk.enhanced.dynamodb.model.QueryConditional;
import software.amazon.awssdk.enhanced.dynamodb.model.QueryEnhancedRequest;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
import java.util.List;
// snippet-end:[dynamodb.java2.get_item_index.import]
/**
* Before running this Java V2 code example, set up your development environment, including your credentials.
*
* For more information, see the following documentation topic:
*
* https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html
*
* To get an item from an Amazon DynamoDB table using the AWS SDK for Java V2, its better practice to use the
* Enhanced Client, see the EnhancedGetItem example.
*
* Create the Movies table by running the Scenario example and loading the Movies data from the JSON file. Next create a secondary
* index for the Movies table that uses only the year column. Name the index **year-index**. For more information, see:
*
* https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GSI.html
*/
public class EnhancedGetItemUsingIndex {
public static void main(String[] args) {
String tableName = "Movies" ; //args[0];
ProfileCredentialsProvider credentialsProvider = ProfileCredentialsProvider.create();
Region region = Region.US_EAST_1;
DynamoDbClient ddb = DynamoDbClient.builder()
.credentialsProvider(credentialsProvider)
.region(region)
.build();
queryIndex(ddb, tableName);
ddb.close();
}
// snippet-start:[dynamodb.java2.get_item_index.main]
public static void queryIndex(DynamoDbClient ddb, String tableName) {
try {
// Create a DynamoDbEnhancedClient and use the DynamoDbClient object.
DynamoDbEnhancedClient enhancedClient = DynamoDbEnhancedClient.builder()
.dynamoDbClient(ddb)
.build();
//Create a DynamoDbTable object based on Movies.
DynamoDbTable<Movies> table = enhancedClient.table("Movies", TableSchema.fromBean(Movies.class));
String dateVal = "2013";
DynamoDbIndex<Movies> secIndex =
enhancedClient.table("Movies", TableSchema.fromBean(Movies.class))
.index("year-index");
AttributeValue attVal = AttributeValue.builder()
.n(dateVal)
.build();
// Create a QueryConditional object that's used in the query operation.
QueryConditional queryConditional = QueryConditional
.keyEqualTo(Key.builder().partitionValue(attVal)
.build());
// Get items in the table.
SdkIterable<Page<Movies>> results = secIndex.query(
QueryEnhancedRequest.builder()
.queryConditional(queryConditional)
.limit(300)
.build());
//Display the results.
results.forEach(page -> {
List<Movies> allMovies = page.items();
for (Movies myMovies: allMovies) {
System.out.println("The movie title is " + myMovies.getTitle() + ". The year is " + myMovies.getYear());
}
});
} catch (DynamoDbException e) {
System.err.println(e.getMessage());
System.exit(1);
}
}
// snippet-end:[dynamodb.java2.get_item_index.main]
}
This now returns all Movies where the year is 2013.
Second way - Use the V2 Service Client
package com.example.dynamodb;
// snippet-start:[dynamodb.java2.query_items_sec_index.import]
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
import software.amazon.awssdk.services.dynamodb.model.QueryRequest;
import software.amazon.awssdk.services.dynamodb.model.QueryResponse;
import java.util.HashMap;
import java.util.Map;
// snippet-end:[dynamodb.java2.query_items_sec_index.import]
/**
* Before running this Java V2 code example, set up your development environment, including your credentials.
*
* For more information, see the following documentation topic:
*
* https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html
*
* Create the Movies table by running the Scenario example and loading the Movies data from the JSON file. Next create a secondary
* index for the Movies table that uses only the year column. Name the index **year-index**. For more information, see:
*
* https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GSI.html
*/
public class QueryItemsUsingIndex {
public static void main(String[] args) {
String tableName = "Movies" ; //args[0];
ProfileCredentialsProvider credentialsProvider = ProfileCredentialsProvider.create();
Region region = Region.US_EAST_1;
DynamoDbClient ddb = DynamoDbClient.builder()
.credentialsProvider(credentialsProvider)
.region(region)
.build();
queryIndex(ddb, tableName);
ddb.close();
}
// snippet-start:[dynamodb.java2.query_items_sec_index.main]
public static void queryIndex(DynamoDbClient ddb, String tableName) {
try {
Map<String,String> expressionAttributesNames = new HashMap<>();
expressionAttributesNames.put("#year","year");
Map<String, AttributeValue> expressionAttributeValues = new HashMap<>();
expressionAttributeValues.put(":yearValue", AttributeValue.builder().n("2013").build());
QueryRequest request = QueryRequest.builder()
.tableName(tableName)
.indexName("year-index")
.keyConditionExpression("#year = :yearValue")
.expressionAttributeNames(expressionAttributesNames)
.expressionAttributeValues(expressionAttributeValues)
.build();
System.out.println("=== Movie Titles ===");
QueryResponse response = ddb.query(request);
response.items()
.forEach(movie-> System.out.println(movie.get("title").s()));
} catch (DynamoDbException e) {
System.err.println(e.getMessage());
System.exit(1);
}
}
// snippet-end:[dynamodb.java2.query_items_sec_index.main]
}
**Third way - Use PartQL
Of course, you can query the partition key using PartiQL. For example.
public static void queryTable(DynamoDbClient ddb) {
String sqlStatement = "SELECT * FROM MoviesPartiQ where year = ? ORDER BY info";
try {
List<AttributeValue> parameters = new ArrayList<>();
AttributeValue att1 = AttributeValue.builder()
.n(String.valueOf("2013"))
.build();
parameters.add(att1);
// Get items in the table and write out the ID value.
ExecuteStatementResponse response = executeStatementRequest(ddb, sqlStatement, parameters);
System.out.println("ExecuteStatement successful: "+ response.toString());
} catch (DynamoDbException e) {
System.err.println(e.getMessage());
System.exit(1);
}
}
The following python code passes ["hello", "world"] into the universal sentence encoder and returns an array of floats denoting their encoded representation.
import tensorflow as tf
import tensorflow_hub as hub
module = hub.KerasLayer("https://tfhub.dev/google/universal-sentence-encoder/4")
model = tf.keras.Sequential(module)
print("model: ", model(["hello", "world"]))
This code works but I'd now like to do the same thing using the Java API. I've successfully loaded the module, but I am unable to pass inputs into the model and extract the output. Here is what I've got so far:
import org.tensorflow.Graph;
import org.tensorflow.SavedModelBundle;
import org.tensorflow.Session;
import org.tensorflow.Tensor;
import org.tensorflow.Tensors;
import org.tensorflow.framework.ConfigProto;
import org.tensorflow.framework.GPUOptions;
import org.tensorflow.framework.GraphDef;
import org.tensorflow.framework.MetaGraphDef;
import org.tensorflow.framework.NodeDef;
import org.tensorflow.util.SaverDef;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
public final class NaiveBayesClassifier
{
public static void main(String[] args)
{
new NaiveBayesClassifier().run();
}
protected SavedModelBundle loadModule(Path source, String... tags) throws IOException
{
return SavedModelBundle.load(source.toAbsolutePath().normalize().toString(), tags);
}
public void run()
{
try (SavedModelBundle module = loadModule(Paths.get("universal-sentence-encoder"), "serve"))
{
Graph graph = module.graph();
try (Session session = new Session(graph, ConfigProto.newBuilder().
setGpuOptions(GPUOptions.newBuilder().setAllowGrowth(true)).
setAllowSoftPlacement(true).
build().toByteArray()))
{
Tensor<String> input = Tensors.create(new byte[][]
{
"hello".getBytes(StandardCharsets.UTF_8),
"world".getBytes(StandardCharsets.UTF_8)
});
List<Tensor<?>> result = session.runner().feed("serving_default_inputs", input).
addTarget("???").run();
}
}
catch (IOException e)
{
e.printStackTrace();
}
}
}
I used https://stackoverflow.com/a/51952478/14731 to scan the model for possible input/output nodes. I believe the input node is "serving_default_inputs" but I can't figure out the output node. More importantly, I don't have to specify any of these values when invoking the code in python through Keras so is there a way to do the same using the Java API?
UPDATE: Thanks to roywei I can now that confirm the input node is serving_default_input and output node is StatefulPartitionedCall_1 but when I plug these names into the aforementioned code I get:
2020-05-22 22:13:52.266287: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at lookup_table_op.cc:809 : Failed precondition: Table not initialized.
Exception in thread "main" java.lang.IllegalStateException: [_Derived_]{{function_node __inference_pruned_6741}} {{function_node __inference_pruned_6741}} Error while reading resource variable EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_25 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/EncoderDNN/DNN/ResidualHidden_0/dense/kernel/part_25/class tensorflow::Var does not exist.
[[{{node EncoderDNN/DNN/ResidualHidden_0/dense/kernel/ConcatPartitions/concat/ReadVariableOp_25}}]]
[[StatefulPartitionedCall_1/StatefulPartitionedCall]]
at libtensorflow#1.15.0/org.tensorflow.Session.run(Native Method)
at libtensorflow#1.15.0/org.tensorflow.Session.access$100(Session.java:48)
at libtensorflow#1.15.0/org.tensorflow.Session$Runner.runHelper(Session.java:326)
at libtensorflow#1.15.0/org.tensorflow.Session$Runner.run(Session.java:276)
Meaning, I still cannot invoke the model. What am I missing?
I figured it out after roywei pointed me in the right direction.
I needed to use SavedModuleBundle.session() instead of constructing my own instance. This is because the loader initializes the graph variables.
Instead of passing a ConfigProto to the Session constructor, I passed it into the SavedModelBundle loader instead.
I needed to use fetch() instead of addTarget() to retrieve the output tensor.
Here is the working code:
public final class NaiveBayesClassifier
{
public static void main(String[] args)
{
new NaiveBayesClassifier().run();
}
public void run()
{
try (SavedModelBundle module = loadModule(Paths.get("universal-sentence-encoder"), "serve"))
{
try (Tensor<String> input = Tensors.create(new byte[][]
{
"hello".getBytes(StandardCharsets.UTF_8),
"world".getBytes(StandardCharsets.UTF_8)
}))
{
MetaGraphDef metadata = MetaGraphDef.parseFrom(module.metaGraphDef());
Map<String, Shape> nameToInput = getInputToShape(metadata);
String firstInput = nameToInput.keySet().iterator().next();
Map<String, Shape> nameToOutput = getOutputToShape(metadata);
String firstOutput = nameToOutput.keySet().iterator().next();
System.out.println("input: " + firstInput);
System.out.println("output: " + firstOutput);
System.out.println();
List<Tensor<?>> result = module.session().runner().feed(firstInput, input).
fetch(firstOutput).run();
for (Tensor<?> tensor : result)
{
{
float[][] array = new float[tensor.numDimensions()][tensor.numElements() /
tensor.numDimensions()];
tensor.copyTo(array);
System.out.println(Arrays.deepToString(array));
}
}
}
}
catch (IOException e)
{
e.printStackTrace();
}
}
/**
* Loads a graph from a file.
*
* #param source the directory containing to load from
* #param tags the model variant(s) to load
* #return the graph
* #throws NullPointerException if any of the arguments are null
* #throws IOException if an error occurs while reading the file
*/
protected SavedModelBundle loadModule(Path source, String... tags) throws IOException
{
// https://stackoverflow.com/a/43526228/14731
try
{
return SavedModelBundle.loader(source.toAbsolutePath().normalize().toString()).
withTags(tags).
withConfigProto(ConfigProto.newBuilder().
setGpuOptions(GPUOptions.newBuilder().setAllowGrowth(true)).
setAllowSoftPlacement(true).
build().toByteArray()).
load();
}
catch (TensorFlowException e)
{
throw new IOException(e);
}
}
/**
* #param metadata the graph metadata
* #return the first signature, or null
*/
private SignatureDef getFirstSignature(MetaGraphDef metadata)
{
Map<String, SignatureDef> nameToSignature = metadata.getSignatureDefMap();
if (nameToSignature.isEmpty())
return null;
return nameToSignature.get(nameToSignature.keySet().iterator().next());
}
/**
* #param metadata the graph metadata
* #return the output signature
*/
private SignatureDef getServingSignature(MetaGraphDef metadata)
{
return metadata.getSignatureDefOrDefault("serving_default", getFirstSignature(metadata));
}
/**
* #param metadata the graph metadata
* #return a map from an output name to its shape
*/
protected Map<String, Shape> getOutputToShape(MetaGraphDef metadata)
{
Map<String, Shape> result = new HashMap<>();
SignatureDef servingDefault = getServingSignature(metadata);
for (Map.Entry<String, TensorInfo> entry : servingDefault.getOutputsMap().entrySet())
{
TensorShapeProto shapeProto = entry.getValue().getTensorShape();
List<Dim> dimensions = shapeProto.getDimList();
long firstDimension = dimensions.get(0).getSize();
long[] remainingDimensions = dimensions.stream().skip(1).mapToLong(Dim::getSize).toArray();
Shape shape = Shape.make(firstDimension, remainingDimensions);
result.put(entry.getValue().getName(), shape);
}
return result;
}
/**
* #param metadata the graph metadata
* #return a map from an input name to its shape
*/
protected Map<String, Shape> getInputToShape(MetaGraphDef metadata)
{
Map<String, Shape> result = new HashMap<>();
SignatureDef servingDefault = getServingSignature(metadata);
for (Map.Entry<String, TensorInfo> entry : servingDefault.getInputsMap().entrySet())
{
TensorShapeProto shapeProto = entry.getValue().getTensorShape();
List<Dim> dimensions = shapeProto.getDimList();
long firstDimension = dimensions.get(0).getSize();
long[] remainingDimensions = dimensions.stream().skip(1).mapToLong(Dim::getSize).toArray();
Shape shape = Shape.make(firstDimension, remainingDimensions);
result.put(entry.getValue().getName(), shape);
}
return result;
}
}
There are two ways to get the names:
1) Using Java:
You can read the input and output names from the org.tensorflow.proto.framework.MetaGraphDef stored in saved model bundle.
Here is an example on how to extract the information:
https://github.com/awslabs/djl/blob/master/tensorflow/tensorflow-engine/src/main/java/ai/djl/tensorflow/engine/TfSymbolBlock.java#L149
2) Using python:
load the saved model in tensorflow python and print the names
loaded = tf.saved_model.load("path/to/model/")
print(list(loaded.signatures.keys()))
infer = loaded.signatures["serving_default"]
print(infer.structured_outputs)
I recommend to take a look at Deep Java Library, it automatically handle the input, output names.
It supports TensorFlow 2.1.0 and allows you to load Keras models as well as TF Hub Saved Model. Take a look at the documentation here and here
Feel free to open an issue if you have problem loading your model.
You can load TF model with Deep Java Library
System.setProperty("ai.djl.repository.zoo.location", "https://storage.googleapis.com/tfhub-modules/google/universal-sentence-encoder/1.tar.gz?artifact_id=encoder");
Criteria.Builder<NDList, NDList> builder =
Criteria.builder()
.setTypes(NDList.class, NDList.class)
.optArtifactId("ai.djl.localmodelzoo:encoder")
.build();
ZooModel<NDList, NDList> model = ModelZoo.loadModel(criteria);
See https://github.com/awslabs/djl/blob/master/docs/load_model.md#load-model-from-a-url for detail
I need to do the same, but seems still lots of missing pieces RE DJL usage. E.g., what to do after this?:
ZooModel<NDList, NDList> model = ModelZoo.loadModel(criteria);
I finally found an example in the DJL source code. The key take-away is to not use NDList for the input/output at all:
Criteria<String[], float[][]> criteria =
Criteria.builder()
.optApplication(Application.NLP.TEXT_EMBEDDING)
.setTypes(String[].class, float[][].class)
.optModelUrls(modelUrl)
.build();
try (ZooModel<String[], float[][]> model = ModelZoo.loadModel(criteria);
Predictor<String[], float[][]> predictor = model.newPredictor()) {
return predictor.predict(inputs.toArray(new String[0]));
}
See https://github.com/awslabs/djl/blob/master/examples/src/main/java/ai/djl/examples/inference/UniversalSentenceEncoder.java for the complete example.
I am using Weka libraries for feature selection problem solution. I have read data as follows : The data set is in arff format
BufferedReader reader = new BufferedReader(new FileReader("D:\\Exp\\golf.arff"));
Instances data = new Instances(reader);
reader.close();
data.setClassIndex(data.numAttributes() - 1);
For example, attributes are Temperature, humidity, Windy, Outlook and the last one is class.
Now I want to store Temperature attribute as instances. The new data will consists only temperature (and it should be instance type because in further processing I have to use methods of instance)
Quick google search brought this:
Instead of picking the features just remove all features that you don't need.
import java.io.File;
import weka.core.Instances;
import weka.core.converters.ArffLoader;
import weka.core.converters.ArffSaver;
import weka.filters.Filter;
import weka.filters.unsupervised.attribute.Remove;
public class Convert4 {
public static void main(String[] args) {
// TODO Auto-generated method stub
try
{
ArffLoader loader2= new ArffLoader();
loader2.setSource(new File("C:/Users/RAHUL/Desktop/stack.arff"));
Instances data2= loader2.getDataSet();
//Load Arff
String[] options = new String[2];
options[0] = "-R"; // "range"
options[1] = "1"; // first attribute
Remove remove = new Remove(); // new instance of filter
remove.setOptions(options); // set options
remove.setInputFormat(data2); // inform filter about dataset **AFTER** setting options
Instances newData2 = Filter.useFilter(data2, remove); // apply filter
ArffSaver saver = new ArffSaver();
saver.setInstances(newData2);
saver.setFile(new File("C:/Users/RAHUL/Desktop/stack2.arff"));
saver.writeBatch();
}
catch (Exception e)
{}
}
}
Answer comes from:
How to remove particular attributes from arff file and produce modified arff?
I am using Riak KV with Java client and I am unable to write on the RiakNode, although I have created a Bucket with the name of the space I want to create an object on.
I basically have the TasteOfRiak.java class, which has been provided by the basho developer website: https://raw.githubusercontent.com/basho/basho_docs/master/extras/code-examples/TasteOfRiak.java
import com.basho.riak.client.api.RiakClient;
import com.basho.riak.client.api.commands.kv.DeleteValue;
import com.basho.riak.client.api.commands.kv.FetchValue;
import com.basho.riak.client.api.commands.kv.StoreValue;
import com.basho.riak.client.api.commands.kv.UpdateValue;
import com.basho.riak.client.core.RiakCluster;
import com.basho.riak.client.core.RiakNode;
import com.basho.riak.client.core.query.Location;
import com.basho.riak.client.core.query.Namespace;
import com.basho.riak.client.core.query.RiakObject;
import com.basho.riak.client.core.util.BinaryValue;
import java.net.UnknownHostException;
public class TasteOfRiak {
// A basic POJO class to demonstrate typed exchanges with Riak
public static class Book {
public String title;
public String author;
public String body;
public String isbn;
public Integer copiesOwned;
}
// This will allow us to update the book object handling the
// entire fetch/modify/update cycle.
public static class BookUpdate extends UpdateValue.Update<Book> {
private final Book update;
public BookUpdate(Book update){
this.update = update;
}
#Override
public Book apply(Book t) {
if(t == null) {
t = new Book();
}
t.author = update.author;
t.body = update.body;
t.copiesOwned = update.copiesOwned;
t.isbn = update.isbn;
t.title = update.title;
return t;
}
}
// This will create a client object that we can use to interact with Riak
private static RiakCluster setUpCluster() throws UnknownHostException {
// This example will use only one node listening on localhost:10017
RiakNode node = new RiakNode.Builder()
.withRemoteAddress("127.0.0.1")
.withRemotePort(8087)
.build();
// This cluster object takes our one node as an argument
RiakCluster cluster = new RiakCluster.Builder(node)
.build();
// The cluster must be started to work, otherwise you will see errors
cluster.start();
return cluster;
}
public static void main( String[] args ) {
try {
// First, we'll create a basic object storing a movie quote
RiakObject quoteObject = new RiakObject()
// We tell Riak that we're storing plaintext, not JSON, HTML, etc.
.setContentType("text/plain")
// Objects are ultimately stored as binaries
.setValue(BinaryValue.create("You're dangerous, Maverick"));
System.out.println("Basic object created");
// In the new Java client, instead of buckets you interact with Namespace
// objects, which consist of a bucket AND a bucket type; if you don't
// supply a bucket type, "default" is used; the Namespace below will set
// only a bucket, without supplying a bucket type
Namespace quotesBucket = new Namespace("quotes");
// With our Namespace object in hand, we can create a Location object,
// which allows us to pass in a key as well
Location quoteObjectLocation = new Location(quotesBucket, "Iceman");
System.out.println("Location object created for quote object");
// With our RiakObject in hand, we can create a StoreValue operation
StoreValue storeOp = new StoreValue.Builder(quoteObject)
.withLocation(quoteObjectLocation)
.build();
System.out.println("StoreValue operation created");
// And now we can use our setUpCluster() function to create a cluster
// object which we can then use to create a client object and then
// execute our storage operation
RiakCluster cluster = setUpCluster();
RiakClient client = new RiakClient(cluster);
System.out.println("Client object successfully created");
StoreValue.Response storeOpResp = client.execute(storeOp);
System.out.println("Object storage operation successfully completed");
// Now we can verify that the object has been stored properly by
// creating and executing a FetchValue operation
FetchValue fetchOp = new FetchValue.Builder(quoteObjectLocation)
.build();
RiakObject fetchedObject = client.execute(fetchOp).getValue(RiakObject.class);
assert(fetchedObject.getValue().equals(quoteObject.getValue()));
System.out.println("Success! The object we created and the object we fetched have the same value");
// Now update the fetched object
fetchedObject.setValue(BinaryValue.create("You can be my wingman any time."));
StoreValue updateOp = new StoreValue.Builder(fetchedObject)
.withLocation(quoteObjectLocation)
.build();
StoreValue.Response updateOpResp = client.execute(updateOp);
updateOpResp = client.execute(updateOp);
// And we'll delete the object
DeleteValue deleteOp = new DeleteValue.Builder(quoteObjectLocation)
.build();
client.execute(deleteOp);
System.out.println("Quote object successfully deleted");
Book mobyDick = new Book();
mobyDick.title = "Moby Dick";
mobyDick.author = "Herman Melville";
mobyDick.body = "Call me Ishmael. Some years ago...";
mobyDick.isbn = "1111979723";
mobyDick.copiesOwned = 3;
System.out.println("Book object created");
// Now we'll assign a Location for the book, create a StoreValue
// operation, and store the book
Namespace booksBucket = new Namespace("books");
Location mobyDickLocation = new Location(booksBucket, "moby_dick");
StoreValue storeBookOp = new StoreValue.Builder(mobyDick)
.withLocation(mobyDickLocation)
.build();
client.execute(storeBookOp);
System.out.println("Moby Dick information now stored in Riak");
// And we'll verify that we can fetch the info about Moby Dick and
// that that info will match the object we created initially
FetchValue fetchMobyDickOp = new FetchValue.Builder(mobyDickLocation)
.build();
Book fetchedBook = client.execute(fetchMobyDickOp).getValue(Book.class);
System.out.println("Book object successfully fetched");
assert(mobyDick.getClass() == fetchedBook.getClass());
assert(mobyDick.title.equals(fetchedBook.title));
assert(mobyDick.author.equals(fetchedBook.author));
// And so on...
// Now to update the book with additional copies
mobyDick.copiesOwned = 5;
BookUpdate updatedBook = new BookUpdate(mobyDick);
UpdateValue updateValue = new UpdateValue.Builder(mobyDickLocation)
.withUpdate(updatedBook).build();
UpdateValue.Response response = client.execute(updateValue);
System.out.println("Success! All of our tests check out");
// Now that we're all finished, we should shut our cluster object down
cluster.shutdown();
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
}
Whenever Eclipse executes this code: "StoreValue.Response storeOpResp = client.execute(storeOp);
System.out.println("Object storage operation successfully completed");"
I get an error that "ERROR com.basho.riak.client.core.RiakNode - Write failed on RiakNode".
Before running that program I have already created a quotesBucket bucket and have activated it.
Does anyone know where the problem is?
Can you store an object through http? Try this in terminal:
curl -XPUT \
-H "Content-Type: text/plain" \
-d "You're dangerous, Maverick" \
http://localhost:8098/types/default/buckets/quotes/keys/Iceman?returnbody=true
I am trying to get the Flickr data by using API key and secret key provided by flickr . I have written a java code for it.
`package com.flickr.project;
import java.io.IOException;
import javax.xml.parsers.ParserConfigurationException;
import org.xml.sax.SAXException;
import com.aetrion.flickr.Flickr;
import com.aetrion.flickr.FlickrException;
import com.aetrion.flickr.REST;
import com.aetrion.flickr.auth.Permission;
import com.aetrion.flickr.photos.SearchParameters;
import com.aetrion.flickr.photos.PhotoList;
import com.aetrion.flickr.photos.PhotosInterface;
import com.aetrion.flickr.photos.Photo;
import com.flickr4java.flickr.Auth;
import com.flickr4java.flickr.RequestContext;
public class SampleProgram{
public static void main(String[] args) {
try {
searchImages();
} catch (Exception e) {
e.printStackTrace();
}
}
public static void searchImages() {
// Search photos with tag keywords and get result
try{
//Set api key
String key="";
String svr="www.flickr.com";
String secret="";
RequestContext requestContext = RequestContext.getRequestContext();
Auth auth = new Auth();
auth.setPermission(Permission.READ);
auth.setToken("");
auth.setTokenSecret("");
requestContext.setAuth(auth);
REST rest=new REST();
rest.setHost(svr);
//initialize Flickr object with key and rest
Flickr flickr=new Flickr(key,secret,rest);
Flickr.debugRequest = false;
Flickr.debugStream = false;
Flickr.debugStream=false;
//initialize SearchParameter object, this object stores the search keyword
SearchParameters searchParams=new SearchParameters();
searchParams.setSort(SearchParameters.INTERESTINGNESS_DESC);
//Create tag keyword array
String[] tags=new String[]{"Dog","Doberman"};
searchParams.setTags(tags);
//Initialize PhotosInterface object
PhotosInterface photosInterface=flickr.getPhotosInterface();
//Execute search with entered tags
// PhotoList photoList=null;
PhotoList photoList=photosInterface.search(searchParams,20,1);
System.out.println("here");
//get search result and fetch the photo object and get small square imag's url
if(photoList!=null){
//Get search result and check the size of photo result
for(int i=0;i<photoList.size();i++){
//get photo object
Photo photo=(Photo)photoList.get(i);
//Get small square url photo
StringBuffer strBuf=new StringBuffer();
strBuf.append("<a href=\"\">");
strBuf.append("<img border=\"0\" src=\""+photo.getSmallSquareUrl()+"\">");
strBuf.append("</a>\n");
// ....
}
}
}catch(Exception e){
e.printStackTrace();
}
}
public void userAuthentication(){
/*InputStream in = null;
try {
in = getClass().getResourceAsStream("/setup.properties");
// properties = new Properties();
// properties.load(in);
} finally {
IOUtilities.close(in);
}
f = new Flickr(properties.getProperty("apiKey"), properties.getProperty("secret"), new REST());
requestContext = RequestContext.getRequestContext();
Auth auth = new Auth();
auth.setPermission(Permission.READ);
auth.setToken(properties.getProperty("token"));
auth.setTokenSecret(properties.getProperty("tokensecret"));
requestContext.setAuth(auth);
Flickr.debugRequest = false;
Flickr.debugStream = false;*/
}
} `
I need to fetch all data including images from flickr using the key words i mentioned in the program.
From the look of things, Photo has several methods like the following:
"getSmallSquareAsInputStream" - This returns an input stream, this can be used to fetch the image data.
The full API's list for Photo class is available here