We Keep Coding

extracting all fields from a Lucene8 index

Given an index created with Lucene-8, but without knowledge of the fields used, how can I programmatically extract all the fields? (I'm aware that the Luke browser can be used interactively (thanks to #andrewjames) Examples for using latest version of Lucene. ) The scenario is that, during a development phase, I have to read indexes without prescribed schemas.
I'm using
IndexReader reader = DirectoryReader.open(FSDirectory.open(Paths.get(index)));
IndexSearcher searcher = new IndexSearcher(reader);
The reader has methods such as:
reader.getDocCount(field);
but this requires knowing the fields in advance.
I understand that documents in the index may be indexed with different fields; I'm quite prepared to iterate over all documents and extract the fields on a regular basis (these indexes are not huge).
I'm using Lucene 8.5.* so post and tutorials based on earlier Lucene versions may not work.

You can access basic field info as follows:
import java.util.List;
import java.io.IOException;
import java.nio.file.Paths;
import org.apache.lucene.document.Document;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.IndexableField;
import org.apache.lucene.store.FSDirectory;
public class IndexDataExplorer {
private static final String INDEX_PATH = "/path/to/index/directory";
public static void doSearch() throws IOException {
IndexReader reader = DirectoryReader.open(FSDirectory.open(Paths.get(INDEX_PATH)));
for (int i = 0; i < reader.numDocs(); i++) {
Document doc = reader.document(i);
List<IndexableField> fields = doc.getFields();
for (IndexableField field : fields) {
// use these to get field-related data:
//field.name();
//field.fieldType().toString();
}
}
}
}

Cannot instantiate the type Instance in Java Weka class

When I try to run this example Weka Java code in Eclipse I get the error
Cannot instantiate the type Instance
Even though there's no error on import weka.core.Instance:
package wekatest;
import weka.classifiers.Classifier;
import weka.classifiers.Evaluation;
import weka.classifiers.bayes.NaiveBayes;
import weka.core.Attribute;
import weka.core.FastVector;
import weka.core.Instance;
import weka.core.Instances;
public class Driver {
public static void main(String[] args) throws Exception{
// Declare two numeric attributes
Attribute Attribute1 = new Attribute("firstNumeric");
Attribute Attribute2 = new Attribute("secondNumeric");
// Declare a nominal attribute along with its values
FastVector fvNominalVal = new FastVector(3);
fvNominalVal.addElement("blue");
fvNominalVal.addElement("gray");
fvNominalVal.addElement("black");
Attribute Attribute3 = new Attribute("aNominal", fvNominalVal);
// Declare the class attribute along with its values
FastVector fvClassVal = new FastVector(2);
fvClassVal.addElement("positive");
fvClassVal.addElement("negative");
Attribute ClassAttribute = new Attribute("theClass", fvClassVal);
// Declare the feature vector
FastVector fvWekaAttributes = new FastVector(4);
fvWekaAttributes.addElement(Attribute1);
fvWekaAttributes.addElement(Attribute2);
fvWekaAttributes.addElement(Attribute3);
fvWekaAttributes.addElement(ClassAttribute);
// Create an empty training set
Instances isTrainingSet = new Instances("Rel", fvWekaAttributes, 10);
// Set class index
isTrainingSet.setClassIndex(3);
// Create the instance
Instance iExample = new Instance(4);
iExample.setValue((Attribute)fvWekaAttributes.elementAt(0), 1.0);
iExample.setValue((Attribute)fvWekaAttributes.elementAt(1), 0.5);
iExample.setValue((Attribute)fvWekaAttributes.elementAt(2), "gray");
iExample.setValue((Attribute)fvWekaAttributes.elementAt(3), "positive");
// add the instance
isTrainingSet.add(iExample);
Classifier cModel = (Classifier)new NaiveBayes();
cModel.buildClassifier(isTrainingSet);
// Test the model
Evaluation eTest = new Evaluation(isTrainingSet);
eTest.evaluateModel(cModel, isTrainingSet);
// Print the result à la Weka explorer:
String strSummary = eTest.toSummaryString();
System.out.println(strSummary);
// Get the confusion matrix
double[][] cmMatrix = eTest.confusionMatrix();
for(int row_i=0; row_i<cmMatrix.length; row_i++){
for(int col_i=0; col_i<cmMatrix.length; col_i++){
System.out.print(cmMatrix[row_i][col_i]);
System.out.print("|");
}
System.out.println();
}
}
}

Take a look at the docs, below. Your mistake is completely understandable, and easy to fix.
You attempted to import the class Instances:
http://weka.sourceforge.net/doc.dev/weka/core/Instances.html
What you really want is the Interface Instance (note singular form in name):
http://weka.sourceforge.net/doc.dev/weka/core/Instance.html

Select classes with given annotation

I have a bunch of source files for java classes. I want to find those classes which are annotated by a given annotation class. The names of those classes should be written to a service provider list file.
Is there any machinery I could use to help me with this task? Or do I have to implement this myself from scratch?
If I had to do this myself, there are several approaches I can think of.
Write an Ant Task in Java. Have it create a ClassLoader using a suitable (probably configurable) class path. Use that loader to (attempt to) load the classes matching the input files, in order to inspect their annotations. Requires annotation retention at runtime, and full initialization of all involved classes and their dependencies.
Use javap to inspect the classes. Since I don't know of a programmatic interface to javap (do you?), this probably means iterating over the files and running a new process for each of them, then massaging the created output in a suitable way. Perhaps a <scriptdef>-ed task could be used for this. This would work with class-file annotation retention, and require no initialization.
Use an annotation processor to collect the information at compile-time. This should be able to work with sourcecode-only retention. But I have no experience writing or using annotation compilers, so I'm not sure this will work, and will need a lot of research to figure out some of the details. In particular how to activate the task for use by ant (Java 6 annotation processing configuration with Ant gives some pointers on this, as does What is the default annotation processors discovery process?) and when to create the output file (in each round, or only in the last round).
Which of these do you think has the greatest chances of success? Can you suggest code samples for one of these, which might be close to what I want and which I could adapt appropriately?

Encouraged by Thomas' comment, I gave approach 3 a try and got the following annotation processor working reasonably well:
import java.io.IOException;
import java.io.Writer;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import javax.annotation.processing.AbstractProcessor;
import javax.annotation.processing.ProcessingEnvironment;
import javax.annotation.processing.RoundEnvironment;
import javax.annotation.processing.SupportedSourceVersion;
import javax.lang.model.SourceVersion;
import javax.lang.model.element.Element;
import javax.lang.model.element.QualifiedNameable;
import javax.lang.model.element.TypeElement;
import javax.tools.StandardLocation;
#SupportedSourceVersion(SourceVersion.RELEASE_7)
public class AnnotationServiceProcessor extends AbstractProcessor {
// Map name of the annotation to name of the corresponding service interface
private static Map<String, String> annotationToServiceMap = new HashMap<>();
static {
// Adapt this to your use, or make it configurable somehow
annotationToServiceMap.put("Annotation1", "Service1");
annotationToServiceMap.put("Annotation2", "Service2");
}
#Override public Set<String> getSupportedAnnotationTypes() {
return annotationToServiceMap.keySet();
}
// Map name of the annotation to list of names
// of the classes which carry that annotation
private Map<String, List<String>> classLists;
#Override public void init(ProcessingEnvironment env) {
super.init(env);
classLists = new HashMap<>();
for (String ann: getSupportedAnnotationTypes())
classLists.put(ann, new ArrayList<String>());
}
public boolean process(Set<? extends TypeElement> annotations,
RoundEnvironment env) {
for (TypeElement ann: annotations) {
List<String> classes =
classLists.get(ann.getQualifiedName().toString());
for (Element elt: env.getElementsAnnotatedWith(ann)) {
QualifiedNameable qn = (QualifiedNameable)elt;
classes.add(qn.getQualifiedName().toString());
}
}
if (env.processingOver()) { // Only write results at the end
for (String ann: getSupportedAnnotationTypes()) {
try {
write(ann, classLists.get(ann));
} catch (IOException e) {
throw new RuntimeException(e); // UGLY!
}
}
}
return true;
}
// Write the service file for each annotation we found
private void write(String ann, List<String> classes) throws IOException {
if (classes.isEmpty())
return;
String service = annotationToServiceMap.get(ann);
Writer w = processingEnv.getFiler()
.createResource(StandardLocation.CLASS_OUTPUT,
"", "META-INF/services/" + service)
.openWriter();
classes.sort(null); // Make the processing order irrelevant
for (String cls: classes) {
w.write(cls);
w.write('\n');
}
w.close();
}
}
So far I've hooked this up to ant using <compilerarg>s from https://stackoverflow.com/a/3644624/1468366. I'll try something better and if I succeed will edit this post to include some ant snippet.

Dynamic Typecasting in Java

I'm writing a plugin for the Minecraft server implementation CraftBukkit, and I've come across a problem where I need to cast to a class that is found through reflection.
Here's the deal. The original code I wrote looked like this, with irrelevant parts removed:
import java.util.ArrayList;
import java.util.Collection;
import java.util.Iterator;
import java.util.List;
import java.util.Random;
import net.minecraft.server.v1_7_R3.EntityAnimal;
import net.minecraft.server.v1_7_R3.EntityHuman;
import org.bukkit.craftbukkit.v1_7_R3.entity.CraftAnimals;
import org.bukkit.craftbukkit.v1_7_R3.entity.CrafteEntity;
import org.bukkit.World;
import org.bukkit.entity.Animals;
import org.bukkit.entity.Entity;
import org.bukkit.event.Listener;
import org.bukkit.plugin.java.JavaPlugin;
import org.bukkit.scheduler.BukkitRunnable;
public class Task extends BukkitRunnable {
private static final int MATING_DISTANCE = 14;
private final JavaPlugin plugin;
private final Random randomizer;
private boolean mateMode;
private double chance;
public Task(JavaPlugin plugin, double chance, boolean mateMode) {
this.plugin = plugin;
this.randomizer = new Random();
this.chance = chance;
this.mateMode = mateMode;
this.theTaskListener = listener;
}
public void run() {
List<World> worlds = plugin.getServer().getWorlds();
Iterator<World> worldIterator = worlds.iterator();
while (worldIterator.hasNext()) {
World world = worldIterator.next();
Collection<Animals> animals = world.getEntitiesByClass(Animals.class);
Iterator<Animals> animalIterator = animals.iterator();
while (animalIterator.hasNext()) {
Animals animal = (Animals) animalIterator.next();
EntityAnimal entity = (EntityAnimal) ((CraftEntity) ((CraftAnimals) animal)).getHandle();
EntityHuman feeder = null;
entity.f(feeder);
}
}
}
}
However, as you can see in the imports, this code imported the classes from only one version of the Minecraft server package - v1_7_R3. Now the problem is, I want to add support for more than that, and I want to be able to do that without creating separate versions of my plugin for each version of Minecraft. Despite the fact that most of the classes in the package are the same (at least ALL of those that I need) the package names are different, and therefore it can't be done with static imports (or at least I think so?)
So, I decided to use reflection in order to get the correct classes I need (this code is in another class):
private static final String[] requiredClasses = {
"net.minecraft.server.%s.EntityAnimal",
"net.minecraft.server.%s.EntityHuman",
"org.bukkit.craftbukkit.%s.entity.CraftAnimals",
"org.bukkit.craftbukkit.%s.entity.CraftEntity"
};
public static final String[] supportedVersions = {
"v1_7_R3",
"v1_7_R4"
};
public Class<?>[] initializeClasses() {
String correctVersion = null;
for (int i = 0; i < supportedVersions.length; i++) {
String version = supportedVersions[i];
boolean hadIssues = false;
for (int j = 0; j < requiredClasses.length; j++) {
String className = requiredClasses[j];
try {
Class.forName(String.format(className, version));
} catch (ClassNotFoundException e) {
getLogger().log(Level.INFO, String.format("The correct version isn't %s.", version));
hadIssues = true;
break;
}
}
if (!hadIssues) {
correctVersion = version;
break;
}
}
Class[] classes = new Class[requiredClasses.length];
if (correctVersion != null) {
getLogger().log(Level.INFO, String.format("The correct version is %s.", correctVersion));
for (int i = 0; i < requiredClasses.length; i++) {
String className = requiredClasses[i];
try {
classes[i] = Class.forName(String.format(className, correctVersion));
} catch (ClassNotFoundException e) {}
}
} else {
getLogger().log(Level.WARNING, "The version of Minecraft on this server is not supported.");
getLogger().log(Level.WARNING, "Due to this, the plugin will self-disable.");
getLogger().log(Level.WARNING, "To fix this issue, get build that supports your version.");
this.setEnabled(false);
}
return classes;
}
Now, this approach successfully retrieves the required classes in both versions currently supported. I passed these to the rewritten Task class using instance variables and an edited constructor, and I removed the version-specific imports:
import java.util.ArrayList;
import java.util.Collection;
import java.util.Iterator;
import java.util.List;
import java.util.Random;
import org.bukkit.World;
import org.bukkit.entity.Animals;
import org.bukkit.entity.Entity;
import org.bukkit.plugin.java.JavaPlugin;
import org.bukkit.scheduler.BukkitRunnable;
public class Task extends BukkitRunnable {
private static final int MATING_DISTANCE = 14;
private final JavaPlugin plugin;
private final Random randomizer;
private boolean mateMode;
private double chance;
private Class entityAnimal;
private Class entityHuman;
private Class craftAnimals;
private Class craftEntity;
public Task(JavaPlugin plugin, Class[] classes, double chance, boolean mateMode) {
this.plugin = plugin;
this.randomizer = new Random();
this.chance = chance;
this.mateMode = mateMode;
this.entityAnimal = classes[0];
this.entityHuman = classes[1];
this.craftAnimals = classes[2];
this.craftEntity = classes[3];
}
Now, how can I rewrite the Task.run() method so that it will use the reflection classes? There is a whole lot of typecasting involved and unfortunately it's all necessary due to the ridiculous amount of overloading in the Minecraft code. For example, the entity.f(EntityHuman human) method cannot simply be called by doing entity.f(null) because there are other overloading entity.f(Object object) methods.
I am open to all suggestions as I'm facing a dead-end here. If there is a better approach to the problem, I could change to that as well.
Thank you!

In an object oriented language, we have access to various design patterns that have been developed for exactly this purpose. We use two patterns in particular.
Adapter Pattern is used to provide the same interface to a number of different implementations. It is sometimes called a shim. You create one class per version of each server, importing libraries to each. The class implements an interface that they hold in common.
Factory Pattern is used to select among the adapter classes. You use whatever method you need to determine which server version you have, and it will create an object implementing the proper interface. The main code remains the same. It calls the factory to get an object that knows how to deal with the server.
The advantages of this approach are several. You don't pollute the name space by importing overlapping libraries. The main code is much less susceptible to change as new server versions are added; the only code that needs to be written is the new server shim and the factory that determines which adapter to produce.

Just a brainstorming idea. What if:
importing all supported versions
fully referencing the appropriate package's types
checking for the version that's targeted at a particular runtime (assumed it can be obtained somehow)
import net.minecraft.server.v1_7_R3.*;
import net.minecraft.server.v1_7_R4.*;
enum Version {
V1_7_R3,
V1_7_R4
}
Version currentVersion;
net.minecraft.server.v1_7_R3.EntityAnimal animal3;
net.minecraft.server.v1_7_R4.EntityAnimal animal4;
// obtain currentVersion
switch ( currentVersion ) {
case V1_7_R3:
animal3.method();
break;
case V1_7_R4:
animal4.method();
break;
}
This is somehow ugly, of course, but under the given circumstances it's the possibility that came into my mind first.

After reading Gerold Broser's response, I realized that I would have to somehow modify my approach in order to create some sort of a handler class that would carry out the version-specific operation - of course this would be an interface that would separately be implemented by a class per version.
However, this became a problem when I realized Maven wouldn't let me call two versions of the same groupid.artifactid object.
I quickly did some research and found mbaxter's Multiple Versions Tutorial as well as the AbstractionExamplePlugin implementation which perfectly demonstrates this approach.
The approach works perfectly and is what every Bukkit developer should use. Here's my finished plugin for further reference if necessary.

Custom builtin in Jena

This is the code I have written but the new built-in does not seem to work. I get the error:
Exception in thread "main" com.hp.hpl.jena.reasoner.rulesys.impl.LPRuleSyntaxException: Syntax error in backward rule: matematica Unknown builtin operation mysum
Can anyone tell me where the error is? Here is my code:
package JenaRules;
import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.Arrays;
import java.util.List;
import org.semanticweb.owlapi.model.OWLOntologyCreationException;
import org.semanticweb.owlapi.model.OWLOntologyStorageException;
import com.hp.hpl.jena.graph.Node;
import com.hp.hpl.jena.query.Query;
import com.hp.hpl.jena.query.QueryExecution;
import com.hp.hpl.jena.query.QueryExecutionFactory;
import com.hp.hpl.jena.query.QueryFactory;
import com.hp.hpl.jena.query.ResultSet;
import com.hp.hpl.jena.query.ResultSetFormatter;
import com.hp.hpl.jena.rdf.model.InfModel;
import com.hp.hpl.jena.rdf.model.Model;
import com.hp.hpl.jena.rdf.model.ModelFactory;
import com.hp.hpl.jena.rdf.model.Resource;
import com.hp.hpl.jena.reasoner.Reasoner;
import com.hp.hpl.jena.reasoner.rulesys.*;
import com.hp.hpl.jena.reasoner.rulesys.builtins.BaseBuiltin;
import com.hp.hpl.jena.util.FileManager;
import com.hp.hpl.jena.vocabulary.RDFS;
import com.hp.hpl.jena.vocabulary.ReasonerVocabulary;
public class RulesOntology_MT {
public static void main(String[] args) throws OWLOntologyStorageException,
OWLOntologyCreationException, IOException {
BuiltinRegistry.theRegistry.register(new BaseBuiltin() {
#Override
public String getName() {
return "mysum";
}
#Override
public int getArgLength() {
return 2;
}
#Override
public boolean bodyCall(Node[] args, int length, RuleContext context) {
checkArgs(length, context);
BindingEnvironment env = context.getEnv();
Node n1 = getArg(0, args, context);
Node n2 = getArg(1, args, context);
if (n1.isLiteral() && n2.isLiteral()) {
Object v1 = n1.getLiteralValue();
Object v2 = n2.getLiteralValue();
Node sum = null;
if (v1 instanceof Number && v2 instanceof Number) {
Number nv1 = (Number)v1;
Number nv2 = (Number)v2;
int sumInt = nv1.intValue()+nv2.intValue();
sum = Util.makeIntNode(sumInt);
return env.bind(args[2], sum);
}
}
return false;
}
});
// NON SERVE
// final String exampleRuleString2 =
// "[mat1: equal(?s ?p )\n\t-> print(?s ?p ?o),\n\t (?s ?p ?o)\n]"+
// "";
final String exampleRuleString =
"[matematica:"+
"(?p http://www.semanticweb.org/prova_rules_M#totale_crediti ?x)"+
" -> " +
"(?p rdf:type http://www.semanticweb.org/prova_rules_M#:Persona)"+
"(?e rdf:type http://www.semanticweb.org/prova_rules_M#:Esame)"+
"(?p http://www.semanticweb.org/prova_rules_M#:haSostenutoEsameDi ?e)"+
"(?e http://www.semanticweb.org/prova_rules_M/persona#crediti_esame ?cr)"+
"mysum(?cr,2)"+
"]";
System.out.println(exampleRuleString);
/* I tend to use a fairly verbose syntax for parsing out my rules when I construct them
* from a string. You can read them from whatever other sources.
*/
final List<Rule> rules;
try( final BufferedReader src = new BufferedReader(new InputStreamReader(new ByteArrayInputStream(exampleRuleString.getBytes()))) ) {
rules = Rule.parseRules(Rule.rulesParserFromReader(src));
}
/* Construct a reasoner and associate the rules with it */
// create an empty non-inferencing model
GenericRuleReasoner reasoner = (GenericRuleReasoner) GenericRuleReasonerFactory.theInstance().create(null);
reasoner.setRules(rules);
/* Create & Prepare the InfModel. If you don't call prepare, then
* rule firings and inference may be deferred until you query the
* model rather than happening at insertion. This can make you think
* that your Builtin is not working, when it is.
*/
InfModel infModel = ModelFactory.createInfModel(reasoner, ModelFactory.createDefaultModel());
infModel.prepare();
infModel.createResource(RDFS.Class);
//write down the result in RDFXML form
infModel.write(System.out);
}
}

Using the code that you provided, and Apache Jena 2.11.1, I cannot replicate the exception you are getting. Do note that when you call BuiltinRegistry.theRegistry.register(...), you are telling the reasoner that the builtin exists.
Solution
The exception that you are getting is likely because, in your actual code, you are not calling BuiltinRegistry.theRegistry.register(...) prior to calling Rule.parseRules(Rule.rulesParserFromReader(src));, so as far as the rule parser is concerned, you are using a Builtin which doesn't exist. To fix it, merely call register before parsing your rules. The toy example provided does not have this problem.
Using the example provided
I also noted that the provided code example did not include anything that would actually stimulate the rule to fire, so, in lieu of infModel.createResource(RDFS.Class);, I added the following lines:
final Resource s = infModel.createResource();
final Property p = infModel.createProperty("http://www.semanticweb.org/prova_rules_M#totale_crediti");
final Resource o = infModel.createResource();
infModel.add(s,p,o);
This stimulated the rule to fire, and led to the following exception trace:
com.hp.hpl.jena.reasoner.rulesys.BuiltinException: Error in clause of rule (matematica) mysum: builtin mysum not usable in rule heads
at com.hp.hpl.jena.reasoner.rulesys.builtins.BaseBuiltin.headAction(BaseBuiltin.java:86)
at com.hp.hpl.jena.reasoner.rulesys.impl.RETEConflictSet.execute(RETEConflictSet.java:184)
at com.hp.hpl.jena.reasoner.rulesys.impl.RETEConflictSet.add(RETEConflictSet.java:81)
at com.hp.hpl.jena.reasoner.rulesys.impl.RETEEngine.requestRuleFiring(RETEEngine.java:249)
at com.hp.hpl.jena.reasoner.rulesys.impl.RETETerminal.fire(RETETerminal.java:80)
at com.hp.hpl.jena.reasoner.rulesys.impl.RETEClauseFilter.fire(RETEClauseFilter.java:227)
at com.hp.hpl.jena.reasoner.rulesys.impl.RETEEngine.inject(RETEEngine.java:469)
at com.hp.hpl.jena.reasoner.rulesys.impl.RETEEngine.runAll(RETEEngine.java:451)
at com.hp.hpl.jena.reasoner.rulesys.impl.RETEEngine.add(RETEEngine.java:174)
at com.hp.hpl.jena.reasoner.rulesys.FBRuleInfGraph.performAdd(FBRuleInfGraph.java:654)
at com.hp.hpl.jena.graph.impl.GraphBase.add(GraphBase.java:202)
at com.hp.hpl.jena.rdf.model.impl.ModelCom.add(ModelCom.java:1138)
at SO.test(SO.java:108)
As a note: my test class is SO.java and line 108 is where we call infModel.add(s,p,o).
The exception that I get is different than the exception you encountered, but it is worth explaining. The implementation that you provided implements Builtin#bodyCall(...), but not Builtin#headAction(...). We can see the exception is thrown from BaseBuiltin#headAction(...). This default behavior assumes that you didn't implement the method because your Builtin doesn't support it. In the toy problem, this is correct behavior because the example implementation cannot be used in rule heads.