I need to use Lucene MoreLikeThis to find similar documents given a paragraph of text. I am new to Lucene and followed the code here
I have already indexed the documents at the directory - "C:\Users\lucene_index_files\v2"
I am using "They are computer engineers and they like to develop their own tools. The program in languages like Java, CPP." as the document to which I want to find similar documents.
public class LuceneSearcher2 {
public static void main(String[] args) throws IOException {
LuceneSearcher2 m = new LuceneSearcher2();
System.out.println("1");
m.start();
System.out.println("2");
//m.writerEntries();
m.findSilimar("They are computer engineers and they like to develop their own tools. The program in languages like Java, CPP.");
System.out.println("3");
}
private Directory indexDir;
private StandardAnalyzer analyzer;
private IndexWriterConfig config;
public void start() throws IOException{
//analyzer = new StandardAnalyzer(Version.LUCENE_42);
//config = new IndexWriterConfig(Version.LUCENE_42, analyzer);
analyzer = new StandardAnalyzer();
config = new IndexWriterConfig(analyzer);
config.setOpenMode(OpenMode.CREATE_OR_APPEND);
indexDir = new RAMDirectory(); //don't write on disk
//https://stackoverflow.com/questions/36542551/lucene-in-java-method-not-found?rq=1
indexDir = FSDirectory.open(FileSystems.getDefault().getPath("C:\\Users\\lucene_index_files\\v2")); //write on disk
//System.out.println(indexDir);
}
private void findSilimar(String searchForSimilar) throws IOException {
IndexReader reader = DirectoryReader.open(indexDir);
IndexSearcher indexSearcher = new IndexSearcher(reader);
System.out.println("2a");
MoreLikeThis mlt = new MoreLikeThis(reader);
mlt.setMinTermFreq(0);
mlt.setMinDocFreq(0);
mlt.setFieldNames(new String[]{"title", "content"});
mlt.setAnalyzer(analyzer);
System.out.println("2b");
StringReader sReader = new StringReader(searchForSimilar);
//Query query = mlt.like(sReader, null);
//Throws error - The method like(String, Reader...) in the type MoreLikeThis is not applicable for the arguments (StringReader, null)
Query query = mlt.like("computer");
System.out.println("2c");
System.out.println(query.toString());
TopDocs topDocs = indexSearcher.search(query,10);
for ( ScoreDoc scoreDoc : topDocs.scoreDocs ) {
Document aSimilar = indexSearcher.doc( scoreDoc.doc );
String similarTitle = aSimilar.get("title");
String similarContent = aSimilar.get("content");
System.out.println("====similar finded====");
System.out.println("title: "+ similarTitle);
System.out.println("content: "+ similarContent);
}
System.out.println("2d");
}}
I am unsure as to what is causing the system to not generate an output/
What is your output ? I am assuming your not finding similar documents. The reason could be that the query you are creating is empty.
First of all to run your code in a meaningful way this line
Query query = mlt.like(sReader, null);
needs a String[] of field names as the argument, so it should work like this
Query query = mlt.like(sReader, new String[]{"title", "content"});
Now, in order to use MoreLikeThis in Lucene, your stored Fields have to have the set the option to store term vectors "setStoreTermVectors(true);" true when creating fields, for instance like this:
FieldType fieldType = new FieldType();
fieldType.setStored(true);
fieldType.setStoreTermVectors(true);
fieldType.setTokenized(true);
Field contentField = new Field("contents", this.getBlurb(), fieldType);
doc.add(contentField);
Leaving this out could result in an empty query string and consequently no results for the query
Related
I'm a bit new to Lucene, I am using a huge database which I indexed previously. The problem is that it is not an efficient way to index the whole table/ database every time if something new is added into it. I'm using lucene3.6.2. I want to make an indexing function which adds the new data to the existing Lucene indexed files, without the need to updateDocument(or delete and re-index in lucene). I mean to say it should not create new files to store the new documents rather should insert them into the previous index files without deleting the previous data inside the index files and without re-indexing the whole database. Whose index should start from the last index location of the previously indexed item, and should be searchable along with the previously generated indexes. This is my indexer code for creating index:
public String TestIndex() throws IOException,SQLException
{
System.out.println("preparing dictionary");
String output="";
Long i=0l;
ResultSet rs = null;
URL u = this.getClass().getClassLoader(). getResource(SearchConstant.INDEX_DIRECTORY_DICTIONARYDETAILS);
String dirLoc = u.getPath().replace("%20", " ");
Directory index = FSDirectory.open(new File(dirLoc)); //new RAMDirectory();
StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_30);
IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_30,analyzer);
config.setOpenMode(OpenMode.CREATE);
IndexWriter w = new IndexWriter(index, config);
try {
String SQL = "Select * from test";
cm = new DbUtility();
rs = cm.getData(SQL);
// 1. create the index
while (rs.next()) {
Document doc = new Document();
doc.add(new Field("id",rs.getObject(1).toString() , Field.Store.YES, Field.Index.ANALYZED));
doc.add(new Field("Heading",rs.getObject(2).toString() , Field.Store.YES, Field.Index.ANALYZED));
w.addDocument(doc);
i = i + 1;
}
System.out.println("I " + i.toString());
}
catch (Exception e) {
System.out.println("I in Error " + i.toString());
System.out.println("Error while retrieving data: "+e.getMessage());
}
w.close();
rs.close();
return output;
}
I have a problem with Lucene Highlighter. I found some code on Stackoverflow and on other, but this code does not work in my program. This is a method where I try search and higlight words, but when I search something, program gives me exception.
Method:
private static void useIndex(String query, String field, String option)
throws ParseException, CorruptIndexException, IOException, InvalidTokenOffsetsException {
// StandardAnalyzer analyzer = new StandardAnalyzer();
Query q = new QueryParser(field, analyzer).parse(query);
int hitsPerPage = 5;
IndexReader reader = DirectoryReader.open(index);
IndexSearcher searcher = new IndexSearcher(reader);
TopScoreDocCollector collector = TopScoreDocCollector.create(hitsPerPage);
searcher.search(q, collector);
ScoreDoc[] hits = collector.topDocs().scoreDocs;
SimpleHTMLFormatter htmlFormatter = new SimpleHTMLFormatter();
Highlighter highlighter = new Highlighter(htmlFormatter, new QueryScorer(q));
// display results
System.out.println("Found " + hits.length + " hits for " + query);
for (int i = 0; i < hits.length; ++i) {
int docId = hits[i].doc;
Document d = searcher.doc(docId);
String docURL = d.get("url");
String docContent = d.get("content");
TokenStream tokenStream = TokenSources.getAnyTokenStream(reader, docId, "content", analyzer);
TextFragment[] frag = highlighter.getBestTextFragments(tokenStream, docContent, false, 4);
String docFrag="";
if ((frag[0] != null) && (frag[0].getScore() > 0)) {
docFrag=frag[0].toString();
}
model.addRow(new Object[] { docURL, findSilimar(docId), docFrag });
}
reader.close();
}
Exception:
Exception in thread "AWT-EventQueue-0" java.lang.NoClassDefFoundError: org/apache/lucene/index/memory/MemoryIndex
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.index.memory.MemoryIndex
I tried everything, but I don't know what is wrong.
P.S. Sorry for my English.
A NoClassDefFoundError means that class isn't in your classpath, so you should figure out what jar you need to add to get it. MemoryIndex is in: lucene-memory-x.x.x.jar
By the way, at a glance, it doesn't appear that this exception would be thrown in the code you've provided.
I want to give user the option to do case case sensitive or case insensitive search.
My idea is use a case sensitive analyzer to index the data and then use sensitive or insensitive analyzer to search depending on user input.
So I created my case sensitive analyzer and here is a simple of my code:
public final class CaseSensitiveStandardAnalyzer extends StopwordAnalyzerBase {
#Override
protected TokenStreamComponents createComponents(final String fieldName, final Reader reader) {
final StandardTokenizer src = new StandardTokenizer(matchVersion, reader);
src.setMaxTokenLength(maxTokenLength);
TokenStream tok = new StandardFilter(matchVersion, src);
tok = new StopFilter(matchVersion, tok, stopwords);
return new TokenStreamComponents(src, tok) {
#Override
protected void setReader(final Reader reader) throws IOException {
src.setMaxTokenLength(CaseSensitiveStandardAnalyzer.this.maxTokenLength);
super.setReader(reader);
}
};
}
For indexing I used this:
Analyzer analyzer = new CaseSensitiveStandardAnalyzer(Version.LUCENE_46);
IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_46,analyzer);
IndexWriter indexWriter = new IndexWriter(indexDir,config);
indexWriter.addDocument(document);
For searching I used:
Analyzer analyzer;
if(caseSentive)
analyzer = new CaseSensitiveStandardAnalyzer(Version.LUCENE_46);
else
analyzer = new StandardAnalyzer(Version.LUCENE_46);
QueryParser queryParser = new QueryParser(Version.LUCENE_46,"content", analyzer);
Query query = queryParser.parse(searchString);
//Search
TopDocs results = indexSearcher.search(query,10000);
ScoreDoc[] hits = results.scoreDocs;
When I tired this, the sensitive case worked, but the insensitive case didn't.
After more researching, I found that using a case-sensitive analyzer with a lower-care query will not work. Case-sensitive analyzer indexed work with case-sensitive query and case-insensitive analyzer indexed work with case-insensitive query, can anyone confirm this?
It seems to me the only reliable way to search both case-sensitive and case-insensitive is to index twice, one for each case, is this correct?
It seems to me the only reliable way to search both case-sensitive and case-insensitive is to index twice, one for each case, is this correct?
That would be a possible solution, but there are more optimal solutions for that use case: https://stackoverflow.com/a/2490441/867816
This might help, too: http://www.hascode.com/2014/07/lucene-by-example-specifying-analyzers-on-a-per-field-basis-and-writing-a-custom-analyzertokenizer/
I'm using Lucene 5.1.0. After Analyzing and indexing a document, I would like to get a list of all the terms indexed that belong to this specific document.
{
File[] files = FILES_TO_INDEX_DIRECTORY.listFiles();
for (File file : files) {
Document document = new Document();
Reader reader = new FileReader(file);
document.add(new TextField("fieldname",reader));
iwriter.addDocument(document);
}
iwriter.close();
IndexReader indexReader = DirectoryReader.open(directory);
int maxDoc=indexReader.maxDoc();
for (int i=0; i < maxDoc; i++) {
Document doc=indexReader.document(i);
String[] terms = doc.getValues("fieldname");
}
}
the terms return null. Is there a way to get the saved terms per document?
Here is a sample code for the answer, using a TokenStream
TokenStream ts= analyzer.tokenStream("myfield", reader);
// The Analyzer class will construct the Tokenizer, TokenFilter(s), and CharFilter(s),
// and pass the resulting Reader to the Tokenizer.
OffsetAttribute offsetAtt = ts.addAttribute(OffsetAttribute.class);
CharTermAttribute charTermAttribute = ts.addAttribute(CharTermAttribute.class);
try {
ts.reset(); // Resets this stream to the beginning. (Required)
while (ts.incrementToken()) {
// Use AttributeSource.reflectAsString(boolean)
// for token stream debugging.
System.out.println("token: " + ts.reflectAsString(true));
String term = charTermAttribute.toString();
System.out.println(term);
}
ts.end(); // Perform end-of-stream operations, e.g. set the final offset.
} finally {
ts.close(); // Release resources associated with this stream.
}
I want to do a search for a query within a file "fdictionary.txt" containing a list of words (230,000 words) written line by line. any suggestion why this code is not working?
The spell checking part is working and gives me the list of suggestions (I limited the length of the list to 1). what I want to do is to search that fdictionary and if the word is already in there, do not call spell checking. My Search function is not working. It does not give me error! Here is what I have implemented:
public class SpellCorrection {
public static File indexDir = new File("/../idxDir");
public static void main(String[] args) throws IOException, FileNotFoundException, CorruptIndexException, ParseException {
Directory directory = FSDirectory.open(indexDir);
SpellChecker spell = new SpellChecker(directory);
IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_20, null);
File dictionary = new File("/../fdictionary00.txt");
spell.indexDictionary(new PlainTextDictionary(dictionary), config, true);
String query = "red"; //kne, console
String correctedQuery = query; //kne, console
if (!search(directory, query)) {
String[] suggestions = spell.suggestSimilar(query, 1);
if (suggestions != null) {correctedQuery=suggestions[0];}
}
System.out.println("The Query was: "+query);
System.out.println("The Corrected Query is: "+correctedQuery);
}
public static boolean search(Directory directory, String queryTerm) throws FileNotFoundException, CorruptIndexException, IOException, ParseException {
boolean isIn = false;
IndexReader indexReader = IndexReader.open(directory);
IndexSearcher indexSearcher = new IndexSearcher(indexReader);
Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_20);
Term term = new Term(queryTerm);
Query termQuery = new TermQuery(term);
TopDocs hits = indexSearcher.search(termQuery, 100);
System.out.println(hits.totalHits);
if (hits.totalHits > 0) {
isIn = true;
}
return isIn;
}
}
where are you indexing the content from fdictionary00.txt?
You can search using IndexSearcher, only when you have index. If you are new to lucene, you might want to check some quick tutorials. (like http://lucenetutorial.com/lucene-in-5-minutes.html)
You never built the index.
You need to setup the index...
Directory directory = FSDirectory.open(indexDir);
Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_20);
IndexWriter writer = new IndexWriter(directory,analyzer,true,IndexWriter.MaxFieldLength.UNLIMITED );
You then need to create a document and add each term to the document as an analyzed field..
Document doc = new Document();
doc.Add(new Field("name", word , Field.Store.YES, Field.Index.ANALYZED));
Then add the document to the index
writer.AddDocument(doc);
writer.Optimize();
Now build the index and close the index writer.
writer.Commit();
writer.Close();
You could make your SpellChecker instance available in a service and use spellChecker.exist(word).
Be aware that the SpellChecker will not index words 2 characters or less. To get around this you can add them to the index after you have created it (add them into SpellChecker.F_WORD field).
If you want to add to your live index and make them available for exist(word) then you will need to add them to the SpellChecker.F_WORD field. Of course, because you're not adding to all the other fields such as gram/start/end etc then your word will not appear as a suggestion for other misspelled words.
In this case you'd have had to add the word into your file so when you re-create the index it would then be available as a suggestion. It would be great if the project made SpellChecker.createDocument(...) public/protected, rather than private, as this method accomplishes everything with adding words.
After all this your need to call spellChecker.setSpellIndex(directory).