I have n users (at the moment only 1000 it should be more than 100'000 in the future so it shouldn't be too inefficient).
The users are visualized as m (usually 1-10) documents. In my first attempt I concat the documents in one String. The String contains all different letters, numbers and special character ("/", "\r\n", "&", "+", ...) everything can be possible.
In the end I want to have a nxn matrix with a similarity score that compare each document to each other. (the diagonal should be the highest score because it's most similar to himself)
Example:
user/user| userA | userB | userC |
userA | 1.00 | 0.94 | 0.33 |
userB | 0.92 | 1.00 | 0.12 |
userC | 0.35 | 0.22 | 1.00 |
That's what I want to achieve. I'm using Lucene but I think I can switch to another framework too when Lucene doesn't provide it.
I have done this:
public class Similarity {
public static void main(String[] args) throws IOException {
UserFactory userFactory = UserFactory.getInstance();
UserBase base = userFactory.getUserFromCsv("user.csv");
Similarity sim = new Similarity();
sim.indexing(base);
}
private StandardAnalyzer analyzer = null;
private Directory index = null;
private IndexWriterConfig config = null;
private IndexWriter w = null;
public Similarity() throws IOException {
analyzer = new StandardAnalyzer();
index = new RAMDirectory();
config = new IndexWriterConfig(analyzer);
w = new IndexWriter(index, config);
}
public void query(User user){
// How??
}
public void indexing(UserBase base) throws IOException {
for(User user : base.getUsers()){
addDoc(w, user.getText(), user.getId());
}
}
w.close();
}
private void addDoc(IndexWriter w, String text, String id) throws IOException {
Document doc = new Document();
doc.add(new TextField("text", text, Field.Store.YES));
doc.add(new StringField("id", id, Field.Store.YES));
w.addDocument(doc);
}
}
The User class is really simple and only have 2 fields text and id. I want to compare the text for each user: user.getText()
My first attempt was to try a normal Query parser:
public void query(User user){
IndexReader reader = DirectoryReader.open(index);
IndexSearcher searcher = new IndexSearcher(reader);
Query q = new QueryParser("text",analyzer).parse(user.getText());
TopDocs docs = searcher.search(q,10000);
for(ScoreDoc hit : docs.scoreDocs){
Document d = searcher.doc(hit.doc);
System.out.println(d.getId() + " " + hit.score);
}
}
The problem in this attempt was the following:
I have to replace all special character before I could let it run (replace "/" - probably more I only tested it with one document and I never know which character could be a problem in the future)
The most similar user is not the user I used as a query - more worse he doesn't even appear in the list...
But when I use only a subset (first 20 character) he find the user I used as a query - strange? probably a flaw in my thinking...
But yeah I don't think that's the best approach to solve this problem...
I also tried it with MoreLikeThis (already deleted the code sorry... but it didn't work I couldn't even get a result it was always empty even if I compare everything with everyone).
I'm a beginner with Lucene so yeah could be a few flaws in my thinking.
Related
I am trying to do autocomplete using lucene search functionality. I have the following code which searches by the query prefix but along with that it also gives me all the sentences containing that word while I want it to display only sentence or word starting exactly with that prefix.
ex: m
--holiday mansion houseboat
--eye muscles
--movies of all time
--machine
I want it to show only last 2 queries. How to do it am stucked here also I am new to lucene. Please can any one help me in this. Thanks in advance.
addDoc(IndexWriter w, String title, String isbn) throws IOException {
Document doc = new Document();
doc.add(new Field("title", title, Field.Store.YES, Field.Index.ANALYZED));
// use a string field for isbn because we don't want it tokenized
doc.add(new Field("isbn", isbn, Field.Store.YES, Field.Index.ANALYZED));
w.addDocument(doc);
}
Main:
try {
// 0. Specify the analyzer for tokenizing text.
// The same analyzer should be used for indexing and searching
StandardAnalyzer analyzer = new StandardAnalyzer();
// 1. create the index
Directory index = FSDirectory.open(new File(indexDir));
IndexWriter writer = new IndexWriter(index, new StandardAnalyzer(Version.LUCENE_30), true, IndexWriter.MaxFieldLength.UNLIMITED); //3
for (int i = 0; i < source.size(); i++) {
addDoc(writer, source.get(i), + (i + 1) + "z");
}
writer.close();
// 2. query
Term term = new Term("title", querystr);
//create the term query object
PrefixQuery query = new PrefixQuery(term);
// 3. search
int hitsPerPage = 20;
IndexReader reader = IndexReader.open(index);
IndexSearcher searcher = new IndexSearcher(reader);
TopScoreDocCollector collector = TopScoreDocCollector.create(hitsPerPage, true);
searcher.search(query, collector);
ScoreDoc[] hits = collector.topDocs().scoreDocs;
// 4. Get results
for (int i = 0; i < hits.length; ++i) {
int docId = hits[i].doc;
Document d = searcher.doc(docId);
System.out.println(d.get("title"));
}
reader.close();
} catch (Exception e) {
System.out.println("Exception (LuceneAlgo.getSimilarString()) : " + e);
}
}
}
I see two solutions:
as suggested by Yahnoosh, save the title field twice, Once as TextField (=analyzed) and once as StringField (not analyzed)
save it just as TextField, but When Querying use SpanFirstQuery
// 2. query
Term term = new Term("title", querystr);
//create the term query object
PrefixQuery pq = new PrefixQuery(term);
SpanQuery wrapper = new SpanMultiTermQueryWrapper<PrefixQuery>(pq);
Query final = new SpanFirstQuery(wrapper, 1);
If I understand your scenario correctly, you want to autocomplete on the title field.
The solution is to have two fields: one analyzed, to enable querying over it, one non-analyzed to have titles indexed without breaking them into individual terms.
Your autocomplete logic should issue prefix queries against the non-analyzed field to match only on the first word. Your term queries should be issued against the analyzed field for matches within the title.
I hope that makes sense.
I have a web application which stores customers usernames, emails and phone numbers.
I want customers to search for other users using email, phone or username for a start just to understand the whole lucene concept. then later on i will add functionality to search within a user an item he posts. I am following this example on www.lucenetutorial.com/lucene-in-5-minutes.html
public class HelloLucene {
public static void main(String[] args) throws IOException, ParseException {
// 0. Specify the analyzer for tokenizing text.
// The same analyzer should be used for indexing and searching
StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_40);
// 1. create the index
Directory index = new RAMDirectory();
IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_40, analyzer);
IndexWriter w = new IndexWriter(index, config);
addDoc(w, "Lucene in Action", "193398817");
addDoc(w, "Lucene for Dummies", "55320055Z");
addDoc(w, "Managing Gigabytes", "55063554A");
addDoc(w, "The Art of Computer Science", "9900333X");
w.close();
// 2. query
String querystr = args.length > 0 ? args[0] : "lucene";
// the "title" arg specifies the default field to use
// when no field is explicitly specified in the query.
Query q = new QueryParser(Version.LUCENE_40, "title", analyzer).parse(querystr);
// 3. search
int hitsPerPage = 10;
IndexReader reader = DirectoryReader.open(index);
IndexSearcher searcher = new IndexSearcher(reader);
TopScoreDocCollector collector = TopScoreDocCollector.create(hitsPerPage, true);
searcher.search(q, collector);
ScoreDoc[] hits = collector.topDocs().scoreDocs;
// 4. display results
System.out.println("Found " + hits.length + " hits.");
for(int i=0;i<hits.length;++i) {
int docId = hits[i].doc;
Document d = searcher.doc(docId);
System.out.println((i + 1) + ". " + d.get("isbn") + "\t" + d.get("title"));
}
// reader can only be closed when there
// is no need to access the documents any more.
reader.close();
}
private static void addDoc(IndexWriter w, String title, String isbn) throws IOException {
Document doc = new Document();
doc.add(new TextField("title", title, Field.Store.YES));
// use a string field for isbn because we don't want it tokenized
doc.add(new StringField("isbn", isbn, Field.Store.YES));
w.addDocument(doc);
}
}
I want new customers to be added to index automatically on registration. customerId is timestamp. so should i add a new document for each field on the customers details or should i concatenate all fields into a string and add as a single document? Please go easy on me I am really new.
This is a good place to start with Lucene indexing mechanism
http://www.ibm.com/developerworks/library/wa-lucene/
In the bottom line when lucene index the document, it first converts it into lucene document form. This lucene document comprises of set of fields and each field is a set of terms. Term are nothing but stream of bytes.
The document which is to be index to pass to analyzer which forms these terms out of it, and these terms keywords which are match during searching process.
When we perform a search process the query is analyzed through the same analyzer and then is match against the terms.
So you dont have to create a document for each field, rather you should create a single document for each user.
I'm using lucene 4.0 with java. I'm trying to search for a string inside a string. If we look at the lucene hello world example, I wish to find the text "lucene" inside the phrase "inLuceneAction". I want it to find me two matches in this case instead of one.
Any Idea on how to do it?
Thanks
public class HelloLucene {
public static void main(String[] args) throws IOException, ParseException {
// 0. Specify the analyzer for tokenizing text.
// The same analyzer should be used for indexing and searching
StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_40);
// 1. create the index
Directory index = new RAMDirectory();
IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_40, analyzer);
IndexWriter w = new IndexWriter(index, config);
addDoc(w, "inLuceneAction", "193398817");
addDoc(w, "Lucene for Dummies", "55320055Z");
addDoc(w, "Managing Gigabytes", "55063554A");
addDoc(w, "The Art of Computer Science", "9900333X");
w.close();
// 2. query
String querystr = args.length > 0 ? args[0] : "lucene";
// the "title" arg specifies the default field to use
// when no field is explicitly specified in the query.
Query q = new QueryParser(Version.LUCENE_40, "title", analyzer).parse(querystr);
// 3. search
int hitsPerPage = 10;
IndexReader reader = DirectoryReader.open(index);
IndexSearcher searcher = new IndexSearcher(reader);
TopScoreDocCollector collector = TopScoreDocCollector.create(hitsPerPage, true);
searcher.search(q, collector);
ScoreDoc[] hits = collector.topDocs().scoreDocs;
// 4. display results
System.out.println("Found " + hits.length + " hits.");
for(int i=0;i<hits.length;++i) {
int docId = hits[i].doc;
Document d = searcher.doc(docId);
System.out.println((i + 1) + ". " + d.get("isbn") + "\t" + d.get("title"));
}
// reader can only be closed when there
// is no need to access the documents any more.
reader.close();
}
private static void addDoc(IndexWriter w, String title, String isbn) throws IOException {
Document doc = new Document();
doc.add(new TextField("title", title, Field.Store.YES));
// use a string field for isbn because we don't want it tokenized
doc.add(new StringField("isbn", isbn, Field.Store.YES));
w.addDocument(doc);
}
}
If you index the terms in the default way, meaning inLuceneAction is one term, Lucene won't be able to seek to this term given Lucene because it has a different prefix. Analyze this string so that it results in three indexed terms: in Lucene Action and then you'll have it fetched. You'll either find a ready-made analyzer for this or you'll have to write your own. Writing own analyzers is a bit out of scope for a single StackOverflow answer, but an excellent place to start is the package info at the bottom of the org.apache.lucene.analysis package Javadoc page.
In short, I am in need to exchange the mapping of multiple field and values from one Index to the resulting Index.
The following is the scenario.
Index 1 Structure
[Field => Values] [Stored]
Doc 1
keys => keyword1;
Ids => id1, id1, id2, id3, id7, id11, etc..
Doc 2
keys => keyword2;
Ids => id3, id11, etc..
Index 2 Structure
[Field => Values] [Stored]
Doc 1
ids => id1
keys => keyword1, keyword1
Doc 3
ids => id3
keys => keyword1, keyword2, etc..
Please note that the keys<->ids mapping is reversed in the resulting Index.
What do you think the most effective way to accomplish this in terms of time complexity? ..
The only way I could think of is that..
1) index1Reader.terms();
2) Process only terms belonging to "Ids" field
3) For each term, get TermDocs
4) For each doc, load it, get "keys" field info
5) Create a new Lucene Doc, add 'Id', multi Keys, write it to index2.
6) Go to step 2.
Since the fields are stored, I'm sure that there are multiple ways of doing it.
Please guide me with any performance techniques. Even the slightest improvement will have a huge impact in my scenario considering that the Index1 size is ~ 6GB.
Total no. of unique keywords: 18 million;
Total no. of unique ids: 0.9 million
Interesting UPDATE
Optimization 1
While adding a new doc, instead of creating multiple duplicate 'Field' objects, creating a single StringBuffer with " " delimiter, and then adding entire as a single Field seems to have up to 25% improvement.
UPDATE 2: Code
public void go() throws IOException, ParseException {
String id = null;
int counter = 0;
while ((id = getNextId()) != null) { // this method is not taking time..
System.out.println("Node id: " + id);
updateIndex2DataForId(id);
if(++counter > 10){
break;
}
}
index2Writer.close();
}
private void updateIndex2DataForId(String id) throws ParseException, IOException {
// Get all terms containing the node id
TermDocs termDocs = index1Reader.termDocs(new Term("id", id));
// Iterate
Document doc = new Document();
doc.add(new Field("id", id, Store.YES, Index.NOT_ANALYZED));
int docId = -1;
while (termDocs.next()) {
docId = termDocs.doc();
doc.add(getKeyDataAsField(docId, Store.YES, Index.NOT_ANALYZED));
}
index2Writer.addDocument(doc);
}
private Field getKeyDataAsField(int docId, Store storeOption, Index indexOption) throws CorruptIndexException,
IOException {
Document doc = index1Reader.document(docId, fieldSelector); // fieldSel has "key"
Field f = new Field("key", doc.get("key"), storeOption, indexOption);
return f;
}
Usage of FieldCache worked like a charm... But, we need to allot more and more RAM to accommodate all the fields on the heap.
I've updated the above updateIndex2DataForId() with the following snippet..
private void updateIndex2DataForId(String id) throws ParseException, IOException {
// Get all terms containing the node id
TermDocs termDocs = index1Reader.termDocs(new Term("id", id));
// Iterate
Document doc = new Document();
doc.add(new Field("id", id, Store.YES, Index.NOT_ANALYZED));
int docId = -1;
StringBuffer buffer = new StringBuffer();
while (termDocs.next()) {
docId = termDocs.doc();
buffer .append(keys[docId] + " "); // keys[] is pre-populated using FieldCache
}
doc.add(new Field("id", buffer.trim().toString(), Store.YES, Index.ANALYZED));
index2Writer.addDocument(doc);
}
String[] keys = FieldCache.DEFAULT.getStrings(index1Reader, "keywords");
It made everything faster, I cannot tell you the exact metrics but I must say very substantial.
Now the program is completing in a bit of reasonable time. Anyways, further guidance is highly appreciated.
What is the best way to find out which terms in a query matched against a given document returned as a hit in lucene?
I have tried a weird method involving hit highlighting package in lucene contrib and also a method that searches for every word in the query against the top most document ("docId: xy AND description: each_word_in_query").
Do not get satisfactory results?
Hit highlighting does not report some of the words that matched for a document other than the first one.
I'm not sure if the second approach is the best alternative.
The method explain in the Searcher is a nice way to see which part of a query was matched and how it affects the overall score.
Example taken from the book Lucene In Action 2nd Edition:
public class Explainer {
public static void main(String[] args) throws Exception {
if (args.length != 2) {
System.err.println("Usage: Explainer <index dir> <query>");
System.exit(1);
}
String indexDir = args[0];
String queryExpression = args[1];
Directory directory = FSDirectory.open(new File(indexDir));
QueryParser parser = new QueryParser(Version.LUCENE_CURRENT,
"contents", new SimpleAnalyzer());
Query query = parser.parse(queryExpression);
System.out.println("Query: " + queryExpression);
IndexSearcher searcher = new IndexSearcher(directory);
TopDocs topDocs = searcher.search(query, 10);
for (int i = 0; i < topDocs.totalHits; i++) {
ScoreDoc match = topDocs.scoreDocs[i];
Explanation explanation = searcher.explain(query, match.doc);
System.out.println("----------");
Document doc = searcher.doc(match.doc);
System.out.println(doc.get("title"));
System.out.println(explanation.toString());
}
}
}
This will explain the score of each document that matches the query.
Not tried yet, but have a look at the implementation of org.apache.lucene.search.highlight.QueryTermExtractor.