I'm Using Lucence V3.1 & Java 1.6.
I'm trying to write code (using java and lucene) that allows me to do multi-field phrase search. However, i don't want the phrase to exactly match the value in the field. What i want is to check if the phrase is actually a substring of the value in the field. I tried the following but no luck yet:
IndexReader reader = IndexReader.Open("<lucene dir>");
Searcher searcher = new IndexSearcher(reader);
BooleanQuery booleanQuery = new BooleanQuery();
Query query1 = new TermQuery(new Term("<field-name>", "<text>"));
booleanQuery.add(query1, BooleanClause.Occur.SHOULD);
Hits hits = searcher.Search(booleanQuery);
Just use quotes? Like "this is the substring". This surely works with the lucene QueryParser
If to be used in a Query use a PhraseQuery. See also http://lucene.apache.org/core/old_versioned_docs/versions/3_1_0/api/core/org/apache/lucene/search/PhraseQuery.html
which analyzer u used while indexing??
if u used Standard Analyzer, you should not face a problem like this...
PS: always use same analyzer for both indexing and searching
Related
For example:
title:lucene+((author:jack)^300.0 (bookname:how to use lucene)^200.0 (price:[100 TO 200])^100.0)~1
Is there anyWay parse the lucene query string to Query Object like Query query = Function(String queryString) in lucene?
You can use the classic QueryParser to build a parser:
QueryParser parser = new QueryParser(default_field_name, analyzer);
If you provide field names in your query string (as you do in your example), then the default field name is not used.
The analyzer should typically be the same as the analyzer which was used to build the index. For example, the StandardAnalyzer:
Analyzer analyzer = new StandardAnalyzer();
And then you can use the string containing your query as follows:
Query query = parser.parse(your_query_string);
The demo code provided as part of Lucene shows an example of this approach. See lines 118 and 135 in the SearchFiles.html code.
Right now, I have successfully configured a basic Hibernate Search index to be able to search for full words on various fields of my JPA entity:
#Entity
#Indexed
class Talk {
#Field String title
#Field String summary
}
And my query looks something like this:
List<Talk> search(String text) {
FullTextEntityManager fullTextEntityManager = Search.getFullTextEntityManager(entityManager)
QueryBuilder queryBuilder = fullTextEntityManager.getSearchFactory().buildQueryBuilder().forEntity(Talk).get()
Query query = queryBuilder
.keyword()
.onFields("title", "summary")
.matching(text)
.createQuery()
FullTextQuery jpaQuery = fullTextEntityManager.createFullTextQuery(query, Talk)
return jpaQuery.getResultList()
}
Now I would like to fine-tune this setup so that when I search for "test" it still finds talks where title or summary contains "test" even as the prefix of another word. So talks titled "unit testing", or whose summary contains "testicle" should still appear in the search results, not just talks whose title or summary contains "test" as a full word.
I've tried to look at the documentation, but I can't figure out if I should change something to the way my entity is indexed, or whether it has something to do with the query. Note that I wanted to do something like the following, but then it's hard to search on several fields:
Query query = queryBuilder
.keyword().wildcard()
.onField("title")
.matching(text + "*")
.createQuery()
EDIT:
Based on Hardy's answer, I configured my entity like so:
#Indexed
#Entity
#AnalyzerDefs([
#AnalyzerDef(name = "ngram",
tokenizer = #TokenizerDef(factory = StandardTokenizerFactory.class),
filters = [
#TokenFilterDef(factory = LowerCaseFilterFactory.class),
#TokenFilterDef(factory = NGramFilterFactory.class,
params = [
#Parameter(name = "minGramSize",value = "3"),
#Parameter(name = "maxGramSize",value = "3")
])
])
])
class Talk {
#Field(analyzer=#Analyzer(definition="ngram")) String title
#Field(analyzer=#Analyzer(definition="ngram")) String summary
}
Thanks to that configuration, when I search for 'arti', I get Talks where title or summary contains words whose 'arti' is a subword of (artist, artisanal, etc.). Unfortunately, after those I also get Talks where title or summary contain words that contains subwords of my search term (arts, fart, etc.). There's probably some fine-tuning to eliminate those, but at least I get results sooner now, and they are in a sensible order.
There are multiple things you can do here. A lot can be done via the proper analyzing during index time.
For example, you want to apply a stemmer appropriate for your language. For English this is generally the Snowball stemmer.The idea is that during indexing all words are reduced to their stem, testing and tested to _test for example. This gets you a bit along your way.
The other thing you can look into is ngramm indexing. According to your description you want to find matching in unrelated words as well. The idea here is to index "subwords" of each words, so that they later can be found.
Regarding analyzers you want to look at the named analyzerssection of the Hibernate Search docs. The key here is the #AnalyzerDef annotation.
On the query side you can also apply some "tricks". Indeed you can use wildcard queries, however, if you are using the Hibernate Search query DSL, you cannot use a keyword query, but you need to use a wildcard query. Again, check the Hibernate Search docs.
You should use Ngram or EdgeNGram Filter for indexin as you correctly noted in your answer. But you should use different analyzer for your queries as suggested in lucene documentation (see search_analyzer):
https://www.elastic.co/guide/en/elasticsearch/guide/current/_index_time_search_as_you_type.html
This way your search query wouldn't be tokenized to ngrams and your results would be more like %text% or text% in SQL.
Unfortunately for unknown reasons Hibernate Search currently doesn't support search_analyzer specification on fields. You can only specific analyzer for indexing, which would be also used for search query analysis.
I plan to implement this functionality myself.
EDIT:
You can specify search-time analyzer (search_analyzer) like this:
List<Talk> search(String text) {
FullTextEntityManager fullTextEntityManager = Search.getFullTextEntityManager(entityManager)
EntityContext entityContext = fullTextEntityManager.getSearchFactory().buildQueryBuilder().forEntity(Talk);
entityContext.overridesForField("myField", "myNamedAnalyzerDef");
QueryBuilder queryBuilder = ec.get()
Query query = queryBuilder
.keyword()
.onFields("title", "summary")
.matching(text)
.createQuery()
FullTextQuery jpaQuery = fullTextEntityManager.createFullTextQuery(query, Talk)
return jpaQuery.getResultList()
}
I have used this technique to effectively simulate Lucene search_analyzer property.
In Lucene version 4.9 I used the EnglishAnalyzer for this. I think it is a English only implementation of the SnowballAnalyzer, but not 100% certain. I used it for both creating and searching the indexes. There is nothing special needed to use it.
Analyzer analyzer = new EnglishAnalyzer(Version.LUCENE_4_9);
IndexWriterConfig iwc = new IndexWriterConfig(Version.LUCENE_4_9, analyzer);
and
analyzer = new EnglishAnalyzer(Version.LUCENE_4_9);
parser = new StandardQueryParser(analyzer);
You can see it in action at Guided Code Search. This runs exclusively off Lucene.
Lucene can be integrated into Hibernate searches, but I haven't yet tried to do that myself. I seems like it would be powerful, but I don't know: See Apache Lucene™ Integration.
I've also read that lucene can be patched into SQL engines, but I haven't tried that either. Example: Indexing Databases with Lucene.
I am using Hibernate Search to search for titles of tv shows on my web app.
I can use the method fuzzy() on keyword() in order to perfom fuzzy searches on keywords, but I need to take into account the whole title, so I am using phrase() instead of keyword(). The method fuzzy() is not defined for phrase(), so I was wondering if there is an easy way to achieve fuzzy searches on phrases using Hibernate Search.
If you just need a PhraseQuery with slop (that is, extra words thrown in), then you can set the slop on a phrase query like:
queryBuilder.phrase()
.setSlop(2)
.onField("myField")
.sentance("this sentence missing something")
.createQuery();
However, I'm not aware of anything in the Hibernate APIs that supports embedding fuzzy queries in phrases, but in Lucene, you can work with the SpanQuery API to build that. SpanMultiTermQueryWrapper and SpanNearQuery, in particular, are what you would need. Something like:
FuzzyQuery query1 = new FuzzyQuery(new Term("field", "fuzy"));
FuzzyQuery query2 = new FuzzyQuery(new Term("field", "phrse"));
Query wrappedQuery1 = new SpanMultiTermQueryWrapper<FuzzyQuery>(query1);
Query wrappedQuery2 = new SpanMultiTermQueryWrapper<FuzzyQuery>(query2);
SpanQuery[] clauses = {wrappedQuery1, wrappedQuery2};
SpanNearQuery(clauses, 0, true);
Title asks it all... I want to do a multi field - phrase search in Lucene.. How to do it ?
for example :
I have fields as String s[] = {"title","author","content"};
I want to search harry potter across all fields.. How do I do it ?
Can someone please provide an example snippet ?
Use MultiFieldQueryParser, its a QueryParser which constructs queries to search multiple fields..
Other way is to use Create a BooleanQuery consisting of TermQurey (in your case phrase query).
Third way is to include the content of other fields into your default content field.
Add
Generally speaking, querying on multiple fields isn’t the best practice for user-entered queries. More commonly, all words you want searched are indexed into a contents or keywords field by combining various fields.
Update
Usage:
Query query = MultiFieldQueryParser.parse(Version.LUCENE_30, new String[] {"harry potter","harry potter","harry potter"}, new String[] {"title","author","content"},new SimpleAnalyzer());
IndexSearcher searcher = new IndexSearcher(...);
Hits hits = searcher.search(query);
The MultiFieldQueryParser will resolve the query in this way: (See javadoc)
Parses a query which searches on the
fields specified. If x fields are
specified, this effectively
constructs:
(field1:query1) (field2:query2)
(field3:query3)...(fieldx:queryx)
Hope this helps.
intensified googling revealed this :
http://lucene.472066.n3.nabble.com/Phrase-query-on-multiple-fields-td2292312.html.
Since it is latest and best, I'll go with his approach I guess.. Nevertheless, it might help someone who is looking for something like I am...
You need to use MultiFieldQueryParser with escaped string. I have tested it with Lucene 8.8.1 and it's working like magic.
String queryStr = "harry potter";
queryStr = "\"" + queryStr.trim() + "\"";
Query query = new MultiFieldQueryParser(new String[]{"title","author","content"}, new StandardAnalyzer()).parse(queryStr);
System.out.println(query);
It will print.
(title:"harry potter") (author:"harry potter") (content:"harry potter")
I have a collection of documents consisting of several fields, and I need to perform queries with several terms coming from multiple fields.
What do you suggest me to use ? MultiFieldQueryParser or MultiPhraseQuery ?
thanks
How about BooleanQuery?
http://lucene.apache.org/java/3_0_2/api/core/org/apache/lucene/search/BooleanQuery.html
Choice of Analyzer
First of all, watch out which analyzer you are using. I was stumped for a while only to realise that the StandardAnalyzer filters out common words like 'the' and 'a'. This is a problem when your field has the value 'A'. You might want to consider the KeywordAnalyzer:
See this post around the analyzer.
// Create an analyzer:
// NOTE: We want the keyword analyzer so that it doesn't strip or alter any terms:
// In our example, the Standard Analyzer removes the term 'A' because it is a common English word.
// https://stackoverflow.com/a/9071806/231860
KeywordAnalyzer analyzer = new KeywordAnalyzer();
Query Parser
Next, you can either create your query using the QueryParser:
See this post around overriding the default operator.
// Create a query parser without a default field in this example (the first argument):
QueryParser queryParser = new QueryParser("", analyzer);
// Optionally, set the default operator to be AND (we leave it the default OR):
// https://stackoverflow.com/a/9084178/231860
// queryParser.setDefaultOperator(QueryParser.Operator.AND);
// Parse the query:
Query multiTermQuery = queryParser.parse("field_name1:\"field value 1\" AND field_name2:\"field value 2\"");
Query API
Or you can achieve the same by constructing the query yourself using their API:
See this tutorial around creating the BooleanQuery.
BooleanQuery multiTermQuery = new BooleanQuery();
multiTermQuery.add(new TermQuery(new Term("field_name1", "field value 1")), BooleanClause.Occur.MUST);
multiTermQuery.add(new TermQuery(new Term("field_name2", "field value 2")), BooleanClause.Occur.MUST);
Delete the Documents that Match the Query
Then we finally pass the query to the writer to delete documents that match the query:
See my answer here, related to this answer.
See the answer to this question.
// Remove the document by using a multi key query:
// http://www.avajava.com/tutorials/lessons/how-do-i-combine-queries-with-a-boolean-query.html
writer.deleteDocuments(multiTermQuery);