We Keep Coding

Elasticsearch + Java - Can you further optimize this query?

We have an elasticsearch that contains millions of records and we are using it for a global searching. However, our query takes 2-4 seconds to return result. Can somebody help or advice how to further optimize the following query:
NativeSearchQueryBuilder query = new NativeSearchQueryBuilder()
.withQuery(QueryBuilders.boolQuery()
.should(QueryBuilders.termQuery("name", searchText))
.should(QueryBuilders.termQuery("key", searchText))
.should(QueryBuilders.termQuery("sort_alpha", searchText))
.should(QueryBuilders.termQuery("sort_number", searchText))
).withPageable(PageRequest.of(0, 100))
.withHighlightFields(new HighlightBuilder.Field("name"),
new HighlightBuilder.Field("key"),
new HighlightBuilder.Field("sort_alpha"),
new HighlightBuilder.Field("sort_number")
.build();

Jest elastic search Query builder query with multiple match strings

I am trying to query elastic search in Java using Jest. I am using a query builder to construct the query.
QueryBuilder matchQuery = QueryBuilders.boolQuery()
.must(QueryBuilders.matchQuery("id", "3434"))
.must(QueryBuilders.matchQuery("name", "name"))
.must(QueryBuilders.matchQuery("action", "login"))
.must(rangeQuery);
//i have this map now
Map<String , String> parameters = new HashMap<>();
parameters.put("id", "3433");
parameters.put("name", "name");
parameters.put("action", "login");
It would be great if some one can tell me if this can be constructed dynamically , like the matchQueries. For instance i would have three match queries , but i would have more. I can put my match query attributes in a Map. But if i iterate that how would we can define this boolQuery ? Anyone has any idea on this ?
Looking for something like this
QueryBuilder matchQuery = QueryBuilders.boolQuery()
.must(QueryBuilders.matchQuery(map.key, map.value))
.must(rangeQuery);
but how would i contruct this dynamically ?

You can simply iterate over your map and add each condition to the query, like this:
BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
for(String key : parameters.keySet()){
boolQuery.must(QueryBuilders.matchQuery(key, parameters.get(key)));
}
boolQuery.must(rangeQuery);

Elasticsearch 2+ JAVA API orFilter

I am trying to write some Java code to query an ES5 instance. In ES1, you could OR and AND filters, orFilter and andFilter. What is the equivalent in ES5? I have tried -
QueryBuilder fb1 = QueryBuilders.termQuery("term1", "value1");
QueryBuilder fb2 = QueryBuilders.termQuery("term1", "value2");
QueryBuilder fb3 = QueryBuilders.termQuery("term2", "value3");
QueryBuilder fb = QueryBuilders.boolQuery()
.must(fb1)
.should(fb2)
This gives results that match fb1, so this is not an OR
QueryBuilder fb = QueryBuilders.boolQuery()
.must(fb1)
.must(fb2)
This gives no results, so the AND worked.
I want to figure out ways of constructing AND, OR queries

In order to make an AND query, put all the conditions into "must".
QueryBuilder fb = QueryBuilders.boolQuery().must(fb1).must(fb2);
In order to make an OR query, put all the conditions into "should".
QueryBuilder fb = QueryBuilders.boolQuery().should(fb1).should(fb2);

From ElasticSearch documentation (https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-compound-queries.html):
QueryBuilder qb = boolQuery()
.must(termQuery("content", "test1"))
.must(termQuery("content", "test4"))
.mustNot(termQuery("content", "test2"))
.should(termQuery("content", "test3"))
.filter(termQuery("content", "test5"));
must seems like it might be the equivalent of the AND and should like it's the equivalent of the OR.

You can use BoolQueryBuiler for these filters :
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery().must(QueryBuilders.termQuery("customerId", customerId))
boolQueryBuilder.mustNot(QueryBuilders.termsQuery("reason", "failed"));
boolQueryBuilder.should(QueryBuilders.termsQuery("action", "deny")).minimumNumberShouldMatch(1);
QueryBuilder queryBuilder=boolQueryBuilder;
new SearchContext().prepareSearch(sc.getIndexNames()).setTypes(sc.getTypeName()).setSearchType(SearchType.DFS_QUERY_THEN_FETCH).setQuery(queryBuilder);

Grouping Solr results in Solr 3.6.1 API causes NullPointerException when parsing result

As long as I limit my query to:
SolrQuery solrQuery = new SolrQuery();
solrQuery.set("q", query); //where query is solr query string (e.g. *:*)
solrQuery.set("start", 0);
solrQuery.set("rows", 10);
everything works fine - results are returned and so on.
Things are getting worse when I try to group results by my field "Token_group" to avoid duplicates:
SolrQuery solrQuery = new SolrQuery();
solrQuery.set("q", query); //where query is solr query string (e.g. *:*)
solrQuery.set("start", 0);
solrQuery.set("rows", 10);
solrQuery.set("group", true);
solrQuery.set("group.field", "token_group");
solrQuery.set("group.ngroups", true);
solrQuery.set("group.limit", 20);
Using this results in HttpSolrServer no exceptions are being thrown, but trying to access results ends up in NPE.
My querying Solr method:
public SolrDocumentList query(SolrQuery query) throws SolrServerException {
QueryResponse response = this.solr.query(query); //(this.solr is handle to HttpSolrSelver)
SolrDocumentList list = response.getResults();
return list;
}
note that similar grouping (using the very same field) is made in our other apps (PHP) and works fine, so this is not a schema issue.

I solved my issue. In case someone needs this in future:
When you perform a group query, you should use different methods to get and parse results.
While in ungrouped queries
QueryResponse response = this.solr.query(query); //(this.solr is handle to HttpSolrSelver)
SolrDocumentList list = response.getResults();
will work, when you want to query for groups, it won't.
So, how do I make and parse query?
Below code for building query is perfectly fine:
SolrQuery solrQuery = new SolrQuery();
solrQuery.set("q", query); //where query is solr query string (e.g. *:*)
solrQuery.set("start", 0);
solrQuery.set("rows", 10);
solrQuery.set("group", true);
solrQuery.set("group.field", "token_group");
solrQuery.set("group.ngroups", true);
solrQuery.set("group.limit", 20);
where last four lines define that Solr should group results and parameters of grouping. In this case group.limit will define how many maximum results within a group you want, and rows will tell how many max results should be there.
Making grouped query looks like this:
List<GroupCommand> groupCommands = this.solr.query(query).getGroupResponse().getValues();
referring to documentation, GroupCommand contains info about grouping as well as list of results, divided by groups.
Okay, I want to get to the results. How to do it?
Well, in my example there's only one position in List<GroupCommand> groupCommands, so to get list of found groups within it:
GroupCommand groupCommand = groupCommands.get(0);
List<Group> groups = groupCommand.getValues();
This will result in list of groups. Each group contains its own SolrDocumentList. To get it:
for(Group g : groups){
SolrDocumentList groupList = g.getResult();
(...)
}
Having this, well just proceed with SolrDocumentList for each group.
I used grouping query to get list of distinct results. How to do it?
This was exacly my case. It seems easy but there's a tricky part that can catch you if you're refactoring already running code that uses getNumFound() from SolrDocumentList.
Just analyze my code:
/**
* Gets distinct resultlist from grouped query
*
* #param query
* #return results list
* #throws SolrServerException
*/
public SolrDocumentList queryGrouped(SolrQuery query) throws SolrServerException {
List<GroupCommand> groupCommands = this.solr.query(query).getGroupResponse().getValues();
GroupCommand groupCommand = groupCommands.get(0);
List<Group> groups = groupCommand.getValues();
SolrDocumentList list = new SolrDocumentList();
if(groups.size() > 0){
long totalNumFound = groupCommand.getNGroups();
int iteratorLimit = 1;
for(Group g : groups){
SolrDocumentList groupList = g.getResult();
list.add(groupList.get(0));
//I wanted to limit list to 10 records
if(iteratorLimit++ > 10){
break;
}
}
list.setNumFound(totalNumFound);
}
return list;
}

How to build FacetQuery using spring-data-solr which returns all documents (*:*) from solr index

I want to write a FacetQuery which may not have any criteria except one filter condition (fq). Following query is an example which I want to build using spring-data-solr API.
http://localhost:8983/solr/jpevents/select?q=*:*&fq=categoryIds:(1101)&facet=true&facet.mincount=1&facet.limit=1&facet.field=primaryCategoryId
How can I set query parameter (q=*:*) in FacetQuery?
Environment: I'm writing a Spring MVC based Search API using spring-data-solr 1.0.0.RELEASE with Solr 4.4.0 and Spring 3.2.4.RELEASE.

you can do this combining #Query and #Facet
#Facet(fields={"primaryCategoryId"}, minCount=1, limit=1)
#Query(value="*:*", filters="categoryIds:(?0)")
public FacetPage<JPEvents> XYZ(List<Long> categories, Pageable page);
or execute FacetQuery using SolrTemplate.
FacetQuery query = new SimpleFacetQuery(new SimpleStringCriteria("*:*"))
.setFacetOptions(new FacetOptions("primaryCategoryId")
.setFacetMinCount(1).setFacetLimit(1));
query.setPageRequest(pageable);
solrTemplate.queryForFacetPage(query, JPEvents.class);

I have done something like this :
public static void main()
{
String url = "http://localhost:8983/solr/autocomplete";
SolrServer solrServer = new HttpSolrServer(url);
SolrQuery query = new SolrQuery();
query.set("q", "*");
query.addFilterQuery("name:*");
query.setFacet(true);
query.addFacetField("name");
System.out.println(query);
QueryResponse queryResponse = solrServer.query(query);
List<FacetField> facetFields = queryResponse.getFacetFields();
FacetField cnameMainFacetField = queryResponse.getFacetField("name");
for (Count cnameAndCount : cnameMainFacetField.getValues()) {
String cnameMain = cnameAndCount.getName();
System.out.println(cnameMain);
System.out.println(cnameAndCount.getCount());
}
This gives me correct counts of faceted fields.
Hope you are able to understand what I am doing. Adding output for better understanding:
q=*&fq=name%3A*&facet=true&facet.field=name
a
10
an
7
w
7
m
6
and
5
c
5
p
5
d
4