We Keep Coding

Do not retrieve all columns with Elastic search query

I have an Elastic search query and I would like to retrieve a certain column, not all.
I make my request in java with BoolQUEryBuilder which gives:
BoolQueryBuilder query = boolQuery();
query.must(wildcardQuery('value', value + "*"));
return findAll(query);
The method findAll :
protected List<T> findAll(final BoolQueryBuilder query) {
Query searchQuery = (new NativeSearchQueryBuilder()).withQuery(query).build();
SearchHits<T> searchHits = this.elasticsearchRestTemplate.search(searchQuery, this.getClazz(), this.elasticsearchRestTemplate.getIndexCoordinatesFor(this.getClazz()));
return (List)SearchHitSupport.unwrapSearchHits(searchHits);
}
I would like to add a filter on the columns. To illustrate in SQL this gives:
Select column_one, column_two from table;

Refer source filtering to fetch only few fields from Elasticsearch query results.
As explained in the same document example below code shows which fields to include and which to exclude.
String[] includeFields = new String[] {"title", "innerObject.*"};
String[] excludeFields = new String[] {"user"};
sourceBuilder.fetchSource(includeFields, excludeFields);

With Spring Data Elasticsearch, you should try this instead:
...
//include only specific fields
final SourceFilter sourceFilter = new FetchSourceFilter(new String[]{"column_one", "column_two"}, null);
// assemble the query
Query searchQuery = new NativeSearchQueryBuilder().withQuery(query).build();
searchQuery.addSourceFilter(sourceFilter);
...

How to define custom analyzer to do global search with hibernate-search and elasticsearch

I have an implementation of hibernate-search-orm (5.9.0.Final) with hibernate-search-elasticsearch (5.9.0.Final).
I defined a custom analyzer on an entity (see beelow) and I indexed two entities :
id: "1"
title: "Médiatiques : récit et société"
abstract:...
id: "2"
title: "Mediatique Com'7"
abstract:...
The search works fine when I search on title field :
"title:médiatique" => 2 results.
"title:mediatique" => 2 results.
My problem is when I do a global search with accents (or not) :
search on "médiatique => 1 result (id:1)
search on "mediatique => 1 result (id:2)
Is there a way to resolve this?
Thanks.
Entity definition:
#Entity
#Table(name="bibliographic")
#DynamicUpdate
#DynamicInsert
#Indexed(index = "bibliographic")
#FullTextFilterDefs({
#FullTextFilterDef(name = "fieldsElasticsearchFilter",
impl = FieldsElasticsearchFilter.class)
})
#AnalyzerDef(name = "customAnalyzer",
tokenizer = #TokenizerDef(factory = StandardTokenizerFactory.class),
filters = {
#TokenFilterDef(factory = LowerCaseFilterFactory.class),
#TokenFilterDef(factory = ASCIIFoldingFilterFactory.class),
})
#Analyzer(definition = "customAnalyzer")
public class BibliographicHibernate implements Bibliographic {
...
#Column(name="title", updatable = false)
#Fields( {
#Field,
#Field(name = "titleSort", analyze = Analyze.NO, store = Store.YES)
})
#SortableField(forField = "titleSort")
private String title;
...
}
Search method :
FullTextEntityManager ftem = Search.getFullTextEntityManager(entityManager);
QueryBuilder qb = ftem.getSearchFactory().buildQueryBuilder().forEntity(Bibliographic.class).get();
QueryDescriptor q = ElasticsearchQueries.fromQueryString(queryString);
FullTextQuery query = ftem.createFullTextQuery(q, Bibliographic.class).setFirstResult(start).setMaxResults(rows);
if (filters!=null){
filters.stream().map((filter) -> filter.split(":")).forEach((f) -> {
query.enableFullTextFilter("fieldsElasticsearchFilter")
.setParameter("field", f[0])
.setParameter("value", f[1]);
}
);
}
if (facetFields!=null){
facetFields.stream().map((facet) -> facet.split(":")).forEach((f) ->{
query.getFacetManager()
.enableFaceting(qb.facet()
.name(f[0])
.onField(f[0])
.discrete()
.orderedBy(FacetSortOrder.COUNT_DESC)
.includeZeroCounts(false)
.maxFacetCount(10)
.createFacetingRequest() );
}
);
}
List<Bibliographic> bibs = query.getResultList();

To be honest I'm more surprised document 1 would match at all, since there's a trailing "s" on "Médiatiques" and you don't use any stemmer.
You are in a special case here: you are using a query string and passing it directly to Elasticsearch (that's what ElasticsearchQueries.fromQueryString(queryString) does). Hibernate Search has very little impact on the query being run, it only impacts the indexed content and the Elasticsearch mapping here.
When you run a QueryString query on Elasticsearch and you don't specify any field, it uses all fields in the document. I wouldn't bet that the analyzer used when analyzing your query is the same analyzer that you defined on your "title" field. In particular, it may not be removing accents.
An alternative solution would be to build a simple query string query using the QueryBuilder. The syntax of queries is a bit more limited, but is generally enough for end users. The code would look like this:
FullTextEntityManager ftem = Search.getFullTextEntityManager(entityManager);
QueryBuilder qb = ftem.getSearchFactory().buildQueryBuilder().forEntity(Bibliographic.class).get();
Query q = qb.simpleQueryString()
.onFields("title", "abstract")
.matching(queryString)
.createQuery();
FullTextQuery query = ftem.createFullTextQuery(q, Bibliographic.class).setFirstResult(start).setMaxResults(rows);
Users would still be able to target specific fields, but only in the list you provided (which, by the way, is probably safer, otherwise they could target sort fields and so on, which you probably don't want to allow). By default, all the fields in that list would be targeted.
This may lead to the exact same result as the query string, but the advantage is, you can override the analyzer being used for the query. For instance:
FullTextEntityManager ftem = Search.getFullTextEntityManager(entityManager);
QueryBuilder qb = ftem.getSearchFactory().buildQueryBuilder().forEntity(Bibliographic.class)
.overridesForField("title", "customAnalyzer")
.overridesForField("abstract", "customAnalyzer")
.get();
Query q = qb.simpleQueryString()
.onFields("title", "abstract")
.matching(queryString)
.createQuery();
FullTextQuery query = ftem.createFullTextQuery(q, Bibliographic.class).setFirstResult(start).setMaxResults(rows);
... and this will use your analyzer when querying.
As an alternative, you can also use a more advanced JSON query by replacing ElasticsearchQueries.fromQueryString(queryString) with ElasticsearchQueries.fromJsonQuery(json). You will have to craft the JSON yourself, though, taking some precautions to avoid any injection from the user (use Gson to build the Json), and taking care to follow the Elasticsearch query syntax.
You can find more information about simple query string queries in the official documentation.
Note: you may want to add FrenchMinimalStemFilterFactory to your list of token filters in your custom analyzer. It's not the cause of your problem, but once you manage to use your analyzer in search queries, you will very soon find it useful.

MongoDb in Java to find users

I have these three users in my DB with the following keys:
Jon-111
Jon-222
Jon-333
The code I wrote currently can only find one user in the DB by his key.
How can I return all these users?
public void getMongoDB() {
DB db = mongo.getDB("people");
DBCollection table = db.getCollection("users");
BasicDBObject searchQuery = new BasicDBObject();
searchQuery.put("key", "Jon-111"); // what do I do here to return all the 'Jon' users?
DBCursor cursor = table.find(searchQuery);
int dbSize = cursor.size(); // size is 1
while (cursor.hasNext()) {
logger.debug(""+cursor.next()+"\n");
}
logger.debug("users table");
}
Thanks in advance

If you want your search to be 'Starts with Jon', try:
searchQuery.put("key", java.util.regex.Pattern.compile("^Jon"));
You will need to use regex to do this kind of query, as shown in my example.
Some more examples for your benefit:
'Ends with Jon':
searchQuery.put("key", java.util.regex.Pattern.compile("Jon$"));
'Contains Jon':
searchQuery.put("key", java.util.regex.Pattern.compile(".*Jon.*"));

You can use regex Pattern to pass in the value. Something like below:
searchQuery.put("key", java.util.regex.Pattern.compile("Jon-\\.*"));

Grouping Solr results in Solr 3.6.1 API causes NullPointerException when parsing result

As long as I limit my query to:
SolrQuery solrQuery = new SolrQuery();
solrQuery.set("q", query); //where query is solr query string (e.g. *:*)
solrQuery.set("start", 0);
solrQuery.set("rows", 10);
everything works fine - results are returned and so on.
Things are getting worse when I try to group results by my field "Token_group" to avoid duplicates:
SolrQuery solrQuery = new SolrQuery();
solrQuery.set("q", query); //where query is solr query string (e.g. *:*)
solrQuery.set("start", 0);
solrQuery.set("rows", 10);
solrQuery.set("group", true);
solrQuery.set("group.field", "token_group");
solrQuery.set("group.ngroups", true);
solrQuery.set("group.limit", 20);
Using this results in HttpSolrServer no exceptions are being thrown, but trying to access results ends up in NPE.
My querying Solr method:
public SolrDocumentList query(SolrQuery query) throws SolrServerException {
QueryResponse response = this.solr.query(query); //(this.solr is handle to HttpSolrSelver)
SolrDocumentList list = response.getResults();
return list;
}
note that similar grouping (using the very same field) is made in our other apps (PHP) and works fine, so this is not a schema issue.

I solved my issue. In case someone needs this in future:
When you perform a group query, you should use different methods to get and parse results.
While in ungrouped queries
QueryResponse response = this.solr.query(query); //(this.solr is handle to HttpSolrSelver)
SolrDocumentList list = response.getResults();
will work, when you want to query for groups, it won't.
So, how do I make and parse query?
Below code for building query is perfectly fine:
SolrQuery solrQuery = new SolrQuery();
solrQuery.set("q", query); //where query is solr query string (e.g. *:*)
solrQuery.set("start", 0);
solrQuery.set("rows", 10);
solrQuery.set("group", true);
solrQuery.set("group.field", "token_group");
solrQuery.set("group.ngroups", true);
solrQuery.set("group.limit", 20);
where last four lines define that Solr should group results and parameters of grouping. In this case group.limit will define how many maximum results within a group you want, and rows will tell how many max results should be there.
Making grouped query looks like this:
List<GroupCommand> groupCommands = this.solr.query(query).getGroupResponse().getValues();
referring to documentation, GroupCommand contains info about grouping as well as list of results, divided by groups.
Okay, I want to get to the results. How to do it?
Well, in my example there's only one position in List<GroupCommand> groupCommands, so to get list of found groups within it:
GroupCommand groupCommand = groupCommands.get(0);
List<Group> groups = groupCommand.getValues();
This will result in list of groups. Each group contains its own SolrDocumentList. To get it:
for(Group g : groups){
SolrDocumentList groupList = g.getResult();
(...)
}
Having this, well just proceed with SolrDocumentList for each group.
I used grouping query to get list of distinct results. How to do it?
This was exacly my case. It seems easy but there's a tricky part that can catch you if you're refactoring already running code that uses getNumFound() from SolrDocumentList.
Just analyze my code:
/**
* Gets distinct resultlist from grouped query
*
* #param query
* #return results list
* #throws SolrServerException
*/
public SolrDocumentList queryGrouped(SolrQuery query) throws SolrServerException {
List<GroupCommand> groupCommands = this.solr.query(query).getGroupResponse().getValues();
GroupCommand groupCommand = groupCommands.get(0);
List<Group> groups = groupCommand.getValues();
SolrDocumentList list = new SolrDocumentList();
if(groups.size() > 0){
long totalNumFound = groupCommand.getNGroups();
int iteratorLimit = 1;
for(Group g : groups){
SolrDocumentList groupList = g.getResult();
list.add(groupList.get(0));
//I wanted to limit list to 10 records
if(iteratorLimit++ > 10){
break;
}
}
list.setNumFound(totalNumFound);
}
return list;
}

How to build FacetQuery using spring-data-solr which returns all documents (*:*) from solr index

I want to write a FacetQuery which may not have any criteria except one filter condition (fq). Following query is an example which I want to build using spring-data-solr API.
http://localhost:8983/solr/jpevents/select?q=*:*&fq=categoryIds:(1101)&facet=true&facet.mincount=1&facet.limit=1&facet.field=primaryCategoryId
How can I set query parameter (q=*:*) in FacetQuery?
Environment: I'm writing a Spring MVC based Search API using spring-data-solr 1.0.0.RELEASE with Solr 4.4.0 and Spring 3.2.4.RELEASE.

you can do this combining #Query and #Facet
#Facet(fields={"primaryCategoryId"}, minCount=1, limit=1)
#Query(value="*:*", filters="categoryIds:(?0)")
public FacetPage<JPEvents> XYZ(List<Long> categories, Pageable page);
or execute FacetQuery using SolrTemplate.
FacetQuery query = new SimpleFacetQuery(new SimpleStringCriteria("*:*"))
.setFacetOptions(new FacetOptions("primaryCategoryId")
.setFacetMinCount(1).setFacetLimit(1));
query.setPageRequest(pageable);
solrTemplate.queryForFacetPage(query, JPEvents.class);

I have done something like this :
public static void main()
{
String url = "http://localhost:8983/solr/autocomplete";
SolrServer solrServer = new HttpSolrServer(url);
SolrQuery query = new SolrQuery();
query.set("q", "*");
query.addFilterQuery("name:*");
query.setFacet(true);
query.addFacetField("name");
System.out.println(query);
QueryResponse queryResponse = solrServer.query(query);
List<FacetField> facetFields = queryResponse.getFacetFields();
FacetField cnameMainFacetField = queryResponse.getFacetField("name");
for (Count cnameAndCount : cnameMainFacetField.getValues()) {
String cnameMain = cnameAndCount.getName();
System.out.println(cnameMain);
System.out.println(cnameAndCount.getCount());
}
This gives me correct counts of faceted fields.
Hope you are able to understand what I am doing. Adding output for better understanding:
q=*&fq=name%3A*&facet=true&facet.field=name
a
10
an
7
w
7
m
6
and
5
c
5
p
5
d
4