We Keep Coding

Do not retrieve all columns with Elastic search query

I have an Elastic search query and I would like to retrieve a certain column, not all.
I make my request in java with BoolQUEryBuilder which gives:
BoolQueryBuilder query = boolQuery();
query.must(wildcardQuery('value', value + "*"));
return findAll(query);
The method findAll :
protected List<T> findAll(final BoolQueryBuilder query) {
Query searchQuery = (new NativeSearchQueryBuilder()).withQuery(query).build();
SearchHits<T> searchHits = this.elasticsearchRestTemplate.search(searchQuery, this.getClazz(), this.elasticsearchRestTemplate.getIndexCoordinatesFor(this.getClazz()));
return (List)SearchHitSupport.unwrapSearchHits(searchHits);
}
I would like to add a filter on the columns. To illustrate in SQL this gives:
Select column_one, column_two from table;

Refer source filtering to fetch only few fields from Elasticsearch query results.
As explained in the same document example below code shows which fields to include and which to exclude.
String[] includeFields = new String[] {"title", "innerObject.*"};
String[] excludeFields = new String[] {"user"};
sourceBuilder.fetchSource(includeFields, excludeFields);

With Spring Data Elasticsearch, you should try this instead:
...
//include only specific fields
final SourceFilter sourceFilter = new FetchSourceFilter(new String[]{"column_one", "column_two"}, null);
// assemble the query
Query searchQuery = new NativeSearchQueryBuilder().withQuery(query).build();
searchQuery.addSourceFilter(sourceFilter);
...

Elastic searchquery by multiple value

I am using java QueryBuilder to search values in elasticQuery.Below is my code snippet.
Sample 2 json store in elastic search
{
"dataLayer":"sourcefeed"
}
{
"dataLayer":"sandbox"
}
BoolQueryBuilder qb = QueryBuilders.boolQuery();
qb.must(QueryBuilders.matchQuery("dataLayer",inputValue);
The input value can be either sourcefeed or sandbox or both,Depending on my input Value it should return the output.For single value it is working fine,But if I give input as InputValue="sourceFeed,sandbox" it is not working.Tried termsQuery also but it is not fetching correctly.Kindly help me to solve the problem.
Thanks Prakash

SearchSourceBuilder query = new SearchSourceBuilder();
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
List<QueryBuilder> queryBuilders1 = new ArrayList<QueryBuilder>();
TermsQueryBuilder termsQuery = QueryBuilders.termsQuery("dataLayer.keyword", listOfValues);
queryBuilders1.add(termsQuery);
boolQueryBuilder.must(queryBuilders1.get(0));
query.query(boolQueryBuilder);
add sourceFeed and sandbox etc in one list and pass it in QueryBuilders.termsQuery.
Try this.Hope this will work.

How to define custom analyzer to do global search with hibernate-search and elasticsearch

I have an implementation of hibernate-search-orm (5.9.0.Final) with hibernate-search-elasticsearch (5.9.0.Final).
I defined a custom analyzer on an entity (see beelow) and I indexed two entities :
id: "1"
title: "Médiatiques : récit et société"
abstract:...
id: "2"
title: "Mediatique Com'7"
abstract:...
The search works fine when I search on title field :
"title:médiatique" => 2 results.
"title:mediatique" => 2 results.
My problem is when I do a global search with accents (or not) :
search on "médiatique => 1 result (id:1)
search on "mediatique => 1 result (id:2)
Is there a way to resolve this?
Thanks.
Entity definition:
#Entity
#Table(name="bibliographic")
#DynamicUpdate
#DynamicInsert
#Indexed(index = "bibliographic")
#FullTextFilterDefs({
#FullTextFilterDef(name = "fieldsElasticsearchFilter",
impl = FieldsElasticsearchFilter.class)
})
#AnalyzerDef(name = "customAnalyzer",
tokenizer = #TokenizerDef(factory = StandardTokenizerFactory.class),
filters = {
#TokenFilterDef(factory = LowerCaseFilterFactory.class),
#TokenFilterDef(factory = ASCIIFoldingFilterFactory.class),
})
#Analyzer(definition = "customAnalyzer")
public class BibliographicHibernate implements Bibliographic {
...
#Column(name="title", updatable = false)
#Fields( {
#Field,
#Field(name = "titleSort", analyze = Analyze.NO, store = Store.YES)
})
#SortableField(forField = "titleSort")
private String title;
...
}
Search method :
FullTextEntityManager ftem = Search.getFullTextEntityManager(entityManager);
QueryBuilder qb = ftem.getSearchFactory().buildQueryBuilder().forEntity(Bibliographic.class).get();
QueryDescriptor q = ElasticsearchQueries.fromQueryString(queryString);
FullTextQuery query = ftem.createFullTextQuery(q, Bibliographic.class).setFirstResult(start).setMaxResults(rows);
if (filters!=null){
filters.stream().map((filter) -> filter.split(":")).forEach((f) -> {
query.enableFullTextFilter("fieldsElasticsearchFilter")
.setParameter("field", f[0])
.setParameter("value", f[1]);
}
);
}
if (facetFields!=null){
facetFields.stream().map((facet) -> facet.split(":")).forEach((f) ->{
query.getFacetManager()
.enableFaceting(qb.facet()
.name(f[0])
.onField(f[0])
.discrete()
.orderedBy(FacetSortOrder.COUNT_DESC)
.includeZeroCounts(false)
.maxFacetCount(10)
.createFacetingRequest() );
}
);
}
List<Bibliographic> bibs = query.getResultList();

To be honest I'm more surprised document 1 would match at all, since there's a trailing "s" on "Médiatiques" and you don't use any stemmer.
You are in a special case here: you are using a query string and passing it directly to Elasticsearch (that's what ElasticsearchQueries.fromQueryString(queryString) does). Hibernate Search has very little impact on the query being run, it only impacts the indexed content and the Elasticsearch mapping here.
When you run a QueryString query on Elasticsearch and you don't specify any field, it uses all fields in the document. I wouldn't bet that the analyzer used when analyzing your query is the same analyzer that you defined on your "title" field. In particular, it may not be removing accents.
An alternative solution would be to build a simple query string query using the QueryBuilder. The syntax of queries is a bit more limited, but is generally enough for end users. The code would look like this:
FullTextEntityManager ftem = Search.getFullTextEntityManager(entityManager);
QueryBuilder qb = ftem.getSearchFactory().buildQueryBuilder().forEntity(Bibliographic.class).get();
Query q = qb.simpleQueryString()
.onFields("title", "abstract")
.matching(queryString)
.createQuery();
FullTextQuery query = ftem.createFullTextQuery(q, Bibliographic.class).setFirstResult(start).setMaxResults(rows);
Users would still be able to target specific fields, but only in the list you provided (which, by the way, is probably safer, otherwise they could target sort fields and so on, which you probably don't want to allow). By default, all the fields in that list would be targeted.
This may lead to the exact same result as the query string, but the advantage is, you can override the analyzer being used for the query. For instance:
FullTextEntityManager ftem = Search.getFullTextEntityManager(entityManager);
QueryBuilder qb = ftem.getSearchFactory().buildQueryBuilder().forEntity(Bibliographic.class)
.overridesForField("title", "customAnalyzer")
.overridesForField("abstract", "customAnalyzer")
.get();
Query q = qb.simpleQueryString()
.onFields("title", "abstract")
.matching(queryString)
.createQuery();
FullTextQuery query = ftem.createFullTextQuery(q, Bibliographic.class).setFirstResult(start).setMaxResults(rows);
... and this will use your analyzer when querying.
As an alternative, you can also use a more advanced JSON query by replacing ElasticsearchQueries.fromQueryString(queryString) with ElasticsearchQueries.fromJsonQuery(json). You will have to craft the JSON yourself, though, taking some precautions to avoid any injection from the user (use Gson to build the Json), and taking care to follow the Elasticsearch query syntax.
You can find more information about simple query string queries in the official documentation.
Note: you may want to add FrenchMinimalStemFilterFactory to your list of token filters in your custom analyzer. It's not the cause of your problem, but once you manage to use your analyzer in search queries, you will very soon find it useful.

Elasticsearch 2+ JAVA API orFilter

I am trying to write some Java code to query an ES5 instance. In ES1, you could OR and AND filters, orFilter and andFilter. What is the equivalent in ES5? I have tried -
QueryBuilder fb1 = QueryBuilders.termQuery("term1", "value1");
QueryBuilder fb2 = QueryBuilders.termQuery("term1", "value2");
QueryBuilder fb3 = QueryBuilders.termQuery("term2", "value3");
QueryBuilder fb = QueryBuilders.boolQuery()
.must(fb1)
.should(fb2)
This gives results that match fb1, so this is not an OR
QueryBuilder fb = QueryBuilders.boolQuery()
.must(fb1)
.must(fb2)
This gives no results, so the AND worked.
I want to figure out ways of constructing AND, OR queries

In order to make an AND query, put all the conditions into "must".
QueryBuilder fb = QueryBuilders.boolQuery().must(fb1).must(fb2);
In order to make an OR query, put all the conditions into "should".
QueryBuilder fb = QueryBuilders.boolQuery().should(fb1).should(fb2);

From ElasticSearch documentation (https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-compound-queries.html):
QueryBuilder qb = boolQuery()
.must(termQuery("content", "test1"))
.must(termQuery("content", "test4"))
.mustNot(termQuery("content", "test2"))
.should(termQuery("content", "test3"))
.filter(termQuery("content", "test5"));
must seems like it might be the equivalent of the AND and should like it's the equivalent of the OR.

You can use BoolQueryBuiler for these filters :
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery().must(QueryBuilders.termQuery("customerId", customerId))
boolQueryBuilder.mustNot(QueryBuilders.termsQuery("reason", "failed"));
boolQueryBuilder.should(QueryBuilders.termsQuery("action", "deny")).minimumNumberShouldMatch(1);
QueryBuilder queryBuilder=boolQueryBuilder;
new SearchContext().prepareSearch(sc.getIndexNames()).setTypes(sc.getTypeName()).setSearchType(SearchType.DFS_QUERY_THEN_FETCH).setQuery(queryBuilder);

Elasticsearch multiple condition query using java api

There are multiple documents containing around 100 fields each. I'd like to perform a following search trough elasticsearch Java API 5.x:
There are 3 fields I'd like to use for this search i.e.
department
job
name
I'd like to search the return documents that match fields like "department: D1", "department: D2", "job: J1", "job: J2" "name: N1"
I've been trying to do it this way
String[] departments = ["d1","d2","d3"];
String[] jobs = ["j1","j2","j3"];
String[] names = ["n1"];
MultiSearchRequestBuilder requestbuilder;
requestBuilder.add(client.prepareSearch().setQuery(QueryBuilders.termsQuery("department", departments)));
requestBuilder.add(client.prepareSearch().setQuery(QueryBuilders.termsQuery("job", jobs)));
requestBuilder.add(client.prepareSearch().setQuery(QueryBuilders.termsQuery("name", names)));
MultiSearchResponse response = requestBuilder.get();
However the queries are executed as if each was an individual query, i.e. in this example when j3 exists in d4, the document with d4 will be matched aswell
How to perform the search the way I mentioned? I've been trying numerous different queries and nothing seems to work, is there something I am missing?

You don't want to use MultiSearchRequestBuilder, you simply need to combine your three constraints in a bool/filter query:
BoolQueryBuilder query = QueryBuilders.boolQuery()
.filter(QueryBuilders.termsQuery("department", departments))
.filter(QueryBuilders.termsQuery("job", jobs))
.filter(QueryBuilders.termsQuery("name", names));
SearchResponse resp = client.prepareSearch().setQuery(query).get();

For Elasticsearch 5.6.4 of using HighRestClient, add required number of sourcebuilder...
static RestHighLevelClient client = new RestHighLevelClient(RestClient.builder(new HttpHost("localhost", 9200, "http")));
public static void multisearch() throws IOException{
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(QueryBuilders.termQuery("name", "vijay1"));
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("posts-1","posts-2").source(sourceBuilder);
SearchResponse searchResponse = client.search(searchRequest);
RestStatus status = searchResponse.status();
System.out.println(searchResponse.toString());