We Keep Coding

Elasticsearch: Java Client for Composite Aggregation using Terms queries

I have created a composite query for aggregating on 2 different attributes as below -
{
"from": 0,
"size": 0,
"query": {
"bool": {
"must": [
{
"nested": {
"query": {
"script": {
"script": {
"source": "params.territoryIds.contains(doc['territoryHierarchy.id'].value) ",
"lang": "painless",
"params": {
"territoryIds": [
12345678
]
}
},
"boost": 1.0
}
},
"path": "territoryHierarchy",
"ignore_unmapped": false,
"score_mode": "none",
"boost": 1.0
}
},
{
"bool": {
"should": [
{
"nested": {
"query": {
"script": {
"script": {
"source": "doc['forecastHeaders.id'].value == params.id && doc['forecastHeaders.revenueCategory'].value == params.revenueCategory ",
"lang": "painless",
"params": {
"revenueCategory": 0,
"id": 987654321
}
},
"boost": 1.0
}
},
"path": "forecastHeaders",
"ignore_unmapped": false,
"score_mode": "none",
"boost": 1.0
}
},
{
"nested": {
"query": {
"script": {
"script": {
"source": "doc['forecastHeaders.id'].value == params.id && doc['forecastHeaders.revenueCategory'].value == params.revenueCategory ",
"lang": "painless",
"params": {
"revenueCategory": 0,
"id": 987654321
}
},
"boost": 1.0
}
},
"path": "forecastHeaders",
"ignore_unmapped": false,
"score_mode": "none",
"boost": 1.0
}
}
],
"adjust_pure_negative": true,
"boost": 1.0
}
},
{
"terms": {
"revnWinProbability": [
40,
50
],
"boost": 1.0
}
},
{
"terms": {
"revenueStatus.keyword": [
"OPEN"
],
"boost": 1.0
}
},
{
"range": {
"recordUpdateTime":{
"gte":1655117440000
}
}
}
],
"adjust_pure_negative": true,
"boost": 1.0
}
},
"version": true,
"aggregations": {
"TopLevelAggregation": {
"composite" : {
"size" : 10000,
"sources" : [
{
"directs": {
"terms": {
"script": {
"source": "def territoryNamesList = new ArrayList(); def name; def thLength = params._source.territoryHierarchy.length; for(int i = 0; i< thLength;i++) { def thRecord = params._source.territoryHierarchy[i]; if (params.territoryIds.contains(thRecord.id) && i+params.levelToReturn < thLength) { territoryNamesList.add(params._source.territoryHierarchy[i+params.levelToReturn].name);} } return territoryNamesList;",
"lang": "painless",
"params": {
"territoryIds": [
12345678
],
"levelToReturn": 1
}
}
}
}
},
{
"qtr" : {
"terms" : {
"field" : "quarter.keyword",
"missing_bucket" : false,
"order" : "asc"
}
}
}
]
},
"aggregations": {
"revnRevenueAmount": {
"sum": {
"script": {
"source": "doc['revenueTypeCategory.keyword'].value != 'Other' ? doc['revnRevenueAmount']:doc['revnRevenueAmount']",
"lang": "painless"
},
"value_type": "long"
}
}
}
}
}
}
So this query does a composite aggregation based on two different terms aggregations, directs and qtr, and it works fine.
Now I am trying to create a corresponding spring data java client implementation for it. So I have created the code as below -
BoolQueryBuilder baseQueryBuilder = getQueryBuilder(searchCriteria);
List<TermsAggregationBuilder> aggregationBuilders = getMultiBaseAggregationBuilders(searchCriteria, baseQueryBuilder);
Where the bool query supplies the first part of the bool query and the getMultiBaseAggregationBuilders method returns the 2 different terms aggregations shown in the query above - directs and qtr. Now I am not finding any API to send this list of terms aggregations to the composite aggregation builder. Would be really grateful if someone can give me a pointer as to how this list of terms aggregations can be used inside the composite aggregation builder so the same can be achieved in the java code as it shows in the elastic query above. Thanks in advance.

How to get single field in mongodb query?

I have data like this:
{ id : 1,
book: "Flash",
chapters: [
{
chap_no: "1",
sub_chapter: [
{sub_no: 1, description: "<description>"
},
{sub_no: 2, description: "<description>"
},
]
}
]
}
i want to show one field like this base on book -> chapter_no -> sub_no
{
sub_no: 2, description: "<description>"
}
in mongodb query.

$match
$unwind
$unwind
$match
$replaceRoot
db.collection.aggregate([
{
"$match": {
"chapters.sub_chapter.sub_no": 2
}
},
{
"$unwind": "$chapters"
},
{
"$unwind": "$chapters.sub_chapter"
},
{
"$match": {
"chapters.sub_chapter.sub_no": 2
}
},
{
"$replaceRoot": {
"newRoot": "$chapters.sub_chapter"
}
}
])
mongoplayground

you can make like this
db.collection.aggregate([
{
"$match": {
$and: [
{
"book": "Flash3"
},
{
"chapters.chap_no": "2"
},
{
"chapters.sub_chapter.sub_no": "1"
}
]
}
},
{
"$unwind": "$chapters"
},
{
"$unwind": "$chapters.sub_chapter"
},
{
"$match": {
$and: [
{
"book": "Flash3"
},
{
"chapters.chap_no": "2"
},
{
"chapters.sub_chapter.sub_no": "1"
}
]
}
},
{
"$replaceRoot": {
"newRoot": "$chapters.sub_chapter"
}
}
])

Query for Elastic search aggregations based on nested array

I do have the documents like below in my index
{
"bookName" : "coolbook",
"timeStamp" : "2018-11-19T12:52:17.000Z",
"referenceId" : "auth_test_01_01_000004",
"peoplestatus" : [
{
"personId" : "p1",
"status" : "like"
},
{
"personId" : "p2",
"status" : "dislike"
},{
"personId" : "p3",
"status" : "netrual"
}
]
}
Now I want to query the aggregations of book count for person p1,p2 like below
the counts of books
p1-liked but p2-disliked
p1,p2 both liked
p2-disliked but p1-liked
p1,b2 both disliked
Thanks for your help

Since you require buckets with different filter for each bucket, filters aggregation is best fit for this.
As per your comment there will be two person ids to be compared following is the query for your following two combinations:
P1 liked but P2 disliked
P1 and P2 both liked
{
"query": {
"match_all": {}
},
"aggs": {
"books": {
"filters": {
"filters": {
"P1L_P2DL": {
"bool": {
"must": [
{
"nested": {
"path": "peoplestatus",
"query": {
"bool": {
"must": [
{
"bool": {
"must": [
{
"term": {
"peoplestatus.personId": "p1"
}
},
{
"term": {
"peoplestatus.status": "like"
}
}
]
}
}
]
}
}
}
},
{
"nested": {
"path": "peoplestatus",
"query": {
"bool": {
"must": [
{
"bool": {
"must": [
{
"term": {
"peoplestatus.personId": "p2"
}
},
{
"term": {
"peoplestatus.status": "dislike"
}
}
]
}
}
]
}
}
}
}
]
}
},
"L1N3": {
"bool": {
"must": [
{
"nested": {
"path": "peoplestatus",
"query": {
"bool": {
"must": [
{
"bool": {
"must": [
{
"term": {
"peoplestatus.personId": "p1"
}
},
{
"term": {
"peoplestatus.status": "like"
}
}
]
}
}
]
}
}
}
},
{
"nested": {
"path": "peoplestatus",
"query": {
"bool": {
"must": [
{
"bool": {
"must": [
{
"term": {
"peoplestatus.personId": "p2"
}
},
{
"term": {
"peoplestatus.status": "like"
}
}
]
}
}
]
}
}
}
}
]
}
}
}
}
}
},
"size": 0
}

Aggregation operation to modify the key of a property from a subdocument of documents

I have in mongo db a collection with documents like this:
{
"prop1": "val1",
"prop2": {
"prop21": {
"toBeChanged1_xyz": "val2",
"toBeChanged2_zyx": "val3",
},
"prop22": {
"whatever": "something"
}
}
}
Trying to create aggregation operation in Java Spring to remove from the keys of prop21's properties the substring after _. This if what I need to have:
{
"prop1": "val1",
"prop2": {
"prop21": {
"toBeChanged1": "val2",
"toBeChanged2": "val3",
},
"prop22": {
"whatever": "something"
}
}
}
Tried something like this:
List<AggregationOperation> operations = new ArrayList<AggregationOperation>();
operations.add( Aggregation.project( "code" ).and( "code" ).minus( "_"+"$entityType" ).as( "newId" ));
It didn't worked, of course. I have no idea how can it be done or can it be done easier in other ways.

You should update your structure to below.
{
"prop1": "val1",
"prop2": [
{
"prop": "prop21",
"value": [
{
"prop": "toBeChanged1_xyz",
"value": "val2"
},
{
"prop": "toBeChanged2_zyx",
"value": "val3"
}
]
},
{
"prop": "prop22",
"value": [
{
"prop": "whatever",
"value": "something"
}
]
}
]
}
You can try below query in 3.4. The query will iterate over property elements and locate prop2 and $split to modify the property name.
[
{
"$addFields": {
"prop2": {
"$map": {
"input": "$prop2",
"as": "prop_2",
"in": {
"prop": "$$prop_2.prop",
"value": {
"$cond": [
{
"$eq": [
"$$prop_2.prop",
"prop21"
]
},
{
"$map": {
"input": "$$prop_2.value",
"as": "prop_21",
"in": {
"prop": {
"$arrayElemAt": [
{
"$split": [
"$$prop_21.prop",
"_"
]
},
0
]
},
"value": "$$prop_21.value"
}
}
},
"$$prop_2.value"
]
}
}
}
}
}
}
]
You can use the below query if you cant modify the structure from 3.4.4 version.
The query uses $arrayToObject and $objectToArray operator to modify the documents to above structure with same logic to modify the property name.
[
{
"$addFields": {
"prop2": {
"$arrayToObject": {
"$map": {
"input": {
"$objectToArray": "$prop2"
},
"as": "prop_2",
"in": {
"k": "$$prop_2.k",
"v": {
"$cond": [
{
"$eq": [
"$$prop_2.k",
"prop21"
]
},
{
"$arrayToObject": {
"$map": {
"input": {
"$objectToArray": "$$prop_2.v"
},
"as": "prop_21",
"in": {
"k": {
"$arrayElemAt": [
{
"$split": [
"$$prop_21.k",
"_"
]
},
0
]
},
"v": "$$prop_21.v"
}
}
}
},
"$$prop_2.v"
]
}
}
}
}
}
}
}
]

How to merge geo distance filter with bool term query

How to use Elasticsearch 1.6/1.7 version geo distance filter with bool term query like this. How and here two merge these two queries
Original query:
{
"query": {
"bool": {
"must": [
{
"term": {
"categories": "tv"
}
}
],
"should": [
{
"term": {
"subCategory": "led"
}
}
],
"minimum_should_match": 1,
"boost": 2
}
}
}
I want to search products with above bool query with distance of 10 miles
{
"filtered": {
"filter": {
"geo_distance": {
"distance": "10km",
"sellerInfoES.address.sellerLocation": "28.628978,77.21971479999999"
}
}
}
}
Thanks Val! Query is working, I am not getting any query parsing error. However this geo query is not returning and distance range result. I am using Elasticsearch 1.6 and stored sellerLocation as geo_point.Mapping:
{
"SellerInfoES": {
"type": "nested",
"properties": {
"sellerLocation": {
"type": "geo_point"
}
}
}
}
This geo_query is not working
{
"geo_distance": {
"distance": "100km",
"sellerLocation": {
"lat": 28.628978,
"lon": 77.21971479999999
}
}
}

You can combine both query/filters like this:
{
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"term": {
"categories": "tv"
}
},
{
"nested": {
"path": "sellerInfoES",
"filter": {
"geo_distance": {
"distance": "10km",
"sellerInfoES.sellerLocation": {
"lat": "28.628978",
"lon":"77.21971479999999"
}
}
}
}
}
],
"should": [
{
"term": {
"subCategory": "led"
}
}
],
"minimum_should_match": 1,
"boost": 2
}
}
}
}
}