We Keep Coding

Elasticsearch: Java Client for Composite Aggregation using Terms queries

I have created a composite query for aggregating on 2 different attributes as below -
{
"from": 0,
"size": 0,
"query": {
"bool": {
"must": [
{
"nested": {
"query": {
"script": {
"script": {
"source": "params.territoryIds.contains(doc['territoryHierarchy.id'].value) ",
"lang": "painless",
"params": {
"territoryIds": [
12345678
]
}
},
"boost": 1.0
}
},
"path": "territoryHierarchy",
"ignore_unmapped": false,
"score_mode": "none",
"boost": 1.0
}
},
{
"bool": {
"should": [
{
"nested": {
"query": {
"script": {
"script": {
"source": "doc['forecastHeaders.id'].value == params.id && doc['forecastHeaders.revenueCategory'].value == params.revenueCategory ",
"lang": "painless",
"params": {
"revenueCategory": 0,
"id": 987654321
}
},
"boost": 1.0
}
},
"path": "forecastHeaders",
"ignore_unmapped": false,
"score_mode": "none",
"boost": 1.0
}
},
{
"nested": {
"query": {
"script": {
"script": {
"source": "doc['forecastHeaders.id'].value == params.id && doc['forecastHeaders.revenueCategory'].value == params.revenueCategory ",
"lang": "painless",
"params": {
"revenueCategory": 0,
"id": 987654321
}
},
"boost": 1.0
}
},
"path": "forecastHeaders",
"ignore_unmapped": false,
"score_mode": "none",
"boost": 1.0
}
}
],
"adjust_pure_negative": true,
"boost": 1.0
}
},
{
"terms": {
"revnWinProbability": [
40,
50
],
"boost": 1.0
}
},
{
"terms": {
"revenueStatus.keyword": [
"OPEN"
],
"boost": 1.0
}
},
{
"range": {
"recordUpdateTime":{
"gte":1655117440000
}
}
}
],
"adjust_pure_negative": true,
"boost": 1.0
}
},
"version": true,
"aggregations": {
"TopLevelAggregation": {
"composite" : {
"size" : 10000,
"sources" : [
{
"directs": {
"terms": {
"script": {
"source": "def territoryNamesList = new ArrayList(); def name; def thLength = params._source.territoryHierarchy.length; for(int i = 0; i< thLength;i++) { def thRecord = params._source.territoryHierarchy[i]; if (params.territoryIds.contains(thRecord.id) && i+params.levelToReturn < thLength) { territoryNamesList.add(params._source.territoryHierarchy[i+params.levelToReturn].name);} } return territoryNamesList;",
"lang": "painless",
"params": {
"territoryIds": [
12345678
],
"levelToReturn": 1
}
}
}
}
},
{
"qtr" : {
"terms" : {
"field" : "quarter.keyword",
"missing_bucket" : false,
"order" : "asc"
}
}
}
]
},
"aggregations": {
"revnRevenueAmount": {
"sum": {
"script": {
"source": "doc['revenueTypeCategory.keyword'].value != 'Other' ? doc['revnRevenueAmount']:doc['revnRevenueAmount']",
"lang": "painless"
},
"value_type": "long"
}
}
}
}
}
}
So this query does a composite aggregation based on two different terms aggregations, directs and qtr, and it works fine.
Now I am trying to create a corresponding spring data java client implementation for it. So I have created the code as below -
BoolQueryBuilder baseQueryBuilder = getQueryBuilder(searchCriteria);
List<TermsAggregationBuilder> aggregationBuilders = getMultiBaseAggregationBuilders(searchCriteria, baseQueryBuilder);
Where the bool query supplies the first part of the bool query and the getMultiBaseAggregationBuilders method returns the 2 different terms aggregations shown in the query above - directs and qtr. Now I am not finding any API to send this list of terms aggregations to the composite aggregation builder. Would be really grateful if someone can give me a pointer as to how this list of terms aggregations can be used inside the composite aggregation builder so the same can be achieved in the java code as it shows in the elastic query above. Thanks in advance.

How to find distinct value with count of other array inside dhe document in MongoDB

I want to get distinct categoryCode and categoryName while this document also contains list of accessories I want to count of this accessories in the response.
Sample data:
[
{
"categoryCode":"categoryCode1",
"categoryName":"categoryName1",
"accessories":[{"a_id":1},{"a_id":2}]
},
{
"categoryCode":"categoryCod2",
"categoryName":"categoryName2",
"accessories":[{"a_id":1},{"a_id":2},{"a_id":3}]
},
{
"categoryCode":"categoryCode1",
"categoryName":"categoryNam1",
"accessories":[{"a_id":1},{"a_id":2}]
}
]
Expected result:
[
{
"categoryCode":"categoryCode1",
"categoryName":"categoryName1",
"accessoriesCount":2
},
{
"categoryCode":"categoryCod2",
"categoryName":"categoryName2",
"accessoriesCount":3
}
]

https://mongoplayground.net/p/q6AZOaTwo5a
db.collection.aggregate([
{
"$group": {
"_id": {
categoryCode: "$categoryCode",
"categoryName": "$categoryName"
},
"accessories": {
"$addToSet": "$accessories"
}
}
},
{
"$project": {
categoryCode: "$_id.categoryCode",
categoryName: "$_id.categoryName",
accessoriesCount: {
$size: "$accessories"
},
_id: 0
}
}
])

Query
group to have the distinct values
$push the accesories arrays (we dont have $concat accumulator)
reduce those arrays to union them, keep only the distinct members, and take the count.
Test code here
db.collection.aggregate([
{
"$group": {
"_id": {
"categoryCode": "$categoryCode",
"categoryName": "$categoryName"
},
"accessories": {
"$push": "$accessories"
}
}
},
{
"$set": {
"accessoriesCount": {
"$size": {
"$reduce": {
"input": "$accessories",
"initialValue": [],
"in": {
"$setUnion": [
"$$value",
"$$this"
]
}
}
}
}
}
},
{
"$project": {
"_id": 0,
"categoryCode": "$_id.categoryCode",
"categoryName": "$_id.categoryName",
"accessoriesCount": "$accessoriesCount"
}
}
])

How to output java annotation's attributes and values as JSON format?

I hava defined annotation and using it on Java Interface, my code is:
#Retention(RetentionPolicy.RUNTIME)
#Target(ElementType.FIELD)
#Documented
public #interface ThingPropertyDefinition {
String identifier();
String name() default "";
EnumDataType dataType();
EnumReadWrite rw();
String description() default "";
ConstraintDefinition constraint() default #ConstraintDefinition();
boolean isScene() default true;
boolean isRequired() default true;
}
and using the annotation on interface, my code:
#ThingService(name = "SmartGateway")
public interface ISmartGatewayService extends IThingService {
//==================define properties==================
/**
* node
*/
#ThingPropertyDefinition(identifier = "Nodes",
name = "节点列表",
description = "网关当前管理的节点列表",
dataType = EnumDataType.STRUCT,
rw = EnumReadWrite.READ_ONLY,
constraint = #ConstraintDefinition(unit = EnumUnit.UNKNOWN),
isRequired = true)
List<ThingDeviceNode> nodes = new ArrayList<>();
//...
}
and using java reflection to output JSON format.
Map<String, Map<String, List<Annotation>>> clazzAnnotation = new ConcurrentHashMap<>();
cacheClassMap.entrySet()
.parallelStream()
.forEachOrdered((clazz) -> {
List<Annotation> props = new ArrayList<>();
List<Annotation> actions = new ArrayList<>();
List<Annotation> events = new ArrayList<>();
Field[] fields = clazz.getValue().getFields();
Stream.of(fields).forEach((f) -> {
f.setAccessible(true);
Arrays.stream(f.getAnnotations()).forEach((a) -> {
if (a.annotationType().equals(ThingPropertyDefinition.class)) {
props.add(a);
}
});
});
Method[] methods = clazz.getValue().getMethods();
Stream.of(methods).forEach((m) -> {
m.setAccessible(true);
Arrays.stream(m.getDeclaredAnnotations()).forEach((a) -> {
if (a.annotationType().equals(ThingActionDefinition.class)) {
actions.add(a);
}
if (a.annotationType().equals(ThingEventDefinition.class)) {
events.add(a);
}
});
});
Map<String, List<Annotation>> model = new ConcurrentHashMap<>();
model.put("actions", actions);
model.put("events", events);
model.put("properties", props);
clazzAnnotation.put(clazz.getKey(), model);
});
ObjectMapper mapper = new ObjectMapper();
//TODO!
log.debug("{}", mapper.writeValueAsString(clazzAnnotation));
I use jackson mapper object to output the List , and output is not whole like this:
"home.gateway.ISmartGatewayService": {
"actions": [
{
"required": true
},
{
"required": true
},
{
"required": true
},
{
"required": true
},
{
"required": true
},
{
"required": true
},
{
"required": true
},
{
"required": true
},
{
"required": true
},
{
"required": true
},
{
"required": true
},
{
"required": true
},
{
"required": true
},
{
"required": true
}
],
"events": [
{},
{},
{},
{},
{},
{},
{}
],
"properties": [
{
"required": true,
"scene": true
}
]
},
how to make annotation's all attributes and values to output? I test fastjson, it works, anyone ideas?

How to write query to find percentage in elasticsearch?

I have data in elasticsearch.
this is my actual doc https://docs.google.com/document/d/1DKID90I9ulUcut-S8UfrnSjY-3citEwmyfnJJmrIRU8/edit?usp=sharing
doc:
{
store_id:"abc",
event_timestamp:"2019-06-05 13:00:05",
event_type:"heartbeat"
}
I have store_id, range of dates and event type in the input.in output, I need the percentage amount of time device was online for that hour.
This is how we consider device online.
If there is an event="heartbeat" for a store_id in an hour then we say the store is online.
example 1.
so if the range is of "2019-05-07" to "2019-05-08" and there are 14 docs with different hour then the percentage will be (14/(2*24))*100
example 2.
doc:
{
store_id:"abc",
event_timestamp:"2019-06-05 13:00:05",
event_type:"heartbeat"
}
doc:
{
store_id:"abc",
event_timestamp:"2019-06-05 14:00:05",
event_type:"heartbeat"
}
doc:
{
store_id:"abc",
event_timestamp:"2019-06-05 14:00:05",
event_type:"heartbeat"
}
if input was store_id="abc" and date_range="2019-06-05" to ""2019-06-05" and event_type="heartbeat" then output would be (2/(1*24)) because there are only two different hour with event=heartbeat of that store.
this is my query for the cumulative sum.If some How I can divide the final cumulative sum with difference between dates.
GET /internship38/_search
{
"query":
{
"bool":
{
"must":
[
{
"match" :
{
"attributes.store_id" : "41b15888-0c2f-48f9-89d0-dc7aad19f52b"
}
},
{
"match":
{
"event_type":"app_sent_heartbeat"
}
}
]
}
},
"aggs":
{
"my_date_histo":{
"date_histogram":{
"field":"arrival_timestamp",
"interval":"day"
},
"aggs":
{
"distinct_hours": {
"cardinality": {
"script": {
"lang": "painless",
"source": "doc[params.date_field].value.hourOfDay;",
"params": {
"date_field": "arrival_timestamp"
}
}
}
},
"cumulative_hours": {
"cumulative_sum": {
"buckets_path": "distinct_hours"
}
}
}
}
}
}
Can It be done in java? for example https://www.programcreek.com/java-api-examples/?api=org.elasticsearch.script.Script

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-bucket-script-aggregation.html
Above link in the elasticsearch documentation would help if you can reformat your query into "buckets" using the "aggs" functionality.
from link:
{
"size": 0,
"aggs" : {
"sales_per_month" : {
"date_histogram" : {
"field" : "date",
"calendar_interval" : "month"
},
"aggs": {
"total_sales": {
"sum": {
"field": "price"
}
},
"t-shirts": {
"filter": {
"term": {
"type": "t-shirt"
}
},
"aggs": {
"sales": {
"sum": {
"field": "price"
}
}
}
},
"t-shirt-percentage": {
"bucket_script": {
"buckets_path": {
"tShirtSales": "t-shirts>sales",
"totalSales": "total_sales"
},
"script": "params.tShirtSales / params.totalSales * 100"
}
}
}
}
}
}

Elasticsearch returns an error of exceeding the limit fielddata

I have a problem with sorting namely, sorting work but only for price field. When I try to sort by start_date, end_date, uid, cat title get the message about exceeding the limit:
Data too large, the date for [ "name of field here"] would be larger than the limit of [19798897459 / 18.4gb]]
I do not know why this is happening code looks correct sample query for elastica looks like this:
Mapping:
"auctions": {
"_all": { "enabled": false },
"properties": {
"cat": { "store": true, "type": "long" },
"curr": { "index": "not_analyzed", "store": true, "type": "string" },
"end_date": { "store": true, "type": "long" },
"price": { "store": true, "type": "long" },
"start_date": { "store": true, "type": "long" },
"tcat": { "store": true, "type": "long" },
"title": { "store": true, "type": "string" },
"uid": { "store": true, "type": "long" }
}
},
Request:
/search?uids=335,547&title=Karta&orderBy=uid&orderDir=asc
Method:
private NativeSearchQueryBuilder getSearchQuery(AuctionIndexSearchParams searchParams, Pageable pageable) {
final List<FilterBuilder> filters = Lists.newArrayList();
final NativeSearchQueryBuilder searchQuery = new NativeSearchQueryBuilder();
Optional.ofNullable(searchParams.getCategoryId()).ifPresent(v -> filters.add(boolFilter().must(termFilter("cat", v))));
Optional.ofNullable(searchParams.getCurrency()).ifPresent(v -> filters.add(boolFilter().must(termFilter("curr", v))));
Optional.ofNullable(searchParams.getTreeCategoryId()).ifPresent(v -> filters.add(boolFilter().must(termFilter("tcat", v))));
Optional.ofNullable(searchParams.getUid()).ifPresent(v -> filters.add(boolFilter().must(termFilter("uid", v))));
final BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();
//access for many uids
if (searchParams.getUids() != null) {
if (searchParams.getItemId() != null || searchParams.getTitle() != null) {
Optional.ofNullable(searchParams.getUids().split(",")).ifPresent(v -> {
filters.add(boolFilter().must(termsFilter("uid", v)));
});
} else {
for (String user : searchParams.getUids().split(",")) {
boolQueryBuilder.should(queryStringQuery(user).field("uid"));
}
}
}
//access for many categories
if (searchParams.getCategories() != null) {
Optional.ofNullable(searchParams.getCategories().split(",")).ifPresent(v -> {
filters.add(boolFilter().must(termsFilter("cat", v)));
});
}
if (searchParams.getItemId() != null) {
boolQueryBuilder.must(queryStringQuery(searchParams.getItemId()).field("_id"));
}
if (Optional.ofNullable(searchParams.getTitle()).isPresent()) {
boolQueryBuilder.must(queryStringQuery(searchParams.getTitle()).analyzeWildcard(true).field("title"));
}
if (Optional.ofNullable(searchParams.getStartDateFrom()).isPresent()
|| Optional.ofNullable(searchParams.getStartDateTo()).isPresent()) {
filters.add(rangeFilter("start_date").from(searchParams.getStartDateFrom()).to(searchParams.getStartDateTo()));
}
if (Optional.ofNullable(searchParams.getEndDateFrom()).isPresent()
|| Optional.ofNullable(searchParams.getEndDateTo()).isPresent()) {
filters.add(rangeFilter("end_date").from(searchParams.getEndDateFrom()).to(searchParams.getEndDateTo()));
}
if (Optional.ofNullable(searchParams.getPriceFrom()).isPresent()
|| Optional.ofNullable(searchParams.getPriceTo()).isPresent()) {
filters.add(rangeFilter("price").from(searchParams.getPriceFrom()).to(searchParams.getPriceTo()));
}
searchQuery.withQuery(boolQueryBuilder);
FilterBuilder[] filterArr = new FilterBuilder[filters.size()];
filterArr = filters.toArray(filterArr);
searchQuery.withFilter(andFilter(filterArr));
if (searchParams.getOrderBy() != null && searchParams.getOrderDir() != null) {
if (searchParams.getOrderDir().toLowerCase().equals("asc")) {
searchQuery.withSort(SortBuilders.fieldSort(searchParams.getOrderBy()).order(SortOrder.ASC));
} else {
searchQuery.withSort(SortBuilders.fieldSort(searchParams.getOrderBy()).order(SortOrder.DESC));
}
}
if (pageable != null) {
searchQuery.withPageable(pageable);
}
System.out.println(searchQuery.build().getQuery());
System.out.println(searchQuery.build().getFilter());
System.out.println(searchQuery.build().getSort());
return searchQuery;
}
System.out.println(searchQuery.build().getQuery());
{
  "bool": {
    "must": {
      "query_string": {
        "query", "card"
        "fields": [ "title"]
        "analyze_wildcard": true
      }
    }
  }
}
System.out.println (searchQuery.build().getFilter());
{
  "and" {
    "filters": [{
      "bool": {
        "must": {
          "terms": {
            "uid" [ "335", "547"]
          }
        }
      }
    }]
  }
}
System.out.println(searchQuery.build().getSort());
null
Any ideas what might cause this exception?
I should add that I've tried these solutions:
FIELDDATA Data is too large
But the effect was even worse, then no query did not work as quickly.
For any help I will be extremely grateful!
/_stats/fielddata?fields=*
{
"_shards": {
"total": 10,
"successful": 5,
"failed": 0
},
"_all": {
"primaries": {
"fielddata": {
"memory_size_in_bytes": 19466671904,
"evictions": 0,
"fields": {
"_id": {
"memory_size_in_bytes": 0
},
"cat": {
"memory_size_in_bytes": 0
},
"price": {
"memory_size_in_bytes": 3235221240
},
"title": {
"memory_size_in_bytes": 16231450664
}
}
}
},
"total": {
"fielddata": {
"memory_size_in_bytes": 19466671904,
"evictions": 0,
"fields": {
"_id": {
"memory_size_in_bytes": 0
},
"cat": {
"memory_size_in_bytes": 0
},
"price": {
"memory_size_in_bytes": 3235221240
},
"title": {
"memory_size_in_bytes": 16231450664
}
}
}
}
},
"indices": {
"allek": {
"primaries": {
"fielddata": {
"memory_size_in_bytes": 19466671904,
"evictions": 0,
"fields": {
"_id": {
"memory_size_in_bytes": 0
},
"cat": {
"memory_size_in_bytes": 0
},
"price": {
"memory_size_in_bytes": 3235221240
},
"title": {
"memory_size_in_bytes": 16231450664
}
}
}
},
"total": {
"fielddata": {
"memory_size_in_bytes": 19466671904,
"evictions": 0,
"fields": {
"_id": {
"memory_size_in_bytes": 0
},
"cat": {
"memory_size_in_bytes": 0
},
"price": {
"memory_size_in_bytes": 3235221240
},
"title": {
"memory_size_in_bytes": 16231450664
}
}
}
}
}
}
Edit:
I solved the problem as follows:
After discernment, it turned out that I'm using version 1.7. The documentation I found information that doc_values must be set in the mapping to true if you want to sort or aggregate. Fields strings need to add another field multifield.
So after a map change to something more or less like this:
{
"_all": {
"enabled": false
},
"properties": {
"cat": {
"store": true,
"type": "long",
"doc_values": true
},
"curr": {
"index": "not_analyzed",
"store": true,
"type": "string",
"doc_values": true
},
"end_date": {
"store": true,
"type": "long",
"doc_values": true
},
"price": {
"store": true,
"type": "long",
"doc_values": true
},
"start_date": {
"store": true,
"type": "long",
"doc_values": true
},
"tcat": {
"store": true,
"type": "long",
"doc_values": true
},
"title": {
"store": true,
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed",
"ignore_above": 256,
"doc_values": true
}
}
},
"uid": {
"store": true,
"type": "long",
"doc_values": true
}
}
}
Sorting work, but slowed down the whole sysytem search, I will say that much, although the documentation is from about 10-20%.
You should also remember to reindex data!
Thanks!