We Keep Coding

Issue with AccumulatorOperators Max when creating aggregation pipeline

I'm trying to convert following Mongo shell aggregation query into Spring Data MongoDB Aggregation:
db.getCollection('Orders').aggregate([
{ $group: {
"_id": {"book":"$bookingId"},
"docs": {$push: '$$ROOT'}
}
},
{$project: {
latestOrd: {
$filter: {
input: "$docs",
as: "item",
cond: { $eq: ["$$item.bookingVersion", { $max: "$docs.bookingVersion" }] }
}
}
}
},
{ $unwind: "$latestOrd" },
{ $replaceRoot:{newRoot:"$latestOrd"}}
])
Query fetches all orders with highest booking version (for example one bookingId may have many documents with version 3).
This query works fine in Mongo shell but I have issue with Spring Data version of it:
Aggregation.group("bookingId").push(Aggregation.ROOT).as("docs");
Aggregation.project().and(filter("docs")
.as("item")
.by(valueOf("item.bookingVersion")
.equalToValue(AccumulatorOperators.Max.maxOf("docs.bookingVersion"))))
.as("latestOrd");
Aggregation.unwind("latestOrd");
Aggregation.replaceRoot("latestOrd");
Spring generates Mongo query similar to one that I've provided above except $max accumulator:
{ "$max" : "$$docs.bookingVersion" }
for some reason it adds double dollar sign instead of single dollar sign and as a result I have following error:
'Use of undefined variable: docs' on server 127.0.0.1:27017
I'm using spring-boot-starter 2.1.0.RELEASE and 4.2 version of Mongo server.
I appreciate any help on this.
Input documents:
[
{
"_id": "5f847811ebcd1a51a0196736",
"status": "REJECTED",
"bookingId": "1",
"bookingVersion": 4,
"operation": "CREATE"
},
{
"_id": "5f847811ebcd1a51a0196735",
"status": "CREATED",
"bookingId": "1",
"bookingVersion": 4,
"docNumber": "7",
"operation": "CREATE"
},
{
"_id": "5f847811ebcd1a51a0196734",
"status": "CREATED",
"bookingId": "1",
"bookingVersion": 3,
"docNumber": "6",
"operation": "CREATE"
},
{
"_id": "5f847811ebcd1a51a0196738",
"status": "CREATED",
"bookingId": "2",
"bookingVersion": 1,
"docNumber": "8",
"operation": "CREATE"
},
{
"_id": "5f847811ebcd1a51a0196737",
"status": "CREATED",
"bookingId": "2",
"bookingVersion": 2,
"docNumber": "9",
"operation": "CREATE"
}
]
Expected output:
[
{
"_id": "5f847811ebcd1a51a0196736",
"status": "REJECTED",
"bookingId": "1",
"bookingVersion": 4,
"operation": "CREATE"
},
{
"_id": "5f847811ebcd1a51a0196735",
"status": "CREATED",
"bookingId": "1",
"bookingVersion": 4,
"docNumber": "7",
"operation": "CREATE"
},
{
"_id": "5f847811ebcd1a51a0196737",
"status": "CREATED",
"bookingId": "2",
"bookingVersion": 2,
"docNumber": "9",
"operation": "CREATE"
}
]

Issue might be with 2.1.0.RELEASE version of spring boot but upgrading it is not an option for me.
To make this work I've created AggregationExpression instead of AccumulatorOperators.Max.maxOf("docs.bookingVersion") part.
Here is a method for it:
private AggregationExpression buildMaxExpression() {
return new AggregationExpression() {
#Override
public Document toDocument(final AggregationOperationContext context) {
return new Document("$max", "$docs.bookingVersion");
}
};
}

Add data from Api response into Firebase Database in Android

This I want to add data to firebase database through Volley API response in Android.
"data": [{
"id": 1,
"full_name": "abc",
"email": "abc999#gmail.com",
"country_code": "+91",
"phone": 2147483647,
"profile_pic": "",
"type": 0,
"status": 1,
"reset_token": "",
"verify_token": "$2y$10$YXCZ1yteimLatQnAszJTi.HOGDZrr9xjKJtIDNs3uagX3elFUlC.2",
"created_at": "2019-05-07 07:53:29",
"updated_at": "2019-05-08 12:57:45",
"deleted_at": null
}, {
"id": 2,
"full_name": "xyz",
"email": "xyz#gm.com",
"country_code": "+91",
"phone": 2147483647,
"profile_pic": "",
"type": 0,
"status": 1,
"reset_token": null,
"verify_token": "$2y$10$Dtk.BdqBgHFyGcpj9bHyI.JRPJlc90Qmhxx0Imm0Mzzd3x6QchDMi",
"created_at": "2019-05-07 08:34:39",
"updated_at": "2019-05-07 08:34:39",
"deleted_at": null
}, {
"id": 3,
"full_name": "abc",
"email": "abc#gmail.com",
"country_code": "091",
"phone": 123456,
"profile_pic": "",
"type": 0,
"status": 1,
"reset_token": "$2y$10$mT9MqON6gMre0rKtoK0ON.VApZYBZP0PY55uM017Cz74E69qBILjm",
"verify_token": "$2y$10$HMBteSyYTKZ3XgYviUdNORKOw1Bpan5m0UcqIyx3dZrYUsNajou",
"created_at": "2019-05-07 08:47:29",
"updated_at": "2019-05-17 05:55:00",
"deleted_at": null
}
]
Now I want firebase data should look like this
firebasedemo
.
+ Users
.
. . name:"abc"
I am writing this line in Api call For loop
Rootref= FirebaseDatabase.getInstance().getReference();
Rootref.child("Users").child("name").setValue(name);
But I am getting only one record in firebase not all
Any help will be highly appreciated

Try the following:
DatabaseReference rootref = FirebaseDatabase.getInstance().getReference();
rootref.child("Users").push().setValue(name);
From the docs:
public DatabaseReference push ()
Create a reference to an auto-generated child location. The child key is generated client-side and incorporates an estimate of the server's time for sorting purposes. Locations generated on a single client will be sorted in the order that they are created, and will be sorted approximately in order across all clients.

Elasticsearch 5.1 Bulk Action

I'm try to make a bulk update
Method: Post
Url: /customer/external/_bulk
Json Body:
{"index":{"_id":"1"}}
{"name": "John Doe" }
{"index":{"_id":"2"}}
{"name": "Jane Doe" }
Id 1 is updated but id 2 didnt update. I dont know why?
Response is here:
{
"took": 138,
"errors": false,
"items": [
{
"index": {
"_index": "customer",
"_type": "external",
"_id": "1",
"_version": 15,
"result": "updated",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"created": false,
"status": 200
}
}
]
}

As #Val mentioned, you should be having the new line character \n at the end of the last line in your json body:
{"index":{"_id":"1"}}
{"name": "John Doe" }
{"index":{"_id":"2"}}
{"name": "Jane Doe" }\n
as per mentioned in bulk_api. Hope it helps!

ElasticSearch Java Api: Update Existing Document

I am currently in the process of attempting to update an ElasticSearch document via the Java API. I have a groovy script with the following code:
static updateRequestById(String agencyIndex, String type, String id, def policy) {
UpdateRequest updateRequest = new UpdateRequest()
updateRequest.docAsUpsert(true);
updateRequest.parent("agentNumber");
updateRequest.index(agencyIndex)
updateRequest.type(type)
updateRequest.id(id)
updateRequest.doc("policies", policy)
elasticsearchClient.update(updateRequest).get()
}
The problem with I am having is that I want to update an array within the following document:
{
"took": 4,
"timed_out": false,
"_shards": {
"total": 10,
"successful": 10,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "int-b-agency",
"_type": "jacket",
"_id": "99808.1.27.09_4644",
"_score": 1,
"_source": {
"agentNumber": "99808.1.27.09",
"fileNumber": "4644",
"policies": [
{
"agentNumber": "99808.1.27.09",
"fileNumber": "4644",
"policyNumber": "2730609-91029084",
"checkNumber": "0",
"checkAmount": 0,
"createdOn": null,
"createdBy": "traxuser621",
"propertyTypeCode": "",
"propertyTypeDesc": "1-4 FAMILY RESIDENTIAL",
"ppaddress": "110 Allan Ct ",
"ppcity": "Jacksonville",
"ppstate": "FL",
"ppzip": "32226",
"ppcounty": "Duval",
"policytype": "",
"status": "Active",
"effectiveDate": "2015-04-01T00:00:00-05:00",
"formType": "BASIC OWNERS - ALTA Owners Policy 06_306_FL - FL Original Rate",
"rateCode": "FLOR",
"rateCodeDesc": "FL Original Rate",
"policyTypeCode": "1",
"policyTypeCodeDesc": "BASIC OWNERS",
"amount": 200000,
"hoiAgentNumber": "",
"proForma": false,
"pdfLocation": "\\\\10.212.61.206\\FNFCenter\\legacy_jacket_pdfs\\2015_4_FL6465\\Policy_2730609-91029084.pdf",
"legacyPolicy": "true",
"associatedPolNbr": null
}
]
}
}
]
}
}
In the document above I have a document that has an array called "policies" with a single object. I want to be able to update the "policies" array with additional objects. The end result should look something like the following:
{
"took": 4,
"timed_out": false,
"_shards": {
"total": 10,
"successful": 10,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "int-b-agency",
"_type": "jacket",
"_id": "41341.1.81.38_41340103",
"_score": 1,
"_source": {
"agentNumber": "41341.1.81.38",
"fileNumber": "41340103",
"policies": [
{
"agentNumber": "41341.1.81.38",
"fileNumber": "41340103",
"policyNumber": "8122638-91036874",
"checkNumber": "0",
"checkAmount": 0,
"createdOn": null,
"createdBy": "traxuser621",
"propertyTypeCode": "",
"propertyTypeDesc": "1-4 FAMILY RESIDENTIAL",
"ppaddress": "1800 Smith St ",
"ppcity": "sicklerville",
"ppstate": "PA",
"ppzip": "08105",
"ppcounty": "Dauphin",
"policytype": "",
"status": "Active",
"effectiveDate": "2016-02-01T00:00:00-06:00",
"formType": "TestData",
"rateCode": "PASALERATE",
"rateCodeDesc": "Sale Rate - Agent",
"policyTypeCode": "26",
"policyTypeCodeDesc": "SALE OWNERS",
"amount": 180000,
"hoiAgentNumber": "",
"proForma": false,
"pdfLocation": "SomeLocation1",
"legacyPolicy": "true",
"associatedPolNbr": null
},
{
"agentNumber": "41341.1.81.38",
"fileNumber": "41340103",
"policyNumber": "8122638-91036875",
"checkNumber": "0",
"checkAmount": 0,
"createdOn": null,
"createdBy": "traxuser621",
"propertyTypeCode": "",
"propertyTypeDesc": "1-4 FAMILY RESIDENTIAL",
"ppaddress": "1800 Smith St ",
"ppcity": "sicklerville",
"ppstate": "PA",
"ppzip": "08105",
"ppcounty": "Dauphin",
"policytype": "",
"status": "Active",
"effectiveDate": "2016-02-01T00:00:00-06:00",
"formType": "Test Data",
"rateCode": "PASALERATE",
"rateCodeDesc": "Sale Rate - Agent",
"policyTypeCode": "26",
"policyTypeCodeDesc": "SALE OWNERS",
"amount": 180000,
"hoiAgentNumber": "",
"proForma": false,
"pdfLocation": "SomeLocation2",
"legacyPolicy": "true",
"associatedPolNbr": null
}
]
}
}
]
}
}
What am I doing wrong?

You can use a scripted update:
Put your new policy in a parameter, for example policy
Use a script like the following :
if (!ctxt._source.policies) { ctxt._source.policies = [] }
ctxt._source.policies += policy
See this documentation : https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html

Updates in inverted indexes are deletes and replacements of documents. There is no in-place update like you find in a db. ES uses Lucene under the hood which in-turn implements a kick-ass inverted index.

Elasticsearch grouping by nested fields

Is there a way to group by nested fields and perform aggregation on a non-nested fields??
I have data like this in ES
{
"_index": "bighalf",
"_type": "excel",
"_id": "AVE0rgXqe0-x669Gsae3",
"_score": 1,
"_source": {
"Name": "Marsh",
"date": "2015-11-07T10:47:14",
"grade": 9,
"year": 2016,
"marks": 70,
"subject": "Mathematics",
"Gender": "male",
"dob": "22/11/2000",
"sprint": [
{
"sprintdate": "2015-11-06T22:30:00",
"sprintname": "changed",
"sprintpoints": 52
}
]
}
},
{
"_index": "bighalf",
"_type": "excel",
"_id": "AVE0rvTHe0-x669Gsae5",
"_score": 1,
"_source": {
"Name": "Taylor",
"date": "2015-11-07T10:47:14",
"grade": 9,
"year": 2016,
"marks": 54,
"subject": "Mathematics",
"Gender": "male",
"dob": "22/11/2000",
"sprint": [
{
"sprintdate": "2015-11-07T22:30:00",
"sprintname": "jira",
"sprintpoints": 52
}
]
}
}
I wanted to group by sprintname and find sum of marks
I tried like this:
SumBuilder sumGrades = AggregationBuilders.sum("sum_grade").field("grade");
NestedBuilder nested = AggregationBuilders.nested("nested").path("sprint")
.subAggregation(AggregationBuilders.terms("by_sprint").field("sprint.sprintname").subAggregation(sumGrades));
String names[] = { "changed", "jira" };
QueryBuilder query = QueryBuilders.boolQuery().must(
QueryBuilders.nestedQuery("sprint",QueryBuilders.boolQuery().must(QueryBuilders.termsQuery("sprint.sprintname", names))));
FilterAggregationBuilder aggregation = AggregationBuilders.filter("agg").filter(query).subAggregation(nested);
the sum_grade did not work for me. But I changed field(grade) with nested field (sprintpoints) and it worked But my requirement is to find sum("grade") and group by sprint.sprintname.

Since your sprint field is of nested type, in your aggregation you need to use the reverse_nested aggregation in order to "jump back" at the root document from within your nested ones. It goes like this:
SumBuilder sumGrades = AggregationBuilders.sum("sum_grade").field("grade");
ReverseNestedBuilder backToGrades = AggregationBuilders.reverseNested("spring_to_grade")
.subAggregation(sumGrades);
TermsBuilder bySprint = AggregationBuilders.terms("by_sprint")
.field("sprint.sprintname").subAggregation(backToGrades)
NestedBuilder nested = AggregationBuilders.nested("nested").path("sprint")
.subAggregation(bySprint);
String names[] = { "changed", "jira" };
QueryBuilder query = QueryBuilders.boolQuery().must(
QueryBuilders.nestedQuery("sprint",QueryBuilders.boolQuery().must(QueryBuilders.termsQuery("sprint.sprintname", names))));
FilterAggregationBuilder aggregation = AggregationBuilders.filter("agg").filter(query).subAggregation(nested);