We Keep Coding

Nested field references in group operations - SpringData MongoDB. Trying to get the sum of cpu and memory_in_gb of all documents

I have the following mongodb document:
{
"_id" : ObjectId("5f283e7d39187d9ab77e7ece"),
"resourceType" : "VM",
"resourceInstanceName" : "virtual_machine_1",
"properties" :
{ "name" : "CentOS-VM", "cpu" : 2, "memory_in_gb" : 2, }
},
{
"_id" : ObjectId("5f28jh58hjf9ab77e7ece"),
"resourceType" : "VM",
"resourceInstanceName" : "virtual_machine_2",
"properties" :
{ "name" : "CentOS-VM", "cpu" : 8, "memory_in_gb" : 8, }
}
I use the following query in mongo shell which works fine
db.collection.aggregate({
$match:
{ "resourceType":"VM" }
}, {
$group: {
_id: '',
instance:
{ $sum: 1 }
,
cpu:
{ $sum: '$properties.cpu' }
,
memory_in_gb:
{ $sum: '$properties.memory_in_gb'}
}
})
and the output was
{ "_id" : "", "instance" : 2.0, "cpu" : 10, "memory_in_gb" : 10 }
using spring-data I have written the following code to produce the same result but it ends up in an Exception:
MatchOperation matchOperation = Aggregation.match(Criteria.where(RESOURCE_TYPE).is("VM"));
UnwindOperation unwindOperation = Aggregation.unwind("properties");
GroupOperation groupOperation = Aggregation.group(ID);
GroupOperation instanceOperation = Aggregation.group().count().as("instance");
GroupOperation cpuOperation = Aggregation.group("properties").sum("properties.cpu").as("cpu");
GroupOperation memoryOperation = Aggregation.group("properties").sum("properties.memory_in_gb").as("memory_in_gb");
Aggregation aggregation = Aggregation.newAggregation(matchOperation,unwindOperation,groupOperation,
instanceOperation, cpuOperation, memoryOperation);
return mongoTemplate.aggregate(aggregation, COLLECTION_NAME, Map.class).getMappedResults();
Here is the stackTrace:
[http-nio-9083-exec-2] ERROR c.s.c.c.v.service.ComputeService.getComputeSummary - Error in Cloud Account-summary :java.lang.IllegalArgumentException: Invalid reference 'properties'!
at org.springframework.data.mongodb.core.aggregation.ExposedFieldsAggregationOperationContext.getReference(ExposedFieldsAggregationOperationContext.java:114)
at org.springframework.data.mongodb.core.aggregation.ExposedFieldsAggregationOperationContext.getReference(ExposedFieldsAggregationOperationContext.java:77)
at org.springframework.data.mongodb.core.aggregation.AbstractAggregationExpression.unpack(AbstractAggregationExpression.java:74)

how to give $toObjectId in Java Mongo Projection Operation

I am new bee to mongo. Below is the Aggregation operation I am doing in mongodb shell . But In my java ProjectionAggregation , am not able to give the $toObjectId. Please correct me what am I missing.
db shell query
db.getCollection('UserData').aggregate([
{
$project : {
"username" : "$username",
"beneficiaries" : "$beneficiaries"
}
},
{
$unwind : {
path : "$beneficiaries",
preserveNullAndEmptyArrays: true
}
},
{
$project : {
"username" : "$username",
"beneficiaries" : "$beneficiaries",
---- dont know how to give $toObjectId in java ProjectionOperation .
"beneficiaryStudId" : { $toObjectId : "$beneficiaries.studentId" }
}
},
{
$lookup:
{
from: "StudentProfileData",
localField: "beneficiaryStudId",
foreignField: "_id",
as: "studProfile"
}
}
])
Java code Projection Operation
ProjectionOperation projectUserAndBeneficiaries = Aggregation.project()
.andExpression("username").as("username")
.andExpression("beneficiaries").as("beneficiaries");
ProjectionOperation projectUserAndOtherDetails = Aggregation.project()
.andExpression("username").as("username")
.andExpression("beneficiaries").as("beneficiaries")
---- How to give $toObjectId in projection operation .andExpression("beneficiaries.studentId").as("beneficiaryStudId");
LookupOperation lookupOperation = LookupOperation.newLookup().
from("StudentProfileData").
localField("beneficiaryStudId").
foreignField("_id").
as("studProfile");
Aggregation agg = Aggregation.newAggregation(projectUserAndBeneficiaries, unwindBeneficiars,
projectUserAndOtherDetails
,lookupOperation);
AggregationResults<UserAndStudentData> output
= mongotemplate.aggregate(agg, "UserData", UserAndStudentData.class);
Sample Ouput
Output in db shell
{
"_id" : ObjectId("5d2f08574de2690001c281ac"),
"username" : "ks241#goo.com",
"beneficiaries" : {
"studentId" : "5d2f0e9c3bcf3e0001a7e562",
"mcBeneficiaryId" : "597418",
"enabled" : true
},
"beneficiaryStudId" : ObjectId("5d2f0e9c3bcf3e0001a7e562"),
"studProfile" : [
{
"_id" : ObjectId("5d2f0e9c3bcf3e0001a7e562"),
"lastName" : "Sharma",
"firstName" : "Kapil",
"studentRegisterCustomFieldValues" : [
{
"bcfdValue" : "One",
"bcfdName" : "Year"
}
],
"gender" : "M",
"merchantId" : "38788943"
}
]
}
where as in java
the studProfile array is always empty if I add $toObjectId the above java aggregation projection query and ran it produces the values as same as that db shell .

Spring data doesn't support to few type of methods. This problem may include into it. But we may use this solution.
Aggregation aggregation=newAggregation(
p-> new Document("$project",
new Document()
.append("username","$username"),
.append("beneficiaries","$beneficiaries)
.append("beneficiaryStudId",
new Document("$toObjectId","$beneficiaries.studentId")
)
)
)

There is a way to do this with Spring's ProjectionOperation and ConvertOperators classes:
ProjectionOperation projection = Aggregation.project()
.andInclude("username", "beneficiaries")
.and(ConvertOperators.ToObjectId.toObjectId("$beneficiaries.studentId"))
.as("beneficiaryStudId");

Two Aggregate Totals in One Group

I wrote a query in MongoDB as follows:
db.getCollection('student').aggregate(
[
{
$match: { "student_age" : { "$ne" : 15 } }
},
{
$group:
{
_id: "$student_name",
count: {$sum: 1},
sum1: {$sum: "$student_age"}
}
}
])
In others words, I want to fetch the count of students that aren't 15 years old and the summary of their age. The query works fine and I get two data items.
In my application, I want to do the query by Spring Data.
I wrote the following code:
Criteria where = Criteria.where("AGE").ne(15);
Aggregation aggregation = Aggregation.newAggregation(
Aggregation.match(where),
Aggregation.group().sum("student_age").as("totalAge"),
count().as("countOfStudentNot15YearsOld"));
When this code is run, the output query will be:
"aggregate" : "MyDocument", "pipeline" :
[ { "$match" { "AGE" : { "$ne" : 15 } } },
{ "$group" : { "_id" : null, "totalAge" : { "$sum" : "$student_age" } } },
{ "$count" : "countOfStudentNot15YearsOld" }],
"cursor" : { "batchSize" : 2147483647 }
Unfortunately, the result is only countOfStudentNot15YearsOld item.
I want to fetch the result like my native query.

If your're asking to return the grouping for both "15" and "not 15" as a result then you're looking for the $cond operator which will allow a "branching" based on conditional evaluation.
From the "shell" content you would use it like this:
db.getCollection('student').aggregate([
{ "$group": {
"_id": null,
"countFiteen": {
"$sum": {
"$cond": [{ "$eq": [ "$student_age", 15 ] }, 1, 0 ]
}
},
"countNotFifteen": {
"$sum": {
"$cond": [{ "$ne": [ "$student_age", 15 ] }, 1, 0 ]
}
},
"sumNotFifteen": {
"$sum": {
"$cond": [{ "$ne": [ "$student_age", 15 ] }, "$student_age", 0 ]
}
}
}}
])
So you use the $cond to perform a logical test, in this case whether the "student_age" in the current document being considered is 15 or not, then you can return a numerical value in response which is 1 here for "counting" or the actual field value when that is what you want to send to the accumulator instead. In short it's a "ternary" operator or if/then/else condition ( which in fact can be shown in the more expressive form with keys ) you can use to test a condition and decide what to return.
For the spring mongodb implementation you use ConditionalOperators.Cond to construct the same BSON expressions:
import org.springframework.data.mongodb.core.aggregation.*;
ConditionalOperators.Cond isFifteen = ConditionalOperators.when(new Criteria("student_age").is(15))
.then(1).otherwise(0);
ConditionalOperators.Cond notFifteen = ConditionalOperators.when(new Criteria("student_age").ne(15))
.then(1).otherwise(0);
ConditionalOperators.Cond sumNotFifteen = ConditionalOperators.when(new Criteria("student_age").ne(15))
.thenValueOf("student_age").otherwise(0);
GroupOperation groupStage = Aggregation.group()
.sum(isFifteen).as("countFifteen")
.sum(notFifteen).as("countNotFifteen")
.sum(sumNotFifteen).as("sumNotFifteen");
Aggregation aggregation = Aggregation.newAggregation(groupStage);
So basically you just extend off of that logic, using .then() for a "constant" value such as 1 for the "counts", and .thenValueOf() where you actually need the "value" of a field from the document, so basically equal to the "$student_age" as shown for the common shell notation.
Since ConditionalOperators.Cond shares the AggregationExpression interface, this can be used with .sum() in the form that accepts an AggregationExpression as opposed to a string. This is an improvement on past releases of spring mongo which would require you to perform a $project stage so there were actual document properties for the evaluated expression prior to performing a $group.
If all you want is to replicate the original query for spring mongodb, then your mistake was using the $count aggregation stage rather than appending to the group():
Criteria where = Criteria.where("AGE").ne(15);
Aggregation aggregation = Aggregation.newAggregation(
Aggregation.match(where),
Aggregation.group()
.sum("student_age").as("totalAge")
.count().as("countOfStudentNot15YearsOld")
);

Get count of unique ObjectId from array MongoDB

I'm new to working with MongoDb and do not know a lot of things.
I need to write an aggregation request.
Here is the JSON document structure.
{
"_id" : ObjectId("5a72f7a75ef7d430e8c462d2"),
"crawler_id" : ObjectId("5a71cbb746e0fb0007adc6c2"),
"skill" : "stack",
"created_date" : ISODate("2018-02-01T13:19:03.522+0000"),
"modified_date" : ISODate("2018-02-01T13:22:23.078+0000"),
"connects" : [
{
"subskill" : "we’re",
"weight" : NumberInt(1),
"parser_id" : [
ObjectId("5a71d88d5ef7d41964fbec11")
]
},
{
"subskill" : "b1",
"weight" : NumberInt(2),
"parser_id" : [
ObjectId("5a71d88d5ef7d41964fbec11"),
ObjectId("5a71d88d5ef7d41964fbec1b")
]
},
{
"subskill" : "making",
"weight" : NumberInt(2),
"parser_id" : [
ObjectId("5a71d88d5ef7d41964fbec1b"),
ObjectId("5a71d88d5ef7d41964fbec1c")
]
},
{
"subskill" : "delivery",
"weight" : NumberInt(2),
"parser_id" : [
ObjectId("5a71d88d5ef7d41964fbec1c"),
ObjectId("5a71d88d5ef7d41964fbec1e")
]
}
]
}
I need the result return the name of skill and the number of unique parser_id.
In this case, the result should be:
[
{
"skill": "stack",
"quantity": 4
}
]
where "stack" - skill name,
and "quantity" - count of unique parser_id.
ObjectId("5a71d88d5ef7d41964fbec11")
ObjectId("5a71d88d5ef7d41964fbec1b")
ObjectId("5a71d88d5ef7d41964fbec1c")
ObjectId("5a71d88d5ef7d41964fbec1e")
Can some one help me with this request ???

Given the document supplied in your question, this command ...
db.collection.aggregate([
{ $unwind: "$connects" },
// count all occurrences
{ "$group": { "_id": {skill: "$skill", parser_id: "$connects.parser_id"}, "count": { "$sum": 1 } }},
// sum all occurrences and count distinct
{ "$group": { "_id": "$_id.skill", "quantity": { "$sum": 1 } }},
// (optional) rename the '_id' attribute to 'skill'
{ $project: { 'skill': '$_id', 'quantity': 1, _id: 0 } }
])
... will return:
{
"quantity" : 4,
"skill" : "stack"
}
The above command groups by skill and connects.parser_id and then gets a distinct count of those groups.
Your command includes the java tag so I suspect you are looking to execute the same command using the MongoDB Java driver. The code below (using MongoDB Java driver v3.x) will return the same result:
MongoClient mongoClient = ...;
MongoCollection<Document> collection = mongoClient.getDatabase("...").getCollection("...");
List<Document> documents = collection.aggregate(Arrays.asList(
Aggregates.unwind("$connects"),
new Document("$group", new Document("_id", new Document("skill", "$skill").append("parser_id", "$connects.parser_id"))
.append("count", new Document("$sum", 1))),
new Document("$group", new Document("_id", "$_id.skill").append("quantity", new Document("$sum", 1))),
new Document("$project", new Document("skill", "$_id").append("quantity", 1).append("_id", 0))
)).into(new ArrayList<>());
for (Document document : documents) {
logger.info("{}", document.toJson());
}
Note: this code deliberately uses the form new Document(<pipeline aggregator>, ...) instead of the Aggregators utilities to make it easier to see the translation between the shell command and its Java equivalent.

try $project with $reduce
$setUnion is used to keep only the distinct ids and finally $size used to get the distinct array count
db.col.aggregate(
[
{$project : {
_id : 0,
skill : 1,
quantity : {$size :{$reduce : {input : "$connects.parser_id", initialValue : [] , in : {$setUnion : ["$$value", "$$this"]}}}}
}
}
]
).pretty()
result
{ "skill" : "stack", "quantity" : 4 }

Spring Data Mongo Template - Counting an array

Mongo document:
{
"_id" : "1",
"array" : [
{
"item" : "item"
},
{
"item" : "item"
}
]
}
My mongo shell query looks like so:
db.getCollection('collectionName').aggregate(
{$match: { _id: "1"}},
{$project: { count: { $size:"$array" }}}
)
Is there anyway to implement this using the Mongo Template from Spring?
So far I have this:
MatchOperation match = new MatchOperation(Criteria.where("_id").is("1"));
ProjectionOperation project = new ProjectionOperation();
Aggregation aggregate = Aggregation.newAggregation(match, project);
mongoTemplate.aggregate(aggregate, collectionName, Integer.class);
I think I am only missing the project logic but I'm not sure if it is possible to do $size or equivalent here.

It's quite possible, the $size operator is supported (see DATAMONGO-979 and its implementation here). Your implementation could follow this example:
import static org.springframework.data.mongodb.core.aggregation.Aggregation.*;
Aggregation agg = new Aggregation(
match(where("_id").is("1")), //
project() //
.and("array") //
.size() //
.as("count")
);
AggregationResults<IntegerCount> results = mongoTemplate.aggregate(
agg, collectionName, Integer.class
);
List<IntegerCount> intCount = results.getMappedResults();

Please find below the sample code. You can change it accordingly for your requirement with collection name, collection name class and array field name.
MatchOperation match = new MatchOperation(Criteria.where("_id").is("1"));
Aggregation aggregate = Aggregation.newAggregation(match, Aggregation.project().and("array").project("size").as("count"));
AggregationResults<CollectionNameClass> aggregateResult = mongoOperations.aggregate(aggregate, "collectionName", <CollectionNameClass>.class);
if (aggregateResult!=null) {
//You can find the "count" as an attrribute inside "result" key
System.out.println("Output ====>" + aggregateResult.getRawResults().get("result"));
System.out.println("Output ====>" + aggregateResult.getRawResults().toMap());
}
Sample output:-
Output ====>[ { "_id" : "3ea8e671-1e64-4cde-bd78-5980049a772b" , "count" : 47}]
Output ====>{serverUsed=127.0.0.1:27017, waitedMS=0, result=[ { "_id" : "3ea8e671-1e64-4cde-bd78-5980049a772b" , "count" : 47}], ok=1.0}

You can write query as
Aggregation aggregate = Aggregation.newAggregation(Aggregation.match(Criteria.where("_id").is(1)),
Aggregation.project().and("array").size().as("count")); mongoTemplate.aggregate(aggregate, collectionName, Integer.class);
It will execute the following query { "aggregate" : "collectionName" , "pipeline" : [ { "$match" : { "_id" : 1}} , { "$project" : { "count" : { "$size" : [ "$array"]}}}]}