I am trying to convert MongoDB Aggregation function to Java Aggregation function.
My MongoDB Query is
[
{
"$match": {
"indicesId": "VUSSTAPNETFF"
}
},
{
"$unwind": "$dataSets"
},
{
"$match": {
"dataSets.date": {
"$lte": ISODate("2013-12-31T18:30:00.000Z"),
"$gte": ISODate("2008-12-31T18:30:00.000Z")
}
}
},
{
"$group": {
"_id": "$indicesId",
"mean": {
"$avg": "$dataSets.data"
},
"indices": {
"$push": "$$ROOT"
}
}
},
{
"$unwind": "$indices"
},
{
"$project": {
"_id": 1,
"mean": 1,
"firstResult": {
"$multiply": [
{
"$subtract": [
"$indices.dataSets.data",
"$mean"
]
},
{
"$subtract": [
"$indices.dataSets.data",
"$mean"
]
}
]
}
}
},
{
"$group":{
"_id":"",
"secondResult": {
"$avg": "$firstResult"
},
"mean":{
"$first": "$mean"
}
}
},
{
"$project":{
"data":{
"$sqrt":"$secondResult"
},
"mean": "$mean"
}
}
]
I tried bellow code for conversion in java.
BasicDBList subtractDBList= new BasicDBList();
subtractDBList.add("$indices.dataSets.data");
subtractDBList.add("$mean");
BasicDBList multiplyDBList= new BasicDBList();
multiplyDBList.add(new BasicDBObject("$subtract",subtractDBList));
multiplyDBList.add(new BasicDBObject("$subtract",subtractDBList));
Aggregation meanAggregation= Aggregation.newAggregation(Aggregation.match(Criteria.where(IndicesUtil.INDICES_ID).is(indicesId)),
Aggregation.unwind(IndicesUtil.PREFIX+IndicesUtil.DATA_SETS),
Aggregation.match(Criteria.where(IndicesUtil.DATA_SETS_DATE).gte(startDate).lte(indicesDataSet.getDate())),
Aggregation.group(IndicesUtil.INDICES_ID).avg(averageParameter).as(IndicesUtil.MEAN).push("$$ROOT").as("indices"),
Aggregation.unwind(IndicesUtil.PREFIX+IndicesUtil.INDICES),
new AggregationOperation() {
#Override
public DBObject toDBObject(AggregationOperationContext arg0) {
return new BasicDBObject("$project",
new BasicDBObject("_id",1).append("mean", IndicesUtil.PREFIX+IndicesUtil.MEAN).append("firstResult",
new BasicDBObject("$multiply",multiplyDBList)));
}
},
Aggregation.group().avg("$firstResult").as("secondResult").first(IndicesUtil.PREFIX+IndicesUtil.MEAN).as(IndicesUtil.MEAN),
new AggregationOperation() {
#Override
public DBObject toDBObject(AggregationOperationContext arg0) {
return new BasicDBObject("$project",
new BasicDBObject(IndicesUtil.DATA,
new BasicDBObject("$sqrt","$secondResult").append("mean", IndicesUtil.PREFIX+IndicesUtil.MEAN)));
}
}
);
I am getting problem with bellow line
Aggregation.group().avg("$firstResult").as("secondResult").first(IndicesUtil.PREFIX+IndicesUtil.MEAN).as(IndicesUtil.MEAN),
And error as per bellow
Invalid reference '$firstResult'!
Thanks in advance.
You can simplify your aggregation on 1.10.1-Release spring mongo version. You should avoid using BasicDBObject/Document and use spring provided helper methods.
Aggregation meanAggregation = Aggregation.newAggregation(
Aggregation.match(Criteria.where(IndicesUtil.INDICES_ID).is(indicesId)),
Aggregation.unwind(IndicesUtil.PREFIX+IndicesUtil.DATA_SETS),
Aggregation.match(Criteria.where(IndicesUtil.DATA_SETS_DATE).gte(startDate).lte(indicesDataSet.getDate())),
Aggregation.group(IndicesUtil.INDICES_ID).avg(averageParameter).as(IndicesUtil.MEAN).push("$$ROOT").as("indices"),
Aggregation.unwind(IndicesUtil.PREFIX+IndicesUtil.INDICES),
Aggregation.project("_id", "mean").andExpression("(indices.dataSets.data - mean) * (indices.dataSets.data - mean)").as("firstResult"),
Aggregation.group().avg("$firstResult").as("secondResult").first(IndicesUtil.PREFIX+IndicesUtil.MEAN).as(IndicesUtil.MEAN),
Aggregation.project("mean").and("secondResult").sqrt().as(IndicesUtil.DATA)
);
Related
I have data like this:
{ id : 1,
book: "Flash",
chapters: [
{
chap_no: "1",
sub_chapter: [
{sub_no: 1, description: "<description>"
},
{sub_no: 2, description: "<description>"
},
]
}
]
}
i want to show one field like this base on book -> chapter_no -> sub_no
{
sub_no: 2, description: "<description>"
}
in mongodb query.
$match
$unwind
$unwind
$match
$replaceRoot
db.collection.aggregate([
{
"$match": {
"chapters.sub_chapter.sub_no": 2
}
},
{
"$unwind": "$chapters"
},
{
"$unwind": "$chapters.sub_chapter"
},
{
"$match": {
"chapters.sub_chapter.sub_no": 2
}
},
{
"$replaceRoot": {
"newRoot": "$chapters.sub_chapter"
}
}
])
mongoplayground
you can make like this
db.collection.aggregate([
{
"$match": {
$and: [
{
"book": "Flash3"
},
{
"chapters.chap_no": "2"
},
{
"chapters.sub_chapter.sub_no": "1"
}
]
}
},
{
"$unwind": "$chapters"
},
{
"$unwind": "$chapters.sub_chapter"
},
{
"$match": {
$and: [
{
"book": "Flash3"
},
{
"chapters.chap_no": "2"
},
{
"chapters.sub_chapter.sub_no": "1"
}
]
}
},
{
"$replaceRoot": {
"newRoot": "$chapters.sub_chapter"
}
}
])
I want to get distinct categoryCode and categoryName while this document also contains list of accessories I want to count of this accessories in the response.
Sample data:
[
{
"categoryCode":"categoryCode1",
"categoryName":"categoryName1",
"accessories":[{"a_id":1},{"a_id":2}]
},
{
"categoryCode":"categoryCod2",
"categoryName":"categoryName2",
"accessories":[{"a_id":1},{"a_id":2},{"a_id":3}]
},
{
"categoryCode":"categoryCode1",
"categoryName":"categoryNam1",
"accessories":[{"a_id":1},{"a_id":2}]
}
]
Expected result:
[
{
"categoryCode":"categoryCode1",
"categoryName":"categoryName1",
"accessoriesCount":2
},
{
"categoryCode":"categoryCod2",
"categoryName":"categoryName2",
"accessoriesCount":3
}
]
https://mongoplayground.net/p/q6AZOaTwo5a
db.collection.aggregate([
{
"$group": {
"_id": {
categoryCode: "$categoryCode",
"categoryName": "$categoryName"
},
"accessories": {
"$addToSet": "$accessories"
}
}
},
{
"$project": {
categoryCode: "$_id.categoryCode",
categoryName: "$_id.categoryName",
accessoriesCount: {
$size: "$accessories"
},
_id: 0
}
}
])
Query
group to have the distinct values
$push the accesories arrays (we dont have $concat accumulator)
reduce those arrays to union them, keep only the distinct members, and take the count.
Test code here
db.collection.aggregate([
{
"$group": {
"_id": {
"categoryCode": "$categoryCode",
"categoryName": "$categoryName"
},
"accessories": {
"$push": "$accessories"
}
}
},
{
"$set": {
"accessoriesCount": {
"$size": {
"$reduce": {
"input": "$accessories",
"initialValue": [],
"in": {
"$setUnion": [
"$$value",
"$$this"
]
}
}
}
}
}
},
{
"$project": {
"_id": 0,
"categoryCode": "$_id.categoryCode",
"categoryName": "$_id.categoryName",
"accessoriesCount": "$accessoriesCount"
}
}
])
Assume I have two mongo collections as follows.
Collection A
{
"_id" : ObjectId("582abcd85d2dfa67f44127e0"),
"level" : "super"
"dataReference" : Object
B : DbRef(B, 5b618a570550de0021aaa2ef, undefined)
}
Collection B
{
"_id" : ObjectId("5b618a570550de0021aaa2ef"),
"role" : "admin"
}
What I need is retrieve the records from Collection A, which records have "level" field's value as "super" and its related Collection B record's "role" value as "admin".
For this, I am trying to use aggregation and java mongoTemplate.
Following is the code that I tried but it returns 0 records.
final TypedAggregation<A> typedAggregation = Aggregation.newAggregation(A.class,
Aggregation.match(Criteria.where("level").equals(level)),
Aggregation.lookup("B", "_id", "dataReference.B.$id", "Basb"),
Aggregation.match(new Criteria().andOperator(
Criteria.where("B.role").regex("admin")
)));
final AggregationResults<Map> A = mongoTemplate.aggregate(typedAggregation, "A", Map.class);
Please note that I am new to Mongo aggregation.
It's quiet ugly solution:
MongoTemplate
You cannot use TypedAggregation because we need to transform A collection to be able join with B collection
Aggregation typedAggregation = Aggregation.newAggregation(
Aggregation.match(Criteria.where("level").is("super")),
new AggregationOperation() {
#Override
public Document toDocument(AggregationOperationContext context) {
return Document.parse("{\"$addFields\":{\"dataReference\":{\"$reduce\":{\"input\":{\"$objectToArray\":\"$dataReference\"},\"initialValue\":null,\"in\":{\"$cond\":[{\"$eq\":[{\"$type\":\"$$this.v\"},\"objectId\"]},\"$$this.v\",\"$$value\"]}}}}}");
}
},
Aggregation.lookup("B", "dataReference", "_id", "B"),
Aggregation.match(new Criteria().andOperator(
Criteria.where("B.role").regex("admin")
)
)
);
final AggregationResults<Document> A = mongoTemplate.aggregate(typedAggregation, "A", Document.class);
MongoDB Aggregation
db.A.aggregate([
{
"$match": {
"level": "super"
}
},
{
"$addFields": {
"B": {
"$reduce": {
"input": {
"$objectToArray": "$dataReference"
},
"initialValue": null,
"in": {
"$cond": [
{
"$eq": [
{
"$type": "$$this.v"
},
"objectId"
]
},
"$$this.v",
"$$value"
]
}
}
}
}
},
{
"$lookup": {
"from": "B",
"localField": "B",
"foreignField": "_id",
"as": "B"
}
},
{
"$match": {
"$and": [
{
"B.role": {
"$regex": "admin",
"$options": ""
}
}
]
}
}
])
MongoPlayground
I have the below list that has duplicate elements
I want to remove all duplicates from the list based on the version and date property
This means, if there's a diplicate element, i get the one that has state actif, if no one has state actif then i get the one with the recent date
[
{
"_id" : ObjectId("5e1832df02f04352705457dd"),
"product":"1",
"version":{
"state":"Actif",
"name":"1.0.0"
},
"createdDate":"01/01/2020"
},
{
"_id" : ObjectId("5e1832df02f04352705457ff"),
"product":"1",
"version":{
"state":"A faire",
"name":"3.0.0"
},
"createdDate":"01/01/2020"
},
{
"_id" : ObjectId("5e1832df02f04352705457ee"),
"product":"1",
"version":{
"state":"Archiver",
"name":"2.0.0"
},
"createdDate":"02/01/2020"
},
{
"_id" : ObjectId("5e1832df02f04352705457gg"),
"product":"2",
"version":null,
"createdDate":"01/01/2020"
},
{
"_id" : ObjectId("5e1832df02f04352705457yy"),
"product":"2",
"version":{
"state":"Archiver",
"name":"2.0.0"
},
"createdDate":"02/01/2020"
},
{
"_id" : ObjectId("5e1832df02f04352705455ss"),
"product":"3",
"version":{
"state":"Archiver",
"name":"2.0.0"
},
"createdDate":"01/01/2020"
}
]
The output should look like:
[
{
"_id" : ObjectId("5e1832df02f04352705457dd"),
"product":"1",
"version":{
"state":"Actif",
"name":"1.0.0"
},
"createdDate":"01/01/2020"
},
{
"_id" : ObjectId("5e1832df02f04352705457yy"),
"product":"2",
"version":{
"state":"Archiver",
"name":"2.0.0"
},
"createdDate":"02/01/2020"
},
{
"_id" : ObjectId("5e1832df02f04352705455ss"),
"product":"3",
"version":{
"state":"Archiver",
"name":"2.0.0"
},
"createdDate":"01/01/2020"
}
]
public List<Product> search() {
final Query query = new Query().with(new Sort(new Order(Direction.DESC, "createdDate")));
return mongoOperations.find(query, Product.class);
}
How can we do this ?
You need to use aggregate method.
Java code
Not tested!
If AggregationOperation expects to implement toDocument method, change all BasicDBObject to Document.
import static org.springframework.data.mongodb.core.aggregation.Aggregation.*;
Aggregation aggregation = newAggregation(
sort(Sort.Direction.ASC, "product", "version", "createdDate"),
group("product").last("$$ROOT").as("root"),
sort(Sort.Direction.ASC, "_id"),
//$replaceRoot
);
return mongoTemplate.aggregate(aggregation, Product.class, Product.class).getMappedResults();
MongoDB shell
db.Product.aggregate([
{
$sort: {
product: 1,
version: -1,
createdDate: -1
}
},
{
$group: {
_id: "$product",
root: {
$push: "$$ROOT"
}
}
},
{
$sort: {
_id: 1
}
},
{
$replaceRoot: {
newRoot: {
$arrayElemAt: [
{
$concatArrays: [
{
$filter: {
input: "$root",
cond: {
$eq: [
"$$this.version.state",
"Actif"
]
}
}
},
[
{
$arrayElemAt: [
"$root",
0
]
}
]
]
},
0
]
}
}
}
])
MongoPlayground
The aggregation using Spring Data 2.2 and MongoDB 3.4 compatible aggregation operators.
MongoOperations mongoOps = new MongoTemplate(MongoClients.create(), "spr_test");
Aggregation agg = newAggregation(
project("product", "version")
.and(Concat.valueOf(SubstrCP.valueOf("createdDate").substringCP(6, 4))
.concat("-")
.concatValueOf(SubstrCP.valueOf("createdDate").substringCP(3, 2))
.concat("-")
.concatValueOf(SubstrCP.valueOf("createdDate").substringCP(0, 2)) )
.as("createdDate"),
group("product")
.push("$$ROOT").as("details")
.push("createdDate").as("createdDates"),
facet(
project("details")
.and(Size.lengthOfArray("details")).as("arrSize"),
match(where("arrSize").is(new Integer(1))))
.as("c1")
.and(
match(where("details.version.state").is("Actif" )),
project()
.and(filter("details")
.as("d")
.by(Eq.valueOf("d.version.state").equalToValue("Actif" )))
.as("details"))
.as("c2")
.and(
project("details", "createdDates")
.and(Size.lengthOfArray("details")).as("arrSize"),
match(where("arrSize").gt(new Integer(1))
.and("details.version.state").ne("Actif")),
project()
.and(filter("details")
.as("d")
.by(Eq.valueOf("d.createdDate").equalTo(Max.maxOf("createdDates"))))
.as("details"))
.as("c3"),
project()
.and(arrayOf("c1").concat("c2").concat("c3"))
.as("result"),
unwind("result"),
project()
.and(arrayOf("result.details").elementAt(0)).as("result")
);
AggregationResults<Document> results = mongoOps.aggregate(agg, "test", Document.class);
results.forEach(System.out::println);
I have a document in collection as :
{
"_id": ObjectId("5ab273ed31fa764560a912f8"),
"hourNumber": 21,
"errorSegments": [{
"agentName": "agentX"
},
{
"agentName": "agentY"
}]
}
I am trying to perform following aggregation function in spring boot where i want to retrieve "errorSegments" of a particular hour that matches an agent , which is working fine in mongo shell :
Working shell :
db.errorsegment.aggregate([{
"$match": {
"hourNumber": 21,
"errorSegments.agentName": "agentX"
}
},
{
"$project": {
"errorSegments": {
"$filter": {
"input": "$errorSegments",
"as": "e",
"cond": {
"$eq": ["$$e.agentName",
"agentX"]
}
}
}
}
},
{
"$unwind": "$errorSegments"
},
{
"$replaceRoot": {
"newRoot": "$errorSegments"
}
}])
So it provides output only , which is desired result :
{ "agentName" : "agentX" }
But my following code in spring gives error :
MatchOperation match = Aggregation.match(Criteria.where("hourNumber").is(21).and("errorSegments.agentName").is("agentX"));
ProjectionOperation project = Aggregation.project()
.and(new AggregationExpression() {
#Override
public DBObject toDbObject(AggregationOperationContext context) {
DBObject filterExpression = new BasicDBObject();
filterExpression.put("input", "$errorSegments");
filterExpression.put("as", "e");
filterExpression.put("cond", new BasicDBObject("$eq", Arrays.<Object> asList("$$e.agentName","agentX")));
return new BasicDBObject("$filter", filterExpression);
} }).as("prop");
UnwindOperation unwind = Aggregation.unwind("$errorSegments");
ReplaceRootOperation replaceRoot = Aggregation.replaceRoot("$errorSegments");
Aggregation aggregation = Aggregation.newAggregation(match,project,unwind,replaceRoot);
AggregationResults<ErrorSegment> errorSegments = mongoOps.aggregate(aggregation, SegmentAudit.class , ErrorSegment.class);
Following is the Logs :
java.lang.IllegalArgumentException: Invalid reference 'errorSegments'!
at org.springframework.data.mongodb.core.aggregation.ExposedFieldsAggregationOperationContext.getReference(ExposedFieldsAggregationOperationContext.java:99) ~[spring-data-mongodb-1.10.3.RELEASE.jar:na]
at org.springframework.data.mongodb.core.aggregation.ExposedFieldsAggregationOperationContext.getReference(ExposedFieldsAggregationOperationContext.java:71) ~[spring-data-mongodb-1.10.3.RELEASE.jar:na]
at org.springframework.data.mongodb.core.aggregation.UnwindOperation.toDBObject(UnwindOperation.java:95) ~[spring-data-mongodb-1.10.3.RELEASE.jar:na]
at org.springframework.data.mongodb.core.aggregation.AggregationOperationRenderer.toDBObject(AggregationOperationRenderer.java:56) ~[spring-data-mongodb-1.10.3.RELEASE.jar:na]
at org.springframework.data.mongodb.core.aggregation.Aggregation.toDbObject(Aggregation.java:580) ~[spring-data-mongodb-1.10.3.RELEASE.jar:na]
at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:1567) ~[spring-data-mongodb-1.10.3.RELEASE.jar:na]
at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:1502) ~[spring-data-mongodb-1.10.3.RELEASE.jar:na]
The source of error is the alias that you use with filter operation. It should be errorSegments instead of prop but you you other problems too. Use fieldname i.e $ prefix is not right.
Here is the updated aggregation. You can use $filter helper.
MatchOperation match = Aggregation.match(Criteria.where("hourNumber").is(21).and("errorSegments.agentName").is("agentX"));
ProjectionOperation project = Aggregation.
project().and(ArrayOperators.Filter.filter("errorSegments")
.as("e")
.by(ComparisonOperators.Eq.valueOf(
"e.agentName")
.equalToValue(
"agentX")))
.as("errorSegments");
UnwindOperation unwind = Aggregation.unwind("errorSegments");
ReplaceRootOperation replaceRoot = Aggregation.replaceRoot("errorSegments");
Aggregation aggregation = Aggregation.newAggregation(match,project,unwind,replaceRoot);
Below is the generated query.
[
{
"$match": {
"hourNumber": 21,
"errorSegments.agentName": "agentX"
}
},
{
"$project": {
"errorSegments": {
"$filter": {
"input": "$errorSegments",
"as": "e",
"cond": {
"$eq": [
"$$e.agentName",
"agentX"
]
}
}
}
}
},
{
"$unwind": "$errorSegments"
},
{
"$replaceRoot": {
"newRoot": "$errorSegments"
}
}
]