MongoDB Data:
{
"_id" : ObjectId("123"),
"attr" : [
{
"nameLable" : "First Name",
"userEnteredValue" : [
"Amanda"
],
"rowNumber":"1"
},
{
"nameLable" : "Last Name",
"userEnteredValue" : [
"Peter"
],
"rowNumber":"1"
},
{
"nameLable" : "First Name",
"userEnteredValue" : [
"Sandra"
],
"rowNumber":"2"
},
{
"nameLable" : "Last Name",
"userEnteredValue" : [
"Peter"
],
"rowNumber":"2"
}
]
}
Matching (First Name equals "Amanda" && Last Name equals "Peter") -> Match should happen within rowNumber so that i will get rowNumber1 record but now i am getting both rows as "Peter" happens to be in both "rowNumber" attribute.
Criteria Code:
Criteria cr = Criteria.where("attr").elemMatch(Criteria.where("nameLable").is(map.get("value1")).and("userEnteredValue").regex(map.get("value2").trim(), "i"); //Inside loop
AggregationOperation match = Aggregation.match(Criteria.where("testId").is("test").andOperator(cr.toArray(new Criteria[criteria.size()])));
DB Query for above search Criteria Match:
db.Col1.aggregate([
{
"$match":{
"testId":"test",
"$and":[
{
"attr":{
"$elemMatch":{
"nameLable":"First Name",
"userEnteredValue":{
"$regex":"Amanda",
"$options":"i"
}
}
}
},
{
"attr":{
"$elemMatch":{
"nameLable":"Last Name",
"userEnteredValue":{
"$regex":"Peter",
"$options":"i"
}
}
}
}
]
}
}
]
)
Please let me know how can we do match within "rowNumber" attribute.
Let me start by recommending you reconsider your document structure, I do not know your product but this structure is very unique and definitely makes most "simple" access patterns I can think of to very cumbersome to execute. This will be noticeable in my answer.
So the current query you have just required 2 separate elements in the array exist, as you mentioned you want the same rowNumber, due to the document structure this isn't really queryable, we will have to first use your query to match "potential" matching documents. At that point we can filter our the matched rows and see if we have both a first name and a last name matching.
Finally we could filter out the none matching rows from the result, here is the pipeline:
db.collection.aggregate([
{
"$match": {
"testId": "test",
"$and": [
{
"attr": {
"$elemMatch": {
"nameLable": "First Name",
"userEnteredValue": {
"$regex": "Amanda",
"$options": "i"
}
}
}
},
{
"attr": {
"$elemMatch": {
"nameLable": "Last Name",
"userEnteredValue": {
"$regex": "Peter",
"$options": "i"
}
}
}
}
]
}
},
{
$addFields: {
goodRows: {
"$setIntersection": [
{
$map: {
input: {
$filter: {
input: "$attr",
cond: {
$and: [
{
$eq: [
"$$this.nameLable",
"First Name"
]
},
{
"$regexMatch": {
"input": {
"$arrayElemAt": [
"$$this.userEnteredValue",
0
]
},
"regex": "Amanda",
"options": "i"
}
}
]
}
}
},
in: "$$this.rowNumber"
}
},
{
$map: {
input: {
$filter: {
input: "$attr",
cond: {
$and: [
{
$eq: [
"$$this.nameLable",
"Last Name"
]
},
{
"$regexMatch": {
"input": {
"$arrayElemAt": [
"$$this.userEnteredValue",
0
]
},
"regex": "Peter",
"options": "i"
}
}
]
}
}
},
in: "$$this.rowNumber"
}
}
]
}
}
},
{
$match: {
$expr: {
$gt: [
{
$size: "$goodRows"
},
0
]
}
}
},
{
$addFields: {
attr: {
$filter: {
input: "$attr",
cond: {
$in: [
"$$this.rowNumber",
"$goodRows"
]
}
}
}
}
}
])
Mongo Playground
I have data like this:
{ id : 1,
book: "Flash",
chapters: [
{
chap_no: "1",
sub_chapter: [
{sub_no: 1, description: "<description>"
},
{sub_no: 2, description: "<description>"
},
]
}
]
}
i want to show one field like this base on book -> chapter_no -> sub_no
{
sub_no: 2, description: "<description>"
}
in mongodb query.
$match
$unwind
$unwind
$match
$replaceRoot
db.collection.aggregate([
{
"$match": {
"chapters.sub_chapter.sub_no": 2
}
},
{
"$unwind": "$chapters"
},
{
"$unwind": "$chapters.sub_chapter"
},
{
"$match": {
"chapters.sub_chapter.sub_no": 2
}
},
{
"$replaceRoot": {
"newRoot": "$chapters.sub_chapter"
}
}
])
mongoplayground
you can make like this
db.collection.aggregate([
{
"$match": {
$and: [
{
"book": "Flash3"
},
{
"chapters.chap_no": "2"
},
{
"chapters.sub_chapter.sub_no": "1"
}
]
}
},
{
"$unwind": "$chapters"
},
{
"$unwind": "$chapters.sub_chapter"
},
{
"$match": {
$and: [
{
"book": "Flash3"
},
{
"chapters.chap_no": "2"
},
{
"chapters.sub_chapter.sub_no": "1"
}
]
}
},
{
"$replaceRoot": {
"newRoot": "$chapters.sub_chapter"
}
}
])
Assume I have two mongo collections as follows.
Collection A
{
"_id" : ObjectId("582abcd85d2dfa67f44127e0"),
"level" : "super"
"dataReference" : Object
B : DbRef(B, 5b618a570550de0021aaa2ef, undefined)
}
Collection B
{
"_id" : ObjectId("5b618a570550de0021aaa2ef"),
"role" : "admin"
}
What I need is retrieve the records from Collection A, which records have "level" field's value as "super" and its related Collection B record's "role" value as "admin".
For this, I am trying to use aggregation and java mongoTemplate.
Following is the code that I tried but it returns 0 records.
final TypedAggregation<A> typedAggregation = Aggregation.newAggregation(A.class,
Aggregation.match(Criteria.where("level").equals(level)),
Aggregation.lookup("B", "_id", "dataReference.B.$id", "Basb"),
Aggregation.match(new Criteria().andOperator(
Criteria.where("B.role").regex("admin")
)));
final AggregationResults<Map> A = mongoTemplate.aggregate(typedAggregation, "A", Map.class);
Please note that I am new to Mongo aggregation.
It's quiet ugly solution:
MongoTemplate
You cannot use TypedAggregation because we need to transform A collection to be able join with B collection
Aggregation typedAggregation = Aggregation.newAggregation(
Aggregation.match(Criteria.where("level").is("super")),
new AggregationOperation() {
#Override
public Document toDocument(AggregationOperationContext context) {
return Document.parse("{\"$addFields\":{\"dataReference\":{\"$reduce\":{\"input\":{\"$objectToArray\":\"$dataReference\"},\"initialValue\":null,\"in\":{\"$cond\":[{\"$eq\":[{\"$type\":\"$$this.v\"},\"objectId\"]},\"$$this.v\",\"$$value\"]}}}}}");
}
},
Aggregation.lookup("B", "dataReference", "_id", "B"),
Aggregation.match(new Criteria().andOperator(
Criteria.where("B.role").regex("admin")
)
)
);
final AggregationResults<Document> A = mongoTemplate.aggregate(typedAggregation, "A", Document.class);
MongoDB Aggregation
db.A.aggregate([
{
"$match": {
"level": "super"
}
},
{
"$addFields": {
"B": {
"$reduce": {
"input": {
"$objectToArray": "$dataReference"
},
"initialValue": null,
"in": {
"$cond": [
{
"$eq": [
{
"$type": "$$this.v"
},
"objectId"
]
},
"$$this.v",
"$$value"
]
}
}
}
}
},
{
"$lookup": {
"from": "B",
"localField": "B",
"foreignField": "_id",
"as": "B"
}
},
{
"$match": {
"$and": [
{
"B.role": {
"$regex": "admin",
"$options": ""
}
}
]
}
}
])
MongoPlayground
I am new with Spring Data MongoDB and I am trying to implement an aggregation query in Java with Spring Data MongoDB. I have tried searching from this problem and approached it using MongoTemplate, but still to no result.
The format of my data:
[{
"_id" : ObjectId("5e1aea6c275360baf96bac29"),
"title" : "postim",
"upvotesBy" : [
"5e18b4c12753608718dfa007",
"5e19ac0f5161a4994ded1f35"
],
"file" : "test",
"description" : "description",
"postedBy" : "5e18b4c12753608718dfa007",
"createdAt" : ISODate("2020-01-12T09:44:12.119+0000"),
"_class" : "com.socialnetwork.post.Post"
},
{
"_id" : ObjectId("5e1aeaf8275360bb4bb47325"),
"title" : "postim2",
"upvotesBy" : [
"5e18b4c12753608718dfa007",
"5e19ac0f5161a4994ded1f35"
],
"file" : "test2",
"description" : "description2",
"postedBy" : "5e18b4c12753608718dfa007",
"createdAt" : ISODate("2020-01-12T09:46:32.909+0000"),
"_class" : "com.socialnetwork.post.Post"
}]
My query:
db.post.aggregate([
{
$match: {}
},
{
$lookup: {
from: "users",
localField: "postedBy",
foreignField: "_id",
as: "user"
}
},
{
$group: {
_id: {
username: "$user.name",
title: "$title",
description: "$description",
upvotes: { $size: "$upvotesBy" },
upvotesBy: "$upvotesBy",
isUpvoted: { $in: [req.query.userId, "$upvotesBy"] },
isPinned: {
$cond: {
if: { $gte: [{ $size: "$upvotesBy" }, 3] },
then: true,
else: false
}
},
file: "$file",
createdAt: {
$dateToString: {
format: "%H:%M %d-%m-%Y",
timezone: "+01",
date: "$createdAt"
}
},
id: "$_id"
}
}
},
{ $sort: { "_id.isPinned": -1, "_id.createdAt": -1 } }
])
This is the query I use in my Javascript backend and I can do this fairly easy with Mongoose. However I am having some difficulty with the Java implementation of it.
private LookupOperation getLookupOperation() {
return LookupOperation.newLookup().from("user")
.localField("postedBy")
.foreignField("_id")
.as("user");
}
#Override
public List<PostSummary> aggregate() {
LookupOperation lookupOperation = getLookupOperation();
return mongoTemplate.aggregate(Aggregation.newAggregation(lookupOperation, Aggregation.group("id")
.addToSet("user.name").as("username")
.addToSet("title").as("title")
.addToSet("description").as("description")
.addToSet("id").as("id")
.push("upvotesBy").as("upvotesBy")
.addToSet("file").as("file")
.addToSet("createdAt").as("createdAt")
), Post.class, PostSummary.class).getMappedResults();
}
When I try to run this I get the following error:
"Cannot convert [] of type class java.util.ArrayList into an instance of class java.lang.Object! Implement a custom Converter<class java.util.ArrayList, class java.lang.Object> and register it with the CustomConversions. Parent object was: com.socialnetwork.post.PostSummary#7159d908"
When I delete the .addToSet("user.name").as("username") from the group aggregation I also get an error from .push("upvotesBy").as("upvotesBy") as it can not convert [] of type class java.util.ArrayList into an instance of class java.lang.String
Also the implementation of the Post Class and the PostSummary Class is simple:
Post.java:
#Document
public class Post {
#Id
private String id;
private String title;
private List<String> upvotesBy;
private String file;
private String description;
private String postedBy;
private Date createdAt = new Date();
// ... Getters and Setters for each field
}
PostSummary.java:
public class PostSummary {
private String username;
private String title;
private String description;
private List<String> upvotesBy;
private String file;
private String createdAt;
private String id;
//... Getters and Setters for the class
}
I also need to implement the isUpvoted and isPinned part of the query, but getting the idea on how to approach the first problem would be a great start.
EDIT: My desired output:
[
{
"username" : "user1",
"title" : "postim2",
"upvotesBy" : [
"5e18b4c12753608718dfa007",
"5e19ac0f5161a4994ded1f35"
],
"file": "file1",
id: "5e18b4c12753608718dber01"
... Other fields of the original post
},
{
"username" : "user2",
"title" : "postim2",
"upvotesBy" : [
"5e18b4c12753608718dfa007",
"5e19ac0f5161a4994ded1f35"
],
id: "5e18b4c12753608718dber02",
"file": "file2",
... Other fields of the original post
}
]
So from the lookup operation I need only to get the name of the user.
Let's do it
We need to update your aggregation to make it work.
Errors:
users's _id is ObjectId type, but in your post you have stored as String, so $lookup should be changed to Uncorrelated sub-queries
We replace $group by '$addFields' which fits better
We add as last stage $project operator to exclude all unsed fields.
db.post.aggregate([
{
$match: {}
},
{
$lookup: {
from: "users",
let: {
postedBy: "$postedBy"
},
pipeline: [
{
$match: {
$expr: {
$eq: [
{
"$toString": "$_id"
},
"$$postedBy"
]
}
}
}
],
as: "user"
}
},
{
$unwind: "$user"
},
{
$addFields: {
id: {
$toString: "$_id"
},
username: "$user.name",
upvotes: {
$size: "$upvotesBy"
},
isUpvoted: {
$in: [
"5e18b4c12753608718dfa007",
"$upvotesBy"
]
},
isPinned: {
$cond: [
{
$gte: [
{
$size: "$upvotesBy"
},
3
]
},
true,
false
]
},
createdAt: {
$dateToString: {
format: "%H:%M %d-%m-%Y",
timezone: "+01",
date: "$createdAt"
}
}
}
},
{
$sort: {
"isPinned": -1,
"createdAt": -1
}
},
{
$project: {
_id: 0,
user: 0,
upvotesBy: 0,
_class: 0
}
}
])
Now, we transform this query to Spring-Data syntax.
Java Implementation
package postman;
import static org.springframework.data.mongodb.core.aggregation.Aggregation.match;
import static org.springframework.data.mongodb.core.aggregation.Aggregation.project;
import static org.springframework.data.mongodb.core.aggregation.Aggregation.sort;
import static org.springframework.data.mongodb.core.aggregation.Aggregation.unwind;
import java.util.Arrays;
import java.util.List;
import org.bson.Document;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Sort.Direction;
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.data.mongodb.core.aggregation.Aggregation;
import org.springframework.data.mongodb.core.aggregation.AggregationOperation;
import org.springframework.data.mongodb.core.aggregation.AggregationOperationContext;
import org.springframework.data.mongodb.core.query.Criteria;
import org.springframework.stereotype.Service;
#Service
public class PostmanService {
#Autowired
private MongoTemplate mongoTemplate;
public List<PostSummary> find(String userId){
Aggregation aggregation = Aggregation.newAggregation(
match(new Criteria()),
//lookup("users", "postedBy", "_id", "user")
new AggregationOperation() {
#Override
public Document toDocument(AggregationOperationContext context) {
return new Document("$lookup",
new Document("from", "users")
.append("let", new Document("postedBy", "$postedBy"))
.append("pipeline", Arrays.asList(
new Document("$match",
new Document("$expr",
new Document("$eq", Arrays.asList(
new Document("$toString", "$_id"),
"$$postedBy"
))))))
.append("as", "user"));
}
},
unwind("$user"),
new AggregationOperation() {
#Override
public Document toDocument(AggregationOperationContext context) {
return new Document("$addFields",
new Document("id", new Document("$toString", "$_id"))
.append("username", "$user.name")
.append("upvotes", new Document("$size", "$upvotesBy"))
.append("isUpvoted", new Document("$in", Arrays.asList(userId, "$upvotesBy")))
.append("isPinned", new Document("$cond",
Arrays.asList(new Document("$gte",
Arrays.asList(new Document("$size", "$upvotesBy"), 3)), Boolean.TRUE, Boolean.FALSE)))
.append("createdAt", new Document("$dateToString",
new Document("format", "%H:%M %d-%m-%Y")
.append("timezone", "+01")
.append("date", "$createdAt")
)));
}
},
sort(Direction.DESC, "isPinned", "createdAt"),
project().andExclude("user", "_class")
);
System.out.println("Aggregation: " + aggregation.toString());
return mongoTemplate.aggregate(aggregation, mongoTemplate.getCollectionName(Post.class), PostSummary.class).getMappedResults();
}
}
Now, we call aggregation pipeline:
List<PostSummary> l = postmanService.find("5e18b4c12753608718dfa007");
for(PostSummary post: l) {
ObjectWriter ow = new ObjectMapper().writer().withDefaultPrettyPrinter();
System.out.println(ow.writeValueAsString(post));
}
2020-01-12 16:15:22.043 INFO 11148 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2020-01-12 16:15:22.047 INFO 11148 --- [ main] Postman.PostmanApplication : Started PostmanApplication in 4.602 seconds (JVM running for 5.301)
Aggregation: { "aggregate" : "__collection__", "pipeline" : [{ "$match" : {}}, { "$lookup" : { "from" : "users", "let" : { "postedBy" : "$postedBy"}, "pipeline" : [{ "$match" : { "$expr" : { "$eq" : [{ "$toString" : "$_id"}, "$$postedBy"]}}}], "as" : "user"}}, { "$unwind" : "$user"}, { "$addFields" : { "id" : { "$toString" : "$_id"}, "username" : "$user.name", "upvotes" : { "$size" : "$upvotesBy"}, "isUpvoted" : { "$in" : ["5e18b4c12753608718dfa007", "$upvotesBy"]}, "isPinned" : { "$cond" : [{ "$gte" : [{ "$size" : "$upvotesBy"}, 3]}, true, false]}, "createdAt" : { "$dateToString" : { "format" : "%H:%M %d-%m-%Y", "timezone" : "+01", "date" : "$createdAt"}}}}, { "$sort" : { "isPinned" : -1, "createdAt" : -1}}, { "$project" : { "user" : 0, "_class" : 0}}]}
2020-01-12 16:15:22.161 INFO 11148 --- [ main] org.mongodb.driver.connection : Opened connection [connectionId{localValue:2, serverValue:277}] to localhost:27017
{
"username" : "user1",
"title" : "postim2",
"description" : "description2",
"upvotesBy" : [ "5e18b4c12753608718dfa007", "5e19ac0f5161a4994ded1f35" ],
"file" : "test2",
"createdAt" : "10:46 12-01-2020",
"id" : "5e1aeaf8275360bb4bb47325"
}
{
"username" : "user1",
"title" : "postim",
"description" : "description",
"upvotesBy" : [ "5e18b4c12753608718dfa007", "5e19ac0f5161a4994ded1f35" ],
"file" : "test",
"createdAt" : "10:44 12-01-2020",
"id" : "5e1aea6c275360baf96bac29"
}
The following is a app_relation Collection :
{
"_id" : ObjectId("5bf518bb1e9f9d2f34a8299b"),
"app_id" : "123456789",
"dev_id" : "1",
"user_id" : "1",
"status" : "active",
"created" : NumberLong(1542789294)
}
The other collection is app :
{
"_id" : ObjectId("5bd02abb1e9f9d2adc211138"),
"app_id" : "123456789",
"custom_app_name" : "Demo",
"price" : 10,
"created" : NumberLong(1540369083)
}
Using Lookup in mongodb I want to embed App collection in AppRelation
For the same my mongodb query is:
db.app_relation.aggregate([
{
$lookup: {
"from": "app",
"localField": "app_id",
"foreignField": "app_id",
"as": "data"
}
},
{
$match: {
"data": {
"$size": 1
}
}
}
])
The equivalent code in Spring Java is :
LookupOperation lookupOperation = LookupOperation.newLookup().from("app").localField("app_id")
.foreignField("app_id").as("data");
AggregationOperation match = Aggregation.match(Criteria.where("data").size(1));
Aggregation aggregation = Aggregation.newAggregation(lookupOperation, match)
.withOptions(Aggregation.newAggregationOptions().cursor(new BasicDBObject()).build());
List<AppRelation> results = mongoTemplate.aggregate(aggregation, AppRelation.class, AppRelation.class)
.getMappedResults();
When executing the above code it provides the empty collection, whereas executing mongo db query it provides proper result.
The query generated in Debug logs is:
{
"aggregate": "app_relation",
"pipeline": [
{
"$lookup": {
"from": "app",
"localField": "app_id",
"foreignField": "app_id",
"as": "data"
}
},
{
"$match": {
"data": {
"$size": 1
}
}
}
],
"cursor": {}
}