I use the Java Spring MongoDB repository in my project.
I have this collection in MongoDB called Info:
{ "_id" : 1, "hosting" : "hostgator.com", count:7 }
{ "_id" : 2, "hosting" : "aws.amazon.com", count:7}
{ "_id" : 3, "hosting" : "aws.amazon.com", count:3}
{ "_id" : 4, "hosting" : "hostgator.com", count:5 }
{ "_id" : 5, "hosting" : "aws.amazon.com", count:1 }
{ "_id" : 6, "hosting" : "cloud.google.com", count:1 }
{ "_id" : 7, "hosting" : "aws.amazon.com", count:5 }
{ "_id" : 8, "hosting" : "hostgator.com", count:2 }
{ "_id" : 9, "hosting" : "cloud.google.com", count:3 }
{ "_id" : 10,"hosting" : "godaddy.com", count:7 }
...
{ "_id" : 100, "hosting" : "godaddy.com", count:5 }
Here is DTO definition:
public class Info{
public int _id;
public String hosting;
public int count;
}
I need to write a query and to get from the database all values of count property and remove the duplications. For example, the result that I expect according to the collection above is:
List<int> counts = [1,2,3,5,7];
For this purpose I use the aggregation group method and MongoTemplate:
GroupOperation groupOperation = Aggregation.group("count");
Aggregation aggregation = Aggregation.newAggregation(groupOperation);
var result = template.aggregate(aggregation, Info.class, Info[].class);
System.out.println(result.getMappedResults());
But the result that I get is an empty array - [].
Why I don't get the expected result?
Related
I have the following mongodb document:
{
"_id" : ObjectId("5f283e7d39187d9ab77e7ece"),
"resourceType" : "VM",
"resourceInstanceName" : "virtual_machine_1",
"properties" :
{ "name" : "CentOS-VM", "cpu" : 2, "memory_in_gb" : 2, }
},
{
"_id" : ObjectId("5f28jh58hjf9ab77e7ece"),
"resourceType" : "VM",
"resourceInstanceName" : "virtual_machine_2",
"properties" :
{ "name" : "CentOS-VM", "cpu" : 8, "memory_in_gb" : 8, }
}
I use the following query in mongo shell which works fine
db.collection.aggregate({
$match:
{ "resourceType":"VM" }
}, {
$group: {
_id: '',
instance:
{ $sum: 1 }
,
cpu:
{ $sum: '$properties.cpu' }
,
memory_in_gb:
{ $sum: '$properties.memory_in_gb'}
}
})
and the output was
{ "_id" : "", "instance" : 2.0, "cpu" : 10, "memory_in_gb" : 10 }
using spring-data I have written the following code to produce the same result but it ends up in an Exception:
MatchOperation matchOperation = Aggregation.match(Criteria.where(RESOURCE_TYPE).is("VM"));
UnwindOperation unwindOperation = Aggregation.unwind("properties");
GroupOperation groupOperation = Aggregation.group(ID);
GroupOperation instanceOperation = Aggregation.group().count().as("instance");
GroupOperation cpuOperation = Aggregation.group("properties").sum("properties.cpu").as("cpu");
GroupOperation memoryOperation = Aggregation.group("properties").sum("properties.memory_in_gb").as("memory_in_gb");
Aggregation aggregation = Aggregation.newAggregation(matchOperation,unwindOperation,groupOperation,
instanceOperation, cpuOperation, memoryOperation);
return mongoTemplate.aggregate(aggregation, COLLECTION_NAME, Map.class).getMappedResults();
Here is the stackTrace:
[http-nio-9083-exec-2] ERROR c.s.c.c.v.service.ComputeService.getComputeSummary - Error in Cloud Account-summary :java.lang.IllegalArgumentException: Invalid reference 'properties'!
at org.springframework.data.mongodb.core.aggregation.ExposedFieldsAggregationOperationContext.getReference(ExposedFieldsAggregationOperationContext.java:114)
at org.springframework.data.mongodb.core.aggregation.ExposedFieldsAggregationOperationContext.getReference(ExposedFieldsAggregationOperationContext.java:77)
at org.springframework.data.mongodb.core.aggregation.AbstractAggregationExpression.unpack(AbstractAggregationExpression.java:74)
I have the following documents in one collection named as mail_test. Some of them have a tags field which is an array:
/* 1 */
{
"_id" : ObjectId("601a7c3a57c6eb4c1efb84ff"),
"email" : "aaaa#bbb.com",
"content" : "11111"
}
/* 2 */
{
"_id" : ObjectId("601a7c5057c6eb4c1efb8590"),
"email" : "aaaa#bbb.com",
"content" : "22222"
}
/* 3 */
{
"_id" : ObjectId("601a7c6d57c6eb4c1efb8675"),
"email" : "aaaa#bbb.com",
"content" : "33333",
"tags" : [
"x"
]
}
/* 4 */
{
"_id" : ObjectId("601a7c8157c6eb4c1efb86f4"),
"email" : "aaaa#bbb.com",
"content" : "4444",
"tags" : [
"yyy",
"zzz"
]
}
There are two documents with non-empty-tags, so I want the result to be 2.
I use the the following statement to aggregate and get the correct tag_count:
db.getCollection('mail_test').aggregate([{$group:{
"_id":null,
"all_count":{$sum:1},
"tag_count":{"$sum":{$cond: [ { $ne: ["$tags", undefined] }, 1, 0]}}
//if replace `undefined` with `null`, I got the tag_count as 4, that is not what I want
//I also have tried `$exists`, but it cannot be used here.
}}])
and the result is:
{
"_id" : null,
"all_count" : 4.0,
"tag_count" : 2.0
}
and I use spring data mongo in java to do this:
private void test(){
Aggregation agg = Aggregation.newAggregation(
Aggregation.match(new Criteria()),//some condition here
Aggregation.group(Fields.fields()).sum(ConditionalOperators.when(Criteria.where("tags").ne(null)).then(1).otherwise(0)).as("tag_count")
//I need an `undefined` instead of `null`,or is there are any other solution?
);
AggregationResults<MailTestGroupResult> results = mongoTemplate.aggregate(agg, MailTest.class, MailTestGroupResult.class);
List<MailTestGroupResult> mappedResults = results.getMappedResults();
int tag_count = mappedResults.get(0).getTag_count();
System.out.println(tag_count);//get 4,wrong
}
I need an undefined instead of null but I don't know how to do this,or is there are any other solution?
You can use Aggregation operators to check if the field tags exists or not with one of the following constructs in the $group stage of your query (to calculate the tag_count value):
"tag_count":{ "$sum": { $cond: [ { $gt: [ { $size: { $ifNull: ["$tags", [] ] }}, 0 ] }, 1, 0] }}
// - OR -
"tag_count":{ "$sum": { $cond: [ $eq: [ { $type: "$tags" }, "array" ] }, 1, 0] }
Both, return the same result (as you had posted).
I have a question about how to retrieve data from a mongodb and how to store it in my Java code. I select all the documents (there are only 5) from my mongodb and then store its content in my Java application.
So my code is as follows:
public class MongoReader {
private Invoice invoice;
public void mongoReader () {
MongoClientURI uri = new MongoClientURI("my-url");
try (MongoClient mongoClient = new MongoClient(uri)) {
MongoDatabase database = mongoClient.getDatabase("BiFiBEP02");
MongoCollection<Document> mongoCollection = database.getCollection("bifi");
FindIterable<Document> documents = mongoCollection.find();
for (Document document : documents){
invoice.setCustomerId(document.getInteger("customerId"));
invoice.setDate(document.getDate("date"));
invoice.setInvoiceId(document.getInteger("invoiceId"));
invoice.setInvoiceLines(document.getList("invoiceLines", ArrayList<InvoiceLine>));
invoice.setNote(document.getString("note"));
invoice.setPersonId(document.getInteger("personId"));
}
}
catch (MongoException mongoException) {
mongoException.printStackTrace();
}
}
}
The document.getList() doesn't seem to be satisfied with my input.
An example of the the mongo document:
"customerId" : 2,
"date" : ISODate("2018-05-16T10:23:40.049Z"),
"invoiceId" : 1,
"invoiceLines" : [
{
"btwCode" : "hoog",
"productId" : 1,
"productName" : "BiFi worstjes voordeelstrip",
"quantity" : 20,
"totalPrice" : 30,
"unit" : "kg"
},
{
"btwCode" : "hoog",
"productId" : 2,
"productName" : "BiFi worstjes kip",
"quantity" : 20,
"totalPrice" : 30,
"unit" : "kg"
},
{
"btwCode" : "laag",
"productId" : 3,
"productName" : "BiFi worstjes extra scherp",
"quantity" : 30,
"totalPrice" : 100.22,
"unit" : "kg"
},
{
"btwCode" : "geen",
"productId" : 1,
"productName" : "BiFi worstjes promotiestand",
"quantity" : -1,
"totalPrice" : 30.32,
"unit" : "kg"
}
],
"note" : "This invoice is very important!",
"personId" : 2
So my question here is: I have to put the invoiceLines in arraylist in my Java object, but I cannot get the array items out of the mongoDb. How do I do this?
I'm new to working with MongoDb and do not know a lot of things.
I need to write an aggregation request.
Here is the JSON document structure.
{
"_id" : ObjectId("5a72f7a75ef7d430e8c462d2"),
"crawler_id" : ObjectId("5a71cbb746e0fb0007adc6c2"),
"skill" : "stack",
"created_date" : ISODate("2018-02-01T13:19:03.522+0000"),
"modified_date" : ISODate("2018-02-01T13:22:23.078+0000"),
"connects" : [
{
"subskill" : "we’re",
"weight" : NumberInt(1),
"parser_id" : [
ObjectId("5a71d88d5ef7d41964fbec11")
]
},
{
"subskill" : "b1",
"weight" : NumberInt(2),
"parser_id" : [
ObjectId("5a71d88d5ef7d41964fbec11"),
ObjectId("5a71d88d5ef7d41964fbec1b")
]
},
{
"subskill" : "making",
"weight" : NumberInt(2),
"parser_id" : [
ObjectId("5a71d88d5ef7d41964fbec1b"),
ObjectId("5a71d88d5ef7d41964fbec1c")
]
},
{
"subskill" : "delivery",
"weight" : NumberInt(2),
"parser_id" : [
ObjectId("5a71d88d5ef7d41964fbec1c"),
ObjectId("5a71d88d5ef7d41964fbec1e")
]
}
]
}
I need the result return the name of skill and the number of unique parser_id.
In this case, the result should be:
[
{
"skill": "stack",
"quantity": 4
}
]
where "stack" - skill name,
and "quantity" - count of unique parser_id.
ObjectId("5a71d88d5ef7d41964fbec11")
ObjectId("5a71d88d5ef7d41964fbec1b")
ObjectId("5a71d88d5ef7d41964fbec1c")
ObjectId("5a71d88d5ef7d41964fbec1e")
Can some one help me with this request ???
Given the document supplied in your question, this command ...
db.collection.aggregate([
{ $unwind: "$connects" },
// count all occurrences
{ "$group": { "_id": {skill: "$skill", parser_id: "$connects.parser_id"}, "count": { "$sum": 1 } }},
// sum all occurrences and count distinct
{ "$group": { "_id": "$_id.skill", "quantity": { "$sum": 1 } }},
// (optional) rename the '_id' attribute to 'skill'
{ $project: { 'skill': '$_id', 'quantity': 1, _id: 0 } }
])
... will return:
{
"quantity" : 4,
"skill" : "stack"
}
The above command groups by skill and connects.parser_id and then gets a distinct count of those groups.
Your command includes the java tag so I suspect you are looking to execute the same command using the MongoDB Java driver. The code below (using MongoDB Java driver v3.x) will return the same result:
MongoClient mongoClient = ...;
MongoCollection<Document> collection = mongoClient.getDatabase("...").getCollection("...");
List<Document> documents = collection.aggregate(Arrays.asList(
Aggregates.unwind("$connects"),
new Document("$group", new Document("_id", new Document("skill", "$skill").append("parser_id", "$connects.parser_id"))
.append("count", new Document("$sum", 1))),
new Document("$group", new Document("_id", "$_id.skill").append("quantity", new Document("$sum", 1))),
new Document("$project", new Document("skill", "$_id").append("quantity", 1).append("_id", 0))
)).into(new ArrayList<>());
for (Document document : documents) {
logger.info("{}", document.toJson());
}
Note: this code deliberately uses the form new Document(<pipeline aggregator>, ...) instead of the Aggregators utilities to make it easier to see the translation between the shell command and its Java equivalent.
try $project with $reduce
$setUnion is used to keep only the distinct ids and finally $size used to get the distinct array count
db.col.aggregate(
[
{$project : {
_id : 0,
skill : 1,
quantity : {$size :{$reduce : {input : "$connects.parser_id", initialValue : [] , in : {$setUnion : ["$$value", "$$this"]}}}}
}
}
]
).pretty()
result
{ "skill" : "stack", "quantity" : 4 }
I'm looking for a way to implement graphlookup using Java in a Resful Web API. I'm trying to implement the hierarchy like on the MongoDB (https://docs.mongodb.com/v3.4/reference/operator/aggregation/graphLookup/)
{ "_id" : 1, "name" : "Dev" }
{ "_id" : 2, "name" : "Eliot", "reportsTo" : "Dev" }
{ "_id" : 3, "name" : "Ron", "reportsTo" : "Eliot" }
{ "_id" : 4, "name" : "Andrew", "reportsTo" : "Eliot" }
{ "_id" : 5, "name" : "Asya", "reportsTo" : "Ron" }
{ "_id" : 6, "name" : "Dan", "reportsTo" : "Andrew" }
This is the employee collection, what I want is to be able to create this structure stored in MongoDB
{
"_id" : 1,
"name" : "Dev",
"reportingHierarchy" : [ ]
}
{
"_id" : 2,
"name" : "Eliot",
"reportsTo" : "Dev",
"reportingHierarchy" : [
{ "_id" : 1, "name" : "Dev" }
]
}
{
"_id" : 3,
"name" : "Ron",
"reportsTo" : "Eliot",
"reportingHierarchy" : [
{ "_id" : 1, "name" : "Dev" },
{ "_id" : 2, "name" : "Eliot", "reportsTo" : "Dev" }
]
}
I've seen examples like this for aggregation, but nothing on graphlookup
Aggregation agg = newAggregation(
match(Criteria.where("pageId").is("2210")),
unwind("postIds"),
group("_id").sum("1").as("sum")
//project("$sum").and("pageId").previousOperation()
);
Is there a way to get graphlookup into a format like this? Where instead of using match, unwind, group, I can use GraphLookupOperation and then something like get map result.
AggregationOperation aggregationOperation = new AggregationOperation() {
#Override public DBObject toDBObject(AggregationOperationContext aggregationOperationContext) {
DBObject graphLookup = new BasicDBObject(
"from", "individual").append(
"startWith", "$reportsTo").append(
"connectFromField", "reportsTo").append(
"connectToField", "firstName").append(
"maxDepth", 2).append(
"as", "reportingHierarchy");
return new BasicDBObject("$graphLookup", graphLookup);
This code allows for a workaround since i'm using an older version, I think 1.10.0 spring mongo. Now I have the issues of the "reportingHierarchy" not looking like I want it to. It's not giving me a name. Just reportsTo and reportingHierarchy that is also including _class which I don't want.