We Keep Coding

Get Average from Mongo Collection using Aggrerate

I am trying to learn how to use the Mongo in Java and have been able to make some simple queries but I have been having trouble with the aggregate operator.
The document structure is a simple one, as the following:
{
"_id" : ObjectId("57dbe94f0507a4d8710ac5b2"),
"name" : "Name1",
"age" : 23
}
{
"_id" : ObjectId("57dbe9750507a4d8710ac5b3"),
"name" : "Name2",
"age" : "String for examble"
}
{
"_id" : ObjectId("57dbee630507a4d8710ac5b5"),
"name" : "Name3",
"age" : 24
}
All I want to do is get the average of the ages in the collection ( name example ).
Simply using mongo I can get the desirable result with the following consult:
db.example.aggregate([
{
$group: {
_id: null,
averageAge: { $avg: "$age" }
}
}
]);
I have tried the following:
BasicDBObject groupFields = new BasicDBObject("_id", "null");
BasicDBObject media = new BasicDBObject("$avg", "$age");
groupFields.put("mediaIdade", media);
BasicDBObject group = new BasicDBObject("$group", groupFields );
AggregateIterable<org.bson.Document> agregate = db.getCollection("exemplo").aggregate(Arrays.asList (group));
Which is almost a direct translation but got a "java.lang.NoSuchMethodError: org.bson.codecs.BsonTypeClassMap.keys()Ljava/util/Set;" , unsurprisingly.
But I cannot translate that to Java. I have checked and found this question but could not understand it due to the use of opperators such as $unwind. So I'm trying to make query as simple as possible to better understand how the Java framework for aggregation works.
Can someone help?

Try something like this.
MongoCollection<Document> dbCollection = db.getCollection("example", Document.class);
AggregateIterable<org.bson.Document> aggregate = dbCollection.aggregate(Arrays.asList(Aggregates.group("_id", new BsonField("averageAge", new BsonDocument("$avg", new BsonString("$age"))))));
Document result = aggregate.first();
double age = result.getDouble("averageAge");
Input:
{ "_id" : ObjectId("58136d6ed96cc299c224d529"), "name" : "Name1", "age" : 23 }
{ "_id" : ObjectId("58136d6ed96cc299c224d52a"), "name" : "Name2", "age" : 26 }
{ "_id" : ObjectId("58136d6ed96cc299c224d52b"), "name" : "Name3", "age" : 24 }
Output:
24.333333333333332

I noticed a small typo in your code:
db.getCollection("exemplo").aggregate(Arrays.asList (group));
Should be example instead of examplo
Also the easiest way to translate working mongo expression is using Document.parse method.
In your case it could be:
db.getCollection("exemplo").aggregate(Arrays.asList(
Document.parse("{ $group: { _id: null, averageAge: { $avg: '$age' } } }")
));
Your impl also almost correct, two minor issues:
replace "null" string with just null value
Use "averageAge" instead of "mediaIdade"
BasicDBObject groupFields = new BasicDBObject("_id", null);
BasicDBObject media = new BasicDBObject("$avg", "$age");
groupFields.put("averageAge", media);
BasicDBObject group = new BasicDBObject("$group", groupFields );
AggregateIterable<org.bson.Document> agregate = db.getCollection("exemple").aggregate(Arrays.asList (group));
To get result:
agregate.first().getDouble("averageAge");

Try this:
DBObject groupFields = new BasicDBObject( "_id", 0);
groupFields.put("average", new BasicDBObject( "$avg", "$age"));
DBObject group = new BasicDBObject("$group", groupFields);
AggregationOutput output = db.getCollection("exemplo").aggregate(group);
Iterable<DBObject> list = output.results();
If required add a filter to the query you can add a match parameter:
DBObject match = new BasicDBObject();
match.put("age", new BasicDBObject("$gte", 25));
DBObject groupFields = new BasicDBObject( "_id", 0);
groupFields.put("average", new BasicDBObject( "$avg", "$age"));
DBObject group = new BasicDBObject("$group", groupFields);
AggregationOutput output = db.getCollection("exemplo").aggregate(match, group);
Iterable<DBObject> list = output.results();

How do I group the results of mongodb query and implement in Java?

I have a collection in a Mongo database that looks like this:
{
"_id" : ObjectId("561b42d4e4b0d4227d011d2c"),
"product_related_data" : {
"depth" : 6,
"height" : 23,
"product_barcode" : "54491472",
"product_name" : "xyz product",
"product_uuid" : "009b9846-b3ad-49f7-a7a0-d35a04f83480",
"width" : 6
},
"sensostrip_data" : {
"barcode" : "130150208299",
"battery_level" : 2.894
},
"stock_related_data" : {
"calculated_max_product_percentage" : 15.625,
"calculated_min_product_percentage" : 12.5,
"current_stock" : 2,
"max_stock" : 6,
"stock_difference_percentage" : 0,
"stock_difference_value" : 0,
"stock_percentage" : 37.5
},
"store_data" : {
"city_uuid" : "cbb4dfe8-172b-11e4-a1f0-00163ed23ec2",
"ip" : "10.0.1.1",
"retailer_uuid" : "8c33c32c-5903-11e4-a1f0-00163ed23ec2",
"store_name" : "xyz store",
"store_uuid" : "15a6cc90-081f-11e5-b213-001e6745ff8d"
},
"time" : {
"check_date" : "2015-10-11 11:53:55",
"previous_check_date" : "2015-10-11 11:48:57"
},
"id" : "6be54bef-0aa3-456c-b912-1731f8154e7d"
}
The mongo query I'm currently executing returns all the documents for a list of store_uuid's and for a product_uuid, and looks like this:
db.readings
.find({ $and:[ {"store_data.store_uuid": {$in:["15a6cc90-081f-11e5-b213-001e6745ff8d","217b983b-5904-11e4-a1f0-00163ed23ec2","5337d78d-5904-11e4-a1f0-00163ed23ec2"]}}
,{"product_related_data.product_uuid": "f44aa29d-09ce-4902-bf12-d45d44b3dfd0"}]})
My current Java implementation (where I make use of projection) looks like this:
DBCollection table = databaseConncetion().getCollection("readings");
BasicDBObject sensorReturn = new BasicDBObject("sensostrip_data.barcode",1);
BasicDBObject clause1 = new BasicDBObject("store_data.store_uuid", new BasicDBObject("$in", StoreIds));
BasicDBObject clause2 = new BasicDBObject("product_related_data.product_uuid", productId);
BasicDBList and = new BasicDBList();
and.add(clause1);
and.add(clause2);
DBObject query = new BasicDBObject("$and", and);
DBCursor cursor = table.find(query, sensorReturn
.append("stock_related_data.stock_percentage",1)
.append("store_data.store_uuid",1)
.append("time.check_date", 1))
.sort(new BasicDBObject("time.check_date", -1))
.limit(100);
However I need this query to group the results for the latest check_date by barcode

The aggregation framework is at your disposal. Running the following aggregation pipeline will give you the desired result:
Mongo shell:
pipeline = [
{
"$match": {
"store_data.store_uuid": {
"$in": [
"15a6cc90-081f-11e5-b213-001e6745ff8d",
"217b983b-5904-11e4-a1f0-00163ed23ec2",
"5337d78d-5904-11e4-a1f0-00163ed23ec2"
]
},
"product_related_data.product_uuid": "f44aa29d-09ce-4902-bf12-d45d44b3dfd0"
}
},
{ "$sort": { "time.check_date": -1 } },
{
"$group": {
"_id": "$sensostrip_data.barcode",
"stock_percentage": { "$first": "$stock_related_data.stock_percentage" },
"store_uuid": { "$first": "$store_data.store_uuid" },
"check_date": { "$first": "$time.check_date" }
}
},
{ "$limit": 100 }
];
db.readings.aggregate(pipeline);
Java test implementation
public class JavaAggregation {
public static void main(String args[]) throws UnknownHostException {
MongoClient mongo = new MongoClient();
DB db = mongo.getDB("test");
DBCollection coll = db.getCollection("readings");
// create the pipeline operations, first with the $match
DBObject match = new BasicDBObject("$match",
new BasicDBObject("store_data.store_uuid", new BasicDBObject("$in", StoreIds))
.append("product_related_data.product_uuid", productId)
);
// sort pipeline
DBObject sort = new BasicDBObject("$sort",
new BasicDBObject("time.check_date", -1)
);
// build the $group operations
DBObject groupFields = new BasicDBObject( "_id", "$sensostrip_data.barcode"); // group by barcode
groupFields.put("stock_percentage", new BasicDBObject( "$first", "$stock_related_data.stock_percentage")); // get the first when ordered documents are grouped
groupFields.put("store_uuid", new BasicDBObject( "$first", "$store_data.store_uuid"));
groupFields.put("check_date", new BasicDBObject( "$first", "$time.check_date"));
// append any other necessary fields
DBObject group = new BasicDBObject("$group", groupFields);
// limit step
DBObject limit = new BasicDBObject("$limit", 100);
// put all together
List<DBObject> pipeline = Arrays.asList(match, sort, group, limit);
AggregationOutput output = coll.aggregate(pipeline);
for (DBObject result : output.results()) {
System.out.println(result);
}
}
}

Aggregation Query adjusting dates for timezone

I want to run this mongodb query using java code.
Query :
db.log.aggregate([
{ $match : {
"ts" : {
$gte: ISODate("2015-07-31T18:30:00.000Z"),
$lt: ISODate("2015-08-01T18:30:00.000Z")
},
"dup": {$exists:false}
}},
{ $project : {
'lts': {
'$add': ['$ts',5.5*3600*1000]
}
}},
{ $group : {
_id : {
day: { $dayOfMonth: "$lts" },
month: { $month: "$lts" },
year: { $year: "$lts" }
},
count: { $sum: 1 }
}}
])
i have tried with this code but its not worked
DBObject query = QueryBuilder.start().put("ts").greaterThanEquals(startdate).lessThanEquals(enddate).and("dup").exists(false).get();
DBObject match = new BasicDBObject("$match", query);
DBObject project=new BasicDBObject("$project", new BasicDBObject("ts",new BasicDBObject("$add",5.5*3600*1000)));
DBObject group = new BasicDBObject("$group", new BasicDBObject("_id",new BasicDBObject("day", new BasicDBObject("$dayOfMonth", "$ts"))
).append("count", new Document("$sum", 1)));
AggregationOutput output = collection.aggregate(match,group);
for (DBObject result : output.results()) {
System.out.println(result);
}

Your inital query could have been better written. Presumably you want to group results by "day" whilst also ajusting from UTC to a local timzezone. It usually is better to just stick with the date math rather than mix in the date aggregation operators in such a case, and also this means that the $project here is not required, and you can just go straight to $group:
db.log.aggregate([
{ "$match": {
"ts": {
"$gte": ISODate("2015-07-31T18:30:00.000Z"),
"$lt": ISODate("2015-08-01T18:30:00.000Z")
},
"dup": { "$exists": false }
}},
{ "$group": {
"_id": {
"$add": [
{ "$subtract": [
{ "$subtract": [
{ "$add": [ "$ts", 5.5*1000*60*60 ] },
new Date(0)
]},
{ "$mod": [
{ "$subtract": [
{ "$add": [ "$ts", 5.5*1000*60*60 ] },
new Date(0)
]},
1000*60*60*24
]}
]},
new Date(0)
]
},
"count": { "$sum": 1 }
}}
])
Note the additional Date(0) statements in there. This is "epoch" and just like when you $add a numeric value to a date you get a Date type in return, when you $subtract you get the integer value. So correcting by adding back "epoch" makes everything a Date type again.
Hence the "date math" is employed everywhere, to "round" to a "day":
( 1000 millseconds * 60 seconds * 60 minutes * 24 hours )
And the modulo $mod operator works out the remainder of this which is subtracted from the current date value, to come to the "rounded" date.
All done in $group and efficiently as well:
Translating to Java is just following the same indentation:
Date startdate = new DateTime(2015, 7, 31, 18, 30, 0, DateTimeZone.UTC).toDate();
Date enddate = new DateTime(2015, 8, 1, 18, 30, 0, DateTimeZone.UTC).toDate();
DBObject query = QueryBuilder.start()
.put("ts").greaterThanEquals(startdate)
.lessThan(enddate)
.and("dup").exists(false).get();
DBObject match = new BasicDBObject("$match",query);
DBObject group = new BasicDBObject("$group",
new BasicDBObject("_id",
new BasicDBObject(
"$add", Arrays.asList(
new BasicDBObject(
"$subtract", Arrays.asList(
new BasicDBObject(
"$subtract", Arrays.asList(
new BasicDBObject(
"$add",Arrays.asList( "$ts", 5.5*1000*60*60 )
),
new Date(0)
)
),
new BasicDBObject(
"$mod", Arrays.asList(
new BasicDBObject(
"$subtract", Arrays.asList(
new BasicDBObject(
"$add",Arrays.asList( "$ts", 5.5*1000*60*60 )
),
new Date(0)
)
),
1000*60*60*24
)
)
)
),
new Date(0)
)
)
)
.append("count", new BasicDBObject("$sum",1))
);
AggregationOutput output = collection.aggregate(Arrays.asList(match, group));
for ( DBObject result : output.results() ) {
System.out.println(result);
}
Also note that your "timezeone adjustment" here is presuming 5.5 hours "behind" UTC, which I hope is correct. If you mean "after" or positive, then that operation is a "subtraction" and not an addition to correct the UTC hours to days.
So all is good, and your grouping keys are real Date objects as well, rather than the composite values returned by date aggregation operators.

change field ts to east 8 timezone:
BasicDBObject addEightHour = new BasicDBObject("$add", Arrays.asList( "$ts", 8*1000*60*60 ));
final BasicDBObject day = new BasicDBObject("$dateToString", new BasicDBObject("format", "\"%Y%m%d\"").append("date", addEightHour));
DBObject groupFields = new BasicDBObject("_id", new BasicDBObject(
"value", day));
DBObject group = new BasicDBObject("$group", groupFields);

How to calculate unique elements in collections?

I have collection like
id : 1, url : youtube.com
also one url can be many times,
I need get whole collections and count unique elements
like
youtube 10
google 8
lycos 5
here is the code
public List<URLEntity> findAll() {
List<URLEntity> list = new ArrayList<URLEntity>();
String sort = "searchDate";
String order = "desc";
DBObject sortCriteria = new BasicDBObject(sort, "desc".equals(order) ? -1 : 1);
BasicDBObject query = new BasicDBObject();
DBCursor cursor = mongoCoreService.getDomainCollection().find(query).sort(sortCriteria);
try {
while (cursor.hasNext()) {
DBObject document = cursor.next();
URLEntity entity = new URLEntity();
entity = Converter.toObject(URLEntity.class, document);
list.add(entity);
}
} finally {
cursor.close();
}
return list;
}
thanks

You can achieve this by using the aggregate framework from MongoDB.
db.yourcollection.aggregate({ $group: { _id: '$url', total: {$sum: 1} }})
You'll obtain something like this, that you can manipulate in JAVA:
{
"result" : [
{
"_id" : "youtube.com",
"total" : 10
},
{
"_id" : "google.com",
"total" : 8
},
{
"_id" : "lycos",
"total" : 5
}
],
"ok" : 1
}

Today I found and test answer in java, here is a code
DBCollection mycoll= db.getCollection("domain");
DBObject fields = new BasicDBObject("domain", 1);
DBObject project = new BasicDBObject("$project", fields );
// Now the $group operation
DBObject groupFields = new BasicDBObject( "_id", "$domain");
groupFields.put("total", new BasicDBObject( "$sum", 1));
DBObject group = new BasicDBObject("$group", groupFields);
// run aggregation
AggregationOutput output = mycoll.aggregate(project, group);
System.out.println(output.getCommand().toString());
for (DBObject dbObject : output.results()) {
System.out.println(dbObject);
}

How to implement a group function in mongoDB using JAVA

The dataset can be found in http://docs.mongodb.org/manual/tutorial/aggregation-zip-code-data-set/
Now , I am trying to implement the following function "Return Average City population by State":
db.zips.aggregate( { $group :
{ _id : { state : "$state", city : "$city" },
pop : { $sum : "$pop" } } },
{ $group :
{ _id : "$_id.state",
avgCityPop : { $avg : "$pop" } } } )
with an output of:
{
"result" : [
{
"_id" : "RI",
"avgCityPop" : 18933.28301886793
}
Please help me in JAVA, Thank you

Try this:
// First $group operation
DBObject compoundId = new BasicDBObject( "state", "$state");
compoundId.append("city", "$city")
DBObject groupFields1 = new BasicDBObject( "_id", compoundId );
groupFields1.put("pop", new BasicDBObject( "$sum", "$pop"));
DBObject group1 = new BasicDBObject("$group", groupFields1);
// Second $group operation
DBObject groupFields2 = new BasicDBObject( "_id", "$_id.state");
groupFields2.put("avgCityPop", new BasicDBObject( "$avg", "$pop"));
DBObject group2 = new BasicDBObject("$group", groupFields2);
// Run aggregation
AggregationOutput output = collection.aggregate( group1, group2 );