I want to run this mongodb query using java code.
Query :
db.log.aggregate([
{ $match : {
"ts" : {
$gte: ISODate("2015-07-31T18:30:00.000Z"),
$lt: ISODate("2015-08-01T18:30:00.000Z")
},
"dup": {$exists:false}
}},
{ $project : {
'lts': {
'$add': ['$ts',5.5*3600*1000]
}
}},
{ $group : {
_id : {
day: { $dayOfMonth: "$lts" },
month: { $month: "$lts" },
year: { $year: "$lts" }
},
count: { $sum: 1 }
}}
])
i have tried with this code but its not worked
DBObject query = QueryBuilder.start().put("ts").greaterThanEquals(startdate).lessThanEquals(enddate).and("dup").exists(false).get();
DBObject match = new BasicDBObject("$match", query);
DBObject project=new BasicDBObject("$project", new BasicDBObject("ts",new BasicDBObject("$add",5.5*3600*1000)));
DBObject group = new BasicDBObject("$group", new BasicDBObject("_id",new BasicDBObject("day", new BasicDBObject("$dayOfMonth", "$ts"))
).append("count", new Document("$sum", 1)));
AggregationOutput output = collection.aggregate(match,group);
for (DBObject result : output.results()) {
System.out.println(result);
}
Your inital query could have been better written. Presumably you want to group results by "day" whilst also ajusting from UTC to a local timzezone. It usually is better to just stick with the date math rather than mix in the date aggregation operators in such a case, and also this means that the $project here is not required, and you can just go straight to $group:
db.log.aggregate([
{ "$match": {
"ts": {
"$gte": ISODate("2015-07-31T18:30:00.000Z"),
"$lt": ISODate("2015-08-01T18:30:00.000Z")
},
"dup": { "$exists": false }
}},
{ "$group": {
"_id": {
"$add": [
{ "$subtract": [
{ "$subtract": [
{ "$add": [ "$ts", 5.5*1000*60*60 ] },
new Date(0)
]},
{ "$mod": [
{ "$subtract": [
{ "$add": [ "$ts", 5.5*1000*60*60 ] },
new Date(0)
]},
1000*60*60*24
]}
]},
new Date(0)
]
},
"count": { "$sum": 1 }
}}
])
Note the additional Date(0) statements in there. This is "epoch" and just like when you $add a numeric value to a date you get a Date type in return, when you $subtract you get the integer value. So correcting by adding back "epoch" makes everything a Date type again.
Hence the "date math" is employed everywhere, to "round" to a "day":
( 1000 millseconds * 60 seconds * 60 minutes * 24 hours )
And the modulo $mod operator works out the remainder of this which is subtracted from the current date value, to come to the "rounded" date.
All done in $group and efficiently as well:
Translating to Java is just following the same indentation:
Date startdate = new DateTime(2015, 7, 31, 18, 30, 0, DateTimeZone.UTC).toDate();
Date enddate = new DateTime(2015, 8, 1, 18, 30, 0, DateTimeZone.UTC).toDate();
DBObject query = QueryBuilder.start()
.put("ts").greaterThanEquals(startdate)
.lessThan(enddate)
.and("dup").exists(false).get();
DBObject match = new BasicDBObject("$match",query);
DBObject group = new BasicDBObject("$group",
new BasicDBObject("_id",
new BasicDBObject(
"$add", Arrays.asList(
new BasicDBObject(
"$subtract", Arrays.asList(
new BasicDBObject(
"$subtract", Arrays.asList(
new BasicDBObject(
"$add",Arrays.asList( "$ts", 5.5*1000*60*60 )
),
new Date(0)
)
),
new BasicDBObject(
"$mod", Arrays.asList(
new BasicDBObject(
"$subtract", Arrays.asList(
new BasicDBObject(
"$add",Arrays.asList( "$ts", 5.5*1000*60*60 )
),
new Date(0)
)
),
1000*60*60*24
)
)
)
),
new Date(0)
)
)
)
.append("count", new BasicDBObject("$sum",1))
);
AggregationOutput output = collection.aggregate(Arrays.asList(match, group));
for ( DBObject result : output.results() ) {
System.out.println(result);
}
Also note that your "timezeone adjustment" here is presuming 5.5 hours "behind" UTC, which I hope is correct. If you mean "after" or positive, then that operation is a "subtraction" and not an addition to correct the UTC hours to days.
So all is good, and your grouping keys are real Date objects as well, rather than the composite values returned by date aggregation operators.
change field ts to east 8 timezone:
BasicDBObject addEightHour = new BasicDBObject("$add", Arrays.asList( "$ts", 8*1000*60*60 ));
final BasicDBObject day = new BasicDBObject("$dateToString", new BasicDBObject("format", "\"%Y%m%d\"").append("date", addEightHour));
DBObject groupFields = new BasicDBObject("_id", new BasicDBObject(
"value", day));
DBObject group = new BasicDBObject("$group", groupFields);
Related
I am trying to put the following aggregation
db.getCollection("order").aggregate(
[
{
"$project" : {
"_id" : -1.0,
"customerId" : "$customer.customerId",
"hasOrderInT0" : {
"$cond" : [
{
"$and" : [
{
"$gte" : [
"$date",
1577829600.0
]
},
{
"$lte" : [
"$date",
1580507999.0
]
}
]
},
1,
0
]
}
}
}
]
);
in a Java app, where I am using Morphia as ORM. Basically if the date is between 2 timestamps, it will put 1 in the hasOrderInT0 field and 0 otherwise.
long initialStart = 1577829600;
long initialEnd = 1580507999;
AggregationPipeline pipeline = databaseService.getConnection().createAggregation(Order.class)
.project(
Projection.projection("_id", "-1"),
Projection.projection("customerId", "$customer.customerId"),
Projection.projection("hasOrderInT0",
Projection.expression(
"$cond",
Arrays.<Object>asList(
new BasicDBObject(
"$and", Arrays.<Object>asList(
new BasicDBObject(
"$gte", Arrays.<Object>asList("$date", initialStart)
),
new BasicDBObject(
"$lte", Arrays.<Object>asList("$date", initialEnd)
)
)
),
1,
0
)
)
)
);
When running the above code, I get the following error:
Caused by: java.lang.ClassCastException: java.util.Arrays$ArrayList cannot be cast to com.mongodb.DBObject
at xyz.morphia.aggregation.AggregationPipelineImpl.toExpressionArgs(AggregationPipelineImpl.java:296)
at xyz.morphia.aggregation.AggregationPipelineImpl.toDBObject(AggregationPipelineImpl.java:249)
at xyz.morphia.aggregation.AggregationPipelineImpl.toDBObject(AggregationPipelineImpl.java:240)
at xyz.morphia.aggregation.AggregationPipelineImpl.project(AggregationPipelineImpl.java:191)
This is my first time using Projection with Morphia and I don't know if this is the right way to implement the command that works in mongo console.
PS: The $project is just a pipeline from a bigger aggregate, but this is the part that is of interest and which is giving the error, so I simplified it for demonstration purpose.
It turns out there is no need to wrap the condition from $cond into in Arrays.<Object>asList. Projection.expression already accepts an arbitrary number of arguments, so the working code is:
long initialStart = 1577829600;
long initialEnd = 1580507999;
AggregationPipeline pipeline = databaseService.getConnection().createAggregation(Order.class)
.project(
Projection.projection("_id", "-1"),
Projection.projection("customerId", "$customer.customerId"),
Projection.projection("hasOrderInT0",
Projection.expression(
"$cond",
new BasicDBObject(
"$and", Arrays.<Object>asList(
new BasicDBObject(
"$gte", Arrays.<Object>asList("$date", initialStart)
),
new BasicDBObject(
"$lte", Arrays.<Object>asList("$date", initialEnd)
)
)
),
1,
0
)
)
);
The BasicDBObject is the first arg, 1 and 0 are the 2nd and 3rd and they will be correctly interpreted by Morphia wrapper.
I am trying to learn how to use the Mongo in Java and have been able to make some simple queries but I have been having trouble with the aggregate operator.
The document structure is a simple one, as the following:
{
"_id" : ObjectId("57dbe94f0507a4d8710ac5b2"),
"name" : "Name1",
"age" : 23
}
{
"_id" : ObjectId("57dbe9750507a4d8710ac5b3"),
"name" : "Name2",
"age" : "String for examble"
}
{
"_id" : ObjectId("57dbee630507a4d8710ac5b5"),
"name" : "Name3",
"age" : 24
}
All I want to do is get the average of the ages in the collection ( name example ).
Simply using mongo I can get the desirable result with the following consult:
db.example.aggregate([
{
$group: {
_id: null,
averageAge: { $avg: "$age" }
}
}
]);
I have tried the following:
BasicDBObject groupFields = new BasicDBObject("_id", "null");
BasicDBObject media = new BasicDBObject("$avg", "$age");
groupFields.put("mediaIdade", media);
BasicDBObject group = new BasicDBObject("$group", groupFields );
AggregateIterable<org.bson.Document> agregate = db.getCollection("exemplo").aggregate(Arrays.asList (group));
Which is almost a direct translation but got a "java.lang.NoSuchMethodError: org.bson.codecs.BsonTypeClassMap.keys()Ljava/util/Set;" , unsurprisingly.
But I cannot translate that to Java. I have checked and found this question but could not understand it due to the use of opperators such as $unwind. So I'm trying to make query as simple as possible to better understand how the Java framework for aggregation works.
Can someone help?
Try something like this.
MongoCollection<Document> dbCollection = db.getCollection("example", Document.class);
AggregateIterable<org.bson.Document> aggregate = dbCollection.aggregate(Arrays.asList(Aggregates.group("_id", new BsonField("averageAge", new BsonDocument("$avg", new BsonString("$age"))))));
Document result = aggregate.first();
double age = result.getDouble("averageAge");
Input:
{ "_id" : ObjectId("58136d6ed96cc299c224d529"), "name" : "Name1", "age" : 23 }
{ "_id" : ObjectId("58136d6ed96cc299c224d52a"), "name" : "Name2", "age" : 26 }
{ "_id" : ObjectId("58136d6ed96cc299c224d52b"), "name" : "Name3", "age" : 24 }
Output:
24.333333333333332
I noticed a small typo in your code:
db.getCollection("exemplo").aggregate(Arrays.asList (group));
Should be example instead of examplo
Also the easiest way to translate working mongo expression is using Document.parse method.
In your case it could be:
db.getCollection("exemplo").aggregate(Arrays.asList(
Document.parse("{ $group: { _id: null, averageAge: { $avg: '$age' } } }")
));
Your impl also almost correct, two minor issues:
replace "null" string with just null value
Use "averageAge" instead of "mediaIdade"
BasicDBObject groupFields = new BasicDBObject("_id", null);
BasicDBObject media = new BasicDBObject("$avg", "$age");
groupFields.put("averageAge", media);
BasicDBObject group = new BasicDBObject("$group", groupFields );
AggregateIterable<org.bson.Document> agregate = db.getCollection("exemple").aggregate(Arrays.asList (group));
To get result:
agregate.first().getDouble("averageAge");
Try this:
DBObject groupFields = new BasicDBObject( "_id", 0);
groupFields.put("average", new BasicDBObject( "$avg", "$age"));
DBObject group = new BasicDBObject("$group", groupFields);
AggregationOutput output = db.getCollection("exemplo").aggregate(group);
Iterable<DBObject> list = output.results();
If required add a filter to the query you can add a match parameter:
DBObject match = new BasicDBObject();
match.put("age", new BasicDBObject("$gte", 25));
DBObject groupFields = new BasicDBObject( "_id", 0);
groupFields.put("average", new BasicDBObject( "$avg", "$age"));
DBObject group = new BasicDBObject("$group", groupFields);
AggregationOutput output = db.getCollection("exemplo").aggregate(match, group);
Iterable<DBObject> list = output.results();
I have a collection in a Mongo database that looks like this:
{
"_id" : ObjectId("561b42d4e4b0d4227d011d2c"),
"product_related_data" : {
"depth" : 6,
"height" : 23,
"product_barcode" : "54491472",
"product_name" : "xyz product",
"product_uuid" : "009b9846-b3ad-49f7-a7a0-d35a04f83480",
"width" : 6
},
"sensostrip_data" : {
"barcode" : "130150208299",
"battery_level" : 2.894
},
"stock_related_data" : {
"calculated_max_product_percentage" : 15.625,
"calculated_min_product_percentage" : 12.5,
"current_stock" : 2,
"max_stock" : 6,
"stock_difference_percentage" : 0,
"stock_difference_value" : 0,
"stock_percentage" : 37.5
},
"store_data" : {
"city_uuid" : "cbb4dfe8-172b-11e4-a1f0-00163ed23ec2",
"ip" : "10.0.1.1",
"retailer_uuid" : "8c33c32c-5903-11e4-a1f0-00163ed23ec2",
"store_name" : "xyz store",
"store_uuid" : "15a6cc90-081f-11e5-b213-001e6745ff8d"
},
"time" : {
"check_date" : "2015-10-11 11:53:55",
"previous_check_date" : "2015-10-11 11:48:57"
},
"id" : "6be54bef-0aa3-456c-b912-1731f8154e7d"
}
The mongo query I'm currently executing returns all the documents for a list of store_uuid's and for a product_uuid, and looks like this:
db.readings
.find({ $and:[ {"store_data.store_uuid": {$in:["15a6cc90-081f-11e5-b213-001e6745ff8d","217b983b-5904-11e4-a1f0-00163ed23ec2","5337d78d-5904-11e4-a1f0-00163ed23ec2"]}}
,{"product_related_data.product_uuid": "f44aa29d-09ce-4902-bf12-d45d44b3dfd0"}]})
My current Java implementation (where I make use of projection) looks like this:
DBCollection table = databaseConncetion().getCollection("readings");
BasicDBObject sensorReturn = new BasicDBObject("sensostrip_data.barcode",1);
BasicDBObject clause1 = new BasicDBObject("store_data.store_uuid", new BasicDBObject("$in", StoreIds));
BasicDBObject clause2 = new BasicDBObject("product_related_data.product_uuid", productId);
BasicDBList and = new BasicDBList();
and.add(clause1);
and.add(clause2);
DBObject query = new BasicDBObject("$and", and);
DBCursor cursor = table.find(query, sensorReturn
.append("stock_related_data.stock_percentage",1)
.append("store_data.store_uuid",1)
.append("time.check_date", 1))
.sort(new BasicDBObject("time.check_date", -1))
.limit(100);
However I need this query to group the results for the latest check_date by barcode
The aggregation framework is at your disposal. Running the following aggregation pipeline will give you the desired result:
Mongo shell:
pipeline = [
{
"$match": {
"store_data.store_uuid": {
"$in": [
"15a6cc90-081f-11e5-b213-001e6745ff8d",
"217b983b-5904-11e4-a1f0-00163ed23ec2",
"5337d78d-5904-11e4-a1f0-00163ed23ec2"
]
},
"product_related_data.product_uuid": "f44aa29d-09ce-4902-bf12-d45d44b3dfd0"
}
},
{ "$sort": { "time.check_date": -1 } },
{
"$group": {
"_id": "$sensostrip_data.barcode",
"stock_percentage": { "$first": "$stock_related_data.stock_percentage" },
"store_uuid": { "$first": "$store_data.store_uuid" },
"check_date": { "$first": "$time.check_date" }
}
},
{ "$limit": 100 }
];
db.readings.aggregate(pipeline);
Java test implementation
public class JavaAggregation {
public static void main(String args[]) throws UnknownHostException {
MongoClient mongo = new MongoClient();
DB db = mongo.getDB("test");
DBCollection coll = db.getCollection("readings");
// create the pipeline operations, first with the $match
DBObject match = new BasicDBObject("$match",
new BasicDBObject("store_data.store_uuid", new BasicDBObject("$in", StoreIds))
.append("product_related_data.product_uuid", productId)
);
// sort pipeline
DBObject sort = new BasicDBObject("$sort",
new BasicDBObject("time.check_date", -1)
);
// build the $group operations
DBObject groupFields = new BasicDBObject( "_id", "$sensostrip_data.barcode"); // group by barcode
groupFields.put("stock_percentage", new BasicDBObject( "$first", "$stock_related_data.stock_percentage")); // get the first when ordered documents are grouped
groupFields.put("store_uuid", new BasicDBObject( "$first", "$store_data.store_uuid"));
groupFields.put("check_date", new BasicDBObject( "$first", "$time.check_date"));
// append any other necessary fields
DBObject group = new BasicDBObject("$group", groupFields);
// limit step
DBObject limit = new BasicDBObject("$limit", 100);
// put all together
List<DBObject> pipeline = Arrays.asList(match, sort, group, limit);
AggregationOutput output = coll.aggregate(pipeline);
for (DBObject result : output.results()) {
System.out.println(result);
}
}
}
I need to convert mongodb query into java.
I have "conversations" collection.
Mongo db query this way and it's work.
{
messages: { source: 1, $elemMatch: { date: { $gte: ISODate("2013-07-25 00:00:00"), $lt: ISODate("2013-08-26 00:00:00")}}}
}
And I am tring into java BasicDBObject
elemMatch = new BasicDBObject();
BasicDBObject retrievedField = new BasicDBObject();
elemMatch.put("date", new BasicDBObject("$gte",StartDate).append("$lte",EndDate));
elemMatch.put("source", 1);
BasicDBObject up = new BasicDBObject();
up.put("$elemMatch",elemMatch);
retrievedField.put("messages", up);
DBCursor cursor = this.dbCollection.find( retrievedField).limit(10);
but this code not working
My collection's data is
{
"_id" : ObjectId("51f130d6e4b0bf50e9bfc038"),
"saleId" : 2.43564e+07,
"saleCode" : "905155440001",
"randomId" : 9630,
"creatorId" : 8.21048e+06,
"recipientId" : 4.83831e+06,
"creatorType" : 1,
"returnReasonId" : 11,
"conversationClosed" : false,
"operatorCalled" : false,
"sellerNotified" : false,
"buyerNotified" : false,
"operatorCalledDate" : null,
"creationDate" : ISODate("2013-07-25T14:06:14.967Z"),
"lastUpdateDate" : ISODate("2013-08-15T08:46:10.115Z"),
"messages" : [
{
"senderId" : 8.21048e+06,
"source" : 1,
"seenByBuyer" : true,
"seenBySeller" : true,
"seenByOperator" : true,
"date" : ISODate("2013-07-25T14:06:39.968Z"),
"messageBody" : "asdad"
},
{
"senderId" : 8.21048e+06,
"source" : 1,
"seenByBuyer" : true,
"seenBySeller" : true,
"seenByOperator" : true,
"date" : ISODate("2013-07-25T14:06:59.978Z"),
"messageBody" : "asdasdawdwa"
},
{
"senderId" : 0,
"source" : 4,
"seenByBuyer" : true,
"seenBySeller" : true,
"seenByOperator" : true,
"date" : ISODate("2013-07-25T14:07:20.044Z"),
"messageBody" : "ad"
}
]
}
Any help?
Calendar calStartDate = Calendar.getInstance();
Calendar calEndDate = Calendar.getInstance();
SimpleDateFormat simpleDateFormat = new SimpleDateFormat ("yyyy-MM-dd HH:mm:ss");
Date StartDate = simpleDateFormat.parse(messageStartDate);
Date EndDate = simpleDateFormat.parse(messageEndDate);
calStartDate.setTime(StartDate);
calEndDate.setTime(EndDate);
Date sdate = new DateTime(calStartDate.get(Calendar.YEAR),calStartDate.get(Calendar.MONTH), calStartDate.get(Calendar.DAY_OF_MONTH),calStartDate.get(Calendar.HOUR),calStartDate.get(Calendar.MINUTE), DateTimeZone.UTC).toDate();
Date edate = new DateTime(calEndDate.get(Calendar.YEAR), calEndDate.get(Calendar.MONTH), calEndDate.get(Calendar.DAY_OF_MONTH), calEndDate.get(Calendar.HOUR), calEndDate.get(Calendar.MINUTE), DateTimeZone.UTC).toDate();
BasicDBObject query = new BasicDBObject(
"messages",
new BasicDBObject("$elemMatch",new BasicDBObject("source", 1).append("date", new BasicDBObject("$gte", sdate).append("$lt", edate)))
);
DBCursor cursor = this.dbCollection.find( query).limit(10);
Your shell query was not correct anyway. It should be:
{
"messages": {
"$elemMatch": {
"source": 1,
"date": {
"$gte": ISODate("2013-07-25 00:00:00"),
"$lt": ISODate("2013-07-26 00:00:00")
}
}
}
}
And it helps to follow the same nesting rules so you don't get the flow incorrect:
BasicDBObject query = new BasicDBObject(
"messages",
new BasicDBObject(
"$elemMatch",
new BasicDBObject( "source", 1 )
.append( new BasicDBObject(
"date",
new BasicDBObject( "$gte", StartDate )
.append( "$lt", EndDate )
)
)
)
);
Make sure that "dates" are of type java.util.Date returned from something like jodatime to be valid for serialization as well. And make sure that are UTC times you are constructing from.
Example:
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
Date StartDate = new DateTime(2013, 7, 25, 0, 0, DateTimeZone.UTC).toDate();
The dataset can be found in http://docs.mongodb.org/manual/tutorial/aggregation-zip-code-data-set/
Now , I am trying to implement the following function "Return Average City population by State":
db.zips.aggregate( { $group :
{ _id : { state : "$state", city : "$city" },
pop : { $sum : "$pop" } } },
{ $group :
{ _id : "$_id.state",
avgCityPop : { $avg : "$pop" } } } )
with an output of:
{
"result" : [
{
"_id" : "RI",
"avgCityPop" : 18933.28301886793
}
Please help me in JAVA, Thank you
Try this:
// First $group operation
DBObject compoundId = new BasicDBObject( "state", "$state");
compoundId.append("city", "$city")
DBObject groupFields1 = new BasicDBObject( "_id", compoundId );
groupFields1.put("pop", new BasicDBObject( "$sum", "$pop"));
DBObject group1 = new BasicDBObject("$group", groupFields1);
// Second $group operation
DBObject groupFields2 = new BasicDBObject( "_id", "$_id.state");
groupFields2.put("avgCityPop", new BasicDBObject( "$avg", "$pop"));
DBObject group2 = new BasicDBObject("$group", groupFields2);
// Run aggregation
AggregationOutput output = collection.aggregate( group1, group2 );