I am trying to create a query using MongoDB Java Driver as part of an aggregation command. Currently I allow a date range or an array of specific dates as an argument. eg
<date>
<start>2013-12-10 00:00:00.000</start>
<end>2013-12-12 23:59:59.999</end>
</date>
or
<date>
<specificDates>2013-12-10 00:00:00.000,2013-12-13 00:00:00.000</specificDates>
</date>
The date range query works fine, I parse and convert the xml into a DBObject that produces the following query in mongo;
{ "$match" : { "d" : { "$gte" : { "$date" : "2013-10-01T00:00:00.000Z"} , "$lt" : { "$date" : "2013-10-04T00:00:00.000Z"}}}}
For the specificDates I want to return only results that occur between 00:00:00.000 on the given day and 00:00:00.000 of the next day. From my pretty basic knowledge of mongo querys i had hoped to do a similar $match as the date range, but have it use $in on an array of date ranges similar to the following;
{ "$match" : { "d" : { "$in" : [ { "$gte" : { "$date" : "2013-10-01T00:00:00.000Z"} , "$lt" : { "$date" : "2013-10-02T00:00:00.000Z"}} , { "$gte" : { "$date" : "2013-10-03T00:00:00.000Z"} , "$lt" : { "$date" : "2013-10-04T00:00:00.000Z"}}]}}}
The above query fails to return any results. I have noticed that $in is not listed in the mongodb manual under the Mongo Aggregation Framework section, but its not throwing any kind of errors that I would have expected for an unsupported operation.
I think the issue may come from this line in the MongoDB Manual;
If the field holds an array, then the $in operator selects the documents whose field holds an array that contains at least one element that matches a value in the specified array (e.g. , , etc.)
In my collection the date isn't stored in an array, I suppose I could store it in the collections in an single element array? (Actually, decided to try this quickly before I posted, no documents returned when the date entry in the document is stored in a single element array)
Document entry example
{ "_id" : ObjectId("52aea5b0065991de1a56d5b0"), "d" : ISODate("2013-12-15T00:00:11.088Z"), "t" : 1501824, "s" : 0, "e" : 601, "tld" : "uk", "y" : "domain:check", "n" : "removed.co.uk" }
Is anyone able to give me some advice as to how I should do this query? Thank you.
EDIT: I left the Java tag here in case anyone needs my DBObject creation code, though it shouldn't be necessary as the queries posted have been generated by my build.
EDIT2: So as Alan Spencer pointed out I should be using $or rather than $in, a working $or function is below (ignore the different formatting like the use of ISODate(), its just copy pasted from the mongo shell rather than getting output from my program)
{ $match : { $or : [ { d : { $gte : ISODate("2013-10-01T00:00:00.000Z"), $lt : ISODate("2013-10-02T00:00:00.000Z") } }, { d : { $gte : ISODate("2013-10-03T00:00:00.000Z"), $lt : ISODate("2013-10-04T00:00:00.000Z") } } ] } }
I think you're inverting the meaning of the $in.
$in is used to match exactly against a list of possible values, like
{"color":{"$in": ["red","green","blue"]}}
For your use case, you are trying to match if it satisfies the first or second, etc. So, you can use $or - http://docs.mongodb.org/manual/reference/operator/query/or/
{ "$match" : { "d" : { "$or" : [ { "$gte" : { "$date" : "2013-10-01T00:00:00.000Z"} , "$lt" : { "$date" : "2013-10-02T00:00:00.000Z"}} , { "$gte" : { "$date" : "2013-10-03T00:00:00.000Z"} , "$lt" : { "$date" : "2013-10-04T00:00:00.000Z"}}]}}}
Related
i have Mongodb collection which contains single field , each day i am receiving 31000 documents and in the collection i have almost 6 months data
Here is how my data looks like in database
{
"_id" : ObjectId("59202aa3f32dfba00d0773c3"),
"Data" : "20-05-2017 18:38:13 SYSTEM_000_00_SAVING ",
"__v" : 0
}
{
"_id" : ObjectId("59202aa3f32dfba00d0773c4"),
"Data" : "20-05-2017 18:38:13 SyTime_000_09_00:00 ",
"__v" : 0
}
here is my code for query
DBObject query = new BasicDBObject();
Pattern regex = Pattern.compile("20-05-2017");
query.put("Data", regex);
i have created index but its still slow
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "NOB_SRB.fdevices"
},
{
"v" : 1,
"unique" : true,
"key" : {
"Data" : 1.0
},
"name" : "Data_1",
"ns" : "NOB_SRB.fdevices"
}
]
Add a start of input anchor ^ to the start of the regex:
Pattern regex = Pattern.compile("^20-05-2017");
Because your regex does not have an anchor, the entire field is searched for the date anywhere in it, which requires every character in the field to be compared.
I have this two documents in my mongoDB database:
db.DocumentFile.find().pretty()
{
"_id" : ObjectId("587f39910cc0fec092bdb10c"),
"_class" : "com.smartinnotec.legalprojectmanagement.dao.domain.DocumentFile",
"fileName" : "DocumentFile1",
"ending" : "jpg",
"projectId" : "587f39910cc0fec092bdb10b",
"active" : true,
"userIdBlackList" : [
"587f39910cc0fec092bdb10a"
]
}
{
"_id" : ObjectId("587f39910cc0fec092bdb10d"),
"_class" : "com.smartinnotec.legalprojectmanagement.dao.domain.DocumentFile",
"fileName" : "DocumentFile2",
"ending" : "jpg",
"projectId" : "587f39910cc0fec092bdb10b",
"active" : true,
"userIdBlackList" : [ ]
}
I have this code in order to get amount of query:
final Query query = new Query();
query.addCriteria(Criteria.where("userIdBlackList").nin(userId));
final Long amount = mongoTemplate.count(query, DocumentFile.class);
return amount.intValue();
The amount is 2 in this case what is wrong - it should be 1.
The query in Query object looks like this:
Query: { "userIdBlackList" : { "$nin" : [ "587f39910cc0fec092bdb10a"]}}
If I copy this query and made a query for the mongodb console like this:
db.DocumentFile.find({ "userIdBlackList" : { "$nin" : [ "587f39910cc0fec092bdb10a"]}}).pretty()
I get an amount of two, what if wrong because one document includes 587f39910cc0fec092bdb10a in userIdBlackList -> it should be one.
With this query command:
db.DocumentFile.find({userIdBlackList: { "$nin": ["587f39910cc0fec092bdb10a"] } }).pretty();
I get the right result, I am really confused at the moment.
Does anyone have any idea?
Maybe the problem ist that one time userIdBlackList is with quotation mark ("userIdBlackList") and the other time it isn't.
I think the problem is with the unintentional formatting picked up for "userIdBlackList". Your string is interpreted with non printing characters in the "??userIdBlackList" for all your search queries. I see little transparent square boxes when I copy your queries to mongo shell.
That tells me their is some encoding issue. Clear that formatting and see if that helps you.
Both $ne and $nin should work!
I have a mongo collection named firma which has one of the document structure as below:
{
"_id" : ObjectId("5729af099b3ebf1d0ca7ff05"),
"musteriler" : [
{
"_id" : "de0bf813-b707-4a8d-afc2-9752e05c3aa5",
"yetkiliListesi" : [
{
"_id" : "a5e487fa-2034-4817-94f2-3bd837b76284",
"ad" : "Burak",
"soyad" : "Duman 1",
"cepTel" : "3333333333333",
"mail" : "asdf#asdf.com"
},
{
"_class" : "com.bisoft.entity.MusteriYetkili",
"_id" : "bc4b537d-522a-4c9a-9f67-8ca243e18f46",
"ad" : "Ridvan",
"soyad" : "ENİŞ",
"cepTel" : "222222222222",
"mail" : "asdf#asdf.com"
}
]
}
],
"defaultTimezone" : "Europe/Istanbul"
}
In the above json, I need to update element of second array(yetkiliListesi) which _id = "a5e487fa-2034-4817-94f2-3bd837b76284"
Since I am using a java application(using mongo java driver and spring boot MongoTemplate) to access it and execute this query :
mongoTemplate.updateFirst(Query.query(Criteria.where("_id").is("5729af099b3ebf1d0ca7ff05").and("musteriler.yetkiliListesi._id").is("a5e487fa-2034-4817-94f2-3bd837b76284")),
new Update().set("musteriler.yetkiliListesi.$", yetkiliDBO), Firma.class);
In the above query, yetkiliDBO is a BasicDBObject and its content :
yetkiliDBO = {
'_class': 'com.bisoft.entity.MusteriYetkili',
'_id': "a5e487fa-2034-4817-94f2-3bd837b76284",
'ad': 'wer',
'soyad': 'xyz',
'cepTel': "222222222222",
mail: "asdf#asdf.com"
}
when execute my query I have an error
com.mongodb.WriteConcernException: { "serverUsed" : "192.168.2.250:27017" , "ok" : 1 , "n" : 0 , "updatedExisting" : false , "err" : "cannot use the part (musteriler of musteriler.yetkiliListesi.0) to traverse the element
What I need to do?
You can not use the '$' placeholder when traversing nested arrays.
The positional $ operator cannot be used for queries which traverse more than one array, such as queries that traverse arrays nested within other arrays, because the replacement for the $ placeholder is a single value
source
I would suggest restructuring your data into separate, less-nested collections.
I have a collection containing following documents -
{
"_id" : ObjectId("56e7a51b4a66e30330151847"),
"host" : "myTestHost.com",
"sessionId" : "daxxxxxx-xxxx-xxxx-xxxx-xxxxxxx1",
"ssoId" : "xxxxxx#gmail.com",
"days" : {
"13" : NumberLong(130),
"11" : NumberLong(457),
"10" : NumberLong(77)
},
"count" : NumberLong(664),
"timeStamp" : NumberLong("1458021713370")
}
I am using mongo java driver 3.2.1.
This document contains an embedded document 'days', that holds a specific count for each day of month.
I need to find the number of days for which a count is present.For example - for above document mentioned, the number of days for which count is present is 3 (13th, 11th and 10th day of month).
I know how to get the count on mongo console -
mongos>var count = 0;
mongos> db.monthData.find({},{days:1}).forEach(function(record){for(f in record.days) { count++;}});
mongos> count;
I need to convert this to java code.
Maybe you can reshape your schema as follow:
{
"_id" : ObjectId("56e7a51b4a66e30330151847"),
"host" : "myTestHost.com",
"sessionId" : "daxxxxxx-xxxx-xxxx-xxxx-xxxxxxx1",
"ssoId" : "xxxxxx#gmail.com",
"days" : [
{
"13" : NumberLong(130)
},
{
"11" : NumberLong(457)
},
{
"10" : NumberLong(77)
}
],
"count" : NumberLong(664),
"timeStamp" : NumberLong(1458021713370)
}
days becomes an array of objects and in this way you can easily use aggregation pipeline to know how many elements are in days array:
>db.collection.aggregate(
[
{$match:{days:{"$exists":1}}},
{$project:{
numberOfDays: {$size:"$days"},
_id:1}
}
]
)
The aggregation returns:
{ "_id" : ObjectId("56e7a51b4a66e30330151847"), "numberOfDays" : 3 }
To use Aggregation Pipeline with Java driver see aggregate, AggregateIterable, Block and read Data Aggregation with Java Driver
I have a collection with a dataset that looks like:
{
"resource" : "10.10.10.10",
"statistics" : {
"connections" : 17
}
}
{
"resource" : "10.10.10.10",
"statistics" : {
"connections" : 24
}
}
I want to use Mongo's $group/$push mechanism to return a dataset that looks like:
{ "_id" : "10.10.10.10", "statTotals" : [ 17, 24 ] }
In Mongo shell, I can that by doing:
db.testcol.aggregate([ { "$project" : { "resource" : 1 , "total" : "$statistics.connections"}} , { "$group" : { "_id" : "$resource" , "statTotals" : { "$push" : "$total"}}} ])
Now I want to do this using Spring's Mongo data solution in Java. The operations I'm currently trying to use are:
ProjectionOperation projOper = Aggregation.project("resource").and("statistics.connections").as("total");
GroupOperation groupOper = Aggregation.group("resource").push("total").as("statTotals");
Unfortunately, this is generating a pipeline which looks like:
{ "aggregate" : "event" , "pipeline" : [ { "$project" : { "resource" : 1 , "total" : "$statistics.connections"}} , { "$group" : { "_id" : "$resource" , "statTotals" : { "$push" : "$statistics.connections"}}}]}
In the $group, the $push is being done against $statistics.connections instead of $total so the results come back blank.
Any help would be greatly appreciated on this. At first I thought this might be https://jira.spring.io/browse/DATAMONGO-1254 but I've tried using both 1.7.2 as well as 1.8.1 and get the same results.