I have a collection for which i want to make queries with pagination by sorting it with dateCreated field as decending to get latest documents.
Am holding lastKnownCommentId(Object id) for performance purpose. Having lastKnownCommentId would avoid loading documents from starting again during next paginations, if not then applying limit on query will cause performance issue.
Query<Comment> query = datastore.createQuery(Comment.class)
.field(POST_ID).equal(postId);
if (lastKnownCommentId != null) {
query.field(FieldConstants.OBJECT_ID).greaterThan(new ObjectId(lastKnownCommentId));
}
query.order(Sort.descending(FieldConstants.DATE_CREATED));
return query.asList(new FindOptions().limit(10));
Now i have 12 documents in this collection which matches to one postId. When this query is excecuted for first pagination with lastKnownCommentId=null it gives me 10 documents sorted by date, 2 documents are still not in pagination picture.
For second pagination with lastKnownCommentId=someId (someId is object id of last document obtained from first pagination), it gives me again 9 documents as result instead of 2 documents which remained in first pagination.
Things are working fine if i dont do sorting by date, i can completely skip this sorting and do sorting on array list instead. Am not understanding why this happening with sorting in query.
I tried to cross check with aggregation and results same ouput.
db.comment.aggregate(
// Pipeline
[
// Stage 1
{
$match: {
"postId":{"$eq":"5fb2090fe4d37312a4c3ce59"}
}
},
// Stage 2
{
$sort: {
"dateCreated":-1
}
},
// Stage 3
{
$match: {
"_id":{"$gt":ObjectId("5fb0e53392ad724f9026d2f7")}
}
},
// Stage 4
{
$limit: // positive integer
10
},
],
// Options
{
cursor: {
batchSize: 50
}
}
);
What does query look like just before you return it? (print out the value, or do a for loop if it's an object so you can see what the values are).
Related
I have entity of this type saved to elasticSearch:
class Value {
private Integer priority;
private Date dueDate;
}
For priority possible values are 1 and 2(I know that enum should be used but let's not concentrate on that).
Now using elastic search I want to get the following result:
How many values I have with priority 1 that have dueDate today or in the past
How many values I have with priority 1 that have dueDate null or in the future
How many values I have with priority 2 that have dueDate today or in the past
How many values I have with priority 2 that have dueDate null or in the future
What I tried doing is the following:
SearchRequest request = new SearchRequest.Builder()
.index(MY_INDEX)
.query(getQuery(someKeys))
.aggregations(ELASTIC_AGG_PRIORITY, Aggregation.of(a ->
a.terms(terms -> terms.field(ELASTIC_AGG_PRIORITY).missing(""))))
.build();
This will get me aggregations that are grouped by priority and will know the number of values with priority 1 and with priority 2. But I need to add nested aggregations and here is my issue. How to separate these buckets that I am getting into one with dueDate in past or today and with dueDate in future or null. Additionally I am having issue with writing the range query:
public static Query getRangeQuery(String field) {
return QueryBuilders.range().field(field).lte(JsonData.of(new Date())).build()._toQuery();
}
I am not able to pass the year and I don't know how to handle the nulls when I need to select everything in the future and null values. I tried the following but not getting the result I need:
SearchRequest request = new SearchRequest.Builder()
.index(MY_INDEX)
.query(getQuery(someKeys))
.aggregations(ELASTIC_AGG_ASSIGNEE, Aggregation.of(a ->
a.terms(terms -> terms.field(ELASTIC_AGG_ASSIGNEE).missing(""))))
.aggregations(ELASTIC_AGG_PRIORITY, Aggregation.of(a ->
a.terms(terms -> terms.field(ELASTIC_AGG_PRIORITY).missing(""))
.aggregations(ELASTIC_AGG_PRIORITY, Aggregation.of(filterAgg -> filterAgg
.filter(QueryUtil.getRangeQuery("dueDate"))))
)
).build();
where QueryUtil.getRangeQuery( is the method described above.
So to summarise my question is how to get the 4 above mentioned results?
You're close but you need to implement the "date_range_aggregation" to get the results you are looking for.
Your terms aggregation is ok, all you need to do is add the date_range_aggregation as a subaggregation to the terms aggregation.
For your "null" values, set a "missing" value as something in the past.That way you can pick that up as a third bucket.
Sample[in json]:
{
"aggs": {
"range": {
"date_range": {
"field": "date",
"missing": "1976/11/30",
"ranges": [
{
"key": "Oldest",
"to" : "1976/11/30"
},
{
"key": "Past",
"from" : "1976/11/30"
"to": "now"
},
{
"key": "Future",
"from": "now"
}
]
}
}
}
}
Let me know if this works for you.
So, I have this repository in MongoDB that holds movies with this structure:
title: String
description: String
likes: Set<String>
hates: Set<String>
The likes & hates are a Set because they hold a list of UserIds - where the user with those UserIds are the ones that liked/hated the movie.
I am trying to have my service get all movies from the database, sorted by the number of likes/hates. Previously, my structure was different, and likes/hates were just Integers. Then, getting all sorted movies was easy:
public List<MovieDocument> getSortedMovies(SortProperty sortBy, Order order) {
return moviesRepository.findAll(Sort.by(fromString(order.toString()), sortBy.toString()))
}
Where sortBy was either likes or hates and order was either asc or desc, provided by the client of the API.
In the above case, MoviesRepository didn't have any custom methods:
#Repository
public interface MoviesRepository extends MongoRepository<MovieDocument, String> {}
How am I supposed to do that now that likes and hates are Set objects?
Again, what I want is to get all movies sorted by the size of the likes/hates sets.
Can I do that using any of the built-in MongoRepository methods? I had a look and didn't see anything useful.
Looking at other StackOverflow posts, I saw there is an option to add methods to my MoviesRepository with an Aggregation annotation. This would look something like:
#Aggregation("{$project: { 'like_count': { $size: '$likes' } }}, {$sort: {'like_count': -1}}]")
List<String> getSortedMovieIdsByLikesDesc();
However, this does not return the whole MovieDocument, but rather it returns the number of likes. In addition to that, it looks like I'd have to create a new custom method for each property/order combination i.e. likes-asc, likes-desc, hates-asc, hates-desc. This feels tedious and not very extensible.
How would I fix the above to return whole documents and is there any other way to do this I'm not considering?
EDIT
I tried the following based on input from #rickhg12hs.
#Aggregation("{$set: { like_count: { $size: $likes } }}, {$sort: {like_count: -1}}")
List<MovieDocument> getSortedMovieIdsByLikesDesc();
#Aggregation("{$set: { like_count: { $size: $likes } }}, {$sort: {like_count: 1}}")
List<MovieDocument> getSortedMovieIdsByLikesAsc();
#Aggregation("{$set: { hate_count: { $size: $hates } }}, {$sort: {hate_count: -1}}")
List<MovieDocument> getSortedMovieIdsByHatesDesc();
#Aggregation("{$set: { hate_count: { $size: $hates } }}, {$sort: {hate_count: 1}}")
List<MovieDocument> getSortedMovieIdsByHatesAsc();
Unfortunately, all four of those methods seem to return the exact same thing when called. Specifically they return the two items that are in the database unordered.
You seem to be doing almost everything right. Here's an example that does what I think you want.
db.collection.aggregate([
{
// Count likes and hates
"$set": {
"likeCount": {
"$size": "$likes"
},
"hateCount": {
"$size": "$hates"
}
}
},
{
// Most likes first, split ties with
// least hates first
"$sort": {
"likeCount": -1,
"hateCount": 1
}
},
// {
// "$project": {
// "likeCount": 0,
// "hateCount": 0
// }
// }
])
You can uncomment the "$project" stage if you want to remove the counts too.
Try it on mongoplayground.net.
I have a node called quotes in Firebase. I'm facing issues while fetching data in Android for a particular range. I want to fetch 3 continues quotes id starting from 2. Here is my query database:
"quotes" : {
"-L75elQJaD3EYPsd4oWS" : {
"authorName" : "Hellen v",
"famousQuote" : "When one door of happiness closes, another opens; but often we look so long at the closed door that we do not see the one which has been opened for us.",
"id" : "1",
"uploadedBy" : "Admin"
},
"-L7GOvDNI-o_H8RvNwoN" : {
"authorName" : "Rocky Balboa",
"famousQuote" : "It's not about how hard you can hit; it's about how hard you can get hit and keep moving forward.",
"id" : "2",
"uploadedBy" : "Admin"
},
"-L7GP9oBv5NR1T6HlDd4" : {
"authorName" : "African proverb",
"famousQuote" : "If you want to go fast, go alone. If you want to go far, go together.",
"id" : "3",
"uploadedBy" : "Admin"
},
"-L7GPjM1F3_7Orcz0Q1q" : {
"authorName" : "A.P.J Abdul Kalam",
"famousQuote" : "Don’t take rest after your first victory because if you fail in second, more lips are waiting to say that your first victory was just luck.",
"id" : "4",
"uploadedBy" : "Admin"
},
Below is the rule which I'm using for quotes
"quotes": {
".indexOn": ".value"
}
How can I get quotes which has id 2,3 and 4?
If you have more than 4 records in your database, to solve this, you can use a query in which you should combine startAt() and endAt() methods to limit both ends of your query like this:
DatabaseReference rootRef = FirebaseDatabase.getInstance().getReference();
Query query = rootRef.child("quotes").orderByChild("id").startAt("2").endAt("4");
query.addListenerForSingleValueEvent(/* ... */);
Please see here more informations about Firebase Query's startAt() method:
Create a query constrained to only return child nodes with a value greater than or equal to the given value, using the given orderBy directive or priority as default.
And here are more informations about Firebase Query's endAt() method:
Create a query constrained to only return child nodes with a value less than or equal to the given value, using the given orderBy directive or priority as default.
Edit: According to your comment, if you only want the items that have the id property set to 2, 3 and 4, you should use nested queries like this:
Query queryTwo = rootRef.child("quotes").orderByChild("id").equalsTo("2");
queryTwo.addListenerForSingleValueEvent(
List<Item> list = new ArrayList();
list.add(itemTwo);
Query queryThree = rootRef.child("quotes").orderByChild("id").equalsTo("3");
queryThree.addListenerForSingleValueEvent(
list.add(itemThree);
Query queryFour = rootRef.child("quotes").orderByChild("id").equalsTo("4");
queryFour.addListenerForSingleValueEvent(
list.add(itemFour);
//Do what you need to do with the list that contains three items
);
);
);
Suppose I have a collection with multiple documents with the following structure:
{
_id: "id12345",
objects: ["123","456", "789"]
}
Now I want to create a query where I pass an array of objects and get the count of occurrence in documents for each of the elements of the array, something like this:
input of query: ["123","321"]
output:
{
"123": 1,
"321": 0
}
I want to do this just by one query, passing an array, I know I could do this easily one by one instead of an array but that is not the purpose.
You can use below aggregation in 3.6.
$match to filter documents where there is at least one array match between input array and objects array.
$project with $map to iterate the input array and compare with each array element in objects array and return 0 and 1 depending on the match.
$unwind with $group to count the occurrences of array element across all documents.
db.colname.aggregate([
{"$match":{"objects":{"$in":["123","321"]}}},
{"$project":{
"keyandcount":{
"$map":{
"input":["123","321"],
"in":{
"k":"$$this",
"v":{"$cond":[{"$in":["$$this","$objects"]},1,0]}
}
}
}
}},
{"$unwind":"$keyandcount"},
{"$group":{"_id":"$keyandcount.k","count":{"$sum":"$keyandcount.v"}}}
])
You can add below two stages at the end to format the response to key value pair.
{"$group":{_id: null,"keycountpair":{"$mergeObjects":{"$arrayToObject":[[["$_id","$count"]]]}}}},
{"$replaceRoot":{"newRoot":"$keycountpair"}}
In general the query to get this is pretty straight forward since it is not recommended to switch values to keys in the data layer but more in the presentation one.
So this:
db.collection.aggregate([
{
$unwind: "$objects"
},
{
$group: {
_id: "$objects",
count: {
$sum: 1
}
}
}
])
Would give you the numbers as _id and a count field with the number of occurrences. Then you can massage that in your app to make it look anyway you would like.
Here is a simple example of the result
I have a method which takes in a Collection of Objects that are to be deleted.
This is the way I am deleting them now
public void deleteAll(Collection<Object> objs){
for(Object obj : objs) {
collection.remove("{ _id: # }", obj.getId());
}
}
I am doing something very similar for update where I am looping through the passed collection of Objects. This seems to be very time consuming.
Is there a better way of doing the update/delete?
It's possible to both remove and update multiple documents with a single query.
remove
You need to use a query with a selector using $in, and an array of _id values to match.
With Jongo, you can build the list to match with $in into the query in a couple of different ways
// pass an array of ids
ObjectId[] ids = {id1, id2, id3};
collection.remove("{ _id: { $in: # } }", ids);
// or pass each id separately
collection.remove("{ _id: { $in:[#, #, #] }}", id1, id2, id3);
update
Exact same concept as above using $in to select the objects you want to update, however you also have to set the multi option, so that the update applies to all the documents it matches against, not just the first.
With Jongo this is done like so
ObjectId[] ids = {id1, id2, id3};
collection
.update("{ _id: { $in: # } }", ids)
.multi()
.with({ $set: { foo: "bar" });