We Keep Coding

Creating Query like "A is "b" and C is '' " (Spring Boot/Spring Data w/Mongo)

I'm making a simple application with Spring Boot (2.3.4) using MongoDB with Spring Data for MongoDB. I usually create queries for the app using the #Query annotation and it works very fine. But for an Aggregation I want to use, I built a query with the Criteria class. The criteria I need is like
where("primary").is(value).and("secondary").is("").
I need all entries where primary is equal to 'value' and secondary is empty. The query entered in MOngoDB Compass
{ $and: [ { primary: 'value' }, { secondary: ''} ] }
works as expected, but when I try to use the Criteria with Spring, it looks like the and part with the secondary is completely dropped. I get any results with 'value' in primary and with anything in secondary. This means an empty fields or anything else. Replacing the .is("") part with .regex("^$") didn't help.
This looks pretty basic to me, so what am I missing here? I don't want to replace the empty secondary with an "empty flag", because that feels wrong.
Update:
This is the code in question
Criteria crit;
if(!primary.equals(secondary)) {
crit = where("primary").is(primary.name()).and("secondary").is(secondary.name());
} else {
crit = where("primary").is(primary.name()).and("secondary").is("");
}
MatchOperation matchStage = Aggregation.match(crit);
GroupOperation groupStage = Aggregation.group("grouping").count().as("sum");
SortOperation sortStage = new SortOperation(Sort.by("_id"));
Aggregation aggregation = Aggregation.newAggregation(matchStage, groupStage, sortStage);
AggregationResults<TypePerGroup> results = mongoTemplate.aggregate(aggregation, "dataCollection", TypePerGroup.class);

This works with mongodb - Not sure what abstraction compass adds. Both queries don't generate the same json query but they are equal.
Generated query
where("primary").is(value).and("secondary").is("").
is
{"primary":value, "secondary": ""}
Perhaps compass doesn't like this variant ?
Anyways to generate query similar to what you have you input in compass you can use below code
Criteria criteria = new Criteria();
criteria.andOperator(Criteria.where("primary").is("hello"), Criteria.where("secondary").is(""));
Query query = Query.query(criteria);

You are not missing anything. where("primary").is(value).and("secondary").is("") is correct and is functionally equivalent to { $and: [ { primary: 'value' }, { secondary: ''} ] }. You should turn on debug level logging for MongoTemplate to see the generated query.

A have connected to Atlas using Mongo DBCompas and added 4 records to collection:
[{
"primary": "A",
"secondary": "A"
},{
"primary": "A",
"secondary": ""
},{
"primary": "B",
"secondary": "B"
},{
"primary": "B",
"secondary": ""
}]
both queries:
List<Data> firstResults = mongoTemplate.query(Data.class)
.matching(Query.query(Criteria.where("primary").is("B").and("secondary").is("")))
.all();
System.out.println(firstResults);
Criteria criteria = new Criteria();
criteria.andOperator(Criteria.where("primary").is("B"), Criteria.where("secondary").is(""));
List<Data> secondResults = mongoTemplate.query(Data.class)
.matching(Query.query(criteria))
.all();
System.out.println(secondResults);
gave the same result:
[Data{primary='B', secondary=''}]
Campfire can you please provide example of your code to analyze?

How to add Bucket Sort to Query Aggregation

I have a ElasticSearch Query that is working well (curl), is my first Query,
First I am filtering by Organization (Multitenancy), then group by Customer, Finally sum the amount of the sales but I only want to have the 3 best customers.
My question is.. How to build the aggregation with the AggregationBuilders to get "bucket_sort" statement. I got the sales grouping by customer with Java API.
Elastic Query is:
curl -X POST 'http://localhost:9200/sales/sale/_search?pretty' -H 'Content-Type: application/json' -d '
{
"aggs": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"term": {
"organization_id": "15"
}
}
]
}
},
"aggs": {
"by_customer": {
"terms": {
"field": "customer_id"
},
"aggs": {
"sum_total" : {
"sum": {
"field": "amount"
}
},
"total_total_sort": {
"bucket_sort": {
"sort": [
{"sum_total": {"order": "desc"}}
],
"size": 3
}
}
}
}
}
}
}
}'
My Java Code:
#Test
public void queryBestCustomers() throws UnknownHostException {
Client client = Query.client();
AggregationBuilder sum = AggregationBuilders.sum("sum_total").field("amount");
AggregationBuilder groupBy = AggregationBuilders.terms("by_customer").field("customer_id").subAggregation(sum);
AggregationBuilder aggregation =
AggregationBuilders
.filters("filtered",
new FiltersAggregator.KeyedFilter("must", QueryBuilders.termQuery("organization_id", "15"))).subAggregation(groupBy);
SearchRequestBuilder requestBuilder = client.prepareSearch("sales")
.setTypes("sale")
.addAggregation(aggregation);
SearchResponse response = requestBuilder.execute().actionGet();
}

I hope I got your question right.
Try adding "order" to your groupBy agg:
AggregationBuilder groupBy = AggregationBuilders.terms("by_customer").field("customer_id").subAggregation(sum).order(Terms.Order.aggregation("sum_total", false));
One more thing, if you want the top 3 clients than your .size(3) should be set on groupBy agg as well and not on sorting. like that:
AggregationBuilder groupBy = AggregationBuilders.terms("by_customer").field("customer_id").subAggregation(sum).order(Terms.Order.aggregation("sum_total", false)).size(3);

As another answer mentioned, "order" does work for your use case.
However there are other use cases where one may want to use bucket_sort. For example if someone wanted to page through the aggregation buckets.
As bucket_sort is a pipeline aggregation you cannot use the AggregationBuilders to instantiate it. Instead you'll need to use the PipelineAggregatorBuilders.
You can read more information about the bucket sort/pipeline aggregation here.
The ".from(50)" in the following code is an example of how you can page through the buckets. This causes the items in the bucket to start from item 50 if applicable. Not including "from" is the equivalent of ".from(0)"
BucketSortPipelineAggregationBuilder paging = PipelineAggregatorBuilders.bucketSort(
"paging", List.of(new FieldSortBuilder("sum_total").order(SortOrder.DESC))).from(50).size(10);
AggregationBuilders.terms("by_customer").field("customer_id").subAggregation(sum).subAggregation(paging);

MongoDB get number of matches by $in command

Definition
I'm creating searching application and mongo db is used to store searching information. This is example dataset of collection "Resource".
{
_id:"5b3b84e02360a26f9a9ae96e",
name:"Advanced Java",
keywords:[
"java", "thread", "state", "public", "void"
]
},
{
_id:"5b3b84e02360a26f9a9ae96f",
name:"Java In Simple",
keywords:[
"java", "runnable", "thread", "sleep", "array"
]
}
This contains name of books and most frequent words (in keywords array) of each. I'm using spring framework with mongo template. If I run below code,
MongoOperations mongoOperations = new MongoTemplate(new MongoClient("127.0.0.1", 27017), "ResourceDB");
Query query = new Query(where("keywords").in("java", "thread", "sleep"));
List<Resource> resources = mongoOperations.find(query, Resource.class);
It results both "Advanced Java" and "Java In Simple" and its ok.
Problem
But in my case, I need them in order. Because "Java In Simple" match 3 words and "Advanced Java" matches only 2 words. So possibility of most relevant book should be "Java In Simple" and it should be in first.
Expecting Order
Java In Simple
Advanced Java
Is it possible to get result in matching order. Or is there any way to get number of matches for each item. For example If is search for ("java", "thread", "sleep"), I'm expecting output like below.
Advanced Java - 2 matches
Java in Simple - 3 matches
Any help appreciated.

$in doesn't match 3 or 2 items. It stops after first match. You need to use aggregation pipeline to calculate intersection of keywords and the array from the query and order by size of the result:
db.collection.aggregate([
{ $addFields: {
matchedTags: { $size: {
$setIntersection: [ "$keywords", [ "java", "thread", "sleep" ] ]
} }
} },
{ $match: { matchedTags: { $gt: 0 } } },
{ $sort: { matchedTags: -1 } }
])

This is for someone who looking to run #Alex Blex's query in java. It looks like mongo template does not have implementation for intersection. So I have done it using mongoDB java client.
List<String> keywords = Arrays.asList("java", "thread", "sleep");
BasicDBList intersectionList = new BasicDBList();
intersectionList.add("$keywords");
intersectionList.add(keywords);
AggregateIterable<Document> aggregate = new MongoClient("127.0.0.1", 27017).getDatabase("ResourceDB").getCollection("Resource").aggregate(
Arrays.asList(
new BasicDBObject("$addFields",
new BasicDBObject("matchedTags",
new BasicDBObject("$size",
new BasicDBObject("$setIntersection", intersectionList)))),
new BasicDBObject("$match",
new BasicDBObject("matchedTags",
new BasicDBObject("$gt", 0))),
new BasicDBObject("$sort",
new BasicDBObject("matchedTags", -1))
)
);
MongoCursor<Document> iterator = aggregate.iterator();
while (iterator.hasNext()){
Document document = iterator.next();
System.out.println(document.get("name")+" - "+document.get("matchedTags"));
}

java : Case insensitive search in Elasticsearch

I'm trying to find out the documents in the index regardless of whether if it's field values are lowercase or uppercase in the index.
This is the index structure, I have designed with the custom analyzer. I'm new to analyzers and I might be wrong. This is how it looks :
POST arempris/emptagnames
{
"settings": {
"analyzer": {
"lowercase_keyword": {
"type": "custom",
"tokenizer": "keyword",
"filter": "lowercase"
}
}
},
"mappings" : {
"emptags":{
"properties": {
"employeeid": {
"type":"integer"
},
"tagName": {
"type": "text",
"fielddata": true,
"analyzer": "lowercase_keyword"
}
}
}
}
}
In the java back-end, I'm using BoolQueryBuilder to find tagnames using employeeids first. This is what I've coded to fetch the values :
BoolQueryBuilder query = new BoolQueryBuilder();
query.must(new WildcardQueryBuilder("tagName", "*June*"));
query.must(new TermQueryBuilder("employeeid", 358));
SearchResponse response12 = esclient.prepareSearch(index).setTypes("emptagnames")
.setQuery(query)
.execute().actionGet();
SearchHit[] hits2 = response12.getHits().getHits();
System.out.println(hits2.length);
for (SearchHit hit : hits2) {
Map map = hit.getSource();
System.out.println((String) map.get("tagName"));
}
It works fine when I specify the tag to be searched as "june" in lowercase, but when I specify it as "June" in the WildCardQueryBuilder with an uppercase for an alphabet, I'm not getting any match.
Let me know where have I committed the mistake. Would greatly appreciate your help and thanks in advance.

There are two type of queries in elasticsearch
Term level queries -> in which exact term is searched. https://www.elastic.co/guide/en/elasticsearch/reference/current/term-level-queries.html
Full text queries -> which first analyzes the query term and then search it. https://www.elastic.co/guide/en/elasticsearch/reference/current/full-text-queries.html
The rules for full text queries is
First it looks for search_analyzer in query
If not mentioned then it uses index time analyzer for that field for searching.
So in this case you need to change your query to this
BoolQueryBuilder query = new BoolQueryBuilder();
query.must(new QueryStringQueryBuilder("tagName:*June*"));
query.must(new TermQueryBuilder("employeeid", 358));
SearchResponse response12 = esclient.prepareSearch(index).setTypes("emptagnames")
.setQuery(query)
.execute().actionGet();
SearchHit[] hits2 = response12.getHits().getHits();
System.out.println(hits2.length);
for (SearchHit hit : hits2) {
Map map = hit.getSource();
System.out.println((String) map.get("tagName"));
}

Case insensitive sorting in MongoDB

How can I sort a MongoDB collection by a given field, case-insensitively? By default, I get A-Z before a-z.

Update:
As of now mongodb have case insensitive indexes:
Users.find({})
.collation({locale: "en" })
.sort({name: 1})
.exec()
.then(...)
shell:
db.getCollection('users')
.find({})
.collation({'locale':'en'})
.sort({'firstName':1})
Update: This answer is out of date, 3.4 will have case insensitive indexes. Look to the JIRA for more information https://jira.mongodb.org/browse/SERVER-90
Unfortunately MongoDB does not yet have case insensitive indexes: https://jira.mongodb.org/browse/SERVER-90 and the task has been pushed back.
This means the only way to sort case insensitive currently is to actually create a specific "lower cased" field, copying the value (lower cased of course) of the sort field in question and sorting on that instead.

Sorting does work like that in MongoDB but you can do this on the fly with aggregate:
Take the following data:
{ "field" : "BBB" }
{ "field" : "aaa" }
{ "field" : "AAA" }
So with the following statement:
db.collection.aggregate([
{ "$project": {
"field": 1,
"insensitive": { "$toLower": "$field" }
}},
{ "$sort": { "insensitive": 1 } }
])
Would produce results like:
{
"field" : "aaa",
"insensitive" : "aaa"
},
{
"field" : "AAA",
"insensitive" : "aaa"
},
{
"field" : "BBB",
"insensitive" : "bbb"
}
The actual order of insertion would be maintained for any values resulting in the same key when converted.

This has been an issue for quite a long time on MongoDB JIRA, but it is solved now. Take a look at this release notes for detailed documentation. You should use collation.
User.find()
.collation({locale: "en" }) //or whatever collation you want
.sort({name:1})
.exec(function(err, users) {
// use your case insensitive sorted results
});

Adding the code .collation({'locale':'en'}) helped to solve my issue.

As of now (mongodb 4), you can do the following:
mongo shell:
db.getCollection('users')
.find({})
.collation({'locale':'en'})
.sort({'firstName':1});
mongoose:
Users.find({})
.collation({locale: "en" })
.sort({name: 1})
.exec()
.then(...)
Here are supported languages and locales by mongodb.

In Mongoose:-
Customer.find()
.collation({locale: "en" })
.sort({comapany: 1})

Here it is in Java. I mixed no-args and first key-val variants of BasicDBObject just for variety
DBCollection coll = db.getCollection("foo");
List<DBObject> pipe = new ArrayList<DBObject>();
DBObject prjflds = new BasicDBObject();
prjflds.put("field", 1);
prjflds.put("insensitive", new BasicDBObject("$toLower", "$field"));
DBObject project = new BasicDBObject();
project.put("$project", prjflds);
pipe.add(project);
DBObject sort = new BasicDBObject();
sort.put("$sort", new BasicDBObject("insensitive", 1));
pipe.add(sort);
AggregationOutput agg = coll.aggregate(pipe);
for (DBObject result : agg.results()) {
System.out.println(result);
}

If you want to sort and return all data in a document, you can add document: "$$ROOT"
db.collection.aggregate([
{
$project: {
field: 1,
insensitive: { $toLower: "$field" },
document: "$$ROOT"
}
},
{ $sort: { insensitive: 1 } }
]).toArray()

Tried all the above and answers
Consolidating the result
Answer-1:
db.collection.aggregate([
{ "$project": {
"field": 1,
"insensitive": { "$toLower": "$field" }
}},
{ "$sort": { "insensitive": 1 } } ])
Aggregate query converts the field into lower, So performance is low for large data.
Answer-2:
db.collection.find({}).collation({locale: "en"}).sort({"name":1})
By default mongo follows uft-8 encoding(Z has high piriority then a) rules ,So overriding with language-specific rules.
Its fast compare to above query
Look into an official document to customize rules
https://docs.mongodb.com/manual/reference/collation/

We solve this problem with the help of .sort function in JavaScript array
Here is the code
function foo() {
let results = collections.find({
_id: _id
}, {
fields: {
'username': 1,
}
}).fetch();
results.sort((a, b)=>{
var nameA = a.username.toUpperCase();
var nameB = b.username.toUpperCase();
if (nameA nameB) {
return 1;
}
return 0;
});
return results;
}