We Keep Coding

Elastic termsQuery not giving expected result

I have an index where each of my objects has status field which can have some predefined values. I want to fetch all of them which has statusINITIATED, UPDATED, DELETED, any match with these and hence created this query by java which I got printing on console, using Querybuilder and nativeSearchQuery, executing by ElasticsearchOperations:
{
"bool" : {
"must" : [
{
"terms" : {
"status" : [
"INITIATED",
"UPDATED",
"DELETED"
],
"boost" : 1.0
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
I have data in my index with 'INITIATED' status but not getting anyone with status mentioned in the query. How to fix this query, please?
If you need anything, please let me know.
Update: code added
NativeSearchQueryBuilder nativeSearchQueryBuilder=new NativeSearchQueryBuilder();
QueryBuildersingleQb=QueryBuilders.boolQuery().must(QueryBuilders.termsQuery("status",statusList));
Pageable pageable = PageRequest.of(0, 1, Sort.by(Defs.START_TIME).ascending());
FieldSortBuilder sort = SortBuilders.fieldSort(Defs.START_TIME).order(SortOrder.ASC);
nativeSearchQueryBuilder.withQuery(singleQb);
nativeSearchQueryBuilder.withSort(sort);
nativeSearchQueryBuilder.withPageable(pageable);
nativeSearchQueryBuilder.withIndices(Defs.SCHEDULED_MEETING_INDEX);
nativeSearchQueryBuilder.withTypes(Defs.SCHEDULED_MEETING_INDEX);
NativeSearchQuery searchQuery = nativeSearchQueryBuilder.build();
List<ScheduledMeetingEntity> scheduledList=elasticsearchTemplate.queryForList(searchQuery, ScheduledMeetingEntity.class);
Update 2: sample data:
I got this from kibana query on this index:
"hits" : [
{
"_index" : "index_name",
"_type" : "type_name",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"createTime" : "2021-03-03T13:09:59.198",
"createTimeInMs" : 1614755399198,
"createdBy" : "user1#domain.com",
"editTime" : "2021-03-03T13:09:59.198",
"editTimeInMs" : 1614755399198,
"editedBy" : "user1#domain.com",
"versionId" : 1,
"id" : "1",
"meetingId" : "47",
"userId" : "129",
"username" : "user1#domain.com",
"recipient" : [
"user1#domain.com"
],
"subject" : "subject",
"body" : "hi there",
"startTime" : "2021-03-04T07:26:00.000",
"endTime" : "2021-03-04T07:30:00.000",
"meetingName" : "name123",
"meetingPlace" : "placeName",
"description" : "sfsafsdafsdf",
"projectName" : "",
"status" : "INITIATED",
"failTry" : 0
}
}
]

Confirm your mapping:
GET /yourIndexName/_mapping
And see if it is valid
Your mapping needs to have keyword for TermsQuery to work.
{
"status": {
"type" "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
ES can automatically do the mapping for you (without you having to do it yourself) when you first push a document. However you probably have finer control if you do the mapping yourself.
Either way, you need to have keyword defined for your status field.
=====================
Alternative Solution: (Case Insensitive)
If you have a Field named (status), and the values you want to search for are (INITIATED or UPDATED, or DELETED).
Then you can do it like this:
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery()
.must(createStringSearchQuery());
public QueryBuilder createStringSearchQuery(){
QueryStringQueryBuilder queryBuilder = QueryBuilders.queryStringQuery(" INITIATED OR UPDATED OR DELETED ");
queryBuilder.defaultField("status");
return queryBuilder;
}
Printing the QueryBuilder:
{
"query_string" : {
"query" : "INITIATED OR UPDATED OR DELETED",
"default_field" : "status",
"fields" : [ ],
"type" : "best_fields",
"default_operator" : "or",
"max_determinized_states" : 10000,
"enable_position_increments" : true,
"fuzziness" : "AUTO",
"fuzzy_prefix_length" : 0,
"fuzzy_max_expansions" : 50,
"phrase_slop" : 0,
"escape" : false,
"auto_generate_synonyms_phrase_query" : true,
"fuzzy_transpositions" : true,
"boost" : 1.0
}
}

java how get last date (or max date) from mongodb collection where userName=currentUserName

I have a collection of logins with the following format:
{ "_id" : ObjectId("541d9aee784269c2e9f3c092"), "userName" : "dani",
"loginDate" : ISODate("2020-01-22T21:08:28.994Z")},
{ "_id" : ObjectId("541d9aee7842sdfsdfsdddsf"), "userName" : "bar",
"loginDate" : ISODate("2020-01-20T21:02:33.994Z")},
{ "_id" : ObjectId("541d9aee784dfgsdfgdfgdfg"), "userName" : "yacuv",
"loginDate" : ISODate("2020-01-17T21:01:11.994Z")},
{ "_id" : ObjectId("541d9aee784dfgdfgdfgdfgd"), "userName" : "ran",
"loginDate" : ISODate("2020-01-15T21:02:33.994Z")},
{ "_id" : ObjectId("541d9aee784269cvvchgfghh"), "userName" : "bar",
"loginDate" : ISODate("2020-01-12T21:02:54.994Z")}}
How do I get the last entry date per user?

Assuming the collection is called "yoval":
db.yoval.aggregate([{$group: {"_id" : "$userName", maxDate: {$max: "$loginDate"}}}])
produces:
{ "_id" : "yacuv", "maxDate" : ISODate("2020-01-17T21:01:11.994Z") }
{ "_id" : "ran", "maxDate" : ISODate("2020-01-15T21:02:33.994Z") }
{ "_id" : "bar", "maxDate" : ISODate("2020-01-20T21:02:33.994Z") }
{ "_id" : "dani", "maxDate" : ISODate("2020-01-22T21:08:28.994Z") }

Mongo DB Query in Arraylist

In my Java Play framework application, I want to store the ArrayList values in mongoDB.
{
"_id" : ObjectId("5832f29bd4c6721e4e8ba4a7"),
"_class" : "com.netas.innovation.entity.Idea",
"title" : "fsaf",
"desc" : "adgg",
"keyWords" : "dgds",
"createdDate" : ISODate("2016-11-21T13:11:55.823Z"),
"checkbox1" : false,
"checkbox2" : false,
"checkbox3" : false,
"scopeOfIdea" : "Herkes",
"template" : false,
"creatorUser" : {
"$ref" : "user",
"$id" : ObjectId("5832f27dd4c6721e4e8ba4a5")
},
"owners" : [
{
"$ref" : "user",
"$id" : ObjectId("5832f27dd4c6721e4e8ba4a5")
}
],
"answer" : {
"$ref" : "answer",
"$id" : ObjectId("5832f29bd4c6721e4e8ba4a6")
},
"fileList" : []
}
i want to search in owners.
My query doesnt work
if(owners != null && !owners.isEmpty()) {
for(int i=0; i<owners.size(); i++) {
criteriaList.add(new **Criteria().elemMatch(Criteria.where("owners.$id").is(owners.get(i).getId())));**
}
}
How can i fix?
i can search by owners.
owners can be two people or three
Thanks for answers

"owners" : [
{
"$ref" : "user(list)",
"$id" : ObjectId("5832ecdb0deb78cc88392c83")
}
$ref : ozgurk,volkany ...

http://prntscr.com/ddz3c9 this example for output... title desc vs vsdate and owners

I'm not a mongodb expert but, I guess you should do somethink like:
List<Criteria> criteriaList = new ArrayList<Criteria>();
...
List idList = new ArrayList();
for (int i = 0; i < owners.size(); i++) {
idList.add(owners.get(i).getId());
}
criteriaList.add(new Criteria().where("owners.$id").in(idList));
...
If you add each owner=owners.get(i).getId() criteria to the list one by one and finally combine all criterias with AND operation you will not get your desired output.
I guess you use Spring data. I tried to write following mongodb query:
db.getCollection('idea').find(
{
"owners.$id": {
"$in" : [
ObjectId("58451c5f13c97bdde9950641"),
ObjectId("28451c5f13c97bdde9950642")
]
}
}
)

Embedded document index for date is not used

I'm newbie using MongoDB and I have a collection for this type of document:
{
"_id" : {
"coordinate" : {
"latitude" : 532144,
"longitude" : -33333
},
"margin" : "N"
},
"prices" : [
{
"type" : "GAS_95",
"price" : 1370,
"date" : ISODate("2014-05-03T18:39:13.635Z")
},
{
"type" : "DIESEL_A",
"price" : 1299,
"date" : ISODate("2014-05-03T18:39:13.635Z")
},
{
"type" : "DIESEL_A_NEW",
"price" : 1350,
"date" : ISODate("2014-05-03T18:39:13.635Z")
},
{
"type" : "GAS_98",
"price" : 1470,
"date" : ISODate("2014-05-03T18:39:13.635Z")
}
]
}
I need to retrieve the prices for specific date, so then I run this query:
db.gasStation.aggregate(
{ "$unwind" : "$prices"},
{ "$match" : {
"_id" : {
"coordinate" : {
"latitude" : 532144 ,
"longitude" : -33333} ,
"margin" : "N"
} ,
"prices.date" : {
"$gte" : ISODate("2014-05-02T23:00:00.000Z") ,
"$lte" : ISODate("2014-05-03T22:59:59.999Z")
}
}
});
All works fine, I retrieve the documents but I presume that my can be improved, I tried to create an index for _id and prices.date:
db.gasStation.ensureIndex( {
"_id" : 1,
"prices.date" : 1
} )
After that I try to see if the index is being used in my query with the explain option but is not using any index:
{
"stages" : [
{
"$cursor" : {
"query" : {
},
"plan" : {
"cursor" : "BasicCursor",
"isMultiKey" : false,
"scanAndOrder" : false,
"allPlans" : [
{
"cursor" : "BasicCursor",
"isMultiKey" : false,
"scanAndOrder" : false
}
]
}
}
},
{
"$unwind" : "$prices"
},
{
"$match" : {
"_id" : {
"coordinate" : {
"latitude" : 532144,
"longitude" : -33333
},
"margin" : "N"
},
"prices.date" : {
"$gte" : ISODate("2014-05-02T23:00:00Z"),
"$lte" : ISODate("2014-05-03T22:59:59.999Z")
}
}
}
],
"ok" : 1
}
is there any reason that my query is not suitable to use the index? I read on MongoDB documentation that the only pipeline that is not using indexes is $group but I'm not using that feature.

Try re-arranging your aggegration pipeline operators. For instance, this query:
db.gasStation.aggregate([
{ "$match" : {
"_id" : {
"coordinate" : {
"latitude" : 532144 ,
"longitude" : -33333} ,
"margin" : "N"
}
}},
{ "$unwind" : "$prices"},
{ "$match" : {
"prices.date" : {
"$gte" : ISODate("2014-05-02T23:00:00.000Z") ,
"$lte" : ISODate("2014-05-03T22:59:59.999Z")
}
}}
], {explain:true});
produces this output, which does show some index usage now:
{
"stages" : [
{
"$cursor" : {
"query" : {
"_id" : {
"coordinate" : {
"latitude" : 532144,
"longitude" : -33333
},
"margin" : "N"
}
},
"plan" : {
"cursor" : "IDCursor",
"indexBounds" : {
"_id" : [
[
{
"coordinate" : {
"latitude" : 532144,
"longitude" : -33333
},
"margin" : "N"
},
{
"coordinate" : {
"latitude" : 532144,
"longitude" : -33333
},
"margin" : "N"
}
]
]
}
}
}
},
{
"$unwind" : "$prices"
},
{
"$match" : {
"prices.date" : {
"$gte" : ISODate("2014-05-02T23:00:00Z"),
"$lte" : ISODate("2014-05-03T22:59:59.999Z")
}
}
}
],
"ok" : 1
The point is to try to get pipeline operators like $match and $sort up front at the beginning of the pipeline to use indexes to limit how much data is accessed and passed on into the rest of the aggregation. There is more that you can do with the above example to improve performance but this should give you a good idea of how to approach it.

Im going to quote the docs on this:
The $match and $sort pipeline operators can take advantage of an index
when they occur at the beginning of the pipeline.
source: http://docs.mongodb.org/manual/core/aggregation-pipeline/#pipeline-operators-and-indexes
You don't have a $match or $sort at the beginning of the pipeline, you have the $unwind operation. Thus, indexes are useless here.
Edit - detailed explanation:
Still, it is possible to move part of the matching condition to the beginning of the pipeline so that an index will be used.
db.gasStation.aggregate([
{ "$match" : {
"_id" : {
"coordinate" : {
"latitude" : 532144 ,
"longitude" : -33333} ,
"margin" : "N"
}
}},
{ "$project": { "prices" : 1, "_id" : 0 } },
{ "$unwind" : "$prices"},
{ "$match" : {
"prices.date" : {
"$gte" : ISODate("2014-05-02T23:00:00.000Z") ,
"$lte" : ISODate("2014-05-03T22:59:59.999Z")
}
}}
],{explain:true});
However, here this index is unnecessary:
{"_id":1, "prices.date":1}
Why? Because the $match at the beginning of the pipeline only filters by the _id. In mongodb a document's _id is automatically indexed, and that's the index that will be used on this case.
Also, you can further optimize your query by removing unnecessary fields using the $project operator. If you don't need a field, remove it as soon as possible.

How to retrieve a document by its own sub document or array?

I have such structure of document:
{
"_id" : "4e76fd1e927e1c9127d1d2e8",
"name" : "***",
"embedPhoneList" : [
{
"type" : "家庭",
"number" : "00000000000"
},
{
"type" : "手机",
"number" : "00000000000"
}
],
"embedAddrList" : [
{
"type" : "家庭",
"addr" : "山东省诸城市***"
},
{
"type" : "工作",
"addr" : "深圳市南山区***"
}
],
"embedEmailList" : [
{
"email" : "********#gmail.com"
},
{
"email" : "********#gmail.com"
},
{
"email" : "********#gmail.com"
},
{
"email" : "********#gmail.com"
}
]
}
What I wan't to do is find the document by it's sub document,such as email in embedEmailList field.
Or if I have structure like this
{
"_id" : "4e76fd1e927e1c9127d1d2e8",
"name" : "***",
"embedEmailList" : [
"123#gmail.com" ,
"********#gmail.com" ,
]
}
the embedEmailList is array,how to find if there is 123#gmail.com?
Thanks.

To search for a specific value in an array, mongodb supports this syntax:
db.your_collection.find({embedEmailList : "foo#bar.com"});
See here for more information.
To search for a value in an embedded object, it supports this syntax:
db.your_collection.find({"embedEmailList.email" : "foo#bar.com"});
See here for more information.