We Keep Coding

Spring data elasticsearch bulk index on Percolator

Using spring data elasticsearch I want to do the following (bulk Indexing)
Step 1
PUT surname
{
"settings" : {
"index" : {
"number_of_shards" : 1,
"number_of_replicas" : 0
}
},
"mappings": {
"properties": {
"message": {
"type": "text"
},
"query": {
"type": "percolator"
}
}
}
}
Step 2:
PUT surname/_bulk
{ "index" : { "_index" : "surname", "_type" : "_doc", "_id" : "1" } }
{ "query" : { "match_phrase" : { "message" : "Aal" } }}
{ "index" : { "_index" : "surname", "_type" : "_doc", "_id" : "2" } }
{ "query" : { "match_phrase" : { "message" : "Aalbers" } }}
Step 1 is done with the help of - https://stackoverflow.com/a/67724048/4068218
For Step 2 I tried
IndexQuery and UpdateQuery
Query query = new CriteriaQuery(new Criteria("match_phrase").subCriteria(new Criteria("message").is("Aal")));
UpdateQuery updateQuery = builder(query).build();
elasticsearchOperations.bulkUpdate(ImmutableList.of(updateQuery),IndexCoordinates.of(indexName.get()));
but both do not work. If I use UpdateQuery I get Validation Failed: 1: id is missing;2: script or doc is missing.
If I use IndexQuery I get malformed query.
How do I go about Step 2 in spring data elasticsearch? Any lead is much appreciated.

Below code worked for me. Sharing here as it might help someone
IndexQuery indexQuery = new IndexQueryBuilder()
.withId("12")
.withSource(String.format("{ \"query\" : { \"match_phrase\" : { \"message\" : \"%s\" } }}", "Aal"))
.build();
elasticsearchOperations.bulkIndex(ImmutableList.of(indexQuery),IndexCoordinates.of(indexName.get()));

Elastic termsQuery not giving expected result

I have an index where each of my objects has status field which can have some predefined values. I want to fetch all of them which has statusINITIATED, UPDATED, DELETED, any match with these and hence created this query by java which I got printing on console, using Querybuilder and nativeSearchQuery, executing by ElasticsearchOperations:
{
"bool" : {
"must" : [
{
"terms" : {
"status" : [
"INITIATED",
"UPDATED",
"DELETED"
],
"boost" : 1.0
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
I have data in my index with 'INITIATED' status but not getting anyone with status mentioned in the query. How to fix this query, please?
If you need anything, please let me know.
Update: code added
NativeSearchQueryBuilder nativeSearchQueryBuilder=new NativeSearchQueryBuilder();
QueryBuildersingleQb=QueryBuilders.boolQuery().must(QueryBuilders.termsQuery("status",statusList));
Pageable pageable = PageRequest.of(0, 1, Sort.by(Defs.START_TIME).ascending());
FieldSortBuilder sort = SortBuilders.fieldSort(Defs.START_TIME).order(SortOrder.ASC);
nativeSearchQueryBuilder.withQuery(singleQb);
nativeSearchQueryBuilder.withSort(sort);
nativeSearchQueryBuilder.withPageable(pageable);
nativeSearchQueryBuilder.withIndices(Defs.SCHEDULED_MEETING_INDEX);
nativeSearchQueryBuilder.withTypes(Defs.SCHEDULED_MEETING_INDEX);
NativeSearchQuery searchQuery = nativeSearchQueryBuilder.build();
List<ScheduledMeetingEntity> scheduledList=elasticsearchTemplate.queryForList(searchQuery, ScheduledMeetingEntity.class);
Update 2: sample data:
I got this from kibana query on this index:
"hits" : [
{
"_index" : "index_name",
"_type" : "type_name",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"createTime" : "2021-03-03T13:09:59.198",
"createTimeInMs" : 1614755399198,
"createdBy" : "user1#domain.com",
"editTime" : "2021-03-03T13:09:59.198",
"editTimeInMs" : 1614755399198,
"editedBy" : "user1#domain.com",
"versionId" : 1,
"id" : "1",
"meetingId" : "47",
"userId" : "129",
"username" : "user1#domain.com",
"recipient" : [
"user1#domain.com"
],
"subject" : "subject",
"body" : "hi there",
"startTime" : "2021-03-04T07:26:00.000",
"endTime" : "2021-03-04T07:30:00.000",
"meetingName" : "name123",
"meetingPlace" : "placeName",
"description" : "sfsafsdafsdf",
"projectName" : "",
"status" : "INITIATED",
"failTry" : 0
}
}
]

Confirm your mapping:
GET /yourIndexName/_mapping
And see if it is valid
Your mapping needs to have keyword for TermsQuery to work.
{
"status": {
"type" "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
ES can automatically do the mapping for you (without you having to do it yourself) when you first push a document. However you probably have finer control if you do the mapping yourself.
Either way, you need to have keyword defined for your status field.
=====================
Alternative Solution: (Case Insensitive)
If you have a Field named (status), and the values you want to search for are (INITIATED or UPDATED, or DELETED).
Then you can do it like this:
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery()
.must(createStringSearchQuery());
public QueryBuilder createStringSearchQuery(){
QueryStringQueryBuilder queryBuilder = QueryBuilders.queryStringQuery(" INITIATED OR UPDATED OR DELETED ");
queryBuilder.defaultField("status");
return queryBuilder;
}
Printing the QueryBuilder:
{
"query_string" : {
"query" : "INITIATED OR UPDATED OR DELETED",
"default_field" : "status",
"fields" : [ ],
"type" : "best_fields",
"default_operator" : "or",
"max_determinized_states" : 10000,
"enable_position_increments" : true,
"fuzziness" : "AUTO",
"fuzzy_prefix_length" : 0,
"fuzzy_max_expansions" : 50,
"phrase_slop" : 0,
"escape" : false,
"auto_generate_synonyms_phrase_query" : true,
"fuzzy_transpositions" : true,
"boost" : 1.0
}
}

how to give $toObjectId in Java Mongo Projection Operation

I am new bee to mongo. Below is the Aggregation operation I am doing in mongodb shell . But In my java ProjectionAggregation , am not able to give the $toObjectId. Please correct me what am I missing.
db shell query
db.getCollection('UserData').aggregate([
{
$project : {
"username" : "$username",
"beneficiaries" : "$beneficiaries"
}
},
{
$unwind : {
path : "$beneficiaries",
preserveNullAndEmptyArrays: true
}
},
{
$project : {
"username" : "$username",
"beneficiaries" : "$beneficiaries",
---- dont know how to give $toObjectId in java ProjectionOperation .
"beneficiaryStudId" : { $toObjectId : "$beneficiaries.studentId" }
}
},
{
$lookup:
{
from: "StudentProfileData",
localField: "beneficiaryStudId",
foreignField: "_id",
as: "studProfile"
}
}
])
Java code Projection Operation
ProjectionOperation projectUserAndBeneficiaries = Aggregation.project()
.andExpression("username").as("username")
.andExpression("beneficiaries").as("beneficiaries");
ProjectionOperation projectUserAndOtherDetails = Aggregation.project()
.andExpression("username").as("username")
.andExpression("beneficiaries").as("beneficiaries")
---- How to give $toObjectId in projection operation .andExpression("beneficiaries.studentId").as("beneficiaryStudId");
LookupOperation lookupOperation = LookupOperation.newLookup().
from("StudentProfileData").
localField("beneficiaryStudId").
foreignField("_id").
as("studProfile");
Aggregation agg = Aggregation.newAggregation(projectUserAndBeneficiaries, unwindBeneficiars,
projectUserAndOtherDetails
,lookupOperation);
AggregationResults<UserAndStudentData> output
= mongotemplate.aggregate(agg, "UserData", UserAndStudentData.class);
Sample Ouput
Output in db shell
{
"_id" : ObjectId("5d2f08574de2690001c281ac"),
"username" : "ks241#goo.com",
"beneficiaries" : {
"studentId" : "5d2f0e9c3bcf3e0001a7e562",
"mcBeneficiaryId" : "597418",
"enabled" : true
},
"beneficiaryStudId" : ObjectId("5d2f0e9c3bcf3e0001a7e562"),
"studProfile" : [
{
"_id" : ObjectId("5d2f0e9c3bcf3e0001a7e562"),
"lastName" : "Sharma",
"firstName" : "Kapil",
"studentRegisterCustomFieldValues" : [
{
"bcfdValue" : "One",
"bcfdName" : "Year"
}
],
"gender" : "M",
"merchantId" : "38788943"
}
]
}
where as in java
the studProfile array is always empty if I add $toObjectId the above java aggregation projection query and ran it produces the values as same as that db shell .

Spring data doesn't support to few type of methods. This problem may include into it. But we may use this solution.
Aggregation aggregation=newAggregation(
p-> new Document("$project",
new Document()
.append("username","$username"),
.append("beneficiaries","$beneficiaries)
.append("beneficiaryStudId",
new Document("$toObjectId","$beneficiaries.studentId")
)
)
)

There is a way to do this with Spring's ProjectionOperation and ConvertOperators classes:
ProjectionOperation projection = Aggregation.project()
.andInclude("username", "beneficiaries")
.and(ConvertOperators.ToObjectId.toObjectId("$beneficiaries.studentId"))
.as("beneficiaryStudId");

how to add values in a single field in mongodb

i have a following Document in mongodb
{
"_id" : "1999",
"Email" : "mail#example.com",
"FirstName" : "personFirstNmae",
"LastName" : "personLastName",
"UserStatus" : "INACTIVE",
"FollowingItems" : [
{
"FollowingItemUuid" : "g12345",
"FollowingItemUuidType" : "GALLERY"
}
]
}
i want to achive this
{
"_id" : "1999",
"Email" : "mail#example.com",
"FirstName" : "personFirstNmae",
"LastName" : "personLastName",
"UserStatus" : "INACTIVE",
"FollowingItems" : [
{
"FollowingItemUuid" : "g12345",
"FollowingItemUuidType" : "GALLERY"
},
{
"FollowingItemUuid" : "M121",
"FollowingItemUuidType" : "MUSEUM"
}
]
}
here is my code
val q=QueryBuilder.start("_id").is("1999")
var update=collection.update(q.get,new BasicDBObject("$set",new BasicDBObject("FollowingItems.$.FollowingItemUuid","M121").append("FollowingItems.$.FollowingItemUuidType","MUSEUM")))
but it throws following exception
com.mongodb.WriteConcernException: { "serverUsed" : "Localhost:27017" , "ok" : 1 , "n" : 0 , "updatedExisting" : false , "err" : "cannot use the part (FollowingItems of FollowingItems.FollowingItemUuid) to traverse the element ({FollowingItems: [ { FollowingItemUuid: \"g12345\", FollowingItemUuidType: \"GALLERY\" } ]})" , "code" : 16837}
at com.mongodb.CommandResult.getWriteException(CommandResult.java:90)
at com.mongodb.CommandResult.getException(CommandResult.java:79)
at com.mongodb.DBCollectionImpl.translateBulkWriteException(DBCollectionImpl.java:316)
at com.mongodb.DBCollectionImpl.update(DBCollectionImpl.java:274)
at com.mongodb.casbah.MongoCollectionBase$class.update(MongoCollection.scala:882)
at com.mongodb.casbah.MongoCollection.update(MongoCollection.scala:1162)
Please guide me how can i achive my desried result and what i am doing wrong

You need to use the $push operator to use this. This is the MongoDB shell command:
db.data.update({
"_id": "1999"
}, {
"$push": {
"FollowingItems": {
"FollowingItemUuid": "M121",
"FollowingItemUuidType": "MUSEUM"
}
}
})
And this is your equivalent QueryBuilder syntax:
val q=QueryBuilder.start("_id").is("1999")
var update=collection.update(q.get,new BasicDBObject("$push",new BasicDBObject("FollowingItems.$.FollowingItemUuid","M121").append("FollowingItems.$.FollowingItemUuidType","MUSEUM")))

retrieve values from nested json array in mongodb

My mongo collection has entries in the following format
{
"myobj" : {
"objList" : [
{ "location" : "Texas" },
{ "location" : "Houston"},
{ "name":"Sam" }
]
},
"category" : "cat1"
}
{
"myobj" :
{
"objList" : [
{ "location" : "Tennesy" },
{ "location" : "NY"},
{ "location" : "SF" }
]
},
"category" : "cat2"
}
I want to extract the "**category**" where location is "Houston". In case of simple JSON object I have to just pass it as query like:
BasicDBObject place = new BasicDBObject();
place.put("location", "Houston");
But in case of nested JSON I don't know how to pass it as a query and get the appropriate category. ie If I pass my location as"Houston" then it should return it's appropriate category "cat1"...i hope my question is clear now....

Ok, you have your documents:
db.coll1.insert({
"myobj" : {
"objList" : [
{ "location" : "Texas" },
{ "location" : "Houston"},
{ "name":"Sam" }
]
},
"category" : "cat1"
})
and
db.coll1.insert({
"myobj" : {
"objList" : [
{ "location" : "Tennesy" },
{ "location" : "Houston"},
{ "location" : "SF" }
]
},
"category" : "cat1"
})
Now you can find what you want using the dot operator:
db.coll1.find({"myobj.objList.location": "Texas"}).pretty() will return one object which has Texas
db.coll1.find({"myobj.objList.location": "SF"}).pretty() will return one object which has SF
db.coll1.find({"myobj.objList.location": "Houston"}).pretty() will return both objects
And now I hope you will be able to write it in Java. I have never used Java, but based on this question you can do something like this. If it will not work, just look how to use dot operator in java driver for mongo:
DBCursor cursor = coll1.find(new BasicDBObject("myobj.objList.location", "Texas"));
P.S. you told, that you wanted to retrieve category. In such a way, you will need to use a projection db.coll1.find({<the query I provided}, {category: 1, _id: 0})