We Keep Coding

How to sum sizes of arrays in MongoDB Java driver?

I have documents similar to below in my People collection:
{
"_id":{"$oid": XYZ},
"id": {"$numberLong":"1"},
"name":"XYZ",
"friends": [...],
"likes": [...]
}
I want to count sum of sizes of friends and likes array for each of document. In MongoDB I created an aggregation query:
{"$project":
{
id: "$id",
neighbour_count: {$sum: [{$size: "$likes"}, {$size: "$friends"}]}
}
}
and got results:
{
"_id": XYZ,
"id":2,
"neighbour_count":1601
}
Now I want simmilar results in my Java MongoDB driver. I tried to do something with Aggregates.count and Projections.fields, but didn't get proper results.
My current code:
DBCollection peopleCollection = database.getCollection("People");
BasicDBList sum = new BasicDBList();
sum.add(new BasicDBObject("$size", "$likes"));
sum.add(new BasicDBObject("$size", "$friends"));
Iterable<DBObject> output = peopleCollection.aggregate(Arrays.asList(
new BasicDBObject("$project", new BasicDBObject("id","$id").append("$sum", sum))))
.results();
throws error:
Invalid $project :: caused by :: FieldPath field names may not start with '$'.
How to do it in proper way?

Elasticsearch 7.13 - elastic search response with old data after update api

We using elastic 7.13
we are doing periodical update to index using upsert
The sequence of operations
create new index with dynamic mapping all strings mapped as text
"dynamic_templates": [
{
"strings_as_keywords": {
"match_mapping_type": "string",
"mapping": {
"type": "text",
"analyzer": "autocomplete",
"search_analyzer": "search_term_analyzer",
"copy_to": "_all",
"fields": {
"keyword": {
"type": "keyword",
"normalizer": "lowercase_normalizer"
}
}
}
}
}
]
upsert bulk with the attached code (I don't have equivalent with rest)
doing search on specific filed
localhost:9200/mdsearch-vitaly123/_search
{
"query": {
"match": {
"fullyQualifiedName": `value_test`
}
}
}
got 1 result
upsert again now "fullyQualifiedName": "value_test1234" (as in step 2)
do search as in step 3
got 2 results 1 doc with "fullyQualifiedName": "value_test"
and other "fullyQualifiedName": "value_test1234"
snippet below of upsert (step 2):
#Override
public List<BulkItemStatus> updateDocumentBulk(String indexName, List<JsonObject> indexDocuments) throws MDSearchIndexerException {
BulkRequest request = new BulkRequest().setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE);
ofNullable(indexDocuments).orElseThrow(NullPointerException::new)
.forEach(x -> {
var id = x.get("_id").getAsString();
x.remove("_id");
request.add(new UpdateRequest(indexName, id)
.docAsUpsert(true)
.doc(x.toString(), XContentType.JSON)
.retryOnConflict(3)
);
});
BulkResponse bulk = elasticsearchRestClient.bulk(request, RequestOptions.DEFAULT);
return stream(bulk.getItems())
.map(r -> new BulkItemStatus(r.getId(), isSuccess(r), r.getFailureMessage()))
.collect(Collectors.toList());
}
I can search by updated properties.
But the problem is that searches retrieve "updated fields" and previous one as well.
How can I solve it ?
maybe limit somehow the version number to be only 1.
I set setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE) but it didn't helped
Here in picture we can see result
P.S - old and updated data retrieved as well
Suggestions ?
Regards,

What is happening is that the following line must yield null:
var id = x.get("_id").getAsString();
In other words, there is no _id field in the JSON documents you pass in indexDocuments. It is not allowed to have fields with an initial underscore character in the source documents. If it was the case, you'd get the following error:
Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters.
Hence, your update request cannot update any document (since there's no ID to identify the document to update) and will simply insert a new one (i.e. what docAsUpsert does), which is why you're seeing two different documents.

Creating Query like "A is "b" and C is '' " (Spring Boot/Spring Data w/Mongo)

I'm making a simple application with Spring Boot (2.3.4) using MongoDB with Spring Data for MongoDB. I usually create queries for the app using the #Query annotation and it works very fine. But for an Aggregation I want to use, I built a query with the Criteria class. The criteria I need is like
where("primary").is(value).and("secondary").is("").
I need all entries where primary is equal to 'value' and secondary is empty. The query entered in MOngoDB Compass
{ $and: [ { primary: 'value' }, { secondary: ''} ] }
works as expected, but when I try to use the Criteria with Spring, it looks like the and part with the secondary is completely dropped. I get any results with 'value' in primary and with anything in secondary. This means an empty fields or anything else. Replacing the .is("") part with .regex("^$") didn't help.
This looks pretty basic to me, so what am I missing here? I don't want to replace the empty secondary with an "empty flag", because that feels wrong.
Update:
This is the code in question
Criteria crit;
if(!primary.equals(secondary)) {
crit = where("primary").is(primary.name()).and("secondary").is(secondary.name());
} else {
crit = where("primary").is(primary.name()).and("secondary").is("");
}
MatchOperation matchStage = Aggregation.match(crit);
GroupOperation groupStage = Aggregation.group("grouping").count().as("sum");
SortOperation sortStage = new SortOperation(Sort.by("_id"));
Aggregation aggregation = Aggregation.newAggregation(matchStage, groupStage, sortStage);
AggregationResults<TypePerGroup> results = mongoTemplate.aggregate(aggregation, "dataCollection", TypePerGroup.class);

This works with mongodb - Not sure what abstraction compass adds. Both queries don't generate the same json query but they are equal.
Generated query
where("primary").is(value).and("secondary").is("").
is
{"primary":value, "secondary": ""}
Perhaps compass doesn't like this variant ?
Anyways to generate query similar to what you have you input in compass you can use below code
Criteria criteria = new Criteria();
criteria.andOperator(Criteria.where("primary").is("hello"), Criteria.where("secondary").is(""));
Query query = Query.query(criteria);

You are not missing anything. where("primary").is(value).and("secondary").is("") is correct and is functionally equivalent to { $and: [ { primary: 'value' }, { secondary: ''} ] }. You should turn on debug level logging for MongoTemplate to see the generated query.

A have connected to Atlas using Mongo DBCompas and added 4 records to collection:
[{
"primary": "A",
"secondary": "A"
},{
"primary": "A",
"secondary": ""
},{
"primary": "B",
"secondary": "B"
},{
"primary": "B",
"secondary": ""
}]
both queries:
List<Data> firstResults = mongoTemplate.query(Data.class)
.matching(Query.query(Criteria.where("primary").is("B").and("secondary").is("")))
.all();
System.out.println(firstResults);
Criteria criteria = new Criteria();
criteria.andOperator(Criteria.where("primary").is("B"), Criteria.where("secondary").is(""));
List<Data> secondResults = mongoTemplate.query(Data.class)
.matching(Query.query(criteria))
.all();
System.out.println(secondResults);
gave the same result:
[Data{primary='B', secondary=''}]
Campfire can you please provide example of your code to analyze?

Mongodb inner join in java spring using AggregationOperation :Error [The 'cursor' option is required, except for aggregate with the explain argument]

I am excecuting following mongodb query I am new to mongo db ,please tell me what i am doing wrong
db.entityCounter.aggregate([
{
$lookup:
{
from: "fields",
localField: "code",
foreignField: "fieldCode",
as: "fieldsresult"
}
},{
$match:{
$and: [{
"fieldsresult.isVisible":"1"
},{"type":"field"
}]
}
}])
below is java spring code
LookupOperation lookupOperation = LookupOperation.newLookup()
.from("fields")
.localField("code")
.foreignField("fieldCode")
.as("fieldsresult");
AggregationOperation match1 = Aggregation.match(Criteria.where("fieldsresult.isVisible").is("1"));
// AggregationOptions aggregationOptions = Aggregation.newAggregationOptions();
DBObject ob=new BasicDBObject();
((BasicDBObject) ob).put("batchSize",10);
Aggregation aggregation = Aggregation.newAggregation(lookupOperation,match1).withOptions(Aggregation.newAggregationOptions().cursor(ob).build());
long val=0;
try {
AggregationResults<EntityCounter> result = mongoOperations.aggregate(aggregation, Fields.class, EntityCounter.class);
// val= result.getMappedResults();
}catch (Exception e){
e.printStackTrace();
}
but I am getting below error
org.springframework.dao.InvalidDataAccessApiUsageException: Command execution failed: Error [The 'cursor' option is required, except for aggregate with the explain argument], Command = { "aggregate" : "entityCounter" , "pipeline" : [ { "$match" : { "fieldsresult.isVisible" : "1"}} , { "$lookup" : { "from" : "fields" , "localField" : "code" , "foreignField" : "fieldCode" , "as" : "fieldsresult"}}]}; nested exception is com.mongodb.MongoCommandException: Command failed with error 9: 'The 'cursor' option is required, except for aggregate with the explain argument' on server localhost:27017. The full response is { "ok" : 0.0, "errmsg" : "The 'cursor' option is required, except for aggregate with the explain argument", "code" : 9, "codeName" : "FailedToParse" }

The lookup was introduced in mongodb 3.4 please upgrade your dB

Mongodb Morphia aggregation

I'm having trouble creating aggregation in Morphia, the documentation is really not clear. This is the original query:
db.collection('events').aggregate([
{
$match: {
"identifier": {
$in: [
userId1, userId2
]
},
$or: [
{
"info.name": "messageType",
"info.value": "Push",
"timestamp": {
$gte: newDate("2015-04-27T19:53:13.912Z"),
$lte: newDate("2015-08-27T19:53:13.912Z")
}
}
]
}{
$unwind: "$info"
},
{
$match: {
$or: [
{
"info.name": "messageType",
"info.value": "Push"
}
]
}
]);
The only example in their docs was using out and there's some example here but I couldn't make it to work.
I didn't even made it past the first match, here's what I have:
ArrayList<String> ids = new ArrayList<>();
ids.add("199941");
ids.add("199951");
Query<Event> q = ads.getQueryFactory().createQuery(ads);
q.and(q.criteria("identifier").in(ids));
AggregationPipeline pipeline = ads.createAggregation(Event.class).match(q);
Iterator<Event> iterator = pipeline.aggregate(Event.class);
Some help or guidance and how to start with the query or how it works will be great.

You need to create the query for the match() pipeline by breaking your code down into manageable pieces that will be easy to follow. So let's start
with the query to match the identifier field, you have done the great so far. We need to then combine with the $or part of the query.
Carrying on from where you left, create the full query as:
Query<Event> q = ads.getQueryFactory().createQuery(ads);
Criteria[] arrayA = {
q.criteria("info.name").equal("messageType"),
q.criteria("info.value").equal("Push"),
q.field("timestamp").greaterThan(start);
q.field("timestamp").lessThan(end);
};
Criteria[] arrayB = {
q.criteria("info.name").equal("messageType"),
q.criteria("info.value").equal("Push")
};
q.and(
q.criteria("identifier").in(ids),
q.or(arrayA)
);
Query<Event> query = ads.getQueryFactory().createQuery(ads);
query.or(arrayB);
AggregationPipeline pipeline = ads.createAggregation(Event.class)
.match(q)
.unwind("info")
.match(query);
Iterator<Event> iterator = pipeline.aggregate(Event.class);
The above is untested but will guide you somewhere closer home, so make some necessary adjustments where appropriate. For some references, the following SO questions may give you some pointers:
Complex AND-OR query in Morphia
Morphia query with or operator
and of course the AggregationTest.java Github page