We Keep Coding

MongoTemplate upsert property snapshot

I'm using mongo 4.2.15
Here is entry:
{
"keys": {
"country": "US",
"channel": "c999"
},
"counters": {
"sale": 0
},
"increments": null
}
I want to be able to initialize counter set as well as increment counters.sale value and save increment result snapshot to increments property. Something like that:
db.getCollection('counterSets').update(
{ "$and" : [
{ "keys.country" : "US"},
{ "keys.channel" : "c999"}
]
},
{ "$inc" :
{ "counters.sale" : 10
},
"$set" :
{ "keys" :
{ "country" : "US", "channel" : "c999"},
"increments":
{ "3000c058-b8a7-4cff-915b-4979ef9a6ed9": {"counters" : "$counters"} }
}
},
{upsert: true})
The result is:
{
"_id" : ObjectId("61965aba1501d6eb40588ba0"),
"keys" : {
"country" : "US",
"channel" : "c999"
},
"counters" : {
"sale" : 10.0
},
"increments" : {
"3000c058-b8a7-4cff-915b-4979ef9a6ed9" : {
"counters" : "$counters"
}
}
}
Does it possible to do such update which is some how copy increment result from root object counters to child increments.3000c058-b8a7-4cff-915b-4979ef9a6ed9.counters with a single upsert. I want to implement safe inrement. Maybe you can suggest some another design?

In order to use expressions, your $set should be part of aggregation pipeline. So your query should look like
NOTE: I've added square brackets to the update
db.getCollection('counterSets').update(
{ "$and" : [
{ "keys.country" : "US"},
{ "keys.channel" : "c999"}
]
},
[ {"$set": {"counters.sale": {"$sum":["$counters.sale", 10]}}}, {"$set": {"increments.x": "$counters"}}],
{upsert: true})
I haven't found any information about the atomicity of aggregation pipelines, so use this carefully.

Get count of unique ObjectId from array MongoDB

I'm new to working with MongoDb and do not know a lot of things.
I need to write an aggregation request.
Here is the JSON document structure.
{
"_id" : ObjectId("5a72f7a75ef7d430e8c462d2"),
"crawler_id" : ObjectId("5a71cbb746e0fb0007adc6c2"),
"skill" : "stack",
"created_date" : ISODate("2018-02-01T13:19:03.522+0000"),
"modified_date" : ISODate("2018-02-01T13:22:23.078+0000"),
"connects" : [
{
"subskill" : "we’re",
"weight" : NumberInt(1),
"parser_id" : [
ObjectId("5a71d88d5ef7d41964fbec11")
]
},
{
"subskill" : "b1",
"weight" : NumberInt(2),
"parser_id" : [
ObjectId("5a71d88d5ef7d41964fbec11"),
ObjectId("5a71d88d5ef7d41964fbec1b")
]
},
{
"subskill" : "making",
"weight" : NumberInt(2),
"parser_id" : [
ObjectId("5a71d88d5ef7d41964fbec1b"),
ObjectId("5a71d88d5ef7d41964fbec1c")
]
},
{
"subskill" : "delivery",
"weight" : NumberInt(2),
"parser_id" : [
ObjectId("5a71d88d5ef7d41964fbec1c"),
ObjectId("5a71d88d5ef7d41964fbec1e")
]
}
]
}
I need the result return the name of skill and the number of unique parser_id.
In this case, the result should be:
[
{
"skill": "stack",
"quantity": 4
}
]
where "stack" - skill name,
and "quantity" - count of unique parser_id.
ObjectId("5a71d88d5ef7d41964fbec11")
ObjectId("5a71d88d5ef7d41964fbec1b")
ObjectId("5a71d88d5ef7d41964fbec1c")
ObjectId("5a71d88d5ef7d41964fbec1e")
Can some one help me with this request ???

Given the document supplied in your question, this command ...
db.collection.aggregate([
{ $unwind: "$connects" },
// count all occurrences
{ "$group": { "_id": {skill: "$skill", parser_id: "$connects.parser_id"}, "count": { "$sum": 1 } }},
// sum all occurrences and count distinct
{ "$group": { "_id": "$_id.skill", "quantity": { "$sum": 1 } }},
// (optional) rename the '_id' attribute to 'skill'
{ $project: { 'skill': '$_id', 'quantity': 1, _id: 0 } }
])
... will return:
{
"quantity" : 4,
"skill" : "stack"
}
The above command groups by skill and connects.parser_id and then gets a distinct count of those groups.
Your command includes the java tag so I suspect you are looking to execute the same command using the MongoDB Java driver. The code below (using MongoDB Java driver v3.x) will return the same result:
MongoClient mongoClient = ...;
MongoCollection<Document> collection = mongoClient.getDatabase("...").getCollection("...");
List<Document> documents = collection.aggregate(Arrays.asList(
Aggregates.unwind("$connects"),
new Document("$group", new Document("_id", new Document("skill", "$skill").append("parser_id", "$connects.parser_id"))
.append("count", new Document("$sum", 1))),
new Document("$group", new Document("_id", "$_id.skill").append("quantity", new Document("$sum", 1))),
new Document("$project", new Document("skill", "$_id").append("quantity", 1).append("_id", 0))
)).into(new ArrayList<>());
for (Document document : documents) {
logger.info("{}", document.toJson());
}
Note: this code deliberately uses the form new Document(<pipeline aggregator>, ...) instead of the Aggregators utilities to make it easier to see the translation between the shell command and its Java equivalent.

try $project with $reduce
$setUnion is used to keep only the distinct ids and finally $size used to get the distinct array count
db.col.aggregate(
[
{$project : {
_id : 0,
skill : 1,
quantity : {$size :{$reduce : {input : "$connects.parser_id", initialValue : [] , in : {$setUnion : ["$$value", "$$this"]}}}}
}
}
]
).pretty()
result
{ "skill" : "stack", "quantity" : 4 }

Issue in extracting the sum aggregation in ElasticSearch multi field group by using JavaAPI

Using ElasticSearch 5.2 and a group by is being done similer to
select city,institutionId, SUM(appOpenCount) from XYZ where ( time > 123 && appOpenCount > 0 ) group by city, institutionId.
I have it working when i do using curl method, but when the same is being converted to java api i am missing something that is causing me not get the last part of sum aggregation.
I have a type temp_type with mapping given below.
{
"temp_index" : {
"mappings" : {
"temp_type" : {
"properties" : {
"appOpenCount" : {
"type" : "integer"
},
"city" : {
"type" : "keyword"
}
"institutionId" : {
"type" : "keyword"
},
"time" : {
"type" : "long"
}
}
}
}
}
}
and my aggregation XGET call looks like this.
curl -XGET "http://localhost:9200/temp_index/temp_type/_search?pretty" -d'
{
"size":0,
"_source":false,
"from" : 0,
"query": {
"bool": {
"must": [
{"range": { "time": { "gte": 1513744603000 } } },
{ "range": { "appOpenCount": { "gt": 0 } } }
]
}
},
"aggregations": {
"city-aggs": {
"terms": { "field": "city"},
"aggregations": {
"intitution-agg": {
"terms": { "field": "institutionId" },
"aggregations": {
"appOpenCount": { "sum": { "field": "appOpenCount" }}}
}
}
}
}
}'
The response is perfect ( the aggregated number mathematically makes sense )
{
"took" : 57,
"timed_out" : false,
"_shards" : { ... },
"hits" : {... },
"aggregations" : {
"city-aggs" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "city-1",
"doc_count" : 25,
"intitution-agg" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "inst-1",
"doc_count" : 5,
"appOpenCount" : {
"value" : 15.0
}
}
]
}
}
]
}
}
Using this as template i converted this to Java API call and it i am able to execute it and access city-agg key and institution-agg key but am not sure how to access the appOpenCount agg. Basically getting null for Sum aggregation.
// bool query
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
List<QueryBuilder> mustQueries = boolQueryBuilder.must();
mustQueries.add(QueryBuilders.rangeQuery("time").gte(startTime));
mustQueries.add(QueryBuilders.rangeQuery("appOpenCount").gt(0));
queryBuilder = boolQueryBuilder;
// aggregationbuilder
AggregationBuilder aggregationBuilder = null;
TermsAggregationBuilder cityAggs = AggregationBuilders.terms("city-aggs").field("city");
TermsAggregationBuilder institutionAggs = AggregationBuilders.terms(
"institution-agg").field("institutionId");
SumAggregationBuilder fieldAggBuilder = AggregationBuilders.sum("appOpenCount").field("appOpenCount");
aggregationBuilder = cityAggs.subAggregation(institutionAggs).subAggregation(fieldAggBuilder);
// search call
SearchResponse searchResponse = client.prepareSearch(indexName)
.setTypes(typeName)
.setQuery(queryBuilder)
.addAggregation(aggregationBuilder)
.setFrom(0)
.setSize(0)
.execute().actionGet();
// Iterate the searchResponse
Terms cityAggsTerms = searchResponse.getAggregations().get("city-aggs");
List<Terms.Bucket> mainCityBuckets = cityAggsTerms.getBuckets();
for (Terms.Bucket mainCityBucket : mainCityBuckets) {
String cityName = mainCityBucket.getKeyAsString();
LOGGER.info("CityName : " + cityName); // all good
Terms institutionTerms = mainCityBucket.getAggregations().get("institution-agg");
List<Terms.Bucket> institutionBuckets = institutionTerms.getBuckets();
for (Terms.Bucket institutionBucket : institutionBuckets) {
String institutionName = institutionBucket.getKeyAsString();
LOGGER.info("InstitutionName : " + institutionName ); // all good
Sum appOpenCountSum = institutionBucket.getAggregations().get("appOpenCount");
if(appOpenCountSum != null) {
double appOpenCount = appOpenCountSum.getValue();
LOGGER.info("InstitutionName : " + institutionName +
" and appOpenCount is " + appOpenCount);
} else {
LOGGER.info("appOpenCountSum is null");
}
} // institution for
}// city for
How can i access the value of appOpenCount aggregation. I am hitting the case where my "appOpenCountSum" variable is null. Any help would be appreciated. I am able to access the city-agg and institution-agg and get proper values too. Not sure how to access the appOpenCount aggregation inside Term.Bucket
I followed the example provided in elastic search docs for this
https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/_metrics_aggregations.html#java-aggs-metrics-sum
Have given in-depth breakdown and hopefully it helps others too.
EDIT : Issue was the way i was building the aggregation query in java. The fieldAggBuilder should be added to institutionAggs and not the way i had done previously. The corrected code below.
// aggregationbuilder
AggregationBuilder aggregationBuilder = null;
TermsAggregationBuilder cityAggs = AggregationBuilders.terms("cityaggs").field("city");
TermsAggregationBuilder institutionAggs = AggregationBuilders.terms(
"institution-agg").field("institutionId");
SumAggregationBuilder fieldAggBuilder =
AggregationBuilders.sum("appOpenCount").field("appOpenCount");
institutionAggs.subAggregation(fieldAggBuilder); // this was missing previously
aggregationBuilder = cityAggs.subAggregation(institutionAggs);

MongoDB $regex query for "end with" particular char

I am not able to remove object from an array named Matrix for a Key match
BasicDBObject where = new BasicDBObject();
where.put("INSTITUTION_ID", instid);
where.put("RuleID", ruleid);
BasicDBObject obj1 = new BasicDBObject();
obj1.put("Matrix.Key",new BasicDBObject("$regex","/"+json.getString("Code")+"$/"));
collection.update(where,new BasicDBObject("$pull", obj1));
The code above is not removing object from array. The structure of the array can be found below
"Matrix" : [
{
"Key" : "6M",
"value" : "Queue"
},
{
"Key" : "6N",
"value" : "Queue"
},
{
"Key" : "6O",
"value" : "Queue"
}]

Command-line client
I suggest that before writing queries in Java notation, you first test them in the mongo console, with the regular JavaScript syntax. The following query works for me.
Data
db.matrix.insert(
{
INSTITUTION_ID: 1,
RuleID: 2,
Matrix: [
{
"Key": "6M",
"value": "Queue"
},
{
"Key": "6N",
"value": "Queue"
},
{
"Key": "6O",
"value": "Queue"
}
]
})
Query
db.matrix.update(
{
INSTITUTION_ID: 1,
RuleID: 2,
},
{
$pull:
{
Matrix:
{
Key:
{
$regex: /M$/
}
}
}
})
Data after the update
{
"INSTITUTION_ID" : 1.0000000000000000,
"RuleID" : 2.0000000000000000,
"Matrix" : [
{
"Key" : "6N",
"value" : "Queue"
},
{
"Key" : "6O",
"value" : "Queue"
}
]
}
Java
I am not sure how this update query should be represented in Java, but try this:
BasicDBObject where =
new BasicDBObject()
.put("INSTITUTION_ID", instid);
.put("RuleID", ruleid);
BasicDBObject update =
new BasicDBObject("$pull",
new BasicDBObject("Matrix",
new BasicDBObject("Key",
new BasicDBObject("$regex",
java.util.regex.Pattern.compile(json.getString("Code") + "$")))));
collection.update(where, update);

retrieve values from nested json array in mongodb

My mongo collection has entries in the following format
{
"myobj" : {
"objList" : [
{ "location" : "Texas" },
{ "location" : "Houston"},
{ "name":"Sam" }
]
},
"category" : "cat1"
}
{
"myobj" :
{
"objList" : [
{ "location" : "Tennesy" },
{ "location" : "NY"},
{ "location" : "SF" }
]
},
"category" : "cat2"
}
I want to extract the "**category**" where location is "Houston". In case of simple JSON object I have to just pass it as query like:
BasicDBObject place = new BasicDBObject();
place.put("location", "Houston");
But in case of nested JSON I don't know how to pass it as a query and get the appropriate category. ie If I pass my location as"Houston" then it should return it's appropriate category "cat1"...i hope my question is clear now....

Ok, you have your documents:
db.coll1.insert({
"myobj" : {
"objList" : [
{ "location" : "Texas" },
{ "location" : "Houston"},
{ "name":"Sam" }
]
},
"category" : "cat1"
})
and
db.coll1.insert({
"myobj" : {
"objList" : [
{ "location" : "Tennesy" },
{ "location" : "Houston"},
{ "location" : "SF" }
]
},
"category" : "cat1"
})
Now you can find what you want using the dot operator:
db.coll1.find({"myobj.objList.location": "Texas"}).pretty() will return one object which has Texas
db.coll1.find({"myobj.objList.location": "SF"}).pretty() will return one object which has SF
db.coll1.find({"myobj.objList.location": "Houston"}).pretty() will return both objects
And now I hope you will be able to write it in Java. I have never used Java, but based on this question you can do something like this. If it will not work, just look how to use dot operator in java driver for mongo:
DBCursor cursor = coll1.find(new BasicDBObject("myobj.objList.location", "Texas"));
P.S. you told, that you wanted to retrieve category. In such a way, you will need to use a projection db.coll1.find({<the query I provided}, {category: 1, _id: 0})