I want to find users by most recent date (Assume the User object has a date field). The data is stored in MongoDB and accessed via a Spring MongoTemplate.
Example of the raw data:
{userId:1, date:10}
{userId:1, date:20}
{userId:2, date:50}
{userId:2, date:10}
{userId:3, date:10}
{userId:3, date:30}
The query should return
{{userId:1, date:20}, {userId:2, date:50}, {userId:3, date:30}}
The aggregation method Ï am using is
db.table1.aggregate({$group:{'_id':'$userId', 'max':{$max:'$date'}}},
{$sort:{'max':1}}).result
You could Sort it first by date DESC and select the first while grouping by userID
final Aggregation aggregation = newAggregation(
Aggregation.sort(Sort.Direction.DESC, "date"),
Aggregation.group("userId").first("date").as("Date")
);
final AggregationResults<User> results = mongoTemplate.aggregate(aggregation, "user", User.class);
Related
I'm having an issue with an old aggregation query.
I have a mongo collection which has many documents containing information about credits / gifts given / awarded to our users.
Currently I am able to run an Aggregated Query to get all the users who have received a credit / gift within the last x number of days and sum the total value of those credits for each user. However, my problem is that now I want to project more fields for my mapping class which I am unable to do.
Here is the document in Mongo
_id: ObjectId("61c36a8a21047124c4181271"),
transactionId: UUID("6fbf536e-7a53-442c-9615-53e32362608b"),
userId: 'xxxxx',
transactionMessage: 'Account credited with: 1',
transactionType: 'CREDIT',
transactionAction: 'GIFT',
inComingPaymentFromUserId: 'xxxx',
awardForContentId : "abcd123242"
transactionAmount: Decimal128("1"),
customMessage: "blah'",
createdDate: ISODate("2021-12-22T18:12:26.812Z"),
lastUpdatedAt: ISODate("2021-12-22T18:12:26.812Z"),
I can run a aggregation which gives me the correct mapping like so:
Aggregation agg = Aggregation.newAggregation(match(Criteria.where("createdDate").gt(LocalDate.now().minusDays(range))
.andOperator(Criteria.where("transactionAction").is("GIFT"))), sort(Sort.Direction.DESC, "createdDate"),
group("userId").sum("transactionAmount").as("totalValueAwarded"),
skip((long) pageable.getPageNumber() * pageable.getPageSize()),
limit(pageable.getPageSize()));
mongoTemplate.aggregate(agg, Transaction.class, RecentlyRewardedUsers.class).getMappedResults();
For Context my mapped RecentlyRewarded.class looks like this:
#Getter
#Setter
public class RecentlyRewardedUsers {
#Field("userId")
private String userId;
private String totalValueAwarded;
}
And correctly the data is mapped to the two fields when the above aggregation runs.
Now, I find I need to add more fields to my RecentlyRewarded class:
#Getter
#Setter
#JsonInclude(JsonInclude.Include.NON_NULL)
public class RecentlyRewardedUsers {
#Field("id")
#Id
private String userId;
private BigDecimal totalValueAwarded;
private String awardForContentId; //THIS IS THE NEW FIELD I've ADDED
}
I thought I would be able to just add the new field "awardforContentId" to my group query and it would be correctly mapped but that is not happening:
Aggregation agg = Aggregation.newAggregation(match(Criteria.where("createdDate").gt(LocalDate.now().minusDays(range))
.andOperator(Criteria.where("transactionAction").is("GIFT"))), sort(Sort.Direction.DESC, "createdDate"),
group("userId", "awardForContentId").sum("transactionAmount").as("totalValueAwarded"),
skip((long) pageable.getPageNumber() * pageable.getPageSize()),
limit(pageable.getPageSize()));
mongoTemplate.aggregate(agg, Transaction.class, RecentlyRewardedUsers.class).getMappedResults();
What happens is that my userId field in my POJO is now set to:
{"userId": "auth0|61b9e2f7d1fc9f0071d508f1", "awardForContentId": "637b98a85dde3949fbb4314f"}
and the awardForContentId in my class is null.
I have also tried just adding the awardForConentId to the project operation like so:
Aggregation agg = Aggregation.newAggregation(match(Criteria.where("createdDate").gt(LocalDate.now().minusDays(range))
.andOperator(Criteria.where("transactionAction").is("GIFT"))), sort(Sort.Direction.DESC, "createdDate"),
group("userId").sum("transactionAmount").as("totalValueAwarded"),
project("userId", "totalValueAwarded", "awardForContentId"),
skip((long) pageable.getPageNumber() * pageable.getPageSize()),
limit(pageable.getPageSize()));
However that results in the following error:
Method threw 'java.lang.IllegalArgumentException' exception.
Invalid reference 'awardForContentId'!
What am I doing stupid here?
Many thanks
When you group by multiple fields, the result will look something like this.
{
"_id": {
"userId": ,
"awardForContentId": ,
},
"transactionAmount":
}
So, you have to project it correctly to match you Java class.
Aggregation agg = Aggregation.newAggregation(
match(
Criteria.where("createdDate").gt(LocalDate.now().minusDays(range))
.andOperator(Criteria.where("transactionAction").is("GIFT"))
),
sort(Sort.Direction.DESC, "createdDate"),
group("userId", "awardForContentId").sum("transactionAmount").as("totalValueAwarded"),
project()
.and("_id.userId").as("userId")
.and("_id.awardForContentId").as("awardForContentId")
.andInclude("totalValueAwarded")
skip((long) pageable.getPageNumber() * pageable.getPageSize()),
limit(pageable.getPageSize())
);
You can remove #Field("id") #Id from the returning class.
I have an implementation of hibernate-search-orm (5.9.0.Final) with hibernate-search-elasticsearch (5.9.0.Final).
I defined a custom analyzer on an entity (see beelow) and I indexed two entities :
id: "1"
title: "Médiatiques : récit et société"
abstract:...
id: "2"
title: "Mediatique Com'7"
abstract:...
The search works fine when I search on title field :
"title:médiatique" => 2 results.
"title:mediatique" => 2 results.
My problem is when I do a global search with accents (or not) :
search on "médiatique => 1 result (id:1)
search on "mediatique => 1 result (id:2)
Is there a way to resolve this?
Thanks.
Entity definition:
#Entity
#Table(name="bibliographic")
#DynamicUpdate
#DynamicInsert
#Indexed(index = "bibliographic")
#FullTextFilterDefs({
#FullTextFilterDef(name = "fieldsElasticsearchFilter",
impl = FieldsElasticsearchFilter.class)
})
#AnalyzerDef(name = "customAnalyzer",
tokenizer = #TokenizerDef(factory = StandardTokenizerFactory.class),
filters = {
#TokenFilterDef(factory = LowerCaseFilterFactory.class),
#TokenFilterDef(factory = ASCIIFoldingFilterFactory.class),
})
#Analyzer(definition = "customAnalyzer")
public class BibliographicHibernate implements Bibliographic {
...
#Column(name="title", updatable = false)
#Fields( {
#Field,
#Field(name = "titleSort", analyze = Analyze.NO, store = Store.YES)
})
#SortableField(forField = "titleSort")
private String title;
...
}
Search method :
FullTextEntityManager ftem = Search.getFullTextEntityManager(entityManager);
QueryBuilder qb = ftem.getSearchFactory().buildQueryBuilder().forEntity(Bibliographic.class).get();
QueryDescriptor q = ElasticsearchQueries.fromQueryString(queryString);
FullTextQuery query = ftem.createFullTextQuery(q, Bibliographic.class).setFirstResult(start).setMaxResults(rows);
if (filters!=null){
filters.stream().map((filter) -> filter.split(":")).forEach((f) -> {
query.enableFullTextFilter("fieldsElasticsearchFilter")
.setParameter("field", f[0])
.setParameter("value", f[1]);
}
);
}
if (facetFields!=null){
facetFields.stream().map((facet) -> facet.split(":")).forEach((f) ->{
query.getFacetManager()
.enableFaceting(qb.facet()
.name(f[0])
.onField(f[0])
.discrete()
.orderedBy(FacetSortOrder.COUNT_DESC)
.includeZeroCounts(false)
.maxFacetCount(10)
.createFacetingRequest() );
}
);
}
List<Bibliographic> bibs = query.getResultList();
To be honest I'm more surprised document 1 would match at all, since there's a trailing "s" on "Médiatiques" and you don't use any stemmer.
You are in a special case here: you are using a query string and passing it directly to Elasticsearch (that's what ElasticsearchQueries.fromQueryString(queryString) does). Hibernate Search has very little impact on the query being run, it only impacts the indexed content and the Elasticsearch mapping here.
When you run a QueryString query on Elasticsearch and you don't specify any field, it uses all fields in the document. I wouldn't bet that the analyzer used when analyzing your query is the same analyzer that you defined on your "title" field. In particular, it may not be removing accents.
An alternative solution would be to build a simple query string query using the QueryBuilder. The syntax of queries is a bit more limited, but is generally enough for end users. The code would look like this:
FullTextEntityManager ftem = Search.getFullTextEntityManager(entityManager);
QueryBuilder qb = ftem.getSearchFactory().buildQueryBuilder().forEntity(Bibliographic.class).get();
Query q = qb.simpleQueryString()
.onFields("title", "abstract")
.matching(queryString)
.createQuery();
FullTextQuery query = ftem.createFullTextQuery(q, Bibliographic.class).setFirstResult(start).setMaxResults(rows);
Users would still be able to target specific fields, but only in the list you provided (which, by the way, is probably safer, otherwise they could target sort fields and so on, which you probably don't want to allow). By default, all the fields in that list would be targeted.
This may lead to the exact same result as the query string, but the advantage is, you can override the analyzer being used for the query. For instance:
FullTextEntityManager ftem = Search.getFullTextEntityManager(entityManager);
QueryBuilder qb = ftem.getSearchFactory().buildQueryBuilder().forEntity(Bibliographic.class)
.overridesForField("title", "customAnalyzer")
.overridesForField("abstract", "customAnalyzer")
.get();
Query q = qb.simpleQueryString()
.onFields("title", "abstract")
.matching(queryString)
.createQuery();
FullTextQuery query = ftem.createFullTextQuery(q, Bibliographic.class).setFirstResult(start).setMaxResults(rows);
... and this will use your analyzer when querying.
As an alternative, you can also use a more advanced JSON query by replacing ElasticsearchQueries.fromQueryString(queryString) with ElasticsearchQueries.fromJsonQuery(json). You will have to craft the JSON yourself, though, taking some precautions to avoid any injection from the user (use Gson to build the Json), and taking care to follow the Elasticsearch query syntax.
You can find more information about simple query string queries in the official documentation.
Note: you may want to add FrenchMinimalStemFilterFactory to your list of token filters in your custom analyzer. It's not the cause of your problem, but once you manage to use your analyzer in search queries, you will very soon find it useful.
Following code doesn't give the expected results, i've tried with multiple documents, those documents structured as follows.
_id: 5a7714d44c75220958e6aa01
imei:355227045347655
point: [3.143453333333333,80.10954]
heading: 0
speed:0
timestamp: 2018-02-04 19:42:36.000
point_distance: 525.25
Now i need to calculate sum of point_distance of every record match with given imei and time period. I've tried to achieve this with following code but returns nothing even if required data exists.
public Object findDistance(long imei, Date from, Date to) {
List aggregationOps = new ArrayList<>();
//operations
aggregationOps.add(match(Criteria.where("imei").is(imei)));
aggregationOps.add(match(Criteria.where("timestamp").gte(from)));
aggregationOps.add(match(Criteria.where("timestamp").lte(to)));
aggregationOps.add(group("imei").sum("point_distance").as("distance"));
aggregationOps.add(project("imei").and("distance").previousOperation());
AggregationOptions agOps = new AggregationOptions.Builder().allowDiskUse(true).cursor(new BasicDBObject()).build();
return (DistanceInfo) getMongoTemplate()
.aggregate(newAggregation(aggregationOps).withOptions(agOps), Location.class, DistanceInfo.class)
.getUniqueMappedResult();
}
Date > from and to are formatted as yyyy-MM-dd hh:mm:ss
DistanceInfo class
public class DistanceInfo {
long imei;
double distance;
}
I'm new to this mongodb stuff and no idea what did i do wrong, how can i correct this ? any help is much appreciated.
Try this. this should work
MatchOperation matchOperation = match(Criteria.where("imei").is(imei)
.and("timestamp").gte(from.getTime()).lte(to.getTime()));
GroupOperation groupOperation = group("imei").sum("point_distance").as("distance");
ProjectionOperation projectionOperation = project().andExpression("imei").as("imei")
.andExpression("distance").as("distance");
Aggregation aggregation = newAggregation(matchOperation, groupOperation, projectionOperation);
AggregationResults<DistanceInfo> results = mongoTemplate.aggregate(aggregation, "location", DistanceInfo.class);
return results.getMappedResults();
I am using spring-data-mongodb and I want to use a cursor for an aggregate operation.
MongoTemplate.stream() gets a Query, so I tried creating the Aggregation instance, convert it to a DbObject using Aggregation.toDbObject(), created a BasicQuery using the DbObject and then invoke the stream() method.
This returns an empty cursor.
Debugging the spring-data-mongodb code shows that MongoTemplate.stream() uses the FindOperation, which makes me thinkspring-data-mongodb does not support streaming an aggregation operation.
Has anyone been able to stream the results of an aggregate query using spring-data-mongodb?
For the record, I can do it using the Java mongodb driver, but I prefer using spring-data.
EDIT Nov 10th - adding sample code:
MatchOperation match = Aggregation.match(Criteria.where("type").ne("AType"));
GroupOperation group = Aggregation.group("name", "type");
group = group.push("color").as("colors");
group = group.push("size").as("sizes");
TypedAggregation<MyClass> agg = Aggregation.newAggregation(MyClass.class, Arrays.asList(match, group));
MongoConverter converter = mongoTemplate.getConverter();
MappingContext<? extends MongoPersistentEntity<?>, MongoPersistentProperty> mappingContext = converter.getMappingContext();
QueryMapper queryMapper = new QueryMapper(converter);
AggregationOperationContext context = new TypeBasedAggregationOperationContext(MyClass.class, mappingContext, queryMapper);
// create a BasicQuery to be used in the stream() method by converting the Aggregation to a DbObject
BasicQuery query = new BasicQuery(agg.toDbObject("myClass", context));
// spring-mongo attributes the stream() method to find() operationsm not to aggregate() operations so the stream returns an empty cursor
CloseableIterator<MyClass> iter = mongoTemplate.stream(query, MyClass.class);
// this is an empty cursor
while(iter.hasNext()) {
System.out.println(iter.next().getName());
}
The following code, not using the stream() method, returns the expected non-empty result of the aggregation:
AggregationResults<HashMap> result = mongoTemplate.aggregate(agg, "myClass", HashMap.class);
For those who are still trying to find the answer to this:
From spring-data-mongo version 2.0.0.M4 onwards (AFAIK) MongoTemplate got an aggregateStream method.
So you can do the following:
AggregationOptions aggregationOptions = Aggregation.newAggregationOptions()
// this is very important: if you do not set the batch size, you'll get all the objects at once and you might run out of memory if the returning data set is too large
.cursorBatchSize(mongoCursorBatchSize)
.build();
data = mongoTemplate.aggregateStream(Aggregation.newAggregation(
Aggregation.group("person_id").count().as("count")).withOptions(aggregationOptions), collectionName, YourClazz.class);
How to Get the difference between the two date values in Mongo criteria and return value must be the difference of two date(i.e 10.01.2015 firs date and 20.01.2015 is second date) I need the value as 10 ? can any one help me how to get this in java using spring framework critieria?)
From this article What's new in Spring Data MongoDB 1.4 M1, you can use the andExpression to get the difference between the two date values:
.andExpression("endDateTime - startDateTime").as("duration")
As an example (untested):
import static org.springframework.data.mongodb.core.aggregation.Aggregation.*;
TypedAggregation<Trip> agg = newAggregation(Trip.class,
match(Criteria.where("userId").is("54e5cead3ab5c1dd97422d86")),
project("endDateTime", "startDateTime")
.andExpression("endDateTime - startDateTime").as("duration")
);
AggregationResults<DBObject> result = mongoTemplate.aggregate(agg, DBObject.class);
List<DBObject> resultList = result.getMappedResults();
In Mongo shell, you can do the following:
db.collection.aggregate( {$match:{ your-match } },
{ $group : {
_id : "$userId",
duration : { $subtract:["$endDateTime", "$startDateTime"]}},
} } );
using spring boot reactive: This will give you duration in mili seconds like 864000000
List<AggregationOperation> operations = new ArrayList<>();
operations.add(project().andExclude("_id")
.and(ArithmeticOperators.valueOf("duration")
.subtract("startDateTime"))
.as("endDateTime"));
operations.add(groupOperation);
//response
reactiveMongoTemplate.aggregate(newAggregation(operations), "collName",
Class.class);
Rather than using minus operation inside projection, I will suggest you to use ArithmaticOperation.Subtract inside projection pipeline.
ProjectionOperation projectionOperation=Aggregation.project().and(Subtract.valueOf(Date.from(Instant.now())).subtract("fieldname"));