Background
I am trying to fetch data from mongo by matching custom class with few fields.
Mongo Collection:
"_id":{
"sNo":"1001",
"name": "Sameer",
"city": "Pune",
"state": "Maharashtra"
}
Pojo to match above details
#Document
class Id {
private String sNo;
private String name;
private String city;
private String state;
// getter & setters
}
Java code to fetch data from mongo where I want to fetch data based on sNo, name & city and want to skip state for matching:
ArrayList<Id> ids = new ArrayList<Id>();
...// code to populate Ids
MatchOperation match = Aggregation.match(Criteria.where("_id").in(ids));
Aggregation agg = Aggregation.newAggregation(match);
AggregationResults<Document> result = mongoTemplate.aggregate(agg, detailsCollection,
Document.class);
Problem:
I didn't find any way to skip few of the fields while match.
for now I am getting all data from collection.
Please help to suggest, how this can be achieved?
Related
I'm having an issue with an old aggregation query.
I have a mongo collection which has many documents containing information about credits / gifts given / awarded to our users.
Currently I am able to run an Aggregated Query to get all the users who have received a credit / gift within the last x number of days and sum the total value of those credits for each user. However, my problem is that now I want to project more fields for my mapping class which I am unable to do.
Here is the document in Mongo
_id: ObjectId("61c36a8a21047124c4181271"),
transactionId: UUID("6fbf536e-7a53-442c-9615-53e32362608b"),
userId: 'xxxxx',
transactionMessage: 'Account credited with: 1',
transactionType: 'CREDIT',
transactionAction: 'GIFT',
inComingPaymentFromUserId: 'xxxx',
awardForContentId : "abcd123242"
transactionAmount: Decimal128("1"),
customMessage: "blah'",
createdDate: ISODate("2021-12-22T18:12:26.812Z"),
lastUpdatedAt: ISODate("2021-12-22T18:12:26.812Z"),
I can run a aggregation which gives me the correct mapping like so:
Aggregation agg = Aggregation.newAggregation(match(Criteria.where("createdDate").gt(LocalDate.now().minusDays(range))
.andOperator(Criteria.where("transactionAction").is("GIFT"))), sort(Sort.Direction.DESC, "createdDate"),
group("userId").sum("transactionAmount").as("totalValueAwarded"),
skip((long) pageable.getPageNumber() * pageable.getPageSize()),
limit(pageable.getPageSize()));
mongoTemplate.aggregate(agg, Transaction.class, RecentlyRewardedUsers.class).getMappedResults();
For Context my mapped RecentlyRewarded.class looks like this:
#Getter
#Setter
public class RecentlyRewardedUsers {
#Field("userId")
private String userId;
private String totalValueAwarded;
}
And correctly the data is mapped to the two fields when the above aggregation runs.
Now, I find I need to add more fields to my RecentlyRewarded class:
#Getter
#Setter
#JsonInclude(JsonInclude.Include.NON_NULL)
public class RecentlyRewardedUsers {
#Field("id")
#Id
private String userId;
private BigDecimal totalValueAwarded;
private String awardForContentId; //THIS IS THE NEW FIELD I've ADDED
}
I thought I would be able to just add the new field "awardforContentId" to my group query and it would be correctly mapped but that is not happening:
Aggregation agg = Aggregation.newAggregation(match(Criteria.where("createdDate").gt(LocalDate.now().minusDays(range))
.andOperator(Criteria.where("transactionAction").is("GIFT"))), sort(Sort.Direction.DESC, "createdDate"),
group("userId", "awardForContentId").sum("transactionAmount").as("totalValueAwarded"),
skip((long) pageable.getPageNumber() * pageable.getPageSize()),
limit(pageable.getPageSize()));
mongoTemplate.aggregate(agg, Transaction.class, RecentlyRewardedUsers.class).getMappedResults();
What happens is that my userId field in my POJO is now set to:
{"userId": "auth0|61b9e2f7d1fc9f0071d508f1", "awardForContentId": "637b98a85dde3949fbb4314f"}
and the awardForContentId in my class is null.
I have also tried just adding the awardForConentId to the project operation like so:
Aggregation agg = Aggregation.newAggregation(match(Criteria.where("createdDate").gt(LocalDate.now().minusDays(range))
.andOperator(Criteria.where("transactionAction").is("GIFT"))), sort(Sort.Direction.DESC, "createdDate"),
group("userId").sum("transactionAmount").as("totalValueAwarded"),
project("userId", "totalValueAwarded", "awardForContentId"),
skip((long) pageable.getPageNumber() * pageable.getPageSize()),
limit(pageable.getPageSize()));
However that results in the following error:
Method threw 'java.lang.IllegalArgumentException' exception.
Invalid reference 'awardForContentId'!
What am I doing stupid here?
Many thanks
When you group by multiple fields, the result will look something like this.
{
"_id": {
"userId": ,
"awardForContentId": ,
},
"transactionAmount":
}
So, you have to project it correctly to match you Java class.
Aggregation agg = Aggregation.newAggregation(
match(
Criteria.where("createdDate").gt(LocalDate.now().minusDays(range))
.andOperator(Criteria.where("transactionAction").is("GIFT"))
),
sort(Sort.Direction.DESC, "createdDate"),
group("userId", "awardForContentId").sum("transactionAmount").as("totalValueAwarded"),
project()
.and("_id.userId").as("userId")
.and("_id.awardForContentId").as("awardForContentId")
.andInclude("totalValueAwarded")
skip((long) pageable.getPageNumber() * pageable.getPageSize()),
limit(pageable.getPageSize())
);
You can remove #Field("id") #Id from the returning class.
I have documents with dynamic fields and I would need to find a count of matching records for a given complex query criteria
Example Entity
#Document(collection = "UserAttributes")
public class UserAttributesEntity {
#Id
#Getter
private String id;
#NotNull
#Size(min = 1)
#Getter #Setter
private String userId;
#NotNull
#Getter #Setter
private Map<String, Object> attributes = new HashMap<>();
}
Example Data:
{
"_id" : ObjectId("6164542362affb14f3f2fef6"),
"userId" : "89ee6942-289a-48c9-b0bb-210ea7c06a88",
"attributes" : {
"age" : 61,
"name" : "Name1"
}
},
{
"_id" : ObjectId("6164548045cc4456792d5325"),
"userId" : "538abb29-c09d-422e-97c1-df702dfb5930",
"attributes" : {
"age" : 40,
"name" : "Name2",
"location" : "IN"
}
}
Expected Query Criteria:
"((userAttributes.name == 'Name1' && userAttributes.age > 40) OR (userAttributes.location == 'IN'))
Building such complex Criteria using $match would be too much of implementation, so I was trying to use SPEL evolution through $project like below:
private Mono<Long> aggregate() {
final Aggregation aggregation = Aggregation
.newAggregation(
Aggregation.project("userAttributes.playerLevel", "userAttributes.name")
.andExpression("((userAttributes.name == 'Name1' && userAttributes.age > 40) OR (userAttributes.location == 'IN'))")
.as("result"),
Aggregation.match(Criteria.where("result").is(true)),
Aggregation.group().count().as("count"));
return mongoTemplate.aggregate(aggregation, UserAttributesEntity.class, Map.class)
.map(result -> Long.valueOf(result.get("count").toString()))
.next();
}
However, the above logic failing due to an exception:
org.springframework.data.mapping.MappingException: Couldn't find PersistentEntity for type java.lang.Object!
at org.springframework.data.mapping.context.MappingContext.getRequiredPersistentEntity(MappingContext.java:119)
at org.springframework.data.mapping.context.PersistentPropertyPathFactory.getPair(PersistentPropertyPathFactory.java:226)
at org.springframework.data.mapping.context.PersistentPropertyPathFactory.createPersistentPropertyPath(PersistentPropertyPathFactory.java:199)
at org.springframework.data.mapping.context.PersistentPropertyPathFactory.lambda$getPersistentPropertyPath$1(PersistentPropertyPathFactory.java:172)
at java.base/java.util.concurrent.ConcurrentMap.computeIfAbsent(ConcurrentMap.java:330)
at org.springframework.data.mapping.context.PersistentPropertyPathFactory.getPersistentPropertyPath(PersistentPropertyPathFactory.java:171)
at org.springframework.data.mapping.context.PersistentPropertyPathFactory.from(PersistentPropertyPathFactory.java:71)
at org.springframework.data.mapping.context.AbstractMappingContext.getPersistentPropertyPath(AbstractMappingContext.java:295)
at org.springframework.data.mongodb.core.aggregation.TypeBasedAggregationOperationContext.getReferenceFor(TypeBasedAggregationOperationContext.java:163)
at org.springframework.data.mongodb.core.aggregation.TypeBasedAggregationOperationContext.getReference(TypeBasedAggregationOperationContext.java:107)
at org.springframework.data.mongodb.core.aggregation.AggregationExpressionTransformer$AggregationExpressionTransformationContext.getFieldReference(AggregationExpressionTransformer.java:82)
at org.springframework.data.mongodb.core.aggregation.SpelExpressionTransformer$CompoundExpressionNodeConversion.convert(SpelExpressionTransformer.java:541)
at org.springframework.data.mongodb.core.aggregation.SpelExpressionTransformer.transform(SpelExpressionTransformer.java:113)
at org.springframework.data.mongodb.core.aggregation.SpelExpressionTransformer.transform(SpelExpressionTransformer.java:58)
at org.springframework.data.mongodb.core.aggregation.SpelExpressionTransformer$ExpressionNodeConversion.transform(SpelExpressionTransformer.java:215)
at org.springframework.data.mongodb.core.aggregation.SpelExpressionTransformer$ExpressionNodeConversion.transform(SpelExpressionTransformer.java:205)
at org.springframework.data.mongodb.core.aggregation.SpelExpressionTransformer$OperatorNodeConversion.convert(SpelExpressionTransformer.java:257)
at org.springframework.data.mongodb.core.aggregation.SpelExpressionTransformer.transform(SpelExpressionTransformer.java:113)
at org.springframework.data.mongodb.core.aggregation.SpelExpressionTransformer.transform(SpelExpressionTransformer.java:58)
at org.springframework.data.mongodb.core.aggregation.SpelExpressionTransformer$ExpressionNodeConversion.transform(SpelExpressionTransformer.java:215)
at org.springframework.data.mongodb.core.aggregation.SpelExpressionTransformer$ExpressionNodeConversion.transform(SpelExpressionTransformer.java:205)
at org.springframework.data.mongodb.core.aggregation.SpelExpressionTransformer$OperatorNodeConversion.convert(SpelExpressionTransformer.java:251)
at org.springframework.data.mongodb.core.aggregation.SpelExpressionTransformer.transform(SpelExpressionTransformer.java:113)
at org.springframework.data.mongodb.core.aggregation.SpelExpressionTransformer.transform(SpelExpressionTransformer.java:105)
at org.springframework.data.mongodb.core.aggregation.ProjectionOperation$ExpressionProjectionOperationBuilder$ExpressionProjection.toMongoExpression(ProjectionOperation.java:438)
at org.springframework.data.mongodb.core.aggregation.ProjectionOperation$ExpressionProjectionOperationBuilder$ExpressionProjection.toDocument(ProjectionOperation.java:433)
at org.springframework.data.mongodb.core.aggregation.ProjectionOperation.toDocument(ProjectionOperation.java:261)
at org.springframework.data.mongodb.core.aggregation.AggregationOperation.toPipelineStages(AggregationOperation.java:55)
at org.springframework.data.mongodb.core.aggregation.AggregationOperationRenderer.toDocument(AggregationOperationRenderer.java:56)
at org.springframework.data.mongodb.core.aggregation.AggregationPipeline.toDocuments(AggregationPipeline.java:81)
at org.springframework.data.mongodb.core.aggregation.Aggregation.toPipeline(Aggregation.java:705)
at org.springframework.data.mongodb.core.AggregationUtil.createPipeline(AggregationUtil.java:105)
at org.springframework.data.mongodb.core.ReactiveMongoTemplate.aggregate(ReactiveMongoTemplate.java:1001)
at org.springframework.data.mongodb.core.ReactiveMongoTemplate.aggregate(ReactiveMongoTemplate.java:970)
I tried digging in further, it seems to fail while its trying to map the fields used within the project to a proper entity object and in this case the projection fields here are actually are of Map object and its unable to identify the matching Entity (of-course the Map is within the UserAttributesEntity) - Is there a better way to solve my problem other than the above approach?
Using the raw aggregation resolved the issue:
return mongoTemplate.aggregate(aggregation, mongoTemplate.getCollectionName(UserAttributesEntity.class), Map.class)
.map(result -> Long.valueOf(result.get("count").toString()))
.next()
I'm trying to parse some JSON data with mongodb. I have no data found I think that the problem is with the difference in field in mongodb data
{"imei":"865566048694354","_id":"5e7c996fd6eb5f039c50bd26","createdAt":"2020-03-26T12:00:47.021Z","updatedAt":"2020-03-26T12:00:47.021Z","__v":0},{"imei":{"test":{"tactileState":"ignore","pixelState":"ignore"},"name":"h12hhhhgkhh"},"_id":"5ea8357d8c562b3dd8fe5bf1","createdAt":"2020-04-28T13:54:05.094Z","updatedAt":"2020-04-28T13:54:05.094Z","__v":0},{"imei":{"test":{"tactileState":"ignore","pixelState":"ignore"},"name":"h12hhhhgkhh"},"_id":"5ea8366741a5e527446744a2","createdAt":"2020-04-28T13:57:59.035Z","updatedAt":"2020-04-28T13:57:59.035Z","__v":0},{"imei":{"test":{"tactileState":"ignore","pixelState":"ignore","greyState":"ignore"},"name":"h12hhhhgkhh"},"_id":"5ea837614cf7ed30f0163c38","createdAt":"2020-04-28T14:02:09.395Z","updatedAt":"2020-04-28T14:02:09.395Z","__v":0},{"imei":{"test":{"bafleState":"1","microState":"1","vibreurState":"1"},"name":"h12hhhhgkhh"},"_id":"5ea837854cf7ed30f0163c39","createdAt":"2020-04-28T14:02:45.287Z","updatedAt":"2020-04-28T14:02:45.287Z","__v":0}
If you can help me how to write the class and how to write the method in flutter because everything which I made always snapshot has no data I think that the problem in the difference in fields in mongodb data makes the problem because all tutorial and article which I see didn't use different database field always the same structure even with embedded document.
Suppose you have a json like this
{
"name": "John Smith",
"email": "john#example.com"
}
With `dart:convert, you can serialize this JSON model in two ways.
Map<String, dynamic> user = jsonDecode(jsonString);
print('Howdy, ${user['name']}!');
Or create a model like this
class User {
final String name;
final String email;
User(this.name, this.email);
User.fromJson(Map<String, dynamic> json)
: name = json['name'],
email = json['email'];
Map<String, dynamic> toJson() =>
{
'name': name,
'email': email,
};
}
And then use it like this:
Map userMap = jsonDecode(jsonString);
var user = User.fromJson(userMap);
print('Howdy, ${user.name}!');
Reference is here.
I am using java maven plugin.I want to fetch employee.csv file records in pojo class.
this pojo class I am generating from employee.csv header and all fields of pojo class are String type.now I want to map employee.csv to generated pojo class.my requirement is I dont want to specify column names manually.because if I change csv file then again I have to chane my code so it should dynamically map with any file. for instance
firstName,lastName,title,salary
john,karter,manager,54372
I want to map this to pojo which I have already
public class Employee
{
private String firstName;
private String lastName;
.
.
//getters and setters
//toString()
}
uniVocity-parsers allows you to map your pojo easily.
class Employee {
#Trim
#LowerCase
#Parsed
private String firstName;
#Parsed
private String lastName;
#NullString(nulls = { "?", "-" }) // if the value parsed in the quantity column is "?" or "-", it will be replaced by null.
#Parsed(defaultNullRead = "0") // if a value resolves to null, it will be converted to the String "0".
private Integer salary; // The attribute name will be matched against the column header in the file automatically.
...
}
To parse:
BeanListProcessor<Employee> rowProcessor = new BeanListProcessor<Employee>(Employee.class);
CsvParserSettings parserSettings = new CsvParserSettings();
parserSettings.setRowProcessor(rowProcessor);
parserSettings.setHeaderExtractionEnabled(true);
CsvParser parser = new CsvParser(parserSettings);
//And parse!
//this submits all rows parsed from the input to the BeanListProcessor
parser.parse(new FileReader(new File("/path/to/your.csv")));
List<Employee> beans = rowProcessor.getBeans();
Disclosure: I am the author of this library. It's open-source and free (Apache V2.0 license).
you can use openCSV jar to read the data and then you can map the each column values with the class attributes.
Due to security reason, i can not share my code with you.
I have a doc in my mongodb that looks like this -
public class AppCheckInRequest {
private String _id;
private String uuid;
private Date checkInDate;
private Double lat;
private Double lon;
private Double altitude;
}
The database will contain multiple documents with the same uuid but different checkInDates
Problem
I would like to run a mongo query using java that gives me one AppCheckInRequest doc(all fields) per uuid who's checkInDate is closest to the current time.
I believe I have to the aggregation framework, but I can't figure out how to get the results I need. Thanks.
In the mongo shell :-
This will give you the whole groupings:
db.items.aggregate({$group : {_id : "$uuid" , value : { $push : "$somevalue"}}} )
And using $first instead of $push will only put one from each (which is what you want i think?):
db.items.aggregate({$group : {_id : "$uuid" , value : { $first : "$somevalue"}}} )
Can you translate this to the Java api? or i'll try to add that too.
... ok, here's some Java:
Assuming the docs in my collection are {_id : "someid", name: "somename", value: "some value"}
then this code shows them grouped by name:
Mongo client = new Mongo("127.0.0.1");
DBCollection col = client.getDB("ajs").getCollection("items");
AggregationOutput agout = col.aggregate(
new BasicDBObject("$group",
new BasicDBObject("_id", "$name").append("value", new BasicDBObject("$push", "$value"))));
Iterator<DBObject> results = agout.results().iterator();
while(results.hasNext()) {
DBObject obj = results.next();
System.out.println(obj.get("_id")+" "+obj.get("value"));
}
and if you change $push to $first, you'll only get 1 per group. You can then add the rest of the fields once you get this query working.