I have requirement to perform group by on nested fields in mongo.
The second level nested field is annotated with #Field. I am using projection with groupBy.
Example
ProjectionOperation projectionOperation = Aggregation.project("id")
.and("author.eid").as("user");
GroupOperation groupOperation = Aggregation.group(aggregationBy, "user").count().as("total");
Aggregation aggregation =
Aggregation.newAggregation(projectionOperation groupOperation);
AggregationResults<Document> aggregationResults = myRepository.getMongoTemplate().aggregate(aggregation, MyClass.class, Document.class);
On execution I am getting error "org.springframework.data.mapping.PropertyReferenceException: No property eid found for type User!"
public class MyClass {
User author;
}
public class User {
#Field("eid")
#JsonProperty("eid") // fasterxml
public String externalId;
}
What I can think of is when casting the aggregation result to MyClass, it is unable to find "eid" because it is annotated.
How to handle this usecase ?
The #Field annotation parsed to replace the pojo property with the field name.
So you should be using
ProjectionOperation projectionOperation = Aggregation.project("id")
.and("author.externalId").as("user");
the query generated will be
{ "$project" : { "user" : "$author.eid" }
Related
I have a DAOImpl with an override function. I'm trying to use an aggregation to first filter then match then to project.
public Map<Long, Integer> getUsersByAccount() {
MatchOperation filterByAccountId = match(new Criteria(ACCOUNT_ID).nin(Arrays.asList(null, "")));
GroupOperation groupByMasterAccount = group(DEFAULT_MASTER_ACCOUNT_ID).count().as("countOfFooUsers");
ProjectionOperation projectByIDandCount = project()
.and("_id")
.as("defaultMasterAccountId");
projectByIDandCount.and("countOfFooUsers");
Aggregation aggregation = newAggregation(
filterByAccountId,
groupByMasterAccount,
projectByIDandCount
);
AggregationResults<MAandUsers> maUsersResult = mongoTemplate
.aggregate(aggregation, USERS_COLLECTION, MAandUsers.class);
The inner class above MAandUsers within the IMPL:
#Data
public static class MAandUsers {
private Long defaultMasterAccountId;
private Integer countOfFooUsers;
}
However, my maUsersResultreturns a null value for countOfFooUsers when, in my test case, there should be 1. I'm assuming my projection operation is off ---> projectByIDandCount.and("countOfFooUsers");
Is there a way for me to use spring's mongotemplate to receive back multiple values on projection operations? I'm fairly new to this
Try using, andInclude
ProjectionOperation projectByIDandCount = project()
.and("_id").as("defaultMasterAccountId")
.andInclude("countOfFooUsers");
I'm getting the following exception when trying to use a named query with Spring Data Elasticsearch.
ClassCastException: org.springframework.data.elasticsearch.core.aggregation.impl.AggregatedPageImpl cannot be cast to org.springframework.data.elasticsearch.core.SearchPage
The query I'm trying to make is:
public interface PlayerRepository extends ElasticsearchRepository<PlayerEntity, String> {
#Query("{\"bool\":{\"must\":[{\"terms\":{\"playerNumber.keyword\": ?0}}]}}")
SearchPage<PlayerEntity> fetchPlayers(JSONArray playerNumbers, Pageable pageable);
}
If I do not use the #Query annotation and instead let Spring derive the query from the method name like so:
SearchPage<PlayerEntity> findPlayerEntityByPlayerNumberIn(List<String> playerNumbers, Pageable pageable);
It works as expected. However, the PlayerNumber field is a #MultiField that supports the the field types of Text and Keyword like so:
#Document(indexName = "#{#playersIndexName}")
public class PlayerEntity {
#MultiField(
mainField = #Field(type = Text, name = "playerNumber"),
otherFields = {#InnerField(suffix = "keyword", type = Keyword)})
private String playerNumber;
...
}
And I need to use the keyword mapping here for the query and not the text mapping. As far as I can tell, Spring Data Elasticsearch cannot derive queries from method names on InnerField, which is why I went with the named query approach. But it seems like the using the declared query approach, detailed here, only supports a subset of return types as detailed here
In addition, I need to use the SearchPage return type as well, because there is metadata there that I need to make decisions on.
So I guess there are a couple of questions that come out of this:
Is it possible to use InnerFields in derived query methods? i.e. something like SearchPage<PlayerEntity> findPlayerEntityByPlayerNumber_KeywordIn(List<String> playerNumbers, Pageable pageable);
Is it possible for a named query to return a SearchPage? I think this might be possible with a custom Repository implementation, but if I could get either approach above to work that would be ideal.
Thanks for any help!!
spring-data-elasticsearch version: 4.0.3.RELEASE
spring-boot-starter-parent version: 2.3.3.RELEASE
elasticsearch version: 7.11.1
To answer your second question (Is it possible for a named query to return a SearchPage?): This is a bug that it does not work with #Query annotated methods. I fixed that yesterday for the main, 4.2.x, 4.1.x and 4.0.x branches so it will work when the next service releases are out.
To answer the first one, I will need to do some research and tests before I can say anything about that - it would be great if it would work. I think I' can give more information later this weekend.
Edit/Addition:
The query derivation from the method name is based on the properties of the Java class and is done in the Spring Data base which knows nothing about these inner fields that only exist in Elasticsearch.
But you can use the following custom repository fragment:
public interface CustomPlayerRepository {
SearchPage<PlayerEntity> findPlayerEntityByPlayerNumberKeywordIn(List<String> playerNumbers, Pageable pageable);
}
public class CustomPlayerRepositoryImpl implements CustomPlayerRepository {
private final ElasticsearchOperations operations;
public CustomPlayerRepositoryImpl(ElasticsearchOperations operations) {
this.operations = operations;
}
#Override
public SearchPage<PlayerEntity> findPlayerEntityByPlayerNumberKeywordIn(
List<String> playerNumbers, Pageable pageable) {
var criteriaQuery = new CriteriaQuery(new Criteria("playerNumber.keyword").in(playerNumbers), pageable);
var searchHits = operations.search(criteriaQuery, PlayerEntity.class);
return SearchHitSupport.searchPageFor(searchHits, pageable);
}
}
I have the following
#Data //Lombok annotation to generate getters and setters
#Entity
public class TradeLog {
#Id
#Column(name="P_TRADE_ID")
private String tradeId;
}
tradeLog.setTradeId("1");
SqlParameterSource insertParam = new BeanPropertySqlParameterSource(tradeLog);
System.out.println(insertProc.execute(insertParam));
And I get this error Exception in thread "main" org.springframework.dao.InvalidDataAccessApiUsageException: Required input parameter 'P_TRADE_ID' is missing
I know that I could do a mapping directly on the jdbc template, but is there any way I could use the java persistence annotations or something like that to handle that for me?
JDBC template does not provide such thing, But you can easily use reflection to scan all the #Column fields and populate its value to MapSqlParameterSource which is another implementation of SqlParameterSource. Something like below , you can wrap it to a function for convenient :
tradeLog.setTradeId("1");
//Create MapSqlParameterSource based on tradeLog
MapSqlParameterSource param = new MapSqlParameterSource();
for(Field field : TradeLog.class.getDeclaredFields()) {
Column column = field.getAnnotation(Column.class);
if(column != null) {
field.setAccessible(true);
param.addValue(column.name(), field.get(tradeLog));
}
}
insertProc.execute(param)
I have Entity object :
#Entity(name = "table")
public class SomeEntity {
#Id
#Column(name = "id_column_name")
public final BigDecimal entityId;
#Column(name = "table_column_name")
public final String entityFieldName;
}
And I have database view defined like this:
CREATE OR REPLACE FORCE EDITIONABLE VIEW "V_TABLE" ("ID_COLUMN_NAME", "TABLE_COLUMN_NAME", "SOME_OTHER_COLUMN") AS ... (some SQL magic)
And I have repository with custom query:
#RepositoryRestResource
interface SomeEntityRepository extends PagingAndSortingRepository<SomeEntity, BigDecimal> {
#Query(value = "select id_column_name, table_column_name FROM V_TABLE where some_other_column = ?#{#parameter} order by ?#{#pageable}",
countQuery = "SELECT count(*) from V_TABLE v where some_other_column = ?#{#parameter}",
nativeQuery = true)
Page<SomeEntity> findBySomeParameter(#Param("parameter") long parameter, Pageable pageable);
}
Everything works fine when I request standard data with url:
http://localhost:8080/someEntity/search/findBySomeParameter?parameter=25&page=0&size=20
But when I add sorting information it doesn't work:
http://localhost:8080/someEntity/search/findBySomeParameter?parameter=25&page=0&size=20&sort=entityFieldName,asc
will throw following exception (I'm using Oracle database):
Caused by: java.sql.SQLSyntaxErrorException: ORA-00904: "ENTITYFIELDNAME": invalid identifier
It seems like sorting field are not translated with #Column(name), but are inlined into SQL query.
Is there any way to make pageable sort translated, so that it will use not field name but column name?
This article sheds light on the issue. Read from section 3.1 on.
Apparently dynamic sorting is not supported for native queries. Actually, if you change your findBySomeParameter method to take a Sort instead of a Pageable you will get org.springframework.data.jpa.repository.query.InvalidJpaQueryMethodException: Cannot use native queries with dynamic sorting.
Using pageable you don't get the exception, and pagination actually seems to work fine, but dynamic sorting does not substitute the column name as you found. Looks to me like the only solution is to use JPQL instead of native query, which is not a problem as long as the query you need to make is the one you provide. You would need to map the view though to a SomeEntityView class in order to use JPQL.
EDIT
I thought the issue was not documented but it actually is here in the official doc
Spring Data JPA does not currently support dynamic sorting for native queries, because it would have to manipulate the actual query declared, which it cannot do reliably for native SQL. You can, however, use native queries for pagination by specifying the count query yourself, as shown in the following example:
This workaround works for me in SpringBoot 2.4.3:
#PersistenceContext
private EntityManager entityManager;
// an object ptoperty name to a column name adapter
private Pageable adaptSortColumnNames(Pageable pageable) {
if (pageable.getSort().isSorted()) {
SessionFactory sessionFactory;
if (entityManager == null || (sessionFactory = entityManager.getEntityManagerFactory().unwrap(SessionFactory.class)) == null)
return pageable;
AbstractEntityPersister persister = (AbstractEntityPersister) ((MetamodelImplementor) sessionFactory.getMetamodel()).entityPersister(CommentEntity.class);
Sort adaptedSort = pageable.getSort().get().limit(1).map(order -> {
String propertyName = order.getProperty();
String columnName = persister.getPropertyColumnNames(propertyName)[0];
return Sort.by(order.getDirection(), columnName);
}).findFirst().get();
return PageRequest.of(pageable.getPageNumber(), pageable.getPageSize(), adaptedSort);
}
return pageable;
}
#GetMapping()
public ResponseEntity<PagedResponse<CommentResponse>> findByTextContainingFts(#RequestParam(value = "text", required = false) String text, Pageable pageable) {
// apply this adapter in controller
pageable = adaptSortColumnNames(pageable);
Page<CommentEntity> page = commentRepository.find(text, pageable);
return ResponseEntity.ok().body(domainMapper.fromPageToPagedResponse(page));
}
I have a collection of documents called 'appointment', and I am trying to group together the number of appointments per day and populate an List of objects call AppointmentSummary, where there will be one object per day, I am using Spring boot to try and achieve this however I keep running into issues.
I have created the following three classes in the same package
AppointmentSummaryRepository.java
public interface AppointmentSummaryRepository extends
MongoRepository<Appointment,String>, AppointmentSummaryRepositoryCustom {
}
AppointmentSummaryRepositoryCustom.java
public interface AppointmentSummaryRepositoryCustom {
List<AppointmentSummary> aggregate(LocalDate startDate, LocalDate endDate);
}
AppointmentSummaryRepositoryImpl.java
public class AppointmentSummaryRepositoryImpl implements AppointmentSummaryRepositoryCustom {
private final MongoTemplate mongoTemplate;
private final Logger log = LoggerFactory.getLogger(AppointmentSummaryRepositoryImpl.class);
#Autowired
public AppointmentSummaryRepositoryImpl(MongoTemplate mongoTemplate){
this.mongoTemplate = mongoTemplate;
}
#Override
public List<AppointmentSummary> aggregate(LocalDate startDate, LocalDate endDate){
log.debug("This is a request to aggerate appointment summary between {} to {}", startDate.toString(), endDate.toString());
MatchOperation matchOperation = getMatchOperation(startDate, endDate);
GroupOperation groupOperation = getGroupOperation();
log.debug("End group operaton");
ProjectionOperation projectionOperation = getProjectOperation();
return mongoTemplate.aggregate(Aggregation.newAggregation(
matchOperation,
groupOperation,
projectionOperation
), Appointment.class, AppointmentSummary.class).getMappedResults();
}
private MatchOperation getMatchOperation(LocalDate startDate, LocalDate endDate) {
log.debug("Begin Match Operation");
Criteria appointmentCriteria = where("appointment_date").gt(startDate).andOperator(where("appointment_date").lt(endDate));
log.debug("End Match Operation");
return match(appointmentCriteria);
}
private GroupOperation getGroupOperation() {
log.debug("Performing Group Operation");
return group("appointment_date")
.last("appointment_date").as("appointment_date")
.addToSet("id").as("appointmentIds")
.sum("id").as("count");
}
private ProjectionOperation getProjectOperation() {
log.debug("Begin project operation");
return project("appointment_date","appointmentIds","count")
.and("appointment_date").previousOperation();
}
Whenever I run the it, I keep getting the following error:
org.springframework.data.mapping.PropertyReferenceException: No property appointment found for type Appointment!
I believe the issue is happening in the following code segment, my understanding is that I initialize the different stages of the pipeline and pass them to the mongoTemplate and the 'getMappedResults' will map the fields from the two objects and populate the AppointmentSummary.class with the output of the aggregation pipeline?
return mongoTemplate.aggregate(Aggregation.newAggregation(
matchOperation,
groupOperation,
projectionOperation
), Appointment.class, AppointmentSummary.class).getMappedResults();
To note that the object Appointment does not have a field/property appointment. I added this in but when I ran the code I received another error message complaining of cannot find type date for Appointment.
Thanks in advance for your help!
Use below variant of mongo template aggregate which takes collection name instead of class.
mongoTemplate.aggregate(Aggregation.newAggregation(
matchOperation,
groupOperation,
projectionOperation
), "appointment", AppointmentSummary.class).getMappedResults();
The reason was that when you use typed variant spring runs validation on the fields used in the aggregation pipeline to match the field names in the pojo and fails when it doesn't find the alias.