We Keep Coding

Distinct values ignoring a field in Spring Data Mongodb

In my spring boot app utilizing mongodb I am trying to get a list of distinct nested objects.
The document looks like this (other details removed fore brevity):
class Movie {
otherproperties...;
List<CastRoleLink> cast;
...
}
//structure of CastRoleLink
class CastRoleLink {
String name;
String urlName;
String roleID;
}
//additional class for structure of results
class CastLink {
String name;
String urlName;
}
What I really want is a list of all the unique CastLink from all of the Movies matching my criteria, which means that I need all of the distinct CastRoleLink objects without the roleID. The tricky part is that for different movies I could have the same name and urlName properties but a different roleID property. In these cases I would consider them the same because the CastLink would be the same. My current solution is close but not quite right.
//query is Query object with criteria for Movies
mongoTemplate.findDistinct(query, "cast", Movies.class, CastLink.class);
This gives me duplicates when the name and urlName properties are the same but the roleID property is different. Is there a way that I can find distinct name and urlName objects while ignoring the roleID property?

I figured it out. The key is to use Aggregation.
Aggregation aggregation = newAggregation(
match(criteria),
unwind("cast"),
group("cast.urlName").addToSet("cast.name").as("name").addToSet("cast.urlName").as("urlName")
project("name").and("urlName").previousOpertaion(),
sort(Sort.Direction.ASC, "name")
)
return mongoTemplate.aggregate(aggregation, Movie.class, CastLink.class).getMappedResults();

How to know the missing items from Spring Data JPA's findAllById method in an efficient way?

Consider this code snippet below:
List<String> usersList = Arrays.asList("john", "jack", "jill", "xxxx", "yyyy");
List<User> userEntities = userRepo.findAllById(usersList);
User class is a simple Entity object annotated with #Entity and has an #Id field which is of String datatype.
Assume that in db I have rows corresponding to "john", "jack" and "jill". Even though I passed 5 items in usersList(along with "xxxx" and "yyyy"), findAllById method would only return 3 items/entities corresponding to "john","jack",and "jill".
Now after the call to findAllById method, what's the best, easy and efficient(better than O(n^2) perhaps) way to find out the missing items which findAllById method did not return?(In this case, it would be "xxxx" and "yyyy").

Using Java Sets
You could use a set as the source of filtering:
Set<String> usersSet = new HashSet<>(Arrays.asList("john", "jack", "jill", "xxxx", "yyyy"));
And now you could create a predicate to filter those not present:
Set<String> foundIds = userRepo.findAllById(usersSet)
.stream()
.map(User::getId)
.collect(Collectors.toSet());
I assume the filter should be O(n) to go over the entire results.
Or you could change your repository to return a set of users ideally using a form of distinct clause:
Set<String> foundIds = userRepo.findDistinctById(usersSet)
.stream()
.map(User::getId)
.collect(Collectors.toSet());;
And then you can just apply a set operator:
usersSet.removeAll(foundIds);
And now usersSet contains the users not found in your result.
And a set has a O(1) complexity to find an item. So, I assume this should be O(sizeOf(userSet)) to remove them all.
Alternatively, you could iterate over the foundIds and gradually remove items from the userSet. Then you could short-circuit the loop algorithm in the event you realize that there are no more userSet items to remove (i.e. the set is empty).
Filtering Directly from Database
Now to avoid all this, you can probably define a native query and run it in your JPA repository to retrieve only users from your list which didn't exist in the database. The query would be somewhat as follows that I did in PostgreSQL:
WITH my_users AS(
SELECT 'john' AS id UNION SELECT 'jack' UNION SELECT 'jill'
)
SELECT id FROM my_users mu
WHERE NOT EXISTS(SELECT 1 FROM users u WHERE u.id = mu.id);
Spring Data: JDBC Example
Since the query is dynamic (i.e. the filtering set could be of different sizes every time), we need to build the query dynamically. And I don't believe JPA has a way to do this, but a native query might do the trick.
You could either pack a JdbcTemplate query directly into your repository or use JPA native queries manually.
#Repository
public class UserRepository {
private final JdbcTemplate jdbcTemplate;
public UserRepository(JdbcTemplate jdbcTemplate) {this.jdbcTemplate = jdbcTemplate;}
public Set<String> getUserIdNotFound(Set<String> userIds) {
StringBuilder sql = new StringBuilder();
for(String userId : userIds) {
if(sql.length() > 0) {
sql.append(" UNION ");
}
sql.append("SELECT ? AS id");
}
String query = String.format("WITH my_users AS (%sql)", sql) +
"SELECT id FROM my_users mu WHERE NOT EXISTS(SELECT 1 FROM users u WHERE u.id = mu.id)";
List<String> myUsers = jdbcTemplate.queryForList(query, userIds.toArray(), String.class);
return new HashSet<>(myUsers);
}
}
Then we just do:
Set<String> usersIds = Set.of("john", "jack", "jill", "xxxx", "yyyy");
Set<String> notFoundIds = userRepo.getUserIdNotFound(usersIds);
There is probably a way to do it with JPA native queries. Let me see if I can do one of those and put it in the answer later on.

You can write your own algorithm that finds missing users. For example:
List<String> missing = new ArrayList<>(usersList);
for (User user : userEntities){
String userId = user.getId();
missing.remove(userId);
}
In the result you will have a list of user-ids that are missing:
"xxxx" and "yyyy"

You can just add a method to your repo:
findByIdNotIn(Collection<String> ids) and Spring will make the query:
See here:
https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#jpa.query-methods
Note (from the docs):
In and NotIn also take any subclass of Collection as aparameter as well as arrays or varargs.

Dynamic search term SQL query with Spring JPA or QueryDSL

I am trying to learn QueryDSL in order to return the results of a single search term from Postgres:
#GetMapping("/product/contains/{matchingWords}")
public List<ProductModel> findByTitleContaining(#PathVariable String matchingWords) {
QProductModel product = QProductModel.productModel;
JPAQuery<?> query = new JPAQuery<>(em);
List<ProductModel> products = query.select(product)
.from(product)
.where(product.title.toLowerCase()
.contains(matchingWords.toLowerCase()))
.fetch();
return products;
}
But I also want to search for any number of search terms, for example, say this is my list of search terms divided by the plus symbol:
String[] params = matchingWords.split("[+]");
How can I dynamically create contains(params[0]) AND/OR contains(params[1] AND/OR ... contains(params[n]) using either QueryDSL or any Java/Spring query framework? I see QueryDSL has a predicate system, but I still don't understand how to dynamically create a query based on a variable number of parameters searching in the same column.

I figured it out. It's a little non-intuitive, but using BooleanBuilder and JPAQuery<?> you can create a dynamic series of boolean predicates, which return a list of matches.
Example:
QProductModel product = QProductModel.productModel;
JPAQuery<?> query = new JPAQuery<>(//entity manager goes here//);
// example list of search parameters separated by +
String[] params = matchingWords.split("[+]");
BooleanBuilder builder = new BooleanBuilder();
for(String param : params) {
builder.or(product.title.containsIgnoreCase(param));
}
// then you can put together the query like so:
List<ProductModel> products = query.select(product)
.from(product)
.where(builder)
.fetch();
return products;

How do we fetch the ResultSet in the desired format when the fields to be fetched are coming as part of the request in spring data jpa?

I have an entity with around 20 columns. At any point in time I don't want to fetch all the columns but only the selected columns and I don't know what could those columns be since it's coming from the request as a parameter. What's the right way to achieve this ?
I have tried implementing this using the native queries but having a hard time parsing the result set to the desired response format.
Query query = entityManager.createNativeQuery("SELECT name, model, desc FROM Product");
query.getResultList();
Now this result set is so generic that I am not able to convert it to the required product model by passing Product.class as a parameter to the query as it fetches only the 3 fields and the rest are missing. And I can't have a predefined projection as it's coming from the API request as a parameter.

Here's how I've done it. Apparently you can get the ResultSet as a list of Tuple. I just had to iterate over this list and make a Map for each entry in result set so I can map it using ModelMapper. Here tags are a list of columns to be read.
Query query = entityManager.createNativeQuery(preparedQuery,Tuple.class);
List<Tuple> resultList = query.getResultList();
List<Product> resultDto = new ArrayList<>();
for (Tuple tuple : resultList) {
HashMap<String, Object> data = new HashMap<>();
for (String tag : tags) {
data.put(tag, tuple.get(tag));
}
resultDto.add(mapper.map(data, Product.class));
}
return resultDto;
Thanks for all who responded to this.

The response you get will be in this format
Map< String,Object >.
Create a ProductDTO class and convert this map into ProductDTO with the
help of ModelMapper.
ProductDTO productDto = new ModelMapper().map(resultMap, ProductDTO.class);

Suppose you want other fields means just add the #JsonInclude annotation at top of the entity class.
#JsonInclude(Include.NON_NULL)
public class Product{
Edit:
Create the Constructor in Product Class
public Product(String name,String model,String desc){
this.name=name;
this.model=model;
this.desc=desc;
}
String query = String.format("select new pakageName.Product(e.name, e.model,e.desc) from Product as e);
Query queryObject = (Query) em.createQuery(query);
List<Product> result = queryObject.getResultList();
Native query only returns the Object. So try this like the below mentioned one.

Spring mongo repository slice

I am using spring-sata-mongodb 1.8.2 with MongoRepository and I am trying to use the mongo $slice option to limit a list size when query, but I can't find this option in the mongorepository.
my classes look like this:
public class InnerField{
public String a;
public String b;
public int n;
}
#Document(collection="Record")
punlic class Record{
public ObjectId id;
public List<InnerField> fields;
public int numer;
}
As you can see I have one collection name "Record" and the document contains the InnerField. the InnerField list is growing all the time so i want to limit the number of the selected fields when I am querying.
I saw that: https://docs.mongodb.org/v3.0/tutorial/project-fields-from-query-results/
which is exactly what I need but I couldn't find the relevant reference in mongorepository.
Any ideas?

Providing an abstraction for the $slice operator in Query is still an open issue. Please vote for DATAMONGO-1230 and help us prioritize.
For now you still can fall back to using BasicQuery.
String qry = "{ \"_id\" : \"record-id\"}";
String fields = "{\"fields\": { \"$slice\": 2} }";
BasicQuery query = new BasicQuery(qry, fields);

Use slice functionality as provided in Java Mongo driver using projection as in below code.
For Example:
List<Entity> list = new ArrayList<Entity>();
// Return the last 10 weeks data only
FindIterable<Document> list = db.getDBCollection("COLLECTION").find()
.projection(Projections.fields(Projections.slice("count", -10)));
MongoCursor<Document> doc = list.iterator();
while(doc.hasNext()){
list.add(new Gson().fromJson(doc.next().toJson(), Entity.class));
}
The above query will fetch all documents of type Entity class and the "field" list of each Entity class document will have only last 10 records.

I found in unit test file (DATAMONGO-1457) way to use slice. Some thing like this.
newAggregation(
UserWithLikes.class,
match(new Criteria()),
project().and("likes").slice(2)
);