We Keep Coding

How to know the missing items from Spring Data JPA's findAllById method in an efficient way?

Consider this code snippet below:
List<String> usersList = Arrays.asList("john", "jack", "jill", "xxxx", "yyyy");
List<User> userEntities = userRepo.findAllById(usersList);
User class is a simple Entity object annotated with #Entity and has an #Id field which is of String datatype.
Assume that in db I have rows corresponding to "john", "jack" and "jill". Even though I passed 5 items in usersList(along with "xxxx" and "yyyy"), findAllById method would only return 3 items/entities corresponding to "john","jack",and "jill".
Now after the call to findAllById method, what's the best, easy and efficient(better than O(n^2) perhaps) way to find out the missing items which findAllById method did not return?(In this case, it would be "xxxx" and "yyyy").

Using Java Sets
You could use a set as the source of filtering:
Set<String> usersSet = new HashSet<>(Arrays.asList("john", "jack", "jill", "xxxx", "yyyy"));
And now you could create a predicate to filter those not present:
Set<String> foundIds = userRepo.findAllById(usersSet)
.stream()
.map(User::getId)
.collect(Collectors.toSet());
I assume the filter should be O(n) to go over the entire results.
Or you could change your repository to return a set of users ideally using a form of distinct clause:
Set<String> foundIds = userRepo.findDistinctById(usersSet)
.stream()
.map(User::getId)
.collect(Collectors.toSet());;
And then you can just apply a set operator:
usersSet.removeAll(foundIds);
And now usersSet contains the users not found in your result.
And a set has a O(1) complexity to find an item. So, I assume this should be O(sizeOf(userSet)) to remove them all.
Alternatively, you could iterate over the foundIds and gradually remove items from the userSet. Then you could short-circuit the loop algorithm in the event you realize that there are no more userSet items to remove (i.e. the set is empty).
Filtering Directly from Database
Now to avoid all this, you can probably define a native query and run it in your JPA repository to retrieve only users from your list which didn't exist in the database. The query would be somewhat as follows that I did in PostgreSQL:
WITH my_users AS(
SELECT 'john' AS id UNION SELECT 'jack' UNION SELECT 'jill'
)
SELECT id FROM my_users mu
WHERE NOT EXISTS(SELECT 1 FROM users u WHERE u.id = mu.id);
Spring Data: JDBC Example
Since the query is dynamic (i.e. the filtering set could be of different sizes every time), we need to build the query dynamically. And I don't believe JPA has a way to do this, but a native query might do the trick.
You could either pack a JdbcTemplate query directly into your repository or use JPA native queries manually.
#Repository
public class UserRepository {
private final JdbcTemplate jdbcTemplate;
public UserRepository(JdbcTemplate jdbcTemplate) {this.jdbcTemplate = jdbcTemplate;}
public Set<String> getUserIdNotFound(Set<String> userIds) {
StringBuilder sql = new StringBuilder();
for(String userId : userIds) {
if(sql.length() > 0) {
sql.append(" UNION ");
}
sql.append("SELECT ? AS id");
}
String query = String.format("WITH my_users AS (%sql)", sql) +
"SELECT id FROM my_users mu WHERE NOT EXISTS(SELECT 1 FROM users u WHERE u.id = mu.id)";
List<String> myUsers = jdbcTemplate.queryForList(query, userIds.toArray(), String.class);
return new HashSet<>(myUsers);
}
}
Then we just do:
Set<String> usersIds = Set.of("john", "jack", "jill", "xxxx", "yyyy");
Set<String> notFoundIds = userRepo.getUserIdNotFound(usersIds);
There is probably a way to do it with JPA native queries. Let me see if I can do one of those and put it in the answer later on.

You can write your own algorithm that finds missing users. For example:
List<String> missing = new ArrayList<>(usersList);
for (User user : userEntities){
String userId = user.getId();
missing.remove(userId);
}
In the result you will have a list of user-ids that are missing:
"xxxx" and "yyyy"

You can just add a method to your repo:
findByIdNotIn(Collection<String> ids) and Spring will make the query:
See here:
https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#jpa.query-methods
Note (from the docs):
In and NotIn also take any subclass of Collection as aparameter as well as arrays or varargs.

How do we fetch the ResultSet in the desired format when the fields to be fetched are coming as part of the request in spring data jpa?

I have an entity with around 20 columns. At any point in time I don't want to fetch all the columns but only the selected columns and I don't know what could those columns be since it's coming from the request as a parameter. What's the right way to achieve this ?
I have tried implementing this using the native queries but having a hard time parsing the result set to the desired response format.
Query query = entityManager.createNativeQuery("SELECT name, model, desc FROM Product");
query.getResultList();
Now this result set is so generic that I am not able to convert it to the required product model by passing Product.class as a parameter to the query as it fetches only the 3 fields and the rest are missing. And I can't have a predefined projection as it's coming from the API request as a parameter.

Here's how I've done it. Apparently you can get the ResultSet as a list of Tuple. I just had to iterate over this list and make a Map for each entry in result set so I can map it using ModelMapper. Here tags are a list of columns to be read.
Query query = entityManager.createNativeQuery(preparedQuery,Tuple.class);
List<Tuple> resultList = query.getResultList();
List<Product> resultDto = new ArrayList<>();
for (Tuple tuple : resultList) {
HashMap<String, Object> data = new HashMap<>();
for (String tag : tags) {
data.put(tag, tuple.get(tag));
}
resultDto.add(mapper.map(data, Product.class));
}
return resultDto;
Thanks for all who responded to this.

The response you get will be in this format
Map< String,Object >.
Create a ProductDTO class and convert this map into ProductDTO with the
help of ModelMapper.
ProductDTO productDto = new ModelMapper().map(resultMap, ProductDTO.class);

Suppose you want other fields means just add the #JsonInclude annotation at top of the entity class.
#JsonInclude(Include.NON_NULL)
public class Product{
Edit:
Create the Constructor in Product Class
public Product(String name,String model,String desc){
this.name=name;
this.model=model;
this.desc=desc;
}
String query = String.format("select new pakageName.Product(e.name, e.model,e.desc) from Product as e);
Query queryObject = (Query) em.createQuery(query);
List<Product> result = queryObject.getResultList();
Native query only returns the Object. So try this like the below mentioned one.

JPA Hibernate Order by alphanumeric field

I am using Jpa/Hibernate to access MariaDB in a Spring Boot application
I am strugling to sort data by an alphanumeric field containng numbers that might end with one letter (pattern \d+[a-z]?)
e.g.
10
104
20a
100b
and I need them ordered like this
10
20a
100b
104
I a bulding my own query with the Criteria Api because I also have to do some complex filtering.
#Transactional(readOnly = true)
public class EntryRepositoryImpl implements EntryRepositoryCustom {
#PersistenceContext
private EntityManager entityManager;
#Override
public Page<Entry> get(MultiValueMap<String, String> parameters, Pageable pageable) {
CriteriaBuilder criteriaBuilder = entityManager.getCriteriaBuilder();
// count total number of filterd entries
Long totalResultCount = getResultCount(parameters, criteriaBuilder);
// build query to get filterd entries
CriteriaQuery<Entry> selectQuery = criteriaBuilder.createQuery(Entry.class);
Root<Entry> getRoot = selectQuery.from(Entry.class);
CriteriaQuery<Entry> select = selectQuery.select(getRoot);
addFilters(parameters, criteriaBuilder, getRoot, select);
// add sorting
List<javax.persistence.criteria.Order> sortOrders = JpaUtils.translateSorting(pageable,
getRoot);
select.orderBy(sortOrders);
// get one page of filterd entries
List<Entry> results = getPageResults(pageable, select);
return new PageImpl<>(results, pageable, totalResultCount);
}
/**
* Translate spring to jpa sorting.
*/
public static List<javax.persistence.criteria.Order> translateSorting(Pageable pageable,
Root<Entry> root) {
List<Sort.Order> orders = new ArrayList<>();
if (pageable.getSort() != null) {
pageable.getSort().iterator().forEachRemaining(orders::add);
}
return orders.stream().
map(order -> {
String[] parts = order.getProperty().split("\\.");
String field = parts[0];
Path path = parts.length == 2 ? root.join(field).get(parts[1]) : root.get(field);
return new OrderImpl(path, order.isAscending());
})
.collect(Collectors.toList());
}
I already have a custom comparator but it seems, there is no way to translate it so the DB could use it.
So far I found the following solutions/ideas
using #SortComparator, but it is not feasible for my use case because the ordering has to happen in the database, because there are over 500k complex rows.
this sql base solution but don't know how to translate it into the Criteria Api.
after looking at the function of CriteriaBuilder (javadoc) I got the idea to split the value into the numeric and string parts and apply to orders but there is not function to split with a regular expression.
Edit:
For now I did split the field into 2 and use two sort expression.

Spring Data Mongo - Query methods and Distinct field

I'm currently working on a project using Spring Data Mongo.
My repository is just an interface extending MongoRepository. I would like to add a custom query method in order to retrieve all distinct values for one of my collection's fields.
I tried something like this:
#RepositoryRestResource(path = "devices", collectionResourceRel = "deviceInfos")
public interface DeviceInfoRepository extends MongoRepository<DeviceInfo, String> {
#RestResource(path = "distinctUnitIds")
List<String> findDistinctUnitIdBy();
}
With that code, Spring give me an error because it's not able to build my list. So I tried this:
#RepositoryRestResource(path = "devices", collectionResourceRel = "deviceInfos")
public interface DeviceInfoRepository extends MongoRepository<DeviceInfo, String> {
#RestResource(path = "distinctUnitIds")
List<DeviceInfo> findDistinctUnitIdBy();
}
That code works but the distinct seems to be totally ignored.
The documentation about Distinct in query method is really not clear...
Did I do something wrong? What's the best way to solve get the distinct values of a field using Spring Data?
Thanks!

You will have to use Spring Data MongoTemplate - the MongoRepository interfaces are made only for basic functionality and for more fine grain control of what you are querying, its best to use MongoTemplate.
Here is an example of how one would get distinct values from a collection:
Criteria criteria = new Criteria();
criteria.where("dataset").is("d1");
Query query = new Query();
query.addCriteria(criteria);
List list = mongoTemplate.getCollection("collectionName")
.distinct("source",query.getQueryObject());
Here is the link to more info: mongodb mongoTemplate get distinct field with some criteria

in SpringBoot2 you can do the following :
DistinctIterable<String> iterable = mongoTemplate.getCollection(COLLECTION_NAME).distinct("source",in(FieldValue,query.getQueryObject(), String.class);
MongoCursor<String> cursor = iterable.iterator();
List<String> list = new ArrayList<>();
while (cursor.hasNext()) {
list.add(cursor.next());
}
return list;

Getting count and list of ids using ElasticsearchTemplate in spring-data-elasticsearch

I am using spring-data-elasticsearch for a project to provide it with full text search functionality. We keep the real data in a relational database and relevant metadata along with respective id in elasticsearch. So for search results, only id field is required as the actual data will be retrieved from the relational database.
I am building the search query based on search criteria and then performing a queryForIds():
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withIndices(indexName)
.withTypes(typeName)
.withQuery(getQueryBuilder(searchParams))
.withPageable(pageable)
.build();
return elasticsearchTemplate.queryForIds(searchQuery);
If I also need the total count for that specific searchQuery, I can do another elasticsearchTemplate.count(searchQuery) call, but that will be redundant as I understand. I think there is a way to get both the list of id and total count by using something like elasticsearchTemplate.queryForPage() in a single call.
Also, can I use a custom class in queryForPage(SearchQuery query, Class<T> clazz, SearchResultMapper mapper) which is not annotated with #Document? The actual document class is really big, and if I am not sure if passing large classes will cause any extra load on the engine since there are over 100 fields to be json mapped, but all I need is the id field. I will have a .withFields("id") in the query builder anyway.

If you want to prevent two calls to elasticsearch, i would suggest to write an custom ResultsExtractor:
SearchQuery searchQuery = new NativeSearchQueryBuilder().withIndices(indexName)
.withTypes(typeName)
.withQuery(queryBuilder)
.withPageable(pageable)
.build();
SearchResult result = template.query(searchQuery, new ResultsExtractor<SearchResult>() {
#Override
public SearchResult extract(SearchResponse response) {
long totalHits = response.getHits()
.totalHits();
List<String> ids = new ArrayList<String>();
for (SearchHit hit : response.getHits()) {
if (hit != null) {
ids.add(hit.getId());
}
}
return new SearchResult(ids, totalHits);
}
});
System.out.println(result.getIds());
System.out.println(result.getCount());
where SearchResult is a custom class:
public class SearchResult {
List<String> ids;
long count;
//getter and setter
}
This way you can get the information you need from the elasticsearch SearchResponse.
Regarding your second question: As far as I can see, when calling queryForPage(SearchQuery query, Class<T> clazz, SearchResultMapper mapper)
the passed class is not checked for the #Document annotation. Just try it out!

One may also consider using AggregatedPage<T>. You can get the total number of records, total pages, current page records, etc. just like in Pageable<T>.
SearchQuery searchQuery = new NativeSearchQueryBuilder().withIndices(indexName)
.withTypes(typeName)
.withQuery(queryBuilder)
.withPageable(pageable)
.build();
AggregatedPage<ElasticDTO> queryResult = elasticsearchTemplate.queryForPage(searchQuery , ElasticDTO.class);