We Keep Coding

How to know the missing items from Spring Data JPA's findAllById method in an efficient way?

Consider this code snippet below:
List<String> usersList = Arrays.asList("john", "jack", "jill", "xxxx", "yyyy");
List<User> userEntities = userRepo.findAllById(usersList);
User class is a simple Entity object annotated with #Entity and has an #Id field which is of String datatype.
Assume that in db I have rows corresponding to "john", "jack" and "jill". Even though I passed 5 items in usersList(along with "xxxx" and "yyyy"), findAllById method would only return 3 items/entities corresponding to "john","jack",and "jill".
Now after the call to findAllById method, what's the best, easy and efficient(better than O(n^2) perhaps) way to find out the missing items which findAllById method did not return?(In this case, it would be "xxxx" and "yyyy").

Using Java Sets
You could use a set as the source of filtering:
Set<String> usersSet = new HashSet<>(Arrays.asList("john", "jack", "jill", "xxxx", "yyyy"));
And now you could create a predicate to filter those not present:
Set<String> foundIds = userRepo.findAllById(usersSet)
.stream()
.map(User::getId)
.collect(Collectors.toSet());
I assume the filter should be O(n) to go over the entire results.
Or you could change your repository to return a set of users ideally using a form of distinct clause:
Set<String> foundIds = userRepo.findDistinctById(usersSet)
.stream()
.map(User::getId)
.collect(Collectors.toSet());;
And then you can just apply a set operator:
usersSet.removeAll(foundIds);
And now usersSet contains the users not found in your result.
And a set has a O(1) complexity to find an item. So, I assume this should be O(sizeOf(userSet)) to remove them all.
Alternatively, you could iterate over the foundIds and gradually remove items from the userSet. Then you could short-circuit the loop algorithm in the event you realize that there are no more userSet items to remove (i.e. the set is empty).
Filtering Directly from Database
Now to avoid all this, you can probably define a native query and run it in your JPA repository to retrieve only users from your list which didn't exist in the database. The query would be somewhat as follows that I did in PostgreSQL:
WITH my_users AS(
SELECT 'john' AS id UNION SELECT 'jack' UNION SELECT 'jill'
)
SELECT id FROM my_users mu
WHERE NOT EXISTS(SELECT 1 FROM users u WHERE u.id = mu.id);
Spring Data: JDBC Example
Since the query is dynamic (i.e. the filtering set could be of different sizes every time), we need to build the query dynamically. And I don't believe JPA has a way to do this, but a native query might do the trick.
You could either pack a JdbcTemplate query directly into your repository or use JPA native queries manually.
#Repository
public class UserRepository {
private final JdbcTemplate jdbcTemplate;
public UserRepository(JdbcTemplate jdbcTemplate) {this.jdbcTemplate = jdbcTemplate;}
public Set<String> getUserIdNotFound(Set<String> userIds) {
StringBuilder sql = new StringBuilder();
for(String userId : userIds) {
if(sql.length() > 0) {
sql.append(" UNION ");
}
sql.append("SELECT ? AS id");
}
String query = String.format("WITH my_users AS (%sql)", sql) +
"SELECT id FROM my_users mu WHERE NOT EXISTS(SELECT 1 FROM users u WHERE u.id = mu.id)";
List<String> myUsers = jdbcTemplate.queryForList(query, userIds.toArray(), String.class);
return new HashSet<>(myUsers);
}
}
Then we just do:
Set<String> usersIds = Set.of("john", "jack", "jill", "xxxx", "yyyy");
Set<String> notFoundIds = userRepo.getUserIdNotFound(usersIds);
There is probably a way to do it with JPA native queries. Let me see if I can do one of those and put it in the answer later on.

You can write your own algorithm that finds missing users. For example:
List<String> missing = new ArrayList<>(usersList);
for (User user : userEntities){
String userId = user.getId();
missing.remove(userId);
}
In the result you will have a list of user-ids that are missing:
"xxxx" and "yyyy"

You can just add a method to your repo:
findByIdNotIn(Collection<String> ids) and Spring will make the query:
See here:
https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#jpa.query-methods
Note (from the docs):
In and NotIn also take any subclass of Collection as aparameter as well as arrays or varargs.

What is the LIMIT clause alternative in JPQL?

I'm working with PostgreSQL query implementing in JPQL.
This is a sample native psql query which works fine,
SELECT * FROM students ORDER BY id DESC LIMIT 1;
The same query in JPQL doesnt work,
#Query("SELECT s FROM Students s ORDER BY s.id DESC LIMIT 1")
Students getLastStudentDetails();
seems like LIMIT clause doesn't work in JPQL.
According to JPA documentation we can use setMaxResults/setFirstResult, Can anyone tell me how can I use that in my above query?

You are using JPQL which doesn't support limiting results like this. When using native JPQL you should use setMaxResults to limit the results.
However you are using Spring Data JPA which basically makes it pretty easy to do. See here in the reference guide on how to limit results based on a query. In your case the following, find method would do exactly what you want.
findFirstByOrderById();
You could also use a Pageable argument with your query instead of a LIMIT clause.
#Query("SELECT s FROM Students s ORDER BY s.id DESC")
List<Students> getLastStudentDetails(Pageable pageable);
Then in your calling code do something like this (as explained here in the reference guide).
getLastStudentDetails(PageRequest.of(0,1));
Both should yield the same result, without needing to resort to plain SQL.

As stated in the comments, JPQL does not support the LIMIT keyword.
You can achieve that using the setMaxResults but if what you want is just a single item, then use the getSingleResult - it throws an exception if no item is found.
So, your query would be something like:
TypedQuery<Student> query = entityManager.createQuery("SELECT s FROM Students s ORDER BY s.id DESC", Student.class);
query.setMaxResults(1);
If you want to set a specific start offset, use query.setFirstResult(initPosition); too

Hello for fetching single row and using LIMIT in jpql we can tell the jpql if it's a native query.
( using - nativeQuery=true )
Below is the use
#Query("SELECT s FROM Students s ORDER BY s.id DESC LIMIT 1", nativeQuery=true)
Students getLastStudentDetails();

You can not use Limit in HQL because Limit is database vendor dependent so Hibernate doesn't allow it through HQL query.
A way you can implement is using a subquery:
#Query("FROM Students st WHERE st.id = (SELECT max(s.id) FROM Students s)")
Students getLastStudentDetails();

The correct way is to write your JPA interface method like this
public interface MyRepository extends PagingAndSortingRepository<EntityClass, KeyClass> {
List<EntityClass> findTop100ByOrderByLastModifiedDesc();
}
In the method name, "100" denotes how many rows you want which you would have otherwise put in the limit clause. also "LastModified" is the column which you want to sort by.
PagingAndSortingRepository or CrudRepository, both will work for this.
For the sake of completeness, OP's interface method would be
List<Students> findTop1ByIdDesc();

JPQL does not allow to add the limit keyword to the query generated by the HQL. You would get the following exception.
org.hibernate.hql.internal.ast.QuerySyntaxException: unexpected token:
LIMIT near line 1
But don't worry there is an alternative to use the limit keyword in the query generated by the HQL by using the following steps.
Sort.by(sortBy).descending() // fetch the records in descending order
pageSize = 1 // fetch the first record from the descending order result set.
Refer the following service class
Service:
#Autowired
StudentRepository repository;
public List<Student> getLastStudentDetails(Integer pageNo, Integer pageSize, String sortBy)
{
Integer pageNo = 0;
Integer pageSize = 1;
String sortBy = "id";
Pageable paging = PageRequest.of(pageNo, pageSize, Sort.by(sortBy).descending());
Slice<Student> pagedResult = repository.findLastStudent(paging);
return pagedResult.getContent();
}
Your repository interface should implement the PagingAndSortingRepository
Repository:
public interface StudentRepository extends JpaRepository<Student,Long>, PagingAndSortingRepository<Student,Long>{
#Query("select student from Student student")
Slice<Student> findLastStudent(Pageable paging);
}
This will add the limit keyword to you query which you can see in the console. Hope this helps.

Hardcode the pagination(new PageRequest(0, 1)) to achieve fetch only one record.
#QueryHints({ #QueryHint(name = "org.hibernate.cacheable", value = "true") })
#Query("select * from a_table order by a_table_column desc")
List<String> getStringValue(Pageable pageable);
you have to pass new PageRequest(0, 1)to fetch records and from the list fetch the first record.

Here a Top Ten Service (it's a useful example)
REPOSITORY
(In the Query, I parse the score entity to ScoreTo ( DTO class) by a constructor)
#Repository
public interface ScoreRepository extends JpaRepository<Scores, UUID> {
#Query("SELECT new com.example.parameters.model.to.ScoreTo(u.scoreId , u.level, u.userEmail, u.scoreLearningPoints, u.scoreExperiencePoints, u.scoreCommunityPoints, u.scoreTeamworkPoints, u.scoreCommunicationPoints, u.scoreTotalPoints) FROM Scores u "+
"order by u.scoreTotalPoints desc")
List<ScoreTo> findTopScore(Pageable pageable);
}
SERVICE
#Service
public class ScoreService {
#Autowired
private ScoreRepository scoreRepository;
public List<ScoreTo> getTopScores(){
return scoreRepository.findTopScore(PageRequest.of(0,10));
}
}

You can use something like this:
#Repository
public interface ICustomerMasterRepository extends CrudRepository<CustomerMaster, String>
{
#Query(value = "SELECT max(c.customer_id) FROM CustomerMaster c ")
public String getMaxId();
}

As your query is simple, you can use the solution of the accepted answer, naming your query findFirstByOrderById();
But if your query is more complicated, I also found this way without need to use a native query:
#Query("SELECT MAX(s) FROM Students s ORDER BY s.id DESC")
Students getLastStudentDetails();
Here a practical example where the named query method cannot be used.

Spring Data Mongo - Query methods and Distinct field

I'm currently working on a project using Spring Data Mongo.
My repository is just an interface extending MongoRepository. I would like to add a custom query method in order to retrieve all distinct values for one of my collection's fields.
I tried something like this:
#RepositoryRestResource(path = "devices", collectionResourceRel = "deviceInfos")
public interface DeviceInfoRepository extends MongoRepository<DeviceInfo, String> {
#RestResource(path = "distinctUnitIds")
List<String> findDistinctUnitIdBy();
}
With that code, Spring give me an error because it's not able to build my list. So I tried this:
#RepositoryRestResource(path = "devices", collectionResourceRel = "deviceInfos")
public interface DeviceInfoRepository extends MongoRepository<DeviceInfo, String> {
#RestResource(path = "distinctUnitIds")
List<DeviceInfo> findDistinctUnitIdBy();
}
That code works but the distinct seems to be totally ignored.
The documentation about Distinct in query method is really not clear...
Did I do something wrong? What's the best way to solve get the distinct values of a field using Spring Data?
Thanks!

You will have to use Spring Data MongoTemplate - the MongoRepository interfaces are made only for basic functionality and for more fine grain control of what you are querying, its best to use MongoTemplate.
Here is an example of how one would get distinct values from a collection:
Criteria criteria = new Criteria();
criteria.where("dataset").is("d1");
Query query = new Query();
query.addCriteria(criteria);
List list = mongoTemplate.getCollection("collectionName")
.distinct("source",query.getQueryObject());
Here is the link to more info: mongodb mongoTemplate get distinct field with some criteria

in SpringBoot2 you can do the following :
DistinctIterable<String> iterable = mongoTemplate.getCollection(COLLECTION_NAME).distinct("source",in(FieldValue,query.getQueryObject(), String.class);
MongoCursor<String> cursor = iterable.iterator();
List<String> list = new ArrayList<>();
while (cursor.hasNext()) {
list.add(cursor.next());
}
return list;

Spring mongo repository slice

I am using spring-sata-mongodb 1.8.2 with MongoRepository and I am trying to use the mongo $slice option to limit a list size when query, but I can't find this option in the mongorepository.
my classes look like this:
public class InnerField{
public String a;
public String b;
public int n;
}
#Document(collection="Record")
punlic class Record{
public ObjectId id;
public List<InnerField> fields;
public int numer;
}
As you can see I have one collection name "Record" and the document contains the InnerField. the InnerField list is growing all the time so i want to limit the number of the selected fields when I am querying.
I saw that: https://docs.mongodb.org/v3.0/tutorial/project-fields-from-query-results/
which is exactly what I need but I couldn't find the relevant reference in mongorepository.
Any ideas?

Providing an abstraction for the $slice operator in Query is still an open issue. Please vote for DATAMONGO-1230 and help us prioritize.
For now you still can fall back to using BasicQuery.
String qry = "{ \"_id\" : \"record-id\"}";
String fields = "{\"fields\": { \"$slice\": 2} }";
BasicQuery query = new BasicQuery(qry, fields);

Use slice functionality as provided in Java Mongo driver using projection as in below code.
For Example:
List<Entity> list = new ArrayList<Entity>();
// Return the last 10 weeks data only
FindIterable<Document> list = db.getDBCollection("COLLECTION").find()
.projection(Projections.fields(Projections.slice("count", -10)));
MongoCursor<Document> doc = list.iterator();
while(doc.hasNext()){
list.add(new Gson().fromJson(doc.next().toJson(), Entity.class));
}
The above query will fetch all documents of type Entity class and the "field" list of each Entity class document will have only last 10 records.

I found in unit test file (DATAMONGO-1457) way to use slice. Some thing like this.
newAggregation(
UserWithLikes.class,
match(new Criteria()),
project().and("likes").slice(2)
);

Return result as Map from JdbcTemplate

I was wondering if there is easy way to generate Map for the results from the query using JdbcTemplate. I have a query which returns two columns so I want to make map from the result directly.
SQL query:
SELECT id, name FROM mytable WHERE somecol='someval';
Expected map:
{123,ABC},{456,DEF} .... {890, XYZ}

Yes, it is possible to return a map of column-name: column-value pairs using JdbcTemplate, but what you have described in your question is not an actual map.
public List<Map<String, Object>> getList() {
return this.jdbcTemplate.queryForList("SELECT id, name FROM mytable");
}
Would return:
[{'id': 123, 'name': 'ABC'}, {...}, ... ]
See the official documentation for more.

I think that you can use this instead of JDBCTemplate. And use method queryForMap.
And your param put like this:
SqlParameterSource namedParameters = new MapSqlParameterSource("somecol",someval);