We Keep Coding

How to know the missing items from Spring Data JPA's findAllById method in an efficient way?

Consider this code snippet below:
List<String> usersList = Arrays.asList("john", "jack", "jill", "xxxx", "yyyy");
List<User> userEntities = userRepo.findAllById(usersList);
User class is a simple Entity object annotated with #Entity and has an #Id field which is of String datatype.
Assume that in db I have rows corresponding to "john", "jack" and "jill". Even though I passed 5 items in usersList(along with "xxxx" and "yyyy"), findAllById method would only return 3 items/entities corresponding to "john","jack",and "jill".
Now after the call to findAllById method, what's the best, easy and efficient(better than O(n^2) perhaps) way to find out the missing items which findAllById method did not return?(In this case, it would be "xxxx" and "yyyy").

Using Java Sets
You could use a set as the source of filtering:
Set<String> usersSet = new HashSet<>(Arrays.asList("john", "jack", "jill", "xxxx", "yyyy"));
And now you could create a predicate to filter those not present:
Set<String> foundIds = userRepo.findAllById(usersSet)
.stream()
.map(User::getId)
.collect(Collectors.toSet());
I assume the filter should be O(n) to go over the entire results.
Or you could change your repository to return a set of users ideally using a form of distinct clause:
Set<String> foundIds = userRepo.findDistinctById(usersSet)
.stream()
.map(User::getId)
.collect(Collectors.toSet());;
And then you can just apply a set operator:
usersSet.removeAll(foundIds);
And now usersSet contains the users not found in your result.
And a set has a O(1) complexity to find an item. So, I assume this should be O(sizeOf(userSet)) to remove them all.
Alternatively, you could iterate over the foundIds and gradually remove items from the userSet. Then you could short-circuit the loop algorithm in the event you realize that there are no more userSet items to remove (i.e. the set is empty).
Filtering Directly from Database
Now to avoid all this, you can probably define a native query and run it in your JPA repository to retrieve only users from your list which didn't exist in the database. The query would be somewhat as follows that I did in PostgreSQL:
WITH my_users AS(
SELECT 'john' AS id UNION SELECT 'jack' UNION SELECT 'jill'
)
SELECT id FROM my_users mu
WHERE NOT EXISTS(SELECT 1 FROM users u WHERE u.id = mu.id);
Spring Data: JDBC Example
Since the query is dynamic (i.e. the filtering set could be of different sizes every time), we need to build the query dynamically. And I don't believe JPA has a way to do this, but a native query might do the trick.
You could either pack a JdbcTemplate query directly into your repository or use JPA native queries manually.
#Repository
public class UserRepository {
private final JdbcTemplate jdbcTemplate;
public UserRepository(JdbcTemplate jdbcTemplate) {this.jdbcTemplate = jdbcTemplate;}
public Set<String> getUserIdNotFound(Set<String> userIds) {
StringBuilder sql = new StringBuilder();
for(String userId : userIds) {
if(sql.length() > 0) {
sql.append(" UNION ");
}
sql.append("SELECT ? AS id");
}
String query = String.format("WITH my_users AS (%sql)", sql) +
"SELECT id FROM my_users mu WHERE NOT EXISTS(SELECT 1 FROM users u WHERE u.id = mu.id)";
List<String> myUsers = jdbcTemplate.queryForList(query, userIds.toArray(), String.class);
return new HashSet<>(myUsers);
}
}
Then we just do:
Set<String> usersIds = Set.of("john", "jack", "jill", "xxxx", "yyyy");
Set<String> notFoundIds = userRepo.getUserIdNotFound(usersIds);
There is probably a way to do it with JPA native queries. Let me see if I can do one of those and put it in the answer later on.

You can write your own algorithm that finds missing users. For example:
List<String> missing = new ArrayList<>(usersList);
for (User user : userEntities){
String userId = user.getId();
missing.remove(userId);
}
In the result you will have a list of user-ids that are missing:
"xxxx" and "yyyy"

You can just add a method to your repo:
findByIdNotIn(Collection<String> ids) and Spring will make the query:
See here:
https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#jpa.query-methods
Note (from the docs):
In and NotIn also take any subclass of Collection as aparameter as well as arrays or varargs.

Storing the keysets from a JPQL query result in a java list

I was successfully able to execute a jpql query and print the result which is stored in a queryResults variable. What I want to achieve next is storing just the IDs (primary key column) in a list without the date (value), but I am not too sure if this is possible; perhaps using something like a java map. Is it possible? If yes, how can this be easily achieved?
private static final TestDao Test_DAO = new TestDao();
#Test
public void testById() {
List<TestEntity> queryResults = TEST_DAO.findById(""); //The record from the sql query is stored in queryResults and findById("") is the method that executes the query in a TestDao class and it is called here
for (TestEntity qResult: queryResults) { // looping through the query result to print the rows
System.out.println(qResult.getId());
System.out.println(qResult.getDate());
}
System.out.println("This is the sql result " + queryResults );
}
Output:
This is the result [TestEntity(id=101, date=2020-01-19 15:12:32.447), TestEntity(id=102, date=2020-09-01 11:04:10.0)]// I want to get the IDs 101 and 102 and store in a list without the Dates
I tried using a map this way:
Map<Integer, Timestamp> map= (Map<Integer, Timestamp>) queryResults.get(0); but I got an exception:
java.lang.ClassCastException: TestEntity cannot be cast to java.util.Map

There are some points before the implementation.
Why are you defining DAO as static? I think this is a bad implementation unless I am missing a particular reason you declared it static. You should define this as a member variable and not a static member
The naming of the method - findById() translated in English is - find Something by this Id, but you are fetching a list of Records, so naming is not correct.
Point 2 becomes invalid if ID property is not a Primary Key in your table, then it makes sense, but still naming is bad. Id is something we use to define Primary Key in the Database and should be and will be unique. But your comments suggest that ID is unique and the Primary Key. So read about how Databases work
And even if not unique, if you pass an Id to find some records, why will get different ids in the Records !!!
About implementation:
Changing in your existing code:
private TestDao Test_DAO = new TestDao();
#Test
public void testById() {
List<TestEntity> queryResults = TEST_DAO.findById("");
List<Long> listOfIds = new ArrayList<>(); // Assuming Id is Long type, same logic for any type
for (TestEntity qResult: queryResults) {
System.out.println(qResult.getId());
listOfIds.add(qResult.getId()); // Just add it to the list
System.out.println(qResult.getDate());
}
}
In case you want to be efficient with the query:
You can use JPQL and hibernate
You can then write a query like:
String query = "select te.id from TestEntity te";
// Create the TypedQuery using EntityManager and then get ResultSet back
List<Long> ids = query.getResultList();
In case of using Spring-Data-Jpa, you can define the repository and define the method and pass the query with #Query annotation. Spring Data JPA

Using SpringBatch JdbcCursorItemReader with List as NamedParameters

I am using below bean definition to configure a reader to read some data from the database table in a Spring Batch project. It is using a named param in SQL. I am passing A java.util.List as a parameter. However, I am getting Invalid Column type error when it tries to run the SQL.
If I just hard code one single value (namedParameters.put("keys", "138219"); ) instead of passing a list, it works.
#Bean
public JdbcCursorItemReader<MyDTO> myReader() {
JdbcCursorItemReader<MyDTO> itemReader = new JdbcCursorItemReader<>();
itemReader.setDataSource(myDatasource);
itemReader.setRowMapper(return new RowMapper<MyDTO>() {
#Override
public MyDTO mapRow(ResultSet resultSet, int rowNum) throws SQLException {
return toMyDto(resultSet);
}
};);
Map<String, Object> namedParameters = new HashMap<>();
List<Long> keys= //Some List
Map<String, List<Long>> singletonMap = Collections.singletonMap("keys", keys);
namedParameters.putAll(singletonMap);
itemReader.setSql(NamedParameterUtils.substituteNamedParameters("SELECT A FROM MYTABLE WHERE KEY IN (:keys)",new MapSqlParameterSource(namedParameters)));
ListPreparedStatementSetter listPreparedStatementSetter = new ListPreparedStatementSetter();
listPreparedStatementSetter.setParameters(
Arrays.asList(NamedParameterUtils.buildValueArray("SELECT A FROM MYTABLE WHERE KEY IN (:keys)", namedParameters)));
itemReader.setPreparedStatementSetter(listPreparedStatementSetter);
return itemReader;
}
I referred sample code snippet here as a response to one of the questions asked - it is what seems to be working when we pass one value. However, my issue is with passing a list instead of one value in the param. This is where it seems to fail.

The ListPreparedStatementSetter is not aware of parameters types. If a parameter is an array or a collection, it will set it as is to the first placeholder, leaving other placeholders unset. Hence the error. In your example, if List<Long> keys = Arrays.asList(1, 2), your sql statement will be:
SELECT A FROM MYTABLE WHERE KEY IN (?, ?)
If you pass your singletonMap to the ListPreparedStatementSetter, it will set the value of keys (which is of type List) to the first placeholder and that's it. The second placeholder will still be unset and the preparation of the statement will fail.
You can flatten parameters before passing them to the ListPreparedStatementSetter and it should work fine. I added a sample with how to flatten parameters before passing them to the prepared statement setter here (See flatten method).
Hope this helps.

How to best map results from an SQL query to a non-entity Java object using Hibernate?

I have a Hibernate managed Java entity called X and a native SQL function (myfunc) that I call from a Hibernate SQL query along these lines:
SQLQuery q = hibernateSession.createSQLQuery(
"SELECT *, myfunc(:param) as result from X_table_name"
);
What I want to do is to map the everything returned from this query to a class (not necessarily managed by Hibernate) called Y. Y should contain all properties/fields from X plus the result returned by myfunc, e.g. Y could extend class X and add a "result" field.
What I've tried:
I've tried using q.addEntity(Y.class) but this fails with:
org.hibernate.MappingException: Unknown entity com.mycompany.Y
q.setResultTransformer(Transformers.aliasToBean(Y.class)); but this fails with: org.hibernate.PropertyNotFoundException: Could not find setter for some_property. X has a field called someProperty with the appropriate getter and setter but in this case it doesn't seem like Hibernate maps the column name (some_property) to the correct field name.
q.setResultTransformer(Criteria.ALIAS_TO_ENTITY_MAP); returns a Map but the values are not always of the type expected by the corresponding field in X. For example fields in X of type enum and Date cannot be mapped directly from the Map returned by the SQL query (where they are Strings).
What's the appropriate way to deal with this situation?

See the chapter of the documentation about SQL queries.
You can use the addScalar() method to specify which type Hibernat should use for a given column.
And you can use aliases to map the results with the bean properties:
select t.some_property as someProperty, ..., myfunc(:param) as result from X_table_name t
Or, (and although it require some lines of code, it's my preferred solution), you can simply do the mapping yourself:
List<Object[]> rows = query.list();
for (Object[] row : rows) {
Foo foo = new Foo((Long) row[0], (String) row[1], ...);
}
This avoids reflection, and lets you control everything.

Easy. Cast the rows to Map<String, Object>:
final org.hibernate.Query q = session.createSQLQuery(sql);
q.setParameter("geo", geo);
q.setResultTransformer(Transformers.ALIAS_TO_ENTITY_MAP);
final List<Map<String, Object>> src = q.list();
final List<VideoEntry> results = new ArrayList<VideoEntry>(src.size());
for (final Map<String, Object> map:src) {
final VideoEntry entry = new VideoEntry();
BeanUtils.populate(entry, map);
results.add(entry);
}

First of all you need to declare the entity in the hibernate configuration xml file something like this: .....
class="path to your entity"
Or you can do the same thing programatically before you make the query.

How to get Map data using JDBCTemplate.queryForMap

How to load data from JDBCTemplate.queryForMap() and it returns the Map Interface.How the maintained the query data internally in map.I trying to load but i got below exception i.e., org.springframework.dao.IncorrectResultSizeDataAccessException: Incorrect result
Code:-
public List getUserInfoByAlll() {
List profilelist=new ArrayList();
Map m=new HashMap();
m=this.jdbctemplate.queryForMap("SELECT userid,username FROM USER");
Set s=m.keySet();
Iterator it=s.iterator();
while(it.hasNext()){
String its=(String)it.next();
Object ob=(Object)m.get(its);
log.info("UserDAOImpl::getUserListSize()"+ob);
}
return profilelist;
}
Plz help me

queryForMap is appropriate if you want to get a single row. You are selecting without a where clause, so you probably want to queryForList. The error is probably indicative of the fact that queryForMap wants one row, but you query is retrieving many rows.
Check out the docs. There is a queryForList that takes just sql; the return type is a
List<Map<String,Object>>.
So once you have the results, you can do what you are doing. I would do something like
List results = template.queryForList(sql);
for (Map m : results){
m.get('userid');
m.get('username');
}
I'll let you fill in the details, but I would not iterate over keys in this case. I like to explicit about what I am expecting.
If you have a User object, and you actually want to load User instances, you can use the queryForList that takes sql and a class type
queryForList(String sql, Class<T> elementType)
(wow Spring has changed a lot since I left Javaland.)

I know this is really old, but this is the simplest way to query for Map.
Simply implement the ResultSetExtractor interface to define what type you want to return. Below is an example of how to use this. You'll be mapping it manually, but for a simple map, it should be straightforward.
jdbcTemplate.query("select string1,string2 from table where x=1", new ResultSetExtractor<Map>(){
#Override
public Map extractData(ResultSet rs) throws SQLException,DataAccessException {
HashMap<String,String> mapRet= new HashMap<String,String>();
while(rs.next()){
mapRet.put(rs.getString("string1"),rs.getString("string2"));
}
return mapRet;
}
});
This will give you a return type of Map that has multiple rows (however many your query returned) and not a list of Maps. You can view the ResultSetExtractor docs here: http://docs.spring.io/spring-framework/docs/2.5.6/api/org/springframework/jdbc/core/ResultSetExtractor.html

To add to #BrianBeech's answer, this is even more trimmed down in java 8:
jdbcTemplate.query("select string1,string2 from table where x=1", (ResultSet rs) -> {
HashMap<String,String> results = new HashMap<>();
while (rs.next()) {
results.put(rs.getString("string1"), rs.getString("string2"));
}
return results;
});

You can do something like this.
List<Map<String, Object>> mapList = jdbctemplate.queryForList(query));
return mapList.stream().collect(Collectors.toMap(k -> (Long) k.get("userid"), k -> (String) k.get("username")));
Output:
{
1: "abc",
2: "def",
3: "ghi"
}