We Keep Coding

How to query DynamoDB and return the first matching result instead of querying the whole collection?

We have some old code that is doing a query of the DynamoDB to find list of matching records.
Sample code below:
final DynamoDBQueryExpression<MyObject> queryExp = new DynamoDBQueryExpression<MyObject>()
.withHashKeyValues(myObject)
.withIndexName(indexName)
.withScanIndexForward(false)
.withConsistentRead(true)
.withLimit(rowsPerPage);
final PaginatedQueryList<MyObject> ruleInstanceList = dynamoDBMapper.query(MyObject.class, queryExp);
This is a slow operation since this query will return a list of matching MyObject, and I noticed all we used it for is to check if this list is empty or not.
So what I want to do is simply doing the query to find the first element or even a different type of query to simply make sure the count is greater than 0, all I need to verify is that the record exist so that I can reduce the latency.
My question is, how do I do it in order to achieve this?

The documentation for getLimit() indicates:
Note that when calling DynamoDBMapper.query, multiple requests are made to DynamoDB if needed to retrieve the entire result set. Setting this will limit the number of items retrieved by each request, NOT the total number of results that will be retrieved. Use DynamoDBMapper.queryPage to retrieve a single page of items from DynamoDB.
To limit the number of results, you can use queryPage() instead of query(). And apply withLimit(1) to your query expression.

Use existing JPAQuery as subquery

I have an instance of JPAQuery<?> and need to retrieve the count. However, since the table may contain many items (millions), I want to limit the count to a given maximum, say 50,000.
The current QueryDSL-Code effectively does this:
query.fetchCount();
Now my desired modifications are quite trivial in raw sql:
select count(*) from (<whatever query> limit 50000);
However, I do not know how I would express this in querydsl. The following code is not correct, because .from() takes an entity path, but query is a query:
JPAExpressions.select(Wildcard.all)
.from(query.limit(50000))
.fetchCount();
I am using querydsl 4.

JPAExpressions.select(Wildcard.all) returns a child of SimplyQuery, which you can call limit on.
JPAExpressions.select(Wildcard.all)
.from(entity)
.limit(50000)
.fetchCount();

CLOB and CriteriaQuery

I have an entity that has a CLOB attribute:
public class EntityS {
...
#Lob
private String description;
}
To retrieve certain EntityS from the DB we use a CriteriaQuery where we need the results to be unique, so we do:
query.where(builder.and(predicates.toArray(new Predicate[predicates.size()]))).distinct(true).orderBy(builder.asc(root.<Long> get(EntityS_.id)));
If we do that we get the following error:
ORA-00932: inconsistent datatypes: expected - got CLOB
I know that's because you cannot use distinct when selecting a CLOB. But we need the CLOB. Is there a workaround for this using CriteriaQuery with Predicates and so on?
We are using an ugly workaround getting rid of the .unique(true) and then filtering the results, but that's crap. We are using it only to be able to keep on developing the app, but we need a better solution and I don't seem to find one...

In case you are using Hibernate as persistence provider, you can specify the following query hint:
query.setHint(QueryHints.HINT_PASS_DISTINCT_THROUGH, false);
This way, "distinct" is not passed through to the SQL command, but Hibernate will take care of returning only distinct values.
See here for more information: https://thoughts-on-java.org/hibernate-tips-apply-distinct-to-jpql-but-not-sql-query/

Thinking outside the box - I have no idea if this will work, but perhaps it is worth a shot. (I tested it and it seems to work, but I created a table with just one column, CLOB data type, and two rows, both with the value to_clob('abcd') - of course it should work on that setup.)
To de-duplicate, compute a hash of each clob, and instruct Oracle to compute a row number partitioned by the hash value and ordered by nothing (null). Then select just the rows where the row number is 1. Something like below (t is the table I created, with one CLOB column called c).
I expect that execution time should be reasonably good. The biggest concern, of course, is collisions. How important is it that you not miss ANY of the CLOBs, and how many rows do you have in the base table in the first place? Is something like "one chance in a billion" of having a collision acceptable?
select c
from (
select c, row_number() over (partition by dbms_crypto.hash(c, 3) order by null) as rn
from t
)
where rn = 1;
Note - the user (your application, in your case) must have EXECUTE privilege on SYS.DBMS_CRYPTO. A DBA can grant it if needed.

Get rowCount before execute query JPA

I'm trying to build a Pagination inside my system. In DAO i'm using "setFirstResult()" and "setMaxResults()" to limit the amount of rows returned.
Look:
Query query = entityManager.createNamedQuery(namedQuery);
if (firstResult != null) {
query.setFirstResult(firstResult);
}
if (maxResult != null) {
query.setMaxResults(maxResult);
}
List returnList = query.getResultList();
But for pagination works i need to know the amount of rows without limitation (firstResult() and maxResults()).
If i have this query:
SELECT * FROM MyEntity e WHERE e.car = :carParam OFFSET 10 LIMIT 20
i would like to count like this
SELECT Count(*) FROM MyEntity e WHERE e.car = :carParam
But I want to avoid create another query manually for each Entity, how can i do a count() without force creating a new count() query ?

There is no way to calculate the total count of results without using Criteria API or (as you said) creating another query manually. Both of them will cause to a separate query against database to calculate the total counts. I had some experiences in this context. It may doubles your response time if your query is going to be run on a huge data set or if you have a large number of concurrent users.
I think the best way to prevent such an overhead on your DBMS is to display a "load more" link at the end of the search results. I highly recommend such approach if displaying total count is not part of your main business or is not forced by the client.
Take a look at this link (if you've not checked it before)

Implementing result paging in hibernate (getting total number of rows)

How do I implement paging in Hibernate? The Query objects has methods called setMaxResults and setFirstResult which are certainly helpful. But where can I get the total number of results, so that I can show link to last page of results, and print things such as results 200 to 250 of xxx?

You can use Query.setMaxResults(int results) and Query.setFirstResult(int offset).
Editing too: There's no way to know how many results you'll get. So, first you must query with "select count(*)...". A little ugly, IMHO.

You must do a separate query to get the max results...and in the case where between time A of the first time the client issues a paging request to time B when another request is issued, if new records are added or some records now fit the criteria then you have to query the max again to reflect such. I usually do this in HQL like this
Integer count = (Integer) session.createQuery("select count(*) from ....").uniqueResult();
for Criteria queries I usually push my data into a DTO like this
ScrollableResults scrollable = criteria.scroll(ScrollMode.SCROLL_INSENSITIVE);
if(scrollable.last()){//returns true if there is a resultset
genericDTO.setTotalCount(scrollable.getRowNumber() + 1);
criteria.setFirstResult(command.getStart())
.setMaxResults(command.getLimit());
genericDTO.setLineItems(Collections.unmodifiableList(criteria.list()));
}
scrollable.close();
return genericDTO;

you could perform two queries - a count(*) type query, which should be cheap if you are not joining too many tables together, and a second query that has the limits set. Then you know how many items exists but only grab the ones being viewed.

You can do one thing. just prepare Criteria query as per your busness requirement with all Predicates , sorting , searching etc.
and then do as below :-
CriteriaBuilder criteriaBuilder = em.getCriteriaBuilder();
CriteriaQuery<Feedback> criteriaQuery = criteriaBuilder.createQuery(Feedback.class);
//Just Prepare your all Predicates as per your business need.
//eg :-
yourPredicateAsPerYourBusnessNeed = criteriaBuilder.equal(Root.get("applicationName"), applicationName);
criteriaQuery.where(yourPredicateAsPerYourBusnessNeed).distinct(true);
TypedQuery<Feedback> criteriaQueryWithPredicate = em.createQuery(criteriaQuery);
//Getting total Count Here
Long totalCount = criteriaQueryWithPredicate.getResultStream().distinct().count();
Now we have our actual data with us as above with total count , right.
So now we can apply pagination on the data we have in our hand above , as below :-
List<Feedback> feedbackList = criteriaQueryWithPredicate.setFirstResult(offset).setMaxResults(pageSize).getResultList();
Now You can prepare a wrapper with your List return by DB along with the totalCount , startingPageNo that is offset here in this case, page Size etc and can return to your service / controller class.
I am 101 % sure , this will solve your problem, Because I was facing same problem and sorted it out same way.
Thanks- Sunil Kumar Mali

You can just setMaxResults to the maximum number of rows you want returned. There is no harm in setting this value greater than the number of actual rows available. The problem the other solutions is they assume the ordering of records remains the same each repeat of the query, and there are no changes going on between commands.
To avoid that if you really want to scroll through results, it is best to use the ScrollableResults. Don't throw this object away between paging, but use it to keep the records in the same order. To find out the number of records from the ScrollableResults, you can simply move to the last() position, and then get the row number. Remember to add 1 to this value, since row numbers start counting at 0.

I personally think you should handle the paging in the front-end. I know this isn't that efficiënt but at least it would be less error prone.
If you would use the count(*) thing what would happen if records get deleted from the table in between requests for a certain page? Lots of things could go wrong this way.