We Keep Coding

Unit test for a large SELECT query with jOOQ

I am using jOOQ for working with a relational database. I have a SELECT query for which I need to write unit tests with mocking. Based on this doc and this post, I need to define my own data provider, which should look something like this:
class MyProvider implements MockDataProvider {
DSLContext create = DSL.using(SQLDialect.MYSQL);
#Override
public MockResult[] execute(MockExecuteContext mockExecuteContext) throws SQLException {
MockResult[] mock = new MockResult[1];
String sql = mockExecuteContext.sql();
if (sql.startsWith("select")) {
Result<Record2<String, String>> result = create.newResult(COL_1, COL_2);
result.add(create.newRecord(COL_1, COL_2)
.values("val1", "val2"));
mock[0] = new MockResult(1, result);
}
return mock;
}
}
where COL_1 and COL_2 are defined as follows:
Field<String> COL_1 = field("Column1", String.class);
Field<String> COL_2 = field("Column2", String.class);
It's quite simple and straightforward when SELECT is a small one (as in the above example, just 2 columns). I am wondering how it should be done in case of complex and large selects. For instance I have a SELECT statement which selects 30+ columns from multiple table joins. Seems the same approach of
Result<Record_X<String, ...>> result = create.newResult(COL_1, ...);
result.add(create.newRecord(COL_1, ...)
.values("val1", ...));
does not work in case of more than 22 columns.
Any help is appreciated.

Answering your question
There is no such limitation as a maximum of 22 columns. As documented here:
Higher-degree records
jOOQ chose to explicitly support degrees up to 22 to match Scala's typesafe tuple, function and product support. Unlike Scala, however, jOOQ also supports higher degrees without the additional typesafety.
You can still construct a record with more than 22 fields using DSLContext.newRecord(Field...). Now, there is no values(Object...) method on the Record type, because the Record type is the super type of all the Record1 - Record22 types. If such an overload were present, then the type safety on the sub types would be lost, because the values(Object...) method is applicable for all types of arguments. This might be fixed in the future by introducing a new RecordN subtype.
But you can load data into your record with other means, e.g. by calling Record.fromArray(Object...):
Record record = create.newRecord(COL_1, ...);
record.fromArray("val1", ...);
result.add(record);
The values() method being mere convenience (adding type safety) for fromArray().
Disclaimer:
I'm assuming you read the disclaimer on the documentation page you've linked. I'm posting it here anyway for other readers of this question, who might not have read the disclaimer:
Disclaimer: The general idea of mocking a JDBC connection with this jOOQ API is to provide quick workarounds, injection points, etc. using a very simple JDBC abstraction. It is NOT RECOMMENDED to emulate an entire database (including complex state transitions, transactions, locking, etc.) using this mock API. Once you have this requirement, please consider using an actual database instead for integration testing, rather than implementing your test database inside of a MockDataProvider.
It seems you're about to re-implement a database which can "run" any type of query, including a query with 23+ columns, and every time you change the query under test, you will also change this test here. I still recommend you do integration testing instead, using testcontainers or even with H2, which will help cover many more queries than any such unit test approach. Here's a quick example showing how to do that: https://github.com/jOOQ/jOOQ/tree/main/jOOQ-examples/jOOQ-testcontainers-example
Also, integration tests will help test query correctness. Unit tests like these will only provide dummy results, irrespective of the actual query. It is likely that such mocks can be implemented much more easily on a higher level than the SQL level, i.e. by mocking the DAO, or repository, or whatever methods, instead.

JPA query from java Object

How can i use jpa for query over an object (not an entity)?
For example this simple code:
String [] theList = {a,b,c,d}.
Query q = new Query("Select tl from theList tl")
Reason behind: the queries are dynamically created and executed, but the objects in the from clause of the jpql query aren't necessarily mapped tables. In some cases there are just an Object, So the actual behavior needed is modify the query during execution of the program to meet the criteria, but i don't know how to modify the query.
Edit: I Don't use native queries because of portability of code. It will be the last option.

What you're looking for is called LINQ, and unfortunately (?) it is available only in C#.
However, you can partially emulate it with Stream(s).
A Stream offers basically all the operators you need
.filter() where
.max() max
.sorted() orderby
.limit() limit
.skip() offset
.collect(groupingBy()) group by
And so on. Just give a look at the Javadoc!

I think 'JdbcTemplate' would suffice your requirement.
JdbcTemplate gives you the flexibility to run native queries and map them to a Java class.
However, you'll have to explicitly map your Java class with the column names in the database.

I have solved using joSQL. Is a powerfull opensource tool that allows you to query over java objects using "sql". It is not jpa but satisfied my needs.
Another tool i have seen that do that is called querydsl.

Elasticsearch, Nested "ANDS" and "ORS"

I am having some difficulty structuring the exact Elasticsearch query that I am looking for, specifically using the java api.
It seems like if I construct a fieldsearch using the java api, I can only use a single field and a single term. If I use a querystring, it looks like I can apply an entire query to a set of fields. What I want to do is apply a specific query to one field, and another query to a different field.
This is confusing I know. This is the type of query I would like to construct
(name contains "foo" or name contains "bar") AND ( date equals today)
I am really loving Elasticsearch for it's speed and flexibility, but the docs on http://www.elasticsearch.org/ are kind of tough to parse (I noticed "introduction" and "concepts" have no links, but the API section does) If anyone has some good resources on mastering these queries, I'd love to see them. Thanks!

Sounds like a bool query with 2 must clause:
matchQuery("name", "foo bar")
rangeQuery("date").from("2013-02-05").to("2013-02-06")
Does it help?

App Engine datastore does not support operator OR

I am trying to query the google datastore for something like (with pm --> persistanceManager):
String filters = "( field == 'value' || field == 'anotherValue' )";
Query query = pm.newQuery(myType.class, filters);
When I execute - I am getting back: App Engine datastore does not support operator OR.
What's the best approach in people experience for this kind of queries?
Any help appreciated!

Perform multiple queries. The Datastore, like all other databases, isn't able to efficiently execute disjunctions. Unlike other databases, it exposes this difficulty to the user, to make it clear that what you're doing isn't efficient. Your only solution is to execute multiple queries - one for each or - and combine them.

I don't know if GAE's JDO and JPA implementations support this, but using the low-level API, you can use the operator IN for this, in one query.
Query query = new Query("Issue");
List<String> list = Arrays.asList("NEW", "OPEN", "ACCEPTED");
query.addFilter("status", FilterOperator.IN, list);
DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
PreparedQuery preparedQuery = datastore.prepare(query);
for (Entity entity : preparedQuery.asIterable()) {
// should iterate over 'NEW', 'OPEN' and 'ACCEPTED' issues
}

According to Google App Engine - Queries and Indexes:
Query Filters
A filter specifies a field name,
an operator, and a value. The value
must be provided by the app; it cannot
refer to another property, or be
calculated in terms of other
properties. The operator can be any of
the following: < <= == >= >
Note: The Java datastore interface does not support the != and
IN filter
operators that are implemented in the
Python datastore interface. (In the
Python interface, these operators are
implemented in the client-side
libraries as multiple datastore
queries; they are not features of the
datastore itself.)
The subject of a filter can be any
object field, including the primary
key and the entity group parent (see
Transactions).
An entity must match all filters to be
a result. In the JDOQL string syntax,
multiple filters are specified
separated by && (logical "and").
Other logical combinations of filters
(logical "or", "not") are not
supported.
Due to the way the App Engine
datastore executes queries, a single
query cannot use inequality filters
(< <= >= >) on more than one
property. Multiple inequality filters
on the same property (such as querying
for a range of values) are permitted.
See Restrictions on Queries.
Basically you're either going to have to restructure your data so that you can find what you're looking for with one condition or multiple "and" conditions or you're going to have to retrieve the data via two (or more) queries and filter/combine it in your code.

Sorry I'm late to the game.. I just ran across your question today.
Another way to "simulate" 'IN' and 'OR' behavior is to use the "low level" Datastore API. The DatastoreService supports a get() method that accepts a collection of Keys and returns a Map of all Entities that matched the passed in Keys. It's an interface, but there's a handy DatastoreServiceFactory available that will dispense a ready-to-use instance.
Unfortunately, Google decided that they don't want to promote this low-level API approach and prefer that developers use JDO or JPA, so there's no documentation available other than the JavaDocs and whatever code samples that you might find when you Google "DatastoreService".
TL

Late breaking News.. at least I'm just getting it. As I was downloading the latest Java SDK for GAE I noticed on the Release Notes that "Issue 29: Expose batch gets" was fixed in the latest release (v1.2.1). Basically it seems that we (I'm looking for the same support it seems) may have a JDO based alternative rather than having to drop down to the "low-level" Datastore API. I've just downloaded the latest Java GAE SDK so I haven't had an opportunity to test anything yet, but I wanted to give you a heads-up ASAP. I'll post anything more I learn after I've had a chance to confirm this "fix".
Please accept my apologies if I've broken StackOverflow etiquette by re-posting my comment as an answer, but I decided to do it for two reasons. Firstly because, even though it's me addressing the same issue again, IMHO this new information appears to provide a completely different "answer" to the problem. And secondly, I was concerned that the comment form might not get your attention before you'd spent a great deal of time looking into the first answer that I provided.
Next time I'll think more carefully before acting.
TL

One way to simplify having to "do it yourself" might be to use parameterized queries:
Query query = pm.newQuery(mytype.class);
query.setFilter("field == autoParam");
query.declareParameters("String autoParam");
List<String> params = myListOfThingsFieldCanBeEqualTo;
Set merged = new HashSet();
for (String f : params) {
merged.addAll(q.execute(f));
}

Contrary to cletus' answer, OR-ing works, in more recent version of App Engine anyway.
Indeed, I found OR-ing not working in App Engine 1.3.0 that I had, but according to Google App Engine - Queries and Indexes (the same source cletus referred to in his answer),
An entity must match all filters to be a result. In the JDOQL string syntax, you can separate multiple filters with || (logical "or") and && (logical "and"), although keep in mind that || can only be employed when the filters it separates all have the same field name. In other words, || is only legal in situations where the filters it separates can be combined into a single contains() filters.
I figured since his answer (and since I last updated my App Engine), App Engine must have been upgraded on this matter.
Update App Engine to 1.3.4, and the OR-ing works! Though with the limitation.
Thanks to cletus anyway:)

You can use the contains method
String filters = "( :values.contains(field) )";
Query query = pm.newQuery(myType.class, filters);

Is it valid for Hibernate list() to return duplicates?

Is anyone aware of the validity of Hibernate's Criteria.list() and Query.list() methods returning multiple occurrences of the same entity?
Occasionally I find when using the Criteria API, that changing the default fetch strategy in my class mapping definition (from "select" to "join") can sometimes affect how many references to the same entity can appear in the resulting output of list(), and I'm unsure whether to treat this as a bug or not. The javadoc does not define it, it simply says "The list of matched query results." (thanks guys).
If this is expected and normal behaviour, then I can de-dup the list myself, that's not a problem, but if it's a bug, then I would prefer to avoid it, rather than de-dup the results and try to ignore it.
Anyone got any experience of this?

Yes, getting duplicates is perfectly possible if you construct your queries so that this can happen. See for example Hibernate CollectionOfElements EAGER fetch duplicates elements

I also started noticing this behavior in my Java API as it started to grow. Glad there is an easy way to prevent it. Out of practice I've started out appending:
.setResultTransformer(Criteria.DISTINCT_ROOT_ENTITY)
To all of my criteria that return a list. For example:
List<PaymentTypeAccountEntity> paymentTypeAccounts = criteria()
.setResultTransformer(Criteria.DISTINCT_ROOT_ENTITY)
.list();

If you have an object which has a list of sub objects on it, and your criteria joins the two tables together, you could potentially get duplicates of the main object.
One way to ensure that you don't get duplicates is to use a DistinctRootEntityResultTransformer. The main drawback to this is if you are using result set buffering/row counting. The two don't work together.

I had the exact same issue with Criteria API. The simple solution for me was to set distinct to true on the query like
CriteriaQuery<Foo> query = criteriaBuilder.createQuery(Foo.class);
query.distinct(true);
Another possible option that came to my mind before would be to simply pass the resulting list to a Set which will also by definition have just an object's single instance.