We Keep Coding

How to make a Hibernate SearchSession return results with unique attributes?

I am working on using the Hibernate SearchSession class in Java to perform a search against a database, the code I currently have to search a table looks something like this:
SearchSession searchSession = Search.session(entityManagerFactory.unwrap(SessionFactory.class).withOptions()
.tenantIdentifier("locations").openSession());
SearchResult<Location> result = searchSession.search(Location.class)
.where( f -> f.bool()
.must( f.match()
.field("locationName")
.matching((phrase)).fuzzy())
).fetch(page * limit, limit);
This search works and properly returns results from the database, but there is no uniqueness constraint on the locationName column and the database holds multiple records with the same value in locationName. As a result, when we try to display them on the UI of the application it looks like there are duplicate values, even though they're unique in the database.
Is there a way to make a SearchSession only return a result if another result with an identical value (such as locationName) has not been returned before? Applying a uniqueness constraint to the database table isn't an option in this scenario, and we were hoping there's a way to handle filtering out duplicate values in the session over taking the results from the search and removing duplicate values separately.

Is there a way to make a SearchSession only return a result if another result with an identical value (such as locationName) has not been returned before?
Not really, at least not at the moment.
If you're using the Elasticsearch backend and are fine with going native, you can insert native JSON into the Elasticsearch request, in particular collapsing.
I think something like this might work:
SearchResult<Location> result = searchSession.search( Location.class )
.extension( ElasticsearchExtension.get() )
.where( f -> f.bool()
.must( f.match()
.field("locationName")
.matching((phrase)).fuzzy())
)
.requestTransformer( context -> {
JsonObject collapse = new JsonObject();
collapse.addProperty("field", "locationName_keyword")
JsonObject body = context.body();
body.add( "collapse", collapse );
} )
// You probably need a sort, as well:
.sort(f -> f.field("id"))
.fetch( page * limit, limit );
You will need to add a locationName_keyword field to your Location entity:
#Indexed
#Entity
public class Location {
// ...
#Id
#GenericField(sortable = Sortable.YES) // Add this
private Long id;
// ...
#FullTextField
#KeywordField(name = "locationName_keyword", sortable = Sortable.YES) // Add this
private String locationName;
// ...
}
(You may need to also assign a custom normalizer to the locationName_keyword field, if the duplicate locations have a slightly different locationName (different case, ...))
Note however that the "total hit count" in the Search result will indicate the number of hits before collapsing. So if there's only one matching locationName, but 5 Location instances with that name, the total hit count will be 5, but users will only see one hit. They'll be confused for sure.
That being said, it might be worth having another look at your situation to determine whether collapsing is really necessary here:
As a result, when we try to display them on the UI of the application it looks like there are duplicate values, even though they're unique in the database.
If you have multiple documents with the same locationName, then surely you have multiple rows in the database with the same locationName? Duplication doesn't appear spontaneously when indexing.
I would say the first thing to do would be to step back, and consider whether you really want to query the Location entity, or if another, related entity wouldn't make more sense. When two locations have the same name, do they have a relationship to another, common entity instance (e.g. of type Shop, ...)?
=> If so, you should probably query that entity type instead (.search(Shop.class)), and take advantage of #IndexedEmbedded to allow filtering based on Location properties (i.e. add #IndexedEmbedded to the location association in the Shop entity type, then use the field location.locationName when adding a predicate that should match the location name).
If there is no such related, common entity instance, then I would try to find out why locations are duplicated exactly, and more importantly why that duplication makes sense in the database, but not to users:
Are the users not interested in all the locations? Then maybe you should add another filter to your query (by "type", ...) that would help remove duplicates. If necessary, you could even run multiple search queries: first one with very strict filters, and if there are no hits, fall back to another one with less strict filters.
Are you using some kind of versioning or soft deletion? Then maybe you should avoid indexing soft-deleted entities or older versions; you can do that with conditional indexing or, if that doesn't work, with a filter in your search query.
If your data really is duplicated (legacy database, ...) without any way to pick a duplicate over another except by "just picking the first one", you could consider whether you need an aggregation instead of full-blown search. Are you just looking for the top location names, or maybe a count of locations by name? Then aggregations are the right tool.

Is there a way to improve the performance of Hibernate-generated queries when stored functions are involved?

I have a page which shows some records in a grid in a paged manner. I have the table my_table and the entity MyTable linked to it:
#SuppressWarnings("serial")
#Entity
#Table(name="my_table")
#Inheritance(strategy=InheritanceType.JOINED)
public class MyTable extends BaseEntity implements Auditable, Serializable {
//...
private Integer myAttribute; //This does not exist in the table
//...
#Formula(value = "(myFunction(attr1, attr2))")
public Integer getMyAttribute() {
return myAttribute;
}
public void setMyAttribute(Integer myAttribute) {
this.myAttribute = myAttribute;
}
//...
}
When I intend to query by "normal" fields, everything works greatly, but when I attempt to filter by myAttribute, like:
queryInput.addAndCriterion(Restrictions.eq("myAttribute", v));
where for instance v is an Integer with the value of 123, the search will time out. If I run the stored function in MySQL directly, then it is executed instantly. I think this code sends a separate request on each item, which could explain the problem. Is there a way to ensure that I can filter by my stored function in a performant way (maybe the call for the stored function will be generated into the query)? I would need to define a criteria which specifies that for each record a certain stored function needs to be called and attr1 and attr2 are passed, which are fields of the records?

As noted in the comment below, this answer doesn't fix the problem of troubleshooting Hibernate, but the OP liked it anyway.
Answer follows...
Querying on any function, a stored function or a builtin function, is always a table-scan.
For example this would not be able to use an index on create_date, even if one existed:
SELECT * FROM MyTable WHERE MONTH(create_date) = 2
The same is true anytime you use an indexed column as arguments to a function.
The workaround for MySQL 5.7 and later is to use a generated column for that expression, and then index the generated column.
ALTER TABLE MyTable
ADD COLUMN created_month INT AS (MONTH(create_date)),
ADD INDEX (created_month);
Once you do that, you can query for created_month = 2 or you can even query the original expression MONTH(create_date) = 2 and it will use the index.
Unfortunately, you can use this feature only with builtin MySQL functions.
https://dev.mysql.com/doc/refman/5.7/en/create-table-generated-columns.html says:
Generated column expressions must adhere to the following rules. An error occurs if an expression contains disallowed constructs.
Stored functions and user-defined functions are not permitted.
An alternative solution would be for you to create a new concrete column to store the result of the stored function, supposing the value is deterministic from its arguments and doesn't depend on the state of data in other tables.
ALTER TABLE MyTable
ADD COLUMN myAttribute INT,
ADD INDEX (myAttribute);
CREATE TRIGGER att_ins BEFORE INSERT ON MyTable
FOR EACH ROW SET NEW.myAttribute = MyFunction(NEW.attr1, NEW.attr2);
CREATE TRIGGER att_upd BEFORE UPDATE ON MyTable
FOR EACH ROW SET NEW.myAttribute = MyFunction(NEW.attr1, NEW.attr2);
Then you would query the new column instead of the expression.
That's kind of a hassle, but it's the only way to get an indexed lookup against the result of your stored function.

How to speed up eager loading of foreign collections?

I have a database table mapped with ORMlite, it contains some data (18 columns) and it also contains ForeignCollectionField(eager = true).
Problem is when loading all data from this table ... ORMlite is creating query for every item instead using joins. Which is resulting in 67124 queries and taking forever to load all objects from this table.
This could be however done in right join query under few seconds? Why to generate thousands of queries instead?
How can I speed it up? Do I have to write raw query and then RawRowMapper , which makes using ORM pointless..
How to deal with loading eager collections in ormlite? Because queryForAll is not way..

Problem is when loading all data from this table ... ORMlite is creating query for every item instead using joins. Which is resulting in 67124 queries and taking forever to load all objects from this table.
It's ORM_Lite_ for a reason. Lot of people have asked for the join support on foreign collections but I've not gotten to it yet. It's not easy.
If you still want to use ORMLite then I'd recommend not using eager = true and doing 2 queries instead. One query for your main item and then another query using the DAO associated with the collection entity using IN. Something like:
qb = accountDao.queryBuilder();
qb.where()...;
List<Account> accounts = qb.query();
// build a list of account-ids
List<Long> accountIds = new ArrayList<>();
for (Account account : accounts) {
accountIds.add(account.getId());
}
// now use this list of ids to get your other entities
List<Order> orders = orderDao.queryBuilder().where().in("accountId", accountIds).query();
// now you have a list of orders for all of your ids
// you will need to associate each order with its account
Hope this helps.

Spring Data JPA Distinct - Return results from a single column

I have some data that contains a STATE field (String/Text) that defines what state a given request is currently in (e.g. pending, approved denied etc.) and to get all the unique values from that column I can run the following TSQL query
SELECT DISTINCT STATE FROM CALLOUT_REQUEST
where CALLOUT_REQUEST is my table name and STATE being the field which returns something like:
STATE
approved
denied
pending
...
However I don't understand how I would turn that into a query in my repository as it seems I need a "by" statement or some other filter mechanism which i can get the STATE based on?
What I am looking to return - as shown in the raw TSQL query above - is some kind of List or Array object which contains all the unique/distinct values in all of the STATE fields.
So in pseudo code i think i am looking for something like this:
String[] states = repository.findDisinctState();
where findDistinctState() would then return an array of sorts.
Hope that makes sense - I am very new to Java and Spring in general so I think I am missing some conceptual knowledge to utilise the above.
UPDATE:
The 'state' concept is closed so i could implement that as an enum - only problem is i dont know how to do that :) Ill look into how i can do that as i think it fits perfectly with what i am trying to achieve.
The List i get from the query provided is intended to be used to get a count of all the occurrences. I had this code before to get a total count for each of the 'states':
Map stats = new HashMap();
String[] states = {"approved", "denied", "pending", "deactivated"};
for (int i = 0; i < states.length; i++) {
stats.put(states[i], repository.countByState(states[i]));
}
Am i correct in understanding that the states Array that i have in the above code snippet could be turned into an enum and then i dont even need the custom #Query anymore?

If that state concept is closed - you know its possible set of values - it should be an enum.
After that you can create queries that you invoke like:
repository.findByState(State.APPROVED)
If you can't create an enum, you need a separate method to get the distinct values, which can't be provided by JPA, because you need a list of strings and not a list of CalloutRequests.
Then you need to specify a query manually like:
#Query("SELECT DISTINCT State FROM CALLOUT_REQUEST")
List<String> findDistinctStates();

You can use a JPQL query for this, with the #org.springframework.data.jpa.repository.Query annotation:
#Query("select distinct state from CalloutRequest")
List<String> findDistinctStates();

If you don't want to use #Query then one solution is there to create an interface "StateOnlyInterface" with method named "getState()".
Then create method in your repo with name, getDistinctState(). Return type of this method to be kept as ArrayList of StateOnlyInterface.

Does EclipseLink store ordered entity List incorrect?

I have entity named Test1, it has 2 fields id and name.
I put about 1000 entity Test1 into a ArrayList with first entity has id = 1, second entity has id = 2, and so on... to 1000.
I use EclipseLink 2.4 provider to communication Postgresql database.
When I store List into database using following code:
EntityManager em;
List<Test1> list = new ArrayList<Test1>();
//populate data into list
for(Test1 test1 : list){
em.merge(test1);
}
When I check database, the rows is not ordered as expected. The first row does not have id = 1, and the second row does not have id = 2 and so on...
What's wrong with EclipseLink or I got mistake?

From what you wrote, you cannot be sure who is responsible for that: either your DB / DB client (when you check the order), or the EcliplseLink. Anyway, if you want a certain order for your JPA operations, you should use EntityManager.flush() in order to hit the DB, otherwise the persistence provider can decide when and in which order to flush its query cache. Also, if you want a certain order in your client, you should sort the results, as some databases (like MySQL) do not guarantee that the rows are returned in the same order in which they have been inserted.

I think it depends, depends on what?
Depends if the List has been marked with #OrderColumn annotation, default is not store the order of the list, but it depends on your model an eclipselink can be configured to store order and be the responsible for the correct update of that field.
Also use OrderColumn could be a hit in the performance so use with caution