I have a task that require some "special" users to be able to switch between accounts without the need to login. As a starting point I have a join table that consists only of Users ID-s. In form of PRIMARY_USER_ID, and SECONDARY_USER_ID as a foreign keys from USERS table. The first thing that needs to be implemented is GET of all connections between Users. [{"primary_username", "primary_email","secondary_username","secondary_email"}].
I have created a many-to-many relationship on User entity, where both sides of relationship are on User.
#EqualsAndHashCode.Exclude
#ToString.Exclude
#ManyToMany(fetch = FetchType.LAZY, cascade = {
CascadeType.PERSIST,
CascadeType.MERGE
})
#JoinTable(name = "CONTACTS_ONE_LOGIN",
joinColumns = { #JoinColumn(name = "PRIMARY_CONTACT")},
inverseJoinColumns = {#JoinColumn(name = "SECONDARY_CONTACT")}
)
private Set<Contact> secondaryContacts = new HashSet<>();
#EqualsAndHashCode.Exclude
#ToString.Exclude
#ManyToMany(cascade = {
CascadeType.REMOVE
},
mappedBy = "secondaryContacts")
private Set<Contact> primaryContacts = new HashSet<>();
Now the problem is, when i want to get all connections between contacts, I would need to get first all the information from the join table, and then go through each PRIMARY_CONTACT_ID to get it's connected contacts. Which would result in very low performances.
I wanted to change this to have an CONNECTED_USERS entity, which would have instead of two USER ID-s have two Many-to-one relationships on USER.
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "PRIMARY_CONTACT_ID")
private Contact contact;
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "SECONDARY_CONTACT_ID")
private Contact contact;
My question is, is this going to add performance, since in my dev DB i do not have a lot of users to test it properly? Or is there any better way to do this?
It's even harder for us to guess how well your model will perform as we know nothing about your application, its data or its business rules. From your question it seems that only a few users are affected by this requirement. So unless your total user population is in the hundreds of thousands you probably don't need to worry.
Either way it's unlikely the performance benefits of the separate intersection table you propose will justify the overhead of maintaining it. What I do suggest is you build a compound function based index something like this (caveat: right now I have no access to a database so the following is untested and may contain syntax error):
create index connected_users_fbi on your_table (
case when secondary_contact_id is not null then primary_contact_id end,
secondary_contact_id);
This index will be useful for identifying primary contacts with secondary contacts. It may also support finding all the primary contacts which are connected to a secondary contact (if you need that feature) through Index Skip Scan.
Obviously don't take my word for it but try to benchmark it with realistic volumes of data. Your project should have a performance environment where you can do such tests. If it doesn't then it's pretty much doomed.
Related
The problem which i am trying to solve is avoid duplicate items inside a list attribute in hibernate.
Consider the below domain.
public class Account
{
#OneToMany(fetch = FetchType.LAZY)
#JoinTable(name = "FI_COMPANY_ACCOUNT", joinColumns = #JoinColumn(name = "ACCOUNT_ID", referencedColumnName = "ID"), inverseJoinColumns = #JoinColumn(name = "COMPANY_ID", referencedColumnName = "ID"))
private List<Company> companies;
#OneToMany(fetch = FetchType.LAZY, mappedBy = "account", cascade = CascadeType.ALL, orphanRemoval = true)
private List<AccountDesc> accountDescList;
}
public class Company {}
public class AccountDesc
{
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "PARENT_ID", referencedColumnName = "ID")
private Account account;
}
I use a Criteria API to fetch Account. In the query i perform fetch using left join for companies and inner join for accountDescList attribute. This help me to get both attributes in first select, and which avoid further selects.
Root<Account> root = criteriaQuery.from(Account.class);
root.fetch("companies", JoinType.LEFT);
root.fetch("accountDescList");
I know the root entity (here Account) can be repeated in the results. I can solve the issue using multiple ways like,
http://in.relation.to/2016/08/04/introducing-distinct-pass-through-query-hint/
https://howtoprogramwithjava.com/how-to-fix-duplicate-data-from-hibernate-queries/
But issue i face is the attribute companies inside the Account has also duplicate entities. This happen if we have more than one entry for accountDescList.
To solve the issue of duplicates in the attribute companies, I feel only solution is to use Set. Could you please clarify on the below questions.
Is there a way other than using Set (for the attribute companies), to solve this issue.
Even if i use can i instruct hibernate to use OrderedSetType (which uses LinkedHashSet). So that i can retain the order of the items as it returned from database. Unfortunately I do not have a attribute to use in OrderBy. I need the whatever default order returned by database.
Thanks in advance.
But the issue I face is the attribute companies inside the Account has also duplicate entities.
That shouldn't happen unless you have duplicate Company entities assigned to the same account.
Using DISTINCT in the Criteria API query will remove root duplicates. However, in your case, it's not worth using JOIN FETCH on both #OneToMany relations since this will cause a Cartesian Product.
You should fetch at most one collection at a time, and maybe use #Subselect fetching for the second collection.
I think that it is much better use Set because a set doesn't allow elements duplicated, also you can overwrite equals method of Company and put it on what fields will be validated when two elements are equals.
The other way would be in setCompanies(List companies) method you can make something logic before this.companies = companies.stream().distinct().collect(Collectors.toList()); or
this.companies = new ArrayList<>(new HashSet(companies)) ;
I am trying to improve the performance of a repository-method. I have a OneToMany-relationship in one of my entities, UserEntity, with a set of AddressEntities that are loaded lazily.
In AddressEntity:
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = EntityConstants.COLUMN_USER_ID, referencedColumnName = EntityConstants.COLUMN_USER_ID)
private UserEntity user;
In UserEntity:
#OneToMany(mappedBy = "user", fetch = FetchType.LAZY)
private List<AddressEntity> addresses;
The problem is that when I fetch AddressEntity, a query is made to the database to the user table as well. From what I understand this is to check for the existence of the UserEntity, and to create a proxy object to it.
This takes time, and I am not interested in whether or not the user entity exists in this case. Is there any way to prevent hibernate to do this extra query and simply leave userEntity to null?
Thanks in advance,
Markus
I solved this by only selecting the attributes that I needed:
SELECT ad.longitude, ad.latitude FROM AddressEntity ad
This does not cause hibernate to check for proxy objects on relations that are not selected.
I have a problem that loading my lazy collections produces a lot of SQL-Statements and I wonder if there is no more efficient way of loading the data.
Situation:
Parent has a lazy collection of Child called children. It is actually a Many-To-Many relation.
I load a list of Parents with a CrudRepository and I need to get all child_ids for each Parent. So every time I access the children collection i executes a new SQL-Statement.
If i load 200 Parents there are 201 queries executes (1 for the list of Parents and 1 for each Parent's children).
Any idea how i can load the data with just one query?
EDIT 1
Parent/Child is probably a bad naming here. In fact i have a Many-To-Many relation.
Here is some code:
#Entity
public class Tour {
#Id
#GeneratedValue(generator = "system-uuid")
#GenericGenerator(name="system-uuid",
strategy = "uuid2")
#Column(length = 60)
private String id;
#ManyToMany
#JoinTable(
name="parent_images",
joinColumns = #JoinColumn(name="tour_id", referencedColumnName = "id"),
inverseJoinColumns = #JoinColumn(name="image_id", referencedColumnName = "id"),
foreignKey = #ForeignKey(name = "FK_TOUR_IMAGE_TOUR"),
inverseForeignKey = #ForeignKey(name = "FK_TOUR_IMAGE_IMAGE")
)
private List<Image> images = new ArrayList<>();
}
#Entity
public class Image {
#Id
#GeneratedValue(generator = "system-uuid")
#GenericGenerator(name="system-uuid",
strategy = "uuid2")
#Column(length = 40)
private String id;
//....
}
// Code to fetch:
#Autowired
TourRepository repo;
List<Tour> tours = repo.findBy(....);
List<String> imageIds = new ArrayList<>();
for(Tour tour : tours){
imageIds.addAll(tour.getImages().stream().map(b -> b.getId()).collect(Collectors.toList()));
}
As another answer suggested, JOIN FETCH is usually the way to solve similar problem. What happens internally for join-fetch is that the generated SQL will contains columns of the join-fetched entities.
However, you shouldn't blindly treat join-fetch being the panacea.
One common case is you want to retrieve entities with 2 One-To-Many relationships. For example, you have User, and each User may have multiple Address and Phone
If you naively do a from User user join fetch user.phones join fetch users.addresses, Hibernate will either report problem in your query, or generate a inefficient query which contains Cartesian product of addresses and phones.
In the above case, one solution is to break it into multiple queries:
from User user join fetch user.phones where .... followed by from User user join fetch user.addresses where .....
Keep in mind: less number of SQL does not always means better performance. In some situation, breaking up queries may improve performance.
That's the whole idea behind lazy collections :)
Meaning, a lazy collection will only be queried if the getter for that collection is called, what you're saying is that you load all entities and something (code, framework, whatever) calls the getChildren (assumption) for that entity; This will produce those queries.
Now, if this is always happening, then first of all, there's no point in having a lazy collection, set them as EAGER. - EDIT: as said in the comments, EAGER is rarely the solution, in this case in particular it definitely does not seem like it, the join is though :)
Either way, for your case that won't help, what you want is to load all data at once I assume, for that, when you do the query you have to make the join explicit, example with JPQL:
SELECT p FROM Parent p LEFT JOIN FETCH p.children
I have an entity with string id:
#Table
#Entity
public class Stock {
#Id
#Column(nullable = false, length = 64)
private String index;
#Column(nullable = false)
private Integer price;
}
And JpaRepository for it:
public interface StockRepository extends JpaRepository<Stock, String> {
}
When I call stockRepository::findAll, I have N + 1 problem:
logs are simplified
select s.index, s.price from stock s
select s.index, s.price from stock s where s.index = ?
The last line from the quote calls about 5K times (the size of the table). Also, when I update prices, I do next:
stockRepository.save(listOfStocksWithUpdatedPrices);
In logs I have N inserts.
I haven't seen similar behavior when id was numeric.
P.S. set id's type to numeric is not the best solution in my case.
UPDATE1:
I forgot to mention that there is also Trade class that has many-to-many relation with Stock:
#Table
#Entity
public class Trade {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private Integer id;
#Column
#Enumerated(EnumType.STRING)
private TradeType type;
#Column
#Enumerated(EnumType.STRING)
private TradeState state;
#MapKey(name = "index")
#ManyToMany(fetch = FetchType.EAGER)
#JoinTable(name = "trade_stock",
joinColumns = { #JoinColumn(name = "id", referencedColumnName = "id") },
inverseJoinColumns = { #JoinColumn(name = "stock_index", referencedColumnName = "index") })
private Map<String, Stock> stocks = new HashMap<>();
}
UPDATE2:
I added many-to-many relation for the Stock side:
#ManyToMany(cascade = CascadeType.ALL, mappedBy = "stocks") //lazy by default
Set<Trade> trades = new HashSet<>();
But now it left joins trades (but they're lazy), and all trade's collections (they are lazy too). However, generated Stock::toString method throws LazyInitializationException exception.
Related answer: JPA eager fetch does not join
You basically need to set #Fetch(FetchMode.JOIN), because fetch = FetchType.EAGER just specifies that the relationship will be loaded, not how.
Also what might help with your problem is
#BatchSize annotation, which specifies how many lazy collections will be loaded, when the first one is requested. For example, if you have 100 trades in memory (with stocks not initializes) #BatchSize(size=50) will make sure that only 2 queries will be used. Effectively changing n+1 to (n+1)/50.
https://docs.jboss.org/hibernate/orm/4.3/javadocs/org/hibernate/annotations/BatchSize.html
Regarding inserts, you may want to set
hibernate.jdbc.batch_size property and set order_inserts and order_updates to true as well.
https://vladmihalcea.com/how-to-batch-insert-and-update-statements-with-hibernate/
However, generated Stock::toString method throws
LazyInitializationException exception.
Okay, from this I am assuming you have generated toString() (and most likely equals() and hashcode() methods) using either Lombok or an IDE generator based on all fields of your class.
Do not override equals() hashcode() and toString() in this way in a JPA environment as it has the potential to (a) trigger the exception you have seen if toString() accesses a lazily loaded collection outside of a transaction and (b) trigger the loading of extremely large volumes of data when used within a transaction. Write a sensible to String that does not involve associations and implement equals() and hashcode() using (a) some business key if one is available, (b) the ID (being aware if possible issues with this approach or (c) do not override them at all.
So firstly, remove these generated methods and see if that improves things a bit.
With regards to the inserts, I do notice one thing that is often overlooked in JPA. I don't know what Database you use, but you have to be careful with
#GeneratedValue(strategy = GenerationType.AUTO)
For MySQL I think all JPA implementations map to an auto_incremented field, and once you know how JPA works, this has two implication.
Every insert will consist of two queries. First the insert and then a select query (LAST_INSERT_ID for MySQL) to get the generated primary key.
It also prevents any batch query optimization, because each query needs to be done in it's own insert.
If you insert a large number of objects, and you want good performance, I would recommend using table generated sequences, where you let JPA pre-allocate IDs in large chunks, this also allows the SQL driver do batch Insert into (...) VALUES(...) optimizations.
Another recommendation (not everyone agrees with me on this one). Personally I never use ManyToMany, I always decompose it into OneToMany and ManyToOne with the join table as a real entity. I like the added control it gives over cascading and fetch, and you avoid some of the ManyToMany traps that exist with bi-directional relations.
I have a Group entity that has a list of User entities in a many to many relationship. It is mapped by a typical join table containing the two IDs. This list may be very large, a million or more users in a group.
I need to add a new user to the group, typically that will be something like
group.getUsers().add(user);
user.getGroups().add(group);
em.merge(group);
em.merge(user);
If I understand typical JPA operation, will this require pulling down the entire list of 1 million+ users into the collection in order to add the new user and then save? That doesn't sound very scalable to me.
Should I simply not be defining this relationship in JPA? Should I be manipulating the join table entries directly in a case like this?
Please forgive the loose syntax, I'm actually using Spring Data JPA so I don't directly use the entity manager directly very often, but the question seems to be general to JPA so I wanted to pose it that way.
Design your models like this and play with UserGroup for associations.
#Entity
public class User {
#OneToMany(cascade = CascadeType.ALL, mappedBy = "user",fetch = FetchType.LAZY)
#OnDelete(action = OnDeleteAction.CASCADE)
private Set<UserGroup> userGroups = new HashSet<UserGroup>();
}
#Entity
#Table(name="user_group",
uniqueConstraints = {#UniqueConstraint(columnNames = {"user_id", "group_id"})})
public class UserGroup {
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "user_id", nullable = false)
#ForeignKey(name = "usergroup_user_fkey")
private User user;
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "group_id", nullable = false)
#ForeignKey(name = "usergroup_group_fkey")
private Group group;
}
#Entity
public class Group {
#OneToMany(cascade = CascadeType.ALL, mappedBy="group", fetch = FetchType.LAZY )
#OnDelete(action = OnDeleteAction.CASCADE)
private Set<UserGroup> userGroups = new HashSet<UserGroup>();
}
Do like this.
User user = findUserId(id); //All groups wont be loaded they are marked lazy
Group group = findGroupId(id); //All users wont be loaded they are marked lazy
UserGroup userGroup = new UserGroup();
userGroup.setUser(user);
userGroup.setGroup(group);
em.save(userGroup);
Using the ManyToMany mapping effectively is caching the collection in the entity, so you might not want to do this for large collections, as displaying it or passing the entity around with it triggered will kill performance.
Instead you might remove the mapping on both sides, and create an entity for the relation table that you can use in queries when you do need to access the relationship. Using an intermediate entity will allow you to use paging and cursors, so that you can limit the data that might be brought back into usable chunks, and you can insert a new entity to represent new relationships with ease.
EclipseLink's attribute change tracking though does allow adding to collections without the need to trigger the relationship, as well as other performance enhancements. This is enabled with weaving and available on collection types that do not maintain order.
The collection classes returned by getUsers() and getGroups() don't have to have their contents resident in memory, and if you have lazy fetching turned on, as I assume you do for such a large relationship, the persistence provider should be smart enough to recognize that you're not trying to read the contents but just adding a value. (Similarly, calling size() on the collection will typically cause a SQL COUNT query rather than actually loading and counting the elements.)