We Keep Coding

Unique constraint with saveAll from JpaRepository

My application receives arrays containing objects, which are mapped to entities and then persisted.
The object has an id property that is mapped to a column with a unique constraint placed on it. Objects with duplicate ids may be sent to the application.
I'm using the saveAll method, but it throws during insertion if any of the objects happens to violate the unique constraint.
Is there an easy way to make it insert all the non-duplicates and ignore the duplicates, or simply update them?
I've tried overriding hash and equals, but that didn't help.
This application receives dozens of requests per second and he amount of duplicates I expect to receive is low.
entity:
#Table(name = "table", uniqueConstraints = #UniqueConstraint(columnNames = "natural_id"))
public class Entity {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
#Column(name = "natural_id", unique = true)
private String naturalId;
// ...
}
repository:
public interface Repository extends JpaRepository<Entity, Long>, JpaSpecificationExecutor<Entity> {}
saving method:
void save(List<Entities> entities) {
try {
repository.saveAll(entities);
} catch (Exception e) {
log.error("...");
throw e;
}
}
If I were to change the saving method to:
#Transactional
void save(List<Entities> entities) {
for (Entities et: entities) {
try {
repository.save(et);
} catch (Exception e) {
log.error("...");
}
}
}
Would that make it all happen within a single transaction? I'm worried because the save method usually creates a new transaction every time it is called, and that would make it extremely slow for an application that receives a decent number of requests per second.
I might ultimately have to query the database before insertion to filter the duplicates out for insertion, and do that again every time a constraint violation exception is caught. It should work, but is quite ugly.

You have to save individual records because save all will not be resume/recover if faced problem with some record and then other record will also not saved as all are in same transaction.
As you suggested
query the database before insertion to filter the duplicates out instead of that
better try for loop with try catch as you mentioned ''dozens of requests per second'' so it wont give much overhead.

If you already implemented hashCode and equals then use Set instead of List to collect your items before saving them.

Implementing news feed on GAE - Should I use Prospective Search?

I have an issue i'm struggling with for some time now. Im trying to implement a news feed feature in my app using GAE cloud endpoints and java. The common concept is of followers and followees, where an action of a followee can be seen by his followers. A new follower should also see his followees past actions, not only from the time he started following.
I made a few tries with the following components. Each try worked great but was lacking something:
On each user action i added a 'log' entity into the datastore with the user id included. When a user was displaying his news feed i just queried for all those entities by their user ids according to the user's followees list. Everything was fine until i realized that a 'IN' query cannot be cursored. So this option was gone.
On this try, which is also the current state of the application, im using the Search API. Upon every user action im not storing a 'log' entity into the datastore anymore but a document into a search index. Complex queries can be cursored here and the world is smiling again. But... im not too sure that, billing wise, this is a smart descision. It seems that the costs of searching/adding/deleting documents along side the documented daily limitations is making the whole thing a bit too sketchy.
The next try should be Prospective Search API. From what i'm reading in the documents it seems the right component to pick for that purpose. Unfortunately, the documentation is really poor and give very little examples. Also the billing information is unclear.
So im asking for the advice of the stackoverflow community. Can you please advise me about this matter ? and if Prospective Search is the right option to choose, can you please provide some clear sample java code that uses cloud endpoints?
EDIT : Just to emphasize the main design requirement here - The news feed feature need to have the ability to fetch sorted followees actions using a cursor (in order avoid querying the whole batch).

Use a pull-aggregate-per-follower model: periodically (or on demand) query all followees actions once and then cache them inside a dedicated per-follower entity. Remember the time of last query, so next time you just query from that point on (assuming actions can not be added/changed to the past times).
This will give you the following features (and limitations):
If query is on-demand, than you will not need to query for users that are inactive.
Since the query is "new-only" (looks for new actions only), it would cost you nothing if it returned zero results.
You will only query each followee actions per follower once. After that all recent actions would be cached inside one entity and loaded into memory with one get. This should be a substantial cost and time saving.
You could sort/filter actions in memory any way you wish.
Limitations:
Entities have a 1MB limit, so there is a max no of actions that you can cache in one entity. So you will either need to limit caching of recent actions per user or spread out action caching over multiple entities.
You will need to use IN query over followees (max 30) and also use parallel threads to achieve decent performance. This could easily hit 3-5 seconds when querying over 1000-2000 followees. Also, you could easily hit RPC limit (aka max concurrent API calls) per instance when serving multiple users at the same time.

I hope I understand the question correctly - you want to implement a news feed into your application and allow users to follow each other. The new followers need to be able to see the users actions. I am sure there are multiple other ways of solving this problem, but I will attempt to help you out by providing a solution that makes use of JAVA JDO to access the datastore.
I would first design the entity relationships in JDO as follows:
1 User to many actions.
1 User to many followers (User).
1 User to many following (User).
Here are simple JDO classes:
User Class:
#PersistenceCapable(identityType=IdentityType.APPLICATION)
public class User {
#PrimaryKey
#Persistent(valueStrategy=IdGeneratorStrategy.IDENTITY)
private Key key;
#Persistent
private String userId; // Google unique user ID, could also store user email.
#Persistent
private Set<Key> actions;
#Persistent
private Set<Key> followers;
#Persistent
private List<Key> following;
public User(Key key, String userId) {
this.key = key;
this.userId = userId;
this.actions = new HashSet<Key>();
this.followers = new HashSet<Key>();
this.following = new HashSet<Key>();
}
public Key getKey() {
return this.key;
}
public void addAction(Key actionKey) {
this.actions.add(actionKey);
}
public void addActions(Set<Key> actionKeys) {
this.actions.addAll(actionKeys);
}
public Set<Key> getActions() {
return this.actions;
}
public void addFollower(Key followerKey) {
this.followers.add(followerKey);
}
public void addFollowers(Set<Key> followerKeys) {
this.followers.addAll(followerKeys);
}
public Set<Key> getFollowers() {
return this.followers;
}
public void addFollowing(Key followingKey) {
this.following.add(followingKey);
}
public void addAllFollowing(Set<Key> followingKeys) {
this.following.addAll(followingKeys);
}
public Set<Key> getFollowing() {
return this.following;
}
}
Action Class:
#PersistenceCapable(identityType=IdentityType.APPLICATION)
public class Action {
#PrimaryKey
#Persistent(valueStrategy=IdGeneratorStrategy.IDENTITY)
private Key key;
#Persistent
Date date;
#Persistent
private String title;
public Action(Key key, String title) {
this.key = key;
this.title = title;
this.date = new Date(); // date of creation (now).
}
public Key getKey() {
return this.key;
}
public void setTitle(String title) {
this.title = title;
}
public String getTitle() {
return this.title;
}
}
The Action class makes use of a Date property, you can refer to the documentation for applicable data types in the datastore. When an action is created, a Date object is allocated and initialized so that it represents the time at which it was allocated, measured to the nearest millisecond.
In my example above I linked the entities by their Keys, you could instead link them by their classes as follows:
List<Action> actions;
The relationship in my example is one of an unowned one-to-many relationship, perhaps it should be owned one-to-many. More information here for your to take a look and perhaps decide which would be best for your solution.
Once the relationships have been defined, you can create your endpoint classes around the JDO model classes. This will create basic api methods. You might want to change the endpoint class methods to suit your needs, for example change the way an action is created. A basic example would be to create the key from the actions title as follows (ActionEnpoint.java):
...
#ApiMethod(name = "insertAction")
public Action insertAction( #Named("title") String title ) {
PersistenceManager pm = getPersistenceManager();
Key key = KeyFactory.createKey(Action.class.getSimpleName(), title);
Action action = null;
try {
action = new Action(key, title);
pm.makePersistent(action);
} finally {
pm.close();
}
return action;
}
...
If you want to, you can add a method to your UserEndpoint class to query the datastore and return all actions belonging to that user and per date using the datastore query objects.
You need to add a method to your UserEndpoint class that allows you to add an action to that user, here is a simple example:
...
#ApiMethod(name = "addActionToUser")
public Achiever addActionToUser(
#Named("userId") String userId,
#Named("actionTitle") String actionTitle) {
PersistenceManager pm = getPersistenceManager();
Key userKey = KeyFactory.createKey(User.class.getSimpleName(), userId);
Key actionKey = KeyFactory.createKey(Action.class.getSimpleName(), actionTitle);
User user = null;
try {
user = (User) pm.getObjectById(User.class, userKey);
user.addAction(actionKey);
pm.makePersistent(user);
} catch (Exception e) {
}
return user;
}
...
Once all of the above is complete you can easily get the list of actions per user by calling the getUser method in your UserEndpoint class, which returns a User object. You can then call [ReturnedUserObject].getActions(). A new follower can now view all of the "followees" actions by just calling the api method to get that "followees" object and get his/her actions. You can then just sort the actions by date or however you envision it.
I hope I understood your question correctly, I was unsure about the first component you mentioned, but it seemed as though you got your relationships mixed up. I hope this solution points you in the right direction at least :).
If you need any additional help or clarification, or my answer was completely off point to what you were looking for then please let me know.
Kind regards,
Miki

ManyToMany NOT NULL check constraint when using Named query

I have an Entity (Layer) that maps a list of other Entities (Member). This List may have no entries / be null. Yet, when I query for the Entity I get a NOT NULL check constraint error from the database.
It seems to be connected to the NamedQueries as I can read the Entity from DB if I query by id.
#Entity
#NamedQueries({
#NamedQuery(name="getChildLayers",-
query = "SELECT la
FROM Layer la
WHERE la.parent = :parent AND la.deletedDate IS NULL")})
public class Layer extends CommonModel {
/*... other field */
#ManyToOne(fetch = FetchType.LAZY, targetEntity = Layer.class, optional = true)
private Layer parent;
#ManyToMany(fetch = FetchType.LAZY, targetEntity = MyUser.class)
private List<MyUser> members;
public List<MyUser> getMembers() {
return members;
}
public void setMembers(List<MyUser> members) {
this.members = members;
}
/*... other getters and setters */
}
I get this error: integrity constraint violation: NOT NULL check constraint; SYS_CT_10298 table: LAYER_MYUSER column: MEMBERS_ID
I am able to create the entry, though.
When I run my tests then all tests fail that read the Entity (but creation works). If I add the following line in the creation method:
layer.setMembers(new ArrayList<MyUser>());
then the methods that test the alternation of the members work (meaning, I can create a Layer and alter its members by adding and removing elements from the list).
It seems to me that reading the Entity from Database fails whenever there are no Member to the Layer.
I did try adding #JoinColumn(nullable=true) to the field, but it changed nothing.
I import javax.persistence classes.
Example as to how I access the variable (in LayerService)
// this method works as expected
public Layer getById(Long id) {
Session s = sessionFactory.getCurrentSession();
return (Layer)s.get(Layer.class, id);
}
// this does not.
public List<Layer> getChildren(Layer layer) {
Query childrenQuery = sessionFactory.getCurrentSession().getNamedQuery("getChildLayers");
childrenQuery.setParameter("parent", layer);
return (List<Layer>) childrenQuery.list();
}
Code changed after Jason Cs answer:
Layer
...
private final List<OCWUser> members = new ArrayList<>();
...
public void setMembers(List<OCWUser> members) {
this.members.clear();
this.members.addAll(members);
}
Problem still exists.

It can be so simple. I forgot to add #JoinTable
#JoinTable(name = "LAYER_USER", joinColumns = #JoinColumn(nullable = true))

One important thing to be aware of is you shouldn't replace this.members with another list in setMembers unless you know you are doing it before you call persist(). Instead you need to clear this.members then add all the specified elements to it. The reason is that Hibernate can and will use its own proxied / instrumented collections classes when [de]serializing an entity, and you blow that away when overwriting the collection class. You should declare members as final and always initialize it to a non-null empty List.
See for example (3.6 but still relevant): http://docs.jboss.org/hibernate/core/3.6/reference/en-US/html/collections.html#collections-persistent, In particular:
Notice how in Example 7.2, “Collection mapping using #OneToMany and
#JoinColumn” the instance variable parts was initialized with an
instance of HashSet. This is the best way to initialize collection
valued properties of newly instantiated (non-persistent) instances.
When you make the instance persistent, by calling persist(), Hibernate
will actually replace the HashSet with an instance of Hibernate's own
implementation of Set.
As long as you are messing with collection fields in this way, any number of strange things can happen.
Also, in general, you want to be careful about stating your invariants and such when accessing collections in this way, as it's easily possible to, e.g., create two Layers that reference the same collection internally, so that actions on one affect the other, or external actions on the passed-in collection affect the layer, e.g. the following code probably doesn't behave like you want it to:
List<MyUser> u = new ArrayList<MyUser>();
Layer a = new Layer();
Layer b = new Layer();
u.add(...);
a.setMembers(u);
b.setMembers(u);
u.clear();
Further, when you persist() one of the layers there, and Hibernate overwrites the field with its own collection class, the behavior then changes as the objects are no longer referencing the same collection:
// not only did u.clear() [possibly undesirably] affect a and b above, but:
session.persist(a);
u.add(...); // ... now it only affects b.

One-to-many relationship: Update removed children with JPA 2.0

I have a bidirectional one-to-many relationship.
0 or 1 client <-> List of 0 or more product orders.
That relationship should be set or unset on both entities:
On the client side, I want to set the List of product orders assigned to the client; the client should then be set / unset to the orders chosen automatically.
On the product order side, I want to set the client to which the oder is assigned; that product order should then be removed from its previously assiged client's list and added to the new assigned client's list.
I want to use pure JPA 2.0 annotations and one "merge" call to the entity manager only (with cascade options). I've tried with the following code pieces, but it doesn't work (I use EclipseLink 2.2.0 as persistence provider)
#Entity
public class Client implements Serializable {
#OneToMany(mappedBy = "client", cascade= CascadeType.ALL)
private List<ProductOrder> orders = new ArrayList<>();
public void setOrders(List<ProductOrder> orders) {
for (ProductOrder order : this.orders) {
order.unsetClient();
// don't use order.setClient(null);
// (ConcurrentModificationEx on array)
// TODO doesn't work!
}
for (ProductOrder order : orders) {
order.setClient(this);
}
this.orders = orders;
}
// other fields / getters / setters
}
#Entity
public class ProductOrder implements Serializable {
#ManyToOne(cascade= CascadeType.ALL)
private Client client;
public void setClient(Client client) {
// remove from previous client
if (this.client != null) {
this.client.getOrders().remove(this);
}
this.client = client;
// add to new client
if (client != null && !client.getOrders().contains(this)) {
client.getOrders().add(this);
}
}
public void unsetClient() {
client = null;
}
// other fields / getters / setters
}
Facade code for persisting client:
// call setters on entity by JSF frontend...
getEntityManager().merge(client)
Facade code for persisting product order:
// call setters on entity by JSF frontend...
getEntityManager().merge(productOrder)
When changing the client assignment on the order side, it works well: On the client side, the order gets removed from the previous client's list and is added to the new client's list (if re-assigned).
BUT when changing on the client side, I can only add orders (on the order side, assignment to the new client is performed), but it just ignores when I remove orders from the client's list (after saving and refreshing, they are still in the list on the client side, and on the order side, they are also still assigned to the previous client.
Just to clarify, I DO NOT want to use a "delete orphan" option: When removing an order from the list, it should not be deleted from the database, but its client assignment should be updated (that is, to null), as defined in the Client#setOrders method. How can this be archieved?
EDIT: Thanks to the help I received here, I was able to fix this problem. See my solution below:
The client ("One" / "owned" side) stores the orders that have been modified in a temporary field.
#Entity
public class Client implements Serializable, EntityContainer {
#OneToMany(mappedBy = "client", cascade= CascadeType.ALL)
private List<ProductOrder> orders = new ArrayList<>();
#Transient
private List<ProductOrder> modifiedOrders = new ArrayList<>();
public void setOrders(List<ProductOrder> orders) {
if (orders == null) {
orders = new ArrayList<>();
}
modifiedOrders = new ArrayList<>();
for (ProductOrder order : this.orders) {
order.unsetClient();
modifiedOrders.add(order);
// don't use order.setClient(null);
// (ConcurrentModificationEx on array)
}
for (ProductOrder order : orders) {
order.setClient(this);
modifiedOrders.add(order);
}
this.orders = orders;
}
#Override // defined by my EntityContainer interface
public List getContainedEntities() {
return modifiedOrders;
}
On the facade, when persisting, it checks if there are any entities that must be persisted, too. Note that I used an interface to encapsulate this logic as my facade is actually generic.
// call setters on entity by JSF frontend...
getEntityManager().merge(entity);
if (entity instanceof EntityContainer) {
EntityContainer entityContainer = (EntityContainer) entity;
for (Object childEntity : entityContainer.getContainedEntities()) {
getEntityManager().merge(childEntity);
}
}

JPA does not do this and as far as I know there is no JPA implementation that does this either. JPA requires you to manage both sides of the relationship. When only one side of the relationship is updated this is sometimes referred to as "object corruption"
JPA does define an "owning" side in a two-way relationship (for a OneToMany this is the side that does NOT have the mappedBy annotation) which it uses to resolve a conflict when persisting to the database (there is only one representation of this relationship in the database compared to the two in memory so a resolution must be made). This is why changes to the ProductOrder class are realized but not changes to the Client class.
Even with the "owning" relationship you should always update both sides. This often leads people to relying on only updating one side and they get in trouble when they turn on the second-level cache. In JPA the conflicts mentioned above are only resolved when an object is persisted and reloaded from the database. Once the 2nd level cache is turned on that may be several transactions down the road and in the meantime you'll be dealing with a corrupted object.

You have to also merge the Orders that you removed, just merging the Client is not enough.
The issue is that although you are changing the Orders that were removed, you are never sending these orders to the server, and never calling merge on them, so there is no way for you changes to be reflected.
You need to call merge on each Order that you remove. Or process your changes locally, so you don't need to serialize or merge any objects.
EclipseLink does have a bidirectional relationship maintenance feature which may work for you in this case, but it is not part of JPA.

Another possible solution is to add the new property on your ProductOrder, I named it detached in the following examples.
When you want to detach the order from the client you can use a callback on the order itself:
#Entity public class ProductOrder implements Serializable {
/*...*/
//in your case this could probably be #Transient
private boolean detached;
#PreUpdate
public void detachFromClient() {
if(this.detached){
client.getOrders().remove(this);
client=null;
}
}
}
Instead of deleting the orders you want to delete you set detached to true. When you will merge & flush the client, the entity manager will detect the modified order and execute the #PreUpdate callback effectively detaching the order from the client.

JPA 2 - OneToOne cross reference

I have been struggling with this issue for a while now, and can't seem to find a solution anywhere. Maybe I have misunderstood something, but would really appreciate it if someone could clear up things for me.
I am trying to create a OneToOne bidirectional relationsship between a User.class and a FacebookProfile.class.
That is I want to be able to look up the corresponding User entity from the FacebookProfile entity, and vice versa:
userInstance.getFbprofile();
fbprofileInstance.getUser();
My User.class:
#Entity
#Access(value=AccessType.FIELD)
public class User implements UserDetails {
#OneToOne(mappedBy="user")
private FacebookProfile fbprofile;
}
My FacebookProfile.class:
#Entity
#Access(value=AccessType.FIELD)
public class FacebookProfile {
#OneToOne (cascade=CascadeType.ALL)
#JoinColumn(name="USER")
private User user;
}
The new FacebookProfile and User instances are created in a Controller, and then sent through a Service layer to a DAO class which persists the objects.
I first create and persist a new User object.
I then create a new FacebookProfile and puts the newly created User in it (setUser). I then persist the FacebookProfile object.
The User ID is now stored in the FacebookProfile database table, but there is no reference from the User to the FacebookProfile so the following code returns a NullPointerException:
User tempUser = userService.findUserById(newUser.getId());
System.out.println("ID "+tempUser.getFbprofile().getId());
Below is the code from the Controller if something is unclear:
//Check if the Facebook user already exist
FacebookProfile fbprofile = facebookProfileService.findFacebookProfileById(fbId);
User newUser;
//Create a new FacebookProfile if it doesn't exist
if(fbprofile == null){
//Check if there is a user registered with the facebook email
newUser = userService.findUserByEmail(fbEmail);
// No User and No FacebookProfile exists
if(!newUser.isEnabled()){
newUser = new User();
newUser.setFirstname(fbFirstname);
newUser.setLastname(fbLastname);
//Set email
Email mail = new Email();
mail.setAddress(fbEmail);
mail.setName("Facebook");
newUser.addEmail(mail);
//Set gender
if(fbGender.equalsIgnoreCase("female")){
newUser.setGender(Gender.FEMALE);
}else if(fbGender.equalsIgnoreCase("male")) {
newUser.setGender(Gender.MALE);
}
userService.createUser(newUser);
}
FacebookProfile newProfile = new FacebookProfile();
newProfile.setId(Long.parseLong(fbId));
newProfile.setUsername(fbUsername);
newProfile.setFirstname(fbFirstname);
newProfile.setLastname(fbLastname);
newProfile.setName(fbName);
newProfile.setEmail(fbEmail);
if(!fbHometown.equals("")){
newProfile.setHometown(fbHometown);
newProfile.setHometownID(Long.parseLong(fbHometownID));
}
if(!fbLocation.equals("")){
newProfile.setLocation(fbLocation);
newProfile.setLocationID(Long.parseLong(fbLocationID));
}
newProfile.setLink(fbLink);
if(fbGender.equalsIgnoreCase("male")){
newProfile.setGender(Gender.MALE);
}else if (fbGender.equalsIgnoreCase("female")) {
newProfile.setGender(Gender.FEMALE);
}
newProfile.setUser(newUser);
this.facebookProfileService.createNewFacebookProfile(newProfile);
//newUser.setFbprofile(newProfile);
}else { //There already exists a FacebookProfile
newUser = fbprofile.getUser();
}
I feel like I have tried every possible solution to this and haven't got it to work.
I suspect that in one of my earlier attempts that the fbprofile reference was set in the User object, but never persisted because it was set after the User object was persisted. I am using JPA2 and EclipseLink.
If someone's got a solution to this it would be much appreciated.

Bi-Directional relations is not maintained by JPA. JPA designers had thought that maintaining bi-directional relations can cause deadlocks, inconsistencies... Toplink was supporting bi-directional relations before JPA.. I have used Toplink for many years, I did come accross some inconsistencies related to bi-directional relations.. You should maintain bi-directional relations manually.. Such as ;
newProfile.setUser(newUser);
newUser.setFbProfile(newProfile);
EDIT :
Well, you want me to elaborate my answer.. You should change your code that persist FbProfile entity such as ;
newProfile.setUser(newUser);
newUser.setFbprofile(newProfile);
this.facebookProfileService.createNewFacebookProfile(newProfile);
As an alternative , you could also change FbProfile setUser method ;
public void setUser(User aUser) {
if (this.user==aUser)
return;
if (this.user != null) {
this.user.setFbProfile(null);
}
this.user = aUser;
if (this.user!=null) {
this.user.setFbprofile(this);
}
}