I have encountered an issue where I save a list of entities all at once, sometimes some rows have values being written to wrong columns.
Basically, I have a Movie entity, which extends Show (annotated with #MappedSuperclass), which extends TraceableEntity that is also annotated with #MappedSuperclass as shown below:
#MappedSuperclass
#EntityListeners(TraceableEntity.TraceableEntityListener.class)
public abstract class TraceableEntity {
#Column
private Date createdOn;
#Column
private Date dateUpdated;
public Date getCreatedOn() {
return createdOn;
}
public void setCreatedOn(Date createdOn) {
this.createdOn = createdOn;
}
public Date getDateUpdated() {
return dateUpdated;
}
public void setDateUpdated(Date dateUpdated) {
this.dateUpdated = dateUpdated;
}
public static class TraceableEntityListener {
#PrePersist
public void beforeInsert(TraceableEntity entity) {
if (entity.getCreatedOn() == null) {
entity.setCreatedOn(new Date());
}
}
#PreUpdate
public void beforeUpdate(TraceableEntity entity) {
entity.setDateUpdated(new Date());
}
}
}
Now, on some occasions, the value of createdOn ends up in dateUpdated, as shown in this screenshot.
In a nutshell, my application is a scraper that retrieves data from an API. I'm using RestTemplate in CompletableFuture to download data concurrently, and then save everything in one go. The method in which .save(...) is invoked is annotated with #Transactional. When the size of the list is under approximately 1500 the saving is fine, it seems that things go wrong when the size exceeds 1500 for some reason. I'd really appreciate your time and help in this matter!
Does it always happen at the same place? Are you missing some rows for example? perhaps the text of some of the stuff you're scraping has special characters that were not escaped properly? You may want to turn on your logging to see exactly what is being sent to the server.
The reason that if you scrape just the "problematic" movies and it turns out fine is probably because the actual problem occurred before the movies in question.
Related
I have entity Article I'm trying to retrieve one random record from database collection.
This is entity Article:
#Data
#Document(value = "article")
public class Article {
#Id
private String articleId;
private String title;
private String description;
private String fullArticle;
This is service to save it:
#Override
public Article save(Article article) {
return articleRepository.save(article);
}
And repository:
#Repository
public interface ArticleRepository extends MongoRepository<Article, String> {
}
So, now I'm trying to create a method that will get me one random record from my collection Article also, I want to create a controller so when I go to some endpoint and submit some get method to retrieve one record from the collection and so I can check it in postman or with Swagger. I find some answers to similar question to mine but no one solved my problem, I want to have API for something like that.
You can use $sample in an aggregation query to get a random document:
db.collection.aggregate([
{
"$sample": {
"size": 1
}
}
])
Example here
I've tested the code and it works as expected.
You can create add this method into repository:
#Aggregation(pipeline={"{$sample:{size:1}}"})
AggregationResults<Article> random();
And call from service like this:
#Override
public Article random(){
return articleRepository.random().getMappedResults().stream().findFirst().orElse(null);
// also you can use .orElseThrow() or whatever you want
}
Like/dislike system.
App has entity Post. Post has field List likes and it joins table, which has columns post_id, user_id.
When User presses button "like" app will add authenticated user in List in PostService. But I need to have the "isLiked" boolean field. This will define what the Like button will look like in frontend.
I can get value for field countLike just call method size() from field likes.
But I don't know now I can get value for field "isLiked".
Help me with it, please.
#Entity
public class Post {
//some fields...
//there I saved users, who has posed "like"
private List<User> likes;
#Transient
private int countLike;
//there I want to save status - liked/disliked;
#Transient
private boolean isLiked;
}
#Service
public class PostService {
//some fields and methods...
public void createLike(int postId, User authenticatedUser) {
Post post = postRepository.getOne(postId);
post.getLikes().add(authenticatedUser);
this.update(post);
}
}
While the approach hinted to by yourself and the comment by "Lino - Vote don't say Thanks" will work, I don't think they are a very good idea.
Working but wastful.
You can create methods like the following:
public int likeCount{}{
return getLikes().size();
}
public boolean isLiked(){
return getLikes().size() > 0;
}
The problem with that is, it will load a lot of data just for providing a single number of even just a single bit of information.
More efficient in most scenarios
Instead I recommend loading the information from the database.
Assuming you are using Hibernate as the JPA implementation you can do that with the #Formula annotation. With this the relevant code looks like this:
#Entity
public class Post {
#Formula("(select count(user_id) from Likes l where l.post_id = post_id)")
private int countLike;
public boolean isLiked(){
return getCountLike() > 0;
}
}
For a simple Java REST-API I created a save function to persist my model to a DynamoDB table.
The model uses a auto generated range key as you can see here:
#DynamoDBTable(tableName = "Events")
public class EventModel {
private int country;
private String id;
// ...
#DynamoDBHashKey
public int getCountry() {
return country;
}
public void setCountry(int country) {
this.country = country;
}
#DynamoDBRangeKey
#DynamoDBAutoGeneratedKey
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
//...
}
Unfortunately the the DynamoDBMappers .save() method does not return anything. I want to return the created item to set the proper location header in my 201 HTTP response.
public EventModel create(EventModel event) {
mapper.save(event);
return null;
}
How can I make that work? Any suggestions? Of course I could generate the id on the client but I don´t want to do this because solving the potential atomicity issue needs additional logic on client- and server-side.
I´m using the aws-java-sdk-dynamodb in version 1.11.86.
Never mind, I figured out how to do it. The .save() method updates the reference of the object. After calling mapper.save(event); the id property is populated and has its value.
So the way to get it work is just:
public EventModel create(EventModel event) {
mapper.save(event);
return event;
}
That´s it!
There is direct way through dynamo db mapper to get what is saved in dynamodb after put/update Approach mentioned by m4xy# would work if you are saving with DynamoDBConfig as CLOBBER or UPDATE. If you are using UPDATE_SKIP_NULL_ATTRIBUTES, this approach won't work.
If you are using mapper, you have to specifically call db again to get existing value (which might have been updated if there are multiple writers and you might get unxpected result). To ensure read that you expect you can implement locking for write such that if lock is acquired by a given thread, no other thread can write for a given key. But, this approach as a downside of slowing down your application.
Alternatively, you can use dynamoDBClient that has apis to support return db values after write.
https://sdk.amazonaws.com/java/api/2.0.0-preview-11/index.html?software/amazon/awssdk/services/dynamodb/DynamoDbClient.html
I am developing a program with play 2.3(with Ebean and H2) for java. I have a model like this:
#Entity
public class DeviceModel extends Model implements PathBindable<DeviceModel> {
#Id
public Long id;
#Lob
#Basic(fetch=FetchType.LAZY)
public byte[] picture;
...
In my controller I have a function which writes a picture as byte[] inside a DeviceModel object and calls the update() function. so now the picture should be saved in database.
And i have this function to show the picture:
public static Result picture(Long id) {
final DeviceModel deviceModel = DeviceModel.findByID(id);
if (deviceModel == null){
return notFound();
}
return ok(deviceModel.picture);
}
the funny thing is that deviceModel.picture is null!
but in my view, I have this:
#if(deviceModel.picture != null) {
show the picture!
} else{
do something else
}
but here, deviceModel.picture is not null!!! and MOST OF THE TIMES the picture will be shown correctly!!
I deleted the #Basic(fetch=FetchType.LAZY) but it didn't solve the problem.
any Idea why is it like this?
I found a work around for this issue, but I still like to know the reason, why accessing the picture field directly, returns null.
here is the work around:
I just made my picture field private, and made getter and setter my self. now in my Controller, with getPicture() I always get the data
#Entity
public class DeviceModel extends Model implements PathBindable<DeviceModel> {
#Id
public Long id;
#Lob
#Basic(fetch=FetchType.LAZY)
private byte[] picture;
public byte[] getPicture() {
return picture;
}
public void setPicture(byte[] picture) {
this.picture = picture;
}
...
The reason for that behavior is the FetchType.LAZY (which apparently is also the default for Lobs). It tells Ebean to fetch the data lazily, i.e. not immediately when you load the object, but only when you actually access it.
As Ebean cannot detect the access when you go directly at the field (picture), the loading doesn't happen at all and you get null.
By using getPicture(), the code of which is enhanced by Ebean, it knows to load the data before returning the value.
You can overcome that behavior by simply using FetchType.EAGER. But you should only do that, if you're sure you need the data always, because it takes more time and memory (e.g. in your example, if you have 100s of pictures and you only want to show a list of names, it would not be necessary to load all the pictures' actual image data as well.
Let's for example say I have the following objectify model:
#Cache
#Entity
public class CompanyViews implements Serializable, Persistence {
#Id
private Long id;
private Date created;
private Date modified;
private Long companyId;
........
private Integer counter;
........
#Override
public void persist() {
persist(false);
}
#Override
public void persist(Boolean async) {
ObjectifyService.register(Feedback.class);
// setup some variables
setUuid(UUID.randomUUID().toString().toUpperCase());
setModified(new Date());
if (getCreated() == null) {
setCreated(new Date());
}
// do the persist
if (async) {
ofy().save().entity(this);
} else {
ofy().save().entity(this).now();
}
}
}
I want to use the counter field to track the number of views, or number opens or basically count something using an integer field.
What happens now is that for one GAE instance, the following will be called:
A:
CompanyViews views = CompanyViews.findByCompanyId(...);
views.setCounter(views.getCounter() + 1);
views.persist();
and for another instance:
B:
CompanyViews views = CompanyViews.findByCompanyId(...);
views.setCounter(views.getCounter() + 1);
views.persist();
If they both read the counter at the same time or read the counter before the other instance has persisted it, they will overwrite each other.
In MySQL / Postgres you get row-level locking, how does one do a "row-level lock" for Objectify entities on GAE?
You need to use transactions when concurrently updating entities.
Note that since you update same entity you will have a limitation of about 1 write/s. To work around that look into sharding counters.