I want to insert a list of Objects in my db. In a special case I know that they primary key (not auto-generated) is not already there. Since I need to insert a big collection, the save(Iterable<Obj> objects) is to slow.
Therefore I consider using a native query. native insert query in hibernate + spring data
In the previous answer, it does not say how to insert a collection of objects. Is this possible?
#Query("insert into my_table (date, feature1, feature2, quantity) VALUES <I do not know what to add here>", nativeQuery = true)
void insert(List<Obj> objs);
Of course if you have a better solution overall, Its even better.
I ended up implementing my own repository. The performance of this is really good, 2s instead of 35s before to insert 50000 elements. The problem with this code is that it does not prevent sql injections.
I also tryed to build a query using setParameter(1, ...) but somehow JPA takes a long time to do that.
class ObjectRepositoryImpl implements DemandGroupSalesOfDayCustomRepository {
private static final int INSERT_BATCH_SIZE = 50000;
#Autowired
private EntityManager entityManager;
#Override
public void blindInsert(List<SomeObject> objects) {
partition(objects, INSERT_BATCH_SIZE).forEach(this::insertAll);
}
private void insertAll(List<SomeObject> objects) {
String values = objects.stream().map(this::renderSqlForObj).collect(joining(","));
String insertSQL = "INSERT INTO mytable (date, feature1, feature2, quantity) VALUES ";
entityManager.createNativeQuery(insertSQL + values).executeUpdate();
entityManager.flush();
entityManager.clear();
}
private String renderSqlForObj(Object obj) {
return "('" + obj.getDate() + "','" +
obj.getFeature1() + "','" +
obj.getFeature2() + "'," +
obj.getQuantity() + ")";
}
}
Related
I am using a Native query in my JPA Repository to run INSERT query - because, I couldn't run a few queries through JPA.
String INSERT_USER_IN_TO_DATABASE = "INSERT INTO USER_MASTER " +
"(" +
"MOBILE_X,USER_TYPE,COUNTRYCODE,USER_N,USER_LAST_N,GENDER,DOB) " +
"VALUES ( " +
"EncryptByKey(Key_GUID(convert(varchar,?1)), ?2)," +
"1,+91,?3,?4,?5,?6" +
")";
#Query(value = INSERT_USER_IN_TO_DATABASE, nativeQuery = true)
#Modifying
UserMasterEntity saveNewUser(String encryptionKey,
String phoneNumber,
String name,
String lastName,
String gender,
String dob);
My intention is to execute the INSERT statement and get me the userMaster entity. But I get the below exception
Caused by: java.lang.IllegalArgumentException: Modifying queries can only use void or int/Integer as return type! Offending method: public abstract com.officeride.fileprocess.job.bulkuser.UserMasterEntity com.officeride.fileprocess.job.bulkuser.UserMasterRepository.saveNewUser(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String)
I used #ColumnTransformer stuff too in my entity and nothing is working out for me.
How to tackle this?
You cannot make the INSERT INTO statement return anything. Depending on your database's support, on that end you could execute multiple statements, e.g., INSERT INTO ...; SELECT ..., but Spring Data JDBC does not support this.
What you can do is implement saveNewUser and perform the insert then select, sequentially and synchronously. See this: https://docs.spring.io/spring-data/jdbc/docs/current/reference/html/#repositories.single-repository-behavior.
If your database supports it, you could create a stored procedure that performs the insert then select.
I have an issue with mapping retrieved data via JDBi3 using PostgreSQL query in my DAO interface.
In my Dropwizard app I have Book DTO class which is has Many-To-Many relation with Author and Category DTO classes and have a problem with mapping queried rows onto BookDTO class. Here are the code snippets of DTO classes:
class BookDTO {
private Long bookId;
// other fields are left for code brevity
private List<Long> authors;
private List<Long> categories;
// empty constructor + constructor with all fields excluding Lists + getters + setters
}
class AuthorDTO {
private Long authorId;
// other fields are left for code brevity
private List<Long> books;
// empty constructor + constructor with all fields excluding List + getters + setters
}
class CategoryDTO {
private Long categoryId;
// other fields are left for code brevity
private List<Long> books;
// empty constructor + constructor with all fields excluding List + getters + setters
}
...and since I am using JDBi3 DAO interfaces for performing CRUD operations this is how my method for querying all books in database looks like:
#Transaction
#UseRowMapper(BookDTOACMapper.class)
#SqlQuery("SELECT book.book_id AS b_id, book.title, book.price, book.amount, book.is_deleted, author.author_id AS aut_id, category.category_id AS cat_id FROM book " +
"LEFT JOIN author_book ON book.book_id = author_book.book_id " +
"LEFT JOIN author ON author_book.author_id = author.author_id " +
"LEFT JOIN category_book ON book.book_id = category_book.book_id " +
"LEFT JOIN category ON category_book.category_id = category.category_id ORDER BY b_id ASC, aut_id ASC, cat_id ASC")
List<BookDTO> getAllBooks();
...and this is map method of BookDTOACMapper class look like:
public class BookDTOACMapper implements RowMapper<BookDTO> {
#Override
public BookDTO map(ResultSet rs, StatementContext ctx) throws SQLException {
final long bookId = rs.getLong("b_id");
// normally retrieving values by using appropriate rs.getXXX() methods
Set<Long> authorIds = new HashSet<>();
Set<Long> categoryIds = new HashSet<>();
long authorId = rs.getLong("aut_id");
if (authorId > 0) {
authorIds.add(authorId);
}
long categoryId = rs.getLong("cat_id");
if (categoryId > 0) {
categoryIds.add(categoryId);
}
while (rs.next()) {
if (rs.getLong("b_id") != bookId) {
break;
} else {
authorId = rs.getLong("aut_id");
if (authorId > 0) { authorIds.add(authorId); }
categoryId = rs.getLong("cat_id");
if (categoryId > 0) { categoryIds.add(categoryId); }
}
}
final List<Long> authorIdsList = new ArrayList<>(authorIds);
final List<Long> categoryIdsList = new ArrayList<>(categoryIds);
return new BookDTO(bookId, title, price, amount, is_deleted, authorIdsList, categoryIdsList);
}
}
Problem I encounter is that when invoking my GET method (defined in Resource class which invokes getAllBooks() method from BookDAO class) displays inconsistent results while the query itself returns proper results.
Many questions that I've managed to find on Stackoverflow, official JDBi3 Docs API and Google Groups are considering One-To-Many relationship and using #UseRowReducer annotation which contains class which impelements LinkedHashMapRowReducer<TypeOfEntityIdentifier, EntityName> but for this case I could not find a way to implement it. Any example/suggestion is welcome. :)
Thank you in advance.
Versions of used tools:
Dropwizard framework 1.3.8
PostgreSQL 11.7
Java8
This will be too long for a comment:
This is basically a debugging question. Why?
while (rs.next()) {
if (rs.getLong("b_id") != bookId) {
break;
} else {
The firstif after the while is eating the row after the current (the one that wass current when the row mapper is called). You are skipping the processing there (putting the data in the Java objects) for the bookId, authorId, etc. That's why you get
inconsistent results while the query itself returns proper results.
So you need to revisit how you process the data. I see two paths:
Revisit the logic of the processing loop to store the data when stopping the processing for given bookId. It is possible to achieve this with scrollable ResultSets - i.e. request a scrollable ResultSet and before the brake; call rs.previous(). On the next call to the row mapper the processing will start from the correct line in the result set.
Use the power of the SQL/PostgreSQL and do it properly: https://dba.stackexchange.com/questions/173831/convert-right-side-of-join-of-many-to-many-into-array Aggregate and shape the data in the database. The database is the best tool for this job.
Also take your time and check the other answers of https://dba.stackexchange.com/users/3684/erwin-brandstetter. They give invaluable insights in the SQL and PostgreSQL.
As zloster mentioned in his answer I've chosen 2nd option (by this answer for Many-To-Many relationships) which was to use edit my PostgreSQL query #SqlQuery annotation above List<BookDTO> getAllBooks(); method. Query now uses array_agg aggregate function in SELECT statement to group my results in an ARRAY and now looks like this:
#UseRowMapper(BookDTOACMapper.class)
#SqlQuery("SELECT b.book_id AS b_id, b.title, b.price, b.amount, b.is_deleted, ARRAY_AGG(aut.author_id) as aut_ids, ARRAY_AGG(cat.category_id) as cat_ids " +
"FROM book b " +
"LEFT JOIN author_book ON author_book.book_id = b.book_id " +
"LEFT JOIN author aut ON aut.author_id = author_book.author_id " +
"LEFT JOIN category_book ON category_book.book_id = b.book_id " +
"LEFT JOIN category cat ON cat.category_id = category_book.category_id " +
"GROUP BY b_id " +
"ORDER BY b_id ASC")
List<BookDTO> getAllBooks();
Therefore map(..) method of BookDTOACMapper class had to be edited and now looks like this:
#Override
public BookDTO map(ResultSet rs, StatementContext ctx) throws SQLException {
final long bookId = rs.getLong("b_id");
String title = rs.getString("title");
double price = rs.getDouble("price");
int amount = rs.getInt("amount");
boolean is_deleted = rs.getBoolean("is_deleted");
Set<Long> authorIds = new HashSet<>();
Set<Long> categoryIds = new HashSet<>();
/* rs.getArray() retrives java.sql.Array and after it getArray gets
invoked which returns array of Object(s) which are being casted
into array of Long elements */
Long[] autIds = (Long[]) (rs.getArray("aut_ids").getArray());
Long[] catIds = (Long[]) (rs.getArray("cat_ids").getArray());
Collections.addAll(authorIds, autIds);
Collections.addAll(categoryIds, catIds);
final List<Long> authorIdsList = new ArrayList<>(authorIds);
final List<Long> categoryIdsList = new ArrayList<>(categoryIds);
return new BookDTO(bookId, title, price, amount, is_deleted, authorIdsList, categoryIdsList);
}
Now all results are consistent and here's a screenshot of query in pgAdmin4.
I have for loop in my program where I save new objects to database. It looks like
for (String value: readvalue.readValue()) {
Value value= getValueForSomething(something);
System.out.println(value);
valueRepository.save(value);
}
And this fragment of code is executed every 30s and saving to database all values. Some values in database have two same fields and one other. How can I update values in h2 database instead of insert new?
I would suggest that within your for loop, create a method that checks to see if the object already exists in your H2 DB by querying for it using an unique identifier like an id. Use the following example RDB query as a reference:
private static final String PRODUCT_ALREADY_EXISTS_QUERY = "SELECT EXISTS(SELECT 1"
+ " FROM inventory.products "
+ " WHERE 1 = 1"
+ " AND id = :id)";
Then, if the record does exists, then call an update method that utilizes a query to UPDATE using the unique identifier. An RDB example query would be:
private static final String UPDATE_QUERY = "UPDATE inventory.products"
+ " SET (company_id, name, price, type, quantity, created_date, last_modified_date) = "
+ " (:companyId, :productName, :price, :productType, :quantity, :createdDate, :lastModifiedDateTime)"
+ " WHERE id = :id ";
If the record doesn't exist, then just create the record like you are.
I'm making the switch away from ORMs in Java and I was wondering what was the best way of dealing with many-to-one and many-to-one relationships in a non-ORM setting.
In my Customer.java class I have:
private Long id;
private String name;
private Date dob;
//About 10 more fields
private List<Pet> pets;
In Pet.java I have:
private String id;
private String name;
private Customer owner;
My database table for Pet looks like this
id BIGSERIAL PRIMARY KEY,
name VARCHAR(20),
owner_id BIGSERIAL REFERENCES...
Now I realize that if I run a query that joins the two tables, I get a "flat" data structure returned which contains the fields for both Customer and Pet as well as any foreign keys. What is the common/most efficient way to treat the data in this scenario?
a. Rebuild the object graph manually by calling customer.setName(resultSet.getString(("name"))...?
b. Use the returned data as is by converting it to a Map<String, Object>?
The data flow is: Data is read from the database -> rendered to JSON for use by an AngularJS front end -> modified data is sent back to the server for validation -> domain logic applied -> saved to database.
If you want to read both Customer and Pet in a single query for better performance, you can do something like this:
List<Customer> customers = new ArrayList<>();
String sql = "SELECT c.id AS cust_id" +
", c.name AS cust_name" +
", c.dob AS cust_dob" +
", p.id AS pet_id" +
", p.name AS pet_name" +
" FROM Customer c" +
" LEFT JOIN Pet p ON p.owner_id = c.id" +
" WHERE c.name LIKE ?" +
" ORDER BY c.id";
try (PreparedStatement stmt = conn.prepareStatement(sql)) {
stmt.setString(1, "%DOE%");
try (ResultSet rs = stmt.executeQuery()) {
Customer customer = null;
while (rs.next()) {
long cust_id = rs.getLong("cust_id");
if (customer == null || customer.getId() != cust_id) {
customer = new Customer();
customer.setId(cust_id);
customer.setName(rs.getString("cust_name"));
customer.setDob(rs.getDate("cust_dob"));
customers.add(customer);
}
long pet_id = rs.getLong("pet_id");
if (pet_id != 0) {
Pet pet = new Pet();
pet.setId(pet_id);
pet.setName(rs.getString("pet_name"));
pet.setOwner(customer);
customer.addPet(pet);
}
}
}
}
The best option at this time are:
Spring JDBC (it has convenience of ORM like bean to object mapping etc.)
iBatis (allows to write SQL queries manually although it is ORM but a thin layer)
Write your own DAO layer implementation.
In all these cases you write your own sql queries and mostly they will result in join queries. By the way the example you have given are not nested objects.
I have several entities that have transient properties that I would like to populate from the sql query. I have tried several things but have not found the correct combination yet.
here is an example entity
#Id
#GeneratedValue(strategy=GenerationType.AUTO)
long id;
#Size(max=1000)
String image_url;
#Transient
boolean liked;
#Transient
long numLikes;
**getters and setters** for persistent properties
public boolean isLiked() {
return liked;
}
public void setLiked(boolean liked) {
this.liked = liked;
}
public long getNumLikes() {
return numLikes;
}
public void setNumLikes(long numLikes) {
this.numLikes = numLikes;
}
I have tried using the #Basic anotation and setting it's fetch method to eager
#Basic(fetch = FetchType.EAGER)
on the getters but that didn't seem to do anything. I've seen it set to lazy as well.
Am I missing anything in here that would cause a column name not be mapped to one of these transient properties?
here is my impl code.
SQLQuery query = getCurrentSession().createSQLQuery("" +
"SELECT * ,count(likes1.liked) as numLikes " +
"FROM aboutme " +
"left JOIN " +
"(" +
" select liked, likedObject_id " +
" from likes " +
" where liked = 1" +
") as likes1 ON aboutme.id = likes1.likedObject_id " +
"left join " +
"(" +
" select likedObject_id, liked " +
" from likes " +
" where liked = 1 and profile_id = :id" +
") as likes2 on likes2.likedObject_id = aboutme.id " +
"WHERE aboutme.profile_id = :id " +
"group by aboutme.id");
query.addEntity(AboutMe.class);
query.setParameter("id",id);
return query.list();
I've tried using the
SQLQuery query = ..."Select count(value that returns properly) as {object.numLikes}..."
query.addEntity("object",Object.class),
I get an error here that says it can't find the column name for property [property] for alias [alias]
What am I doing wrong?
Transient properties are probably not the way to go for what I am trying to do. what I need to figure out is how to map these derived columns to an object that I can return to the front end. how do I set up an object that hibernate can map these aliased columns to properties?
I figured it out. You can set the transient property though it's kind of a pain.
using the addScalar Method on the SQLQuery object you can set the transient properties. However if it is done this way you have to set ALL the properties in the object this way (Hibernate doesn't autofill any properties)
Also you will have to use the setResultTransformer Method also on the SQLQuery object. So to finish off my previous example I would have to add the following code in my impl.
// query.addEntity(AboutMe.class); <- this contradicts the setResultTransformer
query.setParameter("id",id);
query.addScalar("id",StandardBasicTypes.LONG);
query.addScalar("image_url",StandardBasicTypes.LONG);
query.addScalar("liked",StandardBasicTypes.LONG);
query.addScalar("numLikes",StandardBasicTypes.LONG);
query.setResultTransformer(Transformers.aliasToBean(Object.class));
where Object is the class that I am wanting to use that has all these properties declared.
Hibernate will never populate your transient fields and what is most important transient fields are not even part of the java serialization process. What I would try if you really need this - is the ResultTransformer.alliasToBean which is "injecting" the values into the class with reflection.
setResultTransformer(Transformers.aliasToBean(Yourclass.class));
Just a quick idea, not sure whether it will work.