I am using this StringBuilder in order to add content in a query:
Integer lastEntryInEntityId = 1;//acquired through another query
Integer tmpValueForEntityId;
Integer lastEntryInEntity2Id = 1;//acquired through another query
StringBuilder queryString = new StringBuilder("insert
into entity(column,column_1,column_2,column_3) values");
StringBuilder queryString2 = new StringBuilder("insert
into entity2(column,column_1,column_2,column_3) values");
for(Object[] entityToCopy : entitiesToCopy){
Entity entity= (Entity )entityToCopy[0];
tmpValueForEntityId= lastEntryInEntityId ;
queryString.append("("+ lastEntryInEntityId ++ +","+entity.getProperty()+","+entity[1]+","+entity.getProperty2()+"),");
for(Entity2 entity2 : entity.getEntity2Collection()){
queryString2.append("("+lastEntryInEntity2Id ++ +","+tmpValueForEntityId+","+entity.getProperty2()+","+entity.getProperty3()+"),");
}
}
This code takes both too much time and memory. It actually throws an OutOfMemoryException on adding to the second StringBuilder after some time (when entitiesToCopy are too many).
How else can I write this code in order to make it faster and use less memory?
NOTE: A java 8 solution would be preferred.
NOTE 2: I use EntityManager.
You should use concat() instead + inside StringBuilder
for(Object[] entityToCopy : entitiesToCopy){
Entity entity= (Entity )entityToCopy[0];
tmpValueForEntityId= lastEntryInEntityId ;
queryString.append("(").append(lastEntryInEntityId++).append(",").append(entity.getProperty()).append(",").append(entity[1]).append(",").append(entity.getProperty2()).append("),");
for(Entity2 entity2 : entity.getEntity2Collection()){
queryString2.append("(").append(lastEntryInEntity2Id ++).append(",").append(tmpValueForEntityId).append(",").append(entity.getProperty2()).append(",").append(entity.getProperty3()).append("),");
}
}
For better performance, use PreparedStatement in transaction:
dbCon.setAutoCommit(false);
var pst = dbCon.prepareStatement("insert into entity (columnID, column_1, column_2, column_3) values (?, ?, ?, ?)";
for(Object[] entityToCopy : entitiesToCopy){
var entity = (Entity )entityToCopy[0];
tmpValueForEntityId = lastEntryInEntityId;
pst.setInt(1, lastEntryInEntityId);
pst.setString(2, entity.getProperty());
pst.setString(3, entity[1]);
pst.setString(4, entity.getProperty2());
pst.addBatch();
}
pst.executeBatch();
dbCon.commit();
dbCon.setAutoCommit(true);
Each ? represents a column. The first one represents the ID, the second one represents column_1, etc. Keep the order of each one.
Note: If you are using Java prior to 1.10, change var to PreparedStatement
With concurrent connections (more than one thread insert into database):
Don't close the database connection after commit (close on program exit)
The method that inserts data should be synchronized
Don't use prepareStatement(), instead use createStatement() with Pattern (regex) to avoid SQLinjections.
Note: PreparedStatement is good, fast and secure.
The database keep a pool of prepared statements to avoid create new every time. But in concurrence, after one thread have a reference to existing statement -> PreparedStatement, another thread can use it and the transactioin is slow (waits for new instance or new reference to existing). In concurrence this happens many, many times.
EntityManager example:
var em = emf.createEntityManager();
EntityTransaction transaction = null;
try {
transaction = em.getTransaction();
transaction.begin();
for(Object[] entityToCopy : entitiesToCopy){
var entity = (Entity )entityToCopy[0];
...//insert here
}
tx.commit();
} catch (RuntimeException e) {
if (transaction != null && transaction.isActive()) {
tx.rollback();
e.printStackTrace();
}
} finally {
em.close();
}
I executed a query every x iterations so that the query doesn't get too big. This solved my problem.
int count = 0;
for(Object[] entityToCopy : entitiesToCopy){
Entity entity= (Entity )entityToCopy[0];
tmpValueForEntityId= lastEntryInEntityId ;
queryString.append("("+ lastEntryInEntityId ++
+","+entity.getProperty()+","+entity[1]+","+entity.getProperty2()+"),");
for(Entity2 entity2 : entity.getEntity2Collection()){
queryString2.append("("+lastEntryInEntity2Id ++
+","+tmpValueForEntityId+","+entity.getProperty2()+","+entity.getProperty3()+"),");
}
count++;
if(count%2000 == 0 || entitiesToCopy.size() == count){
em.executeQuery(queryString);
queryString = "";
em.executeQuery(queryString2);
queryString2 = "";
}
}
Related
I have this setup
#Table(name ="A")
EntityA {
Long ID;
List<EntityB> children;
}
#Table(name ="B")
EntityB {
Long ID;
EntityA parent;
EntityC grandchild;
}
#Table(name ="C")
EntityC {
Long ID;
}
The SQL query is this (I omitted irrelevant details):
select top 300 from A where ... and ID in (select parent from B where ... and grandchild in (select ID from C where ...)) order by ...
The sql query in direct database or through Hibernate (3.5) SQL runs 1000 faster than using Criteria or HQL to express this.
The SQL generated is identical from HQL and Criteria and the SQL I posted there.
[EDIT]: Correction - the sql was not identical. I didn't try the Hibernate style parameter setting on the management studio side because I did not realize this until later - see my answer.
If I separate out the subqueries into separate queries, then it is fast again.
I tried
removing all mappings of child, parent, ect.. and just use Long Id references - same thing, so its not a fetching, lazy,eager related.
using joins instead of subqueries, and got the same slow behaviour with all combinations of fetching and loading.
setting a projection on ID instead of retrieving entities, so there is no object conversion - still slow
I looked at Hibernate code and it is doing something astounding. It has a loop through all 300 results that end up hitting the database.
private List doQuery(
final SessionImplementor session,
final QueryParameters queryParameters,
final boolean returnProxies) throws SQLException, HibernateException {
final RowSelection selection = queryParameters.getRowSelection();
final int maxRows = hasMaxRows( selection ) ?
selection.getMaxRows().intValue() :
Integer.MAX_VALUE;
final int entitySpan = getEntityPersisters().length;
final ArrayList hydratedObjects = entitySpan == 0 ? null : new ArrayList( entitySpan * 10 );
final PreparedStatement st = prepareQueryStatement( queryParameters, false, session );
final ResultSet rs = getResultSet( st, queryParameters.hasAutoDiscoverScalarTypes(), queryParameters.isCallable(), selection, session );
// would be great to move all this below here into another method that could also be used
// from the new scrolling stuff.
//
// Would need to change the way the max-row stuff is handled (i.e. behind an interface) so
// that I could do the control breaking at the means to know when to stop
final EntityKey optionalObjectKey = getOptionalObjectKey( queryParameters, session );
final LockMode[] lockModesArray = getLockModes( queryParameters.getLockOptions() );
final boolean createSubselects = isSubselectLoadingEnabled();
final List subselectResultKeys = createSubselects ? new ArrayList() : null;
final List results = new ArrayList();
try {
handleEmptyCollections( queryParameters.getCollectionKeys(), rs, session );
EntityKey[] keys = new EntityKey[entitySpan]; //we can reuse it for each row
if ( log.isTraceEnabled() ) log.trace( "processing result set" );
int count;
for ( count = 0; count < maxRows && rs.next(); count++ ) {
if ( log.isTraceEnabled() ) log.debug("result set row: " + count);
Object result = getRowFromResultSet(
rs,
session,
queryParameters,
lockModesArray,
optionalObjectKey,
hydratedObjects,
keys,
returnProxies
);
results.add( result );
if ( createSubselects ) {
subselectResultKeys.add(keys);
keys = new EntityKey[entitySpan]; //can't reuse in this case
}
}
if ( log.isTraceEnabled() ) {
log.trace( "done processing result set (" + count + " rows)" );
}
}
finally {
session.getBatcher().closeQueryStatement( st, rs );
}
initializeEntitiesAndCollections( hydratedObjects, rs, session, queryParameters.isReadOnly( session ) );
if ( createSubselects ) createSubselects( subselectResultKeys, queryParameters, session );
return results; //getResultList(results);
}
In this code
final ResultSet rs = getResultSet( st, queryParameters.hasAutoDiscoverScalarTypes(), queryParameters.isCallable(), selection, session );
it hits the database with the full SQL, but there are no results collected anywhere.
Then it proceeds to go through this loop
for ( count = 0; count < maxRows && rs.next(); count++ ) {
Where for every one of the expected 300 results, it ends up hitting the database to get the actual result.
This seems insane, since it should already have all the results after 1 query. Hibernate logs do not show any additional SQL being issued during all that time.
Anyone have any insight? The only option I have is to go to native SQL query through Hibernate.
I finally managed to get to the bottom of this. The problem was being caused by Hibernate setting the parameters separately from the actual SQL query that involved subqueries. So native SQL or not, the performance will be slow if this is done. For example this will be slow:
String sql = some sql that has named parameter = :value
SQLQuery sqlQuery = session.createSQLQuery(sql);
sqlQuery.setParameter ("value", someValue);
List<Object[]> list = (List<Object[]>)sqlQuery.list();
And this will be fast
String sql = some native sql where parameter = 'actualValue'
SQLQuery sqlQuery = session.createSQLQuery(sql);
List<Object[]> list = (List<Object[]>)sqlQuery.list();
It seems that for some reason with letting Hibernate take care of the parameters it ends up getting stuck in the resultSet fetching. This is probably because the underlying query on the database is taking much longer being parameterized. I ended up writing the equivalent of Hibernate Criteria and Restrictions code that sets the parameters directly as above.
We noticed a similar behaviour in our system.
And also encountered that writing the query with hardcoded parameters instead of using setParameter() would fixed the issue.
We are using MS SQL Server and after further investigation we noticed the the root cause of our issue is a default configuration of the sql server driver that transmits the query parameters as unicode. This lead to our indices being ignored since they were based on the ascii values on the queried columns.
The solution was to setup this property in the jdbc url : sendStringParametersAsUnicode=false
More details can be found here.
https://stackoverflow.com/a/32867579
Is it possible to execute an update while using Criteria in Hibernate? For example:
Session session = getSession();
Criteria crit = session.createCriteria(User.class);
crit.add(Restrictions.eq("token", sessionToken));
User user= new User();
Transaction tx = session.getTransaction();
try
{
tx.begin();
session.updateWithCriteria(user, crit); //my imaginary function
tx.commit();
}
catch (Exception e)
{
e.printStackTrace();
tx.rollback();
}
session.close();
There is a very powerful feature called:
15.4. DML-style operations
small cite from doc:
... However, Hibernate provides methods for bulk SQL-style DML statement execution that is performed through the Hibernate Query Language...
So, while this is not about criteria - we still can use our domain model for querying, because it is about HQL. This is a snippet showing the power:
Session session = sessionFactory.openSession();
Transaction tx = session.beginTransaction();
String hqlUpdate = "update Customer c set c.name = :newName where c.name = :oldName";
// or String hqlUpdate = "update Customer set name = :newName where name = :oldName";
int updatedEntities = s.createQuery( hqlUpdate )
.setString( "newName", newName )
.setString( "oldName", oldName )
.executeUpdate();
tx.commit();
session.close();
SUMMARY: Having that in place:
we can use query to filter results
we can apply bulk update on it
we won't need to load these rows in memory, into the session...
Now we can do something like this for bulk update and delete. New api's released for criteriaUpdate and CriteriaDelete
CriteriaBuilder cb = this.em.getCriteriaBuilder();
// create update
CriteriaUpdate<Order> update = cb.createCriteriaUpdate(Order.class);
// set the root class
Root e = update.from(Order.class);
// set update and where clause
update.set("amount", newAmount);
update.where(cb.greaterThanOrEqualTo(e.get("amount"), oldAmount));
// perform update
this.em.createQuery(update).executeUpdate();
First you should get the object then modify and update:
Query q = session.createQuery("from StockTransaction where tranId = :tranId ");
q.setParameter("tranId", 11);
StockTransaction stockTran = (StockTransaction)q.list().get(0);
stockTran.setVolume(4000000L);
session.update(stockTran);
If you want to use partial/dynamic update feature then put
#org.hibernate.annotations.Entity(
dynamicUpdate = true
)
annotation on top of the dao class.
Example from: http://www.mkyong.com/hibernate/hibernate-dynamic-update-attribute-example/
Note: The Question is "with criteria" but the accepted answer is NOT "with criteria" but SQL.
I've got a requirment where I need to apply a search keyword to name and and state of Parent entity (Customer) and also name attribute of child entity Order.
List<Criterion> aliasRestrictionsCriterion = new ArrayList<Criterion>();
Junction quotedLikeRestrictions = Restrictions.disjunction();
quotedLikeRestrictions.add(Restrictions.ilike("customerName", token, MatchMode.ANYWHERE));
quotedLikeRestrictions.add(Restrictions.ilike("customerState", token, MatchMode.ANYWHERE));
restrictions.add(quotedLikeRestrictions);
and to include the child table in the query I'm am doing the following
Criteria alias = criteria.createAlias("order", "order")
.add(Restrictions.ilike("order.name", token, MatchMode.ANYWHERE));
Hibernate is generating the following query but what I need is to have order.name like comparison get added to the parent block.
CUSTOMER this_ inner join ORDER order_ on this_.ORDER_ID=order_.ORDER_ID where
(lcase(this_.name) like ? or lcase(this_.state) like ?) and (lcase(order_.NAME)
like ?)
what I want is to have (lcase(order_.NAME) like ?) inside the parent block. So I want is the following
(lcase(this_.name) like ? or lcase(this_.state) like ? or lcase(order_.NAME) like ?)
How can I achieve this ?
Update: this is how I'm calling the criteria
protected List getEntities(List<Criterion> restrictions, String aliasSearchToken) {
Session session = currentSession();
Criteria criteria = session.createCriteria(getEntityClass());
// apply restrictions.
if (restrictions != null) {
for (Criterion criterion : restrictions) {
criteria.add(criterion);
}
}
Criteria alias = criteria.createAlias("order", "order").add(Restrictions.ilike("order.name", searchToken, MatchMode.ANYWHERE));
List list = criteria.list();
if (list != null) {
return list;
}
return Collections.emptyList();
}
Update 2:
protected List getEntities(List<Criterion> restrictions, String aliasSearchToken) {
Session session = currentSession();
Criteria criteria = session.createCriteria(getEntityClass());
Junction or = Restrictions.disjunction();
or.add(Restrictions.ilike("order.name", searchToken, MatchMode.ANYWHERE));
criteria.createAlias("order", "order").add(or);
// apply restrictions.
if (restrictions != null) {
for (Criterion criterion : restrictions) {
criteria.add(criterion);
}
}
List list = criteria.list();
if (list != null) {
return list;
}
return Collections.emptyList();
}
produced
select ....... from CUSTOMER this_ inner join ORDER order_ on this_.ORDER_ID=order_.uid where (lcase(order_.NAME) like ?) and (lcase(this_.name) like ? or lcase(this_.state) like ?)
the orderName is just before but still not in the second or statement ... I'm so confused.
Well, instea of adding the restriction to the Criteria (which creates an AND), you must add it to the quotedLikeRestrictions disjunction:
quotedLikeRestrictions.add(Restrictions.ilike("order.name", token, MatchMode.ANYWHERE));
I have 1M rows in a table and I want to get all of them. But when I try to get all rows with jpa by pagination then I get java heap error. Do you think that am I missing something? Any advice
int counter = 0;
while (counter >= 0) {
javax.persistence.EntityManager em = javax.persistence.Persistence
.createEntityManagerFactory("MyPU")
.createEntityManager();
Query query = em.createQuery("select m from mytable m");
java.util.Collection<MyEntity> data = query
.setFirstResult(counter).setMaxResults(1000).getResultList();
for(MyEntity yobj : data){
System.out.println(obj);
}
counter += 1000;
data.clear();
em.clear();
em.close();
}
Since you use native SQL anyway, can't you specify the LIMIT :counter, 1000 (or ROWNUM BETWEEN :counter AND 1000 if using Oracle) directly in your SQL statement?
Note that you create new instance of EntityManagerFactory at each iteration, but don't close it. Perhaps it would be better to use a single factory instead:
int counter = 0;
EntityManagerFactory emf = javax.persistence.Persistence.createEntityManagerFactory("MyPU");
while (counter >= 0) {
javax.persistence.EntityManager em = emf.createEntityManager();
...
}
I am running an aggregate function in java through hibernate and for some reason it is giving me this error:
INFO Binary:182 - could not read column value from result set: l_date; Column 'l_date' not found.
When I run the MySQL query the column names are l_date and logins and I can not figure out why it is not finding that.
I have tested the query in MySQL and verified that it does work. My function looks as follows.
public List<Logins> getUserLoginsByMonth() {
Session session = getSessionFactory().openSession();
ArrayList<Logins> loginList = null;
try {
String SQL_QUERY = "SELECT l_date as l_month, SUM(logins) as logins FROM (SELECT DATE_FORMAT(login_time, '%M') as l_date, COUNT(DISTINCT users) as logins FROM user_logins WHERE login_time > DATE_SUB(NOW(), INTERVAL 1 YEAR) GROUP BY DATE(login_time)) AS Z GROUP BY(l_month)";
Query query = session.createSQLQuery(SQL_QUERY);
List results = query.list();
for(ListIterator iter = results.listIterator(); iter.hasNext() ) {
Objects[] row = (Object[])iter.next();
System.out.println((Date)row[0}]);
System.out.println((BigInteger)row[1]);
}
}
catch(HibernateException e) {
throw new PersistenceDaoException(e);
}
finally {
session.close();
}
}
You need to add aliases as:
query.addScalar("l_month");
query.addScalar("logins");
and then call query.list();
It should work fine. addScalar is part of SqlQuery.