I am dealing with an issue when I attempt to retrieve a large amount of records from a database. It seems that when the amount of records exceed 90.000, the elements can not be retrieved.
When that happens I get the following exception:
com.sun.jdi.ObjectCollectedException occurred while retrieving value.
The code that I am using is the following one:
Session objSession;
List<GroupEntity> colResults;
objSession = this.objSessionFactory.openSession();
try
{
objQuery = objSession.createQuery("FROM GroupEntity WHERE (strDomain = :Domain)")
.setParameter("Domain", strDomain)
.list();
}
catch (Exception objException)
{
throw new GroupException("Could not retrieve the list of WebFiltering groups to scan");
}
objSession.close();
return colResults;
I attempt to page the results retrieved by sets of 1.000, using this method when I insert up to 89.999 records the list is fine. however when I exceed 90.000 I get the same exception.
Any idea about how to face this issue?
In case you process such a big amount of data I'd recommend that you use batch processing with ScrollableResults: https://grokonez.com/hibernate/resolve-hibernate-outofmemoryerror-problem-hibernate-batch-processing
Session session = factory.openSession();
Transaction tx = null;
try {
tx = session.beginTransaction();
ScrollableResults dataCursor = session.createQuery("FROM Data").scroll();
int count = 1;
while (dataCursor.next()) {
Data data = (Data) dataCursor.get(0);
String newText = Utilities.generatedRandomString();
data.setText(newText);
session.update(data);
if (count % 50 == 0) {
System.out.println("============================log: count = " + count);
session.flush();
session.clear();
}
count++;
}
tx.commit();
} catch (Exception e) {
if (null != tx) {
tx.rollback();
}
} finally {
session.close();
}
In this case session will not keep all 90000 records in memory.
"com.sun.jdi.ObjectCollectedException" happens when the object you referring to is garbage collected.
there is no such limit of size on java arrayList.
Related
I get the below error when I ran the below code. May I know what is wrong in my code. Thanks in advance . I am using hibernate and java
HTTP Status 500 - Handler processing failed; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded
#Autowired
SessionFactory sessionFactory;
Session session = null;
Transaction tx = null;
public String getEntityList(String userIds, String callerID) throws Exception {
session = sessionFactory.openSession();
tx = session.beginTransaction();
List<User> userList = session.createCriteria(User.class)
.list();
//Query the database
CallableStatement stmt = null;
String returnVal = "";
ResultSet rs = null;
try {
stmt = ((Connection) session).prepareCall("{?=call WS_Distributionlist(?,?)}");
} catch (SQLException e) {
e.printStackTrace();
}
try {
// left over accounts processing
stmt.executeUpdate();
returnVal += stmt.getString(1) + "|";
System.out.println("RETURN VALUE in tail end :::::: "
+ returnVal);
//returnVal = returnVal.substring(0, returnVal.length() - 1);
System.out.println("Return Value " + returnVal);
session.close();
//return returnVal;
} catch (SQLException e) {
System.out.println("Error while executing the database function ");
} finally {
session.close();
}
tx.commit();
session.close();
return returnVal;
}
A lot of useful details are missing:
userIds is an input parameter of the function where is it used?
userList, how many users are there and where are they used?
WS_Distributionlist(?,?) has 2 input parameters, where do you set them?
I cannot understand the following line of code:
((Connection) session).prepareCall("{?=call WS_Distributionlist(?,?)}");
I've seen and used many technique to get a connection out of an hibernate session, none of which where casting the session itself to connection, but this is more or less a personal curiosity.
The point is that there is a lot of missing code, so I can just speculate that one of this calls produces an huge dataset, or that you are passing some huge parameter to a statement.
Either way, if you deal with HUGE dataset you should probably use a StatelessSession , because stateless session have nearly zero memory overhead (no caching).
If you build your own statement from connection , use FORWARD_ONLY_CURSOR.
Try setting the fetchSize on the statements that returns resultset.
Consider using either a ScrollableResult or an Iterator instead of a List as a result.
I made One Class for executing hibernate select operations my code is
working fine but i just need some help
I am passing hibernate select query from some other class to get the
result if my select query contains more than one column than I call the
method getListbylimit(String query,int limit) its returns
List but when my select query column contains only one than it
gives exception java.lang.String cannot be cast to
[Ljava.lang.Object;
for that I made second method List
getListForSingleColumn(String query) to get the result for single
column
is there any way to write method for this so that I can call only
one method. Rather my select query contain one column or more than one columns.
can I get return type List<Object[]> if I select only one column instead of List<String> so that I can use only one method for select operation
Here is my code
public class ContentDomain {
Session session;
public List<Object[]> getListbylimit(String query,int limit){
session = HibernateUtil.getSessionFactory().getCurrentSession();
/* session = HibernateUtil.getSessionFactory().openSession();
*/
List<Object[]> ls_ob = new ArrayList<Object[]>();
Transaction tx = null;
try {
tx = session.beginTransaction();
Query q = session.createQuery(query);
q.setMaxResults(limit);
ls_ob = (List<Object[]>)q.list();
}catch (HibernateException ex) {
if (tx != null) {
System.out.println("Exception in getList method " + ex);
tx.rollback();
ex.printStackTrace();
}
System.out.println("Exception getList tx open" + ex);
} finally {
session.close();
}
return ls_ob;
}
public List<String> getListForSingleColumn(String query){
session = HibernateUtil.getSessionFactory().getCurrentSession();
/* session = HibernateUtil.getSessionFactory().openSession();*/
List<String> ls_ob = new ArrayList<String>();
Transaction tx = null;
try {
tx = session.beginTransaction();
Query q = session.createQuery(query);
ls_ob = q.list();
}catch (HibernateException ex) {
if (tx != null) {
System.out.println("Exception in getList method " + ex);
tx.rollback();
ex.printStackTrace();
}
System.out.println("Exception getList tx open" + ex);
} finally {
session.close();
}
return ls_ob;
}
}
If you use native queries (session.createSQLQuery("yournnativequery)) you'll always get a List of Object[] with .list()... then it's easy to do a single private parsing method to handle both outputs.
You can replace ls_ob = (List<Object[]>)q.list(); with:
List<Object> result = q.list();
if (!result.isEmpty() && !(result.get(0) instanceof Object[])) {
ls_ob = result.stream().map(o -> new Object[] {o}).collect(Collectors.toList());
} else {
ls_osb = (List) result;
}
Basically, transform the single column query results manually.
I have a requirement where I am updating rows according to uuid's where one uuid may be associated to more than one rows.
Here's the scenario:
4a90558c-4a5b-4af7-8c68-60ff81f74ef3 is my uuid and it exists in 8 columns in my DB.
and my java code is as follows:
try{
session = sessionFactory.getCurrentSession();
tx = session.getTransaction();
criteria = session.createCriteria(Archive.class);
criteria.add(Restrictions.eq("bagUuid", "4a90558c-4a5b-4af7-8c68-60ff81f74ef3"));
ScrollableResults items = criteria.scroll();
while ( items.next() ) {
Archive archive = (Archive)items.get(0);
archive.setDecision(1);
archive.setOperatorAssigned("test");
session.saveOrUpdate(archive);
session.flush();
session.clear();
}
tx.commit();
LOGGER.info("Archive Record is updated: "+archive.getFilePath());
}catch(Exception e){
recordUpdated = false;
tx.rollback();
LOGGER.error("Archive Record failed to update due to exception in updateArchiveRecord method: "+e.getMessage());
}
Here sometimes all records associated to UUID is updating but sometimes failing.
I think it may be a issue with the Hibernate API.
Does anybody else has faced the same issue.
This line looks suspicious:
Archive archive = (Archive)items.get(0);
It means thart regardless of the number of items in the ScrollableResults, you will be updating only the first Archive object. If I understood what you are trying to do correctly, it should be a current record:
Archive archive = (Archive)items.get();
Also, I'd move out/delete session.flush() and session.clear(). The final code would look like this:
try{
session = sessionFactory.getCurrentSession();
tx = session.getTransaction();
criteria = session.createCriteria(Archive.class);
criteria.add(Restrictions.eq("bagUuid", "4a90558c-4a5b-4af7-8c68-60ff81f74ef3"));
ScrollableResults items = criteria.scroll();
while ( items.next() ) {
Archive archive = (Archive)items.get();
archive.setDecision(1);
archive.setOperatorAssigned("test");
session.saveOrUpdate(archive);
}
tx.commit();
session.close();
LOGGER.info("Archive Record is updated: "+archive.getFilePath());
}catch(Exception e){
recordUpdated = false;
tx.rollback();
LOGGER.error("Archive Record failed to update due to exception in updateArchiveRecord method: "+e.getMessage());
}
Slava Imeshev
Why session.createCriteria(classtype).list() return more object than in list?
Returned list is contains repeating objects in random order.
public Collection getAll() {
List list = null;
Session session = null;
Transaction tx = null;
try {
session = HibernateUtil.getSessionFactory().openSession();
tx = session.beginTransaction();
list = session.createCriteria(getClassType()).list();
tx.commit();
} catch (HibernateException ex) {
if (tx != null) {
tx.rollback();
}
LOGGER.error("HibernateException in getAll");
} finally {
if (session != null && session.isOpen()) {
session.close();
}
}
return list;
}
I'm assuming your session.createCriteria(classtype).list() call is returning some of the objects of this class multiple times.
This can occur when you have a OneToMany or ManyToMany relation that is eagerly fetched.
One way to solve this, as JB Nizet correctly points out, is to use the Criteria.DISTINCT_ROOT_ENTITY ResultTransformer.
This, however, will do the work at the 'java side': all objects will be fetched from the database and then all duplicates are removed.
It would be much better to make the OneToMany or ManyToMany lazy (which is the default) instead of eager.
It's probably because the loaded entity has a toMany association that is eagerly fetched using a join. Use a distinct root entity result transformer to only get each root entity once in the list:
criteria.setResultTransformer(DistinctRootEntityResultTransformer.INSTANCE);
or return a Set rather than a List if the order is not important.
Thanks for help, i used it, and resolve problem this way:
...
try {
session = HibernateUtil.getSessionFactory().openSession();
tx = session.beginTransaction();
Criteria criteria = session.createCriteria(getClassType())
.setProjection(Projections.id())
.setFirstResult(getStart())
.setMaxResults(getLength());
HashSet<Long> ids = new HashSet( criteria.list() );
criteria = session.createCriteria(getClassType())
.add(Restrictions.in(ID_COLUMN_NAME, ids))
TreeSet<Employee> items = new TreeSet( criteria.list() );
list = new ArrayList<Employee>(items);
tx.commit();
} catch (HibernateException ex) {
...
Why is this giving me a lock timeout:
for(int i = 0; i < playersCount ; i++) {
StatUser stats = (StatUser) selectedUsers.get(i).getStatUsers().iterator().next();
int gc = stats.getGamesPlayed();
int gm = stats.getMakeCount();
Session session = HibernateUtil.getSessionFactory().getCurrentSession();
Transaction tx = session.beginTransaction();
if(i == winnerIndex) {
stats.setGamesPlayed(gc++);
stats.setMakeCount(gm++);
session.update(stats);
session.flush();
tx.commit();
customTextBean.sendMail(selectedUsers.get(i).getEmail(), "Tillykke du har vundet");
}
else {
stats.setGamesPlayed(gc++);
session.update(stats);
session.flush();
tx.commit();
customTextBean.sendMail(selectedUsers.get(i).getEmail(), "Tillykke " + winnersName + " skal lave kaffe");
}
}
If you create a new transaction (session.beginTransaction();), then a new DB connection is created. So you have a transaction which has a read-lock on stats (from the for loop) and inside of that you try to write to the same row -> Deadlock.
To fix that, you first fetch all StatUsers with a second loop, close the first transaction and then iterate over the result in the code above. If you can't do that because you run out of memory, then Hibernate is no longer your friend and you must use custom SQL.
Other solutions: Use optimistic locking or read the data to be changed with custom SQL and instantiate the objects in the loop.