I was reading a blog regarding bulk fetching with hibernate http://java.dzone.com/articles/bulk-fetching-hibernate.
In this, ScrollableResults is used as a solution. Here still we need to evict objects from session.
I don't understand how using ScrollableResults(or scroll()) is different from using list().
In other words, how the below statements differ in terms of performance
List<Employee> empList = session.createCriteria(Employee.class).list();
ScrollableResults sc = session.createCriteria(Employee.class).scroll();
Please let me know.
The above code seems to be missing some settings.
Query query = session.createQuery(query);
query.setReadOnly(true);
// MIN_VALUE gives hint to JDBC driver to stream results
query.setFetchSize(Integer.MIN_VALUE);
ScrollableResults results = query.scroll(ScrollMode.FORWARD_ONLY);
// iterate over results
while (results.next()) {
Object row = results.get();
// process row then release reference
// you may need to flush() as well
}
results.close();
Follow this link for more details and explanation.
I referred Hibernate API documentation to understand the difference between list() and scroll().
ScrollableResults is like a cursor . An important feature of ScrollableResults is that it allows accessing ith object in the current row of results, without initializing any other results in the row through get(int i) function.
It also allows moving back and forth the result set using next() and previous() function.
Example :
ScrollableResults sc = session.createQuery("select e.employeeName,e.employeeDept FROM Employee e").scroll(ScrollMode.SCROLL_INSENSITIVE);
while(sc.next()) {
String empName = (String)sc.get(0);
System.out.println(empName);
String empdept = (String)sc.get(1);
System.out.println(empdept);
}
The output of above programm will values of employeeName and employeeDept .
Now suppose you want to get the last record of the resultSet. With list() you would need to iterate the whole result. With ScrollableResults, you can use last() function.
Note that for ScrollableResults sc = session.createCriteria(Employee.class).scroll(); should be iterated as
while(sc.next()) {
Employee emp = (Employee)sc.get(0);
System.out.println(emp.getname);
}
Related
I have a view in MySQL that joins a 1-* between Schemas and Properties.
The view works fine. When I run a query in MySQL workbench I get the expected results.
However, when I run the same query in my Java program using HQL, the expected number of rows are being returned, but only the 1st row has data in it. That first row is correct, content-wise.
My code is fairly straightforward (I hope...). Stepping through in debug I can see that queryResult.list has the right # rows that I would expect to be returned for the input objectIdentifier(of the schema), but only the first row has on object in it. The rest are empty. There should be one row for each joined property. As I say, the # of rows is correct.
public List<SchemaAndPropertiesResolvedNames> getSchemaAndPropertiesResolvedNamesView(
String schemaAndPropertiesResolvedNamesId, ServiceHeaderVO context)
throws DALException {
List<SchemaAndPropertiesResolvedNames> result = new ArrayList();
try{
Session session = sessionFactory.getCurrentSession();
session.beginTransaction();
String hql = "FROM SchemaAndPropertiesResolvedNames schemaandpropertiesresolvednames WHERE schemaandpropertiesresolvednames.objectIdentifier =:objectIdentifier";
Query queryResult = session.createQuery(hql);
queryResult.setParameter("objectIdentifier", schemaAndPropertiesResolvedNamesId);
List<SchemaAndPropertiesResolvedNames> listrows = queryResult.list();
Iterator iterator = listrows.iterator();
while (iterator.hasNext()) {
SchemaAndPropertiesResolvedNames obj = (SchemaAndPropertiesResolvedNames) iterator.next();
result.add(obj);
}
session.getTransaction().commit();
return result;
It appears to be partially working to the extent that it always has the correct number of rows in the queryResult (if I try different objectIdentifiers as input). It is just they are not populated apart from the first one.
Is the content of the view causing this?
The annoying this is I could have sworn it worked earlier. Then I went on to make other changes elsewhere, but when I tested it again it had this problem.
Any suggestions? This is the first time I have used HQL.
I was going through the old code in one of our application and found the below scenario.
public <T> List<T> getList(boolean isTrue,String name,String empid)
{
Criteria criteria = session.createCriteria(myType);
criteria.add(Expression.eq("name",name)) ;
if(isTrue&&empid!=null){
criteria.add(Expression.eq("id",empid)) ;
}
List<T> result = criteria.list();
return list;
}
Also the function is called two times like below.
List<T> employeeList = getList(true,name,empId);
if(employeeList.size()==0)
employeeList = getList(false,name,null);
The reason behind this scenario is that sometimes the employeeId won't be correct. In that case we need to do a search only on employee name.
My doubts.
1) If am executing it this way am I querying the database twice ?
2) If yes then is there any way I can optimize it ?
In this case Hibernate will execute the query twice since the parameters are differents.
You can configure query cache and second level cache to improve the performance.
The query cache will return the employeeIds and the second level cache will return the Employee data.
I need to fetch the result of the following query but i am getting a typecast exception. Kindly help out!
SELECT COUNT(*) FROM ( SELECT DISTINCT a.PROPSTAT_CODE,a.PROPSTAT_DESC,a.PROPSTAT_TYPE FROM CNFGTR_PROPSTAT_MSTR a WHERE 1 = 1 )
My code is given below,
Query query = session.createSQLQuery(sqlQuery);
listRes = query.list();
int ans = ((Integer)listRes.get(0)).intValue();
Thanks in advance
Since you say that you are wrapping the above query in another query that returns the count, then this will give you want, without having to convert to any other data types.
Integer count = (Integer) session.createSQLQuery("select count(*) as num_results from (SELECT DISTINCT a.PROPSTAT_CODE,a.PROPSTAT_DESC,a.PROPSTAT_TYPE FROM CNFGTR_PROPSTAT_MSTR a WHERE 1 = 1)")
.addScalar("num_results", new IntegerType())
.uniqueResult();
System.err.println(count);
The trick is the call to "addScalar". This tells Hibernate you want the data type of "num_results" pre-converted to an Integer, regardless of what your specific DB implementation or JDBC driver prefers. Without this, Hibernate will use the type preferred by the JDBC driver, which explains why different answers here have different casts. Setting the desired result type specifically removes all guesswork about your returned data type, gives you the correct results, and has the added bonus of being more portable, should you ever wish to run your application against a different relational database. If you make the call to "list" instead of "uniqueResult" then you can assign the results directly to a List
Use long instead of int. Hibernate returns count(*) as long not int.
Query query = session.createSQLQuery(sqlQuery);
listRes = query.list();
long ans = (long)listRes.get(0);
Well.. I suppose this should work:
Query query = session.createSQLQuery(sqlQuery);
List listRes = query.list();
int ans = ((BigDecimal) listRes.get(0)).intValue();
Note: you need to import java.math.BigDecimal
List number=session.createSQLQuery("SELECT COUNT(*) FROM devicemaster WHERE ClientId="+id).list();
session.getTransaction().commit();
int ans = ((java.math.BigInteger) number.get(0)).intValue();
i am building a shopping cart using jsp and hibernate.
i am filtering the content by brand size and price using checkboxes
the checked checkboxes are returned to the class where hql query exists.
so i want i single hql query that can handle this.
as like if one of the parameter like size is empty (means user doesnt uses it to filter the content ) than an empty string is passed to the hql query which returns any value...
so is there anything possible that all values can be retrived in where clause for empty string or some other alternative except coding different methods for different parameter...
I typically use the Criteria api for things like this... if the user does not specify a size, do not add it to the criteria query.
Criteria criteria = session.createCriteria(MyClass.class);
if(size != null && !size.isEmpty()){
criteria.add(Restrictions.eq("size", size);
}
To have multiple restrictions via an OR statement, you use Disjunction. For an AND, you use Conjunction.
Criteria criteria = session.createCriteria(MyClass.class);
Disjunction sizeDisjunction = Restrictions.disjunction();
String[] sizes = { "small", "medium", "large" };
for(int i = 0; i < sizes.length; i++){
sizeDisjunction.add(Restrictions.eq("size", sizes[i]);
}
criteria.add(sizeDisjunction );
First, good practices say that instead of passing and empty String to the query, you should pass null instead. That said, this hql should help you:
from Product p
where p.brand = coalesce(:brand, p.brand)
and p.size = coalesce(:size, p.size)
and p.price = coalesce (:price, p.price)
Friends!
I am using MongoDB in java project via spring-data. I use Repository interfaces to access data in collections. For some processing I need to iterate over all elements of collection. I can use fetchAll method of repository, but it always return ArrayList.
However, it is supposed that one of collections would be large - up to 1 million records several kilobytes each at least. I suppose I should not use fetchAll in such cases, but I could not find neither convenient methods returning some iterator (which may allow collection to be fetched partially), nor convenient methods with callbacks.
I've seen only support for retrieving such collections in pages. I wonder whether it is the only way for working with such collections?
Late response, but maybe will help someone in the future. Spring data doesn't provide any API to wrap Mongo DB Cursor capabilities. It uses it within find methods, but always returns completed list of objects. Options are to use Mongo API directly or to use Spring Data Paging API, something like that:
final int pageLimit = 300;
int pageNumber = 0;
Page<T> page = repository.findAll(new PageRequest(pageNumber, pageLimit));
while (page.hasNextPage()) {
processPageContent(page.getContent());
page = repository.findAll(new PageRequest(++pageNumber, pageLimit));
}
// process last page
processPageContent(page.getContent());
UPD (!) This method is not sufficient for large sets of data (see #Shawn Bush comments) Please use Mongo API directly for such cases.
Since this question got bumped recently, this answer needs some more love!
If you use Spring Data Repository interfaces, you can declare a custom method that returns a Stream, and it will be implemented by Spring Data using cursors:
import java.util.Stream;
public interface AlarmRepository extends CrudRepository<Alarm, String> {
Stream<Alarm> findAllBy();
}
So for the large amount of data you can stream them and process the line by line without memory limitation.
See https://docs.spring.io/spring-data/mongodb/docs/current/reference/html/#mongodb.repositories.queries
you can still use mongoTemplate to access the Collection and simply use DBCursor:
DBCollection collection = mongoTemplate.getCollection("boundary");
DBCursor cursor = collection.find();
while(cursor.hasNext()){
DBObject obj = cursor.next();
Object object = obj.get("polygons");
..
...
}
Use MongoTemplate::stream() as probably the most appropriate Java wrapper to DBCursor
Another way:
do{
page = repository.findAll(new PageRequest(pageNumber, pageLimit));
pageNumber++;
}while (!page.isLastPage());
Check new method to handle results per document basis.
http://docs.spring.io/spring-data/mongodb/docs/current/api/org/springframework/data/mongodb/core/MongoTemplate.html#executeQuery-org.springframework.data.mongodb.core.query.Query-java.lang.String-org.springframework.data.mongodb.core.DocumentCallbackHandler-
You may want to try the DBCursor way like this:
DBObject query = new BasicDBObject(); //setup the query criteria
query.put("method", method);
query.put("ctime", (new BasicDBObject("$gte", bTime)).append("$lt", eTime));
logger.debug("query: {}", query);
DBObject fields = new BasicDBObject(); //only get the needed fields.
fields.put("_id", 0);
fields.put("uId", 1);
fields.put("ctime", 1);
DBCursor dbCursor = mongoTemplate.getCollection("collectionName").find(query, fields);
while (dbCursor.hasNext()){
DBObject object = dbCursor.next();
logger.debug("object: {}", object);
//do something.
}
The best way to iterator over a large collection is to use the Mongo API directly. I used the below code and it worked like a charm for my use-case.
I had to iterate over more than 15M records and the document size was huge for some of those.
The following code is in Kotlin Spring Boot App (Spring Boot Version: 2.4.5)
fun getAbcCursor(batchSize: Int, from: Long?, to: Long?): MongoCursor<Document> {
val collection = xyzMongoTemplate.getCollection("abc")
val query = Document("field1", "value1")
if (from != null) {
val fromDate = Date(from)
val toDate = if (to != null) { Date(to) } else { Date() }
query.append(
"createTime",
Document(
"\$gte", fromDate
).append(
"\$lte", toDate
)
)
}
return collection.find(query).batchSize(batchSize).iterator()
}
Then, from a service layer method, you can just keep calling MongoCursor.next() on returned cursor till MongoCursor.hasNext() returns true.
An Important Observation: Please do not miss adding batchSize on 'FindIterable' (the return type of MongoCollection.find()). If you won't provide the batch size, the cursor will fetch initial 101 records and will hang after that (it tries to fetch all the remaining records at once).
For my scenario, I used the batch size as 2000, as it gave the best results during testing. This optimized batch size will be impacted by the average size of your records.
Here is the equivalent code in Java (removing createTime from query as it is specific to my data model).
MongoCursor<Document> getAbcCursor(Int batchSize) {
MongoCollection<Document> collection = xyzMongoTemplate.getCollection("your_collection_name");
Document query = new Document("field1", "value1");// query --> {"field1": "value1"}
return collection.find(query).batchSize(batchSize).iterator();
}
This answer is based on: https://stackoverflow.com/a/22711715/5622596
That answer needs a bit of an update as PageRequest has changed how it is being constructed.
With that said here is my modified response:
int pageNumber = 1;
//Change value to whatever size you want the page to have
int pageLimit = 100;
Page<SomeClass> page;
List<SomeClass> compondList= new LinkedList<>();
do{
PageRequest pageRequest = PageRequest.of(pageNumber, pageLimit);
page = repository.findAll(pageRequest);
List<SomeClass> listFromPage = page.getContent();
//Do something with this list example below
compondList.addAll(listFromPage);
pageNumber++;
}while (!page.isLast());
//Do something with the compondList: example below
return compondList;