We Keep Coding

How to clear a batch in JOOQ?

I am trying to reuse a prepared statement when executing multi-inserts. Something like
InsertValuesStepN<Record> batch = create.insertInto(table, fields);
for(int i=0; i<100000; i++) {
batch.values();
if(i % 1000 == 0) {
batch.execute();
// need to call clearBatch here so we don't insert records twice
}
}
but I don't see any way to have InsertValuesStepN clear it's records after calling execute. Is this possible?

Create a new statement for each batch:
You could create a new statement on each batch, instead of reusing the previous one.
InsertValuesStepN<Record> batch = null;
for(int i=0; i<100000; i++) {
if (batch == null)
batch = create.insertInto(table, fields);
batch.values();
if(i % 1000 == 0) {
batch.execute();
batch = null;
}
}
Using the bind() API
This is usually not recommended because of performance issue #6616, but since your bottleneck (as per your comment) is with creating a new prepared statement, you might try to use the Query.bind() API which you could use on your 2nd, 3rd, etc. batch to replace existing bind values with new ones in an existing query. Call Query.bind() like this:
// Create the initial statement with dummy values for your batch
Query batch = create.insertInto(table, fields).values(...).keepStatement(true);
for(int i=0; i<100000; i += 1000) {
// Call this once for each bind value
batch.bind(...);
batch.execute();
// Handle the last insertions, where you have less than 1000 rows per insert
// ...
}
Proxy JDBC
You could implement a proxy JDBC PreparedStatement that doesn't actually close the delegate statement when jOOQ calls PreparedStatement.close(), but keeps it open and offers the same statement again to jOOQ when jOOQ tries to prepare it again.
There's a pending feature request to offer such a PreparedStatement cache out of the box: https://github.com/jOOQ/jOOQ/issues/7327, or maybe, your JDBC driver already has one (e.g. Oracle does).
Using the import API
But perhaps, you're actually using for jOOQ's import API, which allows for specifying the batch, bulk, and commit sizes, e.g.
create
.loadInto(table)
.bulkAfter(1000)
.loadArrays(...) // There are other possible data sources
.fields(fields)
.execute();

Hibernate caches the query while data is being changed

I'm developing an application which connects to an outside service to fetch new SMS. Theses messages are stored in a local database by Hibernate. My client can search these messages based on numerous parameters such as time, number, and etc.
After calling the search method with a list of parameters called 'List1', I get the desired result without any problems. Though while I'm waiting for this result a new message has arrived.
Soon after, I call the search method with same parameter list again and I'm expecting to get the new message as well, but I get the previous result.
I have checked my database and the new message is present so all I can think of is Hibernate caching. Since both queries are exactly the same, I guess hibernate return the same result set as before.
In case my assumption is correct, how can I overcome this problem? If not, so what exactly is going on?
Edit
here is relevant part of my source code. Following two methods will be invoked when client initiates a search request:
smsService.refresh();
JSONArray result = smsService.retrieveMessages(...);
#Transactional
public JSONArray retrieveMessages(Long periodBegining, Long periodEnd, String order, Integer limit, String profile, Boolean unread, String correspondent) {
List<ShortMessage> messageList = shortMessageDAO.find(beginDate, endDate, order, limit, profile, unread, correspondent);
JSONArray result = new JSONArray();
for (ShortMessage message : messageList)
result.put(message.toJSON());
shortMessageDAO.markRead(messageList);
return result;
}
#Transactional
public void refresh() {
webService.authentication(serviceUsername, servicePassword);
while(webService.hasUnread() > 0) {
SMS sms = webService.retrieveMessage();
ShortMessage message = new ShortMessage(sms.getHash(), sms.getFrom(), sms.getTo(), "DEFAULT", sms.getMessage(), new Date(sms.getTime()), true);
shortMessageDAO.insert(message);
}
}
}
public List<ShortMessage> find(Date beginDate, Date endDate, String order, Integer limit, String profile, Boolean unread, String correspondent) {
Criteria criteria = sessionFactory.getCurrentSession().createCriteria(ShortMessage.class);
criteria.add(Restrictions.ge("time", beginDate));
criteria.add(Restrictions.le("time", endDate));
criteria.add(Restrictions.eq("profile", profile));
if (unread)
criteria.add(Restrictions.eq("unread", true));
if (correspondent != null)
criteria.add(Restrictions.eq("origin", correspondent));
criteria.addOrder(order.equals("ASC") ? Order.asc("time") : Order.desc("time"));
criteria.setMaxResults(limit);
criteria.setCacheMode(CacheMode.IGNORE);
return (ArrayList<ShortMessage>) criteria.list();
}

Yes it looks like hibernate is caching your query and returning cached results.
Please give us a overview of your code to suggest better.
Below listed are two ways of controlling the caching behaviour of queries:-
1) At the main named query level:-
#NamedQuery(
name = "myNamedQuery"
query = "SELECT u FROM USER WHERE u.items is EMPTY"
hints = {#QueryHint(name = "org.hibernate.cacheMode", value = "IGNORE")}
)
2) At individual query level :-
Query q = session.createQuery("from User")
.setCacheMode(CacheMode.IGNORE);

After running numerous test, I found out that this problem is not cache related at all. Upon receiving each message I would have stored the time of arrival based on data provided by SMS panel and not my own machine time.
There was a slight time difference (20 seconds to be exact) between those 2 which was the reason behind the query not returning the new received message.

Trying to optmize a Hibernate Query with read only

I'm trying to optmize a query that is currently taking a little longer than expected. The query returns about 11000 entities, but since they are a bit complicated and have nested entities it's somewhat slow. Since I'm not going to modify the entities, I tried setting the query/session to read-only, but it hasn't helped, it still takes just as long, maybe I'm doing something wrong. Below is a simplified code, sorry it's a little messy:
#Entity
#NamedQueries(value = {#NamedQuery(name = "demand.all", query = "select d from Demand d")})
public class Demand {
private Long ID;
private Division division;
private Client client;
private Product product;
private String code;
...
}
#Transactional(readOnly=true)
public List<Demand> getAll() {
SessionImpl sessionImpl = ((SessionImpl)em.getDelegate());
Session session = sessionImpl.getSessionFactory().openSession();
Transaction tx = session.beginTransaction();
try {
sessionImpl.connection().setReadOnly(true);
Query query = session.getNamedQuery("demand.all");
List<Demand> resultList = query.setReadOnly(true).setCacheable(false).setFlushMode(FlushMode.MANUAL).list();
sessionImpl.connection().setReadOnly(false);
tx.commit();
} catch(Exception e) {
resultList = null;
}
session.close();
return resultList;
}
I read that making the query read-only is not enough, so I tried setting the connection and transaction read-only too, but I'm not sure if it's necessary. Anyways, am I doing something wrong? What other way is there to optimize this query?

One way of doing this faster would be fetching the objects in a lazy way or depending on which are necessary and which are not. Like maybe you only need to show 5 columns in a table, instead of every single object in the hierarchy, so you create DTO to get them. If you need more information on one of them.. lets say the user clicks on a row, then you would bring the whole object hierarchy of it..
It may not apply in your case, but its one way of efficiently getting data.

Persistence strategies for High Replication environment (Google App Engine)

I have this code thtat works just fine w/o HR:
protected Entity createEntity(Key key, Map<String, Object> props){
Entity result = null;
try {
Entity e = new Entity(key);
Iterator it = props.entrySet().iterator();
while (it.hasNext()) {
Map.Entry<String, Object> entry = (Map.Entry<String, Object>) it.next();
String propName = entry.getKey();
Object propValue = entry.getValue();
setProperty(e, propName, propValue);
}
key = _ds.put(e);
if (key != null)
result = _ds.get(key);
} catch (EntityNotFoundException e1) {
}
return result;
}
This is just a simple method where its function is to create a new Entity out a a given key, just return NULL otherwise. This works fine without the HR configuration in JUnit however when I configured it, I am always getting an error, where _ds.get(key) can't find the key throwing:
EntityNotFoundException: No entity was found matching the key:
Specifically when doing:
while(it.hasNext()){
// stuff
createEntity(key, map);
// stuff
}
I assume that the problem in my code is that it tries to fetch the entity too soon. If thats is the case, how can I deal with this wihout resorting to Memcache or anything like that.
Update:
When the createEntity is executed within a transaction, it fails. However if I remove it outside of the transaction if fails miserably. I need to be able to run within a transaction, since my higher level API put lots of objects that needs to be there as a group.
Update:
I followed Strom's advise however I found a weird side effect, not doing a _ds.get(key) on the method, makes my PreparedQuery countEntities to fail. Where if add a _ds.get(key) even I don't do anything or save the Entity return from that get countEntities return the expected count. Why is that?

You try to create a new entity and then read back that entity within the same transaction? Can't be done.
Queries and gets inside transactions see a single, consistent snapshot of the datastore that lasts for the duration of the transaction. 1
In a transaction, all reads reflect the current, consistent state of the Datastore at the time the transaction started. This does not include previous puts and deletes inside the transaction. Queries and gets inside a transaction are guaranteed to see a single, consistent snapshot of the Datastore as of the beginning of the transaction. 2
This consistent snapshot view also extends to reads after writes inside transactions. Unlike with most databases, queries and gets inside a Datastore transaction do not see the results of previous writes inside that transaction. Specifically, if an entity is modified or deleted within a transaction, a query or get returns the original version of the entity as of the beginning of the transaction, or nothing if the entity did not exist then. 2
PS. Your assumption is worng, it's impossible to fetch an entity by key "too soon". Fetches by key are strongly consistent.
Also, why do you need to retrieve the entity again anyway? You just put it in the datastore yourself, so you already have its contents.
So change this part:
key = _ds.put(e);
if (key != null)
result = _ds.get(key);
To this:
key = _ds.put(e);
if (key != null)
result = e; // key.equals(e.getKey()) == true

Welcome in GAE environment, try to read it more times before you give up :
int counter = 0;
while (counter < NUMBER_OF_TRIES){
try {
//calling storage or any other non-reliable thing
if(success) {break;} //escape away if success
} catch(EntityNotFoundException e){
//log exception
counter++;
}
}
Important note from google documentation : "the rate at which you can write to the same entity group is limited to 1 write to the entity group per second."
source : https://developers.google.com/appengine/docs/java/gettingstarted/usingdatastore

Avoiding JDBC call

The scenario is like this:
for loop // runs say 200000 times
{
// here, i do a select from a database, fetching few rows which are expected to increase with every new iteration of for loop
// currently i am doing this select using simple JDBC call (using JDBC only is NOT a requirement)
// then i do some string matching stuff and then i either insert or update a particular row (in 95% cases i will insert)
// this insert or update is being done using Hibernate (using Hibernate over here is a requirement)
}
So the problem is, in every loop, I have to consider the each and every previously inserted/updated row. Due to this requirement, I have to do a JDBC call in each and every loop. And this JDBC call is taking the maximum time, bringing down the performance.
I want to know, is there any method using which I do not have to make a JDBC call in each iteration, but still I will be able to consider all the records including the one in the just previous insert/update? Anything like caching or some in-memory data structure or something like that?
Here is the code:
for loop // runs say 2000 times
{
String query = pdi.selectAllPatients(patientInfo);
Statement st = conn.createStatement();
ResultSet patientRs = st.executeQuery(query);
while (patientRs.hasNext())
{
// some string ops
}
// Create session for DB No.2
Session sessionEmpi = sessionFactoryEmpi.getCurrentSession();
sessionEmpi.beginTransaction();
if(some condition)
patientDao.insertPatient(patientInfo, sessionEmpi);
else
patientDao.insertref(patientInfo.getref(), sessionEmpi);
conn.commit();
}
public int insertPatient(PatientInfo input, Session session) throws SQLException {
try {
session.save(input.getPatient());
session.flush();
session.save(input.getref());
session.getTransaction().commit();
return 1;
} catch (Exception ex) {
session.getTransaction().rollback();
ex.printStackTrace();
return 0;
}
}

Is the performance of the SELECT consistent? Unless your data is fairly small, you'll likely have trouble caching all your changes in memory. You may also be able to batch the SELECTs, effectively unrolling the loop.

You can use the PreparedStatement interface instead of Statement interface as it avoids the unnecessary calls for firing the query to the database you just have to bind the data in for loop this will help you to improve performance!!
example:
PreparedStatement s =con.prepareStatement("select * from student_master where stu_id = ?");
for()
{
s.setString(1,"s002");
ResultSet rs = s.executeQuery();
}