We Keep Coding

getting data from result set is too slow

Fetching data from PostgreSQL database with Result Set is too slow.
Here is my code.
for (int i = 0; i < qry_list.size(); i++) {
try {
resultSet = statement.executeQuery(qry_list.get(i));
resultSet.setFetchSize(0);
while (resultSet.next()) {
totalfilecrated = totalfilecrated
+ resultSet.getInt(columnname);
}
} catch (SQLException e) {
e.printStackTrace();
}
}
Here I try to fetch data inside a for loop.is it good?
Here is my query.
For getting ID of individual Organisations(org_unit_id).
"select org_unit_id from emd.emd_org_unit where org_unit_id
in(select org_unit_id from emd.emd_org_unit_detail where entity_type_id=1 and is_active=true) and
is_active=true order by org_unit_name_en";
Then i want to get the count of files with each org_unit_id
select count(*) as totalfilecreatedelectronic from fl_file ff
left join vw_employee_details_with_department epd on epd.post_detail_id=ff.file_opened_by_post_fk
where ff.file_nature = 'E' and ((ff.migration_date>='2011-01-01' and ff.migration_date<='2015-01-01') or
(ff.opening_date >='2011-01-01' and ff.opening_date <='2015-01-01')) and
epd.departmentid=org_unit_id";

Seeing how your second query already contains a reference to a column that's an org_unit_id, you might think joining emd_org_unit table in directly:
select org.org_unit_id, count(*) as totalfilecreatedelectronic
from fl_file ff
left join vw_employee_details_with_department epd on epd.post_detail_id=ff.file_opened_by_post_fk
-- join to active entries in emd_org_unit
inner join from emd.emd_org_unit org ON epd.departmentid=org.org_unit_id
AND org.is_active=true
where ff.file_nature = 'E'
and (
(ff.migration_date>='2011-01-01' and ff.migration_date<='2015-01-01') or
(ff.opening_date >='2011-01-01' and ff.opening_date <='2015-01-01'))
-- and now group by org_unit_id to get the counts
group by org_unit_id
If you'd create a SQLFiddle for this, things would get much clearer I guess.

Hibernate is 1000 times slower than sql query

I have this setup
#Table(name ="A")
EntityA {
Long ID;
List<EntityB> children;
}
#Table(name ="B")
EntityB {
Long ID;
EntityA parent;
EntityC grandchild;
}
#Table(name ="C")
EntityC {
Long ID;
}
The SQL query is this (I omitted irrelevant details):
select top 300 from A where ... and ID in (select parent from B where ... and grandchild in (select ID from C where ...)) order by ...
The sql query in direct database or through Hibernate (3.5) SQL runs 1000 faster than using Criteria or HQL to express this.
The SQL generated is identical from HQL and Criteria and the SQL I posted there.
[EDIT]: Correction - the sql was not identical. I didn't try the Hibernate style parameter setting on the management studio side because I did not realize this until later - see my answer.
If I separate out the subqueries into separate queries, then it is fast again.
I tried
removing all mappings of child, parent, ect.. and just use Long Id references - same thing, so its not a fetching, lazy,eager related.
using joins instead of subqueries, and got the same slow behaviour with all combinations of fetching and loading.
setting a projection on ID instead of retrieving entities, so there is no object conversion - still slow
I looked at Hibernate code and it is doing something astounding. It has a loop through all 300 results that end up hitting the database.
private List doQuery(
final SessionImplementor session,
final QueryParameters queryParameters,
final boolean returnProxies) throws SQLException, HibernateException {
final RowSelection selection = queryParameters.getRowSelection();
final int maxRows = hasMaxRows( selection ) ?
selection.getMaxRows().intValue() :
Integer.MAX_VALUE;
final int entitySpan = getEntityPersisters().length;
final ArrayList hydratedObjects = entitySpan == 0 ? null : new ArrayList( entitySpan * 10 );
final PreparedStatement st = prepareQueryStatement( queryParameters, false, session );
final ResultSet rs = getResultSet( st, queryParameters.hasAutoDiscoverScalarTypes(), queryParameters.isCallable(), selection, session );
// would be great to move all this below here into another method that could also be used
// from the new scrolling stuff.
//
// Would need to change the way the max-row stuff is handled (i.e. behind an interface) so
// that I could do the control breaking at the means to know when to stop
final EntityKey optionalObjectKey = getOptionalObjectKey( queryParameters, session );
final LockMode[] lockModesArray = getLockModes( queryParameters.getLockOptions() );
final boolean createSubselects = isSubselectLoadingEnabled();
final List subselectResultKeys = createSubselects ? new ArrayList() : null;
final List results = new ArrayList();
try {
handleEmptyCollections( queryParameters.getCollectionKeys(), rs, session );
EntityKey[] keys = new EntityKey[entitySpan]; //we can reuse it for each row
if ( log.isTraceEnabled() ) log.trace( "processing result set" );
int count;
for ( count = 0; count < maxRows && rs.next(); count++ ) {
if ( log.isTraceEnabled() ) log.debug("result set row: " + count);
Object result = getRowFromResultSet(
rs,
session,
queryParameters,
lockModesArray,
optionalObjectKey,
hydratedObjects,
keys,
returnProxies
);
results.add( result );
if ( createSubselects ) {
subselectResultKeys.add(keys);
keys = new EntityKey[entitySpan]; //can't reuse in this case
}
}
if ( log.isTraceEnabled() ) {
log.trace( "done processing result set (" + count + " rows)" );
}
}
finally {
session.getBatcher().closeQueryStatement( st, rs );
}
initializeEntitiesAndCollections( hydratedObjects, rs, session, queryParameters.isReadOnly( session ) );
if ( createSubselects ) createSubselects( subselectResultKeys, queryParameters, session );
return results; //getResultList(results);
}
In this code
final ResultSet rs = getResultSet( st, queryParameters.hasAutoDiscoverScalarTypes(), queryParameters.isCallable(), selection, session );
it hits the database with the full SQL, but there are no results collected anywhere.
Then it proceeds to go through this loop
for ( count = 0; count < maxRows && rs.next(); count++ ) {
Where for every one of the expected 300 results, it ends up hitting the database to get the actual result.
This seems insane, since it should already have all the results after 1 query. Hibernate logs do not show any additional SQL being issued during all that time.
Anyone have any insight? The only option I have is to go to native SQL query through Hibernate.

I finally managed to get to the bottom of this. The problem was being caused by Hibernate setting the parameters separately from the actual SQL query that involved subqueries. So native SQL or not, the performance will be slow if this is done. For example this will be slow:
String sql = some sql that has named parameter = :value
SQLQuery sqlQuery = session.createSQLQuery(sql);
sqlQuery.setParameter ("value", someValue);
List<Object[]> list = (List<Object[]>)sqlQuery.list();
And this will be fast
String sql = some native sql where parameter = 'actualValue'
SQLQuery sqlQuery = session.createSQLQuery(sql);
List<Object[]> list = (List<Object[]>)sqlQuery.list();
It seems that for some reason with letting Hibernate take care of the parameters it ends up getting stuck in the resultSet fetching. This is probably because the underlying query on the database is taking much longer being parameterized. I ended up writing the equivalent of Hibernate Criteria and Restrictions code that sets the parameters directly as above.

We noticed a similar behaviour in our system.
And also encountered that writing the query with hardcoded parameters instead of using setParameter() would fixed the issue.
We are using MS SQL Server and after further investigation we noticed the the root cause of our issue is a default configuration of the sql server driver that transmits the query parameters as unicode. This lead to our indices being ignored since they were based on the ascii values on the queried columns.
The solution was to setup this property in the jdbc url : sendStringParametersAsUnicode=false
More details can be found here.
https://stackoverflow.com/a/32867579

Get inserted row to Oracle with java

I am building a java program to insert data to my oracle database.
My problem is that I need to insert into two tables, and to reach unique rows I use in TABLE_A triggers for id before insert get next val in a sequence.
But i need the same id for the TABLE_B for connection.
( i cant get getval because what if another user uses the program... )
So I need to reach somehow that when I use executeql(sql) command in return I see what I have submit.
Now I use that I have name and date, so I select the id where name and date is the just inserted.
But its not the best because in one day I can insert more names. So now this will not unique.
like :
insert into table a ( name,date) val ( 'Ryan','2014.01.01')
id here is autoincremented by sequence
than another sql run:
inert into table_b ( id,someval) val ( select id from table_a where
name ='Ryan', date='2014.01.01, 23)
so i need something like:
system.out.println(smtp.executesql(sql).whatIinsertednow())
*than console:* '1 row insered (id,name,date) : ( 1, Ryan, 2014.01.01)

PreparedStatement prepareStatement = connection.prepareStatement("insert...",
new String[] { "your_primary_key_column_name" });
prepareStatement.executeUpdate();
ResultSet generatedKeys = prepareStatement.getGeneratedKeys();
if (null != generatedKeys && generatedKeys.next()) {
Long primaryKey = generatedKeys.getLong(1);
}
I have found the answer this is perfectly works. I can insert from JAVA and its return with the key.
Full version:
CREATE TABLE STUDENTS
(
STUDENT_ID NUMBER NOT NULL PRIMARY KEY,
NAME VARCHAR2 (50 BYTE),
EMAIL VARCHAR2 (50 BYTE),
BIRTH_DATE DATE
);
CREATE SEQUENCE STUDENT_SEQ
START WITH 0
MAXVALUE 9999999999999999999999999999
MINVALUE 0;
And the Java code
String QUERY = "INSERT INTO students "+
" VALUES (student_seq.NEXTVAL,"+
" 'Harry', 'harry#hogwarts.edu', '31-July-1980')";
// load oracle driver
Class.forName("oracle.jdbc.driver.OracleDriver");
// get database connection from connection string
Connection connection = DriverManager.getConnection(
"jdbc:oracle:thin:#localhost:1521:sample", "scott", "tiger");
// prepare statement to execute insert query
// note the 2nd argument passed to prepareStatement() method
// pass name of primary key column, in this case student_id is
// generated from sequence
PreparedStatement ps = connection.prepareStatement(QUERY,
new String[] { "student_id" });
// local variable to hold auto generated student id
Long studentId = null;
// execute the insert statement, if success get the primary key value
if (ps.executeUpdate() > 0) {
// getGeneratedKeys() returns result set of keys that were auto
// generated
// in our case student_id column
ResultSet generatedKeys = ps.getGeneratedKeys();
// if resultset has data, get the primary key value
// of last inserted record
if (null != generatedKeys && generatedKeys.next()) {
// voila! we got student id which was generated from sequence
studentId = generatedKeys.getLong(1);
}
}
source : http://viralpatel.net/blogs/oracle-java-jdbc-get-primary-key-insert-sql/

You can accomplish that by using the RETURNING clause in your INSERT statement:
INSERT INTO table_a ( name,date) val ( 'Ryan','2014.01.01') RETURNING id INTO ?

Multiple updates in one sql query

I have a table in H2 DB
Order
--------
id (key)
MarketId1
MarketId2
MarketId3
ListName1
ListName2
ListName3
From XML I'm getting list of ListOrder
public final class ListOrder
{
public long listId;
public String Name;
}
So I have 3 prepared statements
"UPDATE Order set " + ListName1 + " = ? WHERE " + MarketId1 + " = ?"
"UPDATE Order set " + ListName2 + " = ? WHERE " + MarketId2 + " = ?"
"UPDATE Order set " + ListName3 + " = ? WHERE " + MarketId3 + " = ?"
The in a method I prepare a list of PreparedStament to execute
final PreparedStatement statement1 = connection.prepareStatement(QUERY1);
final PreparedStatement statement2 = connection.prepareStatement(QUERY2);
final PreparedStatement statement3 = connection.prepareStatement(QUERY3);
for (ListOrder listOrder: listOrders)
{
statement1.setString(1, listOrder.Name);
statement1.setLong(2, listOrder.listId);
statement1.addBatch();
statement2.setString(1, listOrder.Name);
statement2.setLong(2, listName.listId);
statement2.addBatch();
statement3.setString(1, listName.Name);
statement3.setLong(2, listOrder.listId);
statement3.addBatch();
}
return new ArrayList<PreparedStatement>(){{add(statement1); add(statement2); add(statement3);}};
I'm a SQL noob. Is there any better way of doing it? I assume that MarketId 1 2 3 could be the same. ListNames could be null (there will be at least one)
UPDATE:
In code I would write something like this (prob change to HashMap)
for (ListOrder listOrder: listOrders)
{
for(Order order : orders)
{
if(order.marketID1 == listOrder.listID)
order.listName1 = listOrder.Name; //break if no dups
if(order.marketID2 == listOrder.listID)
order.listName2 = listOrder.Name;
if(order.marketID3 == listOrder.listID)
order.listName3 = listOrder.Name;
}
}

You can use update comma separated
UPDATE <TABLE>
SET COL1 = <VAL1>,
COL2= <VAL2>
WHERE <CONDITION>
Is it this what you expect as one update query?

Unless you are trying to update the same record, then there is no way to do this easily or efficiently in a single query. Otherwise, assuming this is the desired result, you could use an OR (or an AND if that is desired) statement such as:
UPDATE Order
SET ListName1=?, ListName2=?, ListName3=?
WHERE MarketId1=? OR MarketId2=? OR MarketId3=?
You might also consider updating your table to use a one:many relationship which might make your queries easier. For example:
Order
--------
id (key)
name
etc
Market_List
--------
id (key)
order_id (fk)
market
listname

JPA: How to INSERT setting PK to MAX(PK) + 1

Scenario: I came across some code that is mixing JPA with JDBC within a transaction. The JDBC is doing an INSERT into a table with basically a blank row, setting the Primary Key to (SELECT MAX(PK) + 1) and the middleName to a temp timestamp. The method is then selecting from that same table for max(PK) + that temp timestamp to check if there was a collision. If successful, it then nulls out the middleName and updates. The method returns the newly created Primary Key.
Question:
Is there a better way to insert an entity into the database, setting the PK to max(pk) + 1 and gaining access to that newly created PK (preferably using JPA)?
Environment:
Using EclipseLink and need to support several versions of both Oracle and MS SqlServer databases.
Bonus Background: The reason I'm asking this question is because I run into a java.sql.BatchUpdateException when calling this method as part of a chain when running integration tests. The upper part of the chain uses JPA EntityManager to persist some objects.
Method in question
#Override
#TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
public int generateStudentIdKey() {
final long now = System.currentTimeMillis();
int id = 0;
try {
try (final Connection connection = dataSource.getConnection()) {
if (connection.getAutoCommit()) {
connection.setAutoCommit(false);
}
try (final Statement statement = connection.createStatement()) {
// insert a row into the generator table
statement.executeUpdate(
"insert into student_demo (student_id, middle_name) " +
"select (max(student_id) + 1) as student_id, '" + now +
"' as middle_name from student_demo");
try (final ResultSet rs = statement.executeQuery(
"select max(student_id) as student_id " +
"from student_demo where middle_name = '" + now + "'")) {
if (rs.next()) {
id = rs.getInt(1);
}
}
if (id == 0) {
connection.rollback();
throw new RuntimeException("Key was not generated");
}
statement.execute("update student_demo set middle_name = null " +
"where student_id = " + id);
} catch (SQLException statementException) {
connection.rollback();
throw statementException;
}
}
} catch (SQLException exception) {
throw new RuntimeException(
"Exception thrown while trying to generate new student_ID", exception);
}
return id;
}

First off: it hurts to answer this. But I know, sometimes you have to deal with the devil :(
So technically, it's not JPA, but if you are using Hibernate as JPA-Provider, you can go with
#org.hibernate.annotations.GenericGenerator(
name = “incrementGenerator”,
strategy = “org.hibernate.id.IncrementGenerator”)
#GeneratedValue(generator="incrementGenerator")
private Long primaryKey;
The Hibernate solution is "thread-safe", but not "cluster-safe", i.e. if you run your application on several hosts, this may fail. You may catch the appropriate exception and try again.
If you stick with your solution: close the ResultSet, Statement and the Connection. Sorry, didn't catch the try-with-resources initially.

The JDBC code is pathological, makes no sense, and will not work in a multi user environment.
I would strongly recommend fixing the code to use a sequence object, or sequence table.
In JPA you can just use sequencing.
See,
http://en.wikibooks.org/wiki/Java_Persistence/Identity_and_Sequencing#Sequencing
If you really want to do your own sequencing, you can either assign the Id yourself, use PrePersist to assign your own id, or in EclipseLink implement your own Sequence subclass that does whatever you desire. You will need to register this Sequence object using a SessionCustomizer.
See,
http://wiki.eclipse.org/EclipseLink/Examples/JPA/CustomSequencing