Having strange performance issue using Hibernate 3.3.2GA behind JPA (and the rest of the Hibernate packages included in JBoss 5.)
I'm using Native Query, and assembling SQL into a prepared statement.
EntityManager em = getEntityManager(MY_DS);
final Query query = em.createNativeQuery(fullSql, entity.getClass());
The SQL has a lot of joins, but is actually very basic, with a single parameter. Like:
SELECT field1, field2, field3 FROM entity left join entity2 on... left join entity3 on
WHERE stringId like ?
and the query runs in under a second on MSSQL Studio.
If I add
query.setParameter(0, "ABC123%");
The query will pause for 9 seconds
2012-01-20 14:36:21 - TRACE: - AbstractBatcher.getPreparedStatement:(484) | preparing statement
2012-01-20 14:36:21 - TRACE: - StringType.nullSafeSet:(133) | binding 'ABC123%' to parameter: 1
2012-01-20 14:36:30 - DEBUG: - AbstractBatcher.logOpenResults:(382) | about to open ResultSet (open ResultSets: 0, globally: 0)
However, if I just replace the "?" with the value (making it not a Prepared Statement, but just a straight SQL query.
fullSql = fullSql.replace("?", "'ABC123%'");
the query will complete in less that a second.
I would really prefer to us a Prepared Statement (the input for the parameters is being extracted from user data) to prevent injection attacks.
Tracing down the slow point in the code, I arrived deep within the jtds-1.2.2 package. The offending line seems to be SharedSocket line 841 "getIn().readFully(hdrBuf);" Nothing really obvious there though...
private byte[] readPacket(byte buffer[])
throws IOException {
//
// Read rest of header
try {
getIn().readFully(hdrBuf);
} catch (EOFException e) {
throw new IOException("DB server closed connection.");
}
Arrived to through this stack...
at net.sourceforge.jtds.jdbc.SharedSocket.readPacket(SharedSocket.java:841)
at net.sourceforge.jtds.jdbc.SharedSocket.getNetPacket(SharedSocket.java:722)
at net.sourceforge.jtds.jdbc.ResponseStream.getPacket(ResponseStream.java:466)
at net.sourceforge.jtds.jdbc.ResponseStream.read(ResponseStream.java:103)
at net.sourceforge.jtds.jdbc.ResponseStream.peek(ResponseStream.java:88)
at net.sourceforge.jtds.jdbc.TdsCore.wait(TdsCore.java:3928)
at net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:1045)
at net.sourceforge.jtds.jdbc.TdsCore.microsoftPrepare(TdsCore.java:1178)
at net.sourceforge.jtds.jdbc.ConnectionJDBC2.prepareSQL(ConnectionJDBC2.java:657)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:776)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1808)
at org.hibernate.loader.Loader.doQuery(Loader.java:697)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.doList(Loader.java:2228)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125)
at org.hibernate.loader.Loader.list(Loader.java:2120)
at org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:312)
at org.hibernate.impl.SessionImpl.listCustomQuery(SessionImpl.java:1722)
at org.hibernate.impl.AbstractSessionImpl.list(AbstractSessionImpl.java:165)
at org.hibernate.impl.SQLQueryImpl.list(SQLQueryImpl.java:175)
at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:67)
I'll leave this question and answer out here in case anyone has the same issue in the future.
The issue is in the way the JTDS drivers send the parameter strings to MSSQL. Apparently Java will attempt to send the parameters Unicode by default, and MSSQL will translate it to Ascii. Why that takes 9 seconds, I do not know.
Lot's of references to this out there, but nothing that helped my till I was able to isolate that it was an issue with the driver to MSSQL connection.
This link was helpful:
[http://server.pramati.com/blog/2010/06/02/perfissues-jdbcdrivers-mssqlserver/]
This is the string using the Microsoft driver.
jdbc:sqlserver://localhost\SQLEXPRESS;
DatabaseName=TESTDB;
sendStringParametersAsUnicode=false
You just need to get the sendStringParametersAsUnicode=false passed to your driver URL setup and you are good.
Check the query plans that SQL server is producing. Prepared statements can be especially problematic.
Let me explain...
If you do this:
SELECT field1, field2, field3 FROM entity left join entity2 on... left join entity3 on
WHERE stringId like 'ABC123%';
and you have an index on "stringId" SQL server knows it can use it.
However if you do this:
SELECT field1, field2, field3 FROM entity left join entity2 on... left join entity3 on
WHERE stringId like ?;
SQL server doesn't know it can use the index when it creates the prepared statement (as you could fill in the parameter with '%ABC123' instead of 'ABC123%') and thus may choose a completely different query plan.
And another answer for people potentially using Oracle with a similar Unicode problem...
Check to make sure someone hasn't set the property oracle.jdbc.defaultNChar=true
This is sometimes done to resolve unicode problems but it means all columns are treated as nvarchars. If you have an index on a varchar column, it won't be used because oracle has to use a function to convert the character encoding.
Related
Before I explain my problem I would like to say that I know the basics of JDBC but not really used to it.
I am using an updatable result set to hold data from 2 different tables, as in the following sample code:
searchQry = "SELECT ct.CustomerName, ct.Email, ct.PhoneNo, ot.ItemName
FROM CUSTOMER_TABLE ct JOIN ORDER_Table ot
ON ct.OrderID = ot.OrderID";
prestmt = dbcon.prepareStatement(searchQry, ResultSet.TYPE_SCROLL_SENSITIVE, ResultSet.CONCUR_UPDATABLE);
uprs = prestmt.executeQuery();
uprs.updateLong("ut.PhoneNo", 7240987456L);
uprs.updateString("otItemName", "GTA5");
uprs.updateRow();
I would like to know if I will update the database from somewhere else (not using the same result set object) while the result set, upsr, connected to the database, whether uprs will get updated with it or it will throw an error or it will go with the old data itself. Sorry if it a newbie question but I can't really test that on my DB without knowing the outcomes and safe measures.
Please, suggest me if there is any better way to update the underlining db along with the data in the ResultSet without having any transaction issues when changing from different places.
Using:
Oracle Database for JDBC connection.
This is something that I've been scratching my head with - especially since it's infuriating to deal with.
Consider the following code:
String query = "UPDATE ORDERS SET VOLUME=?,CONTRACT_ID=?,PROJECT_ID=?,WORKSITE_ID=?,DROPZONE_ID=?,DESCRIPTION_ID=?,MANAGER_ID=?,DELIVERY_DATE=?,REVISION=REVISION+1) WHERE ID=?";
jdbcTemplate.update(query, orderEntity.getVolume(), orderEntity.getContractNo(), orderEntity.getProjectID(), orderEntity.getWorksiteID(), orderEntity.getDropzoneID(), orderEntity.getDescriptionID(), orderEntity.getManagerID(), orderEntity.getDeliveryDate(), id);
We can see that the SQL query is incorrect - and will therefore throw some SQL error but one might have missed that. Spring will (for me) throw a QueryTimeoutException in response to this. I'm sort of okay with that but it's not helpful.
Now let's try
String query = "INSERT INTO ORDERS(ID,REISION,CONTRACT_ID,PROJECT_ID,WORKSITE_ID,DROPZONE_ID,DESCRIPTION_ID,MANAGER_ID,VOLUME,DELIVERY_DATE) VALUES(?,?,?,?,?,?,?,?,?,?)";
jdbcTemplate.update(query, id, revision, etc);
Another spelling mistake that's easily missed - REVISION is misspelled as REISION) Spring throws another QueryTimeoutException again. This now means that if I get that exception I don't actually know what it is. Is it a syntax error? Is it a column spelling error? Is it the (much harder to notice) fact that the foreign key constraint not being followed?
While debugging, this is quite possibly the most infuriating thing ever - all I know is that my query failed to run. How can I get something useful? Is there something I've not added to my pom.xml file?
EDIT:
Here's a nicer example. I have a DESCRIPTIONS table, with an ID, REVISION and TEXT column. All of those are marked as not being nullable.
DescriptionEntity descriptionEntity = new DescriptionEntity("newDesc", 1, null);
String query = "INSERT INTO DESCRIPTIONS (ID,REVISION,TEXT) VALUES(?,?,?)";
jdbcTemplate.update(query, descriptionEntity.getID(), 1, descriptionEntity.getText());
That will also throw a query timeout exception, when running the query in mysql gives me ERROR 1048 (23000): Column 'TEXT' cannot be null
This is, to put it politely, a bit of a pain.
It's not a spelling mistake in the first example, as you left out the opening paren. I would say this isn't an issue with Spring or JDBC, but rather your DB is trying to process the SQL, waiting for more input or something, and never returning.
In the second one, I am not sure what you are talking about since I don't know the table design. I have to assume what you mean is ID is not unique? Again, I wouldn't blame Spring or JDBC, maybe the drive, most likely the database server.
Keep in mind, in a lot of cases, the way SQL is handled in the user Client UI is not the same as how it gets handled through JDBC. For instance, in SQL Server the SQL is set as a string, the passed in parameters set as variables, and it uses sp_executesql() to run it. I discovered that when I had a report that ran PERFECTLY fine through SQL Studio Manager client, but blew up when I ran it live because the query plan optimizer took a different path due to the differences in how the SQL was ran.
This is quite possibly the most stupid error I've ever come across: the issue was in how Maven resolved all the dependencies.
The requirement for Spring Security was placed before the JDBC requirement. That made Spring Security pull down org.springframework:spring-tx:jar:3.0.7.RELEASE:compile which satisfied the dependency for JDBC. Moving the JDBC requirement up meant JDBC pulled down org.springframework:spring-tx:jar:3.2.2.RELEASE:compile.
I have asked this question and wanted to edit it , however StackOverflow for some reason did not allow me to edit . So here is the edited version
For example a query :
create volatile table testTable as (select * from ... blah blah) ;
select top 10 * from testTable ;
drop table testTable ;
It executes perfect in sql assistance as one session. I am sure it is possible to execute it in Java in one session.
Goal : need to execute it in one session similar to sql assistant so that it is possible to refer to the volatile table in the subsequent select statement. Also the data from the select statement should be saved in the ResultSet
PS
I saw one answer to a similar question about mysql. The trick is to turn on allow multiple queries
String dbUrl = "jdbc:mysql:///test?allowMultiQueries=true";
For teradata specifically,
what is the solution ?
I tried
String dbUrl = "jdbc:odbc:dsn?allowMultiQueries=true";
What is exactly failing?
Is there an error message "testtable doesn't exist"? Then your program closes the connection after each request.
Is the table empty when you do the SELECT? Then you forgot to add ON COMMIT PRESERVE ROWS to the CREATE.
I want to run a native SQL from a file using Hibernate. The SQL can contain several statements creating the database structure (i.e. tables, constraints but no insert/update/delete statements).
Example, very simple query is below (which contains the following two SQL statements)
CREATE DATABASE test;
CREATE TABLE test.testtbl( id int(5));
I am using MySQL db, and when I run the above query I am gettng syntax error returned. When I run them one by one, its ok.
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
You have an error in your SQL syntax; check the manual that corresponds to your
MySQL server version for the right syntax to use near
'CREATE TABLE test.testtbl( id int(5))' at line 1
The code to run the query is below (above statement is assigned to 'sql' variable):
session = sf.openSession();
session.beginTransaction();
Query qry = session.createSQLQuery(sql);
qry.executeUpdate();
session.getTransaction().commit();
Any help would be appreciated.
As others have explained
You must run these queries one by one.
The hibernate code gets translated into running one update statement on JDBC.
But you provided two update statements.
In addition,
I personally prefer to have the code that creates tables outside of the Java application, in some DB scripts.
The parameters of the method createSQLQuery is t-sql code;
t-sql code to ensure that in the mysql interface analyzer correctly.
You can try changed the sql :'CREATE TABLE testtbl(id int(5));'
by the way you can use JDBC Connection api (Don't recommend to do so)
Such as:
java.sql.Connection conn=session.connection();
How is it possible?
We are executing EJBQL on Toplink(DB is Oracle) and query.getResultList is empty.
But!
When i switched log level to FINE and received Sql query, that TopLink generates, i tried to execute this query on database and (miracle!) i got a non-empty result!
What could be the reason and how is it treated?
Thanks in advance!
P.S. No exceptions.
UPDATE:
Query log:
SELECT DISTINCT t0.ID, t0.REG_NUM, t0.REG_DATE, t0.OBJ_NAME, t1.CAD_NUM, t1.CAD_NUM_EGRO, t2.ID, t2.DICT_TYPE, t2.ARCHIVE_DATE, t2.IS_DEFAULT, t2.IS_ACTUAL, t2.NAME, t0.INVENTORY_NUM FROM CODE_NAME_TREE_DICTIONARY t3, DEFAULTABLE_DICTIONARY t2, IMMOVABLE_PROP t1, ABSTRACT_PROPERTY t0 WHERE ((t3.ID IN (SELECT DISTINCT t4.ID FROM CODE_NAME_TREE_DICTIONARY t5, CODE_NAME_TREE_DICTIONARY t4, type_property_parents t6 WHERE (((t5.ID = ?) AND (t4.DICT_TYPE = ?)) AND ((t6.type_property_id = t4.ID) AND (t5.ID = t6.parent_id)))) AND ((t1.ID = t0.ID) AND (t0.PROP_TYPE_DISCR = ?))) AND ((t3.ID = t0.PROP_TYPE) AND ((t2.ID (+) = t1.STATUS_ID) AND (t2.DICT_TYPE = ?)))) ORDER BY t0.REG_NUM ASC
bind => [4537, R, R, realty_status]|#]
This query returns 100k rows, but toplink believes that it is not...
With log level to FINE can you verify that you are connecting to the same database? How simple is your testcase; can you verify that it is this exact JPQL that is being translated to that SQL?
VPD (http://download.oracle.com/docs/cd/B28359_01/network.111/b28531/vpd.htm)? Policies?
Is something of this flavor defined on the schema? These features transparently add dynamic where clauses to the statement that is executed in the database session, so the query results depend on the state of the session in this case.
When reformatting the query the following conditions seemed strange:
AND t2.ID (+) = t1.STATUS_ID
AND t2.DICT_TYPE = ?
The (+) indicates an outer join of t2 (DEFAULTABLE_DICTIONARY), but this table seems to be non-optional since it has to have a non-null DICT_TYPE for the second condition.
On closer looking, the bind parameters also seem to be off, the fields are in order
CODE_NAME_TREE_DICTIONARY.ID
CODE_NAME_TREE_DICTIONARY.DICT_TYPE
ABSTRACT_PROPERTY.PROP_TYPE_DISCR
DEFAULTABLE_DICTIONARY.DICT_TYPE
With the given parameters (4537, R, R, realty_status), the first DICT_TYPE would be 'R' while the second is the string "realty_status" which seems inconsistent.
Transactions? Oracle never gives you a "dirty read" which database speak for access to uncommitted data. If you send data on one connection you cannot access it on any other connection until it is committed. If you try the query later by hand, the data has been committed and you get the expected result.
This situation can arise if you are updating the data in more than one connection, and the data manipulation is not set to "auto commit". JPA defaults to auto-commit, but flushing at transaction boundaries can give you a cleaner design.
I can't tell exactly, but I am a little surprised that the string parameters are not quoted. Is it possible that interactively there are some automatic conversions, but over this connection instead of the string 'R' it was converted to the INT ascii for R?
I found the reason!
The reason is Oracle! I've tried the same code on Postgres and its worked!
I dont know why, but in some magic cases oracle ignores query parameters and query returns empty result.