We Keep Coding

Liquibase Slowness When Creating Indexes

I recently upgraded my Java Liquibase version from 3.5.3 to 3.6.3
I have a very heavy environment where there are lots of databases and tables (I am using Oracle).
On this environment, I am trying to execute a huge changelog file where I create tables and indices.
Find below a small part of the changelog.
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.2.xsd">
...
...
...
<changeSet author="me" id="tableCreation78">
<preConditions onFail="MARK_RAN">
<not>
<tableExists tableName="MY_TABLE_NAME" />
</not>
</preConditions>
<comment>Creating table MY_TABLE_NAME</comment>
<createTable tableName="MY_TABLE_NAME">
<column name="M_ID" type="bigint">
<constraints nullable="false" primaryKey="true" primaryKeyName="PK_MY_TABLE_NAME_190" />
</column>
<column name="M_FORMAT" type="int" />
</createTable>
</changeSet>
...
...
...
<changeSet author="me" id="indexCreation121">
<preConditions onFail="MARK_RAN">
<tableExists tableName="MY_TABLE_NAME"/>
<not>
<indexExists tableName="MY_TABLE_NAME" columnNames="M_FEEDER_ID"/>
</not>
</preConditions>
<comment>Creating index for MY_TABLE_NAME</comment>
<createIndex tableName="MY_TABLE_NAME" indexName="MY_INDEX_NAME">
<column name="M_ID_INDEX"/>
</createIndex>
</changeSet>
...
...
...
</databaseChangeLog>
On the Liquibase 3.5.3, creating the index used to be quick.
When I migrated to Liquibase 3.6.3, I had a severe regression in performance.
What used to run in 1-2 minutes, now takes up to 20 minutes to complete.
The changelog does not define Unique Constraints.
While debugging, I noticed one of the many differences between the two versions. In the 3.5.3, the listConstraints and listColumns methods from UniqueConstraintSnapshotGenerator are not called.
In the 3.6.3 version, these methods are called a lot, even though no unique constraints are defined in the changelog. I am guessing that they are here from the previously defined tables of the environment.
Some of these queries (see below) are called multiple times with the exact same parameters. I don't know if it's a maintenance step that was added in the 3.6.3.
2020-08-13 17:03:52,270 INFO [main] select ucc.owner as constraint_container, ucc.constraint_name as constraint_name, ucc.column_name, f.validated as constraint_validate from all_cons_columns ucc INNER JOIN all_constraints f ON ucc.owner = f.owner AND ucc.constraint_name = f.constraint_name where ucc.constraint_name='UC' and ucc.owner='DB' and ucc.table_name not like 'BIN$%' order by ucc.position
I am not sure if this is the cause of the regression but honestly, I am out of ideas.
Does anybody know if this might be the cause of this regression?
Did they add new maintenance steps in Liquibase 3.6.3 that might be causing this big performance degradation?
Thank you so much!

You may need to perform maintenance on your Oracle data dictionary. Databases that use Liquibase tend to drop and create more objects than the average Oracle database, which can cause performance problems with metadata queries.
First, gather optimizer statistics for fixed objects (V$ objects) and the data dictionary (ALL_ objects). This information helps Oracle build good execution plans for metadata queries. The below statement will take a few minutes but may only need to be run once a year:
begin
dbms_stats.gather_fixed_objects_stats;
dbms_stats.gather_dictionary_stats;
end;
/
Another somewhat-common reason for data dictionary query problems is a large number of objects in the recycle bin. The recycle bin is great on production systems, where it lets you instantly recover from dropping the wrong table. But on a development environment, if thousands of objects are constantly dropped but not purged, those old objects can slow down some metadata queries.
--Count the number of objects in the recycle bin.
select count(*) from dba_recyclebin;
--Purge all of them if you don't need them. Must be run as SYS.
purge dba_recyclebin;
Those are two quick and painless solutions to some data dictionary problems. If that doesn't help, you may need to tune specific SQL statements, which may require a lot of information. For example - exactly how long does it take your system to run that query against ALL_CONS_COLUMNS? (On my database, it runs in much less than a second.)
Run Liquibase and then use a query like the one below to find the slowest metadata queries:
select elapsed_time/1000000 seconds, executions, sql_id, sql_fulltext, gv$sql.*
from gv$sql
order by elapsed_time desc;

Oracle->MSSQL: How to set identifier generator?

I'm migrating OracleDB11g to MSSQL2014.
Currently getting following error when trying to save new data:
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Cannot
insert the value NULL into column 'ID', table 'testDB.FILE_SETTINGS';
column does not allow nulls. INSERT fails.
My interpretation is that this is caused as the "native" ID generator differs between Oracle & MSSQL (Sequence vs. Identity).
In Oracle we had small customization to HIBERNATE_SEQUENCE:
alter sequence hibernate_sequence increment by 5;
...but thats all.
hibernate-mapping is originally like this:
<id name="id" column="ID" type="java.lang.Long">
<generator class="native">
</generator>
</id>
In MSSQL I have tried it like this with no luck:
<id name="id" column="ID" type="java.lang.Long">
<generator class="sequence">
<param name="sequence">HIBERNATE_SEQUENCE</param>
</generator>
</id>
And in MSSQL server I have sequence (migration tool created):
HIBERNATE_SEQUENCE in testDB->Views->sys.sequences
Also found in (migration tool created):
testDB->Views->INFORMATION_SCHEMA.SEQUENCES
How should this be done properly as I want to retain the same way to generate identity as was in Oracle? Something wrong in MSSQL or in hibernate settings?
Hibernate version is quite old: 2.1.8

Your interpretation of the error is wrong. This error tells you nothing about sequence. It tells you that your testDB.FILE_SETTINGS has id column defined as NOT NULL but you try to insert NULL value here.
I don't see your code but I think there is something like this:
create table dbo.MyTbl_wrong (id int NOT NULL, col1 varchar(100) );
insert into dbo.MyTbl_wrong(col1) values ('1 str'), ('2 str'), ('3 str');
Cannot insert the value NULL into column 'id', table
'db2.dbo.MyTbl_wrong'; column does not allow nulls. INSERT fails. The
statement has been terminated.
What you should do instead is to use sequence in the default for your id column like this:
create sequence dbo.MySeq
start with 1;
create table dbo.MyTbl (id int NOT NULL default(next value for dbo.MySeq), col1 varchar(100) );
insert into dbo.MyTbl(col1) values ('1 str'), ('2 str'), ('3 str');
--select *
--from dbo.MyTbl;
-------
--id col1
--1 1 str
--2 2 str
--3 3 str

Answering own question if some day someone has similar issues:
Apparently there was nothing wrong with the settings I mentioned in original question, only that hibernate 2.1.8 seemed not to support the SEQUENCE id genererator. Also old hibernate obviously did not support the needed SQLServer2012Dialect.
Ended up updating the hibernate version to 4.3.11 (with its dependencies). This version got selected as it required the least amount of refactoring.
I had some issues with the dependencies as this old project was not using Maven.
Also faced this error as any DB query was attempted:
org.hibernate.LazyInitializationException: could not initialize proxy - no Session
I found out that new hibernate version was defaulted to use lazy loading which was not supported in the old version. So I ended up fixing it with setting "lazy=false" in hibernate setting files.
You may see "org.hibernate.LazyInitializationException: could not initialize proxy - no Session" while upgrading from hibernate 2.1 to hibernate 3.0. You will suddenly find yourself puzzling what happened, it was working before update. Reasons is, Hibernate 3 introduced lazy loading as the default i.e. lazy="true". If you want it to work the same as before you can mark everything as lazy="false". Alternatively you'll have to start eagerly initialising your entities and associations.
Read more: http://javarevisited.blogspot.com/2014/04/orghibernatelazyinitializationException-Could-not-initialize-proxy-no-session-hibernate-java.html#ixzz56ahBmiBl

Java/Hibernate + HSQLDB java.sql.BatchUpdateException: data exception: string data, right truncation

I am testing an application locally using a memory-based HSQLDB.
So far everything went fine, however, when executing a testcase with a String bigger than 256 chars I ran into an error.
Caused by: java.sql.BatchUpdateException: data exception: string data, right truncation; table: TABLENAME column: COLNAME
at org.hsqldb.jdbc.JDBCPreparedStatement.executeBatch(Unknown Source)
at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268)
... 30 more
I gathered that the cause of this error is usually an "overflow" of the datatype one is using.
What bothers me is that I explicitly defined the column to be 4000 chars big, using the hbm.xml files.
<property name="translation" type="java.lang.String" length="4000">
<column name="COLNAME" not-null="false" />
</property>
When I cut the test string down to 256 chars or less everything starts working again. 257+ chars and the error is thrown. I don't really see the reason why this is happening. Why would HSQLDB define this column as length="256", when I explicitly state that it's supposed to be 4000...
Can anyone help?
Best regards,
daZza

Well, for whatever reason HSQLDB seems to do a bad mapping for certain types and values specified in the config file to the real tables it creates.
I was able to fix the issue by changing the "type" attribute to type=text. Now everything works perfectly. I just hope that after I finish testing the application it still works with the original MSSQL DB and maps text to a varchar then...

Hibernate mapped property to SQL formula fails after update to 3.6.5

I have just updated to Hibernate 3.6.5.Final from 3.3.0.GA and have run into a problem with a SQL formula call on an XML mapped property:
<property
name="endDate"
type="java.util.Date"
formula="TIMESTAMPADD(SECOND, (quantity*60*60), transactionDate)"
/>
I have changed nothing in the *.xml.hbm nor have I changed the database design. Where previously my endDate was nicely calculated I now get a MySQLSyntaxErrorException:
You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'this_.SECOND,(this_.quantity*60*60),this_.transactionDate) as formula0_0_ from t' at line 1
The problem is pretty obvious in that the this_.SECOND should be SECOND. It seems to me that Hibernate recognizes the TIMESTAMPADD as a formula but not the SECOND as a static passed parameter and thinks it thus must be a column in the table. I am unsure how to tell hibernate it should use SECOND as is.
I've tried registerFunction and registerKeyword on my Dialect but without any luck as these seem related to HQL function definitions and not native SQL which is used here in the formula.
Could anyone point me in the right direction or tell me what Hibernate does different between these versions and how I can fix it?

I just upgraded to Hibernate 4.1.2 and this same problem started coming back. The solution of [SECOND] no longer works and I had to register the keyword in my own custom Dialect. Like:
public class ExtendedMySQL5InnoDBDialect extends MySQL5InnoDBDialect {
public ExtendedMySQL5InnoDBDialect() {
super();
//make sure to register it in lowercase as uppercase does not work (took me 4 hours to realize)
registerKeyword("second");
}
}

I had the same type of problem in Sql Server - a similar solution might work. .
Here is what I found. .
https://forum.hibernate.org/viewtopic.php?p=2427791
So, try putting quotes around SECOND
<property
name="endDate"
type="java.util.Date"
formula="TIMESTAMPADD("SECOND", (quantity*60*60), transactionDate)"
/>
I am actually not sure how you would escape double quotes in this xml attribute, but I would try " first and if that doesn't work \"

Basic HQL statement always fails

I've recently started trying to use Hibernate, but am doing so in Netbeans. This has left me having to use this example project to try and get me up and running.
Unfortunately, at the step "Enumerating Film Titles and Retrieving Actors Using an HQL Query" my HQL queries do not give results and instead fail, with the exception:
org.hibernate.exception.SQLGrammarException: could not execute query
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'from limit 100' at line 1
It seems that the HQL isn't generating proper MySQL statements but I can't seem to see why, as I've followed the example to the letter thus far.
I am attempting to connect to a local MySQL database named 'sakila', with the following details:
jdbc:mysql://localhost:3306/sakila
which seems to work correctly as I am able to browse the tables from within Netbeans no problem.
My hibernate.cfg.xml is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-configuration PUBLIC "-//Hibernate/Hibernate Configuration DTD 3.0//EN" "http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="hibernate.dialect">org.hibernate.dialect.MySQLInnoDBDialect</property>
<property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.connection.url">jdbc:mysql://localhost:3306/sakila</property>
<property name="hibernate.connection.username">root</property>
<property name="hibernate.show_sql">true</property>
<property name="transaction.factory_class">org.hibernate.transaction.JDBCTransactionFactory</property>
<property name="hibernate.current_session_context_class">thread</property>
<property name="hibernate.query.factory_class">org.hibernate.hql.classic.ClassicQueryTranslatorFactory</property>
<mapping resource="dvdrental/Language.hbm.xml"/>
<mapping resource="dvdrental/FilmActor.hbm.xml"/>
<mapping resource="dvdrental/FilmCategory.hbm.xml"/>
<mapping resource="dvdrental/Category.hbm.xml"/>
<mapping resource="dvdrental/Film.hbm.xml"/>
<mapping resource="dvdrental/Actor.hbm.xml"/>
</session-factory>
</hibernate-configuration>
When I'm using the HQL Query window, the SQL it seems to generate only ever says 'select from ' which is obviously wrong, but I can't see why this is being caused?
Edit 1: The HQL Query tab showing my input

Completely stupid reason- I hadn't even thought about the fact I hadn't built the project, so the example code from the bottom of the example ran properly (cause it came pre-compiled). Works no problem now.

I think the main cause of the error is as stated earlier from User, the U in User should be uppercase/capitalized as hibernate uses the POJO. I tried recompiling the code and it works correctly for the sample Swing application: https://netbeans.org/kb/docs/java/hibernate-java-se.html.
However the HQL Query editor still gives this error. It is strange but seems like this editor is not reliable.

The error message 'from limit 100' from your posted exception tells me, that you forgot to add Film after from. Please provide the HQL query that throws the exception.
BTW: the examples in the example project are bad practice. You should never use inline arguments in HQL queries:
from Film as film where film.filmId between '"+startID+"' and '"+endID+"'
You should use query parameters:
Query q = session.createQuery("from Film as film where film.filmId between :startID and :endID");
q.setParameter("startID", startID);
q.setParameter("endID", endID);

When I'm using the HQL Query window, the SQL it seems to generate only ever says 'select from ' which is obviously wrong, but I can't see why this is being caused?
A simple
from Film
is valid HQL and should work. Could you try without the following line:
<property name="hibernate.query.factory_class">org.hibernate.hql.classic.ClassicQueryTranslatorFactory</property>
Not sure this will solve the problem but there is no reason to use the "old" parser anyway.
That replaces the problem with a new one I'm afraid. Now the SQL tab of the HQL pane reports 'Invalid Query', and if I run the query I get org.hibernate.hql.ast.QuerySyntaxException. Any further pointers?
As I said, I was not sure this would solve the problem. What I would do is write a "simple" JUnit test to validate the setup is working. But an obvious bug like this in NetBeans HQL Query window is unlikely, IMO.

Completely stupid reason- I hadn't even thought about the fact I hadn't built the project, so the example code from the bottom of the example ran properly (cause it came pre-compiled). Works no problem now.
I experienced exactly the same problem but I think your stated solution is not correct. Even your project is not compiled HQL will show you results.
One should be careful of Table name in hibernate.cfg.xml file
as it should be same in query (case sensitive) also for example in your case it has table name as "Actor" so
from Actor will work, not from actor