We Keep Coding

Liquibase Slowness When Creating Indexes

I recently upgraded my Java Liquibase version from 3.5.3 to 3.6.3
I have a very heavy environment where there are lots of databases and tables (I am using Oracle).
On this environment, I am trying to execute a huge changelog file where I create tables and indices.
Find below a small part of the changelog.
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.2.xsd">
...
...
...
<changeSet author="me" id="tableCreation78">
<preConditions onFail="MARK_RAN">
<not>
<tableExists tableName="MY_TABLE_NAME" />
</not>
</preConditions>
<comment>Creating table MY_TABLE_NAME</comment>
<createTable tableName="MY_TABLE_NAME">
<column name="M_ID" type="bigint">
<constraints nullable="false" primaryKey="true" primaryKeyName="PK_MY_TABLE_NAME_190" />
</column>
<column name="M_FORMAT" type="int" />
</createTable>
</changeSet>
...
...
...
<changeSet author="me" id="indexCreation121">
<preConditions onFail="MARK_RAN">
<tableExists tableName="MY_TABLE_NAME"/>
<not>
<indexExists tableName="MY_TABLE_NAME" columnNames="M_FEEDER_ID"/>
</not>
</preConditions>
<comment>Creating index for MY_TABLE_NAME</comment>
<createIndex tableName="MY_TABLE_NAME" indexName="MY_INDEX_NAME">
<column name="M_ID_INDEX"/>
</createIndex>
</changeSet>
...
...
...
</databaseChangeLog>
On the Liquibase 3.5.3, creating the index used to be quick.
When I migrated to Liquibase 3.6.3, I had a severe regression in performance.
What used to run in 1-2 minutes, now takes up to 20 minutes to complete.
The changelog does not define Unique Constraints.
While debugging, I noticed one of the many differences between the two versions. In the 3.5.3, the listConstraints and listColumns methods from UniqueConstraintSnapshotGenerator are not called.
In the 3.6.3 version, these methods are called a lot, even though no unique constraints are defined in the changelog. I am guessing that they are here from the previously defined tables of the environment.
Some of these queries (see below) are called multiple times with the exact same parameters. I don't know if it's a maintenance step that was added in the 3.6.3.
2020-08-13 17:03:52,270 INFO [main] select ucc.owner as constraint_container, ucc.constraint_name as constraint_name, ucc.column_name, f.validated as constraint_validate from all_cons_columns ucc INNER JOIN all_constraints f ON ucc.owner = f.owner AND ucc.constraint_name = f.constraint_name where ucc.constraint_name='UC' and ucc.owner='DB' and ucc.table_name not like 'BIN$%' order by ucc.position
I am not sure if this is the cause of the regression but honestly, I am out of ideas.
Does anybody know if this might be the cause of this regression?
Did they add new maintenance steps in Liquibase 3.6.3 that might be causing this big performance degradation?
Thank you so much!

You may need to perform maintenance on your Oracle data dictionary. Databases that use Liquibase tend to drop and create more objects than the average Oracle database, which can cause performance problems with metadata queries.
First, gather optimizer statistics for fixed objects (V$ objects) and the data dictionary (ALL_ objects). This information helps Oracle build good execution plans for metadata queries. The below statement will take a few minutes but may only need to be run once a year:
begin
dbms_stats.gather_fixed_objects_stats;
dbms_stats.gather_dictionary_stats;
end;
/
Another somewhat-common reason for data dictionary query problems is a large number of objects in the recycle bin. The recycle bin is great on production systems, where it lets you instantly recover from dropping the wrong table. But on a development environment, if thousands of objects are constantly dropped but not purged, those old objects can slow down some metadata queries.
--Count the number of objects in the recycle bin.
select count(*) from dba_recyclebin;
--Purge all of them if you don't need them. Must be run as SYS.
purge dba_recyclebin;
Those are two quick and painless solutions to some data dictionary problems. If that doesn't help, you may need to tune specific SQL statements, which may require a lot of information. For example - exactly how long does it take your system to run that query against ALL_CONS_COLUMNS? (On my database, it runs in much less than a second.)
Run Liquibase and then use a query like the one below to find the slowest metadata queries:
select elapsed_time/1000000 seconds, executions, sql_id, sql_fulltext, gv$sql.*
from gv$sql
order by elapsed_time desc;

Oracle->MSSQL: How to set identifier generator?

I'm migrating OracleDB11g to MSSQL2014.
Currently getting following error when trying to save new data:
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Cannot
insert the value NULL into column 'ID', table 'testDB.FILE_SETTINGS';
column does not allow nulls. INSERT fails.
My interpretation is that this is caused as the "native" ID generator differs between Oracle & MSSQL (Sequence vs. Identity).
In Oracle we had small customization to HIBERNATE_SEQUENCE:
alter sequence hibernate_sequence increment by 5;
...but thats all.
hibernate-mapping is originally like this:
<id name="id" column="ID" type="java.lang.Long">
<generator class="native">
</generator>
</id>
In MSSQL I have tried it like this with no luck:
<id name="id" column="ID" type="java.lang.Long">
<generator class="sequence">
<param name="sequence">HIBERNATE_SEQUENCE</param>
</generator>
</id>
And in MSSQL server I have sequence (migration tool created):
HIBERNATE_SEQUENCE in testDB->Views->sys.sequences
Also found in (migration tool created):
testDB->Views->INFORMATION_SCHEMA.SEQUENCES
How should this be done properly as I want to retain the same way to generate identity as was in Oracle? Something wrong in MSSQL or in hibernate settings?
Hibernate version is quite old: 2.1.8

Your interpretation of the error is wrong. This error tells you nothing about sequence. It tells you that your testDB.FILE_SETTINGS has id column defined as NOT NULL but you try to insert NULL value here.
I don't see your code but I think there is something like this:
create table dbo.MyTbl_wrong (id int NOT NULL, col1 varchar(100) );
insert into dbo.MyTbl_wrong(col1) values ('1 str'), ('2 str'), ('3 str');
Cannot insert the value NULL into column 'id', table
'db2.dbo.MyTbl_wrong'; column does not allow nulls. INSERT fails. The
statement has been terminated.
What you should do instead is to use sequence in the default for your id column like this:
create sequence dbo.MySeq
start with 1;
create table dbo.MyTbl (id int NOT NULL default(next value for dbo.MySeq), col1 varchar(100) );
insert into dbo.MyTbl(col1) values ('1 str'), ('2 str'), ('3 str');
--select *
--from dbo.MyTbl;
-------
--id col1
--1 1 str
--2 2 str
--3 3 str

Answering own question if some day someone has similar issues:
Apparently there was nothing wrong with the settings I mentioned in original question, only that hibernate 2.1.8 seemed not to support the SEQUENCE id genererator. Also old hibernate obviously did not support the needed SQLServer2012Dialect.
Ended up updating the hibernate version to 4.3.11 (with its dependencies). This version got selected as it required the least amount of refactoring.
I had some issues with the dependencies as this old project was not using Maven.
Also faced this error as any DB query was attempted:
org.hibernate.LazyInitializationException: could not initialize proxy - no Session
I found out that new hibernate version was defaulted to use lazy loading which was not supported in the old version. So I ended up fixing it with setting "lazy=false" in hibernate setting files.
You may see "org.hibernate.LazyInitializationException: could not initialize proxy - no Session" while upgrading from hibernate 2.1 to hibernate 3.0. You will suddenly find yourself puzzling what happened, it was working before update. Reasons is, Hibernate 3 introduced lazy loading as the default i.e. lazy="true". If you want it to work the same as before you can mark everything as lazy="false". Alternatively you'll have to start eagerly initialising your entities and associations.
Read more: http://javarevisited.blogspot.com/2014/04/orghibernatelazyinitializationException-Could-not-initialize-proxy-no-session-hibernate-java.html#ixzz56ahBmiBl

Liquibase always generating changesets for creating/dropping index

I am just starting to use Liquibase and I am wondering: why is it that when I run ./mvnw compile liquibase:diff are change sets generated to first drop existing indexes and then recreate them if they already exist?
Ex:
<changeSet author="me (generated)" id="1486157347995-13">
<dropIndex indexName="my_idx" tableName="notification"/>
<createIndex indexName="my_idx" tableName="notification">
<column name="index_col"/>
</createIndex>
</changeSet>

Probably out of "laziness".
This is a simple way to make sure the index created is the same (not only the name, but the columns used) than the one in the reference database.
It handles two diff cases in one:
missing index name in the target db,
same index name but with a different definition.

Hibernate slqjdbc4 column with spaces

I need to map in Java with hibernate a table in sqlserver that has spaces in the column names.
I use Sqlserver 2008 jre7 com.microsoft.sqlserver.sqljdbc4 with Hibernate 3.3.2
I found a lot of similar questions in different forums but none of the solutions I read worked.
I tried using the backtick
<property name="nameSpaces" type="string" >
<column name="`Name with spaces`" not-null="false" />
</property>
this is the error
com.microsoft.sqlserver.jdbc.SQLServerException: Incorrect syntax near '`'.
I also tried to use the [] brakets but hibernate converted it in ` again.
<column name="[Name with spaces]" not-null="false" />
com.microsoft.sqlserver.jdbc.SQLServerException: Incorrect syntax near '`'.
Finally I also tried both of them
<column name="`[Name with spaces]`" not-null="false" />
or
without success.
Can someone help me? Is there any hibernate configuration I can change to work with these column names.
Thanks

The answer (by BilalDja) is hidden above in a comment. I found his answer helpful so I thought I would add the answer here for others with the same problem.
This issue is hard to debug, I had to dig really deep into the hibernate and jdbc code to finally find the "real" SQL statement that is created to get data from the database. I then copied its value and ran it manually on my SQL Server database. Straight away the syntax error was obvious and sure enough it was cause by a space in a column name.
To fix this I just escaped the double quotes around the problematic column name as explained in BilalDja's comments and expanded slightly here...
#Column(name = "\"Column name\"")
private String variableColumnWillMapTo;
That's it, problem solved, no more syntax error.

hibernate - strange error on foreign key and primary key when updating

I receive a strange error that I can't reproduce in eclipse but appears on server.
Says:
Caused by: java.sql.BatchUpdateException: Batch entry 0
update public.user set
is_at='[B#136ef65',
user_name='elk',
password='33e79218965eb72c92a549dd5a330113',
location_precision=NULL,
location_simulated=NULL,
active=NULL,
last_seen_at='2013-02-20 08:13:45.992000 +01:00:00',
app_uuid=NULL,
avatar_url='http://server.org/img/avatars/10.png',
email='mail#mail.com',
score=NULL,
phrase='hibernate fail',
state=NULL,
reputation=NULL,
reputation_desc=NULL,
privacy_settings='visibleForAll',
gender='male',
email_verified='1',
geom='SRID=4326;POINT(2.209740161895752 41.400447845458984)'
where user_id='[B#19ee400'
was aborted. Call getNextException to see the cause.
Following by
java.lang.IllegalArgumentException: id to load is required for loading
It fails because (hibernate?) fails to convert the attribute is_at and user_id to strings. It seems to try to parametrize the query with sort of a binary representation on the object.
Relevant hbm.xml snippet:
<id name="userId" type="java.lang.Long">
<column name="user_id" />
<generator class="sequence">
<param name="sequence">user_user_id_seq</param>
</generator>
</id>
<many-to-one name="checkin" class="org.server.entity.Checkin" fetch="select">
<column name="is_at" />
</many-to-one>
I cannot see nothing strange, maybe more eyes see more? The strange thing is that in my IDE is working. Somebody had ever seen similar error?
hibernate: 3.6.2
version tomcat 6.0.33
java openjdk 1.6.0.18
postgresql: 8.3.9
UPDATE:
The Java code for this:
// queryHistory is to be persisted
// user is not null
queryHistory.setUser(user);
queryHistory.setApiKey(apiKey);
// create point
Coordinate coordinate = new Coordinate(queryHeader.getLocLon(),queryHeader.getLocLat());
GeometryFactory geometryFactory = new GeometryFactory(new PrecisionModel(),4326);
Point point = geometryFactory.createPoint(coordinate);
queryHistory.setGeom(point);
if(user!=null) {
user.setGeom(point);
user.setLastSeenAt(new Timestamp(new Date().getTime()));
hSession.update(user);
}
hSession.persist(queryHistory);
t.commit();
Thanks in advance