I have flyway integrated in one of my projects. I have many migrations and it takes a long time to migrate a new empty database, mainly because there are seed data added along the way as well. Now I want to change that. Unfortunately those migrations were already pushed to production (and yes, at some point the seed data was migrated there as well).
My idea was to set a baseline for the current version of the production system and clean up the old migrations afterwards: Squash the schema-migrations and move the seed- and test-data to a new location, that is not deployed to production.
Now my questions are:
How can I set a baseline in my production database, without affecting all others? Call flyway baseline ... on the database directly? Or can I use any kind of special migration file? Maybe write the baseline line directly to the schema_version table of the database? How would such a query look like?
My newest migration is V4_6_3__.... So my baseline needs to be on V5__...? Or is V4__... enough and all migrations with the same major version are included?
When the baseline is set, is it possible/save to add, edit, and remove migrations, older than the baseline, without breaking my production database on the next migration task?
Sorry for the basic questions, but it seems to me, that the flyway documentation is no help at all...
Thanks in advance!
Sry for the late answer. I have done a pretty similar thing to the one #markdsievers suggested:
I assured that the production environment was at least on version X (flyway.setTarget(X)). Then I changed to a new schema version table (flyway.setTable('temporary_schema_version')) and ran a single migration, that deleted the old table. Afterwards I changed the schema version table back to the original one, set a baseline to version Y > X and ran another migration that deleted the temporary table.
Now I can edit/squash/delete all migrations with a version lower than Y without crashing my production-deployment.
I went through a very similar scenario, and even wrote our own in house tool called the "Rebaser" which does most of what you want. Our main motivation was to upgrade from Flyway 3.x to 4.3 but we also had a large history which needed to be squashed. The gist of it is this:
Squash all your migrations into however many makes sense. I typically have a V2__base_ddl.sql and V3__base_data.sql (Flyway can use the first couple of version numbers for schema creation etc). This is the manual part.
Your rebase tool detects the old schema_version table and deletes it.
Your rebase tool then runs init + migrate with your new baseline version set.
Rebase tool leaves behind a footprint (a rebase key in a custom table) that indicates it has been done.
For my integration test builds (that spin up a vanilla database and migrate forwards to latest) I add an extra folder of test data SQL scripts using Flyways locations argument, thus ensuring I have test data for integration tests but not in any non-test environments.
Our Rebaser is just a thin wrapper around the Flyway Java API, adding in the prestep to do the rebase if configured and then delegate to Flyway.
Flyway doesn't have a notion of rebasing but it's something we have found is necessary to do as your history gets large and contains obsolete data / DDL. So far this system has worked flawlessly.
Related
We are using Flyway for migration of our DB which and we literally have thousands of schemas (a.k.a Silos).
When we deploy a new build, the DB migration may take 10 minutes even if there is no migration needed for the software build.
I wonder if we could configure Flyway to do a lazy DB migration: check each the schema_version table for all the schemas, if latest DB version in the table is equal to that of the current software build, then there is no need to do anything; otherwise, if db version in software builder is newer, just do the necessary migration starting from latest version in the table.
If we can do the above, the migration time could shrink from 10 minutes to a few seconds.
Could anyone shed any light on how to do this? Thanks!
As stated by JB Nizet, Flyway checks the current version, and does not perform any update unless required.
However, if your goal is getting rid of this check because it takes time, you could maybe switch to Java-based migrations instead of SQL-based ones, because unlike the latter ones, Java migrations do not have checksums, and this is were you might reduce some execution time; still has to be tested though.
This is from the official doc in this matter:
Unlike SQL migrations, Java migrations by default do not have a
checksum and therefore do not participate in the change detection of
Flyway’s validation. This can be remedied by implementing the
MigrationChecksumProvider interface.
I do not have an idea if this will reduce execution time, but it may be worth testing at least.
I am managing a web-based project based on java, subversion and svn with 8 developers. Unfortunately, mangling DB changes is a big problem for the project. In our case, every user may update the tables and forgot to put the change scripts in svn. So, it takes lots of our time to see and debug an issue raised because of an un-updated table or view.
So, I wonder, is there any method, tool or plug-in for oracle 11g to keep all DB changes as scripts for us somewhere, e.g. on svn?
Edit 1: Getting a dump from the whole db does not solve my problem, because in the real environment I cannot discard customer data and go back to a new dump.
I think this is just what you need. An open source database change management system. Liquibase.
http://www.liquibase.org/
Do not store change scripts, only scripts that drop and recreate all your objects. Developers should change and run those scripts on a local instance, run automated unit tests, and then check-in their changes.
Rebuilding from scratch is so much better than constantly running alter scripts. You'll never be in control of your application until everyone can easily rebuild the entire system from scratch.
(I assume you're asking about development on trunk, where you have lots of little changes. For major upgrades, like moving from version 1.1 to version 1.2, you'll still need to use change scripts to help preserve data.)
More cheap and worse solution, than Liquibase, according to Oracle: exporting only schema topic, can be
post-commit hook, which
expdp ... DUMPFILE=file.dmp CONTENT=METADATA_ONLY into dir, which is WC or special location in repository
commit this file.dmp
There are two aspects to maintaining database changes. One, as you mentioned, in in the form of scripts that can be applied to an older schema to upgrade it. However, this is part of the answer, as it is really hard for a developer to look at scripts, parse them, and figure out how recent schema changes may affect their work.
So, in addition to change scripts, I would suggest that you also check in a human-readable version of the database metadata, in a text file. SchemaCrawler is one such free tool that is designed for this purpose, and produces rich metadata information in a format that is designed to be diffed. I have found that database metadata changes over time become traceable if you make it a nightly process to automate check-ins of schema metadata.
Please try this tool: www.dbapply.com
It has both GUI for manual deployment of scripts from Subversion repository and command line interface for continuous integration.
Supports Subversion branches.
Can work on Windows and Linux (you need JRE 8).
I have a much used project that I am working on currently updating. There are several places where this project can be installed, and in the future it is not certain what version is used where and to what version one might be updated to in the future. Right now they are all the same, though.
My problem stems from the fact that there might be many changes to the hibernate entity classes, and it must be easy to update to a newer version without any hassle, and no loss of database content. Just replace WAR and start and it should migrate itself.
To my knowledge Hibernate does no altering of tables unless hibernate.hbm2ddl.auto=create, but which actually throws away all the data?
So right now when the Spring context has fully loaded, it executes a bean that will migrate the database to the current version by going through all the changes from versionX to versionY (what version it previously was is saved in the database), and manually alter the table.
It's not much hassle doing a few hard-coded ALTER TABLE to add some columns, but when it comes to adding complete new tables, it feels silly to have to write all that...
So my question(s) is this:
Is there any way to send an entity class and a dialect to Hibernate
code somewhere, and get back a valid SQL query for creating a table?
And even better, somehow create an SQL string for adding a column to a table, dialect-safe?
I hope this is not a silly question, and I have not missed something obvious when it comes to Hibernate...
have you tried
hibernate.hbm2ddl.auto=update
it retains all the database with the data and append only columns and tables you have changed in entity.
I don't think you'll be able to fully automate this. Hibernate has the hbm2ddl tool (available as an ant task or a maven plugin) to generate the required DDL statements from your hibernate configuration to create an empty database but I'm not aware of any tools that can do an automatic "diff" between two versions. In any case you're probably better off doing the diff carefully by hand, as only you know your object model well enough to be able to pick the right defaults for new properties of existing entities etc.
Once you have worked out your diffs you can use a tool like liquibase to manage them and handle actually applying the updates to a database at application start time.
Maybe you should try a different approach. In stead of generating an schema at runtime update, make one 'by hand' (could be based on a hibernate generated script though).
Store a version number in the database and create an update script for every next version. The only thing you have to do now is determine in which version the database currently is and sequentially run the necessary update scripts to get it to the current version.
To make it extra robust you can make a unit/integration test which runs every possible database update and checks the integrity of the resulting database.
I used this method for an application I build and it works flawlessly. An other example of an implementation of this pattern is Android. They have an upgrade method in their API
http://developer.android.com/reference/android/database/sqlite/SQLiteOpenHelper.html#onUpgrade(android.database.sqlite.SQLiteDatabase, int, int)
Don't use Hibernate's ddl. It throws away your data if you want to migrate. I suggest you take a look at Liquibase. Liquibase is a database version control. It works using changesets. Each changeset can be created manually or you can let Liquibase read your Hibernate config and generate a changeset.
Liquibase can be started via Spring so it should fit right in with your project ;-)
MyBatis migrations splits each SQL file into two sections:
One for migrating forward one version
One for migrating back one version
How does one roll back versions using Flyway?
While Flyway supports rollbacks (as a commercial-only feature) its use is discouraged:
https://flywaydb.org/documentation/command/undo
While the idea of undo migrations is nice, unfortunately it sometimes breaks down in practice. As soon as you have destructive changes (drop, delete, truncate, …), you start getting into trouble. And even if you don’t, you end up creating home-made alternatives for restoring backups, which need to be properly tested as well.
Undo migrations assume the whole migration succeeded and should now be undone. This does not help with failed versioned migrations on databases without DDL transactions. Why? A migration can fail at any point. If you have 10 statements, it is possible for the 1st, the 5th, the 7th or the 10th to fail. There is simply no way to know in advance. In contrast, undo migrations are written to undo an entire versioned migration and will not help under such conditions.
An alternative approach which we find preferable is to maintain backwards compatibility between the DB and all versions of the code currently deployed in production. This way a failed migration is not a disaster. The old version of the application is still compatible with the DB, so you can simply roll back the application code, investigate, and take corrective measures.
This should be complemented with a proper, well tested, backup and restore strategy. It is independent of the database structure, and once it is tested and proven to work, no migration script can break it. For optimal performance, and if your infrastructure supports this, we recommend using the snapshot technology of your underlying storage solution. Especially for larger data volumes, this can be several orders of magnitude faster than traditional backups and restores.
This is supported since Flyway 5.0. Sadly it's a commercial only feature though.
https://flywaydb.org/documentation/command/undo
I assume you need a rollback strategy, when e.g. a partner fails at production stage and his deployment is essential for your release.
You could name your flyway SQL scripts like these:
V< YourReleaseNumber >.000_< description >.sql
Now you can leave
V< YourReleaseNumber >.998_rollback.sql for rollback
and make V< YourReleaseNumber >.999_reenroll.sql to reenroll.
In your CI/CD Environment you need 2 more Jobs (manually triggered) after your deployment job. One for rollback, which runs the rollback process including flyway migrate. Other for reenroll.
You just have to care for the target configuration in flyway.
For your deployment job your target should be < YourReleaseNumber >.997
For your rollback job < YourReleaseNumber >.998
When you start a new release, make sure you won't run the rollback/reenroll script of the old release.
As said before a well tested, backup and restore strategy is the recommended solution.
(sry for bad english)
I've been working on a Django project using South to track and manage database schema changes. I'm starting a new Java project using Google Web Toolkit and wonder if there is an equivalent tool. For those who don't know, here's what South does:
Automatically recognize changes to my Python database models (add/delete columns, tables etc.)
Automatically create SQL statements to apply those changes to my database
Track the applied schema migrations and apply them in order
Allow data migrations using Python code. For example, splitting a name field into a first-name and last-name field using the Python split() function
I haven't decided on my Java ORM yet, but Hibernate looks like the most popular. For me, the ability to easily make database schema changes will be an important factor.
Wow, South sounds pretty awesome! I'm not sure of anything out-of-the-box that will help you nearly as much as that does, however if you choose Hibernate as your ORM solution you can build your own incremental data migration suite without a lot of trouble.
Here was the approach that I used in my own project, it worked fairly well for me for over a couple of years and several updates/schema changes:
Maintain a schema_version table in the database that simply defines a number that represents the version of your database schema. This table can be handled outside of the scope of Hibernate if you wish.
Maintain the "current" version number for your schema inside your code.
When the version number in code is newer than what's the database, you can use Hibernate's SchemaUpdate utility which will detect any schema additions (NOTE, just additions) such as new tables, columns, and constraints.
Finally I maintained a "script" if you will of migration steps that were more than just schema changes that were identified by which schema version number they were required for. For instance new columns needed default values applied or something of that nature.
This may sounds like a lot of work, especially when coming from an environment that took care of a lot of it for you, but you can get a setup like this rolling pretty quickly with Hibernate and it is pretty easy to add onto as you go on. I never ended up making any changes to my incremental update framework over that time except to add new migration tasks.
Hopefully someone will come along with a good answer for a more "hands-off" approach, but I thought I'd share an approach that worked pretty well for me.
Good luck to you!
as I'm looking for the same heres what I've achieved so far.
We first used dbdeploy. It manages the most stuff for you but you will have to write all the transition scripts by yourself! That means every change you make has to be in its own script which you will have to write from scratch. Not very handy, but works very reliable.
The second thing I encountered is liquibase. It stores the configuration in one single xml file. Not very intuitive to read, but managable. Plus there is an Intellij Idea plugin for it. At the moment of writing it still has some minor issues, but as the author assured me, they will be fixed soon.
The perfect solution would be to get south working with your java environment. That really would be a tool to marry :D
Maybe try flyaway. Seems like a good alternative.
I've been thinking about using django-jython just for db migrations in our legacy Java application. The latest Jython version is 2.5.4rc1, but I think I can mitigate the risk by just using it for South migrations.
Especially since I can use inspectdb to generate the models for me. And then replace parts of the Java with Python "seamlessly".
If you're using hibernate, then checkout liquibase
http://www.liquibase.org/databases.html
It's been around for 10 years so it's pretty solid. It may support other ORM's, just have a dig around on their website. Checkout the liquibase+hibernate extension here:
https://github.com/liquibase/liquibase-hibernate