MyBatis migrations splits each SQL file into two sections:
One for migrating forward one version
One for migrating back one version
How does one roll back versions using Flyway?
While Flyway supports rollbacks (as a commercial-only feature) its use is discouraged:
https://flywaydb.org/documentation/command/undo
While the idea of undo migrations is nice, unfortunately it sometimes breaks down in practice. As soon as you have destructive changes (drop, delete, truncate, …), you start getting into trouble. And even if you don’t, you end up creating home-made alternatives for restoring backups, which need to be properly tested as well.
Undo migrations assume the whole migration succeeded and should now be undone. This does not help with failed versioned migrations on databases without DDL transactions. Why? A migration can fail at any point. If you have 10 statements, it is possible for the 1st, the 5th, the 7th or the 10th to fail. There is simply no way to know in advance. In contrast, undo migrations are written to undo an entire versioned migration and will not help under such conditions.
An alternative approach which we find preferable is to maintain backwards compatibility between the DB and all versions of the code currently deployed in production. This way a failed migration is not a disaster. The old version of the application is still compatible with the DB, so you can simply roll back the application code, investigate, and take corrective measures.
This should be complemented with a proper, well tested, backup and restore strategy. It is independent of the database structure, and once it is tested and proven to work, no migration script can break it. For optimal performance, and if your infrastructure supports this, we recommend using the snapshot technology of your underlying storage solution. Especially for larger data volumes, this can be several orders of magnitude faster than traditional backups and restores.
This is supported since Flyway 5.0. Sadly it's a commercial only feature though.
https://flywaydb.org/documentation/command/undo
I assume you need a rollback strategy, when e.g. a partner fails at production stage and his deployment is essential for your release.
You could name your flyway SQL scripts like these:
V< YourReleaseNumber >.000_< description >.sql
Now you can leave
V< YourReleaseNumber >.998_rollback.sql for rollback
and make V< YourReleaseNumber >.999_reenroll.sql to reenroll.
In your CI/CD Environment you need 2 more Jobs (manually triggered) after your deployment job. One for rollback, which runs the rollback process including flyway migrate. Other for reenroll.
You just have to care for the target configuration in flyway.
For your deployment job your target should be < YourReleaseNumber >.997
For your rollback job < YourReleaseNumber >.998
When you start a new release, make sure you won't run the rollback/reenroll script of the old release.
As said before a well tested, backup and restore strategy is the recommended solution.
(sry for bad english)
Related
I am trying to understand when to use schema.sql db creation technique and when to rely on Spring boot's creation based on my entity classes. How to decide?
Let's leave for the moment the schema.sql out.
ORM automatic schema creation (create or update or create-drop) is normally useful during the development of application. Even during release candidates and QA reviews it is still useful because the changes which happen from development team with issues arising could be live much faster.
When the application reaches a critical phase and is mature in production, then any changes happening automatically from ORM could be considered dangerous.
At this stage you would normally in big companies rollout in production database only some sql scripts that affect the database, which should be reviewed first and tested before rollout happens. Also a rollback sql might be necessary.
So normally ORM having effect in database schema is only during early stages of an application and not when it has matured enough in production.
Let's now come back to schema.sql. This file can be used just once to create the database from some sql commands. This also would not be expected to be run anytime the application executes. At least not in majority of applications.
I think a logical approach would be to use the ORM during intial stages of development and QA and then when you are about to reach a mature phase, you inspect what type of database the ORM has created and then you make a manual review just to be sure of everything and any optimizations which would make sense and at this stage you can create with the already existing schema of ORM your own permanent schema.sql.
The obvious reason for the above is that with schema.sql you have 100% control of how the database is created. Using ORM you depend on the ORM provider to build this for you. This ORM provider might provide some new library that affects how the previous was used and many other things. The result is that with ORM you don't have 100% control on the database to be created.
We are using Flyway for migration of our DB which and we literally have thousands of schemas (a.k.a Silos).
When we deploy a new build, the DB migration may take 10 minutes even if there is no migration needed for the software build.
I wonder if we could configure Flyway to do a lazy DB migration: check each the schema_version table for all the schemas, if latest DB version in the table is equal to that of the current software build, then there is no need to do anything; otherwise, if db version in software builder is newer, just do the necessary migration starting from latest version in the table.
If we can do the above, the migration time could shrink from 10 minutes to a few seconds.
Could anyone shed any light on how to do this? Thanks!
As stated by JB Nizet, Flyway checks the current version, and does not perform any update unless required.
However, if your goal is getting rid of this check because it takes time, you could maybe switch to Java-based migrations instead of SQL-based ones, because unlike the latter ones, Java migrations do not have checksums, and this is were you might reduce some execution time; still has to be tested though.
This is from the official doc in this matter:
Unlike SQL migrations, Java migrations by default do not have a
checksum and therefore do not participate in the change detection of
Flyway’s validation. This can be remedied by implementing the
MigrationChecksumProvider interface.
I do not have an idea if this will reduce execution time, but it may be worth testing at least.
I have flyway integrated in one of my projects. I have many migrations and it takes a long time to migrate a new empty database, mainly because there are seed data added along the way as well. Now I want to change that. Unfortunately those migrations were already pushed to production (and yes, at some point the seed data was migrated there as well).
My idea was to set a baseline for the current version of the production system and clean up the old migrations afterwards: Squash the schema-migrations and move the seed- and test-data to a new location, that is not deployed to production.
Now my questions are:
How can I set a baseline in my production database, without affecting all others? Call flyway baseline ... on the database directly? Or can I use any kind of special migration file? Maybe write the baseline line directly to the schema_version table of the database? How would such a query look like?
My newest migration is V4_6_3__.... So my baseline needs to be on V5__...? Or is V4__... enough and all migrations with the same major version are included?
When the baseline is set, is it possible/save to add, edit, and remove migrations, older than the baseline, without breaking my production database on the next migration task?
Sorry for the basic questions, but it seems to me, that the flyway documentation is no help at all...
Thanks in advance!
Sry for the late answer. I have done a pretty similar thing to the one #markdsievers suggested:
I assured that the production environment was at least on version X (flyway.setTarget(X)). Then I changed to a new schema version table (flyway.setTable('temporary_schema_version')) and ran a single migration, that deleted the old table. Afterwards I changed the schema version table back to the original one, set a baseline to version Y > X and ran another migration that deleted the temporary table.
Now I can edit/squash/delete all migrations with a version lower than Y without crashing my production-deployment.
I went through a very similar scenario, and even wrote our own in house tool called the "Rebaser" which does most of what you want. Our main motivation was to upgrade from Flyway 3.x to 4.3 but we also had a large history which needed to be squashed. The gist of it is this:
Squash all your migrations into however many makes sense. I typically have a V2__base_ddl.sql and V3__base_data.sql (Flyway can use the first couple of version numbers for schema creation etc). This is the manual part.
Your rebase tool detects the old schema_version table and deletes it.
Your rebase tool then runs init + migrate with your new baseline version set.
Rebase tool leaves behind a footprint (a rebase key in a custom table) that indicates it has been done.
For my integration test builds (that spin up a vanilla database and migrate forwards to latest) I add an extra folder of test data SQL scripts using Flyways locations argument, thus ensuring I have test data for integration tests but not in any non-test environments.
Our Rebaser is just a thin wrapper around the Flyway Java API, adding in the prestep to do the rebase if configured and then delegate to Flyway.
Flyway doesn't have a notion of rebasing but it's something we have found is necessary to do as your history gets large and contains obsolete data / DDL. So far this system has worked flawlessly.
I am managing a web-based project based on java, subversion and svn with 8 developers. Unfortunately, mangling DB changes is a big problem for the project. In our case, every user may update the tables and forgot to put the change scripts in svn. So, it takes lots of our time to see and debug an issue raised because of an un-updated table or view.
So, I wonder, is there any method, tool or plug-in for oracle 11g to keep all DB changes as scripts for us somewhere, e.g. on svn?
Edit 1: Getting a dump from the whole db does not solve my problem, because in the real environment I cannot discard customer data and go back to a new dump.
I think this is just what you need. An open source database change management system. Liquibase.
http://www.liquibase.org/
Do not store change scripts, only scripts that drop and recreate all your objects. Developers should change and run those scripts on a local instance, run automated unit tests, and then check-in their changes.
Rebuilding from scratch is so much better than constantly running alter scripts. You'll never be in control of your application until everyone can easily rebuild the entire system from scratch.
(I assume you're asking about development on trunk, where you have lots of little changes. For major upgrades, like moving from version 1.1 to version 1.2, you'll still need to use change scripts to help preserve data.)
More cheap and worse solution, than Liquibase, according to Oracle: exporting only schema topic, can be
post-commit hook, which
expdp ... DUMPFILE=file.dmp CONTENT=METADATA_ONLY into dir, which is WC or special location in repository
commit this file.dmp
There are two aspects to maintaining database changes. One, as you mentioned, in in the form of scripts that can be applied to an older schema to upgrade it. However, this is part of the answer, as it is really hard for a developer to look at scripts, parse them, and figure out how recent schema changes may affect their work.
So, in addition to change scripts, I would suggest that you also check in a human-readable version of the database metadata, in a text file. SchemaCrawler is one such free tool that is designed for this purpose, and produces rich metadata information in a format that is designed to be diffed. I have found that database metadata changes over time become traceable if you make it a nightly process to automate check-ins of schema metadata.
Please try this tool: www.dbapply.com
It has both GUI for manual deployment of scripts from Subversion repository and command line interface for continuous integration.
Supports Subversion branches.
Can work on Windows and Linux (you need JRE 8).
My application is always developing, so occasionally - when the version upgrades - some tables need to be created/altered/deleted, some data modified, etc. Generally some sql code needs to be executed.
Is there a Java library that can be used to keep my database structure up to date (by analyzing something like "db structure version" information and executing custom sql to code to update from one version to another)?
Also it would be great to have some basic actions (like add/remove column) ready to use with minimal configuration, ie name/type and no sql code.
Try DBDeploy. Although I haven't used it in the past, it sounds like this project would help in your case. DBDeploy is a database refactoring manager that:
"Automates the process of establishing
which database refactorings need to be
run against a specific database in
order to migrate it to a particular
build."
It is known to integrate with both Ant and Maven.
Try Liquibase.
Liquibase is an open source (Apache
2.0 Licensed), database-independent library for tracking, managing and
applying database changes. It is built
on a simple premise: All database
changes are stored in a human readable
yet trackable form and checked into
source control.
Supported features:
Extensibility
Merging changes from multiple developers
Code branches
Multiple Databases
Managing production data as well as various test datasets
Cluster-safe database upgrades
Automated updates or generation of SQL scripts that can be approved and
applied by a DBA
Update rollbacks
Database ”diff“s
Generating starting change logs from existing databases
Generating database change documentation
We use a piece of software called Liquibase for this. It's very flexible and you can set it up pretty much however you want it. We have it integrated with Maven so our database is always up to date.
You can also check Flyway (400 questions tagged on SOW) or mybatis (1049 questions tagged). To add to the comparison the other options mentioned: Liquibase (663 questions tagged) and DBDeploy (24 questions tagged).
Another resource that you can find useful is the feature comparison in the Flyway website (There are other related projects mentioned there).
You should take a look into OR Mapping libraries, e.g. Hibernate
Most ORM mappers have logic to do schema upgrades for you, I have successfully used Hibernate which gets at least the basic stuff right automatically.