I am managing a web-based project based on java, subversion and svn with 8 developers. Unfortunately, mangling DB changes is a big problem for the project. In our case, every user may update the tables and forgot to put the change scripts in svn. So, it takes lots of our time to see and debug an issue raised because of an un-updated table or view.
So, I wonder, is there any method, tool or plug-in for oracle 11g to keep all DB changes as scripts for us somewhere, e.g. on svn?
Edit 1: Getting a dump from the whole db does not solve my problem, because in the real environment I cannot discard customer data and go back to a new dump.
I think this is just what you need. An open source database change management system. Liquibase.
http://www.liquibase.org/
Do not store change scripts, only scripts that drop and recreate all your objects. Developers should change and run those scripts on a local instance, run automated unit tests, and then check-in their changes.
Rebuilding from scratch is so much better than constantly running alter scripts. You'll never be in control of your application until everyone can easily rebuild the entire system from scratch.
(I assume you're asking about development on trunk, where you have lots of little changes. For major upgrades, like moving from version 1.1 to version 1.2, you'll still need to use change scripts to help preserve data.)
More cheap and worse solution, than Liquibase, according to Oracle: exporting only schema topic, can be
post-commit hook, which
expdp ... DUMPFILE=file.dmp CONTENT=METADATA_ONLY into dir, which is WC or special location in repository
commit this file.dmp
There are two aspects to maintaining database changes. One, as you mentioned, in in the form of scripts that can be applied to an older schema to upgrade it. However, this is part of the answer, as it is really hard for a developer to look at scripts, parse them, and figure out how recent schema changes may affect their work.
So, in addition to change scripts, I would suggest that you also check in a human-readable version of the database metadata, in a text file. SchemaCrawler is one such free tool that is designed for this purpose, and produces rich metadata information in a format that is designed to be diffed. I have found that database metadata changes over time become traceable if you make it a nightly process to automate check-ins of schema metadata.
Please try this tool: www.dbapply.com
It has both GUI for manual deployment of scripts from Subversion repository and command line interface for continuous integration.
Supports Subversion branches.
Can work on Windows and Linux (you need JRE 8).
Related
I have a java app which I deploy on various plateforms (using ansible).
This app uses a database, which sometimes needs to get schema updates, which I perform and log/version with flyway (as a software dependency).
I now face the need to update data on all plateforms, but with different values depending on the plateforms. This is not a schema update, but is nonetheless data (list of other apps to which it connects) that forms the main structure of my app, and as such I want it to be versioned, in a similar way to what flyway does.
At first I was thinking I should input the different data in my ansible configuration, which seemed to make sense as it's ansible that knows about the various plateforms. And then I thought that this information would get passed to flyway somehow so that it performs the required updates.
However if that is handled using 'versioned migrations', I could end up with version conflicts because one environment requires an update and another doesn't (common versioning vs environment versioning).
There is a slight mention of this issue in the flyway FAQ, and one can set the flyway.locations property, or maybe I could use flyway placeholders that are set by ansible ?
Am I on the right track ? Or should I not use flyway altogether (is it meant to be used with DML, or should it be reserved for DDL) ?
Flyway can be used for both schema and data updates. Although it's primary purpose is around versioning schema updates.
It sounds like you need a way to deploy some scripts only in certain environments. Flyway provides functionality that will support this workflow. However, you'll need to decide on the approach that works best for you.
Here are some ideas.
Use different locations
The simplest way I can think of is to have environment specific scripts in their own locations. You can also have a location for 'common' scripts.
When you deploy, you can specify the 'common' location, alongside the environment specific one. Something like:
flyway migration -locations=common/sql, test/sql
flyway migration -locations=common/sql, production/sql
And so on.
shouldExecute script config & placeholders
Another way is to use the Flyway Teams feature shouldExecute. This let's you define a boolean expression to determine if a script should be run. You can inject a value from a placeholder. There is a blog post that explains more about it.
Use the cherryPick configuration option
Another Teams Edition feature is cherryPick, which allows you to specify exactly which scripts to deploy. So you might have a configuration file per environment with a cherryPick config that specifies the exact scripts to run. This one might be unwieldy since you need to explicitly list every script, but it does give you complete control.
I have flyway integrated in one of my projects. I have many migrations and it takes a long time to migrate a new empty database, mainly because there are seed data added along the way as well. Now I want to change that. Unfortunately those migrations were already pushed to production (and yes, at some point the seed data was migrated there as well).
My idea was to set a baseline for the current version of the production system and clean up the old migrations afterwards: Squash the schema-migrations and move the seed- and test-data to a new location, that is not deployed to production.
Now my questions are:
How can I set a baseline in my production database, without affecting all others? Call flyway baseline ... on the database directly? Or can I use any kind of special migration file? Maybe write the baseline line directly to the schema_version table of the database? How would such a query look like?
My newest migration is V4_6_3__.... So my baseline needs to be on V5__...? Or is V4__... enough and all migrations with the same major version are included?
When the baseline is set, is it possible/save to add, edit, and remove migrations, older than the baseline, without breaking my production database on the next migration task?
Sorry for the basic questions, but it seems to me, that the flyway documentation is no help at all...
Thanks in advance!
Sry for the late answer. I have done a pretty similar thing to the one #markdsievers suggested:
I assured that the production environment was at least on version X (flyway.setTarget(X)). Then I changed to a new schema version table (flyway.setTable('temporary_schema_version')) and ran a single migration, that deleted the old table. Afterwards I changed the schema version table back to the original one, set a baseline to version Y > X and ran another migration that deleted the temporary table.
Now I can edit/squash/delete all migrations with a version lower than Y without crashing my production-deployment.
I went through a very similar scenario, and even wrote our own in house tool called the "Rebaser" which does most of what you want. Our main motivation was to upgrade from Flyway 3.x to 4.3 but we also had a large history which needed to be squashed. The gist of it is this:
Squash all your migrations into however many makes sense. I typically have a V2__base_ddl.sql and V3__base_data.sql (Flyway can use the first couple of version numbers for schema creation etc). This is the manual part.
Your rebase tool detects the old schema_version table and deletes it.
Your rebase tool then runs init + migrate with your new baseline version set.
Rebase tool leaves behind a footprint (a rebase key in a custom table) that indicates it has been done.
For my integration test builds (that spin up a vanilla database and migrate forwards to latest) I add an extra folder of test data SQL scripts using Flyways locations argument, thus ensuring I have test data for integration tests but not in any non-test environments.
Our Rebaser is just a thin wrapper around the Flyway Java API, adding in the prestep to do the rebase if configured and then delegate to Flyway.
Flyway doesn't have a notion of rebasing but it's something we have found is necessary to do as your history gets large and contains obsolete data / DDL. So far this system has worked flawlessly.
I have a Java-based server, transmitting data from many remote devices to one app via TCP/IP. I need to develop several versions of it. How can I develop and then dwell them without need in coding for 2 projects?I'm asking not only for that project, but for different approaches.
Where the behaviour differs, make the behaviour "data driven" - typically by externalizing the data the drives the behaviour to properties files that are read at runtime/startup.
The goal is to have a single binary whose behaviour varies depending on the properties files found in the runtime environment.
Java supports this pattern through the Properties class, which offers convenient ways of loading properties. In fact, most websites operate in this way, for example the production database user/pass details are never (should never be) in the code. The sysadmins will edit a properties file that is read at start up, and which is protected by the operating system's file permissions.
Other options are to use a database to store the data that drives behaviour.
It can be a very powerful pattern, but it can be abused too, so some discretion is advised.
I think you need to read up on Source Control Management (SCM) and Version Control Systems (VCS).
I would recommend setting up a git or Subversion repository and adding the code initially to trunk and then branching it off to the number of branches (versions you'll be working on).
The idea of different versions is this:
You're developing your code and have it in your SCM's trunk (or otherwise known as a HEAD). At some point you consider the code stable enough for a release. You therefore create a tag (let's call it version 1.0). You cannot (should not) make changes to tags -- they're only there as a marker in time for you. If you have a client who has version 1.0 and reports bugs which you would like to fix, you create a branch based on a copy of your tag. The produced version would (normally) be 1.x (1.1, 1.2, etc). When you're done with your fixes, you tag again and release the new version.
Usually, most of the development happens on your trunk.
When you are ready with certain fixes, or know that certain fixes have already been applied to your trunk, you can merge these changes to other branches, if necessary.
Make any other version based on previous one by reusing code base, configurations and any other asset. In case if several versions should be in place at one time use configuration management practices. Probably you should consider some routing activities and client version checks on server side. This is the place where 'backward compatibility' comes to play.
The main approach is first to find and extract the code that won't change from one version to another. The best is to maximize this part to share the maximum of code base and to ease the maintenance (correcting a bug for one means correcting for all).
Then it depends on what really changes from one version to another. The best is that on the main project you can use some abstract classes or interfaces that you will be able to implement for each specific project.
My application is always developing, so occasionally - when the version upgrades - some tables need to be created/altered/deleted, some data modified, etc. Generally some sql code needs to be executed.
Is there a Java library that can be used to keep my database structure up to date (by analyzing something like "db structure version" information and executing custom sql to code to update from one version to another)?
Also it would be great to have some basic actions (like add/remove column) ready to use with minimal configuration, ie name/type and no sql code.
Try DBDeploy. Although I haven't used it in the past, it sounds like this project would help in your case. DBDeploy is a database refactoring manager that:
"Automates the process of establishing
which database refactorings need to be
run against a specific database in
order to migrate it to a particular
build."
It is known to integrate with both Ant and Maven.
Try Liquibase.
Liquibase is an open source (Apache
2.0 Licensed), database-independent library for tracking, managing and
applying database changes. It is built
on a simple premise: All database
changes are stored in a human readable
yet trackable form and checked into
source control.
Supported features:
Extensibility
Merging changes from multiple developers
Code branches
Multiple Databases
Managing production data as well as various test datasets
Cluster-safe database upgrades
Automated updates or generation of SQL scripts that can be approved and
applied by a DBA
Update rollbacks
Database ”diff“s
Generating starting change logs from existing databases
Generating database change documentation
We use a piece of software called Liquibase for this. It's very flexible and you can set it up pretty much however you want it. We have it integrated with Maven so our database is always up to date.
You can also check Flyway (400 questions tagged on SOW) or mybatis (1049 questions tagged). To add to the comparison the other options mentioned: Liquibase (663 questions tagged) and DBDeploy (24 questions tagged).
Another resource that you can find useful is the feature comparison in the Flyway website (There are other related projects mentioned there).
You should take a look into OR Mapping libraries, e.g. Hibernate
Most ORM mappers have logic to do schema upgrades for you, I have successfully used Hibernate which gets at least the basic stuff right automatically.
Please help… Old Programmer looking to use subversion assembla at my firm. I am doing alot of Java in Eclipse and my issues is the following. I am going to make it very easy.
1) I build a web site in Eclipse with JSP. I check it in and commit. it is live
2) I start working on version two of the site but someone finds a bug in one of the prod JSPs. how do I checkout that version of the jsp update it and then commit it to the project. please tell me the right steps
Here's the workflow that most organizations use:
When you make a production release, you tag it. In SVN, this is done with svn cp, copying trunk into a named directory under tags.
If you need to make bugfixes to a production release, you use svn cp to copy the tagged revision into a branch under branches. You then check out this named revision, make your changes, and check in.
If you're going to push the changes out to production, you can tag them from the branch, again using svn cp. Tags are cheap in Subversion.
If the fixes you made in the branch need to go back into trunk, you can merge them.
This is covered in the docs (this is a link to the chapter on branching and merging, but I recommend you read the introductory material if you're not familiar with SVN).
http://svnbook.red-bean.com/en/1.5/index.html
Read chapters 2 and 3 and that will be enough to hit the ground running. The command you are likely looking for is svn update -rNNN however, without some background on SVN odds are excellent that you'll misuse it as SVN is very like (yet in some ways different) than the old school CVS, RCS, SCCS like systems.
You might want to skim chapter 1 too, as the revisioning model SVN uses is a little different than tight locking models (if you've been using one of those).