{"id":24,"date":"2017-06-12T08:34:00","date_gmt":"2017-06-12T08:34:00","guid":{"rendered":"\/colin\/morethanfour\/post\/Flyway-DB"},"modified":"2017-06-12T08:34:00","modified_gmt":"2017-06-12T08:34:00","slug":"flyway-db","status":"publish","type":"post","link":"https:\/\/spacefold.com\/colin\/morethanfour\/2017\/06\/12\/flyway-db\/","title":{"rendered":"Flyway DB"},"content":{"rendered":"<p>For years I&#8217;ve used a database build script framework written in-house to recompile all schema objects and run unit tests. However, the actual DDL code to migrate a schema from one version to the next was always left as a hand-rolled script that is executed up-front before the normal build process.<\/p>\n<p>A colleague introduced me to Flyway DB and I am now a fan. The introduction, documentation, and tutorial material is top-notch.<\/p>\n<h2>Installing Flyway on Linux<\/h2>\n<p>Flyway DB can be found here:<\/p>\n<ul>\n<li> <a title=\"Flyway DB Web Site\" href=\"https:\/\/flywaydb.org\/\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/flywaydb.org\/<\/a><\/li>\n<\/ul>\n<p>Although it has an API for integration, I specifically like the command-line utility version:  <a title=\"FlyWay DB command line tool\" href=\"https:\/\/flywaydb.org\/getstarted\/firststeps\/commandline\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/flywaydb.org\/getstarted\/firststeps\/commandline<\/a><\/p>\n<p>Unzip the command-line tool version into your preferred program location. In my Linux VM, I decided on ~\/bin:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" title=\"image\" style=\"display: inline\" alt=\"image\" src=\"http:\/\/www.spacefold.com\/colin\/morethanfour\/wp-content\/uploads\/image_3.png\" width=\"462\" height=\"163\"><\/p>\n<p>and created a simlink to put flyway into the path:<\/p>\n<div id=\"codeSnippetWrapper\">\n<pre id=\"codeSnippet\" style=\"border-top-style: none; font-size: 8pt; overflow: visible; border-left-style: none; font-family: 'Courier New', courier, monospace; width: 100%; border-bottom-style: none; color: black; padding-bottom: 0px; direction: ltr; text-align: left; padding-top: 0px; border-right-style: none; padding-left: 0px; margin: 0em; line-height: 12pt; padding-right: 0px; background-color: #f4f4f4\">$ ln -s ~\/bin\/flyway-4.2.0\/flyway ~\/bin\/flyway<\/pre>\n<p><\/div>\n<p>Now I can run it from any directory.<\/p>\n<h2>How Flyway works<\/h2>\n<p>Flyway scans the contents of a \u201cmigration scripts\u201d folder using a file naming convention to understand which scripts belong to which specific version.<\/p>\n<p>It then uses a special table in the target database called \u201cschema_version\u201d that contains metadata about the current version of the database.<\/p>\n<h2>Using Flyway<\/h2>\n<p>The basic syntax is<\/p>\n<p>&gt; flyway VERB<\/p>\n<p>where VERB is one of:<\/p>\n<ul>\n<li>migrate<\/li>\n<li>clean<\/li>\n<li>info<\/li>\n<li>validate<\/li>\n<li>baseline<\/li>\n<li>repair<\/li>\n<\/ul>\n<p>In our case, we need to:<\/p>\n<ul>\n<li>&#8220;baseline&#8221; our database so that Flyway understands the current starting &#8220;version&#8221;;<\/li>\n<li>write a DDL script to move the DB to the next version;<\/li>\n<li>&#8220;migrate&#8221; our database instance to that version.<\/li>\n<\/ul>\n<h3>Setting up a working project directory<\/h3>\n<ul>\n<li>Create a working folder<\/li>\n<li>copy the <em>{install}\/conf\/flyway.conf<\/em> file into the working directory, for project-specific overrides;<\/li>\n<li>create a sub-folder called <em>sql<\/em><\/li>\n<\/ul>\n<p>The working directory I set up was:<\/p>\n<div id=\"codeSnippetWrapper\">\n<pre id=\"codeSnippet\" style=\"border-top-style: none; font-size: 8pt; overflow: visible; border-left-style: none; font-family: 'Courier New', courier, monospace; width: 100%; border-bottom-style: none; color: black; padding-bottom: 0px; direction: ltr; text-align: left; padding-top: 0px; border-right-style: none; padding-left: 0px; margin: 0em; line-height: 12pt; padding-right: 0px; background-color: #f4f4f4\">\/home\/colin\/Projects\/flyway<br>\/home\/colin\/Projects\/flyway\/sql<\/pre>\n<p><\/div>\n<p>I copied the flyway.conf file, ready for local configuration.<\/p>\n<h3>Selecting the target DB<\/h3>\n<p>I&#8217;ve been working through the &#8220;MySQL Cookbook&#8221; tutorial, so I have a user and a database installed in the VM locally, called &#8220;cookbook&#8221;, with one table &#8220;limbs&#8221; already created.<\/p>\n<p>Editing the .conf file:<\/p>\n<div id=\"codeSnippetWrapper\">\n<pre id=\"codeSnippet\" style=\"border-top-style: none; font-size: 8pt; overflow: visible; border-left-style: none; font-family: 'Courier New', courier, monospace; width: 100%; border-bottom-style: none; color: black; padding-bottom: 0px; direction: ltr; text-align: left; padding-top: 0px; border-right-style: none; padding-left: 0px; margin: 0em; line-height: 12pt; padding-right: 0px; background-color: #f4f4f4\">flyway.url=jdbc:mysql:\/\/localhost:3306\/cookbook<br>flyway.user=root<br>flyway.locations=filesystem:.\/sql<\/pre>\n<p><\/div>\n<p>We could supply the password as well, but for interactive use it is safer to have flyway prompt for it. For automated processes, you&#8217;d want to put the password in the .conf file.<\/p>\n<p>Note the syntax of specifying the location of the SQL migration scripts, with a &#8220;filesystem:&#8221; prefix. <\/p>\n<p>Just for kicks, let&#8217;s run the &#8220;migrate&#8221; command:<\/p>\n<div id=\"codeSnippetWrapper\">\n<pre id=\"codeSnippet\" style=\"border-top-style: none; font-size: 8pt; overflow: visible; border-left-style: none; font-family: 'Courier New', courier, monospace; width: 100%; border-bottom-style: none; color: black; padding-bottom: 0px; direction: ltr; text-align: left; padding-top: 0px; border-right-style: none; padding-left: 0px; margin: 0em; line-height: 12pt; padding-right: 0px; background-color: #f4f4f4\">$ flyway migrate<br>Flyway 4.2.0 by Boxfuse<br> <br>Database password: ****<br>Database: jdbc:mysql:\/\/localhost:3306\/cookbook (MySQL 5.7)<br>Successfully validated 0 migrations (execution time 00:00.003s)<br>ERROR: Found non-empty schema(s) `cookbook` without metadata table! Use baseline() or set baselineOnMigrate to true to initialize the<br>$<\/pre>\n<p><\/div>\n<p>We see a couple of issues:<\/p>\n<ul>\n<li>the target DB is missing the metadata table, and therefore Flyway doesn&#8217;t know what version it is;<\/li>\n<li>we don&#8217;t have any upgrade scripts to process.<\/li>\n<\/ul>\n<h3>Baselining our target DB<\/h3>\n<p>We can update the .conf file to include information about the database. Let&#8217;s assume we are starting at version 5:<\/p>\n<div id=\"codeSnippetWrapper\">\n<pre id=\"codeSnippet\" style=\"border-top-style: none; font-size: 8pt; overflow: visible; border-left-style: none; font-family: 'Courier New', courier, monospace; width: 100%; border-bottom-style: none; color: black; padding-bottom: 0px; direction: ltr; text-align: left; padding-top: 0px; border-right-style: none; padding-left: 0px; margin: 0em; line-height: 12pt; padding-right: 0px; background-color: #f4f4f4\">flyway.baselineVersion=5<br>flyway.baselineDescription=Baseline prior to Flyway<\/pre>\n<p><\/div>\n<p>Now we can use the &#8220;baseline&#8221; command to create the metadata:<\/p>\n<div id=\"codeSnippetWrapper\">\n<pre id=\"codeSnippet\" style=\"border-top-style: none; font-size: 8pt; overflow: visible; border-left-style: none; font-family: 'Courier New', courier, monospace; width: 100%; border-bottom-style: none; color: black; padding-bottom: 0px; direction: ltr; text-align: left; padding-top: 0px; border-right-style: none; padding-left: 0px; margin: 0em; line-height: 12pt; padding-right: 0px; background-color: #f4f4f4\">$ flyway baseline<br>Flyway 4.2.0 by Boxfuse<br> <br>Database password: ****<br>Database: jdbc:mysql:\/\/localhost:3306\/cookbook (MySQL 5.7)<br>Creating Metadata table: `cookbook`.`schema_version`<br>Successfully baselined schema with version: 5<br>$<\/pre>\n<p><\/div>\n<p>Querying MySQL shows the table now exists:<\/p>\n<div id=\"codeSnippetWrapper\">\n<pre id=\"codeSnippet\" style=\"border-top-style: none; font-size: 8pt; overflow: visible; border-left-style: none; font-family: 'Courier New', courier, monospace; width: 100%; border-bottom-style: none; color: black; padding-bottom: 0px; direction: ltr; text-align: left; padding-top: 0px; border-right-style: none; padding-left: 0px; margin: 0em; line-height: 12pt; padding-right: 0px; background-color: #f4f4f4\">mysql&gt; show tables;<br>+--------------------+<br>| Tables_in_cookbook |<br>+--------------------+<br>| limbs              |<br>| schema_version     |<br>+--------------------+<br>2 rows <span style=\"color: #0000ff\">in<\/span> set (0.00 sec)<\/pre>\n<p><\/div>\n<p>And the contents:<\/p>\n<div id=\"codeSnippetWrapper\">\n<pre id=\"codeSnippet\" style=\"border-top-style: none; font-size: 8pt; overflow: visible; border-left-style: none; font-family: 'Courier New', courier, monospace; width: 100%; border-bottom-style: none; color: black; padding-bottom: 0px; direction: ltr; text-align: left; padding-top: 0px; border-right-style: none; padding-left: 0px; margin: 0em; line-height: 12pt; padding-right: 0px; background-color: #f4f4f4\">mysql&gt; select * from schema_version;<br>+----------------+---------+--------------------------+----------+--------------------------+----------+--------------+--------------<br>| installed_rank | version | description              | type     | <span style=\"color: #0000ff\">script<\/span>                   | checksum | installed_by | installed_on<br>+----------------+---------+--------------------------+----------+--------------------------+----------+--------------+--------------<br>|              1 |       5 | Baseline prior to Flyway | BASELINE | Baseline prior to Flyway |     NULL | root         | 2017-06-12 14<br>+----------------+---------+--------------------------+----------+--------------------------+----------+--------------+--------------<br>1 row <span style=\"color: #0000ff\">in<\/span> set (0.00 sec)<\/pre>\n<p><\/div>\n<p>We can use the &#8220;info&#8221; verb on the command-line to review the current version according to Flyway:<\/p>\n<div>\n<pre id=\"codeSnippet\" style=\"border-top-style: none; font-size: 8pt; overflow: visible; border-left-style: none; font-family: 'Courier New', courier, monospace; width: 100%; border-bottom-style: none; color: black; padding-bottom: 0px; direction: ltr; text-align: left; padding-top: 0px; border-right-style: none; padding-left: 0px; margin: 0em; line-height: 12pt; padding-right: 0px; background-color: #f4f4f4\">$ flyway info<br>Flyway 4.2.0 by Boxfuse<br> <br>Database password: ****<br>Database: jdbc:mysql:\/\/localhost:3306\/cookbook (MySQL 5.7)<br>+---------+--------------------------+---------------------+---------+<br>| Version | Description              | Installed on        | State   |<br>+---------+--------------------------+---------------------+---------+<br>|       5 | Baseline prior to Flyway | 2017-06-12 14:23:41 | Baselin |<br>+---------+--------------------------+---------------------+---------+<br>$<\/pre>\n<\/div>\n<div>Both schema and table name can be changed in the .conf, but it is probably better to leave it at default. For testing migrations on a freshly-built test database, there is a setting in the .conf that allows automatic default baselining if the schema table is not detected:<\/div>\n<\/p>\n<div id=\"codeSnippetWrapper\">\n<pre id=\"codeSnippet\" style=\"border-top-style: none; font-size: 8pt; overflow: visible; border-left-style: none; font-family: 'Courier New', courier, monospace; width: 100%; border-bottom-style: none; color: black; padding-bottom: 0px; direction: ltr; text-align: left; padding-top: 0px; border-right-style: none; padding-left: 0px; margin: 0em; line-height: 12pt; padding-right: 0px; background-color: #f4f4f4\">flyway.baselineOnMigrate=true<\/pre>\n<p><\/div>\n<p>This can be very handy, along with specifying the DB user password, if security is not a priority.<\/p>\n<h3>Migrating the DB to a new version<\/h3>\n<p>Our current DB version is 5, but I&#8217;m going to create scripts for the versions either side of that, in order to see what Flyway does.<\/p>\n<p>The script filenaming convention is documented here:<\/p>\n<ul>\n<li><a href=\"https:\/\/flywaydb.org\/documentation\/migration\/sql\">https:\/\/flywaydb.org\/documentation\/migration\/sql<\/a><\/li>\n<\/ul>\n<p>There are a number of possibilities including versioning scripts; repeatable scripts; and SQL callback scripts for invoking at various points during the process. For now, we&#8217;re just looking at versioning scripts.<\/p>\n<p>Consider the following script files:<\/p>\n<div id=\"codeSnippetWrapper\">\n<pre id=\"codeSnippet\" style=\"border-top-style: none; font-size: 8pt; overflow: visible; border-left-style: none; font-family: 'Courier New', courier, monospace; width: 100%; border-bottom-style: none; color: black; padding-bottom: 0px; direction: ltr; text-align: left; padding-top: 0px; border-right-style: none; padding-left: 0px; margin: 0em; line-height: 12pt; padding-right: 0px; background-color: #f4f4f4\">$ ls -l sql\/*.sql<br>-rw-rw-r-- 1 colin colin 64 Jun 12 14:35 sql\/V4__other_objects.sql<br>-rw-rw-r-- 1 colin colin 80 Jun 12 14:35 sql\/V5__limbs.sql<br>-rw-rw-r-- 1 colin colin 62 Jun 12 14:36 sql\/V6__test_table2.sql<br>$<\/pre>\n<p><\/div>\n<p>Note: it may be tricky to remember that the name convention requires a double-underscore (&#8220;dunder&#8221;) between the version number and the file name!<\/p>\n<p>Now, if we run the &#8220;info&#8221; verb:<\/p>\n<div id=\"codeSnippetWrapper\">\n<pre id=\"codeSnippet\" style=\"border-top-style: none; font-size: 8pt; overflow: visible; border-left-style: none; font-family: 'Courier New', courier, monospace; width: 100%; border-bottom-style: none; color: black; padding-bottom: 0px; direction: ltr; text-align: left; padding-top: 0px; border-right-style: none; padding-left: 0px; margin: 0em; line-height: 12pt; padding-right: 0px; background-color: #f4f4f4\">Flyway 4.2.0 by Boxfuse<br>Database: jdbc:mysql:\/\/localhost:3306\/cookbook (MySQL 5.7)<br> <br>+---------+--------------------------+---------------------+---------+<br>| Version | Description              | Installed on        | State   |<br>+---------+--------------------------+---------------------+---------+<br>| 4       | other objects            |                     | &lt;Baseln |<br>| 5       | Baseline prior to Flyway | 2017-06-12 14:23:41 | Success |<br>| 6       | test table2              |                     | Pending |<br>+---------+--------------------------+---------------------+---------+<br>$<\/pre>\n<p><\/div>\n<p>Flyway detects this information from a) the script file naming convention, and b) the contents of the schema_version table in the target DB.<\/p>\n<p>Questions:<\/p>\n<ul>\n<li>Can we have more than one script belonging to a specific version?<\/li>\n<li>If so, what is the execution sequence?<\/li>\n<li>Can we have sub-versions (e.g. 5.1, 5.2 etc)?<\/li>\n<\/ul>\n<p>The answer is that Flyway will throw an error if it detects two scripts associated with the same version number, so in fact you MUST use sub-version numbers for multiple scripts, like this:<\/p>\n<div id=\"codeSnippetWrapper\">\n<pre id=\"codeSnippet\" style=\"border-top-style: none; font-size: 8pt; overflow: visible; border-left-style: none; font-family: 'Courier New', courier, monospace; width: 100%; border-bottom-style: none; color: black; padding-bottom: 0px; direction: ltr; text-align: left; padding-top: 0px; border-right-style: none; padding-left: 0px; margin: 0em; line-height: 12pt; padding-right: 0px; background-color: #f4f4f4\">V6_0__test_table2.sql<br>V6_1_1_test_table1.sql<br>V6_1_2__test_table3.sql<\/pre>\n<p><\/div>\n<p>Of course, this now dictates the execution sequence:<\/p>\n<div id=\"codeSnippetWrapper\">\n<pre id=\"codeSnippet\" style=\"border-top-style: none; font-size: 8pt; overflow: visible; border-left-style: none; font-family: 'Courier New', courier, monospace; width: 100%; border-bottom-style: none; color: black; padding-bottom: 0px; direction: ltr; text-align: left; padding-top: 0px; border-right-style: none; padding-left: 0px; margin: 0em; line-height: 12pt; padding-right: 0px; background-color: #f4f4f4\">$ flyway info<br>Flyway 4.2.0 by Boxfuse<br>Database: jdbc:mysql:\/\/localhost:3306\/cookbook (MySQL 5.7)<br>+---------+--------------------------+---------------------+---------+<br>| Version | Description              | Installed on        | State   |<br>+---------+--------------------------+---------------------+---------+<br>| 4       | other objects            |                     | &lt;Baseln |<br>| 5       | Baseline prior to Flyway | 2017-06-12 14:23:41 | Success |<br>| 6.0     | test table2              |                     | Pending |<br>| 6.1.1   | test table1              |                     | Pending |<br>| 6.1.2   | test table3              |                     | Pending |<br>+---------+--------------------------+---------------------+---------+<\/pre>\n<p><\/div>\n<p>Now, let&#8217;s migrate!<\/p>\n<div id=\"codeSnippetWrapper\">\n<pre id=\"codeSnippet\" style=\"border-top-style: none; font-size: 8pt; overflow: visible; border-left-style: none; font-family: 'Courier New', courier, monospace; width: 100%; border-bottom-style: none; color: black; padding-bottom: 0px; direction: ltr; text-align: left; padding-top: 0px; border-right-style: none; padding-left: 0px; margin: 0em; line-height: 12pt; padding-right: 0px; background-color: #f4f4f4\">$ flyway migrate<br>Flyway 4.2.0 by Boxfuse<br> <br>Database: jdbc:mysql:\/\/localhost:3306\/cookbook (MySQL 5.7)<br>Successfully validated 5 migrations (execution time 00:00.018s)<br>Current version of schema `cookbook`: 5<br>Migrating schema `cookbook` to version 6.0 - test table2<br>Migrating schema `cookbook` to version 6.1.1 - test table1<br>Migrating schema `cookbook` to version 6.1.2 - test table3<br>Successfully applied 3 migrations to schema `cookbook` (execution time 00:00.115s).<br>$<br> <br>mysql&gt; show tables;<br>+--------------------+<br>| Tables_in_cookbook |<br>+--------------------+<br>| limbs              |<br>| schema_version     |<br>| test1              |<br>| test2              |<br>| test3              |<br>+--------------------+<br>5 rows <span style=\"color: #0000ff\">in<\/span> set (0.00 sec)<br> <br>mysql&gt; select * from schema_version;<br>+----------------+---------+--------------------------+----------+--------------------------+-------------+--------<br>| installed_rank | version | description              | type     | <span style=\"color: #0000ff\">script<\/span>                   | checksum    | install<br>+----------------+---------+--------------------------+----------+--------------------------+-------------+--------<br>|              1 | 5       | Baseline prior to Flyway | BASELINE | Baseline prior to Flyway |        NULL | root<br>|              2 | 6.0     | test table2              | SQL      | V6_0__test_table2.sql    |   224077121 | root<br>|              3 | 6.1.1   | test table1              | SQL      | V6_1_1__test_table1.sql  |   669488585 | root<br>|              4 | 6.1.2   | test table3              | SQL      | V6_1_2__test_table3.sql  | -1574979834 | root<br>+----------------+---------+--------------------------+----------+--------------------------+-------------+--------<br>4 rows <span style=\"color: #0000ff\">in<\/span> set (0.00 sec)<br><\/pre>\n<p><\/div>\n<div>And now:<\/div>\n<\/p>\n<div id=\"codeSnippetWrapper\">\n<pre id=\"codeSnippet\" style=\"border-top-style: none; font-size: 8pt; overflow: visible; border-left-style: none; font-family: 'Courier New', courier, monospace; width: 100%; border-bottom-style: none; color: black; padding-bottom: 0px; direction: ltr; text-align: left; padding-top: 0px; border-right-style: none; padding-left: 0px; margin: 0em; line-height: 12pt; padding-right: 0px; background-color: #f4f4f4\">$ flyway info<br>Flyway 4.2.0 by Boxfuse<br>  <br>Database: jdbc:mysql:\/\/localhost:3306\/cookbook (MySQL 5.7)<br>+---------+--------------------------+---------------------+---------+<br>| Version | Description              | Installed on        | State   |<br>+---------+--------------------------+---------------------+---------+<br>| 4       | other objects            |                     | &lt;Baseln |<br>| 5       | Baseline prior to Flyway | 2017-06-12 14:23:41 | Success |<br>| 6.0     | test table2              | 2017-06-12 15:01:22 | Success |<br>| 6.1.1   | test table1              | 2017-06-12 15:01:22 | Success |<br>| 6.1.2   | test table3              | 2017-06-12 15:01:22 | Success |<br>+---------+--------------------------+---------------------+---------+<br><\/pre>\n<p><\/div>\n<div>It&#8217;s pretty cool!<\/div>\n","protected":false},"excerpt":{"rendered":"<p>For years I&#8217;ve used a database build script framework written in-house to recompile all schema objects and run unit tests. However, the actual DDL code to migrate a schema from one version to the next was always left as a hand-rolled script that is executed up-front before the normal build process. A colleague introduced me [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[],"class_list":["post-24","post","type-post","status-publish","format-standard","hentry","category-database-management","post-preview"],"_links":{"self":[{"href":"https:\/\/spacefold.com\/colin\/morethanfour\/wp-json\/wp\/v2\/posts\/24","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/spacefold.com\/colin\/morethanfour\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/spacefold.com\/colin\/morethanfour\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/spacefold.com\/colin\/morethanfour\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/spacefold.com\/colin\/morethanfour\/wp-json\/wp\/v2\/comments?post=24"}],"version-history":[{"count":0,"href":"https:\/\/spacefold.com\/colin\/morethanfour\/wp-json\/wp\/v2\/posts\/24\/revisions"}],"wp:attachment":[{"href":"https:\/\/spacefold.com\/colin\/morethanfour\/wp-json\/wp\/v2\/media?parent=24"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/spacefold.com\/colin\/morethanfour\/wp-json\/wp\/v2\/categories?post=24"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/spacefold.com\/colin\/morethanfour\/wp-json\/wp\/v2\/tags?post=24"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}