ContainerRecord should support (zero-downtime) migrations

[ronaldtse]

Each container should contain a schema_versions table, to record what migrations have been run within the Container.

For example, each of the N Company containers will have the same internal schema (say, files, messages).

When we want to migrate the files table across all company_* databases, the company_n/files table (located in the company_n database) is migrated from version X to X+1.

However, since migration of all company_n databases (from X to X+1) can be a lengthy process, we need to achieve zero-downtime with the container schema migrations.

This means that each company_1/files (e.g. version X+1) and company_2/files (e.g. version X) may be at different schema versions at any single time, and that the application should deal with this issue without hiccups.

[nattfodd]

Just for the record: this might only work for a very simple migrations (adding a column, creating/deleting a table, updating some records etc.) which can be done in a few seconds. Otherwise those on-the-go migrations might not act as expected...

For a versatile solution, I would made migrations in a form of a separate rake task (similar to rake db:migrate), and run it for all companies. Checking in ContainerRecord if the schema is actual or not, and if not - throw error until the specific company database is fully migrated (and catch this error whenever we use company databases).

We'll still achieve zero-downtime, because there's no need to restart the server to make it work. We'll just move the responsibility of migrations from the gem to the proper external raketask.

What do you think?

[ronaldtse]

@nattfodd I'm afraid that if we keep the company_n databases invalid until migration is complete will cause a lot of downtime by itself.

I'm thinking that we need a migration table per company_n database?

[nattfodd]

@ronaldtse I'm just cautious about the following case:

• there's a long running migration for "container" databases
• some code is executed, like Company.find(1).files
• ContainerRecord detects there're pending migrations and start this long-running one, keeping the connection alive (up to 60 seconds by default I guess?)
• at the same time, another request has been made, running the same code: Company.find(1).files - what should we return in that case? Keep another connection alive, looping endlessly until the migration is finished? Or throw some error?

[ronaldtse]

@nattfodd ah!

• there's a long running migration for "container" databases
• some code is executed, like Company.find(1).files
• ContainerRecord detects there're pending migrations and start this long-running one, keeping the connection alive (up to 60 seconds by default I guess?)

• Company.find(1).files should return immediately as it would have worked with a previous schema version.

[ronaldtse]

Actually, there are two types of migrations:

• Non-blocking: One that is does not break existing functionality, where the code handling ContainerRecords should be able to interact with data still at a previous schema.
• Blocking: One that involves schema changes and may not be backwards compatible.

My previous comment was premature, and applied only to the first type.

For the first type, the migration can be run anytime.

For the second type, the migration must be run before the code can use the data. However, since there are company_n databases, these databases can be migrated in parallel if we apply run migration tasks, so they can be done quickly.