Skip to content

Releases: peterldowns/pgmigrate

v0.4.0+commit.54dbdfe

24 Feb 15:33
54dbdfe

Choose a tag to compare

nit: consistently close sql rows, check for errors (#34)

[Detail](https://detail.dev) reported a number of edge-case problems
with the data-loading logic in internal/schema. Fundamentally, I was
being lazy, and omitted some basic error checking.

- Need to `defer rows.Close()` after making a query
- Need to check `rows.Err()` after iterating through the rows.

This PR addresses these issues consistently throughout all the
data-loading code.

Additionally, with newer versions of golang, the compiler is able to
infer more
types when calling generic functions, so I've fixed all the editor
warnings
related to not needing to infer the types when calling the `Sort()`
method.

## Testing
- CI and existing unit tests.

v0.4.0+commit.7c72d44

23 Feb 15:30
7c72d44

Choose a tag to compare

fix: create migrations schema if missing (#32)

When `Migrator.TableName` references a non-existent schema (for example
`new_schema.pgmigrate_migrations`), migration setup failed because
`ensureMigrationsTable` only attempted to create the table. This change
creates the schema first when needed, then creates the migrations table.

I also refactored schema/table parsing into an exported helper,
`pgtools.ParseTableName`, and updated migrator code to use it so this
behavior is reusable in other packages.

## Testing
- [x] `go test -count=1 -run TestCreateMigrationsTableInMissingSchema
./...`
- [x] `go test -count=1 ./internal/pgtools`

v0.4.0+commit.25f7510

23 Feb 15:46
25f7510

Choose a tag to compare

fix(cli): respect configured table name in ops subcommands (#33)

Fixes a bug where `pgmigrate ops` subcommands ignored the configured
`table_name` (from config/env/flag) and always operated on the default
migrations table.

- Updated ops subcommands to use a configured `Migrator` instead of
package-level helpers:
  - `mark-applied`
  - `mark-unapplied`
  - `set-checksum`
  - `recalculate-checksum`

We probably want to remove the pgmigrate package methods for running
migrations and instead force callers to create an explicit migrator
instance. That's a breaking change so I'll handle that in later work.

## Testing
- unit tests and CI

v0.4.0+commit.8963dd6

20 Oct 18:00
8963dd6

Choose a tag to compare

fix: go1.24.0 Sprintf failure

v0.4.0+commit.66c998b

20 Oct 17:33
66c998b

Choose a tag to compare

fix: nix build, v0.4.0 for cmd/pgmigrate and example

v0.4.0+commit.18de11f

20 Oct 17:25
18de11f

Choose a tag to compare

feat: multi-schema support when parsing/dumping (#20)

Customers may want to use pgmigrate's parsing/dumping logic with
migrations that interact with multiple different Postgres _schemas_
within the same database. Currently, the dumping logic doesn't support
this — the code assumes that you're interacting with a single schema,
which it assumes by default is named `public`. This PR changes that
assumption and makes it so that you can parse/dump multiple schemas
within the same database.

In order to make multiple-schemas work, I had to update the way the
internal schema-parsing code refers to objects — now objects must be
fully referenced including their schema, not just their name. This
prevents conflicts between objects with the same name in different
schemas, and in general opens the door to broader multi-schema support.

The code needs to make sure that schemas exist, which means quoting and
escaping their names correctly. Testing this led to minor
backwards-incompatible changes in the general quoting/escaping logic
used throughout the codebase. now, `hyphenated-object-names` will be
escaped as `"hyphenated-object-names"`; previously, they would not have
been escaped. I don't think this ever would have worked before, but it's
fixed now.

The `pgmigrate dump` configuration and its documentation needed updating
to make it clear that objects must be written as fully qualified
`schema.name` instead of `name`. The `schema: public` key was updated to
be a list of strings. These are backwards-incompatible changes.

Finally, I added the ability to add header and footer lines to the
dumped SQL schema when running `pgmigrate dump`, motivated by support
for functions that make queries against tables. pgmigrate always dumps
function definitions in the same order at the top of the file. While we
could make it possible for functions to depend on tables, this leads to
very complicated manually-specified dependency trees. It's simpler to
just use the new header options to set a transaction-local
[check_function_bodies](https://postgresqlco.nf/doc/en/param/check_function_bodies/)
parameter to let you create functions without validating their
definitions.

## Testing
- [x] lots of automated tests

## Breaking Changes
- `.pgmigrate.yaml` config format
- quoting/escaping logic now escapes `hyphenated-object-names`,
previously didn't.

## Followups
I'll have to tag versions `v0.4.0` and `cmd/pgmigrate/v0.4.0` manually
after this is merged. That should fix all the build stuff.

v0.3.1+commit.d3ecf8e

29 Jul 16:18
d3ecf8e

Choose a tag to compare

feat: add ls as alias for list (#27)

This PR makes it possible to run `pgmigrate ls` as an alias for
`pgmigrate list`, which is already an alias for `pgmigrate applied`.
Solves a long-standing pet peeve of mine, I think all CLIs should let
`ls` == `list`.

## Tests
I manually built and ran `pgmigrate ls` and `pgmigrate list` and saw
that they did the same thing. We don't have any tests on the CLI.

v0.3.1+commit.80ada56

15 Jul 19:56
80ada56

Choose a tag to compare

version(cli,example): v0.3.1

v0.3.1+commit.3f7b24c

15 Jul 19:51
3f7b24c

Choose a tag to compare

feat: spin when acquiring session locks (#26)

This PR makes it so that pgmigrate uses a spinlock approach to acquire
the session (Postgres advisory) lock before applying migrations. This
fixes https://github.com/peterldowns/pgmigrate/issues/25.

## Background
When applications run migrations at startup, it's good to configure the
database's connection and lock timeout statements to prevent your
application from hanging indefinitely. Long-running migrations can
starve out existing traffic and cause user-facing service disruptions.
Postgres allows clients to [configure various
timeouts](https://www.postgresql.org/docs/current/runtime-config-client.html)
so that if certain types of query behavior take longer than expected,
the query will be failed rather than hanging indefinitely. The most
relevant timeouts are:

* `lock_timeout`: the maximum amount of time a query can wait to acquire
a lock.
* `statement_timeout`: the maximum amount of time a single statement can
run.
* `transaction_timeout`: the maximum amount of time a single transaction
can run.

pgmigrate runs each migration within a transaction, so clients may use
`transaction_timeout` to specify the maximum amount of time for which a
migration should run before it should be considered failed. In some
cases, clients may want to explicitly set the `lock_timeout` or
`statement_timeout` as well, for more finegrained control.

## Problem
pgmigrate uses both in-memory locks and Postgres session/advisory locks
in order to guarantee that no more than a single instance of pgmigrate
is running migrations at any point time. The goal is to allow
applications to always call `Migrate()` at startup — or, equivalently,
to run migrations as a kubernetes init container, a pre-launch script,
or whatever. If there are multiple app instances starting up at the same
time, only one of them should be running migrations; the others should
wait until the migrations have finished running, then start up without
performing any other work.

This is an essential feature that makes pgmigrate quite nice to use in
the real world!

Unfortunately, pgmigrate was using `pg_advisory_lock` to acquire the
Postgres session/advisory lock. If this lock is already held, then the
statement will block indefinitely until that other holder releases the
lock. That's the expected behavior, but this doesn't play nicely with
`lock_timeout` and `statement_timeout` — those timeouts apply to this
library's session/advisory lock acquisition, too, not just the locks
within the client's migrations!

So, this leads to a problem during normal operations:

- Your application runs migrations at startup
- You have multiple instances of your application running/launching at
once
- You have a statement_timeout or lock_timeout of some maximum time
- You add a new migration that takes longer than statement_timeout or
lock_timeout
- You deploy your application and see lots of crashloops because
pgmigrate takes longer than the timeout to acquire its own internal
session lock, about which you shouldn't need to care.

## Solution
This PR solves the problem by spinning using `pg_try_advisory_lock` to
wait indefinitely to acquire the session lock. This query returns
immediately, and tells you whether or not you've acquired the lock. If
pgmigrate did not acquire the lock, it sleeps for `sessionlock.SpinWait`
(100ms) and then tries again, indefinitely, until it can acquire the
lock.

This means that pgmigrate's lock acquisition logic is now only affected
by the `idle_session_timeout` and the `statement_timeout` settings,
which is true of literally any connection and can't be worked around.
Clients will need to make sure their `idle_session_timeout` is greater
than the spin-wait here (100ms), which is reasonable. As the docs say,

> Unlike the case with an open transaction, an idle session without a
transaction imposes no large costs on the server, so there is less need
to enable this timeout than `idle_in_transaction_session_timeout`.
>
> Be wary of enforcing this timeout on connections made through
connection-pooling software or other middleware, as such a layer may not
react well to unexpected connection closure. It may be helpful to enable
this timeout only for interactive sessions, perhaps by applying it only
to particular users.

## Testing

I manually tested these changes with a locally running database, two
concurrent processes running `pgmigrate apply`, and connection string
parameters of
`?idle_session_timeout=200&lock_timeout=100&transaction_timeout=6001&statement_timeout=6001&sslmode=disable"`.
The migrations being applied had a few that explicitly called `SELECT
pg_sleep(5);` to sleep for 5 seconds. The result was as expected: with
this PR, the migrations succeeded in one process. The second process
waited until that happened, then exited successfully.

I also added an automated test to prove to myself that the spinlock
works. The concurrency logic is a little hairy, but there are plenty of
comments in the code. I checked and confirmed that the previous
`sessionlock.With` logic causes this test to fail.

## Followups
- It might be nice to allow clients to configure the `SpinWait`
duration.
- Clients may also want to be able to set a maximum time to wait before
giving up on lock acquisition.
- Although it can be done by setting a deadline on the context passed to
pgmigrate, CLI users may want a way to set a total deadline on all
pgmigrate operations.

v0.3.1+commit.1360af2

15 Jul 20:56
1360af2

Choose a tag to compare

fix: flake (vendorHash)