Releases: peterldowns/pgmigrate
Releases · peterldowns/pgmigrate
v0.4.0+commit.54dbdfe
nit: consistently close sql rows, check for errors (#34) [Detail](https://detail.dev) reported a number of edge-case problems with the data-loading logic in internal/schema. Fundamentally, I was being lazy, and omitted some basic error checking. - Need to `defer rows.Close()` after making a query - Need to check `rows.Err()` after iterating through the rows. This PR addresses these issues consistently throughout all the data-loading code. Additionally, with newer versions of golang, the compiler is able to infer more types when calling generic functions, so I've fixed all the editor warnings related to not needing to infer the types when calling the `Sort()` method. ## Testing - CI and existing unit tests.
v0.4.0+commit.7c72d44
fix: create migrations schema if missing (#32) When `Migrator.TableName` references a non-existent schema (for example `new_schema.pgmigrate_migrations`), migration setup failed because `ensureMigrationsTable` only attempted to create the table. This change creates the schema first when needed, then creates the migrations table. I also refactored schema/table parsing into an exported helper, `pgtools.ParseTableName`, and updated migrator code to use it so this behavior is reusable in other packages. ## Testing - [x] `go test -count=1 -run TestCreateMigrationsTableInMissingSchema ./...` - [x] `go test -count=1 ./internal/pgtools`
v0.4.0+commit.25f7510
fix(cli): respect configured table name in ops subcommands (#33) Fixes a bug where `pgmigrate ops` subcommands ignored the configured `table_name` (from config/env/flag) and always operated on the default migrations table. - Updated ops subcommands to use a configured `Migrator` instead of package-level helpers: - `mark-applied` - `mark-unapplied` - `set-checksum` - `recalculate-checksum` We probably want to remove the pgmigrate package methods for running migrations and instead force callers to create an explicit migrator instance. That's a breaking change so I'll handle that in later work. ## Testing - unit tests and CI
v0.4.0+commit.8963dd6
fix: go1.24.0 Sprintf failure
v0.4.0+commit.66c998b
fix: nix build, v0.4.0 for cmd/pgmigrate and example
v0.4.0+commit.18de11f
feat: multi-schema support when parsing/dumping (#20) Customers may want to use pgmigrate's parsing/dumping logic with migrations that interact with multiple different Postgres _schemas_ within the same database. Currently, the dumping logic doesn't support this — the code assumes that you're interacting with a single schema, which it assumes by default is named `public`. This PR changes that assumption and makes it so that you can parse/dump multiple schemas within the same database. In order to make multiple-schemas work, I had to update the way the internal schema-parsing code refers to objects — now objects must be fully referenced including their schema, not just their name. This prevents conflicts between objects with the same name in different schemas, and in general opens the door to broader multi-schema support. The code needs to make sure that schemas exist, which means quoting and escaping their names correctly. Testing this led to minor backwards-incompatible changes in the general quoting/escaping logic used throughout the codebase. now, `hyphenated-object-names` will be escaped as `"hyphenated-object-names"`; previously, they would not have been escaped. I don't think this ever would have worked before, but it's fixed now. The `pgmigrate dump` configuration and its documentation needed updating to make it clear that objects must be written as fully qualified `schema.name` instead of `name`. The `schema: public` key was updated to be a list of strings. These are backwards-incompatible changes. Finally, I added the ability to add header and footer lines to the dumped SQL schema when running `pgmigrate dump`, motivated by support for functions that make queries against tables. pgmigrate always dumps function definitions in the same order at the top of the file. While we could make it possible for functions to depend on tables, this leads to very complicated manually-specified dependency trees. It's simpler to just use the new header options to set a transaction-local [check_function_bodies](https://postgresqlco.nf/doc/en/param/check_function_bodies/) parameter to let you create functions without validating their definitions. ## Testing - [x] lots of automated tests ## Breaking Changes - `.pgmigrate.yaml` config format - quoting/escaping logic now escapes `hyphenated-object-names`, previously didn't. ## Followups I'll have to tag versions `v0.4.0` and `cmd/pgmigrate/v0.4.0` manually after this is merged. That should fix all the build stuff.
v0.3.1+commit.d3ecf8e
feat: add ls as alias for list (#27) This PR makes it possible to run `pgmigrate ls` as an alias for `pgmigrate list`, which is already an alias for `pgmigrate applied`. Solves a long-standing pet peeve of mine, I think all CLIs should let `ls` == `list`. ## Tests I manually built and ran `pgmigrate ls` and `pgmigrate list` and saw that they did the same thing. We don't have any tests on the CLI.
v0.3.1+commit.80ada56
version(cli,example): v0.3.1
v0.3.1+commit.3f7b24c
feat: spin when acquiring session locks (#26) This PR makes it so that pgmigrate uses a spinlock approach to acquire the session (Postgres advisory) lock before applying migrations. This fixes https://github.com/peterldowns/pgmigrate/issues/25. ## Background When applications run migrations at startup, it's good to configure the database's connection and lock timeout statements to prevent your application from hanging indefinitely. Long-running migrations can starve out existing traffic and cause user-facing service disruptions. Postgres allows clients to [configure various timeouts](https://www.postgresql.org/docs/current/runtime-config-client.html) so that if certain types of query behavior take longer than expected, the query will be failed rather than hanging indefinitely. The most relevant timeouts are: * `lock_timeout`: the maximum amount of time a query can wait to acquire a lock. * `statement_timeout`: the maximum amount of time a single statement can run. * `transaction_timeout`: the maximum amount of time a single transaction can run. pgmigrate runs each migration within a transaction, so clients may use `transaction_timeout` to specify the maximum amount of time for which a migration should run before it should be considered failed. In some cases, clients may want to explicitly set the `lock_timeout` or `statement_timeout` as well, for more finegrained control. ## Problem pgmigrate uses both in-memory locks and Postgres session/advisory locks in order to guarantee that no more than a single instance of pgmigrate is running migrations at any point time. The goal is to allow applications to always call `Migrate()` at startup — or, equivalently, to run migrations as a kubernetes init container, a pre-launch script, or whatever. If there are multiple app instances starting up at the same time, only one of them should be running migrations; the others should wait until the migrations have finished running, then start up without performing any other work. This is an essential feature that makes pgmigrate quite nice to use in the real world! Unfortunately, pgmigrate was using `pg_advisory_lock` to acquire the Postgres session/advisory lock. If this lock is already held, then the statement will block indefinitely until that other holder releases the lock. That's the expected behavior, but this doesn't play nicely with `lock_timeout` and `statement_timeout` — those timeouts apply to this library's session/advisory lock acquisition, too, not just the locks within the client's migrations! So, this leads to a problem during normal operations: - Your application runs migrations at startup - You have multiple instances of your application running/launching at once - You have a statement_timeout or lock_timeout of some maximum time - You add a new migration that takes longer than statement_timeout or lock_timeout - You deploy your application and see lots of crashloops because pgmigrate takes longer than the timeout to acquire its own internal session lock, about which you shouldn't need to care. ## Solution This PR solves the problem by spinning using `pg_try_advisory_lock` to wait indefinitely to acquire the session lock. This query returns immediately, and tells you whether or not you've acquired the lock. If pgmigrate did not acquire the lock, it sleeps for `sessionlock.SpinWait` (100ms) and then tries again, indefinitely, until it can acquire the lock. This means that pgmigrate's lock acquisition logic is now only affected by the `idle_session_timeout` and the `statement_timeout` settings, which is true of literally any connection and can't be worked around. Clients will need to make sure their `idle_session_timeout` is greater than the spin-wait here (100ms), which is reasonable. As the docs say, > Unlike the case with an open transaction, an idle session without a transaction imposes no large costs on the server, so there is less need to enable this timeout than `idle_in_transaction_session_timeout`. > > Be wary of enforcing this timeout on connections made through connection-pooling software or other middleware, as such a layer may not react well to unexpected connection closure. It may be helpful to enable this timeout only for interactive sessions, perhaps by applying it only to particular users. ## Testing I manually tested these changes with a locally running database, two concurrent processes running `pgmigrate apply`, and connection string parameters of `?idle_session_timeout=200&lock_timeout=100&transaction_timeout=6001&statement_timeout=6001&sslmode=disable"`. The migrations being applied had a few that explicitly called `SELECT pg_sleep(5);` to sleep for 5 seconds. The result was as expected: with this PR, the migrations succeeded in one process. The second process waited until that happened, then exited successfully. I also added an automated test to prove to myself that the spinlock works. The concurrency logic is a little hairy, but there are plenty of comments in the code. I checked and confirmed that the previous `sessionlock.With` logic causes this test to fail. ## Followups - It might be nice to allow clients to configure the `SpinWait` duration. - Clients may also want to be able to set a maximum time to wait before giving up on lock acquisition. - Although it can be done by setting a deadline on the context passed to pgmigrate, CLI users may want a way to set a total deadline on all pgmigrate operations.
v0.3.1+commit.1360af2
fix: flake (vendorHash)