Add dbtool GUI plus diff/exec/fold and broader migration tooling#474
Draft
christianparpart wants to merge 46 commits intomasterfrom
Draft
Add dbtool GUI plus diff/exec/fold and broader migration tooling#474christianparpart wants to merge 46 commits intomasterfrom
christianparpart wants to merge 46 commits intomasterfrom
Conversation
github-actions
Bot
added
documentation
christianparpart
force-pushed
the
feature/migrations-gui
branch
from
6e1de5a to
d1c22d7
Compare
christianparpart
force-pushed
the
feature/migrations-gui
branch
5 times, most recently
from
103a585 to
156d6a7
Compare
christianparpart
changed the title
christianparpart
force-pushed
the
feature/migrations-gui
branch
3 times, most recently
from
69c4852 to
2fcf48c
Compare
christianparpart
force-pushed
the
feature/migrations-gui
branch
from
2fcf48c to
020e68f
Compare
christianparpart
force-pushed
the
feature/migrations-gui
branch
from
020e68f to
d9f9a16
Compare
christianparpart
force-pushed
the
feature/migrations-gui
branch
2 times, most recently
from
49cd796 to
301326c
Compare
christianparpart
force-pushed
the
feature/migrations-gui
branch
2 times, most recently
from
abcded1 to
8860114
Compare
Adds `lup2dbtool`, a CLI that ingests Lastrada's `init_m_*.sql` /
`upd_m_*.sql` corpus and emits one C++ file per migration, ready to
be compiled into a Lightweight migration plugin that dbtool dlopens
at runtime.
The implementation grew incrementally as the LUP corpus exposed
dialect quirks; what's bundled here is the consolidated end state:
- LupVersionConverter: parses `init_m_<M>_<m>_<p>.sql` /
`upd_m_<M>_<m>_<p>[__<…>].sql` filenames, derives the migration
timestamp (`20000000<MMmmpp>`) and the dotted release string used
by the `LIGHTWEIGHT_SQL_RELEASE` markers.
- LupSqlParser: top-level statement splitter that handles the corpus's
windows-1252 encoding, missing trailing semicolons, sybase
`print '…'` directives, and multi-line statements.
- SqlStatementParser: structured parser for CREATE TABLE / ALTER TABLE
ADD COLUMN / ADD/DROP FOREIGN KEY / CREATE INDEX / DROP TABLE /
INSERT / UPDATE / DELETE shapes actually seen in the corpus. UPDATE
recognises `IS [NOT] NULL`, simple `(col op val)` triples, and
composite WHEREs (AND/OR/NOT/EXISTS/IN) routed to the
WhereClauseParser. Paren-aware top-level WHERE detection so an
inner `WHERE` inside a `NOT EXISTS (SELECT … WHERE …)` does not
get mis-split into the SET clause.
- WhereClauseParser: small recursive-descent parser that renders the
WHERE body into a canonical, dialect-portable string with every
identifier double-quoted and uppercased.
- CodeGenerator: emits the Lightweight DSL calls. Always uppercases
identifiers (tables, columns, FK targets, index columns, SET targets,
WHERE columns, plus barewords inside expression bodies). Emits
idempotent `AddColumnIfNotExists` / `AddNotRequiredColumnIfNotExists`
unconditionally so overlapping LUP releases (e.g. 4_7_6 vs 5_0_0)
apply linearly. Detects literal-vs-expression UPDATE SET RHSs and
routes the latter through `SetExpression` with bareword
identifiers re-quoted. Prefixes index names with their table
(`<TABLE>_<index>`) to dodge SQLite/PG global-index-namespace
collisions; strips `ASC/DESC` from index column lists.
- main: argument parsing, file iteration, multi-file output with
`{version}` substitution, optional CMake/Plugin emission, the
`--max-lines-per-file` split for the giant `2_3_6` migration, and
the `--force-unicode` / `--varchar-scale N` knobs that widen
parameterised character columns (NVARCHAR for MSSQL,
scale-multiplied size for PG which counts characters not bytes).
- StringUtils: shared `Trim`, `ToUpper`, `RemoveQuotes`,
`NormalizeWhitespace`, `IsSqlReservedWord`, and
`CanonicalizeIdentifiersInSql` helpers — the last is the bareword-
to-quoted-uppercase rewriter used by both the WHERE-clause parser
and the codegen for embedded SQL fragments.
Tests in Lup2DbtoolTests cover the version parser, every major SQL
shape the parser recognises, the codegen mappings (including the
forceUnicode and varcharScale variants), and the WHERE canonicaliser.
Signed-off-by: Christian Parpart <christian@parpart.family>
Adds infrastructure for plugins to scope compat behaviour to their own migration timestamp range, plus the actual `lup-truncate` render path used by LUP legacy migrations. - `MigrationManager::SetCompatPolicy` / `ComposeCompatPolicy` / `CompatFlagsFor`: a per-migration `CompatPolicy` callback returns the set of compat flags active for a given migration. Composition unions flag sets so multiple plugins can contribute policies in the same process. - `MigrationRenderContext` (in `SqlQuery/MigrationPlan.hpp`): mutable per-run state threaded through a new context-aware `ToSql` overload. Carries a `(schema, table, column) -> ColumnWidth` cache populated from `CreateTable`/`AlterTable` plan steps and an optional lazy `widthLookup` callback for tables created by earlier runs. - `lup-truncate` renderer: clips oversize INSERT/UPDATE string values to the destination column's declared width, with UTF-8 byte-vs-char unit awareness so MSSQL `varchar(N)` (bytes) and `nvarchar(N)` (chars) are both honoured. Each truncation is logged via `SqlLogger::OnWarning` — silent in LUpd, deliberately not silent here so the data loss stays auditable. - `ApplySingleMigration` / `PreviewMigration` get context-aware overloads; `ApplyPendingMigrations` / `PreviewPendingMigrations` share one context across the whole sequence so an INSERT in migration N+k can still see widths declared by a CREATE TABLE in migration N. - `MigrationManager::RewriteChecksums(dryRun)`: re-stamps `schema_migrations.checksum` rows that have drifted vs. the current `MigrationBase::ComputeChecksum` output. Recovery primitive used after a regen changes byte shape but not logical effect. Tests cover: strict mode leaves values intact, lup-truncate clips and warns, UTF-8 multi-byte char counting, `string_view` (char-array literal) input, DROP TABLE evicting cache entries, and policy composition unioning flag sets per migration. Signed-off-by: Christian Parpart <christian@parpart.family>
…t-out Flips the generator's `forceUnicode` default to `true` so emitted DSL uses `NCHAR`/`NVARCHAR` everywhere. This is the right default: MSSQL gets char-counted widths so multi-byte source data (German umlauts) no longer overflows byte-counted budgets, and SQLite/PostgreSQL formatters downgrade back to `CHAR`/`VARCHAR` so it is a semantic no-op there. `--force-unicode` remains accepted (no-op) for backwards compatibility; `--no-force-unicode` is the new opt-out for callers that need the legacy narrow types. Tests updated accordingly. Signed-off-by: Christian Parpart <christian@parpart.family>
Registers a `CompatPolicy` on the singleton `MigrationManager` at plugin load time: every LUP migration with timestamp strictly less than the first 6.0.0 migration (`20'000'000'060'000`) is rendered with `lup-truncate` active; everything at or above that cutoff gets strict, standards-compliant behaviour. The threshold is intentionally hard-coded here, not exposed as a runtime knob — compat scope is a property of the legacy *code*, not of the deployment, so operators should never need to configure it. Signed-off-by: Christian Parpart <christian@parpart.family>
`CollectMigrations` now propagates each plugin's installed `CompatPolicy` onto the central manager via `ComposeCompatPolicy`, so multiple plugins can contribute their own policies in the same dbtool process. Signed-off-by: Christian Parpart <christian@parpart.family>
Adds a `rewrite-checksums` admin command that prints the drift diff in dry-run mode and requires explicit `--yes` to write. Used after a regen of generated migrations changes byte shape but not logical effect (the Unicode-default flip in lup2dbtool is the canonical case). Adds a `--yes` / `-y` flag to confirm destructive actions. Signed-off-by: Christian Parpart <christian@parpart.family>
- Section 1: Unicode-default generator is now the lup2dbtool default rather than a per-CMake flag. - Section 2: compat policy moved to plugin-side ownership rather than per-profile YAML config — the LUP plugin installs a timestamp-keyed policy on its own `MigrationManager` singleton, and `dbtool` composes it onto the central manager via `CollectMigrations`. The operator never sees this knob. - Section 3: documents the shipped `dbtool rewrite-checksums` recovery tool and the verification run against the staging MSSQL database (rewrote 150 drifted checksums; the migrate run proceeds past the 4_07_05 boundary that originally blocked it). - Out-of-scope: defers logical-equivalence checksum hashes and documents that Unicode-regen FK shape mismatches need per-database remediation, not a generic compat flag. Signed-off-by: Christian Parpart <christian@parpart.family>
Extracts the line-budget bin-packing and chunked file emission logic
into src/Lightweight/CodeGen/SplitFileWriter.{hpp,cpp}, plus an
EmitPluginCmake helper that drops a CMakeLists.txt and Plugin.cpp
suitable for a drop-in migration plugin.
lup2dbtool's GroupBlocksByLineBudget becomes a thin wrapper around
the shared helper, so the existing split-on-large-migration behaviour
stays exactly the same. The new module is also picked up by the fold
emitter in a follow-up commit.
Signed-off-by: Christian Parpart <christian@parpart.family>
The N'...' prefix tells SQL Server to *store* a literal as Unicode but
does not change how the bytes between the quotes are decoded — those
go through the connection's narrow code page (CP-1252 by default). A
UTF-8 byte pair like 0xC3 0xBC ('ü') was decoded as two separate
Latin-1 characters ü, both garbling the stored value and inflating
the perceived character count. Legitimate 100-codepoint German strings
overflowed NVARCHAR(100) with 'String or binary data would be
truncated' (error 2628), even after lup-truncate clipped them to the
column's declared width.
Override SqlServerQueryFormatter::StringLiteral to UTF-8-decode
client-side and emit each non-ASCII codepoint as an NCHAR(N) call
glued onto the surrounding ASCII runs with `+`:
"für" -> N'f' + NCHAR(252) + N'r'
ASCII fast-paths through unchanged. BMP non-ASCII becomes one NCHAR().
Supplementary-plane codepoints (>U+FFFF) become a UTF-16 surrogate
pair (two NCHAR() calls). Per-codepoint encoding lives in three small
private statics so escaping rules stay in one place.
Tests cover ASCII runs, empty literals, single-quote escaping, BMP
non-ASCII, pure-non-ASCII inputs, the single-char overload, and a
sanity check that SQLite/PostgreSQL still emit plain '...' literals.
The existing UTF-8-multibyte lup-truncate test asserts on the new
NCHAR-concatenated emission. Migration Insert/Update expectations
updated to use N'...' literals on MSSQL.
Verified end-to-end on lastrada-mssql: PROBEN_PRUEFUNGEN row 1008
(106-codepoint German content clipped to 100) now stores correctly
and round-trips German umlauts intact.
Signed-off-by: Christian Parpart <christian@parpart.family>
Adds a pure plan-walk primitive,
MigrationManager::FoldRegisteredMigrations(formatter, upToInclusive),
that folds the effect of every registered migration into a per-table
view of the final shape plus a chronological list of data steps,
indexes, and releases. Never executes SQL or opens a database
connection.
Used by:
- dbtool fold (extracted to feature/dbtool-fold) — emits a
self-contained baseline (.cpp plugin or .sql script).
- hard-reset (added in a follow-up commit) — to know which tables
the migrations would have created.
- unicode-upgrade-tables (added in a follow-up commit) — to know
which char/varchar columns the migrations now declare wide.
Also adds a `RunAdminCommand<Result>` template in dbtool that factors
the shared dry-run / --yes / diff UX, and rewrites `rewrite-checksums`
on top of it for net-negative lines. The two upcoming admin commands
will reuse this template.
Tests: fold unit tests cover create/altercolumn/drop-table cleanup,
data-step chronological order, --up-to truncation, RawSql passthrough,
column rename FK propagation, release-range filtering.
Signed-off-by: Christian Parpart <christian@parpart.family>
Adds an admin command that drops every migration-owned table (preserves user-created tables) plus schema_migrations, in reverse creation order with cascade=true ifExists=true. Pair with `migrate` for a clean re-deploy. The implementation walks `MigrationManager::FoldRegisteredMigrations` to compute the migration-owned set, intersects with the live schema via `SqlSchema::ReadAllTables`, then drops the matching live tables inside a single transaction. Tables present in the live DB but not in the migration plan are reported under `preservedTables` so operators spot them. Wired into dbtool through the shared `RunAdminCommand<Result>` template introduced earlier — dry-run prints the diff, `--yes` confirms the destructive action. Tests: SQLite end-to-end coverage for dropping migrated tables + schema_migrations, preserving user tables, and dry-run being observationally pure. Signed-off-by: Christian Parpart <christian@parpart.family>
Adds an admin command that rewrites legacy VARCHAR/CHAR columns to NVARCHAR/NCHAR where the registered migrations now declare wide types. Drops + re-adds touched FKs, with a SQLite-specific path via `RebuildSqliteTable` for in-place column-type rewrite. Compares the folded plan's intended column types against `SqlSchema::ReadAllTables` output; an upgrade is triggered iff intended is `NVarchar`/`NChar` AND live is `Varchar`/`Char` with the same `size`. Foreign keys touching any upgrade column are dropped before the alter and re-added afterwards. Cross-backend. Wired into dbtool through the shared `RunAdminCommand<Result>` template — dry-run prints the diff, `--yes` confirms the destructive action. Tests: SQLite coverage for dry-run drift reporting plus an idempotent roundtrip (running unicode-upgrade-tables twice in a row produces no second-run drift). Signed-off-by: Christian Parpart <christian@parpart.family>
The original cutoff at the first 6.0.0 timestamp assumed post-6.0.0 LUP SQL files would be cleaned up before generation. They aren't: upd_m_6_01_12.sql writes a 61-char value into TEXTBAUSTEINGRUPPEN.NAME which is NCHAR(60), and similar over-long inserts appear elsewhere in the 6.x.x range. LUpd silently client-side-truncated those historically; without lup-truncate active for these migrations, MSSQL strict NCHAR sizing rejects them with error 2628. Raise the cutoff to 20'000'070'000'000 so every LUP-sourced migration (timestamp range < 7.00.00) gets the legacy truncation policy. The 9999_99_99 sentinel and any future native modern migrations remain strict. Comment block updated to record the rationale and the concrete case. Signed-off-by: Christian Parpart <christian@parpart.family>
Thin diagnostics helper for inspecting INFORMATION_SCHEMA, sanity- checking row counts, or running one-off queries from CI / shell scripts. Streams the result set as tab-separated values to stdout; row count goes to stderr. The query is read from the command-line argument or, when no argument is supplied (or `-` is passed), from stdin until EOF. Multi-statement scripts pass through to ExecuteDirect — the ODBC driver advances through subsequent result sets automatically. dbtool --profile X exec "SELECT * FROM INFORMATION_SCHEMA.COLUMNS" echo "SELECT COUNT(*) FROM users" | dbtool --profile X exec Used during the MSSQL UTF-8 literal investigation to confirm that MSSQL was decoding raw `N'für'` as four CP-1252 codepoints rather than three Unicode codepoints, which led to the NCHAR-concatenation fix shipped in the previous commit. Signed-off-by: Christian Parpart <christian@parpart.family>
The Examples block bloated the bottom of `dbtool --help`, pushing the options listing off the screen. Hoist it behind a dedicated `--show-examples` flag and leave a one-line pointer at the end of help output so discoverability is preserved without forcing the whole example wall on every `--help` reader. Signed-off-by: Christian Parpart <christian@parpart.family>
Ship a Qt 6 / QML desktop GUI for dbtool's migration workflow alongside
the CLI. Users get a connection-aware view of pending vs applied
migrations, a release-grouped timeline, one-click apply / rollback /
backup, and an SQL preview before any destructive action.
Architecture:
- AppController is the top-level QML-exposed model: owns the
SqlConnection, drives MigrationRunner / BackupRunner workers off the
GUI thread, and exposes Q_PROPERTY surfaces consumed by the QML side.
Persists profile selection, view-mode (Simple vs Expert) and window
geometry via QSettings.
- MigrationRunner / BackupRunner are QObject workers running on
dedicated QThread instances. Both forward structured progress and
log lines via Qt signals so the UI stays responsive during long
operations. MigrationRunner consumes the structured
MigrationException surface (timestamp, step index, driver message,
failed SQL) for actionable failure reporting.
- QmlProgressManager bridges Lightweight's IProgressManager interface
to QML, mirroring the StandardProgressManager used by the CLI.
- Models (MigrationListModel, ReleaseListModel, ProfileListModel,
OdbcDataSourceListModel) expose typed list interfaces for the QML
views; each is backed by domain data from Lightweight + ProfileStore
+ DataSourceEnumerator.
- ThemeController owns palette + accent state and persists theme
preference via QSettings. The Theme.qml singleton consumes it.
Views:
- Simple view: single centred column (connection -> status -> run ->
progress -> success/failure) targeting downstream operators who just
want "bring my DB up to date". Status card surfaces current/target
release labels in large type, with a green up-to-date pill when no
work is pending. Run card offers one primary button plus a
"Back up first" checkbox that chains backup -> apply.
- Expert view: full three-pane timeline with per-migration controls,
bulk operations, release groupings, log panel, SQL preview, and
backup/restore dialog.
- ToolBar exposes a single Simple/Expert toggle button labelled with
the destination view ("Switch to Expert view" / "Switch to Simple
view"). FontMetrics reserves width for the longer label so the
toolbar layout never reflows on toggle.
- Per-view window sizing: each view stores its own preferred window
geometry so toggling does not yank the user into a layout that
doesn't fit their content.
Build:
- cmake/FindQt.cmake locates a Qt 6 install, top-level CMakeLists
optionally descends into the GUI when Qt is found, and
src/tools/CMakeLists.txt only registers the migrations-gui subdir
when the optional dependency is satisfied. The CLI build is
unaffected when Qt is absent.
Docs:
- docs/migrations-gui-plan.md captures the design rationale.
- docs/migrations-gui-mockup.html is the static mockup the QML layout
was derived from.
Signed-off-by: Christian Parpart <christian@parpart.family>
A single transaction over hundreds of tables exhausts Postgres's per-transaction lock pool: each `DROP TABLE ... CASCADE` takes an AccessExclusiveLock (plus more from CASCADE-triggered drops), capped by `max_locks_per_transaction * (max_connections + max_prepared_transactions)`. Resetting a 678-table schema surfaced as `ERROR: out of shared memory` (SQLSTATE 53200). Drop in chunks of 32 via `std::views::chunk`, committing each batch so locks are released between batches. `schema_migrations` gets its own small transaction. Drops are idempotent (`IF EXISTS`) and ordered in reverse creation order, so a partial-failure re-run still completes. Signed-off-by: Christian Parpart <christian@parpart.family>
Switches the `status` summary to a uniform 26-char label column so "Registered/Applied/Pending migrations:", "Unknown applied:" and the "Latest applied/available release:" lines all line up the same way. Also surfaces the highest declared release alongside the latest applied one, making "are we caught up?" answerable at a glance. Signed-off-by: Christian Parpart <christian@parpart.family>
…w status wording `dbtool status` now prints "Latest applied release:" instead of the old "Latest release:". Bring the WriteReleaseMarker doxygen example in line so future readers do not chase a label that no longer exists. Signed-off-by: Christian Parpart <christian@parpart.family>
Replaces the single global --input-encoding switch (which silently double-encoded whichever group of files disagreed with the flag) with per-file detection. New default mode is `auto`: each file is classified by validating its SQL payload (with `--` and `/* … */` comments stripped from the detection signal so author notes in legacy banners do not skew the result). Files that validate as UTF-8 are passed through unchanged; the rest are converted from Windows-1252. The explicit `utf-8` and `windows-1252` modes now act as assertions: a mismatch is reported as a per-file error (with the offending byte and its offset) and propagates to a non-zero exit code, so a stale flag in the build cannot quietly mangle a UTF-8 file or vice versa. The plugin CMake default is flipped to `auto` to match. ParseSqlFile now returns std::expected<ParsedMigration, std::string>; the lup2dbtool driver collects per-file errors, emits all of them, and still exits non-zero if any file failed even when the surviving inputs generated cleanly. Tests cover auto-mode pass-through of UTF-8, auto-mode conversion of Windows-1252, the strict-mode rejections in both directions, and the "non-ASCII only inside comments" case that motivated the comment-stripped detection signal. Signed-off-by: Christian Parpart <christian@parpart.family>
On a fresh checkout (or after a clean of the generated directory) the configure-time `file(GLOB lup_*.cpp)` produced no inputs, the plugin was linked with an empty placeholder, and dbtool reported "nothing to migrate" against an empty database — even though hundreds of SQL migrations were sitting in LUPMIGRATION_SQL_DIR. Reconfiguring after the first build was the documented escape hatch but a real footgun. Configure-time bootstrap now drives lup2dbtool synchronously when the generation stamp is missing: the tool is built, located in the build tree, and invoked with the same arguments as the build-time custom command, so `lup_*.cpp` files exist before the glob runs. A sentinel file guards against the re-entrant `cmake --regenerate-during-build` pass that ninja issues before each build (without it the inner configure would recurse into `cmake --build`, deadlocking the tree). The placeholder source and the post-bootstrap reconfigure step are gone. The glob now uses CONFIGURE_DEPENDS, so adding a new SQL release (or splitter changes that produce extra `_partNN` files) lights up the new sources on the next build without a manual re-run of cmake. Signed-off-by: Christian Parpart <christian@parpart.family>
…uild is mid-configure
The previous cold-start bootstrap called `cmake --build ${CMAKE_BINARY_DIR}
--target lup2dbtool` from inside the parent's first configure, but `build.ninja`
is only emitted after configure finishes — so on a true fresh build directory
ninja aborted with "loading 'build.ninja': The system cannot find the file
specified" and the configure failed before any migrations could be generated.
Detect whether the parent has a usable build script (`build.ninja` or
`Makefile`) and split the path:
* Warm cold-start (parent build present, only `generated/` wiped) keeps the
original `cmake --build ${CMAKE_BINARY_DIR}` invocation. The binary search
now also covers `CMAKE_RUNTIME_OUTPUT_DIRECTORY` and per-config subdirs so
Windows (where DLLs/EXEs land in `target/`) and multi-config generators
resolve the freshly-built binary.
* True cold-start spins up a self-contained scratch sub-build at
`${CMAKE_BINARY_DIR}/_lup_bootstrap`. The scratch configure inherits the
toolchain file, compilers, generator, build type, vcpkg triplet, and make
program from the parent, but disables every optional component (tests,
examples, GUI, benchmark, large-db tool, docs, clang-tidy, pedantic-werror)
and clears LUPMIGRATION_SQL_DIR to prevent the bootstrap from recursing
into itself. The scratch dir persists so subsequent cold-starts (after
`rm -rf generated/`) are incremental.
Verified end-to-end against `D:/Lastrada/src/model4_JP` (408 SQL files): fresh
configure on an empty build dir generates all 408 migrations and produces
`build.ninja`, a follow-up wipe of `generated/` reuses the warm path, and
`cmake --build --target LupMigrationsPlugin` links the resulting DLL with
every generated TU compiled in.
Signed-off-by: Christian Parpart <christian@parpart.family>
…stamp Adds `ApplyPendingMigrationsUpTo(target, cb)` and `PreviewPendingMigrationsUpTo(target, cb)` to MigrationManager. Both honor the same dependency-respecting topological order as their unbounded counterparts and thread a single render context across the run so column- width state from earlier CREATE TABLEs is visible to later compat-aware INSERT/UPDATE rendering. A new anonymous `FilterPendingUpTo` helper filters the topo-sorted pending list to migrations with `ts <= targetInclusive` and refuses partial states whose included migrations declare a dependency on an excluded (`ts > target`) pending migration — applying such a set would silently violate the dependency contract. Covered by five new unit tests: boundary apply, off-boundary target, no-op when already at target, preview SQL identity, and dependency- cross-boundary refusal. Signed-off-by: Christian Parpart <christian@parpart.family>
…elease Adds `dbtool migrate-to-release <VERSION>`, the forward-direction counterpart of the existing `rollback-to-release`. Resolves the version to its declared `highestTimestamp` via `FindReleaseByVersion` and delegates to `MigrationManager::ApplyPendingMigrationsUpTo`. Forward-only: when the database is already at or past the target, the command is a no-op and prints a hint pointing at `rollback-to-release` rather than silently reverting. Honors `--dry-run` (`-n`) and `--no-lock` like the other migration commands. Wires the command into `DispatchDbCommand`, adds it to `PrintUsage` and two examples in `PrintExamples`, documents it in `docs/dbtool.md`, and extends `test_dbtool.py` with four integration scenarios (dry-run, unknown release, already-at-target no-op, happy path). Signed-off-by: Christian Parpart <christian@parpart.family>
…e across engines Compare live tables to the migration plan by name only. The engine resolves an unqualified plan (`schemaName=""`) into its default schema (`dbo` on MSSQL, `public` on Postgres, none on SQLite), so the live row's schema is engine-specific while the migration plan keeps its declared schema. Matching the dropped-tables half of this same function, which already keys off the unqualified table name. Signed-off-by: Christian Parpart <christian@parpart.family>
An explicit `--connection-string` without `--profile` is the user's way of saying "don't auto-apply any profile defaults" — the connection string they typed pins them to a specific backend, and silently merging another profile's `schema` / `pluginsDir` / etc. is at best surprising and at worst wrong (e.g. picking up `schema: dbo` from a SQL Server profile while pointing at SQLite, which then injects `"dbo"."<table>"` into every introspection query). Signed-off-by: Christian Parpart <christian@parpart.family>
…lTables Replace post-migration verification queries against `sqlite_schema` with `SqlSchema::ReadAllTables`, so HardReset and UnicodeUpgradeTables test cases run on every backend the suite is parameterized over and not just SQLite. Signed-off-by: Christian Parpart <christian@parpart.family>
Apply clang-format-22 across all modified C++ files. No behavioral changes — purely whitespace/wrapping normalisation against the project .clang-format style. Signed-off-by: Christian Parpart <christian@parpart.family>
- Add Doxygen comments for new public members (HardResetResult, UnicodeUpgradeResult, ColumnUpgradeEntry, PlanFoldingResult inner structs, ColumnDiff, TableDiff, RowDiff, TableDataDiff, DiffProgressEvent, DataSourceInfo, DriverInfo, SecretResolver move ops, MigrationRenderContext::ColumnKey/TableKey, SplitFileWriter CodeBlock). - Replace `\ref` with backticks in SqlSchemaDiff/SqlDataDiff comments so doxygen stops complaining about unresolved references. - Reword `@overload` directive in MigrationPlan.hpp so doxygen does not interpret the trailing words as a symbol name. - Move `@copydoc` recursion in SqlStatement.hpp by giving the primary declarations their own brief docstrings. - Exclude `src/Lightweight/tui/` from doxygen — vendored code that follows the upstream comment style. - Migrate `.find(x) != std::string::npos` to `.contains(x)` (and the inverse) across all PR-introduced code, satisfying clang-tidy 22's `readability-container-contains` check. - Convert positional aggregate initializers to designated form for `StatementWithComments` and `MigrationRenderContext::ColumnKey`, satisfying `modernize-use-designated-initializers`. No behavioral changes — all transformations are mechanical. Signed-off-by: Christian Parpart <christian@parpart.family>
christianparpart
force-pushed
the
feature/migrations-gui
branch
from
8860114 to
742a13a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The original goal of this branch — and still its main visible deliverable — is a Qt 6 GUI on top of
dbtool. While building it, the supporting infrastructure (profile / secrets handling, ODBC enumeration, cross-backend migration semantics) grew large enough to be useful on its own, and the CLI grew several new commands (exec,diff,fold,hard-reset,unicode-upgrade-tables,rewrite-checksums) that the GUI either drives directly or shares code with. Hence the size of the diff.Headline change — dbtool GUI
src/tools/dbtool-gui/(opt-in viaLIGHTWEIGHT_BUILD_GUI=ON): Qt 6 / QML front-end fordbtoolagainst any configured connection profile. Includes:LIGHTWEIGHT_SQL_RELEASEmarkers, with per-migration SQL preview and one-click apply / revert / backup-restore actions (MigrationRunner,BackupRunnerrunning onQThreadworkers).Config::ProfileStore+Secrets::SecretResolver, with ODBC DSN auto-discovery and an optionalqtkeychain-based credential backend.SqlQueryRunner,SqlSyntaxHighlighter,SqlResultModel) so the GUI is not migration-only.ThemeControllerand a small reusable QML component set (Card,StatusPill,KineticListView,WheelScrollAmplifier, ...).Reusable Lightweight infrastructure
Config::ProfileStore: YAML-backed store for named connection profiles, used by bothdbtooland the GUI so credentials no longer live in plaintext on disk.Secrets::SecretResolverwithEnvBackend/FileBackend/StdinBackend: pluggable indirection for credentials referenced from profiles.FileBackendrefuses files with mode wider than0600.Odbc::DataSourceEnumerator: wrapsSQLDataSourcesand driver enumeration so callers can populate a DSN dropdown.MigrationFold/: shared engine that the GUI anddbtool folduse to collapse a chain of migrations into a single C++ or SQL artifact (CppEmitter,SqlEmitter,Folder).CodeGen/SplitFileWriter: shared codegen helper used bylup2dbtoolandMigrationFoldto emit one C++ file per migration with a shared CMake snippet.tui/: vendored markdown-table / SGR / unicode-width helpers (also pulls inlibunicodeviaFindOrFetchLibunicode.cmake) — used by the newdbtool diffrenderer.MigrationException: carries operation / timestamp / title / step index / failed SQL / driver message, so both the CLI and GUI can show actionable error context without parsingwhat().WhereExpression/SetExpressiononUpdate/Delete, idempotent variants ofAddColumn/DropColumn/AddForeignKey/DropIndexacross SQLite / PostgreSQL / MSSQL, deterministic FK constraint names, a SQLiteALTER TABLErebuild path forAddForeignKey/DropForeignKey, and per-migration compat policy with alup-truncaterenderer.NCHARconcatenation so non-BMP characters survive round-trip.SqlSchemacross-engine introspection fixes:SQLForeignKeysrow grouping now keys onFK_NAME(with a SQLitePRAGMAfallback), plus assorted reader gaps that previously caused identical migrations to look like drift across engines.SqlSchemaDiff/SqlDataDiff: cross-engine logical equivalence — pairs tables / columns by name, projects toLogicalKind/LogicalType, and folds engine-specific incidental detail (CharvsNChar,Tinyint→Smallint,Realprecision drift, unbounded text,VarBinarysize loss). Per-side table descriptors so a Postgres-vs-MSSQL data diff issues the rightSELECTagainst each side.SqlConnectionnow surfaces the original driver diagnostic whenConnect()fails;SqlStatementtoleratesSQL_NO_DATAfromSQLExecDirect;SqlConnectInfo::EnsureSqliteDatabaseFileExistsbootstraps a missing file-based SQLite database.dbtool— new commands and reworksConfig::ProfileStore+Secrets::SecretResolver(drops the directyaml-cpplink and bespoke~/.config/dbtool/dbtool.ymlparsing); adds--profile <name>and surfaces the latest applied release instatus.dbtool exec <QUERY>— ad-hoc SQL execution against a profile.dbtool diff <profileA> <profileB>— schema + data diff between two databases, rendered through the newDiffRenderer(markdown tables, schema labels, canonical type variants).dbtool fold— collapse a migration chain into a single artifact viaMigrationFold.dbtool hard-reset— drop all tables in batched transactions (used by the GUI and CI).dbtool unicode-upgrade-tables— bulk UTF-8 upgrade for legacy MSSQL schemas.dbtool rewrite-checksums— rewrite stored migration checksums after an authorised content change, with plugin policy propagation.--show-examples— long-form examples moved out of the default--help.lup2dbtoolandLupMigrationsPluginLupSqlParser,WhereClauseParser, expandedSqlStatementParserandCodeGeneratorto lift the legacyinit_m_*.sql/upd_m_*.sqlcorpus into one C++ file per migration (lup_{version}.cpp), with--emit-cmakefor the shared CMake snippet.--force-unicodeis now the default (with--no-force-unicodeopt-out).LupMigrationsPlugininstalls thelup-truncatecompat policy by timestamp (cutoff extended to all LUP migrations) and scrubs proprietary paths from placeholder comments.Documentation
docs/migrations-gui-plan.md,docs/migrations-gui-mockup.html— design + clickable mockup that drove the GUI work.docs/lup-legacy-migration-plan.md— updated to reflect the as-shipped design.docs/dbtool.md— covers the new subcommands and profile flow.Risk
LIGHTWEIGHT_BUILD_GUI=ON) and the newdbtoolsubcommands are additive.dbtoolno longer reads~/.config/dbtool/dbtool.ymldirectly — existing users must migrate to a profile. The new schema-diff is intentionally lossier within a single engine (size-only drift on a column is no longer flagged); intra-engine size drift is still caught via the canonical type variant for sized types.DiffTableDatasignature changed (singleTable const&→ per-side descriptors); only the in-tree dbtool caller and four tests are affected and have been updated.Coverage
ConfigProfileStoreTests,SecretResolverTests,DataSourceEnumeratorTests,SqlSchemaDiffTests,SqlDataDiffTests, plus large additions toLup2DbtoolTests,MigrationTests, andQueryBuilderTests.dbtool diff.Test plan
LIGHTWEIGHT_BUILD_GUI=OFF(default) and confirm CLI-only build is unchanged.LIGHTWEIGHT_BUILD_GUI=ON, launchdbtool-gui, exercise apply / revert / backup-restore against a SQLite profile.dbtool diffbetween SQLite and PostgreSQL of the same migrated schema — expect zero differences.dbtool exec,dbtool fold,dbtool hard-reset,dbtool unicode-upgrade-tables,dbtool rewrite-checksumsagainst a throwaway profile.