Move spec tests into the LLB graph#898
Open
cpuguy83 wants to merge 3 commits intoproject-dalec:mainfrom
Open
Conversation
cpuguy83
force-pushed
the
be_lazy
branch
2 times, most recently
from
b3b9d7f to
68c2f6b
Compare
cpuguy83
force-pushed
the
be_lazy
branch
27 times, most recently
from
ce19276 to
99d2499
Compare
cpuguy83
force-pushed
the
be_lazy
branch
21 times, most recently
from
1316da1 to
e8be354
Compare
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR moves Dalec spec/package tests into the normal BuildKit LLB graph so test execution happens as part of the solve, rather than running separate test containers after the main build.
Changes:
- Introduces a new
internal/testrunnerimplementation that runs test steps and file/stdio checks inside the LLB graph. - Refactors Linux distro target handlers (package/container/sysext) to wire tests into the produced LLB and ensure test execution is part of the solve.
- Updates test harness/CI to stream solve status differently and to build docs examples via the existing
buildx bake examplestarget.
Reviewed changes
Copilot reviewed 48 out of 49 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
internal/testrunner/runner.go |
Adds the core LLB-based validation runner + utilities for composing validations. |
internal/testrunner/step.go |
Implements step execution and stdout/stderr capture files used by LLB checks. |
internal/testrunner/tests.go |
Orchestrates per-test execution and merges validations into the graph. |
tests.go |
Refactors source-mapping for tests/checks to return llb.ConstraintsOpt for LLB integration. |
frontend/test_runner.go |
Switches frontend test execution to the new LLB-based testrunner flow. |
targets/linux/distro_handler.go |
Changes target handlers to solve graphs that include tests (and final-state dependency chaining). |
targets/linux/rpm/distro/pkg.go / targets/linux/deb/distro/pkg.go |
Updates distro-specific test dependency installation + test runner hookup. |
test/testenv/buildx.go / test/testenv/build.go |
Refactors solve option setup and solve-status streaming helpers. |
cmd/frontend/main.go |
Moves subcommand handling to the plugin-based command registry. |
.github/workflows/ci.yml |
Uses docker buildx bake examples for docs examples build in CI. |
Move spec/package test execution out of the frontend's manual container-run path and into the normal LLB graph. The previous test runner solved the package build, then used the BuildKit API to run test containers and inspect filesystem state from outside the graph. That made the build less lazy, introduced extra solve behavior, and broke clients such as `docker buildx dap build` that expect the frontend solve to remain debuggable as a normal graph. This change represents test execution as BuildKit operations instead. The frontend binary is mounted into test containers and invoked through internal subcommands for filesystem checks. Each assertion is emitted as a separate exec op so failures can retain useful source mapping. Running tests this way keeps the test work attached to the build graph, lets BuildKit schedule and cache it normally, and avoids forcing package outputs to be materialized just so the frontend can run tests manually. Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This moves Dalec spec/package tests into the normal LLB graph instead of running test containers manually through the BuildKit API after the main solve.
The main implementation details and rationale are in
tests: run package tests in LLB graph.This also includes small supporting fixes to protect shared test spec mutation and to build docs examples through the existing bake target.