[Spike] Pipeline generation: Core abstractions and GitHub Actions scheduling#15643
Draft
mitchdenny wants to merge 24 commits intomainfrom
Draft
[Spike] Pipeline generation: Core abstractions and GitHub Actions scheduling#15643mitchdenny wants to merge 24 commits intomainfrom
mitchdenny wants to merge 24 commits intomainfrom
Conversation
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 15643Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 15643" |
Contributor
|
Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
|
Contributor
|
Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
|
…uling Add core pipeline environment abstractions (IPipelineEnvironment, IPipelineStepTarget, PipelineEnvironmentCheckAnnotation) to Aspire.Hosting, along with a new Aspire.Hosting.Pipelines.GitHubActions package implementing workflow resource model and scheduling resolver. Key changes: - IPipelineEnvironment marker interface for CI/CD environments - IPipelineStepTarget for scheduling steps onto workflow jobs - PipelineStep.ScheduledBy property for step-to-job assignment - GetEnvironmentAsync() with annotation-based environment resolution - GitHubActionsWorkflowResource and GitHubActionsJob types - SchedulingResolver projecting step DAG onto job dependency graph - 29 unit tests covering environment resolution and scheduling Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add WorkflowYaml model types (WorkflowYaml, JobYaml, StepYaml, etc.) - Add hand-rolled WorkflowYamlSerializer (no external dependencies) - Add WorkflowYamlGenerator: scheduling result → complete workflow YAML - Boilerplate steps: checkout, setup-dotnet, install Aspire CLI - State artifact upload/download between dependent jobs - Per-job aspire do --continue --job <jobId> execution - Add TryRestoreStepAsync on PipelineStep for CI/CD state restore - Executor calls restore callback before Action - If restore returns true, step is skipped (already completed) - Add 9 YAML generator unit tests - Add 4 Verify snapshot tests with complete YAML output - Add 5 step state restore integration tests - Update spec doc with YAML generation and state restore design Total: 47 tests passing (12 hosting + 35 GitHub Actions) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds ScheduleStep(string stepName, IPipelineStepTarget target) to IDistributedApplicationPipeline. This allows consumers to schedule built-in steps (e.g., WellKnownPipelineSteps.Build) onto CI/CD jobs without having to create them — useful when integrations register steps and the AppHost just needs to assign them to workflow jobs. - Interface and implementation in Aspire.Hosting - 7 unit tests: basic scheduling, override, built-in steps, null guards, error on missing step Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Revert IDistributedApplicationPipeline to its original signature to preserve binary compatibility. The interface method AddStep retains its original 4-parameter signature. New members (ScheduleStep, GetEnvironmentAsync, and the 5-param AddStep overload with scheduledBy) are on DistributedApplicationPipeline only. Adding an optional parameter to an interface method changes the IL signature, which is a binary breaking change even though it's source compatible. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mitchdenny
force-pushed
the
spike/pipeline-generation
branch
from
d1185d9 to
3a360ae
Compare
…urce
- GitHubActionsJob → GitHubActionsJobResource (extends Resource)
- New GitHubActionsStageResource (extends Resource) for logical job grouping
- workflow.AddStage("build-stage") → stage.AddJob("build")
- workflow.AddJob("build") still works for direct job creation
- Jobs created via stages are also registered on the workflow
- 6 new tests for stage/resource APIs
- Total: 60 tests passing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…jobs
- GitHubActionsWorkflowResource and GitHubActionsStageResource now
implement IPipelineStepTarget, so steps can be scheduled onto them
- When a step targets a workflow, a default stage + default job are
auto-created to host it
- When a step targets a stage, a default job is auto-created within
that stage (named '{stage}-default')
- SchedulingResolver.ResolveJobForStep uses pattern matching to
resolve workflow/stage/job targets down to concrete jobs
- GetOrAddDefaultStage() on workflow, GetOrAddDefaultJob() on stage
- DefaultJob in SchedulingResult is now nullable (not created when
all steps are explicitly scheduled)
- 6 new resolver tests covering workflow/stage scheduling targets
- Total: 66 tests passing (19 hosting + 47 GH Actions)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- PipelineCommand: group command (like 'mcp') at 'aspire pipeline' - PipelineInitCommand: subcommand extending PipelineCommandBase - Passes --operation publish --step pipeline-init to AppHost - Follows same pattern as deploy/publish commands - Added WellKnownPipelineSteps.PipelineInit constant - Registered in DI (Program.cs) and RootCommand - Resource strings (.resx + .Designer.cs) with localization xlf files - HelpGroup.Deployment for discoverability Usage: aspire pipeline init [--apphost <path>] [--output-path <path>] Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix DI resolution failure: 'Unable to resolve service for type PipelineCommand while attempting to activate RootCommand'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Register a built-in 'pipeline-init' step in DistributedApplicationPipeline that discovers IPipelineEnvironment resources and calls their PipelineWorkflowGeneratorAnnotation to generate CI/CD workflow files. - Add PipelineWorkflowGeneratorAnnotation and PipelineWorkflowGenerationContext - Register generator annotation in AddGitHubActionsWorkflow extension - Pipeline-init step iterates environments and invokes generators Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Detect repository root via git rev-parse, fall back to walking up directories for aspire.config.json, and confirm with the user via IInteractionService. The resolved root is passed to generators as RepositoryRootDirectory (replaces OutputDirectory). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move ScheduleStep from concrete class only to the interface so it is discoverable via builder.Pipeline. Add convenience AddStep extension method with scheduledBy parameter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ActionsWorkflowResource> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ng default Unscheduled steps (ScheduledBy=null) are now resolved by BFS through reverse dependencies to find the nearest explicitly-scheduled consumer, and placed on that consumer's job. This eliminates unnecessary default stages/jobs when explicit stages exist. Fallback behavior: if no consumer is found and explicit targets exist, orphan steps go to the first available stage/job. If no explicit targets exist at all, the original default job behavior is preserved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- PipelineScopeAnnotation: auto-detects execution scope from CI env vars
(GITHUB_RUN_ID + GITHUB_RUN_ATTEMPT for RunId, GITHUB_JOB for JobId)
- PipelineScopeMapAnnotation: maps scope IDs to step names from scheduling
- ContinuationStateManager: per-scope writes to .aspire/state/{RunId}/{JobId}.json,
merged reads from all files in RunId directory for fan-in scenarios
- Executor enters continuation mode automatically when scope is detected:
in-scope steps execute, out-of-scope steps skip (or restore from state)
- YAML generator simplified: 'aspire do' (no --continue --job flags),
artifact names use 'aspire-do-state-' prefix, paths include run ID
- 17 new tests: scope annotation, continuation state manager, scope filtering
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…steps Two issues fixed: 1. The pipeline-init step was passing only well-known steps (_steps) to the workflow generator, missing resource annotation steps that create the actual cross-job dependencies. Fixed by using _lastResolvedSteps which contains all steps after annotation collection and RequiredBy normalization. 2. The pull-based scheduling assigned orphan steps (no consumer chain to an explicit target) to the first available job, which could create spurious cross-job dependencies leading to cycles. For example, publish-prereq (orphan) went to publish-default but depended on process-parameters in deploy-default, while deploy-default already depended on publish-default. Fixed by adding Phase 2b: orphan steps are now co-located with their dependencies instead of the first available job. Uses iterative resolution to handle chains of orphan-to-orphan dependencies. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…stomization - Add WellKnownDependencyTags class (requires-dotnet, requires-nodejs, requires-docker, requires-azure-cli, requires-aspire-cli) that declare what tooling a pipeline step needs on the CI machine - Tag existing step creation sites: ProjectResource and Container build/push steps get requires-dotnet/requires-docker, Azure steps get requires-azure-cli - WorkflowYamlGenerator now scans dependency tags per job and conditionally emits setup steps (setup-dotnet, setup-node, azure/login) only when needed - Channel-aware Aspire CLI install: reads aspire.config.json for channel (stable/preview/dev/daily) and emits curl|bash with quality arg - Make WorkflowYaml/JobYaml/StepYaml/etc public with XML docs and [Experimental] attributes for developer customization - Add ConfigureWorkflow extension method on IResourceBuilder<GitHubActionsWorkflowResource> with WorkflowCustomizationAnnotation for post-generation YAML mutation - Add 14 new tests: tag collection, conditional emission, channel install, ConfigureWorkflow callback tests - Update 4 snapshot tests for new install command format Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- URL: aspire.dev/install.sh (not aka.ms/install-aspire.sh) - Quality flags: -q dev (daily channel), -q staging (staging channel) - Unknown channels (e.g. 'preview') fall back to default (stable) - Remove unused PrInstallScriptUrl constant (PR install is future work) - Add staging channel test, rename preview test to unknown channel test - Update 4 snapshots with correct URL Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Compute terminal steps per job in SchedulingResolver and emit 'aspire do <step-name>' instead of bare 'aspire do' which requires a step argument - Add pull_request trigger to generated YAML (targeting main) - Add PullRequestTrigger to YAML model and serializer - Make CLI install step PR-aware: on pull_request events, use get-aspire-cli-pr.sh with PR number; on push/manual, use channel-based install from aspire.dev/install.sh - Update 4 snapshot tests and 4 unit tests for new behavior Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
🎬 CLI E2E Test Recordings — 52 recordings uploaded (commit View recordings
📹 Recordings uploaded automatically from CI run #23708676354 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Spike/proof-of-concept for pipeline generation — the ability to generate CI/CD workflow YAML files (e.g., GitHub Actions) from the Aspire application model.
What's Included
Core Abstractions (
Aspire.Hosting)IPipelineEnvironment— Marker interface extendingIResourcefor CI/CD environmentsIPipelineStepTarget— Interface for scheduling targets (workflow jobs implement this)PipelineEnvironmentCheckAnnotation— Annotation-based environment relevance checkLocalPipelineEnvironment— Internal fallback for local executionPipelineStep.ScheduledBy— Property linking steps to workflow jobsGetEnvironmentAsync()— Resolution of active pipeline environment from modelGitHub Actions Package (
Aspire.Hosting.Pipelines.GitHubActions)GitHubActionsWorkflowResource—Resource + IPipelineEnvironmentrepresenting a workflowGitHubActionsJob— Job within a workflow, implementsIPipelineStepTargetGitHubActionsWorkflowExtensions—builder.AddGitHubActionsWorkflow(name)extensionSchedulingResolver— Projects step DAG onto job dependency graph with cycle detectionTests (29 total)
Spec Document
docs/specs/pipeline-generation.md— Full architecture, API primitives, and future roadmapExample Usage
Not Yet Implemented
aspire pipeline init)aspire do --continue --job <jobId>CLI supportRelated
See
docs/specs/pipeline-generation.mdfor full design details and open questions.