Skip to content

Implement storage of plan artifacts in the same storage as state files#1425

Merged
christian-calabrese merged 28 commits intomainfrom
poc-store-plan-artifact-in-csp
Mar 23, 2026
Merged

Implement storage of plan artifacts in the same storage as state files#1425
christian-calabrese merged 28 commits intomainfrom
poc-store-plan-artifact-in-csp

Conversation

@christian-calabrese
Copy link
Copy Markdown
Contributor

@christian-calabrese christian-calabrese commented Mar 11, 2026

Add two new TypeScript actions to read/write Terraform plan bundles on Azure Blob Storage and S3. The infra_apply workflow uses them to store the plan next to the state file (same container/bucket, plan-artifacts/ prefix), replacing the previous GitHub Artifact flow.

Compared to the old encrypted-artifact approach, storing the plan on the CSP backend means access is controlled by the same federated identity already trusted for state operations — no encryption key to provision or rotate, and the plan never leaves the CSP security boundary.

The download action includes a post: step that automatically deletes the remote object after a successful apply.

Resolves CES-1727

@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented Mar 11, 2026

⚠️ No Changeset found

Latest commit: 3410876

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@christian-calabrese christian-calabrese changed the title Poc store plan artifact in csp PoC store plan artifact in CSP storage Mar 11, 2026
Comment thread .github/workflows/infra_apply.yaml Fixed
Comment thread .github/workflows/infra_apply.yaml Fixed
@christian-calabrese christian-calabrese changed the title PoC store plan artifact in CSP storage Implement PoC to store plan artifacts in the state storage Mar 11, 2026
Comment thread .github/workflows/infra_apply.yaml Outdated
Comment thread actions/cloud-storage-download/src/main.ts Outdated
Comment thread actions/terraform-plan-storage-download/src/post.ts
@christian-calabrese
Copy link
Copy Markdown
Contributor Author

christian-calabrese commented Mar 11, 2026

Here is a full test of the flow: dx-playground

@christian-calabrese christian-calabrese marked this pull request as ready for review March 11, 2026 14:07
@christian-calabrese christian-calabrese requested a review from a team as a code owner March 11, 2026 14:07
Comment thread .github/workflows/infra_apply.yaml Fixed
Comment thread .github/workflows/infra_apply.yaml Fixed
@christian-calabrese
Copy link
Copy Markdown
Contributor Author

The workflow reads the Terraform backend configuration from the local file .terraform/terraform.tfstate (after terraform init) using jq to extract backend.type and the backend config (e.g. key, storage_account_name/container for Azure or bucket/region for S3).

It then computes the upload path by taking the state key, splitting it into directory and basename, and placing the plan next to the state under a sibling folder: <state-dir>/plan-artifacts/<state-basename>.<GITHUB_RUN_ID> (for flat keys the folder becomes plan-artifacts/<state-basename>.<GITHUB_RUN_ID>).

Finally the plan is uploaded to the same storage provider/container/bucket using the cloud-storage-upload action at that computed path, and the cloud-storage-download action later fetches it from the same location.

After a successful apply, the download action’s post: step (controlled by delete-on-completion: "true") deletes the plan artifact from the remote state storage and also removes the local file from the runner.

@christian-calabrese christian-calabrese changed the title Implement PoC to store plan artifacts in the state storage Implement storage of plan artifacts in the same storage as state files Mar 11, 2026
Comment thread actions/terraform-plan-storage-download/tsup.config.ts
Comment thread actions/terraform-plan-storage-download/package.json
Comment thread actions/terraform-plan-upload/action.yaml Outdated
Comment thread actions/terraform-plan-upload/action.yaml Outdated
Comment thread actions/terraform-plan-download/action.yaml Outdated
Comment thread actions/cloud-storage-upload/action.yaml Outdated
Comment thread actions/cloud-storage-download/action.yaml Outdated
Comment thread actions/cloud-storage-download/src/main.ts Outdated
Comment thread actions/cloud-ephemeral-download/action.yaml Outdated
Comment thread actions/cloud-ephemeral-download/src/post.ts Outdated
Comment thread .github/workflows/infra_apply.yaml Fixed
Comment thread .github/workflows/infra_apply.yaml Fixed
Comment thread .github/workflows/infra_apply.yaml Fixed
Comment thread actions/terraform-plan-storage-download/src/main.ts Fixed
Comment thread actions/terraform-plan-storage-download/src/main.ts Fixed
@christian-calabrese christian-calabrese marked this pull request as draft March 13, 2026 13:11
# Outputs (provider, plan-path, credentials) are passed to the apply job.
- name: Upload Terraform Plan
id: upload_plan
uses: pagopa/dx/actions/terraform-plan-storage-upload@poc-store-plan-artifact-in-csp

Check warning

Code scanning / CodeQL

Unpinned tag for a non-immutable Action in workflow Medium

Unpinned 3rd party Action 'infra_apply.yaml' step
Uses Step: upload_plan
uses 'pagopa/dx/actions/terraform-plan-storage-upload' with ref 'poc-store-plan-artifact-in-csp', not a pinned commit hash
# Download and extract the plan bundle (plan file + .terraform.lock.hcl + .terraform/modules/).
# Storage coordinates come directly from the tf_plan job outputs — no terraform init required.
- name: Download Terraform Plan
uses: pagopa/dx/actions/terraform-plan-storage-download@poc-store-plan-artifact-in-csp

Check warning

Code scanning / CodeQL

Unpinned tag for a non-immutable Action in workflow Medium

Unpinned 3rd party Action 'infra_apply.yaml' step
Uses Step
uses 'pagopa/dx/actions/terraform-plan-storage-download' with ref 'poc-store-plan-artifact-in-csp', not a pinned commit hash
Comment thread actions/terraform-plan-storage-download/src/main.ts Fixed
Comment thread actions/terraform-plan-storage-download/src/main.ts Fixed
@christian-calabrese christian-calabrese marked this pull request as ready for review March 16, 2026 13:32
Comment thread actions/terraform-plan-storage-download/action.yaml
@christian-calabrese christian-calabrese force-pushed the poc-store-plan-artifact-in-csp branch from e6e3ce3 to 3410876 Compare March 20, 2026 13:22
@christian-calabrese christian-calabrese merged commit f1a1a73 into main Mar 23, 2026
8 checks passed
@christian-calabrese christian-calabrese deleted the poc-store-plan-artifact-in-csp branch March 23, 2026 08:38
christian-calabrese added a commit that referenced this pull request Mar 30, 2026
#1425)

Add two new TypeScript actions to read/write Terraform plan bundles on
Azure Blob Storage and S3. The infra_apply workflow uses them to store
the plan next to the state file (same container/bucket, plan-artifacts/
prefix), replacing the previous GitHub Artifact flow.

Compared to the old encrypted-artifact approach, storing the plan on the
CSP backend means access is controlled by the same federated identity
already trusted for state operations — no encryption key to provision or
rotate, and the plan never leaves the CSP security boundary.

The download action includes a post: step that automatically deletes the
remote object after a successful apply.

Resolves CES-1727
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants