Skip to content

Add code coverage#961

Open
kartikjoshi21 wants to merge 2 commits intoproject-dalec:mainfrom
kartikjoshi21:kartikjoshi21/code-coverage
Open

Add code coverage#961
kartikjoshi21 wants to merge 2 commits intoproject-dalec:mainfrom
kartikjoshi21:kartikjoshi21/code-coverage

Conversation

@kartikjoshi21
Copy link
Copy Markdown
Member

@kartikjoshi21 kartikjoshi21 commented Feb 5, 2026

Add Go coverage reporting to CI (end-to-end)

This PR adds automated Go coverage reporting to the CI pipeline.

  • Unit and integration tests now generate coverage profiles (one per integration matrix suite).
  • It also collects coverage for the Dalec BuildKit frontend by building it with coverage enabled and exporting its covdata during solves.
  • A new coverage-report job downloads all profiles, merges them, and uploads a combined report (including an index.html coverage page) as a workflow artifact, plus a summary in the Actions run.

Fixes: #889

@kartikjoshi21 kartikjoshi21 force-pushed the kartikjoshi21/code-coverage branch 3 times, most recently from 25adb30 to 0ab8eae Compare February 6, 2026 10:03
@kartikjoshi21 kartikjoshi21 marked this pull request as ready for review February 6, 2026 10:05
Copilot AI review requested due to automatic review settings February 6, 2026 10:05
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds comprehensive code coverage tracking to the CI workflow to support OpenSSF Silver Badge requirements. It instruments both unit and integration tests to collect coverage data, merges the profiles, and generates coverage reports that are displayed in GitHub's step summary.

Changes:

  • Modified integration tests to collect coverage data with -covermode=atomic -coverpkg=./... flags
  • Modified unit tests to collect coverage data, excluding the /test directory packages
  • Added new coverage-report job that merges all coverage profiles and generates summary reports

Comment thread .github/workflows/ci.yml
Comment on lines +501 to +507
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093
with:
name: coverage-unit
path: coverage

- name: Download integration coverage artifacts
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093
Copy link

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing version comment for the download-artifact action. All other action uses in this workflow include version comments (e.g., # v6.0.0). Please add a version comment to maintain consistency with the established convention in this file.

Suggested change
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093
with:
name: coverage-unit
path: coverage
- name: Download integration coverage artifacts
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # download-artifact pinned SHA
with:
name: coverage-unit
path: coverage
- name: Download integration coverage artifacts
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # download-artifact pinned SHA

Copilot uses AI. Check for mistakes.
Comment thread .github/workflows/ci.yml
Comment on lines +501 to +507
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093
with:
name: coverage-unit
path: coverage

- name: Download integration coverage artifacts
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093
Copy link

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing version comment for the download-artifact action. All other action uses in this workflow include version comments (e.g., # v6.0.0). Please add a version comment to maintain consistency with the established convention in this file.

Suggested change
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093
with:
name: coverage-unit
path: coverage
- name: Download integration coverage artifacts
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.1.8
with:
name: coverage-unit
path: coverage
- name: Download integration coverage artifacts
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.1.8

Copilot uses AI. Check for mistakes.
Comment thread .github/workflows/ci.yml
runs-on: ubuntu-22.04
needs:
- unit
- integration
Copy link

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The coverage-report job will not run if either the unit or integration jobs fail, even though those jobs use if: always() to upload coverage artifacts on failure. This means coverage reports won't be generated when tests fail. Consider adding if: always() to the coverage-report job (or using a more specific condition like if: success() || failure()) to ensure coverage reports are generated even when tests fail, which would be valuable for understanding coverage gaps in failing test scenarios.

Suggested change
- integration
- integration
if: ${{ always() }}

Copilot uses AI. Check for mistakes.
Comment thread .github/workflows/ci.yml Outdated
exit 1
fi

"$(go env GOPATH)/bin/gocovmerge" coverage/unit.out ${integration_profiles} > coverage/all.out
Copy link

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The unquoted variable expansion ${integration_profiles} could potentially cause issues with shell word splitting if any filenames contain spaces or special characters. While the current suite names don't contain spaces, it's a best practice to use an array or quote the variable to make the script more robust. Consider using a bash array: integration_profiles=($(find...)) and then "${integration_profiles[@]}" in the gocovmerge command.

Copilot uses AI. Check for mistakes.
Comment thread .github/workflows/ci.yml Outdated
@kartikjoshi21 kartikjoshi21 force-pushed the kartikjoshi21/code-coverage branch from 0ab8eae to 853ecfb Compare February 9, 2026 09:32
Comment thread .github/workflows/ci.yml
@kartikjoshi21 kartikjoshi21 force-pushed the kartikjoshi21/code-coverage branch 12 times, most recently from de76fc1 to 905a3e6 Compare March 3, 2026 10:01
@kartikjoshi21 kartikjoshi21 force-pushed the kartikjoshi21/code-coverage branch from 905a3e6 to b94d946 Compare March 3, 2026 19:08
@kartikjoshi21 kartikjoshi21 requested a review from cpuguy83 March 3, 2026 19:12
@cpuguy83
Copy link
Copy Markdown
Collaborator

cpuguy83 commented Mar 3, 2026

@kartikjoshi21 CI didn't run on this because there was temporarily a conflict (commit is now reverted so no conflict).
Can you rebase to trigger CI?

@kartikjoshi21 kartikjoshi21 force-pushed the kartikjoshi21/code-coverage branch 2 times, most recently from 124edfa to fe8704e Compare March 5, 2026 08:44
@kartikjoshi21
Copy link
Copy Markdown
Member Author

@kartikjoshi21 CI didn't run on this because there was temporarily a conflict (commit is now reverted so no conflict). Can you rebase to trigger CI?

@cpuguy83 Updated the PR, Thanks

Comment thread cmd/frontend/coverage.go Outdated
Comment thread cmd/frontend/coverage.go Outdated
@kartikjoshi21 kartikjoshi21 force-pushed the kartikjoshi21/code-coverage branch 2 times, most recently from 693bbc9 to 2b605d0 Compare March 30, 2026 05:55
@kartikjoshi21 kartikjoshi21 requested a review from cpuguy83 March 30, 2026 05:55
Comment thread cmd/frontend/coverage.go Outdated
This PR adds automated Go coverage reporting to the CI pipeline.
Unit and integration tests now generate coverage profiles
(one per integration matrix suite).
It also collects coverage for the Dalec BuildKit frontend by
building it with coverage enabled and exporting its covdata
during solves.

A new coverage-report job downloads all profiles, merges them,
and uploads a combined report (including an index.html coverage page)
as a workflow artifact, plus a summary in the Actions run.

Fixes: project-dalec#889
Signed-off-by: Kartik Joshi <karikjoshi21@gmail.com>
@kartikjoshi21 kartikjoshi21 force-pushed the kartikjoshi21/code-coverage branch from 2b605d0 to 4d0717a Compare April 22, 2026 10:46
Signed-off-by: Kartik Joshi <karikjoshi21@gmail.com>
@kartikjoshi21 kartikjoshi21 force-pushed the kartikjoshi21/code-coverage branch from 4d0717a to 0f7b13d Compare April 27, 2026 11:32
@kartikjoshi21 kartikjoshi21 requested a review from cpuguy83 April 28, 2026 06:18
@cpuguy83
Copy link
Copy Markdown
Collaborator

cpuguy83 commented May 1, 2026

Comment from GPT-5.5:

I think frontend coverage is currently being converted with metadata but no counters.

Evidence:

This PR writes counter files as covcounters.<hash>.<pid>.<timestamp>.<random> in test/testenv/frontend_coverage.go, so they do not match the anchored Go regex and are ignored by go tool covdata textfmt.

That matches the latest CI artifacts: all 14 frontend-*.out profiles are present, but each has 0 covered statements (covered=0, total=6835, pct=0.00). So the frontend profiles are currently contributing denominator, but no executed frontend coverage.

Suggested fix: write counter files using exactly covcounters.<hash>.<pid>.<timestamp> and add an assertion that go tool covdata textfmt produces at least one non-zero frontend coverage block, rather than only checking that covmeta.* / covcounters.* files exist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[REQ] Add code coverage testing

4 participants