Add post-release smoke tests to release workflow#760
Open
Conversation
Adds a smoke-test job that runs after PyPI publish to validate the released hf-xet version end-to-end against the real Hub. Tests both repo and bucket upload/download paths with SHA-256 integrity checks. Includes Slack notifications on success/failure. Also adds a temporary standalone test-smoke-test.yml workflow to validate the setup before merging (to be deleted before merge). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Allows testing the smoke test job and Slack notifications from the PR without needing the workflow on the default branch for workflow_dispatch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The smoke test token doesn't have repo creation rights under the personal account. Add --namespace flag to the test script and default to xet-team org in CI workflows. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Smoke tests and Slack notifications validated successfully via PR trigger. The release.yml smoke-test job is the permanent home for this functionality. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
seanses
reviewed
Apr 2, 2026
seanses
reviewed
Apr 2, 2026
seanses
reviewed
Apr 2, 2026
seanses
reviewed
Apr 2, 2026
- Pin actions to commit SHAs (checkout, setup-uv, slack-github-action) - Expand OS matrix to ubuntu, macos, windows (run in parallel) - Replace uv pip compile with PyPI JSON API for availability check - Add run URL to success Slack notification - Single Slack notification after all platforms complete, with failed platform names on failure (via GitHub API job query) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Addressed all review feedback:
Additional improvements:
|
The wait-for-pypi job polls PyPI once on a single runner, then all 3 platform smoke tests start in parallel once the package is available. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
| } | ||
|
|
||
| - name: Notify Slack on success | ||
| if: ${{ needs.smoke-test.result == 'success' }} |
There was a problem hiding this comment.
No Slack notification when smoke tests are skipped
Medium Severity
The smoke-test-notify job only handles needs.smoke-test.result values of 'failure' and 'success'. If wait-for-pypi fails (e.g., the package never appears on PyPI within 15 minutes), the smoke-test job is skipped, producing a result of 'skipped'. Neither notification step fires, and the team receives no Slack alert that post-release validation didn't actually run. The same silent gap occurs if the smoke tests are cancelled. This means a release can complete with zero smoke test coverage and zero notification.
Additional Locations (1)
seanses
reviewed
Apr 2, 2026
| run: | | ||
| VERSION="${{ steps.version.outputs.version }}" | ||
| ./scripts/smoke_tests/run.sh --hf-xet-version "$VERSION" --namespace xet-team | ||
| ./scripts/smoke_tests/run.sh --hf-xet-version "${{ needs.wait-for-pypi.outputs.version }}" --namespace xet-team |
Collaborator
There was a problem hiding this comment.
Running "run.sh" in bash on Windows -- worth triggering a test in this PR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
smoke-testjob torelease.ymlthat runs after PyPI publishtest-smoke-test.ymlfor validating the setup before merge (will be deleted)Secrets required
HF_SMOKE_TEST_TOKENSLACK_SMOKE_TEST_WEBHOOK_URLTesting plan
HF_SMOKE_TEST_TOKENto repo secretsSLACK_SMOKE_TEST_WEBHOOK_URLto repo secrets1.4.0)test-smoke-test.ymlbefore merging🤖 Generated with Claude Code
Note
Medium Risk
Adds new post-release automation that gates on PyPI propagation and exercises real Hub operations using privileged secrets, so failures or misconfiguration could break the manual release workflow and leak/overuse credentials.
Overview
Adds a post-PyPI release validation pipeline to
.github/workflows/release.yml: wait for the publishedhf_xetversion to appear on PyPI, then run cross-platform smoke tests (Ubuntu/macOS/Windows) against the released package, followed by Slack notifications on success/failure (including which platforms failed and a link to the run).Updates
scripts/smoke_tests/test_upload_download.pyto accept an explicit--namespaceso smoke tests can run under an org/user namespace (defaulting to the token owner), aligning with the workflow’s use of a dedicated bot token.Written by Cursor Bugbot for commit 971d52d. This will update automatically on new commits. Configure here.