-
Notifications
You must be signed in to change notification settings - Fork 31
proposal: add proposal for label automation #66
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
jan--f
wants to merge
1
commit into
prometheus:main
Choose a base branch
from
jan--f:label-automation
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,339 @@ | ||
| # Label Automation Tooling | ||
|
|
||
| * **Owners:** | ||
| * `@jan--f` | ||
|
|
||
| * **Implementation Status:** `Not implemented` | ||
|
|
||
| * **Related Issues and PRs:** | ||
| * [PROM-65](https://github.com/prometheus/proposals/pull/65) | ||
|
|
||
| * **Other docs or links:** | ||
| * [Prom-Prow Bot Implementation](https://github.com/jan--f/prom-prow) (to be | ||
| moved to `prometheus` org) | ||
| * [GitHub Actions: Add Labels to Issues](https://docs.github.com/en/actions/tutorials/manage-your-work/add-labels-to-issues) | ||
| * [CNCF Hosted Tools](https://contribute.cncf.io/resources/services/hosted-tools/) | ||
| * [Kubernetes Prow](https://docs.prow.k8s.io/) | ||
| * [GitHub CODEOWNERS Documentation](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners) | ||
|
|
||
| > TL;DR: This proposal recommends using a custom lightweight chat-ops bot | ||
| > (prom-prow) for Prometheus repositories. The bot provides Prow-style commands | ||
| > (`/lgtm`, `/cc`, `/label`, `/hold`) without the complexity of OWNERS files or | ||
| > running a full Prow instance. It uses GitHub's native collaborator permissions | ||
| > for command authorization and CODEOWNERS for automatic review assignment, | ||
| > providing bidirectional sync between `/lgtm` commands and GitHub UI approvals. | ||
|
|
||
| ## Why | ||
|
|
||
| The label system proposed in PROM-65 introduces structured labels for issue and | ||
| PR workflow management. However, manually applying and managing these labels | ||
| would create significant overhead for maintainers and contributors. Automation | ||
| is essential to make the label system practical and effective. | ||
|
|
||
| The Kubernetes ecosystem has demonstrated that chat-ops style label management | ||
| (using `/label`, `/approve`, `/lgtm` commands in PR comments) can provide an | ||
| efficient and intuitive workflow for maintainers. This approach should be | ||
| adopted for Prometheus projects. | ||
|
|
||
| ### Pitfalls of the current solution | ||
|
|
||
| Currently, Prometheus projects rely entirely on manual label application: | ||
|
|
||
| - Maintainers must navigate GitHub's UI to apply labels | ||
| - Difficult to enforce consistent labeling practices | ||
| - No automatic labeling based on PR content or actions | ||
|
|
||
| ## Goals | ||
|
|
||
| Goals and use cases for the solution as proposed in [How](#how): | ||
|
|
||
| * Enable chat-ops style label management through PR comments | ||
| * Provide a lightweight, maintainable automation solution | ||
| * Support the label taxonomy defined in PROM-65 | ||
| * Align with CNCF's preference for GitHub Actions where possible | ||
|
|
||
| ### Audience | ||
|
|
||
| This proposal targets: | ||
|
|
||
| - Prometheus maintainers and contributors who need efficient label management | ||
| tools | ||
|
|
||
| ## Non-Goals | ||
|
|
||
| * Running a full Kubernetes-based Prow instance | ||
| * Implementing all Prow plugins and features | ||
| * Automatic merge functionality (at least initially) | ||
| * Replacing manual label operations | ||
|
|
||
| ## How | ||
|
|
||
| ### Proposed Architecture | ||
|
|
||
| This proposal recommends using the **prom-prow** bot, a custom lightweight chat-ops bot designed specifically for Prometheus repositories. The bot is implemented as a GitHub Action and provides Prow-style commands without requiring OWNERS files or running a full Prow instance. | ||
|
|
||
| ### Prom-Prow Bot | ||
|
|
||
| The **prom-prow bot** is a custom GitHub Action that provides Prow-style chat-ops commands optimized for Prometheus workflows. Unlike existing Prow GitHub Actions, prom-prow is designed to work seamlessly with GitHub's native permission system and CODEOWNERS without requiring OWNERS files. | ||
|
|
||
| #### Supported Commands | ||
|
|
||
| The following commands are available in PR comments: | ||
|
|
||
| - **`/lgtm`**: Approve the PR | ||
| - Anyone: Submits an approving GitHub review (if repository is configured to | ||
| allow reviews by Actions) | ||
| - Collaborators: Also adds `review/lgtm` label | ||
| - PR authors cannot approve their own changes | ||
| - **`/lgtm cancel`**: Cancel approval | ||
| - Anyone: Dismisses the user's review | ||
| - Collaborators: Also removes `review/lgtm` label (only if no other collaborators have approved) | ||
| - **`/cc @user1 @user2`**: Request reviews from specified users (requires write access) | ||
| - **`/label <label>`**: Add label(s) to the PR (requires write access) | ||
| - **`/hold`**: Add `blocked/hold` label to prevent merging (requires write access) | ||
| - **`/unhold`**: Remove `blocked/hold` label (requires write access) | ||
|
|
||
| #### Key Features | ||
|
|
||
| - **No OWNERS Files Required**: Uses GitHub's native collaborator permissions | ||
| - **Bidirectional /lgtm Sync**: Both `/lgtm` command and GitHub UI approvals add the `review/lgtm` label | ||
| - **Automatic LGTM Removal**: Removes `review/lgtm` when new commits are pushed | ||
| - **Self-Approval Prevention**: PR authors cannot approve their own changes | ||
| - **Welcome Comments**: Posts helpful command documentation on new PRs | ||
| - **Two-Tier Permissions**: | ||
| - Anyone can submit reviews via `/lgtm` | ||
| - Only collaborators can manage labels | ||
| - **Works alongside CODEOWNERS**: Codeowners can be used for automatic review | ||
| assignment | ||
|
|
||
| #### Example Configuration | ||
|
|
||
| GitHub Actions workflow (`.github/workflows/prometheus-bot.yml`): | ||
|
|
||
| ```yaml | ||
| name: Prometheus Bot | ||
|
|
||
| on: | ||
| issue_comment: | ||
| types: [created] | ||
| pull_request_review: | ||
| types: [submitted] | ||
| pull_request: | ||
| types: [opened, synchronize] | ||
|
|
||
| jobs: | ||
| bot: | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| contents: read | ||
| issues: write | ||
| pull-requests: write | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: prometheus/prom-prow@main | ||
| with: | ||
| github-token: ${{ secrets.GITHUB_TOKEN }} | ||
| ``` | ||
|
|
||
| Note that this currently uses the Dockerfile integration, i.e. builds the image | ||
| on every run. For a production setup we wouls push the action image to a | ||
| registry and pull from there. | ||
|
|
||
| #### Why Custom Bot Instead of Prow GitHub Actions or Prow? | ||
|
|
||
| - Works with existing CODEOWNERS (no migration needed) | ||
| - Uses GitHub's native collaborator permissions (no new files) | ||
| - Bidirectional sync: `/lgtm` command ↔ GitHub UI approval both add label | ||
| - Simpler permission model aligned with GitHub's access levels | ||
| - Lightweight and maintainable (single Go binary, ~300 lines of core logic) | ||
|
|
||
| See [below for alternatives](#alternatives) considered. | ||
|
|
||
| ### Component 2: Initial Label Workflows | ||
|
|
||
| Simple GitHub Actions workflows automatically apply initial labels to new issues | ||
| and pull requests using the GitHub CLI (`gh`). | ||
|
|
||
| #### Issue Labeling | ||
|
|
||
| Automatically add `triage/needs-triage` to new issues. | ||
|
|
||
| `.github/workflows/label-issues.yml`: | ||
|
|
||
| ```yaml | ||
| name: Label new issues | ||
| on: | ||
| issues: | ||
| types: | ||
| - opened | ||
|
|
||
| jobs: | ||
| label_issues: | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| issues: write | ||
| steps: | ||
| - run: gh issue edit "$NUMBER" --add-label "$LABELS" | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| GH_REPO: ${{ github.repository }} | ||
| NUMBER: ${{ github.event.issue.number }} | ||
| LABELS: triage/needs-triage | ||
| ``` | ||
|
|
||
| #### Pull Request Labeling | ||
|
|
||
| Automatically add `review/needs-review` to new pull requests. | ||
|
|
||
| `.github/workflows/label-prs.yml`: | ||
|
|
||
| ```yaml | ||
| name: Label new pull requests | ||
| on: | ||
| pull_request: | ||
| types: | ||
| - opened | ||
|
|
||
| jobs: | ||
| label_prs: | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| pull-requests: write | ||
| steps: | ||
| - run: gh pr edit "$NUMBER" --add-label "$LABELS" | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| GH_REPO: ${{ github.repository }} | ||
| NUMBER: ${{ github.event.pull_request.number }} | ||
| LABELS: review/needs-review | ||
| ``` | ||
|
|
||
| #### Benefits of This Approach | ||
|
|
||
| - **Simple**: Uses only GitHub CLI commands, no external actions needed | ||
| - **Maintainable**: Clear, straightforward YAML that's easy to understand | ||
| - **Flexible**: Easy to add multiple labels or conditional logic | ||
| - **Native**: Uses GitHub's built-in CLI tool | ||
| - **Lightweight**: No dependencies on third-party actions | ||
|
|
||
| #### Optional: Component Label Automation | ||
|
|
||
| If desired, component labels could be added based on file patterns using a simple | ||
| script, but this is optional since maintainers can easily apply component labels | ||
| using `/label component/X` commands during review. | ||
|
|
||
| ### Integration with CODEOWNERS | ||
|
|
||
| The prom-prow bot is designed to work seamlessly with GitHub's native CODEOWNERS: | ||
|
|
||
| - **CODEOWNERS handles automatic review assignment**: When a PR is opened, GitHub automatically requests reviews from code owners based on the files changed | ||
| - **Prom-prow handles chat-ops commands**: Maintainers use `/lgtm`, `/cc`, `/label`, etc. for workflow management | ||
| - **Full compatibility**: CODEOWNERS can use teams, complex patterns, and all GitHub features | ||
|
|
||
| ### Stale Issue Management (Optional) | ||
|
|
||
| If lifecycle labels are adopted, GitHub's `actions/stale` can automate stale | ||
| marking: | ||
|
|
||
| ```yaml | ||
| name: Mark Stale Issues | ||
| on: | ||
| schedule: | ||
| - cron: '0 0 * * *' | ||
|
|
||
| jobs: | ||
| stale: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/stale@v9 | ||
| with: | ||
| stale-issue-label: 'lifecycle/stale' | ||
| exempt-issue-labels: 'lifecycle/frozen' | ||
| days-before-stale: 90 | ||
| days-before-close: 30 | ||
| stale-issue-message: 'This issue has been automatically marked as stale...' | ||
| ``` | ||
|
|
||
| ### Implementation Plan | ||
|
|
||
| 1. **Pilot in prometheus/prometheus**: | ||
| - Deploy `.github/workflows/prometheus-bot.yml` workflow | ||
| - Deploy simple initial labeling workflows for new issues and PRs | ||
| - Document commands in CONTRIBUTING.md and link to that in welcome message | ||
|
|
||
| 2. **Evaluate and Iterate**: | ||
| - Gather feedback from maintainers after 1-2 months | ||
| - Adjust bot configuration based on usage patterns | ||
| - Consider additional commands if needed | ||
|
|
||
| 3. **Roll Out to Other Repositories**: | ||
| - Apply to other prometheus/* repositories | ||
| - Customize per repository as needed (e.g., additional labels) | ||
|
|
||
| ### Validation | ||
|
|
||
| Success will be measured by: | ||
|
|
||
| - Increased consistency in label application | ||
| - Positive feedback from maintainers on workflow efficiency | ||
|
|
||
| ## Alternatives | ||
|
|
||
| ### Alternative 1: Full Prow Instance | ||
|
|
||
| Run a complete Kubernetes-based Prow instance, as offered by CNCF. | ||
|
|
||
| **Advantages:** | ||
| - Full feature set including tide (automatic merging), automatic label management | ||
| - Battle-tested in Kubernetes ecosystem | ||
| - Most powerful automation capabilities | ||
| - Full plugin ecosystem | ||
|
|
||
| **Disadvantages:** | ||
| - Significant operational overhead (Kubernetes cluster required) | ||
| - CNCF steers projects toward GitHub Actions | ||
| - More complex to maintain and configure | ||
|
|
||
| **Rationale for rejection**: Too expensive to run and maintain relative to | ||
| benefits. CNCF recommends GitHub Actions, and prom-prow provides sufficient | ||
| functionality without operational overhead. | ||
|
|
||
| ### Alternative 2: Prow GitHub Actions | ||
|
|
||
| Use the existing [Prow GitHub Actions](https://github.com/jpmcb/prow-github-actions) project. | ||
|
|
||
| **Advantages:** | ||
| - Community-maintained tool | ||
| - Provides Prow-style commands | ||
| - No Kubernetes infrastructure needed | ||
|
|
||
| **Disadvantages:** | ||
| - Requires OWNERS files (only supports single root OWNERS, no per-directory) | ||
| - No GitHub team support in OWNERS (individual usernames only) | ||
| - Does not interact with CODEOWNERS for review assignment | ||
| - No bidirectional sync (GitHub UI approvals don't trigger label automation) | ||
| - Would require maintaining both OWNERS and CODEOWNERS | ||
|
|
||
| **Rationale for rejection**: Requires migrating to OWNERS files or maintaining | ||
| dual configuration. Prom-prow provides better integration with GitHub's native | ||
| features (CODEOWNERS, collaborator permissions, UI approvals) while being | ||
| simpler to maintain. | ||
|
|
||
| ## Action Plan | ||
|
|
||
| * [ ] Review and accept this proposal | ||
| * [ ] Deploy prom-prow bot | ||
| * [ ] Create `.github/workflows/prometheus-bot.yml` | ||
| * [ ] Deploy initial label workflows | ||
| * [ ] Create `.github/workflows/label-issues.yml` (adds `triage/needs-triage`) | ||
| * [ ] Create `.github/workflows/label-prs.yml` (adds `review/needs-review`) | ||
| * [ ] Documentation | ||
| * [ ] Update CONTRIBUTING.md with chat-ops commands | ||
| * [ ] Announce to prometheus-developers mailing list | ||
| * [ ] Pilot period (1-2 months) | ||
| * [ ] Gather maintainer feedback | ||
| * [ ] Monitor for any issues | ||
| * [ ] Adjust if needed | ||
| * [ ] Roll out to other repositories | ||
| * [ ] Apply to alertmanager, client_golang, etc. on maintainers request | ||
| * [ ] Optional: Add stale issue automation (if lifecycle labels adopted) | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I learned the hard way that testing things in prometheus/prometheus can be disruptive, let's do it in a test repo.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes indeed, I did a small test already on the repo mentioned in the proposal, e.g. https://github.com/jan--f/prom-prow/pull/1. But happy to do a larger scale test on a Prometheus clone as well.