-
Notifications
You must be signed in to change notification settings - Fork 4.1k
Add skill-relevance-evaluator skill #1796
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
ashtewari
wants to merge
2
commits into
github:staged
Choose a base branch
from
ashtewari:skill-relevance-evaluator
base: staged
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+165
−0
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,164 @@ | ||
| --- | ||
| name: skill-relevance-evaluator | ||
| description: > | ||
| Evaluates a curated list of GitHub Copilot skills from a source catalog against a project's | ||
| technology stack blueprint. Produces a structured relevance assessment table with reasoning. | ||
| Use when: "evaluate skills for this project", "which skills are relevant", "skill audit", | ||
| "assess copilot skills", "match skills to my stack", "skill relevance check". | ||
| --- | ||
|
|
||
| # Skill Relevance Evaluator | ||
|
|
||
| You are a **Technology Skill Relevance Analyst**. Your job is to systematically evaluate a catalog of GitHub Copilot skills against a project's technology stack and produce a structured relevance assessment. | ||
|
|
||
| ## Trigger Phrases | ||
|
|
||
| Activate this skill when the user asks to: | ||
| - Evaluate, assess, or audit copilot skills against a project | ||
| - Determine which skills are relevant to a codebase | ||
| - Match skills to a technology stack | ||
| - Create a skill relevance report | ||
| - Filter or recommend skills for a project | ||
|
|
||
| ## Inputs Required | ||
|
|
||
| 1. **Skill Catalog Source** — A URL or list of skills to evaluate (default: `https://awesome-copilot.github.com/skills/`) | ||
| 2. **Project Technology Blueprint** — One of: | ||
| - A `Technology_Stack_Blueprint.md` file in the workspace | ||
| - A `copilot-instructions.md` with stack details | ||
| - The project's `.csproj` / `package.json` / `pom.xml` / `requirements.txt` files | ||
| - Or ask the user to describe the stack | ||
|
|
||
| ## Evaluation Process | ||
|
|
||
| ### Phase 1: Extract Project Fingerprint | ||
|
|
||
| Analyze the technology blueprint or codebase to extract: | ||
|
|
||
| ``` | ||
| PROJECT_FINGERPRINT: | ||
| - Primary Language(s): [e.g., C#, TypeScript, Python] | ||
| - Framework(s): [e.g., .NET 8, Azure Functions v4, ASP.NET Core] | ||
| - Architecture Pattern(s): [e.g., Clean Architecture, CQRS, MediatR] | ||
| - Database(s): [e.g., SQL Server, PostgreSQL, Cosmos DB] | ||
| - Messaging: [e.g., Azure Service Bus, Kafka, RabbitMQ] | ||
| - Cloud Platform: [e.g., Azure, AWS, GCP] | ||
| - CI/CD: [e.g., Azure DevOps, GitHub Actions] | ||
| - Testing: [e.g., NUnit, xUnit, Jest, pytest] | ||
| - Key Libraries: [e.g., Polly, Hangfire, EF Core, MediatR] | ||
| - Hosting Model: [e.g., Serverless, Containers, VMs] | ||
| - Has Frontend: [Yes/No — what framework] | ||
| - Has AI/ML: [Yes/No — what kind] | ||
| - Data Sensitivity: [e.g., PII, PHI, PCI, None] | ||
| - Issue Tracking: [e.g., GitHub Issues, Azure DevOps, Jira] | ||
| ``` | ||
|
|
||
| ### Phase 2: Fetch and Parse Skill Catalog | ||
|
|
||
| For each skill in the catalog, extract: | ||
| - **Name**: The skill identifier | ||
| - **Description**: What it does | ||
| - **Technology Scope**: What languages/frameworks/platforms it targets | ||
| - **Use Case**: When to invoke it | ||
|
|
||
| ### Phase 3: Apply Relevance Criteria | ||
|
|
||
| For each skill, apply these decision rules in order: | ||
|
|
||
| | Rule | Result | | ||
| |------|--------| | ||
| | Skill targets a **different language** (e.g., Java skill for a Python project) | **No** | | ||
| | Skill targets a **different platform** (e.g., AWS skill for Azure project) | **No** | | ||
| | Skill targets a **different framework** not in the stack (e.g., React for backend-only) | **No** | | ||
| | Skill targets a **specific product** not used (e.g., Salesforce, Qdrant, Power BI) | **No** | | ||
| | Skill is **language-agnostic process/workflow** applicable to software development | Evaluate further | | ||
| | Skill targets the **exact technology** in the stack | **Yes** | | ||
| | Skill addresses a **cross-cutting concern** present in the project (security, docs, testing) | **Yes** | | ||
| | Skill targets **Azure services** used by the project | **Yes** | | ||
| | Skill is a **meta/productivity** tool with no project-specific benefit | **No** | | ||
| | Skill addresses **planning/architecture** applicable to the architecture pattern | **Yes** | | ||
|
ashtewari marked this conversation as resolved.
|
||
|
|
||
| ### Phase 4: Generate Output | ||
|
|
||
| Produce a markdown table with columns: | ||
|
|
||
|
ashtewari marked this conversation as resolved.
|
||
| ```markdown | ||
| | # | Skill Name | Relevant | Reasoning | | ||
| |---|---|---|---| | ||
| | 1 | skill-name | Yes/No | One-line explanation tied to project fingerprint | | ||
| ``` | ||
|
|
||
| After the table, include: | ||
| - **Summary**: X relevant / Y not relevant out of Z total | ||
| - **Top 10 Recommended**: The most impactful relevant skills, ordered by likely value | ||
| - **Categories Breakdown**: Group relevant skills by category (Development, Testing, Security, Documentation, DevOps, Architecture) | ||
|
|
||
| ## Output Format | ||
|
|
||
| ```markdown | ||
| # Skill Relevance Assessment — {Project Name} | ||
|
|
||
| **Generated:** {date} | ||
| **Catalog Source:** {url or description} | ||
| **Project:** {name} | {primary language} | {framework} | {cloud platform} | ||
|
|
||
| ## Project Fingerprint | ||
|
|
||
| {extracted fingerprint} | ||
|
|
||
| ## Relevance Assessment | ||
|
|
||
| | # | Skill Name | Relevant | Reasoning | | ||
| |---|---|---|---| | ||
| | ... | ... | ... | ... | | ||
|
|
||
| ## Summary | ||
|
|
||
| - **Total Skills Evaluated:** {N} | ||
| - **Relevant:** {X} ({percentage}%) | ||
| - **Not Relevant:** {Y} ({percentage}%) | ||
|
|
||
| ## Top 10 Recommended Skills | ||
|
|
||
| 1. **{skill}** — {why it's high-value for this specific project} | ||
| 2. ... | ||
|
|
||
| ## Relevant Skills by Category | ||
|
|
||
| ### Development ({count}) | ||
| - ... | ||
|
|
||
| ### Testing ({count}) | ||
| - ... | ||
|
|
||
| ### Security ({count}) | ||
| - ... | ||
|
|
||
| ### Documentation ({count}) | ||
| - ... | ||
|
|
||
| ### DevOps & Deployment ({count}) | ||
| - ... | ||
|
|
||
| ### Architecture & Planning ({count}) | ||
| - ... | ||
| ``` | ||
|
|
||
| ## Constraints | ||
|
|
||
| - Never mark a skill as relevant solely because it's "generally useful" — tie every Yes to a specific technology, pattern, or concern in the project fingerprint | ||
| - If the project has no frontend, ALL frontend skills are No | ||
| - If the project uses Language X, skills for Language Y are No (unless the skill is language-agnostic) | ||
| - Platform-specific skills (AWS, GCP) are No for projects on a different cloud | ||
| - When uncertain, default to No with reasoning "Not directly applicable — {gap}" | ||
| - Prefer fetching the latest skill catalog rather than relying on cached/memorized lists; if browsing/fetching is unavailable or the catalog cannot be retrieved, ask the user to paste the catalog contents or provide a local file path, then evaluate from that source | ||
| - Output the full table — do not truncate or summarize with "and N more..." | ||
|
|
||
| ## Example Invocations | ||
|
|
||
| ``` | ||
| User: Evaluate awesome-copilot skills for this project | ||
| User: Which copilot skills should I install for this repo? | ||
| User: Run a skill relevance check against my Technology_Stack_Blueprint.md | ||
| User: Assess all awesome-copilot skills — output a table with yes/no and reasoning | ||
| ``` | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.