VS Code extension: ship portable, built-in custom agents (codeql-query-developer, codeql-workshop-author)#281
Merged
Conversation
10 tasks
Copilot
AI
changed the title
[WIP] Add portable, built-in custom agents to VS Code extension
VS Code extension: ship portable, built-in custom agents (codeql-query-developer, codeql-workshop-author)
May 18, 2026
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR packages two Copilot Chat custom agents (plus selected prompts/skills) directly into the VS Code extension VSIX, adds a bundling script/config to copy and rename whitelisted content at build time, and expands the extension’s tests/docs to cover and describe the bundled customizations.
Changes:
- Add
bundle:customizationsbuild step that copies curated.agent.md,.prompt.md, and skill content intoagents/,prompts/, andskills/for VSIX packaging (with optional overlay support and a generated manifest). - Contribute bundled agents/prompts/skills via
contributes.chatAgents/chatPromptFiles/chatSkills, and add a “Show Built-in Custom Agents Status” command. - Add unit + integration tests for bundling, presence, and bundled markdown link validity; update docs/changelog and adjust various
.github/*markdown sources.
Show a summary per file
| File | Description |
|---|---|
| extensions/vscode/test/suite/bundled-markdown-links.integration.test.ts | Integration test to validate relative markdown links in bundled customizations. |
| extensions/vscode/test/suite/agents.integration.test.ts | Integration tests for bundled agents presence and contributed manifest entries. |
| extensions/vscode/test/customizations/bundle-customizations.test.ts | Unit tests for bundling behavior, overlays, manifest emission, and link checks. |
| extensions/vscode/src/extension.ts | Adds codeql-mcp.showAgentsStatus command and documents agent contribution approach. |
| extensions/vscode/scripts/bundle-customizations.js | New bundler to copy/rename agents/prompts/skills into VSIX-shipped output dirs + manifest. |
| extensions/vscode/README.md | Documents built-in agents and build-time overlay flow. |
| extensions/vscode/package.json | Adds contributes.chatAgents/chatPromptFiles/chatSkills, new command, and bundling scripts wiring. |
| extensions/vscode/examples/team-customizations/skills/example-team-skill/SKILL.md | Example overlay skill content. |
| extensions/vscode/examples/team-customizations/README.md | Example overlay documentation and folder structure. |
| extensions/vscode/examples/team-customizations/prompts/example-team.prompt.md | Example overlay prompt file. |
| extensions/vscode/examples/team-customizations/agents/example-override.agent.md | Example overlay agent file. |
| extensions/vscode/eslint.config.mjs | Adds timer globals to ESLint environment. |
| extensions/vscode/esbuild.config.js | Adds new integration test entry points to the test bundle build. |
| extensions/vscode/customizations/bundle-customizations.config.js | Whitelist/rename map for prompts and skills to bundle into the extension. |
| extensions/vscode/customizations/agents/ql-mcp-ext-workshop-author.agent.md | New shipped workshop-author agent definition. |
| extensions/vscode/customizations/agents/ql-mcp-ext-query-developer.agent.md | New shipped query-developer agent definition. |
| extensions/vscode/.vscodeignore | Excludes customization sources/examples from VSIX (ships only the generated outputs). |
| extensions/vscode/.gitignore | Ignores generated agents/, prompts/, skills/, and dist-customizations-manifest.json outputs. |
| CHANGELOG.md | Adds [Unreleased] entry describing the new VS Code extension customization packaging. |
| .markdownlintignore | Ignores generated extension customizations outputs for markdownlint. |
| .github/skills/validate-ql-mcp-server-tools-queries/SKILL.md | Updates links/related resources to avoid broken references. |
| .github/skills/create-codeql-query-development-workshop/workshop-structure-reference.md | Markdown formatting adjustments (code fences/language tags). |
| .github/skills/create-codeql-query-development-workshop/stage-decomposition-examples.md | Markdown formatting adjustments (headings). |
| .github/skills/create-codeql-query-development-workshop/SKILL.md | Markdown formatting adjustments and removes “Related Skills” section. |
| .github/skills/create-codeql-query-development-workshop/mcp-tools-reference.md | Markdown formatting adjustments (code fence language tags). |
| .github/skills/create-codeql-query-development-workshop/java-quality-queries-workshop-1/README.md | Markdown formatting + link updates. |
| .github/skills/create-codeql-query-development-workshop/examples/example-cpp-simple/README.md | Markdown formatting adjustments. |
| .github/skills/create-codeql-query-development-workshop/examples/codeql-sap-js-ui5-xss/README.md | Markdown formatting adjustments. |
| .github/prompts/validate-ql-mcp-server-tools-via-workshop.prompt.md | Removes model: from prompt frontmatter. |
| .github/prompts/ql-mcp-server-fix-build-and-test.prompt.md | Removes model: from prompt frontmatter. |
| .github/agents/ql-mcp-tool-tester.md | Removes model: from agent frontmatter. |
| .github/agents/ql-mcp-tool-developer.md | Removes model: from agent frontmatter. |
| .github/agents/ql-agent-skills-developer.md | Removes model: from agent frontmatter. |
| .github/agents/mcp-enabled-ql-workshop-developer.md | Removes model: from agent frontmatter. |
| .github/agents/mcp-enabled-ql-query-developer.md | Removes model: from agent frontmatter. |
Copilot's findings
- Files reviewed: 35/35 changed files
- Comments generated: 9
…le packaging - Add codeql-query-developer and codeql-workshop-author .agent.md files - Add bundle-customizations.js script with overlay support - Add AgentRegistrar class managing chat.agentFilesLocations registration - Add bundle:customizations npm script; update vscode:prepublish and clean - Add codeql-mcp.agents.enabled and additionalAgentDirs settings - Add codeql-mcp.showAgentsStatus command - Add contributes.chatAgents, chatPromptFiles, chatSkills static declarations - Add Vitest unit tests for bundler and AgentRegistrar - Add Mocha integration test suite for agents - Add team-customizations example overlay - Update README.md with Built-in Custom Agents section - Update CHANGELOG.md [Unreleased] with new VS Code Extension features Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com>
…ills/, manifest) Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com>
…n tests Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com>
Switch to `contributes.chatAgents` (declarative) only — the same channel GitHub.copilot-chat resolves its own Plan/Ask/Explore agents through. Changes: - Remove `src/customizations/agent-registrar.ts` and its unit tests. - Drop `codeql-mcp.agents.enabled` and `codeql-mcp.additionalAgentDirs` settings (unsupported; users set `chat.agentFilesLocations` to a workspace-relative path directly). - Repurpose `codeql-mcp.showAgentsStatus` to report agents from `contributes.chatAgents` in the manifest. - Add red→green integration tests: assert no absolute path leaks into `chat.agentFilesLocations`; assert `contributes.chatAgents` lists both bundled agents and that each referenced file exists on disk. - Remove stale toggle/`additionalAgentDirs` integration tests. - Update README + CHANGELOG to reflect declarative-only registration. - Strip `model: Claude Opus 4.6 (copilot)` from the two `.github/prompts/` files that still set it (no static model in any agent or prompt). Tests: 185/185 unit, 78/78 integration, lint clean.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Nathan Randall <70299490+data-douser@users.noreply.github.com>
Across the VS Code extension bundler and the server-side CLI executor,
rename "whitelist" to "allowlist" (and follow the same convention for any
future "blacklist" → "denylist") in:
- Doc comments and header descriptions
- JSDoc and inline comments
- Identifiers:
- bundle-customizations.js:
promptWhitelist → promptAllowlist
skillWhitelist → skillAllowlist
- Error messages
- Test descriptions and comments
Files updated:
- extensions/vscode/customizations/bundle-customizations.config.js
- extensions/vscode/scripts/bundle-customizations.js
- extensions/vscode/test/customizations/bundle-customizations.test.ts
- server/src/lib/cli-executor.ts
- server/test/src/lib/cli-executor.test.ts
Out of scope (untouched):
- server/dist/*.js.map (generated; will pick up new strings on next build)
- server/ql/rust/**/*.testproj/**/rust/library/core/**.rs (vendored Rust
stdlib test fixtures)
- .codeql/ql-mcp/query-logs/** (gitignored runtime logs)
- client/gh-ql-mcp-client (compiled binary)
Tests: 186/186 vscode unit, 78/78 server cli-executor unit, lint clean.
No behavior changes.
Replace placeholder NNN PR references with the actual PR number, now that the bundled-customizations entries sit in the [Unreleased] section alongside the supply-chain hardening work from #279.
data-douser
force-pushed
the
copilot/ship-portable-custom-agents
branch
from
ab091e7 to
6709cc1
Compare
- README: tighten the 'Extending at Build Time' intro so the framing matches the limitation documented just below (manifest patch is required for net-new contributions; overlays only override or extend the already-contributed list). - agents.integration.test.ts: rewrite the obsolete file header. It used to claim the tests verify chat.agentFilesLocations is 'updated correctly', but the suite now asserts the extension does NOT write the bundled-agents path there (VS Code rejects absolute paths). - bundle-customizations.test.ts: the 'real source tree' link-validity test now copies only the allowlisted prompts and skill directories (read from the actual bundler config) instead of the entire server/src/prompts/ and .github/skills/ trees. Run time on this test drops from ~1.2s to ~0.7s and the I/O is bounded by the allowlist rather than by the size of the example workshop fixtures. Tests: 186 unit (all pass); integration noWorkspace 83/83 exit 0.
- extension.ts: extract `readBundledAgentsStatus(context)` helper that reads the manifest from `context.extensionUri.fsPath` directly, removing the previous `context.extension.id` dependency. `showAgentsStatus` now calls the helper, and the same helper is exposed on `ExtensionApi.getBundledAgentsStatus()` for test introspection. - agents.integration.test.ts (TDD): new test asserts the extension API exposes `getBundledAgentsStatus()` and that it returns both bundled agent paths (`ql-mcp-ext-query-developer.agent.md`, `ql-mcp-ext-workshop-author.agent.md`) plus a `bundledDir` resolved via `extensionUri`. Red against the previous code, green after the helper extraction. - CHANGELOG.md: reorder `[Unreleased]` sections to match the repo's changelog guidance (`Highlights` -> `Added` -> `Changed` -> `Fixed` -> `Security` -> `Dependencies`). The new `Added > VS Code Extension` bullets now sit immediately after `Highlights`. Tests: 186 unit (vitest); integration noWorkspace 84, singleFolder 86, multiRoot 89; all exit 0.
…gents Workflow prompts now ship exclusively via the `ql-mcp` MCP server's `prompts/list` and surface as slash commands in Copilot Chat. The four bundled `ql-mcp-ext-*` `.prompt.md` files duplicated content the server already exposes under canonical IDs (`/ql_tdd_basic`, etc.), creating two slash commands for the same workflow. Bundled agents and skills are unchanged in form; only their prose changes to reference a richer palette of MCP slash IDs. Changes: - bundle-customizations.config.js: `prompts` allowlist is now empty. Header doc explains why and notes the array is kept for future reversibility. - bundle-customizations.js: prompts copy loop is preserved but gated on a non-empty allowlist; the `prompts/` target dir is only created when at least one prompt would be copied. - package.json: drop `contributes.chatPromptFiles` entirely. - customizations/agents/ql-mcp-ext-query-developer.agent.md: split the old 'Bundled Skills and Prompts' section into 'Bundled Skills' (2 bullets) and 'MCP Prompts' (7 bullets) referencing canonical slash IDs. Adds `/ql_lsp_iterative_development`, `/explain_codeql_query`, `/document_codeql_query`, `/data_extension_development` to the workflow palette. - customizations/agents/ql-mcp-ext-workshop-author.agent.md: same split. The MCP Prompts list now has 5 bullets, adding `/explain_codeql_query` and `/document_codeql_query`. - README.md 'Built-in Custom Agents': describes agents + skills only; a single sentence points users at the MCP-served slash commands. - CHANGELOG.md [Unreleased]: rename the 'Bundled prompts and skills' bullet to 'Bundled skills'; add a 'Changed > VS Code Extension' bullet that explains the prompts-via-MCP migration. TDD (new integration tests in agents.integration.test.ts): - 'package.json does NOT contribute chatPromptFiles' — asserts `contributes.chatPromptFiles` is absent or empty. - 'Bundled prompts/ directory is not produced by the bundler' — asserts `<extensionPath>/prompts/` does not exist. - 'Every MCP prompt slash ID referenced by shipped agents maps to a real MCP prompt' — scans each agent's body for `/<id>` tokens, asserts each is in the 15-ID set registered by server/src/prompts/workflow-prompts.ts, and asserts each agent references at least one. All three failed against the previous tip and pass now. Also: bundled-markdown-links.integration.test.ts splits its 'required' vs 'optional' bundled-dir lists so the absence of the `prompts/` dir no longer trips the vacuous-pass guard. Tests: 186 unit (vitest); integration noWorkspace 87, singleFolder 89, multiRoot 92; all exit 0.
- src/extension.ts: extend `ExtensionApi` to include `environmentBuilder` and `serverManager`. The activate() return value was always shaped this way (the bridge/workspace integration tests rely on these hooks), but the type only listed `mcpProvider` and `getBundledAgentsStatus`, which is an excess-property violation under TS strict checks. Document both as part of the public contract. - test/suite/extension.integration.test.ts: new TDD assertion that the runtime `ExtensionApi` exposes `mcpProvider`, `environmentBuilder`, `serverManager`, and a callable `getBundledAgentsStatus`. Guards against any future drift between the declared interface and the returned shape. - README.md "Adding Your Own Agents at Runtime": clarify exactly which path forms `chat.agentFilesLocations` accepts (workspace-relative and home-relative `~/foo` form) and which it rejects (absolute, drive-letter, backslash separators, glob characters). Resolves the reviewer's complaint that `~/.copilot/agents` was listed as a workspace-relative example. - CHANGELOG.md `[Unreleased]` Highlights: condense the second supply-chain hardening bullet to a brief lead-in that points at the Security section, eliminating the duplicate prose previously flagged by repo changelog rules (no entry repeated across multiple sections). Tests: 186 unit (vitest); integration noWorkspace 88, singleFolder 90, multiRoot 93; all exit 0. The three remaining unresolved reviewer comments (`extension.ts:85`, `cli-executor.ts:35`, `.github/agents/...:4`) all concern the PR description text on GitHub claiming a narrower scope than the actual diff. The code is consistent; the PR description should be updated via the GitHub UI.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Nathan Randall <70299490+data-douser@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Copilot's findings
Comments suppressed due to low confidence (1)
server/src/lib/cli-executor.ts:43
- The PR description/scope checklist claims there are no
server/src/changes, but this PR updates server code here (and updates the bundledserver/distoutput accordingly). Please either update the PR description to include these server changes in scope, or split the server allowlist/wording changes into a separate PR.
- Files reviewed: 38/40 changed files
- Comments generated: 3
data-douser
approved these changes
May 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #280.
Outline of Changes
Agent and Prompt Configuration Updates:
modelfield from the following agent and prompt YAML files to decouple agent definitions from a specific language model:.github/agents/mcp-enabled-ql-query-developer.md.github/agents/mcp-enabled-ql-workshop-developer.md.github/agents/ql-agent-skills-developer.md.github/agents/ql-mcp-tool-developer.md.github/agents/ql-mcp-tool-tester.md.github/prompts/ql-mcp-server-fix-build-and-test.prompt.md.github/prompts/validate-ql-mcp-server-tools-via-workshop.prompt.mdDocumentation and Formatting Improvements:
.github/skills/create-codeql-query-development-workshop/stage-decomposition-examples.mdfrom bold to proper markdown headings for better structure and readability. [1] [2] [3] [4]Resource Link Updates:
Content Cleanup:
These changes improve maintainability, clarity, and consistency across agent definitions and workshop documentation.