Create Plugin / Plugin E2e: Improve agentic e2e testing experience

[sunker]

Problem

AI coding agents are increasingly used to write and maintain e2e tests for Grafana plugins. Today, an agent dropped into a plugin project has no idea how to use @grafana/plugin-e2e. The existing scaffolded instructions.md says "use @grafana/plugin-e2e" but provides no guidance on the API surface - fixtures, page models, custom matchers, provisioning patterns or version-aware selectors.

This means agents can fall back to writing raw Playwright code (page.goto, page.click) instead of idiomatic @grafana/plugin-e2e code (createDataSourceConfigPage, panelEditPage.datasource.set, toBeOK). The result is brittle tests that break across Grafana versions - exactly the problem plugin-e2e was built to solve.

Solution

Two parts, in order of complexity:

Part 1: Plugin-type-specific e2e testing docs for agents (low-hanging fruit)

Ship a testing-focused context file (e.g. tests/AGENTS.md or .config/AGENTS/e2e-testing.md) in newly scaffolded plugins via create-plugin. This file would be plugin-type-specific (datasource, panel, app) and include:

• The @grafana/plugin-e2e fixture API relevant to that plugin type (e.g. datasource plugins get createDataSourceConfigPage, panelEditPage, readProvisionedDataSource; app plugins get gotoAppPage, appConfigPage, the fixtures.ts extension pattern)
• Common test patterns as copyable code examples (config editor, query editor, provisioned resources, error scenarios, mock responses)
• Custom matcher reference (toBeOK, toHaveAlert, toDisplayPreviews)
• Tricky parts: the selectors.apis.DataSource.proxy() URL construction, force: true for checkboxes/switches, provisioning file structure
• Links to the live docs via the llms.txt convention for deeper exploration

This works with every AI tool (no MCP dependency), is version-controlled alongside the plugin and costs nothing at runtime. Templates already exist per plugin type (templates/datasource/, templates/app/, templates/panel/) so this fits the existing scaffolding model.

Part 2: Grafana plugin test agents (planner, generator, healer)

Inspired by Playwright Test Agents, scaffold Claude Code subagent definitions in .claude/agents/ that are specialized for Grafana plugin e2e testing:

• Planner - Reads plugin.json to determine plugin type, knows the standard test surface (config editor, query editor, variables, annotations, alerts for datasources; pages and config for app plugins; options and rendering for panels), produces a structured test plan mapped to @grafana/plugin-e2e fixtures
• Generator - Writes idiomatic @grafana/plugin-e2e tests using the correct fixtures, page models and matchers. Uses Playwright MCP browser tools to verify locators against the live app before writing assertions
• Healer - Understands Grafana-specific failure modes (version-aware selector changes, checkbox actionability quirks, provisioning file mismatches), debugs and fixes broken tests

These are just markdown files (~1,000-1,500 tokens each) that follow the Claude Code subagent format. They reference the existing Playwright MCP tools for browser interaction - no custom MCP server needed. The agents would be templated per plugin type, matching the existing create-plugin template structure.

Additionally, we can create an add cmd script that wires this up for existing plugins.

[mckn]

I think this sounds great! It would be great to try to do this in a standardised way so we don't commit too much to a single platform.

https://agentskills.io/what-are-skills
https://agents.md/

Or use sub-agents but try to reference other docs as much as possible so you don't need to repeat the information multiple times.

[m13v]

the standardized agent approach (agentskills.io, agents.md) is the right call. one thing we've learned building test agents for web apps: the planner agent needs access to both the plugin.json metadata AND a live accessibility snapshot of the running plugin to generate accurate test plans. static analysis alone misses runtime behavior. for the healer agent specifically, Grafana's version-aware selector problem is a great use case for tiered selector fallbacks (data-testid, aria, role, text, css in priority order) with automatic adaptation when the primary fails. the tricky part is distinguishing 'this selector drifted but the element is still there' vs 'this element was genuinely removed in a new Grafana version'.

[m13v]

we've been building exactly this kind of test agent system, the selector fallback and browser orchestration layer we use for multi-browser parallel test execution: https://github.com/m13v/browser-lock/blob/main/playwright-lock.sh