feat: new manifest format

[atilafassina]

Summary

Evolves the AppKit plugin manifest contract from v1.0 to v2.0 and swaps the canonical authoring surface from hand-written JSON Schema to Zod via the Standard Schema interface. AJV, the json-schema-to-typescript codegen pipeline, and the standalone semantic-validator delete; refinement and .transform() on Zod schemas replace them.

Mid-review (2026-05-18) the design picked up a substitutability-gate amendment that deleted postScaffold entirely in favor of a unified plugin-level scaffolding.rules block, and audited every prose rule against the gate. See "2026-05-18 amendment" below.

Important

~91% of insertions are auto-regenerated artifacts — two JSON Schema files (docs/static/schemas/{plugin-manifest,template-plugins}.schema.json) account for +10,400 lines.

.gitattributes marks all generated artifacts linguist-generated=true. GitHub collapses them in the file list — substantive review surface is ~32 files / ~2,300 insertions.

What changes vs main

Contract

• discoveryDescriptor is a discriminated union: { type: "kind", resourceKind, ... } | { type: "cli", cliCommand, ... }. Six known kinds: warehouse, genie_space, volume, postgres_project, postgres_branch, postgres_database.
• cli variant rejects shell metacharacters (;, |, &, backtick, $, newlines) on cliCommand and shortcut. .describe() notes direct authors to kind for first-party resources and flag the cli shape as minimal-and-may-tighten.
• RESOURCE_KIND_COMMANDS map next to the schema; commands verified against Databricks CLI v0.299.0. volume kind carries parents: ["catalog", "schema"] — declares free-text prompts the runner must collect before invoking discovery (replaces the prior prose rule).
• origin on template fields is a .transform() output (drift-impossible).
• rules.must / rules.should / rules.never items capped at maxLength: 120. Per-bucket and cross-bucket dedup enforced via .superRefine().
• 4 core plugin manifests (analytics, files, genie, lakebase) migrated to the kind variant; lakebase gains a project field.

Authoring surface

• New: packages/shared/src/schemas/manifest.ts (Zod schemas + RESOURCE_KIND_COMMANDS + TEMPLATE_SCAFFOLDING).
• New: tools/generate-json-schema.ts (Zod → draft-07 JSON Schema, emits to docs/static/schemas/).
• New runtime dep on packages/shared: @standard-schema/spec.

Deletions

• Deps removed: ajv, ajv-formats, json-schema-to-typescript.
• packages/shared/src/schemas/{plugin-manifest.schema.json,template-plugins.schema.json,plugin-manifest.generated.ts}.
• tools/generate-schema-types.ts, docs/scripts/copy-schemas.ts.
• validate-manifest.ts (498 → 177 lines): AJV plumbing + runSemanticValidation + helpers.
• enrichFieldsWithOrigin, validateDiscoveryOrigin.
• postScaffoldStepSchema and postScaffold field (post-amendment — see below).

Migrated to Zod-direct

• schema-resources.ts and tools/generate-registry-types.ts (no more runtime JSON-schema reads).
• manifest-types.ts is now a re-export shim (z.infer types + StandardSchemaV1).

Plugin Rules (per core plugin, scaffolding.rules)

Plugin	must	should
analytics	Before init, ensure the SQL Warehouse passed via --set analytics.sql-warehouse.id is running	After init, ensure config/queries/ has at least one .sql file before running npm run typegen
files	Before init, verify your Unity Catalog volume exists and you have WRITE_VOLUME permission	—
genie	After init, configure the 'spaces' map in plugin config with alias-to-Space-ID mappings	—
jobs	(no rules)	(no rules)
lakebase	After init, run database migrations via 'pnpm drizzle:migrate' before first request	After init, verify Lakebase connectivity with 'psql $PGHOST -c "select 1"'

Template-level scaffolding.rules

{
  "must": [
    "Keep all secrets and credentials only in app.yaml, databricks.yml, and/or .env"
  ],
  "should": [
    "ask user when in doubt of resource to use for plugin"
  ],
  "never": [
    "guess resources when multiple or no options are available",
    "embed secrets in files that will go to the client-bundle"
  ]
}

Each entry passes the substitutability gate (describes a cross-cutting agent behavior at decision time, not derivable from structured manifest data). Each ≤120 chars. Conflict-detection-safe against the current plugin rule corpus (no must ↔ never overlaps).

2026-05-18 amendment — substitutability gate

The original v2.0 plan shipped postScaffold arrays ({ instruction, required }) at the plugin level. During v2.0 implementation we found the format couldn't participate in the databricks-apps agent skill's command-construction flow — prose doesn't compose across plugins, duplicates work the skill drives from structured resources, and { instruction, required } adds zero parseable signal.

The amendment introduces one design principle and applies it uniformly:

A prose rule in the manifest is legitimate only if it cannot be expressed as structured data on a resource, plugin config, or template.

Changes that landed under the amendment:

• postScaffold deleted. postScaffoldStepSchema and the postScaffold field removed from both pluginManifestSchema and the template plugin schema. All four core plugins migrated to plugin-level scaffolding.rules.
• Plugin-level scaffolding.rules with must / should / never arrays (≤120 chars each, no duplicates within a bucket, no overlap across buckets — all enforced via .superRefine()).
• Template-level scaffoldingRulesSchema gains should[]? for parity with plugin-level.
• TEMPLATE_SCAFFOLDING audit — original 4 must entries pruned (every one was substitutable: skill content, derivable from requiredByTemplate, derivable from field.env, or a discovery-descriptor concern). Original 3 never entries replaced with 2 sharper rules covering credentials hygiene and resource-selection decision points; one entry deleted as unreachable. See "Template-level scaffolding.rules" above.
• A6 schema-extension decisions:
  • (a) Volume catalog+schema prerequisite → EXTEND: RESOURCE_KIND_COMMANDS.volume.parents = ["catalog", "schema"]. Models transient query inputs as a kind-level property without conflating with dependsOn (which references sibling fields).
  • (b) Config-driven --set derivation for genie spaces → ACCEPT AS PROSE pending Track B confirmation of the skill's consumption model.
  • (c) Liveness preconditions (analytics warehouse) → ACCEPT AS PROSE. Runtime state checks aren't structural data.
• Per-plugin rule audit: 7 substitutability-gate violations introduced during experimentation were caught and reverted (permission duplications, resource-existence tautologies, inactionable-at-scaffold-time directives). Net effect: zero new plugin rules survive the gate that weren't already in the post-Phase-1 baseline.

.claude/commands/ skill extensions

Outside the packages/ tree but relevant to reviewers — the in-repo plugin commands gained procedural v2.0 semantic checks:

• audit-core-plugin.md — Step 3.6 (3.6a substitutability-gate patterns, 3.6b discovery descriptor completeness, 3.6c parents vs dependsOn consistency). All findings feed Category 1 (Manifest Design).
• review-core-plugin.md — Step 5.6 with the same checks scoped to changed manifest files only.
• create-core-plugin.md — Step 4e.1 post-scaffold enrichment that walks each user-supplied field to add a discovery block and gates any scaffolding.rules against 5 explicit reject patterns.

No changes to plugin-best-practices.md or plugin-review-guidance.md — the additions are procedural workflow steps, not narrative rules.

Track B — consumer skill (cross-repo)

The databricks-apps skill in databricks/databricks-agent-skills#79 ("docs(apps): update AppKit scaffolding setup for Manifest 2.0") adds a new "Scaffolding Rules Protocol" subsection to skills/databricks-apps/SKILL.md (skill version bump 0.1.1 → 0.1.2). It consumes the rules emitted by this PR:

1. Gather rules from selected --features plugins + every plugin with requiredByTemplate: true + template-level
2. Precedence — manifest rules override skill-baked directives
3. Phase ordering — Before init / After init prefixes
4. Conflict detection — plugin must ↔ template never stops the flow
5. Reporting — surface merged guardrails to the user before init

Sequencing: Track A (this PR) ships first. Track B is opened, not blocking merge — plugin-level rules are advisory metadata in the transition state. Failure mode if Track B lands later: degraded (third-party plugin guardrails don't fire), not broken (resources still drive the build).

End-to-end integration runbook lives in ~/.xavier/research/manifest-2-skill-integration-test.md (vault note, not in this repo).

Deferred to follow-up PRs

Change	Note
TypeScript-authored manifests (defineManifest({ ... }))	Future work; no stacked PR yet
Docs for the v2.0 format	#370 (open, branch manifest-docs)
Branded field-name keys for compile-time dependsOn checks	Not a must-have, larger implementation change
cli variant full hardening	Shell-metacharacter denylist + describe notes shipped here. Remaining: argv-array form (executor-side spawn-not-shell), output-shape contract (replace jq path with declared field names + unwrap). Tied to the executor PR.
Track B skill PR	databricks-agent-skills#79 — opens for skill author review; not required to merge before Track A

Note

Plugin authors continue to write manifest.json. See the docs PR (#370) for the authoring guide.

Note on the generated files diff

1. Zod emits keys in a different order than the hand-written JSON Schema did. For example, you'll see "description" flip from before "type" to after "type", and "required": [...] arrays appear in different positions. z.toJSONSchema() has its own key ordering, and that's what now writes the file.
2. Discovery is reshaped as a discriminated union. That's why $ref: "#/$defs/resourceRequirement" got expanded into giant oneOf: [...] blocks per resource type — the schema's actual shape changed, not just its layout.

Biome's JSON formatter only normalizes whitespace; it doesn't reorder keys. So the noise you're seeing is the cost of moving from hand-authored JSON Schema → Zod-generated JSON.

[Copilot]

Pull request overview

This PR upgrades the AppKit plugin manifest ecosystem to a v2.0 template manifest format to support smarter databricks apps init scaffolding, including field discovery metadata, post-scaffold user instructions, computed field origins, and semantic (cross-field) validation during plugin validate.

Changes:

• Bump template plugin manifest version to 2.0 and add a top-level scaffolding descriptor.
• Extend plugin/template schemas with discovery (CLI command-based value discovery) and postScaffold steps; generate/propagate computed origin into template manifests during sync.
• Add semantic validation (dependsOn cycles/dangling refs, <PROFILE> placeholder, discovery/origin coherence, postScaffold structure) and associated tests/docs updates.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
template/appkit.plugins.json	Updates template manifest to v2.0 and annotates built-in plugins with discovery/origin/postScaffold plus scaffolding descriptor.
packages/shared/src/schemas/template-plugins.schema.json	Expands template manifest schema to support v2.0 and scaffolding descriptor; inlines field/requirement defs for origin.
packages/shared/src/schemas/plugin-manifest.schema.json	Adds discovery to resource fields and postScaffold to plugin manifests.
packages/shared/src/schemas/plugin-manifest.generated.ts	Updates generated TS types to include discovery/postScaffold types.
packages/shared/src/plugin.ts	Re-exports new generated types (DiscoveryDescriptor, PostScaffoldStep).
packages/shared/src/cli/commands/plugin/validate/validate.ts	Runs semantic validation and formats semantic errors/warnings in CLI output.
packages/shared/src/cli/commands/plugin/validate/validate-manifest.ts	Implements semantic validation rules and issue formatting.
packages/shared/src/cli/commands/plugin/validate/validate-manifest.test.ts	Adds test coverage for new semantic validation behavior.
packages/shared/src/cli/commands/plugin/sync/sync.ts	Computes/injects origin, bumps template manifest to v2.0, and adds scaffolding descriptor on write.
packages/shared/src/cli/commands/plugin/sync/sync.test.ts	Adds unit tests for origin computation.
packages/shared/src/cli/commands/plugin/manifest-types.ts	Adds scaffolding descriptor types and exports new manifest-related types.
packages/appkit/src/plugins/lakebase/manifest.json	Adds discovery descriptors and postScaffold steps to the lakebase built-in plugin manifest.
packages/appkit/src/plugins/genie/manifest.json	Adds schema reference, discovery descriptor, and postScaffold steps to genie plugin manifest.
packages/appkit/src/plugins/files/manifest.json	Adds discovery descriptor and postScaffold steps to files plugin manifest.
packages/appkit/src/plugins/analytics/manifest.json	Adds discovery descriptor and postScaffold steps to analytics plugin manifest.
docs/static/schemas/template-plugins.schema.json	Publishes updated template plugins schema for docs site.
docs/static/schemas/plugin-manifest.schema.json	Publishes updated plugin manifest schema for docs site.
docs/static/appkit-ui/styles.gen.css	Updates generated UI styles (tailwind output) used by docs UI.
docs/docs/api/appkit/Interface.ResourceFieldEntry.md	Documents the new discovery field on ResourceFieldEntry.
docs/docs/api/appkit/Interface.PluginManifest.md	Documents the new postScaffold field on PluginManifest.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[keugenek]

@atilafassina this one looks ok and evolution, however little heavy code-wise (let me post comments from isaac about these) and mixing concepts little bit. What do you think of this:

Alternatives worth weighing

Zod as single source of truth (my default for a TS-first ecosystem):

• Types, runtime validation, and semantic checks in one file.
• zod-to-json-schema emits the $schema JSON for VSCode intellisense.
• Cycle/dangling-ref checks become .refine() calls in the same module as the
  schema.
• Deletes plugin-manifest.generated.ts and the Ajv layer.

TS-authored manifests compiled to JSON:

• manifest.ts → defineManifest({...}) with branded types for field names
  (compile-time dependsOn checking, no DFS needed).
• sync lowers to .json for non-TS consumers.
• Strongest author ergonomics; agent still reads JSON.

Split concerns into two files:

• manifest.json — strict contract (resources, fields, env), stable schema.
• scaffold.yaml or .md — agent hints (discovery, prompts, postScaffold), free
  to evolve.
• Today they're fused, which is why the schema keeps growing.

[keugenek]

Overall direction looks right — v2.0 versioning via if/then is correct and co-located manifest.json per plugin is the right locality. A few shape-level concerns worth addressing before or soon after this lands. Happy to chat on any of them.