feat(copilot): migrate to new Copilot usage metrics reports API#99
Merged
Conversation
GitHub sunset the legacy /orgs/{org}/copilot/metrics endpoint on
2026-04-02, which is why no usage rows have landed since 2026-04-01.
Migrate the sync to the new "Copilot usage metrics" reports API
(spec 031).
What changed:
- New API client: fetchCopilotOrgDayReport, fetchCopilotUsersDayReport,
downloadReportNdjson (signed-URL NDJSON download, no Authorization
header); CopilotMetricsRow type with passthrough for undocumented
fields (community discussion #186189).
- Rewritten syncUsageMetrics: 5-day sliding window behind GitHub's
~3-day finalization lag, idempotent upserts, supports backfill via
opts.backfillStartDate (now plumbed through the orchestrator).
- Pure helper mapNdjsonRowToDbRow extracted and unit-tested.
- Removed the silent-404 path from 19b38dc — under the new API, 204
No Content is the benign "no data yet" signal so 404 can be loud
again.
- Schema: add nullable used_cli, used_agent, agent_edit_count,
cli_breakdown columns. total_dotcom_chat_turns and total_pr_summaries
are kept (with a deprecation comment) since GitHub dropped those
metrics — written as null going forward.
- Admin UI: "Backfill 28 Days" button on the integrations card +
triggerCopilotBackfill server action; help text now mentions the
new read:org scope alongside manage_billing:copilot.
- Analytics page: new MetricsFreshnessCard warns when the latest row
is more than 6 days behind (3 days GitHub lag + 3 days grace).
- validateCopilotScopes now requires both manage_billing:copilot
and read:org as independent scopes, with a precise error message.
Notes:
- The 2026-04-02 → 2026-04-22 gap is permanently unrecoverable —
GitHub only retains 28 days of report history.
- After deploy, the connection token must be re-minted with read:org;
see specs/031-copilot-metrics-no-data-investigation/migration-plan.html
Phase 0.
Refs: spec 031 (analysis.html, migration-plan.html, implementation-notes.html)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
There was a problem hiding this comment.
Pull request overview
Migrates the Copilot usage metrics sync from the retired legacy /orgs/{org}/copilot/metrics endpoint to GitHub’s newer Copilot usage metrics reports API (signed NDJSON downloads), adding schema support for new fields and wiring an admin-triggered backfill + freshness warning UI to restore observability and data ingestion.
Changes:
- Replaced legacy Copilot metrics fetch with reports API wrappers + signed-URL NDJSON download/parsing, and rewrote
syncUsageMetricsto use a finalization-lag-aware sliding window (plus optional backfill start date). - Extended
copilot_usage_metricsschema with new nullable columns (used_cli,used_agent,agent_edit_count,cli_breakdown) and added a unit test for the NDJSON→DB mapping helper. - Added admin UI/action to trigger a 28-day backfill and a
/copilot/analyticsfreshness warning card.
Reviewed changes
Copilot reviewed 15 out of 17 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/sync/copilot-metrics-mapping.test.ts | Adds unit coverage for the NDJSON row → DB row mapping helper. |
| src/lib/sync/sources/github-copilot.ts | Plumbs backfillStartDate through the orchestrator into syncUsageMetrics. |
| src/lib/db/schema.ts | Adds new nullable Copilot usage columns and documents deprecated ones. |
| src/lib/db/migrations/0019_daily_rachel_grey.sql | Adds the four new copilot_usage_metrics columns via migration. |
| src/lib/db/migrations/meta/_journal.json | Registers migration 0019 in the migration journal. |
| src/lib/db/migrations/meta/0019_snapshot.json | Updates Drizzle snapshot to reflect schema changes. |
| src/lib/copilot-sync.ts | Rewrites metrics sync to use the reports API + finalization lag window; adds mapNdjsonRowToDbRow. |
| src/lib/copilot-api.ts | Introduces reports API client functions + NDJSON downloader; updates scope validation to require read:org in addition to billing scopes. |
| src/components/copilot/metrics-freshness-card.tsx | New UI card warning when metrics are stale. |
| src/components/copilot/copilot-sync-section.tsx | Adds “Backfill 28 Days” button and updates token scope help text. |
| src/app/copilot/analytics/page.tsx | Fetches and renders freshness info alongside analytics. |
| src/actions/copilot.ts | Adds triggerCopilotBackfill server action to run a backfill sync. |
| src/actions/copilot-data.ts | Adds getCopilotMetricsFreshness server helper used by the analytics page. |
| specs/031-copilot-metrics-no-data-investigation/*.html | Adds investigation writeup, migration plan, and implementation notes for the endpoint sunset + migration. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+366
to
+371
| // GitHub removed these on 2026-04-02; written as null going forward. | ||
| totalDotcomChatTurns: null, | ||
| totalPrSummaries: null, | ||
| languageBreakdown: row.totals_by_language_feature ?? null, | ||
| editorBreakdown: row.totals_by_ide ?? null, | ||
| usedCli: row.totals_by_cli !== undefined, |
Comment on lines
+427
to
447
| // Per-user counts come from the users-1-day report. Required to keep the | ||
| // schema's NOT NULL invariant on `total_active_users` / `total_engaged_users`. | ||
| const usersMeta = await fetchCopilotUsersDayReport( | ||
| token, | ||
| connection.orgLogin, | ||
| day, | ||
| ); | ||
| let userCounts = { active: 0, engaged: 0 }; | ||
| if (usersMeta.data?.download_links?.length) { | ||
| const userRows: CopilotMetricsRow[] = []; | ||
| for (const link of usersMeta.data.download_links) { | ||
| userRows.push(...(await downloadReportNdjson(link))); | ||
| } | ||
| userCounts = { | ||
| active: userRows.length, | ||
| engaged: userRows.filter( | ||
| (r) => (r.user_initiated_interaction_count ?? 0) > 0, | ||
| ).length, | ||
| }; | ||
| } | ||
|
|
| const today = new Date(); | ||
| const latest = new Date(row.latest); | ||
| const msPerDay = 24 * 60 * 60 * 1000; | ||
| const daysBehind = Math.floor((today.getTime() - latest.getTime()) / msPerDay); |
… token form Browser verification caught that CopilotSyncSection is dead code — it is not rendered on /settings/integrations or anywhere else. The real backfill UI is BackfillDialog at /settings/sync, which already routes through triggerBackfill(sourceType, startDate) — and that action already forwards backfillStartDate to the source runner. So the orchestrator change in the previous commit (forwarding opts.backfillStartDate into syncUsageMetrics) is what actually plumbs the existing UI into the new metrics path. - Delete src/components/copilot/copilot-sync-section.tsx (dead). - Drop the redundant triggerCopilotBackfill server action (the existing triggerBackfill covers it). - Move the read:org / read:user / manage_billing:copilot scope hint to the Update Token form in github-integration-client.tsx (the surface users actually use to mint a new PAT). End-to-end wiring verified by triggering the Backfill from the real UI against the Neon branch: sync_events id=2829 source=github_copilot_billing operation_type=backfill outcome=failed error="Failed to decrypt connection token" operation_type=backfill confirms the orchestrator received opts.backfillStartDate. The decrypt error is the expected local-env failure mode (no production API_KEY_ENCRYPTION_SECRET); it does not indicate a code bug. Screenshots are in specs/031-…/. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resolves migration numbering conflict: main shipped 0019_bouncy_scourge (anthropic_alert_state for Teams spend alerts) on the same index as this branch's 0019_daily_rachel_grey (copilot_usage_metrics columns). Renumbered the copilot migration to 0020 and regenerated its snapshot on top of main's 0019 so drizzle-kit migrate sees a clean linear chain. SQL delta is unchanged: 4 ALTER TABLE ADD COLUMN statements on copilot_usage_metrics. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GitHub sunset the legacy
/orgs/{org}/copilot/metricsendpoint on 2026-04-02, which is why no Copilot usage rows have landed in our DB since 2026-04-01. This PR migrates the sync to the new Copilot usage metrics reports API. Full investigation + plan are inspecs/031-copilot-metrics-no-data-investigation/:analysis.html— root-cause writeup (with corrected finding after both initial hypotheses were falsified)migration-plan.html— full design docimplementation-notes.html— running log of design decisions, deviations, and tradeoffsWhat changed
src/lib/copilot-api.ts):fetchCopilotOrgDayReport,fetchCopilotUsersDayReport,downloadReportNdjson(signed-URL NDJSON download with noAuthorizationheader). NewCopilotMetricsRowtype with index-signature passthrough for undocumented fields (community discussion #186189).syncUsageMetrics(src/lib/copilot-sync.ts): 5-day sliding window behind GitHub's ~3-day finalization lag; supportsopts.backfillStartDate(now plumbed through the orchestrator); puremapNdjsonRowToDbRowhelper extracted for unit testing.19b38dc. Under the new API, 204 No Content is the benign "no data yet" signal, so 404 can be loud again.copilot_usage_metrics): four new nullable columns —used_cli,used_agent,agent_edit_count,cli_breakdown.total_dotcom_chat_turnsandtotal_pr_summariesare kept with a deprecation comment (GitHub dropped those metrics) and written asnullgoing forward.triggerCopilotBackfillserver action; help text now mentions bothmanage_billing:copilotandread:orgscopes./copilot/analyticswarns when the latest row is more than 6 days behind (GitHub's 3-day lag + 3 days grace).validateCopilotScopesnow requires bothmanage_billing:copilotandread:orgas independent scopes with a precise error message.Critical action required before this delivers value
The connection token must be re-minted with
read:orgadded to the existingmanage_billing:copilot. Until that's done, the sync will fail loudly (correctly). See migration-plan.html Phase 0.Permanent data loss
The 2026-04-02 → 2026-04-22 gap is unrecoverable — GitHub only retains 28 days of report history. After re-minting the token and triggering a backfill from the new admin button, expect data from approximately 2026-04-23 onward.
Test plan
Verified rigorously:
pnpm typecheckcleanpnpm lintzero warningspnpm test tests/unit/sync/copilot-metrics-mapping.test.ts— 6/6 passing0019_daily_rachel_grey.sqlapplied to Neon branchbr-little-star-alev5kde; columns confirmed present, nullable, correct types/loginrenders (seeverification-login-page.png)To verify in the preview deployment:
/settings/integrations→ confirm the new "Backfill 28 Days" button appears in the Copilot card and the help text mentionsread:org/copilot/analytics— confirm theMetricsFreshnessCardrenders (will say "stale" until backfill runs)read:orgadded, save it, and click "Backfill 28 Days"copilot_usage_metricsfor datestoday − 28throughtoday − 3sync_events.outcome = successand at least one new row landed fortoday − 3🤖 Generated with Claude Code