Skip to content

Run benchmarks during Pages build so the benchmarks page shows real data#159

Merged
mrjf merged 2 commits into
mainfrom
copilot/regenerate-benchmarks-results-json
Apr 21, 2026
Merged

Run benchmarks during Pages build so the benchmarks page shows real data#159
mrjf merged 2 commits into
mainfrom
copilot/regenerate-benchmarks-results-json

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 21, 2026

benchmarks/results.json on main is the empty stub, so benchmarks.html renders "No benchmark data available yet". PR #154 tried to regenerate it inside the perf-comparison autoloop, but that sandbox has no bun, so every iteration evaluates to 0 and the autoloop is stuck.

Move benchmark execution to the Pages workflow (which already has Bun) and revert the autoloop's evaluation to the pre-#154 file-count metric.

  • .github/workflows/pages.yml — Setup Python earlier, run bash benchmarks/run_benchmarks.sh, then unconditionally copy the regenerated benchmarks/results.json into the playground artifact. The existence guard is dropped — if benchmarks fail, the build should fail loudly.

  • .autoloop/programs/perf-comparison/program.md — Restore the original Evaluation section (counts min(bench_*.ts, bench_*.py) files) so iterations can ratchet again without needing to execute benchmarks. Update the iteration prose to note that the Pages workflow regenerates results.json on merge to main; the autoloop only needs to add benchmark scripts.

Trade-offs

  • Pages builds get slower (~10–30 min for 600 pairs), but only on push to main, so PRs aren't blocked. Can be sharded with a matrix later if painful.
  • Broken benchmarks surface on merge instead of in-PR. Acceptable since the autoloop still validates that benchmark files are syntactically valid.

…mparison eval to file-count

Agent-Logs-Url: https://github.com/githubnext/tsessebe/sessions/bc8c8dec-26dd-46a4-89e0-62e7f4d245c3

Co-authored-by: mrjf <180956+mrjf@users.noreply.github.com>
Copilot AI changed the title [WIP] Regenerate benchmarks/results.json during Pages build Run benchmarks during Pages build so the benchmarks page shows real data Apr 21, 2026
Copilot AI requested a review from mrjf April 21, 2026 03:59
Copilot finished work on behalf of mrjf April 21, 2026 03:59
@mrjf mrjf marked this pull request as ready for review April 21, 2026 04:11
@mrjf mrjf merged commit e49fb2b into main Apr 21, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Regenerate benchmarks/results.json during Pages build so the benchmarks page shows real data

2 participants