Skip to content

chore: implement a project overhead scale test#615

Open
savme wants to merge 1 commit into
mainfrom
bench/project-control-plane
Open

chore: implement a project overhead scale test#615
savme wants to merge 1 commit into
mainfrom
bench/project-control-plane

Conversation

@savme
Copy link
Copy Markdown

@savme savme commented May 12, 2026

Summary

Phase one of #596.

  • Adds a scale build-tag test in test/scale/ that measures per-project memory, goroutine, and etcd watcher overhead against a live Milo API server
  • Captures before/after snapshots (heap, sys, goroutines, etcd watchers) with pprof profile export to dev/profiles/
  • Adds task test:scale and excludes generated profiles from git
  • Uses existing dev cluster setup and go test

This establishes a baseline (the test does not define any thresholds) to help us quantify the current per-project costs before implementing any architectural changes.

How to run

task dev:setup                     # provision dev infra 
task test:scale                    # 20 projects (default)
# or to run with n projects
SCALE_PROJECTS=50 task test:scale

Profiles are written to dev/profiles/heap-{N}p-{before,after}.pb.gz and can be compared with go tool pprof.

Test plan

  • task test:scale completes against a task dev:setup environment
  • go test ./... (without -tags scale) is unaffected

Example Run

Log
=== RUN   TestProjectControlPlaneOverhead
    project_overhead_test.go:107: connected to milo apiserver at https://localhost:30443
    project_overhead_test.go:108: port-forward milo-system/svc/etcd: localhost:63269 -> 2379
    project_overhead_test.go:112: resources: 52 listable types
    project_overhead_test.go:114: [before] Δgoroutines=1661 apiserver[heap=125.1MiB sys=183.7MiB goroutines=1661] etcd[watchers=54]
    project_overhead_test.go:114: [before] Δgoroutines=1 apiserver[heap=129.9MiB sys=183.7MiB goroutines=1662] etcd[watchers=54]
    project_overhead_test.go:114: [before] Δgoroutines=9 apiserver[heap=134.8MiB sys=183.7MiB goroutines=1653] etcd[watchers=54]
    project_overhead_test.go:114: [before] Δgoroutines=1 apiserver[heap=111.4MiB sys=183.7MiB goroutines=1654] etcd[watchers=54]
    project_overhead_test.go:115: latency before: n=52 errors=0 p50=1ms p90=2ms p99=3ms max=8ms
    project_overhead_test.go:160: bootstrap: 104/1040
    project_overhead_test.go:160: bootstrap: 208/1040
    project_overhead_test.go:160: bootstrap: 312/1040
    project_overhead_test.go:160: bootstrap: 416/1040
    project_overhead_test.go:160: bootstrap: 520/1040
    project_overhead_test.go:160: bootstrap: 624/1040
    project_overhead_test.go:160: bootstrap: 728/1040
    project_overhead_test.go:160: bootstrap: 832/1040
    project_overhead_test.go:160: bootstrap: 936/1040
    project_overhead_test.go:160: bootstrap: 1040/1040
    project_overhead_test.go:119: [after] Δgoroutines=21782 apiserver[heap=333.1MiB sys=484.9MiB goroutines=21782] etcd[watchers=1054]
    project_overhead_test.go:119: [after] Δgoroutines=1097 apiserver[heap=341.0MiB sys=484.9MiB goroutines=20685] etcd[watchers=1054]
    project_overhead_test.go:119: [after] Δgoroutines=0 apiserver[heap=348.8MiB sys=484.9MiB goroutines=20685] etcd[watchers=1054]
    project_overhead_test.go:119: [after] Δgoroutines=8 apiserver[heap=361.6MiB sys=484.9MiB goroutines=20693] etcd[watchers=1054]
    project_overhead_test.go:119: [after] Δgoroutines=5 apiserver[heap=371.1MiB sys=484.9MiB goroutines=20698] etcd[watchers=1054]
    project_overhead_test.go:120: latency after:  n=52 errors=0 p50=1ms p90=1ms p99=3ms max=5ms
    project_overhead_test.go:123: results: projects=20 resources_per_project=52
    project_overhead_test.go:124:   before:      apiserver[heap=111.4MiB sys=183.7MiB goroutines=1654] etcd[watchers=54]
    project_overhead_test.go:125:   after:       apiserver[heap=371.1MiB sys=484.9MiB goroutines=20698] etcd[watchers=1054]
    project_overhead_test.go:126:   per-project: apiserver[heap=13.0MiB sys=15.1MiB goroutines=952] etcd[watchers=50]
    project_overhead_test.go:128: profiles: saved milo/dev/profiles/heap-20p.pb.gz
    project_overhead_test.go:128: profiles: saved milo/dev/profiles/goroutine-20p.txt
--- PASS: TestProjectControlPlaneOverhead (73.27s)

@savme savme force-pushed the bench/project-control-plane branch from 97cca5b to 74d6e40 Compare May 12, 2026 11:59
@savme savme marked this pull request as ready for review May 12, 2026 12:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant