perf(db): reduce idle CPU, memory, and cloud API costs#1211
perf(db): reduce idle CPU, memory, and cloud API costs#1211
Conversation
PR Build Metrics✅ All clear — no issues detected
Binary Size
Dependency ChangesNo dependency changes. govulncheck OutputBuild Info
History (9 previous)
🤖 Updated on each push. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d21ccc1c90
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ca50569457
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b27b0864fd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7909a1c764
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d3e01cad50
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3c2b5ebfc0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
Profiling Results: Idle CPU Optimization at 400 DatabasesWe profiled Litestream with 400 idle databases (800+ goroutines) to measure the impact of the optimizations in this PR and identify remaining bottlenecks. MethodologyUsed a new automated profiling test (
PROFILE_DB_COUNT=400 PROFILE_DURATION=60s PROFILE_OUTPUT_DIR=./profiles/pr1211 \
go test -tags='profile,integration' -run TestIdleProfileSuite -timeout=5m -v ./tests/integration/CPU Profile Analysis (400 idle databases, 30s sample)Before optimization (initial PR): After optimization (final PR): Result: 34% reduction in total idle CPU (5.18% → 3.42% at 400 databases). What Changed
Idle Path Syscalls Per Tick
Runtime Metrics at 400 Databases (60s steady-state)
Goroutine Breakdown (400 DBs)Remaining CPU Hotspot
Verification
|
Litestream consumes unnecessary resources on idle databases: the notify channel fires unconditionally every 1s waking all replicas, Sync() performs ~15 syscalls per tick even when the WAL hasn't changed, and the monitor uses fixed 1s polling regardless of activity. Changes: - Gate notify channel on `synced` return value so replicas only wake when data was actually synced (eliminates spurious wakeups) - Add WAL change detection: compare WAL file size against lastSyncedWALOffset to skip expensive verify+sync when idle - Add adaptive idle backoff in db.monitor(): exponentially increase polling interval from MonitorInterval to MaxIdleInterval (default 60s) when no changes detected, reset immediately on activity - Add Prometheus metrics: litestream_sync_skipped_total counter and litestream_db_idle gauge for observability - Add max-idle-interval config option to control backoff ceiling - Clear WAL cache in ResetLocalState for correct fresh-snapshot behavior Fixes #1210 Closes #992 Closes #1171
The adaptive idle backoff in monitor() delayed syncs during burst write patterns, causing oversized L0 files. The WAL change detection in Sync() already makes idle syncs cheap (1 os.Stat), so the monitor continues polling at MonitorInterval for low sync latency while Sync() skips expensive work when idle.
The fast-path that skips verifyAndSync() now also checks WAL header salt values (bytes 16-24) in addition to file size. This properly handles checkpoint-induced WAL resets where content changes while size stays constant (salt rotation after RESTART/FULL checkpoint, followed by new writes that refill to the same byte length). Changes: - Add lastWALSalt1, lastWALSalt2 fields to DB struct - Fast-path reads WAL header and compares salts before skipping - Update salt cache at end of each Sync() - Clear salt cache in ResetLocalState() Addresses PR review feedback.
1. Fix replica retry after upload failures (P1): - Add needsRetry flag to Replica.monitor() - Skip notify wait when retrying after error - Ensures transient failures on idle DBs can still retry 2. Remove unused max-idle-interval config (P3): - Remove DefaultMaxIdleInterval constant - Remove MaxIdleInterval from DB struct and DBConfig - Remove wiring in NewDBFromConfig() - Config was a no-op, can be re-added with fsnotify work
Move replica notification to happen right after verifyAndSync() succeeds with synced=true, before checkpointIfNeeded() and Pos() which can fail. This ensures replicas are notified about new TXIDs even when subsequent operations fail, preventing idle database replicas from being left behind after transient post-sync errors.
…rtbeats P1: Defer replica notification until all LTX creation completes - Add syncedDuringCurrentSync field to track LTX creation across Sync() - verifyAndSync() sets this flag when LTX is created (incl. checkpoint) - Use defer to notify replicas at end of Sync(), after checkpoint - Ensures replicas don't miss checkpoint-generated LTX files P2: Keep heartbeat timestamps fresh during idle periods - Add IdleWakeupInterval = 1 minute constant - Replica monitor wakes up periodically even without notify - Ensures Replica.Sync() is called, updating LastSuccessfulSyncAt - Prevents health checks from failing on idle-but-healthy databases
P2: Add db.mu.Lock() around WAL cache mutations in ResetLocalState() - Prevents data race with Sync() which reads/writes these fields under db.mu - Critical for auto-recovery which runs ResetLocalState() from Replica.monitor() P3: Replace time.After() with reusable time.Timer in replica monitor - Create single idleTimer before the loop, defer Stop() - Drain timer channel when notify fires first - Reset timer after each select - Reduces GC pressure with many replicas under sustained writes
The fast path that skips verifyAndSync() when WAL is unchanged now runs full verification at least once per minute. This ensures corrupted/missing LTX files are detected during idle periods, rather than going unnoticed until the next write. Changes: - Add FullVerifyInterval = 1 minute constant - Add lastFullVerifyTime field to track verification time - Fast path checks if full verification is due before skipping - Update timestamp when verifyAndSync() runs
…ed monitors Profile-guided optimizations based on 400-DB CPU profiling: 1. Combine WAL change detection into single open+fstat+read (was stat+stat+open+read): - Skip ensureWALExists() when lastSyncedWALOffset > 0 (WAL known to exist) - Use f.Stat() on already-open fd instead of separate os.Stat() - Eliminates 2 syscalls per idle tick (os.Stat x2) 2. Add jittered start delay to db.monitor(): - Random delay [0, MonitorInterval) before first tick - Spreads 400 synchronized wakeups across the interval - Reduces scheduler contention from burst patterns 3. Add automated profiling test (TestIdleProfileSuite): - Configurable DB count, duration, output directory - Collects CPU profile, heap profile, goroutine dump - Optional runtime/trace capture - JSON metrics export for before/after comparison Results at 400 idle databases (30s CPU profile): Total CPU: 5.18% → 3.42% (-34%) os.Stat: 0.30s → 0s (eliminated) ensureWALExists: 0.20s → 0s (eliminated) Scheduler: 0.37s → 0.29s (-22%)
Treat the fresh-database no-position state as a notify wait instead of a retry path so the first real upload is not held behind backoff or SyncInterval. Keep the first DB monitor sync at MonitorInterval and apply jitter on the following wakeup so startup latency does not regress while steady-state wakeups still dephase. Add regression coverage for both review findings and fix soak-test helpers needed for local validation.
corylanou
force-pushed
the
issue-1210-perf-reduce-idle-cpu-memory-and-cloud-api-costs
branch
from
83e8e9a to
d51580e
Compare
1 participant
Summary
Reduces unnecessary CPU, memory, and cloud storage API usage on idle databases while preserving prompt startup replication behavior.
Fixes #1210
Closes #992
Closes #1171
Review Follow-up
Addressed the two review findings from this PR:
no position, waiting for dataas a notify-wait state instead of a retry state, so the first real upload is not delayed behind exponential backoff or an extraSyncInterval.MonitorInterval, and the phase shift is applied on the following wakeup.Additional Validation Fixes
While running local soak coverage for this PR, I fixed two soak-harness issues so the documented validation paths could actually run:
criticalErrorsreferences in the soak testsCreateSoakConfig()for MinIO/S3-backed soak runsValidation
go test ./...go test ./... -run 'TestDB_Monitor_FirstSyncDoesNotWaitExtraInterval|TestReplica_Monitor_FreshDBWaitsOnNotify|TestStore_Integration'SOAK_DEBUG=1 go test -v -tags='integration,soak' -run=TestComprehensiveSoak -test.short -timeout=1h ./tests/integrationSOAK_AUTO_PURGE=yes SOAK_DEBUG=1 go test -v -tags='integration,soak,docker' -run=TestMinIOSoak -test.short -timeout=20m ./tests/integrationResults
wrong # of entries in index idx_load_test_timestamp. That issue was surfaced during validation and is not addressed in this PR.