probe: futex tracing — 3 idle waiters, not contention

[WaylandYang]

Summary

Third follow-up to #128. bpftrace on the futex syscall pair, aggregating per-uaddr wait time during a BRANCH window. Adds `probe-futex-trace.sh` and a new "Follow-up: futex tracing" section to PROBE-multi-branch-anomaly.md.

Finding

8 s window over one 153 ms BRANCH:

```
@wait_ns[0x...7648, 137]: 152.35 ms (3 calls)
@wait_ns[0x...8c88, 137]: 152.49 ms (2 calls)
@wait_ns[0x...8d08, 137]: 152.56 ms (2 calls)
@wait_ns[0x...52a0, 137]: 1.14 ms (6 calls) ← FC .data/.bss
```

Op 137 = `FUTEX_WAIT_BITSET_PRIVATE`. Three different futexes each accumulated ~152 ms ≈ the entire pause window. Cross-referenced against `/proc/$pid/maps`: three of them fall in anonymous heap mappings adjacent to FC binary's `.bss` (consistent with Rust heap-allocated synchronization primitives — `parking_lot::Mutex`/`Condvar` inner atomics). One falls in FC's `.data/.bss` (possibly a `static`-held mutex, lower amplitude).

Interpretation flip

The previous probe pass said "futex contention". This one corrects: it's not contention, it's 3 idle waiters parked on the snapshot worker's completion signal. Each thread sleeps the entire pause then wakes when the worker finishes. Eliminating the futex calls wouldn't speed anything up — the bottleneck is whatever the snapshot worker is doing single-threaded.

Revised #118 implication (third pass)

• Phase 2 (io_uring) addresses ~2 % of the window — narrows further from "Phase 2 still helps disk write" to "Phase 2 doesn't help the dominant cost"
• Phase 3 (1 s tick) is orthogonal to the waiters
• Next operational step: build FC with DWARF symbols and perf-record-flamegraph the snapshot worker thread. The current FC binary is static-pie-linked without frame pointers; bpftrace ustack can't symbolize. Until that's set up, we can't tell which Rust function in `vmm::persist::create_snapshot` (~21 KB of compiled code) holds the per-snapshot growing loop.

Files

• `bench/pause-window/probe-futex-trace.sh` — the probe script
• `bench/pause-window/PROBE-multi-branch-anomaly.md` — new section + revised next-steps

Refs #118.

🤖 Generated with Claude Code