probe: multi-BRANCH anomaly — thread-level attribution refinement

[WaylandYang]

Summary

Follow-up to #128. The original `strace -c` probe couldn't distinguish user-space CPU from off-CPU blocked-waiting. This pass added bpftrace + per-thread `/proc/$pid/task/*/stack` polling and substantially refined the picture.

TL;DR — original "user-space CPU" claim was too strong

bpftrace at 199 Hz on the FC pid during a slow BRANCH: only ~18 samples in 1.6 s out of ~320 expected = FC is off-CPU ~94 % of the BRANCH window.

What FC's 5 threads do during a slow BRANCH (kernel-sleep histogram)

```
ep_poll x90 main thread (HTTP idle)
[kvm] x88 vCPU thread parked in kvm_vcpu_halt — the pause working
vhost_task_fn x50 vhost-net (idle)
futex_wait_queue x17 ← actual signal: thread blocked on userspace futex
submit_bio_wait x3 snapshot writer waiting on block IO
jbd2_log_wait x2 ext4 journal commit
```

Full kvm stack: `kvm_vcpu_block ← kvm_vcpu_halt ← ... ← __x64_sys_ioctl`
Full futex stack: `futex_wait_queue ← futex_wait ← do_futex ← __x64_sys_futex` (kernel can't say which futex from a static stack).

Revised picture — 3 contributors

1. Userspace futex contention — a worker waits on a mutex; lock hold-time may scale with accumulated snapshot count
2. ext4 journal / block IO writeback — ~2 % of the window
3. User-space CPU on the snapshot worker — ~70 % of off-CPU thread-poll points returned empty kernel stack (thread was in user mode). FC's static-pie release build lacks frame pointers → bpftrace can't symbolize. Needs DWARF unwinding (perf, when available) or a debug FC rebuild.

Implications for #118 Phase 2/3 (refined)

• Phase 2 (`io_uring` writer) addresses contributor RNG re-seed on restore (security correctness) #2 only (~2 % of the window) — narrows the win further
• Phase 3 (1 s pre-emptive tick) would compound the futex contention if the snapshot worker holds the contended lock. Identifying the lock should be a prereq.
• New candidate work: bpftrace on `tracepoint:syscalls:sys_enter_futex` with `args.uaddr` capture, correlated with the waker's stack → if contention is fixable, may cut pause growth without touching the IO path.

Files

• `bench/pause-window/probe-bpftrace-fc.sh` — user-stack sampler at 199 Hz
• `bench/pause-window/probe-syscall-poll.sh` — /proc/$pid/syscall poll loop
• `bench/pause-window/PROBE-multi-branch-anomaly.md` — "Follow-up" section appended with the refined picture and revised Phase 2 + 3 pause-window: NVMe + io_uring + pre-emptive background snapshot #118 scope

Refs #118.

🤖 Generated with Claude Code