v0.3 shipped: BRANCH source-pause 29 s → 205 ms (143x) on 4 GiB SSD

[WaylandYang]

forkd v0.3 ships diff-snapshot BRANCH today (v0.3.0 in the morning + v0.3.1 with multi-BRANCH support in the afternoon). Source-pause window on POST /v1/sandboxes/:id/branch drops from a memory-bandwidth-bound write of the full memory image to just the diff of pages dirtied since the previous snapshot.

Measured pause-window (mean of 3 trials, mem-2048 SSD)

Source memory	v0.2 Full	v0.3 Diff	Speedup
256 MiB	1.8 s	241 ms	7.5 ×
512 MiB	3.4 s	226 ms	15 ×
1024 MiB	6.9 s	229 ms	30 ×
2048 MiB	14.5 s	222 ms	65 ×
4096 MiB	29.3 s	205 ms	143 ×

The Diff column is roughly constant because Firecracker's snapshot_type: "Diff" writes only dirtied pages (~900 KB for an idle source). For real workloads we swept dirty footprint separately: typical agent workloads with 30-300 MiB dirty see 6-15 ×; crossover (Diff loses) is at ~65 % source RAM dirty.

What changed

• Opt-in "diff": true on the BRANCH request.
• Daemon parallelizes the source-tag memory.bin copy with the source still running, takes a Diff snapshot during a sub-second pause, resumes, then merges the diff onto the pre-copied output. Source TCP / kvmclock / timers see a ~200 ms gap instead of seconds.
• Total BRANCH API latency is unchanged (the cp still runs in the background); only source DOWNTIME shrinks. Right trade-off for live BRANCH from a long-running agent.
• v0.3.1 lifts the first-BRANCH-only restriction via a previous-output chain (each BRANCH's memory.bin is, by construction, source's state at that BRANCH's pause time — exactly the base the next diff needs). Zero extra storage, no shadow file.

We considered forking Firecracker and chose not to

Original v0.3 plan was memfd + uffd_wp live-fork. After phase 1's 143 × on vanilla upstream, the cost-benefit on maintaining a deeplethe/firecracker fork (own CI, rebase per upstream tag, track CVEs, weakened trust story) stopped making sense for the remaining ~175 ms of Firecracker control-plane floor. Decision + revival criteria in issue #101 and docs/design/userfaultfd.md § "Why we won't fork Firecracker".

Data + design

• bench/pause-window/RESULTS-v0.3.md — 117 trials across phases 1a/1b/1c/1d, all CSVs in-tree.
• docs/design/diff-snapshots.md — phase 1 design including the rejected alternatives.
• Releases: v0.3.0, v0.3.1.

Try it

pip install forkd==0.3.1
# spawn source, then:
curl -X POST $URL/v1/sandboxes/$ID/branch \
    -H "Authorization: Bearer $TOKEN" \
    -d '{"diff": true}'
# pause_ms ~200 ms instead of seconds

Questions, real workloads to point the bench at, and PRs welcome.