loop: four anomaly-handling fixes — missing-signals pause, stop-kills-subprocess, agents reap, post-reflection stop guard

[fstamatelopoulos]

Summary

Five distinct fixes across two anomaly-handling features, all surfaced by a single gmbot dogfood run. Bundled because they share infrastructure (active-processes registry, per-role state-store flips, killProcessTree) and the later fixes close gaps that the earlier ones exposed.

What the gmbot dogfood exposed

• Iter 10: codex hit a usage limit; dev + judge exited without writing signals. cfcf treated it as "failed iteration" and silently moved on. Iter 11 then had to re-discover the missing work from scratch.
• Iter 19: reflection started at 3:41 pm. User checked at 8 pm — dashboard still showed "reflection running." Loop state was phase: stopped, but the reflection codex (PID 99729) was still alive with the cfcf server as its parent.
• After PA helped diagnose, user ran kill -9 99729 — and observed the loop continuing to the next iteration despite the stop. That was the giveaway that even fixing stop's auto-kill wouldn't be sufficient — there was a downstream cascade clobbering the stop signal.

Feature 1: Harness-level missing-signals pause + retry_iteration resume action

parseSignalFile() and parseJudgeSignals() returning null (file missing, JSON parse failure, or schema validation failure — all collapsed to one signal) now pauses the loop with pauseReason: \"missing_signals\" instead of silently treating the iteration as failed and continuing.

The harness deliberately doesn't classify the root cause. Quota cap / crash / OOM / killed all collapse to "agent exited without signals → can't safely continue → pause." The user reads the log to identify the cause and chooses recovery.

New resume action retry_iteration re-spawns dev on the same iteration after the user unblocks the underlying cause (typically: quota window resets). Rolls the iteration counter back via new decrementIteration() helper; existing branch is re-created off HEAD by the normal iteration body.

Pause-action allowlist for missing_signals: [retry_iteration, continue, stop_loop_now]. No finish/refine/consult — those all assume a meaningful iteration result.

Feature 2: cfcf stop honours its name end-to-end

Four layered bugs:

Bug A — stopLoop() only flipped the state flag. It set state.phase = \"stopped\" and saved. Mid-flight subprocesses kept running because nothing signaled them. The main while-loop's isStopped(state) exits the next iteration boundary cleanly, but a subprocess started before stop was simply abandoned.

Bug B — The per-role state-store stayed executing. When the subprocess was orphaned, the runner's normal completion path didn't fire, so reflect-state.json was never flipped. Dashboard's getActiveAgent() reads that file; kept claiming reflection was alive.

Bug C — No CLI way to kill live-server children. Existing cfcf server reap filters on PPID==1 (dead-server orphans). A live-server stranded subprocess didn't qualify.

Bug D — External "stopped" flag overwritten by post-reflection phase mutations. After reflection's await unblocked (whether from our new auto-kill or an external kill -9), the post-reflection code (archive, commit, Clio ingest) ran cleanly — and then state.phase = \"deciding\" clobbered the \"stopped\" flag, makeDecision returned, the outer loop's isLoopDone(state) saw \"deciding\" (not \"stopped\") and continued to the next iteration. The isStopped checks existed after dev's await (iteration-loop.ts:1852) and judge's await (line 2011), but were missing after reflection.

Fixes (one per bug)

Bug	Fix
A	stopLoop() maps phase → AgentRole via new loopActivePhaseToRole(phase) helper; looks up active-processes registry; calls existing killProcessTree() (SIGTERM → 1.5s → SIGKILL).
B	New markReflectStateFailed / markDocumentStateFailed / markReviewStateFailed helpers (idempotent — no-op when no state or already terminal). Called by stop + reap.
C	New cfcf agents reap [--workspace <name>] [--yes] + GET /api/active-processes + POST /api/active-processes/:wsId/:role/kill. Mirrors cfcf server reap UX.
D	One isStopped(state) guard before the DECIDE block (line 2283). Mirrors the dev/judge guard pattern.

PA / HA safety guarantee

PA (cfcf spec) and HA (cfcf help assistant) run interactively with stdio: \"inherit\" outside the cfcf server. They are NOT in the active-processes registry, and the registry's AgentRole type (packages/core/src/log-storage.ts:20) doesn't include them. Two layers of defence at type + registry. None of the kill paths in this PR can reach them.

Wall-clock timeout deliberately rejected

PA's initial design for Feature 2 included a per-role timeout. Declined:

• Conflicts with codex's rolling-window quota recovery (~1hr legit waits).
• Adds config knobs that are hard to tune.
• Conflicts with cf²'s "human on the loop, not in it" principle. cfcf agents reap is the user-driven version of the same intent.

Files touched (~360 LoC + tests + CHANGELOG)

Core:

• process-manager.ts — export killProcessTree
• reflection-runner.ts, documenter-runner.ts, architect-runner.ts — mark<X>StateFailed helpers
• iteration-loop.ts — loopActivePhaseToRole helper, stopLoop extended, isStopped guard before DECIDE, missing-signals pause helper, retry_iteration dispatch, allowlist
• workspaces.ts — decrementIteration helper
• types.ts — extend ResumeAction

Server: app.ts — 2 new endpoints (/api/active-processes)

CLI: commands/agents.ts (new file), index.ts (register), commands/resume.ts (add retry_iteration)

Web: types.ts, api.ts, components/FeedbackForm.tsx — mirror new pauseReason + ResumeAction variants. Existing pendingQuestions[0] rendering surfaces the generic-anomaly message; no UI copy added.

Test plan

• bun run test — 1038 tests pass (+16 new)
• bun run typecheck — clean
• CLI bundles; cfcf agents reap --help renders cleanly
• Manual smoke (post-merge):
  • Start tiny loop → wait for any agent phase → cfcf stop --workspace <name> → verify subprocess dies within ~1.5s AND dashboard chip clears.
  • On an idle workspace: cfcf agents reap → expect "No active agent processes."
  • Manufacture a missing-signals scenario (chmod 000 an agent binary or similar) → verify pause with the new reason text → resume with --action retry_iteration → verify counter rolled back + iteration re-runs cleanly.

Three commits, cleanly separable

• 1f24e90 Feature 1 (missing-signals pause + retry_iteration)
• 6f95f02 Feature 2, Bugs A/B/C (kill on stop + agents reap)
• 6f4664b Bug D (isStopped guard before DECIDE)

If reviewers prefer to merge incrementally, the commits are independent. Default plan is to merge as one.

🤖 Generated with Claude Code