bug: openab as PID 1 leaves zombie processes from agent subprocesses

[thekkagent]

Description

When openab runs as PID 1 in a container (the default for the published openab-codex image), zombie processes from agent grandchildren are never reaped, because openab does not implement a SIGCHLD reaper expected of init processes.

Observed in a K3s pod running ghcr.io/openabdev/openab-codex:0.7.2 with codex-acp as the agent. After ~85 minutes of normal Discord usage, 44 zombie processes had accumulated — all with PPid = 1 (i.e. openab itself).

$ cat /proc/1/comm
openab

$ for p in /proc/[0-9]*; do
    s=$(awk '/^State:/{print $2}' $p/status 2>/dev/null)
    pp=$(awk '/^PPid:/{print $2}' $p/status 2>/dev/null)
    [ "$s" = "Z" ] && echo "Z pid=$(basename $p) ppid=$pp"
  done | head
Z pid=10070 ppid=1
Z pid=10838 ppid=1
Z pid=11431 ppid=1
Z pid=12169 ppid=1
Z pid=12750 ppid=1
Z pid=13621 ppid=1
Z pid=14232 ppid=1
Z pid=14936 ppid=1
Z pid=15160 ppid=1
Z pid=1575  ppid=1

# state distribution across all processes
      3 (kernel/empty)
     12 S
     44 Z

The chain:

1. openab (PID 1) spawns codex-acp
2. codex-acp spawns shell tools (git, grep, next, ...) for tool
   calls
3. When codex-acp exits or restarts a session, its remaining children get
   reparented to PID 1 (kernel default)
4. openab does not call wait() on arbitrary children → they become
   zombies and stay forever

This is distinct from #269 — that one is about kiro-cli-chat orphans (alive, ~300 MB RSS each, real memory leak). This issue is about
reaped-but-not-cleaned zombies (almost no memory, but they consume PID table entries).

Steps to Reproduce

1. Deploy openab with the codex-acp agent using the standard
   ghcr.io/openabdev/openab-codex:0.7.2 image (no init wrapper)
2. Use the Discord bot for normal coding sessions involving shell tool
   calls (git status, pnpm build, next build, etc.)
3. After ~30–60 minutes of activity, exec into the pod:

   kubectl exec <pod> -- sh -c '
     for p in /proc/[0-9]*; do
       awk "/^State:/{print \$2}" $p/status
     done | sort | uniq -c'

4. Observe the count of Z (zombie) state processes growing roughly
   linearly with session activity, all with PPid = 1

Expected Behavior

Zombie processes should be reaped automatically. Either:

• Option A (recommended, image-level fix): ship the image with a proper
  init binary as PID 1. tini is the standard answer; one apt install +
  ENTRYPOINT ["tini", "--", "openab"] in Dockerfile.codex. This also
  fixes signal forwarding for free.
• Option B (app-level fix): have openab install a SIGCHLD handler
  that reaps any waitable children whenever it detects it is running as PID

1. The Rust ecosystem has crates for this, or a small signal-hook loop
   calling waitpid(-1, WNOHANG).

Option A is much smaller and is the standard container best practice. It also helps every other agent image (openab-claude, openab-gemini, openab-copilot) since the same Dockerfile structure exists across them.

Environment

• chart: openab 0.7.2 (Helm)
• image: ghcr.io/openabdev/openab-codex:0.7.2
• agent: codex-acp (@zed-industries/codex-acp@0.9.5)
• runtime: K3s (containerd, not docker)
• pod securityContext: runAsNonRoot: true, runAsUser: 1000,
  capabilities.drop: [ALL]
• no shareProcessNamespace, no init wrapper
• args for codex passed via Helm:

  -c approval_policy="never"
  -c sandbox_mode="workspace-write"
  -c sandbox_workspace_write.network_access=true
  
  

Screenshots / Logs

Process listing snippet at the time of measurement (PPid column omitted for brevity, full command lines shown for live processes only):

PID 1     S  openab /etc/openab/config.toml
PID 13    S  node /usr/local/bin/codex-acp -c approval_policy="never" -c
sandbox_mode="workspace-write" -c
sandbox_workspace_write.network_access=true
PID 16715 S  node /usr/local/bin/corepack pnpm build
PID 16747 S  node /usr/local/bin/corepack pnpm -r build
PID 16783 R  node .../packages/web/.../next build
... plus 44 Z (zombie) entries, all with PPid=1, empty cmdline ...

Total state distribution: 12 S, 1 R, 44 Z after ~85 minutes uptime, ~30 zombies/hour during normal use. With Linux's default pid_max=32768 this means PID exhaustion at roughly 1000 hours uptime. Long-lived pods will eventually fail to fork.

Workaround

• Periodically restart the deployment with kubectl rollout restart deploy/openab
• Or build a downstream image that wraps the entrypoint in tini:

FROM ghcr.io/openabdev/openab-codex:0.7.2
USER root
RUN apt-get update && apt-get install -y --no-install-recommends tini &&
apt-get clean
USER node
ENTRYPOINT ["tini", "--", "openab"]

[thekkagent]

Additional context

Relationship to #269 / PR #270

This is structurally related to #269 but is a distinct bug that PR #270 does not cover:

	#269 / PR #270	#289 (this)
Process state	S (running)	Z (zombie)
Still alive?	Yes — grandchild never killed	No — already exited
Leaked resource	RAM (~300 MB / process)	PID table entries (~0 bytes / process)
Trigger	Session shutdown kills kiro-cli but not kiro-cli-chat	Normal shell tool calls that exit cleanly
Required fix	process_group(0) + killpg(-PGID) on drop	PID 1 SIGCHLD reaper (or init wrapper)

PR #270's killpg approach only runs when a session ends. Zombies from tool-call subprocesses that exit mid-session are still accumulating with PPid=1 because nothing calls wait() on them. The two fixes are independent and a maintainer should not close #289 as a duplicate.

The less obvious sibling bug: signal forwarding

While researching this I noticed a second consequence of running openab directly as PID 1 that may not be on anyone's radar yet. The Linux kernel does not apply default signal dispositions to PID 1:

If a process running as PID 1 hasn't registered a handler for the signal, the kernel won't fall back to default behavior, and nothing happens.

— dumb-init README

In practice this means: if openab has no explicit SIGTERM handler (I haven't audited the source yet — worth confirming), then kubectl delete pod sends SIGTERM and it is effectively swallowed. Kubernetes then waits the full grace period (default 30s) before falling back to SIGKILL. Every pod deletion and every rollout pays that 30-second penalty silently.

Same root cause as this issue (openab is PID 1 but not acting like init). Same fixes resolve both.

Canonical approaches to PID 1 in containers

The three established paths, from most decoupled to most idiomatic:

1. Minimal init as PID 1 (image-level)

Three interchangeable binaries, all ~100 KB and single-purpose:

• tini — bundled with Docker 1.13+ (docker run --init injects it)
• dumb-init — Yelp, the canonical write-up of both PID 1 problems
• catatonit — openSUSE/Podman default, uses signalfd(2) instead of sigwait(2)

Dockerfile change for any of the openab-* images:

RUN apt-get update \
 && apt-get install -y --no-install-recommends tini \
 && apt-get clean
ENTRYPOINT ["tini", "--", "openab"]

Lowest-risk option. Does not require any Rust changes.

2. Runtime-injected init (not applicable here)

docker run --init auto-injects tini on the Docker runtime. Kubernetes has no equivalent spec.containers[].init: true field, so this path is unavailable for the openab deployment model. Noted only for completeness.

3. App-level PID 1 handling

Implement reaping and signal forwarding inside openab itself. In Tokio:

use tokio::signal::unix::{signal, SignalKind};

// Reap any orphaned children that exit, at any time.
let mut sigchld = signal(SignalKind::child())?;
tokio::spawn(async move {
    loop {
        sigchld.recv().await;
        // Non-blocking reap loop.
        while unsafe { libc::waitpid(-1, std::ptr::null_mut(), libc::WNOHANG) } > 0 {}
    }
});

// Graceful shutdown on SIGTERM / SIGINT.
let mut sigterm = signal(SignalKind::terminate())?;
let mut sigint  = signal(SignalKind::interrupt())?;
tokio::select! {
    _ = sigterm.recv() => graceful_shutdown().await,
    _ = sigint.recv()  => graceful_shutdown().await,
    _ = run()          => {}
}

Heavier to implement and test, but it is the only option that lets graceful shutdown itself be async — which PR #270's review discussion already identified as a follow-up requirement:

implementing a 'wait-and-kill' strategy would require refactoring to include an async fn shutdown() before the AcpConnection is dropped

Suggested scoping

Since PR #270's discussion already flags an async fn shutdown() refactor as follow-up work, option 3 naturally folds three separate issues into one refactor:

1. [BUG] Memory Leak: Stale 'kiro-cli-chat' processes orphaned after session termination #269 / PR fix: memory leak by using process groups to kill orphaned grandchildren #270 — killpg on session drop (done once Drop becomes async shutdown)
2. bug: openab as PID 1 leaves zombie processes from agent subprocesses #289 (this issue) — SIGCHLD reaper loop in main
3. Implicit SIGTERM handler — fixes the 30-second-grace-period penalty under Kubernetes

If that refactor is going to happen anyway, doing all three together is probably less total churn than shipping tini as a separate Dockerfile change across four agent images (openab-codex, openab-claude, openab-gemini, openab-copilot) and then revisiting the same area later.

Either direction works. Flagging the options so the maintainers can pick based on roadmap preference — happy to take a stab at a PR in whichever direction is preferred.

References

• Kubernetes pod termination flow (SIGTERM → grace period → SIGKILL): https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/
• tini design notes: https://github.com/krallin/tini#why-tini
• dumb-init — canonical write-up of both PID 1 problems: https://github.com/Yelp/dumb-init#readme
• catatonit — same space, different signal-handling primitive: https://github.com/openSUSE/catatonit
• Docker --init flag: https://docs.docker.com/reference/cli/docker/container/run/#init

[CHC-Agent]

Issue at a Glance

Container (K8s pod)
       │
       PID 1: openab (no init, no SIGCHLD handler)
       │
       ├── spawn ──► codex-acp (child)
       │                 │
       │                 ├── git status    ──► exits ──► reparented to PID 1 ──► 💀 zombie
       │                 ├── pnpm build    ──► exits ──► reparented to PID 1 ──► 💀 zombie
       │                 ├── next build    ──► exits ──► reparented to PID 1 ──► 💀 zombie
       │                 └── ...
       │
       └── nobody calls wait() ──► zombies accumulate (~30/hr)
                                    PID exhaustion at ~1000 hrs

Hi @thekkagent,

Thorough report and follow-up analysis — the distinction from #269 (live orphans eating RAM vs. dead zombies eating PID entries) is clear and well-documented.

No existing PR addresses this directly. PR #310 (merged) fixed process group kill for session teardown but does not cover PID 1 reaping of mid-session tool-call children.

Related: #269 (orphaned kiro-cli-chat processes, different root cause), #289 (prior report, closed as incomplete).