Session pool leaks memory: orphaned kiro-cli processes and no eviction

[chaodu-agent]

Description

When running openab with kiro-cli on a constrained host (e.g. 3.6 GB RAM on Zeabur), the session pool fills up with idle sessions that are never reclaimed in time. Once max_sessions is reached, new requests are rejected and the host eventually OOMs.

Each kiro-cli acp spawns a child kiro-cli-chat acp process (~230-390 MB each). When the pool drops a session, kill_on_drop only kills the direct child — the grandchild kiro-cli-chat process becomes orphaned and keeps consuming memory.

Observed on a live deployment — 10 stale kiro-cli-chat acp processes consuming 3 GB total:

PID	Started	RSS
459820	Apr12	290 MB
581161	Apr12	306 MB
625730	Apr12	300 MB
633382	Apr12	282 MB
673360	Apr12	388 MB
724688	00:43	273 MB
872305	08:48	274 MB
872764	08:50	236 MB
907784	10:39	227 MB
913618	11:00	230 MB

Four root causes identified:

1. Orphaned grandchild processes — kill_on_drop(true) only SIGKILLs the direct child PID. The grandchild kiro-cli-chat survives and leaks memory. Fix: use process groups (setsid/setpgid) and kill the entire group on cleanup.

2. No cleanup on Discord thread archive — EventHandler only implements message and ready. Archiving a thread leaves the session alive until TTL. Fix: implement thread_update handler.

3. No LRU eviction — when pool is full, get_or_create() rejects with "pool exhausted" instead of evicting the oldest idle session. Fix: evict oldest last_active session when at capacity.

4. Default TTL too long — session_ttl_hours defaults to 24. On a 3.6 GB host with 10 sessions × ~300 MB = 3 GB of idle processes. Fix: lower default or document memory implications.

Industry Comparison

A survey of agent harnesses from Picrew/awesome-agent-harness shows openab sits in the riskiest position — process-level isolation without proper process group management:

Harness	Isolation	Orphan Risk	Cleanup Strategy
Gemini CLI a2a	In-process Map	None ✅	task.dispose() + Map delete
openab	Process	HIGH ☠️	kill_on_drop (broken for grandchildren)
acpx	Process	Low ✅	3-stage shutdown (stdin.end() → SIGTERM → SIGKILL) + self-terminating TTL
Scion	Container	None ✅	docker rm -f kills everything
Daytona / E2B	VM/microVM	None ✅	Destroy sandbox API

Key insight from acpx: they use a 3-stage graceful shutdown (stdin.end() → SIGTERM 1.5s → SIGKILL 1s → detach all handles) and self-terminating queue-owner processes that exit when idle. This eliminates both the orphan problem and the need for a central cleanup task.

Key insight from Scion: container-per-agent makes orphans impossible by design (docker rm -f kills the entire process tree). This is the most robust long-term architecture but requires more infrastructure.

Steps to Reproduce

1. Deploy openab with kiro-cli on a host with limited RAM (e.g. 3.6 GB)
2. Send messages from Discord that create multiple threads (up to max_sessions)
3. Archive/close the Discord threads
4. Observe that kiro-cli and kiro-cli-chat processes remain running
5. Run ps aux | grep kiro-cli — orphaned processes accumulate
6. Eventually the host runs out of memory and the pod/container is killed

Expected Behavior

• When a Discord thread is archived, the associated session and all its child processes should be terminated
• When the pool is full, the oldest idle session should be evicted to make room
• When a session is dropped, all descendant processes (including grandchildren) should be killed via process group signal
• Default TTL should be reasonable for small hosts, or clearly documented

[chaodu-agent]

Suggested Solution

A single PR addressing all four root causes in src/acp/connection.rs and src/acp/pool.rs:

1. Process group kill (critical)

In AcpConnection::spawn(), create a new process group before exec so all descendants share the same PGID:

// connection.rs — inside spawn()
unsafe {
    cmd.pre_exec(|| {
        libc::setpgid(0, 0); // new process group = own PID
        Ok(())
    });
}

Then implement Drop for AcpConnection (or a manual shutdown method) that kills the entire group:

// Kill entire process group, not just direct child
unsafe { libc::kill(-(child_pid as i32), libc::SIGTERM); }

Remove kill_on_drop(true) since we handle cleanup ourselves.

2. LRU eviction in pool

In SessionPool::get_or_create(), when pool is full, evict the oldest idle session instead of rejecting:

if conns.len() >= self.max_sessions {
    if let Some(oldest_key) = conns.iter()
        .min_by_key(|(_, c)| c.last_active)
        .map(|(k, _)| k.clone())
    {
        info!(evicted = %oldest_key, "pool full, evicting oldest idle session");
        conns.remove(&oldest_key); // Drop triggers process group kill
    } else {
        return Err(anyhow!("pool exhausted"));
    }
}

3. Thread archive cleanup

Add thread_update to the serenity EventHandler in discord.rs:

async fn thread_update(&self, _ctx: Context, _old: Option<GuildChannel>, new: GuildChannel) {
    if let Some(meta) = &new.thread_metadata {
        if meta.archived {
            let thread_key = new.id.get().to_string();
            self.pool.remove(&thread_key).await;
        }
    }
}

Add a remove() method to SessionPool that removes and drops the connection for a given key.

4. Lower default TTL

Change default_ttl_hours() from 24 to 4 in config.rs. Add a comment in config.toml.example and values.yaml noting memory implications.

---

Considered Alternatives

A. prctl(PR_SET_PDEATHSIG) instead of process groups

Linux-specific. Sets a signal to be sent to the child when its parent dies. Problem: only works one level deep — if kiro-cli forks kiro-cli-chat, the grandchild does not inherit the death signal. Also does not help with explicit cleanup (only parent death). Rejected — process groups are more reliable and cover all descendants.

B. cgroups for process containment

Spawn each session inside a dedicated cgroup, then kill the entire cgroup on cleanup. More robust than process groups (handles double-forks, daemonization). Rejected for now — requires cgroup v2 setup, elevated permissions, and adds significant complexity. Process groups solve the immediate problem. Could revisit if agents start spawning background daemons.

C. Periodic pkill by PID file or naming convention

Have openab write PID files or tag processes with environment variables, then periodically scan and kill orphans. Rejected — fragile, race-prone, and does not solve the root cause. Process groups are the correct OS primitive.

D. Single long-lived kiro-cli process with multiplexed sessions

Instead of spawning one process per session, run a single kiro-cli process and multiplex all ACP sessions over it. Would eliminate the process management problem entirely. Rejected for now — requires changes to kiro-cli's ACP implementation to support multiple concurrent sessions on a single stdin/stdout. Worth exploring as a future optimization but out of scope for this bug fix.

E. Memory-based eviction instead of time-based LRU

Monitor RSS of child processes and evict the largest consumer when memory pressure is high. Rejected — adds complexity (polling /proc/[pid]/status), platform-specific, and LRU eviction already solves the pool exhaustion problem. The real memory fix is killing orphaned grandchildren.

[chaodu-agent]

Visual Explanation

The Problem: Orphaned Grandchildren

                         openab (SessionPool)
                                │
                  ┌─────────────┼─────────────┐
                  │             │             │
            kill_on_drop  kill_on_drop  kill_on_drop
                  │             │             │
                  ▼             ▼             ▼
             kiro-cli      kiro-cli      kiro-cli        ← direct children
             (25 MB)       (25 MB)       (25 MB)            killed on drop ✅
                  │             │             │
                  │ fork        │ fork        │ fork
                  ▼             ▼             ▼
           kiro-cli-chat  kiro-cli-chat  kiro-cli-chat   ← grandchildren
             (300 MB)      (300 MB)      (300 MB)           ORPHANED! ☠️

When openab drops a session, kill_on_drop kills kiro-cli (the direct child).
But kiro-cli-chat (the grandchild) is not in the kill scope — it gets
reparented to PID 1 and keeps running, eating ~300 MB each.

  After pool cleanup:

             kiro-cli      ← killed ✅ (25 MB freed)
                  │
                  ╳ parent died
                  │
                  ▼
           kiro-cli-chat   ← still alive ☠️ (300 MB leaked!)
           (reparented to PID 1)

  × 10 sessions = 3 GB leaked

The Fix: Process Groups

                         openab (SessionPool)
                                │
                  ┌─────────────┼─────────────┐
                  │             │             │
              PGID=100     PGID=200     PGID=300
            ┌─────────┐  ┌─────────┐  ┌─────────┐
            │         │  │         │  │         │
            │ kiro-cli│  │ kiro-cli│  │ kiro-cli│    ← setpgid(0,0)
            │ (25 MB) │  │ (25 MB) │  │ (25 MB) │       at spawn
            │    │    │  │    │    │  │    │    │
            │    ▼    │  │    ▼    │  │    ▼    │
            │kiro-cli-│  │kiro-cli-│  │kiro-cli-│    ← inherits
            │  chat   │  │  chat   │  │  chat   │       parent's PGID
            │(300 MB) │  │(300 MB) │  │(300 MB) │
            └─────────┘  └─────────┘  └─────────┘
             process       process       process
              group         group         group

On cleanup: kill(-PGID, SIGTERM) → kills the entire group at once.

  After pool cleanup with process groups:

            ┌─────────┐
            │ ██████  │  ← kill(-100, SIGTERM)
            │ ██████  │     kills BOTH processes ✅
            │ ██████  │     325 MB freed!
            └─────────┘

  × 10 sessions = 3.25 GB freed ✅

Pool Lifecycle (with all fixes)

  Discord          openab SessionPool          OS Processes
  ───────          ──────────────────          ────────────

  Thread A     ──►  get_or_create("A")    ──►  spawn PGID=100
  created           pool: [A]                  kiro-cli + kiro-cli-chat

  Thread B     ──►  get_or_create("B")    ──►  spawn PGID=200
  created           pool: [A, B]               kiro-cli + kiro-cli-chat

    ...             pool: [A..J]               10 process groups
                    (10/10 full)

  Thread K     ──►  get_or_create("K")
  created           pool full!
                    ├─ LRU evict "A"      ──►  kill(-100, SIGTERM) 💀
                    ├─ spawn PGID=1100    ──►  kiro-cli + kiro-cli-chat
                    pool: [B..K]

  Thread B     ──►  thread_update(archived)
  archived          ├─ pool.remove("B")   ──►  kill(-200, SIGTERM) 💀
                    pool: [C..K]               slot freed immediately

  4h passes    ──►  cleanup_idle()
                    ├─ evict stale C,D,E  ──►  kill groups 💀💀💀
                    pool: [F..K]

[chaodu-agent]

How openclaw/acpx Solves This

acpx uses a fundamentally different architecture: ephemeral clients + a shared queue-owner process per session.

acpx Architecture

                          acpx CLI
                             │
                     ┌───────┴───────┐
                     │               │
                  "send"          "send"
                 (prompt 1)      (prompt 2)
                     │               │
                     ▼               ▼
              ┌──────────────────────────┐
              │   Queue Owner Process    │  ← ONE long-lived process
              │   (per session)          │     per session, with TTL
              │                          │
              │  ┌────────────────────┐  │
              │  │  Shared AcpClient  │  │  ← single AcpClient reused
              │  │                    │  │     across all prompts
              │  │  agent ──► stdin   │  │
              │  │  agent ◄── stdout  │  │
              │  └────────────────────┘  │
              │                          │
              │  IPC server (Unix sock)  │  ← accepts tasks from CLI
              │  Task queue (depth ≤16)  │
              │  Heartbeat + lease       │
              │  TTL idle shutdown       │
              └──────────────────────────┘
                          │
                          │ spawns exactly 1
                          ▼
                    ┌────────────┐
                    │   Agent    │  (e.g. kiro-cli, claude)
                    │  Process   │
                    └────────────┘

Key Differences from openab

  openab (current)                    acpx
  ────────────────                    ────

  Discord msg ──► spawn new           CLI "send" ──► IPC to queue owner
                  kiro-cli process                    (reuse existing)

  1 process per session               1 process per session
  Process lives until TTL (24h)       Process lives until TTL idle
  No IPC — direct stdin/stdout        IPC via Unix socket
  kill_on_drop (direct child only)    Graceful 3-stage shutdown:
                                        1. stdin.end()
                                        2. SIGTERM (1.5s grace)
                                        3. SIGKILL (1s grace)
                                      + detach all stdio handles

acpx's 3-Stage Graceful Shutdown

  AcpClient.close()
       │
       ▼
  ┌─────────────────────┐
  │ 1. child.stdin.end() │  ← "please exit gracefully"
  │    wait up to Xms    │     (agent-specific grace period)
  └──────────┬──────────┘
             │ still alive?
             ▼
  ┌─────────────────────┐
  │ 2. child.kill(TERM)  │  ← "exit now please"
  │    wait 1500ms       │
  └──────────┬──────────┘
             │ still alive?
             ▼
  ┌─────────────────────┐
  │ 3. child.kill(KILL)  │  ← "you're done"
  │    wait 1000ms       │
  └──────────┬──────────┘
             │
             ▼
  ┌─────────────────────┐
  │ 4. detachAgentHandles│  ← destroy stdin/stdout/stderr
  │    child.unref()     │     prevent zombie handles
  └─────────────────────┘

acpx's Queue Owner TTL (Self-Terminating)

  Queue Owner spawned (detached, unref'd)
       │
       ▼
  ┌──────────────────────────────────────┐
  │  Loop:                               │
  │    task = owner.nextTask(ttlMs)      │  ← blocks until task or timeout
  │    if no task within TTL → break     │     DEFAULT_TTL = idle timeout
  │    else → run prompt turn            │
  │           save session record        │
  │           loop again                 │
  └──────────────┬───────────────────────┘
                 │ TTL expired, no more tasks
                 ▼
  ┌──────────────────────────────────────┐
  │  Cleanup:                            │
  │    owner.close()                     │
  │    sharedClient.close()  ← 3-stage   │
  │    releaseQueueOwnerLease()          │
  │    process exits cleanly             │
  └──────────────────────────────────────┘

Why acpx Doesn't Have openab's Problem

  openab's problem:

    openab ──spawn──► kiro-cli ──fork──► kiro-cli-chat
                         │                    │
                    kill_on_drop          ORPHANED ☠️
                    (kills this)         (not killed)


  acpx's approach:

    acpx ──spawn──► agent (direct child)
                      │
                 stdin.end() → SIGTERM → SIGKILL
                 (escalating shutdown on the direct child)
                      │
                 child.unref() + destroy all handles
                 (no zombie file descriptors)

  But more importantly:

    acpx spawns the queue-owner as detached + unref'd.
    The queue-owner owns the agent lifecycle entirely.
    When TTL expires, queue-owner shuts down the agent
    AND itself — nothing is left behind.

    openab keeps the agent in a HashMap inside its own
    process. If openab crashes or the HashMap entry leaks,
    the agent process is orphaned forever.

What openab Could Adopt

  Priority 1: Process group kill (immediate fix)
  ─────────────────────────────────────────────
  Cheapest fix. setpgid + kill(-pgid). Solves orphan grandchildren.

  Priority 2: 3-stage graceful shutdown (from acpx)
  ─────────────────────────────────────────────────
  stdin.end() → SIGTERM → SIGKILL with timeouts.
  More graceful than kill_on_drop's raw SIGKILL.

  Priority 3: Self-terminating sessions (from acpx)
  ─────────────────────────────────────────────────
  Instead of relying on a central cleanup task,
  each session could have its own idle timer that
  triggers self-shutdown. Decentralized = more robust.

[chaodu-agent]

Agent Harness Process Management — Landscape Survey

Based on Picrew/awesome-agent-harness, here's how the major harnesses solve the "agent process lifecycle" problem that openab faces.

---

The Spectrum of Isolation

  Less Isolation                                          More Isolation
  ◄──────────────────────────────────────────────────────────────────────►

  In-Process        Process per        Queue-Owner       Container per
  (shared memory)   Session            (self-terminating) Session
       │                 │                   │                  │
       ▼                 ▼                   ▼                  ▼
  ┌──────────┐    ┌──────────────┐    ┌──────────────┐    ┌──────────────┐
  │ Gemini   │    │   openab     │    │    acpx      │    │   Scion      │
  │ CLI a2a  │    │  (current)   │    │  (openclaw)  │    │  (Google)    │
  │          │    │              │    │              │    │              │
  │ Tasks in │    │ spawn per    │    │ detached     │    │ container    │
  │ a Map<>  │    │ thread,      │    │ queue-owner  │    │ per agent,   │
  │ in one   │    │ kill_on_drop │    │ per session, │    │ git worktree │
  │ Node.js  │    │              │    │ self-TTL     │    │ per agent    │
  │ process  │    │              │    │              │    │              │
  └──────────┘    └──────────────┘    └──────────────┘    └──────────────┘
    0 MB/task      ~300 MB/session     ~300 MB/session     ~500 MB/container
    No isolation   PID isolation       PID + self-cleanup   Full OS isolation

---

1. Gemini CLI (a2a-server) — In-Process Task Map

                    Node.js Process
                    ┌──────────────────────────────┐
                    │  CoderAgentExecutor           │
                    │                               │
                    │  tasks: Map<id, TaskWrapper>  │
                    │  ┌─────┐ ┌─────┐ ┌─────┐     │
                    │  │ T1  │ │ T2  │ │ T3  │     │
                    │  └─────┘ └─────┘ └─────┘     │
                    │                               │
                    │  executingTasks: Set<id>       │
                    │  (prevents double-execution)  │
                    │                               │
                    │  Cleanup: task.dispose()      │
                    │  on terminal state            │
                    │  (canceled/failed/completed)  │
                    └──────────────────────────────┘

  Pros: Zero process overhead, instant task creation
  Cons: One crash kills ALL tasks, no memory isolation
  Cleanup: Explicit dispose() + Map.delete() on terminal states

No child processes at all. Everything runs in one Node.js event loop. Tasks are just objects in a Map. When a task reaches a terminal state (canceled/failed/completed), it calls task.dispose() and deletes from the map. No orphan problem because there are no child processes.

---

2. openab (current) — Process-per-Session with Central Pool

                    openab (Rust/tokio)
                    ┌──────────────────────────────┐
                    │  SessionPool                  │
                    │  connections: HashMap<String,  │
                    │               AcpConnection>  │
                    │                               │
                    │  ┌─────┐ ┌─────┐ ┌─────┐     │
                    │  │ C1  │ │ C2  │ │ C3  │     │
                    │  └──┬──┘ └──┬──┘ └──┬──┘     │
                    └─────┼───────┼───────┼────────┘
                          │       │       │
                          ▼       ▼       ▼
                       kiro-cli kiro-cli kiro-cli    ← direct children
                          │       │       │
                          ▼       ▼       ▼
                       kiro-cli-chat (grandchildren)  ← ORPHAN RISK ☠️

  Cleanup: kill_on_drop (direct child only) + 24h TTL sweep
  Problem: Grandchildren survive, no eviction, no thread-close hook

---

3. acpx (openclaw) — Self-Terminating Queue Owner

  CLI "send"  ──►  IPC submit  ──►  Queue Owner (detached)
                                    ┌──────────────────────┐
                                    │  sharedClient         │
                                    │  ┌────────────────┐   │
                                    │  │  agent process  │   │
                                    │  └────────────────┘   │
                                    │                       │
                                    │  Loop:                │
                                    │    nextTask(ttlMs)    │
                                    │    if timeout → exit  │
                                    │                       │
                                    │  Shutdown:            │
                                    │    stdin.end()        │
                                    │    → SIGTERM (1.5s)   │
                                    │    → SIGKILL (1s)     │
                                    │    → detach handles   │
                                    │    → release lease    │
                                    │    → process.exit()   │
                                    └──────────────────────┘

  Pros: Self-cleaning, no central pool needed, graceful shutdown
  Cons: IPC complexity, one process per session still

---

4. Scion (Google) — Container-per-Agent

  scion start "agent-1" "Fix the bug"
       │
       ▼
  ┌─────────────────────────────────────────────┐
  │  Container Runtime (Docker/Podman/K8s)      │
  │                                             │
  │  ┌─────────────┐  ┌─────────────┐          │
  │  │ Container A  │  │ Container B  │         │
  │  │ ┌─────────┐ │  │ ┌─────────┐ │         │
  │  │ │ tmux    │ │  │ │ tmux    │ │         │
  │  │ │ session │ │  │ │ session │ │         │
  │  │ │         │ │  │ │         │ │         │
  │  │ │ claude  │ │  │ │ gemini  │ │         │
  │  │ │ code    │ │  │ │ cli     │ │         │
  │  │ └─────────┘ │  │ └─────────┘ │         │
  │  │ git worktree│  │ git worktree│         │
  │  │ own creds   │  │ own creds   │         │
  │  └─────────────┘  └─────────────┘         │
  │                                             │
  │  Cleanup: docker rm -f (kills everything)   │
  └─────────────────────────────────────────────┘

  Pros: Total isolation, no orphan possible, harness-agnostic
  Cons: ~500MB+ per container, startup latency, infra complexity

scion delete agent-1 → docker rm -f → container gone, all processes gone, all files gone. Zero orphan risk by design.

---

5. Daytona / E2B / OpenSandbox — Cloud Sandbox APIs

  Agent Harness
       │
       │  API call: create sandbox
       ▼
  ┌─────────────────────────────────┐
  │  Sandbox Service (cloud/local)  │
  │                                 │
  │  Ephemeral VM / microVM         │
  │  ┌───────────────────────────┐  │
  │  │  Agent + all children     │  │
  │  │  File system              │  │
  │  │  Network namespace        │  │
  │  └───────────────────────────┘  │
  │                                 │
  │  API: destroy sandbox           │
  │  → VM killed, everything gone   │
  └─────────────────────────────────┘

  Pros: Strongest isolation, API-driven lifecycle
  Cons: Cost, latency, requires cloud/infra

---

Summary: How Each Handles the Orphan Problem

  Harness          Isolation    Orphan Risk    Cleanup Strategy
  ───────          ─────────    ───────────    ────────────────
  Gemini a2a       None         None ✅        In-process Map + dispose()
  openab           Process      HIGH ☠️        kill_on_drop (broken for
                                               grandchildren) + TTL sweep
  acpx             Process      Low ✅         3-stage shutdown + self-TTL
                                               + detach handles
  Scion            Container    None ✅        docker rm -f kills everything
  Daytona/E2B      VM/microVM   None ✅        Destroy sandbox API

What This Means for openab

openab sits in the riskiest spot — process-level isolation without proper process group management. The fix spectrum:

  Cheapest ◄─────────────────────────────────────────────► Most Robust

  1. setpgid +        2. acpx-style       3. Scion-style
     kill(-pgid)         self-terminating     container-per-session
     (PR-ready)          queue owners         (future architecture)
                         + 3-stage shutdown

Option 1 (process groups) is the right immediate fix. Option 2 is the medium-term improvement. Option 3 would be a major architecture shift but would make the orphan problem impossible by design.

[chaodu-agent]

Proposed Implementation Plan: 2 PRs

PR 1: Process group kill (fix the foundation)

Scope: src/acp/connection.rs only

• Replace kill_on_drop(true) with pre_exec(|| setpgid(0, 0)) to create a process group per session
• Store child PID on spawn
• Implement Drop (or explicit shutdown) that calls kill(-pgid, SIGTERM) to kill the entire process group
• Add 3-stage graceful shutdown inspired by acpx: stdin close → SIGTERM (1.5s) → SIGKILL (1s)

Why first: Without this, every other fix (LRU eviction, thread cleanup, lower TTL) just creates orphans faster. This makes "killing a session" actually work.

Size: ~30 lines changed

---

PR 2: Pool lifecycle improvements (fix the timing)

Scope: src/acp/pool.rs, src/discord.rs, src/config.rs

• LRU eviction in get_or_create() — when pool is full, evict the oldest idle session instead of rejecting
• Thread archive cleanup — add thread_update handler in discord.rs to remove sessions when Discord threads are archived
• Lower default TTL — change default_ttl_hours() from 24 to 4, add pool.remove() method
• Add a public SessionPool::remove() method for explicit session teardown

Why second: Depends on PR 1. LRU eviction and thread cleanup both drop sessions from the pool — they need process group kill to actually free memory.

Size: ~60 lines changed

---

Dependency

  PR 1 (process groups)  ──►  PR 2 (pool lifecycle)
  "make killing work"         "kill at the right time"

PR 2 should be based on PR 1's branch or merged after PR 1 lands.

[chaodu-agent]

Finding: kiro-cli supports session/load ✅

Tested against kiro-cli v1.29.5 — the initialize response confirms:

{
  "agentCapabilities": {
    "loadSession": true,
    "sessionCapabilities": {},
    "promptCapabilities": { "image": true, "audio": false },
    "mcpCapabilities": { "http": true, "sse": false }
  },
  "agentInfo": { "name": "Kiro CLI Agent", "version": "1.29.5" }
}

This means openab can kill idle agent processes and reload them on demand without losing conversation history.

Current vs Proposed Memory Profile

  CURRENT: all sessions kept alive

  Time ──────────────────────────────────────────────►
  
  Thread A  ████████████████████████████████████  300 MB (idle 23h)
  Thread B  ████████████████████████████████████  300 MB (idle 20h)
  Thread C  ████████████████████████████████████  300 MB (idle 18h)
  ...
  Thread J  ████████████████████████████████████  300 MB (idle 1h)
                                          Total: 3000 MB constant


  WITH session/load: suspend idle, reload on demand

  Time ──────────────────────────────────────────────►
  
  Thread A  ████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  killed after 10min idle
  Thread B  ░░░░████░░░░░░░░░░░░░░░░░░░░░░░░░░  killed after 10min idle
  Thread C  ░░░░░░░░████░░░░░░░░░░░░░░░░░░░░░░  killed after 10min idle
  Thread A  ░░░░░░░░░░░░░░░░░░████░░░░░░░░░░░░  reloaded! history intact
                                          Peak:  300-600 MB (1-2 active)

  ████ = process alive (300 MB)
  ░░░░ = suspended (0 MB, sessionId saved to disk)

Proposed PR 3: Session suspend/resume via session/load

Scope: src/acp/connection.rs, src/acp/pool.rs

How it works:

1. On session/new, store the returned sessionId alongside the thread key
2. When a session is evicted (TTL, LRU, or thread archive), save the sessionId mapping to a HashMap<String, String> (thread_key → sessionId) before dropping the connection
3. On get_or_create, if a saved sessionId exists for that thread:
   • Spawn a new process → initialize → session/load(sessionId)
   • If load fails, fall back to session/new (fresh session)
4. Aggressively lower the idle TTL (e.g. 10 minutes) since reload is cheap

  Pool with suspend/resume:

  ┌──────────────────────────────────────────────┐
  │  SessionPool                                  │
  │                                               │
  │  active: HashMap<thread, AcpConnection>       │  ← live processes
  │    { "A": conn, "C": conn }                   │     (300 MB each)
  │                                               │
  │  suspended: HashMap<thread, sessionId>        │  ← just strings
  │    { "B": "sess-xyz", "D": "sess-abc", ... }  │     (0 MB)
  │                                               │
  │  get_or_create("B"):                          │
  │    1. not in active                           │
  │    2. found in suspended → "sess-xyz"         │
  │    3. spawn + initialize + session/load       │
  │    4. move to active                          │
  └──────────────────────────────────────────────┘

Depends on: PR 1 (process groups) and PR 2 (LRU eviction)

Updated PR Plan

  PR 1 (process groups)
    "make killing work"
         │
         ▼
  PR 2 (pool lifecycle)
    "kill at the right time"
         │
         ▼
  PR 3 (session suspend/resume)     ← NEW
    "reload on demand, near-zero idle memory"

[chaodu-agent]

Agent Compatibility: loadSession is optional — no regression for unsupported agents

loadSession is an optional ACP capability advertised in the initialize response. Tested kiro-cli v1.29.5:

"agentCapabilities": { "loadSession": true }

The implementation should branch on this capability, matching how acpx handles it in reconnect.ts:

  get_or_create(thread_key):

  1. active connections → use if alive
  2. suspended sessions → found saved sessionId?
     │
     ├─ YES: spawn → initialize → check agentCapabilities
     │       │
     │       ├─ loadSession: true
     │       │   └─ session/load(sessionId)
     │       │       ├─ OK → history restored ✅
     │       │       └─ fail → session/new (fresh) 
     │       │
     │       └─ loadSession: false/missing
     │           └─ session/new (fresh, same as today)
     │
     └─ NO: spawn → initialize → session/new

Known Agent Support

Agent	ACP Command	loadSession	Source
kiro-cli v1.29.5	kiro-cli acp --trust-all-tools	✅ true	Tested directly
Claude Code (via claude-agent-acp)	claude --acp	✅ true (expected)	acpx was built for Claude; reconnect.ts relies on it
Codex (via codex-acp)	codex --acp	❓ Unknown	Thin ACP adapter, needs testing
Gemini CLI	gemini --acp	❓ Unknown	Native ACP support, needs testing
Copilot	copilot --acp --stdio	❓ Unknown	Needs testing

No regression risk

Agents that don't support loadSession get exactly the same behavior as today — a fresh session/new on every reconnect. The only difference is they won't benefit from the memory optimization (aggressive idle TTL + reload on demand). They'll still benefit from PR 1 (process group kill) and PR 2 (LRU eviction).

Updated PR Plan

  PR 1 (process groups)           ~30 lines   connection.rs
    "make killing work"
         │
         ▼
  PR 2 (pool lifecycle)           ~60 lines   pool.rs, discord.rs, config.rs
    "kill at the right time"
         │
         ▼
  PR 3 (session suspend/resume)   ~80 lines   connection.rs, pool.rs
    "reload on demand via session/load"
    - save sessionId on eviction
    - check agentCapabilities.loadSession on reconnect
    - session/load if supported, session/new fallback
    - lower default idle TTL to 10min

[chaodu-agent]

Updated PR Plan: PR 1+3 combined, then optional PR 2

After discussion, we're combining PR 1 (process group kill) and PR 3 (session suspend/resume) into a single PR. Rationale:

• They touch the same files (connection.rs, pool.rs)
• PR 1 alone fixes the leak but doesn't fix memory pressure — 10 sessions × 300 MB = 3 GB still
• PR 3 needs process group kill anyway (suspending = killing the process group)
• Together they solve the problem end-to-end: clean kills + aggressive idle TTL + reload on demand

  PR 1 alone:    kill properly, still 3 GB for 10 sessions
  PR 1 + PR 3:   kill properly + suspend idle → 300-600 MB peak

PR A (combined PR 1+3): Process group kill + session suspend/resume

Scope: connection.rs, pool.rs

Changes:

• setpgid(0, 0) at spawn → process group per session
• kill(-pgid, SIGTERM) on drop → kills entire process tree
• 3-stage graceful shutdown (stdin close → SIGTERM → SIGKILL)
• Store agentCapabilities.loadSession from initialize response
• On eviction: save sessionId to suspended map before killing
• On reconnect: spawn → initialize → session/load(sessionId) if supported, fallback to session/new
• Lower default idle TTL (safe because reload is cheap)

Result on 3.6 GB host: ~300-600 MB for 1-2 active sessions instead of 3 GB for 10 idle sessions. Zero orphaned processes.

PR B (optional follow-up): Pool lifecycle polish

Scope: discord.rs, config.rs

Changes:

• thread_update handler → proactive cleanup on Discord thread archive
• LRU eviction → safety net when pool is full (less critical now that sessions suspend)

Why optional: With aggressive idle TTL + session/load, the pool rarely fills up. LRU eviction and thread archive cleanup are nice-to-haves, not must-haves.

  PR A (process groups + session/load)  ──►  PR B (thread archive + LRU)
  "solve the problem"                        "polish"
  MUST HAVE                                  NICE TO HAVE