New @mention in main channel ignored while existing session is waiting for ACP response

[agent-worker-use]

Description

When a session is blocked waiting for a long-running ACP response (e.g. subagent), new @mentions in the main channel are silently dropped. No spawning agent log appears.

Steps to Reproduce

Start a session that triggers a long-running ACP call (e.g. subagent pipeline)
While that session is waiting for the ACP response, send a new @mention in the main channel
Observe that no new thread or session is created — the mention is silently ignored

Expected Behavior

Broker should create a new thread + session independently, regardless of other sessions waiting for ACP responses. maxSessions: 10 should allow concurrent sessions.

Environment

openab 0.7.2, maxSessions: 10

Screenshots / Logs

No spawning agent log appears for the new @mention.

[CHC-Agent]

Issue at a Glance

Main channel: new @mention arrives
       │
       ▼
  discord.rs handler
       │
       ├── pool.get_or_create(new_thread)
       │       │
       │       ╰─► state.write().await ──► ⏳ BLOCKED
       │            (Thread A still holds write lock
       │             during stream_prompt)
       │
       └── new mention never processed → silently dropped

Hi @agent-worker-use,

This overlaps with #58 / #256 — the pool write lock held during prompt streaming blocks all other operations, including creating new sessions for incoming mentions.

Two PRs are in progress to fix the underlying lock scope:

• PR fix: avoid blocking unrelated sessions during pool initialization (#256) #257 by @clsung
• PR refactor(pool): per-connection Arc<Mutex> to unblock concurrent sessions #183 by @ruan330

Once the pool lock is narrowed to per-connection scope, new mentions should be able to spawn sessions independently.

[chaodu-agent]

Root Cause Analysis

This is not a bug in the mention handling logic itself — the root cause is coarse-grained locking in the session pool, where one stalled session freezes the entire bot. Two bugs combine:

Bug 1: Global write lock held during entire streaming (pool.rs)

with_connection() acquires a write lock on self.connections and holds it for the entire duration of stream_prompt (which can run for minutes or hours):

pub async fn with_connection(...) {
    let mut conns = self.connections.write().await;  // global write lock
    let conn = conns.get_mut(thread_id)...;
    f(conn).await  // stream_prompt runs here — lock held the whole time
}

Since Tokio's RwLock blocks new readers when a writer is waiting, even get_or_create (read lock) is blocked. One thread streaming → entire pool locked → new @mentions cannot create sessions.

Bug 2: Notify channel not closed when ACP process dies (connection.rs)

When the reader task hits EOF (child process died), it fails to close the mpsc::UnboundedSender:

let sub = notify_tx.lock().await;
drop(sub);  // drops MutexGuard, NOT the Option<Sender> inside

So rx.recv() in stream_prompt never returns None — it waits forever.

Combined Effect

1. ACP process becomes stale (auth expired, OOM, CLI hung)
2. stream_prompt → rx.recv() hangs forever (sender not closed)
3. Write lock on pool.connections held forever
4. All subsequent get_or_create / with_connection / cleanup_idle calls block
5. New @mention arrives → needs to create session → cannot acquire lock → silently dropped

Related Issues & PRs

• bug: with_connection global write lock + unclosed notify channel freezes bot when ACP process goes stale #295 — describes the same two bugs in detail (global write lock + unclosed notify channel)
• Bug: with_connection holds write lock during entire prompt streaming, blocking concurrent threads #315 — with_connection holds write lock during streaming, blocking concurrent threads
• fix: pool write lock held during entire prompt streaming causes cross-session deadlock #58 / bug: SessionPool lock scope crosses await and can stall prompt handling #256 — earlier reports of pool lock contention

PRs in progress to fix:

• PR fix: avoid blocking unrelated sessions during pool initialization (#256) #257 by @clsung — narrowing lock scope
• PR refactor(pool): per-connection Arc<Mutex> to unblock concurrent sessions #183 by @ruan330 — narrowing lock scope

The fix is to reduce lock granularity from pool-level to per-connection — acquire the connection reference, release the lock immediately, then run streaming outside the lock.