ds4-server: watchdog thread + decode-loop SSE keepalive

[Allen091080]

Summary

Builds on the SSE keepalive merged in f027269 (PR #194). That patch
covered prefill silence; this one covers the other two failure modes
on the same surface:

1. Worker thread stalls in GPU/Metal kernel calls with no
   cancellation points. SIGTERM cannot drain a stuck turn — we observed
   a 12+ hour ds4-server process wedged on a single chat request,
   requiring SIGKILL.
2. Decode-loop silence during reasoning-only stretches (<think>...)
   or slow tool-input phases: the prefill keepalive does not fire, the
   decode loop produces no streamable bytes for a while, and the client
   TCP idle-timeout closes the connection.

Fix

Two cooperating pieces, both in ds4_server.c:

A) Watchdog thread

A dedicated watchdog_main polls worker_last_progress every 5s
while worker_in_job is set.

• Soft stall (default 60s): set worker_abort_requested so the
  decode loop bails on its next iteration; the client gets an error
  finish_reason and the worker continues serving future jobs.
• Hard stall (default 120s): call _exit(137). The launchd
  KeepAlive supervisor restarts immediately; on-disk KV cache
  survives so the restarted process can usually resume the client's
  last prefix from cache instead of re-prefilling.

B) Decode-loop additions

The decode loop now:

• Checks worker_abort_requested at the top of each iteration, breaks
  with a clear finish=error when the watchdog has requested abort.
• Emits a : decode\n\n SSE comment line at most every 15s when
  j->req.stream is set (matches the prefill keepalive cadence).
• Refreshes worker_last_progress after each produced token via
  relaxed __atomic_store_n.

server_progress_cb also refreshes worker_last_progress on every
prefill_chunk callback — without this, big prefills (≥ soft threshold)
would be mistaken for stalls. Chunks fire every ~7s in practice and
reset the timestamp well within the 60s soft limit, so prefill is
never falsely aborted.

State additions

pthread_t worker_tid;
pthread_t watchdog_tid;
bool watchdog_running;
int watchdog_stuck_soft_s;  // default 60
int watchdog_stuck_hard_s;  // default 120

// relaxed __atomic_* accessed across threads
volatile long worker_last_progress;
volatile int worker_in_job;
volatile int worker_abort_requested;

The watchdog only acts while worker_in_job is set — idle workers
don't advance progress and we must not misinterpret that as a stall.

Verification

Machine: MacBook Pro M5 Max, 128 GiB RAM
Backend: Metal
Model: DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf (q2-imatrix)
Server: ./ds4-server --host 0.0.0.0 --port 8000 --ctx 500000 --kv-disk-dir … --kv-disk-space-mb 204800

• make — clean build, no new warnings.
• ./ds4_test --server — server: OK / ds4 tests: ok.
• Real running ds4-server protected: in an earlier multi-hour run we
  saw a hard wedge that needed SIGKILL. With the patch applied, a
  follow-up run hit the soft trigger (WATCHDOG soft stall 62s >= 60s — requesting decode-loop abort) and the client got a clean error
  within 70s instead of hanging indefinitely. launchd KeepAlive
  picked the process back up.
• 35s decode burst with reasoning: client receives : decode comment
  lines every ~15s, no client disconnects.

Test plan

• CI runs ./ds4_test --server
• Manual: run a heavy chat request long enough to provoke a real
  or simulated stall (e.g. add a temporary sleep in ds4_session_eval)
  and confirm the soft trigger fires within ~65s and _exit(137)
  fires within ~125s.
• Confirm kill -15 still drains cleanly via the existing
  g_stop_requested shutdown path — the watchdog is purely
  additive.

[antirez]

Fixed more organically in recent commits. See the original issue for more info. Thanks.

[Allen091080]

Thanks for the quick reply! Agreed — f91c12b's prefill_display event is the right shape for prefill stalls and lines up better with the project's minimalism.

Closing this PR. I'll send a much smaller follow-up that keeps only the decode-loop SSE keepalive piece (no watchdog thread, no _exit), since long-thinking and long tool-input phases during decode produce no streamable bytes for a while either and the current prefill keepalive doesn't cover those.