Vector aggregator fails to reconnect after NATS lame duck mode

[scotwells]

Problem

The Vector aggregator deployment gets stuck and never reconnects to the NATS server after the NATS server enters lame duck mode. The error message observed is:

lame duck mode detected

Impact

• Activity processing pipeline stops working
• Audit logs and events are not processed until Vector pods are manually restarted
• Requires manual intervention to restore functionality

Expected Behavior

Vector should automatically reconnect to the NATS server after it exits lame duck mode or switches to a new leader.

Investigation Areas

1. Vector NATS source configuration - reconnection settings
2. NATS client library behavior in lame duck mode
3. Kubernetes deployment/liveness probe configuration
4. Potential need for connection retry logic or pod restart policy

Reproduction

Occurs when NATS server enters lame duck mode (e.g., during rolling updates or node drains).

[scotwells]

Investigation Summary

Root Cause

Vector's NATS source fails to reconnect after NATS lame duck mode due to a known limitation in the async_nats JetStream client. When the connection drops during lame duck, the internal subscriber channel closes before reconnection completes, causing the Stream iterator to terminate. Vector's source has no retry loop - it exits cleanly with Ok(()).

Code Path

1. NATS server enters lame duck mode → sends INFO with lame_duck_mode: true
2. async_nats logs "lame duck mode detected" (the error we see)
3. Server closes connection during lame duck window
4. async_nats reconnects at connection level ✅
5. JetStream Stream's subscriber channel closes during reconnection ❌
6. poll_recv() returns None → Stream terminates
7. Vector's run_nats_jetstream() returns Ok(()) - no restart loop

Upstream Issues

This is a known problem in async_nats:

• nats-io/nats.rs#224 - "JS Consumer lost after disconnect/reconnect" (2022)
• nats-io/nats.rs#557 - Same issue reported again (2023)
• nats-io/nats.rs#610 - Partial fix merged, but doesn't fully solve it

Community workaround is to wrap the consumer in an application-level retry loop, which Vector doesn't do.

Options

Option	Effort	Notes
A. Configure multiple NATS URLs	Low	Faster failover, may reduce but not eliminate issue
B. Upstream PR to Vector	Medium	Add reconnection loop to NATS source
C. Extend activity-processor	Medium	Write directly to ClickHouse, bypass Vector
D. Custom ingestion service	High	Reinvent Vector's batching/buffering

Version Info

• Vector Helm chart: 0.49.x (runs Vector app 0.51.1)
• async_nats in Vector: 0.42.0
• Latest async_nats: 0.46.0 (no fix for this issue)

Relevant Files

• config/components/vector-aggregator/vector-hr.yaml
• Vector source: src/sources/nats/source.rs - run_nats_jetstream() has no retry loop
• async_nats: src/jetstream/consumer/pull.rs - Stream terminates when subscriber closes