sendMessageDraft 429 cascade during long streaming replies in DMs

[fitz123]

Summary

During long streaming replies in DM bindings (kind: "dm"), the bot generates large bursts of bot_telegram_api_errors_total{method="sendMessageDraft", error_code="429"} and bot_telegram_api_errors_total{method="sendChatAction", error_code="429"} entries. The user-visible outcome is fine — the final sendMessage always delivers — but the log noise is heavy and the Prometheus TelegramAPIErrors alert fires repeatedly during normal usage.

Today (over one day of moderately heavy DM usage) the bot logged 428 rate-limit lines, vs. 28-31 per day in the preceding baseline week. The single chat-volume spike drove a ~14× increase in error counter values, with no actual delivery failures observed.

Evidence

Counter snapshot (anonymized):

bot_telegram_api_errors_total{method="sendMessageDraft",error_code="429"} 378
bot_telegram_api_errors_total{method="sendChatAction",error_code="429"} 50

Sample log lines from one burst:

2026-05-15T19:10:29.607Z WARN [telegram-api] Rate limited: method=sendMessageDraft retry_after=3
2026-05-15T19:10:31.706Z WARN [telegram-api] Rate limited: method=sendMessageDraft retry_after=3
2026-05-15T19:10:31.758Z WARN [telegram-api] Rate limited: method=sendMessageDraft retry_after=3
...

Hour-by-hour burst counts on a high-volume day:

UTC hour	rate-limit lines
09	17
12	67
15	19
16	71
17	145
18	43
19	66

16 burst clusters. Each cluster aligns 1:1 with the end of a single assistant turn whose final reply length exceeded ~2 KB. Linear correlation between final-reply size and burst size:

Final reply (chars)	Avg rate-limit lines per turn
< 2,000	1–4
2,000–2,800	17–24
2,800–3,500	20–24
3,500–4,000	36–43
> 4,000	66 (max sample)

Root cause

Three multiplying factors:

1. DRAFT_DEBOUNCE_MS = 300 (bot/src/stream-relay.ts:124) — while a reply streams, draft updates fire roughly every 300 ms = up to ~3.3 calls/sec per chat.
2. Per-chat Telegram limit on draft API — empirically Telegram throttles sendMessageDraft to ~1 call/sec for the same chat_id, matching the documented general per-chat send-rate (Bot API FAQ "Broadcasting to users"). The official sendMessageDraft method spec does not document a draft-specific exemption.
3. AUTO_RETRY_OPTIONS.maxRetryAttempts = 5 (bot/src/telegram-bot.ts:497-501) combined with @grammyjs/auto-retry honoring each retry_after (3-10 s) — every rate-limited logical sendMessageDraft call balloons into up to 5 additional retries, each logged.

For a 2.5 KB reply streaming over ~10-25 s the math is roughly 30-80 draft attempts × ~80% rejection rate × autoRetry 5× = the observed burst sizes.

What this is NOT

Verified during investigation:

• DM guard works. bot/src/telegram-adapter.ts:60-71 correctly short-circuits sendDraft when binding.kind !== "dm". No sendMessageDraft ever reaches the wire from non-DM bindings; all 378 rate-limit lines today were attributable to a single DM binding.
• Not an agent-side loop or tool storm. The largest tool-heavy turn observed (6× Bash plus Write/Read) produced zero rate-limit errors, because its output was tool-bound rather than streamed prose.
• Not retry-on-error in the agent. No turn-level error/retry patterns appear in the affected session JSONLs.
• Not delivery failure. Every affected turn delivered its final sendMessage successfully.

The problem is purely the cascading retry behavior on a cosmetic operation.

Proposals

Two independent options. Either alone would substantially reduce the cascade; together they would eliminate it.

Option A — raise draft debounce

bot/src/stream-relay.ts:124: increase DRAFT_DEBOUNCE_MS from 300 to 750-1000 so the steady-state draft rate stays under Telegram's per-chat threshold.

Trade-off: slightly choppier draft updates in the user's chat (drafts refresh ~1×/sec instead of ~3×/sec). The final sendMessage is unaffected.

Option B — disable autoRetry for draft methods

Drafts are cosmetic and fire-and-forget (see bot/src/stream-relay.ts:180 — failures are caught and discarded). A 429 on a draft has no recovery value: by the time autoRetry waits 3-10 s, the streaming text has already moved on. autoRetry on sendMessageDraft purely amplifies log noise.

Either lower AUTO_RETRY_OPTIONS.maxRetryAttempts globally (5 → 1), or selectively skip autoRetry for sendMessageDraft via a per-method filter.

Trade-off: lower global retry limit affects all methods — final sendMessage would also retry less aggressively (currently up to 5×). A per-method skip is more targeted.

Recommended order

1. Wait for the in-flight observability PR (chat_id in error logs + new bot_telegram_api_calls_total counter labelled by binding) to land and gather one day of data.
2. Compute the actual error-to-total ratio for sendMessageDraft from the new counter.
3. Decide based on that ratio whether one option suffices or both are warranted.

Out of scope for this issue

• The observability work (chat_id in logs + call counter) is being implemented separately and is not blocked by this issue.
• The Prometheus alert expression at monitoring/prometheus/rules.yml:22-23 lives in a private monitoring repo and is not changed by this issue. It will likely be revised to use the new ratio once the counter is available.

References

• Migration that introduced this code path: Replace editMessageText streaming with sendMessageDraft for DMs #71
• Bot API method spec: https://core.telegram.org/bots/api#sendmessagedraft
• Bot API broadcast rate limits: https://core.telegram.org/bots/faq#my-bot-is-hitting-limits-how-do-i-avoid-this

[fitz123]

Independent review (codex CLI + a fresh Opus session, read source separately, no shared context)

Both reviewers converged on Option B as the primary fix, via a thin wrapper transformer that skips @grammyjs/auto-retry for sendMessageDraft only. Both produced the same implementation pattern independently:

const retry = autoRetry(AUTO_RETRY_OPTIONS);
bot.api.config.use(async (prev, method, payload, signal) => {
  if (method === "sendMessageDraft") return prev(method, payload, signal);
  return retry(prev, method, payload, signal);
});

Rationale: @grammyjs/auto-retry exposes no per-method filter (verified in node_modules/@grammyjs/auto-retry/out/mod.js — only the four top-level options). The wrapper preserves full retry semantics for sendMessage and every other method, while drafts pass through once and fail fast. The inner logging transformer continues to write one Rate limited: line per logical call instead of one per retry attempt, collapsing ~80% of the log volume at the root.

Disagreement on whether Option A is also needed

• One reviewer: Option B alone is sufficient. Option A reduces the source rate of drafts but each one still incurs up to 5 retry attempts × 3–10 s delay until B is in place — the bulk of the 429 log lines come from retry amplification (~5×), not from initial draft frequency. Expected reduction from A alone is bounded at ~17%.
• Other reviewer: Do both. With B alone, steady-state draft rate is still ~3.3/sec per active DM stream, which still exceeds Telegram's per-chat soft limit; A=1000 ms keeps the source rate under the threshold. The two are complementary, not substitutes.

Both technically correct: B removes the amplification, A reduces the source. Implementing only B first and measuring the residual via the in-flight observability counter is a reasonable sequencing.

Findings not covered in the original issue body

1. Stacked in-flight retries. The current debounce gate at bot/src/stream-relay.ts:186 (if (draftTimer) return) only prevents queued timers — it does not prevent concurrent HTTP calls already retrying inside autoRetry. During a 30 s autoRetry stall, new sendDraftNow() calls continue firing every ~300 ms, so draftPromises[] can grow unboundedly during a single stream. Raising DRAFT_DEBOUNCE_MS mitigates but does not eliminate this; per-chat concurrency control (e.g. an inFlightDraft: Promise<void> per chat that newer drafts await rather than stack on) would be cleaner.

2. Misleading inline comment. bot/src/stream-relay.ts:123 currently reads "Drafts are cosmetic — no rate limits.", which appears to be the assumption that originally motivated the 300 ms value. Worth correcting in the same PR as Option B to prevent future regression.

3. Orphan drafts on bot restart. If the bot exits mid-stream, the most recent draft persists server-side until a sendMessage with the same draft_id clears it. Behavior of Telegram's draft-clearing on missing final sendMessage is unclear from the current Bot API spec — worth verifying whether a follow-up sendMessage (with the same draft_id) is required to clear, or whether drafts auto-expire after the documented 30-second preview window without any explicit clear. Not blocking Option B; tracked as a follow-up.

4. rethrowHttpErrors: false semantics. Per node_modules/@grammyjs/auto-retry/out/mod.js:49-65, transport-level HTTP errors are retried inside a tighter inner loop without the configured maxRetryAttempts applying the same way. This is also undesirable for drafts and is automatically resolved by the wrapper-transformer approach for B.

Caveats and corrections to the original issue

• File:line references in the issue body were captured before the in-flight observability work modified bot/src/telegram-bot.ts. Behavioral claims still hold; line numbers need refreshing when the fix PR is opened.
• "Wait for observability PR before deciding" — both reviewers pushed back on this if interpreted as indefinite. The fix is ~10 lines, low-risk (drafts already silently fail at the call site), and diagnosis is strong enough to act now. Reasonable sequencing: ship Option B promptly; use the new bot_telegram_api_calls_total counter (once landed) to decide whether Option A is also warranted.

Updated recommendation

1. Ship Option B (wrapper transformer skipping autoRetry for sendMessageDraft) + fix the misleading comment at bot/src/stream-relay.ts:123 in the same PR.
2. Wait for one full day of data from the new counter to measure residual draft 429 ratio.
3. Decide on Option A and on the concurrency-lock follow-up based on that data.