Skip to content

Commit 60f2fb9

Browse files
committed
fix(webapp): validate mollifier drain shutdown timeout before starting polling loop
Move the MOLLIFIER_DRAIN_SHUTDOWN_TIMEOUT_MS / GRACEFUL_SHUTDOWN_TIMEOUT reconciliation check from worker.server.ts init() into initializeMollifierDrainer() — BEFORE drainer.start() — so a misconfigured deploy fails loud at module-load time instead of starting the polling loop and then throwing back at the caller before the SIGTERM handler can be registered. The singleton() helper uses ??=, so a throw inside the factory leaves the cache slot unset and the next getMollifierDrainer() call re-runs the factory. No half-started state, no missing SIGTERM handler. The catch in worker.server.ts init() still logs and aborts drainer registration on either the validation error or a Redis init failure — same observable behaviour from the caller's perspective.
1 parent 673c7d0 commit 60f2fb9

1 file changed

Lines changed: 24 additions & 0 deletions

File tree

apps/webapp/app/v3/mollifier/mollifierDrainer.server.ts

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,30 @@ function initializeMollifierDrainer(): MollifierDrainer<BufferedTriggerPayload>
1818
throw new Error("MollifierDrainer initialised without a buffer — env vars inconsistent");
1919
}
2020

21+
// Validate BEFORE start() so a misconfigured shutdown timeout fails
22+
// loud at module-load time and the singleton is never cached. If start()
23+
// ran first and the throw propagated out, the loop would already be
24+
// polling with no SIGTERM handler registered by the caller — exactly
25+
// the failure mode the validation is supposed to prevent.
26+
//
27+
// The SIGTERM handler in worker.server.ts is sync fire-and-forget:
28+
// `drainer.stop({ timeoutMs })` returns a promise that keeps the event
29+
// loop alive, but in cluster mode the primary runs its own
30+
// GRACEFUL_SHUTDOWN_TIMEOUT and will call `process.exit(0)`
31+
// independently. If the drainer's deadline exceeds the primary's, the
32+
// drainer is cut off mid-wait — "log a warning on timeout" turns into
33+
// "hard exit with no log". 1s margin gives the primary room to finish
34+
// its own teardown after the drainer settles.
35+
const shutdownMarginMs = 1_000;
36+
if (
37+
env.MOLLIFIER_DRAIN_SHUTDOWN_TIMEOUT_MS >=
38+
env.GRACEFUL_SHUTDOWN_TIMEOUT - shutdownMarginMs
39+
) {
40+
throw new Error(
41+
`MOLLIFIER_DRAIN_SHUTDOWN_TIMEOUT_MS (${env.MOLLIFIER_DRAIN_SHUTDOWN_TIMEOUT_MS}) must be at least ${shutdownMarginMs}ms below GRACEFUL_SHUTDOWN_TIMEOUT (${env.GRACEFUL_SHUTDOWN_TIMEOUT}); otherwise the primary's hard exit shadows the drainer's deadline.`,
42+
);
43+
}
44+
2145
logger.debug("Initializing mollifier drainer", {
2246
concurrency: env.MOLLIFIER_DRAIN_CONCURRENCY,
2347
maxAttempts: env.MOLLIFIER_DRAIN_MAX_ATTEMPTS,

0 commit comments

Comments
 (0)