feat(redis): add reconnectOnError for READONLY / LOADING reply errors

During an ElastiCache role swap or node-type change, the TCP/TLS
connection stays open but the server starts answering with READONLY
(the client is talking to a node that became a replica) or LOADING
(node still loading data from disk). Without an explicit hook, these
errors surface to caller code as ReplyError instances — every write
op on the affected connection fails until the cluster cuts over.

Returning 2 from reconnectOnError tells ioredis to tear down the
connection, reconnect, and re-issue the failed command. After
reconnect, DNS / SG state routes the new socket to a writable node.

The shared createRedisClient helper picks this up automatically.
defaultReconnectOnError is exported for direct ioredis call sites
that bypass createRedisClient.

Author: ericallam

.server-changes/redis-reconnect-on-readonly-loading.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: improvement
+---
+
+Add `reconnectOnError` to the shared ioredis client config so READONLY / LOADING reply errors during ElastiCache node-type changes trigger a disconnect-reconnect-retry cycle instead of surfacing to caller code.

internal-packages/redis/src/index.ts

@@ -3,12 +3,32 @@ import { Logger } from "@trigger.dev/core/logger";
 
 export { Redis, type Callback, type RedisOptions, type Result, type RedisCommander } from "ioredis";
 
+/**
+ * Reply-error -> reconnect mapping. Without this hook, an ElastiCache
+ * vertical scale-up surfaces tens of thousands of READONLY / LOADING
+ * reply errors to caller code over a healthy TCP/TLS connection (the
+ * client keeps talking to a node whose role swapped underneath it).
+ *
+ * Returning 2 tells ioredis to disconnect, reconnect, and retry the
+ * command that triggered the error. After reconnect, DNS / SG routing
+ * should land on a writable primary.
+ *
+ * Empirical confirmation on the harness in TRI-8878: this option
+ * reduced a scale-up event from ~437,000 caller-surfaced errors to 2.
+ */
+export function defaultReconnectOnError(err: Error): boolean | 1 | 2 {
+  const msg = err.message ?? "";
+  if (msg.startsWith("READONLY") || msg.startsWith("LOADING")) return 2;
+  return false;
+}
+
 const defaultOptions: Partial<RedisOptions> = {
   retryStrategy: (times: number) => {
     const delay = Math.min(times * 50, 1000);
     return delay;
   },
   maxRetriesPerRequest: process.env.GITHUB_ACTIONS ? 50 : process.env.VITEST ? 5 : 20,
+  reconnectOnError: defaultReconnectOnError,
 };
 
 const logger = new Logger("Redis", "debug");