fix(library-identity-consumer): stringify lastVerifiedAt to defeat Drizzle's date-serializer override

[jakebromberg]

Summary

First prod run of library-identity-consumer on 2026-05-20 had 14,405 / 14,405 UPSERTs fail with zero library_identity rows landing. LML resolution was healthy (matched dry-run exactly); the failure was 100% writer-side.

Root cause is a Drizzle / postgres-js interaction documented at porsager/postgres#761 and present in node_modules/drizzle-orm/postgres-js/driver.js:19:

const transparentParser = (val) => val;
for (const type of ['1184', '1082', '1083', '1114', '1182', '1185', '1115', '1231']) {
  client.options.parsers[type] = transparentParser;
  client.options.serializers[type] = transparentParser;  // ← also rebinds outbound serializer
}

The override is intended for inbound parsers (so reads return raw text for Drizzle to convert itself), but the same loop also rebinds the outbound serializer for OID 1184 (timestamptz). The native postgres-js serializer would call .toISOString() on Date objects; the transparent passthrough returns the Date unchanged. Then postgres-js's Bind() → b.str(date) → Buffer.byteLength(date) → ERR_INVALID_ARG_TYPE: The "string" argument must be of type string or an instance of Buffer or ArrayBuffer. Received an instance of Date.

Why no other job hit this

library-canonical-entity-backfill, flowsheet-metadata-backfill, library-artwork-url-backfill, and flowsheet-dj-name-backfill all sidestep the bug by either using now() in SQL or by going through db.insert(...).values({...}) (Drizzle's column-aware encoder path, which doesn't hit the transparent serializer). The consumer's writeSingleArtist is the only writer in the repo that passes a JS Date as ${value} inside a sql\`` raw template.

The fix

new Date() → new Date().toISOString(). A pre-stringified ISO 8601 string passes through the transparent serializer unchanged, and PG parses the literal as a timestamptz happily.

Diff is +11/-1 in a single file, with a long comment block explaining the citation so future readers don't need to re-walk this investigation.

Test plan

• Direct repro of the failure via a Node script using the same drizzle() + tx.execute(sql\`)` code path as the writer — fails with the documented ERR_INVALID_ARG_TYPE.
• End-to-end verification of the fix against prod: both the library_identity_source and library_identity UPSERTs land cleanly when lastVerifiedAt is pre-stringified.
• npx tsc --noEmit -p jobs/library-identity-consumer/tsconfig.json clean.

Deploy + re-run

After this lands:

1. Deploy via Manual Build & Deploy target=library-identity-consumer.
2. Pull the new image on EC2: docker pull 203767826763.dkr.ecr.us-east-1.amazonaws.com/library-identity-consumer:<new tag>.
3. Re-run the consumer (no DRY_RUN) — expected: 14,405 `rows_resolved`, ~30K source rows, 15,376 `rows_unresolved`, 0 `writer_error`.
4. Confirms library_identity populated for the 14,405 resolved rows.

Recommend also shipping the companion observability fix (PR #975) so future writer failures actually surface the PG code/constraint in logs instead of just "Failed query".

Related

• Closes (in spirit) — [§4 step 2 post-pivot] Consume LML POST /api/v1/identity/bulk-resolve-libraries #802 didn't track this since the issue was in the post-merge first-run.
• Companion observability fix: PR fix(library-identity-consumer): log error.cause on writer failure so PG diagnostics surface #975 (cause-logging on writer failures).
• Followups already filed: BS#974 (predicate scope for NULL canonical_entity_id rows), LML#355 (51.6% unresolved rate investigation).

[jakebromberg]

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

• Fix is minimal (+11/-1) and the root-cause analysis matches the postgres-js / Drizzle behavior cited in the body.
• new Date().toISOString() is the same pattern already used elsewhere in jobs/ for ISO-8601-into-raw-SQL paths.
• The two other new Date() callsites in jobs (flowsheet-etl/job.ts, library-etl/job.ts) go through db.insert(...).values({...}) and so don't hit the transparent serializer — consistent with the PR body's diagnosis.
• No other ${...Date...} interpolations in sql\`raw templates acrossjobs/, apps/, shared/` — the consumer's writer was the only site.