[Tracker] BS open-issue triage — P0 sequencing, cross-cluster conflicts

[jakebromberg]

End-of-day triage of all open issues in this repo. Entry point for any contributor or agent picking up the work: where to start, what blocks what, which proposals overlap.

This tracker is intended to be edited in place, not appended to. When triage state changes, update the relevant section. Don't add a new "Updates YYYY-MM-DD" block.

Org-level entry point: WXYC/wiki#5 (cross-repo P0 sequencing + patterns).

P0 sequencing

Land before the LML metadata backfill epic (#631 family) ships in earnest. Rationale on each issue.

1. Production flowsheet table is catastrophically bloated (1.4 GB heap, 24 live rows) #525 — flowsheet table 1.4 GB heap, 24 live rows, never autovacuumed. Silently degrades everything else. Code-side fix is operational (VACUUM FULL + per-table autovacuum settings ALTER TABLE) — no migration shipped, so this stays open until the operational pass runs.
2. npm audit — 18 vulnerabilities (2 critical, 8 high) as of 2026-04-14 #75 — npm audit: 2 critical, 8 high vulns (per the 2026-04-14 comment; original title still says "moderate"). Title refresh requested in 2026-04-28 comment.
3. Run LML entity-resolution pipeline to populate entity.identity in prod #572 — single-action unblocker (run LML entity-resolution pipeline in prod). Per the 2026-04-29 comment, superseded by Epic [Epic] E2 — Canonical identity record (library_identity) + LML write contract #663 (E2-BS) and slated to close as part of E2-BS step 1; treat as folded.
4. Backfill library.artwork_url for Discogs-resolvable rows #637 — backfill library.artwork_url for Discogs-resolvable rows. Gates draft PR Denormalize library.artwork_url onto flowsheet on bin-pick #643 (held until library.artwork_url population is materially > 0.2%) and the bin-pick surface of Re-emit liveFs SSE event after metadata enrichment UPDATE lands #628.

After P0: the #631 family's remaining children — #640 (pilot — results recorded 2026-04-30 with provisional partition recommendation; both observability companions BS#669 and LML#229 have now merged, so the recommendation can be re-confirmed against actual cache_stats Sentry attributes if anyone wants to upgrade it from provisional) → #641 (rollout — native blockedBy now narrows to just #640 since #649 closed via PR #683) → #642 (close-out).

Open PRs held intentionally

• PR Denormalize library.artwork_url onto flowsheet on bin-pick #643 — Denormalize library.artwork_url onto flowsheet on bin-pick. Draft, held on Backfill library.artwork_url for Discogs-resolvable rows #637 (artwork population is 0.2% today; denorm SELECT writes ≈zero rows until backfill ships).
• PR [mojibake/m2-4-bs-cache] Audit + invalidate BS metadata caches keyed by mojibake'd names #568 — [mojibake/m2-4-bs-cache] audit + invalidate BS metadata caches. Non-draft, green CI from 2026-04-27, but issue M2.4 — Backend-Service metadata + artwork cache verify #528 it relates to is itself blocked on tubafrenzy#489/[A.5] Search service rewrite: tsvector + plays for Both mode #490/[B-0.5] Recover broken-FK flowsheet entries (~292K, 15%) #493/[B-1.2] Library backfill: resolve canonical entity per row via LML #495 + docs#11.

Stale draft PRs

Scanner cluster (PRs #212, #213, #215, #217) was swept and closed 2026-05-09. Feature branches preserved on origin (feature/library-code-lookup-reviews, feature/scanner-gemini-integration, feature/scanner-batch-processing, feature/scanner-list-batch-jobs); reopen with fresh PRs if the scanner work resumes.

Severity

• CRITICAL: Production flowsheet table is catastrophically bloated (1.4 GB heap, 24 live rows) #525.
• HIGH: npm audit — 18 vulnerabilities (2 critical, 8 high) as of 2026-04-14 #75, Audit ETL post-pass UPDATEs for the unbounded resolveDjNames pattern #516, Flowsheet ETL: shows.legacy_dj_name occasionally not populated even when tubafrenzy has DJ_NAME #605, Re-emit liveFs SSE event after metadata enrichment UPDATE lands #628, [Epic] Historical metadata backfill for ~1.86M flowsheet rows #631 (epic), Re-emit on LML linkage UPDATE (album_id race; sibling of #628) #644, Duplicate hour-boundary breakpoints: dj-site manual breakpoint + tubafrenzy auto-breakpoint #699 (filed 2026-05-01: duplicate hour-boundary breakpoints, awaiting dj-site#479 product call).

Cross-cluster conflicts (decide once before starting)

Artwork / metadata storage location

• Extract artwork and album-level metadata into a dedicated album_metadata table #532 wants to move artwork/metadata columns off flowsheet and library into a new album_metadata table.
• Backfill library.artwork_url for Discogs-resolvable rows #637 backfills library.artwork_url against the current schema.
• Extract artwork and album-level metadata into a dedicated album_metadata table #532 is natively blockedBy [A.3] Phased rollout of flowsheet-metadata-backfill #641 (rollout-plateau is the unblock signal). Conflict against [A.1.b] Add metadata_attempt_at marker and flip backfill to recurring drift-repair #639's metadata_attempt_at column resolved when [A.1.b] Add metadata_attempt_at marker and flip backfill to recurring drift-repair #639 closed 2026-04-29 in current shape; conflict against Backfill library.artwork_url for Discogs-resolvable rows #637 remains and is the active sequencing call.

Decision (unchanged): defer #532 until #631's drain (#641) plateaus, then re-evaluate. Sequencing notes are on #637.

LML refactor cluster

• Refactor: extract LML client into a shared workspace package #608 (extract LML client to shared workspace), Refactor: extract shared batched-UPDATE backfill skeleton #613 (batched-UPDATE skeleton), Refactor: collapse 11 near-identical job Dockerfiles into a parameterized one #614 (Dockerfile collapse — body says 11 files, current count is 13 in repo root / 10 once Dockerfile.{backend,auth,migrate} are excluded), Refactor: move flowsheet-etl/logger.ts into @wxyc/database for reuse #615 (logger relocation) all carve shared/ workspaces out of jobs/. Could batch as a focused refactor week.
• [Parked] Refactor: extract shared id-cursor backfill orchestrator #612 ([Parked]) and Factor backfill-runtime shared abstraction across one-shot/recurring jobs #651 overlap on backfill-runtime extraction. Factor backfill-runtime shared abstraction across one-shot/recurring jobs #651's 2026-04-29 re-evaluation comment recommends narrowing to three small extractions (shared/job-utils/{partition,logger,lml-fetch}.ts) rather than a new workspace. Still pending fold/close decision.
• Sweep: parameterize schema name in remaining hardcoded wxyc_schema raw SQL sites #681 is the mechanical sweep of the eight remaining hardcoded wxyc_schema raw-SQL sites — orthogonal to the workspace/extraction debate; can land standalone.

Health-probe diagnosability

• Epic [Epic] Operational diagnosability of /health probes (cross-repo) #665's cross-repo prerequisite (LML#226) closed. BS-side child Surface specific DB failure cause from /healthcheck probe #662 (typed /healthcheck cause vocabulary) is now unblocked; can proceed independently.

Per-show play_order semantics (resolved 2026-05-02)

• nextPlayOrder() should scope to current show_id, not global max #693 fix (f71b3ef) converted inserts to per-show but not mutations, leaving migrations.spec.js shape-fixture test broken on main, blocking every PR's CI #712 visible (changeOrder bumped play_orders across all shows). Resolved by PR Scope flowsheet changeOrder play_order renumber to entry's show (closes #712) #713 (closes migrations.spec.js shape-fixture test broken on main, blocking every PR's CI #712). Read-side analogue (getEntriesByRange ordering by desc(play_order) across id-filtered rows that span shows) closed 2026-05-09 via getEntriesByRange orderBy(desc(play_order)) is meaningless across shows post-#693 #714 → PR fix(flowsheet): order id-range fetch by id, not per-show play_order (#714) #786.

Cross-repo dependencies (native + body prose)

BS issue	Blocked by	Status
#528	tubafrenzy#489, #490, #493, #495; docs#11	All open
#644	#628	Open (sibling fix)
#250, #251	#249	Open (graph projection chain)
#641	#640	(#649 closed via PR #683 2026-05-02; only the pilot remains)
#642	#641, #640	(#638, #639 closed; #649 closed)
#532	#641	Open (rollout plateau is the signal)
#637	#572 (soft)	#572 superseded by #663
#655	tubafrenzy#496, docs#13, docs#16	All open
#665	library-metadata-lookup#226	#226 closed — BS-side #662 is unblocked

Native blockedBy relationships are visible in each issue's Relationships pill and queryable via GraphQL blockedBy.

Cross-repo runtime risks

Not strict blockers but cap realized hit rate / stability of the LML metadata backfill (recorded in #631 / #637 / #638 bodies):

• library-metadata-lookup#197 — library.db literal 'NULL' strings in label column defeat Discogs strict search on a large fraction of rows.
• library-metadata-lookup#200 — /lookup doesn't enrich direct library hits with Discogs artwork. Affects Backfill library.artwork_url for Discogs-resolvable rows #637's observed hit rate.
• library-metadata-lookup#207 — wires up MusicBrainz/Wikidata/Spotify identity sources. When this lands the no-match floor drops.
• library-metadata-lookup#210, Replace addEntry/updateEntry SQL mirrors with HTTP calls #237 — trio-artist lookup misses (validation-side shipped via Replace addEntry/updateEntry SQL mirrors with HTTP calls to tubafrenzy #236).
• library-metadata-lookup#244 — pin mypy in CI. Per WXYC/wiki#5 P0, blocks every LML deploy chain after the 2026-05-01 FD-leak fixes.
• library-metadata-lookup#229 — cache_stats projection onto LML's transaction. Closed 2026-05-02 (paired with BS PR Wrap lookupMetadata in Sentry span and project cache_stats #669, also merged); E2E observability path is now wired.

Cluster map

• LML metadata backfill (Epic [Epic] Historical metadata backfill for ~1.86M flowsheet rows #631): 631, 640, 641, 642, 644, 628, 650, 651, 661, 692.
• SSE re-emit race (Epic [Epic] Fix SSE re-emit race after async UPDATEs #654): 654, 628, 644.
• LML cache_stats observability (Epic [Epic] LML cache_stats E2E observability (LML → shared types → BS Sentry spans) #656): closed 2026-05-03 — fully shipped via BS#646 + LML#213/Re-evaluate Elasticsearch for library catalog if tsvector proves insufficient #229. wxyc-shared#86 (typed CacheStats schema) is the leftover hygiene item; not blocking.
• Operational diagnosability of /health probes (Epic [Epic] Operational diagnosability of /health probes (cross-repo) #665): 665, 662; cross-repo prereq library-metadata-lookup#226 closed.
• Cross-cache-identity epics (E2/E3/E4): 663, 664, 666, 667, 668, 521, 624, 572 (subsumed by [Epic] E2 — Canonical identity record (library_identity) + LML write contract #663).
• Catalog search: 233, 624 (subsumed), 637, 670 (transferred from dj-site#475).
• ETL / data integrity: 516, 521, 525, 528, 605.
• Refactor / chore: 608, 610, 611, 612, 613, 614, 615, 651, 661, 681.
• Graph (BS serving projection): 249, 250, 251 (blocked on WXYC/semantic-index#13).
• Auth / permissions / accounts: 275, 393, 671 (transferred from dj-site#178).
• Stability / security: 75, 99 (PostHog metrics, labeled stale).
• Operational / incidents: 699 (duplicate breakpoints — paired with dj-site#479).
• CI / chore: 133, 207.
• Search backend / bugs: 233.
• Mojibake propagation: 528 (M2.4, blocked on tuba M1 + docs#11), 655 (M3.3, blocked on tuba#496 + docs#13/Updated env variable to be more accurate and descriptive #16).

Labels in use for cross-repo touch

lml (≈12), tubafrenzy (≈12), semantic-index (3), linkage, catalog-search-perf, mojibake, epic, Blocked!, flowsheet-metadata-backfill, cross-cache-identity, epic:e2-bs/epic:e2-lml/epic:e3-normalization/epic:e4-instrumentation. One-word repo-name labels (lml, tubafrenzy) are the convention.

How to find work

• All blocked issues carry the Blocked! label and a "## Blocked by" body section.
• Native blockedBy relationships render under the issue's Relationships pill.
• Per-issue triage commentary is on each issue's page (look for "Triage finding" / "Status update" / "Sequencing concern" / "Updated finding" comments).
• The flowsheet-metadata-backfill label groups the [Epic] Historical metadata backfill for ~1.86M flowsheet rows #631 cluster (including pre-flight finding Factor fetchTopArtwork helper to consolidate LML lookup-and-pick-first pattern #650).
• Closures since the last full triage cycle (drained from this tracker on 2026-05-02): Fix critical memory leaks causing EC2 instance crashes #319, Refactor: audit linkage threshold constants drift (forward path ↔ library-canonical-entity-backfill) #609, Reconcile monitor: switch tubafrenzy direction to SSE #622, Boot-time assertion: fail loud when library.artist_name has any NULL rows #625, Tubafrenzy webhook upserts skip LML metadata enrichment #626, Document SQL-direct bridges for B-2.2 flowsheet linkage (Discogs bridge + trigram fuzzy) #629, album_plays materialized-view refresh hits DB_STATEMENT_TIMEOUT_MS #632, Identify SIGTERM source on EC2 host before next long-running backfill #633, Remove the LML-driven flowsheet-lml-link-backfill job #635, [A.1.a] flowsheet-metadata-backfill historical drain job #638, [A.1.b] Add metadata_attempt_at marker and flip backfill to recurring drift-repair #639, Wrap lookupMetadata in a Sentry span and project LML cache_stats onto it #646, Backfill library.artwork_url via LML (precondition for #643) #647, Replication scripts leave orphaned slots on RDS #648, Strip Discogs spacer.gif placeholders before persisting artwork_url #649, [Epic] LML cache_stats E2E observability (LML → shared types → BS Sentry spans) #656 (epic), Partial index on flowsheet.metadata_attempt_at IS NULL for #638 / #639 Phase 2 sweep #659, Reverse-direction reconciliation: refresh local MySQL clone on a schedule #672, Backend-service Sentry receives zero events — silent for ≥30 days (errors + transactions + spans) #678, Audit filterSpacerGif scope: do non-artwork URL fields carry Discogs spacer.gif placeholders? #679, Parameterize schema name in library-canonical-entity-backfill raw SQL #680, Honor SENTRY_TRACES_SAMPLE_RATE env in flowsheet job loggers #682, Catalog search 503 cached on a single NULL row — backend restart cycle traps DJs #685, Flowsheet POST 500: nextPlayOrder() seq-scans 2.6M rows, exceeds 5s statement timeout #687, Rotation dropdown drops 147 active rows on JOIN, surfaces 35 albums as duplicates #689, Filter LmlClientError from Sentry's express auto-capture #690, nextPlayOrder() should scope to current show_id, not global max #693, Rotation table has duplicate active rows for the same album+bucket (Little Brother x9 in H) #694, Seed integration tests with realistic data shape, not just empty DB #701, Annotate sync-target tables in schema.ts with SOURCE comments + ESLint rule for constraint changes #702, PR-review bot: post data-shape report when PRs add constraints #703, Emit CloudWatch metric for mutation 4xx (replacement signal post-#691 Sentry filter) #704, Constraint-adding migrations should include precondition guards #705, migrations.spec.js shape-fixture test broken on main, blocking every PR's CI #712, getEntriesByRange orderBy(desc(play_order)) is meaningless across shows post-#693 #714, Wrap lookupMetadata in Sentry.startSpan inside flowsheet-metadata-backfill job #715, provision-dryrun-aws: iamPolicy aborts on policy drift instead of upgrading via create-policy-version #769. Plus PRs Strip Discogs spacer.gif placeholders at metadata chokepoints #683, Wrap lookupMetadata in Sentry span and project cache_stats #669, Scope flowsheet changeOrder play_order renumber to entry's show (closes #712) #713 merged 2026-05-02; PRs ci(provisioner): upgrade IAM policy on drift instead of aborting (#769) #785/fix(flowsheet): order id-range fetch by id, not per-show play_order (#714) #786/feat(metadata-backfill): wrap lookupMetadata in Sentry.startSpan (#715) #787 (closing provision-dryrun-aws: iamPolicy aborts on policy drift instead of upgrading via create-policy-version #769/getEntriesByRange orderBy(desc(play_order)) is meaningless across shows post-#693 #714/Wrap lookupMetadata in Sentry.startSpan inside flowsheet-metadata-backfill job #715), 5 Dependabot PRs (Bump the dev-dependencies group across 1 directory with 8 updates #771, Bump the production-dependencies group across 1 directory with 9 updates #724, Bump actions/setup-node from 4 to 6 #723, Bump actions/github-script from 8 to 9 #598, Bump actions/checkout from 4 to 6 #597), and stale-draft sweep (closed PRs Add library code lookup, reviews API, and album update #212, Add Gemini scanner service for vinyl record metadata extraction #213, Add batch scan processing endpoints #215, Add GET /library/scan/batch endpoint to list batch jobs #217) merged/closed 2026-05-09.

[jakebromberg]

Filed #678 — backend-service Sentry has been silent (zero events / transactions / spans) for ≥30 days, surfaced while cross-validating the post-#658 metadata-stamp gap. Listed as a blocker of #640 and #646 via the Relationships API. Likely operational env-var drift on the EC2 host (SENTRY_DSN missing or stale); code path under apps/backend/instrument.ts looks correct since #244.

[jakebromberg]

Status update (2026-05-02): an org-wide tracker now sits above this one at WXYC/wiki#5. It covers cross-repo patterns and dependencies that don't belong solely in this repo (decoder-drift class, CI tool-pinning class, Pipeline Hardening Project #19 spine, mojibake/canonicalization across 6 repos, Sentry rollout, storage-refactor sequencing, ancient-issue sweep). #645 stays the authoritative tracker for Backend-Service-local sequencing.

Concrete deltas vs this tracker that should refresh #645's body when next edited:

• 2026-05-01 incident issues (nextPlayOrder() should scope to current show_id, not global max #693, Rotation table has duplicate active rows for the same album+bucket (Little Brother x9 in H) #694, Flowsheet POST 500: nextPlayOrder() seq-scans 2.6M rows, exceeds 5s statement timeout #687, Rotation dropdown drops 147 active rows on JOIN, surfaces 35 albums as duplicates #689, Catalog search 503 cached on a single NULL row — backend restart cycle traps DJs #685) all closed AND merged to main: commits f71b3ef (nextPlayOrder() should scope to current show_id, not global max #693), 0b7f4d2 (Rotation table has duplicate active rows for the same album+bucket (Little Brother x9 in H) #694, Rotation dropdown drops 147 active rows on JOIN, surfaces 35 albums as duplicates #689), 988110e (Flowsheet POST 500: nextPlayOrder() seq-scans 2.6M rows, exceeds 5s statement timeout #687), 2f96d64 (Catalog search 503 cached on a single NULL row — backend restart cycle traps DJs #685). These are no longer load-bearing for P0.
• Filter LmlClientError from Sentry express auto-capture #691 (Sentry filter) closed and merged 2026-05-02; side effect is that 4xx is now hidden from Sentry — debug 4xx via the Sentry trace explorer / spans, not via the issues feed. This pairs with Emit CloudWatch metric for mutation 4xx (replacement signal post-#691 Sentry filter) #704 (CloudWatch fallback metric) which also closed.
• The "fire 1.86M-row backfill at scale" path still depends on the P0 list here (Production flowsheet table is catastrophically bloated (1.4 GB heap, 24 live rows) #525, Fix critical memory leaks causing EC2 instance crashes #319, npm audit — 18 vulnerabilities (2 critical, 8 high) as of 2026-04-14 #75, Run LML entity-resolution pipeline to populate entity.identity in prod #572, Backfill library.artwork_url for Discogs-resolvable rows #637); none of those have moved.
• Cross-repo deps newly worth encoding: [A.2] Pilot run for flowsheet-metadata-backfill #640 ← library-metadata-lookup#213 (cache_stats trace must merge first); Duplicate hour-boundary breakpoints: dj-site manual breakpoint + tubafrenzy auto-breakpoint #699 ↔ dj-site#479 (paired product decision).