Skip to content

fix(probe): deactivate dead Caltrans URLs instead of aborting#15

Open
NewCoder3294 wants to merge 2 commits into
mainfrom
prod/probe-allow-rot
Open

fix(probe): deactivate dead Caltrans URLs instead of aborting#15
NewCoder3294 wants to merge 2 commits into
mainfrom
prod/probe-allow-rot

Conversation

@NewCoder3294
Copy link
Copy Markdown
Owner

Summary

Raise `probeCameraLiveness` `abortThreshold` from 0.5 to 0.95. Caltrans D4's CDN routinely 404s 60–80% of advertised URLs; the old threshold treated that as a transient outage and skipped every write, so dead URLs accumulated and `/wall` went empty.

Root cause (from /investigate report continuation)

  • `syncCameras` already refreshes `stream_url` daily from Caltrans (good).
  • `probeCameraLiveness` runs right after to mark dead URLs `is_active=false`.
  • But `abortThreshold: 0.5` aborts ALL writes when >half of probed cameras fail. Sample probe showed 4 of 5 D4 URLs dead (80%) — well past threshold. Aborts every run, nothing gets deactivated.

Even `packages/sync/src/sync.ts`'s comment concedes: "many streams CalTrans reports as in-service are 404 at the CDN."

Test plan

  • Manually trigger `/api/cron/sync-cameras` post-deploy (or wait for 9 AM UTC daily run).
  • Confirm response shows `probe.aborted: false` and a non-zero `probe.dead` count.
  • Reload `/wall` — should show live tiles instead of all-offline.

Follow-up (not in this PR)

Per-camera consecutive-failure tracking (`failed_probe_count` column) — deactivate after N=2 failed probes instead of relying on a global ratio. Better resilience to genuine 1-day transient outages.

🤖 Generated with Claude Code

NewCoder3294 and others added 2 commits May 19, 2026 15:29
- Remove Supabase auth middleware and (app) layout gate; the dashboard
  is now reachable without sign-in.
- Delete /login route, (auth) layout, and /auth/callback handler.
- Replace the business-owner landing's "HLS feed preview" placeholder
  with a real Caltrans D4 stream rendered via the existing LiveStream
  component (proxied through /api/hls). Falls back to an offline
  overlay if the upstream feed is unavailable.

Supabase client libs are kept — contributor flows still read from them.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Caltrans D4's CDN routinely 404s 60–80% of the stream URLs it advertises
in cctvStatusD04.json. The 0.5 abortThreshold treated that as a transient
outage and skipped every write, so dead URLs accumulated indefinitely and
/wall went empty in prod.

0.95 still catches a near-total worker egress failure (every probe times
out) but lets the daily sync deactivate the actual rot. Cameras that come
back online are still revived on the next run via the existing toRevive
path, so the change is reversible per-camera.

Recommended follow-up: per-camera consecutive-failure tracking instead
of a global ratio. Out of scope here.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@vercel
Copy link
Copy Markdown

vercel Bot commented May 19, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
watchdog Ready Ready Preview, Comment May 19, 2026 11:06pm

Request Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant