feat: pluggable storage backends (pg/redis/mongodb) — v0.3.0#1
Open
robertziel wants to merge 1 commit into
Open
feat: pluggable storage backends (pg/redis/mongodb) — v0.3.0#1robertziel wants to merge 1 commit into
robertziel wants to merge 1 commit into
Conversation
Make the durable-queue store selectable via configure(db_backend="pg"|"redis"|"mongodb"). Additive + opt-in: pg (default) is byte-compatible and the legacy Postgres engine is untouched; selecting redis/mongo does NOT re-home the orchestrator/worker (a later milestone). - backends/base.py: StorageBackend port — enqueue / claim-exactly-once / lease+reclaim / idempotent terminals / atomic outbox (complete_with_event/fail_with_event) / wake / heartbeat / ON-OFF control. No driver object in any signature (anti-leakage); each backend is namespace-bound (multi-tenant isolation). - backends/postgres.py: reference adapter — FOR UPDATE SKIP LOCKED, RETURNING CAS, one-transaction outbox, LISTEN/NOTIFY, on its own qw_* tables. - backends/redis.py: Lua-atomic claim/terminal/outbox, sorted-set priority+FIFO, pub/sub wake (single instance, not Cluster). - backends/mongodb.py: find_one_and_update claim, multi-doc-transaction outbox, change-stream wake (replica set required). - tests/test_backend_contract.py: one parametrized contract suite, green against all three live servers (incl. an 8-thread claim-once contention stress test); a backend whose server is unreachable is skipped. - config: db_backend + db_namespace + redis/mongo URL envs; [redis]/[mongodb] optional extras (drivers import lazily — pg-only deploys need neither). - docs/storage_backends.md: contract, capability matrix + caveats, audit, and the integration boundary. CHANGELOG 0.3.0; CLAUDE.md/AGENTS.md/README updated. Audit (design/regressions/safety/leakage): fixed Redis counts() drift (decrement prior status, not always running); added the concurrency test; no engine-suite regressions (415 green). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a host-agnostic
StorageBackendSPI so the durable-queue store is selectable:One provider per file (
queue_workflows/backends/), following the codebase's "injected hook + safe default" philosophy. Additive and opt-in:pg(default) is byte-compatible and the legacy Postgres engine is untouched.The contract (identical across backends)
enqueue · claim exactly-once · lease/renew/reclaim · idempotent terminals · atomic outbox (
complete_with_event/fail_with_event— both-or-neither) · wake · heartbeat · operator ON/OFF · per-namespace tenant isolation.pgredismongodbFOR UPDATE SKIP LOCKEDZPOPMINin Luafind_one_and_updateLISTEN/pg_notifynamespacecolumnTests (TDD)
tests/test_backend_contract.py— one parametrized suite green against all three live servers (a backend whose server is unreachable is skipped), incl. an 8-thread claim-once contention stress test.Audit (design / regressions / safety / internal data leakage)
counts()could drift; the Lua now decrements the job's prior status (pg/mongo derive counts from live status, so they're immune).Integration boundary (honest scope)
Selecting
redis/mongodbdoes not re-home the orchestrator / claim-worker / dispatcher — those still run on Postgres. v0.3.0 ships the SPI + three contract-complete backends; wiring the DAG orchestrator end-to-end onto a non-PG backend is a later milestone. Caveats (Redis = single instance not Cluster; Mongo = replica set required) are indocs/storage_backends.md.Release
Version bumped to 0.3.0 + CHANGELOG cut (additive ⇒ SemVer minor). Recommend cutting the
v0.3.0git tag after merge (a tag on this branch would be orphaned by a squash-merge).🤖 Generated with Claude Code