Skip to content

feat: auto-derive idempotency keys on @queue.task#143

Merged
pratyush618 merged 3 commits into
masterfrom
feat/idempotent-tasks
May 8, 2026
Merged

feat: auto-derive idempotency keys on @queue.task#143
pratyush618 merged 3 commits into
masterfrom
feat/idempotent-tasks

Conversation

@pratyush618
Copy link
Copy Markdown
Collaborator

Summary

Turns the existing unique_key deduplication infrastructure (already present
end-to-end in SQLite, Postgres, and Redis with partial unique indexes / Lua
scripts) into a first-class, opt-in task feature. No Rust changes, no schema
migration, no new Storage methods — pure-Python wiring.

@queue.task(idempotent=True)
def charge(customer_id: int, amount: int) -> str: ...

charge.delay(42, 1000)
charge.delay(42, 1000)   # → same job ID, no duplicate execution
  • Auto-key format: auto:{sha256(task_name|serialized_payload)[:32]}. Hashed
    after the task's serializer runs, so the wire-payload contract is what
    decides equivalence.
  • Per-call idempotency_key="…" overrides the auto-derived key.
  • Per-call idempotent=False disables auto-derivation for one submission even
    on a task registered with idempotent=True.
  • Existing unique_key parameter kept as a documented alias for back-compat.
  • Mirrored on enqueue_many() via idempotency_keys: list[str | None] and a
    uniform idempotent flag.
  • on_enqueue middleware sees and can mutate the new keys.

Slot release semantics are unchanged — the dedup window is held only while the
job is pending or running; once it terminates, a fresh call with the same
arguments creates a new job. This is enforced by the existing partial unique
index, not new code.

Files

  • py_src/taskito/app.py_task_idempotent map, _resolve_unique_key()
    helper, new kwargs on enqueue() / enqueue_many().
  • py_src/taskito/mixins/decorators.pyidempotent: bool = False on
    @queue.task().
  • py_src/taskito/task.pyTaskWrapper.apply_async() plumbs the new kwargs.
  • tests/python/test_idempotent.py — 8 new tests.
  • docs/content/docs/guides/reliability/idempotency.mdx + meta.json — guide.

Test plan

  • uv run python -m pytest tests/python/test_idempotent.py tests/python/test_unique.py -v (12 passed)
  • uv run python -m pytest tests/python/ -x (506 passed, 9 skipped)
  • uv run ruff check py_src/ tests/python/test_idempotent.py
  • uv run mypy py_src/taskito/ --no-incremental (clean across 121 files)
  • cargo check --workspace (clean)
  • CI green across SQLite / Postgres / Redis Rust jobs (no Rust touched, but worth confirming)

@pratyush618 pratyush618 merged commit f108a62 into master May 8, 2026
21 checks passed
@pratyush618 pratyush618 deleted the feat/idempotent-tasks branch May 8, 2026 00:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant