Auto-import jobs lost after container restart (RQ depends_on + in-memory Redis)

[flowcool]

Summary

When using autotag: "auto" in inbox config, albums get stuck in PREVIEWED status forever after a container restart. The dependent import job created via RQ's depends_on is lost because Redis runs in-memory inside the container.

Environment

• beets-flask: latest (Docker)
• RQ version: 2.6.1
• Redis: embedded in container (no persistence)
• Config: autotag: "auto", auto_threshold: null

Steps to reproduce

1. Configure an inbox folder with autotag: "auto"
2. Let beets-flask process albums — they go through the full chain: PENDING → PREVIEWING → PREVIEWED → IMPORTING → IMPORTED ✓
3. Restart the container (docker restart beets-flask)
4. New albums arrive in the inbox
5. Albums that were mid-processing get stuck at PREVIEWED forever

Root cause analysis

enqueue_import_auto() in enqueue.py creates a two-step RQ job chain:

job1 = preview_queue.enqueue(run_preview, ...)  # step 1
job2 = _enqueue(import_queue, run_import_auto, ..., depends_on=job1)  # step 2

• job1 (preview) runs on the preview queue
• job2 (import) is placed in RQ's deferred registry, waiting for job1 to finish
• When job1 completes, RQ moves job2 to the import queue

The problem: Redis runs in-memory with no persistence. On container restart, Redis is wiped clean. All deferred import jobs are lost. Preview jobs that were already running complete and set the session status to PREVIEWED, but their dependent import jobs no longer exist in Redis.

Evidence

After a container restart, we observed:

Metric	Value
Import worker (PID 66)	Alive, registered in Redis, idle since birth
successful_job_count on import worker	0 (never processed a single job)
rq:finished:import	does not exist (no import ever completed)
DeferredJobRegistry(import)	0 jobs
Albums stuck at PREVIEWED	76 (including albums with 85–93% match scores — well above threshold)
Albums successfully IMPORTED (before restart)	684

The watchdog compounds the problem. When it detects folders that already have a session in the database, auto_tag() in watchdog/inbox.py skips re-enqueueing:

state = SessionStateInDb.get_by_hash_and_path(hash=None, path=folder.full_path)
should_enqueue = False
if state is None:
    should_enqueue = True
else:
    if enq_kind == invoker.EnqueueKind.PREVIEW and folder.hash != state.folder_hash:
        should_enqueue = True

Since the session exists in the database (persistent), the watchdog sees it and skips. But the corresponding RQ job is gone from Redis (volatile). There is no recovery path — the album stays PREVIEWED forever.

Manual verification

We confirmed the mechanism works correctly when triggered manually via the API:

# Before: 76 PREVIEWED, 684 IMPORTED, import worker: 0 successful jobs

POST /api_v1/session/enqueue
{ "folder_hashes": ["..."], "folder_paths": ["..."], "kind": "IMPORT_AUTO" }
# → "1 added as kind: IMPORT_AUTO"
# → Preview runs → import job created via depends_on → import executes

# After: 75 PREVIEWED, 685 IMPORTED, import worker: 1 successful job ✓

Suggested areas to investigate

1. Redis persistence: Could Redis be configured with AOF or RDB snapshots to survive restarts?
2. Startup recovery: On startup, scan for sessions in PREVIEWED/PENDING status and re-enqueue them as IMPORT_AUTO
3. Watchdog should_enqueue logic: Consider allowing re-enqueue when the session status is PREVIEWED (import step not completed) rather than only when the folder hash changed