Add staged background ingest promotion

CHANGELOG.md

@@ -4,6 +4,7 @@ All notable changes to RecallForge will be documented in this file.
 
 ## [Unreleased]
 
+- Added staged background reindex promotion so document, video, audio, and conversation replacements stay hidden until their parent/child memory batches are complete.
 - Added deterministic memory graph enrichment with entity/relation side tables and new `memory_graph_entities` / `memory_graph_related` MCP tools.
 - Replaced the tiny UAT video clips with compact episodic-memory fixtures, richer transcript sidecars, related artifact metadata, and regression coverage for the video corpus.
 - Added `memory_add_conversation` so conversation threads ingest as canonical parent memories with turn-level child memories and standard memory rollups.

README.md

@@ -161,7 +161,7 @@ See [docs/mcp-tools.md](docs/mcp-tools.md) for the full tool reference.
 
 ## How it works
 
-RecallForge encodes text, images, video frames, documents, conversation turns, and audio transcripts into the same 2048-dimensional vector space using Qwen3-VL. It also extracts lightweight entity and relation metadata so agents can navigate from one memory to other memories that mention the same people, projects, tickets, URLs, and organizations. This means "find notes about this diagram" works whether the diagram is text, an image, a conversation thread, or a frame from a video. A 3-stage pipeline handles the rest:
+RecallForge encodes text, images, video frames, documents, conversation turns, and audio transcripts into the same 2048-dimensional vector space using Qwen3-VL. It also extracts lightweight entity and relation metadata so agents can navigate from one memory to other memories that mention the same people, projects, tickets, URLs, and organizations. Reindexes for documents, video, audio, and conversations are staged as hidden batches first, then promoted together so agents keep seeing the previous complete memory until the replacement is ready. This means "find notes about this diagram" works whether the diagram is text, an image, a conversation thread, or a frame from a video. A 3-stage pipeline handles the rest:
 
 ```mermaid
 graph TD

docs/ARCHITECTURE.md

@@ -106,24 +106,27 @@ embedded_at    | vector[2048]
 user/session/project/profile namespace fields
 memory_id      | memory_role | memory_root_path
 importance     | ttl_seconds | tags       | expires_at
+active          | index_batch_id
 ```
 
 **documents** (registry)
 ```
 id | collection | file_path | title | content_hash | content_type | active
-created_at | updated_at
+created_at | updated_at | index_batch_id
 ```
 
 **entities** (memory graph mentions)
 ```
 id | collection | entity_key | name | entity_type | memory_id | memory_root_path
 file_path | content_hash | hash_seq | seq | evidence | namespace fields | created_at
+active | index_batch_id
 ```
 
 **relations** (lightweight graph edges)
 ```
 id | collection | subject_key | subject_name | object_key | object_name | relation_type
 memory_id | memory_root_path | file_path | content_hash | hash_seq | evidence | namespace fields
+active | index_batch_id
 ```
 
 Conversation memories use the same parent/child layout as media-derived memories:
@@ -226,6 +229,8 @@ Key runtime details:
     │   ├── data/*.parquet
     │   └── _indices/text_body_fts/    # Tantivy FTS
     ├── documents/
+    ├── entities/
+    ├── relations/
     ├── content/
     └── cache/
 ```
@@ -252,6 +257,18 @@ path + text
          ensure_fts_index()  ──> Tantivy index rebuild
 ```
 
+Complex file and conversation reindexes use a staging/promotion variant of this flow:
+
+```
+new root + children
+     │
+     ├──> write hidden rows (`active=0`, `index_batch_id=batch_*`)
+     ├──> keep old active rows visible to search/list/get/graph reads
+     └──> promote batch: activate new rows and deactivate old rows for the memory path
+```
+
+Failed staged ingests delete only the hidden batch, so readers continue to see the last complete memory.
+
 ## Data Flow: Audio
 
 Audio is transcript-first in this release. The storage layer accepts common audio extensions only when a sibling `.srt`, `.vtt`, `.txt`, or `.transcript.json` sidecar exists.

docs/MEMORY_POLICY.md

@@ -27,10 +27,17 @@ RecallForge stores complex files as a root memory plus child assets:
 
 The root memory uses `memory_role="root"` and child assets use `memory_role="child"` with `memory_root_path` pointing back to the root.
 
+## Ingest Consistency
+
+Complex reindexes are staged before they become visible. Documents, video, audio, and conversation memories write replacement parent/child rows with `active=0` and a private `index_batch_id`; search, memory listing, memory lookup, and graph navigation only read active rows. Once the replacement parent and all child assets are stored, RecallForge promotes the batch in one visibility step and deactivates the old rows for that memory path.
+
+This keeps agents from seeing half-updated memories during background ingest. If a staged ingest fails before promotion, RecallForge deletes the hidden batch and the previous active memory remains readable.
+
 ## Ranking
 
 RecallForge combines BM25 and vector search through RRF, then reranks in `hybrid` mode. Before fusion, storage search applies memory policy:
 
+- Hidden staging rows are filtered out until promotion.
 - Expired rows are filtered out with `expires_at`.
 - `importance` can add up to a 15% boost before score normalization.
 - Fresh rows receive a small recency boost.

docs/mcp-tools.md

@@ -471,7 +471,7 @@ Example MCP client config (Claude Desktop):
 - `INVALID_INPUT`: returned by storage/validation for invalid combinations/inputs.
 - `INTERNAL_ERROR`: uncaught exceptions.
 
-**Notes:** This is the recommended ingest entry point. `set_config` can change default `collection`/`max_file_size_mb` used when omitted.
+**Notes:** This is the recommended ingest entry point. `set_config` can change default `collection`/`max_file_size_mb` used when omitted. Reindexing complex memories (documents, video, audio) stages replacement rows hidden from reads, then promotes the parent and child assets together when ingest succeeds.
 
 ---
 
@@ -710,7 +710,7 @@ Turn objects accept:
 - `BACKEND_ERROR`: when the storage backend does not support conversation memories.
 - `INTERNAL_ERROR`: uncaught exceptions.
 
-**Notes:** The root path becomes the stable `memory_id` identity seed. Child turns are stored at `path::turn:0001`, `path::turn:0002`, and so on with the same `memory_id` and `memory_root_path`.
+**Notes:** The root path becomes the stable `memory_id` identity seed. Child turns are stored at `path::turn:0001`, `path::turn:0002`, and so on with the same `memory_id` and `memory_root_path`. Replacing a conversation stages the new root and turn children first, so clients keep seeing the previous complete thread until the replacement is promoted.
 
 ---
 

docs/research/recallforge-memory-mcp-roadmap.md

@@ -67,9 +67,7 @@ Why it comes before advanced ranking:
 Goal:
 - Make RecallForge feel like memory retrieval, not generic search.
 
-Current Linear fit:
-- `REC-84`
-- `REC-83`
+Shipped Linear work:
 - `REC-75`
 - `REC-78`
 
@@ -87,7 +85,7 @@ Why it comes here:
 Goal:
 - Push more intelligence into ingest-time structure instead of query-time cost.
 
-Current Linear fit:
+Shipped Linear work:
 - `REC-76`
 
 Likely follow-ons:

src/recallforge/storage/base.py

@@ -115,6 +115,8 @@ def insert_document(
         memory_id: Optional[str] = None,
         memory_role: str = "root",
         memory_root_path: Optional[str] = None,
+        active: int = 1,
+        index_batch_id: Optional[str] = None,
     ) -> str:
         """
         Insert or update a document.
@@ -175,6 +177,8 @@ def insert_embedding(
         importance: Optional[float] = None,
         ttl_seconds: Optional[int] = None,
         tags: Optional[List[str]] = None,
+        active: int = 1,
+        index_batch_id: Optional[str] = None,
     ) -> None:
         """Insert an embedding for a document chunk."""
         pass
@@ -273,6 +277,8 @@ def upsert_memory(
         _skip_delete: bool = False,
         memory_role: str = "root",
         memory_root_path: Optional[str] = None,
+        _active: int = 1,
+        _index_batch_id: Optional[str] = None,
     ) -> str:
         """Create or update a memory entry and its embeddings."""
         pass