Add memory graph enrichment

CHANGELOG.md

@@ -4,6 +4,7 @@ All notable changes to RecallForge will be documented in this file.
 
 ## [Unreleased]
 
+- Added deterministic memory graph enrichment with entity/relation side tables and new `memory_graph_entities` / `memory_graph_related` MCP tools.
 - Replaced the tiny UAT video clips with compact episodic-memory fixtures, richer transcript sidecars, related artifact metadata, and regression coverage for the video corpus.
 - Added `memory_add_conversation` so conversation threads ingest as canonical parent memories with turn-level child memories and standard memory rollups.
 

README.md

@@ -33,7 +33,7 @@ One query. Any modality. All local.
 | Audio transcript ingest | ✅ | ❌ | ❌ | ❌ | ❌ |
 | Document ingest (PDF/DOCX/PPTX) | ✅ | ❌ | ❌ | ❌ | ❌ |
 | Built-in reranking | ✅ Multimodal | ❌ | ❌ | ✅ ColBERT | ✅ Modules |
-| MCP-native | ✅ 24 tools | ❌ | ❌ | ❌ | ❌ |
+| MCP-native | ✅ 26 tools | ❌ | ❌ | ❌ | ❌ |
 | 100% local | ✅ | ✅ | ⚠️ Cloud default | ✅ | ✅ Docker |
 | Apple Silicon optimized | ✅ MLX 4-bit | ❌ | ❌ | ❌ | ❌ |
 | Cloud option | ❌ | ✅ | ✅ | ✅ | ✅ |
@@ -146,7 +146,7 @@ Run over HTTP/SSE:
 recallforge serve --http --host 127.0.0.1 --port 7433 --mode embed
 ```
 
-RecallForge now exposes **24 MCP tools** across search, ingest, memory, collection admin, and runtime config. HTTP/SSE mode also exposes `/health`, `/sse`, and `/messages/`.
+RecallForge now exposes **26 MCP tools** across search, ingest, memory graph navigation, collection admin, and runtime config. HTTP/SSE mode also exposes `/health`, `/sse`, and `/messages/`.
 
 See [docs/mcp-tools.md](docs/mcp-tools.md) for the full tool reference.
 
@@ -161,7 +161,7 @@ See [docs/mcp-tools.md](docs/mcp-tools.md) for the full tool reference.
 
 ## How it works
 
-RecallForge encodes text, images, video frames, documents, conversation turns, and audio transcripts into the same 2048-dimensional vector space using Qwen3-VL. This means "find notes about this diagram" works whether the diagram is text, an image, a conversation thread, or a frame from a video. A 3-stage pipeline handles the rest:
+RecallForge encodes text, images, video frames, documents, conversation turns, and audio transcripts into the same 2048-dimensional vector space using Qwen3-VL. It also extracts lightweight entity and relation metadata so agents can navigate from one memory to other memories that mention the same people, projects, tickets, URLs, and organizations. This means "find notes about this diagram" works whether the diagram is text, an image, a conversation thread, or a frame from a video. A 3-stage pipeline handles the rest:
 
 ```mermaid
 graph TD
@@ -276,7 +276,7 @@ src/recallforge/
 │   └── lancedb_backend.py # LanceDB + Tantivy FTS
 ├── cache.py              # LRU embedding cache
 ├── search.py             # Hybrid search pipeline (BM25 + vector + RRF)
-├── server.py             # MCP server (24 tools, stdio + HTTP/SSE)
+├── server.py             # MCP server (26 tools, stdio + HTTP/SSE)
 ├── documents.py          # PDF/DOCX/PPTX extraction
 ├── video.py              # Frame/transcript extraction
 ├── audio.py              # Transcript-first audio ingest

docs/ARCHITECTURE.md

@@ -114,6 +114,18 @@ id | collection | file_path | title | content_hash | content_type | active
 created_at | updated_at
 ```
 
+**entities** (memory graph mentions)
+```
+id | collection | entity_key | name | entity_type | memory_id | memory_root_path
+file_path | content_hash | hash_seq | seq | evidence | namespace fields | created_at
+```
+
+**relations** (lightweight graph edges)
+```
+id | collection | subject_key | subject_name | object_key | object_name | relation_type
+memory_id | memory_root_path | file_path | content_hash | hash_seq | evidence | namespace fields
+```
+
 Conversation memories use the same parent/child layout as media-derived memories:
 
 ```
@@ -122,7 +134,7 @@ conversation root path
      └──> turn child memories (`path::turn:0001`, `path::turn:0002`, ...)
 ```
 
-All turn children share the root `memory_id` and `memory_root_path`, so matching turns strengthen the parent conversation result through the standard memory rollup path.
+All turn children share the root `memory_id` and `memory_root_path`, so matching turns strengthen the parent conversation result through the standard memory rollup path. Entity and relation rows keep the same IDs and evidence paths, which lets MCP clients navigate same-entity memories without relying only on lexical overlap.
 
 **content** (bodies, content-addressed)
 ```
@@ -193,7 +205,7 @@ backend = recallforge.get_backend()
 ### 5. MCP Server (`src/recallforge/server.py`)
 
 ```
-Tools: 24 MCP tools across search, ingest, memory, collection admin, batch, and runtime config
+Tools: 26 MCP tools across search, ingest, memory, memory graph, collection admin, batch, and runtime config
 Transport: stdio (default) or HTTP/SSE (`/health`, `/sse`, `/messages/`)
 Startup: backend.warm_up() for predictable latency
 Signals: SIGTERM/SIGINT graceful shutdown

docs/MEMORY_POLICY.md

@@ -47,3 +47,9 @@ Raw audio transcription and dedicated audio encoders are not part of the shipped
 ## Conversations
 
 Use `memory_add_conversation` when an agent or app wants to persist a thread. RecallForge stores the parent at the supplied `path` and stores each turn at `path::turn:0001`, `path::turn:0002`, and so on. All turns share the parent `memory_id`, so if several turns match a query, search and explanation output roll them up into the parent conversation with evidence paths.
+
+## Memory Graph
+
+Every indexed text-bearing evidence unit can also produce lightweight entity mentions and co-mention relation edges. That includes normal text memories, OCR/document sections, transcripts, captions, and conversation turns. Graph rows store `memory_id`, `memory_root_path`, `file_path`, and a short evidence snippet, so same-entity navigation remains traceable to the source memory.
+
+Use `memory_graph_entities` to inspect entities for a memory, path, or entity key. Use `memory_graph_related` to find other memories that share extracted entities with a seed memory or entity. This graph enrichment is intentionally local and deterministic: it adds navigation and grouping without introducing an external NLP service.

docs/mcp-tools.md

@@ -55,6 +55,8 @@ Example MCP client config (Claude Desktop):
 - `memory_update`
 - `memory_delete`
 - `memory_get`
+- `memory_graph_entities`
+- `memory_graph_related`
 - `list_memories`
 
 ### Admin / Introspection
@@ -853,6 +855,124 @@ Turn objects accept:
 
 ---
 
+## memory_graph_entities
+
+**Description:** List entity mentions extracted from indexed memory text, OCR text, transcripts, captions, and conversation turns. Each mention includes the source memory/path and evidence snippet that produced it.
+
+**Parameters:**
+| Name | Type | Required | Default | Description |
+|------|------|----------|---------|-------------|
+| memory_id | string | Conditionally* | — | Stable memory identifier |
+| path | string | Conditionally* | — | Root or child memory path |
+| entity | string | Conditionally* | — | Entity name or normalized entity key |
+| collection | string | No | server default collection | Collection filter |
+| limit | integer | No | 100 | Max entity mentions |
+| user_id | string | No | — | User namespace |
+| session_id | string | No | — | Session namespace |
+| project_id | string | No | — | Project namespace |
+| profile | string | No | — | Profile namespace |
+
+\* Provide at least one of `memory_id`, `path`, or `entity`.
+
+**Example Request:**
+```json
+{
+  "path": "threads/customer-renewal",
+  "collection": "default"
+}
+```
+
+**Example Response:**
+```json
+{
+  "success": true,
+  "count": 2,
+  "entities": [
+    {
+      "entity_key": "acme_robotics",
+      "name": "Acme Robotics",
+      "entity_type": "proper_noun",
+      "memory_id": "b4f7...",
+      "memory_root_path": "threads/customer-renewal",
+      "file_path": "threads/customer-renewal::turn:0002",
+      "evidence": "Acme Robotics asked to review the renewal timeline..."
+    }
+  ]
+}
+```
+
+**Errors:**
+- `INVALID_INPUT`: when no seed (`memory_id`, `path`, or `entity`) is provided.
+- `BACKEND_ERROR`: when the storage backend does not support graph entities.
+- `INTERNAL_ERROR`: uncaught exceptions.
+
+**Notes:** Entity keys are normalized for lookup. Evidence snippets are stored beside the graph rows so related-memory navigation can be traced back to source memories.
+
+---
+
+## memory_graph_related
+
+**Description:** Find memories related by shared extracted entities. This is useful when two memories mention the same person, project, organization, ticket, URL, or topic even if ordinary lexical search would not rank them together.
+
+**Parameters:**
+| Name | Type | Required | Default | Description |
+|------|------|----------|---------|-------------|
+| memory_id | string | Conditionally* | — | Stable memory identifier to use as the seed |
+| path | string | Conditionally* | — | Root or child memory path to use as the seed |
+| entity | string | Conditionally* | — | Entity name or normalized entity key to use as the seed |
+| collection | string | No | server default collection | Collection filter |
+| limit | integer | No | 20 | Max related memories |
+| user_id | string | No | — | User namespace |
+| session_id | string | No | — | Session namespace |
+| project_id | string | No | — | Project namespace |
+| profile | string | No | — | Profile namespace |
+
+\* Provide at least one of `memory_id`, `path`, or `entity`.
+
+**Example Request:**
+```json
+{
+  "entity": "Acme Robotics",
+  "collection": "default",
+  "limit": 5
+}
+```
+
+**Example Response:**
+```json
+{
+  "success": true,
+  "count": 1,
+  "related_memories": [
+    {
+      "memory_id": "af31...",
+      "collection": "default",
+      "path": "notes/acme-budget",
+      "score": 11,
+      "shared_entities": [
+        { "entity_key": "acme_robotics", "name": "Acme Robotics", "entity_type": "proper_noun" }
+      ],
+      "evidence": [
+        {
+          "path": "notes/acme-budget",
+          "entity": "Acme Robotics",
+          "text": "The budget memo says Acme Robotics approved new sensors."
+        }
+      ]
+    }
+  ]
+}
+```
+
+**Errors:**
+- `INVALID_INPUT`: when no seed (`memory_id`, `path`, or `entity`) is provided.
+- `BACKEND_ERROR`: when the storage backend does not support related memory graph lookup.
+- `INTERNAL_ERROR`: uncaught exceptions.
+
+**Notes:** Relatedness is based on shared graph entity keys and includes evidence from the related memories. The score is intentionally simple: more distinct shared entities and mentions rank higher.
+
+---
+
 ## list_memories
 
 **Description:** List canonical root memories for a collection or namespace.
@@ -1185,7 +1305,12 @@ Operation object schema:
 2. Query with `search` or `explain_results`; matching turns roll up to the parent `memory_id`.
 3. Inspect the full memory with `memory_get` when the agent needs turn evidence.
 
-### 4) Configure mode, then ingest
+### 4) Navigate the memory graph
+1. Call `memory_graph_entities` for a `memory_id`, `path`, or entity name to inspect extracted entities and evidence.
+2. Call `memory_graph_related` with the same seed to find memories that share those entities.
+3. Use the returned evidence paths with `memory_get` or `search` when a client needs the full memory context.
+
+### 5) Configure mode, then ingest
 1. Inspect config using `get_config`.
 2. Set desired runtime defaults with `set_config` (for mode, default collection, max file size).
 3. Run `ingest` without repeating shared defaults.

src/recallforge/entities.py

@@ -0,0 +1,157 @@
+"""Lightweight entity and relation extraction for memory graph enrichment."""
+
+from __future__ import annotations
+
+import hashlib
+import re
+from dataclasses import dataclass
+from itertools import combinations
+from typing import Iterable, Optional
+
+
+_MAX_ENTITIES_PER_TEXT = 24
+_MAX_ENTITY_LEN = 80
+_MAX_EVIDENCE_CHARS = 360
+
+_STOP_ENTITIES = {
+    "A", "An", "And", "Are", "As", "At", "By", "For", "From", "In", "Into", "Is",
+    "It", "Of", "On", "Or", "The", "This", "To", "With",
+    "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday",
+    "January", "February", "March", "April", "May", "June", "July", "August",
+    "September", "October", "November", "December",
+}
+_STOP_ENTITY_KEYS = {_entity.lower() for _entity in _STOP_ENTITIES}
+
+_PROPER_NOUN_RE = re.compile(
+    r"\b[A-Z][A-Za-z0-9&._-]{1,}(?:\s+[A-Z][A-Za-z0-9&._-]{1,}){0,4}\b"
+)
+_ACRONYM_RE = re.compile(r"\b[A-Z][A-Z0-9]{1,}(?:-[A-Z0-9]+)*\b")
+_HANDLE_RE = re.compile(r"(?<!\w)@[A-Za-z0-9_.-]{2,}")
+_ISSUE_RE = re.compile(r"\b[A-Z]{2,10}-\d{1,6}\b")
+_URL_RE = re.compile(r"\bhttps?://[^\s)>\]]+")
+_DOMAIN_RE = re.compile(r"\b(?:[A-Za-z0-9-]+\.)+[A-Za-z]{2,}\b")
+
+
+@dataclass(frozen=True)
+class ExtractedEntity:
+    """One normalized entity mention with source evidence."""
+
+    name: str
+    entity_key: str
+    entity_type: str
+    evidence: str
+
+
+@dataclass(frozen=True)
+class ExtractedRelation:
+    """A lightweight relation edge between two entity mentions."""
+
+    subject_key: str
+    subject_name: str
+    object_key: str
+    object_name: str
+    relation_type: str
+    evidence: str
+
+
+def _clean_entity(raw: str) -> str:
+    text = re.sub(r"\s+", " ", str(raw or "").strip())
+    return text.strip(".,;:!?()[]{}\"'`")
+
+
+def normalize_entity_key(name: str) -> str:
+    """Normalize an entity mention to a stable lookup key."""
+    lowered = name.lower()
+    lowered = re.sub(r"^@", "", lowered)
+    lowered = re.sub(r"[^a-z0-9]+", "_", lowered).strip("_")
+    return lowered
+
+
+def _classify_entity(name: str) -> str:
+    if name.startswith("@"):
+        return "person"
+    if _ISSUE_RE.fullmatch(name):
+        return "ticket"
+    if _URL_RE.fullmatch(name) or _DOMAIN_RE.fullmatch(name):
+        return "url"
+    if _ACRONYM_RE.fullmatch(name):
+        return "acronym"
+    if any(token in name.lower().split() for token in ("project", "program", "initiative")):
+        return "project"
+    return "proper_noun"
+
+
+def _evidence_for(text: str, start: int, end: int) -> str:
+    left = max(0, start - 120)
+    right = min(len(text), end + 180)
+    snippet = re.sub(r"\s+", " ", text[left:right]).strip()
+    if len(snippet) > _MAX_EVIDENCE_CHARS:
+        snippet = snippet[: _MAX_EVIDENCE_CHARS - 3].rsplit(" ", 1)[0].strip() + "..."
+    return snippet
+
+
+def _iter_entity_matches(text: str):
+    for pattern in (_URL_RE, _HANDLE_RE, _ISSUE_RE, _ACRONYM_RE, _PROPER_NOUN_RE, _DOMAIN_RE):
+        yield from pattern.finditer(text)
+
+
+def extract_entities(text: str, *, max_entities: int = _MAX_ENTITIES_PER_TEXT) -> list[ExtractedEntity]:
+    """Extract deterministic entity mentions from text without external NLP deps."""
+    if not isinstance(text, str) or not text.strip():
+        return []
+
+    found: dict[str, ExtractedEntity] = {}
+    occupied_spans: list[tuple[int, int]] = []
+    for match in sorted(_iter_entity_matches(text), key=lambda item: (item.start(), -(item.end() - item.start()))):
+        if any(match.start() < end and match.end() > start for start, end in occupied_spans):
+            continue
+        name = _clean_entity(match.group(0))
+        if not name or len(name) > _MAX_ENTITY_LEN or name in _STOP_ENTITIES:
+            continue
+        key = normalize_entity_key(name)
+        if len(key) < 2 or key in _STOP_ENTITY_KEYS or key in found:
+            continue
+        found[key] = ExtractedEntity(
+            name=name,
+            entity_key=key,
+            entity_type=_classify_entity(name),
+            evidence=_evidence_for(text, match.start(), match.end()),
+        )
+        occupied_spans.append((match.start(), match.end()))
+        if len(found) >= max_entities:
+            break
+    return list(found.values())
+
+
+def extract_relations(
+    entities: Iterable[ExtractedEntity],
+    *,
+    max_pairs: int = 48,
+) -> list[ExtractedRelation]:
+    """Create co-mention relation edges for entities found in the same evidence unit."""
+    unique: dict[str, ExtractedEntity] = {}
+    for entity in entities:
+        unique.setdefault(entity.entity_key, entity)
+
+    relations: list[ExtractedRelation] = []
+    for left, right in combinations(list(unique.values())[:12], 2):
+        evidence = left.evidence if len(left.evidence) >= len(right.evidence) else right.evidence
+        relations.append(
+            ExtractedRelation(
+                subject_key=left.entity_key,
+                subject_name=left.name,
+                object_key=right.entity_key,
+                object_name=right.name,
+                relation_type="co_mentions",
+                evidence=evidence,
+            )
+        )
+        if len(relations) >= max_pairs:
+            break
+    return relations
+
+
+def stable_graph_id(*parts: Optional[str]) -> str:
+    """Build a stable hash ID for graph rows."""
+    seed = "\x1f".join(str(part or "") for part in parts)
+    return hashlib.sha256(seed.encode("utf-8")).hexdigest()