diff --git a/.lore.md b/.lore.md index 3a6d072..ca366cd 100644 --- a/.lore.md +++ b/.lore.md @@ -73,12 +73,27 @@ * **Always ask for session history recall at the start of new sessions**: Session continuity: At session start, recall specific prior-session details (PR titles, branch names, test counts, file names, commit details, lint fixes) as precisely as possible. State explicitly when details were not captured. Proactively log granular session artifacts (exact error messages, PR titles, file paths, line numbers, commit hashes, test names, root causes) so future recall is reliable. User relies on this to resume context quickly. When information is unavailable, say so explicitly and suggest where to find it (GitHub, logs, commit history). Never fabricate or infer details as confirmed facts — always distinguish verified session records from generated/uncertain information. + +* **Always carry forward workflow directives and incident context as persistent hard rules across sessions**: 🔴 HARD RULES (every session): 1. Two-phase Plan/Build: write plan file first, wait for approval, then implement. 2. Pre-merge self-review: rate all changed files by severity with exact paths/line numbers; fix all criticals; confirm CI green. 3. Eval fairness: demand deep fairness analysis before accepting eval results. 4. LTM cache delete must be inside 'changes made' guard — unconditional placement is a bug. 5. Re-state hard rules + completed work at session start with exact file paths, line numbers, commit hashes, PR URLs — flag missing details with 🟡. 6. Root cause first: RCA must precede any proposed solution. 7. Never run \`git remote -v\` with client-controlled cwd. 8. IDs starting with \`LOREAI-GATEWAY-\` are Sentry issue IDs. 9. Pull from origin/main before any codebase exploration or edits. 10. Resolve \`.lore.md\` rebase conflicts with \`--ours\`. 11. When multiple fixes identified, apply together and immediately re-run evals. 12. Before implementing features/debugging, explore relevant codebase area first. 13. Production incidents: acknowledge → investigate → hotfix → code fix → preventive measures. 14. 'Let's go with that plan.' = full approval, proceed immediately. 15. Pre-c \[truncated — entry too long] + + +* **Always fix cache memory leaks with TTL eviction, size cap, and scheduled pruning**: Cache memory leak fix pattern: (1) TTL check in \`.get()\` — delete and return undefined if \`Date.now() - entry.ts > this.ttlMs\`; (2) LRU eviction in \`.set()\` — delete oldest key when \`store.size >= maxEntries\`; (3) \`setInterval(() => this.prune(), 60\_000)\` in constructor. Defaults: \`maxEntries = 10\_000\`, \`ttlMs = 300\_000\` (5 min). Applies to all cache modules. + + +* **Always include database migration versioning context when discussing schema changes**: When discussing schema changes, always preserve migration versioning context: which version introduced the change, which reverted it, the incident that caused the revert, and the rollback migration filename. Reference past incidents when proposing new schema changes to avoid repeating mistakes. + * **Always investigate root causes thoroughly before approving implementation changes**: Always investigate root causes thoroughly before proposing changes: explain why the original design decision was made, downstream effects, and why removing it is safe. Read actual source files, trace full execution paths, report precise file names, line numbers, and exact code snippets verbatim. Pull from origin/main before any codebase exploration or edits. When using git worktrees: stash uncommitted changes, rebase onto origin/main, then restore. Always resolve \`.lore.md\` rebase conflicts by keeping main's version (\`--ours\`). Before considering a PR done, perform a critical self-review: code correctness, test quality, PR description accuracy, accidentally staged files, stale comments, debug artifacts. Fix all issues (labeled C1, C2, etc.), amend commit, force-push. End sessions with a structured summary: (1) all PRs shipped with numbers and one-line descriptions; (2) current eval scores showing Lore vs baseline with delta percentages. When multiple fixes are identified, apply them together and immediately re-run evals — treat eval runs as the primary feedback loop, analyze results per-category. IDs starting with \`LOREAI-GATEWAY-\` are Sentry issue IDs. + +* **Always reject proper-lockfile in favor of flock for advisory locking**: Always use \`flock\` advisory locking instead of \`proper-lockfile\`. \`proper-lockfile@4.1.2\` only checks if a PID is alive on the same machine — fails in containerized environments where PID namespaces reset on restart, leaving stale locks. \`flock\` is automatically released on process exit. Never suggest \`proper-lockfile\` in this codebase. + * **Always request critical code reviews with specific file paths, line numbers, and severity classifications**: Code review standard: provide exact file paths, line ranges, severity classifications (critical/medium/low), root causes, and concrete fix recommendations. Must-fix items called out explicitly before merge. Before merging any PR: (1) run critical self-review covering all changed files; (2) fix all criticals; (3) confirm CI green. Reviews must be skeptical — actively look for subtle bugs (state not cleared on fallback paths, consume-once flag semantics, circuit breaker bypass, concurrency edge cases). Produce explicit verdict alongside ranked findings. Before implementing features or debugging, read all named files deeply and report findings with precise references. Always analyze root causes before proposing solutions. When starting eval-related work, enumerate concrete gaps before proposing solutions. Track which evals have been run vs. pending. After root-cause analysis or bug fix, propose eval extensions covering the newly discovered failure mode. When presented with a GitHub issue, challenge unsubstantiated claims — verify against actual code. + +* **Always request worker tests with a consistent 7-case spec covering compute, missing-record, cleanup retention, and sync scenarios**: Worker test files follow a consistent 7-case spec: (1) compute job — DB lookup + update, (2) missing record — skip without throw, (3) cleanup — hard-delete records archived >30 days, (4) cleanup — preserve recently archived records, (5) sync — process a batch, (6) sync — skip missing records, (7) sync — respect dryRun flag. Tests mock DB and Redis. Applies uniformly across all worker modules. + * **Lore eval scores must beat or match tail-window — scoring below it means lost information**: Lore eval scores must beat or match tail-window baseline — scoring below means lost information (treat as bug). \`inflateScenario(scenario, opts?)\` in \`packages/eval/src/inflate.ts\` — opts is \`{ targetTokens?, excludeKeywords? }\`, NOT positional args; silently fails. Token estimation: chars/4 (scenario convention; chars/3 in baselines.ts for budget safety). Auto-extracts protected keywords from question+referenceAnswer. Adjusts \`question.metadata.turnIndex\` after inflation. 8 replay fixtures, 16 scenarios, 130 questions, 6 baselines in CI. \`--inflate\` incompatible with replay mode — run inflated scenarios in live mode only. Inflator buries preference-change turns (known issue). diff --git a/packages/core/src/config.ts b/packages/core/src/config.ts index 93d6ffc..acedf97 100644 --- a/packages/core/src/config.ts +++ b/packages/core/src/config.ts @@ -80,16 +80,22 @@ export const LoreConfig = z.object({ metaThreshold: z.number().min(3).default(20), /** Max chars per tool output when rendering temporal messages for distillation input. * Outputs longer than this are replaced with a compact annotation preserving line - * count, error signals, and file paths. Default: 2000 (matches upstream OpenCode's - * TOOL_OUTPUT_MAX_CHARS during compaction). Set to 0 to disable. */ - toolOutputMaxChars: z.number().min(0).default(2_000), + * count, error signals, and file paths. Default: 4000. Raised from 2000 to preserve + * error messages and stack traces that exceed 2K chars. See #417. Set to 0 to disable. */ + toolOutputMaxChars: z.number().min(0).default(4_000), + /** Number of most-recent gen-0 segments to keep un-archived when + * meta-distillation fires. These segments retain full detail in the + * context prefix while older segments are consolidated. Default: 5. + * See #417. */ + recentSegmentsToKeep: z.number().min(0).default(5), }) .default({ minMessages: 5, minSegmentTokens: 64, maxSegmentTokens: 16384, metaThreshold: 20, - toolOutputMaxChars: 2_000, + toolOutputMaxChars: 4_000, + recentSegmentsToKeep: 5, }), knowledge: z .object({ diff --git a/packages/core/src/distillation.ts b/packages/core/src/distillation.ts index adba030..2f4988c 100644 --- a/packages/core/src/distillation.ts +++ b/packages/core/src/distillation.ts @@ -101,21 +101,23 @@ export function workerTokenBudget( /** * Compute the max_tokens budget for gen-0 distillation of raw messages. * - * Uses a √N-based formula (8 × √N) instead of a linear ratio so that the + * Uses a √N-based formula (10 × √N) instead of a linear ratio so that the * budget grows sub-linearly with input size. This naturally constrains the * LLM to produce output at ~R ≈ 2–4 (the square-root boundary) and avoids * expansion on small segments where a linear 0.25 ratio + 1024 floor gave * the model far too much room. * - * The multiplier (8) gives ~4× headroom above the R=2.0 target, accounting + * The multiplier (10) gives ~5× headroom above the R=2.0 target, accounting * for the detailed observation format (emoji markers, timestamps, entity - * tags, exact numbers) required by the distillation prompt. + * tags, exact numbers) required by the distillation prompt. Raised from 8 + * to improve retention of specific identifiers (error messages, file paths, + * version numbers) in long sessions (400K+ tokens). See #417. * * @param sourceTokens Estimated source token count from raw messages * @returns Token budget clamped to [256, 4096] */ export function distillTokenBudget(sourceTokens: number): number { - const MULTIPLIER = 8; + const MULTIPLIER = 10; const FLOOR = 256; const CAP = 4096; return Math.max(FLOOR, Math.min(Math.ceil(MULTIPLIER * Math.sqrt(sourceTokens)), CAP)); @@ -1084,19 +1086,31 @@ async function metaDistillInner(input: { // since the last meta-distill — no overlap with the anchor body. const priorMeta = latestMeta(input.projectPath, input.sessionID); + // Partition: keep the most recent N gen-0 segments un-archived so their + // full detail stays in the context prefix. Only consolidate older segments. + // This prevents the two-stage compression from dropping specific identifiers + // (error messages, file paths, version numbers) in long sessions. See #417. + const cfg = config(); + const recentToKeep = cfg.distillation.recentSegmentsToKeep; + const toConsolidate = recentToKeep > 0 && existing.length >= recentToKeep + ? existing.slice(0, -recentToKeep) + : existing; + // Threshold: first meta needs ≥3 gen-0 segments to consolidate. Subsequent // anchored metas only need ≥1 new gen-0 since the prior meta already covers // earlier history; without this distinction, every meta-distill round would // need a fresh pile of segments and we'd lose the incremental-update benefit. + // Apply threshold to toConsolidate (not existing) to prevent the kept recent + // segments from re-triggering consolidation on the next idle tick. if (priorMeta) { - if (existing.length === 0) return null; + if (toConsolidate.length === 0) return null; } else { - if (existing.length < 3) return null; + if (toConsolidate.length < 3) return null; } - const userContent = recursiveUser(existing, priorMeta?.observations); + const userContent = recursiveUser(toConsolidate, priorMeta?.observations); - const model = input.model ?? config().model; + const model = input.model ?? cfg.model; const inputTokens = Math.ceil(userContent.length / 3); const maxTokens = workerTokenBudget(inputTokens, 0.25, 1024, 8192); const responseText = await input.llm.prompt( @@ -1117,10 +1131,10 @@ async function metaDistillInner(input: { // covers new gen-0 since the last meta — we must consult the prior meta's // generation explicitly to keep the chain monotonic. const maxGen = Math.max( - ...existing.map((d) => d.generation), + ...toConsolidate.map((d) => d.generation), priorMeta?.generation ?? 0, ); - const allSourceIDs = existing.flatMap((d) => d.source_ids); + const allSourceIDs = toConsolidate.flatMap((d) => d.source_ids); // Atomic: store the new meta row + archive the merged gen-0 rows in one // transaction. Without this, a crash between the two would leave stale @@ -1139,10 +1153,11 @@ async function metaDistillInner(input: { generation: maxGen + 1, callType: input.callType, }); - // Archive the gen-0 distillations that were merged into gen-1+. - // They remain searchable via BM25 recall but are excluded from the - // in-context prefix and (post-F2) from `loadForSession`'s default path. - archiveDistillations(existing.map((d) => d.id)); + // Archive only the consolidated gen-0 distillations — recent segments + // kept via recentSegmentsToKeep remain non-archived in the prefix. + // Archived rows remain searchable via BM25 recall but are excluded from + // the in-context prefix and (post-F2) from `loadForSession`'s default path. + archiveDistillations(toConsolidate.map((d) => d.id)); db().exec("COMMIT"); } catch (e) { db().exec("ROLLBACK"); diff --git a/packages/core/src/gradient.ts b/packages/core/src/gradient.ts index 1088056..4e3e2aa 100644 --- a/packages/core/src/gradient.ts +++ b/packages/core/src/gradient.ts @@ -1229,23 +1229,44 @@ function importanceBonus(d: Distillation): number { return Math.min(bonus, 1.0); } -function selectDistillations(all: Distillation[], limit: number): Distillation[] { +export function selectDistillations(all: Distillation[], limit: number): Distillation[] { if (all.length <= limit) return all; - // Recency: normalize to [0, 0.7] where oldest = 0.0, newest = 0.7. - // Use (length - 1) as divisor so the last entry gets full recency weight. - const maxIdx = all.length - 1; - const scored = all.map((d, i) => ({ + // Always include meta distillations (gen >= 1) — they contain the + // consolidated session history and must not be evicted by recency-weighted + // gen-0 segments. Without this guarantee, layer 3 (distLimit=5) would drop + // the meta in favor of 5 recent gen-0 segments, losing older context. #417. + const meta = all.filter((d) => d.generation >= 1); + const gen0 = all.filter((d) => d.generation === 0); + const remainingSlots = limit - meta.length; + + // If meta entries alone fill or exceed the limit, keep them all by score. + if (remainingSlots <= 0) { + const maxIdx = meta.length - 1; + const scored = meta.map((d, i) => ({ + d, + score: (maxIdx > 0 ? (i / maxIdx) : 1) * 0.7 + importanceBonus(d) * 0.3, + })); + return scored + .sort((a, b) => b.score - a.score) + .slice(0, limit) + .map((s) => s.d) + .sort((a, b) => a.created_at - b.created_at); + } + + // Fill remaining slots from gen-0 by recency + importance scoring. + const maxIdx = gen0.length - 1; + const scored = gen0.map((d, i) => ({ d, score: (maxIdx > 0 ? (i / maxIdx) : 1) * 0.7 + importanceBonus(d) * 0.3, })); - - // Keep top N by score, then re-sort chronologically (cache-safe). - return scored + const topGen0 = scored .sort((a, b) => b.score - a.score) - .slice(0, limit) - .map((s) => s.d) - .sort((a, b) => a.created_at - b.created_at); + .slice(0, remainingSlots) + .map((s) => s.d); + + // Merge and re-sort chronologically (cache-safe). + return [...meta, ...topGen0].sort((a, b) => a.created_at - b.created_at); } // Build a synthetic message pair containing the distilled history. diff --git a/packages/core/test/distillation.test.ts b/packages/core/test/distillation.test.ts index f9ab587..651a244 100644 --- a/packages/core/test/distillation.test.ts +++ b/packages/core/test/distillation.test.ts @@ -832,6 +832,190 @@ describe("metaDistill — anchored second round", () => { }); }); +// ─── recentSegmentsToKeep — preserve recent gen-0 detail (#417) ───────────── + +describe("metaDistill — recentSegmentsToKeep", () => { + beforeEach(() => { + const pid = ensureProject(META_PROJECT); + db().query("DELETE FROM distillations WHERE project_id = ?").run(pid); + }); + + test("keeps recent gen-0 segments un-archived when more than recentSegmentsToKeep exist (first round)", async () => { + const pid = ensureProject(META_PROJECT); + // Insert 8 gen-0 rows with ascending timestamps so ordering is deterministic. + const ids: string[] = []; + for (let i = 0; i < 8; i++) { + ids.push( + insertGen0({ + projectId: pid, + sessionID: META_SESSION, + observations: `obs-${i}`, + createdAt: Date.now() + i * 1000, + }), + ); + } + + const llm = makeStubLLM("\nconsolidated older segments\n"); + const result = await metaDistill({ + llm, + projectPath: META_PROJECT, + sessionID: META_SESSION, + }); + + expect(result).not.toBeNull(); + + // With recentSegmentsToKeep=5 and 8 gen-0 rows: + // toConsolidate = first 3 (ids 0-2), toKeep = last 5 (ids 3-7) + const rows = db() + .query( + "SELECT id, archived, generation FROM distillations WHERE project_id = ? AND session_id = ? ORDER BY created_at ASC", + ) + .all(pid, META_SESSION) as Array<{ id: string; archived: number; generation: number }>; + + // First 3 gen-0 rows should be archived. + for (let i = 0; i < 3; i++) { + const row = rows.find((r) => r.id === ids[i]); + expect(row).toBeDefined(); + expect(row!.archived).toBe(1); + } + // Last 5 gen-0 rows should remain non-archived. + for (let i = 3; i < 8; i++) { + const row = rows.find((r) => r.id === ids[i]); + expect(row).toBeDefined(); + expect(row!.archived).toBe(0); + } + // A new gen-1 meta row should exist. + const meta = rows.find((r) => r.generation === 1); + expect(meta).toBeDefined(); + expect(meta!.archived).toBe(0); + + // Only the first 3 segments should appear in the LLM prompt. + expect(llm.prompts).toHaveLength(1); + expect(llm.prompts[0]!.user).toContain("obs-0"); + expect(llm.prompts[0]!.user).toContain("obs-1"); + expect(llm.prompts[0]!.user).toContain("obs-2"); + expect(llm.prompts[0]!.user).not.toContain("obs-3"); + expect(llm.prompts[0]!.user).not.toContain("obs-7"); + }); + + test("does not re-trigger consolidation when kept segments exist with a prior meta", async () => { + const pid = ensureProject(META_PROJECT); + // Set up: prior meta exists, 5 recent gen-0 rows (exactly recentSegmentsToKeep). + insertMeta({ + projectId: pid, + sessionID: META_SESSION, + observations: "PRIOR_META", + generation: 1, + }); + for (let i = 0; i < 5; i++) { + insertGen0({ + projectId: pid, + sessionID: META_SESSION, + observations: `kept-obs-${i}`, + createdAt: Date.now() + i * 1000, + }); + } + + const llm = makeStubLLM("should not be called"); + const result = await metaDistill({ + llm, + projectPath: META_PROJECT, + sessionID: META_SESSION, + }); + + // With 5 gen-0 rows and recentSegmentsToKeep=5, toConsolidate is empty. + // The threshold check should short-circuit. + expect(result).toBeNull(); + expect(llm.prompts).toHaveLength(0); + }); + + test("returns null when gen-0 count equals recentSegmentsToKeep (nothing to consolidate)", async () => { + const pid = ensureProject(META_PROJECT); + // Insert exactly 5 gen-0 rows (== recentSegmentsToKeep). + // toConsolidate is empty, so meta-distillation short-circuits. + for (let i = 0; i < 5; i++) { + insertGen0({ + projectId: pid, + sessionID: META_SESSION, + observations: `all-obs-${i}`, + createdAt: Date.now() + i * 1000, + }); + } + + const llm = makeStubLLM("should not be called"); + const result = await metaDistill({ + llm, + projectPath: META_PROJECT, + sessionID: META_SESSION, + }); + + // With 5 gen-0 rows and recentSegmentsToKeep=5, toConsolidate is empty. + expect(result).toBeNull(); + expect(llm.prompts).toHaveLength(0); + + // All 5 gen-0 rows remain non-archived. + const rows = db() + .query( + "SELECT archived FROM distillations WHERE project_id = ? AND session_id = ? AND generation = 0", + ) + .all(pid, META_SESSION) as Array<{ archived: number }>; + expect(rows.every((r) => r.archived === 0)).toBe(true); + expect(rows).toHaveLength(5); + }); + + test("anchored round with recentSegmentsToKeep keeps recent segments", async () => { + const pid = ensureProject(META_PROJECT); + // Prior meta + 7 new gen-0 rows. + insertMeta({ + projectId: pid, + sessionID: META_SESSION, + observations: "PRIOR_META", + generation: 1, + }); + const ids: string[] = []; + for (let i = 0; i < 7; i++) { + ids.push( + insertGen0({ + projectId: pid, + sessionID: META_SESSION, + observations: `anchored-obs-${i}`, + createdAt: Date.now() + i * 1000, + }), + ); + } + + const llm = makeStubLLM("\nanchored update\n"); + const result = await metaDistill({ + llm, + projectPath: META_PROJECT, + sessionID: META_SESSION, + }); + + expect(result).not.toBeNull(); + + // toConsolidate = first 2 (ids 0-1), toKeep = last 5 (ids 2-6). + const gen0Rows = db() + .query( + "SELECT id, archived FROM distillations WHERE project_id = ? AND session_id = ? AND generation = 0 ORDER BY created_at ASC", + ) + .all(pid, META_SESSION) as Array<{ id: string; archived: number }>; + + // First 2 archived. + expect(gen0Rows.find((r) => r.id === ids[0])!.archived).toBe(1); + expect(gen0Rows.find((r) => r.id === ids[1])!.archived).toBe(1); + // Last 5 remain non-archived. + for (let i = 2; i < 7; i++) { + expect(gen0Rows.find((r) => r.id === ids[i])!.archived).toBe(0); + } + + // Prompt should contain anchor + only the first 2 segments. + expect(llm.prompts[0]!.user).toContain(""); + expect(llm.prompts[0]!.user).toContain("anchored-obs-0"); + expect(llm.prompts[0]!.user).toContain("anchored-obs-1"); + expect(llm.prompts[0]!.user).not.toContain("anchored-obs-2"); + }); +}); + // ─── detectSegments (token-aware splitting) ───────────────────────────────── describe("detectSegments", () => { @@ -1372,21 +1556,21 @@ describe("run() expansion guard and tiny-segment handling", () => { describe("distillTokenBudget", () => { test("returns floor (256) for small inputs", () => { - // 8 * √64 = 64 → below floor + // 10 * √64 = 80 → below floor expect(distillTokenBudget(64)).toBe(256); expect(distillTokenBudget(100)).toBe(256); expect(distillTokenBudget(0)).toBe(256); }); test("returns √N-based value for medium inputs", () => { - // 8 * √2000 = 8 * 44.72 = 358 - expect(distillTokenBudget(2000)).toBe(358); - // 8 * √4000 = 8 * 63.25 = 506 - expect(distillTokenBudget(4000)).toBe(506); + // 10 * √2000 = 10 * 44.72 = 448 + expect(distillTokenBudget(2000)).toBe(448); + // 10 * √4000 = 10 * 63.25 = 633 + expect(distillTokenBudget(4000)).toBe(633); }); test("returns cap (4096) for very large inputs", () => { - // 8 * √(300000) = 8 * 547.7 = 4382 → above cap + // 10 * √(300000) = 10 * 547.7 = 5477 → above cap expect(distillTokenBudget(300000)).toBe(4096); }); @@ -1401,11 +1585,11 @@ describe("distillTokenBudget", () => { test("is much smaller than old linear budget for typical segments", () => { // Old: workerTokenBudget(2000, 0.25, 1024, 8192) = 1024 (floor) - // New: distillTokenBudget(2000) = 358 + // New: distillTokenBudget(2000) = 448 expect(distillTokenBudget(2000)).toBeLessThan(workerTokenBudget(2000, 0.25, 1024, 8192)); // Old: workerTokenBudget(8192, 0.25, 1024, 8192) = 2048 - // New: distillTokenBudget(8192) = 724 + // New: distillTokenBudget(8192) = 906 expect(distillTokenBudget(8192)).toBeLessThan(workerTokenBudget(8192, 0.25, 1024, 8192)); }); }); diff --git a/packages/core/test/gradient.test.ts b/packages/core/test/gradient.test.ts index eca0fb8..ad81b03 100644 --- a/packages/core/test/gradient.test.ts +++ b/packages/core/test/gradient.test.ts @@ -24,6 +24,7 @@ import { setCachePricing, shouldCompress, getTier, + selectDistillations, } from "../src/gradient"; import type { LoreMessage, LorePart, LoreMessageWithParts } from "../src/types"; import { isToolPart } from "../src/types"; @@ -2480,3 +2481,81 @@ describe("tier-based context management", () => { }); }); }); + +// ─── selectDistillations — meta preservation guarantee (#417) ─────────────── + +describe("selectDistillations", () => { + /** Create a distillation stub with the fields selectDistillations uses. */ + function dist(id: string, generation: number, createdAt: number, observations = ""): { + id: string; observations: string; generation: number; token_count: number; created_at: number; session_id: string; + } { + return { id, observations, generation, token_count: 100, created_at: createdAt, session_id: "sel-sess" }; + } + + test("returns all when count <= limit", () => { + const all = [dist("a", 0, 1), dist("b", 0, 2), dist("c", 0, 3)]; + expect(selectDistillations(all, 5)).toEqual(all); + expect(selectDistillations(all, 3)).toEqual(all); + }); + + test("always includes meta (gen>=1) even when it has lowest recency", () => { + // 1 meta (oldest) + 5 gen-0 (newer) → limit=5 should keep the meta + 4 gen-0 + const meta = dist("meta", 1, 100, "decided to use flock"); + const gen0 = Array.from({ length: 5 }, (_, i) => + dist(`g0-${i}`, 0, 200 + i * 10), + ); + const all = [meta, ...gen0]; // chronological order + + const selected = selectDistillations(all, 5); + expect(selected).toHaveLength(5); + // Meta must be present. + expect(selected.some((d) => d.id === "meta")).toBe(true); + // The oldest gen-0 should be dropped (lowest recency among gen-0). + expect(selected.some((d) => d.id === "g0-0")).toBe(false); + // Result should be chronologically sorted. + for (let i = 1; i < selected.length; i++) { + expect(selected[i]!.created_at).toBeGreaterThanOrEqual(selected[i - 1]!.created_at); + } + }); + + test("preserves multiple meta entries when they exist", () => { + const meta1 = dist("meta1", 1, 100); + const meta2 = dist("meta2", 2, 150); + const gen0 = Array.from({ length: 5 }, (_, i) => + dist(`g0-${i}`, 0, 200 + i * 10), + ); + const all = [meta1, meta2, ...gen0]; + + const selected = selectDistillations(all, 5); + expect(selected).toHaveLength(5); + expect(selected.some((d) => d.id === "meta1")).toBe(true); + expect(selected.some((d) => d.id === "meta2")).toBe(true); + // 3 gen-0 slots remaining, filled by most recent. + expect(selected.some((d) => d.id === "g0-4")).toBe(true); + expect(selected.some((d) => d.id === "g0-3")).toBe(true); + expect(selected.some((d) => d.id === "g0-2")).toBe(true); + }); + + test("selects by recency when all entries are gen-0 (no meta)", () => { + const all = Array.from({ length: 8 }, (_, i) => + dist(`g0-${i}`, 0, 100 + i * 10), + ); + const selected = selectDistillations(all, 3); + expect(selected).toHaveLength(3); + // Most recent 3 gen-0 should win. + expect(selected.map((d) => d.id)).toEqual(["g0-5", "g0-6", "g0-7"]); + }); + + test("emergency limit=2 keeps meta + most recent gen-0", () => { + const meta = dist("meta", 1, 100, "architecture decision"); + const gen0 = Array.from({ length: 5 }, (_, i) => + dist(`g0-${i}`, 0, 200 + i * 10), + ); + const all = [meta, ...gen0]; + + const selected = selectDistillations(all, 2); + expect(selected).toHaveLength(2); + expect(selected[0]!.id).toBe("meta"); + expect(selected[1]!.id).toBe("g0-4"); // most recent gen-0 + }); +});