perf: improve distillation detail retention at 400K+ token sessions#423
Merged
Conversation
Three targeted changes to address information loss in long sessions: 1. Increase gen-0 budget multiplier (8→10): gives the observer LLM 25% more tokens per segment to preserve specific identifiers (error messages, file paths, version numbers). 2. Increase tool output truncation limit (2K→4K): lets the distillation LLM see actual error messages and stack traces instead of compact annotations. 3. Protect recent gen-0 segments from meta-distillation: keeps the 5 most recent gen-0 segments un-archived so their full detail stays in the context prefix while older segments are consolidated. The partition logic in metaDistillInner() uses the threshold check on toConsolidate.length (not existing.length) to prevent kept segments from re-triggering consolidation on idle ticks. Closes #417
When recentSegmentsToKeep keeps 5 gen-0 segments un-archived, the meta distillation (gen>=1) sits at index 0 with the lowest recency score. At layer 3 (distLimit=5), selectDistillations would evict the meta in favor of 5 newer gen-0 segments, losing the consolidated session history. Fix: always include gen>=1 entries first, then fill remaining slots from gen-0 by recency+importance scoring. This ensures the meta is never dropped under compression pressure. Adds 5 tests for selectDistillations meta preservation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses information loss in Lore's distillation pipeline during long sessions (400K+ tokens), where two-stage compression (gen-0 + meta) produces ~53-64:1 total compression, dropping specific identifiers that compaction's single-pass preserves.
Closes #417
Changes
recentSegmentsToKeepconfig, default 5) so their full detail stays in the context prefix while older segments are consolidated by the reflectorselectDistillationsat layer 3: Always includes gen>=1 entries (consolidated session history) before filling remaining slots from gen-0 by recency+importance scoring. Without this, the meta would be evicted at layer 3 (distLimit=5) since 5 newer gen-0 segments outscore it on recencyKey Design Details
Partition logic:
metaDistillInner()applies the threshold check totoConsolidate.length(notexisting.length) to prevent kept segments from re-triggering consolidation on idle ticks — especially under bust pressure whereeffectiveMetaThresholdcan drop to 5.Meta preservation:
selectDistillations()now separates meta (gen>=1) and gen-0 entries. Meta entries always get included first; remaining slots are filled from gen-0 by the existing recency+importance scoring. This ensures the consolidated session history is never dropped under compression pressure.Tests
distillTokenBudgetassertions for MULTIPLIER=10recentSegmentsToKeep: first-round partition, re-trigger prevention, boundary case (count == keep), anchored round with partitionselectDistillationsmeta preservation: basic passthrough, meta guaranteed inclusion, multiple metas, gen-0-only fallback, emergency limit=2