Skip to content

design: stream-based session & backup loading to reduce memory pressure#135

Open
dev01lay2 wants to merge 1 commit intomainfrom
design/streaming-session-backup-loading
Open

design: stream-based session & backup loading to reduce memory pressure#135
dev01lay2 wants to merge 1 commit intomainfrom
design/streaming-session-backup-loading

Conversation

@dev01lay2
Copy link
Collaborator

Design Document — Stream-Based Session & Backup Loading

仅设计文档,不包含代码更改。

问题

当前 session 分析和 backup 操作将所有数据一次性加载到内存:

  • analyze_sessions 读取并解析所有 .jsonl 文件 → O(total session bytes) 峰值内存
  • preview_session 读取整个文件到内存 → O(file size)
  • Backup/restore 同步复制,UI 无进度反馈

对于有数百个 session 的实例(Pi、小 VPS),峰值内存可达 50MB+,且 UI 在操作期间无响应。

方案概要

  1. Tauri Event Channel 模式 — 新增 *_stream 命令变体,通过 Tauri event 分批推送数据(默认 batch_size=50),前端渐进式渲染
  2. SSH 流式读取层SshConnectionPool::exec_streaming(),用 bounded mpsc channel 分块传输 stdout + StreamDeserializer 增量解析
  3. Backup 进度事件 — 按阶段 emit 进度(config → agents → memory)
  4. 取消支持 — 全局 CancellationToken 注册表 + cancel_stream 命令
  5. 向后兼容 — 保留现有 eager 命令,前端根据文件数量自动选择

内存预算

场景 当前峰值 流式后
Local 500 sessions ~50MB ~2MB
Remote 500 sessions ~5MB ~500KB
Preview 10MB JSONL ~10MB ~200KB

详见 docs/design/streaming-session-backup-loading.md

@github-actions
Copy link
Contributor

📊 Test Coverage Report

Metric Base (main) PR (design/streaming-session-backup-loading) Delta
Lines 74.34% (6134/8251) 74.34% (6134/8251) ⚪ ±0.00%
Functions 68.88% (704/1022) 68.88% (704/1022) ⚪ ±0.00%
Regions 75.86% (10156/13388) 75.86% (10156/13388) ⚪ ±0.00%

Coverage measured by cargo llvm-cov (clawpal-core + clawpal-cli).

@github-actions
Copy link
Contributor

📦 PR Build Artifacts

Platform Download Size
Windows-x64 📥 clawpal-Windows-x64 25.7 MB
macOS-x64 📥 clawpal-macOS-x64 13.3 MB
Linux-x64 📥 clawpal-Linux-x64 103.8 MB
macOS-ARM64 📥 clawpal-macOS-ARM64 12.6 MB

🔨 Built from 19e25ea · View workflow run
⚠️ Unsigned development builds — for testing only

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant