There still seems to be some problems with cache hits缓存命中方面似乎还是有些问题

[pmsleepcheck]

Problem

对同样的项目进行修改
Make modifications to the same project

Is yesterday's input_cache_miss rate bug already in the releases 0.8.17 ?
If it's not yet in version 0.8.17, then this is a false positive.

Proposed solution

Continue to look for other reasons that may cause lower cache hit ratios

Additional context

We can see that cc's hit rate is as high as over 90% Based on experience, cc will even get higher and higher Tui is only 0.81% This leads to huge differences in cost and speed 可以看到cc的命中率高达90%以上（0.91） 根据经验 cc甚至会越来越高 而tui 只有0.84% 这导致了极大的成本和速度差异

By the way, yesterday's data was only about 50% cache hit rate
Although I can't be sure, I believe the last cache hit rate fix is already in release
顺便昨天的数据是只有50%左右的缓存命中率
虽然不能确信但我相信上个缓存命中率修复已经在发行版本

[Rhapsody0x1]

昨天观察到了类似的问题。使用 DeepSeek TUI 的 harness 运行 Terminal Bench 2.0，缓存命中率相当低：
Observed a similar issue yesterday. Running Terminal Bench 2.0 using the DeepSeek TUI harness resulted in a very low cache hit rate.

[pmsleepcheck]

昨天观察到了类似的问题。使用 DeepSeek TUI 的 harness 运行 Terminal Bench 2.0，缓存命中率相当低： Observed a similar issue yesterday. Running Terminal Bench 2.0 using the DeepSeek TUI harness resulted in a very low cache hit rate.

昨天的版本应该有一个缓存方面的修复 感觉大概未命中从40%-到20%的样子 但是还是不够好
如果你是昨天测的似乎可以重来一下看看了 或者等进一步修复
似乎还有其他烧token的现象

[Hmbown]

thank you guys for sharing this - it's very helpful. I have found another bug related to the API and am hoping this hot fix in 0.8.19 will fix most of this. There are also some missed cache tokens in rlm when it's used, it seems

[Rhapsody0x1]

我让 Codex 审查了仓库的代码，以下是它的分析结果，希望这会有帮助。

I asked Codex to review the code. Below is its analysis. Hope this can be useful.

1. High-Confidence Causes / 高置信的原因

第一个核心原因是 Agent 模式默认延迟加载了很多编辑相关工具。当前 always-loaded allowlist 主要包含读文件、搜索、诊断、计划和少量基础 shell 工具，但不包含 apply_patch、write_file、edit_file、git history/inspection 类工具以及测试相关工具。因此一次正常的编码循环中，模型很可能先读取和搜索，然后再通过 ToolSearch 或显式调用激活编辑工具，导致请求顶部的 tool array 发生变化。见 tool_catalog.rs。

The first core cause is that many editing-oriented tools are deferred by default in Agent mode. The always-loaded allowlist mostly covers read/search/diagnostics/planning and basic shell tools, but excludes tools such as apply_patch, write_file, edit_file, git inspection/history tools, and test tools. A normal coding loop can therefore start with read/search tools and later activate editing tools through ToolSearch or requested deferred tool activation, changing the serialized tool array near the top of the request. See tool_catalog.rs.

let always_loaded_in_action_modes = matches!(mode, AppMode::Agent)
    && matches!(
        name,
        "exec_shell"
            | "exec_shell_wait"
            | "exec_shell_interact"
            | "exec_wait"
            | "exec_interact"
    );

!matches!(
    name,
    "read_file"
        | "list_dir"
        | "grep_files"
        | "file_search"
        | "diagnostics"
        | "rlm"
        | "recall_archive"
        | MULTI_TOOL_PARALLEL_NAME
        | "update_plan"
        | "checklist_write"
        | "todo_write"
        | "task_create"
        | "task_list"
        | "task_read"
        | "task_gate_run"
        | "task_shell_start"
        | "task_shell_wait"
        | "github_issue_context"
        | "github_pr_context"
        | REQUEST_USER_INPUT_NAME
)

第二个核心原因是 active tool list 会在对话中途增长。代码已经明确承认：激活 deferred tool 会改变后续工具的 byte offset，并从该位置开始破坏 cached prefix。虽然实现采用 head/tail 两段来减少位移，但新增 deferred tool 仍然会追加到 tool array 尾部，使后续请求相比前一轮出现新的顶部结构。见 tool_catalog.rs 和 tool_catalog.rs。

The second core cause is that the active tool list can grow during the conversation. The code comment explicitly acknowledges that activating deferred tools shifts byte offsets and busts the cached prefix from that point onward. The two-pass head/tail ordering mitigates later-tool displacement, but any newly activated deferred tool is still appended to the tool array, so the next request has new top-of-request schema content. See tool_catalog.rs and tool_catalog.rs.

fn active_tool_list_from_catalog(catalog: &[Tool], active: &HashSet<String>) -> Vec<Tool> {
    // Two-pass for prefix-cache stability (#263). Always-loaded tools come
    // first in their stable catalog order; tools that started life deferred
    // and were activated mid-conversation by ToolSearch get appended at the
    // tail. Otherwise activating a deferred tool shifts every later tool's
    // byte offset and busts the cached prefix from that point onwards.
    let mut head: Vec<Tool> = Vec::new();
    let mut tail: Vec<Tool> = Vec::new();
    // ...
    head.extend(tail);
    head
}

if !def.defer_loading.unwrap_or(false) || active_tools.contains(tool_name) {
    return false;
}

active_tools.insert(tool_name.to_string())

for name in &discovered {
    active_tools.insert(name.clone());
}

第三个核心原因是大上下文模型的 tool-result compaction 过于宽松。对于 large-context model，单个 tool output 的 hard cap 是 180_000 字符，noisy-tool soft cap 是 60_000 字符，压缩后 snippet 仍可保留 40_000 字符。更关键的是，只有 shell、parallel 和 web_search 被视为 noisy，read_file 不在其中。见 context.rs、context.rs 和 context.rs。

The third core cause is that tool-result compaction is far too permissive for large-context models. For large-context models, a single tool output can reach a 180_000 character hard cap, a 60_000 character noisy-tool soft cap, and still keep a 40_000 character snippet after compaction. More importantly, only shell, parallel, and web_search outputs are classified as noisy; read_file is not. See context.rs, context.rs, and context.rs.

const LARGE_CONTEXT_TOOL_RESULT_HARD_LIMIT_CHARS: usize = 180_000;
const LARGE_CONTEXT_TOOL_RESULT_SOFT_LIMIT_CHARS: usize = 60_000;
const LARGE_CONTEXT_TOOL_RESULT_SNIPPET_CHARS: usize = 40_000;

fn tool_result_is_noisy(tool_name: &str) -> bool {
    matches!(
        tool_name,
        "exec_shell"
            | "exec_shell_wait"
            | "exec_shell_interact"
            | "multi_tool_use.parallel"
            | "web_search"
    )
}

let should_compact = raw_chars > limits.hard_limit_chars
    || (tool_result_is_noisy(tool_name) && raw_chars > limits.noisy_soft_limit_chars);
if !should_compact {
    return raw.to_string();
}

中文：第四个核心原因是 read_file 默认返回完整文件内容，并且该工具结果不被视为 noisy。中大型源码文件会被原样注入上下文，并在后续请求中作为历史 tool result 重放。如果每轮读取不同文件，或者读取文件后又有编辑和测试输出，uncached tail 会快速膨胀。见 file.rs。

English: The fourth core cause is that read_file returns the full file body by default and its result is not classified as noisy. Medium or large source files can be injected verbatim into context and replayed as historical tool results in later requests. If each turn reads different files, or combines file reads with edit/test outputs, the uncached tail grows quickly. See file.rs.

let contents = fs::read_to_string(&file_path).map_err(|e| {
    ToolError::execution_failed(format!("Failed to read {}: {}", file_path.display(), e))
})?;

Ok(ToolResult::success(contents))

2. Secondary / Latent Issues / 次要或潜在问题

每个用户 turn 都会追加新的 <turn_meta> 块，其中包含当前日期和 working-set summary。working-set summary 又包含 workspace path、repo summary 和 active paths。这不是解释 0.8% cache hit 的主因，因为它位于请求尾部且内容通常较短；但它会增加每轮 fresh uncached tail，尤其当 active paths 频繁变化时。见 engine.rs 和 working_set.rs。

Each user turn appends a fresh <turn_meta> block containing the current local date and working-set summary. The working-set summary includes workspace path, repo summary, and active paths. This does not by itself explain a 0.8% cache hit ratio because it appears near the request tail and is usually short, but it does add fresh uncached tail content every turn, especially when active paths change often. See engine.rs and working_set.rs.

let summary = if let Some(working_set_summary) = working_set_summary {
    format!("Current local date: {today}\n{working_set_summary}")
} else {
    format!("Current local date: {today}")
};

ContentBlock::Text {
    text: format!("<turn_meta>\n{summary}\n</turn_meta>"),
    cache_control: None,
}

lines.push("## Repo Working Set".to_string());
lines.push(format!("Workspace: {}", workspace.display()));
// ...
lines.push("Active paths (prioritize these):".to_string());

另一个潜在问题是 cache-control metadata 在请求序列化路径中可能丢失。system_to_instructions 会把 SystemPrompt::Blocks flatten 成普通字符串，从而丢弃 block metadata；chat request builder 再把这个字符串作为 system message content 插入；tool_to_chat 序列化工具名称、描述、参数和少量附加字段，但没有传播 tool.cache_control。这不一定是 DeepSeek 自动 prefix caching 的主因，但会影响兼容显式 cache-control 的 provider/path。见 client.rs、chat.rs 和 chat.rs。

Another latent issue is that cache-control metadata appears to be lost in the request serialization path. system_to_instructions flattens SystemPrompt::Blocks into a plain string, discarding block metadata; the chat request builder inserts that string as system message content; tool_to_chat serializes tool name, description, parameters, and a few extra fields, but does not propagate tool.cache_control. This may not be the main cause for DeepSeek automatic prefix caching, but it can prevent explicit cache-control from working on compatible providers/paths. See client.rs, chat.rs, and chat.rs.

Some(SystemPrompt::Blocks(blocks)) => {
    let joined = blocks
        .into_iter()
        .map(|b| b.text)
        .collect::<Vec<_>>()
        .join("\n\n---\n\n");
    // metadata is not preserved in the flattened string
}

out.push(json!({
    "role": "system",
    "content": instructions,
}));

pub(super) fn tool_to_chat(tool: &Tool) -> Value {
    let mut value = json!({
        "type": "function",
        "function": {
            "name": to_api_tool_name(&tool.name),
            "description": tool.description,
            "parameters": tool.input_schema,
        }
    });
    // no tool.cache_control propagation here
    value
}

3. Why This Hurts Cache Hit Ratio / 为什么这会降低命中率

Prompt cache / prefix cache 通常依赖请求前缀 byte-for-byte 稳定。稳定的 system prompt 只是必要条件之一；如果 system 之后、messages 之前或工具 schema 区域的 tool array 发生变化，provider 能复用的 cached prefix 会显著缩短。Deferred tool 在对话中途激活时，会把新的 tool schema 追加到请求顶部附近，因此即使用户重复做同一个项目的修改，前缀也不再和上一轮完全一致。

Prompt cache / prefix cache usually depends on byte-for-byte stability of the request prefix. A stable system prompt is only one requirement; if the tool schema section changes after the system prompt and before messages, the provider can reuse much less cached prefix. When deferred tools are activated mid-conversation, new tool schemas are appended near the top of the request, so even repeated edits in the same project no longer share the exact same prefix as prior requests.

同时，full-file read_file 输出和宽松的 compaction 策略会让历史 tool-result replay 变得非常大。这些内容通常位于较新的 conversation tail，并且随每轮读取、执行命令、修改文件而变化。结果是：可缓存的 prefix 变短，fresh uncached tail 变长，账单中可见的 cache hit ratio 降低，实际延迟和成本收益也下降。

At the same time, full-file read_file outputs and permissive compaction make historical tool-result replay very large. These contents usually live in the newer conversation tail and change with each file read, command execution, and edit. The result is a shorter reusable prefix, a larger fresh uncached tail, lower visible cache hit ratio, and weaker latency/cost benefits.

这也解释了为什么与 Claude Code / cc 对比时差异会很明显。如果 cc 保持更稳定的 tool list，并且更积极地摘要或限制 tool outputs，那么重复编码任务中的 provider-visible prefix 会更长、更稳定，cache hit ratio 就可能保持在高位。而 DeepSeek-TUI 当前路径中，tool-array churn 加上 oversized fresh tool outputs 足以把命中率压到很低；turn metadata 只是额外放大 tail 的因素，不应被过度归因为唯一原因。

This also explains why the comparison with Claude Code / cc can be stark. If cc keeps a more stable tool list and summarizes or limits tool outputs more aggressively, its provider-visible prefix remains longer and more stable across repeated coding tasks, allowing cache hit ratio to stay high. In the current DeepSeek-TUI path, tool-array churn plus oversized fresh tool outputs are enough to drive the hit ratio very low; turn metadata only adds extra tail growth and should not be overclaimed as the sole explanation.

4. Validation / Benchmark Plan / 验证与基准测试计划

我建议添加一个 provider-request snapshot 测试：固定在同一 workspace、同一 system prompt、同一用户任务下，记录连续多轮请求的 serialized tool array、system section、message prefix 和 tool-result tail 大小。重点观察 deferred tool 激活前后，tool array 是否变化，以及变化发生在请求的哪个 byte offset。

I suggest adding a provider-request snapshot test: with the same workspace, same system prompt, and same user task, record the serialized tool array, system section, message prefix, and tool-result tail size across multiple turns. Pay special attention to whether the tool array changes before and after deferred tool activation, and at which byte offset the change appears.

[lloydzhou]

tools应该保持稳定！tools通常会在system prompt之后messages之前被输入给模型，所以按前缀匹配kv cache的话，一旦tools改变，意味着cache失效

[douglarek]

不要仅仅只放缓存命中的数据，你的例子呢，控制变量才能方便查找问题。怎么复现你的缓存命中和你的 cc ，他们是否走了一样的命中路径？

[pmsleepcheck]

不要仅仅只放缓存命中的数据，你的例子呢，控制变量才能方便查找问题。怎么复现你的缓存命中和你的 cc ，他们是否走了一样的命中路径？

我没有预算去做完全相同的对比 抱歉
从经验上来讲 处理相同项目缓存命中率是大致相同的
cc是一个比较一般的参考物 他毕竟不是针对ds优化的

[qinxianyuzou]

I just feel the cost is too high right now — I'm spending $1–2 per day on tokens, and all I'm doing is asking about tiny project questions, like a translation plugin with a few thousand lines of code.

[D7x7z49]

@Hmbown at https://github.com/D7x7z49/llm-context-idea we have many similar ideas.

A cache-optimized execution layer. It is language-agnostic and model-agnostic. It is a rule-based syntax. This may solve your problem.

see: github/spec-kit#2440
see: https://github.com/D7x7z49/waf

[lloydzhou]

llm-context-idea

Research notes on maintaining and reasoning about LLM context windows across long-lived projects. Speculative ideas, heuristic boundaries, and design sketches without empirical validation.

[guancn]

当前的最新版本有改善，但是还不够理想，距离claude code和opencode还是差距较大。暂时先回opencode了，再等两轮更新。

[D7x7z49]

llm-context-idea

Research notes on maintaining and reasoning about LLM context windows across long-lived projects. Speculative ideas, heuristic boundaries, and design sketches without empirical validation.

how?