Prerequisites
Feature Description
I am running into an issue with prompt caching/checkpointing when handling long contexts using a "truncate middle" strategy.
When my context window fills up, the frontend drops the middle of the conversation. This creates a new prompt structure: [Beginning] + [Post-Truncation Context] + [New Tokens].
For all subsequent turns, the [Beginning] + [Post-Truncation Context] block is completely static and joined together. The only thing that changes is the [New Tokens] at the end. However, the checkpointing system seems to ignore this newly formed static block. It only matches the [Beginning] part, and forces the model to reprocess the entire [Post-Truncation Context] on every single subsequent generation.
Motivation
Because the prompt processing doesn't recognize the new joined prefix, Time To First Token (TTFT) is severely degraded on every single turn after the first truncation event. Allowing the cache to properly update and match the new longest common prefix would restore fast inference speeds for long-running sessions.
Possible Implementation
Since the [Beginning] and the [Post-Truncation Context] are entered joined and remain completely static across multiple turns, I don't necessarily need complex RoPE/KV shifting on the fly.
I just want the prompt caching mechanism to properly save, recognize, and match this new combined static prefix once it has been processed. If the exact same joined tokens are sent in the next turn, the cache should hit for the entire [Beginning] + [Post-Truncation Context] block, so the model only has to process the newest tokens at the end.
Prerequisites
Feature Description
I am running into an issue with prompt caching/checkpointing when handling long contexts using a "truncate middle" strategy.
When my context window fills up, the frontend drops the middle of the conversation. This creates a new prompt structure: [Beginning] + [Post-Truncation Context] + [New Tokens].
For all subsequent turns, the [Beginning] + [Post-Truncation Context] block is completely static and joined together. The only thing that changes is the [New Tokens] at the end. However, the checkpointing system seems to ignore this newly formed static block. It only matches the [Beginning] part, and forces the model to reprocess the entire [Post-Truncation Context] on every single subsequent generation.
Motivation
Because the prompt processing doesn't recognize the new joined prefix, Time To First Token (TTFT) is severely degraded on every single turn after the first truncation event. Allowing the cache to properly update and match the new longest common prefix would restore fast inference speeds for long-running sessions.
Possible Implementation
Since the [Beginning] and the [Post-Truncation Context] are entered joined and remain completely static across multiple turns, I don't necessarily need complex RoPE/KV shifting on the fly.
I just want the prompt caching mechanism to properly save, recognize, and match this new combined static prefix once it has been processed. If the exact same joined tokens are sent in the next turn, the cache should hit for the entire [Beginning] + [Post-Truncation Context] block, so the model only has to process the newest tokens at the end.