Add portable PromptCacheOptions for provider-agnostic cache configura…#5660
Add portable PromptCacheOptions for provider-agnostic cache configura…#5660sobychacko wants to merge 1 commit intospring-projects:mainfrom
Conversation
sobychacko
force-pushed
the
feature/core-prompt-cache-types
branch
4 times, most recently
from
be238c5 to
5379f3a
Compare
sobychacko
force-pushed
the
feature/core-prompt-cache-types
branch
from
5379f3a to
4481e68
Compare
Prompt caching requires provider-specific types today — identical strategies live in separate enums (AnthropicCacheStrategy, BedrockCacheStrategy), forcing rewrites when switching providers. Add PromptCacheStrategy and PromptCacheChatOptions mixin interface following the StructuredOutputChatOptions pattern: cache fields as flat builder methods (.promptCacheStrategy(), .promptCacheTtl(), etc.), no wrapper object. Only Anthropic and Bedrock implement it, enabling instanceof detection. Full feature parity with native Anthropic options: strategy, TTL, min content length, per-type TTL/min-length maps, multi-block system caching, content length function. PromptCacheProperties (spring.ai.prompt-cache.*) enables caching from application.properties with zero Java code. Deprecate native types for removal in 2.0.0-M5: CacheOptions, CacheStrategy, CacheTtl classes and their getter/setter/builder methods on both providers. Deprecated types only appear in backward-compat fallback methods that convert native to portable. Production code works entirely through portable types: CacheEligibilityResolver accepts PromptCacheChatOptions directly, BedrockProxyChatModel reads PromptCacheStrategy directly. No deprecated types in createRequest() for either provider. Bedrock warns on unsupported options. Core doc page categorizes providers (explicit, automatic prefix, external lifecycle). Provider pages trimmed to specifics only. All ITs migrated to portable API. Signed-off-by: Soby Chacko <soby.chacko@broadcom.com>
sobychacko
force-pushed
the
feature/core-prompt-cache-types
branch
from
4481e68 to
d21a624
Compare
tzolov
left a comment
tzolov
left a comment
There was a problem hiding this comment.
Looks great. The options abstraction nicely unifies caching capabilities, and the documentation clearly explaining explicit (Anthropic), implicit (OpenAI), and dedicated-service (Gemini) types is helpful — I wasn't aware the others don't expose direct caching control.
@sobychacko, before we merge, though: is this generalization actually warranted? I initially supported it assuming multiple providers would implement caching differently — similar to structured output — but if it'll only ever apply to Anthropic, do we really need the abstraction layer?
|
@tzolov Thats a fair point, something I also struggled a bit. However, it is applied to both Anthropic and Bedrock. They both provide similar API's for prompt caching and we have duplication across those two modules. Bedrock provides prompt caching for both nova and claude models. So, this is up to you. I am ok if we decide not to merge; we just need to live with the redundancy in those two modules, which may not be an unreasonable thing to do in this case. |
3 participants
…tion
Today, configuring prompt caching requires learning provider-specific APIs — AnthropicCacheOptions with AnthropicCacheStrategy for Anthropic, BedrockCacheOptions with BedrockCacheStrategy for Bedrock. Users who switch providers must rewrite their cache configuration, and the conceptually identical strategies (SYSTEM_ONLY, TOOLS_ONLY, etc.) live in separate, unrelated enums.
Introduce PromptCacheStrategy and PromptCacheOptions as core types on the ChatOptions interface, allowing users to configure prompt caching once using portable types that work across providers. Each supporting provider translates these to its native cache mechanism — Anthropic maps strategy, TTL, and minContentLength to AnthropicCacheOptions; Bedrock maps strategy to BedrockCacheOptions (logging that TTL is ignored since Bedrock uses a fixed 5-minute TTL).
Provider-specific cache options take precedence when both are set, preserving full backward compatibility and access to provider-specific features like Anthropic's multi-block system caching or per-message-type TTL configuration.