fix(llm-client): enable Anthropic prompt caching by default - #233
fix(llm-client): enable Anthropic prompt caching by default#233nachiketb-nvidia wants to merge 1 commit into
Conversation
WalkthroughThe change adds request-level caching control to ChangesPrompt caching
Estimated code review effort: 4 (Complex) | ~45 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
crates/libsy-llm-client/src/client.rs (1)
936-952: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winBroaden test coverage for the new caching branches.
The new test only covers a single plain-string message content block being marked and then unmarked. It does not exercise: preserving a caller-supplied
cache_controlwhen caching is already enabled (thehas_anthropic_cache_controlearly return), thesystem-prompt fallback, the last-tool fallback, or removal from atoolsarray withinput_schema. Given this logic gates a cost-sensitive feature enabled by default, add cases for these branches.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/libsy-llm-client/src/client.rs` around lines 936 - 952, Expand the test `anthropic_prompt_caching_is_injected_and_can_be_disabled` with cases covering preservation of caller-supplied `cache_control` via the `has_anthropic_cache_control` early return, system-prompt fallback, last-tool fallback, and removal of `cache_control` from tools containing `input_schema`. Assert each branch’s expected JSON behavior while retaining the existing plain-string injection and disablement coverage.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/libsy-llm-client/src/client.rs`:
- Around line 718-779: Add concise comments to the private helpers
mark_last_text_cacheable, has_anthropic_cache_control, remove_cache_controls,
and is_anthropic_cacheable explaining their traversal purpose, the meaning of
Anthropic-cacheable content, the reverse traversal used to select the last text
block, and why input_schema is excluded from recursion. Keep the implementation
unchanged and align the wording with the nearby apply_anthropic_prompt_caching
comment.
- Around line 685-740: Update apply_anthropic_prompt_caching and
mark_last_text_cacheable to identify and mark the last block accepted by
is_anthropic_cacheable, including trailing image, document, tool_use, and
tool_result blocks rather than only text blocks. Add tests covering trailing
tool_use and tool_result blocks, and ensure the implementation does not create
redundant manual cache breakpoints when supported automatic top-level
cache_control is used.
---
Nitpick comments:
In `@crates/libsy-llm-client/src/client.rs`:
- Around line 936-952: Expand the test
`anthropic_prompt_caching_is_injected_and_can_be_disabled` with cases covering
preservation of caller-supplied `cache_control` via the
`has_anthropic_cache_control` early return, system-prompt fallback, last-tool
fallback, and removal of `cache_control` from tools containing `input_schema`.
Assert each branch’s expected JSON behavior while retaining the existing
plain-string injection and disablement coverage.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 58c8b986-f59b-421b-918f-54e783240d64
📒 Files selected for processing (2)
crates/libsy-llm-client/src/client.rscrates/protocol/src/llm.rs
| // Applies Switchyard's prompt-caching policy to an Anthropic request body. | ||
| fn apply_anthropic_prompt_caching(body: &mut Value, enabled: bool) { | ||
| if !enabled { | ||
| remove_cache_controls(body); | ||
| return; | ||
| } | ||
| if has_anthropic_cache_control(body) { | ||
| return; | ||
| } | ||
|
|
||
| if let Some(messages) = body.get_mut("messages").and_then(Value::as_array_mut) { | ||
| for message in messages.iter_mut().rev() { | ||
| if message | ||
| .get_mut("content") | ||
| .is_some_and(mark_last_text_cacheable) | ||
| { | ||
| return; | ||
| } | ||
| } | ||
| } | ||
| if body.get_mut("system").is_some_and(mark_last_text_cacheable) { | ||
| return; | ||
| } | ||
| if let Some(tool) = body | ||
| .get_mut("tools") | ||
| .and_then(Value::as_array_mut) | ||
| .and_then(|tools| tools.last_mut()) | ||
| .and_then(Value::as_object_mut) | ||
| { | ||
| tool.insert("cache_control".to_string(), json!({"type": "ephemeral"})); | ||
| } | ||
| } | ||
|
|
||
| fn mark_last_text_cacheable(content: &mut Value) -> bool { | ||
| match content { | ||
| Value::String(text) => { | ||
| *content = json!([{ | ||
| "type": "text", | ||
| "text": std::mem::take(text), | ||
| "cache_control": {"type": "ephemeral"} | ||
| }]); | ||
| true | ||
| } | ||
| Value::Array(blocks) => blocks.iter_mut().rev().any(|block| { | ||
| let Some(object) = block.as_object_mut() else { | ||
| return false; | ||
| }; | ||
| if object.get("type").and_then(Value::as_str) != Some("text") { | ||
| return false; | ||
| } | ||
| object.insert("cache_control".to_string(), json!({"type": "ephemeral"})); | ||
| true | ||
| }), | ||
| _ => false, | ||
| } | ||
| } |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
Does Anthropic's Messages API support a top-level automatic cache_control field, and is it compatible with existing per-block cache_control breakpoints in the same request?
💡 Result:
Yes, Anthropic's Messages API supports a top-level cache_control field, which enables automatic prompt caching [1][2]. When you include a cache_control field at the top level of your request, the system automatically applies a cache breakpoint to the last cacheable block and updates it as the conversation grows [1][3]. This method is designed for simplicity, particularly in multi-turn conversations [1][2]. Furthermore, automatic caching is compatible with explicit, per-block cache_control breakpoints within the same request [4][3]. When both are used together, the automatic cache breakpoint consumes one of the available four breakpoint slots per request [3]. This allows developers to combine the convenience of automatic, top-level caching with the fine-grained control provided by manual block-level breakpoints [4][3].
Citations:
- 1: https://platform.claude.com/docs/en/build-with-claude/prompt-caching
- 2: https://platform.claude.com/docs/en/build-with-claude/prompt-caching?f80ce999_sort_date=desc
- 3: https://platform.claude.com/docs/en/build-with-claude/prompt-caching?%3F__hstc=43401018.71aa366c60c32c7e3032e45be702fadd.1753488000274.1753488000275.1753488000276.1&f80ce999_sort_date=desc
- 4: https://platform.claude.com/cookbook/misc-prompt-caching
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- client.rs symbols ---'
rg -n -C 8 'apply_anthropic_prompt_caching|mark_last_text_cacheable|is_anthropic_cacheable|remove_cache_controls|has_anthropic_cache_control' crates/libsy-llm-client/src/client.rs
printf '%s\n' '--- Anthropic API/version references ---'
rg -n -i -C 3 'anthropic|cache_control|prompt.?cach|messages' crates/libsy-llm-client Cargo.toml Cargo.lock 2>/dev/null | head -n 300
printf '%s\n' '--- package files ---'
git ls-files '*Cargo.toml' '*Cargo.lock'Repository: NVIDIA-NeMo/Switchyard
Length of output: 27863
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- helper implementation and tests ---'
sed -n '685,780p' crates/libsy-llm-client/src/client.rs
sed -n '920,1035p' crates/libsy-llm-client/src/client.rs
printf '%s\n' '--- Anthropic wire-format definitions and conversions ---'
rg -n -C 5 'AnthropicMessages|tool_use|tool_result|document|cache_control|content' crates/libsy-llm-client/src
printf '%s\n' '--- request configuration and extra_body documentation ---'
rg -n -C 5 'struct HttpBackendConfig|extra_body|anthropic-version|caching' crates/libsy-llm-client/src crates/libsy-llm-client/README.mdRepository: NVIDIA-NeMo/Switchyard
Length of output: 50380
🏁 Script executed:
#!/bin/bash
set -e
python3 - <<'PY'
import json
def is_cacheable(obj):
return "input_schema" in obj or obj.get("type") in {
"text", "image", "document", "tool_use", "tool_result"
}
def mark_last_text_cacheable(content):
if isinstance(content, str):
return [{"type": "text", "text": content,
"cache_control": {"type": "ephemeral"}}]
if isinstance(content, list):
for block in reversed(content):
if not isinstance(block, dict) or block.get("type") != "text":
continue
block["cache_control"] = {"type": "ephemeral"}
return True
return False
cases = [
["text", "tool_use"],
["text", "tool_result"],
["image", "tool_result"],
["tool_result"],
]
for types in cases:
body = {"messages": [{"role": "assistant", "content":
[{"type": t} for t in types]}]}
mark_last_text_cacheable(body["messages"][0]["content"])
markers = [
(i, b.get("type"))
for i, b in enumerate(body["messages"][0]["content"])
if isinstance(b, dict) and "cache_control" in b
]
print(json.dumps({"input": types, "marked": markers}))
PYRepository: NVIDIA-NeMo/Switchyard
Length of output: 368
🌐 Web query:
site:platform.claude.com/docs prompt caching top-level cache_control automatic cache_control Messages API anthropic-version 2023-06-01 tool_use tool_result
💡 Result:
Prompt caching in the Anthropic Messages API allows you to cache large prompt prefixes—including system instructions, tool definitions, and long conversation histories—to reduce latency and costs [1]. You can implement this using either top-level automatic caching or explicit cache breakpoints [1]. Top-Level vs. Explicit Cache Control - Top-Level Cache Control: By adding a single cache_control field at the top level of your request body, the API automatically applies a cache breakpoint to the last cacheable block in your request [1][2]. This is the simplest method and is well-suited for growing message histories [1]. - Explicit Cache Breakpoints: For more precise control, you can place a cache_control object directly on individual content blocks (such as specific text blocks or tool definitions) [1][2]. This allows you to cache different sections that change at different frequencies [1]. The cache_control object requires a type (currently always "ephemeral") and an optional ttl (time-to-live), which can be set to "5m" (5 minutes) or "1h" (1 hour) [3][2]. Tool Use and Cache Interaction - Automatic Breakpoints with Server Tools: If your request uses a server tool (e.g., web search, code execution), the API automatically places a cache breakpoint on the tool result before the next iteration of the agentic loop [4]. This automatic breakpoint uses a default 5-minute TTL, regardless of the TTLs you manually configure for other blocks [4]. - Tool Results: You can enable caching for tool results by attaching a cache_control object to the tool_result block [5]. When handling tool calls, ensure your tool_result blocks are placed first in the user message content array [6]. - Tool Definitions: Prompt caching references the full prompt prefix, which includes the tools, system instructions, and messages in that specific order [1]. Changing the tools array (such as adding, removing, or reordering tools) will invalidate the cache for that prefix [7]. For optimal performance, always include the anthropic-version header (e.g., 2023-06-01 or later) to ensure access to these features [3]. You can monitor the effectiveness of your caching strategy by checking the cache_creation and cache_read_input_tokens fields in the response usage object [8][9][7].
Citations:
- 1: https://platform.claude.com/docs/en/build-with-claude/prompt-caching
- 2: https://platform.claude.com/docs/en/api/messages/create
- 3: https://platform.claude.com/docs/en/api/python/messages
- 4: https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-use-with-prompt-caching
- 5: https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-runner
- 6: https://platform.claude.com/docs/en/agents-and-tools/tool-use/handle-tool-calls
- 7: https://platform.claude.com/docs/en/build-with-claude/cache-diagnostics
- 8: https://platform.claude.com/docs/en/api/cli/messages
- 9: https://platform.claude.com/docs/en/api/cli/messages/create
Mark the last Anthropic cacheable block, not only text blocks.
is_anthropic_cacheable includes image, document, tool_use, and tool_result, but mark_last_text_cacheable checks only text. A trailing tool result therefore causes an earlier text block to be marked, or no message block to be marked. Reuse is_anthropic_cacheable and add tests for trailing tool_use and tool_result blocks.
Anthropic also supports top-level automatic cache_control for the Messages API, including use with explicit breakpoints. Consider using it instead of this traversal if all targeted providers support it; otherwise, prevent redundant manual breakpoints.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/libsy-llm-client/src/client.rs` around lines 685 - 740, Update
apply_anthropic_prompt_caching and mark_last_text_cacheable to identify and mark
the last block accepted by is_anthropic_cacheable, including trailing image,
document, tool_use, and tool_result blocks rather than only text blocks. Add
tests covering trailing tool_use and tool_result blocks, and ensure the
implementation does not create redundant manual cache breakpoints when supported
automatic top-level cache_control is used.
| fn mark_last_text_cacheable(content: &mut Value) -> bool { | ||
| match content { | ||
| Value::String(text) => { | ||
| *content = json!([{ | ||
| "type": "text", | ||
| "text": std::mem::take(text), | ||
| "cache_control": {"type": "ephemeral"} | ||
| }]); | ||
| true | ||
| } | ||
| Value::Array(blocks) => blocks.iter_mut().rev().any(|block| { | ||
| let Some(object) = block.as_object_mut() else { | ||
| return false; | ||
| }; | ||
| if object.get("type").and_then(Value::as_str) != Some("text") { | ||
| return false; | ||
| } | ||
| object.insert("cache_control".to_string(), json!({"type": "ephemeral"})); | ||
| true | ||
| }), | ||
| _ => false, | ||
| } | ||
| } | ||
|
|
||
| fn has_anthropic_cache_control(value: &Value) -> bool { | ||
| match value { | ||
| Value::Object(object) => { | ||
| (is_anthropic_cacheable(object) && object.contains_key("cache_control")) | ||
| || object | ||
| .iter() | ||
| .filter(|(key, _)| key.as_str() != "input_schema") | ||
| .any(|(_, value)| has_anthropic_cache_control(value)) | ||
| } | ||
| Value::Array(values) => values.iter().any(has_anthropic_cache_control), | ||
| _ => false, | ||
| } | ||
| } | ||
|
|
||
| fn remove_cache_controls(body: &mut Value) { | ||
| match body { | ||
| Value::Object(object) => { | ||
| if is_anthropic_cacheable(object) { | ||
| object.remove("cache_control"); | ||
| } | ||
| for (key, value) in object { | ||
| if key != "input_schema" { | ||
| remove_cache_controls(value); | ||
| } | ||
| } | ||
| } | ||
| Value::Array(values) => values.iter_mut().for_each(remove_cache_controls), | ||
| _ => {} | ||
| } | ||
| } | ||
|
|
||
| fn is_anthropic_cacheable(object: &Map<String, Value>) -> bool { | ||
| object.contains_key("input_schema") | ||
| || matches!( | ||
| object.get("type").and_then(Value::as_str), | ||
| Some("text" | "image" | "document" | "tool_use" | "tool_result") | ||
| ) | ||
| } |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Add comments for these private traversal helpers.
mark_last_text_cacheable, has_anthropic_cache_control, remove_cache_controls, and is_anthropic_cacheable implement non-trivial recursive JSON traversal with an input_schema exclusion rule, but none carry a comment explaining intent (why input_schema is excluded, what "cacheable" means, why traversal order matters). apply_anthropic_prompt_caching has a one-line comment; these four do not.
As per coding guidelines, "Add concise Rust documentation comments (///) for public items and comments for module intent, non-obvious private helpers, ... and complex validation, routing, configuration, async, lifecycle, or concurrency logic."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/libsy-llm-client/src/client.rs` around lines 718 - 779, Add concise
comments to the private helpers mark_last_text_cacheable,
has_anthropic_cache_control, remove_cache_controls, and is_anthropic_cacheable
explaining their traversal purpose, the meaning of Anthropic-cacheable content,
the reverse traversal used to select the last text block, and why input_schema
is excluded from recursion. Keep the implementation unchanged and align the
wording with the nearby apply_anthropic_prompt_caching comment.
Source: Coding guidelines
d7636bd to
04ad90e
Compare
nachiketb-nvidia
changed the title
Signed-off-by: nachiketb <nachiketb@nvidia.com>
04ad90e to
6ddd961
Compare
nachiketb-nvidia
changed the title
1 participant
What
Enable Anthropic prompt caching by default in
TranslatingLlmClient.Why
Native Anthropic requests require an explicit cache breakpoint to use provider prompt caching.
How
For
Backend::Anthropic, the client marks the final message content block with{"cache_control": {"type": "ephemeral"}}after request translation. Other backend formats are unchanged.What to review
Validation
cargo test -p switchyard-llm-client anthropic_prompt_caching_marks_final_messagecargo clippy -p switchyard-llm-client --all-targets -- -D warnings