Skip to content

docs(providers): document prompt cache batch 19 - #6817

Open
rekram1-node wants to merge 2 commits into
devfrom
cache-audit-19
Open

docs(providers): document prompt cache batch 19#6817
rekram1-node wants to merge 2 commits into
devfrom
cache-audit-19

Conversation

@rekram1-node

Copy link
Copy Markdown
Collaborator

Batch 19 prompt-cache audit (11 providers). Header-only changes to provider.toml.

provider verdict mechanism
wandb UNKNOWN first-party OpenAI-compatible API; no cache docs or extra-field policy found
watsonx NATIVE_OTHER native wx.ai chat API via watsonx-ai-provider; no chat-level cache params documented
xai NATIVE_OTHER automatic prefix cache; x-grok-conv-id header steers chat hits, prompt_cache_key body key is Responses-only
xiaomi NATIVE_OTHER native cached-input billing (cache_read); no client-facing cache key documented
xiaomi-token-plan-ams NATIVE_OTHER same Xiaomi API family; cached-input billing, no client key
xiaomi-token-plan-cn NATIVE_OTHER same Xiaomi API family; cached-input billing, no client key
xiaomi-token-plan-sgp NATIVE_OTHER same Xiaomi API family; cached-input billing, no client key
xpersona TOLERATES OpenAI-compatible chat endpoint; schema lists model/messages/reasoning only, no documented cache control
zai NATIVE_OTHER automatic implicit context caching via usage.prompt_tokens_details.cached_tokens; no client key
zai-coding-plan NATIVE_OTHER same automatic context caching on coding chat endpoint; plan credits bill cached input
zeldoc UNKNOWN private LLM plus router; no cache params or cached-token fields documented, caching depends on downstream

bun validate passes.

@github-actions

Copy link
Copy Markdown
Contributor

No actionable findings.

@github-actions github-actions Bot added the reviewer: ready Automated review found no actionable items label Sep 11, 2026
@github-actions github-actions Bot removed the reviewer: ready Automated review found no actionable items label Sep 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Action items

  • [medium] [possible mistake] providers/wandb/provider.toml:3 - Check: Provider cache policy must not contradict this provider’s published model costs. Why: The new header says there is no documented prompt cache and to treat usage as uncached, but many providers/wandb/models/** entries already publish non-zero cost.cache_read (e.g. DeepSeek V4 Flash at 0.07). That leaves the catalog internally inconsistent for clients using these comments. Action: Reconcile the policy line with existing cache_read pricing—either rephrase to automatic/undocumented hit billing without a client key (if prices are real) or verify and fix the model costs if cache is not actually billed.
  • [medium] [possible mistake] providers/xpersona/provider.toml:4 - Check: Provider cache policy must not contradict this provider’s published model costs. Why: The new header says no documented cache and to treat usage as uncached, while multiple providers/xpersona/models/** entries define cost.cache_read (e.g. 0.15 on GPT-class models). Action: Reconcile the policy with those cache_read values—acknowledge billed/implicit cache hits if valid, or correct the model costs if cache pricing should not be listed.

@github-actions

Copy link
Copy Markdown
Contributor

Action items

  • [medium] [possible mistake] providers/wandb/provider.toml:1 - Check: Prompt-cache docs must stay consistent with catalog pricing/usage facts. Why: The new header says chat usage has “no cached-token field” and to “treat usage as uncached,” but this provider already publishes non-zero cost.cache_read on many models (e.g. DeepSeek V4 Pro, GPT-OSS, Qwen). That is an internal contradiction: either caching/cached billing exists and the comment overstates UNKNOWN, or the existing cache_read rows are wrong. Action: Reconcile the header with the model costs—acknowledge automatic/cached-input billing (and any usage field) if cache_read is correct, or remove/zero those cache_read values if caching truly is not billed/observable.
  • [medium] [possible mistake] providers/xpersona/provider.toml:1 - Check: Prompt-cache docs must stay consistent with catalog pricing facts. Why: The new header says there is “no documented prompt cache” and to “treat usage as uncached,” while every Xpersona model already has a non-zero cost.cache_read (e.g. gpt-5.4, Claude Sonnet 4.6). Consumers get opposite signals from provider docs vs pricing. Action: Soften or correct the header to match real billing (e.g. automatic cache hits / priced cache_read with no client cache key), or drop the cache_read costs if caching is not actually offered.

@github-actions

Copy link
Copy Markdown
Contributor

Action items

  • [medium] [possible mistake] providers/wandb/provider.toml:1 - Check: Material factual claims in hand-authored provider docs must be consistent with the PR’s stated audit and supported by the cited source. Why: The PR body still classifies wandb as UNKNOWN with “no cache docs or extra-field policy found,” but the final header claims automatic prefix caching, a chat body field cache_salt, and https://docs.wandb.ai/inference/response-settings/prefix-caching. Those cannot both be right; if cache_salt is wrong, clients will send a non-existent control. Action: Verify the W&B prefix-caching docs for chat cache_salt and usage.prompt_tokens_details.cached_tokens, keep the TOML only if accurate, and update the PR body verdict/mechanism to match.
  • [medium] [possible mistake] providers/xai/provider.toml:3 - Check: Prompt-cache header comments must distinguish request controls from response telemetry for this host’s chat API. Why: Final text puts hits under Use body: hits via usage.prompt_tokens_details.cached_tokens, which reads as a request-body cache field. The prior revision correctly said chat has no body cache key and that prompt_cache_key is Responses-only; dropping that can send callers looking for a chat body parameter that does not exist. Action: Restore the chat vs Responses split (headers / no chat body key / response cached_tokens), and only list real request body fields under Use body.
  • [low] [possible mistake] providers/xiaomi/provider.toml:2 (same pattern on xiaomi-token-plan-*, zai, zai-coding-plan) - Check: Use body should document request-body cache controls, not only response usage. Why: These hosts are described as automatic prefix cache with no client cache key, yet Use body only names usage.prompt_tokens_details.cached_tokens (response). That blurs “how to request a hit” vs “how to observe a hit.” Action: Rephrase so request body is empty/none (or omitted under the positive-only rule) and hit detection is clearly response/usage-side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant