fix(providers): normalize OpenAI-compatible cache usage#536
Open
Raymond8196 wants to merge 1 commit into
Open
Conversation
Parse standard prompt-token details, OpenRouter cache writes, and DeepSeek hit/miss fields through one shared normalizer used by non-streaming, streaming, and SSE-reassembly paths. Keep prompt tokens cache-exclusive for ORGII accounting so persisted cache usage, context size, and hit-rate metrics stay consistent. Pre-commit hook ran. Total eslint: 0, total circular: 0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Parse prompt-cache usage through one shared normalizer across every OpenAI-compatible path (non-streaming, streaming, SSE reassembly), so persisted cache usage, context-size, and hit-rate metrics stay consistent regardless of which vendor cache dialect the provider returns.
Why this is needed. The OpenAI-compatible client is shared by OpenAI, Zhipu (GLM), DashScope, Groq, xAI, MiniMax, DeepSeek, and aggregator relays (OpenRouter, etc.). Each reports cache hits in a different shape, but the three execution paths previously only forwarded raw
prompt_tokens/completion_tokens/total_tokens— so cache counters were silently dropped andhit_rateread as a constant 0% on providers that do report cache.Shared normalizer (
Usage::to_usage_map) keys on field schema (not provider id) and resolves three real-world dialects by priority:prompt_tokensincl.prompt_tokens_details.cached_tokensprompt = raw − cache_read, emitscache_readcache_write_tokensprompt = raw − cache_read − cache_write, emitscache_writeprompt_cache_hit_tokens+prompt_cache_miss_tokensprompt, hit →cache_read(missing half derived from inclusive total)cache_read_tokens/cache_write_tokens(prompt excludes cache)Tie-break rule: a populated nested shape wins; a non-empty DeepSeek split wins over an empty
{}nested; already-normalized top-level counters are only used when neither vendor shape is present. All fields accept camelCase aliases. Saturating subtraction guards against malformed counts producing negative prompt.Contract kept cache-exclusive.
prompt_tokensis normalized to uncached/billable input;cache_readandcache_writelive in their own counters. DownstreamUsageTotalsreconstructs the real context-window fill (last_prompt = prompt + cache_read + cache_write) so a fully-cached prefix no longer reports as "2 tokens used", andcache_hit_rate = cache_read / (cache_read + prompt)reflects actual cost savings.Test plan
cargo test -p agent_core --lib providers::openai_compat: 78 passed / 0 failed, covering:cached_tokens, OpenRouter nestedcache_write_tokenscargo clippy -p agent_core --lib: 0 warnings in the 5 changed fileshit_rate95–98% on warm rounds;cache_readgenuinely non-zero (60224 / 58432 / 59968 …), confirming the nestedcached_tokensshape is captured end-to-end.