Skip to content

fix(providers): normalize OpenAI-compatible cache usage#536

Open
Raymond8196 wants to merge 1 commit into
org2AI:developfrom
Raymond8196:codex/openai-compat-cache-usage
Open

fix(providers): normalize OpenAI-compatible cache usage#536
Raymond8196 wants to merge 1 commit into
org2AI:developfrom
Raymond8196:codex/openai-compat-cache-usage

Conversation

@Raymond8196

Copy link
Copy Markdown
Contributor

Summary

Parse prompt-cache usage through one shared normalizer across every OpenAI-compatible path (non-streaming, streaming, SSE reassembly), so persisted cache usage, context-size, and hit-rate metrics stay consistent regardless of which vendor cache dialect the provider returns.

Why this is needed. The OpenAI-compatible client is shared by OpenAI, Zhipu (GLM), DashScope, Groq, xAI, MiniMax, DeepSeek, and aggregator relays (OpenRouter, etc.). Each reports cache hits in a different shape, but the three execution paths previously only forwarded raw prompt_tokens / completion_tokens / total_tokens — so cache counters were silently dropped and hit_rate read as a constant 0% on providers that do report cache.

Shared normalizer (Usage::to_usage_map) keys on field schema (not provider id) and resolves three real-world dialects by priority:

Upstream shape Example fields Normalized
Standard nested (OpenAI / Zhipu / DashScope / Groq / xAI / MiniMax) prompt_tokens incl. prompt_tokens_details.cached_tokens prompt = raw − cache_read, emits cache_read
OpenRouter cache write nested cache_write_tokens prompt = raw − cache_read − cache_write, emits cache_write
DeepSeek hit/miss split prompt_cache_hit_tokens + prompt_cache_miss_tokens miss → prompt, hit → cache_read (missing half derived from inclusive total)
Already-normalized relay top-level cache_read_tokens / cache_write_tokens (prompt excludes cache) passed through, no subtraction

Tie-break rule: a populated nested shape wins; a non-empty DeepSeek split wins over an empty {} nested; already-normalized top-level counters are only used when neither vendor shape is present. All fields accept camelCase aliases. Saturating subtraction guards against malformed counts producing negative prompt.

Contract kept cache-exclusive. prompt_tokens is normalized to uncached/billable input; cache_read and cache_write live in their own counters. Downstream UsageTotals reconstructs the real context-window fill (last_prompt = prompt + cache_read + cache_write) so a fully-cached prefix no longer reports as "2 tokens used", and cache_hit_rate = cache_read / (cache_read + prompt) reflects actual cost savings.

Test plan

  • cargo test -p agent_core --lib providers::openai_compat: 78 passed / 0 failed, covering:
    • standard nested cached_tokens, OpenRouter nested cache_write_tokens
    • DeepSeek hit+miss split, hit-only / miss-only derivation from inclusive total, empty-nested-vs-split tie-break
    • camelCase aliases, plain usage (no cache fields), top-level already-normalized counters
    • malformed nested count saturates prompt to 0 (never negative)
    • end-to-end wiremock: OpenAI non-streaming, DeepSeek streaming, SSE reassembly
  • cargo clippy -p agent_core --lib: 0 warnings in the 5 changed files
  • Production-verified on live GLM-5.2 traffic: per-call hit_rate 95–98% on warm rounds; cache_read genuinely non-zero (60224 / 58432 / 59968 …), confirming the nested cached_tokens shape is captured end-to-end.

Parse standard prompt-token details, OpenRouter cache writes, and DeepSeek hit/miss fields through one shared normalizer used by non-streaming, streaming, and SSE-reassembly paths.

Keep prompt tokens cache-exclusive for ORGII accounting so persisted cache usage, context size, and hit-rate metrics stay consistent.

Pre-commit hook ran. Total eslint: 0, total circular: 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant