fix(pricing): carry long-context tier rates through bundling, applied where billing evidence exists - #1478
fix(pricing): carry long-context tier rates through bundling, applied where billing evidence exists#1478ozymandiashh wants to merge 4 commits into
Conversation
iamtoruk
left a comment
iamtoruk
left a comment
There was a problem hiding this comment.
Reviewed against the real corpus. The mechanism is right, but two things block it, and both move money.
Blocker 1: audit-report.ts:143 tiers whole buckets instead of calls
tieredCostsFor(bucket.model, rates, displayed.inputTokens + displayed.cacheReadTokens, bucket.provider) feeds the summed tokens of a (provider, model) bucket into a per-call threshold. Buckets aggregate every call for that pair, so any tiered codex model clears 272k on the sum alone and gets tiered wholesale.
Measured, audit --provider codex --period lifetime --to 2026-09-17:
| recomputed | attributed | |
|---|---|---|
| main | $5,226.99 | $5,226.99 |
| this branch | $10,206.94 | $5,228.00 |
That is a $4,978.94 phantom gap where main has none. Per model on the branch: gpt-5.5 $4,090.56 to $7,964.25, codex-auto-review $775.43 to $1,543.89, gpt-5.6-sol $348.56 to $676.67.
The PR body says this hunk keeps recomputedTotalUSD tracking attributedCostUSD. It does the opposite. The (c4) test does not catch it because copilot is not in TIERED_PRICING_PROVIDERS, so the invariant is only exercised on the untiered provider.
Fix: accumulate the tiered cost per call while the bucket is being built. The loop already walks every call, so the per-call prompt total is right there. Do not re-derive it from bucket sums.
Blocker 2: the tier never reaches warm history
DAILY_CACHE_VERSION stays 33 and MIN_SUPPORTED_VERSION stays 28. On my real warm cache, codex lifetime is $5,889.57 on main and $5,889.57 on this branch. Identical to twelve decimal places. The tier changes nothing for anyone who upgrades with a warm cache.
That is not a small window. 6,487 codex calls, roughly $663, exist only in the daily cache. Their sources have aged out and nothing else can re-derive them.
Every prior pricing fix bumped it: v23 for #1075, v24 for #1077, v29 for #1118. This one needs 34 with MIN_SUPPORTED_VERSION raised to match.
The session cache is fine. cachedCallToApiCall passes call.provider, so fresh parse and cache read agree. I measured +$1.012964 with the session cache warm, with it wiped, and with only codex-results warm. Same number all three times. No CODEX_CACHE_VERSION or parse-version bump needed.
Split the snapshot refresh out
Commit efb4e03 is already the refresh on its own. Please land it as its own PR first. The mechanism is inert without its tier data, so it has to go first anyway, and it moves money that has nothing to do with #1076.
The refresh is the only thing that moves a repriced model on my data. MiniMaxAI/MiniMax-M3 loses its published cache-read rate of 6e-8, so buildCosts falls back to input * 0.1 and prices it at 3e-8. Lifetime delta is -$2.99 across 1,574 calls. The cause is upstream adding nebius/MiniMaxAI/MiniMax-M3, which carries no cache-read price, and the bundler's pass 2 handing the stripped alias to it on first write wins. A published rate quietly became an inferred one.
The refresh also takes pricing-fallback.json from 203 entries to 51, which the PR body does not mention. 146 of the 154 dropped are now in the primary snapshot, which is fine. Nine models lose pricing entirely: qwen3.5-plus, mercury-2.5-preview, inkling-small:batch, nex-n2-mini, nex-n2-pro, gpt-latest, gemma-4-31b-it:batch, gpt-oss-20b:batch, and grok-latest. That last one is why tests/grok-parser-pipeline.test.ts had to be rewritten. None of the nine are in my data today, but that is luck, not design.
One more wording fix. Only 4 of the "11 removed" are actually gone upstream, the two friendliai llama-3.1 entries in their bare and prefixed forms. The other 7 are twelvelabs embedding and gpt-image-2 rows that are still published. The bundler now filters them for lacking a per-token output cost. That is a filter rule change, not an upstream deletion, and the changelog should say so.
Three corrections to the numbers
The tier delta on my corpus is +$1.012964 on 8 of 51,801 codex calls, not the +$1.28 on 45,638 from the issue. The 8 are all gpt-5.6-sol on 2026-07-12 with prompts from 276,331 to 288,675. Please re-measure rather than quoting the issue, the base and tier rates both moved in the refresh.
The Copilot counterexample is 1.91x, not 2x. Input and cache read double, but output only goes 1.2e-5 to 1.8e-5. The (c4) session prices at $4.4687 base and $8.5573 tiered. The point stands and it is good evidence, the multiplier is just wrong in the description.
Parity between fresh parse and cache read is confirmed, as above. Worth saying so in the PR body since it is the thing that usually goes wrong here.
Nits
tests/models.test.ts tightens the kimi assertion from toBeCloseTo(8, 5) to toBeCloseTo(8, 9). Unrelated to this change, drop it or say why.
The comment on tieredCostsFor has an orphaned fragment: "the override row is rebuilt without one, exact or aliased. The". Looks like a bad rewrap.
In audit-report.ts, webSearch: tiered ? ... reads as if web search participates in the tier. It never does. Leave it on rates.
Otherwise
The mechanism is sound and I checked it properly. Threshold parsed from the suffix, largest threshold wins, _priority and _flex correctly excluded, prompt tokens as input plus cached input. I hand-computed gpt-5.6 at 272,001 prompt tokens and got $2.176008 against a base of $1.088004, and the tests pin 271,999 and 272,000 on the right sides of the line. Snapshot counts all check out exactly: 5,938 entries, 1,210 added, 11 removed, 298 repriced, 394 tier bearing at 184/170/26/12/2. I compared three repriced models against upstream LiteLLM today, including their tier fields, and all three match. tsc is clean, the suite is green at 4,277, locks are 36, and the app typechecks.
Fix the two blockers, split the refresh, correct the figures, and this is good to go.
af3de52 to
e5669f6
Compare
|
Both blockers fixed, the refresh split into #1483 (landing first, stacked), and the numbers re-measured rather than quoted. Blocker 1: the audit recompute is now per call — the bucket loop accumulates each call's components through the same tier swap Blocker 2: #1483 carries the refresh alone plus the two rules it needed: completeness-wins for prefixed aliases (MiniMax cache-read restored to 6e-8), and carry-forward of previously-priced fallback entries — exactly the nine you listed; qwen3.5-plus needed the date-stripped-non-coverage distinction ( Numbers: tier-only on my corpus is +$35.43 lifetime codex (branch vs the refresh branch, same data) — this machine runs many more sessions past 272k than your 8-of-51,801; both measurements are real. Copilot corrected to 1.91×. Parity restated in the PR body. The kimi tightening is reverted, the comment rewrapped. One residual: |
|
Thanks for this. The tier mechanism itself is right: double-gating on the provider being codex plus the prompt-token threshold, keeping below-threshold and non-codex pricing byte-identical, reading rates from the largest threshold, letting an exact override still win, and the parser mirroring the bundler regex. The tests are real and cover the cases that matter, and the cost-calc and parsing changes merge cleanly into current main. Issue #1076 is genuinely still unfixed on main, so this is needed. The blocker is the bundler and data, not the mechanism. This branch is stacked on an older version of the snapshot refresh that has since been hardened on main (#1483). The two chore(pricing) commits in this branch revert that hardening: they bring back completeness-wins and drop the previous-snapshot carry-forward. As a result the branch's snapshot reprices base rates that main deliberately keeps, for example grok-3 goes $3/$15 to $1.25/$2.5 per million and mistral-large-latest $8/$24 to $0.5/$1.5. Merging as-is, or resolving the conflict by taking this side, would silently undo #1483 for ~43 models. To land it:
The engineering is sound; it just needs to sit on the current base with regenerated data. Ping me once it's rebased and I'll take another pass. |
…e them where billing evidence exists LiteLLM publishes *_above_<n>k_tokens rates for 418 models - OpenAI's gpt-5.4/5.5/5.6 family above 272k, Anthropic above 200k, and 128k/256k/ 512k tiers elsewhere - but the bundler flattened every entry to the fixed five-slot tuple and calculateCost had no threshold dimension, so a request past its threshold priced at the base rate forever (getagentseal#1076). The tuple gains an optional sixth slot {threshold, input, output, cacheWrite, cacheRead}, the threshold parsed from the key suffix because LiteLLM ships no numeric field (272k means 272,000; getagentseal#1075 verified assuming 128k fabricates +64% spend). Old bundles parse unchanged; CACHE_SCHEMA_VERSION moves to 4 so a stale on-disk live-pricing cache cannot reintroduce the old behavior; the live-fetch parser picks the same tier up; service-tier _priority/_flex variants and the 1-hour cache-write combination are not context thresholds and stay out. A tier applies when the request's prompt tokens (input + cached input, the existing grok-4.6 definition) reach the threshold, each published rate replacing its base and omitted slots keeping the base. Application is gated on billing evidence via TIERED_PRICING_PROVIDERS: codex today, where the maintainer's corpus measurement shows the surcharge passing through; a real Copilot session on gpt-5.6-terra with ~6M-token prompts billed at the BASE rate, so Copilot keeps base pricing until such evidence exists - applying it there would fabricate spend, the exact class getagentseal#1075 warned about. The audit report recomputes through the same tier swap so recomputedTotalUSD keeps tracking attributedCostUSD.
…ex tier test Repair pass from review: both tierOf implementations now collect rates per threshold and read only from the largest one, so a hypothetical two-tier entry can never mix the smaller tier's rates under the bigger threshold; the bundler validates tier values exactly like the live parser (finite, non-negative). A codex end-to-end test drives a real rollout past gpt-5.5's 272k prompt threshold and asserts the tiered arithmetic explicitly (a typo'd provider string in codex.ts would leave the call at base rates and fail it), alongside the no-provider base-rate control that keeps the real Copilot billing pin intact. Comment fixes: the priceOverride wording, the grok pin's actual resolution branch, and the threading note on TIERED_PRICING_PROVIDERS for whoever adds the next provider.
…ily caches Both review blockers: - audit-report no longer feeds a bucket's SUMMED tokens into the per-call 272k threshold (a \,979 phantom gap on the reviewer's corpus, any tiered codex model clearing the threshold on the sum alone). The recompute accumulates per call, through the same tier swap calculateCost applies, with each call's own prompt total; a new fixture mixes one 300k call with 200 small ones whose sum blows past the threshold and pins that only the big call tiers. Web search stays on the base rates - it never participates in a tier. - DAILY_CACHE_VERSION 33 -> 34 with MIN_SUPPORTED_VERSION raised to match, the cfdaa68 convention: a 33 file written under tierless pricing is below the floor, is not adopted as the finalized base, and its days re-derive off the warm session cache (\ of codex spend on the reviewer's corpus lives only in the daily cache). The migration tests pin the new floor contract: no version-diff repairs merge into a below-floor file, and the v30 adoption test tracks CURRENT. Nits from review: the kimi precision tightening is reverted to (8, 5), and the tieredCostsFor comment is rewrapped. Measured on this machine's corpus, tier-only (branch vs the refresh branch, same data): +\.43 lifetime codex - this corpus runs far more sessions past 272k than the reviewer's 8-of-51,801 (+\.012964); both numbers are real, the corpora differ.
e5669f6 to
be98132
Compare
…TL and fast-mode fixes Takes the in-flight upstream implementation of OpenAI's long-context pricing (getagentseal#1478, fixes getagentseal#1076) over the one added here, and drops ours. Theirs is strictly better on every axis: - The tier is data-driven, parsed from LiteLLM's own `*_above_<n>k_tokens` keys into a 6th snapshot slot, so it covers 401 tier-bearing models and picks up Anthropic's above-200k rows too. Ours hardcoded 8 OpenAI models and their thresholds by hand from the vendor's pricing page. - It solves the per-request/aggregate trap properly. Ours gated on a `PromptScope` the CALLER had to assert, which meant nothing could safely be wired up, so the tier shipped inert. Theirs gates on TIERED_PRICING_PROVIDERS - a provider allowlist backed by billing evidence - and threads the provider through the codex call sites and the parser recompute, so it actually applies. Both of us independently hit the same Copilot rollup (tests/parser.test.ts "(c4)"), where a 12.6M-token session SUM is not a long-context request; their note records it as the reason the allowlist exists. Kept from ours, because getagentseal#1478 leaves both open by name ("Fast-mode and the 1-hour cache-write rate remain visible gaps on purpose"): - Cache writes split by TTL. Anthropic bills a 1-hour write at 2x base input and a 5-minute write at 1.25x, and Claude Code writes mostly 1h, so collapsing them understated the audit recompute on every Claude corpus (Opus 5 read $305.29 against $327.86 attributed). Rendered as two columns. - The fast-mode multiplier, applied per call. With both folded into their per-call recompute, recomputedTotalUSD reconciles to attributedCostUSD exactly - verified to floating point on every row of a real 30-day corpus across claude, codex and opencode. Our tier tests are dropped as redundant; getagentseal#1478 covers the same ground data-driven under "long-context tiers (getagentseal#1076)". The published cache read/write rate pins stay, including Fable 5.1's 0.025x cache-hit rate. Root suite: 4,452 passed, 5 skipped, 0 failed. Typecheck clean.
2 participants
Fixes #1076.
Codex calls at a model's published long-context threshold now use its tier rates. Below-threshold calls and other providers keep base pricing; explicit overrides still win. The audit recomputes each call independently, and daily-cache version/minimum 34 forces warm history to re-derive. The live-pricing cache schema advances to 4.
Rebased onto
mainat4cf18855, dropping the two embedded snapshot-refresh commits (434d73d6,882284af). This PR is no longer stacked on #1483.The bundler retains #1483's input/output/cache
fillsOnlyguards, previous-primary plus previous-fallback carry-forward, and exact-key-only coverage. Tier presence is excluded from base-row completeness and replacement guards: slot 5 was previously absent, but became an object whose reference comparison blocked six legitimate cache-slot fills. The selected source row now supplies both its base rates and its tier.Regenerated from captured live LiteLLM, models.dev and OpenRouter responses: 5,946 primary entries, 401 tier-bearing, 212 fallback. Fallback regeneration is byte-identical to main. Replaying main's bundler and this bundler against the same responses produces zero differences in the first five slots for every primary and fallback entry, and no previously priced key disappears.
The live source has moved since main's committed snapshot: four primary keys were added and 19 primary tuples changed upstream. Those exact changes also occur with the unchanged main bundler; none comes from tier-based alias selection. No primary key was removed. The changelog entry for #1076 is restored.
Validation:
codex-auto-reviewparses to $4.045; the base-rate control is $2.03. Audit fixture with one 300k call and 200 small calls reconciles attributed and recomputed totals at $3.27.