Skip to content

OpenAI: sort request body keys so prompt caches can hit - #432

Open
gtokman wants to merge 1 commit into
huggingface:mainfrom
gtokman:sort-request-body-keys
Open

gtokman wants to merge 1 commit into
huggingface:mainfrom
gtokman:sort-request-body-keys

Conversation

@gtokman

@gtokman gtokman commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Requests are encoded with a plain JSONEncoder, so the same tools and messages can come out with their keys in a different order on each request. OpenAI's prompt cache, and Azure's, only match an exact byte prefix. When the order changes, no request ever reads from the cache, and each one is billed as a fresh cache write.

This sorts the keys (.sortedKeys) when encoding request bodies, in all three places they're encoded: Chat Completions, Responses, and streaming. It also adds a test.

What we measured: we ran a 15-turn tool-using benchmark on gpt-6.2-sol through Azure:

  • Before: about 1.5M input tokens were written to the cache and none were read, at $4.05 a run.
  • After: 71–93% of input tokens were read from the cache, at about $0.50 a run.

The key-order differences were easy to see in a diff of two consecutive raw requests: one showed up at character 16 of the system message, and another at character 190 of the tools.

@gtokman
gtokman force-pushed the sort-request-body-keys branch from b133b51 to ca63241 Compare November 19, 2026 12:23
Requests were encoded with a plain JSONEncoder, so the same tools and messages came out
in a different key order each time. Providers whose prompt cache matches an exact prefix
(OpenAI's, Azure's) never hit: every call was billed as a cache write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gtokman
gtokman force-pushed the sort-request-body-keys branch from ca63241 to 8117244 Compare November 19, 2026 12:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant