Add Cocoon Stack Gateway to the AI gateway benchmarks - #409
Open
CMGS wants to merge 1 commit into
Open
Conversation
Adds `cocoonstack-gateway` (github.com/cocoonstack/gateway, served from gw-bench.simular.cloud in us-east4) to all four families: - Anthropic: native `/v1/messages`, claude-haiku-4-5-20251001 - OpenAI: `/v1/responses`, gpt-5.4-mini - Gemini: OpenAI-compatible `/v1/chat/completions`, gemini-3.6-flash (the gateway translates to Gemini's native API; there is no native Gemini inbound) - Kimi: `/v1/chat/completions`, kimi-k3 on Moonshot's international endpoint; `temperature` omitted (kimi-k3 accepts only 1) and `reasoningCountsAsFirstToken: true` (the stream carries `reasoning_content`) One credential, COCOONSTACK_GATEWAY_API_KEY, wired through .env.example, provider-vars.json and the workflow's vault list; a missing key reports SKIPPED as usual. Confirmed live with this repo's runner from us-east4: `--iterations 50` on all four families, 800/800 probes succeeded, and every wire ends with a usage frame so tokens/sec resolves.
Contributor License AgreementAll contributors are covered by a CLA. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Cocoon Stack Gateway (
cocoonstack-gateway, github.com/cocoonstack/gateway) to all four AI-gateway families. The participant is served fromgw-bench.simular.cloud(us-east4).anthropic/v1/messagesclaude-haiku-4-5-20251001responses/v1/responsesgpt-5.4-miniopenai/v1/chat/completionsgemini-3.6-flashopenai/v1/chat/completionskimi-k3Notes on the entries:
streamGenerateContent; there is no native Gemini inbound, same as the other translation-layer participants in that family. Thinking tokens are reported undercompletion_tokens_details.reasoning_tokensand folded intocompletion_tokens.temperature(kimi-k3 accepts only 1, as documented at the top ofproviders-kimi.ts) and setsreasoningCountsAsFirstToken: true; the stream carriesreasoning_contentin Moonshot's own shape.stream_options.include_usageis honored on the chat wire; Responses and Messages report usage natively), so tokens/sec resolves for all four families.Credential: a single env var,
COCOONSTACK_GATEWAY_API_KEY(a bearer /x-api-keyvalue issued by us). It is wired through.env.example,provider-vars.jsonand the workflow's vault list; a missing key reports the participant as SKIPPED as usual. We will send the key to the maintainers out of band.Verification:
pnpm typecheckclean; this repo's runner from a us-east4 client with--iterations 50(50 cold + 50 warm) on all four families, interleaved with the family's direct baseline, 800/800 probes succeeded (--no-ingest, results not included here).