Skip to content

fix(amazon-bedrock): correct metadata, pricing, and profiles - #6637

Closed
rekram1-node wants to merge 2 commits into
devfrom
feat/bedrock-audit
Closed

fix(amazon-bedrock): correct metadata, pricing, and profiles#6637
rekram1-node wants to merge 2 commits into
devfrom
feat/bedrock-audit

Conversation

@rekram1-node

Copy link
Copy Markdown
Collaborator

Summary

Full audit of all 153 existing Bedrock entries, using five vendor-scoped audits, independent review, AWS/lab documentation, current public AWS price dimensions, and targeted live API checks.

  • 48 existing price records corrected, including 23 Claude source-region price quartets, Nova cache-write accounting, and seven Qwen/Gemma/Voxtral price records.
  • 126 existing entries have resolved metadata or pricing corrections. Additional changes add citations or replace duplicate inline metadata with base_model.
  • 15 missing entries added, bringing Bedrock to 168 entries; eight additions are explicitly marked deprecated/legacy.
  • No existing model IDs removed, API routes changed, or lifecycle statuses changed. The previously discussed questionable bare/prefixed IDs remain.

All changes are catalog TOML. New provider files use complete shared lab metadata and host-specific overrides.

Audit coverage

Scope Existing entries audited
Anthropic 60
Amazon Nova 18
OpenAI 20
Meta / Writer / Mistral 25
DeepSeek / Qwen / xAI / NVIDIA / Google / MiniMax / Z.AI / Moonshot 30
Total 153

Important corrections

Pricing

  • Match Claude geography-specific rates to exact AWS source-region Standard dimensions; do not apply a universal 1.1x multiplier. Opus 4.1 stays at its correct $15/$75; AU Opus 4.6's erroneous $16.50/$82.50 becomes $5.50/$27.50.
  • Nova cache writes are billed at the input rate, with no additional surcharge. Live usage reports cache-write tokens separately from ordinary input tokens, so omitting their price undercounts processing. Add the full input-equivalent cache-write rate and the missing bare Nova 2 Lite cache-read price.
  • Correct Qwen Coder 480B, Coder Next, Next 80B, VL 235B, Gemma 3 12B/27B, and Voxtral Small reference prices.
  • Replace inferred Astra pricing citations with its now-published AWS card, which confirms all existing prices and long-context tiers.

Price comments identify the reference source Region. One scalar catalog cost object cannot represent every eligible source Region's rate. Standard/Global/Geo, 1K-versus-1M units, cache TTL, and service-tier dimensions were checked independently.

Metadata and capabilities

  • Correct Nova 2 Lite, Palmyra X4, Opus 4.5, Mistral Large 3 and other confirmed release metadata; distinguish lab defaults from Bedrock caps.
  • Live boundaries establish Nova Micro/Lite/Pro output 10,000, Nova 2 Lite Bedrock output 65,535 (lab guide remains 65,536), GPT-OSS Runtime output 128,000, and Llama 4 Bedrock output 8,192.
  • All six Sonnet 4.6 Bedrock routes accept 128,000 output settings; US and AU reject 128,001 with an explicit 128,000 maximum. Keep this as a host override rather than changing the native lab default.
  • Correct vision/attachment/open-weight facts and host-specific modalities, including text-only Llama 3.3 and Writer's native-versus-Bedrock differences.
  • Correct missing reasoning controls using actual host fields; verify representative on/off pairs. Gemma 4 uses effort none/high; GPT-OSS/Safeguard and closed OpenAI Runtime models have different wire shapes, documented in the files.
  • Live JSON-schema checks confirm US/Global Astra and Luna support; Sonnet 5 rejects constrained format on US/Global Converse. The Astra correction overlaps fix: Remove structured output restriction for Bedrock GPT-6 Astra #6593.
  • Reverse documentation-only limit corrections when direct API checks contradicted them; acceptance alone is not presented as proof of an absolute ceiling.

Added entries

Current-generation entries

  • in.openai.gpt-5.6-luna, in.openai.gpt-5.6-terra — India profiles; successful live requests from Mumbai.
  • us-gov.openai.gpt-oss-20b-1:0, us-gov.openai.gpt-oss-120b-1:0 — exact AWS-published GovCloud profiles and source-region prices; not live-invoked with commercial credentials.
  • google.gemma-4-31b, google.gemma-4-26b-a4b, google.gemma-4-e2b — live-verified Mantle Responses routes, reasoning and structured JSON.

Legacy entries, explicitly status = "deprecated"

  • us.amazon.nova-premier-v1:0.
  • Sonnet 4: us., eu., apac., global. profiles.
  • Haiku 3: us., eu., apac. profiles.

Legacy invocations can require use within the previous 30 days. Haiku 3 reaches its documented EOL on 2026-09-10; Sonnet 4 on 2026-10-14. These additions record existing profiles for eligible users and are not represented as new active models.

Deferred / unresolved

  • Withdrew tentative Llama 3.2 additions after five profiles returned explicit end-of-life errors. An ACTIVE profile listing alone was insufficient.
  • Deferred new Mantle Chat IDs with client API-selection dependencies, Daybreak variants lacking complete control/cap evidence, and unresolved NVIDIA GovCloud reasoning variants.
  • Existing NVIDIA Nano3/Super3 controls and some exact Mistral/open-model output ceilings remain unresolved. Existing values are retained where sources conflict; the audit is not a claim that every field was freshly live-verified.
  • Existing questionable routes remain for the separately requested removal discussion.

Sources and validation

Each changed model's leading comments cite its primary sources. Principal references:

Checks completed:

  • bun validate passes on the combined final catalog.
  • Generated-catalog comparison against an immutable baseline: 0 removals, 0 existing route changes, 0 existing status changes.
  • Verified saved pricing against 4,448 current AWS product price records, with no differences; independently matched 262 Claude dimensions by exact rate code/region/unit.
  • Reviewed the 39 resolved records on other providers affected by shared lab corrections (dates, Llama 3.3 attachment, Nova limits and inherited metadata).
  • git diff --check passes; changed-path check confirms catalog TOML only.

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Action items

  • [high] [violation] models/mistral/magistral-small-2509.toml:17 - Check: New lab metadata must be complete (limit.context + limit.output, plus required capability fields). Why: Several new models/ files used as base_model targets only set limit.context and omit limit.output (magistral-small-2509, all three ministral-3-*-instruct-2512, both voxtral-*-2507, openai/gpt-oss-safeguard-20b). AGENTS.md requires complete lab entries when hosts inherit them; host-only output overrides leave lab metadata incomplete and make inheritance fragile. Action: Add verified limit.output (and any other missing required lab fields) on every new lab TOML before relying on Bedrock overrides.
  • [high] [possible mistake] providers/amazon-bedrock/models/anthropic.claude-opus-4-6-v1.toml:6 - Check: Bare/unprefixed Bedrock IDs should carry the correct source-region Standard rates, not a mismatched geo. Why: Bare anthropic.claude-opus-4-6-v1 and anthropic.claude-sonnet-4-6 are re-priced to the 1.1× band ($5.50/$27.50 and $3.30/$16.50) with eu-west-2 price-list citations, while the Global twins stay on the $5/$25 and $3/$15 band. That split is internally inconsistent unless the bare IDs are truly EU-only on-demand. Action: Confirm the bare model IDs’ actual source-region Standard dimensions; if they are US/global on-demand, restore US list rates and keep 1.1× only on AU/EU/JP/US geo profiles.
  • [medium] [possible mistake] providers/amazon-bedrock/models/mistral.ministral-3-14b-instruct.toml:1 - Check: Provider files must override lab modalities/attachment when the host surface differs. Why: Switching these entries to base_model = "mistral/ministral-3-*-instruct-2512" and dropping local [modalities]/attachment makes Ministral 14B and 8B inherit lab attachment = true and input = ["text", "image"]. The previous Bedrock definitions were text-only / attachment = false (3B already kept image). Action: If Bedrock Converse for 14B/8B is text-only, restore explicit attachment = false and text-only [modalities] overrides (or document host multimodal support with a first-party Bedrock citation).
  • [medium] [possible mistake] providers/amazon-bedrock/models/google.gemma-3-27b-it.toml:8 - Check: tool_call overrides must match the Bedrock host, not only the lab default. Why: Lab google/gemma-3-*-it sets tool_call = false, but Bedrock keeps/sets tool_call = true on Gemma 3 27B and 4B while 12B inherits false. That split is suspicious without host evidence. Action: Align tool_call across the Gemma 3 Bedrock trio to whatever Converse actually supports, and cite the Bedrock model card/API behavior.
  • [low] [possible mistake] providers/amazon-bedrock/models/deepseek.r1-v1:0.toml:8 - Check: Capability overrides need host evidence when they flip lab defaults. Why: tool_call = false is newly forced on bare and US DeepSeek R1 while lab deepseek/deepseek-r1 is tool_call = true. Comments cover always-on reasoning and output caps, not tools. Action: Verify Bedrock R1 tool calling; keep the override only with a card/API note, otherwise drop it so the lab value inherits.

@rekram1-node

Copy link
Copy Markdown
Collaborator Author

Addressed the five review items in 8ffef3a, with additional primary-source checks and direct Converse probes:

  1. Complete new labs — fixed. All 11 new lab entries now contain every required field, including both context and output. Added the seven missing outputs: Magistral 131,072; Ministral 262,144; Voxtral 32,768; Safeguard 20B 131,072. Headers cite exact native configs/examples. For open-weight models without a separately published completion cap, these are explicitly documented shared-window generation ceilings, not invented hosted-API maxima. Bedrock's smaller output overrides remain separate. Voxtral context was also made precise at 32,768 from native params.json.

  2. Bare Claude pricing — confirmed, retained. Both bare IDs succeed in London. Fresh us-east-1 calls reject both with HTTP 400: “on-demand throughput isn't supported … use … an inference profile.” The London Standard dimensions are:

    • Opus 4.6: $5.50/$27.50, SKUs 49XYAK32S4QDU7VP / K7N8ZERVSHSFXCVC.
    • Sonnet 4.6: $3.30/$16.50, SKUs ZMG85E5S2RJPKJCH / CH7Y686BV8D7WX7K.
    • The cheaper Global twins use separate _Global dimensions. Their price difference is expected, not a US/EU mismatch.
    • London source prices, Opus card, Sonnet card.
  3. Ministral 8B/14B vision — confirmed, retained. Native Converse accepted generated solid-color PNGs and correctly answered Blue (8B) and Yellow (14B), HTTP 200. The 8B and 14B cards affirm Image input and Converse. Added live-evidence comments.

  4. Gemma 3 tools — aligned false across 4B/12B/27B. All three can emit genuine toolUse blocks under forced ANY selection, but all three reject valid native toolResult continuations with a backend role-alternation error. An independent retry adding assistant text also failed. Removed the positive overrides: an agent-facing tool capability should not promise a complete tool-use loop when its result continuation is broken. Comments explicitly preserve the distinction between forced call generation and the failed roundtrip.

  5. DeepSeek R1 tools — false confirmed. us.deepseek.r1-v1:0 explicitly rejects toolConfig with “This model doesn't support tool use.” Confirmed using AUTO and with toolChoice omitted, so this is not merely a forced-choice restriction. Both retained R1 overrides now cite that exact result and the AWS parameter guide.

Combined bun validate, explicit new-lab completeness checks, and git diff --check pass. Generated-catalog comparison still reports zero existing model removals, route changes, or lifecycle-status changes.

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Action items

  • [high] [possible mistake] providers/amazon-bedrock/models/jp.anthropic.claude-opus-5.toml:6 - Check: Regional Claude geo pricing consistency (JP 1.1x vs US/global 1.0x). Why: This PR updates JP Opus 4.7/4.8, Haiku 4.5, Sonnet 4.5, and Sonnet 4.6 to ap-northeast-1 1.1x rates (5.50/27.50, etc.), and US/AU Sonnet 5 / Opus 5 follow the same geo pattern, but JP Opus 5 stays at 5/25 and JP Sonnet 5 stays at 2/10. That understates JP on-demand cost for those two profiles relative to the PR’s own pricing rule. Action: Align jp.anthropic.claude-opus-5 and jp.anthropic.claude-sonnet-5 costs (and cache rates) with the same JP source-region dimensions used for the other JP Claude entries, or document why these two SKUs are intentionally 1.0x.
  • [medium] [possible mistake] models/meta/llama-4-scout-17b-instruct.toml:14 - Check: Host-specific context limits after lab metadata changes. Why: Lab context is raised 3_500_00010_000_000, and Bedrock bare/US Scout files only override limit.output (to 8_192), so resolved Bedrock context becomes 10M. Other hosts already cap Scout far lower (often ~128k–1.3M). The Bedrock edits cite live maxTokens checks, not a 10M context verification. Action: Confirm the AWS Bedrock Scout context window; if Bedrock is below 10M, add an explicit Bedrock [limit].context override on meta.llama4-scout-17b-instruct-v1:0 and us.meta.llama4-scout-17b-instruct-v1:0 (and keep the 10M value only on the lab entry if that is the native Meta figure).
  • [medium] [possible mistake] providers/amazon-bedrock/models/moonshotai.kimi-k2.5.toml (temperature line in diff) - Check: Override-only base_model files must not invent capability deltas. Why: Shared lab models/moonshotai/kimi-k2.5.toml has temperature = false, but the Bedrock entry keeps temperature = true after switching to base_model, with no Bedrock wire/live note for temperature (unlike the reasoning toggle / maxTokens notes). Action: Drop temperature so it inherits false, or keep the override only with a leading comment citing Bedrock acceptance of temperature on this ID.

@rekram1-node

Copy link
Copy Markdown
Collaborator Author

Superseded by smaller, independently validated PRs based on current dev:

The latest reviewer findings are included in these splits: Tokyo Claude 5 Standard pricing is corrected, Scout’s inherited 10M Bedrock context is cited, and Kimi K2.5’s temperature override has live Converse evidence. No existing model removals are included.

@rekram1-node

Copy link
Copy Markdown
Collaborator Author

Correction to the split mapping above: #6791 is Meta / Mistral / Writer, and #6793 is Nova. The other three mappings are unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant