Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cmd/root/eval.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ import (
"github.com/docker/docker-agent/pkg/telemetry"
)

const defaultJudgeModel = "anthropic/claude-opus-4-5-20251101"
const defaultJudgeModel = "anthropic/claude-opus-5"

type evalFlags struct {
evaluation.Config
Expand Down
2 changes: 1 addition & 1 deletion docs/concepts/models/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ for details.
| Provider | Key | Example Models | API Key Env Var |
| ------------------- | ---------------- | ------------------------------------ | ----------------------------------- |
| OpenAI | `openai` | gpt-5, gpt-5-mini, gpt-4o | `OPENAI_API_KEY` |
| Anthropic | `anthropic` | claude-sonnet-4-5, claude-opus-4-7 | `ANTHROPIC_API_KEY` |
| Anthropic | `anthropic` | claude-sonnet-4-5, claude-opus-5 | `ANTHROPIC_API_KEY` |
| Google | `google` | gemini-3.5-flash, gemini-3-pro | `GOOGLE_API_KEY` / `GEMINI_API_KEY` |
| AWS Bedrock | `amazon-bedrock` | Claude, Nova, Llama models | AWS credentials |
| Docker Model Runner | `dmr` | ai/qwen3, ai/llama3.2 | None (local) |
Expand Down
2 changes: 1 addition & 1 deletion docs/features/cli/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -455,7 +455,7 @@ $ docker agent eval <agent-file>|<registry-ref> [<eval-dir>|./evals] [flags]
| Flag | Default | Description |
| ------------------- | ------------------------------------ | -------------------------------------------------------------------------- |
| `-c, --concurrency` | num CPUs | Number of concurrent evaluation runs |
| `--judge-model` | `anthropic/claude-opus-4-5-20251101` | Model for LLM-as-a-judge relevance scoring (format: `provider/model`) |
| `--judge-model` | `anthropic/claude-opus-5` | Model for LLM-as-a-judge relevance scoring (format: `provider/model`) |
| `--output <dir>` | `<eval-dir>/results` | Directory for results, logs, and session databases |
| `--only <pattern>` | (all) | Only run evals with file names matching these patterns (repeatable) |
| `--base-image` | (default) | Custom base Docker image for eval containers |
Expand Down
2 changes: 1 addition & 1 deletion docs/features/evaluation/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ $ docker agent eval <agent-file>|<registry-ref> [<eval-dir>|./evals]
| Flag | Default | Description |
| ------------------- | --------------------------- | ----------------------------------------------------------------- |
| `-c, --concurrency` | num CPUs | Number of concurrent evaluation runs |
| `--judge-model` | `anthropic/claude-opus-4-5-20251101` | Model for LLM-as-a-judge relevance scoring |
| `--judge-model` | `anthropic/claude-opus-5` | Model for LLM-as-a-judge relevance scoring |
| `--output` | `<eval-dir>/results` | Directory for results, logs, and session databases |
| `--only` | (all) | Only run evals with file names matching these patterns |
| `--base-image` | (default) | Custom base Docker image for eval containers (see [Custom Base Images](#custom-base-images)) |
Expand Down
3 changes: 2 additions & 1 deletion docs/providers/anthropic/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,8 @@ models:

| Model ID | Description |
| ------------------- | --------------------------------------------------- |
| `claude-opus-4-7` | Highest-capability Opus model; supports task budget |
| `claude-opus-5` | Highest-capability Opus model; full effort ladder (low–max) |
| `claude-opus-4-7` | Previous Opus flagship; supports task budget |
| `claude-sonnet-4-5` | Most capable Sonnet; supports extended thinking |
| `claude-sonnet-4-0` | Previous Sonnet generation, still supported |
| `claude-haiku-4-5` | Fast and inexpensive, good for tight loops |
Expand Down
37 changes: 19 additions & 18 deletions pkg/modelinfo/modelinfo.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
//
// Some providers must specialize their behavior depending on the underlying
// model: pick OpenAI's Responses API for o-series and gpt-5, switch Claude
// Opus 4.6/4.7/4.8 to adaptive thinking, use level-based thinking for Gemini 3+,
// Opus 4.6/4.7/4.8/5 to adaptive thinking, use level-based thinking for Gemini 3+,
// auto-enable interleaved thinking for any Claude model regardless of host
// (Anthropic, Bedrock, Vertex AI Model Garden), decide which attachment MIME
// types can be forwarded natively, and so on.
Expand Down Expand Up @@ -113,23 +113,24 @@ func AlwaysReasons(modelID string) bool {
return isOSeries(normalizeOpenAI(modelID))
}

// claudeOpus46To48Prefixes lists the bare Claude Opus model families that
// adaptiveOnlyOpusPrefixes lists the bare Claude Opus model families that
// reject token-based thinking ([RejectsTokenThinking]) and instead require
// adaptive thinking. This is an API behavior quirk that models.dev does not
// describe, so it stays hard-coded here (unlike context windows, which come
// from the catalogue).
var claudeOpus46To48Prefixes = []string{"claude-opus-4-6", "claude-opus-4-7", "claude-opus-4-8"}

// isClaudeOpus46To48 reports whether modelID names a Claude Opus 4.6, 4.7 or
// 4.8 model (or a dated variant like claude-opus-4-7-20251101). Bedrock-style
// identifiers such as "global.anthropic.claude-opus-4-8" are recognised by
// stripping the inference-profile prefix first.
func isClaudeOpus46To48(modelID string) bool {
var adaptiveOnlyOpusPrefixes = []string{"claude-opus-4-6", "claude-opus-4-7", "claude-opus-4-8", "claude-opus-5"}

// isAdaptiveOnlyOpus reports whether modelID names a Claude Opus model that
// rejects token-based thinking (e.g. claude-opus-4-6, claude-opus-5, or a
// dated variant like claude-opus-4-7-20251101). Bedrock-style identifiers
// such as "global.anthropic.claude-opus-5" are recognised by stripping the
// inference-profile prefix first.
func isAdaptiveOnlyOpus(modelID string) bool {
m := normalize(modelID)
if bare, ok := bedrockClaudeModelName(m); ok {
m = bare
}
for _, prefix := range claudeOpus46To48Prefixes {
for _, prefix := range adaptiveOnlyOpusPrefixes {
if m == prefix || strings.HasPrefix(m, prefix+"-") {
return true
}
Expand All @@ -141,15 +142,15 @@ func isClaudeOpus46To48(modelID string) bool {
// `thinking.type=enabled` (token-based extended thinking) and instead requires
// `thinking.type=adaptive`.
//
// Currently Claude Opus 4.6, 4.7 and 4.8 (and dated variants like
// claude-opus-4-7-20251101). Bedrock-style identifiers such as
// "global.anthropic.claude-opus-4-8" are recognised too.
// Applies to Claude Opus 4.6, 4.7, 4.8, 5, and their dated variants (e.g.
// claude-opus-4-7-20251101, claude-opus-5-20260724). Bedrock-style identifiers
// such as "global.anthropic.claude-opus-5" are recognised too.
// For these models the agent transparently switches a token-based budget to
// adaptive thinking.
//
// See https://platform.claude.com/docs/en/build-with-claude/adaptive-thinking
func RejectsTokenThinking(modelID string) bool {
return isClaudeOpus46To48(modelID)
return isAdaptiveOnlyOpus(modelID)
}

// SupportsAdaptiveThinking reports whether an Anthropic Claude model accepts
Expand All @@ -161,10 +162,10 @@ func RejectsTokenThinking(modelID string) bool {
// Claude 3.x) reject `thinking.type=adaptive`/`output_config.effort` with a 400
// and must use token-based extended thinking (`thinking.type=enabled`) instead.
//
// Supported: Opus 4.6/4.7/4.8 (which additionally reject token budgets, see
// [RejectsTokenThinking]), Sonnet 4.6, the Claude 5 families (e.g. Sonnet 5),
// and the codenamed frontier models (Fable, Mythos). Bedrock-style identifiers
// such as "global.anthropic.claude-sonnet-4-6" are recognised too.
// Supported: Opus 4.6/4.7/4.8/5 (which additionally reject token budgets, see
// [RejectsTokenThinking]), Sonnet 4.6, the Claude 5 families (e.g. Sonnet 5,
// Opus 5), and the codenamed frontier models (Fable, Mythos). Bedrock-style
// identifiers such as "global.anthropic.claude-sonnet-4-6" are recognised too.
//
// The set is a superset of [RejectsTokenThinking]: a model that rejects token
// budgets must accept adaptive thinking.
Expand Down
12 changes: 10 additions & 2 deletions pkg/modelinfo/modelinfo_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ func TestSupportsDeferredTools(t *testing.T) {
model string
want bool
}{
{"anthropic", "claude-opus-5", true},
{"anthropic", "claude-opus-4-5", true},
{"anthropic", "claude-opus-4-5-20251101", true},
{"anthropic", "claude-sonnet-4-5", true},
Expand Down Expand Up @@ -300,6 +301,10 @@ func TestRejectsTokenThinking(t *testing.T) {
{"claude-opus-4-60", false}, // must not match
{"claude-opus-4-70", false},
{"claude-opus-4-80", false},
// Claude Opus 5 rejects token-based thinking (adaptive-only generation).
{"claude-opus-5", true},
{"claude-opus-5-20260724", true},
{"global.anthropic.claude-opus-5-20260724-v1:0", true},
{"", false},
// An "openai/" qualifier must NOT be stripped here: normalize() is
// generic (not OpenAI-specific), so a qualified id naming a Claude
Expand Down Expand Up @@ -388,9 +393,11 @@ func TestSupportsAdaptiveThinkingSupersetOfRejects(t *testing.T) {

models := []string{
"claude-opus-4-5", "claude-opus-4-6", "claude-opus-4-7", "claude-opus-4-8",
"claude-opus-4-8-20260601", "claude-sonnet-4-5", "claude-sonnet-4-6",
"claude-opus-4-8-20260601", "claude-opus-5", "claude-opus-5-20260724",
"claude-sonnet-4-5", "claude-sonnet-4-6",
"claude-sonnet-5", "claude-haiku-4-5", "claude-fable-5",
"anthropic.claude-opus-4-8-v1:0", "global.anthropic.claude-opus-4-6-v1:0",
"global.anthropic.claude-opus-5-20260724-v1:0",
}
for _, m := range models {
t.Run(m, func(t *testing.T) {
Expand All @@ -414,10 +421,11 @@ func TestSupportsFullThinkingDisplay(t *testing.T) {
{"claude-sonnet-4-5", true},
{"claude-haiku-4-5", true},
{"claude-opus-4-5", true},
// The adaptive generation (4.6+) only accepts summarized/omitted.
// The adaptive generation (4.6+, and Opus 5) only accepts summarized/omitted.
{"claude-opus-4-6", false},
{"claude-opus-4-7", false},
{"claude-opus-4-8", false},
{"claude-opus-5", false},
{"claude-sonnet-4-6", false},
{"claude-sonnet-5", false},
{"claude-fable-5", false},
Expand Down
6 changes: 6 additions & 0 deletions pkg/modelinfo/thinking_levels.go
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,12 @@ func anthropicTopTierSupport(m string) (hasXHigh, hasMax bool) {
// release adds xhigh while the preview tops out at max.
return !strings.Contains(m, "preview"), true
}
// Claude Opus 5+ ships the full effort ladder (xhigh and max), matching Opus 4.7+.
if strings.Contains(m, "opus") {
if major, _, ok := claudeOpusSonnetVersion(m); ok && major >= 5 {
return true, true
}
}
family, minor, ok := claudeFamilyMinor(m)
if !ok {
return false, false
Expand Down
15 changes: 15 additions & 0 deletions pkg/modelinfo/thinking_levels_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,18 @@ func TestSupportedThinkingLevels(t *testing.T) {
modelID: "anthropic.claude-sonnet-4-5-20250929-v1:0",
want: []effort.Level{effort.None, effort.Low, effort.Medium, effort.High},
},
{
name: "claude opus 5 gets xhigh and max",
provider: "anthropic",
modelID: "claude-opus-5",
want: []effort.Level{effort.None, effort.Low, effort.Medium, effort.High, effort.XHigh, effort.Max},
},
{
name: "bedrock claude opus 5 gets xhigh and max",
provider: "amazon-bedrock",
modelID: "us.anthropic.claude-opus-5-20260724",
want: []effort.Level{effort.None, effort.Low, effort.Medium, effort.High, effort.XHigh, effort.Max},
},
{
name: "claude fable gets xhigh and max",
provider: "anthropic",
Expand Down Expand Up @@ -419,6 +431,9 @@ func TestAnthropicTopEfforts(t *testing.T) {
// Both xhigh and max.
{"claude-opus-4-7", []effort.Level{effort.XHigh, effort.Max}},
{"claude-opus-4-8", []effort.Level{effort.XHigh, effort.Max}},
{"claude-opus-5", []effort.Level{effort.XHigh, effort.Max}},
{"claude-opus-5-20260724", []effort.Level{effort.XHigh, effort.Max}},
{"us.anthropic.claude-opus-5", []effort.Level{effort.XHigh, effort.Max}},
{"claude-fable-5", []effort.Level{effort.XHigh, effort.Max}},
{"claude-mythos-5", []effort.Level{effort.XHigh, effort.Max}},
// Bedrock-style identifiers with regional prefixes: the prefix is
Expand Down
2 changes: 1 addition & 1 deletion pkg/modelsdev/snapshot.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion pkg/modelsdev/snapshot_date.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2026-07-20T07:08:23Z
2026-07-24T19:29:40Z
21 changes: 21 additions & 0 deletions pkg/modelsdev/snapshot_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,27 @@ func TestEmbeddedSnapshotIncludesGPT56Family(t *testing.T) {
}
}

// TestEmbeddedSnapshotIncludesClaudeOpus5 is the regression test for the
// Claude Opus 5 snapshot refresh (released 2026-07-24): it guards against a
// stale embedded snapshot silently dropping offline model resolution /
// pricing / context limits, the same way TestEmbeddedSnapshotIncludesGPT56Family
// spot-checks the gpt-5.6 family.
func TestEmbeddedSnapshotIncludesClaudeOpus5(t *testing.T) {
t.Parallel()

db := embeddedSnapshot()
require.NotNil(t, db)
anthropic, ok := db.Providers["anthropic"]
require.True(t, ok, "embedded snapshot must contain the anthropic provider")

m, ok := anthropic.Models["claude-opus-5"]
require.True(t, ok, "embedded snapshot must contain anthropic/claude-opus-5")
assert.Equal(t, 1000000, m.Limit.Context, "claude-opus-5 must report the 1M context window")
assert.Equal(t, int64(128000), m.Limit.Output, "claude-opus-5 must report the 128k max output")
assert.True(t, m.Reasoning, "claude-opus-5 must be marked as a reasoning model")
assert.Equal(t, "claude-opus", m.Family, "claude-opus-5 must belong to the claude-opus family")
}

// TestEmbeddedSnapshotParses ensures the snapshot baked into the binary is
// valid JSON and carries a non-trivial catalog. A broken snapshot would
// silently degrade every offline lookup, so we guard it at build time.
Expand Down
Loading