diff --git a/README.md b/README.md index e6c7fcf..afc2070 100644 --- a/README.md +++ b/README.md @@ -127,7 +127,7 @@ go test ./router/ -bench=. -benchmem This is a Go port of [BlockRunAI/ClawRouter](https://github.com/BlockRunAI/ClawRouter). Applicable routing, proxy and spending-control changes are ported from upstream source snapshots. OpenClaw/desktop lifecycle and provider-specific account services are excluded; x402 signing remains unimplemented. See [UPSTREAM_SYNC.md](UPSTREAM_SYNC.md) for details. -**Current sync**: v0.12.279 release (`becb296`, checked 2026-09-22). No new Go-applicable core changes since the previous snapshot; see the tracker for adapted and deferred features. +**Current sync**: ClawRouter 0.12.297 source (`78e489a`, checked 2026-10-22; latest published release: v0.12.294). Includes updated chat metadata, gateway catalog refresh, bounded outcome memory, request-scoring fixes and local proxy protections. See the tracker for adaptations and exclusions. ## License diff --git a/UPSTREAM_SYNC.md b/UPSTREAM_SYNC.md index 579d8e8..94d4e56 100644 --- a/UPSTREAM_SYNC.md +++ b/UPSTREAM_SYNC.md @@ -2,7 +2,7 @@ **Upstream**: [BlockRunAI/ClawRouter](https://github.com/BlockRunAI/ClawRouter) (TypeScript) **This repo**: [DOS/DOSRouter](https://github.com/DOS/DOSRouter) (Go port) -**Last synced**: v0.12.279 release (`becb2968711bb203a1498958de27a2b1be3780c3`, checked 2026-09-22; no new Go-applicable core changes since `05de1e0`, exclusions below) +**Last synced**: 0.12.297 source (`78e489a`, checked 2026-10-22; latest published release/tag is v0.12.294, source-only changes through 0.12.297 included; adaptations and exclusions below) ## Sync Workflow @@ -30,6 +30,66 @@ These upstream areas are excluded (TS/npm-specific): ## Sync Log +### 2026-10-22 - Sync through 0.12.297 source + +Owner: Codex, branch `codex/sync-clawrouter-20261022`. Compared the previous +release commit `becb2968711bb203a1498958de27a2b1be3780c3` with upstream main +`78e489a` (package version 0.12.297). GitHub's latest published release and +tag at capture time is v0.12.294; 0.12.295-297 are source versions, not newly +published GitHub releases. The sync includes the subsequent retired-Surf cleanup. + +Pinned dependency sources: router-core +`8773a9954b091bd0090acdfec67c75a006c0e156` and model-catalog +`b95efe2f025c9968742e6479bc9ec71b6a40c0f4` (catalog 2026.10.20.4). + +| Upstream change | Go disposition | +| --- | --- | +| v0.12.280-281 / 294-297 chat catalog | Add 17 chat records and 34 aliases, correct Sonnet 5 and DeepSeek Vision prices, carry Mistral Large 4's list price, remove Nano 30B from the free fallback lineup, and mark Hy3/retired GPT/Grok IDs. Preserve DOS-specific aliases, exact Nano 30B pins and retired free-to-free redirects. Bare `deepseek` follows V4.1 Flash; free V4 Flash aliases remain free. | +| v0.12.287 gateway catalog | Each serving proxy refreshes only its configured gateway's `/v1/models?format=json` at startup and every five minutes. Validate complete updates; preserve the last good snapshot after failures; refuse redirects and free-product billing changes. New chat models are explicit picks, with no inferred tool eligibility or automatic tier promotion. Each request retains its admission-price snapshot through settlement. No remote ordering policy is fetched. | +| v0.12.281 outcome memory | Per-server, mutex-protected last 10 outcomes for each model/chat-or-tools pair, 30-minute expiry, 512 keys. Three failures at a 50% rate move a model behind reliable candidates before the five-attempt cap. Preserve explicit pins and all-unreliable ordering. Count server failures, incomplete bodies, empty turns, attempt timeouts and observed SSE outcomes; ignore caller cancellation and payment errors. Empty non-streaming turns use the full bounded chain. | +| v0.12.291-297 verified tools | Add the 13 probe-verified recovery candidates behind configured eligible models, ordered by estimated token price outside Premium. Preserve proven capability metadata; catalog categories cannot grant tool support. All 16 configured tier chains retain the previous primaries. | +| v0.12.292 scoring | Size CJK characters at roughly one token each, bound intent scans to both ends of 8,000 characters, match word boundaries/inflections instead of unrelated substrings, use explicit `response_format` for structured output, and stop claiming savings for unknown prices. | +| v0.12.282 local trust | Bind the CLI server to `127.0.0.1`; reject non-local Host, cross-site/opaque Origin and cross-site browser requests on every route. `Server.Handler()` applies the same HTTP guards for embedders. | +| v0.12.283 strict spend reservations | Existing Go reservations already account atomically for in-flight attempts and conservatively settle lost/5xx responses. Keep that implementation. Gateway claims of zero pricing outside local `free/*` products cannot bypass configured limits. Quote-before-sign accounting is inapplicable while x402 signing remains unimplemented. | +| v0.12.284 / 286, media, host integration | Skip account-ledger reconciliation, Solana batch settlement, local img2img filesystem/URL readers, paid OpenClaw sender commands, frozen OpenClaw configuration and Desktop lifecycle: these surfaces do not exist in this Go runtime. Existing image passthrough and Pyth-backed partners remain unchanged. | +| npm/Node, brand, picker, Surf | Skip TS dependencies/build/publishing, Desktop UI, brand counters and picker-only changes. DOSRouter has no Surf integration to remove. Retain the existing Go CI/security workflows and independent automated review. | + +Intentional limitations: the existing Go rules strategy remains in place; the +optional TypeScript portfolio strategy, empirical speed-probe decay and its +task-affinity model are not introduced by this sync. Go does not yet identify +repetitive-output loops as a separate failure signal. No x402 signer, ledger +reconciliation or new paid media/provider contract is enabled. + +Prices and availability in the bundled snapshot are upstream source metadata, +not live DOS provider verification. Gemini 4 Argon's bundled $2/$10 promotion +ends after 2026-11-30 (then $4/$20); offline date-based repricing is not +implemented. Successful catalog refreshes use the configured gateway's rates. +The existing Gemini Flash promotion caveat below still applies offline. + +Validation: regression tests, `go test ./...`, `go vet ./...` and +`go build ./...` passed locally. Native Windows has no cgo compiler, so the +race run is covered by the existing Linux CI job. CI race tests, vulnerability +scan, CodeQL, final-head review and merge acceptance are recorded on the sync +PR. Scope is DOSRouter source and documentation only; no deployment, paid +provider request, credential change or cross-repository write is included. + +CI unblock: the first scanner run on `35a4b8c` found 10 reachable standard +library advisories in Go 1.26.6, including GO-2026-6809 (fixed in 1.26.10) +and the Go 1.26.9 HTTP/TLS fixes. No open PR owns this module change. The +smallest unblock raises only the preferred toolchain to Go 1.26.10; the Go +language minimum and module dependencies stay unchanged. Re-run the existing +security, build/race and CodeQL jobs on the resulting head before merge. +The same govulncheck command then passed locally with zero reachable +vulnerabilities (18 module-level advisories remain in code not called here). +Review hardening: bound non-streaming chat bodies to 64 MiB and error bodies +to 1 MiB. Incomplete/oversized bodies stop fallback and preserve accounting; +boundary tests and the existing truncated-response settlement regression pass. +The completed baseline CodeRabbit review identified three Go adaptation edges: +retarget bare Tencent/Hy3/Hunyuan aliases directly to the declared Qwen +successor (keep exact `tencent/hy3` pins), reject remote zero flat-price claims +under spend limits, and use Go's 500-character scan default for omitted or +negative limits. Each issue has a failing-then-passing regression test. + ### 2026-09-22 - Sync to the v0.12.279 release Verified the upstream release list, fetched tags, and compared the previous diff --git a/docs/configuration.md b/docs/configuration.md index ee27118..7ea2259 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -2,6 +2,23 @@ Complete reference for DOSRouter configuration options. +The standalone server listens on `127.0.0.1`. All routes reject non-local +Host headers, cross-site or opaque Origin headers, and cross-site browser +requests. Native local clients can omit Origin. Embedded servers should bind +to loopback and use `Server.Handler()` to retain these HTTP protections. + +While serving, model metadata refreshes from the configured upstream's +`/v1/models?format=json` at startup and every five minutes, using only the +configured upstream API key. A failed or malformed refresh keeps the last good +snapshot and logs a generic warning. No other catalog host is contacted. +Aliases, free-product identity, proven tool support and automatic routing +policy remain defined by DOSRouter source. Newly discovered chat models are +available by explicit ID; remote zero-price claims cannot bypass spend limits. + +Non-streaming chat responses are bounded to 64 MiB, and upstream error bodies +to 1 MiB. An oversized or incomplete response fails without repeating the +send; accounting conservatively retains the known charge or reservation. + ## Table of Contents - [Environment Variables](#environment-variables) diff --git a/go.mod b/go.mod index 48e19f8..fc9ceea 100644 --- a/go.mod +++ b/go.mod @@ -2,7 +2,7 @@ module github.com/DOS/DOSRouter go 1.25.0 -toolchain go1.26.6 +toolchain go1.26.10 require ( golang.org/x/crypto v0.50.0 // indirect diff --git a/models/catalog_test.go b/models/catalog_test.go index 0bdaf2e..1d55335 100644 --- a/models/catalog_test.go +++ b/models/catalog_test.go @@ -22,12 +22,16 @@ func TestSyncedCatalogEconomicsAndLimits(t *testing.T) { {"openai/chat-latest", 5, 30, 128000, 128000}, {"anthropic/claude-fable-5", 10, 50, 1000000, 128000}, {"anthropic/claude-sonnet-4.6", 3, 15, 1000000, 128000}, - {"anthropic/claude-sonnet-5", 3, 15, 1000000, 128000}, + {"anthropic/claude-sonnet-5", 2, 10, 1000000, 128000}, {"moonshot/kimi-k3", 3, 15, 1048576, 65536}, {"google/gemini-3.8-flash", 0.75, 3.75, 1048576, 65536}, {"google/gemini-3.6-flash", 0.75, 3.75, 1048576, 65536}, {"deepseek/deepseek-v4-pro", 1.32, 3.96, 1048576, 65536}, - {"deepseek/deepseek-v4-flash-vision-exp", 0.44, 1.32, 1048576, 65536}, + {"deepseek/deepseek-v4-flash-vision-exp", 0.30, 1.20, 1048576, 65536}, + {"openai/gpt-6.1-sol", 2, 10, 1050000, 128000}, + {"deepseek/deepseek-v4.1-flash", 0.14, 0.28, 1048576, 65536}, + {"mistral/mistral-large-4", 1.36, 4.18, 524288, 262144}, + {"google/gemini-4-argon", 2, 10, 1048576, 65536}, {"qwen/qwen3.8-flash", 0.15, 0.47, 1000000, 131072}, {"xiaomi/mimo-v2.5", 0.14, 0.28, 1048576, 131072}, {"zai/glm-5.3", 1.4, 4.4, 1000000, 131072}, @@ -49,6 +53,18 @@ func TestSyncedCatalogEconomicsAndLimits(t *testing.T) { } } +func TestRetiredTencentShorthandsResolveActiveMetadata(t *testing.T) { + for _, alias := range []string{"hunyuan", "hy3", "tencent"} { + m := GetModel(ResolveModelAlias(alias)) + if m == nil || m.Deprecated { + t.Errorf("%s still targets retired metadata", alias) + } + } + if ResolveModelAlias("tencent/hy3") != "tencent/hy3" { + t.Fatal("explicit compatibility pin changed") + } +} + func TestCatalogIdentityAndRetirementInvariants(t *testing.T) { seen := make(map[string]bool) for _, m := range Models { diff --git a/models/models.go b/models/models.go index 926b218..d690ee6 100644 --- a/models/models.go +++ b/models/models.go @@ -36,6 +36,40 @@ type PromoDef struct { // ModelAliases maps shorthand names to full model IDs. var ModelAliases = map[string]string{ + "muse-glimmer-30b": "free/muse-glimmer-30b", + "muse-glimmer": "free/muse-glimmer-30b", + "muse": "free/muse-glimmer-30b", + "nvidia/muse-glimmer-30b": "free/muse-glimmer-30b", + "laguna-s-2.1": "poolside/laguna-s-2.1", + "laguna-s": "poolside/laguna-s-2.1", + "mistral-large-4": "mistral/mistral-large-4", + "grok-4-6": "xai/grok-4.6", + "grok-4.6": "xai/grok-4.6", + "grok-4-7": "xai/grok-4.7", + "grok-4.7": "xai/grok-4.7", + "hy4-preview": "tencent/hy4-preview", + "hy4": "tencent/hy4-preview", + "qwen3-8-max": "qwen/qwen3.8-max", + "qwen-3.8-max": "qwen/qwen3.8-max", + "qwen3.8-max": "qwen/qwen3.8-max", + "v4.1-flash": "deepseek/deepseek-v4.1-flash", + "deepseek-v4-1-flash": "deepseek/deepseek-v4.1-flash", + "deepseek-v4.1-flash": "deepseek/deepseek-v4.1-flash", + "gpt-5.1": "openai/gpt-5.1", + "gpt-6-1-sol": "openai/gpt-6.1-sol", + "gpt-6.1-sol": "openai/gpt-6.1-sol", + "gpt-6-luna": "openai/gpt-6-luna", + "gpt-6-sol": "openai/gpt-6-sol", + "gpt-6-astra": "openai/gpt-6-astra", + "anthropic/claude-opus-5-5": "anthropic/claude-opus-5.5", + "opus-5-5": "anthropic/claude-opus-5.5", + "opus-5.5": "anthropic/claude-opus-5.5", + "anthropic/claude-fable-5-1": "anthropic/claude-fable-5.1", + "fable-5-1": "anthropic/claude-fable-5.1", + "fable-5.1": "anthropic/claude-fable-5.1", + "anthropic/claude-sonnet-5-5": "anthropic/claude-sonnet-5.5", + "sonnet-5-5": "anthropic/claude-sonnet-5.5", + "sonnet-5.5": "anthropic/claude-sonnet-5.5", // Upstream chat aliases plus DOS-compatible shorthand targets. // Bare Claude/Sonnet, o1, Gemini, Flash and paid DeepSeek Pro retain DOS behavior. // Retired free shorthands resolve directly to an active free successor. @@ -69,7 +103,7 @@ var ModelAliases = map[string]string{ "coder-free": "free/north-mini-code", "codex": "openai/gpt-5.3-codex", "cohere/north-mini-code": "free/north-mini-code", - "deepseek": "deepseek/deepseek-chat", + "deepseek": "deepseek/deepseek-v4.1-flash", "deepseek-chat": "deepseek/deepseek-chat", "deepseek-free": "free/nemotron-3.5-lightning", "deepseek-v4-flash": "free/nemotron-3.5-lightning", @@ -146,10 +180,10 @@ var ModelAliases = map[string]string{ "grok-build": "xai/grok-build-0.1", "grok-code": "xai/grok-build-0.1", "grok-code-fast-1": "deepseek/deepseek-chat", - "grok-fast": "xai/grok-4-fast-reasoning", + "grok-fast": "xai/grok-4-1-fast-reasoning", "haiku": "anthropic/claude-haiku-4.5", - "hunyuan": "tencent/hy3", - "hy3": "tencent/hy3", + "hunyuan": "qwen/qwen3.7-flash", + "hy3": "qwen/qwen3.7-flash", "kimi": "moonshot/kimi-k2.7", "kimi-k2": "moonshot/kimi-k2.6", "kimi-k2.5": "moonshot/kimi-k2.5", @@ -180,7 +214,7 @@ var ModelAliases = map[string]string{ "mistral-small": "free/nemotron-3.5-lightning", "moonshot": "moonshot/kimi-k2.7", "nano": "openai/gpt-5.4-nano", - "nano-30b": "free/nemotron-3-nano-30b", + "nano-30b": "free/nemotron-3-nano-omni-30b-a3b-reasoning", "nano-omni": "free/nemotron-3-nano-omni-30b-a3b-reasoning", "nano-vl": "free/nemotron-3-nano-omni-30b-a3b-reasoning", "nemotron": "free/nemotron-3.5-lightning", @@ -189,9 +223,9 @@ var ModelAliases = map[string]string{ "nemotron-3.5-lightning": "free/nemotron-3.5-lightning", "nemotron-49b": "free/nemotron-3.5-lightning", "nemotron-lightning": "free/nemotron-3.5-lightning", - "nemotron-nano": "free/nemotron-3-nano-30b", - "nemotron-nano-30b": "free/nemotron-3-nano-30b", - "nemotron-nano-9b": "free/nemotron-3-nano-30b", + "nemotron-nano": "free/nemotron-3-nano-omni-30b-a3b-reasoning", + "nemotron-nano-30b": "free/nemotron-3-nano-omni-30b-a3b-reasoning", + "nemotron-nano-9b": "free/nemotron-3-nano-omni-30b-a3b-reasoning", "nemotron-nano-vl": "free/nemotron-3-nano-omni-30b-a3b-reasoning", "nemotron-omni": "free/nemotron-3-nano-omni-30b-a3b-reasoning", "nemotron-super": "free/nemotron-3.5-lightning", @@ -212,7 +246,7 @@ var ModelAliases = map[string]string{ "nvidia/llama-4-maverick": "free/llama-4-maverick", "nvidia/mistral-large-3-675b": "free/nemotron-3.5-lightning", "nvidia/mistral-nemotron": "free/mistral-nemotron", - "nvidia/nemotron-3-nano-30b": "free/nemotron-3-nano-30b", + "nvidia/nemotron-3-nano-30b": "free/nemotron-3-nano-omni-30b-a3b-reasoning", "nvidia/nemotron-3-nano-omni-30b-a3b-reasoning": "free/nemotron-3-nano-omni-30b-a3b-reasoning", "nvidia/nemotron-3-super-120b": "free/nemotron-3.5-lightning", "nvidia/nemotron-3-ultra-550b": "free/nemotron-3-ultra-550b", @@ -280,14 +314,14 @@ var ModelAliases = map[string]string{ "sonnet-5.0": "anthropic/claude-sonnet-5", "step-3.7-flash": "free/nemotron-3.5-lightning", "step-flash": "free/nemotron-3.5-lightning", - "tencent": "tencent/hy3", + "tencent": "qwen/qwen3.7-flash", "terra-pro": "openai/gpt-5.6-terra-pro", "ultra-550b": "free/nemotron-3-ultra-550b", "v4-flash": "free/nemotron-3.5-lightning", "v4-flash-vision": "deepseek/deepseek-v4-flash-vision-exp", "v4-pro": "free/nemotron-3.5-lightning", "vision-free": "free/nemotron-3-nano-omni-30b-a3b-reasoning", - "xai/grok-3-fast": "xai/grok-4-fast-reasoning", + "xai/grok-3-fast": "xai/grok-4-1-fast-reasoning", "xai/grok-code-fast-1": "deepseek/deepseek-chat", "xiaomi": "xiaomi/mimo-v2.5-pro", } @@ -300,7 +334,7 @@ var Models = []ModelDef{ // Routing profiles {ID: "auto", Name: "Auto (Smart Router - Balanced)", InputPrice: 0, OutputPrice: 0, ContextWindow: 1050000, MaxOutput: 128000}, - {ID: "free", Name: "Free - Nemotron 3.5 Lightning", InputPrice: 0, OutputPrice: 0, ContextWindow: 1000000, MaxOutput: 16384, Reasoning: true}, + {ID: "free", Name: "Free → Nemotron 3.5 Lightning", InputPrice: 0, OutputPrice: 0, ContextWindow: 1000000, MaxOutput: 16384, Reasoning: true}, {ID: "eco", Name: "Eco (Smart Router - Cost Optimized)", InputPrice: 0, OutputPrice: 0, ContextWindow: 1050000, MaxOutput: 128000}, {ID: "premium", Name: "Premium (Smart Router - Best Quality)", InputPrice: 0, OutputPrice: 0, ContextWindow: 2000000, MaxOutput: 200000}, @@ -308,21 +342,21 @@ var Models = []ModelDef{ {ID: "openai/gpt-5.2", Name: "GPT-5.2", Version: "5.2", InputPrice: 1.75, OutputPrice: 14.0, ContextWindow: 400000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, {ID: "openai/gpt-5-mini", Name: "GPT-5 Mini", Version: "5.0", InputPrice: 0.25, OutputPrice: 2.0, ContextWindow: 200000, MaxOutput: 128000, ToolCalling: true}, {ID: "openai/gpt-5-nano", Name: "GPT-5 Nano", Version: "5.0", InputPrice: 0.05, OutputPrice: 0.4, ContextWindow: 128000, MaxOutput: 128000, ToolCalling: true, Deprecated: true, FallbackModel: "openai/gpt-5.4-nano"}, - {ID: "openai/gpt-5.2-pro", Name: "GPT-5.2 Pro", Version: "5.2", InputPrice: 21.0, OutputPrice: 168.0, ContextWindow: 400000, MaxOutput: 128000, Reasoning: true, Vision: true, ToolCalling: true}, + {ID: "openai/gpt-5.2-pro", Name: "GPT-5.2 Pro", Version: "5.2", InputPrice: 21, OutputPrice: 168, ContextWindow: 400000, MaxOutput: 128000, Reasoning: true, Vision: true}, {ID: "openai/gpt-5.6-sol", Name: "GPT-5.6 Sol", Version: "5.6", InputPrice: 4.0, OutputPrice: 20.0, ContextWindow: 1050000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, {ID: "openai/gpt-5.6-terra", Name: "GPT-5.6 Terra", Version: "5.6", InputPrice: 2.0, OutputPrice: 12.0, ContextWindow: 1050000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, {ID: "openai/gpt-5.6-luna", Name: "GPT-5.6 Luna", Version: "5.6", InputPrice: 0.2, OutputPrice: 1.2, ContextWindow: 1050000, MaxOutput: 128000, Vision: true, Agentic: true, ToolCalling: true}, {ID: "openai/gpt-5.6-sol-pro", Name: "GPT-5.6 Sol Pro", Version: "5.6", InputPrice: 4.0, OutputPrice: 20.0, ContextWindow: 1050000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, {ID: "openai/gpt-5.6-terra-pro", Name: "GPT-5.6 Terra Pro", Version: "5.6", InputPrice: 2.0, OutputPrice: 12.0, ContextWindow: 1050000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, - {ID: "openai/gpt-5.6-luna-pro", Name: "GPT-5.6 Luna Pro", Version: "5.6", InputPrice: 0.2, OutputPrice: 1.2, ContextWindow: 1050000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, + {ID: "openai/gpt-5.6-luna-pro", Name: "GPT-5.6 Luna Pro", Version: "5.6", InputPrice: 0.2, OutputPrice: 1.2, ContextWindow: 1050000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true}, {ID: "openai/gpt-5.5", Name: "GPT-5.5", Version: "5.5", InputPrice: 5.0, OutputPrice: 30.0, ContextWindow: 1050000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, - {ID: "openai/gpt-5.5-pro", Name: "GPT-5.5 Pro", Version: "5.5", InputPrice: 30.0, OutputPrice: 180.0, ContextWindow: 1050000, MaxOutput: 128000, Reasoning: true, Vision: true, ToolCalling: true}, + {ID: "openai/gpt-5.5-pro", Name: "GPT-5.5 Pro", Version: "5.5", InputPrice: 30, OutputPrice: 180, ContextWindow: 1050000, MaxOutput: 128000, Reasoning: true, Vision: true}, {ID: "openai/chat-latest", Name: "ChatGPT Instant (GPT-5.5)", Version: "5.5", InputPrice: 5.0, OutputPrice: 30.0, ContextWindow: 128000, MaxOutput: 128000, Vision: true, ToolCalling: true}, {ID: "openai/gpt-5.4", Name: "GPT-5.4", Version: "5.4", InputPrice: 2.5, OutputPrice: 15.0, ContextWindow: 1050000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, {ID: "openai/gpt-5.4-mini", Name: "GPT-5.4 Mini", Version: "5.4", InputPrice: 0.75, OutputPrice: 4.5, ContextWindow: 400000, MaxOutput: 128000, Vision: true, Agentic: true, ToolCalling: true}, - {ID: "openai/gpt-5.4-pro", Name: "GPT-5.4 Pro", Version: "5.4", InputPrice: 30.0, OutputPrice: 180.0, ContextWindow: 1050000, MaxOutput: 128000, Reasoning: true, Vision: true, ToolCalling: true}, + {ID: "openai/gpt-5.4-pro", Name: "GPT-5.4 Pro", Version: "5.4", InputPrice: 30, OutputPrice: 180, ContextWindow: 1050000, MaxOutput: 128000, Reasoning: true, Vision: true}, {ID: "openai/gpt-5.4-nano", Name: "GPT-5.4 Nano", Version: "5.4", InputPrice: 0.2, OutputPrice: 1.25, ContextWindow: 1050000, MaxOutput: 128000, ToolCalling: true}, - {ID: "openai/gpt-5.3", Name: "GPT-5.3", Version: "5.3", InputPrice: 1.75, OutputPrice: 14.0, ContextWindow: 128000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, + {ID: "openai/gpt-5.3", Name: "GPT-5.3", Version: "5.3", InputPrice: 1.75, OutputPrice: 14, ContextWindow: 128000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true, Deprecated: true, FallbackModel: "openai/gpt-5.2"}, {ID: "openai/gpt-5.3-codex", Name: "GPT-5.3 Codex", Version: "5.3", InputPrice: 1.75, OutputPrice: 14.0, ContextWindow: 400000, MaxOutput: 128000, Agentic: true, ToolCalling: true}, {ID: "openai/gpt-4.1", Name: "GPT-4.1", Version: "4.1", InputPrice: 2.0, OutputPrice: 8.0, ContextWindow: 128000, MaxOutput: 32768, Vision: true, ToolCalling: true}, {ID: "openai/gpt-4.1-mini", Name: "GPT-4.1 Mini", Version: "4.1", InputPrice: 0.4, OutputPrice: 1.6, ContextWindow: 128000, MaxOutput: 32768, ToolCalling: true}, @@ -339,7 +373,7 @@ var Models = []ModelDef{ {ID: "anthropic/claude-haiku-4.5", Name: "Claude Haiku 4.5", Version: "4.5", InputPrice: 1.0, OutputPrice: 5.0, ContextWindow: 200000, MaxOutput: 64000, Vision: true, Agentic: true, ToolCalling: true}, {ID: "anthropic/claude-sonnet-4.5", Name: "Claude Sonnet 4.5", Version: "4.5", InputPrice: 3.0, OutputPrice: 15.0, ContextWindow: 200000, MaxOutput: 64000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, {ID: "anthropic/claude-sonnet-4.6", Name: "Claude Sonnet 4.6", Version: "4.6", InputPrice: 3.0, OutputPrice: 15.0, ContextWindow: 1000000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, - {ID: "anthropic/claude-sonnet-5", Name: "Claude Sonnet 5", Version: "5", InputPrice: 3.0, OutputPrice: 15.0, ContextWindow: 1000000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, + {ID: "anthropic/claude-sonnet-5", Name: "Claude Sonnet 5", Version: "5", InputPrice: 2, OutputPrice: 10, ContextWindow: 1000000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, {ID: "anthropic/claude-opus-4.5", Name: "Claude Opus 4.5", Version: "4.5", InputPrice: 5.0, OutputPrice: 25.0, ContextWindow: 200000, MaxOutput: 64000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, {ID: "anthropic/claude-opus-4.6", Name: "Claude Opus 4.6", Version: "4.6", InputPrice: 5.0, OutputPrice: 25.0, ContextWindow: 1000000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, {ID: "anthropic/claude-fable-5", Name: "Claude Fable 5", Version: "5", InputPrice: 10.0, OutputPrice: 50.0, ContextWindow: 1000000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, @@ -356,15 +390,15 @@ var Models = []ModelDef{ {ID: "google/gemini-3.6-flash", Name: "Gemini 3.6 Flash", Version: "3.6", InputPrice: 0.75, OutputPrice: 3.75, ContextWindow: 1048576, MaxOutput: 65536, Reasoning: true, Vision: true, ToolCalling: true}, {ID: "google/gemini-3.5-flash", Name: "Gemini 3.5 Flash", Version: "3.5", InputPrice: 1.5, OutputPrice: 9.0, ContextWindow: 1048576, MaxOutput: 65536, Reasoning: true, Vision: true, ToolCalling: true}, {ID: "google/gemini-3.5-flash-lite", Name: "Gemini 3.5 Flash Lite", Version: "3.5", InputPrice: 0.3, OutputPrice: 2.5, ContextWindow: 1048576, MaxOutput: 65536, Reasoning: true, ToolCalling: true}, - {ID: "google/gemini-3-flash-preview", Name: "Gemini 3 Flash Preview", Version: "3.0", InputPrice: 0.5, OutputPrice: 3.0, ContextWindow: 1048576, MaxOutput: 65536, Reasoning: true, Vision: true}, + {ID: "google/gemini-3-flash-preview", Name: "Gemini 3 Flash Preview", Version: "3.0", InputPrice: 0.5, OutputPrice: 3, ContextWindow: 1048576, MaxOutput: 65536, Reasoning: true, Vision: true, ToolCalling: true}, {ID: "google/gemini-2.5-pro", Name: "Gemini 2.5 Pro", Version: "2.5", InputPrice: 1.25, OutputPrice: 10.0, ContextWindow: 1048576, MaxOutput: 65536, Reasoning: true, Vision: true, ToolCalling: true}, {ID: "google/gemini-2.5-flash", Name: "Gemini 2.5 Flash", Version: "2.5", InputPrice: 0.3, OutputPrice: 2.5, ContextWindow: 1048576, MaxOutput: 65536, Vision: true, ToolCalling: true}, {ID: "google/gemini-2.5-flash-lite", Name: "Gemini 2.5 Flash Lite", Version: "2.5", InputPrice: 0.1, OutputPrice: 0.4, ContextWindow: 1048576, MaxOutput: 65536, Vision: true, ToolCalling: true}, {ID: "google/gemini-3.1-flash-lite", Name: "Gemini 3.1 Flash Lite", Version: "3.1", InputPrice: 0.25, OutputPrice: 1.5, ContextWindow: 1048576, MaxOutput: 65536, ToolCalling: true}, // deepseek - {ID: "deepseek/deepseek-chat", Name: "DeepSeek V4 Flash Chat", Version: "4-flash", InputPrice: 0.14, OutputPrice: 0.28, ContextWindow: 1048576, MaxOutput: 65536, ToolCalling: true}, - {ID: "deepseek/deepseek-reasoner", Name: "DeepSeek V4 Flash Reasoner", Version: "4-flash", InputPrice: 0.14, OutputPrice: 0.28, ContextWindow: 1048576, MaxOutput: 65536, Reasoning: true, ToolCalling: true}, + {ID: "deepseek/deepseek-chat", Name: "DeepSeek Chat (V4.1 Flash)", Version: "4.1-flash", InputPrice: 0.14, OutputPrice: 0.28, ContextWindow: 1048576, MaxOutput: 65536, ToolCalling: true}, + {ID: "deepseek/deepseek-reasoner", Name: "DeepSeek Reasoner (V4.1 Flash)", Version: "4.1-flash", InputPrice: 0.14, OutputPrice: 0.28, ContextWindow: 1048576, MaxOutput: 65536, Reasoning: true, ToolCalling: true}, {ID: "deepseek/deepseek-v4-pro", Name: "DeepSeek V4 Pro", Version: "4-pro", InputPrice: 1.32, OutputPrice: 3.96, ContextWindow: 1048576, MaxOutput: 65536, Reasoning: true, Agentic: true, ToolCalling: true}, // moonshot @@ -384,14 +418,14 @@ var Models = []ModelDef{ {ID: "nvidia/kimi-k2.5", Name: "Kimi K2.5 (NVIDIA, retired)", Version: "k2.5", InputPrice: 0.6, OutputPrice: 3.0, ContextWindow: 262144, MaxOutput: 16384, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true, Deprecated: true, FallbackModel: "moonshot/kimi-k2.5"}, // xai - {ID: "xai/grok-3", Name: "Grok 3", Version: "3", InputPrice: 3.0, OutputPrice: 15.0, ContextWindow: 131072, MaxOutput: 16384, Reasoning: true, ToolCalling: true}, - {ID: "xai/grok-3-mini", Name: "Grok 3 Mini", Version: "3-mini", InputPrice: 0.3, OutputPrice: 0.5, ContextWindow: 131072, MaxOutput: 16384, ToolCalling: true}, - {ID: "xai/grok-4-fast-reasoning", Name: "Grok 4 Fast Reasoning", Version: "4", InputPrice: 0.2, OutputPrice: 0.5, ContextWindow: 2000000, MaxOutput: 16384, Reasoning: true, ToolCalling: true}, - {ID: "xai/grok-4-fast-non-reasoning", Name: "Grok 4 Fast", Version: "4", InputPrice: 0.2, OutputPrice: 0.5, ContextWindow: 2000000, MaxOutput: 16384, ToolCalling: true}, + {ID: "xai/grok-3", Name: "Grok 3", Version: "3", InputPrice: 3, OutputPrice: 15, ContextWindow: 131072, MaxOutput: 16384, Reasoning: true, ToolCalling: true, Deprecated: true, FallbackModel: "xai/grok-4-1-fast-non-reasoning"}, + {ID: "xai/grok-3-mini", Name: "Grok 3 Mini", Version: "3-mini", InputPrice: 0.3, OutputPrice: 0.5, ContextWindow: 131072, MaxOutput: 16384, ToolCalling: true, Deprecated: true, FallbackModel: "xai/grok-4-1-fast-non-reasoning"}, + {ID: "xai/grok-4-fast-reasoning", Name: "Grok 4 Fast Reasoning", Version: "4", InputPrice: 0.2, OutputPrice: 0.5, ContextWindow: 2000000, MaxOutput: 16384, Reasoning: true, ToolCalling: true, Deprecated: true, FallbackModel: "xai/grok-4-1-fast-reasoning"}, + {ID: "xai/grok-4-fast-non-reasoning", Name: "Grok 4 Fast", Version: "4", InputPrice: 0.2, OutputPrice: 0.5, ContextWindow: 2000000, MaxOutput: 16384, ToolCalling: true, Deprecated: true, FallbackModel: "xai/grok-4-1-fast-non-reasoning"}, {ID: "xai/grok-4-1-fast-reasoning", Name: "Grok 4.1 Fast Reasoning", Version: "4.1", InputPrice: 0.2, OutputPrice: 0.5, ContextWindow: 2000000, MaxOutput: 16384, Reasoning: true, ToolCalling: true}, {ID: "xai/grok-4-1-fast-non-reasoning", Name: "Grok 4.1 Fast", Version: "4.1", InputPrice: 0.2, OutputPrice: 0.5, ContextWindow: 2000000, MaxOutput: 16384, ToolCalling: true}, {ID: "xai/grok-4-0709", Name: "Grok 4 (0709)", Version: "4-0709", InputPrice: 3.0, OutputPrice: 15.0, ContextWindow: 256000, MaxOutput: 16384, Reasoning: true, ToolCalling: true}, - {ID: "xai/grok-2-vision", Name: "Grok 2 Vision", Version: "2", InputPrice: 2.0, OutputPrice: 10.0, ContextWindow: 32768, MaxOutput: 16384, Vision: true, ToolCalling: true}, + {ID: "xai/grok-2-vision", Name: "Grok 2 Vision", Version: "2", InputPrice: 2, OutputPrice: 10, ContextWindow: 32768, MaxOutput: 16384, Vision: true, ToolCalling: true, Deprecated: true, FallbackModel: "xai/grok-4.3"}, {ID: "xai/grok-4.20-reasoning", Name: "Grok 4.20 Reasoning", Version: "4.20", InputPrice: 2.0, OutputPrice: 6.0, ContextWindow: 2000000, MaxOutput: 16384, Reasoning: true, ToolCalling: true}, {ID: "xai/grok-4.20-non-reasoning", Name: "Grok 4.20", Version: "4.20", InputPrice: 2.0, OutputPrice: 6.0, ContextWindow: 2000000, MaxOutput: 16384, ToolCalling: true}, {ID: "xai/grok-4.20-multi-agent", Name: "Grok 4.20 Multi-Agent", Version: "4.20", InputPrice: 2.0, OutputPrice: 6.0, ContextWindow: 2000000, MaxOutput: 16384, Reasoning: true, ToolCalling: true}, @@ -400,7 +434,7 @@ var Models = []ModelDef{ {ID: "xai/grok-build-0.1", Name: "Grok Build 0.1", Version: "0.1", InputPrice: 1.0, OutputPrice: 2.0, ContextWindow: 256000, MaxOutput: 16384, Agentic: true, ToolCalling: true}, // tencent - {ID: "tencent/hy3", Name: "Tencent Hy3", Version: "hy3", InputPrice: 0.132, OutputPrice: 0.528, ContextWindow: 262144, MaxOutput: 128000, Reasoning: true, ToolCalling: true}, + {ID: "tencent/hy3", Name: "Tencent Hy3", Version: "hy3", InputPrice: 0.132, OutputPrice: 0.528, ContextWindow: 262144, MaxOutput: 128000, Reasoning: true, ToolCalling: true, Deprecated: true, FallbackModel: "qwen/qwen3.7-flash"}, // xiaomi {ID: "xiaomi/mimo-v2.5-pro", Name: "Xiaomi MiMo-V2.5 Pro", Version: "v2.5-pro", InputPrice: 0.435, OutputPrice: 0.87, ContextWindow: 1048576, MaxOutput: 131072, Reasoning: true, ToolCalling: true}, @@ -424,7 +458,7 @@ var Models = []ModelDef{ {ID: "free/seed-oss-36b", Name: "[Free] Seed-OSS 36B", Version: "oss-36b", InputPrice: 0, OutputPrice: 0, ContextWindow: 131072, MaxOutput: 16384, Deprecated: true, FallbackModel: "free/nemotron-3.5-lightning"}, {ID: "free/mistral-nemotron", Name: "[Free] Mistral Nemotron", Version: "nemotron", InputPrice: 0, OutputPrice: 0, ContextWindow: 131072, MaxOutput: 16384, Deprecated: true, FallbackModel: "free/nemotron-3.5-lightning"}, {ID: "free/step-3.7-flash", Name: "[Free] StepFun Step 3.7 Flash", Version: "3.7-flash", InputPrice: 0, OutputPrice: 0, ContextWindow: 131072, MaxOutput: 16384, Reasoning: true, Deprecated: true, FallbackModel: "free/nemotron-3.5-lightning"}, - {ID: "free/nemotron-nano-9b-v2", Name: "[Free] Nemotron Nano 9B v2", Version: "nano-9b-v2", InputPrice: 0, OutputPrice: 0, ContextWindow: 131072, MaxOutput: 16384, Reasoning: true, Deprecated: true, FallbackModel: "free/nemotron-3-nano-30b"}, + {ID: "free/nemotron-nano-9b-v2", Name: "[Free] Nemotron Nano 9B v2", Version: "nano-9b-v2", InputPrice: 0, OutputPrice: 0, ContextWindow: 131072, MaxOutput: 16384, Reasoning: true, Deprecated: true, FallbackModel: "free/nemotron-3.5-lightning"}, {ID: "free/nemotron-nano-12b-v2-vl", Name: "[Free] Nemotron Nano 12B v2 VL", Version: "nano-12b-v2-vl", InputPrice: 0, OutputPrice: 0, ContextWindow: 131072, MaxOutput: 16384, Reasoning: true, Deprecated: true, FallbackModel: "free/nemotron-3-nano-omni-30b-a3b-reasoning"}, {ID: "free/nemotron-3.5-lightning", Name: "[Free] Nemotron 3.5 Lightning", Version: "3.5-lightning", InputPrice: 0, OutputPrice: 0, ContextWindow: 1000000, MaxOutput: 16384, Reasoning: true}, {ID: "free/nemotron-3-nano-30b", Name: "[Free] Nemotron 3 Nano 30B", Version: "3-nano-30b", InputPrice: 0, OutputPrice: 0, ContextWindow: 131072, MaxOutput: 16384, Reasoning: true}, @@ -434,10 +468,10 @@ var Models = []ModelDef{ {ID: "free/laguna-xs-2.1", Name: "[Free] Poolside Laguna XS 2.1", Version: "xs-2.1", InputPrice: 0, OutputPrice: 0, ContextWindow: 131072, MaxOutput: 16384}, // qwen - {ID: "qwen/qwen3.8-flash", Name: "Qwen3.8 Flash", Version: "3.8-flash", InputPrice: 0.15, OutputPrice: 0.47, ContextWindow: 1000000, MaxOutput: 131072, Reasoning: true, Vision: true, ToolCalling: true}, + {ID: "qwen/qwen3.8-flash", Name: "Qwen3.8 Flash", Version: "3.8-flash", InputPrice: 0.15, OutputPrice: 0.47, ContextWindow: 1000000, MaxOutput: 131072, Reasoning: true, Vision: true}, // deepseek - {ID: "deepseek/deepseek-v4-flash-vision-exp", Name: "DeepSeek V4 Flash Vision", Version: "v4-flash-vision-exp", InputPrice: 0.44, OutputPrice: 1.32, ContextWindow: 1048576, MaxOutput: 65536, Reasoning: true, Vision: true, ToolCalling: true}, + {ID: "deepseek/deepseek-v4-flash-vision-exp", Name: "DeepSeek V4 Flash Vision", Version: "v4-flash-vision-exp", InputPrice: 0.3, OutputPrice: 1.2, ContextWindow: 1048576, MaxOutput: 65536, Reasoning: true, Vision: true}, // xiaomi {ID: "xiaomi/mimo-v2.5", Name: "Xiaomi MiMo V2.5", Version: "2.5", InputPrice: 0.14, OutputPrice: 0.28, ContextWindow: 1048576, MaxOutput: 131072, Reasoning: true, Vision: true, ToolCalling: true}, @@ -462,6 +496,26 @@ var Models = []ModelDef{ {ID: "free/nemotron-super-49b", Name: "nemotron-super-49b (retired)", ContextWindow: 131072, MaxOutput: 16384, InputPrice: 0, OutputPrice: 0, Deprecated: true, FallbackModel: "free/nemotron-3.5-lightning"}, {ID: "free/nemotron-3-super-120b", Name: "nemotron-3-super-120b (retired)", ContextWindow: 131072, MaxOutput: 16384, InputPrice: 0, OutputPrice: 0, Deprecated: true, FallbackModel: "free/nemotron-3.5-lightning"}, {ID: "free/devstral-2-123b", Name: "devstral-2-123b (retired)", ContextWindow: 131072, MaxOutput: 16384, InputPrice: 0, OutputPrice: 0, Deprecated: true, FallbackModel: "free/nemotron-3.5-lightning"}, + // Added from ClawRouter 78e489a (0.12.297 source). + {ID: "openai/gpt-6-astra", Name: "GPT-6 Astra", Version: "6", InputPrice: 10, OutputPrice: 50, ContextWindow: 1050000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, + {ID: "openai/gpt-6-sol", Name: "GPT-6 Sol", Version: "6", InputPrice: 2, OutputPrice: 10, ContextWindow: 1050000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, + {ID: "openai/gpt-6.1-sol", Name: "GPT-6.1 Sol", Version: "6.1", InputPrice: 2, OutputPrice: 10, ContextWindow: 1050000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, + {ID: "openai/gpt-6-luna", Name: "GPT-6 Luna", Version: "6", InputPrice: 0.1, OutputPrice: 0.5, ContextWindow: 1050000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, + {ID: "openai/gpt-5.1", Name: "GPT-5.1", Version: "5.1", InputPrice: 1.25, OutputPrice: 10, ContextWindow: 400000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, + {ID: "anthropic/claude-fable-5.1", Name: "Claude Fable 5.1", Version: "5.1", InputPrice: 10, OutputPrice: 50, ContextWindow: 1000000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, + {ID: "anthropic/claude-opus-5.5", Name: "Claude Opus 5.5", Version: "5.5", InputPrice: 4, OutputPrice: 20, ContextWindow: 1000000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, + {ID: "anthropic/claude-sonnet-5.5", Name: "Claude Sonnet 5.5", Version: "5.5", InputPrice: 2, OutputPrice: 10, ContextWindow: 1000000, MaxOutput: 128000, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, + {ID: "google/gemini-4-argon", Name: "Gemini 4 Argon", Version: "4", InputPrice: 2, OutputPrice: 10, ContextWindow: 1048576, MaxOutput: 65536, Reasoning: true, Vision: true}, + {ID: "deepseek/deepseek-v4.1-flash", Name: "DeepSeek V4.1 Flash", Version: "4.1-flash", InputPrice: 0.14, OutputPrice: 0.28, ContextWindow: 1048576, MaxOutput: 65536, ToolCalling: true}, + {ID: "xai/grok-4.7", Name: "Grok 4.7", Version: "4.7", InputPrice: 2, OutputPrice: 6, ContextWindow: 500000, MaxOutput: 16384, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, + {ID: "xai/grok-4.6", Name: "Grok 4.6", Version: "4.6", InputPrice: 2, OutputPrice: 6, ContextWindow: 500000, MaxOutput: 16384, Reasoning: true, Vision: true, Agentic: true, ToolCalling: true}, + {ID: "free/muse-glimmer-30b", Name: "[Free] Muse Glimmer 30B", Version: "30b", InputPrice: 0, OutputPrice: 0, ContextWindow: 131072, MaxOutput: 16384, Reasoning: true}, + {ID: "qwen/qwen3.8-max", Name: "Qwen3.8 Max", Version: "3.8-max", InputPrice: 2, OutputPrice: 6, ContextWindow: 1000000, MaxOutput: 131072, Reasoning: true}, + {ID: "tencent/hy4-preview", Name: "Tencent Hy4 Preview", Version: "hy4-preview", InputPrice: 0.834, OutputPrice: 2.501, ContextWindow: 1048576, MaxOutput: 64000, Reasoning: true}, + {ID: "mistral/mistral-large-4", Name: "Mistral Large 4", Version: "large-4", InputPrice: 1.36, OutputPrice: 4.18, ContextWindow: 524288, MaxOutput: 262144, Reasoning: true, Vision: true}, + {ID: "poolside/laguna-s-2.1", Name: "Poolside Laguna S 2.1", Version: "s-2.1", InputPrice: 0.09, OutputPrice: 0.18, ContextWindow: 1048576, MaxOutput: 131072}, + // Added from ClawRouter 78e489a (0.12.297 source). + } // modelIndex is built on init for fast lookups. diff --git a/proxy/catalog.go b/proxy/catalog.go new file mode 100644 index 0000000..b214977 --- /dev/null +++ b/proxy/catalog.go @@ -0,0 +1,211 @@ +package proxy + +import ( + "context" + "encoding/json" + "errors" + "io" + "log" + "math" + "net/http" + "regexp" + "slices" + "strings" + "time" + + "github.com/DOS/DOSRouter/models" + "github.com/DOS/DOSRouter/router" +) + +type catalogSnapshot struct { + models []models.ModelDef + pricing map[string]router.ModelPricing +} + +func (s *Server) catalogView() catalogSnapshot { + s.catalogMu.RLock() + defer s.catalogMu.RUnlock() + if s.catalog != nil { + return *s.catalog + } + return catalogSnapshot{models.Models, s.modelPricing} +} + +func (c catalogSnapshot) model(id string) *models.ModelDef { + for i := range c.models { + if c.models[i].ID == id { + return &c.models[i] + } + } + return nil +} + +type catalogRow struct { + ID string `json:"id"` + Name string `json:"name"` + Categories []string `json:"categories"` + Available *bool `json:"available"` + Billing string `json:"billing_mode"` + Pricing struct { + Input *float64 `json:"input"` + Output *float64 `json:"output"` + Flat *float64 `json:"flat"` + } `json:"pricing"` + Context *int `json:"context_window"` + MaxOutput *int `json:"max_output"` +} + +var catalogID = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._+/-]*$`) + +func validPrice(p *float64) bool { return p != nil && *p >= 0 && !math.IsNaN(*p) && !math.IsInf(*p, 0) } + +// projectCatalog changes metadata only; aliases, routing policy and proven tools stay local. +func projectCatalog(body []byte) (catalogSnapshot, error) { + var payload struct { + Data []catalogRow `json:"data"` + } + invalid := errors.New("invalid gateway catalog") + if json.Unmarshal(body, &payload) != nil || len(payload.Data) == 0 { + return catalogSnapshot{}, invalid + } + next := catalogSnapshot{models: append([]models.ModelDef(nil), models.Models...), pricing: models.BuildPricingMap()} + byUpstream := map[string]int{} + for i, m := range next.models { + id := m.ID + if strings.HasPrefix(id, "free/") { + id = "nvidia/" + strings.TrimPrefix(id, "free/") + } + if m.ID == "free/north-mini-code" { + id = "cohere/north-mini-code" + } + if m.ID == "free/laguna-xs-2.1" { + id = "poolside/laguna-xs-2.1" + } + byUpstream[id] = i + } + seen := map[string]bool{} + for _, row := range payload.Data { + if !catalogID.MatchString(row.ID) || seen[row.ID] || row.Name == "" || row.Categories == nil { + return catalogSnapshot{}, invalid + } + seen[row.ID] = true + if row.Context != nil && *row.Context <= 0 || row.MaxOutput != nil && *row.MaxOutput <= 0 { + return catalogSnapshot{}, invalid + } + p := router.ModelPricing{} + switch row.Billing { + case "paid": + if !validPrice(row.Pricing.Input) || !validPrice(row.Pricing.Output) { + return catalogSnapshot{}, invalid + } + p.InputPrice = *row.Pricing.Input + p.OutputPrice = *row.Pricing.Output + case "free": + if row.Pricing.Input != nil && *row.Pricing.Input != 0 || row.Pricing.Output != nil && *row.Pricing.Output != 0 { + return catalogSnapshot{}, invalid + } + case "flat": + if !validPrice(row.Pricing.Flat) { + return catalogSnapshot{}, invalid + } + p.FlatPrice = row.Pricing.Flat + default: + continue + } + if !slices.Contains(row.Categories, "chat") || row.Available != nil && !*row.Available || !strings.Contains(row.ID, "/") || strings.HasPrefix(row.ID, "blockrun/") { + continue + } + idx, known := byUpstream[row.ID] + var m models.ModelDef + if known { + m = next.models[idx] + if strings.HasPrefix(m.ID, "free/") && row.Billing != "free" { + return catalogSnapshot{}, errors.New("gateway changed a free model's billing") + } + } else { + if _, alias := models.ModelAliases[row.ID]; alias { + continue + } + // No remote source may mint a DOSRouter free-product name. + if strings.HasPrefix(row.ID, "free/") || strings.HasPrefix(row.ID, "dosrouter/") { + continue + } + m = models.ModelDef{ID: row.ID, ContextWindow: 32768, MaxOutput: 4096, Reasoning: slices.Contains(row.Categories, "reasoning"), Vision: slices.Contains(row.Categories, "vision")} + } + m.Name = row.Name + m.InputPrice = p.InputPrice + m.OutputPrice = p.OutputPrice + m.Promo = nil + if row.Context != nil { + m.ContextWindow = *row.Context + } + if row.MaxOutput != nil { + m.MaxOutput = *row.MaxOutput + } + if known { + next.models[idx] = m + } else { + byUpstream[row.ID] = len(next.models) + next.models = append(next.models, m) + } + next.pricing[m.ID] = p + } + return next, nil +} + +func (s *Server) refreshCatalog(ctx context.Context) error { + ctx, cancel := context.WithTimeout(ctx, 5*time.Second) + defer cancel() + req, err := http.NewRequestWithContext(ctx, http.MethodGet, strings.TrimRight(s.config.UpstreamBase, "/")+"/v1/models?format=json", nil) + if err != nil { + return err + } + if s.config.UpstreamAPIKey != "" { + req.Header.Set("Authorization", "Bearer "+s.config.UpstreamAPIKey) + } + resp, err := s.httpClient.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return errors.New("gateway catalog unavailable") + } + body, err := io.ReadAll(io.LimitReader(resp.Body, 4*1024*1024+1)) + if err != nil || len(body) > 4*1024*1024 { + return errors.New("invalid gateway catalog body") + } + next, err := projectCatalog(body) + if err != nil { + return err + } + s.catalogMu.Lock() + s.catalog = &next + s.catalogMu.Unlock() + return nil +} + +func (s *Server) startCatalogRefresh() { + s.catalogStart.Do(func() { + go func() { + ticker := time.NewTicker(5 * time.Minute) + defer ticker.Stop() + failed := false + for { + if s.catalogCtx.Err() != nil { + return + } + err := s.refreshCatalog(s.catalogCtx) + if err != nil && !failed && s.catalogCtx.Err() == nil { + log.Print("DOSRouter: model catalog refresh failed; keeping the last good snapshot") + } + failed = err != nil + select { + case <-s.catalogCtx.Done(): + return + case <-ticker.C: + } + } + }() + }) +} diff --git a/proxy/catalog_test.go b/proxy/catalog_test.go new file mode 100644 index 0000000..e0242d6 --- /dev/null +++ b/proxy/catalog_test.go @@ -0,0 +1,145 @@ +package proxy + +import ( + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + "testing" + + "github.com/DOS/DOSRouter/models" + "github.com/DOS/DOSRouter/spendcontrol" +) + +func TestCatalogProjectionKeepsProductPolicy(t *testing.T) { + body := []byte(`{"data":[ + {"id":"openai/gpt-6-sol","name":"Updated Sol","categories":["chat"],"billing_mode":"paid","pricing":{"input":3,"output":11},"context_window":900000,"max_output":100000}, + {"id":"vendor/new","name":"New","categories":["chat","vision","coding"],"billing_mode":"flat","pricing":{"flat":0.25}}, + {"id":"nvidia/nemotron-3.5-lightning","name":"Free","categories":["chat"],"billing_mode":"free","pricing":{}} + ]}`) + next, err := projectCatalog(body) + if err != nil { + t.Fatal(err) + } + if m := next.model("openai/gpt-6-sol"); m == nil || m.InputPrice != 3 || !m.ToolCalling || m.ContextWindow != 900000 { + t.Fatalf("known capability or metadata lost: %+v", m) + } + if m := next.model("vendor/new"); m == nil || m.ToolCalling || m.Agentic || !m.Vision { + t.Fatalf("new model acquired unverified tools: %+v", m) + } + if p := next.pricing["vendor/new"]; p.FlatPrice == nil || *p.FlatPrice != 0.25 { + t.Fatal("flat price omitted") + } + if models.GetModel("openai/gpt-6-sol").InputPrice != 2 || models.GetModel("vendor/new") != nil { + t.Fatal("catalog escaped server ownership") + } + if models.ResolveModelAlias("gpt5") != "openai/gpt-5.6-terra" || next.pricing["free/nemotron-3.5-lightning"].InputPrice != 0 { + t.Fatal("product alias/free policy changed") + } +} + +func TestCatalogRejectsWholeInvalidUpdate(t *testing.T) { + for _, body := range []string{ + `{"data":[]}`, + `{"data":[{"id":"new/model","name":"New","categories":["chat"],"billing_mode":"paid","pricing":{"input":1}}]}`, + `{"data":[{"id":"new/model","name":"New","categories":["chat"],"billing_mode":"flat","pricing":{"flat":-1}}]}`, + `{"data":[{"id":"nvidia/nemotron-3.5-lightning","name":"Free","categories":["chat"],"billing_mode":"paid","pricing":{"input":1,"output":1}}]}`, + `{"data":[{"id":"bad\n/model","name":"Bad","categories":["chat"],"billing_mode":"free","pricing":{}}]}`, + } { + if _, err := projectCatalog([]byte(body)); err == nil { + t.Errorf("accepted invalid catalog: %s", body) + } + } +} + +func TestCatalogRefreshIsolatedAndKeepsLastGood(t *testing.T) { + var valid atomic.Bool + valid.Store(true) + srv, _ := syncTestServer(t, func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/v1/models" || r.URL.Query().Get("format") != "json" { + t.Errorf("wrong catalog URL: %s", r.URL) + } + if valid.Load() { + io.WriteString(w, `{"data":[{"id":"vendor/new","name":"New","categories":["chat"],"billing_mode":"paid","pricing":{"input":1,"output":2}}]}`) + } else { + io.WriteString(w, `{"data":[]}`) + } + }, nil) + other, _ := syncTestServer(t, func(http.ResponseWriter, *http.Request) { t.Fatal("another server was contacted") }, nil) + if err := srv.refreshCatalog(context.Background()); err != nil { + t.Fatal(err) + } + old := srv.catalogView() + if old.model("vendor/new") == nil || other.catalogView().model("vendor/new") != nil { + t.Fatal("cross-server catalog contamination") + } + valid.Store(false) + if err := srv.refreshCatalog(context.Background()); err == nil { + t.Fatal("bad update accepted") + } + if srv.catalogView().model("vendor/new") == nil { + t.Fatal("last good catalog discarded") + } + w := httptest.NewRecorder() + srv.handleModels(w, httptest.NewRequest("GET", "/v1/models", nil)) + if !strings.Contains(w.Body.String(), "vendor/new") { + t.Fatal("new model absent from public listing") + } +} + +func TestCatalogRefreshDoesNotFollowRedirects(t *testing.T) { + destination := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { t.Error("catalog followed redirect") })) + defer destination.Close() + srv, _ := syncTestServer(t, func(w http.ResponseWriter, r *http.Request) { http.Redirect(w, r, destination.URL, 302) }, nil) + if err := srv.refreshCatalog(context.Background()); err == nil { + t.Fatal("redirect accepted") + } +} + +func TestRemoteFreeClaimCannotBypassSpendLimits(t *testing.T) { + for _, row := range []string{ + `{"id":"vendor/free-claim","name":"Free claim","categories":["chat"],"billing_mode":"free","pricing":{}}`, + `{"id":"vendor/free-claim","name":"Free claim","categories":["chat"],"billing_mode":"flat","pricing":{"flat":0}}`, + } { + srv, sc := syncTestServer(t, nil, nil) + if err := sc.SetLimit(spendcontrol.WindowSession, 1); err != nil { + t.Fatal(err) + } + next, err := projectCatalog([]byte(`{"data":[` + row + `]}`)) + if err != nil { + t.Fatal(err) + } + srv.catalog = &next + w := httptest.NewRecorder() + if _, allowed := srv.reserveChat(w, chatRequest{}, []byte(`{}`), "vendor/free-claim"); allowed || w.Code != 429 { + t.Fatalf("untrusted free claim bypassed limits: %d", w.Code) + } + } +} + +func TestCatalogReasoningMetadataReachesForwardedRequest(t *testing.T) { + srv, _ := syncTestServer(t, func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodGet { + io.WriteString(w, `{"data":[{"id":"vendor/reasoner","name":"Reasoner","categories":["chat","reasoning"],"billing_mode":"paid","pricing":{"input":1,"output":2}}]}`) + return + } + var req chatRequest + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + t.Error(err) + } + if len(req.Messages) < 2 || req.Messages[1].ReasoningContent == nil { + t.Error("catalog reasoning metadata was ignored") + } + syncTestOK(w) + }, nil) + if err := srv.refreshCatalog(context.Background()); err != nil { + t.Fatal(err) + } + w := syncTestChat(t, srv, `{"model":"vendor/reasoner","messages":[{"role":"user","content":"hi"},{"role":"assistant","content":"hello"},{"role":"user","content":"continue"}]}`) + if w.Code != 200 { + t.Fatalf("status=%d", w.Code) + } +} diff --git a/proxy/chain.go b/proxy/chain.go new file mode 100644 index 0000000..d182cd6 --- /dev/null +++ b/proxy/chain.go @@ -0,0 +1,65 @@ +package proxy + +import ( + "encoding/json" + "slices" + + "github.com/DOS/DOSRouter/router" +) + +// Function-calling probe results from router-core 8773a99. These are recovery +// candidates only; they never replace a configured, eligible primary. +var verifiedToolFallbacks = []string{ + "anthropic/claude-fable-5.1", "anthropic/claude-opus-5.5", "anthropic/claude-sonnet-5.5", + "deepseek/deepseek-v4.1-flash", "google/gemini-3.8-flash", "openai/gpt-5.1", + "openai/gpt-6-astra", "openai/gpt-6-luna", "openai/gpt-6-sol", "openai/gpt-6.1-sol", + "xai/grok-4.6", "xai/grok-4.7", "xiaomi/mimo-v2.5", +} + +func requiresTools(req chatRequest) bool { + if !requestHasTools(req.Tools) { + return false + } + var choice string + if json.Unmarshal(req.Extra["tool_choice"], &choice) == nil && choice == "none" { + return false + } + return true +} + +func smartFallbackChain(req chatRequest, decision *router.RoutingDecision, catalog catalogSnapshot) []string { + chain := router.GetFallbackChain(decision.Tier, decision.TierConfigs) + tools := requiresTools(req) + if tools { + additional := append([]string(nil), verifiedToolFallbacks...) + if decision.Profile != "premium" { + slices.SortStableFunc(additional, func(a, b string) int { + pa, pb := catalog.pricing[a], catalog.pricing[b] + ca, cb := pa.InputPrice+pa.OutputPrice, pb.InputPrice+pb.OutputPrice + if ca < cb { + return -1 + } + if ca > cb { + return 1 + } + return 0 + }) + } + chain = append(chain, additional...) + } + // Retain full-chain ordering before outcome demotion and the attempt cap. + seen := map[string]bool{} + result := []string{} + for _, id := range chain { + if seen[id] { + continue + } + seen[id] = true + m := catalog.model(id) + if m == nil || m.Deprecated || tools && !m.ToolCalling { + continue + } + result = append(result, id) + } + return result +} diff --git a/proxy/chain_test.go b/proxy/chain_test.go new file mode 100644 index 0000000..8370a1e --- /dev/null +++ b/proxy/chain_test.go @@ -0,0 +1,34 @@ +package proxy + +import ( + "encoding/json" + "testing" + + "github.com/DOS/DOSRouter/router" +) + +func TestVerifiedToolsAreRecoveryOnly(t *testing.T) { + srv, _ := syncTestServer(t, nil, nil) + decision := &router.RoutingDecision{Tier: router.TierSimple, Profile: "auto", TierConfigs: map[router.Tier]router.TierConfig{router.TierSimple: {Primary: "openai/gpt-4o-mini", Fallback: []string{"free/nemotron-3.5-lightning"}}}} + req := chatRequest{Tools: json.RawMessage(`[{"type":"function","function":{"name":"test"}}]`)} + chain := smartFallbackChain(req, decision, srv.catalogView()) + if chain[0] != "openai/gpt-4o-mini" { + t.Fatal("verified fallback displaced configured primary") + } + found := false + for _, id := range chain { + if id == "free/nemotron-3.5-lightning" { + t.Fatal("tool-ineligible free model retained") + } + if id == "openai/gpt-6.1-sol" { + found = true + } + } + if !found { + t.Fatal("new verified tool recovery model omitted") + } + req.Extra = map[string]json.RawMessage{"tool_choice": json.RawMessage(`"none"`)} + if len(smartFallbackChain(req, decision, srv.catalogView())) != 2 { + t.Fatal("disabled tools still changed chain") + } +} diff --git a/proxy/local_guard.go b/proxy/local_guard.go new file mode 100644 index 0000000..890ff59 --- /dev/null +++ b/proxy/local_guard.go @@ -0,0 +1,45 @@ +package proxy + +import ( + "encoding/json" + "net/http" + "net/url" + "strings" +) + +func localURL(raw string) bool { + u, err := url.Parse(raw) + if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.User != nil || u.RawQuery != "" || u.Fragment != "" || u.Path != "" { + return false + } + host := strings.TrimSuffix(strings.ToLower(u.Hostname()), ".") + return host == "localhost" || host == "127.0.0.1" || host == "::1" +} + +func localOnly(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + trusted := r.Host != "" && localURL("http://"+r.Host) + origins, hasOrigin := r.Header["Origin"] + if hasOrigin { + trusted = trusted && len(origins) == 1 && localURL(origins[0]) + } + if strings.EqualFold(r.Header.Get("Sec-Fetch-Site"), "cross-site") { + trusted = false + } + if !trusted { + http.Error(w, "Local clients only", http.StatusForbidden) + return + } + next.ServeHTTP(w, r) + }) +} + +func requiresStructuredOutput(req chatRequest) bool { + var format struct { + Type string `json:"type"` + } + if json.Unmarshal(req.Extra["response_format"], &format) != nil { + return false + } + return format.Type == "json_object" || format.Type == "json_schema" +} diff --git a/proxy/local_guard_test.go b/proxy/local_guard_test.go new file mode 100644 index 0000000..9e78fbe --- /dev/null +++ b/proxy/local_guard_test.go @@ -0,0 +1,50 @@ +package proxy + +import ( + "net/http" + "net/http/httptest" + "testing" +) + +func TestLocalTrustBoundary(t *testing.T) { + for _, tc := range []struct { + host, origin, site string + want int + }{ + {"127.0.0.1:8080", "", "", 204}, + {"localhost:8080", "http://localhost:3000", "same-site", 204}, + {"[::1]:8080", "https://[::1]", "", 204}, + {"attacker.example", "", "", 403}, + {"127.0.0.1", "https://attacker.example", "", 403}, + {"127.0.0.1", "null", "", 403}, + {"127.0.0.1", "file://localhost", "", 403}, + {"127.0.0.1", "http://localhost/path", "", 403}, + {"127.0.0.1", "", "cross-site", 403}, + } { + t.Run(tc.host+tc.origin+tc.site, func(t *testing.T) { + req := httptest.NewRequest("POST", "http://"+tc.host+"/v1/images/generations", nil) + if tc.origin != "" { + req.Header.Set("Origin", tc.origin) + } + req.Header.Set("Sec-Fetch-Site", tc.site) + w := httptest.NewRecorder() + localOnly(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(204) })).ServeHTTP(w, req) + if w.Code != tc.want { + t.Fatalf("status=%d want=%d", w.Code, tc.want) + } + }) + } +} + +func TestAllRoutesRejectRemoteBrowsers(t *testing.T) { + srv, _ := syncTestServer(t, func(http.ResponseWriter, *http.Request) { t.Fatal("untrusted request reached upstream") }, nil) + for _, path := range []string{"/health", "/v1/models", "/v1/chat/completions", "/v1/images/generations", "/v1/crypto/price/BTC"} { + req := httptest.NewRequest("POST", "http://localhost"+path, nil) + req.Header.Set("Origin", "https://attacker.example") + w := httptest.NewRecorder() + srv.Handler().ServeHTTP(w, req) + if w.Code != 403 { + t.Errorf("%s: status %d", path, w.Code) + } + } +} diff --git a/proxy/outcomes.go b/proxy/outcomes.go new file mode 100644 index 0000000..883f02a --- /dev/null +++ b/proxy/outcomes.go @@ -0,0 +1,96 @@ +package proxy + +import ( + "regexp" + "sync" + "time" +) + +type modelOutcome struct { + at time.Time + ok bool +} +type outcomeEntry struct { + samples []modelOutcome + touched uint64 +} + +// outcomeMemory is a bounded, process-local hint, scoped to one upstream server. +type outcomeMemory struct { + mu sync.Mutex + entries map[string]outcomeEntry + sequence uint64 +} + +func (m *outcomeMemory) recent(key string, now time.Time) outcomeEntry { + entry := m.entries[key] + live := entry.samples[:0] + for _, sample := range entry.samples { + if now.Sub(sample.at) < 30*time.Minute { + live = append(live, sample) + } + } + entry.samples = live + if len(live) == 0 { + delete(m.entries, key) + } else { + m.entries[key] = entry + } + return entry +} + +func (m *outcomeMemory) record(model, kind string, ok bool, now time.Time) { + m.mu.Lock() + defer m.mu.Unlock() + if m.entries == nil { + m.entries = make(map[string]outcomeEntry) + } + key := model + "|" + kind + entry := m.recent(key, now) + entry.samples = append(entry.samples, modelOutcome{now, ok}) + if len(entry.samples) > 10 { + entry.samples = entry.samples[len(entry.samples)-10:] + } + m.sequence++ + entry.touched = m.sequence + m.entries[key] = entry + if len(m.entries) > 512 { + oldest := "" + var seq uint64 = ^uint64(0) + for k, e := range m.entries { + if e.touched < seq { + oldest = k + seq = e.touched + } + } + delete(m.entries, oldest) + } +} + +func (m *outcomeMemory) demote(chain []string, kind, keep string, now time.Time) []string { + m.mu.Lock() + defer m.mu.Unlock() + good, bad := []string{}, []string{} + for _, model := range chain { + samples := m.recent(model+"|"+kind, now).samples + failures := 0 + for _, sample := range samples { + if !sample.ok { + failures++ + } + } + if model != keep && failures >= 3 && failures*2 >= len(samples) { + bad = append(bad, model) + } else { + good = append(good, model) + } + } + if len(good) == 0 { + return chain + } + return append(good, bad...) +} + +var paymentFailurePattern = regexp.MustCompile(`(?is)payment.*(?:verification|settlement).*failed|insufficient.*funds|transaction_simulation_failed|payment_in_doubt`) + +func paymentFailure(body []byte) bool { return paymentFailurePattern.Match(body) } diff --git a/proxy/outcomes_test.go b/proxy/outcomes_test.go new file mode 100644 index 0000000..b00bfac --- /dev/null +++ b/proxy/outcomes_test.go @@ -0,0 +1,54 @@ +package proxy + +import ( + "fmt" + "reflect" + "sync" + "testing" + "time" +) + +func TestOutcomeWindowExpiryAndKinds(t *testing.T) { + m := &outcomeMemory{} + now := time.Now() + chain := []string{"bad", "good"} + for i := 0; i < 3; i++ { + m.record("bad", "tools", false, now) + } + if got := m.demote(chain, "tools", "", now); !reflect.DeepEqual(got, []string{"good", "bad"}) { + t.Fatalf("demotion=%v", got) + } + for _, tc := range []struct { + kind, keep string + at time.Time + }{{"chat", "", now}, {"tools", "bad", now}, {"tools", "", now.Add(30 * time.Minute)}} { + if got := m.demote(chain, tc.kind, tc.keep, tc.at); !reflect.DeepEqual(got, chain) { + t.Fatalf("unexpected demotion=%v", got) + } + } + for i := 0; i < 600; i++ { + m.record(fmt.Sprint(i), "chat", false, now) + } + if len(m.entries) != 512 { + t.Fatalf("unbounded outcomes: %d", len(m.entries)) + } +} + +func TestOutcomeMemoryConcurrent(t *testing.T) { + var memory outcomeMemory + var workers sync.WaitGroup + for i := 0; i < 12; i++ { + workers.Add(1) + go func() { + defer workers.Done() + for j := 0; j < 30; j++ { + memory.record("model", "chat", j%2 == 0, time.Now()) + memory.demote([]string{"model", "other"}, "chat", "", time.Now()) + } + }() + } + workers.Wait() + if len(memory.entries["model|chat"].samples) != 10 { + t.Fatal("outcome window was not bounded") + } +} diff --git a/proxy/proxy.go b/proxy/proxy.go index 564eec2..8d1e4f8 100644 --- a/proxy/proxy.go +++ b/proxy/proxy.go @@ -73,6 +73,11 @@ type Server struct { routingConfig router.RoutingConfig modelPricing map[string]router.ModelPricing httpClient *http.Client + catalogMu sync.RWMutex + catalog *catalogSnapshot + catalogCtx context.Context + catalogCancel context.CancelFunc + catalogStart sync.Once // Middleware components dedup *dedup.Deduplicator @@ -81,6 +86,7 @@ type Server struct { journal *journal.SessionJournal spendControl *spendcontrol.SpendControl spendError error + outcomes outcomeMemory // Deferred startup for OpenClaw plugin config (upstream v0.12.142) // When OpenClaw calls Register() twice, the first call has empty pluginConfig. @@ -106,7 +112,9 @@ func New(cfg Config) *Server { if sc == nil { sc, spendErr = spendcontrol.New(spendcontrol.NewFileStorage()) } + catalogCtx, catalogCancel := context.WithCancel(context.Background()) return &Server{ + catalogCtx: catalogCtx, catalogCancel: catalogCancel, config: cfg, routingConfig: rc, modelPricing: models.BuildPricingMap(), @@ -126,6 +134,7 @@ func New(cfg Config) *Server { // Close shuts down the proxy server and its components. func (s *Server) Close() { + s.catalogCancel() s.startMu.Lock() if s.deferTimer != nil { s.deferTimer.Stop() @@ -181,6 +190,14 @@ func (s *Server) IsRegistered() bool { // ListenAndServe starts the proxy server. func (s *Server) ListenAndServe() error { + s.startCatalogRefresh() + addr := fmt.Sprintf("127.0.0.1:%d", s.config.Port) + log.Printf("DOSRouter proxy listening on %s (upstream: %s)", addr, s.config.UpstreamBase) + return http.ListenAndServe(addr, s.Handler()) +} + +// Handler exposes the same local-client protections for embedded servers. +func (s *Server) Handler() http.Handler { mux := http.NewServeMux() mux.HandleFunc("/v1/chat/completions", s.handleChatCompletions) mux.HandleFunc("/v1/images/generations", s.handleImageGen) @@ -193,9 +210,7 @@ func (s *Server) ListenAndServe() error { // Registers: /v1/stocks/*, /v1/crypto/price/*, /v1/fx/price/*, /v1/commodity/price/*. partners.NewMarketHandler().Routes(mux) - addr := fmt.Sprintf(":%d", s.config.Port) - log.Printf("DOSRouter proxy listening on %s (upstream: %s)", addr, s.config.UpstreamBase) - return http.ListenAndServe(addr, mux) + return localOnly(mux) } // chatRequest is the OpenAI-compatible request format. @@ -217,6 +232,7 @@ type chatMessage struct { } func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) { + catalog := s.catalogView() if r.Method != http.MethodPost { http.Error(w, "Method not allowed", http.StatusMethodNotAllowed) return @@ -321,10 +337,11 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) { } d, err := router.Route(prompt, systemPrompt, maxOutputTokens, router.RouterOptions{ - Config: s.routingConfig, - ModelPricing: s.modelPricing, - RoutingProfile: routingProfile, - HasTools: requestHasTools(req.Tools), + Config: s.routingConfig, + ModelPricing: catalog.pricing, + RoutingProfile: routingProfile, + HasTools: requestHasTools(req.Tools), + RequiresStructuredOutput: requiresStructuredOutput(req), }) if err != nil { http.Error(w, "Routing error: "+err.Error(), http.StatusInternalServerError) @@ -383,11 +400,6 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) { // Rewrite model in request body req.Model = resolvedModel - // Normalize assistant messages for reasoning models (upstream v0.12.92) - if models.IsReasoningModel(resolvedModel) { - req.Messages = normalizeMessagesForThinking(req.Messages) - } - // Validate request can be marshaled (actual marshaling happens per-model in fallback loop) if _, err := json.Marshal(req); err != nil { http.Error(w, "Failed to marshal request", http.StatusInternalServerError) @@ -429,11 +441,27 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) { // Build fallback chain: if smart-routed, try all models in the tier var fallbackChain []string if decision != nil { - fallbackChain = router.GetFallbackChain(decision.Tier, decision.TierConfigs) + fallbackChain = smartFallbackChain(req, decision, catalog) + if len(fallbackChain) == 0 { + http.Error(w, "No eligible routing candidates", http.StatusServiceUnavailable) + return + } } if len(fallbackChain) == 0 { fallbackChain = []string{resolvedModel} } + requestKind := "chat" + if requestHasTools(req.Tools) { + requestKind = "tools" + } + keep := "" + if userExplicit { + keep = resolvedModel + } + fallbackChain = s.outcomes.demote(fallbackChain, requestKind, keep, time.Now()) + if len(fallbackChain) > 5 { + fallbackChain = fallbackChain[:5] + } type attemptResult struct { model string @@ -441,6 +469,7 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) { } var attempts []attemptResult var resp *http.Response + var respBody []byte var spend *requestSpend accounted := make(map[*requestSpend]bool) finalStatus := "interrupted" @@ -469,8 +498,12 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) { req.MaxTokens = 4096 } req.Model = tryModel - tryBody, _ := json.Marshal(req) - currentSpend, ok := s.reserveChat(w, req, tryBody, tryModel) + tryReq := req + if m := catalog.model(tryModel); m != nil && m.Reasoning { + tryReq.Messages = normalizeMessagesForThinking(req.Messages) + } + tryBody, _ := json.Marshal(tryReq) + currentSpend, ok := s.reserveChatWithPricing(w, tryReq, tryBody, tryModel, catalog.pricing) if !ok { return } @@ -488,8 +521,13 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) { // clearTimeout-on-success behavior. Derived from r.Context() so a client // disconnect still cancels the in-flight attempt. attemptCtx, cancelAttempt := context.WithCancel(r.Context()) - timer := time.AfterFunc(perModelTimeout(tryModel), cancelAttempt) + attemptTimeout := perModelTimeout(tryModel) + if m := catalog.model(tryModel); m != nil && m.Reasoning { + attemptTimeout = perModelTimeoutReasoning + } + timer := time.AfterFunc(attemptTimeout, cancelAttempt) tryResp, tryErr := retry.Do(attemptCtx, makeReqFor(tryBody), retry.WithClient(s.httpClient), retry.WithNetworkRetries(false), retry.WithMaxRetries(0)) + timedOut := attemptCtx.Err() != nil if tryResp != nil && tryResp.StatusCode >= 300 { tryErr = nil } @@ -503,6 +541,9 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) { if r.Context().Err() != nil { return } + if timedOut { + s.outcomes.record(tryModel, requestKind, false, time.Now()) + } attempts = append(attempts, attemptResult{model: tryModel, reason: tryErr.Error()}) break } @@ -515,7 +556,19 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) { currentSpend.release() accounted[currentSpend] = true } - errBody, _ := io.ReadAll(tryResp.Body) + errBody, readErr := readResponseBody(tryResp.Body, maxErrorResponseBytes) + if readErr != nil { + tryResp.Body.Close() + timer.Stop() + cancelAttempt() + if r.Context().Err() == nil { + http.Error(w, "Incomplete upstream error response", http.StatusBadGateway) + } + return + } + if r.Context().Err() == nil && tryResp.StatusCode >= 500 && !paymentFailure(errBody) && !strings.Contains(strings.ToLower(string(errBody)), "overload") { + s.outcomes.record(tryModel, requestKind, false, time.Now()) + } tryResp.Body.Close() timer.Stop() cancelAttempt() @@ -536,15 +589,52 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) { // Success: stop the per-model timer so streaming is not cut at the bound, // and keep the context alive (cancel deferred after the loop). timer.Stop() + if !req.Stream { + var readErr error + if id := gatewayRequestID(tryResp.Header); id != "" { + w.Header().Set("X-DOSRouter-Request-Id", id) + } + respBody, readErr = readResponseBody(tryResp.Body, maxChatResponseBytes) + tryResp.Body.Close() + currentSpend.header = tryResp.Header + if readErr != nil { + finalizeSpend(currentSpend, "interrupted") + cancelAttempt() + if r.Context().Err() == nil { + s.outcomes.record(tryModel, requestKind, false, time.Now()) + message := "Incomplete upstream response" + if len(attempts) > 0 { + message = "Incomplete fallback response" + } + http.Error(w, message, http.StatusBadGateway) + } + return + } + if isEmptyTurn(respBody) && !paymentFailure(respBody) { + s.outcomes.record(tryModel, requestKind, false, time.Now()) + if decision != nil { + currentSpend.readUsage(respBody) + finalizeSpend(currentSpend, "empty") + cancelAttempt() + attempts = append(attempts, attemptResult{model: tryModel, reason: "empty turn"}) + continue + } + } else if !paymentFailure(respBody) { + s.outcomes.record(tryModel, requestKind, true, time.Now()) + } + } cancelResp = cancelAttempt resp = tryResp spend = currentSpend spend.header = resp.Header resolvedModel = tryModel - if tryModel != fallbackChain[0] { + if decision != nil && tryModel != decision.Model { w.Header().Set("X-DOSRouter-Fallback", tryModel) w.Header().Set("X-DOSRouter-Model", tryModel) } + if decision != nil && sessionID != "" { + s.sessions.SetSession(sessionID, tryModel, decision.Tier, false) + } break } if cancelResp != nil { @@ -596,13 +686,21 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) { scanner := bufio.NewScanner(resp.Body) scanner.Buffer(make([]byte, 64*1024), 1024*1024) var streamInputTok, streamOutputTok int + streamHasOutput, streamComplete, streamFailure, streamPaymentFailure := false, false, false, false proseFilters := make(map[int]*proseFilter) for scanner.Scan() { line := scanner.Text() + if line == "data: [DONE]" { + streamComplete = true + } // Parse and rewrite streaming chunks: inject model name + track usage if strings.HasPrefix(line, "data: ") && line != "data: [DONE]" { var chunk map[string]interface{} if json.Unmarshal([]byte(line[6:]), &chunk) == nil { + if chunk["error"] != nil { + streamFailure = true + streamPaymentFailure = streamPaymentFailure || paymentFailure([]byte(line)) + } // Track usage tokens if u, ok := chunk["usage"].(map[string]interface{}); ok { pt, inputOK := validTokenCount(u["prompt_tokens"]) @@ -636,6 +734,15 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) { delta["content"] = cleaned mutated = true } + if strings.TrimSpace(cleaned) != "" { + streamHasOutput = true + } + if calls, ok := delta["tool_calls"].([]interface{}); ok && len(calls) > 0 { + streamHasOutput = true + } + if finish != "" { + streamComplete = true + } } if choiceEndsWithToolCalls(choice) && !forwardToolCallProse() { if delta, ok := choice["delta"].(map[string]interface{}); ok { @@ -665,7 +772,7 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) { // Intercept [DONE] to inject cost usage chunk if line == "data: [DONE]" { if decision != nil && (streamInputTok > 0 || streamOutputTok > 0) { - cb := buildCostBreakdown(resolvedModel, string(decision.Tier), decision.Profile, s.modelPricing, streamInputTok, streamOutputTok) + cb := buildCostBreakdown(resolvedModel, string(decision.Tier), decision.Profile, catalog.pricing, streamInputTok, streamOutputTok) if cb != nil { usageChunk := map[string]interface{}{ "id": fmt.Sprintf("chatcmpl-%d", time.Now().UnixMilli()), @@ -695,91 +802,16 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) { flusher.Flush() } if scanner.Err() != nil || r.Context().Err() != nil { - return - } - spend.input, spend.output = streamInputTok, streamOutputTok - } else { - respBody, readErr := io.ReadAll(resp.Body) - if readErr != nil { - if r.Context().Err() == nil { - http.Error(w, "Incomplete upstream response", http.StatusBadGateway) + if r.Context().Err() == nil && !streamPaymentFailure { + s.outcomes.record(resolvedModel, requestKind, false, time.Now()) } return } - - // --- Empty turn fallback detection --- - // If the response has empty content, no tool_calls, and finish_reason "stop", - // treat as degraded and try the next model in the fallback chain. - if resp.StatusCode == http.StatusOK && decision != nil && isEmptyTurn(respBody) { - nextModel := "" - for i, m := range fallbackChain { - if m == resolvedModel && i+1 < len(fallbackChain) { - nextModel = fallbackChain[i+1] - break - } - } - if nextModel != "" { - log.Printf("degraded response: empty turn from %s, falling back to %s", resolvedModel, nextModel) - req.Model = nextModel - fbBody, _ := json.Marshal(req) - - // Re-pin session to fallback model - if sessionID != "" { - s.sessions.SetSession(sessionID, nextModel, decision.Tier, userExplicit) - } - - spend.readUsage(respBody) - finalizeSpend(spend, "empty") - spend = nil - nextSpend, allowed := s.reserveChat(w, req, fbBody, nextModel) - if !allowed { - return - } - spend = nextSpend - fbResp, fbErr := retry.Do(r.Context(), makeReqFor(fbBody), retry.WithClient(s.httpClient), retry.WithNetworkRetries(false), retry.WithMaxRetries(0)) - if fbResp != nil && fbResp.StatusCode >= 300 { - fbErr = nil - } - if fbErr != nil { - finalizeSpend(nextSpend, "error") - if fbResp != nil { - fbResp.Body.Close() - } - if r.Context().Err() == nil { - http.Error(w, "Fallback request failed", http.StatusBadGateway) - } - return - } - defer fbResp.Body.Close() - if fbResp.StatusCode >= 300 { - nextSpend.header = fbResp.Header - if _, settled := settledCost(fbResp.Header); settled || fbResp.StatusCode >= 500 { - finalizeSpend(nextSpend, "error") - } else { - nextSpend.release() - accounted[nextSpend] = true - } - http.Error(w, "Fallback request rejected", http.StatusBadGateway) - return - } - spend = nextSpend - spend.header = fbResp.Header - w.Header().Del("X-DOSRouter-Request-Id") - if id := gatewayRequestID(fbResp.Header); id != "" { - w.Header().Set("X-DOSRouter-Request-Id", id) - } - respBody, readErr = io.ReadAll(fbResp.Body) - if readErr != nil { - http.Error(w, "Incomplete fallback response", http.StatusBadGateway) - return - } - resp = fbResp - resolvedModel = nextModel - w.Header().Set("X-DOSRouter-Fallback", nextModel) - w.Header().Set("X-DOSRouter-Model", nextModel) - - } + if !streamPaymentFailure { + s.outcomes.record(resolvedModel, requestKind, streamHasOutput && streamComplete && !streamFailure, time.Now()) } + spend.input, spend.output = streamInputTok, streamOutputTok + } else { spend.readUsage(respBody) // Inject usage.cost into non-streaming response (upstream v0.12.146) @@ -834,7 +866,7 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) { if usage, ok := parsed["usage"].(map[string]interface{}); ok && decision != nil { inputTok, _ := usage["prompt_tokens"].(float64) outputTok, _ := usage["completion_tokens"].(float64) - cb := buildCostBreakdown(resolvedModel, string(decision.Tier), decision.Profile, s.modelPricing, int(inputTok), int(outputTok)) + cb := buildCostBreakdown(resolvedModel, string(decision.Tier), decision.Profile, catalog.pricing, int(inputTok), int(outputTok)) if cb != nil { usage["cost"] = cb } @@ -948,7 +980,7 @@ func (s *Server) handleModels(w http.ResponseWriter, r *http.Request) { } var data []modelEntry - for _, m := range models.Models { + for _, m := range s.catalogView().models { if m.Deprecated { continue } @@ -1037,7 +1069,7 @@ func (s *Server) handleDebug(w http.ResponseWriter, r *http.Request) { decision, _ := router.Route(req.Prompt, req.SystemPrompt, maxTokens, router.RouterOptions{ Config: s.routingConfig, - ModelPricing: s.modelPricing, + ModelPricing: s.catalogView().pricing, RoutingProfile: profile, HasTools: req.HasTools, }) @@ -1163,8 +1195,8 @@ func isEmptyTurn(body []byte) bool { var resp struct { Choices []struct { Message struct { - Content string `json:"content"` - ToolCalls json.RawMessage `json:"tool_calls"` + Content string `json:"content"` + ToolCalls []json.RawMessage `json:"tool_calls"` } `json:"message"` FinishReason string `json:"finish_reason"` } `json:"choices"` @@ -1172,10 +1204,12 @@ func isEmptyTurn(body []byte) bool { if json.Unmarshal(body, &resp) != nil || len(resp.Choices) == 0 { return false } - c := resp.Choices[0] - return c.Message.Content == "" && - (len(c.Message.ToolCalls) == 0 || string(c.Message.ToolCalls) == "null") && - c.FinishReason == "stop" + for _, c := range resp.Choices { + if strings.TrimSpace(c.Message.Content) != "" || len(c.Message.ToolCalls) > 0 || c.FinishReason != "stop" { + return false + } + } + return true } // CostBreakdown is the usage.cost payload injected into every routed response. diff --git a/proxy/response.go b/proxy/response.go new file mode 100644 index 0000000..3652dd8 --- /dev/null +++ b/proxy/response.go @@ -0,0 +1,22 @@ +package proxy + +import ( + "errors" + "io" +) + +const ( + maxChatResponseBytes int64 = 64 << 20 + maxErrorResponseBytes int64 = 1 << 20 +) + +func readResponseBody(body io.Reader, limit int64) ([]byte, error) { + data, err := io.ReadAll(io.LimitReader(body, limit+1)) + if err != nil { + return nil, err + } + if int64(len(data)) > limit { + return nil, errors.New("upstream response too large") + } + return data, nil +} diff --git a/proxy/response_test.go b/proxy/response_test.go new file mode 100644 index 0000000..bae1eab --- /dev/null +++ b/proxy/response_test.go @@ -0,0 +1,25 @@ +package proxy + +import ( + "strings" + "testing" +) + +func TestUpstreamBodyBound(t *testing.T) { + for _, size := range []int{7, 8, 9, 20} { + input := strings.NewReader(strings.Repeat("x", size)) + body, err := readResponseBody(input, 8) + if size <= 8 { + if err != nil || len(body) != size { + t.Fatalf("valid body rejected: %d", size) + } + } else { + if err == nil || len(body) != 0 { + t.Fatalf("oversized response retained: size=%d read=%d", size, len(body)) + } + if input.Size()-int64(input.Len()) > 9 { + t.Fatal("reader consumed past the bound") + } + } + } +} diff --git a/proxy/spending.go b/proxy/spending.go index d560080..0b8cade 100644 --- a/proxy/spending.go +++ b/proxy/spending.go @@ -24,16 +24,22 @@ type requestSpend struct { header http.Header finished bool source string + price router.ModelPricing } func (s *Server) reserveChat(w http.ResponseWriter, req chatRequest, body []byte, model string) (*requestSpend, bool) { + return s.reserveChatWithPricing(w, req, body, model, s.catalogView().pricing) +} + +func (s *Server) reserveChatWithPricing(w http.ResponseWriter, req chatRequest, body []byte, model string, pricing map[string]router.ModelPricing) (*requestSpend, bool) { if s.spendError != nil || s.spendControl == nil { http.Error(w, "Spending state unavailable", http.StatusServiceUnavailable) return nil, false } - price, known := s.modelPricing[model] + price, known := pricing[model] limited := len(s.spendControl.GetLimits()) > 0 - if !known && limited { + zeroCost := price.FlatPrice == nil && price.InputPrice == 0 && price.OutputPrice == 0 || price.FlatPrice != nil && *price.FlatPrice == 0 + if limited && (!known || zeroCost && !strings.HasPrefix(model, "free/")) { http.Error(w, "Unknown model cost under configured spend limits", http.StatusTooManyRequests) return nil, false } @@ -79,7 +85,7 @@ func (s *Server) reserveChat(w http.ResponseWriter, req chatRequest, body []byte json.NewEncoder(w).Encode(map[string]any{"error": check.Reason, "blockedBy": check.BlockedBy, "remaining": check.Remaining}) return nil, false } - return &requestSpend{server: s, id: id, model: model, estimate: estimate, cost: estimate, source: "estimate", completions: n}, true + return &requestSpend{server: s, id: id, model: model, estimate: estimate, cost: estimate, source: "estimate", completions: n, price: price}, true } func (sp *requestSpend) readUsage(body []byte) { @@ -106,7 +112,8 @@ func (sp *requestSpend) finish(body []byte) { if cost, ok := settledCost(sp.header); ok { sp.cost, sp.source = cost, "gateway" } else if (sp.usageKnown || sp.input > 0 || sp.output > 0) && sp.input >= 0 && sp.output >= 0 { - if p, ok := sp.server.modelPricing[sp.model]; ok { + { + p := sp.price sp.cost = (float64(sp.input)*p.InputPrice + float64(sp.output)*p.OutputPrice) / 1_000_000 if p.FlatPrice != nil { sp.cost = *p.FlatPrice * float64(sp.completions) diff --git a/proxy/sync_october_test.go b/proxy/sync_october_test.go new file mode 100644 index 0000000..fe333fd --- /dev/null +++ b/proxy/sync_october_test.go @@ -0,0 +1,80 @@ +package proxy + +import ( + "encoding/json" + "fmt" + "io" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/DOS/DOSRouter/router" +) + +func TestSyncRecurringFailuresAreTriedLast(t *testing.T) { + calls := []string{} + srv, _ := syncTestServer(t, func(w http.ResponseWriter, r *http.Request) { + var req chatRequest + json.NewDecoder(r.Body).Decode(&req) + calls = append(calls, req.Model) + if req.Model == "openai/gpt-4o-mini" { + http.Error(w, "model failure", 502) + return + } + syncTestOK(w) + }, func(c *Config) { + cfg := router.DefaultRoutingConfig() + for tier := range cfg.Tiers { + cfg.Tiers[tier] = router.TierConfig{Primary: "openai/gpt-4o-mini", Fallback: []string{"deepseek/deepseek-chat"}} + } + c.RoutingConfig = &cfg + }) + for i := 0; i < 4; i++ { + req := httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(fmt.Sprintf(`{"model":"auto","messages":[{"role":"user","content":"hello %d"}]}`, i))) + req.Header.Set("Authorization", "Bearer test-only") + w := httptest.NewRecorder() + srv.handleChatCompletions(w, req) + if w.Code != 200 { + t.Fatalf("request %d: %d %s", i, w.Code, w.Body.String()) + } + } + if len(calls) != 7 || calls[len(calls)-1] != "deepseek/deepseek-chat" { + t.Fatalf("recurring failure was not demoted: %v", calls) + } + w := syncTestChat(t, srv, `{"model":"openai/gpt-4o-mini","messages":[{"role":"user","content":"pinned"}]}`) + if w.Code != 502 || calls[len(calls)-1] != "openai/gpt-4o-mini" { + t.Fatal("explicit pin displaced") + } +} + +func TestSyncEmptyTurnsUseWholeFallbackChain(t *testing.T) { + calls := 0 + srv, _ := syncTestServer(t, func(w http.ResponseWriter, r *http.Request) { + calls++ + if calls < 3 { + io.WriteString(w, `{"choices":[{"message":{"content":""},"finish_reason":"stop"}]}`) + return + } + syncTestOK(w) + }, func(c *Config) { + cfg := router.DefaultRoutingConfig() + for tier := range cfg.Tiers { + cfg.Tiers[tier] = router.TierConfig{Primary: "openai/gpt-4o-mini", Fallback: []string{"deepseek/deepseek-chat", "google/gemini-3.8-flash"}} + } + c.RoutingConfig = &cfg + }) + w := syncTestChat(t, srv, `{"model":"auto","messages":[{"role":"user","content":"hello"}]}`) + if w.Code != 200 || calls != 3 || !strings.Contains(w.Body.String(), "Done.") { + t.Fatalf("empty turn recovery: status=%d calls=%d body=%s", w.Code, calls, w.Body.String()) + } +} + +func TestEmptyToolArraysAreEmptyTurns(t *testing.T) { + if !isEmptyTurn([]byte(`{"choices":[{"message":{"content":" ","tool_calls":[]},"finish_reason":"stop"}]}`)) { + t.Fatal("empty tool array hid an empty response") + } + if isEmptyTurn([]byte(`{"choices":[{"message":{"content":""},"finish_reason":"stop"},{"message":{"content":"usable"},"finish_reason":"stop"}]}`)) { + t.Fatal("usable alternate completion was discarded") + } +} diff --git a/router/config.go b/router/config.go index 85dcf8d..fba1af1 100644 --- a/router/config.go +++ b/router/config.go @@ -17,7 +17,7 @@ func DefaultRoutingConfig() RoutingConfig { Scoring: defaultScoringConfig(), - // Tier chains from router-core 5ee7c23c993013a8052588191569db5cf7fb793c. + // Tier chains from router-core 8773a9954b091bd0090acdfec67c75a006c0e156. // NVIDIA free IDs use the DOS catalog namespace. Tiers: map[Tier]TierConfig{ TierSimple: { @@ -84,7 +84,7 @@ func DefaultRoutingConfig() RoutingConfig { TierSimple: { Primary: "free/nemotron-3.5-lightning", Fallback: []string{ - "free/nemotron-3-nano-30b", + "free/nemotron-3-nano-omni-30b-a3b-reasoning", "google/gemini-2.5-flash-lite", "zai/glm-5.3-flash", "openai/gpt-5.6-luna", diff --git a/router/rules.go b/router/rules.go index 2a90393..ceca8e6 100644 --- a/router/rules.go +++ b/router/rules.go @@ -70,7 +70,7 @@ func ClassifyByRules(prompt, systemPrompt string, estimatedTokens int, config Sc // Count reasoning markers for override - only check USER prompt var reasoningMatches int for _, kw := range config.ReasoningKeywords { - if strings.Contains(userText, strings.ToLower(kw)) { + if matchesKeyword(userText, kw) { reasoningMatches++ } } @@ -161,7 +161,7 @@ func scoreKeywordMatch(text string, keywords []string, name, signalLabel string, var matches []string for _, kw := range keywords { - if strings.Contains(text, strings.ToLower(kw)) { + if matchesKeyword(text, kw) { matches = append(matches, kw) } } @@ -224,7 +224,7 @@ func scoreAgenticTask(text string, keywords []string) (DimensionScore, float64) var signals []string for _, kw := range keywords { - if strings.Contains(text, strings.ToLower(kw)) { + if matchesKeyword(text, kw) { matchCount++ if len(signals) < 3 { signals = append(signals, kw) diff --git a/router/selector.go b/router/selector.go index 5cae29d..e11b8f9 100644 --- a/router/selector.go +++ b/router/selector.go @@ -3,11 +3,11 @@ package router import "math" const ( - baselineModelID = "anthropic/claude-opus-4.6" - baselineInputPrice = 5.0 // per 1M tokens - baselineOutputPrice = 25.0 // per 1M tokens - serverMarginPercent = 5 - minPaymentUSD = 0.001 + baselineModelID = "anthropic/claude-opus-4.7" + baselineInputPrice = 5.0 // per 1M tokens + baselineOutputPrice = 25.0 // per 1M tokens + serverMarginPercent = 5 + minPaymentUSD = 0.001 ) // SelectModel picks the primary model for a tier and builds a RoutingDecision. @@ -52,7 +52,7 @@ func SelectModel( float64(maxOutputTokens)/1_000_000*opusOut var savings float64 - if routingProfile != "premium" && baselineCost > 0 { + if hasPricing && routingProfile != "premium" && baselineCost > 0 { savings = math.Max(0, (baselineCost-costEstimate)/baselineCost) } @@ -111,7 +111,7 @@ func CalculateModelCost( baselineCost = float64(estimatedInputTokens)/1_000_000*opusIn + float64(maxOutputTokens)/1_000_000*opusOut - if routingProfile != "premium" && baselineCost > 0 { + if hasPricing && routingProfile != "premium" && baselineCost > 0 { savings = math.Max(0, (baselineCost-costEstimate)/baselineCost) } return diff --git a/router/strategy.go b/router/strategy.go index c3d6847..7e73710 100644 --- a/router/strategy.go +++ b/router/strategy.go @@ -3,8 +3,6 @@ package router import ( "fmt" "log" - "math" - "regexp" "strings" "time" ) @@ -85,9 +83,10 @@ func (s *RulesStrategy) Name() string { return "rules" } func (s *RulesStrategy) Route(prompt string, systemPrompt string, maxOutputTokens int, options RouterOptions) RoutingDecision { config := options.Config - // Estimate input tokens (~4 chars per token) + // Size all context while scanning a bounded sample for request intent. fullText := systemPrompt + " " + prompt - estimatedTokens := int(math.Ceil(float64(len(fullText)) / 4)) + estimatedTokens := EstimateTokens(fullText) + prompt = sampleForScan(prompt, config.Classifier.PromptTruncationChars) // Rule-based classification ruleResult := ClassifyByRules(prompt, systemPrompt, estimatedTokens, config.Scoring) @@ -186,11 +185,7 @@ func (s *RulesStrategy) Route(prompt string, systemPrompt string, maxOutputToken } // Structured output detection - hasStructuredOutput := false - if systemPrompt != "" { - structuredRe := regexp.MustCompile(`(?i)json|structured|schema`) - hasStructuredOutput = structuredRe.MatchString(systemPrompt) - } + hasStructuredOutput := options.RequiresStructuredOutput var tier Tier var confidence float64 diff --git a/router/sync_test.go b/router/sync_test.go new file mode 100644 index 0000000..c463a51 --- /dev/null +++ b/router/sync_test.go @@ -0,0 +1,58 @@ +package router + +import ( + "strings" + "testing" +) + +func TestSyncKeywordBoundaries(t *testing.T) { + for _, pair := range [][2]string{{"classic", "class"}, {"history", "story"}, {"credit", "edit"}, {"prefix", "fix"}, {"step 10", "step 1"}} { + d := scoreKeywordMatch(pair[0], []string{pair[1]}, "test", "test", 1, 2, 0, 0.5, 1) + if d.Score != 0 { + t.Errorf("%q matched %q", pair[0], pair[1]) + } + } + for _, pair := range [][2]string{{"classes", "class"}, {"debugging", "debug"}, {"implementing", "implement"}, {"请step by step证明", "step by step"}} { + d := scoreKeywordMatch(pair[0], []string{pair[1]}, "test", "test", 1, 2, 0, 0.5, 1) + if d.Score == 0 { + t.Errorf("%q did not match %q", pair[0], pair[1]) + } + } +} + +func TestSyncHostSchemasDoNotUpgradeUserTurn(t *testing.T) { + cfg := DefaultRoutingConfig() + cfg.Scoring.ConfidenceThreshold = 0 + cfg.Overrides.StructuredOutputMinTier = TierReasoning + opt := RouterOptions{Config: cfg, ModelPricing: map[string]ModelPricing{}} + a := (&RulesStrategy{}).Route("hello", "", 16, opt) + b := (&RulesStrategy{}).Route("hello", "Tools use JSON structured schema. Be quick.", 16, opt) + if a.Tier != b.Tier { + t.Fatalf("host schema changed tier: %s -> %s", a.Tier, b.Tier) + } +} + +func TestSyncCJKCapacityEstimate(t *testing.T) { + cfg := DefaultRoutingConfig() + cfg.Overrides.MaxTokensForceComplex = 900 + d := (&RulesStrategy{}).Route(strings.Repeat("中", 1000), "", 16, RouterOptions{Config: cfg}) + if !strings.Contains(d.Reasoning, "Input exceeds") { + t.Fatalf("CJK request undercounted: %s", d.Reasoning) + } +} + +func TestSyncUnknownPriceDoesNotClaimSavings(t *testing.T) { + d := SelectModel(TierSimple, 1, "rules", "", map[Tier]TierConfig{TierSimple: {Primary: "unknown/model"}}, nil, 100, 100, "auto", 0) + if d.Savings != 0 { + t.Fatalf("unknown price claims savings: %g", d.Savings) + } +} + +func TestUnsetScanLimitUsesUsefulDefault(t *testing.T) { + text := strings.Repeat("x ", 300) + "prove" + for _, limit := range []int{0, -1} { + if got := sampleForScan(text, limit); !strings.Contains(got, "prove") || len(got) > 501 { + t.Fatalf("invalid default sample: %q", got) + } + } +} diff --git a/router/text.go b/router/text.go new file mode 100644 index 0000000..940a3ac --- /dev/null +++ b/router/text.go @@ -0,0 +1,108 @@ +package router + +import ( + "strings" + "unicode" + "unicode/utf16" + "unicode/utf8" +) + +// EstimateTokens follows router-core 0.3's CJK-aware, model-agnostic estimate. +func EstimateTokens(text string) int { + cjk, other := 0, 0 + for _, r := range text { + if isCJK(r) { + cjk++ + } else { + other += utf16.RuneLen(r) + } + } + return cjk + (other+3)/4 +} + +func isCJK(r rune) bool { + return r >= 0x3000 && r <= 0x30ff || r >= 0x3400 && r <= 0x4dbf || r >= 0x4e00 && r <= 0x9fff || r >= 0xac00 && r <= 0xd7af || r >= 0xf900 && r <= 0xfaff || r >= 0xff00 && r <= 0xffef +} + +func sampleForScan(text string, limit int) string { + if limit < 1 { + limit = 500 // Match the configured Go default when a caller omits this field. + } + if limit > 8000 { + limit = 8000 + } + r := []rune(text) + if len(r) <= limit { + return text + } + left := (limit + 1) / 2 + return string(r[:left]) + "\n" + string(r[len(r)-(limit-left):]) +} + +func segmentedWord(r rune) bool { + return unicode.Is(unicode.Latin, r) || unicode.Is(unicode.Cyrillic, r) || unicode.Is(unicode.Greek, r) || unicode.IsNumber(r) +} + +// matchesKeyword preserves stems and inflections without matching unrelated words. +func matchesKeyword(text, keyword string) bool { + kw := strings.ToLower(keyword) + if kw == "" { + return true + } + for _, r := range kw { + if isCJK(r) || r >= 0x600 && r <= 0x6ff { + return strings.Contains(text, kw) + } + } + last := kw[strings.LastIndex(kw, " ")+1:] + ascii := true + for _, r := range kw { + if !(r >= 'a' && r <= 'z' || r >= '0' && r <= '9' || r == '\'' || r == ' ' || r == '-') { + ascii = false + } + } + forms := []string{kw} + end, _ := utf8.DecodeLastRuneInString(kw) + bounded := ascii && len(last) <= 6 && (unicode.IsLetter(end) || unicode.IsNumber(end)) + if bounded && !unicode.IsDigit(end) { + suffixes := []string{"s", "ed", "ing"} + if len(last) <= 3 { + if strings.HasSuffix(kw, "x") || strings.HasSuffix(kw, "s") || strings.HasSuffix(kw, "z") || strings.HasSuffix(kw, "ch") || strings.HasSuffix(kw, "sh") { + suffixes = append(suffixes, "es") + } + } else { + suffixes = []string{"s", "es", "ed", "d", "ing", "er", "ers", "ly", "ment", "ments", "ation", "ations", "e", "en", "n", "a", "as", "o", "os", "em", "et", "ung", "ungen"} + if end == 'e' { + for _, suffix := range []string{"ing", "ed", "es", "er", "ers", "ation", "ations"} { + forms = append(forms, kw[:len(kw)-1]+suffix) + } + } else if end == 'y' { + forms = append(forms, kw[:len(kw)-1]+"ies", kw[:len(kw)-1]+"ied") + } else if len(kw) >= 2 && strings.ContainsRune("aeiou", rune(kw[len(kw)-2])) && strings.ContainsRune("bdgmnprt", end) { + for _, suffix := range []string{"ing", "ed", "er", "ers"} { + forms = append(forms, kw+string(end)+suffix) + } + } + } + for _, suffix := range suffixes { + forms = append(forms, kw+suffix) + } + } + first, _ := utf8.DecodeRuneInString(kw) + for _, form := range forms { + for from := 0; from <= len(text); { + pos := strings.Index(text[from:], form) + if pos < 0 { + break + } + pos += from + before, _ := utf8.DecodeLastRuneInString(text[:pos]) + after, _ := utf8.DecodeRuneInString(text[pos+len(form):]) + if (!(unicode.IsLetter(first) || unicode.IsNumber(first)) || pos == 0 || !segmentedWord(before)) && (!bounded || !segmentedWord(after)) { + return true + } + from = pos + 1 + } + } + return false +} diff --git a/router/types.go b/router/types.go index 28445b9..db74f44 100644 --- a/router/types.go +++ b/router/types.go @@ -49,17 +49,17 @@ type DimensionScore struct { // RoutingDecision is the final output: which model to use and why. type RoutingDecision struct { - Model string // selected model ID - Tier Tier // classified tier - Confidence float64 // confidence in classification - Method string // "rules" or "llm" - Reasoning string // human-readable reasoning - CostEstimate float64 // estimated cost in USD - BaselineCost float64 // what Claude Opus would cost - Savings float64 // 0-1 percentage saved vs baseline - AgenticScore float64 // 0-1 agentic task score - TierConfigs map[Tier]TierConfig // tier configs used for this decision - Profile string // "auto", "eco", "premium", "agentic" + Model string // selected model ID + Tier Tier // classified tier + Confidence float64 // confidence in classification + Method string // "rules" or "llm" + Reasoning string // human-readable reasoning + CostEstimate float64 // estimated cost in USD + BaselineCost float64 // what Claude Opus would cost + Savings float64 // 0-1 percentage saved vs baseline + AgenticScore float64 // 0-1 agentic task score + TierConfigs map[Tier]TierConfig // tier configs used for this decision + Profile string // "auto", "eco", "premium", "agentic" } // TierConfig maps a tier to its primary model and fallback chain. @@ -70,11 +70,12 @@ type TierConfig struct { // RouterOptions carries configuration for a single routing call. type RouterOptions struct { - Config RoutingConfig - ModelPricing map[string]ModelPricing - RoutingProfile string // "eco", "auto", "premium" - HasTools bool - Now *int64 // unix millis override for promotion window checks (testing) + Config RoutingConfig + ModelPricing map[string]ModelPricing + RoutingProfile string // "eco", "auto", "premium" + HasTools bool + RequiresStructuredOutput bool // Explicit response_format, never inferred from host prose. + Now *int64 // unix millis override for promotion window checks (testing) } // ModelPricing holds per-model pricing info. @@ -86,15 +87,15 @@ type ModelPricing struct { // RoutingConfig is the top-level configuration. type RoutingConfig struct { - Version string `json:"version"` - Classifier ClassifierConfig `json:"classifier"` - Scoring ScoringConfig `json:"scoring"` - Tiers map[Tier]TierConfig `json:"tiers"` - AgenticTiers map[Tier]TierConfig `json:"agenticTiers,omitempty"` - EcoTiers map[Tier]TierConfig `json:"ecoTiers,omitempty"` - PremiumTiers map[Tier]TierConfig `json:"premiumTiers,omitempty"` - Promotions []Promotion `json:"promotions,omitempty"` - Overrides OverridesConfig `json:"overrides"` + Version string `json:"version"` + Classifier ClassifierConfig `json:"classifier"` + Scoring ScoringConfig `json:"scoring"` + Tiers map[Tier]TierConfig `json:"tiers"` + AgenticTiers map[Tier]TierConfig `json:"agenticTiers,omitempty"` + EcoTiers map[Tier]TierConfig `json:"ecoTiers,omitempty"` + PremiumTiers map[Tier]TierConfig `json:"premiumTiers,omitempty"` + Promotions []Promotion `json:"promotions,omitempty"` + Overrides OverridesConfig `json:"overrides"` } // ScoringConfig controls the weighted dimension scorer. @@ -125,34 +126,34 @@ type ScoringConfig struct { ComplexReasoning float64 `json:"complexReasoning"` } `json:"tierBoundaries"` - ConfidenceSteepness float64 `json:"confidenceSteepness"` - ConfidenceThreshold float64 `json:"confidenceThreshold"` + ConfidenceSteepness float64 `json:"confidenceSteepness"` + ConfidenceThreshold float64 `json:"confidenceThreshold"` } // ClassifierConfig controls the LLM fallback classifier. type ClassifierConfig struct { - LLMModel string `json:"llmModel"` - LLMMaxTokens int `json:"llmMaxTokens"` - LLMTemperature float64 `json:"llmTemperature"` - PromptTruncationChars int `json:"promptTruncationChars"` - CacheTTLMs int64 `json:"cacheTtlMs"` + LLMModel string `json:"llmModel"` + LLMMaxTokens int `json:"llmMaxTokens"` + LLMTemperature float64 `json:"llmTemperature"` + PromptTruncationChars int `json:"promptTruncationChars"` + CacheTTLMs int64 `json:"cacheTtlMs"` } // OverridesConfig holds override rules. type OverridesConfig struct { - MaxTokensForceComplex int `json:"maxTokensForceComplex"` - StructuredOutputMinTier Tier `json:"structuredOutputMinTier"` - AmbiguousDefaultTier Tier `json:"ambiguousDefaultTier"` + MaxTokensForceComplex int `json:"maxTokensForceComplex"` + StructuredOutputMinTier Tier `json:"structuredOutputMinTier"` + AmbiguousDefaultTier Tier `json:"ambiguousDefaultTier"` AgenticMode *bool `json:"agenticMode,omitempty"` } // Promotion is a time-windowed tier override. type Promotion struct { - Name string `json:"name"` - StartDate string `json:"startDate"` - EndDate string `json:"endDate"` - TierOverrides map[Tier]PartialTierConfig `json:"tierOverrides"` - Profiles []string `json:"profiles,omitempty"` + Name string `json:"name"` + StartDate string `json:"startDate"` + EndDate string `json:"endDate"` + TierOverrides map[Tier]PartialTierConfig `json:"tierOverrides"` + Profiles []string `json:"profiles,omitempty"` } // PartialTierConfig allows partial override of a TierConfig.