diff --git a/PERF.md b/PERF.md index e6b9c9bfb2..a0bb07078b 100644 --- a/PERF.md +++ b/PERF.md @@ -1136,8 +1136,8 @@ and an explicit zero temperature or output limit is preserved. Explicit choices remain explicit. `CODEAF_REASONING`, `CODEAF_EXEC_REASONING`, a model or crew value with `:low`, `:medium` or `:high`, a saved task or standing-work rung, and an embedder's `ai.Option` still -travel. The three shipped crew presets contain bare model ids and add no effort -level. `cmd/harness-design` is a development command with explicit CLI-sized +travel. A crew seat nobody pinned is routed to a bare model id and adds no +effort level. `cmd/harness-design` is a development command with explicit CLI-sized requests and retains its caps. The local process remains bounded independently of provider generation: diff --git a/README.md b/README.md index dee128d01d..6866074014 100644 --- a/README.md +++ b/README.md @@ -149,22 +149,27 @@ repository. ## The right model for each call -One session, many models. The model you talk to is one seat. Five more, the -crew, take the calls you did not type: +One session, many models. The model you talk to is one seat. Every task you hand +off runs on a crew of three more, picked for that task from what kind of work it +is — a bug fix, a complex fix, open-ended work, or something else: -| seat | what it answers | +| seat | what it does | | --- | --- | -| reflex | memory, titles, the safety gate. Near free, reads every turn. | -| small work | digests, task names, yes-or-no checks | -| worker | every task you hand off. Most of the bill. | -| careful work | checks on finished work, the brief a task is shaped into, vision | -| mastermind | plans runs and designs subharnesses | - -`/crew frugal`, `balanced` or `max` sets all five in one word. Any seat can be -pinned. - -Every finished task is graded by the check it already had to pass. Work that -keeps failing on the worker seat moves up to careful work on its own, and each +| worker | does the work. Most of the bill. | +| planner | plans runs and designs subharnesses | +| checker | checks finished work before it lands | + +The crew is auto by default. Each seat is scored from its catalog metadata by +learned weights, moved by how this install's own tasks ended, over every model your +connected providers serve. `/crew` shows it: the seats, the allowed models, the +providers, a per-task limit ($5 unless set) and an optional daily cap, and today's +spend. `/crew pin checker ` pins one seat, `/crew models open` limits every +seat to open-weight models, and `/task --best` or `/task --cheap` moves one task. +Each task says its crew and what it cost against the estimate, and `/redo +stronger` runs it again a step up. Two cheap rows, reflex and small work, take +the small calls you did not type: memory, titles, digests, the safety gate. + +Every finished task is graded by the check it already had to pass, and each request goes to the provider that has been fastest for that kind of call. `codeaf models` prints the ratings. @@ -175,7 +180,7 @@ ChatGPT plan, Ollama and any OpenAI-compatible endpoint. ## Model Pool -The picker can choose models from what other installs have found. It is on by +Installs can pool what they have measured about models. It is on by default: what an install sends is computed, text-free numbers about the models it ran (role, model, a number, which model judged, door, size bucket, day) under a per-install nonce, never code, @@ -183,7 +188,7 @@ prompts, paths or an identity, and `codeaf pool status` shows exactly what is waiting to go. Turn it off with `model_pool = off` on the settings sheet or `CODEAF_MODEL_POOL=off`; `read` uses the pool and sends nothing, and `CODEAF_TELEMETRY=off` caps it at `read` along with the usage counts. The relay -publishes a signed index the crew picker reads under `picked from = learn`. The index is mirrored on the `model-pool` branch at +publishes a signed index of what the installs measured. The index is mirrored on the `model-pool` branch at `pool/index.json`. The design is [Pareto Crewing](docs/design/model-pool/pareto-crewing.pdf); the relay's code is under `relay/`, with a [runbook](docs/design/model-pool/RUNBOOK.md) that includes running your own. diff --git a/audit-notes/settings-inventory.md b/audit-notes/settings-inventory.md index 21ab4cc0a9..4a087e071c 100644 --- a/audit-notes/settings-inventory.md +++ b/audit-notes/settings-inventory.md @@ -39,7 +39,7 @@ sentence of the registry's own Hint (`settings.go:663`). rows; a search → every tab's matches under faint headings, and the tab bar follows the first match. - Providers is led by `modelsSection` (`settings.go:700`): your model, - provider, speed guard, routing, prompt profile, crew, then the five tier + provider, speed guard, routing, prompt profile, then the five tier rows in `roles.Tiers` order (reflex, low, worker, high, mastermind), then pinned roles. Every other row follows in registry order (`tabRows`, `settings.go:1430`). The registry builds the 10 model rows @@ -229,12 +229,11 @@ rows land between the roles section and "looking"): | lane.guard | speed guard | toggle | bool | on (settings.go:935) | "an answer that is slow to start is asked of the next-best provider as well, and you read whichever replies first. One extra call, under a tenth of spend." | registry `settings.go:2055`, skin `settings.go:703` | | routing | routing | cycle | choice: simple/latency/price/off | simple | "one model is served by many providers. simple is the one it ships with and sends no preference of ours — no pinned provider means the router's own default answers, and a pinned provider is the whole request; latency asks for the fastest and demotes one that keeps being slow; price asks for the cheapest; off asks for nothing, measures nothing, and leaves the two rows above it with no provider to name. a change here takes effect on your next message." | registry `settings.go:1999`, skin `settings.go:779` | | prompt.profile | prompt profile | cycle | choice: auto/lean/full | auto | "how much codeaf tells the model before you type. auto reads the model's context window and goes lean under 32,000 tokens; lean and full say so yourself, for a provider that reports a window its model does not really have." | registry `settings.go:2022`, skin `settings.go:710` | -| models.crew | crew | cycle | choice: frugal/balanced/max (+ reads custom) | balanced — derived; the five shipped tiers are exactly the balanced preset (`settings.go:1017-1031`) | dynamic, built by `crewAbout` (`settings.go:716`): "the five below, chosen as one word: frugal — …; balanced — …; max — …. Answer one yourself and this reads custom." (the preset sentences come from `config.CrewLineFor`) | registry `settings.go:2269`, skin `settings.go:322` | | models.tiers.reflex | reflex | select | model | mistralai/mistral-nemo (shipped, settings.go:1017) | "near-free · reads every turn — memory, titles, safety" | registry `settings.go:2297`, skin `settings.go:334` | | models.tiers.low | small work | select | model | deepseek/deepseek-v4-flash-0731 (shipped) | "cheap · the small calls — names, digests, the safety gate" | registry `settings.go:2306`, skin `settings.go:364` | -| models.tiers.worker | worker | select | model | z-ai/glm-5.3-flash (shipped) | "does the work · every task, its parts, every run node — most of the bill" | registry `settings.go:2317`, skin `settings.go:368` | -| models.tiers.high | careful work | select | model | moonshotai/kimi-k3 (shipped) | "careful · checks what must not be wrong — audits, briefs, vision" | registry `settings.go:2327`, skin `settings.go:372` | -| models.tiers.mastermind | mastermind | text | model, may carry :low/:medium/:high | moonshotai/kimi-k3 (shipped) | "thinks · plans runs and designs harnesses — add :low, :medium or :high" | registry `settings.go:2338`, skin `settings.go:380` | +| models.tiers.worker | worker | select | model | empty — auto, routed per task | "does the work · every task, its parts, every run node — most of the bill" | registry `settings.go:2317`, skin `settings.go:368` | +| models.tiers.high | checker | select | model | empty — auto, routed per task | "checks what must not be wrong — audits, briefs, vision" | registry `settings.go:2327`, skin `settings.go:372` | +| models.tiers.mastermind | planner | text | model, may carry :low/:medium/:high | empty — auto, routed per task | "plans runs and designs harnesses — add :low, :medium or :high" | registry `settings.go:2338` | | models.roles | pinned roles | text | text (role:model pairs) | none | "exceptions to the five rows above, one per role: title:openai/gpt-5-mini." | registry `settings.go:2347`, skin `settings.go:386` | | model.plan | planning | select | model | blank, "follows execution" | firstSentence of hint: "the model that plans and reviews the work." (full hint `settings.go:2965`: "the model that plans and reviews the work. Empty follows the work model.") | registry `settings.go:2686` (modelRow), skin `settings.go:683` (init) | | model.work | execution | select | model | the work model | "the model that does the work." (full hint: "the model that does the work. It changes on the next job.") | registry modelRow; skin init | @@ -249,7 +248,6 @@ rows land between the roles section and "looking"): | effort | thinking | cycle | choice: auto + the effort rungs (auto, plus five explicit levels; `EffortChoices`, `config/effort.go:29`) | auto (effort.Ship = None, `effort/effort.go:63`) | "how hard the model thinks, unless something nearer the work says otherwise. ctrl+v moves the rung of whatever you stand on — the rung beside the model above the message box for one conversation, a task, or a standing item — and ctrl+t in /model dials one model. This row answers for everything nobody dialled." | registry `settings.go:1789`, skin `settings.go:757` | | document_engine | reading | cycle | choice: auto/local/free/ocr | auto (config.go:62) | "which rung reads your documents. auto walks local, then free, then paid OCR." | registry `settings.go:1798`, skin `settings.go:680` | | api_key | openrouter key | text | text, secret | not set | "the key codeaf talks to models with. A missing default key opens connect openrouter in your browser; paste a replacement here if needed. A change lands on this conversation at once." | registry `settings.go:1829`, skin `settings.go:476` | -| models.crew.source | model family | cycle | choice: open/all | open (crew.go:86) | "which models the crew word draws from: open weights, or the whole catalog with closed and frontier models in it. Open is the default." | registry `settings.go:2284`, skin `settings.go:341` | | reply.guard | reply guard | cycle | choice: on/off | on | "on cuts a reply that has come apart — one line or one letter repeated, alphabets mixed inside words — throws it away and asks once more. Code blocks are never judged." | registry `settings.go:2417`, skin `settings.go:427` | ### The roles section (generated, not registry rows) @@ -266,8 +264,8 @@ registered from internal/session init functions. 23 rows in 5 groups: | roles · reflex | reflex | | roles · small work | title, caption, router, consolidate, task-name, job-name, intake, guardian, sentinel, spellout | | roles · worker | worker | -| roles · careful work | careful, auditor, repair, shaper, vision | -| roles · mastermind | planner, designer, mark-reader, handoff, division, router-confirm | +| roles · checker | careful, auditor, repair, shaper, vision | +| roles · planner | planner, designer, mark-reader, handoff, division, router-confirm | Each row shows the model the role resolves to right now; a pinned row also says "pinned". The one line under a row (verbatim): unpinned → " · @@ -378,12 +376,11 @@ branch can, because its registry tops out at 77 (78 with split_pct). |---|---|---| | model_pool | CategoryModels, choice, label "model pool" (dev settings.go:1911) | fde49a587, #1194 | | models.pool.public_key | CategoryModels, text, label "pool key" (dev settings.go:1926) | fde49a587, #1194 | -| models.crew.pick | CategoryModels, choice, label "picked from" (dev settings.go:2400) | fde49a587, #1194 | | telemetry | CategoryInterface, bool, label "telemetry", default on (dev settings.go:2627) | cc8bea7e9, #1095 | -The `Key:` diff between this branch's build and dev's is exactly those four +The `Key:` diff between this branch's build and dev's is exactly those three rows and nothing else: every key this branch has, dev has too, and no key dev -has is missing here but those four. The four-row difference is fixed rows that +has is missing here but those three. The three-row difference is fixed rows that landed on dev after the fork, not conditional rows and not rows this branch retired. The deletion comment at settings.go:3046 names two rows gone outright (`practice_demand_pct`, `propose_new_skills`), and an earlier draft of this @@ -465,16 +462,9 @@ for the word "budget" will not find it — though the search does match the key the panel moved the row to Spending as "per conversation" — a person reading the Session tab's promise ("this conversation and only this conversation") will not find the row that bounds this conversation. -- THE CREW ROW'S NEIGHBOUR MOVED. The skin comment on "model family" - (models.crew.source) claims "it sits directly under the crew word" - (settings.go:337), but `modelsSection` does not lead it, so on the drawn - Providers tab it reads after the openrouter-key row (and after the roles and - services sections when they stand), ~25 rows below the crew word it changes. - The comment describes registry order, not the tab's reading order — a - redesign should either lead it or fix the comment. -- "thinking" (effort) vs mastermind's ":high". Two rows both about how hard a +- "thinking" (effort) vs the planner row's ":high". Two rows both about how hard a model thinks; effort's about says it "answers for everything nobody dialled" - and names ctrl+v and /model's ctrl+t, and the mastermind row's about says + and names ctrl+v and /model's ctrl+t, and the planner row's about says "add :low, :medium or :high". The distinction (install-wide rung vs one tier's level) is stated only in the about lines. - The Connections tab name does double duty: it is the accounts tab, and the diff --git a/cmd/codeaf/chat.go b/cmd/codeaf/chat.go index c6b2774801..bb24b413f0 100644 --- a/cmd/codeaf/chat.go +++ b/cmd/codeaf/chat.go @@ -166,14 +166,14 @@ func buildBrain(w *chatWindow, session string, opts brainOptions) (*chatBrain, e // then let the machine form print underneath, which said one fact twice. return nil, err } - // A tier row that says auto is answered from this catalog (config.AutoModels), - // so the word is wired BEFORE the seats handed in are applied — a door whose - // ladder answered the word before this line resolved it from nothing. The - // read is the same non-blocking one, never a fetch. + // The crew router picks its seats from this catalog (config.CrewCatalog), + // so it is wired BEFORE the seats handed in are applied — a door whose + // ladder routed before this line would have routed from nothing. The read + // is the same non-blocking one, never a fetch. modelCatalog := catalog.LoadLazy(context.Background(), catalog.Options{ BaseURL: settings.BaseURL, APIKey: settings.APIKey, Dir: settings.ProfileDir, }) - config.AutoModels = modelCatalog.ModelsNow + seatCrewRows(modelCatalog.ModelsNow) wirePoolIndex(settings.ProfileDir) if opts.seats != nil { applySeats(&settings, *opts.seats) diff --git a/cmd/codeaf/chatv3.go b/cmd/codeaf/chatv3.go index ea05abd537..e3d8a35132 100644 --- a/cmd/codeaf/chatv3.go +++ b/cmd/codeaf/chatv3.go @@ -4,6 +4,7 @@ import ( "context" "errors" "fmt" + "github.com/Agent-Field/codeaf/internal/crewroute" "io" "os" "path/filepath" @@ -937,7 +938,7 @@ func openV3Launch(proc *v3Process, opts v3Options) (*v3Launch, error) { // engine's own nothing. The ask is built once and a client is made from it // per call, each billed to the judge's own seat. taskLanded := poolJudgeHook(settings, settings.ProfileDir, workspace, - config.AutoModels, poolJudgeAsk(settings, settings.ProfileDir), time.Now, "task") + config.CrewCatalog, poolJudgeAsk(settings, settings.ProfileDir), time.Now, "task") // The runs a live process would have judged but a process death left unjudged, // and the headless doors that never had this hook: at start, on a goroutine // nobody waits on, judge the resumed session's own final-state nodes and the @@ -945,7 +946,7 @@ func openV3Launch(proc *v3Process, opts v3Options) (*v3Launch, error) { // process tracker cancels and joins it at close. poolErrandGoCtx(settings.ProfileDir, "pool/judge-sweep", func(ctx context.Context) { poolJudgeSweepRun(ctx, settings, settings.ProfileDir, found.Place.Tasks(), - config.AutoModels, poolJudgeAsk(settings, settings.ProfileDir), time.Now) + config.CrewCatalog, poolJudgeAsk(settings, settings.ProfileDir), time.Now) }) cfg := session.Config{ @@ -1627,6 +1628,16 @@ func applyV3Governance(cfg session.Config, profileDir string, yolo, oneModel boo cfg.ApprovalPolicy = policy cfg.RolesSource = source cfg.OneModel = oneModel + // EVERY TASK THIS CONVERSATION STARTS IS ROUTED ITS OWN CREW — worker, + // planner, checker picked for that task from the profile's allowed models + // and pins (internal/config's RouteCrew, internal/session's taskcrew.go). + // Under `--one-model` there is no crew: every call rides the conversation's + // model, which is what the flag says, so no router is handed over. + if !oneModel { + cfg.RouteCrew = func(ask config.CrewAsk) (crewroute.Decision, error) { + return config.RouteCrew(profileDir, ask) + } + } cfg.SpendRailUSD = rail // The fallback chain reads PROFILE-ONLY, like the search keys below and // unlike the three rows above it. A repository that could answer this could @@ -2129,7 +2140,7 @@ func v3SearchSeam(brain *store.Store) tui3.SearchStore { // IT IS LIVE. It used to be resolved once, at boot, on the argument that two // calls in one conversation must not answer to different settings — and the // crew is what makes that argument the wrong way round. A person who types -// `/crew max` because the planner is not thinking hard enough has said something +// `/crew pin planner …` because the planner is not thinking hard enough has said something // about the run they are about to start, not about the next launch, and a source // that made them restart to be heard would be a knob that does nothing on the // surface that offers it. @@ -2260,6 +2271,12 @@ func (c *v3Crew) snapshot() (map[string]string, error) { // the cheapest question in it. The mastermind tier: it plans adaptive runs // and designs saved harnesses, and a repository that could point it at a // model would be spending a visitor's credit on the run it asked for. + // A CHECKER NO PROJECT NAMED IS THE CREW'S: its pin, or the router's + // standing pick ([config.TierModelAt]) — never an empty row that would + // fall to the conversation's model. + if strings.TrimSpace(high) == "" { + high = config.TierModelAt(c.profileDir, config.ModelTierHigh) + } values := map[string]string{ roles.TierKey(roles.TierReflex): config.TierModelAt(c.profileDir, config.ModelTierReflex), roles.TierKey(roles.TierMastermind): config.TierModelAt(c.profileDir, config.ModelTierMastermind), diff --git a/cmd/codeaf/chatv3_crew_test.go b/cmd/codeaf/chatv3_crew_test.go index dd9300eec2..adf8b5493f 100644 --- a/cmd/codeaf/chatv3_crew_test.go +++ b/cmd/codeaf/chatv3_crew_test.go @@ -9,6 +9,7 @@ import ( "testing" "github.com/Agent-Field/codeaf/internal/config" + "github.com/Agent-Field/codeaf/internal/crewroute" "github.com/Agent-Field/codeaf/internal/roles" ) @@ -17,8 +18,9 @@ import ( // // It used to be resolved once at boot, on the argument that two calls in one // conversation must not answer to different settings. /crew is what makes that -// the wrong way round — somebody who types `/crew max` because the planner is not -// thinking hard enough has said something about the run they are about to start. +// the wrong way round — somebody who types `/crew pin planner …` because the +// planner is not thinking hard enough has said something about the run they +// are about to start. func TestAMidSessionCrewChangeIsHonoredByTheNextCall(t *testing.T) { dir := t.TempDir() @@ -27,28 +29,16 @@ func TestAMidSessionCrewChangeIsHonoredByTheNextCall(t *testing.T) { t.Fatalf("v3RolesSource: %v", err) } - // The shipped crew, resolved once. Everything after this is a change made - // under a source that is already in use. - before, err := roles.Resolve(roles.Source(source), roles.RolePlanner, "vendor/conversation") - if err != nil { - t.Fatal(err) - } - shipped, _ := roles.SplitEffort(config.DefaultMastermindModel) - if before != shipped { - t.Fatalf("the planner started on %q, want the shipped mastermind's %q", before, shipped) - } - - if err := config.ApplyCrew(dir, config.CrewFrugal); err != nil { - t.Fatalf("setting the crew to frugal: %v", err) + // A pin written under a source that is already in use. + if err := config.SetCrewPin(dir, crewroute.Planner, "vendor/first-planner"); err != nil { + t.Fatalf("pinning the planner: %v", err) } after, err := roles.Resolve(roles.Source(source), roles.RolePlanner, "vendor/conversation") if err != nil { t.Fatal(err) } - frugal, _ := config.CrewModels(config.CrewFrugal) - want, _ := roles.SplitEffort(frugal[config.ModelTierMastermind]) - if after != want { - t.Fatalf("after /crew frugal the planner resolves to %q, want %q", after, want) + if after != "vendor/first-planner" { + t.Fatalf("after /crew pin the planner resolves to %q, want vendor/first-planner", after) } // A single class answered by hand is seen the same way, and so is a pin — @@ -71,7 +61,7 @@ func TestAMidSessionCrewChangeIsHonoredByTheNextCall(t *testing.T) { t.Fatal(err) } if call.Model != "vendor/thinker" || call.Effort != "high" { - t.Fatalf("after a hand-set mastermind the planner resolves to %+v", call) + t.Fatalf("after a hand-set planner row the planner resolves to %+v", call) } pins, found := registry.Row(config.KeyModelRoles) @@ -109,7 +99,7 @@ func TestABrokenRowLeavesTheLastGoodCrewInPlace(t *testing.T) { // is then moved by an unrelated write, so the source does rebuild and does // meet the broken row. handEdit(t, dir, "models.roles", "planner") - if err := config.ApplyCrew(dir, config.CrewMax); err != nil { + if err := config.SetCrewPin(dir, crewroute.Planner, "vendor/max-planner"); err != nil { t.Fatal(err) } if model, _ := roles.Resolve(roles.Source(source), roles.RolePlanner, "vendor/conversation"); model != "vendor/pinned" { @@ -119,13 +109,11 @@ func TestABrokenRowLeavesTheLastGoodCrewInPlace(t *testing.T) { // And once the row parses again, the new crew is picked up — a refusal is not // a latch. handEdit(t, dir, "models.roles", "") - if err := config.ApplyCrew(dir, config.CrewMax); err != nil { + if err := config.SetCrewPin(dir, crewroute.Planner, "vendor/max-planner"); err != nil { t.Fatal(err) } - want, _ := config.CrewModels(config.CrewMax) - if model, _ := roles.Resolve(roles.Source(source), roles.RolePlanner, "vendor/conversation"); model != want[config.ModelTierMastermind] { - t.Fatalf("after the row parsed again the planner resolves to %q, want the max crew's %q", - model, want[config.ModelTierMastermind]) + if model, _ := roles.Resolve(roles.Source(source), roles.RolePlanner, "vendor/conversation"); model != "vendor/max-planner" { + t.Fatalf("after the row parsed again the planner resolves to %q, want the pinned vendor/max-planner", model) } } @@ -178,8 +166,8 @@ func TestTheCrewSourceIsSafeUnderConcurrentTurns(t *testing.T) { wait.Add(1) go func() { defer wait.Done() - for _, preset := range []string{config.CrewFrugal, config.CrewMax, config.CrewBalanced} { - if err := config.ApplyCrew(dir, preset); err != nil { + for _, pin := range []string{"vendor/a", "vendor/b", "vendor/c"} { + if err := config.SetCrewPin(dir, crewroute.Worker, pin); err != nil { t.Error(err) return } diff --git a/cmd/codeaf/chatv3_host.go b/cmd/codeaf/chatv3_host.go index 7e9b8c6257..b5a3f4de88 100644 --- a/cmd/codeaf/chatv3_host.go +++ b/cmd/codeaf/chatv3_host.go @@ -620,11 +620,11 @@ func hostOptions(fleet *engineFleet, welcome remote.Welcome, pick bool) (tui3.Op // lets go (#1274): its fetch writes a cache when it lands, and one nobody // joined could write after this window had closed. fleet.own(models.Close) - // A tier row that says auto is answered from this catalog (config.AutoModels): + // The crew router picks its seats from this catalog (config.CrewCatalog): // the same non-blocking read, never a fetch, and set once at start-up. - config.AutoModels = models.ModelsNow - // The pool's index is seated beside it, read once here and refreshed in the - // background, against the same profile the catalog was read from. + seatCrewRows(models.ModelsNow) + // The pool's errands start beside it, against the same profile the catalog + // was read from. wirePoolIndex(profileDir) // The refresh key in /model asks the same router THIS machine's list came // from, and refills the same shelf — the list is this laptop's list of diff --git a/cmd/codeaf/chatv3_process.go b/cmd/codeaf/chatv3_process.go index 7376fe4a71..93c78d8d34 100644 --- a/cmd/codeaf/chatv3_process.go +++ b/cmd/codeaf/chatv3_process.go @@ -244,9 +244,9 @@ func openV3ProcessWith(door string, askKey bool) (*v3Process, error) { } processCtx, processStop := context.WithCancel(context.Background()) models := catalog.LoadLazy(processCtx, discovery) - // A tier row that says auto is answered from this catalog (config.AutoModels): + // The crew router picks its seats from this catalog (config.CrewCatalog): // the same non-blocking read, never a fetch, and set once at start-up. - config.AutoModels = models.ModelsNow + seatCrewRows(models.ModelsNow) wirePoolIndex(settings.ProfileDir) shelf := newV3ModelShelf(models, discovery) shelf.setSources(settings.Sources) diff --git a/cmd/codeaf/chatv3_roles_test.go b/cmd/codeaf/chatv3_roles_test.go index 7c07bcaff8..d41a41830d 100644 --- a/cmd/codeaf/chatv3_roles_test.go +++ b/cmd/codeaf/chatv3_roles_test.go @@ -4,6 +4,7 @@ import ( "testing" "github.com/Agent-Field/codeaf/internal/config" + "github.com/Agent-Field/codeaf/internal/crewroute" "github.com/Agent-Field/codeaf/internal/roles" ) @@ -59,21 +60,31 @@ func TestTheDoorsRoleLadderCarriesEveryTier(t *testing.T) { t.Fatalf("the reflex role resolves to %q after the tier was pinned to vendor/tiny", reflex) } - // EVERY CLASS IS WIRED, each to its own shipped model. The failure this holds - // shut is a class the door forgot: a role on it would resolve to the - // conversation's model, which is the mistake the reflex arm was added for. - // The mastermind's value may carry a level, and Resolve hands back the id - // alone — a colon in a model field is a request for a model nobody serves. - mastermind, _ := roles.SplitEffort(config.DefaultMastermindModel) + // EVERY CLASS IS WIRED. The two small rows ship pointed at a model; the + // three crew seats are the crew's — a pin when one is written, the + // router's pick otherwise — so pinning them here makes every class answer + // with its own model and none with the conversation's. The planner's value + // may carry a level, and Resolve hands back the id alone. + for seat, pin := range map[crewroute.Seat]string{ + crewroute.Worker: "vendor/worker", crewroute.Checker: "vendor/checker", crewroute.Planner: "vendor/planner:high", + } { + if err := config.SetCrewPin(dir, seat, pin); err != nil { + t.Fatalf("pinning the %s: %v", seat, err) + } + } + source, err = v3RolesSource(t.TempDir(), dir) + if err != nil { + t.Fatalf("v3RolesSource: %v", err) + } for _, c := range []struct { role roles.Role want string }{ {roles.RoleTitle, config.DefaultLowModel}, - {roles.RoleWorker, config.DefaultWorkerModel}, - {roles.RoleAuditor, config.DefaultHighModel}, - {roles.RolePlanner, mastermind}, - {roles.RoleDesigner, mastermind}, + {roles.RoleWorker, "vendor/worker"}, + {roles.RoleAuditor, "vendor/checker"}, + {roles.RolePlanner, "vendor/planner"}, + {roles.RoleDesigner, "vendor/planner"}, } { model, err := roles.Resolve(roles.Source(source), c.role, "vendor/conversation") if err != nil || model != c.want { @@ -82,8 +93,8 @@ func TestTheDoorsRoleLadderCarriesEveryTier(t *testing.T) { } // And the level reaches the caller as its own half. call, err := roles.ResolveCall(roles.Source(source), roles.RolePlanner, "vendor/conversation") - if _, level := roles.SplitEffort(config.DefaultMastermindModel); err != nil || call.Effort != level { - t.Fatalf("the planner resolved to %+v (%v), want the shipped level %q", call, err, level) + if err != nil || call.Effort != "high" { + t.Fatalf("the planner resolved to %+v (%v), want the pinned level high", call, err) } } diff --git a/cmd/codeaf/crewword_door_test.go b/cmd/codeaf/crewword_door_test.go deleted file mode 100644 index 1a28b3b4bb..0000000000 --- a/cmd/codeaf/crewword_door_test.go +++ /dev/null @@ -1,96 +0,0 @@ -package main - -import ( - "testing" - - "github.com/Agent-Field/codeaf/internal/catalog" - "github.com/Agent-Field/codeaf/internal/config" - "github.com/Agent-Field/codeaf/internal/roles" -) - -// THE TASK DOOR HONORS THE CONFIGURED CREW WORD. -// -// The worker seat of every task a proposal hands off is read through -// internal/roles ([session]'s defaultTaskModel asks for [roles.TierWorker]) off -// the key map this door builds ([v3RolesSource]). A run that wrote ONE word — -// `models.crew=frugal`, `models.crew=max` — into config.json used to read the -// balanced crew for every arm, because the word was never read: the preset is -// derived from the five tier rows, and the run wrote none of them. The worker -// and the checker (the careful-work seat) must answer the word the run asked -// for, not the default it fell to. -func TestTheTaskDoorHonorsTheConfiguredCrewWord(t *testing.T) { - for _, preset := range []string{config.CrewFrugal, config.CrewMax} { - dir := t.TempDir() - handEdit(t, dir, config.KeyCrew, preset) - - read, err := v3RolesSource(t.TempDir(), dir) - if err != nil { - t.Fatalf("building the door's key map for crew=%s: %v", preset, err) - } - row, ok := config.CrewModelsForSource(config.CrewSourceAt(dir), preset) - if !ok { - t.Fatalf("there is no %s preset in the family the profile names", preset) - } - // The door's own two questions, in its own words. - worker, ok := roles.TierModel(roles.Source(read), roles.TierWorker) - if !ok || worker != row[config.ModelTierWorker] { - t.Errorf("crew=%s: the door's worker seat reads %q, want the preset's own %q", - preset, worker, row[config.ModelTierWorker]) - } - checker, ok := roles.TierModel(roles.Source(read), roles.TierHigh) - if !ok || checker != row[config.ModelTierHigh] { - t.Errorf("crew=%s: the door's checker seat reads %q, want the preset's own %q", - preset, checker, row[config.ModelTierHigh]) - } - } -} - -// AND A PIN THE RUN WROTE SURVIVES THE LEARNED PICK AT THE SAME DOOR. The knee -// arm pins the crew table's own ids and runs under `models.crew.pick=learn`; -// the pick used to recompute both seats. The door must answer exactly the ids -// the run pinned. -func TestTheTaskDoorKeepsAPinnedSeatUnderLearn(t *testing.T) { - balanced, ok := config.CrewModelsForSource(config.DefaultCrewSource, config.CrewBalanced) - if !ok { - t.Fatalf("there is no %s preset in the %s family", config.CrewBalanced, config.DefaultCrewSource) - } - // A catalog is held, so the learned pick has something to recompute with — - // the shape the benchmark ran, where the pick overrode the pin. - previous := config.AutoModels - config.AutoModels = func() []catalog.Model { return seatPickRows() } - defer func() { config.AutoModels = previous }() - - for _, tc := range []struct { - name string - withWord bool - }{ - {"with the stored crew word", true}, - {"with the pins alone", false}, - } { - dir := t.TempDir() - handEdit(t, dir, config.KeyCrewPick, config.CrewPickLearn) - if tc.withWord { - handEdit(t, dir, config.KeyCrew, config.CrewBalanced) - } - handEdit(t, dir, config.KeyTierWorkerModel, balanced[config.ModelTierWorker]) - handEdit(t, dir, config.KeyTierHighModel, balanced[config.ModelTierHigh]) - - read, err := v3RolesSource(t.TempDir(), dir) - if err != nil { - t.Fatalf("%s: building the door's key map: %v", tc.name, err) - } - for _, seat := range []struct { - tier roles.Tier - want string - }{ - {roles.TierWorker, balanced[config.ModelTierWorker]}, - {roles.TierHigh, balanced[config.ModelTierHigh]}, - } { - got, ok := roles.TierModel(roles.Source(read), seat.tier) - if !ok || got != seat.want { - t.Errorf("%s: the door's %s seat under %s reads %q, want the pin %q", - tc.name, seat.tier, config.CrewPickLearn, got, seat.want) - } - } - } -} diff --git a/cmd/codeaf/do.go b/cmd/codeaf/do.go index 50ed45e609..8dc512df8c 100644 --- a/cmd/codeaf/do.go +++ b/cmd/codeaf/do.go @@ -7,6 +7,8 @@ import ( "encoding/json" "errors" "fmt" + "github.com/Agent-Field/codeaf/internal/crewroute" + "github.com/Agent-Field/codeaf/internal/router" "io" "log" "os" @@ -170,7 +172,7 @@ type headlessOutcome struct { Learned []string `json:"learned,omitempty"` // Model and PlanModel are the two seats this errand ran on, and the two // Source fields name the rung that chose each — `--model`, `CODEAF_MODEL`, - // `crew frugal`, `default` (config.ResolveSeats). They are here because the + // `pinned`, `routed` (config.ResolveSeats). They are here because the // defect that produced them was invisible from outside: a campaign that // believed its profile's crew was in force had no way to read back that the // run had resolved its models somewhere else entirely (#166). A caller @@ -184,6 +186,14 @@ type headlessOutcome struct { PlanModel string `json:"plan_model"` ModelSource string `json:"model_source"` PlanModelSource string `json:"plan_model_source"` + // crew is the router's decision for this run — the class it read the task + // as, every seat's pick and the estimate — and nil when nothing routed. + // The envelope carries it as `class`, `crew`, `est_usd` beside `spend`, + // which is the actual (envelope.go's [legacyErrandFields]). + crew *crewroute.Decision + // checkModel and checkModelSource are the third seat, beside the two + // above. + checkModel, checkModelSource string // Subharness is the worker that took the deliverable, read back from the // durable row rather than from what was asked for. It is always present and // never empty, because an absent key is indistinguishable from an older @@ -322,6 +332,14 @@ func runDo(args []string) error { model := flags.String("model", "", modelFlagHelp) planModel := flags.String("plan-model", "", planModelFlagHelp) checkModel := flags.String("check-model", "", checkModelFlagHelp) + // HOW HARD TO TRY THIS ONE TASK, said on the command line and sticking to + // nothing: --best puts the strongest crew the allowed models make on it, + // --cheap the cheapest, and --pin seats one seat for this run alone. The + // three seat flags above are one-task pins too (config.ResolveSeats). + best := flags.Bool("best", false, "run this task on the strongest crew your allowed models make") + cheap := flags.Bool("cheap", false, "run this task on the cheapest crew your allowed models make") + var pins pinFlags + flags.Var(&pins, "pin", "use one model for this run only: worker=model[@provider], planner=… or checker=… (repeatable)") // A FLAG IS DOCUMENTED BY WHAT IT DOES, NOT BY WHAT IT SETS. These two said // "…; sets CODEAF_CONTEXT_FILL_PCT for this run", which is the // implementation, and hard-coded their defaults in prose while their own @@ -364,7 +382,18 @@ func runDo(args []string) error { if err != nil { return err } + if *best && *cheap { + return fmt.Errorf("--best and --cheap ask for two different crews · say one") + } + effort := crewroute.EffortKnee + switch { + case *best: + effort = crewroute.EffortBest + case *cheap: + effort = crewroute.EffortCheap + } return doErrand(doRequest{ + effort: effort, pins: pins.pins, task: task, run: run, database: *database, keep: *keep, workspace: *workspace, timeout: wall.wall, asJSON: *asJSON, yesSpend: *yesSpend, model: *model, planModel: *planModel, checkModel: *checkModel, @@ -390,6 +419,10 @@ type doRequest struct { model string planModel string checkModel string + // effort and pins are the one-task crew words: --best or --cheap, and + // every --pin. They move this run's crew and nothing after it. + effort crewroute.Effort + pins map[crewroute.Seat]config.CrewPin // contextFill and completionReserve are this run's two dials on the window // law (internal/ctxbudget). They are integers rather than a struct because // zero has to mean "not asked for": the law's own defaults are the answer @@ -499,17 +532,66 @@ func doErrand(request doRequest) error { if err == nil { useAutoSeats(settings) } - seats := config.ResolveSeats(config.ProfileDir(), request.model, request.planModel) - // THE CHECK SEAT RESOLVES AT THE DOOR: its flag, its environment, a plan - // seat pinned by flag or environment, then the crew's careful row. - seats.Check = config.CheckSeat(request.checkModel, seats.Plan) + profileDir := config.ProfileDir() + // A PROFILE WRITTEN BEFORE CREWS WERE ROUTED IS MIGRATED ONCE, and the one + // line saying so is said here, on stderr, where a person reads the models + // line (internal/config's crewmigrate.go). + if line, _ := config.MigrateCrew(profileDir); line != "" { + fmt.Fprintln(request.stderr, line) + } + // THE CREW IS ROUTED FOR THIS TASK: the flags and the environment are + // one-task pins, a --pin is one too, and every seat nothing named is picked + // for what the task reads as (config.ResolveSeats). THE CHECK SEAT IS ITS + // OWN SEAT and never inherits the planner's model. + repo := request.workspace + if repo == "" { + repo, _ = os.Getwd() + } + if abs, err := filepath.Abs(repo); err == nil { + repo = abs + } + seats, err := config.ResolveSeats(profileDir, config.SeatFlags{ + Model: request.model, PlanModel: request.planModel, CheckModel: request.checkModel, + }, config.CrewAsk{ + Task: crewroute.Task{Text: request.task}, Effort: request.effort, Pins: request.pins, Repo: repo, + }) + if err != nil && !errors.Is(err, config.ErrCrewAtCap) { + // A SEAT NOTHING ALLOWED CAN SIT, or a pin that will not route, is said + // before anything is opened: there is no crew to run on. + if !request.asJSON { + return err + } + outcome := failedErrand(err, started) + outcome.run = request.run + return reportErrand(request, outcome) + } + if errors.Is(err, config.ErrCrewAtCap) && !request.yesSpend { + // AT THE DAILY CAP A HEADLESS RUN REFUSES: nobody is there to ask, and a + // cap that spends anyway is not a cap. -yes-spend is the one way past. + capErr := fmt.Errorf("today's crew spend has reached the daily cap of %s · raise it with `/crew cap`, run with --cheap, or pass -yes-spend", + crewroute.Money(config.CrewCapAt(profileDir))) + if !request.asJSON { + return capErr + } + outcome := failedErrand(capErr, started) + outcome.seated(seats) + outcome.run = request.run + return reportErrand(request, outcome) + } fmt.Fprintln(request.stderr, seats.Report()) + call := router.CrewCallID(request.run) + if seats.Crew != nil { + config.LogCrewDecision(profileDir, call, *seats.Crew, repo, crewTitle(request.task)) + } outcome, err := errandRun(request, seats, started) if err != nil { if !request.asJSON { if outcome.recordKept != "" { fmt.Fprintf(request.stderr, "record kept at %s\n", outcome.recordKept) } + if seats.Crew != nil { + config.LogCrewOutcome(profileDir, call, *seats.Crew, repo, crewTitle(request.task), router.CrewNotKept, 0) + } return err } recordKept := outcome.recordKept @@ -517,6 +599,17 @@ func doErrand(request doRequest) error { outcome.recordKept = recordKept } outcome.seated(seats) + // THE CREW'S OUTCOME, beside its decision in the router's log: accepted + // when the run came home done, not kept otherwise — and the summary line + // with the actual beside the estimate. + if seats.Crew != nil { + settled := router.CrewNotKept + if outcome.resolvedStop() == stopDone { + settled = router.CrewAccepted + } + config.LogCrewOutcome(profileDir, call, *seats.Crew, repo, crewTitle(request.task), settled, outcome.Spend) + fmt.Fprintln(request.stderr, "crew: "+seats.Crew.Line(config.PinMark, outcome.Spend)) + } // THE RUN NAMES ITSELF ON EVERY PATH, including the one where nothing // worked: the id is what joins this object to the rows the model-call log // wrote and to the debug record's folder, and a run that fell over after @@ -555,6 +648,56 @@ func (o *headlessOutcome) seated(seats config.Seats) { o.PlanModel = seats.Plan.Model o.ModelSource = seats.Work.Rung() o.PlanModelSource = seats.Plan.Rung() + o.checkModel, o.checkModelSource = seats.Check.Model, seats.Check.Rung() + o.crew = seats.Crew +} + +// crewTitle is the first line of a task, cut short: what the router's log +// names a headless task by. +func crewTitle(task string) string { + line, _, _ := strings.Cut(strings.TrimSpace(task), "\n") + if runes := []rune(line); len(runes) > 80 { + line = string(runes[:80]) + "…" + } + return line +} + +// pinFlags is the repeatable --pin: seat=model[@provider], one per flag. +type pinFlags struct { + pins map[crewroute.Seat]config.CrewPin +} + +func (p *pinFlags) String() string { + if p == nil || len(p.pins) == 0 { + return "" + } + var said []string + for _, seat := range crewroute.Seats { + if pin, ok := p.pins[seat]; ok { + said = append(said, string(seat)+"="+pin.String()) + } + } + return strings.Join(said, ",") +} + +func (p *pinFlags) Set(raw string) error { + seatWord, value, ok := strings.Cut(raw, "=") + seat, known := config.ParseCrewSeat(seatWord) + if !ok || !known { + return fmt.Errorf("--pin takes seat=model[@provider], and the seat is worker, planner or checker") + } + pin, auto, err := config.ParseCrewPin(value) + if err != nil { + return err + } + if auto { + return fmt.Errorf("--pin %s=auto pins nothing · leave the flag off to have the seat routed", seat) + } + if p.pins == nil { + p.pins = map[crewroute.Seat]config.CrewPin{} + } + p.pins[seat] = pin + return nil } // errandRun is the errand itself: everything from opening a store to composing @@ -3529,6 +3672,13 @@ func runErrand(request doRequest, seats config.Seats) (outcome headlessOutcome, } completerFor = crewCompleters(settings, newClient) } + // EVERY SEAT CALL IS PRICED BEFORE IT IS MADE (internal/session's + // spendguard.go): the day's cap — the daily spending limit too unless + // -yes-spend lifted it — the per-task limit, and the checker's own + // ceiling on this run. + guard := doSpendGuard(config.ProfileDir(), seats.Crew, spendPreauthorized(request.yesSpend, env.Value)) + unguarded := completerFor + completerFor = func(model string) session.Completer { return guard.Wrap(model, unguarded(model)) } // THE REVIEW ROUND IS ON for every `do` run: a leaf that lands done is // checked against its acceptance, and a check that does not hold becomes a // fix task under the leaf's parent the run waits on. @@ -3720,6 +3870,16 @@ type runSpend struct { refused bool } +// doSpendGuard is the guard a `do` run's calls are held to. preauthorized +// (-yes-spend) lifts the daily spending limit from it and nothing else: the +// per-task limit holds either way, with or without a routed crew. +func doSpendGuard(profileDir string, crew *crewroute.Decision, preauthorized bool) *session.SpendGuard { + if crew == nil { + return session.TaskSpendGuard(profileDir) + } + return session.CrewSpendGuard(profileDir, *crew, !preauthorized) +} + // runSpendBound is THE SPENDING CONTRACT `--yes-spend` promises // ([yesSpendFlagHelp]): without it, a run stops at the plan-price question's // figure and at what is left of today's limit, whichever is nearer; with it, diff --git a/cmd/codeaf/do_engine_contract_test.go b/cmd/codeaf/do_engine_contract_test.go index 5db8b6677e..f9e640f9c5 100644 --- a/cmd/codeaf/do_engine_contract_test.go +++ b/cmd/codeaf/do_engine_contract_test.go @@ -24,6 +24,7 @@ import ( "time" "github.com/Agent-Field/agentfield/sdk/go/ai" + "github.com/Agent-Field/codeaf/internal/config" "github.com/Agent-Field/codeaf/internal/session" ) @@ -209,6 +210,11 @@ func TestDoOnTheRunEngineYesSpendRunsPastThePlanPrice(t *testing.T) { t.Setenv("CODEAF_PLANDB_BIN", beltPlandbDoor(t)) t.Setenv("CODEAF_PLAN_CONSENT", "0.5") t.Setenv("CODEAF_DAILY_BUDGET", "0.5") + // The per-task limit is not what this run is about, and --yes-spend does + // not lift it: it is raised past what the scripted calls are priced at. + if err := config.SetCrewTaskCap(config.ProfileDir(), "1000"); err != nil { + t.Fatal(err) + } workspace := beltRepoWorkspace(t) var stdout, stderr strings.Builder diff --git a/cmd/codeaf/do_engine_test.go b/cmd/codeaf/do_engine_test.go index a2a01bfca2..90df20d0dc 100644 --- a/cmd/codeaf/do_engine_test.go +++ b/cmd/codeaf/do_engine_test.go @@ -435,7 +435,7 @@ func TestDoOnTheRunEngineSeatsEveryLaunchOnTheDoorsModels(t *testing.T) { var stdout, stderr strings.Builder err = doErrand(doRequest{ task: "write out.txt and say what you did", workspace: workspace, asJSON: true, - timeout: 60 * time.Second, slots: bound(1), model: workModel, planModel: planModel, + timeout: 60 * time.Second, slots: bound(1), model: workModel, planModel: planModel, checkModel: planModel, stdout: &stdout, stderr: &stderr, newBeltCompleter: newBelt, }) if err != nil { diff --git a/cmd/codeaf/do_lock_test.go b/cmd/codeaf/do_lock_test.go index d0e8f1d5d3..d9c7ccdee5 100644 --- a/cmd/codeaf/do_lock_test.go +++ b/cmd/codeaf/do_lock_test.go @@ -31,6 +31,8 @@ func TestDoFailsFastWhenAnotherProcessHoldsTheResidentLock(t *testing.T) { err = doErrand(doRequest{ task: "write the release note", database: path, + // Every seat is named, so the crew needs no router to reach the lock. + model: "test/model", planModel: "test/model", checkModel: "test/model", // The wall is long and the bound is short on purpose: the thing under // test is that the run leaves on the bound rather than on the wall. timeout: 5 * time.Minute, diff --git a/cmd/codeaf/do_machine_gate_test.go b/cmd/codeaf/do_machine_gate_test.go index 2fe0989a01..d220f3a0dd 100644 --- a/cmd/codeaf/do_machine_gate_test.go +++ b/cmd/codeaf/do_machine_gate_test.go @@ -39,7 +39,7 @@ func TestDoRunHonoursProfileMachineFloor(t *testing.T) { outcome, err := runErrand(doRequest{ task: "do the work", workspace: workspace, timeout: 120 * time.Millisecond, stderr: &stderr, newBeltCompleter: func(string) session.Completer { return seat }, - }, config.ResolveSeats(profile, "", "")) + }, stubErrandSeats()) if err != nil { t.Fatal(err) } @@ -55,7 +55,7 @@ func TestDoRunHonoursProfileMachineFloor(t *testing.T) { outcome, err = runErrand(doRequest{ task: "do the work", workspace: workspace, timeout: 30 * time.Second, stderr: &stderr, newBeltCompleter: func(string) session.Completer { return seat }, - }, config.ResolveSeats(profile, "", "")) + }, stubErrandSeats()) if err != nil || outcome.Nodes == 0 { t.Fatalf("zeroed machine gate did not start: nodes=%d err=%v", outcome.Nodes, err) } @@ -88,7 +88,7 @@ func TestDoHeldByTheMachineSaysWhyOnStderrAndInItsStop(t *testing.T) { asJSON: true, stdout: &stdout, stderr: &stderr, newBeltCompleter: func(string) session.Completer { return seat }, } - outcome, err := runErrand(request, config.ResolveSeats(profile, "", "")) + outcome, err := runErrand(request, stubErrandSeats()) if err != nil { t.Fatal(err) } @@ -167,3 +167,14 @@ func TestDoSaysAMachineHoldOnceAndItsEndOnce(t *testing.T) { t.Fatal("no gate at all must reach the engine as nil, not as a wrapper around nothing") } } + +// stubErrandSeats is the crew a stubbed errand runs on: three seats named +// outright, as flags name them, so the run needs no catalog and no router — +// the seat's completer is the test's own. +func stubErrandSeats() config.Seats { + return config.Seats{ + Work: config.Seat{Role: config.SeatWork, Model: "stub/worker", Source: config.SeatFlag}, + Plan: config.Seat{Role: config.SeatPlan, Model: "stub/planner", Source: config.SeatFlag}, + Check: config.Seat{Role: config.SeatCheck, Model: "stub/checker", Source: config.SeatFlag}, + } +} diff --git a/cmd/codeaf/do_private_store_test.go b/cmd/codeaf/do_private_store_test.go index f12581ab1f..3103f06934 100644 --- a/cmd/codeaf/do_private_store_test.go +++ b/cmd/codeaf/do_private_store_test.go @@ -10,7 +10,6 @@ import ( "time" "github.com/Agent-Field/agentfield/sdk/go/ai" - "github.com/Agent-Field/codeaf/internal/config" "github.com/Agent-Field/codeaf/internal/session" ) @@ -65,7 +64,7 @@ func TestDoRunRefusesFileDirectoryBeforeOpeningRecord(t *testing.T) { } seat := &beltSeat{} _, err := runErrand(doRequest{task: "work", workspace: file, timeout: time.Second, - newBeltCompleter: func(string) session.Completer { return seat }}, config.ResolveSeats(config.ProfileDir(), "", "")) + newBeltCompleter: func(string) session.Completer { return seat }}, stubErrandSeats()) if err == nil || !strings.Contains(err.Error(), file) { t.Fatalf("bad --dir error = %v", err) } @@ -98,7 +97,7 @@ func TestDoRunCreatesMissingDirectoryWithoutRepositoryRecord(t *testing.T) { seat := finishingSeat(0) var stderr strings.Builder outcome, err := runErrand(doRequest{task: "work", workspace: want, timeout: 30 * time.Second, - stderr: &stderr, newBeltCompleter: func(string) session.Completer { return seat }}, config.ResolveSeats(config.ProfileDir(), "", "")) + stderr: &stderr, newBeltCompleter: func(string) session.Completer { return seat }}, stubErrandSeats()) if err != nil || outcome.Nodes == 0 { t.Fatalf("missing --dir did not run: nodes=%d err=%v", outcome.Nodes, err) } diff --git a/cmd/codeaf/do_test.go b/cmd/codeaf/do_test.go index 7f80c84bd1..1e076b466e 100644 --- a/cmd/codeaf/do_test.go +++ b/cmd/codeaf/do_test.go @@ -919,6 +919,7 @@ func TestJSONPrintsAnObjectWhenTheErrandCannotEvenStart(t *testing.T) { var stdout, stderr strings.Builder err := doErrand(doRequest{ task: "write the release note", database: filepath.Join(blocked, "graph.db"), + model: "test/model", planModel: "test/model", checkModel: "test/model", asJSON: true, timeout: 10 * time.Second, stdout: &stdout, stderr: &stderr, }) @@ -964,6 +965,7 @@ func TestAFailedErrandWithoutJSONStillJustReturnsTheError(t *testing.T) { var stdout, stderr strings.Builder err := doErrand(doRequest{ task: "write the release note", database: filepath.Join(blocked, "graph.db"), + model: "test/model", planModel: "test/model", checkModel: "test/model", timeout: 10 * time.Second, stdout: &stdout, stderr: &stderr, }) if err == nil || !strings.Contains(err.Error(), "store directory") { diff --git a/cmd/codeaf/envelope.go b/cmd/codeaf/envelope.go index 20d8f16fa6..23a94c2297 100644 --- a/cmd/codeaf/envelope.go +++ b/cmd/codeaf/envelope.go @@ -550,6 +550,30 @@ func legacyErrandFields(outcome headlessOutcome) map[string]any { "subharness": outcome.Subharness, "workspace": outcome.workspace, } + // THE CREW THE TASK WAS ROUTED TO, and the third seat beside the two + // above: the class the task was read as, each seat's model, route and + // whether it was pinned, and the estimate beside `spend`, which is the + // actual. Present on every run that was routed, absent on one that + // failed before a crew existed. + if strings.TrimSpace(outcome.checkModel) != "" { + fields["check_model"] = outcome.checkModel + fields["check_model_source"] = outcome.checkModelSource + } + if outcome.crew != nil { + fields["class"] = string(outcome.crew.Class) + fields["est_usd"] = outcome.crew.EstUSD + crew := map[string]any{} + for _, pick := range outcome.crew.Crew { + crew[string(pick.Seat)] = map[string]any{ + "model": pick.Model, "provider": pick.Provider, "kind": string(pick.Kind), + "pinned": pick.Pinned, "est_usd": pick.CostUSD, + } + } + fields["crew"] = crew + if outcome.crew.Effort != "" { + fields["effort"] = string(outcome.crew.Effort) + } + } if fields["artifacts"] == nil { fields["artifacts"] = []string{} } diff --git a/cmd/codeaf/exec.go b/cmd/codeaf/exec.go index 24adfb75ee..10f3e4c3af 100644 --- a/cmd/codeaf/exec.go +++ b/cmd/codeaf/exec.go @@ -117,7 +117,7 @@ func runExec(args []string) error { // would be the parity it claims in name only. Only the work seat is printed, // because only the work seat runs anything. useAutoSeats(settings) - seats := config.ResolveSeats(settings.ProfileDir, *model, *planModel) + seats := doorSeats(settings, *model, *planModel, prompt) applySeats(&settings, seats) fmt.Fprintln(os.Stderr, seats.Work.Report()) modelCatalog := sharedCatalog(settings) diff --git a/cmd/codeaf/logs_test.go b/cmd/codeaf/logs_test.go index e1b1e4835b..ab034a15c9 100644 --- a/cmd/codeaf/logs_test.go +++ b/cmd/codeaf/logs_test.go @@ -13,6 +13,8 @@ import ( "time" "github.com/Agent-Field/codeaf/internal/calllog" + "github.com/Agent-Field/codeaf/internal/catalog" + "github.com/Agent-Field/codeaf/internal/config" ) // TestMain switches the model-call log OFF for this package and gives the @@ -46,6 +48,14 @@ func TestMain(m *testing.M) { os.Setenv("CODEAF_MODEL_POOL_SUBMIT_URL", "http://127.0.0.1:1/v1/rows") } restore := isolateTestEnvironment() + // AND THE CREW ROUTER READS ONE FIXED CATALOG. The process's catalog is + // seated once, by whichever test first builds it, from whatever server that + // test stood up — so every later door would route its unpinned seats against + // a stranger's rows, or none. Two priced, tool-serving rows make routing + // deterministic for the whole binary; seatCrewCatalog still overrides it for + // a test that means a catalog of its own. + config.CrewCatalog = func() []catalog.Model { return crewDoorCatalog() } + seatCrewRows = func(func() []catalog.Model) {} // AND THE TELEMETRY OFF SWITCH IS CLEARED, because since the pool learned // to hear it (config.ModelPoolResolved) a shell that exports it quiets the // pool to `read`, and every pool test here that means the default would diff --git a/cmd/codeaf/main.go b/cmd/codeaf/main.go index cb7154e89d..a1554d49d6 100644 --- a/cmd/codeaf/main.go +++ b/cmd/codeaf/main.go @@ -13,6 +13,7 @@ import ( "errors" "flag" "fmt" + "github.com/Agent-Field/codeaf/internal/crewroute" "io" "log" "os" @@ -800,7 +801,7 @@ func runPlanNew(name string, args []string) error { return err } useAutoSeats(settings) - seats := config.ResolveSeats(settings.ProfileDir, *model, *planModel) + seats := doorSeats(settings, *model, *planModel, goal) applySeats(&settings, seats) workClient, err := settings.Client() if err != nil { @@ -980,7 +981,7 @@ func runRevise(name string, args []string) error { return err } useAutoSeats(settings) - seats := config.ResolveSeats(settings.ProfileDir, *model, *planModel) + seats := doorSeats(settings, *model, *planModel, graph.Goal+"\n\n"+event) applySeats(&settings, seats) workClient, err := settings.Client() if err != nil { @@ -1212,22 +1213,21 @@ func applyModelFlags(settings *config.Config, model, planModel string) { } } -// THE TWO MODEL FLAGS SAY THE SAME THING AT EVERY DOOR, so they say it once. +// THE THREE MODEL FLAGS SAY THE SAME THING AT EVERY DOOR, so they say it once. // -// The wording they replaced was `(default CODEAF_MODEL)`, which named one rung -// of four and hid the two that decide most runs: a profile's crew, and this -// build's own default when nobody has said anything at all. A help string that -// names the whole ladder is the shortest place a person can learn that their -// crew reaches this command (config.ResolveSeats). +// Each is a ONE-TASK PIN: the flag, then its variable, then the crew — a pin +// the profile holds (/crew pin) or the router's pick for this task +// (config.ResolveSeats). The check seat never falls to the plan seat, and no +// seat falls to a model this build chose for everybody. const ( - workLadderHelp = "flag › CODEAF_MODEL › crew › default" - planLadderHelp = "flag › CODEAF_PLAN_MODEL › crew mastermind › the work model" - checkLadderHelp = "flag › CODEAF_CHECK_MODEL › plan pinned by flag or environment › crew careful" - modelFlagHelp = "work model for this run (" + workLadderHelp + ")" - planModelFlagHelp = "model that plans, when different from the work model (" + planLadderHelp + ")" - // The check seat's ladder names its environment rung and the resolved - // plan seat fallback before the crew's careful row. - checkModelFlagHelp = "model that checks finished work (" + checkLadderHelp + ")" + workLadderHelp = "flag › CODEAF_MODEL › crew pin › crew routed per task" + planLadderHelp = "flag › CODEAF_PLAN_MODEL › crew pin › crew routed per task" + checkLadderHelp = "flag › CODEAF_CHECK_MODEL › crew pin › crew routed per task" + modelFlagHelp = "work model for this run, a one-task pin (" + workLadderHelp + ")" + planModelFlagHelp = "model that plans, a one-task pin (" + planLadderHelp + ")" + // The check seat's ladder is its own: a pinned planner says something about + // planning and nothing about who grades the work. + checkModelFlagHelp = "model that checks finished work, a one-task pin (" + checkLadderHelp + ")" ) // yesSpendFlagHelp is what `--yes-spend` MEANS, said once, on both doors that @@ -1294,6 +1294,32 @@ func debugRecordRoot() string { // environment reading underneath. One assignment per seat, so the models a // door's receipt names and the clients it then builds cannot be different // models. +// doorSeats is the crew every headless door that is not `codeaf do` runs on: +// the flags as one-task pins, and every seat nothing named routed for the task +// text the door has (empty reads as open-ended work, the router's safe +// default). A profile written before crews were routed is migrated first, with +// its one line. +// +// AT THE DAILY CAP THESE DOORS WARN AND GO ON. `codeaf do` refuses there +// unless told -yes-spend, because it is the door campaigns run through; these +// are a person at a terminal running one plan step or one program, and the +// line on stderr is said before anything is spent. A seat nothing allowed can +// sit is said too, and the seat is left for the door's own model to fill. +func doorSeats(settings config.Config, model, planModel, task string) config.Seats { + if line, _ := config.MigrateCrew(settings.ProfileDir); line != "" { + fmt.Fprintln(os.Stderr, line) + } + seats, err := config.ResolveSeats(settings.ProfileDir, config.SeatFlags{Model: model, PlanModel: planModel}, + config.CrewAsk{Task: crewroute.Task{Text: task}}) + switch { + case errors.Is(err, config.ErrCrewAtCap): + fmt.Fprintln(os.Stderr, "note: today's crew spend has reached the daily cap · this run goes ahead; `codeaf do` would have stopped") + case err != nil: + fmt.Fprintln(os.Stderr, "note: "+err.Error()) + } + return seats +} + func applySeats(settings *config.Config, seats config.Seats) { settings.Model = seats.Work.Model settings.PlanModel = seats.Plan.Model diff --git a/cmd/codeaf/pool_test.go b/cmd/codeaf/pool_test.go index 493289f07c..b9469e1538 100644 --- a/cmd/codeaf/pool_test.go +++ b/cmd/codeaf/pool_test.go @@ -17,8 +17,6 @@ import ( "testing" "time" - "github.com/Agent-Field/codeaf/internal/config" - "github.com/Agent-Field/codeaf/internal/crewpick" "github.com/Agent-Field/codeaf/internal/home" "github.com/Agent-Field/codeaf/internal/pool/index" "github.com/Agent-Field/codeaf/internal/pool/outbox" @@ -1155,8 +1153,6 @@ func TestPoolVerifyRefusesADocumentItsKeyDoesNotTrust(t *testing.T) { } } -// ── THE SEATED INDEX ──────────────────────────────────────────────────────── - // writePoolDoc puts a document where poolIndexFor reads the cache: doc.json // under the profile's pool directory. func writePoolDoc(t *testing.T, dir, doc string) { @@ -1170,58 +1166,6 @@ func writePoolDoc(t *testing.T, dir, doc string) { } } -// With no cache the reader answers the seed this build carries, and a worker -// cell is in it — so `learn` has numbers on day one. -func TestPoolIndexForAnswersTheSeedWithNoCache(t *testing.T) { - held := poolIndexFor(t.TempDir(), poolcfg.Resolve("", "", noEnv), poolClock(t))() - if held == nil { - t.Fatal("no cache and no seed: the reader answered nothing") - } - worker := false - for _, cell := range held.Cells("role_quality") { - if cell.Role == "worker" { - worker = true - } - } - if !worker { - t.Fatal("the seed answered no worker cell") - } -} - -// A cached document whose generated day is newer than the seed's wins: the -// reader hands back the cached numbers rather than the embedded ones. -func TestPoolIndexForKeepsANewerCache(t *testing.T) { - dir := t.TempDir() - writePoolDoc(t, dir, `{ - "schema": 1, - "generated": "2026-09-20", - "min_installs": 1, - "metrics": {"role_quality": {"kind": "gaussian", "dims": ["role", "model"]}}, - "cells": [{"metric": "role_quality", "role": "worker", "model": "z-ai/glm-5.3", "mean": 75, "sd": 7, "n": 30}] - }`) - held := poolIndexFor(dir, poolcfg.Resolve("", "", noEnv), poolClock(t))() - if held == nil || held.Generated().Format("2006-01-02") != "2026-09-20" { - t.Fatalf("a newer cache did not win: %v", held) - } -} - -// A cache that does not parse is not a cache: the seed stands in its place. -func TestPoolIndexForIgnoresAnUnparsableCache(t *testing.T) { - dir := t.TempDir() - writePoolDoc(t, dir, "{ this is not a document") - held := poolIndexFor(dir, poolcfg.Resolve("", "", noEnv), poolClock(t))() - if held == nil || held.Generated().Format("2006-01-02") != seedDay(t) { - t.Fatalf("an unparsable cache did not fall back to the seed: %v", held) - } -} - -// A mode that forbids reading answers no index at all. -func TestPoolIndexForAnswersNothingWhenTheModeIsOff(t *testing.T) { - if held := poolIndexFor(t.TempDir(), poolcfg.Resolve("off", "", noEnv), poolClock(t))(); held != nil { - t.Fatal("a mode that forbids reading answered an index") - } -} - // No key in the build means no fetch: the refresh starts no goroutine, which // is every run on a build with no key compiled in. A key starts the one fetch, // and the mode still has to allow reading. @@ -1711,77 +1655,6 @@ func TestPoolShowReadsAnUnparsableOwnSheetAsNone(t *testing.T) { } } -// The wired reader answers the own sheet's cells — the install's own evidence, -// read once and parsed once — and an off mode seats nothing at all. -func TestWirePoolIndexSeatsTheOwnSheetsCells(t *testing.T) { - prevIndex, prevOwn := config.AutoIndex, config.AutoOwnCells - t.Cleanup(func() { config.AutoIndex, config.AutoOwnCells = prevIndex, prevOwn }) - stubPoolRefresh(t) - - dir := t.TempDir() - seedOwnSheet(t, dir) - wirePoolIndex(dir) - if config.AutoOwnCells == nil { - t.Fatal("the own sheet was not seated") - } - own := config.AutoOwnCells() - want := []crewpick.Cell{ - {Role: "worker", Model: "a/one", Mean: 85, N: 2}, - {Role: "worker", Model: "b/two", Mean: 70, N: 1}, - } - if len(own) != len(want) { - t.Fatalf("the seated cells are %+v, want %+v", own, want) - } - for i := range want { - if own[i] != want[i] { - t.Fatalf("cell %d is %+v, want %+v", i, own[i], want[i]) - } - } - if config.AutoIndex == nil || config.AutoIndex() == nil { - t.Fatal("the index was not seated beside the own sheet") - } -} - -// A mode that forbids reading seats nothing: the own sheet is the pool's own -// reading, and off is off for the whole of it. -func TestWirePoolIndexSeatsNothingWhenThePoolIsOff(t *testing.T) { - prevIndex, prevOwn := config.AutoIndex, config.AutoOwnCells - t.Cleanup(func() { config.AutoIndex, config.AutoOwnCells = prevIndex, prevOwn }) - t.Setenv("CODEAF_MODEL_POOL", "off") - stubPoolRefresh(t) - - dir := t.TempDir() - seedOwnSheet(t, dir) - wirePoolIndex(dir) - if config.AutoOwnCells != nil { - t.Fatal("a pool that forbids reading seated the own sheet") - } - if config.AutoIndex != nil && config.AutoIndex() != nil { - t.Fatal("a pool that forbids reading seated an index") - } -} - -// An own sheet that does not parse is a loss, not a fault a pick stops for: -// the wired reader answers nothing rather than a broken sheet's half. -func TestWirePoolIndexSeatsNothingForAnUnparsableOwnSheet(t *testing.T) { - prevOwn := config.AutoOwnCells - t.Cleanup(func() { config.AutoOwnCells = prevOwn }) - stubPoolRefresh(t) - - dir := t.TempDir() - poolDir := filepath.Join(dir, "pool") - if err := os.MkdirAll(poolDir, 0o755); err != nil { - t.Fatal(err) - } - if err := os.WriteFile(filepath.Join(poolDir, "own.json"), []byte("not a document"), 0o600); err != nil { - t.Fatal(err) - } - wirePoolIndex(dir) - if got := config.AutoOwnCells; got != nil && got() != nil { - t.Fatal("a broken own sheet was seated") - } -} - // ── THE MIRROR ────────────────────────────────────────────────────────────── // poolEnv is an environment holding exactly the names given, so a test pins diff --git a/cmd/codeaf/poolindex.go b/cmd/codeaf/poolindex.go index cdf8ce6432..1d588c5f38 100644 --- a/cmd/codeaf/poolindex.go +++ b/cmd/codeaf/poolindex.go @@ -1,18 +1,16 @@ -// The Model Pool's index, seated at start-up. +// The Model Pool's index, kept fresh at start-up. // -// The index is a measurement document the crew picker reads a seat's quality -// from (config.PoolQualityMetric, through config.AutoIndex). A machine that -// has fetched one keeps it under the profile's pool directory; a machine that -// has never fetched one still has the index the binary carries -// (internal/pool/index's Seed), so a tier row that says `auto` has numbers on -// its first run rather than none. +// The index is a measurement document the pool publishes: how each model did +// in each seat across the installs that share their judged runs. A machine +// that fetches one keeps it under the profile's pool directory, where +// `codeaf pool` reads it. The one fetch this file makes runs in a goroutine +// started where the pool is wired, and it never blocks a run: it writes the +// puller's own cache and the change is read at the NEXT start. // -// READ ONCE, NEVER ON THE RUN'S PATH. [poolIndexFor] parses the cache beside -// the seed exactly once and hands back a function answering the value it -// already has, so a pick asks no disk and no lock. The one fetch this file -// makes runs in a goroutine started where the loader is seated, and it never -// answers a pick: it writes the puller's own cache and the change is read at -// the NEXT start. A run's picks must not move under it. +// Nothing here seats a crew. The crew is routed per task from an evidence +// table the build carries (internal/crewroute); reading the pool's cells into +// that router as measured evidence is follow-up work, not something this file +// half-does. package main import ( @@ -20,55 +18,16 @@ import ( "crypto/ed25519" "errors" "log" - "os" - "path/filepath" "sync" "time" "github.com/Agent-Field/codeaf/internal/config" - "github.com/Agent-Field/codeaf/internal/crewpick" "github.com/Agent-Field/codeaf/internal/guard" - "github.com/Agent-Field/codeaf/internal/pool/index" "github.com/Agent-Field/codeaf/internal/pool/poolcfg" "github.com/Agent-Field/codeaf/internal/pool/pull" - "github.com/Agent-Field/codeaf/internal/pool/record" "github.com/Agent-Field/codeaf/internal/trace" ) -// poolIndexFor builds this process's one index reader. The read is -// start-up work and nothing on a run's path: the document is parsed here, -// once, and the returned function answers the value already in hand, so a -// pick over many tiers does no disk and no decode per call. -// -// THE CACHE WINS ONLY WHEN IT IS FRESHER. A cached doc.json is handed to -// index.Fallback beside the embedded seed, which trusts whichever declares -// the newer generated day — a young cache keeps its numbers, an old or -// unparsable one leaves the seed in place. A mode that forbids reading -// (poolcfg.Off) answers no index at all, which is the same nothing the seam -// read before this file existed. -// -// THE ONE ERROR IT CANNOT RECOVER FROM is an embedded seed that does not -// parse; that answers nil rather than panicking, though a test in the index -// package parses the very bytes this build carries. -func poolIndexFor(profileDir string, cfg poolcfg.Config, now func() time.Time) func() *index.Index { - if !cfg.CanRead() { - return func() *index.Index { return nil } - } - var cached []byte - if doc, err := os.ReadFile(filepath.Join(config.ProfilePath(profileDir, "pool"), "doc.json")); err == nil { - cached = doc - } - held, err := index.Fallback(cached, index.Seed()) - if err != nil { - seed, seedErr := index.SeedIndex() - if seedErr != nil { - return func() *index.Index { return nil } - } - held = seed - } - return func() *index.Index { return held } -} - // poolRefreshGo is [guard.Go] behind a variable so a test can prove that no // goroutine is started when the build carries no key. var poolRefreshGo = func(scope string, fn func()) { guard.Go(scope, fn) } @@ -244,17 +203,11 @@ func poolIndexPull(ctx context.Context, url, poolDir string, ttl time.Duration, return err } -// wirePoolIndex seats this process's pool readings beside its catalog: it -// hands the index reader to seat resolution (config.AutoIndex), the install's -// own judged scores beside them (config.AutoOwnCells, read off the own sheet -// under the pool directory), and starts the refresh beside both. It is called -// once at start-up, from the same places config.AutoModels is set, so a tier -// row that says `auto` resolves against a prior on every door that resolves a -// seat. +// wirePoolIndex starts this process's pool errands: the index refresh, and +// the push of rows a previous run judged and could not hand over. It is +// called once at start-up, from the places the catalog is seated. func wirePoolIndex(profileDir string) { cfg := config.ModelPoolAt(profileDir) - config.AutoIndex = poolIndexFor(profileDir, cfg, time.Now) - config.AutoOwnCells = poolOwnCellsFor(profileDir, cfg) // The errands below run on this profile's tracker so the process that // seated them can join them when it closes ([stopPoolErrands]). held := poolErrandsStart(profileDir) @@ -267,28 +220,3 @@ func wirePoolIndex(profileDir string) { poolErrandGo(profileDir, "pool/push", func() { poolPush(held.ctx, profileDir, cfg, poolPushBudget) }) } } - -// poolOwnCellsFor builds this process's one reader of the install's own judged -// scores. THE READ IS START-UP WORK AND NOTHING ON A RUN'S PATH, the same -// posture the index reader keeps: the own sheet under the profile's pool -// directory is read and parsed here, once, and the returned function answers -// the cells already in hand, so a pick does no disk and no decode per call. -// A mode that forbids reading answers nil — the same nothing the seam reads -// before any sheet exists — and so does a sheet that does not parse, which is -// said under the debug record's switch and read as absent rather than fatal: -// an own sheet is the install's own evidence, and a broken one is a loss, not -// a fault a pick should stop for. -func poolOwnCellsFor(profileDir string, cfg poolcfg.Config) func() []crewpick.Cell { - if !cfg.CanRead() { - return nil - } - sheet, err := record.LoadSheet(record.OwnSheetPath(config.ProfilePath(profileDir, "pool"))) - if err != nil { - if trace.Enabled() { - log.Printf("model pool: own sheet: %v", err) - } - return nil - } - cells := record.Cells(sheet) - return func() []crewpick.Cell { return cells } -} diff --git a/cmd/codeaf/pooljudge_close_test.go b/cmd/codeaf/pooljudge_close_test.go index 00c05c18da..7cf45ae9a7 100644 --- a/cmd/codeaf/pooljudge_close_test.go +++ b/cmd/codeaf/pooljudge_close_test.go @@ -167,7 +167,6 @@ func TestJudgeLandingLeftUnjudgedWhenCancelledMidJudge(t *testing.T) { t.Setenv("CODEAF_HOME", t.TempDir()) t.Setenv("CODEAF_MODEL_POOL", "on") t.Setenv("CODEAF_MODEL_POOL_SUBMIT_URL", "http://127.0.0.1:1/submit") - restoreOwnCells(t) profileDir := t.TempDir() poolDir := config.ProfilePath(profileDir, "pool") diff --git a/cmd/codeaf/poolrecord.go b/cmd/codeaf/poolrecord.go index a463039294..b8cbeedd96 100644 --- a/cmd/codeaf/poolrecord.go +++ b/cmd/codeaf/poolrecord.go @@ -30,7 +30,6 @@ import ( "github.com/Agent-Field/agentfield/sdk/go/ai" "github.com/Agent-Field/codeaf/internal/catalog" "github.com/Agent-Field/codeaf/internal/config" - "github.com/Agent-Field/codeaf/internal/crewpick" "github.com/Agent-Field/codeaf/internal/guard" "github.com/Agent-Field/codeaf/internal/lane" "github.com/Agent-Field/codeaf/internal/pool/judge" @@ -277,12 +276,6 @@ func poolJudgeLandingContext(ctx context.Context, settings config.Config, profil // never judges it again. The live hook writes this but does not read it, so a // resettle still re-judges; only the sweep reads it. markJudged(poolDir, landing.ID, landing.Attempt) - // The next pick in this process reads the new cells at once, the same way - // the picker reads them at start-up (poolindex.go's poolOwnCellsFor): a - // closing one is the install's own evidence and is never held to the - // index's min_installs. - cells := record.Cells(sheet) - config.AutoOwnCells = func() []crewpick.Cell { return cells } // The sheet is saved, so what the recorder appended is the install's own // evidence now; the copies waiting in the outbox leave for the relay // here, on this hook's own goroutine (the session runs TaskLanded on diff --git a/cmd/codeaf/poolrecord_test.go b/cmd/codeaf/poolrecord_test.go index e1ddaf1efa..6d1b763cd9 100644 --- a/cmd/codeaf/poolrecord_test.go +++ b/cmd/codeaf/poolrecord_test.go @@ -83,14 +83,6 @@ func poolTildeCatalog() []catalog.Model { } } -// restoreOwnCells puts the picker's own-cells seam back after a test that -// repointed it, so no other test in the process reads this test's sheet. -func restoreOwnCells(t *testing.T) { - t.Helper() - previous := config.AutoOwnCells - t.Cleanup(func() { config.AutoOwnCells = previous }) -} - // TestPoolJudgeHookScoresALandedTaskIntoItsOwnSheetAndAnswersTheNewCells runs // one landing through the real hook and reads back the whole of it: one cell // per held seat in the install's own sheet, the seam answering those cells at @@ -100,7 +92,6 @@ func TestPoolJudgeHookScoresALandedTaskIntoItsOwnSheetAndAnswersTheNewCells(t *t t.Setenv("CODEAF_HOME", t.TempDir()) t.Setenv("CODEAF_MODEL_POOL", "on") t.Setenv("CODEAF_MODEL_POOL_SUBMIT_URL", "http://127.0.0.1:1/submit") - restoreOwnCells(t) profileDir := t.TempDir() settings := config.Config{} @@ -132,13 +123,6 @@ func TestPoolJudgeHookScoresALandedTaskIntoItsOwnSheetAndAnswersTheNewCells(t *t } } - if config.AutoOwnCells == nil { - t.Fatal("the picker's own-cells seam was not repointed after the landing") - } - if answered := config.AutoOwnCells(); len(answered) != 2 { - t.Fatalf("the seam answers %d cells, want the 2 the sheet now holds", len(answered)) - } - if len(asked) != 2 || asked[0] != "other/judge" || asked[1] != "other/judge" { t.Fatalf("the judge asked %v, want only other/judge — never a model the crew held", asked) } @@ -174,7 +158,6 @@ func TestPoolJudgeHookSpellsASeatByItsBareModelID(t *testing.T) { t.Setenv("CODEAF_HOME", t.TempDir()) t.Setenv("CODEAF_MODEL_POOL", "on") t.Setenv("CODEAF_MODEL_POOL_SUBMIT_URL", "http://127.0.0.1:1/submit") - restoreOwnCells(t) profileDir := t.TempDir() settings := config.Config{} @@ -265,7 +248,6 @@ func TestWritePendingLandingSpellsThePoolSeatsBare(t *testing.T) { // records locally only. func TestPoolJudgeHookAppendsOutboxRowsOnlyWhenTheModeSends(t *testing.T) { t.Setenv("CODEAF_HOME", t.TempDir()) - restoreOwnCells(t) profileDir := t.TempDir() outboxFile := filepath.Join(config.ProfilePath(profileDir, "pool"), "outbox.jsonl") @@ -319,7 +301,6 @@ func TestPoolJudgeHookAppendsOutboxRowsOnlyWhenTheModeSends(t *testing.T) { func TestPoolJudgeHookDoesNothingUnderModeOff(t *testing.T) { t.Setenv("CODEAF_HOME", t.TempDir()) t.Setenv("CODEAF_MODEL_POOL", "off") - restoreOwnCells(t) profileDir := t.TempDir() settings := config.Config{} @@ -379,7 +360,6 @@ func TestPoolJudgeHookGivesEachSeatsQuestionItsOwnShareOfTheLandingTime(t *testi t.Setenv("CODEAF_HOME", t.TempDir()) t.Setenv("CODEAF_MODEL_POOL", "on") t.Setenv("CODEAF_MODEL_POOL_SUBMIT_URL", "http://127.0.0.1:1/submit") - restoreOwnCells(t) profileDir := t.TempDir() settings := config.Config{} @@ -462,7 +442,6 @@ func TestPoolJudgeHookStillScoresTheSecondSeatWhenTheFirstSeatsShareRunsOut(t *t t.Setenv("CODEAF_HOME", t.TempDir()) t.Setenv("CODEAF_MODEL_POOL", "on") t.Setenv("CODEAF_MODEL_POOL_SUBMIT_URL", "http://127.0.0.1:1/submit") - restoreOwnCells(t) profileDir := t.TempDir() settings := config.Config{} @@ -525,7 +504,6 @@ func TestPoolJudgeHookRecordsTheJudgeThatScoredTheLanding(t *testing.T) { t.Setenv("CODEAF_HOME", t.TempDir()) t.Setenv("CODEAF_MODEL_POOL", "on") t.Setenv("CODEAF_MODEL_POOL_SUBMIT_URL", "http://127.0.0.1:1/submit") - restoreOwnCells(t) profileDir := t.TempDir() settings := config.Config{} @@ -564,7 +542,6 @@ func TestPoolJudgeHookRecordsTheCandidatesAndReasonWhenEveryJudgeFails(t *testin t.Setenv("CODEAF_HOME", t.TempDir()) t.Setenv("CODEAF_MODEL_POOL", "on") t.Setenv("CODEAF_MODEL_POOL_SUBMIT_URL", "http://127.0.0.1:1/submit") - restoreOwnCells(t) profileDir := t.TempDir() settings := config.Config{} @@ -594,7 +571,6 @@ func TestPoolJudgeHookRecordsTheDeclineWhenThereIsNoCandidate(t *testing.T) { t.Setenv("CODEAF_HOME", t.TempDir()) t.Setenv("CODEAF_MODEL_POOL", "on") t.Setenv("CODEAF_MODEL_POOL_SUBMIT_URL", "http://127.0.0.1:1/submit") - restoreOwnCells(t) profileDir := t.TempDir() settings := config.Config{} @@ -689,7 +665,6 @@ func TestPoolJudgeHookMovesToTheNextCandidateWhenTheCheapestAnswersNothing(t *te t.Setenv("CODEAF_HOME", t.TempDir()) t.Setenv("CODEAF_MODEL_POOL", "on") t.Setenv("CODEAF_MODEL_POOL_SUBMIT_URL", "http://127.0.0.1:1/submit") - restoreOwnCells(t) profileDir := t.TempDir() settings := config.Config{} @@ -767,7 +742,6 @@ func TestPoolJudgeHookWritesNothingWhenEveryCandidateAnswersNothing(t *testing.T t.Setenv("CODEAF_HOME", t.TempDir()) t.Setenv("CODEAF_MODEL_POOL", "on") t.Setenv("CODEAF_MODEL_POOL_SUBMIT_URL", "http://127.0.0.1:1/submit") - restoreOwnCells(t) profileDir := t.TempDir() settings := config.Config{} @@ -839,7 +813,6 @@ func TestPoolJudgeSweepJudgesThePendingRowsAndLeavesARecordOfItself(t *testing.T t.Setenv("CODEAF_HOME", t.TempDir()) t.Setenv("CODEAF_MODEL_POOL", "on") t.Setenv("CODEAF_MODEL_POOL_SUBMIT_URL", "http://127.0.0.1:1/submit") - restoreOwnCells(t) profileDir := t.TempDir() settings := config.Config{APIKey: "test-key"} @@ -889,7 +862,6 @@ func TestPoolJudgeSweepCutByItsBudgetLeavesTheRestAndSaysSo(t *testing.T) { t.Setenv("CODEAF_HOME", t.TempDir()) t.Setenv("CODEAF_MODEL_POOL", "on") t.Setenv("CODEAF_MODEL_POOL_SUBMIT_URL", "http://127.0.0.1:1/submit") - restoreOwnCells(t) profileDir := t.TempDir() settings := config.Config{APIKey: "test-key"} @@ -944,7 +916,6 @@ func TestPoolJudgeSweepCutByItsBudgetLeavesTheRestAndSaysSo(t *testing.T) { func TestPoolJudgeSweepLeavesNoRecordWhenThePoolCannotRead(t *testing.T) { t.Setenv("CODEAF_HOME", t.TempDir()) t.Setenv("CODEAF_MODEL_POOL", "off") - restoreOwnCells(t) profileDir := t.TempDir() poolJudgeSweep(config.Config{APIKey: "test-key"}, profileDir, "", poolTestCatalog, poolTestAsk(config.Config{}, new([]string)), time.Now) diff --git a/cmd/codeaf/poolsweep_test.go b/cmd/codeaf/poolsweep_test.go index dc71e7481e..1d5fdfcf51 100644 --- a/cmd/codeaf/poolsweep_test.go +++ b/cmd/codeaf/poolsweep_test.go @@ -25,7 +25,6 @@ func TestPoolJudgeSweepJudgesAPendingRowOnceThenNeverAgain(t *testing.T) { t.Setenv("CODEAF_HOME", t.TempDir()) t.Setenv("CODEAF_MODEL_POOL", "on") t.Setenv("CODEAF_MODEL_POOL_SUBMIT_URL", "http://127.0.0.1:1/submit") - restoreOwnCells(t) profileDir := t.TempDir() poolDir := config.ProfilePath(profileDir, "pool") @@ -70,7 +69,6 @@ func TestPoolJudgeSweepWithNoKeyLeavesRowsWaiting(t *testing.T) { t.Setenv("CODEAF_HOME", t.TempDir()) t.Setenv("CODEAF_MODEL_POOL", "on") t.Setenv("CODEAF_MODEL_POOL_SUBMIT_URL", "http://127.0.0.1:1/submit") - restoreOwnCells(t) profileDir := t.TempDir() poolDir := config.ProfilePath(profileDir, "pool") @@ -99,7 +97,6 @@ func TestPoolJudgeSweepToleratesATornPendingLine(t *testing.T) { t.Setenv("CODEAF_HOME", t.TempDir()) t.Setenv("CODEAF_MODEL_POOL", "on") t.Setenv("CODEAF_MODEL_POOL_SUBMIT_URL", "http://127.0.0.1:1/submit") - restoreOwnCells(t) profileDir := t.TempDir() poolDir := config.ProfilePath(profileDir, "pool") @@ -155,7 +152,6 @@ func TestPoolJudgeSweepCutShortKeepsALeftoverClaimsRows(t *testing.T) { t.Setenv("CODEAF_HOME", t.TempDir()) t.Setenv("CODEAF_MODEL_POOL", "on") t.Setenv("CODEAF_MODEL_POOL_SUBMIT_URL", "http://127.0.0.1:1/submit") - restoreOwnCells(t) profileDir := t.TempDir() poolDir := config.ProfilePath(profileDir, "pool") diff --git a/cmd/codeaf/role_ladder.go b/cmd/codeaf/role_ladder.go index f4cfdded2c..d733b6c867 100644 --- a/cmd/codeaf/role_ladder.go +++ b/cmd/codeaf/role_ladder.go @@ -25,7 +25,7 @@ import ( // never quietly undoes what somebody chose in the palette. // // planOrigin is who named it, in the words a person would recognize — -// `CODEAF_PLAN_MODEL`, `--plan-model`, `crew frugal` — and it is passed in +// `CODEAF_PLAN_MODEL`, `--plan-model`, `routed`, `pinned` — and it is passed in // rather than guessed at here, because since #166 the answer may have come from // the profile's crew and a seed that named a variable nobody set would be a // false record in the journal. diff --git a/cmd/codeaf/role_ladder_test.go b/cmd/codeaf/role_ladder_test.go index 993a1a3546..8b29319c41 100644 --- a/cmd/codeaf/role_ladder_test.go +++ b/cmd/codeaf/role_ladder_test.go @@ -86,7 +86,7 @@ func TestPlanKnobSeedsTheGlobalPlanBindingOnce(t *testing.T) { // Another origin names itself instead, and a value the initializer has not // said before does move the binding. The origin is whatever named the model - // — a flag here, a crew preset on a headless run (config.ResolveSeats). + // — a flag here, a crew pin or the router on a headless run (config.ResolveSeats). installRoleLadder(graph, "talk/model", "flagged/model", "work/model", "flagged/model", "--plan-model") binding, _, err = graph.RoleBindingAt(store.RolePlan, store.ScopeGlobal) if err != nil { diff --git a/cmd/codeaf/run.go b/cmd/codeaf/run.go index c6fa7377c0..d3ca63dfba 100644 --- a/cmd/codeaf/run.go +++ b/cmd/codeaf/run.go @@ -191,7 +191,7 @@ func runGraph(name string, args []string) error { return err } useAutoSeats(settings) - seats := config.ResolveSeats(settings.ProfileDir, *model, *planModel) + seats := doorSeats(settings, *model, *planModel, graph.Goal) applySeats(&settings, seats) // A graph may be loaded from disk and expanded again after an overrun, so // run installs the measured ruler before any planning-capable work starts. diff --git a/cmd/codeaf/seats_wiring_test.go b/cmd/codeaf/seats_wiring_test.go index 81456400d0..0e2281a4e1 100644 --- a/cmd/codeaf/seats_wiring_test.go +++ b/cmd/codeaf/seats_wiring_test.go @@ -1,22 +1,17 @@ package main import ( - "context" "encoding/json" - "io" - "net/http" - "net/http/httptest" "os" - "path/filepath" "strings" "sync" "testing" "time" - "github.com/Agent-Field/agentfield/sdk/go/ai" + "github.com/Agent-Field/codeaf/internal/catalog" "github.com/Agent-Field/codeaf/internal/config" - "github.com/Agent-Field/codeaf/internal/roles" - "github.com/Agent-Field/codeaf/internal/store" + "github.com/Agent-Field/codeaf/internal/crewroute" + "github.com/Agent-Field/codeaf/internal/router" ) // ONE LADDER, EVERY HEADLESS DOOR, AND NO SECOND COPY OF IT. @@ -25,112 +20,114 @@ import ( // resolving their two models their own way, so a crew that reached the chat // reached none of them (#166). The next door will be written by copying one of // these, and the copy is only safe while the resolution is a call rather -// than a paragraph worth of lookups. So this reads the source: every door asks -// config.ResolveSeats, and no door reaches past it for the environment or the -// build's default on its own. +// than a paragraph worth of lookups. So this reads the source: `codeaf do` +// asks config.ResolveSeats itself, every other door asks doorSeats (which asks +// it once), and no door reaches past it for the environment on its own. func TestEveryHeadlessDoorResolvesItsSeatsThroughTheOneLadder(t *testing.T) { - // main.go carries two doors — plan and revise — and the other three files - // carry one each. - doors := map[string]int{ - "do.go": 1, "exec.go": 1, "run.go": 1, "main.go": 2, "subharness_run.go": 1, + ladder := map[string]int{"do.go": 1, "main.go": 1} + doors := map[string]int{"exec.go": 1, "run.go": 1, "main.go": 3, "subharness_run.go": 1} + for name, wanted := range ladder { + raw, err := os.ReadFile(name) + if err != nil { + t.Fatal(err) + } + if got := strings.Count(string(raw), "config.ResolveSeats("); got != wanted { + t.Errorf("%s climbs the ladder %d times, want %d", name, got, wanted) + } } for name, wanted := range doors { raw, err := os.ReadFile(name) if err != nil { t.Fatal(err) } - source := string(raw) - if got := strings.Count(source, "config.ResolveSeats("); got != wanted { - t.Errorf("%s climbs the ladder %d times, want %d — a door that resolves its models "+ + if got := strings.Count(string(raw), "doorSeats("); got != wanted { + t.Errorf("%s asks doorSeats %d times, want %d — a door that resolves its models "+ "another way is a door the crew does not reach", name, got, wanted) } - // A door either seats the answer on its own settings or hands it to the - // brain that does (brainOptions.seats). What it may not do is resolve - // the seats and then build its clients from something else. - if !strings.Contains(source, "applySeats(") && !strings.Contains(source, "seats:") { + } + for _, name := range []string{"do.go", "exec.go", "run.go", "main.go", "subharness_run.go"} { + raw, err := os.ReadFile(name) + if err != nil { + t.Fatal(err) + } + source := string(raw) + if !strings.Contains(source, "applySeats(") && !strings.Contains(source, "seats:") && !strings.Contains(source, "seats.Work.Model") { t.Errorf("%s resolves the seats and never seats them", name) } for _, reach := range []string{ `os.Getenv("CODEAF_MODEL")`, `os.Getenv("CODEAF_PLAN_MODEL")`, + `os.Getenv("CODEAF_CHECK_MODEL")`, "os.Getenv(config.ModelEnv)", "os.Getenv(config.PlanModelEnv)", } { if strings.Contains(source, reach) { - t.Errorf("%s reads %s for itself; the ladder reads the environment, "+ - "and it is the only rung that can tell the environment from the default", name, reach) + t.Errorf("%s reads %s for itself; the ladder reads the environment", name, reach) } } - } - - // The build's default is the ladder's bottom rung and nothing else's. It - // survives in main.go exactly once — in the usage table, where the row for - // CODEAF_MODEL prints the default it falls back to — and that is prose, not - // resolution. - for _, name := range []string{"do.go", "exec.go", "run.go", "subharness_run.go"} { - raw, err := os.ReadFile(name) - if err != nil { - t.Fatal(err) - } - if strings.Contains(string(raw), "config.DefaultModel") { - t.Errorf("%s names the build's default model itself instead of falling to the ladder's last rung", name) - } - } - raw, err := os.ReadFile("main.go") - if err != nil { - t.Fatal(err) - } - for _, line := range strings.Split(string(raw), "\n") { - if strings.Contains(line, "config.DefaultModel") && !strings.Contains(line, "CODEAF_MODEL") { - t.Errorf("main.go reads the build's default outside the usage table: %q", strings.TrimSpace(line)) + // NO SEAT FALLS TO A MODEL THIS BUILD CHOSE FOR EVERYBODY. + if name != "main.go" && strings.Contains(source, "config.DefaultModel") { + t.Errorf("%s names the build's default model itself", name) } } } -// The help text is where a person learns that their crew reaches this command, -// so it names the whole ladder rather than one rung of it. +// The help text is where a person learns that the flag is a one-task pin and +// that the crew is routed per task when nothing is pinned. func TestTheModelFlagsNameTheWholeLadder(t *testing.T) { - for _, name := range []string{"do.go", "exec.go", "run.go", "main.go", "subharness_run.go"} { - raw, err := os.ReadFile(name) - if err != nil { - t.Fatal(err) - } - if strings.Contains(string(raw), "run (default CODEAF_MODEL)") { - t.Errorf("%s still says the work model defaults to CODEAF_MODEL, which was one rung of four", name) + for _, help := range []string{modelFlagHelp, planModelFlagHelp, checkModelFlagHelp} { + for _, want := range []string{"one-task pin", "crew pin", "routed per task"} { + if !strings.Contains(help, want) { + t.Errorf("a model flag's help does not say %q: %q", want, help) + } } } - for _, want := range []string{"CODEAF_MODEL", "crew", "default"} { - if !strings.Contains(modelFlagHelp, want) { - t.Errorf("--model's help does not mention %q: %q", want, modelFlagHelp) - } + if !strings.Contains(checkModelFlagHelp, "CODEAF_CHECK_MODEL") { + t.Errorf("--check-model's help does not name its variable: %q", checkModelFlagHelp) } - if !strings.Contains(planModelFlagHelp, "CODEAF_PLAN_MODEL") || !strings.Contains(planModelFlagHelp, "crew") { - t.Errorf("--plan-model's help does not name its own ladder: %q", planModelFlagHelp) + // THE CHECK SEAT'S LADDER NEVER PASSES THROUGH THE PLAN SEAT. + if strings.Contains(checkLadderHelp, "plan") { + t.Errorf("the check seat's ladder still names the plan seat: %q", checkLadderHelp) } - for _, want := range []string{"CODEAF_CHECK_MODEL", "plan", "crew"} { - if !strings.Contains(checkModelFlagHelp, want) { - t.Errorf("--check-model's help does not mention %q: %q", want, checkModelFlagHelp) - } +} + +// crewDoorCatalog is a catalog the router can price: a cheap model and a +// dear, better one, both open and both serving tools. +func crewDoorCatalog() []catalog.Model { + return []catalog.Model{ + {ID: "vendor/cheap", PromptPrice: 0.1e-6, CompletionPrice: 0.4e-6, CodingIndex: 40, AgenticIndex: 40, IntelligenceIndex: 40, + Parameters: []string{"tools"}, ContextLength: 200000, OpenWeights: true}, + {ID: "vendor/strong", PromptPrice: 3e-6, CompletionPrice: 15e-6, CodingIndex: 70, AgenticIndex: 70, IntelligenceIndex: 70, + Parameters: []string{"tools"}, ContextLength: 200000, OpenWeights: true}, } } -// THE ACCEPTANCE CASE, END TO END: a profile that says `frugal` and a command -// line that says nothing. +// seatCrewCatalog points the router at a catalog for one test. +func seatCrewCatalog(t *testing.T, rows []catalog.Model) { + t.Helper() + previous, seat := config.CrewCatalog, seatCrewRows + config.CrewCatalog = func() []catalog.Model { return rows } + seatCrewRows = func(func() []catalog.Model) {} + t.Cleanup(func() { config.CrewCatalog, seatCrewRows = previous, seat }) +} + +// THE ACCEPTANCE CASE, END TO END: a command line that says nothing, on a +// profile that pinned only the checker. // -// The errand must plan on the profile's mastermind and work on its working tier -// — the two seats the chat's planner and worker ride (roles.DefaultAssignment) -// — and it must SAY SO, on the opening line and in the object a harness reads. -// Both halves matter: a benchmark that cannot read back which crew ran is the -// position this issue was reported from. -func TestAnErrandWithNoFlagsRunsTheProfilesCrewAndSaysSo(t *testing.T) { +// The worker and the planner are ROUTED for this task and the checker is the +// pin, and the run says so — on the opening lines, in the summary line with the +// actual beside the estimate, and in the object a harness reads (`class`, +// `crew`, `est_usd`, `check_model`). The decision and its outcome land in the +// router's log. +func TestAnErrandWithNoFlagsRunsARoutedCrewAndSaysSo(t *testing.T) { script := newScriptedBrain(t) defer script.close() - // The variables are cleared rather than left to the shell this suite runs - // in: the rung under test is the one below them. t.Setenv(config.ModelEnv, "") t.Setenv(config.PlanModelEnv, "") - if err := config.ApplyCrew(script.dir, config.CrewFrugal); err != nil { - t.Fatalf("writing the frugal crew: %v", err) + t.Setenv(config.CheckModelEnv, "") + seatCrewCatalog(t, crewDoorCatalog()) + if err := config.SetCrewPin(script.dir, crewroute.Checker, "vendor/strong"); err != nil { + t.Fatalf("pinning the checker: %v", err) } var mu sync.Mutex @@ -138,6 +135,7 @@ func TestAnErrandWithNoFlagsRunsTheProfilesCrewAndSaysSo(t *testing.T) { var stdout, stderr strings.Builder err := doErrand(doRequest{ task: "write the release note and include the migration steps", + run: "crew-door-run", timeout: 60 * time.Second, asJSON: true, stdout: &stdout, @@ -152,63 +150,48 @@ func TestAnErrandWithNoFlagsRunsTheProfilesCrewAndSaysSo(t *testing.T) { if err != nil { t.Fatalf("the errand did not settle cleanly: %v\nstderr:\n%s", err, stderr.String()) } - - // THE WORK SEAT IS THE WORKER ROW, which is what [config.ResolveSeats] fills - // it from. It used to be read off the small-work row here and pass anyway, - // because the frugal preset happened to put one id on both. - work := config.TierModelAt(script.dir, config.ModelTierWorker) - plan := config.TierModelAt(script.dir, config.ModelTierMastermind) - for _, model := range []string{work, plan} { - found := false - mu.Lock() - for _, built := range built { - found = found || built == model - } - models := append([]string(nil), built...) - mu.Unlock() - if !found { - t.Fatalf("no client was built on %q; the run used %v", model, models) - } + if !strings.Contains(stderr.String(), "crew: ") || !strings.Contains(stderr.String(), config.PinMark+" strong") { + t.Fatalf("the run never said its crew with the pinned checker marked:\n%s", stderr.String()) } - if line := config.ResolveSeats(script.dir, "", "").Line(); !strings.Contains(stderr.String(), line) { - t.Fatalf("the opening lines never named the crew:\nwant %q\nstderr:\n%s", line, stderr.String()) + var fields map[string]any + if err := json.Unmarshal([]byte(stdout.String()), &fields); err != nil { + t.Fatalf("--json did not print one object: %v\n%s", err, stdout.String()) } - if !strings.Contains(stderr.String(), "crew frugal") { - t.Fatalf("the receipt did not name the preset:\n%s", stderr.String()) + for _, key := range []string{"class", "crew", "est_usd", "check_model", "model_source"} { + if _, ok := fields[key]; !ok { + t.Fatalf("--json carries no %q:\n%s", key, stdout.String()) + } } - - var outcome headlessOutcome - if err := json.Unmarshal([]byte(stdout.String()), &outcome); err != nil { - t.Fatalf("--json did not print one object: %v\n%s", err, stdout.String()) + if fields["check_model"] != "vendor/strong" || fields["check_model_source"] != "pinned" { + t.Fatalf("--json named the checker %v (%v), want the pin", fields["check_model"], fields["check_model_source"]) } - if outcome.Model != work || outcome.PlanModel != plan { - t.Fatalf("--json named model %q and plan_model %q, want %q and %q", - outcome.Model, outcome.PlanModel, work, plan) + if fields["model_source"] != "routed" { + t.Fatalf("--json named the worker's rung %v, want routed", fields["model_source"]) } - if outcome.ModelSource != "crew frugal" || outcome.PlanModelSource != "crew frugal" { - t.Fatalf("--json named the rungs %q and %q, want the crew for both", - outcome.ModelSource, outcome.PlanModelSource) + + log := router.ReadCrewLog(config.ProfilePath(script.dir, ""), time.Now()) + if len(log.Recent) != 1 || !log.Recent[0].Settled { + t.Fatalf("the router's log holds %+v, want one settled crew", log.Recent) } } -// The flag still wins, and the receipt still says which rung answered — the two -// halves of the ladder that a benchmark pinning one seat depends on. -func TestAFlaggedSeatOutranksTheCrewAndTheReceiptSaysWhich(t *testing.T) { +// The flag still wins, and it is a ONE-TASK PIN: the receipt says which rung +// answered, and the check seat does not inherit a flagged planner. +func TestAFlaggedPlannerNeverSeatsTheChecker(t *testing.T) { script := newScriptedBrain(t) defer script.close() t.Setenv(config.ModelEnv, "") t.Setenv(config.PlanModelEnv, "") - if err := config.ApplyCrew(script.dir, config.CrewFrugal); err != nil { - t.Fatal(err) - } + t.Setenv(config.CheckModelEnv, "") + seatCrewCatalog(t, crewDoorCatalog()) var stdout, stderr strings.Builder err := doErrand(doRequest{ task: "write the release note and include the migration steps", timeout: 60 * time.Second, asJSON: true, - model: "vendor/pinned-worker", + planModel: "vendor/flagged-planner", stdout: &stdout, stderr: &stderr, newClient: script.client, @@ -220,346 +203,59 @@ func TestAFlaggedSeatOutranksTheCrewAndTheReceiptSaysWhich(t *testing.T) { if err := json.Unmarshal([]byte(stdout.String()), &outcome); err != nil { t.Fatalf("--json did not print one object: %v\n%s", err, stdout.String()) } - if outcome.Model != "vendor/pinned-worker" || outcome.ModelSource != "--model" { - t.Fatalf("the flagged seat reads %q (%s)", outcome.Model, outcome.ModelSource) + if outcome.PlanModel != "vendor/flagged-planner" || outcome.PlanModelSource != "--plan-model" { + t.Fatalf("the flagged seat reads %q (%s)", outcome.PlanModel, outcome.PlanModelSource) } - // And the seat nobody flagged still climbs to the crew. - if outcome.PlanModelSource != "crew frugal" { - t.Fatalf("the plan seat reads %q (%s)", outcome.PlanModel, outcome.PlanModelSource) + var fields map[string]any + _ = json.Unmarshal([]byte(stdout.String()), &fields) + if fields["check_model"] == "vendor/flagged-planner" { + t.Fatal("the check seat inherited the flagged planner") } } -// A crew preset fills the run and durable role binding with its bare model id. -// Presets choose models and leave generation behavior to them. -func TestACrewsBareModelReachesTheRunWhole(t *testing.T) { +// AT THE DAILY CAP `codeaf do` REFUSES, and -yes-spend is the one way past. +func TestAnErrandAtTheDailyCapRefusesUnlessToldToSpend(t *testing.T) { script := newScriptedBrain(t) defer script.close() t.Setenv(config.ModelEnv, "") t.Setenv(config.PlanModelEnv, "") - if err := config.ApplyCrew(script.dir, config.CrewBalanced); err != nil { + t.Setenv(config.CheckModelEnv, "") + seatCrewCatalog(t, crewDoorCatalog()) + if err := config.SetCrewCap(script.dir, "1"); err != nil { t.Fatal(err) } - written := config.TierModelAt(script.dir, config.ModelTierMastermind) - if _, level := roles.SplitEffort(written); level != "" { - t.Fatalf("the balanced mastermind reads %q and imposes level %q", written, level) - } + previous := config.CrewHistory + config.CrewHistory = func(string) config.CrewDay { return config.CrewDay{SpentUSD: 2} } + t.Cleanup(func() { config.CrewHistory = previous }) - // A durable store, because the plan role's binding is the record under test - // and an ephemeral one evaporates with the run. - database := filepath.Join(t.TempDir(), "graph.db") - var mu sync.Mutex - var built []string var stdout, stderr strings.Builder err := doErrand(doRequest{ - task: "write the release note and include the migration steps", - database: database, - timeout: 60 * time.Second, - asJSON: true, - stdout: &stdout, - stderr: &stderr, - newClient: func(settings config.Config, model string) (*liveClient, error) { - mu.Lock() - built = append(built, model) - mu.Unlock() - return script.client(settings, model) - }, + task: "write the release note and include the migration steps", timeout: 60 * time.Second, + stdout: &stdout, stderr: &stderr, newClient: script.client, }) - if err != nil { - t.Fatalf("the errand did not settle cleanly: %v\nstderr:\n%s", err, stderr.String()) - } - - mu.Lock() - models := append([]string(nil), built...) - mu.Unlock() - found := false - for _, model := range models { - found = found || model == written - } - if !found { - t.Fatalf("the plan seat was filled with %v, none of them the crew's own %q", models, written) - } - - // The binding the ladder resolves calls through carries the level, and says - // the crew put it there rather than naming a variable nobody set. - graph, err := store.Open(database) - if err != nil { - t.Fatal(err) - } - defer graph.Close() - binding, ok, err := graph.RoleBindingAt(store.RolePlan, store.ScopeGlobal) - if err != nil || !ok { - t.Fatalf("plan binding: found=%t err=%v", ok, err) - } - if binding.Value != written { - t.Fatalf("the plan role is bound to %q, want the crew's %q", binding.Value, written) - } - if binding.Origin != store.RoleSeedOriginPrefix+"crew balanced" { - t.Fatalf("the binding says %q named the model", binding.Origin) - } - - // And what the run reports is what the person wrote in the sheet. - var outcome headlessOutcome - if err := json.Unmarshal([]byte(stdout.String()), &outcome); err != nil { - t.Fatalf("--json did not print one object: %v\n%s", err, stdout.String()) - } - if outcome.PlanModel != written { - t.Fatalf("--json named plan_model %q, want %q", outcome.PlanModel, written) - } - if !strings.Contains(stderr.String(), "plan "+written+" (crew balanced)") { - t.Fatalf("the receipt did not print the value as the sheet holds it:\n%s", stderr.String()) - } -} - -// The plan seat resolved from an unoverridden balanced profile sends its bare -// model on both contexts a headless run uses, with no reasoning object. -func TestACrewsBareModelReachesTheHeadlessWire(t *testing.T) { - dir := t.TempDir() - t.Setenv(config.ModelEnv, "") - t.Setenv(config.PlanModelEnv, "") - if err := config.ApplyCrew(dir, config.CrewBalanced); err != nil { - t.Fatal(err) - } - - type requestShape struct { - Model string `json:"model"` - Reasoning json.RawMessage `json:"reasoning"` + if err == nil || !strings.Contains(err.Error(), "daily cap") { + t.Fatalf("an errand at the cap was not refused: %v", err) } - var received []requestShape - server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { - if request.Method != http.MethodPost { - writer.Header().Set("Content-Type", "application/json") - _, _ = io.WriteString(writer, `[]`) - return - } - raw, err := io.ReadAll(request.Body) - if err != nil { - t.Errorf("read request: %v", err) - http.Error(writer, `{"error":"unreadable"}`, http.StatusBadRequest) - return - } - var body requestShape - if err := json.Unmarshal(raw, &body); err != nil { - t.Errorf("decode request: %v", err) - http.Error(writer, `{"error":"invalid"}`, http.StatusBadRequest) - return - } - received = append(received, body) - writer.Header().Set("Content-Type", "application/json") - _, _ = io.WriteString(writer, `{"model":"stub","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"ok"}}],"usage":{"prompt_tokens":1,"completion_tokens":1}}`) - })) - defer server.Close() - - settings := config.Config{ - APIKey: "test-key", - BaseURL: server.URL, - Timeout: config.DefaultTimeout, - Reasoning: config.DefaultReasoning, - ExecReasoning: config.DefaultExecReasoning, - ProfileDir: dir, - } - seats := config.ResolveSeats(dir, "", "") - applySeats(&settings, seats) - client, err := settings.ClientFor(settings.PlanModelResolved()) - if err != nil { - t.Fatal(err) - } - contexts := []context.Context{ - settings.Context(context.Background(), "plan this"), - settings.ExecContext(settings.Context(context.Background(), "do this")), - } - for _, ctx := range contexts { - if _, err := client.CompleteWithMessages(ctx, - []ai.Message{{Role: "user", Content: []ai.ContentPart{{Type: "text", Text: "hello"}}}}); err != nil { - t.Fatal(err) - } - } - - wantModel, wantEffort := roles.SplitEffort(config.DefaultMastermindModel) - if wantEffort != "" { - t.Fatalf("shipped mastermind %q imposes effort %q", config.DefaultMastermindModel, wantEffort) - } - if seats.Plan.Model != config.DefaultMastermindModel { - t.Fatalf("balanced plan seat = %q, want %q", seats.Plan.Model, config.DefaultMastermindModel) - } - if len(received) != len(contexts) { - t.Fatalf("provider received %d requests, want %d", len(received), len(contexts)) - } - for index, body := range received { - if body.Model != wantModel || len(body.Reasoning) != 0 { - t.Errorf("call %d carried model %q and reasoning %s, want bare %q and no reasoning object", - index, body.Model, body.Reasoning, wantModel) - } + if script.count("chat") != 0 { + t.Fatal("an errand at the cap spent before it refused") } } -// THE WHOLE ROAD, ON A PROFILE OLDER THAN THE WORKER SEAT (#302). -// -// The unit test pins the rung; this pins what a person actually gets: a config -// written before the worker row existed — the four keys and no fifth — must send -// every client the errand builds to the crew's own models, never to the build's -// default, and the run must SAY on its way past that the seat was inherited. The -// second half is the part that makes the first half checkable from outside, -// which is the property the defect took away. -func TestAnErrandOnACrewOlderThanTheWorkerSeatNeverTouchesTheBuildsDefault(t *testing.T) { - script := newScriptedBrain(t) - defer script.close() - t.Setenv(config.ModelEnv, "") - t.Setenv(config.PlanModelEnv, "") - - // The pre-#278 crew shape, written as a profile of that vintage holds it. - pinned := "vendor/pinned-small-work" - profile := map[string]string{ - config.KeyTierReflexModel: "vendor/pinned-reflex", - config.KeyTierLowModel: pinned, - config.KeyTierHighModel: "vendor/pinned-careful", - config.KeyTierMastermindModel: "vendor/pinned-thinking", - } - raw, err := json.Marshal(profile) - if err != nil { +// --pin is repeatable, names a seat, and refuses what is not a pin. +func TestThePinFlagReadsSeatEqualsModel(t *testing.T) { + var pins pinFlags + if err := pins.Set("checker=moonshotai/kimi-k3@openrouter"); err != nil { t.Fatal(err) } - if err := os.WriteFile(config.BudgetConfigPath(script.dir), raw, 0o600); err != nil { + if err := pins.Set("worker=z-ai/glm-5.3-flash"); err != nil { t.Fatal(err) } - - var mu sync.Mutex - var built []string - var stdout, stderr strings.Builder - if err := doErrand(doRequest{ - task: "write the release note and include the migration steps", - timeout: 60 * time.Second, - asJSON: true, - stdout: &stdout, - stderr: &stderr, - newClient: func(settings config.Config, model string) (*liveClient, error) { - mu.Lock() - built = append(built, model) - mu.Unlock() - return script.client(settings, model) - }, - }); err != nil { - t.Fatalf("the errand did not settle cleanly: %v\nstderr:\n%s", err, stderr.String()) + if got := pins.pins[crewroute.Checker]; got.Model != "moonshotai/kimi-k3" || got.Provider != "openrouter" { + t.Fatalf("the checker pin reads %+v", got) } - - mu.Lock() - models := append([]string(nil), built...) - mu.Unlock() - for _, model := range models { - if model == config.DefaultModel { - t.Fatalf("a client was built on the build's default %q; the run used %v", - config.DefaultModel, models) + for _, bad := range []string{"judge=vendor/x", "worker", "worker=auto"} { + if err := (&pinFlags{}).Set(bad); err == nil { + t.Errorf("--pin %q was accepted", bad) } } - worked := false - for _, model := range models { - worked = worked || model == pinned - } - if !worked { - t.Fatalf("nothing ran on the small-work model the crew pinned; the run used %v", models) - } - - // The receipt: the work seat says the crew answered and that the row was - // inherited, and the one line saying why is printed ONCE. - seats := config.ResolveSeats(script.dir, "", "") - if !strings.Contains(stderr.String(), "work "+pinned+" (crew custom, inherited)") { - t.Fatalf("the opening line does not name the inherited seat:\n%s", stderr.String()) - } - if strings.Contains(stderr.String(), "work "+config.DefaultModel) { - t.Fatalf("the opening line still seats the build's default:\n%s", stderr.String()) - } - notice := seats.Notice() - if notice == "" { - t.Fatal("the seats resolved by inheritance and the run has nothing to say about it") - } - if got := strings.Count(stderr.String(), notice); got != 1 { - t.Fatalf("the run said the line %d times, want once:\n%s", got, stderr.String()) - } - - // And the object a script reads carries the same fact. - var outcome headlessOutcome - if err := json.Unmarshal([]byte(stdout.String()), &outcome); err != nil { - t.Fatalf("--json did not print one object: %v\n%s", err, stdout.String()) - } - if outcome.Model != pinned || outcome.ModelSource != "crew custom, inherited" { - t.Fatalf("--json named model %q (%s), want the inherited crew row", - outcome.Model, outcome.ModelSource) - } -} - -// AND THE CONVERSATION SEATS THE SAME ROW (#312). -// -// The chat resolves its five classes through [v3RolesSource], which is a -// different road from [config.ResolveSeats] on purpose — a conversation has no -// flag and no campaign variable for its crew, only a profile — but it must end -// at the same model, or a person's crew means one thing in `codeaf do` and -// another in the window they actually work in. This is the ladder read the way -// the door reads it: the key internal/roles asks for, on a profile older than -// the worker seat. -func TestTheChatRoleMapSeatsTheInheritedWorkerRow(t *testing.T) { - pinned := "vendor/pinned-small-work" - dir := writeVintageProfile(t, map[string]string{ - config.KeyTierReflexModel: "vendor/pinned-reflex", - config.KeyTierLowModel: pinned, - config.KeyTierHighModel: "vendor/pinned-careful", - config.KeyTierMastermindModel: "vendor/pinned-thinking", - }) - - read, err := v3RolesSource(t.TempDir(), dir) - if err != nil { - t.Fatalf("building the conversation's role map: %v", err) - } - worker, ok := read(roles.TierKey(roles.TierWorker)) - if !ok || worker != pinned { - t.Fatalf("the conversation's worker class reads %q (held=%t), want the small-work model the crew pinned", - worker, ok) - } - if worker == config.DefaultWorkerModel { - t.Fatalf("the conversation's worker class fell to the build's default %q", config.DefaultWorkerModel) - } - // The rows the profile does hold are untouched by any of it. - if got, _ := read(roles.TierKey(roles.TierMastermind)); got != "vendor/pinned-thinking" { - t.Fatalf("the thinking class reads %q", got) - } - - // A row written by hand wins over the lineage, and a row cleared on purpose - // still means "follow the conversation" — which on this surface is a class - // the role ladder does not hold at all. - own := writeVintageProfile(t, map[string]string{ - config.KeyTierLowModel: pinned, - config.KeyTierWorkerModel: "vendor/my-own-worker", - }) - read, err = v3RolesSource(t.TempDir(), own) - if err != nil { - t.Fatal(err) - } - if got, _ := read(roles.TierKey(roles.TierWorker)); got != "vendor/my-own-worker" { - t.Fatalf("a pinned worker row reads %q", got) - } - cleared := writeVintageProfile(t, map[string]string{ - config.KeyTierLowModel: pinned, - config.KeyTierWorkerModel: "", - }) - read, err = v3RolesSource(t.TempDir(), cleared) - if err != nil { - t.Fatal(err) - } - if got, held := read(roles.TierKey(roles.TierWorker)); held { - t.Fatalf("a worker row cleared on purpose is held as %q instead of following the conversation", got) - } -} - -// writeVintageProfile writes a profile holding exactly these rows: a value for a -// row somebody wrote, the empty string for one they cleared, and no key at all -// for a class of a vintage that never had one. -func writeVintageProfile(t *testing.T, rows map[string]string) string { - t.Helper() - dir := t.TempDir() - raw, err := json.Marshal(rows) - if err != nil { - t.Fatal(err) - } - if err := os.WriteFile(config.BudgetConfigPath(dir), raw, 0o600); err != nil { - t.Fatal(err) - } - return dir } diff --git a/cmd/codeaf/seatsrows_test.go b/cmd/codeaf/seatsrows_test.go deleted file mode 100644 index cdcfbdc440..0000000000 --- a/cmd/codeaf/seatsrows_test.go +++ /dev/null @@ -1,409 +0,0 @@ -package main - -import ( - "encoding/json" - "net/http" - "net/http/httptest" - "os" - "strings" - "sync" - "testing" - "time" - - "github.com/Agent-Field/codeaf/internal/catalog" - "github.com/Agent-Field/codeaf/internal/config" -) - -// THE HEADLESS DOORS WAIT FOR THE ROWS A PICK NEEDS. -// -// A pick taken off the table, and a tier row that says auto, are answered from -// the rows the process holds (config.AutoModels) — and before the doors waited, -// every cold start read that seam while the warm was still in flight, so the -// seat fell to the family's table row on a machine whose catalog was sitting in -// its own cache file. The chat surface never met the defect because its picks -// happen after the warm has landed. These tests run the doors' own two calls — -// useAutoSeats and config.ResolveSeats, the same sequence `codeaf exec` and -// `codeaf do` climb — against a profile whose catalog is cached, absent and -// unpickable, and read the rung word off the receipt either way. - -// seatPickRows is the small catalog a test cache holds: three priced rows, -// spread across the bill and across three vendors — a crew's worker and its -// careful seat may not share a vendor, and a candidate below 80% of the seat's -// best falls off its shortlist — so the rows sit close in quality, far apart in -// price, and each able to sit every seat (a window of 200k, tools, images). -// None of them is a family's table id. -func seatPickRows() []catalog.Model { - return []catalog.Model{ - {ID: "north/penny", Name: "Penny", ContextLength: 262144, - PromptPrice: 0.00001, CompletionPrice: 0.00002, CacheReadPrice: 0.000001, - IntelligenceIndex: 85, CodingIndex: 85, AgenticIndex: 85, - InputModalities: []string{"text", "image"}, OutputModalities: []string{"text"}, Parameters: []string{"tools"}}, - {ID: "south/pound", Name: "Pound", ContextLength: 262144, - PromptPrice: 0.002, CompletionPrice: 0.004, CacheReadPrice: 0.0002, - IntelligenceIndex: 86, CodingIndex: 86, AgenticIndex: 86, - InputModalities: []string{"text", "image"}, OutputModalities: []string{"text"}, Parameters: []string{"tools"}}, - {ID: "east/crown", Name: "Crown", ContextLength: 262144, - PromptPrice: 0.05, CompletionPrice: 0.1, CacheReadPrice: 0.005, - IntelligenceIndex: 100, CodingIndex: 100, AgenticIndex: 100, - InputModalities: []string{"text", "image"}, OutputModalities: []string{"text"}, Parameters: []string{"tools"}}, - } -} - -// blockedCatalogServer is a stand-in base whose /models fetch never returns: -// the cold cache behind a slow network, reduced to the one fact that matters — -// the warm does not land within any bound a test is willing to spend. The -// handler is let go exactly once, because the server refuses to close behind a -// wedged request. -func blockedCatalogServer(t *testing.T) string { - t.Helper() - release := make(chan struct{}) - var once sync.Once - letGo := func() { once.Do(func() { close(release) }) } - server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - <-release - })) - t.Cleanup(server.Close) - t.Cleanup(letGo) - return server.URL -} - -// seatTheTestCatalog replaces the process's memoised catalog with a fresh one -// and restores the seams it seats when the test ends. The once is inside the -// accessor, so tests in one process would otherwise share a catalog pointed at -// whichever of them called first — which is the memoising this file is not -// about. -func seatTheTestCatalog(t *testing.T) { - t.Helper() - // The pool's refresh reaches the relay over the network and writes its - // own cache into the profile as it lands — exactly the late write a - // TempDir cleanup races. These tests read no measurements, so the pool is - // off for them: the same off nothing a row saying auto can read, which is - // the answer the seam already holds when there is no index at all. - t.Setenv("CODEAF_MODEL_POOL", "off") - previous := sharedCatalog - models, index, ownCells := config.AutoModels, config.AutoIndex, config.AutoOwnCells - sharedCatalog = newSharedCatalog() - t.Cleanup(func() { - sharedCatalog = previous - config.AutoModels, config.AutoIndex, config.AutoOwnCells = models, index, ownCells - }) -} - -// boundTheSeatWait shortens [autoSeatRowsBound] for one test and restores it, -// so the bound can be watched without spending the three seconds it carries. -func boundTheSeatWait(t *testing.T, bound time.Duration) { - t.Helper() - previous := autoSeatRowsBound - autoSeatRowsBound = bound - t.Cleanup(func() { autoSeatRowsBound = previous }) -} - -// cachedRowHeld fails the test unless the seat resolved to one of the cached -// rows — the brief's "a model from the catalog rows, not the table row". -func cachedRowHeld(t *testing.T, modelID string) { - t.Helper() - for _, row := range seatPickRows() { - if row.ID == modelID { - return - } - } - t.Fatalf("the seat resolved %q, which is not one of the cached rows", modelID) -} - -// A pick off the table is computed from the cached rows, and the receipt says -// which rung answered: `computed from the catalog` under the catalog word, -// `learned` under learn. Both doors resolve through this exact pair of calls, -// and `codeaf exec` prints the work seat's line as its whole report. -func TestADoorWithAPickWaitsForTheCachedRowsAndSaysTheRung(t *testing.T) { - for _, pick := range []string{config.CrewPickCatalog, config.CrewPickLearn} { - t.Run(pick, func(t *testing.T) { - dir := t.TempDir() - if err := catalog.Remember(catalog.Options{Dir: dir}, seatPickRows()); err != nil { - t.Fatal(err) - } - if err := config.SetCrewPick(dir, pick); err != nil { - t.Fatal(err) - } - seatTheTestCatalog(t) - t.Setenv(config.ModelEnv, "") - t.Setenv(config.PlanModelEnv, "") - - useAutoSeats(config.Config{ProfileDir: dir}) - seats := config.ResolveSeats(dir, "", "") - - want := "crew balanced, computed from the catalog" - if pick == config.CrewPickLearn { - want = "crew balanced, learned" - } - for _, seat := range []config.Seat{seats.Work, seats.Plan} { - if seat.Rung() != want { - t.Fatalf("the %s seat reads %q (%s), want the %q rung", seat.Role, seat.Model, seat.Rung(), want) - } - cachedRowHeld(t, seat.Model) - } - // And not the family's table row: [config.TierModelAt] climbs the same - // ladder and would answer the computed id here too, so the table is read - // straight off the family. - table, ok := config.CrewModelsForSource(config.CrewSourceAt(dir), config.DefaultCrew) - if !ok || seats.Work.Model == table[config.ModelTierWorker] { - t.Fatalf("the work seat fell to the family's table row %q over a catalog it holds", table[config.ModelTierWorker]) - } - // And the one line the exec door prints names the rung too. - if !strings.Contains(seats.Work.Report(), want) { - t.Fatalf("the work seat's report does not name the rung:\n%s", seats.Work.Report()) - } - }) - } -} - -// A tier row that says auto is the other way a seat needs the rows, and the -// door waits for them with no pick row in the profile at all. -func TestADoorWithAnAutoRowWaitsForTheCachedRows(t *testing.T) { - dir := t.TempDir() - if err := catalog.Remember(catalog.Options{Dir: dir}, seatPickRows()); err != nil { - t.Fatal(err) - } - rows, err := json.Marshal(map[string]string{config.KeyTierWorkerModel: config.AutoValue}) - if err != nil { - t.Fatal(err) - } - if err := os.WriteFile(config.BudgetConfigPath(dir), rows, 0o600); err != nil { - t.Fatal(err) - } - seatTheTestCatalog(t) - t.Setenv(config.ModelEnv, "") - t.Setenv(config.PlanModelEnv, "") - - useAutoSeats(config.Config{ProfileDir: dir}) - seats := config.ResolveSeats(dir, "", "") - - if seats.Work.Rung() != "crew balanced, computed from the catalog" { - t.Fatalf("an auto row resolved to %q (%s), want the computed rung", seats.Work.Model, seats.Work.Rung()) - } - cachedRowHeld(t, seats.Work.Model) - table, ok := config.CrewModelsForSource(config.CrewSourceAt(dir), config.DefaultCrew) - if !ok || seats.Work.Model == table[config.ModelTierWorker] { - t.Fatalf("the auto row fell to the family's table row %q over a catalog it holds", table[config.ModelTierWorker]) - } -} - -// A pick left at the table reads no rows at all, and waits for nothing — not -// even for a warm this test never lets land. The bound is raised to five -// seconds so that a wrongly added wait cannot pass by timing out quietly -// first: the door must be back in under a second. -func TestADoorAtTheTablePickWaitsForNothing(t *testing.T) { - url := blockedCatalogServer(t) - dir := t.TempDir() - if err := config.ApplyCrew(dir, config.CrewBalanced); err != nil { - t.Fatal(err) - } - seatTheTestCatalog(t) - boundTheSeatWait(t, 5*time.Second) - t.Setenv(config.ModelEnv, "") - t.Setenv(config.PlanModelEnv, "") - - started := time.Now() - useAutoSeats(config.Config{ProfileDir: dir, BaseURL: url}) - if waited := time.Since(started); waited > time.Second { - t.Fatalf("a door with no pick and no auto row waited %s for rows it does not read", waited) - } - seats := config.ResolveSeats(dir, "", "") - for _, seat := range []config.Seat{seats.Work, seats.Plan} { - if seat.Rung() != "crew balanced" { - t.Fatalf("the %s seat reads %q, want the crew's own table row", seat.Role, seat.Rung()) - } - } - if seats.Work.Model != config.TierModelAt(dir, config.ModelTierWorker) { - t.Fatalf("the work seat read %q, want the table row %q", seats.Work.Model, config.TierModelAt(dir, config.ModelTierWorker)) - } -} - -// A cold cache behind a network that never answers falls to the table row -// within the bound, and the receipt says so: `table` is a rung word on purpose, -// so a run that fell says it fell. -func TestADoorOnAColdCacheFallsToTheTableRowAndSaysSo(t *testing.T) { - url := blockedCatalogServer(t) - dir := t.TempDir() - if err := config.SetCrewPick(dir, config.CrewPickCatalog); err != nil { - t.Fatal(err) - } - seatTheTestCatalog(t) - boundTheSeatWait(t, 250*time.Millisecond) - t.Setenv(config.ModelEnv, "") - t.Setenv(config.PlanModelEnv, "") - - started := time.Now() - useAutoSeats(config.Config{ProfileDir: dir, BaseURL: url}) - seats := config.ResolveSeats(dir, "", "") - if waited := time.Since(started); waited > 2*time.Second { - t.Fatalf("the door waited %s, past the bound it was given", waited) - } - if !strings.Contains(seats.Report(), "(crew balanced, table)") { - t.Fatalf("a run that fell to the table row must say so:\n%s", seats.Report()) - } - if seats.Work.Model != config.TierModelAt(dir, config.ModelTierWorker) { - t.Fatalf("the work seat read %q, want the table row %q", seats.Work.Model, config.TierModelAt(dir, config.ModelTierWorker)) - } -} - -// A KEY FROM THE ENVIRONMENT IS THE KEY THE SEAT IS RESOLVED WITH. The -// provider key is supplied through the environment, never written into the -// profile, and the seat must still be computed off the cached rows: the run's -// catalog is the same one every door shares, and a key the environment holds -// is a key [config.LoadKeyless] holds too. The seat names the `learned` rung -// under the learn pick, and it is the id the run actually sits somebody in. -func TestAnErrandWhoseKeyComesFromTheEnvironmentNamesTheLearnedRung(t *testing.T) { - script := newScriptedBrain(t) - defer script.close() - t.Setenv(config.ModelEnv, "") - t.Setenv(config.PlanModelEnv, "") - // The key is in the ENVIRONMENT and nowhere in the profile — which is the - // one arrangement the seat has to survive. - t.Setenv(config.APIKeyEnv, "sk-env-only") - if err := config.SetCrewPick(script.dir, config.CrewPickLearn); err != nil { - t.Fatal(err) - } - if err := catalog.Remember(catalog.Options{Dir: script.dir, BaseURL: script.server.URL}, seatPickRows()); err != nil { - t.Fatal(err) - } - seatTheTestCatalog(t) - - var mu sync.Mutex - var built []string - var stdout, stderr strings.Builder - err := doErrand(doRequest{ - task: "write the release note and include the migration steps", - timeout: 60 * time.Second, - asJSON: true, - stdout: &stdout, - stderr: &stderr, - newClient: func(settings config.Config, model string) (*liveClient, error) { - mu.Lock() - built = append(built, model) - mu.Unlock() - return script.client(settings, model) - }, - }) - if err != nil { - t.Fatalf("the errand did not settle cleanly: %v\nstderr:\n%s", err, stderr.String()) - } - - if !strings.Contains(stderr.String(), "(crew balanced, learned)") { - t.Fatalf("the opening lines never named the learned rung:\n%s", stderr.String()) - } - var outcome headlessOutcome - if err := json.Unmarshal([]byte(stdout.String()), &outcome); err != nil { - t.Fatalf("--json did not print one object: %v\n%s", err, stdout.String()) - } - if outcome.ModelSource != "crew balanced, learned" { - t.Fatalf("--json named the rung %q, want the learned rung", outcome.ModelSource) - } - cachedRowHeld(t, outcome.Model) - - mu.Lock() - models := append([]string(nil), built...) - mu.Unlock() - for _, model := range models { - if model == outcome.Model { - return - } - } - t.Fatalf("no client was built on the learned seat %q; the run used %v", outcome.Model, models) -} - -// AND WITH NO KEY ANYWHERE. The seat line is part of the door's promise: it is -// printed before the run reaches a provider, so a profile with no key still -// says which models it meant to use — and the seats still come off the cached -// rows, which need no key to read. The keyless read is the second rung here, -// under the run's own settings, so the ordering this pins is exactly the one -// the seating picked. -func TestAnErrandWithNoKeySeatsTheCachedRowsThenSaysTheKeyIsMissing(t *testing.T) { - script := newScriptedBrain(t) - defer script.close() - t.Setenv(config.ModelEnv, "") - t.Setenv(config.PlanModelEnv, "") - t.Setenv(config.APIKeyEnv, "") - t.Setenv("OPENAI_API_KEY", "") - if err := config.SetCrewPick(script.dir, config.CrewPickLearn); err != nil { - t.Fatal(err) - } - if err := catalog.Remember(catalog.Options{Dir: script.dir, BaseURL: script.server.URL}, seatPickRows()); err != nil { - t.Fatal(err) - } - seatTheTestCatalog(t) - - var stdout, stderr strings.Builder - err := doErrand(doRequest{ - task: "write the release note and include the migration steps", - timeout: 60 * time.Second, - stdout: &stdout, - stderr: &stderr, - }) - if err == nil || !strings.Contains(err.Error(), config.APIKeyEnv) { - t.Fatalf("a keyless errand must end on the missing-key sentence, got %v", err) - } - if !strings.Contains(stderr.String(), "(crew balanced, learned)") { - t.Fatalf("the seat line must be printed before the key is missed:\n%s", stderr.String()) - } -} - -// THE ERRAND'S RECEIPT IS THE PART A HARNESS READS. A profile with the pick -// off the table and a catalog cached beside it runs on the computed ids, and -// both places the run names its rungs — the opening stderr lines and the -// --json object — say `computed from the catalog`. -func TestAnErrandNamesTheRungTheCatalogAnswered(t *testing.T) { - script := newScriptedBrain(t) - defer script.close() - t.Setenv(config.ModelEnv, "") - t.Setenv(config.PlanModelEnv, "") - if err := config.SetCrewPick(script.dir, config.CrewPickCatalog); err != nil { - t.Fatal(err) - } - if err := catalog.Remember(catalog.Options{Dir: script.dir, BaseURL: script.server.URL}, seatPickRows()); err != nil { - t.Fatal(err) - } - seatTheTestCatalog(t) - - var mu sync.Mutex - var built []string - var stdout, stderr strings.Builder - err := doErrand(doRequest{ - task: "write the release note and include the migration steps", - timeout: 60 * time.Second, - asJSON: true, - stdout: &stdout, - stderr: &stderr, - newClient: func(settings config.Config, model string) (*liveClient, error) { - mu.Lock() - built = append(built, model) - mu.Unlock() - return script.client(settings, model) - }, - }) - if err != nil { - t.Fatalf("the errand did not settle cleanly: %v\nstderr:\n%s", err, stderr.String()) - } - - if !strings.Contains(stderr.String(), "(crew balanced, computed from the catalog)") { - t.Fatalf("the opening lines never named the computed rung:\n%s", stderr.String()) - } - var outcome headlessOutcome - if err := json.Unmarshal([]byte(stdout.String()), &outcome); err != nil { - t.Fatalf("--json did not print one object: %v\n%s", err, stdout.String()) - } - if outcome.ModelSource != "crew balanced, computed from the catalog" { - t.Fatalf("--json named the rung %q, want the computed rung", outcome.ModelSource) - } - cachedRowHeld(t, outcome.Model) - - // And the computed id is the one the run actually sat somebody in. - mu.Lock() - models := append([]string(nil), built...) - mu.Unlock() - for _, model := range models { - if model == outcome.Model { - return - } - } - t.Fatalf("no client was built on the computed seat %q; the run used %v", outcome.Model, models) -} diff --git a/cmd/codeaf/subharness.go b/cmd/codeaf/subharness.go index 3c6db1b44b..eff94bcc56 100644 --- a/cmd/codeaf/subharness.go +++ b/cmd/codeaf/subharness.go @@ -441,52 +441,41 @@ func newSharedCatalog() func(config.Config) *catalog.Catalog { resolved = catalog.LoadLazy(context.Background(), catalog.Options{ BaseURL: settings.BaseURL, APIKey: settings.APIKey, Dir: settings.ProfileDir, }) - // A tier row that says auto is answered from this catalog (config.AutoModels): - // the same non-blocking read, never a fetch, and set once at start-up so every - // headless door resolves the word against the list it already holds. - config.AutoModels = resolved.ModelsNow - // and the pool's index beside it, in the same one-time manner: a tier - // row that says auto is answered from the index this process was seated - // with, the seed when no cache is fresher, and the one fetch it makes - // runs in the background and never blocks this read. + // The crew router picks its seats from this catalog + // (config.CrewCatalog): the same non-blocking read, never a fetch, + // and set once at start-up so every headless door routes against + // the list it already holds. + seatCrewRows(resolved.ModelsNow) + // and the pool's errands beside it, in the same one-time manner. wirePoolIndex(settings.ProfileDir) }) return resolved } } +// seatCrewRows hands the router the catalog's rows. It is a variable so a +// test that seats a catalog of its own is not overwritten by the first door +// that warms the shared one. +var seatCrewRows = func(rows func() []catalog.Model) { config.CrewCatalog = rows } + // autoSeatRowsBound is how long a headless door waits for the catalog's rows -// when the profile's pick or a tier row needs them. It is sized to cover the -// disk read of a cached catalog and nothing more. It is a variable because the -// test of the bound must not spend three seconds proving the bound is honoured. +// before it routes its crew. It is sized to cover the disk read of a cached +// catalog and nothing more. It is a variable because the test of the bound must +// not spend three seconds proving the bound is honoured. var autoSeatRowsBound = 3 * time.Second -// useAutoSeats seats this process's catalog under the seat ladder, and is what +// useAutoSeats seats this process's catalog under the crew router, and is what // a headless door calls BEFORE it resolves its seats. // -// The order is the whole of it. A tier row that says `auto` is answered from -// the rows already in hand ([config.AutoModels]), and every headless door -// climbed the ladder before it asked for a catalog at all — so the word read -// against nothing and landed on the family's table row on every run, on a -// machine whose catalog was sitting in its own cache file. It is the same lazy, -// memoised catalog every one of those doors goes on to use; asking for it a few -// lines earlier waits for nothing. -// -// AND WHEN THE ANSWER NEEDS THE ROWS, THE DOOR WAITS FOR THEM, within -// [autoSeatRowsBound]: a pick taken off the table ([config.CrewPickAt]) and a -// tier row that says auto ([config.AnyTierAutoAt]) are both computed from those -// rows, and the chat surface never met the defect because its picks happen after -// the warm has landed. The bound is a bound on the wait, not on the fetch — when -// it runs out the warm carries on in the background, the resolver falls to the -// family's table row exactly as it did before, and the seat's receipt names the -// rung that answered (`table`), so a run that fell says it fell. A profile with -// neither a pick nor an auto row reads no rows at all, and waits for nothing, -// the way it always has. +// THE ORDER IS THE WHOLE OF IT. Every seat nobody pinned is routed per task +// from the catalog's rows ([config.CrewCatalog]), and a door that climbed the +// ladder before it asked for a catalog would route from nothing. So the door +// waits for the rows, within [autoSeatRowsBound]: the bound is a bound on the +// wait, not on the fetch — when it runs out the warm carries on in the +// background and the router works from the models it can still price (a pin, a +// flag), and says so in the error it answers when it cannot seat a seat. func useAutoSeats(settings config.Config) { resolved := sharedCatalog(settings) - if config.CrewPickAt(settings.ProfileDir) == config.CrewPickTable && !config.AnyTierAutoAt(settings.ProfileDir) { - return - } ctx, cancel := context.WithTimeout(context.Background(), autoSeatRowsBound) defer cancel() resolved.Warmed(ctx) diff --git a/cmd/codeaf/subharness_run.go b/cmd/codeaf/subharness_run.go index d0f332a326..e614b9eac6 100644 --- a/cmd/codeaf/subharness_run.go +++ b/cmd/codeaf/subharness_run.go @@ -118,7 +118,7 @@ func runSubharnessCommand(args []string) error { // same ladder every other headless door climbs, so a profile's crew reaches // this one too (config.ResolveSeats). useAutoSeats(settings) - seats := config.ResolveSeats(settings.ProfileDir, *model, "") + seats := doorSeats(settings, *model, "", "") applySeats(&settings, seats) fmt.Fprintln(os.Stderr, seats.Work.Report()) // The measured ruler is seated for the same reason `run` seats it: this is a diff --git a/cmd/codeaf/taskcap_test.go b/cmd/codeaf/taskcap_test.go new file mode 100644 index 0000000000..f08aba2fa9 --- /dev/null +++ b/cmd/codeaf/taskcap_test.go @@ -0,0 +1,60 @@ +package main + +import ( + "context" + "errors" + "testing" + + "github.com/Agent-Field/agentfield/sdk/go/ai" + + "github.com/Agent-Field/codeaf/internal/config" + "github.com/Agent-Field/codeaf/internal/crewroute" + "github.com/Agent-Field/codeaf/internal/session" +) + +// costlyCompleter answers every call at a fixed provider-reported cost. +type costlyCompleter struct { + usd float64 + calls int +} + +func (c *costlyCompleter) CompleteWithMessages(context.Context, []ai.Message, ...ai.Option) (*ai.Response, error) { + c.calls++ + cost := c.usd + return &ai.Response{Choices: []ai.Choice{{Message: ai.Message{Role: "assistant"}}}, + Usage: &ai.Usage{PromptTokens: 100, CompletionTokens: 10, Cost: &cost}}, nil +} + +// -YES-SPEND DOES NOT LIFT THE PER-TASK LIMIT: the guard a `do` run is held +// to carries it with and without the flag, with and without a routed crew, +// and a run under the flag stops at it. +func TestYesSpendKeepsThePerTaskLimit(t *testing.T) { + dir := t.TempDir() + crew := &crewroute.Decision{} + for _, c := range []struct { + crew *crewroute.Decision + preauthorized bool + }{{crew, true}, {crew, false}, {nil, true}, {nil, false}} { + guard := doSpendGuard(dir, c.crew, c.preauthorized) + if guard.TaskCap != 5 || guard.TaskAction != config.CrewTaskCapAction(5) { + t.Fatalf("crew=%v yes-spend=%v: per-task limit %v, %q", c.crew != nil, c.preauthorized, guard.TaskCap, guard.TaskAction) + } + } + + guard := doSpendGuard(dir, crew, true) + guard.Price = func(string) (float64, float64, float64, bool) { return 1e-6, 1e-5, 0, true } + guard.Day = session.NewSpendDay(0) + seat := &costlyCompleter{usd: 2} + wrapped := guard.Wrap("vendor/worker", seat) + var stopped session.ErrSpendStopped + var err error + for i := 0; i < 10 && err == nil; i++ { + _, err = wrapped.CompleteWithMessages(t.Context(), []ai.Message{{Role: "user"}}) + } + if !errors.As(err, &stopped) || stopped.Action != "this task reached its $5 limit · raise it in /crew" { + t.Fatalf("a run under -yes-spend ended on %v", err) + } + if seat.calls != 2 { + t.Fatalf("%d calls were made under a $5 limit at $2 a call, want 2", seat.calls) + } +} diff --git a/docs/DESIGN-LANGUAGE.md b/docs/DESIGN-LANGUAGE.md index 4660e1f8b0..4c71ce1b94 100644 --- a/docs/DESIGN-LANGUAGE.md +++ b/docs/DESIGN-LANGUAGE.md @@ -386,10 +386,10 @@ opposite ends. This surface says a great deal on its own account — a note, a hint, a legend, an announcement — and every word of it was written in the reading tiers, because none of it is the conversation. That is right about the LINE and it was wrong -about what the line is for. `crew → balanced · brain kimi-k3:low · hands -deepseek-v4-flash · checks qwen3.8-27b` was one flat dim run from end to end: -the words a person already knew, and the four model ids they typed the command -to learn, at exactly the same weight. The sentence was legible and the ANSWER +about what the line is for. `checker ⌖ moonshotai/kimi-k3 · every task until +you unpin it · you are still talking to deepseek-v4-flash` was one flat dim run +from end to end: the words a person already knew, and the two model ids they +typed the command to learn, at exactly the same weight. The sentence was legible and the ANSWER inside it was not. So the prose of an informational line stays where it is, and **each load-bearing @@ -441,8 +441,8 @@ mentioned would have spent the budget forty times a minute. a line is bright then nothing in it is, and a rule that lifted every noun would have bought back the flat line it started from. So the hint slot lifts the key and never the verb beside it; `/status` lifts the figure and never its label; -the crew line lifts the three ids and leaves `crew →`, the preset word the -person has just typed, and the three role words where they were — those are the +the `/crew` panel lifts the model ids and leaves the seat words, `auto · now` +and the words the person has just typed where they were — those are the question, and the ids are the answer. **A datum is named, never guessed.** `internal/tui3/payload.go` holds the whole diff --git a/docs/GUIDE.md b/docs/GUIDE.md index 91f927d48d..5664b0e263 100644 --- a/docs/GUIDE.md +++ b/docs/GUIDE.md @@ -303,7 +303,7 @@ sign in once in your browser. openrouter makes the default service's key for thi Where it is not, the same page is headed `your openrouter key` and reads `codeaf talks to models on its default service through openrouter, on your key and your card. nothing is sent until you do.` Either way the foot takes a pasted key and `esc` skips setup. -The second page is `Daily limit`, `Chat model` and `Work crew`. +The second page is `Daily limit` and `Chat model`. @@ -320,7 +320,7 @@ object, while a pinned call asks for exactly that lane. `latency` and `price` re opt-in settings. The default daily rail is `$500`; setting that row to `0` removes it. First run asks for -`Daily limit`, `Chat model`, and `Work crew`. +`Daily limit` and `Chat model`; a task's crew is picked per task, and `/crew` shows it. State lives under `$CODEAF_HOME`, or `~/.codeaf` when it is unset or empty: settings and credentials in `config.json`, memory in `graph.db`, and project sessions under `v3/projects/`. diff --git a/docs/HEADLESS.md b/docs/HEADLESS.md index 3b10d0c88e..2c4f55acc3 100644 --- a/docs/HEADLESS.md +++ b/docs/HEADLESS.md @@ -17,6 +17,7 @@ schema, and a person needs to know which command actually thinks. ``` codeaf do "" [--dir dir] [--db path] [--keep] [--timeout D] [--json] [--yes-spend] [--model slug] [--plan-model slug] + [--check-model slug] [--best|--cheap] [--pin seat=model[@provider]] [--context-fill N] [--completion-reserve N] ``` @@ -40,9 +41,13 @@ work done. | `--keep` | off | Keep the private store instead of deleting it; the path is printed to stderr. | | `--timeout D` | `15m` | Hard wall, as a duration with a unit: `5m`, `2h`, `90s`. A bare number is still read as seconds for one release, so `--timeout 900` keeps working. A wall, not a schedule — the length of rope at which a wedged run is more useful dead. | | `--json` | off | Print one machine-readable object instead of the prose deliverable. | -| `--yes-spend` | off | Spend past today's limit and past the plan-price question, without stopping to ask. The same flag with the same one sentence on `codeaf plan run`. Equivalent to `CODEAF_PREAUTHORIZE_SPEND=1`. | -| `--model slug` | the ladder below | The work model for this run. | -| `--plan-model slug` | the ladder below | Model that plans, replans, writes contracts, and runs the delivery gate, when it should differ from the model executing leaves. | +| `--yes-spend` | off | Spend past today's limit and past the plan-price question, without stopping to ask. The same flag with the same one sentence on `codeaf plan run`. Equivalent to `CODEAF_PREAUTHORIZE_SPEND=1`. It does not lift the per-task limit (`/crew cap task`, $5 unless set). | +| `--model slug` | the ladder below | The worker for this run — a one-task pin. | +| `--plan-model slug` | the ladder below | The planner for this run — a one-task pin: plans, replans, writes contracts, and runs the delivery gate. | +| `--check-model slug` | the ladder below | The checker for this run — a one-task pin. It never inherits the planner. | +| `--best` | off | Run this task on the strongest crew your allowed models make. | +| `--cheap` | off | Run this task on the cheapest crew your allowed models make. | +| `--pin seat=model[@provider]` | none | Pin one seat for this run only — `worker=`, `planner=` or `checker=`. Repeatable, once per seat. | | `--context-fill N` | `60` | How full a model's context window may get before it is compacted, in percent; the law clamps it to 10–90. Setting it is what makes it govern a conversation's fold line as well — unset, that line follows the model's window. | | `--completion-reserve N` | `65536` | Tokens every call keeps free for its visible answer *and its reasoning*. Raise it for a reasoning-heavy model that truncates; lower it to buy prompt room on a small window. | @@ -118,47 +123,66 @@ question was not the compiler's to answer at all. The corollary for a harness: put the answer in the ask. Anything you leave implicit is something `do` will decide for you and tell you it decided. -### Which models a run uses — one ladder, four rungs +### Which models a run uses — three seats, one ladder each -The two seats — the model that **works** and the model that **plans** — resolve -the same way at every headless door (`do`, `exec`, `run`, `plan new`, -`plan revise`, `plan run`). First rung that answers wins, per seat: +A run has a crew of three seats — the **worker** that does the work, the +**planner** that plans it and the **checker** that reads the result — and every +headless door (`do`, `exec`, `run`, `plan new`, `plan revise`, `plan run`) +resolves them the same way. First rung that answers wins, per seat: -| | work seat | plan seat | -| --- | --- | --- | -| 1 | `--model slug` | `--plan-model slug` | -| 2 | `CODEAF_MODEL` | `CODEAF_PLAN_MODEL` | -| 3 | the profile's crew — the **small work** row | the profile's crew — the **mastermind** row | -| 4 | the build's default (`codeaf --help`) | empty: the work model plans too | - -**Rung 3 is what `/crew` writes** (`models.tiers.*` in the profile's -`config.json`), and it is the rung that used to be missing: until #166 a headless -run read the flags and the environment and never opened the profile, so a machine -told `frugal` in the chat ran something else the moment the same brain ran -headless. The two rows are the ones the chat's own planner and worker ride, so -the crew now means the same thing on both surfaces. - -The crew answers only where a crew was actually **written**. A profile nobody has -touched falls to rung 4 — the four shipped tier values are the `balanced` row, so -reading them as a crew would make rung 4 unreachable and change the default work -model for everybody. `CODEAF_HOME` / `CODEAF_PROFILE_DIR` decide which profile is +| | worker | planner | checker | +| --- | --- | --- | --- | +| 1 | `--model slug` | `--plan-model slug` | `--check-model slug` | +| 2 | `CODEAF_MODEL` | `CODEAF_PLAN_MODEL` | `CODEAF_CHECK_MODEL` | +| 3 | a `/crew pin` in the profile | a `/crew pin` | a `/crew pin` | +| 4 | routed for this task | routed for this task | routed for this task | + +**Rungs 1 and 2 are one-task pins.** They are handed to the router as pins for +this run, so the crew line, the estimate and the logged decision describe the +crew that actually ran. **The checker never inherits the planner**: a pinned +planner says something about planning and nothing about who grades the work. +`codeaf do --pin checker=moonshotai/kimi-k3@openrouter` is the same pin spelled +per seat, with the route after `@`. + +**Rung 3 is what `/crew pin` writes** in the chat, so a checker pinned there is +the checker here. `CODEAF_HOME` / `CODEAF_PROFILE_DIR` decide which profile is asked, so an isolated run is isolated here too. -A crew row may carry a thinking level (`moonshotai/kimi-k3:low`), and so may a -flag or a variable. The value travels whole and the level is applied per call by -the role ladder, exactly as it is in the chat; the slug sent to the provider is -the model alone. (Until this landed it was sent whole, so `--plan-model -kimi-k3:low` asked OpenRouter for a model id nobody publishes.) +**Rung 4 is the router.** Every seat nothing named is picked for this task: the +task text is read as a class of work (`bugfix`, `openended` or `other`), and each +seat gets the model that serves that class best for its cost among the models +the profile allows (`/crew models` — `all` when nothing was written). `--best` +and `--cheap` move one run to the strongest or the cheapest crew allowed. + +A seat's value may carry a thinking level (`moonshotai/kimi-k3:low`). The value +travels whole and the level is applied per call by the role ladder, exactly as +it is in the chat; the slug sent to the provider is the model alone. -**Every run says which rung answered**, on stderr, before anything else: +**Every run says which rung answered**, on stderr, before anything else, and +under it the crew line — the class, the worker and its route, the checker (📌 +on a pinned seat) and the estimate: ``` -models: work z-ai/glm-5.3-flash (crew frugal) · plan z-ai/glm-5.3-flash (crew frugal) -models: work anthropic/whatever (--model) · plan follows the work model (default) +models: worker z-ai/glm-5.3-flash (routed) · planner z-ai/glm-5.3-flash (routed) · checker moonshotai/kimi-k3 (pinned) +crew: bugfix · worker glm-5.3-flash (openrouter) · checker 📌 kimi-k3 · est $0.023 ``` +When the run ends the `crew:` line is said again with the actual beside the +estimate — `$0.021 (est $0.023)`. + +**The daily cap.** `/crew cap` sets what crews may spend in a day. At the cap +`codeaf do` refuses before spending anything — `today's crew spend has reached +the daily cap of $5.00 · raise it with `/crew cap`, run with --cheap, or pass +-yes-spend` — and `--yes-spend` is the one way past. The other doors warn on +stderr and go on. + +**The per-task limit.** Every run is held to the most one task may spend, $5 +unless `/crew cap task` set another. A call that would take the run past it is +not made, and the run stops on `this task reached its $5 limit · raise it in +/crew`. `--yes-spend` does not lift it. + so a campaign can verify what actually ran instead of trusting the shell it -launched from. `do --json` carries the same four facts as fields. +launched from. `do --json` carries the same facts as fields. ### Exit codes — one ladder, and it is the same one on all three commands @@ -269,13 +293,19 @@ there now. Test its value, or read `ok`. Fields that belong to `do` and stay: `spend_work` and `spend_overhead` — what the work cost against what it cost to decide what the work should be — `blocked_on`, `learned`, `plan_model`, `model_source`, `plan_model_source`, +`check_model`, `check_model_source`, `class`, `crew`, `est_usd`, `effort`, `subharness` and `workspace`. | Field | Contract | | --- | --- | | `blocked_on` | The question it could not answer, verbatim. Non-empty **only** alongside a non-zero exit and an empty `answer`. | | `learned` | The job's blackboard: discoveries, pitfalls, a sibling's failure and why. On an ephemeral store this is the only piece of what the run understood that would otherwise die with it — capture it if you care about the run's reasoning. | -| `model_source` / `plan_model_source` | Which rung of the ladder above chose each: `--model`, `CODEAF_MODEL`, `crew frugal`, `default`. Pin these in a campaign's records — they are the only way to tell two cells apart that were launched from different profiles. | +| `model_source` / `plan_model_source` / `check_model_source` | Which rung of the ladder above chose each seat: the flag (`--model`, `--plan-model`, `--check-model`), the variable (`CODEAF_MODEL`, …), `pinned` or `routed`. Pin these in a campaign's records — they are the only way to tell two cells apart that were launched from different profiles. | +| `check_model` | The checker's model. Present whenever a checker was seated. | +| `class` | The class of work the task was read as: `bugfix`, `openended` or `other`. Present on a routed run. | +| `crew` | Each seat — `worker`, `planner`, `checker` — as `{model, provider, kind, pinned, est_usd}`; `kind` is how the route bills: `metered`, `plan` or `local`. | +| `est_usd` | The crew's estimate for the task, beside `spend_usd`, which is the actual. | +| `effort` | `best` or `cheap` when `--best` or `--cheap` was given; absent otherwise. | | `workspace` | The directory the run worked in, absolute, edited in place. Always present; empty on a run that never opened one or was handed to an existing resident whose workspace this invocation cannot establish. | ### Stream discipline @@ -739,8 +769,9 @@ The full list is `codeaf help env`. What matters headless: | Variable | Default | Why a harness cares | | --- | --- | --- | | `OPENROUTER_API_KEY` | — | Required. | -| `CODEAF_MODEL` | see `--help` | The work model. `--model` overrides per run; unset, the profile's crew answers before the built-in default — see the ladder in section 1. | -| `CODEAF_PLAN_MODEL` | unset | Plans, replans, contracts, and the gate on a stronger model while a smaller one executes leaves. Unset, the profile's crew mastermind answers; with no crew written, the work model plans too. | +| `CODEAF_MODEL` | unset | Pins the worker for every run it is set on. `--model` overrides per run; unset, a `/crew pin` or the router answers — see the ladder in section 1. | +| `CODEAF_PLAN_MODEL` | unset | Pins the planner: plans, replans, contracts and the delivery gate. Unset, a `/crew pin` or the router answers. | +| `CODEAF_CHECK_MODEL` | unset | Pins the checker. Unset, a `/crew pin` or the router answers — never the planner. | | `CODEAF_MODELS` | unset | A panel instead of one model: calls cascade cheapest-first and escalate when a verifier catches a failure. Comma-separated slugs or a JSON path. **Changes what a run costs and how it fails — pin it when measuring.** | | `CODEAF_DAILY_BUDGET` | `20.0` | The day's spending limit in dollars; `0` is unlimited. A run that reaches it stops. | | `CODEAF_PREAUTHORIZE_SPEND` | unset | `1` is `--yes-spend` for every run. | @@ -783,15 +814,16 @@ Rules that came from getting them wrong: - **`--model` alone does not pin a chat cell to one model.** The tier rows and role pins answer the auxiliary calls, so a campaign attributing spend and quality to a named model must pass `--one-model` — or measure a profile it - did not record. `do` takes its two seats from the same tier rows when nothing - else names them, so a headless cell is pinned by passing both flags (or both - variables), and `model_source` in the `--json` object says whether they took. + did not record. `do` routes every seat nothing names, so a headless cell is + pinned by passing all three flags (or all three variables, or `--pin` per seat), + and `model_source`, `plan_model_source` and `check_model_source` in the `--json` + object say whether they took. A run whose numbers are compared across the two shapes should say which is which. Verify rather than assume: the `usage` records in the session transcript name the model that actually served each call. - **The profile is part of the measurement.** Two cells run from two profiles - with different crews are two configurations, not one. Record `model_source` - and `plan_model_source` beside the score, or point every cell at one + with different pins or allowed models are two configurations, not one. Record + `model_source`, `plan_model_source`, `check_model_source` and `crew` beside the score, or point every cell at one `CODEAF_PROFILE_DIR`. - **`--timeout` is part of the result.** A cell that hit the wall measured the wall as much as the work. Report the timeout rate beside the score or the diff --git a/docs/LIMITS.md b/docs/LIMITS.md index 1e94d7d591..a246613637 100644 --- a/docs/LIMITS.md +++ b/docs/LIMITS.md @@ -42,6 +42,14 @@ today's spend beside the word: `no limit · $4.25 today`. spell "practice without a bound" — self-origin work runs while nobody is watching, so it is the one pocket that always has a bottom. +**The crew's two limits live on `/crew`, not on this tab.** The **per task** row +here reads the per-task limit and opens nothing new: it is set with +`/crew cap task ` or on the panel's cap row (`per task $5 · daily none`). +It is the second rail where `0` is not "no limit" — a task always has a limit, so +`0` and `none` are refused and an emptied box is $5 again. The crew's **daily +cap** beside it is unset until somebody sets it with `/crew cap `, and +`/crew cap off` takes it away. + ## Every rail in the build | # | Rail | Where the number lives | Was | Now | Unit | What it blocks | `0` = no limit | @@ -57,6 +65,9 @@ watching, so it is the one pocket that always has a bottom. | 9 | workflow ceiling | `internal/craft/types.go` `MaxRunBudgetUSD` | 10 | **500** | USD | the most a workflow file may grant itself | n/a | | 10 | workflow wall | `internal/craft/types.go` `DefaultWallClock` / `MaxWallClock` | 30m / 2h | **6h / 24h** | time | stops opening new rounds; running leaves finish | no | | 11 | discard consent gate | `internal/store/surgery.go` `SurgerySpendGateUSD` | 0.25 | **5** | USD already spent | asks before throwing running work away — raising it asks **less** | n/a | +| 12 | per-task limit | `internal/config/crew.go` `CrewTaskCapDefault` | none | **5** | USD per task | the call that would pass it is not made; the task stops on `this task reached its $5 limit · raise it in /crew`. `-yes-spend` does not lift it | **no — a task always has one** | +| 13 | crew daily cap | `internal/config/crew.go` `CrewCapAt` (`models.crew.cap`) | none | **none** | USD/day of crew spend | a task does not start, and a call that would cross it is not made; `codeaf do` refuses unless `-yes-spend` | yes (`off`) | +| 14 | checker ceiling | `internal/config/crewspend.go` `crewCheckCeilingTimes` / `crewCheckCeilingFloor` | none | **3× the checker's estimate, at least $0.05** | USD per check | the check stops and the task ends unchecked | n/a | ### Rails that already shipped unbounded, and stay that way @@ -71,8 +82,8 @@ watching, so it is the one pocket that always has a bottom. `CODEAF_NODE_BUDGET` (60 nodes) and `--budget` on `codeaf run` / `codeaf exec` (150 000 **tokens**) are counts. A task's own bounds are steps and time — `taskDeadline` 60m renewable four times, `taskMaxSteps` 200, `taskNoProgress` 6 -— and a task carries **no dollar cap of its own**: its money bound is whatever -rail the conversation that started it carries. `costHintUSD` (10¢) is the point +— and a task's dollar cap is the per-task limit set in `/crew` (`models.crew.task_cap`, +$5 by default), inside whatever rail the conversation that started it carries. `costHintUSD` (10¢) is the point at which a first-run tip arms, not a ceiling. ## Where each number is set, in the order it wins @@ -92,6 +103,8 @@ explicit instruction and still errors. | standing per-firing | — | `per_run_usd` on the `stand` tool, per item | | adaptive-run tank | — | the composer's third line, per run | | unattended budget | `CODEAF_MAX_COST` / `CODEAF_MAX_HOURS` | no — flags | +| per-task limit | — | `/crew cap task` (`models.crew.task_cap`) | +| crew daily cap | — | `/crew cap` (`models.crew.cap`) | ## One source of truth diff --git a/docs/TELEMETRY.md b/docs/TELEMETRY.md index a0d1e1a665..3551ea7e52 100644 --- a/docs/TELEMETRY.md +++ b/docs/TELEMETRY.md @@ -110,7 +110,8 @@ The usage counts are not the only thing this binary sends to AgentField. With `model_pool` set to `on` — the default — a judge scores each crew seat after a task lands, and one row per seat leaves for `https://codeaf.agentfield.ai/pool/v1/rows`: the model slug that held the -seat, the judge's slug, the seat (worker, high or mastermind), a 0-100 score, +seat, the judge's slug, the seat (the worker, checker or planner, spelled on the wire as `worker`, +`high` and `mastermind`), a 0-100 score, the door the run came in by (task, do, exec or run), the crew size and the UTC day, under a random per-install nonce in an `X-Codeaf-Install` header. No prompt, code, path or name rides in a row. **Every way of turning the counts off turns this stream off too** — `CODEAF_TELEMETRY=off`, diff --git a/docs/changes/unreleased/1436-crew-routed-per-task.md b/docs/changes/unreleased/1436-crew-routed-per-task.md new file mode 100644 index 0000000000..80ccc77049 --- /dev/null +++ b/docs/changes/unreleased/1436-crew-routed-per-task.md @@ -0,0 +1,79 @@ +--- +kind: changed +title: a task's crew is routed per task, and /crew is the panel of what is allowed and pinned +pr: 1436 +surface: [engine, chat, docs] +invalidates: + - "The crew was a preset word — `frugal`, `balanced`, `max` — that wrote all five tier rows at once, with a family row (`models.crew.source`: `open` or `all`) and a pick row (`models.crew.pick`: `table`, `catalog`, `learn`) beside it, and a tier row could say `auto`. None of those words exists now: every seat nobody pinned is picked per task by internal/crewroute. `/crew ` is refused with the four forms the command does take." + - "The preset tables, the catalog picker package, `config/auto.go` and its catalog, index and own-cells seams, and the three shipped crew-seat default models are gone. The router's catalog seam is `config.CrewCatalog`; the Model Pool's index is no longer a crew picker." + - "The check seat inherited the plan seat's model when only the planner was named by flag or environment. It never does: `--check-model`, `CODEAF_CHECK_MODEL`, a pin, or the router's checker." + - "A seat nothing pinned fell to a model this build chose for everybody. It is routed; a seat nothing allowed can sit is an error that says so." + - "A worker row a profile never wrote was read as inherited from the small-work row, with a one-time notice. There is no inheritance and no notice: an unwritten crew row is auto." + - "The onboarding controls screen asked for a crew. It asks for the daily limit and the chat model; the crew asks nothing up front." + - "The settings rows for the working, careful and planning tiers are one `seats` row that opens the `/crew` panel. The `crew`, `model family` and `picked from` rows are gone." + - "`codeaf do`'s `model_source`/`plan_model_source` read `crew ` or `default`. They read `--model`, the variable, `pinned` or `routed`, and `-json` also carries `class`, `crew`, `est_usd`, `check_model` and `check_model_source`." + - "A task carried no dollar limit of its own, and the Spending tab's `per task` row read `no limit of its own`. A task is held to the per-task limit set in `/crew` ($5 by default), and the row reads that figure." + - "On an OpenRouter balance read as low, the five tier rows nobody set read a hand-picked free crew and `/status` called that crew `free`. Only reflex and small work read free models now; the three crew seats see the OpenRouter account as out of credit before any call and are routed to free pools, and the crew line says `free routes in use (may log prompts) · credit unavailable on openrouter`." + - "Remote protocol version 17 is replaced by 18: `Task.Start` carries the one-task effort word and `Task.RedoStronger` runs a task again on a stronger crew. An older engine refuses at the handshake rather than starting the task on the crew the person asked it not to use." +--- +A task's crew — the worker that does the work, the planner that structures it +and the checker that reads the result — is picked for that task. The router +classifies the task as a bugfix, a complex fix (a bugfix whose report shows +reach), open-ended work or other, prices every +allowed model on every connected route (a subscription plan or a local model +costs nothing to route to), and sits each seat where quality minus λ times cost +is highest, λ at the knee of the curve. `--best` and `--cheap` (on `/task`, on +the conversation's hand-off as `effort`, and on `codeaf do`) move λ for one task +only. The design behind the defaults is in +`docs/design/model-pool/pareto-crewing.pdf`. + +**Every model in the catalog is scored from its catalog row.** Fitted weights +shipped in `internal/crewroute/prior.json` turn a row into a quality per seat +and class with its variance. A row that publishes any index (the AA indexes or +arena Elo) is scored on its indexes alone, so a model no dearer and at least as +good on every shared index never ranks below another; a row with none is +scored from context, release date (the catalog's `created` field, now read), +licence and family, never above the population mean. Price is never read as +ability. Seats weigh a model at its score less one standard deviation of +ability; a worker's and a checker's mean ability must reach the floor when any +allowed model's does. On open-ended and other work at the knee a support +upgrade goes to the checker, the planner staying on the base model; `--cheap` +takes a stronger worker within 1.5× the cheapest one's cost. A model whose row is too thin for a finite score is not picked unless +pinned. Reach is read over a one-paragraph ask's whole text, a security fix is +a reach signal, and "wrongly", "rejected", "instead of" and a rename or version +bump read as fixes. No per-model table ships, and +estimates come from catalog prices times each seat's token profile times the +install's own cost factor. Each task's outcome — accepted, kept, redone, +failed — moves that model's score in that seat by a small bounded step, +recorded on the decision row. + +**What persists is what the panel says.** `/crew` opens an interactive panel: +the three seats (`auto · usually ` or a pin), the allowed-models rule +stepped in place (`all`, `open`, a price ceiling typed into two boxes, or a +custom checklist of providers and models), the daily cap typed in place, and a +line for today's spend. A seat's list starts with auto and the suggested model, +opens a model's routes, and offers to widen the rule when the pick is outside +it. Every change saves at once, ticks its row and can be undone with `z` for a +few seconds. `/crew pin `, `/crew unpin `, +`/crew models ` and `/crew cap ` write the same state and +open the panel on the changed row. In `/settings` → Providers the three seat +rows are one `seats` row that opens the panel. A pin outside the allowed models is refused. At the cap a chat task +does not start and `codeaf do` refuses unless given `-yes-spend`. + +**No task may cost more than its limit**, $5 unless set on the panel's cap row +(`per task $5 · daily none`, `tab` between the two) or with `/crew cap task `. +Every priced call of the task — each seat and the helpers made for it — is held +to one tally before it is made; the call that would pass the limit is not made +and the task stops on `this task reached its $5 limit · raise it in /crew`. +`-yes-spend` does not lift it, and `codeaf do` holds every run to it. + +**Redo is how a crew learns.** `/redo stronger` runs the last task again with +every unpinned seat one step stronger, and the router's log records that this +class of work in this repository was under-served, so the next such task starts +a step higher, at most three steps; each accepted task of that kind takes a +step back off. Nothing escalates on +its own. + +**Old profiles migrate once**, with one line: preset, pick and `auto` rows +become auto, the ids a person wrote stay pinned, and the `open` family becomes +the allowed rule `open`. diff --git a/docs/changes/unreleased/1501-crew-pinned-checker-gap.md b/docs/changes/unreleased/1501-crew-pinned-checker-gap.md new file mode 100644 index 0000000000..644747ef16 --- /dev/null +++ b/docs/changes/unreleased/1501-crew-pinned-checker-gap.md @@ -0,0 +1,8 @@ +--- +kind: fixed +title: the /crew panel's weak-checker warning counts a pinned checker +pr: 1501 +surface: [chat, docs] +invalidates: + - "The `/crew` panel's `no strong checker among the models you allow` warning was asked of the allowed models only. A strong checker pinned over a weak set left the warning on screen, and a weak checker pinned over a strong set had none. A pinned checker is now the one asked: a strong pin clears the warning, and a weak pin reads `checker pinned to · open-ended work will be checked weakly`." +--- diff --git a/docs/design/home-rethink/DATA-AUDIT.md b/docs/design/home-rethink/DATA-AUDIT.md index 6c610b6988..8bf6dd050d 100644 --- a/docs/design/home-rethink/DATA-AUDIT.md +++ b/docs/design/home-rethink/DATA-AUDIT.md @@ -511,7 +511,7 @@ Display (:77), Providers (:80), Connections (`connectcaps.go:109`). The other concepts are **separate overlays** with their own commands: `/connect` → `openConnect` (`connectpanel.go:463`), `/permissions` → `openPermissions` (`permissions.go:265`), `/harness` → `openHarness` (`harnesspanel.go:209`), -`/crew` → `crewPicker` (`crew.go:183`). `/fixes` and `/accounts` do not exist. +`/crew` → the crew chooser of the time (`crew.go:183`; since replaced by the `/crew` panel). `/fixes` and `/accounts` do not exist. | Section | Verdict | | --- | --- | diff --git a/docs/design/model-pool/Makefile b/docs/design/model-pool/Makefile deleted file mode 100644 index 08d0fd4ade..0000000000 --- a/docs/design/model-pool/Makefile +++ /dev/null @@ -1,27 +0,0 @@ -# Builds pareto-crewing.pdf: the figures from data/*.csv, then the paper. -# make stale figures + pdf -# make figures redraw every figure and table -# make clean remove build products but keep the pdf - -FIGURES := fig-front.pdf fig-cells.pdf fig-shrink.pdf fig-regret.pdf fig-semibandit.pdf tab-regret.tex tab-semibandit.tex - -all: pareto-crewing.pdf - -figures: - python3 figures.py - -# One run of the script writes every figure and table; the stamp is what make -# dates them by, so old makes without grouped targets work too. -$(FIGURES): figures.stamp -figures.stamp: figures.py data/measured-crews.csv data/seed-cells.csv - python3 figures.py - touch $@ - -pareto-crewing.pdf: pareto-crewing.tex $(FIGURES) - latexmk -pdf -interaction=nonstopmode -halt-on-error pareto-crewing.tex - -clean: - latexmk -c pareto-crewing.tex - rm -f pareto-crewing.bbl figures.stamp - -.PHONY: all figures clean diff --git a/docs/design/model-pool/REVIEW-2.md b/docs/design/model-pool/REVIEW-2.md deleted file mode 100644 index d18220d551..0000000000 --- a/docs/design/model-pool/REVIEW-2.md +++ /dev/null @@ -1,60 +0,0 @@ -# Second review of docs/design/model-pool/pareto-crewing.tex - -Referee report on the revision; the first is `REVIEW.md` (written against the draft then called `learn.tex`), the authors' answer `REVIEW-RESPONSE.md`. Every check below was made against this branch's code and data: `internal/pool/{index,judge,record,tally,outbox,pull,poolcfg}`, `internal/crewpick`, `cmd/codeaf/pool*.go`, `internal/config/{settings,auto}.go`, `relay/src/*.js`, `relay/wrangler.toml`, `.github/workflows/pool-mirror.yml`, `docs/design/model-pool/{figures.py,RUNBOOK.md,data/*.csv}`, `README.md` and the settings hint in `internal/tui3/settings.go`. Nothing was executed; no test suite, no `figures.py`. - -## Verdict - -**Minor revision.** The central mathematics is now right: Proposition 4's capped posterior derives correctly from the stated hierarchy including the bias term (checked end to end below), the drift moment no longer halves the estimate, and Proposition 3's statement is repaired. Every number quoted in Section 8 matches the committed tables exactly, the experiments run on all 23 chat-door crews, the row schema matches `record.Row` and the relay's validator field for field, and the Status paragraph is for the first time an accurate map of the branch — every claim in it checks against the tree except one word. What remains is a set of sentences that describe the relay as more merged and more complete than the code under `relay/` is (the join on the write path, drift and the constants in the hourly job, pool status's relay line, the read-side installs floor), one false sentence left inside a corrected proof, and one provenance number (72 runs over nine tasks) with no source in the repository. Each is a sentence or a small patch; none touches the paper's results. Minor revision. - -**Checks that passed.** (i) *Proposition 4* (:331–338). Under hierarchy (6), pool rows carry marginal variance $\sigma^2+\tau_1^2$ about $\mu_{rm}$, so their effective count at variance $\sigma^2$ is $\tilde n=n^{\mathrm{pool}}\sigma^2/(\sigma^2+\tau_1^2)=n^{\mathrm{pool}}\ell/(\ell+1)$; the posterior of $\mu_{rm}$ has precision $K/\sigma^2$, $K=k+\tilde n$, mean the $k{:}\tilde n$ weighted average of $\hat q_r(m)+b$ and $\bar y^{\mathrm{pool}}$; integrating $\mu_{rm}$ out gives $\mu^u$ an external precision $1/(\sigma^2/K+\tau_1^2)=W/\sigma^2$ with $W=K\ell/(K+\ell)\le\ell$, and combining with the own rows yields exactly (15)–(16) with $k^{\ast}+n^{\ast}=W$ in the ratio $k{:}\tilde n$. The Corollary is the $\ell=\infty$ limit ($W\to k+n^{\mathrm{pool}}$, $b=0$) and matches `crewpick.blendQuality` with `PriorWeightAt = 30` (crewpick.go:479) and the fold in internal/config/auto.go:169–176. (ii) *Drift* (:264): adjacent-day means have variance $\omega^2+\sigma^2(1/n_d+1/n_{d-1})$; the bracket subtracts exactly that, so (13) now estimates $\omega^2$, and the steady-state relation is untouched. (iii) *Section 8 against the tables*: Thompson 0.64 vs table 0.97 at $N=400$ (gap \$0.33, a third of 0.97), $\varepsilon$-greedy 0.76 (\$0.12 behind), crew-mean-score 0.31, crew-clean-bit 0.54, role 0.15 (half of 0.31, a third of 0.54) — all match `tab-regret.tex` and `tab-semibandit.tex` exactly; 23 chat-door crews (data/measured-crews.csv), 200 replicates, $k{=}30$, $H{=}\$10$, noise $0.15^2$ all match figures.py; 16 unknowns (6 workers + 4 checkers + 6 planners) behind 138 crews (one same-vendor worker/checker pair removes 6 of 144) match `role_truth`. (iv) *Row fields*: paper :199–201 = `record.Row` (record.go:106–116) = `schema.js` payload, nonce on the batch not the row (:201; outbox envelope, `X-Codeaf-Install`). (v) *Status* (:550) against the tree — see defect 4. (vi) README and the settings hint — see new defect 8. - -## The seventeen defects and the overstated claims - -1. **Prop 4, capped counts — fixed.** One capped external $W=K\ell/(K+\ell)$, $K=k+\tilde n$, split $k/K$ and $\tilde n/K$ (:331–333, proof :338); the "at most $\ell$ own rows" sentence now scopes to catalog and pool together (:345). Derivation verified, including the $b$ term and the $\sigma^2+\tau_1^2$ pool variance the first report found missing. -2. **Drift $\tfrac12$ — fixed.** (13) at :264 carries the matching $\sigma^2(1/n_d+1/n_{d-1})$ noise term; no factor of one half. -3. **Additive sheet vs max-join — partly fixed.** §5.2 (:224) now names both stores and the own sheet's description matches `tally.Merge` (tally.go:227) and its cell key (tally.go:123); but "The relay's store is the sheet of Definition 2, keyed by install and day and merged by (11)" is false of the code — the relay folds rows additively (relay/src/worker.js:160) and the join has no caller outside its test — see new defect 1. The abstract's join-semilattice sentence (:64) inherits the gap. -4. **Status vs tree — fixed.** The judge hook is wired to chat landings (cmd/codeaf/poolrecord.go:46–66, the `session.TaskLanding` seam), rows are recorded into the own sheet and the outbox and pushed (record.go, cmd/codeaf/poolinstall.go:71–99), the built-in key is compiled in (cmd/codeaf/pool.go:31), the blend is the Corollary's at $k{=}30$, and the relay and mirror exist. One word overstates: "the join" in the relay clause — new defect 1. -5. **Relay code — fixed.** `relay/` lands with `src/{worker,schema,sheet,index}.js`, tests and `verify.mjs`; `wrangler.toml` binds the `/pool/*` route. Residual: the ten-step runbook is at `docs/design/model-pool/RUNBOOK.md`, not "in that directory" (:493) — new defect 6. -6. **Public key — fixed.** `poolPublicKeys` carries the signer's key (cmd/codeaf/pool.go:31, matching relay/wrangler.toml's `POOL_PUBLIC_KEY`), a stored key overrides it (`poolTrustedKeys`, cmd/codeaf/pool.go:44–53), and the background refresh starts under it (cmd/codeaf/poolindex.go:75–81). -7. **Run counts — partly fixed.** The abstract no longer quotes a number (:64); Figure 6's "14 cells from 120 scored runs" (:478) is the defensible reading (data/seed-cells.csv: 14 cells, $n$ sums to 225, about 120 judged runs at ~1.9 scored roles). But the new number, "fitted by a ridge model on 72 scored runs over nine tasks" (:186, :442), is unverifiable on this branch: the runs column of measured-crews.csv sums to 1224, and `figures.py` reads the fitted columns without fitting anything, so nothing in the repository produces 72 or nine. -8. **Role-level ratio — fixed.** :445 and :461 state both comparisons and every number matches: 0.54 clean bit, 0.31 mean score, 0.15 role at $N{=}400$; "half" and "a third" hold. -9. **Prop 3 — partly fixed.** The statement is corrected (:243: one constant and one constraint per component; when $G$ is connected the single constraint identifies everything), but the proof still ends "With the single centring constraint, exactly one component is fully identified" (:246) — the removed falsehood, still false for $c\ge2$ (one constraint leaves a $(c{-}1)$-dimensional gauge; no component is pinned) — new defect 2. -10. **Subset of crews — fixed.** "all 23 chat-door crews" (:445); `figures.py` `fig_regret` takes every `door=="task"` row, and the CSV holds 23; the `[:12]` slice is gone. -11. **`min_installs` — partly fixed.** The relay computes and publishes a distinct-install count per cell (relay/src/sheet.js `aggregate` → `installs`; relay/src/index.js `buildIndex`), and §5.4's robustness claim is reduced to "a minority of installs that are broken or adversarial" (:253). But "A cell backed by fewer than three installs is published but not read" is still false of the code: both readers floor on the observation count (internal/pool/index/index.go:265 `n < x.minInstalls`; crewpick.go:235 `c.N < minInstalls`), `index.Cell` never reads a cell's `installs` field, and the seed ships `min_installs: 1` with no `installs` fields (internal/pool/index/seed.json:4). §9's flood sentence survives unchanged (:499) — the response says it was reduced; only the §5.4 half was. -12. **Row fields — fixed.** Row (role, model, score, judge, door, size, day) (:199–201) = `record.Row` (record.go:106–116) = `schema.js`'s validated payload; nonce drawn once per profile, on the batch header not the row (:201; outbox.go, cmd/codeaf/poolinstall.go:44–69); no price or duration anywhere in a row; Definition 2's cell (:206), the relay's `cellKey` (sheet.js:24) and §9's "one triple per (install, cell, day)" (:493) now spell the cell one way. -13. **Abstract — fixed.** The constants sentence scopes itself to statistical constants and names the two user prices (:64), matching Table 2 up to its last clause (new defect 5); the modes read as the code behaves (off reads nothing: cmd/codeaf/poolindex.go `poolIndexFor`). -14. **`explore` — fixed.** "specified, not on this head" (:542); no such key exists in internal/config, consistently. -15. **$E$ as surrogate — fixed.** :176 derives (3) as (2) at $r=1$, states the regime, and allows (2) directly where a delivered rate is measured. -16. **Imputation — fixed.** :110 describes the implemented rule (crewpick.go `fill`: per present index, median ratio over at least three donors, offers averaged, capped at the pool's maximum; scale-proportional standing below three donors). -17. **Loader — fixed.** :490 "the fresher of the cached index and the embedded seed index, by generated day" matches `index.Fallback` (index.go:96–104). - -Overstated claims: - -- **Abstract privacy — fixed** (:64 states what a row contains, not what can be inferred). -- **Prop 5 — fixed** (product posterior; the proof names the ignored dependence and scopes exactness to the product approximation, :424). -- **Exploration tax — partly fixed** (:445). Thompson is no longer the worst row (greedy is), but "no better than the table" is contradicted by `tab-regret.tex` at $N{=}10$ (0.97 vs 1.01) and $N{=}40$ (0.93 vs 0.95); only at $N{=}20$ is Thompson worse (0.97 vs 0.95). The differences are small; the sentence should say "within noise of the table" or quote the numbers. -- **figures.py — partly fixed** (:482). The constants at its head are $H$, $\lambda$, $K$, REPS; fig-shrink's catalog 80, measured 65 and $\sigma{=}15$, and fig-front's iso-line anchors, are function-local, so "from the two CSV files it reads and the constants written at its head" still overstates two figures. -- **pool status/show — partly fixed** (:546). `pool show` matches the code (each address with its source, the index summary); status counts pending, but "says whether the relay answered" is not implemented — status never reaches the network and prints the mode-derived can-send/can-read (cmd/codeaf/pool.go:213). New defect 7. -- **Rollback — fixed in part** (:496). The version refusal is real (internal/pool/pull/pull.go `good`); the join clause leans on the unwired join — new defect 1. -- **Judge floor — fixed** (:195, absolute; judge.go:45, applied absolutely in `Pick`). -- **"cents per run" — fixed** (:195, "one short completion per role"). - -## New defects - -1. **§5.2 (:224) and §9 (:496) claim the relay's store merges by the join (11); it does not.** The worker folds each received row additively into the per-(install, day, cell) triple (`fold`, relay/src/worker.js:160; relay/src/sheet.js:34–42), and `join` (relay/src/sheet.js:35) is called by no file except relay/test/sheet.test.js. The outbox marks a batch sent only on a 2xx (internal/pool/outbox/outbox.go, `sendHTTP`), so a batch whose response is lost after the server folded it is re-sent and folded twice: Proposition 2(iii)'s "with any duplication" and "the join of Section 5.2 makes a replayed old publication harmless" do not hold of the wire as shipped, and Status's word "the join" (:550) overstates with them. Fix: have `POST /v1/rows` carry the sender's running totals per cell and merge by `join`, or add batch-level idempotency. -2. **Proof of Proposition 3 (:246), last sentence.** "With the single centring constraint, exactly one component is fully identified" is false when $G$ has $c\ge2$ components: the constraint $\sum_C w_C\delta_C=0$ (weights $w_C$ = judge mass per component) leaves a $(c{-}1)$-dimensional gauge, so *no* component's levels are pinned under a single constraint. Delete the sentence or write what the statement's last line already says. -3. **"The relay centres $\beta$ within each component" (:249).** relay/src/sheet.js's `judgeSeverity` (:57) re-centres $\beta$ by one global weighted mean each sweep (:76–86), so the sentence does not describe the shipped fit, and on a disconnected judge–cell graph the alternating fit's fixed point remains initialisation-dependent — the first report's note, still true of the code. -4. **§9's hourly job and Figure 2 (:286, :493).** The scheduled job runs the judge-effect fit and the Huber aggregation and no drift estimate (relay/src/sheet.js; Status :550 itself defers (13)), yet §9 says the job "runs the judge-effect, robust-aggregation and drift estimates" and the figure's relay node prints "drift (13)". The mirroring is not the job's either: it is a separate GitHub workflow on a six-hour cron (.github/workflows/pool-mirror.yml:11), while the relay's own cron is hourly (relay/wrangler.toml:17). -5. **§6 (:355) and Table 2's caption (:391).** "These are published in the index beside the cells ... re-estimated at every relay cycle", and the ten-degrees-of-freedom withholding rule, are not on the branch: the relay's index document carries only schema, version, generated, min_installs, judges, rubrics, aliases, metrics and cells (relay/src/index.js `buildIndex`), no estimator lives in relay/src, and Status defers the moment estimators (:550). The caption also counts $v_r$ — measured from token accounting, not estimated — among "the rest are empirical-Bayes estimates". State these as specified-not-landed, as Status does. -6. **§9 (:493): "the runbook in that directory is ten steps."** The runbook has ten steps (docs/design/model-pool/RUNBOOK.md, Deploy 1–10) but lives beside this document, not in `relay/`; relay/README.md carries no steps. -7. **§10 (:546): "pool status counts the rows waiting in the outbox and says whether the relay answered."** The count is real (cmd/codeaf/pool.go:213, `pending %d`), but nothing in status contacts the relay; can-send/can-read are the mode's permissions (cfg.CanSend/CanRead), not an answer. Only `pool verify` reaches the network. -8. **README vs the paper on what leaves an install.** The README's Model Pool section enumerates "(role, model, a number, a day)" (README.md:201–203); the paper's row also carries judge, door and size (:199–201). The settings hint avoids the enumeration and is consistent (internal/tui3/settings.go:680–683). Name the row's fields in the README or point at the paper's list, so the two surfaces say the same thing. - -## For a camera-ready version - -- Make the relay's write path match §5.2 — snapshot triples merged by the join, or batch ids — so Proposition 2(iii) holds of the wire and Status's "the join" stands. -- Delete Proposition 3's proof-final sentence; implement per-component centring in the relay's fit or move the sentence to the design. -- Land or reword the four present-tense relay claims: drift in the hourly job (§9, :286), the constants and ten-DOF rule (§6, Table 2 caption), pool status's relay line (§10). -- Carry the distinct-install count through the readers (an `installs` field on `index.Cell`, the crewpick floor) so §5.4's three-install sentence is true of the code; revisit §9's flood sentence once fabricated installs can be bounded. -- Give "72 scored runs over nine tasks" a checkable home (state how the CSV's runs column, 1224 over 56 crews, relates to it, and that `rows[1]` — ds-v4.1-flash with qwen3.8-max — is the shipped balanced pick), and make §5.6's $\hat b=-15.7$, 75 and 231 reproducible from `figures.py` and the committed CSVs, which today carry no catalog indexes. -- Align the README's field list with the row's seven fields. diff --git a/docs/design/model-pool/REVIEW-RESPONSE.md b/docs/design/model-pool/REVIEW-RESPONSE.md deleted file mode 100644 index af8e3e7221..0000000000 --- a/docs/design/model-pool/REVIEW-RESPONSE.md +++ /dev/null @@ -1,54 +0,0 @@ -# Response to the referee - -The report in `REVIEW.md` was written against the draft at the head before this one. The paper was `learn.tex` then and is `pareto-crewing.tex` now; each numbered defect is answered below with what changed; the section numbers are the revised document's. - -## Defects - -1. **Proposition 4, the split of the cap.** Corrected. The external count is now one capped quantity `W = Kℓ/(K+ℓ)` with `K = k + ñ`, `ñ = n_pool·ℓ/(ℓ+1)`, shared between catalog and pool as `k/K` and `ñ/K` of `W`; the proof integrates `μ_rm` out once and states the precision-weighted mean. The corollary's statement ("worth at most ℓ own rows") now applies to catalog and pool together, and the sentence that follows says which form the shipped pick is (`ℓ = ∞`, `b = 0`). -2. **Drift estimator.** The factor ½ is gone from (13); the difference of adjacent day means has variance `ω² + σ²(1/n_d + 1/n_{d-1})`. -3. **Additive local sheet versus max-join.** Section 5.2 now names both stores: the install's own sheet (`internal/pool/tally`) merges by addition because each row is folded once; the relay's store is keyed by (install, day, cell) and merges by the join. -4. **Status paragraph.** Rewritten for the head this document ships with (judge hook, install nonce, push, relay, mirror) and lists what is specified but not landed. -5. **Relay code.** `relay/` lands on this head; Section 9 describes it as present. -6. **Public key.** Compiled into the binary on this head; the private-relay setting overrides it. -7. **Run counts.** The crew table is "56 combinations fitted by a ridge model on 72 scored runs over nine tasks" (caption of Figure 1 and Section 8); the seed cells come from the ledger's 120 runs (Figure 6). The abstract no longer quotes a number. -8. **Role-level ratio.** Section 8 and the abstract now state both comparisons: the role-level learner sits at half the crew-level learner fed the same scores and a third of the one fed the clean bit. -9. **Proposition 3.** Restated: identified up to one constant per connected component, one centring constraint per component; proof names the kernel. -10. **Subset of crews.** The regret experiment now runs on all 23 chat-door crews; the table and figure were regenerated and the text quotes the new numbers. -11. **`min_installs`.** Section 5.4 says the relay publishes a distinct-install count per cell and that the reader's floor applies to it; the flood claim is reduced to what the design bounds (a minority of broken or hostile installs) and says what it does not (fabricated majorities). -12. **Row fields.** The row is (role, model, score, judge, door, size, day); the install nonce is on the batch; dollars and minutes are stated to stay in the install's ledger. The cell is spelled the same way in Definition 2, the join and the row. -13. **Abstract.** The constants sentence says what the user still sets (the price of a defect and of time); "stay out and still learn" is replaced by the three modes as they behave. -14. **`explore`.** Marked "specified, not on this head" in the settings table. -15. **`E` as a surrogate.** Section 4 states that (3) is `E_H` at `r = 1`, when the ranking agrees, and that the picker can use `E_H` directly where a delivered rate is measured. -16. **Imputation.** Section 3 describes the implemented rule (median ratio over at least three donors per present index, averaged, capped at the pool's maximum, scale-proportional standing below three donors). -17. **Loader.** Section 9 says the fresher of cache and seed is read, by generated day. - -## Overstated claims - -- Privacy in the abstract is now a statement of what a row contains, not of what can be inferred. -- Proposition 5 is stated for the product posterior and says which dependence the draw ignores. -- The exploration tax is stated as forty runs in which Thompson is no better than the table. -- `figures.py` is said to reproduce the figures from the CSVs and the constants at its head. -- `pool status` and `pool show` are described as they print. -- Rollback is caught by the version; the join makes a replayed old publication harmless. -- The judge floor is stated as absolute. -- "Cents per run" is replaced by "one short completion per role". - -## What the referee asked for and is not in this revision - -- A catalog bias term `b` is added to the hierarchy with its estimator, and the seed's own `b̂ = −13.7`, `τ̂₀² = 0` are reported with the reason the day-one `k = 30` stands. -- Experiments on judge effects, drift, the settled probability and the pair term, and a flood analysis, remain future work; the Status paragraph lists them among what is specified but not landed. - -## Second round (`REVIEW-2.md`) - -Verdict was minor revision; the eight new defects are answered here. - -1. The relay folds rows additively; the paper said the join. §5.2 now says which merge is which: addition within a store (own sheet, relay), the join between copies of sheets, and names the retry double-count and the quota that bounds it. Status and §9 no longer say the relay's store is join-merged. -2. Proposition 3's proof: the sentence the referee quotes is not in the revised proof; the proof ends at the kernel and the per-component constraint. -3. §5.3 now says the reference relay centres by one global weighted mean and that per-component levels are a convention on a disconnected graph. -4. Drift is out of §9's job description and both figures' relay nodes; it stays in Status as next to land. -5. §6 now states the constants' publication and the ten-degrees-of-freedom rule as the design, and that this head's index carries the cells and the day-one constants. -6. "the runbook beside this document". -7. `pool status` reaches the relay on this head (client B, after the referee's clone); the sentence stands. -8. README's field list now names the row's seven fields. - -Camera-ready items not taken here: join-merged relay writes or batch ids; per-component centring in the relay; the distinct-install count through the readers; a checkable home for the 72 runs; the seed's offset numbers computed by `figures.py`. diff --git a/docs/design/model-pool/REVIEW.md b/docs/design/model-pool/REVIEW.md deleted file mode 100644 index b91c6fa63d..0000000000 --- a/docs/design/model-pool/REVIEW.md +++ /dev/null @@ -1,67 +0,0 @@ -# Review of docs/design/model-pool/learn.tex - -## Summary - -The paper builds the crew picker in three layers: a catalog-only pick (a pareto front in bill and quality with a log-scale knee), a privacy-preserving measurement pool that replaces the catalog's proxy for quality, and a Thompson-sampling learner over role-level posteriors. The first layer is implemented and matches the prose in every particular checked: (1) is `crewpick.SeatCost` (crewpick.go:258), (2) is `quality` (crewpick.go:463) with the same pool-normalised scale, (3) is the shortlist at `Floor = 0.80` (crewpick.go:183, 454), (4)–(5) are `FrontWith`/`Presets`/`AtKnob`/`knee` (crewpick.go:315, 376, 388, 407) including the id tie order and the epsilon, the seat shapes and the vendor rule match `DefaultShapes` and the crew loop, and the Corollary's blend with `k = 30` is `blendQuality` with `PriorWeightAt = 30` (crewpick.go:479, 189). The judge rule of 5.1 matches `judge.Pick` (judge.go:159) with floor 60 (judge.go:45), the pool modes match `poolcfg`, the signed pull with version refusal matches `internal/pool/pull`, and the seed index matches Figure 3's construction exactly: its 14 cells are Beta(1,1) posterior means and sds of a no-major-defect bit times 100 (verified against `internal/pool/index/seed.json` and `data/seed-cells.csv`, e.g. worker/glm-5.3-flash n=62, mean 65.6, sd 5.9). Lemmas 1 and 2, Propositions 1, 2 and 5, the ICC formula of 5.4, estimators (17) and (18), and the steady-state Kalman relation of 5.5 are correct as mathematics. The central statistical claim is not: the capped counts of Proposition 4 are not the posterior its own proof derives, the drift estimator (14) returns half of what it estimates, and Proposition 3's closing sentence is false. Against the branch, the Status section overstates the wiring in six places, and two of the quoted experiment numbers do not match the committed tables. Major revision. - -## Defects, by severity - -1. **Proposition 4, eq (15)–(16) (learn.tex:325–335). The capped counts are not the posterior the proof derives.** The proof computes the catalog-and-pool precision about $\mu^u$ as $(k+n^{\mathrm{pool}})\ell/(k+n^{\mathrm{pool}}+\ell)$ and then claims this "splits into the two terms of (16) in proportion to $k$ and $n^{\mathrm{pool}}$". It does not: $k\ell/(k+\ell) + n\ell/(n+\ell) \ne (k+n)\ell/(k+n+\ell)$ for any positive $k, n, \ell$. The correct split under the proof's own model is $k\ell/(k+n^{\mathrm{pool}}+\ell)$ on the catalog and $n^{\mathrm{pool}}\ell/(k+n^{\mathrm{pool}}+\ell)$ on the pool. With $k=30$, $n^{\mathrm{pool}}=100$, $\ell=30$ the printed weights sum to 38.1 against the proof's 24.4, and as $k$ and $n^{\mathrm{pool}}$ grow the printed formula lets catalog and pool together approach $2\ell$ own rows, breaking the proof's own bound of $\ell$ and the text's claim after (16) that the pool's evidence is "worth at most $\ell$ own rows" (true only of the sum). A second error sits in the proof's first stage: precision $(k+n^{\mathrm{pool}})/\sigma^2$ treats pool rows as $\sigma^2$-noise observations of $\mu_{rm}$, but under the hierarchy (6) they are other installs' rows with marginal variance $\sigma^2+\tau_1^2$, so their contribution is $n^{\mathrm{pool}}/(\sigma^2+\tau_1^2)$ — the same inconsistency biases (19). Fix: state the posterior precision as one external term, $W = K\ell/(K+\ell)$ with $K = k + n^{\mathrm{pool}}\sigma^2/(\sigma^2+\tau_1^2)$ (or $K = k+n^{\mathrm{pool}}$ under the stated simplification), weight the catalog and pool as $k/K$ and $n^{\mathrm{pool}}/K$ of $W$, keep "external evidence $\le \ell$" for the sum, and make (19)'s noise term match. The Corollary is unaffected (it is the $\ell=\infty$ limit, which both forms share). - -2. **Eq (14) (learn.tex:264). The drift estimator carries a spurious factor of one half.** For adjacent-day means, $\operatorname{Var}(\bar y_d - \bar y_{d-1}) = \omega^2 + \sigma^2(1/n_d + 1/n_{d-1})$; the bracket subtracts exactly that noise, so (14) returns $\hat\omega^2/2$ and the filter forgets at half the rate the data show. The $\tfrac12$ is correct only for differences taken at lag two. Fix: drop it, or difference at lag 2 with the matching noise term. The steady-state relation in the same paragraph, $\alpha^2/(1-\alpha) = \bar n\,\omega^2/\sigma^2$, is correct. - -3. **Eq (13)/Proposition 2 (learn.tex:209–219) against `internal/pool/tally/tally.go`: the shipped sheet does not merge by the join.** `Sheet.Merge` adds ("adds other into s", tally.go:201; "Merging a sheet into itself doubles it", tally.go:199) — commutative and associative but not idempotent — and the cell address (metric, role, model, dims) carries no install key (tally.go:123). Proposition 2 itself is correct mathematics for per-install running totals; addition is safe only for pairwise-disjoint sheets, so a re-delivered or duplicated publication double-counts and claim (iii)'s "with any duplication" does not hold of the shipped merge. The abstract's "sufficient statistics that form a join-semilattice, so any number of installs and a relay can merge them in any order and reach the same state" is a design, not a description of anything on the branch, and the Status paragraph (learn.tex:548) puts "the outbox and the sheet" on the branch. Fix: either say the shipped sheet is additive and duplicate safety is the outbox's per-node sent markers (a guarantee that does not federate), or land the install-keyed store with the max-join before making the federation claims. - -4. **Status (learn.tex:548). The judge, the sheet and the outbox are on the branch as libraries with no call sites.** `internal/pool/judge` is imported by nothing (no caller of `judge.Judge` or `judge.Pick` outside the package), `internal/pool/tally` is imported by nothing, and `outbox.Append` has no caller — `cmd/codeaf/pool.go` only reads `Pending`. No run is judged, no row is written, no sheet accrues, so the closing sentence "Until the relay exists the pool is the seed index plus each install's own sheet" is false: on the branch the pool is the seed or cached index alone, blended through `config.autoPrior` (internal/config/auto.go:144). Fix: say the three packages exist unwired and that *learn* on the branch means catalog plus index. - -5. **"The relay's code is in this repository" (learn.tex:544, with the deploy sentence at 492). There is no `relay/` directory.** Status (learn.tex:548) itself lists the relay as "specified here and next to land"; sections 9 and "What the user controls" contradict it in the present tense. Fix: future tense, or land the directory. - -6. **"The signing key's public half is compiled into the binary" (learn.tex:495, 544). It is not.** `var poolPublicKeys []ed25519.PublicKey` is empty, with the comment "EMPTY ON THIS BRANCH" (cmd/codeaf/pool.go:37); `pool verify` refuses without `--key` (cmd/codeaf/pool.go, `verifyPool`), and the background refresh never starts because `startPoolIndexRefresh` returns when the key list is empty (cmd/codeaf/poolindex.go:77) — so "a refresh runs in the background" (learn.tex:489) is inert on this branch. Fix: say the mechanism is in place and the key ships beside the first published index, or compile a key. - -7. **"120 scored runs" (abstract, learn.tex:64; section 8, learn.tex:441; captions at 186 and 477). The number matches neither committed dataset where it is used.** The `runs` column of `data/measured-crews.csv` sums to 1224 across the 56 crews, and the seed's 14 cells carry n summing to 225 role-level scores (consistent with about 120 judged runs at 1.9 roles each — Figure 3's usage is the defensible one). But the abstract, the Figure 1 caption and section 8 attach the 120 to the 56-crew table, whose rows are backed by 1224 runs, and the section 8 experiments take that table as ground truth. Fix: say what the 120 counts (judged runs) and correct the three locations that attach it to the fitted-crew table. - -8. **Section 8 (learn.tex:460). "at four hundred runs the role-level learner's regret is about a third of the crew-level learner's on the same scores."** `tab-semibandit.tex` at $N=400$ reads 0.15 (role), 0.31 (crew, mean judge score), 0.54 (crew, clean bit). Against the learner the sentence names — fed the same scores — the ratio is about one half. A third holds only against the clean-bit learner, which is not "on the same scores"; the abstract's "factor of three" matches that other comparison. Fix: name the baseline and quote it; as written the sentence is wrong against its own table. - -9. **Proposition 3 (learn.tex:246). "With the single centring constraint, exactly one component is fully identified."** False when $G$ has $c \ge 2$ components: the kernel of the two-way design is $c$-dimensional and the single constraint $\sum_j n_{\cdot\cdot j}\beta_j = 0$ leaves an $(c{-}1)$-dimensional gauge that shifts $\mu$ in every component jointly (shifts weighted by $\sum_C N_C\delta_C = 0$), so no component's levels are pinned — only the within-component contrasts, as the preceding sentence correctly says. Identifying a component's levels needs one centring constraint per component. Note also that the alternating re-centring (learn.tex:235–239) is global, so on a disconnected $G$ its fixed point depends on the initialisation. Fix: one constraint per component, or state that identification holds only when $G$ is connected. - -10. **Section 8 (learn.tex:444). "the twelve chat-door crews" — the data has 23; `figures.py` keeps the first 12 in file order (figures.py:136, `[:12]`).** No criterion for the twelve is stated anywhere; which twelve they are sets the oracle ($1.94 all-in) and every number in `tab-regret.tex`. Fix: state the subset rule, or use all 23. - -11. **Section 5.4 (learn.tex:253). "A cell backed by fewer than three installs is published but not read" — nothing on the branch counts installs.** The index reader drops a cell when its n — an observation count — falls below the document's `min_installs` (internal/pool/index/index.go:265), `crewpick.PriorFromCells` applies the same test to a count (crewpick.go:231), and the shipped seed sets `min_installs` to 1 (internal/pool/index/seed.json:4); one install contributing three rows passes. The related claim in section 9 (learn.tex:498) that "the min-installs and robust-aggregation rules bound what a flood of fabricated nonces can move" is unsupported: the Huber rule's breakdown point of 1/2 is in installs, fabricated installs are unauthenticated, and a flood that outnumbers genuine installs passes the breakdown point while each nonce clears any rows-per-install cap at trivial cost. Fix: carry a distinct-install count per cell in the published document, set the shipped floor to it, and either analyse the flood bound or drop the claim. - -12. **Section 5.1 (learn.tex:199–201). The row fields exist nowhere; the shipped row is an envelope.** The outbox stores `{schema, day, nonce, payload}` with a fresh 16-byte nonce per row (internal/pool/outbox/outbox.go:49, 135) and an opaque payload that nothing builds; the per-profile install nonce of learn.tex:201 exists in no file; the sheet's cell is (metric, role, model, dims) with the triple (n, sum, sumsq) (internal/pool/tally/tally.go:123, 266), which can carry judge/day/size/door as dims but carries none today. Relatedly, Definition 2 (learn.tex:203) puts day inside the cell and section 5.5 says "The day is a cell dimension", while section 9 (learn.tex:492) says the store holds "one triple per (install, cell, day)" — the two spellings of cell differ. Fix: present the row schema as specified and add it to Status, and pick one spelling of the cell. - -13. **Abstract (learn.tex:64), two claims contradicted by the paper's own later sections.** "every constant except the user's price of a defect is an empirical-Bayes estimate" — Table 2 says $\lambda$ is the user's, $\phi=0.8$ and the judge floor 60 are structural, and $v_r$ is measured; only $\sigma$, $k$, $\ell_f$, $\omega^2$ and $\beta$ are estimated. "stay out, and still learns from its own runs" — with the pool off the code reads no index at all (cmd/codeaf/poolindex.go:50) and the paper itself says "with it off, only the catalog prior remains" (learn.tex:544). Fix both sentences. - -14. **"What the user controls" (learn.tex:540). The `explore` setting does not exist.** `internal/config` carries `model_pool` (settings.go:119), the pick words and the crew presets, but no `explore` key anywhere; nothing toggles a draw. Fix: mark the row "next to land", as Status does for the draw. - -15. **Section 4 (learn.tex:171–173). The working objective $E$ is neither derived form.** $E = c+\lambda t + H(1-p_{\mathrm{clean}})$ equals $E_H$ with $r$ set to 1 and $p$ replaced by $p_{\mathrm{clean}}$: non-deliveries are priced as shipped defects, and the rerun term $(1-r)(c_f+\lambda t_f)/r$ — and $E_0$'s $1/(rp)$ factor — are gone. For a cheap but flaky crew with $r$ well below 1, $E$ can rank it above what $E_0$ or $E_H$ would; the paper never states the surrogate. Fix: one sentence deriving $E$ as $E_H$ under $r=1$ (or adding the rerun term), and the regime where the surrogate is safe ($H$ large next to run cost, $r$ not far below 1). - -16. **Section 3 (learn.tex:110). The imputation sentence does not describe the implemented imputation.** `crewpick` uses the donor median only with three or more donors, averages the offers from every present index, caps the estimate at the pool's largest measured value of the missing index, and falls back to scale-proportional standing below three donors (crewpick.go:525, 582). The sentence describes none of this. Fix: describe the implemented rule, or mark the sentence as simplified. - -17. **Section 9 (learn.tex:489). "The loader reads the cached index if one is present and the embedded seed index otherwise."** The cache wins only when it is not older than the seed: `index.Fallback` compares the generated days and prefers the seed on an older cache (internal/pool/index/index.go:96–104). Fix: "the fresher of the cache and the seed by generated day." - -## Correct but overstated - -- Abstract (learn.tex:64): "nothing an install sends can be read back into what it was working on." Absence of text is not unlinkability: a row's (dollars, minutes, size, door, day, role, model) is a quasi-identifying tuple against anyone who knows the task schedule. Section 9's store-scoped claim (learn.tex:498) is the defensible form; the abstract goes further than it. -- Proposition 5 (learn.tex:420): "is exactly Thompson sampling over the crews' posterior costs." Exact only if the per-(role, model) posteriors are independent; 5.3's judge-effect fit and 5.5's drift model both induce cross-cell dependence that the independent draw ignores. It is Thompson sampling on a product approximation to the posterior. -- Section 8 (learn.tex:444): "Thompson pays a small exploration tax in the first twenty runs." `tab-regret.tex` puts Thompson at 0.98 and 0.97 at $N=10$ and 20 — the worst row of the table, above even the fixed table (0.91, 0.93). The tax is real; "small" is not. -- Section 8 (learn.tex:481): "reproduces every figure and table from the two CSV files it reads." Figure 5's constants (catalog 80, measured 65, $\sigma=15$) and Figure 1's iso-line anchors are hardcoded in `figures.py`. -- Sections 9 and "What the user controls" (learn.tex:489, 544): `pool status` prints a pending count ("pending %d", cmd/codeaf/pool.go:172), not the rows; `pool show` prints the index's generated day and counts, not its cells. -- Section 9 (learn.tex:495): "(Section 5.2's join makes a rollback detectable)." A rollback is detected by the version number the puller compares (`Puller.good`, internal/pool/pull/pull.go); the join makes old data harmless, not detectable. -- Section 5.1 (learn.tex:195): "The floor exists because a judge below the worker's class rubber-stamps." The rule is an absolute floor of 60 (judge.go:45), not a floor at the worker's class: a coding-61 judge over a coding-90 worker passes it. The rationale is relative, the rule absolute. -- Section 5.1 (learn.tex:195): "at present prices it is cents per run" — not checkable from anything in the repository. - -## What a reader would still want - -- The posterior under the full hierarchy (pool rows at variance $\sigma^2+\tau_1^2$): the corrected (16) and (19), and a first $\hat k$ computed from the seed's own 14 cells — the paper's data can produce one and never does. -- The regime in which (7) ranks crews as $E_0$ and $E_H$ do: a worked example with $r<1$ and a cheap, flaky crew. -- Experiments for the untested sections: the judge-effect fit of 5.3 (the seed carries a single judge id, so the shipped data cannot exercise it), the drift model of 5.5, and the settled probability $\pi^\ast$ of section 7 — no trajectory is shown for any of them. -- The pair and interaction term of 6.1: specified with an estimator, never simulated. -- The subset rule behind "the twelve chat-door crews" and the identity of the "fixed table row" (the second line of `data/measured-crews.csv`: glm-5.3-flash with qwen3.8-max, all-in regret \$0.96 against the subset oracle), with `tab-regret.tex`'s sensitivity to both choices. -- The fabricated-install flood analysed rather than asserted (defect 11): what the Huber rule bounds once fabricated installs are the majority, and the cost per nonce to get there. -- Instance-dependent gaps: the paper's own caption says a dozen crews sit within cents; the whole regret scale (\$0.96 for the table against a \$1.94 oracle all-in) is governed by gaps the paper never reports, so "a third" and "factor of three" are dataset-specific with no gap structure given. -- Where the day-one constants come from: $\sigma=15$ and $\ell_f=30$ in Table 2 have no derivation. - -## Verdict - -Major revision. The catalog half of the paper is sound and matches the code line for line, and the CRDT, fixed-effects and robustness designs are right as designs. But the central estimator is wrong as printed (defect 1), one moment estimator is off by a factor of two (defect 2), one proposition's closing claim is false (defect 9), and the paper's account of what is implemented overstates the branch in six places (defects 3–6, 12, 14). The experiments support the direction of every claim they are attached to, but two quoted numbers do not match the committed tables (defects 7–8). Fix the mathematics, reconcile the Status paragraph with the tree, and re-derive or re-run the affected numbers; the architecture is worth publishing once every claim is as accurate as the catalog section already is. diff --git a/docs/design/model-pool/data/measured-crews.csv b/docs/design/model-pool/data/measured-crews.csv deleted file mode 100644 index 75e0f98120..0000000000 --- a/docs/design/model-pool/data/measured-crews.csv +++ /dev/null @@ -1,57 +0,0 @@ -door,worker,planner,checker,delivered,clean,usd,minutes,runs -task,ds-v4.1-flash,-,qwen3.8-max,0.75,0.69,0.13,0,5 -task,glm-5.3-flash,-,qwen3.8-max,0.95,0.78,0.27,4,17 -task,ds-v4.1-flash,-,kimi-k3,0.75,0.69,0.24,2,7 -task,glm-5.3,-,qwen3.8-max,0.91,0.85,0.53,1,11 -task,ds-v4.1-flash,-,fable-5.1,0.75,0.87,0.45,0,8 -task,glm-5.3-flash,-,kimi-k3,0.95,0.79,0.50,11,21 -do,ds-v4.1-flash,opus-5,-,0.67,0.9,0.46,1,12 -do,ds-v4.1-flash,glm-5.3-flash,-,0.67,0.84,0.36,0,16 -task,ds-v4.1-flash,-,opus-5,0.75,0.64,0.40,0,7 -do,ds-v4.1-flash,qwen3.8-max,-,0.67,0.56,0.22,0,11 -do,ds-v4.1-flash,glm-5.3,-,0.67,0.51,0.23,0,9 -task,muse-spark-1.3,-,qwen3.8-max,0.5,0.95,0.41,0,16 -task,glm-5.3-flash,-,fable-5.1,0.95,0.95,0.95,1,25 -task,glm-5.3,-,kimi-k3,0.91,0.85,0.98,6,14 -task,glm-5.3-flash,-,opus-5,0.95,0.74,0.84,4,20 -do,muse-spark-1.3,qwen3.8-max,-,0.75,0.82,0.72,0,32 -do,ds-v4.1-flash,kimi-k3,-,0.67,0.39,0.18,0,11 -do,muse-spark-1.3,glm-5.3,-,0.75,0.77,0.75,0,26 -do,glm-5.3,qwen3.8-max,-,0.92,0.72,0.93,2,22 -task,muse-spark-1.3,-,kimi-k3,0.5,0.95,0.76,0,19 -do,muse-spark-1.3,kimi-k3,-,0.75,0.65,0.59,0,32 -do,glm-5.3-flash,opus-5,-,0.64,0.95,0.96,1,37 -do,glm-5.3-flash,qwen3.8-max,-,0.64,0.66,0.47,2,34 -do,glm-5.3-flash,glm-5.3,-,0.64,0.6,0.49,1,27 -do,glm-5.3,kimi-k3,-,0.92,0.55,0.76,1,23 -do,ds-v4.1-flash,fable-5.1,-,0.67,0.71,0.89,0,13 -do,muse-spark-1.3,glm-5.3-flash,-,0.75,0.95,1.16,0,48 -do,glm-5.3,glm-5.3-flash,-,0.92,0.95,1.50,1,33 -do,muse-spark-1.3,opus-5,-,0.75,0.95,1.45,0,35 -task,glm-5.3,-,fable-5.1,0.91,0.95,1.87,1,16 -do,glm-5.3-flash,kimi-k3,-,0.64,0.49,0.39,3,34 -do,glm-5.3,opus-5,-,0.92,0.95,1.88,2,24 -task,glm-5.3,-,opus-5,0.91,0.8,1.65,1,13 -task,muse-spark-1.3,-,opus-5,0.5,0.9,1.28,0,19 -task,qwen3.8-max,-,kimi-k3,0.83,0.69,1.31,2,22 -task,muse-spark-1.3,-,fable-5.1,0.5,0.95,1.44,0,24 -do,qwen3.8-max,glm-5.3-flash,-,0.86,0.84,2.01,1,55 -do,glm-5.3-flash,fable-5.1,-,0.64,0.81,1.87,5,41 -task,qwen3.8-max,-,fable-5.1,0.83,0.87,2.50,2,27 -do,qwen3.8-max,glm-5.3,-,0.86,0.5,1.30,1,29 -task,ds-v4-pro-0813,-,qwen3.8-max,0.5,0.35,0.47,0,7 -do,qwen3.8-max,opus-5,-,0.86,0.9,2.53,2,40 -do,muse-spark-1.3,fable-5.1,-,0.75,0.95,2.84,2,39 -do,ds-v4-pro-0813,glm-5.3-flash,-,0.67,0.5,1.32,0,20 -do,ds-v4-pro-0813,opus-5,-,0.67,0.56,1.66,0,14 -do,qwen3.8-max,kimi-k3,-,0.86,0.39,1.02,0,37 -task,qwen3.8-max,-,opus-5,0.83,0.64,2.22,0,22 -task,ds-v4-pro-0813,-,fable-5.1,0.5,0.54,1.64,0,10 -task,ds-v4-pro-0813,-,kimi-k3,0.5,0.36,0.86,0,8 -do,glm-5.3,fable-5.1,-,0.92,0.87,3.68,4,27 -do,ds-v4-pro-0813,qwen3.8-max,-,0.67,0.23,0.82,0,13 -task,ds-v4-pro-0813,-,opus-5,0.5,0.3,1.46,0,8 -do,ds-v4-pro-0813,glm-5.3,-,0.67,0.17,0.85,0,11 -do,qwen3.8-max,fable-5.1,-,0.86,0.71,4.94,1,44 -do,ds-v4-pro-0813,fable-5.1,-,0.67,0.38,3.24,1,16 -do,ds-v4-pro-0813,kimi-k3,-,0.67,0.06,0.67,0,13 diff --git a/docs/design/model-pool/fig-cells.pdf b/docs/design/model-pool/fig-cells.pdf deleted file mode 100644 index 4ba5204fe9..0000000000 Binary files a/docs/design/model-pool/fig-cells.pdf and /dev/null differ diff --git a/docs/design/model-pool/fig-front.pdf b/docs/design/model-pool/fig-front.pdf deleted file mode 100644 index e6931cdfef..0000000000 Binary files a/docs/design/model-pool/fig-front.pdf and /dev/null differ diff --git a/docs/design/model-pool/fig-regret.pdf b/docs/design/model-pool/fig-regret.pdf deleted file mode 100644 index 16c0897db2..0000000000 Binary files a/docs/design/model-pool/fig-regret.pdf and /dev/null differ diff --git a/docs/design/model-pool/fig-semibandit.pdf b/docs/design/model-pool/fig-semibandit.pdf deleted file mode 100644 index 5cda759ece..0000000000 Binary files a/docs/design/model-pool/fig-semibandit.pdf and /dev/null differ diff --git a/docs/design/model-pool/fig-shrink.pdf b/docs/design/model-pool/fig-shrink.pdf deleted file mode 100644 index 981e0c5a5b..0000000000 Binary files a/docs/design/model-pool/fig-shrink.pdf and /dev/null differ diff --git a/docs/design/model-pool/figures.py b/docs/design/model-pool/figures.py deleted file mode 100644 index d33f7b1a2e..0000000000 --- a/docs/design/model-pool/figures.py +++ /dev/null @@ -1,288 +0,0 @@ -#!/usr/bin/env python3 -"""Draws the figures in pareto-crewing.tex from data/*.csv. Standard library, numpy and matplotlib only. - - python3 figures.py # writes fig-*.pdf and tab-*.tex beside this file - -The simulations are the ones the paper describes. Ground truth is the table of measured crews fitted -from our own scored runs (data/measured-crews.csv). A learner picks a crew per run, sees the run's -outcome and its dollars, and pays the all-in cost E = c + lambda*t + H*(1 - clean). Regret is E minus -the oracle crew's, averaged over replicates. Every replicate draws its own noisy catalog prior. -""" -import csv, os, random -import numpy as np -import matplotlib -matplotlib.use("Agg") -import matplotlib.pyplot as plt - -HERE = os.path.dirname(os.path.abspath(__file__)) -TABLE_PROVENANCE = "% generated by figures.py from data/measured-crews.csv; do not edit\n" -H, LAM, K = 10.0, 0.01, 30 -REPS = 200 -plt.rcParams.update({"font.size": 9, "font.family": "serif", "axes.spines.top": False, "axes.spines.right": False}) - - -def crews(): - rows = list(csv.DictReader(open(os.path.join(HERE, "data", "measured-crews.csv")))) - for r in rows: - for k in ("delivered", "clean", "usd", "minutes"): - r[k] = float(r[k]) - r["runs"] = int(r["runs"]) - r["name"] = r["worker"] + ("+" + r["checker"] if r["checker"] != "-" else "") + ("/" + r["planner"] if r["planner"] != "-" else "") - r["run"] = r["usd"] + LAM * r["minutes"] - r["p"] = r["delivered"] * r["clean"] - return rows - - -def allin(r, h=H): - return r["run"] + h * (1 - r["p"]) - - -def fig_front(): - rows = crews() - fig, ax = plt.subplots(figsize=(5.4, 3.5)) - for door, mark, label in (("task", "o", "chat door (worker + checker)"), ("do", "s", "headless door (worker + planner)")): - pts = [r for r in rows if r["door"] == door] - ax.scatter([r["run"] for r in pts], [100 * r["p"] for r in pts], s=18, marker=mark, alpha=.75, label=label) - front, best = [], -1 - for r in sorted(rows, key=lambda r: r["run"]): - if r["p"] > best: - front.append(r) - best = r["p"] - ax.plot([r["run"] for r in front], [100 * r["p"] for r in front], "k-", lw=.8, label="pareto front") - for r in front: - ax.annotate(r["name"], (r["run"], 100 * r["p"]), fontsize=6, xytext=(4, -9), textcoords="offset points") - xs = np.linspace(0.05, 4, 100) - for h, e, x in ((1, 1.2, 1.0), (10, 3.0, 2.4), (50, 12.0, 3.2)): - ax.plot(xs, 100 * (1 - (e - xs) / h), color="#999", lw=.6, ls="--") - ax.text(x, 100 * (1 - (e - x) / h) + 1.5, f"E = \\${e:.0f} at H = \\${h}", fontsize=6, color="#777") - ax.set_xlim(0, 4) - ax.set_ylim(0, 100) - ax.set_xlabel("cost of one run, $ (dollars + λ · minutes)") - ax.set_ylabel("P(delivered and clean), %") - ax.legend(fontsize=7, loc="lower left") - fig.tight_layout() - fig.savefig(os.path.join(HERE, "fig-front.pdf")) - - -def fig_cells(): - rows = list(csv.DictReader(open(os.path.join(HERE, "data", "seed-cells.csv")))) - fig, axes = plt.subplots(1, 3, figsize=(6.2, 2.6)) - for ax, role in zip(axes, ("worker", "high", "mastermind")): - cells = sorted([r for r in rows if r["role"] == role], key=lambda r: -float(r["mean"])) - names = [r["model"].split("/")[-1] for r in cells] - ax.barh(names, [float(r["mean"]) for r in cells], xerr=[float(r["sd"]) for r in cells], color="#8a8fb0", ecolor="#333", capsize=2) - for i, r in enumerate(cells): - ax.text(2, i, f"n={r['n']}", va="center", fontsize=6, color="white") - ax.set_title(role, fontsize=9) - ax.set_xlim(0, 100) - ax.invert_yaxis() - ax.tick_params(axis="y", labelsize=7) - axes[0].set_xlabel("role_quality, posterior mean ± sd") - fig.tight_layout() - fig.savefig(os.path.join(HERE, "fig-cells.pdf")) - - -def fig_shrink(): - q_cat, s_bar, sigma = 80.0, 65.0, 15.0 - n = np.arange(0, 121) - mean = (K * q_cat + n * s_bar) / (K + n) - sd = sigma / np.sqrt(K + n) - fig, ax = plt.subplots(figsize=(5.2, 2.8)) - ax.fill_between(n, mean - 1.96 * sd, mean + 1.96 * sd, color="#8a8fb0", alpha=.3, label="95% posterior band") - ax.plot(n, mean, "k-", label="posterior mean") - ax.axhline(q_cat, ls=":", color="#555", lw=.8) - ax.axhline(s_bar, ls="--", color="#555", lw=.8) - ax.text(2, q_cat + 0.8, "catalog prior", fontsize=7) - ax.text(100, s_bar + 0.8, "measured mean", fontsize=7) - ax.axvline(K, color="#a55", lw=.8) - ax.text(K + 1.5, 62, "n = k = 30: half weight each", fontsize=7, color="#a55") - ax.set_xlabel("effective observations n") - ax.set_ylabel("role quality") - ax.set_xlim(0, 120) - ax.set_ylim(60, 84) - ax.legend(fontsize=7, loc="upper right") - fig.tight_layout() - fig.savefig(os.path.join(HERE, "fig-shrink.pdf")) - - -def simulate(strategy, N, seed, rows, oracle): - """Crew-level learners: every crew is one arm with one Beta posterior on P(clean).""" - rnd = random.Random(seed) - truth = {r["name"]: r["p"] for r in rows} - prior = {n: min(.99, max(.01, p + rnd.gauss(0, .15))) for n, p in truth.items()} - a = {n: 1 + K * prior[n] for n in truth} - b = {n: 1 + K * (1 - prior[n]) for n in truth} - fixed = {r["name"]: r["run"] for r in rows} - total, explored = 0.0, 0 - for t in range(N): - mean_best = min(truth, key=lambda n: fixed[n] + H * (1 - a[n] / (a[n] + b[n]))) - if strategy == "table": - pick = rows[1]["name"] - elif strategy == "catalog": - pick = min(truth, key=lambda n: fixed[n] + H * (1 - prior[n])) - elif strategy == "greedy": - pick = mean_best if rnd.random() >= 0.1 else rnd.choice(list(truth)) - else: - pick = min(truth, key=lambda n: fixed[n] + H * (1 - rnd.betavariate(a[n], b[n]))) - explored += pick != mean_best - clean = rnd.random() < truth[pick] - total += fixed[pick] + H * (1 - clean) - oracle - if strategy in ("greedy", "thompson"): - a[pick] += clean - b[pick] += not clean - return total / N, explored / N - - -def fig_regret(): - rows = [r for r in crews() if r["door"] == "task"] - oracle = min(allin(r) for r in rows) - Ns = [10, 20, 40, 70, 100, 150, 200, 300, 400] - fig, (ax, ax2) = plt.subplots(1, 2, figsize=(6.2, 2.8)) - table = {} - for s, style in (("table", "k:"), ("catalog", "k--"), ("greedy", "C1-"), ("thompson", "C0-")): - reg, exp = [], [] - for N in Ns: - rs = [simulate(s, N, seed, rows, oracle) for seed in range(REPS)] - reg.append(np.mean([r[0] for r in rs])) - exp.append(np.mean([r[1] for r in rs])) - table[s] = reg - ax.plot(Ns, reg, style, label={"table": "fixed table row", "catalog": "catalog only", "greedy": "ε-greedy, ε = 0.1", "thompson": "Thompson"}[s]) - if s in ("greedy", "thompson"): - ax2.plot(Ns, [100 * e for e in exp], style, label=s) - ax.set_xlabel("runs N") - ax.set_ylabel("regret vs oracle, $ per task") - ax.set_ylim(0, None) - ax.legend(fontsize=7) - ax2.set_xlabel("runs N") - ax2.set_ylabel("runs off the posterior-mean pick, %") - ax2.set_ylim(0, None) - ax2.legend(fontsize=7) - fig.tight_layout() - fig.savefig(os.path.join(HERE, "fig-regret.pdf")) - with open(os.path.join(HERE, "tab-regret.tex"), "w") as f: - f.write(TABLE_PROVENANCE) - f.write("\\begin{tabular}{l" + "r" * len(Ns) + "}\\toprule\n$N$ & " + " & ".join(str(n) for n in Ns) + "\\\\\\midrule\n") - for s in ("table", "catalog", "greedy", "thompson"): - f.write(s + " & " + " & ".join(f"{v:.2f}" for v in table[s]) + "\\\\\n") - f.write("\\bottomrule\\end{tabular}\n") - - -VENDOR = {"ds-v4.1-flash": "deepseek", "ds-v4-pro-0813": "deepseek", "ds-v4-flash-0731": "deepseek", "glm-5.3-flash": "z-ai", - "glm-5.3": "z-ai", "muse-spark-1.3": "meta", "qwen3.8-max": "qwen", "kimi-k3": "moonshot", "fable-5.1": "anthropic", - "opus-5": "anthropic"} - - -def role_truth(): - """A three-role ground truth read off the measured rows under the linear rule of the paper: a role quality - q_r(m) in [0, 100] per (role, model) and a dollar surcharge per (role, model); a crew's P(clean) is the mean of - its role qualities over 100 and its cost the sum of its surcharges. Workers and checkers come from the chat - rows (delivered is the worker's, clean the checker's), planners from the headless rows.""" - rows = crews() - chat = [r for r in rows if r["door"] == "task"] - do = [r for r in rows if r["door"] == "do"] - mean = lambda xs: float(np.mean(xs)) - q = {"worker": {}, "checker": {}, "planner": {}} - cost = {"worker": {}, "checker": {}, "planner": {}} - for w in sorted({r["worker"] for r in chat}): - mine = [r for r in chat if r["worker"] == w] - q["worker"][w] = 100 * mean([r["delivered"] for r in mine]) - cost["worker"][w] = mean([r["run"] for r in mine]) - base = mean([r["run"] for r in chat]) - for c in sorted({r["checker"] for r in chat}): - mine = [r for r in chat if r["checker"] == c] - q["checker"][c] = 100 * mean([r["clean"] for r in mine]) - cost["checker"][c] = mean([r["run"] for r in mine]) - base - dobase = mean([r["run"] for r in do]) - for p in sorted({r["planner"] for r in do}): - mine = [r for r in do if r["planner"] == p] - q["planner"][p] = 100 * mean([r["clean"] for r in mine]) - cost["planner"][p] = max(0.0, mean([r["run"] for r in mine]) - dobase) - grid = {} - for w in q["worker"]: - for c in q["checker"]: - if VENDOR.get(w) == VENDOR.get(c): - continue - for p in q["planner"]: - k = (w, c, p) - grid[k] = dict(p=(q["worker"][w] + q["checker"][c] + q["planner"][p]) / 300, - run=cost["worker"][w] + cost["checker"][c] + cost["planner"][p]) - return q, grid - - -def simulate_roles(level, N, seed, q, grid, sigma=15.0): - """Thompson sampling under three kinds of feedback on the same truth: one Beta arm per crew fed the run's - clean/defect bit; one Gaussian arm per crew fed the mean of the judge's role scores; one Gaussian posterior per - (role, model) fed each role's own judge score (semi-bandit). Judge scores are the role quality plus N(0, sigma).""" - rnd = random.Random(seed) - arms = list(grid) - roles = ("worker", "checker", "planner") - oracle = min(grid[a]["run"] + H * (1 - grid[a]["p"]) for a in arms) - noise = lambda x: min(99.0, max(1.0, x + rnd.gauss(0, 15))) - prior_q = {r: {m: noise(q[r][m]) for m in q[r]} for r in roles} - prior_p = {a: sum(prior_q[r][a[i]] for i, r in enumerate(roles)) / 300 for a in arms} - if level == "crew-bit": - a_ = {a: 1 + K * prior_p[a] for a in arms} - b_ = {a: 1 + K * (1 - prior_p[a]) for a in arms} - elif level == "crew-score": - n_ = {a: float(K) for a in arms} - s_ = {a: K * 100 * prior_p[a] for a in arms} - else: - n_ = {(r, m): float(K) for r in roles for m in q[r]} - s_ = {(r, m): K * prior_q[r][m] for r in roles for m in q[r]} - total = 0.0 - for t in range(N): - if level == "crew-bit": - pick = min(arms, key=lambda a: grid[a]["run"] + H * (1 - rnd.betavariate(a_[a], b_[a]))) - elif level == "crew-score": - pick = min(arms, key=lambda a: grid[a]["run"] + H * (1 - rnd.gauss(s_[a] / n_[a], sigma / n_[a] ** .5) / 100)) - else: - draw = {k: rnd.gauss(s_[k] / n_[k], sigma / n_[k] ** .5) for k in n_} - pick = min(arms, key=lambda a: grid[a]["run"] + H * (1 - sum(draw[(r, a[i])] for i, r in enumerate(roles)) / 300)) - g = grid[pick] - clean = rnd.random() < g["p"] - total += g["run"] + H * (1 - clean) - oracle - scores = [q[r][pick[i]] + rnd.gauss(0, sigma) for i, r in enumerate(roles)] - if level == "crew-bit": - a_[pick] += clean; b_[pick] += not clean - elif level == "crew-score": - n_[pick] += 1; s_[pick] += float(np.mean(scores)) - else: - for i, r in enumerate(roles): - n_[(r, pick[i])] += 1; s_[(r, pick[i])] += scores[i] - return total / N - - -def fig_semibandit(): - q, grid = role_truth() - Ns = [10, 20, 40, 70, 100, 150, 200, 300, 400] - fig, ax = plt.subplots(figsize=(4.8, 2.9)) - table = {} - for level, style, label in (("crew-bit", "C0:", "one arm per crew, clean/defect bit"), - ("crew-score", "C0--", "one arm per crew, mean judge score"), - ("role", "C2-", "one posterior per (role, model), role scores")): - reg = [np.mean([simulate_roles(level, N, seed, q, grid) for seed in range(REPS)]) for N in Ns] - table[level] = reg - ax.plot(Ns, reg, style, label=label) - d = sum(len(q[r]) for r in q) - ax.set_title(f"{len(grid)} crews from {len(q['worker'])} workers × {len(q['checker'])} checkers × {len(q['planner'])} planners; {d} unknowns", fontsize=8) - ax.set_xlabel("runs N") - ax.set_ylabel("regret vs oracle, $ per task") - ax.set_ylim(0, None) - ax.legend(fontsize=7) - fig.tight_layout() - fig.savefig(os.path.join(HERE, "fig-semibandit.pdf")) - with open(os.path.join(HERE, "tab-semibandit.tex"), "w") as f: - f.write(TABLE_PROVENANCE) - f.write("\\begin{tabular}{l" + "r" * len(Ns) + "}\\toprule\n$N$ & " + " & ".join(str(n) for n in Ns) + "\\\\\\midrule\n") - for level, name in (("crew-bit", "crew, clean bit"), ("crew-score", "crew, mean score"), ("role", "role, semi-bandit")): - f.write(name + " & " + " & ".join(f"{v:.2f}" for v in table[level]) + "\\\\\n") - f.write("\\bottomrule\\end{tabular}\n") - - -if __name__ == "__main__": - fig_front() - fig_cells() - fig_shrink() - fig_regret() - fig_semibandit() - print("figures written") diff --git a/docs/design/model-pool/pareto-crewing.pdf b/docs/design/model-pool/pareto-crewing.pdf index 7771ef6318..397dac7e62 100644 Binary files a/docs/design/model-pool/pareto-crewing.pdf and b/docs/design/model-pool/pareto-crewing.pdf differ diff --git a/docs/design/model-pool/pareto-crewing.tex b/docs/design/model-pool/pareto-crewing.tex deleted file mode 100644 index c57a7ac586..0000000000 --- a/docs/design/model-pool/pareto-crewing.tex +++ /dev/null @@ -1,567 +0,0 @@ -\documentclass[10pt,a4paper]{article} -\usepackage[margin=2.54cm]{geometry} -\usepackage{amsmath,amssymb,amsthm,mathtools} -\usepackage{newtxtext,newtxmath} -\usepackage{booktabs} -\usepackage{graphicx} -\usepackage{microtype} -\usepackage{authblk} -\usepackage{algorithm} -\usepackage{algpseudocode} -\usepackage{tikz} -\usetikzlibrary{arrows.meta,positioning,shapes.geometric,calc,fit} -\usepackage{fancyhdr} -\usepackage[hidelinks]{hyperref} -\usepackage{caption} -\usepackage{float} -\captionsetup{font=small,labelfont=bf,width=.92\textwidth} -\usepackage{abstract} -\renewcommand{\abstractnamefont}{\normalfont\bfseries} -\setlength{\absleftindent}{1.2cm}\setlength{\absrightindent}{1.2cm} -\setlength{\parskip}{2pt} - -\pagestyle{fancy} -\fancyhf{} -\fancyhead[L]{\small\textsc{Preprint}} -\fancyhead[R]{\small Pareto Crewing} -\fancyfoot[C]{\small\thepage} -\renewcommand{\headrulewidth}{0.2pt} - -\newcommand{\E}{\mathbb{E}} -\newcommand{\R}{\mathbb{R}} -\newcommand{\N}{\mathbb{N}} -\newcommand{\roles}{\mathcal{R}} -\newcommand{\mods}{\mathcal{M}} -\newcommand{\crews}{\mathcal{K}} -\newcommand{\front}{\mathcal{F}} -\newcommand{\seat}[1]{\mathrm{#1}} -\newcommand{\code}[1]{\texttt{#1}} -\newcommand{\pclean}{P_{\mathrm{c}}} -\DeclareMathOperator*{\argmin}{arg\,min} -\DeclareMathOperator*{\argmax}{arg\,max} -\newtheorem{theorem}{Theorem} -\newtheorem{proposition}{Proposition} -\newtheorem{lemma}{Lemma} -\newtheorem{corollary}{Corollary} -\theoremstyle{definition} -\newtheorem{definition}{Definition} -\newtheorem{assumption}{Assumption} -\theoremstyle{remark} -\newtheorem{remark}{Remark} - -\title{\textbf{Pareto Crewing}\\[6pt] -\large Decentralised, privacy-preserving measurement of language-model teams,\\ and a learner that picks them on the cost-quality front, per role and per task} -\author{Santosh Kumar Radha} -\affil{AgentField AI} -\date{September 2026} - -\begin{document} -\maketitle -\thispagestyle{fancy} - -\begin{abstract} -\noindent -\emph{Pareto crewing} is the practice of picking the language-model team for a task on the cost-quality front, one model per role, from evidence about how models have performed in that role rather than from a fixed table. An agent harness runs a task through a \emph{crew}: an assignment of one language model to each of $n$ roles (in our instance a worker that does the work, a checker that reads it and a planner that cuts it). The crew that pays best is neither the cheapest nor the strongest on paper; it is the one that minimises the expected cost of one clean delivery once failed runs, wall time and shipped defects are priced. We give that objective in closed form, show how a crew is computed from a published catalog of prices and benchmark indexes alone (a pareto front in bill and quality, with a scale-free knee), and then how measurements from real runs replace the catalog's proxy. Runs are scored role by role by a model outside the crew. Each score becomes a row that carries no text, no file name, no price and no identity beyond a per-install nonce, so what an install sends is a handful of coarse fields about a model, not about the work; rows reduce to sufficient statistics that form a join-semilattice, so any number of installs and a relay can merge them in any order and reach the same state, the relay is a convenience rather than an authority, and the published index is signed so that every reader can check what it is blending in. Participation is a setting: an install can contribute, read and learn from its own runs without sending, or stay out. Judge severity is removed by a two-way fixed-effects fit on those statistics, drift by a local-level model whose forgetting rate is estimated rather than set, and heterogeneity between installs by a hierarchical prior whose variance is estimated by moments; every statistical constant is an empirical-Bayes estimate, and what remains for the user to set is the price of a defect and the price of time. The crew is chosen by Thompson sampling with semi-bandit feedback: because the objective is linear in role qualities and every role is scored separately, a run on one crew informs every crew that shares a model with it, and the sample complexity scales with the number of models rather than the number of crews. On a ground truth fitted to our own runs the learner beats a fixed table by a third of the remaining regret within a few hundred runs, and role-level feedback halves the regret of a crew-level learner fed the same scores at four hundred runs. -\end{abstract} - -\vspace{4pt} -\noindent\textbf{Keywords:} Pareto crewing, model selection, pareto front, empirical Bayes, Thompson sampling, combinatorial semi-bandits, CRDT, judge calibration, privacy, decentralised measurement. - -\section{Introduction} - -A harness that drives language models through software tasks has to decide, before every task, which model sits in which role. The decision is made under three kinds of information that arrive at different times and in different units: published prices and benchmark indexes, available for every model on the day it appears; the harness's own past runs, few but on the right distribution; and the runs of everyone else using the same harness, many but on other people's tasks and scored by other people's judges. This note is about turning the three into one decision. - -We call the resulting discipline \emph{Pareto crewing}: the crew is picked on the cost-quality front, per role and per task, from whatever evidence exists on the day, and the evidence is pooled across installs without any install giving up what it worked on. Our contributions are the following. -\begin{enumerate} -\item A closed-form objective for a crew, the expected cost of one clean delivery under reruns and a priced defect (Section~\ref{sec:objective}), and the observation that on a catalog alone the objective cannot be evaluated, which is why the catalog picker is a pareto front with a scale-free knee (Section~\ref{sec:catalog}). -\item A decentralised measurement of role quality (Section~\ref{sec:pool}): text-free rows, sufficient statistics that form a state-based CRDT (Proposition~\ref{prop:crdt}), judge fixed effects identified from the connectivity of the judge-cell graph (Proposition~\ref{prop:ident}), robust aggregation across installs, and drift handled by a local-level model. -\item An estimator with no tuned constant except the user's price of a defect (Section~\ref{sec:eb}): the catalog's prior weight, the between-install variance, the noise scale and the forgetting rate are all estimated from the pool by moments. -\item A picker (Section~\ref{sec:decide}) that is Thompson sampling in a combinatorial semi-bandit: the crew objective is linear in role qualities, roles are scored separately, so one run updates every crew sharing a model with it. -\item An empirical check on our own measured runs (Section~\ref{sec:exp}), reproducible from the script beside this document. -\end{enumerate} - -\section{Setting} -\label{sec:setting} - -\begin{definition}[Roles, crews] -Let $\roles$ be a finite set of $n$ \emph{roles} and $\mods$ a finite set of models. A \emph{crew} is a map $\kappa:\roles\to\mods$. Each role $r$ carries a \emph{shape}: a token share $v_r\ge 0$ with $\sum_r v_r=1$, index weights $w_r\in\Delta^{2}$, a cache share $\gamma_r\in[0,1]$, an input-to-output ratio $\rho_r>0$, and a predicate $\mathrm{fit}_r(m)$ saying whether model $m$ can hold the role (tools, images, context). A symmetric relation $D\subseteq\roles\times\roles$ names the pairs of roles that may not be held by models of the same vendor. -\end{definition} - -Our instance has $\roles=\{\seat{W},\seat{H},\seat{M}\}$ (worker, high, mastermind), token shares $(0.90,0.08,0.02)$, weights $(0.2,0.3,0.5)$, $(\tfrac13,\tfrac13,\tfrac13)$, $(0.6,0,0.4)$ on the indexes (intelligence, coding, agentic), cache shares $(0.75,0.75,0)$, ratios $(100,100,10)$ and $D=\{(\seat W,\seat H)\}$: the two roles that read the same work never share one shop. On the chat door the planner role is absent and $n=2$. Nothing below depends on $n=3$. - -A \emph{task} is one unit of work; a \emph{run} is one attempt at it by a crew. A run ends \emph{delivered} or not, and a delivered task is later found \emph{clean} (no major defect) or not. Money is in dollars and time is priced at $\lambda$ dollars per minute. - -The \emph{catalog} lists, per model, a prompt price $\pi_m$, a cache-read price $\pi^c_m$, a completion price $\kappa_m$ per million tokens, and benchmark indexes $I_m\in\R^3$. The \emph{pool} is the set of measurements described in Section~\ref{sec:pool}; the \emph{own sheet} is one install's part of it. - -\section{The crew computed from the catalog} -\label{sec:catalog} - -\paragraph{Role cost and role quality.} -Per million task tokens on role $r$, model $m$ costs -\begin{equation} -c_r(m) = (1-\gamma_r)\,\pi_m + \gamma_r\,\pi^{c}_m + \kappa_m/\rho_r , -\label{eq:seatcost} -\end{equation} -and its catalog quality is the weighted, pool-normalised index -\begin{equation} -\hat q_r(m) = 100 \sum_{i=1}^{3} w_{r,i}\,\frac{I_{m,i}}{\max_{m'\in\mods} I_{m',i}} . -\label{eq:catq} -\end{equation} -A missing index is imputed from the models that publish it: for each index the model does publish, the median over at least three donor models of the ratio between the missing and the present index gives an offer, the offers are averaged, and the estimate is capped at the largest measured value of the missing index in the pool; with fewer than three donors the model takes the standing its present indexes give it on the pool's scale. An imputed quality is flagged and loses ties. - -\paragraph{Shortlists.} -Role $r$'s shortlist is -\begin{equation} -S_r=\{m:\ \mathrm{fit}_r(m)\ \wedge\ \hat q_r(m)\ge \phi\max_{m'} \hat q_r(m')\},\qquad \phi=0.8, -\label{eq:floor} -\end{equation} -the floor being the statement that a model which cannot do the job is not cheap at any price. The crew set is $\crews=\{\kappa\in\prod_r S_r:\ \mathrm{vendor}(\kappa(r))\ne\mathrm{vendor}(\kappa(r'))\ \forall (r,r')\in D\}$. - -\paragraph{Bill, quality, front.} -\begin{equation} -B(\kappa)=\sum_{r} v_r\,c_r(\kappa(r)),\qquad Q(\kappa)=\sum_{r}\nu_r\, q_r(\kappa(r)),\qquad \nu_r\ge0,\ \textstyle\sum_r\nu_r=1, -\label{eq:crew} -\end{equation} -with $\nu$ uniform by default. The \emph{front} $\front\subseteq\crews$ is the set of crews not dominated in $(B,Q)$. Sorting $\crews$ by $(B\uparrow, Q\downarrow, \text{imputed count}\uparrow, \text{id}\uparrow)$ and keeping a crew when its $Q$ exceeds every kept crew's yields $\front$ in $O(|\crews|\log|\crews|)$, and the tie order makes $\front$ a function of the set $\crews$ rather than of any input order. In practice $|S_r|$ is a few dozen and $|\crews|$ a few thousand. - -\paragraph{Why a front and not a score.} -The objective of Section~\ref{sec:objective} needs a probability of clean delivery. The catalog has none, so any scalarisation of $(B,Q)$ would fix an exchange rate between a price and a benchmark index that nobody has measured. Dominance is the only order that needs no such rate; every crew a user could sensibly want is on $\front$, which is small, and the remaining choice is one-dimensional along it. - -\paragraph{Knee and knob.} -Index $\front$ by bill, $B_0\le\dots\le B_1$. The three named picks are \emph{frugal} $=\argmin_\front B$, \emph{max} $=\argmax_\front Q$, and \emph{balanced}, the crew farthest above the chord of the front in $(\ln B,Q)$: -\begin{equation} -\text{balanced}=\argmax_{\kappa\in\front}\ \Big[(Q(\kappa)-Q_0) - (Q_1-Q_0)\frac{\ln B(\kappa)-\ln B_0}{\ln B_1-\ln B_0}\Big]. -\label{eq:knee} -\end{equation} -\begin{proposition}[Scale invariance] -The balanced crew is invariant under $B\mapsto aB$ for any $a>0$ and under $Q\mapsto Q+b$. -\end{proposition} -\begin{proof} -$\ln(aB)-\ln(aB_0)=\ln B-\ln B_0$ and $\ln(aB_1)-\ln(aB_0)=\ln B_1-\ln B_0$; the bracket is unchanged. A shift of $Q$ cancels in both differences. -\end{proof} -The log scale is not cosmetic: catalog prices span two orders of magnitude, and the knee should not move when a currency changes or every vendor cuts prices by the same factor. A knob $k\in[0,1]$ picks the best crew with $B\le B_0(B_1/B_0)^k$, so equal steps in $k$ are equal ratios in the bill. - -\section{The objective: the cost of one clean delivery} -\label{sec:objective} - -Let a run of crew $\kappa$ cost $c$ dollars and $t$ minutes, deliver with probability $r$, and let a delivered task be clean with probability $p$; a run that fails to deliver costs $c_f$ and $t_f$. - -\begin{lemma}[Rerun until clean] -If every defect is caught and the task rerun, the expected cost to one clean delivery is -\begin{equation} -E_0(\kappa)=\frac{(1-r)(c_f+\lambda t_f)+r\,(c+\lambda t)}{r\,p}. -\label{eq:E0} -\end{equation} -\end{lemma} -\begin{proof} -Attempts are i.i.d.\ with success probability $rp$; the count to the first success is geometric with mean $1/(rp)$, and by Wald's identity the expected total cost is the mean count times the expected cost of one attempt, which is the numerator. -\end{proof} - -\begin{lemma}[Priced defect] -If a shipped defect costs $H$ to find and mend and only non-deliveries are rerun, -\begin{equation} -E_H(\kappa)=\frac{(1-r)(c_f+\lambda t_f)}{r} + (c+\lambda t) + (1-p)\,H . -\label{eq:EH} -\end{equation} -\end{lemma} -\begin{proof} -The count of failed attempts before the first delivery is geometric on $\{0,1,\dots\}$ with mean $(1-r)/r$; the delivering attempt costs $c+\lambda t$; the defect is paid once with probability $1-p$. -\end{proof} - -$H$ is the one quantity that belongs to the user, and the budget word sets it: frugal $H=\$1$, balanced $\$10$, max $\$50$. Both forms are monotone in the same three quantities, the run's cost, its time and $\pclean=rp$, and the picker works with -\begin{equation} -E(\kappa)=c(\kappa)+\lambda\,t(\kappa)+H\,\big(1-\pclean(\kappa)\big). -\label{eq:E} -\end{equation} -Equation~\eqref{eq:E} is~\eqref{eq:EH} with $r$ set to one: a run that fails to deliver is priced as a shipped defect rather than rerun. The surrogate ranks crews as $E_H$ does when $H$ is large next to a run's cost and $r$ is not far below one, which is the regime of every crew on the front of Figure~\ref{fig:front}; when a role's delivered rate is measured, as the worker's is, the picker can use~\eqref{eq:EH} directly and nothing below changes. Figure~\ref{fig:front} draws our measured crews with the iso-lines of~\eqref{eq:E}. The knob $k$ and the price $H$ parametrise the same trade-off; $k$ is what a catalog admits, $H$ is what measurements admit. - -\begin{assumption}[Linearity in roles] -\label{ass:linear} -$\pclean(\kappa)=\sum_r \nu_r\,q_r(\kappa(r))/100$, where $q_r(m)$ is the pool's role quality: the run's outcome credited to the role, on a 0--100 scale. -\end{assumption} -The assumption is what makes~\eqref{eq:E} linear in the role qualities, which Section~\ref{sec:decide} uses. It is a working rule rather than a law: a product over roles would count one outcome $n$ times, and a fitted interaction becomes available once judged scores separate the roles' contributions. Nothing in Sections~\ref{sec:pool}--\ref{sec:eb} depends on it. - -\begin{figure}[t] -\centering\includegraphics{fig-front.pdf} -\caption{Our measured crews: 56 (worker, checker or planner) combinations, each fitted by a ridge model on 72 scored runs over nine tasks; cost of one run against the probability that it delivers clean. Solid: the pareto front. Dashed: iso-cost lines of~\eqref{eq:E}; a larger $H$ tilts them so that quality buys more.} -\label{fig:front} -\end{figure} - -\section{The Model Pool: a decentralised measure of role quality} -\label{sec:pool} - -\subsection{Judged runs and rows} - -After a run finishes, one model outside the crew reads the record (brief, deliverable, the checker's report and claim, the files touched) and answers one question per role with a score in $[0,100]$ and one sentence. The judge is chosen deterministically: the cheapest catalog model with coding index at least $60$ that can call tools, holds no role in the crew, and is not from the worker's vendor; a role the judge's own model holds is skipped. The floor is absolute, not relative to the worker; its reason is that a judge well below the class of the work it reads rubber-stamps. The judge's calls are billed to a role of their own, so learning is never free by accident; the cost is one short completion per role. - -One judged role of one run is one \emph{row} -\[ -x=(\text{role},\ \text{model},\ \text{score},\ \text{judge},\ \text{door},\ \text{size}\in\{S,M,L\},\ \text{day}), -\] -sent under an \emph{install} nonce drawn once per profile and carried on the batch, not in the row. No text, no price and no duration is in a row; dollars and minutes stay in the install's own ledger. Rows wait in a local outbox, which can be inspected, until they are sent. - -\subsection{Sufficient statistics and order-free merging} - -\begin{definition}[Cell, sheet] -A \emph{cell} is a tuple $\theta=(\text{role},\text{model},\text{judge},\text{door},\text{size})$. A \emph{sheet} is a map from (install, day, cell) to a triple $T=(n,S,S^2)\in\N\times\R_{\ge0}\times\R_{\ge0}$, the count, sum and sum of squares of the scores an install has produced in that cell. -\end{definition} - -Every quantity computed below is a function of the sheet. Sheets are merged by the join -\begin{equation} -(A\sqcup B)(u,d,\theta)=\big(\max(n_A,n_B),\ \max(S_A,S_B),\ \max(S^2_A,S^2_B)\big)\quad\text{componentwise on } A(u,d,\theta),\,B(u,d,\theta), -\label{eq:join} -\end{equation} -with a missing entry read as $(0,0,0)$. - -\begin{proposition}[The sheet is a state-based CRDT] -\label{prop:crdt} -Let each install $u$ only ever publish its own entries $(u,d,\theta)$ and let every published triple be the running total of $u$'s rows in $\theta$ on day $d$. Then (i) $\sqcup$ is commutative, associative and idempotent; (ii) an install's triples are monotone non-decreasing in time, so a later publication dominates an earlier one under $\sqcup$; (iii) any two nodes that have received the same set of publications, in any order and with any duplication, hold the same sheet. -\end{proposition} -\begin{proof} -(i) is the componentwise max on a product of totally ordered sets. (ii) Scores lie in $[0,100]$, so $n$, $S$ and $S^2$ each grow by a non-negative amount per row. (iii) follows from (i) and (ii): the join of a set of publications is their least upper bound, which does not depend on order or multiplicity, and by (ii) the least upper bound of an install's publications is its latest. -\end{proof} - -The cell means and pooled variances are functions of $\sum_u T(u,\theta)$, so by Proposition~\ref{prop:crdt} a relay and every install compute the same estimates from the same publications; there is no coordinator whose order of arrival matters, and a relay that is lost is rebuilt from any node that kept the publications. This is the sense in which the measure is decentralised: the relay is a convenience for bandwidth and signing, not a source of truth. Two kinds of merge exist, and the join is the law for one of them. Within a store, rows are folded by addition: the install's own sheet (\code{internal/pool/tally}) adds each of its rows once, as it is judged, and the reference relay adds each row it receives into the sender's triple for that cell and day. Addition is not idempotent, so a batch retried after a lost acknowledgement can be counted twice there; the relay bounds that by its daily row quota per install, and a relay that wants exactness keeps the batch's nonce. Between stores, when a relay, a mirror or an install holds a copy of another's sheet, the join~\eqref{eq:join} is the merge, and Proposition~\ref{prop:crdt} is what makes copies converge whatever the order. - -\subsection{Judges are not the same ruler} - -Let $y$ be a score of role $r$, model $m$, by judge $j$. We posit the two-way additive model -\begin{equation} -y=\mu_{rm}+\beta_j+\varepsilon,\qquad \E\varepsilon=0,\ \operatorname{Var}\varepsilon=\sigma^2, -\label{eq:twoway} -\end{equation} -with $\mu_{rm}$ the quantity of interest and $\beta_j$ the judge's severity, constrained by $\sum_j n_{\cdot\cdot j}\beta_j=0$ (the judges' weighted mean is the ruler). Weighted least squares needs only the cell means $\bar y_{rmj}=S_{rmj}/n_{rmj}$ and counts. The normal equations are solved by alternating -\begin{equation} -\beta_j\leftarrow\frac{\sum_{r,m} n_{rmj}(\bar y_{rmj}-\mu_{rm})}{\sum_{r,m} n_{rmj}},\qquad -\mu_{rm}\leftarrow\frac{\sum_{j} n_{rmj}(\bar y_{rmj}-\beta_j)}{\sum_{j} n_{rmj}}, -\label{eq:als} -\end{equation} -re-centring $\beta$ after each sweep. Each half-step minimises a convex quadratic in its block, so the residual is non-increasing and the iteration converges to the least-squares solution; five sweeps suffice in practice. - -\begin{proposition}[Identifiability] -\label{prop:ident} -Let $G$ be the bipartite graph with the judges on one side, the pairs $(r,m)$ on the other, and an edge where $n_{rmj}>0$. The parameters of~\eqref{eq:twoway} are identified up to one additive constant per connected component of $G$: within a component every contrast is identified, and one centring constraint per component fixes its levels. When $G$ is connected the single constraint identifies everything. -\end{proposition} -\begin{proof} -The design matrix of a two-way additive model has rank equal to (number of vertices of $G$) minus (number of connected components), and its kernel is spanned by the vectors that shift $\mu$ up and $\beta$ down by one constant on one component; one linear constraint per component removes them. With the single centring constraint, exactly one component is fully identified. -\end{proof} - -The reference relay centres $\beta$ by one weighted mean, which fixes the levels only when $G$ is connected; on a disconnected graph the between-component offsets are a convention of the fit, and the index says which judges each cell was scored by so a reader can tell. The judge rule of Section~5.1 makes $G$ connected quickly in practice: it sends most installs to the same few cheap judges. A model scored by a single judge so far has $\mu_{rm}$ equal to its own mean and the index says so. - -\subsection{Installs are not independent, and some are wrong} - -Rows from one install share a task mix, a codebase and a way of writing briefs. Under an intra-class correlation $\varrho$, $n_u$ rows from install $u$ carry the information of $n_u/(1+(n_u-1)\varrho)$ independent ones, which saturates at $1/\varrho$. The pool's cell estimate therefore treats installs, not rows, as the unit: it forms the install means $\bar y_{u\theta}$ and combines them with a location estimator that is robust to a minority of installs that are broken or adversarial. We use the Huber $M$-estimate with scale $s=1.4826\cdot\mathrm{MAD}$ and the conventional tuning constant $1.345$ (95\% efficiency at the normal), which has breakdown point $1/2$ in location, and fall back to the precision-weighted mean when a cell has fewer than five installs. A cell backed by fewer than three installs is published but not read. - -\subsection{Drift} - -Models change under the same name. Let $\bar y_d$ be a cell's day-$d$ mean over $n_d$ rows and model the true level as a random walk, -\begin{equation} -\mu_d=\mu_{d-1}+\eta_d,\quad \eta_d\sim\mathcal N(0,\omega^2);\qquad \bar y_d=\mu_d+\bar\varepsilon_d,\quad \bar\varepsilon_d\sim\mathcal N(0,\sigma^2/n_d). -\label{eq:localevel} -\end{equation} -The Kalman filter for this local-level model is, in steady state with constant $n_d=\bar n$, an exponentially weighted average whose forgetting factor $1-\alpha$ solves $\alpha^2/(1-\alpha)=\bar n\omega^2/\sigma^2$; the effective memory is about $1/\alpha$ days. The drift variance is estimated by moments from the day-to-day differences, -\begin{equation} -\hat\omega^2=\Big[\widehat{\operatorname{Var}}(\bar y_d-\bar y_{d-1})-\sigma^2\big(\tfrac1{n_d}+\tfrac1{n_{d-1}}\big)\Big]_+, -\label{eq:drift} -\end{equation} -pooled across cells. A cell with no evidence of drift ($\hat\omega^2=0$) keeps everything; a cell that drifts forgets at the rate the data show. The day is a cell dimension for exactly this reason. - -\begin{figure}[t] -\centering -\begin{tikzpicture}[x=1cm,y=1cm, - box/.style={draw,rounded corners=2pt,inner sep=4pt,font=\small,align=center,minimum height=8mm}, - store/.style={box,fill=black!6}, - net/.style={box,fill=blue!6}, - arr/.style={-{Latex[length=2mm]},thick}, - lbl/.style={font=\scriptsize,fill=white,inner sep=1.5pt} -] -\node[box] (cat) at (1.4,0) {catalog\\prices, indexes}; -\node[store] (post) at (4.7,0) {posterior per\\(role, model)}; -\node[box] (ts) at (8.1,0) {Thompson draw\\$\argmin_\kappa \tilde E(\kappa)$}; -\node[box] (run) at (10.7,0) {run}; -\node[box] (judge) at (13.3,0) {judge\\outside the crew}; -\node[store] (outbox) at (10.7,-1.6) {outbox}; -\node[store] (own) at (13.3,-1.6) {own sheet}; -\node[store] (index) at (4.7,-4.1) {signed index\\(mirrored)}; -\node[net] (relay) at (9.4,-4.1) {relay: fold, judge effects \eqref{eq:als},\\Huber over installs, sign}; -\node[box,dashed] (others) at (13.3,-4.1) {other installs}; -\draw[arr] (cat) -- (post); -\draw[arr] (post) -- (ts); -\draw[arr] (ts) -- (run); -\draw[arr] (run) -- (judge); -\draw[arr] (judge) -- (own); -\draw[arr] ([xshift=-7mm]judge.south) |- (outbox.east); -\draw[arr] (own.south) -- ++(0,-0.45) -| node[lbl,pos=0.12,below]{at once} (post.south); -\draw[arr] (outbox.south) -- node[lbl,right]{rows, no text} (outbox.south |- relay.north); -\draw[arr] (relay) -- (index); -\draw[arr] (index.north) -- node[lbl,pos=0.35,right]{read at start} (post.south); -\draw[arr] (others.west) -- node[lbl,above]{rows} (relay.east); -\draw[arr] (index.south) -- ++(0,-0.5) -| node[lbl,pos=0.25,below]{the same index} (others.south); -\draw[dotted,thick] (-0.1,0.75) rectangle (14.9,-2.55); -\node[font=\scriptsize] at (0.9,0.95) {one install}; -\end{tikzpicture} -\caption{Where each step runs. Inside the dotted box is one install; the only thing that crosses it outward is a row without text. The relay merges publications with the join of Proposition~\ref{prop:crdt} and publishes a signed index; any install can host a relay, and any node that kept the publications can rebuild one.} -\label{fig:flow} -\end{figure} - -\section{Estimation with no tuned constant} -\label{sec:eb} - -We now combine the three sources for one install $u$. Write $\hat q_r(m)$ for the catalog quality~\eqref{eq:catq}, $\bar y^{\mathrm{pool}}_{rm}$ and $n^{\mathrm{pool}}_{rm}$ for the pool's cell after Sections~5.3--5.5, and $\bar y^{u}_{rm}$, $n^{u}_{rm}$ for the own sheet. - -\begin{assumption}[Hierarchy] -\begin{equation} -\mu_{rm}\sim\mathcal N\big(\hat q_r(m)+b,\,\tau_0^2\big),\qquad -\mu^{u}_{rm}\mid\mu_{rm}\sim\mathcal N\big(\mu_{rm},\,\tau_1^2\big),\qquad -y\mid\mu^{u}_{rm}\sim\mathcal N\big(\mu^{u}_{rm},\,\sigma^2\big). -\label{eq:hier} -\end{equation} -\end{assumption} -The first line says the catalog predicts the population quality up to one offset $b$ shared by every model (the catalog's scale is not the judge's) and a noise $\tau_0$; the second that install $u$'s own quality for the model scatters around the population's; the third is judge noise after Section~5.3. The install cares about $\mu^u_{rm}$. - -\begin{proposition}[Posterior for the install] -\label{prop:post} -Under~\eqref{eq:hier}, with $k=\sigma^2/\tau_0^2$, $\ell=\sigma^2/\tau_1^2$ and $\tilde n=n^{\mathrm{pool}}\,\ell/(\ell+1)$ (pool rows are other installs' rows, so each carries variance $\sigma^2+\tau_1^2$ about $\mu_{rm}$), the posterior of $\mu^u_{rm}$ is normal with -\begin{equation} -\E[\mu^u_{rm}\mid\cdot]=\frac{k^{\ast}\,(\hat q_r(m)+b) + n^{\ast}\,\bar y^{\mathrm{pool}}_{rm} + n^u\,\bar y^u_{rm}}{k^{\ast}+n^{\ast}+n^u},\qquad -\operatorname{Var}[\mu^u_{rm}\mid\cdot]=\frac{\sigma^2}{k^{\ast}+n^{\ast}+n^u}, -\label{eq:three} -\end{equation} -where the catalog and the pool share one \emph{capped} external count $W$ in proportion to what each contributed, -\begin{equation} -W=\frac{K\,\ell}{K+\ell}\ \le\ \ell,\qquad K=k+\tilde n,\qquad -k^{\ast}=\frac{k}{K}\,W,\qquad n^{\ast}=\frac{\tilde n}{K}\,W . -\label{eq:cap} -\end{equation} -\end{proposition} -\begin{proof} -Integrate out $\mu_{rm}$. The catalog and the pool inform $\mu^u_{rm}$ only through $\mu_{rm}$, whose posterior given them is normal with precision $K/\sigma^2$, $K=k+\tilde n$, and mean the $k:\tilde n$ precision-weighted average of $\hat q_r(m)+b$ and $\bar y^{\mathrm{pool}}_{rm}$; adding the between-install variance $\tau_1^2$ gives, for $\mu^u_{rm}$, a precision $1/(\sigma^2/K+\tau_1^2)=W/\sigma^2$ with the same mean. The own rows add precision $n^u/\sigma^2$ directly. Precision-weighting the two gives~\eqref{eq:three}, with the external count $W$ split as $k^{\ast}+n^{\ast}=W$ in the ratio $k:\tilde n$. -\end{proof} - -\begin{corollary} -With $\tau_1=0$ ($\ell=\infty$) the posterior mean is $(1-w)\hat q_r(m)+w\bar y$ with $w=n/(n+k)$, $n=n^{\mathrm{pool}}+n^u$: the plain shrinkage blend, which is what the picker applies today with $k=30$. -\end{corollary} - -The cap~\eqref{eq:cap} is the point of the hierarchy. However many rows the pool holds, everything external to this install, catalog and pool together, is worth at most $\ell=\sigma^2/\tau_1^2$ own rows, because other installs' tasks are not this install's tasks; and the own rows are never capped. This replaces an ad hoc "own rows count triple" by a quantity with a meaning and an estimator. The pick shipped today is the Corollary's form: the pool's and the install's cells are folded together by observation count against the catalog at $k=30$, which is~\eqref{eq:three} at $\ell=\infty$ and $b=0$. - -\paragraph{Estimating the constants.} -Every constant in~\eqref{eq:three} is estimated from the sheet by moments; none is set by hand. -\begin{align} -\hat\sigma^2 &= \frac{\sum_{u,\theta}\big(S^2_{u\theta}-S_{u\theta}^2/n_{u\theta}\big)}{\sum_{u,\theta}(n_{u\theta}-1)} &&\text{pooled within-cell variance,}\label{eq:sigma}\\ -\hat\tau_1^2 &= \Big[\,\widehat{\operatorname{Var}}_u\big(\bar y_{u\theta}\big)-\hat\sigma^2\,\overline{(1/n_{u\theta})}\,\Big]_+ &&\text{between-install variance, pooled over cells,}\label{eq:tau1}\\ -\hat b &= \overline{\bar y^{\mathrm{pool}}_{rm}-\hat q_r(m)} &&\text{the catalog's offset,}\label{eq:bias}\\ -\hat\tau_0^2 &= \Big[\,\widehat{\operatorname{Var}}_{(r,m)}\big(\bar y^{\mathrm{pool}}_{rm}-\hat q_r(m)\big)-\overline{(\hat\sigma^2+\hat\tau_1^2)/n^{\mathrm{pool}}_{rm}}\,\Big]_+ &&\text{how far the catalog misses,}\label{eq:tau0} -\end{align} -each a within-minus-noise moment estimator of the kind used by Efron and Morris~\cite{efron1975}. Then $k=\hat\sigma^2/\hat\tau_0^2$ and $\ell=\hat\sigma^2/\hat\tau_1^2$, and $\hat\omega^2$ of~\eqref{eq:drift} sets the forgetting. On the seed index alone the estimator already says something: its thirteen cells with a published intelligence index give $\hat b=-13.7$ (the judged scale sits 14 points below the catalog's) and a variance of the residuals, 109, below the noise they carry, 167, so $\hat\tau_0^2$ clips to zero: thirteen cells of bit-valued scores ($\hat\sigma\approx43$) cannot resolve a catalog error under about 13 points, and the day-one $k=30$ stands until judged scores, with their smaller $\sigma$, replace the bits. The design publishes them in the index beside the cells so that every install blends with the same constants, re-estimated at every relay cycle and withheld until each denominator has at least ten degrees of freedom, with the day-one values of Table~\ref{tab:const} until then; the index on this head carries the cells and the day-one constants, and the estimators are the next thing the relay's job runs. - -\subsection{Covariates, and why no row is ever wasted} -\label{sec:covariates} - -A row carries more than (role, model, score): a door, a size bucket, a judge, a day, an install, and any field added later, such as the harness version or a task family. Call the vector of such fields the row's \emph{profile} $c\in\mathcal C=\prod_f L_f$, a product of factor levels. The hierarchy~\eqref{eq:hier} extends factor by factor: -\begin{equation} -\mu_{rm,c}=\mu_{rm}+\sum_{f}\delta^{f}_{rm,c_f},\qquad \delta^{f}_{rm,\cdot}\sim\mathcal N(0,\tau_f^2)\ \text{i.i.d.\ over levels}, -\label{eq:factors} -\end{equation} -so that a (role, model) has one population quality and one partially pooled offset per level of each factor. The install factor is the case $f=\text{install}$ of~\eqref{eq:factors}, and Proposition~\ref{prop:post} is its two-level instance; the same derivation gives, for a query profile $c$, a posterior mean that is a precision-weighted combination of the cell means at every level that shares a coordinate with $c$, each capped by its own $\ell_f=\sigma^2/\tau_f^2$, with $\hat\tau_f^2$ estimated by the between-level moment~\eqref{eq:tau1} applied to factor $f$. Two properties follow. - -\emph{Old rows keep their value when the harness changes.} If the door algorithm is rewritten, rows from the old door are a level of the door factor; they still inform $\mu_{rm}$ with the full weight the estimated $\tau_{\text{door}}^2$ allows, and the new door starts at $\mu_{rm}$ plus nothing and specialises as its own rows arrive. If the two doors turn out to differ little, $\hat\tau_{\text{door}}^2\approx 0$, $\ell_{\text{door}}\to\infty$, and the factor pools completely; if they differ a lot, the cap keeps the old door from speaking for the new one. The data decide, per factor, how much they transfer. - -\emph{The schema is open.} A row with a field absent (older rows, before the field existed) is at a level of its own, \emph{unknown}, which the estimator treats like any other level. Adding a field therefore invalidates nothing already collected, and the sheet's join in Proposition~\ref{prop:crdt} is unaffected because a cell key is only a tuple. What can be extracted from the rows is the set of contrasts the factors expose; a factor that is never recorded cannot be recovered later, which is why the row carries every cheap field now and the text of none. - -Regressors that are not factors, a task's token count for instance, enter the same way as a linear term with a partially pooled slope; the moment estimators are the same with a design column in place of an indicator. - -\paragraph{Interactions: the crew as a group.} -Assumption~\ref{ass:linear} scores roles one at a time. Whether a particular worker and a particular checker do better \emph{together} than their role qualities predict is a question about the pair, and the same hierarchy answers it: a pair $(m_r,m_{r'})$ is a level of a factor over $\mods\times\mods$, with its own partially pooled offset $\delta^{rr'}_{(m_r,m_{r'})}\sim\mathcal N(0,\tau_{rr'}^2)$. A pair that has never been run sits at zero offset and the additive prediction stands; a pair with rows moves by exactly the amount its rows justify against $\hat\tau_{rr'}^2$; if pairs in general add nothing, $\hat\tau_{rr'}^2\to0$ and the term disappears. This is the whole difference between learning crews as groups and learning roles: the group term is there, but it is charged for. A crew-level learner with no structure has to visit each of $|\crews|$ crews to say anything about it; the additive model needs $\sum_r|S_r|$ observations to say something about every crew, and the pair term then spends further observations only where a pair is actually run. Some group effects are constraints rather than parameters: the vendor rule $D$ is one, and it is applied to $\crews$ before any learning happens. - -\begin{table}[h] -\centering\small -\begin{tabular}{p{2.0cm}p{5.2cm}p{4.0cm}p{2.6cm}}\toprule -constant & meaning & estimator & day-one value\\\midrule -$\sigma$ & judge noise & \eqref{eq:sigma} & 15\\ -$k=\sigma^2/\tau_0^2$ & weight of the catalog & \eqref{eq:tau0} & 30\\ -$\ell_f=\sigma^2/\tau_f^2$ & cap on evidence transferred across a factor (install, door, size, \dots) & \eqref{eq:tau1} per factor & 30\\ -$\omega^2$ & drift per day & \eqref{eq:drift} & 0 (no forgetting)\\ -$\beta_j$ & judge severity & \eqref{eq:als} & 0\\ -$v_r$ & role token shares & measured from run token accounting & $0.90/0.08/0.02$\\ -\midrule -$H$ & price of a shipped defect & the user's, by budget word & \$1 / \$10 / \$50\\ -$\lambda$ & price of a minute & the user's & \$0.01\\ -$\phi$, judge floor & structural floors & fixed by design & $0.8$, $60$\\ -\bottomrule\end{tabular} -\caption{Every constant with its estimator. Two are the user's by design; two are structural floors. The rest are empirical-Bayes estimates that the relay publishes with the index.} -\label{tab:const} -\end{table} - -\begin{figure}[t] -\centering\includegraphics{fig-shrink.pdf} -\caption{Shrinkage of one role quality from the catalog prior (80) toward the measured mean (65), $k=30$, $\sigma=15$. The band is what the Thompson draw of Section~\ref{sec:decide} samples from.} -\label{fig:shrink} -\end{figure} - -\section{Deciding: Thompson sampling with semi-bandit feedback} -\label{sec:decide} - -Given a posterior $\mathcal N(\mu'_{rm},\sigma'^2_{rm})$ per role and model from~\eqref{eq:three}, the pick could minimise~\eqref{eq:E} at the posterior means. That never tries a crew the prior underrates, and the catalog prior is wrong by a lot for some models (Figure~\ref{fig:cells}). Thompson sampling~\cite{thompson1933,russo2018} draws one plausible world from the posterior and acts as if it were true: a crew is chosen with the probability that it is the best, exploration is proportional to what is unknown, and it stops on its own as the posteriors tighten. No exploration rate exists to be set. - -\begin{algorithm}[h] -\caption{One pick under \emph{learn}} -\begin{algorithmic}[1] -\Require shortlists $S_r$ on the posterior means, budget word $\Rightarrow H$, task size bucket $\Rightarrow$ expected tokens $T$ and minutes $t(\kappa)$ -\For{each role $r$ and $m\in S_r$} - \State $\tilde q_{rm}\sim\mathcal N(\mu'_{rm},\sigma'^2_{rm})$ \Comment{one draw per run; never redrawn during the run} -\EndFor -\For{each $\kappa\in\crews$} - \State $\tilde E(\kappa)\gets T\,B(\kappa)+\lambda\,t(\kappa)+H\,\big(1-\sum_r\nu_r\tilde q_{r\kappa(r)}/100\big)$ -\EndFor -\State \Return $\argmin_\kappa\tilde E(\kappa)$ \Comment{ties: fewer imputed indexes, then id order} -\end{algorithmic} -\end{algorithm} - -\begin{proposition}[Role draws are crew draws] -Under Assumption~\ref{ass:linear}, drawing $\tilde q_{rm}$ independently per (role, model) and minimising $\tilde E$ over $\crews$ is Thompson sampling over the crews' posterior costs under the product posterior on role qualities. -\end{proposition} -\begin{proof} -$\tilde E(\kappa)$ is an affine function of the vector $(\tilde q_{rm})$, so its joint law over $\crews$ is the push-forward of the posterior on role qualities; a sample of the latter is a sample of the former. The judge-effect fit of Section~5.3 and the drift model of Section~5.5 induce dependence between cells that the independent draw ignores; the draw is exact for the product approximation the index publishes. -\end{proof} - -The structure that matters is the feedback. A run of crew $\kappa$ is judged role by role, so it yields one observation for each of its $n$ (role, model) pairs, and the posterior of every crew that shares a pair with $\kappa$ moves. This is a combinatorial bandit with semi-bandit feedback~\cite{chen2013,wen2015}: the unknowns are the $\sum_r|S_r|$ role qualities, not the $|\crews|$ crew values, and the regret of Thompson sampling scales with the former~\cite{wen2015}. Section~\ref{sec:exp} measures the difference on our runs. For Gaussian posteriors the Bayesian regret of Thompson sampling is $O(\sqrt{dT\log T})$ in the dimension $d$ of the unknowns over $T$ runs~\cite{russo2014}. - -\paragraph{The learned front.} -Under \emph{learn} the front of Section~\ref{sec:catalog} is recomputed on the posterior: the bill $B(\kappa)$ is read at the run's expected tokens, and the quality is the posterior mean of $\pclean(\kappa)$, so the front is the pareto set in (expected dollars per task, expected probability of a clean delivery) with an interval on each point from~\eqref{eq:three}. The budget word's $H$ is a line of slope $-1/H$ in that plane and the pick is where the line first touches the front; Algorithm~1 is the same pick on one draw of the front rather than on its mean, which is what makes it explore. The pool command prints the mean front with its intervals, and the catalog front beside it, so that a user can see where measurement moved a pick and by how much. - -\paragraph{When is it settled.} -The same posterior answers whether learning is done: the probability $\pi^\ast$ that the posterior-mean crew is the best, estimated by repeating the draw of Algorithm~1 a few hundred times. The pool command reports $\pi^\ast$; a value near one means further exploration is nearly free, a value near $1/|\front|$ means the ranking is still open. Costs enter as observed dollars per run and are not sampled; their noise is small next to the quality's. - -\begin{remark}[A new model] -A model with no rows has $n=0$ and sits at its catalog prior with the prior variance $\sigma^2/k^{\ast}$. If that puts it near the front it is drawn into crews soon, judged, and confirmed or pushed down by its first rows. Nothing is done by hand when the catalog changes. -\end{remark} - -\section{Experiments on our measured runs} -\label{sec:exp} - -Ground truth is the table of Figure~\ref{fig:front}: 56 crews with $\pclean$, dollars and minutes fitted by a ridge model on 72 scored runs of our own over nine tasks. Each replicate draws a catalog prior per arm as the truth plus $\mathcal N(0,0.15^2)$ noise, roughly the disagreement between indexes and measurements in Figure~\ref{fig:cells}, holds it with $k=30$, and runs $N$ tasks; a strategy picks, observes the outcome and pays~\eqref{eq:E} with $H=\$10$. Regret is the excess over the oracle crew, averaged over 200 replicates. - -\paragraph{Crew-level learners.} -Table~\ref{tab:regret} and Figure~\ref{fig:regret} compare, on all 23 chat-door crews of the table, a fixed table row (the shipped balanced crew), the catalog prior alone, $\varepsilon$-greedy and Thompson, all with one arm per crew. After 400 runs Thompson is \$0.33 per task under the fixed table, a third of the table's regret, and still falling; $\varepsilon$-greedy is \$0.12 behind it. Thompson pays an exploration tax in its first forty runs, when it is no better than the table; $k=30$ is what bounds it. $\varepsilon$-greedy pays more, because a tenth of its runs go to a random crew among 23, and its rate neither grows when a new model appears nor shrinks when the ranking is settled. - -\begin{table}[h] -\centering\small -\input{tab-regret.tex} -\caption{Regret against the oracle, dollars per task after $N$ runs, one arm per crew.} -\label{tab:regret} -\end{table} - -\begin{figure}[h] -\centering\includegraphics{fig-regret.pdf} -\caption{Left: regret per task. Right: share of runs on a crew other than the current posterior-mean best. Thompson explores heavily while many crews sit within cents of each other and less as the posteriors separate; $\varepsilon$-greedy explores at its fixed rate.} -\label{fig:regret} -\end{figure} - -\paragraph{Role-level feedback.} -Figure~\ref{fig:semibandit} and Table~\ref{tab:semibandit} isolate the structure of Section~\ref{sec:decide}. The ground truth is three-role: a role quality per (role, model) read off the measured rows (a worker's delivered rate, a checker's and a planner's clean rate), combined by the linear rule of Assumption~\ref{ass:linear} over every crew the vendor rule admits, with dollars additive over roles. Three learners run Thompson sampling with the same prior noise and the same $H$ and differ only in what they see after a run: one arm per crew fed the run's clean-or-defect bit; one arm per crew fed the mean of the judge's role scores; one posterior per (role, model) fed each role's own score, which is what the judge of Section~5.1 provides. The last is the semi-bandit. Richer feedback per run helps: the crew-level learner fed the mean score sits at \$0.31 after four hundred runs against \$0.54 for the one fed only the clean bit. The structure helps again as much: the role-level learner, fed the same scores as the second, sits at \$0.15, half of it and a third of the first, because every run updates the crews that share a model with the one that ran, and there are sixteen unknowns behind 138 crews. - -\begin{table}[h] -\centering\small -\input{tab-semibandit.tex} -\caption{Regret against the oracle, dollars per task after $N$ runs, on the three-role truth; same prior noise and $H$ for all three learners.} -\label{tab:semibandit} -\end{table} - -\begin{figure}[h] -\centering\includegraphics{fig-semibandit.pdf} -\caption{Thompson sampling under three kinds of feedback on the same three-role truth. Dotted and dashed: one arm per crew, fed the clean bit and the mean role score respectively. Solid: one posterior per (role, model) fed the role scores, the combinatorial semi-bandit of Section~\ref{sec:decide}.} -\label{fig:semibandit} -\end{figure} - -\begin{figure}[t] -\centering\includegraphics{fig-cells.pdf} -\caption{The seed index shipped in the binary: 15 cells from 194 scored runs, \code{role\_quality} $=100\cdot P(\text{no major defect})$ under a Beta(1,1) posterior. The spread between models a catalog rates alike is why the pool exists.} -\label{fig:cells} -\end{figure} - -The truth in these experiments is our own fitted table from one team's tasks, so the size of the gain is the shape of the effect and not a promise for every install; the regret scale is set by how close the crews sit, which the front of Figure~\ref{fig:front} shows and no bound here uses. The script \code{figures.py} beside this document reproduces every figure and table from the two CSV files it reads and the constants written at its head. - -\section{System architecture} -\label{sec:arch} - -The statistics of Sections~\ref{sec:pool}--\ref{sec:decide} fix what the system computes; this section fixes where the bytes live, so that the privacy claims can be checked against the deployment rather than taken on trust. Figure~\ref{fig:arch} shows the four components. The design rule is that the relay holds nothing worth stealing and nothing that cannot be rebuilt. - -\paragraph{The binary.} -Everything that touches a task runs inside the install: the catalog read, the posterior, the draw, the run, the judge call, the own sheet and the outbox. The judge is an ordinary model call through the install's own provider credentials; the record it reads never leaves the machines the install already trusts with its work. The outbox is a newline-delimited file under the profile directory; \code{codeaf pool status} prints its pending rows, and a user who wants to see exactly what would be sent can read the file. The loader reads the fresher of the cached index and the embedded seed index, by generated day, once at start-up; a refresh runs in the background and takes effect at the next start, so a running task's picks never move under it. - -\paragraph{The relay.} -The reference relay is a single edge function with an object store behind it, deployable in minutes on a commodity edge platform (\code{wrangler deploy} from \code{relay/} in the repository; the runbook beside this document is ten steps). It exposes two routes. \code{POST /v1/rows} accepts a batch of rows, validates the schema, folds each row into the sender's triple for its cell and day, and discards the row: the store holds the sheet of Definition~2 and nothing else, one triple per (install, cell, day), so its size is bounded by installs times cells times days and not by traffic, and a row that has been folded cannot be recovered from it. \code{GET /index.json} serves the last published index and its detached signature. An hourly scheduled job reads the sheet, runs the judge-effect fit and the robust aggregation of Section~5 (the drift estimate of Section~5.5 is specified and next to land), writes the index, signs it with an Ed25519 key held as a platform secret, and mirrors the pair of files to a public git branch so that they are browsable, diffable and forkable. The function keeps no request logs and stores no network address; the install nonce is the only identity in the store, and rate limiting is by nonce (a cap on rows per day) rather than by address. - -\paragraph{Priming, and one document at a time.} -The pool does not start empty. The same measured runs the seed of Figure~\ref{fig:cells} encodes were loaded once as ordinary rows, by a committed script in \code{relay/tools}, under one install nonce per originating session and one judge id for the reviewer that scored them; so the first published index carries the cells the binary already knows, and an install that reaches the relay reads numbers on day one rather than an empty document. Nothing is counted twice, because an install holds exactly one index document at a time: the cached one when it parses and is not older, by its generated day, than the seed the binary carries, and the seed otherwise. The seed is the offline fallback, never a second source blended with the fetched one. That the priming rows were scored by review and later rows by a judge model is the judge-severity situation of Section~\ref{sec:pool} already: one more judge id, fitted like the others. - -\paragraph{Keys and verification.} -The signing key's public half is compiled into the binary as a list, so a key can be rotated by shipping a binary that carries both; the private half exists only in the platform secret store. Verification is client-side: the binary checks the signature before it reads a single cell, refuses a version lower than the one it holds (a rollback is caught by the version; a replayed old index is refused the same way), and on any failure keeps the cached copy beside an error. Because verification is local, the index can be served from anywhere: the mirror, a static file host, or a directory on disk, which is what the \code{file://} address is for. - -\paragraph{Trust and failure.} -What a compromised relay learns is the sheet: counts, sums and sums of squares of scores per (install nonce, role, model, judge, day, size, door). It learns no task, no code, no prompt, no user, and no address. What a compromised relay can do is publish a bad index, which the signature stops unless the key is also taken; a taken key is retired by shipping the next key, and the min-installs and robust-aggregation rules bound what a flood of fabricated nonces can move in the meantime. What an unreachable relay costs is nothing at all: the cache, then the seed, answer every read, and the outbox waits. Any install can run the same function against its own store and point its \code{CODEAF\_MODEL\_POOL\_URL} and \code{\_SUBMIT\_URL} at it; two relays that receive the same publications produce the same sheet (Proposition~\ref{prop:crdt}), so a private relay and the public one can be federated later by a one-way copy of publications with no reconciliation step. - -\begin{figure}[H] -\centering -\resizebox{\textwidth}{!}{% -\begin{tikzpicture}[x=1cm,y=1cm, - box/.style={draw,rounded corners=2pt,inner sep=4pt,font=\small,align=center,minimum height=8mm}, - store/.style={box,fill=black!6}, - net/.style={box,fill=blue!6}, - arr/.style={-{Latex[length=2mm]},thick}, - lbl/.style={font=\scriptsize,fill=white,inner sep=1.5pt} -] -\node[box,text width=3.2cm] (bin) at (2.0,0) {binary\\posterior, draw, run, judge}; -\node[store,text width=3.2cm] (outbox) at (2.0,-1.8) {outbox\\rows, local file}; -\node[store,text width=3.2cm] (cache) at (2.0,-3.6) {index cache\\and embedded seed}; -\node[net,text width=6.1cm] (fn) at (8.2,-1.8) {edge function\\\code{POST /v1/rows}: fold a row into its triple\\\code{GET /index.json}: serve the last index}; -\node[store,text width=6.1cm] (kv) at (8.2,-4.2) {object store\\one triple per (install, cell, day)}; -\node[net,text width=3.4cm] (cron) at (13.7,-4.2) {hourly job\\fold, \eqref{eq:als}, Huber, sign}; -\node[store,text width=3.4cm] (mirror) at (13.7,-1.8) {public mirror\\\code{index.json} + \code{.sig}}; -\node[box,dashed,text width=3.4cm] (secret) at (13.7,-6.0) {platform secret\\Ed25519 private key}; -\draw[arr] (bin) -- (outbox); -\draw[arr] (cache.west) -- ++(-0.4,0) |- node[lbl,pos=0.25,left]{read at start} (bin.west); -\draw[arr] (outbox.east) -- node[lbl,above]{https, batched} (fn.west |- outbox.east); -\draw[arr] (fn) -- node[lbl,right]{fold} (kv); -\draw[arr] (kv) -- node[lbl,above]{sheet} (cron); -\draw[arr] (cron) -- node[lbl,right]{signed index} (mirror); -\draw[arr] (secret) -- (cron); -\draw[arr] (mirror.west) -- node[lbl,below,pos=0.6]{pull, verify} (fn.east |- mirror.west); -\draw[arr] ([xshift=-2.2cm]fn.south) |- node[lbl,pos=0.75,above]{cached index} (cache.east); -\node[font=\scriptsize] at (8.0,0.2) {no request logs, no addresses stored, rows discarded after folding}; -\end{tikzpicture}} -\caption{Deployment. The relay is one edge function, one object store holding only the sheet, and one scheduled signing job; the install verifies every index locally and never blocks on the network. Any of the three network components can be replaced by a static file, a directory, or a self-hosted copy.} -\label{fig:arch} -\end{figure} - -\section{What the user controls} - -\begin{center}\small -\begin{tabular}{p{2.1cm}p{4.3cm}p{8.0cm}}\toprule -setting & values & meaning\\\midrule -\code{crew} & frugal $\mid$ balanced $\mid$ max $\mid$ custom & how much to spend: $H$ and the knob $k$\\ -\code{picked from} & table $\mid$ catalog $\mid$ learn & the rows we measured; recomputed from today's catalog; catalog plus pool plus own sheet\\ -\code{model\_pool} & on $\mid$ read $\mid$ off & read the index and send rows (default); read only; neither\\ -\code{explore} & on $\mid$ off & Thompson draw (default) or the posterior mean; \emph{learn} only; specified, not on this head\\ -\bottomrule\end{tabular} -\end{center} - -With the pool set to \emph{read}, the own sheet still accrues and enters~\eqref{eq:three}; with it \emph{off}, only the catalog prior remains and the pick is the catalog's. The index is a JSON document signed with an Ed25519 key whose public half is in the binary; \code{codeaf pool verify} checks it, \code{pool show} prints its summary and each address with its source, \code{pool status} counts the rows waiting in the outbox and says whether the relay answered. The binary carries a seed index of our own scored runs (Figure~\ref{fig:cells}) so that \emph{learn} has numbers on day one. The relay's code is in this repository; its addresses are settings, so an install can read from and send to a relay of its own, or to none. - -\section{Status} - -On the head this document ships with: the catalog crew of Section~\ref{sec:catalog}; the shrinkage blend of the Corollary with $k=30$; the \code{picked from} row and its ladder; the index document, its reader and signed pull with the built-in key; the outbox, the install nonce and the push; the judge, its deterministic choice, and the hook that calls it when a chat task lands; the own sheet, whose cells enter the prior beside the index's by observation count; the settings; the seed index; the relay of Section~\ref{sec:arch} with its store keyed by install and day, the join, the judge-effect fit and the Huber aggregation; the mirror workflow. Specified here and next to land: the drift estimate~\eqref{eq:drift}; the hierarchical cap~\eqref{eq:cap} with the moment estimators~\eqref{eq:sigma}--\eqref{eq:tau0} in place of the count fold; the Thompson draw with role-level updates, the \code{explore} setting and the settled probability $\pi^\ast$; the pair term of Section~6.1; the headless doors' hook. The referee's report on an earlier draft, and what changed in answer to it, are beside this file. - -\begin{thebibliography}{9}\small -\bibitem{thompson1933} W.~R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. \emph{Biometrika} 25:285--294, 1933. -\bibitem{russo2018} D.~Russo, B.~Van Roy, A.~Kazerouni, I.~Osband, Z.~Wen. A tutorial on Thompson sampling. \emph{Foundations and Trends in Machine Learning} 11(1):1--96, 2018. -\bibitem{russo2014} D.~Russo, B.~Van Roy. Learning to optimize via posterior sampling. \emph{Mathematics of Operations Research} 39(4):1221--1243, 2014. -\bibitem{chen2013} W.~Chen, Y.~Wang, Y.~Yuan. Combinatorial multi-armed bandit: general framework and applications. \emph{ICML}, 2013. -\bibitem{wen2015} Z.~Wen, B.~Kveton, A.~Ashkan. Efficient learning in large-scale combinatorial semi-bandits. \emph{ICML}, 2015. -\bibitem{efron1975} B.~Efron, C.~Morris. Data analysis using Stein's estimator and its generalizations. \emph{Journal of the American Statistical Association} 70(350):311--319, 1975. -\bibitem{shapiro2011} M.~Shapiro, N.~Pregui\c{c}a, C.~Baquero, M.~Zawirski. Conflict-free replicated data types. \emph{SSS}, 2011. -\bibitem{huber1964} P.~J. Huber. Robust estimation of a location parameter. \emph{Annals of Mathematical Statistics} 35(1):73--101, 1964. -\bibitem{harvey1989} A.~C. Harvey. \emph{Forecasting, Structural Time Series Models and the Kalman Filter}. Cambridge University Press, 1989. -\end{thebibliography} - -\end{document} diff --git a/docs/design/model-pool/tab-regret.tex b/docs/design/model-pool/tab-regret.tex deleted file mode 100644 index 463ceb71f2..0000000000 --- a/docs/design/model-pool/tab-regret.tex +++ /dev/null @@ -1,8 +0,0 @@ -% generated by figures.py from data/measured-crews.csv; do not edit -\begin{tabular}{lrrrrrrrrr}\toprule -$N$ & 10 & 20 & 40 & 70 & 100 & 150 & 200 & 300 & 400\\\midrule -table & 1.01 & 0.95 & 0.95 & 0.93 & 0.95 & 0.93 & 0.94 & 0.96 & 0.97\\ -catalog & 0.99 & 0.94 & 0.94 & 0.91 & 0.92 & 0.88 & 0.88 & 0.90 & 0.91\\ -greedy & 1.20 & 1.10 & 1.01 & 0.99 & 0.94 & 0.90 & 0.86 & 0.80 & 0.76\\ -thompson & 0.97 & 0.97 & 0.93 & 0.87 & 0.88 & 0.84 & 0.79 & 0.71 & 0.64\\ -\bottomrule\end{tabular} diff --git a/docs/design/model-pool/tab-semibandit.tex b/docs/design/model-pool/tab-semibandit.tex deleted file mode 100644 index 5a7ac2a294..0000000000 --- a/docs/design/model-pool/tab-semibandit.tex +++ /dev/null @@ -1,7 +0,0 @@ -% generated by figures.py from data/measured-crews.csv; do not edit -\begin{tabular}{lrrrrrrrrr}\toprule -$N$ & 10 & 20 & 40 & 70 & 100 & 150 & 200 & 300 & 400\\\midrule -crew, clean bit & 0.47 & 0.50 & 0.50 & 0.51 & 0.53 & 0.54 & 0.55 & 0.55 & 0.54\\ -crew, mean score & 0.29 & 0.43 & 0.39 & 0.38 & 0.36 & 0.35 & 0.34 & 0.34 & 0.31\\ -role, semi-bandit & 0.36 & 0.35 & 0.31 & 0.27 & 0.26 & 0.22 & 0.20 & 0.17 & 0.15\\ -\bottomrule\end{tabular} diff --git a/internal/catalog/catalog.go b/internal/catalog/catalog.go index 94775538e2..d73402fd00 100644 --- a/internal/catalog/catalog.go +++ b/internal/catalog/catalog.go @@ -123,6 +123,11 @@ type Model struct { IntelligenceIndex float64 `json:"intelligence_index,omitempty"` CodingIndex float64 `json:"coding_index,omitempty"` AgenticIndex float64 `json:"agentic_index,omitempty"` + // Created is when the model was listed, in Unix seconds — the release date + // the crew router reads a model's age from. Zero means the row did not say + // or was cached before this field was kept; a reader then falls back to a + // date in the canonical slug, or to none. + Created int64 `json:"created,omitempty"` // OpenWeights says the row's weights are published — OpenRouter's // `hugging_face_id`, kept as the one-word answer to whether the weights are // public. A row cached before this field existed reads false, which every @@ -1157,6 +1162,7 @@ func fetch(ctx context.Context, options Options) ([]Model, error) { CacheReadPrice: cacheRead, PriceUnknown: !promptOK || !completionOK, ArenaElo: arenaElo(item.Benchmarks), + Created: int64(item.Created), IntelligenceIndex: analysisScore(item.Benchmarks, "intelligence_index"), CodingIndex: analysisScore(item.Benchmarks, "coding_index"), AgenticIndex: analysisScore(item.Benchmarks, "agentic_index"), @@ -1177,7 +1183,7 @@ func fetch(ctx context.Context, options Options) ([]Model, error) { } // modelWire is one row of OpenRouter's /models listing, in the shape this -// package reads it. Everything absent from it — description, created, +// package reads it. Everything absent from it — description, // per_request_limits, top_provider, links — is either prose nobody renders or // provider bookkeeping, and a field added here is a field something on screen // has to be able to explain. @@ -1194,7 +1200,9 @@ type modelWire struct { } `json:"alias_target"` Name string `json:"name"` ContextLength int `json:"context_length"` - Architecture struct { + // Created is when the row was listed, in Unix seconds. + Created float64 `json:"created"` + Architecture struct { // Modality is the coarse "text->text" string. It is read for nothing: // input_modalities and output_modalities say the same thing as lists, // and a list is what every question this package answers is asked in. diff --git a/internal/config/auto.go b/internal/config/auto.go deleted file mode 100644 index 21b15d1a5e..0000000000 --- a/internal/config/auto.go +++ /dev/null @@ -1,513 +0,0 @@ -package config - -import ( - "strings" - - "github.com/Agent-Field/codeaf/internal/catalog" - "github.com/Agent-Field/codeaf/internal/crewpick" - "github.com/Agent-Field/codeaf/internal/pool/index" -) - -// THE WORD THAT ASKS THE CATALOG TO ANSWER A TIER ROW. -// -// A tier row names a model id, and a model id is a bet the person made one day -// against a catalog that keeps moving under it. `auto` is the row that -// declines to bet: the seat's model is computed from the catalog's own -// published figures — the three capability indexes against the three prices, -// under each seat's call shape (internal/crewpick) — every time the row is -// read, through [autoRow], the one seam both ladders pass through. -// -// NOTHING IS REWRITTEN. The row stays `auto` on disk and the id is derived on -// every read, so a catalog that moves moves the seat with it, and a profile -// never holds a model id this build chose for somebody. - -// AutoValue is the bare word a tier row holds to have its seat computed from -// the catalog. -const AutoValue = "auto" - -// IsAuto reports whether a tier value is the bare word, case folded with the -// surrounding space ignored. `auto:high` is not it: the rows own the -// `:` notation, and a suffixed word is a model id with a level, handed -// on whole the way every other one is. -func IsAuto(value string) bool { - return strings.ToLower(strings.TrimSpace(value)) == AutoValue -} - -// AutoModels is how the catalog reaches seat resolution: the binary holding -// the catalog sets it ONCE AT START-UP, from its non-blocking read, and never -// a fetch — a tier row that says auto resolves from whatever the catalog -// already holds, the same posture every other catalog reader here keeps. Nil, -// and a func answering with no rows, are ordinary states rather than errors: -// a row that says auto then reads the family's table row ([autoRow]), which -// is the answer it had before this word existed. -var AutoModels func() []catalog.Model - -// AutoIndex is how the Model Pool's measurement index reaches seat -// resolution: the binary holding the index sets it ONCE AT START-UP, from -// whatever read it already made, and never a fetch — a tier row that says -// auto resolves from the index already in hand, the same posture AutoModels -// keeps. Nil is an ordinary state, not an error: a row that says auto then -// reads the catalog's own figures alone, which is what it read before this -// seam existed. -var AutoIndex func() *index.Index - -// AutoOwnCells is how an install's own judged scores reach seat resolution: -// the binary holding the pool sets it ONCE AT START-UP, from the own sheet it -// read from disk under the pool directory, the same posture AutoIndex keeps. -// Nil is an ordinary state, not an error: the prior then reads the index's -// cells alone, which is what it read before this seam existed. The cells are -// never held to the index's min_installs — an install's own scores are its -// own evidence, one observation of which is worth having. -var AutoOwnCells func() []crewpick.Cell - -// PoolQualityMetric names the index metric the picker reads a measured seat -// quality from: a gaussian metric whose mean is on the same 0-100 scale -// crewpick scores a seat on, with one cell per role and model. -const PoolQualityMetric = "role_quality" - -// AutoPick is the pick with the measured quality the Model Pool holds ALWAYS -// carried as a prior: [AutoPickWith] with [autoPrior], whatever a profile's -// pick word says. No seat resolves through it any more — the bare `auto` row -// ([autoRow]) and the pick row's words ([pickedModel]) decide the prior -// together in [priorFor] — and it stays for a caller that wants the measured -// computation regardless of the pick word. -func AutoPick(tier, family, preset string, models []catalog.Model) (modelID string, ok bool) { - return AutoPickWith(tier, family, preset, models, autoPrior(autoIndex())) -} - -// AutoPickWith is the pick itself, with the measured quality NAMED: prior is -// carried into the front as a rating seats read on top of the catalog's own -// published scores, and a nil prior leaves every seat on those scores alone — -// the answer the `catalog` pick word carries, where `learn` passes the pool's -// measurements and the person's own judged runs ([autoPrior]). -// -// It is PURE: no disk, no network, the rows are only read, and the same rows -// give the same answer however often it is asked and in whatever order they -// arrive (crewpick breaks its ties by id, not by order). Each row is read as -// a crewpick candidate — the three indexes, the three prices AS PUBLISHED (a -// uniform scale changes no pick on a front sorted by bill), whether the -// provider published a cache-read price at all, the window, the modalities -// and the parameters — and a row whose price the provider did not publish is -// left out, because a model that may cost anything has no place in a pick -// that is about cost. -// -// tier names the seat the pick is read from: worker, high and mastermind have -// answers, reflex and low and any other word do not. family narrows the -// shelf: `open` picks off the open-weight rows, every other word off the -// whole catalog. preset is one of the three crew words, and any other word -// has no answer. No rows, an empty front or an empty id are no answer too; -// no answer is ok false and an empty id, which is the caller's cue to read -// the family's table row instead ([autoRow], [pickedSeat]). -func AutoPickWith(tier, family, preset string, models []catalog.Model, prior crewpick.Prior) (modelID string, ok bool) { - seat, ok := autoSeat(tier) - if !ok || len(models) == 0 { - return "", false - } - fam := crewpick.All - if normalCrewSource(family) == CrewSourceOpen { - fam = crewpick.Open - } - candidates := autoCandidates(models) - if len(candidates) == 0 { - return "", false - } - frugal, balanced, max := crewpick.Presets(crewpick.FrontWith(candidates, crewpick.DefaultShapes(), fam, prior)) - var pick crewpick.Crew - switch strings.ToLower(strings.TrimSpace(preset)) { - case CrewFrugal: - pick = frugal - case CrewBalanced: - pick = balanced - case CrewMax: - pick = max - default: - return "", false - } - var id string - switch seat { - case crewpick.Worker: - id = pick.Worker - case crewpick.High: - id = pick.High - case crewpick.Mastermind: - id = pick.Mastermind - } - return id, id != "" -} - -// autoIndex is [AutoIndex] read with its ordinary absence folded into one -// answer. -func autoIndex() *index.Index { - if AutoIndex == nil { - return nil - } - return AutoIndex() -} - -// autoPrior reads the measured quality of the index AND of this install's own -// sheet into a crewpick prior. The index's cells of PoolQualityMetric are kept -// only when the metric is declared gaussian — a mean on any other scale would -// be blended against figures it does not share units with — and dropped by -// PriorFromCells below the index's own min_installs, resolved through the -// index's canonical ids so an alias meets its candidate. The own sheet's cells -// are this install's own evidence: they are read through a second -// PriorFromCells at a floor of one and folded into the index's prior seat by -// seat, the means combined by observation count. Their scores are on the -// 0-100 scale a judge answers on whatever the index's metric says, so they -// are read even beside an index whose role_quality is not gaussian. A nil -// index and a nil seam answer no prior, which leaves every seat on the -// catalog quality. -func autoPrior(idx *index.Index) crewpick.Prior { - canonical := autoCanonical(idx) - var indexPrior crewpick.Prior - if idx != nil { - if kind, ok := idx.Kind(PoolQualityMetric); ok && kind == "gaussian" { - cells := idx.Cells(PoolQualityMetric) - measured := make([]crewpick.Cell, 0, len(cells)) - for _, c := range cells { - measured = append(measured, crewpick.Cell{Role: c.Role, Model: c.Model, Mean: c.Mean, N: c.N}) - } - indexPrior = crewpick.PriorFromCells(measured, idx.MinInstalls(), idx.Canonical) - } - } - var ownPrior crewpick.Prior - if own := autoOwnCells(); len(own) > 0 { - ownPrior = crewpick.PriorFromCells(own, 1, canonical) - } - return crewpick.MergePriors(indexPrior, ownPrior) -} - -// autoCanonical is the index's canonical ids, nil when there is no index. The -// own sheet's ids are the ones this install resolved its seats to, and a nil -// canonical leaves them as they stand. -func autoCanonical(idx *index.Index) func(string) string { - if idx == nil { - return nil - } - return idx.Canonical -} - -// autoOwnCells is [AutoOwnCells] read with its ordinary absence folded into -// one answer. -func autoOwnCells() []crewpick.Cell { - if AutoOwnCells == nil { - return nil - } - return AutoOwnCells() -} - -// autoSeat is the tier word that names which of crewpick's three seats the -// pick is read from. The two tiers that never vary — reflex and small work — -// have no answer here, the same law their columns in the shipped tables keep. -func autoSeat(tier string) (crewpick.Seat, bool) { - switch strings.ToLower(strings.TrimSpace(tier)) { - case ModelTierWorker: - return crewpick.Worker, true - case ModelTierHigh: - return crewpick.High, true - case ModelTierMastermind: - return crewpick.Mastermind, true - } - return 0, false -} - -// autoCandidates reads every published row as a candidate. Rows with no -// published price are dropped here, where the reason is nameable, rather than -// being carried into the picker to fall out of its own candidacy law. -func autoCandidates(models []catalog.Model) []crewpick.Candidate { - candidates := make([]crewpick.Candidate, 0, len(models)) - for _, model := range models { - if model.PriceUnknown { - continue - } - candidates = append(candidates, crewpick.Candidate{ - ID: model.ID, - Open: model.OpenWeights, - Intelligence: model.IntelligenceIndex, - Coding: model.CodingIndex, - Agentic: model.AgenticIndex, - PromptPrice: model.PromptPrice, - CompletionPrice: model.CompletionPrice, - CacheReadPrice: model.CacheReadPrice, - HasCacheRead: model.CacheReadPrice > 0, - Context: model.ContextLength, - Images: listHolds(model.InputModalities, "image"), - Tools: listHolds(model.Parameters, "tools"), - }) - } - return candidates -} - -// listHolds says whether a row's own list carries the word, case folded — the -// catalog cleans its lists, but a reader that folds is one fewer assumption. -func listHolds(words []string, word string) bool { - for _, held := range words { - if strings.EqualFold(strings.TrimSpace(held), word) { - return true - } - } - return false -} - -// ── the seam ──────────────────────────────────────────────────────────────── - -// autoRow is THE ONE SEAM both ladders resolve a tier row that says auto -// through — [tierSeatUnder], which the conversation, the settings sheet and -// the crew word read, and [resolveSeat], which every headless door climbs. A -// second copy in each would be a seat that means one thing in chat and -// another headless, which is the defect this file exists to prevent one -// ladder at a time. -// -// The answer is the computed id, on its own rung, when the catalog can -// compute one; otherwise the family's table row FOR THE PRESET, on the table -// rung — the same id a preset write would have landed, and never `auto` and -// never empty. -func autoRow(profileDir, family, tier string) (model string, source SeatSource, preset string) { - preset = crewPresetUnder(profileDir, family) - // THE ANSWER IS THE PICK WORD'S OWN ANSWER, computed the way a - // pick-computed seat's is ([pickedModel]): the prior is decided once, in - // [priorFor] — the pool's measurements under `learn`, the published - // figures alone under every other word — so the two seams cannot disagree - // about what `catalog` means, and the rung names the word that ran - // ([computedRung]) so they cannot name one computation two ways either. - pick := CrewPickAt(profileDir) - if id, ok := AutoPickWith(tier, family, preset, autoCatalogRows(), priorFor(pick)); ok { - return id, computedRung(pick), preset - } - if row, ok := CrewModelsForSource(family, preset); ok { - if id := strings.TrimSpace(row[tier]); id != "" { - return id, SeatTable, preset - } - } - // Unreachable for a tier this build knows — every preset row answers all - // five — but the never-empty law is carried rather than assumed: the - // family's default row is the last word. - return defaultTierModel(family, tier), SeatTable, preset -} - -// pickedSeat is THE PICK ROW'S HALF OF THE LADDER: where a seat's model comes -// from when the pick is off the table and the tier row does not name a model -// of its own. Both ladders call it — [tierSeatUnder] for the conversation and -// the settings sheet, [resolveSeat] for every headless door — so a pick -// cannot mean one thing in chat and another headless. -// -// The rule, in the order it is applied: -// -// - the two seats that read every turn are NOT asked: reflex and small work -// always read the table, the same law their columns in the shipped tables -// keep, and the caller decides which tiers reach here; -// - a WRITTEN dial row is a person's own model, and it wins — the pick answers -// for the seats nobody named, not over the names they typed. A written row -// is a pin when the profile's five tier rows are NOT all written (a `/crew` -// APPLY writes all five at once, a hand pin writes one), when the profile -// STORES its own crew word ([storedCrewWord]), or when the row's id is not -// the preset's own table value; -// - every other seat among worker, careful work and mastermind is computed -// at the crew's preset ([crewPresetUnder]), on the rung the pick names: -// [SeatComputed] under the `catalog` word, [SeatLearned] under `learn`; -// - when the catalog cannot compute one — no rows, no pick off the front — -// the preset's own table row is the answer, on the table rung, the same -// fallback [autoRow] keeps. Never `auto` and never empty. -// -// ok is false only when the pick has nothing to say — the pick row at its -// default, a hand-typed id, or a tier that always reads the table — and the -// caller answers with the row's own rung instead. The DEFAULT PICK IS THE -// FIRST CHECK: a profile that has never answered the row reads every seat -// exactly as it read before the row existed, which is what a default is for. -func pickedSeat(profileDir, family, tier, model string, written bool) (Seat, bool) { - if CrewPickAt(profileDir) == CrewPickTable { - return Seat{}, false - } - if tier != ModelTierWorker && tier != ModelTierHigh && tier != ModelTierMastermind { - return Seat{}, false - } - // A WRITTEN DIAL ROW IS A PIN THE PICK DOES NOT RECOMPUTE when it is not the - // preset answering. The preset answering is a `/crew` APPLY, which writes all - // five tier rows at once ([ApplyCrew]); so a written row wins when the five - // rows are not all written (a person pinned one by hand), when the profile - // stores its own crew word ([storedCrewWord] — a run that named a budget and - // pinned the crew table's OWN ids, which the value rule below would - // recompute), or when the row's id is not the preset's own. A crew applied - // WHOLE with no stored word keeps the old rule: a row holding the preset's - // own table value is the preset answering and the pick computes it. - if written { - if _, worded := storedCrewWord(profileDir); worded || !allTiersWritten(profileDir) { - return Seat{}, false - } - } - preset := crewPresetUnder(profileDir, family) - if row, ok := CrewModelsForSource(family, preset); ok { - if model != "" && !strings.EqualFold(strings.TrimSpace(model), row[tier]) { - return Seat{}, false - } - } - id, source := pickedModel(tier, family, preset, CrewPickAt(profileDir)) - return Seat{Role: tierSeatRole(tier), Model: id, Source: source, Crew: preset}, true -} - -// allTiersWritten reports whether the profile's five tier rows were EACH WRITTEN -// — every key held and non-empty. It is how the pick tells a crew APPLY, which -// writes all five at once ([ApplyCrew]), from a person pinning one model by -// hand: only a whole crew is the preset answering, and a partial write is the -// person's own rows. -func allTiersWritten(profileDir string) bool { - for _, tier := range ModelTiers { - model, _, source, cleared := crewRow(profileDir, tier) - if cleared || source != SeatCrew || strings.TrimSpace(model) == "" { - return false - } - } - return true -} - -// unwrittenSeat is what an UNWRITTEN tier row reads when the profile STORES a -// crew word: that word's own table row, under the family named, on the table -// rung. It is shared by both ladders — [tierSeatUnder] for the conversation and -// the settings sheet, [resolveSeat] for every headless door — so a word that -// seats one surface seats the other, and a profile that stores no word answers -// nothing here and keeps its own bottom rung. -func unwrittenSeat(profileDir, family, tier string) (Seat, bool) { - word, ok := storedCrewWord(profileDir) - if !ok { - return Seat{}, false - } - row, ok := CrewModelsForSource(family, word) - if !ok { - return Seat{}, false - } - id := strings.TrimSpace(row[tier]) - if id == "" { - return Seat{}, false - } - return Seat{Role: tierSeatRole(tier), Model: id, Source: SeatTable, Crew: word}, true -} - -// priorFor is the prior a seat computed at the crew's preset carries, read -// from the pick word alone: [autoPrior] under `learn` — the Model Pool's -// measurements and this install's own judged runs — and nil under every other -// word, which leaves each seat on the published figures alone. The bare -// `auto` row ([autoRow]) and the pick word's own seat ([pickedModel]) both -// answer through it, so the two seams cannot disagree again about what -// `catalog` means. -func priorFor(pick string) crewpick.Prior { - if normalCrewPick(pick) == CrewPickLearn { - return autoPrior(autoIndex()) - } - return nil -} - -// pickedModel is the pick word's own answer for one tier: the catalog's pick -// at the preset, on the rung the word names; and, when nothing can be -// computed, the preset's own table row on the table rung — the id a preset -// write would have landed, and never `auto` and never empty. -func pickedModel(tier, family, preset, pick string) (modelID string, source SeatSource) { - pick = normalCrewPick(pick) - if id, ok := AutoPickWith(tier, family, preset, autoCatalogRows(), priorFor(pick)); ok { - return id, computedRung(pick) - } - if row, ok := CrewModelsForSource(family, preset); ok { - if id := strings.TrimSpace(row[tier]); id != "" { - return id, SeatTable - } - } - // Unreachable for a tier this build knows — every preset row answers all - // five — but the never-empty law is carried rather than assumed, the same - // last word [autoRow] keeps. - return defaultTierModel(family, tier), SeatTable -} - -// computedRung is the rung a seat computed at the crew's preset answers on, -// read from the pick word alone: [SeatLearned] under `learn`, whose pick -// carries the Model Pool's measurements and this install's own judged runs -// ([autoPrior]), and [SeatComputed] under `catalog` — and under the table word, -// which computes nothing of its own. IT ASKS WHICH PICK RAN, never whether the -// prior moved the id: [SeatLearned]'s own comment says the rung names the -// computation, and a rung that reported an actual move would be a second rule. -// The pick's own seat ([pickedModel]) and a bare `auto` row ([autoRow]) both -// answer through it, so the two seams cannot name one computation two ways. -func computedRung(pick string) SeatSource { - if normalCrewPick(pick) == CrewPickLearn { - return SeatLearned - } - return SeatComputed -} - -// autoCatalogRows is [AutoModels] read with its two ordinary absences folded -// into one answer. -func autoCatalogRows() []catalog.Model { - if AutoModels == nil { - return nil - } - return AutoModels() -} - -// crewPresetUnder is the preset the profile's five STORED rows make, in the -// family given — read from the rows and never through seat resolution, so -// the seam cannot ask the ladder that is asking it. -// -// Each stored tier value is read through the ladder's own row reader -// ([crewRow]), which is what keeps a row reached through the lineage saying -// it the same way. A row the reader cannot answer — cleared, or never held — -// reads the family's default-preset id, which is what that tier runs until -// somebody writes it. A row that says auto matches WHICHEVER preset is being -// compared, because auto is the one row with no opinion of its own: it is -// asking to be computed at the budget the other four rows name. -// -// The answer is the first preset whose every row matches, the default preset -// tried first and then the crew's own order; when none matches, the default — -// the budget an undecided profile runs at is the budget an auto seat runs at. -// -// THE DEFAULT PRESET WINS EVERY TIE, and ties are ordinary rather than rare: -// the presets differ in only a few cells, so an auto row on a seat the two -// presets share leaves the rest of the rows matching both. Max differs from -// balanced only in the worker seat today, which makes a crew with an auto -// worker and the shipped rows elsewhere exactly that tie — it reads balanced. -// Trying the default first is what decides it, so the order above is the rule -// and not an accident of iteration. -func crewPresetUnder(profileDir, family string) string { - // A STORED CREW WORD IS THE BUDGET, not the rows under it. A run that wrote - // one word into config.json instead of the five rows ([storedCrewWord]) - // named its budget directly, so the computed seats read at that budget - // whether or not the rows were written — this is the seam the word reaches - // the task door's seats through. - if word, ok := storedCrewWord(profileDir); ok { - return word - } - defaults := crewTableFor(family)[DefaultCrew] - read := make(map[string]string, len(ModelTiers)) - for _, tier := range ModelTiers { - value, _, source, cleared := crewRow(profileDir, tier) - switch { - case cleared || source == "": - read[tier] = strings.ToLower(strings.TrimSpace(defaults[tier])) - case IsAuto(value): - read[tier] = AutoValue - default: - read[tier] = strings.ToLower(strings.TrimSpace(value)) - } - } - matches := func(preset string) bool { - row, ok := CrewModelsForSource(family, preset) - if !ok { - return false - } - for _, tier := range ModelTiers { - if read[tier] == AutoValue { - continue - } - if read[tier] != strings.ToLower(strings.TrimSpace(row[tier])) { - return false - } - } - return true - } - if matches(DefaultCrew) { - return DefaultCrew - } - for _, preset := range CrewPresets { - if preset != DefaultCrew && matches(preset) { - return preset - } - } - return DefaultCrew -} diff --git a/internal/config/auto_test.go b/internal/config/auto_test.go deleted file mode 100644 index 08131c91ab..0000000000 --- a/internal/config/auto_test.go +++ /dev/null @@ -1,599 +0,0 @@ -package config - -import ( - "encoding/json" - "math" - "os" - "strings" - "testing" - - "github.com/Agent-Field/codeaf/internal/catalog" - "github.com/Agent-Field/codeaf/internal/crewpick" - "github.com/Agent-Field/codeaf/internal/pool/index" -) - -// THE WORD, AND WHAT IS NOT THE WORD. `auto` is a bare word a tier row holds, -// and everything that only resembles one is a model id: a suffixed word is the -// rows' own `:` notation, a slashed one is a vendor's namespace, and -// `automatic` is somebody's model that happens to begin with it. -func TestIsAutoReadsTheBareWordAlone(t *testing.T) { - for word, want := range map[string]bool{ - "auto": true, " AUTO ": true, "Auto": true, "\tauto\n": true, - "": false, "auto:high": false, "vendor/auto": false, "automatic": false, - } { - if got := IsAuto(word); got != want { - t.Errorf("IsAuto(%q) = %t, want %t", word, got, want) - } - } -} - -// autoTestRows is a small catalog with every fact a pick reads, and one row -// the provider priced "-1" on — which is not a candidate, for the reason a -// pick that is about cost cannot count a model whose cost nobody published. -func autoTestRows() []catalog.Model { - return []catalog.Model{ - {ID: "a/cheap", OpenWeights: true, IntelligenceIndex: 30, CodingIndex: 35, AgenticIndex: 40, - PromptPrice: 0.0000002, CompletionPrice: 0.0000009, CacheReadPrice: 0.00000002, - ContextLength: 200_000, InputModalities: []string{"text", "image"}, Parameters: []string{"tools"}}, - {ID: "a/mid", OpenWeights: true, IntelligenceIndex: 45, CodingIndex: 50, AgenticIndex: 48, - PromptPrice: 0.000001, CompletionPrice: 0.000003, CacheReadPrice: 0.0000001, - ContextLength: 200_000, InputModalities: []string{"text"}, Parameters: []string{"tools"}}, - {ID: "b/care", IntelligenceIndex: 60, CodingIndex: 55, AgenticIndex: 50, - PromptPrice: 0.000004, CompletionPrice: 0.00002, CacheReadPrice: 0.0000004, - ContextLength: 400_000, InputModalities: []string{"text", "image"}, Parameters: []string{"tools"}}, - {ID: "c/mind", IntelligenceIndex: 75, CodingIndex: 70, AgenticIndex: 65, - PromptPrice: 0.00001, CompletionPrice: 0.00005, ContextLength: 200_000, - InputModalities: []string{"text"}, Parameters: []string{"tools"}}, - {ID: "r/router", PriceUnknown: true, IntelligenceIndex: 80, CodingIndex: 80, AgenticIndex: 80, - PromptPrice: -1, CompletionPrice: -1, ContextLength: 200_000}, - } -} - -// THE PICK IS PURE AND ITS ANSWER IS THE FRONT'S. The same rows give the same -// id however often it is asked, the rows are never touched, and a row whose -// price nobody published is in no seat. The front is read the way the shipped -// tables were (crew.go owns the method), so the pick answers exactly what -// crewpick answers for these rows — asserted here against crewpick itself -// rather than against a figure copied out of one. -func TestAutoPickAnswersTheFront(t *testing.T) { - rows := autoTestRows() - workers := make([]string, 0, 3) - for i := 0; i < 3; i++ { - id, ok := AutoPick(ModelTierWorker, CrewSourceAll, CrewFrugal, rows) - if !ok || id == "" { - t.Fatalf("AutoPick answered ok=%t id=%q", ok, id) - } - workers = append(workers, id) - } - for i := 1; i < len(workers); i++ { - if workers[i] != workers[0] { - t.Fatalf("the same rows answered %q then %q — the pick is not pure", workers[0], workers[i]) - } - } - _ = workers - if rows[4].ID != "r/router" || !rows[4].PriceUnknown { - t.Fatal("AutoPick modified the rows it was handed") - } - - frugal, _, maxCrew := crewpick.Presets(crewpick.Front(autoCandidates(rows), crewpick.DefaultShapes(), crewpick.All)) - if len(frugal.Worker) == 0 { - t.Fatal("the test rows field no crew — the rows are broken") - } - if got, _ := AutoPick(ModelTierWorker, CrewSourceAll, CrewFrugal, rows); got != frugal.Worker { - t.Fatalf("worker frugal = %q, want the front's own %q", got, frugal.Worker) - } - if got, _ := AutoPick(ModelTierMastermind, CrewSourceAll, CrewMax, rows); got != maxCrew.Mastermind { - t.Fatalf("mastermind max = %q, want the front's own %q", got, maxCrew.Mastermind) - } -} - -// THE TWO TIERS THAT NEVER VARY HAVE NO ANSWER HERE, the same law their -// columns in the shipped tables keep — reflex and small work are the same -// near-free models in all three presets, and a word that computes for them -// would be a second opinion about a row that has none. -func TestAutoPickRefusesTheTiersWithoutOpinions(t *testing.T) { - for _, tier := range []string{ModelTierReflex, ModelTierLow, "", "banana"} { - if id, ok := AutoPick(tier, CrewSourceAll, CrewFrugal, autoTestRows()); ok || id != "" { - t.Errorf("AutoPick(%q) answered %q, want no answer", tier, id) - } - } -} - -// THE THREE WORDS ARE THE ONLY PRESETS, `open` IS THE ONLY OTHER FAMILY, AND -// NOTHING IS AN ERROR: no answer is ok false and an empty id, on no rows, -// on an empty front, on a word that is not a preset, and on a tier the -// catalog cannot field. -func TestAutoPickAnswersNothingWhenNothingCanBePicked(t *testing.T) { - if id, ok := AutoPick(ModelTierWorker, CrewSourceAll, "banana", autoTestRows()); ok || id != "" { - t.Errorf("an unknown preset answered %q", id) - } - if id, ok := AutoPick(ModelTierWorker, CrewSourceAll, CrewFrugal, nil); ok || id != "" { - t.Errorf("no rows answered %q", id) - } - if id, ok := AutoPick(ModelTierHigh, CrewSourceAll, CrewFrugal, []catalog.Model{{ID: "a/cheap", PromptPrice: 1}}); ok || id != "" { - t.Errorf("rows that field no crew answered %q", id) - } - // A family word this build does not know reads as the default family, - // the way [CrewSourceAt] reads one — and that family answers here. - if _, ok := AutoPick(ModelTierWorker, "misplaced", CrewFrugal, autoTestRows()); !ok { - t.Error("an unknown family word read as no family at all") - } -} - -// ── the ladder ────────────────────────────────────────────────────────────── - -// A ROW THAT SAYS AUTO RESOLVES THROUGH THE ONE SEAM, ON BOTH LADDERS, to the -// computed id with the rung that says so — and with the catalog absent, to -// the family's table row for the preset the other four rows name, on the -// table rung. It never resolves to `auto` and never to empty. -func TestAnAutoRowResolvesOnBothLadders(t *testing.T) { - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - restore := AutoModels - AutoModels = func() []catalog.Model { return autoTestRows() } - defer func() { AutoModels = restore }() - - rows := writeProfileRows(t, map[string]string{ - KeyTierWorkerModel: " auto ", - KeyTierMastermindModel: "AUTO", - }) - for _, seat := range []Seat{ - TierSeatAt(rows, ModelTierWorker), - ResolveSeats(rows, "", "").Work, - } { - if seat.Source != SeatComputed { - t.Errorf("conversation ladder: the seat reads %s, want computed", seat.Rung()) - } - if strings.TrimSpace(seat.Model) == "" || IsAuto(seat.Model) { - t.Errorf("conversation ladder: the seat reads %q, want a computed id", seat.Model) - } - } - plan := ResolveSeats(rows, "", "").Plan - if plan.Source != SeatComputed { - t.Errorf("headless ladder: the plan seat reads %s, want computed", plan.Rung()) - } - - // The catalog reaches nothing: the family's table row answers, on the - // table rung, at the preset the stored rows make. These rows pin nothing - // else, so they are the default crew — and the worker column of that - // preset in the DEFAULT family is the answer the table owes. - AutoModels = nil - table, _ := CrewModelsForSource(DefaultCrewSource, DefaultCrew) - want := table[ModelTierWorker] - for name, seat := range map[string]Seat{ - "conversation": TierSeatAt(rows, ModelTierWorker), - "headless": ResolveSeats(rows, "", "").Work, - } { - if seat.Model != want || seat.Source != SeatTable { - t.Errorf("%s ladder with no catalog: %q (%s), want %q (table)", name, seat.Model, seat.Rung(), want) - } - } -} - -// A REACHED-THROUGH-THE-LINEAGE ROW SAYS AUTO THE SAME WAY. The worker row -// landed after profiles held only the small-work row, and a profile of that -// vintage whose small-work row says auto hands its work seat the same answer -// the conversation hands it — computed when the catalog can, and never the -// bare word. -func TestAnInheritedAutoRowResolvesTheSameWay(t *testing.T) { - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - restore := AutoModels - AutoModels = func() []catalog.Model { return autoTestRows() } - defer func() { AutoModels = restore }() - - dir := writeProfileRows(t, map[string]string{KeyTierLowModel: AutoValue}) - for name, seat := range map[string]Seat{ - "conversation": TierSeatAt(dir, ModelTierWorker), - "headless": ResolveSeats(dir, "", "").Work, - } { - if seat.Model == AutoValue || seat.Model == "" { - t.Errorf("%s ladder: the seat reads %q, want what the row was reached through", name, seat.Model) - } - } -} - -// THE FLAG AND THE VARIABLE STILL OUTRANK A ROW THAT SAYS AUTO, and a flag -// whose text is the word is handed on whole — a person who named a model -// called auto got a model called auto, exactly as they did before this rung -// existed. -func TestTheInvocationRungsStillOutrankAnAutoRow(t *testing.T) { - t.Setenv(ModelEnv, "vendor/from-the-environment") - t.Setenv(PlanModelEnv, "vendor/plans-from-the-environment") - dir := writeProfileRows(t, map[string]string{KeyTierWorkerModel: AutoValue}) - seats := ResolveSeats(dir, "", "") - if seats.Work.Model != "vendor/from-the-environment" || seats.Work.Source != SeatEnv { - t.Errorf("the environment rung read %q (%s)", seats.Work.Model, seats.Work.Rung()) - } - if seats.Plan.Model != "vendor/plans-from-the-environment" || seats.Plan.Source != SeatEnv { - t.Errorf("the plan seat read %q (%s)", seats.Plan.Model, seats.Plan.Rung()) - } - seats = ResolveSeats(dir, AutoValue, "") - if seats.Work.Model != AutoValue || seats.Work.Source != SeatFlag { - t.Errorf("a flag that says auto read %q (%s), want the word handed on whole", seats.Work.Model, seats.Work.Rung()) - } -} - -// THE PRESET AN AUTO SEAT RUNS AT IS READ FROM THE STORED ROWS, never through -// seat resolution — the seam cannot ask the ladder that is asking it — and a -// row that says auto matches whichever preset the other rows name. -// -// MAX DIFFERS FROM BALANCED ONLY IN THE WORKER SEAT NOW, so an auto row on the -// worker can no longer tell the two apart — balanced, the default preset, wins -// that tie — and the identifying shape is the other way round: the worker -// pinned to max's own id, which only max names, with the auto row on a seat -// above it. -func TestThePresetAnAutoSeatRunsAtIsReadFromTheStoredRows(t *testing.T) { - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - models, _ := CrewModelsForSource(DefaultCrewSource, CrewMax) - rows := map[string]string{KeyTierMastermindModel: AutoValue} - for _, tier := range ModelTiers { - if tier != ModelTierMastermind { - rows[tierKeyFor(tier)] = models[tier] - } - } - dir := writeProfileRows(t, rows) - restore := AutoModels - AutoModels = func() []catalog.Model { return autoTestRows() } - seat := TierSeatAt(dir, ModelTierMastermind) - AutoModels = restore - if seat.Crew != CrewMax { - t.Errorf("max's rows and an auto row read as %q", seat.Crew) - } - if seat.Rung() != "crew "+CrewMax+", computed from the catalog" { - t.Errorf("the rung reads %q, want crew max, computed from the catalog", seat.Rung()) - } - // A profile with no auto row anywhere reads as it always read — the - // default rung, no crew word — which is the unchanged-behaviour law: the - // seam fires only on a row that says the word. - seat = TierSeatAt(t.TempDir(), ModelTierWorker) - if seat.Rung() != "default" || seat.Crew != "" { - t.Errorf("an untouched profile reads %q (%s), want the default rung as before", seat.Rung(), seat.Crew) - } -} - -// THE DEFAULT PRESET WINS EVERY TIE. An auto row matches whichever preset is -// being compared, so a profile whose OTHER rows match two presets at once has -// no single answer from the rows alone — and the answer is the default preset, -// balanced, the budget an undecided profile already runs at. -// -// The tie is built here rather than assumed: the worker holds auto and the -// other four rows are taken from max, and the test first states that those -// four are balanced's rows too, which is what makes this a tie at all. Should -// the tables move so the two presets differ somewhere above the worker, that -// first check fails and says so, rather than the test quietly pinning nothing. -func TestTheDefaultPresetWinsWhenTheStoredRowsMatchTwoPresets(t *testing.T) { - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - balanced, ok := CrewModelsForSource(DefaultCrewSource, CrewBalanced) - if !ok { - t.Fatalf("there is no %s preset in the %s family", CrewBalanced, DefaultCrewSource) - } - max, ok := CrewModelsForSource(DefaultCrewSource, CrewMax) - if !ok { - t.Fatalf("there is no %s preset in the %s family", CrewMax, DefaultCrewSource) - } - rows := map[string]string{KeyTierWorkerModel: AutoValue} - for _, tier := range ModelTiers { - if tier == ModelTierWorker { - continue - } - if balanced[tier] != max[tier] { - t.Fatalf("the %s seat differs between balanced (%s) and max (%s), so an auto worker is no longer a tie", - tier, balanced[tier], max[tier]) - } - rows[tierKeyFor(tier)] = max[tier] - } - if balanced[ModelTierWorker] == max[ModelTierWorker] { - t.Fatalf("balanced and max name the same worker, so the two presets are not two") - } - - dir := writeProfileRows(t, rows) - restore := AutoModels - AutoModels = func() []catalog.Model { return autoTestRows() } - seat := TierSeatAt(dir, ModelTierWorker) - AutoModels = restore - - if seat.Crew != CrewBalanced { - t.Errorf("rows matching both presets read as %q, want %s, the default preset", seat.Crew, CrewBalanced) - } - if seat.Rung() != "crew "+CrewBalanced+", computed from the catalog" { - t.Errorf("the rung reads %q, want crew %s, computed from the catalog", seat.Rung(), CrewBalanced) - } - // And the same tie on the table rung, with nothing to compute from: the - // default preset decides the fallback id too, not just the budget word. - dir = writeProfileRows(t, rows) - seat = TierSeatAt(dir, ModelTierWorker) - if seat.Crew != CrewBalanced || seat.Source != SeatTable { - t.Errorf("with no catalog the tie reads %q (%s), want balanced on the table rung", seat.Crew, seat.Rung()) - } - if seat.Model != balanced[ModelTierWorker] { - t.Errorf("the tie's table id is %q, want balanced's own worker %q", seat.Model, balanced[ModelTierWorker]) - } -} - -// A SETTINGS ROW SHOWS THE MODEL RUNNING, because it reads through the -// resolver — and a profile that says auto nowhere is untouched by any of -// this: the same rows, the same ids, the same rungs. -func TestAutoNowhereChangesNothing(t *testing.T) { - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - restore := AutoModels - AutoModels = func() []catalog.Model { return autoTestRows() } - defer func() { AutoModels = restore }() - - dir := t.TempDir() - if err := ApplyCrew(dir, CrewBalanced); err != nil { - t.Fatal(err) - } - before := map[string]string{} - for _, tier := range ModelTiers { - before[tier] = TierModelAt(dir, tier) - } - raw, err := os.ReadFile(BudgetConfigPath(dir)) - if err != nil { - t.Fatal(err) - } - var stored map[string]any - if err := json.Unmarshal(raw, &stored); err != nil { - t.Fatal(err) - } - if len(stored) == 0 { - t.Fatal("the write stored nothing") - } - for tier, want := range before { - if got := TierModelAt(dir, tier); got != want { - t.Errorf("%s read %q then %q — a read rewrote a row", tier, want, got) - } - } - if got := CrewAt(dir); got != CrewBalanced { - t.Errorf("the crew word reads %q, want balanced", got) - } -} - -// THE SEAM'S TABLE RUNG IS THE PRESET'S OWN ID, not merely a non-empty -// answer: with the catalog gone, an auto row under rows that pin max reads -// to that preset's id for the tier, not to the default preset's. The worker -// is the pinned seat — it is the one cell where max differs from balanced — -// and the auto row rides the mastermind. -func TestTheTableRungAnswersThePresetTheRowsName(t *testing.T) { - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - models, _ := CrewModelsForSource(DefaultCrewSource, CrewMax) - rows := map[string]string{KeyTierMastermindModel: AutoValue} - for _, tier := range ModelTiers { - if tier != ModelTierMastermind { - rows[tierKeyFor(tier)] = models[tier] - } - } - dir := writeProfileRows(t, rows) - seat := TierSeatAt(dir, ModelTierMastermind) - if seat.Model != models[ModelTierMastermind] || seat.Source != SeatTable { - t.Errorf("with no catalog the seat reads %q (%s), want max's own mastermind id on the table rung", - seat.Model, seat.Rung()) - } -} - -// mustIndex parses a measurement document or fails the test. -func mustIndex(t *testing.T, doc string) *index.Index { - t.Helper() - x, err := index.Parse([]byte(doc)) - if err != nil { - t.Fatalf("parse index: %v", err) - } - return x -} - -// priorDocument is the smallest index document that rates one catalog row on -// the worker seat: a gaussian role_quality metric and a cell well above the -// document's min_installs. -const priorDocument = `{ - "schema": 1, - "generated": "2026-09-17", - "min_installs": 5, - "metrics": {"role_quality": {"kind": "gaussian", "dims": ["role", "model"]}}, - "cells": [ - {"metric": "role_quality", "role": "worker", "model": "a/cheap", "mean": 95, "sd": 5, "n": 1000} - ] -}` - -// THE INDEX REACHES THE PICK THROUGH THE PRIOR. An index whose role_quality -// metric rates one catalog row well above its published worker quality moves -// AutoPick's worker answer to that row, and with no index the pick answers -// exactly what the front answers, as before. -func TestAutoPickReadsAMeasuredQualityPrior(t *testing.T) { - restore := AutoIndex - defer func() { AutoIndex = restore }() - - rows := autoTestRows() - - AutoIndex = nil - frugal, _, _ := crewpick.Presets(crewpick.Front(autoCandidates(rows), crewpick.DefaultShapes(), crewpick.All)) - before, ok := AutoPick(ModelTierWorker, CrewSourceAll, CrewFrugal, rows) - if !ok || before != frugal.Worker { - t.Fatalf("with no index the worker pick reads %q, want the front's own %q", before, frugal.Worker) - } - - AutoIndex = func() *index.Index { return mustIndex(t, priorDocument) } - after, ok := AutoPick(ModelTierWorker, CrewSourceAll, CrewFrugal, rows) - if !ok { - t.Fatal("the pick answers nothing with the index in hand") - } - if after == before { - t.Fatalf("the rated row left the worker pick at %q; the index never reached the quality", after) - } - if after != "a/cheap" { - t.Fatalf("the rated row did not take the worker seat: got %q", after) - } -} - -// THE PRIOR IS THE ARGUMENT, AND NOTHING HIDES IT. AutoPickWith carries -// whatever prior it is given: nil leaves the worker on the front's own -// answer, and the same measured cell the AutoPick test reads moves it — which -// is exactly the difference between the `catalog` and `learn` pick words -// (crew.go's [CrewPickAt]). -func TestAutoPickWithCarriesThePriorItIsGiven(t *testing.T) { - rows := autoTestRows() - front, _, _ := crewpick.Presets(crewpick.Front(autoCandidates(rows), crewpick.DefaultShapes(), crewpick.All)) - bare, ok := AutoPickWith(ModelTierWorker, CrewSourceAll, CrewFrugal, rows, nil) - if !ok || bare != front.Worker { - t.Fatalf("with no prior the worker pick reads %q, want the front's own %q", bare, front.Worker) - } - measured, ok := AutoPickWith(ModelTierWorker, CrewSourceAll, CrewFrugal, rows, autoPrior(mustIndex(t, priorDocument))) - if !ok { - t.Fatal("the pick answers nothing with the prior in hand") - } - if measured == bare { - t.Fatalf("the measured cell left the worker pick at %q; the prior never reached the front", measured) - } - if measured != "a/cheap" { - t.Fatalf("the rated row did not take the worker seat: got %q", measured) - } -} - -// ── the two seams ─────────────────────────────────────────────────────────── - -// A BARE AUTO ROW AND THE PICK WORD'S OWN SEAT ANSWER ONE CATALOG THE SAME -// WAY. Under `picked from = catalog` both read the catalog's published -// figures alone — the Model Pool's measurements enter neither seat — and -// under `learn` both carry them, on the learned rung. The fixture still -// splits the two answers, asserted rather than assumed, so rows or a prior -// that stop moving the worker pick fail here saying so instead of pinning -// nothing. -func TestABareAutoRowAnswersTheCatalogThePickWordNames(t *testing.T) { - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - restoreModels, restoreIndex := AutoModels, AutoIndex - defer func() { AutoModels, AutoIndex = restoreModels, restoreIndex }() - AutoModels = func() []catalog.Model { return autoTestRows() } - AutoIndex = func() *index.Index { return mustIndex(t, priorDocument) } - - published, ok := AutoPickWith(ModelTierWorker, DefaultCrewSource, DefaultCrew, autoTestRows(), nil) - if !ok || published == "a/cheap" { - t.Fatalf("the fixture does not split the two answers: the published pick reads %q (ok %t), so the assertions below cannot tell them apart", published, ok) - } - - dir := writeProfileRows(t, map[string]string{ - KeyTierWorkerModel: AutoValue, - KeyCrewPick: CrewPickCatalog, - }) - model, source, preset := autoRow(dir, DefaultCrewSource, ModelTierWorker) - want, _ := pickedModel(ModelTierWorker, DefaultCrewSource, preset, CrewPickCatalog) - if model != want { - t.Fatalf("under picked from = catalog the bare auto row answers %q and the pick word's own seat answers %q — the two seams disagreed about what the catalog means", model, want) - } - if source != SeatComputed { - t.Errorf("under picked from = catalog the bare row's rung reads %q, want computed", source) - } - - dir = writeProfileRows(t, map[string]string{ - KeyTierWorkerModel: AutoValue, - KeyCrewPick: CrewPickLearn, - }) - model, source, preset = autoRow(dir, DefaultCrewSource, ModelTierWorker) - want, _ = pickedModel(ModelTierWorker, DefaultCrewSource, preset, CrewPickLearn) - if model != want || model != "a/cheap" { - t.Errorf("under picked from = learn the bare auto row answers %q and the pick word's own seat answers %q, want the measured pick a/cheap", model, want) - } - if source != SeatLearned { - t.Errorf("under picked from = learn the bare row's rung reads %q, want learned", source) - } -} - -// ── the install's own sheet ───────────────────────────────────────────────── - -// ownCells is the install's own evidence for the prior tests: one cell the -// index document also carries, one it does not, both on the worker seat. -func ownCells() []crewpick.Cell { - return []crewpick.Cell{ - {Role: "worker", Model: "a/cheap", Mean: 90, N: 3}, - {Role: "worker", Model: "b/only", Mean: 40, N: 1}, - } -} - -// With no index in hand the own sheet's cells are the whole prior — and a -// one-observation cell of this install's own is worth having, where an -// index's min_installs would have dropped it. -func TestAutoPriorReadsTheOwnCellsAloneWithNoIndex(t *testing.T) { - restore := AutoOwnCells - defer func() { AutoOwnCells = restore }() - AutoOwnCells = ownCells - - prior := autoPrior(nil) - if len(prior[crewpick.Worker]) != 2 { - t.Fatalf("the own cells made a prior of %d worker ratings, want two: %v", len(prior[crewpick.Worker]), prior) - } - if r := prior[crewpick.Worker]["b/only"]; r.Mean != 40 || r.N != 1 { - t.Fatalf("a one-observation own cell did not survive: %v", r) - } - if r := prior[crewpick.Worker]["a/cheap"]; r.Mean != 90 || r.N != 3 { - t.Fatalf("the shared cell came back %v, want mean 90 over 3", r) - } -} - -// A model both the index and the own sheet rate folds by observation count: -// the counts add, the mean is the mean of the means weighted by them, and the -// cells either one holds alone are carried beside it. -func TestAutoPriorFoldsTheOwnCellsIntoTheIndexByObservationCount(t *testing.T) { - restoreIndex, restoreOwn := AutoIndex, AutoOwnCells - defer func() { AutoIndex, AutoOwnCells = restoreIndex, restoreOwn }() - AutoIndex = func() *index.Index { return mustIndex(t, priorDocument) } - AutoOwnCells = ownCells - - prior := autoPrior(AutoIndex()) - shared := prior[crewpick.Worker]["a/cheap"] - wantMean := (1000*95.0 + 3*90.0) / 1003 - if shared.N != 1003 { - t.Fatalf("the shared cell folded to N %d, want 1003", shared.N) - } - if math.Abs(shared.Mean-wantMean) > 1e-9 { - t.Fatalf("the shared cell folded to mean %v, want %v", shared.Mean, wantMean) - } - if r := prior[crewpick.Worker]["b/only"]; r.Mean != 40 || r.N != 1 { - t.Fatalf("the own-only cell did not survive the fold: %v", r) - } -} - -// A nil seam changes nothing: the index's prior is exactly what it was, and -// with no index there is no prior at all. -func TestANilOwnCellsSeamLeavesThePriorAlone(t *testing.T) { - restoreIndex, restoreOwn := AutoIndex, AutoOwnCells - defer func() { AutoIndex, AutoOwnCells = restoreIndex, restoreOwn }() - AutoIndex = func() *index.Index { return mustIndex(t, priorDocument) } - AutoOwnCells = nil - - prior := autoPrior(AutoIndex()) - if len(prior[crewpick.Worker]) != 1 || prior[crewpick.Worker]["a/cheap"] != (crewpick.Rating{Mean: 95, N: 1000}) { - t.Fatalf("a nil seam changed the index prior: %v", prior) - } - if prior := autoPrior(nil); prior != nil { - t.Fatalf("a nil seam and no index made a prior: %v", prior) - } -} - -// The own sheet's scores are on the 0-100 scale a judge answers on whatever -// the index's metric says, so they are read even beside an index whose -// role_quality is not gaussian — while the index's own cells are not. -func TestAutoPriorReadsTheOwnCellsBesideANonGaussianIndex(t *testing.T) { - restoreIndex, restoreOwn := AutoIndex, AutoOwnCells - defer func() { AutoIndex, AutoOwnCells = restoreIndex, restoreOwn }() - AutoIndex = func() *index.Index { - return mustIndex(t, `{ - "schema": 1, - "generated": "2026-09-17", - "min_installs": 1, - "metrics": {"role_quality": {"kind": "tally", "dims": ["role", "model"]}}, - "cells": [{"metric": "role_quality", "role": "worker", "model": "a/cheap", "mean": 5, "n": 900}] - }`) - } - AutoOwnCells = ownCells - - prior := autoPrior(AutoIndex()) - if r, ok := prior[crewpick.Worker]["a/cheap"]; !ok || r.Mean != 90 || r.N != 3 { - t.Fatalf("the own cell was not read beside a non-gaussian index: %v ok %v", prior[crewpick.Worker]["a/cheap"], ok) - } - if _, ok := prior[crewpick.Worker]["b/only"]; !ok { - t.Fatalf("the second own cell did not survive: %v", prior) - } -} diff --git a/internal/config/budget.go b/internal/config/budget.go index 5a2ac90638..d60e4bc4b4 100644 --- a/internal/config/budget.go +++ b/internal/config/budget.go @@ -117,6 +117,17 @@ func writeProfileValue(profileDir, key string, value any) error { // still exactly one place that knows how a setting reaches the disk. var profileWriteMu sync.Mutex +// removeProfileKey, as a value in [writeProfileValues], takes the key OUT of +// the file rather than writing it. It exists for the rows whose absence is an +// answer of its own — an unpinned crew seat is a row that is not there, and +// writing an empty string would be the different answer "cleared" — so the +// same one transaction can remove one row while it writes another. +var removeProfileKey = profileKeyRemoval{} + +// profileKeyRemoval is [removeProfileKey]'s type, unexported so no caller can +// spell a removal any other way. +type profileKeyRemoval struct{} + func writeProfileValues(profileDir string, updates map[string]any) error { if len(updates) == 0 { return nil @@ -126,7 +137,12 @@ func writeProfileValues(profileDir string, updates map[string]any) error { // failure nobody could search for. key := errorKey(updates) encodedUpdates := make(map[string]json.RawMessage, len(updates)) + var removed []string for name, value := range updates { + if _, remove := value.(profileKeyRemoval); remove { + removed = append(removed, name) + continue + } encodedValue, err := json.Marshal(value) if err != nil { return fmt.Errorf("write config %s: %w", name, err) @@ -165,6 +181,9 @@ func writeProfileValues(profileDir string, updates map[string]any) error { for name, encodedValue := range encodedUpdates { values[name] = encodedValue } + for _, name := range removed { + delete(values, name) + } encoded, err := json.MarshalIndent(values, "", " ") if err != nil { return fmt.Errorf("write config %s: %w", key, err) diff --git a/internal/config/config.go b/internal/config/config.go index 9ce82f04aa..68188c2d42 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -372,6 +372,17 @@ var nonSettingProfileFields = []string{ // [LaneBorrowAt] reads it. Left off this list, every launch after the row // was saved told the person their profile carried an ignored key. LaneBorrowKey(LaneSlotTalk), + // THE CREW'S STANDING ROWS. They are written by /crew's shortcuts and + // panel and read by the router (crew.go), and no settings-registry row owns + // them: the panel is where a rule, a cap and the providers turned off are + // read beside the day's spend. + KeyCrewAllowed, + KeyCrewCap, + KeyCrewTaskCap, + // And the free-routes switch beside them, which the crew's providers list + // turns on and off. + KeyCrewFreeRoutes, + KeyCrewProvidersOff, } // retiredProfileKeys are top-level config.json keys that a shipped version once @@ -391,6 +402,12 @@ var retiredProfileKeys = map[string]bool{ "practice_demand_pct": true, // reader removed by 84ba8503e "propose_new_skills": true, // reader removed by 84ba8503e "attribution": true, // reader removed on 2026-09-23: signing has no off + // The retired crew rows are read once more, by the migration that removes + // them and says so in its own line (crewmigrate.go's [MigrateCrew]); the + // unread check must not say it a second time in a worse sentence. + legacyKeyCrew: true, // presets retired on 2026-09-24: seats are routed per task + legacyKeyCrewSource: true, // family row retired on 2026-09-24: `open` became the allowed rule + legacyKeyCrewPick: true, // pick row retired on 2026-09-24 } // retiredRowNotes are the retired keys a person set ON PURPOSE, each with the diff --git a/internal/config/credits_law_test.go b/internal/config/credits_law_test.go index 5cabb886bc..1dc15eb9ee 100644 --- a/internal/config/credits_law_test.go +++ b/internal/config/credits_law_test.go @@ -18,8 +18,8 @@ func TestEveryImplicitModelRungUsesTheBalanceCheck(t *testing.T) { {"credits.go", "ChatDefaultAt", "useFreeDefaultsAt"}, {"freecrew.go", "freeTierModelAt", "useFreeDefaultsAt"}, {"config.go", "load", "ChatDefaultAt"}, - {"seats.go", "tierSeatUnder", "freeTierModelAt"}, - {"seats.go", "ResolveSeats", "ChatDefaultAt"}, + {"seats.go", "TierSeatAt", "freeTierModelAt"}, + {"crewhealth.go", "crewHealthAt", "useFreeDefaultsAt"}, {"../../internal/tui3/modelservices.go", "reachableModelAfterDisconnect", "ChatDefaultAt"}, {"../../cmd/codeaf/chatv3.go", "v3TalkModel", "ChatDefaultAt"}, {"../../cmd/codeaf/chatv3.go", "openV3Launch", "v3TalkModel"}, @@ -64,7 +64,6 @@ func TestNoUnlistedRuntimeReaderUsesThePaidDefault(t *testing.T) { "cmd/codeaf/main.go:": "prints the compile-time fallback in environment help", "cmd/codeaf/competence.go:runCompetenceTo": "the separate competence command has its own fallback", "internal/config/credits.go:ChatDefaultAt": "owns the paid side of the profile-aware bottom rung", - "internal/config/seats.go:resolveSeat": "a deliberately cleared work row keeps its former fallback", "internal/tui3/credits.go:refreshCreditWarnings": "recognises an untouched paid conversation before moving it", } seen := make(map[string]int) diff --git a/internal/config/credits_test.go b/internal/config/credits_test.go index 059a2f8da1..a6c49ee472 100644 --- a/internal/config/credits_test.go +++ b/internal/config/credits_test.go @@ -6,8 +6,11 @@ import ( "strings" "testing" + "github.com/Agent-Field/codeaf/internal/catalog" "github.com/Agent-Field/codeaf/internal/credits" + "github.com/Agent-Field/codeaf/internal/crewroute" "github.com/Agent-Field/codeaf/internal/modelsource" + "github.com/Agent-Field/codeaf/internal/router" ) func TestCreditsRecordChangesImplicitDefaultsWithoutSavingModels(t *testing.T) { @@ -22,10 +25,10 @@ func TestCreditsRecordChangesImplicitDefaultsWithoutSavingModels(t *testing.T) { if err := WriteCreditsReading(dir, "secret-key", credits.Reading{Known: true, Low: true}); err != nil { t.Fatal(err) } - if SettingsGeneration() <= before || !CreditsLowAt(dir) || ChatDefaultAt(dir) != FreeChatModel || CrewAt(dir) != CrewFree { - t.Fatalf("low record did not move chat and crew defaults: chat=%q crew=%q", ChatDefaultAt(dir), CrewAt(dir)) + if SettingsGeneration() <= before || !CreditsLowAt(dir) || ChatDefaultAt(dir) != FreeChatModel || !crewHealthAt(dir).unaffordable[modelsource.DefaultID] { + t.Fatalf("low record did not move chat default and crew health: chat=%q", ChatDefaultAt(dir)) } - for tier, want := range freeCrewModels { + for tier, want := range freeHelperModels { if got := TierModelAt(dir, tier); got != want { t.Errorf("%s = %q, want %q", tier, got, want) } @@ -58,7 +61,7 @@ func TestCreditsRecordChangesImplicitDefaultsWithoutSavingModels(t *testing.T) { if err := WriteCreditsReading(dir, "secret-key", credits.Reading{Known: true}); err != nil { t.Fatal(err) } - if CreditsLowAt(dir) || ChatDefaultAt(dir) != DefaultModel || CrewAt(dir) != DefaultCrew || CreditsNeedRead(dir, "secret-key") { + if CreditsLowAt(dir) || ChatDefaultAt(dir) != DefaultModel || crewHealthAt(dir).unaffordable[modelsource.DefaultID] || CreditsNeedRead(dir, "secret-key") { t.Fatal("healthy re-read did not restore implicit defaults") } } @@ -75,7 +78,7 @@ func TestCannotPayKeepsTheWholeVendorSentence(t *testing.T) { } } -func TestFreeCrewIsAReadingAndExplicitRowsWin(t *testing.T) { +func TestFreeHelpersAreAReadingAndExplicitRowsWin(t *testing.T) { dir := t.TempDir() if err := WriteAPIKey(dir, "key"); err != nil { t.Fatal(err) @@ -89,35 +92,30 @@ func TestFreeCrewIsAReadingAndExplicitRowsWin(t *testing.T) { if ChatModelAt(dir) != "openai/gpt-4" { t.Fatal("saved talk model changed") } - if err := ApplyCrew(dir, CrewMax); err != nil { - t.Fatal(err) - } - if CrewAt(dir) != CrewMax { - t.Fatalf("explicit crew became %q", CrewAt(dir)) - } - if TierModelAt(dir, ModelTierWorker) == FreeChatModel { - t.Fatal("a written seat was replaced") - } - if strings.Split(freeCrewModels[ModelTierHigh], "/")[0] == strings.Split(freeCrewModels[ModelTierWorker], "/")[0] { - t.Fatal("careful and worker share a vendor") - } - for _, model := range freeCrewModels { + for _, model := range freeHelperModels { if !strings.HasSuffix(model, ":free") { t.Fatalf("%q is not free", model) } } + if strings.Split(freeHelperModels[ModelTierReflex], "/")[0] == strings.Split(freeHelperModels[ModelTierLow], "/")[0] { + t.Fatal("reflex and small work share a vendor") + } } -func TestLowBalanceDoesNotReplaceClearedStoredOrComputedSeats(t *testing.T) { +// A ROW SOMEBODY WROTE IS NEVER REPLACED by the low reading: a crew pin stays +// the pin, a written helper row stays theirs, a cleared helper row stays +// empty, and no crew seat is handed a model this build chose. +func TestLowBalanceDoesNotReplaceWrittenOrClearedRows(t *testing.T) { for _, tc := range []struct { - name string - write func(string) error - tier string - want string + name string + key string + raw string + tier string + want string }{ - {"cleared work row", func(dir string) error { return writeProfileValue(dir, KeyTierWorkerModel, "") }, ModelTierWorker, ""}, - {"stored crew word", func(dir string) error { return writeProfileValue(dir, KeyCrew, CrewMax) }, ModelTierWorker, crewAllModels[CrewMax][ModelTierWorker]}, - {"catalog pick", func(dir string) error { return SetCrewPick(dir, CrewPickCatalog) }, ModelTierReflex, DefaultReflexModel}, + {"pinned worker", KeyTierWorkerModel, "openai/gpt-4", ModelTierWorker, "openai/gpt-4"}, + {"written reflex row", KeyTierReflexModel, "vendor/reflex", ModelTierReflex, "vendor/reflex"}, + {"cleared small-work row", KeyTierLowModel, "", ModelTierLow, ""}, } { t.Run(tc.name, func(t *testing.T) { dir := t.TempDir() @@ -127,14 +125,11 @@ func TestLowBalanceDoesNotReplaceClearedStoredOrComputedSeats(t *testing.T) { if err := WriteCreditsReading(dir, "key", credits.Reading{Known: true, Low: true}); err != nil { t.Fatal(err) } - if err := tc.write(dir); err != nil { + if err := writeProfileValue(dir, tc.key, tc.raw); err != nil { t.Fatal(err) } if got := TierModelAt(dir, tc.tier); got != tc.want { - t.Fatalf("explicit seat became %q, want %q", got, tc.want) - } - if tc.name == "cleared work row" && ResolveSeats(dir, "", "").Work.Model != DefaultModel { - t.Fatal("cleared headless work row became the free implicit default") + t.Fatalf("explicit row became %q, want %q", got, tc.want) } }) } @@ -154,7 +149,7 @@ func TestLowCreditsBelongOnlyToTheCurrentKey(t *testing.T) { if err := WriteAPIKey(dir, "key-B"); err != nil { t.Fatal(err) } - if CreditsLowAt(dir) || ChatDefaultAt(dir) != DefaultModel || CrewAt(dir) != DefaultCrew { + if CreditsLowAt(dir) || ChatDefaultAt(dir) != DefaultModel || crewHealthAt(dir).unaffordable[modelsource.DefaultID] { t.Fatal("key A's low record changed key B's defaults before B was read") } if !CreditsNeedRead(dir, APIKeyAt(dir)) { @@ -180,3 +175,41 @@ func TestFreeModelNeedsASuffixOrAKnownZeroTariff(t *testing.T) { } } } + +// A BALANCE READ AS LOW PUTS THE CREW ON FREE ROUTES before any call: the +// three seats are routed to free pools, the decision says so once, and no +// paid OpenRouter route is tried first. A healthy reading routes as usual. +func TestALowBalanceRoutesTheCrewToFreePools(t *testing.T) { + dir := crewProfile(t) + rows := CrewCatalog() + rows = append(rows, catalog.Model{ID: "z-ai/glm-5.3-flash:free", OpenWeights: true, IntelligenceIndex: 41.8, CodingIndex: 71.5, + AgenticIndex: 50.9, ArenaElo: 1348, ContextLength: 1310720, Parameters: []string{"tools"}}) + CrewCatalog = func() []catalog.Model { return rows } + var history []router.CrewRouteOutcome + withRouteHistory(t, &history) + if err := WriteCreditsReading(dir, APIKeyAt(dir), credits.Reading{Known: true, Low: true}); err != nil { + t.Fatal(err) + } + d, err := RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: fixTask}}) + if err != nil { + t.Fatal(err) + } + for _, seat := range crewroute.Seats { + if pick := d.Seat(seat); !crewroute.IsFree(pick.Send) { + t.Errorf("%s on a low balance sends %q, want a free pool", seat, pick.Send) + } + } + if strings.Count(d.Note, "free routes in use (may log prompts)") != 1 { + t.Errorf("the decision note is %q, want the free-routes notice once", d.Note) + } + if err := WriteCreditsReading(dir, APIKeyAt(dir), credits.Reading{Known: true}); err != nil { + t.Fatal(err) + } + d, err = RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: fixTask}}) + if err != nil { + t.Fatal(err) + } + if crewroute.IsFree(d.Seat(crewroute.Worker).Send) || strings.Contains(d.Note, "free routes") { + t.Errorf("a healthy balance still routed free: %s · %q", d.Seat(crewroute.Worker).Send, d.Note) + } +} diff --git a/internal/config/crew.go b/internal/config/crew.go index 03b3f655d4..5acf6a1d17 100644 --- a/internal/config/crew.go +++ b/internal/config/crew.go @@ -1,729 +1,1358 @@ package config import ( + "encoding/json" + "errors" "fmt" "strings" + "time" + "github.com/Agent-Field/codeaf/internal/catalog" + "github.com/Agent-Field/codeaf/internal/crewroute" + "github.com/Agent-Field/codeaf/internal/modelsource" "github.com/Agent-Field/codeaf/internal/roles" + "github.com/Agent-Field/codeaf/internal/router" ) -// THE CREW: five classes of model, answered as one word. +// THE CREW: THREE SEATS, EACH PINNED OR PICKED FOR THE TASK IN FRONT OF IT. // -// The five tier rows are the honest shape of the decision — a class of call has -// a class of model, and a new call joins a class instead of growing a knob — and -// they are still five model ids somebody has to know. Nobody arrives at a -// settings sheet wanting to name five model ids. They arrive wanting to spend -// pennies, or wanting to spend what it takes. So there is one row above the five -// that takes that sentence and writes all of them. +// A task is done by a crew of three — the worker that does it, the planner +// that cuts and steers it, the checker that reads the result against what was +// asked — and which model sits each seat is decided PER TASK by the router +// (internal/crewroute), from the kind of work the task is. Nothing about that +// decision is stored. What IS stored, and is the whole of what a person +// configures, is what the router may pick from: // -// THREE PRESETS AND NO MORE. A fourth would be a fourth thing to explain, and -// the axis they move along has exactly three interesting points: everything -// cheap, one model thinking over cheap ones working, everything capable and -// asked to think. +// - a PIN per seat, `model` or `model@provider`, which that seat always runs +// and the router never overrides; +// - the ALLOWED MODELS, one rule ([crewroute.ParseAllowed]) — `all` unless +// somebody narrowed it; +// - an optional DAILY CAP on what crews spend, which the router paces toward; +// - the PROVIDERS TURNED OFF, a set of connections the crew may not route +// through — empty unless somebody switched one off. // -// TWO FAMILIES BEHIND THE SAME THREE WORDS. Every preset exists twice: once in -// the all family, the same three words resolved over the whole catalog with -// closed and frontier models in it; and once in the open family, every seat an -// open-weight model, for the person who wants the three words to stay on that -// shelf whatever the catalog does around it. Which family the words draw from -// is one row, [KeyCrewSource], and all is what it reads when nobody has -// answered it. Both the derived reading and the write resolve through the row -// ([CrewSourceAt]), so the word on the sheet can never mean one family while -// the five values it summarizes were drawn from the other. +// The providers a crew can use are the connections the person made, read off +// the profile the way every call reads them; what persists is only which of +// them the crew leaves alone, so a connection made tomorrow is on the day it +// is made. That is the one-sentence model the whole feature is built on — the +// /crew panel says what is allowed and it persists; the words in an ask say +// how hard to try that one task; nothing else sticks. // -// THE PRESET IS DERIVED AND NEVER STORED. [CrewAt] reads the five live tier -// values and answers which preset they are, or "custom". A stored word would be -// a claim about five other rows that any one of them could falsify, and a sheet -// that said "balanced" over a hand-pinned mastermind would be lying in exactly -// the place somebody went to check. This is the one-source-of-truth law applied -// to a summary: a summary that can drift from what it summarizes is not a -// summary. -// -// THE BUILD WRITES NO WORD, AND READS ONE THAT IS THERE ([storedCrewWord]). Every -// profile this product shapes derives the preset from the rows; a run that wrote -// the word itself — a harness, a hand edit — named a budget, and a word on disk -// that nothing read is a crew word that reached nobody. The word is the budget -// when present, and the class rows under it are then that run's own pins. - -// The preset words. They are the values [KeyCrew] takes, and they are strings -// on disk in the same sense every other choice row's words are — spelled here -// once, read by the row, the command and the manual. +// THE PINS LIVE IN THE TIER ROWS THE CREW HAS ALWAYS LIVED IN. The worker +// seat is the `worker` tier row, the planner is the `mastermind` row, the +// checker is the `high` row — the rows the role ladder already reads, so the +// auxiliary calls that ride those tiers (the brief a task is shaped into, an +// image read for a model that cannot see one, the plan of an adaptive run) +// follow a pin without a second place to write it. An unwritten row is `auto`: +// the seat is routed. The reflex and small-work rows are not crew seats and +// keep their shipped defaults. + +// Crew rows, spelled once. const ( - CrewFrugal = "frugal" - CrewBalanced = "balanced" - CrewMax = "max" - // CrewCustom is a READING and never a write. It is what the row says when - // the five tier values are somebody's own arrangement rather than one of the - // three, which is what happens the moment a person answers one tier row - // directly. It is deliberately absent from [CrewPresets]: "set the crew to - // custom" is not a sentence with a meaning — custom is what you get, not - // what you ask for. - CrewCustom = "custom" + // KeyCrewAllowed is the allowed-models rule. PROFILE-ONLY for the worker + // row's own reason: a repository that could widen it could send a + // visitor's work, and their credit, to a model nobody on that machine chose. + KeyCrewAllowed = "models.crew.allowed" + // KeyCrewCap is the daily cap on what crews spend, in dollars; absent or + // zero is no cap. + KeyCrewCap = "models.crew.cap" + // KeyCrewTaskCap is the most one task may spend, in dollars; absent or + // zero is [CrewTaskCapDefault]. + KeyCrewTaskCap = "models.crew.task_cap" + // KeyCrewFreeRoutes is whether the crew may use providers' free pools + // (a `:free` route of a model). PROFILE-ONLY and OFF unless somebody + // turned it on: a free pool may log or train on what it is sent, which is + // a choice about a person's code, not a price. + KeyCrewFreeRoutes = "models.crew.free_routes" + // KeyCrewProvidersOff is the connected providers a person turned off for + // the crew, a list of provider ids; absent is every provider on. It is a + // row of its own beside the allowed rule and never a `-x` inside it + // (crewroute's providers.go says why), and PROFILE-ONLY for the allowed + // rule's reason. + KeyCrewProvidersOff = "models.crew.providers.off" ) -// CrewPresets lists the words a person may WRITE, cheapest first. [CrewCustom] -// is not among them; see its own comment. -var CrewPresets = []string{CrewFrugal, CrewBalanced, CrewMax} +// CrewAuto is the word a seat reads when it is not pinned. +const CrewAuto = "auto" -// DefaultCrew is what a profile nobody has touched reads. It is balanced because -// the five shipped tier defaults ARE the balanced row of the DEFAULT FAMILY — -// see [crewAllModels] — and that identity is asserted by a test rather than -// trusted. -const DefaultCrew = CrewBalanced +// CrewCommand is the one door a person reaches the crew through, spelled once +// so a sentence naming it cannot drift from the command that answers. +const CrewCommand = "/crew" -// The two families the preset words can draw from. They are the values -// [KeyCrewSource] takes, spelled here once and read by the row, the resolver -// and the manual. -const ( - // CrewSourceOpen is the open-weight family. - CrewSourceOpen = "open" - // CrewSourceAll is the whole catalog, closed and frontier models included, - // and the family a profile that has answered nothing resolves. - CrewSourceAll = "all" -) +// CrewSeatTier is the tier row a seat's pin is written in. +func CrewSeatTier(seat crewroute.Seat) string { + switch seat { + case crewroute.Planner: + return ModelTierMastermind + case crewroute.Checker: + return ModelTierHigh + } + return ModelTierWorker +} -// CrewSources lists them, open first, which is the order the row widens in: the -// narrower shelf, then the whole catalog. The default is the second of them, -// [DefaultCrewSource], because this list is about width and not about which one -// a profile starts on. -var CrewSources = []string{CrewSourceOpen, CrewSourceAll} +// CrewSeatKey is the registry row a seat's pin is written in — the settings +// sheet's one way of telling a seat's row from the other tier rows. +func CrewSeatKey(seat crewroute.Seat) string { return tierKeyFor(CrewSeatTier(seat)) } -// DefaultCrewSource is all: the three words are read off the whole catalog -// unless the row says otherwise, and the five shipped tier defaults are that -// family's balanced row. -const DefaultCrewSource = CrewSourceAll +// CrewTierSeat is the seat a tier row pins, false for a tier that is not a +// crew seat (reflex, small work). +func CrewTierSeat(tier string) (crewroute.Seat, bool) { + switch tier { + case ModelTierWorker: + return crewroute.Worker, true + case ModelTierMastermind: + return crewroute.Planner, true + case ModelTierHigh: + return crewroute.Checker, true + } + return "", false +} -// ── where the seats are picked from ──────────────────────────────────────── +// ParseCrewSeat reads a person's word for a seat. Only the three seat names +// are accepted: the tier words behind them are machinery. +func ParseCrewSeat(word string) (crewroute.Seat, bool) { + switch strings.ToLower(strings.TrimSpace(word)) { + case "worker": + return crewroute.Worker, true + case "planner": + return crewroute.Planner, true + case "checker": + return crewroute.Checker, true + } + return "", false +} -// THE THIRD ROW THE CREW WORDS ARE ANSWERED THROUGH. The crew row says how -// much to spend and the family row says which shelf those budgets name; the -// pick row says where the models for that money come from when a tier row -// does not hold a model id of its own: -// -// - `table` — the rows this build measured and shipped ([crewModels] and -// [crewAllModels]), which is what an unwritten seat has always read; -// - `catalog` — the same three budgets recomputed off the catalog's own -// published prices and scores, on every read, with no measurement of -// anybody's own runs in it ([AutoPickWith] with no prior); -// - `learn` — the catalog computation plus the Model Pool's measurements -// and the person's own judged runs, carried as a quality prior -// ([autoPrior]). -// -// THE DEFAULT IS THE TABLE because the table is what a profile has always -// read: an unwritten seat names the preset's own row, and nothing about a -// profile that has answered nothing moves until somebody answers a row. The -// other two words are an opt-in to a read that keeps moving — a seat that -// follows the catalog follows it whether or not the shipped rows do — and a -// person has to say so. -// -// A PICK NEVER OVERRIDES A MODEL ID. The row answers for the seats a person -// did not name, and the seats they did — written by hand, or by a preset — -// keep their ids until the crew is picked again, except that a row holding -// the preset's own table value is the preset answering, not a person pinning -// one model by id. The rule is [pickedSeat]'s to apply and the manual's to -// state. -const ( - // CrewPickTable is the measured rows this build ships, and the default. - CrewPickTable = "table" - // CrewPickCatalog is the catalog's own published figures, with nothing - // measured on top. - CrewPickCatalog = "catalog" - // CrewPickLearn is the catalog computation plus the Model Pool's - // measurements and the person's own judged runs. - CrewPickLearn = "learn" -) +// ── pins ──────────────────────────────────────────────────────────────────── -// CrewPicks lists the words a person may WRITE, narrowest first: the shipped -// table, then the catalog on its own, then the catalog with what runs -// measured. A pick word is not a preset and names no budget — the crew row -// above it still does that. -var CrewPicks = []string{CrewPickTable, CrewPickCatalog, CrewPickLearn} +// CrewPin is one pinned seat: a model id, and the provider route it was +// pinned to when somebody wrote `model@provider`. +type CrewPin struct { + Model string + Provider string +} -// DefaultCrewPick is the table: the rows this build measured are where an -// unwritten seat's model comes from until somebody answers the row. -const DefaultCrewPick = CrewPickTable +// String is the pin the way it is written and stored: `model[@provider]`. +func (p CrewPin) String() string { + if p.Provider == "" { + return p.Model + } + return p.Model + "@" + p.Provider +} -// knownCrewPick folds a word and says whether it is one of the picks this -// build knows. It is the ONE place the fold is spelled: [SetCrewPick] and the -// ladder's [pickedSeat] both go through it, so a pick added to [CrewPicks] is -// accepted everywhere at once. A reader folds a word it does not know to the -// default pick; a writer refuses it. -func knownCrewPick(pick string) (string, bool) { - pick = strings.ToLower(strings.TrimSpace(pick)) - for _, known := range CrewPicks { - if pick == known { - return known, true +// ParseCrewPin reads a written pin. Blank and `auto` are not pins — auto is +// true — and a pin whose model carries a thinking level (`vendor/model:high`) +// is checked by the gate every tier row shares ([ValidateTierValue]). +func ParseCrewPin(raw string) (pin CrewPin, auto bool, err error) { + raw = strings.TrimSpace(raw) + if raw == "" || strings.EqualFold(raw, CrewAuto) { + return CrewPin{}, true, nil + } + model, provider := raw, "" + if at := strings.LastIndex(raw, "@"); at > 0 { + model, provider = strings.TrimSpace(raw[:at]), strings.ToLower(strings.TrimSpace(raw[at+1:])) + if provider == "" { + return CrewPin{}, false, fmt.Errorf("%q: name the provider after @, or leave the @ off", raw) } } - return "", false + // A FREE POOL IS A ROUTE, NOT A THINKING LEVEL: `…:free` names the model's + // free route, and only what is left of the id is a tier value. + if err := ValidateTierValue(strings.TrimSuffix(model, ":free")); err != nil { + return CrewPin{}, false, err + } + return CrewPin{Model: model, Provider: provider}, false, nil } -// normalCrewPick folds a pick word to one of the three this build knows, -// reading a word it does not know as the default pick. -func normalCrewPick(pick string) string { - if known, ok := knownCrewPick(pick); ok { - return known +// CrewPinAt is one seat's pin, false when the seat is auto. +// +// A row that says `auto`, a row that is empty and a retired preset word are +// all auto; any model id is a pin. A row that does not parse is auto too, and the +// panel says so, because a seat that silently ran a half-read id would be the +// one thing worse than a seat that ignores it. +func CrewPinAt(profileDir string, seat crewroute.Seat) (CrewPin, bool) { + value, held := persistedString(profileDir, tierKeyFor(CrewSeatTier(seat))) + if !held { + return CrewPin{}, false } - return DefaultCrewPick + pin, auto, err := ParseCrewPin(value) + if auto || err != nil { + return CrewPin{}, false + } + return pin, true } -// CrewPickAt is where the seats are picked from on this profile, -// [DefaultCrewPick] when the row is absent. A word this build does not know -// reads as the default pick, silently, the way a retired choice reads -// everywhere else on this sheet. -func CrewPickAt(profileDir string) string { - if value, ok := persistedString(profileDir, KeyCrewPick); ok { - return normalCrewPick(value) +// crewSeatRow is a seat's settings row: the pin as written, or empty — which +// the row draws as `auto`. +func crewSeatRow(profileDir string, seat crewroute.Seat) string { + if pin, ok := CrewPinAt(profileDir, seat); ok { + return pin.String() } - return DefaultCrewPick + return "" } -// AnyTierAutoAt answers whether any of the five tier rows reads `auto` on this -// profile — written directly, or reaching the word through an older row -// ([crewRow]) — which is the second way a seat resolution is computed from the -// catalog's rows, beside the pick row ([CrewPickAt]). A door that resolves -// seats asks both before it resolves, because both answers are computed from -// the rows the process already holds, and a door that asks before they land -// reads the family's table row over a profile that never chose it. -func AnyTierAutoAt(profileDir string) bool { - for _, tier := range ModelTiers { - if model, _, _, _ := crewRow(profileDir, tier); IsAuto(model) { - return true +// CrewPinsAt is every pinned seat. +func CrewPinsAt(profileDir string) map[crewroute.Seat]CrewPin { + pins := map[crewroute.Seat]CrewPin{} + for _, seat := range crewroute.Seats { + if pin, ok := CrewPinAt(profileDir, seat); ok { + pins[seat] = pin } } - return false + return pins } -// SetCrewPick writes the pick row ALONE, in one file write. The word is -// refused the way every choice row refuses one, so a typo cannot land a pick -// nothing reads. It writes no tier row: the pick says where seats are read -// from, and the seats keep the ids on disk until the crew is picked again. -func SetCrewPick(profileDir, pick string) error { - known, ok := knownCrewPick(pick) - if !ok { - return fmt.Errorf("pick one of: %s", strings.Join(CrewPicks, ", ")) +// SetCrewPin pins one seat, IN ONE FILE WRITE. `auto` or blank unpins it. +// +// A PIN OUTSIDE THE ALLOWED MODELS IS REFUSED WITH THE REASON, never written +// and quietly ignored: the rule is the person's own and a pin that broke it +// would be the one decision on the panel that contradicted another. A pin +// naming a provider must name one that is connected, for the same reason. +func SetCrewPin(profileDir string, seat crewroute.Seat, raw string) error { + pin, auto, err := ParseCrewPin(raw) + if err != nil { + return err + } + if auto { + return ClearCrewPin(profileDir, seat) + } + if err := CrewPinAllowed(profileDir, pin); err != nil { + return err + } + values := map[string]any{tierKeyFor(CrewSeatTier(seat)): pin.String()} + // A profile carrying retired rows is migrated in the same write. + for key, value := range legacyCrewClearing(profileDir) { + if _, set := values[key]; !set { + values[key] = value + } } - return writeProfileValue(profileDir, KeyCrewPick, known) + return writeProfileValues(profileDir, values) } -// crewModels is the open-weight table: one row per preset, one model per class. -// -// THE WORKER COLUMN IS THE DIAL. It holds glm-5.3-flash through balanced, and -// max is the preset that takes it to glm-5.3, because it is the seat that pays -// most of a task's bill, and a preset that moved every other seat while leaving -// it alone would change everything about a task except its cost. The careful -// column always sees images (the vision role rides it) and is a second vendor -// from balanced upward; frugal keeps worker and careful on the same -// glm-5.3-flash, because at that bill the open-weight front has no second -// vendor to take the careful seat. The reflex and low columns never vary: they -// are the same near-free models in all three presets, and a column that never -// varies is not a dial. -// -// HOW THE IDS WERE READ OFF, on 2026-09-16 and seat by seat. Every open-weight -// row of the catalog was placed on two axes: the expected bill that seat's own -// call shape runs up, built from the catalog's published prompt, completion and -// cache-read prices, against that seat's quality, taken from its published -// intelligence, coding and agentic indexes. THE CALL SHAPE IS PART OF THE -// PRICE. The worker and the careful seats were costed as LONG CACHED LOOPS — a -// large prompt read back turn after turn, so the cache-read price carries most -// of the weight — and the mastermind as ONE-SHOT CALLS, where the prompt is -// paid in full each time and there are few of them. Each preset then takes, for -// each seat, a point on the pareto front of that plot at the bill it is willing -// to run: nothing on the front costs less at the same quality, and nothing at -// the same bill scores higher. -// -// That is why the columns do not climb together. Under one-shot pricing -// glm-5.3-flash is on the front at frugal's bill and glm-5.3 is the next point -// above it, so the mastermind column reads flash, glm-5.3, glm-5.3; under -// long-loop pricing the same plot puts kimi-k3 on the careful seat from -// balanced upward, which is also the second vendor that seat has to be. -// -// No closed model is here: the open family is the shelf that stands on the -// open-weight rows alone, and the `all` family is where a closed model goes. -var crewModels = map[string]map[string]string{ - CrewFrugal: { - ModelTierReflex: "mistralai/mistral-nemo", - ModelTierLow: "deepseek/deepseek-v4-flash-0731", - ModelTierWorker: "z-ai/glm-5.3-flash", - ModelTierHigh: "z-ai/glm-5.3-flash", - ModelTierMastermind: "z-ai/glm-5.3-flash", - }, - CrewBalanced: { - ModelTierReflex: "mistralai/mistral-nemo", - ModelTierLow: "deepseek/deepseek-v4-flash-0731", - ModelTierWorker: "z-ai/glm-5.3-flash", - ModelTierHigh: "moonshotai/kimi-k3", - ModelTierMastermind: "z-ai/glm-5.3", - }, - CrewMax: { - ModelTierReflex: "mistralai/mistral-nemo", - ModelTierLow: "deepseek/deepseek-v4-flash-0731", - ModelTierWorker: "z-ai/glm-5.3", - ModelTierHigh: "moonshotai/kimi-k3", - ModelTierMastermind: "z-ai/glm-5.3", - }, -} - -// crewAllModels is the same three presets answered from the whole catalog -// rather than its open-weight shelf, which is what the `all` family of -// [KeyCrewSource] draws from AND WHAT A PROFILE THAT HAS ANSWERED NOTHING -// RESOLVES ([DefaultCrewSource]). The careful column is a DIFFERENT VENDOR -// from the worker in every preset here — the open family's frugal row is the -// one standing exception — and the reflex and low columns still never vary. -// Closed models live here and only here. -// -// The ids come off the same plot [crewModels] describes, run on 2026-09-16 over -// every row of the catalog rather than the open-weight ones: expected task bill -// against seat quality, the bill built from the published prompt, completion -// and cache-read prices under each seat's own call shape — the worker and the -// careful seats as long cached loops, the mastermind as one-shot calls — and -// the quality from the published intelligence, coding and agentic indexes. -// -// THE WORKER STAYS ON glm-5.3-flash THROUGH BALANCED, and that is the whole -// shape of this table. The worker seat carries most of a task's tokens, so a -// step there multiplies through the entire bill while a step on the careful or -// the mastermind seat is paid a handful of times. The money therefore goes to -// the two low-volume seats first: frugal to balanced moves the careful seat to -// claude-fable-5.1 and the mastermind to claude-opus-5, and max moves the -// worker itself, with the careful seat staying on fable beside it. -var crewAllModels = map[string]map[string]string{ - CrewFrugal: { - ModelTierReflex: "google/gemini-2.5-flash", - ModelTierLow: "deepseek/deepseek-v4-flash-0731", - ModelTierWorker: "z-ai/glm-5.3-flash", - ModelTierHigh: "qwen/qwen3.8-max-0902", - ModelTierMastermind: "z-ai/glm-5.3-flash", - }, - CrewBalanced: { - ModelTierReflex: "google/gemini-2.5-flash", - ModelTierLow: "deepseek/deepseek-v4-flash-0731", - ModelTierWorker: "z-ai/glm-5.3-flash", - ModelTierHigh: "anthropic/claude-fable-5.1", - ModelTierMastermind: "anthropic/claude-opus-5", - }, - CrewMax: { - ModelTierReflex: "google/gemini-2.5-flash", - ModelTierLow: "deepseek/deepseek-v4-flash-0731", - ModelTierWorker: "z-ai/glm-5.3", - ModelTierHigh: "anthropic/claude-fable-5.1", - ModelTierMastermind: "anthropic/claude-opus-5", - }, -} - -// crewTableFor is the family one source word names. It is total: `open` names -// the open-weight family, and every other reading (blank, misspelt, a word a -// later build retired) names the catalog-wide one, because an unreadable answer -// resolves the same family an absent answer does ([DefaultCrewSource]). -func crewTableFor(source string) map[string]map[string]string { - if normalCrewSource(source) == CrewSourceOpen { - return crewModels - } - return crewAllModels -} - -// knownCrewSource folds a word and says whether it is one of the families this -// build knows. It is the ONE place the fold is spelled: [normalCrewSource], -// [ApplyCrewUnder] and [SetCrewSource] all go through it, so a family added to -// [CrewSources] is accepted everywhere at once. A reader folds a word it does -// not know to the default family; a writer refuses it. -func knownCrewSource(source string) (string, bool) { - source = strings.ToLower(strings.TrimSpace(source)) - for _, known := range CrewSources { - if source == known { - return known, true +// ClearCrewPin unpins one seat: the row is removed, and the seat is routed. +func ClearCrewPin(profileDir string, seat crewroute.Seat) error { + values := legacyCrewClearing(profileDir) + values[tierKeyFor(CrewSeatTier(seat))] = removeProfileKey + return writeProfileValues(profileDir, values) +} + +// ClearCrewPins unpins every seat in one write. +func ClearCrewPins(profileDir string) error { + values := legacyCrewClearing(profileDir) + for _, seat := range crewroute.Seats { + values[tierKeyFor(CrewSeatTier(seat))] = removeProfileKey + } + return writeProfileValues(profileDir, values) +} + +// CrewPinAllowed says why a pin may not be written, or nil. The model must be +// one the allowed rule admits — by its figures when the catalog knows it, by +// name otherwise — and a pinned provider must be connected and must reach it. +func CrewPinAllowed(profileDir string, pin CrewPin) error { + rule := CrewAllowedAt(profileDir) + model, known := crewCatalogModel(pin.Model) + switch { + case known && !rule.AdmitsModel(model): + return fmt.Errorf("%s is outside the models you allow (%s) · widen them with /crew models +%s", + pin.Model, rule.String(), crewroute.ShortModel(pin.Model)) + case !known && rule.Base != crewroute.BaseAll && !rule.NamesModel(pin.Model): + return fmt.Errorf("%s is not in the catalog, so the rule %s cannot admit it by price or licence · name it with /crew models +%s", + pin.Model, rule.String(), pin.Model) + case !known && rule.Base == crewroute.BaseAll && !rule.AdmitsModel(crewroute.Model{ID: pin.Model}): + return fmt.Errorf("%s is outside the models you allow (%s)", pin.Model, rule.String()) + } + if pin.Provider != "" { + provider, ok := crewProviderByID(CrewProvidersAt(profileDir), pin.Provider) + if !ok { + return fmt.Errorf("%s is not a connected provider · connect it with /connect, or pin the model without @%s", pin.Provider, pin.Provider) + } + if !rule.AdmitsRoute(provider.ID) { + return fmt.Errorf("%s is a provider your allowed models exclude (%s)", provider.ID, rule.String()) + } + if !provider.On { + return fmt.Errorf("%s is turned off for the crew · turn it on in /crew's providers row", provider.ID) + } + if _, ok := provider.route(pin.Model, model, known); !ok { + return fmt.Errorf("%s does not serve %s", provider.Name, pin.Model) } } - return "", false + return nil } -// normalCrewSource folds a source word to one of the two this build knows, -// reading a word it does not know as the default family. -func normalCrewSource(source string) string { - if known, ok := knownCrewSource(source); ok { - return known +// ── allowed models and the cap ────────────────────────────────────────────── + +// CrewAllowedAt is the allowed-models rule. A rule that does not parse — a +// hand edit — reads as the default, the way a retired choice reads everywhere +// else on this sheet; the writer refuses one. +func CrewAllowedAt(profileDir string) crewroute.Allowed { + value, _ := persistedString(profileDir, KeyCrewAllowed) + rule, err := crewroute.ParseAllowed(value) + if err != nil { + rule, _ = crewroute.ParseAllowed(crewroute.DefaultAllowed) } - return DefaultCrewSource + return rule } -// CrewSourceAt is which family the preset words draw from on this profile, -// [DefaultCrewSource] when the row is absent. A word this build does not know -// reads as the default family, silently, the way a retired choice reads -// everywhere else on this sheet. -func CrewSourceAt(profileDir string) string { - if value, ok := persistedString(profileDir, KeyCrewSource); ok { - return normalCrewSource(value) +// SetCrewAllowed writes the rule in its canonical spelling, refusing one that +// does not parse and one that would leave a pinned seat outside it. +func SetCrewAllowed(profileDir, raw string) error { + rule, err := crewroute.ParseAllowed(raw) + if err != nil { + return err } - return DefaultCrewSource + return writeCrewAllowed(profileDir, rule) } -// storedCrewWord is the crew word a profile STORES on the crew row, when it -// names one of this build's presets. -// -// THIS BUILD NEVER WRITES IT. The crew row is derived from the five tier rows, -// not stored ([KeyCrew] argues it), so on every profile this product shapes the -// row is absent. But a run that writes ONE word into config.json instead of the -// five rows — a harness driven arm, a hand edit — is asking for a budget, and a -// word sitting on disk that nothing reads is a crew word that reached nobody. -// The word is read here, at the one place a preset is decided ([crewPresetUnder] -// and [crewStoredAt]), so a word that is present seats the crew it names whether -// the five rows were written or not. +// ModifyCrewAllowed is `/crew models +x` and `-x`: the rule with one word +// added or taken away. +func ModifyCrewAllowed(profileDir string, add bool, word string) error { + word = strings.TrimSpace(word) + if word == "" { + return errors.New("name a model or a provider after the + or -") + } + return writeCrewAllowed(profileDir, CrewAllowedAt(profileDir).With(add, word)) +} + +// writeCrewAllowed is the rule's one writer. +func writeCrewAllowed(profileDir string, rule crewroute.Allowed) error { + for _, seat := range crewroute.Seats { + pin, ok := CrewPinAt(profileDir, seat) + if !ok { + continue + } + if model, known := crewCatalogModel(pin.Model); (known && !rule.AdmitsModel(model)) || (!known && rule.Base != crewroute.BaseAll && !rule.NamesModel(pin.Model)) { + return fmt.Errorf("your %s is pinned to %s, which that rule leaves out · /crew unpin %s first, or add +%s", + seat, pin.Model, seat, crewroute.ShortModel(pin.Model)) + } + } + return writeProfileValue(profileDir, KeyCrewAllowed, rule.String()) +} + +// CrewCapAt is the daily cap on crew spend, zero for none. +func CrewCapAt(profileDir string) float64 { + value, ok := persistedFloat(profileDir, KeyCrewCap) + if !ok || value < 0 { + return 0 + } + return value +} + +// SetCrewCap writes the cap: a dollar amount, or `none`. +func SetCrewCap(profileDir, raw string) error { + return writeDollars(profileDir, KeyCrewCap, raw) +} + +// CrewTaskCapDefault is the per-task limit when none is set. +const CrewTaskCapDefault = 5.0 + +// CrewTaskCapAt is the most one task may spend, in dollars: the stored figure +// when it is above zero, [CrewTaskCapDefault] otherwise. There is always one. +func CrewTaskCapAt(profileDir string) float64 { + value, ok := persistedFloat(profileDir, KeyCrewTaskCap) + if !ok || value <= 0 { + return CrewTaskCapDefault + } + return value +} + +// SetCrewTaskCap writes the per-task limit: a dollar amount above zero. A +// task always has a limit, so `none` and zero are refused. +func SetCrewTaskCap(profileDir, raw string) error { + value, err := parseDollars(raw) + if err != nil { + return err + } + if value <= 0 { + return errors.New("a task always has a limit — a dollar amount above zero") + } + return writeProfileValue(profileDir, KeyCrewTaskCap, value) +} + +// CrewFreeRoutesAt is whether the crew may route to providers' free pools. +// Absent is off. // -// A word this build does not know names no preset and is read as no word at all, -// the way the family and pick rows fold a retired choice. The family is still -// [CrewSourceAt]'s: the word names a budget, not a shelf. -func storedCrewWord(profileDir string) (string, bool) { - value, held := persistedString(profileDir, KeyCrew) - if !held { - return "", false +// A FREE ROUTE IS A ROUTE, NOT A MODEL. On, a model's free pool joins its other +// routes ([crewroute.Free]) and is weighed at what it is expected to cost — +// its refusals included — and a seat on it falls through to the same model's +// paid route before any other model. Off, the free pools are not routes at +// all; a person can still pin one by name. +func CrewFreeRoutesAt(profileDir string) bool { + on, _ := persistedBool(profileDir, KeyCrewFreeRoutes) + return on +} + +// SetCrewFreeRoutes turns the free routes on or off. +func SetCrewFreeRoutes(profileDir string, on bool) error { + return writeProfileValue(profileDir, KeyCrewFreeRoutes, on) +} + +// ── the providers turned off ──────────────────────────────────────────────── + +// ErrCrewLastProvider is the refusal to turn off the last provider on: a crew +// with nothing to route through is not a narrower crew but no crew, and the +// way to say "none of these" is to disconnect them, which /connect does. +var ErrCrewLastProvider = errors.New("at least one provider must stay on") + +// ErrCrewNoRoutableProvider is the same refusal one step wider: the providers +// left on would route no seat — only a custom endpoint, say, which a pin +// reaches and the router never picks — while every seat is not pinned to a +// provider still on. +var ErrCrewNoRoutableProvider = errors.New("at least one provider that can route a seat must stay on") + +// CrewProvidersOffAt is the providers turned off for the crew. A row that does +// not read — a hand edit — reads as none off, the way a rule that does not +// parse reads as the default. +func CrewProvidersOffAt(profileDir string) crewroute.ProvidersOff { + raw, ok := persistedValue(profileDir, KeyCrewProvidersOff) + if !ok { + return nil + } + var ids []string + if json.Unmarshal(raw, &ids) != nil { + return nil } - word := strings.ToLower(strings.TrimSpace(value)) - for _, preset := range CrewPresets { - if word == preset { - return preset, true + off := crewroute.ProvidersOff{} + for _, id := range ids { + if id = strings.ToLower(strings.TrimSpace(id)); id != "" { + off[id] = true } } - return "", false + return off } -// CrewModelsForSource is the five models one preset would set under one -// family, by tier word, false for a word that is not a preset. It returns a -// copy for [CrewModels]'s reason. -func CrewModelsForSource(source, preset string) (map[string]string, bool) { - row, ok := crewTableFor(source)[strings.ToLower(strings.TrimSpace(preset))] +// SetCrewProviderOn turns one connected provider on or off for the crew, IN +// ONE FILE WRITE, refusing to turn off the last one on ([ErrCrewLastProvider]) +// and a provider that is not connected. +// +// THE ROW KEEPS ONLY CONNECTED PROVIDERS. An id left behind by a connection +// since removed is dropped on the next write, so a provider disconnected and +// connected again is on — the same as any connection made after the row was +// written — and the row never grows a list of names nobody can see. +func SetCrewProviderOn(profileDir, id string, on bool) error { + providers := CrewProvidersAt(profileDir) + provider, ok := crewProviderByID(providers, id) if !ok { - return nil, false + return fmt.Errorf("%s is not a connected provider · connect it with /connect", strings.TrimSpace(id)) + } + var ids []string + still := 0 + for _, p := range providers { + stays := p.On + if p.ID == provider.ID { + stays = on + } + if stays { + still++ + continue + } + ids = append(ids, p.ID) + } + if still == 0 { + return ErrCrewLastProvider + } + if !on && !crewStillRoutes(profileDir, providers, provider.ID) { + return ErrCrewNoRoutableProvider + } + if len(ids) == 0 { + return writeProfileValues(profileDir, map[string]any{KeyCrewProvidersOff: removeProfileKey}) + } + return writeProfileValue(profileDir, KeyCrewProvidersOff, ids) +} + +// crewStillRoutes is whether turning one more provider off leaves the crew +// able to seat every seat: some allowed model a seat can sit is reached by a +// provider still on, or every seat is pinned to a provider still on. A +// profile that could route nothing before the change is not refused for +// routing nothing after it — the refusal is about THIS change. +func crewStillRoutes(profileDir string, providers []CrewProvider, id string) bool { + rule := CrewAllowedAt(profileDir) + all := crewCandidates(rule, providers) + before, after := crewroute.ProvidersOff{}, crewroute.ProvidersOff{} + for _, p := range providers { + if !p.On { + before[p.ID], after[p.ID] = true, true + } + } + after[id] = true + if !crewSeatable(before.Candidates(all)) || crewSeatable(after.Candidates(all)) { + return true + } + pins := CrewPinsAt(profileDir) + for _, seat := range crewroute.Seats { + pin, ok := pins[seat] + if !ok { + return false + } + route := resolveCrewPin(pin, providers) + if _, connected := crewProviderByID(providers, route.Provider); !connected || !after.On(route.Provider) { + return false + } } - out := make(map[string]string, len(row)) - for tier, model := range row { - out[tier] = model + return true +} + +// crewSeatable is whether any candidate can sit a seat: a model that takes +// tools, on at least one route. +func crewSeatable(candidates []crewroute.Candidate) bool { + for _, c := range candidates { + if c.Model.Tools && len(c.Routes) > 0 { + return true + } } - return out, true + return false } -// crewLines is the one line each preset says about itself, IN THE FAMILY IT -// DRAWS FROM. It is what /crew prints beside each option and what the settings -// chooser shows under it, so it must name the models the preset actually picks -// in the family on screen: a line naming open models above frontier ids is the -// contradiction the chooser exists to prevent. -var crewLines = map[string]map[string]string{ - CrewSourceOpen: { - CrewFrugal: "glm-flash works, checks and thinks · pennies a day", - CrewBalanced: "glm-flash works, kimi-k3 checks, glm-5.3 thinks", - CrewMax: "glm-5.3 works and thinks, kimi-k3 checks", - }, - CrewSourceAll: { - CrewFrugal: "glm-flash works and thinks, qwen-max checks", - CrewBalanced: "glm-flash works, fable checks, opus thinks", - CrewMax: "glm-5.3 works, fable checks, opus thinks", - }, +// ── what the router may pick from ─────────────────────────────────────────── + +// CrewCatalog is how the catalog reaches crew routing: the binary holding the +// catalog sets it ONCE AT START-UP, from its non-blocking read, and never a +// fetch — a task is routed on whatever the catalog already holds. Nil, and a +// func answering no rows, are ordinary states rather than errors: the router +// then has no candidates, and a task's crew comes from its rescue ladder — +// the last crew that worked here, the model the person is talking to. +var CrewCatalog func() []catalog.Model + +// crewCatalogRows is [CrewCatalog] read with its ordinary absences folded. +func crewCatalogRows() []catalog.Model { + if CrewCatalog == nil { + return nil + } + return CrewCatalog() } -// CrewLine is one preset's own line in the DEFAULT family, empty for a word -// that is not a preset. The family-aware spelling is [CrewLineFor]. -func CrewLine(preset string) string { - return CrewLineFor(DefaultCrewSource, preset) +// crewModelOf reads one catalog row the way the router reads a model. +func crewModelOf(row catalog.Model) crewroute.Model { + return crewroute.Model{ + ID: row.ID, Open: row.OpenWeights, + PromptPrice: row.PromptPrice, CompletionPrice: row.CompletionPrice, CacheReadPrice: row.CacheReadPrice, + Intelligence: row.IntelligenceIndex, Coding: row.CodingIndex, Agentic: row.AgenticIndex, + ArenaElo: row.ArenaElo, Released: crewReleased(row), + Context: row.ContextLength, Tools: crewTakesTools(row) && crewSpeaksText(row), + } } -// CrewLineFor is one preset's own line in one family, empty for a word that is -// not a preset. A family this build does not know reads as the default one, the -// way the row does. -func CrewLineFor(source, preset string) string { - return crewLines[normalCrewSource(source)][strings.ToLower(strings.TrimSpace(preset))] +// crewReleased is when a catalog row's model was released: the row's own +// listing time, or the date a canonical slug ends on (`…-20260826`), or the +// zero time when the row says neither. +func crewReleased(row catalog.Model) time.Time { + if row.Created > 0 { + return time.Unix(row.Created, 0).UTC() + } + slug := strings.TrimSpace(row.CanonicalSlug) + if at := strings.LastIndex(slug, "-"); at >= 0 && len(slug)-at-1 == 8 { + if day, err := time.Parse("20060102", slug[at+1:]); err == nil && day.Year() >= 2020 { + return day + } + } + return time.Time{} } -// CrewModels is the five models one preset would set in the DEFAULT family, by -// tier word: the family a profile nobody has touched reads, and the spelling -// the callers hold. The family-aware spelling is -// [CrewModelsForSource]. It returns a copy, because a caller printing the -// table must not be able to edit it. -func CrewModels(preset string) (map[string]string, bool) { - return CrewModelsForSource(DefaultCrewSource, preset) +// crewIndexesFrom is m with every published figure it lacks — an index, the +// arena Elo, the release date — read from other, a row of the same model. +func crewIndexesFrom(m, other crewroute.Model) crewroute.Model { + if m.Intelligence <= 0 { + m.Intelligence = other.Intelligence + } + if m.Coding <= 0 { + m.Coding = other.Coding + } + if m.Agentic <= 0 { + m.Agentic = other.Agentic + } + if m.ArenaElo <= 0 { + m.ArenaElo = other.ArenaElo + } + if m.Released.IsZero() { + m.Released = other.Released + } + return m } -// CrewAt is the crew as the five live tier values make it: the preset they are, -// or [CrewCustom]. -// -// It reads through [TierModelAt], so a profile that has never been touched reads -// the shipped defaults and therefore reads [DefaultCrew] — the five defaults are -// the balanced row and nothing here needs to know that separately. A tier a -// person cleared on purpose reads empty, matches no preset, and turns the answer -// to custom, which is the truth: "one of these follows the conversation" is not -// any of the three. +// crewTakesTools is whether a catalog row says the model takes tool calls. // -// The comparison runs against the family [CrewSourceAt] names, so the reading -// moves with the row and never behind it: flip the family and a crew the old -// family wrote matches nothing, which reads as custom and is true, because one -// family's five ids are not any preset of the other. -func CrewAt(profileDir string) string { - family := CrewSourceAt(profileDir) - if CrewPickAt(profileDir) != CrewPickTable { - // WITH THE PICK OFF THE TABLE the three dial seats are computed ids the - // preset tables do not hold, and comparing the live seats would read - // custom over a crew the person chose. The word answers what the STORED - // rows make instead — the budget the seats are computed at ([pickedSeat] - // reads the same rows) — so the word on the sheet stays the decision it - // summarizes while the ids underneath move with the catalog. - return crewStoredAt(profileDir, family) - } - live := make(map[string]string, len(ModelTiers)) - for _, tier := range ModelTiers { - live[tier] = tierSeatUnder(profileDir, family, tier).Model - } - if sameCrew(live, freeCrewModels) { - return CrewFree - } - table := crewTableFor(family) - for _, preset := range CrewPresets { - if sameCrew(live, table[preset]) { - return preset - } - } - return CrewCustom -} - -// crewStoredAt is the crew the STORED five rows make, in the family given: -// the preset they are, or [CrewCustom]. It is [CrewAt]'s reading when the -// pick row takes the seats off the table, where the live comparison would -// compare computed ids. +// A ROW THAT LISTS NO PARAMETERS HAS SAID NOTHING, and a crew seat is an agent +// loop: a model whose tool support is unknown is not sent to find out in the +// middle of somebody's task. +func crewTakesTools(row catalog.Model) bool { + return len(row.Parameters) > 0 && listHolds(row.Parameters, "tools") +} + +// crewSpeaksText is whether a catalog row reads text and writes text. A crew +// seat is a conversation of text and tool calls: a speech, transcription or +// image model that lists tool parameters is still no seat. A row that lists +// no modalities has said nothing either way and is judged on its tools alone. +func crewSpeaksText(row catalog.Model) bool { + if len(row.InputModalities) > 0 && !listHolds(row.InputModalities, "text") { + return false + } + return len(row.OutputModalities) == 0 || listHolds(row.OutputModalities, "text") +} + +// crewCatalogModel is one model as the router would read it, from the +// catalog, false when the catalog does not know it. // -// Each row is read through the ladder's own row reader ([crewRow]). A row the -// reader cannot answer because the key was NEVER HELD reads the family's -// default-preset id, which is what that tier runs until somebody writes it; -// a row CLEARED ON PURPOSE reads empty and matches nothing, because -// "follows the conversation" is not any of the three; a row that says auto is -// skipped, because auto is the one row with no opinion of its own — it runs -// at whatever budget the rows around it name ([crewPresetUnder]). -func crewStoredAt(profileDir, family string) string { - // A STORED CREW WORD NAMES THE BUDGET OUTRIGHT. A run that wrote one word - // instead of the five rows made its decision on the word, and the five rows - // under it are that word's own table rows — so the word is the answer, and - // the row comparison below is only for a profile whose budget is the rows. - if word, ok := storedCrewWord(profileDir); ok { - return word - } - defaults := crewTableFor(family)[DefaultCrew] - stored := make(map[string]string, len(ModelTiers)) - for _, tier := range ModelTiers { - value, _, source, cleared := crewRow(profileDir, tier) - switch { - case cleared: - stored[tier] = "" - case source == "": - stored[tier] = strings.ToLower(strings.TrimSpace(defaults[tier])) - case IsAuto(value): - stored[tier] = AutoValue - default: - stored[tier] = strings.ToLower(strings.TrimSpace(value)) +// THE EXACT ID WINS. A pin names what the person wrote, and a catalog that +// lists that id serves that id — never a dated snapshot that happens to share +// its lineage and to be listed first. Only an id the catalog does not list is +// read through its lineage. +func crewCatalogModel(id string) (crewroute.Model, bool) { + exact := stripCrewRoute(id) + rows := crewCatalogRows() + for _, row := range rows { + if strings.EqualFold(row.ID, exact) && !row.PriceUnknown { + return crewModelOf(row), true } } - table := crewTableFor(family) - // THE DEFAULT PRESET WINS EVERY TIE, the law [crewPresetUnder] states: a - // profile whose every row says auto is a crew with no opinion of its own, - // and it reads balanced rather than whichever preset the loop met first. - if crewStoredMatches(stored, table[DefaultCrew]) { - return DefaultCrew + lineage := crewroute.Lineage(exact) + for _, row := range rows { + if crewroute.Lineage(row.ID) == lineage && !row.PriceUnknown { + return crewModelOf(row), true + } } - for _, preset := range CrewPresets { - if preset != DefaultCrew && crewStoredMatches(stored, table[preset]) { - return preset + return crewroute.Model{}, false +} + +// stripCrewRoute takes a connection's prefix off an id a person wrote with +// one (`openrouter/z-ai/glm-5.3-flash`), leaving the catalog id. +func stripCrewRoute(id string) string { + return strings.TrimPrefix(strings.TrimSpace(id), modelsource.DefaultID+"/") +} + +// listHolds says whether a row's own list carries the word, case folded. +func listHolds(words []string, word string) bool { + for _, held := range words { + if strings.EqualFold(strings.TrimSpace(held), word) { + return true } } - return CrewCustom + return false } -// crewStoredMatches compares a stored reading with one preset row, skipping -// the tiers whose row says auto. -func crewStoredMatches(stored, preset map[string]string) bool { - for _, tier := range ModelTiers { - if stored[tier] == AutoValue { +// CrewProvider is one connected provider as the crew can use it. +type CrewProvider struct { + // ID is the connection's own id — `openrouter`, `z-ai`, `codex`, + // `ollama` — and the word an `@provider` pin names. + ID string + // Name is how the panel says it. + Name string + // Written is the id prefix that sends a call to this connection. + Written string + // Kind is how it bills: metered, a subscription plan, or local. + Kind crewroute.RouteKind + // Serves is the models a plan door serves, empty for every model the + // vendor lists. + Serves []string + // On is whether the crew may route through it: every connection is, until + // a person turns it off ([SetCrewProviderOn]). + On bool + // vendors are the catalog vendor prefixes this connection serves directly. + vendors []string + // collides lists the catalog vendors whose ids this connection's Written + // prefix would capture, so the default route spells them `openrouter/…`. + collides map[string]bool +} + +// crewVendors maps a direct connection to the catalog vendor prefixes it +// serves. A connection whose Written is the vendor's own prefix needs no row. +var crewVendors = map[string][]string{ + "moonshot": {"moonshotai"}, + "codex": {"openai"}, +} + +// CrewProvidersAt is every provider the crew can route through: each +// connection this profile has with a key, in the person's own order, the +// default service first when it has one. It is DERIVED, never a setting — +// connecting a provider is what adds it. +func CrewProvidersAt(profileDir string) []CrewProvider { + off := CrewProvidersOffAt(profileDir) + set := ResolveSources(profileDir, APIKeyAt(profileDir), DefaultBaseURL) + written := map[string]bool{} + for _, service := range set.All() { + written[strings.ToLower(service.Source.Written)] = true + } + var out []CrewProvider + for _, service := range set.All() { + source := service.Source + if strings.TrimSpace(service.Key) == "" && !source.KeyOptional { continue } - if stored[tier] != strings.ToLower(strings.TrimSpace(preset[tier])) { - return false + p := CrewProvider{ID: strings.ToLower(source.ID), Name: source.Name, Written: source.Written, Kind: crewroute.Metered} + p.On = off.On(p.ID) + switch { + case strings.EqualFold(source.ID, modelsource.DefaultID): + p.collides = map[string]bool{} + for w := range written { + if w != modelsource.DefaultID { + p.collides[w] = true + } + } + case strings.EqualFold(source.ID, "codex"): + p.Kind = crewroute.Plan + p.Serves = []string{source.Preferred} + case strings.EqualFold(source.ID, "ollama"): + p.Kind = crewroute.Local + case modelsource.IsCustomID(source.ID): + // A custom endpoint serves models the catalog does not describe; + // it is reachable by a pin that names it and by nothing else. + case service.Door.ID != "" && !service.Door.Metered: + p.Kind = crewroute.Plan + p.Serves = append([]string(nil), service.Door.Models...) + } + if !strings.EqualFold(source.ID, modelsource.DefaultID) && !modelsource.IsCustomID(source.ID) && !strings.EqualFold(source.ID, "ollama") { + p.vendors = append([]string{strings.ToLower(source.Written)}, crewVendors[strings.ToLower(source.ID)]...) } + out = append(out, p) } - return true + return out } -// sameCrew compares two crews class by class, case-folded, because a model id is -// matched case-insensitively everywhere else on this surface. -func sameCrew(live, preset map[string]string) bool { - for _, tier := range ModelTiers { - if !strings.EqualFold(strings.TrimSpace(live[tier]), preset[tier]) { - return false +// crewProviderByID finds a connected provider by the word a pin names. +func crewProviderByID(providers []CrewProvider, id string) (CrewProvider, bool) { + id = strings.ToLower(strings.TrimSpace(id)) + for _, p := range providers { + if p.ID == id || strings.EqualFold(p.Written, id) { + return p, true } } - return true + return CrewProvider{}, false } -// ApplyCrew writes all five tier rows from one preset, IN ONE FILE WRITE. +// route is how this provider reaches one catalog model, false when it does +// not. The send is the id that makes an ordinary call go this way — the +// connection-routing grammar every call already obeys ([modelsource.Split]). +func (p CrewProvider) route(id string, model crewroute.Model, known bool) (crewroute.Route, bool) { + id = stripCrewRoute(id) + vendor, tail := id, id + if slash := strings.Index(id, "/"); slash >= 0 { + vendor, tail = strings.ToLower(id[:slash]), id[slash+1:] + } + if p.ID == modelsource.DefaultID { + if !known { + return crewroute.Route{}, false + } + send := id + if p.collides[vendor] { + send = modelsource.DefaultID + "/" + id + } + return crewroute.Route{Provider: p.ID, Send: send, Kind: p.Kind}, true + } + // A pin written with this connection's own prefix is already its send. + if strings.EqualFold(vendor, p.Written) && !known { + return crewroute.Route{Provider: p.ID, Send: id, Kind: p.Kind}, true + } + served := false + for _, v := range p.vendors { + if v == vendor { + served = true + break + } + } + if !served { + return crewroute.Route{}, false + } + if len(p.Serves) > 0 { + ok := false + for _, s := range p.Serves { + if strings.EqualFold(crewroute.Lineage(s), crewroute.Lineage(tail)) { + ok = true + break + } + } + if !ok { + return crewroute.Route{}, false + } + } + return crewroute.Route{Provider: p.ID, Send: p.Written + "/" + tail, Kind: p.Kind}, true +} + +// CrewCandidatesAt is what the router may pick from on this profile: every +// catalog model the allowed rule admits, with every route a connected +// provider offers it on — plans and local models first, then a direct +// connection, then the default service, which is the order a tie between +// routes of equal cost is broken in. A model no connected provider reaches is +// not a candidate, however cheap. // -// The five keys land together or not at all. Five separate writes would leave a -// window — one process crash, one full disk — in which two classes belong to the -// old crew and two to the new, and the crew row would read "custom" about a -// state nobody chose. It is also the only shape in which a reader that happens -// to be resolving a role while somebody presses enter cannot see half a crew. +// A provider the person turned off carries none of those routes +// ([crewroute.ProvidersOff]), so a model only it reached is no candidate +// either — the route cost the router weighs is always a route it may take. +func CrewCandidatesAt(profileDir string) []crewroute.Candidate { + candidates, _ := crewCandidatesNoticed(profileDir, crewHealthAt(profileDir)) + return candidates +} + +// crewCandidates is [CrewCandidatesAt] over a rule and a set of providers +// already read, which is how [CrewOffersAt] asks the same question under `all`. +// It is asked with no free routes. +func crewCandidates(rule crewroute.Allowed, providers []CrewProvider) []crewroute.Candidate { + return crewCandidatesWith(rule, providers, crewRouteFacts{}) +} + +// crewCandidatesWith is [crewCandidates] with the profile's route facts. // -// The preset is resolved under the family [CrewSourceAt] names, so the row and -// the write cannot disagree about which table the word means: flip to `all`, -// press the crew again, and the five ids that land are the all-family ones. -func ApplyCrew(profileDir, preset string) error { - // An empty family means KEEP THE ONE THE PROFILE HOLDS, so this path reads the - // profile once inside ApplyCrewUnder rather than once here and again there, and - // the write is exactly the five tiers. One resolve-and-write serves both callers - // rather than two that can drift apart. - return ApplyCrewUnder(profileDir, "", preset) -} - -// ApplyCrewUnder writes a FAMILY AND A PRESET AS ONE DECISION, IN ONE FILE -// WRITE. The family row and the five tier rows are one state, so writing them -// apart leaves a window in which a reader sees the family set to `all` while the -// rows still hold open ids, which is the half-written crew [ApplyCrew] forbids, read as -// `custom` about a state nobody chose. +// ONE MODEL IS ONE CANDIDATE, WHATEVER IT IS SPELLED. Catalog rows are grouped +// by their canonical identity ([crewroute.Canonical]), so a model's free pool +// (`…:free`) is not a second, free model beside it but one more route of the +// same one — kept only when free routes are on ([CrewFreeRoutesAt]). A free +// pool whose paid model the catalog does not list is a model with that one +// route, weighed at what the route is expected to cost, never at zero. +func crewCandidatesWith(rule crewroute.Allowed, providers []CrewProvider, facts crewRouteFacts) []crewroute.Candidate { + var models []crewroute.Model + var free map[string]string // canonical id → the free row's id + if rows := crewCatalogRows(); len(rows) > 0 { + paid := map[string]bool{} + at := map[string]int{} + for _, row := range rows { + if crewUnpriced(row) || strings.HasPrefix(row.ID, "~") || crewroute.IsFree(row.ID) { + continue + } + lineage := crewroute.Lineage(row.ID) + if paid[lineage] { + // The same model again — a dated snapshot beside its name — + // is one candidate, read from the row that lists the model + // by its own name when there is one, so a seat is sent the + // name and not whichever snapshot the catalog listed first. + // THE FIGURES ARE THE MODEL'S, whichever row carried them: an + // index the snapshot's row published is not lost because the + // model's own row did not repeat it. + if strings.EqualFold(row.ID, lineage) { + models[at[lineage]] = crewIndexesFrom(crewModelOf(row), models[at[lineage]]) + } else { + models[at[lineage]] = crewIndexesFrom(models[at[lineage]], crewModelOf(row)) + } + continue + } + paid[lineage], at[lineage] = true, len(models) + models = append(models, crewIndexesFrom(crewModelOf(row), crewroute.Model{})) + } + if facts.free { + free = map[string]string{} + for _, row := range rows { + if !crewroute.IsFree(row.ID) || strings.HasPrefix(row.ID, "~") || crewUnpriced(row) { + continue + } + key := crewroute.Lineage(row.ID) + free[key] = row.ID + if !paid[key] { + paid[key] = true + m := crewModelOf(row) + m.ID = strings.TrimSuffix(m.ID, ":free") + models = append(models, m) + } + } + } + } + var out []crewroute.Candidate + for _, m := range models { + if !rule.AdmitsModel(m) { + continue + } + var plans, direct, fallback []crewroute.Route + for _, p := range providers { + if !rule.AdmitsRoute(p.ID) { + continue + } + r, ok := p.route(m.ID, m, true) + if !ok { + continue + } + switch { + case r.Kind != crewroute.Metered: + plans = append(plans, r) + case p.ID == modelsource.DefaultID: + fallback = append(fallback, r) + default: + direct = append(direct, r) + } + } + if send, ok := free[crewroute.Lineage(m.ID)]; ok { + for _, p := range providers { + if p.ID == modelsource.DefaultID && rule.AdmitsRoute(p.ID) { + plans = append(plans, crewroute.Route{Provider: p.ID, Send: send, Kind: crewroute.Free}) + } + } + } + // Only a free pool: the model is reachable on nothing else. + fallback = withoutPaidPool(fallback, free[crewroute.Lineage(m.ID)], m) + // AND ONLY THE ROUTES THAT ARE HEALTHY: a route quarantined, cooling + // down, on a disconnected provider, or paid on an account out of + // credit is not offered, and the rest carry their learned failure + // rate ([crewHealth.usable]). + routes := facts.health.usable(append(append(plans, direct...), fallback...)) + if len(routes) == 0 { + continue + } + out = append(out, crewroute.Candidate{Model: m, Routes: routes}) + } + return out +} + +// crewUnpriced is a catalog row the router may not weigh: one whose provider +// published no price, or a paid id priced at nothing — a stealth or preview +// model, a meta-router — whose zero promises nothing about the next call and +// would win every seat at $0. ONLY A FREE POOL (`…:free`) PRICED AT AN +// EXPLICIT ZERO IS FREE. A pin still names such a model by its id; the router +// never picks one on its own. +func crewUnpriced(row catalog.Model) bool { + if row.PriceUnknown { + return true + } + return !crewroute.IsFree(row.ID) && row.PromptPrice <= 0 && row.CompletionPrice <= 0 && row.RequestPrice <= 0 +} + +// withoutPaidPool drops the default service's metered route for a model the +// catalog lists ONLY as a free pool: there is no paid route to it, and a +// metered route at the free row's zero prices would be the free pool again +// under another kind. +func withoutPaidPool(routes []crewroute.Route, freeSend string, m crewroute.Model) []crewroute.Route { + if freeSend == "" || m.PromptPrice > 0 || m.CompletionPrice > 0 { + return routes + } + out := routes[:0] + for _, r := range routes { + if r.Provider != modelsource.DefaultID { + out = append(out, r) + } + } + return out +} + +// CrewOffer is one model a seat could be pinned to on this profile: a model +// some connected provider reaches, WHETHER OR NOT THE ALLOWED RULE ADMITS IT. // -// The family row rides along ONLY WHEN IT CHANGES, so an enter that keeps the -// family does not pin a setting the person never answered: the five tier rows are -// written and the family row stays unanswered, free to follow a later default -// family. -func ApplyCrewUnder(profileDir, source, preset string) error { - // An empty source keeps the family the profile holds, which is how [ApplyCrew] - // asks for the five tiers alone. The profile is read ONCE here, so there is no - // window between a check and the write in which the family could move. - persisted := CrewSourceAt(profileDir) - if strings.TrimSpace(source) == "" { - source = persisted - } - known, ok := knownCrewSource(source) - if !ok { - return fmt.Errorf("pick one of: %s", strings.Join(CrewSources, ", ")) +// THE PICKER LISTS WHAT THE RULE LEAVES OUT, AND SAYS SO. A list that hid +// every model outside the rule would answer "why can't I pick kimi?" with +// silence; one that shows it dim, with the one key that lets it in, answers +// the question on the row where it was asked ([SetCrewPin] still refuses a pin +// outside the rule — the panel's key widens the rule first, out loud). +type CrewOffer struct { + Model crewroute.Model + // Routes are every connected provider reaching it, in [CrewCandidatesAt]'s + // order — plans and local first, the default service last — and the + // cheapest route is the first one. + Routes []crewroute.Route + // Served is whether a provider that is ON reaches it: a model only a + // provider turned off reaches is still listed, and is not allowed. + Served bool + // Allowed is whether the rule admits the model on at least one route + // through a provider that is on. + Allowed bool +} + +// CrewOffersAt is every model a connected provider reaches, allowed or not, +// in the catalog's order. It is [CrewCandidatesAt] asked under `all` and then +// read against the rule in force, so the two can never disagree about which +// route reaches what. +func CrewOffersAt(profileDir string) []CrewOffer { + rule := CrewAllowedAt(profileDir) + off := CrewProvidersOffAt(profileDir) + providers := CrewProvidersAt(profileDir) + var out []CrewOffer + facts := crewRouteFacts{free: CrewFreeRoutesAt(profileDir), health: crewHealthAt(profileDir)} + for _, c := range crewCandidatesWith(crewroute.Allowed{Base: crewroute.BaseAll}, providers, facts) { + // ── ONLY WHAT A SEAT CAN RUN ── + // A picker offers exactly what the router could seat: a model that takes + // tool calls and holds a seat's context — not an embedding, image or + // video model — on a route that is healthy (quarantined and cooling + // routes were left out above). + if !crewroute.Seatable(crewroute.Planner, c) { + continue + } + // ── end of the seat check ── + offer := CrewOffer{Model: c.Model, Routes: c.Routes} + offer.Served = len(off.Routes(c.Routes)) > 0 + if rule.AdmitsModel(c.Model) { + for _, r := range off.Routes(c.Routes) { + if rule.AdmitsRoute(r.Provider) { + offer.Allowed = true + break + } + } + } + out = append(out, offer) } - preset = strings.ToLower(strings.TrimSpace(preset)) - models, ok := CrewModelsForSource(known, preset) - if !ok { - return fmt.Errorf("pick one of: %s", strings.Join(CrewPresets, ", ")) + return out +} + +// SetCrewAllowedRule writes a rule the panel composed ([crewroute.Allowed]'s +// edits) through the rule's one writer, so a checklist tick is refused for +// exactly the reason the typed form would be — a pinned seat it would leave +// outside. +func SetCrewAllowedRule(profileDir string, rule crewroute.Allowed) error { + return writeCrewAllowed(profileDir, rule) +} + +// CrewState is the crew's persisted rows as they stand — the three seats, the +// allowed rule, the cap, the providers turned off and the free-routes switch, +// raw — which is what the panel's undo puts back. +// +// IT IS THE ROWS AND NOT A READING OF THEM. An undo that re-wrote what the +// readers made of the rows would turn a hand-written `auto` into an absent +// row, or a preset's legacy row into a pin; putting the bytes back is the only +// undo that restores exactly what was there. +type CrewState struct { + values map[string]json.RawMessage +} + +// crewStateKeys are the rows a [CrewState] carries. +func crewStateKeys() []string { + keys := []string{KeyCrewAllowed, KeyCrewCap, KeyCrewTaskCap, KeyCrewProvidersOff, KeyCrewFreeRoutes} + for _, seat := range crewroute.Seats { + keys = append(keys, tierKeyFor(CrewSeatTier(seat))) } - values := make(map[string]any, len(models)+1) - for tier, model := range models { - values[tierKeyFor(tier)] = model + return keys +} + +// CrewStateAt reads the crew's rows as they stand. +func CrewStateAt(profileDir string) CrewState { + state := CrewState{values: map[string]json.RawMessage{}} + for _, key := range crewStateKeys() { + if raw, ok := persistedValue(profileDir, key); ok { + state.values[key] = raw + } } - if known != persisted { - values[KeyCrewSource] = known + return state +} + +// RestoreCrewState writes a [CrewState] back, IN ONE FILE WRITE: a row that +// was absent is removed, and every other row gets its old bytes. +func RestoreCrewState(profileDir string, state CrewState) error { + values := map[string]any{} + for _, key := range crewStateKeys() { + if raw, ok := state.values[key]; ok { + values[key] = raw + continue + } + values[key] = removeProfileKey } return writeProfileValues(profileDir, values) } -// writeCrew is the row's writer: the same refusal wording every choice row uses, -// and then the atomic write. -func writeCrew(profileDir, raw string) error { - return ApplyCrew(profileDir, raw) +// CrewGapsAt names what the allowed models leave uncovered, for the panel's +// one-line warning. A pinned seat is judged by its pin alone, read from the +// catalog whether or not a route to it is healthy right now, because the pin +// is what the seat runs. +func CrewGapsAt(profileDir string) []crewroute.Gap { + pins := map[crewroute.Seat]crewroute.Model{} + for seat, pin := range CrewPinsAt(profileDir) { + model, known := crewCatalogModel(pin.Model) + if !known { + model = crewroute.Model{ID: pin.Model} + } + pins[seat] = model + } + return crewroute.Gaps(CrewCandidatesAt(profileDir), pins) +} + +// ── one task's crew ───────────────────────────────────────────────────────── + +// CrewAsk is one task's request for a crew. +type CrewAsk struct { + Task crewroute.Task + // Effort is the one-task word: best, cheap, or the knee. + Effort crewroute.Effort + // Pins are ONE-TASK pins, laid over the profile's: `--pin` and the seat + // flags. They never persist. + Pins map[crewroute.Seat]CrewPin + // Sends are seats a door has already filled with an id to send as it + // stands — a flag, a variable — which the router treats as pins. + Sends map[crewroute.Seat]string + // Stronger is the crew that ran, for a redo that asks for a stronger one. + Stronger *crewroute.Decision + // Again is the crew that ran and never started, for a redo of it: the + // next-best models at the same cost, not a stronger crew. + Again *crewroute.Decision + // ChatModel is the model the person is talking to — proven reachable — + // the last rung of a seat's ladder when nothing routed can start. + ChatModel string + // Repo keys the learned offset: a repository whose work of one class was + // redone stronger starts that class a step higher. + Repo string } -// SetCrewSource writes the family row ALONE, in one file write. The settings -// row is its caller; the chooser commits the family and the preset together -// through [ApplyCrewUnder]. The word is refused the way every choice row refuses -// one, so a typo cannot land a family nothing reads. -func SetCrewSource(profileDir, source string) error { - known, ok := knownCrewSource(source) - if !ok { - return fmt.Errorf("pick one of: %s", strings.Join(CrewSources, ", ")) +// ErrCrewAtCap is a crew asked for with the day's crew spend already at the +// daily cap. The decision still comes back: the chat asks the person, and a +// headless run refuses unless told `-yes-spend`. +var ErrCrewAtCap = errors.New("today's crew spend has reached the daily cap") + +// CrewHistory is how the router's log reaches crew routing: today's crew +// spend and the learned offsets, read out of the profile's router-events log +// (internal/router's crew.go). It is a variable so a test can hand a day of +// its own; nil reads as a day with nothing spent and nothing learned — the +// state of a fresh install, not an error. +var CrewHistory = func(profileDir string) CrewDay { + log := router.ReadCrewLog(ProfilePath(profileDir, ""), time.Now()) + return CrewDay{SpentUSD: log.SpentUSD, Offsets: log.Offsets, CostFactor: log.CostFactor, Learned: crewLearned(log.Quality)} +} + +// crewLearned is the log's learned quality moves keyed the way the router +// reads them ([crewroute.LearnKey]): the ids a seat ran folded to their +// lineage, the moves of one lineage averaged. +func crewLearned(quality map[string]float64) map[string]float64 { + if len(quality) == 0 { + return nil } - return writeProfileValue(profileDir, KeyCrewSource, known) + sums, counts := map[string]float64{}, map[string]int{} + for key, move := range quality { + parts := strings.SplitN(key, "\x00", 3) + if len(parts) != 3 { + continue + } + k := crewroute.LearnKey(crewroute.Class(parts[0]), crewroute.Seat(parts[1]), parts[2]) + sums[k] += move + counts[k]++ + } + out := make(map[string]float64, len(sums)) + for k, sum := range sums { + out[k] = sum / float64(counts[k]) + } + return out } -// CrewSummary is the one line a crew change confirms itself with: -// -// crew → balanced · brain claude-opus-5 · hands glm-5.3-flash · checks claude-fable-5.1 -// -// The three names are the classes a person actually asked about — what thinks, -// what works, what checks — and HANDS IS THE WORKER: the seat that does the -// task, which is what everybody reading the word took it to mean back when it -// named the small-work tier. The reflex and small-work models are deliberately -// absent: they are the same near-free models in all three presets, so naming -// them would be facts that never vary. The ids are shortened to their base names -// because the vendor prefix is the half nobody reads twice. -func CrewSummary(profileDir string) string { - return crewSummaryWith(profileDir, "") -} - -// CrewSummaryPick is the confirmation with the pick named when it is not the -// default one: -// -// crew → balanced · learn · brain claude-opus-5 · hands glm-5.3-flash · checks claude-fable-5.1 +// CrewRecordOf is one decision as the router's log keeps it. +func CrewRecordOf(d crewroute.Decision, repo, title string) router.CrewRecord { + record := router.CrewRecord{ + TaskClass: string(d.Class), Repo: repo, Title: title, Effort: string(d.Effort), Steps: d.Steps, + Seats: map[string]string{}, Providers: map[string]string{}, Kinds: map[string]string{}, EstUSD: d.EstUSD, + Redo: d.Redo, EstBase: d.EstUSD, TaskSubclass: d.Subclass, TaskReach: d.Reach, + } + if d.CostFactor > 0 { + record.EstBase = d.EstUSD / d.CostFactor + } + for _, pick := range d.Crew { + if pick.Learned != 0 { + if record.Learned == nil { + record.Learned = map[string]float64{} + } + record.Learned[string(pick.Seat)] = pick.Learned + } + record.Seats[string(pick.Seat)] = pick.Send + if pick.Provider != "" { + record.Providers[string(pick.Seat)] = pick.Provider + } + if pick.Kind != "" { + record.Kinds[string(pick.Seat)] = string(pick.Kind) + } + if pick.Pinned { + record.Pinned = append(record.Pinned, string(pick.Seat)) + } + } + return record +} + +// LogCrewDecision writes one task's crew decision into the profile's router +// log, under the call id the outcome will settle. +func LogCrewDecision(profileDir, call string, d crewroute.Decision, repo, title string) { + record := CrewRecordOf(d, repo, title) + record.Top = crewTop(profileDir, d) + router.LogCrewDecision(ProfilePath(profileDir, ""), call, record, CrewCandidateNames(profileDir)) +} + +// LogCrewOutcome settles it: accepted, redone stronger, or not kept. +func LogCrewOutcome(profileDir, call string, d crewroute.Decision, repo, title, outcome string, costUSD float64) { + router.LogCrewOutcome(ProfilePath(profileDir, ""), call, CrewRecordOf(d, repo, title), outcome, costUSD) +} + +// CrewCandidateNames is what a logged decision was made among. // -// The pick rides the preset word because the two are one decision read at two -// heights — how much to spend, and where the models for that money come from -// — and a confirmation that said only `balanced` would drop the half the -// person just changed. At the default pick this is [CrewSummary] itself, so a -// profile nobody has taught the pick to confirms exactly as it always has. -func CrewSummaryPick(profileDir string) string { - return crewSummaryWith(profileDir, CrewPickAt(profileDir)) +// ONLY WHAT COULD SIT A SEAT IS NAMED: a speech, image or embedding model the +// catalog lists is no candidate for a crew, and a log row naming it would +// claim the router weighed it. +func CrewCandidateNames(profileDir string) []string { + return crewroute.Names(crewSeatableOnly(CrewCandidatesAt(profileDir))) } -// crewSummaryWith is the line both summaries are built from: the pick named -// between the preset and the three classes when one was given that is not the -// default, and never otherwise — a profile at the default pick confirms in -// the words it has always confirmed in. -func crewSummaryWith(profileDir, pick string) string { - head := "crew → " + CrewAt(profileDir) - if pick != "" && pick != CrewPickTable { - head += " · " + pick +// crewSeatableOnly is the candidates that can sit at least one seat. +func crewSeatableOnly(candidates []crewroute.Candidate) []crewroute.Candidate { + var out []crewroute.Candidate + for _, c := range candidates { + for _, seat := range crewroute.Seats { + if crewroute.Seatable(seat, c) { + out = append(out, c) + break + } + } } - return head + " · " + CrewClasses(profileDir) + return out } -// CrewClasses is the three class names alone: -// -// brain claude-opus-5 · hands glm-5.3-flash · checks claude-fable-5.1 -// -// It is the tail of [CrewSummary] lifted out because a second surface prints the -// crew now — /status, where the word already has a label of its own and "crew →" -// in front of it would say the word twice. ONE SOURCE OF TRUTH: the three names, -// their order and their separator are spelled here once, so the confirmation a -// person reads after /crew and the line they read in /status cannot drift into -// naming the same four models differently. -func CrewClasses(profileDir string) string { - ids := CrewClassModels(profileDir) - return "brain " + ids[0] + " · hands " + ids[1] + " · checks " + ids[2] -} - -// CrewClassModels is the three ids [CrewClasses] names, in that order and -// without the role words in front of them: -// -// claude-opus-5, glm-5.3-flash, claude-fable-5.1 +// crewTop is a decision's best three per seat, re-weighed at its λ over the +// candidates it was routed among, for the log row. +func crewTop(profileDir string, d crewroute.Decision) map[string][]router.CrewScore { + candidates, _ := crewCandidatesNoticed(profileDir, crewHealthAt(profileDir).probing()) + var learned map[string]float64 + if CrewHistory != nil { + learned = CrewHistory(profileDir).Learned + } + out := map[string][]router.CrewScore{} + for seat, ranked := range crewroute.Explain(d.Class, d.Lambda, candidates, learned, crewTopN) { + for _, s := range ranked { + route := s.Provider + if s.Kind != "" { + route += ":" + string(s.Kind) + } + out[string(seat)] = append(out[string(seat)], router.CrewScore{Model: s.Model, Route: route, Q: s.Quality, C: s.CostUSD, S: s.Score}) + } + } + return out +} + +// crewTopN is how many candidates a log row keeps per seat. +const crewTopN = 3 + +// CrewLogAt is the router log's account of crews on this profile, for the +// panel: today's spend and tasks and the recent ones. +func CrewLogAt(profileDir string) router.CrewLog { + return router.ReadCrewLog(ProfilePath(profileDir, ""), time.Now()) +} + +// CrewDay is what the log says about crews: today's spend, and the learned +// escalation offset per repository and class. +type CrewDay struct { + SpentUSD float64 + Offsets map[string]int + // CostFactor is the learned estimate factor per class + // ([router.CrewLog.CostFactor]). + CostFactor map[string]float64 + // Learned is this install's learned quality moves, keyed + // [crewroute.LearnKey] ([router.CrewLog.Quality]). + Learned map[string]float64 +} + +// OffsetKey is the key [CrewDay.Offsets] is read under. +func OffsetKey(repo string, class crewroute.Class) string { + return strings.TrimSpace(repo) + "\x00" + string(class) +} + +// RouteCrew picks one task's crew on this profile: the seats pinned there or +// for this task run their pins, and every other seat is routed. // -// It exists because a surface drawing the crew line has to be able to say which -// runs of it are the ANSWER — the ids a person typed /crew to change — and which -// are the labels around them (internal/tui3's payload.go). Reading them back out -// of the sentence would be a second parser for a string this file just built, so -// the sentence is built from this list instead and the two cannot disagree about -// how many models there are or which order they come in. -func CrewClassModels(profileDir string) []string { - return []string{ - shortModel(TierModelAt(profileDir, ModelTierMastermind)), - shortModel(TierModelAt(profileDir, ModelTierWorker)), - shortModel(TierModelAt(profileDir, ModelTierHigh)), - } -} - -// shortModel is a model id without its vendor prefix, and the level kept. THE -// EMPTINESS LAW: a class that follows the conversation has no id to print, and -// says so in words rather than leaving a gap a reader has to interpret. -func shortModel(value string) string { - value = strings.TrimSpace(value) - if value == "" { - return "the conversation" +// The class is read first, so the learned offset for this repository and this +// class of work can move the price of a point before the seats are picked. A +// day at its cap still answers with the crew it would run, beside +// [ErrCrewAtCap]; a seat nothing allowed can sit is a [crewroute.NoCandidateError]. +func RouteCrew(profileDir string, ask CrewAsk) (crewroute.Decision, error) { + reading := crewroute.Classify(ask.Task) + providers := CrewProvidersAt(profileDir) + pins := map[crewroute.Seat]crewroute.Pin{} + for seat, pin := range CrewPinsAt(profileDir) { + pins[seat] = resolveCrewPin(pin, providers) } - if at := strings.LastIndex(value, "/"); at >= 0 { - return value[at+1:] + for seat, pin := range ask.Pins { + pins[seat] = resolveCrewPin(pin, providers) } - return value + for seat, send := range ask.Sends { + if send = strings.TrimSpace(send); send != "" { + pins[seat] = resolveCrewPin(CrewPin{Model: send}, providers) + } + } + day := CrewDay{} + if CrewHistory != nil { + day = CrewHistory(profileDir) + } + pace, atCap := crewroute.Pace(day.SpentUSD, CrewCapAt(profileDir)) + health := crewHealthAt(profileDir) + // AN ACCOUNT OUT OF CREDIT, OR A KEY REFUSED, IS PROBED BY THE NEXT TASK: + // its paid routes are routed as usual, so the task's first seat call is + // the probe. A 200 clears the account on the spot; another refusal moves + // the seat down its ladder inside the task (to a free pool, when nothing + // paid is left), costing one refused call. Without the probe an account + // topped up would never be asked again. + candidates, notice := crewCandidatesNoticed(profileDir, health.probing()) + req := crewroute.Request{ + Task: ask.Task, + Class: reading.Class, + Reading: &reading, + Candidates: candidates, + Pins: pins, + Effort: ask.Effort, + Steps: day.Offsets[OffsetKey(ask.Repo, reading.Class)], + Pace: pace, + Stronger: ask.Stronger, + Again: ask.Again, + Avoid: health.demoted, + Learned: day.Learned, + CostFactor: day.CostFactor[string(reading.Class)], + TaskCap: CrewTaskCapAt(profileDir), + } + d, err := crewroute.Decide(req) + if err != nil && !errors.Is(err, crewroute.ErrStrongest) { + // NO CREW CAN BE FORMED: the seat nothing can sit goes down the rest of + // the ladder — the last crew that worked here, the person's own model + // — and with nothing left the answer is the one thing to do. + d, err = crewRescued(profileDir, req, health, ask.ChatModel, err) + } + if err != nil { + return crewroute.Decision{}, err + } + if notice != "" { + d.Note = strings.TrimSpace(strings.TrimPrefix(d.Note+" · "+notice, " · ")) + } + d.Why, d.Sure = reading.Why, reading.Sure + d.Redo = ask.Stronger != nil || ask.Again != nil + if atCap { + return d, ErrCrewAtCap + } + return d, nil +} + +// resolveCrewPin is a pin with the route it will run on: the provider it +// names, or the connection its own prefix names, or the default service. +func resolveCrewPin(pin CrewPin, providers []CrewProvider) (out crewroute.Pin) { + out = crewroute.Pin{Model: stripCrewRoute(pin.Model), Provider: pin.Provider, Send: pin.Model, Kind: crewroute.Metered} + if crewroute.IsFree(out.Model) { + // A pinned free pool runs on that pool and is weighed as one. + defer func() { out.Kind = crewroute.Free }() + } + model, known := crewCatalogModel(pin.Model) + if known && !crewroute.IsFree(out.Model) && !strings.EqualFold(model.ID, out.Model) { + // THE CATALOG DOES NOT LIST THE ID AS WRITTEN, only a variant of it: + // the pin is sent as that variant, and the line says so. + pin.Model, out.Send = model.ID, model.ID + } + if pin.Provider != "" { + if p, ok := crewProviderByID(providers, pin.Provider); ok { + if r, ok := p.route(pin.Model, model, known); ok { + out.Provider, out.Send, out.Kind = r.Provider, r.Send, r.Kind + } + } + return out + } + // No provider named: the id itself says where it goes, the way every call + // reads it — a connection's own prefix, or the default service. + for _, p := range providers { + if p.ID == modelsource.DefaultID { + continue + } + if prefix := strings.ToLower(p.Written) + "/"; strings.HasPrefix(strings.ToLower(pin.Model), prefix) { + out.Provider, out.Kind = p.ID, p.Kind + return out + } + } + for _, p := range providers { + if p.ID == modelsource.DefaultID { + out.Provider = p.ID + } + } + return out +} + +// standingCrewSeat is a routed seat for the calls that ride a crew seat's +// tier without a task in front of them — the conversation's own planner and +// careful calls. They are routed as work of no particular class, so an +// unpinned seat never falls back to a model this build chose for everybody. +// A profile with nothing the router can pick answers empty, which is the role +// ladder's own floor: the model the person is talking to. +func standingCrewSeat(profileDir string, seat crewroute.Seat) string { + d, err := crewroute.Decide(crewroute.Request{Class: crewroute.Other, Candidates: CrewCandidatesAt(profileDir)}) + if err != nil { + return "" + } + return d.Seat(seat).Send } // ── the gate on a tier value ──────────────────────────────────────────────── -// writeTierModel is the writer all five tier rows share: validate the notation, -// then persist. It is one function rather than five closures so a sixth tier -// cannot arrive with a gate somebody forgot to put on it. +// writeTierModel is the writer the two rows that are not crew seats share: +// validate the notation, then persist. The crew's three rows write through +// [SetCrewPin], which runs the same gate and the allowed-models rule besides. func writeTierModel(profileDir, tier, raw string) error { raw = strings.TrimSpace(raw) if err := ValidateTierValue(raw); err != nil { diff --git a/internal/config/crew_test.go b/internal/config/crew_test.go index 70b2c20a4f..6b458d65ad 100644 --- a/internal/config/crew_test.go +++ b/internal/config/crew_test.go @@ -1,643 +1,548 @@ package config import ( - "encoding/json" - "os" + "errors" + "slices" "strings" "testing" - "github.com/Agent-Field/codeaf/internal/roles" + "github.com/Agent-Field/codeaf/internal/catalog" + "github.com/Agent-Field/codeaf/internal/crewroute" ) -// THE CREW, from the three sides a person meets it: what a preset writes, what -// the row says about four values it did not write, and what the row refuses. - -// THE SHIPPED DEFAULTS ARE A PRESET AND NOT FOUR OPINIONS. This is the identity -// the whole derived reading rests on: if the four defaults were not exactly one -// preset's row, a profile nobody had touched would read "custom" about values -// this build chose itself. -func TestTheShippedTierDefaultsAreExactlyTheBalancedCrew(t *testing.T) { - balanced, ok := CrewModels(CrewBalanced) - if !ok { - t.Fatal("there is no balanced preset") - } - for _, c := range []struct{ tier, want string }{ - {ModelTierReflex, DefaultReflexModel}, - {ModelTierLow, DefaultLowModel}, - {ModelTierWorker, DefaultWorkerModel}, - {ModelTierHigh, DefaultHighModel}, - {ModelTierMastermind, DefaultMastermindModel}, - } { - if balanced[c.tier] != c.want { - t.Errorf("balanced sets %s to %q, and the shipped default is %q", - c.tier, balanced[c.tier], c.want) - } +// crewProfile is a profile with an OpenRouter key and a small catalog, the +// ordinary state a crew is routed in. Every variable a seat or a key reads +// is cleared so the machine running the test cannot answer for it. +func crewProfile(t *testing.T) string { + t.Helper() + for _, name := range []string{APIKeyEnv, "OPENAI_API_KEY", ModelEnv, PlanModelEnv, CheckModelEnv, "CODEAF_BASE_URL", + "DEEPSEEK_API_KEY", "ZHIPU_API_KEY", "MOONSHOT_API_KEY", "MINIMAX_API_KEY", "DASHSCOPE_API_KEY"} { + t.Setenv(name, "") } - if got := CrewAt(t.TempDir()); got != DefaultCrew { - t.Fatalf("a profile nobody has touched reads the crew as %q, want %q", got, DefaultCrew) + dir := t.TempDir() + if err := WriteAPIKey(dir, "sk-or-v1-crewtest-0123456789"); err != nil { + t.Fatal(err) } + previous := CrewCatalog + t.Cleanup(func() { CrewCatalog = previous }) + rows := []catalog.Model{ + {ID: "z-ai/glm-5.3-flash", CanonicalSlug: "z-ai/glm-5.3-flash-20260826", OpenWeights: true, PromptPrice: 1.5e-7, CompletionPrice: 5e-7, CacheReadPrice: 5e-8, + IntelligenceIndex: 41.8, CodingIndex: 71.5, AgenticIndex: 50.9, ArenaElo: 1348, ContextLength: 1310720, Parameters: []string{"tools"}}, + {ID: "moonshotai/kimi-k3", CanonicalSlug: "moonshotai/kimi-k3-20260715", OpenWeights: true, PromptPrice: 3e-6, CompletionPrice: 1.5e-5, CacheReadPrice: 3e-7, + IntelligenceIndex: 43.6, CodingIndex: 76.2, AgenticIndex: 50, ArenaElo: 1421, ContextLength: 1048576, Parameters: []string{"tools"}}, + {ID: "deepseek/deepseek-v4-flash", CanonicalSlug: "deepseek/deepseek-v4-flash-20260423", OpenWeights: true, PromptPrice: 8.246e-8, CompletionPrice: 1.6492e-7, CacheReadPrice: 1.6492e-8, + IntelligenceIndex: 24.2, CodingIndex: 56.2, AgenticIndex: 22.2, ArenaElo: 1216, ContextLength: 1048576, Parameters: []string{"tools"}}, + {ID: "anthropic/claude-opus-5", CanonicalSlug: "anthropic/claude-opus-5-20260723", PromptPrice: 5e-6, CompletionPrice: 2.5e-5, CacheReadPrice: 5e-7, + IntelligenceIndex: 50.8, CodingIndex: 78, AgenticIndex: 56.5, ArenaElo: 1372, ContextLength: 1000000, Parameters: []string{"tools"}}, + {ID: "vendor/no-tools", PromptPrice: 1e-9, CompletionPrice: 1e-9, Parameters: []string{"temperature"}}, + {ID: "vendor/unpriced", PriceUnknown: true, Parameters: []string{"tools"}}, + } + CrewCatalog = func() []catalog.Model { return rows } + return dir } -// THE OPEN FAMILY IS LOCKED TOO, and this is its page. It is read through -// [CrewModelsForSource] and not [CrewModels], because the bare spelling answers -// the DEFAULT family and the default family is the catalog-wide one — the all -// table's own page is [TestTheAllFamilyNamesTheLockedModels]. -func TestCrewPresetsNameTheApprovedModels(t *testing.T) { - want := map[string]map[string]string{ - CrewFrugal: { - ModelTierReflex: "mistralai/mistral-nemo", - ModelTierLow: "deepseek/deepseek-v4-flash-0731", - ModelTierWorker: "z-ai/glm-5.3-flash", - ModelTierHigh: "z-ai/glm-5.3-flash", - ModelTierMastermind: "z-ai/glm-5.3-flash", - }, - CrewBalanced: { - ModelTierReflex: "mistralai/mistral-nemo", - ModelTierLow: "deepseek/deepseek-v4-flash-0731", - ModelTierWorker: "z-ai/glm-5.3-flash", - ModelTierHigh: "moonshotai/kimi-k3", - ModelTierMastermind: "z-ai/glm-5.3", - }, - CrewMax: { - ModelTierReflex: "mistralai/mistral-nemo", - ModelTierLow: "deepseek/deepseek-v4-flash-0731", - ModelTierWorker: "z-ai/glm-5.3", - ModelTierHigh: "moonshotai/kimi-k3", - ModelTierMastermind: "z-ai/glm-5.3", - }, - } - for preset, expected := range want { - got, ok := CrewModelsForSource(CrewSourceOpen, preset) - if !ok { - t.Fatalf("there is no %s preset", preset) +const ( + fixTask = "fix: crash when the config file is empty\n\nTraceback (most recent call last):\nValueError: empty" + openTask = "Add a --json flag to the status command so scripts can read it" +) + +func TestAnUntouchedProfileRoutesEverySeat(t *testing.T) { + dir := crewProfile(t) + seats, err := ResolveSeats(dir, SeatFlags{}, CrewAsk{Task: crewroute.Task{Text: fixTask}}) + if err != nil { + t.Fatal(err) + } + // A fix: the model the catalog reads as able for its price in every seat — + // the cheaper v4-flash publishes weaker indexes, and a checker's ability + // must reach the floor. + for seat, want := range map[Seat]string{seats.Work: "z-ai/glm-5.3-flash", seats.Plan: "z-ai/glm-5.3-flash", seats.Check: "z-ai/glm-5.3-flash"} { + if seat.Source != SeatRouted || seat.Model != want { + t.Errorf("%s: %+v, want routed to %s on a fix", seat.Role, seat, want) } - for _, tier := range ModelTiers { - if got[tier] != expected[tier] { - t.Errorf("%s sets %s to %q, want %q", preset, tier, got[tier], expected[tier]) - } + } + if seats.Crew == nil || seats.Crew.Class != crewroute.Bugfix { + t.Fatalf("crew %+v, want a bugfix decision", seats.Crew) + } + fixEst := seats.Crew.EstUSD + seats, err = ResolveSeats(dir, SeatFlags{}, CrewAsk{Task: crewroute.Task{Text: openTask}}) + if err != nil { + t.Fatal(err) + } + if seats.Work.Model != "z-ai/glm-5.3-flash" || seats.Crew.EstUSD <= fixEst { + t.Errorf("open-ended: worker %s at $%.3f; want the flash worker on a dearer crew than the fix's $%.3f", + seats.Work.Model, seats.Crew.EstUSD, fixEst) + } + for _, m := range []string{seats.Work.Model, seats.Plan.Model, seats.Check.Model} { + if strings.Contains(m, "opus") || strings.Contains(m, "fable") { + t.Errorf("a seat fell to %s", m) } } } -// Every preset names every class, and every model in it is a whole slug. A -// preset with a gap in it would write a blank into a tier row, which means -// "follow the conversation" — the opposite of choosing a crew. -func TestEveryPresetNamesEveryClass(t *testing.T) { - for _, preset := range CrewPresets { - models, ok := CrewModels(preset) - if !ok { - t.Fatalf("%s is in CrewPresets and has no models", preset) - } - for _, tier := range ModelTiers { - model := models[tier] - if strings.TrimSpace(model) == "" { - t.Errorf("%s leaves %s blank, which means follow the conversation", preset, tier) - } - if !strings.Contains(model, "/") { - t.Errorf("%s sets %s to %q, which is not a provider-qualified id", preset, tier, model) - } - // A level is only ever legible on a value the gate accepts. - if err := ValidateTierValue(model); err != nil { - t.Errorf("%s sets %s to %q, which its own row would refuse: %v", preset, tier, model, err) - } - } - if CrewLine(preset) == "" { - t.Errorf("%s has no line to say about itself", preset) - } +// THE CHECK SEAT NEVER QUIETLY INHERITS THE PLANNER. A person who pinned the +// planner said something about planning; the checker is routed for the task. +func TestAPlanFlagDoesNotSeatTheChecker(t *testing.T) { + dir := crewProfile(t) + seats, err := ResolveSeats(dir, SeatFlags{PlanModel: "anthropic/claude-opus-5"}, CrewAsk{Task: crewroute.Task{Text: fixTask}}) + if err != nil { + t.Fatal(err) } - if CrewLine(CrewCustom) != "" { - t.Error("custom has a line, and custom is a reading rather than a choice") + if seats.Plan.Source != SeatFlag || seats.Plan.Model != "anthropic/claude-opus-5" { + t.Errorf("planner %+v, want the flag", seats.Plan) } - if _, ok := CrewModels(CrewCustom); ok { - t.Error("custom names four models, and it is not a preset") + if seats.Check.Source != SeatRouted || seats.Check.Model == "anthropic/claude-opus-5" { + t.Errorf("checker %+v inherited the planner's flag", seats.Check) + } + if !seats.Crew.Seat(crewroute.Planner).Pinned { + t.Error("the flag did not reach the router as a one-task pin") } } -// Setting a preset writes all five tier keys, and the row reads that preset back. -func TestSettingAPresetWritesAllFiveTiers(t *testing.T) { - dir := t.TempDir() - rows := registry(t, dir) - crew := mustRow(t, rows, KeyCrew) - - if err := crew.Apply(CrewMax); err != nil { - t.Fatalf("setting the crew to max: %v", err) - } - want, _ := CrewModels(CrewMax) - for _, tier := range ModelTiers { - if got := TierModelAt(dir, tier); got != want[tier] { - t.Errorf("after max, the %s tier reads %q, want %q", tier, got, want[tier]) - } - } - if got := CrewAt(dir); got != CrewMax { - t.Fatalf("the crew reads %q after max was set", got) +func TestTheLadderFlagThenEnvThenPinThenRouter(t *testing.T) { + dir := crewProfile(t) + if err := SetCrewPin(dir, crewroute.Checker, "moonshotai/kimi-k3"); err != nil { + t.Fatal(err) } - // THE FIVE KEYS ARE ON DISK, all of them, in one file — a preset is not a - // word stored beside four rows it claims to have written. - values := map[string]json.RawMessage{} - raw, err := os.ReadFile(BudgetConfigPath(dir)) + t.Setenv(ModelEnv, "deepseek/deepseek-v4-flash") + seats, err := ResolveSeats(dir, SeatFlags{}, CrewAsk{Task: crewroute.Task{Text: fixTask}}) if err != nil { t.Fatal(err) } - if err := json.Unmarshal(raw, &values); err != nil { - t.Fatal(err) + if seats.Work.Source != SeatEnv || seats.Work.Rung() != ModelEnv { + t.Errorf("worker %+v, want the variable", seats.Work) } - for _, tier := range ModelTiers { - if _, written := values[tierKeyFor(tier)]; !written { - t.Errorf("%s is not in config.json after a preset was applied", tierKeyFor(tier)) - } + if seats.Check.Source != SeatPinned || seats.Check.Model != "moonshotai/kimi-k3" { + t.Errorf("checker %+v, want the profile pin", seats.Check) } - if _, written := values[KeyCrew]; written { - t.Error("models.crew was stored — the reading is derived, and a stored word can become a lie") + if seats.Plan.Source != SeatRouted { + t.Errorf("planner %+v, want routed", seats.Plan) } - - if err := crew.Apply(CrewFrugal); err != nil { - t.Fatalf("setting the crew to frugal: %v", err) + seats, _ = ResolveSeats(dir, SeatFlags{Model: "z-ai/glm-5.3-flash"}, CrewAsk{Task: crewroute.Task{Text: fixTask}, + Pins: map[crewroute.Seat]CrewPin{crewroute.Checker: {Model: "deepseek/deepseek-v4-flash"}}}) + if seats.Work.Source != SeatFlag { + t.Errorf("a flag lost to the variable: %+v", seats.Work) + } + if seats.Check.Model != "deepseek/deepseek-v4-flash" || seats.Check.Source != SeatPinned { + t.Errorf("a one-task --pin lost to the profile pin: %+v", seats.Check) } - if got := CrewAt(dir); got != CrewFrugal { - t.Fatalf("the crew reads %q after frugal was set", got) + // And the one-task pin did not persist. + if pin, _ := CrewPinAt(dir, crewroute.Checker); pin.Model != "moonshotai/kimi-k3" { + t.Errorf("the profile pin moved to %q", pin.Model) } } -// ANSWERING ONE TIER ROW TURNS THE CREW TO CUSTOM, because that is what happened. -func TestAnsweringOneTierRowMakesTheCrewCustom(t *testing.T) { - dir := t.TempDir() - rows := registry(t, dir) - - if err := mustRow(t, rows, KeyCrew).Apply(CrewBalanced); err != nil { +func TestPinsRoundTripAndRefuseWhatTheRuleLeavesOut(t *testing.T) { + dir := crewProfile(t) + if err := SetCrewPin(dir, crewroute.Worker, "z-ai/glm-5.3-flash@openrouter"); err != nil { t.Fatal(err) } - if err := mustRow(t, rows, KeyTierMastermindModel).Apply("openai/gpt-5"); err != nil { - t.Fatal(err) - } - if got := CrewAt(dir); got != CrewCustom { - t.Fatalf("the crew reads %q over a hand-set mastermind, want %q", got, CrewCustom) + pin, ok := CrewPinAt(dir, crewroute.Worker) + if !ok || pin.Model != "z-ai/glm-5.3-flash" || pin.Provider != "openrouter" { + t.Fatalf("pin read back %+v, %v", pin, ok) } - // And the row itself says so, which is the whole point: the summary cannot - // disagree with the four rows it summarizes. - if got := mustRow(t, registry(t, dir), KeyCrew).Value(); got != CrewCustom { - t.Fatalf("the crew row reads %q", got) + if got := mustRow(t, registry(t, dir), KeyTierWorkerModel).Value(); got != "z-ai/glm-5.3-flash@openrouter" { + t.Errorf("the worker row reads %q", got) } - - // A tier CLEARED on purpose is custom too — "one of these follows the - // conversation" is not any of the three presets. - if err := mustRow(t, rows, KeyTierMastermindModel).Apply(""); err != nil { + if err := SetCrewAllowed(dir, "open"); err != nil { t.Fatal(err) } - if got := CrewAt(dir); got != CrewCustom { - t.Fatalf("the crew reads %q with a cleared mastermind, want %q", got, CrewCustom) + if err := SetCrewPin(dir, crewroute.Planner, "anthropic/claude-opus-5"); err == nil || !strings.Contains(err.Error(), "outside the models you allow") { + t.Errorf("a closed model pinned under `open`: %v", err) } - - // Putting the preset back is one keystroke and heals all five. - if err := mustRow(t, registry(t, dir), KeyCrew).Apply(CrewBalanced); err != nil { + if err := SetCrewPin(dir, crewroute.Planner, "moonshotai/kimi-k3@fireworks"); err == nil || !strings.Contains(err.Error(), "not a connected provider") { + t.Errorf("a pin on an unconnected provider: %v", err) + } + // A rule that would strand a pin is refused and names the pin. + if err := SetCrewAllowed(dir, "≤0.1/0.2"); err == nil || !strings.Contains(err.Error(), "pinned to z-ai/glm-5.3-flash") { + t.Errorf("a rule that strands the worker pin: %v", err) + } + if err := SetCrewPin(dir, crewroute.Worker, "auto"); err != nil { t.Fatal(err) } - if got := CrewAt(dir); got != CrewBalanced { - t.Fatalf("the crew reads %q after balanced was set again", got) + if _, ok := CrewPinAt(dir, crewroute.Worker); ok { + t.Error("`auto` did not unpin the worker") + } + if got := mustRow(t, registry(t, dir), KeyTierWorkerModel).Value(); got != CrewAuto { + t.Errorf("an unpinned worker row reads %q, want auto", got) } } -// The row refuses a word that is not a preset, in the words every choice row -// refuses in — and it refuses "custom", which is a reading and not a choice. -func TestTheCrewRowRefusesAWordThatIsNotAPreset(t *testing.T) { - dir := t.TempDir() - crew := mustRow(t, registry(t, dir), KeyCrew) - for _, word := range []string{"cheap", CrewCustom, ""} { - err := crew.Apply(word) - if err == nil { - t.Fatalf("the crew row took %q", word) - } - if !strings.Contains(err.Error(), CrewBalanced) { - t.Errorf("refusing %q does not name the choices: %v", word, err) +func TestTheAllowedRuleNarrowsTheCandidates(t *testing.T) { + dir := crewProfile(t) + ids := func() []string { return crewroute.Names(CrewCandidatesAt(dir)) } + all := ids() + for _, want := range []string{"anthropic/claude-opus-5", "z-ai/glm-5.3-flash"} { + if !slices.Contains(all, want) { + t.Errorf("all: %v is missing %s", all, want) } } - if got := CrewAt(dir); got != DefaultCrew { - t.Fatalf("a refused write changed the crew to %q", got) + if slices.Contains(all, "vendor/unpriced") { + t.Errorf("an unpriced row is a candidate: %v", all) } -} - -// ── the family the three words draw from ──────────────────────────────────── - -// THE CAREFUL SEAT IS A SECOND VENDOR, IN EVERY PRESET OF THE ALL FAMILY. -// A check from the same vendor as the work it checks is a check that shares -// that vendor's blind spots. The open family's frugal row is the one standing -// exception: worker and careful both sit on glm-5.3-flash there, because at -// that preset's bill the open-weight front has no second vendor to take the -// careful seat. -func TestTheCarefulSeatIsAlwaysASecondVendorInEveryFamily(t *testing.T) { - vendor := func(id string) string { - if at := strings.Index(id, "/"); at > 0 { - return id[:at] - } - return id + if err := ModifyCrewAllowed(dir, false, "anthropic"); err != nil { + t.Fatal(err) } - for _, family := range CrewSources { - for _, preset := range CrewPresets { - if family == CrewSourceOpen && preset == CrewFrugal { - continue - } - models, ok := CrewModelsForSource(family, preset) - if !ok { - t.Fatalf("there is no %s preset in the %s family", preset, family) - } - if vendor(models[ModelTierWorker]) == vendor(models[ModelTierHigh]) { - t.Errorf("%s under %s has %q working and %q checking, and a check from "+ - "the same vendor catches what that vendor lets through", - preset, family, models[ModelTierWorker], models[ModelTierHigh]) - } - } + if got := ids(); slices.Contains(got, "anthropic/claude-opus-5") { + t.Errorf("-anthropic left %v", got) + } + if got := CrewAllowedAt(dir).String(); got != "all -anthropic" { + t.Errorf("rule reads %q", got) } } -// THE ALL FAMILY IS LOCKED, not derived: this is the page that says which ids -// the three words mean. A wrong id here is a person's bill, and a silent -// rewrite of the locked table would be a decision made by a diff. -func TestTheAllFamilyNamesTheLockedModels(t *testing.T) { - want := map[string]map[string]string{ - CrewFrugal: { - ModelTierReflex: "google/gemini-2.5-flash", - ModelTierLow: "deepseek/deepseek-v4-flash-0731", - ModelTierWorker: "z-ai/glm-5.3-flash", - ModelTierHigh: "qwen/qwen3.8-max-0902", - ModelTierMastermind: "z-ai/glm-5.3-flash", - }, - CrewBalanced: { - ModelTierReflex: "google/gemini-2.5-flash", - ModelTierLow: "deepseek/deepseek-v4-flash-0731", - ModelTierWorker: "z-ai/glm-5.3-flash", - ModelTierHigh: "anthropic/claude-fable-5.1", - ModelTierMastermind: "anthropic/claude-opus-5", - }, - CrewMax: { - ModelTierReflex: "google/gemini-2.5-flash", - ModelTierLow: "deepseek/deepseek-v4-flash-0731", - ModelTierWorker: "z-ai/glm-5.3", - ModelTierHigh: "anthropic/claude-fable-5.1", - ModelTierMastermind: "anthropic/claude-opus-5", - }, - } - for preset, expected := range want { - got, ok := CrewModelsForSource(CrewSourceAll, preset) - if !ok { - t.Fatalf("there is no %s preset in the all family", preset) - } - for _, tier := range ModelTiers { - if got[tier] != expected[tier] { - t.Errorf("%s under all sets %s to %q, want %q", preset, tier, got[tier], expected[tier]) - } - if !strings.Contains(got[tier], "/") { - t.Errorf("%s under all sets %s to %q, which is not a provider-qualified id", preset, tier, got[tier]) - } - if err := ValidateTierValue(got[tier]); err != nil { - t.Errorf("%s under all sets %s to %q, which its own row would refuse: %v", preset, tier, got[tier], err) - } - } +func TestADailyCapPacesAndThenStops(t *testing.T) { + dir := crewProfile(t) + if err := SetCrewCap(dir, "1"); err != nil { + t.Fatal(err) } - // THE LINES ARE PART OF THE TABLE, and they are the words a person reads while - // deciding to spend frontier money: a line naming a model the preset does not - // pick is the contradiction the chooser exists to prevent. - wantLines := map[string]map[string]string{ - CrewSourceOpen: { - CrewFrugal: "glm-flash works, checks and thinks · pennies a day", - CrewBalanced: "glm-flash works, kimi-k3 checks, glm-5.3 thinks", - CrewMax: "glm-5.3 works and thinks, kimi-k3 checks", - }, - CrewSourceAll: { - CrewFrugal: "glm-flash works and thinks, qwen-max checks", - CrewBalanced: "glm-flash works, fable checks, opus thinks", - CrewMax: "glm-5.3 works, fable checks, opus thinks", - }, - } - for family, lines := range wantLines { - for preset, line := range lines { - if got := CrewLineFor(family, preset); got != line { - t.Errorf("the %s family's %s line is %q, want %q", family, preset, got, line) - } - } + previous := CrewHistory + t.Cleanup(func() { CrewHistory = previous }) + spent := 0.99 + CrewHistory = func(string) CrewDay { return CrewDay{SpentUSD: spent} } + d, err := RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: openTask}}) + if err != nil { + t.Fatal(err) + } + if d.Seat(crewroute.Checker).Model == "moonshotai/kimi-k3" { + t.Error("at 99% of the cap an open-ended task still bought the dear checker") + } + spent = 1.2 + if _, err := RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: openTask}}); !errors.Is(err, ErrCrewAtCap) { + t.Errorf("over the cap: %v, want ErrCrewAtCap", err) } } -// A SOURCE NOBODY SET IS THE DEFAULT FAMILY, which is the catalog-wide one. The -// open table exists beside it, but a profile that never answered the row -// resolves `all`: the shipped defaults, the all presets under the same words, -// and a write that lands the all ids. This is the default-preservation law for -// the FAMILY. An unknown word is held to the same law, silently, the way a -// retired choice reads everywhere else. -func TestASourceNobodyAnsweredReadsTheDefaultFamily(t *testing.T) { - dir := t.TempDir() - if got := CrewSourceAt(dir); got != DefaultCrewSource { - t.Fatalf("an untouched profile reads the source as %q, want %q", got, DefaultCrewSource) +func TestALearnedOffsetStartsARedoneClassHigher(t *testing.T) { + dir := crewProfile(t) + previous := CrewHistory + t.Cleanup(func() { CrewHistory = previous }) + CrewHistory = func(string) CrewDay { + return CrewDay{Offsets: map[string]int{OffsetKey("repo", crewroute.Bugfix): 1}} } - if got := CrewAt(dir); got != DefaultCrew { - t.Fatalf("an untouched profile reads the crew as %q, want %q", got, DefaultCrew) + d, err := RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: fixTask}, Repo: "repo"}) + if err != nil { + t.Fatal(err) + } + plain, _ := RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: fixTask}, Repo: "elsewhere"}) + if d.Quality <= plain.Quality || d.EstUSD <= plain.EstUSD { + t.Errorf("a fix class redone here starts at quality %.2f ($%.3f), not above another repository's %.2f ($%.3f)", + d.Quality, d.EstUSD, plain.Quality, plain.EstUSD) } - if err := ApplyCrew(dir, CrewMax); err != nil { + if plain.Seat(crewroute.Worker).Model != "z-ai/glm-5.3-flash" { + t.Errorf("another repository inherited the offset: %s", plain.Seat(crewroute.Worker).Model) + } +} + +// A CONNECTED PLAN IS PREFERRED: its marginal cost is nothing, and the send +// goes out through the plan's own prefix. +func TestAPlanRouteIsFreeAndCollidingIdsKeepTheirRouterSpelling(t *testing.T) { + dir := crewProfile(t) + if err := writeProfileValue(dir, keyModelSources, []PersistedSource{{ID: "z-ai", Written: "z-ai", Key: "zai-key-0123456789", Door: "coding-plan", Order: 1}}); err != nil { t.Fatal(err) } - want, _ := CrewModelsForSource(DefaultCrewSource, CrewMax) - for _, tier := range ModelTiers { - if got := TierModelAt(dir, tier); got != want[tier] { - t.Errorf("with no source set, max wrote %s to %q, want the default family's %q", tier, got, want[tier]) + var zai CrewProvider + for _, p := range CrewProvidersAt(dir) { + if p.ID == "z-ai" { + zai = p } } - // A WORD THIS BUILD DOES NOT KNOW READS AS THE DEFAULT FAMILY, silently. - // Being told a preference from an older build is now an error is the one - // thing a retirement must never do, and a family is not a thing to guess - // either. - if err := writeProfileValue(dir, KeyCrewSource, "wide"); err != nil { - t.Fatal(err) + if zai.Kind != crewroute.Plan { + t.Fatalf("z-ai on its coding plan: %+v", zai) } - if got := CrewSourceAt(dir); got != DefaultCrewSource { - t.Fatalf("an unknown source word reads as %q, want %q", got, DefaultCrewSource) + d, err := RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: fixTask}}) + if err != nil { + t.Fatal(err) } - if got := CrewAt(dir); got != CrewMax { - t.Fatalf("an unknown source word moved the crew to %q", got) + w := d.Seat(crewroute.Worker) + if w.Provider != "z-ai" || w.CostUSD != 0 || w.Send != "z-ai/glm-5.3-flash" { + t.Errorf("worker %+v, want glm-5.3-flash on the coding plan", w) } -} - -// FLIPPING TO ALL MAKES THE SAME WORDS MEAN THE ALL FAMILY. Seats nobody pinned -// move at once, because an unwritten seat is the default crew resolved in the -// current family; the five ids already written are theirs, so a flip is a -// meaning change to them rather than a write, and they read custom against the -// new family until one keystroke re-applies them. -func TestFlippingToAllMakesTheWordsMeanTheAllFamily(t *testing.T) { - dir := t.TempDir() - rows := registry(t, dir) - if err := mustRow(t, rows, KeyCrewSource).Apply(CrewSourceAll); err != nil { - t.Fatalf("setting the family to all: %v", err) - } - if got := CrewSourceAt(dir); got != CrewSourceAll { - t.Fatalf("the source row reads %q after all was set", got) - } - // The crew row never writes the source: a stored tier id is not a family claim, - // and the family is not stored in the crew row either. - if err := mustRow(t, registry(t, dir), KeyCrew).Apply(CrewBalanced); err != nil { - t.Fatalf("setting the crew to balanced: %v", err) - } - want, _ := CrewModelsForSource(CrewSourceAll, CrewBalanced) - for _, tier := range ModelTiers { - if got := TierModelAt(dir, tier); got != want[tier] { - t.Errorf("under all, balanced wrote %s to %q, want the all-family %q", tier, got, want[tier]) - } + // A z-ai model pinned to OpenRouter is spelled so the call goes there. + if err := SetCrewPin(dir, crewroute.Planner, "z-ai/glm-5.3-flash@openrouter"); err != nil { + t.Fatal(err) } - if got := CrewAt(dir); got != CrewBalanced { - t.Fatalf("the crew reads %q after balanced was applied under all", got) + d, _ = RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: fixTask}}) + if p := d.Seat(crewroute.Planner); p.Send != "openrouter/z-ai/glm-5.3-flash" || p.Provider != "openrouter" { + t.Errorf("planner pinned @openrouter: %+v", p) } +} - // And the same five ids are NOT balanced in the open family, which is the - // point of the toggle: the words changed family, and the reading follows. - if err := mustRow(t, registry(t, dir), KeyCrewSource).Apply(CrewSourceOpen); err != nil { +func TestMigratingARetiredCrew(t *testing.T) { + dir := crewProfile(t) + // A balanced preset applied in the `open` family: all five rows written, + // the three seat rows exactly the preset's, and the family and pick rows. + // The ids stay pins: no row says which hand wrote it. + if err := writeProfileValues(dir, map[string]any{ + KeyTierReflexModel: "mistralai/mistral-nemo", KeyTierLowModel: "deepseek/deepseek-v4-flash-0731", + KeyTierWorkerModel: "z-ai/glm-5.3-flash", KeyTierHighModel: "moonshotai/kimi-k3", KeyTierMastermindModel: "z-ai/glm-5.3", + legacyKeyCrewSource: "open", legacyKeyCrewPick: "learn", + }); err != nil { t.Fatal(err) } - if got := CrewAt(dir); got != CrewCustom { - t.Fatalf("after flipping back to open, the all-family five read %q, want %q", got, CrewCustom) - } - // One keystroke heals it, and heals it open. - if err := mustRow(t, registry(t, dir), KeyCrew).Apply(CrewBalanced); err != nil { + line, err := MigrateCrew(dir) + if err != nil { t.Fatal(err) } - if got := CrewAt(dir); got != CrewBalanced { - t.Fatalf("the crew reads %q after balanced was applied under open again", got) + if !strings.Contains(line, "kept your pins: worker z-ai/glm-5.3-flash, planner z-ai/glm-5.3, checker moonshotai/kimi-k3") || !strings.Contains(line, "allowed models: open") { + t.Errorf("notice %q", line) + } + if got := CrewAllowedAt(dir).String(); got != "open" { + t.Errorf("the open family became %q", got) } - open, _ := CrewModelsForSource(CrewSourceOpen, CrewBalanced) - for _, tier := range ModelTiers { - if got := TierModelAt(dir, tier); got != open[tier] { - t.Errorf("after re-applying under open, %s reads %q, want the open %q", tier, got, open[tier]) + for _, key := range []string{legacyKeyCrewSource, legacyKeyCrewPick} { + if _, held := persistedValue(dir, key); held { + t.Errorf("%s survived the migration", key) } } + if pins := CrewPinsAt(dir); len(pins) != 3 { + t.Errorf("the seat rows are pins %v, want all three kept", pins) + } + if got := TierModelAt(dir, ModelTierReflex); got != "mistralai/mistral-nemo" { + t.Errorf("the reflex row, not a crew seat, moved to %q", got) + } + // Once: the second run has nothing to say. + if again, _ := MigrateCrew(dir); again != "" { + t.Errorf("a second migration said %q", again) + } } -// ── the level on a tier value ─────────────────────────────────────────────── - -// A LEVEL IS ACCEPTED AND ANY OTHER SUFFIX IS REFUSED IN WORDS. -func TestTheTierRowsTakeALevelAndRefuseAMisspeltOne(t *testing.T) { - dir := t.TempDir() - row := mustRow(t, registry(t, dir), KeyTierMastermindModel) - - for _, value := range []string{ - "moonshotai/kimi-k3:low", - "moonshotai/kimi-k3:medium", - "moonshotai/kimi-k3:high", - "deepseek/deepseek-v4-flash", - "", - } { - if err := row.Apply(value); err != nil { - t.Errorf("the mastermind row refused %q: %v", value, err) - } +// THE OWNER'S PROFILE: the three seats all hold the id an old preset shipped, +// beside written low and reflex rows. Nothing retired is on it, so nothing +// moves: all three stay pins, every run. +func TestAPresetsIdsAPersonWroteStayPinned(t *testing.T) { + dir := crewProfile(t) + if err := writeProfileValues(dir, map[string]any{ + KeyTierReflexModel: "mistralai/mistral-nemo", KeyTierLowModel: "z-ai/glm-5.3-flash", + KeyTierWorkerModel: "z-ai/glm-5.3-flash", KeyTierHighModel: "z-ai/glm-5.3-flash", KeyTierMastermindModel: "z-ai/glm-5.3-flash", + }); err != nil { + t.Fatal(err) } - - for _, value := range []string{ - // off is an effort word this build knows and a tier value may not carry - // it: it is a different request shape, and some endpoints refuse it. - "moonshotai/kimi-k3:off", - "moonshotai/kimi-k3:max", - "moonshotai/kimi-k3:none", - "moonshotai/kimi-k3:xhigh", - "moonshotai/kimi-k3:ultra", - "moonshotai/kimi-k3:", - } { - err := row.Apply(value) - if err == nil { - t.Errorf("the mastermind row took %q", value) - continue + for run := 0; run < 2; run++ { + if line, err := MigrateCrew(dir); err != nil || line != "" { + t.Fatalf("run %d migrated a profile with nothing retired: %q %v", run, line, err) } - for _, level := range roles.Efforts { - if !strings.Contains(err.Error(), level) { - t.Errorf("refusing %q does not name %q: %v", value, level, err) - } + } + for _, seat := range crewroute.Seats { + if pin, ok := CrewPinAt(dir, seat); !ok || pin.Model != "z-ai/glm-5.3-flash" { + t.Errorf("the %s is %+v, %v, want its pin kept", seat, pin, ok) } } - - // EVERY TIER ROW HAS THE SAME GATE. A sixth tier must not be able to arrive - // with the check missing. - for _, key := range []string{KeyTierReflexModel, KeyTierLowModel, KeyTierWorkerModel, KeyTierHighModel} { - if err := mustRow(t, registry(t, dir), key).Apply("some/model:off"); err == nil { - t.Errorf("%s took a level it cannot honour", key) + for _, key := range []string{KeyTierWorkerModel, KeyTierHighModel, KeyTierMastermindModel, KeyTierLowModel, KeyTierReflexModel} { + if _, held := persistedValue(dir, key); !held { + t.Errorf("%s was deleted", key) } } } -// A level written into a tier row reaches [roles.ResolveCall] as its own half, -// and never as part of the model id. -func TestALevelOnATierRowResolvesAsAnEffortAndNotAsAnId(t *testing.T) { - dir := t.TempDir() - if err := mustRow(t, registry(t, dir), KeyTierMastermindModel).Apply("moonshotai/kimi-k3:high"); err != nil { +// A CHECKER PINNED ALONE BESIDE A RETIRED WORD IS KEPT, and the line says so; +// migrating again changes nothing. +func TestAPinnedCheckerSurvivesMigration(t *testing.T) { + dir := crewProfile(t) + if err := writeProfileValues(dir, map[string]any{KeyTierHighModel: "moonshotai/kimi-k3", legacyKeyCrew: "balanced"}); err != nil { t.Fatal(err) } - source := roles.Source(func(key string) (string, bool) { - if key == roles.TierKey(roles.TierMastermind) { - return TierModelAt(dir, ModelTierMastermind), true - } - return "", false - }) - call, err := roles.ResolveCall(source, roles.RolePlanner, "vendor/conversation") - if err != nil { - t.Fatal(err) + line, err := MigrateCrew(dir) + if err != nil || !strings.Contains(line, "kept your pins: checker moonshotai/kimi-k3") { + t.Fatalf("notice %q %v", line, err) + } + if again, _ := MigrateCrew(dir); again != "" { + t.Errorf("a second migration said %q", again) } - if call.Model != "moonshotai/kimi-k3" || call.Effort != "high" { - t.Fatalf("the planner resolved to %+v", call) + if pin, ok := CrewPinAt(dir, crewroute.Checker); !ok || pin.Model != "moonshotai/kimi-k3" { + t.Errorf("the checker is %+v, %v", pin, ok) } } -// Every persisted write bumps the generation, which is the signal a live crew -// source invalidates its snapshot on (cmd/codeaf's v3RolesSource). -func TestEverySettingsWriteBumpsTheGeneration(t *testing.T) { - dir := t.TempDir() - before := SettingsGeneration() - if err := mustRow(t, registry(t, dir), KeyCrew).Apply(CrewMax); err != nil { +func TestMigrationKeepsAHandWrittenIdAsAPin(t *testing.T) { + dir := crewProfile(t) + if err := writeProfileValues(dir, map[string]any{ + KeyTierHighModel: "anthropic/claude-opus-5", KeyTierWorkerModel: "auto", legacyKeyCrew: "frugal", + }); err != nil { t.Fatal(err) } - after := SettingsGeneration() - if after <= before { - t.Fatalf("the generation went %d → %d across a write", before, after) + line, err := MigrateCrew(dir) + if err != nil { + t.Fatal(err) } - // A refused write changes nothing, so it must not move the counter — a - // reader that rebuilt on every rejected keystroke would be paying for - // somebody's typing. - _ = mustRow(t, registry(t, dir), KeyCrew).Apply("nonsense") - if got := SettingsGeneration(); got != after { - t.Fatalf("a refused write moved the generation to %d", got) + if !strings.Contains(line, "kept your pins: checker anthropic/claude-opus-5") { + t.Errorf("notice %q", line) } -} - -// THE FAMILY ROW REFUSES A WORD THAT IS NOT A FAMILY, the way the crew row -// refuses a word that is not a preset, and a refused write leaves the row absent -// rather than half-written. -func TestSetCrewSourceRefusesAWordThatIsNotAFamily(t *testing.T) { - dir := t.TempDir() - if err := SetCrewSource(dir, "wide"); err == nil { - t.Fatal("SetCrewSource accepted a word that is not a family") + if pin, ok := CrewPinAt(dir, crewroute.Checker); !ok || pin.Model != "anthropic/claude-opus-5" { + t.Errorf("the hand-written checker is %+v, %v", pin, ok) } - if got := CrewSourceAt(dir); got != DefaultCrewSource { - t.Fatalf("a refused write left the row reading %q", got) + if _, ok := CrewPinAt(dir, crewroute.Worker); ok { + t.Error("an `auto` row became a pin") } - if CrewConfigured(dir) { - t.Fatal("a refused write marked the crew as answered") + if _, held := persistedValue(dir, legacyKeyCrew); held { + t.Error("the preset word survived") } } -// AN UNWRITTEN SEAT FOLLOWS THE FAMILY, which is what makes the hint true: a -// profile on the all family with no rows runs the frontier crew without anyone -// picking it, and the crew word already reads the default preset. -func TestAnUnwrittenSeatFollowsTheFamily(t *testing.T) { - dir := t.TempDir() - if err := SetCrewSource(dir, CrewSourceAll); err != nil { - t.Fatal(err) - } - want, ok := CrewModelsForSource(CrewSourceAll, DefaultCrew) - if !ok { - t.Fatal("there is no default preset in the all family") - } - for tier, model := range want { - if got := TierModelAt(dir, tier); got != model { - t.Fatalf("%s on an all profile with no row is %q, want the all default %q", tier, got, model) +func TestParseCrewPin(t *testing.T) { + cases := []struct { + raw string + pin CrewPin + auto bool + fails bool + }{ + {"", CrewPin{}, true, false}, + {"auto", CrewPin{}, true, false}, + {"moonshotai/kimi-k3", CrewPin{Model: "moonshotai/kimi-k3"}, false, false}, + {"moonshotai/kimi-k3:high@OpenRouter", CrewPin{Model: "moonshotai/kimi-k3:high", Provider: "openrouter"}, false, false}, + {"moonshotai/kimi-k3@", CrewPin{}, false, true}, + {"moonshotai/kimi-k3:hgih", CrewPin{}, false, true}, + } + for _, tc := range cases { + pin, auto, err := ParseCrewPin(tc.raw) + if (err != nil) != tc.fails || auto != tc.auto || (!tc.fails && pin != tc.pin) { + t.Errorf("ParseCrewPin(%q) = %+v, %v, %v", tc.raw, pin, auto, err) } } - if got := CrewAt(dir); got != CrewBalanced { - t.Fatalf("an all profile with no rows reads the crew as %q, want %q", got, CrewBalanced) - } } -// ── the pick row ──────────────────────────────────────────────────────────── - -// THE PICK ROW DEFAULTS TO THE TABLE, and a word that is not one of the three -// is refused the way every choice row refuses one. -func TestCrewPickDefaultsToTheTable(t *testing.T) { - dir := t.TempDir() - if got := CrewPickAt(dir); got != CrewPickTable { - t.Fatalf("an untouched profile reads the pick %q, want %q", got, CrewPickTable) - } - if got := CrewAt(dir); got != DefaultCrew { - t.Fatalf("an untouched profile reads the crew %q, want %q", got, DefaultCrew) +// THE PICKER'S OFFERS ARE EVERY REACHABLE MODEL, and the rule only marks them: +// a model the rule leaves out is still offered, as not allowed. +func TestCrewOffersMarkWhatTheRuleLeavesOut(t *testing.T) { + dir := crewProfile(t) + if err := SetCrewAllowed(dir, "open"); err != nil { + t.Fatal(err) } - if err := SetCrewPick(dir, CrewPickLearn); err != nil { - t.Fatalf("learn would not set: %v", err) + allowed := map[string]bool{} + for _, offer := range CrewOffersAt(dir) { + if len(offer.Routes) == 0 { + t.Errorf("%s was offered with no route", offer.Model.ID) + } + allowed[offer.Model.ID] = offer.Allowed } - if got := CrewPickAt(dir); got != CrewPickLearn { - t.Fatalf("the pick reads %q after learn was set", got) + if got, ok := allowed["anthropic/claude-opus-5"]; !ok || got { + t.Fatalf("a closed model under `open` reads offered=%v allowed=%v", ok, got) } - if err := SetCrewPick(dir, "measured"); err == nil { - t.Fatal("a word that is not one of the three was accepted") + if !allowed["moonshotai/kimi-k3"] { + t.Fatal("an open model under `open` is not allowed") } - if got := CrewPickAt(dir); got != CrewPickLearn { - t.Fatalf("a refused write left the pick reading %q", got) + if _, ok := allowed["vendor/unpriced"]; ok { + t.Fatal("an unpriced row was offered") } } -// THE AUTO ROW IS THE OTHER WAY A SEAT IS COMPUTED FROM THE CATALOG'S ROWS, -// beside the pick row this section's tests read. A pick off the table and a -// tier row that says auto both compute from the rows the process holds, and -// the door that waits for them asks [AnyTierAutoAt] for the second. A row -// cleared on purpose says nothing about models, and a crew of model ids asks -// for nothing — but the word reaches the read through the lineage too, so a -// profile older than the worker seat counts its small-work row as well. -func TestAnyTierAutoAtAnswersWhenARowSaysAuto(t *testing.T) { - dir := t.TempDir() - if AnyTierAutoAt(dir) { - t.Fatal("an untouched profile has no auto row to wait for") - } - if err := ApplyCrew(dir, CrewBalanced); err != nil { +// UNDO PUTS THE ROWS BACK, absent ones included, in one write. +func TestCrewStateRestoresTheRows(t *testing.T) { + dir := crewProfile(t) + before := CrewStateAt(dir) + if err := SetCrewPin(dir, crewroute.Checker, "moonshotai/kimi-k3"); err != nil { t.Fatal(err) } - if AnyTierAutoAt(dir) { - t.Fatal("a crew of model ids has no auto row to wait for") - } - if err := writeProfileValue(dir, tierKeyFor(ModelTierWorker), AutoValue); err != nil { + if err := SetCrewCap(dir, "5"); err != nil { t.Fatal(err) } - if !AnyTierAutoAt(dir) { - t.Fatal("a worker row that says auto went unnoticed") + if err := SetCrewAllowedRule(dir, crewroute.Allowed{Base: crewroute.BaseOpen}); err != nil { + t.Fatal(err) } - if err := writeProfileValue(dir, tierKeyFor(ModelTierWorker), ""); err != nil { + if err := RestoreCrewState(dir, before); err != nil { t.Fatal(err) } - if AnyTierAutoAt(dir) { - t.Fatal("a row cleared on purpose read as an auto row") + if _, ok := CrewPinAt(dir, crewroute.Checker); ok { + t.Fatal("the restored profile still pins the checker") } - vintage := t.TempDir() - if err := writeProfileValue(vintage, tierKeyFor(ModelTierLow), AutoValue); err != nil { - t.Fatal(err) + if CrewCapAt(dir) != 0 || CrewAllowedAt(dir).String() != "all" { + t.Fatalf("restored cap %v rule %q", CrewCapAt(dir), CrewAllowedAt(dir).String()) } - if !AnyTierAutoAt(vintage) { - t.Fatal("an auto row reached through the lineage went unnoticed") + if _, held := persistedValue(dir, KeyCrewCap); held { + t.Fatal("an absent row came back as a written one") } } -// THE PICK-AWARE CREW WORD READS THE BUDGET THE SEATS ARE COMPUTED AT. The -// live seats are computed ids the preset tables do not hold, so the word is -// read off the STORED rows — a profile that applied a crew and then set the -// pick keeps saying the crew it picked — and a row the person emptied on -// purpose still reads custom, because "follows the conversation" is not any -// of the three. -func TestTheCrewWordReadsThePresetThePickComputesAt(t *testing.T) { - dir := t.TempDir() - if err := ApplyCrew(dir, CrewMax); err != nil { +// A PROVIDER TURNED OFF IS A ROUTE TAKEN AWAY, in a row of its own: the router +// never picks through it, the offers mark a model only it reached as not +// allowed, the last provider on cannot be turned off, a pin naming it is +// refused, and the undo puts the row back with the others. +func TestCrewProvidersTurnedOff(t *testing.T) { + dir := crewProfile(t) + if err := writeProfileValue(dir, keyModelSources, []PersistedSource{{ID: "z-ai", Written: "z-ai", Key: "zai-key-0123456789", Door: "coding-plan", Order: 1}}); err != nil { t.Fatal(err) } - if err := SetCrewPick(dir, CrewPickLearn); err != nil { + before := CrewStateAt(dir) + for _, p := range CrewProvidersAt(dir) { + if !p.On { + t.Fatalf("a connection nobody turned off reads off: %+v", p) + } + } + if err := SetCrewProviderOn(dir, "z-ai", false); err != nil { t.Fatal(err) } - if got := CrewAt(dir); got != CrewMax { - t.Fatalf("a max crew under learn reads %q, want %q", got, CrewMax) + if got := CrewProvidersOffAt(dir); !got["z-ai"] || len(got) != 1 { + t.Fatalf("the row reads %v", got) } - // And a hand-typed row makes the word custom, as it always has: the stored - // comparison is the honest one, not a second way to say balanced. - rows, err := os.ReadFile(BudgetConfigPath(dir)) + if rule := CrewAllowedAt(dir).String(); rule != "all" { + t.Fatalf("turning a provider off wrote the allowed rule: %q", rule) + } + for _, c := range CrewCandidatesAt(dir) { + for _, r := range c.Routes { + if r.Provider == "z-ai" { + t.Fatalf("%s is still routed through a provider that is off", c.Model.ID) + } + } + } + d, err := RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: fixTask}}) if err != nil { t.Fatal(err) } - values := map[string]string{} - if err := json.Unmarshal(rows, &values); err != nil { + if w := d.Seat(crewroute.Worker); w.Provider != "openrouter" { + t.Fatalf("the worker rode %s with z-ai off", w.Provider) + } + if err := SetCrewPin(dir, crewroute.Worker, "z-ai/glm-5.3-flash@z-ai"); err == nil || !strings.Contains(err.Error(), "turned off") { + t.Fatalf("a pin through a provider that is off: %v", err) + } + // THE LAST ONE ON STAYS ON. + if err := SetCrewProviderOn(dir, "openrouter", false); !errors.Is(err, ErrCrewLastProvider) { + t.Fatalf("turning off the last provider on: %v", err) + } + if err := SetCrewProviderOn(dir, "nobody", false); err == nil { + t.Fatal("a provider that is not connected was turned off") + } + // ONLY OPENROUTER ON, AND ONLY IT SERVES: every offer is still served. + for _, offer := range CrewOffersAt(dir) { + if !offer.Served { + t.Errorf("%s reads unserved while openrouter is on", offer.Model.ID) + } + } + if err := RestoreCrewState(dir, before); err != nil { t.Fatal(err) } - values[tierKeyFor(ModelTierMastermind)] = "openai/gpt-5" - raw, err := json.Marshal(values) - if err != nil { + if _, held := persistedValue(dir, KeyCrewProvidersOff); held { + t.Fatal("undo left the providers row written") + } + // TURNING THE LAST ONE BACK ON REMOVES THE ROW rather than writing an empty list. + if err := SetCrewProviderOn(dir, "z-ai", false); err != nil { t.Fatal(err) } - if err := os.WriteFile(BudgetConfigPath(dir), raw, 0o600); err != nil { + if err := SetCrewProviderOn(dir, "z-ai", true); err != nil { t.Fatal(err) } - if got := CrewAt(dir); got != CrewCustom { - t.Fatalf("a hand-typed row under learn reads the crew %q, want custom", got) + if _, held := persistedValue(dir, KeyCrewProvidersOff); held { + t.Fatal("every provider on left the row written") + } +} + +// THE GUARD IS ON ROUTING, NOT ON COUNTING: a custom endpoint left on alone +// routes no seat, so turning OpenRouter off beside it is refused — unless +// every seat is pinned to a provider still on. +func TestCrewProvidersKeepOneThatRoutes(t *testing.T) { + dir := crewProfile(t) + custom := PrepareCustomSource(dir, "http://127.0.0.1:9001/v1", "my-vllm") + custom.Key, custom.Order = "sk-my-vllm-0123456789", 1 + if err := WriteSources(dir, []PersistedSource{custom}); err != nil { + t.Fatal(err) + } + if err := SetCrewProviderOn(dir, "openrouter", false); !errors.Is(err, ErrCrewNoRoutableProvider) { + t.Fatalf("turning off the one provider that routes beside a custom endpoint: %v", err) + } + for _, seat := range crewroute.Seats { + if err := SetCrewPin(dir, seat, "my-vllm/qwen-coder"); err != nil { + t.Fatal(err) + } + } + if err := SetCrewProviderOn(dir, "openrouter", false); err != nil { + t.Fatalf("every seat pinned to the custom endpoint, and still refused: %v", err) } } diff --git a/internal/config/crewfresh_test.go b/internal/config/crewfresh_test.go new file mode 100644 index 0000000000..13779871be --- /dev/null +++ b/internal/config/crewfresh_test.go @@ -0,0 +1,72 @@ +package config + +import ( + "os" + "path/filepath" + "testing" + + "github.com/Agent-Field/codeaf/internal/catalog" + "github.com/Agent-Field/codeaf/internal/crewroute" +) + +// A FRESH PROFILE STARTS ON AUTO. With HOME and CODEAF_HOME both empty +// directories and no profile override, every crew seat is routed — worker, +// planner and checker — nothing is pinned, the allowed rule is `all`, a task +// is held to $5, and there is no daily crew cap until one is set. Reading all +// of it writes nothing. +func TestAFreshProfileStartsOnAutoWithTheDefaultLimits(t *testing.T) { + crewProfile(t) // clears the seat and key variables, installs a catalog + t.Setenv("HOME", t.TempDir()) + t.Setenv("CODEAF_HOME", t.TempDir()) + t.Setenv(ProfileDirEnv, "") + dir := ProfileDir() + if dir != "" { + t.Fatalf("profile override leaked: %q", dir) + } + if pins := CrewPinsAt(dir); len(pins) != 0 { + t.Fatalf("a fresh profile has pins: %v", pins) + } + if rule := CrewAllowedAt(dir).String(); rule != "all" { + t.Fatalf("a fresh profile allows %q, want all", rule) + } + if limit := CrewTaskCapAt(dir); limit != 5 { + t.Fatalf("a fresh profile's per-task limit is $%.2f, want $5", limit) + } + if daily := CrewCapAt(dir); daily != 0 { + t.Fatalf("a fresh profile has a daily crew cap of $%.2f, want none", daily) + } + for _, seat := range crewroute.Seats { + row := TierSeatAt(dir, CrewSeatTier(seat)) + if row.Source != SeatRouted { + t.Errorf("%s reads %+v on a fresh profile, want auto (routed)", seat, row) + } + } + if _, err := os.Stat(filepath.Join(os.Getenv("CODEAF_HOME"), "config.json")); !os.IsNotExist(err) { + t.Fatalf("reading a fresh profile wrote config.json: %v", err) + } +} + +// NO SEAT FALLS BACK TO A MODEL THIS BUILD CHOSE. With nothing the router can +// seat, a crew seat reads empty — the role ladder's floor is the model the +// person is talking to — and a task's crew is an error that says so, never a +// hard-coded default. +func TestNoCrewSeatFallsBackToABuiltInModel(t *testing.T) { + dir := crewProfile(t) + CrewCatalog = func() []catalog.Model { return nil } + builtIn := map[string]bool{DefaultModel: true, DefaultReflexModel: true, DefaultLowModel: true, FreeChatModel: true} + for _, seat := range crewroute.Seats { + row := TierSeatAt(dir, CrewSeatTier(seat)) + if row.Model != "" || builtIn[row.Model] { + t.Errorf("%s with nothing routable reads %q, want empty", seat, row.Model) + } + } + seats, err := ResolveSeats(dir, SeatFlags{}, CrewAsk{Task: crewroute.Task{Text: fixTask}}) + if err == nil { + t.Fatalf("a crew with nothing routable resolved: %+v", seats) + } + for _, seat := range []Seat{seats.Work, seats.Plan, seats.Check} { + if builtIn[seat.Model] { + t.Errorf("%s fell back to the built-in %s", seat.Role, seat.Model) + } + } +} diff --git a/internal/config/crewhealth.go b/internal/config/crewhealth.go new file mode 100644 index 0000000000..a237b0e896 --- /dev/null +++ b/internal/config/crewhealth.go @@ -0,0 +1,539 @@ +package config + +import ( + "errors" + "sort" + "strings" + "sync" + "time" + + "github.com/Agent-Field/codeaf/internal/crewroute" + "github.com/Agent-Field/codeaf/internal/modelsource" + "github.com/Agent-Field/codeaf/internal/router" +) + +// ROUTE HEALTH — WHAT THIS INSTALL HAS LEARNED ABOUT EACH ROUTE. +// +// A route is a provider, a send id and a price, and a crew seat's first call on +// it either answers or fails with a kind of failure (internal/provider's +// RouteFailure: payment, auth, forbidden, quota, unavailable, transient). Those +// outcomes are written to the router's log as they happen ([LogCrewRoute]) and +// read back here, oldest first, into what every decision is routed around: +// +// - forbidden → the route is QUARANTINED for [crewQuarantineFor]: this route +// will not take this model for this account, and asking again tomorrow +// will not change that; +// - quota → the route COOLS DOWN until the reset it said, or for +// [crewCooldownFor]; +// - unavailable → the route cools down, and the model is DEMOTED off +// unpinned seats: it is not served there any more; +// - payment → every PAID route on that provider's account is +// UNAFFORDABLE, until a paid call on it answers again; +// - auth → the provider is DISCONNECTED, until a call on it answers +// again — or the person reconnects it; +// - transient → nothing but the route's failure rate. +// +// Every outcome feeds the route's learned chance of refusing a first call, +// which is part of what the route is expected to cost (crewroute's routeCost). +// NOTHING HERE NAMES A PROVIDER: the kinds are the provider layer's, and a +// provider that says its credit is gone before a call is made can say so +// through [CrewAccountState]. + +// The health windows, spelled once. +const ( + // crewQuarantineFor is how long a forbidden route is kept out. + crewQuarantineFor = 7 * 24 * time.Hour + // crewCooldownFor is how long a quota or unavailable route rests when it + // said no reset of its own. + crewCooldownFor = time.Hour + // crewUnavailableFor is how long a route whose model was not served rests. + crewUnavailableFor = 6 * time.Hour + // crewDemoteFor is how far back failures count toward demoting a model, + // and crewDemoteAfter how many start failures (not limits) it takes. + crewDemoteFor = 7 * 24 * time.Hour + crewDemoteAfter = 2 + // A route's learned failure rate is read against a prior worth + // crewPriorTasks tasks at crewPriorRate, so one lucky call does not make a + // route look safe. + crewPriorTasks = 3.0 + crewPriorRate = 0.3 +) + +// CrewAccountState is what a provider adapter can say about an account before +// any call is made: its remaining credit, and when its limit resets. +type CrewAccountState struct { + CreditKnown bool + CreditUSD float64 + ResetAt time.Time +} + +// CrewAccounts is the OPTIONAL adapter that primes route health for providers +// that can report their account state. Nil — the default — is providers that +// cannot, whose health is learned from their calls alone. +var CrewAccounts func(profileDir, provider string) (CrewAccountState, bool) + +// CrewRouteHistory is how route outcomes reach the health reading: the router +// log's recent first-call outcomes. A variable so a test can hand its own. +var CrewRouteHistory = func(profileDir string) []router.CrewRouteOutcome { + return router.ReadCrewLog(ProfilePath(profileDir, ""), time.Now()).Routes +} + +// crewHealth is the reading. +type crewHealth struct { + now time.Time + // blocked is send → until, for a route quarantined or cooling down. + blocked map[string]time.Time + // unaffordable and disconnected are providers. + unaffordable map[string]bool + disconnected map[string]bool + // demoted are model lineages kept off unpinned seats. + demoted map[string]bool + // fail is send → learned chance of refusing a first call. + fail map[string]float64 + // resets is provider → the latest reset a cooling route said. + resets map[string]time.Time + // lowBalance are providers whose balance was READ as low before any call + // ([CreditsLowAt]). They stay unaffordable while a task is probing: the + // reading is taken again at every launch while it is low + // ([CreditsNeedRead]), so a top-up is noticed without a refused call. + lowBalance map[string]bool +} + +// crewRouteFacts are what a candidate list is built with beyond the rule and +// the providers: whether free pools are routes, and route health. +type crewRouteFacts struct { + free bool + health crewHealth +} + +// crewHealthAt reads this profile's route health now. +func crewHealthAt(profileDir string) crewHealth { + var history []router.CrewRouteOutcome + if CrewRouteHistory != nil { + history = CrewRouteHistory(profileDir) + } + h := crewHealthOf(history, time.Now()) + // AN OPENROUTER BALANCE KNOWN TO BE LOW ([CreditsLowAt]) is an account out + // of credit before any call is made: the seats reach for free routes and + // the crew line says so, instead of a first paid call being refused. + if useFreeDefaultsAt(profileDir) { + h.unaffordable[modelsource.DefaultID] = true + h.lowBalance = map[string]bool{modelsource.DefaultID: true} + } + if CrewAccounts != nil { + for _, p := range CrewProvidersAt(profileDir) { + if state, ok := CrewAccounts(profileDir, p.ID); ok && state.CreditKnown && state.CreditUSD <= 0 { + h.unaffordable[p.ID] = true + } + } + } + return h +} + +// crewHealthOf reads route outcomes, oldest first, as health at now. +func crewHealthOf(history []router.CrewRouteOutcome, now time.Time) crewHealth { + h := crewHealth{now: now, blocked: map[string]time.Time{}, unaffordable: map[string]bool{}, + disconnected: map[string]bool{}, demoted: map[string]bool{}, fail: map[string]float64{}, resets: map[string]time.Time{}} + sorted := append([]router.CrewRouteOutcome(nil), history...) + sort.SliceStable(sorted, func(i, j int) bool { return sorted[i].At.Before(sorted[j].At) }) + starts, fails := map[string]int{}, map[string]int{} + startFails := map[string]int{} + block := func(send string, until time.Time) { + if until.After(now) && until.After(h.blocked[send]) { + h.blocked[send] = until + } + } + for _, o := range sorted { + if o.Kind == "" { + starts[o.Send]++ + delete(h.disconnected, o.Provider) + if o.Paid { + delete(h.unaffordable, o.Provider) + } + continue + } + fails[o.Send]++ + lineage := crewroute.Lineage(o.Send) + switch o.Kind { + case "payment": + h.unaffordable[o.Provider] = true + case "auth": + h.disconnected[o.Provider] = true + case "forbidden": + block(o.Send, o.At.Add(crewQuarantineFor)) + if now.Sub(o.At) < crewDemoteFor { + startFails[lineage]++ + } + case "quota": + until := o.Until + if until.IsZero() { + until = o.At.Add(crewCooldownFor) + } + block(o.Send, until) + if until.After(h.resets[o.Provider]) { + h.resets[o.Provider] = until + } + case "unavailable": + block(o.Send, o.At.Add(crewUnavailableFor)) + if now.Sub(o.At) < crewDemoteFor { + h.demoted[lineage] = true + } + } + } + for lineage, n := range startFails { + if n >= crewDemoteAfter { + h.demoted[lineage] = true + } + } + for send := range fails { + n := float64(starts[send] + fails[send]) + h.fail[send] = (float64(fails[send]) + crewPriorRate*crewPriorTasks) / (n + crewPriorTasks) + } + return h +} + +// probing is this health with the accounts it holds out of credit and the +// providers whose key it saw refused put back on trial: what a new task is +// routed under, so its first call asks them again ([RouteCrew]). An account +// whose balance was read as low is not put on trial ([crewHealth.lowBalance]). +func (h crewHealth) probing() crewHealth { + h.unaffordable, h.disconnected = map[string]bool{}, map[string]bool{} + for id := range h.lowBalance { + h.unaffordable[id] = true + } + return h +} + +// CrewHealthySend is the model an auxiliary call — a summary, a brief, a +// landing's answer, anything that is not a crew seat's own call nor the +// person's turn — should ask instead of send, when send's route is one route +// health says will not answer: quarantined or cooling, on an account out of +// credit, on a provider whose key was refused. It is the router's standing +// worker pick, then the seat's rescue; send itself when its route is +// healthy or nothing better is reachable. +// +// AN AUXILIARY CALL NEVER PROBES. Only a task's first seat call asks an +// account out of credit again ([RouteCrew]); a helper that did would spend a +// refusal on every summary. +func CrewHealthySend(profileDir, send, chatModel string) string { + send = strings.TrimSpace(send) + if send == "" { + return send + } + health := crewHealthCached(profileDir) + if health.answers(send, CrewProvidersAt(profileDir)) { + return send + } + if seat := standingCrewSeat(profileDir, crewroute.Worker); seat != "" && seat != send { + return seat + } + for _, rung := range CrewRescue(profileDir, crewroute.Other, crewroute.Worker, chatModel) { + if rung.Send != send { + return rung.Send + } + } + return send +} + +// CrewRouteAnswers is whether route health expects send's route to answer: +// not quarantined or cooling, not on an account out of credit, not on a +// provider whose key was refused. A profile with nothing wrong on it answers +// at once, without reading its providers. +func CrewRouteAnswers(profileDir, send string) bool { + send = strings.TrimSpace(send) + if send == "" { + return true + } + health := crewHealthCached(profileDir) + if len(health.blocked) == 0 && len(health.unaffordable) == 0 && len(health.disconnected) == 0 { + return true + } + return health.answers(send, CrewProvidersAt(profileDir)) +} + +// answers is whether this health expects send's route to answer. +func (h crewHealth) answers(send string, providers []CrewProvider) bool { + if _, blocked := h.blocked[send]; blocked { + return false + } + pin := resolveCrewPin(CrewPin{Model: send}, providers) + return !h.disconnected[pin.Provider] && !(pin.Kind == crewroute.Metered && h.unaffordable[pin.Provider]) +} + +// forgetCrewHealth drops the cached route health. +func forgetCrewHealth() { + crewHealthCache.mu.Lock() + crewHealthCache.at = time.Time{} + crewHealthCache.mu.Unlock() +} + +// crewHealthCached is [crewHealthAt] read at most once a few seconds per +// profile: auxiliary calls ask it on every request, and the router's log is a +// file. +func crewHealthCached(profileDir string) crewHealth { + crewHealthCache.mu.Lock() + defer crewHealthCache.mu.Unlock() + if crewHealthCache.dir == profileDir && time.Since(crewHealthCache.at) < crewHealthFresh { + return crewHealthCache.health + } + crewHealthCache.dir, crewHealthCache.at, crewHealthCache.health = profileDir, time.Now(), crewHealthAt(profileDir) + return crewHealthCache.health +} + +// crewHealthFresh is how long a cached health reading stands. +const crewHealthFresh = 3 * time.Second + +var crewHealthCache struct { + mu sync.Mutex + dir string + at time.Time + health crewHealth +} + +// usable is the routes this health leaves, with their learned failure rates. +func (h crewHealth) usable(routes []crewroute.Route) []crewroute.Route { + out := routes[:0] + for _, r := range routes { + if _, blocked := h.blocked[r.Send]; blocked { + continue + } + if h.disconnected[r.Provider] || (r.Kind == crewroute.Metered && h.unaffordable[r.Provider]) { + continue + } + r.FailRate = h.fail[r.Send] + out = append(out, r) + } + return out +} + +// crewCandidatesNoticed is [CrewCandidatesAt] with the one notice a decision +// carries when it had to reach for free routes. +// +// WHEN NOTHING PAID CAN BE REACHED, THE FREE POOLS ARE USED — whatever the +// free-routes row says — because a task on a free pool is better than a task +// that cannot start, and the line says so: a free pool may log what it is +// sent. A paid route coming back (a paid call answering, a provider reporting +// credit) puts the next task back on normal routing on its own. +func crewCandidatesNoticed(profileDir string, health crewHealth) ([]crewroute.Candidate, string) { + rule, providers, off := CrewAllowedAt(profileDir), CrewProvidersAt(profileDir), CrewProvidersOffAt(profileDir) + facts := crewRouteFacts{free: CrewFreeRoutesAt(profileDir), health: health} + candidates := off.Candidates(crewCandidatesWith(rule, providers, facts)) + if facts.free || len(health.unaffordable) == 0 || anyPaidRoute(candidates) { + return candidates, "" + } + facts.free = true + withFree := off.Candidates(crewCandidatesWith(rule, providers, facts)) + if len(withFree) == len(candidates) { + return candidates, "" + } + return withFree, "free routes in use (may log prompts) · credit unavailable on " + strings.Join(crewSet(health.unaffordable), ", ") +} + +// anyPaidRoute is whether any candidate is reachable on a route that bills. +func anyPaidRoute(candidates []crewroute.Candidate) bool { + for _, c := range candidates { + for _, r := range c.Routes { + if r.Kind == crewroute.Metered { + return true + } + } + } + return false +} + +// crewSet is a set's members in order. +func crewSet(set map[string]bool) []string { + out := make([]string, 0, len(set)) + for k, on := range set { + if on { + out = append(out, k) + } + } + sort.Strings(out) + return out +} + +// crewAction is the ONE thing a person can do when no crew can be formed or a +// seat has nowhere left to go, read off what the routes said — never the +// refusal's own text. +func (h crewHealth) crewAction() string { + switch { + case len(h.unaffordable) > 0: + return "add credit on " + strings.Join(crewSet(h.unaffordable), ", ") + " to continue" + case len(h.disconnected) > 0: + return "reconnect " + strings.Join(crewSet(h.disconnected), ", ") + " with /connect" + } + var latest time.Time + for _, at := range h.resets { + if at.After(latest) { + latest = at + } + } + if !latest.IsZero() && latest.After(h.now) { + return "the limit resets at " + latest.Local().Format("15:04") + " — try again then" + } + return "no allowed model on a connected provider can start · widen /crew models or pin one with /crew" +} + +// pinnedBefore is whether a seat was the person's pin rather than a rescue. +func pinnedBefore(seat crewroute.Seat, pins map[crewroute.Seat]crewroute.Pin, rescued []string) bool { + for _, said := range rescued { + if strings.HasPrefix(said, string(seat)+" on ") { + return false + } + } + _, ok := pins[seat] + return ok +} + +// ErrCrewUnreachable is a crew nothing reachable can form; its text is the one +// action ([crewHealth.crewAction]). +type ErrCrewUnreachable struct{ Action string } + +func (e ErrCrewUnreachable) Error() string { return e.Action } + +// LogCrewRoute writes one seat's first-call outcome on its route: answered +// (kind empty), or failed with a route-failure kind and when the route said it +// may be asked again. +func LogCrewRoute(profileDir, call string, d crewroute.Decision, repo, title string, seat crewroute.Seat, pick crewroute.Pick, kind string, until time.Time) { + if kind != "" { + // A FAILURE IS SEEN BY THE VERY NEXT CALL, not a few seconds later: + // the cache that spares every request a read of the log is dropped. + forgetCrewHealth() + } + router.LogCrewRoute(ProfilePath(profileDir, ""), call, CrewRecordOf(d, repo, title), router.CrewRouteOutcome{ + Seat: string(seat), Send: pick.Send, Provider: pick.Provider, + Paid: pick.Kind == crewroute.Metered, Kind: kind, Until: until, + }) +} + +// CrewRescue is the last of a seat's ladder, after everything the router +// offered: the seat as the last crew that completed a task on this install +// ran it, then the model the person is talking to — each only when its route +// is healthy and the model can sit the seat. The router's own rungs come first +// ([crewroute.Decision.Ladder]). +func CrewRescue(profileDir string, class crewroute.Class, seat crewroute.Seat, chatModel string) []crewroute.Pick { + health := crewHealthAt(profileDir) + var out []crewroute.Pick + seen := map[string]bool{} + add := func(send string) { + send = strings.TrimSpace(send) + if send == "" || seen[send] { + return + } + seen[send] = true + if _, blocked := health.blocked[send]; blocked { + return + } + pin := resolveCrewPin(CrewPin{Model: send}, CrewProvidersAt(profileDir)) + // NOT EVEN THE PERSON'S OWN MODEL rides an account that just said it + // is out of credit, or a key that was just refused: the conversation + // running on it proves the model, not the account. + if health.disconnected[pin.Provider] || (pin.Kind == crewroute.Metered && health.unaffordable[pin.Provider]) { + return + } + if model, known := crewCatalogModel(send); known && !crewroute.Seatable(seat, crewroute.Candidate{Model: model, Routes: []crewroute.Route{{Send: pin.Send}}}) { + return + } + out = append(out, crewroute.Pick{Seat: seat, Model: pin.Model, Provider: pin.Provider, Send: pin.Send, Kind: pin.Kind}) + } + if last := CrewLastGood(profileDir); last != nil { + add(last.Seats[string(seat)]) + } + out = append(out, crewFreeRescue(profileDir, class, seat, health)...) + add(chatModel) + return out +} + +// crewFreeRescue is the seat on a free pool when EVERY PAID ROUTE IS OUT OF +// REACH — the free-routes switch off or on. With the switch on the pools are +// already routes on the seat's ladder; with it off they are used only here, +// and the crew line says so ([crewCandidatesNoticed]'s notice). The best few +// free picks for the seat, each a different model, so a pool at its limit +// leaves the next. +func crewFreeRescue(profileDir string, class crewroute.Class, seat crewroute.Seat, health crewHealth) []crewroute.Pick { + candidates, notice := crewCandidatesNoticed(profileDir, health) + if notice == "" { + return nil + } + var free []crewroute.Candidate + for _, c := range candidates { + var routes []crewroute.Route + for _, r := range c.Routes { + if r.Kind == crewroute.Free { + routes = append(routes, r) + } + } + // THE RESCUE KEEPS A FLOOR: a model tuned for one domain (finance, + // medicine, law) or too small to do a seat's work is no rescue at all, + // however little else is left — a task stopped on its one action is + // better than one sent to a model that cannot do it. + if len(routes) > 0 && !crewroute.DomainTuned(c.Model.ID) && !crewroute.Tiny(c.Model.ID) { + free = append(free, crewroute.Candidate{Model: c.Model, Routes: routes}) + } + } + var out []crewroute.Pick + avoid := map[string]bool{} + for len(out) < crewFreeRescues && len(free) > 0 { + d, err := crewroute.Decide(crewroute.Request{Class: class, Candidates: free, Avoid: avoid, Rescue: true}) + if err != nil { + break + } + pick := d.Seat(seat) + if pick.Send == "" || avoid[crewroute.Lineage(pick.Model)] { + break + } + pick.Pinned = false + out = append(out, pick) + avoid[crewroute.Lineage(pick.Model)] = true + } + return out +} + +// crewFreeRescues is how many free models a seat's rescue tries. +const crewFreeRescues = 3 + +// CrewLastGood is the crew of the newest task whose result was kept. +var CrewLastGood = func(profileDir string) *router.CrewRecord { + return router.ReadCrewLog(ProfilePath(profileDir, ""), time.Now()).LastGood +} + +// crewRescued is a decision the router could not form for want of a seat, +// formed with that seat on its rescue: the last good crew's model, then the +// person's own. Nothing to rescue it with is the one action. +func crewRescued(profileDir string, req crewroute.Request, health crewHealth, chatModel string, cause error) (crewroute.Decision, error) { + var missing crewroute.NoCandidateError + if !errors.As(cause, &missing) { + return crewroute.Decision{}, cause + } + pins := map[crewroute.Seat]crewroute.Pin{} + for seat, pin := range req.Pins { + pins[seat] = pin + } + var rescued []string + for tries := 0; tries < len(crewroute.Seats); tries++ { + rescue := CrewRescue(profileDir, req.Class, missing.Seat, chatModel) + if len(rescue) == 0 { + return crewroute.Decision{}, ErrCrewUnreachable{Action: health.crewAction()} + } + r := rescue[0] + pins[missing.Seat] = crewroute.Pin{Model: r.Model, Provider: r.Provider, Send: r.Send, Kind: r.Kind} + rescued = append(rescued, string(missing.Seat)+" on "+crewroute.ShortModel(r.Model)) + req.Pins = pins + d, err := crewroute.Decide(req) + if err == nil { + // The rescues rode in as pins to be seated; they are not the person's. + for i := range d.Crew { + if _, was := req.Pins[d.Crew[i].Seat]; was && !pinnedBefore(d.Crew[i].Seat, pins, rescued) { + d.Crew[i].Pinned = false + } + } + d.Note = strings.TrimSpace(d.Note + " running on fallback crew · " + strings.Join(rescued, ", ")) + return d, nil + } + if !errors.As(err, &missing) { + return crewroute.Decision{}, err + } + } + return crewroute.Decision{}, ErrCrewUnreachable{Action: health.crewAction()} +} diff --git a/internal/config/crewhealth_test.go b/internal/config/crewhealth_test.go new file mode 100644 index 0000000000..d74757d980 --- /dev/null +++ b/internal/config/crewhealth_test.go @@ -0,0 +1,652 @@ +package config + +import ( + "errors" + "math" + "os" + "strings" + "testing" + "time" + + "github.com/Agent-Field/codeaf/internal/catalog" + "github.com/Agent-Field/codeaf/internal/crewroute" + "github.com/Agent-Field/codeaf/internal/router" +) + +// withRouteHistory hands the health reading a history of its own, and no +// last-good crew unless the test says. +func withRouteHistory(t *testing.T, history *[]router.CrewRouteOutcome) { + t.Helper() + previous, previousGood := CrewRouteHistory, CrewLastGood + t.Cleanup(func() { CrewRouteHistory, CrewLastGood = previous, previousGood }) + CrewRouteHistory = func(string) []router.CrewRouteOutcome { return *history } + CrewLastGood = func(string) *router.CrewRecord { return nil } +} + +// EACH KIND OF FAILURE MOVES A ROUTE'S HEALTH ITS OWN WAY, and a later success +// on the same account puts it back. +func TestRouteHealthTransitions(t *testing.T) { + now := time.Date(2026, 9, 24, 12, 0, 0, 0, time.UTC) + at := func(ago time.Duration) time.Time { return now.Add(-ago) } + reset := now.Add(90 * time.Minute) + h := crewHealthOf([]router.CrewRouteOutcome{ + {At: at(time.Hour), Send: "a/forbidden:free", Provider: "p1", Kind: "forbidden"}, + {At: at(10 * 24 * time.Hour), Send: "a/old-forbidden", Provider: "p1", Kind: "forbidden"}, + {At: at(time.Minute), Send: "a/limited", Provider: "p2", Kind: "quota", Until: reset}, + {At: at(time.Minute), Send: "a/gone", Provider: "p2", Kind: "unavailable"}, + {At: at(time.Hour), Send: "a/pricey", Provider: "broke", Kind: "payment"}, + {At: at(time.Hour), Send: "a/keyless", Provider: "badkey", Kind: "auth"}, + {At: at(2 * time.Hour), Send: "a/twice", Provider: "p3", Kind: "forbidden"}, + {At: at(time.Hour), Send: "openrouter/a/twice", Provider: "p4", Kind: "forbidden"}, + {At: at(time.Hour), Send: "a/flaky", Provider: "p5", Kind: "transient"}, + {At: at(time.Minute), Send: "a/flaky", Provider: "p5"}, + }, now) + if _, ok := h.blocked["a/forbidden:free"]; !ok { + t.Error("a forbidden route was not quarantined") + } + if _, ok := h.blocked["a/old-forbidden"]; ok { + t.Error("a quarantine outlived its week") + } + if until := h.blocked["a/limited"]; !until.Equal(reset) { + t.Errorf("a limited route cools until %v, want its reset %v", until, reset) + } + if !h.demoted[crewroute.Lineage("a/gone")] || !h.demoted[crewroute.Lineage("a/twice")] { + t.Errorf("demoted %v: want the withdrawn model and the one refused on two routes", h.demoted) + } + if !h.unaffordable["broke"] || !h.disconnected["badkey"] { + t.Errorf("unaffordable %v, disconnected %v", h.unaffordable, h.disconnected) + } + if rate := h.fail["a/flaky"]; rate <= 0 || rate >= 1 { + t.Errorf("a route that failed once and answered once reads %v", rate) + } + // A paid call answering on the account clears it; so does a call on the + // provider whose key was refused. + cleared := crewHealthOf([]router.CrewRouteOutcome{ + {At: at(time.Hour), Send: "a/pricey", Provider: "broke", Kind: "payment"}, + {At: at(time.Minute), Send: "a/pricey", Provider: "broke", Paid: true}, + {At: at(time.Hour), Send: "a/keyless", Provider: "badkey", Kind: "auth"}, + {At: at(time.Minute), Send: "a/keyless", Provider: "badkey"}, + }, now) + if cleared.unaffordable["broke"] || cleared.disconnected["badkey"] { + t.Errorf("a later success did not clear: %v %v", cleared.unaffordable, cleared.disconnected) + } +} + +// THE OWNER'S SEQUENCE: a route refused a seat with a 403, and the next task +// was routed onto it again. A forbidden route is quarantined, and the next +// task's crew does not sit on it. +func TestAForbiddenRouteIsNotRoutedToAgain(t *testing.T) { + dir := crewProfile(t) + var history []router.CrewRouteOutcome + withRouteHistory(t, &history) + first, err := RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: fixTask}}) + if err != nil { + t.Fatal(err) + } + worker := first.Seat(crewroute.Worker) + history = append(history, router.CrewRouteOutcome{At: time.Now(), Seat: "worker", Send: worker.Send, Provider: worker.Provider, Kind: "forbidden"}) + second, err := RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: fixTask}}) + if err != nil { + t.Fatal(err) + } + for _, pick := range second.Crew { + if pick.Send == worker.Send { + t.Fatalf("the %s seat went back to the quarantined route %s", pick.Seat, pick.Send) + } + } +} + +// THE PANEL'S GAP COUNTS A PINNED CHECKER. A strong checker pinned over a set +// whose only other model is weak leaves no warning — even while the pin's route +// is quarantined and the candidates no longer carry it — and a weak one pinned +// over a strong set is warned about by name. +func TestTheGapCountsAPinnedChecker(t *testing.T) { + dir := crewProfile(t) + var history []router.CrewRouteOutcome + withRouteHistory(t, &history) + if err := SetCrewAllowed(dir, "deepseek-v4-flash, kimi-k3"); err != nil { + t.Fatal(err) + } + if gaps := CrewGapsAt(dir); len(gaps) != 0 { + t.Fatalf("a set with kimi-k3 in it: gaps %+v", gaps) + } + if err := SetCrewPin(dir, crewroute.Checker, "moonshotai/kimi-k3"); err != nil { + t.Fatal(err) + } + history = append(history, router.CrewRouteOutcome{At: time.Now(), Seat: "checker", Send: "moonshotai/kimi-k3", Provider: "openrouter", Kind: "forbidden"}) + for _, c := range CrewCandidatesAt(dir) { + if crewroute.Lineage(c.Model.ID) == crewroute.Lineage("moonshotai/kimi-k3") && len(c.Routes) > 0 { + t.Fatalf("the quarantined pin is still a candidate: %+v", c) + } + } + if gaps := CrewGapsAt(dir); len(gaps) != 0 { + t.Errorf("kimi-k3 pinned as checker: gaps %+v, want none", gaps) + } + history = nil + if err := SetCrewPin(dir, crewroute.Checker, "deepseek/deepseek-v4-flash"); err != nil { + t.Fatal(err) + } + gaps := CrewGapsAt(dir) + want := "checker pinned to deepseek-v4-flash · open-ended work will be checked weakly" + if len(gaps) != 1 || gaps[0].Line != want { + t.Errorf("v4-flash pinned as checker: gaps %+v, want %q", gaps, want) + } + if err := ClearCrewPin(dir, crewroute.Checker); err != nil { + t.Fatal(err) + } + if gaps := CrewGapsAt(dir); len(gaps) != 0 { + t.Errorf("the checker unpinned again: gaps %+v", gaps) + } +} + +// A ZERO-CREDIT ACCOUNT, end to end: a paid call says payment → the next +// task is still routed on the paid route, which its first call probes → the +// seat's rescue is the free pool → with that pool at its limit and the chat +// model on the same account, nothing → the one action. A paid call answering +// clears it, and the rescue has no free pool to offer any more. +func TestAZeroCreditAccountWalksTheLadder(t *testing.T) { + dir := crewProfile(t) + rows := CrewCatalog() + rows = append(rows, catalog.Model{ID: "z-ai/glm-5.3-flash:free", OpenWeights: true, + IntelligenceIndex: 41.8, CodingIndex: 71.5, AgenticIndex: 50.9, ContextLength: 1310720, Parameters: []string{"tools"}}) + CrewCatalog = func() []catalog.Model { return rows } + var history []router.CrewRouteOutcome + withRouteHistory(t, &history) + ask := CrewAsk{Task: crewroute.Task{Text: fixTask}} + + normal, err := RouteCrew(dir, ask) + if err != nil { + t.Fatal(err) + } + w := normal.Seat(crewroute.Worker) + if w.Kind != crewroute.Metered { + t.Fatalf("with credit and free routes off the worker is %+v", w) + } + + history = append(history, router.CrewRouteOutcome{At: time.Now(), Seat: "worker", Send: w.Send, Provider: w.Provider, Kind: "payment"}) + probe, err := RouteCrew(dir, ask) + if err != nil { + t.Fatalf("an account out of credit refused the task at decision time: %v", err) + } + if got := probe.Seat(crewroute.Worker); got.Kind != crewroute.Metered { + t.Fatalf("the next task's worker is %+v, want the paid route, probed", got) + } + rescue := CrewRescue(dir, crewroute.Bugfix, crewroute.Worker, "somelab/the-chat-model") + if len(rescue) == 0 || rescue[0].Kind != crewroute.Free { + t.Fatalf("the rescue is %+v, want the free pool", rescue) + } + + history = append(history, router.CrewRouteOutcome{At: time.Now(), Seat: "worker", Send: rescue[0].Send, Provider: w.Provider, Kind: "quota"}) + for _, r := range CrewRescue(dir, crewroute.Bugfix, crewroute.Worker, "somelab/the-chat-model") { + if r.Send == rescue[0].Send || r.Model == "somelab/the-chat-model" { + t.Fatalf("the rescue offers %+v: a pool at its limit, or the chat model on the account out of credit", r) + } + } + if got := crewHealthAt(dir).crewAction(); got != "add credit on openrouter to continue" { + t.Errorf("the one action is %q", got) + } + + history = append(history, router.CrewRouteOutcome{At: time.Now(), Seat: "worker", Send: w.Send, Provider: w.Provider, Paid: true}) + for _, r := range CrewRescue(dir, crewroute.Bugfix, crewroute.Worker, "somelab/the-chat-model") { + if r.Kind == crewroute.Free { + t.Fatalf("with credit back the rescue still reaches for a free pool: %+v", r) + } + } +} + +// AN AUXILIARY CALL NEVER ASKS A ROUTE THAT WILL NOT ANSWER: a helper that +// would call a quarantined route, or a paid route on an account out of +// credit, is handed the router's healthy pick instead. +func TestAnAuxiliaryCallIsHandedAHealthyRoute(t *testing.T) { + dir := crewProfile(t) + history := []router.CrewRouteOutcome{{At: time.Now(), Send: "z-ai/glm-5.3-flash", Provider: "openrouter", Kind: "forbidden"}} + withRouteHistory(t, &history) + crewHealthCache.mu.Lock() + crewHealthCache.dir = "" + crewHealthCache.mu.Unlock() + if got := CrewHealthySend(dir, "z-ai/glm-5.3-flash", "somelab/chat"); got == "z-ai/glm-5.3-flash" || got == "" { + t.Errorf("a helper on the quarantined route is sent to %q", got) + } + if got := CrewHealthySend(dir, "moonshotai/kimi-k3", "somelab/chat"); got != "moonshotai/kimi-k3" { + t.Errorf("a healthy route was moved to %q", got) + } +} + +// A PICKER OFFERS WHAT A SEAT CAN RUN: no model without tool calls, and no +// model whose every route is quarantined. +func TestCrewOffersOnlyWhatASeatCanRun(t *testing.T) { + dir := crewProfile(t) + history := []router.CrewRouteOutcome{{At: time.Now(), Send: "moonshotai/kimi-k3", Provider: "openrouter", Kind: "forbidden"}} + withRouteHistory(t, &history) + for _, offer := range CrewOffersAt(dir) { + switch offer.Model.ID { + case "vendor/no-tools": + t.Error("a model that takes no tool calls was offered") + case "moonshotai/kimi-k3": + t.Error("a model on a quarantined route was offered") + } + } +} + +// A DIRECT CONNECTION IS A ROUTE BESIDE THE DEFAULT SERVICE: a model a +// connected provider serves itself is one candidate with both routes, and a +// payment failure on one account leaves the other. +func TestADirectConnectionIsASecondRouteToTheSameModel(t *testing.T) { + dir := crewProfile(t) + if err := writeProfileValue(dir, keyModelSources, []PersistedSource{{ID: "deepseek", Written: "deepseek", Key: "sk-deepseek-0123456789", Order: 1}}); err != nil { + t.Fatal(err) + } + var history []router.CrewRouteOutcome + withRouteHistory(t, &history) + var routes []crewroute.Route + for _, c := range CrewCandidatesAt(dir) { + if crewroute.Lineage(c.Model.ID) == "deepseek/deepseek-v4-flash" { + routes = c.Routes + } + } + if len(routes) != 2 || routes[0].Provider != "deepseek" || routes[1].Provider != "openrouter" { + t.Fatalf("deepseek-v4-flash routes: %+v, want the direct connection then the default service", routes) + } + history = append(history, router.CrewRouteOutcome{At: time.Now(), Send: routes[1].Send, Provider: "openrouter", Kind: "payment"}) + for _, c := range CrewCandidatesAt(dir) { + for _, r := range c.Routes { + if r.Provider == "openrouter" && r.Kind == crewroute.Metered { + t.Fatalf("a paid route on the account out of credit is still offered: %+v", r) + } + } + } +} + +// THE FREE SWITCH DECIDES WHETHER A FREE POOL IS A ROUTE AT ALL. Off (the +// default), no candidate, offer or routed seat carries a :free route and no +// :free id is offered; on, the pool is one more route of its paid model. +func TestTheFreeSwitchGovernsFreeRoutesEverywhere(t *testing.T) { + dir := crewProfile(t) + rows := CrewCatalog() + rows = append(rows, + catalog.Model{ID: "z-ai/glm-5.3-flash:free", OpenWeights: true, IntelligenceIndex: 41.8, CodingIndex: 71.5, AgenticIndex: 50.9, ContextLength: 1310720, Parameters: []string{"tools"}}, + catalog.Model{ID: "thinkingmachines/inkling-small:free", ContextLength: 262144, CodingIndex: 60, Parameters: []string{"tools"}}) + CrewCatalog = func() []catalog.Model { return rows } + var history []router.CrewRouteOutcome + withRouteHistory(t, &history) + freeRoutes := func() (ids []string, routes int) { + for _, c := range CrewCandidatesAt(dir) { + for _, r := range c.Routes { + if r.Kind == crewroute.Free { + routes++ + } + } + } + for _, o := range CrewOffersAt(dir) { + if crewroute.IsFree(o.Model.ID) { + ids = append(ids, o.Model.ID) + } + for _, r := range o.Routes { + if r.Kind == crewroute.Free { + routes++ + } + } + } + return ids, routes + } + if ids, routes := freeRoutes(); len(ids) > 0 || routes > 0 { + t.Fatalf("with free routes off: offered %v, %d free routes", ids, routes) + } + d, err := RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: openTask}}) + if err != nil { + t.Fatal(err) + } + for _, pick := range d.Crew { + if pick.Kind == crewroute.Free || strings.Contains(pick.Send, ":free") || strings.Contains(pick.Model, "inkling") { + t.Fatalf("with free routes off the %s seat is %+v", pick.Seat, pick) + } + } + if err := SetCrewFreeRoutes(dir, true); err != nil { + t.Fatal(err) + } + ids, routes := freeRoutes() + if len(ids) > 0 || routes == 0 { + t.Fatalf("with free routes on: offered ids %v (want none, a pool is a route), %d free routes (want some)", ids, routes) + } +} + +// A PICKER NEVER OFFERS A MODEL THAT CANNOT HOLD A CONVERSATION: speech, +// transcription and image models are no seat, whatever parameters they list. +func TestCrewOffersNoSpeechOrImageModels(t *testing.T) { + dir := crewProfile(t) + rows := CrewCatalog() + rows = append(rows, + catalog.Model{ID: "deepgram/aura-2", PromptPrice: 1e-9, CompletionPrice: 1e-9, ContextLength: 262144, InputModalities: []string{"text"}, OutputModalities: []string{"audio"}, Parameters: []string{"tools"}}, + catalog.Model{ID: "deepgram/nova-3", PromptPrice: 1e-9, CompletionPrice: 1e-9, ContextLength: 262144, InputModalities: []string{"audio"}, OutputModalities: []string{"text"}, Parameters: []string{"tools"}}, + catalog.Model{ID: "vendor/flux-image", PromptPrice: 1e-9, CompletionPrice: 1e-9, ContextLength: 262144, InputModalities: []string{"text"}, OutputModalities: []string{"image"}, Parameters: []string{"tools"}}) + CrewCatalog = func() []catalog.Model { return rows } + var history []router.CrewRouteOutcome + withRouteHistory(t, &history) + var deepseek bool + for _, offer := range CrewOffersAt(dir) { + switch offer.Model.ID { + case "deepgram/aura-2", "deepgram/nova-3", "vendor/flux-image": + t.Errorf("%s was offered for a seat", offer.Model.ID) + case "deepseek/deepseek-v4-flash": + deepseek = true + } + } + if !deepseek { + t.Error("a chat model with tools was not offered") + } +} + +// THE FREE RUNG RUNS WITH THE SWITCH OFF when every paid route is out of +// reach: a seat's rescue after a payment failure is a free pool, and the +// person's own model on the account that said no is not. +func TestTheRescueFallsToAFreePoolWhenNothingPaidIsReachable(t *testing.T) { + dir := crewProfile(t) + rows := CrewCatalog() + rows = append(rows, catalog.Model{ID: "z-ai/glm-5.3-flash:free", OpenWeights: true, + IntelligenceIndex: 41.8, CodingIndex: 71.5, AgenticIndex: 50.9, ContextLength: 1310720, Parameters: []string{"tools"}}) + CrewCatalog = func() []catalog.Model { return rows } + history := []router.CrewRouteOutcome{{At: time.Now(), Seat: "worker", Send: "z-ai/glm-5.3-flash", Provider: "openrouter", Kind: "payment"}} + withRouteHistory(t, &history) + if CrewFreeRoutesAt(dir) { + t.Fatal("the switch should be off") + } + rescue := CrewRescue(dir, crewroute.Bugfix, crewroute.Worker, "moonshotai/kimi-k3") + if len(rescue) == 0 || rescue[0].Kind != crewroute.Free { + t.Fatalf("rescue %+v, want a free pool first", rescue) + } + for _, r := range rescue { + if r.Model == "moonshotai/kimi-k3" { + t.Errorf("the chat model rode the account out of credit: %+v", r) + } + } +} + +// A FREE POOL CAN BE PINNED BY NAME: `:free` is a route, not a thinking level, +// and the pin runs on the pool. +func TestAFreePoolCanBePinnedByName(t *testing.T) { + pin, auto, err := ParseCrewPin("thinkingmachines/inkling-small:free@openrouter") + if err != nil || auto || pin.Model != "thinkingmachines/inkling-small:free" || pin.Provider != "openrouter" { + t.Fatalf("parse: %+v %v %v", pin, auto, err) + } + if _, _, err := ParseCrewPin("vendor/model:loud"); err == nil { + t.Error("an unknown suffix was taken") + } + dir := crewProfile(t) + got := resolveCrewPin(CrewPin{Model: "z-ai/glm-5.3-flash:free"}, CrewProvidersAt(dir)) + if got.Kind != crewroute.Free || got.Send != "z-ai/glm-5.3-flash:free" { + t.Errorf("a pinned free pool resolves to %+v", got) + } +} + +// A PIN SENDS THE ID THE PERSON WROTE when the catalog lists it, though a +// dated snapshot of the same model is listed first; an id the catalog does +// not list is sent as the variant it resolves to, and the line says which. +func TestAPinSendsExactlyTheIdWritten(t *testing.T) { + dir := crewProfile(t) + rows := append([]catalog.Model{{ID: "deepseek/deepseek-v4-flash-0731", OpenWeights: true, PromptPrice: 8e-8, CompletionPrice: 1.6e-7, + IntelligenceIndex: 24.2, CodingIndex: 56.2, AgenticIndex: 22.2, ContextLength: 1048576, Parameters: []string{"tools"}}}, CrewCatalog()...) + CrewCatalog = func() []catalog.Model { return rows } + var history []router.CrewRouteOutcome + withRouteHistory(t, &history) + if err := SetCrewPin(dir, crewroute.Checker, "deepseek/deepseek-v4-flash"); err != nil { + t.Fatal(err) + } + d, err := RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: fixTask}}) + if err != nil { + t.Fatal(err) + } + if got := d.Seat(crewroute.Checker); got.Send != "deepseek/deepseek-v4-flash" || got.Model != "deepseek/deepseek-v4-flash" { + t.Fatalf("the pinned checker is %+v, want exactly the id written", got) + } + if strings.Contains(d.Line("", -1), "→") { + t.Errorf("an exact pin reads as resolved: %q", d.Line("", -1)) + } + + if err := SetCrewPin(dir, crewroute.Checker, "z-ai/glm-5.3-flash-latest"); err != nil { + t.Fatal(err) + } + d, err = RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: fixTask}}) + if err != nil { + t.Fatal(err) + } + if got := d.Seat(crewroute.Checker); got.Send != "z-ai/glm-5.3-flash" { + t.Fatalf("a pin the catalog lists only as a variant is %+v", got) + } + if line := d.Line("", -1); !strings.Contains(line, "glm-5.3-flash-latest → glm-5.3-flash") { + t.Errorf("the resolved pin is not said: %q", line) + } + + for _, c := range CrewCandidatesAt(dir) { + if crewroute.Lineage(c.Model.ID) == "deepseek/deepseek-v4-flash" && c.Model.ID != "deepseek/deepseek-v4-flash" { + t.Errorf("the candidate is read from the snapshot %s, not the model's own row", c.Model.ID) + } + } +} + +// A MODEL IS PRICED FROM ITS OWN ROW, never from a route spelling of it the +// catalog listed first (`:floor` is the cheapest provider's price, not the +// model's list price). +func TestAModelIsPricedFromItsOwnRow(t *testing.T) { + dir := crewProfile(t) + rows := append([]catalog.Model{{ID: "moonshotai/kimi-k3:floor", OpenWeights: true, PromptPrice: 8.8e-7, CompletionPrice: 1.053e-5, + IntelligenceIndex: 43.6, CodingIndex: 76.2, AgenticIndex: 50, ContextLength: 1048576, Parameters: []string{"tools"}}}, CrewCatalog()...) + CrewCatalog = func() []catalog.Model { return rows } + var history []router.CrewRouteOutcome + withRouteHistory(t, &history) + for _, offer := range CrewOffersAt(dir) { + if crewroute.Lineage(offer.Model.ID) == "moonshotai/kimi-k3" && (offer.Model.ID != "moonshotai/kimi-k3" || offer.Model.PromptPrice != 3e-6) { + t.Errorf("kimi-k3 is offered as %s at %v/token in", offer.Model.ID, offer.Model.PromptPrice) + } + } +} + +// A DECISION ROW EXPLAINS ITSELF: only candidates that can sit a seat are +// named, and each seat carries its best three with route, quality, cost and +// score. +func TestADecisionRowNamesOnlySeatableCandidatesAndItsTopThree(t *testing.T) { + dir := crewProfile(t) + var history []router.CrewRouteOutcome + withRouteHistory(t, &history) + for _, name := range CrewCandidateNames(dir) { + if name == "vendor/no-tools" { + t.Errorf("a model that cannot sit a seat is named a candidate") + } + } + d, err := RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: fixTask}}) + if err != nil { + t.Fatal(err) + } + top := crewTop(dir, d) + worker := top["worker"] + if len(worker) == 0 || len(worker) > 3 || worker[0].Model != d.Seat(crewroute.Worker).Model || worker[0].Route == "" { + t.Errorf("the worker's top is %+v, want the pick first with its route", worker) + } +} + +// A RESCUE TAKES A GENERAL OR A CODE MODEL BEFORE ONE TUNED FOR A DOMAIN, and +// says free routes once. +func TestTheFreeRescuePrefersGeneralModels(t *testing.T) { + dir := crewProfile(t) + rows := CrewCatalog() + rows = append(rows, + catalog.Model{ID: "inclusionai/ling-3.0-flash-fin:free", ContextLength: 262144, IntelligenceIndex: 60, CodingIndex: 80, AgenticIndex: 60, Parameters: []string{"tools"}}, + catalog.Model{ID: "poolside/laguna-s-2.1:free", ContextLength: 262144, CodingIndex: 40, Parameters: []string{"tools"}}) + CrewCatalog = func() []catalog.Model { return rows } + history := []router.CrewRouteOutcome{{At: time.Now(), Seat: "worker", Send: "z-ai/glm-5.3-flash", Provider: "openrouter", Kind: "payment"}} + withRouteHistory(t, &history) + for _, seat := range crewroute.Seats { + rescue := CrewRescue(dir, crewroute.Bugfix, seat, "") + if len(rescue) == 0 || crewroute.DomainTuned(rescue[0].Model) { + t.Errorf("the %s rescue is %+v, want a general model first", seat, rescue) + } + } + _, notice := crewCandidatesNoticed(dir, crewHealthAt(dir)) + if strings.Count(notice, "free routes") != 1 { + t.Errorf("the notice says free routes other than once: %q", notice) + } +} + +// A SEAT THAT CANNOT START MOVES SIDEWAYS BEFORE IT MOVES UP: with the cheap +// worker's route quarantined, the next rung is a qualified model at a similar +// cost, not the dearest measured one — even when the model's own catalog row +// left its indexes to a dated snapshot's row. +func TestTheNextRungIsAtASimilarCost(t *testing.T) { + dir := crewProfile(t) + var rows []catalog.Model + for _, r := range CrewCatalog() { + if r.ID == "deepseek/deepseek-v4-flash" { + bare := r + bare.IntelligenceIndex, bare.CodingIndex, bare.AgenticIndex = 0, 0, 0 + rows = append(rows, catalog.Model{ID: "deepseek/deepseek-v4-flash-0731", OpenWeights: true, PromptPrice: r.PromptPrice, CompletionPrice: r.CompletionPrice, + IntelligenceIndex: r.IntelligenceIndex, CodingIndex: r.CodingIndex, AgenticIndex: r.AgenticIndex, ContextLength: r.ContextLength, Parameters: r.Parameters}, bare) + continue + } + rows = append(rows, r) + } + CrewCatalog = func() []catalog.Model { return rows } + var history []router.CrewRouteOutcome + withRouteHistory(t, &history) + d, err := RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: fixTask}}) + if err != nil { + t.Fatal(err) + } + for _, seat := range []crewroute.Seat{crewroute.Worker, crewroute.Planner} { + pick, ladder := d.Seat(seat), d.Ladder[seat] + if len(ladder) == 0 { + t.Fatalf("the %s has no ladder", seat) + } + if next := ladder[0]; next.CostUSD > 3*pick.CostUSD || crewroute.Lineage(next.Model) == "moonshotai/kimi-k3" { + t.Errorf("the %s's next rung from %s ($%.4f) is %s ($%.4f)", seat, pick.Model, pick.CostUSD, next.Model, next.CostUSD) + } + } +} + +// A CREW'S CALLS ARE HELD TO THE LOWER OF ITS OWN CAP AND THE DAY'S LIMIT, +// and an open-ended kimi checker to three times its estimate. +func TestTheSpendLinesASeatCallIsHeldTo(t *testing.T) { + dir := crewProfile(t) + if err := writeProfileValues(dir, map[string]any{KeyCrewCap: 1.0, KeyDailyBudget: 0.25}); err != nil { + t.Fatal(err) + } + if capUSD, action := CrewSpendCap(dir, true); capUSD != 0.25 || !strings.Contains(action, "/budget") { + t.Errorf("with the daily limit: %v %q", capUSD, action) + } + if capUSD, action := CrewSpendCap(dir, false); capUSD != 1.0 || !strings.Contains(action, "/crew cap") { + t.Errorf("without it: %v %q", capUSD, action) + } + var history []router.CrewRouteOutcome + withRouteHistory(t, &history) + if err := SetCrewPin(dir, crewroute.Checker, "moonshotai/kimi-k3"); err != nil { + t.Fatal(err) + } + // The planner on another model, so the checker's model is its own. + if err := SetCrewPin(dir, crewroute.Planner, "z-ai/glm-5.3-flash"); err != nil { + t.Fatal(err) + } + d, err := RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: openTask}}) + if err != nil && !errors.Is(err, ErrCrewAtCap) { + t.Fatal(err) + } + ceilings := CrewSeatCeilings(d) + if est, got := d.Seat(crewroute.Checker).EstUSD, ceilings[d.Seat(crewroute.Checker).Send]; est <= 0 || math.Abs(got-3*est) > 1e-9 { + t.Errorf("the open-ended kimi checker's ceiling is $%.3f (estimate $%.3f)", got, d.Seat(crewroute.Checker).EstUSD) + } + // One model in every seat: the spend the guard keeps is the model's, so + // the checker's line would stop the worker, and there is none. + one := crewroute.Decision{Crew: []crewroute.Pick{ + {Seat: crewroute.Worker, Model: "vendor/cheap", Send: "vendor/cheap"}, + {Seat: crewroute.Checker, Model: "vendor/cheap", Send: "vendor/cheap", EstUSD: 0.01}, + }} + if got := CrewSeatCeilings(one); got != nil { + t.Errorf("a checker sharing the worker's model has a ceiling: %v", got) + } +} + +// A MODEL PRICED AT NOTHING THAT IS NOT A FREE POOL IS NO CANDIDATE: a stealth +// preview listed at "0"/"0" scored as $0 and won a planner's rescue. Neither +// it nor a row with no published price is picked — in the crew or in the free +// rescue — while an explicit-zero `:free` pool still is. +func TestAnUnpricedModelIsNeverPicked(t *testing.T) { + dir := crewProfile(t) + rows := CrewCatalog() + rows = append(rows, + catalog.Model{ID: "stealth/space-bunny-alpha", ContextLength: 1048576, + IntelligenceIndex: 70, CodingIndex: 80, AgenticIndex: 70, Parameters: []string{"tools"}}, + catalog.Model{ID: "vendor/unpublished", PriceUnknown: true, ContextLength: 1048576, + IntelligenceIndex: 70, CodingIndex: 80, AgenticIndex: 70, Parameters: []string{"tools"}}, + catalog.Model{ID: "stealth/space-bunny-alpha-2:free", PriceUnknown: true, ContextLength: 1048576, + IntelligenceIndex: 70, CodingIndex: 80, AgenticIndex: 70, Parameters: []string{"tools"}}, + catalog.Model{ID: "z-ai/glm-5.3-flash:free", OpenWeights: true, ContextLength: 1310720, + IntelligenceIndex: 41.8, CodingIndex: 71.5, AgenticIndex: 50.9, Parameters: []string{"tools"}}) + CrewCatalog = func() []catalog.Model { return rows } + unpriced := func(id string) bool { + return strings.Contains(id, "space-bunny") || strings.Contains(id, "unpublished") + } + var history []router.CrewRouteOutcome + withRouteHistory(t, &history) + d, err := RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: fixTask}}) + if err != nil { + t.Fatal(err) + } + for _, pick := range d.Crew { + if unpriced(pick.Model) { + t.Errorf("the %s is %s", pick.Seat, pick.Model) + } + for _, rung := range d.Ladder[pick.Seat] { + if unpriced(rung.Model) { + t.Errorf("the %s's ladder holds %s", pick.Seat, rung.Model) + } + } + } + history = []router.CrewRouteOutcome{{At: time.Now(), Seat: "worker", Send: "z-ai/glm-5.3-flash", Provider: "openrouter", Kind: "payment"}} + for _, seat := range crewroute.Seats { + rescue := CrewRescue(dir, crewroute.Bugfix, seat, "") + if len(rescue) == 0 { + t.Fatalf("the %s has no free rescue", seat) + } + for _, r := range rescue { + if unpriced(r.Model) { + t.Errorf("the %s's rescue holds %s", seat, r.Model) + } + } + } +} + +// A COMPLEX FIX IS ROUTED AS ONE ON THE REAL ROAD: RouteCrew classifies first +// (for the learned offset) and hands the whole reading on, so the hermes +// report runs its worker a rung above a one-line fix's, and the log row says +// complex. ResolveSeats is the road codeaf do takes. +func TestRouteCrewRunsAComplexFixOnAStrongerWorker(t *testing.T) { + dir := crewProfile(t) + var history []router.CrewRouteOutcome + withRouteHistory(t, &history) + report, err := os.ReadFile("../crewroute/testdata/hermes-7680.md") + if err != nil { + t.Fatal(err) + } + simple, err := RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: "fix: typo in the loop bound of paginate() skips the last page"}}) + if err != nil { + t.Fatal(err) + } + hard, err := RouteCrew(dir, CrewAsk{Task: crewroute.Task{Text: string(report)}}) + if err != nil { + t.Fatal(err) + } + if simple.Subclass != "simple" || hard.Subclass != "complex" { + t.Fatalf("subclasses %q and %q", simple.Subclass, hard.Subclass) + } + if record := CrewRecordOf(hard, "repo", "t"); record.TaskSubclass != "complex" || record.TaskReach == "" { + t.Errorf("the log row carries %q / %q", record.TaskSubclass, record.TaskReach) + } + sw, hw := simple.Seat(crewroute.Worker), hard.Seat(crewroute.Worker) + if hw.Quality <= sw.Quality { + t.Errorf("complex worker %s (q %.2f), simple %s (q %.2f)", hw.Model, hw.Quality, sw.Model, sw.Quality) + } + seats, err := ResolveSeats(dir, SeatFlags{}, CrewAsk{Task: crewroute.Task{Text: string(report)}}) + if err != nil { + t.Fatal(err) + } + if seats.Crew == nil || seats.Crew.Subclass != "complex" || seats.Crew.Seat(crewroute.Worker).Model != hw.Model { + t.Errorf("codeaf do's seats: %+v", seats.Crew) + } +} diff --git a/internal/config/crewmigrate.go b/internal/config/crewmigrate.go new file mode 100644 index 0000000000..0c974738b2 --- /dev/null +++ b/internal/config/crewmigrate.go @@ -0,0 +1,116 @@ +package config + +import ( + "strings" + + "github.com/Agent-Field/codeaf/internal/crewroute" +) + +// A PROFILE WRITTEN BEFORE THE CREW WAS ROUTED, READ THE WAY ITS OWNER MEANT IT. +// +// Until 2026-09-24 a crew was a preset word — frugal, balanced, max — that +// wrote all five tier rows at once, a family row (`open` or `all`) saying +// which shelf the word drew from, a pick row (`table`, `catalog`, `learn`) +// saying where unwritten seats were computed from, and a tier row could say +// `auto` to be computed. None of those words means anything now: every seat +// nobody pinned is routed per task. So a profile carrying them is read like +// this, and migrated once, with one line saying so: +// +// - the preset word, the pick word and an `auto` row are read as AUTO; +// - the family row's `open` becomes the allowed-models rule `open`, because +// it said which models the person would accept, which is exactly what that +// rule says now; `all` is the default and needs no row; +// - a seat row holding ANY model id is A PIN — even one a retired preset +// once wrote, and even three rows holding the same id. There is no telling +// from a row which hand wrote it, and a pick never overrides a model id +// that may have been a person's: the rule the old picker kept, kept. +// +// The reading is applied on every read until the migration writes it down +// ([MigrateCrew]), so a build that has not yet migrated a profile still runs +// the crew its owner meant. + +// The retired rows, spelled here and nowhere else. +const ( + legacyKeyCrew = "models.crew" + legacyKeyCrewSource = "models.crew.source" + legacyKeyCrewPick = "models.crew.pick" +) + +// legacyWords are the retired preset and pick words a seat row may hold. +var legacyWords = map[string]bool{ + "frugal": true, "balanced": true, "max": true, "table": true, "catalog": true, "learn": true, CrewAuto: true, +} + +// crewSeatTiers are the three tier rows that hold crew pins. +var crewSeatTiers = []string{ModelTierWorker, ModelTierHigh, ModelTierMastermind} + +// legacyCrewClearing is the write that migrates a profile: every retired row +// removed, the family's `open` carried onto the allowed rule, and every seat +// row that is not a person's pin removed. It is empty for a profile with +// nothing retired on it, and any crew write folds it into its own one write. +func legacyCrewClearing(profileDir string) map[string]any { + values := map[string]any{} + for _, key := range []string{legacyKeyCrew, legacyKeyCrewPick} { + if _, held := persistedValue(profileDir, key); held { + values[key] = removeProfileKey + } + } + if source, held := persistedString(profileDir, legacyKeyCrewSource); held { + values[legacyKeyCrewSource] = removeProfileKey + if strings.EqualFold(strings.TrimSpace(source), "open") { + if _, answered := persistedValue(profileDir, KeyCrewAllowed); !answered { + values[KeyCrewAllowed] = "open" + } + } + } + for _, tier := range crewSeatTiers { + value, held := persistedString(profileDir, tierKeyFor(tier)) + if !held { + continue + } + word := strings.ToLower(strings.TrimSpace(value)) + if word == "" || legacyWords[word] { + values[tierKeyFor(tier)] = removeProfileKey + } + } + return values +} + +// MigrateCrew writes a retired crew's reading down, once, and answers the one +// line that says so — empty for a profile with nothing to migrate, which is +// every profile after the first run of this build. +// +// THE LINE IS SAID ONCE BECAUSE THE MIGRATION IS DONE ONCE: after it, the +// retired rows are gone and nothing is left to say it about. It names what +// stayed pinned, because a person whose crew changed shape under them is owed +// the half that did not. +func MigrateCrew(profileDir string) (string, error) { + values := legacyCrewClearing(profileDir) + if len(values) == 0 { + return "", nil + } + if err := writeProfileValues(profileDir, values); err != nil { + return "", err + } + // The seats in the order the manual and the panel list them. + var kept []string + pins := CrewPinsAt(profileDir) + for _, seat := range crewroute.Seats { + if pin, ok := pins[seat]; ok { + kept = append(kept, string(seat)+" "+pin.String()) + } + } + line := "your crew is auto now · codeaf picks the worker, planner and checker for each task" + if len(kept) > 0 { + // A PERSON WHOSE SEATS STAYED PINNED IS TOLD THAT FIRST: nothing they + // wrote was changed, and only the seats they left open are picked. + line = "kept your pins: " + strings.Join(kept, ", ") + if len(kept) < len(crewSeatTiers) { + line += " · the other seats are picked for each task" + } + } + if rule, held := values[KeyCrewAllowed]; held { + line += " · allowed models: " + rule.(string) + } + return line + " · " + CrewCommand + " to see it", nil +} diff --git a/internal/config/crewspend.go b/internal/config/crewspend.go new file mode 100644 index 0000000000..384729c4e6 --- /dev/null +++ b/internal/config/crewspend.go @@ -0,0 +1,106 @@ +package config + +import ( + "fmt" + "math" + + "github.com/Agent-Field/codeaf/internal/crewroute" +) + +// WHAT A SEAT CALL IS HELD TO BEFORE IT IS MADE (internal/session's +// spendguard.go does the holding): the model's prices, the day's cap, and each +// seat's own ceiling on one task. + +// CrewCallPrice is a model's prices per token — prompt, completion, cache +// read — as the catalog or the evidence table knows them. ok is false for a +// model neither prices, and for a free pool. +func CrewCallPrice(model string) (prompt, completion, cacheRead float64, ok bool) { + m, known := crewCatalogModel(model) + if !known || (m.PromptPrice <= 0 && m.CompletionPrice <= 0) { + return 0, 0, 0, false + } + return m.PromptPrice, m.CompletionPrice, m.CacheReadPrice, true +} + +// CrewSpendCap is the day's cap a crew's seat calls are held to, and the one +// sentence a call it stops ends on: the crew's own daily cap +// ([CrewCapAt]), or — where withDaily says the run is bound by it — the +// day's spending limit ([DailyBudgetUSDAt]), whichever is lower. Zero is no +// cap. +func CrewSpendCap(profileDir string, withDaily bool) (float64, string) { + capUSD, action := CrewCapAt(profileDir), "" + if capUSD > 0 { + action = "today's crew spend has reached the daily cap of " + crewroute.Money(capUSD) + " · raise it with /crew cap" + } + if withDaily { + if daily, err := DailyBudgetUSDAt(profileDir); err == nil && daily > 0 && (capUSD <= 0 || daily < capUSD) { + capUSD = daily + action = fmt.Sprintf("today's spending limit of $%.2f is reached · raise it with /budget", daily) + } + } + return capUSD, action +} + +// CrewTaskSpendCap is the per-task limit every priced call of one task is +// held to ([CrewTaskCapAt]) and the sentence a call it stops ends on. It is +// not lifted by --yes-spend: that flag answers the day's questions, not this. +func CrewTaskSpendCap(profileDir string) (float64, string) { + capUSD := CrewTaskCapAt(profileDir) + return capUSD, CrewTaskCapAction(capUSD) +} + +// CrewTaskCapAction is the sentence a call the per-task limit stops ends on. +func CrewTaskCapAction(capUSD float64) string { + return "this task reached its " + CrewTaskMoney(capUSD) + " limit · raise it in /crew" +} + +// CrewTaskMoney is a per-task limit in words: whole dollars without cents +// ($5), anything else to the cent ($2.50). +func CrewTaskMoney(usd float64) string { + if usd == math.Trunc(usd) { + return fmt.Sprintf("$%.0f", usd) + } + return fmt.Sprintf("$%.2f", usd) +} + +// CrewSeatCeilings is each seat's own spend ceiling on one task, keyed by the +// ids the seat is asked for: the CHECKER'S, a few times what it is expected to +// cost and never under a floor. A check is the seat whose length nothing else +// bounds — it reads until it is satisfied — and one that ran to eleven times +// its estimate on a dear model was the whole of a day's overshoot. A seat on a +// route that bills nothing has no ceiling. +func CrewSeatCeilings(d crewroute.Decision) map[string]float64 { + checker := d.Seat(crewroute.Checker) + if checker.EstUSD <= 0 { + return nil + } + ceiling := checker.EstUSD * crewCheckCeilingTimes + if ceiling < crewCheckCeilingFloor { + ceiling = crewCheckCeilingFloor + } + // A CHECKER ON A MODEL ANOTHER SEAT ALSO SITS has no ceiling: the guard + // keeps a model's spend, not a seat's, and a crew whose three seats are one + // model would stop its WORKER at the checker's line. + for _, pick := range d.Crew { + if pick.Seat != crewroute.Checker && (pick.Send == checker.Send || pick.Model == checker.Model) { + return nil + } + } + out := map[string]float64{} + for _, id := range []string{checker.Send, checker.Model} { + if id != "" { + out[id] = ceiling + } + } + return out +} + +// CrewCheckCeilingAction is the sentence a check its ceiling ends says, with +// the ceiling's dollars. +const CrewCheckCeilingAction = "the check stopped at its spend ceiling of $%.2f, three times its estimate, before it finished" + +// The checker's ceiling: how many times its estimate, and the least it is. +const ( + crewCheckCeilingTimes = 3.0 + crewCheckCeilingFloor = 0.05 +) diff --git a/internal/config/crewword_test.go b/internal/config/crewword_test.go deleted file mode 100644 index 1e5b2d6d1b..0000000000 --- a/internal/config/crewword_test.go +++ /dev/null @@ -1,155 +0,0 @@ -package config - -import ( - "testing" - - "github.com/Agent-Field/codeaf/internal/catalog" - "github.com/Agent-Field/codeaf/internal/crewpick" -) - -// A CREW WORD A RUN STORES IS THE BUDGET ITS SEATS RUN AT. -// -// The crew row is derived from the five tier rows and this build never stores -// it — but a run that writes ONE word into config.json instead of the five -// rows (a harness, a hand edit) is asking for a budget, and a word that sits on -// disk unread is a crew word that reached nobody. The seats nobody wrote then -// read that word's own table rows, under whatever family the profile names. -func TestAStoredCrewWordSeatsItsPreset(t *testing.T) { - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - - // Frugal and max are the two the default family tells apart from balanced - // in a seat a task pays for: frugal's careful seat and max's worker are the - // cells that move. Balanced is the preset a profile reads anyway, so it - // proves nothing here. - for _, preset := range []string{CrewFrugal, CrewMax} { - row, ok := CrewModelsForSource(DefaultCrewSource, preset) - if !ok { - t.Fatalf("there is no %s preset in the %s family", preset, DefaultCrewSource) - } - dir := writeProfileRows(t, map[string]string{KeyCrew: preset}) - for _, tier := range []string{ModelTierWorker, ModelTierHigh} { - seat := TierSeatAt(dir, tier) - if seat.Model != row[tier] { - t.Errorf("crew=%s: the %s seat reads %q, want the preset's own %q", - preset, tier, seat.Model, row[tier]) - } - } - // And the word is the word: the crew reads as the preset the run asked - // for, not the default it fell to. - if got := CrewAt(dir); got != preset { - t.Errorf("crew=%s: the crew word reads %q", preset, got) - } - } - - // A word this build does not know is no word at all, and an untouched - // profile is unchanged — the seam fires only on a preset on disk. - unknown := writeProfileRows(t, map[string]string{KeyCrew: "knee"}) - if got := TierSeatAt(unknown, ModelTierWorker).Model; got != DefaultWorkerModel { - t.Errorf("an unknown crew word seated %q, want the build's own default %q", got, DefaultWorkerModel) - } - untouched := t.TempDir() - if got := TierSeatAt(untouched, ModelTierWorker); got.Source != SeatDefault || got.Crew != "" { - t.Errorf("an untouched profile reads %q (%s), want the default rung as before", got.Model, got.Rung()) - } -} - -// A PINNED ROW SURVIVES THE LEARNED PICK. -// -// The knee of a Pareto arm pins the crew table's OWN ids — `glm-5.3-flash` for -// the worker, `fable-5.1` for the careful seat — and the pick used to answer -// them anyway, on the value-comparison rule that a row holding the preset's own -// table value is the preset answering rather than a pin. A hand pin writes ONE -// row, where a `/crew` APPLY writes all five, so a partial write is the person's -// own rows and the pick leaves them alone — with or without a stored crew word. -func TestAPinnedRowSurvivesTheLearnedPick(t *testing.T) { - restore := AutoModels - AutoModels = func() []catalog.Model { return autoTestRows() } - defer func() { AutoModels = restore }() - - balanced, ok := CrewModelsForSource(DefaultCrewSource, CrewBalanced) - if !ok { - t.Fatalf("there is no %s preset in the %s family", CrewBalanced, DefaultCrewSource) - } - pins := map[string]string{ - tierKeyFor(ModelTierWorker): balanced[ModelTierWorker], - tierKeyFor(ModelTierHigh): balanced[ModelTierHigh], - } - // Both the Pareto arm's shape (a stored crew word beside its pins) and the - // bare pin the brief writes are pins: the stored word is not what makes them - // win, but it must not cost them their rung either. - for _, tc := range []struct { - name string - rows map[string]string - }{ - {"a stored word beside the pins", map[string]string{KeyCrew: CrewBalanced}}, - {"the pins alone", map[string]string{}}, - } { - rows := map[string]string{KeyCrewPick: CrewPickLearn} - for k, v := range tc.rows { - rows[k] = v - } - for k, v := range pins { - rows[k] = v - } - dir := writeProfileRows(t, rows) - for _, tier := range []string{ModelTierWorker, ModelTierHigh} { - seat := TierSeatAt(dir, tier) - if seat.Model != pins[tierKeyFor(tier)] { - t.Errorf("%s: a pinned %s under %s reads %q, want the pin %q", - tc.name, tier, CrewPickLearn, seat.Model, pins[tierKeyFor(tier)]) - } - } - } - - // THE CONTROL: a seat nobody wrote is still computed — the pick answers for - // the seats the person did not name. - dir := writeProfileRows(t, map[string]string{ - KeyCrewPick: CrewPickLearn, - tierKeyFor(ModelTierWorker): balanced[ModelTierWorker], - }) - mind := TierSeatAt(dir, ModelTierMastermind) - if mind.Source != SeatLearned { - t.Errorf("an unwritten seat under %s reads %q (%s), want a learned seat", - CrewPickLearn, mind.Model, mind.Rung()) - } -} - -// A CREW APPLIED WHOLE STILL COMPUTES ITS DIAL SEATS UNDER A PICK. -// -// `/crew balanced` writes all five tier rows at once, at the preset's own ids, -// and the pick is meant to compute those three dial seats at the crew's budget. -// Only a PARTIAL write is a person's pin, so a whole crew keeps the old rule: a -// row holding the preset's own table value is the preset answering. -func TestAWholeCrewApplyStillComputesUnderTheLearnedPick(t *testing.T) { - restore := AutoModels - AutoModels = func() []catalog.Model { return autoTestRows() } - defer func() { AutoModels = restore }() - - dir := t.TempDir() - if err := ApplyCrew(dir, CrewBalanced); err != nil { - t.Fatal(err) - } - if err := SetCrewPick(dir, CrewPickLearn); err != nil { - t.Fatal(err) - } - if !allTiersWritten(dir) { - t.Fatal("a crew apply did not write all five tier rows") - } - - _, front, _ := crewpick.Presets(crewpick.Front(autoCandidates(autoTestRows()), crewpick.DefaultShapes(), crewpick.All)) - for _, tc := range []struct { - tier string - want string - }{ - {ModelTierWorker, front.Worker}, - {ModelTierHigh, front.High}, - {ModelTierMastermind, front.Mastermind}, - } { - seat := TierSeatAt(dir, tc.tier) - if seat.Model != tc.want || seat.Source != SeatLearned { - t.Errorf("the applied crew's %s under %s reads %q (%s), want the computed %q (learned)", - tc.tier, CrewPickLearn, seat.Model, seat.Source, tc.want) - } - } -} diff --git a/internal/config/custom_test.go b/internal/config/custom_test.go index e4569b0c54..5ebffcada8 100644 --- a/internal/config/custom_test.go +++ b/internal/config/custom_test.go @@ -244,7 +244,9 @@ func TestRenameConnectionWithAnInvalidTierValueLeavesEveryRowUnchanged(t *testin if got := TierModelAt(dir, ModelTierLow); got != "homelab/b:low" { t.Fatalf("the low tier moved before the failure: %q", got) } - if got := TierModelAt(dir, ModelTierHigh); got != "homelab/a:mid" { + // The checker row is a crew seat, and a value the tier gate refuses is not + // a pin it runs — so the row is read as it stands on disk. + if got, _ := persistedString(dir, KeyTierHighModel); got != "homelab/a:mid" { t.Fatalf("the high tier did not keep the value the rename refused: %q", got) } if got := ModelFallbacksAt(dir); got != "homelab/a, openai/x" { diff --git a/internal/config/derivation_test.go b/internal/config/derivation_test.go index c0728fe6cb..69b3253a54 100644 --- a/internal/config/derivation_test.go +++ b/internal/config/derivation_test.go @@ -139,10 +139,6 @@ var settingReaders = map[string]string{ // the word into poolcfg beside the mode, and the puller's keys resolve // from there (cmd/codeaf's poolTrustedKeys). KeyModelPoolPublicKey: "ModelPoolAt", - // The crew's pick row names its resolver: the surfaces read the pick to - // say it beside the crew word, and the ladder turns it into computed seats - // ([pickedSeat]) — the identifier proves the word reaches a surface. - KeyCrewPick: "CrewPickAt", // The prompt-profile row names the session field the word lands in, which // is what the far side actually reads: cmd/codeaf's applyV3Governance puts // PromptProfileAt's answer there and internal/session's promptprofile.go @@ -268,17 +264,6 @@ var settingReaders = map[string]string{ // low row prove this one. cmd/codeaf's crew source and internal/tui3's // settings skin both reach it by this name. KeyTierMastermindModel: "TierMastermind", - // The crew row is the four tier rows answered as one word, and what reads it - // is /crew (internal/tui3's crew.go) — the row's own reader, named here - // rather than the writer, because a row that could be set and never read - // would be exactly the dial-wired-to-nothing this table exists to catch. - KeyCrew: "CrewAt", - // The source row's own reader is CrewSourceAt, and the picker reaches it by - // that name (internal/tui3's crew.go), so the row names its reader rather - // than the crew derivation one hop away. The crew's writer resolves through - // the same reader, so the row and the five ids it produces cannot disagree - // about which family the preset words mean. - KeyCrewSource: "CrewSourceAt", // The reflex row names the ROLE that reaches it rather than TierKey, which // its two neighbours share: internal/reflex resolves [roles.RoleReflex] and // nothing else lands on this key, so naming the shared spelling a third diff --git a/internal/config/firstrun.go b/internal/config/firstrun.go index bb96fa8cec..975fdbac85 100644 --- a/internal/config/firstrun.go +++ b/internal/config/firstrun.go @@ -60,23 +60,6 @@ func APIKeyConfigured(profileDir string) bool { return APIKeyAt(profileDir) != "" } -// CrewConfigured is whether a person has ever answered the crew: any of the -// tier rows, or the family row above them, is in the profile file. It is any -// rather than all because [ApplyCrew] writes the tier rows together, and a -// person who pinned one tier by hand, or chose a family, has an opinion the -// setup must not paper over with a preset. -func CrewConfigured(profileDir string) bool { - if _, ok := persistedString(profileDir, KeyCrewSource); ok { - return true - } - for _, tier := range ModelTiers { - if _, ok := persistedString(profileDir, tierKeyFor(tier)); ok { - return true - } - } - return false -} - // DailyBudgetConfigured is whether a person has answered the daily ceiling: the // environment pins it, or the profile file holds it. func DailyBudgetConfigured(profileDir string) bool { diff --git a/internal/config/firstrun_test.go b/internal/config/firstrun_test.go index c8b8c749ed..abf6f5b71c 100644 --- a/internal/config/firstrun_test.go +++ b/internal/config/firstrun_test.go @@ -16,25 +16,25 @@ func TestAFreshProfileIsMissingAllThreeAndAnAnsweredOneIsNot(t *testing.T) { t.Setenv("CODEAF_DAILY_BUDGET", "") dir := t.TempDir() - if APIKeyConfigured(dir) || CrewConfigured(dir) || DailyBudgetConfigured(dir) { - t.Fatal("a profile nobody has touched must read as unanswered on all three") + if APIKeyConfigured(dir) || DailyBudgetConfigured(dir) { + t.Fatal("a profile nobody has touched must read as unanswered on both") } - // The crew and the ceiling both RESOLVE on that profile — to the defaults — - // and that is exactly what must not count. - if CrewAt(dir) != DefaultCrew { - t.Fatalf("crew resolves to %q, want the default", CrewAt(dir)) + // The crew RESOLVES on that profile — every seat auto — and that is + // exactly what must not count. + if pins := CrewPinsAt(dir); len(pins) != 0 { + t.Fatalf("an untouched profile has pins %v", pins) } if err := WriteAPIKey(dir, " sk-or-v1-abc "); err != nil { t.Fatal(err) } - if err := ApplyCrew(dir, CrewMax); err != nil { + if err := SetCrewCap(dir, "5"); err != nil { t.Fatal(err) } if err := WriteDailyBudgetUSD(dir, 7); err != nil { t.Fatal(err) } - if !APIKeyConfigured(dir) || !CrewConfigured(dir) || !DailyBudgetConfigured(dir) { + if !APIKeyConfigured(dir) || !DailyBudgetConfigured(dir) { t.Fatal("every answered row must read as configured") } if got := PersistedAPIKey(dir); got != "sk-or-v1-abc" { @@ -149,25 +149,16 @@ func TestTheAPIKeyRowMasksReadsTheShellFirstAndWritesTheProfile(t *testing.T) { } } -// THE FAMILY IS AN OPINION TOO. A person who chose a family and never pinned a -// tier has still answered the crew, so the setup must not paper over it with a -// preset: the law the tier rows already carry, extended to the row above them. -// And a family nobody chose is the default, which is NOT an answer. -func TestTheFamilyRowCountsAsAnAnsweredCrew(t *testing.T) { +// THE ALLOWED RULE NOBODY WROTE IS `all`, and narrowing it is read back. +func TestTheAllowedRuleDefaultsToAllAndReadsBack(t *testing.T) { dir := t.TempDir() - if CrewConfigured(dir) { - t.Fatal("a profile nobody has touched must read as unanswered") - } - if got := CrewSourceAt(dir); got != DefaultCrewSource { - t.Fatalf("an untouched profile reads the family as %q, want the default", got) + if got := CrewAllowedAt(dir).String(); got != "all" { + t.Fatalf("an untouched profile allows %q, want all", got) } - if err := SetCrewSource(dir, CrewSourceAll); err != nil { + if err := SetCrewAllowed(dir, "open"); err != nil { t.Fatal(err) } - if !CrewConfigured(dir) { - t.Fatal("a profile whose family was chosen must read as answered") - } - if got := CrewSourceAt(dir); got != CrewSourceAll { - t.Fatalf("the family row reads %q after it was written", got) + if got := CrewAllowedAt(dir).String(); got != "open" { + t.Fatalf("a narrowed rule reads back %q", got) } } diff --git a/internal/config/freecrew.go b/internal/config/freecrew.go index 439eee77e4..7d18f08604 100644 --- a/internal/config/freecrew.go +++ b/internal/config/freecrew.go @@ -3,27 +3,26 @@ package config // FreeChatModel is the build's chat bottom rung while a known account is low. const FreeChatModel = "qwen/qwen3.8-27b:free" -// CrewFree is a derived reading of the five free seats, never a saved preset. -const CrewFree = "free" - -// The free table is hand-picked because the ordinary crew picker rejects zero -// tariffs; each seat must remain usable without an account top-up. -var freeCrewModels = map[string]string{ - ModelTierReflex: "nvidia/nemotron-3.5-lightning:free", - ModelTierLow: "thinkingmachines/inkling-small:free", - ModelTierWorker: FreeChatModel, - ModelTierHigh: "thinkingmachines/inkling:free", - ModelTierMastermind: FreeChatModel, +// freeHelperModels are the free models the two rows that are not crew seats — +// reflex and small work — read while a known account is low. The table is +// hand-picked because an ordinary pick rejects zero tariffs; each row must +// stay usable without an account top-up. +// +// The three crew seats are not here: they are routed per task, and a known-low +// account reaches the router as an OpenRouter account out of credit +// ([crewHealthAt]), which moves the seats onto free routes with the router's +// own notice. +var freeHelperModels = map[string]string{ + ModelTierReflex: "nvidia/nemotron-3.5-lightning:free", + ModelTierLow: "thinkingmachines/inkling-small:free", } -// freeTierModelAt is the only low-credit check beneath an unwritten table seat. -func freeTierModelAt(profileDir, family, tier string) string { - if useFreeDefaultsAt(profileDir) && CrewPickAt(profileDir) == CrewPickTable { - if _, stored := storedCrewWord(profileDir); !stored { - if model := freeCrewModels[tier]; model != "" { - return model - } +// freeTierModelAt is the only low-credit check beneath an unwritten helper row. +func freeTierModelAt(profileDir, tier string) string { + if useFreeDefaultsAt(profileDir) { + if model := freeHelperModels[tier]; model != "" { + return model } } - return defaultTierModel(family, tier) + return builtinTierModel(tier) } diff --git a/internal/config/projectconfig_test.go b/internal/config/projectconfig_test.go index a82bfc5cf0..298ba974e0 100644 --- a/internal/config/projectconfig_test.go +++ b/internal/config/projectconfig_test.go @@ -93,12 +93,14 @@ func TestEveryProjectRowResolvesProjectOverProfileOverDefault(t *testing.T) { }{ {KeyToolApprovalMode, "deny", "allow", DefaultToolApprovalMode}, {KeyToolApprovals, "bash:deny", "bash:allow", ""}, - // The two tier rows ARRIVE WITH A MODEL IN THEM since the crew landed - // (crew.go), so their built-in reading is this build's own choice for - // that class of work rather than nothing. The ladder is unchanged: the - // repository's answer, then the person's, then the shipped one. + // The small-work row ARRIVES WITH A MODEL IN IT, so its built-in + // reading is this build's own near-free model. The checker row is a + // crew seat and has no shipped model: unpinned, it is routed, and a + // profile with no provider connected has nothing to route to — empty, + // which follows the conversation. The ladder is unchanged: the + // repository's answer, then the person's, then the built-in one. {KeyTierLowModel, "project/cheap", "profile/cheap", DefaultLowModel}, - {KeyTierHighModel, "project/capable", "profile/capable", DefaultHighModel}, + {KeyTierHighModel, "project/capable", "profile/capable", ""}, {KeyModelRoles, "title:project/title", "title:profile/title", ""}, } { got, err := ProjectStringAt(project, profile, row.key) diff --git a/internal/config/seats.go b/internal/config/seats.go index 315fc4cd26..635cf9a263 100644 --- a/internal/config/seats.go +++ b/internal/config/seats.go @@ -1,311 +1,82 @@ package config import ( + "errors" "strings" + "github.com/Agent-Field/codeaf/internal/crewroute" "github.com/Agent-Field/codeaf/internal/env" ) -// THE TWO SEATS EVERY DOOR SITS SOMEBODY IN, RESOLVED ONE WAY. +// THE THREE SEATS EVERY DOOR SITS SOMEBODY IN, RESOLVED ONE WAY. // -// A run needs two models named before it can start: the one that plans and the -// one that works. The chat surface has always asked the profile for them — a -// role rides a tier, a tier is a crew row, and /crew writes all four -// ([ApplyCrew], internal/roles) — while every headless door resolved its own -// pair from a flag and an environment variable and never opened the profile at -// all. So "my crew is frugal", said in the product's own vocabulary, held in the -// conversation and was silently lost the moment the same brain ran under `do` -// (#166). The benchmark that found it had to hand-translate a preset into two -// slugs, which is the one-source-of-truth violation the crew exists to prevent. +// A run needs its crew named before it can start: the worker that does the +// work, the planner that structures it, the checker that reads the result. +// Every door — `codeaf do`, `plan run`, `plan new`, `exec`, a saved program — +// resolves them here, through one ladder, so a pin said in the product's own +// vocabulary holds on every surface (#166 was the day it did not). // -// [ResolveSeats] is the whole ladder, in one place, for every door: the flag, the -// environment, the crew, the shipped default. It lives beside [CrewAt] rather -// than in the commands because a fifth door must inherit it by calling it, not -// by remembering to re-derive it — a rung copied into four files is a rung that -// will differ in one of them. +// THE LADDER, ONCE PER SEAT: // -// IT ALSO REPORTS WHICH RUNG ANSWERED. That is not decoration: the whole defect -// was invisible from outside, and a run whose receipt names the rung can be -// checked by the script that ran it. [Config.Load] cannot answer that question — -// it folds the environment and the default together into one string — which is -// why the ladder reads the environment itself rather than reading it back off a -// loaded config. +// 1. the flag, which is what this invocation said — a ONE-TASK PIN; +// 2. the environment, which is what this campaign said — a pin too; +// 3. a one-task `--pin`, then the profile's pin (/crew pin); +// 4. THE ROUTER, for every seat nothing above named: the task is classified +// and the seat picked for it (internal/crewroute, [RouteCrew]). +// +// THERE IS NO FIFTH RUNG. A seat nothing pinned is routed; it never falls +// back to a model this build chose for everybody, and the check seat never +// quietly inherits the planner's model because only the planner was named — +// a person who pinned the planner said something about planning. +// +// IT ALSO REPORTS WHICH RUNG ANSWERED, because the defect this file was +// written for was invisible from outside, and a run whose receipt names the +// rung can be checked by the script that ran it. -// SeatRole is which of the two seats one answer fills. It is on the [Seat] so a -// seat can name its own flag and its own variable rather than being handed them. +// SeatRole is which of the three seats one answer fills, by the name a person +// reads. It is on the [Seat] so a seat can name its own flag and variable. type SeatRole string const ( - // SeatWork is the model that does the work. - SeatWork SeatRole = "work" - // SeatPlan is the model that structures it — plans, replans, contracts, the - // delivery gate. Empty is legal here and means "the work model plans too"; - // see [Config.PlanModelResolved]. - SeatPlan SeatRole = "plan" - // SeatCheck is the model that checks finished work. It has no flagless - // rung of its own here: an empty check seat is the crew's careful row, - // read by the run's crew factory rather than resolved here, because the - // check is the one seat a two flag run must not let a third model fill. - SeatCheck SeatRole = "check" + // SeatWork is the worker: the model that does the work. + SeatWork SeatRole = "worker" + // SeatPlan is the planner: plans, replans, contracts, the delivery gate. + SeatPlan SeatRole = "planner" + // SeatCheck is the checker: the model that reads finished work. + SeatCheck SeatRole = "checker" ) -// SeatSource is the rung that answered, and the five constants are the ladder in -// order. It is an enum rather than a sentence because a caller putting it in -// JSON is making a promise a script parses. +// crewSeatOf is the router's seat for a role. +func crewSeatOf(role SeatRole) crewroute.Seat { + switch role { + case SeatPlan: + return crewroute.Planner + case SeatCheck: + return crewroute.Checker + } + return crewroute.Worker +} + +// SeatSource is the rung that answered. It is an enum rather than a sentence +// because a caller putting it in JSON is making a promise a script parses. type SeatSource string const ( - // SeatFlag is `--model` or `--plan-model`: the most recent thing the person - // said, and it always wins. + // SeatFlag is `--model`, `--plan-model` or `--check-model`: the most recent + // thing the person said, and it always wins. SeatFlag SeatSource = "flag" - // SeatEnv is the environment: automation's override, set once for a campaign - // instead of threaded onto every invocation. + // SeatEnv is the environment: automation's override, set once for a + // campaign instead of threaded onto every invocation. SeatEnv SeatSource = "env" - // SeatCrew is the profile's own tier row — what /crew wrote. - SeatCrew SeatSource = "crew" - // SeatInherited is the profile's crew answering through an OLDER row than - // this seat's own, because the profile was written before this seat existed - // ([tierLineage]). It is still the crew answering — the model came from what - // the person chose — and it is a separate rung because a person is owed the - // difference between "the row you wrote" and "the row your row was split out - // of". - SeatInherited SeatSource = "inherited" - // SeatComputed is a tier row that says `auto` answering from the catalog: - // the seat's model computed off the published figures at read time - // ([AutoPick]). It is a rung of the crew's own, not the build's default — - // the person wrote the row — and it says so because a receipt that called it - // `crew` would hide the one fact a reader of the run is checking: the id was - // derived, not named. - SeatComputed SeatSource = "computed" - // SeatTable is a tier row that says `auto` answering from the family's - // table row ([autoRow], [pickedModel]) because nothing could be computed — - // no catalog, or no pick off it. The id is the preset's own, and the rung - // says which kind of answer it was. - SeatTable SeatSource = "table" - // SeatLearned is a seat computed at the crew's budget under the `learn` - // pick ([CrewPickAt]): the catalog's own figures, plus the Model Pool's - // measurements and the person's own judged runs carried as a quality - // prior. It is a rung of its own, beside [SeatComputed], because the one - // fact a reader of a run is checking — was this id derived, and from - // what — is a different answer under the two words: the catalog alone, - // or the catalog plus what runs measured. - SeatLearned SeatSource = "learned" - // SeatDefault is this build's choice, for a profile that has never said - // anything about models at all. + // SeatPinned is a pin: the profile's own (/crew pin) or a one-task `--pin`. + SeatPinned SeatSource = "pinned" + // SeatRouted is the router's pick for this task. + SeatRouted SeatSource = "routed" + // SeatDefault is this build's own model for a tier that is not a crew + // seat — reflex and small work — on a profile that never wrote the row. SeatDefault SeatSource = "default" ) -// tierLineage is THE TABLE THAT MAKES A NEW SEAT SAFE TO ADD, and every tier -// word has a row in it whether or not it has an ancestor. -// -// A seat added to the ladder after people have profiles is a seat their profiles -// have no key for, and the ladder would fall past the crew they chose to this -// build's default — which is what happened when the worker row landed (#302): -// four pinned rows, and all the work quietly running on a model nobody picked. -// So the rule is written once, here, rather than discovered again per seat: A -// TIER WITH NO KEY OF ITS OWN INHERITS FROM THE ROW IT WAS SPLIT OUT OF, AND -// SAYS THAT IT DID. A tier that is nobody's descendant declares that by naming -// no ancestor, explicitly, so the next tier cannot join the table by forgetting -// to. -// -// Words is what the settings sheet calls the row, because the one line a person -// reads about the substitution has to name a row they can go and find. -// -// AND THE READ NEVER REPAIRS THE PROFILE. Writing the inherited key back would -// end the substitution for good after one run, and it would also mean every -// `codeaf do` — several at once on the same profile, on a machine where the disk -// may be full — rewrites the file a person's crew lives in, to record a decision -// this build made rather than one they did. [ApplyCrew] already writes every -// row, so a person who answers the line ends it themselves, once, deliberately. -var tierLineage = map[string]struct { - // Inherits is the tier this one falls back to, or empty for a tier that has - // always existed and inherits from nothing. - Inherits string - // Words is this row's name on the settings sheet. - Words string -}{ - ModelTierReflex: {Words: "reflex"}, - ModelTierLow: {Words: "small work"}, - // The worker seat was split out of the small-work row in #278: before it, - // the work seat of every headless run and every node of an adaptive run - // resolved through `models.tiers.low`. That row is therefore the honest - // ancestor — a profile pinned before #278 meant its small-work model when it - // said what the work should run on. - ModelTierWorker: {Inherits: ModelTierLow, Words: "worker"}, - ModelTierHigh: {Words: "careful work"}, - ModelTierMastermind: {Words: "thinking"}, -} - -// tierWords is a row's name on the settings sheet, for the one line that has to -// send somebody to it. An unknown tier answers with its own word, which is what -// a table this file owns can promise about a word it does not. -func tierWords(tier string) string { - if row, ok := tierLineage[tier]; ok && row.Words != "" { - return row.Words - } - return tier -} - -// inheritedTier is the crew's answer for a tier the profile has no key for: the -// nearest ancestor row that was actually WRITTEN, and which row that was. -// -// It walks the lineage rather than taking one step, so a seat split out of a -// seat still reaches the row a profile older than both of them holds. The walk -// is bounded by the number of tiers, which makes a table with a cycle in it slow -// nothing down — [TestEveryTierDeclaresItsAncestry] is what refuses the cycle. -// -// AN UNWRITTEN ANCESTOR IS NOT AN ANSWER. Reading the ancestor through -// [TierModelAt] would hand back this build's default for that row and report it -// as the crew, which is the same lie one row over. A row cleared on purpose -// reads empty — "follow the conversation", which a headless run cannot — and -// falls through too. -func inheritedTier(profileDir, tier string) (model, from string, ok bool) { - for hops := 0; hops < len(tierLineage); hops++ { - row, known := tierLineage[tier] - if !known || row.Inherits == "" { - return "", "", false - } - tier = row.Inherits - if written, held := persistedString(profileDir, tierKeyFor(tier)); held { - if value := strings.TrimSpace(written); value != "" { - return value, tier, true - } - return "", "", false - } - } - return "", "", false -} - -// tierSeatRole is what one tier's seat is CALLED in the line a person reads -// about it. The two seats a run sits somebody in have names of their own -// ([SeatWork], [SeatPlan]) and both surfaces have to spell them the same way: a -// conversation whose work seat was inherited must read the sentence `codeaf do` -// prints, not a second wording for one fact. Every other row answers with its -// own name on the settings sheet, so a tier that joins the lineage tomorrow has -// a sentence before anybody writes one. -func tierSeatRole(tier string) SeatRole { - switch tier { - case ModelTierWorker: - return SeatWork - case ModelTierMastermind: - return SeatPlan - } - return SeatRole(tierWords(tier)) -} - -// crewRow is THE PROFILE HALF OF THE LADDER, and the one place that tells a row -// somebody WROTE from a row they CLEARED from a key that was NEVER HELD. -// -// The distinction is the whole mechanism and it is decided here so that the two -// ladders above it cannot decide it differently: [resolveSeat], which climbs a -// flag and a variable first for a run, and [TierSeatAt], which is what a -// conversation and the settings sheet read. A second copy of these three cases -// is how a person's crew comes to mean one thing headless and another in chat, -// which is the defect this file was written for one surface at a time (#166, -// #302, #312). -// -// source is empty when the profile has nothing this ladder can use, and -// `cleared` tells the two ways of having nothing apart: a row emptied ON PURPOSE -// says "follow the conversation", which a conversation can do and a headless run -// cannot, so each caller's own bottom rung answers for it. -func crewRow(profileDir, tier string) (model, from string, source SeatSource, cleared bool) { - if written, held := persistedString(profileDir, tierKeyFor(tier)); held { - if value := strings.TrimSpace(written); value != "" { - return value, "", SeatCrew, false - } - return "", "", "", true - } - // The key was never held, which on a profile older than this seat means the - // crew was chosen before the seat existed. It still answers, through the row - // this row was split out of ([tierLineage]). - if value, ancestor, ok := inheritedTier(profileDir, tier); ok { - return value, ancestor, SeatInherited, false - } - return "", "", "", false -} - -// TierSeatAt is ONE TIER ROW READ AS A SEAT, for the surfaces that seat roles -// rather than run a door: the conversation's role map (cmd/codeaf's v3Crew), the -// five rows of the settings sheet, and the crew word derived from them. -// -// IT IS [resolveSeat] WITHOUT THE TWO RUNGS THAT BELONG TO AN INVOCATION. A flag -// is something a command line said and a conversation has no command line for -// its crew; CODEAF_MODEL names the model a person TALKS TO ([Load] folds it into -// Config.Model), and a variable that also filled the work seat of every task -// handed off in that conversation would be one word quietly moving two dials. -// What is left is the profile — which is where a conversation's seats have -// always come from. -// -// Where it differs from a run is the BOTTOM, and only there: -// -// - a row the person WROTE is the crew answering; -// - a row they CLEARED reads empty, and stays empty, because on this surface -// that is an answer — "follow the conversation" — and refusing it would make -// a default into a rule ([TierModelAt] argues it at length); -// - a key NEVER HELD asks the lineage before the bottom rung, so a profile -// older than the worker seat hands a task the same model `codeaf do` hands -// it (#302 headless, #312 in the conversation), and the seat carries the fact -// so a surface can say it once ([Seat.Notice]); -// - anything else is this build's own choice for that class of work. -// -// THE PRESET WORD IS DELIBERATELY NOT READ HERE. [Seat.Crew] stays empty: -// [CrewAt] derives the preset from all five rows THROUGH this function, so a -// seat that filled it in would be the ladder asking the summary that is computed -// from the ladder — five extra file reads per row, and a cycle. A surface that -// wants both facts asks for both. -func TierSeatAt(profileDir, tier string) Seat { - return tierSeatUnder(profileDir, CrewSourceAt(profileDir), tier) -} - -// tierSeatUnder is [TierSeatAt] with the family already read, so a caller that -// walks every tier ([CrewAt]) reads the profile ONCE for the family instead of -// once per unwritten seat, and the five seats cannot be resolved under one -// family while the preset table is resolved under another. -func tierSeatUnder(profileDir, family, tier string) Seat { - model, from, source, cleared := crewRow(profileDir, tier) - // WHETHER THE ROW ITSELF WAS WRITTEN, read before the fallback below can - // stand in for it. A row a person typed is a pin the pick must not recompute - // ([pickedSeat]), and that is a fact about the row's presence, not about - // whether its id happens to match the preset it sits in. - written := source == SeatCrew - // THE BUDGET A ROW NOBODY WROTE RUNS AT. A build this one shapes writes no - // crew word, and an unwritten row reads the default preset's own id — but a - // run that STORED a word on the crew row ([storedCrewWord]) named a budget, - // and the seat reads that word's table row instead. The pick may still - // compute it when the pick is off the table. - var byWord string - if source == "" { - source = SeatDefault - if !cleared { - model = freeTierModelAt(profileDir, family, tier) - if seat, ok := unwrittenSeat(profileDir, family, tier); ok { - model, source, byWord = seat.Model, seat.Source, seat.Crew - } - } - } - // A row that says auto resolves through the one seam both ladders share - // ([autoRow]). Only a row somebody WROTE — directly, or through the lineage - // — can say it: the default rung names a model id, and a cleared row names - // nothing at all. - if IsAuto(model) { - var preset string - model, source, preset = autoRow(profileDir, family, tier) - return Seat{Role: tierSeatRole(tier), Model: model, Source: source, Crew: preset} - } - // AND THE PICK ROW ANSWERS FOR THE SEATS NOBODY NAMED, before the row's - // own rung is read: a pick of catalog or learn computes the dial seats at - // the crew's preset ([pickedSeat]), and leaves everything it does not - // answer — a hand-typed id, a cleared row, the two seats that always read - // the table — exactly as it was. A cleared row is a deliberate answer - // ("follow the conversation") and no pick unsays it on this surface. - if !cleared { - if seat, ok := pickedSeat(profileDir, family, tier, model, written); ok { - return seat - } - } - return Seat{Role: tierSeatRole(tier), Model: model, Source: source, From: from, Crew: byWord} -} - // ModelEnv, PlanModelEnv, and CheckModelEnv are the variables the seats read. // They are spelled here once because the ladder, [Load], and receipts name them. const ( @@ -314,7 +85,8 @@ const ( CheckModelEnv = "CODEAF_CHECK_MODEL" ) -// Seat is one seat's answer: the model, and where it came from. +// Seat is one seat's answer: the model, where it came from, and the provider +// it runs on when the router or a pin named one. type Seat struct { // envSpelling is the variable that answered when Source is [SeatEnv] — // the current spelling or the former one — recorded at resolution so the @@ -323,21 +95,7 @@ type Seat struct { Role SeatRole Model string Source SeatSource - // From is the tier word this seat's model was INHERITED from, and is empty - // unless Source is [SeatInherited]. It is carried rather than re-derived - // because the line that tells a person what happened has to name the row it - // read, and a second walk of the lineage at print time could name a - // different one. - From string - // Crew is the preset word the profile's five tier rows make — `frugal`, - // `balanced`, `max` or `custom` ([CrewAt]) — and is empty unless Source is - // [SeatCrew], [SeatInherited], [SeatComputed] or [SeatTable]. It is read - // rather than stored, exactly as the settings sheet reads it, so a receipt - // and the sheet cannot disagree about which crew ran. On the computed and - // table rungs it is the preset the auto row is computed at, read from the - // stored rows ([crewPresetUnder]) rather than derived through the resolver, - // which would be the seam asking itself. - Crew string + Provider string } // Flag is the flag that fills this seat. @@ -353,16 +111,19 @@ func (s Seat) Flag() string { // Env is the variable that fills this seat. func (s Seat) Env() string { - if s.Role == SeatPlan { + switch s.Role { + case SeatPlan: return PlanModelEnv + case SeatCheck: + return CheckModelEnv } return ModelEnv } // Rung is the answering rung as a receipt says it: the flag or the variable by -// name, the crew by preset, and otherwise the one word `default`. Naming the -// flag and the variable rather than saying "flag" and "env" costs nothing and -// tells a reader which of the two they would have to change. +// name, and otherwise the rung's own word. Naming the flag and the variable +// rather than saying "flag" and "env" costs nothing and tells a reader which +// of the two they would have to change. func (s Seat) Rung() string { switch s.Source { case SeatFlag: @@ -372,347 +133,182 @@ func (s Seat) Rung() string { return s.envSpelling } return s.Env() - case SeatCrew: - return s.crewRung() - case SeatInherited: - // The crew answered, through an older row. Both halves are said, - // because a receipt that said only `crew custom` would hide exactly the - // substitution this rung exists to report. - return s.crewRung() + ", inherited" - case SeatComputed: - return s.crewRung() + ", computed from the catalog" - case SeatLearned: - return s.crewRung() + ", learned" - case SeatTable: - return s.crewRung() + ", table" - } - return "default" -} - -// crewRung is the crew half of a rung: the preset the profile's rows make, or -// the bare word for a profile whose preset cannot be read. -func (s Seat) crewRung() string { - if strings.TrimSpace(s.Crew) == "" { - return "crew" - } - return "crew " + s.Crew -} - -// Notice is the ONE line a person reads when this seat was not filled by a row -// they wrote, and it is empty for every other seat. -// -// IT IS SAID ONCE, WHERE THE SEATS ARE REPORTED, and never at the point of a -// call: this seat answers every request a run makes, and a line that arrived -// with all of them would be noise a person learns to read past — which leaves -// them exactly where the silence did. [Report] is how a door prints it, so the -// door does not have to remember. -// -// It is the register the surface already speaks in — an observation, a middle -// dot, a promise, lowercase, no full stop, nothing about machinery -// (internal/session's taskEscalationNote and checkpointCeilingNote). The -// observation is the true thing: the profile is older than the seat. The promise -// is what is running instead and what ends it. -// -// AND THE PROMISE NAMES THE DOOR, because it used to end in one nobody could -// find. `until you pick a crew again` is a remedy with no address on it, and -// this line is printed on FOUR HEADLESS DOORS — `codeaf do`, `codeaf plan run`, -// `codeaf exec` and `codeaf run` — where there is no way at all to pick a crew: -// [CrewPreset] is written by `/crew` and by the settings sheet's Providers row, -// both of which are the conversation, and no flag and no terminal verb sets one. -// So a person reading this in a terminal was told to do something, given no way -// to do it, and left to discover on their own that the answer was a different -// surface. Cause plus what to do is the law on both surfaces; a cause plus a -// dead end is the defect. -// -// ONE FORM, TRUE FROM BOTH PLACES IT IS PRINTED. Naming `/crew` reads as the -// next keystroke in the conversation and as a destination from the terminal, and -// it is the same sentence in both — which is what keeps somebody who has seen -// one surface recognising the other. `seat` STAYS: it is this product's own -// noun for a row of the crew, taught under that name on the manual's own page -// and spoken by the settings sheet and the model picker, and the vocabulary law -// is about MACHINERY — the program's words for its own process — not about a -// domain noun the product teaches. -// -// AND THE PROMISE ATTRIBUTES THE MODEL RATHER THAN DESCRIBING IT. This read -// `it is running on your small work model`, printed directly under a models -// line naming `deepseek/deepseek-v4-pro` — so two consecutive lines called one -// model by its name and then called it small, and the developer who met them -// could not tell which model the lane was actually on. The two lines were never -// in disagreement about the FACT: whenever the source is [SeatInherited] the -// model on the line above IS the inherited one, always, because that is what -// inheriting means. What differed was the grammar. `small work` is the NAME OF -// A SEAT on the settings sheet, one of the five this product seats, and putting -// it in front of `model` turns a seat's name into an adjective about the model -// it holds. -// -// So the notice never characterises the model a second time — it says which -// SEAT lent it. `your small work seat's model` cannot be read as a claim about -// deepseek-v4-pro, and it answers the question the person actually has: this -// seat has no row of its own, so it is borrowing that one's. A description was -// always going to contradict the line above it, since the two are one model. -func (s Seat) Notice() string { - if s.Source != SeatInherited { - return "" + case "": + return string(SeatDefault) } - return "your crew was set before the " + string(s.Role) + " seat existed · " + - "it is running on your " + tierWords(s.From) + " seat's model until you pick a crew with " + - CrewCommand + " in the conversation" + return string(s.Source) } -// CrewCommand is the one door that ends the inheritance [Notice] describes. It -// is a constant so the sentence and the surface that answers to it cannot drift -// apart — the notice is printed by four commands that cannot reach it, and a -// remedy naming a door that has been renamed is worse than one naming none. -const CrewCommand = "/crew" - -// FromWords is the row this seat's model was inherited from, in the words the -// settings sheet calls that row by — empty unless the source is -// [SeatInherited]. It is exported for the surface that has its own sentence to -// build about the same fact ([Notice] is the sentence; this is the noun), so two -// surfaces cannot invent two names for one row. -func (s Seat) FromWords() string { - if s.Source != SeatInherited { - return "" - } - return tierWords(s.From) -} - -// Report is what a door prints: the seat, and the line that says a row was -// inherited when one was. -func (s Seat) Report() string { return withNotice(s.Line(), s.Notice()) } - // Describe is one seat in a receipt's voice: // -// work z-ai/glm-5.3-flash (crew frugal) -// plan follows the work model (default) +// worker z-ai/glm-5.3-flash (routed) +// checker moonshotai/kimi-k3 (pinned) // -// THE EMPTINESS LAW, as the crew row already keeps it: an unfilled plan seat is -// said in words rather than left as a gap somebody has to interpret. +// THE EMPTINESS LAW: a seat with no model says so in words rather than +// leaving a gap somebody has to interpret. func (s Seat) Describe() string { model := strings.TrimSpace(s.Model) if model == "" { - model = "follows the work model" + model = "no model" } return string(s.Role) + " " + model + " (" + s.Rung() + ")" } // modelsLabel is the word a door puts in front of the sentence. It is spelled -// once so the four doors cannot label the same fact differently. +// once so the doors cannot label the same fact differently. const modelsLabel = "models: " // Line is one seat labelled, for the door that seats only one — `exec`, which // executes and never plans. func (s Seat) Line() string { return modelsLabel + s.Describe() } -// Seats is both seats of one run, and the check seat beside them when a door -// resolved one. Work and Plan are the ladder's two answers -// ([ResolveSeats]); Check is the door's own answer for the review round -// ([CheckSeat]), and empty on a door that named nothing, which the run's crew -// factory reads as the profile's careful row. +// Report is what a door prints for one seat. +func (s Seat) Report() string { return s.Line() } + +// Seats is one run's crew as the door resolved it, and the router's decision +// beside it when a seat was routed. type Seats struct { Work Seat Plan Seat Check Seat + // Crew is the router's decision for this task — the class it read, every + // seat's pick and the estimate — nil only when no router was asked. + Crew *crewroute.Decision } -// Sentence is both seats, unlabelled, for a door whose opening lines have a -// label column of their own: +// Sentence is the crew, unlabelled: // -// work z-ai/glm-5.3-flash (crew frugal) · plan z-ai/glm-5.3-flash (crew frugal) +// worker z-ai/glm-5.3-flash (routed) · planner z-ai/glm-5.3-flash (routed) · checker moonshotai/kimi-k3 (pinned) // -// ONE SHAPE AND NOT TWO. It would read a little better to collapse a run whose -// seats came from the same crew into one clause, and it would mean a script -// parsing this has two grammars to know and a person comparing two runs has two -// shapes to compare. Every run says both seats and names the rung beside each. +// ONE SHAPE AND NOT TWO: every run says every seat and names the rung beside +// each, so a script parsing this has one grammar to know. func (s Seats) Sentence() string { - return s.Work.Describe() + " · " + s.Plan.Describe() + return s.Work.Describe() + " · " + s.Plan.Describe() + " · " + s.Check.Describe() } -// Line is the one line a headless run opens with: -// -// models: work z-ai/glm-5.3-flash (crew frugal) · plan z-ai/glm-5.3-flash (crew frugal) +// Line is the one line a headless run opens with. func (s Seats) Line() string { return modelsLabel + s.Sentence() } -// Notice is the inheritance line for whichever seat was filled by an older row, -// and empty when neither was. Both seats are asked and their lines joined, -// because the table decides which tiers have ancestors and this must not have to -// be edited when a second one does. -func (s Seats) Notice() string { - lines := make([]string, 0, 2) - for _, seat := range []Seat{s.Work, s.Plan} { - if notice := seat.Notice(); notice != "" { - lines = append(lines, notice) - } +// Report is what a door prints: the models line, and under it the crew's own +// line — the class the task was read as and the estimate — when it was routed. +func (s Seats) Report() string { + if s.Crew == nil { + return s.Line() } - return strings.Join(lines, "\n") + return s.Line() + "\ncrew: " + s.Crew.Line(PinMark, -1) } -// Report is what a door prints instead of [Line]: the models line, and beneath -// it the one line that says a seat was inherited. ONE PLACE, so a door added -// tomorrow cannot print the models and swallow the reason. -func (s Seats) Report() string { return withNotice(s.Line(), s.Notice()) } +// PinMark is the mark a pinned seat wears on a headless line. The chat +// surface draws its own, from its icon vocabulary. +const PinMark = "📌" -// withNotice puts a notice under a line, and is the whole of the composition so -// the two Report methods cannot disagree about it. -func withNotice(line, notice string) string { - if notice == "" { - return line - } - return line + "\n" + notice +// SeatFlags are the three seat flags a door took. +type SeatFlags struct { + Model string + PlanModel string + CheckModel string } -// ResolveSeats climbs the ladder once per seat. +// ResolveSeats climbs the ladder for all three seats and routes every seat +// nothing named, for the task in ask. // -// 1. the flag, which is what this invocation said; -// 2. the environment, which is what this campaign said; -// 3. the profile's crew — the planning seat takes the MASTERMIND tier and the -// work seat takes the WORKER tier, because that is where the two roles ride -// in chat: roles.DefaultAssignment puts RolePlanner on TierMastermind and -// RoleWorker on TierWorker, and a chat task's own worker resolves through -// the same row (internal/session's defaultTaskModel), so a headless run, an -// adaptive run and a task handed off in conversation all call the same -// model on the same profile; -// 4. the crew again, through an OLDER ROW, for a profile written before this -// seat existed — the worker row's ancestor is the small-work row it was -// split out of ([tierLineage]), and the seat says it was inherited rather -// than reporting the model as though the person had pinned it; -// 5. this build's default, which a profile that has said nothing about models -// at all is the only thing left for. +// A flag and a variable are handed to the router AS PINS FOR THIS TASK, so the +// decision it answers — the crew line, the estimate, the logged row — is the +// crew that actually runs, and a seat the flag filled still says `--model` +// on the receipt. A value is handed on WHOLE: a flag may read +// `moonshotai/kimi-k3:low`, and the level is split off where every level is, +// at the point of the call ([roles.SplitEffort]). // -// profileDir is the profile to ask, [ProfileDir] for an ordinary process. The -// environment is read HERE, at call time, and not carried in from [Load]: Load -// resolves CODEAF_MODEL and DefaultModel into one field and the difference -// between them is exactly what the receipt has to report. -// -// A tier value may carry a thinking level (`moonshotai/kimi-k3:low`) and is -// handed on WHOLE, exactly as a flag or a variable carrying one is. The level is -// applied where every other level is applied — the role ladder splits it into a -// model and an effort at the point of the call ([roles.SplitEffort], -// internal/session's roleRequest) — and the client seam takes the level off the -// slug it sends ([Config.providerConfig]). Neither of those is this function's -// business, and a rung that quietly shortened the value would be the second -// model policy this whole file exists to remove. -func ResolveSeats(profileDir, flagModel, flagPlanModel string) Seats { - return Seats{ - Work: resolveSeat(SeatWork, profileDir, flagModel, ModelTierWorker, ChatDefaultAt(profileDir)), - Plan: resolveSeat(SeatPlan, profileDir, flagPlanModel, ModelTierMastermind, ""), +// The error is the router's: a seat nothing allowed can sit, or +// [ErrCrewAtCap] — beside which the seats still come back, because what a +// door does at the cap (ask, refuse, or be told to spend) is the door's. +func ResolveSeats(profileDir string, flags SeatFlags, ask CrewAsk) (Seats, error) { + seats := Seats{Work: Seat{Role: SeatWork}, Plan: Seat{Role: SeatPlan}, Check: Seat{Role: SeatCheck}} + if ask.Sends == nil { + ask.Sends = map[crewroute.Seat]string{} } -} - -// CheckSeat is the check seat's own ladder, resolved at the door. -// The flag wins, then the check environment. A plan seat pinned by either its -// flag or its environment answers next. Any other plan source leaves the seat -// empty for the run's crew factory to fill from the careful row. -func CheckSeat(flagCheck string, plan Seat) Seat { - seat := Seat{Role: SeatCheck} - if value := strings.TrimSpace(flagCheck); value != "" { - seat.Model, seat.Source = value, SeatFlag - return seat + for _, slot := range []struct { + seat *Seat + flag string + }{{&seats.Work, flags.Model}, {&seats.Plan, flags.PlanModel}, {&seats.Check, flags.CheckModel}} { + seat := slot.seat + if value := strings.TrimSpace(slot.flag); value != "" { + seat.Model, seat.Source = value, SeatFlag + } else if value := strings.TrimSpace(env.Value(seat.Env())); value != "" { + seat.Model, seat.Source = value, SeatEnv + seat.envSpelling, _ = env.Spelling(seat.Env()) + } else { + continue + } + ask.Sends[crewSeatOf(seat.Role)] = seat.Model } - if value := strings.TrimSpace(env.Get(CheckModelEnv)); value != "" { - seat.Model, seat.Source = value, SeatEnv - return seat + decision, err := RouteCrew(profileDir, ask) + if err != nil && !errors.Is(err, ErrCrewAtCap) { + return seats, err } - if plan.Source == SeatFlag || plan.Source == SeatEnv { - seat.Model, seat.Source = plan.Model, plan.Source + seats.Crew = &decision + for _, seat := range []*Seat{&seats.Work, &seats.Plan, &seats.Check} { + pick := decision.Seat(crewSeatOf(seat.Role)) + seat.Provider = pick.Provider + if seat.Source != "" { + continue + } + seat.Model = pick.Send + if pick.Pinned { + seat.Source = SeatPinned + } else { + seat.Source = SeatRouted + } } - return seat + return seats, err } -// resolveSeat is the ladder itself, once, for either seat. The two seats differ -// in three values — which flag, which tier, what the bottom rung is — and in -// nothing else, which is why there is one function and not two. -// -// UNTOUCHED IS NOT THE CREW ANSWERING. A tier is read through [crewRow] rather -// than [TierModelAt] so that a profile which has never held the key falls -// THROUGH to the default rather than being reported as `crew balanced` — the -// five shipped tier defaults are the balanced row, so TierModelAt would answer -// for a person who has never said anything, the bottom rung would become -// unreachable, and this build's headless default model would quietly change for -// everybody. `crew` here means somebody chose a crew. A row cleared on purpose -// reads empty, which is "follow the conversation" — a sentence a headless run -// has no conversation to answer with — so it falls through too. +// CheckEnvModel is the check seat's environment rung alone, for the chat's +// runs: a conversation has no flag, and CODEAF_CHECK_MODEL reaches a `/task` +// run the way the manual says it reaches a headless one. Empty when unset. +func CheckEnvModel() string { + return strings.TrimSpace(env.Get(CheckModelEnv)) +} + +// TierSeatAt is ONE TIER ROW READ AS A SEAT, for the surfaces that seat roles +// rather than run a door: the conversation's role map (cmd/codeaf's v3Crew), +// the settings sheet's rows, and a run's crew factory for a seat its door did +// not name. // -// AND A KEY THAT WAS NEVER HELD IS NOT THE SAME AS A PROFILE THAT SAID NOTHING. -// The two were one case until the worker row landed (#302) and a profile that -// had pinned every row it knew about fell all the way to the default for the -// seat that does the work. So an unheld key asks the lineage before the bottom -// rung: the crew answers through the ancestor row when the person wrote one, on -// its own rung, and the run says so in one line ([Seat.Notice]). +// A CREW SEAT'S ROW — worker, planner (mastermind), checker (high) — is its +// pin when one is written, and otherwise the router's pick for work of no +// particular class ([standingCrewSeat]): the calls that ride those tiers +// without a task in front of them are routed too, never handed a model this +// build chose for everybody. // -// EVERY RUNG HANDS ITS VALUE ON WHOLE. A tier value may read -// `moonshotai/kimi-k3:low` and a flag may say the same thing; both travel from -// here unchanged, are seeded into the plan role's binding unchanged, and are -// split into a model and a thinking effort by the role ladder at the point of -// the call. A crew rung that shortened the value would leave a person planning -// at `low` in the conversation and at nothing headless — the same divergence -// #166 is about, one knob over. -func resolveSeat(role SeatRole, profileDir, flag, tier, fallback string) Seat { - seat := Seat{Role: role} - if value := strings.TrimSpace(flag); value != "" { - seat.Model, seat.Source = value, SeatFlag - return seat - } - if value := strings.TrimSpace(env.Value(seat.Env())); value != "" { - seat.Model, seat.Source = value, SeatEnv - // The receipt names the spelling that answered — the former one is - // still read for a release, and a person who set it must not be told - // to look at a row they never wrote. - seat.envSpelling, _ = env.Spelling(seat.Env()) - return seat - } - // AND THE PROFILE IS READ THROUGH THE ROW A CONVERSATION READS IT THROUGH - // ([crewRow]), so the three cases a tier key can be in are decided once for - // both surfaces. Nothing the profile can say fills this seat with the - // fallback except silence: a row cleared on purpose says "follow the - // conversation", which a headless run has no conversation to answer with, - // and it lands on the same bottom rung a profile that said nothing does. - model, from, source, cleared := crewRow(profileDir, tier) - if source == "" { - if cleared && role == SeatWork { - fallback = DefaultModel - } - // THE PICK ANSWERS BEFORE THE BOTTOM RUNG: an unwritten seat under a - // pick of catalog or learn is computed at the crew's preset - // ([pickedSeat]), the same answer the conversation reads, and a row - // cleared on purpose still falls through — the pick answers for seats - // nobody named, and a cleared row is somebody naming emptiness. - if !cleared { - if seat, ok := pickedSeat(profileDir, CrewSourceAt(profileDir), tier, "", false); ok { - return seat - } - } - // A STORED CREW WORD SEATS THE HEADLESS DOOR TOO ([unwrittenSeat]): the - // word names a budget, and the row nobody wrote reads that budget's own - // table row rather than the run's fallback — the same seat the - // conversation reads, so a word cannot mean one thing here and another in - // chat. - if !cleared { - if seat, ok := unwrittenSeat(profileDir, CrewSourceAt(profileDir), tier); ok { - return seat - } +// THE OTHER TWO ROWS — reflex and small work — are not crew seats and read the +// way they always have: a row somebody WROTE is theirs, a row they CLEARED +// reads empty and follows the conversation ([TierModelAt] argues why UNSET +// and CLEARED are different answers), and a key never held reads this build's +// own near-free model — or, on an account known to be low, a free one +// ([freeTierModelAt]). +func TierSeatAt(profileDir, tier string) Seat { + if seat, ok := CrewTierSeat(tier); ok { + role := SeatRole(seat) + if pin, ok := CrewPinAt(profileDir, seat); ok { + return Seat{Role: role, Model: pin.Model, Source: SeatPinned, Provider: pin.Provider} } - seat.Model, seat.Source = fallback, SeatDefault - return seat + return Seat{Role: role, Model: standingCrewSeat(profileDir, seat), Source: SeatRouted} } - // A row that says auto resolves through the one seam both ladders share - // ([autoRow]) — on the rung it earned, with the preset it was computed at - // carried for the receipt — and never to `auto` and never to empty, which - // is what a headless run could do nothing with. - if IsAuto(model) { - var preset string - seat.Model, seat.Source, preset = autoRow(profileDir, CrewSourceAt(profileDir), tier) - seat.Crew = preset - return seat + role := SeatRole(tierWords(tier)) + if written, held := persistedString(profileDir, tierKeyFor(tier)); held { + return Seat{Role: role, Model: strings.TrimSpace(written), Source: SeatPinned} } - // AND BETWEEN THE ROW AND ITS OWN RUNG, the pick: a written row holding - // the preset's own table value is the preset answering rather than a pin, - // and the pick computes it ([pickedSeat]) — unless the profile stores a crew - // word, which makes every written row a pin. A hand-typed id keeps its rung - // either way. - if seat, ok := pickedSeat(profileDir, CrewSourceAt(profileDir), tier, model, source == SeatCrew); ok { - return seat + return Seat{Role: role, Model: freeTierModelAt(profileDir, tier), Source: SeatDefault} +} + +// tierWords is a non-crew row's name on the settings sheet. +func tierWords(tier string) string { + switch tier { + case ModelTierReflex: + return "reflex" + case ModelTierLow: + return "small work" } - seat.Model, seat.Source, seat.From, seat.Crew = model, source, from, CrewAt(profileDir) - return seat + return tier } diff --git a/internal/config/seats_test.go b/internal/config/seats_test.go deleted file mode 100644 index 8b9086d9ab..0000000000 --- a/internal/config/seats_test.go +++ /dev/null @@ -1,1042 +0,0 @@ -package config - -import ( - "context" - "encoding/json" - "github.com/Agent-Field/codeaf/internal/env" - "io" - "io/fs" - "net/http" - "net/http/httptest" - "os" - "path/filepath" - "strings" - "testing" - - "github.com/Agent-Field/agentfield/sdk/go/ai" - "github.com/Agent-Field/codeaf/internal/catalog" - "github.com/Agent-Field/codeaf/internal/crewpick" - "github.com/Agent-Field/codeaf/internal/router" -) - -// THE LADDER, PINNED RUNG BY RUNG. The bug it exists to prevent is not a wrong -// model — it is a rung that answers when a rung above it should have, which is -// invisible in a run's output and shows up as a benchmark measuring a crew it -// was not using (#166). So every rung is asserted for both seats, including the -// order between them. -func TestTheSeatLadderAnswersInItsOwnOrder(t *testing.T) { - for _, test := range []struct { - name string - // crew is written with the same writer /crew uses, or left empty for a - // profile nobody has touched. - crew string - hand map[string]string - env, planEnv string - flag, planFlag string - - work, workRung string - plan, planRung string - }{ - { - name: "a profile nobody has touched answers with the build's default", - work: DefaultModel, workRung: "default", - plan: "", planRung: "default", - }, - { - name: "the crew answers when it is the only thing said", - crew: CrewFrugal, - work: "z-ai/glm-5.3-flash", workRung: "crew frugal", - plan: "z-ai/glm-5.3-flash", planRung: "crew frugal", - }, - { - name: "the environment outranks the crew", - crew: CrewFrugal, - env: "vendor/from-the-environment", - planEnv: "vendor/plans-from-the-environment", - work: "vendor/from-the-environment", workRung: ModelEnv, - plan: "vendor/plans-from-the-environment", planRung: PlanModelEnv, - }, - { - name: "the flag outranks the environment", - crew: CrewFrugal, - env: "vendor/from-the-environment", - planEnv: "vendor/plans-from-the-environment", - flag: "vendor/from-the-flag", - planFlag: "vendor/plans-from-the-flag", - work: "vendor/from-the-flag", workRung: "--model", - plan: "vendor/plans-from-the-flag", planRung: "--plan-model", - }, - { - // The two seats climb independently: one flag does not settle both. - name: "one seat may be flagged while the other still climbs", - crew: CrewFrugal, - planEnv: "vendor/plans-from-the-environment", - flag: "vendor/from-the-flag", - work: "vendor/from-the-flag", workRung: "--model", - plan: "vendor/plans-from-the-environment", planRung: PlanModelEnv, - }, - { - // Presets choose models without silently adding a thinking level. - // An operator can still put a level on a flag or tier row. - name: "the crew leaves generation behavior to the model", - crew: CrewMax, - work: "z-ai/glm-5.3", workRung: "crew max", - plan: "anthropic/claude-opus-5", planRung: "crew max", - }, - { - // And a flag carrying one is not shortened either, so the two rungs - // hand the same kind of value to everything downstream. - name: "a flagged level travels the same way", - planFlag: "moonshotai/kimi-k3:low", - work: DefaultModel, workRung: "default", - plan: "moonshotai/kimi-k3:low", planRung: "--plan-model", - }, - { - // One row answered by hand is a custom crew, and the receipt says - // so rather than naming a preset the four rows do not make. - name: "a hand-written row reads as the custom crew it makes", - crew: CrewFrugal, - // THE WORK SEAT IS THE WORKER ROW, and not the small-work row beside - // it: the same row a task handed off in conversation rides. - hand: map[string]string{ModelTierWorker: "vendor/my-own-worker"}, - work: "vendor/my-own-worker", workRung: "crew custom", - plan: "z-ai/glm-5.3-flash", planRung: "crew custom", - }, - { - // A row cleared on purpose means "follow the conversation", and a - // headless run has no conversation to follow. - name: "a cleared row falls through to the default", - crew: CrewFrugal, - hand: map[string]string{ModelTierWorker: "", ModelTierMastermind: ""}, - work: DefaultModel, workRung: "default", - plan: "", planRung: "default", - }, - } { - t.Run(test.name, func(t *testing.T) { - dir := t.TempDir() - if test.crew != "" { - if err := ApplyCrew(dir, test.crew); err != nil { - t.Fatalf("writing the %s crew: %v", test.crew, err) - } - } - for tier, model := range test.hand { - if err := writeTierModel(dir, tier, model); err != nil { - t.Fatalf("writing the %s row: %v", tier, err) - } - } - // Set at call time, both of them, so a case that names neither is - // running against a shell that has. - t.Setenv(ModelEnv, test.env) - t.Setenv(PlanModelEnv, test.planEnv) - - seats := ResolveSeats(dir, test.flag, test.planFlag) - if seats.Work.Model != test.work || seats.Work.Rung() != test.workRung { - t.Errorf("work seat = %q (%s), want %q (%s)", - seats.Work.Model, seats.Work.Rung(), test.work, test.workRung) - } - if seats.Plan.Model != test.plan || seats.Plan.Rung() != test.planRung { - t.Errorf("plan seat = %q (%s), want %q (%s)", - seats.Plan.Model, seats.Plan.Rung(), test.plan, test.planRung) - } - }) - } -} - -// AN UNTOUCHED PROFILE IS NOT THE CREW ANSWERING, and this is the decision -// written down where it can be argued with. -// -// The four shipped tier defaults ARE the balanced row (crew.go), so [CrewAt] -// says `balanced` about a profile nobody has ever opened. Reading the seats -// through [TierModelAt] would therefore let the crew rung answer for a person -// who has never said anything about models — the bottom rung would become -// unreachable, and this build's headless default model would silently change -// from [DefaultModel] to the low tier's. So the ladder asks whether the key was -// WRITTEN, and `crew` on a receipt means somebody chose a crew. -func TestAnUntouchedProfileReadsTheDefaultAndNotTheBalancedCrew(t *testing.T) { - dir := t.TempDir() - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - - if got := CrewAt(dir); got != DefaultCrew { - t.Fatalf("an untouched profile reads crew %q, so this test is about the wrong thing", got) - } - seats := ResolveSeats(dir, "", "") - if seats.Work.Source != SeatDefault || seats.Work.Model != DefaultModel { - t.Fatalf("work seat = %q (%s), want the build default", seats.Work.Model, seats.Work.Rung()) - } - if seats.Plan.Source != SeatDefault || seats.Plan.Model != "" { - t.Fatalf("plan seat = %q (%s), want an empty seat that follows the work model", - seats.Plan.Model, seats.Plan.Rung()) - } - - // And the same profile once somebody has actually said `balanced`: the same - // four models, and now the crew is what answered. - if err := ApplyCrew(dir, CrewBalanced); err != nil { - t.Fatal(err) - } - seats = ResolveSeats(dir, "", "") - if seats.Work.Source != SeatCrew || seats.Work.Rung() != "crew balanced" { - t.Fatalf("after /crew balanced the work seat reads %s", seats.Work.Rung()) - } -} - -// The receipt is the whole point of resolving the rung, so its sentence is -// pinned: both seats, each with what chose it, on one line. -func TestTheReceiptNamesBothSeatsAndTheirRungs(t *testing.T) { - dir := t.TempDir() - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "vendor/plans") - if err := ApplyCrew(dir, CrewFrugal); err != nil { - t.Fatal(err) - } - line := ResolveSeats(dir, "", "").Line() - frugal, _ := CrewModels(CrewFrugal) - want := "models: work " + frugal[ModelTierWorker] + " (crew frugal) · plan vendor/plans (" + PlanModelEnv + ")" - if line != want { - t.Fatalf("the opening line reads\n\t%s\nwant\n\t%s", line, want) - } - - // An unfilled plan seat is said in words rather than left as a gap. - t.Setenv(PlanModelEnv, "") - empty := ResolveSeats(t.TempDir(), "", "") - if !strings.Contains(empty.Line(), "plan follows the work model (default)") { - t.Fatalf("an empty plan seat reads %q", empty.Line()) - } -} - -// THE LEVEL COMES OFF AT THE WIRE AND NOWHERE EARLIER. -// -// `moonshotai/kimi-k3:low` is how a tier row, a flag and a variable all say -// "that model, thinking a little". The value travels whole to the client seam, -// which keeps the level beside the adapter while asking the provider for the -// model alone. A slug with the level on it is one no provider publishes; sent -// whole it is a 404 on every call the seat makes, which is what the flag path -// did before this seam knew the difference. -func TestTheThinkingLevelNeverReachesTheProviderAsPartOfTheSlug(t *testing.T) { - var asked string - server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - raw, _ := io.ReadAll(r.Body) - var body struct { - Model string `json:"model"` - } - _ = json.Unmarshal(raw, &body) - asked = body.Model - w.Header().Set("Content-Type", "application/json") - _, _ = io.WriteString(w, `{"choices":[{"message":{"content":"hi"},"finish_reason":"stop"}],"usage":{}}`) - })) - defer server.Close() - - settings := Config{APIKey: "k", BaseURL: server.URL, Timeout: DefaultTimeout} - ask := func(client router.Client, err error) string { - t.Helper() - if err != nil { - t.Fatal(err) - } - asked = "" - if _, err := client.CompleteWithMessages(context.Background(), - []ai.Message{{Role: "user", Content: []ai.ContentPart{{Type: "text", Text: "hi"}}}}); err != nil { - t.Fatal(err) - } - return asked - } - - // The plan seat, however it was filled. - if got := ask(settings.ClientFor("moonshotai/kimi-k3:low")); got != "moonshotai/kimi-k3" { - t.Fatalf("the plan client asked for %q, want the model id alone", got) - } - // And the work seat, which reaches the same seam through Client(). - settings.Model = "moonshotai/kimi-k3:high" - if got := ask(settings.Client()); got != "moonshotai/kimi-k3" { - t.Fatalf("the work client asked for %q, want the model id alone", got) - } - // A suffix that is not a level is part of the id and is left alone. - if got := ask(settings.ClientFor("vendor/model:free")); got != "vendor/model:free" { - t.Fatalf("a non-level suffix was eaten: %q", got) - } - // The settings keep what the person wrote, so the role ladder still sees - // the level it has to apply. - if settings.Model != "moonshotai/kimi-k3:high" { - t.Fatalf("building a client rewrote the seat to %q", settings.Model) - } -} - -// A PROFILE OLDER THAN A SEAT STILL GETS THE CREW IT CHOSE, AND IS TOLD SO. -// -// This is #302 written down. A crew applied before the worker row existed pinned -// the four rows there were, and the ladder read the fifth key, found nothing, -// and fell past the whole profile to the build's default — so every headless run -// worked on a model the person had never named while their planning ran on the -// one they had. The only tell was one word on a line nobody reads twice. -// -// The rung is the crew, through the row the worker row was split out of, and it -// says `inherited` rather than borrowing the plain crew rung: the person is owed -// the difference between the row they wrote and the row they did not. -func TestACrewWrittenBeforeTheWorkerSeatStillAnswersForItAndSaysSo(t *testing.T) { - dir := t.TempDir() - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - // The pre-#278 crew shape, written the way it was written then: four rows - // and no worker row. - for tier, model := range map[string]string{ - ModelTierReflex: "mistralai/mistral-nemo", - ModelTierLow: "deepseek/deepseek-v4-flash", - ModelTierHigh: "qwen/qwen3.8-27b", - ModelTierMastermind: "deepseek/deepseek-v4-flash", - } { - if err := writeTierModel(dir, tier, model); err != nil { - t.Fatalf("writing the %s row: %v", tier, err) - } - } - if _, held := persistedString(dir, KeyTierWorkerModel); held { - t.Fatal("the fixture holds a worker row, so it is not the profile this is about") - } - - seats := ResolveSeats(dir, "", "") - if seats.Work.Model != "deepseek/deepseek-v4-flash" { - t.Errorf("the work seat runs on %q, want the small-work model the person pinned", seats.Work.Model) - } - if seats.Work.Model == DefaultModel { - t.Errorf("the work seat fell to the build's default %q with a whole crew written above it", DefaultModel) - } - if seats.Work.Source != SeatInherited || seats.Work.From != ModelTierLow { - t.Errorf("the work seat says %s from %q, want the crew through the small-work row", - seats.Work.Source, seats.Work.From) - } - if got := seats.Work.Rung(); got != "crew custom, inherited" { - t.Errorf("the receipt reads %q, want the crew and the fact that it was inherited", got) - } - // The seat the profile does hold a row for is untouched by any of this. - if seats.Plan.Source != SeatCrew || seats.Plan.Model != "deepseek/deepseek-v4-flash" { - t.Errorf("the plan seat reads %q (%s)", seats.Plan.Model, seats.Plan.Rung()) - } -} - -// AND THE PLAIN-INSTALL PATH IS BYTE-IDENTICAL. A profile that has said nothing -// about models has nothing to inherit from, so the bottom rung answers and -// nothing is said about it — the third acceptance of #302, and the rung whose -// unreachability the ladder's own comment warns about. -func TestAProfileThatHasSaidNothingStillRunsTheBuildsDefaultAndSaysNothing(t *testing.T) { - dir := t.TempDir() - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - - seats := ResolveSeats(dir, "", "") - if seats.Work.Source != SeatDefault || seats.Work.Model != DefaultModel { - t.Fatalf("the work seat reads %q (%s), want the build's default", seats.Work.Model, seats.Work.Rung()) - } - if notice := seats.Notice(); notice != "" { - t.Fatalf("a profile with no crew was told about an inheritance that did not happen: %q", notice) - } - if seats.Report() != seats.Line() { - t.Fatalf("the report grew a line: %q", seats.Report()) - } - - // One row written by hand, and the seat that has no row of its own inherits - // it — the same profile, one key later. - if err := writeTierModel(dir, ModelTierLow, "vendor/my-small-work"); err != nil { - t.Fatal(err) - } - seats = ResolveSeats(dir, "", "") - if seats.Work.Source != SeatInherited || seats.Work.Model != "vendor/my-small-work" { - t.Fatalf("the work seat reads %q (%s), want the small-work row it inherits from", - seats.Work.Model, seats.Work.Rung()) - } - // A row cleared ON PURPOSE is an answer, and the answer is "follow the - // conversation" — which a headless run cannot, so it falls to the default - // rather than being reported as a crew that chose it. - if err := writeTierModel(dir, ModelTierLow, ""); err != nil { - t.Fatal(err) - } - if seats := ResolveSeats(dir, "", ""); seats.Work.Source != SeatDefault { - t.Fatalf("a cleared small-work row was inherited anyway: %q (%s)", - seats.Work.Model, seats.Work.Rung()) - } -} - -// THE LINE IS SAID ONCE, AND ONLY WHEN IT IS TRUE. -// -// A substitution nobody is told about is the defect; a warning on every one of a -// run's calls is the same defect wearing a hat, because a person learns to read -// past it. So it belongs exactly where the seats are reported, once, and the -// register is the one the surface already speaks in — an observation, a middle -// dot, a promise, lowercase, no full stop, no machinery. -func TestTheInheritedSeatIsAnnouncedOnceInTheHousesOwnVoice(t *testing.T) { - dir := t.TempDir() - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - if err := writeTierModel(dir, ModelTierLow, "vendor/my-small-work"); err != nil { - t.Fatal(err) - } - seats := ResolveSeats(dir, "", "") - - notice := seats.Notice() - if notice == "" { - t.Fatal("the seat was inherited and nothing was said about it") - } - report := seats.Report() - if got := strings.Count(report, notice); got != 1 { - t.Fatalf("the line is printed %d times in one report, want once:\n%s", got, report) - } - if !strings.HasPrefix(report, seats.Line()+"\n") { - t.Fatalf("the report is not the models line and then the reason:\n%s", report) - } - if !strings.Contains(notice, "small work") { - t.Errorf("the line does not name the row a person would go and look at: %q", notice) - } - if !strings.Contains(notice, "work seat") { - t.Errorf("the line does not name the seat that was filled: %q", notice) - } - if notice != strings.ToLower(notice) { - t.Errorf("the line is not lowercase: %q", notice) - } - if strings.HasSuffix(notice, ".") { - t.Errorf("the line ends in a full stop, which makes a remark into an announcement: %q", notice) - } - if strings.Count(notice, " · ") != 1 { - t.Errorf("the line is not the observation-then-promise the surface speaks in: %q", notice) - } - if strings.Contains(notice, "\n") { - t.Errorf("the line is more than one line: %q", notice) - } - for _, machinery := range []string{"tier", "models.tiers", "fallback", "resolve", "seatinherited", "config"} { - if strings.Contains(notice, machinery) { - t.Errorf("the line says %q, which is machinery: %q", machinery, notice) - } - } - - // And a crew that pins every row hears nothing at all: the notice is a fact - // about this profile, not decoration on the models line. - pinned := t.TempDir() - if err := ApplyCrew(pinned, CrewFrugal); err != nil { - t.Fatal(err) - } - if notice := ResolveSeats(pinned, "", "").Notice(); notice != "" { - t.Fatalf("a fully pinned crew was told about an inheritance: %q", notice) - } - // Nor does a seat somebody filled by hand this minute. - t.Setenv(ModelEnv, "vendor/from-the-environment") - if notice := ResolveSeats(dir, "", "").Notice(); notice != "" { - t.Fatalf("a seat filled from the environment was called inherited: %q", notice) - } -} - -// EVERY TIER DECLARES ITS ANCESTRY, OR DECLARES THAT IT HAS NONE. -// -// The worker row will not be the last seat added, and the failure it caused is -// invisible from outside — so the rule lives in one table and this reads the -// table rather than the code that uses it. A tier word added to [ModelTiers] -// without a row here is the same defect again, one seat later, and it fails -// here instead of in somebody's benchmark. -func TestEveryTierDeclaresItsAncestry(t *testing.T) { - known := make(map[string]bool, len(ModelTiers)) - for _, tier := range ModelTiers { - known[tier] = true - } - for _, tier := range ModelTiers { - row, ok := tierLineage[tier] - if !ok { - t.Errorf("the %s tier declares no ancestry — name the row it was split out of, "+ - "or an empty ancestor to say it was always here", tier) - continue - } - if row.Words == "" { - t.Errorf("the %s tier has no name a person would recognise, so the line about it "+ - "cannot send them to a row they can find", tier) - } - if row.Inherits != "" && !known[row.Inherits] { - t.Errorf("the %s tier inherits from %q, which is not a tier", tier, row.Inherits) - } - // A ring of rows would make the walk depend on where it started. - seen := map[string]bool{tier: true} - for step := row.Inherits; step != ""; step = tierLineage[step].Inherits { - if seen[step] { - t.Fatalf("the lineage from %s comes back to %s", tier, step) - } - seen[step] = true - } - } - for tier := range tierLineage { - if !known[tier] { - t.Errorf("the table holds %q, which is not one of this build's tiers", tier) - } - } - // The one ancestry there is today, named rather than merely well-formed: the - // worker seat came out of the small-work row (#278), and a change of mind - // about that is a change of what old profiles run on. - if got := tierLineage[ModelTierWorker].Inherits; got != ModelTierLow { - t.Errorf("the worker row inherits from %q, want the small-work row it was split out of", got) - } -} - -// APPLYING A CREW PINS EVERY SEAT, so a profile written by this build never -// needs the lineage at all — the inheritance is for the profiles that already -// exist, and it must not become the ordinary path. -func TestApplyingACrewLeavesNoSeatToInherit(t *testing.T) { - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - for _, preset := range CrewPresets { - dir := t.TempDir() - if err := ApplyCrew(dir, preset); err != nil { - t.Fatal(err) - } - for _, tier := range ModelTiers { - if _, held := persistedString(dir, tierKeyFor(tier)); !held { - t.Errorf("%s leaves the %s row unwritten, so a seat on it inherits", preset, tier) - } - } - if seats := ResolveSeats(dir, "", ""); seats.Work.Source != SeatCrew { - t.Errorf("%s: the work seat reads %s", preset, seats.Work.Rung()) - } - } -} - -// THE SAME LADDER, READ THE WAY A CONVERSATION READS IT. -// -// #312 is #302 on the other surface. The chat's role map, the settings sheet and -// the crew word all resolve a tier row through [TierModelAt], which knew two -// answers — the row somebody wrote, and this build's choice — so a profile older -// than the worker seat handed every task started in a conversation to the build's -// default while the same profile handed `codeaf do` the crew's own model. The -// rung is the same rung; what this table pins is that adding it moved NOTHING -// else: a row written, a row cleared and a profile with no rows at all read -// exactly what they read before, on every tier. -func TestATierRowIsSeatedTheWayAConversationReadsIt(t *testing.T) { - const small = "vendor/my-small-work" - for _, test := range []struct { - name string - // rows is the profile's config.json, written as a profile of that - // vintage holds it: a key present with a value, a key present and empty, - // or a key that is simply not there. - rows map[string]string - tier string - model string - source SeatSource - from string - }{ - { - name: "a key that was never held on a profile older than the seat takes the row it was split out of", - rows: map[string]string{KeyTierLowModel: small}, - tier: ModelTierWorker, - model: small, source: SeatInherited, from: ModelTierLow, - }, - { - name: "a row somebody wrote is the crew answering and nothing climbs past it", - rows: map[string]string{KeyTierLowModel: small, KeyTierWorkerModel: "vendor/my-worker"}, - tier: ModelTierWorker, - model: "vendor/my-worker", source: SeatCrew, - }, - { - // The decision this whole rung is fenced against: emptying a row is - // somebody saying "follow the conversation", and a conversation is - // the one thing that CAN. It must not start inheriting. - name: "a row cleared on purpose still follows the conversation", - rows: map[string]string{KeyTierLowModel: small, KeyTierWorkerModel: ""}, - tier: ModelTierWorker, - model: "", source: SeatDefault, - }, - { - name: "a profile that has said nothing still reads this build's choice", - rows: map[string]string{}, - tier: ModelTierWorker, - model: DefaultWorkerModel, source: SeatDefault, - }, - { - // An ancestor that was itself cleared is not an answer either: the - // walk stops at the row that was written, and an empty one is not. - name: "an ancestor cleared on purpose is not inherited from", - rows: map[string]string{KeyTierLowModel: ""}, - tier: ModelTierWorker, - model: DefaultWorkerModel, source: SeatDefault, - }, - { - // And a tier with no ancestor is untouched by any of it. - name: "a row that was always here has nothing to inherit", - rows: map[string]string{KeyTierLowModel: small}, - tier: ModelTierHigh, - model: DefaultHighModel, source: SeatDefault, - }, - } { - t.Run(test.name, func(t *testing.T) { - dir := writeProfileRows(t, test.rows) - seat := TierSeatAt(dir, test.tier) - if seat.Model != test.model || seat.Source != test.source || seat.From != test.from { - t.Errorf("the %s row seats %q (%s from %q), want %q (%s from %q)", - test.tier, seat.Model, seat.Source, seat.From, test.model, test.source, test.from) - } - // AND THE MODEL EVERY OTHER CALLER READS IS THIS SEAT'S MODEL. The - // sheet, the crew word and the conversation's role map all go - // through TierModelAt, and a seat that disagreed with it would be a - // receipt about a model nothing runs on. - if got := TierModelAt(dir, test.tier); got != seat.Model { - t.Errorf("TierModelAt reads %q while the seat reads %q", got, seat.Model) - } - }) - } -} - -// THE LINE IS THE SAME LINE ON BOTH SURFACES, and the noun in it is the row a -// person can go and find. A second wording for one fact is how somebody -// concludes the two surfaces are doing two different things. -func TestAnInheritedTierSeatSaysTheSameSentenceTheRunSays(t *testing.T) { - dir := writeProfileRows(t, map[string]string{KeyTierLowModel: "vendor/my-small-work"}) - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - - seat := TierSeatAt(dir, ModelTierWorker) - if got, want := seat.Notice(), ResolveSeats(dir, "", "").Work.Notice(); got != want { - t.Fatalf("the conversation reads\n\t%s\nand the run reads\n\t%s", got, want) - } - if got := seat.FromWords(); got != "small work" { - t.Errorf("the row is called %q here and `small work` on the settings sheet", got) - } - // Every other shape says nothing at all, which is what makes the line worth - // reading when it is there. - for name, rows := range map[string]map[string]string{ - "a pinned row": {KeyTierWorkerModel: "vendor/my-worker"}, - "a cleared row": {KeyTierLowModel: "vendor/my-small-work", KeyTierWorkerModel: ""}, - "no rows": {}, - } { - seat := TierSeatAt(writeProfileRows(t, rows), ModelTierWorker) - if notice := seat.Notice(); notice != "" { - t.Errorf("%s was told about an inheritance that did not happen: %q", name, notice) - } - if words := seat.FromWords(); words != "" { - t.Errorf("%s names a row it did not inherit from: %q", name, words) - } - } -} - -// ONE READER OF A TIER ROW, AND EVERY SURFACE BEHIND IT. -// -// The defect was not that the ladder was wrong — it was that the conversation -// had a second, shorter one. So this reads the sources and states the law twice: -// inside this package, only the file that owns the ladder turns a tier key into -// a MODEL (asking whether a key exists is a different question, and -// [CrewConfigured] asks it), and outside it nobody spells a tier key at all. A -// surface that wants a model asks [TierModelAt]; one that also wants the rung -// asks [TierSeatAt]; nobody opens the file for themselves. -func TestEveryReadOfATierRowGoesThroughTheLadder(t *testing.T) { - root := moduleRoot(t) - inside, err := filepath.Glob(filepath.Join(root, "internal", "config", "*.go")) - if err != nil { - t.Fatal(err) - } - for _, path := range inside { - base := filepath.Base(path) - if strings.HasSuffix(base, "_test.go") || base == "seats.go" { - continue - } - raw, err := os.ReadFile(path) - if err != nil { - t.Fatal(err) - } - for _, line := range strings.Split(string(raw), "\n") { - if !strings.Contains(line, "tierKeyFor(") || !strings.Contains(line, "persistedString(") { - continue - } - // `_, ok :=` is the presence question and is nobody's second ladder. - if !strings.Contains(line, "_, ok :=") && !strings.Contains(line, "_, held :=") { - t.Errorf("%s reads the VALUE of a tier row for itself: %q\n\tthe ladder is seats.go's, and a "+ - "second reader is how `unset` comes to mean one thing here and another there", - base, strings.TrimSpace(line)) - } - } - } - // And outside this package, nothing spells a tier key at all. - _ = filepath.WalkDir(root, func(path string, entry fs.DirEntry, err error) error { - switch { - case err != nil: - return nil - case entry.IsDir(): - if name := entry.Name(); name == ".git" || name == "bin" || name == "bench" { - return filepath.SkipDir - } - return nil - case !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "_test.go"): - return nil - case strings.HasPrefix(path, filepath.Join(root, "internal", "config")): - return nil - } - raw, err := os.ReadFile(path) - if err != nil { - return nil - } - if strings.Contains(string(raw), `"models.tiers.`) { - rel, _ := filepath.Rel(root, path) - t.Errorf("%s spells a tier key by hand instead of asking the ladder", rel) - } - return nil - }) - // AND THE CONVERSATION'S OWN ROLE MAP IS ON IT. This is the wiring #312 was - // about: the three rows the chat resolves from the profile go through this - // package, which is what makes the worker row inherit there too. - raw, err := os.ReadFile(filepath.Join(root, "cmd", "codeaf", "chatv3.go")) - if err != nil { - t.Fatal(err) - } - for _, tier := range []string{"ModelTierReflex", "ModelTierMastermind", "ModelTierWorker"} { - if want := "config.TierModelAt(c.profileDir, config." + tier + ")"; !strings.Contains(string(raw), want) { - t.Errorf("the chat door does not seat the %s row through the ladder (%s)", tier, want) - } - } -} - -// writeProfileRows makes a throwaway profile holding exactly these keys — a -// value for a row somebody wrote, the empty string for a row they cleared, and -// nothing at all for a key of a vintage that never had one. -func writeProfileRows(t *testing.T, rows map[string]string) string { - t.Helper() - dir := t.TempDir() - raw, err := json.Marshal(rows) - if err != nil { - t.Fatal(err) - } - if err := os.WriteFile(BudgetConfigPath(dir), raw, 0o600); err != nil { - t.Fatal(err) - } - return dir -} - -// moduleRoot walks up to the directory holding go.mod, for the two tests that -// read the tree they are part of. -func moduleRoot(t *testing.T) string { - t.Helper() - dir, err := os.Getwd() - if err != nil { - t.Fatal(err) - } - for { - if _, err := os.Stat(filepath.Join(dir, "go.mod")); err == nil { - return dir - } - parent := filepath.Dir(dir) - if parent == dir { - t.Fatalf("no go.mod above %s", dir) - } - dir = parent - } -} - -// ── THE INHERITED-SEAT NOTICE ENDED IN A DOOR NOBODY COULD FIND ───────────── -// -// The line is printed on four HEADLESS doors — `codeaf do`, `codeaf plan run`, -// `codeaf exec` and `codeaf run` — and it ended `until you pick a crew again`. -// There is no way to pick a crew from a terminal: a crew is written by `/crew` -// and by the settings sheet's Providers row, both of which are the -// conversation, and no flag and no verb in the binary sets one. So the one line -// whose whole job is to explain a surprising model choice told somebody to do -// something and gave them nowhere to do it. -// -// The law on both surfaces is CAUSE PLUS WHAT TO DO. A cause plus a dead end is -// the defect, and the fix is the address, not the noun. -func TestTheInheritedSeatNoticeNamesADoorAPersonCanActuallyReach(t *testing.T) { - dir := writeProfileRows(t, map[string]string{KeyTierLowModel: "vendor/my-small-work"}) - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - notice := ResolveSeats(dir, "", "").Work.Notice() - if notice == "" { - t.Fatal("a profile older than the worker seat was told nothing at all") - } - if !strings.Contains(notice, CrewCommand) { - t.Fatalf("the remedy names no door at all, so a person reading it in a terminal "+ - "has nowhere to go.\n said: %q\n want it to name %q", notice, CrewCommand) - } - // AND IT SAYS WHERE THAT DOOR IS. `/crew` alone is the next keystroke to - // somebody already in the conversation and a riddle to somebody in a shell, - // and the shell is where four of the five printers of this line stand. - if !strings.Contains(notice, CrewCommand+" in the conversation") { - t.Fatalf("the door is named without saying where it is.\n said: %q", notice) - } - // THE REGISTER THE FUNCTION'S OWN COMMENT SETS OUT: an observation, a middle - // dot, a promise, lowercase, no full stop. - if !strings.Contains(notice, " · ") { - t.Errorf("the observation and the promise are not separated by the middle dot: %q", notice) - } - if strings.HasSuffix(notice, ".") { - t.Errorf("the line ends in a full stop, which nothing else in this register does: %q", notice) - } - if first := notice[:1]; first != strings.ToLower(first) { - t.Errorf("the line opens upper-case: %q", notice) - } -} - -// AND `seat` IS THE PRODUCT'S OWN NOUN, NOT MACHINERY. -// -// It was proposed for renaming as machinery vocabulary. It is not: the crew and -// its seats are what the settings sheet, the model picker and the manual's own -// page all teach a person to say, and the vocabulary law bans the PROGRAM'S -// words for its own process — `auditor`, `verdict`, `refuted` — not a domain -// noun the product names things with. Renaming it here would have left every -// other surface, and the page that explains this exact line, speaking a word the -// binary no longer printed. -func TestTheNoticeKeepsTheProductsOwnWordForARowOfTheCrew(t *testing.T) { - dir := writeProfileRows(t, map[string]string{KeyTierLowModel: "vendor/my-small-work"}) - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - notice := ResolveSeats(dir, "", "").Work.Notice() - for _, want := range []string{"crew", "seat"} { - if !strings.Contains(notice, want) { - t.Fatalf("the notice stopped saying %q, and every other crew surface still does:\n%s", - want, notice) - } - } -} - -// TestTheInheritedNoticeNamesTheSeatItBorrowedFromRatherThanDescribingTheModel -// is the defect a developer met on their first run. -// -// The banner read -// -// models: work deepseek/deepseek-v4-pro (crew custom, inherited) -// your crew was set before the work seat existed · it is running on your small work model until you pick a crew with /crew in the conversation -// -// — two consecutive lines, one naming a model and the next calling that same -// model small, with no way for the reader to tell which model the lane was -// actually on. The two were never in disagreement about the fact: an inherited -// seat's model IS the model on the line above, always. So the notice may not -// describe the model at all; it names the SEAT the model was borrowed from, -// which is the only thing the line above does not already say. -func TestTheInheritedNoticeNamesTheSeatItBorrowedFromRatherThanDescribingTheModel(t *testing.T) { - dir := writeProfileRows(t, map[string]string{KeyTierLowModel: "deepseek/deepseek-v4-pro"}) - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - - report := ResolveSeats(dir, "", "").Report() - if !strings.Contains(report, "deepseek/deepseek-v4-pro") { - t.Fatalf("the banner never names the model the work seat is on:\n%s", report) - } - // The row's own name is still said — a person has to know which seat to go - // and look at — but never as an adjective in front of `model`. - if !strings.Contains(report, "your small work seat's model") { - t.Errorf("the notice does not say which seat lent the model, so the reader "+ - "cannot find the row to change:\n%s", report) - } - for _, describes := range []string{"small work model", "worker model", "careful work model", "thinking model", "reflex model"} { - if strings.Contains(report, describes) { - t.Errorf("the banner names %s on one line and calls it %q on the next — "+ - "a seat's name used as an adjective reads as a claim about the model, "+ - "and the two lines are about ONE model:\n%s", - "deepseek/deepseek-v4-pro", "your "+describes, report) - } - } -} - -// A person who set the former spelling of the model variable is told that -// spelling on the receipt, not the current one they never wrote; with both -// set, the current one answered and is named. -func TestTheReceiptNamesTheSpellingOfTheVariableThatAnswered(t *testing.T) { - dir := t.TempDir() - legacy := env.Legacy(ModelEnv) // AFORGE_MODEL — legacy-name - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - t.Setenv(legacy, "vendor/former") - seats := ResolveSeats(dir, "", "") - if seats.Work.Source != SeatEnv || seats.Work.Model != "vendor/former" { - t.Fatalf("work seat = %q (%s), want the former spelling's value from the env rung", seats.Work.Model, seats.Work.Rung()) - } - if got := seats.Work.Rung(); got != legacy { - t.Fatalf("rung = %q, want %q — the receipt must name the variable the person set", got, legacy) - } - t.Setenv(ModelEnv, "vendor/current") - seats = ResolveSeats(dir, "", "") - if got := seats.Work.Rung(); got != ModelEnv || seats.Work.Model != "vendor/current" { - t.Fatalf("rung = %q model = %q, want %q and its value", got, seats.Work.Model, ModelEnv) - } -} - -// ── the pick row's half of the ladder ─────────────────────────────────────── - -// A PICK OFF THE TABLE COMPUTES THE DIAL SEATS AND LEAVES THE OTHER TWO ALONE. -// The three seats that the presets dial — worker, careful work, mastermind — -// are computed at the crew's preset, `learned` under the learn word and -// `computed from the catalog` under the catalog word; reflex and small work -// always read the table, which is what makes a pick a dial over three seats -// and not a second crew. The ids are asserted against crewpick's own front, -// the way the shipped tables are (auto_test.go owns the method). -func TestAPickOffTheTableComputesTheDialSeats(t *testing.T) { - restore := AutoModels - AutoModels = func() []catalog.Model { return autoTestRows() } - defer func() { AutoModels = restore }() - - _, balanced, _ := crewpick.Presets(crewpick.Front(autoCandidates(autoTestRows()), crewpick.DefaultShapes(), crewpick.All)) - table, _ := CrewModelsForSource(CrewSourceAt(""), DefaultCrew) - - for _, pick := range []struct { - word string - source SeatSource - }{ - {CrewPickLearn, SeatLearned}, - {CrewPickCatalog, SeatComputed}, - } { - dir := writeProfileRows(t, map[string]string{KeyCrewPick: pick.word}) - for _, c := range []struct { - tier string - want string - }{ - {ModelTierWorker, balanced.Worker}, - {ModelTierHigh, balanced.High}, - {ModelTierMastermind, balanced.Mastermind}, - } { - seat := TierSeatAt(dir, c.tier) - if seat.Model != c.want || seat.Source != pick.source { - t.Errorf("%s: the %s seat reads %q (%s), want %q (%s)", pick.word, c.tier, - seat.Model, seat.Source, c.want, pick.source) - } - if seat.Crew != DefaultCrew { - t.Errorf("%s: the %s seat ran at %q, want the untouched profile's %q", pick.word, c.tier, seat.Crew, DefaultCrew) - } - } - // THE PICK COMPUTES RATHER THAN READS: the front's ids are not the - // preset table's own, so a seat carrying the front's id is a seat the - // catalog answered and not the table in disguise. - if balanced.Worker == table[ModelTierWorker] { - t.Fatal("the test rows field a front identical to the shipped table; the pick would be asserting nothing") - } - // REFLEX AND SMALL WORK ALWAYS READ THE TABLE, and a pick does not - // touch them: the same rung an untouched profile has always read, with - // the table's own id. - for _, tier := range []string{ModelTierReflex, ModelTierLow} { - seat := TierSeatAt(dir, tier) - if seat.Model != table[tier] { - t.Errorf("%s: the %s seat reads %q, want the table's %q", pick.word, tier, seat.Model, table[tier]) - } - if seat.Source != SeatDefault { - t.Errorf("%s: the %s seat reads rung %q, want the table's own default", pick.word, tier, seat.Source) - } - } - } -} - -// A HAND-TYPED MODEL SURVIVES THE PICK. The pick answers for the seats nobody -// named, not over the names they typed — and the headless ladder says the same -// thing the conversation says, because a pick that meant one thing in chat and -// another under `codeaf do` would be the divergence the ladder exists to -// prevent. -func TestAHandTypedSeatSurvivesThePick(t *testing.T) { - restore := AutoModels - AutoModels = func() []catalog.Model { return autoTestRows() } - defer func() { AutoModels = restore }() - - dir := writeProfileRows(t, map[string]string{ - KeyCrewPick: CrewPickLearn, - tierKeyFor(ModelTierWorker): "someone/else", - }) - - seat := TierSeatAt(dir, ModelTierWorker) - if seat.Model != "someone/else" || seat.Source != SeatCrew { - t.Fatalf("a hand-typed worker reads %q (%s), want someone/else (crew)", seat.Model, seat.Source) - } - // The seats the pick does answer are computed at the preset the stored - // rows still make — the hand-typed row costs them their preset match, so - // the budget falls back to the default preset, which is what an undecided - // profile runs at. - _, balanced, _ := crewpick.Presets(crewpick.Front(autoCandidates(autoTestRows()), crewpick.DefaultShapes(), crewpick.All)) - high := TierSeatAt(dir, ModelTierHigh) - if high.Model != balanced.High || high.Source != SeatLearned { - t.Fatalf("the careful seat reads %q (%s), want %q (learned)", high.Model, high.Source, balanced.High) - } - - // AND THE HEADLESS LADDER SAYS THE SAME. The work seat is the worker tier - // and the plan seat is the mastermind tier, and both must answer exactly - // what the conversation's rows answer — the receipt's own claim. - seats := ResolveSeats(dir, "", "") - if seats.Work.Model != "someone/else" || seats.Work.Source != SeatCrew { - t.Fatalf("headless, the work seat reads %q (%s), want someone/else (crew)", seats.Work.Model, seats.Work.Source) - } - if seats.Plan.Model != balanced.Mastermind || seats.Plan.Source != SeatLearned { - t.Fatalf("headless, the plan seat reads %q (%s), want %q (learned)", seats.Plan.Model, seats.Plan.Source, balanced.Mastermind) - } -} - -// THE PICK'S RUNGS SAY THEIR WORDS, so a receipt never has to be guessed at: -// the catalog word computes, the learn word learned, and the table's own -// fallback keeps its one word. -func TestThePickRungsSayTheirWords(t *testing.T) { - seat := Seat{Crew: CrewBalanced, Source: SeatComputed} - if got := seat.Rung(); got != "crew "+CrewBalanced+", computed from the catalog" { - t.Errorf("the computed rung reads %q, want crew %s, computed from the catalog", got, CrewBalanced) - } - seat.Source = SeatLearned - if got := seat.Rung(); got != "crew "+CrewBalanced+", learned" { - t.Errorf("the learned rung reads %q, want crew %s, learned", got, CrewBalanced) - } -} - -// A BARE `auto` ROW IS COMPUTED UNDER THE PICK TOO, and its rung names the word -// that ran. The pool's measurements reach the auto row through the same prior -// the pick carries — [AutoPick] wraps [autoPrior] — so a row computed under the -// learn word is a learned seat and must say so, not the catalog word's -// `computed from the catalog`. [autoRow] is the one seam both ladders resolve an -// auto row through, so the conversation's row and the headless work seat must -// name the same rung: a row that meant one thing in chat and another headless is -// the divergence the seam exists to prevent. -func TestAnAutoRowUnderLearnReadsTheLearnRung(t *testing.T) { - t.Setenv(ModelEnv, "") - t.Setenv(PlanModelEnv, "") - restore := AutoModels - AutoModels = func() []catalog.Model { return autoTestRows() } - defer func() { AutoModels = restore }() - - dir := writeProfileRows(t, map[string]string{ - KeyCrewPick: CrewPickLearn, - tierKeyFor(ModelTierWorker): AutoValue, - }) - want := "crew " + CrewBalanced + ", learned" - for name, seat := range map[string]Seat{ - "conversation": TierSeatAt(dir, ModelTierWorker), - "headless": ResolveSeats(dir, "", "").Work, - } { - if got := seat.Rung(); got != want { - t.Errorf("%s: a bare auto row under %s reads rung %q, want %q", name, CrewPickLearn, got, want) - } - if strings.TrimSpace(seat.Model) == "" || IsAuto(seat.Model) { - t.Errorf("%s: the auto seat reads model %q, want a computed id", name, seat.Model) - } - } -} - -// THE CHECK SEAT IS RESOLVED AT THE DOOR, rung by rung: its flag, its -// environment, a plan seat pinned by flag or environment, then empty for the -// crew factory's careful row. A plan filled by the crew does not answer. -func TestCheckSeatClimbsItsOwnLadder(t *testing.T) { - planFlag := Seat{Role: SeatPlan, Model: "vendor/named-plan-flag", Source: SeatFlag} - planEnv := Seat{Role: SeatPlan, Model: "vendor/named-plan-env", Source: SeatEnv} - planCrew := Seat{Role: SeatPlan, Model: "vendor/crew-plan", Source: SeatCrew} - - t.Run("check flag", func(t *testing.T) { - t.Setenv(CheckModelEnv, "vendor/named-check-env") - seat := CheckSeat("vendor/named-check-flag", planFlag) - if seat.Model != "vendor/named-check-flag" || seat.Source != SeatFlag { - t.Fatalf("check seat = %q (%s), want the check flag", seat.Model, seat.Rung()) - } - }) - t.Run("check environment", func(t *testing.T) { - t.Setenv(CheckModelEnv, "vendor/named-check-env") - seat := CheckSeat("", planFlag) - if seat.Model != "vendor/named-check-env" || seat.Source != SeatEnv { - t.Fatalf("check seat = %q (%s), want the check environment", seat.Model, seat.Rung()) - } - }) - t.Run("plan flag", func(t *testing.T) { - t.Setenv(CheckModelEnv, "") - seat := CheckSeat("", planFlag) - if seat.Model != planFlag.Model || seat.Source != SeatFlag { - t.Fatalf("check seat = %q (%s), want the flagged plan", seat.Model, seat.Rung()) - } - }) - t.Run("plan environment", func(t *testing.T) { - t.Setenv(CheckModelEnv, "") - seat := CheckSeat("", planEnv) - if seat.Model != planEnv.Model || seat.Source != SeatEnv { - t.Fatalf("check seat = %q (%s), want the environment plan", seat.Model, seat.Rung()) - } - }) - t.Run("crew careful", func(t *testing.T) { - t.Setenv(CheckModelEnv, "") - for _, plan := range []Seat{{Role: SeatPlan}, planCrew} { - if seat := CheckSeat("", plan); seat.Model != "" { - t.Fatalf("check seat = %q, want empty for the crew's careful row", seat.Model) - } - } - }) -} diff --git a/internal/config/settings.go b/internal/config/settings.go index 9bda64e36b..7ca1cc07ae 100644 --- a/internal/config/settings.go +++ b/internal/config/settings.go @@ -10,6 +10,7 @@ import ( "strings" "time" + "github.com/Agent-Field/codeaf/internal/crewroute" "github.com/Agent-Field/codeaf/internal/ctxbudget" "github.com/Agent-Field/codeaf/internal/env" "github.com/Agent-Field/codeaf/internal/pool/poolcfg" @@ -220,50 +221,6 @@ const ( // [ValidateTierValue] is the same gate on all five — but this is the one the // shipped crew writes it into. KeyTierMastermindModel = "models.tiers.mastermind" - // KeyCrew is the five tiers answered as ONE DECISION. Nobody arrives wanting - // to name five model ids; they arrive wanting to spend pennies, or to spend - // what it takes. So the row takes one word — frugal, balanced, max — and - // writes all five tier rows from it. - // - // IT IS NOT STORED. The row's reading is DERIVED from the five live tier - // values: they match a preset and it says so, or they do not and it says - // custom. A stored word would be a claim about five other rows that any one - // of them could falsify by being edited, and a settings sheet that told you - // "balanced" over a hand-pinned tier would be lying in the one place a person - // went to check. - // - // THE BUILD WRITES NO WORD HERE, AND READS ONE THAT IS. The derivation above is - // what every profile this product shapes reads — but a run that wrote the - // word itself (a harness, a hand edit) named a budget, and a crew word that - // reached nobody is worse than a row five others can falsify: [storedCrewWord] - // reads it as the budget its seats run at, and the class rows under it are - // that run's own pins ([pickedSeat]). - KeyCrew = "models.crew" - // KeyCrewSource is which family the crew words draw from: `open`, the - // open-weight table this build ships, or `all`, the same three words - // resolved over the whole catalog with closed and frontier models in it. - // IT IS A ROW RATHER THAN A SECOND VOCABULARY because the three words are - // the only thing anybody learns: the question a person arrives with is how - // much to spend, and which shelf the answer comes off is one more answer to - // the same question, not six new preset words. The row is read by the - // crew's own machinery (crew.go's [CrewSourceAt]) and never by a caller - // spelling the ids itself, so the family and the tables cannot disagree - // about what a preset means. It is PROFILE-ONLY with the tier rows, for the - // worker row's own reason: a repository that could answer it could send a - // visitor's work, and their credit, to a vendor they never chose. - KeyCrewSource = "models.crew.source" - // KeyCrewPick is where the crew's seats are picked from when a tier row - // does not hold a model id of its own. The three words are read and - // answered by the crew's own machinery (crew.go's [CrewPickAt] and - // [SetCrewPick]) beside the words [KeyCrew] and [KeyCrewSource] take: the - // crew row says how much to spend, the family row says which shelf those - // budgets name, and this row says where the models for that money come - // from — the rows this build measured, or a computation off the catalog - // made again on every read, with or without what the Model Pool measured. - // It is PROFILE-ONLY with the crew and family rows, for the worker row's - // own reason: a repository that could answer it could send a visitor's - // work, and their credit, to a model nobody on that machine chose. - KeyCrewPick = "models.crew.pick" // KeyMouse is whether the surface reports the mouse at all. ON is the // default ([DefaultMouse]), because hover, click and the wheel are v3's own // language and the thing they cost is bought back by a key: an alt-screen @@ -1030,45 +987,24 @@ const ( // [tierKeyFor] is total over it. var ModelTiers = []string{ModelTierReflex, ModelTierLow, ModelTierWorker, ModelTierHigh, ModelTierMastermind} -// THE SHIPPED CREW. All five tiers arrive pointed at a model, and the five -// together are exactly the `balanced` row of the DEFAULT FAMILY (crew.go's -// [crewAllModels], named by [DefaultCrewSource]) — which is what makes the crew -// row read "balanced" on a profile nobody has touched instead of reading -// "custom" about its own defaults. +// THE TWO ROWS THAT ARE NOT CREW SEATS arrive pointed at a model. The reflex +// and small-work tiers carry the calls a conversation makes on its own behalf — +// twice a turn for the reflex — so a person who never opened the sheet gets a +// model that costs near nothing rather than the one they are talking to. The +// three crew seats have no shipped model at all: an unpinned seat is routed per +// task (crew.go), and a build-chosen worker, planner or checker would be the +// implicit fallback the router exists to remove. // // Each is a bare OpenRouter id, spelled ONCE here and read by every caller -// through [TierModelAt], so the model this build considers near-free is one -// string rather than a figure repeated in a row, a resolver and a page. -// -// Blank is still an answer on every one of them: a row a person emptied on -// purpose reads empty and the roles on it follow the model the person is talking +// through [TierModelAt]. Blank is still an answer on both: a row a person +// emptied on purpose reads empty and follows the model the person is talking // to, which is [roles.Resolve]'s floor. UNSET and CLEARED are different answers -// here, and that distinction is the whole mechanism ([TierModelAt] says how). -// -// The ids are read off the catalog's own published rows — its intelligence, -// coding and agentic indexes against its prompt, completion and cache-read -// prices, priced under each seat's own call shape (crew.go's [crewAllModels] -// comment owns the method and the date). The low row is pinned to a DATED build -// on purpose: the bare `deepseek/deepseek-v4-flash` id resolves to the April +// here ([TierModelAt] says how). The low row is pinned to a DATED build on +// purpose: the bare `deepseek/deepseek-v4-flash` id resolves to the April // build, and the July build costs the same. const ( DefaultReflexModel = "google/gemini-2.5-flash" DefaultLowModel = "deepseek/deepseek-v4-flash-0731" - // The worker is the seat that pays most of a task's bill, so it is the last - // seat a preset spends on: a step here multiplies through every token a task - // runs up, where a step on the two low-volume seats is paid a handful of - // times. glm-5.3-flash is the point on the long-cached-loop front that - // balanced runs at, and it can see images, which the parent can hand it - // without a vision detour. - DefaultWorkerModel = "z-ai/glm-5.3-flash" - // The careful tier is ALWAYS A DIFFERENT VENDOR FROM THE WORKER, in every - // preset, and always a model that sees images: a check from a second vendor - // catches what the first vendor's blind spots let through, and the vision - // role rides this row. - DefaultHighModel = "anthropic/claude-fable-5.1" - // The mastermind names a capable planning model. Its generation behavior is - // left to the provider unless an operator adds a level to the model id. - DefaultMastermindModel = "anthropic/claude-opus-5" ) // DocumentEngines are the four rungs CODEAF_DOC_ENGINE accepts. @@ -2420,62 +2356,13 @@ func (s *Settings) build() []Setting { read: func() string { return formatDuration(resolvedDuration(BriefAfterAt(dir))) }, write: func(raw string) error { return writeDuration(dir, KeyBriefAfter, raw) }, }, - // THE CREW, AND THEN THE FOUR CLASSES IN IT. The tiers are what a person - // actually configures for the calls codeaf makes on its own — the name it - // gives a session, the check on work a task says is finished, the plan an - // adaptive run steers by (internal/roles). Four rows, not one per feature: a new - // call joins a class and needs no row of its own. - // - // The crew row comes FIRST because it is the only one most people will - // ever touch: one word writes all four (crew.go). The four below it are - // what that word wrote, and each is answerable on its own — which is what - // turns the crew reading to "custom". - Setting{ - Key: KeyCrew, Category: CategoryModels, Kind: SettingChoice, - Label: "crew", Choices: CrewPresets, - Hint: "the five models codeaf works with, chosen as one word. `frugal`, " + - "`balanced` and `max` each pick their own roster, listed by /crew and under " + - "the crew row, and which shelf they draw from is the `model family` row " + - "below. Change one of the five rows below and this reads `custom`.", - read: func() string { return CrewAt(dir) }, - write: func(raw string) error { return writeCrew(dir, raw) }, - }, - // WHERE THE SEATS ARE PICKED FROM, one row under the crew. The crew row - // says how much to spend and this says where the models for that money - // come from when a tier row does not hold a person's own id: the rows - // this build measured and shipped, or the same three budgets recomputed - // off the catalog on every read, with or without what the Model Pool - // and the person's own judged runs measured. The words are the crew's - // own (crew.go), so the row and the ladder cannot disagree about what a - // pick means. - Setting{ - Key: KeyCrewPick, Category: CategoryModels, Kind: SettingChoice, - Label: "picked from", Choices: CrewPicks, - Hint: "where the crew's models come from. table: the rows we measured. " + - "catalog: recomputed from today's published prices and scores at your " + - "crew's budget. learn: catalog plus the Model Pool's measurements and " + - "your own judged runs.", - read: func() string { return CrewPickAt(dir) }, - write: func(raw string) error { return SetCrewPick(dir, raw) }, - }, - // THE FAMILY THE THREE WORDS DRAW FROM, one row under the crew. It sits - // beside the crew row because it is the same decision read one level up: - // the crew row says which five models, and this says which shelf those - // five come off. `all` is the default and what the shipped five are the - // balanced row of; `open` narrows the same three words to open weights. - Setting{ - Key: KeyCrewSource, Category: CategoryModels, Kind: SettingChoice, - Label: "model family", Choices: CrewSources, - Hint: "which family the crew words draw from. `all` is the default: frugal, " + - "balanced and max read off the whole catalog, closed and frontier models " + - "included, and cost what those models cost. `open` reads the same three " + - "words off the open-weight rows only, so no seat is a bet on one vendor's " + - "pricing. Seats nobody pinned move with the family at " + - "once, because an unwritten seat is the default crew; rows already written " + - "keep their ids until you pick the crew again.", - read: func() string { return CrewSourceAt(dir) }, - write: func(raw string) error { return SetCrewSource(dir, raw) }, - }, + // THE TWO ROWS THAT ARE NOT THE CREW, AND THEN THE CREW'S THREE PINS. + // The tiers are what a person configures for the calls codeaf makes on + // its own — the name it gives a session, the check on work a task says is + // finished, the plan an adaptive run steers by (internal/roles). The + // crew's three rows are PINS: blank is `auto`, the router picking that + // seat for each task, and /crew is where they are read and set with the + // allowed models and the cap beside them (crew.go). Setting{ Key: KeyTierReflexModel, Category: CategoryModels, Kind: SettingText, Label: "reflex", EmptyLabel: "follows the conversation", @@ -2494,37 +2381,32 @@ func (s *Settings) build() []Setting { read: func() string { return TierModelAt(dir, ModelTierLow) }, write: func(raw string) error { return writeTierModel(dir, ModelTierLow, raw) }, }, - // The worker row is the one most people will change second, after the - // crew: it is the seat that does the work and pays most of a task's bill. Setting{ Key: KeyTierWorkerModel, Category: CategoryModels, Kind: SettingText, - Label: "worker", EmptyLabel: "follows the conversation", - Hint: "the model that does the work — every task you hand off, the parts it " + - "divides into, and the nodes of an adaptive run. Most of what a task costs " + - "is spent here. Leave it blank and tasks ride the model you are talking to; " + - "the `task model` row under Tasks, when set, wins over this one.", - read: func() string { return TierModelAt(dir, ModelTierWorker) }, - write: func(raw string) error { return writeTierModel(dir, ModelTierWorker, raw) }, + Label: "worker", EmptyLabel: CrewAuto, + Hint: "the model that does the work of every task — blank is auto: codeaf picks it " + + "for each task from the kind of work it is. Pin one with a model id, or " + + "`model@provider` to pin the route too. /crew shows all three seats.", + read: func() string { return crewSeatRow(dir, crewroute.Worker) }, + write: func(raw string) error { return SetCrewPin(dir, crewroute.Worker, raw) }, }, Setting{ Key: KeyTierHighModel, Category: CategoryModels, Kind: SettingText, - Label: "careful work", EmptyLabel: "follows the conversation", - Hint: "the capable model for the things that must not be wrong — the check on " + - "finished task work, the brief a task is shaped into, reading an image.", - read: func() string { return TierModelAt(dir, ModelTierHigh) }, - write: func(raw string) error { return writeTierModel(dir, ModelTierHigh, raw) }, + Label: "checker", EmptyLabel: CrewAuto, + Hint: "the model that checks finished task work, and reads an image for a model " + + "that cannot — blank is auto: a strong checker for open-ended work, a cheap one " + + "for a narrow fix. Pin one with a model id, or `model@provider`.", + read: func() string { return crewSeatRow(dir, crewroute.Checker) }, + write: func(raw string) error { return SetCrewPin(dir, crewroute.Checker, raw) }, }, - // The fourth tier is the one whose value may name a LEVEL as well as a - // model, because it is the one class of call where how hard the model - // thinks is the point rather than the price. Setting{ Key: KeyTierMastermindModel, Category: CategoryModels, Kind: SettingText, - Label: "mastermind", EmptyLabel: "follows the conversation", - Hint: "the model that plans adaptive runs and designs saved harnesses — the one " + - "answer that decides what every other call does. Add `:low`, `:medium` or " + + Label: "planner", EmptyLabel: CrewAuto, + Hint: "the model that plans a task's work and designs saved harnesses — blank is " + + "auto. Pin one with a model id, or `model@provider`; add `:low`, `:medium` or " + "`:high` to ask it to think that hard: `moonshotai/kimi-k3:high`.", - read: func() string { return TierModelAt(dir, ModelTierMastermind) }, - write: func(raw string) error { return writeTierModel(dir, ModelTierMastermind, raw) }, + read: func() string { return crewSeatRow(dir, crewroute.Planner) }, + write: func(raw string) error { return SetCrewPin(dir, crewroute.Planner, raw) }, }, Setting{ Key: KeyModelRoles, Category: CategoryModels, Kind: SettingText, @@ -4038,33 +3920,20 @@ func ParseToolApprovals(raw string) (map[string]string, error) { return pairs, nil } -// TierModelAt resolves the model one auxiliary tier runs on. Empty means the -// tier follows the session's own model, which is internal/roles' floor. +// TierModelAt resolves the model one tier runs on. Empty means the tier follows +// the session's own model, which is internal/roles' floor. // -// UNSET AND CLEARED ARE DIFFERENT ANSWERS, on all five tiers. A profile that has -// never held the key gets this build's own choice for that class of work -// ([DefaultReflexModel] and its four neighbours), because a person who never -// opened the sheet should not have the whole crew answering on the most -// expensive model in the build — which is what following the conversation means -// once there is a mastermind tier in it. A row somebody emptied ON PURPOSE reads -// empty and follows the conversation, because refusing to let them turn it off -// would make a default into a rule. +// A CREW SEAT'S TIER — worker, mastermind (the planner), high (the checker) — +// reads its pin, or the router's standing pick when it has none ([TierSeatAt]), +// so the calls that ride those tiers outside a task follow the crew without a +// second place to set it. // -// AND UNSET HAS TWO READINGS OF ITS OWN, which is the rung this function learned -// in #312. A key that was never held on a profile OLDER THAN ITS SEAT is not a -// person declining to answer — it is a crew chosen before the row existed — so -// the read climbs [TierSeatAt], where an unheld key asks the row it was split -// out of first ([tierLineage]) and only a profile with nothing above it reaches -// the build's choice. Every caller of this function therefore reads the model a -// conversation ACTUALLY runs that class of work on: the role map cmd/codeaf -// builds, the settings sheet's five rows, and [CrewAt], which is why the crew -// word and the work cannot disagree. A caller that also needs to say WHERE the -// answer came from asks [TierSeatAt] for the seat instead of this for the model. -// -// The reflex tier was the first row written this way, for the reason its key -// still states: a call made twice a turn is a bill nobody agreed to. The other -// three joined it when the crew landed, and the four defaults together are one -// preset rather than four opinions (crew.go). +// THE OTHER TWO ROWS KEEP THE RULE THEY HAVE ALWAYS HAD: UNSET AND CLEARED ARE +// DIFFERENT ANSWERS. A profile that has never held the key gets this build's +// own near-free model ([DefaultReflexModel], [DefaultLowModel]), because a call +// made twice a turn is a bill nobody agreed to; a row somebody emptied ON +// PURPOSE reads empty and follows the conversation, because refusing to let +// them turn it off would make a default into a rule. // // The value may carry a level (`moonshotai/kimi-k3:low`) and IS RETURNED WHOLE. // Splitting is [roles.SplitEffort]'s job at the point of resolution, because a @@ -4089,35 +3958,12 @@ func tierKeyFor(tier string) string { return KeyTierLowModel } -// defaultTierModel is what a tier answers on a profile that has never held its -// key: the DEFAULT CREW, resolved in the family the profile chose. It is not the -// build's constant alone, because a profile that answered the family row and no -// tier row would otherwise run one family's crew under the other's label: the -// family saying one thing and the ladder another. Under the default family the -// answer is the constant, which is what [DefaultWorkerModel] and its kin name, -// because that family's default row and the builtin five name the same set by -// construction (crew_test.go pins it). -func defaultTierModel(family, tier string) string { - if table, ok := CrewModelsForSource(family, DefaultCrew); ok { - if model, held := table[tier]; held { - return model - } - } - return builtinTierModel(tier) -} - -// builtinTierModel is the build's own five. It is reachable only for a tier word -// no family table holds, because both tables answer every tier this build knows. +// builtinTierModel is the build's own model for the two rows that are not crew +// seats. A crew seat's tier has none — it is routed — and reads the small-work +// model only if a caller asks for a tier word this build does not know. func builtinTierModel(tier string) string { - switch tier { - case ModelTierReflex: + if tier == ModelTierReflex { return DefaultReflexModel - case ModelTierWorker: - return DefaultWorkerModel - case ModelTierHigh: - return DefaultHighModel - case ModelTierMastermind: - return DefaultMastermindModel } return DefaultLowModel } diff --git a/internal/config/settings_test.go b/internal/config/settings_test.go index ad1e0aef42..abd0f78761 100644 --- a/internal/config/settings_test.go +++ b/internal/config/settings_test.go @@ -773,14 +773,11 @@ func TestMaskCredentialHidesTheKeyAndItsLength(t *testing.T) { // The three throttle rows read their defaults, take a person's answer, and hand // it back to the accessor the session door calls — which is the whole of what a // settings row has to do. -// EVERY TIER ROW SHIPS POINTED AT A MODEL, and the two answers a person can give -// one are different from each other: never touching it is this build's own -// choice, emptying it on purpose is "follow the conversation". -// -// The reflex row was the first written this way, for the reason its key states. -// The other three joined it when the crew landed (crew.go), because a whole crew -// following the conversation means the most expensive model in the build -// answering the cheapest questions in it. +// THE TWO ROWS THAT ARE NOT CREW SEATS SHIP POINTED AT A MODEL, and the two +// answers a person can give one are different from each other: never touching +// it is this build's own choice, emptying it on purpose is "follow the +// conversation". The crew's three seats ship with NO model: unpinned, a seat is +// routed, and a profile with no provider connected has nothing to route to. func TestEveryTierShipsWithAModelAndCanStillBeCleared(t *testing.T) { dir := t.TempDir() row := mustRow(t, registry(t, dir), KeyTierReflexModel) @@ -791,15 +788,12 @@ func TestEveryTierShipsWithAModelAndCanStillBeCleared(t *testing.T) { if got := row.Value(); got != DefaultReflexModel { t.Fatalf("the reflex row reads %q in an untouched profile, want %q", got, DefaultReflexModel) } - // And so do its three neighbours, each with the model this build chose for - // that class of work. - for _, c := range []struct{ tier, want string }{ - {ModelTierLow, DefaultLowModel}, - {ModelTierHigh, DefaultHighModel}, - {ModelTierMastermind, DefaultMastermindModel}, - } { - if got := TierModelAt(dir, c.tier); got != c.want { - t.Fatalf("the %s tier resolves %q in an untouched profile, want %q", c.tier, got, c.want) + if got := TierModelAt(dir, ModelTierLow); got != DefaultLowModel { + t.Fatalf("the low tier resolves %q in an untouched profile, want %q", got, DefaultLowModel) + } + for _, tier := range []string{ModelTierWorker, ModelTierHigh, ModelTierMastermind} { + if seat := TierSeatAt(dir, tier); seat.Source != SeatRouted { + t.Fatalf("the %s tier reads %+v in an untouched profile, want a routed seat", tier, seat) } } diff --git a/internal/config/sources_test.go b/internal/config/sources_test.go index 77a7a2ac9d..4c25124121 100644 --- a/internal/config/sources_test.go +++ b/internal/config/sources_test.go @@ -469,7 +469,7 @@ func TestReconnectingACollidingServiceKeepsTheFirstSuggestedName(t *testing.T) { func TestAnUnqualifiedIdResolvesExactlyAsItDidBefore(t *testing.T) { models := []string{ DefaultModel, DefaultVoiceModel, DefaultReflexModel, DefaultLowModel, - DefaultWorkerModel, DefaultHighModel, DefaultMastermindModel, + "z-ai/glm-5.3-flash", "moonshotai/kimi-k3", "anthropic/claude-opus-5", } for _, base := range []string{DefaultBaseURL, "https://company.example/v1"} { for _, model := range models { diff --git a/internal/config/testdata/profile-keys.ledger b/internal/config/testdata/profile-keys.ledger index 4d6c828b2f..ec695bced4 100644 --- a/internal/config/testdata/profile-keys.ledger +++ b/internal/config/testdata/profile-keys.ledger @@ -45,8 +45,13 @@ model.work model_pool model_sources models.crew +models.crew.allowed +models.crew.cap +models.crew.free_routes models.crew.pick +models.crew.providers.off models.crew.source +models.crew.task_cap models.fallbacks models.pool.public_key models.roles diff --git a/internal/crewpick/crewpick.go b/internal/crewpick/crewpick.go deleted file mode 100644 index a8e6942667..0000000000 --- a/internal/crewpick/crewpick.go +++ /dev/null @@ -1,715 +0,0 @@ -// Package crewpick picks one model per seat for a three-seat crew by reading -// the pareto front of expected task bill against crew quality. -// -// It deliberately imports nothing from the rest of the tree and touches no -// disk and no network: the caller owns the candidate list and the seat -// shapes, and the same candidates give the same front however often they are -// asked about and in whatever order they arrive. -// -// The law it implements: every (worker, high, mastermind) combination the -// candidates can field is scored by the bill its seats' token volumes run up -// and by the mean of its three seat qualities. A candidate is in the running -// for a seat only when it meets the family and the seat's needs and its -// quality reaches Floor of that seat's best in the family, and a crew is -// refused outright when its high seat comes from the worker's own vendor. -// The crews that no other crew beats on both bill and quality survive as the -// front, sorted by bill; Presets and AtKnob read picks off that front. -// -// A candidate that publishes some but not all of its three indexes is scored -// on the ones it publishes: each missing index is estimated from the call's -// candidates that publish both it and one of its measured ones — the donors — -// never above the call's largest measured value of that index, and the -// result says which of a pick's indexes were estimated. A candidate that -// publishes none of its indexes, or no price at all, is out of the running. -package crewpick - -import ( - "math" - "slices" - "sort" - "strings" -) - -// An Index is one of the three capability indexes quality is read from — -// intelligence, coding, agentic — in the order the seat weights and a -// candidate's reading carry them. -type Index int - -const ( - Intelligence Index = iota - Coding - Agentic -) - -// A Candidate is one model up for a seat. Prices are dollars per 1M tokens. -// -// A candidate carrying none of the three indexes — zero values, which is how -// unpublished indexes read — or carrying a zero prompt price is not a -// candidate at all: it never occupies a seat, never clears a floor, and -// raises no ceiling. A candidate carrying some but not all of its indexes is -// scored on the ones it publishes, each missing one estimated from the -// donors among its fellow candidates, and the result says which of a pick's -// indexes those were. A model that publishes no cache-read price says so with -// HasCacheRead false and pays the prompt price on the cache share too. -type Candidate struct { - // ID names the candidate; the text before its first slash is the vendor. - ID string - // Open marks a candidate the open family may draw from. - Open bool - // Intelligence, Coding and Agentic are the three capability indexes - // quality is read from. - Intelligence float64 - Coding float64 - Agentic float64 - // PromptPrice, CompletionPrice and CacheReadPrice are the dollars per - // 1M tokens of each kind. - PromptPrice float64 - CompletionPrice float64 - CacheReadPrice float64 - // HasCacheRead reports whether CacheReadPrice is published at all. - HasCacheRead bool - // Context is the model's context window, in tokens. - Context int - // Images and Tools are what the model can take and call. - Images bool - Tools bool -} - -// A Seat is one of a crew's three roles: the worker carries the volume of -// the work, the high seat judges and repairs what the worker writes, and the -// mastermind spends a little context steering the whole task. -type Seat int - -const ( - Worker Seat = iota - High - Mastermind -) - -// A SeatShape is how one seat uses whatever model sits it: the weights -// quality is read from, the token mix cost is computed on, what the model must -// be able to take and call, the window it must have, and the share of a -// task's tokens the seat burns. -type SeatShape struct { - // Weights reads quality from the three indexes: intelligence, coding, - // agentic. - Weights [3]float64 - // InOut is the seat's input tokens per output token; the completion - // price is spread over it. - InOut float64 - // CacheShare is the share of the seat's input tokens served from cache. - CacheShare float64 - // NeedTools and NeedImages are what a model must take and call to sit - // the seat. - NeedTools bool - NeedImages bool - // MinContext is the window a model must have to sit the seat. - MinContext int - // Volume is the seat's share of a task's tokens, and the weight its - // cost carries in a crew's bill. - Volume float64 -} - -// DefaultShapes returns the three seat shapes crews are picked for. The -// worker takes half its quality from the agentic index and reads barely -// anything but cached input; the high seat judges the three indexes evenly, -// needs tools and images both, and sees the same token mix on a sliver of the -// volume; the mastermind reads intelligence and agentic only, on a terse -// ten-in-one mix with no cache and a fraction of the volume. -func DefaultShapes() map[Seat]SeatShape { - return map[Seat]SeatShape{ - Worker: { - Weights: [3]float64{0.2, 0.3, 0.5}, - InOut: 100, - CacheShare: 0.75, - NeedTools: true, - MinContext: 200_000, - Volume: 0.90, - }, - High: { - Weights: [3]float64{1.0 / 3.0, 1.0 / 3.0, 1.0 / 3.0}, - InOut: 100, - CacheShare: 0.75, - NeedTools: true, - NeedImages: true, - MinContext: 200_000, - Volume: 0.08, - }, - Mastermind: { - Weights: [3]float64{0.6, 0, 0.4}, - InOut: 10, - MinContext: 200_000, - Volume: 0.02, - }, - } -} - -// A Family is which candidates a crew may be drawn from. -type Family int - -const ( - // Open draws only from candidates marked Open. - Open Family = iota - // All draws from every candidate. - All -) - -// A Crew is one (worker, high, mastermind) pick. Bill is the expected dollars -// per 1M task tokens once each seat's share of the task is paid for at its -// seat cost; Quality is the mean of the three seat qualities on a 0-100 -// scale. -type Crew struct { - Worker string - High string - Mastermind string - Bill float64 - Quality float64 - // Estimated marks, for the model in each seat, which of its indexes - // were scored by estimate rather than measurement: the row is the seat - // and the column the index. A crew picked on measured indexes alone is - // all false. - Estimated [3][3]bool - // Measured marks, for the model in each seat, whether a pool's own - // rating for that model entered the seat's quality: the row is the seat - // and the value says the seat's model carried a rating with a positive - // observation count. A crew picked with no prior, or whose seats the - // prior never rated, is all false. - Measured [3]bool -} - -// Floor is the share of a seat's best quality in the family that a candidate -// must reach to stay in the running for that seat. A model that cannot do -// the job is not cheap, whatever it costs. -const Floor = 0.80 - -// PriorWeightAt is the observation count at which a pool's own rating for a -// model carries the same weight in a seat's quality as the catalog's -// published indexes: at N observations the rating is worth half the seat and -// the catalog the other half. A rating with no observations never enters. -const PriorWeightAt = 30 - -// A Rating is a pool's measured quality for one model on one seat, on the -// same 0-100 scale quality is read on, with the number of observations behind -// it. -type Rating struct { - Mean float64 - N int -} - -// A Prior is a pool's measured quality per seat, keyed by canonical model id: -// the rating a seat's quality is blended towards, by how many observations -// back it. A nil Prior is no prior at all, and every seat keeps the catalog -// quality. -type Prior map[Seat]map[string]Rating - -// A Cell is one measurement a caller hands PriorFromCells: a role, a model id -// and the pool's mean quality with the observations behind it. It is the shape -// a measurement document reads into, so crewpick need not import the reader. -type Cell struct { - Role string - Model string - Mean float64 - N int -} - -// seatWords names the roles a Cell's role word maps to a seat by, folded to -// one spelling so a document's own case never matters. Any other word names -// no seat and is ignored. -var seatWords = map[string]Seat{ - "worker": Worker, - "high": High, - "mastermind": Mastermind, -} - -// PriorFromCells reads a pool's measurements into a Prior: a cell names a -// seat by its role word, a model by an id that canonical resolves to the -// canonical id a candidate is looked up by, and carries a mean and a count. -// A cell whose role names no seat, or whose count is below minInstalls, is -// ignored, and so is a count of zero or less. A nil canonical leaves the id -// as it stands. When no cell survives the prior is nil, which is the same as -// no prior at all. -// -// The floor here keeps counting rows rather than the installs a measurement -// document carries beside them: the reader applies the document's floor -// before these cells arrive — on installs where a cell spells them — and -// rows cannot be fewer than the installs that produced them, so a cell that -// floor kept meets this one too. The cells built outside a document carry -// rows alone, so an installs field on Cell would sit unset on every caller, -// and there is none. -func PriorFromCells(cells []Cell, minInstalls int, canonical func(string) string) Prior { - var prior Prior - for _, c := range cells { - seat, ok := seatWords[strings.ToLower(strings.TrimSpace(c.Role))] - if !ok || c.N < minInstalls || c.N <= 0 { - continue - } - model := c.Model - if canonical != nil { - model = canonical(model) - } - if prior == nil { - prior = Prior{} - } - if prior[seat] == nil { - prior[seat] = map[string]Rating{} - } - prior[seat][model] = Rating{Mean: c.Mean, N: c.N} - } - return prior -} - -// MergePriors folds two priors into one. Seat by seat, a rating both hold for -// a model is combined by observation count — the two means weighted by the -// counts behind them, the counts added — and a rating either holds alone is -// carried whole. A nil prior on either side is the other, and two nils are -// nil. The priors given are read and never changed, so the answer is never -// one of them. -func MergePriors(a, b Prior) Prior { - if a == nil { - return b - } - if b == nil { - return a - } - merged := make(Prior) - for seat, ratings := range a { - for model, r := range ratings { - if merged[seat] == nil { - merged[seat] = map[string]Rating{} - } - merged[seat][model] = r - } - } - for seat, ratings := range b { - for model, r := range ratings { - got, ok := merged[seat][model] - if !ok { - if merged[seat] == nil { - merged[seat] = map[string]Rating{} - } - merged[seat][model] = r - continue - } - n := got.N + r.N - if n <= 0 { - merged[seat][model] = Rating{} - continue - } - merged[seat][model] = Rating{ - Mean: (float64(got.N)*got.Mean + float64(r.N)*r.Mean) / float64(n), - N: n, - } - } - } - return merged -} - -// SeatCost is the seat's expected cost of a candidate, in dollars per 1M -// input-equivalent tokens: the prompt price blended with the cache-read -// price by the seat's cache share, plus the completion price spread over the -// seat's input-to-output ratio. A candidate that publishes no cache-read -// price pays the prompt price on the cache share too. -func SeatCost(c Candidate, shape SeatShape) float64 { - cache := c.PromptPrice - if c.HasCacheRead { - cache = c.CacheReadPrice - } - return (1-shape.CacheShare)*c.PromptPrice + shape.CacheShare*cache + c.CompletionPrice/shape.InOut -} - -// SeatQuality is the candidate's quality for the seat on a 0-100 scale: each -// index counted against the pool's best for that index and weighted by the -// seat's weights. The pool sets the scale, so the same candidate scores -// differently in a stronger pool. A missing index is estimated from the -// pool's donors exactly as in Front. A row that is not a candidate at all, -// and any row against a pool with no candidates to raise a scale, score -// zero. -func SeatQuality(c Candidate, shape SeatShape, pool []Candidate) float64 { - return SeatQualityWith(c, shape, pool, Worker, nil) -} - -// SeatQualityWith is SeatQuality with the pool's prior blended in for the seat -// given: a rating the prior holds for the candidate's canonical id, with a -// positive observation count, moves the seat's quality towards the rating's -// mean by N/(N+PriorWeightAt), and a seat the prior never rates — or rates -// with no observations — keeps the catalog quality unchanged. -func SeatQualityWith(c Candidate, shape SeatShape, pool []Candidate, seat Seat, prior Prior) float64 { - if !isCandidate(c) { - return 0 - } - scale := maxima(pool) - r := fill(c, readTally(pool), scale) - q, _ := blendQuality(c.ID, quality(r.idx, shape, scale), seat, prior) - return q -} - -// Front returns the pareto front of (bill, quality) over every crew the -// candidates can field in the family, sorted by bill: a crew on it is beaten -// by no other crew on both, and every crew off it is beaten by one on it. -// -// shapes must carry an entry for every seat. Per seat the running is -// narrowed twice before crews are enumerated — family and the seat's needs -// first, then Floor against that seat's best in the family — and a crew is -// refused while its high seat comes from the worker's own vendor, so the two -// seats that see the same work never share one shop. Ties in (bill, quality) -// break to the crew scored on fewer estimated indexes, and ties past that by -// id order — worker, then high, then mastermind — which is what makes the -// front a property of the candidates rather than of their order. -func Front(candidates []Candidate, shapes map[Seat]SeatShape, fam Family) []Crew { - return FrontWith(candidates, shapes, fam, nil) -} - -// FrontWith is Front with the pool's prior blended into each seat's quality: a -// rating the prior holds for a seat's model, with a positive observation -// count, moves that seat's quality towards the rating's mean by -// N/(N+PriorWeightAt), the seat's measured flag says so, and the floor and the -// crew's quality read the blended value. A nil prior leaves every seat on the -// catalog quality, exactly as Front does. The front is a property of the -// candidates and the prior, never of the order they arrive in. -func FrontWith(candidates []Candidate, shapes map[Seat]SeatShape, fam Family, prior Prior) []Crew { - scale := maxima(candidates) - t := readTally(candidates) - readings := make([]reading, 0, len(candidates)) - for _, c := range candidates { - readings = append(readings, fill(c, t, scale)) - } - workers := shortlist(readings, shapes[Worker], fam, scale, Worker, prior) - highs := shortlist(readings, shapes[High], fam, scale, High, prior) - minds := shortlist(readings, shapes[Mastermind], fam, scale, Mastermind, prior) - vw := shapes[Worker].Volume - vh := shapes[High].Volume - vm := shapes[Mastermind].Volume - - crews := make([]Crew, 0, len(workers)*len(highs)*len(minds)) - for _, w := range workers { - for _, h := range highs { - if vendor(w.c.ID) == vendor(h.c.ID) { - continue - } - for _, m := range minds { - crews = append(crews, Crew{ - Worker: w.c.ID, - High: h.c.ID, - Mastermind: m.c.ID, - Bill: vw*w.cost + vh*h.cost + vm*m.cost, - Quality: (w.quality + h.quality + m.quality) / 3, - Estimated: [3][3]bool{w.est, h.est, m.est}, - Measured: [3]bool{w.measured, h.measured, m.measured}, - }) - } - } - } - - // The front: cheapest bill first, best quality first within a bill, and - // a crew survives when its quality beats everything cheaper so far. - sort.SliceStable(crews, func(i, j int) bool { - if crews[i].Bill != crews[j].Bill { - return crews[i].Bill < crews[j].Bill - } - if crews[i].Quality != crews[j].Quality { - return crews[i].Quality > crews[j].Quality - } - return estimateCount(crews[i]) < estimateCount(crews[j]) - }) - var front []Crew - best := math.Inf(-1) - for _, crew := range crews { - if crew.Quality > best+1e-9 { - front = append(front, crew) - best = crew.Quality - } - } - return front -} - -// Presets reads the three named picks off a front: frugal is the cheapest -// crew, max the dearest, and balanced the knee between them — the crew -// farthest above the straight line from one end of the front to the other in -// (ln bill, quality), which is where money stops buying quality in earnest. -// A front of one crew is all three; an empty front is none. -func Presets(front []Crew) (frugal, balanced, max Crew) { - if len(front) == 0 { - return Crew{}, Crew{}, Crew{} - } - frugal, balanced, max = front[0], knee(front), front[len(front)-1] - return frugal, balanced, max -} - -// AtKnob reads one pick off the front at knob k in [0, 1]: the budget runs -// geometrically from the frugal bill to the max bill — lo*(hi/lo)^k — and -// the pick is the best-quality crew whose bill fits it, so k of 0 is frugal -// and k of 1 is max. An empty front picks nothing. -func AtKnob(front []Crew, k float64) Crew { - if len(front) == 0 { - return Crew{} - } - lo, hi := front[0].Bill, front[len(front)-1].Bill - budget := lo * math.Pow(hi/lo, k) - var pick Crew - for _, crew := range front { - if crew.Bill <= budget*(1+1e-9) { - pick = crew - } - } - return pick -} - -// knee returns the front's knee: the crew farthest above the straight line -// from the first crew to the last in (ln bill, quality). Ties go to the -// earlier crew, and a front whose two ends share a bill has no line to be -// above — its best crew carries the name. -func knee(front []Crew) Crew { - x0, y0 := math.Log(front[0].Bill), front[0].Quality - x1, y1 := math.Log(front[len(front)-1].Bill), front[len(front)-1].Quality - if x1 == x0 { - return front[len(front)-1] - } - pick := front[0] - best := math.Inf(-1) - for _, crew := range front { - d := (crew.Quality - y0) - (y1-y0)*(math.Log(crew.Bill)-x0)/(x1-x0) - if d > best { - pick, best = crew, d - } - } - return pick -} - -// A seatPick is a candidate already priced and scored for one seat, on the -// candidate's reading: measured and estimated indexes alike. measured says -// whether the pool's own rating for the seat entered the quality. -type seatPick struct { - c Candidate - idx [3]float64 - est [3]bool - cost float64 - quality float64 - measured bool -} - -// shortlist returns a seat's running: the candidates that pass family and the -// seat's needs, kept when their quality reaches Floor of the family's best -// for the seat, in id order. The seat's quality is read with the pool's prior -// blended in, and measured marks the picks the prior touched. -func shortlist(readings []reading, shape SeatShape, fam Family, scale [3]float64, seat Seat, prior Prior) []seatPick { - var running []seatPick - top := 0.0 - for _, r := range readings { - if !eligible(r.c, shape, fam) { - continue - } - q, measured := blendQuality(r.c.ID, quality(r.idx, shape, scale), seat, prior) - pick := seatPick{c: r.c, idx: r.idx, est: r.est, cost: SeatCost(r.c, shape), quality: q, measured: measured} - running = append(running, pick) - top = math.Max(top, pick.quality) - } - var kept []seatPick - for _, pick := range running { - if pick.quality >= Floor*top { - kept = append(kept, pick) - } - } - sort.Slice(kept, func(i, j int) bool { return kept[i].c.ID < kept[j].c.ID }) - return kept -} - -// quality is SeatQuality against pre-read maxima, on a candidate's reading. -func quality(idx [3]float64, shape SeatShape, scale [3]float64) float64 { - if scale[Intelligence] <= 0 || scale[Coding] <= 0 || scale[Agentic] <= 0 { - return 0 - } - w := shape.Weights - return 100 * (w[Intelligence]*idx[Intelligence]/scale[Intelligence] + - w[Coding]*idx[Coding]/scale[Coding] + - w[Agentic]*idx[Agentic]/scale[Agentic]) -} - -// blendQuality blends the pool's prior for one seat into the catalog quality: -// a rating the prior holds for the model on the seat, with a positive -// observation count, carries weight N/(N+PriorWeightAt) against the catalog's -// own quality, and the second result reports whether the rating entered. A -// seat the prior never rates, or rates with no observations, keeps the -// catalog quality and reads measured false. -func blendQuality(model string, catalog float64, seat Seat, prior Prior) (float64, bool) { - if prior == nil { - return catalog, false - } - rating, ok := prior[seat][model] - if !ok || rating.N <= 0 { - return catalog, false - } - w := float64(rating.N) / float64(rating.N+PriorWeightAt) - return (1-w)*catalog + w*rating.Mean, true -} - -// maxima reads the pool's largest measured value of each index — the scale -// quality is measured against and the ceiling an estimate never crosses. -// Only real candidates raise a ceiling, and a row that publishes only some -// of its indexes raises only those. -func maxima(pool []Candidate) (scale [3]float64) { - for _, c := range pool { - if !isCandidate(c) { - continue - } - v := measured(c) - for i, val := range v { - scale[i] = math.Max(scale[i], val) - } - } - return scale -} - -// measured is the candidate's three indexes as it published them: an -// unpublished index reads as zero. -func measured(c Candidate) [3]float64 { - return [3]float64{c.Intelligence, c.Coding, c.Agentic} -} - -// A tally is one call's donor statistics: for every ordered pair of indexes, -// how many candidates — the donors — carry both measured, and the median of -// the first over the second among them. A missing index is estimated from -// these. -type tally struct { - count [3][3]int - median [3][3]float64 -} - -// readTally reads the pool's tally; the donors of a pair of indexes are the -// candidates carrying both measured. -func readTally(pool []Candidate) tally { - var ratios [3][3][]float64 - for _, c := range pool { - if !isCandidate(c) { - continue - } - v := measured(c) - for _, x := range [3]Index{Intelligence, Coding, Agentic} { - for _, y := range [3]Index{Intelligence, Coding, Agentic} { - if x != y && v[x] > 0 && v[y] > 0 { - ratios[x][y] = append(ratios[x][y], v[x]/v[y]) - } - } - } - } - var t tally - for _, x := range [3]Index{Intelligence, Coding, Agentic} { - for _, y := range [3]Index{Intelligence, Coding, Agentic} { - t.count[x][y] = len(ratios[x][y]) - if len(ratios[x][y]) >= 3 { - t.median[x][y] = medianOf(ratios[x][y]) - } - } - } - return t -} - -// medianOf is the middle of the values in order, or the mean of the two -// middles when there is no single middle. -func medianOf(values []float64) float64 { - slices.Sort(values) - n := len(values) - if n%2 == 1 { - return values[n/2] - } - return (values[n/2-1] + values[n/2]) / 2 -} - -// A reading is a candidate's three indexes ready for scoring — the measured -// ones as published, the missing ones estimated from the call's donors — -// with est remembering which of the three were estimated rather than -// measured. -type reading struct { - c Candidate - idx [3]float64 - est [3]bool -} - -// fill reads the candidate's three indexes for scoring. A measured index is -// taken as published; a missing one is estimated from the ones present: each -// present index Y offers Y times the tally's median of the missing index over -// Y among the donors carrying both — or, when fewer than three donors carry -// the pair, equal standing, the missing index read where its present one -// stands, X over the pool's largest X equal to Y over the pool's largest Y — -// and the mean of the offers is kept, never above the pool's largest measured -// value of the missing index. A row that is no candidate at all fills to -// zeros. -func fill(c Candidate, t tally, scale [3]float64) reading { - var r reading - if !isCandidate(c) { - return r - } - r.c = c - v := measured(c) - r.idx = v - for _, x := range [3]Index{Intelligence, Coding, Agentic} { - if v[x] > 0 { - continue - } - var offers []float64 - for _, y := range [3]Index{Intelligence, Coding, Agentic} { - if y == x || v[y] <= 0 { - continue - } - if t.count[x][y] >= 3 { - offers = append(offers, v[y]*t.median[x][y]) - } else { - offers = append(offers, v[y]/scale[y]*scale[x]) - } - } - sum := 0.0 - for _, offer := range offers { - sum += offer - } - r.idx[x] = math.Min(sum/float64(len(offers)), scale[x]) - r.est[x] = true - } - return r -} - -// estimateCount reads how many of a crew's picks were scored on estimated -// indexes — the tiebreak between crews that tie on bill and quality. -func estimateCount(c Crew) int { - n := 0 - for _, seat := range c.Estimated { - for _, est := range seat { - if est { - n++ - } - } - } - return n -} - -// isCandidate reports whether a row is a candidate at all: at least one of -// the three indexes present and a prompt price above zero. Everything -// downstream — eligibility, the floor, the maxima — starts from this. -func isCandidate(c Candidate) bool { - return (c.Intelligence > 0 || c.Coding > 0 || c.Agentic > 0) && c.PromptPrice > 0 -} - -// eligible reports whether a candidate may sit the seat: family first, then -// what the seat needs the model to take and call, then the window it needs. -func eligible(c Candidate, shape SeatShape, fam Family) bool { - if !isCandidate(c) { - return false - } - if fam == Open && !c.Open { - return false - } - if shape.NeedTools && !c.Tools { - return false - } - if shape.NeedImages && !c.Images { - return false - } - return c.Context >= shape.MinContext -} - -// vendor is the id text before the first slash, or the whole id when there is -// no slash. -func vendor(id string) string { - if i := strings.Index(id, "/"); i >= 0 { - return id[:i] - } - return id -} diff --git a/internal/crewpick/crewpick_test.go b/internal/crewpick/crewpick_test.go deleted file mode 100644 index f884530002..0000000000 --- a/internal/crewpick/crewpick_test.go +++ /dev/null @@ -1,833 +0,0 @@ -package crewpick - -import ( - "encoding/json" - "fmt" - "math" - "os" - "slices" - "testing" -) - -// A fixture row in its own field names: a null index reads as zero — -// missing, the way the package reads it — and a null cache-read price reads -// as unpublished. The partially published row and the rows with unpublished -// cache prices stay in the fixture, so the estimate and exclusion rules -// have something to bite on. -type row struct { - ID string `json:"id"` - Open bool `json:"open"` - I float64 `json:"I"` - C float64 `json:"C"` - A float64 `json:"A"` - Pi float64 `json:"pi"` - Po float64 `json:"po"` - Cr *float64 `json:"cr"` - Ctx int `json:"ctx"` - Img bool `json:"img"` - Tools bool `json:"tools"` -} - -// loadCandidates reads the fixture into candidates. -func loadCandidates(t *testing.T) []Candidate { - t.Helper() - raw, err := os.ReadFile("testdata/candidates.json") - if err != nil { - t.Fatal(err) - } - var rows []row - if err := json.Unmarshal(raw, &rows); err != nil { - t.Fatal(err) - } - cands := make([]Candidate, 0, len(rows)) - for _, r := range rows { - c := Candidate{ - ID: r.ID, - Open: r.Open, - Intelligence: r.I, - Coding: r.C, - Agentic: r.A, - PromptPrice: r.Pi, - CompletionPrice: r.Po, - Context: r.Ctx, - Images: r.Img, - Tools: r.Tools, - } - if r.Cr != nil { - c.CacheReadPrice = *r.Cr - c.HasCacheRead = true - } - cands = append(cands, c) - } - return cands -} - -// model builds a candidate that can sit any seat: every index at 100, tools -// and images, a 200k window, and a published cache price. -func model(id string) Candidate { - return Candidate{ - ID: id, - Open: true, - Intelligence: 100, - Coding: 100, - Agentic: 100, - PromptPrice: 1, - CompletionPrice: 4, - CacheReadPrice: 0.1, - HasCacheRead: true, - Context: 200_000, - Images: true, - Tools: true, - } -} - -// donors builds n fully measured candidates that can sit no seat — a window -// too small for any of them — so they feed the estimates and the ceilings -// and nothing else. -func donors(prefix string, n int) []Candidate { - rows := make([]Candidate, 0, n) - for i := range n { - d := model(fmt.Sprintf("%s/d%d", prefix, i)) - d.Context = 1000 - rows = append(rows, d) - } - return rows -} - -// wantCrew checks a pick against its worker|high|mastermind line. -func wantCrew(t *testing.T, name string, got Crew, want string) { - t.Helper() - line := got.Worker + "|" + got.High + "|" + got.Mastermind - if line != want { - t.Fatalf("%s = %s, want %s", name, line, want) - } -} - -// The fixture's front is a fixed property of its rows: seven crews in the -// open family, eighteen in all, and these are the picks each front reads. -// The all-family balanced crew is where money stops buying quality — a bill -// of 0.590 at quality 95.4. -func TestPresetsReadTheFixtureFront(t *testing.T) { - cands := loadCandidates(t) - shapes := DefaultShapes() - - open := Front(cands, shapes, Open) - if len(open) != 7 { - t.Fatalf("open front holds %d crews, want 7", len(open)) - } - frugal, balanced, max := Presets(open) - wantCrew(t, "open frugal", frugal, "deepseek/deepseek-v4-flash-0731|z-ai/glm-5.3-flash|z-ai/glm-5.3-flash") - wantCrew(t, "open balanced", balanced, "z-ai/glm-5.3-flash|moonshotai/kimi-k3|z-ai/glm-5.3") - wantCrew(t, "open max", max, "z-ai/glm-5.3|moonshotai/kimi-k3|z-ai/glm-5.3") - - all := Front(cands, shapes, All) - if len(all) != 18 { - t.Fatalf("all front holds %d crews, want 18", len(all)) - } - frugal, balanced, max = Presets(all) - wantCrew(t, "all frugal", frugal, "z-ai/glm-5.3-flash|google/gemini-3.8-flash|z-ai/glm-5.3-flash") - wantCrew(t, "all balanced", balanced, "z-ai/glm-5.3-flash|anthropic/claude-fable-5.1|anthropic/claude-fable-5.1") - wantCrew(t, "all max", max, "anthropic/claude-fable-5.1|openai/gpt-6-astra|anthropic/claude-fable-5.1") - if math.Abs(balanced.Bill-0.590) > 0.005 { - t.Fatalf("all balanced bill = %f, want about 0.590", balanced.Bill) - } - if math.Abs(balanced.Quality-95.4) > 0.05 { - t.Fatalf("all balanced quality = %f, want about 95.4", balanced.Quality) - } -} - -// The high seat sees the same work the worker does, so a crew never draws it -// from the worker's own vendor. Every crew on both fixture fronts holds -// that line, and a pool that cannot field two vendors fields no crew at all. -func TestTheHighSeatNeverSharesAVendorWithTheWorker(t *testing.T) { - cands := loadCandidates(t) - for _, fam := range []Family{Open, All} { - for _, crew := range Front(cands, DefaultShapes(), fam) { - if vendor(crew.Worker) == vendor(crew.High) { - t.Fatalf("worker %s and high %s share a vendor", crew.Worker, crew.High) - } - } - } - - oneVendor := []Candidate{model("one/first"), model("one/second")} - if front := Front(oneVendor, DefaultShapes(), All); len(front) != 0 { - t.Fatalf("a one-vendor pool fielded %d crews, want none", len(front)) - } -} - -// The floor keeps a seat's running to candidates that can do the job: a -// model whose quality falls below Floor of the seat's best in the family is -// not cheap and never reaches a crew. The boundary is inclusive — a -// candidate sitting exactly on the floor runs, and at a tenth of the price -// of anything above it, it runs first. -func TestTheFloorDropsAModelThatCannotDoTheJob(t *testing.T) { - pool := []Candidate{model("a/m"), model("c/h")} - - // Cheap enough to be every crew's frugal worker, and short one index: - // 70 of the worker seat's 100, 76 of the mastermind's, and ineligible - // for the high seat, which needs images. - cheap := model("b/w") - cheap.Agentic = 40 - cheap.Images = false - cheap.PromptPrice = 0.01 - cheap.CompletionPrice = 0.02 - cheap.HasCacheRead = false - pool = append(pool, cheap) - - // Exactly on the worker seat's floor: 80 of 100. - edge := model("e/b") - edge.Agentic = 60 - edge.PromptPrice = 0.02 - edge.CompletionPrice = 0.04 - edge.HasCacheRead = false - pool = append(pool, edge) - - front := Front(pool, DefaultShapes(), All) - if len(front) == 0 { - t.Fatal("the pool must field crews") - } - frugal, _, _ := Presets(front) - if frugal.Worker != "e/b" { - t.Fatalf("frugal worker = %s, want the on-floor candidate e/b", frugal.Worker) - } - for _, crew := range front { - for _, id := range []string{crew.Worker, crew.High, crew.Mastermind} { - if id == "b/w" { - t.Fatalf("the below-floor model b/w reached a crew: %+v", crew) - } - } - } -} - -// A row carrying none of its three indexes, or priced at zero, is not a -// candidate at all: it never reaches a seat and it raises no ceiling. Both -// rules bite where the floor cannot — a free model would clear any floor, -// and a bare row carries no index to estimate from. -func TestARowWithNoIndexesAtAllOrNoPriceIsNoCandidate(t *testing.T) { - good := []Candidate{model("a/m"), model("c/h")} - good[1].Intelligence = 90 - good[1].PromptPrice = 2 - good[1].CompletionPrice = 8 - good[1].CacheReadPrice = 0.2 - - bare := model("d/x") - bare.Intelligence = 0 - bare.Coding = 0 - bare.Agentic = 0 - bare.PromptPrice = 0.01 - bare.CompletionPrice = 0.02 - bare.HasCacheRead = false - - free := model("e/y") - free.PromptPrice = 0 - - pool := append(slices.Clone(good), bare, free) - front := Front(pool, DefaultShapes(), All) - for _, crew := range front { - for _, id := range []string{crew.Worker, crew.High, crew.Mastermind} { - if id == "d/x" || id == "e/y" { - t.Fatalf("a non-candidate row reached a crew: %+v", crew) - } - } - } - // The non-candidate rows leave the real candidates' front exactly what - // it was without them. - if want := Front(good, DefaultShapes(), All); !slices.Equal(front, want) { - t.Fatalf("the non-candidate rows moved the front:\n got %+v\nwant %+v", front, want) - } -} - -// A model publishing only its intelligence index — a release scored in -// stages — is scored on it, with the coding and agentic indexes estimated -// from the donors, and at a hundredth of the price it takes the worker seat. -func TestAModelWithOnlyAnIntelligenceIndexIsChosen(t *testing.T) { - pool := []Candidate{model("a/w"), model("c/h")} - newRelease := model("b/x") - newRelease.Coding = 0 - newRelease.Agentic = 0 - newRelease.PromptPrice = 0.01 - newRelease.CompletionPrice = 0.02 - newRelease.HasCacheRead = false - pool = append(pool, newRelease) - pool = append(pool, donors("v", 3)...) - - front := Front(pool, DefaultShapes(), All) - if len(front) == 0 { - t.Fatal("the pool must field crews") - } - frugal, _, _ := Presets(front) - if frugal.Worker != "b/x" { - t.Fatalf("frugal worker = %s, want the partially scored b/x", frugal.Worker) - } - want := [3][3]bool{{false, true, true}, {}, {false, true, true}} - if frugal.Estimated != want { - t.Fatalf("the b/x crew's Estimated = %v, want %v", frugal.Estimated, want) - } -} - -// A model publishing only its coding index — many an older model — is -// scored on it, with the intelligence and agentic indexes estimated, and at -// a hundredth of the price it takes the worker seat too. -func TestAModelWithOnlyACodingIndexIsChosen(t *testing.T) { - pool := []Candidate{model("a/w"), model("c/h")} - older := model("b/x") - older.Intelligence = 0 - older.Agentic = 0 - older.PromptPrice = 0.01 - older.CompletionPrice = 0.02 - older.HasCacheRead = false - pool = append(pool, older) - pool = append(pool, donors("v", 3)...) - - front := Front(pool, DefaultShapes(), All) - if len(front) == 0 { - t.Fatal("the pool must field crews") - } - frugal, _, _ := Presets(front) - if frugal.Worker != "b/x" { - t.Fatalf("frugal worker = %s, want the partially scored b/x", frugal.Worker) - } - want := [3][3]bool{{true, false, true}, {}, {true, false, true}} - if frugal.Estimated != want { - t.Fatalf("the b/x crew's Estimated = %v, want %v", frugal.Estimated, want) - } -} - -// The donor estimate is the median of the missing index over the present -// one among the donors carrying both — not the mean, which the far-out -// fourth donor drags away — and the rows missing either index of the pair -// never donate. The row missing its intelligence index still raises the -// coding ceiling with its own 400. -func TestTheDonorEstimateIsTheMedianRatio(t *testing.T) { - shape := SeatShape{Weights: [3]float64{0, 1, 0}} - pool := []Candidate{ - {ID: "a/d1", Intelligence: 10, Coding: 10, Agentic: 10, PromptPrice: 1}, - {ID: "a/d2", Intelligence: 10, Coding: 20, Agentic: 10, PromptPrice: 1}, - {ID: "a/d3", Intelligence: 10, Coding: 30, Agentic: 10, PromptPrice: 1}, - {ID: "a/d4", Intelligence: 10, Coding: 300, Agentic: 10, PromptPrice: 1}, - {ID: "b/noc", Intelligence: 10, Agentic: 10, PromptPrice: 1}, - {ID: "b/noi", Coding: 400, Agentic: 10, PromptPrice: 1}, - } - x := Candidate{ID: "c/x", Intelligence: 10, PromptPrice: 1} - - // The donors' coding over intelligence ratios run 1, 2, 3, 30: the - // median reads 2.5 and the estimate 25 of a 400 ceiling, where the mean - // would read 9 and the estimate 90. - if got := SeatQuality(x, shape, pool); got != 6.25 { - t.Fatalf("SeatQuality on the median estimate = %v, want 6.25", got) - } -} - -// With fewer than three donors carrying a pair, the estimate falls back to -// equal standing — the missing index read at the present one's share of -// the pool's ceilings — not to the median of two donors, and not to -// exclusion. -func TestFewerThanThreeDonorsEstimatesByEqualStanding(t *testing.T) { - shape := SeatShape{Weights: [3]float64{0, 1, 0}} - pool := []Candidate{ - {ID: "a/d1", Intelligence: 100, Coding: 50, Agentic: 10, PromptPrice: 1}, - {ID: "a/d2", Intelligence: 50, Coding: 100, Agentic: 10, PromptPrice: 1}, - } - x := Candidate{ID: "b/x", Intelligence: 80, PromptPrice: 1} - - // Equal standing reads x's coding at x's share of the intelligence - // ceiling, 80 of 100; the two donors' median would read 100. - if got := SeatQuality(x, shape, pool); got != 80 { - t.Fatalf("SeatQuality on equal standing = %v, want 80", got) - } -} - -// An estimate never crosses the pool's largest measured value of its -// index: the donors' coding over intelligence median runs 20, and 50 times -// that would read 1000 of a 300 ceiling. -func TestAnEstimateIsCappedAtTheLargestMeasuredIndex(t *testing.T) { - shape := SeatShape{Weights: [3]float64{0, 1, 0}} - pool := []Candidate{ - {ID: "a/d1", Intelligence: 10, Coding: 100, Agentic: 10, PromptPrice: 1}, - {ID: "a/d2", Intelligence: 10, Coding: 200, Agentic: 10, PromptPrice: 1}, - {ID: "a/d3", Intelligence: 10, Coding: 300, Agentic: 10, PromptPrice: 1}, - } - x := Candidate{ID: "b/x", Intelligence: 50, PromptPrice: 1} - - if got := SeatQuality(x, shape, pool); got != 100 { - t.Fatalf("SeatQuality on the capped estimate = %v, want 100", got) - } -} - -// The estimate is never silent: the crew says, for the model in each seat, -// which of its three indexes were scored by estimate and which measured. -// The worker publishes only its intelligence index and the mastermind only -// its coding one; the high seat is measured throughout. -func TestTheResultMarksEstimatedIndexes(t *testing.T) { - pool := []Candidate{model("a/w"), model("c/h")} - worker := model("b/x") - worker.Coding = 0 - worker.Agentic = 0 - worker.Images = false - worker.PromptPrice = 0.01 - worker.CompletionPrice = 0.02 - worker.HasCacheRead = false - mind := model("d/m") - mind.Intelligence = 0 - mind.Agentic = 0 - mind.Tools = false - mind.PromptPrice = 0.005 - mind.CompletionPrice = 0.01 - mind.HasCacheRead = false - pool = append(pool, worker, mind) - pool = append(pool, donors("v", 3)...) - - front := Front(pool, DefaultShapes(), All) - if len(front) == 0 { - t.Fatal("the pool must field crews") - } - frugal, _, _ := Presets(front) - if frugal.Worker != "b/x" || frugal.Mastermind != "d/m" { - t.Fatalf("frugal = %s|%s|%s, want b/x on the worker seat and d/m on the mastermind's", frugal.Worker, frugal.High, frugal.Mastermind) - } - want := [3][3]bool{{false, true, true}, {}, {true, false, true}} - if frugal.Estimated != want { - t.Fatalf("the crew's Estimated = %v, want %v", frugal.Estimated, want) - } -} - -// Two candidates for a seat otherwise tied on quality and cost run to the -// one with fewer estimated indexes — even when its id reads later, and -// whichever way round the pool arrives. -func TestATieOnQualityAndCostRunsToFewerEstimatedIndexes(t *testing.T) { - pool := []Candidate{model("a/w"), model("c/h")} - // Twins for the mastermind seat, identical in every number the picker - // reads, but aaa runs on two estimated indexes and zzz on none. - aaa := model("aaa/two") - aaa.Intelligence = 0 - aaa.Coding = 80 - aaa.Agentic = 0 - aaa.Tools = false - aaa.Images = false - aaa.PromptPrice = 0.1 - aaa.CompletionPrice = 0.2 - aaa.HasCacheRead = false - zzz := aaa - zzz.ID = "zzz/one" - zzz.Intelligence = 80 - zzz.Agentic = 80 - pool = append(pool, aaa, zzz) - pool = append(pool, donors("v", 3)...) - - front := Front(pool, DefaultShapes(), All) - if len(front) == 0 { - t.Fatal("the pool must field crews") - } - if front[0].Mastermind != "zzz/one" { - t.Fatalf("a tie went to the estimated twin: %+v", front[0]) - } - for _, crew := range front { - if crew.Mastermind == "aaa/two" { - t.Fatalf("the estimated twin survived its tie: %+v", crew) - } - } - - reversed := slices.Clone(pool) - slices.Reverse(reversed) - if two := Front(reversed, DefaultShapes(), All); !slices.Equal(front, two) { - t.Fatal("the tie broke differently when the pool arrived reversed") - } -} - -// Estimates, caps and ties read the same whatever order the candidates -// arrive in: a pool of partially published models gives the same front -// shuffled as it does in hand order. -func TestPartialCandidatesGiveTheSameFrontInAnyOrder(t *testing.T) { - newRelease := model("b/new") - newRelease.Coding = 0 - newRelease.Agentic = 0 - newRelease.PromptPrice = 0.01 - newRelease.CompletionPrice = 0.02 - newRelease.HasCacheRead = false - older := model("d/old") - older.Intelligence = 0 - older.Agentic = 0 - older.Tools = false - staged := model("e/staged") - staged.Agentic = 0 - pool := []Candidate{model("a/w"), model("c/h"), newRelease, older, staged} - pool = append(pool, donors("v", 3)...) - - one := Front(pool, DefaultShapes(), All) - if len(one) == 0 { - t.Fatal("the pool must field crews") - } - if one[0].Estimated == ([3][3]bool{}) { - t.Fatal("the cheapest crew carries no estimated index; the partial candidates never reached it") - } - reversed := slices.Clone(pool) - slices.Reverse(reversed) - if two := Front(reversed, DefaultShapes(), All); !slices.Equal(one, two) { - t.Fatal("the front changed when the partial candidates arrived reversed") - } - rotated := append(slices.Clone(pool[3:]), pool[:3]...) - if three := Front(rotated, DefaultShapes(), All); !slices.Equal(one, three) { - t.Fatal("the front changed when the pool arrived rotated") - } -} - -// An input with no index missing picks on measurements alone: the fixture's -// partially published row left out, both fronts hold the same sizes and -// the same frugal, balanced and max picks the whole fixture holds, and no -// estimated flag is set anywhere — the partial row never raised a ceiling, -// so leaving it out moves nothing measured. -func TestNothingMissingLeavesEveryPickUnchanged(t *testing.T) { - cands := loadCandidates(t) - var full []Candidate - for _, c := range cands { - if c.Intelligence > 0 && c.Coding > 0 && c.Agentic > 0 { - full = append(full, c) - } - } - shapes := DefaultShapes() - - open := Front(full, shapes, Open) - if len(open) != 7 { - t.Fatalf("the full open front holds %d crews, want 7", len(open)) - } - frugal, balanced, max := Presets(open) - wantCrew(t, "open frugal", frugal, "deepseek/deepseek-v4-flash-0731|z-ai/glm-5.3-flash|z-ai/glm-5.3-flash") - wantCrew(t, "open balanced", balanced, "z-ai/glm-5.3-flash|moonshotai/kimi-k3|z-ai/glm-5.3") - wantCrew(t, "open max", max, "z-ai/glm-5.3|moonshotai/kimi-k3|z-ai/glm-5.3") - - all := Front(full, shapes, All) - if len(all) != 18 { - t.Fatalf("the full all front holds %d crews, want 18", len(all)) - } - frugal, balanced, max = Presets(all) - wantCrew(t, "all frugal", frugal, "z-ai/glm-5.3-flash|google/gemini-3.8-flash|z-ai/glm-5.3-flash") - wantCrew(t, "all balanced", balanced, "z-ai/glm-5.3-flash|anthropic/claude-fable-5.1|anthropic/claude-fable-5.1") - wantCrew(t, "all max", max, "anthropic/claude-fable-5.1|openai/gpt-6-astra|anthropic/claude-fable-5.1") - if math.Abs(balanced.Bill-0.590) > 0.005 { - t.Fatalf("all balanced bill = %f, want about 0.590", balanced.Bill) - } - if math.Abs(balanced.Quality-95.4) > 0.05 { - t.Fatalf("all balanced quality = %f, want about 95.4", balanced.Quality) - } - for _, front := range [][]Crew{open, all} { - for _, crew := range front { - if crew.Estimated != ([3][3]bool{}) { - t.Fatalf("a fully measured pool raised an estimated flag: %+v", crew) - } - } - } -} - -// The knob runs from one end of the front to the other: at 0 the budget is -// the cheapest bill and the pick is frugal, at 1 it is the dearest and the -// pick is max. -func TestAtKnobAtTheEndsReadsFrugalAndMax(t *testing.T) { - cands := loadCandidates(t) - for _, fam := range []Family{Open, All} { - front := Front(cands, DefaultShapes(), fam) - frugal, _, max := Presets(front) - if got := AtKnob(front, 0); got != frugal { - t.Fatalf("AtKnob at 0 = %+v, want the frugal %+v", got, frugal) - } - if got := AtKnob(front, 1); got != max { - t.Fatalf("AtKnob at 1 = %+v, want the max %+v", got, max) - } - } -} - -// The front is a property of the candidates, not of their order: the running -// for each seat is read in id order, so the same list gives the same front -// however it arrives. -func TestTheSameCandidatesGiveTheSameFrontInAnyOrder(t *testing.T) { - cands := loadCandidates(t) - shapes := DefaultShapes() - one := Front(cands, shapes, All) - - reversed := slices.Clone(cands) - slices.Reverse(reversed) - if two := Front(reversed, shapes, All); !slices.Equal(one, two) { - t.Fatal("the front changed when the candidates arrived reversed") - } -} - -// Two crews that tie in (bill, quality) are one point on the front, and the -// point keeps the crew whose ids read first — worker, then high, then -// mastermind — whichever way round the pool arrived. -func TestTiesInBillAndQualityBreakInIdOrder(t *testing.T) { - // Two candidates identical in every way but the id, and able to sit - // only the mastermind seat: every crew one finishes ties with the crew - // the other finishes. - aaa := model("aaa/x") - aaa.Tools = false - aaa.Images = false - aaa.PromptPrice = 0.1 - aaa.CompletionPrice = 0.2 - aaa.HasCacheRead = false - bbb := aaa - bbb.ID = "bbb/x" - - pool := []Candidate{model("a/m"), model("c/h"), aaa, bbb} - one := Front(pool, DefaultShapes(), All) - if len(one) == 0 { - t.Fatal("the pool must field crews") - } - for _, crew := range one { - if crew.Mastermind != "aaa/x" { - t.Fatalf("a tie went to the wrong twin: %+v", crew) - } - } - - reversed := slices.Clone(pool) - slices.Reverse(reversed) - if two := Front(reversed, DefaultShapes(), All); !slices.Equal(one, two) { - t.Fatal("the tie broke differently when the pool arrived reversed") - } -} - -// The seat's cost is its token mix priced: the prompt price blended with the -// cache-read price by the cache share, and the completion price spread over -// the input-to-output ratio. A model that publishes no cache-read price pays -// the prompt price on the cache share too. -func TestSeatCostPricesTheTokenMix(t *testing.T) { - shape := SeatShape{InOut: 2, CacheShare: 0.5} - published := Candidate{PromptPrice: 2, CompletionPrice: 8, CacheReadPrice: 0.5, HasCacheRead: true} - if got := SeatCost(published, shape); got != 5.25 { - t.Fatalf("SeatCost with a published cache price = %f, want 5.25", got) - } - unpublished := Candidate{PromptPrice: 2, CompletionPrice: 8} - if got := SeatCost(unpublished, shape); got != 6 { - t.Fatalf("SeatCost with no published cache price = %f, want 6", got) - } -} - -// Quality reads each index against the pool's best for that index, so the -// same candidate scores lower in a stronger pool. A row that is not a -// candidate scores nothing, and so does any row against a pool with no -// candidates to raise a scale. -func TestSeatQualityReadsAgainstThePoolsBest(t *testing.T) { - shape := SeatShape{Weights: [3]float64{0.25, 0.25, 0.5}} - pool := []Candidate{ - {ID: "best", Intelligence: 100, Coding: 100, Agentic: 100, PromptPrice: 1}, - {ID: "half", Intelligence: 50, Coding: 100, Agentic: 100, PromptPrice: 1}, - } - if got := SeatQuality(pool[0], shape, pool); got != 100 { - t.Fatalf("the pool's best scored %f, want 100", got) - } - if got := SeatQuality(pool[1], shape, pool); got != 87.5 { - t.Fatalf("the half scored %f, want 87.5", got) - } - - stronger := append(slices.Clone(pool), - Candidate{ID: "top", Intelligence: 200, Coding: 100, Agentic: 100, PromptPrice: 1}) - if got := SeatQuality(pool[1], shape, stronger); got != 81.25 { - t.Fatalf("the half scored %f against a stronger pool, want 81.25", got) - } - - blank := Candidate{ID: "blank", Coding: 0, PromptPrice: 1} - if got := SeatQuality(blank, shape, pool); got != 0 { - t.Fatalf("a row missing an index scored %f, want 0", got) - } - if got := SeatQuality(pool[0], shape, []Candidate{blank}); got != 0 { - t.Fatalf("a pool with no candidates scored %f, want 0", got) - } -} - -// A pool's own rating for a seat moves that seat's quality towards the -// rating's mean, and the more observations back it the farther it moves: a -// rating with a large count across a seat whose catalog quality is 100 and -// whose pool mean is 50 lands somewhere between the two. -func TestALargeRatingMovesASeatTowardsThePoolMean(t *testing.T) { - shape := SeatShape{Weights: [3]float64{1, 0, 0}} - pool := []Candidate{model("m")} - if base := SeatQuality(pool[0], shape, pool); base != 100 { - t.Fatalf("the catalog quality is %f, want 100", base) - } - prior := Prior{Worker: {"m": {Mean: 50, N: 1000}}} - if got := SeatQualityWith(pool[0], shape, pool, Worker, prior); got <= 50 || got >= 100 { - t.Fatalf("a large rating scored %f, want between the pool mean 50 and the catalog 100", got) - } -} - -// The blend carries the rating and the catalog at equal weight at exactly -// PriorWeightAt observations, so the seat reads the midpoint of the two; a -// seat the prior never rates, and a rating with no observations, keep the -// catalog quality. -func TestTheBlendAtTheRatingWeightIsTheMidpoint(t *testing.T) { - shape := SeatShape{Weights: [3]float64{1, 0, 0}} - pool := []Candidate{model("m")} - - half := Prior{Worker: {"m": {Mean: 40, N: PriorWeightAt}}} - if got := SeatQualityWith(pool[0], shape, pool, Worker, half); got != 70 { - t.Fatalf("the blend at N = PriorWeightAt scored %f, want the midpoint 70", got) - } - unrated := Prior{Worker: {"other": {Mean: 40, N: 1000}}} - if got := SeatQualityWith(pool[0], shape, pool, Worker, unrated); got != 100 { - t.Fatalf("a seat the prior never rated scored %f, want the catalog 100", got) - } - empty := Prior{Worker: {"m": {Mean: 40, N: 0}}} - if got := SeatQualityWith(pool[0], shape, pool, Worker, empty); got != 100 { - t.Fatalf("a rating with no observations scored %f, want the catalog 100", got) - } -} - -// A prior is read from role-worded cells: a role that names no seat and a -// count below min_installs are both ignored, and only a role that names a -// seat with enough observations behind it becomes a rating. -func TestPriorFromCellsIgnoresACountBelowMinInstalls(t *testing.T) { - prior := PriorFromCells([]Cell{ - {Role: "worker", Model: "a/kept", Mean: 80, N: 40}, - {Role: "worker", Model: "b/dropped", Mean: 90, N: 10}, - {Role: "engineer", Model: "c/ignored", Mean: 95, N: 100}, - }, 30, nil) - if len(prior) != 1 || len(prior[Worker]) != 1 { - t.Fatalf("prior = %v, want one worker rating", prior) - } - if _, ok := prior[Worker]["a/kept"]; !ok { - t.Fatalf("a rating at the floor was dropped: %v", prior) - } - if _, ok := prior[Worker]["b/dropped"]; ok { - t.Fatalf("a rating below min_installs survived: %v", prior) - } -} - -// The result says which seats a pool's rating entered: a seat whose model the -// prior rated reads measured, and a seat it did not — here the mastermind, -// which no rating names — does not. -func TestMeasuredIsSetOnlyOnTheSeatsThePriorTouched(t *testing.T) { - pool := []Candidate{model("a/w"), model("c/h"), model("b/m")} - prior := Prior{ - Worker: {"a/w": {Mean: 100, N: 100}}, - High: {"b/m": {Mean: 100, N: 100}}, - } - front := FrontWith(pool, DefaultShapes(), All, prior) - if len(front) == 0 { - t.Fatal("the pool must field crews") - } - wantCrew(t, "frugal", front[0], "a/w|b/m|a/w") - if front[0].Measured != ([3]bool{true, true, false}) { - t.Fatalf("Measured = %v, want the worker and high seats alone", front[0].Measured) - } -} - -// A prior changes quality alone: every bill a crew is given is its seats' -// token mixes priced, exactly as it is without a prior. -func TestAPriorNeverChangesTheBill(t *testing.T) { - pool := []Candidate{model("a/w"), model("c/h"), model("b/m")} - shapes := DefaultShapes() - prior := Prior{ - Worker: {"a/w": {Mean: 20, N: 10_000}}, - High: {"c/h": {Mean: 140, N: 10_000}}, - } - front := FrontWith(pool, shapes, All, prior) - if len(front) == 0 { - t.Fatal("the pool must field crews") - } - for _, crew := range front { - want := 0.0 - for _, seat := range []struct { - model string - seat Seat - }{{crew.Worker, Worker}, {crew.High, High}, {crew.Mastermind, Mastermind}} { - for _, c := range pool { - if c.ID == seat.model { - want += shapes[seat.seat].Volume * SeatCost(c, shapes[seat.seat]) - } - } - } - if math.Abs(crew.Bill-want) > 1e-9 { - t.Fatalf("crew %s|%s|%s bill = %f, want its seats' priced mix %f", - crew.Worker, crew.High, crew.Mastermind, crew.Bill, want) - } - } -} - -// A prior is a property of the candidates, not of their order: with a rating -// present the same pool gives the same front shuffled, rotated and reversed. -func TestAPriorKeepsTheFrontTheSameInAnyOrder(t *testing.T) { - cands := loadCandidates(t) - prior := Prior{Worker: {"z-ai/glm-5.3": {Mean: 90, N: 100}}} - one := FrontWith(cands, DefaultShapes(), All, prior) - if len(one) == 0 { - t.Fatal("the pool must field crews") - } - reversed := slices.Clone(cands) - slices.Reverse(reversed) - if two := FrontWith(reversed, DefaultShapes(), All, prior); !slices.Equal(one, two) { - t.Fatal("the front changed when the candidates arrived reversed") - } - rotated := append(slices.Clone(cands[3:]), cands[:3]...) - if three := FrontWith(rotated, DefaultShapes(), All, prior); !slices.Equal(one, three) { - t.Fatal("the front changed when the pool arrived rotated") - } -} - -// An alias resolved through canonical meets its candidate: a cell spelling a -// model's alias lands on the candidate's canonical id, so the rating reaches -// it, while the same cell with no canonical step lands on the alias and -// touches nothing. -func TestAnAliasResolvedThroughCanonicalMeetsItsCandidate(t *testing.T) { - shape := SeatShape{Weights: [3]float64{1, 0, 0}} - pool := []Candidate{model("z-ai/glm-5.3")} - canonical := func(id string) string { - if id == "glm-5.3" { - return "z-ai/glm-5.3" - } - return id - } - cell := Cell{Role: "worker", Model: "glm-5.3", Mean: 40, N: PriorWeightAt} - resolved := PriorFromCells([]Cell{cell}, 5, canonical) - if got := SeatQualityWith(pool[0], shape, pool, Worker, resolved); got != 70 { - t.Fatalf("the aliased rating scored %f, want the midpoint 70", got) - } - unresolved := PriorFromCells([]Cell{cell}, 5, nil) - if got := SeatQualityWith(pool[0], shape, pool, Worker, unresolved); got != 100 { - t.Fatalf("the unresolved alias scored %f, want the catalog 100 untouched", got) - } -} - -// Two priors fold into one: a rating both hold for a model is weighted by the -// counts behind it, a rating either holds alone is carried whole, and a nil on -// either side is the other side. -func TestMergePriorsFoldsTwoPriorsByObservationCount(t *testing.T) { - if got := MergePriors(nil, nil); got != nil { - t.Fatalf("two nils folded to %v, want nil", got) - } - a := Prior{Worker: {"a/one": {Mean: 80, N: 20}}} - if got := MergePriors(a, nil); got == nil || got[Worker]["a/one"].N != 20 { - t.Fatalf("a nil on one side is not the other: %v", got) - } - b := Prior{High: {"b/two": {Mean: 60, N: 5}}} - if got := MergePriors(nil, b); got == nil || got[High]["b/two"].N != 5 { - t.Fatalf("a nil on one side is not the other: %v", got) - } - - // A model only one prior names is carried whole, in either order. - onlyA := Prior{Worker: {"a/only": {Mean: 70, N: 4}}} - onlyB := Prior{Worker: {"b/only": {Mean: 90, N: 6}}} - got := MergePriors(onlyA, onlyB) - if len(got[Worker]) != 2 { - t.Fatalf("no shared key folded to %v, want the union", got) - } - if r := got[Worker]["a/only"]; r.Mean != 70 || r.N != 4 { - t.Fatalf("a carried rating moved: %v", r) - } - if r := got[Worker]["b/only"]; r.Mean != 90 || r.N != 6 { - t.Fatalf("a carried rating moved: %v", r) - } - - // A model both name: the counts add and the mean is the mean of the means - // weighted by them — (10*50 + 30*90) / 40 = 80. - sharedA := Prior{Worker: {"c/both": {Mean: 50, N: 10}}} - sharedB := Prior{Worker: {"c/both": {Mean: 90, N: 30}}} - if r := MergePriors(sharedA, sharedB)[Worker]["c/both"]; r.Mean != 80 || r.N != 40 { - t.Fatalf("a shared rating folded to %v, want mean 80 over 40", r) - } - - // The priors given are read and never changed. - if r := sharedA[Worker]["c/both"]; r.Mean != 50 || r.N != 10 { - t.Fatalf("the merge moved its first prior: %v", r) - } - if r := sharedB[Worker]["c/both"]; r.Mean != 90 || r.N != 30 { - t.Fatalf("the merge moved its second prior: %v", r) - } -} diff --git a/internal/crewpick/testdata/candidates.json b/internal/crewpick/testdata/candidates.json deleted file mode 100644 index 493a31a519..0000000000 --- a/internal/crewpick/testdata/candidates.json +++ /dev/null @@ -1 +0,0 @@ -[{"id": "inclusionai/ling-3.0-flash-vl", "open": true, "I": 25, "C": 57, "A": 30, "pi": 0.06, "po": 0.18, "cr": 0.012, "ctx": 131072, "img": true, "tools": true, "so": true, "created": 1789056114, "exp": null, "alias": null}, {"id": "deepseek/deepseek-v4.1-flash", "open": true, "I": 39.5, "C": null, "A": null, "pi": 0.3, "po": 1.2, "cr": 0.006, "ctx": 1048576, "img": true, "tools": true, "so": true, "created": 1789021285, "exp": null, "alias": null}, {"id": "openai/gpt-6-astra", "open": false, "I": 52.8, "C": 76.9, "A": 51.5, "pi": 10.0, "po": 50.0, "cr": 1.0, "ctx": 1050000, "img": true, "tools": true, "so": true, "created": 1788552838, "exp": null, "alias": null}, {"id": "qwen/qwen3.8-max-0902", "open": false, "I": 45.4, "C": 76.2, "A": 56.1, "pi": 2.0, "po": 6.0, "cr": 0.25, "ctx": 1000000, "img": true, "tools": true, "so": true, "created": 1788469704, "exp": null, "alias": null}, {"id": "google/gemini-3.8-flash", "open": false, "I": 41.2, "C": 76.3, "A": 41.1, "pi": 0.75, "po": 3.75, "cr": 0.075, "ctx": 1048576, "img": true, "tools": true, "so": true, "created": 1788362056, "exp": null, "alias": null}, {"id": "anthropic/claude-fable-5.1", "open": false, "I": 53.4, "C": 81.6, "A": 58, "pi": 10.0, "po": 50.0, "cr": 0.25, "ctx": 1000000, "img": true, "tools": true, "so": true, "created": 1788285838, "exp": null, "alias": null}, {"id": "ibm-granite/granite-4.2-8b", "open": true, "I": 11.8, "C": 22.4, "A": 3.7, "pi": 0.06, "po": 0.25, "cr": 0.015, "ctx": 131072, "img": false, "tools": true, "so": true, "created": 1788206780, "exp": null, "alias": null}, {"id": "inclusionai/ling-3.0-flash-fin", "open": false, "I": 23, "C": 55.6, "A": 29.3, "pi": 0.06, "po": 0.18, "cr": 0.012, "ctx": 262144, "img": false, "tools": true, "so": true, "created": 1787846290, "exp": null, "alias": null}, {"id": "z-ai/glm-5.3-flash", "open": true, "I": 41.9, "C": 71.5, "A": 51.2, "pi": 0.09, "po": 0.3, "cr": 0.018, "ctx": 1310720, "img": true, "tools": true, "so": true, "created": 1787752741, "exp": null, "alias": null}, {"id": "z-ai/glm-5.3", "open": true, "I": 44.9, "C": 74.8, "A": 53.4, "pi": 1.4, "po": 4.4, "cr": 0.26, "ctx": 1310720, "img": false, "tools": true, "so": true, "created": 1787086655, "exp": null, "alias": null}, {"id": "qwen/qwen3.8-27b", "open": true, "I": 33.9, "C": 68.1, "A": 46.5, "pi": 0.21400000000000002, "po": 2.5500000000000003, "cr": 0.15, "ctx": 1000000, "img": true, "tools": true, "so": true, "created": 1786722910, "exp": null, "alias": null}, {"id": "google/gemini-3.7-flash", "open": false, "I": 39.4, "C": 76.1, "A": 36.4, "pi": 0.75, "po": 3.75, "cr": 0.075, "ctx": 1048576, "img": true, "tools": true, "so": true, "created": 1786640581, "exp": null, "alias": null}, {"id": "qwen/qwen3.8-2.4t-a95b", "open": true, "I": 40, "C": 71.9, "A": 50.4, "pi": 2.0, "po": 6.0, "cr": 0.25, "ctx": 1048576, "img": false, "tools": true, "so": true, "created": 1786551702, "exp": null, "alias": null}, {"id": "deepseek/deepseek-v4-pro-0813", "open": true, "I": 36.3, "C": 68.8, "A": 42.3, "pi": 1.32, "po": 3.9600000000000004, "cr": 0.044, "ctx": 1048576, "img": false, "tools": true, "so": true, "created": 1786549364, "exp": null, "alias": null}, {"id": "x-ai/grok-4.6", "open": false, "I": 44.4, "C": 76.8, "A": 53.4, "pi": 2.0, "po": 6.0, "cr": 0.5, "ctx": 500000, "img": true, "tools": true, "so": true, "created": 1786548957, "exp": null, "alias": null}, {"id": "nvidia/nemotron-3.5-lightning", "open": true, "I": 13.6, "C": 26.8, "A": 6.1, "pi": 0.08, "po": 0.19999999999999998, "cr": 0.04, "ctx": 262144, "img": false, "tools": true, "so": true, "created": 1786452751, "exp": null, "alias": null}, {"id": "meta/muse-spark-1.2", "open": false, "I": 39.8, "C": 72.2, "A": 44, "pi": 1.25, "po": 4.25, "cr": 0.15, "ctx": 1048576, "img": true, "tools": true, "so": true, "created": 1785959287, "exp": null, "alias": null}, {"id": "deepseek/deepseek-v4-flash-0731", "open": true, "I": 34.5, "C": 69.1, "A": 41.7, "pi": 0.06, "po": 0.12, "cr": 0.012, "ctx": 1310720, "img": false, "tools": true, "so": true, "created": 1785478908, "exp": null, "alias": null}, {"id": "thinkingmachines/inkling-small", "open": true, "I": 26.1, "C": 52.9, "A": 25, "pi": 0.44999999999999996, "po": 1.2, "cr": 0.09999999999999999, "ctx": 1048576, "img": true, "tools": true, "so": false, "created": 1785443117, "exp": null, "alias": null}, {"id": "anthropic/claude-opus-5", "open": false, "I": 50.7, "C": 78, "A": 56.2, "pi": 5.0, "po": 25.0, "cr": 0.5, "ctx": 1000000, "img": true, "tools": true, "so": true, "created": 1784912544, "exp": null, "alias": null}, {"id": "inclusionai/ling-3.0-flash", "open": true, "I": 20.6, "C": 50.6, "A": 21, "pi": 0.020999999999999998, "po": 0.063, "cr": 0.004200000000000001, "ctx": 262144, "img": false, "tools": true, "so": false, "created": 1784818580, "exp": null, "alias": null}, {"id": "google/gemini-3.6-flash", "open": false, "I": 34.3, "C": 69.2, "A": 30.2, "pi": 0.75, "po": 3.75, "cr": 0.075, "ctx": 1048576, "img": true, "tools": true, "so": true, "created": 1784646733, "exp": null, "alias": null}, {"id": "google/gemini-3.5-flash-lite", "open": false, "I": 22.7, "C": 49.3, "A": 15.9, "pi": 0.3, "po": 2.5, "cr": 0.03, "ctx": 1048576, "img": true, "tools": true, "so": true, "created": 1784646726, "exp": null, "alias": null}, {"id": "meituan/longcat-2.0", "open": true, "I": 19.7, "C": 45.3, "A": 15.9, "pi": 0.3, "po": 1.2, "cr": 0.006, "ctx": 1048756, "img": false, "tools": true, "so": false, "created": 1784554658, "exp": null, "alias": null}, {"id": "thinkingmachines/inkling", "open": true, "I": 25.5, "C": 52.1, "A": 24.3, "pi": 1.0, "po": 4.05, "cr": 0.16999999999999998, "ctx": 1048576, "img": true, "tools": true, "so": false, "created": 1784325956, "exp": null, "alias": null}, {"id": "moonshotai/kimi-k3", "open": true, "I": 43.8, "C": 76.2, "A": 50.6, "pi": 3.0, "po": 15.0, "cr": 0.3, "ctx": 1048576, "img": true, "tools": true, "so": true, "created": 1784215858, "exp": null, "alias": null}, {"id": "meta/muse-spark-1.1", "open": false, "I": 34.3, "C": 71.3, "A": 27.5, "pi": 1.25, "po": 4.25, "cr": 0.15, "ctx": 1048576, "img": true, "tools": true, "so": true, "created": 1784215741, "exp": null, "alias": null}, {"id": "openai/gpt-5.6-luna", "open": false, "I": 37.5, "C": 71.4, "A": 42.7, "pi": 0.19999999999999998, "po": 1.2, "cr": 0.02, "ctx": 1050000, "img": true, "tools": true, "so": true, "created": 1783590864, "exp": null, "alias": null}, {"id": "openai/gpt-5.6-terra", "open": false, "I": 42.3, "C": 76.7, "A": 43.7, "pi": 2.0, "po": 12.0, "cr": 0.19999999999999998, "ctx": 1050000, "img": true, "tools": true, "so": true, "created": 1783590857, "exp": null, "alias": null}, {"id": "openai/gpt-5.6-sol", "open": false, "I": 47.1, "C": 77.4, "A": 50.5, "pi": 2.0, "po": 10.0, "cr": 0.19999999999999998, "ctx": 1050000, "img": true, "tools": true, "so": true, "created": 1783590850, "exp": null, "alias": null}, {"id": "x-ai/grok-4.5", "open": false, "I": 39.1, "C": 72.4, "A": 42.1, "pi": 2.0, "po": 6.0, "cr": 0.3, "ctx": 500000, "img": true, "tools": true, "so": true, "created": 1783523154, "exp": null, "alias": null}, {"id": "anthropic/claude-sonnet-5", "open": false, "I": 38.4, "C": 71.5, "A": 44.3, "pi": 2.0, "po": 10.0, "cr": 0.19999999999999998, "ctx": 1000000, "img": true, "tools": true, "so": true, "created": 1782843083, "exp": null, "alias": null}, {"id": "z-ai/glm-5.2", "open": true, "I": 34, "C": 68.8, "A": 39.4, "pi": 1.4, "po": 4.4, "cr": 0.14, "ctx": 1048576, "img": false, "tools": true, "so": true, "created": 1781631930, "exp": null, "alias": null}, {"id": "moonshotai/kimi-k2.7-code", "open": true, "I": 26.3, "C": 60.8, "A": 22.5, "pi": 0.7062, "po": 3.21, "cr": 0.18, "ctx": 262144, "img": true, "tools": true, "so": true, "created": 1781266361, "exp": null, "alias": null}, {"id": "anthropic/claude-fable-5", "open": false, "I": 49.7, "C": 76.5, "A": 51, "pi": 10.0, "po": 50.0, "cr": 1.0, "ctx": 1000000, "img": true, "tools": true, "so": true, "created": 1781007515, "exp": null, "alias": null}, {"id": "nvidia/nemotron-3-ultra-550b-a55b", "open": true, "I": 23.4, "C": 49.3, "A": 21.7, "pi": 0.625, "po": 3.125, "cr": 0.1875, "ctx": 262144, "img": false, "tools": true, "so": true, "created": 1780551208, "exp": null, "alias": null}, {"id": "qwen/qwen3.7-plus", "open": false, "I": 25.8, "C": 55.9, "A": 19.7, "pi": 0.32, "po": 1.28, "cr": 0.064, "ctx": 1000000, "img": true, "tools": true, "so": true, "created": 1780491783, "exp": null, "alias": null}, {"id": "minimax/minimax-m3", "open": true, "I": 29.6, "C": 58.6, "A": 30.8, "pi": 0.3, "po": 1.2, "cr": 0.06, "ctx": 1048576, "img": true, "tools": true, "so": true, "created": 1780245374, "exp": null, "alias": null}, {"id": "anthropic/claude-opus-4.8", "open": false, "I": 42, "C": 74.3, "A": 42.6, "pi": 5.0, "po": 25.0, "cr": 0.5, "ctx": 1000000, "img": true, "tools": true, "so": true, "created": 1779905091, "exp": null, "alias": null}, {"id": "qwen/qwen3.7-max", "open": false, "I": 29.9, "C": 66, "A": 23.9, "pi": 1.475, "po": 4.425, "cr": 0.295, "ctx": 1000000, "img": false, "tools": true, "so": true, "created": 1779376861, "exp": null, "alias": null}, {"id": "google/gemini-3.5-flash", "open": false, "I": 33, "C": 70.1, "A": 27.3, "pi": 1.5, "po": 9.0, "cr": 0.15, "ctx": 1048576, "img": true, "tools": true, "so": true, "created": 1779193800, "exp": null, "alias": null}, {"id": "x-ai/grok-4.3", "open": false, "I": 25.4, "C": 42.2, "A": 17.2, "pi": 1.25, "po": 2.5, "cr": 0.19999999999999998, "ctx": 1000000, "img": true, "tools": true, "so": true, "created": 1777591821, "exp": null, "alias": null}, {"id": "mistralai/mistral-medium-3-5", "open": false, "I": 14.9, "C": 46.9, "A": 9.4, "pi": 1.5, "po": 7.5, "cr": null, "ctx": 262144, "img": true, "tools": true, "so": true, "created": 1777570439, "exp": null, "alias": null}, {"id": "qwen/qwen3.6-35b-a3b", "open": true, "I": 18.8, "C": 41.9, "A": 15, "pi": 0.09999999999999999, "po": 0.8999999999999999, "cr": 0.049999999999999996, "ctx": 262144, "img": true, "tools": true, "so": true, "created": 1777260255, "exp": null, "alias": null}, {"id": "qwen/qwen3.6-27b", "open": true, "I": 21.9, "C": 53.7, "A": 20.1, "pi": 0.3, "po": 2.0, "cr": 0.03, "ctx": 262144, "img": true, "tools": true, "so": true, "created": 1777255064, "exp": null, "alias": null}, {"id": "openai/gpt-5.5", "open": false, "I": 38.6, "C": 74.9, "A": 37.3, "pi": 5.0, "po": 30.0, "cr": 0.5, "ctx": 1050000, "img": true, "tools": true, "so": true, "created": 1777051893, "exp": null, "alias": null}, {"id": "deepseek/deepseek-v4-pro", "open": true, "I": 30.9, "C": 59.4, "A": 27.7, "pi": 1.5999999999999999, "po": 3.1999999999999997, "cr": 0.135, "ctx": 1048576, "img": false, "tools": true, "so": true, "created": 1777000679, "exp": null, "alias": null}, {"id": "deepseek/deepseek-v4-flash", "open": true, "I": 24.8, "C": 52, "A": 27.9, "pi": 0.088606, "po": 0.177212, "cr": 0.017721200000000003, "ctx": 1048576, "img": false, "tools": true, "so": true, "created": 1777000666, "exp": null, "alias": null}, {"id": "tencent/hy3-preview", "open": true, "I": 25.8, "C": 58.8, "A": 25.6, "pi": 0.18, "po": 0.6, "cr": 0.06, "ctx": 262144, "img": false, "tools": true, "so": false, "created": 1776878150, "exp": null, "alias": null}, {"id": "xiaomi/mimo-v2.5-pro", "open": true, "I": 26.4, "C": 60.2, "A": 22.7, "pi": 0.435, "po": 0.87, "cr": 0.0036, "ctx": 1050000, "img": false, "tools": true, "so": true, "created": 1776874273, "exp": null, "alias": null}, {"id": "xiaomi/mimo-v2.5", "open": true, "I": 22.3, "C": 56.8, "A": 17.4, "pi": 0.14, "po": 0.28, "cr": 0.0028, "ctx": 1050000, "img": true, "tools": true, "so": true, "created": 1776874269, "exp": null, "alias": null}, {"id": "moonshotai/kimi-k2.6", "open": true, "I": 27.5, "C": 61.8, "A": 22.1, "pi": 0.95, "po": 4.0, "cr": 0.16, "ctx": 262144, "img": true, "tools": true, "so": true, "created": 1776699402, "exp": null, "alias": null}, {"id": "z-ai/glm-5.1", "open": true, "I": 26.4, "C": 55.8, "A": 25.2, "pi": 0.966, "po": 3.036, "cr": 0.1794, "ctx": 204800, "img": false, "tools": true, "so": true, "created": 1775578025, "exp": null, "alias": null}, {"id": "google/gemma-4-31b-it", "open": true, "I": 15.4, "C": 43.4, "A": 6.7, "pi": 0.09, "po": 0.33999999999999997, "cr": 0.049999999999999996, "ctx": 262144, "img": true, "tools": true, "so": true, "created": 1775148486, "exp": null, "alias": null}, {"id": "arcee-ai/trinity-large-thinking", "open": true, "I": 10.9, "C": 25.8, "A": 1.2, "pi": 0.25, "po": 0.7999999999999999, "cr": 0.06, "ctx": 262144, "img": false, "tools": true, "so": false, "created": 1775058318, "exp": null, "alias": null}, {"id": "minimax/minimax-m2.7", "open": true, "I": 23.2, "C": 52.6, "A": 16.8, "pi": 0.3, "po": 1.2, "cr": 0.06, "ctx": 204800, "img": false, "tools": true, "so": true, "created": 1773836697, "exp": null, "alias": null}, {"id": "openai/gpt-5.4-nano", "open": false, "I": 21.2, "C": 56.1, "A": 17.7, "pi": 0.19999999999999998, "po": 1.25, "cr": 0.02, "ctx": 400000, "img": true, "tools": true, "so": true, "created": 1773748187, "exp": null, "alias": null}, {"id": "openai/gpt-5.4-mini", "open": false, "I": 24.6, "C": 56.1, "A": 19.7, "pi": 0.75, "po": 4.5, "cr": 0.075, "ctx": 400000, "img": true, "tools": true, "so": true, "created": 1773748178, "exp": null, "alias": null}, {"id": "mistralai/mistral-small-2603", "open": true, "I": 11.5, "C": 26.6, "A": 1.4, "pi": 0.15, "po": 0.6, "cr": 0.015, "ctx": 262144, "img": true, "tools": true, "so": true, "created": 1773695685, "exp": null, "alias": null}, {"id": "nvidia/nemotron-3-super-120b-a12b", "open": true, "I": 13.6, "C": 37.7, "A": 4.1, "pi": 0.08, "po": 0.44999999999999996, "cr": null, "ctx": 262144, "img": false, "tools": true, "so": true, "created": 1773245239, "exp": null, "alias": null}, {"id": "inception/mercury-2", "open": false, "I": 11.5, "C": 31.1, "A": 4, "pi": 0.25, "po": 0.75, "cr": 0.024999999999999998, "ctx": 128000, "img": false, "tools": true, "so": true, "created": 1772636275, "exp": null, "alias": null}, {"id": "google/gemini-3.1-flash-lite-preview", "open": false, "I": 16, "C": 34.7, "A": 3.2, "pi": 0.25, "po": 1.5, "cr": 0.024999999999999998, "ctx": 1048576, "img": true, "tools": true, "so": true, "created": 1772512673, "exp": null, "alias": null}, {"id": "qwen/qwen3.5-122b-a10b", "open": true, "I": 16.2, "C": 45.7, "A": 9.6, "pi": 0.26, "po": 2.08, "cr": null, "ctx": 262144, "img": true, "tools": true, "so": true, "created": 1772053789, "exp": null, "alias": null}, {"id": "google/gemini-3.1-pro-preview", "open": false, "I": 30.4, "C": 68.8, "A": 10.3, "pi": 2.0, "po": 12.0, "cr": 0.19999999999999998, "ctx": 1048576, "img": true, "tools": true, "so": true, "created": 1771509627, "exp": null, "alias": null}, {"id": "anthropic/claude-sonnet-4.6", "open": false, "I": 30.5, "C": 63, "A": 33.1, "pi": 3.0, "po": 15.0, "cr": 0.3, "ctx": 1000000, "img": true, "tools": true, "so": true, "created": 1771342990, "exp": null, "alias": null}, {"id": "qwen/qwen3.5-397b-a17b", "open": true, "I": 19.1, "C": 48.2, "A": 10.6, "pi": 0.55, "po": 3.5, "cr": 0.22499999999999998, "ctx": 262144, "img": true, "tools": true, "so": true, "created": 1771223018, "exp": null, "alias": null}, {"id": "qwen/qwen3-coder-next", "open": true, "I": 10.1, "C": 36.2, "A": 3.6, "pi": 0.12, "po": 0.7999999999999999, "cr": 0.07, "ctx": 262144, "img": false, "tools": true, "so": true, "created": 1770164101, "exp": null, "alias": null}, {"id": "upstage/solar-pro-3", "open": false, "I": 7.8, "C": 16.2, "A": 1.4, "pi": 0.15, "po": 0.6, "cr": 0.015, "ctx": 131072, "img": false, "tools": true, "so": true, "created": 1769481200, "exp": null, "alias": null}, {"id": "nvidia/nemotron-3-nano-30b-a3b", "open": true, "I": 8.9, "C": 14.4, "A": 1, "pi": 0.06, "po": 0.24, "cr": null, "ctx": 262144, "img": false, "tools": true, "so": true, "created": 1765731275, "exp": null, "alias": null}, {"id": "mistralai/devstral-2512", "open": true, "I": 9.4, "C": 31.3, "A": 4.9, "pi": 0.39999999999999997, "po": 2.0, "cr": 0.04, "ctx": 262144, "img": false, "tools": true, "so": true, "created": 1765285419, "exp": null, "alias": null}, {"id": "mistralai/ministral-14b-2512", "open": true, "I": 6, "C": 14.4, "A": 1.1, "pi": 0.19999999999999998, "po": 0.19999999999999998, "cr": 0.02, "ctx": 262144, "img": true, "tools": true, "so": true, "created": 1764681735, "exp": null, "alias": null}, {"id": "mistralai/ministral-8b-2512", "open": true, "I": 5.5, "C": 9.7, "A": 0.6, "pi": 0.15, "po": 0.15, "cr": 0.015, "ctx": 262144, "img": true, "tools": true, "so": true, "created": 1764681654, "exp": null, "alias": null}, {"id": "mistralai/ministral-3b-2512", "open": true, "I": 4.8, "C": 4.8, "A": 0.8, "pi": 0.09999999999999999, "po": 0.09999999999999999, "cr": 0.01, "ctx": 131072, "img": true, "tools": true, "so": true, "created": 1764681560, "exp": null, "alias": null}, {"id": "mistralai/mistral-large-2512", "open": false, "I": 9.7, "C": 20.1, "A": 2.4, "pi": 0.5, "po": 1.5, "cr": 0.049999999999999996, "ctx": 262144, "img": true, "tools": true, "so": true, "created": 1764624472, "exp": null, "alias": null}, {"id": "anthropic/claude-haiku-4.5", "open": false, "I": 17.6, "C": 43.9, "A": 10.3, "pi": 1.0, "po": 5.0, "cr": 0.09999999999999999, "ctx": 200000, "img": true, "tools": true, "so": true, "created": 1760547638, "exp": null, "alias": null}, {"id": "anthropic/claude-sonnet-4.5", "open": false, "I": 21.2, "C": 52.1, "A": 17.5, "pi": 3.0, "po": 15.0, "cr": 0.3, "ctx": 1000000, "img": true, "tools": true, "so": true, "created": 1759161676, "exp": null, "alias": null}, {"id": "deepseek/deepseek-v3.1-terminus", "open": true, "I": 15.4, "C": 43.5, "A": 8.9, "pi": 0.27, "po": 1.0, "cr": 0.135, "ctx": 163840, "img": false, "tools": true, "so": true, "created": 1758548275, "exp": null, "alias": null}, {"id": "qwen/qwen3-30b-a3b-thinking-2507", "open": true, "I": 9.8, "C": 12.1, "A": 0.9, "pi": 0.19999999999999998, "po": 2.4, "cr": null, "ctx": 81920, "img": false, "tools": true, "so": false, "created": 1756399192, "exp": null, "alias": null}, {"id": "mistralai/mistral-medium-3.1", "open": false, "I": 9.5, "C": 20.5, "A": 3.1, "pi": 0.39999999999999997, "po": 2.0, "cr": 0.04, "ctx": 131072, "img": true, "tools": true, "so": true, "created": 1755095639, "exp": null, "alias": null}, {"id": "openai/gpt-5-mini", "open": false, "I": 17.4, "C": 15.6, "A": 8.9, "pi": 0.25, "po": 2.0, "cr": 0.024999999999999998, "ctx": 400000, "img": true, "tools": true, "so": true, "created": 1754587407, "exp": null, "alias": null}, {"id": "openai/gpt-oss-120b", "open": true, "I": 12.3, "C": 30.4, "A": 6.2, "pi": 0.037, "po": 0.16999999999999998, "cr": null, "ctx": 131072, "img": false, "tools": true, "so": true, "created": 1754414231, "exp": null, "alias": null}, {"id": "openai/gpt-oss-20b", "open": true, "I": 9, "C": 20.7, "A": 1.4, "pi": 0.03, "po": 0.13, "cr": 0.03, "ctx": 131072, "img": false, "tools": true, "so": true, "created": 1754414229, "exp": null, "alias": null}, {"id": "qwen/qwen3-235b-a22b-thinking-2507", "open": true, "I": 12.7, "C": 22.1, "A": 1.3, "pi": 0.22999999999999998, "po": 2.3, "cr": null, "ctx": 131072, "img": false, "tools": true, "so": false, "created": 1753449557, "exp": null, "alias": null}, {"id": "google/gemini-2.5-pro", "open": false, "I": 16.7, "C": 33.3, "A": 3.5, "pi": 1.25, "po": 10.0, "cr": 0.125, "ctx": 1048576, "img": true, "tools": true, "so": true, "created": 1750169544, "exp": null, "alias": null}, {"id": "qwen/qwen3-8b", "open": true, "I": 5.2, "C": 9, "A": 0.8, "pi": 0.117, "po": 0.45499999999999996, "cr": null, "ctx": 131072, "img": false, "tools": true, "so": false, "created": 1745876632, "exp": null, "alias": null}, {"id": "qwen/qwen3-14b", "open": true, "I": 6.4, "C": 13.8, "A": 0.9, "pi": 0.12, "po": 0.24, "cr": null, "ctx": 131072, "img": false, "tools": true, "so": true, "created": 1745876478, "exp": null, "alias": null}, {"id": "qwen/qwen3-32b", "open": true, "I": 7.2, "C": 15.3, "A": 0.9, "pi": 0.08, "po": 0.28, "cr": null, "ctx": 131072, "img": false, "tools": true, "so": true, "created": 1745875945, "exp": null, "alias": null}, {"id": "meta-llama/llama-4-maverick", "open": true, "I": 9.3, "C": 16.3, "A": 0.6, "pi": 0.1875, "po": 0.6525, "cr": null, "ctx": 1048576, "img": true, "tools": true, "so": true, "created": 1743881822, "exp": null, "alias": null}, {"id": "meta-llama/llama-4-scout", "open": true, "I": 6.5, "C": 8.2, "A": 0.5, "pi": 0.09999999999999999, "po": 0.3, "cr": null, "ctx": 1310720, "img": true, "tools": true, "so": true, "created": 1743881519, "exp": null, "alias": null}, {"id": "deepseek/deepseek-chat-v3-0324", "open": true, "I": 9.7, "C": 21.2, "A": 0.8, "pi": 0.25, "po": 1.0, "cr": null, "ctx": 163840, "img": false, "tools": true, "so": true, "created": 1742824755, "exp": null, "alias": null}, {"id": "google/gemma-3-12b-it", "open": true, "I": 3.8, "C": 5.8, "A": 0.1, "pi": 0.049999999999999996, "po": 0.15, "cr": null, "ctx": 131072, "img": true, "tools": true, "so": true, "created": 1741902625, "exp": null, "alias": null}, {"id": "cohere/command-a", "open": true, "I": 13.9, "C": 27.8, "A": 3.6, "pi": 2.5, "po": 10.0, "cr": null, "ctx": 256000, "img": false, "tools": false, "so": true, "created": 1741894342, "exp": null, "alias": null}, {"id": "google/gemma-3-27b-it", "open": true, "I": 4.9, "C": 10.1, "A": 0.1, "pi": 0.08, "po": 0.44999999999999996, "cr": 0.04, "ctx": 131072, "img": true, "tools": true, "so": true, "created": 1741756359, "exp": null, "alias": null}, {"id": "openai/o3-mini-high", "open": false, "I": 11, "C": 16.3, "A": 0.9, "pi": 1.1, "po": 4.4, "cr": 0.55, "ctx": 200000, "img": false, "tools": true, "so": true, "created": 1739372611, "exp": null, "alias": null}, {"id": "deepseek/deepseek-r1", "open": true, "I": 11.4, "C": 24.6, "A": 1.1, "pi": 0.7, "po": 2.5, "cr": null, "ctx": 64000, "img": false, "tools": true, "so": false, "created": 1737381095, "exp": null, "alias": null}] \ No newline at end of file diff --git a/internal/crewroute/allowed.go b/internal/crewroute/allowed.go new file mode 100644 index 0000000000..8b92d1c5fd --- /dev/null +++ b/internal/crewroute/allowed.go @@ -0,0 +1,344 @@ +package crewroute + +import ( + "fmt" + "strconv" + "strings" +) + +// WHICH MODELS THE CREW MAY BE PICKED FROM — ONE RULE, WRITTEN ONCE. +// +// The /crew panel says what is allowed and the router picks inside it, and +// this is the rule's grammar and its one reading. It is a RULE rather than a +// list wherever it can be, because a rule keeps meaning what it meant when the +// catalog moves: `open` admits tomorrow's open-weight release the day it is +// listed, and a list of today's ids would need somebody to notice it. +// +// all everything a connected provider can reach (the default) +// open open-weight models only +// ≤1/5 at most $1 per million tokens in and $5 out (`<=1/5` too) +// glm-5.3-flash, kimi-k3 exactly these models — an id, a model's own name +// after its vendor, or a vendor or provider name +// +// Any of the four may be followed by `+x` and `-x`, read left to right, the +// last word about a model winning: `open -deepseek` is every open model but +// DeepSeek's, `≤1/5 +moonshotai/kimi-k3` is the price rule with one exception +// let in. A `+x` or `-x` naming a PROVIDER — `-openrouter` — takes that +// provider's routes away rather than models, so a model it serves stays +// allowed on any other route that reaches it. + +// Base is the first word of a rule: what is allowed before any `+x` or `-x`. +type Base string + +const ( + BaseAll Base = "all" + BaseOpen Base = "open" + BasePrice Base = "price" + BaseList Base = "list" +) + +// DefaultAllowed is the rule a profile that never answered reads. +const DefaultAllowed = "all" + +// Allowed is a parsed rule. +type Allowed struct { + Base Base + // MaxIn and MaxOut are the price rule's two ceilings, dollars per million + // tokens. Zero on every other base. + MaxIn float64 + MaxOut float64 + // Members are an explicit list's words. + Members []string + // Mods are the `+x` / `-x` words, in the order they were written. + Mods []Mod +} + +// Mod is one `+x` or `-x`. +type Mod struct { + Add bool + Word string +} + +// ParseAllowed reads a rule. An empty rule is [DefaultAllowed]. A rule that +// does not parse is refused with the reason, because a typo that silently +// allowed everything would be a setting somebody thinks is protecting them. +func ParseAllowed(raw string) (Allowed, error) { + words := splitRule(raw) + if len(words) == 0 { + return Allowed{Base: BaseAll}, nil + } + var rule Allowed + first := strings.ToLower(words[0]) + rest := words + switch { + case first == string(BaseAll): + rule.Base, rest = BaseAll, words[1:] + case first == string(BaseOpen): + rule.Base, rest = BaseOpen, words[1:] + case strings.HasPrefix(first, "≤") || strings.HasPrefix(first, "<="): + in, out, err := parsePriceRule(first) + if err != nil { + return Allowed{}, err + } + rule.Base, rule.MaxIn, rule.MaxOut, rest = BasePrice, in, out, words[1:] + case strings.HasPrefix(first, "+") || strings.HasPrefix(first, "-"): + // A rule that opens with a modifier modifies everything: `-openrouter` + // alone is all, minus one provider. + rule.Base = BaseAll + default: + rule.Base = BaseList + } + for _, word := range rest { + switch { + case strings.HasPrefix(word, "+") && len(word) > 1: + rule.Mods = append(rule.Mods, Mod{Add: true, Word: strings.ToLower(word[1:])}) + case strings.HasPrefix(word, "-") && len(word) > 1: + rule.Mods = append(rule.Mods, Mod{Add: false, Word: strings.ToLower(word[1:])}) + case rule.Base == BaseList: + rule.Members = append(rule.Members, strings.ToLower(word)) + default: + return Allowed{}, fmt.Errorf("%q: after %s, name a model or provider with + or -", word, rule.Base) + } + } + return rule, nil +} + +// splitRule reads a rule's words: spaces and commas both separate them. +func splitRule(raw string) []string { + return strings.FieldsFunc(strings.TrimSpace(raw), func(r rune) bool { + return r == ',' || r == ' ' || r == '\t' || r == '\n' + }) +} + +// parsePriceRule reads `≤1/5` or `<=1/5`: two ceilings, dollars per million. +func parsePriceRule(word string) (in, out float64, err error) { + body := strings.TrimPrefix(strings.TrimPrefix(word, "≤"), "<=") + body = strings.ReplaceAll(body, "$", "") + parts := strings.Split(body, "/") + if len(parts) != 2 { + return 0, 0, fmt.Errorf("%q: a price rule is ≤/ dollars per million tokens, like ≤1/5", word) + } + in, errIn := strconv.ParseFloat(strings.TrimSpace(parts[0]), 64) + out, errOut := strconv.ParseFloat(strings.TrimSpace(parts[1]), 64) + if errIn != nil || errOut != nil || in < 0 || out < 0 { + return 0, 0, fmt.Errorf("%q: a price rule is ≤/ dollars per million tokens, like ≤1/5", word) + } + return in, out, nil +} + +// String writes the rule back in its canonical spelling, which is what the +// panel shows and what is stored. +func (a Allowed) String() string { + var words []string + switch a.Base { + case BaseOpen: + words = append(words, string(BaseOpen)) + case BasePrice: + words = append(words, "≤"+trimFloat(a.MaxIn)+"/"+trimFloat(a.MaxOut)) + case BaseList: + words = append(words, strings.Join(a.Members, ", ")) + default: + if len(a.Mods) == 0 { + return string(BaseAll) + } + words = append(words, string(BaseAll)) + } + for _, mod := range a.Mods { + sign := "-" + if mod.Add { + sign = "+" + } + words = append(words, sign+mod.Word) + } + return strings.Join(words, " ") +} + +// trimFloat spells a ceiling without trailing zeros. +func trimFloat(v float64) string { return strconv.FormatFloat(v, 'f', -1, 64) } + +// With is the rule with one more `+x` or `-x` on its end — what `/crew models +// +x` writes. A word the rule already carries with the same sign is not +// written twice, and the same word with the other sign is replaced, so the +// rule stays the shortest spelling of what the person meant. +func (a Allowed) With(add bool, word string) Allowed { + word = strings.ToLower(strings.TrimSpace(word)) + out := a + out.Mods = nil + for _, mod := range a.Mods { + if mod.Word != word { + out.Mods = append(out.Mods, mod) + } + } + out.Mods = append(out.Mods, Mod{Add: add, Word: word}) + return out +} + +// AdmitsModel says whether the rule allows a model, before routes are asked. +func (a Allowed) AdmitsModel(m Model) bool { + admitted := false + switch a.Base { + case BaseAll, "": + admitted = true + case BaseOpen: + admitted = m.Open + case BasePrice: + // Prices are per token in the catalog; the rule is per million. + admitted = m.PromptPrice*1e6 <= a.MaxIn+1e-9 && m.CompletionPrice*1e6 <= a.MaxOut+1e-9 + case BaseList: + for _, member := range a.Members { + if matchesModel(member, m.ID) { + admitted = true + break + } + } + } + for _, mod := range a.Mods { + if matchesModel(mod.Word, m.ID) { + admitted = mod.Add + } + } + return admitted +} + +// AdmitsRoute says whether the rule allows a provider's routes. Only a `-x` +// naming the provider takes them away; a later `+x` gives them back. +func (a Allowed) AdmitsRoute(provider string) bool { + provider = strings.ToLower(strings.TrimSpace(provider)) + admitted := true + for _, mod := range a.Mods { + if mod.Word == provider { + admitted = mod.Add + } + } + return admitted +} + +// NamesModel says whether the rule speaks about this model by name — a list +// member or a modifier matching it. It is how a pin outside a price or open +// rule can still be let in by name. +func (a Allowed) NamesModel(id string) bool { + for _, member := range a.Members { + if matchesModel(member, id) { + return true + } + } + for _, mod := range a.Mods { + if mod.Add && matchesModel(mod.Word, id) { + return true + } + } + return false +} + +// matchesModel reads one rule word against one model id: the whole id, the +// model's own name after its vendor, or the vendor itself. The comparison is +// on lineage, so `deepseek/deepseek-v4-flash` names the dated build too. +func matchesModel(word, id string) bool { + word = strings.ToLower(strings.TrimSpace(word)) + if word == "" { + return false + } + full := Lineage(id) + if Lineage(word) == full { + return true + } + vendor, tail := full, full + if slash := strings.Index(full, "/"); slash >= 0 { + vendor, tail = full[:slash], full[slash+1:] + } + return word == vendor || Lineage(word) == tail +} + +// ── THE PANEL'S EDITS, EACH THE SHORTEST RULE THAT SAYS IT ── +// +// The /crew panel edits the rule a toggle at a time — a model ticked on or off +// in its checklist, a provider taken away, the base walked from `all` to +// `open` — and every one of those writes the rule back through the same one +// writer the typed form uses. What these add is the arithmetic of saying the +// change in as few words as possible, because the rule is SHOWN: a checklist +// that left `+x -x +x` behind after three presses would be a panel writing a +// rule nobody could read back. + +// Custom says whether the rule is more than one of the three plain bases: an +// explicit list, or a base with exceptions on it. It is the fourth answer the +// panel's models row walks between, and the one its checklist edits. +func (a Allowed) Custom() bool { + return a.Base == BaseList || len(a.Mods) > 0 +} + +// Rebased is the rule on a new base with nothing carried over — the `‹ open ›` +// the panel's row steps to is `open`, and not `open` with whatever exceptions +// the rule it stepped from happened to have. The price ceilings are taken for +// [BasePrice] and ignored for every other base. +func (a Allowed) Rebased(base Base, maxIn, maxOut float64) Allowed { + out := Allowed{Base: base} + if base == BasePrice { + out.MaxIn, out.MaxOut = maxIn, maxOut + } + return out +} + +// Without is the rule with every `+x` or `-x` naming word taken off, which +// hands that word's verdict back to the base. +func (a Allowed) Without(word string) Allowed { + word = strings.ToLower(strings.TrimSpace(word)) + out := a + out.Mods = nil + for _, mod := range a.Mods { + if mod.Word != word { + out.Mods = append(out.Mods, mod) + } + } + return out +} + +// Toggled is the rule with one model's verdict flipped to admit, in the +// shortest spelling: an explicit list gains or loses a member, and any other +// base loses the exception it already had for the model when that alone gives +// the wanted verdict, or gains one when it does not. It is what a tick in the +// panel's checklist writes. +// +// AN EXPLICIT LIST MAY NOT BE EMPTIED. `a, b` less both is not a rule the +// grammar can spell — an empty rule reads as `all`, which is the opposite of +// what unticking the last model means — so the second answer is false and the +// rule comes back as it was. +func (a Allowed) Toggled(m Model, admit bool) (Allowed, bool) { + word := strings.ToLower(strings.TrimSpace(m.ID)) + if a.Base == BaseList { + out := a.Without(word) + var members []string + for _, member := range a.Members { + if !matchesModel(member, m.ID) { + members = append(members, member) + } + } + if admit { + members = append(members, word) + } + if len(members) == 0 { + return a, false + } + out.Members = members + if out.AdmitsModel(m) != admit { + // A vendor-wide exception outranks the member: say this model by name + // after it, which is the last word and therefore the verdict. + out = out.With(admit, word) + } + return out, true + } + out := a.Without(word) + if out.AdmitsModel(m) != admit { + out = out.With(admit, word) + } + return out, true +} + +// RouteToggled is the rule with one provider's routes given back or taken +// away, in the shortest spelling, the way [Allowed.Toggled] says a model. +func (a Allowed) RouteToggled(provider string, admit bool) Allowed { + out := a.Without(provider) + if out.AdmitsRoute(provider) != admit { + out = out.With(admit, provider) + } + return out +} diff --git a/internal/crewroute/allowed_test.go b/internal/crewroute/allowed_test.go new file mode 100644 index 0000000000..34b745ef12 --- /dev/null +++ b/internal/crewroute/allowed_test.go @@ -0,0 +1,120 @@ +package crewroute + +import "testing" + +func TestAllowedRules(t *testing.T) { + flash := Model{ID: "z-ai/glm-5.3-flash", Open: true, PromptPrice: 0.15e-6, CompletionPrice: 0.5e-6} + kimi := Model{ID: "moonshotai/kimi-k3", Open: true, PromptPrice: 3e-6, CompletionPrice: 15e-6} + opus := Model{ID: "anthropic/claude-opus-5", PromptPrice: 5e-6, CompletionPrice: 25e-6} + v4 := Model{ID: "deepseek/deepseek-v4-flash-0731", Open: true, PromptPrice: 0.04e-6, CompletionPrice: 0.64e-6} + cases := []struct { + rule string + admit map[string]bool + }{ + {"", map[string]bool{flash.ID: true, kimi.ID: true, opus.ID: true, v4.ID: true}}, + {"all", map[string]bool{flash.ID: true, kimi.ID: true, opus.ID: true, v4.ID: true}}, + {"open", map[string]bool{flash.ID: true, kimi.ID: true, opus.ID: false, v4.ID: true}}, + {"open -deepseek", map[string]bool{flash.ID: true, kimi.ID: true, opus.ID: false, v4.ID: false}}, + {"≤1/5", map[string]bool{flash.ID: true, kimi.ID: false, opus.ID: false, v4.ID: true}}, + {"<=1/5 +moonshotai/kimi-k3", map[string]bool{flash.ID: true, kimi.ID: true, opus.ID: false, v4.ID: true}}, + {"glm-5.3-flash, kimi-k3", map[string]bool{flash.ID: true, kimi.ID: true, opus.ID: false, v4.ID: false}}, + {"deepseek/deepseek-v4-flash", map[string]bool{v4.ID: true, flash.ID: false}}, + {"anthropic", map[string]bool{opus.ID: true, flash.ID: false}}, + {"-anthropic", map[string]bool{opus.ID: false, flash.ID: true}}, + } + for _, tc := range cases { + rule, err := ParseAllowed(tc.rule) + if err != nil { + t.Fatalf("%q: %v", tc.rule, err) + } + for _, m := range []Model{flash, kimi, opus, v4} { + want, asked := tc.admit[m.ID] + if !asked { + continue + } + if got := rule.AdmitsModel(m); got != want { + t.Errorf("%q admits %s = %v, want %v", tc.rule, m.ID, got, want) + } + } + } +} + +func TestAllowedRefusesWhatItCannotRead(t *testing.T) { + for _, bad := range []string{"≤1", "<=a/b", "open kimi-k3", "≤-1/5"} { + if _, err := ParseAllowed(bad); err == nil { + t.Errorf("%q parsed; want a refusal", bad) + } + } +} + +func TestAllowedRoundTripsAndModifies(t *testing.T) { + for _, rule := range []string{"all", "open", "≤1/5", "open -deepseek +deepseek/deepseek-v4-flash", "glm-5.3-flash, kimi-k3"} { + parsed, err := ParseAllowed(rule) + if err != nil { + t.Fatal(err) + } + if parsed.String() != rule { + t.Errorf("%q round-trips as %q", rule, parsed.String()) + } + } + open, _ := ParseAllowed("open -deepseek") + if got := open.With(true, "deepseek").String(); got != "open +deepseek" { + t.Errorf("With flips a sign: %q", got) + } + if got := open.With(false, "openrouter"); got.AdmitsRoute("openrouter") || !got.AdmitsRoute("z-ai") { + t.Errorf("-openrouter: %q", got.String()) + } +} + +// THE PANEL'S EDITS ARE THE SHORTEST RULE THAT SAYS THEM: a tick undone is the +// rule it was, and a list keeps at least one member. +func TestAllowedTogglesInTheShortestSpelling(t *testing.T) { + kimi := Model{ID: "moonshotai/kimi-k3", Open: true} + opus := Model{ID: "anthropic/claude-opus-5"} + all, _ := ParseAllowed("all") + + off, ok := all.Toggled(kimi, false) + if !ok || off.String() != "all -moonshotai/kimi-k3" { + t.Fatalf("unticking kimi from all wrote %q", off.String()) + } + back, _ := off.Toggled(kimi, true) + if back.String() != "all" || back.Custom() { + t.Fatalf("ticking it again wrote %q, want the plain base back", back.String()) + } + + open, _ := ParseAllowed("open") + in, _ := open.Toggled(opus, true) + if in.String() != "open +anthropic/claude-opus-5" || !in.AdmitsModel(opus) { + t.Fatalf("ticking a closed model on open wrote %q", in.String()) + } + + list, _ := ParseAllowed("kimi-k3, glm-5.3-flash") + grown, _ := list.Toggled(opus, true) + if !grown.AdmitsModel(opus) || grown.Base != BaseList { + t.Fatalf("ticking onto a list wrote %q", grown.String()) + } + shrunk, _ := grown.Toggled(kimi, false) + if shrunk.AdmitsModel(kimi) || len(shrunk.Mods) != 0 { + t.Fatalf("unticking a member wrote %q, want it off the list and no exception", shrunk.String()) + } + one, _ := ParseAllowed("kimi-k3") + if kept, ok := one.Toggled(kimi, false); ok || kept.String() != one.String() { + t.Fatalf("emptying a list was accepted as %q", kept.String()) + } + + away := all.RouteToggled("openrouter", false) + if away.String() != "all -openrouter" || away.AdmitsRoute("openrouter") { + t.Fatalf("taking openrouter away wrote %q", away.String()) + } + if again := away.RouteToggled("openrouter", true); again.String() != "all" { + t.Fatalf("giving it back wrote %q", again.String()) + } + + price, _ := ParseAllowed("open -deepseek") + if re := price.Rebased(BasePrice, 1, 5); re.String() != "≤1/5" { + t.Fatalf("rebasing onto a price wrote %q", re.String()) + } + if !price.Custom() { + t.Fatal("a base with an exception is not custom") + } +} diff --git a/internal/crewroute/canonical.go b/internal/crewroute/canonical.go new file mode 100644 index 0000000000..f163bda46f --- /dev/null +++ b/internal/crewroute/canonical.go @@ -0,0 +1,310 @@ +package crewroute + +import ( + "strconv" + "strings" + "sync" + + "github.com/Agent-Field/codeaf/internal/pool/index" +) + +// ONE MODEL, MANY SPELLINGS — WHAT IS THE SAME MODEL ACROSS PROVIDERS. +// +// Evidence and quality belong to a MODEL; a price, a limit and a send id belong +// to a ROUTE to it. OpenRouter spells GLM `z-ai/glm-5.3-flash`, Fireworks +// `accounts/fireworks/models/glm-5p3-flash`, a Hugging Face mirror +// `zai-org/GLM-5.3-Flash`, OpenRouter's free pool `z-ai/glm-5.3-flash:free`, +// and a local Ollama pull `glm-5.3-flash:q4_k_m`. The router must read those +// as one model with five routes — or, for the local pull, as a distinct +// VARIANT of the one model — or it scores a model once and prices it five +// times as five strangers. +// +// [Canonical] is that reading. It is PURE AND FAST — string operations, no +// pattern engine, no network — because it runs for every catalog row of every +// decision. It composes, in order: +// +// 1. the provider namespaces a re-hosting provider puts in front of a model +// (`openrouter/`, `accounts//models/`), taken off; +// 2. the route suffixes that name a way of serving the same weights +// (`:free`, `:nitro`, `:floor`), taken off — a route is not a model; +// 3. a local quantisation tag (`:q4_k_m`, `:fp8`, `-gguf`), taken off into a +// VARIANT: the same weights squeezed, which inherit the model's evidence +// at a discount ([quantDiscount]) but are never the same model; +// 4. spelling: lowercase, `_` read as `-`, and a version written with a `p` +// between two digits (`5p3`) read as the dot it stands for; +// 5. the owner: a Hugging Face organisation read as the catalog's vendor +// ([vendorOrgs]), and a bare model name given the vendor its family +// belongs to ([familyVendors]); +// 6. the pool index's own alternates (internal/pool/index), and the small +// table of known mismatches below ([modelAliases]); +// 7. [Lineage]'s rules: a thinking level, `-latest` and a dated snapshot. +// +// AN UNKNOWN SPELLING STAYS ITS OWN MODEL. Nothing here merges two ids because +// they look alike: every rule is a fact about how a provider spells an id, and +// a model no rule recognises is read on its own catalog figures, the way +// every model is. + +// Canon is a model's canonical identity: the id evidence is kept under, and +// the quantisation variant when the id names a squeezed local copy of it. +type Canon struct { + ID string + Variant string +} + +// String is the identity as one key: the id, and the variant after `@` when +// there is one — so a quantised copy is never the model it came from. +func (c Canon) String() string { + if c.Variant == "" { + return c.ID + } + return c.ID + "@" + c.Variant +} + +// quantDiscount is how much of a model's quality a quantised local copy is +// credited with. Quantisation costs something on agentic work — tool calls +// are where a squeezed model slips first — so the credit is a conservative +// share: a local copy sits a seat when it is free AND nearly as good, never +// because its model's score was borrowed whole. +const quantDiscount = 0.85 + +// routeSuffixes are the `:word` endings that name a way of serving a model +// rather than a model: OpenRouter's free pool, its throughput and price +// variants. +var routeSuffixes = map[string]bool{"free": true, "nitro": true, "floor": true} + +// thinkingSuffixes are the `:word` endings that name how hard a model thinks +// on a call — the same weights, asked differently. +var thinkingSuffixes = map[string]bool{"low": true, "medium": true, "high": true, "minimal": true, "max": true} + +// vendorOrgs are Hugging Face organisations whose models the catalog lists +// under another vendor word. A TABLE OF FACTS: each is the organisation that +// publishes the weights and the vendor word OpenRouter spells for it. +var vendorOrgs = map[string]string{ + "zai-org": "z-ai", + "deepseek-ai": "deepseek", + "moonshot": "moonshotai", + "meta": "meta-llama", +} + +// familyVendors give a bare model name — a provider that hosts it without an +// owner in the id, as Fireworks and Ollama do — the vendor its family belongs +// to. A TABLE OF FACTS about which company names its models this way; a name +// no entry starts is left without a vendor and stays its own model. +var familyVendors = []struct{ prefix, vendor string }{ + {"glm-", "z-ai"}, + {"kimi-", "moonshotai"}, + {"deepseek-", "deepseek"}, + {"qwen", "qwen"}, + {"gpt-oss", "openai"}, + {"llama-", "meta-llama"}, + {"mistral-", "mistralai"}, + {"devstral", "mistralai"}, + {"codestral", "mistralai"}, + {"minimax-", "minimax"}, + {"gemma-", "google"}, +} + +// modelAliases are known mismatches no rule above reads: a provider's own name +// for a model the catalog lists under another. A TABLE OF FACTS, each entry +// added when a provider was seen to spell a model this way — never a guess +// that two names probably mean one model. +var modelAliases = map[string]string{ + // Together serves Llama 3.3 70B under a "-turbo" name for the same + // instruct weights the catalog lists. + "meta-llama/llama-3.3-70b-instruct-turbo": "meta-llama/llama-3.3-70b-instruct", +} + +var ( + poolIndexOnce sync.Once + poolIndex *index.Index +) + +// poolAliases is the embedded pool index, read once; nil when it will not +// parse, which leaves its alternates unread and nothing else changed. +func poolAliases() *index.Index { + poolIndexOnce.Do(func() { + if x, err := index.SeedIndex(); err == nil { + poolIndex = x + } + }) + return poolIndex +} + +// Canonical is the id a model's evidence is kept under, across every +// provider's spelling of it ([CanonicalOf] without the variant). +func Canonical(id string) string { return CanonicalOf(id).ID } + +// canonMemo holds what [CanonicalOf] has read, because a decision asks the +// same few hundred ids several times each and the answer never changes. It +// is bounded: past canonMemoMax ids it stops remembering new ones. +var canonMemo = struct { + sync.RWMutex + ids map[string]Canon +}{ids: map[string]Canon{}} + +const canonMemoMax = 8192 + +// CanonicalOf reads one id as the model it names — see the file comment for +// the rules, in order. +func CanonicalOf(id string) Canon { + canonMemo.RLock() + c, ok := canonMemo.ids[id] + canonMemo.RUnlock() + if ok { + return c + } + c = canonicalOf(id) + canonMemo.Lock() + if len(canonMemo.ids) < canonMemoMax { + canonMemo.ids[id] = c + } + canonMemo.Unlock() + return c +} + +// canonicalOf is [CanonicalOf] without the memo. +func canonicalOf(id string) Canon { + s := strings.ToLower(strings.TrimSpace(id)) + s = strings.TrimPrefix(s, "~") + s = strings.TrimPrefix(s, "openrouter/") + if strings.HasPrefix(s, "accounts/") { + // accounts//models/: Fireworks' namespace for a model it + // hosts, with no owner in it. + if at := strings.Index(s, "/models/"); at >= 0 { + s = s[at+len("/models/"):] + } + } + var variant string + // The `:word` endings, last first: a route, a thinking level, a quant tag. + for { + at := strings.LastIndex(s, ":") + if at <= 0 { + break + } + tail := s[at+1:] + switch { + case routeSuffixes[tail], thinkingSuffixes[tail], tail == "latest": + s = s[:at] + continue + case quantTag(tail): + variant, s = tail, s[:at] + continue + } + break + } + if at := strings.LastIndex(s, "-"); at > 0 && (s[at+1:] == "gguf" || quantTag(s[at+1:])) && variant == "" { + variant, s = s[at+1:], s[:at] + } + vendor, name := "", s + if slash := strings.Index(s, "/"); slash >= 0 { + vendor, name = s[:slash], s[slash+1:] + } + name = versionDots(strings.ReplaceAll(name, "_", "-")) + if mapped, ok := vendorOrgs[vendor]; ok { + vendor = mapped + } + if vendor == "" { + for _, f := range familyVendors { + if strings.HasPrefix(name, f.prefix) { + vendor = f.vendor + break + } + } + } + out := name + if vendor != "" { + out = vendor + "/" + name + } + if alias, ok := modelAliases[out]; ok { + out = alias + } + if x := poolAliases(); x != nil { + out = strings.ToLower(x.Canonical(out)) + } + return Canon{ID: lineageTail(out), Variant: variant} +} + +// quantTag is whether a word names a quantisation: q4_k_m, q8_0, iq3_xs, +// fp8, fp16, bf16, int4, int8, awq, gptq. +func quantTag(word string) bool { + switch { + case word == "awq" || word == "gptq" || word == "bf16" || word == "gguf": + return true + case strings.HasPrefix(word, "fp") && allDigits(word[2:]): + return true + case strings.HasPrefix(word, "int") && allDigits(word[3:]): + return true + case strings.HasPrefix(word, "iq") && len(word) > 2 && word[2] >= '0' && word[2] <= '9': + return true + case strings.HasPrefix(word, "q") && len(word) > 1 && word[1] >= '0' && word[1] <= '9': + return true + } + return false +} + +// versionDots reads a version written with a `p` between two digits as the +// dot it stands for: `glm-5p3-flash` is `glm-5.3-flash`. +func versionDots(name string) string { + if !strings.Contains(name, "p") { + return name + } + b := []byte(name) + for i := 1; i+1 < len(b); i++ { + if b[i] == 'p' && b[i-1] >= '0' && b[i-1] <= '9' && b[i+1] >= '0' && b[i+1] <= '9' { + b[i] = '.' + } + } + return string(b) +} + +// DomainTuned is whether a model's own id says it was tuned for one domain +// other than code and general work — finance, medicine, law, a single subject, +// role-play — by a tag among the words of its name. Such a model is a poor +// seat for software work whatever its figures, and a rescue that must take a +// model it knows little about takes a general or a code model first. +func DomainTuned(id string) bool { + name := strings.ToLower(ShortModel(strings.TrimSuffix(id, ":free"))) + for _, word := range strings.FieldsFunc(name, func(r rune) bool { + return r == '-' || r == '_' || r == '.' || r == ':' || r == '/' + }) { + if domainTags[word] { + return true + } + } + return false +} + +// domainTags are the name words that mark a model tuned for one domain. +var domainTags = map[string]bool{ + "fin": true, "finance": true, "financial": true, "med": true, "medical": true, "medicine": true, + "bio": true, "biomed": true, "clinical": true, "health": true, "legal": true, "law": true, + "math": true, "maths": true, "chem": true, "chemistry": true, "roleplay": true, "rp": true, + "story": true, "storytelling": true, "novel": true, "creative": true, "translate": true, + "translation": true, "guard": true, "safety": true, "sante": true, "medic": true, + "pharma": true, "doctor": true, "bank": true, "banking": true, "trading": true, "tax": true, + "legalbench": true, "romance": true, "companion": true, "erp": true, +} + +// Tiny is whether a model's own id gives it fewer parameters than a crew seat +// can use — `lfm-2.5-2.6b`, `gemma-3n-e4b` — read off the largest size tag in +// its name (`30b-a3b` is thirty billion, three active). A name that gives no +// size is not called tiny. +func Tiny(id string) bool { + name := strings.ToLower(ShortModel(strings.TrimSuffix(id, ":free"))) + largest := 0.0 + for _, word := range strings.FieldsFunc(name, func(r rune) bool { + return r == '-' || r == '_' || r == ':' || r == '/' + }) { + word = strings.TrimPrefix(word, "e") + if !strings.HasSuffix(word, "b") { + continue + } + if n, err := strconv.ParseFloat(strings.TrimSuffix(word, "b"), 64); err == nil && n > largest { + largest = n + } + } + return largest > 0 && largest < tinyBelow +} + +// tinyBelow is the fewest billions of parameters a crew seat is given to. +const tinyBelow = 8.0 diff --git a/internal/crewroute/canonical_test.go b/internal/crewroute/canonical_test.go new file mode 100644 index 0000000000..05307e84ef --- /dev/null +++ b/internal/crewroute/canonical_test.go @@ -0,0 +1,104 @@ +package crewroute + +import ( + "math" + "strings" + "testing" +) + +// ONE MODEL, EVERY SPELLING: OpenRouter, Fireworks, a Hugging Face mirror, +// Together, a free pool, a dated snapshot and a local quantised pull of the +// same weights all read as one model — the +// quantised pull as a distinct variant of it. +func TestCanonicalReadsEveryProvidersSpellingOfOneModel(t *testing.T) { + cases := []struct { + id, want, variant string + }{ + {"z-ai/glm-5.3-flash", "z-ai/glm-5.3-flash", ""}, + {"openrouter/z-ai/glm-5.3-flash", "z-ai/glm-5.3-flash", ""}, + {"accounts/fireworks/models/glm-5p3-flash", "z-ai/glm-5.3-flash", ""}, + {"zai-org/GLM-5.3-Flash", "z-ai/glm-5.3-flash", ""}, + {"z-ai/glm_5.3_flash", "z-ai/glm-5.3-flash", ""}, + {"z-ai/glm-5.3-flash:free", "z-ai/glm-5.3-flash", ""}, + {"z-ai/glm-5.3-flash:nitro", "z-ai/glm-5.3-flash", ""}, + {"z-ai/glm-5.3-flash-20260301", "z-ai/glm-5.3-flash", ""}, + {"~z-ai/glm-5.3-flash-latest", "z-ai/glm-5.3-flash", ""}, + {"z-ai/glm-5.3-flash:high", "z-ai/glm-5.3-flash", ""}, + {"glm-5.3-flash:q4_k_m", "z-ai/glm-5.3-flash", "q4_k_m"}, + {"deepseek-ai/DeepSeek-V4-Flash", "deepseek/deepseek-v4-flash", ""}, + {"accounts/fireworks/models/kimi-k3", "moonshotai/kimi-k3", ""}, + {"meta-llama/Llama-3.3-70B-Instruct-Turbo", "meta-llama/llama-3.3-70b-instruct", ""}, + {"moonshotai/kimi-k3-fp8", "moonshotai/kimi-k3", "fp8"}, + } + for _, tc := range cases { + got := CanonicalOf(tc.id) + if got.ID != tc.want || got.Variant != tc.variant { + t.Errorf("CanonicalOf(%q) = %+v, want %q variant %q", tc.id, got, tc.want, tc.variant) + } + } +} + +// AN UNKNOWN SPELLING STAYS ITS OWN MODEL: nothing is merged on a likeness. +func TestCanonicalNeverMergesOnALikeness(t *testing.T) { + for _, pair := range [][2]string{ + {"z-ai/glm-5.3-flash", "z-ai/glm-5.3"}, + {"somelab/coder-7b", "otherlab/coder-7b"}, + {"thinkingmachines/inkling-small", "thinkingmachines/inkling"}, + } { + if Canonical(pair[0]) == Canonical(pair[1]) { + t.Errorf("%q and %q were read as one model", pair[0], pair[1]) + } + } + if got := Canonical("somelab/mystery-model"); got != "somelab/mystery-model" { + t.Errorf("an unknown id was rewritten to %q", got) + } +} + +// A QUANTISED COPY IS NOT ITS MODEL, and is credited a share of its model's +// quality. +func TestAQuantisedCopyIsCreditedAShareOfItsModel(t *testing.T) { + tab := prior() + local := glmFlash + local.ID = "glm-5.3-flash:q4_k_m" + if Lineage(local.ID) == Lineage(glmFlash.ID) || !strings.Contains(Lineage(local.ID), "@q4_k_m") { + t.Fatalf("the quantised copy shares the model's identity: %q", Lineage(local.ID)) + } + full, _ := tab.quality(OpenEnded, Worker, glmFlash) + got, _ := tab.quality(OpenEnded, Worker, local) + if want := full * quantDiscount; full <= 0 || math.Abs(got-want) > 1e-9 { + t.Fatalf("the copy reads %v, want %v", got, want) + } +} + +// A MODEL'S NAME SAYS WHEN IT WAS TUNED FOR ONE DOMAIN, and a general or a +// code model's does not. +func TestDomainTunedModelsAreToldByTheirNames(t *testing.T) { + for _, id := range []string{"inclusionai/ling-3.0-flash-fin:free", "vendor/med-llama-70b", "acme/legal-7b", "x/qwen-math-72b", "y/roleplay-13b:free"} { + if !DomainTuned(id) { + t.Errorf("%s reads as general", id) + } + } + for _, id := range []string{"poolside/laguna-s-2.1:free", "z-ai/glm-5.3-flash", "qwen/qwen3-coder:free", "google/gemma-4-31b-it:free", "cohere/north-mini-code:free"} { + if DomainTuned(id) { + t.Errorf("%s reads as domain-tuned", id) + } + } +} + +// A MODEL'S NAME SAYS WHEN IT IS TOO SMALL FOR A SEAT, and a name with no size +// or a large one does not. +func TestTinyModelsAreToldByTheirNames(t *testing.T) { + for _, id := range []string{"liquid/lfm-2.5-2.6b:free", "google/gemma-3n-e4b-it:free", "meta-llama/llama-3.2-3b-instruct"} { + if !Tiny(id) { + t.Errorf("%s reads as big enough", id) + } + } + for _, id := range []string{"nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free", "z-ai/glm-5.3-flash", "qwen/qwen3-32b", "poolside/laguna-s-2.1:free"} { + if Tiny(id) { + t.Errorf("%s reads as tiny", id) + } + } + if !DomainTuned("inclusionai/ling-3.0-flash-sante:free") { + t.Error("a health-tuned model reads as general") + } +} diff --git a/internal/crewroute/classify.go b/internal/crewroute/classify.go new file mode 100644 index 0000000000..86cadb2d90 --- /dev/null +++ b/internal/crewroute/classify.go @@ -0,0 +1,424 @@ +package crewroute + +import ( + "regexp" + "strings" +) + +// WHAT KIND OF WORK A TASK IS, READ OFF ITS OWN WORDS. +// +// The router routes on a task's CLASS and on nothing finer, by design +// (docs/design/model-pool/pareto-crewing.pdf): the class decides the crew. A +// narrow fix goes to the cheapest qualified crew, and open-ended work (a +// feature, a refactor, docs, a design) gets a strong checker behind it. So the +// question this file answers is the only one the router asks of the text: +// which of the two is this? +// +// ── THE SIGNALS ── +// +// It is rules, not a model, and every rule is written down here so a person +// asking "why did it call my task a bugfix" is answered by reading them: +// +// - LABELS weigh most. An issue labelled `bug` or `enhancement` is a person +// telling us the answer. +// - THE TITLE weighs more than the body. A conventional-commit prefix +// (`fix:`, `feat:`, `refactor:`), a `[bug]` tag or a leading verb +// (`add`, `support`, `document`) is the author classifying their own work. +// - THE BODY adds the rest: a stack trace, an exception's name, "crashes", +// "regression", "steps to reproduce" lean narrow; "feature request", +// "would be great", "consider adding", "refactor" lean open-ended. +// +// A harness's own wrapper around an issue — "Implement issue #12: …" and the +// standing instruction to keep the test suite green that follows every task +// it hands out — is read past, because it is the same sentence on every task +// and a signal that fires on everything tells the router nothing. +// +// ── UNCERTAIN IS OPEN-ENDED ── +// +// When the signals do not clearly favour one class, the answer is +// [OpenEnded], and that is the safe side on purpose: calling a narrow fix +// open-ended costs a strong checker it did not need — cents — while calling +// open-ended work a narrow fix sends it to a crew without the checker that +// work needs. The two mistakes are not the same size, so the tie does not go +// to the cheaper one. + +// Class is the kind of work a task is. It is a string because it is written +// into the router's event log and onto a task's card, and a script reading +// either reads the word. +type Class string + +const ( + // Bugfix is a NARROW, VERIFIABLE change: a defect, a regression, a missing + // check, a test that should exist. It goes to the cheapest qualified crew. + Bugfix Class = "bugfix" + // OpenEnded is work whose shape the task does not fix: a feature, a + // refactor, documentation, a design. It gets a strong checker: the checker + // is the seat worth paying for here. + OpenEnded Class = "openended" + // Other is work that changes nothing in particular — a question, an + // investigation, a review. It is read on the average of the other two + // classes' links, because the weights carry none of its own. + Other Class = "other" +) + +// Classes lists the three, in the order a report prints them. +// Word is the class as a person reads it: `open-ended` rather than the +// logged `openended`, which stays the key the router's log and the evidence +// table are written in. +func (c Class) Word() string { + if c == OpenEnded { + return "open-ended" + } + return string(c) +} + +var Classes = []Class{Bugfix, OpenEnded, Other} + +// Task is what the classifier reads: the words a person or an issue gave the +// work, and the labels the issue carried when it came from a tracker. +type Task struct { + Text string + Labels []string +} + +// Reading is the classifier's answer: the class, the one line saying which +// signal decided it, and whether the signals were clear. Sure false is the +// uncertain case, which always reads [OpenEnded]. +type Reading struct { + Class Class + Why string + Sure bool + // Complex is, for a bugfix, the signals that make it a fix with reach + // ([complexFix]), in words; empty is a simple fix. It names no class of + // its own — the task is still a bugfix on every line a person reads — and + // it moves only the worker ([Decide]). + Complex string +} + +// signal is one rule: a pattern, the class it leans toward, how hard, and +// the words the reading names it by when it decides. +type signal struct { + pattern *regexp.Regexp + class Class + weight int + name string +} + +// The weights, spelled once. A label is a person's own answer and outweighs +// anything read off prose; a title is the author's own summary and outweighs +// a body, where the same word may be describing somebody else's code. +const ( + weightLabel = 4 + weightTitle = 2 + weightBody = 1 + // clearMargin is how far one class must lead the other before the reading + // is SURE. One point is a single body word, which is not a decision. + clearMargin = 2 +) + +// titleSignals are read off the first line of the task, after any harness +// wrapper is removed ([taskTitle]). +var titleSignals = []signal{ + {regexp.MustCompile(`(?i)^\s*(fix|bugfix|hotfix)(\([^)]*\))?!?:`), Bugfix, 2, "a fix: title"}, + {regexp.MustCompile(`(?i)^\s*(test|tests|ci|chore|build)(\([^)]*\))?!?:`), Bugfix, 1, "a narrow chore title"}, + {regexp.MustCompile(`(?i)\[\s*bug\s*\]|^\s*bug\s*[:\-]|^\s*BUG\b`), Bugfix, 2, "a bug tag in the title"}, + {regexp.MustCompile(`(?i)\b(crash(es|ed|ing)?|error|exception|fails?|failing|broken|regression|drops?|rejects?|(is |are |gets? |being )?rejected|ignored|incorrect(ly)?|wrong(ly)?|overflow\w*|hangs?|leaks?)\b`), Bugfix, 1, "a failure word in the title"}, + {regexp.MustCompile(`(?i)\binstead of\b`), Bugfix, 1, "what happens set against what should"}, + // A MECHANICAL CHANGE — a rename, a version bump — is small and exact + // work however far across the tree it reaches: the fix's crew, not the + // open-ended one. + {mechanicalEdit, Bugfix, 2, "a mechanical change"}, + {regexp.MustCompile(`\b[A-Z][A-Za-z]+(Error|Exception)\b`), Bugfix, 1, "an exception named in the title"}, + // A DEFECT SAID IN PLAIN WORDS is a fix whether or not it wears a `fix:` + // prefix: a leading verb that repairs something, the word bug or broken, + // and a sentence that sets what happens against what should. + {regexp.MustCompile(`(?i)^\s*(please\s+)?(fix|repair|correct|resolve|patch)\b`), Bugfix, 2, "a title that asks for a repair"}, + {regexp.MustCompile(`(?i)\b(bug|bugs|buggy|broken)\b`), Bugfix, 1, "the word bug or broken"}, + {defectSentence, Bugfix, 2, "what happens set against what should"}, + // A TITLE THAT SAYS WHAT THE CODE DOES NOT DO is a defect report: "never + // keeps a square image", "does not close the file", "fails to parse", + // "should reject". "should support" and its kin ask for something new and + // are weighed back on the open-ended side below. + {regexp.MustCompile(`(?i)\b(never|doesn't|doesn’t|does not|don't|don’t|do not|fails to|failed to|cannot|can't|can’t|won't|won’t|no longer|should(n't|n’t| not)?)\b`), Bugfix, 1, "a title that says what the code does not do"}, + // A CALL WRITTEN OUT IN THE TITLE — `RandomResizedCrop(scale=(1, 1))`, + // `parse("")` — is somebody quoting the input that misbehaves. + {regexp.MustCompile(`\b[A-Za-z_][\w.]*\((?:[^()]|\([^()]*\))+\)`), Bugfix, 1, "a call written out in the title"}, + {regexp.MustCompile(`(?i)\bshould (also )?(support|allow|accept|provide|offer|expose|have|be able)\b`), OpenEnded, 1, "a title asking for something the code should also do"}, + {regexp.MustCompile(`(?i)^\s*(feat|feature|refactor|perf|docs?)(\([^)]*\))?!?:`), OpenEnded, 2, "a feature or refactor title"}, + {regexp.MustCompile(`(?i)^\s*(add|support|implement|introduce|enable|allow|create|design|define|document|redesign|rework|restructure|migrate|extend|expose|make)\b`), OpenEnded, 2, "a title that asks for something new"}, + {regexp.MustCompile(`(?i)\b(consider|proposal|rfc|epic|feature|enhancement|refactor|docs|documentation|more pythonic|should (probably )?(default|be|support|allow)|fractional|metadata)\b`), OpenEnded, 1, "an open-ended word in the title"}, +} + +// bodySignals are read off everything after the title. +var bodySignals = []signal{ + {regexp.MustCompile(`Traceback \(most recent call last\)|(?m)^\s+at [\w.$]+\(|panic: |goroutine \d+ \[`), Bugfix, 2, "a stack trace"}, + {regexp.MustCompile(`\b[A-Z][A-Za-z]*(Error|Exception)\b`), Bugfix, 1, "an exception in the text"}, + {regexp.MustCompile(`(?i)\b(steps to reproduce|to reproduce|describe the bug|expected behaviou?r|actual behaviou?r|minimal (reproducible )?example)\b`), Bugfix, 2, "a bug report's own headings"}, + {defectSentence, Bugfix, 2, "what happens set against what should"}, + {regexp.MustCompile(`(?i)\b(failing|failed|fails) (unit |integration )?tests?\b|\btests? (is |are )?(failing|fail|broken)\b`), Bugfix, 1, "a failing test named"}, + {regexp.MustCompile(`(?i)\b(crash(es|ed)?|regression|exit code \d+|stack ?trace|segfault|fails with|raises|silently (drops?|discards?|ignores?)|returns? (the )?wrong)\b`), Bugfix, 1, "a failure described in the text"}, + {regexp.MustCompile(`(?i)\b(is your feature request|feature request|would be (really )?(great|nice|useful|helpful)|it'?d be great|consider adding|nice to have|enhancement|design (doc|proposal)|user stor(y|ies))\b`), OpenEnded, 2, "a feature request's own words"}, + {regexp.MustCompile(`(?i)\b(refactor(ing)?|restructur(e|ing)|redesign|extensib(le|ility)|new (api|option|parameter|feature|command|endpoint|page|metric)s?|not well documented|documentation)\b`), OpenEnded, 1, "open-ended work described in the text"}, +} + +// otherSignals mark work that changes nothing in particular. They are read +// off the title only, because a question's body is full of the words a bug +// report uses. +var otherTitle = regexp.MustCompile(`(?i)^\s*(explain|investigate|why\b|how (do|does|can|should)|what (is|does|are)|review|audit|summari[sz]e|analy[sz]e|research|benchmark|compare)\b`) + +// labelWords maps an issue label, folded, to the class it names. +var labelWords = map[string]Class{ + "bug": Bugfix, "type: bug": Bugfix, "kind/bug": Bugfix, "defect": Bugfix, "regression": Bugfix, + "crash": Bugfix, "type:bug": Bugfix, "bugfix": Bugfix, "fix": Bugfix, + "enhancement": OpenEnded, "feature": OpenEnded, "feature request": OpenEnded, "type: feature": OpenEnded, + "kind/feature": OpenEnded, "refactor": OpenEnded, "documentation": OpenEnded, "docs": OpenEnded, + "design": OpenEnded, "proposal": OpenEnded, "rfc": OpenEnded, "epic": OpenEnded, + "question": Other, "investigation": Other, "discussion": Other, +} + +// labelKeys are the words a tracker's own label spellings are built on — +// `bug :bug:`, `type/bug`, `Kind: Enhancement ✨` — read when the label is +// not one of [labelWords] as written. Labels with none of them (`help +// wanted`, `good first issue`) say nothing about the kind of work. +var labelKeys = map[string]Class{ + "bug": Bugfix, "bugs": Bugfix, "defect": Bugfix, "regression": Bugfix, "crash": Bugfix, "bugfix": Bugfix, + "enhancement": OpenEnded, "feature": OpenEnded, "refactor": OpenEnded, "documentation": OpenEnded, + "docs": OpenEnded, "proposal": OpenEnded, "rfc": OpenEnded, "epic": OpenEnded, + "question": Other, "investigation": Other, "discussion": Other, +} + +// labelWordRun is one run of letters in a folded label. +var labelWordRun = regexp.MustCompile(`[a-z]+`) + +// labelClass is the class a label names: as written first, then word by +// word, with an emoji shortcode (`:bug:`) read as its word. +func labelClass(label string) (Class, bool) { + folded := strings.ToLower(strings.TrimSpace(label)) + if class, ok := labelWords[folded]; ok { + return class, true + } + if strings.Contains(folded, "🐛") { + return Bugfix, true + } + for _, word := range labelWordRun.FindAllString(folded, -1) { + if class, ok := labelKeys[word]; ok { + return class, true + } + } + return "", false +} + +// wrapperLine is the harness's own sentence around an issue's title: +// "Implement issue #12: [bug] export drops the last row". The verb is +// the harness talking and the tail is the issue's own title. +var wrapperLine = regexp.MustCompile(`(?i)^\s*(implement|fix|resolve|address|solve|close|work on)\s+(github\s+)?issue\s+#?\d+\s*[:\-—]\s*(.*)$`) + +// defectSentence is a sentence that sets what happens against what should: +// "returns a - b instead of a + b", "should keep the order but drops it", +// "expected 3, got 4". +var defectSentence = regexp.MustCompile(`(?i)\b(returns?|gives?|prints?|produces?|yields?|shows?)\b[^.]*\binstead of\b|\bshould\b[^.]*\bbut\b|\bexpected\b[^.]*\b(got|but|received|instead)\b`) + +// mechanicalEdit is a title that asks for a rename or a version bump. +var mechanicalEdit = regexp.MustCompile(`(?i)^\s*(please\s+)?(rename|bump|update (the )?(version|copyright)|upgrade (the )?version)\b`) + +// tinyEdit is a change too small to be open-ended whatever verb it opens +// with: a docstring, a typo, a comment, one line. It reads as work of no +// particular class — the cheap crew — and it outweighs a leading "add". +var tinyEdit = regexp.MustCompile(`(?i)\b(docstring|docstrings|typo|typos|spelling|misspell\w*|one[- ]line|single[- ]line|whitespace|a comment|code comment|trailing comma)\b`) + +// boilerplate is the standing instruction a harness appends to every task. +// It is cut before the body is read, because it is the same on every task. +var boilerplate = regexp.MustCompile(`(?is)work in this repository\..*$`) + +// Classify reads a task's class. It is pure and allocation-light: the rules +// are compiled once at load, and a task is read in a single pass per rule. +func Classify(task Task) Reading { + title, body := taskTitle(task.Text) + var score [3]int + best := map[Class]signal{} + note := func(s signal, weight int) { + score[classIndex(s.class)] += weight + if held, ok := best[s.class]; !ok || weight > held.weight { + best[s.class] = signal{class: s.class, weight: weight, name: s.name} + } + } + labelled := map[Class]bool{} + for _, label := range task.Labels { + if class, ok := labelClass(label); ok { + labelled[class] = true + note(signal{class: class, name: "the `" + strings.TrimSpace(label) + "` label"}, weightLabel) + } + } + if otherTitle.MatchString(title) { + note(signal{class: Other, name: "a title asking a question rather than for a change"}, weightTitle+1) + } + if tinyEdit.MatchString(title) { + note(signal{class: Other, name: "a small scoped edit"}, 2*weightTitle+1) + } + for _, s := range titleSignals { + if s.pattern.MatchString(title) { + note(s, s.weight*weightTitle) + } + } + for _, s := range bodySignals { + if s.pattern.MatchString(body) { + note(s, s.weight*weightBody) + } + } + reading := decide(score, best, labelled) + if reading.Class == Bugfix { + reading.Complex = complexFix(reachText(title, body)) + } + return reading +} + +// decide is the reading the scores make. A task a person labelled a bug and +// nobody labelled anything else is a bugfix whatever its prose leaves unsaid: +// the label is the answer, and the open-ended default is for tasks nobody +// answered. +func decide(score [3]int, best map[Class]signal, labelled map[Class]bool) Reading { + bug, open, other := score[0], score[1], score[2] + switch { + case other > bug && other > open: + return Reading{Class: Other, Why: best[Other].name, Sure: other-max(bug, open) >= clearMargin} + case bug >= open+clearMargin: + return Reading{Class: Bugfix, Why: best[Bugfix].name, Sure: true} + case open >= bug+clearMargin: + return Reading{Class: OpenEnded, Why: best[OpenEnded].name, Sure: true} + case labelled[Bugfix] && !labelled[OpenEnded]: + return Reading{Class: Bugfix, Why: best[Bugfix].name, Sure: true} + case bug == 0 && open == 0: + return Reading{Class: OpenEnded, Why: "nothing in the task says which kind of work it is", Sure: false} + } + return Reading{Class: OpenEnded, Why: "the task reads both ways, so it gets the stronger checker", Sure: false} +} + +// classIndex is a class's slot in the score array. +func classIndex(class Class) int { + switch class { + case Bugfix: + return 0 + case OpenEnded: + return 1 + } + return 2 +} + +// taskTitle splits a task into the line that names it and the rest. The +// first non-blank line is the title, a Markdown heading's hashes and a +// harness wrapper removed; when the wrapper carries the title, the heading +// line above it (usually the same words) is dropped as a duplicate. +func taskTitle(text string) (title, body string) { + text = boilerplate.ReplaceAllString(text, "") + lines := strings.Split(text, "\n") + at := -1 + for i, line := range lines { + if strings.TrimSpace(line) != "" { + at = i + break + } + } + if at < 0 { + return "", "" + } + title = strings.TrimSpace(strings.TrimLeft(strings.TrimSpace(lines[at]), "#")) + rest := lines[at+1:] + if m := wrapperLine.FindStringSubmatch(title); m != nil { + title = strings.TrimSpace(m[3]) + } + // A wrapper line further down names the same issue again; it is read as + // the title's second spelling and not as body text. + kept := rest[:0:0] + for _, line := range rest { + if m := wrapperLine.FindStringSubmatch(strings.TrimSpace(line)); m != nil { + if title == "" { + title = strings.TrimSpace(m[3]) + } + continue + } + kept = append(kept, line) + } + return title, strings.Join(kept, "\n") +} + +// A FIX WITH REACH IS NOT A ONE-LINE FIX. The cheapest worker is the pick for +// fixes of the ordinary size, and a defect whose repair crosses files, a wire +// contract or a language's rules +// is a different job wearing the same word: the cheap worker's patch is the +// narrow one, and the checker then rejects it or — worse — passes it. So a +// bugfix is read once more for REACH, off its body, on signals that say what +// the repair must hold together rather than what broke: +// +// - more than one source file named; +// - an API, an endpoint, a status code or a protocol; +// - language rules — i18n, locales, plurals, grammar, Unicode; +// - a long report, or more than one reproduction; +// - existing tests, guards or behaviour that must keep passing; +// - a security defect — traversal, injection, a bypass — whose repair must +// close every path to it. +// +// Two of them make a complex fix; one alone is how an ordinary report reads. +// A complex fix moves its worker one rung up the seat's front ([Decide]); its +// planner and checker are the fix's own, and a pin is never overruled. + +// reachSignal is one of the signals above, and the words it is named by. +type reachSignal struct { + name string + hit func(body string) bool +} + +// The patterns the reach signals read. +var ( + sourceFile = regexp.MustCompile(`\b[\w./-]+\.(py|go|ts|tsx|js|jsx|mjs|rs|java|kt|rb|c|cc|cpp|h|hpp|cs|php|swift|scala|ex|exs|vue|svelte)\b`) + wireWords = regexp.MustCompile(`(?i)\b(api|apis|endpoints?|status codes?|http ?[1-5]\d\d|[1-5]\d\d (error|response)|protocols?|grpc|websockets?|rpc|wire format|openapi)\b`) + languageRul = regexp.MustCompile(`(?i)\b(i18n|l10n|locales?|locali[sz]ation|internationali[sz]ation|translations?|plurali[sz]ation|plurals?|grammar|language rules?|unicode|utf-?8|diacritics?|right-to-left|rtl)\b`) + keepPassing = regexp.MustCompile(`(?i)\b(existing|current|other|all)( [\w-]+){0,2} (tests?|checks?|guards?|behaviou?r)\b[^.]{0,60}\b(pass|passing|keep|kept|still|remain|break|breaking|unchanged)|\bwithout breaking\b|\bbackwards? compat`) + reproMark = regexp.MustCompile(`(?im)^\s*#+\s*(repro|reproduction|to reproduce|steps to reproduce)\b|^\s*(repro|reproduction) \d`) + manyRepros = regexp.MustCompile(`(?i)\b(two|three|four|five|several|multiple|many|\d+) (repros|reproductions|reproducers|failing cases)\b`) + securityFix = regexp.MustCompile(`(?i)\b(security|vulnerab\w*|cve-\d+|path traversal|directory traversal|injection|xss|csrf|ssrf|auth(entication|orization)? bypass|privilege escalation|sanitis\w*|sanitiz\w*|escap(e|ing) (user )?input)\b`) +) + +// complexLongBody is how long a report is, after the title, before its length +// alone is a signal of reach: a few paragraphs, or a trace and a table. +const complexLongBody = 2000 + +var reachSignals = []reachSignal{ + {"more than one file", func(body string) bool { + seen := map[string]bool{} + for _, f := range sourceFile.FindAllString(body, -1) { + seen[f] = true + } + return len(seen) > 1 + }}, + {"an API or protocol", wireWords.MatchString}, + {"language rules", languageRul.MatchString}, + {"a long report or several repros", func(body string) bool { + return len(body) > complexLongBody || strings.Count(body, "```") >= 4 || len(reproMark.FindAllString(body, -1)) > 1 || + manyRepros.MatchString(body) + }}, + {"a security fix", securityFix.MatchString}, + {"existing tests that must keep passing", keepPassing.MatchString}, +} + +// reachText is what a fix is read for reach in: its body, or — for a task +// written as one paragraph, which is the whole of most asks — its title. +func reachText(title, body string) string { + if strings.TrimSpace(body) == "" { + return title + } + return body +} + +// complexFix is the reach a fix's body shows, in words; empty is a simple fix. +func complexFix(body string) string { + var hits []string + for _, s := range reachSignals { + if s.hit(body) { + hits = append(hits, s.name) + } + } + if len(hits) < 2 { + return "" + } + return strings.Join(hits, ", ") +} diff --git a/internal/crewroute/classify_test.go b/internal/crewroute/classify_test.go new file mode 100644 index 0000000000..fbc183dc58 --- /dev/null +++ b/internal/crewroute/classify_test.go @@ -0,0 +1,105 @@ +package crewroute + +import "testing" + +func TestClassifyReadsTheKindOfWork(t *testing.T) { + cases := []struct { + name string + task Task + class Class + sure bool + }{ + {"a fix: title", Task{Text: "fix: avoid an overflow when formatting very large integers"}, Bugfix, true}, + {"a bug tag", Task{Text: "[bug] export drops the last row of a table"}, Bugfix, true}, + {"a stack trace", Task{Text: "config loader\n\nTraceback (most recent call last):\n File \"x.py\", line 3\nKeyError: 'a'"}, Bugfix, true}, + {"a crash in the title", Task{Text: "the diff tool crashes on repositories without an initial commit"}, Bugfix, true}, + {"a bug label", Task{Text: "the chart looks odd", Labels: []string{"bug"}}, Bugfix, true}, + {"a feature request", Task{Text: "Consider adding a percentile metric\n\n**Is your feature request related to a problem?**"}, OpenEnded, true}, + {"a refactor title", Task{Text: "refactor(api): split the handler registry into modules"}, OpenEnded, true}, + {"add something", Task{Text: "Add a --json flag to the status command"}, OpenEnded, true}, + {"docs", Task{Text: "docs: explain the retry settings"}, OpenEnded, true}, + {"an enhancement label", Task{Text: "the chart looks odd", Labels: []string{"enhancement"}}, OpenEnded, true}, + {"a question", Task{Text: "Why does the scheduler skip the first job?"}, Other, true}, + {"nothing to go on", Task{Text: "the widget"}, OpenEnded, false}, + {"empty", Task{}, OpenEnded, false}, + } + for _, tc := range cases { + got := Classify(tc.task) + if got.Class != tc.class || got.Sure != tc.sure { + t.Errorf("%s: %s (sure %v, %q), want %s (sure %v)", tc.name, got.Class, got.Sure, got.Why, tc.class, tc.sure) + } + if got.Why == "" { + t.Errorf("%s: a reading with no reason", tc.name) + } + } +} + +// A HARNESS'S WRAPPER IS READ PAST: every task it hands out says "Implement +// issue #N" and "make the existing test suite pass", and a signal that fires on +// everything tells the router nothing. +func TestClassifyReadsPastAHarnessWrapper(t *testing.T) { + wrapped := "Implement issue #12: [bug] export drops the last row of a table\n\nThe loop stops one short.\n\n" + + "Work in this repository. Implement the change and make the existing test suite pass. Do not weaken or delete tests to make them pass." + if got := Classify(Task{Text: wrapped}); got.Class != Bugfix { + t.Errorf("wrapped fix read as %s (%q)", got.Class, got.Why) + } + feature := "Implement issue #34: Remember the window size between sessions\n\nIt would be really useful to keep it for next time.\n\n" + + "Work in this repository. Implement the change and make the existing test suite pass." + if got := Classify(Task{Text: feature}); got.Class != OpenEnded { + t.Errorf("wrapped feature read as %s (%q)", got.Class, got.Why) + } +} + +func TestParseEffortTakesAPersonsWords(t *testing.T) { + for word, want := range map[string]Effort{"": EffortKnee, "best": EffortBest, "thorough": EffortBest, "cheap": EffortCheap} { + if got, ok := ParseEffort(word); !ok || got != want { + t.Errorf("ParseEffort(%q) = %q, %v", word, got, ok) + } + } + if _, ok := ParseEffort("medium"); ok { + t.Error("an unknown effort word was accepted") + } +} + +func TestLineageFoldsSnapshotsAndLevels(t *testing.T) { + cases := map[string]string{ + "deepseek/deepseek-v4-flash-0731": "deepseek/deepseek-v4-flash", + "~deepseek/deepseek-v4-flash-latest": "deepseek/deepseek-v4-flash", + "moonshotai/kimi-k3:high": "moonshotai/kimi-k3", + "Z-AI/GLM-5.3-Flash": "z-ai/glm-5.3-flash", + "openai/gpt-5.5": "openai/gpt-5.5", + "qwen/qwen3.8-max-0902": "qwen/qwen3.8-max", + } + for in, want := range cases { + if got := Lineage(in); got != want { + t.Errorf("Lineage(%q) = %q, want %q", in, got, want) + } + } +} + +// A DEFECT SAID IN PLAIN WORDS IS A FIX, a tiny edit is small whatever verb it +// opens with, and a genuinely open ask stays open-ended. +func TestClassifyPlainDefectsAndTinyEdits(t *testing.T) { + cases := []struct { + text string + class Class + }{ + {"Fix the bug in calc.py: add(a, b) returns a - b instead of a + b.", Bugfix}, + {"Repair the export so it keeps the header row", Bugfix}, + {"The pager is broken on narrow terminals", Bugfix}, + {"sorting should keep equal items in order but it reorders them", Bugfix}, + {"parse_date: expected 2024-01-02, got 2024-02-01", Bugfix}, + {"tests/test_lists.py is failing: natural_list drops middle items", Bugfix}, + {"add a one-line docstring to the helper in utils.py", Other}, + {"fix a typo in the README", Other}, + {"Add support for YAML config files", OpenEnded}, + {"refactor the storage layer into separate packages", OpenEnded}, + {"Design a plugin system for exporters", OpenEnded}, + {"do a deep research on opensource coding agents and harnesses please", OpenEnded}, + } + for _, tc := range cases { + if got := Classify(Task{Text: tc.text}); got.Class != tc.class { + t.Errorf("%q: %s (%q), want %s", tc.text, got.Class, got.Why, tc.class) + } + } +} diff --git a/internal/crewroute/fixtures_test.go b/internal/crewroute/fixtures_test.go new file mode 100644 index 0000000000..736517a897 --- /dev/null +++ b/internal/crewroute/fixtures_test.go @@ -0,0 +1,53 @@ +package crewroute + +import "time" + +// released is a catalog row's release date. +func released(year int, month time.Month, day int) time.Time { + return time.Date(year, month, day, 0, 0, 0, 0, time.UTC) +} + +// Catalog rows as a live catalog lists them: per-token prices, context, open +// weights, the published indexes and design-arena Elo, and the release date. +var ( + glmFlash = Model{ID: "z-ai/glm-5.3-flash", Open: true, PromptPrice: 4.5e-8, CompletionPrice: 6e-7, CacheReadPrice: 2.85e-8, + Intelligence: 41.8, Coding: 71.5, Agentic: 50.9, ArenaElo: 1348, Context: 1310720, Released: released(2026, 8, 26), Tools: true} + kimiK3 = Model{ID: "moonshotai/kimi-k3", Open: true, PromptPrice: 3e-6, CompletionPrice: 1.5e-5, CacheReadPrice: 3e-7, + Intelligence: 43.6, Coding: 76.2, Agentic: 50, ArenaElo: 1421, Context: 1048576, Released: released(2026, 7, 15), Tools: true} + v4Flash = Model{ID: "deepseek/deepseek-v4-flash", Open: true, PromptPrice: 7.168e-8, CompletionPrice: 1.4336e-7, CacheReadPrice: 1.4336e-8, + Intelligence: 24.2, Coding: 56.2, Agentic: 22.2, ArenaElo: 1216, Context: 1048576, Released: released(2026, 4, 23), Tools: true} + glm53 = Model{ID: "z-ai/glm-5.3", Open: true, PromptPrice: 1.4e-6, CompletionPrice: 4.4e-6, CacheReadPrice: 2.6e-7, + Intelligence: 44.8, Coding: 74.8, Agentic: 53.1, ArenaElo: 1387, Context: 1310720, Released: released(2026, 8, 16), Tools: true} + opus5 = Model{ID: "anthropic/claude-opus-5", PromptPrice: 5e-6, CompletionPrice: 2.5e-5, CacheReadPrice: 5e-7, + Intelligence: 50.8, Coding: 78, Agentic: 56.5, ArenaElo: 1372, Context: 1000000, Released: released(2026, 7, 23), Tools: true} + fable51 = Model{ID: "anthropic/claude-fable-5.1", PromptPrice: 1e-5, CompletionPrice: 5e-5, CacheReadPrice: 2.5e-7, + Intelligence: 53.4, Coding: 81.6, Agentic: 57.9, ArenaElo: 1421, Context: 1000000, Released: released(2026, 8, 31), Tools: true} + astra6 = Model{ID: "openai/gpt-6-astra", PromptPrice: 1e-5, CompletionPrice: 5e-5, CacheReadPrice: 1e-6, + Intelligence: 52.7, Coding: 76.9, Agentic: 51, Context: 1050000, Released: released(2026, 9, 3), Tools: true} +) + +// candidateOf is a model reachable on the default service only. +func candidateOf(m Model) Candidate { + return Candidate{Model: m, Routes: []Route{{Provider: "openrouter", Send: m.ID, Kind: Metered}}} +} + +// catalogCandidates are three open models across the price range, each on the +// default service. +func catalogCandidates() []Candidate { + return []Candidate{candidateOf(glmFlash), candidateOf(kimiK3), candidateOf(v4Flash)} +} + +// frontierCandidates are the three open models with a mid-price model and the +// dearest frontier rows beside them. +func frontierCandidates() []Candidate { + return append(catalogCandidates(), candidateOf(glm53), candidateOf(opus5), candidateOf(fable51), candidateOf(astra6)) +} + +// catalogRow is a model with published figures and no release date. +func catalogRow(id string, open bool, in, out float64, intel, coding, agentic float64) Candidate { + return Candidate{ + Model: Model{ID: id, Open: open, PromptPrice: in / 1e6, CompletionPrice: out / 1e6, CacheReadPrice: in / 1e7, + Intelligence: intel, Coding: coding, Agentic: agentic, Context: 1_000_000, Tools: true}, + Routes: []Route{{Provider: "openrouter", Send: id, Kind: Metered}}, + } +} diff --git a/internal/crewroute/law_test.go b/internal/crewroute/law_test.go new file mode 100644 index 0000000000..3ea2efa050 --- /dev/null +++ b/internal/crewroute/law_test.go @@ -0,0 +1,45 @@ +package crewroute + +import ( + "go/ast" + "go/parser" + "go/token" + "os" + "strings" + "testing" +) + +// NO PROVIDER IS NAMED IN THE ROUTER. A route is a provider, a send id, a kind +// and a price; which provider is behind it is the caller's to know, and a +// branch on one provider's name here would be a rule the next provider does +// not get. The id-spelling tables (canonical.go) read how providers SPELL +// model ids, which is data about ids, and are the one file allowed to. +func TestTheRouterNamesNoProvider(t *testing.T) { + names := []string{"openrouter", "fireworks", "together", "ollama", "codex", "chatgpt", "groq", "deepinfra", "anthropic", "cloudflare"} + entries, err := os.ReadDir(".") + if err != nil { + t.Fatal(err) + } + for _, entry := range entries { + name := entry.Name() + if !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") || name == "canonical.go" { + continue + } + file, err := parser.ParseFile(token.NewFileSet(), name, nil, 0) + if err != nil { + t.Fatal(err) + } + ast.Inspect(file, func(n ast.Node) bool { + lit, ok := n.(*ast.BasicLit) + if !ok || lit.Kind != token.STRING { + return true + } + for _, provider := range names { + if strings.Contains(strings.ToLower(lit.Value), provider) { + t.Errorf("%s names the provider %q in %s", name, provider, lit.Value) + } + } + return true + }) + } +} diff --git a/internal/crewroute/prior.go b/internal/crewroute/prior.go new file mode 100644 index 0000000000..db5bf180ca --- /dev/null +++ b/internal/crewroute/prior.go @@ -0,0 +1,698 @@ +package crewroute + +import ( + _ "embed" + "encoding/json" + "math" + "sort" + "strings" + "sync" + "time" +) + +// WHAT A SEAT IS WORTH, BEFORE THIS INSTALL HAS RUN ANYTHING. +// +// The router needs, for every class of work, every seat and every model it +// might sit there, two numbers: how much that model in that seat adds to the +// work's quality, and what it costs. Both are read off the model's own +// catalog row through fitted weights (prior.json, embedded). No model has a +// row of its own there: every model, whoever makes it, is scored the same way. +// +// 1. ABILITY FROM METADATA. A model's ability is a latent number predicted +// from the fields its catalog row carries. The weights are a joint +// Gaussian over the ability and those fields ([weights.Mean], +// [weights.Cov]); a row is scored by conditioning on the fields it HAS, +// and a field it lacks is integrated out, which widens the variance. +// A row that publishes any index — the intelligence, coding or agentic +// index, or the design-arena Elo ([weights.IndexFeatures]) — is scored +// from those indexes alone: nothing else on the row raises or lowers it, +// so a model at least as good on every index and no dearer is never +// scored below another. A row that publishes none is scored from its +// context length, release date and licence ([weights.BaseFeatures]), and +// its product family ([familyOf]) moves the mean by the family's fitted +// offset. Price is never read as ability: it is the cost. +// +// 2. ABILITY TO A SEAT. The ability is mapped to an expected solve rate u in +// (0, 1) ([weights.Ability]), and each (class, seat) reads u through its +// own linear link: quality = level + slope·(u − u_ref), on the 0–10 crew +// scale ([seatLink]). The slope is what the seat pays for ability in that +// class of work; its standard deviation is how sure the fit is of it. +// A seat is weighed at the ability's mean less [riskKappa] of its +// standard deviation, so a row the weights know less about is weighed +// below one they know well at the same mean. +// +// 3. THIS INSTALL'S OUTCOMES. What a person kept or redid, per class, seat +// and model, moves the quality by a bounded amount ([Request.Learned]). +// +// A model the weights cannot say enough about — its variance barely below the +// prior's, because its row carries almost nothing ([weights.EvidenceMaxRatio]) +// — sits no seat unless a person pins it. Nor does a model whose ability's +// upper bound falls under the weakest ability the weights credit with doing +// the work ([weights.UFloor]). +// +// The cost of a seat is the seat's token shape — how much it reads fresh, how +// much it reads back from a warm cache, how much it writes on an ordinary task +// — priced at the route's published per-token prices and scaled per class and +// seat ([weights.CostScale]). + +//go:embed prior.json +var priorJSON []byte + +// shape is one seat's tokens on an ordinary task. +type shape struct { + Prompt float64 `json:"prompt"` + Cached float64 `json:"cached"` + Completion float64 `json:"completion"` +} + +// seatLink is one (class, seat)'s link from ability to quality. +type seatLink struct { + Level float64 `json:"level"` + Slope float64 `json:"slope"` + SlopeSD float64 `json:"slope_sd"` +} + +// classLink is one class's links, around the ability u_ref they are centred on. +type classLink struct { + URef float64 `json:"u_ref"` + Seats map[Seat]seatLink `json:"seats"` +} + +// weights is prior.json read. +type weights struct { + Knee float64 `json:"knee_per_usd"` + // Features names the metadata columns after the ability, in the order of + // Loc, Scale, Mean and Cov (whose index 0 is the ability). + Features []string `json:"features"` + // IndexFeatures are the published indexes a row is scored from when it + // carries any; BaseFeatures what a row with none is scored from. + IndexFeatures []string `json:"index_features"` + BaseFeatures []string `json:"base_features"` + Loc []float64 `json:"loc"` + Scale []float64 `json:"scale"` + Mean []float64 `json:"mean"` + Cov [][]float64 `json:"cov"` + // VarScale calibrates the conditional variance. + VarScale float64 `json:"var_scale"` + // FamilyRho is the share of residual variance a family explains, and + // Families each family's offset (in residual standard deviations) and the + // variance left in that offset. + FamilyRho float64 `json:"family_rho"` + Families map[string][2]float64 `json:"families"` + // Ability maps the latent ability to an expected solve rate: + // u = σ(A·θ + B). + Ability struct { + A float64 `json:"a"` + B float64 `json:"b"` + } `json:"ability"` + // UFloor is the least solve rate a model's upper bound must reach to sit + // a seat. + UFloor float64 `json:"u_floor"` + // EvidenceMaxRatio is the most the ability's variance may keep of the + // prior's for a model to be scored at all. + EvidenceMaxRatio float64 `json:"evidence_max_ratio"` + Link map[Class]classLink `json:"link"` + Shapes map[Seat]shape `json:"shapes"` + CostScale map[Class]map[Seat]float64 `json:"cost_scale"` + index map[string]int // feature name → column + cols [len(featureNames)]int // featureNames' columns, zero for one the weights lack + isIndex, isBase [len(featureNames)]bool // featureNames' roles + priorVar float64 // the ability's own variance, standardised +} + +// table is one decision's view of the weights: the weights, the abilities +// already read this decision, this install's learned moves, and the cost floor +// of each seat over the candidates in front of it. +type table struct { + *weights + learned map[string]float64 + floors map[Seat]float64 + // priced is [table.pricedLadder], built from candidates on first use. + priced map[Seat][]pricePoint + candidates []Candidate + cache map[string]ability + // rescue is a table asked for a seat's last rungs ([Request.Rescue]). + rescue bool + // abilityFloor is whether any candidate's mean ability reaches the + // floor ([table.seatCredible]). + abilityFloor bool + // cheapTolerance, above one, is how many times a cheap pick's cost a + // worker may cost and still be preferred for being stronger + // ([Request.Effort] cheap). + cheapTolerance float64 +} + +var ( + loaded *weights + loadOnce sync.Once +) + +// load reads the weights once. Weights that do not parse are a build that +// must not ship, so it panics at first use the way a malformed embedded asset +// does everywhere else in this tree, and a test reads it on every run. +func load() *weights { + loadOnce.Do(func() { + var w weights + if err := json.Unmarshal(priorJSON, &w); err != nil { + panic("crewroute: prior.json: " + err.Error()) + } + n := len(w.Features) + 1 + if len(w.Loc) != n || len(w.Scale) != n || len(w.Mean) != n || len(w.Cov) != n { + panic("crewroute: prior.json: the ability model's dimensions disagree") + } + w.index = map[string]int{} + for i, f := range w.Features { + w.index[f] = i + 1 + } + for i, f := range featureNames { + w.cols[i] = w.index[f] + w.isIndex[i] = holds(w.IndexFeatures, f) + w.isBase[i] = holds(w.BaseFeatures, f) + } + w.priorVar = w.Cov[0][0] + if w.VarScale <= 0 { + w.VarScale = 1 + } + loaded = &w + }) + return loaded +} + +// holds is whether a list names a word. +func holds(words []string, word string) bool { + for _, w := range words { + if w == word { + return true + } + } + return false +} + +// prior is a fresh decision's table: nothing learned, no candidates seen. +func prior() *table { + w := load() + return &table{weights: w, cache: map[string]ability{}} +} + +// forDecision is the table one decision reads: this install's learned moves +// and each seat's cost floor over the candidates. +func forDecision(candidates []Candidate, learned map[string]float64) *table { + t := prior() + t.cache = make(map[string]ability, len(candidates)) + t.learned = learned + t.floors = t.floorsOf(candidates) + return t +} + +// ability is what the weights say about one model: the solve rate u, its +// variance, the solve rate a seat weighs it at (u at the ability's mean less +// [riskKappa] standard deviations), the latent ability's variance, whether +// the row carried enough to be scored at all and whether it was scored from +// published indexes — with the model's lineage and whether it is a quantised +// copy, read once. +type ability struct { + U, VarU float64 + UScore float64 + Theta float64 + VarTheta float64 + Indexed bool + Known bool + lineage string + quant bool +} + +// abilityOf reads one model's ability, once per decision. +func (t *table) abilityOf(m Model) ability { + if a, ok := t.cache[m.ID]; ok { + return a + } + a := remembered(t.weights, m) + if t.cache != nil { + t.cache[m.ID] = a + } + return a +} + +// The abilities of the rows seen lately, keyed by the whole row: a row that +// changes — a new price, an index published — is a new key and is read +// again. The memo is dropped whole when it grows past memoRows. +var ( + memoMu sync.Mutex + memo = map[Model]ability{} +) + +const memoRows = 8192 + +// remembered is [weights.abilityOf] through the memo. +func remembered(w *weights, m Model) ability { + memoMu.Lock() + a, ok := memo[m] + memoMu.Unlock() + if ok { + return a + } + a = w.abilityOf(m) + memoMu.Lock() + if len(memo) >= memoRows { + memo = map[Model]ability{} + } + memo[m] = a + memoMu.Unlock() + return a +} + +// featureNames are the metadata fields the router can read off a model, in +// the order [weights.features] reads them. +var featureNames = [...]string{"lp_in", "lp_out", "lctx", "date", "open", "aa_int", "aa_cod", "aa_ag", "elo"} + +// features is the metadata a model is scored from, in the weights' columns, +// NaN where the row does not carry the field or the field is not read: a row +// that carries any index is read on its indexes only, and a row with none on +// its base fields. The second answer is whether the row took the index path. +func (w *weights) features(m Model) ([]float64, bool) { + x := make([]float64, len(w.Features)) + for i := range x { + x[i] = math.NaN() + } + open := 0.0 + if m.Open { + open = 1 + } + values := [len(featureNames)]float64{ + math.Log10(m.PromptPrice * 1e6), math.Log10(m.CompletionPrice * 1e6), math.Log2(float64(m.Context)), + yearsSince2024(m.Released), open, m.Intelligence, m.Coding, m.Agentic, m.ArenaElo, + } + has := [len(featureNames)]bool{ + m.PromptPrice > 0, m.CompletionPrice > 0, m.Context > 0, !m.Released.IsZero(), true, + m.Intelligence > 0, m.Coding > 0, m.Agentic > 0, m.ArenaElo > 0, + } + indexed := false + for i, col := range w.cols { + if col > 0 && has[i] && w.isIndex[i] { + indexed = true + } + } + for i, col := range w.cols { + read := w.isBase[i] + if indexed { + read = w.isIndex[i] + } + if col > 0 && has[i] && read { + x[col-1] = values[i] + } + } + return x, indexed +} + +// yearsSince2024 is a release date as the weights read it. +func yearsSince2024(at time.Time) float64 { + return at.Sub(time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC)).Hours() / 24 / 365.25 +} + +// riskKappa is how many standard deviations of ability a seat weighs a model +// below its mean. +const riskKappa = 1.0 + +// abilityOf conditions the joint Gaussian on the fields a row is read by, +// adds the family's offset to a row without indexes, and maps the ability to +// a solve rate. +func (w *weights) abilityOf(m Model) ability { + x, indexed := w.features(m) + var obs [maxFeatures]int + var z, so0 [maxFeatures]float64 + var soo [maxFeatures][maxFeatures]float64 + k := 0 + for i, v := range x { + if !math.IsNaN(v) && k < maxFeatures { + col := i + 1 + obs[k] = col + z[k] = (v-w.Loc[col])/w.Scale[col] - w.Mean[col] + k++ + } + } + for i := 0; i < k; i++ { + for j := 0; j < k; j++ { + soo[i][j] = w.Cov[obs[i]][obs[j]] + } + so0[i] = w.Cov[obs[i]][0] + } + mean, variance := w.Mean[0], w.priorVar + // K = S_0o S_oo⁻¹: solve S_oo k = S_o0. + if gain, ok := solveSPD(&soo, &so0, k); ok { + for i := 0; i < k; i++ { + mean += gain[i] * z[i] + variance -= gain[i] * so0[i] + } + } + variance = math.Max(variance, 1e-6) + known := variance/w.priorVar <= w.EvidenceMaxRatio+1e-9 + canon := CanonicalOf(m.ID) + theta := mean*w.Scale[0] + w.Loc[0] + v := variance * w.Scale[0] * w.Scale[0] + if !indexed { + if off, ok := w.Families[familyKey(canon.ID)]; ok { + theta += math.Sqrt(v) * off[0] + v *= 1 - w.FamilyRho + off[1] + } + // A ROW WITH NO PUBLISHED INDEX IS NEVER SCORED ABOVE THE POPULATION'S + // MEAN: what its date, context, licence and family would add stands in + // for indexes it does not publish, so it widens the variance instead. + if pop := w.Mean[0]*w.Scale[0] + w.Loc[0]; theta > pop { + v += (theta - pop) * (theta - pop) + theta = pop + } + } + v *= w.VarScale + a, b := w.Ability.A, w.Ability.B + spread := math.Sqrt(1 + math.Pi*a*a*v/8) + u := sigmoid((a*theta + b) / spread) + d := a * u * (1 - u) + score := sigmoid((a*(theta-riskKappa*math.Sqrt(v)) + b) / spread) + return ability{U: u, VarU: d * d * v, UScore: score, Theta: theta, VarTheta: v, Known: known, Indexed: indexed, + lineage: canon.String(), quant: canon.Variant != ""} +} + +// maxFeatures bounds the metadata columns the ability is read from. +const maxFeatures = 16 + +// solveSPD solves A x = b for the leading n×n block of a small symmetric +// positive-definite A by Cholesky, false when it is not. +func solveSPD(a *[maxFeatures][maxFeatures]float64, b *[maxFeatures]float64, n int) ([maxFeatures]float64, bool) { + var l [maxFeatures][maxFeatures]float64 + var y, x [maxFeatures]float64 + if n == 0 { + return x, false + } + for i := 0; i < n; i++ { + for j := 0; j <= i; j++ { + sum := a[i][j] + for k := 0; k < j; k++ { + sum -= l[i][k] * l[j][k] + } + if i == j { + if sum <= 0 { + return x, false + } + l[i][i] = math.Sqrt(sum) + } else { + l[i][j] = sum / l[j][j] + } + } + } + for i := 0; i < n; i++ { + sum := b[i] + for k := 0; k < i; k++ { + sum -= l[i][k] * y[k] + } + y[i] = sum / l[i][i] + } + for i := n - 1; i >= 0; i-- { + sum := y[i] + for k := i + 1; k < n; k++ { + sum -= l[k][i] * x[k] + } + x[i] = sum / l[i][i] + } + return x, true +} + +func sigmoid(x float64) float64 { return 1 / (1 + math.Exp(-x)) } + +// familyOf is a model's product family: its vendor and its name with the +// version numbers taken out — `z-ai/glm-5.3-flash` is `z-ai/glm-flash`, and a +// token such as `v4`, `k3` or `30b` that is mostly a number is dropped whole. +// A new version of a family starts from what the family's earlier versions +// were fitted to. +func familyOf(id string) string { return familyKey(CanonicalOf(id).ID) } + +// familyKey is [familyOf] on an id already read by [CanonicalOf]. +func familyKey(canon string) string { + vendor, name := "", canon + if at := strings.LastIndex(canon, "/"); at >= 0 { + vendor, name = canon[:at+1], canon[at+1:] + } + var keep []string + for _, tok := range strings.FieldsFunc(name, func(r rune) bool { return r == '-' || r == '.' }) { + stripped := strings.Map(func(r rune) rune { + if r >= '0' && r <= '9' { + return -1 + } + return r + }, tok) + if stripped != tok && len(stripped) <= 2 { + continue + } + if stripped != "" { + keep = append(keep, stripped) + } + } + return vendor + strings.Join(keep, "-") +} + +// linkOf is a class's link, the average of the others for a class the weights +// do not carry. +func (t *table) linkOf(class Class) classLink { + if l, ok := t.Link[class]; ok { + return l + } + return t.Link[Other] +} + +// quality is what one model adds in one seat for one class of work — the +// link at the solve rate the seat weighs the model at, plus what this install +// learned — and its standard deviation. It is a score on the crew scale, not +// clamped to it, so a weaker model is always scored below a stronger one. +func (t *table) quality(class Class, seat Seat, m Model) (q, sd float64) { + return t.qualityOf(class, seat, t.abilityOf(m)) +} + +// qualityOf is [table.quality] on an ability already read. +func (t *table) qualityOf(class Class, seat Seat, a ability) (q, sd float64) { + link := t.linkOf(class) + s := link.Seats[seat] + du := a.U - link.URef + q = s.Level + s.Slope*(a.UScore-link.URef) + sd = math.Sqrt(s.Slope*s.Slope*a.VarU + du*du*s.SlopeSD*s.SlopeSD) + if a.quant { + // A QUANTISED LOCAL COPY is credited with a share of its model's + // quality: the weights read the model's row, and the copy is squeezed. + q *= quantDiscount + } + if len(t.learned) > 0 { + q += t.learned[learnKey(class, seat, a.lineage)] + } + return q, sd +} + +// credible is whether a model may sit a seat: its row says enough to score +// it, and the upper bound of its ability reaches the weakest ability the +// weights credit with doing the work. +func (t *table) credible(m Model) bool { return t.credibleAt(t.abilityOf(m)) } + +// credibleAt is [table.credible] on an ability already read. +func (t *table) credibleAt(a ability) bool { + return a.Known && a.U+2*math.Sqrt(a.VarU) >= t.UFloor +} + +// seatCredible is [table.credibleAt] for one seat. THE WORKER AND THE +// CHECKER ARE NEVER "ANY MODEL WILL DO": one does the work and the other +// decides whether it is accepted, so each one's ability mean — not only its +// upper bound — must reach the floor, whenever any candidate's does. A set of +// allowed models with none that reaches it is still crewed, and [Gaps] says +// the checker is weak. +func (t *table) seatCredible(seat Seat, a ability) bool { + if !t.credibleAt(a) { + return false + } + return seat == Planner || !t.abilityFloor || a.U >= t.UFloor +} + +// pricePoint is one ability on a seat's price ladder and the least a +// credible priced model at least that able costs there. +type pricePoint struct { + U, Cost float64 +} + +// floorsOf is each seat's cost floor over the candidates: the cheapest flat +// cost of a credible model with a published price. It keeps the candidates +// for the price ladder [table.priceFor] builds when an unpriced model asks. +func (t *table) floorsOf(candidates []Candidate) map[Seat]float64 { + floors := map[Seat]float64{} + t.candidates = candidates + for _, c := range candidates { + if c.Model.PromptPrice <= 0 && c.Model.CompletionPrice <= 0 { + continue + } + a := t.abilityOf(c.Model) + if !t.credibleAt(a) { + continue + } + if a.U >= t.UFloor && seatable(Checker, c) { + t.abilityFloor = true + } + for _, seat := range Seats { + if !seatable(seat, c) { + continue + } + cost := t.seatCost(seat, c.Model) + if cost <= 0 { + continue + } + if held, ok := floors[seat]; !ok || cost < held { + floors[seat] = cost + } + } + } + return floors +} + +// pricedLadder is, per seat, the credible priced candidates by ability, best +// first, each with the least cost of any at least that able. +func (t *table) pricedLadder() map[Seat][]pricePoint { + if t.priced != nil { + return t.priced + } + t.priced = map[Seat][]pricePoint{} + for _, c := range t.candidates { + if c.Model.PromptPrice <= 0 && c.Model.CompletionPrice <= 0 { + continue + } + a := t.abilityOf(c.Model) + if !t.credibleAt(a) { + continue + } + for _, seat := range Seats { + if cost := t.seatCost(seat, c.Model); cost > 0 && seatable(seat, c) { + t.priced[seat] = append(t.priced[seat], pricePoint{U: a.U, Cost: cost}) + } + } + } + for seat, points := range t.priced { + sort.Slice(points, func(i, j int) bool { return points[i].U > points[j].U }) + for i := 1; i < len(points); i++ { + points[i].Cost = math.Min(points[i].Cost, points[i-1].Cost) + } + t.priced[seat] = points + } + return t.priced +} + +// costFloor is the least flat cost of a credible priced model in a seat. +func (t *table) costFloor(seat Seat) float64 { return t.floors[seat] } + +// priceFor is what a model that publishes no price is weighed at in a seat: +// A PRICE OF ZERO IS NOT EVIDENCE OF VALUE, so it is the least any credible +// priced model at least as able costs there — or, when none is as able, the +// dearest of them. +func (t *table) priceFor(seat Seat, u float64) float64 { + points := t.pricedLadder()[seat] + if len(points) == 0 { + return 0 + } + // points run best first; the last one at least as able holds the least + // cost among all at least as able. + at := sort.Search(len(points), func(i int) bool { return points[i].U < u }) + if at == 0 { + dearest := 0.0 + for _, p := range points { + dearest = math.Max(dearest, p.Cost) + } + return dearest + } + return points[at-1].Cost +} + +// seatCost is what one model is expected to cost in one seat on an ordinary +// task at the prices it publishes, scaled for the seat. A route that bills +// nothing per token (a subscription, a local model) is priced by the caller, +// not here. Classes share the flat shape; [table.classCost] scales it. +func (t *table) seatCost(seat Seat, m Model) float64 { + s := t.Shapes[seat] + cached := m.CacheReadPrice + if cached <= 0 { + // A provider that publishes no cache-read price is paid the prompt + // price on what it reads back — the dearer reading, never a free one. + cached = m.PromptPrice + } + return s.Prompt*m.PromptPrice + s.Cached*cached + s.Completion*m.CompletionPrice +} + +// classCost is a seat's expected dollars for one task of a class: the flat +// shape's cost times the class and seat's fitted scale. +func (t *table) classCost(class Class, seat Seat, m Model) float64 { + return t.seatCost(seat, m) * t.costScale(class, seat) +} + +// costScale is a class and seat's fitted ratio of dollars to the flat shape, +// one when the weights carry none. +func (t *table) costScale(class Class, seat Seat) float64 { + if byseat, ok := t.CostScale[class]; ok { + if s := byseat[seat]; s > 0 { + return s + } + } + if s := t.CostScale[Other][seat]; s > 0 { + return s + } + return 1 +} + +// LearnKey is the key a learned quality move is kept under: the class, the +// seat and the model's lineage. +func LearnKey(class Class, seat Seat, model string) string { + return learnKey(class, seat, Lineage(model)) +} + +// learnKey is [LearnKey] on a lineage already read. +func learnKey(class Class, seat Seat, lineage string) string { + return string(class) + "\x00" + string(seat) + "\x00" + lineage +} + +// learnedOf is the learned move for one model in one seat, zero for none. +func (t *table) learnedOf(class Class, seat Seat, a ability) float64 { + if len(t.learned) == 0 { + return 0 + } + return t.learned[learnKey(class, seat, a.lineage)] +} + +// Lineage is the id a model's quality is kept under: its canonical identity +// across every provider's spelling ([CanonicalOf]) — lowercase, a provider's +// namespace and a route suffix (`:free`, `:nitro`) taken off, a thinking +// level, a floating alias's `~` and `-latest`, and a dated snapshot suffix +// taken off. `deepseek/deepseek-v4-flash-0731` is the same lineage as +// `deepseek/deepseek-v4-flash`, and so is its free pool: a route is not a +// model. A quantised local copy keeps its variant after `@`, so it is never +// mistaken for the model it was squeezed from. +func Lineage(id string) string { return CanonicalOf(id).String() } + +// lineageTail is [Lineage]'s own rules on an id already read by +// [CanonicalOf]: `-latest` and a dated snapshot suffix taken off. +func lineageTail(id string) string { + id = strings.TrimSuffix(id, "-latest") + if at := strings.LastIndex(id, "-"); at > 0 { + if tail := id[at+1:]; (len(tail) == 4 || len(tail) == 8) && allDigits(tail) { + id = id[:at] + } + } + return id +} + +// allDigits is whether a word is made of ASCII digits only. +func allDigits(word string) bool { + for i := 0; i < len(word); i++ { + if word[i] < '0' || word[i] > '9' { + return false + } + } + return word != "" +} + +// Scorable is whether the weights can score a model at all from what its row +// carries — enough metadata for a finite-variance ability — for a caller that +// offers models and wants to say which the router may pick unpinned. +func Scorable(m Model) bool { return load().abilityOf(m).Known } + +// Knee is the default price of a quality point, in points per dollar — see +// [Route] for why this is the knee of the quality-cost front. +func Knee() float64 { return load().Knee } diff --git a/internal/crewroute/prior.json b/internal/crewroute/prior.json new file mode 100644 index 0000000000..1aa2b82164 --- /dev/null +++ b/internal/crewroute/prior.json @@ -0,0 +1 @@ +{"ability":{"a":1.69289,"b":-2.39415},"about":"crew prior: ability from catalog metadata (joint Gaussian; a row with any published index is conditioned on its indexes only, a row with none on its base fields plus a family offset), and a per-(class, seat) linear link from ability to seat quality","base_features":["lctx","date","open"],"cost_scale":{"bugfix":{"checker":0.797,"planner":1.636,"worker":2.928},"openended":{"checker":2.304,"planner":3.027,"worker":5.829},"other":{"checker":1.55,"planner":2.332,"worker":4.378}},"cov":[[1.030461,0.416575,0.583581,0.501484,0.747589,-0.305773,0.847848,0.897998,0.807642,0.86204],[0.416575,0.993129,1.029271,0.361442,0.139927,-0.40675,0.584541,0.55341,0.524119,0.439355],[0.583581,1.029271,1.506385,0.466973,0.257412,-0.58702,0.783987,0.75326,0.702375,0.614275],[0.501484,0.361442,0.466973,1.081343,0.371667,-0.202468,0.602851,0.607486,0.567244,0.470639],[0.747589,0.139927,0.257412,0.371667,1.073949,-0.211271,0.714962,0.793748,0.689152,0.797933],[-0.305773,-0.40675,-0.58702,-0.202468,-0.211271,1.011656,-0.437372,-0.413516,-0.366752,-0.334855],[0.847848,0.584541,0.783987,0.602851,0.714962,-0.437372,1.692432,1.04365,1.020982,0.939495],[0.897998,0.55341,0.75326,0.607486,0.793748,-0.413516,1.04365,1.600639,0.998652,1.003052],[0.807642,0.524119,0.702375,0.567244,0.689152,-0.366752,1.020982,0.998652,1.577331,0.895046],[0.86204,0.439355,0.614275,0.470639,0.797933,-0.334855,0.939495,1.003052,0.895046,1.650216]],"evidence_max_ratio":0.7,"families":{"01-ai/yi-chat":[-0.8166,0.2498],"01-ai/yi-coder":[-0.6075,0.2498],"01-ai/yi-coder-chat":[-0.4706,0.2498],"01-ai/yi-lightning":[-0.0358,0.2498],"allenai/olmo-instruct":[-0.7382,0.2498],"amazon/nova-pro":[-0.4706,0.2498],"amazon/nova-pro-preview":[0.0235,0.2498],"ant-group/ring":[-0.0835,0.2498],"anthropic/claude":[0.204,0.1681],"anthropic/claude-code":[0.4376,0.2498],"anthropic/claude-fable":[0.8626,0.1681],"anthropic/claude-haiku":[0.1272,0.1016],"anthropic/claude-haiku-latest+claude-sonnet":[-0.0034,0.2498],"anthropic/claude-opus":[0.4676,0.032],"anthropic/claude-opus-+gpt":[-0.1558,0.2498],"anthropic/claude-opus-[high]":[0.4659,0.2498],"anthropic/claude-opus-high":[0.0917,0.2498],"anthropic/claude-sonnet":[0.5528,0.0366],"anthropic/claude-sonnet-+claude-haiku-+gemini-pro-preview-+gemini-flash-preview-mini-+gpt-mini+text-embedding-small":[0.0619,0.2498],"anthropic/claude-sonnet-+claude-opus+gpt-global+gemini-pro":[0.2895,0.2498],"anthropic/claude-sonnet-+claude-opus-+claude-sonnet-+gemini-pro-preview":[0.2744,0.2498],"anthropic/claude-sonnet-+claude-sonnet":[-0.0289,0.2498],"anthropic/claude-sonnet-+claude-sonnet-+gemini-pro+gpt":[0.2252,0.2498],"anthropic/claude-sonnet-+deepseek":[0.0442,0.2498],"anthropic/claude-sonnet-+gemini-pro-preview":[0.0853,0.2498],"anthropic/claude-sonnet-+gpt":[0.3465,0.0849],"anthropic/claude-sonnet-+gpt-preview":[-0.0589,0.2498],"anthropic/claude-sonnet-+o+gemini-pro":[0.2163,0.2498],"anthropic/claude-sonnet-+o+o-mini":[0.2185,0.2498],"anthropic/claude-sonnet-[high]":[0.3953,0.2498],"anthropic/claude-sonnet-high":[-0.1903,0.2498],"anthropic/claude-sonnet-mini":[0.2165,0.1267],"anthropic/claude-sonnet-mini+o-mini":[0.1473,0.1681],"anthropic/fable-[high]":[0.4776,0.2498],"anysphere/cursor":[0.3564,0.2498],"arcee-ai/trinity-large":[-0.3179,0.2498],"bytedance-seed/doubao":[-0.0753,0.2498],"bytedance-seed/doubao-pro":[-0.2219,0.1681],"bytedance-seed/doubao-seed-code+doubao-seed":[0.3212,0.2498],"bytedance-seed/seed-pro-preview":[0.2453,0.2498],"cognition/swe":[0.3524,0.1267],"cohere/cai-command-a":[-0.1953,0.2498],"cohere/command-a":[-0.2322,0.2498],"cohere/command-a-plus":[-0.299,0.2498],"cohere/command-a-quality":[-0.3743,0.2498],"cohere/command-r":[-0.6885,0.1267],"cohere/command-r+":[-0.7138,0.2498],"cohere/command-r-plus":[-0.561,0.1267],"cursor/composer":[0.2738,0.2498],"deep-cogito/cogito":[0.1509,0.2498],"deepseek/deepseek":[0.4873,0.0849],"deepseek/deepseek-+-claude-sonnet":[0.0638,0.2498],"deepseek/deepseek-chat":[0.221,0.0849],"deepseek/deepseek-coder":[0.1863,0.1681],"deepseek/deepseek-coder-instruct":[-1.0365,0.1267],"deepseek/deepseek-coder-lite-instruct":[-0.3438,0.2498],"deepseek/deepseek-distill-llama":[0.2045,0.2498],"deepseek/deepseek-distill-qwen":[-0.2833,0.2498],"deepseek/deepseek-exp":[0.2972,0.2498],"deepseek/deepseek-flash":[0.4046,0.1681],"deepseek/deepseek-flash-[high]":[0.1776,0.2498],"deepseek/deepseek-flash-high-preview":[0.1582,0.2498],"deepseek/deepseek-lite-chat":[-1.0617,0.2498],"deepseek/deepseek-lite-preview":[-0.0789,0.2498],"deepseek/deepseek-local":[0.0257,0.1681],"deepseek/deepseek-preview":[0.1261,0.2498],"deepseek/deepseek-pro":[0.093,0.2498],"deepseek/deepseek-pro-[high]":[0.2165,0.2498],"deepseek/deepseek-pro-high-preview":[0.2605,0.2498],"deepseek/deepseek-speciale":[0.0984,0.2498],"deepseek/deepseek-terminus":[0.356,0.2498],"google/gemini-exp":[0.3081,0.2498],"google/gemini-flash":[-0.1397,0.0426],"google/gemini-flash-exp":[-0.479,0.1267],"google/gemini-flash-lite":[-0.4325,0.1267],"google/gemini-flash-lite-preview":[-0.1802,0.1016],"google/gemini-flash-preview":[0.222,0.0728],"google/gemini-flash-thinking-exp":[0.0202,0.2498],"google/gemini-pro":[0.2119,0.0728],"google/gemini-pro-exp":[0.2358,0.1267],"google/gemini-pro-preview":[0.509,0.1016],"google/gemini-pro-preview-customtools":[0.134,0.2498],"google/gemma":[0.2137,0.1681],"google/gemma-it":[-0.4465,0.0638],"https:/deepswe-preview":[0.0238,0.2498],"ibm-granite/granite":[-0.8232,0.1681],"inception/mercury":[-0.5088,0.2498],"jetbrains/junie":[0.4553,0.2498],"kat-coder-pro":[-0.1509,0.2498],"meta-llama/llama":[-0.5451,0.2498],"meta-llama/llama-+qwen":[-0.0684,0.2498],"meta-llama/llama-chat-hf":[-1.3179,0.2498],"meta-llama/llama-instruct":[0.1213,0.0728],"meta-llama/llama-maverick":[-0.2921,0.2498],"meta-llama/llama-maverick-instruct":[-0.0685,0.1681],"meta-llama/llama-nemotron-instruct":[0.0048,0.2498],"meta-llama/llama-nemotron-instruct-hf":[-0.4491,0.2498],"meta-llama/llama-nemotron-nano":[-0.0623,0.2498],"meta-llama/llama-nemotron-ultra":[0.5961,0.2498],"meta-llama/llama-scout-instruct":[-0.5931,0.1681],"meta-llama/llama-swe-rl":[-0.0627,0.2498],"meta-llama/muse-glimmer":[-0.0425,0.2498],"meta-llama/muse-spark":[0.3923,0.1016],"microsoft/phi":[-0.0766,0.2498],"microsoft/phi-medium-instruct":[-0.239,0.1681],"microsoft/phi-mini-instruct":[-1.0062,0.1016],"microsoft/phi-moe-instruct":[-0.357,0.2498],"microsoft/phi-small-instruct":[-0.4657,0.1681],"microsoft/wizardlm":[0.0645,0.2498],"minimax/minimax":[0.4869,0.0728],"minimax/minimax-preview":[0.4037,0.2498],"mistralai/codestral":[-0.7278,0.1681],"mistralai/codestral-k-m":[-0.5618,0.2498],"mistralai/devstral":[-0.0415,0.1681],"mistralai/devstral-instruct":[0.222,0.2498],"mistralai/devstral-small":[-0.1252,0.2498],"mistralai/devstral-small-instruct":[0.1526,0.2498],"mistralai/magistral-medium":[0.2501,0.2498],"mistralai/magistral-small":[0.1558,0.2498],"mistralai/mistral-instruct":[-0.4577,0.1681],"mistralai/mistral-large":[-0.1068,0.1267],"mistralai/mistral-medium":[0.6154,0.1681],"mistralai/mistral-nemo":[-0.4306,0.2498],"mistralai/mistral-small":[-0.9968,0.2498],"mistralai/mixtral-instruct":[0.0242,0.1681],"moonshotai/kimi":[0.7317,0.1016],"moonshotai/kimi-code":[0.3721,0.2498],"moonshotai/kimi-instant":[0.404,0.2498],"moonshotai/kimi-instruct":[0.1935,0.2498],"moonshotai/kimi-ioi":[0.2203,0.2498],"moonshotai/kimi-preview":[0.6195,0.1681],"moonshotai/kimi-thinking":[0.2664,0.2498],"moonshotai/kimi-thinking-turbo":[0.3329,0.2498],"nousresearch/hermes-llama":[0.244,0.1681],"nvidia/nemotron-nano":[-0.6892,0.2498],"nvidia/nemotron-super":[-0.1573,0.2498],"nvidia/nemotron-ultra":[-0.0308,0.2498],"ollama/codegeex":[-0.8137,0.2498],"ollama/codestral":[-0.5953,0.2498],"ollama/granite-dense":[-1.0518,0.2498],"ollama/hermes":[-0.9229,0.2498],"ollama/llama":[-0.9238,0.2498],"ollama/mistral-small":[-0.7138,0.2498],"ollama/opencodeinterpreter":[-0.8526,0.2498],"ollama/qwen":[-0.436,0.2498],"ollama/qwen-coder":[-0.2591,0.2498],"ollama/qwen-coder-instruct-gguf":[-0.3163,0.2498],"ollama/tulu":[-0.9238,0.2498],"ollama/yi-coder":[-0.6306,0.2498],"openai/":[0.5653,0.1681],"openai/chatgpt":[0.3604,0.1267],"openai/codex":[0.4206,0.2498],"openai/davinci":[-0.7954,0.2498],"openai/gpt":[0.4259,0.0394],"openai/gpt-+claude-sonnet":[0.2794,0.2498],"openai/gpt-astra":[0.7491,0.2498],"openai/gpt-chat":[0.053,0.2498],"openai/gpt-codex":[0.367,0.0849],"openai/gpt-codex-mini":[-0.2002,0.2498],"openai/gpt-instant":[0.1348,0.2498],"openai/gpt-luna":[0.1056,0.2498],"openai/gpt-luna-[medium]":[0.2309,0.2498],"openai/gpt-mini":[-0.1864,0.0849],"openai/gpt-nano":[-0.549,0.1267],"openai/gpt-oss":[-0.1304,0.1681],"openai/gpt-oss-rl":[-0.1966,0.2498],"openai/gpt-pre-release":[0.4335,0.2498],"openai/gpt-preview":[0.6458,0.1267],"openai/gpt-pro":[0.154,0.1681],"openai/gpt-sol":[0.4233,0.2498],"openai/gpt-sol-[medium]":[0.4857,0.2498],"openai/gpt-terra":[0.147,0.2498],"openai/gpt-turbo":[0.4115,0.1681],"openai/gpt-turbo-instruct":[0.8094,0.2498],"openai/high-+-gpt":[0.1836,0.2498],"openai/low-prompt-old":[0.0748,0.2498],"openai/mini":[0.4278,0.1267],"openai/preview":[0.5113,0.2498],"openai/pro":[0.4635,0.2498],"openrouter/horizon-alpha":[-0.1406,0.2498],"openrouter/horizon-beta":[-0.1272,0.2498],"perplexity/sonar":[-0.1422,0.2498],"perplexity/sonar-pro":[0.2022,0.2498],"poolside/laguna-m":[0.0891,0.2498],"poolside/laguna-xs":[-0.0356,0.2498],"qwen/qwen":[0.0019,0.027],"qwen/qwen-chat":[-1.361,0.0728],"qwen/qwen-coder":[-0.1599,0.1681],"qwen/qwen-coder-instruct":[-0.3525,0.0568],"qwen/qwen-coder-next":[0.0791,0.2498],"qwen/qwen-diff,-no-think,-alibaba-api":[0.032,0.2498],"qwen/qwen-flash":[-0.323,0.2498],"qwen/qwen-flash-next":[0.6628,0.2498],"qwen/qwen-ins":[-0.469,0.2498],"qwen/qwen-instruct":[-0.5857,0.0465],"qwen/qwen-instruct-turbo":[-0.6228,0.1681],"qwen/qwen-math-instruct":[-0.5571,0.2498],"qwen/qwen-max":[0.4669,0.1016],"qwen/qwen-max-preview":[0.2416,0.2498],"qwen/qwen-next-instruct":[-0.0461,0.2498],"qwen/qwen-plus":[-0.0475,0.1016],"qwen/qwen-plus-preview":[0.2073,0.2498],"qwen/qwen-thinking":[0.0994,0.1681],"qwen/qwq":[0.1247,0.2498],"qwen/qwq-+-qwen-coder-instruct":[-0.2111,0.2498],"qwen/qwq-max-preview":[0.1621,0.2498],"qwen/qwq-preview":[-0.0759,0.2498],"qwen/qwq-temp":[0.0578,0.2498],"stepfun/step-flash":[0.2902,0.1681],"stepfun/step-preview":[0.432,0.2498],"tencent/":[0.37,0.2498],"tencent/hunyuan-preview":[0.1183,0.2498],"tencent/preview":[0.5534,0.2498],"thinkingmachines/inkling":[-0.0579,0.2498],"thinkingmachines/inkling-small":[-0.0718,0.2498],"trinity-large":[-0.0982,0.2498],"ucsb-surfi/termigen":[-0.1533,0.2498],"unknown/acm-rewrite-qwen-chat":[-0.4565,0.2498],"unknown/agent":[0.3785,0.2498],"unknown/agentless-deepseek-qwen-vl":[-0.1778,0.2498],"unknown/agentless-qwen-coderb-qwen-vl":[-0.1423,0.1267],"unknown/amazon-nova-lite":[-0.5735,0.2498],"unknown/amazon-nova-micro":[-0.7354,0.2498],"unknown/amazon-nova-premier":[-0.0513,0.2498],"unknown/amazon-nova-pro":[-0.35,0.2498],"unknown/azerogpt":[-0.7646,0.1681],"unknown/codearts-glm":[0.6218,0.2498],"unknown/codearts-minimax":[0.408,0.2498],"unknown/codegen-qwen-instruct-chk":[-0.2455,0.2498],"unknown/codeqwen":[-0.7745,0.2498],"unknown/coding-amcfull-apifull-mmluk-meta-llama-instruct-chk":[-0.3682,0.2498],"unknown/coding-meta-llama-instruct-chk":[-0.3619,0.2498],"unknown/deepcoder-preview":[0.0554,0.2498],"unknown/dracarys-instruct":[0.1982,0.2498],"unknown/dracarys-llama-instruct":[0.1871,0.2498],"unknown/exaone":[0.1779,0.2498],"unknown/frogboss":[0.0524,0.2498],"unknown/frogmini":[-0.027,0.2498],"unknown/gemma":[-0.7496,0.2498],"unknown/human":[0.1684,0.2498],"unknown/hunyuan-turbos":[-0.2095,0.2498],"unknown/iswe-openmodels":[0.342,0.2498],"unknown/lcb-math-qwen-instructv-merged":[-0.4395,0.2498],"unknown/learnlm-pro-experimental":[-0.3162,0.2498],"unknown/mathstral":[-0.8491,0.2498],"unknown/mcts-refine":[-0.2899,0.2498],"unknown/metastone":[-0.1841,0.2498],"unknown/multiple":[0.5624,0.2498],"unknown/nan":[-0.0018,0.2498],"unknown/open-mistral-nemo":[-0.7216,0.2498],"unknown/open-mixtral":[-1.0543,0.1681],"unknown/openhands-lm":[0.3543,0.2498],"unknown/openhermes-mistral":[-0.9015,0.2498],"unknown/openreasoning-nemotron":[0.1794,0.2498],"unknown/optimus-alpha":[-0.0145,0.2498],"unknown/perplexity-sonar":[-1.7795,0.2498],"unknown/quasar-alpha":[-0.0022,0.2498],"unknown/reflection":[-0.6538,0.2498],"unknown/reflection-llama":[-0.6215,0.2498],"unknown/sdk":[0.3981,0.2498],"unknown/sky-preview":[-0.4314,0.2498],"unknown/smaug-qwen-instruct":[-0.5569,0.2498],"unknown/starling-lm-beta":[-0.9126,0.2498],"unknown/step":[-0.2236,0.2498],"unknown/swe-llama":[-0.9052,0.2498],"unknown/trinity-large":[0.0201,0.2498],"unknown/unknown":[-0.0322,0.2498],"unknown/vicuna":[-1.5824,0.1681],"unknown/wbot":[-0.2281,0.2498],"unknown/xbai":[0.1638,0.2498],"unknown/zephyr-alpha":[-1.0089,0.2498],"unknown/zephyr-beta":[-1.1369,0.2498],"upstage/solar-pro":[-0.8581,0.1267],"x-ai/grok":[0.1404,0.0465],"x-ai/grok-[high]":[0.4832,0.2498],"x-ai/grok-beta":[-0.0009,0.1267],"x-ai/grok-build":[0.0689,0.2498],"x-ai/grok-code-fast":[-0.219,0.2498],"x-ai/grok-fast":[-0.3376,0.1681],"x-ai/grok-mini":[0.1328,0.1681],"x-ai/grok-mini-beta":[0.0647,0.2498],"x-ai/grok-mini-reasoning-beta":[-0.2483,0.2498],"xiaomi/mimo":[0.1791,0.2498],"xiaomi/mimo-flash":[0.225,0.2498],"xiaomi/mimo-pro":[0.3028,0.1681],"z-ai/glm":[0.8345,0.0638],"z-ai/glm-[high]":[0.4265,0.2498],"z-ai/glm-air":[0.256,0.2498],"z-ai/glm-flash":[0.3295,0.1681],"z-ai/glm-turbo":[0.0735,0.2498]},"family_rho":0.486,"features":["lp_in","lp_out","lctx","date","open","aa_int","aa_cod","aa_ag","elo"],"index_features":["aa_int","aa_cod","aa_ag","elo"],"knee_per_usd":5.0,"link":{"bugfix":{"seats":{"checker":{"level":1.634,"slope":2.118,"slope_sd":1.264},"planner":{"level":0.98,"slope":2.118,"slope_sd":1.264},"worker":{"level":3.921,"slope":2.2,"slope_sd":1.272}},"u_ref":0.6095},"openended":{"seats":{"checker":{"level":1.345,"slope":8.556,"slope_sd":2.749},"planner":{"level":0.807,"slope":8.556,"slope_sd":2.749},"worker":{"level":3.227,"slope":8.565,"slope_sd":2.77}},"u_ref":0.6018},"other":{"seats":{"checker":{"level":1.49,"slope":5.337,"slope_sd":2.006},"planner":{"level":0.893,"slope":5.337,"slope_sd":2.006},"worker":{"level":3.574,"slope":5.383,"slope_sd":2.021}},"u_ref":0.6057}},"loc":[0.0,-0.221944,0.295021,17.303728,1.842834,0.377389,26.412941,44.133613,19.901136,1193.91129],"mean":[0.203091,-0.080757,-0.062968,-0.079207,-0.125447,0.055773,-0.568404,-0.348314,-0.440756,-0.369667],"noise":1.7217,"scale":[1.0,0.957046,0.726154,2.545528,0.768919,0.484733,14.103448,22.709659,18.571976,124.41988],"shapes":{"checker":{"cached":80000,"completion":2200,"prompt":12000},"planner":{"cached":10000,"completion":1500,"prompt":4000},"worker":{"cached":250000,"completion":4000,"prompt":15000}},"u_floor":0.3695,"var_scale":0.51,"version":2} \ No newline at end of file diff --git a/internal/crewroute/providers.go b/internal/crewroute/providers.go new file mode 100644 index 0000000000..076e386b08 --- /dev/null +++ b/internal/crewroute/providers.go @@ -0,0 +1,66 @@ +package crewroute + +import "strings" + +// WHICH PROVIDERS THE CREW MAY ROUTE THROUGH — A SET BESIDE THE RULE, NOT IN IT. +// +// The allowed rule ([Allowed]) says which MODELS a seat may be picked from. The +// providers a person turned off say which ROUTES may carry them, and the two +// are kept apart on purpose: a `-x` in the rule naming a provider would also +// read as a vendor (`-openai` takes OpenAI's models away, not just a route), +// and stepping the rule onto a new base drops every exception it had, so a +// provider switched off there would switch itself back on the next time the +// models row was walked. +// +// THE SET IS OF PROVIDERS TURNED OFF, never of providers turned on, so a +// provider connected tomorrow is on the day it is connected — the same thing +// connecting one has always meant — and nobody has to remember a second place +// to switch it on. +// +// The filter is the last step before a model becomes a candidate: a model's +// routes through providers that are off are taken away, and a model left with +// no route is no candidate, however cheap — exactly the verdict a model no +// connected provider reaches already gets. + +// ProvidersOff is the set of provider ids a person turned off, lower case. +// The zero value is every provider on. +type ProvidersOff map[string]bool + +// On says whether a provider's routes may be used. +func (off ProvidersOff) On(provider string) bool { + return !off[strings.ToLower(strings.TrimSpace(provider))] +} + +// Routes is the routes a provider that is on carries, in their own order. +func (off ProvidersOff) Routes(routes []Route) []Route { + if len(off) == 0 { + return routes + } + var out []Route + for _, r := range routes { + if off.On(r.Provider) { + out = append(out, r) + } + } + return out +} + +// Candidates is the candidates with the routes of every provider that is off +// taken away, and every candidate left with no route dropped. The route order +// each candidate came with — plans and local first, the default service last — +// is kept, because it is the order a tie between routes of equal cost is +// broken in, and a filter is no reason to break it differently. +func (off ProvidersOff) Candidates(candidates []Candidate) []Candidate { + if len(off) == 0 { + return candidates + } + var out []Candidate + for _, c := range candidates { + routes := off.Routes(c.Routes) + if len(routes) == 0 { + continue + } + out = append(out, Candidate{Model: c.Model, Routes: routes}) + } + return out +} diff --git a/internal/crewroute/providers_test.go b/internal/crewroute/providers_test.go new file mode 100644 index 0000000000..e23b32df63 --- /dev/null +++ b/internal/crewroute/providers_test.go @@ -0,0 +1,31 @@ +package crewroute + +import "testing" + +// A PROVIDER TURNED OFF TAKES ITS ROUTES AWAY AND NOTHING ELSE: a model it +// shares with a provider still on keeps that route, in its own order, and a +// model only it reached is no candidate. +func TestProvidersOffFiltersRoutesNotModels(t *testing.T) { + shared := Candidate{Model: Model{ID: "z-ai/glm-5.3-flash"}, Routes: []Route{ + {Provider: "z-ai", Kind: Plan}, {Provider: "ollama", Kind: Local}, {Provider: "openrouter", Kind: Metered}, + }} + only := Candidate{Model: Model{ID: "openai/gpt-6"}, Routes: []Route{{Provider: "codex", Kind: Plan}}} + all := []Candidate{shared, only} + + if got := ProvidersOff(nil).Candidates(all); len(got) != 2 || len(got[0].Routes) != 3 { + t.Fatalf("no provider off changed the candidates: %+v", got) + } + got := ProvidersOff{"codex": true, "ollama": true}.Candidates(all) + if len(got) != 1 || got[0].Model.ID != shared.Model.ID { + t.Fatalf("a model only an off provider reaches stayed a candidate: %+v", got) + } + if routes := got[0].Routes; len(routes) != 2 || routes[0].Provider != "z-ai" || routes[1].Provider != "openrouter" { + t.Fatalf("the routes left are %+v, want z-ai then openrouter", routes) + } + if len(all[0].Routes) != 3 { + t.Fatal("the filter changed the candidates it was handed") + } + if !(ProvidersOff{"codex": true}).On("OpenRouter") || (ProvidersOff{"codex": true}).On(" Codex ") { + t.Fatal("On does not read a provider id case folded") + } +} diff --git a/internal/crewroute/reach_test.go b/internal/crewroute/reach_test.go new file mode 100644 index 0000000000..7bee656f98 --- /dev/null +++ b/internal/crewroute/reach_test.go @@ -0,0 +1,151 @@ +package crewroute + +import ( + "os" + "testing" +) + +// THE KORNIA CASE: an issue labelled `bug :bug:` — the tracker's own spelling — +// whose title names the misbehaving call and says what it never does was read +// as open-ended, because no rule knew the label's spelling and no title rule +// fired. Either alone now reads it as a bugfix. +func TestAKorniaStyleDefectIsABugfix(t *testing.T) { + title := "augmentation: RandomResizedCrop(scale=(1, 1)) never keeps a square or portrait image: strict candidate test and an inverted-ratio fallback" + report, err := os.ReadFile("testdata/kornia-4814.md") + if err != nil { + t.Fatal(err) + } + labels := []string{"bug :bug:", "help wanted", "good first issue"} + for name, task := range map[string]Task{ + "the title alone": {Text: title}, + "the labels on a bare text": {Text: "the crop looks wrong", Labels: labels}, + "the issue as filed": {Text: string(report), Labels: labels}, + } { + if got := Classify(task); got.Class != Bugfix || !got.Sure { + t.Errorf("%s: %s (sure %v, %q), want a sure bugfix", name, got.Class, got.Sure, got.Why) + } + } + for _, text := range []string{"The export does not close the file", "parse_date fails to read ISO weeks", "Settings should reject a negative timeout"} { + if got := Classify(Task{Text: text}); got.Class != Bugfix { + t.Errorf("%q: %s (%q), want bugfix", text, got.Class, got.Why) + } + } + if got := Classify(Task{Text: "The exporter should support YAML"}); got.Class == Bugfix { + t.Errorf("a request for something new read as a bugfix (%q)", got.Why) + } +} + +// A FIX WITH REACH IS COMPLEX, A ONE-LINE FIX IS SIMPLE, and only the worker +// moves: one rung, at the same λ, never over a pin. +func TestAComplexFixGetsAStrongerWorker(t *testing.T) { + report, err := os.ReadFile("testdata/hermes-7680.md") + if err != nil { + t.Fatal(err) + } + complexTask := Task{Text: string(report)} + simpleTask := Task{Text: "fix: typo in the loop bound of paginate() skips the last page"} + if got := Classify(complexTask); got.Class != Bugfix || got.Complex == "" { + t.Fatalf("the hermes report: %s, reach %q, want a complex bugfix", got.Class, got.Complex) + } + if got := Classify(simpleTask); got.Class != Bugfix || got.Complex != "" { + t.Fatalf("a one-line typo: %s, reach %q, want a simple bugfix", got.Class, got.Complex) + } + cands := append(catalogCandidates(), candidateOf(glm53)) + simple, err := Decide(Request{Task: simpleTask, Candidates: cands}) + if err != nil { + t.Fatal(err) + } + hard, err := Decide(Request{Task: complexTask, Candidates: cands}) + if err != nil { + t.Fatal(err) + } + if simple.Subclass != "simple" || hard.Subclass != "complex" || hard.Class != Bugfix { + t.Errorf("subclasses %q and %q (class %s)", simple.Subclass, hard.Subclass, hard.Class) + } + if hard.Lambda != simple.Lambda { + t.Errorf("λ moved: %v against %v", hard.Lambda, simple.Lambda) + } + sw, hw := simple.Seat(Worker), hard.Seat(Worker) + if hw.Quality <= sw.Quality || Lineage(hw.Model) == Lineage(sw.Model) { + t.Errorf("complex worker %s (%.2f), simple worker %s (%.2f): want a stronger one", hw.Model, hw.Quality, sw.Model, sw.Quality) + } + for _, seat := range []Seat{Planner, Checker} { + if hard.Seat(seat).Model != simple.Seat(seat).Model { + t.Errorf("%s moved: %s against %s", seat, hard.Seat(seat).Model, simple.Seat(seat).Model) + } + } + pinned, err := Decide(Request{Task: complexTask, Candidates: cands, Pins: map[Seat]Pin{Worker: {Model: sw.Model}}}) + if err != nil { + t.Fatal(err) + } + if got := pinned.Seat(Worker).Model; Lineage(got) != Lineage(sw.Model) { + t.Errorf("a pinned worker became %s", got) + } +} + +// A CLASS GIVEN IS NOT A REACH FORGOTTEN: a caller that classified first and +// hands on only the class — or the whole reading — still gets the complex +// fix's worker, read off the same words. +func TestAGivenBugfixClassStillReadsReach(t *testing.T) { + report, err := os.ReadFile("testdata/hermes-7680.md") + if err != nil { + t.Fatal(err) + } + task := Task{Text: string(report)} + cands := catalogCandidates() + plain, err := Decide(Request{Class: Bugfix, Task: Task{Text: "fix: typo in the loop bound of paginate() skips the last page"}, Candidates: cands}) + if err != nil { + t.Fatal(err) + } + reading := Classify(task) + for name, r := range map[string]Request{ + "the class alone": {Class: Bugfix, Task: task, Candidates: cands}, + "the whole reading": {Class: Bugfix, Reading: &reading, Candidates: cands}, + } { + d, err := Decide(r) + if err != nil { + t.Fatal(err) + } + if d.Subclass != "complex" || d.Reach == "" { + t.Errorf("%s: subclass %q reach %q, want complex", name, d.Subclass, d.Reach) + } + if d.Seat(Worker).Quality <= plain.Seat(Worker).Quality { + t.Errorf("%s: worker %s, no stronger than the simple fix's %s", name, d.Seat(Worker).Model, plain.Seat(Worker).Model) + } + } +} + +// A ONE-PARAGRAPH ASK IS READ FOR REACH TOO. Most asks have no body under a +// title line, and the signals that make a fix complex sit in that one +// paragraph. +func TestAOneParagraphFixIsReadForReach(t *testing.T) { + cases := []struct{ text, why string }{ + {"The /api/v2/orders endpoint returns HTTP 500 instead of 404 when the order id does not exist. The handler in api/orders.py and the repository in db/orders_repo.py both need changes, and the existing tests in tests/test_orders_api.py must keep passing. Repro: GET /api/v2/orders/999999 on a fresh database.", "two files and an endpoint"}, + {"Security: the file upload endpoint accepts path traversal in the filename (../../etc/passwd). Sanitize filenames in upload/handlers.py and add a regression test.", "a security fix on an endpoint"}, + } + for _, tc := range cases { + r := Classify(Task{Text: tc.text}) + if r.Class != Bugfix || r.Complex == "" { + t.Errorf("%s: read as %s, reach %q; want a complex fix", tc.why, r.Class, r.Complex) + } + } + if r := Classify(Task{Text: "Fix typo in README: \"recieve\" should be \"receive\" in the install section."}); r.Complex != "" { + t.Errorf("a typo read as reaching: %q", r.Complex) + } +} + +// A DEFECT SAID WITHOUT THE WORD BUG is still a fix, and a mechanical change +// — a rename — is a small fix, not open-ended work. +func TestDefectWordsAndMechanicalChangesReadAsFixes(t *testing.T) { + for _, text := range []string{ + "Session titles are wrongly rejected for Japanese chats when the title contains English product names.", + "The export is incorrectly rounded when the total has three decimals.", + "Search returns the archived items instead of the live ones.", + "Rename the function get_usr to get_user across the codebase.", + "Bump the version to 2.4.1 in every package manifest.", + } { + if r := Classify(Task{Text: text}); r.Class != Bugfix || r.Complex != "" { + t.Errorf("%q read as %s (%s), reach %q; want a simple fix", text, r.Class, r.Why, r.Complex) + } + } +} diff --git a/internal/crewroute/route.go b/internal/crewroute/route.go new file mode 100644 index 0000000000..0f92afb0a1 --- /dev/null +++ b/internal/crewroute/route.go @@ -0,0 +1,1434 @@ +// Package crewroute picks the crew for one task: which model sits the worker, +// the planner and the checker seat, on which provider's route, for this piece +// of work and nothing longer. +// +// ── THE ONE SENTENCE IT IMPLEMENTS ── +// +// The /crew panel says what is allowed (it persists); the words in the ask say +// how hard to try this one task. Nothing else sticks. So there is no preset +// here and no mode: a caller hands over the task, the models the person allows +// on the providers they have connected, the seats they pinned, and at most one +// word of effort, and gets back one crew for this task. +// +// ── WHY IT ROUTES ON THE CLASS OF WORK ── +// +// The design is in docs/design/model-pool/pareto-crewing.pdf. The rule it +// follows: narrow fixes use the cheapest qualified crew, and open-ended work +// gets a stronger checker, the one seat worth paying for there. The policy is +// read off a table ([prior]) rather than written into branches: classify the +// task ([Classify]), then pick each seat to maximise quality minus λ times +// cost. +// +// ── WHAT IT DOES NOT DO ── +// +// It does not escalate on its own. A done-verdict that misses real solves and +// passes failures would make an escalation loop spend on the wrong tasks, so +// [AutoEscalate] is off, and a stronger crew is something a person asks for +// (`redo stronger`). +// +// It is PURE: no disk, no network, no clock. The same request gives the same +// crew, in the same order, every time — which is what lets a decision be +// logged, replayed and argued with. A decision costs well under a millisecond. +package crewroute + +import ( + "errors" + "fmt" + "math" + "sort" + "strconv" + "strings" + "time" +) + +// AutoEscalate is whether a crew is ever made stronger without somebody +// asking. It is OFF, and a constant rather than a setting, because whether an +// automatic escalation pays for itself depends on how far the checker's +// verdict can be trusted, which is a property of the checker, not a +// preference. +const AutoEscalate = false + +// Seat is one of the crew's three seats, by the name a person reads. +type Seat string + +const ( + // Worker does the work: the leaves of a task, every tool turn of it. + Worker Seat = "worker" + // Planner cuts the work into pieces and steers the run. + Planner Seat = "planner" + // Checker reads finished work against what it was supposed to do. + Checker Seat = "checker" +) + +// Seats lists the three in the order a crew is read out. +var Seats = []Seat{Worker, Planner, Checker} + +// Model is a catalog row as the router reads it. Prices are dollars per +// token, as the catalog publishes them. +type Model struct { + ID string + Open bool + PromptPrice float64 + CompletionPrice float64 + CacheReadPrice float64 + Intelligence float64 + Coding float64 + Agentic float64 + // ArenaElo is the best design-arena Elo the row publishes, zero for none. + ArenaElo float64 + Context int + // Released is when the model was released, the zero time when the row + // does not say. + Released time.Time + // Tools is whether the model takes tool calls. A crew seat is an agent + // loop, so a model that cannot call a tool cannot sit one. + Tools bool +} + +// RouteKind is how a route bills. +type RouteKind string + +const ( + // Metered is billed per token at the model's published prices. + Metered RouteKind = "metered" + // Plan is a subscription login — a coding plan, a ChatGPT account — whose + // marginal cost for one more task is taken as zero. + Plan RouteKind = "plan" + // Local is a model on this machine, which costs nothing per token. + Local RouteKind = "local" + // Free is a provider's free pool for a model (OpenRouter's `:free`): no + // price, but rate-limited hard, dropped without notice, and allowed to log + // or train on what it is sent. It is weighed at what it is EXPECTED to cost + // ([routeCost]), used only when the person turned free routes on, and a + // seat on it falls through to the same model's paid route first. + Free RouteKind = "free" +) + +// freeFailPrior is the chance a free route refuses a task's first call, +// before this install has any outcome of its own for that route: free pools +// are rate-limited by design, so the prior is pessimistic. +const freeFailPrior = 0.3 + +// freeLimitPenalty is what a free route is charged for its limits even when it +// answers — the waits and retries of a rate-limited pool — in dollars a task. +const freeLimitPenalty = 0.001 + +// freeLimitShare is the same charge as a share of the paid route the seat +// falls to, whichever is more: waits and retries cost more on a dearer seat. +const freeLimitShare = 0.1 + +// Route is one way to reach a model: the provider, the id to send so the call +// goes that way, and how it bills. +type Route struct { + Provider string + Send string + Kind RouteKind + // FailRate is this install's learned chance that the route refuses a + // task's first call, zero when nothing is known — a free route then reads + // [freeFailPrior]. + FailRate float64 +} + +// Candidate is a model the person allows, with every route a connected +// provider offers it on, the caller's preferred route first. +type Candidate struct { + Model Model + Routes []Route +} + +// Pin is a seat the person fixed. Model is the id they wrote, Provider the +// route they pinned when they wrote `model@provider`, and Send and Kind the +// route the caller resolved for it. +type Pin struct { + Model string + Provider string + Send string + Kind RouteKind +} + +// Effort is the one word about how hard to try this task. +type Effort string + +const ( + // EffortKnee is the default: the knee of the quality-cost front. + EffortKnee Effort = "" + // EffortBest buys the most quality the weights believe in, whatever it + // costs (λ → 0, ties to the cheaper). + EffortBest Effort = "best" + // EffortCheap buys quality only where it is nearly free. + EffortCheap Effort = "cheap" +) + +// cheapTolerance is how many times the cheapest fitting worker's cost a +// cheap crew pays for a stronger worker. +const cheapTolerance = 1.5 + +// ParseEffort reads a person's word for effort. Empty is the knee; anything +// else that is not one of the two words is refused by the caller's own form. +func ParseEffort(word string) (Effort, bool) { + switch strings.ToLower(strings.TrimSpace(word)) { + case "": + return EffortKnee, true + case "best", "thorough", "strong", "strongest": + return EffortBest, true + case "cheap", "cheapest": + return EffortCheap, true + } + return EffortKnee, false +} + +// The price of quality, spelled once. +// +// THE KNEE is [Knee] quality points per dollar (prior.json). It sits where the +// quality-cost front bends: a step up that buys fewer than [Knee] points per +// dollar is not taken, and one that buys more is. Because the open-ended +// checker's link pays for ability and a fix's support seats' do not, that +// buys a strong checker on open-ended work and nothing on a fix, and it does +// so because of the weights and the prices, not because of a branch. +// +// cheapFactor makes cheap ten times as stingy: quality must come at under +// half a cent a point. stepFactor is one escalation step: a quarter of the +// price per point, so each step buys what cost up to four times as much. +const ( + cheapFactor = 10.0 + stepFactor = 4.0 + maxSteps = 6 +) + +// Request is everything one decision reads. +type Request struct { + Task Task + // Class, when set, is taken as given and the classifier is not asked — a + // replay, or a caller that already knows. + Class Class + // Reading, when set, is the classifier's whole answer as the caller + // already read it — the class, why, how sure, and a fix's reach — and is + // taken as it stands. A caller that classifies first to key its learned + // offset hands the reading on here rather than the class alone, which + // dropped the reach and ran every complex fix on the simple fix's worker. + Reading *Reading + // Candidates are the allowed models reachable on a connected provider. + Candidates []Candidate + // Pins are the seats the person fixed. A pinned seat always runs its pin. + Pins map[Seat]Pin + Effort Effort + // Steps is how many escalation steps above the knee to start: the learned + // offset for this repository and class, which decays as tasks are accepted. + Steps int + // Pace multiplies λ as a daily cap is approached ([Pace]); zero is one. + Pace float64 + // Stronger is the crew that ran, when the person asked to redo the task + // stronger: every unpinned seat is picked at least as strong, and the crew + // as a whole strictly stronger, or the decision says it cannot be. + Stronger *Decision + // Again is the crew that ran and NEVER STARTED — its seats' first calls + // were refused — when the person asks to redo it: the task never ran, so it + // is asked again at the same λ on the next-best models, not escalated. + Again *Decision + // Avoid are model lineages that failed to start on this install recently + // ([Lineage]). An unpinned seat is not given one while anything else can + // sit it; a pin is never overruled. + Avoid map[string]bool + // Learned is this install's move to a model's quality in a seat, by + // [LearnKey]: what its tasks kept or redid there ([router.CrewLog]). + Learned map[string]float64 + // TaskCap is the most one task may be estimated to cost; zero is none. A + // crew estimated over it is not chosen, whatever the effort word. + TaskCap float64 + // CostFactor is this install's learned ratio of what its tasks of the + // class cost to their estimates; zero is none learned. + CostFactor float64 + // Rescue is a seat's LAST RUNG being picked — the free pools when nothing + // paid can be reached — rather than a crew being chosen: any model that + // can sit the seat (tools, context, a route) is taken, best first, though + // it publishes too little for the router to trust it as a first pick. A + // seat that runs on a thinly described model says so on its line; a seat + // that does not run at all says nothing useful. + Rescue bool +} + +// Pick is one seat's answer. +type Pick struct { + Seat Seat + Model string + Provider string + Send string + Kind RouteKind + Pinned bool + Quality float64 + // SD is the standard deviation of Quality under the weights. + SD float64 `json:",omitempty"` + // Learned is the part of Quality this install's own outcomes moved. + Learned float64 `json:",omitempty"` + CostUSD float64 + // EstUSD is what this seat is expected to cost the task ([table.classCost]): + // nothing on a plan, a local model or a free pool. + EstUSD float64 `json:",omitempty"` +} + +// Decision is one task's crew. +type Decision struct { + Class Class + Why string + Sure bool + Effort Effort + Steps int + // Lambda is the price of a quality point the crew was picked at. + Lambda float64 + Crew []Pick + // EstUSD is the crew's estimated cost for an ordinary task of its class. + EstUSD float64 + Quality float64 + // OneOff is a stronger redo of a crew whose every seat was pinned: the + // pins were stepped over for this one run and are unchanged. + OneOff bool + // Considered is how many candidates the decision chose among. + Considered int + // Ladder is, for each unpinned seat, where the seat goes when its call + // fails to start, in order: the same model on its next routes, then the + // next qualified models at a similar cost. The caller adds what only it + // knows — the last crew that worked, the model the person is talking to. + Ladder map[Seat][]Pick `json:"-"` + // Retried are the seats that moved down their ladder during the task, in + // the order they moved, so the line can say so. + Retried []Retry `json:",omitempty"` + // Rungs are what a redo or an effort word changed against the crew it + // would otherwise have run, seat by seat. + Rungs []Retry `json:",omitempty"` + // Note is one plain sentence the line ends on: an effort word that changed + // nothing, or the free routes taken because nothing paid could be. + Note string `json:",omitempty"` + // Stopped is the one action a task stopped on when a seat had nowhere + // left to go and the failure said why — credit, a key, a limit. The line + // leads with it and offers no stronger redo, which could not help. + Stopped string `json:",omitempty"` + // Redo is whether this crew was asked for by a redo of a task that ran + // before. Its own acceptance is the redo's, not a later task's: it must + // not take back the step the redo just taught. + Redo bool `json:",omitempty"` + // Subclass is, for a bugfix, "complex" or "simple" ([complexFix]), with + // Reach the signals that made it complex. The log keeps it; every line a + // person reads still says bugfix. + Subclass string `json:",omitempty"` + Reach string `json:",omitempty"` + // CostFactor is what this install's own tasks of the class have cost + // against their estimates ([router.CrewLog.CostFactor]); EstUSD carries + // it. Zero is none learned yet. + CostFactor float64 `json:",omitempty"` +} + +// Unspent is the actual cost a line is drawn with when there is none worth +// saying: a task that stopped before anything was spent. The line then names +// no money at all, rather than a $0.000 that reads as a free success. +const Unspent = -2.0 + +// Retry is one seat moved from one model to another — down its ladder during +// a task, or up a rung on a redo — and why, when a failure moved it. +type Retry struct { + Seat Seat + From string + To string + Why string `json:",omitempty"` +} + +// WithRung is the decision with one seat moved to another pick for the rest +// of the task — a rung of its ladder, or one the caller found — the estimate +// re-added and the move recorded, with why, for the line. +func (d Decision) WithRung(seat Seat, next Pick, why string) Decision { + out := d + out.Crew = append([]Pick(nil), d.Crew...) + next.Seat = seat + out.Retried = append(append([]Retry(nil), d.Retried...), Retry{Seat: seat, From: d.Seat(seat).Model, To: next.Model, Why: why}) + out.EstUSD, out.Quality = 0, 0 + for i := range out.Crew { + if out.Crew[i].Seat == seat { + out.Crew[i] = next + } + out.EstUSD += estOf(out.Crew[i]) + out.Quality += out.Crew[i].Quality + } + if out.CostFactor > 0 { + out.EstUSD *= out.CostFactor + } + return out +} + +// Seat is one seat's pick, the zero Pick for a seat the decision has none for. +func (d Decision) Seat(seat Seat) Pick { + for _, pick := range d.Crew { + if pick.Seat == seat { + return pick + } + } + return Pick{} +} + +// NoCandidateError is a seat nobody pinned and nothing allowed can sit. +type NoCandidateError struct{ Seat Seat } + +func (e NoCandidateError) Error() string { + return fmt.Sprintf("no allowed model on a connected provider can sit the %s seat", e.Seat) +} + +// ErrStrongest is a redo that asked for a stronger crew than the strongest +// one the allowed models make. +var ErrStrongest = errors.New("this is already the strongest crew the models you allow can make") + +// Decide picks the crew for one task. +func Decide(r Request) (Decision, error) { + t := forDecision(r.Candidates, r.Learned) + t.rescue = r.Rescue + if r.Effort == EffortCheap { + t.cheapTolerance = cheapTolerance + } + reading := readingOf(r) + d := Decision{Class: reading.Class, Why: reading.Why, Sure: reading.Sure, Effort: r.Effort, Steps: r.Steps, Considered: len(r.Candidates)} + pins := r.Pins + if r.Stronger != nil && allPinned(pins) { + // EVERY SEAT PINNED AND A STRONGER RUN ASKED FOR: the pins stay what + // they are, and this one run steps over them. + pins, d.OneOff = nil, true + } + lambda := baseLambda(t, r) + d.Lambda = lambda + candidates := r.Candidates + avoid := r.Avoid + if r.Again != nil { + avoid = map[string]bool{} + for lineage := range r.Avoid { + avoid[lineage] = true + } + for _, pick := range r.Again.Crew { + if !pick.Pinned { + avoid[Lineage(pick.Model)] = true + } + } + } + if len(avoid) > 0 { + candidates = avoiding(candidates, avoid) + } + crew, err := pickCrew(t, d.Class, candidates, pins, lambda) + if err != nil { + return Decision{}, err + } + // A LEARNED OFFSET IS RUNGS, NOT A PRICE: each step is one rung up the + // front from the crew the knee picks ([stronger]), so a repository whose + // work was redone once starts one rung higher — never at the top. + for i := 0; i < r.Steps && i < maxSteps && r.Stronger == nil && r.Again == nil; i++ { + ran := Decision{Crew: crew} + up, _, err := stronger(t, d.Class, candidates, pins, lambda, &ran) + if err != nil { + break + } + crew = up + } + if d.Class == Bugfix { + d.Subclass, d.Reach = "simple", reading.Complex + if reading.Complex != "" { + d.Subclass = "complex" + if r.Stronger == nil && r.Again == nil { + crew = withReachingWorker(t, crew, candidates) + } + } + } + if r.Stronger != nil { + crew, d.Lambda, err = stronger(t, d.Class, candidates, pins, lambda, r.Stronger) + if err != nil { + return Decision{}, err + } + } + // NO CREW OVER THE TASK LIMIT: a crew whose estimate, at this install's + // cost factor, is over the limit is not chosen — not under --best, not on + // a redo. The price of a point is raised until the crew fits; pins are + // kept as they are. + if r.TaskCap > 0 && crewEst(t, d.Class, crew, r.CostFactor) > r.TaskCap+1e-12 { + var fits bool + crew, d.Lambda, fits = underCap(t, d.Class, candidates, pins, d.Lambda, r.TaskCap, r.CostFactor) + if fits { + d.Note = "held under the " + Money(r.TaskCap) + " task limit" + } else { + d.Note = "every crew is estimated over the " + Money(r.TaskCap) + " task limit · the cheapest runs" + } + } + d.Crew = crew + // AN EFFORT WORD SAYS WHAT IT CHANGED, or that it changed nothing: the crew + // the knee would have picked is the one to compare with. + var knee *Decision + if (r.Effort == EffortBest || r.Effort == EffortCheap) && r.Stronger == nil && r.Again == nil { + plain := r + plain.Effort, plain.Steps = "", 0 + kneeT := *t + kneeT.cheapTolerance = 0 + if kneeCrew, err := pickCrew(&kneeT, d.Class, candidates, pins, baseLambda(t, plain)); err == nil { + knee = &Decision{Crew: kneeCrew} + if sameCrew(kneeCrew, crew) && d.Note == "" { + if r.Effort == EffortBest { + d.Note = "best · already the strongest crew allowed" + } else { + d.Note = "cheap · already the cheapest crew" + } + } + } + } + // A REDO SAYS WHAT IT CHANGED, seat by seat, so the card can show the rung + // it took rather than a new crew to compare by eye. + if ran := firstOf(firstOf(r.Stronger, r.Again), knee); ran != nil { + for _, pick := range crew { + if was := ran.Seat(pick.Seat); was.Model != "" && Lineage(was.Model) != Lineage(pick.Model) { + d.Rungs = append(d.Rungs, Retry{Seat: pick.Seat, From: was.Model, To: pick.Model}) + } + } + } + // EACH UNPINNED SEAT CARRIES ITS NEXT PICK, so a model that fails to start + // is replaced inside the task rather than ending it: the best candidate of + // another lineage at the λ the crew was picked at. + for _, pick := range crew { + if pick.Pinned { + continue + } + if ladder := ladderFor(t, d.Class, pick, candidates, d.Lambda); len(ladder) > 0 { + if d.Ladder == nil { + d.Ladder = map[Seat][]Pick{} + } + d.Ladder[pick.Seat] = ladder + } + } + for i := range d.Crew { + d.Crew[i].EstUSD = pickEst(t, d.Class, d.Crew[i]) + } + for _, pick := range d.Crew { + d.EstUSD += estOf(pick) + d.Quality += pick.Quality + } + // THE ESTIMATE LEARNS FROM THIS INSTALL: what its tasks of the class + // actually cost against their estimates moves the next one's. + if r.CostFactor > 0 { + d.CostFactor = r.CostFactor + d.EstUSD *= r.CostFactor + } + return d, nil +} + +// pickEst is one pick's expected cost for the estimate: nothing where the +// route bills nothing, the seat's expected cost for the class otherwise. +func pickEst(t *table, class Class, pick Pick) float64 { + switch pick.Kind { + case Plan, Local, Free: + return 0 + } + if math.IsInf(pick.CostUSD, 0) { + return 0 + } + return pick.CostUSD +} + +// crewEst is a crew's estimate against a task limit: each seat's expected +// cost, times this install's cost factor when it is above one — the reading +// that errs dear. +func crewEst(t *table, class Class, crew []Pick, factor float64) float64 { + var sum float64 + for _, pick := range crew { + sum += pickEst(t, class, pick) + } + if factor > 1 { + sum *= factor + } + return sum +} + +// underCap is the crew picked at the least price of a point, from λ up, whose +// estimate fits the task limit, and whether one fits. With none, it is the +// cheapest crew the doubling reached. +func underCap(t *table, class Class, candidates []Candidate, pins map[Seat]Pin, lambda, limit, factor float64) ([]Pick, float64, bool) { + at := math.Max(lambda, t.Knee/64) + var last []Pick + for i := 0; i < 40; i++ { + at *= 2 + crew, err := pickCrew(t, class, candidates, pins, at) + if err != nil { + break + } + last = crew + if crewEst(t, class, crew, factor) <= limit+1e-12 { + return crew, at, true + } + } + return last, at, false +} + +// estOf is a pick's expected cost: its estimate, or — for a pick made where +// no estimate was read, a rescue — what it was weighed at. +func estOf(pick Pick) float64 { + switch { + case pick.EstUSD > 0: + return pick.EstUSD + case pick.Kind == Plan || pick.Kind == Local || pick.Kind == Free: + return 0 + } + return pick.CostUSD +} + +// baseLambda is the price of a quality point this request starts at: the +// knee, or the effort's own, moved by the learned steps and the day's pace. +func baseLambda(t *table, r Request) float64 { + lambda := t.Knee + switch r.Effort { + case EffortBest: + return 0 + case EffortCheap: + lambda *= cheapFactor + } + if r.Pace > 0 { + lambda *= r.Pace + } + return lambda +} + +// firstOf is the first of two decisions that is set. +func firstOf(a, b *Decision) *Decision { + if a != nil { + return a + } + return b +} + +// ladderRivals is how many other models a seat's ladder offers after its +// own model's routes. +const ladderRivals = 2 + +// ladderLook is how many times ladderRivals the ladder weighs before it puts +// those at a similar cost first; similarCost is how much dearer than the +// seat's pick "similar" allows. +const ( + ladderLook = 3 + similarCost = 3.0 +) + +// ladderFor is where a seat goes when its call fails to start, in order: +// +// 1. THE SAME MODEL ON ITS NEXT ROUTES, cheapest first. A route refusing says +// something about the route, and the model was picked on its own merits. +// 2. THE NEXT QUALIFIED MODELS for the seat at the same λ — the picks the +// seat would have had without this model — so the fallback costs about +// what the pick did rather than whatever is left. +func ladderFor(t *table, class Class, pick Pick, candidates []Candidate, lambda float64) []Pick { + // A SEAT THAT CANNOT START falls back to a credible model at a similar + // cost; the ability floor a first pick must reach does not send it to a + // model many times dearer. + fallback := *t + fallback.abilityFloor = false + t = &fallback + var ladder []Pick + own := Lineage(pick.Model) + for _, c := range candidates { + if t.abilityOf(c.Model).lineage != own { + continue + } + rest := append([]Route(nil), c.Routes...) + used := map[string]bool{pick.Send: true} + for { + var left []Route + for _, r := range rest { + if !used[r.Send] { + left = append(left, r) + } + } + if len(left) == 0 { + break + } + c.Routes = left + next := pickOf(t, class, pick.Seat, c) + used[next.Send] = true + ladder = append(ladder, next) + rest = left + } + break + } + // The rivals, in ONE pass: the best few other lineages by the score + // [bestFor] ranks on, kept in order as they are found — and then those at + // a SIMILAR COST to the seat's pick ahead of dearer ones, so a seat that + // cannot start moves sideways before it moves up: a failed call is not a + // request for a stronger crew. + keep := ladderRivals * ladderLook + type rival struct { + pick Pick + score float64 + } + var rivals []rival + for _, c := range candidates { + if t.abilityOf(c.Model).lineage == own { + continue + } + next, ok := eligible(t, class, pick.Seat, c) + if !ok { + continue + } + score := next.Quality - lambda*weighedCost(t, class, pick.Seat, next) + at := len(rivals) + for at > 0 && score > rivals[at-1].score+1e-12 { + at-- + } + if at >= keep { + continue + } + rivals = append(rivals, rival{}) + copy(rivals[at+1:], rivals[at:]) + rivals[at] = rival{pick: next, score: score} + if len(rivals) > keep { + rivals = rivals[:keep] + } + } + near := math.Max(pick.CostUSD, t.costFloor(pick.Seat)*t.costScale(class, pick.Seat)) * similarCost + sort.SliceStable(rivals, func(i, j int) bool { + return rivals[i].pick.CostUSD <= near && rivals[j].pick.CostUSD > near + }) + if len(rivals) > ladderRivals { + rivals = rivals[:ladderRivals] + } + for _, r := range rivals { + ladder = append(ladder, r.pick) + } + return ladder +} + +// sameCrew is whether two crews seat the same models. +func sameCrew(a, b []Pick) bool { + if len(a) != len(b) { + return false + } + for _, pick := range a { + if Lineage(seatOf(b, pick.Seat).Model) != Lineage(pick.Model) { + return false + } + } + return true +} + +// seatOf is a crew's pick for a seat. +func seatOf(crew []Pick, seat Seat) Pick { + if i := seatIndex(crew, seat); i >= 0 { + return crew[i] + } + return Pick{} +} + +// avoiding is the candidates without the lineages named — unless that would +// leave none, because a crew on a model that failed once is better than no +// crew at all, and the task then says why it stopped. +func avoiding(candidates []Candidate, avoid map[string]bool) []Candidate { + out := make([]Candidate, 0, len(candidates)) + for _, c := range candidates { + if !avoid[Lineage(c.Model.ID)] { + out = append(out, c) + } + } + if len(out) == 0 { + return candidates + } + return out +} + +// allPinned is whether every seat is pinned. +func allPinned(pins map[Seat]Pin) bool { + for _, seat := range Seats { + if _, ok := pins[seat]; !ok { + return false + } + } + return true +} + +// pickCrew picks every seat at one λ. The seats are independent — a crew's +// quality is the sum of its seats' and so is its cost — so the best crew is +// each seat's best pick, and no combination has to be searched. +func pickCrew(t *table, class Class, candidates []Candidate, pins map[Seat]Pin, lambda float64) ([]Pick, error) { + crew := make([]Pick, 0, len(Seats)) + for _, seat := range Seats { + if pin, ok := pins[seat]; ok { + crew = append(crew, pinned(t, class, seat, pin, candidates)) + continue + } + pick, ok := bestFor(t, class, seat, candidates, lambda, math.Inf(-1)) + if !ok { + return nil, NoCandidateError{Seat: seat} + } + if seat == Worker && t.cheapTolerance > 1 { + pick = strongerWithin(t, class, seat, candidates, pick, t.cheapTolerance) + } + crew = append(crew, pick) + } + if lambda > 0 && class != Bugfix { + crew = checkerFirst(t, class, crew, candidates, pins) + } + return crew, nil +} + +// checkerFirst puts a support upgrade in the checker's seat. ON WORK WHOSE +// SUPPORT SEATS THE WEIGHTS DO NOT TELL APART — their slopes within one +// standard deviation of each other — the stronger of the two support picks +// goes to the checker, which decides whether the work is accepted, and the +// weaker to the planner. A pinned seat is never moved, and a model that +// cannot sit the other seat stays where it was. +func checkerFirst(t *table, class Class, crew []Pick, candidates []Candidate, pins map[Seat]Pin) []Pick { + if _, ok := pins[Planner]; ok { + return crew + } + if _, ok := pins[Checker]; ok { + return crew + } + link := t.linkOf(class) + p, c := link.Seats[Planner], link.Seats[Checker] + if math.Abs(p.Slope-c.Slope) > math.Max(p.SlopeSD, c.SlopeSD) { + return crew + } + pi, ci := -1, -1 + for i, pick := range crew { + switch pick.Seat { + case Planner: + pi = i + case Checker: + ci = i + } + } + if pi < 0 || ci < 0 || Lineage(crew[pi].Model) == Lineage(crew[ci].Model) { + return crew + } + byID := map[string]Candidate{} + for _, cand := range candidates { + byID[cand.Model.ID] = cand + } + planCand, okP := byID[crew[pi].Model] + checkCand, okC := byID[crew[ci].Model] + if !okP || !okC { + return crew + } + asChecker, okC := eligible(t, class, Checker, planCand) + asPlanner, okP := eligible(t, class, Planner, checkCand) + if !okC || !okP || asChecker.Quality <= crew[ci].Quality+1e-9 { + return crew + } + out := append([]Pick(nil), crew...) + out[pi], out[ci] = asPlanner, asChecker + return out +} + +// strongerWithin is a cheap seat's pick traded for the strongest eligible +// model that costs at most tolerance times as much: a model barely dearer and +// clearly stronger is the better buy even when quality is nearly free. +func strongerWithin(t *table, class Class, seat Seat, candidates []Candidate, pick Pick, tolerance float64) Pick { + limit := weighedCost(t, class, seat, pick) * tolerance + best := pick + for _, c := range candidates { + other, ok := eligible(t, class, seat, c) + if !ok || weighedCost(t, class, seat, other) > limit+1e-12 || other.Quality <= best.Quality+1e-9 { + continue + } + best = other + } + return best +} + +// bestFor is one seat's best pick at λ among the models whose quality is +// above floor. Ties go to the cheaper pick and then to the smaller id, so the +// answer never depends on the order the candidates arrived in. +func bestFor(t *table, class Class, seat Seat, candidates []Candidate, lambda, floor float64) (Pick, bool) { + var best Pick + var bestScore float64 + found := false + for _, c := range candidates { + pick, ok := eligible(t, class, seat, c) + if !ok || pick.Quality <= floor+1e-9 { + continue + } + score := pick.Quality - lambda*weighedCost(t, class, seat, pick) + switch { + case !found, score > bestScore+1e-12: + case score < bestScore-1e-12: + continue + case pick.CostUSD < best.CostUSD-1e-12: + case pick.CostUSD > best.CostUSD+1e-12: + continue + case pick.Model < best.Model: + default: + continue + } + best, bestScore, found = pick, score, true + } + return best, found +} + +// Scored is one candidate as a seat weighed it: the model, the route it +// would ride, its quality and cost as the router read them, and the score +// (quality − λ·cost) it was ranked by. +type Scored struct { + Model string `json:"m"` + Provider string `json:"p,omitempty"` + Kind RouteKind `json:"k,omitempty"` + Quality float64 `json:"q"` + CostUSD float64 `json:"c"` + Score float64 `json:"s"` +} + +// Explain is each seat's best n candidates at the λ a decision was made at, +// best first — what a router log row carries so a crew can be explained from +// the log alone. +func Explain(class Class, lambda float64, candidates []Candidate, learned map[string]float64, n int) map[Seat][]Scored { + t := forDecision(candidates, learned) + out := map[Seat][]Scored{} + for _, seat := range Seats { + var ranked []Scored + for _, c := range candidates { + pick, ok := eligible(t, class, seat, c) + if !ok { + continue + } + cost := weighedCost(t, class, seat, pick) + ranked = append(ranked, Scored{Model: pick.Model, Provider: pick.Provider, Kind: pick.Kind, + Quality: round3(pick.Quality), CostUSD: round3(cost), Score: round3(pick.Quality - lambda*cost)}) + } + sort.SliceStable(ranked, func(i, j int) bool { return ranked[i].Score > ranked[j].Score }) + if len(ranked) > n { + ranked = ranked[:n] + } + out[seat] = ranked + } + return out +} + +// round3 keeps a logged figure to three places. +func round3(x float64) float64 { return math.Round(x*1000) / 1000 } + +// seatable is whether a candidate can sit a seat at all: it takes tool calls, +// it has a route, and its context holds the seat's work. A free pool is a +// ROUTE of its model, not a model ([Free]): whether the model may sit the seat +// is decided on the model's own merits, and the route is chosen after. +func seatable(seat Seat, c Candidate) bool { + if !c.Model.Tools || len(c.Routes) == 0 { + return false + } + return c.Model.Context <= 0 || c.Model.Context >= seatContext[seat] +} + +// Seatable is [seatable] for a caller that offers models for a seat — a +// picker, a rescue — so it offers exactly what the router could seat. +func Seatable(seat Seat, c Candidate) bool { return seatable(seat, c) } + +// seatContext is the least context window a model needs to sit each seat, in +// tokens: the worker holds a long agent loop over a repository, the checker +// reads the work and its diff, the planner reads a brief and writes a plan. A +// model that publishes no window is not refused for it. +var seatContext = map[Seat]int{Worker: 64_000, Checker: 64_000, Planner: 32_000} + +// IsFree is whether an id names a free, rate-limited route (`…:free`). +func IsFree(id string) bool { + return strings.HasSuffix(strings.ToLower(strings.TrimSpace(id)), ":free") +} + +// eligible is one candidate in one seat, and whether it may sit it: seatable, +// and credible under the weights ([table.credible]) — its row says enough to +// score it, and its ability may reach what the seat needs. +func eligible(t *table, class Class, seat Seat, c Candidate) (Pick, bool) { + if !seatable(seat, c) { + return Pick{}, false + } + a := t.abilityOf(c.Model) + pick := pickAt(t, class, seat, c, a) + if t.rescue { + return pick, true + } + return pick, t.seatCredible(seat, a) +} + +// weighedCost is the cost a pick is weighed at: its expected cost on its +// route, an unpriced model's already floored ([pickOf]). +func weighedCost(t *table, class Class, seat Seat, pick Pick) float64 { + return pick.CostUSD +} + +// pickOf is one candidate in one seat on its cheapest route. Routes of equal +// cost keep the caller's order, which puts the provider the person connected +// for that vendor ahead of a router. +func pickOf(t *table, class Class, seat Seat, c Candidate) Pick { + return pickAt(t, class, seat, c, t.abilityOf(c.Model)) +} + +// pickAt is [pickOf] on the candidate's ability already read. +func pickAt(t *table, class Class, seat Seat, c Candidate, a ability) Pick { + q, sd := t.qualityOf(class, seat, a) + metered := t.classCost(class, seat, c.Model) + unpriced := c.Model.PromptPrice <= 0 && c.Model.CompletionPrice <= 0 + floor := 0.0 + if unpriced { + // An unpriced model is weighed at what a priced model as able costs, + // never at zero. + floor = t.priceFor(seat, a.U) * t.costScale(class, seat) + metered = floor + } + fallback := metered + pick := Pick{Seat: seat, Model: c.Model.ID, Quality: q, SD: sd, Learned: t.learnedOf(class, seat, a), + CostUSD: math.Inf(1)} + for _, route := range c.Routes { + cost := routeCost(route, metered, fallback) + if unpriced && route.Kind == Free { + // A free pool of a model nobody prices is weighed above the + // cheapest priced model: its zero is not a price. + cost = math.Max(cost, floor+freeLimitPenalty) + } + if cost < pick.CostUSD-1e-12 { + pick.Provider, pick.Send, pick.Kind, pick.CostUSD = route.Provider, route.Send, route.Kind, cost + } + } + return pick +} + +// routeCost is what one more task is EXPECTED to cost on a route: what it +// bills — the metered price, nothing on a plan, a local model or a free pool — +// plus what its refusals cost: the chance it refuses a first call times the +// paid route the seat then falls to (fallback). A free pool's chance starts +// pessimistic and it pays a charge for its limits. +// +// A FREE ROUTE IS NEVER PRICED AT ZERO. Zero is what made a free pool win a +// worker seat and die in its first second; its expected cost is what a +// person actually pays for choosing it. +func routeCost(route Route, metered, fallback float64) float64 { + switch route.Kind { + case Plan, Local: + return route.FailRate * fallback + case Free: + rate := route.FailRate + if rate <= 0 { + rate = freeFailPrior + } + return rate*fallback + math.Max(freeLimitPenalty, freeLimitShare*fallback) + } + // A ROUTE THAT REFUSES SOME FIRST CALLS costs what it bills plus what its + // refusals send elsewhere, at this install's learned rate. + return metered + route.FailRate*fallback +} + +// pinned is a pinned seat's pick: the pin, always, on the route the pin names +// when it names one. A pinned model the candidates do not carry — outside the +// catalog, or on a provider the router cannot price — still sits the seat on +// the send the caller resolved; its quality is the weights' reading of what +// its row carries, and its cost what its route publishes, or nothing. +func pinned(t *table, class Class, seat Seat, pin Pin, candidates []Candidate) Pick { + pick := Pick{Seat: seat, Model: pin.Model, Provider: pin.Provider, Send: pin.Send, Kind: pin.Kind, Pinned: true} + for _, c := range candidates { + if Lineage(c.Model.ID) != Lineage(pin.Model) { + continue + } + found := pickOf(t, class, seat, c) + if pin.Provider != "" { + for _, route := range c.Routes { + if strings.EqualFold(route.Provider, pin.Provider) { + found.Provider, found.Send, found.Kind = route.Provider, route.Send, route.Kind + metered := t.classCost(class, seat, c.Model) + found.CostUSD = routeCost(route, metered, math.Max(metered, t.costFloor(seat)*t.costScale(class, seat))) + } + } + } + if math.IsInf(found.CostUSD, 1) { + found.CostUSD = 0 + } + if pin.Send != "" && (found.Send == "" || strings.EqualFold(found.Provider, pin.Provider)) { + // THE PIN'S OWN SEND ON ITS OWN ROUTE: the id the person wrote (or + // the variant it resolved to), never the candidate's first row. + found.Send = pin.Send + if found.Kind == "" || pin.Kind == Free { + found.Kind = pin.Kind + } + } + found.Model = pin.Model + found.Pinned = true + return found + } + // A pin the candidates do not carry is read on its id alone: the weights' + // prior for a model they know nothing of, at no known cost. + pick.Quality, pick.SD = t.quality(class, seat, Model{ID: pin.Model}) + if pick.Send == "" { + pick.Send = pin.Model + } + return pick +} + +// stronger is a redo's crew: ONE RUNG UP, on one seat. +// +// Every seat keeps what ran; then the unpinned seat whose next rung buys the +// most — quality gained less the dearer price at one step below this λ — moves +// up to that rung and no further. The next rung is the least step up the +// seat's front ([nextRung]): the next model above it by quality that no other +// model beats on both quality and cost. A redo that jumped to the top of the +// catalog skipped every mid-price model and multiplied the bill for one +// sentence of dissatisfaction; a ladder is walked a rung at a time, and a +// second redo takes the next rung. Ties go to the checker, then the worker, +// then the planner — the checker is the seat the routing rule upgrades first. +func stronger(t *table, class Class, candidates []Candidate, pins map[Seat]Pin, lambda float64, ran *Decision) ([]Pick, float64, error) { + crew := make([]Pick, 0, len(Seats)) + for _, seat := range Seats { + if pin, ok := pins[seat]; ok { + crew = append(crew, pinned(t, class, seat, pin, candidates)) + continue + } + was := ran.Seat(seat) + if was.Model == "" { + pick, ok := bestFor(t, class, seat, candidates, lambda, math.Inf(-1)) + if !ok { + return nil, 0, NoCandidateError{Seat: seat} + } + was = pick + } + // A one-off over every pin starts from the pins as they ran. + was.Pinned = false + crew = append(crew, was) + } + step := lambda / stepFactor + at, gain := -1, math.Inf(-1) + var up Pick + for _, seat := range []Seat{Checker, Worker, Planner} { + i := seatIndex(crew, seat) + if i < 0 || crew[i].Pinned { + continue + } + next, ok := nextRung(t, class, seat, candidates, crew[i]) + if !ok { + continue + } + g := (next.Quality - crew[i].Quality) - step*(weighedCost(t, class, seat, next)-weighedCost(t, class, seat, crew[i])) + if g > gain+1e-12 { + at, gain, up = i, g, next + } + } + if at < 0 { + return nil, 0, ErrStrongest + } + crew[at] = up + return crew, step, nil +} + +// readingOf is the reading a decision is made on: the caller's own when it +// handed one on, the class given with the reach read off the task's words +// when it gave only the class, and the classifier's otherwise. A CLASS GIVEN +// IS NOT A REACH FORGOTTEN: a bugfix named by a replay or a caller is still +// read for reach whenever the task's words are there to read. +func readingOf(r Request) Reading { + switch { + case r.Reading != nil && (r.Class == "" || r.Class == r.Reading.Class): + return *r.Reading + case r.Class == "": + return Classify(r.Task) + } + reading := Reading{Class: r.Class, Why: "given", Sure: true} + if r.Class == Bugfix { + reading.Complex = complexFix(reachText(taskTitle(r.Task.Text))) + } + return reading +} + +// withReachingWorker is a complex fix's crew: its worker one rung up the +// seat's front ([nextRung]) from the one the knee picked — the next model +// above it by quality that nothing beats on both quality and cost, not the +// top of the catalog. A pinned worker stays the pin, and a worker already at +// the top stays where it is. +func withReachingWorker(t *table, crew []Pick, candidates []Candidate) []Pick { + i := seatIndex(crew, Worker) + if i < 0 || crew[i].Pinned { + return crew + } + next, ok := nextRung(t, Bugfix, Worker, candidates, crew[i]) + if !ok { + return crew + } + out := append([]Pick(nil), crew...) + out[i] = next + return out +} + +// seatIndex is where a seat's pick sits in a crew, -1 when it is not there. +func seatIndex(crew []Pick, seat Seat) int { + for i, pick := range crew { + if pick.Seat == seat { + return i + } + } + return -1 +} + +// nextRung is the least step up a seat's front from the pick it has: among the +// models above it by quality, those no other model beats on both quality and +// weighed cost, and of those the one with the least quality — ties to the +// cheaper. +func nextRung(t *table, class Class, seat Seat, candidates []Candidate, from Pick) (Pick, bool) { + own := Lineage(from.Model) + var above []Pick + for _, c := range candidates { + pick, ok := eligible(t, class, seat, c) + if !ok || pick.Quality <= from.Quality+1e-9 || t.abilityOf(c.Model).lineage == own { + continue + } + above = append(above, pick) + } + // The front, best first: a pick is on it when nothing at least as good + // costs no more (and one of the two strictly). Sweeping by quality from + // the top, a pick is dominated exactly when an earlier one is no dearer. + sort.SliceStable(above, func(i, j int) bool { + if math.Abs(above[i].Quality-above[j].Quality) > 1e-9 { + return above[i].Quality > above[j].Quality + } + return weighedCost(t, class, seat, above[i]) < weighedCost(t, class, seat, above[j]) + }) + var best Pick + found := false + cheapest := math.Inf(1) + for i := 0; i < len(above); { + // One quality level at a time: within it only the cheapest can be on + // the front, and it is dominated by anything above it no dearer. + j := i + for j < len(above) && math.Abs(above[j].Quality-above[i].Quality) <= 1e-9 { + j++ + } + p := above[i] + pc := weighedCost(t, class, seat, p) + if pc < cheapest-1e-12 { + best, found = p, true + } + for k := i; k < j; k++ { + cheapest = math.Min(cheapest, weighedCost(t, class, seat, above[k])) + } + i = j + } + return best, found +} + +// nextUp is the cheapest model whose quality in the seat is above floor. +func nextUp(t *table, class Class, seat Seat, candidates []Candidate, floor float64) (Pick, bool) { + var best Pick + found := false + for _, c := range candidates { + pick, ok := eligible(t, class, seat, c) + if !ok || pick.Quality <= floor+1e-9 { + continue + } + if !found || pick.Quality < best.Quality-1e-9 || + (math.Abs(pick.Quality-best.Quality) <= 1e-9 && (pick.CostUSD < best.CostUSD || (pick.CostUSD == best.CostUSD && pick.Model < best.Model))) { + best, found = pick, true + } + } + return best, found +} + +// Pace is how a daily cap moves the price of a quality point: nothing until +// half the cap is spent, then λ grows as what is left shrinks — twice as +// stingy with a quarter of the cap left, five times with a tenth — so a day +// that is running hot drifts to cheaper crews before it reaches the wall. +// atCap is the wall itself: the day's spend has reached the cap. A cap of +// zero is no cap. +func Pace(spent, cap float64) (multiplier float64, atCap bool) { + if cap <= 0 { + return 1, false + } + if spent >= cap { + return 1, true + } + frac := spent / cap + if frac <= 0.5 { + return 1, false + } + return 1 / (2 * (1 - frac)), false +} + +// Gap is one class of work the allowed models leave without a seat the +// routing rule requires. +type Gap struct { + Class Class + Seat Seat + Line string +} + +// Gaps names what the allowed models cannot cover. Today that is one thing, +// the seat the routing rule depends on: open-ended work with no strong checker +// — none credible whose ability reaches the middle the open-ended link is +// centred on. +// +// A PINNED SEAT IS THE SEAT. A pin always runs, so a pinned checker is the only +// model the question is asked of: a strong pin leaves no gap however weak the +// rest of the allowed models are, and a weak pin is a gap however strong they +// are, said with the pin's name because the pin is what a person would change. +// pins carry each pinned seat's model as the catalog reads it, or only its id +// when the catalog does not carry it; a candidate of the same lineage is read +// in its place, the way [pinned] reads it. An unpinned checker is asked of +// every model allowed, as before. +func Gaps(candidates []Candidate, pins map[Seat]Model) []Gap { + t := forDecision(candidates, nil) + strong := t.linkOf(OpenEnded).URef + isStrong := func(m Model) bool { + a := t.abilityOf(m) + return t.credibleAt(a) && a.U+math.Sqrt(a.VarU) >= strong + } + if pin, ok := pins[Checker]; ok && strings.TrimSpace(pin.ID) != "" { + for _, c := range candidates { + if Lineage(c.Model.ID) == Lineage(pin.ID) { + pin = c.Model + break + } + } + if isStrong(pin) { + return nil + } + return []Gap{{Class: OpenEnded, Seat: Checker, Line: "checker pinned to " + ShortModel(pin.ID) + " · open-ended work will be checked weakly"}} + } + for _, c := range candidates { + if seatable(Checker, c) && isStrong(c.Model) { + return nil + } + } + return []Gap{{Class: OpenEnded, Seat: Checker, Line: "no strong checker among the models you allow · open-ended work will be checked weakly"}} +} + +// ── how a decision reads ──────────────────────────────────────────────────── + +// Line is the one line a task card and a headless run's summary print: +// +// bugfix · worker glm-5.3-flash (openrouter) · checker glm-5.3-flash · $0.021 (est $0.023) +// +// pinMark is drawn in front of a pinned seat's model; the chat surface hands +// its own glyph and a headless door hands 📌. actual below zero is not known +// yet, and the line then ends on the estimate alone (the emptiness law: an +// unknown is absent, never $0.00). +func (d Decision) Line(pinMark string, actual float64) string { + var b strings.Builder + if len(d.Retried) > 0 { + // A SEAT THAT MOVED DURING THE TASK IS SAID FIRST, plainly: the crew + // running is not the crew picked, and why — before any figure a reader + // could take for the picked crew's success. + // + // EACH SEAT SAYS ITS NET MOVE, once: where it started, where it is + // now, the FIRST reason — the cause, which a free pool at its limit + // further down never is — and how many it tried between — never the + // whole trail, which is the router log's to keep. A line that listed + // every rung of a long ladder wrapped past the card. + b.WriteString("running on fallback crew") + for _, seat := range Seats { + var first, last Retry + moves := 0 + for _, r := range d.Retried { + if r.Seat != seat { + continue + } + if moves == 0 { + first = r + } + last, moves = r, moves+1 + } + if moves == 0 { + continue + } + b.WriteString(" · " + string(seat) + " " + ShortModel(first.From) + " → " + ShortModel(last.To)) + var note []string + if first.Why != "" { + note = append(note, first.Why) + } + if moves > 1 { + note = append(note, "+"+strconv.Itoa(moves-1)+" tried") + } + if len(note) > 0 { + b.WriteString(" (" + strings.Join(note, "; ") + ")") + } + } + b.WriteString(" · ") + } + b.WriteString(d.Class.Word()) + worker, planner, checker := d.Seat(Worker), d.Seat(Planner), d.Seat(Checker) + // EACH SEAT IS SAID ONCE, WITH ITS CHANGE: a seat a redo or an effort word + // moved reads `planner glm-5.3-flash → kimi-k3` where the seat stands, + // never the seat and then its rung again at the end. + rung := map[Seat]Retry{} + for _, r := range d.Rungs { + rung[r.Seat] = r + } + said := func(pick Pick) string { + if r, ok := rung[pick.Seat]; ok { + return ShortModel(r.From) + " → " + seatModel(pick, pinMark) + } + return seatModel(pick, pinMark) + } + b.WriteString(" · worker ") + b.WriteString(said(worker)) + if worker.Provider != "" { + route := worker.Provider + if worker.Kind == Free { + route += " · free" + } + b.WriteString(" (" + route + ")") + } + // THE PLANNER IS NAMED WHEN IT IS NOT THE WORKER. A crew whose planner is + // the worker's model says nothing a person reading the line needs; one + // whose planner is another model is a crew of three, and says so. + _, plannerMoved := rung[Planner] + if planner.Model != "" && (plannerMoved || Lineage(planner.Model) != Lineage(worker.Model)) { + b.WriteString(" · planner ") + b.WriteString(said(planner)) + } + b.WriteString(" · checker ") + b.WriteString(said(checker)) + switch { + case actual == Unspent: + case actual >= 0: + b.WriteString(" · " + Money(actual) + " (est " + Money(d.EstUSD) + ")") + default: + b.WriteString(" · est " + Money(d.EstUSD)) + } + if d.Note != "" { + b.WriteString(" · " + d.Note) + } + return b.String() +} + +// seatModel is one seat's model the way the line names it: the name after the +// vendor, with the pin mark in front when the seat was pinned. +func seatModel(pick Pick, pinMark string) string { + name := ShortModel(pick.Model) + if sent := ShortModel(pick.Send); pick.Pinned && sent != "" && sent != name { + // A PIN SENT AS ANOTHER ID SAYS WHICH: `deepseek-v4-flash → -0731`. + if strings.HasPrefix(sent, name) { + sent = sent[len(name):] + } + name += " → " + sent + } + switch { + case pick.Pinned && strings.TrimSpace(pinMark) != "": + return pinMark + " " + name + case pick.Pinned: + // NO MARK IS NO BLANK: a surface with no glyph for a pin says the word. + return name + " (pinned)" + } + return name +} + +// ShortModel is a model id without its vendor: the half a person reads. +func ShortModel(id string) string { + id = strings.TrimSpace(id) + if at := strings.LastIndex(id, "/"); at >= 0 { + return id[at+1:] + } + return id +} + +// Money spells a task's dollars: three places under a dollar, where the +// difference between routed crews lives, and two above it. +func Money(usd float64) string { + if usd >= 1 { + return fmt.Sprintf("$%.2f", usd) + } + return fmt.Sprintf("$%.3f", usd) +} + +// Names lists the candidate ids a decision chose among, sorted — the field a +// logged decision carries so it can be analysed after the fact. +func Names(candidates []Candidate) []string { + names := make([]string, 0, len(candidates)) + for _, c := range candidates { + names = append(names, c.Model.ID) + } + sort.Strings(names) + return names +} diff --git a/internal/crewroute/route_test.go b/internal/crewroute/route_test.go new file mode 100644 index 0000000000..f398dc9e90 --- /dev/null +++ b/internal/crewroute/route_test.go @@ -0,0 +1,698 @@ +package crewroute + +import ( + "errors" + "math" + "math/rand" + "strings" + "testing" + "time" +) + +func TestThePriorWeightsParseAndAreSmall(t *testing.T) { + if len(priorJSON) > 2<<20 { + t.Fatalf("prior.json is %d bytes; the embedded weights must stay under 2 MB", len(priorJSON)) + } + w := load() + if w.Knee <= 0 || len(w.Shapes) != 3 || len(w.Features) == 0 { + t.Fatalf("weights incomplete: knee %v, %d shapes, %d features", w.Knee, len(w.Shapes), len(w.Features)) + } + for _, class := range Classes { + for _, seat := range Seats { + if _, ok := w.Link[class].Seats[seat]; !ok { + t.Errorf("no link for %s/%s", class, seat) + } + } + } +} + +// THE WEIGHTS CARRY NO MODEL OF THEIR OWN: every model, the ones a person +// knows best included, is scored from its catalog row. +func TestTheWeightsCarryNoModelRows(t *testing.T) { + for _, m := range []Model{glmFlash, kimiK3, v4Flash} { + if strings.Contains(string(priorJSON), m.ID) || strings.Contains(string(priorJSON), ShortModel(m.ID)) { + t.Errorf("prior.json names %s", m.ID) + } + } +} + +// THE ROUTED POLICY, read off the weights and the prices rather than branches: +// open-ended work pays more for ability in every seat than a fix does, so +// there are prices at which open-ended work buys the stronger checker and a +// fix does not — and none at which a fix buys it and open-ended work does not. +func TestOpenEndedWorkBuysAStrongerCheckerAtPricesAFixDoesNot(t *testing.T) { + tab := prior() + for _, seat := range Seats { + if fix, open := tab.linkOf(Bugfix).Seats[seat].Slope, tab.linkOf(OpenEnded).Seats[seat].Slope; open <= fix { + t.Errorf("%s: open-ended slope %.3f not above the fix's %.3f", seat, open, fix) + } + } + weak := candidateOf(glmFlash) + split := false + for mult := 0.25; mult <= 64; mult *= 1.25 { + strong := glm53 + strong.ID = "acme/strong-checker" + strong.PromptPrice, strong.CompletionPrice, strong.CacheReadPrice = glmFlash.PromptPrice*mult, glmFlash.CompletionPrice*mult, glmFlash.CacheReadPrice*mult + cands := []Candidate{weak, candidateOf(strong)} + fix, err := Decide(Request{Class: Bugfix, Candidates: cands}) + if err != nil { + t.Fatal(err) + } + open, err := Decide(Request{Class: OpenEnded, Candidates: cands}) + if err != nil { + t.Fatal(err) + } + fixBuys, openBuys := fix.Seat(Checker).Model == strong.ID, open.Seat(Checker).Model == strong.ID + if fixBuys && !openBuys { + t.Errorf("at %.2fx the flash price a fix buys the strong checker and open-ended work does not", mult) + } + split = split || (openBuys && !fixBuys) + } + if !split { + t.Error("no price at which open-ended work buys the stronger checker and a fix does not") + } + for _, cands := range [][]Candidate{catalogCandidates(), frontierCandidates()} { + fix, _ := Decide(Request{Class: Bugfix, Candidates: cands}) + open, _ := Decide(Request{Class: OpenEnded, Candidates: cands}) + if fix.EstUSD >= open.EstUSD { + t.Errorf("a fix estimates $%.3f, open-ended work $%.3f: the fix should be the cheaper crew", fix.EstUSD, open.EstUSD) + } + } +} + +// A CHEAPER MODEL AT LEAST AS GOOD ON EVERY PUBLISHED INDEX NEVER LOSES. For +// rows that publish the same indexes, A no dearer than B and at least as good +// on each of them, B is never scored above A in any seat nor cheaper, and a +// decision over the two never seats B — whatever their release dates, +// context, licence or family say. +func TestADominatedModelNeverBeatsItsDominator(t *testing.T) { + rng := rand.New(rand.NewSource(7)) + names := []string{"deepseek/deepseek-v9-flash", "z-ai/glm-9", "acme/model", "moonshotai/kimi-k9", "anthropic/claude-opus-9"} + for i := 0; i < 400; i++ { + var a, b Model + b = Model{ID: names[rng.Intn(len(names))], Open: rng.Intn(2) == 0, Tools: true, + PromptPrice: (0.05 + rng.Float64()*5) / 1e6, CompletionPrice: (0.1 + rng.Float64()*25) / 1e6, + Context: 1 << (17 + rng.Intn(4)), Released: released(2025+rng.Intn(2), time.Month(1+rng.Intn(12)), 1+rng.Intn(28))} + b.CacheReadPrice = b.PromptPrice / 10 + a = Model{ID: names[rng.Intn(len(names))] + "-a", Open: rng.Intn(2) == 0, Tools: true, + PromptPrice: b.PromptPrice * rng.Float64(), CompletionPrice: b.CompletionPrice * rng.Float64(), + Context: 1 << (17 + rng.Intn(4)), Released: released(2025+rng.Intn(2), time.Month(1+rng.Intn(12)), 1+rng.Intn(28))} + a.CacheReadPrice = a.PromptPrice / 10 + fields := []struct { + get func(*Model) *float64 + lo float64 + }{{func(m *Model) *float64 { return &m.Intelligence }, 20}, {func(m *Model) *float64 { return &m.Coding }, 40}, + {func(m *Model) *float64 { return &m.Agentic }, 20}, {func(m *Model) *float64 { return &m.ArenaElo }, 1150}} + published := 0 + for _, f := range fields { + if rng.Intn(2) == 0 { + continue + } + published++ + v := f.lo + rng.Float64()*f.lo*0.8 + *f.get(&b) = v + *f.get(&a) = v + rng.Float64()*f.lo*0.2 + } + if published == 0 { + a.Coding, b.Coding = 70, 60 + } + tab := prior() + for _, class := range []Class{Bugfix, OpenEnded, Other} { + for _, seat := range Seats { + qa, _ := tab.quality(class, seat, a) + qb, _ := tab.quality(class, seat, b) + if qb > qa+1e-9 { + t.Fatalf("%s %s: dominated %+v scored %.4f above %+v at %.4f", class, seat, b, qb, a, qa) + } + if tab.classCost(class, seat, b) < tab.classCost(class, seat, a) { + t.Fatalf("%s %s: the dominated model costs less", class, seat) + } + } + d, err := Decide(Request{Class: class, Candidates: []Candidate{candidateOf(b), candidateOf(a)}}) + if err != nil { + continue + } + for _, p := range d.Crew { + if p.Model == b.ID { + t.Fatalf("%s: %s seated the dominated %+v over %+v", class, p.Seat, b, a) + } + } + } + } + // The row that started it: a flash model no dearer on any price than + // another and better on every index the other publishes, which publishes + // one index to its four. + thin := Model{ID: "deepseek/deepseek-v4.1-flash", Open: true, PromptPrice: 1.5e-7, CompletionPrice: 6e-7, CacheReadPrice: glmFlash.CacheReadPrice, + Intelligence: 39.5, Context: 1048576, Released: released(2026, 9, 10), Tools: true} + for _, class := range []Class{Bugfix, OpenEnded} { + for _, effort := range []Effort{EffortCheap, EffortKnee} { + d, err := Decide(Request{Class: class, Effort: effort, Candidates: []Candidate{candidateOf(thin), candidateOf(glmFlash)}}) + if err != nil { + t.Fatal(err) + } + if got := d.Seat(Worker).Model; got != glmFlash.ID { + t.Errorf("%s %q: worker %s over %s", class, effort, got, glmFlash.ID) + } + } + } +} + +// THE SUPPORT SEATS OF A FIX STILL PAY FOR ABILITY: a model that publishes one +// middling index at a fraction of a cent does not check a fix while a model +// strong on every index costs a few tenths of a cent more. +func TestAFixIsNotCheckedByWhateverIsCheapest(t *testing.T) { + if s := prior().linkOf(Bugfix).Seats[Checker].Slope; s <= 0 { + t.Fatalf("a fix's checker pays nothing for ability (slope %.3f)", s) + } + thin := Model{ID: "inclusionai/ling-3.0-flash", Open: true, PromptPrice: 2.1e-8, CompletionPrice: 6.3e-8, CacheReadPrice: 4.2e-9, + Coding: 50.6, Context: 262144, Released: released(2026, 7, 23), Tools: true} + d, err := Decide(Request{Class: Bugfix, Candidates: []Candidate{candidateOf(thin), candidateOf(glmFlash)}}) + if err != nil { + t.Fatal(err) + } + if got := d.Seat(Checker).Model; got != glmFlash.ID { + t.Errorf("a fix's checker is %s", got) + } + // A CHECKER'S MEAN ABILITY MUST REACH THE FLOOR when any candidate's does. + tab := forDecision([]Candidate{candidateOf(thin), candidateOf(v4Flash), candidateOf(glmFlash)}, nil) + for _, m := range []Model{thin, v4Flash, glmFlash} { + a := tab.abilityOf(m) + if pick, ok := eligible(tab, Bugfix, Checker, candidateOf(m)); ok && a.U < tab.UFloor { + t.Errorf("%s may check with mean ability %.3f under the floor %.3f", pick.Model, a.U, tab.UFloor) + } + } +} + +// --BEST BUYS QUALITY IN EVERY SEAT, up to the task limit: no seat of a best +// crew is left on a model another candidate outscores there. +func TestBestBuysTheStrongestModelInEverySeat(t *testing.T) { + cands := frontierCandidates() + for _, class := range []Class{Bugfix, OpenEnded} { + d, err := Decide(Request{Class: class, Candidates: cands, Effort: EffortBest, TaskCap: 5}) + if err != nil { + t.Fatal(err) + } + tab := forDecision(cands, nil) + for _, p := range d.Crew { + for _, c := range cands { + if other, ok := eligible(tab, class, p.Seat, c); ok && other.Quality > p.Quality+1e-9 { + t.Errorf("%s best: %s is %s at %.3f while %s scores %.3f", class, p.Seat, p.Model, p.Quality, other.Model, other.Quality) + } + } + } + } +} + +// WITH ONLY CATALOG METADATA, a model whose published indexes are stronger +// wins the seat from one at the same price, context and release date. +func TestAStrongerIndexedSimilarPricedModelWinsASeat(t *testing.T) { + weak := catalogRow("acme/coder-a", true, 0.3, 1.2, 35, 60, 35) + strong := catalogRow("zeta/coder-b", true, 0.3, 1.2, 48, 76, 55) + for _, c := range []*Candidate{&weak, &strong} { + c.Model.Released = released(2026, 8, 1) + } + for _, class := range []Class{OpenEnded, Other} { + d, err := Decide(Request{Class: class, Candidates: []Candidate{weak, strong}}) + if err != nil { + t.Fatal(err) + } + for _, seat := range []Seat{Worker, Checker} { + if got := d.Seat(seat).Model; got != strong.Model.ID { + t.Errorf("%s %s went to %s, want the stronger-indexed model", class, seat, got) + } + } + } +} + +// A MODEL WITH NO USABLE METADATA is not scored, so it is neither a pick nor a +// rung — however cheap — and it still sits a seat a person pins it to. +func TestAModelWithNoUsableMetadataIsNotPickedUnlessPinned(t *testing.T) { + bare := Candidate{Model: Model{ID: "somelab/mystery", Context: 1_000_000, Tools: true}, + Routes: []Route{{Provider: "openrouter", Send: "somelab/mystery", Kind: Metered}}} + if Scorable(bare.Model) { + t.Fatal("a row with nothing but a context window was scored") + } + cands := append(catalogCandidates(), bare) + for _, class := range Classes { + for _, effort := range []Effort{EffortCheap, EffortKnee, EffortBest} { + d, err := Decide(Request{Class: class, Candidates: cands, Effort: effort}) + if err != nil { + t.Fatal(err) + } + for _, pick := range d.Crew { + if pick.Model == bare.Model.ID { + t.Errorf("%s/%q: the %s went to a model with no usable metadata", class, effort, pick.Seat) + } + for _, rung := range d.Ladder[pick.Seat] { + if rung.Model == bare.Model.ID { + t.Errorf("%s/%q: the %s's ladder holds it", class, effort, pick.Seat) + } + } + } + } + } + d, err := Decide(Request{Class: Bugfix, Candidates: cands, Pins: map[Seat]Pin{ + Worker: {Model: bare.Model.ID, Send: bare.Model.ID, Kind: Metered}, + }}) + if err != nil || d.Seat(Worker).Model != bare.Model.ID || !d.Seat(Worker).Pinned { + t.Fatalf("the pin was not honoured: %+v %v", d.Seat(Worker), err) + } +} + +// A MISSING INDEX WIDENS THE READING: a row that publishes one index is read +// with more doubt than one that publishes all three. +func TestAMissingIndexWidensTheVariance(t *testing.T) { + one := catalogRow("acme/one-index", true, 0.3, 1.2, 0, 70, 0).Model + all := catalogRow("acme/all-indexes", true, 0.3, 1.2, 45, 70, 50).Model + w := load() + if a, b := w.abilityOf(one), w.abilityOf(all); a.VarTheta <= b.VarTheta { + t.Errorf("one index read with variance %.4g, three with %.4g", a.VarTheta, b.VarTheta) + } +} + +// A ROW THAT GAINS INDEXES IS READ AGAIN: nothing about a model is remembered +// between decisions, so the next catalog refresh re-scores it — from its +// indexes now, and with less doubt. +func TestARowThatGainsIndexesIsRescored(t *testing.T) { + bare := glmFlash + bare.Intelligence, bare.Coding, bare.Agentic, bare.ArenaElo = 0, 0, 0, 0 + w := load() + before, after := w.abilityOf(bare), w.abilityOf(glmFlash) + if before.Indexed || !after.Indexed { + t.Fatalf("index path: before %v, after %v", before.Indexed, after.Indexed) + } + if after.VarTheta >= before.VarTheta { + t.Errorf("published indexes did not narrow the reading: variance %.3f then %.3f", before.VarTheta, after.VarTheta) + } + qb, _ := prior().quality(OpenEnded, Worker, bare) + qa, _ := prior().quality(OpenEnded, Worker, glmFlash) + if qa <= qb { + t.Errorf("strong published indexes scored %.3f, not above the bare row's %.3f", qa, qb) + } +} + +// A ROW WITH NO PUBLISHED INDEX IS NEVER SCORED ABOVE THE POPULATION'S MEAN, +// however new it is: its date and family widen the reading instead. +func TestANewRowWithoutIndexesIsNotScoredAboveAverage(t *testing.T) { + w := load() + pop := w.Mean[0]*w.Scale[0] + w.Loc[0] + fresh := Model{ID: "openai/gpt-9-luna-pro", PromptPrice: 1e-7, CompletionPrice: 5e-7, Context: 1050000, + Released: released(2026, 9, 22), Tools: true} + if a := w.abilityOf(fresh); a.Theta > pop+1e-9 || a.Indexed { + t.Errorf("an index-less row read at ability %.3f over the population's %.3f", a.Theta, pop) + } + if d, err := Decide(Request{Class: Bugfix, Candidates: []Candidate{candidateOf(fresh), candidateOf(glmFlash)}}); err != nil || d.Seat(Worker).Model != glmFlash.ID { + t.Errorf("worker %s over the indexed flash model (%v)", d.Seat(Worker).Model, err) + } +} + +// THIS INSTALL'S OUTCOMES MOVE A SCORE WITHOUT FREEZING IT: the learned move +// adds to the model's reading, and the reading still follows its catalog row. +func TestInstallEvidenceMovesAScoreWithoutFreezingIt(t *testing.T) { + key := LearnKey(OpenEnded, Checker, glmFlash.ID) + plain, _ := Decide(Request{Class: OpenEnded, Candidates: []Candidate{candidateOf(glmFlash)}}) + learned, _ := Decide(Request{Class: OpenEnded, Candidates: []Candidate{candidateOf(glmFlash)}, Learned: map[string]float64{key: 0.5}}) + if got := learned.Seat(Checker).Quality - plain.Seat(Checker).Quality; math.Abs(got-0.5) > 1e-9 || learned.Seat(Checker).Learned != 0.5 { + t.Fatalf("a learned +0.5 moved the checker by %.3f (recorded %.3f)", got, learned.Seat(Checker).Learned) + } + cheaper := glmFlash + cheaper.Agentic = 30 + moved, _ := Decide(Request{Class: OpenEnded, Candidates: []Candidate{candidateOf(cheaper)}, Learned: map[string]float64{key: 0.5}}) + if moved.Seat(Checker).Quality == learned.Seat(Checker).Quality { + t.Error("with a learned move in place, a changed catalog row no longer moves the score") + } +} + +// A redo lowers what a model is worth to this install and an accepted task +// raises it: the learned move changes who sits the seat. +func TestALearnedMoveCanChangeThePick(t *testing.T) { + cands := catalogCandidates() + base, _ := Decide(Request{Class: OpenEnded, Candidates: cands}) + was := base.Seat(Checker).Model + down, _ := Decide(Request{Class: OpenEnded, Candidates: cands, Learned: map[string]float64{LearnKey(OpenEnded, Checker, was): -3}}) + if down.Seat(Checker).Model == was { + t.Errorf("a checker this install marked down by 3 points still sits the seat") + } +} + +// FRONTIER MODELS ARE CANDIDATES: --best may pick them, and no crew is +// chosen whose estimate is over the task limit. +func TestBestNeverPicksACrewOverTheTaskLimit(t *testing.T) { + cands := frontierCandidates() + best, err := Decide(Request{Class: OpenEnded, Candidates: cands, Effort: EffortBest, TaskCap: 5}) + if err != nil { + t.Fatal(err) + } + frontier := false + for _, pick := range best.Crew { + if strings.HasPrefix(pick.Model, "anthropic/") || strings.HasPrefix(pick.Model, "openai/") { + frontier = true + } + } + if !frontier { + t.Errorf("--best on open-ended work picked no frontier model: %s", best.Line("", -1)) + } + for _, limit := range []float64{5, 1, 0.2} { + for _, class := range Classes { + for _, effort := range []Effort{EffortKnee, EffortBest} { + for _, factor := range []float64{0, 1, 2.5} { + d, err := Decide(Request{Class: class, Candidates: cands, Effort: effort, TaskCap: limit, CostFactor: factor}) + if err != nil { + t.Fatal(err) + } + if est := crewEst(prior(), class, d.Crew, factor); est > limit+1e-9 { + t.Errorf("%s/%q at $%v (factor %v): crew estimated $%.3f: %s", class, effort, limit, factor, est, d.Line("", -1)) + } + } + } + } + } + tight, _ := Decide(Request{Class: OpenEnded, Candidates: cands, Effort: EffortBest, TaskCap: 0.2}) + if !strings.Contains(tight.Note, "task limit") { + t.Errorf("a crew held under the limit does not say so: %q", tight.Note) + } + if est := tight.EstUSD; est > 0.2+1e-9 { + t.Errorf("held crew estimates $%.3f", est) + } +} + +func TestAPinnedSeatAlwaysRunsItsPin(t *testing.T) { + cands := catalogCandidates() + d, err := Decide(Request{Class: Bugfix, Candidates: cands, Pins: map[Seat]Pin{ + Checker: {Model: "moonshotai/kimi-k3", Send: "moonshotai/kimi-k3", Kind: Metered}, + }}) + if err != nil { + t.Fatal(err) + } + checker := d.Seat(Checker) + if !checker.Pinned || checker.Model != "moonshotai/kimi-k3" { + t.Errorf("checker %+v, want the kimi pin", checker) + } + if d.Seat(Worker).Pinned { + t.Errorf("an unpinned worker was not routed: %+v", d.Seat(Worker)) + } + // A pin the candidates do not carry still sits its seat on its own send. + d, err = Decide(Request{Class: Bugfix, Candidates: cands, Pins: map[Seat]Pin{ + Worker: {Model: "ollama/qwen3-coder", Provider: "ollama", Send: "ollama/qwen3-coder", Kind: Local}, + }}) + if err != nil { + t.Fatal(err) + } + if w := d.Seat(Worker); w.Send != "ollama/qwen3-coder" || w.CostUSD != 0 || !w.Pinned { + t.Errorf("outside pin: %+v", w) + } +} + +func TestTheCheapestRouteWinsAndAPlanCostsNothing(t *testing.T) { + cands := []Candidate{{Model: glmFlash, Routes: []Route{ + {Provider: "openrouter", Send: "openrouter/z-ai/glm-5.3-flash", Kind: Metered}, + {Provider: "z-ai", Send: "z-ai/glm-5.3-flash", Kind: Plan}, + }}} + d, err := Decide(Request{Class: Bugfix, Candidates: cands}) + if err != nil { + t.Fatal(err) + } + for _, pick := range d.Crew { + if pick.Provider != "z-ai" || pick.CostUSD != 0 { + t.Errorf("%s: %+v, want the coding plan at no marginal cost", pick.Seat, pick) + } + } + if d.EstUSD != 0 { + t.Errorf("a crew entirely on a plan estimates $%v", d.EstUSD) + } + // And a pinned provider is kept even when it is the dearer route. + d, _ = Decide(Request{Class: Bugfix, Candidates: cands, Pins: map[Seat]Pin{ + Worker: {Model: "z-ai/glm-5.3-flash", Provider: "openrouter"}, + }}) + if w := d.Seat(Worker); w.Provider != "openrouter" || w.Send != "openrouter/z-ai/glm-5.3-flash" || w.CostUSD == 0 { + t.Errorf("@openrouter pin: %+v", w) + } +} + +func TestNoCandidateIsAnErrorNamingTheSeat(t *testing.T) { + _, err := Decide(Request{Class: Bugfix}) + var missing NoCandidateError + if !errors.As(err, &missing) || missing.Seat != Worker { + t.Fatalf("err %v, want a NoCandidateError for the worker", err) + } +} + +func TestRedoStrongerEscalatesOnlyUnpinnedSeats(t *testing.T) { + cands := catalogCandidates() + first, _ := Decide(Request{Class: Bugfix, Candidates: cands}) + again, err := Decide(Request{Class: Bugfix, Candidates: cands, Stronger: &first}) + if err != nil { + t.Fatal(err) + } + if again.Quality <= first.Quality { + t.Fatalf("redo stronger: %.2f is not above %.2f", again.Quality, first.Quality) + } + pins := map[Seat]Pin{Worker: {Model: "z-ai/glm-5.3-flash"}} + first, _ = Decide(Request{Class: OpenEnded, Candidates: cands, Pins: pins, Effort: EffortCheap}) + first.Rungs, first.Note = nil, "" + again, err = Decide(Request{Class: OpenEnded, Candidates: cands, Pins: pins, Stronger: &first}) + if err != nil { + t.Fatal(err) + } + if w := again.Seat(Worker); !w.Pinned || w.Model != "z-ai/glm-5.3-flash" { + t.Errorf("redo moved a pinned worker: %+v", w) + } + // Every seat pinned: the pins stay, and the one run steps over them. + all := map[Seat]Pin{Worker: {Model: "z-ai/glm-5.3-flash"}, Planner: {Model: "z-ai/glm-5.3-flash"}, Checker: {Model: "z-ai/glm-5.3-flash"}} + first, _ = Decide(Request{Class: OpenEnded, Candidates: cands, Pins: all}) + again, err = Decide(Request{Class: OpenEnded, Candidates: cands, Pins: all, Stronger: &first}) + if err != nil { + t.Fatal(err) + } + if !again.OneOff || again.Quality <= first.Quality { + t.Errorf("all-pinned redo: one-off %v, quality %.2f over %.2f", again.OneOff, again.Quality, first.Quality) + } + // And the strongest crew there is has nowhere to go. + best, _ := Decide(Request{Class: OpenEnded, Candidates: cands, Effort: EffortBest}) + best.Crew = []Pick{ + {Seat: Worker, Model: "acme/top", Quality: 99}, {Seat: Planner, Model: "acme/top", Quality: 99}, {Seat: Checker, Model: "acme/top", Quality: 99}, + } + if _, err := Decide(Request{Class: OpenEnded, Candidates: cands, Stronger: &best}); !errors.Is(err, ErrStrongest) { + t.Errorf("redo of the strongest crew: %v, want ErrStrongest", err) + } +} + +func TestLearnedStepsStartAFixHigher(t *testing.T) { + cands := catalogCandidates() + base, _ := Decide(Request{Class: Bugfix, Candidates: cands}) + d, _ := Decide(Request{Class: Bugfix, Candidates: cands, Steps: 2}) + if d.Quality <= base.Quality { + t.Errorf("two learned steps on a fix: quality %.2f, not above the knee's %.2f", d.Quality, base.Quality) + } +} + +func TestPaceGrowsAsTheCapNears(t *testing.T) { + cases := []struct { + spent, cap, want float64 + atCap bool + }{ + {0, 0, 1, false}, + {1, 10, 1, false}, + {5, 10, 1, false}, + {7.5, 10, 2, false}, + {9, 10, 5, false}, + {10, 10, 1, true}, + {12, 10, 1, true}, + } + for _, tc := range cases { + got, at := Pace(tc.spent, tc.cap) + if math.Abs(got-tc.want) > 1e-9 || at != tc.atCap { + t.Errorf("Pace(%v, %v) = %v, %v; want %v, %v", tc.spent, tc.cap, got, at, tc.want, tc.atCap) + } + } + // Near the cap an open-ended task takes a cheaper crew. + cands := frontierCandidates() + plain, _ := Decide(Request{Class: OpenEnded, Candidates: cands}) + mult, _ := Pace(9.9, 10) + d, _ := Decide(Request{Class: OpenEnded, Candidates: cands, Pace: mult}) + if d.EstUSD >= plain.EstUSD { + t.Errorf("at 99%% of the cap the crew still estimates $%.3f against $%.3f (λ %.0f)", d.EstUSD, plain.EstUSD, d.Lambda) + } +} + +func TestGapsNameAMissingStrongChecker(t *testing.T) { + if gaps := Gaps(catalogCandidates(), nil); len(gaps) != 0 { + t.Errorf("a set with a strong checker has gaps %+v", gaps) + } + gaps := Gaps([]Candidate{candidateOf(v4Flash)}, nil) + if len(gaps) != 1 || gaps[0].Seat != Checker || gaps[0].Class != OpenEnded { + t.Errorf("v4-flash alone: gaps %+v, want the open-ended checker", gaps) + } +} + +// A PINNED CHECKER IS THE CHECKER. The gap is asked of the pin alone: a strong +// pin closes it over a weak set, a weak pin opens it over a strong one and names +// itself, and a pin on another seat changes nothing. +func TestGapsCountAPinnedChecker(t *testing.T) { + weakSet := []Candidate{candidateOf(v4Flash)} + if gaps := Gaps(weakSet, map[Seat]Model{Checker: kimiK3}); len(gaps) != 0 { + t.Errorf("a strong pinned checker over a weak set: gaps %+v, want none", gaps) + } + // The pin is read from the candidates when they carry it, and from the + // figures it came with when they do not. + if gaps := Gaps(append(weakSet, candidateOf(kimiK3)), map[Seat]Model{Checker: {ID: kimiK3.ID}}); len(gaps) != 0 { + t.Errorf("a strong pinned checker the candidates carry: gaps %+v, want none", gaps) + } + gaps := Gaps(catalogCandidates(), map[Seat]Model{Checker: v4Flash}) + want := "checker pinned to deepseek-v4-flash · open-ended work will be checked weakly" + if len(gaps) != 1 || gaps[0].Seat != Checker || gaps[0].Class != OpenEnded || gaps[0].Line != want { + t.Errorf("a weak pinned checker over a strong set: gaps %+v, want %q", gaps, want) + } + if gaps := Gaps(catalogCandidates(), map[Seat]Model{Worker: v4Flash, Planner: v4Flash}); len(gaps) != 0 { + t.Errorf("an unpinned checker over a strong set, other seats pinned weak: gaps %+v", gaps) + } + gaps = Gaps(weakSet, map[Seat]Model{Worker: kimiK3}) + if len(gaps) != 1 || !strings.HasPrefix(gaps[0].Line, "no strong checker among the models you allow") { + t.Errorf("an unpinned checker over a weak set, the worker pinned strong: gaps %+v", gaps) + } +} +func TestTheDecisionLine(t *testing.T) { + cands := catalogCandidates() + d, _ := Decide(Request{Class: OpenEnded, Candidates: cands, Pins: map[Seat]Pin{Checker: {Model: "moonshotai/kimi-k3"}}}) + got := d.Line("📌", 0.108) + want := "open-ended · worker " + ShortModel(d.Seat(Worker).Model) + " (openrouter)" + if !strings.HasPrefix(got, want) || !strings.Contains(got, " · checker 📌 kimi-k3 · $0.108 (est "+Money(d.EstUSD)+")") { + t.Errorf("line %q", got) + } + if got := d.Line("📌", -1); !strings.HasSuffix(got, " · est "+Money(d.EstUSD)) { + t.Errorf("line before the run ends: %q", got) + } +} + +// THE ROUTER'S OWN BUDGET: under two milliseconds a decision, against a +// catalog the size of the real one, classification included. +func TestADecisionTakesUnderTwoMilliseconds(t *testing.T) { + cands := frontierCandidates() + for i := 0; i < 600; i++ { + row := catalogRow("acme/m"+string(rune('a'+i%26))+strings.Repeat("x", i%7)+string(rune('a'+i/26)), i%2 == 0, + 0.1+float64(i%30)/10, 0.5+float64(i%40)/5, 20+float64(i%35), 40+float64(i%45), 20+float64(i%40)) + if i%3 == 0 { + row.Model.Intelligence, row.Model.Agentic = 0, 0 + } + row.Model.Released = released(2025, time.Month(1+i%12), 1+i%28) + cands = append(cands, row) + } + task := Task{Text: "fix: crash when the config has no trailing newline\n\nTraceback (most recent call last):\n ...\nValueError: bad"} + // THE FASTEST OF SEVERAL BATCHES is the decision's own cost: a mean over + // one batch on a shared, loaded machine also counts every time the + // scheduler took the core away, which is not the router's to answer for. + const batches, runs = 5, 40 + best := time.Duration(1<<63 - 1) + for b := 0; b < batches; b++ { + start := time.Now() + for i := 0; i < runs; i++ { + if _, err := Decide(Request{Task: task, Candidates: cands, TaskCap: 5}); err != nil { + t.Fatal(err) + } + } + if per := time.Since(start) / runs; per < best { + best = per + } + } + if best > 2*time.Millisecond { + t.Errorf("a decision took %v at best; the budget is 2ms", best) + } +} + +func TestRouteIsDeterministicWhateverTheOrder(t *testing.T) { + cands := append(frontierCandidates(), catalogRow("acme/twin-a", true, 0.15, 0.5, 41.8, 71.5, 50.9), catalogRow("acme/twin-b", true, 0.15, 0.5, 41.8, 71.5, 50.9)) + for _, class := range Classes { + a, _ := Decide(Request{Class: class, Candidates: cands}) + reversed := make([]Candidate, len(cands)) + for i := range cands { + reversed[len(cands)-1-i] = cands[i] + } + b, _ := Decide(Request{Class: class, Candidates: reversed}) + for _, seat := range Seats { + if a.Seat(seat).Model != b.Seat(seat).Model { + t.Errorf("%s %s: %s one way, %s the other", class, seat, a.Seat(seat).Model, b.Seat(seat).Model) + } + } + } +} + +// THE PORT READS THE WEIGHTS AS THEY WERE FITTED: the family key drops +// version numbers and mostly-numeric tokens. +func TestAFamilyIsTheNameWithoutItsVersion(t *testing.T) { + for id, want := range map[string]string{ + "z-ai/glm-5.3-flash": "z-ai/glm-flash", + "moonshotai/kimi-k3": "moonshotai/kimi", + "deepseek/deepseek-v4-flash": "deepseek/deepseek-flash", + "anthropic/claude-opus-5": "anthropic/claude-opus", + "qwen/qwen3-coder-30b-a3b": "qwen/qwen-coder", + } { + if got := familyOf(id); got != want { + t.Errorf("familyOf(%q) = %q, want %q", id, got, want) + } + } +} + +// ON OPEN-ENDED AND OTHER WORK A SUPPORT UPGRADE GOES TO THE CHECKER: the +// weights do not tell the planner from the checker, and the checker is the +// seat that accepts the work. At the knee the checker is never weaker than +// the planner; a pinned planner stays where it was. +func TestAtTheKneeTheCheckerIsNeverWeakerThanThePlanner(t *testing.T) { + for _, cands := range [][]Candidate{catalogCandidates(), frontierCandidates()} { + byID := map[string]Model{} + for _, c := range cands { + byID[c.Model.ID] = c.Model + } + for _, class := range []Class{OpenEnded, Other} { + d, err := Decide(Request{Class: class, Candidates: cands}) + if err != nil { + t.Fatal(err) + } + w := load() + plan, check := w.abilityOf(byID[d.Seat(Planner).Model]), w.abilityOf(byID[d.Seat(Checker).Model]) + if check.UScore < plan.UScore { + t.Errorf("%s: checker %s (%.3f) weaker than planner %s (%.3f)", class, + d.Seat(Checker).Model, check.UScore, d.Seat(Planner).Model, plan.UScore) + } + } + } + strong := glm53.ID + d, err := Decide(Request{Class: OpenEnded, Candidates: frontierCandidates(), Pins: map[Seat]Pin{ + Planner: {Model: strong, Send: strong, Kind: Metered}}}) + if err != nil || d.Seat(Planner).Model != strong { + t.Errorf("a pinned planner moved: %+v %v", d.Seat(Planner), err) + } +} + +// --CHEAP DOES NOT BUY A WORKER ON PRICE ALONE: the worker's mean ability must +// reach the floor, and a clearly stronger worker within half again the cost +// is preferred. +func TestACheapWorkerIsStillCredible(t *testing.T) { + thin := Model{ID: "inclusionai/ling-3.0-flash", Open: true, PromptPrice: 2.1e-8, CompletionPrice: 6.3e-8, CacheReadPrice: 4.2e-9, + Coding: 50.6, Context: 262144, Released: released(2026, 7, 23), Tools: true} + cands := []Candidate{candidateOf(thin), candidateOf(glmFlash), candidateOf(kimiK3)} + for _, class := range []Class{Bugfix, OpenEnded} { + d, err := Decide(Request{Class: class, Effort: EffortCheap, Candidates: cands}) + if err != nil { + t.Fatal(err) + } + if got := d.Seat(Worker).Model; got != glmFlash.ID { + t.Errorf("%s cheap: worker %s", class, got) + } + } + // Within the tolerance the stronger model wins; beyond it the cheaper one. + tab := forDecision(cands, nil) + base, _ := eligible(tab, Bugfix, Worker, candidateOf(glmFlash)) + dear := glm53 + dear.ID = "acme/near-price" + scale := 1.2 * tab.classCost(Bugfix, Worker, glmFlash) / tab.classCost(Bugfix, Worker, glm53) + dear.PromptPrice, dear.CompletionPrice, dear.CacheReadPrice = glm53.PromptPrice*scale, glm53.CompletionPrice*scale, glm53.CacheReadPrice*scale + if got := strongerWithin(tab, Bugfix, Worker, []Candidate{candidateOf(glmFlash), candidateOf(dear)}, base, cheapTolerance); got.Model != dear.ID { + t.Errorf("a stronger worker at 1.2x the cost lost to %s", got.Model) + } + far := dear + far.ID = "acme/far-price" + far.PromptPrice, far.CompletionPrice, far.CacheReadPrice = dear.PromptPrice*2, dear.CompletionPrice*2, dear.CacheReadPrice*2 + if got := strongerWithin(tab, Bugfix, Worker, []Candidate{candidateOf(glmFlash), candidateOf(far)}, base, cheapTolerance); got.Model != glmFlash.ID { + t.Errorf("a stronger worker at 2.4x the cost was bought: %s", got.Model) + } +} diff --git a/internal/crewroute/seat_test.go b/internal/crewroute/seat_test.go new file mode 100644 index 0000000000..cd228b7e48 --- /dev/null +++ b/internal/crewroute/seat_test.go @@ -0,0 +1,334 @@ +package crewroute + +import ( + "strings" + "testing" +) + +// A FREE POOL OF A MODEL NOBODY PRICES never wins a seat on its price: it is +// weighed above the cheapest priced model, so with figures no better than a +// priced model's it does not take that model's seat. +func TestAFreeUnpricedModelNeverTakesASeatOnPrice(t *testing.T) { + stranger := candidateOf(glmFlash) + stranger.Model.ID = "thinkingmachines/inkling-small" + stranger.Model.PromptPrice, stranger.Model.CompletionPrice, stranger.Model.CacheReadPrice = 0, 0, 0 + stranger.Routes = []Route{{Provider: "openrouter", Send: "thinkingmachines/inkling-small:free", Kind: Free}} + candidates := append(catalogCandidates(), stranger) + for _, class := range Classes { + d, err := Decide(Request{Class: class, Candidates: candidates}) + if err != nil { + t.Fatal(err) + } + for _, pick := range d.Crew { + if pick.Model == stranger.Model.ID { + t.Fatalf("%s %s went to the free stranger: %+v", class, pick.Seat, pick) + } + } + } +} + +// A MODEL THAT CANNOT DO THE SEAT'S WORK DOES NOT SIT IT: no tool calls, or a +// context too short for the seat. +func TestTheCapabilityFiltersKeepUnfitModelsOut(t *testing.T) { + noTools := catalogRow("somelab/no-tools", true, 0.01, 0.02, 90, 90, 90) + noTools.Model.Tools = false + short := catalogRow("somelab/short", true, 0.01, 0.02, 90, 90, 90) + short.Model.Context = 16_000 + for _, c := range []Candidate{noTools, short} { + if _, err := Decide(Request{Class: Bugfix, Candidates: []Candidate{c}}); err == nil { + t.Fatalf("%s sat a seat", c.Model.ID) + } + } +} + +// ONE MODEL, THREE ROUTES: the free pool, a direct connection, and the +// default service. With the free pool trusted, it is the route; the seat's +// fallback is the SAME model on its next route, never another model. +func TestAModelOnThreeRoutesTakesTheRightOneAndFallsThroughFreeToPaid(t *testing.T) { + m := glmFlash + routes := func(freeFail float64) []Route { + return []Route{ + {Provider: "openrouter", Send: "z-ai/glm-5.3-flash:free", Kind: Free, FailRate: freeFail}, + {Provider: "z-ai", Send: "z-ai/glm-5.3-flash", Kind: Metered}, + {Provider: "openrouter", Send: "openrouter/z-ai/glm-5.3-flash", Kind: Metered}, + } + } + one := func(freeFail float64) Decision { + d, err := Decide(Request{Class: Bugfix, Candidates: []Candidate{{Model: m, Routes: routes(freeFail)}}}) + if err != nil { + t.Fatal(err) + } + return d + } + d := one(0.05) + worker := d.Seat(Worker) + if worker.Kind != Free || worker.Send != "z-ai/glm-5.3-flash:free" { + t.Fatalf("a reliable free route was not taken: %+v", worker) + } + ladder := d.Ladder[Worker] + if len(ladder) < 2 || ladder[0].Model != worker.Model || ladder[0].Send != "z-ai/glm-5.3-flash" || ladder[1].Send != "openrouter/z-ai/glm-5.3-flash" { + t.Fatalf("the ladder is %+v, want the same model's direct then default routes first", ladder) + } + moved := d.WithRung(Worker, ladder[0], "free pool limited") + if moved.Seat(Worker).Send != "z-ai/glm-5.3-flash" || len(moved.Retried) != 1 || moved.Retried[0].Why != "free pool limited" { + t.Fatalf("moving the worker gave %+v", moved.Seat(Worker)) + } + // A free route this install has watched refuse most first calls costs more + // than it saves, and the paid route is taken from the start. + if got := one(0.95).Seat(Worker); got.Kind != Metered || got.Send != "z-ai/glm-5.3-flash" { + t.Fatalf("an unreliable free route was still taken: %+v", got) + } +} + +// A FREE ROUTE IS NEVER PRICED AT ZERO: its expected cost is its refusals. +func TestAFreeRoutesExpectedCostIsItsRefusals(t *testing.T) { + if got := routeCost(Route{Kind: Free}, 0.02, 0.02); got <= 0 { + t.Fatalf("a free route costs %v", got) + } + if routeCost(Route{Kind: Free, FailRate: 0.9}, 0.02, 0.02) <= routeCost(Route{Kind: Free, FailRate: 0.1}, 0.02, 0.02) { + t.Fatal("a route that refuses more was not dearer") + } +} + +// A MODEL THAT FAILED TO START IS KEPT OFF UNPINNED SEATS — while anything +// else can sit them — and never off a pin. +func TestAnAvoidedModelSitsNoUnpinnedSeat(t *testing.T) { + avoid := map[string]bool{Lineage("z-ai/glm-5.3-flash"): true} + d, err := Decide(Request{Class: Bugfix, Candidates: catalogCandidates(), Avoid: avoid}) + if err != nil { + t.Fatal(err) + } + for _, pick := range d.Crew { + if Lineage(pick.Model) == Lineage("z-ai/glm-5.3-flash") { + t.Fatalf("%s went to the avoided model", pick.Seat) + } + } + pinned, err := Decide(Request{Class: Bugfix, Candidates: catalogCandidates(), Avoid: avoid, + Pins: map[Seat]Pin{Worker: {Model: "z-ai/glm-5.3-flash", Send: "z-ai/glm-5.3-flash", Kind: Metered}}}) + if err != nil || pinned.Seat(Worker).Model != "z-ai/glm-5.3-flash" { + t.Fatalf("a pin was overruled: %+v %v", pinned.Seat(Worker), err) + } + // Avoiding the only model there is still makes a crew. + only := catalogCandidates()[:1] + if _, err := Decide(Request{Class: Bugfix, Candidates: only, Avoid: avoid}); err != nil { + t.Fatalf("avoiding the only model left no crew: %v", err) + } +} + +// THE LINE: open-ended with its hyphen, the planner when it is another model, +// the free route named, and a seat that moved said once. +func TestTheLineSaysClassPlannerAndARetry(t *testing.T) { + d := Decision{Class: OpenEnded, EstUSD: 0.05, Crew: []Pick{ + {Seat: Worker, Model: "z-ai/glm-5.3-flash", Provider: "openrouter", Kind: Free}, + {Seat: Planner, Model: "moonshotai/kimi-k3"}, + {Seat: Checker, Model: "moonshotai/kimi-k3"}, + }} + line := d.Line("", -1) + for _, want := range []string{"open-ended · ", "(openrouter · free)", " · planner kimi-k3 · checker kimi-k3"} { + if !strings.Contains(line, want) { + t.Errorf("line %q lacks %q", line, want) + } + } + moved := d.WithRung(Worker, Pick{Model: "deepseek/deepseek-v4-flash"}, "credit unavailable on openrouter") + if line := moved.Line("", -1); !strings.Contains(line, "running on fallback crew · worker glm-5.3-flash → deepseek-v4-flash (credit unavailable on openrouter)") { + t.Errorf("the moved line does not say so: %q", line) + } + same := Decision{Class: Bugfix, Crew: []Pick{{Seat: Worker, Model: "a/x"}, {Seat: Planner, Model: "a/x"}, {Seat: Checker, Model: "a/y"}}} + if strings.Contains(same.Line("", -1), "planner") { + t.Errorf("a planner on the worker's model was named: %q", same.Line("", -1)) + } +} + +// THE OWNER'S CATALOG: a free stranger with no price and one published index +// beside the priced models. Every class keeps it out. +func TestTheOwnersCatalogNeverSeatsTheFreeStranger(t *testing.T) { + stranger := Candidate{ + Model: Model{ID: "thinkingmachines/inkling-small", Coding: 52.9, Context: 262_144, Tools: true}, + Routes: []Route{{Provider: "router", Send: "thinkingmachines/inkling-small:free", Kind: Free}}, + } + for _, class := range Classes { + d, err := Decide(Request{Class: class, Candidates: append(catalogCandidates(), stranger)}) + if err != nil { + t.Fatal(err) + } + for _, pick := range d.Crew { + if Lineage(pick.Model) == Lineage(stranger.Model.ID) { + t.Fatalf("%s %s went to the stranger", class, pick.Seat) + } + } + } +} + +// REDO CLIMBS ONE RUNG AT A TIME: monotone, one seat, the next model on the +// seat's front — never the top of the catalog in one step. +func TestRedoClimbsTheLadderOneRungAtATime(t *testing.T) { + cands := frontierCandidates() + dear := catalogRow("somelab/very-dear", false, 30, 150, 60, 85, 70) + dear.Model.Released = released(2026, 9, 1) + cands = append(cands, dear) + ran, err := Decide(Request{Class: OpenEnded, Candidates: cands, Effort: EffortCheap}) + if err != nil { + t.Fatal(err) + } + ran.Rungs, ran.Note = nil, "" + prev := ran + for step := 0; step < 3; step++ { + next, err := Decide(Request{Class: OpenEnded, Candidates: cands, Stronger: &prev}) + if err != nil { + t.Fatal(err) + } + changed := 0 + for _, seat := range Seats { + if next.Seat(seat).Quality < prev.Seat(seat).Quality-1e-9 { + t.Fatalf("step %d: %s got weaker", step, seat) + } + if Lineage(next.Seat(seat).Model) != Lineage(prev.Seat(seat).Model) { + changed++ + } + if next.Seat(seat).Model == dear.Model.ID && step == 0 { + t.Fatalf("the first redo jumped to the dearest model in the catalog on %s", seat) + } + } + if changed != 1 || len(next.Rungs) != 1 || !strings.Contains(next.Line("", -1), "→") { + t.Fatalf("step %d moved %d seats: %q", step, changed, next.Line("", -1)) + } + prev = next + } +} + +// A REDO OF A TASK THAT NEVER STARTED is asked again on the next-best models at +// the same price of a point — not escalated, because nothing ran. +func TestARedoOfATaskThatNeverStartedTakesTheNextBestAtTheSameCost(t *testing.T) { + cands := catalogCandidates() + ran, _ := Decide(Request{Class: Bugfix, Candidates: cands}) + again, err := Decide(Request{Class: Bugfix, Candidates: cands, Again: &ran}) + if err != nil { + t.Fatal(err) + } + if again.Lambda != ran.Lambda { + t.Fatalf("the retry moved λ from %v to %v", ran.Lambda, again.Lambda) + } + for _, seat := range Seats { + if Lineage(again.Seat(seat).Model) == Lineage(ran.Seat(seat).Model) { + t.Errorf("%s is the model that never started (%s)", seat, again.Seat(seat).Model) + } + } +} + +// AN EFFORT WORD SAYS WHAT IT CHANGED, OR THAT IT CHANGED NOTHING. +func TestAnEffortWordSaysWhatItChanged(t *testing.T) { + cands := catalogCandidates() + best, _ := Decide(Request{Class: Bugfix, Candidates: cands, Effort: EffortBest}) + if len(best.Rungs) == 0 || best.Note != "" { + t.Fatalf("--best on a fix changed the crew and said %v / %q", best.Rungs, best.Note) + } + only := []Candidate{candidateOf(kimiK3)} // nothing stronger to be had + top, _ := Decide(Request{Class: Bugfix, Candidates: only, Effort: EffortBest}) + if top.Note != "best · already the strongest crew allowed" || !strings.Contains(top.Line("", -1), top.Note) { + t.Fatalf("--best with nothing stronger said %q", top.Line("", -1)) + } +} + +// A LEARNED OFFSET IS RUNGS: each step is one rung from the knee's crew, and the +// crew never skips to the top. +func TestALearnedOffsetIsRungsNotAJump(t *testing.T) { + dear := catalogRow("somelab/very-dear", false, 30, 150, 60, 85, 70) + dear.Model.Released = released(2026, 9, 1) + cands := append(frontierCandidates(), dear) + base, _ := Decide(Request{Class: Bugfix, Candidates: cands}) + one, _ := Decide(Request{Class: Bugfix, Candidates: cands, Steps: 1}) + moved := 0 + for _, seat := range Seats { + if Lineage(one.Seat(seat).Model) != Lineage(base.Seat(seat).Model) { + moved++ + } + if one.Seat(seat).Model == "somelab/very-dear" { + t.Fatalf("one learned step jumped to the dearest model on %s", seat) + } + } + if moved != 1 { + t.Fatalf("one learned step moved %d seats", moved) + } +} + +// A LINE LEADS WITH ITS STATE: a crew running on its fallback says so before +// the class and the seats, and a task that stopped before spending names no +// money — never a $0.000 that reads as a free success. +func TestTheLineLeadsWithTheFallbackAndNamesNoUnspentMoney(t *testing.T) { + d := Decision{Class: Bugfix, EstUSD: 0.01, Crew: []Pick{ + {Seat: Worker, Model: "z-ai/glm-5.3-flash"}, {Seat: Planner, Model: "z-ai/glm-5.3-flash"}, {Seat: Checker, Model: "moonshotai/kimi-k3"}, + }} + moved := d.WithRung(Worker, Pick{Model: "deepseek/deepseek-v4-flash"}, "credit unavailable on openrouter") + line := moved.Line("", Unspent) + if !strings.HasPrefix(line, "running on fallback crew · worker glm-5.3-flash → deepseek-v4-flash") { + t.Errorf("the fallback is not first: %q", line) + } + if strings.Contains(line, "$") { + t.Errorf("an unspent line names money: %q", line) + } +} + +// A REDO'S LINE SAYS EACH SEAT ONCE, WITH ITS CHANGE: a planner that moved is +// `planner glm-5.3-flash → kimi-k3` where the planner stands, not the +// planner and then its rung again. +func TestARedoLineSaysEachSeatOnce(t *testing.T) { + d := Decision{Class: Bugfix, Crew: []Pick{ + {Seat: Worker, Model: "z-ai/glm-5.3-flash"}, {Seat: Planner, Model: "moonshotai/kimi-k3"}, {Seat: Checker, Model: "moonshotai/kimi-k3"}, + }, Rungs: []Retry{{Seat: Planner, From: "z-ai/glm-5.3-flash", To: "moonshotai/kimi-k3"}}} + line := d.Line("", -1) + if strings.Count(line, "planner") != 1 || !strings.Contains(line, "planner glm-5.3-flash → kimi-k3") { + t.Errorf("the redo line reads %q", line) + } +} + +// A LONG LADDER IS ONE MOVE ON THE LINE: each seat says where it started, +// where it is, the first reason and how many it tried between, and the line +// stays within a card however many rungs were walked. +func TestALongLadderIsSaidAsItsNetMove(t *testing.T) { + d := Decision{Class: Bugfix, EstUSD: 0.02, Crew: []Pick{ + {Seat: Worker, Model: "z-ai/glm-5.3-flash"}, {Seat: Planner, Model: "z-ai/glm-5.3-flash"}, {Seat: Checker, Model: "moonshotai/kimi-k3"}, + }} + for _, seat := range []Seat{Worker, Planner} { + why := "credit unavailable on openrouter" + for _, m := range []string{"poolside/laguna-s-2.1", "poolside/laguna-xs-2.1", "nvidia/nemotron-3-super-120b-a12b", "google/gemma-4-31b-it"} { + d = d.WithRung(seat, Pick{Model: m, Kind: Free}, why) + why = "limit reached on openrouter" + } + } + line := d.Line("", Unspent) + if !strings.Contains(line, "worker glm-5.3-flash → gemma-4-31b-it (credit unavailable on openrouter; +3 tried)") || strings.Contains(line, "laguna") { + t.Errorf("the ladder is not said as its net move: %q", line) + } + if n := len([]rune(line)); n > 260 { + t.Errorf("the line is %d characters after a long ladder: %q", n, line) + } +} + +// A MODEL THAT PUBLISHES NOTHING BUT A PRICE IS NOT RANKED ON IT: a row with +// no index, no date and a near-zero price is neither a pick nor a rung, and a +// seat's last-rung rescue may still take it. +func TestAModelWithNoEvidenceIsNotPickedOnPrice(t *testing.T) { + bare := Candidate{Model: Model{ID: "upstage/solar-mini4", PromptPrice: 1e-9, CompletionPrice: 1e-9, Tools: true}, + Routes: []Route{{Provider: "openrouter", Send: "upstage/solar-mini4", Kind: Metered}}} + cands := append(catalogCandidates(), bare) + for _, effort := range []Effort{"", EffortCheap} { + d, err := Decide(Request{Class: Bugfix, Candidates: cands, Effort: effort}) + if err != nil { + t.Fatal(err) + } + for _, pick := range d.Crew { + if pick.Model == bare.Model.ID { + t.Errorf("effort %q: the %s is a model with no evidence", effort, pick.Seat) + } + for _, rung := range d.Ladder[pick.Seat] { + if rung.Model == bare.Model.ID { + t.Errorf("effort %q: the %s's ladder holds a model with no evidence", effort, pick.Seat) + } + } + } + } + rescue, err := Decide(Request{Class: Bugfix, Candidates: []Candidate{bare}, Rescue: true}) + if err != nil || rescue.Seat(Planner).Model != bare.Model.ID { + t.Errorf("the rescue would not take it: %v %+v", err, rescue.Crew) + } +} diff --git a/internal/crewroute/testdata/hermes-7680.md b/internal/crewroute/testdata/hermes-7680.md new file mode 100644 index 0000000000..2131593a3f --- /dev/null +++ b/internal/crewroute/testdata/hermes-7680.md @@ -0,0 +1,124 @@ +# Critical: Hermes doesn't validate finish_reason for tool_calls, allowing truncated JSON to be processed + +## 🐛 Bug Report + +### Summary +Hermes does not validate `finish_reason` when `tool_calls` are present, allowing truncated/incomplete JSON in tool call arguments to be processed. This causes cascading errors including HTTP 500 responses and wasted retry attempts. + +### Impact +- **Severity:** Critical +- **Affects:** All users using models with output token limits +- **Symptoms:** HTTP 500 errors, invalid JSON in tool calls, session failures + +### Root Cause + +When a model returns `finish_reason="length"` with `tool_calls` present, the current code only checks for truncation when tool_calls are **absent** (`run_agent.py:7803`): + +```python +if finish_reason == "length": + if self.api_mode == "chat_completions": + assistant_message = response.choices[0].message + if not assistant_message.tool_calls: # ← Only handles missing tool_calls + # continuation retry logic +``` + +**Problem:** When `tool_calls` exist, the code skips the length check entirely and proceeds to process potentially incomplete JSON arguments. + +### Reproduction + +1. Use a model with limited output tokens (e.g., `max_tokens=4096`) +2. Request a task that generates a large tool call (e.g., `write_file` with substantial content) +3. Model hits token limit mid-generation +4. Returns `finish_reason="length"` (or router rewrites to `"tool_calls"`) with incomplete JSON: + ```json + {"path": "/tmp/file.md", "content": "very long text... + ``` +5. Hermes processes the broken JSON → validation fails → 3 retries → tool error → HTTP 500 + +### Example from Real Session + +**Session:** `20260411_121436_65708048` + +**Truncated tool call:** +```json +{"path": "/tmp/ai_gateway_comparison_report.md" +``` +(Missing closing brace and `content` parameter) + +**Result:** +- 3 retry attempts (all failed) +- HTTP 500 from router +- Session terminated + +**Context stats:** +- 48K tokens (24% of 200K limit) - context was healthy +- 45 tool calls (44 successful, 1 truncated) +- No compression needed + +### Proposed Fix + +Add validation for truncated tool_calls when `finish_reason="length"`: + +```python +if finish_reason == "length": + if self.api_mode == "chat_completions": + assistant_message = response.choices[0].message + + # NEW: Validate tool_calls JSON completeness + if assistant_message.tool_calls: + incomplete_tools = [] + for tc in assistant_message.tool_calls: + args = tc.function.arguments or "" + if not args.strip(): + continue + try: + json.loads(args) + except json.JSONDecodeError as e: + incomplete_tools.append((tc.function.name, str(e))) + + if incomplete_tools: + tool_name, error = incomplete_tools[0] + # Return user-friendly error instead of processing broken JSON + return { + "final_response": "⚠️ Tool call truncated due to length limit", + "error": f"Tool call truncated: {tool_name} - {error}", + "partial": True + } + + # Existing logic for missing tool_calls + if not assistant_message.tool_calls: + # continuation retry +``` + +### Benefits + +1. **Prevents cascading errors** - catches truncation early +2. **Saves API calls** - no wasted retries on unfixable truncation +3. **Better UX** - clear error message with actionable suggestions +4. **Preserves context** - doesn't pollute message history with broken JSON + +### Testing + +Tested fix locally - successfully catches truncated tool calls and returns helpful error instead of HTTP 500. + +### Files Affected + +- `run_agent.py` (lines 7801-7803) + +### Related Code + +The existing invalid JSON retry logic (lines 8845-8883) doesn't distinguish between: +- Recoverable JSON errors (model mistake) → retry makes sense +- Truncation errors (length limit) → retry is pointless + +This fix addresses the root cause before broken JSON enters the retry loop. + +--- + +**Environment:** +- Hermes version: latest (2026-04-11) +- Model: kr/claude-sonnet-4.5 +- Router: router.neomentor.tech +- Platform: Matrix + +**Analysis:** Full detailed analysis available in session logs. diff --git a/internal/crewroute/testdata/kornia-4814.md b/internal/crewroute/testdata/kornia-4814.md new file mode 100644 index 0000000000..4805395dc7 --- /dev/null +++ b/internal/crewroute/testdata/kornia-4814.md @@ -0,0 +1,46 @@ +# augmentation: RandomResizedCrop(scale=(1, 1)) never keeps a square or portrait image: strict candidate test and an inverted-ratio fallback + +`RandomResizedCrop` keeps the whole image under `scale=(1.0, 1.0)` only for landscape inputs. For a square or portrait input it returns a smaller crop that falls outside both `scale` and `ratio`. torchvision's reference `RandomResizedCrop.get_params` keeps the whole image in these cases. + +Two lines in `kornia/augmentation/random_generator/_2d/crop.py` (on `main` 57cd68e6f) combine to cause this: + +1. **The candidate test is strict** (line 268): `(w < size[1]) * (h < size[0])`. torchvision accepts `0 < w <= width and 0 < h <= height`. With `scale=(1.0, 1.0)` no candidate can be strictly smaller on both axes, so every call falls back. +2. **The fallback inverts the ratio** (lines 283-290). kornia's `ratio` is width/height (`w = sqrt(area * ratio)`), but the fallback computes `in_ratio = H / W`. It compares that against `min(ratio)` in both branches and never against `max(ratio)`. torchvision compares `W / H` against `min(ratio)` and `max(ratio)` and keeps the whole image when the input ratio is already in range. + +## Repro + +```python +import torch +import kornia.augmentation as K + +def crop_hw(shape): + torch.manual_seed(0) + src = K.RandomResizedCrop((4, 4), scale=(1.0, 1.0), p=1.0).forward_parameters((1, 1, *shape))["src"][0] + return int(src[2, 1] - src[1, 1]) + 1, int(src[1, 0] - src[0, 0]) + 1 + +for shape in [(8, 6), (8, 8), (32, 32), (100, 60), (60, 100)]: + print(shape, crop_hw(shape)) +``` + +Compared with torchvision 0.29.0's `RandomResizedCrop.get_params` run on the same shapes (the function taken verbatim from the wheel): + +| input H x W | kornia crop | area | w/h | torchvision crop | +|---|---|---|---|---| +| 8 x 6 | 4 x 6 | 0.50 | 1.50 | 8 x 6 | +| 8 x 8 | 6 x 8 | 0.75 | 1.33 | 8 x 8 | +| 32 x 32 | 24 x 32 | 0.75 | 1.33 | 32 x 32 | +| 100 x 60 | 45 x 60 | 0.45 | 1.33 | 80 x 60 | +| 60 x 100 | 60 x 80 | 0.80 | 1.33 | 60 x 80 | + +In the 8 x 6 row the crop's w/h of 1.5 is outside the default `ratio=(3/4, 4/3)`. It also has half the area that `scale` requests. Landscape inputs agree with torchvision only because the default `ratio` is symmetric (`min = 1 / max`). + +## Expected + +Follow torchvision's rule: a candidate may equal the input size, and the fallback compares `W / H` with both `min(ratio)` and `max(ratio)`, keeping the whole image when it is in range. Each change alone flips the current pin. + +## Where it is documented + +The `RandomResizedCrop` Convention block and `test_convention_random_resized_crop_fallback_can_escape_scale_and_ratio` currently describe this as intended behaviour. #4791 reframes both as a wart linked to this issue. The fix PR inverts the pin and drops the wart sentence. + +Posted on behalf of @ducha-aiki by Claude (Opus 5.5). + diff --git a/internal/e2e/manual_e2e_test.go b/internal/e2e/manual_e2e_test.go index b9394afd0f..fccbea87ab 100644 --- a/internal/e2e/manual_e2e_test.go +++ b/internal/e2e/manual_e2e_test.go @@ -52,7 +52,7 @@ import ( "testing" "time" - "github.com/Agent-Field/codeaf/internal/config" + "github.com/Agent-Field/codeaf/internal/crewroute" "github.com/Agent-Field/codeaf/internal/exec/bare" "github.com/Agent-Field/codeaf/internal/manual" "github.com/Agent-Field/codeaf/internal/session" @@ -311,17 +311,19 @@ func mentionsAny(text string, words ...string) bool { // TestManualQuotesTheCrewsRealSize is #293 §3, which is the failure class this // whole lane exists for: a page that has stopped being true reads exactly like // one that is, and the person asked precisely because they could not check. The -// count is derived from [config.ModelTiers] and cross-checked against every -// preset's own row, so the day a seat is added or removed this test moves with -// the code and the manual is what turns red. +// count is derived from [crewroute.Seats], the router's own list of the seats a +// task runs on, so the day a seat is added or removed this test moves with the +// code and the manual is what turns red. func TestManualQuotesTheCrewsRealSize(t *testing.T) { - seats := crewSeatCount(t) - t.Logf("internal/config owns the figure: %d seats (%v), the same count in all %d presets", - seats, config.ModelTiers, len(config.CrewPresets)) + seats := len(crewroute.Seats) + if seats == 0 { + t.Fatalf("crewroute.Seats is empty") + } + t.Logf("internal/crewroute owns the figure: %d seats (%v)", seats, crewroute.Seats) w := newManualWorld(t) runManualScenario(t, w, "how many models the crew is", - "how many models does codeaf run on its own behalf?", + "how many models is a task's crew?", func(p *probe, out turn, calls []manualCall) { if len(calls) == 0 { p.missf("the model answered a question about codeaf out of memory: it called %v and never opened the manual", out.names()) @@ -329,11 +331,11 @@ func TestManualQuotesTheCrewsRealSize(t *testing.T) { } // NAMING THE SEATS IS SAYING HOW MANY OF THEM THERE ARE, and it is // the more checkable of the two: #293 §3's own worked example is two - // pages that listed four seat names and omitted the one that pays + // pages that listed some seat names and omitted the one that pays // most of a task's bill, which a count alone would not have caught. - // The words are the settings registry's, not this file's. - if named := seatsNamedIn(out.Reply, crewSeatLabels(t)); len(named) == seats { - t.Logf(" the reply names all %d crew seats the registry owns: %v", seats, named) + // The words are the router's, not this file's. + if named := seatsNamedIn(out.Reply, crewSeatLabels()); len(named) == seats { + t.Logf(" the reply names all %d crew seats the router owns: %v", seats, named) return } said := countsClaimedAbout(out.Reply, crewNouns) @@ -350,64 +352,28 @@ func TestManualQuotesTheCrewsRealSize(t *testing.T) { seats, said, out.Reply, indent(manualResults(calls), " ")) return } - t.Logf(" the reply says %d, which is what internal/config says", seats) + t.Logf(" the reply says %d, which is what internal/crewroute says", seats) }) } // crewNouns are the words a claim about the crew's size lands on. A number that // is not next to one of these is a number about something else — how many -// presets there are, how many pages the manual has — and grading it would be +// rows the settings sheet has, how many pages the manual has — and grading it would be // grading a sentence nobody asked about. var crewNouns = []string{"model", "models", "seat", "seats", "tier", "tiers", "crew"} -// crewSeatCount is the figure the CODE owns, read the way the settings sheet -// reads it and checked against the table underneath, so a count that only one -// of the two agreed with could not pass as the answer. -func crewSeatCount(t *testing.T) int { - t.Helper() - seats := len(config.ModelTiers) - if seats == 0 { - t.Fatalf("config.ModelTiers is empty") - } - for _, preset := range config.CrewPresets { - models, found := config.CrewModels(preset) - if !found { - t.Fatalf("config.CrewModels(%q) answers nothing", preset) - } - if len(models) != seats { - t.Fatalf("the %s preset sets %d models where there are %d tiers; the count this test grades is not one number", - preset, len(models), seats) - } - } - return seats -} - -// crewSeatLabels is the word each crew seat is KNOWN BY, read off the settings -// registry rows that front the five tiers — "reflex", "small work", "worker", -// "careful work", "mastermind". They are the registry's because a seat's name is -// the registry's to change, and a test that spelled them here would be the -// second copy that #293 §3 is about. -func crewSeatLabels(t *testing.T) []string { - t.Helper() - registry := config.NewSettings(config.SettingsOptions{}) - labels := make([]string, 0, len(config.ModelTiers)) - for _, tier := range config.ModelTiers { - row, found := registry.Row(tierRowKey(tier)) - if !found || strings.TrimSpace(row.Label) == "" { - t.Fatalf("the settings registry has no labelled row for the %s tier", tier) - } - labels = append(labels, row.Label) +// crewSeatLabels is the word each crew seat is KNOWN BY — "worker", "planner", +// "checker" — read off the router's own list. They are the router's because a +// seat's name is the router's to change, and a test that spelled them here +// would be the second copy that #293 §3 is about. +func crewSeatLabels() []string { + labels := make([]string, 0, len(crewroute.Seats)) + for _, seat := range crewroute.Seats { + labels = append(labels, string(seat)) } return labels } -// tierRowKey is the settings key one tier's model row answers to. The prefix is -// taken off a key the package already exports rather than typed again, so the -// day the rows are renamed this moves with them. -func tierRowKey(tier string) string { - return strings.TrimSuffix(config.KeyTierReflexModel, config.ModelTierReflex) + tier -} - // seatsNamedIn is which of those words a reply carries. func seatsNamedIn(reply string, labels []string) []string { low := strings.ToLower(reply) diff --git a/internal/e2e/tui_e2e_test.go b/internal/e2e/tui_e2e_test.go index b863096ef4..b55d0bd148 100644 --- a/internal/e2e/tui_e2e_test.go +++ b/internal/e2e/tui_e2e_test.go @@ -109,7 +109,6 @@ func TestTUIE2E(t *testing.T) { t.Run("plain_launch_opens_connections_and_harnesses", testPlainLaunchConnectionsAndHarnesses) t.Run("a_nested_landing_asks_and_a_key_answers_it", testNestedGate) t.Run("a_refused_landing_is_incomplete", testRefusedLanding) - t.Run("a_crew_older_than_the_work_seat_says_so_once", testInheritedWorkSeat) t.Run("a_fresh_install_is_shown_the_setup", testFreshInstallSetup) t.Run("a_refused_task_proposal_draws_no_schema_sentence", testRefusedTaskProposal) t.Run("space_in_the_task_room_pages_the_card", testTaskRoomKeepsSpace) @@ -176,6 +175,22 @@ func testPlainLaunchConnectionsAndHarnesses(t *testing.T) { t.Fatalf("the plain launch lost this machine's harness registry:\n%s", harnesses) } t.Logf("the local engine road opened this machine's harness registry:\n%s", harnesses) + + // AND THE CREW PANEL, which is the same kind of door onto this machine's + // profile: /crew opens it framed over the conversation, enter on the first + // seat opens that seat's list on `auto`, and esc steps back out one level at + // a time. Nothing is chosen, so nothing is written and no model is asked. + r.keys("Escape") + r.lit("/crew") + r.keys("Enter") + crew := r.waitFor(20*time.Second, say(t, "crewMainKeys")) + t.Logf("the crew panel opened:\n%s", crew) + r.keys("Enter") + seats := r.waitFor(20*time.Second, say(t, "crewAutoWord")) + t.Logf("the worker's seat list opened on auto:\n%s", seats) + r.keys("Escape") + r.waitFor(20*time.Second, say(t, "crewMainKeys")) + r.keys("Escape") r.quit() } @@ -1736,117 +1751,6 @@ func moneyIn(t *testing.T, screen, needle string, after bool) string { // ── 12 ────────────────────────────────────────────────────────────────────── -// testInheritedWorkSeat is #312's acceptance on the real screen: a crew older -// than the work seat, met where a person actually meets it. -// -// THE PROFILE IS THE DEFECT. A crew applied before the worker class existed -// (#278) holds four `models.tiers.*` rows and no `worker` among them. Headless -// doors learned to read that shape in #311; the conversation did not, so every -// task started from a thread ran on the build's own worker model and nothing -// anywhere said which model that was or why. This subtest builds exactly that -// profile — four real rows, the fifth key deleted — starts one small task, and -// reads back two things a unit test cannot: that the LINE is on the screen once, -// and that the model the node actually called is the row the person pinned. -// -// THE MODEL IS READ OUT OF THE CALL LOG, which is always on and writes one line -// per model call with the tag the caller set (internal/calllog, and session's -// loop.go tags a node's calls `task`). That is the node's own journal, and it is -// the only evidence in this suite that comes off the wire rather than off the -// screen. -// -// IT IS DELIBERATELY THE CHEAPEST SHAPE THERE IS: `/task solo`, which runs one -// worker and makes no sizing call before it, on a brief that is one file. -func testInheritedWorkSeat(t *testing.T) { - // The row the work must land on. It is a DIFFERENT id from the model the - // conversation talks on (newHome pins that) and from this build's own worker - // default, because the whole question is which of the three answered. - const smallWork = "deepseek/deepseek-v4-flash-0731" - home := newHome(t, map[string]any{ - "models.tiers.reflex": "mistralai/mistral-nemo", - "models.tiers.low": smallWork, - "models.tiers.high": smallWork, - "models.tiers.mastermind": smallWork, - }) - dropWorkerRow(t, home) - ws := newWorkspace(t, "seatws", false) - r := start(t, "afe2e_seat", home, ws, tuiWide, 40) - - // Whichever door the launch took — home on a machine with several - // conversations, and straight into a greeted conversation on a fresh one, - // which is what a state root built one minute ago always is. THE GREETED - // CONVERSATION HAS TWO SHAPES and this waits for both: the starter line under - // the wordmark, and the starting POINTS a conversation nobody has typed in - // yet stands on, whose foot is [welcomeStarterKeysWord] — the screen that - // entry's own `why` warns a subtest about waiting past. - r.waitForAny(20*time.Second, say(t, "placeRestWord"), say(t, "starterTaskWord"), - say(t, "welcomeStarterKeysWord")) - r.keys("Escape") - r.lit("/task solo write a file called hello.txt containing the word hello") - r.keys("Enter") - - // THE LINE, WHEN THE WORK STARTS. Both halves of it: the observation about - // the profile and the promise about what ends it. - screen := r.waitFor(4*time.Minute, say(t, "inheritedSeatObservation"), say(t, "inheritedSeatPromise")) - t.Logf("the conversation says which row filled its work seat:\n%s", screen) - - // AND ONCE. A node divides into parts and each part starts; a line that - // arrived with each of them is the noise this mechanism refused headless. - if got := strings.Count(screen, say(t, "inheritedSeatObservation")); got != 1 { - t.Errorf("the line is on the screen %d times, want once:\n%s", got, screen) - } - - // AND THE WORK IS ON THE ROW THE PERSON PINNED. The node's calls carry the - // `task` tag, and no call anywhere may have gone to the build's own worker. - // The log is POLLED rather than read once: the receipt is written when the - // node starts and the node's first call goes out a moment later, and this - // subtest deliberately stops as soon as there is something to read rather - // than paying for the whole piece of work. - models := waitForTaskCalls(t, home, 3*time.Minute) - if len(models) == 0 { - t.Fatalf("no call in the log was tagged as a task's; the log held %v", callModels(t, home)) - } - t.Logf("the node called: %v", models) - for _, model := range models { - if model != smallWork { - t.Errorf("a task call went to %q, want the small-work row %q the crew pinned", model, smallWork) - } - } - for _, model := range callModels(t, home) { - if model == "z-ai/glm-5.3-flash" { - t.Errorf("a call went to this build's own worker model, which is the substitution the issue is about") - } - } - r.quit() -} - -// dropWorkerRow deletes `models.tiers.worker` from a rig's config, so the -// profile is the shape a crew set before that class existed actually has. -// -// It is a DELETE and not an empty string: the two are different answers -// everywhere in this build — a row emptied on purpose means "follow the -// conversation" — and it is the one this suite must write, because [newHome] -// copies the person's own config and theirs may hold the key. -func dropWorkerRow(t *testing.T, home string) { - t.Helper() - path := filepath.Join(home, "config.json") - raw, err := os.ReadFile(path) - if err != nil { - t.Fatalf("config: %v", err) - } - rows := map[string]any{} - if err := json.Unmarshal(raw, &rows); err != nil { - t.Fatalf("config: %v", err) - } - delete(rows, "models.tiers.worker") - out, err := json.MarshalIndent(rows, "", " ") - if err != nil { - t.Fatalf("config: %v", err) - } - if err := os.WriteFile(path, out, 0o600); err != nil { - t.Fatalf("config: %v", err) - } -} - // waitForTaskCalls polls the call log until a node's own call is in it, and // answers every model those calls asked for. func waitForTaskCalls(t *testing.T, home string, within time.Duration) []string { @@ -1931,7 +1835,7 @@ func callLogModels(t *testing.T, home, tag string) []string { // that ran it. // // IT IS DELIBERATELY THE CHEAPEST TASK THERE IS — `/task solo` on a one-file -// brief, the shape [testInheritedWorkSeat] already pays for. The measured run +// brief, the cheapest shape a task comes in. The measured run // lands in about thirty seconds and costs five cents. // // AND IT RUNS SHORT ON PURPOSE ([tuiShortRows]): a record that fits on the diff --git a/internal/e2e/tuiwords_test.go b/internal/e2e/tuiwords_test.go index 473738ae4e..3bc903b619 100644 --- a/internal/e2e/tuiwords_test.go +++ b/internal/e2e/tuiwords_test.go @@ -121,6 +121,14 @@ var tuiWords = map[string]tuiWord{ screen: "harnesses are unavailable here", why: "the refusal that must be absent from an ordinary launch and remain available to the hosted seam", }, + "crewMainKeys": { + screen: "enter change · esc close · ? keys", + why: "/crew opened its panel, framed, with its keys in the bottom edge", + }, + "crewAutoWord": { + screen: "auto — codeaf picks per task", + why: "enter on a seat opened the seat list on its first row, auto", + }, // ── the bounded stop ───────────────────────────────────────────────────── "stoppingWord": { @@ -705,26 +713,6 @@ var tuiWords = map[string]tuiWord{ "planFigures). It is the same `N steps` and `$` the row already spends in its two columns", }, - // ── the seat a crew older than it never wrote ──────────────────────────── - // - // The two halves of one line, and it is the ENGINE'S sentence: the surface - // says it in the thread and every headless door prints it under the models - // line, out of one composer (internal/config's Seat.Notice), so the gate - // looks for it where it is spelled rather than in the surface that relays it. - "inheritedSeatObservation": { - screen: "your crew was set before the work seat existed", - source: "your crew was set before the ", - pkg: "internal/config", - why: "the observation half: a profile older than the seat is told so, once, when work starts on it", - }, - "inheritedSeatPromise": { - screen: "it is running on your small work seat's model", - source: "it is running on your ", - pkg: "internal/config", - why: "the promise half, naming the row the work is actually on. It stops at the row rather " + - "than at `until you pick a crew again` because a transcript line is cut to make room for " + - "the task rail, and the sentence is longer than an ordinary window minus that column", - }, "taskLookWord": { screen: "your call", pkg: "internal/session", diff --git a/internal/manual/chat/adaptive-runs.md b/internal/manual/chat/adaptive-runs.md index 142e9dc8d8..d0f0d854bb 100644 --- a/internal/manual/chat/adaptive-runs.md +++ b/internal/manual/chat/adaptive-runs.md @@ -139,7 +139,7 @@ None of this happens without a folder to weigh in. A run with no `-w`, or a goal no file that exists, is not treated as small — it is treated as unmeasured, and every one of these passes behaves exactly as it did before any of it was read. -## Running one task without the screen — codeaf do, headless, from a script: what flags it takes, what it prints, and what its exit code means +## Running one task without the screen — codeaf do, headless, from a script: what flags it takes, what it prints, what its last line and its exit code mean ``` codeaf do "" @@ -158,9 +158,10 @@ keyboard decides for itself and says on the record that it decided. | `--json` | print one machine-readable object instead of the deliverable | | `--yes-spend` | spend past today's limit and the plan price without stopping | | `--slots ` | how many workers may run at once for this run; `0` is no limit. Unset, it is your `task.parallel` setting, which is no limit out of the box | -| `--model ` | the work model for this run | -| `--plan-model ` | the model that plans, when it should differ from the work model | -| `--check-model ` | the model that checks finished work; then `CODEAF_CHECK_MODEL`, then a plan seat pinned by flag or environment, then the crew's careful row | +| `--model ` | the worker for this run | +| `--plan-model ` | the planner for this run | +| `--check-model ` | the checker for this run; never the planner | +| `--best` / `--cheap` / `--pin` | the crew for this one task — see *running from the terminal* | | `--context-fill ` | how full a model's context window may get before it is compacted | | `--completion-reserve ` | tokens every call keeps free for its answer and its reasoning | @@ -254,7 +255,7 @@ working. `exec` plans nothing, so naming it changes nothing, and it says so once error stream. Without `--json`, standard output is the worker's own text and nothing else. It opens on the -error stream naming one model rather than two — `models: work (crew frugal)` — +error stream naming one model rather than three — `models: worker (routed)` — because only one of them runs anything. ## What one costs and what happens when the money runs out @@ -360,14 +361,14 @@ nobody made a finding about it. The row carries the node's own spend, and a land row carries its digest. **Each node row names the model it runs on, and it is not the planner's.** A run is -deliberately two classes of model: the root row carries the **mastermind** class that cuts -the goal, and every node row under it carries the **worker** class that does it — one -careful call deciding what happens, many cheaper ones doing it. It is the same class a task -handed off in conversation runs on. So opening a node and reading +deliberately two seats: the root row carries the **planner** that cuts the goal, and every +node row under it carries the **worker** that does it — one careful call deciding what +happens, many cheaper ones doing it. They are the run's crew, picked for it the way a task +handed off in conversation has its crew picked. So opening a node and reading `task ` at the foot of the frame is how you see your crew actually working; the run's own row above it will be naming something else, and that is the arrangement rather than a disagreement. The id is settled once, when the run starts, so a `/model` half way through -does not move it. A run whose classes resolved nothing says nothing rather than guessing. +does not move it. A run whose seats resolved nothing says nothing rather than guessing. ## What the rows under a run are called — sub task names that were just the prompt, and workers all named "You are a" diff --git a/internal/manual/chat/commands.md b/internal/manual/chat/commands.md index 8e2a43fbc8..9e01401300 100644 --- a/internal/manual/chat/commands.md +++ b/internal/manual/chat/commands.md @@ -184,11 +184,18 @@ Canonical word, the other words it answers to, its argument form, and what it do | `/memories` | — | — | prints every memory into the conversation | | `/remember` | — | `` | keeps one thing across conversations | | `/forget` | — | `` | forgets the best matching memory | -| `/crew` | — | — | opens the six-seat reading: the model you talk to, then the three crew presets | -| `/crew` | — | `` | sets the crew to `frugal`, `balanced` or `max` | +| `/crew` | — | — | opens the crew panel: the three seats, the allowed models, the providers, the per-task limit and the daily cap, changed in place | +| `/crew` | — | `pin ` | pins the worker, planner or checker to a model; `/model` stays | +| `/crew` | — | `unpin ` | puts a seat back on auto | +| `/crew` | — | `models ` | which models a seat may be picked from — `all`, `open`, `≤in/out`, ids | +| `/crew` | — | `cap ` | the most tasks' crews may spend in a day | +| `/crew` | — | `cap task ` | the most one task may spend — $5 unless set; `-yes-spend` does not lift it | | `/task` | — | — | opens the full-screen task page — the same page as `/history` and ctrl+. | | `/task` | — | `` | starts one worker at once; its brief is written and its width read beside it, and wide work splits | | `/task` | — | `solo ` | starts one worker at once, with no reading of its width | +| `/task` | — | `--best ` | starts the task on the strongest crew the allowed models make, this task only | +| `/task` | — | `--cheap ` | starts the task on the cheapest crew that does the work, this task only | +| `/redo` | — | `stronger` | runs the last task again on a stronger crew, and teaches the crew that kind of work needs more | | `/history` | — | — | opens the full-screen sessions place — every task this machine has run, filterable (also ctrl+.) | | `/status` | `/info`, `/context` | — | prints every fact the status line knows, one per line | | `/status` | `/info`, `/context` | `--json` | prints the same facts as one JSON object, keys in the same order | @@ -670,20 +677,18 @@ whole wall time — the right edge of the row shows the live `38 tok/s` instead) (`2 open · 1 waiting`) and `watching` (the standing count, which is drawn at the foot of the task column). `crew` is a fifth and has its own line above. -The `crew` line sits directly under `model` and reads the preset word — or `custom` — and -the three classes: +The `crew` line sits directly under `model` and says the crew is auto, with any seat you +pinned after it: ``` -crew max · brain claude-opus-5 · hands glm-5.3 · checks claude-fable-5.1 +crew auto · pinned checker moonshotai/kimi-k3 ``` -The crew is **not on the status line**. It was one short segment there — `crew max`, or -`crew custom` — at the head of the telemetry until 2026-09-09, and it came off: the row is -a ledger of things you act on from it, and a preset is changed on a page. The `crew` line -here and on the phone's status sheet is where it is read now, in full. Every ordinary -launch has a crew — one is never unset, only `custom` — so the line is always there; the -one session that shows none is a **remote** one opened with `--host`, where the crew -belongs to the other machine. +The crew is **not on the status line**: the row is a ledger of things you act on from it, +and the crew is changed on its panel. The `crew` line here and on the phone's status sheet +is where it is read. Every ordinary launch has a crew — auto is an answer, not an absence — +so the line is always there; the one session that shows none is a **remote** one opened +with `--host`, where the crew belongs to the other machine. `/status` differs from the on-screen status sheet in two deliberate ways: @@ -824,8 +829,8 @@ the same row the Spending tab writes through. The row names it takes are **`day`** (`daily`, `today`), **`conversation`** (`chat`, `session`), **`plan`** (`plans`, `ask`) and **`practice`** — the four rows that can be -edited. There is deliberately **no `/budget task`**: a task has no dollar limit of its -own, so a command that accepted one would be writing a number nothing reads. +edited. There is **no `/budget task`**: the per-task limit is set in `/crew`, with +`/crew cap task <$>`. A write says back what it landed, in the tab's own words for that row — `per day · $50`, or `per day · no limit`. A figure it cannot read is refused in the row's own words with @@ -1474,76 +1479,54 @@ On a project that has never run a task the page opens on its heading and one lin `work you send off with /task lands here, and its record stays`. The tasks pages describe the page in full. -## /crew — the five models codeaf uses on its own behalf, read beside the one you talk to +## /crew — the crew panel, and the three seats a task runs on -codeaf runs **six model seats**. Seat one is the model you talk to, and `/model` is what -moves it. The other five — reflex, small work, worker, careful work, mastermind — are the -models codeaf uses on its own behalf, for the calls you did not type and for the work -inside every task. `/crew` reads all six and sets the five in one word. **It never moves -seat one.** +The crew is the three seats a task runs on: the **worker** that does the work, the +**planner** that structures it, and the **checker** that reads the result. By default all +three are auto — codeaf picks each one for each task, from what kind of work the task is. +**`/crew` never moves the model you talk to**; only `/model` does. ``` /crew ``` -opens the six-seat reading, bottom-anchored like the model picker. From the top: +opens the crew panel over the conversation — six rows you change in place (the worker, +planner and checker seats, the allowed **models**, the **providers**, the daily **cap**) +and a dim `today` line. `enter` changes the row under the cursor, `esc` closes, `?` lists +every key. On the **providers** row `←`/`→` walk one chip per connected provider and `space` +turns the one under the cursor off or on; the `+` at its end opens `/connect`. The +models page has the panel drawn out and every key (*Crew panel keys*). A seat on auto says +the model it usually runs; a pinned seat wears the pin mark `⌖`. Typed on home, `/crew` +steps off home and `esc` brings you back. -``` -the five models codeaf uses on its own behalf — not the one you chat with - you talk to · deepseek-v4-flash - family ‹ open models · all models › - frugal — glm-flash works and thinks, qwen-max checks - reflex google/gemini-2.5-flash · small work deepseek/deepseek-v4-flash-0731 · worker z-ai/glm-5.3-flash · careful work qwen/qwen3.8-max-0902 · mastermind z-ai/glm-5.3-flash -› balanced — glm-flash works, fable checks, opus thinks - reflex google/gemini-2.5-flash · small work deepseek/deepseek-v4-flash-0731 · worker z-ai/glm-5.3-flash · careful work anthropic/claude-fable-5.1 · mastermind anthropic/claude-opus-5 - max — glm-5.3 works, fable checks, opus thinks - reflex google/gemini-2.5-flash · small work deepseek/deepseek-v4-flash-0731 · worker z-ai/glm-5.3 · careful work anthropic/claude-fable-5.1 · mastermind anthropic/claude-opus-5 -each of the five can be pinned on its own in /settings → Providers -``` - -The first line says what the presets change and what they do not. The second is **seat -one** — `you talk to · `, spelled as the legend above the box spells it — with no marker and -no highlight, because nothing in this chooser can move it. The third is the **family**: **←→** moves `family`, which says which pool the three presets below it draw from, open weights or the -whole catalog. Then the three presets: the one -in force wears a highlighted ground, `›` is where **enter** is aimed and it opens on yours, -↑ / ctrl+p and ↓ / ctrl+n move, and **esc** closes without changing anything. The last -line points at the settings row where one seat can be pinned by itself; the chooser does not -pick seats one at a time. - -If you have pinned one of the five yourself, no preset wears the ground and the chooser says -`yours is none of the three — picking one puts all five back` above the closing line. - -A word that is not one of the three changes nothing and prints the three: -`/crew cheap` answers `/crew cheap · not one of the three` and then the listing. - -What each of the five classes funds, and how to set one of them on its own, is on the models -page. - -## /crew — the confirm line, and the model it leaves alone - -`/crew frugal`, `/crew balanced` or `/crew max` sets the five and confirms in one line: +The shortcuts write the same rows and then open the panel with a tick on the row they +changed: ``` -crew → max · brain claude-opus-5 · hands glm-5.3 · checks claude-fable-5.1 · you are still talking to deepseek-v4-flash — /model changes that +/crew · /crew pin · /crew unpin · /crew models · /crew cap · /crew cap task ``` -**The model ids are drawn brighter than the words around them.** `crew →`, the preset -word, `brain`/`hands`/`checks` and `you are still talking to` stay at the grey every note -is written in; the three crew ids and the model you are talking to step up, because they -are what the command was typed to find out. `/model` wears the command chip, because it is -a command you can type. See "Why is one word in a line brighter than the rest" on the -screen page. +- **`/crew pin `** pins one seat, for every task until you unpin + it. `@provider` sends it through that connection. A pin outside the allowed models is + refused. The confirmation names the model you are still talking to: + `checker ⌖ moonshotai/kimi-k3 · every task until you unpin it · you are still talking to + deepseek-v4-flash — /model changes that`. +- **`/crew unpin `** puts the seat back on auto; `/crew unpin all` puts all three back. +- **`/crew models `** sets which models a seat nobody pinned may be picked from — + `all`, `open`, `≤1/5` for a price ceiling per million tokens in and out, or a list of ids — + and `+id` or `-id` changes the rule in force by one word. Bare `/crew models` says the rule. +- **`/crew cap `** caps what crews may spend in a day. Bare `/crew cap` says the + cap and today's spend. +- **`/crew cap task `** sets the most one task may spend — $5 unless set. A call + that would take a task past it is not made, and `-yes-spend` does not lift it. -The last clause names, by id, the one seat the command did not touch: the model you are -talking to, in the same spelling the legend above the box uses, so you can check it -against the line over your own prompt. `/crew` never changes that model and never offers to; only -`/model` does. When the session has no model yet the clause reads -`the model you talk to is untouched — /model changes that`. +A form that is none of these changes nothing and prints the line of shortcuts: +`/crew cheap · not a crew form · …`. How hard to try **one** task is not a panel setting at +all — it is said with `/task --best` or `/task --cheap`, and `/redo stronger` asks again +after the fact (see *Tasks*). -**The change is live.** The next call codeaf makes on its own uses the new crew — no -relaunch, and no waiting for the next session. To read the crew back afterwards: the live -`/status` prints the `crew` line under `model`, the phone's status sheet has the same row, -`/settings` → Providers has the crew row, and bare `/crew` opens on yours. +**The change is live.** The next task uses it — no relaunch. The whole story, from how a +seat is picked to how a profile from an earlier build is migrated, is on the models page. **That promise is local-session only.** Over `--host` the session resolves its crew from the other machine, and there is no crew write across the connection — so `/crew` refuses @@ -1728,7 +1711,7 @@ Every one of them lands on the next launch rather than on the conversation in fr you, so they are a fact about this machine and not about this session. Nothing you saved moved: the keys they are stored under are unchanged. -The five models codeaf uses on your behalf are **not** here — they are on Providers, with +The models codeaf uses on your behalf are **not** here — they are on Providers, with the row that says which model you are talking to. They used to be on this tab, one tab away from it, which made "which model does the planning" and "which model am I talking to" two errands on two screens. Neither is the conversation's own money limit here any more: it is @@ -1832,17 +1815,18 @@ order: 5. **prompt profile** — how much codeaf tells the model before you type: `auto`, `lean`, `full`. Another cycle row. `auto` reads the model's context window and goes lean under 32,000 tokens (see *Models, context, and what it costs*). -6. **crew** — the five below, chosen as one word: `frugal`, `balanced`, `max`. It is a cycle - row: enter or space walks it. Answer any of the five yourself and it reads `custom`. -7. **reflex** — `near-free · reads every turn — memory, titles, safety` -8. **small work** — `cheap · the small calls — names, digests, the safety gate` -9. **worker** — `does the work · every task, its parts, every run node — most of the bill` -10. **careful work** — `careful · checks what must not be wrong — audits, briefs, vision` -11. **mastermind** — `thinks · plans runs and designs harnesses — add :low, :medium or :high` -12. **pinned roles**, and hanging off it the **roles** list — one row per auxiliary call - codeaf makes for itself, grouped under its class. Those rows come from the running binary +6. **reflex** — `near-free · reads every turn — memory, titles, safety` +7. **small work** — `cheap · the small calls — names, digests, the safety gate` +8. **seats** — `the worker, planner and checker, the models they may be picked from, the providers they may route through, and the per-task and daily caps · enter opens /crew` +9. **pinned roles**, and hanging off it the **roles** list — one row per auxiliary call + codeaf makes for itself, grouped under its row. Those rows come from the running binary rather than the settings registry. +**worker**, **checker** and **planner** are the crew's three seats, and on the tab they are +one row, **seats** (`auto · 1 pinned · models open · 3 of 4 providers · per task $5 · daily $5.00`). +`enter` on it opens the `/crew` panel, where the seats, the allowed models, the providers +and the per-task and daily caps are changed; `esc` there comes back to the row. + **A pin for a role this build no longer has is ignored, and the row stops showing it.** Roles come and go with the calls that use them — `compaction` was one, and a compaction has not asked a model since long before it was deleted. A pin left behind for a word like that is dropped @@ -1863,9 +1847,9 @@ that has come apart is cut and asked again, on by default (see *Models, context, it costs*) — and one row per capability slot added automatically from the settings registry: drawing, speaking, composing, filming, voice. -The first four of the five classes are **select** rows and open the model picker. The -**mastermind** row is a **text** box instead, because its value may carry a thinking level -(`moonshotai/kimi-k3:high`) and a picker hands back a bare id. +The reflex, small work, worker and checker rows are **select** rows and open the model +picker. The **planner** row is a **text** box instead, because its value may carry a +thinking level (`moonshotai/kimi-k3:high`) and a picker hands back a bare id. The connected model services have their own section on the tab, each with its billing door, the safe spelling of its key, its region and its order. The section ends with an @@ -1924,13 +1908,13 @@ one call codeaf makes outside a turn — `title`, `guardian`, `auditor`, one of them you ask for yourself with `ctrl+r` (see the keys page) — drawn as ` `, with `pinned` after it when that role has a model of its own. -The rows are **grouped under their class**, in the same order the five class rows are drawn -above them: `roles · reflex`, `roles · small work`, `roles · worker`, `roles · careful work`, -`roles · mastermind`. The class is the heading, so it is not repeated on every row — which +The rows are **grouped under their row**, in the same order the five model rows are drawn +above them: `roles · reflex`, `roles · small work`, `roles · worker`, `roles · checker`, +`roles · planner`. The row is the heading, so it is not repeated on every role — which leaves the widest part of the row for the model id it is there to show. Stop on a row and the line under the list says **what that role is** and where its answer -came from: `the plan that steers an adaptive run · follows mastermind above. enter pins it +came from: `the plan that steers an adaptive run · follows planner above. enter pins it to a model of its own.` - **enter** opens the model picker and pins the role to what you choose. diff --git a/internal/manual/chat/empty-screen.md b/internal/manual/chat/empty-screen.md index 6591997bdf..190c4d06c9 100644 --- a/internal/manual/chat/empty-screen.md +++ b/internal/manual/chat/empty-screen.md @@ -9,7 +9,7 @@ A new conversation opens on **one centred group** and nothing else. From the top ┌─ ┌─┐ ┌─┤ ┌─┐ │ │ ├─ the codeaf wordmark, drawn CodeAF │ │ │ │ │ ├─╴ ├─┤ │ └─ └─┘ └─┘ └─┘ ╵ ╵ ╵ -anthropic/claude-sonnet-4.5 · balanced crew +anthropic/claude-sonnet-4.5 · auto crew a blank row › _ the message box, cursor in it try "what is in this folder" · /task starts work · / shows commands @@ -20,8 +20,8 @@ recent sessions only when this folder has earlier conversations ``` The line under the wordmark is the model that will answer, as its full routing -address, and the crew preset behind it (`frugal`, `balanced`, `max`, or `custom`) — -the two facts that decide what a turn costs. The group is centred in the window, a +address, and `auto crew` — tasks get their own worker, planner and checker, picked per +task — the two facts that decide what the work costs. The group is centred in the window, a shade above the middle. It shows once, arrives with a slow sweep across the letters over about a second and a quarter, and is then still. diff --git a/internal/manual/chat/getting-started.md b/internal/manual/chat/getting-started.md index bf2ff2396f..9f67dda66a 100644 --- a/internal/manual/chat/getting-started.md +++ b/internal/manual/chat/getting-started.md @@ -17,8 +17,8 @@ an empty prompt and a provider error. It opens in the chat itself, on **two scre under a minute, nothing else on the frame: 1. **connect openrouter** — the default service; `enter` signs in in your browser, and pasting an existing key also works -2. **Models and spending** — one screen with three controls on it: **Daily limit**, - **Chat model** and **Work crew**, each already showing the value that is in force +2. **Models and spending** — one screen with two controls on it, **Daily limit** and + **Chat model**, each already showing the value that is in force The second screen's way out is **`Start a conversation`**. Every control on it opens on the value you already have, so pressing `enter` there agrees to exactly what is on the @@ -45,7 +45,7 @@ are standing on — `tab` walks the rows, `?` opens a control's detail — and o window it is cut by whole clauses rather than mid-word. **On a window too short for the whole screen the explanations are what go**, a whole -sentence at a time and never half of one. The three values, `Start a conversation` and +sentence at a time and never half of one. The two values, `Start a conversation` and the keyboard line are never given up, so a sixteen-row window still shows a screen you can answer and leave. @@ -156,46 +156,37 @@ here. They keep their shipped defaults — plan approval asks first above `$100` conversation ceiling is `no limit` — and `/budget` or `/settings` → Spending changes them when they start to matter. -## The model you talk to and the work crew on the setup screen +## The model you talk to on the setup screen — and why it does not ask about the crew **Chat model** is the model you talk to, shown by name — `DeepSeek V4 Flash` rather than `deepseek/deepseek-v4-flash`. Its line reads *The model you talk to in this -conversation.*, and `?` adds the exact catalog id and that it also handles this -conversation's tool use. Opening the row draws the real catalog: five rows at a time, -`↑`/`↓` scroll the rest past, and **typing narrows it**, so two hundred models are -reachable from a form with five rows on it. The row under the cursor shows its exact id. -The model you are already on is always on that list and the cursor opens on it, even with -no catalog yet, so accepting confirms rather than changes. Choosing one goes through the -same settings row `/model` writes and is kept for the next launch. - -**Work crew** is the five models codeaf uses on its own behalf — *Models used to plan, -run, and check tasks.* Opening it draws the three presets — `Frugal`, `Balanced` and -`Max` — each with a whole one-line description of **the choice** (how much model goes on -the work), never a price: this screen makes no claim about what anything will cost you. -`?` on the row shows the seats the crew is actually made of (`brain … · hands … · -checks …`) and that **a crew change leaves the model you talk to alone**. The crew is the five class rows (`models.tiers.reflex`, `models.tiers.low`, -`models.tiers.worker`, `models.tiers.high`, `models.tiers.mastermind`); the model that -answers you is the row above it, and neither touches the other. - -Two rules keep this screen from writing something you did not ask for: - -- **`esc` out of either list leaves the row exactly as it was.** A cursor inside a list - is provisional until you accept it. -- **A crew you arranged yourself is never overwritten.** If any of the five class rows is - already in your profile — you pinned one by hand, or an earlier `/crew` wrote them — the - row reads `Custom`, `Start a conversation` writes no preset over it, and the list still - says `yours is none of the three — picking one puts all five back` if you want one. - -If a **task model** is pinned (`task.model`), one dim line under the crew says so — -`Tasks are pinned to … · /settings changes that` — because that pin takes the worker seat -out of the preset's hands and a crew row that did not mention it would be selling you a -dial that is disconnected. +conversation.*, and `?` adds the exact catalog id, that it also handles this +conversation's tool use, and that tasks get their own crew, picked per task. Opening the +row draws the real catalog: five rows at a time, `↑`/`↓` scroll the rest past, and +**typing narrows it**, so two hundred models are reachable from a form with five rows on +it. The row under the cursor shows its exact id. The model you are already on is always on +that list and the cursor opens on it, even with no catalog yet, so accepting confirms +rather than changes. Choosing one goes through the same settings row `/model` writes and +is kept for the next launch. + +**There is no crew question**, because the crew is three seats — the worker, the planner and the +checker — and codeaf picks all three for each task from what kind of work it is, so there +is nothing to choose before the first task. `/crew` shows the crew, and pins a seat when +you want one model there every time. The model that answers you is the one above, and +neither touches the other. + +**`esc` out of the list leaves the row exactly as it was.** A cursor inside a list is +provisional until you accept it. + +If a **task model** is pinned (`task.model`), one dim line under the chat model says so — +`Tasks are pinned to … · /settings changes that` — because that pin decides the worker +seat, and a screen that did not mention it would be hiding where tasks run. At **112 columns and wider** a bordered panel stands beside these rows, labelled `○ Example · what you can do` and footed `An illustration. Nothing here has run.` — the only bordered surface codeaf draws, so it cannot be read as more form. It holds one request you could type and what it leads to, and follows the row you are on: beside the -crew it shows `/task Fix the failing tests and explain the changes.` That request **types +review row it shows `/task Fix the failing tests and explain the changes.` That request **types itself out once** on arriving and on `←`/`→`, then settles; typing settles it at once. Under 112 columns it is not drawn and the form is unchanged. @@ -242,23 +233,22 @@ Every answer went through a settings row, so every answer has a door: | What you answered | Where to change it later | | --- | --- | | the default service's openrouter key | clear or remove it and the next local interactive launch offers **connect openrouter** again; `/settings`, Providers tab, the **openrouter key** row still accepts a pasted replacement | -| the crew | `/crew` (bare shows the three, `/crew max` sets one), or the **crew** row on the settings panel | +| the crew | nothing was asked — it is auto. `/crew` shows it, and `/crew pin ` pins a seat | | the daily limit | `/budget` (also `/limits`), or `/settings` → **Spending**. `CODEAF_DAILY_BUDGET` in your shell outranks the row | | the model you talk to | `/model`, or the **Chat model** row on the setup screen — the same settings row either way | | memory, permissions, the task countdown | `/settings`; the setup screen only shows them, under `Other settings` | A credential changed in the settings row reaches the running conversation at once, -exactly as the setup's does. The crew and the budget are read live too: the next call -codeaf makes on its own behalf uses the new crew, and the rail is checked against the -new ceiling. +exactly as the setup's does. A crew pin and the budget are read live too: the next task +starts on the new pin, and the rail is checked against the new ceiling. -**The setup asks about three things and no more.** Memory stays on, tool approvals keep +**The setup asks about two things and no more.** Memory stays on, tool approvals keep prompting, and a proposed task keeps its 15-second countdown — none of them becomes a question there, because none can be answered usefully before you have seen codeaf do anything. They are taught where they happen: the countdown is on the task card, and the first permission question explains the actual tool that asked for something. -Under the three fields is one row that shows them: **`Other settings use defaults · +Under the two fields is one row that shows them: **`Other settings use defaults · review`** on a fresh profile, and **`Review other settings`** on a profile that has already written any of them down — it never claims your own settings are defaults. `enter` on that row opens three read-only rows straight off the settings registry: @@ -322,7 +312,7 @@ composer still names what `enter` does — and the first place you land on says line: `your terminal sends opt as a letter — turn on "use option as meta" in …`, naming the terminal you are actually in. -**The first-run setup says it too.** When the three questions are done, a Mac gets one more +**The first-run setup says it too.** When the questions are done, a Mac gets one more line: `the seven places answer opt+1…opt+7 · if opt types a character instead, turn on "use option as meta" in …`. It is a condition rather than a report — nothing has been pressed yet — and it is said once. diff --git a/internal/manual/chat/home.md b/internal/manual/chat/home.md index 02b798bca1..19d9296c82 100644 --- a/internal/manual/chat/home.md +++ b/internal/manual/chat/home.md @@ -1320,8 +1320,9 @@ one behind your back. This is every fate, in the words the drop-up draws them in | **`opens the page`** | `/settings` `/set` `/config` · `/home` · `/search` · `/spend` · `/standing` · `/memory` `/memories` · `/history` · `/task` (bare) | A place replaces a place, exactly as before. | | **`this list is /resume`** | `/resume` `/sessions` | Says `this list is /resume · enter opens a row` — home *is* that list. | | **`onto home's tray`** | `/attach ` | The file — or picture — rides on home's own tray into the conversation you open next. Home says `attached · notes.md · rides with the next conversation`. A bare `/attach` opens the browser aimed at the next conversation's folder, and a file chosen there lands on the tray. | -| **`opens a conversation here first`** | `/files` · `/folder` `/place` `/dir` · `/manual` · `/crew` (bare) · `/permissions` `/perms` · `/connect` · `/harness` · `/subharness` · `/skill` `/skills` · `/copy` · `/select` · `/rewind` `/undo` `/back` · `/compact` · `/export` `/save` · `/standing ` · `/task ` | Opens a conversation at the target — the folder at the right of the keys row and the model on the rule above the box — then runs there. Home closes, exactly as `enter` closes it. `/manual` is on this road since 2026-09-22: it is a question put to the model, so it needs a conversation to be asked in. `/folder` joined it the same day — it gives THIS conversation a folder, and home has no this; the pin it used to be here is `/project`. | -| **`answers here`** | `/help` · `/status` · `/cost` · `/cache` · `/budget` · `/crew ` · `/debug` · `/stop` · `/remember` · `/forget` · a word nobody defined | Answers with a note, and the first line of that note is put on home's own line under the box. `there is no command called /pricing · / lists them` is now something you can read. | +| **`opens a conversation here first`** | `/files` · `/folder` `/place` `/dir` · `/manual` · `/permissions` `/perms` · `/connect` · `/harness` · `/subharness` · `/skill` `/skills` · `/copy` · `/select` · `/rewind` `/undo` `/back` · `/compact` · `/export` `/save` · `/standing ` · `/task ` | Opens a conversation at the target — the folder at the right of the keys row and the model on the rule above the box — then runs there. Home closes, exactly as `enter` closes it. `/manual` is on this road since 2026-09-22: it is a question put to the model, so it needs a conversation to be asked in. `/folder` joined it the same day — it gives THIS conversation a folder, and home has no this; the pin it used to be here is `/project`. | +| **`answers here`** | `/help` · `/status` · `/cost` · `/cache` · `/budget` · `/debug` · `/stop` · `/remember` · `/forget` · a word nobody defined | Answers with a note, and the first line of that note is put on home's own line under the box. `there is no command called /pricing · / lists them` is now something you can read. | +| **`opens the crew panel`** | `/crew` and every `/crew` shortcut | Home steps aside and the crew panel opens over the conversation behind it; `esc` on the panel brings you back to home. A shortcut opens it with a tick on the row it changed. | | **`runs on the conversation behind home`** | `/land` · `/land ` · `/workspace ` | Acts on the conversation this window is holding behind the screen — not on the one `enter` would open — and its answer is echoed onto home's line. | | **`a fresh conversation behind home`** | `/new` `/clear` `/clean` `/reset` | Replaces the conversation behind the screen and says `started a fresh conversation behind home`. It is not the same act as `enter`, which opens a conversation at the target. | | **`closes the conversation behind home`** | `/quit` `/exit` `/q` | Closes it and says `closed · `. When it was the last conversation this terminal was holding, codeaf leaves. | diff --git a/internal/manual/chat/how-tasks-run.md b/internal/manual/chat/how-tasks-run.md index 66eeeb9083..745a7d702b 100644 --- a/internal/manual/chat/how-tasks-run.md +++ b/internal/manual/chat/how-tasks-run.md @@ -1791,12 +1791,12 @@ The worker is fresh; the working copy is not. The task stays *running* while a r way, and you see one plain line of what is being closed. **A correction round is attempted by a more capable model.** The first attempt runs on the -task's own model; when a check finds gaps, the worker sent back to close them runs on your -crew's **careful work** model — the `repair` role, in `/crew`. It is the one place codeaf +task's own model; when a check finds gaps, the worker sent back to close them runs on the +task's **checker** — the `repair` role, under the checker row. It is the one place codeaf spends more than you asked it to, and it is spent only after something has actually gone wrong, on a job the check has already narrowed to named gaps in a working copy that is -already most of the way there. Two things turn it off by themselves: a crew whose careful -model is the same as the model the work is on repairs on that model and costs nothing +already most of the way there. Two things turn it off by themselves: a crew whose checker +is the same as the model the work is on repairs on that model and costs nothing extra, and a task whose model **you named** — on the card or from inside its room — keeps your model for the correction round too. @@ -2179,7 +2179,7 @@ what was left of the work is **not work a worker can do**: an approving review o person may give, a credential or an account nobody here holds, a decision that is yours to make, or a step that is somebody else's system doing something by itself. -It is found by the same `mastermind` model that reads a task's parts before it splits (the +It is found by the same **planner** that reads a task's parts before it splits (the tasks page, *when a task turns out to be too wide for one worker*). That reading runs **beside the task's worker**, which has already started, and when it comes back saying nobody here can do this, the worker is stopped where it is — the one time a started worker diff --git a/internal/manual/chat/keys.md b/internal/manual/chat/keys.md index 7213acc594..8b373d415c 100644 --- a/internal/manual/chat/keys.md +++ b/internal/manual/chat/keys.md @@ -1779,6 +1779,18 @@ move · `enter` activate. Its foot reads `esc close · ↑↓ move`. `end`/`G` bottom · `ctrl+o` lifts the line cap. Its foot reads exactly `esc close · ↑↓ scroll`, or `esc close · ↑↓ scroll · tap … for the rest`. +**Crew panel** (`/crew`): `up`/`k`/`ctrl+p`, `down`/`j`/`ctrl+n`, `home`, `end` walk the six +rows · `enter` changes the row · `esc` goes back one level and closes · `?` lists every key · +`z` undoes the last change while the bottom edge offers `z undo`. On the **models** row +`←`/`→` step `all`, `open`, `price`, `custom`, and `enter` steps forward like `→` until +`price` or `custom`, where it opens the ceilings or the checklist. On the **providers** row `←`/`→` walk the +chips and `space` turns the one under the cursor off or on — the foot reads +`enter change · space toggle · esc close · ? keys` there and nowhere else — and `space` on +the `+` opens `/connect`, whose `esc` comes back to this row; `enter` opens the providers list, whose foot reads +`space or enter toggle · esc back`. On the **cap** row a digit starts the figure. A click on +a row is `enter`, a click on a provider chip toggles it, and the wheel walks. The models +page has every key (*Crew panel keys*). + All of these are modal: while one is up, every chord except `ctrl+c` belongs to it. `ctrl+c` does not close the panel — it quits codeaf, with the panel still up. diff --git a/internal/manual/chat/models-and-cost.md b/internal/manual/chat/models-and-cost.md index d610020560..1970c68f95 100644 --- a/internal/manual/chat/models-and-cost.md +++ b/internal/manual/chat/models-and-cost.md @@ -461,209 +461,401 @@ another conversation cannot use this model-changing door. `/model` says behind that page. Provider pinning with `@provider` or `auto` remains available from the conversation's `/model`. -Changing the conversation model does not move existing tasks. New tasks resolve -their model from an explicit choice, the task model setting, the crew's worker -class, then the conversation model when that worker class is blank. - -## The crew — which models codeaf uses on my behalf, and /crew - -codeaf makes calls you did not type: naming a session, naming a piece of work on the roster, -the brief a task opens on, the safety gate, the check on finished task work, the second -look before a task starts itself, the reading of a task's parts before they are handed out, -the planner of an adaptive run and the nodes under it, the designer of a saved harness page, -looking at an image. Each of those is a -**role**, and every role sits on one of five **classes** — the **crew** — which you set in -`/settings` → Providers, or in one word with `/crew`: - -- **reflex** — near-free · reads every turn — memory, titles, safety. -- **small work** — cheap · the small calls — names, digests, the safety gate. -- **worker** — does the work · every task you hand off, the parts it divides into, every - node of an adaptive run. Most of what a task costs is spent here. -- **careful work** — careful · checks what must not be wrong — audits, briefs, vision. -- **mastermind** — thinks · plans runs and designs harnesses. - -**All five arrive with a model already in them**, and on an account that is not -known low on OpenRouter credits the five together are the `balanced` preset. -While the account is known low, unwritten table seats use the free crew described -in *OpenRouter credits and free models*, and `/status` calls that crew `free`: - -| class | as shipped | -| --- | --- | -| reflex | `google/gemini-2.5-flash` | -| small work | `deepseek/deepseek-v4-flash-0731` | -| worker | `z-ai/glm-5.3-flash` | -| careful work | `anthropic/claude-fable-5.1` | -| mastermind | `anthropic/claude-opus-5` | - -None of them is the model you are talking to. A crew that followed your conversation would -put the most expensive model in the build on the cheapest questions in it — a call made -twice every turn on a frontier model is a bill nobody agreed to. That is also why the two -seats that read every turn stay on near-free models while the frontier ids sit on the seats -that answer a handful of times. You can pin any vendor's model on any row yourself, and you -can connect that vendor as a direct service; the -[services page](services.md) explains its names, limits and missing Phase 1 cost record. - -**The `model family` row** (`models.crew.source`) decides which family -the three preset words draw from. `all` is the default and is the table above: the whole -catalog, closed and frontier models included, costing what those models cost. `open` reads the -same three words, `frugal`, `balanced` and `max`, off the open-weight rows only, so no seat is a -bet on one vendor's pricing. Flip the row and the seats nobody pinned move with it at once, -because an unwritten seat is the default crew resolved in the family you are on; the rows already -on disk keep their ids until you pick the crew again, and the crew word reads `custom` while they -match no preset in the family you flipped to. - -Under `open`, the same three words resolve to these: - -| class | frugal | balanced | max | -| --- | --- | --- | --- | -| reflex | `mistralai/mistral-nemo` | `mistralai/mistral-nemo` | `mistralai/mistral-nemo` | -| small work | `deepseek/deepseek-v4-flash-0731` | `deepseek/deepseek-v4-flash-0731` | `deepseek/deepseek-v4-flash-0731` | -| worker | `z-ai/glm-5.3-flash` | `z-ai/glm-5.3-flash` | `z-ai/glm-5.3` | -| careful work | `z-ai/glm-5.3-flash` | `moonshotai/kimi-k3` | `moonshotai/kimi-k3` | -| mastermind | `z-ai/glm-5.3-flash` | `z-ai/glm-5.3` | `z-ai/glm-5.3` | - -The worker column is the dial in that family — it holds `glm-5.3-flash` through `balanced`, and -`max` moves it to `glm-5.3` — because it is the seat that pays most of a task's bill and a -preset that moved every other seat would change everything about a task except its cost. - -**Why these ids.** Every row of the catalog was placed on two axes on 2026-09-16: the bill a -seat's own call shape runs up, built from the catalog's published prompt, completion and -cache-read prices, against that seat's quality, taken from its published intelligence, coding -and agentic indexes. The call shape is part of the price, so the seats were priced apart — the -worker and the careful seats as long cached loops, where a large prompt is read back turn -after turn and the cache-read price carries most of the weight, and the mastermind as one-shot -calls, where the prompt is paid in full each time and there are few of them. Each preset then -takes, for each seat, a point on the pareto front of that plot at the bill it is willing to -run: nothing on the front costs less at the same quality, and nothing at the same bill scores -higher. The two families are the same plot over two sets of rows, which is why the columns do -not climb together in either of them. - -In the `all` family the worker stays on `glm-5.3-flash` through `balanced`, because the worker -seat carries most of a task's tokens: a step there multiplies through the whole bill, where a -step on the careful or the mastermind seat is paid a handful of times. So `frugal` to -`balanced` spends on those two low-volume seats, and `max` moves the worker itself. The -careful and mastermind seats are settled by `balanced` and stay there through `max`. - -The careful class always sees images — the vision role rides that row — and is a different -vendor from the worker in every preset of the `all` family; the open family's `frugal` row -is the one standing exception, with worker and careful both on `glm-5.3-flash`, because at -that bill the open-weight front has no second vendor for the careful seat. The small-work row is pinned to -the July build of DeepSeek V4 Flash on purpose: the bare `deepseek/deepseek-v4-flash` id -resolves to the April build, and the July build costs the same. - -**Clearing a row is still an answer.** A class you empty on purpose reads -`follows the conversation`, and every role on it runs on the model you are talking to. That -is the only way to say "use my model for this", and it is deliberately something you have to -say rather than the default. - -### The three presets - -These are the three in the family you are on unless you changed the `model family` row — the -default one, `all`: - -| | frugal | balanced | max | -| --- | --- | --- | --- | -| reflex | `gemini-2.5-flash` | `gemini-2.5-flash` | `gemini-2.5-flash` | -| small work | `deepseek-v4-flash-0731` | `deepseek-v4-flash-0731` | `deepseek-v4-flash-0731` | -| worker | `glm-5.3-flash` | `glm-5.3-flash` | `glm-5.3` | -| careful work | `qwen3.8-max-0902` | `claude-fable-5.1` | `claude-fable-5.1` | -| mastermind | `glm-5.3-flash` | `claude-opus-5` | `claude-opus-5` | - -- **frugal**: glm-flash works and thinks, qwen-max checks -- **balanced**: glm-flash works, fable checks, opus thinks -- **max**: glm-5.3 works, fable checks, opus thinks - -The reflex and small-work columns never vary — they are the -same near-free models in all three — so `/crew` never names them: the confirmation and -`/status` say **brain**, **hands** and **checks**, which are the mastermind, the worker and -the careful class. - -`/crew` opens all three as a chooser with yours marked, under a scope line — `the five -models codeaf uses on its own behalf — not the one you chat with` — and a `you talk to · -` line naming the seat the presets do not touch. ↑ / ctrl+p and ↓ / ctrl+n move; -enter applies and esc cancels. If the five classes make a custom crew, no row is marked and -the chooser says picking one puts all five back. `/crew max` still sets it directly and -confirms in one line, which ends `· you are still talking to deepseek-v4-flash — /model -changes that` — naming the conversation's own model by id, because the crew changes -nothing about it and the model segment on the status line goes on saying what it said -before. `/crew learn`, `/crew catalog` and `/crew table` move the **picked from** row and -nothing else — the three words the row takes, answered without touching a model id — and -when the pick is off `table` the crew word says so beside the preset: `balanced · learn` -in `/status`, `crew balanced · learn` on the status line. The **crew** row in `/settings` -→ Providers is the same thing: enter or space walks it frugal → balanced → max, and the -**picked from** row under it walks table → catalog → learn. - -**The crew row is not stored by codeaf — it is worked out from the five.** Answer any one -of the five rows yourself and the crew row reads `custom`, because that is what is true. -`/crew balanced` puts all five back in one write. A profile that applied a crew before the -worker row existed reads `custom` until a preset is applied again, because its four old -rows and the new fifth are not any of the three. A run that writes the crew word into the -profile itself — nothing this build does, but a harness or a hand edit may — is read as -the budget its seats run at, and the class rows under it are that run's own pins. - -### Where the seats are picked from — table, catalog, learn - -The **picked from** row (`models.crew.pick`, just under the crew row) says where the -seats' models come from when a class row does not hold a model id of its own. The crew -row above it still says how much to spend; this row says where the models for that -money are read from: - -- **table** — the rows we measured: the ids this build shipped with, the same ones every - preset table holds. This is the default, and it is what an unwritten class has always - read. -- **catalog** — recomputed from today's published prices and scores at your crew's - budget, on every read. Nothing is stored; a catalog that moves moves the seat with it, - and your profile never holds a model id this build chose for you. -- **learn** — the catalog computation plus the Model Pool's measurements and your own - judged runs, carried as a quality rating the better-measured models read on top of - their published scores. - -`catalog` and `learn` are Pareto crewing: the crew is picked on the cost-quality front, -per role and per task, from evidence rather than from a fixed table. - -A pick moves the three seats the presets dial — **worker**, **careful work** and -**mastermind** — and never the two that read every turn: **reflex** and **small work** -keep their near-free ids, the same ones in every preset. - -**A model you typed by hand wins.** The pick answers for the seats nobody named. A class -row YOU wrote keeps its model, spelled as you typed it — the crew table's own id included -— and a `models.crew` word stored in the profile marks every class row beside it as yours. -`/crew balanced` writes all five classes at once, so it is the preset answering rather -than a pin and the pick computes its three dial seats. **A flag or an environment variable -still outranks the pick** — the pick reads the profile, and `--model` and `CODEAF_MODEL` -are what an invocation said. - -When the catalog cannot compute a seat — no catalog yet, or no pick off its front — the -seat falls back to the table row for your preset, never to `auto` and never to empty. A -seat the pick computed names it where the preset would be: `crew balanced, computed -from the catalog` under **catalog**, `crew balanced, learned` under **learn**. - -**The per-seat alias is the bare word `auto`.** Any of the five class rows may hold the -bare word `auto` instead of a model id, case folded. The seat's model is then -**computed from the catalog** — the three published capability indexes against the three -published prices, under that seat's own call shape — every time the row is read. The -word stays on disk; the id is worked out on every read. - -**The budget it computes at comes from the other four rows.** `auto` has no opinion about -cost of its own, so it runs at whatever preset the rows around it name: four rows that are -`balanced`'s make an `auto` row a balanced seat, four that are `max`'s make it a max seat. - -**When the other rows match more than one preset, `balanced` wins.** It is the default -preset, and the budget an undecided profile runs at is the budget an `auto` seat runs at. -This is not a rare corner: `max` differs from `balanced` only in the worker seat, so a -crew with `auto` on the worker and the other four rows as shipped matches both, and reads -as `balanced`. Pin the worker to `max`'s own id and put the `auto` row on a seat above it -if you want a computed seat at `max`'s budget. - -A seat on a computed row names both facts where a seat is shown — `crew balanced, -computed from the catalog` — so the reading is never a guess. - -### The roles under each class - -Directly under the **pinned roles** row the panel lists **every registered role**, grouped -under the class answering it, saying which model comes out. As shipped: - -| role | class | what it is | +Changing the conversation model does not move existing tasks. A new task's +worker comes from an explicit choice, then the task model setting, then the +crew's **worker** seat — a pin, or the model the crew picked for that task — +and only when nothing answers, the conversation model. + +## The crew — which models a task runs on, and /crew + +The crew is three seats: the **worker** that does the work, the **planner** that +structures it, and the **checker** that reads the result. **By default all three are auto.** +codeaf picks each seat for each task: it reads what kind of work the task is — a +**bugfix**, a **complex fix**, **open-ended** work, or **other** — and picks the model for +each seat from what its catalog row says about it, weighed against what the model costs. +Every model a connected provider serves is a candidate, frontier models included, as long +as the allowed models admit it. A model that publishes any index — the intelligence, coding +or agentic index, or an arena rating — is scored on those alone, through learned weights +that ship with codeaf, for that seat on that kind of work; price never counts as ability, so a model +no dearer than another and at least as good on every index they both publish is never ranked +below it. A model that publishes none is scored from its context, release date, licence and +family, never above the average model, and less surely. Each seat weighs a model a little +below its score by how unsure the score is, and a row is scored again when the catalog next +lists new figures for it. How this install's own tasks ended — accepted, kept, redone, failed — moves +a model's score in a seat a little each time, within a bound, and never freezes it. A task it +cannot read with confidence counts as open-ended, because that is where a weak crew costs +the most. + +The pick stops where more money stops buying much. Every seat pays for ability, and +open-ended work pays more for it than a fix does, so a small fix usually runs on a cheap crew +and open-ended work buys a stronger model sooner as prices rise. On open-ended and other work +that upgrade goes to the **checker** first — the seat that accepts the work — and the planner +stays on the base model; `--best` upgrades every seat. The worker and the checker are never +simply the cheapest model: their score must reach the ability of the weakest model seen doing +the work, whenever an allowed model's does, and `--cheap` takes a clearly stronger worker when +it costs no more than half again as much. A fix whose report shows **reach** — +more than one file, an API or protocol, language rules, a long report or several repros, +a security defect, existing tests that must keep passing, two of these at least — is a +**complex fix**: its worker is one rung stronger than a simple fix's; its planner and +checker are the fix's own, the line still says bugfix, and a worker you pinned stays +pinned. An issue's own labels count most: +a `bug` label in any spelling (`bug :bug:`, `type/bug`) makes the task a bugfix. The price is the one you +would actually pay: a model you reach through a subscription plan you connected costs +nothing extra, and a local model costs nothing at all, so the crew prefers those routes +whenever one serves the model. + +The crew is three seats because those are the calls a task spends most of its money on. +codeaf also makes smaller calls on your behalf — naming a session, the safety gate, memory +— and those ride two rows of their own in `/settings` → Providers, **reflex** and +**small work**, which ship pointed at near-free models and are not part of the crew. + +**`/model` is untouched.** It is the model you talk to, and nothing about the crew moves +it. + +### The /crew panel + +Bare `/crew` opens the crew panel: a framed panel over the conversation, with six rows you +can change and one line about the day. + +``` +╭─ crew ──────────────────────────────────────────────────────── esc ─╮ +│› worker auto · usually glm-5.3-flash │ +│ planner auto · usually glm-5.3-flash │ +│ checker ⌖ kimi-k3 │ +│ │ +│ models ‹ all › (96) │ +│ providers ✓ openrouter ✓ z-ai sub ✓ ollama local ○ my-vllm + │ +│ cap per task $5 · daily none │ +│ │ +│ today $1.84 · 14 tasks │ +╰─ enter change · esc close · ? keys ─────────────────────────────────╯ +``` + +- **the seats** — `auto · usually ` for a seat codeaf picks, naming the model the + recent tasks — the last eight — ran there most (`likely` before there is any history, + and whatever the router would pick now when the usual model is no longer reachable); or the pin mark `⌖` and the + model, with `@provider` when a route is pinned too. A pin nothing connected can run says + `unavailable`. +- **models** — which models a seat may be picked from, walked with `←`/`→` in place: + `all`, `open`, `price`, `custom`, with the number of models each admits in brackets. The + number counts only models a provider that is on still serves. +- **providers** — one chip per connected provider: `✓` on, `○` off and dim. An API key is + its plain name, a subscription says `sub`, a model on this machine says `local` when there + is room, and a custom endpoint is the name you gave it. The `+` at the end opens + `/connect`. On a narrow window the chips fold to `3 of 4 on`. +- **cap** — `per task $5 · daily none`: the most one task may spend, and the most crews + may spend in a day, `none` for no daily cap. +- **today** — what crews spent today and how many tasks ran. A day with nothing in it has + no line. + +**The allowed models are the models rule less every model no provider that is on serves.** +Turning a provider off takes its routes away, never a model: a model another provider still +serves stays allowed on that route, and the crew's routing and its route costs only ever +use providers that are on. Which providers are off is saved beside the rule, not inside it, +so walking the models row to a new answer never turns a provider back on. A provider you +connect later is on the day you connect it. **At least one provider must stay on**: turning +off the last one is refused under the rows with `at least one provider must stay on`, and +turning off the last one that can route a seat — leaving on only a custom endpoint, which a +pin reaches and the crew never picks — is refused with +`at least one provider that can route a seat must stay on`, unless every seat is pinned to +a provider still on. A +pinned seat whose only provider is off says `provider off` on its row. + +With nothing connected the panel says `no providers connected — /connect adds one`; a pin +whose provider is not connected says `unavailable`. A rule that leaves open-ended work +without a strong checker says so under the rows: +`no strong checker among the models you allow · open-ended work will be checked weakly`. +**A pinned checker is the checker**, so the warning asks about the pin alone: a strong pin +clears it whatever else is allowed, and a weak one names itself — +`checker pinned to deepseek-v4-flash · open-ended work will be checked weakly` — even when a +strong model is allowed beside it. + +Every change is saved the moment you make it, and the next task uses it with no relaunch. +The changed row wears a tick `✓`, and for five seconds the bottom edge offers `z undo`, +which puts things back exactly as they were. + +## Crew panel keys — how to pin, unpin, set the cap or the allowed models on /crew + +`enter` is the one verb: + +- **enter on a seat** opens that seat's list: `auto — codeaf picks per task` first, then + the router's suggestion marked with the star and `suggested`, then every model your + providers reach with its price in and out per million tokens and one provider. Type to + filter; `enter` pins. **Unpinning is choosing `auto`** — the list opens on it, so it is + `enter enter`. `→` on a model shows its routes (`any route · cheapest`, then each + provider); `enter` on one pins the model to that provider, `←` folds them. +- **A model outside your allowed models** is on the list marked `not allowed`. `enter` on it + says ` is not in your allowed models () — enter to allow it`; a second + `enter` adds the model to the rule and pins it. +- **models row**: `←`/`→` step between `all`, `open`, `price` and `custom`, saving each; + `enter` steps forward too, the way `→` does, until `price` or `custom`, where it opens + what that answer holds. + On `price` the row becomes two boxes, `≤ $[ 1 ] in / $[ 5 ] out`; `enter` edits the + first, `enter` (or `tab`) moves to the second, `enter` saves. On `custom`, `enter` opens + a checklist of every model your providers reach — models only; providers are the + providers row's — ticked where the rule admits it: type to filter, `space` or `enter` + ticks and unticks, and each tick is saved as the shortest rule that says it + (`open -deepseek`, `all -moonshotai/kimi-k3`). +- **providers row**: `←`/`→` walk the chips, `space` turns the one under the cursor off or + on (the bottom edge says `space toggle` only on this row), and `space` on `+` opens + `/connect` — `esc` there brings you back to this row. `enter` opens the providers list: one line per provider with how it bills + (`api key`, `subscription`, `local`, `custom endpoint`), how many models it serves, what it + carried today and `on` or `off`; `space` or `enter` toggles the line under the cursor, `z` + undoes, `esc` goes back to the row. On a narrow window the list drops what it carried + today first, then the model count, then how it bills; `on` or `off` always stays. The + list's last line is + `free routes off · rate-limited, may log prompts`: turned on, the crew may also use a + model's free `:free` route, which is rate-limited and may log what it is sent. It is off + until you turn it on, and it is on the list only, never on the panel's rows. +- **cap row**: type a figure (a digit starts it) and `enter`; empty it and `enter` for none. + A figure that is not dollars is refused under the rows. +- `z` undoes the last change while the bottom edge offers it; `?` shows every key; + `esc` goes back exactly one level, and on the panel closes it. +- **Mouse**: a click on a row is `enter`; a click on `‹` or `›` steps the models row; a + click on a provider chip toggles it; the wheel scrolls a list. +- **Filtering a seat's list** finds a model from the front of its name: a prefix of the id + or the name comes first, then a word inside it (`flash`), then the letters anywhere inside + it. A row whose letters only match scattered — `kim` in `grok-imagine` — is shown only + when nothing matched better. + +Typical keystrokes: pin the checker is `/crew ↓ ↓ enter kim enter`; allow open-weight +models only is `/crew ↓ ↓ ↓ →`; a $5 cap is `/crew`, down to `cap`, `5`, `enter`. + +The shortcuts do the same writes from the box and then open the panel with the tick on the +row they changed: + +``` +/crew · /crew pin · /crew unpin · /crew models · /crew cap · /crew cap task +``` + +### Pinning a seat + +`/crew pin checker moonshotai/kimi-k3` pins the checker, and every task from then on runs +its checker on that model until `/crew unpin checker` puts the seat back on auto. +`/crew unpin all` puts all three back. + +**A pin may name the provider too**: `/crew pin worker z-ai/glm-5.3-flash@openrouter` sends +that seat through OpenRouter even when a direct connection also serves the model. Without +`@provider` the crew picks the cheapest route that reaches the pinned model. + +**A pin outside the allowed models is refused**, in words, and nothing is written — a pin +the rule would have to break is not a pin. And a pinned model none of your connections can +reach is not quietly swapped: the task does not start, and says why. + +In `/settings` → Providers the three seats are one row, **seats**, which says how many are +pinned, the allowed rule, how many providers are on, the per-task limit and the daily cap; `enter` on it opens the crew panel, and `esc` there +brings you back to the row. + +### Which models are allowed + +`/crew models` says which models a seat nobody pinned may be picked from. An unwritten rule is `all`. + +- **`all`** — every model a connected provider can reach. +- **`open`** — open-weight models only, so no seat is a bet on one vendor's pricing. +- **`≤1/5`** (or `<=1/5`) — at most $1 per million tokens in and $5 out. +- **a list** — `glm-5.3-flash, kimi-k3`: exactly these. A word may be a full id, the name + after the vendor, or a vendor or provider name. + +Any rule can be followed by `+x` and `-x`, read left to right: `open -deepseek` is every +open model but DeepSeek's, and `≤1/5 +moonshotai/kimi-k3` is the price rule with one +exception let in. `/crew models +kimi-k3` or `/crew models -deepseek` changes the rule in +force by one word. A `-x` naming a **provider** — `-openrouter` — takes that provider's +routes away rather than any model; to switch a provider off for the crew, use the panel's +**providers** row, which keeps the choice when the rule changes. + +A rule that would leave a pinned seat outside it is refused until you unpin the seat, and +a rule that does not parse is refused with the reason — a typo that silently allowed +everything would be a setting somebody thinks is protecting them. When the allowed models +leave a kind of work without a strong enough checker, the panel says so on a `gap` line. + +### The daily cap + +`/crew cap 5` caps what crews may spend in a day at $5; `/crew cap off` takes the cap +away. The crew **paces toward it**: once half the day's cap is spent, a dearer crew costs +more of the day's quality to justify, so the picks lean cheaper as the cap gets close. + +**At the cap a task does not start.** In a conversation the task is refused with the cap, +what was spent, and the two ways on — raise it with `/crew cap`, or ask for that task with +`--cheap`. `codeaf do` refuses the same way unless you pass `-yes-spend`; `codeaf exec`, +`codeaf run` and `codeaf plan` say one line about it and go ahead. + +**Nor does a call that would cross it.** Every call a task makes — its seats', and the +helpers around them, such as the run's closing summary — is priced before it is made: +what the day has spent, plus every call still on its way, plus what this call is expected +to cost, which is never less than what the same model charged for its last call today. +A call that would pass the cap is not made: the task stops on `today's crew spend has reached the daily cap +of $5.00 · raise it with /crew cap`, with what it had done so far. A checker cut off this +way ends on the same sentence. + +**A checker has a ceiling of its own on each task**: three times its estimate, and never +less than $0.05. A check that reaches it stops there, on `the check stopped at its spend +ceiling of $0.26, three times its estimate, before it finished`, and the task ends +unchecked rather than on a bill ten times its estimate. + +This cap is the crew's own. The day's limit under `/settings` → Spending counts everything +codeaf spends, and still applies. + +### The per-task limit + +No task may cost more than its limit: **$5** unless you set another. `/crew cap task 10` +sets it to $10; on the panel it is the first figure on the **cap** row +(`per task $5 · daily none`) — `enter` on the row, `tab` to the per-task box, type, `enter`. +A task always has a limit: `none` and `0` are refused, and an emptied box is $5 again. + +Every priced call of one task — each seat's, on every model, and the helpers made for it — +counts against one figure: what the task has spent, plus every call of it still on its +way, plus what this call is expected to cost. A call that would pass the limit is not made, +and the task stops on `this task reached its $5 limit · raise it in /crew`, with what it had +done so far. The checker's own ceiling still applies inside it. + +`-yes-spend` does not lift it. `codeaf do` holds every run to the same limit, with or +without a routed crew; the flag answers the day's questions and the plan-price question, +not this one. + +### How hard to try one task — --best and --cheap + +The panel says what persists. **How hard to try one task is said in the ask, and sticks to +nothing**: + +- `/task --best ` puts the strongest crew the allowed models make on that task. +- `/task --cheap ` puts the cheapest crew that still does the work on it. +- In a conversation, just say so — "do this properly", "cheapest is fine" — and the task + codeaf hands off carries the word as its `effort` (`best` or `cheap`). +- From a terminal, `codeaf do --best` and `codeaf do --cheap` do the same (see *Running + from the terminal*). + +The next task is back on the ordinary pick. + +### What a task says about its crew + +A routed task says its crew in ONE line, under the line that says it started, and the line +is rewritten in place as the task goes. When it starts: + +``` +task 12 crew · open-ended · worker glm-5.3-flash (openrouter) · planner kimi-k3 · checker ⌖ kimi-k3 · est $0.121 +``` + +and when it lands, the same line with what it actually cost beside the estimate: + +``` +task 12 crew · open-ended · worker glm-5.3-flash (openrouter) · planner kimi-k3 · checker ⌖ kimi-k3 · $0.108 (est $0.121) · not right? /redo stronger +``` + +The estimate is what crews like this one cost: each seat's token profile for that kind of +work at the model's catalog prices, then moved by what this install's own paid tasks of the +kind cost against their estimates. A seat on a plan, a local model or a free pool adds +nothing to it. No crew estimated over the per-task limit is picked, `--best` included: the +pick is the best crew under it, and the line says `held under the $5.00 task limit` when that +changed it. + +A pin sends exactly the id you wrote when the catalog lists it. An id the catalog lists +only as a variant — a dated snapshot — is sent as that variant, and the line says which: +`checker ⌖ deepseek-v4-flash → -0731`. + +The first word is the kind of work the task was read as. The worker's provider is named +because it is where the money goes; the planner is named when it is another model than the +worker; a seat you pinned wears the pin mark `⌖`. A task that failed leads its line with +`failed — /redo stronger runs it again on a stronger crew`, and one that spent nothing +names no money. `/task --best` that changes +nothing says `best · already the strongest crew allowed`, and one that does names the rung +(`worker glm-5.3-flash → kimi-k3`). + +**A seat whose model cannot start moves, inside the task.** When a seat's first call is +refused, the seat goes down its ladder: the same model on its next route, then the next +model for the seat at a similar cost, then the model the last good crew here used, then the +model you are talking to — never a rung on an account that has just said it is out of +credit or refused its key. The line says so first — `running on fallback crew · worker +glm-5.3-flash → deepseek-v4-flash (credit unavailable on openrouter)` — and with nothing left +the task stops on the one thing to do, which leads its line in place of `/redo stronger` +(a stronger crew would meet the same wall): `failed — add credit on openrouter to +continue`, `reconnect openrouter with /connect`, or when the limit resets. What each route did is kept: a route +that refused a model is left out for a week, one at its limit rests until its reset, an +account out of credit is skipped until a paid call on it answers again. Free routes are +off unless you turn them on; when every paid route is out of reach they are used anyway, +and the line says `free routes in use (may log prompts)`. A rescue never takes a model +whose name says it was tuned for one domain (finance, medicine, law) or is too small for +a seat's work, and a seat tries at most three free pools in a task; a pool at its limit +is not waited on — the seat moves on at once, and with nothing left the task stops on +its action within seconds. A pool that answered at its limit is not asked again in that +task by any seat, and no helper call reaches a route that refused — it is refused before +it is sent. A model the catalog lists at no price that is not a free pool (a stealth or +preview model) is never picked unless you pin it, and neither is a model whose catalog row +carries too little to score it: with no index, rating, release date or lineage to read there +is nothing to rank it on but its price. The line says each seat's net move and +its first cause — `worker glm-5.3-flash → gemma-4-31b-it (credit unavailable on +openrouter; +3 tried)` — and the router's log keeps every rung. A seat that cannot start moves to a model at a similar cost before a +dearer one. A task that ran on an account out of credit and failed ends on the credit action, +not on `/redo stronger`. A free pool can be pinned by name — +`/crew pin worker vendor/model:free` — and picking a model's `free` route in a seat's +list pins that pool, not the paid route beside it. + +### Route health + +**What each route did is kept, and the next pick reads it.** Every seat's first call is +logged with how it ended, and routes are weighed by it: + +- a route that **refused a model** outright — no such model, not allowed on this key — is + left out for a week; +- a route **at its rate limit** rests until the reset it gave; +- a route that **fails often** is weighed at what those failures cost, so a cheaper route + that rarely answers can lose to a dearer one that does; +- an **account out of credit** or a **key refused** is skipped by helpers, and the next + task's first seat call asks it again, once: an answer puts it back at once; +- an **OpenRouter balance read as low** (*OpenRouter credits and free models*) counts as out + of credit before any call is made and is not asked again by a task: with no other paid + provider connected, the crew goes to free routes and the line says + `free routes in use (may log prompts) · credit unavailable on openrouter`. The balance is + read again at every launch while it is low, so a top-up puts routing back to ordinary. + +Helper calls — summaries, briefs, a landing's answer — never go to a route health says will +not answer: they are handed the router's own pick for the seat, or its rescue, instead. + +### Redo stronger + +`/redo stronger` runs the last task again ONE RUNG stronger: every seat keeps what it ran, +and the one seat whose next model up buys the most moves to that next model — never to the +top of the catalog in one step. The line says the rung it took (`checker glm-5.3-flash → +deepseek-v4-flash`); a second redo takes the next rung. It is one task's ask — your pins and +your allowed rule are untouched — and a crew already at the strongest the allowed models +make says so and starts nothing. A task still running cannot be redone; stop it first. + +**A task that never started is asked again, not escalated.** When no seat answered a single +call — the route refused it, the account was out of credit — nothing ran to be too weak, so +the redo takes the next-best models at the same cost. + +**It also teaches the crew.** A redo says the crew under-served that kind of work in that +repository, so the next task of the same kind there starts a rung higher — at most three +steps — and a step comes off after one accepted task of that kind, so work that was +under-served once is not overpaid for ever. A task counts as accepted when it finished and +its result was not refused: merged, kept on its branch, or an answer with nothing to land. + +Every decision and how it ended — accepted, redone stronger, or not kept — is written to +the router's log beside your settings, `router-events.jsonl`, which is what the panel's +recent tasks and today's spend are read from. + +### A profile from before the crew was picked per task + +Earlier builds set the crew with a preset word, a model family and a pick word. A profile +that still carries them is migrated once, on the first launch of this build, and told in +one line: + +``` +your crew is auto now · codeaf picks the worker, planner and checker for each task · /crew to see it +``` + +The preset and pick words, and a seat row that said `auto`, become auto. A seat holding +any model id stays pinned — even an id an old preset shipped, since nothing on the row +says which hand wrote it — and a profile with pins is told so instead +(`kept your pins: worker …, planner …, checker …`). A profile with nothing retired on it +is not touched. A family of +`open` becomes the allowed rule `open`; the default family needs no rule. + +### The roles under each row + +Directly under the **pinned roles** row, `/settings` → Providers lists **every registered +role**, grouped under the row answering it, saying which model comes out: + +| role | row | what it is | | --- | --- | --- | | `reflex` | reflex | reads every turn for memory — routing and keeping | | `title` | small work | the name a session gives itself | @@ -672,31 +864,31 @@ under the class answering it, saying which model comes out. As shipped: | `router` | small work | whether a turn should have been work | | `consolidate` | small work | tidies what is remembered while nobody is here | | `taskname` | small work | the two or three words a task is called | -| `auditor` | careful work | whether finished-looking work is actually finished | -| `vision` | careful work | reads images for a model that cannot see them | -| `shaper` | careful work | the brief a task you started yourself is given | -| `careful` | careful work | a part of a task that needs judgement | -| `repair` | careful work | the second go at work a check found gaps in | -| `planner` | mastermind | the plan that steers an adaptive run | -| `designer` | mastermind | writes and reviews a harness page | -| `routerconfirm` | mastermind | a second look before work starts itself | -| `markreader` | mastermind | what is left of an answer that is being taken out of your hands, drawn as parts | -| `handoff` | mastermind | the instruction a handed-over turn gives whoever finishes it | -| `division` | mastermind | the parts a worker hands its own work out in | +| `auditor` | checker | whether finished-looking work is actually finished | +| `vision` | checker | reads images for a model that cannot see them | +| `shaper` | checker | the brief a task you started yourself is given | +| `careful` | checker | a part of a task that needs judgement | +| `repair` | checker | the second go at work a check found gaps in | +| `planner` | planner | the plan that steers an adaptive run | +| `designer` | planner | writes and reviews a harness page | +| `routerconfirm` | planner | a second look before work starts itself | +| `markreader` | planner | what is left of an answer that is being taken out of your hands, drawn as parts | +| `handoff` | planner | the instruction a handed-over turn gives whoever finishes it | +| `division` | planner | the parts a worker hands its own work out in | The list is built from what is registered in the running binary, so it is the truth about this build rather than a table someone kept up to date. Stop on a row and the line under the -list is that role's own description followed by which class it follows. - -**What the mastermind's roles have in common is that one answer decides what all the other -calls do.** `planner` and `designer` used to sit on careful work beside the check on -finished work, which made one model id answer two unrelated bills: the careful calls are -many and short, and these are few. A planner that cuts badly spends a whole run on work nobody wanted; -a designer that writes badly puts a wrong answer on the menu with a name on it; -`routerconfirm` stands between a cheap model's "that should have been work" and a task -starting itself, and it is asked on nothing else, so it costs a call only where something was -about to be spent; `division` reads a task's parts before any of them exists, and every turn -every part ever takes runs on the brief it leaves behind. +list is that role's own description followed by which row it follows. A role on a crew seat +that has no task in front of it — a title, a check outside any task — is answered by the +seat's pin, or by the model the crew would pick for work of no particular kind. + +**What the planner's roles have in common is that one answer decides what all the other +calls do.** A planner that cuts badly spends a whole run on work nobody wanted; a designer +that writes badly puts a wrong answer on the menu with a name on it; `routerconfirm` stands +between a cheap model's "that should have been work" and a task starting itself, and it is +asked on nothing else, so it costs a call only where something was about to be spent; +`division` reads a task's parts before any of them exists, and every turn every part ever +takes runs on the brief it leaves behind. `markreader` and `handoff` are the two calls a long answer makes (*Tasks*). `markreader` is asked **at most once during an answer** — only when that answer can no longer work where it @@ -705,258 +897,92 @@ of any answer that touched a tool at all. It reads the account of the work and s left of your question. **It runs beside the work rather than stopping it**: the next step of the answer goes out immediately and the reading happens alongside it. What it draws is the list the work carries on with; the step it lands beside is stopped either way, because the -decision to stop was taken before it was asked. It used to be awaited, and a measured one held -the work for 8.1 seconds to decide nothing. **A long answer no longer buys one of these +decision to stop was taken before it was asked. **A long answer no longer buys one of these every ten rounds**: the two earlier moments cost no call at all now — codeaf tells the model what its answer has run up and the model decides for itself (*Tasks*, under *An answer that runs long is told*). `handoff` writes the instruction the task -opens on when an answer is handed over. Both sit on mastermind for the same measured reason: +opens on when an answer is handed over. Both sit on the planner for the same measured reason: a cheap model asked "is this finished" answered `(done)` about half-finished work 15 times out of 18, and that is the one answer that quietly drops a handover you were owed. There is no cheaper reading of that question — there is only a wrong one. **`careful` is not a call at all** — it is the model a *part* of a divided task runs on when -the worker graded that part careful (*Tasks*). It sits on careful work beside the audit for +the worker graded that part careful (*Tasks*). It sits on the checker beside the audit for the same reason: the failure it guards against is work that looks finished and is quietly wrong. -## I changed the crew but the model at the bottom did not change — why did my model not change - -That is right, and nothing is broken. **`/crew` does not change the model you are talking -to**, and the readout at the bottom of the frame is that model — the conversation's. The -only thing that moves it is `/model`, the model row in `/settings`, or naming one with -`/model `. The confirmation says so by name: `/crew max` ends -`· you are still talking to deepseek-v4-flash — /model changes that`, and `/status` prints -`model` and `crew` on neighbouring lines so the two dials read as two. - -The crew is a different dial: the five **classes** codeaf makes its own calls on — reflex, -small work, worker, careful work, mastermind — used for titles, memory, the safety gate, -the work inside every task, checks on finished work, the brief a task is shaped into, adaptive-run -planners and their nodes, harness pages, and looking at an image. Setting it writes all -five class rows in one write, and **it is live from that moment**: the next call codeaf -makes on its own uses the new crew, with no relaunch and no new session. A task already -running keeps the model it was admitted on. - -**Where to read the crew back:** - -- `/status` prints a `crew` line directly under `model`: - `crew max · brain claude-opus-5 · hands glm-5.3 · checks claude-fable-5.1`. The word is - the preset, or `custom` when the five classes are your own arrangement. **brain** is the - mastermind, **hands** is the worker, **checks** is the careful class. -- `/settings` → Providers has the **crew** row above the five class rows. -- Bare `/crew` opens the three presets with yours marked, under a `you talk to · ` - line naming the seat they do not touch. -- `/status` and the phone status sheet say `crew max` on their own line — the status row - itself stopped carrying the crew word on 2026-09-09; it is a setting, not a measurement. -- The hint line under the model picker says `crew max` beside its keys, so the picker you - opened looking for the change tells you the crew is a separate thing. - -**Tasks DO follow the crew, through its worker seat.** A plain task runs on the -`task.model` row if you set one, otherwise on the crew's **worker** class, and only when -that row is blank on the conversation's model. An **adaptive run** uses the classes the -same way: its planner takes the **mastermind** class and every node under it takes the -**worker** class. Those ids are settled once, when the task or run is admitted, so changing -the crew — or `/model` — half way through does not move work already going. +## I changed the crew but the model at the bottom did not change — does /crew change my chat model + +No, and nothing is broken. **`/crew` does not change the model you are talking to**, and +the readout at the bottom of the frame is that model — the conversation's. The only things +that move it are `/model`, the model row in `/settings`, or naming one with +`/model `. A pin confirms by naming the model you are still on: +`checker ⌖ moonshotai/kimi-k3 · every task until you unpin it · you are still talking to +deepseek-v4-flash — /model changes that`. + +`/status` prints `model` and `crew` on neighbouring lines so the two dials read as two: +`crew auto · codeaf picks the worker, planner and checker for each task`, or +`auto · pinned checker moonshotai/kimi-k3` when you pinned a seat. The phone status sheet +says the same, and the hint line under the model picker says `crew auto` beside its keys, +so the picker you opened looking for the change tells you the crew is a separate thing. +The one session with no `crew` line at all is a **remote** one opened with `--host`: that +crew lives on the other machine. ## Which model does a task run on — why did my task run on glm-5.3-flash and not my chat model -**The crew's worker seat**, unless you said otherwise. The ladder, first answer wins: +**The crew's worker**, unless you said otherwise. The ladder, first answer wins: 1. a model named in the ask — "do this on deepseek" — or picked on the proposal's chips; 2. the `task model` row under `/settings` → Tasks, when you have set one; -3. the crew's **worker** class — `hands` in the `/crew` confirmation and the `/status` - crew line; -4. the model you are talking to, only when the worker row is blank. - -So on the shipped `balanced` crew a task runs on `z-ai/glm-5.3-flash` whatever you are -chatting on, and `/crew max` moves the next task onto `z-ai/glm-5.3`. While a known -OpenRouter balance is low, an unwritten worker row uses the free worker model -instead; a crew or worker row you chose keeps its model. The task's row on the -roster, its room's status line and its finished card all name the model it actually ran -on. The worker of an adaptive run's nodes is the same seat, and so is the work model of -`codeaf do` — one row, every door. - -This is new: until the worker seat existed a task rode the model you were talking to, and -the crew moved everything about a task except its cost. - -## Does my crew reach codeaf do, or only this conversation — what models a headless run uses - -**It reaches both.** A crew you set here is the crew a run started from a script or a -terminal uses — `codeaf do`, `codeaf exec`, `codeaf run`, `codeaf plan new`, -`codeaf plan revise` and `codeaf plan run`. Set it once with `/crew frugal` and the same policy holds -whether the work is asked for here or run with nobody watching. - -Those runs seat two models, and each one is resolved the same way. The first of these that -answers wins: - -1. a model named on the command line: `--model` for the work, `--plan-model` for the - planning, and `--check-model` for the checks; -2. `CODEAF_MODEL`, `CODEAF_PLAN_MODEL`, or `CODEAF_CHECK_MODEL` in the environment. - Without its own pin, a check rides a plan seat pinned by flag or environment, - so a third model never arrives from the profile; -3. **your crew** — the planning seat takes the **mastermind** class, the work seat takes - the **worker** class, the same row a task handed off in conversation rides; -4. **your crew again, through an older class**, when your profile was set before a class - existed — the worker class inherits the small-work class it was split out of, and the - run says it did; -5. what the build ships with. - -So `--model` is one voice of four rather than the only one. This was not always true: until -recently a run outside the chat read only the flag and the environment, and a crew set here -was silently lost the moment the same brain ran from a script. - -Each of those runs opens by saying which voice answered, so nothing has to be guessed at: - -``` -models: work z-ai/glm-5.3-flash (crew frugal) · plan z-ai/glm-5.3-flash (crew frugal) -``` - -Two details worth knowing. A crew answers only once you have actually set one — a profile -nobody has touched takes the build's default rather than reading its own shipped values back -as a crew. And a class carrying a thinking level, like `kimi-k3:low`, carries it there too: -the run plans on that model at that level, the same as it does here. - -## Why does my run say inherited — my crew is older than the worker class - -The worker class arrived after the other four. A crew set before it exists on disk as four -classes with no worker among them, so a run has no worker row of its own to read. It does -**not** fall back to the shipped default: it takes the class the worker was split out of — -**small work**, the row that used to do this job — and it tells you, in one line under the -models line: - -``` -models: work deepseek/deepseek-v4-flash (crew custom, inherited) · plan deepseek/deepseek-v4-flash (crew custom) -your crew was set before the work seat existed · it is running on your small work seat's model until you pick a crew with /crew in the conversation -``` - -`inherited` beside the class means exactly that: **the model came from your crew, but from a -row you did not write.** The line is said once, when the run opens, and never again — not on -every call. - -The same thing happens in the conversation, where there is no models line to carry the word — -see "Why is my task running on a model I did not pick". - -To end it, set the crew again with `/crew frugal`, `/crew balanced` or `/crew max`, which -writes all five classes including the worker, or pin the worker row alone in `/settings` → -Providers. Either one, and the next run reads `crew frugal` with no second line. +3. the crew's **worker** — your pin, or the model the crew picked for this task; +4. the model you are talking to, only when nothing above answers. -A crew set with this build already pins every class, so this only ever appears on a profile -older than the class. And it is only for a row you never wrote: a row you **emptied on -purpose** means "follow the conversation", which a run outside the chat has no conversation -for, so that falls to the build's default the way it always has. +The task's crew line, its row on the roster, its room's status line and its finished card +all name the model it actually ran on. The seats are settled when the task starts, so +changing the crew — or `/model` — half way through does not move work already going. A +second task handed off while one is still running joins it and rides its crew. -## Why is my task running on a model I did not pick — inherited work seat in the conversation - -Tasks you hand off in a conversation run on the **worker** class, not on the model you are -talking to. If your crew was set before that class existed, you have no worker row — so the -work takes the class the worker was split out of, **small work**, and the thread tells you -once, the first time a task starts: - -``` -your crew was set before the work seat existed · it is running on your small work seat's model until you pick a crew with /crew in the conversation -``` - -**It is said once per session**, when work actually starts, and never per task or per part. -Twenty tasks in one sitting is one line. Start codeaf again tomorrow with the same profile and -you get it again — it is true until you answer it. - -`/crew` shows the same fact about the row itself, under the three presets: - -``` -your work seat is inherited from small work — picking one writes it -``` - -**To end it, pick any crew** — `/crew frugal`, `/crew balanced`, `/crew max`, or the chooser -that bare `/crew` opens. Every preset writes all five classes including the worker, so the -line stops on both surfaces at once. You can also pin the worker row on its own in -`/settings` → Providers. - -Three things this is **not**: - -- It is not the model you talk to. That one is on the status line and only `/model` moves it. -- It is not a row you emptied. A worker row you cleared on purpose means "follow the - conversation", and a task then rides the model you are talking to — that is an answer, and - nothing is said about it. -- It is not a fresh install. A profile that has never named any model runs this build's own - choice for each class, silently, the way it always has. - -The word `inherited` is the same word `codeaf do` prints beside the model on its `models:` -line, so the two surfaces are telling you about one thing. - -## What are the six models — the one you talk to and the five crew seats - -codeaf runs **six model seats**. **Seat one is the model you talk to**: it answers every -message you type, it is the id written above the message box, and `/model` is the only thing -that moves it. The other five are the **crew** — the models codeaf uses on its own behalf, -for calls you did not type: - -| seat | word | what it answers | -| --- | --- | --- | -| 1 | you talk to | your messages — set with `/model` | -| 2 | reflex | memory, titles, the safety gate — near-free, reads every turn | -| 3 | small work | digests, task names, the safety gate's yes-or-no — cheap | -| 4 | worker | every task you hand off, its parts, every run node — most of the bill | -| 5 | careful work | checks on finished work, the brief a task is shaped into, vision | -| 6 | mastermind | plans adaptive runs and designs harnesses — thinks | - -`/crew` shows all six and sets seats two to six in one word — `frugal`, `balanced` or -`max` — and never seat one. Bare `/crew` opens with `you talk to · ` above the three -presets, so the seat the presets do not touch is on the same page as the ones they do. -`/settings` → Providers pins any one of the five on its own, which turns the crew word to -`custom`. Seat one is the model named above the message box; the other five are the -`crew` line of `/status`. - -## Does /crew change my chat model — no, and what crew max on the status line means - -No. `/crew max` moves the five crew seats and leaves the model you talk to exactly where it -was. The confirmation names it: - -``` -crew → max · brain claude-opus-5 · hands glm-5.3 · checks claude-fable-5.1 · you are still talking to deepseek-v4-flash — /model changes that -``` +## Does my crew reach a run from the shell, or only this conversation — what models a headless run uses -`/model`, `/model ` or the model row in `/settings` are the only ways to change the -chat model, and `/crew` never offers to. The two dials also stay separate on the frame: the -chat model is written above the message box, and `crew max` — or `crew balanced`, -`crew frugal`, `crew custom` when you pinned a seat yourself — is a line of `/status` and of -the phone status sheet. It is not on the status row: a setting is not a measurement, and -the row is for numbers now. `/status` prints `model` and `crew` on neighbouring lines at any -width. The one session with no `crew` line at all is a **remote** one opened with -`--host`: that crew lives on the other machine. +**It reaches both.** The pins, the allowed rule and the cap you set here are the ones +`codeaf do`, `codeaf exec`, `codeaf run`, `codeaf plan new`, `codeaf plan revise` and +`codeaf plan run` use, and each of those runs is routed for its own task the same way. +*Running from the terminal* has the flags and the models line a run opens with. -## Asking a class to think harder — a level on a class value +## Asking a seat to think harder — a level on a pin -A class value may carry a thinking level as well as a model: +A pin, or any model row, may carry a thinking level as well as a model: ``` moonshotai/kimi-k3:high ``` -`:low`, `:medium` and `:high` are the three. No shipped crew preset adds one; a level travels only when you add it. +`:low`, `:medium` and `:high` are the three. Nothing codeaf picks adds one; a level travels only when you add it. The level is not part of the model id. It travels as its own request option, exactly as the picker's **ctrl+t** effort does, so the example sends model `moonshotai/kimi-k3` and asks for high thinking separately. -- **Any of the five class rows takes one**, though the mastermind is the one it is for. On - the worker row in a conversation it reaches the one-shot role calls only, never the work - inside a task. At a headless door — `codeaf do`, `exec`, `plan` or `run` — that row fills - a seat instead, and every request the seat sends carries its level. +- **Any pin and any model row takes one**, though the planner is the seat it is for: + `/crew pin planner moonshotai/kimi-k3:high`. On the worker in a conversation it reaches + the one-shot role calls only, never the work inside a task. At a headless door — + `codeaf do`, `exec`, `plan` or `run` — the pin fills a seat instead, and every request + the seat sends carries its level. - **Any other suffix is refused**, in words: *"off" is not a thinking level. Add `low`, `medium`, `high` to a model id, or leave the level off*. It is a different request shape — it asks the provider to suppress thinking outright — and some endpoints refuse it. `:max`, `:none`, `:xhigh` and the other near-misses are refused the same way. That refusal is about this notation alone — the effort ladder has rungs called `xhigh` and `max`, and they are a - separate thing from a suffix on a class value (see *Making the model think harder, deeper, + separate thing from a suffix on a pin (see *Making the model think harder, deeper, or less*). - Where a level is set, the role rows print it after the id, `kimi-k3:low`, which is the same notation the model picker and `/status` use. -The **mastermind** row is a text box because a picker hands back a bare id, while this row may -hold an id with a thinking instruction on it. +The **planner** row in `/settings` → Providers is a text box because a picker hands back a +bare id, while this row may hold an id with a thinking instruction on it. ## Why is my crew thinking at low — the pin is being ignored, effort=low in the log -A level written onto a class value is a pin, and it reaches the wire on **every** request the +A level written onto a pin is a pin too, and it reaches the wire on **every** request the seat that holds it sends — the conversation's one-shot role calls, and every call of a `codeaf do`, `exec`, `plan` or `run`. It is not a preference something further in gets to reconsider. @@ -1009,8 +1035,8 @@ next to the money it is spending: `planner: ` is the model amending the plan after every node — resolved once when the run started, from the model whatever started the run named, then the `planner` role's pin, -then the **mastermind** class, then the model you are talking to. Since the mastermind ships -with a model in it, this is usually *not* the model in the rest of this conversation, which +then the crew's **planner** — its pin, or the model the crew would pick — then the model you +are talking to. That is usually *not* the model in the rest of this conversation, which is why the run's own page says it rather than leaving you to work it out. You cannot name it yourself from a conversation, because a conversation cannot start a run at all — see *adaptive runs*. @@ -1021,7 +1047,7 @@ On a narrow screen (**under 60 columns**) the segment comes off that line and is on the first row of the page instead, above the chips. It is moved, not dropped — what the header sheds first is the goal, which you can still read in the conversation. -The nodes under the planner run on the `worker` role, which sits on the **worker** class — +The nodes under the planner run on the `worker` role, which sits on the crew's **worker** — a different model, and not on this line. `/settings` → Providers lists both. ## Pinning one role to its own model, and unpinning it @@ -1030,7 +1056,7 @@ In `/settings` → Providers, move onto any row of the roles list and press **en opens the model picker — the same one `/model` opens, same filter box, same ranking — and what you choose is **pinned** to that role alone. The row then reads ` pinned`, and the legend at the foot offers **del unpin**. Press -**del** on a pinned row to clear it; the role goes back to following its class. +**del** on a pinned row to clear it; the role goes back to following its row. The picker a role opens asks that role's own question. `vision` offers only models that can see; every other role offers the models you can hold a conversation with. @@ -1042,13 +1068,12 @@ and pinning from the list rewrites the row without disturbing the other pins in **A third door: just ask.** "Use `deepseek/deepseek-v4-pro` for planning and for designing harnesses" is a sentence codeaf acts on — it looks the row up with `settings` and writes it -with `change_setting`, into the same `models.roles` row, after asking you. The five class -rows (`models.tiers.reflex`, `models.tiers.low`, `models.tiers.worker`, `models.tiers.high`, -`models.tiers.mastermind`), the crew word (`models.crew`) and the pins are all writable that -way; only the role **slots** further down the Providers tab are not, because those are -bindings the running session holds rather than values in your profile. +with `change_setting`, into the same `models.roles` row, after asking you. The model rows +of the Providers tab — reflex, small work, and the crew's worker, checker and planner pins — +are writable that way too; only the role **slots** further down the Providers tab are not, +because those are bindings the running session holds rather than values in your profile. -So the ladder for any role, most specific first: **its pin**, then **its class's model**, +So the ladder for any role, most specific first: **its pin**, then **its row's model**, then **the model you are talking to**. Two things worth knowing: @@ -1082,11 +1107,11 @@ Until something has settled at all it says `nothing measured yet. Ratings appear once calls have been graded.` **What codeaf does with it** is one thing only: when a task splits itself, a part the -worker called ordinary work is minted on your **careful work** model instead if work +worker called ordinary work is minted on your **checker**'s model instead if work named like it has been turned down twice or more on the model the task is on. That is the -whole of it — no model is ever swapped out from under you, your chat model is untouched, -and an install with no crew classes set never lifts anything, because there is nowhere -dearer to lift it to. *Tasks*, under *When a task turns out to be too wide for one +whole of it — no model is ever swapped out from under you, and your chat model is +untouched. The crew learns from the same record in one more way, which you ask for: +`/redo stronger` (see *The crew*). *Tasks*, under *When a task turns out to be too wide for one worker*, has the rest. The record lives with your settings, in `router-ledger.json` and `router-events.jsonl`. @@ -1101,7 +1126,7 @@ small model that was down cost you the name and said nothing, while the model yo talking to sat there able to do it. Now the call **falls through one rung of the same ladder** and asks again: pin, then the -class's model, then the model you are talking to. That last rung is the floor, and it is a +row's model, then the model you are talking to. That last rung is the floor, and it is a model that demonstrably works — it is the one answering your own turns. The cost lands against the model that actually answered, not the one that refused, so `/cost` and the usage rows reconcile. @@ -1111,9 +1136,9 @@ would turn one bad minute at a provider into three charges and three waits for a nobody asked for. Nothing is said on screen either way — these are errands you did not ask for, and there is no state for "a small thing did not work". -**Each of these calls also has its own patience**, taken from its class rather than from a -per-call setting: a reflex call has **45s**, a cheap-class call **2m**, a capable-class one -**5m**, and a mastermind call **10m**. Some calls set something tighter still and keep it — +**Each of these calls also has its own patience**, taken from its row rather than from a +per-call setting: a reflex call has **45s**, a small-work call **2m**, a checker call +**5m**, and a planner call **10m**. Some calls set something tighter still and keep it — the guardian answers in ten seconds or not at all, the memory lookup moves to another machine after **two seconds** and stops after **eighteen**, and the two readers at the end of an answer get **20s** and **30s**. What this replaced was the ordinary @@ -1132,7 +1157,7 @@ correct behaviour and a surprise to anyone reading a bill, so this is the flag f where **one model has to answer for the whole run** — comparing two models against each other, timing a benchmark cell, or attributing a cost. -It settles four things on your model: the five crew classes, any role you pinned, the model +It settles four things on your model: the crew's three seats and the two small rows, any role you pinned, the model that work leaving the conversation runs on, and the fallback chain codeaf would otherwise move to when a model cannot answer. Under this flag **nothing hops** — not on a refusal, not on a reply that keeps stalling, not on rate limiting that will not clear — because a @@ -1142,12 +1167,12 @@ ordinary run falls back to the nearest same-class model, and this flag withholds **The two calls that ordinarily refuse the conversation's model ride it too.** The reader that decides whether a long answer is moved to a task, and the writer of the brief that task opens -on, normally run on the thinking tier and on nothing else: with no crew they are skipped +on, normally run on the crew's planner and on nothing else: with no planner they are skipped rather than handed to the model that just wrote the answer. Under this flag they run on your model like everything else, because you have said your model is the crew. Without the flag and -without a thinking-tier row, a move that needs them says `no second model is set`. +without a planner to seat, a move that needs them says `no second model is set`. -**It changes no setting and writes nothing.** Your crew rows and pins are untouched, `/crew` +**It changes no setting and writes nothing.** Your pins and rows are untouched, `/crew` still says what it said, and the next session without the flag reads them exactly as before. It is a posture for one run, not an edit. @@ -1164,28 +1189,18 @@ Two things it deliberately does not do: Standing items never take this posture, whatever the session that set them up was started with. They fire on their own clock long after your run ended, and the crew answers for them. -## What the screen says under `--one-model` — why does the status line say one model, where did my crew word go, no crew receipt when a task starts +## What the screen says under `--one-model` — why does the status line say one model, where did my crew word go, no crew line when a task starts **The crew line names the flag, because the flag is what seats the call.** Under -`--one-model` the crew line of `/status` and the phone sheet reads `one model` rather than the preset your four -rows derive to, and `/status` answers its crew line with `one model · every call rides the model +`--one-model` the crew line of `/status` and the phone sheet reads `one model` rather than +`auto`, and `/status` answers its crew line with `one model · every call rides the model you are talking to`. The model picker's hint slot and the welcome line under the wordmark say the same word. All of them read one answer, so none of them can disagree with another. -**Your crew word is not gone, it is overridden.** The rows are untouched on disk — this flag -writes nothing — and the next session started without it draws `crew balanced`, `crew max` or -`crew custom` again exactly as before. What the flag refuses to do is print a crew that is not -seating anything this run. - -**And no crew receipt is posted.** The line a profile older than the work seat ordinarily gets -when its first task starts — `your crew was set before the work seat existed · it is running on -your small work seat's model until you pick a crew with /crew in the conversation` — is not said -under this flag. That line -reports a substitution, and under the flag there is none: every call is already on the model -you are talking to, which is your own answer to the question it asks. Picking a crew would not -change what runs, so the sentence is not offered, and the `/crew` sheet says nothing about an -inherited work seat either. Without the flag, the same profile draws `crew custom` and says -that line once, exactly as it always did. +**Your crew is not gone, it is overridden.** Your pins are untouched on disk — this flag +writes nothing — and the next session started without it draws `crew auto` again exactly as +before. What the flag refuses to do is print a crew that is not seating anything this run, +so no task says a crew line under it either: nothing was picked. ## What temperature does codeaf use — sampling settings like temperature, top-p and seed @@ -1334,7 +1349,7 @@ Neither happens now. What still travels is what somebody asked for: a level you dialled, an explicit `--reasoning` level, `CODEAF_REASONING` and `CODEAF_EXEC_REASONING` at a headless -door, a crew class value like `moonshotai/kimi-k3:high`, and a rung on a task or +door, a pinned value like `moonshotai/kimi-k3:high`, and a rung on a task or a standing card. `off` on the headless environment settings really does send the disable; `off` on the chat dial is the legacy spelling of `auto`. Errands the session runs for itself — naming a conversation, judging a route — still ask for @@ -1368,7 +1383,7 @@ answer: it cannot think harder than its own ceiling. - A model whose catalog row says it takes no reasoning knob at all is sent nothing about thinking. -These rungs are not the same notation as a thinking level written onto a crew class value +These rungs are not the same notation as a thinking level written onto a pin (`moonshotai/kimi-k3:high`), which still takes only `low`, `medium` and `high`. ## The model went quiet, or stopped answering halfway through — the request is cut when nothing comes back, and how long it waits first @@ -2342,7 +2357,7 @@ same three-second beat every place runs on, and it draws three things: is bound to planning" are opposite facts about the same blank. There is no figure on that row: no line in the ledger names a slot, so there is nothing measured to put there. **Today only the conversation slot is drawn at all.** A window holds a client for the - model you are talking to and for no other; the five crew slots are answered where their + model you are talking to and for no other; the other slots are answered where their own session is opened, so this window cannot tell "nothing is bound" from "I cannot ask" — and the emptiness law says an unknown is drawn as nothing rather than guessed at; - **by topic** — the three things money is ever spent on, because the ledger holds @@ -2503,14 +2518,14 @@ their own words — `background`, `changes`, `spend`, `context`, `cache`, `rate` finally `file`. Labels are padded into two aligned columns. -The `crew` line sits directly under `model` and reads the preset word — or `custom` — with -the three classes after it: +The `crew` line sits directly under `model` and says the crew is auto, with any seat you +pinned after it: ``` -crew max · brain claude-opus-5 · hands glm-5.3 · checks claude-fable-5.1 +crew auto · pinned checker moonshotai/kimi-k3 ``` -On the live status line the crew is one short segment — `crew max`, or `crew custom` — at +On the live status line the crew is one short segment — `crew auto`, or `crew auto · 1 pinned` — at the head of the telemetry beside the model, and among the first a narrow row gives up; the `crew` line here and on the phone's status sheet is the full reading. A **remote** session opened with `--host` has no crew of its own to read — it is the other machine's — and gets @@ -2795,7 +2810,7 @@ They live on **one tab**: `/settings` → **Spending**, which `/budget` opens di | **per day** | `$500` | new work waits for midnight or for you to raise it here | | **per conversation** | `no limit` | this conversation stops starting new turns; the turn in flight always finishes | | **per plan** | `asks first above $100` | a planned job estimated above it quotes its step count and its price and waits for your go-ahead — it asks, it does not stop | -| **per task** | `no limit of its own` | nothing of its own; a task spends against the day and this conversation | +| **per task** | `$5 a task` | that task's next priced call is not made; set in `/crew` | | **per standing run** | `$5 a firing` | that one firing stops there; each order may name its own | | **practice** | `$50 of the day` | codeaf's practice on itself stops until tomorrow, and your own work is untouched | @@ -2936,30 +2951,24 @@ The row was called `ask before spending` when it lived on the Workspace tab, and setting key behind it is still `plan_consent_usd` — the panel's search matches the key as well as the label, so typing either finds it. -## What may a task spend — a task has no dollar limit of its own +## What may a task spend — $5 a task unless you set another -**A task carries no dollar cap of its own.** The Spending tab says so on the `per task` -row, in those words: `no limit of its own`, with the dim receipt `it spends against the -day and this conversation`. +**A task carries a dollar limit of its own: $5 unless you set another.** The Spending tab +shows it on the `per task` row — `$5 a task`, with the dim receipt `set in /crew · it also +spends against the day and this conversation`. It is set on the `/crew` panel's **cap** row +or with `/crew cap task 10`; see [The per-task limit](#the-per-task-limit). -That is not a missing feature — it is what the rail actually is. A task's own bounds are -**steps and time**, not money: a deadline it may renew, a step count, and a limit on how -long it may go without progress. The money it spends is counted against the day's limit -and against the limit on the conversation that started it, which are the two rows above it -on the same tab. +Every priced call of one task counts against it, and a call that would pass it is not +made. A task's other bounds are **steps and time**: a deadline it may renew, a step count, +and a limit on how long it may go without progress. Its money is also counted against the day's limit and +against the limit on the conversation that started it, the two rows above it on the same +tab. -**What you get instead of a per-task limit is seeing it happen.** The `$` on the status -line counts what the tasks are spending while they are spending it, and `/cost` splits that -figure into `conversation` and `tasks`. A task is bounded by the wallet and watched on the -row — it is never stopped on its own dollar count. +The `$` on the status line counts what the tasks are spending while they spend it, and +`/cost` splits that figure into `conversation` and `tasks`. -So **there is no per-task money row to edit**, and `/budget task 20` is not a shape this -command takes. Where you *can* put a figure on one piece of work is the **composer layer**: -`alt+enter` before you send a task, and its third line reads `it may spend up to $100.00 -before it asks`. Type a number there and that errand gets that ceiling — it stops before -its next turn once it reaches it, and any adaptive run it starts is held to a tank no -bigger. A task started any other way — `/task `, a proposal card, codeaf's own -hands — runs under the day's limit and this conversation's. +`/budget task 20` is not a shape this command takes; the per-task figure lives in `/crew`. +The **composer layer** can put a further figure on one errand; see the tasks page. `per standing run` beside it is the same kind of reading for a different reason: it reads `$5 a firing · each order may name its own`, because that rail is written **per standing @@ -3898,7 +3907,7 @@ lean. So an open-weight model with a large window is NOT lean. `glm-5.3-flash` and `glm-5.3` are served with 128,000 tokens of room, so they get the full page, the full tool list and saved memories, exactly like any other -128,000-token model — including when they are the model your crew preset picked +128,000-token model — including when they are the model the crew picked for the `worker` seat, and including when you then choose that same model in chat. Open weights are a licence, not a size. diff --git a/internal/manual/chat/openrouter-credits.md b/internal/manual/chat/openrouter-credits.md index 34058d7bd9..a1135e90eb 100644 --- a/internal/manual/chat/openrouter-credits.md +++ b/internal/manual/chat/openrouter-credits.md @@ -5,17 +5,17 @@ For the default OpenRouter service, codeaf reads the account's credit balance and the key's own spending cap, whichever is smaller. A known remaining balance of $0.50 or less is low. While it is low, a new conversation on a profile where you never -chose a model opens on `qwen/qwen3.8-27b:free`, and the five crew seats nobody set -use `nvidia/nemotron-3.5-lightning:free` for reflex, -`thinkingmachines/inkling-small:free` for small work, `qwen/qwen3.8-27b:free` for -worker, `thinkingmachines/inkling:free` for careful work and -`qwen/qwen3.8-27b:free` for mastermind. `/status` and the setup screen call that -crew `free`; `/crew` still offers `frugal`, `balanced` and `max`, and choosing one -writes it. +chose a model opens on `qwen/qwen3.8-27b:free`, and the two helper rows nobody set +use `nvidia/nemotron-3.5-lightning:free` for reflex and +`thinkingmachines/inkling-small:free` for small work. The crew's three seats — worker, +planner and checker — are picked per task, and a low balance reaches them as an +OpenRouter account out of credit: a seat nobody pinned is routed to a free pool, and the +task's crew line says `free routes in use (may log prompts) · credit unavailable on +openrouter` (*Route health* in *Models and cost*). Free ids are defaults only. A model you chose with `/model` or the setup screen, -`--model`, `CODEAF_MODEL`, a crew you applied or a tier row you set stays chosen, -and the free defaults are never written into your settings. Nothing is spent +`--model`, `CODEAF_MODEL`, a seat you pinned with `/crew pin` or a helper row you set stays +chosen, and the free defaults are never written into your settings. Nothing is spent reading the balance: it is two account lookups and no model call. OpenRouter limits free models to 20 requests per minute, and to 50 requests per @@ -61,8 +61,8 @@ A conversation that has sent nothing and whose model nobody chose follows the default both ways: onto the free model when the account reads low, and back to the usual default when a later read finds more than $0.50. A conversation that has sent a message keeps its model either way. After a top-up, the next new -conversation opens on the usual default and the crew seats nobody set return to the -`balanced` models. +conversation opens on the usual default, the helper rows return to their usual models, +and the crew is routed over paid routes again. ## Low on credits warning under the message box diff --git a/internal/manual/chat/permissions.md b/internal/manual/chat/permissions.md index 2922ae579b..3b6f2dde5f 100644 --- a/internal/manual/chat/permissions.md +++ b/internal/manual/chat/permissions.md @@ -925,10 +925,9 @@ conversation's own model (`model.talk`) is changed with `/model`, and the other role slots — `model.plan`, `model.work`, `model.verify`, `model.scribe` — are bindings the running session holds rather than values in your profile, so neither `change_setting` nor the panel can write them. To send codeaf's own -auxiliary calls somewhere, set one of the five crew classes -(`models.tiers.reflex`, `models.tiers.low`, `models.tiers.worker`, -`models.tiers.high`, `models.tiers.mastermind`), set all five at once with -`models.crew`, or pin one role in `models.roles`. +auxiliary calls somewhere, set one of the five model rows on the Providers tab — +reflex, small work, worker, checker, planner — or pin one role in `models.roles`. A +task's worker, planner and checker are pinned with `/crew pin`. A row your environment has pinned refuses like it does everywhere else: `