A misspelled or unrecognised judge model silently becomes an OpenAI judge instead of failing

[rajarshidattapy]

Description

getModelConfig ends with an unconditional fallback:

// src/utils/models.ts:306-316
// Default fallback
return {
  id: alias,
  provider: "openai",
  displayName: alias,
  supportsTemperature: true,
  ...
}

The orchestrator derives the judge provider from that return value:

// src/orchestrator/index.ts:90-91
const judgeModelInfo = resolveModel(judgeModel)
const judgeName = judgeModelInfo.provider as JudgeName

So any string that is not in MODEL_CONFIGS and does not start with gpt-/o1/o3/o4/claude-/gemini- is routed to OpenAI with the typo'd string used verbatim as the model ID.

Concrete cases:

• --judge sonnet-4-5 (hyphen instead of dot; the registry key is sonnet-4.5) → OpenAI judge, model "sonnet-4-5".
• --judge claude-opus-4.5 typo'd as opus4.5 → OpenAI judge.
• --judge gemini2.5-pro → OpenAI judge.
• Any future Anthropic alias not yet in the registry and not prefixed claude-.

The failure mode is then one of:

• OPENAI_API_KEY is unset → getJudgeConfig("openai") returns an empty key and the run dies deep in the evaluate phase with an OpenAI 401, pointing the user at the wrong provider entirely.
• OPENAI_API_KEY is set → a 404 "model not found" from OpenAI for a model name the user believes is an Anthropic model.

Neither message mentions that the alias was unrecognised.

Related: case handling is inconsistent

getModelConfig lowercases for the registry lookup but tests the original casing in every prefix branch:

const lowerAlias = alias.toLowerCase()
if (MODEL_CONFIGS[lowerAlias]) return MODEL_CONFIGS[lowerAlias]
if (alias.startsWith("gpt-5") || ...) { ... }     // alias, not lowerAlias

So GPT-4.5 misses both the registry and the gpt- prefix branch and lands in the default fallback with supportsTemperature: true — wrong for a reasoning model. Claude-sonnet-4-6 behaves the same way.

Impact

Silent misrouting in the component that decides every score in the benchmark. A run can complete and publish results attributed to a judge that never ran.

Suggested fix

• Use lowerAlias consistently in the prefix checks.
• Replace the final fallback with a thrown error listing listAvailableModels(), or at minimum a loud logger.warn naming the assumed provider. Failing fast at CLI-parse time is much cheaper than failing in the evaluate phase after a full ingest.