feat: options allow overrides

[constantinius]

Add support for object-style option values in framework config.json with per-value config overrides
Option values can now be a plain string (existing behavior) or { "value": "...", "overrides": {...} } to override modelOverrides, skip, or toolNameMapping for that specific option combination
Overrides are deep-merged into the framework config during test matrix expansion

Example

{
  "options": {
    "modelSetup": [
      "single",
      {
        "value": "fallback",
        "overrides": {
          "modelOverrides": { "request": "some-other-model" }
        }
      }
    ]
  }
}

The "single" variant uses the framework's default config. The "fallback" variant overrides modelOverrides.request for all checks in that test run.

Motivation
Some framework variants need different validation expectations (e.g., a different expected model name, different checks to skip, or different tool name mappings). Previously this required duplicating the entire framework folder. With option overrides, a single config.json can express variant-specific config inline.

[sentry]

Bug: Option overrides for skip and toolNameMapping are shallow-merged, causing them to completely replace base configurations instead of merging with them.
_{Severity: MEDIUM}

Suggested Fix

Explicitly deep-merge the skip and toolNameMapping properties from optionOverrides into the framework configuration, similar to how modelOverrides is currently handled. This will ensure that nested values from both the base and override configurations are combined correctly.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: src/orchestrator.ts#L667

Potential issue: When applying option-specific overrides to a framework configuration,
the `skip` and `toolNameMapping` properties are shallow-merged via a spread operator.
This is inconsistent with `modelOverrides`, which is explicitly deep-merged. As a
result, if a base framework config and an option override both define `skip` or
`toolNameMapping` objects, the override's object will completely replace the base
object, leading to the loss of pre-existing rules from the base configuration. This
behavior contradicts the documented intent to deep-merge overrides.

_{Did we get this right? 👍 / 👎 to inform future reviews.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

[cursor]

Shallow override of skip and toolNameMapping loses framework config

Medium Severity

The ...optionOverrides spread on the framework object shallow-replaces skip and toolNameMapping, while modelOverrides gets explicit deep-merge treatment. If an option override specifies a partial skip (e.g., only checks), the framework's existing skip.tests entries are silently lost. Similarly, if an override specifies one toolNameMapping entry, all other framework-level tool name mappings disappear. The same shallow-replacement issue exists in getOptionCombinations when combining overrides across multiple option dimensions.

Additional Locations (1)

• src/orchestrator.ts#L889-L900

 

[github-actions]

🟡 AI SDK Integration Test Results

Status: 286 tests failing (no regressions)

Summary

Metric	main	PR	Change
Total Tests	530	530	—
Passed	231	231	—
Failed	286	286	—

Test Matrix

Agent Tests

SDK	Basic Agent Test	Conversation ID Agent Test	Long Input Agent Test	Tool Call Agent Test	Tool Error Agent Test	Vision Agent Test
browser/langgraph	❌blk, combined ❌blk, compiled ❌blk, custom-state ❌blk, graph ❌blk, langchain ❌str, combined ❌str, compiled ❌str, custom-state ❌str, graph ❌str, langchain	❌blk, combined ❌blk, compiled ❌blk, custom-state ❌blk, graph ❌blk, langchain ❌str, combined ❌str, compiled ❌str, custom-state ❌str, graph ❌str, langchain	❌blk, combined ❌blk, compiled ❌blk, custom-state ❌blk, graph ❌blk, langchain ❌str, combined ❌str, compiled ❌str, custom-state ❌str, graph ❌str, langchain	❌blk, combined ❌blk, compiled ❌blk, custom-state ❌blk, graph ❌blk, langchain ❌str, combined ❌str, compiled ❌str, custom-state ❌str, graph ❌str, langchain	❌blk, combined ❌blk, compiled ❌blk, custom-state ❌blk, graph ❌blk, langchain ❌str, combined ❌str, compiled ❌str, custom-state ❌str, graph ❌str, langchain	❌blk, combined ❌blk, compiled ❌blk, custom-state ❌blk, graph ❌blk, langchain ❌str, combined ❌str, compiled ❌str, custom-state ❌str, graph ❌str, langchain
cloudflare/langgraph	❌	❌	❌	❌	❌	❌
cloudflare/vercel	❌	❌	❌	❌	❌	❌
nextjs/mastra	✅	❌	—	✅	✅	✅
nextjs/vercel	✅blk ✅str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str
node/langgraph	❌	❌	❌	❌	❌	❌
node/manual	✅	❌	✅	✅	✅	✅
node/mastra	❌	❌	❌	❌	❌	❌
node/vercel	✅	❌	❌	❌	❌	❌
php/laravel	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str
python/langgraph	❌a ❌s	❌a ❌s	❌a ❌s	❌a ❌s	❌a ❌s	❌a ❌s
python/manual	✅a ✅s	✅a ✅s	✅a ✅s	✅a ✅s	✅a ✅s	✅a ✅s
python/openai-agents	✅	✅	❌	✅	✅	❌
python/pydantic-ai	✅	✅	❌	❌	❌	❌

Embedding Tests

SDK	Basic Embeddings Test
browser/google-genai	❌
browser/langchain	❌
browser/openai	✅
cloudflare/google-genai	❌
cloudflare/langchain	❌
cloudflare/openai	✅
cloudflare/vercel	❌
nextjs/google-genai	❌
nextjs/langchain	❌
nextjs/openai	✅
nextjs/vercel	❌
node/google-genai	❌
node/langchain	❌
node/openai	✅
node/vercel	❌
php/laravel	❌
python/google-genai	❌a, blk ❌s, blk
python/langchain	❌a, blk ❌s, blk
python/litellm	❌a, blk ✅s, blk
python/manual	✅a, blk ✅s, blk
python/openai	✅a, blk ✅s, blk

LLM Tests

SDK	Basic Error LLM Test	Basic LLM Test	Conversation ID LLM Test	Long Input LLM Test	Multi-Turn LLM Test	Vision LLM Test
browser/anthropic	✅blk ❌str	✅blk ✅str	❌blk ❌str	✅blk ✅str	✅blk ✅str	✅blk ✅str
browser/google-genai	✅blk ✅str	✅blk ✅str	❌blk ❌str	❌blk ❌str	❌blk ❌str	✅blk ✅str
browser/langchain	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str
browser/openai	✅blk ✅str	✅blk ❌str	❌blk ❌str	✅blk ❌str	✅blk ❌str	✅blk ❌str
cloudflare/anthropic	✅blk ❌str	✅blk ✅str	❌blk ❌str	✅blk ✅str	✅blk ✅str	✅blk ✅str
cloudflare/google-genai	✅blk ✅str	✅blk ✅str	❌blk ❌str	✅blk ✅str	✅blk ✅str	✅blk ✅str
cloudflare/langchain	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str
cloudflare/openai	✅blk ✅str	✅blk ✅str	❌blk ❌str	✅blk ✅str	✅blk ✅str	✅blk ✅str
nextjs/anthropic	✅blk ✅str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str
nextjs/google-genai	✅blk ✅str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str
nextjs/langchain	✅blk ✅str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str
nextjs/openai	✅blk ✅str	✅blk ❌str	✅blk ❌str	✅blk ❌str	✅blk ❌str	✅blk ❌str
node/anthropic	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str
node/google-genai	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str
node/langchain	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str
node/manual	—	✅	✅	✅	✅	✅
node/openai	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str
python/anthropic	✅a, blk ❌a, str ✅s, blk ❌s, str	✅a, blk ❌a, str ✅s, blk ❌s, str	✅a, blk ❌a, str ✅s, blk ❌s, str	✅a, blk ❌a, str ✅s, blk ❌s, str	✅a, blk ❌a, str ✅s, blk ❌s, str	✅a, blk ❌a, str ✅s, blk ❌s, str
python/cohere	❌s, blk, v1 ❌s, blk, v2 ❌s, str, v1 ❌s, str, v2	❌s, blk, v1 ❌s, blk, v2 ❌s, str, v1 ❌s, str, v2	❌s, blk, v1 ❌s, blk, v2 ❌s, str, v1 ❌s, str, v2	—	❌s, blk, v1 ❌s, blk, v2 ❌s, str, v1 ❌s, str, v2	—
python/google-genai	✅a, blk ✅a, str ✅s, blk ✅s, str	✅a, blk ✅a, str ✅s, blk ✅s, str	✅a, blk ✅a, str ✅s, blk ✅s, str	✅a, blk ✅a, str ✅s, blk ✅s, str	❌a, blk ❌a, str ❌s, blk ❌s, str	❌a, blk ❌a, str ❌s, blk ❌s, str
python/langchain	❌a, blk ❌a, str ❌s, blk ❌s, str	✅a, blk ✅a, str ✅s, blk ✅s, str	✅a, blk ✅a, str ✅s, blk ✅s, str	✅a, blk ✅a, str ✅s, blk ✅s, str	✅a, blk ✅a, str ✅s, blk ✅s, str	✅a, blk ✅a, str ✅s, blk ✅s, str
python/litellm	❌a, blk ❌a, str ❌s, blk ❌s, str	❌a, blk ❌a, str ✅s, blk ✅s, str	❌a, blk ❌a, str ✅s, blk ✅s, str	❌a, blk ❌a, str ✅s, blk ✅s, str	❌a, blk ❌a, str ✅s, blk ✅s, str	❌a, blk ❌a, str ✅s, blk ✅s, str
python/manual	—	✅a, blk ✅s, blk	✅a, blk ✅s, blk	✅a, blk ✅s, blk	✅a, blk ✅s, blk	✅a, blk ✅s, blk
python/openai	❌a, blk ❌a, str ❌s, blk ❌s, str	✅a, blk ❌a, str ✅s, blk ❌s, str	✅a, blk ❌a, str ✅s, blk ❌s, str	✅a, blk ❌a, str ✅s, blk ❌s, str	✅a, blk ❌a, str ✅s, blk ❌s, str	❌a, blk ❌a, str ❌s, blk ❌s, str

Legend: ✅ Pass | ❌ Fail | ✅🔧 Fixed | ❌📉 Regressed | ✅🆕 New (pass) | ❌🆕 New (fail) | 🗑️ Removed | _str=streaming _blk=blocking _a=async _s=sync _hi=highlevel _lo=lowlevel

---

Generated by AI SDK Integration Tests