Repository navigation
feat: add opt-in SystemOne query model routing - #1218
Conversation
Route simple requests to manually listed low-cost models while preserving the Agent's configured model as the strong tier. Configuration is opt-in through a query_routing block in config.yaml or the Agent query_routing argument. Added conservative Decision-based classification, capability gates, ordered model selection, session upgrade stickiness, shadow mode, strong-model fallback, multilingual documentation, and unit/E2E coverage.
Exercise the real Responses API path with session pinning, high-stakes routing, shadow mode, and Decision failure fallback. Add exact threshold and context-limit boundary tests.
Resolve one concrete model before ADK preprocessing and preserve that binding through tool loops, nested AgentTools and native sub-agent transfers. Reclassify the next user turn instead of retaining session-level strong-model stickiness. Rebuild Responses chains from complete history when targets or protocols change, validate continuation prefixes, and retain full replay for expired response IDs. Support independent model connections and reject transport overrides or fallbacks that would violate the fixed-turn contract. Keep routing opt-in with shadow mode as the default and document both configuration and compatibility boundaries. Validation: - Post-sync regression: 8638 passed, 57 skipped, 4 expected failures. - All-files pre-commit: lint, format, secret scans and dependency policy passed. - Real SystemOne/model E2E: 8 passed. - Changed executable line coverage against origin/main: 98.40% (798/811). - Documentation type checks and production build passed. Live E2E uses stored Responses continuation with server-side caching disabled; cache-service entitlement and production routing quality/cost calibration remain operator-specific. No default behavior changes for unconfigured Agents.
Validate tool arguments with bounded correction under enforce routing, preserve Responses tool-call ordering across models, and clarify candidate capabilities. Add paired public-benchmark adapters, transport usage accounting, recovery ledgers and native scorer regression coverage. Off and shadow behavior remain unchanged. Verification: 8719-test full regression, 638-test affected suite, 98.67% changed production-line coverage, native adapter checks and real ADK HTTP campaigns; model quality and provider limits remain task-dependent.
Expose immutable DecisionConfig at the shared decision layer and Agent root. Remove nested consumer connection fields, former imports, and consumer-specific connection environment variables. Keep routing and content-safety policy and client state independent. Migrate both documentation languages and callers; add secret-safe rejection and credential-boundary regressions. Extend real HTTP evaluation with matched fixed controls and frozen routing policies for independent calibration.
Preserve cloud-drive create and append timestamps during offline replay so native scoring can verify the executed results exactly. Keep upstream utility/security checks and strict result comparison intact; no provider-specific changes. Verification: regression observed red then green; pinned native suite 15 passed; changed scorer lines 11/11 covered; full regression 8750 passed, 91 skipped, 4 xfailed; pre-commit passed. Evaluation-only change, with no public SDK contract impact.
Default complex reasoning, tool orchestration and long-context thresholds to 0.5. Keep high stakes at 0.05, explicit overrides authoritative, and custom-question defaults unchanged. Routing remains opt-in and shadow mode retains strong-model execution. Update both documentation languages and exercise the new boundaries and isolated overrides. Validation: 478 targeted tests; 8756 regression tests passed, 91 skipped, 4 xfailed; configuration module coverage 100%; pre-commit and docs checks passed. Live HTTP acceptance initially had 18 passes and 2 SystemOne timeout failures; both affected cases passed separate follow-up checks, with original timeout records retained.
cuericlee
left a comment
There was a problem hiding this comment.
Review: opt-in SystemOne query model routing (d629698)
Reviewed the full production surface (veadk/routing/*, veadk/decision/*, veadk/configs/routing_configs.py, Agent integration, content-safety refactor, Ark Responses replay) and the contract tests. CI is green (25 checks), test coverage of the routing/decision modules is thorough, and the core design is sound: one decision per root turn persisted into session state, conservative fail-closed classification, explicit capability gates, immutable/secret-aware config, and target-aware portable replay that strips foreign thought_signatures. The nested-AgentTool/transfer handling via contextvars + RoutedLlmAgent views is well thought through, and the cancellation/accounting tests cover the tricky paths.
Comments below, ordered by importance. None are demonstrated runtime defects against the current test suite — #1 is a release-compatibility issue, #2–#4 are design questions, the rest nits.
1. Removing the released content_safety.decision block fails hard with no migration path
The nested tool.content_safety.decision.{api_url,api_key,model} block and the TOOL_CONTENT_SAFETY_DECISION_* environment variables exist in a published release (present in tag 1.1.27). After this PR, ContentSafetyConfig keeps extra="forbid", so an application still shipping the old block gets a raw "extra fields not permitted" startup error, and the old env vars are silently ignored. The PR description says "no compatibility aliases are retained," but users upgrading from 1.1.21–1.1.27 get neither an alias nor a pointer to the replacement (decision root block).
Suggested, in order of preference:
- Accept (and deprecation-warning on) the old block/env for one release, mapping them onto
DecisionConfig; or - At minimum, detect the legacy keys and raise an error that names the new location (e.g. "
tool.content_safety.decisionmoved to the rootdecisionblock"), and add an upgrade note in bothguardrail*.mdxlanguages. Right now the migration is documented only in this PR body.
2. Circuit breaker is dead code; cheap-tier outages fail the user turn with no protection on later turns
ModelRegistry.mark_failed() / unhealthy_until have no callers anywhere in veadk/ or tests/, and test_selected_model_failure_never_changes_model explicitly asserts that a failing selected model propagates instead of falling back to strong. That fixed-model-per-turn choice is correct mid-turn, but the asymmetry is worth a conscious decision:
- classification-time decision/timeout/capability failure → strong;
- execution-time 5xx / outage of the cheap model → exception to the user, and because
mark_failedis never called, every subsequent turn keeps selecting the same broken model until manual config change.
Either wire mark_failed from the execution error path (so at least later turns route around a degraded tier) and keep the cooldown, or remove the breaker fields/method to avoid implying protection that doesn't exist, and state the runtime-error behavior in model*.mdx alongside the other listed limits.
3. Session-state decision record is keyed globally, not per router
_RECORD = "veadk:query_routing:turn" is written to session state and resumed whenever record["invocation_id"] == ctx.invocation_id (veadk/routing/execution.py:46, :272-289). The covered transfer test is root→owned sub-view, where one router owns the turn. Consider a session that reaches a second, independently configured routed agent within the same invocation via native agent transfer (sibling agent, not a sub-agent/AgentTool of the owner): routing_owner resolves to that agent itself, _select finds the first router's record, and fails with "routing configuration changed during a resumed turn" even though that agent has not changed anything — or, if configuration_id happens to match, silently adopts the first router's model_name.
If that topology is supported, namespace the record by router owner/config (e.g. key the state delta by configuration_id or agent name); if it is intentionally out of scope (like live bidirectional execution), please add it to the documented limitations and ideally a guard with a precise error.
4. Tool-argument validation can now break strong-model turns it didn't touch before
In enforce mode validate_tool_arguments runs even when the route is strong. Two failure modes are new for users who enabled routing but never went to a cheap model:
_schema()raisesValueErrorfor any non-local$ref/$dynamicRef(tool_arguments.py:70-72), uncaught, while building the schema for the tool being invoked. A single tool whose schema the validator can't handle kills the whole turn on the strong model, where pre-routing execution worked. Consider failing open for validator-infrastructure problems (log + skip that invocation), reserving the hard failure for genuine argument errors.- Strict Draft 2020-12 checking with only integer-string normalization may reject argument shapes that providers/tools accepted before (floats-as-strings, union quirks). The two-correction budget is per whole root turn across all tools, so a few noisy tools can exhaust it even on
strong.
Worth a test with a strong-only route (shadow=False, judge returns strong) plus a remote-$ref/loose-schema tool to pin the intended behavior.
5. Decision HTTP client is closed after every (sub-)turn
In _scope, when the outermost scope exits with active_runs == 0 it calls classifier.aclose() (execution.py:318-321). ADK re-enters _run_async_impl for each continuation after tool execution; the scope wraps each entry, so a single user turn with two LLM steps closes the pooled httpx.AsyncClient between steps and lazily rebuilds it — defeating the keep-alive pool that DecisionClient was designed to reuse. Consider tying client lifetime to the RouterLlm/Agent (an explicit shutdown hook) or refcounting by invocation rather than by _run_async_impl entry.
Nits / questions
RouteStore.get_or_createcoalescing/TTL looks unreachable in production: the cache key includesinvocation_id, nested agents inheritTurnContextinstead of calling the store, and replay goes through the session record (execution.py:279). If only the in-flight future matters, the store could be much smaller; if cross-invocation reuse was intended, the key defeats it.set_route_selection/get_route_selectionand the direct path inRoutingLlm.generate_content_asyncappear unused outside tests. If that path is ever reached it bypasses portable replay and the capability re-check; consider making the fallback explicit or removing it.routing_requeststores tools undermember.name + "." + tool.nameintools_dictbut appends declarations with their unqualified function names toconfig.tools; the judge therefore sees[available_tools]names that don't match the schemas. Intentional for the classifier prompt?QueryStateBuilder.build_for_routingrendersconfig.tools/response_schemavia barestr(value)on GenAI typed objects — proto/pydantic reprs are noisy and not stable across SDK versions;model_dump/protobuf text format would be a steadier judge input.- Minor:
target_idincludes all of_additional_argsin the hash. It's one-way, but callers can put custom headers there; hashing only the fields that actually affect routing would be safer and less collision-prone across harmless arg changes.
Verification note
I did not re-run the suite locally; this is static review against d629698c plus the stated CI results (all green, including test 3.10/3.12, live tests skipped in ordinary CI). The live E2E covers Responses chaining across model switches, which is the riskiest transport change; the text_before_tool_calls reordering in ark_llm.py now affects every enforce-mode request including strong-route turns, so it would be worth one explicit non-routed-vs-routed strong-route payload diff test if one doesn't already exist.
Keep the legacy content_safety.decision block and TOOL_CONTENT_SAFETY_DECISION_* variables working with deprecation warnings. A root decision block or any DECISION_* override takes precedence, and the two locations are never merged. Wire simple-model failures into the 60-second cooldown through the ADK model error callback, so the current turn still propagates the error while later turns select the strong model. Namespace persisted turn records by router owner so independently configured routed agents in one invocation no longer overwrite each other. The record still carries the configuration hash, so a resumed turn with changed configuration fails explicitly. Skip tool argument validation for strong, shadow, and disabled routing, and fail open on validator infrastructure errors such as non-local schema references. Genuine invalid arguments still return a structured error. Close the decision client with the Agent/Runner lifecycle instead of after every ADK continuation. Drop the unused cross-invocation decision cache, route selection helpers, and active-run tracking; align judge tool names with the model declarations; and render routing state as stable JSON with accurate size accounting.
cuericlee
left a comment
There was a problem hiding this comment.
Re-review of 3878616b (review-feedback fixes)
Re-reviewed the follow-up commit against the five findings from the previous pass. All five are addressed, and so are the nits; each fix comes with focused tests and bilingual doc updates.
Verified resolved
- Released content-safety config migration —
legacy_decisionaccepts the old nesteddecisionblock via alias with a deprecation warning, the releasedTOOL_CONTENT_SAFETY_DECISION_*env vars are mapped with a warning, and a rootdecisionblock (or anyDECISION_*var) takes precedence with no merging (has_explicit_decision_config). Factory precedence is explicit: Agent-supplieddecision→ legacy block (only when root is absent) → root loader. Migration callout added to bothguardrail*.mdxlanguages. Tests cover YAML, env, precedence, and the from_dict bypass. - Circuit breaker wired —
canonical_on_model_error_callbacksis a real google-adk 2.7 hook (verified the framework invokes callbacks by keyword inside the same flow task, and re-raises when they returnNone).mark_model_failuremarks only the failed simple tier for the existing 60s cooldown; strong-route/shadow errors and the pre-call capabilityValueErrorcorrectly do not trip it, and the current turn still propagates. The updated contract test asserts the second turn skips both the broken tier and a new classification. - Per-router decision records — session state is now namespaced as
veadk:query_routing:turn:<owner-agent-name>on both write and resume, with the same-router config-hash guard retained; covered by a two-router test. - Tool-argument validation scope — strong-route turns skip validation entirely, and validator-infrastructure failures (schema build, non-local refs) fail open with a warning instead of failing the turn. Docs updated.
- Decision client lifetime — no longer closed per
_run_async_implentry; closed once viaRoutingLlm.aclose← newAgent.aclose←Runner.close(which still calls ADK'ssuper().close()infinally). The cancellation test now asserts the client survives the turn and closes at runner shutdown.
Nits also handled: store reduced to in-flight coalescing with matching tests; the unused route-selection ContextVar removed; the RoutingLlm direct path is now strong-only with a deep-copied request (tests pin both properties); judge-facing tool declarations now carry the same qualified names as tools_dict; judge state renders typed objects via stable JSON model_dump instead of str(); target_id only hashes transport-identity args.
Remaining (non-blocking)
- The released symbols
veadk.content_safety.DecisionConfigandveadk.content_safety.DEFAULT_DECISION_MODEL(present in 1.1.21–1.1.27) are still not re-exported — only the YAML/env compatibility was restored. Code doingfrom veadk.content_safety import DecisionConfigstill breaks on upgrade; consider a one-release shim re-export with aDeprecationWarning, or state the import-path removal in the release notes alongside the YAML migration. - The deprecation warning for
legacy_decisionfires regardless ofprovider; it only had an effect withprovider=decision. Harmless, just possibly noisy for a dormant block — gate it on the provider if you want strict signal. mark_model_failuretreats every model exception (including request-content 4xx) as a tier outage and cools the model down process-wide for 60s. Effect is only extra strong-model traffic/cost, so acceptable, but distinguishing transport/5xx from 4xx would make the breaker more precise later.- The per-router state key uses the agent name; this relies on agent names being unique within an app, which ADK already effectively requires for session state. Worth keeping if that invariant ever changes.
CI on 3878616b was still pending at review time (26 jobs queued/running); I did not re-run the suite locally — static verification of the diff, google-adk 2.7 hooks, and the added/updated tests only.
No blocking issues from my side; the remaining items can land as follow-ups.
Keep the released veadk.content_safety DecisionConfig and DEFAULT_DECISION_MODEL imports working for one release through a DeprecationWarning shim; the canonical location is veadk.decision. Emit legacy content-safety connection warnings only for provider=decision, the only provider that consumes the deprecated block or environment variables. Other providers ignore both without noise. Do not open the simple-model cooldown for request-specific 4xx responses. 408 and 429 stay transient, as do 5xx and errors without a status code, so the breaker still covers tier outages without shifting next rounds to the strong model for a bad request. Document the new behavior and the ADK unique-agent-name assumption behind the per-router record key.
cuericlee
left a comment
There was a problem hiding this comment.
Re-review of 6a457a55 (compatibility + breaker refinements)
The three remaining items from the previous pass are addressed:
- Import shim — module-level
__getattr__inveadk/content_safety/__init__.pyrestores the releasedDecisionConfigandDEFAULT_DECISION_MODELimports for one release with aDeprecationWarningpointing atveadk.decision; covered by a test that also asserts the symbol is gone fromcontent_safety.config. Migration callout updated in both doc languages. - Warning noise — the legacy-block warning and the legacy-env application now gate on
provider=decision; inert configs stay silent, with tests for both paths. - Breaker precision — 4xx classification added (
408/429transient, other 4xx non-circuit-opening, unknown/no-code failures keep the conservative behavior), documented bilingually, and the per-router record key now carries the ADK agent-name-uniqueness rationale.
One functional gap in the breaker refinement:
The 4xx filter only matches the test double, not production exception shapes
_opens_circuit reads getattr(error, "code", None), but neither routed-model adapter puts the HTTP status there:
- RetryingLiteLlm (the default chat-completions cheap tier) raises litellm exceptions (
litellm/exceptions.py, pinnedlitellm>=1.83.7,<=1.83.14): HTTP status isstatus_code;codeis only set onRateLimitErrorand is not the status. Alitellm.BadRequestErrorhasstatus_code=400,code=None. - ArkLlm Responses raises
volcenginesdkarkruntimeexceptions (openai-fork):ArkBadRequestError.status_code=400, while.codeis the string body code (e.g."InvalidParameter"), andArkAPIConnectionErrorhas no int code.
In all of these cases the int check fails and execution falls through to return True, so a request-specific 400/401/403/404 from a cheap model still opens the 60s cooldown and later turns spend strong-model tokens — the exact behavior this commit sets out to avoid. The new test only passes because ProviderError(code=400) is a bespoke exception that happens to expose the status as code; google-genai style errors (which do use int .code) are not what the routed adapters raise.
Suggested fix: take the status from both shapes, in order, e.g.
code = getattr(error, "status_code", None)
if not isinstance(code, int):
code = getattr(error, "code", None)
if isinstance(code, int) and not isinstance(code, bool) and 400 <= code < 500:
return code in _TRANSIENT_CLIENT_CODES
return True…and add tests using the real classes (or shape-accurate doubles) for litellm.BadRequestError (400 → no cooldown), litellm.RateLimitError/Timeout (429/408 → cooldown), the Ark SDK's 400/429/5xx exceptions, and a connection error (no status → cooldown). The existing same-turn assertions (exception still propagates, route never switches mid-turn) already cover the parts that matter and can stay as-is.
Everything else looks good; this is the only change I'd want before merge, and CI on 6a457a55 was still pending at review time.
LiteLLM and the Ark SDK both expose the response status on status_code; Ark's code is a string business code such as InvalidParameter. Reading only error.code made every client error look stateless, so a 400/401/403/404 still opened the 60-second cooldown and pushed later turns to the strong model. Extract the status from status_code, then response.status_code, then an integer-parseable code. Keep 408 and 429 transient; other 4xx responses no longer open the cooldown. Connection errors and unknown errors remain conservative and still open it. Cover classification with real litellm and Ark SDK exception classes.
cuericlee
left a comment
There was a problem hiding this comment.
Re-review of 0f52edfc — resolved, LGTM
Verified the fix against the actual dependency shapes (pulled litellm 1.83.14 and volcengine-ark-runtime 5.0.56 source):
- litellm (
litellm/exceptions.py):BadRequestError/AuthenticationError/...carry the status onstatus_code;TimeoutandAPIConnectionErrorhave none. The new tests instantiate the real classes withhttpx.Response, so they pin the true shapes. - Ark runtime (
volcenginesdkarkruntime/_exceptions.py):ArkAPIStatusErrorsets bothresponse.status_codeand a class-levelstatus_codeliteral, whileArkAPIError.codeis the body business string (e.g."InvalidParameter");ArkAPIConnectionError/ArkAPITimeoutErrorhave no status. The dedicated test assertserror.code == "InvalidParameter"and that it is still classified as non-circuit-opening — exactly the trap from the previous revision. - genai-style int
.codefallback retained with an explicit test, and bool statuses excluded.
The candidate order (status_code → response.status_code → integer code) is correct for all three adapters, and the resulting policy matches the documented design: request-specific 4xx → no cooldown; 408/429, 5xx, timeout, connection and any status-less failure → cooldown; same-turn behavior is unchanged (error still propagates, no mid-turn model switch). The ProviderError double in test_turn_contract.py was also corrected to mirror status_code.
One trivial nit, not blocking: _http_status uses int(value), which also coerces non-integer numerics (400.9 → 400) and decimal/numeric strings. No real provider exception carries those, but an isinstance(value, int) check before the int() fallback would make the intent exact.
No further findings from me. CI on 0f52edfc was still mostly pending at review time — merge once it's green.
Queries with different capability requirements currently use the same execution model. This adds opt-in SystemOne routing: select a strong or configured candidate model once per root user turn, then retain that choice through subsequent LLM calls and nested tools.
Changes
DecisionConfig, root-level YAMLdecision, andAgent(decision=...). Routing and content safety share connection settings while retaining independent policies and client state. Remove former consumer-specific settings and update both documentation languages.0.5; high stakes remains0.05. Explicit overrides take precedence. Routing is opt-in and defaults toshadow.Validation
Tested on 2026-11-12; final revision:
d629698c.GitHub CI: 25 checks passed, 4 release/publish jobs skipped as configured, 0 failed or pending. This includes Python 3.10/3.12, Codex smoke, license headers, pre-commit, secret scanning, CodeQL and release gates.
uv run --extra dev pytest -n 2 -m "not codex_smoke and not piagent_smoke": 8,697 passed, 73 skipped, 4 xfailed. Local test processes include loopback addresses inNO_PROXY/no_proxy; an initial run was interrupted after the host proxy blocked local HTTP requests. The 39 affected-module tests passed after this environment correction.doubao-seed-2-1-pro-260915anddoubao-seed-2-1-lite-260915.uv run --extra dev pre-commit run --all-files: pass, including Ruff and secret scanning..licenserc.yaml,header check: 0 invalid headers.npm --prefix docs run types:checkandnpm --prefix docs run build: pass.Live checks require explicit opt-in and credentials; ordinary CI skips them. They validate execution contracts, not universal routing quality or cost savings.
Compatibility and limits
Unconfigured routing preserves existing model execution. Users of the former content-safety-specific Decision connection settings must migrate to public
decision; no compatibility aliases are retained.The supported scope is ADK text turns. Applications must serialize concurrent turns in one session. Live bidirectional execution and external runtimes are excluded. Provider context limits, invalid tool calls and decision timeouts remain observable.
Cleanup and CI verification traceability
Comparison:
049d066d; PR base:38d3c6e6. Cleanup changes test scope and documentation, with license-only production edits; no additional runtime API or dependency changes.tests/e2e/test_query_routing_http_live.py.licenserc.yaml, failing License-Checkdocs/content/docs/framework/agent/model*.mdx