[AAASM-5661] 🐛 (python-sdk): Say what the offline quick-start configuration evaluates - #318
Merged
Merged
Conversation
…iguration The AAASM-5529 controls each call install_fake_core() before running init_assembly, supplying an authoritative native runtime that the configuration docs/quick-start.md hands a reader does not have. They are therefore structurally unable to observe what the documented path does. This control runs the page's four keyword arguments verbatim — with no enforcement_mode, because the page never mentions one — against a pure-Python install, and drives the governed call through Agno's own FunctionCall.execute rather than the SDK's internal chain, so "a hook is installed" is observed rather than assumed. Refs AAASM-5661
_governance_unavailable's fail-open branch returns the bare GatewayClient, which exposes no check_tool_start, so the adapters' missing-interceptor fallback allows and a governed tool call reaches its body with no in-process decision behind it. It emitted nothing at all; the only loud signal on that configuration was _warn_agent_unregistered, which is about registration, so a caller who read it still had no way to learn that enforcement was gone too. Emit a stderr notice naming the gap and the remedy, scoped to the fail-open branch so a fail-closed caller is not told the opposite. Refs AAASM-5661
Three framework tabs revert the hook init_assembly() installs and re-apply one wired to the example's own LocalPolicyEngine. Their comments call it a no-op hook, which has been false since AAASM-4760 — it is deny-all — and the tab bodies are generated from quickstart_snippets/, vendored from the examples repo, so their comments are not this repo's to rewrite. The prose that explains the step is, and it needs a control that turns red if an upstream snippet drops it. Also asserts both startup warnings the documented configuration emits: registration on stderr, the enforcement gap through warnings. Refs AAASM-5661
The page told a reader that after init_assembly() "every tool call from this point on is routed through the policy gate". Measured on the configuration the page hands them — no gateway listening, no native agent_assembly._core on a pure-Python install — nothing evaluates any policy: the SDK installs a deny-all fail-closed hook and refuses each governed tool call with a reason naming the absent extension. Installing a real interceptor there is not achievable: with no runtime and no gateway there is no authority to consult, and the SDK cannot synthesise one. So the sentences are corrected rather than the code, in ADR 0033 §6 terms — Degraded, not Evaluated — and a new section says what the reader gets and what the SDK needs to decide instead. Every reworded sentence is rebound to test_quickstart_documented_ configuration.py, which runs the page's arguments with no fake native core, so the claims no longer rest on controls that supply the authority the documented path lacks. Refs AAASM-5661
The claim-bindings gate checks a named control still exists; it cannot check the control still passes. ci.yaml does not run on a docs-only PR, which is precisely the change that rewords a claim, so the named evidence went unexecuted on exactly those PRs. Refs AAASM-5661
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Three unproven_reason referents named AAASM-5661 — the ticket this PR closes. On merge each would have resolved to finished work that never intended to deliver the capability, which is the stale-referent failure AAASM-5750 exists to eliminate. auto-start-probes-and-starts-a-gateway -> AAASM-5760 no-arg-init-connects-and-appears-in-dashboard -> AAASM-5760 gateway-returns-allow-deny-decisions -> AAASM-5758 AAASM-5760 carries both measurements verbatim: the aasm console script shadowing the bundled binary is its defect #1, and the gateway-less call raising rather than degrading is its defect #2. AAASM-5758 runs each documented quick-start from published artifacts against a real gateway, which is the only place a claim about what a gateway returns can be shown true; it lists AAASM-5661 among its blockers, so it cannot close first. The shared _DOCUMENTED_PATH_UNMEASURED body no longer carries a ticket of its own — that is what let one referent serve two claims with different owners. Refs AAASM-5661, AAASM-5760, AAASM-5758
…irst The stale-referent rule compared against one string, so it banned the module's original author and nobody after them — the one author who was never going to break it. AAASM-5661 edited these bindings, pointed three unproven claims at itself, and the gate stayed green for the whole PR because the banned name was still AAASM-5529. Widen it to a tuple every subsequent ticket appends itself to, since each closes on merge and none may be a referent afterwards. Proven able to fail: repointing one claim back at AAASM-5661 turns test_an_unproven_reason_does_not_name_an_implementing_ticket red (exit 1); reverting it green (exit 0). Refs AAASM-5661
The AAASM-5529 controls monkeypatch _start_network_layer away, so nothing observed what mode="sdk-only" actually does. Assert the no-op shutdown hook rather than the network_mode string: the string is what the caller asked for, a started sidecar would leave a real teardown behind. Refs AAASM-5661
A full-page sweep for the class — a sentence asserting enforcement
unconditionally while resting on controls that install a fake native core
— found five. Two were corrected in the first pass; these are the rest.
:672 "the in-process adapter enforces on tool calls with no network
sidecar" — the same claim as the mode section below, left
unconditional while that one was qualified. The enforcement half
now lives only in the qualified sentence.
:675 "**Tool calls were governed.**" — named by nobody; found by the
sweep. A bare past-tense assertion about this example, where the
calls were refused without a policy governing them.
:681 "the policy denied the call" — the sentence a reader meets at the
moment their tool blocks, ten lines above the section saying the
opposite. Its own binding named a control whose point is that
there is no verdict source.
:681 now says what refused it and how to tell which, with a control per
disjunct: the deny controls read the policy text out of the reason, the
documented-path control reads the absent-extension text out of it.
Refs AAASM-5661
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
docs/quick-start.mdtold a reader that afterinit_assembly()"every tool call from thispoint on is routed through the policy gate". Measured on the configuration the page hands them —
init_assembly(gateway_url=..., api_key=..., agent_id=..., mode="sdk-only"), nothing listening onthe gateway URL, and a pure-Python
pip install agent-assemblywith no nativeagent_assembly._core— no policy is evaluated at all. The SDK installs a deny-all_FailClosedInterceptor(AAASM-4760) and refuses each governed tool call with a reason naming theabsent extension. Measured through Agno's own
FunctionCall.execute:So the reader's tools do not run — and nothing decides that they shouldn't.
Which branch of AC1 this takes, and why. The documentation branch. Installing a real
interceptor in that configuration is not achievable:
build_governance_interceptorhas exactlythree outcomes, and each needs an authority the offline example has none of — a connected
RuntimeClient(connect_runtime_clientreturnsNonewith no native extension) or a gateway toregister against. With the native extension present and no gateway,
init_assemblydoes not evenreturn:
register_agentraises andenforcere-raises it asConfigurationError. The SDK cannotsynthesise a policy source, so the honest fix is to correct the sentences, in ADR 0033 §6 terms —
Degraded, not Evaluated — and to say what the reader gets and what is needed to get a
decision from the SDK instead.
AC2.
test/unit/test_quickstart_documented_configuration.pyruns the page's four keywordarguments verbatim (no
enforcement_mode, because the page never mentions one) with noinstall_fake_core(), and drives the governed call through Agno's real chokepoint rather than theSDK's internal chain — so "an interceptor is installed" is observed, not assumed.
AC3.
_governance_unavailable's fail-open branch returned the bareGatewayClient— whichexposes no
check_tool_start, so the adapters allow — and emitted nothing. The only loud signal onthat configuration was about registration. It now writes an enforcement notice naming the gap and
the remedy, scoped to the fail-open branch.
Type of Change
Breaking Changes
New stderr output on the
observe/disabledfail-open path. No API or behaviour change.Related Issues
AAASM-5661unprovenreasons AAASM-5529 left in
test_quickstart_claim_bindings.pyfor the post-init claims.Testing
The new control was proven able to fail. With the enforcement path severed (the fail-open
posture that predates AAASM-4760 restored in
_governance_unavailable):failing on
test_a_governance_hook_is_installed_on_agnos_own_tool_path(the Agno tool body runs andthe file appears),
test_the_refusal_is_the_fail_closed_posture_rather_than_a_policy_decision, andtest_startup_reports_both_the_registration_and_the_enforcement_gap. With the path restored:Full suite:
1351 passed, 16 skipped.ruff check .clean,pre-commit run --all-filesclean forevery file this PR touches.
Checklist
Not done here, and why
docs/quick-start.md:141,:321,:468("init_assembly()installs a no-op hook; we revert it") are code comments inside the
BEGIN GENERATED: quickstart-framework-tabsregion, generated fromquickstart_snippets/, whichis vendored verbatim from
ai-agent-assembly/examples. They are stale rather than merelyimprecise — the hook has been deny-all since AAASM-4760, not a no-op — but rewriting the vendored
copy would silently diverge from the upstream repo that owns them.
TestTheWorkaroundTheFramework TabsCarrypins the step instead, so an upstream change that drops it turns the new prose red.A follow-up in
examplesis needed.auto-start-probes-and-starts-a-gatewaystays unproven in the bindings gate. It names aseparate measured defect (the
[project.scripts] aasmconsole script shadows the bundled binary,and the shadowing one has no
startsubcommand), which is a packaging change, not this one._FailClosedInterceptorsets no_enforceattribute, so_interceptor_enforces()reportsFalsefor it. Harmless today — itscheck_tool_startreturns a knowndeny, which normalisesthe same either way — but the flag is inconsistent with
RuntimeQueryInterceptor.