[AAASM-5750] ✨ (core): Forward the hook-layer audit record to the runtime - #319
Conversation
…ntime Encodes a governed call's outcome as the aa_core::AuditEntry JSON the native GovernanceEvent validates, and hands it to RuntimeClient.send_event. Failures are swallowed: the adapters call the audit hook from inside the governed tool path, so a degraded audit channel must not fail the call. Refs AAASM-5750
The vocabulary had no value for a hook that carries the record onward, so wiring one would have had to declare a retention the SDK cannot observe. Adds it and maps each disposition onto the ADR 0033 section 6 term it earns. Refs AAASM-5750
Neither audit hook resolved on any interceptor this SDK shipped: __getattr__ handed both names to a GatewayClient that has neither, so the adapters found nothing to call and every governed call, allowed as well as denied, produced no evidence. Both now resolve and forward to the runtime. audit_sink is computed from the runtime client so an interceptor without one still declares honestly. Refs AAASM-5750
init_assembly warned for every disposition that was not the caller's own, which silently included forwarded the moment that value existed. Enumerates the two that leave no evidence, and corrects the surrounding claims. Refs AAASM-5750
…t has The fake _core exposed neither, so an interceptor built over it declared absent and the forwarding path was off in every test using it. The event wrapper validates the AuditEntry field set, because a double that accepts any string makes "the record crossed" pass over a payload the real one rejects. Refs AAASM-5750
Inverts the measurement: the SDK's own governed-tool chain over the same boundary, same positive control, now has to find the record on the far side on both branches. Adds the no-runtime input that shows the computed disposition moving, and a control that the boundary double can reject. Refs AAASM-5750
5750 built the sink, so a claim still calling recording Planned under it describes shipped behaviour as unbuilt. Collapses the guarded-site tier into one repository-wide rule and adds a positive control over synthetic input, since a rule expecting no findings is green when the scan is broken. Refs AAASM-5750
The unit suite substitutes a double for the native constructor and can only catch an obviously wrong payload. This asserts the real one accepts the builder's output on both branches, with a negative control, behind the existing native gate. Also rescopes a control comment to what its fixture proves. Refs AAASM-5750
Twelve sites across the docs tree, the README and the repo guide asserted that governed calls leave nothing behind. Each now states what the record's fate depends on, and names the runtime-absent case rather than dropping it. Refs AAASM-5750
read_text raises for a file removed between the directory listing and the read, aborting the whole scan — turning a gate whose verdict is "no findings" into one that produced no verdict. The node SDK's equivalent walk hit exactly that in CI. Skips it instead; a file that no longer exists carries no claim. Refs AAASM-5750
"Every tool call" is on ADR 0033 fd-7's banned list — it overstates coverage, since only a governed tool's call reaches the gate. Corrected while the line was being edited rather than left because it predated the change. Refs AAASM-5750
The first sweep enumerated by phrase and missed sites whose false clause never used the word "audit": the architecture page still said nothing in agent_assembly calls send_event and nothing constructs GovernanceEvent, both of which runtime_audit.py now does. Also two adapter module docstrings, a deny-path comment contradicting its own module docstring, the runnable example, and two summaries in assembly.py contradicted by code 200 lines below them — including the field comment that documented the warning predicate this branch had already fixed. Refs AAASM-5750
|
Second sweep — the first one's method was wrong, and it missed nine sites. The first pass enumerated by phrase (grep for
Plus two adapter module docstrings ( One judgement call, flagged rather than changed silently: Full unit suite (1297 passed), |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
send_event is unacknowledged, so nothing observable from this side establishes a durable event attributed to the action — ADR 0033 section 6 Observed is not earned. Also corrects the event-type comment: the runtime does not key off that tag, it keys off action_type/detail, and report_event leaves those at proto3 zero, so the collapse the comment warned against is what actually happens. Drops the cross-SDK line that goes stale on merge. Refs AAASM-5750
Three of twelve adapters build a record on the denied path. haystack and llamaindex each said otherwise in their own module docstring, at the very adapters that raise before their record helper. Refs AAASM-5750
Six shipped sites said audit covers "allowed calls as much as denied ones". It covers three adapters on the denied path — google_adk, pydantic_ai, openai_agents — and the other eight, LangChain among them, return or raise first. The exception was disclosed in the PR and reached no user-facing text, which is the ADR 0034 failure: an upper layer may simplify, never broaden. Refs AAASM-5750
|
Review round 2 — You were right that it is eight, and that LangChain is among them. I measured it rather than taking the number: 3 of 12 adapters build a record on the denied path — The All six shipped sites scoped, and the eight named in user-facing text. That the disclosure lived only in the PR body was the ADR 0034 failure, and I accept the characterisation — an upper layer may simplify, never broaden, and "allowed calls as much as denied ones" broadened.
The event-type comment is corrected to the opposite of what it said. It claimed the runtime "keys its own handling off this tag". It keys off
Noted, not fixed: python's claim gate scans only Gates: 1297 passed / 4 skipped, |
The gap was described in eight places with no referent, which reads as an explained limitation rather than a tracked one. Points at what 5783 will change — report_event payloads reaching the live stream and the durable entry — and states that no SDK can claim Observed until it lands. Refs AAASM-5750, AAASM-5783
|
1 participant
Description
The SDK's own interceptor now resolves an audit hook and forwards a governed call's outcome to the runtime.
Before this,
RuntimeQueryInterceptor.__getattr__delegatedrecord_result/on_tool_endtoGatewayClient, which has neither — so the adapters'getattrguard found nothing and no record was emitted on any path, allowed or denied.RuntimeQueryInterceptornow defines both hooks and ships the outcome overRuntimeClient.send_event, the same native primitiveregister_agentalready uses.Design, and the code that decided it
native/aa-ffi-python/src/lib.rs:188(RuntimeClient.send_event→aa_sdk_client::AssemblyClient::report_event) was already exported, already benchmarked, and had zero call sites inagent_assembly/. It is the only durable event-publish path reachable from Python: the gRPC stubs exposeCheckAction/BatchCheck/OpControlStream(inbound only), and there is no HTTP audit route.GovernanceEventvalidates its argument asaa_core::AuditEntryJSON, soagent_assembly/core/runtime_audit.pyencodes that shape. The identity and hash-chain fields are zero-filled, deliberately and with the reason written down: onlyevent_typeis read from the entry, the whole JSON travels as an opaquedetailslabel, and the authoritative attribution is the runtime's — derived from the verified identity of the IPC connection. The agent id the SDK does know travels insidepayload, where it is plainly a claim by the SDK.on_tool_endis defined as well asrecord_result, for the one caller that looks up the second name specifically:AssemblyCallbackHandlerforwards its ownon_tool_endto the interceptor's. Without it, the record built on LangChain's callback path was accepted by the handler and dropped one hop later.§6 term earned
AUDIT_SINK_FORWARDED)._FailClosedInterceptor, or any run with no reachable runtime →AUDIT_SINK_ABSENT→ Degraded. Not Unmeasured: where the record stops has been measured.AUDIT_SINK_DISCARDED.Type of Change
Breaking Changes
AuditSinkDispositiongains a value; the two hooks are additive.Related Issues
Testing
pytest test/(unit) — 1297 passed, 4 skipped.ruff check/ruff formatclean.pre-commit run --all-files— all hooks pass (mypy included).Proven end to end at the downstream boundary, with a reachability control (AC2).
test_the_shipped_path_forwards_the_record_across_the_native_boundary[allow|deny]drives the realrun_governed_async_toolchain against a recording native client. The positive control asserts a policy query carrying the probe crossed; the finding asserts thesend_eventchannel specifically carries the record — kept separate so the control cannot satisfy the finding.The payload is validated against the real native constructor, not only a double. I built the extension locally (
maturin develop --release) and confirmedGovernanceEventaccepts the builder's output on both branches. That is pinned astest_the_audit_payload_builder_is_accepted_by_the_real_governance_eventbehind the existing native gate, with a negative control. The built.sois not committed. In the unit suite the constructor is a double that validates the requiredAuditEntryfield set — a replica of the contract, not the validator, and its docstring says so.Proven able to fail (AC3). Replacing
record_result's body withreturn False:No test here injects a sink into the SDK's own path.
Checklist
Known limitation, stated rather than papered over
Seven adapters (CrewAI, LlamaIndex, Haystack, Agno, Smolagents, Microsoft Agent Framework, MCP) construct no record at all on their denied path — they return or raise before their record helper. That is a separate defect of the AAASM-5665 class (an event that is never built), not a sink that drops one, and it is out of this ticket's scope. Those denied paths remain Evaluated, not Observed.
🤖 Generated with Claude Code
https://claude.ai/code/session_01XWLmA8FgULT9e6ntdCo1H2