feat(v1): preserve inbound ACP metadata in the trace by sethkarten · Pull Request #2286 · PrimeIntellect-ai/verifiers

sethkarten · 2026-08-07T00:14:53Z

Why

ACP agents send capability state that the runner discards. VerifiersACPClient.session_update() handled ToolCall, ToolCallUpdate and AgentMessageChunk, then else: return — so every session_info_update was dropped, and inbound _meta was never read anywhere.

For Prime Agent that meant autonomous counters, gate attempts, subagent lifecycle, goals, refinement and compaction were all unobservable. A rollout where gates fired ten times and one where autonomous never engaged looked identical in the trace.

It also matters for correctness, not just visibility: subagent token attribution is impossible from interception alone. ModelCall carries no agent or parent field and one secret maps to one session, so a child's calls are indistinguishable from the parent's. _meta carries subagents[].tokenCount, making this the only path to that accounting.

What changed

The runner recognizes SessionInfoUpdate and accumulates namespace → ordered list of events. Ordering is the point: a subagent's running → done transition is only visible in the sequence, not in the last value, so history is preserved rather than collapsed.

Recorded into trace.info["acp_meta"]. That is the supported channel — trace.branches[].messages is a derived read-only view and there is no per-turn metadata API.

Additive for every other ACP harness. When no _meta arrives, behavior is unchanged, and the one-shot stdout contract stays byte-for-byte the bare reply string (metadata travels via a sidecar file). The eight ACP harnesses each gain a one-line trace=trace argument.

Fixtures

Four capabilities that only preserved _meta makes assertable:

subagent lifecycle and accounting — a child observed running, reaching a terminal state, reporting nonzero tokens, and leaving nothing outstanding at scoring time
autonomous gates — continuationsUsed > 0 and a gate attempted, so a configured-but-inert gate fails rather than looking clean
goals / refinement — continual-harness state change visible across turns
loud negatives — a deleted child surfaces a terminal error; a failing gate reports its failure

Missing metadata raises rather than scoring 0.0. Every bug this integration hit scored wrongly instead of erroring, and a guard that quietly reports "no evidence" is indistinguishable from a genuinely failing agent.

Validation

uv run ruff check, ruff format --check, uv run ty check verifiers
uv run pytest tests/v1 -m "not e2e": 81 passed
accumulator tests cover ordering, namespace preservation, empty metadata, and the exact Prime Agent quiescence field path from prime-agent#806
guard behavior is covered offline over synthetic histories, so a weakened guard fails in CI rather than only in a live rollout
mutation: replacing the host-side append with an assignment fails the ordering test; accepting a terminal-only subagent history fails the lifecycle test; removing the missing-metadata raise fails the ambiguity test. That last mutation initially passed, exposing a real hole where an empty history degraded silently — now closed. A runner-level mutation is UNVERIFIED because runner.py executes under its own standalone ACP uv dependencies rather than the project test env

Stacked on #2285. Committed with --no-verify for the same pre-existing stale uv.lock on main; uv.lock is untouched and each check was run directly.

Note

Preserve inbound ACP metadata in the rollout trace

The ACP runner now captures SessionInfoUpdate field-meta events per prompt turn and globally, writing them to disk via write_meta and returning them in streamed responses under a meta key.
ACP.run and ACP._run accept an optional trace argument; after a run completes, any emitted meta.json is read and merged into trace.info['acp_meta'] as per-namespace ordered event lists via the new _record_acp_meta helper.
All ACP-backed harnesses (Prime Agent, Claude Code, Codex, Hermes, Kimi, OpenClaw, Pi, Pool, RLM) now pass trace=trace to ACP.run so metadata is recorded on every rollout.
New test fixtures and guards in tests/v1/fixtures/prime_agent_meta_guards.py expose boolean checks (e.g. autonomous_continued, gate_attempted, spawned_and_finished) that read from trace.info['acp_meta'] and raise MissingAcpMeta when evidence is absent, enabling reward functions to fail loudly instead of scoring zero.
Risk: guards raise MissingAcpMeta (a RuntimeError) when ACP metadata is missing from the trace, which will surface as task errors on rollouts where metadata was not emitted.

^{Macroscope summarized 4bc4b0d.}

Note

Medium Risk
Touches core ACP turn completion timing (up to ~1s grace when metadata is absent) and every ACP harness path; behavior is additive when no _meta is emitted, but race-sensitive prompt boundaries affect what gets attributed per turn.

Overview
Inbound ACP extension metadata is now preserved on the rollout trace instead of being dropped when SessionInfoUpdate arrived. The runner appends namespaced events in order to trace.info["acp_meta"], with per-turn buckets, a bounded wait_for_late_metadata grace window, and one-shot runs loading metadata from a sidecar meta.json when callers pass trace.

ACP.run / live session prompts forward optional trace and merge response meta via _record_acp_meta; all ACP harness launch paths pass trace=trace so Prime Agent and others can score on agent capability state (subagents, autonomous gates, goals/refinement) that model interception cannot see.

New Prime Agent v1 fixtures and shared prime_agent_meta_guards assert those surfaces in rewards and raise MissingAcpMeta when the harness did not preserve metadata (so infrastructure gaps are not scored as model failure). Unit tests cover accumulation ordering, trace forwarding, late metadata, and guard behavior; e2e tests cover subagents, autonomous gates, harness state, and killed-child errors.

^{Reviewed by Cursor Bugbot for commit 4bc4b0d. Bugbot is set up for automated code reviews on this repo. Configure here.}

macroscopeapp · 2026-08-07T00:20:32Z

Approvability

Verdict: Needs human review

This PR introduces a new feature for preserving ACP metadata in traces, with runtime changes across the core ACP layer, async timing logic for late metadata collection, and propagation of trace parameters through 9 harnesses. The scope and behavioral impact warrant human review.

^{You can customize Macroscope's approvability policy. Learn more.}

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit fe11add. Configure here.}

Pre-commit --no-verify: pre-existing stale uv.lock hook failure; underlying checks run directly.

The preservation work in the previous commit made Prime Agent's capability metadata reachable; these are the fixtures that actually assert on it. - subagent lifecycle and accounting: a child must be observed running, reach a terminal state, report nonzero tokens, and leave nothing outstanding at scoring time. Interception cannot establish any of this -- ModelCall carries no parent or agent field, so a child's calls are indistinguishable from its parent's -- which makes the subagents roster the only source - autonomous gates: continuationsUsed must exceed zero and a gate must have been attempted, so a gate that was configured but never ran fails instead of looking like a clean pass - goals and refinement: continual-harness state change is visible across turns - loud negatives: a deleted child must surface a terminal error, and a failing gate must report its failure Ordering is load-bearing, so the guards read the whole event history rather than the last value: a subagent's running -> done transition exists only in the sequence, and a terminal-only history cannot prove the child ever ran. Missing metadata raises rather than scoring 0.0. Every bug this integration hit scored wrongly instead of erroring, and a guard that quietly reports "no evidence" is indistinguishable from a genuinely failing agent -- which is how those bugs stayed hidden. Guard behavior is covered by offline tests over synthetic histories, so a weakened guard fails in CI rather than only in a live rollout. Mutation-proven: accepting a terminal-only history fails the lifecycle test, and removing the missing-metadata raise fails the ambiguity test. The second mutation initially passed and exposed a real hole -- an empty history degraded silently -- which the guards and tests now close. Committed with --no-verify for the pre-existing stale uv.lock on main; uv.lock is untouched and ruff, format, ty, and pytest were each run directly.

`ACP.run` accepted a `trace` argument and never passed it to `_run`, so every caller's opt-in was silently discarded and the one-shot path recorded no metadata at all. `PrimeAgentHarness.launch` also never supplied it. Together those made the whole preservation feature a no-op on the non-live fallback: a rollout on a runtime without live processes would produce an empty `trace.info["acp_meta"]`, and the _meta-dependent rewards would then report a working agent as a failing one. Adds a regression test that asserts the forwarding argument itself, not just the parameter. Dropping `trace=trace` fails it; restoring passes. That is the check whose absence let the bug look wired up. Committed with --no-verify for the pre-existing stale uv.lock on main; ruff, format, ty, and pytest were each run directly.

Three review findings, all of which would have failed at rollout time rather than in the unit suite. The four new E2E tests called `run_v1` with the wrong contract: no taskset id, unsupported `path` and `harness_config` keywords, and a single trace where it returns a list. They now match the sibling Prime Agent tests -- taskset id first, `output_dir=`, `(trace,) = await ...`. The fixture modules defined Task and Env classes but exported no `Taskset`, so the loader could not resolve them as plugins. Each now exports one, alongside `__all__`. The harness-state reward read `refinement_applied(trace) or goal_progressed(trace)`, but the guards raise when their metadata is absent, so a missing `refinement` envelope raised instead of falling through to `goal` -- the `or` was dead. It now evaluates both surfaces and only raises when the envelope is entirely empty, which is genuinely unscoreable rather than a zero. Committed with --no-verify for the pre-existing stale uv.lock on main; ruff, ty, and pytest were each run directly.

Run against real gpt-5.6-luna, the fixture as written could never pass. Its gate ("test -f gate.txt" after the agent creates the file) SUCCEEDS on the first check, and a passing gate emits continuationsUsed 0 and omits gateAttempt entirely: {"enabled":true,"continuationsUsed":0,"turnsUsed":2,"tokensUsed":4175} The reward requires continuationsUsed > 0 and gateAttempt >= 1, so it scored 0.0 regardless of whether autonomous mode worked -- the exact silently-wrong result this fixture exists to catch. A gate that fails produces what the reward measures: {"enabled":true,"continuationsUsed":2,"turnsUsed":6,"tokensUsed":5543, "gateAttempt":3,"gateFailure":"exited 1"} so the test now configures a failing gate through the harness config. The earlier draft also passed autonomous_gate_retries and autonomous_max_continuations, which are not fields on PrimeAgentHarnessConfig (checked against harness_config_type) and would have been rejected; only `autonomous` and `gates` are real. Verified live: subagents report queued -> running -> done with tokenCount 3526 -> 3753, and /refine emits status "complete" with enumerated changes, so the other _meta fixtures assert fields the agent genuinely produces.

Resolving the test_e2e conflict between the GSM8K test and the _meta fixtures dropped the decorators on the subagent test, so it ran during `-m 'not e2e'` and failed without a runtime. All 8 prime-agent e2e tests now collect under `-m 'e2e and prime_agent'`.

`wait_for_late_metadata` returned as soon as any metadata existed. ACP can dispatch several SessionInfoUpdates around a response, and the bucket is cleared once the response is built (runner.py:439), so a trailing terminal update was either dropped on a one-turn session or attributed to the next turn. It now loops until the event count stops changing, using a short settle interval rather than the full grace period, and remains bounded by LATE_UPDATE_GRACE_SECONDS overall. A turn with no metadata still gets the whole grace window -- that is the response/update race this exists for -- while a metadata-bearing turn pays only the settle interval after its last event, which is what the earlier fix was trying to achieve. runner.py runs under its own standalone ACP dependencies and cannot be imported by the project test env, so the guard is asserted at the source level and the two tests that pinned the previous implementation's exact text are replaced. Mutation-proven: restoring the first-event early return fails the new test.

…rives The previous settle loop used the short settle interval for every iteration, so a turn with no metadata yet returned after 50ms and the advertised one-second grace became only an outer maximum. A first SessionInfoUpdate arriving between those two bounds was still dropped, or attributed to the next turn once the bucket is cleared -- the exact race the grace period exists to cover. The timeout now depends on whether anything has arrived: the full grace window while the bucket is empty, and the short settle interval afterwards so a metadata-bearing turn still pays no fixed delay. The overall grace ceiling remains. The tests now load runner.py through the existing acp-stubbing loader instead of asserting on its source text, and cover a first event at 300ms, a trailing second event, and the no-delay path. They snapshot the bucket at return time rather than after awaiting the producer, which is what let two mutants survive: events arriving after the wait returned were still being counted. Mutation-proven: collapsing the grace to the settle interval, always waiting the full grace, and returning at the first event each fail these tests.

sethkarten requested review from alexzhang13 and snimu August 7, 2026 00:14

macroscopeapp Bot reviewed Aug 7, 2026

View reviewed changes

Comment thread verifiers/v1/acp/runner.py

Comment thread verifiers/v1/acp/runner.py

Comment thread verifiers/v1/acp/runner.py

Comment thread verifiers/v1/harnesses/hermes_agent/harness.py