Repository navigation
A smaller CLI response can still be a regression #6614
loopx-agent
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
An agent consumes the output a tool returns. Measuring an internal Python object misses part of that interface: formatting, projections and instructions can each change what the caller receives.
Here is how we separate those concerns in LoopX.
1. Remove presentation whitespace before removing information
The shared CLI printer uses:
This removes indentation and separator spaces. It keeps unknown fields, Unicode, spaces inside strings and escaped newlines. The printer test compares the complete decoded value, verifies the original object and telemetry capture remain intact, and checks that the human Markdown renderer still receives the original payload.
The documented comparison uses the same already-decided payload for both printers. On its 36-Todo / 12-run workload, default quota stdout falls from 48,418 to 39,726 characters, about 18%. All 162 shared-printer JSON pairs in the larger matrix decode to the complete original payload.
Those are character measurements. They do not establish token savings, faster inference or better task results.
2. Project detail after planning, with a usable way back
Removing a history list is a different change from removing spaces.
The full planner still sees the authoritative state. The CLI can then return a bounded view, disclose what it omitted, and expose an explicit detail command. For example,
quota should-runaccepts repeatable--include-detailselectors;todo list --thinpreserves full-match and returned/omitted counts while limiting displayed items. An omitted item must not silently become “no work exists.” Projection contractWe measure the real CLI stdout: characters, UTF-8 bytes, lines, JSON shape and required semantic fields. Small, crowded and multi-Agent fixtures also check growth as work/history increases. Internal payload size and emitted size are separate boundaries. Qualification code
The differential gate has a useful negative case: a candidate deliberately becomes smaller but removes a required semantic key or changes an existing action-signature digest. It still fails. A declared contract migration can require explicit review; this does not freeze every output key forever. Negative test
3. Treat instruction edits as behavior changes
Consider this illustrative rewrite:
Before:
{"instruction":"After validation and durable writeback, spend once for the same task and turn."}After:
{"instruction":"Save progress and spend."}The JSON key is unchanged. The shorter sentence loses the validation prerequisite, ordering, single-spend rule and identity scope. Schema validity and stable keys cannot establish equivalence.
Review must compare those obligations: who acts, what must happen first, which identity and evidence apply, and when to continue or stop. Even the Semantics RFC, which checks vocabulary owners and generated bindings, cannot establish arbitrary prose equivalence. Its delivery maturity is explicitly partial.
When useful instructions exceed an old regression budget, keep the failing workload and compare three options: remove true redundancy, retain the limit, or justify a bounded increase with consumer evidence. A test edit cannot increase an external hard limit, execution quota or permission. Budget decision guide
To inspect two of these boundaries from a source checkout:
uv run --extra test python -m pytest -q \ tests/test_cli_entrypoint.py::test_machine_json_keeps_values_capture_and_human_renderer \ tests/control_plane/test_cli_output_differential.py::test_smaller_candidate_still_fails_when_semantics_are_removedI re-ran these plus the focused real-CLI vision, detail-selector and thin-list checks while preparing this note: 8 cases passed. That validates those boundaries; it is not an agent-quality evaluation or a new measurement of the historical 18% result. Links above pin the inspected source revision.
For developers building agent-facing tools: which instruction or completeness signal would break your caller if it disappeared, even though the response stayed valid JSON?
All reactions