feat(eco): /eco — RTK-style token compression for Bash tool output#708
Merged
Conversation
Port the token-compression methods from the RTK reference analysis (my-docs/token-compression/RTK/) into clawcodex, toggled per session with a new /eco slash command. When on, the model-bound rendering of Bash results is compressed by deterministic filters; the raw output stays recoverable. New src/eco/ package (imports nothing from tool_system): - filters.py — pure (command, exit_code, text) -> FilterHit|None filters: test-runner failure focus (pytest/cargo/go/jest native output; all-green runs collapse to one line, failures kept verbatim with relevant-line selection, capped at 10), command-family-scoped noise stripping (git advice/progress, pip/npm/cargo/docker/apt/brew ceremony), log dedup with [xN] counts over normalized lines, and a recoverable head cap for >400-line success outputs. - engine.py — first-hit dispatch with RTK's safety invariants: never_worse guard against the exact baseline the mapper would emit, lossy hits require a tee file + recovery hint or are discarded (guard pre-check avoids orphan files), 10 MiB input cap, any exception -> passthrough. - tee.py — per-session raw recovery files (co-located with the Step-11 tool-results dir) with collision-proof names (time_ns+pid+counter+slug), 1 MiB UTF-8-boundary-safe cap, keep-newest-50 rotation, and runnable hints: [full output: ...] / [see remaining: tail -n +N ...]. - guard.py / state.py — chars/4 estimator, never_worse, process-global session toggle + honest savings stats (passthroughs record nothing). Wiring: bash_tool computes eco on the full pre-truncation output and stores ecoContent (wire content only; raw stdout/stderr fields untouched — note the TUI transcript renders the mapped content, so the user sees the same compact string); mapper honors it ahead of the stdout/stderr assembly, byte-identical when absent. /eco command (toggle/on/off/status with per-filter savings), agent-server "eco" control, TUI SLASHES entry + dispatch case. Excluded by construction: interrupted/timeout/background/cd/image paths, non-Bash tools. Exit codes and is_error are never altered. A green test summary with a non-zero exit is treated as untrusted and passed through. Evaluation on a live command corpus: -91% aggregate tokens (pytest -v -96%, failing pytest -74% with failures preserved, find -95%, repetitive logs -98%, small/compact outputs pass through untouched). 53 new tests; design + implementation critic-reviewed (B1 tee-collision blocker and M1-M3 majors fixed with regression tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ericleepi314
added a commit
that referenced
this pull request
Jul 14, 2026
Replace the DeepSeek Prefix Cache banner with the measured /eco story (same session, 80% fewer Bash-output tokens; 92,989 -> 17,767 across 27 real operations), with an explicit #eco-benchmark anchor down to the full benchmark section. The DeepSeek economics remain in the 2026-06-18 news item, which the banner and the /eco section now point at. Add a 2026-07-13 news item (#708 + #712) to README.md, docs/NEWS.md, and README_ZH.md; both README news lists stay at the 10 most recent (the GLM-5.2 item rolls off — already archived in docs/NEWS.md). Wording is engine-exact per critic review: failure summaries keep the error lines that matter (kept lines never rewritten), every LOSSY compression tees the full output to disk (safe-loss ceremony strips do not claim recoverability), and the RTK-model recompute names its -19%. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ericleepi314
added a commit
that referenced
this pull request
Jul 14, 2026
Replace the DeepSeek Prefix Cache banner with the measured /eco story (same session, 80% fewer Bash-output tokens; 92,989 -> 17,767 across 27 real operations), with an explicit #eco-benchmark anchor down to the full benchmark section. The DeepSeek economics remain in the 2026-06-18 news item, which the banner and the /eco section now point at. Add a 2026-07-13 news item (#708 + #712) to README.md, docs/NEWS.md, and README_ZH.md; both README news lists stay at the 10 most recent (the GLM-5.2 item rolls off — already archived in docs/NEWS.md). Wording is engine-exact per critic review: failure summaries keep the error lines that matter (kept lines never rewritten), every LOSSY compression tees the full output to disk (safe-loss ceremony strips do not claim recoverability), and the RTK-model recompute names its -19%. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ericleepi314
added a commit
that referenced
this pull request
Jul 14, 2026
…713) Replace the DeepSeek Prefix Cache banner with the measured /eco story (same session, 80% fewer Bash-output tokens; 92,989 -> 17,767 across 27 real operations), with an explicit #eco-benchmark anchor down to the full benchmark section. The DeepSeek economics remain in the 2026-06-18 news item, which the banner and the /eco section now point at. Add a 2026-07-13 news item (#708 + #712) to README.md, docs/NEWS.md, and README_ZH.md; both README news lists stay at the 10 most recent (the GLM-5.2 item rolls off — already archived in docs/NEWS.md). Wording is engine-exact per critic review: failure summaries keep the error lines that matter (kept lines never rewritten), every LOSSY compression tees the full output to disk (safe-loss ceremony strips do not claim recoverability), and the RTK-model recompute names its -19%. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/eco— RTK-style token compression for Bash tool output, ported from the RTK deep-dive (my-docs/token-compression/RTK/, gitignored). When toggled on, the model-bound rendering of Bash results is compressed by deterministic filters — no model calls, no command rewriting — with every lossy transform backed by an on-disk recovery file. 60–90% fewer tokens on the outputs that dominate agent sessions (test runs, installs, long listings, log streams).Usage:
/eco(toggle) ·/eco on|off·/eco status(session savings, per-filter breakdown). Session-scoped, same state shape as/effort ultracode; applies to subagent Bash calls in the same process.Methods (from RTK)
test_runnernoise-striplog-dedup[×N]-counted unique errors/warnings over normalized lines (ts/uuid/hex/num/path)head-cap[see remaining: tail -n +61 …]Safety invariants (critic-reviewed, tested):
never_worseguard: compressed rendering is compared against the exact baseline the mapper would emit; baseline wins → passthrough. Worst case = 0% savings.tool-results/) and append a[full output: …]/tail -n +Nhint, or the hit is discarded. Tee filenames are collision-proof (time_ns+pid+counter).is_error, interrupted/timeout/background/image/cd paths: untouched. A green test summary with a non-zero exit is treated as untrusted (→ passthrough), never masked behind an all-green one-liner.<persisted-output>already covers giants).Note: the TUI transcript renders the mapped content, so with eco on the user sees the same compact rendering the model does; raw stdout/stderr stay in the output dict and the tee files.
Wiring
src/eco/{state,guard,tee,filters,engine}.py— pure, no tool_system imports.bash_tool.py:_maybe_apply_ecoon the success path (full pre-truncation output) →output["ecoContent"];_bash_map_result_to_apihonors it;_assemble_bash_bodyfactored so the guard's baseline is exactly the wire assembly./ecoLocalCommand (headless-safe) + agent-serverecocontrol + TUISLASHESentry + dispatch case.Evaluation (live command corpus through the real engine)
Testing
tests/test_eco.py,tests/test_bash_eco_integration.py): per-filter fixtures with ≥60% savings floors, never-worse property, tee collision/rotation/UTF-8-boundary, orphan-file avoidance, exit-code cross-checks, git state-line preservation, end-to-end_bash_call+mapper with eco on/off.agent-server --stdiodriven over NDJSON —eco on/status/off/bad-argall correct./ecoin slash menu, toggles on, status reports, toggles off.🤖 Generated with Claude Code