Tighten honest accounting across parser, storage, and aggregation - #8
Open
oxff644 wants to merge 1 commit into
Open
Tighten honest accounting across parser, storage, and aggregation#8oxff644 wants to merge 1 commit into
oxff644 wants to merge 1 commit into
Conversation
- Cap shell read ranges to the lines the tool visibly returned; reads with no visible output and multi-stage pipelines fall back to unknown instead of overclaiming. - Record unresolvable or missing direct-read paths as unknown events instead of dropping them silently. - Skip and disclose corrupt or newer-versioned event-log lines instead of failing every reader on the first bad line. - Redact common credential shapes from commands before they are stored. - Add per-range content digests plus verification that downgrades anchored reads to low confidence when the lines changed since. - Add parser_version so re-backfilled tool calls supersede their older-parser events at aggregation instead of double-counting. - Bound per-file range summaries with explicit elided counts, dedupe repeated search output across compound-command segments, and handle backslash continuations and escaped quotes when splitting segments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
agentcov's value rests on the guessing rule: coverage must never overclaim.
Auditing the pipeline surfaced several paths that still could, and one that
under-reports invisibly:
catof a 100-line file whose tool output was truncated to 3 lines wasrecorded as reading lines 1–100.
cat f | sed -n '1,50p' | head -5was recorded as reading 50 lines whilethe agent saw 5.
cat f > /dev/nullwas recorded as a full read although the agent sawnothing.
events — not even an unknown — so audit trails could not distinguish
"nothing was read" from "the parser lost it".
events.jsonlmadeload_eventsraise, taking down every subcommand including the Stop hook.
Authorization: Bearer …) were storedverbatim in the event log and rendered into shareable reports.
Changes
lines the tool visibly returned; capped ranges are downgraded to
inferredwith the original claim kept in the event
reason. Reads with no visibleoutput, and multi-stage pipelines, fall back to unknown. Responses without
usable text evidence are left untouched (no widening, no guessing).
unknown events with the raw path in
reason.disclosed in coverage output (
event_parse_errors, plus entries inunknown_events) instead of failing every reader, matching the"readers should tolerate older records" contract in AGENTS.md.
sk-/ghp_-style keys, AWS accesskey ids, and
token=…-style pairs are replaced with[REDACTED]beforestorage. Redaction happens before identity computation so backfill dedupe
stays stable. Heuristics include counter-examples in tests
(
sk-learn-notes.md,token=$TOKEN, regex patterns) to bound falsepositives.
LineRangegains an optionalper-range digest of the exact lines read; aggregation downgrades anchored
ranges to
lowconfidence and marks themcontent_changedwhen the linesno longer match. (The stamping side lands in the follow-up live-hooks PR.)
parser_version;aggregation keeps only the highest version per stable tool-call key, so
re-backfilling a transcript after a parser upgrade supersedes the older
interpretation instead of double-counting the union. Events without a
tool_use_idare never superseded. The new fields are additive and excludedfrom the identity tuple; legacy records load unchanged.
read_ranges/search_seen_rangesincoverage JSON keep the newest 200 entries with an explicit
*_elidedcount (a 20k-event log previously produced a ~10 MB coverage.json for a
110-line repo; now ~0.8 MB).
segments is counted once; backslash-newline continuations and escaped
quotes inside double quotes are handled when splitting segments.
Test plan
make checkpasses (120 tests on this branch, 23 added here). Every newbehavior has a positive test and a near-miss test that still comes back as
unknown/unredacted, per the repo's test guidance.
🤖 Generated with Claude Code