Skip to content

Tighten honest accounting across parser, storage, and aggregation - #8

Open
oxff644 wants to merge 1 commit into
trailofbits:mainfrom
oxff644:honest-accounting
Open

Tighten honest accounting across parser, storage, and aggregation#8
oxff644 wants to merge 1 commit into
trailofbits:mainfrom
oxff644:honest-accounting

Conversation

@oxff644

@oxff644 oxff644 commented Aug 24, 2026

Copy link
Copy Markdown

Motivation

agentcov's value rests on the guessing rule: coverage must never overclaim.
Auditing the pipeline surfaced several paths that still could, and one that
under-reports invisibly:

  • A cat of a 100-line file whose tool output was truncated to 3 lines was
    recorded as reading lines 1–100.
  • cat f | sed -n '1,50p' | head -5 was recorded as reading 50 lines while
    the agent saw 5.
  • cat f > /dev/null was recorded as a full read although the agent saw
    nothing.
  • A direct-read tool call for a path outside the repository produced zero
    events
    — not even an unknown — so audit trails could not distinguish
    "nothing was read" from "the parser lost it".
  • One corrupt or newer-versioned line in events.jsonl made load_events
    raise, taking down every subcommand including the Stop hook.
  • Credentials in commands (e.g. Authorization: Bearer …) were stored
    verbatim in the event log and rendered into shareable reports.

Changes

  • Output-evidence capping: read observations are capped to the number of
    lines the tool visibly returned; capped ranges are downgraded to inferred
    with the original claim kept in the event reason. Reads with no visible
    output, and multi-stage pipelines, fall back to unknown. Responses without
    usable text evidence are left untouched (no widening, no guessing).
  • No silent drops: unresolvable or missing direct-read paths now produce
    unknown events with the raw path in reason.
  • Tolerant event-log reads: unreadable lines are skipped, counted, and
    disclosed in coverage output (event_parse_errors, plus entries in
    unknown_events) instead of failing every reader, matching the
    "readers should tolerate older records" contract in AGENTS.md.
  • Credential redaction: bearer tokens, sk-/ghp_-style keys, AWS access
    key ids, and token=…-style pairs are replaced with [REDACTED] before
    storage. Redaction happens before identity computation so backfill dedupe
    stays stable. Heuristics include counter-examples in tests
    (sk-learn-notes.md, token=$TOKEN, regex patterns) to bound false
    positives.
  • Content anchors (verification side): LineRange gains an optional
    per-range digest of the exact lines read; aggregation downgrades anchored
    ranges to low confidence and marks them content_changed when the lines
    no longer match. (The stamping side lands in the follow-up live-hooks PR.)
  • Parser-evolution idempotency: events record a parser_version;
    aggregation keeps only the highest version per stable tool-call key, so
    re-backfilling a transcript after a parser upgrade supersedes the older
    interpretation instead of double-counting the union. Events without a
    tool_use_id are never superseded. The new fields are additive and excluded
    from the identity tuple; legacy records load unchanged.
  • Bounded range summaries: per-file read_ranges/search_seen_ranges in
    coverage JSON keep the newest 200 entries with an explicit *_elided
    count (a 20k-event log previously produced a ~10 MB coverage.json for a
    110-line repo; now ~0.8 MB).
  • Smaller parser fixes: repeated search output across compound-command
    segments is counted once; backslash-newline continuations and escaped
    quotes inside double quotes are handled when splitting segments.

Test plan

make check passes (120 tests on this branch, 23 added here). Every new
behavior has a positive test and a near-miss test that still comes back as
unknown/unredacted, per the repo's test guidance.

🤖 Generated with Claude Code

- Cap shell read ranges to the lines the tool visibly returned; reads
  with no visible output and multi-stage pipelines fall back to unknown
  instead of overclaiming.
- Record unresolvable or missing direct-read paths as unknown events
  instead of dropping them silently.
- Skip and disclose corrupt or newer-versioned event-log lines instead
  of failing every reader on the first bad line.
- Redact common credential shapes from commands before they are stored.
- Add per-range content digests plus verification that downgrades
  anchored reads to low confidence when the lines changed since.
- Add parser_version so re-backfilled tool calls supersede their
  older-parser events at aggregation instead of double-counting.
- Bound per-file range summaries with explicit elided counts, dedupe
  repeated search output across compound-command segments, and handle
  backslash continuations and escaped quotes when splitting segments.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@CLAassistant

CLAassistant commented Aug 24, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants