Skip to content

fix(claude): emit the facts the session sheet reads - #116

Merged
drewstone merged 2 commits into
mainfrom
feat/claude-session-facts
Sep 10, 2026
Merged

fix(claude): emit the facts the session sheet reads#116
drewstone merged 2 commits into
mainfrom
feat/claude-session-facts

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

What this fixes

The deterministic session-facts sheet (traces facts) answers a Codex session exactly and answered a Claude Code session at 0.384 of the same scale.

The sheet was not the problem. Its logic is harness-neutral; every fact it reads was emitted by the Codex adapter alone. The Claude Code adapter marked no folded subagent span, emitted no assistant-message span, no file-change span and no spawn marker, and decided who typed a turn from the text when the harness had already recorded it.

What changed, and why the transcript supports it

A subagent's work is the subagent's. Claude Code writes one transcript per spawned agent under the session directory, and the adapter folds them all into the parent's trace. Nothing marked them, so the parent's tool count was the whole tree's — 2,675 tool spans on a session whose own agent made 332. Every span read from a child transcript now carries traces.span.subagent; the sheet keeps those calls out of toolCalls and counts them in a new subagentToolSpans field, so the exclusion is checkable rather than invisible. The same rule keeps a child's edits, prompts and end time out of the parent's changedFiles, humanTurns and record window.

Spawns are named by the harness's own signal. A Task call selects the agent to run through subagent_type, so a call carrying it spawned a subagent — no allowlist of tool names decides it. Such a call now carries traces.agent.spawn plus the task name, and one synthesized subagent.lifecycle span stands for each child transcript so subagents has a span to be checked against.

The assistant's prose gets a span. It reached the trace only as content on llm.turn, so nothing answered "what did the agent say last". Each response now also emits message.assistant — the shape the sheet already reads — and a child's message is attributed to that child's task instead of the session's.

File changes come from the harness's record. Claude Code backs a file up before it changes it and records the path it resolved; those records become file.change spans, which beats a tool argument and survives a truncated input.

Who typed a turn comes from origin.kind. Claude Code stamps it on every record a person produced, so it decides: human is a person, any other kind is the harness or another agent. In a transcript that stamps it, a user record without it is not a human turn however human its text reads — [Request interrupted by user] is written by the CLI — and each turn that rule reclassifies is stamped so a reader can disagree span by span. A typed slash command is stored as <command-name> plus <command-args>; the human turn is the line that rejoins them, not the storage format. A message sent while a turn is running never becomes a conversation turn at all — Claude Code surfaces it inside the running turn as a queued_command attachment — and is now counted as the turn it is.

Measured

The sheet mapped mechanically onto an eight-question session-audit battery (tool calls, subagents, pull requests, changed files, human turns, final message, session window, token total), scored against an independently adjudicated truth. Same harness, same build pipeline, before at b8ae085 and after at this branch's head.

split sessions before after
Codex dev 8 1.000 1.000
Codex holdout 5 0.996 0.996
Claude Code 2 0.384 1.000

Claude Code, per question, before → after: tool calls 0.000 → 1.000, subagents 0.000 → 1.000, pull requests 0.891 → 1.000, changed files 0.100 → 1.000, human turns 0.330 → 1.000, final message 0.000 → 1.000, session window 0.750 → 1.000, token total 1.000 → 1.000.

Every Codex sheet is byte-identical before and after apart from the new subagentToolSpans field, so the Codex result is genuinely unchanged rather than re-scoring to the same number by chance.

n = 2 for Claude Code. Two sessions is a claim about those two sessions, not a general improvement claim. The rules implemented here are general — they read structural signals the harness records — but their breadth is untested beyond n=2.

What the transcript still cannot support

  • No cumulative token total. Claude Code records per-response usage and no session total, so tokenTotal stays null with that reason rather than summing per-turn deltas, which is a different and smaller number. Both measured sessions score this correctly by refusing to answer.
  • No change kind on a file change. The harness's file-history record names the path and states no add/update/delete, so the span states none and the sheet's existing default applies.
  • No subagent outcome. A child transcript records what the child did, not whether the parent accepted it, so subagent.lifecycle is UNSET; the parent's own tool span carries the status.
  • A spawn whose child transcript is absent is still listed from the call, with no lifecycle span to check it against.

Gates

pnpm check:source, pnpm typecheck, pnpm test (1003 tests), pnpm build, pnpm check:package all green. The two bounded-heap regressions pass. Fixtures are synthetic inline JSONL in the style of the existing Claude tests; no content from any real session is in this change.

🤖 Generated with Claude Code

drewstone and others added 2 commits September 10, 2026 14:44
The deterministic session-facts sheet answers a Codex session exactly and
answered a Claude Code session at 0.384 of the same scale. The sheet was
not the problem: its logic is harness-neutral, but every fact it reads was
emitted by the Codex adapter alone.

Four things the Claude transcript records and the adapter dropped:

- A spawned agent's transcript is folded into the parent's trace with
  nothing marking it, so the parent's tool count was the whole tree's:
  2,675 where the session's own agent made 332. Every span read from a
  child transcript now carries `traces.span.subagent`, and the sheet keeps
  the child's calls out of the parent's totals and counts them in a new
  `subagentToolSpans` field, so the exclusion is checkable rather than
  invisible. The same rule keeps a child's edits, prompts and end time out
  of the parent's changed files, human turns and record window.
- A Task call selects the agent to run through `subagent_type`, which is
  the harness's own signal that a call spawned a subagent. Such a call now
  carries `traces.agent.spawn` and the task name it named, and one
  synthesized `subagent.lifecycle` span stands for each child transcript,
  so `subagents` lists what ran with a span to check it against.
- The assistant's prose reached the trace only as `content` on `llm.turn`,
  so nothing answered "what did the agent say last". Each response now
  also emits `message.assistant`, the shape the sheet already reads, and a
  child's message is attributed to that child's task.
- The harness backs a file up before it changes it and records the path it
  resolved. Those records become `file.change` spans, which is a better
  source than a tool argument and survives a truncated input.

Who typed a turn also came from the text when the harness had already
recorded it. Claude Code stamps `origin.kind` on every record a person
produced, so it decides: `human` is a person, any other kind is the
harness or another agent. In a transcript that stamps it, a user record
without it is not a human turn however human its text reads, and the turns
that reclassifies are stamped so the reading can be checked. A typed slash
command is stored as `<command-name>` plus `<command-args>`; the human turn
is the line that rejoins them, not the storage format. A message sent while
a turn is running never becomes a conversation turn at all — it arrives as
a `queued_command` attachment — and is now counted as the turn it is.

Measured with the deterministic sheet mapped onto an eight-question audit
battery, scored against an independently adjudicated truth:

| split | before | after |
| --- | --- | --- |
| Codex dev (n=8) | 1.000 | 1.000 |
| Codex holdout (n=5) | 0.996 | 0.996 |
| Claude Code (n=2) | 0.384 | 1.000 |

Every Codex sheet is byte-identical before and after apart from the new
field, so the Codex result is unchanged rather than merely re-scoring the
same. n=2 for Claude Code: this is two sessions, not a general claim.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A message a person sends mid-turn arrives as a `queued_command` attachment,
and its `prompt` is a message body: usually a string, but the content
blocks of a message when the person attached an image alongside the words.
The new reader called a string helper on it directly, so a session with one
such message failed to parse at all — five of sixty local transcripts.

It now goes through the same `textOf` the adapter already uses for every
other message body, which takes a string verbatim and joins the text blocks
of an array. The image block is dropped, which is right: the human turn is
what they typed.

Two neighbouring reads of untrusted transcript fields are hardened the same
way rather than left to trip later. `origin.kind` is used only when it is a
non-empty string, so a record carrying some other value falls back to the
text heuristics instead of being trusted as a label, and a subagent
metadata file whose `description` is not a string names no task rather than
taking the parse down.

Found by parsing sixty local Claude Code transcripts with both builds: the
base build read all sixty, this one failed five before the fix and reads
all sixty after it. The measured splits are unchanged — Codex dev 1.000,
Codex holdout 0.996, Claude Code transfer 1.000.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@drewstone

Copy link
Copy Markdown
Contributor Author

Follow-up: a crash found by probing beyond the two measured sessions

Parsing the 60 largest local Claude Code transcripts with both builds caught a regression the two measured sessions did not contain: a queued_command attachment's prompt is a message body, so it is usually a string but is the message's content blocks when the person attached an image alongside the words. The new reader called a string helper on it and 5 of the 60 transcripts failed to parse at all.

It now goes through the same textOf the adapter already uses for every other message body. Two neighbouring reads of untrusted transcript fields are hardened the same way rather than left to trip later: origin.kind is trusted only when it is a non-empty string (otherwise the text heuristics answer), and a subagent metadata file whose description is not a string names no task instead of taking the parse down.

Probe result: base build 60/60 parsed, this branch 55/60 before the fix and 60/60 after. The measured splits are unchanged — Codex dev 1.000, Codex holdout 0.996, Claude Code transfer 1.000.

@drewstone
drewstone merged commit 435d80e into main Sep 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant