Skip to content

feat(fidelity): measure what converting a session into a harness loses - #94

Draft
malihashar wants to merge 2 commits into
skillsynchq:mainfrom
malihashar:feat/fidelity-library
Draft

malihashar wants to merge 2 commits into
skillsynchq:mainfrom
malihashar:feat/fidelity-library

Conversation

@malihashar

@malihashar malihashar commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

In short

txcript check is a new command that tells you what converting a session to another agent would lose, before you do it. This is step 2 of 3: the library that does the measuring (txcript::fidelity). It adds no command yet, and nothing visible changes for users.

Part 2 of 3 for #92. It builds on #93.

How to review this PR

This branch contains the commit from #93 too, until that PR is merged. Please review only the last commit, feat(fidelity): measure what converting a session into a harness loses. After #93 merges, this PR will show only its own changes.

What this PR adds

A new library module, txcript::fidelity. It answers one question: if I convert this session into harness X, what will I lose?

let report = txcript::fidelity::assess(HarnessId::Codex, &transcript)?;
let all = txcript::fidelity::check_all(&transcript);

A Report lists each difference as lost, changed or synthesized (made up by the target), grouped by what it is:

Severity Examples
content text, thinking, tool calls and arguments, results, images, artifacts, error flags
accounting usage, stop reasons, the model on each message, thinking signatures
metadata title, model, git branch, working directory

The types are serializable, and nothing in the module does any I/O or needs a new dependency.

How it measures

  1. Render the transcript into the target's native text (using dispatch from refactor: move the harness text dispatch out of the wasm module #93).
  2. Read that text back.
  3. Compare the original with the result.

So it reports exactly what the real codec keeps, and it cannot drift when a codec changes. Some choices that keep it honest:

  • It compares a flat stream of blocks, not message by message, because targets split and merge messages.
  • JSON that is only re-spaced is not loss. Trailing spaces and empty blocks are ignored. Ids and regenerated timestamps are ignored.
  • A stop reason or tool name that only changed form is reported once as "changed", not as one loss plus one addition.
  • Items that the target only made up are listed, but they never raise the severity.

The measurement is "codec-level": it does not include what a store adds on top, such as artifact files.

Tests

  • 48 unit tests: each category as lost, changed and synthesized; JSON spacing is not loss; split and merged messages are not loss; empty sessions; deterministic output; bounded work on odd input.
  • A golden table of what every harness loses on one rich session. It is also published in docs/formats/README.md, and a test fails if the two disagree, so the docs cannot drift. When a codec fix changes a result, the table is updated on purpose.
  • A property test: no harness panics on generated sessions, and Simple (the lossless format) reports nothing.
  • The full suite passes, and cargo fmt and cargo clippy (0 warnings, with and without default features) are clean.

What you should know

What I could not test

Real agent resumes, OpenCode and Cowork writes (I have no opencode binary or Cowork account here), and the real WASM build.

malihashar and others added 2 commits October 25, 2026 15:07
The two functions that parse or render a harness's native text by
HarnessId lived privately in src/wasm.rs, so only the WASM bindings could
use them. Move them to a new always-compiled txcript::dispatch module and
make wasm.rs call them. No behaviour change.

This lets other code (such as a conversion report) render a transcript
into any harness's format and read it back without going through WASM.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Add txcript::fidelity. assess(target, source) renders a transcript into a
harness's native text, reads it back, and compares the two, so it reports
exactly what that codec keeps, changes, loses, or makes up. check_all runs
it for every harness that can render.

Losses are grouped as content (text, tool calls and results, images),
accounting (usage, stop reasons, per-message model, thinking signatures)
and metadata (title, model, branch, cwd). Blocks are compared as a flat
stream, not message by message, because targets split and merge turns.
JSON re-spacing and regenerated ids are not counted as loss. The result is
serialisable. The measurement is codec-level and does not include what a
store adds, such as artifact files.

Tests: unit tests for each category, one golden table of what every
harness loses on a rich session (also published in docs/formats), a
proptest that no target panics and that Simple reports nothing, and the
docs table is checked against the codecs so it cannot drift.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant