Skip to content

fix(supervise): keep a dropped child's harness transcript, persist it as its own blob, and settle a pointer - #1257

Merged
drewstone merged 10 commits into
mainfrom
fix/fleet-observability
Sep 16, 2026
Merged

drewstone merged 10 commits into
mainfrom
fix/fleet-observability

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Closes #1244. Closes #1248.

The hole

#1243 (0.229.0) reads a sandbox child's harness transcript just before the settled result is built. A child that settles down has no result artifact for it to ride in, so its transcript still died at destroy — and these are the children an operator most wants, because they are the ones that failed. Measured 2026-09-15 on three capability-per-parameter pursuits: 45 children in one evening executed, reasoned, and settled down with nothing; one run lost 38 of 68 inside ten minutes.

Two more facts surfaced while mapping the fix, each verified against source and each changing the shape:

The fix

  • The capture also runs in providerAsExecutor's failure path, at the last point the environment is live: the finally that destroys it runs after.
  • Executor.harnessTranscript() is the pull seam, mirroring metered(): read on settle, valid after execute resolves or throws. The scope carries the answer on every settlement arm, done and down alike.
  • The scope persists each capture under its own content ref in the ResultBlobStore, exactly as the tool-span trace already is, and the settlement carries a receipt: harnessTranscript: { status: 'available', transcriptRef, harness, fileCount, totalBytes, skippedCount }, a sibling of trace on Settled (both arms) and on the journal's settled record. Read it back with harnessTranscriptArtifact(evidence, blobs). No executor learns about storage; no child loses its reasoning to a ceiling.
  • Four absences stay distinct instead of collapsing into one silence, because an operator acts differently on each: execution-never-started (no environment was ever created — the maxUsd refusal of A maxUsd budget refuses every sandbox child because the tangle provider path reports unknown dollar cost #1240), capture-did-not-run (a box existed and the abort path closed the stream before the capture), executor-exposes-no-transcript (a CLI child has none by construction), and the capture's own reasons. transcript-persistence-failed mirrors trace-persistence-failed.
  • Renamed NativeSession*HarnessTranscript*. The contrast is not native vs non-native; it is the harness's own first-person session file against the supervisor's second-hand tool-span trace.
  • Per-child bounds unchanged (2 MiB / 16 MiB / 1000 files), now enforced exactly after each read; the module comment carries the fleet multiplication so the next person sizing a fleet finds it.

Breaking

WorkerTraceEvidence.nativeSession and ProviderLeafOut.nativeSession are removed. Both shipped in 0.229.0; the first was never written, the second is replaced by the settlement receipt. discovery-lab, the only fleet consumer, runs 0.226.0 and reads neither. Minor bump to 0.232.0 (check-version-bump: 16 export changes, paid by 0.229.0 → 0.232.0).

Evidence

🤖 Generated with Claude Code

drewstone and others added 4 commits September 15, 2026 21:16
…he receipt a home an unavailable trace cannot hide

Closes #1244.

0.229.0 reads a sandbox child's harness transcript just before the settled result is
built. A child that settles `down` has no result artifact for it to ride in, so its
transcript still died at destroy — and those are the children an operator most wants,
because they are the ones that failed. Measured 2026-09-15 across three
capability-per-parameter pursuits: 45 children in one evening executed, reasoned, and
settled `down` with nothing; one run lost 38 of 68 inside ten minutes.

- Capture also runs in providerAsExecutor's failure path, at the last point the
  environment is live: the `finally` that destroys it runs after.
- `Executor.harnessTranscript()` is the pull seam, mirroring `metered()` — read on
  settle, valid after `execute` resolves OR throws.
- Removes `WorkerTraceEvidence.nativeSession`, declared in 0.229.0 and written by
  nothing. It could not have worked on the path that needed it: it sat inside the
  `available` arm, and a dropped child has zero tool spans, so its trace is
  `unavailable`. The receipt is now a sibling of `trace` on the settlement itself.
- Renames NativeSession* to HarnessTranscript*. The contrast is not native vs
  non-native; it is the harness's own first-person session file against the
  supervisor's second-hand tool-span trace.
- Four absences stay distinct: execution-never-started, capture-did-not-run,
  executor-exposes-no-transcript, and the capture's own reasons.

typecheck clean; harness-transcript + environment-provider suites 109/109.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…settle a pointer

Closes #1248.

0.229.0 inlined transcript files as strings inside the settled result, so every
replay and resume re-parsed up to 16 MiB per child whether or not anyone wanted it,
and a 132-child fleet at the ceiling carried ~2.1 GiB inside its result blobs.

The scope now persists each capture under its own content ref in the ResultBlobStore,
the way the tool-span trace already is, and the settlement carries a receipt:
{ status: 'available', transcriptRef, harness, fileCount, totalBytes, skippedCount }.
Replay rehydrates a ref. Nobody pays for a transcript until they open it with
harnessTranscriptArtifact(evidence, blobs). No child loses its reasoning to a ceiling.

- ProviderLeafOut.harnessTranscript removed: executors report the in-memory
  HarnessTranscriptCapture through their port and never touch storage, restoring the
  "no blob sink in any provider or destroy site" property.
- Bounds stay 2 MiB / 16 MiB / 1000 files, now enforced exactly after each read; the
  module comment carries the fleet multiplication.
- transcript-persistence-failed mirrors trace-persistence-failed.
- api-surface.json, docs/api regenerated; canonical-api.md version 0.230.0.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ed child journals it

- persistHarnessTranscript settles a receipt with no file bytes on it, and
  harnessTranscriptArtifact brings the files back from the run blobs; a blob write
  that fails names transcript-persistence-failed instead of a dangling pointer.
- A child whose provider stream throws mid-flight settles down through a real
  scope with a pointer that resolves, and the journal's settled record carries the
  same pointer, so replay and Lab read it without the process (#1244).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@drewstone

Copy link
Copy Markdown
Contributor Author

Cloud reliability owner coordinating final Runtime delivery at the user request. #1255 continuity/auth and #1254 SDK cohort are merged; my lanes are integrating #1251 preflight and #1250 validation against current main690c5158 (Runtime0.231.1/Bench0.13.2). #1256 source owner is fixing its parent-completion assertion. Please retain sole #1257 source ownership, merge current main to adopt Knowledge17.0.2 cohort, fix exact CI failures locally, and send verified head + review. Do not start a competing tag/publish. Your0.232.0 breaking API version will be retained if ready for the coordinated cut; no downgrade or backlevel release. Cloud proof is prepared and remains unlaunched until the candidate is frozen.

@drewstone

Copy link
Copy Markdown
Contributor Author

Independent integration review of current8b83c709 found no lifecycle/persistence regression, but two concrete truthfulness/reader gaps to fix on your owned branch before claiming complete capture: (1) harness-transcript.ts find -size -2048k piped to head -1000 silently excludes oversized/overflow files, so skippedCount can be0 for incomplete enumeration. Preserve bounded work but report the omissions/truncation explicitly. (2) harnessTranscriptArtifact validates arrays but accepts malformed entries such as files:[null]; validate entry fields/bytes/content and skipped entries before returning. Focused113tests pass; neither finding is an excuse to expand architecture. Please add the narrow boundary regressions and send exact tested head. Root keeps final merge/tag ownership; tiny cloud candidate is separately testing the already verified continuity/preflight path.

@drewstone

Copy link
Copy Markdown
Contributor Author

Release cut: original pending set #1251/#1250/#1256 is now merged; main449fd3b9 exactly matches locally proven4,038-test candidate. I am publishing Runtime0.231.1 now so cloud and product consumers can proceed. Your0.232.0 remains the subsequent release, not a downgrade; keep sole source ownership and fix the two specific capture/reader findings. Do not re-tag0.231.1 or publish it in parallel. Send readyhead and proof when complete, and I will coordinate next delivery without interrupting the cloud proof.

drewstone and others added 4 commits September 15, 2026 21:51
# Conflicts:
#	CHANGELOG.md
#	docs/api/primitive-catalog.md
#	docs/canonical-api.md
#	package.json
…nciled floor, and executor wrappers

Review of #1257 found the receipt lost on three paths and one absence misnamed:

- replaySpawnTree rebuilt every Settled without harnessTranscript, so a replayed tree
  showed no transcript where the journal said there was one (#1214 one level down).
  Both rehydrated arms now carry it; the end-to-end test replays and asserts it.
- A retained-pending child writes no settled record while its slot stays open, so its
  pointer had no durable home — and that is the #1244 population exactly. The
  `reconciled` floor record now carries the receipt; a later terminal record for the
  node carries it forward.
- gateOnDeliverable, retryPreSpawnRefusals and the completion-gate wrappers forward
  every other optional port and not this one, so a wrapped provider child settled
  `executor-exposes-no-transcript`. They forward it now.
- A capture whose every found file was skipped (an abort, a denylist hit, over budget)
  reported `no-transcript`. It now reports `nothing-carried` with the skipped list, so
  "there was a transcript and it was not kept" never reads as "no transcript".

Also: the `cancelled` journal arm declares the field the settle path was already
writing; captureHarnessTranscript is public so a BYO executor can implement the port;
the module header describes the blob model; the CHANGELOG names the removed 0.229.0
field by its real name (ProviderLeafOut.nativeSession). Tests: the exact 16 MiB
post-read bound, nothing-carried on abort, replay rehydration.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 9e2ce28a8bb2f90be11d7c71f9fccc7d8d3268d3)
… capture, regenerate docs

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ents its parameter

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@drewstone

Copy link
Copy Markdown
Contributor Author

Rebased-by-merge onto main (449fd3b9, after #1253/#1254/#1255/#1256/#1250/#1251), then addressed the adversarial review of the first push. Fixed in 6bc5e2fa:

Deferred, on purpose: NodeSnapshot / observer projection / prepareInterruptedExecutors don't surface the receipt (same gap trace has there today); the steering sandbox executor has no capture at all, so it reports executor-exposes-no-transcript — the reason's doc now says so. Both belong with the #1247 read-side work.

Full-suite number: pending on a frozen worktree at 638a236c; my first two runs were invalidated by peer vitest contention and by my own rebase mutating the tree under test, and I'm not reporting either.

drewstone and others added 2 commits September 15, 2026 22:07
The fixtures embed the package version; CI's testing-fixture round-trip failed on
7f5638a because they still said 0.231.1 after the merge from main.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…lformed transcript blob

Integration review of 8b83c709 found two truthfulness gaps:

- `find -size -2048k | head -1000` dropped oversized files before listing them and cut
  the listing at 1000 with nothing saying so, so a receipt could read skippedCount 0
  on an incomplete capture. One `find -printf '%s\t%p'` now lists size and path: an
  oversized file is skipped by its size without being read, and a listing that
  overflows carries an `enumeration-truncated-at-1000` entry. A stub that answers
  path-only lines is read and bounded after the read, as before.
- harnessTranscriptArtifact accepted `files: [null]`. Every file and skipped entry is
  validated before the blob is returned; a malformed blob is corruption, not absence.

Regressions: oversized listed-not-read, overflow named, all-oversized is
nothing-carried, malformed blob refused.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@drewstone

Copy link
Copy Markdown
Contributor Author

Both integration findings fixed in 30d5cbfd, and this is the verified head I'm sending — I'll leave the merge and tag to you as asked, no competing publish.

  1. Enumeration omissions are now reported. One find -printf '%s\t%p' lists size and path, so an oversized file is skipped by its listed size without being read, and a listing that overflows carries one enumeration-truncated-at-1000 entry; skippedCount can no longer read 0 on an incomplete capture. A box whose find lacks -printf answers path-only lines and is bounded after the read, as before.
  2. harnessTranscriptArtifact validates every entry — path/bytes/content on files, path/reason on skipped; files: [null] throws as corruption instead of returning as a transcript.

Regressions added for both plus the all-oversized and overflow cases: harness-transcript + environment-provider suites 119/119. Local gates all exit 0 on this head: typecheck, biome, build, api-surface, docs:api (0 typedoc warnings), docs:freshness, check:version-bump (16 export changes paid by 0.231.1 → 0.232.0), testing fixtures regenerated for 0.232.0. merge-tree against origin/main clean. The previous head df9ccd03 was fully green in CI (ci, agent-bench, official-optimizers, packed-cohort); CI on 30d5cbfd is running.

Also merged from main since your note: Knowledge 17.0.2 cohort adopted via #1254.

@drewstone

Copy link
Copy Markdown
Contributor Author

Final exact-head recheck at 30d5cbf: locally ran harness-transcript and environment-provider suites, 119/119 passed; prior malformed-entry and omitted-file findings are fixed. All four current-head checks passed and merge-tree with the journal/retry repair is clean. I am adopting this into the coordinated 0.232.0 train, without tagging until the continuation-policy follow-up is verified. Narrow portability limitation remains: enumerate invokes GNU find -printf, so the prose about non-printf environments returning path-only output is not an implemented fallback. This does not block the Tangle Linux target; do not claim BSD/BusyBox capture support from these tests.

@drewstone
drewstone merged commit 1a433ed into main Sep 16, 2026
4 checks passed
@drewstone

Copy link
Copy Markdown
Contributor Author

Sequencing note for the cut: #1259 (fix/reconciled-terminal, closes the runtime half of #1247) is stacked on this branch — base fix/fleet-observability, merge-tree clean against both this head and main. Merge this first, then I retarget #1259 to main (or you do), then Lab discovery-lab#778 takes the dependency bump. No version change beyond 0.232.0 for either.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant