Skip to content

fix(runtime): keep long-running journals and retries reliable - #1258

Merged
drewstone merged 1 commit into
mainfrom
fix/long-pursuit-recovery-final
Sep 16, 2026
Merged

drewstone merged 1 commit into
mainfrom
fix/long-pursuit-recovery-final

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Repairs

Three general long-pursuit defects, repaired inside their existing owners:

  1. Journal append cost grew with the entire history. FileSpawnJournal now maintains a rebuildable validation index while retaining the original JSONL authority. Healthy owned appends no longer reread all preceding events. Restart, replacement, truncation, changed file metadata, and ambiguous write/fsync outcomes rebuild or invalidate the index. Cold validation uses the same incremental checks instead of repeated prefix scans. Cursor, retained-admission, input/result, and materialization checks stay enabled.
  2. Queued observer evidence retained mutable caller objects. Snapshot event and decision input before yielding to the asynchronous write queue, so later hooks cannot rewrite what the durable observer records.
  3. Separated failures accumulated as consecutive failures. Reset the consecutive-failure counter after a completed driver invocation. A continuing pursuit can recover from isolated outages without being stopped by stale failures from earlier successful turns. Explicit total attempts, consecutive failures, deadlines, cancellation, and resource limits remain enforced.

No new execution loop, scheduler, state store, profile, timeout default, or research policy. This builds on the merged retained-session continuity and child-evidence repairs. Runtime version 0.232.0 and generated artifacts follow the existing release scripts. No tag, package publication, main merge, or deployment is performed by this PR.

Verification

  • Failure-first reproduction: original journal/observer focused suite 4 failed / 7 passed; original retry suite 1 failed / 44 passed. Additional boundary cases added afterward.
  • Final focused journal + retry suites: 58 passed.
  • Full local native Runtime suite: 4,059 passed, 6 existing skips; lint, build, source/example typechecks, and documentation freshness passed.
  • Independent native Actions verification 35059657768: frozen install and peer checks, lint, full tests, build, full typecheck, verify:package, and docs:check all passed. The clean single-commit branch has the exact verified product tree; temporary preparation files and their history are absent.
  • Local filesystem benchmark, 2,500 progress appends: original 13.131s, patched 1.153s (~11.39x faster), byte-identical files and all records replayed.
  • Local 50,000-record / 10,005,623-byte stress: every record replayed in order from a fresh accessor; batches of 5,000 stayed between 2.76s and 3.25s; cold replay 0.268s.

The local container required proper orphan-process reaping to run the existing subprocess cleanup tests; no production or test deadline was weakened. The normal Linux Actions run passed without that environment wrapper.

Regression coverage and limits

Real filesystem tests cover repeated append, restart, another accessor, equal-size replacement, in-place rewrite, truncation, torn tails, committed corruption, duplicate cursors, and injected fsync failure. Observer tests cover mutation across the queue/hook boundary. Retry tests cover isolated failures, genuinely consecutive failures, and cumulative limits.

The index is not a distributed fence and does not authorize multiple simultaneous writers. Cold reads still materialize the requested history; this is not bounded-memory replay of an infinite log. Benchmarks establish local journal behavior, not production fleet throughput. Cloud continuation and coordinator-interruption qualification tracked in #1246 remains separate.

Normal PR CI must pass on this exact head before merge.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@drewstone

Copy link
Copy Markdown
Contributor Author

Cloud-director Runtime owner coordination: I found this PR before editing and will reuse your consecutive-failure reset rather than duplicate it. I own the requested completion-driven continuation contract: explicit repromptOnUnmet until-complete mode and successful incomplete turns not consuming the error retry allowance. I will keep your journal/observer surfaces untouched. Please retain ownership of this frozen PR and send any current source/release constraints. Runtime 0.231.1 is published; #1257 and this PR both currently request 0.232.0, so I am reconciling release ordering before publication. Independent read-only journal review underway. No competing tag or ADC dispatch.

@drewstone

Copy link
Copy Markdown
Contributor Author

Independent source review of exact 0cb003b complete: standards and spec reviewers found no substantive findings; 90 local tests passed across journal integrity, observer projection, and retry paths. Report retained at discovery-lab cloud-reliability-20260916/runtime-1258-review.md (local evidence pending its owner PR). Completion-driven correction is based on this head and intentionally separates successful turns from cumulative failed-invocation allowance; no journal edits. All four current-head checks are green. I will merge this verified prerequisite under existing user authorization, without publishing yet; the open 0.232.0 release slot can carry the integrated follow-up per the maintained version gate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant