fix(runtime): keep long-running journals and retries reliable - #1258
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Cloud-director Runtime owner coordination: I found this PR before editing and will reuse your consecutive-failure reset rather than duplicate it. I own the requested completion-driven continuation contract: explicit repromptOnUnmet until-complete mode and successful incomplete turns not consuming the error retry allowance. I will keep your journal/observer surfaces untouched. Please retain ownership of this frozen PR and send any current source/release constraints. Runtime 0.231.1 is published; #1257 and this PR both currently request 0.232.0, so I am reconciling release ordering before publication. Independent read-only journal review underway. No competing tag or ADC dispatch. |
|
Independent source review of exact 0cb003b complete: standards and spec reviewers found no substantive findings; 90 local tests passed across journal integrity, observer projection, and retry paths. Report retained at discovery-lab cloud-reliability-20260916/runtime-1258-review.md (local evidence pending its owner PR). Completion-driven correction is based on this head and intentionally separates successful turns from cumulative failed-invocation allowance; no journal edits. All four current-head checks are green. I will merge this verified prerequisite under existing user authorization, without publishing yet; the open 0.232.0 release slot can carry the integrated follow-up per the maintained version gate. |
Repairs
Three general long-pursuit defects, repaired inside their existing owners:
FileSpawnJournalnow maintains a rebuildable validation index while retaining the original JSONL authority. Healthy owned appends no longer reread all preceding events. Restart, replacement, truncation, changed file metadata, and ambiguous write/fsync outcomes rebuild or invalidate the index. Cold validation uses the same incremental checks instead of repeated prefix scans. Cursor, retained-admission, input/result, and materialization checks stay enabled.No new execution loop, scheduler, state store, profile, timeout default, or research policy. This builds on the merged retained-session continuity and child-evidence repairs. Runtime version 0.232.0 and generated artifacts follow the existing release scripts. No tag, package publication, main merge, or deployment is performed by this PR.
Verification
verify:package, anddocs:checkall passed. The clean single-commit branch has the exact verified product tree; temporary preparation files and their history are absent.The local container required proper orphan-process reaping to run the existing subprocess cleanup tests; no production or test deadline was weakened. The normal Linux Actions run passed without that environment wrapper.
Regression coverage and limits
Real filesystem tests cover repeated append, restart, another accessor, equal-size replacement, in-place rewrite, truncation, torn tails, committed corruption, duplicate cursors, and injected fsync failure. Observer tests cover mutation across the queue/hook boundary. Retry tests cover isolated failures, genuinely consecutive failures, and cumulative limits.
The index is not a distributed fence and does not authorize multiple simultaneous writers. Cold reads still materialize the requested history; this is not bounded-memory replay of an infinite log. Benchmarks establish local journal behavior, not production fleet throughput. Cloud continuation and coordinator-interruption qualification tracked in #1246 remains separate.
Normal PR CI must pass on this exact head before merge.