Implementation Markdown journey intermittently exceeds 120s in CI #1143

The Implementation Markdown journey exceeds its existing 120-second launcher deadline in a local four-process Deno batch, then passes alone at the same source revision without a source fix.

Command: deno task test packages/cli/tests/adversarial-implementation.test.ts. The batch uses xargs -P 4 to run each of ten wrapper files in its own process. The Deno version observed immediately after this local run is 2.9.7, aarch64-apple-darwin (V8 15.0.245.2-rusty, TypeScript 6.0.3).

The failing output identifies runs the Implementation Markdown journeys without native models:

timed out after 120000ms (load average 8.5, 6.6, 5.2)
--- stdout before the deadline: nothing ---
--- stderr before the deadline: nothing ---
FAILED | 0 passed | 1 failed (1 step) (2m0s)

Both run-start head.txt files name the same revision and both worktree.patch files are empty (SHA256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855). This establishes source-identical intermittence under different concurrency/load conditions. Contention, nested startup cost and a stalled inner operation remain possible causes; no cause fix is established. The deadline is unchanged.

This was a local execution, so there is no GitHub Actions run URL. Run location: /private/tmp/xmd-combined-workflow-pilot-20261105/ticket/attempts/0003/runs/{deno-ten,deno-implementation-alone}. Preserved first output, numeric exit ledgers, both worktree snapshots and their hashes are under /private/tmp/xmd-continuation-call0003-20261106/execution-call0003/observed-first-failures; the durable copy is under /Users/tarasmankovski/.codex/visualizations/2026/11/04/01a1a4bf-bfa9-7801-905d-e364192bc859/continuation-call0003-20261106. Codex execution thread: 01a1a4bf-bfa9-7801-905d-e364192bc859.

Keep this issue open until the cause is fixed and focused regression evidence is recorded. A passing rerun does not close it. Related: #1119 (Friction journey timeout).

The same journey also times out in combined delivery CI42213983733 attempt1, Node5/17, at signed commit ede14f092352542b08b84e5d4e1470978d194b4f. The unchanged120000ms wrapper deadline expires with empty captured stdout/stderr; reported load averages1.3,0.9,0.4. Toolchain: Node22.23.4, Deno2.9.5 and Bun1.4.0 on ubuntu-latest. Complete first shard log: 493290 bytes, SHA256 c614e5787f2b6c8af035cef74c406249dde52c9470d000764cd5746c684b2b25, retained before any rerun under /private/tmp/xmd-combined-delivery-20261106/ci-first-failures/node-5-attempt1/ and its durable copy. All first CI logs are also preserved in a complete archive.

A bounded Linux diagnostic42216703321 checks out that exact source with the recorded CI toolchain, keeps the actual wrapper cap unchanged, and instruments the direct Node CLI instead. It completes the Implementation path in99.416 seconds; journal observations place about96.652 seconds in the nested Deno command. The executed outer Node profile is mostly idle. This successful instrumented sample does not identify the cause of the original timeout or count as an unchanged-wrapper rerun. Its240s diagnostic safety cap is separate from the120s test limit. A failed-job CI retry is running on unchanged source; no timeout increase, source correction or exclusion is made. Keep the issue open pending a cause fix and regression evidence.

Delivery diagnosis and bounded correction

The failed-job-only retry also fails: CI42213983733 attempt2, Node5/17 again exceeds120000ms. Complete first retry log:493314bytes, SHA256 8cf52ce159e169d93218c7e07cdfe7195ede79d5cbc2215c05dbaad9ef76a9e5. No third unchanged-source CI retry is planned.

A second bounded Linux comparison42219724791 completes the unchanged Implementation wrapper in82.483s at the same source. It invokes root-imported tsx4.21.0 rather than proving the runtime driver's prepared .bin/tsx4.23.15 parent. This successful diagnostic does not erase either CI failure.

Independent Astra traced an actual deadline overlap: Implementation's outer runCli120s matches the full allowance of its inner Test/child, leaving no startup or cleanup headroom. Under the user's continued delivery authority, a narrow helper correction is being implemented to give only Implementation and Friction outer180s headroom. Inner guards, assertions, execution paths and all other limits remain. This corrects enclosing deadline structure; it does not establish a production cause for the failed observations. Focused evidence, exact-commit independent review and fresh hosted verification remain required. Keep this issue open.

The narrow outer-headroom correction is signed at 0c6b488f4af7dea23953a38181b127d72d9e7cac in PR #1144 and passes exact-commit independent Astra review. Corrected CI42222261409 attempt1 is fully green (60 applicable jobs succeed, expected composability skip); the real Node Implementation wrapper passes in81.276s. Four focused corrected wrappers also pass on local Node22.23.4/Bun1.4.0/Deno. All inner deadlines and assertions remain. These successful observations complete current CI recovery, but their durations remain below the former120s bound, so they do not establish the original production cause. Keep this flake open. Complete coherent corrected-source measurement42222287812 and measured shard calibration are still running/pending; no partial timing record is imported.

Complete coherent measurement evidence

Measurement42222287812 attempt1 completes all three runtimes and assembly at unchanged corrected source 0c6b488f4af7dea23953a38181b127d72d9e7cac; official artifact digests, exact corpus/provenance and values are verified and imported byte-for-byte in PR #1144 metadata5a2b. The actual Node Implementation case completes in121571.756343ms (wholefile122007ms), above the former120000ms outer limit and below the current180000ms limit, with inner guards and assertions unchanged. This shows completed work using the added enclosing headroom in this observation; it is not a universal historical cause fix or a discriminating regression by itself. The complete original Node measurement log and exact case/file receipts are retained in post-pilot-headroom-measurement-evidence.json in the coordinator’s durable bundle.

PR #1144’s final required CI is green after a separately recorded unchanged-head TestAgent recovery (#1146). Its seven-minute/five-run shard timing calibration is explicitly maintainer-deferred for that delivery only, not passed. This issue stays open for the user-authorized cause/regression reconciliation through the workflow accepted by the actual NEW #922 pilot. No new diagnostic or source repair starts from this evidence update.

Additional unchanged-source observation, 2026-11-07

At PR #1163 head 4320ec168a501d1abd13375598b213eaba46367f, CI 42385177050 attempt 1, Deno 18/22 fails packages/cli/tests/adversarial-implementation.test.ts: the inner “Implementation fixture journeys” Test reaches its unchanged 120-second deadline. The whole wrapper reports about 2m1s. This is a timeout observation, with no demonstrated new causal defect in the candidate.

The identical signed revision and clean worktree passes the focused file locally in 1m41s (within a two-file Deno invocation; Preparation follows it). Command: deno task test packages/cli/tests/adversarial-implementation.test.ts packages/cli/tests/adversarial-preparation.test.ts; final exit 0. The first complete CI log is 624,834 bytes, SHA256 53de767ca2196608d1c523607dc3bb70f4bd7bc4192066717a67cd77d6942df5, saved before retry. Local output SHA256 c72644a7cf71694a0ff8c2d6948bb8c60e38b7bf27d21a10e614b3d3178d6310. Full logs, source checks and runtime versions are retained under /Users/tarasmankovski/.codex/visualizations/2026/11/04/01a1a4bf-bfa9-7801-905d-e364192bc859/1163-ci432-timeout-evidence (manifest.json).

The local comparison preserves source and deadlines but uses Darwin arm64, Deno 2.9.7 and Node 26.5.1; the CI failures are on hosted Ubuntu (Deno 2.9.5; Node job 22.23.4). This is qualified same-source passing evidence under a different host/toolchain, not proof of a passing same-job CI rerun or a shared cause. One failed-jobs-only CI retry has been requested; its result is pending. Source, deadlines, exclusions and correction allowances are unchanged. Keep this issue open until a cause fix and regression evidence exist.

Same-head CI retry result, 2026-11-07

On unchanged commit 4320ec168a501d1abd13375598b213eaba46367f, the single failed-jobs retry completed successfully for test-deno (18/22). These are attempt 2 of the same CI run; the first failing output remains preserved as recorded above. This supplies CI intermittence evidence in addition to the qualified local pass. It does not establish a common cause or repair the timeout. The issue remains open. The run's remaining failures are Main's health gate and its aggregate, separate from these product checks.

Additional Bun observation, 2026-11-07

CI 2514, Bun shard 7/10 at 0b78b818adfc4f03ec4b29532c069f6f8126f058 failed the same named Implementation fixture journeys inner 120-second deadline (121,876.24 ms wrapper). All 46 shard files continued to run. The complete first log was saved before any rerun at /private/tmp/xmd-1163-try-delivery-20261107/ci0b78-first-bun-7-10.log. This is a Bun observation, not proof that its cause matches the historical Deno timeout. No retry or timeout/source change was made for this observation; independent review separately requested two consumer corrections. The issue stays open.

The same 0b78b818 CI run also failed Node shard 13/18 at the inner Implementation fixture journeys 120-second deadline. All 42 files ran. Its complete first output is retained at /private/tmp/xmd-1163-try-delivery-20261107/ci0b78-first-node-13-18.log. This adds a Node observation alongside Bun; neither supplies a causal diagnosis or identical-source retry pass. No retry of this source was requested while the separate independent-review correction was being completed.

Node recurrence on PR #1190, 2026-11-08

Node shard 16/18, commit 818cacaafd5101d0b890a94a5d4cd7f7bf58ebd7, failed packages/cli/tests/adversarial-implementation.test.ts: Implementation fixture journeys reached its unchanged 120-second deadline at 18:05:39 UTC. The Node wrapper reported 123866.275398 ms. All 30 assigned files still ran; this was the shard's first failure. No rerun or source/deadline change was made for this observation.

The complete first log is retained locally (695718 bytes, SHA256 b2ca74b2a8d331de9c1facb57a607774cf666698dcc178c8e9e56c03f92297a2). The uploaded artifact, test-journals-node-16-attempt-1, was downloaded and inspected: 67894-byte ZIP, SHA256 660544d32f5501bbf5ee43aa5d1ef768ef18d9231207910d758489ab3d6186ef. It contains only child-9.jsonl and child-10.jsonl under an invocation beginning 2026-11-08T18-05-48.108Z, after the Implementation failure; neither contains the named Test or its 120-second timeout. Its existence therefore does not supply the timed-out execution's trace. Implementation still takes the legacy xmd run carrier path, outside automatic xmd test failure-journal retention.

This adds another Node observation to the existing Deno, Bun and Node history. It does not identify the timed-out inner operation or prove that the causes match. Keep this flake issue open; current main's separate green run is not an exact-revision fix or retry for this old PR head.

Bun recurrence in PR #1203

At signed head ea21243cb186ceb821ceb42eab581098d5ac78fb, CI run 42553057791, Bun shard 5/10 fails packages/cli/tests/adversarial-implementation.test.ts: the inner Implementation fixture journeys Test reaches its unchanged 120-second deadline. The wrapper reports 121,945.11 ms. Command: bun run test:bun 5/10; Linux, Bun 1.4.0 with Deno 2.9.5 and Node 22.23.4. All 46 assigned files still ran.

The complete first raw job log was preserved before further execution: 479,663 bytes, SHA-256 0c08bb5541e7d88dbc6a0346f80d0faf1b0bcdd21f8e3fe9188f5948868cbd39, retained at /private/tmp/xmd-staged-submission-evidence-20261108/ci/repair-bun5-first.log. The workflow artifact query returned no artifacts. The legacy carrier still supplies no interrupted inner-operation attribution.

A single focused local command, bun test --timeout=300000 packages/cli/tests/adversarial-implementation.test.ts, passes in 105.019 seconds on successor head 606845dc597101b804168c3006fb13a1e07eb5d1, macOS, Bun 1.3.14. That successor adds only a literal to the separate expected-wrapper inventory; it changes no Implementation path, assertion or deadline. Local log: 287 bytes, SHA-256 2895b8cc2f19aaf2648c3b4b5bcdc4892cd25dcbd4e3c9e112b1b872bf3815e6. This different-host, different-head pass is qualified evidence, not an exact-head CI retry or causal repair.

No unchanged-head retry of this failure was requested. The older run's unfinished jobs were cancelled after preserving completed failures, because the successor corrects a separate explicit-inventory failure and has its own fresh CI. On successor head 606845dc597101b804168c3006fb13a1e07eb5d1, fresh CI job 128954521507 passes all 46/46 files. Implementation takes 100,258.30 ms and ProductionRole also passes. Complete passing log: 472,358 bytes, SHA-256 fc8d7fc48ec7f258c2c7642ce1be07781c98c65b31e6d708e7b2e735bfaac429, retained as latest-bun5.log. This different-head CI pass changes only the separate expected inventory; it supplies recovery evidence without establishing an exact-head retry or a cause fix. Preserve the existing cause/regression contract, deadlines and assertions; keep this issue open.