A PureRead job ran 43+ minutes at 100% of a core and blocked every route.bind to its root; only a daemon restart cleared it

[iceteaSA]

Sibling to #242/#243, but a different lane — and the reason I'm filing rather than reopening: every census in those two issues was SerialLspStatus. This one is PureRead, which v0.51.3's deferred-response work does not cover. The deployed fix is not regressing; this is a second hole in the same wall.

What happened

A subagent session was working in a scratch tree while its actor root was ~/projects/opencode. One interactive read job stopped returning and never came back:

job=subc-319-344  command=executor::Interactive::PureRead  lane=PureRead
age_ms=2605249            (43.4 min, growing monotonically)
started_before_oldest_writer=true
blockers=[queued_behind_configure(1), waiting_on_readers_stuck(...)]
blocked: route.bind for /home/icetea/projects/opencode
configure_phase_timings=[canonicalize=0ms, worktree_probe=0ms, config_resolve=0ms, ack_ready=3399485ms]

14 censuses over 13 minutes, still live when I intervened. Every phase that does work is 0ms; the entire wait is ack_ready, behind the read gate.

Client-side this rendered as module_id 'aft' did not answer route.bind within 12s, repeatedly. The subagent diagnosed itself correctly before giving up: "after a Playwright command timed out, every subsequent AFT/bash/search call fails."

The job survived its client. The operator killed the process; the daemon-side job kept running. Only a drain-restart of the module cleared it.

It is a runaway, not a leaked in-flight counter

I checked, because the fixes differ:

tid 2083 aft-executor-wo   512 CPU ticks / 5s  ≈ 100% of one core
process disk read          0 KB / 3s
D-state threads            none
daemon RSS                 0.77 GB, 1d18h uptime — otherwise healthy

One executor worker pinned, no blocked threads. Something is genuinely executing, not a flag that failed to decrement.

Note on the 0 KB: that rules out disk I/O, not a filesystem walk — the tree had just been written by a test run, so it was entirely in page cache. I initially read it as proof of pure compute and that was overreach.

The target

The searches immediately before the wedge were aft_search calls with an explicit path into a scratch tree:

/tmp/oc-diffux    148,732 files    2.8 GB    NOT a git root    2,408 node_modules dirs

Served through a borrowed search index — the journal shows borrowed search index for /tmp/oc-diffux has 0 drifted file(s) repeatedly up to 11:42:04, with searches completing normally in 100–800ms. Then a Playwright e2e run (large file churn across that tree) errored, a bash_kill landed at 11:45:01, and the very next job on that channel wedged.

Why the existing guards didn't hold

MAX_FALLBACK_WALK_FILES (50,000) and FALLBACK_WALK_BUDGET (10s) exist and are wired into this exact call — commands/semantic_search.rs:560-561 passes both into search_grep_bounded. The job ran ~260× the walk budget regardless, so whatever consumed the time is not inside the bounded walk.

One negative result that narrows it: I reproduced the same query against the same tree with a standalone binary and an isolated storage dir. It returned in 0 seconds — but with Semantic search is not enabled, so it took the lexical path only. Plain lexical search over that tree is not the culprit.

What I could not determine

I could not localize the hot loop. The release binary is stripped, so eu-stack returned nothing, and strace isn't installed on this host. Two hypotheses I can't separate with what I have:

1. Borrowed-index drift recomputation. The index reported 0 drifted files right up until a test run churned a large fraction of a 148k-file tree. Recomputing that delta could be both unbounded and CPU-shaped.
2. A cold index/semantic build triggered by that drift, with the interactive PureRead job parked waiting on the artifact — in which case the pinned thread is the build and the stuck job is a waiter holding the gate. This would fit the evidence equally well, and it changes the fix.

A debug build would separate these in minutes. Happy to run anything specific on this host; the tree still exists and the shape reproduces on demand.

Why this matters beyond one wedged session

The failure mode is worse than slow. A single interactive read on the PureRead lane can:

• run unbounded, with no aggregate command timeout
• hold the read gate, so has_epoch_reader blocks Lane::Mutating — every write and every new bind to that root queues behind it indefinitely
• survive the death of the client that issued it
• be clearable only by restarting the daemon, which drops warm state for every other root

The started_before_oldest_writer=true field you added for #243 is doing exactly its job here: it says the promotion gate is working and simply cannot preempt an already-running reader. The gap is not admission fairness — it's that nothing bounds or cancels the reader.

Adjacent to #250

ICEY1W32's report is the same root-type problem in a different subsystem: a large non-git directory full of dependency trees is a poor candidate for eager whole-tree work. Your stated direction there — "for such roots the callgraph moves to on-demand" — would want to cover external-path search too, not just the eager callgraph build. Ours was 148k files against their 24k.

The two pieces that would have contained this independently of root-type policy:

1. An aggregate bound on interactive lane occupancy, so no single read can hold the gate indefinitely regardless of what it's doing internally.
2. Route-abandonment cancellation for lanes other than inspect. Blocking-fresh aft_inspect holds the project execution lane; unrelated agents' mutations queue 25-60s behind it #242 gave inspect exactly this; a PureRead whose route is gone has no reason to keep computing.

[aft-alfonso]

Verified against source and accepted as filed — this is a second hole in the same wall, and your lane census makes the two-sided shape exact: admission fairness held (started_before_oldest_writer did its job), and nothing bounds or cancels the running reader. Both of your containment pieces are the accepted direction and are in implementation now: route-abandonment cancellation generalized beyond inspect (a PureRead whose route died stops at the next checkpoint), and the borrowed-path drift/verification work bounded with honest degradation instead of an unbounded synchronous recompute on an interactive lane. Your negative control (instant lexical on an isolated store) is what narrows it — the suspect set is the borrowed-index drift path, and we are reproducing with a debug build on a scaled churned tree to name the exact loop before fixing. Aggregate occupancy bounds on running readers (preemption) are deliberately excluded from this round as a separate design decision; what ships now is bounding + cancellation + a running-age census arm so the next specimen is visible in minutes, not censuses. The external-path point is taken as well: the non-git on-demand direction from the sibling report covers external search verification too.

[alfonso-aft]

Localized, with your negative control doing the heavy lifting. It is a third mechanism - neither drift recomputation nor a cold build: the test run's churn made the artifact OWNER persist a new index generation; your next borrowed query missed the generation-keyed cache and synchronously re-entered SearchIndex::read_from_disk_with_policy, whose per-file loop canonicalizes every path in the corpus (symlink-escape rejection) and, for legacy artifacts, additionally walks every posting to rebuild trigram counts - all before query-mode selection, which is why the walk budgets you cited never applied (they bound candidate verification, downstream of the load). A 30k-file debug replay measured 4ms warm / 3.4s on the churned reload under load, with 96 of 122 on-CPU samples inside fs::canonicalize; at 148k files with 2,400 node_modules dirs and the legacy posting walk, that is your 43-minute shape. Your instant standalone repro confirms it from the other side: an isolated store parses fresh without the borrowed reload path. Fix in review now: the borrowed load/verify path is budgeted with disclosed bounded-lexical degradation, route abandonment cancels in-flight interactive jobs generally (checkpointed - a PureRead whose route died stops computing), and a 60s running-age census line makes the next long occupant visible immediately. Ships in the next release after v0.52.0; this issue closes when it lands.

[iceteaSA]

Third mechanism — both my hypotheses were wrong, and the generation-miss framing explains something neither of them could: why the guards at semantic_search.rs:560-561 were irrelevant. I was looking downstream of the load for a bound that only ever applied to candidate verification.

One thing on top of your profile, since fs::canonicalize dominating 96/122 samples has a sharper edge than "canonicalize is slow":

// search_index.rs:4714
fn canonicalize_existing_or_deleted_path(path: &Path) -> PathBuf {
    if let Ok(canonical) = fs::canonicalize(path) { return canonical; }
    let Some(parent) = path.parent() else { return path.to_path_buf(); };
    let Some(file_name) = path.file_name() else { return path.to_path_buf(); };
    fs::canonicalize(parent).map(|p| p.join(file_name)).unwrap_or_else(|_| path.to_path_buf())
}

A path that no longer exists costs two canonicalize syscalls, not one. The persisted corpus is the pre-churn file list, so after a test run deletes or rewrites a large fraction of the tree, a large fraction of entries take the miss branch. The churn that invalidates the generation is the same churn that maximizes per-path cost — the reload is most expensive exactly in the scenario that triggers it. That's a multiplier your 30k-file replay would understate unless the replay deleted files rather than rewriting them.

The second-order point is the cheap one: the fallback canonicalizes the parent, and paths in a corpus are heavily clustered by directory. For 148k files across a few thousand directories, memoizing parent canonicalization collapses the dominant term from ~one syscall per file to ~one per directory. canonicalize_or_normalize (:4673) and canonicalize_for_search_membership (:4665) sit on the same primitive.

Worth noting this is the same cost class as #220, where you accepted memoizing resolved_path_restriction_root at context.rs:6129. That fix covered the path-restriction site; the index-load sites weren't in scope and are what surfaced here. If a systematic pass over the remaining unmemoized call sites is ever worth scheduling, this is the second time the primitive has produced a user-visible incident.

None of which argues against the bounding work — a budget with disclosed degradation is the right containment regardless, since it holds for corpus shapes no constant-factor win covers. This just changes whether the common case needs to degrade at all.

Happy to test a candidate against the real 148k-file tree; it still exists on this host, and the standalone-vs-daemon split gives a clean A/B.

[aft-alfonso]

Both refinements accepted and in implementation on top of the merged bounding work: the deleted-path double-canonicalize collapses to one syscall (and the measurement replay gains a churn-with-deletions arm precisely because a rewrite-only replay understates that term - your point), and the reload pass gets a per-load parent-directory memo shared by all three sites on the primitive, with symlink-escape rejection semantics byte-identical and the security-domain sites (path restriction validation) deliberately excluded from the shared memo. On the systematic-pass question: this is indeed the primitive's second user-visible incident after the path-restriction memoization, so the remaining unmemoized call sites are getting an audit entry in the perf ledger rather than waiting for a third specimen. The bounding work merged today regardless - as you say, budgets hold for corpus shapes no constant-factor win covers. Both land in the next release with the rest of this issue's fixes.