compute: walk fast-path index peeks off the serving worker by antiguru · Pull Request #38242 · MaterializeInc/materialize

antiguru · 2026-08-17T10:05:58Z

Status: parked, not seeking review

This exists so the experiment that decides its fate can deploy it. It is not on its way in, and it should not be reviewed toward merge. doc/developer/design/20260720_two_runtime_compute/peek-placement.md, added by this PR, carries the argument and the decision rule. Tracked as CPU-217.

There are four candidates for one defect, and the grid is two independent axes rather than one:

	run to completion	yields
on the serving worker	today's default	#38040, cooperative yielding
off the worker	this PR	coherent, unmeasured, unimplemented

The decision rule, registered before the experiment runs: if yielding alone matches this on E1, E11 and E8b, this should be deleted rather than merged. If it wins on E8b alone, its case narrows to the unattributed swap-walk duration and nothing else.

Two measurements already count against it. E2 found the second runtime does not need it, and E12 found it reproducibly worse than doing nothing when a peek queues behind a long operator activation, because dispatch happens in process_peeks after step_or_park returns and retirement costs another step. That cost is paid before the walk starts, so giving the walk yield points cannot recover it.

What it does

Takes an owned, Send snapshot of a peek's cursor and walks it on a blocking task instead of inline on the timely worker that received it, so a long scan no longer delays the peeks queued behind it.

The snapshot owns the Arc batches its cursor covers, which is what makes it Send: the traces this crate maintains are read through an Rc-based reader, so neither borrowing from one nor owning Rc batches would cross a thread. That is a hard dependency on the Arc migration in #37881. It is generic over the cursor source, so a worker-local TraceBundle and a registry SharedTraceHandle feed the same walk.

Bounded by index_peek_offload_max_inflight, since each in-flight walk retains the batches its cursor covers — a memory bound, not a concurrency knob. mz_index_peek_walks_total{substrate} exists so that "the offload changed nothing" and "the offload never engaged" are distinguishable, which cost a round of staging measurement before it existed. An offloaded walk can also divert to the peek response stash partway through, without which the feature is unreachable in a production configuration, because production runs the stash on.

Stacking

Based on #37770, so its diff carries the interactive runtime too until that lands. Only the final commit is this work. #37881 is a prerequisite of both.

The flag naming discussed on #38239 (compute_peek_substrate as a named substrate rather than a boolean, since placement and preemption are independent) is not implemented here. It is specified only, and deliberately not built while the mechanism itself is undecided.

Verification

cargo check --workspace --all-targets clean. cargo test -p mz-compute --lib 88/88, including offloaded_walk_matches_the_inline_walk, which asserts the offloaded walk returns exactly what the inline walk returns on both cursor sources.

🤖 Generated with Claude Code

https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

…aring, interactive runtime) Squashed work-log for the two-runtime compute branch. Migrates production arrangements from Rc to Arc batches, adds cross-runtime arrangement sharing, and stands up a second in-process compute runtime that serves ephemeral peeks off the maintenance runtime's shared arrangements, fronted by a Multiplexer. Includes fixes landed while stabilizing the branch: - keep shared-trace snapshot import consistent so joins don't double-count - guard the shared-index snapshot bound against a Timestamp::MAX as_of - stop the multiplexer from dropping peek responses on non-zero processes - release transient read holds so two-runtime read frontiers advance: forward the interactive runtime's terminal empty-Frontiers report for a dropped transient even after ownership eviction, and report frontiers only for the transient collections the interactive runtime exclusively hosts Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

Arc batches carry more fixed per-batch overhead than the Rc batches this check was written against, so a single-record index arrangement now exceeds the coarse `size < 16 * 1024` "not egregious" bound (the empty arrangement still fits). Bump the bound to 32 KiB, which one Arc batch on top of the empty spine stays under. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

…w is dropped A deferred interactive dataflow whose shared-index dependencies are not yet published is cancelled from `pending_work` without ever building. The controller had already created the collection, initialized its per-replica frontiers to the as_of, and acquired read holds on its storage inputs. It releases those input read holds only once the collection's frontiers all reach the empty antichain, but the cancelled dataflow never reported any frontier, so the holds leaked and pinned the inputs' read frontiers. This surfaced as an MV's `since` never advancing under two-runtime (read_frontier_advancement). Send empty `Frontiers` from `drop_collection` on the deferred-cancel path so the controller can clean the collection up and drop the holds. Also filter interactive-runtime frontier reports on `id.is_transient()` in the multiplexer instead of `owner_of(id)` plus a terminal-empty special case. The interactive runtime reports frontiers only for the transient collections it hosts (see `report_frontiers`), so every such report must forward regardless of `transient_owner`, whose eviction on `AllowCompaction{empty}` races ahead of a dropped transient's trailing frontier reports. This replaces the earlier terminal-transient hack and no longer depends on ownership state. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

Measures peek p50/p99/qps under a saturated maintenance runtime. Two closed-loop `HydrationChurn` actions continuously build, hydrate, and drop heavy maintained MVs, keeping the maintenance workers busy. Two open-loop peeks (a fast-path index point lookup and a slower range-scan reduce) run at a fixed rate against a pre-hydrated indexed table. Run twice, once with `enable_two_runtime_compute=true` and once false, and compare the SELECT latencies. With two runtimes on, the interactive runtime serves the peeks off the shared arrangements, isolated from the maintenance workers, so the open-loop peeks accumulate far less queue-wait latency than when they contend with hydration on the same workers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

Two indexes on the same key make the optimizer build the second as a cross-dataflow re-export of the first: the second index's dataflow imports the first's arrangement and re-exports it under its own id. On the maintenance runtime, render's `ArrangementFlavor::Trace` arm publishes that re-export into the sharing registry via `reexport`, but unlike the `Local` arm it builds no streams of its own and so installs no seal-signal frontier tap. An interactive peek on the re-exported id that arrives when the arrangement's `upper` equals the peek `as_of` defers in `pending_work` waiting for the seal. Nothing ever fires `note_frontier` for the re-exported id when it advances, so the peek is never re-examined and hangs. A `DROP INDEX` of one of two same-key indexes followed by a `SELECT` hit this deterministically. Have the registry track re-export aliases. `reexport(from, to)` records `to` as an alias of `from`, and `notify` marks an id together with the transitive closure of ids that re-export it. The source's own dataflow outlives its catalog drop while a re-export still imports its arrangement, so its live tap keeps sealing the re-export. `remove` prunes an id only as a target, never as a source, so that trailing seal signal survives the source's `DROP INDEX`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

Records the sound architecture for cross-runtime index arrangement sharing: a capture-based lifecycle with two synchronization barriers (seal-gated capture, tombstone teardown), grounded in the invariant that the interactive runtime only performs single-time reads. Removes the incremental replay machinery that caused the row-doubling and delayed-capability-panic bugs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

Replaces the v1 single-time/tombstone sketch with the converged design after three adversarial review rounds and two feasibility spikes. Three principles: build on a correct protocol and panic outside it (no local safeguards, shared fate); the multiplexer splits the controller's one correct protocol into two correct sub-protocols, making the interactive one well-formed with an internal Import command; render in command arrival order with late-bound imports for deterministic construction. Keeps the frontier-tracked replay (deleting it reintroduces row-doubling) and fixes the delayed-capability panic at its root by single-sourcing the replay feed. Late-binding uses pre-allocated publication points (placeholder plus adopt-in-place), spike-validated feasible. Teardown safety rests on the controller's read-hold discipline rather than a compute-level lease, which review showed to be inert under a correct controller. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

The interactive query's Get(id) is already a dataflow import that the interactive runtime resolves from the registry by role, since it holds no traces of its own. Formalize that: the interactive sub-protocol's index imports are shared imports, a self-describing import kind that references a registry id with no prior local creation. This makes the sub-protocol well-defined without a new command. Placeholder teardown rides existing events, the maintenance publisher's close-on-drop for an adopted slot and last-reader eviction for a never-adopted one, so no import verb and no withdrawal are needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

Nine staged TDD tasks from the committed design: single-source replay feed (F1), placeholder plus adopt (F2), registry get-or-create, placeholder close/eviction, bounded-read routing, arrival-order rendering with late-bound imports, since<=as_of assert, delete dead live import(), and regression/concurrency suites. Tasks 1-2 seed from the validated spike branches spike/single-source-publish and spike-f2-placeholder. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

…rontier The publisher derived the published upper from the trace's map_batches, which leads the arrangement stream within a worker step, so a Frontier instruction could be enqueued before a Batch whose hint is below it and the importer's caps.delayed panicked. Use the stream frontier, which never leads the delivered batches, as the authoritative upper for both the published state and the importer Frontier instructions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

…place) A SharedTrace placeholder can be created empty and handed to an importer as a live handle before the arrangement is published. The maintenance publisher later adopts the same Arc in place rather than constructing a fresh one. Publishing becomes the degenerate case of adopting a publication point with no prior reader. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

Whichever runtime touches an id first creates its placeholder slot; the other adopts or reads the same Arc. Replaces create-fresh-and-overwrite so a placeholder a reader already imports is filled in place and never overwritten. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

A placeholder whose index creation is cancelled before it publishes must not wedge its importers. Closing pushes a terminal empty frontier to them and the registry evicts the slot when its last reader leaves. An adopted slot is closed by the maintenance drop instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

Route a CreateDataflow to the interactive runtime only when its until is bounded and it has no subscribe or copy-to sink. Copy-to is finite-until but drives an S3 sink and is refused by reconciliation, so it belongs on maintenance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

…d imports Remove the per-worker deferral, which built dataflows in each worker's own publication order and so allocated timely channel ids in a worker-divergent order. Every dataflow now builds immediately in command arrival order, and an import of a not-yet-published dependency binds through a registry placeholder that adopts later. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

Matches the assert the maintenance import path already makes. A since above the requested as_of means the controller offered an unreadable as_of, a protocol error, so it panics rather than reading coalesced data silently. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

The interactive runtime issues only bounded reads, so the unbounded live import had no non-test callers. import_snapshot_at is the sole interactive import path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

…s-first invariant Add a debug_assert on the interactive CreateDataflow path as a tripwire for a Multiplexer routing bug, matching the bounded-read predicate the multiplexer enforces. Document that evict_unadopted consults only the oks adoption flag, which is sound because every publish site adopts oks before errs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

… read-hold discipline Record where correctness comes from for cleanup of a maintenance index an interactive runtime imports. The controller holds a read hold on the imported id for every reader and frees it only after the reader retires, so the index is never compacted or dropped while an interactive import still reads it, with no cross-runtime signal (Instance::finish_peek). A controller that violated this would be an incorrect protocol instantiation, which the runtime panics on. evict_unadopted is registry hygiene for a bounded, empty leaked placeholder, not a correctness mechanism, and Published::close guards only a wedge that a correct protocol never produces. Both are retained for a future hygiene path and have no production caller today. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

rustdoc under --document-private-items rejects these: SharedTraceHandle and PublishArrangement are not in scope in the linking modules (use full crate paths), Self::notify on the private Inner struct resolves to nothing (name ArrangementSharingRegistry::notify), and public docs on get_or_create_placeholder and evict_unadopted linked to the private insert_shared and adopted_and (demote to plain code spans). cargo check does not run rustdoc, so these passed the per-task build gates. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

The old `ArrangementSharingRegistry::insert`/`insert_shared` and the `PublishArrangement::publish`/`publish_named` primitives are production-dead: every maintenance publish path now binds through `get_or_create_placeholder` plus `adopt`, which fills a placeholder in place rather than overwriting a slot a reader may already have imported. `insert` reintroduced exactly that placeholder-overwrite hazard, so remove it. Delete the four items and port their remaining (test-only) callers to the production pattern: registry-based sites route through `get_or_create_placeholder` + `adopt` + `notify`, and the standalone shared-trace primitive sites use `Published::placeholder` + `adopt` via a new `adopt_fresh` test helper. Behavior is unchanged: `insert(publish, publish)` into a fresh slot is equivalent to placeholder-then-fill. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

`PendingWork` lost its `Dataflow` variant earlier in this branch, leaving a single `Peek(SharedIndexPeek)` wrapper around every value in `pending_work`. Store `SharedIndexPeek` directly in the map, drop the enum, and straight-line the four single-arm match sites (`resolve_dirty`, `handle_cancel_peek`, and the reconciliation drop loop in `server.rs`). No peek behavior changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

psycopg 3.3.3's Cursor.execute accepts LiteralString, bytes, sql.SQL, sql.Composed, or Template, but not a plain str. An f-string with interpolated attributes types as str, so pyright rejected the three HydrationChurn.execute calls. Encode each f-string to bytes, matching the repo convention (for example util.py PgConnInfo.connect and parallel_workload/action.py). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

…ontier-filtered one A late importer's initial snapshot came from state.chain, which the publisher built by filtering map_batches to batch.upper() <= stream_frontier. The stream frontier lags the trace by a scheduling round: the batch data is delivered, the frontier notification catches up a round later. When the Spine merges an already streamed batch with a leading one into a single batch whose upper leads the frontier, that filter drops the whole batch, stranding its historical part. A late importer registering in that window seeds an incomplete snapshot missing rows, so its reducer later sees a retraction without its insert and reports a non-positive DistinctBy multiplicity, or returns wrong rows. Seed with the full map_batches instead. This costs only momentary memory, since batches are Arc-shared, and cannot double-count: the stream emits each original batch once and never re-emits a merged batch, so incremental batches never carry what the seed already holds. The stream frontier still drives the published upper and the incremental Frontier instructions, which is where it is authoritative. Exposed by removing the interactive-dataflow deferral, which made interactive reads register against an actively-updating arrangement rather than a settled one. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

…rators The dataflow-operator introspection now shows the arrangement-sharing operators (PublishShared and the InspectBatch seal taps) and the ArcBatch batch type name (mz_row_spine::arc_batch::ArcBatch instead of alloc::rc::Rc) that the two-runtime publish path installs. This is expected output, not a behavior change. Rewritten with --rewrite-results under enable_two_runtime_compute; verified stable across repeated runs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

…ream tap A fast-path peek parked on a shared index's seal is re-examined only when the registry wakes its interactive worker. That wake came from an inspect_container tap on the arrangement stream, placed upstream of the publisher sink. The tap fired note_frontier when the frontier reached the tap, but the sink advances the published state.upper a scheduling round later, downstream. So the peek woke, read a stale state.upper, found itself NotReady, and parked. If that frontier advance was the arrangement's last (a static index or view, the common case), no further tap fired and the peek hung. The sink's own post-upper wakeups (the importer activators and the upper_changed condvar) do not reach the peek server loop, so nothing re-woke it. Flaky, because it depended on the tap-versus-sink scheduling gap. Move the seal signal into the publisher sink: adopt takes an on_seal callback that the sink fires once per activation on which state.upper advances, after it releases the state lock. Firing after the lock keeps the wakers lock strictly after the trace state lock, matching the reader order (wakers then state) and avoiding a deadlock. Firing after the advance means a peek the wake re-examines reads the advanced upper and completes. This makes the seal notify program ordered after the state.upper advance, so the lost-wakeup contract on ArrangementSharingRegistry::notify covers the seal, not just publication. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

…nal move Moving the seal signal from the upstream inspect_container taps into the publisher sink removed the InspectBatch tap operators from the dataflow-operator introspection, so relations.slt over-counted them. Rewrite the golden to match. Also demote the intra-doc reference to ArrangementSharingRegistry::notify (a pub(crate) item) from a link to a code span, since adopt is public and lint-doc rejects a public-to-private doc link. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

The cross-runtime arrangement publisher derived its writer-driven compaction floor by reaching into `agent.trace_box_unstable()` (an upstream API documented as unstable, with mutation undefined behavior) plus fork-only TraceBox compaction getters, computing the meet of all other referees' holds minus its own. That was the only remaining reason the differential-dataflow fork was required. Replace it with the authoritative sources Materialize already has. The controller pushes logical compaction through `AllowCompaction`, and physical compaction follows the trace upper (as `TraceManager::maintenance` does). So the publisher now takes its logical floor from the last `AllowCompaction` frontier, forwarded into the shared slot via `ArrangementSharingRegistry::note_allow_compaction` from `handle_allow_compaction`, and its physical floor from the stream `upper` it already holds. `SharedTraceState` gains a `writer_logical` field, seeded `None` so the publisher falls back to its own hold (the `as_of`) before the first command arrives. With the trace-box read gone, nothing links against the fork: `mz_row_spine::ArcBatch` already supplies the cross-thread batch impls, and the sharing primitive lives in `shared_trace.rs`. Remove the `[patch.crates-io]` override for differential-dataflow and differential-dogs3, which resolve to crates.io 0.25.1. This also clears the lint-and-rustfmt dd-git-source deny. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

At 200+50 reads/s the combined rate exceeds both the two-runtime and the single-runtime serving drain, so both runs overload and report queue-wait latency growing to the run length. That masks the isolation the scenario means to show. Halve to 100+25/s. This sits above the baseline drain but below the two-runtime drain, so the comparison separates cleanly: two-runtime holds reads flat (p50 ~11ms, near-zero slope) through hydration churn while the baseline backlogs to tens of seconds. The reported latency now reflects service time rather than an undrained open-loop backlog. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

…isher Two `render::interactive_import_tests` drove compaction by advancing a writer trace handle and relied on the publisher observing it through the trace-box read, which no longer exists. Forward the same frontier through `ArrangementSharingRegistry::note_allow_compaction`, the production `handle_allow_compaction` path, so the publisher advances its `since`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CY7GHdrTBfAG4tgvSgPGJ9

The TLA+ sketch stated I1 and was never run: there is no TLC runner in the repository and no CI job for one. It also turned out to admit a four-step counterexample to its own stated invariant with capping on, which is the release-ordering hole the multiplexer now closes. A stated-but-unchecked property is worse than none, because it reads as assurance. `doc/developer/design/20260720_two_runtime_compute/protocol` replaces it with a Lean 4 model that `ci/test/lean-protocol.sh` checks, wired into the pipeline. The library sets `warningAsError`, so an unproved goal fails the build. That makes the specific failure mode above structurally impossible rather than merely noticed, and the proof holds for all times rather than a finite set of them. Three properties, all corollaries of one inductive invariant proved preserved by every step: `since_le_as_of` (an index is never compacted past the `as_of` of a created, not yet rendered dataflow), `physical_le_since` (the publisher never forwards a physical compaction frontier beyond the published `since`), and `no_regression`. `Step` is parameterised by two booleans selecting the behaviours the implementation used to have, and each has a counterexample: `release_on_drop_violates_invariant` and `physical_from_upper_violates_invariant`. A model that can only express the fixed system cannot tell you it fixed anything. The trade against TLC is real and recorded in the README: TLC searches for counterexamples and Lean does not, so this certifies a fix rather than hunting the next defect. The image follows `doc/developer/semantics/Dockerfile`, minus the Mathlib cache, since the model uses core Lean only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

The read hold on an imported shared arrangement disappeared as soon as the dataflow finished building, so the publisher fell back to the writer-driven frontier and was free to compact past the `as_of` the dataflow still read at. The hold a consumer keeps is the returned `Arranged`'s own trace, and only `mz_join_core` keeps one, by moving its input traces into its operator. `as_collection` and the reduce path take the stream and drop the handle, and the `CollectionBundle` holding it lives in the build-time `Context`, which dies when `build_compute_dataflow` returns. So for every consumer but a join, nothing was left. The import's source operator now owns a hold and downgrades it to the frontier it has acknowledged. That operator lives exactly as long as the dataflow, already tracks the frontier in question, and was until now the one participant holding no registration at all: it captured the bare `Arc<SharedTrace>`. Following `acknowledged` is the import's own obligation and nothing more. Everything at or below it has been delivered and will never be replayed, so this import will not read there again. A consumer that reads the returned trace rather than the stream needs accuracy governed by its own progress, which can lag, and it holds a separate registration for that. The publisher forwards the meet, so the slower of the two wins. The setter joins, which also keeps the hold at `as_of` while the seed drains, since seeded coverage can already lead `as_of`. The physical frontier needs no channel of its own. The publisher derives it from the published `since`, which the logical holds drive, so it follows. `interactive_import_holds_after_construction` covers this with a stream-only consumer and nothing kept alive past the builder. It asserts on the registered holds rather than on a read, because a read cannot distinguish a live hold at some frontier from no hold and the publisher forwarding that same frontier from its fallback, which is exactly the difference at stake. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

Found by an adversarial review of the read-hold and compaction code. Each is confirmed by a test that fails without its fix. **An import reported a physical frontier leading the chain's coverage.** `register_at` seeded the handle's physical frontier from the requested `as_of`, but a consumer checks the reported physical compaction against the coverage it derives from `map_batches`, and `mz_join_core` does so with a plain `assert!` (differential's `join_core` carries the same one). An `as_of` legitimately leads the coverage: an import over a placeholder whose publisher has not adopted it yet sees an empty chain, and a read beyond the index's seal leads it too. So an interactive join over a shared index aborted the worker, and under shared fate the process, on a correct import. It now reports the published `since`, which is what the trace guarantees and what `TraceAgent::clone` inherits. **A released reader could permanently discard the publication point's capability.** `Antichain::join` is absorbing for the empty antichain, so a consumer forwarding an empty input frontier zeroed its own hold, which drove `compaction_target` to the empty frontier, which the publisher forwarded to an agent whose setter joins. The capability was then gone for every future reader. `antichain_meet` treats empty as its identity, so nothing in the published frontiers looked wrong. The reduce operator forwards exactly this on every dataflow whose input finishes. An empty request now releases the hold, and the fold skips empty holds regardless. **The published `since` chased the readers' own holds.** With no controller `AllowCompaction` yet, the writer-driven floor fell back to the publisher's agent hold, which the publisher itself drives up from the meet of the reader holds each activation. That closed a feedback loop: `since` climbed to wherever the readers were, and a later read at an earlier time was refused with a panic naming a frontier no writer ever asked for. The floor is now captured once at adoption. Also: an `errs` refusal reported the `oks` arrangement's diagnostics, because the reporting closure was typed to one handle. Diagnostics now come off the publication point that actually refused, which also stops a failure path registering a hold on its way to a panic. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

The read-hold machinery does not enforce I1 where it has to. Three of the gaps are structural rather than incidental: the multiplexer's cap is per-process-0, so processes at index 1 and above get no cross-runtime ordering at all; it is per-connection, so `Hello` discards it while the runtimes still hold old-epoch commands, and reconciliation's locally synthesized compactions never traverse it anyway; and a hold recorded against one id does not cap compaction for a re-export alias of the same publication point. `read-holds.md` proposes putting acquisition on maintenance's own ordered command stream, which is what can order a process the multiplexer never sees, and leaving downgrade and release intra-process. The asymmetry it rests on: being late to downgrade only delays compaction, while being late to acquire means the merge already ran and coalesced times cannot be recovered. `protocol-holds/` is the TLA+ model, checked by `ci/test/tla-holds.sh` and wired into the pipeline. It earned its keep on the first run by refuting the design as originally written: the release was to travel on maintenance's stream, and TLC found in nine steps that it can overtake a create interactive has not processed, so maintenance applies acquire, release and compaction while the dataflow is still queued and it then renders against compacted data. The release is now on interactive's stream, where it is ordered against the create, and maintenance reclaims by observing the registration disappear. The retired design is kept as `HoldsCap.cfg` and is expected to violate I1, with the runner failing if it stops doing so. A model that can only express the proposed design cannot tell you the design fixed anything. TLA+ rather than Lean because the question is counterexample search across interleavings, not certification, and because the process count is a parameter. The earlier choice of Lean was forced by there being no TLC available, not by fit. The Lean model stays as it is for the single-process core. Also fixes copyright headers the previous commit missed. `bin/lint` only sees tracked files, so running it before `git add` passed vacuously. Lean's block comment syntax is not a prefix the checker recognises, so those headers use line comments, and `lean-toolchain` cannot carry a comment at all and is excluded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

Vocabulary only. Nothing emits `AcquireHolds` or `ReleaseHolds` yet and the compute side panics on receiving one, so this changes no behaviour. It is split out because the exhaustive matches it touches are spread across the controller, the metrics, the history and the multiplexer, and reviewing that mechanical part separately from the semantics is easier. Both are synthesized by `mz_compute_client::multiplex::Multiplexer` rather than issued by the controller, which is why `ComputeCommandHistory::reduce` treats them as unreachable. That is deliberate: on reconnection the controller replays the `CreateDataflow` and the multiplexer re-derives the hold, so the hold lands ahead of the replayed compactions without any state having to survive the reconnection. `AcquireHolds` goes to the runtime that owns the held collections, so that it is ordered against their `AllowCompaction`s within that runtime's own stream. `ReleaseHolds` goes to the runtime that renders the holder, so that it is ordered against that dataflow's create. The TLA+ model under `doc/developer/design/20260720_two_runtime_compute/protocol-holds` forced that asymmetry: a release on the owning runtime's stream can overtake a create the rendering runtime has not processed, so the owner would apply acquire, release and compaction while the dataflow was still queued, and it would then render against compacted data. Payload is boxed, as `CreateDataflow` and `Peek` are, because `ComputeCommand` has a size assertion and every command is cloned into the controller's history. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

Step 0 landed in 9babb21b5b. The rest is not split further because step 1 without step 2 leaks a hold, and step 4, which is where the debt gets paid down, is only safe once 1 to 3 are in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

A dataflow on the interactive runtime that imports a maintenance-owned index needs that index held at its as_of, and it cannot install the hold itself: a TraceAgent is neither Send nor reachable across the runtime boundary. AcquireHolds now installs it on the maintenance worker, from a clone of the trace manager's own handle, which sits at the controller's read frontier and so is low enough to represent the as_of. The publisher's agent is not, because its setter joins and it has followed the controller up. Publishing the pin is as load-bearing as taking it. A reader gates on the published since, so a pin the publication point does not record leaves since at the writer's frontier and handle_at refuses the very reader the pin was for. since is therefore derived from the publisher-driven part and the recorded pins together, recomputed by whichever side moves. The hold follows its readers, floored at its own as_of, rather than sitting at the as_of for the dataflow's life. A frozen hold is a permanent pin, and an interactive SUBSCRIBE lives as long as its client, so its index would never compact again. The floor is what makes following safe without attributing registrations to holders: the meet is at or below every registration, so flooring it cannot carry a hold past its own reader. ReleaseHolds travels on the rendering runtime's stream, as the model requires, and is recorded into the per-process registry for the owning runtime to act on. A release recorded before the matching acquisition is applied, which the two independent streams allow, is consumed by that acquisition, which then installs nothing. Nothing emits these commands yet, so no behaviour changes. Two things the sequence in read-holds.md did not account for are recorded there: handing off to the registration is unsafe, and the reduced command history must drop hold commands, since a replica pushes what it receives into its own history and reducing it hit an unreachable!().

G3 was an artifact of the cap being keyed by collection id. A Trace re-export installs a clone of the same TraceBundle under the second id, so both ids' handles are agents on one TraceBox and share one publication point. A real hold on either id pins the arrangement both name, so step 3 needs no alias closure. Also settles open questions 1 to 3, and records the conservative epoch handling.

The multiplexer now emits AcquireHolds to maintenance when it routes an importing create to interactive, and ReleaseHolds to interactive when that export drops. It no longer modifies compaction frontiers. The guarantee is entirely within maintenance's own stream: this is the only point that observes both, and it is sequential, so the acquisition precedes every compaction that follows the create. Nothing about interactive's stream enters the argument, which is what makes it hold when interactive is arbitrarily behind or never processes the create at all. The release goes to interactive so it is ordered behind the holder's own drop there, the asymmetry the TLA+ model forced. One holder per export, not per dataflow, because the drop that releases is per export and a dataflow's exports may drop at different times. Index imports only: a source import is served from persist and carries its own read hold. This deletes hold_floor, interactive_holds, deferred_compaction, pending_compaction, compaction_floor, retire_hold, release_holds, flush_pending_compaction and the retire-on-response trigger, and with them four defects: the per-query compaction_floor leak, recv performing a send, Hello's epoch exposure, and hold_floor's incomparable- antichain comparator. recv no longer sends at all, so its cancel-safety argument shrinks to nothing being sent. capping_never_regresses_a_compaction_frontier is dropped rather than rewritten. It asserted that a cap never lowers a frontier already sent, and nothing is capped now, so frontiers are forwarded verbatim and the hazard is gone by construction. The new acquire_precedes_the_create_and_compaction_is_not_capped covers the verbatim forwarding. The test harness gains a timeline shared across both mock clients. Per-side command lists cannot express the ordering that is the whole mechanism.

The design sections described mechanisms that did not survive contact with the code: per-registration agents, nonce-carrying holds, and an alias closure. Replaced with what landed, including why the acquired hold has to follow its readers and why the release travels on the other stream.

The spec described the design as drafted, which the implementation diverged from three ways. A model describing a system nobody built reads as assurance, so it is worse than no model. Brought into line: the acquired hold follows its reader instead of sitting at the as_of, reclaim is driven by an explicit release the rendering runtime records rather than by watching a registration appear and go away, and the drop and the release are two commands on that runtime's stream in that order rather than one. Three things fell out of doing it. I1 was wrong for the shipped design. It would have flagged a correct downgrade, because once the reader advances the collection may legitimately compact past the original as_of. It is now two windows: protected at the as_of before the dataflow is built on a process, at the reader's current hold afterwards. Sabotaging the downgrade to overshoot its reader violates it, so it discriminates. The no-permanent-pin property cannot be an invariant. Reclaim and downgrade are separate steps, so there is always a legal state where the release has been applied and the hold is still there, and a safety version flags exactly that state. It is now a liveness property against a fair spec. Checking it surfaced a dependency nobody had written down: the release reaches the owning runtime only through the rendering runtime applying a command, so a runtime that stops draining its queue strands the hold forever. Fairness now states that the runtimes drain, which is what each worker's server loop does. MaintStep deliberately treats only the command-acquired hold as a bound and ignores the reader's own registration. That registration is forwarded through one agent whose setter joins, so it cannot represent a frontier below where the agent already sits. Treating it as a bound would assume away the ratchet the acquired hold exists for. HoldsReleaseOnMaint.cfg is new and keeps run one's refutation live: the release on the owning runtime's stream, which the asymmetry in the code exists to avoid. Both refutations still violate I1 and the runner fails if either stops.

Formatting only. The resolution of the upstream index-peek-rows metrics against the peek offload restructure left one initializer wrapped that fits on a line.

Upstream MaterializeInc#38170 removed mz-compute-client from clusterd as an unused dependency, which it was on main. This branch uses it: `lib.rs` constructs the two-runtime `Multiplexer` from it. A semantic conflict git cannot see. Both sides are individually correct and the files do not overlap, so the rebase reported nothing and the breakage only showed up building clusterd, which no per-crate check of the compute crates reaches.

`bin/doc --document-private-items` runs with `-D warnings`, so `rustdoc::private-intra-doc-links` is an error: a public module doc may not link to a private or crate-private item, because the link resolves only under `--document-private-items` and breaks without it. Both references become plain code spans. They point at internal mechanics that the surrounding prose already explains, so nothing is lost by not linking.

A published arrangement forwarded the published `since` as its physical compaction target. That conflates the two compaction kinds: logical compaction decides which times stay distinguishable, physical compaction decides which batches may merge. Holding physical down to `since` kept a batch boundary at every frontier at or above the controller's read frontier, so the spine stopped merging, and it did so for every published index whether or not anything ever imported it. Measured on an index with no importer at all: 5 to 7 batches where an unshared index reached 1. A reader does not need a boundary at its `as_of`. `import_snapshot_at` seeds it with the whole chain and wraps the handle in `TraceFrontier`, which advances times rather than cutting, so a batch straddling `as_of` is harmless. Batches are immutable, so a merge after a seed is captured cannot disturb it, and `acknowledged` starts at that seed's coverage and only rises. What a reader does need is a boundary at each frontier it later passes to `cursor_through`. `mz_join_core` already declares exactly that through `set_physical_compaction`, as differential intends, but `SharedTraceHandle` recorded the request in a local field and dropped it. Mirror it into the publication point as `physical_holds` instead, and forward their meet, falling back to the chain coverage when no reader is registered. A floor starts at the coverage at registration, never at `since` (a logical frontier, which is the conflation this commit removes) and never at `upper`, which can lead the coverage and so sit above the seed a reader registering now receives. The straddle check in `SharedTraceHandle::batches_through` stays. It is the detector for a reader that needs a boundary this forwarding merged away, and deleting it would leave a consumer silently double counting updates at times not before its cut. `stale_as_of_import_over_merged_chain_matches_direct` asserted that no batch straddles the `as_of`. That was a restatement of the old forwarding rather than a property a reader depends on, so it now asserts the straddle, which makes it the regression test for reintroducing a collective floor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

Every hold command in this design reconstructs an ordering that routing destroyed: `CreateDataflow` goes only to the rendering runtime while `AllowCompaction` goes to the owning one, so the two are no longer on a single ordered stream. The alternative is to stop routing compaction and hand it to both runtimes, which restores that ordering directly. Model both variants so TLC decides between them rather than leaving the argument in a comment. `broadcast` hands the frontier to both runtimes and adds nothing else. It violates I1 in five states: the controller creates a dataflow and immediately drops it, as a cancelled peek does, so its own read hold is gone before the rendering runtime has applied either command. Compaction then advances on the owning runtime while the create is still queued, and the dataflow is later built at an `as_of` the collection has compacted past. Restoring the ordering within each stream does not restore it between them. `broadcast-standing` adds the standing per-collection hold: the rendering runtime holds every shared collection at the last frontier it has applied, so the owning runtime cannot realize a frontier that runtime has not reached. It holds over 19940 distinct states. The invariant is that a shared arrangement compacts only as fast as the slowest runtime's stream position, which is derived from the importer rather than from the controller's per-dataflow bookkeeping, and it needs no acquisition, no release and no reclaim. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

Records the conclusion the model reached and what follows from it. The diagnosis in read-holds.md stands: splitting compute keeps the controller's global floor and loses the single ordered command stream. The mechanism does not. Every version of it reconstructs that ordering on top of the routing that lost it, where sending compaction to both runtimes restores it directly. Adds broadcast-compaction.md with the design, the refutation of broadcast without a hold, the invariant it buys, the list of what it deletes, and the two unmeasured costs. Marks read-holds.md as superseded while keeping its diagnosis and its record of what each attempt cost, since that is what makes the simpler mechanism defensible rather than merely simpler. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

An index's `since` must not pass the `as_of` of a dataflow importing it. A single command stream ordered the create against every later compaction. Splitting compute across two runtimes routed those two commands to different streams, which lost that ordering. Broadcast `AllowCompaction` for maintenance-owned collections to the interactive runtime as well. It applies the frontier as a standing hold on the shared arrangement, and the publisher bounds its logical target by that hold, so a shared arrangement compacts only as fast as the slowest runtime's stream position. An importing dataflow whose `CreateDataflow` is still queued on the interactive runtime has registered no reader hold yet, so this is the only thing keeping the arrangement at or below the `as_of` it is about to read at. The hold is seeded at the publisher's compaction frontier at adoption rather than at the minimum time. The controller offers no `as_of` below a collection's own `since`, so no importer can need a frontier below it, and the seed keeps a collection the controller never compacts again from being pinned forever. Tests at three levels, each verified by sabotaging the mechanism it covers: the publication point, the registry and import path, and the multiplexer's routing. The publisher also carries a `debug_assert` that the published `since` never passes the hold. The acquisition layer (`AcquireHolds`/`ReleaseHolds`) is untouched and now redundant with this. Its deletion follows in a separate commit, so the invariant is asserted before anything is removed on the strength of it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

The standing hold makes every acquisition redundant. It bounds the publisher's compaction by the importing runtime's own stream position, so the publisher's agent never sits above an `as_of` that runtime can still present, which is the ratchet the acquisition existed to escape. Deleted: `ComputeCommand::AcquireHolds`/`ReleaseHolds` and `HoldRequest`, `compute_state/command_hold.rs`, `ComputeState::command_holds` and its maintenance pass, the registry's release records (`release_holder`/`reclaim_holder`/`released_holders`/`clear_released`), the multiplexer's `held_exports` and hold synthesis, and the `history.rs` arms. Two more things fell out, both of which existed only because a command hold was a second writer of the published `since`. `SharedTraceState::writer_since` and `refresh_since` go, because with one writer left the publisher assigns `since` directly. So do the four `Published` methods that recorded a hold the point did not own, and with them the distinction between a hold that is a request forwarded through the publisher's agent and a hold that is a grant backed by someone else's handle. Every hold is a request again. The replica no longer discards anything at a reconnection. A standing hold is per collection, carries no dataflow identity, and only rises, so clearing it would drop the bound to the minimum time until the replayed compactions raised it again. Four render tests covering the acquisition go. What they tested, a read at an `as_of` below where the publisher's agent already sits, is covered at the same level by `standing_hold_pins_until_the_importing_runtime_applies`. The multiplexer's ordering test keeps its other half, that compaction is forwarded uncapped, and its `Hello` test now asserts what `reset` still does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

The spec described an acquisition protocol the code no longer has. It now models what the code does: compaction broadcast to both runtimes, and the rendering runtime's applied frontier bounding the publisher. Gone with the acquisition: the `acquire` and `release-on-maint` mechanisms and their variables, the `cap` mechanism and the frontier report it retired on, and the properties that had them as their subject. `Holds.cfg` is the shipped design rather than a retired one, and `HoldsRouted.cfg` replaces `HoldsCap.cfg` as the second refutation: compaction routed to the owning runtime alone, which is the raw defect the runtime split introduced. It fails in the same six states as broadcast-alone, so the pair shows that broadcasting the command restores the ordering within each stream and buys nothing for that counterexample. Applying a compaction and publishing one are now separate actions. The previous spec recomputed the published frontier only when a command arrived, which models a publisher that never notices its standing hold catching up. Splitting them is what makes the liveness question expressible, and `CompactionNotStalled` states it: the bound a lagging rendering runtime imposes is temporary. It holds only because `Fairness` says the runtimes drain, which is the coupling this design introduces. The reader's own registration is a bound now. It could not be one under the acquisition mechanism, since a registration forwarded through a single joining agent cannot represent a frontier below where that agent sits. Bounding the agent by the standing hold removes that ratchet, which is the same reason the acquisition became unnecessary. `Holds.cfg` holds over 7845 distinct states, depth 19. Both refutations violate I1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

The broadcast reached the interactive runtime for every maintenance-owned collection, including the transient ones. Maintenance renders its own transient collections, its subscribes and copy-tos, and those are sinks with no arrangement for anything to import. The interactive runtime has installed nothing under such an id, so applying the frontier for one took the local path and the drop in that sequence panicked in `drop_collection` with "dropped untracked collection". Two fixes, each of which is sufficient on its own. Narrow the broadcast to the collections the interactive runtime can import, which are the non-transient ones maintenance publishes. That is the truthful scope: a standing hold exists to keep an importable arrangement readable. Decide local work from `collections` rather than from the id. The id was the wrong question in both directions: the interactive runtime holds empty local copies of the peer's introspection indexes, whose ids are the peer's to publish, and the peer renders transient collections this runtime has never seen. So a stray broadcast can no longer reach the drop path at all. Reproduced with `bin/sqllogictest --optimized -- --system-parameter-default=enable_two_runtime_compute=true` over subscribe_outputs.slt and cursor.slt, which panic before either fix and pass with either one. 836 assertions pass over the nine SUBSCRIBE-carrying files. Also drop a rustdoc link from a public module doc to a private field, which `ci/test/lint-doc.sh` rejects under `-D rustdoc::private-intra-doc-links`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

A refusal says the arrangement compacted past the dataflow's `as_of`, and the existing diagnostics cannot say whether anything on the replica could have prevented it. The standing hold is the value that decides: equal to the refusing `since` means the importing runtime had already applied that compaction before it built the dataflow, so the create was ordered behind it on that runtime's own stream and the arrangement was legitimately compacted, with no replica-side hold able to help. Below it means the publisher escaped its bound, which is a defect here. The publisher's `debug_assert` covers the second case but compiles out under `--optimized`, which is what sqllogictest and mzcompose run, so the panic path has to carry it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

`enable_two_runtime_compute` named a topology, two runtimes, rather than the thing it turns on. What a reader wants from the name is which mechanism appears, and the mechanism is an interactive runtime that serves reads off the arrangements the maintenance runtime builds. The new name also joins the established `enable_compute_*` family alongside `enable_compute_peek_response_stash` and `enable_compute_replica_expiration`, so it sorts and reads with its siblings. Renames the const and the parameter name together with all three Python registration sites, since `check-test-flags.sh` requires the mzcompose variable parameters and the parallel-workload flag list to agree with the compiled set. NOTE: the parameter name is also the LaunchDarkly key. The old key carries a staging rule enabling the feature for the read-placement evaluation environment, and nothing reads that key after this commit, so the new key has to be created before the next staging measurement or the environment silently falls back to the compiled default of off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

Placement and preemption are independent: a walk off the timely worker can still have yield points, which buys somewhere to observe a cancellation request and a bound on how long one walk pins its batches. So S1 and S5 are not two values of one setting, and the fourth cell of that grid is an untried candidate rather than a contradiction. Records what each cell has measured and names the two that decide the question, both empty: S1 on M1 is predicted throughout and never measured, and nothing at all is known about an off-worker walk that yields. The deciding experiment runs S1 against S5 on E1, E11 and E8b, with the rule registered up front, and needs neither the interactive runtime nor this branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

The design of record, the evaluation record and the two machine-checked protocol models land separately in MaterializeInc#38239, along with their CI checks. They have no dependency on this code and reviewing 4,700 lines of prose against 7,000 lines of compute is worse for both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

The offload walks a fast-path index peek's cursor on a blocking task instead of on the serving worker. It is orthogonal to the interactive runtime: it needs no second runtime, the interactive runtime needs no offload, and the two were only ever in one branch because they were written together. Keeping them together also argues two things at once, and one of them is now a question rather than a claim. Cooperative peek yielding is a second candidate for the same defect, placement and preemption turn out to be independent axes so there is a fourth untried combination, and the experiment that decides between them has not run. Landing a mechanism that measurement may retire is the expensive order, since a merged mechanism is harder to remove than an unmerged one. What this leaves is the interactive runtime alone, whose case is E7 and E9: temporary dataflows under maintenance load, and a replica that stays introspectable while it hydrates. E1 and E11 go with the offload, which is where E2 showed they belonged. Interactive peeks are unaffected. They resolve through `shared_index_peek_response`, and the offload was a conditional in front of that rather than the path itself. Removes `local_snapshot.rs`, `PendingPeek::IndexOffload` and its walk plumbing, the two dyncfgs, `mz_index_peek_walks_total`, `PeekStash::upload_blocking`, and the in-flight accounting. `static_assertions` goes with them, having been used only for the `Send` assertion on the owned snapshot. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

…nct work Resolutions for the rebase onto main after `ComputeRuntimeRole` and the error-multiplicity collapse landed separately. The role enum arrived on main with `Interactive` gated behind `cfg(test)`, since nothing outside tests could construct it yet. This branch constructs it, so the enum and its impl keep this branch's copy, which drops the gate and adds `publishes`. Main's copy is removed rather than merged: the two differ only in the gate and the accompanying notes about it. `distinct_errs` and `bundle_errs` both match exhaustively on `ArrangementFlavor`, and neither knew about `SharedTrace`. A shared trace is an imported arrangement like `Trace`, so it takes the same treatment in both: `distinct_errs` leaves it alone because this dataflow cannot rewrite an imported error trace in place, and `bundle_errs` yields its error collection so a delta join propagates those errors once rather than once per path.

Takes an owned, `Send` snapshot of a peek's cursor and walks it on a blocking task instead of inline on the timely worker that received it, so a long scan no longer delays the peeks queued behind it. The snapshot owns the `Arc` batches its cursor covers, which is what makes it `Send`: the traces this crate maintains are read through an `Rc`-based reader, so borrowing from one, or owning `Rc` batches, would not cross a thread. Generic over the cursor source, so a worker-local `TraceBundle` and a registry `SharedTraceHandle` feed the same walk. Bounded by `index_peek_offload_max_inflight`, because each in-flight walk retains the batches its cursor covers. `mz_index_peek_walks_total{substrate}` exists so that "the offload changed nothing" and "the offload never engaged" are distinguishable, which cost a round of staging measurement before it did. And an offloaded walk can divert to the peek response stash partway through, without which the feature is unreachable in a production configuration, since production runs the stash on. PARKED. This is one of four candidates for the same defect, and the experiment that chooses between them has not run. Cooperative peek yielding (MaterializeInc#38040) is a second, and placement and preemption are independent axes, so an off-worker walk that yields is an untried fourth. Measurement already found this mechanism reproducibly worse than doing nothing when a peek queues behind a long operator activation, and E2 found it is not what the second runtime needs. The decision rule is registered in `peek-placement.md`: if yielding alone matches this on E1, E11 and E8b, this should be deleted rather than merged. It exists as a branch so the experiment can deploy it, not because it is on its way in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019G29DBfgE8LXpE5jamm2Zi

The placement axis is orthogonal to the interactive runtime, so its rationale and its stash-plumbing plan belong with the change that implements it rather than with the two-runtime design document. `peek-placement.md` records why this is parked rather than dropped: placement and preemption are independent axes, the deciding experiment needs only fixtures that already exist, and a merged mechanism is harder to delete than an unmerged one. `peek-offload-stash-plan.md` records the plumbing that lets an offloaded walk use the peek response stash. Both previously lived on the design branch. The evaluation they cite moved to the project's experimental-evaluation document.

antiguru and others added 30 commits August 21, 2026 09:19

antiguru and others added 28 commits August 21, 2026 09:19

compute: format the rebase conflict resolution

87075f8

Formatting only. The resolution of the upstream index-peek-rows metrics against the peek offload restructure left one initializer wrapped that fits on a line.

antiguru force-pushed the mh/peek-placement branch from a5c9803 to f73b5ce Compare August 21, 2026 07:39

Navigation Menu

Uh oh!

compute: walk fast-path index peeks off the serving worker - #38242

compute: walk fast-path index peeks off the serving worker#38242
antiguru wants to merge 120 commits into
MaterializeInc:mainfrom
antiguru:mh/peek-placement

antiguru commented Aug 17, 2026 •

edited by github-actions Bot

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

antiguru commented Aug 17, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Status: parked, not seeking review

What it does

Stacking

Verification

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

antiguru commented Aug 17, 2026 •

edited by github-actions Bot

Loading