compute: turn the interactive runtime on in test configurations - #38393
Draft
antiguru wants to merge 1 commit into
Draft
compute: turn the interactive runtime on in test configurations#38393antiguru wants to merge 1 commit into
antiguru wants to merge 1 commit into
Conversation
antiguru
force-pushed
the
mh/interactive-08-enable
branch
2 times, most recently
from
August 21, 2026 11:50
7cc7c1e to
edb0ed3
Compare
antiguru
force-pushed
the
mh/interactive-08-enable
branch
from
August 21, 2026 12:28
edb0ed3 to
33f6773
Compare
antiguru
force-pushed
the
mh/interactive-08-enable
branch
from
August 21, 2026 13:24
33f6773 to
2f0fd74
Compare
This was referenced Aug 21, 2026
antiguru
force-pushed
the
mh/interactive-08-enable
branch
from
August 21, 2026 13:42
2f0fd74 to
69cfb34
Compare
antiguru
force-pushed
the
mh/interactive-08-enable
branch
from
August 21, 2026 14:31
69cfb34 to
51bbf8a
Compare
pull Bot
pushed a commit
to Arstman/materialize
that referenced
this pull request
Aug 21, 2026
…nt sharing (MaterializeInc#38396) Replaces MaterializeInc#37881, whose head branch lives on a fork and so cannot be the base of a stacked PR in this repository. Same commits, same tree, on an upstream branch instead. This is the root of the stack MaterializeInc#38386 through MaterializeInc#38393, which splits MaterializeInc#37770. ### Motivation Cross-runtime arrangement sharing (the two-runtime read-isolation work, MaterializeInc#37770) needs batches readable from a thread other than the one maintaining the trace. Differential's default spines reference-count batches with `Rc`, which is worker-local. ### Description Introduce `mz_row_spine::ArcBatch`, a local newtype around `Arc<B>` that carries differential's batch traits (the orphan rule forbids the blanket impl on a bare `Arc<B>`), and switch the production spines and their builders — `RowRowSpine`, `RowValSpine`, `RowSpine`, `ValRowSpine`, `ColValSpine`, `ColKeySpine` — from `Rc`/`RcBuilder` to `ArcBatch`/`ArcBuilder`. An `Arc`-backed batch whose contents are `Send + Sync` can be read across threads, which `Rc` cannot do. Only the batch handle becomes atomic; the batch contents are unchanged, so the cost is a marginally more expensive refcount. Also adds generic `ArcOrdVal`/`ArcOrdKeySpine` aliases for callers outside `mz_compute`, adapts batch-size logging (`log_arrangement_size_inner`) to reach through the newtype to the inner `Arc`, and switches the storage sink trace to the `Arc`-backed spine. Builds against released differential-dataflow 0.25 with no fork or `[patch.crates-io]`. ### Verification `cargo check --workspace` passes with no `Cargo.lock` churn. `relations.slt`'s golden is rewritten because the spine type name appears in operator names. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
`enable_compute_interactive_runtime` becomes a variable system parameter defaulting to on, which is what makes every preceding piece of this work reachable: sqllogictest, testdrive, and the mzcompose suites now provision replicas with two runtimes, so peeks and bounded transient dataflows route to the interactive runtime and maintenance publishes its indexes for it to read. Production keeps the dyncfg's own default, which is off. The clusterd mzcompose service grows the second runtime's port and `--interactive-compute-timely-config`, mirroring what the controller passes in a real deployment. Two clusterd-test-driver specs cover an index read and a query dataflow across the runtime boundary, and a parallel-benchmark scenario measures read isolation with the feature on against off. Two goldens move. `relations.slt` gains the publisher operators, which are real operators the maintenance runtime now installs on every published index. `introspection-sources.td` raises a coarse arrangement-size bound from 16 KiB to 32 KiB, because publication raises the reported size of a one-record index past the old bound. Whether that overhead is constant per arrangement or scales with size is not established and wants re-measuring, so the comment records the measurement without claiming a mechanism. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
antiguru
force-pushed
the
mh/interactive-08-enable
branch
from
August 21, 2026 17:54
51bbf8a to
4f02fc3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Last of eight PRs splitting #37770. Stacks on #38392. Tracked by CPU-216.
enable_compute_interactive_runtimemoves out ofUNINTERESTING_SYSTEM_PARAMETERS, where #38391 parked it, and becomes a variable system parameter defaulting to on. That is what makes every preceding piece reachable: sqllogictest, testdrive, and the mzcompose suites provision replicas with two runtimes. Parallel-workload'sFlipFlagsActionis unpinned to both values at the same time. Production keeps the dyncfg's own default, off.The clusterd mzcompose service grows the second runtime's port and
--interactive-compute-timely-config. Two clusterd-test-driver specs cover an index read and a query dataflow across the runtime boundary, and a parallel-benchmark scenario measures read isolation with the feature on against off.relations.sltgains the publisher operators, and pins the flag itself rather than inheriting the environment default. A variable system parameter can be flipped byCI_SYSTEM_PARAMETERS=random, which would otherwise make the golden mismatch about half the time in randomized runs. The flag is read when a replica is provisioned, and the file creates its own cluster afterreset-server, so setting it beforehand is enough; the file now passes with the flag defaulted either way.introspection-sources.tdraises a coarse arrangement-size bound from 16 KiB to 32 KiB, since publication raises the reported size of a one-record index past the old bound. Whether that overhead is constant per arrangement or scales with size is not established and wants re-measuring, so the comment records the measurement without claiming a mechanism.