compute: serve fast-path peeks on the interactive runtime by antiguru · Pull Request #38392 · MaterializeInc/materialize

antiguru · 2026-08-21T10:22:36Z

Seventh of eight PRs splitting #37770. Stacks on #38391. Tracked by CPU-216.

The interactive runtime holds no local traces, so an index peek there resolves against the sharing registry. Such peeks live in pending_work keyed by a WorkId and indexed by the id they wait on, not in pending_peeks. A publication or seal marks that id dirty and wakes the worker, which re-examines exactly the items indexed under the changed ids, so wakeups scale with what changed rather than with total pending work. Fast-path persist peeks still land in pending_peeks on every runtime, because their persist-read task wakes the worker with no dirty signal.

Interactive dataflows build immediately in command arrival order rather than deferring until their dependency is published. An import over an unadopted placeholder produces no data and holds its output frontier at the minimum until a publisher adopts the same slot, so late binding replaces the deferral.

The interactive runtime reports only its transient collections' frontiers, since it shares the identity of every non-transient collection with maintenance and the controller keeps one frontier stream per collection. For the same reason its logging is forced off: it serves introspection peeks from maintenance's published copies. The consequence is that nothing the interactive runtime does appears in introspection, tracked as CPU-222. That wants resolving before the flag is turned on anywhere it would hide real work.

Reconciliation drops pending_work and dep_index. The standing holds in the registry deliberately survive: one is per collection, carries no dataflow identity, and only rises, so clearing it would drop the arrangement's bound to the minimum until replayed compactions raised it again.

Reachable only on a runtime holding the Interactive role, which requires the dyncfg that is still off everywhere.

antiguru · 2026-08-21T12:14:16Z

-    pub(crate) fn role(&self) -> ComputeRuntimeRole {
-        self.role
-    }
-


Function moved for no particular reason.

Agreed, no reason. Moved back to where #38389 introduced it, right after new(). This was churn imported by taking compute_state.rs wholesale from the original branch. (Posted by Claude Code.)

antiguru · 2026-08-21T12:15:53Z

-        let suspension_token = self.compute_state.suspended_collections.remove(&id);
-        drop(suspension_token);
+        //
+        // Every dataflow builds immediately on `CreateDataflow`, inserting its
+        // `suspended_collections` entry before the `Schedule` that follows in arrival order can
+        // reach us. A `Schedule` with no entry is therefore a stray or duplicate command, a silent
+        // no-op.
+        if let Some(suspension_token) = self.compute_state.suspended_collections.remove(&id) {
+            drop(suspension_token);
+        }


Dropping Option<T> versus dropping Some(T) should be equivalent. This change seems avoidable?

Correct, they are equivalent. Reverted to the original two lines and kept only the comment about a Schedule with no entry being a stray. (Posted by Claude Code.)

antiguru · 2026-08-21T12:18:34Z

-    /// Attempts to fulfill the peek and reports success.
-    ///
-    /// To produce output at `peek.timestamp`, we must be certain that
-    /// it is no longer changing. A trace guarantees that all future
-    /// changes will be greater than or equal to an element of `upper`.
-    ///
-    /// If an element of `upper` is less or equal to `peek.timestamp`,
-    /// then there can be further updates that would change the output.
-    /// If no element of `upper` is less or equal to `peek.timestamp`,
-    /// then for any time `t` less or equal to `peek.timestamp` it is
-    /// not the case that `upper` is less or equal to that timestamp,
-    /// and so the result cannot further evolve.


Comment vanished.

Restored verbatim. The signature and body of seek_fulfillment are unchanged from main, so it applies as written. Same wholesale-copy churn as the moved role(). (Posted by Claude Code.)

antiguru · 2026-08-21T12:19:17Z

+            // NOTE: the standing holds in the sharing registry are deliberately NOT cleared here.
+            // One is per collection and carries no dataflow identity, so it cannot go stale across a
+            // reconnection, and it only ever rises. Clearing it would drop the arrangement's bound to
+            // the minimum time until the replayed compactions raised it again. See
+            // `doc/developer/design/20260720_two_runtime_compute/design.md`.


Is this still correct?

Yes, still correct. design.md states it: a standing hold is per collection and carries no dataflow identity, so a replayed dataflow receiving a fresh transient id cannot conflate two holders, and the replica clears nothing at a reconnection. Clearing would reset the hold to the minimum time until the replayed compactions raised it again. The reference used to point at broadcast-compaction.md, which no longer exists; repointed to design.md. Note the hold's shape is about to change: it becomes an ordinary hold in a MutableAntichain in the next pass, so this NOTE will be rewritten. (Posted by Claude Code.)

The interactive runtime holds no local traces, so an index peek there resolves against the sharing registry instead. Such peeks live in `pending_work` keyed by a `WorkId` and indexed by the id they wait on, not in `pending_peeks`. A publication or seal marks that id dirty and wakes the worker, which re-examines exactly the items indexed under the ids that changed, so wakeups scale with what changed rather than with total pending work. Fast-path persist peeks still land in `pending_peeks` on every runtime, because their persist-read task wakes the worker with no dirty signal, which is why the every-step scan stays. Interactive dataflows build immediately in command arrival order rather than deferring until their dependency is published. An import over an unadopted placeholder produces no data and holds its output frontier at the minimum until a publisher adopts the same slot, so late binding replaces the deferral. The interactive runtime reports only its transient collections' frontiers. It shares the identity of every non-transient collection with maintenance, which owns and reports the real frontiers, and the controller keeps one frontier stream per collection, so reporting the shared ones would race the owner and regress it. For the same reason its logging is forced off: it serves introspection peeks from maintenance's published copies, and its own empty copies would clobber them. The consequence is that nothing the interactive runtime does appears in introspection, tracked as CPU-222. Reconciliation drops `pending_work` and `dep_index`, whose peeks belong to the reconciled-away connection. The standing holds in the registry deliberately survive: one is per collection, carries no dataflow identity, and only rises, so clearing it would drop the arrangement's bound to the minimum until replayed compactions raised it again. Reachable only on a runtime holding the `Interactive` role, which requires the dyncfg that is still off everywhere. Tests are out of line in `compute_state/tests.rs`, per the convention in `src/compute/AGENTS.md`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

antiguru mentioned this pull request Aug 21, 2026

compute: turn the interactive runtime on in test configurations #38393

Draft

antiguru force-pushed the mh/interactive-07-peeks branch from cf82e1f to 77e275a Compare August 21, 2026 11:23

antiguru commented Aug 21, 2026

View reviewed changes

antiguru force-pushed the mh/interactive-07-peeks branch 2 times, most recently from 6984b18 to f359f28 Compare August 21, 2026 13:24

antiguru force-pushed the mh/interactive-07-peeks branch from f359f28 to f45dc4c Compare August 21, 2026 13:42

antiguru requested a review from DAlperin August 21, 2026 13:47

antiguru force-pushed the mh/interactive-07-peeks branch from f45dc4c to 40e6744 Compare August 21, 2026 14:31

antiguru force-pushed the mh/interactive-07-peeks branch from 40e6744 to 0ba0d59 Compare August 21, 2026 17:54

Navigation Menu

Uh oh!

compute: serve fast-path peeks on the interactive runtime - #38392

compute: serve fast-path peeks on the interactive runtime#38392
antiguru wants to merge 1 commit into
mh/interactive-06-runtimefrom
mh/interactive-07-peeks

antiguru commented Aug 21, 2026

Uh oh!

antiguru Aug 21, 2026

Uh oh!

antiguru Aug 21, 2026

Uh oh!

antiguru Aug 21, 2026

Uh oh!

antiguru Aug 21, 2026

Uh oh!

antiguru Aug 21, 2026

Uh oh!

antiguru Aug 21, 2026

Uh oh!

antiguru Aug 21, 2026

Uh oh!

antiguru Aug 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

antiguru commented Aug 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant