Skip to content

compute: import published indexes as a shared arrangement - #38390

Draft
antiguru wants to merge 1 commit into
mh/interactive-04-publishfrom
mh/interactive-05-import
Draft

compute: import published indexes as a shared arrangement#38390
antiguru wants to merge 1 commit into
mh/interactive-04-publishfrom
mh/interactive-05-import

Conversation

@antiguru

Copy link
Copy Markdown
Member

Fifth of eight PRs splitting #37770. Stacks on #38389. Tracked by CPU-215.

ArrangementFlavor gains a SharedTrace variant carrying arrangements backed by TraceFrontier<SharedTraceHandle>, and import_index_shared constructs it on the interactive runtime instead of reading the local TraceManager, which holds nothing there. The shared handle shares the RowRow/Err batch and cursor types of a maintenance TraceAgent, so the variant flows through the same generic bodies as Trace and a downstream Get receives a real arrangement with its key and permutation intact, rather than a collection the plan re-derives. Joins, delta joins, thresholds, and hydration logging gain the corresponding arm.

The import is a static snapshot at as_of, bounded one step past it rather than by self.until: interactive work is single-time, so the capability must drop once the shared trace seals past as_of for the one-shot result to complete, and an unbounded until never gets there.

The read hold is the returned Arranged's own trace rather than a separate token, so consumers can downgrade it as their frontiers advance. A token nobody downgrades would be a floor under every hold that is downgraded, since the publisher forwards the meet.

The variant and its constructor land together because a never-constructed variant of an enum inside the private render module trips dead_code, and CI lints with -D warnings. Reachable only when a runtime holds the Interactive role, which nothing constructs yet.

@antiguru
antiguru force-pushed the mh/interactive-05-import branch from cf8f1f6 to 5c374d3 Compare August 21, 2026 11:23
@antiguru
antiguru force-pushed the mh/interactive-05-import branch from 5c374d3 to 1bce309 Compare August 21, 2026 13:24
@antiguru
antiguru force-pushed the mh/interactive-05-import branch from 1bce309 to 35cb94e Compare August 21, 2026 13:42
@antiguru
antiguru requested a review from DAlperin August 21, 2026 13:46
@antiguru
antiguru force-pushed the mh/interactive-05-import branch from 35cb94e to f92f260 Compare August 21, 2026 14:31
`ArrangementFlavor` gains a `SharedTrace` variant carrying arrangements backed by
`TraceFrontier<SharedTraceHandle>`, and `import_index_shared` constructs it on the
interactive runtime instead of reading the local `TraceManager`, which holds
nothing there. Because the shared handle shares the `RowRow`/`Err` batch and
cursor types of a maintenance `TraceAgent`, the variant flows through the same
generic bodies as `Trace`, so a downstream `Get` receives a real arrangement with
its key and permutation intact rather than a collection the plan has to
re-derive. Joins, delta joins, thresholds, and hydration logging gain the
corresponding arm.

The import is a static snapshot at `as_of`, bounded one step past it rather than
by `self.until`. Interactive work is single-time, so the capability must drop once
the shared trace seals past `as_of` for the one-shot result to complete, and an
unbounded `until` never gets there. An `as_of` at `Timestamp::MAX` yields the
empty bound, which is the right reading of "the final state".

The read hold is the returned `Arranged`'s own trace rather than a separate token,
so consumers can downgrade it as their frontiers advance. A token nobody
downgrades would be a floor under every hold that is downgraded, because the
publisher forwards the meet.

The two export paths reject `SharedTrace` as unreachable. Only the interactive
runtime produces the variant, and its exports are freshly rendered query outputs,
never a re-export of an import.

Both `import_index_shared` and the variant are reachable only when a runtime holds
the `Interactive` role, which nothing constructs yet.

The two `sharing` tests that build this variant return here, along with the
`consolidate_capture` helper they share. `render`'s tests move out of line to
`render/tests.rs`, per the convention in `src/compute/AGENTS.md`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@antiguru
antiguru force-pushed the mh/interactive-05-import branch from f92f260 to aa7c8eb Compare August 21, 2026 17:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant