Skip to content

Add persist-backed compute introspection sources#35172

Draft
antiguru wants to merge 1 commit intoMaterializeInc:mainfrom
antiguru:persist_introspection
Draft

Add persist-backed compute introspection sources#35172
antiguru wants to merge 1 commit intoMaterializeInc:mainfrom
antiguru:persist_introspection

Conversation

@antiguru
Copy link
Member

Reintroduces direct persist writes from compute replicas for introspection data, bypassing environmentd for the data path.
The coordinator's role is limited to shard allocation and lifecycle management.

  • Per-replica persist shards for all ~30 log variants via the MV persist sink
  • CREATE CLUSTER ... (PERSIST INTROSPECTION = true) opt-in + enable_persist_introspection dyncfg kill switch
  • LoggingConfig.sink_logs carries shard metadata to replicas
  • Deterministic GlobalId::PersistedIntrospectionSource encoding per replica+variant
  • Shard allocation on replica create, wired through coordinator → controller → compute
  • mzcompose integration tests for syntax, dyncfg, lifecycle, restart, catalog changes

🤖 Generated with Claude Code

@github-actions
Copy link

Thanks for opening this PR! Here are a few tips to help make the review process smooth for everyone.

PR title guidelines

  • Use imperative mood: "Fix X" not "Fixed X" or "Fixes X"
  • Be specific: "Fix panic in catalog sync when controller restarts" not "Fix bug" or "Update catalog code"
  • Prefix with area if helpful: compute: , storage: , adapter: , sql:

Pre-merge checklist

  • The PR title is descriptive and will make sense in the git log.
  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).

@antiguru antiguru force-pushed the persist_introspection branch 7 times, most recently from 18535d5 to c8bd29f Compare February 26, 2026 14:25
Adds per-replica introspection sources backed by persist shards,
enabling queryable introspection data that survives replica restarts.

Each replica gets a dedicated schema (mz_introspection_{cluster}_{replica})
containing sources and views derived from compute log variants. Sources
live in a separate durable collection while views use the regular items
collection. Cleanup is handled via the dependency system through
cluster_replica_dependents → schema_dependents.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@antiguru antiguru force-pushed the persist_introspection branch from c8bd29f to 41d4407 Compare February 26, 2026 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant