adapter: hydrate migrated builtin MVs before 0dt cut-over by mtabebe · Pull Request #38399 · MaterializeInc/materialize

mtabebe · 2026-08-21T14:53:29Z

Problem:
A builtin schema migration using the Replacement mechanism hands the new deployment a fresh persist shard. Nothing writes that shard while the deployment is read-only, because ComputeController::allow_writes no-ops in read-only mode, so the MV's write frontier never advances and the 0dt readiness gate has to drop it and everything downstream of it from the caught-up check. The deployment then promotes with those collections unhydrated, and they all hydrate at once at cut-over, spiking catalog-server CPU and degrading catalog queries.

Solution:
The replacement shard is exclusively owned by this deployment, so the MV can write it while we are still read-only. Do that, and keep the MV in the readiness gate, so it hydrates before cut-over instead of at it.

Previously we didn't do this for builtins derived from mz_catalog because catalog shard frontier was not held. However, as of 26.17 it is, so gate the behaviour on the version of the old leader.

Testing:

New environmentd integration test test_0dt_migrated_builtin_mv_hydrates_before_promotion boots a read-only generation with a forced replacement migration and asserts the migrated builtin MVs mz_databases and mz_clusters are readable before it reports ReadyToPromote, proving they hydrate before cut-over rather than at it.
Extended the 0dt mzcompose workflow builtin-schema-migrations-replacement to read those MVs from the read-only generation before promotion, guarding the same invariant through a real cross-version upgrade.

aljoscha · 2026-08-21T16:08:29Z

Previously we didn't do this for builtins derived from mz_catalog because catalog shard frontier was not held. However, as of 26.17 it is, so gate the behaviour on the version of the old leader.

I didn't follow this one, about the "catalog shard frontier" being held.

Problem: A builtin schema migration using the `Replacement` mechanism hands the new deployment a fresh persist shard. Nothing writes that shard while the deployment is read-only, because `ComputeController::allow_writes` no-ops in read-only mode, so the MV's write frontier never advances and the 0dt readiness gate has to drop it and everything downstream of it from the caught-up check. The deployment then promotes with those collections unhydrated, and they all hydrate at once at cut-over, spiking catalog-server CPU and degrading catalog queries. Solution: The replacement shard is exclusively owned by this deployment, so the MV can write it while we are still read-only. Do that, and keep the MV in the readiness gate, so it hydrates before cut-over instead of at it. Previously we didn't do this for builtins derived from mz_catalog because catalog shard frontier was not held. However, as of 26.17 it is, so gate the behaviour on the version of the old leader. The behaviour is gated behind the `enable_0dt_hydrate_migrated_builtin_mvs` feature flag, on by default, a break-glass revert to the prior exclude-from-gate behaviour without a redeploy. Testing: - New environmentd integration test `test_0dt_migrated_builtin_mv_hydrates_before_promotion` boots a read-only generation with a forced `replacement` migration and asserts the migrated builtin MVs `mz_databases` and `mz_clusters` are readable before it reports `ReadyToPromote`, proving they hydrate before cut-over rather than at it. - Extended the `0dt` mzcompose workflow `builtin-schema-migrations-replacement` to read those MVs from the read-only generation before promotion, guarding the same invariant through a real cross-version upgrade.

ggevay · 2026-08-21T16:35:19Z

@aljoscha:

Previously we didn't do this for builtins derived from mz_catalog because catalog shard frontier was not held. However, as of 26.17 it is, so gate the behaviour on the version of the old leader.

I didn't follow this one, about the "catalog shard frontier" being held.

This refers to #35402. If the old version is before that, MVs might not be able to hydrate, because the catalog shard's upper was not guaranteed to be near the wall clock.

def- · 2026-08-21T18:14:30Z

QA LLM Review

1. MEDIUM -- Neither new test can distinguish "hydrated before `ReadyToPromote`" from "hydrated 60s after it"

src/environmentd/tests/server.rs:2631

Both new tests prove only that the replacement shard gets written at all; they cannot observe the ordering their names and comments assert, so a change that write-enables the migrated MVs but leaves them excluded from the caught-up gate — exactly the "promote with them unhydrated" behaviour this PR removes — leaves both tests green.

Details

The tests rely on a peek against an empty replacement shard blocking rather than erroring. But it blocks equally when the MV is merely late: in read-only mode the peek timestamp comes from the shared EpochMilliseconds oracle (≈ the leader's now), so the peek waits for the MV to reach roughly the caught-up condition — and it is issued only after ReadyToPromote has been observed, with a 60 s budget. In a test environment these MVs hydrate in a second or two. Change .filter(|_| !hydrate_migrated_mvs) at src/adapter/src/coord.rs:5195 to .filter(|_| hydrate_migrated_mvs) and both this test and test/0dt/mzcompose.py:1816 still pass.

The second claimed half — "write-enabling puts these MVs back into the gate, so anything they cannot catch up to now blocks promotion" — is not exercised either, for a separate reason. force_migrations = "replacement" replaces every builtin storage collection that has a registered shard, and mz_cluster_replica_frontiers is a BuiltinSource (src/catalog/src/builtin/mz_catalog.rs:3066), so it gets a fresh shard too, and register_introspection_collection's force_writable makes the read-only generation write its own replica frontiers into it. maybe_check_caught_up reads "live" frontiers from that same shard, so the lag comparison is the new generation against itself and within_lag is trivially satisfied. That is also why the tuned with_0dt_caught_up_check_allowed_lag = 5s / with_0dt_caught_up_check_stability_period = 10s cannot do what the comment at server.rs:2566 says: a collection "frozen at boot_ts + 1" would report a frozen live frontier as well, and would then be dropped by the beyond_all_hope cutoff rather than blocking.

Suggested fix: poll leader status and MV readability in the same loop, and assert the MVs became readable no later than the tick that first reported ReadyToPromote, rather than reading them afterwards with a generous timeout. For the lag half, either drop mz_cluster_replica_frontiers from forced replacement in these tests or state in the test that only the hydration half is covered.

2. LOW -- Exclusion set misses a migrated MV whose input is a builtin collection new in the same release

src/adapter/src/coord.rs:5181

The new comment justifies dropping migrated MVs out of the exclusion set on the grounds that migrated builtin tables keep advancing via read_only_mode_table_worker. A builtin table or source added in the same release gets no writer at all while read-only, so a Replacement-migrated MV reading one can never catch up, and with with_0dt_deployment_max_wait defaulting to a year that blocks cut-over until an operator forces skip-catchup.

Details

register_table_collections (src/storage-controller/src/lib.rs:1498) retains only migrated_storage_collections while read-only, and register_introspection_collection's force_writable is migration-only, so a brand-new builtin table or source sits at upper [0]. exclude_collections covers those collections themselves, but the transitive-dependent walk at coord.rs:5200 is seeded only from new builtin MVs, so dependents of a new builtin table or source are not excluded. Until now that gap was masked for migrated MVs, which were excluded together with their dependents unconditionally.

Not reachable in the tree today: no builtin table or source has been added since v26.38.0, so no currently-applicable Replacement step can read one. The cheapest guard is to seed the todo walk from all of new_builtin_collections rather than only the materialized views among them.

3. LOW -- Break-glass flag is captured at boot, so flipping it does not unstick a running read-only deployment

src/adapter/src/coord.rs:5119

hydrate_migrated_mvs is read once in serve, and exclude_collections is built from it at the same point, so ALTER SYSTEM SET enable_0dt_hydrate_migrated_builtin_mvs = false has no effect on a deployment that is already stuck. The flag's description presents it as an emergency control without saying that.

Details

Capturing the value once is correct — bootstrap and the gate have to agree on it. The gap is only in what the description promises. The operator sequence during an incident is: the read-only generation cannot run ALTER SYSTEM, so the setting has to be changed on the leader, and only a restart of the new generation picks it up. Worth adding to the description at src/adapter-types/src/dyncfgs.rs:83, so someone reaching for it mid-incident does not conclude the flag is broken.

ggevay

(Posting current thoughts, will continue on Monday)

ggevay · 2026-08-21T18:22:10Z

+                // read one (`mz_clusters` joins `mz_cluster_replica_size_internal`):
+                // `read_only_mode_table_worker` keeps advancing the uppers of migrated tables, so
+                // an MV over one still catches up.
                let new_builtin_mvs = new_builtin_collections


Seconding finding 2 of the QA LLM review, which we found independently: the exclusion
closure is seeded from new builtin MVs only, so dependents of new builtin
tables and sources are not walked. Those inputs get no writer while read-only
(register_table retains only migrated tables for the read-only table worker), so a
migrated MV reading a new builtin table can produce nothing past the input's empty
frontier: its own upper stays at the minimum and it fails this gate, and with
hydrate_migrated_mvs it now stays in the caught-up check, blocking promotion until
someone runs skip-catchup (with_0dt_deployment_max_wait defaults to a year) or pulls
the break-glass and restarts the pod. No current conversion MV reads a new builtin
table, so this is latent, but it is one release away. Seeding the walk from all of
new_builtin_collections would close it.

Good catch, walk starts from new_builtin_collections now.

ggevay · 2026-08-21T18:22:10Z

+                        %collection_hydrated,
+                        "collection {id} not in live frontiers"
+                    );
+                    if write_frontier.less_equal(&Timestamp::minimum()) || !collection_hydrated {


Two notes on the strengthened condition:

It is not gated on hydrate_migrated_mvs (or the leader version) and applies to
every collection missing from live_frontiers, so disabling the break-glass flag
does not fully restore the old behavior, while the flag's description says it
"reverts to excluding migrated MVs". As far as we can tell the ungated reach is
narrow (with the flag off, migrated MVs are excluded, and the only newly blocked
class is collections with a frontier past the minimum that cannot hydrate), so
softening the flag description may be enough; gating this condition on the same
bool is the alternative if you want the revert to be exact.

Optional: unlike the strong path below, there is no lag check against now here, so
an MV that hydrates once and then stalls keeps satisfying the gate. The pre-existing
condition had the same shape, so this may be fine, but it seems worth a conscious
decision.

Both adressed!

ggevay · 2026-08-21T18:22:10Z

+    "enable_0dt_hydrate_migrated_builtin_mvs",
+    true,
+    "Write-enable replacement-migrated builtin materialized views while read-only during a 0dt \
+     deployment, so they hydrate before cut-over and keep gating promotion. Emergency break-glass \


On top of finding 3 of the QA LLM review (boot-time capture, restart required, worth
stating in this description, seconded): there is no
LaunchDarkly flag (or KNOWN_MISSING_FROM_LD allowlist entry) for it yet, so in cloud
the break-glass could not actually be pulled today, and the
launchdarkly-flag-consistency check will likely flag it.

I updated the description of the flag.

I think we should defer LD?

ggevay · 2026-08-21T18:22:10Z

+#[mz_ore::test(tokio::test(flavor = "multi_thread"))]
+#[cfg_attr(miri, ignore)] // too slow
+#[allow(clippy::disallowed_methods)]
+async fn test_0dt_migrated_builtin_mv_hydrates_before_promotion() {


Nice test, and it does fail without the fix. This overlaps finding 1 of the QA LLM
review, which we can partly confirm empirically: rerunning the test with logging shows
all 219 "not in live frontiers" hits land in the first caught-up tick (while the
replaced mz_cluster_replica_frontiers shard is still empty), and from the second tick
every collection, including the migrated MVs, passes on the with-live-frontier path
against the deployment's own frontiers. We also ran two mutants: with the
no-live-frontier condition rewritten to wave everything through, the test still passes;
with the flag defaulted off (MVs not write-enabled), it reaches ReadyToPromote and
only fails at the delayed peek, which supports the review's point that the peek cannot
pin the ordering either. The production shape, where the leader hosts the id as a table
so the MV must pass through the no-live-frontier branch's frontier > minimum && hydrated, is never in the deciding position in either test.
Related landmine worth a NOTE: a real Replacement step for
mz_cluster_replica_frontiers would make the production gate self-referential the same
way.

Concrete suggestion: a flag-off variant of this test is cheap and valuable (we ran one:
with enable_0dt_hydrate_migrated_builtin_mvs defaulted off, ReadyToPromote is still
reached and the pre-promote peek times out, so it pins both the fallback path and the
fix). Covering the decisive no-live-frontier path likely needs
--force-builtin-schema-migration to accept a subset of objects, so
mz_cluster_replica_frontiers keeps its real shard while the MVs are replaced.

Ok I think I am going to defer the no live frontier change. Pushing the sub set down is a lot of churn for this bug fix IMO

ggevay · 2026-08-21T18:22:10Z

+    ) -> Result<(), CollectionUpdateError> {
+        // Every builtin eligible for the read-only bypass has a system id. A non-system id means
+        // the caller's `Replacement`-only invariant broke, so degrade to the read-only no-op (cold
+        // collection at cut-over) rather than risk writing a shard the leader still serves. Mirrors


Three small accuracy points on the docs here:

"degrade to the read-only no-op (cold collection at cut-over)": with
hydrate_migrated_mvs on, a collection that hits this tripwire stays in the
caught-up gate (exclusion only happens when the feature is off), and with its shard
unwritten its frontier never leaves the minimum, so the actual failure mode is
blocking promotion, not arriving cold at cut-over.

The storage-side check this claims to mirror is a hard assert!, not a soft panic
with a fallback.

The exclusivity guarantee is per (version, deploy generation), not per process: two
read-only processes at the same version and generation share the shard and both
write it (savepoint opens do not fence). That is the same shape as multi-replica MV
sinks, so it is fine, but the comment could say why it is tolerable.

Also worth a sentence somewhere: the replica-side AllowWrites handler enables persist
compaction process-wide on the clusterd, which this path triggers inside a read-only
deployment for the first time.

ggevay · 2026-08-21T18:22:10Z

 Depending on the existing migration shard entries, the process either decides to use the existing replacement shard, or to create the replacement shard and write its ID into the migration shard, at the current version.
 It sets the new shard ID as the migrated collection's shard in its in-memory catalog and commences bootstrapping using the replacement shard.

+Because this environment exclusively owns the replacement shard, the read-only process force-writes it during bootstrap rather than leaving it read-only until promotion.


This paragraph states the force-write unconditionally. Worth mentioning the two
conditions the code has: the old leader must be at v26.17 or newer (it has to keep the
catalog shard's frontier advancing, see MIN_LEADER_VERSION_FOR_MIGRATED_MV_WRITES),
and the enable_0dt_hydrate_migrated_builtin_mvs break-glass, with exclusion from the
caught-up gate as the fallback in both cases.

ggevay · 2026-08-21T18:22:10Z

-                // this deployment promotes). An excluded dependent may still be hydrating
-                // right after promotion, a brief blip we accept because these MVs are small
-                // and get a writer at cut-over.
+                // check as well. That's new builtin MVs, whose fresh shard has no writer until


Optional: the comment could say WHY new builtin MVs cannot get the same write-enable
treatment: their shard allocation lives only in the read-only savepoint, so the
promoted leader allocates a different shard and preflight writes would be discarded.
Replacement shards survive promotion via the durable migration-shard entry, which is
what makes force-writing them worthwhile.

ggevay · 2026-08-21T18:22:10Z

@@ -2773,7 +2787,22 @@ impl Coordinator {
                    // If this is a replacement MV, it must remain read-only until the replacement


Now that this branch also handles migration-Replacement MVs, the word "replacement"
means two unrelated things within three lines. Suggest disambiguating this comment,
for example: "If this MV is a pending REPLACEMENT FOR another MV, it must stay
read-only until ALTER ... APPLY REPLACEMENT swaps it in." The two sets are disjoint
(builtin MVs never come from user DDL), so this is wording only.

…caught-up path Two gaps in the 0dt caught-up gate, both found in review of MaterializeInc#38399. Seed the transitive-dependent walk from all of `new_builtin_collections`, not just the materialized views among them. A brand-new builtin table or source has no read-only writer either, since `register_table_collections` retains only migrated tables while read-only, so a `Replacement`-migrated MV reading one can never advance past that empty frontier. With migrated MVs now kept in the gate, such an MV would block promotion until an operator intervened. No builtin table or source has been added since v26.38.0, so this is latent today. Require the write frontier to be within the allowed lag of `now` on the no-live-frontier path, the same bound the live-frontier path applies. Hydration is a one-shot property, so a collection that hydrated and then stalled satisfied that branch forever. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Review of MaterializeInc#38399 found several claims that no longer match the code: - The `allow_writes_in_read_only` tripwire falls back to a no-op, which now blocks promotion rather than leaving a cold collection at cut-over, and the storage-side check it points at is a hard assert, not a soft panic with a fallback. Say both, and say why compute is deliberately softer. - Exclusive ownership of a replacement shard holds per (build version, deploy generation), not per process. Two read-only processes of one generation both write the shard, the same shape as a multi-replica MV, which the self-correcting persist sink is built to tolerate. - The replica-side handler enables persist compaction process-wide, which this path is the first thing to trigger inside a read-only deployment. - "Replacement" meant two unrelated things three lines apart in `bootstrap`: user `REPLACEMENT FOR` DDL and the builtin-migration mechanism. The sets are disjoint, so this is wording only. - Record why a *new* builtin MV cannot be write-enabled the same way: its shard allocation lives only in the read-only savepoint, so the promoted leader allocates a different shard. - Cite PR MaterializeInc#35402 for the v26.17 leader floor, and note in `caught_up.rs` that a future `Replacement` of `mz_cluster_replica_frontiers` would make the gate read its own frontiers. The design doc stated the force-write unconditionally; give the two conditions and the exclude-from-gate fallback. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Review of MaterializeInc#38399 found several claims that no longer match the code: - The `allow_writes_in_read_only` tripwire falls back to a no-op, which now blocks promotion rather than leaving a cold collection at cut-over, and the storage-side check it points at is a hard assert, not a soft panic with a fallback. Say both, and say why compute is deliberately softer. - Exclusive ownership of a replacement shard holds per (build version, deploy generation), not per process. Two read-only processes of one generation both write the shard, the same shape as a multi-replica MV, which the self-correcting persist sink is built to tolerate. - The replica-side handler enables persist compaction process-wide, which this path is the first thing to trigger inside a read-only deployment. - "Replacement" meant two unrelated things three lines apart in `bootstrap`: user `REPLACEMENT FOR` DDL and the builtin-migration mechanism. The sets are disjoint, so this is wording only. - Record why a *new* builtin MV cannot be write-enabled the same way: its shard allocation lives only in the read-only savepoint, so the promoted leader allocates a different shard. - Cite PR MaterializeInc#35402 for the v26.17 leader floor, and note that forcing the replacement mechanism across all builtins, as the 0dt tests do, leaves the caught-up gate reading its own frontiers. The design doc stated the force-write unconditionally; give the two conditions and the exclude-from-gate fallback. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…caught-up path Two gaps in the 0dt caught-up gate, both found in review of MaterializeInc#38399. Seed the transitive-dependent walk from all of `new_builtin_collections`, not just the materialized views among them. A brand-new builtin table or source has no read-only writer either, since `register_table_collections` retains only migrated tables while read-only, so a `Replacement`-migrated MV reading one can never advance past that empty frontier. With migrated MVs now kept in the gate, such an MV would block promotion until an operator intervened. No builtin table or source has been added since v26.38.0, so this is latent today. Require the write frontier to be within the allowed lag of `now` on the no-live-frontier path, the same bound the live-frontier path applies. Hydration is a one-shot property, so a collection that hydrated and then stalled satisfied that branch forever. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Review of MaterializeInc#38399 found several claims that no longer match the code: - The `allow_writes_in_read_only` tripwire falls back to a no-op, which now blocks promotion rather than leaving a cold collection at cut-over, and the storage-side check it points at is a hard assert, not a soft panic with a fallback. Say both, and say why compute is deliberately softer. - Exclusive ownership of a replacement shard holds per (build version, deploy generation), not per process. Two read-only processes of one generation both write the shard, the same shape as a multi-replica MV, which the self-correcting persist sink is built to tolerate. - The replica-side handler enables persist compaction process-wide, which this path is the first thing to trigger inside a read-only deployment. - "Replacement" meant two unrelated things three lines apart in `bootstrap`: user `REPLACEMENT FOR` DDL and the builtin-migration mechanism. The sets are disjoint, so this is wording only. - Record why a *new* builtin MV cannot be write-enabled the same way: its shard allocation lives only in the read-only savepoint, so the promoted leader allocates a different shard. - Cite PR MaterializeInc#35402 for the v26.17 leader floor, and note that forcing the replacement mechanism across all builtins, as the 0dt tests do, leaves the caught-up gate reading its own frontiers. The design doc stated the force-write unconditionally; give the two conditions and the exclude-from-gate fallback. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

- Seed the caught-up gate's transitive-dependent walk from all of `new_builtin_collections`, not just the MVs, and require the write frontier to be within the allowed lag of `now` on the no-live-frontier path so a hydrated-then-stalled collection no longer passes the gate forever. - Document the `enable_0dt_hydrate_migrated_builtin_mvs` break-glass flag: it is read once at startup, so flipping it means changing the setting on the leader and restarting, and it is not an exact revert. - Correct the read-only write-enable comments to match the code: the `allow_writes_in_read_only` tripwire blocks promotion, exclusive shard ownership holds per (build version, deploy generation), a new builtin MV cannot be write-enabled, and cite PR MaterializeInc#35402 for the v26.17 leader floor.

mtabebe force-pushed the fix/0dt-migrated-mv-caught-up-gate branch 3 times, most recently from 89c7a98 to 29624d1 Compare August 21, 2026 16:05

mtabebe force-pushed the fix/0dt-migrated-mv-caught-up-gate branch from 29624d1 to ed1200d Compare August 21, 2026 16:35

mtabebe marked this pull request as ready for review August 21, 2026 17:12

mtabebe requested review from a team as code owners August 21, 2026 17:12

ggevay reviewed Aug 21, 2026

View reviewed changes

		@@ -2773,7 +2787,22 @@ impl Coordinator {
		// If this is a replacement MV, it must remain read-only until the replacement

Uh oh!

Conversation

mtabebe commented Aug 21, 2026

Uh oh!

aljoscha commented Aug 21, 2026

Uh oh!

ggevay commented Aug 21, 2026

Uh oh!

def- commented Aug 21, 2026

QA LLM Review

1. MEDIUM -- Neither new test can distinguish "hydrated before ReadyToPromote" from "hydrated 60s after it"

2. LOW -- Exclusion set misses a migrated MV whose input is a builtin collection new in the same release

3. LOW -- Break-glass flag is captured at boot, so flipping it does not unstick a running read-only deployment

Uh oh!

ggevay left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

1. MEDIUM -- Neither new test can distinguish "hydrated before `ReadyToPromote`" from "hydrated 60s after it"