adapter: Compare 0dt hydration against the leader

[aljoscha]

Motivation

Closes: SQL-696

A rollout can wait on hydration the leader does not have either: a new collection the leader has not hydrated, or an undersized replica that never hydrated on the leader. The incoming generation should not need more hydration than the leader.

Description

The caught-up check reads the leader's mz_compute_hydration_times next to its frontiers and waives local hydration only on explicit evidence:

• Collections. A compute collection within the lag bound need not hydrate when every leader replica hosting it reports it unhydrated and none reports it hydrated. The stability gate ignores it too.
• Replicas. A replica the leader reports unhydrated, for a collection the classification did not ignore, only has to be Online. A replica the leader has fully hydrated still has to hydrate here, so we keep the redundancy the leader actually has.

Missing reports never waive anything. Storage collections are unchanged. The hydration source is protected from builtin migration, so the check keeps reading the leader's shard. There is no flag: skip-catchup and manual promotion remain the escape hatch.

Verification

Unit tests pin both waiver rules, including mixed, missing, and absent reports. Two 0dt workflows exercise them with a nonzero stability period. A replica-scoped hydration-concurrency override suspends both leader replicas for the collection rule, and only one leader replica for the replica rule while the other hydrates the same index. Both verify promotion readiness while the corresponding incoming index remains unhydrated.

Release notes

• Zero-downtime rollouts no longer wait for collections or replicas that the current deployment has not hydrated either.

[linear-code]

SQL-696

[ggevay]

I've triggered a relevant subset of Nightly: https://buildkite.com/materialize/nightly/builds/18373

[aljoscha]

Is this across all replicas of some cluster, so the unhydrated/hydrated is mixed, or hydration status of one specific replica. Need to be specific in the code docs and naming here, please

[ggevay]

LGTM, minor comments

[ggevay]

Nothing in the 0dt suite creates a collection or replica the leader hasn't hydrated, and CI runs with a 0 s stability period, so a no-op would pass. Could we add a workflow for each rule (e.g. a leader whose hydration of one object is stalled, and the crash-loop workflow with a replica the leader hasn't fully hydrated)?

[ggevay]

Could this say why one unhydrated collection is enough to exempt the whole replica, i.e. that we don't count a replica as useful until it's fully hydrated?

[def-]

QA LLM Review

1. MEDIUM -- A leader-waived collection that the incoming generation already hydrated exempts its replicas from the stability gate

src/adapter/src/coord/caught_up.rs:903

A compute collection that every leader replica reports unhydrated only goes into ignored_compute_collections while the incoming generation has not hydrated it either. Once the incoming side hydrates it on any replica, the leader's NULL rows for it reach replica_has_unhydrated_collection (caught_up.rs:642), and every replica hosting it becomes Exempt. On a single-replica cluster that switches the stability gate off at the first tick where the incoming replica is momentarily fully hydrated, which is the crash-loop case the gate exists to catch.

Details

Scenario: a single-replica cluster with indexes A and X. On the leader, A is hydrated and X is still hydrating, either because it was created during the rollout or because the leader replica restarted. Either way it is within the 2 h cutoff. The incoming r1 runs the new version and OOMs a few minutes after it hydrates.

• On ticks where incoming r1 has not hydrated X, X is waived and ignored. r1 is Asked about A, and the 10-minute period holds the cluster while r1 keeps restarting. This is the intended behavior.
• On the tick where incoming r1 has hydrated both A and X, collection_hydrated is true, so X is not inserted at :903-908. The leader's (X, r1) NULL row then makes r1 Exempt at :642. observe_stability([Exempt]) yields stable_for_ms = now, so the cluster is ready and promotion lands on a crash-looping replica while the leader was serving A.

So more progress on the incoming side gets the replica less scrutiny. That contradicts the description's rule that the stability gate ignores a leader-waived collection. In multi-replica clusters, every replica hosting such an X is exempted even though no leader replica hydrated X, so the redundancy argument behind the replica rule does not apply. Neither new workflow covers this, because both keep the incoming index unhydrated.

Suggested fix: ignore the collection whenever the leader waiver applies, regardless of local hydration. Then the replica rule only fires on collections that some leader replica did hydrate. This is a superset of today's condition, because readiness == Ready && !collection_hydrated already implies unhydrated_on_leader.

-                if (readiness != CollectionReadiness::Ready || !collection_hydrated)
+                if (readiness != CollectionReadiness::Ready || unhydrated_on_leader)
                     && matches!(collection_type, CollectionType::Compute)