cluster-controller: graceful reconfiguration strategy + ALTER reshape + wait-shim

[aljoscha]

**NOTE: This is PR3 in a stack of PRs, the full branch is at #36738

Move graceful (zero-downtime) cluster reconfiguration into the cluster
controller as a pure strategy, driven by the durable reconfiguration
record, with hydration-aware cut-over, a durable honored timeout, and a
clean audited abort when resources run out. Everything lands dark behind
the enable_cluster_controller master gate. The legacy 3-stage machine
still runs when the gate is off.

Strategy (mz-cluster-controller). New pure
GracefulReconfigurationStrategy, engaged whenever the reconfiguration
record is present. desired_replicas contributes
target.replication_factor replicas at the target shape (size, logging,
AZ list) on top of the baseline's realized set, the hydrate-overlap.
update_state cuts the realized config over to the target and clears the
record once those replicas are all present and hydrated. Success takes
precedence over the deadline. Past the deadline with the target not fully
hydrated it applies the record's on_timeout. COMMIT cuts over to the
still-unhydrated target and clears the record. ROLLBACK clears the
record durably while leaving the realized config untouched and stops
contributing the target replicas, so the cluster reverts to the
pre-reconfiguration set and the strategy disengages. With the record gone
this is a stable state, not a retry loop, and the clear is the durable
transition the audit timed-out event classifies.

Hydration seam. New ClusterControllerCtx::hydrated_replicas(cluster, replicas) -> BTreeSet ("which of these replicas have all current
collections hydrated"), the shape its only consumer needs and that the
underlying controller APIs can express. The controller pulls it on demand,
only while a reconfiguration is in flight, into the live-signal field
ClusterState::hydrated_replicas (excluded from the compare-and-append
witness). The adapter driver backs it per-replica against the compute and
storage controllers, which collapse a replica list to a single "hydrated
on any" bool.

ALTER reshape (gated). With the master gate on, a managed-cluster ALTER
that changes a replica's config shape (SIZE, logging, AVAILABILITY ZONES),
or any ALTER while a record is already in flight, writes or folds the
reconfiguration record onto the realized config and leaves the realized
shape in place. The controller converges and cuts over. A fold overlays
the ALTER onto the in-flight target per dimension: a dimension the
ALTER sets re-targets, one left Unchanged keeps the in-flight target's
value (seeding Unchanged dimensions from the realized config would
silently revert the in-flight transition, since the realized config only
advances at cut-over), while the deadline and on_timeout are replaced
wholesale by the latest ALTER's. Non-shape changes with no record in
flight keep updating the realized config directly. The deadline is now + timeout and on_timeout is threaded from the existing WITH (WAIT ...)
clause: WAIT UNTIL READY (TIMEOUT, ON TIMEOUT ...) verbatim, WAIT FOR
desugars to ON TIMEOUT COMMIT, and omitting WAIT falls back to the
default_cluster_reconfiguration_timeout dyncfg and the default action.
An ALTER that omits ON TIMEOUT carries None through the plan, and
each execution path resolves its own default: the controller-owned paths
use ROLLBACK, the safe choice that never silently induces downtime by
cutting over to an un-hydrated target, while the legacy foreground path
keeps its historical implicit COMMIT, matching the documented default.

Wait-shim. New ClusterStage::AwaitReconfiguration polls the durable
record until the controller resolves it, preserving today's foreground UX
over the same durable mechanism. The controller owns the deadline and the
shim deliberately has no timeout of its own, since erroring while a record
is still present can race the controller and misreport an ON TIMEOUT COMMIT cut-over as a timeout. Once the record clears, the shim reports
success only if the realized config reached the target. A ROLLBACK
timeout, a resource-exhaustion abort, or a concurrent ALTER that
re-targeted the record and settled elsewhere all leave it short and
surface AlterClusterTimeout. With the new
enable_background_alter_cluster dyncfg on, ALTER returns immediately
instead. Session disconnect does not abort a reconfiguration, it only
stops waiting.

Resource budgets and exhaustion. ALTER validates the reshape's
transient peak up front: the controller runs the realized and target sets
side by side until cut-over, so the cluster's peak contribution is both
shapes at once, realized_rf plus target_rf against
max_replicas_per_cluster and credit(realized) plus credit(target)
against max_credit_consumption_rate, all computed from config. The peak
model is simple and conservative (a same-shape overlap is over-counted)
and matches the legacy wait path, which creates the full target set as
pending replicas at ALTER time and so enforces both limits on the
overlap. The credit base is the live consumption of every other cluster,
excluding this cluster's own replicas so a re-target of an in-flight
record never double counts an already-materialized overlap. An ALTER
back to the realized shape materializes nothing and skips the checks,
keeping the cancel escape hatch usable at the limits.

A record can still become unsatisfiable after it is written, when a limit
shrinks or the environment grows before the controller creates the
overlap. The apply then fails resource validation, and rather than parking
the record and retrying every tick, the controller aborts the
reconfiguration: an immediate clearly-audited abort beats a silent park
that may still time out. The kernel drives the reaction and
the coordinator only reports and executes. ApplyOutcome gains
ResourceExhausted, which the ctx impl maps from the adapter error,
declaring the fact and nothing more. On that outcome the kernel sheds the
cluster's most expendable transient strategy and recomputes next tick,
emitting the new Decision::AbortReconfiguration under the same expected
witness. The failed create changed no durable state, so the witness is
still current, and a concurrent user re-target wins the guard and is left
to converge instead of being clobbered. The peel policy lives with the
strategies: a reconfiguration is a discretionary user change that fails
cleanly and can be retried, so it is shed first, while the baseline is
never shed. On the transaction side the abort is a normal record-clear
Op::UpdateClusterConfig, not a dedicated op. The op gains a
writer-declared ClusterConfigUpdateReason, consumed by the audit
classifier for exactly the case the config diff cannot distinguish: an
abort's clear is byte-identical to a ROLLBACK-at-deadline timeout's
clear and would otherwise mislabel as timed-out.

Replica-id preallocation. The legacy managed-to-managed path preallocated
replica ids before checking whether the controller owns the replica set,
so a controller-owned alter (an rf-only change under the master gate)
durably allocated and discarded ids on every such alter and paid an
unnecessary catalog-write-timestamp round-trip. Compute controller
ownership first and allocate no ids when the controller owns the set. It
allocates its own ids when it materializes the change.

Audit. Two levels are recorded. Per replica: new
ReplicaCreateDropReason::GracefulReconfiguration maps to
CreateOrDropClusterReplicaReasonV1::Reconfiguration, carried on the
controller's graceful-desired replica creates. Lifecycle: new
EventDetails::AlterClusterReconfigurationV1 records a started, finalized,
timed-out, resource-exhausted, or cancelled transition with the target
shape and, where it applies, the active deadline. It is emitted from the
single Op::UpdateClusterConfig durable write site, classified purely from
the before/after reconfiguration record, whether the same write advanced
the realized config to the target, and the op's declared reason. Writing or
re-targeting a record is started. An ALTER-back whose new target equals
the realized shape is cancelled. A clear that advanced the realized
config to the target is finalized, covering both a hydrated success and a
COMMIT-at-deadline cut-over. A clear that did not advance it is
timed-out, the ROLLBACK-at-deadline revert, unless the op declares a
resource-exhaustion abort, which classifies as resource-exhausted. Keying
on whether the config advanced, rather than on on_timeout, is what lets
the rollback revert carry the sole timed-out papertrail with no
controller-seam signal. The audit details are defined by the catalog
migrations this stack builds on, v87->v88 for the lifecycle event and
replica reason and v88->v89 for the resource-exhausted transition.

Tests: graceful kernel and flow cases in mz-cluster-controller (in-flight
desire, cut-over, partial hydration, timeout-vs-hydrated precedence,
clear-and-drop on a rollback timeout, COMMIT- vs ROLLBACK-at-timeout,
AZ-only shape change, full overlap then cut-over, ALTER-back, the
fold_reconfiguration_target overlay, and resource-exhaustion sheds),
FakeCtx seam tests that drive reconcile end-to-end past a forced deadline,
classify_reconfiguration_transition unit tests in mz-adapter covering
the declared abort, and an extended cluster-controller.td asserting a
background ALTER cuts the realized size over, that the
omitted/COMMIT/ROLLBACK spellings each drive a record under the gate,
and that a reshape whose transient peak overflows
max_replicas_per_cluster or max_credit_consumption_rate is rejected
synchronously with no record written (the credit limit computed via
set-from-sql against the live replica set, since that budget is
environment-wide). Source-ingest and CDC testdrive files gain
poll-for-convergence guards and a fast tick interval, because the
controller now reconciles replication-factor changes asynchronously.

Implements graceful reconfiguration as a controller strategy, the ALTER
reshape, the durable WITH (WAIT ...) timeout, the resource-exhaustion
abort, and the reconfiguration audit lifecycle from
doc/developer/design/20260522_cluster_autoscaling.md.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com