compute: add a lifecycle event log for compute exports

[antiguru]

Motivation

The compute half of improved hydration visibility needs to say, per maintained object, when it started to hydrate, when it hydrated, and when its output became durable, so that a query can take min/max to determine cluster replica hydration.

Timestamp columns cannot carry that. Two things get in the way. The stages do not share a grain: installed, started and hydrated are per-worker facts, since each worker hydrates its own fragment of the dataflow, while whether the output is durable is a property of the sink as a whole, maintained on one elected worker. And a timestamp cannot say why the next stage has not happened, so a NULL cannot tell a replacement materialized view awaiting a cutover apart from an index that will never write, or from a materialized view whose refresh is in the future.

Design doc: doc/developer/design/20260817_compute_hydration_timestamps.md, reworked in this change. Part of CPU-226.

Description

One append-only log relation, mz_introspection.mz_compute_lifecycle_events_per_worker, per replica, in memory:

export_id    text        not null
worker_id    uint8       not null
event        text        not null
occurred_at  timestamptz not null
reason       text        nullable
details      jsonb       nullable

event	grain	reason
installed	per worker	none
started	per worker	none
hydrated	per worker	none
write_blocked	per object	read_only
write_unblocked	per object	none
written	per object	none

An index emits the first three and stops, which is the index degeneracy of the lifecycle falling out of the model rather than being special-cased. Subscribes and COPY TO stop early for the same reason, and a metric sink folds its output into the metrics registry rather than into a shard, so it has no write stages either.

mz_compute_hydration_times_per_worker is untouched. hydrated_at still reads the reported output frontier and time_ns still fires on the same crossing, so mz_compute_hydration_times, mz_compute_hydration_statuses, mz_hydration_statuses and the blue-green readiness query keep reporting exactly what they reported before. The change is purely additive. An earlier version of this PR redefined hydrated_at; that is dropped.

Grain. worker_id is the worker that observed the event and is never NULL. The per-object events are observed by the single worker that maintains the sink's shared write frontier, so they appear once per object and the row records which worker was elected. Nothing else may read that shared frontier as a measure of writing: mint clears it on every non-elected worker, where it is then the empty antichain and would report having written everything immediately. The election hashed(sink_id) % peers now has one definition, sink::materialized_view::frontier_owner, called both by mint and by the code recording ownership, rather than being spelled out separately in each place that needs it.

hydrated reads the compute probe, not the output frontier. The output frontier is the meet of write and compute frontier, which makes it durability, and per the point above it is not uniform across workers for a sink-backed collection. A collection with no compute probe produces its output by writing it, an index into its own trace, so there the write frontier is the progress.

The write stages are gated on hydration, and this is the one place the shipped behaviour differs from the design sketched in the review comment.

Before hydration the sink has produced nothing, so read-only mode is holding nothing back. Every collection starts read-only and is released by the controller, so reporting a block from installation would put a write_blocked and a write_unblocked on essentially every materialized view, both ahead of hydrated, making write_unblocked - hydrated negative rather than zero. Gating means the pair appears only when something really was held back, which is the state the log exists to expose.

Gating also keeps written ordered after hydrated, which it is not otherwise. apply_refresh rounds a REFRESH materialized view's frontier up to the next refresh time, and does so off its input frontier, before the dataflow has computed anything. So the sink sees a desired frontier ahead of the as-of immediately, mints a description for the pre-refresh window, and appends an empty batch, advancing the shard's upper while the dataflow is still hydrating.

That last point has a second consequence: a refresh schedule advances writing rather than blocking it, so there is no refresh cause for write_blocked to report and the refresh value is not in the vocabulary. test/testdrive/materialized-view-refresh-options.td:376 already demonstrates this from the outside, asserting that a materialized view whose first refresh is ~16 minutes out reports mz_hydration_statuses.hydrated = true, and that flag is time_ns IS NOT NULL, which requires the write frontier to have passed the as-of.

reason is typed, details is not, following mz_source_statuses and mz_sink_statuses. What details carries is the dataflow's as-of, which every stage is defined relative to: without it the interval between two stages says nothing about how much work was done, since a replacement materialized view with a far behind as-of is a completely different amount of work at the same duration. The as-of is rendered as a JSON string rather than a number, because mz_timestamp has no faithful JSON number counterpart and a string round-trips exactly.

Only read_only is attributed. It is the one cause of a write block compute can observe. Two attributions from the design sketch are not available and are called out in the doc as follow-up rather than silently dropped: distinguishing a started that waited on the hydration limiter from one that waited on its inputs needs SequentialHydration to report which, since both appear to the replica as Schedule arriving late; and distinguishing a fresh CreateDataflow from a dataflow retained across reconciliation is not observable here at all, because a retained dataflow emits no new installed event.

The relation is unkeyed, so index_by arranges by the whole row. (export_id, worker_id, event) is unique today, but declaring it a key is a uniqueness claim the optimizer acts on, and a false key is a correctness hazard where a wide index is only a performance one.

Rows are retracted when the export is dropped, and the emitted row pairs are kept on ExportState so they are retracted verbatim rather than re-derived.

Verification

cargo check is clean for mz-compute, mz-compute-client and mz-catalog, the last two with --all-targets. test_compute_event_size passes, which is the check that adding as_of to the Export event and a Lifecycle variant to ComputeEvent did not grow the enum.

test/testdrive/compute-lifecycle-events.td is new. It asserts the per-worker stage counts for an index and the per-object count for a materialized view's written, the ordering within each worker, written never preceding hydrated, occurred_at landing in the recent past rather than being an offset from an arbitrary origin, details->>'as_of' being present, retraction on drop, and three invariants under set-max-tries max-tries=1: the closed event and reason vocabularies, no stage without its predecessors, and no stage reported twice per export and worker.

Two honest limitations.

The testdrive file has not been executed. Running it needs an environmentd image, which this environment cannot build, so CI is the first thing to run it. The assertions are conservative but "expected to pass" is the accurate description.

write_blocked and write_unblocked are not covered, because reaching them needs a replica that is still read-only after a collection has hydrated, which is a 0dt cutover and not reachable from plain testdrive. The invariant assertions are written so that the pair may be absent or complete, never requiring it to be present. A platform-check or a cluster test that drives read-only mode would be the place to cover them.

The goldens were hand-edited rather than produced by --rewrite-results, since that needs a full build: oid.slt, information_schema_tables.slt, cluster.slt, mz_catalog_server_index_accounting.slt, autogenerated/mz_introspection.slt, both test/workload-replay lists, and the mz_introspection docs page, where the relation is marked RELATION_SPEC_UNDOCUMENTED alongside mz_compute_hydration_times_per_worker and the other per-worker logs. Worth a reviewer's eye, and CI will catch any that are wrong.

---

Generated by Claude Code

[linear-code]

CPU-226

[antiguru]

Design: a lifecycle event log instead of wide timestamp columns

Writing up a design we converged on offline, which supersedes the approach in this PR. The code here is sound, but the shape it extends is not the one we want, and the reasons are worth recording before the stacked written_at PR builds on it.

The problem with wide columns

Two structural issues surface as soon as the write stages are added.

The first is that the stages do not share a grain. installed, started and hydrated are genuinely per-worker facts, because each worker's dataflow fragment hydrates independently. Durability is not: the sink maintains its shared frontier on one elected worker, hashed(sink_id) % peers, and clears it on every other, so a durable timestamp is a property of the sink as a whole. A single wide row forces either NULLs on non-elected workers or a second relation, and this PR is currently paying for that with a column whose meaning depends on which worker logged it.

The second is that a timestamp column cannot say why the next stage has not happened. hydrated with no durable timestamp is healthy for a read-only replacement awaiting cutover, healthy for a materialized view whose refresh is in the future, and a bug otherwise. A NULL cannot distinguish those from each other, nor from an index that will never write, nor from a continual task with no compute sink.

Shape

One append-only relation, per replica, in memory.

column	type	notes
export_id	text not null
worker_id	uint8	see the open question below
event	text not null	installed, started, hydrated, write_blocked, write_unblocked, written
occurred_at	timestamptz not null
reason	text	closed vocabulary, filterable
details	jsonb	open ended, documented by example

event	grain	reason vocabulary
installed	per worker	commanded, reconciled
started	per worker	inputs_available, hydration_limit
hydrated	per worker	none
write_blocked	per object	read_only, refresh
write_unblocked	per object	none
written	per object	none

Indexes emit the first three and stop, which is the index degeneracy of the lifecycle falling out of the model rather than being special cased. Subscribes and COPY TO likewise stop early, and a continual task has no compute sink at all.

Why write_blocked is logged on entry

An earlier sketch carried the blocking reason on write_unblocked, which reads naturally but means the reason is only observable once the block ends. If it never ends, which is precisely the state an operator is debugging, there is no row at all. Logging entry into the state makes "which objects are hydrated but not writing, and why" a query over present rows rather than an inference from absence.

Why write_unblocked and not write_started

mint produces a batch description as soon as desired_frontier advances past persist_frontier, and persist_frontier starts at the as-of (src/compute/src/sink/materialized_view.rs:682-685). So for a plain read-write materialized view the first write is minted at hydration, and a separate "write started" timestamp would carry no information.

It only diverges when something gates the desired frontier: read-only mode, where report_frontiers clears the write frontier, or a refresh schedule, where apply_refresh rounds the frontier up. In both cases the informative moment is when writing became permitted. Naming it that way gives each interval exactly one cause:

• started - installed is queueing.
• hydrated - started is compute.
• write_unblocked - hydrated is blocked time, and zero in the common case.
• written - write_unblocked is write work: snapshot size, persist throughput, retries.

Stamp write_unblocked on the first append attempt rather than the successful one, so a retry loop on upper mismatch shows up in the interval instead of being hidden by it.

Why details is jsonb

This follows mz_source_statuses and mz_sink_statuses (src/catalog/src/builtin/mz_internal.rs:2153, :2435), which pair a typed status with a nullable details jsonb whose comment documents it by example rather than promising a schema. The split matters: what people filter and group on stays typed, and only the look-at-one-row detail goes in the json.

Both reason vocabularies pass that test. Which objects are blocked, and on what is WHERE event = 'write_blocked' AND reason = 'read_only'. Did this start late because of the hydration limiter or because inputs were not ready is a GROUP BY reason. Neither should be a json extraction.

The motivating payload for details is as_of. Every event in the log is defined relative to it, since hydrated means the progress frontier passed the as-of. Without it, hydrated - started cannot distinguish a genuinely fast hydration from one whose as-of was already recent, and a replacement materialized view with a far behind as-of is a completely different amount of work at the same duration. It also interacts with reason on installed: a retained dataflow keeps its as-of and a replacement gets a new one, so recording both makes "same object, new dataflow" self evident. None of that earns a column, and all of it is worth having in the row.

Consequence worth stating now: invariant tests must assert on event, reason and occurred_at and never on details, or the first test that pins a field removes the extensibility it is there for.

Bounds

Roughly six rows per object, times workers for the first three events, and a REFRESH EVERY materialized view records only its first write rather than a pair per interval. Everything is retracted when the object is dropped. This is in-memory introspection, so there is no durable growth to reason about.

Recommendation on the existing relation

Leave HydrationTime exactly as it is on main. This PR currently changes what hydrated_at means for elected workers, which is a semantic change to a documented relation and the source of two findings in the QA review above. If the log carries the dataflow stage instead, nothing existing changes meaning, and the change becomes purely additive. A follow-up can migrate the two readers of hydrated_at, both of which are tests, and drop the redundant columns.

Two related corrections to this PR's design doc, independent of the log:

• The read-only argument is false. report_frontiers already excludes the write frontier in read-only mode (src/compute/src/compute_state.rs:965-970), and the comment above it says why, so the old hydrated_at never waited for a cutover. That argument is correct, but it belongs to write_blocked and write_unblocked, which this PR does not add.
• The deleted REFRESH paragraph was over broad rather than wrong. A plain REFRESH EVERY does refresh at creation, because ALIGNED TO defaults to mz_now(), but REFRESH AT and ALIGNED TO '<future>' do not, and the rollup hazard the paragraph warned about still applies to mz_compute_hydration_statuses.hydration_time.

A deterministic test lever

The PR notes that a small materialized view writes its snapshot in milliseconds, so no assertion can separate the stages, and rules out REFRESH on the grounds of the implicit refresh at creation. That holds for REFRESH EVERY but not for REFRESH AT '<future>', which has no implicit refresh. The compute probe is attached at src/compute/src/sink/materialized_view.rs:184, before apply_refresh at :188, so:

CREATE MATERIALIZED VIEW mv IN CLUSTER test
  WITH (REFRESH AT '<now + 1 hour>') AS SELECT ...;
-- a `hydrated` event on every worker, no `written` event, stable for an hour

That assertion fails on the current code and holds for an hour rather than milliseconds. test/testdrive/materialized-view-refresh-options.td already builds future dated refresh materialized views, so the pattern is in tree.

Implementation cost

Scoped, not started:

• A ComputeLog variant and RelationDesc in src/compute-client/src/logging.rs, unkeyed rather than keyed on (export_id, worker_id).
• A log id appended in src/catalog/src/durable/transaction.rs. Existing ids must not be renumbered; doing so panics on restart with a negative capability on IntrospectionSourceIndex.
• A BuiltinLog with a fresh OID and ontology in src/catalog/src/builtin/mz_introspection.rs.
• A demux arrangement in src/compute/src/logging/compute.rs, including packing the jsonb column, which the existing packers do not do.
• Emit sites in src/compute/src/compute_state.rs, mostly present in this PR already.
• Derived views, doc, tests.

Around 400 to 600 lines across seven files. Adding a builtin introspection index wants a restart test rather than sqllogictest alone.

Open question

Whether worker_id is nullable. Nullable makes the relation self describing, since a NULL says an event is sink wide. Non-nullable matches every existing ComputeLog variant, avoids a NULL in a framework that has none, and records which worker was elected for free, at the cost of the per-object grain being documented rather than visible in the row. I lean non-nullable with the grain in the column comment, but it is worth a second opinion.

Filed by Claude Code.

[def-]

QA LLM Review

1. HIGH -- Adding a builtin log also moves mz_catalog.mz_sources, which has no migration step

src/adapter/src/catalog/open/builtin_schema_migration.rs:424

Adding MZ_COMPUTE_LIFECYCLE_EVENTS_PER_WORKER changes the SQL fingerprint of two generated builtin materialized views, not one: make_mz_indexes inlines the builtin-log set, and so does make_mz_sources. Only mz_indexes gets a replacement step here, so upgrading from a released version leaves mz_sources with a changed fingerprint and no migration, and update_fingerprints panics during catalog open, blocking environmentd startup.

Details

make_mz_sources (src/catalog/src/builtin/builtin.rs:361-368) builds one VALUES row per builtin log and chains it into builtin_values, which is interpolated into the MV's sql. Fingerprint for &BuiltinMaterializedView is create_sql() (src/catalog/src/builtin.rs:756-760), which contains that sql, and the function's own doc comment states the intent: "the MV's SQL fingerprint changes whenever a builtin source or log is added or removed, which forces a MigrationStep::replacement for mz_sources".

At catalog open, Migration::run → plan_migration selects only steps with version > source_version. Upgrading from a released 26.39.x, source_version.pre is empty, so the dev-version auto-force at builtin_schema_migration.rs:669-677 does not kick in and no forced plan covers mz_sources. update_fingerprints (builtin_schema_migration.rs:1185-1211) then finds a mismatch for a builtin that is neither migrated, nor ephemeral (Log/View/Index), nor runtime-alterable, and takes the panic!("fingerprint mismatch for builtin ...") arm. The newest mz_sources step in the list is at 26.27.0-dev.0, well below any currently released version.

This is the first builtin log added since mz_sources became a generated MV, which is why no prior step exists to copy from.

Fix — add the companion step next to the mz_indexes one:

         MigrationStep::replacement(
             "26.40.0-dev.0",
             CatalogItemType::MaterializedView,
             MZ_CATALOG_SCHEMA,
             "mz_indexes",
         ),
+        // Same reason for `mz_sources`: `make_mz_sources` inlines one VALUES row per builtin
+        // log alongside the builtin sources, so adding a log moves its fingerprint too.
+        MigrationStep::replacement(
+            "26.40.0-dev.0",
+            CatalogItemType::MaterializedView,
+            MZ_CATALOG_SCHEMA,
+            "mz_sources",
+        ),
     ]

Worth noting for the future: test_mz_indexes_fingerprint_changes_with_new_builtin_index asserts that an extra log moves the mz_indexes fingerprint, but the sibling mz_sources test only exercises an extra source, so nothing on the PR path flags the second dependency. Extending that test with an extra-log case would make the next log addition self-announcing.

[antiguru]

Confirmed and fixed in aff3e16938. I verified each step of the analysis against the code rather than taking it on faith, and it holds: make_mz_sources(source_iter, log_iter) chains log_values into builtin_values (builtin/builtin.rs:361-368), update_fingerprints reaches panic!("fingerprint mismatch for builtin ...") for a builtin that is neither migrated nor ephemeral nor runtime-alterable, and the newest mz_sources step was at 26.27.0-dev.0. Added the companion Replacement step at 26.40.0-dev.0.

I had found the mz_indexes half and stopped there, which was the mistake. Worth recording why it was easy to stop: make_mz_sources's own doc comment already states the rule in full, saying the fingerprint changes whenever a builtin source or log is added and that this "forces a MigrationStep::replacement for mz_sources". The requirement was written down and I read past it.

Also took the test suggestion, and it is sharper than it first looks. test_mz_sources_fingerprint_changes_with_new_builtin_source's doc comment already claimed to verify "a new builtin source or log", but the body only ever appended an extra source. The mz_indexes sibling does cover the extra-log case. So the property was asserted in prose and left unimplemented, which is precisely why nothing on the PR path announced the second dependency. The test now appends an extra log as well, so the next builtin log added trips cargo test instead of a 0dt smoke test.

For the record on the rest of this PR's CI: the five slt shards and two testdrive shards are failing for reasons this does not explain, since both boot a fresh catalog where stored and computed fingerprints agree by construction. I am running bin/sqllogictest locally over the six goldens I touched to get the actual assertion text, because bk, gh and CI_DASHBOARD_TOKEN are all unavailable in this environment and I would rather not keep guessing at them from the diff.

---

Generated by Claude Code

[def-]

QA LLM Review

1. MEDIUM -- The back-filled started lifecycle event is stamped at hydration time, not at install time

src/compute/src/logging/compute.rs:1259

When a dataflow hydrates before its Schedule arrives, handle_hydration back-fills started_at from installed_at for mz_compute_hydration_times_per_worker but logs the lifecycle started event with occurred_at = self.time, which is the hydration instant. For those exports the two relations then contradict each other about the same export and worker: the timestamps relation reports queueing 0 and hydration δ, the event log reports queueing δ and hydration ~0, which is exactly the inversion the surrounding comment says the back-fill exists to avoid.

Details

log_lifecycle (src/compute/src/logging/compute.rs:1266-1269) always stamps occurred_at = self.time, and the back-fill call site has no way to override it. Meanwhile handle_hydration deliberately sets export.hydration_timestamps.started_at = Some(installed_at) two lines above, with the comment "Stamping hydrated_at instead would invert it, charging the whole life to queueing and reporting zero hydration time for a dataflow that only ever hydrated." The lifecycle back-fill added at line 1259 does precisely that, under a comment claiming it is "the same back-fill".

Concrete sequence, for the case the code's own comment names as happening on every bootstrap ("an index over an already-hydrated arrangement reports hydration while still suspended, which happens for a handful of mz_catalog_server indexes"):

• T0: CreateDataflow → Export → installed_at = T0, lifecycle installed at T0. No Schedule yet, since maybe_schedule_collection (src/compute-client/src/controller/instance.rs:1670-1707) withholds it until every dependency is itself scheduled and past the as-of.
• T0+δ: a report_frontiers tick sees the output frontier past the as-of → set_hydrated() → handle_hydration. started_at is back-filled to T0 in the timestamps row; the lifecycle started row is written with occurred_at = T0+δ.
• Same tick: observe_hydration writes lifecycle hydrated at T0+δ.

δ is at least one introspection interval and is bounded by how long the dependency chain takes to be scheduled, so it is not a rounding artifact. Ordering assertions still pass (started is emitted before hydrated in the event stream), so test/testdrive/compute-lifecycle-events.td does not catch it; only the interval values are wrong.

Fix — thread the instant through instead of always reading self.time. new_timestamps is already a Copy of the timestamps at line 1243, so installed_at is in scope at the call site:

     fn log_lifecycle(&mut self, export_id: GlobalId, stage: LifecycleStage) {
-        let ts = self.ts();
-        // Stamp the event time rather than `ts`, as in `handle_export`.
-        let occurred_at = self.time;
+        self.log_lifecycle_at(export_id, stage, self.time);
+    }
+
+    /// As `log_lifecycle`, but for a stage whose instant is not the current event time.
+    fn log_lifecycle_at(&mut self, export_id: GlobalId, stage: LifecycleStage, occurred_at: Duration) {
+        let ts = self.ts();

and at the back-fill:

         if backfilled_start {
-            self.log_lifecycle(export_id, LifecycleStage::Started);
+            // Match the timestamps relation: a dataflow that hydrated before its `Schedule`
+            // was never queued, so `started` belongs at `installed_at`.
+            self.log_lifecycle_at(
+                export_id,
+                LifecycleStage::Started,
+                new_timestamps.installed_at,
+            );
         }

[antiguru]

Confirmed and fixed in 581179de84. log_lifecycle unconditionally stamped self.time, so the back-filled started landed at the hydration instant while the timestamps row put started_at at installed_at. Two relations disagreeing about one export and worker, and in the direction the adjacent comment exists to rule out.

Split it into log_lifecycle and log_lifecycle_at, and the back-fill now passes new_timestamps.installed_at.

The part worth dwelling on: I wrote the comment "The lifecycle log needs the same back-fill" directly above a call that did not perform the same back-fill. The installed_at reasoning was two lines up in the function I was editing, and the comment I authored asserted the property while the code contradicted it. Ordering assertions could never catch this, since the emission order is right and only the interval values are wrong, so thank you for reading the values rather than the sequence.

Two notes on the surrounding state, so the next review has the current picture:

The mz_indexes/mz_sources fingerprint pair from your previous finding is fixed and clippy is green on it. Still outstanding and unexplained: all five slt shards and two testdrive shards fail, which neither fingerprint step accounts for, since both boot a fresh catalog where stored and computed fingerprints agree by construction.

I hand-edited eight goldens for this relation and have since found two errors in my own edits, so I am not going to keep hand-editing them. A local bin/sqllogictest build is nearly done and I will use --rewrite-results over the touched files to regenerate them from a real catalog, then review the diff for changes a new builtin log should not cause. If anything in this PR's goldens still looks hand-derived after that, it is worth flagging.

---

Generated by Claude Code