console: add data for the redesigned cluster page's resource usage

[leedqin]

Motivation

Second PR in the cluster details redesign stack, on top of #39446. It adds the data behind the resource usage card: memory split into RAM and swap, replica status over time, hydration, heap limits and object creation times. The card itself is the next PR (#39508), so this one can be reviewed without the chart code. Nothing renders these queries yet.

Part of CNS-174.

Description

New queries. Each filters on literal replica or object ids, so it plans as an index lookup on mz_catalog_server, except where noted.

Data	Source	Refresh
RAM and swap as shares of the heap limit	swap_of_ram_percent and heap_limit_percent on the console utilization views, 26.44 and later	with the utilization data
Replica status transitions	mz_cluster_replica_status_history	30s
Finished hydrations	mz_replica_hydration_history, 26.43 and later, where it gained process_id. No index, so it only runs for replicas that came back online inside the window.	60s
How many objects each current replica has yet to hydrate	mz_compute_hydration_times	30s
Heap limit per replica	mz_cluster_replica_metrics	once, refetched while a replica's latest limit is null, since a failed metrics fetch records a null
Creation times of the cluster's indexes and materialized views	mz_object_lifetimes, filtered to those object types since other kinds share the u<N> id form	once per set of ids
A replica's per-process samples, for the Processes drawer in #39508 and the workers drawer in #39889	mz_cluster_replica_metrics_history, a lookup on its replica_id index	live SUBSCRIBE while the drawer is open
Each size's process count and per-process CPU, memory and disk allocation, for the same drawers	mz_cluster_replica_sizes, fast path on its index	once

Per-process reads. The resource usage card charts replicas, not processes. This PR adds the per-process subscribe builder and the sizes query that the Processes drawer in #39508 and the workers drawer in #39889 read. The subscribe starts at a literal time one window back, and useRollingWindowStartMs moves that start forward once per window, so an open subscribe holds at most two windows of samples. A mz_now() temporal filter would instead hold each sample's pending retraction in the subscribe on mz_catalog_server: on a local emulator with 2 replicas of 16 processes over 24h, that measured ~12 MiB per open subscribe, against ~2 MiB with the literal bound. EXPLAIN on mz_catalog_server: a lookup on mz_cluster_replica_metrics_history_ind, with the bound as a plain filter. Per-process catalog views are tracked in CNS-203. useUtilizationTier holds the tier choice (3h subscribe, 24h subscribe, or poll) that the replica-level paths share.

useReplicaUtilizationHistory gets two options:

• includeMemoryBreakdown adds the RAM and swap columns. Callers that don't pass it keep their query keys, so the classic page sends the same queries as before.
• includeOfflineEvents: false skips the 20s offline-events poll on windows of 3h or less. The new page reads offline stretches from status history instead.

useClusterObjectsLive (in allObjectsCollection.ts) returns a cluster's objects of the given types as a TanStack DB live query over the existing allObjectsCollection. Filtering useAllObjects gives a new array on every DDL anywhere in the environment; the live query keeps its result until one of this cluster's matching objects changes. It reuses the app-wide objects subscribe, so it adds no catalog_server load.

resourceUsageModel.ts holds the chart logic as pure functions: status timelines, replica colors, memory bar segments, and merging buckets so bars stay readable on wide windows.

The k6 cluster detail replay lists these queries' refresh intervals, since it replays unlisted queries every 5s. The testing guide's scalability commands now use yarn test:e2e:scalability, because the chromium project skips scalability/.

types/materialize.d.ts is edited by hand, with TODOs on the new columns, because yarn gen:types fails against current builds (pg_get_serial_sequence doesn't exist).

A hydration in progress has no recorded start, so the timeline model doesn't draw one. The next PR shows it as a count of objects still hydrating at the now edge instead.

Verification

• replicaUtilizationBinning.test.ts: RAM and swap shares survive binning and come from the bucket's highest-heap sample.
• allObjectsCollection.test.ts: useClusterObjectsLive returns the cluster's objects of the given types, keeps its result through changes on other clusters, and updates when one of its objects changes.
• resourceUsageModel.test.ts: status timelines (state before the window, readings before the first transition once older history is gone, clipping, out-of-memory markers, hydration stretches), replica colors, memory bar segments, the RAM limit, and bucket merging.
• replicaProcessMetrics.test.ts: the per-process subscribe bounds its samples with a literal start and no temporal filter.
• queries.test.ts: useRollingWindowStartMs starts one window back, moves once per window, and never returns a start for another window.
• The plans in the table were checked with EXPLAIN on mz_catalog_server in a local v26.45 build, and the two per-process queries on a cloud environment.

🤖 Generated with Claude Code

[def-]

QA LLM Review

1. MEDIUM -- Hydration history query is gated at 26.41 but needs a column added in 26.43

console/src/platform/clusters/queries.ts:798

REPLICA_HYDRATION_HISTORY_MIN_VERSION is "26.41.0", but buildReplicaHydrationEpisodesQuery filters on mz_replica_hydration_history.process_id, which first shipped in v26.43.0. On environments running v26.41.0 through v26.42.0 the gate lets the query through, and every poll fails with column "process_id" does not exist, so the redesigned card never shows finished hydration stretches there and reports "Hydration history is unavailable."

Details

At v26.41.0, v26.41.1, v26.41.2 and v26.42.0, the table in src/catalog/src/builtin/mz_internal.rs ends at the status column. process_id was added by #38864 (9297cd2), which git tag --contains places first in v26.43.0. The filter at console/src/api/materialize/cluster/replicaStatusTimeline.ts:124 (process_id IS NULL OR process_id = '0') therefore fails to plan on those versions. The IS NULL arm was clearly meant for rows written before the column existed, but that doesn't help when the column itself is missing. The hook keeps refetchInterval: 60_000 while it is in error, so a page left open sends a failing query to the environment every minute, plus react-query's retries. The Requires mz >= 26.41 doc at replicaStatusTimeline.ts:100 and the PR description's version table have the same off-by-two.

The 26.44 gate for the memory breakdown columns is correct: swap_of_ram_percent and heap_limit_percent are present in all three console utilization views at v26.44.0.

Fix: raise the constant to "26.43.0" and update the doc comment. Before 26.43 each episode is a single replica-wide row, so the other option is to add the process_id filter only at 26.43 and later. That keeps 26.41 and 26.42 supported at the cost of a second gate.

[linear-code]

CNS-174

CNS-173

[leedqin]

Thanks, confirmed: process_id first ships in v26.43.0. The hydration history gate is now 26.43.0, with the doc comment updated to match, in 0e48230.

[jdonelson]

On the open question: I think the inference can go wrong in two ways. The collector visits one replica per hydration_history_collection_interval. In an environment with N replicas, a finished episode can take about N minutes to be recorded.

• The stretch can be too long. Suppose a replica restarts and finishes hydrating, but the collector hasn't recorded that episode yet. Then someone creates a new index. The model sees an unhydrated object and no recorded episode since the restart. So it paints the whole time since the restart as hydrating. This lasts until the collector visits the replica.
• The stretch can be missing. Suppose the collector has already recorded the restart episode. Then someone creates a new index. The model finds an episode that started after the restart. So it doesn't paint a hydrating stretch at all. The new index's hydration doesn't appear until its own episode is recorded.

Nothing on mz_catalog_server says when an in-progress hydration started. So any stretch drawn for it is a guess. Two options:

1. Don't draw a stretch for in-progress hydration. Instead, show the current state at the "now" edge, for example "Hydrating · 3 objects". This is accurate, but less visual.
2. Start the inferred stretch at the later of two times: the restart, or the earliest creation time among the replica's unhydrated objects. Then drop the episode check. This fixes both cases, but it's still an approximation. useHydratingReplicaIds would need to keep the object ids. They might also need mapping through mz_object_global_ids to match mz_object_lifetimes.

[leedqin]

Thanks, this is a clear write-up. I went with option 1: the model no longer infers an in-progress stretch, and the status row shows a "Hydrating · N objects" badge at the now edge, counted per current replica from mz_compute_hydration_times (#39508). Finished episodes still draw from history.

[jdonelson]

nit: the commit message says hydration episodes need 26.41 and later, but the gate is 26.43 (for process_id), as the description says.

[leedqin]

Fixed the commit message.

[def-]

QA LLM Review

1. MEDIUM -- Status timeline is blank before the first transition once a replica's older history has aged out

console/src/platform/clusters/ClusterPage/ResourceUsage/resourceUsageModel.ts:153

transformReplicaTimeline gets its starting state only from a transition before the window, and uses sampleRange only when there are no transitions at all. Replica status history older than 30 days is deleted on every environmentd start, including a replica's latest row, so a long-running replica that is OOM-killed inside the window shows an empty status row from the window start up to the kill, although its metrics show it running the whole time.

Details

partially_truncate_status_history with StatusHistoryRetentionPolicy::TimeWindow (src/storage-controller/src/collection_mgmt.rs:1676) retracts every row older than replica_status_history_retention_window (30 days by default). It does not keep the newest row per replica. For a replica that has been stable for more than 30 days, the only rows left after an OOM are offline(t1, oom-killed) and online(t2). The loop then emits segments starting at t1, and [windowStart, t1) has none. The card then draws only the background there, and stateAt returns undefined, so the tooltip shows no status. This is the main case the card is for: a long-running replica that starts running out of memory. The existing test "paints recorded hydration episodes over running time" passes with this gap ([offline(10), online(20)] over [0, 100] starts at 10).

The fix is to fill the stretch before the earliest transition from sampleRange (it has samples, so it was running) whenever that transition falls inside the window, and not only when sorted is empty.

2. LOW -- Heap-limit polling stops on a metrics row whose heap_limit is still null

console/src/platform/clusters/queries.ts:1139

useReplicaHeapLimits stops refetching once every replica has any row in mz_cluster_replica_metrics, even if that row's heap_limit is NULL. Combined with staleTime: Infinity, a replica that is new when the page fetches never gets its heap limit for the rest of the session, so the limit labels lose their byte values.

Details

The replica metrics task ticks right after provisioning (src/controller/src/clusters.rs:962). At that point the pod has no metrics yet, so get_metrics returns ServiceProcessMetrics::default() and the controller writes a row with every column NULL. The next tick is 60s later. A page opened on a just-created or just-resized cluster, or one whose currentReplicaIds change because of a resize, can fetch in that window and see the replica as "reported". This contradicts the PR's "refetched only while a replica hasn't reported one", and the comment at line 1132. The fix is to build reported only from rows with a non-null heapLimit.

[def-]

QA LLM Review

1. MEDIUM -- Creation-times query matches clusters, replicas, roles and schemas that share an index's id

console/src/api/materialize/cluster/objectCreationTimes.ts:30

buildObjectCreationTimesQuery filters mz_object_lifetimes on id and event_type only. Cluster, replica, database, schema, role and network-policy ids use the same u<N> form as catalog item ids, so any of those created in the window under an index's or MV's id comes back as that object's creation. transformDdlEvents then labels it CREATE INDEX <name> at the wrong time, typically at a cluster resize.

Details

mz_object_lifetimes.id is details->>'id' for every audited object type, and details->>'replica_id' for replicas. Each of those kinds has its own id counter. On a fresh v26.47 dev image, I ran CREATE CLUSTER c SIZE '25cc', CREATE MATERIALIZED VIEW mv IN CLUSTER c ... (u2), CREATE INDEX i IN CLUSTER c ... (u3), then ALTER CLUSTER c SET (SIZE = '50cc'). The PR's query for ('u2', 'u3') returned five rows: the MV, the index, u2 cluster-replica, u3 cluster, and u3 cluster-replica. The last one is the resize. The card in #39508 draws a labeled marker for each row, so a resize shows up as CREATE INDEX i at the moment memory changes. That is the correlation the card exists to show. Two events for one id also give the markers duplicate React keys (key={event.id}).

The console's other mz_object_lifetimes joins (sourceList.ts, sinkList.ts, objectDetails.ts, fetchSecretsList.ts) also constrain object_type. The fix is to add .where("object_type", "in", ["index", "materialized-view"]), or to pass the caller's object types through.

[leedqin]

Thanks. 1 is fixed in 6d357be: without a transition before the window, the stretch up to the first transition is drawn as running from the replica's readings (empty readings don't count). 2 was already fixed in the earlier push: the heap limits query keeps polling while a replica's latest limit is null.

[leedqin]

Thanks, fixed in 6d357be: the creation-times query now filters object_type to indexes and materialized views.

[jdonelson]

This adds a per-viewer subscribe dataflow on mz_catalog_server for every multi-process cluster someone has open. The temporal filter bounds it, but it has to hold every row in the window to retract it later. In the 24h tier that's about 1,440 samples per process per open tab, for example around 23,000 rows for 2 replicas of 8 processes.

The EXPLAIN shows the lookup, but not the memory. Was the 24h tier's footprint on mz_catalog_server measured? If it's significant, limiting per-process series to the 3h tier until CNS-203 lands would bound it to an eighth.

[leedqin]

Thanks, it wasn't measured. On a local emulator with 2 replicas of 16 processes, the 24h temporal filter held ~12 MiB per open tab against ~2 MiB with a literal bound. The card no longer subscribes per process, and the builder kept for the workers drawer now starts at a literal time that moves forward once per window, in 9f3a5c2.