console: show how a replica's workers share its CPU on the redesigned cluster page

[leedqin]

Motivation

Sixth PR in the cluster details redesign stack, on top of #39607. The CPU chart shows a replica's CPU as a share of its allocation, averaged over its workers. The CPU troubleshooting guide points out that a replica whose work sits on one worker can peg a core while reporting a modest cpu_percent, and that high CPU with evenly spread work points elsewhere: an expensive object, ad-hoc queries, or an undersized cluster. The Console had no way to tell these apart.

This adds a Workers button to the CPU chart. It opens a drawer that reads the replica's per-worker scheduling counters live and leads with a finding in the guide's terms.

Description

What it reads. The drawer reads only while open.

• Every 5s, it reads each worker's total time in running dataflows. It sums the root operators of mz_scheduling_elapsed_per_worker (list_length(address) = 1), because a root's time includes its children.
• Every 30s, it reads the same counters per dataflow, named through mz_compute_exports. Transient (t) exports pool into one "Ad-hoc queries and subscribes" row.
• Both reads run on the replica being diagnosed, through cluster and cluster_replica on the session.
• They go over one WebSocket session (WebsocketQueryClient), held only while reading. An HTTP read would open a session each time, and a session start costs mz_catalog_server about 35 ms of CPU.
• Guards on the replica's load:
  • statement_timeout of 10s;
  • each read waits at least 10× as long as the last one took;
  • reads pause after 10 minutes and stop when the drawer closes.
• The replica's own CPU comes from a SUBSCRIBE to its raw per-minute metrics. It is an index lookup on replica_id, from 20 minutes before the drawer opens. The card's series can't be reused: past a 1h window they are per-bucket peaks, and past 24h they are polled.
• Readings are kept for 30 minutes in per-environment TanStack DB collections, so a reopened drawer continues from them.

What it shows.

• A verdict over the last 5 minutes:

  • a saturated worker: at least 90% of a core and twice the other workers' average;
  • uneven work: the busiest worker at half a core or more and twice the others;
  • light load: uneven, but no worker near its limit, so no warning;
  • every worker busy: under 10% idle;
  • high replica CPU (75% or more), split into time running dataflows and time outside them, with advice that depends on which dominates;
  • idle;
  • balanced.

  The busiest worker is compared with the others' average, not with an average that includes it, which can't reach 2× on two workers. Only a saturated worker outranks high CPU. Uneven work under high CPU becomes a note on that verdict.

  The skew verdicts name the object that took most of the busiest worker's time. They also give a copyable EXPLAIN ANALYZE CPU WITH SKEW for that object, pinned to the replica. Until 30s of readings exist, the drawer shows a first look from the lifetime counters instead.

• Two charts on one clock axis:

  • replica CPU against time running dataflows, as % of the allocation;
  • the busiest worker against the average worker, as % of a core over a trailing 30s.

• The busiest workers, each with the object it spent most of its time on. When nothing is skewed, this is one range covering all workers.

• Where the CPU went: a UniversalTable with one row per object, measured since the drawer opened or since each dataflow started. Each row has:

  • a bar per worker against a dashed even-share line;
  • the hottest worker, as a multiple of an even share;
  • its extra time;
  • its share and its CPU time.

  Since the drawer opened, the share is % of the replica's CPU allocation, the CPU chart's unit. Objects, an "Other dataflows" footer and a line for work outside running dataflows add up to the replica's CPU. Over each dataflow's lifetime there is no common span, so the share is of all dataflow CPU instead. Objects under 1% of the dataflow CPU aren't flagged as skewed. Header tooltips define each column.

Counter semantics. A dataflow's time leaves the counters when it shuts down (handle_shutdown in src/compute/src/logging/timely.rs). So the counters fall when a dataflow is dropped or the replica restarts.

• A reading where any worker's counter fell starts a new segment, and the verdict doesn't compare across it.
• A per-dataflow read that fell rebases the table's baseline.
• Work in dataflows that start and finish between two readings is never seen. The high-CPU verdict counts it as outside running dataflows and says so.

Supporting changes.

• useObjectLink is shared with the memory by object card.
• TimeAxis takes a tickFormat.
• HoverOverlay and PercentScale are exported.
• SubscribeCollection gains getRows.
• Size allocations include workers.
• SessionVariables includes statement_timeout.
• ChartHeader takes an action, and the CPU chart passes the Workers button through it.
• buildExtendedRequestQuery is exported.

The drawer ships with the redesign, so it is behind the same UI preview flag and off in self-managed.

Cost.

• On mz_catalog_server: one session per stretch of reading, and one metrics SUBSCRIBE per open drawer.
• On the diagnosed replica: one join and reduce over introspection every 5s, plus a per-dataflow read every 30s. That read returns one row per dataflow and worker.

The read time on a large replica isn't measured yet. The backoff and the 10-minute cap bound it.

Verification

• workerCpuModel.test.ts:
  • busy shares between readings and a 30s trailing average, with no line across a pause or a counter reset;
  • counter resets are detected, and the verdict window doesn't reach past one;
  • verdicts for saturated, uneven, light, balanced, every worker busy, idle and high CPU, including high CPU with idle dataflows and high CPU over uneven work;
  • a pegged worker is caught on a two-worker replica, and with peers each well behind it;
  • dataflow time is measured against cores;
  • per-object rows over a window, with ad-hoc dataflows pooled;
  • replica CPU per scrape, averaging processes.
• workerCpuSampler.test.ts:
  • reads every 5s and per-dataflow reads at most every 30s;
  • backs off to ten times a slow read;
  • pauses itself and closes its session at the 10-minute limit;
  • pausing and resuming during a read keeps one cadence and draws no false gap;
  • resuming after a failed read reads again on a new session;
  • a fallen worker counter marks a reset;
  • a fallen per-dataflow counter rebases the baseline;
  • a new sampler continues from earlier readings, marked after a gap;
  • a read that lands after stop is ignored.
• WebsocketQueryClient.test.ts:
  • queued queries run one at a time and resolve with their own rows;
  • an error rejects only its own query;
  • a close rejects the pending query and every later one;
  • numeric parameters are sent as strings.
• ObjectCpuTable.test.tsx:
  • a window is measured against the replica's allocation, with what ran outside dataflows;
  • a lifetime is measured against all dataflow time;
  • an object too small to matter isn't flagged as skewed.
• WorkerCpuVerdict.test.tsx:
  • a saturated worker's verdict offers a copyable EXPLAIN ANALYZE CPU WITH SKEW, pinned to the replica;
  • high CPU outside dataflows links to ad-hoc query troubleshooting;
  • high CPU in dataflows doesn't suggest splitting clusters;
  • low replica CPU is only called out when it is low;
  • the first look names a lifetime-skewed worker and leaves out stale CPU metrics.

🤖 Generated with Claude Code

[def-]

QA LLM Review

1. MEDIUM -- Skew verdicts can't fire on 2-worker replicas, which get "spread evenly" instead

console/src/platform/clusters/ClusterPage/WorkerCpu/workerCpuModel.ts:213

isSkewed compares the busiest worker with an average that includes the busiest worker, so the ratio can never exceed the worker count. On a 2-worker replica (M.1-xsmall, the smallest size that gets the Workers button), saturated and uneven can only fire if the other worker is completely idle. A worker pegged at 100% next to one at 20% is therefore reported as "Work is spread evenly", or as "this isn't worker skew" once replica CPU reaches 75%.

Details

With two workers, busiest >= 2 * (busiest + other) / 2 reduces to other <= 0. The introspection logging dataflows run on every worker, so the other worker never reads exactly zero. For [1.0, 0.2], the average is 0.6 and 1.0 < 1.2, so workerVerdict returns balanced. If the replica is at 75% or more, cpuVerdict takes the highCpu branch, and WorkerCpuVerdict.tsx:202 prints "the busiest worker 100% of its core, so this isn't worker skew." Either way, the EXPLAIN ANALYZE ... WITH SKEW suggestion never appears, and BusiestWorkers collapses into a single range. The lifetime check in the first look (WorkerCpuVerdict.tsx:137) has the same ceiling.

This is the case the PR description motivates the drawer with: one worker pegged while cpu_percent looks modest. Four workers are affected too. In [1.0, 0.4, 0.4, 0.4] one worker is pegged at 2.5× each peer with the replica near 55%, and the verdict is still balanced, because the busiest worker has to outweigh all the others combined. The tests only use four-worker inputs with near-idle peers, such as [0.97, 0.14, 0.12, 0.13].

A fix is to compare the busiest worker against the mean of the other workers. That gives 5× for [1.0, 0.2] and 2.5× for the four-worker case. The existing test cases keep their verdicts: [0.21, 0.17, 0.16, 0.18] comes out at 1.24× and stays balanced.

[def-]

QA LLM Review

1. MEDIUM -- High-CPU verdict headline says "spread evenly" while its own body reports uneven work

console/src/platform/clusters/ClusterPage/WorkerCpu/WorkerCpuVerdict.tsx:249

This push turns uneven work under high replica CPU into a note on the highCpu verdict (workerCpuModel.ts:293). The dataflow-dominated branch still uses the headline "mostly running dataflows spread evenly across its workers", so the drawer's top-line finding contradicts the "Work is uneven, w0 at 85% … against 42% for the other workers" sentence directly below it.

Details

This branch is easy to reach. Take a 2-worker replica at one worker per core, with workers [0.85, 0.42] and the replica at 80% CPU. workerVerdict returns uneven, since 0.85 ≥ 2 × 0.42 and 0.85 < 0.9. cpuVerdict then returns highCpu with dataflowFraction 0.635 and outsideFraction 0.165. Because outside ≤ dataflow, Verdict takes the branch at line 246 and renders "r1 is using 80% of its CPU, mostly running dataflows spread evenly across its workers." Four workers at [0.85, 0.42, 0.42, 0.42] hit the same branch.

Before this push the headline was accurate, because an uneven verdict never reached highCpu. No test renders highCpu with uneven set. The model test at workerCpuModel.test.ts:272 uses [0.7, 0.2, 0.2, 0.2] at 90% CPU, which lands in the "mostly outside" branch instead.

One fix is to drop "spread evenly across its workers" from the label when uneven is set.