Skip to content

console: keep the redesigned cluster page's live data in TanStack DB collections - #39607

Open
leedqin wants to merge 1 commit into
cluster-page-memory-by-objectfrom
cluster-page-live-utilization
Open

leedqin wants to merge 1 commit into
cluster-page-memory-by-objectfrom
cluster-page-live-utilization

Conversation

@leedqin

@leedqin leedqin commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Fifth PR in the cluster details redesign stack, on top of #39510. The resource usage charts and the memory by object card read their live data from TanStack DB collections instead of per-hook subscribe state, so switching windows reads data that's already loaded, and coming back to a cluster paints its last data at once.

Description

Charts. Each cluster gets a collection fed by the 3h or 24h utilization SUBSCRIBE, and each subscribe covers its view's full retention. Switching windows within a tier reads the collection instead of resubscribing. Leaving the Metrics tab closes the subscribe; coming back paints the held rows at once and shows the refreshing spinner until the new snapshot lands. If the held rows have all left the window, the card shows loading rather than "No resource usage". Windows past 24h still poll, through usePolledReplicaUtilizationHistory, which mounts no subscribe hooks and so opens no idle WebSocket. The page no longer calls useReplicaUtilizationHistory, so its includeOfflineEvents option goes away.

Per-process utilization. useLiveProcessUtilization keeps each cluster's per-process samples in a collection too, replacing #39507's useReplicaProcessUtilization. It still subscribes only for replicas whose size runs more than one process.

Memory by object. Each replica gets a collection fed by a SUBSCRIBE to mz_object_arrangement_sizes, an index lookup on replica_id. A live query joins it to allObjectsCollection for names and keeps the ten largest. The card waits for that collection, so objects never show as raw ids, and a replica without a heap limit drops the bar and lists each object's size. The replica id comes from the largest-replica query. Sizes update live, and the card no longer polls sizes every 60s or object names every 5 minutes. Environments before 26.35 keep the polled card.

Collections are keyed by environment and id, because cluster and replica ids repeat across environments, through one createScopedSubscribeCollections helper that drops an earlier environment's collections when the scope changes. They aren't persisted. A subscribe collection's status gains isRefreshing, set while held rows are served and no running subscribe has delivered a newer snapshot. Ending a collection's session sets it, so coming back before the next subscribe starts (for example while the cluster's lineage refetches) shows loading rather than an empty window.

Measured in headless Chrome against a dev server (React dev mode, so absolute times run high), median of 3 runs, time until the charts are drawn:

Step Before After
Open a cluster 594 ms 630 ms
Switch 1h to 3h 222 ms 25 ms
Switch back to 1h 238 ms 24 ms
Objects tab back to Metrics 338 ms 102 ms
Back to the cluster from the list 347 ms 95 ms

The card's first open doesn't get faster: it waits on the largest replica query, which this PR doesn't change. On mz_catalog_server the card's subscribe costs the same as the polls it replaces. Each held-open subscribe adds about 0.2 mCPU per viewer on environmentd for its progress updates.

Verification

  • useLiveClusterUtilization.test.ts: 3h samples bin into the window's buckets, keeping each bucket's peak, and 24h buckets clip to the window in time order.
  • useLiveLargestObjects.test.ts: the largest objects are named from the catalog, largest first; an object missing from the catalog is marked as an orphaned dataflow; only the ten largest are kept.
  • MemoryByObject.test.tsx: on an environment that streams sizes, objects are named from the catalog; a replica without a heap limit shows no share; the card waits for the catalog before naming objects.
  • ResourceUsage.test.tsx: a 1h window draws from the live subscribe.
  • useLiveClusterUtilization.test.ts: coming back with held rows that have left the window reports loading and refreshing on every render until the new snapshot lands, including while an evicted lineage query refetches.
  • useLiveProcessUtilization.test.ts: only multi-process replicas are subscribed, over one WebSocket; a single-process cluster opens none; the socket closes when the cluster is no longer shown and moves to the next cluster's replicas.
  • useSubscribe.test.tsx: ending a collection session marks its rows as refreshing.
  • subscribeCollection.test.ts: a scope change drops the earlier scope's collections.
  • subscribeCollection.test.ts: held rows are marked refreshing until a new subscribe's snapshot lands.

🤖 Generated with Claude Code

@leedqin
leedqin added this pull request to stack #39509 October 7, 2026 02:26
@leedqin
leedqin force-pushed the cluster-page-live-utilization branch 2 times, most recently from 1e4837a to e76d608 Compare October 8, 2026 04:11
name: object?.name ?? size.objectId,
size: size.size,
memoryPercentage:
size.size === null ? null : (size.size / heapLimit) * 100,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

heapLimit is Number(largestReplica.heapLimit), which is 0 when no replica has reported metrics yet (or on the emulator). Dividing by it makes the largest object fill the whole bar and the legend read "Infinity%". Returning a null percentage like the polled path avoids that, but then the bar is empty and formatMemoryUsage shows "-" for every object, hiding sizes we do have. Could the card drop the bar when the heap limit is unknown and list each object's size on its own?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, done in 40df6d7: without a heap limit the card drops the bar and lists each object's size, with "Size on " as the subtitle.

@leedqin
leedqin force-pushed the cluster-page-live-utilization branch from e76d608 to f193b47 Compare October 12, 2026 23:36
@leedqin
leedqin force-pushed the cluster-page-live-utilization branch 3 times, most recently from 902dac4 to 42bb039 Compare October 15, 2026 15:59
@leedqin
leedqin force-pushed the cluster-page-live-utilization branch 3 times, most recently from e578716 to 40df6d7 Compare October 15, 2026 18:11
@leedqin
leedqin marked this pull request as ready for review October 15, 2026 19:28
@leedqin
leedqin requested a review from a team as a code owner October 15, 2026 19:28
@leedqin
leedqin requested a review from jdonelson October 15, 2026 19:28
@linear-code

linear-code Bot commented Oct 15, 2026

Copy link
Copy Markdown

CNS-203

@leedqin
leedqin force-pushed the cluster-page-live-utilization branch from 40df6d7 to 9422d83 Compare October 15, 2026 21:37
…collections

Fifth in the cluster details redesign stack. The resource usage charts and
the memory by object card read their live data from TanStack DB collections
instead of per-hook subscribe state.

Charts: each cluster gets a collection fed by the 3h or 24h utilization
SUBSCRIBE, and each subscribe covers its view's full retention. Switching
windows within a tier reads the collection instead of resubscribing.
Leaving the Metrics tab closes the subscribe; coming back paints the held
rows at once and marks the card as refreshing until the new snapshot lands.
Held rows that have all left the window show as loading, not as a window
with no usage. Windows past 24h still poll, through a poll-only hook that
mounts no subscribe hooks and so opens no idle WebSocket. The page no
longer uses useReplicaUtilizationHistory, so its includeOfflineEvents option
goes away.

Per-process utilization moves to a collection too: useLiveProcessUtilization
replaces useReplicaProcessUtilization, and still subscribes only for replicas
whose size runs more than one process.

Memory by object: each replica gets a collection fed by a SUBSCRIBE to
mz_object_arrangement_sizes, an index lookup on replica_id. A live query
joins it to the all objects collection for names and keeps the ten largest.
The card waits for that collection, so objects never show as raw ids, and
a replica without a heap limit shows sizes without shares. Sizes update
live, and the card no longer polls sizes every 60s or object names every 5
minutes. Environments before 26.35 keep the polled card.

Collections are keyed by environment and id, since cluster and replica ids
repeat across environments, through one createScopedSubscribeCollections
helper that drops an earlier environment's collections when the scope
changes. They are not persisted. A subscribe collection's status gains
isRefreshing for the held-rows case, and ending a collection's session
marks its rows as held, so a return before the next subscribe starts shows
loading rather than an empty window.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants