Repository navigation
Conversation
leedqin
added this pull request to stack #39509
October 7, 2026 02:26
leedqin
force-pushed
the
cluster-page-live-utilization
branch
2 times, most recently
from
October 8, 2026 04:11
1e4837a to
e76d608
Compare
jdonelson
reviewed
Oct 8, 2026
| name: object?.name ?? size.objectId, | ||
| size: size.size, | ||
| memoryPercentage: | ||
| size.size === null ? null : (size.size / heapLimit) * 100, |
Contributor
There was a problem hiding this comment.
heapLimit is Number(largestReplica.heapLimit), which is 0 when no replica has reported metrics yet (or on the emulator). Dividing by it makes the largest object fill the whole bar and the legend read "Infinity%". Returning a null percentage like the polled path avoids that, but then the bar is empty and formatMemoryUsage shows "-" for every object, hiding sizes we do have. Could the card drop the bar when the heap limit is unknown and list each object's size on its own?
Contributor
Author
There was a problem hiding this comment.
Good point, done in 40df6d7: without a heap limit the card drops the bar and lists each object's size, with "Size on " as the subtitle.
leedqin
force-pushed
the
cluster-page-live-utilization
branch
from
October 12, 2026 23:36
e76d608 to
f193b47
Compare
leedqin
force-pushed
the
cluster-page-live-utilization
branch
3 times, most recently
from
October 15, 2026 15:59
902dac4 to
42bb039
Compare
leedqin
force-pushed
the
cluster-page-live-utilization
branch
3 times, most recently
from
October 15, 2026 18:11
e578716 to
40df6d7
Compare
leedqin
marked this pull request as ready for review
October 15, 2026 19:28
leedqin
force-pushed
the
cluster-page-live-utilization
branch
from
October 15, 2026 21:37
40df6d7 to
9422d83
Compare
…collections Fifth in the cluster details redesign stack. The resource usage charts and the memory by object card read their live data from TanStack DB collections instead of per-hook subscribe state. Charts: each cluster gets a collection fed by the 3h or 24h utilization SUBSCRIBE, and each subscribe covers its view's full retention. Switching windows within a tier reads the collection instead of resubscribing. Leaving the Metrics tab closes the subscribe; coming back paints the held rows at once and marks the card as refreshing until the new snapshot lands. Held rows that have all left the window show as loading, not as a window with no usage. Windows past 24h still poll, through a poll-only hook that mounts no subscribe hooks and so opens no idle WebSocket. The page no longer uses useReplicaUtilizationHistory, so its includeOfflineEvents option goes away. Per-process utilization moves to a collection too: useLiveProcessUtilization replaces useReplicaProcessUtilization, and still subscribes only for replicas whose size runs more than one process. Memory by object: each replica gets a collection fed by a SUBSCRIBE to mz_object_arrangement_sizes, an index lookup on replica_id. A live query joins it to the all objects collection for names and keeps the ten largest. The card waits for that collection, so objects never show as raw ids, and a replica without a heap limit shows sizes without shares. Sizes update live, and the card no longer polls sizes every 60s or object names every 5 minutes. Environments before 26.35 keep the polled card. Collections are keyed by environment and id, since cluster and replica ids repeat across environments, through one createScopedSubscribeCollections helper that drops an earlier environment's collections when the scope changes. They are not persisted. A subscribe collection's status gains isRefreshing for the held-rows case, and ending a collection's session marks its rows as held, so a return before the next subscribe starts shows loading rather than an empty window.
leedqin
force-pushed
the
cluster-page-live-utilization
branch
from
October 15, 2026 21:55
9422d83 to
bf415da
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Fifth PR in the cluster details redesign stack, on top of #39510. The resource usage charts and the memory by object card read their live data from TanStack DB collections instead of per-hook subscribe state, so switching windows reads data that's already loaded, and coming back to a cluster paints its last data at once.
Description
Charts. Each cluster gets a collection fed by the 3h or 24h utilization SUBSCRIBE, and each subscribe covers its view's full retention. Switching windows within a tier reads the collection instead of resubscribing. Leaving the Metrics tab closes the subscribe; coming back paints the held rows at once and shows the refreshing spinner until the new snapshot lands. If the held rows have all left the window, the card shows loading rather than "No resource usage". Windows past 24h still poll, through
usePolledReplicaUtilizationHistory, which mounts no subscribe hooks and so opens no idle WebSocket. The page no longer callsuseReplicaUtilizationHistory, so itsincludeOfflineEventsoption goes away.Per-process utilization.
useLiveProcessUtilizationkeeps each cluster's per-process samples in a collection too, replacing #39507'suseReplicaProcessUtilization. It still subscribes only for replicas whose size runs more than one process.Memory by object. Each replica gets a collection fed by a SUBSCRIBE to
mz_object_arrangement_sizes, an index lookup onreplica_id. A live query joins it toallObjectsCollectionfor names and keeps the ten largest. The card waits for that collection, so objects never show as raw ids, and a replica without a heap limit drops the bar and lists each object's size. The replica id comes from the largest-replica query. Sizes update live, and the card no longer polls sizes every 60s or object names every 5 minutes. Environments before 26.35 keep the polled card.Collections are keyed by environment and id, because cluster and replica ids repeat across environments, through one
createScopedSubscribeCollectionshelper that drops an earlier environment's collections when the scope changes. They aren't persisted. A subscribe collection's status gainsisRefreshing, set while held rows are served and no running subscribe has delivered a newer snapshot. Ending a collection's session sets it, so coming back before the next subscribe starts (for example while the cluster's lineage refetches) shows loading rather than an empty window.Measured in headless Chrome against a dev server (React dev mode, so absolute times run high), median of 3 runs, time until the charts are drawn:
The card's first open doesn't get faster: it waits on the largest replica query, which this PR doesn't change. On
mz_catalog_serverthe card's subscribe costs the same as the polls it replaces. Each held-open subscribe adds about 0.2 mCPU per viewer on environmentd for its progress updates.Verification
useLiveClusterUtilization.test.ts: 3h samples bin into the window's buckets, keeping each bucket's peak, and 24h buckets clip to the window in time order.useLiveLargestObjects.test.ts: the largest objects are named from the catalog, largest first; an object missing from the catalog is marked as an orphaned dataflow; only the ten largest are kept.MemoryByObject.test.tsx: on an environment that streams sizes, objects are named from the catalog; a replica without a heap limit shows no share; the card waits for the catalog before naming objects.ResourceUsage.test.tsx: a 1h window draws from the live subscribe.useLiveClusterUtilization.test.ts: coming back with held rows that have left the window reports loading and refreshing on every render until the new snapshot lands, including while an evicted lineage query refetches.useLiveProcessUtilization.test.ts: only multi-process replicas are subscribed, over one WebSocket; a single-process cluster opens none; the socket closes when the cluster is no longer shown and moves to the next cluster's replicas.useSubscribe.test.tsx: ending a collection session marks its rows as refreshing.subscribeCollection.test.ts: a scope change drops the earlier scope's collections.subscribeCollection.test.ts: held rows are marked refreshing until a new subscribe's snapshot lands.🤖 Generated with Claude Code