Skip to content

console: add a memory by object card to the redesigned cluster page - #39510

Open
leedqin wants to merge 1 commit into
cluster-page-resource-usagefrom
cluster-page-memory-by-object
Open

leedqin wants to merge 1 commit into
cluster-page-resource-usagefrom
cluster-page-memory-by-object

Conversation

@leedqin

@leedqin leedqin commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Fourth PR in the cluster details redesign stack, on top of #39508. The design replaces the resource intensive objects table with a memory by object chart, so the Metrics tab shows where a replica's memory goes next to the resource usage card. Fixes CNS-175

Description

  • On the new page's Metrics tab, a card titled Top 10 Objects by Memory (or Objects by Memory when there are fewer than 10) replaces the resource intensive objects table.
  • It reads the same query as the table: the largest objects on the cluster's largest replica (up to 10), each as a share of that replica's heap limit.
  • One bar of the heap limit is split by object, with ticks at 0 to 100%. Hovering a segment names the object. The legend gives each object's size and share, formatted as the table did. Objects are colored by rank with the Freshness page's palette (generateRainbowPalette).
  • Each object links to its Maintained Objects drawer (maintained-objects/:objectId). While maintained-objects-ui-50 is off that route doesn't exist, so the link goes to the object's workflow graph, as the table's rows did. A dropped object's dataflow keeps its name and has no link.
  • The largest replica's loading and error states move from LargestMaintainedQueries into LargestReplicaBoundary, which the table and the card share.
  • The largest replica's heap limit now covers all of its processes: the per-process heap_limit times the size's process count from mz_cluster_replica_sizes. Object sizes are replica-wide, so on a multi-process replica every share was inflated by the process count, which also affected the classic table. The process count comes from the size rather than a sum over mz_cluster_replica_metrics rows, because a process whose metrics fetch failed has no limit to sum. Checked on a staging environment: a 2-process replica's limit doubles, and EXPLAIN adds only a differential join on mz_cluster_replica_sizes_ind.

The design plots memory by object over time. Per-object memory is only reported as a current value by default: mz_object_arrangement_size_history isn't collected unless arrangement_size_history_collection_interval is set, and it samples too coarsely for short windows. So the card shows the current snapshot for now.

Not carried over from the table: the freshness column and the hint that memory grows until hydration finishes. Not included from the design: the replica dropdown. The card uses the largest replica, as the table did.

Verification

image

MemoryByObject.test.tsx covers:

  • the title, the replica, and each object's size and share;
  • links to the Maintained Objects drawer, and to the workflow graph while that page is off;
  • a dropped object's dataflow shown without a link;
  • nothing rendered for a cluster without replicas, once the replica query has answered;
  • the busy-replica message when the objects fail to load;
  • the bar stopping at the heap limit.

LargestMaintainedQueries.test.tsx passes unchanged after the shared boundary moved out.

largestClusterReplica.test.sql.ts covers a replica whose size runs four processes.

🤖 Generated with Claude Code

@leedqin
leedqin added this pull request to stack #39509 October 2, 2026 23:31
@leedqin
leedqin force-pushed the cluster-page-memory-by-object branch from 51e5abe to 32c8f7d Compare October 2, 2026 23:42
@leedqin
leedqin force-pushed the cluster-page-memory-by-object branch from 32c8f7d to 9c821e9 Compare October 2, 2026 23:51
@leedqin
leedqin force-pushed the cluster-page-memory-by-object branch from 9c821e9 to 66f1ba1 Compare October 3, 2026 00:01
@leedqin
leedqin force-pushed the cluster-page-memory-by-object branch 2 times, most recently from f4c71cc to e754e35 Compare October 3, 2026 16:59
@leedqin
leedqin force-pushed the cluster-page-memory-by-object branch 2 times, most recently from 82c98dc to 31eae67 Compare October 5, 2026 15:19
@leedqin
leedqin marked this pull request as ready for review October 5, 2026 15:31
@leedqin
leedqin requested a review from a team as a code owner October 5, 2026 15:31
@leedqin
leedqin requested a review from jdonelson October 5, 2026 15:31
@linear-code

linear-code Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

CNS-173

CNS-175

<VStack spacing={4} alignItems="stretch">
<HStack spacing={2} alignItems="baseline" flexWrap="wrap">
<Text as="h3" textStyle="heading-sm">
Memory consumption by object ({objects.length})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The count is the number the query returns, which caps at 10, so a cluster with hundreds of objects shows "(10)". Something like "Top 10 objects by memory" would avoid reading it as a total.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to "Top 10 objects by memory".

@jdonelson

Copy link
Copy Markdown
Contributor

I have some layout-related feedback:

Screenshot 2026-10-07 at 4 39 40 PM Screenshot 2026-10-07 at 4 39 52 PM

The legends for these 2 graphs are tough to read. On the freshness graph it overflows to the left. Maybe remove them both and move the color swatches to the Freshness table, like we did on the new Freshness page? You'd have to make each object use the same color on both graphs.

@leedqin
leedqin force-pushed the cluster-page-memory-by-object branch from a7b956d to 792d44c Compare October 8, 2026 03:48
@leedqin

leedqin commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

I have some layout-related feedback: @jdonelson

I have a followup PR coming to remove this Freshness section so you can add your Freshness tab/ graph once your page is shipped and I agree that we can use the Freshness Swatch legend

@leedqin
leedqin force-pushed the cluster-page-memory-by-object branch from 792d44c to 1492640 Compare October 8, 2026 04:11
@leedqin

leedqin commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Thanks. This PR now colors the card's objects by rank with the Freshness page's palette (generateRainbowPalette), so it matches that page. The legends and the Freshness table are unchanged here.

@leedqin
leedqin force-pushed the cluster-page-memory-by-object branch from 1492640 to ae5cc95 Compare October 12, 2026 23:36
<VStack spacing={4} alignItems="stretch">
<HStack spacing={2} alignItems="baseline" flexWrap="wrap">
<Text as="h3" textStyle="heading-sm">
Top {TOP_OBJECT_COUNT} objects by memory

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "Top 10" implies there are more than 10, but it also shows when there are fewer (the test fixture has 2). Suggest "Objects by Memory" when there are fewer than TOP_OBJECT_COUNT, and "Top 10 Objects by Memory" otherwise. Also Title Case, to match "Resource Usage" above.

Suggested change
Top {TOP_OBJECT_COUNT} objects by memory
{objects.length < TOP_OBJECT_COUNT
? "Objects by Memory"
: `Top ${TOP_OBJECT_COUNT} Objects by Memory`}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied, with your test changes, in 3957f75.

);
await renderMemoryByObject();

expect(await screen.findByText("Top 10 objects by memory")).toBeVisible();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Goes with the heading suggestion on MemoryByObject.tsx: the 2-object fixture is fewer than TOP_OBJECT_COUNT, so the heading becomes "Objects by Memory". Commit both together, or this test fails.

Suggested change
expect(await screen.findByText("Top 10 objects by memory")).toBeVisible();
expect(await screen.findByText("Objects by Memory")).toBeVisible();

server.use(largestReplicaHandler([]));
await renderMemoryByObject();

expect(screen.queryByText(/objects by memory/)).not.toBeInTheDocument();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also goes with the heading suggestion: this regex is case-sensitive, so after the Title Case change it would match neither heading and the assertion would pass even if one rendered. /Objects by Memory/ matches both forms.

Suggested change
expect(screen.queryByText(/objects by memory/)).not.toBeInTheDocument();
expect(screen.queryByText(/Objects by Memory/)).not.toBeInTheDocument();

@leedqin
leedqin force-pushed the cluster-page-memory-by-object branch from ae5cc95 to f6bfc47 Compare October 15, 2026 15:11
@leedqin
leedqin force-pushed the cluster-page-memory-by-object branch 2 times, most recently from 1cd434f to cd0fecc Compare October 15, 2026 15:59
@leedqin
leedqin force-pushed the cluster-page-memory-by-object branch 3 times, most recently from a652d45 to 3957f75 Compare October 15, 2026 18:11
@def-

def- commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

QA LLM Review

1. MEDIUM -- Bar's 100% is one process's heap limit, but object sizes span every process

console/src/platform/clusters/ClusterPage/MemoryByObject.tsx:88

The card treats each object's memoryPercentage as an additive share of one heap-limit bar. On a replica with more than one process, that number is the replica-wide arrangement size divided by a single process's heap limit, so every share is multiplied by the process count. On those replicas the bar fills to 100% while the replica still has headroom, and segmentWidths cuts smaller objects out of the bar even though the legend still lists them.

Details

The denominator is MAX(crm.heap_limit) (console/src/api/materialize/expressionBuilders.ts:229, used via LargestMaintainedQueries.tsx:178). mz_cluster_replica_metrics has one row per (replica_id, process_id), and heap_limit comes from the per-pod memory and disk limits (src/controller/src/clusters.rs:832). The base PR's sharedHeapLimitBytes says the same thing: "One process's heap limit in bytes". The numerator is a total for the whole replica. mz_object_arrangement_sizes is a COUNT(*) over every worker's raw size logs (src/adapter/src/coord/introspection.rs:656), and the pre-v26.35 query sums the same logs.

Example: a 6400cc replica (scale: 2, misc/helm-charts/operator/values.yaml:252) with each process's heap limit H. Objects A, B and C use 0.6H, 0.5H and 0.4H across both processes, so together they hold 37.5% of the replica's 2H heap. The card reports 60%, 50% and 40%. The bar shows A at 60% and B clamped to 40%, so it reads as completely full. C gets no segment at all. These objects are on the largest clusters, which is where this card matters most.

The classic table already showed the inflated percentage for each row. The new part is the stacked 0–100% axis, plus the clamp that hides the overflow. A fix is to use the replica's total heap as the denominator, for example SUM(heap_limit) across processes or the limit times mz_cluster_replica_sizes.processes. That would correct the table too.

@leedqin
leedqin force-pushed the cluster-page-memory-by-object branch from 3957f75 to b322b4d Compare October 15, 2026 21:37
Fourth in the cluster details redesign stack. On the new page's Metrics
tab, a memory by object card replaces the resource intensive objects
table. It reads the same query: the largest objects on the cluster's
largest replica, as shares of that replica's heap limit. One bar of the
heap limit is split by object, and a legend gives each object's size and
share. Objects are colored by rank with the Freshness page's palette.

The design plots this over time, but per-object memory is only reported as
a current value by default (mz_object_arrangement_size_history isn't
collected unless arrangement_size_history_collection_interval is set), so
the card shows the current snapshot.

Each object links to its Maintained Objects drawer, or to its workflow
graph while the maintained-objects-ui-50 flag is off.

The largest replica's loading and error states move into
LargestReplicaBoundary so the classic table and the card share them.

The largest replica's heap limit now covers all of its processes: the
per-process limit times the size's process count from
mz_cluster_replica_sizes. Object sizes are replica-wide, so on a
multi-process replica every share was inflated by the process count, in
the classic table too.
@leedqin
leedqin force-pushed the cluster-page-memory-by-object branch from b322b4d to a66742d Compare October 15, 2026 21:55

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants