From e06e865549cb07f26020097e330738b17235df7b Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 8 Jul 2026 16:52:14 -0400 Subject: [PATCH 1/2] DOC-226: Document per-entity client quota metrics (26.2) Adds a "Track quota use per entity" subsection to the Monitor client throughput section, documenting ENG-37 (redpanda PR #30832): - kafka_per_entity_quota_metrics cluster property (opt-in, no restart) - The two new per-entity counters on /public_metrics and /metrics - Entity/quota-type label scheme and cardinality bounding behavior - Utilization calculation guidance with a PromQL example Builds on the existing aggregate-metrics content rather than duplicating it. Gated to Self-Managed pending confirmation of Cloud support for the new cluster property. DRAFT-VERIFY comments mark items to confirm against a 26.2 build. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../manage-throughput.adoc | 57 +++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc index ce1e6da001..421e6a4767 100644 --- a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc +++ b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc @@ -727,6 +727,63 @@ TRACE 2024-06-14 15:37:59,195 [shard 2:main] kafka_quotas - connection_context. ---- endif::[] +// DRAFT-VERIFY (DOC-226): This subsection documents ENG-37 (redpanda PR #30832, merged to +// dev 2026-06-24, targets v26.2). Verify all flagged items against a 26.2 build before +// publishing. SME: Mateusz Najda (@mnajda-redpanda). +// DRAFT-VERIFY (DOC-226): Confirm whether Redpanda Cloud exposes the +// kafka_per_entity_quota_metrics cluster property. Gated to Self-Managed until confirmed. +ifndef::env-cloud[] +==== Track quota use per entity + +The quota metrics above aggregate throttling and throughput by quota rule and quota type, so they cannot tell you which specific client is approaching its limit. Per-entity quota metrics label each series with the throttled entity's identity, so you can measure how much of an enforced quota each client actually uses, identify throttled clients by name, and alert on a workload before it saturates its quota. + +Per-entity quota metrics are disabled by default because each throttled entity adds metric series. To enable them: + +[,bash] +---- +rpk cluster config set kafka_per_entity_quota_metrics true +---- + +The change takes effect without a broker restart. + +When enabled, Redpanda exposes two additional counters: + +* Total per-entity quota throttling delay, in milliseconds: +** `/public_metrics` - `redpanda_kafka_quotas_client_quota_throttle_time_ms_by_entity` +** `/metrics` - `vectorized_kafka_quotas_client_quota_throttle_time_ms_by_entity` +* Per-entity quota throughput (bytes for produce and fetch quotas, partition mutations for partition mutation quotas): +** `/public_metrics` - `redpanda_kafka_quotas_client_quota_throughput_by_entity` +** `/metrics` - `vectorized_kafka_quotas_client_quota_throughput_by_entity` + +// DRAFT-VERIFY (DOC-226): Confirm the exposed internal metric names on a 26.2 build, then +// replace the plain code spans above with xrefs to the regenerated metrics reference pages +// (anchors do not exist until /gen-metrics runs for 26.2). + +Each series is labeled with the entity identity and the quota type: + +* `redpanda_quota_type`: Always present. One of `produce_quota`, `fetch_quota`, or `partition_mutation_quota`. +* `redpanda_quota_user`: The user principal, for user-based quotas. +* `redpanda_quota_client_id`: The client ID, for client ID-based quotas. +* `redpanda_quota_group_name`: The client ID prefix, for quotas that apply to a <>. +// DRAFT-VERIFY (DOC-226): Confirm the redpanda_quota_group_name label corresponds to +// client-id-prefix quotas (not consumer groups) on a live build. + +Combined quotas, such as a user with a specific client ID, include each matching label on the same series. + +To keep metric cardinality bounded, Redpanda registers a per-entity series only while an entity is actively being throttled, and removes the series after the entity has been idle. An entity appears in these metrics only after it has been throttled at least once. From that point, Redpanda records the entity's throughput on every request, not only on throttled requests, so a Prometheus `rate()` query reflects the entity's actual throughput. + +To measure how much of its quota an entity is using, divide the entity's throughput rate by its configured limit. For example, the following query returns the produce throughput rate for each throttled client ID: + +[,promql] +---- +sum by (redpanda_quota_client_id) ( + rate(redpanda_kafka_quotas_client_quota_throughput_by_entity{redpanda_quota_type="produce_quota"}[5m]) +) +---- + +Compare the result with the entity's configured limit from xref:reference:rpk/rpk-cluster/rpk-cluster-quotas-describe.adoc[`rpk cluster quotas describe`]. A ratio close to 1 means the client is saturating its quota and its requests are being delayed. +endif::[] + == See also - xref:manage:cluster-maintenance/about-throughput-quotas.adoc[] From 2d65e6bf5e4be800a5bcf6c9bd5906c2930c03ed Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 8 Jul 2026 17:46:56 -0400 Subject: [PATCH 2/2] DOC-226: Rework per-entity metrics intro to contextualize the feature Intro now follows the situation / what-it-does / benefit / when-to-use formula instead of leading with the metrics mechanics. Co-Authored-By: Claude Opus 4.8 (1M context) --- modules/manage/pages/cluster-maintenance/manage-throughput.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc index 421e6a4767..c2587ad027 100644 --- a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc +++ b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc @@ -735,7 +735,7 @@ endif::[] ifndef::env-cloud[] ==== Track quota use per entity -The quota metrics above aggregate throttling and throughput by quota rule and quota type, so they cannot tell you which specific client is approaching its limit. Per-entity quota metrics label each series with the throttled entity's identity, so you can measure how much of an enforced quota each client actually uses, identify throttled clients by name, and alert on a workload before it saturates its quota. +When a workload slows down because a client hits its throughput quota, the aggregate quota metrics can confirm that throttling is happening, but they cannot tell you which client is being throttled or how close each client is to its limit. Per-entity quota metrics answer these questions: Redpanda labels each throttle-time and throughput series with the identity of the throttled entity, so you can measure how much of an enforced quota each client actually uses. Use these metrics to identify throttled clients by name, alert on a workload before it saturates its quota, and right-size quota values based on observed usage. Per-entity quota metrics are disabled by default because each throttled entity adds metric series. To enable them: