kafka: add throttle time metrics and grafana

[3AceShowHand]

What problem does this PR solve?

Issue Number: close #6013

What is changed and how it works?

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

`None`

Summary by CodeRabbit

• New Features

  • Added Kafka broker throttle-time monitoring with average and p99 measurements.
  • Added dashboard panels displaying throttle time by namespace, changefeed, instance, broker, keyspace, and throttle type.
  • Metrics are reported in seconds for clearer performance analysis.

• Improvements

  • Added automatic cleanup of broker throttle-time metrics when no longer active.
  • Updated monitoring dashboards to include the new Kafka throttle-time visualizations.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 084b26f2-9a6c-4d49-94b6-9753be94372b

📥 Commits

Reviewing files that changed from the base of the PR and between 01c2acd and e9c917f.

📒 Files selected for processing (5)

• metrics/grafana/ticdc_new_arch.json
• metrics/nextgengrafana/ticdc_new_arch_next_gen.json
• metrics/nextgengrafana/ticdc_new_arch_with_keyspace_name.json
• pkg/sink/kafka/metrics.go
• pkg/sink/kafka/metrics_collector.go

🚧 Files skipped from review as they are similar to previous changes (5)

• metrics/nextgengrafana/ticdc_new_arch_with_keyspace_name.json
• pkg/sink/kafka/metrics.go
• metrics/nextgengrafana/ticdc_new_arch_next_gen.json
• pkg/sink/kafka/metrics_collector.go
• metrics/grafana/ticdc_new_arch.json

Included review availability: Your plan includes up to 1 review per rolling hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The Kafka sink now exports broker throttle-time averages and p99 values through Prometheus. Three Grafana dashboards display the metric in seconds. Dashboard versions increase from 41 to 42.

Changes

Kafka throttle-time observability

Layer / File(s)	Summary
Metric definition and registration pkg/sink/kafka/metrics.go	Defines and registers the throttleTimeGauge metric with namespace, changefeed, broker, and type labels.
Broker histogram collection pkg/sink/kafka/metrics_collector.go	Collects average and p99 broker throttle-time values in seconds and removes their labels during broker cleanup.
Grafana dashboard panels metrics/grafana/ticdc_new_arch.json, metrics/nextgengrafana/ticdc_new_arch_next_gen.json, metrics/nextgengrafana/ticdc_new_arch_with_keyspace_name.json	Adds Kafka Throttle Time panels with broker and deployment dimensions, seconds-formatted axes, and updated dashboard versions.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to e9c91

This PR adds Kafka throttle-time metrics and dashboard updates; no actionable merge-blocking risk remains based on the available review evidence.

Sequence Diagram(s)

sequenceDiagram
  participant KafkaBroker
  participant MetricsCollector
  participant Prometheus
  participant Grafana
  KafkaBroker->>MetricsCollector: expose throttle-time histogram
  MetricsCollector->>Prometheus: export average and p99 gauge values
  Grafana->>Prometheus: query ticdc_sink_kafka_producer_throttle_time
  Prometheus-->>Grafana: return broker throttle-time series

Loading

Possibly related PRs

• pingcap/ticdc#5873: Both changes modify Kafka metrics definitions and collection.

Suggested reviewers: asddongmen, wk989898

Poem

A rabbit watched the brokers glow,
As throttle-time metrics flow.
Mean and p99 came into view,
With Grafana charts in seconds too.
Three dashboards brightened the night.
“Hop,” said Bun, “the data looks right!”

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description includes an issue number but leaves the change details, tests, compatibility answers, documentation decision, and release note incomplete.	Describe how the metrics work, identify completed tests, answer both questions, state the documentation impact, and add a release note for this new feature.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main changes: Kafka throttle time metrics and Grafana dashboards.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wk989898

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [wk989898]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-08-17 10:00:47.50657945 +0000 UTC m=+3645433.542674516: ☑️ agreed by wk989898.

[3AceShowHand]

/test all