metrics: Enhance the o11y of TiFlash storage layer

[JaySon-Huang]

What problem does this PR solve?

Issue Number: close #10272

Problem Summary:

What is changed and how it works?

* Add metrics about
  * Number of keyspace, Storage instance, Segment instance, MemTable instance
  * Bytes of MemTable and allocated bytes of MemTable
* System table
  * Add `delta_cache_alloc_size` to `system.dt_segments`
  * Add `column_count`,`delta_cache_alloc_size` to `system.dt_tables`

New Grafana panels on TiFlash-Summary:

• "Threads" - a copy of "Threads" from TiFlash-Proxy-Details. We can check whether specified thread request a large IO, etc
• "Imbalanced read/write" - Check whether the CPU usage/read throughput/write throughput is imbalanced between different TiFlash instances
• "Memory trace" - A summary status of objects in memory, check whether there are OOM risk. Number of tables, segments, memtable bytes, etc

With the "Memory trace" panel, we can get more insight about the issue #10253 (comment) that mem-table may take too much memory.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

* deploy a cluster with 2 tiflash
* load chbenchmark 1500
* run chbenchmark workload with 50 tp thread and 1 ap thread for 60 minutes
"tiup bench ch --host 10.2.12.81 -P 8020 --warehouses 1500 run -D chbenchmark -T 50 -t 1 --time 60m"
* collect the metrics below

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Enhance the observability for TiFlash OOM risk under wide-column scenario

[JaySon-Huang]

/test pull-integration-next-gen

[JaySon-Huang]

/test pull-integration

[ti-chi-bot]

@JaySon-Huang: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test pull-integration-next-gen

/test pull-integration-test

/test pull-unit-next-gen

/test pull-unit-test

Use /test all to run all jobs.

Details

In response to this:

/test pull-integration

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[JaySon-Huang]

/test pull-integration-test

[JaySon-Huang]

/test pull-integration-test

[JaySon-Huang]

/test pull-integration-test

[JaySon-Huang]

New added fields:

• column_count
• delta_cache_alloc_size

[JaySon-Huang]

New added fields:

• delta_cache_alloc_size

[CalvinNeo]

lgtm

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2025-07-08 07:56:57.028637041 +0000 UTC m=+1986469.751816022: ☑️ agreed by Lloyd-Pottiger.
• 2025-07-08 08:05:25.817386988 +0000 UTC m=+1986978.540565970: ☑️ agreed by CalvinNeo.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CalvinNeo, JinheLin, Lloyd-Pottiger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [CalvinNeo,JinheLin,Lloyd-Pottiger]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.5: #10288.
But this PR has conflicts, please resolve them!

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-8.1: #10289.
But this PR has conflicts, please resolve them!

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-8.5: #10290.
But this PR has conflicts, please resolve them!