From 1b3db2ace59b76f888861713b8acd1494a5c09db Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Wed, 13 May 2026 15:34:45 +0800 Subject: [PATCH 1/5] Create empty translation PR From 4260fedb2c493d142d7dd2a95e531271bad0dcfe Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Wed, 13 May 2026 07:37:05 +0000 Subject: [PATCH 2/5] Auto-sync: Update English docs from Chinese PR Synced from: https://github.com/pingcap/docs-cn/pull/20955 Target PR: https://github.com/pingcap/docs/pull/22882 AI Provider: azure Co-authored-by: github-actions[bot] --- configure-memory-usage.md | 144 +++++++++++++++++++++++++++++++++++++- system-variables.md | 64 +++++++++++++++++ 2 files changed, 206 insertions(+), 2 deletions(-) diff --git a/configure-memory-usage.md b/configure-memory-usage.md index 25dfbd93bc0f4..2478288811525 100644 --- a/configure-memory-usage.md +++ b/configure-memory-usage.md @@ -208,8 +208,6 @@ The following example uses a memory-consuming SQL statement to demonstrate the d 9 rows in set (1 min 37.428 sec) ``` -## Others - ### Mitigate OOM issues by configuring `GOMEMLIMIT` GO 1.19 introduces an environment variable [`GOMEMLIMIT`](https://pkg.go.dev/runtime@go1.19#hdr-Environment_Variables) to set the memory limit that triggers GC. @@ -227,3 +225,145 @@ To verify the performance of `GOMEMLIMIT`, a test is performed to compare the sp - In TiDB v6.1.3, `GOMEMLIMIT` is set to 40000 MiB. It is found that the simulated workload runs stably for a long time, OOM does not occur in the TiDB server, and the maximum memory usage of the process is stable at around 40.8 GiB: ![v6.1.3 workload no oom with GOMEMLIMIT](/media/configure-memory-usage-613-no-oom.png) + +## Memory arbitrator mode + +In versions earlier than TiDB v9.0.0, the [memory control mechanism](#how-to-configure-the-memory-usage-threshold-of-a-tidb-server-instance) had the following issues: + +- When the memory usage of a TiDB instance exceeded the limit, TiDB might randomly terminate running SQL statements. +- Memory resources were used first and then reported, and memory usage information was isolated between different SQL statements, making it difficult for TiDB to uniformly schedule and control memory resources at the instance level. +- Under high memory pressure, the overhead of Go garbage collection (Garbage Collection, GC) increased significantly, and in severe cases could lead to out of memory (OOM) issues. + +Starting from v9.0.0, TiDB introduces memory arbitrator mode. This mode introduces a global memory arbitrator in each TiDB instance to centrally manage and schedule the memory resources of the instance from top to bottom, mitigating the preceding issues. + +> **Warning:** +> +> This feature is experimental and is not recommended for use in the production environment. This feature might be changed or removed without prior notice. If you find a bug, report it by submitting an [issue](https://github.com/pingcap/tidb/issues) on GitHub. + +You can enable memory arbitrator mode using the [`tidb_mem_arbitrator_mode`](/system-variables.md#tidb_mem_arbitrator_mode-introduced-in-v900) system variable or the `instance.tidb_mem_arbitrator_mode` parameter in the TiDB configuration file. + +- `disable`: disables memory arbitrator mode + +- `standard` or `priority`: enables memory arbitrator mode. After it is enabled: + - Memory resources are used with a subscribe-before-allocation mechanism and are uniformly scheduled by the memory arbitrator in each TiDB instance + - The overall memory usage of the TiDB instance is expected not to exceed the limit of [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-introduced-in-v640), and the [alarm for high memory usage](#alarm-for-high-memory-usage-of-tidb-server) no longer takes effect + - The control behavior of the following system variables remains effective: + - [`tidb_mem_oom_action`](/system-variables.md#tidb_mem_oom_action-introduced-in-v610) + - [`tidb_mem_quota_query`](/system-variables.md#tidb_mem_quota_query) + - [`max_execution_time`](/system-variables.md#max_execution_time) + +When memory resources are insufficient or there is an OOM risk, the arbitrator can reclaim memory resources by terminating SQL, but it does not terminate `DDL`, `DCL`, or `TCL` SQL types. Terminated SQL returns error code `8180` to the client. The error format is: `Query execution was stopped by the global memory arbitrator [reason=?, path=?] [conn=?]`. The related fields are explained as follows: + +- `conn`: connection (session) ID +- `reason`: the specific reason why the SQL was terminated +- `path`: the stage at which the SQL was terminated (if the error does not contain the `path` field, the SQL was terminated during the execution stage) + - `ParseSQL`: parsing + - `CompilePlan`: compiling the execution plan + +Memory arbitrator mode pools the arbitrable memory resources in each TiDB instance. The memory resources corresponding to `tidb_server_memory_limit` are divided into the following four categories, which can be viewed through the `Mem Quota Stats` monitoring metric: + +- `allocated`: allocated memory quota, including but not limited to the memory pool to which the SQL belongs and the memory pools of various public modules +- `available`: allocatable memory quota +- `buffer`: memory quota of the buffer +- `out-of-control`: memory quota not controlled by the arbitrator, including but not limited to: + - unsafe or disabled memory + - memory waiting for garbage collection + - memory used by the Go runtime + - other untracked memory usage + +`out-of-control` is a risk metric that requires special attention. It is dynamically calculated by the arbitrator, with a minimum value of `tidb_server_memory_limit - tidb_mem_arbitrator_soft_limit`. The arbitrator quantifies this type of risky memory quota as accurately as possible and reserves space for it during memory allocation, thereby reducing the risk of OOM in the TiDB instance. + +### `standard` mode + +In `standard` mode, during SQL execution, SQL dynamically subscribes to memory resources from the arbitrator and blocks to wait when resources are insufficient. The arbitrator processes subscription requests on a first in, first out (FIFO) basis. + +- When parsing SQL or compiling the execution plan, the required memory quota is estimated by TiDB and is proportional to the number of SQL keywords. +- If global memory resources are insufficient, the arbitrator fails the subscription request and terminates the SQL. The `reason` field in the returned error is `CANCEL(out-of-quota & standard-mode)`. + +### `priority` mode + +In `priority` mode, during SQL execution, SQL dynamically subscribes to memory resources from the arbitrator and blocks to wait when resources are insufficient. The arbitrator processes subscription requests according to the [resource group priority](/information-schema/information-schema-resource-groups.md) (`LOW | MEDIUM | HIGH`) to which the SQL belongs. + +- When parsing SQL or compiling the execution plan, the required memory quota is estimated by TiDB and is proportional to the number of SQL keywords. Unlike `standard` mode, in `priority` mode, unless there is an `OOM` risk, a failed subscription does not immediately terminate the SQL. +- The arbitrator processes subscription requests in descending order of priority. Requests with the same priority are queued in the order they are initiated. +- When global memory resources are insufficient: + - The arbitrator terminates lower-priority SQL in order (from lower to higher priority, and from larger to smaller memory quota usage) to reclaim resources for higher-priority SQL. The `reason` field in the returned error is `CANCEL(out-of-quota & priority-mode)`. + - If there is no terminable SQL, the subscription request continues to wait until an existing SQL finishes execution and releases resources. + +If you want SQL to avoid the latency overhead caused by waiting for memory resources, you can set the [`tidb_mem_arbitrator_wait_averse`](/system-variables.md#tidb_mem_arbitrator_wait_averse-introduced-in-v900) system variable to `1`. + +- The subscription requests of the related SQL are automatically bound to priority `HIGH`. +- When global memory resources are insufficient, the arbitrator directly terminates the related SQL, and the `reason` field in the returned error is `CANCEL(out-of-quota & wait-averse)`. + +### Memory risk control + +When the memory usage of a TiDB instance reaches the `95%` threshold of `tidb_server_memory_limit`, the arbitrator starts handling memory risks. If the actual memory usage cannot be reduced to a safe level in the short term or the actual memory release rate is too low, the TiDB instance faces an `OOM` risk. The arbitrator forcibly terminates SQL in order (from lower to higher priority, and from larger to smaller memory quota usage), and the `reason` field in the returned error is `KILL(out-of-memory)`. + +If you need to force SQL to run when memory resources are insufficient, you can set the [`tidb_mem_arbitrator_wait_averse`](/system-variables.md#tidb_mem_arbitrator_wait_averse-introduced-in-v900) system variable to `nolimit`. This variable makes the memory usage of the related SQL unrestricted by the arbitrator, but it might cause the TiDB instance to `OOM`. + +### Manually ensuring memory safety + +You can set the upper limit of the arbitrator's memory quota in a TiDB instance using the [`tidb_mem_arbitrator_soft_limit`](/system-variables.md#tidb_mem_arbitrator_soft_limit-introduced-in-v900) system variable or the `instance.tidb_mem_arbitrator_soft_limit` parameter in the TiDB configuration file. The smaller the upper limit, the safer the global memory, but the lower the memory resource utilization. This variable can be used to manually and quickly converge memory risks. + +TiDB internally caches the historical maximum memory usage of some SQL statements and pre-subscribes sufficient memory quota before the next execution of the SQL. If it is known that a SQL statement has a problem of a large amount of uncontrolled memory usage, you can use the [`tidb_mem_arbitrator_query_reserved`](/system-variables.md#tidb_mem_arbitrator_query_reserved-introduced-in-v900) system variable to specify the quota subscribed by the SQL. The larger the value, the safer the global memory, but the lower the memory resource utilization. Pre-subscribing sufficient or excess quota can effectively ensure the isolation of memory resources for SQL. + +### Monitoring and observability metrics + +The `TiDB / Memory Arbitrator` panel is added to `Grafana` monitoring, containing the following metrics: + +- `Work Mode`: the memory management mode of the TiDB instance +- `Arbitration Exec`: statistics on the arbitrator processing various requests +- `Events`: statistics on various events in arbitrator mode. Pay special attention to `mem-risk` (memory risk) and `oom-risk` (`OOM` risk) +- `Mem Quota Stats`: usage of various memory quotas. Pay special attention to `out-of-control`, `wait-alloc` (total memory quota waiting for allocation), and `mem-inuse` (actual memory usage) +- `Mem Quota Arbitration`: processing time of memory resource subscription requests +- `Mem Pool Stats`: the number of various memory pools +- `Runtime Mem Pressure`: memory pressure value, that is, the ratio of actual memory usage to allocated memory quota +- `Waiting Tasks`: the number of various tasks queued and waiting for memory allocation + +[`SLOW_QUERY`](/information-schema/information-schema-slow-query.md) adds the `Mem_arbitration` field, which indicates the total time that SQL waits for memory resources. The `Mem Arbitration` column on the [TiDB Dashboard slow query page](/dashboard/dashboard-slow-query.md) also displays this information. + +[`PROCESSLIST`](/information-schema/information-schema-processlist.md) adds the following fields: + +- `MEM_ARBITRATION`: the total time that SQL has waited for memory resources so far +- `MEM_WAIT_ARBITRATE_START`: the start time of the current memory resource subscription request. If there is currently no waiting subscription request, the value is NULL +- `MEM_WAIT_ARBITRATE_BYTES`: the requested bytes of the current memory resource subscription request. If there is currently no waiting subscription request, the value is NULL + +[`Expensive query`](/identify-expensive-queries.md) adds the `mem_arbitration` field to record the time that SQL waits for memory resources and the information of the current subscription request, for example: + +- `cost_time 2.1s, wait_start 1970-01-02 10:17:36.789 UTC, wait_bytes 123456789123 Bytes (115.0 GB)` + +[`STATEMENTS_SUMMARY`](/statement-summary-tables.md): the `statements_summary`, `statements_summary_history`, `cluster_statements_summary`, and `cluster_statements_summary_history` tables add the following fields. The `Mean Mem Arbitration` column on the [TiDB Dashboard SQL statement analysis execution details page](/dashboard/dashboard-statement-list.md) also displays this information: + +- `AVG_MEM_ARBITRATION`: the average time that SQL waits for memory resources +- `MAX_MEM_ARBITRATION`: the maximum time that SQL waits for memory resources + +### Best practices + +The default value `80%` of the `tidb_server_memory_limit` system variable is relatively small. After enabling memory arbitrator mode, it is recommended to set it to `95%`. + +In single-node TiDB deployment scenarios: + +- If `priority` mode is enabled, it is recommended to bind OLTP-related SQL or business-critical SQL to high-priority resource groups, and bind other SQL to medium-priority or low-priority resource groups as needed +- If `standard` mode is enabled and SQL returns error `8180`, it is recommended to retry the SQL after waiting for a period of time + +In multi-node TiDB deployment scenarios: + +- If `standard` mode is enabled and SQL returns error `8180`, it is recommended to retry the SQL on another TiDB node + +- If `priority` mode is enabled: + - It is recommended to bind OLTP-related SQL or business-critical SQL to high-priority resource groups, and bind other SQL to medium-priority or low-priority resource groups as needed + - Use `max_execution_time` to limit the maximum execution time of SQL + - If a timeout or error `8180` occurs, it is recommended to retry the SQL on another TiDB node + - If you want SQL to fail quickly when memory resources are insufficient and then be retried on another node, you can set `tidb_mem_arbitrator_wait_averse` + +- TiDB instance grouping + - In each group, use the `instance.tidb_mem_arbitrator_mode` parameter in the configuration file to set the memory management mode of TiDB instances + - In each group, use the `instance.tidb_mem_arbitrator_soft_limit` parameter in the configuration file to set the upper limit of the memory quota of TiDB instances as needed + - Distribute SQL to different instance groups according to business requirements, and handle SQL failures or retries based on the arbitrator mode of each instance group + +You can take the following measures to ensure the execution of important SQL: + +- Use `tidb_mem_arbitrator_query_reserved` to pre-subscribe memory quota for important SQL, enhancing its memory resource isolation. +- Bind important SQL to high-priority resource groups. +- Set `tidb_mem_quota_query` to a larger value to reduce the probability that a single SQL is interrupted because it exceeds the memory quota. +- If you can accept the OOM risk, set `tidb_mem_arbitrator_wait_averse` to `nolimit` so that the related SQL bypasses the memory arbitrator limitation. diff --git a/system-variables.md b/system-variables.md index 3e84c5bff2236..1441b7bb54584 100644 --- a/system-variables.md +++ b/system-variables.md @@ -4008,6 +4008,70 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - Unit: Threads - This variable is used to set the maximum concurrency for TiFlash to execute a request. The default value is `-1`, indicating that this system variable is invalid and the maximum concurrency depends on the setting of the TiFlash configuration `profiles.default.max_threads`. When the value is `0`, the maximum number of threads is automatically configured by TiFlash. +### `tidb_mem_arbitrator_mode` New in v9.0.0 + +> **Warning:** +> +> The feature controlled by this variable is experimental and is not recommended for use in the production environment. This feature might be changed or removed without prior notice. If you encounter any bugs, report an [issue](https://github.com/pingcap/tidb/issues) on GitHub. + +- Scope: GLOBAL +- Persists to cluster: Yes +- Is controlled by the Hint [SET_VAR](/optimizer-hints.md#set_varvar_namevar_value): No +- Type: Enum +- Default value: `disable` +- Possible values: `disable`, `standard`, `priority` +- This variable sets the memory management mode of a TiDB instance. For details, see [TiDB memory control](/configure-memory-usage.md#memory-arbitration-mode). The following values are supported: + - `disable` (default): Disables memory arbitration mode and keeps the mechanism of [use first and report later](/system-variables.md#tidb_server_memory_limit-new-in-v640). + - `standard`: Enables standard memory arbitration mode. When SQL needs to use memory resources, it first subscribes to the memory arbitrator and allocates memory resources only after the subscription succeeds. If the subscription fails, the SQL execution is terminated. + - `priority`: Enables priority-based memory arbitration mode. When SQL needs to use memory resources, it first subscribes to the memory arbitrator and allocates memory resources only after the subscription succeeds. TiDB handles memory resource subscription requests according to the SQL [resource group priority](/information-schema/information-schema-resource-groups.md). + +### `tidb_mem_arbitrator_query_reserved` New in v9.0.0 + +> **Warning:** +> +> The feature controlled by this variable is experimental and is not recommended for use in the production environment. This feature might be changed or removed without prior notice. If you encounter any bugs, report an [issue](https://github.com/pingcap/tidb/issues) on GitHub. + +- Scope: SESSION +- Is controlled by the Hint [SET_VAR](/optimizer-hints.md#set_varvar_namevar_value): Yes +- Type: Integer +- Default value: `0` +- Unit: Bytes +- Range: `[0, 9223372036854775807]` +- When [memory arbitration mode](/configure-memory-usage.md#memory-arbitration-mode) is enabled, this variable controls the amount of memory resources that SQL pre-subscribes to from the memory arbitrator before execution. For details, see [TiDB memory control](/configure-memory-usage.md#manually-ensuring-memory-safety). + +### `tidb_mem_arbitrator_soft_limit` New in v9.0.0 + +> **Warning:** +> +> The feature controlled by this variable is experimental and is not recommended for use in the production environment. This feature might be changed or removed without prior notice. If you encounter any bugs, report an [issue](https://github.com/pingcap/tidb/issues) on GitHub. + +- Scope: GLOBAL +- Persists to cluster: Yes +- Is controlled by the Hint [SET_VAR](/optimizer-hints.md#set_varvar_namevar_value): No +- Type: String +- Default value: `0` +- Possible values: `0`, floating-point number `(0, 1]`, integer `(1, 9223372036854775807]` +- When [memory arbitration mode](/configure-memory-usage.md#memory-arbitration-mode) is enabled, this variable controls the upper limit of memory resources that the arbitrator can allocate in a TiDB instance. For details, see [TiDB memory control](/configure-memory-usage.md#manually-ensuring-memory-safety). + - `0`: The default upper limit of memory resources is `95%` of the value of [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640) + - Floating-point number `(0, 1]`: Specifies the upper limit of memory resources as a ratio relative to [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640). For example, `0.8` means the upper limit of memory resources is `tidb_server_memory_limit * 0.8`. + - Integer `(1, 9223372036854775807]`: Specifies the number of bytes + +### `tidb_mem_arbitrator_wait_averse` New in v9.0.0 + +> **Warning:** +> +> The feature controlled by this variable is experimental and is not recommended for use in the production environment. This feature might be changed or removed without prior notice. If you encounter any bugs, report an [issue](https://github.com/pingcap/tidb/issues) on GitHub. + +- Scope: SESSION +- Is controlled by the Hint [SET_VAR](/optimizer-hints.md#set_varvar_namevar_value): No +- Type: Enum +- Default value: `0` +- Possible values: `0`, `1`, `nolimit` +- When [memory arbitration mode](/configure-memory-usage.md#memory-arbitration-mode) is enabled, this variable controls the behavior of SQL when waiting for memory resources. For details, see [TiDB memory control](/configure-memory-usage.md#memory-arbitration-mode). + - `0` (default): Disables this feature, and the variable does not take effect + - `1`: Takes effect only in `priority` mode. SQL is automatically bound to high priority when subscribing to memory resources. When global memory resources are insufficient, the SQL execution is terminated instead of being blocked waiting. + - `nolimit`: The memory usage of SQL is not limited by the arbitrator. This value might increase the risk of OOM on the TiDB instance. + ### tidb_mem_oom_action New in v6.1.0 - Scope: GLOBAL From a2f13bb323eabc6bda510ecbfab01c34f680ea4c Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Thu, 14 May 2026 10:34:38 +0800 Subject: [PATCH 3/5] Apply suggestions from code review --- configure-memory-usage.md | 48 +++++++++++++++++++-------------------- system-variables.md | 8 +++---- 2 files changed, 28 insertions(+), 28 deletions(-) diff --git a/configure-memory-usage.md b/configure-memory-usage.md index 2478288811525..457b44fa1cdbf 100644 --- a/configure-memory-usage.md +++ b/configure-memory-usage.md @@ -228,11 +228,11 @@ To verify the performance of `GOMEMLIMIT`, a test is performed to compare the sp ## Memory arbitrator mode -In versions earlier than TiDB v9.0.0, the [memory control mechanism](#how-to-configure-the-memory-usage-threshold-of-a-tidb-server-instance) had the following issues: +Before TiDB v9.0.0, the [memory control mechanism](#how-to-configure-the-memory-usage-threshold-of-a-tidb-server-instance) has the following issues: -- When the memory usage of a TiDB instance exceeded the limit, TiDB might randomly terminate running SQL statements. -- Memory resources were used first and then reported, and memory usage information was isolated between different SQL statements, making it difficult for TiDB to uniformly schedule and control memory resources at the instance level. -- Under high memory pressure, the overhead of Go garbage collection (Garbage Collection, GC) increased significantly, and in severe cases could lead to out of memory (OOM) issues. +- When the memory usage of a TiDB instance exceeds the limit, TiDB might randomly terminate running SQL statements. +- Memory resources follow a “use-then-report” mechanism, and memory usage is isolated across different SQL statements. As a result, TiDB cannot centrally schedule or control memory resources at the instance level. +- Under high memory pressure, the overhead of Go garbage collection (Garbage Collection, GC) increases significantly, and in severe cases might cause out of memory (OOM) issues. Starting from v9.0.0, TiDB introduces memory arbitrator mode. This mode introduces a global memory arbitrator in each TiDB instance to centrally manage and schedule the memory resources of the instance from top to bottom, mitigating the preceding issues. @@ -245,18 +245,18 @@ You can enable memory arbitrator mode using the [`tidb_mem_arbitrator_mode`](/sy - `disable`: disables memory arbitrator mode - `standard` or `priority`: enables memory arbitrator mode. After it is enabled: - - Memory resources are used with a subscribe-before-allocation mechanism and are uniformly scheduled by the memory arbitrator in each TiDB instance - - The overall memory usage of the TiDB instance is expected not to exceed the limit of [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-introduced-in-v640), and the [alarm for high memory usage](#alarm-for-high-memory-usage-of-tidb-server) no longer takes effect + - Memory resources are used with a subscribe-before-allocation mechanism and are uniformly scheduled by the memory arbitrator in each TiDB instance. + - The overall memory usage of the TiDB instance is expected not to exceed the limit of [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640), and the [alarm for high memory usage](#alarm-for-high-memory-usage-of-tidb-server) no longer takes effect. - The control behavior of the following system variables remains effective: - - [`tidb_mem_oom_action`](/system-variables.md#tidb_mem_oom_action-introduced-in-v610) + - [`tidb_mem_oom_action`](/system-variables.md#tidb_mem_oom_action-new-in-v610) - [`tidb_mem_quota_query`](/system-variables.md#tidb_mem_quota_query) - [`max_execution_time`](/system-variables.md#max_execution_time) When memory resources are insufficient or there is an OOM risk, the arbitrator can reclaim memory resources by terminating SQL, but it does not terminate `DDL`, `DCL`, or `TCL` SQL types. Terminated SQL returns error code `8180` to the client. The error format is: `Query execution was stopped by the global memory arbitrator [reason=?, path=?] [conn=?]`. The related fields are explained as follows: -- `conn`: connection (session) ID -- `reason`: the specific reason why the SQL was terminated -- `path`: the stage at which the SQL was terminated (if the error does not contain the `path` field, the SQL was terminated during the execution stage) +- `conn`: the connection (session) ID +- `reason`: the specific reason why the SQL is terminated +- `path`: the stage at which the SQL is terminated (if the error does not contain the `path` field, the SQL is terminated during the execution stage) - `ParseSQL`: parsing - `CompilePlan`: compiling the execution plan @@ -287,25 +287,25 @@ In `priority` mode, during SQL execution, SQL dynamically subscribes to memory r - When parsing SQL or compiling the execution plan, the required memory quota is estimated by TiDB and is proportional to the number of SQL keywords. Unlike `standard` mode, in `priority` mode, unless there is an `OOM` risk, a failed subscription does not immediately terminate the SQL. - The arbitrator processes subscription requests in descending order of priority. Requests with the same priority are queued in the order they are initiated. - When global memory resources are insufficient: - - The arbitrator terminates lower-priority SQL in order (from lower to higher priority, and from larger to smaller memory quota usage) to reclaim resources for higher-priority SQL. The `reason` field in the returned error is `CANCEL(out-of-quota & priority-mode)`. + - The arbitrator terminates lower-priority SQL in order (from low to high priority, and from large to small memory quota usage) to reclaim resources for higher-priority SQL. The `reason` field in the returned error is `CANCEL(out-of-quota & priority-mode)`. - If there is no terminable SQL, the subscription request continues to wait until an existing SQL finishes execution and releases resources. -If you want SQL to avoid the latency overhead caused by waiting for memory resources, you can set the [`tidb_mem_arbitrator_wait_averse`](/system-variables.md#tidb_mem_arbitrator_wait_averse-introduced-in-v900) system variable to `1`. +If you want SQL to avoid the latency overhead caused by waiting for memory resources, you can set the [`tidb_mem_arbitrator_wait_averse`](/system-variables.md#tidb_mem_arbitrator_wait_averse-new-in-v900) system variable to `1`. - The subscription requests of the related SQL are automatically bound to priority `HIGH`. - When global memory resources are insufficient, the arbitrator directly terminates the related SQL, and the `reason` field in the returned error is `CANCEL(out-of-quota & wait-averse)`. ### Memory risk control -When the memory usage of a TiDB instance reaches the `95%` threshold of `tidb_server_memory_limit`, the arbitrator starts handling memory risks. If the actual memory usage cannot be reduced to a safe level in the short term or the actual memory release rate is too low, the TiDB instance faces an `OOM` risk. The arbitrator forcibly terminates SQL in order (from lower to higher priority, and from larger to smaller memory quota usage), and the `reason` field in the returned error is `KILL(out-of-memory)`. +When the memory usage of a TiDB instance reaches the `95%` threshold of `tidb_server_memory_limit`, the arbitrator starts handling memory risks. If the actual memory usage cannot be reduced to a safe level in the short term or the actual memory release rate is too low, the TiDB instance faces the `OOM` risk. The arbitrator forcibly terminates SQL in order (from low to high priority, and from large to small memory quota usage), and the `reason` field in the returned error is `KILL(out-of-memory)`. -If you need to force SQL to run when memory resources are insufficient, you can set the [`tidb_mem_arbitrator_wait_averse`](/system-variables.md#tidb_mem_arbitrator_wait_averse-introduced-in-v900) system variable to `nolimit`. This variable makes the memory usage of the related SQL unrestricted by the arbitrator, but it might cause the TiDB instance to `OOM`. +If you need to force SQL to run when memory resources are insufficient, you can set the [`tidb_mem_arbitrator_wait_averse`](/system-variables.md#tidb_mem_arbitrator_wait_averse-new-in-v900) system variable to `nolimit`. This variable makes the memory usage of the related SQL unrestricted by the arbitrator, but it might cause the TiDB instance to `OOM`. ### Manually ensuring memory safety -You can set the upper limit of the arbitrator's memory quota in a TiDB instance using the [`tidb_mem_arbitrator_soft_limit`](/system-variables.md#tidb_mem_arbitrator_soft_limit-introduced-in-v900) system variable or the `instance.tidb_mem_arbitrator_soft_limit` parameter in the TiDB configuration file. The smaller the upper limit, the safer the global memory, but the lower the memory resource utilization. This variable can be used to manually and quickly converge memory risks. +You can set the upper limit of the arbitrator's memory quota in a TiDB instance using the [`tidb_mem_arbitrator_soft_limit`](/system-variables.md#tidb_mem_arbitrator_soft_limit-new-in-v900) system variable or the `instance.tidb_mem_arbitrator_soft_limit` parameter in the TiDB configuration file. The smaller the upper limit, the safer the global memory, but the lower the memory resource utilization. You can use this variable to manually and quickly converge memory risks. -TiDB internally caches the historical maximum memory usage of some SQL statements and pre-subscribes sufficient memory quota before the next execution of the SQL. If it is known that a SQL statement has a problem of a large amount of uncontrolled memory usage, you can use the [`tidb_mem_arbitrator_query_reserved`](/system-variables.md#tidb_mem_arbitrator_query_reserved-introduced-in-v900) system variable to specify the quota subscribed by the SQL. The larger the value, the safer the global memory, but the lower the memory resource utilization. Pre-subscribing sufficient or excess quota can effectively ensure the isolation of memory resources for SQL. +TiDB internally caches the historical maximum memory usage of some SQL statements and pre-subscribes sufficient memory quota before the next execution of the SQL. If it is known that a SQL statement has a large amount of uncontrolled memory usage, you can use the [`tidb_mem_arbitrator_query_reserved`](/system-variables.md#tidb_mem_arbitrator_query_reserved-new-in-v900) system variable to specify the quota subscribed by the SQL. The larger the value, the safer the global memory, but the lower the memory resource utilization. Pre-subscribing sufficient or excess quota can effectively ensure the isolation of memory resources for SQL. ### Monitoring and observability metrics @@ -348,18 +348,18 @@ In single-node TiDB deployment scenarios: In multi-node TiDB deployment scenarios: -- If `standard` mode is enabled and SQL returns error `8180`, it is recommended to retry the SQL on another TiDB node +- If `standard` mode is enabled and SQL returns error `8180`, it is recommended to retry the SQL on another TiDB node. - If `priority` mode is enabled: - - It is recommended to bind OLTP-related SQL or business-critical SQL to high-priority resource groups, and bind other SQL to medium-priority or low-priority resource groups as needed - - Use `max_execution_time` to limit the maximum execution time of SQL - - If a timeout or error `8180` occurs, it is recommended to retry the SQL on another TiDB node - - If you want SQL to fail quickly when memory resources are insufficient and then be retried on another node, you can set `tidb_mem_arbitrator_wait_averse` + - It is recommended to bind OLTP-related SQL or business-critical SQL to high-priority resource groups, and bind other SQL to medium-priority or low-priority resource groups as needed. + - Use `max_execution_time` to limit the maximum execution time of SQL. + - If a timeout or error `8180` occurs, it is recommended to retry the SQL on another TiDB node. + - If you want SQL to fail quickly when memory resources are insufficient and then be retried on another node, you can set `tidb_mem_arbitrator_wait_averse`. - TiDB instance grouping - - In each group, use the `instance.tidb_mem_arbitrator_mode` parameter in the configuration file to set the memory management mode of TiDB instances - - In each group, use the `instance.tidb_mem_arbitrator_soft_limit` parameter in the configuration file to set the upper limit of the memory quota of TiDB instances as needed - - Distribute SQL to different instance groups according to business requirements, and handle SQL failures or retries based on the arbitrator mode of each instance group + - In each group, use the `instance.tidb_mem_arbitrator_mode` parameter in the configuration file to set the memory management mode of TiDB instances. + - In each group, use the `instance.tidb_mem_arbitrator_soft_limit` parameter in the configuration file to set the upper limit of the memory quota of TiDB instances as needed. + - Distribute SQL to different instance groups according to business requirements, and handle SQL failures or retries based on the arbitrator mode of each instance group. You can take the following measures to ensure the execution of important SQL: diff --git a/system-variables.md b/system-variables.md index 1441b7bb54584..49fbaaf241678 100644 --- a/system-variables.md +++ b/system-variables.md @@ -4020,7 +4020,7 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - Type: Enum - Default value: `disable` - Possible values: `disable`, `standard`, `priority` -- This variable sets the memory management mode of a TiDB instance. For details, see [TiDB memory control](/configure-memory-usage.md#memory-arbitration-mode). The following values are supported: +- This variable sets the memory management mode of a TiDB instance. For more information, see [TiDB memory control](/configure-memory-usage.md#memory-arbitration-mode). The following values are supported: - `disable` (default): Disables memory arbitration mode and keeps the mechanism of [use first and report later](/system-variables.md#tidb_server_memory_limit-new-in-v640). - `standard`: Enables standard memory arbitration mode. When SQL needs to use memory resources, it first subscribes to the memory arbitrator and allocates memory resources only after the subscription succeeds. If the subscription fails, the SQL execution is terminated. - `priority`: Enables priority-based memory arbitration mode. When SQL needs to use memory resources, it first subscribes to the memory arbitrator and allocates memory resources only after the subscription succeeds. TiDB handles memory resource subscription requests according to the SQL [resource group priority](/information-schema/information-schema-resource-groups.md). @@ -4037,7 +4037,7 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - Default value: `0` - Unit: Bytes - Range: `[0, 9223372036854775807]` -- When [memory arbitration mode](/configure-memory-usage.md#memory-arbitration-mode) is enabled, this variable controls the amount of memory resources that SQL pre-subscribes to from the memory arbitrator before execution. For details, see [TiDB memory control](/configure-memory-usage.md#manually-ensuring-memory-safety). +- When [memory arbitration mode](/configure-memory-usage.md#memory-arbitration-mode) is enabled, this variable controls the amount of memory resources that SQL pre-subscribes to from the memory arbitrator before execution. For more information, see [TiDB memory control](/configure-memory-usage.md#manually-ensuring-memory-safety). ### `tidb_mem_arbitrator_soft_limit` New in v9.0.0 @@ -4051,7 +4051,7 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - Type: String - Default value: `0` - Possible values: `0`, floating-point number `(0, 1]`, integer `(1, 9223372036854775807]` -- When [memory arbitration mode](/configure-memory-usage.md#memory-arbitration-mode) is enabled, this variable controls the upper limit of memory resources that the arbitrator can allocate in a TiDB instance. For details, see [TiDB memory control](/configure-memory-usage.md#manually-ensuring-memory-safety). +- When [memory arbitration mode](/configure-memory-usage.md#memory-arbitration-mode) is enabled, this variable controls the upper limit of memory resources that the arbitrator can allocate in a TiDB instance. For more information, see [TiDB memory control](/configure-memory-usage.md#manually-ensuring-memory-safety). - `0`: The default upper limit of memory resources is `95%` of the value of [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640) - Floating-point number `(0, 1]`: Specifies the upper limit of memory resources as a ratio relative to [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640). For example, `0.8` means the upper limit of memory resources is `tidb_server_memory_limit * 0.8`. - Integer `(1, 9223372036854775807]`: Specifies the number of bytes @@ -4067,7 +4067,7 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - Type: Enum - Default value: `0` - Possible values: `0`, `1`, `nolimit` -- When [memory arbitration mode](/configure-memory-usage.md#memory-arbitration-mode) is enabled, this variable controls the behavior of SQL when waiting for memory resources. For details, see [TiDB memory control](/configure-memory-usage.md#memory-arbitration-mode). +- When [memory arbitration mode](/configure-memory-usage.md#memory-arbitration-mode) is enabled, this variable controls the behavior of SQL when waiting for memory resources. For more information, see [TiDB memory control](/configure-memory-usage.md#memory-arbitration-mode). - `0` (default): Disables this feature, and the variable does not take effect - `1`: Takes effect only in `priority` mode. SQL is automatically bound to high priority when subscribing to memory resources. When global memory resources are insufficient, the SQL execution is terminated instead of being blocked waiting. - `nolimit`: The memory usage of SQL is not limited by the arbitrator. This value might increase the risk of OOM on the TiDB instance. From 7537d206e0128e2c73de86ad9124207ff95186d6 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Thu, 14 May 2026 10:36:12 +0800 Subject: [PATCH 4/5] Update configure-memory-usage.md --- configure-memory-usage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configure-memory-usage.md b/configure-memory-usage.md index 457b44fa1cdbf..d06254120e64e 100644 --- a/configure-memory-usage.md +++ b/configure-memory-usage.md @@ -247,7 +247,7 @@ You can enable memory arbitrator mode using the [`tidb_mem_arbitrator_mode`](/sy - `standard` or `priority`: enables memory arbitrator mode. After it is enabled: - Memory resources are used with a subscribe-before-allocation mechanism and are uniformly scheduled by the memory arbitrator in each TiDB instance. - The overall memory usage of the TiDB instance is expected not to exceed the limit of [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640), and the [alarm for high memory usage](#alarm-for-high-memory-usage-of-tidb-server) no longer takes effect. - - The control behavior of the following system variables remains effective: + - The behavior of the following system variables remains effective: - [`tidb_mem_oom_action`](/system-variables.md#tidb_mem_oom_action-new-in-v610) - [`tidb_mem_quota_query`](/system-variables.md#tidb_mem_quota_query) - [`max_execution_time`](/system-variables.md#max_execution_time) From 9394643d0e6b996236bc390a70d1362df51db06d Mon Sep 17 00:00:00 2001 From: houfaxin Date: Thu, 14 May 2026 14:47:25 +0800 Subject: [PATCH 5/5] Fix memory arbitrator links and wording Update documentation references to use the new "memory arbitrator" naming and corrected anchor targets. Changed cross-doc links in configure-memory-usage.md and system-variables.md (e.g. tidb_mem_arbitrator_mode anchor, memory-arbitrator-mode anchor, and the high-memory alarm anchor) so they point to the renamed headings introduced for v9.0.0 and ensure consistent phrasing across the docs. --- configure-memory-usage.md | 6 +++--- system-variables.md | 14 +++++++------- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/configure-memory-usage.md b/configure-memory-usage.md index d06254120e64e..3d65653f34e5c 100644 --- a/configure-memory-usage.md +++ b/configure-memory-usage.md @@ -228,7 +228,7 @@ To verify the performance of `GOMEMLIMIT`, a test is performed to compare the sp ## Memory arbitrator mode -Before TiDB v9.0.0, the [memory control mechanism](#how-to-configure-the-memory-usage-threshold-of-a-tidb-server-instance) has the following issues: +Before TiDB v9.0.0, the [memory control mechanism](#configure-the-memory-usage-threshold-of-a-tidb-server-instance) has the following issues: - When the memory usage of a TiDB instance exceeds the limit, TiDB might randomly terminate running SQL statements. - Memory resources follow a “use-then-report” mechanism, and memory usage is isolated across different SQL statements. As a result, TiDB cannot centrally schedule or control memory resources at the instance level. @@ -240,13 +240,13 @@ Starting from v9.0.0, TiDB introduces memory arbitrator mode. This mode introduc > > This feature is experimental and is not recommended for use in the production environment. This feature might be changed or removed without prior notice. If you find a bug, report it by submitting an [issue](https://github.com/pingcap/tidb/issues) on GitHub. -You can enable memory arbitrator mode using the [`tidb_mem_arbitrator_mode`](/system-variables.md#tidb_mem_arbitrator_mode-introduced-in-v900) system variable or the `instance.tidb_mem_arbitrator_mode` parameter in the TiDB configuration file. +You can enable memory arbitrator mode using the [`tidb_mem_arbitrator_mode`](/system-variables.md#tidb_mem_arbitrator_mode-new-in-v900) system variable or the `instance.tidb_mem_arbitrator_mode` parameter in the TiDB configuration file. - `disable`: disables memory arbitrator mode - `standard` or `priority`: enables memory arbitrator mode. After it is enabled: - Memory resources are used with a subscribe-before-allocation mechanism and are uniformly scheduled by the memory arbitrator in each TiDB instance. - - The overall memory usage of the TiDB instance is expected not to exceed the limit of [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640), and the [alarm for high memory usage](#alarm-for-high-memory-usage-of-tidb-server) no longer takes effect. + - The overall memory usage of the TiDB instance is expected not to exceed the limit of [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640), and the [alarm for high memory usage](#trigger-the-alarm-of-excessive-memory-usage) no longer takes effect. - The behavior of the following system variables remains effective: - [`tidb_mem_oom_action`](/system-variables.md#tidb_mem_oom_action-new-in-v610) - [`tidb_mem_quota_query`](/system-variables.md#tidb_mem_quota_query) diff --git a/system-variables.md b/system-variables.md index 49fbaaf241678..21d1c0dbfb8b1 100644 --- a/system-variables.md +++ b/system-variables.md @@ -4020,10 +4020,10 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - Type: Enum - Default value: `disable` - Possible values: `disable`, `standard`, `priority` -- This variable sets the memory management mode of a TiDB instance. For more information, see [TiDB memory control](/configure-memory-usage.md#memory-arbitration-mode). The following values are supported: - - `disable` (default): Disables memory arbitration mode and keeps the mechanism of [use first and report later](/system-variables.md#tidb_server_memory_limit-new-in-v640). - - `standard`: Enables standard memory arbitration mode. When SQL needs to use memory resources, it first subscribes to the memory arbitrator and allocates memory resources only after the subscription succeeds. If the subscription fails, the SQL execution is terminated. - - `priority`: Enables priority-based memory arbitration mode. When SQL needs to use memory resources, it first subscribes to the memory arbitrator and allocates memory resources only after the subscription succeeds. TiDB handles memory resource subscription requests according to the SQL [resource group priority](/information-schema/information-schema-resource-groups.md). +- This variable sets the memory management mode of a TiDB instance. For more information, see [TiDB memory control](/configure-memory-usage.md#memory-arbitrator-mode). The following values are supported: + - `disable` (default): Disables memory arbitrator mode and keeps the mechanism of [use first and report later](/system-variables.md#tidb_server_memory_limit-new-in-v640). + - `standard`: Enables standard memory arbitrator mode. When SQL needs to use memory resources, it first subscribes to the memory arbitrator and allocates memory resources only after the subscription succeeds. If the subscription fails, the SQL execution is terminated. + - `priority`: Enables priority-based memory arbitrator mode. When SQL needs to use memory resources, it first subscribes to the memory arbitrator and allocates memory resources only after the subscription succeeds. TiDB handles memory resource subscription requests according to the SQL [resource group priority](/information-schema/information-schema-resource-groups.md). ### `tidb_mem_arbitrator_query_reserved` New in v9.0.0 @@ -4037,7 +4037,7 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - Default value: `0` - Unit: Bytes - Range: `[0, 9223372036854775807]` -- When [memory arbitration mode](/configure-memory-usage.md#memory-arbitration-mode) is enabled, this variable controls the amount of memory resources that SQL pre-subscribes to from the memory arbitrator before execution. For more information, see [TiDB memory control](/configure-memory-usage.md#manually-ensuring-memory-safety). +- When [memory arbitrator mode](/configure-memory-usage.md#memory-arbitrator-mode) is enabled, this variable controls the amount of memory resources that SQL pre-subscribes to from the memory arbitrator before execution. For more information, see [TiDB memory control](/configure-memory-usage.md#manually-ensuring-memory-safety). ### `tidb_mem_arbitrator_soft_limit` New in v9.0.0 @@ -4051,7 +4051,7 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - Type: String - Default value: `0` - Possible values: `0`, floating-point number `(0, 1]`, integer `(1, 9223372036854775807]` -- When [memory arbitration mode](/configure-memory-usage.md#memory-arbitration-mode) is enabled, this variable controls the upper limit of memory resources that the arbitrator can allocate in a TiDB instance. For more information, see [TiDB memory control](/configure-memory-usage.md#manually-ensuring-memory-safety). +- When [memory arbitrator mode](/configure-memory-usage.md#memory-arbitrator-mode) is enabled, this variable controls the upper limit of memory resources that the arbitrator can allocate in a TiDB instance. For more information, see [TiDB memory control](/configure-memory-usage.md#manually-ensuring-memory-safety). - `0`: The default upper limit of memory resources is `95%` of the value of [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640) - Floating-point number `(0, 1]`: Specifies the upper limit of memory resources as a ratio relative to [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640). For example, `0.8` means the upper limit of memory resources is `tidb_server_memory_limit * 0.8`. - Integer `(1, 9223372036854775807]`: Specifies the number of bytes @@ -4067,7 +4067,7 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - Type: Enum - Default value: `0` - Possible values: `0`, `1`, `nolimit` -- When [memory arbitration mode](/configure-memory-usage.md#memory-arbitration-mode) is enabled, this variable controls the behavior of SQL when waiting for memory resources. For more information, see [TiDB memory control](/configure-memory-usage.md#memory-arbitration-mode). +- When [memory arbitrator mode](/configure-memory-usage.md#memory-arbitrator-mode) is enabled, this variable controls the behavior of SQL when waiting for memory resources. For more information, see [TiDB memory control](/configure-memory-usage.md#memory-arbitrator-mode). - `0` (default): Disables this feature, and the variable does not take effect - `1`: Takes effect only in `priority` mode. SQL is automatically bound to high priority when subscribing to memory resources. When global memory resources are insufficient, the SQL execution is terminated instead of being blocked waiting. - `nolimit`: The memory usage of SQL is not limited by the arbitrator. This value might increase the risk of OOM on the TiDB instance.