From 6a2a4f45bc1d62b0444af56a72aff0d4764ad637 Mon Sep 17 00:00:00 2001 From: jmaeagle99 <44687433+jmaeagle99@users.noreply.github.com> Date: Wed, 26 Aug 2026 20:25:03 -0700 Subject: [PATCH 1/2] Document RequestTooLarge WFT failure reason --- docs/references/sdk-metrics.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/references/sdk-metrics.mdx b/docs/references/sdk-metrics.mdx index 5a18129bf2..813f48d913 100644 --- a/docs/references/sdk-metrics.mdx +++ b/docs/references/sdk-metrics.mdx @@ -542,6 +542,7 @@ Valid values for the `failure_reason` tag: - `NonDeterminismError`: The Workflow Task failed due to a non-determinism error. - `GrpcMessageTooLarge`: The Workflow Task response exceeded the gRPC message size limit and could not be delivered. The Temporal Service terminates the Workflow Execution in response. See [Troubleshoot the BlobSizeLimitError](/troubleshooting/blob-size-limit-error). - `PayloadsTooLarge`: A Payload or Memo in the Workflow Task response exceeded the Namespace size limit, so the Worker proactively failed the Workflow Task as retryable. See [Troubleshoot the BlobSizeLimitError](/troubleshooting/blob-size-limit-error#payload-size-limit). +- `RequestTooLarge`: The whole Workflow Task completion exceeded the Namespace limit on the size of a single request, so the Worker proactively failed the Workflow Task as non-retryable rather than send a request the Temporal Service would reject. - `WorkflowError`: The Workflow Task failed for any other reason. ### `workflow_task_execution_latency` From 2c9dae8fb3a39639dcb86a1119faa2a0589cf541 Mon Sep 17 00:00:00 2001 From: jmaeagle99 <44687433+jmaeagle99@users.noreply.github.com> Date: Wed, 26 Aug 2026 20:50:41 -0700 Subject: [PATCH 2/2] Update runbook and alert guidance --- docs/best-practices/worker-alerting.mdx | 1 + docs/references/sdk-metrics.mdx | 2 +- docs/troubleshooting/execution-failures.mdx | 28 ++++++++++++++++++++- 3 files changed, 29 insertions(+), 2 deletions(-) diff --git a/docs/best-practices/worker-alerting.mdx b/docs/best-practices/worker-alerting.mdx index bcc0dddc86..0aea4f565f 100644 --- a/docs/best-practices/worker-alerting.mdx +++ b/docs/best-practices/worker-alerting.mdx @@ -135,6 +135,7 @@ For triage, see [Execution failures](/troubleshooting/execution-failures). | --- | --- | --- | --- | --- | --- | | [Non-determinism error](/troubleshooting/execution-failures#non-determinism-error) | `workflow_task_execution_failed` | `failure_reason=NonDeterminismError` | Any occurrence | 1m | Critical | | [gRPC message too large](/troubleshooting/execution-failures#grpc-message-too-large) | `workflow_task_execution_failed` | `failure_reason=GrpcMessageTooLarge` | Any occurrence | 1m | Critical | +| [Request too large](/troubleshooting/execution-failures#request-too-large) | `workflow_task_execution_failed` | `failure_reason=RequestTooLarge` | Any occurrence | 1m | Critical | | [Workflow Task execution failures elevated](/troubleshooting/execution-failures#workflow-task-execution-failures-elevated) | `workflow_task_execution_failed` | `failure_reason=WorkflowError` | Rate above 10/s | 2m | Warning | | [Workflow Task execution latency high](/troubleshooting/execution-failures#workflow-task-execution-latency-high) | `workflow_task_execution_latency` | `task_queue`, `workflow_type` | p99 above 10s | 5m | Critical | | [Activity execution failures elevated](/troubleshooting/execution-failures#activity-execution-failures-elevated) | `activity_execution_failed` | `activity_type` | Rate above 10/s | 2m | Warning | diff --git a/docs/references/sdk-metrics.mdx b/docs/references/sdk-metrics.mdx index 813f48d913..70f8b9e677 100644 --- a/docs/references/sdk-metrics.mdx +++ b/docs/references/sdk-metrics.mdx @@ -542,7 +542,7 @@ Valid values for the `failure_reason` tag: - `NonDeterminismError`: The Workflow Task failed due to a non-determinism error. - `GrpcMessageTooLarge`: The Workflow Task response exceeded the gRPC message size limit and could not be delivered. The Temporal Service terminates the Workflow Execution in response. See [Troubleshoot the BlobSizeLimitError](/troubleshooting/blob-size-limit-error). - `PayloadsTooLarge`: A Payload or Memo in the Workflow Task response exceeded the Namespace size limit, so the Worker proactively failed the Workflow Task as retryable. See [Troubleshoot the BlobSizeLimitError](/troubleshooting/blob-size-limit-error#payload-size-limit). -- `RequestTooLarge`: The whole Workflow Task completion exceeded the Namespace limit on the size of a single request, so the Worker proactively failed the Workflow Task as non-retryable rather than send a request the Temporal Service would reject. +- `RequestTooLarge`: The whole Workflow Task completion exceeded the Namespace limit on the size of a single request, so the Worker proactively failed the Workflow Task rather than send a request the Temporal Service would reject. The Temporal Service retries the Workflow Task. See [Request too large](/troubleshooting/execution-failures#request-too-large). - `WorkflowError`: The Workflow Task failed for any other reason. ### `workflow_task_execution_latency` diff --git a/docs/troubleshooting/execution-failures.mdx b/docs/troubleshooting/execution-failures.mdx index e033d4b941..92f3d6b7f7 100644 --- a/docs/troubleshooting/execution-failures.mdx +++ b/docs/troubleshooting/execution-failures.mdx @@ -20,7 +20,7 @@ If you got here from a firing alert, the section you landed on is the runbook fo For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics). [`temporal_workflow_task_execution_failed`](/references/sdk-metrics#workflow_task_execution_failed) carries a `failure_reason` tag along with `namespace`, `task_queue`, and `workflow_type`. -The reason decides what the Temporal Service does next, and the difference is worth knowing: two of the three retry forever, one terminates the Execution on the spot. +The reason decides what the Temporal Service does next, and the difference is worth knowing: three of the four retry forever, one terminates the Execution on the spot. Alert on each `failure_reason` separately. ## Non-determinism error {/* #non-determinism-error */} @@ -81,6 +81,32 @@ Check the Workflow terminate rate on your server dashboard. A spike alongside th ::: +## Request too large {/* #request-too-large */} + +**Recommended alert:** `temporal_workflow_task_execution_failed` with `failure_reason=RequestTooLarge` +Fires on any occurrence, held for 1m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#execution-failures). + +The whole Workflow Task completion was bigger than the Namespace limit on a single `RespondWorkflowTaskCompleted` request. +This is not one oversized Payload, it is the total: every command, Payload, message, and piece of metadata the Worker produced for that one Workflow Task, added together. + +Workers that support command pagination split a large completion across several requests, but the recombined size still has to fit the limit. +When it does not, the Worker fails the Workflow Task itself with cause `WORKFLOW_TASK_FAILED_CAUSE_REQUEST_TOO_LARGE` rather than sending pages the Temporal Service is going to reject. + +**Why it matters.** +The Temporal Service retries the Workflow Task, so unlike [gRPC message too large](#grpc-message-too-large) the Execution is not terminated. +Replay produces the same oversized completion every time, so affected Executions stop making progress and stay that way until the cause is fixed. +The retries consume Workflow Worker capacity and can start affecting healthy Executions on the same Task Queue. + +**Triage.** + +1. **Identify the affected Executions.** This metric does not carry a Workflow Id. Worker logs record the Workflow Id and Run Id. In the Temporal UI you can also find affected Executions by querying the `TemporalReportedProblems` Search Attribute, which the Temporal Service sets on Executions experiencing repeated Workflow Task failures. +1. **Find what made the completion large.** The limit applies to the sum, so look for what the Workflow produced in a single step: + - **A large fan-out.** Scheduling many Activities, Child Workflows, or Timers in one Workflow Task. Break the fan-out into smaller batches across multiple Workflow Tasks. + - **Accumulated Signals or Updates.** A large number buffered into one Workflow Task. Rate-limit senders or batch them. + - **Large Payloads in aggregate.** Individually under the Payload limit, but large once combined. Move them out of band with [External Storage](/external-storage). +1. **Check the Namespace limit.** The limit is a Namespace setting on the size of a single Workflow Task completion, separate from the Payload and gRPC message limits. On Temporal Cloud, contact support if the workload legitimately needs a higher limit. Self-hosted operators can adjust it in dynamic config. +1. **Fix and redeploy.** Affected Executions resume on their next Workflow Task retry once the corrected Worker is running. They do not need to be restarted, because they were never terminated. + ## Workflow Task execution failures elevated {/* #workflow-task-execution-failures-elevated */} **Recommended alert:** `temporal_workflow_task_execution_failed` with `failure_reason=WorkflowError`