Skip to content

Document RequestTooLarge WFT failure reason #5191

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
jmaeagle99 wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Document RequestTooLarge WFT failure reason #5191

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6a2a4f4
Document RequestTooLarge WFT failure reason
jmaeagle99 Aug 27, 2026
2c9dae8
Update runbook and alert guidance
jmaeagle99 Aug 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/best-practices/worker-alerting.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ For triage, see [Execution failures](/troubleshooting/execution-failures).
| --- | --- | --- | --- | --- | --- |
| [Non-determinism error](/troubleshooting/execution-failures#non-determinism-error) | `workflow_task_execution_failed` | `failure_reason=NonDeterminismError` | Any occurrence | 1m | Critical |
| [gRPC message too large](/troubleshooting/execution-failures#grpc-message-too-large) | `workflow_task_execution_failed` | `failure_reason=GrpcMessageTooLarge` | Any occurrence | 1m | Critical |
| [Request too large](/troubleshooting/execution-failures#request-too-large) | `workflow_task_execution_failed` | `failure_reason=RequestTooLarge` | Any occurrence | 1m | Critical |
| [Workflow Task execution failures elevated](/troubleshooting/execution-failures#workflow-task-execution-failures-elevated) | `workflow_task_execution_failed` | `failure_reason=WorkflowError` | Rate above 10/s | 2m | Warning |
| [Workflow Task execution latency high](/troubleshooting/execution-failures#workflow-task-execution-latency-high) | `workflow_task_execution_latency` | `task_queue`, `workflow_type` | p99 above 10s | 5m | Critical |
| [Activity execution failures elevated](/troubleshooting/execution-failures#activity-execution-failures-elevated) | `activity_execution_failed` | `activity_type` | Rate above 10/s | 2m | Warning |
Expand Down
1 change: 1 addition & 0 deletions docs/references/sdk-metrics.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -542,6 +542,7 @@ Valid values for the `failure_reason` tag:
- `NonDeterminismError`: The Workflow Task failed due to a non-determinism error.
- `GrpcMessageTooLarge`: The Workflow Task response exceeded the gRPC message size limit and could not be delivered. The Temporal Service terminates the Workflow Execution in response. See [Troubleshoot the BlobSizeLimitError](/troubleshooting/blob-size-limit-error).
- `PayloadsTooLarge`: A Payload or Memo in the Workflow Task response exceeded the Namespace size limit, so the Worker proactively failed the Workflow Task as retryable. See [Troubleshoot the BlobSizeLimitError](/troubleshooting/blob-size-limit-error#payload-size-limit).
- `RequestTooLarge`: The whole Workflow Task completion exceeded the Namespace limit on the size of a single request, so the Worker proactively failed the Workflow Task rather than send a request the Temporal Service would reject. The Temporal Service retries the Workflow Task. See [Request too large](/troubleshooting/execution-failures#request-too-large).
- `WorkflowError`: The Workflow Task failed for any other reason.

### `workflow_task_execution_latency`
Expand Down
28 changes: 27 additions & 1 deletion docs/troubleshooting/execution-failures.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ If you got here from a firing alert, the section you landed on is the runbook fo
For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics).

[`temporal_workflow_task_execution_failed`](/references/sdk-metrics#workflow_task_execution_failed) carries a `failure_reason` tag along with `namespace`, `task_queue`, and `workflow_type`.
The reason decides what the Temporal Service does next, and the difference is worth knowing: two of the three retry forever, one terminates the Execution on the spot.
The reason decides what the Temporal Service does next, and the difference is worth knowing: three of the four retry forever, one terminates the Execution on the spot.
Alert on each `failure_reason` separately.

## Non-determinism error {/* #non-determinism-error */}
Expand Down Expand Up @@ -81,6 +81,32 @@ Check the Workflow terminate rate on your server dashboard. A spike alongside th

:::

## Request too large {/* #request-too-large */}

**Recommended alert:** `temporal_workflow_task_execution_failed` with `failure_reason=RequestTooLarge`
Fires on any occurrence, held for 1m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#execution-failures).

The whole Workflow Task completion was bigger than the Namespace limit on a single `RespondWorkflowTaskCompleted` request.
This is not one oversized Payload, it is the total: every command, Payload, message, and piece of metadata the Worker produced for that one Workflow Task, added together.

Workers that support command pagination split a large completion across several requests, but the recombined size still has to fit the limit.
When it does not, the Worker fails the Workflow Task itself with cause `WORKFLOW_TASK_FAILED_CAUSE_REQUEST_TOO_LARGE` rather than sending pages the Temporal Service is going to reject.

**Why it matters.**
The Temporal Service retries the Workflow Task, so unlike [gRPC message too large](#grpc-message-too-large) the Execution is not terminated.
Replay produces the same oversized completion every time, so affected Executions stop making progress and stay that way until the cause is fixed.
The retries consume Workflow Worker capacity and can start affecting healthy Executions on the same Task Queue.

**Triage.**

1. **Identify the affected Executions.** This metric does not carry a Workflow Id. Worker logs record the Workflow Id and Run Id. In the Temporal UI you can also find affected Executions by querying the `TemporalReportedProblems` Search Attribute, which the Temporal Service sets on Executions experiencing repeated Workflow Task failures.
1. **Find what made the completion large.** The limit applies to the sum, so look for what the Workflow produced in a single step:
- **A large fan-out.** Scheduling many Activities, Child Workflows, or Timers in one Workflow Task. Break the fan-out into smaller batches across multiple Workflow Tasks.
- **Accumulated Signals or Updates.** A large number buffered into one Workflow Task. Rate-limit senders or batch them.
- **Large Payloads in aggregate.** Individually under the Payload limit, but large once combined. Move them out of band with [External Storage](/external-storage).
1. **Check the Namespace limit.** The limit is a Namespace setting on the size of a single Workflow Task completion, separate from the Payload and gRPC message limits. On Temporal Cloud, contact support if the workload legitimately needs a higher limit. Self-hosted operators can adjust it in dynamic config.
1. **Fix and redeploy.** Affected Executions resume on their next Workflow Task retry once the corrected Worker is running. They do not need to be restarted, because they were never terminated.

## Workflow Task execution failures elevated {/* #workflow-task-execution-failures-elevated */}

**Recommended alert:** `temporal_workflow_task_execution_failed` with `failure_reason=WorkflowError`
Expand Down