diff --git a/plugins/temporal/skills/temporal-ops/SKILL.md b/plugins/temporal/skills/temporal-ops/SKILL.md new file mode 100644 index 0000000..23b24d5 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/SKILL.md @@ -0,0 +1,309 @@ +--- +name: temporal-ops +description: 'Administer and diagnose running Temporal Cloud or self-hosted Temporal Server environments via CLI (temporal, tcld) — not SDK code. Operations: namespace CRUD, Cloud capacity/APS, API-key rotation, mTLS cert rotation, workflow health, batch cancel/terminate/reset, export, search attributes, Ops API, billing, audit logs, Terraform, SAML/SCIM, migration. Diagnosis: bottom-up triage of stuck workflows, non-determinism, worker-health, task-queue problems, HA failover, payload size limits, performance bottlenecks, missed schedules. Do NOT trigger for generic TLS/gRPC errors unrelated to Temporal, writing application code (temporal-developer), or worker tuning/sizing (temporal-workertuning).' +version: 0.2.0 +disable-model-invocation: true +--- + +# Skill: temporal-ops + +## Overview + +This skill operates and diagnoses Temporal environments. It has two modes: + +- **Operations:** the user wants to do something — create a namespace, rotate a key, check capacity, find unhealthy workflows, cancel a batch, set up export. The skill executes the right commands and interprets the output. +- **Diagnosis:** the user arrives with a symptom — a stuck workflow, a cert error, a connection timeout, a non-determinism panic. The skill routes the investigation through a layered, bottom-up diagnosis until a root cause is identified with a confidence score. + +It does not teach how to write workflows or activities (use `skill-temporal-developer` for that), and it does not reproduce exhaustive CLI flag tables — run `temporal --help` for those, and see [cli-conventions.md](references/ops/cli-conventions.md) for cross-command CLI conventions. The boundary is: if the user needs to administer or troubleshoot a running Temporal environment, this skill applies. + +## Out of scope + +- **Writing workflows, activities, or SDK code** → `skill-temporal-developer`. +- **Exhaustive CLI flags / command reference** → run `temporal --help`; **cross-command CLI conventions** → [cli-conventions.md](references/ops/cli-conventions.md). +- **Worker performance tuning, sizing, capacity planning** → `skill-temporal-workertuning`. +- **Helm, Kubernetes, database admin, monitoring stack config** for self-hosted — beyond the CLI surface. + +If the conversation drifts into one of these areas, hand off to the relevant sibling skill rather than improvising. + +## Philosophy + +### Operator discipline + +When the user wants to perform an operational task: + +1. **Identify the intent and backend.** Is this a Cloud operation (`tcld`) or a self-hosted operation (`temporal operator`)? Data-plane operations (`temporal workflow`, `temporal batch`, etc.) work on both. **If the backend is ambiguous, ask before proceeding — do not assume Cloud or self-hosted and do not output environment-specific commands until you know.** +2. **Execute commands and interpret output.** Run the documented command, read the result, and report what it means — or act on it if the user asked for an action. Read-only commands (`get`, `list`, `describe`, `count`, `show`) run freely. Anything listed under [Destructive operations](#destructive-operations) is proposed to the user first. +3. **Verify the result.** After a mutating operation, confirm the new state matches the user's intent. + +### Destructive operations + +An operation belongs to this tier if it is **irreversible** (`tcld namespace +delete`), **revokes access for a live identity** (`tcld apikey delete`), **moves +production traffic** (`tcld namespace failover`), or **fans out to every match** +(any `--query` form). Apply the test to the operation in front of you — this is a +rule, not a list, and a command's absence from any list in this skill does not +place it outside the tier. + +For anything in the tier: gather the evidence and **propose**. Do not run it on +your own initiative, and do not run one to find out what it would do. The +reference file for each command states its specific blast radius; read that +before proposing, not after. + +1. **Blast radius as a number, not a description.** For any `--query` form, run + `temporal workflow count --query ''` with the byte-identical query + first and carry the result into the proposal. A filter with no narrowing + predicate beyond `ExecutionStatus="Running"` matches every open Execution in + the Namespace. +2. **Name the target.** State the exact command, the target, and the Namespace it + resolves to. Connection settings can come from `TEMPORAL_*` env vars or a + config-file profile, so the target is frequently not visible in the command + text. If the backend or Namespace was inferred from context rather than stated + by the user, say so — a destructive command aimed at the wrong Namespace is the + most common way this goes wrong. +3. **Ask explicitly, then run it so it completes.** Put the command, the target, + and — for any `--query` form — the count from step 1 to the user as a direct + question, and wait for an answer. Once they approve, run it with `--yes` on the + `--query` batch forms; that flag is what lets an approved batch finish, since + the interactive prompt needs a terminal and without one the command reports + `user denied confirmation` and does nothing. `--yes` belongs in a command the + user approved, never in a retry of one that failed its prompt. Do not substitute + a loop over single-target `workflow terminate --workflow-id`. Approval covers one + command against one target; it does not carry to the next command, a widened + query, or a second Namespace. +4. **Verify, and know the abort path.** Re-run the corresponding `get`, + `describe`, or `count`. A batch job drains asynchronously: + `temporal batch describe --job-id ` shows how far it has gotten and + `temporal batch terminate --job-id ` stops it before it reaches the rest + of its matches. + +When a reversible sibling reaches the same goal, propose it alongside: `cancel` +lets Workflow cleanup code run where `terminate` does not; `apikey disable` is +reversible where `delete` is not; `accepted-client-ca add` appends where `set` +replaces. + +Assume nothing in the environment will stop a destructive command on your behalf. +Credential scope, command denylists, and confirmation prompts may or may not be +configured, and their possible presence is not a reason to skip any step above — +you are the safeguard the user is relying on. + +### Diagnostic discipline + +When the user arrives with a failure or anomaly: + +1. **Bottom-up diagnosis.** Verify the lower layer before blaming the upper one. The layers, from bottom to top: + 1. DNS / network path + 2. TCP / port reachability + 3. TLS handshake + 4. Authentication (API key or mTLS client cert) + 5. gRPC health and Temporal frontend reachability + 6. Temporal namespace, task queues, workers + 7. Workflow code (determinism, signals, timers, child workflows) + + The full ladder lives in [diagnostic-ladder.md](references/triage/diagnostic-ladder.md). + +2. **Always verify the next layer up** rather than prescribing a speculative fix. If TLS works, prove auth works before blaming the workflow. If pollers are present, prove the workflow's last event before blaming the worker. + +3. **Attach a confidence score** (1-10) to every proposed diagnosis: + - 9-10: symptoms, operation, and confirming signals line up cleanly. + - 6-8: evidence is good but at least one alternative remains plausible. + - 1-5: the issue is still ambiguous; the "fix" is the next discriminating check, not a root cause. + +4. **Name ambiguity explicitly.** Errors like `context deadline exceeded` are not self-describing, and a single code such as `RESOURCE_EXHAUSTED` can mean more than one condition (account-limit throttling vs. per-Workflow lock contention). Surface that, gather more context, and scope the next step narrowly. + +These are skill conventions, not docs-derived facts. + +## Intent routing + +### Operations + +Find the row that matches the user's intent. The reference file contains the commands and procedures. + +| Intent | Category | Reference | +|---|---|---| +| Create, get, list, delete a Cloud namespace | Cloud namespace admin | [cloud-namespace-admin.md](references/ops/cloud-namespace-admin.md) | +| Add/remove region, failover, HA config | Cloud namespace admin | [cloud-namespace-admin.md](references/ops/cloud-namespace-admin.md) | +| Set retention, tags, codec-server, connectivity rules | Cloud namespace admin | [cloud-namespace-admin.md](references/ops/cloud-namespace-admin.md) | +| Add or rename Cloud search attributes | Cloud namespace admin | [cloud-namespace-admin.md](references/ops/cloud-namespace-admin.md) | +| Check current APS / capacity mode | Cloud capacity | [cloud-capacity.md](references/ops/cloud-capacity.md) | +| Switch On-Demand ↔ Provisioned, set TRUs | Cloud capacity | [cloud-capacity.md](references/ops/cloud-capacity.md) | +| Understand APS / RPS / OPS limits and throttling | Cloud capacity | [cloud-capacity.md](references/ops/cloud-capacity.md) | +| Create, disable, enable, delete an API key | Cloud IAM | [cloud-iam.md](references/ops/cloud-iam.md) | +| Invite, list, remove users; set roles/permissions | Cloud IAM | [cloud-iam.md](references/ops/cloud-iam.md) | +| Manage user groups and service accounts | Cloud IAM | [cloud-iam.md](references/ops/cloud-iam.md) | +| Generate mTLS certs, upload CA, set cert filters | Cloud certs | [cloud-certs.md](references/ops/cloud-certs.md) | +| Rotate mTLS certificates | Cloud certs | [cloud-certs.md](references/ops/cloud-certs.md) | +| Set up Workflow History Export (S3 / GCS) | Cloud namespace admin | [cloud-namespace-admin.md](references/ops/cloud-namespace-admin.md) | +| Set up PrivateLink / PSC, manage connectivity rules | Cloud connectivity | [cloud-connectivity.md](references/ops/cloud-connectivity.md) | +| Self-hosted cluster health, describe, namespace CRUD | Self-hosted admin | [self-hosted-admin.md](references/ops/self-hosted-admin.md) | +| Self-hosted search attributes, Nexus endpoints | Self-hosted admin | [self-hosted-admin.md](references/ops/self-hosted-admin.md) | +| Check or manage a Cloud Nexus Endpoint's caller-Namespace allowlist; the 1,000-caller default | Cloud namespace admin | [cloud-namespace-admin.md#tcld-nexus-endpoint-allowed-namespace](references/ops/cloud-namespace-admin.md#tcld-nexus-endpoint-allowed-namespace) | +| Find stuck/hung/unhealthy workflows via list queries | Workflow health | [workflow-health.md](references/ops/workflow-health.md) | +| Task queue poller status, workflow counts | Workflow health | [workflow-health.md](references/ops/workflow-health.md) | +| Cancel, terminate, or reset workflows | Workflow recovery | [workflow-stuck.md#recovery-commands](references/triage/workflow-stuck.md#recovery-commands) | +| Bulk / batch operations on workflows (`--query`) | CLI conventions | [cli-conventions.md](references/ops/cli-conventions.md#the---query--batch-job-bridge) | +| Schedule CRUD, time-spec, and operations | CLI conventions | [cli-conventions.md](references/ops/cli-conventions.md#schedule-time-spec-forms) | +| Complete or fail an activity externally | CLI conventions | [cli-conventions.md](references/ops/cli-conventions.md#operation--command-index) | +| Cloud Ops API access, rate limits, Go SDK | Cloud Ops API | [cloud-ops-api.md](references/ops/cloud-ops-api.md) | +| View billing, generate billing report, cost attribution | Cloud billing | [cloud-billing.md](references/ops/cloud-billing.md) | +| Audit Logs: view, query via API, configure sink (AWS/GCP) | Cloud audit logs | [cloud-audit-logs.md](references/ops/cloud-audit-logs.md) | +| Terraform provider: Namespace/User/SA/API Key/Nexus CRUD | Cloud Terraform | [cloud-terraform.md](references/ops/cloud-terraform.md) | +| Expiry alerts (cert, API key, credit), status page | Cloud notifications | [cloud-notifications.md](references/ops/cloud-notifications.md) | +| SAML SSO, SCIM provisioning, IdP integration | Cloud SAML/SCIM | [cloud-saml-scim.md](references/ops/cloud-saml-scim.md) | +| Migrate self-hosted to Cloud (automated or manual), migrate between Cloud regions | Cloud migration | [cloud-migration.md](references/ops/cloud-migration.md) | +| End-to-end ops playbook (setup, rotation, audit, billing, Terraform) | Ops recipes | [ops/recipes.md](references/ops/recipes.md) | + +### Diagnosis + +Find the row that matches the user's symptom. Start the investigation at the first check, then read the linked reference. + +| Symptom | Category | First check | Reference | +|---|---|---|---| +| `connection refused`, cannot reach frontend | Connectivity | `nc -zvw10 7233` | [connectivity.md#connection-refused](references/triage/connectivity.md#connection-refused) | +| `no such host`, DNS resolution fails | Connectivity | `dig +short ` or `nslookup ` | [connectivity.md#dns](references/triage/connectivity.md#dns) | +| `tls: handshake failure`, server rejects handshake | Certificates | `openssl s_client -connect :7233 -servername ` | [authentication.md#things-to-check-when-unauthenticated-is-returned-with-an-api-key](references/triage/authentication.md#things-to-check-when-unauthenticated-is-returned-with-an-api-key) | +| `namespace not found` / wrong namespace string with an API key | Authentication | Confirm Regional Endpoint form `..api.temporal.io:7233` | [authentication.md#address-form-for-api-key-connections](references/triage/authentication.md#address-form-for-api-key-connections) | +| `RESOURCE_EXHAUSTED` gRPC status | Rate limits | Identify which limit fired: throttle metrics on Cloud v1, the `resource_exhausted_cause` label on v0 / self-hosted | [rate-limits.md#identifying-which-limit-was-hit](references/triage/rate-limits.md#identifying-which-limit-was-hit) | +| Task queue shows no pollers | Worker health | `temporal task-queue describe --task-queue ` | [worker-health.md#what-no-pollers-looks-like](references/triage/worker-health.md#what-no-pollers-looks-like) | +| Workflow stuck on a pending activity / timer / child / signal | Workflow stuck | `temporal workflow describe --workflow-id ` | [workflow-stuck.md#the-primary-inspection-command-temporal-workflow-describe](references/triage/workflow-stuck.md#the-primary-inspection-command-temporal-workflow-describe) | +| `NondeterminismError`, repeating `WorkflowTaskFailed` | Non-determinism | Identify the last `WorkflowTaskFailed` cause in the Event History | [non-determinism.md#the-wft-failure-signature-of-non-determinism](references/triage/non-determinism.md#the-wft-failure-signature-of-non-determinism) | +| Replay fails locally but prod workflow was running | Non-determinism | Fetch the history and run the SDK replayer in one test | [replay.md#step-2--run-the-sdk-replayer-all-supported-sdks](references/triage/replay.md#step-2--run-the-sdk-replayer-all-supported-sdks) | +| HA failover did not route traffic to failover region | HA failover | `tcld namespace get --namespace .` vs. DNS CNAME | [ha-failover.md#start-here-establish-ground-truth](references/triage/ha-failover.md#start-here-establish-ground-truth) | +| Serverless Worker (AWS Lambda) stopped processing work after a Namespace failover | HA failover | Confirm a `FailoverNamespace` audit event, then compare the new active region against the Lambda ARN on the Worker Deployment Version | [ha-failover.md#symptom-serverless-workers-kept-running-in-the-old-region-after-failover](references/triage/ha-failover.md#symptom-serverless-workers-kept-running-in-the-old-region-after-failover) | +| `context deadline exceeded` (unknown layer) | Runtime errors | Identify which operation and SDK emitted it | [runtime-errors.md#deadline-exceeded](references/triage/runtime-errors.md#deadline-exceeded) | +| `Workflow is busy` / `ResourceExhausted` on signal/update/query to one Workflow (BusyWorkflow) | Runtime errors | Rule out account-limit throttling, then split `temporal_cloud_v1_resource_exhausted_error_count` by `operation` | [runtime-errors.md#workflow-lock-contention-busyworkflow](references/triage/runtime-errors.md#workflow-lock-contention-busyworkflow) | +| `PAYLOADS_TOO_LARGE`, `exceeds size limit`, payload/gRPC blob size error | Blob size limits | Check whether the issue is payload (2 MB) or gRPC message (4 MB) | [blob-size-limits.md](references/triage/blob-size-limits.md) | +| Workflow stuck in invisible retry loop (gRPC message too large) | Blob size limits | Check Worker logs for `ResourceExhausted`, reduce batch size | [blob-size-limits.md](references/triage/blob-size-limits.md) | +| High schedule-to-start latency, task slot depletion, slow Workflow Tasks | Performance bottlenecks | Check `temporal_workflow_task_schedule_to_start_latency` P95 | [performance-bottlenecks.md](references/triage/performance-bottlenecks.md) | +| High replay latency, cache evictions, deadlock detected | Performance bottlenecks | Check `workflow_task_replay_latency` and sticky cache metrics | [performance-bottlenecks.md](references/triage/performance-bottlenecks.md) | +| Schedule did not fire, missed catchup window | Missed Schedule Actions | Alert on `temporal_cloud_v1_schedule_missed_catchup_window_count` | [schedule-missed.md](references/triage/schedule-missed.md) | + +If a symptom does not map to a row, start at [diagnostic-ladder.md](references/triage/diagnostic-ladder.md) and work up from whichever layer was last known healthy. + +## The process + +### Operations path + +#### Step 1: Identify intent and backend + +Determine what the user wants to do and whether it targets: +- **Temporal Cloud** → use `tcld` commands (requires `tcld login`) +- **Self-hosted cluster** → use `temporal operator` commands +- **Data plane (either backend)** → use `temporal workflow`, `temporal batch`, `temporal schedule`, etc. + +If the backend is unambiguous from context — `.tmprl.cloud` address, `tcld` command, Cloud namespace format `ns.account` → Cloud; Kubernetes/Helm, `docker-compose`, self-hosted config files → self-hosted — proceed without asking. **Otherwise, stop and ask: "Are you on Temporal Cloud or self-hosted?" before outputting any environment-specific commands.** Do not default to either environment. Once known, save the answer to memory so you don't ask again in future conversations. + +#### Step 2: Execute and interpret + +Look up the intent in the Operations table above. Read the linked reference file for the exact commands, flags, and expected output. Run the command and interpret the result for the user. + +#### Step 3: Verify + +After a mutating operation (create, update, delete, rotate), confirm the new state: +- Re-run the corresponding `get` or `describe` command +- Confirm the output matches the user's intent +- Report the result + +### Diagnosis path + +#### Step 1: Identify the symptom + +Ask the user for the exact, copy-pasted error text. Do not accept paraphrases — the exact string often encodes the layer (e.g., `x509:` prefix means TLS/cert layer, `RESOURCE_EXHAUSTED:` prefix means gRPC rate limit, `NondeterminismError` means workflow replay layer). Note that `RESOURCE_EXHAUSTED` alone does not tell you which condition fired — account-limit throttling and per-Workflow lock contention (`Workflow is busy`, i.e. BusyWorkflow) share the code. Split the resource-exhausted metric by its label (`operation` on the Cloud v1 family, `resource_exhausted_cause` on v0 and self-hosted) rather than parsing the free-text message. + +Confirm three things before continuing: +- What command was run, or what SDK call produced the error? +- What environment produced it (local dev server, self-hosted cluster, Temporal Cloud)? If clear from context (addresses, commands, namespace format), don't ask — **but if uncertain, ask now before proceeding with any diagnosis.** Save the answer to memory for future conversations. +- What changed recently (new deploy, new certs, new namespace, new region)? + +#### Step 2: Gather context + +The context the investigation needs depends on the category. At minimum: + +- **For any Cloud auth / connectivity issue:** auth method (API key vs mTLS), exact address, exact namespace, SDK + version. The endpoint family differs by auth method — see [connectivity.md#endpoint-formats](references/triage/connectivity.md#endpoint-formats). For private connectivity (PrivateLink / PSC), TLS server name overrides also vary by auth method — see [cloud-connectivity.md](references/ops/cloud-connectivity.md). +- **For a stuck workflow:** namespace, workflow ID, run ID, and the output of `temporal workflow describe --workflow-id ` (pending-operation state lives here, not in the Event History alone). Event History via `temporal workflow show` is the companion view. +- **For a worker health issue:** worker logs (registration errors, auth errors, panics), the output of `temporal task-queue describe --task-queue `, and `temporal worker describe --task-queue ` for per-worker details. +- **For a non-determinism error:** the worker log line containing the error, the workflow type name, and access to the history JSON for replay. + +#### Step 3: Validate pasted SDK config (if any) + +If the user has pasted SDK connection code — even just the address/namespace/auth fields — review it against [sdk-snippet-review.md](references/triage/sdk-snippet-review.md) **before** descending the ladder. Wrong endpoint family, short namespace, or mismatched auth method will make every network-layer probe below look broken when nothing lower actually is. + +Skip this step when the user has an established, previously-working config and the symptom is new — the snippet is not the culprit, the environment changed. Otherwise treat snippet validation as Layer 0. + +#### Step 4: Descend the ladder + +Use [diagnostic-ladder.md](references/triage/diagnostic-ladder.md) to pick the right starting layer. As a rule of thumb: + +- Auth / connectivity / cert symptom → start at layer 1 (DNS) and walk up. +- Worker / task-queue symptom → start at layer 6 (namespace + pollers). +- Stuck workflow / determinism symptom → start at layer 7 (workflow code), but confirm layer 6 (worker is actually polling) first. + +Each layer has a command that proves it healthy and a failure signature that tells you whether the problem lives at that layer or higher. + +#### Step 5: Fix and verify + +Prescribe the fix scoped to the root cause. Then verify by re-running the layer's healthy-check command and, if possible, the original user operation. Attach the confidence score to the diagnosis. + +If the layer above the fix is still failing, return to step 4 and continue walking upward — the first broken layer is rarely the only one. + +## Prerequisites + +- **Temporal CLI** (`temporal`) — required for data-plane operations and self-hosted admin. Install: `brew install temporal` or see [Temporal CLI docs](https://docs.temporal.io/cli). +- **tcld** — required for Cloud operations. Install: `brew install temporal-cloud-cli` or see [tcld docs](https://docs.temporal.io/cloud/tcld). Authenticate with `tcld login` before use. + +## Reference files + +### Operations + +- [cloud-namespace-admin.md](references/ops/cloud-namespace-admin.md) — Cloud namespace lifecycle via `tcld`: create, get, list, delete, failover, add-region, retention, tags, codec-server, HA config, connectivity rules, search attributes, accepted-client-ca, certificate filters, export, and the `tcld nexus endpoint allowed-namespace` caller allowlist (1,000-caller Access Policy ceiling). +- [cloud-capacity.md](references/ops/cloud-capacity.md) — Capacity modes (On-Demand / Provisioned), APS/RPS/OPS definitions, TRUs, `tcld namespace capacity update`, default limits, throttling, APS management best practices. +- [cloud-iam.md](references/ops/cloud-iam.md) — API key lifecycle (`tcld apikey`), users (`tcld user`), user groups (`tcld user-group`), service accounts, account operations (`tcld account`), roles, namespace permissions. +- [cloud-certs.md](references/ops/cloud-certs.md) — mTLS cert management: generating certs with `tcld generate-certificates`, uploading CAs, certificate filters, cert rotation, switching mTLS ↔ API keys. +- [cloud-connectivity.md](references/ops/cloud-connectivity.md) — Private connectivity (AWS PrivateLink / GCP PSC), connectivity rules: setup, rule parameters, tcld commands, attaching rules to namespaces. +- [cloud-migration.md](references/ops/cloud-migration.md) — Migration paths: automated self-hosted→Cloud (S2S proxy, `tcld migration` commands, 5 phases), manual self-hosted→Cloud (client changes, workflow strategies), within-Cloud region-to-region (HA add-region/failover). +- [cloud-ops-api.md](references/ops/cloud-ops-api.md) — Cloud Ops API: HTTP and gRPC endpoints (`saas-api.tmprl.cloud`), Go SDK, protobuf compilation, rate limits (160 RPS account, 40 user, 80 SA, 10 concurrent async), API version header, use cases. +- [cloud-billing.md](references/ops/cloud-billing.md) — Cloud billing: Billing Center (invoices, credits, plans, cost by namespace), Usage Dashboards, Billing API: async CSV report generation, FOCUS-friendly format, 27-column report schema, date range constraints. +- [cloud-audit-logs.md](references/ops/cloud-audit-logs.md) — Cloud Audit Logs: supported control plane events (Account, API Keys, Connectivity Rules, Namespace, Export, Nexus, Service Accounts, User, User Groups), JSON format, API access (30-day retention), AWS Kinesis and GCP Pub/Sub sink configuration. +- [cloud-terraform.md](references/ops/cloud-terraform.md) — Terraform provider: setup (`TEMPORAL_CLOUD_API_KEY`), `temporalcloud_namespace`/`temporalcloud_user`/`temporalcloud_service_account`/`temporalcloud_apikey`/`temporalcloud_nexus_endpoint` CRUD, import, data sources (regions, namespaces), limitations (API keys not importable, cannot manage Account Owner). +- [cloud-notifications.md](references/ops/cloud-notifications.md) — Cloud notifications: certificate expiry (15/10/5 days), API key expiry (30/20/10 days), credit consumption/expiry alerts, plan changes, failover events, recipient roles, `noreply@temporal.io` sender. +- [cloud-saml-scim.md](references/ops/cloud-saml-scim.md) — SAML SSO (Entra ID, Okta): entity identifier (`urn:auth0:prod-tmprl:ACCOUNT_ID-saml`), callback URL (`login.tmprl.cloud`), IdP configuration steps, support ticket workflow. SCIM: supported vendors, prerequisites (SAML first), 10-minute sync window, group-to-role mapping. +- [self-hosted-admin.md](references/ops/self-hosted-admin.md) — Self-hosted control plane via `temporal operator`: cluster health/describe, namespace CRUD, search-attribute create/list/remove, Nexus endpoint CRUD. +- [workflow-health.md](references/ops/workflow-health.md) — Data-plane health queries: `temporal workflow list` with List Filters, `temporal workflow describe`/`show`/`count`, `temporal task-queue describe` for poller status. +- [cli-conventions.md](references/ops/cli-conventions.md) — Cross-command `temporal` CLI conventions: connection/identity (`TEMPORAL_*` env vars ↔ `--address`/`--namespace`/`--api-key`, `--identity`), output/formatting (`--output`, `--time-format`, payload shorthand), the `--query` ⇒ batch-job bridge (with `temporal batch describe/list/terminate`), and schedule time-spec forms. Ends with an operation→command index that routes each data-plane operation to its owner file. Delegates exhaustive flags to `temporal --help`. +- [ops/recipes.md](references/ops/recipes.md) — End-to-end ops playbooks: set up new namespace, check APS, switch capacity mode, find hung workflows, rotate API key, audit access, rotate mTLS certs, check self-hosted health, view billing / generate billing report, configure audit log sink, provision resources with Terraform, set up SAML SSO. + +### Diagnosis + +- [sdk-snippet-review.md](references/triage/sdk-snippet-review.md) — Layer-0 config check for pasted SDK connection snippets: endpoint form per auth method, namespace format, auth / TLS expectations, `TEMPORAL_*` env vars, common misconfigurations. Run before the diagnostic ladder. +- [diagnostic-ladder.md](references/triage/diagnostic-ladder.md) — the seven-layer bottom-up model, with one canonical command per layer and cross-links into the topical leaves. +- [connectivity.md](references/triage/connectivity.md) — DNS, TCP, endpoint families (Namespace Endpoint for mTLS vs. Regional Endpoint for API keys), firewall/proxy shapes, PrivateLink/PSC, quick diagnostic scripts. +- [certificates.md](references/triage/certificates.md) — x509 and TLS alert strings, expiry / unknown-authority / hostname-mismatch / key-mismatch diagnosis, Cloud accepted-client-CA set via `tcld namespace accepted-client-ca`, Cloud mTLS certificate requirements, rotation and expiry notifications, openssl recipes. +- [authentication.md](references/triage/authentication.md) — `UNAUTHENTICATED` vs `PERMISSION_DENIED`, API-key lifecycle (`tcld apikey` commands, env var propagation, required Regional Endpoint form), mTLS after TLS (certificate filters, identity-to-role mapping), Cloud account-level roles and namespace-level permissions. +- [workflow-stuck.md](references/triage/workflow-stuck.md) — Workflow Execution Status values, `temporal workflow describe` as the primary inspection command, Event History via `temporal workflow show`, pending activities / child workflows / signals / Nexus operations / Workflow Tasks, WorkflowTaskFailed retry loops, recovery commands (signal, terminate, cancel, reset, pause/unpause). +- [non-determinism.md](references/triage/non-determinism.md) — determinism definition, WFT-failure signature, ND-inducing code patterns, per-SDK error shapes, identifying ND from Event History, local replay reproduction, remediation via Worker Versioning / patching / reset. +- [worker-health.md](references/triage/worker-health.md) — no-pollers runbook via `temporal task-queue describe`, reachability and versioning, worker-level describe, schedule-to-start latency, worker task slots, sticky execution and sticky cache, worker heartbeating, Cloud namespace-level poller limits, worker log signatures. +- [rate-limits.md](references/triage/rate-limits.md) — what `RESOURCE_EXHAUSTED` means (and does not), Cloud APS / RPS / OPS under On-Demand and Provisioned capacity modes, self-hosted `frontend.rps` / `frontend.namespaceRPS` dynamic config, identifying which limit fired via the throttle metrics (Cloud v1) or the `resource_exhausted_cause` label (v0 / self-hosted), and separating account-limit throttling from single-resource exhaustion. +- [ha-failover.md](references/triage/ha-failover.md) — Cloud HA routing via the Namespace Endpoint CNAME, verifying the active region (control-plane `tcld namespace get` vs. DNS view), clients that did not follow the failover, PrivateLink after failover, failover-not-executing, handover-window errors, platform limits, RPO/RTO semantics, and Serverless Workers (AWS Lambda) not following a failover because compute-provider configuration is region-scoped. +- [runtime-errors.md](references/triage/runtime-errors.md) — deadline-exceeded disambiguated by operation and by where the call was made, Workflow lock contention (BusyWorkflow) separated from account-limit throttling and confirmed via the `operation` breakdown, routing for `no pollers` / `INVALID_ARGUMENT` / unspecified `UNAVAILABLE`. +- [replay.md](references/triage/replay.md) — fetching Event History with the SDK client (CLI export as fallback), running the SDK replayer in every supported SDK (Go, Python, TypeScript, Java, .NET, Ruby, PHP), `TEMPORAL_DEBUG` and the deadlock detector, interpreting divergent and successful replays, and the TypeScript-only VS Code extension. +- [blob-size-limits.md](references/triage/blob-size-limits.md) — Payload size limit (2 MB) and gRPC message size limit (4 MB): error messages, per-SDK behavior (Python 1.23.0+ vs. others), claim check pattern, External Storage (Pre-release), batch-size reduction. +- [performance-bottlenecks.md](references/triage/performance-bottlenecks.md) — Latency and throughput diagnosis via SDK metrics: schedule-to-start latency, workflow task execution latency, replay latency, activity execution latency, task slot depletion, network request metrics, sticky cache metrics. +- [schedule-missed.md](references/triage/schedule-missed.md) — Missed Schedule Actions: alerting via `temporal_cloud_v1_schedule_missed_catchup_window_count` / `schedule_missed_catchup_window`, investigation via `temporal schedule list` + `temporal schedule describe`, DescribeSchedule fields (`missedCatchupWindow`, `overlapSkipped`, `bufferDropped`), default catchup window (one year), root causes, overlap policies (6 values), backfill remediation. +- [recipes.md](references/triage/recipes.md) — four end-to-end triage walkthroughs: stuck workflow at 3am, cert expired with workers offline, task-queue backlog mystery, non-determinism caught in prod. + +## Feedback + +### Feedback prompt + +When you have **first** loaded this skill, output this message to the user: + +"Thank you for trying out the public preview of the Temporal Ops skill! We would love to hear your feedback - positive or negative - over in the [Community Slack](https://t.mp/slack), in the [#topic-ai channel](https://temporalio.slack.com/archives/C0818FQPYKY)." + +Do not output this message multiple times in the same conversation. + +### Reporting Issues in This Skill + +If you (the AI) find this skill's explanations are unclear, misleading, or missing important information, draft a GitHub issue body describing the problem encountered and what would have helped, then ask the user to file it at https://github.com/temporalio/skill-temporal-ops/issues/new. Do not file the issue autonomously. diff --git a/plugins/temporal/skills/temporal-ops/agents/openai.yaml b/plugins/temporal/skills/temporal-ops/agents/openai.yaml new file mode 100644 index 0000000..b75d213 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/agents/openai.yaml @@ -0,0 +1,6 @@ +interface: + display_name: "Temporal Ops" + short_description: "Administer and diagnose Temporal Cloud and self-hosted environments" + +policy: + allow_implicit_invocation: false diff --git a/plugins/temporal/skills/temporal-ops/references/ops/cli-conventions.md b/plugins/temporal/skills/temporal-ops/references/ops/cli-conventions.md new file mode 100644 index 0000000..7701db5 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/ops/cli-conventions.md @@ -0,0 +1,187 @@ +# Temporal CLI conventions and command index + +Cross-cutting conventions for the `temporal` data-plane CLI (`workflow`, `batch`, +`schedule`, `activity`), plus an index that routes each operation to the file +that owns its judgment. + +> **Scope.** Data-plane commands are documented across several files (see the +> [command index](#operation--command-index) below); this file is *not* the home +> for any single command. It holds the cross-command rules that don't belong to +> one operation, and points at the owner file for everything else. + +## Use `--help` for flags + +This file does not reproduce full flag tables — run `temporal --help` +for the exhaustive, version-current flag set: + +```bash +temporal --help # e.g. temporal workflow reset --help +``` + +It *does* curate the behaviors that are destructive, non-obvious, or buried in +`--help`'s description prose, plus the cross-command rules that belong to no +single command. (When a curated fact is version-volatile, the *behavior* is +stated here and the exact flag spelling/values are left to `--help`.) + +## Connection and identity + +Every `temporal` command reads the same connection settings from three +sources, checked in this order (first match wins): flag, env var, then +config-file profile. + +| Env var | Flag | Purpose | +|---|---|---| +| `TEMPORAL_ADDRESS` | `--address` | Frontend `host:port` (default `localhost:7233`). | +| `TEMPORAL_NAMESPACE` | `--namespace`, `-n` | Namespace (default `default`). | +| `TEMPORAL_API_KEY` | `--api-key` | API-key auth (implies TLS). | + +- **Config-file profile** (Environment Configuration) supplies a value only + when no flag or env var sets it: `[profile.]` in a TOML file at + `$CONFIG_PATH/temporalio/temporal.toml` (OS-specific path; run `temporal + config --help`). Pick the profile with `--profile` / `TEMPORAL_PROFILE` + (default `default`); point at a different file with `--config-file` / + `TEMPORAL_CONFIG_FILE`. Docs: + [Environment Configuration](https://docs.temporal.io/develop/environment-configuration). +- mTLS uses `TEMPORAL_TLS_CERT` / `TEMPORAL_TLS_KEY` (`--tls-cert-path` / + `--tls-key-path`). The endpoint form differs by auth method — see + [../triage/connectivity.md](../triage/connectivity.md). +- `--identity` records who ran a mutating command (default + `temporal-cli:$USER@$HOST`) and shows up in Event History and audit logs. Set + it explicitly in shared automation. + +Full env-var list: [docs.temporal.io/cli/setup-cli](https://docs.temporal.io/cli/setup-cli). + +## Output and formatting + +- `--output`, `-o` — `text` (default), `json`, `jsonl`, `none`. Use `json`/`jsonl` + for scripting and pipe to `jq`; the triage/health runbooks recommend JSON + output whenever a step fans out over many results. +- `--time-format` — `relative` (default), `iso`, `raw`. +- Payload shorthand: JSON output renders payloads inline by default; pass + `--no-json-shorthand-payloads` to emit the raw payload envelope instead. + +## The `--query` ⇒ batch-job bridge + +The one cross-command rule worth memorizing. Passing `--query` (a +[List Filter](workflow-health.md#1-list-filter-fundamentals)) in place of +`--workflow-id` to `temporal workflow cancel | terminate | signal | delete` does +**not** act inline — it starts an asynchronous **batch job** over every matching +Execution. (The `--query` form of `temporal activity reset | unpause` behaves the +same way.) + +### Count before you mutate + +The mutating form does not report how many Executions it matched until the job is +already running. Run the query through `count` first and put that number in front +of the user: + +```bash +temporal workflow count --query 'ExecutionStatus="Running" AND WorkflowType=""' +``` + +Use the byte-identical query string in both commands — a predicate dropped between +the `count` and the `terminate` silently widens the blast radius. A filter with no +narrowing term beyond `ExecutionStatus="Running"` matches every open Execution in +the Namespace. + +### Running an approved batch + +```bash +temporal workflow terminate \ + --query 'ExecutionStatus="Running" AND WorkflowType=""' \ + --reason "" \ + --rps \ # throttle the batch; only valid with --query + --yes # proceed without the interactive prompt +``` + +Without `--yes` the command prompts `Start batch against approximately N +workflow(s)? y/N`. That prompt needs a terminal: with no terminal attached it +reads EOF, reports `user denied confirmation`, exits non-zero, and touches +nothing. So `--yes` is how an already-approved batch actually runs — and it is +also what suppresses the `N`, which is why the `count` above is not optional. +Get the user's approval on the scope, then run it with `--yes`; do not discover +the flag by retrying a command that failed the prompt. + +### Inspecting and aborting a running job + +A batch job drains asynchronously, so one started against too broad a query can +still be stopped before it reaches the rest of its matches: + +```bash +temporal batch list +temporal batch describe --job-id # progress; how far it has drained +temporal batch terminate --job-id --reason "" +``` + +`batch terminate` stops the job, **not** the Executions it already acted on — +those are already terminated, cancelled, or deleted and stopping the job does not +bring them back. + +`--reason`, `--rps`, and `--yes` are accepted only when `--query` is present. For +*which* List Filter to run, see [workflow-health.md](workflow-health.md). + +### Single-target commands have no prompt at all + +A single-target `workflow cancel | terminate | delete | signal` (with +`--workflow-id`) executes immediately — there is no confirmation and no `--yes` +to skip, because the prompt exists **only** on the `--query` batch form above. +The scope is one Execution, but nothing between the command and the effect will +catch a wrong Workflow ID or a wrong Namespace, so confirm both before running. + +`workflow delete` in a multi-region (global) Namespace removes the Execution from +**all replicas**; requests to a passive cluster are forwarded to the active one by +default — pass `--grpc-meta xdc-redirection=false` to target a passive cluster. + +## Schedule time-spec forms + +`temporal schedule create` (and `update`) accept any combination of three spec +flags (run `temporal schedule create --help` for the rest): + +- `--interval` — shorthand duration, e.g. `45m`, or `6h/5h` (every 6h, offset 5h). +- `--calendar` — JSON, e.g. `{"dayOfWeek":"Fri","hour":"3","minute":"30"}`. +- `--cron` — Unix cron or robfig (`@daily`, `@every 1h`), e.g. `"30 12 * * Fri"`. + +`--overlap-policy` takes one of six values (`Skip`, `BufferOne`, `BufferAll`, +`CancelOther`, `TerminateOther`, `AllowAll`); their semantics and the backfill +workflow live in [../triage/schedule-missed.md](../triage/schedule-missed.md). +Concept page: [docs.temporal.io/schedule](https://docs.temporal.io/schedule). + +**Update/delete gotchas** (`--help` buries these in the command description): +`temporal schedule update` **fully replaces** the schedule spec — options you don't +pass reset to defaults, so `describe` first and re-specify everything; `memo` and +search attributes can't be changed after creation. `temporal schedule delete` does +**not** stop already-running Executions — terminate those separately (e.g. +`temporal workflow terminate` by `TemporalScheduledById`). + +**`backfill` is a fan-out.** It replays the Schedule's actions across a past +window, so the Executions it starts scale with the window divided by the interval — +a month backfilled onto a 15-minute schedule is roughly 2,880 of them, and under +`--overlap-policy AllowAll` they start together rather than queueing. Compute that +number from the window and interval and put it in front of the user before running +one, the same way `count` precedes a `--query` mutation. The overlap policy is the +difference between a backfill that drains and one that stampedes the Worker fleet; +see [../triage/schedule-missed.md](../triage/schedule-missed.md). + +## Operation → command index + +One row per common data-plane operation. The linked file owns the judgment (when +to run it, how to read the output); run `temporal --help` for flags. + +| Operation | Skeleton | Owner | +|---|---|---| +| Find / list / count unhealthy workflows | `temporal workflow list --query ''` | [workflow-health.md](workflow-health.md) | +| Inspect one workflow | `temporal workflow describe -w ` / `show` / `stack` | [workflow-health.md](workflow-health.md), [../triage/workflow-stuck.md](../triage/workflow-stuck.md) | +| Recover a stuck workflow (signal / cancel / terminate / reset) | `temporal workflow signal\|cancel\|terminate\|reset ...` | [../triage/workflow-stuck.md](../triage/workflow-stuck.md#recovery-commands) | +| Reset past a non-determinism divergence | `temporal workflow reset -w --event-id ` | [../triage/non-determinism.md](../triage/non-determinism.md) | +| Bulk cancel / terminate / signal / delete | `--query` form → [batch bridge](#the---query--batch-job-bridge) | this file + [workflow-health.md](workflow-health.md) | +| Pause / unpause / reset a stuck activity | `temporal activity pause\|unpause\|reset ...` | [../triage/workflow-stuck.md](../triage/workflow-stuck.md#temporal-activity-pause--unpause--reset) | +| Complete / fail an activity externally | `temporal activity complete\|fail -a -w ` | `temporal activity --help` | +| Schedule CRUD (create / update / toggle / trigger / delete) | `temporal schedule -s ...` | this file ([spec forms](#schedule-time-spec-forms)) | +| Backfill / diagnose missed schedule actions | `temporal schedule backfill -s ...` | [../triage/schedule-missed.md](../triage/schedule-missed.md) | + +`activity complete | fail` inject a result the Activity never produced. The +Workflow resumes on that outcome as if the Activity had really succeeded or +failed, and the Event History records the supplied result with no undo. They are +for an Activity genuinely completing asynchronously outside the Worker — propose +rather than run. To stop a retry loop rather than answer it, use +[`activity pause`](../triage/workflow-stuck.md#temporal-activity-pause--unpause--reset). diff --git a/plugins/temporal/skills/temporal-ops/references/ops/cloud-audit-logs.md b/plugins/temporal/skills/temporal-ops/references/ops/cloud-audit-logs.md new file mode 100644 index 0000000..c814680 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/ops/cloud-audit-logs.md @@ -0,0 +1,211 @@ +# Cloud Audit Logs + +Audit Logs provide forensic access information for operations in the Temporal Cloud control plane. They answer "who, when, and what" questions about Temporal Cloud resources. + +Required role: Account Owner or Global Administrator to view Audit Logs via UI, use the API, or configure an Audit Log Integration. + +**Audit Logs do NOT capture data plane events** (Workflow Start, Workflow Terminate, Schedule Create, etc.). For closed Workflow Histories, use the Export feature instead. + +--- + +## Supported events + +### Account +- `ChangeAccountPlanType`: Change Account Plan Type +- `UpdateAccountAPI`: Configure Audit Logs, Configure Observability Endpoint + +### API Keys +- `CreateAPIKey`: Create API Key +- `DeleteAPIKey`: Delete API Key +- `UpdateAPIKey`: Update API Key + +### Connectivity Rules +- `CreateConnectivityRule`: Create Connectivity Rule +- `DeleteConnectivityRule`: Delete Connectivity Rule + +### Namespace +- `CreateNamespaceAPI`: Create Namespace +- `DeleteNamespaceAPI`: Delete Namespace +- `FailoverNamespacesAPI`: Failover (for High Availability Namespaces) +- `RenameCustomSearchAttributeAPI`: Rename Custom Search Attribute +- `UpdateNamespaceAPI`: Retention period changes, replica edits, authentication method updates, custom search attribute updates, connectivity rule bindings + +### Namespace Export +- `CreateNamespaceExportSink`: Create Namespace Export Sink +- `DeleteNamespaceExportSink`: Delete Namespace Export Sink +- `UpdateNamespaceExportSink`: Update Namespace Export Sink +- `ValidateNamespaceExportSink`: Validate Namespace Export Sink + +### Nexus Endpoint +- `CreateNexusEndpoint`: Create Nexus Endpoint +- `DeleteNexusEndpoint`: Delete Nexus Endpoint +- `UpdateNexusEndpoint`: Update Nexus Endpoint + +### Service Accounts +- `CreateServiceAccount`: Create Service Account +- `CreateServiceAccountAPIKey`: Create Service Account API Key +- `DeleteServiceAccount`: Delete Service Account +- `UpdateServiceAccount`: Update Service Account + +### User +- `CreateUserAPI`: Create Users +- `DeleteUserAPI`: Delete Users +- `InviteUsersAPI`: Invite Users +- `SetUserNamespaceAccessAPI`: Set User Namespace Access +- `UpdateIdentityNamespacePermissionsAPI`: Update Identity Namespace Permissions +- `UpdateUserAPI`: Update User Account-level Roles +- `UpdateUserNamespacePermissionsAPI`: Update User Namespace Permissions + +### User Groups +- `CreateUserGroup`: Create User Group +- `DeleteUserGroup`: Delete User Group +- `SetUserGroupNamespaceAccess`: Set User Group Namespace Access +- `UpdateUserGroup`: Update User Group + +--- + +## Audit Log format + +```json +{ + "operation": // Operation that was performed + "principal": // Information about who initiated the operation + "raw_details": // Details about the request + "x_forwarded_for": // The IP address(es) making the call + "emit_time": // Time the operation was recorded + "log_id": // Unique ID of the log entry + "async_operation_id": // Optional async operation id set by the user when sending a request + "request_id": // DEPRECATED, use async_operation_id + "status": // Status, such as OK or ERROR + "version": // Version of the log entry +} +``` + +**Deprecation notice:** The `request_id` field is deprecated and is planned for removal on or after November 1 2026. Use `async_operation_id` instead. + +The `x_forwarded_for` field uses the `X-Forwarded-For` format: a comma-separated list of IP addresses, evaluated from last to first until meeting the first untrusted IP address. + +--- + +## Viewing Audit Logs + +### Via the Cloud UI + +1. Select **Settings**. +2. On the **Settings** page, select **Audit Logs**. + +Up to 1000 events can be downloaded from the Audit Log UI to a local file. + +### Via the API + +Audit Logs can be accessed using the Cloud Ops API. Use the API to build dashboards for viewing Audit Logs outside of Temporal Cloud. If your goal is to export logs continuously, use an Audit Log sink instead. + +Audit Logs are accessible for the past 30 days using the API. + +API filter parameters: + +| Parameter | Description | +|---|---| +| `StartTimeInclusive` | Filter for UTC time >= (defaults to 30 days ago) - optional | +| `EndTimeExclusive` | Filter for UTC time < (defaults to current time) - optional | +| `PageSize` | Cannot exceed 1000. Defaults to 100. - optional | +| `PageToken` | Page token for continuing from another response - optional | + +--- + +## Audit Log sink configuration + +Audit Logs can be sent to AWS Kinesis or GCP Pub/Sub. + +### AWS Kinesis + +Prerequisites: an AWS account and Kinesis Data Streams. + +An [AWS CloudFormation template](https://temporal-auditlogs-config.s3.us-west-2.amazonaws.com/cloudformation/iam-role-for-temporal-audit-logs.yaml) is available to create an IAM role with access to a Kinesis stream. + +Kinesis has a rate limit of 1,000 messages per second. + +Setup via Cloud UI: + +1. Select **Settings** > **Audit Logs** > **Setup**. +2. Choose your **Access method**: **Auto** (configure CloudFormation from the Cloud UI) or **Manual** (download a template). +3. Enter the **Kinesis ARN**, **Role name**, and **AWS region**. +4. Follow the Auto or Manual steps to complete CloudFormation stack creation. + +Use the **Verify** button to confirm Temporal can write to the stream. + +First logs appear within 10 minutes after configuring the sink. + +### GCP Pub/Sub + +For manual setup: create a Pub/Sub topic and a service account in the same GCP project. + +Setup via Cloud UI: + +1. Select **Settings** > **Audit Logs** > **Setup**. +2. Select **Pub/Sub**. +3. Enter the **service account email** and **Topic name**. +4. Choose **Manual** or **Deploy with Terraform** to configure permissions. +5. Use the **Verify** button to confirm Temporal can write to the topic. +6. Click **Create**. + +Audit Logs appear in Pub/Sub within 10 minutes. + +If using Terraform for deployment, the manual prerequisites (topic and service account creation) can be skipped. + +--- + +## Managing sinks via tcld + +Sinks can also be managed with `tcld account audit-log` (alias `al`), under two provider subgroups: `kinesis` (alias `k`) and `pubsub` (alias `ps`). + +Both subgroups expose the same subcommands: + +| Subcommand | Alias | Purpose | +|---|---|---| +| `create` | `c` | Create a sink (created enabled) | +| `validate` | `v` | Validate sink config without creating it | +| `update` | `u` | Update sink fields or toggle enabled | +| `get` | `g` | Get a sink by name | +| `delete` | `d` | Delete a sink by name | +| `list` | `l` | List sinks | + +### Kinesis create/validate flags + +| Flag | Alias | Required | +|---|---|---| +| `--sink-name` | | Yes | +| `--role-name` | `--rn` | Yes | +| `--destination-uri` | `--du` | Yes | +| `--region` | `--re` | Yes | + +### Pub/Sub create/validate flags + +| Flag | Alias | Required | +|---|---|---| +| `--sink-name` | | Yes | +| `--service-account-email` | `--sae` | Yes | +| `--topic-name` | `--tn` | Yes | + +`update` additionally takes `--enabled` (toggle `true`/`false`) and `--resource-version` / `-v`; provider flags are optional on update. + +`get`, `delete`, and `list` are shared across both subgroups. `get` and `delete` identify the sink with `--sink-name` (`delete` also accepts `--resource-version` / `-v`); `list` accepts `--page-size` and `--page-token`. + +--- + +## Troubleshooting + +### Sink status + +The Audit Logs page of the Cloud UI shows the current status: + +- If an error is detected, a summary appears below the page title. +- If functioning normally, an **On** badge appears next to the page heading. + +Temporal retains Audit Log information for up to 30 days. To retrieve logs up to the past 30 days, file a request. + +If you experience an issue with a sink, Temporal can provide missing audit information via a support ticket. + +### Deleting a sink + +In the Cloud UI: **Settings** > **Audit Logs** > **Edit** > **Delete** at the bottom of the page. After confirmation, the sink is removed and logs stop flowing to the stream. diff --git a/plugins/temporal/skills/temporal-ops/references/ops/cloud-billing.md b/plugins/temporal/skills/temporal-ops/references/ops/cloud-billing.md new file mode 100644 index 0000000..b732f62 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/ops/cloud-billing.md @@ -0,0 +1,185 @@ +# Cloud Billing + +Temporal Cloud provides billing and costs information for your account. Use this information to assess spending patterns, inspect your credit ledger, check invoice histories, update payment details, and manage your current plan. + +--- + +## Tools for measuring usage and billing + +| Tool | What it provides | Who can view | +|---|---|---| +| **Billing Center** | Summary invoices, credits, plan management, account deletion | Account Owners, Finance Admin | +| **Billing API** | Namespace-level cost attribution down to hourly granularity, enriched with Tags and Projects; FOCUS-friendly CSV format | Account Owners, Finance Admin | +| **Usage Dashboards** | Aggregate Actions on a Namespace level with Action categories | Account Owners, Finance Admin, Global Admin (account level); Namespace access holders (namespace level) | +| **Actions in Event History** | Highlights Actions in a given Event History via the Cloud UI (some Actions are not measured in Workflow histories) | Account Owners, Global Admin, Namespace Admin, Developers, Read-Only | +| **Actions Metrics** | High-cardinality billable action metric with labels for Category, Action Type, Workflow Type, Namespace (minute granularity) | Metrics Read-Only service account role | + +--- + +## Billing Center + +Access: Account Owners and Finance Admins. + +### Current balance + +Shows the balance for the current billing cycle and the date it was last updated. This balance adjusts with use. + +Billing cycles normally begin on the first of the month (UTC). The minimum plan fee for your first month is prorated based on your sign-up date. + +### Recent bill + +Displays the previous bill amount. If the account pays through Stripe, a **Pay Now** button appears. Auto-payment accounts do not need to manually pay. + +### Invoices table + +| Column | Description | +|---|---| +| Date (UTC) | Date range covered by the invoice | +| Type | Type of invoice (e.g., credit purchase, cloud usage) | +| Status | Current status (e.g., paid, pending) | +| Credit Granted | Total credits added to the account | +| Credit Purchase Amount | Amount paid for purchasing credits | +| Credit Usage | Credits used during the billing cycle | +| Subtotal | Total amount before adjustments | +| Balance Due | Amount to pay after applying credits | + +Invoices prior to the current calendar month can be downloaded. The current billing period invoice is not finalized and cannot be downloaded. + +### Credits table + +| Column | Description | +|---|---| +| Effective At (UTC) | Date when the credit grant became effective | +| Type | Whether the transaction was a deduction, expiry, or grant | +| Amount | Credit amount granted, deducted, or expired | +| Credits Remaining | Remaining credit available | + +### Plans + +Account Owners and Finance Admins can view plan information, pricing details, entitlements, available plans, and Pay-as-You-Go pricing rates. On a standard agreement they can also upgrade and downgrade between available plans. + +- Upgrades are processed immediately with pro-rated billing. Monthly entitlements reflect the full volume of the upgrade plan for that billing month. After an upgrade, a downgrade cannot be processed until the following billing period. +- Downgrades are processed immediately. Billing and entitlements are backdated to the beginning of the billing period. + +### Account cancellation + +- **Accounts managed by sales team:** Submit a support ticket. +- **Self-signup accounts:** Account owners can delete their accounts on the Billing page, under the **Plan** tab. Permanently deleted accounts immediately cease billing and are scheduled for full deletion within 72 hours. Account Data and Active Storage are permanently deleted. Retained Storage is deleted per its configured retention period. + +--- + +## Usage Dashboards + +Actions usage is tracked across an account in the usage dashboard and is visible to Account Owners, Finance Admin, and Global Admin. Per-namespace usage is visible on the Namespace pages to those with access. + +### Actions in Workflows + +When viewing an Event History, events that represent a Billable Action are annotated with the number consumed by the event in the **Billable Actions** column. These Actions are summarized at the top of the workflow. + +This estimate is useful for projecting cost. Example: 20 Actions per run, 100 runs/day, 30 days = 60,000 Billable Actions per month. + +> **Treat the estimate as an estimate.** The Billable Action estimate is an **experimental feature** and only measures Billable Actions that exist within Workflow event histories. If billable events exist outside of event history, the actual Actions count could be higher. Workflows with the `TemporalNamespaceDivision` Search Attribute set may not have accurate estimates. + +Excluded from the Billable Actions estimate: + +- Query +- Activity Heartbeats +- Rejected Update Workflow Executions +- Export +- Schedule +- Replicated Actions in Namespace replication + +--- + +## Billing API + +The Billing API is part of the Cloud Operations API. It provides Namespace-level cost attribution through on-demand billing reports in CSV format, for ingestion into FinOps tooling and cloud cost management platforms. + +Reports contain: + +- Accurate Namespace-level cost attribution +- Hourly, daily, and monthly granularities +- A FOCUS-friendly data format + +### Report generation flow + +Report generation is **asynchronous**. + +1. Create a billing report using `CreateBillingReport`. The response includes a `billing_report_id` and `async_operation_id`. +2. Poll `GetBillingReport` using the `billing_report_id`. +3. When the report state becomes `BILLING_REPORT_STATE_GENERATED`, retrieve the download URL. +4. Download the report before the URL expires. + +Key identifiers: + +| Identifier | Purpose | +|---|---| +| `billing_report_id` | Identifies the billing report; used to retrieve metadata and download URLs | +| `async_operation_id` | Identifies the background operation responsible for generating the report | + +The async operation follows the standard Cloud Operations async model. See [cloud-ops-api.md](cloud-ops-api.md). + +### Allowed date ranges + +Date ranges must use billing-month boundaries (MM/YYYY). Requests may include the current billing month. Finalized reports include usage up to `current_time` - 24 hours (rounded down to the granularity level). + +Data range limits by granularity: + +| Granularity | Available range | +|---|---| +| Hourly | Current billing month + previous billing month | +| Daily | Current billing month + previous two billing months | +| Monthly | Current billing month + previous eleven billing months | + +### Rate limits and concurrency + +Within a single account, only one billing report is generated at a time. Additional requests are accepted but queued. + +Report generation time varies and is not guaranteed. Factors include the size of the requested date range and overall platform load. + +### Best practices + +- Provide an idempotency key (`async_operation_id`) when retrying requests. +- Poll `GetBillingReport` using exponential backoff. +- Download reports immediately after generation (URLs expire). +- Avoid frequent generation of large overlapping ranges in the current billing period. + +### Report schema (27 columns) + +Each row represents a charge record. + +| Column Name | Description | Example | +|---|---|---| +| `BillingAccountID` | Temporal Cloud account ID | `a2dd6` | +| `BillingAccountName` | Temporal Cloud account name | `temporal` | +| `BillingCurrency` | The currency an account is billed in | `USD (cents)` | +| `BillingPeriodEnd` | Exclusive end bound of a billing period | `2024-02-01T00:00:00Z` | +| `BillingPeriodStart` | Inclusive start bound of a billing period | `2024-01-01T00:00:00Z` | +| `ChargeCategory` | Highest-level classification based on how it is billed | `Usage` | +| `ChargeDescription` | Self-contained summary of the charge's purpose | `Actions - Tier 1` | +| `ChargeFrequency` | How often a charge occurs | `Usage-Based` | +| `ChargePeriodEnd` | Time period end for the charge (correlates to data granularity) | `2025-10-01T01:00:00.000Z` | +| `ChargePeriodStart` | Time period start for the charge (correlates to data granularity) | `2025-10-01T00:00:00.000Z` | +| `ContractedCost` | Cost calculated by multiplying `ContractedUnitPrice` and `PricingQuantity` | `100.00` | +| `ContractedUnitPrice` | Agreed-upon unit price for a single pricing unit, inclusive of negotiated discounts | `10.00` | +| `InvoiceID` | ID of the invoice for this billing period | `in_XXXXXXXXXXXXXXXXXXXX` | +| `InvoiceIssuer` | Entity responsible for issuing payable invoices | `stripe` | +| `PricingQuantity` | Volume of a given SKU used or purchased | `10.00` | +| `PricingUnit` | Measurement unit for `PricingQuantity` | `1 Million Actions` | +| `Provider` | Provider of purchased resources or services | `Temporal Technologies` | +| `Publisher` | Publisher of purchased resources or services | `Temporal Technologies` | +| `ResourceID` | Namespace name + Temporal Cloud account ID | `production.a2dd6` | +| `ResourceName` | Namespace name + Temporal Cloud account ID | `production.a2dd6` | +| `ResourceType` | Type of resource the charge applies to | `Namespace` | +| `ServiceCategory` | Highest-level classification based on core function | `Temporal Cloud` | +| `ServiceName` | Offering that can be purchased from a provider | `Temporal Cloud` | +| `ServiceSubcategory` | Secondary classification based on core function | `Actions` | +| `SKUID` | Unique identifier for a specific SKU | `essentials-actions` | +| `SKUMeter` | Functionality being metered by a particular SKU | `Actions` | +| `Tags` | Provider and customer defined tags associated with resources | `{"$tmprl_project":["project-id"],"namespace-tag-key":["namespace-tag-value"]}` | + +**Key schema notes:** + +- `ResourceID` is `namespace_name.account_id` (e.g., `production.a2dd6`), not just the namespace name. +- `BillingCurrency` values are in cents (e.g., `USD (cents)`). +- The cost column is `ContractedCost`, not `Cost` or `TotalCost`. diff --git a/plugins/temporal/skills/temporal-ops/references/ops/cloud-capacity.md b/plugins/temporal/skills/temporal-ops/references/ops/cloud-capacity.md new file mode 100644 index 0000000..6d4fa5c --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/ops/cloud-capacity.md @@ -0,0 +1,245 @@ +# Temporal Cloud Capacity Modes + +Quick-reference for Temporal Cloud capacity configuration: On-Demand vs Provisioned modes, APS/RPS/OPS definitions, TRUs, CLI commands, default limits, throttling, and APS management best practices. + +--- + +## APS, RPS, and OPS + +These three measures apply at different layers. Do not conflate them. + +| Measure | Full Name | Scope | What It Measures | +|---------|-----------|-------|------------------| +| **APS** | Actions Per Second | Temporal Cloud Namespace | Rate of billable Actions (starting/signaling Workflows, scheduling Activities, etc.) | +| **RPS** | Requests Per Second | Temporal Service (Cloud and self-hosted) | Rate of gRPC requests to the Temporal Service | +| **OPS** | Operations Per Second | Temporal Cloud | Anything a user does directly, or Temporal does on behalf of the user, that produces load on Temporal Server | + +APS is the higher-level, primary limit for Namespaces. RPS and OPS are lower-level measures to control and balance request rates at the service level. + +--- + +## What Counts as an Action? + +An Action is any billable operation within Temporal Cloud. Key categories: + +- **Workflow**: Starting, resetting, Continue-As-New, Child Workflow start, Search Attribute upsert +- **Activity**: Starting, retrying, Heartbeating (only if the heartbeat reaches the Server) +- **Timer**: Timer started (including implicit SDK timers from timeouts) +- **Signal**: Every Signal sent (from Client or Workflow); one Action for Signal-With-Start regardless of whether the Workflow starts +- **Query**: Every Query received by a Worker (`__temporal_workflow_metadata` excluded) +- **Update**: Every accepted or rejected Update +- **Schedule**: Each Schedule execution accrues 3 Actions (2 for Schedule start + 1 for the target Workflow start) +- **Nexus**: Scheduling or canceling a Nexus Operation each counts as 1 Action on the caller Namespace + +Actions during Workflow Replay do **not** count. + +Actions excluded from APS calculations: Export, Capacity-related Actions. + +--- + +## On-Demand Capacity + +Default mode. Namespace capacity scales automatically based on trailing usage. + +### Default limits + +| Measure | Default Limit | +|---------|---------------| +| APS | 500 | +| RPS | 2,000 | +| OPS | 4,000 | + +The limit never falls below the default value. + +### Auto-scaling formula + +Limit = the **greater** of: + +1. Default limit (500 APS) +2. The **lesser** of: + - 4 x APS Mean (over the past 7 days) + - 2 x APS P90 (over the past 7 days) + +**Example**: If your average APS over 7 days is 200 and your P90 is 500: + +- 4 x 200 = 800 +- 2 x 500 = 1,000 +- Lesser of those = 800 +- Greater of 800 vs default 500 = **800 APS limit** + +Under On-Demand you are only charged for the Actions you use. + +--- + +## Provisioned Capacity + +Lets you manually control Namespace limits by requesting Temporal Resource Units (TRUs). + +### Per-TRU rates + +| Measure | Per TRU | +|---------|---------| +| APS | 500 | +| RPS | 1,500 | +| OPS | 4,000 | + +### Valid TRU counts + +**2, 3, 4, 6, 8, 10, 12** -- subject to regional availability. + +TRUs can be adjusted hourly. + +When TRUs are requested, Temporal aims to provision the additional capacity within two minutes. + +For requests in excess of 4 TRUs in regions outside of the US, submit a support ticket to ensure capacity availability. + +### When to use Provisioned Capacity + +- Planned events (promotions, load testing, migrations) +- Unplanned events / usage spikes +- Known but sudden system spikes +- Load testing +- Migrating workloads + +When switching back to On-Demand mode, your APS limit resets to the running average from the last 7 days. If Temporal Support has set a custom limit for your namespace, this limit is persisted across capacity mode changes. + +--- + +## Setting Capacity Modes + +Capacity modes can be set and adjusted by **Global Admin** and **Namespace Admin**. + +### CLI + +Update capacity: + +``` +tcld namespace capacity update \ + --namespace \ + --capacity-mode \ + [--capacity-value ] \ + [--request-id ] \ + [--resource-version ] +``` + +- `--capacity-mode` (`--cm`): `on_demand` for automatic scaling, `provisioned` for fixed allocation. +- `--capacity-value` (`--cv`): throughput value in TRUs. Required and must be greater than 0 when `--capacity-mode` is `provisioned`; ignored for `on_demand`. +- `--request-id`: optional; server assigns one if not specified. +- `--resource-version`: optional; CLI uses the latest version if not set. + +Get current capacity (alias `g`): + +``` +tcld namespace capacity get \ + --namespace +``` + +- `--namespace` (`-n`): required. + +If using API key authentication with `--api-key`, add it directly after `tcld` and before `capacity update`. + +`capacity update` changes both the bill and the throughput ceiling, so propose it +rather than running it: read the current setting with `capacity get` first, and put +the before-and-after mode and TRU count in front of the user. The direction that +causes an incident is downward — lowering TRUs, or switching `provisioned` → +`on_demand` on a Namespace that was provisioned precisely because auto-scaling +could not keep up, throttles production traffic with `RESOURCE_EXHAUSTED` rather +than failing the command. See [Throttling Behavior](#throttling-behavior) and +[../triage/rate-limits.md](../triage/rate-limits.md). + +### UI + +Navigate to the Namespace page in Temporal Cloud UI (`https://cloud.temporal.io/namespaces/`), click **Manage Capacity**, then select On-Demand or Provisioned and configure TRUs via the slider. + +### API + +Call the `UpdateNamespace` API after Namespace creation and define the desired capacity state as part of the capacity spec. + +--- + +## Throttling Behavior + +When your Action rate exceeds your APS (or RPS/OPS) limit, Temporal Cloud throttles requests. + +1. **Priority-based**: Low-priority operations throttled first; higher-priority operations (`StartWorkflowExecution`, `SignalWorkflowExecution`, `UpdateWorkflowExecution`) continue when possible. +2. **Not instantaneous**: Usage may briefly exceed your limit before throttling takes effect. +3. **`ResourceExhausted` errors**: Server returns a `ResourceExhausted` gRPC error; SDK clients automatically retry based on the default gRPC retry policy. +4. **Potential failure**: If throttling persists beyond the SDK's retry limit, client calls fail -- work **can** be lost if you do not handle these failures. + +> For diagnosis of `RESOURCE_EXHAUSTED` errors in triage context, see `../triage/rate-limits.md`. + +**Best practices for handling throttling**: + +- Log any failed `StartWorkflowExecution`, `SignalWorkflowExecution`, or `UpdateWorkflowExecution` calls (including payloads) so you can retry or backfill later. +- Set up Cloud metrics (`temporal_cloud_v0_resource_exhausted_errors`) to alert when throttling occurs. +- Alert at 70-80% utilization to give time to react. + +--- + +## Other Namespace-Level Limits + +| Limit | Default | Notes | +|-------|---------|-------| +| Namespaces per account | 10 (auto-increases) | | +| Schedules RPS | 10 per second | Use jitter to avoid thundering herd | +| Visibility API | 30 calls per second | Not configurable | +| Certificates | 32 KB or 16 certificates (whichever first) | | +| Concurrent Task pollers | 20,000 Activity + 20,000 Workflow Task | Per Namespace | +| Retention period | 30 days default, configurable 1-90 days | | +| Batch jobs | 1 concurrent per Namespace, max 50 Executions/sec | | + +--- + +## APS Management Best Practices + +### Common reasons for hitting APS limits + +1. **Bursty traffic**: Calendar-driven spikes, event-driven surges, recovery thundering herds, timer storms, retry storms. +2. **Cascading Workflows and fan-out**: Parent Workflows spawning many Child Workflows; each child's full action lifecycle counts against the Namespace APS. +3. **Human-in-the-loop at scale**: Long-running Workflows with frequent Queries from UIs for state polling. +4. **Many small Activities**: 1,000 single-record Activities vs 10 batched Activities -- each Activity adds Action overhead. +5. **Multiple use cases in one Namespace**: APS limit is per Namespace, so multiple workloads compound. + +### Mitigation strategies + +- **Stagger and jitter**: Use Schedule jitter and Start Delay to smooth batch starts. +- **Batch Activities**: Combine multiple external calls in a single Activity; process data in chunks. +- **Reduce fan-out depth**: Evaluate whether Child Workflows are necessary; limit fan-out size; flatten deeply nested hierarchies. +- **Push state, don't poll**: Avoid polling patterns where UIs constantly Query Workflow state; push state changes to a database that UIs read. +- **Use longer monitoring intervals**: Check SLAs every 30 minutes instead of every 1 minute; consolidate Timers. +- **Separate Namespaces per use case**: Plan for one set of Namespaces (per environment) per use case. +- **Provision TRUs for known spikes**: Pre-provision before planned events, deprovision after. + +### Automation for TRU scaling + +- Use the Cloud Ops API, Terraform Provider, or `tcld` CLI to programmatically scale capacity. +- Set utilization thresholds (e.g., scale up at 70-80% of limit). +- Schedule capacity changes with Temporal Schedules or Workflows. +- React to upstream leading indicators (queue depth, campaign start) to trigger capacity changes proactively. + +### Monitoring + +- Track `temporal_cloud_v0_resource_exhausted_errors` to detect throttling events. +- Alert at 70-80% utilization. +- Analyze historical patterns to decide between reactive TRU provisioning and proactive automation. +- For Provisioned Namespaces, on-demand envelope metrics show what limits would be under On-Demand mode. + +--- + +## Quick Reference + +| Question | Answer | +|----------|--------| +| Default APS (On-Demand) | 500 | +| Default RPS (On-Demand) | 2,000 | +| Default OPS (On-Demand) | 4,000 | +| APS per TRU | 500 | +| RPS per TRU | 1,500 | +| OPS per TRU | 4,000 | +| Valid TRU counts | 2, 3, 4, 6, 8, 10, 12 | +| TRU provisioning time | Within 2 minutes | +| On-Demand scaling window | Past 7 days | +| On-Demand formula | lesser of 4 x APS Mean or 2 x APS P90 | +| Who can change capacity | Global Admin, Namespace Admin | +| CLI commands | `tcld namespace capacity get`, `tcld namespace capacity update` | +| Throttling error | `ResourceExhausted` gRPC error | diff --git a/plugins/temporal/skills/temporal-ops/references/ops/cloud-certs.md b/plugins/temporal/skills/temporal-ops/references/ops/cloud-certs.md new file mode 100644 index 0000000..c79670b --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/ops/cloud-certs.md @@ -0,0 +1,367 @@ +# Temporal Cloud -- mTLS Certificate Management + +Quick-reference for generating, uploading, filtering, and rotating mTLS certificates on Temporal Cloud using `tcld`. + +> For diagnosing certificate errors (x509 failures, TLS handshake errors), see the triage file `../triage/certificates.md`. + +--- + +## Namespace ID format + +All `--namespace` flags accept a **Namespace ID** in the format `.` (e.g., `your-namespace.a1b2c`). + +If `--namespace` is omitted, the value of the environment variable `$TEMPORAL_CLOUD_NAMESPACE` is used. + +--- + +## 1. Generating certificates with tcld + +### 1a. Generate a CA certificate + +```bash +tcld generate-certificates certificate-authority-certificate \ + --organization \ + --validity-period \ + --ca-certificate-file .pem \ + --ca-key-file .key +``` + +Alias for the subcommand: `ca` + +| Flag | Alias | Purpose | +|---|---|---| +| `--organization` | `--org` | Organization name for the certificate | +| `--validity-period` | `-d` | Duration in `d/h` format (e.g. `30d10h`) | +| `--ca-certificate-file` | `--ca-cert` | Output path for the `.pem` CA certificate | +| `--ca-key-file` | `--ca-key` | Output path for the `.key` private key | +| `--rsa-algorithm` | `--rsa` | Use 4096-bit RSA instead of ECDSA P-384 (disabled by default) | + +Default key algorithm: ECDSA P-384. + +A CA certificate generated by tcld has a maximum duration of 1 year (`-d 1y`) and a minimum duration of 7 days. + +**Shorthand example** (from docs): + +```bash +tcld gen ca --org temporal -d 1y --ca-cert ca.pem --ca-key ca.key +``` + +### 1b. Generate an end-entity (leaf) certificate + +```bash +tcld generate-certificates end-entity-certificate \ + --organization \ + --validity-period \ + --ca-certificate-file .pem \ + --ca-key-file .key \ + --certificate-file .pem \ + --key-file .key +``` + +Alias for the subcommand: `leaf` + +| Flag | Alias | Purpose | +|---|---|---| +| `--organization` | `--org` | Organization name | +| `--organization-unit` | _(none)_ | Optional OU name | +| `--common-name` | _(none)_ | Optional common name | +| `--validity-period` | `-d` | Duration in `d/h` format | +| `--ca-certificate-file` | `--ca-cert` | Path to the signing CA `.pem` | +| `--ca-key-file` | `--ca-key` | Path to the signing CA `.key` | +| `--certificate-file` | `--cert` | Output path for the leaf `.pem` | +| `--key-file` | `--key` | Output path for the leaf `.key` | + +End-entity certificate must expire before its root CA certificate. + +**Shorthand example** (from docs): + +```bash +tcld gen leaf --org temporal -d 364d --ca-cert ca.pem --ca-key ca.key --cert client.pem --key client.key +``` + +--- + +## 2. CA certificate requirements + +CA certificates uploaded to Temporal Cloud must meet all of the following: + +- X.509v3 +- Each certificate must be a root certificate or issued by another certificate in the bundle +- Must include `CA: true` +- Cannot be a well-known CA (e.g. DigiCert, Let's Encrypt) unless certificate filters are also specified +- Signing algorithm: RSA or ECDSA with SHA-256 or stronger (SHA-1 and MD5 rejected) +- Cannot be generated with a passphrase +- Bundle limit: up to 16 CA certificates, max 32 KB payload before base64 encoding + +End-entity certificates must include `CA: false` and Digital Signature key usage. + +Each certificate in the chain (from end-entity to root) must have a unique Distinguished Name. Distinguished Names are not case sensitive. + +--- + +## 3. Uploading CA certificates to a Namespace + +### 3a. Add a CA certificate (appends to existing) + +```bash +tcld namespace accepted-client-ca add \ + --namespace . \ + --ca-certificate-file +``` + +Alias: `a` + +| Flag | Alias | Notes | +|---|---|---| +| `--ca-certificate` | `-c` | Base64-encoded CA certificate PEM string | +| `--ca-certificate-file` | `-f` | Path to CA certificate PEM file | +| `--namespace` | `-n` | Namespace ID | +| `--resource-version` | `-v` | ETag; uses latest if omitted | +| `--request-id` | `-r` | Request identifier for async op | + +If both `--ca-certificate` and `--ca-certificate-file` are specified, only `--ca-certificate` is used. + +### 3b. Set CA certificates (replaces all existing) + +```bash +tcld namespace accepted-client-ca set \ + --namespace . \ + --ca-certificate-file +``` + +Alias: `s` + +Same flags as `add` (`--ca-certificate` / `--ca-certificate-file`, `--namespace`, `--resource-version`, `--request-id`). + +### 3c. List current CA certificates + +```bash +tcld namespace accepted-client-ca list \ + --namespace . +``` + +Alias: `l` + +### 3d. Remove a CA certificate + +```bash +tcld namespace accepted-client-ca remove \ + --namespace . \ + --ca-certificate-file +``` + +Alias: `r` + +Removal can target by certificate content or fingerprint: + +| Flag | Alias | Notes | +|---|---|---| +| `--ca-certificate` | `-c` | Base64-encoded PEM string | +| `--ca-certificate-file` | `-f` | Path to PEM file | +| `--ca-certificate-fingerprint` | `--fp` | Certificate fingerprint (takes precedence; if set, cert/file flags are ignored) | +| `--all` | _(none)_ | Remove all CA certificates; cannot be combined with the cert/file/fingerprint flags; blocked when auth method is `mtls` or `api_key_or_mtls` | + +--- + +## 4. Certificate filters + +Certificate filters restrict which end-entity certificates can connect, based on DN fields. + +A filter can include any combination (at least one) of: + +- `commonName` +- `organization` +- `organizationalUnit` +- `subjectAlternativeName` + +Maximum 25 certificate filters per Namespace. + +Matching is case-insensitive. A single `*` wildcard is allowed at the beginning or end (but not both) of a value. A bare `*` is not valid. + +If a well-known CA certificate is configured, you cannot clear certificate filters. + +JSON format for filter definitions: + +```json +{ "filters": [ { "commonName": "test1" } ] } +``` + +### 4a. Import (set) certificate filters + +Replaces any existing filters with the ones provided. + +```bash +tcld namespace certificate-filters import \ + --namespace . \ + --certificate-filter-file +``` + +Alias: `imp` + +| Flag | Alias | Notes | +|---|---|---| +| `--certificate-filter-file` | `--file`, `-f` | Path to JSON file | +| `--certificate-filter-input` | `--input`, `-i` | Inline JSON string | + +If both `--certificate-filter-file` and `--certificate-filter-input` are specified, the command returns an error. + +### 4b. Add certificate filters (appends) + +```bash +tcld namespace certificate-filters add \ + --namespace . \ + --certificate-filter-file +``` + +| Flag | Alias | Notes | +|---|---|---| +| `--certificate-filter-file` | `-f`, `--file` | Path to JSON file | +| `--certificate-filter-input` | `-i`, `--input` | Inline JSON string | + +### 4c. Export (view) current certificate filters + +```bash +tcld namespace certificate-filters export \ + --namespace . \ + --certificate-filter-file +``` + +Alias: `exp` + +### 4d. Clear all certificate filters + +```bash +tcld namespace certificate-filters clear \ + --namespace . +``` + +Alias: `c` + +Caution: clearing filters allows any client certificate that chains up to a configured CA certificate to connect. + +--- + +## 5. Certificate rotation procedure (zero-downtime) + +This is the rollover process documented for both UI and tcld. + +### Using tcld + +1. Create a single PEM file containing both old and new CA certificates concatenated: + + ``` + -----BEGIN CERTIFICATE----- + ... old CA cert ... + -----END CERTIFICATE----- + -----BEGIN CERTIFICATE----- + ... new CA cert ... + -----END CERTIFICATE----- + ``` + +2. Upload the bundle: + + ```bash + tcld namespace accepted-client-ca set --ca-certificate-file + ``` + +3. Monitor traffic to the old certificate until it ceases. + +4. Create a file containing only the new CA certificate. + +5. Run `set` again with the new-only file to remove the old CA: + + ```bash + tcld namespace accepted-client-ca set --ca-certificate-file + ``` + +### Using the Cloud UI + +1. Navigate to **Namespaces** > select Namespace > **Edit** > **Authentication**. +2. Scroll past the existing certificate's `-----END CERTIFICATE-----` and paste the new PEM block on the next line. +3. **Save**. +4. Wait until all Workers are using the new certificate. +5. Return to **Edit** > **Authentication**, delete the old certificate, and **Save**. + +--- + +## 6. Namespace auth method: mTLS vs API keys + +When creating a Namespace, `--auth-method` selects the authentication mode. + +Valid values: `mtls`, `api_key`, `restricted`, `api_key_or_mtls` + +- `mtls` (default): requires `--ca-certificate` or `--ca-certificate-file` +- `api_key`: no certificate flags needed +- `api_key_or_mtls`: accepts both authentication methods simultaneously (requires flexible auth to be enabled; contact [Temporal Support](https://docs.temporal.io/cloud/support#ticketing) to enable it) + +```bash +tcld namespace create \ + --namespace . \ + --region us-east-1 \ + --auth-method api_key +``` + +Creating with mTLS: + +```bash +tcld namespace create \ + --namespace . \ + --region us-east-1 \ + --ca-certificate-file ca.pem +``` + +Certificate filters can optionally be set at create time via `--certificate-filter-file` or `--certificate-filter-input`. + +--- + +## 7. Handling a compromised end-entity certificate + +Temporal does not support or check certificate revocation lists (CRLs). + +Recommended approach: use short-lived end-entity certificates so a compromised one expires quickly. + +If immediate action is needed: + +1. If using certificate filters, set filters to block the compromised certificate. +2. Otherwise, generate a new CA certificate. +3. Deploy the new CA alongside the existing one (so existing end-entity certs continue working). +4. Regenerate all end-entity certificates from the new CA. +5. Remove the old (compromised) CA certificate from the Namespace. +6. Monitor audit logs for unauthorized access. + +--- + +## 8. Configuring clients with end-entity certificates + +Use the `temporal` CLI with these flags: + +```bash +temporal \ + --tls-ca-path \ + --tls-cert-path \ + --tls-key-path \ + --tls-server-name +``` + +SDK-specific connection guides: + +- Go: `/develop/go/client/temporal-client#connect-to-temporal-cloud` +- Java: `/develop/java/client/temporal-client#connect-to-temporal-cloud` +- Python: `/develop/python/client/temporal-client#connect-to-temporal-cloud` +- TypeScript: `/develop/typescript/client/temporal-client#connect-to-temporal-cloud` +- .NET: `/develop/dotnet/client/temporal-client#connect-to-temporal-cloud` +- PHP: `/develop/php/client/temporal-client#connect-to-a-dev-cluster` + +For Java SDK: convert the private key from PKCS1 to PKCS8 format: + +```bash +openssl pkcs8 -topk8 -inform PEM -outform PEM -in .key -out .pkcs8.key -nocrypt +``` + +--- + +## 9. Expiration notifications + +Temporal Cloud sends email notifications 15 days before certificate expiration. + +An expired root CA invalidates all downstream certificates. An expired end-entity certificate prevents Temporal Clients from connecting or starting Workflow Executions. Workers with expired certs will either stall indefinitely or cause Workflow timeouts. diff --git a/plugins/temporal/skills/temporal-ops/references/ops/cloud-connectivity.md b/plugins/temporal/skills/temporal-ops/references/ops/cloud-connectivity.md new file mode 100644 index 0000000..4c4acf8 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/ops/cloud-connectivity.md @@ -0,0 +1,191 @@ +# Cloud Connectivity + +Quick-reference for private connectivity (AWS PrivateLink / GCP PSC) and Connectivity Rules in Temporal Cloud. + +Citations use stable doc anchors (slug + heading), e.g. `/cloud/connectivity#connectivity-rules` — public URL fragments, not line numbers. + +--- + +## Private Connectivity + +Temporal Cloud supports private connectivity via **AWS PrivateLink** or **GCP Private Service Connect (PSC)** in addition to public internet endpoints. + +Namespace access is always authenticated via API keys or mTLS regardless of connectivity method. + +### Three-step setup process + +1. **Set up the private connection** from your VPC to the region where the Namespace is located. +2. **Update private DNS and/or client configuration** to use the private connection. Activating private connectivity does not change Namespace or Regional Endpoints automatically — clients keep resolving public addresses until you do this. +3. **Create a Connectivity Rule** (required for GCP PSC, optional for AWS PrivateLink) and attach it to the target Namespace(s). + +### AWS PrivateLink key facts + +- The PrivateLink endpoint **must be in the same region** as the Namespace (or, for HA, the same region as one of the replicas). Cross-region endpoints are not supported. +- PrivateLink endpoint services are **regional** -- individual Namespaces do not use separate services. +- Security group must accept **TCP ingress on port 7233**. +- The VPC endpoint can take up to 10 minutes to reach `Available`; configure private DNS or direct VPCE targeting only after that. +- **Direct VPCE targeting** (without per-Namespace DNS): point clients at the VPC Endpoint DNS name and set `ServerName` / SNI override to the Namespace Endpoint. Works with HA too — each Worker targets its own region's VPC Endpoint (different VPCE address per region), while `ServerName` stays the Namespace Endpoint. It does not follow the CNAME, so Temporal Cloud's cross-region forwarding is what keeps passive-region Workers productive and preserves the path across a failover. + +### GCP Private Service Connect key facts + +- PSC endpoint must be in the **same region** as the Namespace (or, for HA, the same region as one of the replicas). +- PSC endpoint stays in **`Pending`** until a matching Connectivity Rule is created -- the Connectivity Rule is the approval step (no separate producer-side approval). +- Automatic Failover via Temporal Cloud DNS is **not currently supported** with GCP PSC; manual worker updates are required on failover. + +### Client configuration without private DNS + +If you cannot set up private DNS, update two settings in your Temporal clients: + +1. Set endpoint server address to the PrivateLink DNS name or PSC IP address, port `7233`. +2. Set the TLS server name override (depends on auth method): + +| Auth method | TLS server name | +|---|---| +| mTLS (single-region) | Namespace Endpoint, e.g. `my-namespace.my-account.tmprl.cloud` | +| API key (single-region) | Regional API endpoint, e.g. `us-east-1.aws.api.temporal.io` (or `us-central1.gcp.api.temporal.io`) | +| Multi-region (mTLS or API key) | Active region endpoint, e.g. `aws-us-east-1.region.tmprl.cloud` | + +Using the wrong TLS server name with API-key auth over PrivateLink/PSC fails the handshake with `connection reset by peer` even though `nc` shows the port open. + +### Control plane connectivity + +- The control plane (`saas-api.tmprl.cloud`) is accessible via public internet and optionally via AWS PrivateLink. Private connectivity does **not** block public internet access — the control plane is always reachable publicly. +- Control plane PrivateLink is in `us-west-2`, service name `com.amazonaws.vpce.us-west-2.vpce-svc-0c57a5930b6f6be0e`. The endpoint ships its own private DNS name, so clients can use it without configuring private DNS (enable the VPC's `Enable DNS hostnames` and `Enable DNS support`). +- Hostnames by surface: `saas-api.tmprl.cloud` for Terraform / `tcld` / Cloud Ops API; `web.onboarding.tmprl.cloud` and `web.saas-api.tmprl.cloud` for the Web UI. +- The PrivateLink service is exposed only in `us-west-2`; reach it from another region via a `us-west-2` VPC Endpoint plus VPC Peering. + +--- + +## Connectivity Rules + +Connectivity Rules restrict the network paths that can reach a Namespace. They are enforced by Temporal Cloud and do not create or modify the underlying network connection. + +### Default behavior + +A Namespace with **zero** Connectivity Rules is reachable over the public internet and any private connections already configured to the region. + +When one or more rules are attached, Temporal Cloud **immediately blocks** any traffic that does not match a rule. + +The Web UI is **not** subject to connectivity rule enforcement — it stays reachable over the public internet even on a private-only Namespace. + +### When you need a Connectivity Rule + +| Provider | Required? | Why | +|---|---|---| +| AWS PrivateLink | Optional -- add only to enforce private-only access | PrivateLink becomes usable when VPC endpoint is `Available` without any rule | +| GCP PSC | **Required** | PSC endpoint stays `Pending` until a matching rule is created | + +### Rule parameters + +**Public rule**: Only **one public rule per account**. + +**AWS PrivateLink private rule** requires: +- `--connection-id`: VPC endpoint identifier (`vpce-...` value), not the endpoint service or DNS name. +- `--region`: Region prefixed with `aws-` (e.g. `aws-us-east-1`). Must match Namespace region. + +**GCP PSC private rule** requires: +- `--connection-id`: PSC connection identifier (numeric string, e.g. `1234567890123456789`). +- `--region`: Region prefixed with `gcp-` (e.g. `gcp-us-east1`). Must match Namespace region. +- `--gcp-project-id`: GCP project where the PSC connection was created. + +> **Connectivity Rules cannot be updated in place.** To change a rule, delete it, create a new one with the desired parameters, and re-attach it to every Namespace that used it. Creating a second public rule alongside an existing one returns an error. + +### Permissions and limits + +- Only **Account Admins and Account Owners** can create/manage connectivity rules (visible to Account Developers and above). +- Default: 5 private rules per Namespace, 50 private rules per account. Contact support to raise limits. + +### tcld connectivity-rule commands + +Alias: `cr`. + +Create a private rule (AWS): + +```bash +tcld connectivity-rule create --connectivity-type private --connection-id "vpce-00939a7ed9EXAMPLE" --region "aws-us-east-1" +``` + +Create a private rule (GCP): + +```bash +tcld connectivity-rule create --connectivity-type private --connection-id "1234567890" --region "gcp-us-central1" --gcp-project-id "my-project-123" +``` + +Create a public rule (once per account): + +```bash +tcld connectivity-rule create --connectivity-type public +``` + +Other subcommands: + +| Subcommand | Purpose | +|---|---| +| `tcld connectivity-rule get --connectivity-rule-id ` | Get a rule | +| `tcld connectivity-rule delete --connectivity-rule-id ` | Delete a rule | +| `tcld connectivity-rule list` | List all rules (optionally filter by `--namespace`) | + +`--connectivity-type` values: `private`, `public`. + +### Attaching rules to a Namespace + +> ⚠️ **Attaching a rule is destructive to existing access.** Once any Connectivity Rule is set, the Namespace is reachable **only** via the connections named in its rules. Removing a rule that workers are using interrupts their traffic. To migrate without lockout: attach a public rule alongside the private rules, move all workers onto private connections, then remove the public rule. + +```bash +tcld namespace set-connectivity-rules \ + --namespace "my-namespace.abc123" \ + --connectivity-rule-ids "rule-id-1" \ + --connectivity-rule-ids "rule-id-2" +``` + +Alias: `tcld n scrs`. + +Rules are attached **as a set** -- to remove one rule while keeping others, re-specify only the rules to keep. + +Remove all rules (makes Namespace public again): + +```bash +tcld namespace set-connectivity-rules --namespace "my-namespace.abc123" --remove-all +``` + +Rules can also be set at Namespace creation time with `--connectivity-rule-ids`: + +```bash +tcld namespace create \ + --namespace test-namespace.a1b2c \ + --region us-east-1 \ + --auth-method api_key \ + --connectivity-rule-ids \ + --connectivity-rule-ids +``` + +View rules for a Namespace: + +```bash +tcld connectivity-rule list -n "my-namespace.abc123" +``` + +Or view them as part of `tcld namespace get`. + +--- + +## Troubleshooting pointers + +### PSC endpoint stuck in Pending + +- Most common cause: no Connectivity Rule exists for the connection ID. +- Check that `--connection-id`, `--region`, and `--gcp-project-id` in the Connectivity Rule match the endpoint exactly. +- The endpoint's region must be a supported Temporal Cloud region. + +### PrivateLink TLS handshake fails + +- If using API key auth over PrivateLink/PSC with the wrong TLS server name, the handshake fails with `connection reset by peer` even though `nc` shows the port is open. +- Verify the TLS server name override matches the auth-method table above. + +### Network connectivity check + +```bash +nc -zv 7233 +``` + +For full connectivity diagnosis, see the triage `../triage/connectivity.md` reference. diff --git a/plugins/temporal/skills/temporal-ops/references/ops/cloud-iam.md b/plugins/temporal/skills/temporal-ops/references/ops/cloud-iam.md new file mode 100644 index 0000000..e2a231f --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/ops/cloud-iam.md @@ -0,0 +1,554 @@ +# Cloud IAM Reference + +Quick-reference for Temporal Cloud identity and access management via `tcld`. +Covers API keys, users, user groups, service accounts, account operations, roles, and namespace permissions. + +> For authentication failures during workflow execution, see the triage ladder in `../triage/authentication.md`. + +--- + +## API Key Lifecycle (`tcld apikey`) + +Alias: `ak` + +### Create + +```bash +tcld apikey create --name \ + --description "" \ + --duration # e.g. 24h; ignored if --expiry set; required unless --expiry + # --expiry # e.g. '2023-11-28T09:23:24-08:00' + # --request-id +``` + +| Flag | Alias | Required | Notes | +|------|-------|----------|-------| +| `--name` | `-n` | Yes | Display name of the API key | +| `--description` | `-desc` | No | | +| `--duration` | `-d` | Conditional | Duration from now until expiry. Ignored if `--expiry` is set. Required when `--expiry` is omitted; must be positive (no `0s` default) | +| `--expiry` | `-e` | Conditional | Absolute expiry timestamp (RFC3339). Required when `--duration` is omitted | +| `--request-id` | `-r` | No | Server assigns one if not set | + +To create an API key for a **Service Account**, add `--service-account-id `: + +```bash +tcld apikey create \ + --name \ + --description "" \ + --duration \ + --service-account-id +``` + +### Get + +```bash +tcld apikey get --id +``` + +| Flag | Alias | Required | +|------|-------|----------| +| `--id` | `-i` | Yes | + +### List + +```bash +tcld apikey list +``` + +Alias: `l` + +| Flag | Alias | Required | Notes | +|------|-------|----------|-------| +| `--owner-id` | `-oid` | No | Filter API keys by owner ID | +| `--owner-type` | `-ot` | No | Filter by owner type: `user` \| `service-account` | + +### Delete + +```bash +tcld apikey delete --id +``` + +| Flag | Alias | Required | Notes | +|------|-------|----------|-------| +| `--id` | `-i` | Yes | | +| `--resource-version` | `-v` | No | ETag; uses latest if not set | +| `--request-id` | `-r` | No | Server assigns if not set | + +Deleting a key immediately breaks every Worker, script, and CI job still +presenting it, and the key cannot be restored. Identify what is using the key +before proposing the delete, and confirm with the user rather than deleting +autonomously. During a rotation, prefer `disable` — it produces the same +`UNAUTHENTICATED` failure for callers but can be reversed with `enable` if you +disabled the wrong key. Delete only once the replacement is verified in use. + +### Disable + +```bash +tcld apikey disable --id +``` + +| Flag | Alias | Required | Notes | +|------|-------|----------|-------| +| `--id` | `-i` | Yes | | +| `--resource-version` | `-v` | No | ETag; uses latest if not set | +| `--request-id` | `-r` | No | Server assigns if not set | + +### Enable + +```bash +tcld apikey enable --id +``` + +| Flag | Alias | Required | Notes | +|------|-------|----------|-------| +| `--id` | `-i` | Yes | | +| `--resource-version` | `-v` | No | ETag; uses latest if not set | +| `--request-id` | `-r` | No | Server assigns if not set | + +### Key Rotation Procedure + +1. Create a new key (you may reuse key names). +2. Verify both original and new key function properly. +3. Switch clients to load the new key. +4. Delete the old key after it is no longer in use. + +### API Key Limits + +- Up to **10** non-expired keys per user. +- Up to **20** non-expired keys per Service Account. +- Maximum expiration time: **2 years**. + +--- + +## API Key Connectivity Setup + +To authenticate SDK or CLI connections to Temporal Cloud using an API key: + +### Environment variable approach (recommended) + +```bash +export TEMPORAL_API_KEY= +temporal workflow list \ + --address ..tmprl.cloud:7233 \ + --namespace . +``` + +### tcld authentication + +Pass the key with `--api-key` or set `TEMPORAL_CLOUD_API_KEY` (tcld source/README). +Public docs sometimes say `TEMPORAL_API_KEY` for tcld; that env var is for Temporal CLI/SDKs, not tcld. + +```bash +tcld --api-key apikey list +# or +export TEMPORAL_CLOUD_API_KEY= +tcld apikey list +``` + +### Namespace gRPC endpoint format + +Recommended Namespace Endpoint (Temporal CLI, SDKs, Workers): + +``` +..tmprl.cloud:7233 +``` + +Regional endpoint (`..api.temporal.io:7233`) is an alternate for advanced HA routing, not the default API-key CLI address. + +--- + +## User Management (`tcld user`) + +Alias: `u` + +### Invite + +```bash +tcld user invite \ + --user-email \ + --account-role \ + --namespace-permission = +``` + +| Flag | Alias | Required | Notes | +|------|-------|----------|-------| +| `--user-email` | `-e` | Yes | Can be supplied multiple times | +| `--account-role` | `--ar` | Yes | Case-insensitive: `Admin` \| `Developer` \| `Read` \| `Owner` \| `FinanceAdmin` \| `MetricsRead` | +| `--namespace-permission` | `-p` | No | Format: `namespace=permission-type`. Can be repeated. Permissions: `Admin` \| `Write` \| `Read` | +| `--request-id` | `-r` | No | | + +Example with multiple namespace permissions: +```bash +tcld user invite \ + --user-email \ + --account-role developer \ + --namespace-permission ns1=Admin \ + --namespace-permission ns2=Write \ + --request-id <123456> +``` + +### Get + +```bash +tcld user get --user-email +# or +tcld user get --user-id +``` + +Must set either `--user-email` or `--user-id`. + +### List + +```bash +tcld user list +``` + +| Flag | Alias | Required | Notes | +|------|-------|----------|-------| +| `--namespace` | `-n` | No | Filter: users with permissions to this namespace | +| `--page-token` | `-p` | No | Pagination token | +| `--page-size` | `-s` | No | Defaults to 10 | + +### Delete + +```bash +tcld user delete --user-email +# or +tcld user delete --user-id +``` + +Must set either `--user-email` or `--user-id`. + +| Flag | Alias | Required | Notes | +|------|-------|----------|-------| +| `--user-email` | | Conditional | | +| `--user-id` | | Conditional | | +| `--request-id` | `-r` | No | | +| `--resource-version` | `-v` | No | ETag; uses latest if not set | + +Removes the user's access to the account and every Namespace they held +permissions on. Confirm the identity with the user before running — `--user-email` +is easy to mistype into a valid address belonging to someone else, so prefer +`tcld user list` to resolve the exact `--user-id` first and propose the delete +against that. To reduce a user's access rather than remove them, update their +account role or Namespace permissions instead. + +### Resend Invite + +```bash +tcld user resend-invite --user-email +# or +tcld user resend-invite --user-id +``` + +Alias: `ri` + +Must set either `--user-email` or `--user-id`. + +### Set Account Role + +```bash +tcld user set-account-role --user-email --account-role +# or +tcld user set-account-role --user-id --account-role +``` + +| Flag | Alias | Required | Notes | +|------|-------|----------|-------| +| `--account-role` | `-ar` | Yes | Case-insensitive: `Admin` \| `Developer` \| `Read` \| `Owner` \| `FinanceAdmin` \| `MetricsRead` | +| `--user-email` | `-e` | Conditional | | +| `--user-id` | `--id` | Conditional | | +| `--request-id` | `-r` | No | | +| `--resource-version` | `-v` | No | ETag | + +### Set Namespace Permissions + +```bash +tcld user set-namespace-permissions \ + --user-email \ + --namespace-permission = +``` + +Alias: `snp` + +| Flag | Alias | Required | Notes | +|------|-------|----------|-------| +| `--user-email` | | Conditional | | +| `--user-id` | | Conditional | | +| `--namespace-permission` | `-p` | No | Format: `namespace=permission-type`. Can be repeated. Permissions: `Admin` \| `Write` \| `Read`. Empty removes all namespace permissions | +| `--request-id` | `-r` | No | | +| `--resource-version` | `-v` | No | ETag | + +--- + +## Roles and Permissions + +### Account-Level Roles + +| Role (tcld value) | Notes | +|--------------------|-------| +| `admin` | Global Administrator | +| `developer` | | +| `read` | Read-only | +| `owner` | Account Owner | +| `financeadmin` | Finance Admin | +| `metricsread` | Metrics read access | +| `none` | User-group only; removes account-level role | + +Account-role values are case-insensitive in `tcld user` commands; canonical forms are +`Admin`, `Developer`, `Read`, `Owner`, `FinanceAdmin`, `MetricsRead`. +`tcld user invite` and `tcld user set-account-role` accept all six of these values at the CLI. +Assignment policy is enforced by the server / account permissions, not by tcld: +Global Admin cannot assign Account Owner; Finance Admin is assignable by Account Owner +(and to Service Accounts by Global Admin). Owner changes may also require Support depending +on account policy. `none` is accepted only by `tcld user-group` commands. + +### Namespace-Level Permissions + +| Permission (tcld value) | Notes | +|--------------------------|-------| +| `Admin` | Full namespace control | +| `Write` | | +| `Read` | Read-only | + +Format for `--namespace-permission` flag: `=` +where `` is the full Cloud namespace ID (e.g. `mynamespace.abc123`). + +--- + +## User Groups (`tcld user-group`) + +Alias: `ug` + +### Create + +```bash +tcld user-group create \ + --display-name \ + --account-role \ + --namespace-role - +``` + +| Flag | Alias | Required | Notes | +|------|-------|----------|-------| +| `--display-name` | | Yes | Display name of the group | +| `--account-role` | | Yes | `admin` \| `read` \| `developer` \| `owner` \| `financeadmin` \| `none` | +| `--namespace-role` | `-nr` | No | Repeatable. Format: `-` where role is `admin` \| `read` \| `write`. Example: `mynamespace.abc123-read` | + +Alias: `c` + +**Important**: the `--namespace-role` format uses a **hyphen** separator (`-`), not `=`. +This differs from `tcld user` commands which use `=`. Auto-generated tcld docs currently omit the format string; behavior is defined in tcld `nsRoleToAccess`. + +### Get + +```bash +tcld user-group get --group-id +``` + +Alias: `g` + +### List + +```bash +tcld user-group list +``` + +| Flag | Alias | Notes | +|------|-------|-------| +| `--page-size` | `-s` | Defaults to 10 | +| `--page-token` | `-p` | | + +### Delete + +```bash +tcld user-group delete --group-id +``` + +Alias: `d` + +Every member loses the Namespace permissions the group conferred, which can +revoke access for many people at once. List the members first and confirm the +scope with the user before proposing the delete. To remove one person, use +`remove-users`; to change what the group grants, update its permissions. For a +SCIM-synced group the IdP owns group create/update/delete and membership, so +deleting it here does not change the IdP — offboard in the IdP instead. See +[cloud-saml-scim.md](cloud-saml-scim.md). + +### Add Users + +```bash +tcld user-group add-users --group-id --user-email +``` + +Alias: `au` + +| Flag | Alias | Notes | +|------|-------|-------| +| `--group-id` | `-id` | Required | +| `--user-email` | `-e` | Can be specified multiple times | + +### Remove Users + +```bash +tcld user-group remove-users --group-id --user-email +``` + +Alias: `ru` + +| Flag | Alias | Notes | +|------|-------|-------| +| `--group-id` | `-id` | Required | +| `--user-email` | `-e` | Can be specified multiple times | + +### List Members + +```bash +tcld user-group list-members --group-id +``` + +Alias: `lm` + +### Set Access + +```bash +tcld user-group set-access --group-id \ + --account-role \ + --namespace-role - +``` + +Alias: `sa` + +| Flag | Alias | Required | Notes | +|------|-------|----------|-------| +| `--group-id` | `-id` | Yes | | +| `--account-role` | | Conditional | Required for replace mode. Omit with `--append`/`--remove` (those modes reject setting account role). Values: `admin` \| `read` \| `developer` \| `owner` \| `financeadmin` \| `none` | +| `--namespace-role` | `-nr` | No | Repeatable. Same `-` format as create | +| `--append` | `-a` | No | Append namespace roles instead of replacing all existing roles | +| `--remove` | `-r` | No | Remove the given namespace roles instead of replacing | + +Without `--append` or `--remove`, set-access **replaces** all existing roles and requires `--account-role`. + +--- + +## Service Accounts (`tcld service-account`) + +Service Accounts are non-human identities that use API keys to authenticate. +Use `tcld service-account --help` for a full list of subcommands. + +### Create + +```bash +tcld service-account create -n "" -d "" --ar "" +# Optional: --np "=" +``` + +Returns a `ServiceAccountId` used for subsequent operations. + +### Create Scoped (Namespace-scoped) + +```bash +tcld service-account create-scoped -n "" --np "=" +``` + +Namespace-scoped Service Accounts always have a `Read` Account Role and are restricted to a single namespace. +Cannot be reassigned to a different namespace after creation. + +### List + +```bash +tcld service-account list +``` + +### Get + +```bash +tcld service-account get --service-account-id "" +``` + +Alias: `g`. `--service-account-id` (alias `--id`) is required. + +### Delete + +```bash +tcld service-account delete --service-account-id "" +``` + +Deleting a Service Account automatically deletes all associated API keys. + +The blast radius is not one identity but every Worker and automation +authenticating with any key the Service Account owns, and none of it is +recoverable. Run `tcld apikey list --owner-type service-account --owner-id +` first, report what would be revoked, and confirm with the user before +proposing the delete. + +### Update + +Three update commands exist: + +```bash +# Update name or description +tcld service-account update --id "" -d "" + +# Update account role +tcld service-account set-account-role --id "" --ar "" + +# Update namespace permissions +tcld service-account set-namespace-permissions --id "" -p "=" +``` + +### Namespace-Scoped Lifecycle + +When a namespace is deleted, all associated Namespace-scoped Service Accounts and their API keys are automatically deleted. + +--- + +## Account Operations (`tcld account`) + +Alias: `a` + +### Get + +```bash +tcld account get +``` + +Returns information about the Temporal Cloud account you are logged into. No modifiers. + +### List Regions + +```bash +tcld account list-regions +``` + +Lists all regions where the account can provision namespaces. Alias: `l` + +### Audit Log + +Subcommands for configuring audit log sinks: + +- `tcld account audit-log kinesis` (alias: `k`) -- Kinesis sinks: create, delete, get, list, update, validate +- `tcld account audit-log pubsub` (alias: `ps`) -- Pub/Sub sinks: create, delete, get, list, update, validate + +### Metrics + +```bash +tcld account metrics enable # Enable metrics endpoint +tcld account metrics disable # Disable metrics endpoint +``` + +End-entity certificates must be configured before enabling. +Managed via `tcld account metrics accepted-client-ca` subcommands: `add`, `list`, `set`, `remove`. + +--- + +## Key Differences: `tcld user` vs `tcld user-group` Namespace Permission Format + +| Context | Flag | Format | Example | +|---------|------|--------|---------| +| `tcld user` commands | `--namespace-permission` | `=` | `ns1.abc123=Admin` | +| `tcld user-group` commands | `--namespace-role` | `-` | `mynamespace.abc123-read` | + +The permission values also differ in case: +- `tcld user`: `Admin` | `Write` | `Read` (title case) +- `tcld user-group`: `admin` | `read` | `write` (lower case) diff --git a/plugins/temporal/skills/temporal-ops/references/ops/cloud-migration.md b/plugins/temporal/skills/temporal-ops/references/ops/cloud-migration.md new file mode 100644 index 0000000..a2c15c9 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/ops/cloud-migration.md @@ -0,0 +1,255 @@ +# Cloud Migration + +Three migration paths exist for Temporal workflows: + +| Path | From | To | Downtime | +|---|---|---|---| +| Automated | Self-hosted | Temporal Cloud | Zero | +| Manual | Self-hosted | Temporal Cloud | Varies | +| Within Cloud | Cloud region A | Cloud region B | Zero | + +--- + +## Automated Migration (Self-Hosted to Cloud) + +Pre-release feature. Contact your Temporal account executive before planning. + +Migrate in order of least-critical to most-critical namespace. Start with testing namespaces where downtime is acceptable. + +### Limitations + +- Self-hosted server version 1.22+ required +- History shard counts must be a power of two (e.g. 512, 1024) +- Multiple self-hosted servers with the same cluster name (default `active`) cannot connect to one migration server simultaneously; either migrate one at a time or use separate migration servers +- Cross-namespace commands (`system.enableCrossNamespaceCommands`) must be disabled and related code removed before migration +- Target cloud namespace must be empty (no workflows); cannot combine manual and auto migration +- If Global Namespace was previously enabled: Initial Failover Version must be <= 1,000,000 and Failover Version Increment must be a divisor of 1,000,000 + +### Phase 1: Prepare + +Collect data and submit to Temporal via support ticket: + +**Cluster configuration** (run per cluster): +``` +temporal operator cluster describe --address --output json # server > 1.28.1 +tctl --address admin cluster describe # server <= 1.28.1 +``` + +**Custom search attributes** (must be Cloud-compatible): +``` +temporal operator search-attribute list # Elasticsearch/OpenSearch +temporal operator search-attribute list --namespace="your_ns" # SQL +``` + +**Namespace metrics**: total open/closed workflows, total storage, current retention policy, peak APS + +**mTLS certificates** for S2S Proxy: `openssl verify -CAfile ca.pem client-cert.pem` + +**Cloud namespaces**: create empty target namespaces, apply custom search attributes, adjust rate limits + +**Submit CSV mapping** to Temporal: +``` +cluster_name, cloud_region, source_namespace, cloud_namespace +cluster1, us-east-1, default, use1.nnnnn +``` + +### Phase 2: Setup + +Proceed only after Temporal approves the migration request. + +**S2S Proxy deployment**: +1. Pull latest image from `temporalio/s2s-proxy` Docker Hub +2. Deploy 3 replicas (min 4 CPU, 512 MB memory per replica). Replica count must match cloud side. +3. Proxy initiates outbound TCP 8233 to cloud-side proxy. Ensure firewalls permit this. +4. Verify connectivity: +``` +temporal operator cluster describe --address {proxy-external-address} +``` + +Monitor proxy health via Prometheus endpoint (`proxy-pod-ip:9090/metrics`), in particular `temporal_s2s_proxy_mux_connection_active`. + +**Dynamic configuration changes**: + +```yaml +frontend.keepAliveMaxConnectionAge: + - value: '2h' +``` + +If Global Namespace was never enabled, enable it: set `clusterMetadata.enableGlobalNamespace: true`, `failoverVersionIncrement: 1000000` (coordinate with Temporal), `initialFailoverVersion: <2-99>` (unique per cluster), and `dcRedirectionPolicy.policy: 'all-apis-forwarding'`. + +Restart all services (frontend first, then history, matching, worker) and verify with `temporal operator cluster describe`. + +For server versions 1.22.x-1.23.x, also enable stream-based replication: +```yaml +history.enableReplicationStream: + - value: true +``` + +Verify your persistence/database layer has sufficient CPU and I/O capacity. + +### Phase 3: Test + +Use a non-production namespace that can tolerate data loss. Run a mix of completed, active, and new workflows. Perform a full end-to-end migration. Testing succeeds if all data migrates to Cloud. + +### Phase 4: Initiate + +**Start migration** (Temporal generates the endpoint-id): +``` +tcld migration start --endpoint-id --source-namespace --target-namespace +``` + +Self-hosted namespace is active, cloud namespace is passive. Workflows replicate self-hosted to cloud. + +Billing for the cloud namespace does not begin until migration is confirmed. + +**Monitor progress**: +``` +tcld migration get --id +``` +Also monitor `replication_stream_stuck` metric from self-hosted side. + +**Handover to Cloud**: +``` +tcld migration handover --id --to-replica-id cloud +``` +Cloud becomes active, self-hosted becomes passive. To hand back to self-hosted: +``` +tcld migration handover --id --to-replica-id on-prem +``` + +### Phase 5: Finalize + +Complete client transfer, then validate: confirm worker access to cloud namespaces, verify metrics access, monitor schedule-to-start latency / start vs. completion rate / sync match rate, and plan a worker tuning session (performance may differ). + +**Confirm migration** (final, cannot be undone; halts replication): +``` +tcld migration confirm --id +``` + +**Abort migration** (rolls back without impacting workflows): +``` +tcld migration abort --id +``` + +### Transfer Clients to Cloud + +**Option 1 (recommended)**: Deploy two sets of clients, one pointing to self-hosted and one to Cloud. +1. Cloud clients connect and poll but receive no tasks initially +2. Start migration: self-hosted active, cloud passive. Cloud client requests forward to self-hosted automatically +3. Handover: cloud active, self-hosted passive. Self-hosted client requests forward to cloud automatically +4. Confirm migration: self-hosted clients stop receiving tasks; shut them down + +**Option 2**: Single set of clients, switch endpoint during migration. Risk: if workers are misconfigured during switch, workflows stop making progress. + +### Key Facts + +All workflows migrate by default; for closed workflows you may specify a date range (top speed optimization). Schedules are supported. Cannot split one source namespace into multiple cloud namespaces. Encrypted payloads remain encrypted through migration. + +--- + +## Manual Migration (Self-Hosted to Cloud) + +Use when automated migration requirements are not met or when migration scope is smaller. + +### Client Code Changes + +Update Worker and Starter connection code: +- Add SSL certificate and private key associated with the namespace +- Set gRPC endpoint to `..tmprl.cloud:port` +- Configure `tcld` with the same address, namespace, and certificate + +### Workflow Execution Strategies + +**New workflows**: Once updated client code is deployed, new executions automatically go to Cloud. Maintain the self-hosted client as long as you need to send Signals or Queries to old executions. + +**Running workflows**: +- Short-running: drain (let them complete), then restart on Cloud +- Long-running / continuous: cancel and pass current state to a new workflow on Cloud. Example implementation: `github.com/temporalio/temporal-migration` (Java) + +During live migration, a Signal and Query execute per workflow. The Query API loads the full history into Workers. Ensure self-hosted Worker capacity supports this memory load. + +**Completed workflows**: Execution history cannot be automatically migrated to Cloud via manual migration. Maintain self-hosted access or export JSON for analytics. + +### Considerations When Resuming Workflows + +- **Idempotency**: Determine whether to skip non-idempotent steps when resuming +- **Elapsed time**: Calculate sleep deltas for resumed executions +- **Child workflows**: Pass child state into parent to resume children correctly; parent/child relationship does not carry over +- **Heartbeat state**: Long-running activities relying on heartbeat details will not receive latest details in target namespace +- **Signal handling**: Handle `NotFound` when signaling between workflows; they may resume out of order +- **Duration between awaitables**: Factor elapsed time accuracy for sleeps between awaitables + +### Other Considerations + +- Add mTLS certificate to Cloud namespace +- Metrics differ between self-hosted and Cloud; review Cloud metrics documentation +- Review security and access implications +- Review current APS load with your AE/SA to set appropriate namespace limits + +--- + +## Migrate Within Cloud (Region to Region) + +Uses Temporal Cloud High Availability features. Zero downtime. + +HA features affect pricing. + +### Prerequisites + +- Namespaces using Export must stop Export and reconfigure for the new region before migration +- If workers use API key authentication, update all client code to use the regional endpoint of the new replica + +### Migration Steps + +1. **Add replica** in target region (see available regions and supported multi-region/multi-cloud configurations) +2. **Wait** for the replica to become active. Cloud UI shows a time estimate; namespace admins receive an email on completion. +3. **Update workers** (API key auth only) to use the new region's regional endpoint +4. **Failover** to the new region via Cloud UI +5. **Remove** the original region's replica + +If using API keys for worker authentication, removing the replica requires a support ticket. + +All replica changes are subject to a cooldown period before further changes can be made. + +--- + +## tcld Migration Command Reference + +| Command | Purpose | +|---|---| +| `tcld migration start --endpoint-id --source-namespace --target-namespace ` | Begin migration | +| `tcld migration get --id ` | Check migration status | +| `tcld migration list` (alias `l`) | List all migrations (no flags) | +| `tcld migration handover --id --to-replica-id cloud` | Hand over to Cloud | +| `tcld migration handover --id --to-replica-id on-prem` | Hand back to self-hosted | +| `tcld migration confirm --id ` | Finalize (irreversible) | +| `tcld migration abort --id ` | Abort and roll back | + +`start`, `handover`, `confirm`, and `abort` accept an optional `--request-id`/`-r`; the server assigns one if unset. + +--- + +## Troubleshooting + +**S2S Proxy not connecting** +- Verify outbound TCP 8233 is open through firewalls +- Check `temporal_s2s_proxy_mux_connection_active` metric on `proxy-pod-ip:9090/metrics` +- Confirm replica count matches between self-hosted and cloud-side proxy + +**Replication appears stuck** +- Monitor `replication_stream_stuck` metric from self-hosted side +- For server 1.22.x-1.23.x, ensure `history.enableReplicationStream` is set to `true` and history pods are restarted + +**Cluster name collision** +- Multiple self-hosted servers with the same cluster name (default `active`) cannot use the same migration server. Migrate one at a time or use separate migration servers. + +**Workflows not progressing after handover** +- Option 2 (single client set) risk: if workers are misconfigured during endpoint switch, workflows stop. Verify all workers connect to Cloud before handover. +- Option 1 (dual client set) is recommended to avoid this scenario + +**Manual migration: Query overloading Workers** +- Query API loads full workflow history into Worker memory. Ensure capacity before migrating large numbers of workflows via `ListFilter`. + +**Within-Cloud: Cannot remove replica** +- If using API keys for worker auth, a support ticket is required to remove the replica +- Replica changes are subject to a cooldown period diff --git a/plugins/temporal/skills/temporal-ops/references/ops/cloud-namespace-admin.md b/plugins/temporal/skills/temporal-ops/references/ops/cloud-namespace-admin.md new file mode 100644 index 0000000..ca546f2 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/ops/cloud-namespace-admin.md @@ -0,0 +1,635 @@ +# Cloud Namespace Administration via tcld + +Quick-reference for Cloud namespace lifecycle operations using `tcld namespace`. + +Alias: `n` + +--- + +## Identity formats + +| Concept | Format | Example | +|---|---|---| +| Namespace Name | `` (2-39 chars, lowercase, letters/numbers/hyphens, must start with letter, end with letter or number) | `accounting-production` | +| Account ID | `` (5+ chars) | `123de` | +| Namespace ID | `.` | `accounting-production.123de` | +| Namespace endpoint | `..tmprl.cloud:7233` | `accounting-production.123de.tmprl.cloud:7233` | +| Regional endpoint | `..api.temporal.io:7233` | `us-east-1.aws.api.temporal.io:7233` | + +All `--namespace` / `-n` flags accept the **Namespace ID** (full form), not the short Namespace Name. + +If `--namespace` is omitted, the environment variable `$TEMPORAL_CLOUD_NAMESPACE` is used. + +--- + +## Limits + +- Default account namespace limit: 10 (auto-increases as you create namespaces; for large-scale needs open a support ticket) +- Retention range: 1-90 days +- Max tags per namespace: 10 +- Tag key/value length: 1-63 characters +- Soft limit of 1000 unique tag keys per account +- Max caller Namespaces per Nexus Endpoint Access Policy: 1,000 (support ticket to raise) +- Max Nexus Endpoints per account: 100 (support ticket to raise) + +--- + +## tcld namespace create + +Alias: `c` + +```bash +tcld namespace create \ + --namespace \ + --region \ + --auth-method api_key +``` + +| Flag | Alias | Required | Notes | +|---|---|---|---| +| `--namespace` | `-n` | Yes | Becomes part of the Namespace ID | +| `--region` | `--re` | Yes | One for standard, two for HA. See Regions docs | +| `--auth-method` | | No | `mtls` (default), `api_key`, `restricted`, or `api_key_or_mtls` | +| `--ca-certificate` | `-c` | Conditional | Required if `--auth-method mtls` and no `--ca-certificate-file` | +| `--ca-certificate-file` | `--cf` | Conditional | Path to PEM file | +| `--certificate-filter-file` | `--cff` | No | JSON file defining cert filters | +| `--certificate-filter-input` | `--cfi` | No | JSON string defining cert filters | +| `--cloud-provider` | `--cp` | No | `aws` (default) or `gcp` | +| `--connectivity-rule-ids` | `--ids` | No | Can be specified multiple times | +| `--enable-delete-protection` | `--edp` | No | Default `false` | +| `--endpoint` | `-e` | No | Codec server endpoint (must be HTTPS) | +| `--include-credentials` | `--ic` | No | Include cross-origin credentials for codec server. Default `false` | +| `--pass-access-token` | `--pat` | No | Pass user access token to codec server. Default `false` | +| `--request-id` | `-r` | No | Async operation request ID | +| `--retention-days` | `--rd` | No | Default `30` | +| `--search-attribute` | `--sa` | No | `name=type` format; can repeat. Types: `Bool`, `Datetime`, `Double`, `Int`, `Keyword`, `Text` | +| `--tag` | `--t` | No | `key=value` format; can repeat | +| `--user-namespace-permission` | `-p` | No | `email=permission` format; `Admin`, `Write`, `Read` | + +Example with HA (two regions), tags, and search attributes: + +```bash +tcld namespace create \ + --namespace my-namespace.a1b2c \ + --region us-east-1 \ + --region us-west-2 \ + --auth-method api_key \ + --retention-days 30 \ + --search-attribute "customer_id=Int" \ + --tag "env=production" \ + --user-namespace-permission "user@example.com=Admin" +``` + +--- + +## tcld namespace get + +Alias: `g` + +```bash +tcld namespace get \ + --namespace +``` + +| Flag | Alias | Required | Notes | +|---|---|---|---| +| `--namespace` | `-n` | No | Falls back to `$TEMPORAL_CLOUD_NAMESPACE` | + +Output is JSON by default (no `--format` flag exists). + +--- + +## tcld namespace list + +Alias: `l` + +```bash +tcld namespace list +``` + +| Flag | Alias | Required | Notes | +|---|---|---|---| +| `--page-size` | | No | Namespaces per page; must be >0 and ≤ max page size | +| `--page-token` | | No | Page token from a previous response | + +Returns JSON with a `namespaces` array and `nextPageToken`. + +--- + +## tcld namespace delete + +Alias: `d` + +```bash +tcld namespace delete \ + --namespace +``` + +| Flag | Alias | Required | Notes | +|---|---|---|---| +| `--namespace` | `-n` | Yes | | +| `--request-id` | `-r` | No | | +| `--resource-version` | `-v` | No | ETag; if omitted uses latest | + +Deletion is permanent. All Workflow Executions and Task Queues are removed immediately. Closed Workflow Histories remain until their retention period expires. + +There is no undo. Never run it autonomously. Before proposing it, report what is +in the Namespace — +`temporal workflow count --query 'ExecutionStatus="Running"'` against that +Namespace — and quote the full Namespace ID (`.`) +back to the user for confirmation, since a bare name can match a Namespace in a +different account than the one they mean. If the intent is to stop work rather +than discard the Namespace, that is a Workflow-level or capacity-level change, +not a delete. + +If the delete is refused, the Namespace has delete protection enabled (below). +Treat that as a deliberate decision by whoever provisioned it: report the block +and stop. Do not disable protection and retry unless the user explicitly asks for +that, as a separate step. + +### Delete protection + +Enable via `--enable-delete-protection` / `--edp` at create time. + +Toggle on an existing namespace (`lifecycle`, alias `lc`): + +```bash +tcld namespace lifecycle set \ + --namespace \ + --enable-delete-protection +``` + +Read the current delete-protection state: + +```bash +tcld namespace lifecycle get \ + --namespace +``` + +--- + +## tcld namespace add-region + +Upgrades a namespace to support High Availability by adding a replica region. + +```bash +tcld namespace add-region \ + --namespace \ + --region +``` + +| Flag | Alias | Required | Notes | +|---|---|---|---| +| `--namespace` | `-n` | Yes | | +| `--region` | `--re` | Yes | Region name, e.g. `us-east-1` | +| `--cloud-provider` | | No | `aws` (default) or `gcp` | +| `--request-id` | `-r` | No | | + +Temporal Cloud sends an email alert once the Namespace is ready. + +--- + +## tcld namespace delete-region + +Removes a replica region, disabling HA. Imposes a mandatory 7-day waiting period before re-enabling HA in the same location. + +```bash +tcld namespace delete-region \ + --namespace \ + --region +``` + +| Flag | Alias | Required | Notes | +|---|---|---|---| +| `--namespace` | `-n` | Yes | | +| `--region` | `--re` | Yes | Region to remove | +| `--cloud-provider` | | No | `aws` (default) or `gcp` | +| `--request-id` | `-r` | No | | + +The 7-day wait is what makes this hard to walk back: the Namespace runs +single-region for a week with no failover target, so a removal done to "clean up" +a replica cannot be reversed if the primary degrades in the meantime. Confirm +with the user that they intend to give up HA for at least that long, and check +which region is currently active first — removing the replica is a different +operation from failing back to it. + +--- + +## tcld namespace failover + +Switches a namespace from its primary region to a replica region (requires HA). + +```bash +tcld namespace failover \ + --namespace \ + --region +``` + +| Flag | Alias | Required | Notes | +|---|---|---|---| +| `--namespace` | `-n` | Yes | | +| `--region` | `--re` | Yes | Region to fail over TO | +| `--cloud-provider` | | No | `aws` (default) or `gcp` | +| `--request-id` | `-r` | No | | + +This moves production traffic and is not a diagnostic step — never trigger one to +test whether failover works, and never trigger one while diagnosing a symptom that +has not been traced to the active region. Confirm the target with the user, and +note that `--region` names the region being failed over **to**, not away from. +Once the request returns an operation ID the failover is guaranteed to proceed and +cannot be called back; clients may see a brief window of retryable errors during +handover. After a user-triggered failover Temporal does **not** fail back +automatically. See [../triage/ha-failover.md](../triage/ha-failover.md). + +--- + +## tcld namespace retention + +Alias: `r` + +### retention get + +Alias: `g` + +```bash +tcld namespace retention get \ + --namespace +``` + +### retention set + +Alias: `s` + +```bash +tcld namespace retention set \ + --namespace \ + --retention-days +``` + +| Flag | Alias | Required | Notes | +|---|---|---|---| +| `--namespace` | `-n` | Yes | | +| `--retention-days` | `--rd` | Yes | Range: 1-90 days | + +Lowering retention is a data deletion, not a settings change: closed Workflow +Histories that fall outside the new window stop being retained and cannot be +recovered by setting the value back. Read the current value with `retention get` +and confirm the new number with the user before proposing it; the deletion does not +appear anywhere in the command's own output. Raising retention is safe but does not +resurrect anything already aged out. + +--- + +## tcld namespace auth-method + +Alias: `am` + +Gets or sets the authentication method for an existing namespace. Changing the method can break existing client connections; tcld prompts for confirmation on disruptive changes. + +### auth-method get + +```bash +tcld namespace auth-method get \ + --namespace +``` + +| Flag | Alias | Required | Notes | +|---|---|---|---| +| `--namespace` | `-n` | Yes | | + +### auth-method set + +```bash +tcld namespace auth-method set \ + --namespace \ + --auth-method +``` + +| Flag | Alias | Required | Notes | +|---|---|---|---| +| `--namespace` | `-n` | Yes | | +| `--auth-method` | `--am` | Yes | One of `restricted`, `mtls`, `api_key`, `api_key_or_mtls` | +| `--request-id` | `-r` | No | | +| `--resource-version` | `-v` | No | ETag; latest if omitted | + +--- + +## tcld namespace export (Workflow History Exports) + +Workflow History Export sinks are managed with `tcld namespace export` (alias `es`), under two provider subgroups: `s3` (AWS) and `gcs` (GCP). + +Both subgroups expose the same subcommands: + +| Subcommand | Alias | Purpose | +|---|---|---| +| `create` | `c` | Create a sink (created enabled) | +| `validate` | `v` | Validate sink config without creating it | +| `update` | `u` | Update sink fields or toggle enabled | +| `get` | `g` | Get a sink by name | +| `delete` | `d` | Delete a sink by name | +| `list` | `l` | List sinks | + +### S3 create/validate flags + +| Flag | Alias | Required | +|---|---|---| +| `--sink-name` | | Yes | +| `--role-arn` | | Yes | +| `--s3-bucket-name` | | Yes | +| `--kms-arn` | | No | +| `--region` | `--re` | No | + +### GCS create/validate flags + +| Flag | Alias | Required | +|---|---|---| +| `--sink-name` | | Yes | +| `--service-account-email` | | Yes | +| `--gcs-bucket` | | Yes | + +`update` additionally takes `--enabled` (toggle `true`/`false`) and `--resource-version` / `-v`; provider flags are optional on update. + +`get`, `delete`, and `list` are shared across both subgroups. `get` and `delete` identify the sink with `--sink-name` (`delete` also accepts `--resource-version` / `-v`); `list` accepts `--page-size` and `--page-token`. + +--- + +## tcld namespace update-codec-server + +Alias: `ucs` + +```bash +tcld namespace update-codec-server \ + --namespace \ + --endpoint +``` + +| Flag | Alias | Required | Notes | +|---|---|---|---| +| `--namespace` | `-n` | Yes | | +| `--endpoint` | `-e` | Yes | Must be HTTPS | +| `--pass-access-token` | `--pat` | No | Default `false` | +| `--include-credentials` | `--ic` | No | Default `false` | + +--- + +## tcld namespace update-high-availability + +Alias: `uha` + +```bash +tcld namespace update-high-availability \ + --namespace \ + --disable-auto-failover=true +``` + +| Flag | Alias | Required | Notes | +|---|---|---|---| +| `--namespace` | `-n` | Yes | | +| `--disable-auto-failover` | | No | `true` or `false` (default). Use `--disable-auto-failover=false` to (re-)enable Temporal-managed failover. | + +--- + +## tcld namespace tags + +Alias: `t` + +### tags upsert + +Add new tags or update existing tag values. Alias: `u` + +```bash +tcld namespace tags upsert \ + --namespace \ + --tag "key1=value1" \ + --tag "key2=updated" +``` + +| Flag | Alias | Required | Notes | +|---|---|---|---| +| `--namespace` | `-n` | Yes | | +| `--tag` | `--t` | Yes | `key=value` format; repeatable | +| `--request-id` | `-r` | No | | + +### tags remove + +Remove tags by key. Alias: `rm` + +```bash +tcld namespace tags remove \ + --namespace \ + --tag-key "key1" \ + --tag-key "key2" +``` + +| Flag | Alias | Required | Notes | +|---|---|---|---| +| `--namespace` | `-n` | Yes | | +| `--tag-key` | `--tk` | Yes | Key string; repeatable | +| `--request-id` | `-r` | No | | + +### Tag constraints + +- Allowed characters: lowercase `a-z`, `0-9`, `.`, `_`, `-`, `@` +- Keys must be unique per namespace +- Only Account Admins and Account Owners can create/edit tags + +--- + +## tcld namespace set-connectivity-rules + +Alias: `scrs` + +| Flag | Alias | Required | Notes | +|---|---|---|---| +| `--namespace` | `n` | Yes | | +| `--connectivity-rule-ids` | `ids` | No | Repeatable. `--ids id1 --ids id2` | +| `--remove-all` | | No | Acknowledges removal of all rules, enabling connectivity from any source | + +--- + +## tcld namespace search-attributes + +Alias: `sa` + +### search-attributes add + +Alias: `a` + +```bash +tcld namespace search-attributes add \ + --namespace \ + --search-attribute "YourSearchAttribute1=Text" \ + --search-attribute "YourSearchAttribute2=Double" +``` + +| Flag | Alias | Required | Notes | +|---|---|---|---| +| `--namespace` | `-n` | No | Falls back to `$TEMPORAL_CLOUD_NAMESPACE` | +| `--search-attribute` | `--sa` | Yes | `name=type` format; repeatable. Types: `Bool`, `Datetime`, `Double`, `Int`, `Keyword`, `Text` | +| `--request-id` | `-r` | No | | +| `--resource-version` | `-v` | No | | + +To delete a search attribute, contact Support at support.temporal.io. + +### search-attributes rename + +```bash +tcld namespace search-attributes rename \ + --namespace \ + --existing-name \ + --new-name +``` + +| Flag | Alias | Required | Notes | +|---|---|---|---| +| `--namespace` | `-n` | No | | +| `--existing-name` | `--en` | Yes | | +| `--new-name` | `--nn` | Yes | | +| `--request-id` | `-r` | No | | +| `--resource-version` | `-v` | No | | + +--- + +## tcld namespace accepted-client-ca + +Manages client CA certificates used to verify mTLS connections. Alias: `ca` + +| Subcommand | Alias | Purpose | +|---|---|---| +| `add` | `a` | Add CA certs | +| `list` | `l` | List current CA certs | +| `set` | `s` | Replace all CA certs (used for rollover) | +| `remove` | `r` | Remove specific CA certs | + +All subcommands accept `--namespace` / `-n`, `--request-id` / `-r`, `--resource-version` / `-v`. + +Certificate can be supplied as: +- `--ca-certificate` / `-c` (base64-encoded string) +- `--ca-certificate-file` / `-f` (path to PEM file) + +If both are specified, `--ca-certificate` takes precedence. + +The `remove` subcommand additionally supports `--ca-certificate-fingerprint` / `--fp` for removal by fingerprint. + +### CA certificate rollover procedure + +1. Create a single PEM file with both old and new CA certificate blocks concatenated +2. Run `tcld namespace accepted-client-ca set --ca-certificate-file ` +3. Monitor traffic until old cert usage ceases +4. Run `set` again with only the new certificate + +Do NOT use a CA certificate signed with SHA-1 -- such signatures are rejected. + +`set` and `remove` both change who can connect, and the failure is fleet-wide +rather than gradual: every client presenting a leaf that chained only to a dropped +CA fails its next handshake, and Workers fail closed with a TLS error rather than +degrading. `set` is the sharper of the two because it replaces the entire bundle — +a PEM that omits a CA still in use silently revokes it. Run `list` first, confirm +which CAs are live, and prefer `add` when the goal is to introduce a new CA. +Reserve `set` for steps 2 and 4 of the rollover above, and do not run step 4 until +step 3 has actually shown old-cert traffic stop. Recovery means re-uploading the +dropped CA, so keep the PEM until the rollover is confirmed complete. + +--- + +## tcld namespace certificate-filters + +Manages certificate filters that authorize client certificates based on DN fields. Alias: `cf` + +| Subcommand | Alias | Purpose | +|---|---|---| +| `add` | `a` | Add certificate filters | +| `import` | `imp` | Set (replace all) certificate filters | +| `export` | `exp` | Export current filters to file | +| `clear` | `c` | Clear all filters (allows any client cert that chains to a configured CA) | + +Filter fields (at least one required): `commonName`, `organization`, `organizationalUnit`, `subjectAlternativeName` + +Filter input via `--certificate-filter-file` / `-f` or `--certificate-filter-input` / `-i`. Cannot specify both. + +JSON format: `{ "filters": [ { "commonName": "test1" } ] }` + +Two of these subcommands change access, in opposite directions, and both are worth +proposing rather than running. `import` replaces the whole filter set rather than +appending to it, so it locks out every identity whose cert matched only a filter +the new file omits — the same fleet-wide shape as `accepted-client-ca set`, and the +reason to `export` to a file first and edit that. `clear` fails the other way: with +no filters, **any** client cert that chains to a configured CA is accepted +, so it silently widens access +instead of removing it. Neither direction is what "clear the filters" sounds like; +state which one you mean when you propose it. + +--- + +## tcld nexus endpoint allowed-namespace + +Manages a Nexus Endpoint's Access Policy — the allowlist of caller Namespaces +permitted to use the Endpoint at runtime. Cloud-only: self-hosted authorization +goes through a custom Authorizer plugin instead. For Endpoint CRUD itself, see +[self-hosted-admin.md § Nexus Endpoint Commands](self-hosted-admin.md#nexus-endpoint-commands), +which maps each `temporal operator nexus` verb to its `tcld nexus` equivalent. + +| Subcommand | Purpose | +|---|---| +| `list` | Show the current allowlist | +| `add` | Add caller Namespaces; entries already present are ignored | +| `remove` | Remove caller Namespaces; entries not present are ignored | +| `set` | Replace the entire allowlist | + +All subcommands take `--name` / `-n` (the Endpoint) and, except `list`, +`--namespace` / `-ns`, which is repeatable: + +```bash +tcld nexus endpoint allowed-namespace add \ + --name \ + --namespace \ + --namespace +``` + +**No callers are allowed by default**, not even from the Endpoint's own target +Namespace. The allowlist is empty at create time unless seeded with +`--allow-namespace` (singular, a repeatable flag on `tcld nexus endpoint create`, +not a subcommand). + +`set` replaces the full list, so any entry you don't pass is dropped — revoking +those callers at their next Nexus Operation. Never run it on your own initiative. +Run `list` first, name the exact entries it would drop, and ask the user directly; +run it only once they have approved, and only against that Endpoint. Prefer `add` +when the goal is to grant. Terraform manages the +same field as `allowed_caller_namespaces`, so a `set` against a +Terraform-provisioned Endpoint will be reverted on the next apply — see +[cloud-terraform.md](cloud-terraform.md). + +--- + +## Endpoint and authentication summary + +| Auth method | Endpoint type | Format | +|---|---|---| +| API key or mTLS | Namespace endpoint (recommended) | `..tmprl.cloud:7233` | +| API key or mTLS | Regional endpoint | `..api.temporal.io:7233` | + +- Namespace endpoints auto-route during HA failover -- Workers and Clients do not need endpoint changes +- When using mTLS with a regional endpoint, set `server_name` to the Namespace endpoint value +- Web UI URL: `https://cloud.temporal.io/namespaces/` + +--- + +## Access and permissions + +- Creating a namespace requires Developer, Account Owner, or Global Admin account-level role +- The creator is automatically granted Namespace Admin permission +- Deleting a namespace requires Namespace Admin permission +- Tags: only Account Admins and Account Owners can create/edit + +--- + +## Common anti-patterns + +| Wrong | Right | Why | +|---|---|---| +| `--namespace my-ns` | `--namespace my-ns.a1b2c` | Namespace ID requires account suffix | +| `tcld namespace get --format json` | `tcld namespace get` | No `--format` flag; output is JSON by default | +| `tcld namespace search-attributes create` | `tcld namespace search-attributes add` | Subcommand is `add`, not `create` | +| `tcld namespace update --retention-days 30` | `tcld namespace retention set --retention-days 30` | Retention is its own subcommand tree | +| Short-name endpoint `my-ns:7233` | `my-ns.a1b2c.tmprl.cloud:7233` | Cloud requires full Namespace ID in endpoint | diff --git a/plugins/temporal/skills/temporal-ops/references/ops/cloud-notifications.md b/plugins/temporal/skills/temporal-ops/references/ops/cloud-notifications.md new file mode 100644 index 0000000..6b23dcb --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/ops/cloud-notifications.md @@ -0,0 +1,55 @@ +# Cloud Notifications + +Temporal Cloud sends notifications about system status and important administrative events. + +--- + +## Status page subscriptions + +In the event of an incident, Temporal updates the [Temporal Cloud status page](https://status.temporal.io/). Users can subscribe to updates in their preferred mode (e.g. email, Slack, SMS, etc.) by visiting this page. + +--- + +## Administrative email notifications + +Temporal Cloud sends emails to notify users of important administrative events. + +| Reason for email | Who receives email | +|---|---| +| Certificate Expiring in 15 days | Global Administrator, Namespace Administrator, Account Owner | +| Certificate Expiring in 10 days | Global Administrator, Namespace Administrator, Account Owner | +| Certificate Expiring in 5 days | Global Administrator, Namespace Administrator, Account Owner | +| API Key Expiring in 30 days | Global Administrator, Account Owner, individual user (if API Key has an owner) | +| API Key Expiring in 20 days | Global Administrator, Account Owner, individual user (if API Key has an owner) | +| API Key Expiring in 10 days | Global Administrator, Account Owner, individual user (if API Key has an owner) | +| Sign up credit expiring in 30 days | Account Owner, Finance Administrator | +| Sign up credit expiring in 14 days | Account Owner, Finance Administrator | +| Sign up credit expiring in 7 days | Account Owner, Finance Administrator | +| Sign up credit expiring in 1 day | Account Owner, Finance Administrator | +| Sign up credit is 50% consumed | Account Owner, Finance Administrator | +| Sign up credit is 90% consumed | Account Owner, Finance Administrator | +| Account plan type changed | Global Administrator, Account Owner, Finance Administrator | +| Namespace Failover Completed/Failed | Global Administrator, Namespace Administrator, Account Owner | + +--- + +## Quick reference: notification thresholds + +| Resource | Notification schedule | +|---|---| +| Certificate expiry | 15, 10, 5 days before expiry | +| API Key expiry | 30, 20, 10 days before expiry | +| Sign up credit expiry | 30, 14, 7, 1 day(s) before expiry | +| Sign up credit consumption | 50%, 90% consumed | + +--- + +## Email sender + +To ensure you receive email notifications, configure your junk-email filters to permit email from `noreply@temporal.io`. + +--- + +## Providing feedback + +To provide feedback on notifications or request changes, create a support ticket. diff --git a/plugins/temporal/skills/temporal-ops/references/ops/cloud-ops-api.md b/plugins/temporal/skills/temporal-ops/references/ops/cloud-ops-api.md new file mode 100644 index 0000000..9fa520e --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/ops/cloud-ops-api.md @@ -0,0 +1,130 @@ +# Cloud Ops API + +The Cloud Ops API provides programmatic management of Temporal Cloud control plane resources, including Namespaces, Users, Service Accounts, API Keys, and others. The Temporal Cloud Terraform Provider, `tcld` CLI, and Web UI all use the Cloud Ops API. + +**Stage:** Public Preview. + +--- + +## Endpoints + +| Interface | URL | Notes | +|---|---|---| +| HTTP | `https://saas-api.tmprl.cloud` | Control-plane HTTP API | +| gRPC | `saas-api.tmprl.cloud:443` | Port 443 for gRPC connections | + +PrivateLink / Private Service Connect: configure private DNS for `saas-api.tmprl.cloud` (and `web.saas-api.tmprl.cloud` for the Web UI). + +- HTTP API docs: [saas-api.tmprl.cloud/docs/httpapi.html](https://saas-api.tmprl.cloud/docs/httpapi.html#description/introduction) +- gRPC API source: [github.com/temporalio/cloud-api](https://github.com/temporalio/cloud-api/tree/main) +- gRPC docs on Buf: [buf.build/temporalio/cloud-api](https://buf.build/temporalio/cloud-api/docs/main:temporal.api.cloud.cloudservice.v1#temporal.api.cloud.cloudservice.v1.CloudService) + +The HTTP API supports the same operations as the gRPC API, but is usable via standard HTTP methods. It does **not** allow interaction with individual Workflows or Activities via HTTP. + +--- + +## Prerequisites + +- A Temporal Cloud User or Service Account +- API Key for authentication (owned by that User or Service Account) + +Required roles/permissions vary by RPC (Account Owner, Global Admin, Developer, Finance Admin, Namespace roles, or Custom Roles). Do not assume Account Admin for every operation. + +--- + +## API version header + +Use the `temporal-cloud-api-version` header to select an API version. The backend uses this version to safely mutate resources. Current version: [github.com/temporalio/cloud-api/blob/main/VERSION](https://github.com/temporalio/cloud-api/blob/main/VERSION). + +### gRPC + +gRPC clients must send a `temporal-cloud-api-version` header on every request. + +### HTTP + +For HTTP clients, the version header is optional. If omitted, the HTTP gateway defaults it to the latest API version. This supports simple `curl` usage without looking up a version first. + +For production HTTP automation, still pin an explicit version so behavior does not change when the gateway’s latest version advances. + +--- + +## Go SDK + +For Go developers, use the [Go SDK](https://github.com/temporalio/cloud-sdk-go). Module path is `go.temporal.io/cloud-sdk`. SDK is under active development; pin a version. + +Install: + +```go +go get go.temporal.io/cloud-sdk@latest +``` + +Import: + +```go +import ( + "go.temporal.io/cloud-sdk/cloudclient" +) +``` + +Go samples: [github.com/temporalio/cloud-samples-go](https://github.com/temporalio/cloud-samples-go) +Cloud Ops API client setup: [client/api/client.go](https://github.com/temporalio/cloud-samples-go/blob/main/client/api/client.go) + +--- + +## Compiling protobuf (non-Go languages) + +For languages other than Go, download the gRPC protobufs from the [Cloud Ops API repository](https://github.com/temporalio/cloud-api/tree/main/temporal/api/cloud) and compile them manually. Prefer [Buf](https://buf.build/temporalio/cloud-api) when possible. + +Example using Python (from the `cloud-api` repository root; protos live under `temporal/`): + +```bash +git clone https://github.com/temporalio/cloud-api.git +cd cloud-api +python -m grpc_tools.protoc \ + -I. \ + --python_out=. \ + --grpc_python_out=. \ + $(find temporal -name '*.proto') +``` + +For operation specifics, refer to `cloudservice/v1/request_response.proto` for gRPC messages and `cloudservice/v1/service.proto` for gRPC services. + +--- + +## Use cases + +Common reasons to use the Cloud Ops API: + +- Provision Namespaces per environment or tenant via pipelines. +- Bootstrap new projects by creating users, assigning roles, and creating Namespaces via custom scripts. +- Rotate service account keys on a schedule with a job. +- Audit and report access across orgs with scheduled HTTP requests. + +--- + +## Rate limits + +| Scope | Limit | +|---|---| +| Account-level total | 160 RPS | +| Per user | 40 RPS | +| Per service account | 80 RPS | +| Concurrent long-running mutating ops | 10 (default; subset of create/update/delete RPCs) | + +Rate limits are enforced across all Temporal Cloud control plane operations (tcld, UI, Cloud Ops API). + +Multiple clients used by the same identity (user or service account) share the same rate limit. + +Authentication method (SSO, API keys) does not affect rate limiting. + +### Requesting limit increases + +If your use case requires higher rate limits, submit a support ticket. Provide your current usage patterns, the specific limits you need increased, and a description of your use case. + +--- + +## Connection setup + +- gRPC: `saas-api.tmprl.cloud:443`. +- HTTP: `https://saas-api.tmprl.cloud`. +- Establish a secure connection. See the [Cloud Ops API client setup in Go](https://github.com/temporalio/cloud-samples-go/blob/main/client/api/client.go). diff --git a/plugins/temporal/skills/temporal-ops/references/ops/cloud-saml-scim.md b/plugins/temporal/skills/temporal-ops/references/ops/cloud-saml-scim.md new file mode 100644 index 0000000..b54ae1d --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/ops/cloud-saml-scim.md @@ -0,0 +1,182 @@ +# Cloud SAML and SCIM + +SAML enables single sign-on (SSO) by allowing your identity provider to authenticate users into Temporal Cloud. SCIM automatically creates, updates, and removes users and groups in Temporal Cloud based on changes in your identity provider. + +Temporal-side SAML/SCIM enablement is performed by Temporal Support (internal IAM/cloud-iam), not via `tcld`. + +--- + +## SAML SSO + +SAML 2.0 integration allows you to authenticate users of your Temporal Cloud account using your organization's IdP. This enforces corporate identity policies such as multi-factor authentication (MFA) and password complexity. + +SAML is included in the Business, Enterprise, and Mission Critical plans. + +### Configuration overview + +1. Locate your Temporal Cloud Account Id (5-6 characters after the period in your Namespace Id, e.g., `f45a2`). +2. Configure SAML with your IdP (Microsoft Entra ID or Okta). +3. Share connection information with Temporal and test the connection. + +### Entity identifier format + +``` +urn:auth0:prod-tmprl:ACCOUNT_ID-saml +``` + +Example: + +``` +urn:auth0:prod-tmprl:f45a2-saml +``` + +### Callback URL format + +``` +https://login.tmprl.cloud/login/callback?connection=ACCOUNT_ID-saml +``` + +Example: + +``` +https://login.tmprl.cloud/login/callback?connection=f45a2-saml +``` + +### Sign on URL format (Entra ID only) + +``` +https://cloud.temporal.io/login/saml?connection=ACCOUNT_ID-saml +``` + +### Microsoft Entra ID configuration + +1. Sign in to Microsoft Entra ID. +2. **Manage Microsoft Entra ID** > **View** > **Add > Enterprise application**. +3. **Create your own application** > name it (e.g., `temporal-cloud`) > select **Integrate any other application you don't find in the gallery**. +4. **Getting Started** > **Set up single sign on** > **SAML**. +5. In **Basic SAML Configuration**: + - Set **Identifier (Entity ID)** to the entity identifier above. + - Set **Reply URL (Assertion Consumer Service URL)** to the callback URL above. + - Set **Sign on URL** to the sign on URL above. +6. In **Attributes & Claims**: + - Set **Unique User Identifier (NameID)** to `user.userprincipalname`. + - Set **NameID format** to `emailAddress`. + - Ensure **Email** and **Name** are present under Additional claims. +7. Collect for Temporal: download **Certificate (Base64)** and copy **Login URL**. + +### Okta configuration + +1. Sign in to Okta Admin Console. +2. **Applications** > **Create App Integration** > **SAML 2.0** > **Next**. +3. Name the application (e.g., `temporal-cloud`). +4. In **Configure SAML**: + - Set **Single sign on URL** to the callback URL above. + - Set **Audience URI (SP Entity ID)** to the entity identifier above. + - Set **Name ID format** to `EmailAddress`. + - Set **Attribute Statements**: `email` and `name`. +5. In the **Feedback** section, select **Finish**. +6. On the application page > **Sign On** tab > **View SAML setup instructions**. Copy IdP settings and download the active certificate. + +### Finish SAML configuration + +Create a support ticket with: + +- The sign-in URL from your application +- The X.509 SAML sign-in certificate (PEM or Base64 are both acceptable) +- One or more IdP domains to map to the SAML connection + +The IdP domain is generally the same as your email domain. Multiple IdP domains can be provided. + +After Temporal confirms configuration is complete, go to the Cloud login page, enter your email, choose **Enterprise identity**, then click **Continue**. Do **not** use **Continue with Google** or **Continue with Microsoft** for SAML SSO. + +### SAML-only enforcement + +Enabling SAML alone does **not** block other login methods. SAML-only is a **separate setting**, configured by Temporal Support when requested. + +When SAML-only is enabled, **only** SAML login is allowed — it blocks email+password+MFA and social login (Google/Microsoft). + +--- + +## SCIM user provisioning + +SCIM lets you integrate your identity provider with Temporal Cloud to automate user provisioning and access. Changes in the IdP are reflected in Temporal Cloud: + +- User creation / onboarding +- User deletion / offboarding +- User membership in groups + +SCIM requires SAML. Pricing: + +- **Business:** SCIM is a paid add-on (+$500/mo) +- **Enterprise / Mission Critical:** SCIM included + +### Supported IdP vendors + +- Okta +- Microsoft Entra ID (Azure AD) +- Google Workspace +- OneLogin +- CyberArk +- JumpCloud +- PingFederate +- Any SCIM 2.0-compliant provider + +### Prerequisites + +1. Configure SAML SSO first. +2. Identify your organization's IdP administrator and specify their contact details in the support ticket (that admin completes Directory Sync setup; they do not need broad Cloud admin rights for SCIM setup alone). +3. Submit a support ticket to enable SCIM. + +### Cloud-managed vs SCIM-managed lifecycle + +| Subject | Who manages create/delete | Who manages group membership | Who assigns Temporal roles | +|---------|---------------------------|------------------------------|----------------------------| +| **Cloud-managed users** | Cloud UI/API invite and delete, until user lifecycle management is disabled | Cloud (or SCIM if later synced into groups) | Cloud UI / `tcld` / Terraform | +| **SCIM-managed users** | IdP-owned; offboard in the IdP. Once **user lifecycle management** is disabled, Cloud UI/API can no longer create/delete users | IdP only | Roles still assigned in Cloud (directly or via synced groups) | +| **SCIM-synced groups** | IdP creates/updates/deletes groups | IdP only | Assign roles in Cloud **after** sync (UI / `tcld` / Terraform). IdP does **not** map Temporal roles | + +Whether users can be added or removed from the Cloud UI/API is governed by the account-level **user lifecycle management** setting: while enabled, you can still invite and remove users outside of SCIM; once disabled, user create/delete is IdP-only. Account Roles can always be changed from the Cloud interface. + +### Okta onboarding flow + +1. Temporal Support enables the SCIM integration on your account. Enabling integration automatically emails a configuration link to the Okta administrator. +2. The Okta administrator opens the link, which leads to step-by-step configuration instructions. +3. Once configured, Temporal Cloud begins receiving SCIM messages and automatically onboards/offboards users and groups. + +### Key behaviors + +- User and group change events are applied within **10 minutes** of being made in the IdP. +- User lifecycle management with SCIM also allows user roles to be derived from group membership (roles assigned on the group in Cloud). +- Once a group has been synced in Temporal Cloud, assign roles to the group via the **Cloud UI**, **`tcld`**, or **Terraform**. See [User Group Management](https://github.com/temporalio/tcld?tab=readme-ov-file#user-group-management). +- Disabling SCIM does **not** remove already-synced users or groups. + +--- + +## Access model context + +Access to Temporal Cloud is governed by role-based access control (RBAC). Each access principal has one account-level role and optionally one or more Namespace-level permissions. + +Access principals: + +- **Users** — Individual user accounts +- **User Groups** — Groups for simplified access management +- **Service Accounts** — Automated access +- **Custom Roles** — Customer-defined permission sets assignable to principals + +SAML and SCIM are identity integration features, not access principals. + +Multiple accounts can coexist on the same email domain, each with its own SAML configuration tied to its unique Account ID. However, each email address can only be associated with a single Temporal Cloud account. + +### Troubleshooting + +- **Lost MFA access:** Click **Try another method** on the MFA screen. Enter your recovery code or receive a verification code via email. Then remove the authenticator via **My Profile** > **Password and Authentication** > **Authenticator App** > **Remove method** (not "reset"). +- **Password reset:** If logged in: **My Profile** > **Password and Authentication** > **Reset Password**. If not logged in: enter email, click **Continue**, then **Forgot password**. +- **Email domain changes:** If your organization changed its email domain, create a support ticket with your previous and new email addresses and your Account Id. + +### Common pitfalls + +| Symptom / trap | Cause / fix | +|----------------|-------------| +| Social login (Google/Microsoft) still visible after SAML | SAML ≠ SAML-only. Ask Temporal Support to enable SAML-only if you need to block non-SAML methods | +| Wrong login path | Prefer **Enterprise identity** → **Continue**, or the Sign on URL with `connection=ACCOUNT_ID-saml` (Entra). Do not use Continue with Google/Microsoft for SAML | +| Stale permissions after role/group change | SCIM events apply within ~10 minutes; role/permission visibility in Cloud may lag briefly after that | diff --git a/plugins/temporal/skills/temporal-ops/references/ops/cloud-terraform.md b/plugins/temporal/skills/temporal-ops/references/ops/cloud-terraform.md new file mode 100644 index 0000000..057783d --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/ops/cloud-terraform.md @@ -0,0 +1,294 @@ +# Cloud Terraform Provider + +The Terraform Temporal Cloud provider allows you to use Terraform to manage resources for Temporal Cloud. It uses the Cloud Ops API. + +Once a resource is managed by Terraform, you should only use Terraform to manage that resource. + +Source of truth for resource/data-source schemas: [Terraform Registry](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs). Public docs at docs.temporal.io can lag the provider. + +--- + +## Prerequisites + +- Terraform CLI +- An API Key for authentication + +--- + +## Setup + +Set the `TEMPORAL_CLOUD_API_KEY` environment variable: + +```bash +export TEMPORAL_CLOUD_API_KEY= +``` + +Or pass it directly in the provider block: + +```hcl +provider "temporalcloud" { api_key = "my-temporalcloud-api-key" } +``` + +Required provider configuration: + +```hcl +terraform { + required_providers { + temporalcloud = { + source = "temporalio/temporalcloud" + } + } +} + +provider "temporalcloud" { + # Optional: TEMPORAL_CLOUD_ENDPOINT, TEMPORAL_CLOUD_ALLOWED_ACCOUNT_ID, + # TEMPORAL_CLOUD_ALLOW_INSECURE +} +``` + +--- + +## Supported resources (provider inventory) + +Prefer the [registry resources list](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs) for the full set. Current resources include: + +- `temporalcloud_namespace` +- `temporalcloud_nexus_endpoint` +- `temporalcloud_user` +- `temporalcloud_service_account` +- `temporalcloud_apikey` +- `temporalcloud_connectivity_rule` +- `temporalcloud_custom_role` +- `temporalcloud_group` +- `temporalcloud_group_access` +- `temporalcloud_group_members` +- `temporalcloud_metrics_endpoint` +- `temporalcloud_namespace_export_sink` +- `temporalcloud_namespace_search_attribute` +- `temporalcloud_namespace_tags` +- `temporalcloud_account_audit_log_sink` + +Sections below cover the most common ops workflows. + +--- + +## Namespace management + +Resource: `temporalcloud_namespace` + +Required identity: Account Owner, Global Admin, or Developer Account Role. + +### Create + +```hcl +resource "temporalcloud_namespace" "namespace" { + name = "terraform" + regions = ["aws-us-east-1"] + accepted_client_ca = base64encode(file("ca.pem")) + retention_days = 14 +} +``` + +Key fields: `name`, `regions`, `retention_days`. Auth: at least one of `accepted_client_ca` (mTLS) or `api_key_auth = true`. Both may be enabled. + +```hcl +resource "temporalcloud_namespace" "api_key_ns" { + name = "terraform-api-key" + regions = ["aws-us-east-1"] + api_key_auth = true + retention_days = 14 +} +``` + +### Update + +Terraform automatically recognizes changes in `.tf` files and applies them. For example, changing `retention_days` triggers an update. + +### Delete + +Remove the `temporalcloud_namespace` resource and all dependent resource configurations from your Terraform files and run `terraform apply`. + +Deletion safeguards: + +- Terraform meta-argument: `prevent_destroy` +- Cloud-side: `namespace_lifecycle.enable_delete_protection` (must set to `false` before destroy) + +### Import + +```bash +terraform import temporalcloud_namespace.terraform namespaceid.acctid +``` + +The Namespace ID is in the format `namespaceid.acctid`, available at the top of the Namespace page in the Cloud UI. + +--- + +## Nexus Endpoint management + +Resource: `temporalcloud_nexus_endpoint` + +Required identity: Developer role (or higher) and Namespace Admin permission on the Endpoint's target Namespace. + +### Create + +```hcl +resource "temporalcloud_nexus_endpoint" "nexus_endpoint" { + name = "terraform-nexus-endpoint" + description = "my-service" + worker_target = { + namespace_id = temporalcloud_namespace.target_namespace.id + task_queue = "terraform-task-queue" + } + allowed_caller_namespaces = [ + temporalcloud_namespace.caller_namespace.id, + ] +} +``` + +Key fields: `name`, `description`, `worker_target` (namespace_id, task_queue), `allowed_caller_namespaces`. + +### Import + +Address must be `TYPE.NAME` matching your resource block: + +```bash +terraform import temporalcloud_nexus_endpoint.nexus_endpoint +``` + +--- + +## User management + +Resource: `temporalcloud_user` + +### Limitations + +- Terraform cannot create, update, or delete the Account Owner role. You can import an Account Owner, but not manage the role itself. +- Namespace access must be managed from the User resource, not from the Namespace resource. +- Account Owners and Global Admins automatically gain access to all Namespaces; you cannot specify Namespace access for these roles. +- Manage a specific user in one and only one `.tf` file to avoid overwriting permissions. +- To import a user, you need the User ID (currently not available in the Cloud UI). Fetch it with `tcld user list` or `data.temporalcloud_users`. + +### Create + +```hcl +resource "temporalcloud_user" "global_admin" { + email = "admin@example.com" + account_access = "Admin" +} + +resource "temporalcloud_user" "namespace_admin" { + email = "developer@example.com" + account_access = "Developer" + + namespace_accesses = [{ + namespace_id = temporalcloud_namespace.namespace.id + permission = "Write" + }] +} +``` + +`account_access` is case-insensitive. Allowed values: `owner` (import only), `admin`, `developer`, `read`, `financeadmin`, `none` (SCIM-managed). + +### Import + +```bash +terraform import temporalcloud_user.user 72360058153949edb2f1d47019c1e85f +``` + +--- + +## Service Account management + +Resource: `temporalcloud_service_account` + +Service Accounts use a `name` instead of `email`. + +### Limitations (not identical to users) + +- No Account Owner role for Service Accounts. `account_access` values: `admin`, `developer`, `read`, `financeadmin`, `metricsread`. +- Namespace access is managed on the Service Account resource (not the Namespace resource), same as users. +- Global Admins (`account_access = "admin"`) automatically gain access to all Namespaces; do not set `namespace_accesses` for them. +- Optional `namespace_scoped_access`: binds the SA to a single namespace (namespace assignment immutable after create; permission is mutable). Cannot combine with `account_access` / `namespace_accesses`. +- Manage a specific Service Account in one and only one `.tf` file. + +--- + +## API Key management + +Resource: `temporalcloud_apikey` + +### Create + +```hcl +resource "temporalcloud_apikey" "global_apikey" { + display_name = "admin" + owner_type = "service-account" + owner_id = temporalcloud_service_account.global_service_account.id + expiry_time = "2024-11-01T00:00:00Z" + disabled = false +} +``` + +To access the API Key token, create an output: + +```hcl +output "apikey_token" { + value = temporalcloud_apikey.global_apikey.token + sensitive = true +} +``` + +Retrieve the token: + +```bash +terraform output -json apikey_token +``` + +### Update + +You can update `display_name`, `description`, and `disabled` in place. Changing `owner_id`, `owner_type`, or `expiry_time` forces resource replacement. Updating does not rotate the token. + +### Import + +API keys **cannot** be imported into Terraform. Once created, the API Key secret is not stored and cannot be retrieved. Create a new API Key using Terraform directly instead. + +--- + +## Data sources + +The provider supports many data sources. Prefer the [registry data sources list](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs). Current set includes: + +- `temporalcloud_regions` +- `temporalcloud_namespaces` / `temporalcloud_namespace` +- `temporalcloud_users` / `temporalcloud_user` +- `temporalcloud_service_accounts` / `temporalcloud_service_account` +- `temporalcloud_nexus_endpoints` / `temporalcloud_nexus_endpoint` +- `temporalcloud_connectivity_rule` +- `temporalcloud_account_audit_log_sink` +- `temporalcloud_scim_group` + +### Regions + +```hcl +data "temporalcloud_regions" "regions" {} + +output "regions" { + value = data.temporalcloud_regions.regions.regions +} +``` + +### Namespaces + +The `temporalcloud_namespaces` data source provides access to available Namespaces in the account. + +### Users + +`temporalcloud_users` returns each user's `id` (useful for import when the Cloud UI does not show User IDs). + +--- + +## Resources + +- Terraform Registry: [registry.terraform.io/providers/temporalio/temporalcloud/latest](https://registry.terraform.io/providers/temporalio/temporalcloud/latest) +- GitHub repository: [github.com/temporalio/terraform-provider-temporalcloud](https://github.com/temporalio/terraform-provider-temporalcloud/tree/main) diff --git a/plugins/temporal/skills/temporal-ops/references/ops/recipes.md b/plugins/temporal/skills/temporal-ops/references/ops/recipes.md new file mode 100644 index 0000000..1f5a584 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/ops/recipes.md @@ -0,0 +1,619 @@ +# Ops Recipes + +End-to-end operational playbooks that chain commands from the ops reference files. +For triage-focused walkthroughs, see `../triage/recipes.md`. + +--- + +## (a) Set up a new Cloud namespace with API key auth (end-to-end) + +**When to use:** Provisioning a new Temporal Cloud namespace from scratch, using API key authentication. + +### Step 1: Create the namespace + +```bash +tcld namespace create \ + --namespace . \ + --region \ + --auth-method api_key \ + --retention-days 30 \ + --enable-delete-protection=true +``` + +Requires Developer, Account Owner, or Global Admin account-level role. +The creator is automatically granted Namespace Admin permission. + +Optional flags: +- `--search-attribute "name=type"` (types: `Bool`, `Datetime`, `Double`, `Int`, `Keyword`, `Text`). +- `--tag "key=value"` (up to 10 tags per namespace). +- `--user-namespace-permission "email=permission"` (permissions: `Admin`, `Write`, `Read`). +- Omit `--enable-delete-protection` (or pass `--enable-delete-protection=false`) to skip delete protection; it is disabled by default. + +### Step 2: Create a service account for Workers + +```bash +tcld service-account create -n "" -d "" --ar "developer" \ + --np ".=Write" +``` + +Note the returned `ServiceAccountId`. + +### Step 3: Create an API key for the service account + +```bash +tcld apikey create \ + --name \ + --description "" \ + --duration \ + --service-account-id +``` + +Save the returned key secret. + +### Step 4: Verify connectivity + +Set the API key and test with the Temporal CLI: + +```bash +export TEMPORAL_API_KEY= +temporal workflow list \ + --address ..tmprl.cloud:7233 \ + --namespace . +``` + +### Step 5: (Optional) Grant additional user access + +```bash +tcld user set-namespace-permissions \ + --user-email \ + --namespace-permission .= +``` + +Permissions: `Admin`, `Write`, `Read`. + +--- + +## (b) Check current APS and capacity mode for a Cloud namespace + +**When to use:** You need to know whether a namespace is On-Demand or Provisioned and what its current APS limit is. + +1. Get the namespace details: + + ```bash + tcld namespace get \ + --namespace . + ``` + + Output is JSON by default (no `--format` flag exists). + +2. In the JSON output, look for the capacity configuration section. Key fields: + + - **Capacity mode**: `on_demand` or `provisioned`. + - **TRU count** (if provisioned): the number of Temporal Resource Units allocated. Valid values: 2, 3, 4, 6, 8, 10, 12. + - **APS limit**: On-Demand default is 500; each TRU provides 500 APS. + +3. To check whether throttling is occurring, look for `temporal_cloud_v0_resource_exhausted_errors` in your metrics. + +--- + +## (c) Switch capacity mode from On-Demand to Provisioned + +**When to use:** You are preparing for a planned spike (load test, promotion, migration) and need to pre-provision capacity beyond the On-Demand auto-scaling limit. + +1. Confirm current capacity mode (see playbook (b)): + + ```bash + tcld namespace get \ + --namespace . + ``` + +2. Switch to Provisioned mode with the desired TRU count: + + ```bash + tcld namespace capacity update \ + --namespace . \ + --capacity-mode provisioned \ + --capacity-value + ``` + + Valid `--capacity-value` values: 2, 3, 4, 6, 8, 10, 12. + + Temporal aims to provision the additional capacity within two minutes. + + For requests in excess of 4 TRUs in regions outside of the US, submit a support ticket to ensure capacity availability. + + Requires **Global Admin** or **Namespace Admin** role. + +3. Verify the change took effect: + + ```bash + tcld namespace get \ + --namespace . + ``` + + Confirm the capacity mode is `provisioned` and the TRU count matches your request. + +4. When the spike is over, switch back to On-Demand: + + ```bash + tcld namespace capacity update \ + --namespace . \ + --capacity-mode on_demand + ``` + + When switching back to On-Demand mode, your APS limit resets to the running average from the last 7 days. Plan for this if your workload is sensitive to the transition. + +--- + +## (d) Find and triage all hung workflows in a namespace + +**When to use:** You suspect workflows are stuck and need to locate them and understand why they are not making progress. + +### Step 1: Count potentially stuck workflows + +```bash +temporal workflow count \ + --query "ExecutionStatus = 'Running' AND StartTime < '2024-01-15T09:00:00Z'" +``` + +Replace the timestamp with your threshold for "too long". A count greater than zero indicates workflows that have been running longer than expected. + +### Step 2: List the stuck workflows + +```bash +temporal workflow list \ + --query "ExecutionStatus = 'Running' AND StartTime < '2024-01-15T09:00:00Z'" \ + --limit 20 +``` + +You can narrow further by Task Queue or Workflow Type: + +```bash +temporal workflow list \ + --query "ExecutionStatus = 'Running' AND TaskQueue = 'my-task-queue' AND StartTime < '2024-01-15T09:00:00Z'" +``` + +### Step 3: Check worker health on the relevant Task Queue + +```bash +temporal task-queue describe \ + --task-queue my-task-queue +``` + +Look for: active pollers present, `LastAccessTime` within the last minute, no growing `ApproximateBacklogCount`. + +If there are no pollers, no Workers are running for this Task Queue and workflows on this queue cannot make progress. + +### Step 4: Inspect individual stuck workflows + +```bash +temporal workflow describe --workflow-id +``` + +```bash +temporal workflow show --workflow-id --reverse +``` + +```bash +temporal workflow stack --workflow-id +``` + +### Step 5: Diagnose root cause + +For diagnosing *why* a specific workflow is stuck (pending activities, pending child workflows, non-determinism, etc.), follow the triage procedures in `../triage/workflow-stuck.md`. + +--- + +## (e) Rotate an API key without downtime + +**When to use:** An API key is approaching expiration or needs to be rotated for security hygiene. + +1. Create a new API key (for a user): + + ```bash + tcld apikey create --name \ + --description "" \ + --duration + ``` + + Or for a Service Account: + + ```bash + tcld apikey create \ + --name \ + --description "" \ + --duration \ + --service-account-id + ``` + + You may reuse key names. + + Save the returned key secret -- it is only shown once. + +2. Verify both the original and new key function properly: + + ```bash + temporal workflow list \ + --address ..tmprl.cloud:7233 \ + --namespace . + ``` + + Set `TEMPORAL_API_KEY` to each key in turn and confirm the command succeeds. + +3. Update clients and workers to load the new key. + +4. Once no traffic uses the old key, delete it: + + ```bash + tcld apikey delete --id + ``` + + Alternatively, disable before deleting to validate nothing breaks: + + ```bash + tcld apikey disable --id + ``` + +**Limits:** Up to 10 non-expired keys per user; up to 20 non-expired keys per Service Account. Maximum expiration: 2 years. + +--- + +## (f) Audit namespace access (users + keys + service accounts) + +**When to use:** You need a complete picture of who and what can access a namespace -- humans, API keys, and service accounts. + +### Step 1: List all users with access to the namespace + +```bash +tcld user list --namespace . +``` + +This filters to users with direct permissions on the specified namespace. Users with account-level roles (e.g., Account Owner, Global Admin) have implicit access to all namespaces but may not appear in this filtered list — check `tcld user list` (without `--namespace`) and inspect account roles to get the full picture. + +### Step 2: Inspect individual user permissions + +```bash +tcld user get --user-email +``` + +Check the account role (`admin`, `developer`, `read`) and namespace-level permissions (`Admin`, `Write`, `Read`). + +### Step 3: List all user groups + +```bash +tcld user-group list +``` + +For each group with namespace access, list its members: + +```bash +tcld user-group list-members --group-id +``` + +### Step 4: List all service accounts + +```bash +tcld service-account list +``` + +Review the output for service accounts that have permissions on the target namespace. Namespace-scoped Service Accounts always have a `Read` Account Role and are restricted to a single namespace. + +### Step 5: List all API keys + +```bash +tcld apikey list +``` + +Cross-reference the API key owners (user IDs or service account IDs) against the users and service accounts identified above. + +--- +## (g) Rotate mTLS certificates + +**When to use:** A CA certificate is approaching expiration, or you need to switch to a new CA without disrupting running Workers. + +Temporal Cloud sends email notifications 15 days before certificate expiration. + +### Step 1: Generate a new CA certificate + +```bash +tcld generate-certificates certificate-authority-certificate \ + --organization \ + --validity-period \ + --ca-certificate-file .pem \ + --ca-key-file .key +``` + +Default key algorithm is ECDSA P-384. Maximum duration: 1 year. + +### Step 2: Generate new end-entity (leaf) certificates + +```bash +tcld generate-certificates end-entity-certificate \ + --organization \ + --validity-period \ + --ca-certificate-file .pem \ + --ca-key-file .key \ + --certificate-file .pem \ + --key-file .key +``` + +End-entity certificate must expire before its root CA certificate. + +### Step 3: Create a combined PEM bundle with old and new CA certificates + +Concatenate both CA certificates into a single PEM file: + +``` +-----BEGIN CERTIFICATE----- +... old CA cert ... +-----END CERTIFICATE----- +-----BEGIN CERTIFICATE----- +... new CA cert ... +-----END CERTIFICATE----- +``` + +### Step 4: Upload the combined bundle (replaces all existing CAs) + +```bash +tcld namespace accepted-client-ca set \ + --namespace . \ + --ca-certificate-file .pem +``` + +Both old and new end-entity certificates will now be accepted. + +### Step 5: Roll out new end-entity certificates to Workers and Clients + +Deploy the new leaf certificates to all Workers and Clients. Monitor traffic to the old certificate until it ceases. + +### Step 6: Remove the old CA certificate + +Create a file containing only the new CA certificate and run `set` again: + +```bash +tcld namespace accepted-client-ca set \ + --namespace . \ + --ca-certificate-file .pem +``` + +### Step 7: Verify the namespace only has the new CA + +```bash +tcld namespace accepted-client-ca list \ + --namespace . +``` + +Do NOT use a CA certificate signed with SHA-1 -- such signatures are rejected. + +--- + +## (h) Check self-hosted cluster health + +**When to use:** You want to verify that a self-hosted Temporal cluster is operational and inspect its configuration. + +The full command reference for self-hosted cluster operations lives in [self-hosted-admin.md](self-hosted-admin.md). This recipe chains the key commands into a quick health check. + +1. **Cluster health:** `temporal operator cluster health` — returns `SERVING` if healthy. +2. **Cluster details:** `temporal operator cluster describe --detail` — Cluster Name, persistence, visibility, shard count. +3. **Namespaces:** `temporal operator namespace list` — all Namespaces on the Service. +4. **Worker health on key Task Queues:** `temporal task-queue describe --task-queue ` — look for active pollers and no growing `ApproximateBacklogCount`. +5. **Spot-check stuck workflows:** `temporal workflow count --query "ExecutionStatus = 'Running' AND StartTime < ''"` — if high, follow playbook (d) above. + +For remote clusters, pass `--address :` (and TLS flags if enabled). See [self-hosted-admin.md → Global flags](self-hosted-admin.md#global-flags-summary) for the full flag set. + +--- + +## (i) View billing and generate a billing report + +**When to use:** You need to understand your Temporal Cloud costs at the namespace level, or generate a CSV billing report for FinOps tooling. + +### Step 1: Review billing in the Cloud UI + +Navigate to the **Billing** page in the Temporal Cloud UI. Account Owners and Finance Admins can view: + +- Current balance and recent bill +- Invoices table (with downloadable invoices for prior months) +- Credits table +- Cost by Namespace (per-namespace proportional cost breakdown) + +### Step 2: Generate a billing report via the Billing API + +The Billing API provides namespace-level cost attribution in CSV format. + +Report generation is asynchronous: + +1. Call `CreateBillingReport` with the desired date range (billing-month boundaries) and granularity. The response includes a `billing_report_id` and `async_operation_id`. +2. Poll `GetBillingReport` using the `billing_report_id` with exponential backoff. +3. When the state is `BILLING_REPORT_STATE_GENERATED`, retrieve the download URL. +4. Download the CSV before the URL expires. + +Date range limits by granularity: + +| Granularity | Available range | +|---|---| +| Hourly | Current + previous billing month | +| Daily | Current + previous two billing months | +| Monthly | Current + previous eleven billing months | + +### Step 3: Interpret the report + +Key columns to understand: + +- `ContractedCost`: The actual cost (not `Cost` or `TotalCost`). +- `ResourceID`: `namespace_name.account_id` (e.g., `production.a2dd6`), not just the namespace name. +- `BillingCurrency`: Values are in cents (e.g., `USD (cents)`). + +Only one billing report per account is generated at a time; additional requests are queued. + +--- + +## (j) Configure an Audit Log sink + +**When to use:** You need to stream Temporal Cloud control plane Audit Logs to your infrastructure for compliance or monitoring. + +Audit Logs capture control plane events only -- they do NOT capture data plane events (Workflow Start, etc.). + +Required role: Account Owner or Global Administrator. + +### Option A: AWS Kinesis + +1. Ensure you have a Kinesis Data Stream in your AWS account. An [AWS CloudFormation template](https://temporal-auditlogs-config.s3.us-west-2.amazonaws.com/cloudformation/iam-role-for-temporal-audit-logs.yaml) is available to create the required IAM role. + +2. In the Cloud UI: **Settings** > **Audit Logs** > **Setup**. + +3. Choose **Auto** (configure CloudFormation from the UI) or **Manual** (download a template). + +4. Enter the **Kinesis ARN**, **Role name**, and **AWS region**. + +5. Complete the CloudFormation stack creation. + +6. Use the **Verify** button to confirm Temporal can write to the stream. + +First logs appear within 10 minutes. + +### Option B: GCP Pub/Sub + +1. Create a Pub/Sub topic and set up a service account in the same GCP project (or skip if using Terraform). + +2. In the Cloud UI: **Settings** > **Audit Logs** > **Setup** > **Pub/Sub**. + +3. Enter the **service account email** and **Topic name**. + +4. Choose **Manual** or **Deploy with Terraform** to configure permissions. + +5. Use the **Verify** button, then click **Create**. + +Audit Logs appear in Pub/Sub within 10 minutes. + +### Verify the sink is working + +The Audit Logs page of the Cloud UI shows the current status: an **On** badge if functioning normally, or an error summary if an issue is detected. + +### Accessing logs via API + +Audit Logs are accessible for the past 30 days without a sink. + +**[Cloud Ops API](https://docs.temporal.io/ops) (retrieving log records):** Use `StartTimeInclusive`, `EndTimeExclusive`, `PageSize` (max 1000, default 100), and `PageToken` for pagination. + +tcld does not retrieve log records; it only manages export sinks. List the configured sinks with: + +```bash +tcld account audit-log kinesis list +tcld account audit-log pubsub list +``` + +--- + +## (k) Provision resources with Terraform + +**When to use:** You want to automate Temporal Cloud resource management (Namespaces, Users, Service Accounts, API Keys, Nexus Endpoints) using infrastructure as code. + +Provider source: [github.com/temporalio/terraform-provider-temporalcloud](https://github.com/temporalio/terraform-provider-temporalcloud). + +### Step 1: Set up the Terraform provider + +```bash +export TEMPORAL_CLOUD_API_KEY= +``` + +```hcl +terraform { + required_providers { + temporalcloud = { + source = "temporalio/temporalcloud" + } + } +} + +provider "temporalcloud" { + +} +``` + +### Step 2: Define resources + +Example Namespace: + +```hcl +resource "temporalcloud_namespace" "namespace" { + name = "terraform" + regions = ["aws-us-east-1"] + accepted_client_ca = base64encode(file("ca.pem")) + retention_days = 14 +} +``` + +Example User with namespace access: + +```hcl +resource "temporalcloud_user" "developer" { + email = "developer@example.com" + account_access = "Developer" + + namespace_accesses = [{ + namespace_id = temporalcloud_namespace.namespace.id + permission = "Write" + }] +} +``` + +### Step 3: Apply + +```bash +terraform init +terraform apply +``` + +### Key limitations + +- Once a resource is managed by Terraform, manage it only through Terraform. +- Terraform cannot create, update, or delete the Account Owner role. +- Namespace access must be managed from the User resource, not the Namespace resource. +- API keys cannot be imported into Terraform -- create new keys instead. +- The Terraform resource for API keys is `temporalcloud_apikey` (no underscore between `api` and `key`). + +--- + +## (l) Set up SAML SSO + +**When to use:** You want to enable single sign-on for your organization's Temporal Cloud account using your corporate identity provider. + +SAML is available as an add-on for any Temporal Cloud plan. + +### Step 1: Locate your Account Id + +Find your Account Id (5-6 characters after the period in your Namespace Id, e.g., `f45a2`). Available from the Cloud UI profile dropdown or from any Namespace Id. + +### Step 2: Construct the SAML URLs + +Entity identifier: + +``` +urn:auth0:prod-tmprl:ACCOUNT_ID-saml +``` + +Callback URL: + +``` +https://login.tmprl.cloud/login/callback?connection=ACCOUNT_ID-saml +``` + +Replace `ACCOUNT_ID` with your actual Account Id. + +### Step 3: Configure your IdP + +**Microsoft Entra ID:** Create an Enterprise application, configure SAML with the entity identifier, callback URL, and sign on URL (`https://cloud.temporal.io/login/saml?connection=ACCOUNT_ID-saml`). Set NameID to `user.userprincipalname` with format `emailAddress`. Collect the Certificate (Base64) and Login URL. + +**Okta:** Create a SAML 2.0 app integration. Set Single sign on URL to the callback URL. Set Audience URI to the entity identifier. Set Name ID format to `EmailAddress` with `email` and `name` attribute statements. Collect IdP settings and download the active certificate. + +### Step 4: Submit a support ticket + +Include: + +- The sign-in URL from your application +- The X.509 SAML sign-in certificate in PEM format +- One or more IdP domains to map to the SAML connection + +### Step 5: Verify + +After Temporal confirms configuration, log in with your email and click **Continue** to be redirected to your IdP. diff --git a/plugins/temporal/skills/temporal-ops/references/ops/self-hosted-admin.md b/plugins/temporal/skills/temporal-ops/references/ops/self-hosted-admin.md new file mode 100644 index 0000000..b0126f5 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/ops/self-hosted-admin.md @@ -0,0 +1,367 @@ +# Self-Hosted Admin — `temporal operator` + +Control-plane operations for self-hosted Temporal Services. +All commands use the `temporal operator` CLI prefix and connect to `--address` (default `localhost:7233`). + +> **Cloud vs Self-Hosted**: `temporal operator` manages self-hosted clusters. +> Cloud equivalents use `tcld` — see [comparison table](#cloud-equivalent-comparison) at the end. + +--- + +## Cluster Commands + +### Health check + +```bash +temporal operator cluster health +``` + +Returns health status of the Temporal Service. No subcommand-specific flags; uses [global flags](#global-flags-summary) only. + +### Describe cluster + +```bash +temporal operator cluster describe [--detail] +``` + +Shows Cluster Name, persistence store, and visibility store. + +| Flag | Required | Type | Description | +|------|----------|------|-------------| +| `--detail` | No | **bool** | Show history shard count and Cluster/Service version information. | + +### System info + +```bash +temporal operator cluster system +``` + +Shows Server version, scheduling support, and more. Defaults to local Service; use `--frontend-address` to target a remote endpoint. + +### List clusters + +```bash +temporal operator cluster list [--limit max-count] +``` + +Lists remote Temporal Clusters registered to the local Service. Reports: name, ID, address, History Shard count, Failover version, availability. + +| Flag | Required | Type | Description | +|------|----------|------|-------------| +| `--limit` | No | **int** | Maximum number of Clusters to display. | + +### Remove cluster + +Removes a registered remote Cluster from the local Service. De-registering it +affects everything relying on replication to that Cluster, across every Namespace +configured against it — the blast radius is Service-wide, not per-Namespace. +Confirm the Cluster name and what still depends on it with the user before +proposing this. + +```bash +temporal operator cluster remove --name YourClusterName +``` + +| Flag | Required | Type | Description | +|------|----------|------|-------------| +| `--name` | Yes | **string** | Cluster/Service name. | + +### Upsert cluster + +```bash +temporal operator cluster upsert \ + --frontend-address "YourRemoteEndpoint:YourRemotePort" \ + --enable-connection false +``` + +Add, remove, or update a registered remote Cluster. + +| Flag | Required | Type | Description | +|------|----------|------|-------------| +| `--enable-connection` | No | **bool** | Set the connection to "enabled". | +| `--enable-replication` | No | **bool** | Set the replication to "enabled". | +| `--frontend-address` | Yes | **string** | Remote endpoint. | + +--- + +## Namespace Commands + +### Create namespace + +```bash +temporal operator namespace create \ + --namespace YourNewNamespaceName \ + [options] +``` + +Create a Namespace with multi-region replication: + +```bash +temporal operator namespace create \ + --global \ + --namespace YourNewNamespaceName +``` + +| Flag | Required | Type | Description | +|------|----------|------|-------------| +| `--active-cluster` | No | **string** | Active Cluster (Service) name. | +| `--cluster` | No | **string[]** | Cluster names. Can be passed multiple times. | +| `--data` | No | **string[]** | Namespace data as `KEY=VALUE` pairs. Keys must be identifiers, values must be JSON. | +| `--description` | No | **string** | Namespace description. | +| `--email` | No | **string** | Owner email. | +| `--global` | No | **bool** | Enable multi-region data replication. | +| `--history-archival-state` | No | **string-enum** | Accepted values: `disabled`, `enabled`. Default `disabled`. | +| `--history-uri` | No | **string** | Archive history to this URI. Once enabled, can't be changed. | +| `--retention` | No | **duration** | Time to preserve closed Workflows before deletion. Default `72h`. | +| `--visibility-archival-state` | No | **string-enum** | Accepted values: `disabled`, `enabled`. Default `disabled`. | +| `--visibility-uri` | No | **string** | Archive visibility to this URI. Once enabled, can't be changed. | + +Note: URI values for archival states can't be changed once enabled. + +### Delete namespace + +Deletion is permanent, and it takes the Namespace's Workflow Executions and Task +Queues with it. Never run it autonomously. Before proposing it, report what the +Namespace holds — `temporal workflow count --query 'ExecutionStatus="Running"'` +against that Namespace — and confirm the Namespace name with the user. + +```bash +temporal operator namespace delete --namespace YourNamespaceName +``` + +| Flag | Required | Type | Description | +|------|----------|------|-------------| +| `--yes`, `-y` | No | **bool** | Don't prompt to confirm deletion. | + +The confirmation is stronger than the usual `y/N`: the CLI asks for the Namespace +name to be typed back. Without `--yes`, and with no terminal to answer the prompt, +the command exits non-zero with `user denied confirmation or mistyped the +namespace name` and deletes nothing. So `--yes` belongs only in a command the user +has already approved — never in a retry of one that failed its prompt. + +### Describe namespace + +```bash +temporal operator namespace describe --namespace YourNamespaceName +``` + +Can also identify by `--namespace-id`: + +| Flag | Required | Type | Description | +|------|----------|------|-------------| +| `--namespace-id` | No | **string** | Namespace ID. | + +### List namespaces + +```bash +temporal operator namespace list +``` + +Displays a detailed listing for all Namespaces on the Service. No subcommand-specific flags. + +### Update namespace + +```bash +temporal operator namespace update --namespace YourNamespaceName [options] +``` + +Examples: + +```bash +# Assign active cluster +temporal operator namespace update \ + --namespace YourNamespaceName \ + --active-cluster NewActiveCluster + +# Promote for multi-region replication +temporal operator namespace update \ + --namespace YourNamespaceName \ + --promote-global +``` + +| Flag | Required | Type | Description | +|------|----------|------|-------------| +| `--active-cluster` | No | **string** | Active Cluster (Service) name. | +| `--cluster` | No | **string[]** | Cluster (Service) names. | +| `--data` | No | **string[]** | Namespace data as `KEY=VALUE` pairs. | +| `--description` | No | **string** | Namespace description. | +| `--email` | No | **string** | Owner email. | +| `--history-archival-state` | No | **string-enum** | Accepted values: `disabled`, `enabled`. | +| `--history-uri` | No | **string** | Archive history URI. Once enabled, can't be changed. | +| `--promote-global` | No | **bool** | Enable multi-region data replication. | +| `--replication-state` | No | **string-enum** | Accepted values: `normal`, `handover`. | +| `--retention` | No | **duration** | Time to preserve closed Workflows before deletion. | +| `--visibility-archival-state` | No | **string-enum** | Accepted values: `disabled`, `enabled`. | +| `--visibility-uri` | No | **string** | Archive visibility URI. Once enabled, can't be changed. | + +--- + +## Search Attribute Commands + +Supported types: `Text`, `Keyword`, `Int`, `Double`, `Bool`, `Datetime`, `KeywordList`. + +### Create search attribute + +```bash +temporal operator search-attribute create \ + --name YourAttributeName \ + --type Keyword +``` + +| Flag | Required | Type | Description | +|------|----------|------|-------------| +| `--name` | Yes | **string[]** | Search Attribute name. | +| `--type` | Yes | **string-enum[]** | Accepted values: `Text`, `Keyword`, `Int`, `Double`, `Bool`, `Datetime`, `KeywordList`. | + +### List search attributes + +```bash +temporal operator search-attribute list +``` + +Displays active Search Attributes that can be assigned or used in Workflow Queries. No subcommand-specific flags. + +### Remove search attribute + +Confirm with the user before proposing a removal: every List Filter, saved query, +and Workflow Query referencing the attribute stops resolving, and the effect is +Namespace-wide rather than scoped to one Workflow. + +```bash +temporal operator search-attribute remove --name YourAttributeName +``` + +Once the user has approved the removal, `--yes` skips the confirmation prompt: + +```bash +temporal operator search-attribute remove --name YourAttributeName --yes +``` + +| Flag | Required | Type | Description | +|------|----------|------|-------------| +| `--name` | Yes | **string[]** | Search Attribute name. | +| `--yes`, `-y` | No | **bool** | Don't prompt to confirm removal. | + +> **Self-hosted deletion note**: `remove` de-registers custom attributes from the queryable set ("Remove custom Search Attributes from the options that can be assigned or used with Workflow Queries"). Permanent deletion from the backing store may require additional steps. The Cloud docs note: "If you wish to delete a Search Attribute, please contact Support." + +--- + +## Nexus Endpoint Commands + +### Create endpoint + +```bash +temporal operator nexus endpoint create \ + --name your-endpoint \ + --target-namespace your-namespace \ + --target-task-queue your-task-queue \ + --description-file DESCRIPTION.md +``` + +Target is either a Worker (`--target-namespace` + `--target-task-queue`) or an external URL (`--target-url`). + +Fails if an Endpoint with the same name already exists. + +| Flag | Required | Type | Description | +|------|----------|------|-------------| +| `--description` | No | **string** | Endpoint description. May use Markdown. | +| `--description-file` | No | **string** | Path to description file. May use Markdown. | +| `--name` | Yes | **string** | Endpoint name. | +| `--target-namespace` | No | **string** | Namespace where handler Worker polls for Nexus tasks. | +| `--target-task-queue` | No | **string** | Task Queue that handler Worker polls for Nexus tasks. | +| `--target-url` | No | **string** | External endpoint URL. _(Experimental)_ | + +### Delete endpoint + +```bash +temporal operator nexus endpoint delete --name your-endpoint +``` + +| Flag | Required | Type | Description | +|------|----------|------|-------------| +| `--name` | Yes | **string** | Endpoint name. | + +### Get endpoint (EXPERIMENTAL) + +```bash +temporal operator nexus endpoint get --name your-endpoint +``` + +| Flag | Required | Type | Description | +|------|----------|------|-------------| +| `--name` | Yes | **string** | Endpoint name. | + +### List endpoints + +```bash +temporal operator nexus endpoint list +``` + +No subcommand-specific flags. + +### Update endpoint + +```bash +temporal operator nexus endpoint update \ + --name your-endpoint \ + --target-task-queue your-other-queue +``` + +Patches the endpoint; existing fields for which flags are not provided are left unchanged. + +| Flag | Required | Type | Description | +|------|----------|------|-------------| +| `--description` | No | **string** | Endpoint description. May use Markdown. | +| `--description-file` | No | **string** | Path to description file. May use Markdown. | +| `--name` | Yes | **string** | Endpoint name. | +| `--target-namespace` | No | **string** | Namespace where handler Worker polls for Nexus tasks. | +| `--target-task-queue` | No | **string** | Task Queue that handler Worker polls for Nexus tasks. | +| `--target-url` | No | **string** | External endpoint URL. _(Experimental)_ | +| `--unset-description` | No | **bool** | Unset the description. | + +--- + +## Global Flags Summary + +Key global flags applicable to all `temporal operator` commands: + +| Flag | Type | Default | Description | +|------|------|---------|-------------| +| `--address` | **string** | `localhost:7233` | Temporal Service gRPC endpoint. | +| `--namespace`, `-n` | **string** | `default` | Temporal Service Namespace. | +| `--api-key` | **string** | | API key for request. | +| `--tls` | **bool** | | Enable base TLS encryption. Defaulted to true if api-key or other TLS options are present. | +| `--tls-ca-path` | **string** | | Path to server CA certificate. | +| `--tls-cert-path` | **string** | | Path to x509 certificate. | +| `--tls-key-path` | **string** | | Path to x509 private key. | +| `--tls-server-name` | **string** | | Override target TLS server name. | +| `--output`, `-o` | **string-enum** | `text` | Non-logging data output format. Accepted values: `text`, `json`, `jsonl`, `none`. | +| `--log-level` | **string-enum** | `never` | Log level. Accepted values: `debug`, `info`, `warn`, `error`, `never`. | +| `--env` | **string** | `default` | Active environment name. | +| `--config-file` | **string** | | TOML config file path. | + +See `docs/cli/operator.mdx` lines 520-558 for the full list. + +--- + +## Cloud-Equivalent Comparison + +| Operation | Self-Hosted (`temporal operator`) | Cloud (`tcld`) | +|-----------|-----------------------------------|----------------| +| Check cluster health | `temporal operator cluster health` | N/A (Cloud-managed) | +| Describe cluster | `temporal operator cluster describe` | N/A (Cloud-managed) | +| Server system info | `temporal operator cluster system` | N/A (Cloud-managed) | +| List clusters | `temporal operator cluster list` | N/A (Cloud-managed) | +| Create namespace | `temporal operator namespace create` | `tcld namespace create` | +| Delete namespace | `temporal operator namespace delete` | `tcld namespace delete` | +| Describe namespace | `temporal operator namespace describe` | `tcld namespace get` | +| List namespaces | `temporal operator namespace list` | `tcld namespace list` | +| Update namespace | `temporal operator namespace update` | No single equivalent; use per-attribute subcommands (`retention set`, `capacity update`, `auth-method set`, `tags`, …) | +| Create search attribute | `temporal operator search-attribute create` | `tcld namespace search-attributes add` | +| List search attributes | `temporal operator search-attribute list` | No tcld subcommand; use Cloud UI or Cloud Ops API | +| Remove search attribute | `temporal operator search-attribute remove` | `tcld namespace search-attributes rename` | +| Create Nexus endpoint | `temporal operator nexus endpoint create` | `tcld nexus endpoint create` | +| Delete Nexus endpoint | `temporal operator nexus endpoint delete` | `tcld nexus endpoint delete` | +| Get Nexus endpoint | `temporal operator nexus endpoint get` | `tcld nexus endpoint get` | +| List Nexus endpoints | `temporal operator nexus endpoint list` | `tcld nexus endpoint list` | +| Update Nexus endpoint | `temporal operator nexus endpoint update` | `tcld nexus endpoint update` | diff --git a/plugins/temporal/skills/temporal-ops/references/ops/workflow-health.md b/plugins/temporal/skills/temporal-ops/references/ops/workflow-health.md new file mode 100644 index 0000000..a4de11d --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/ops/workflow-health.md @@ -0,0 +1,450 @@ +# Workflow Health Queries + +Data-plane commands for **finding** unhealthy Workflow Executions. +Backend-agnostic (works on Temporal Cloud and self-hosted). + +> **Scope.** This file covers *locating* stuck/hung/failed workflows. +> To *diagnose why* a specific workflow is stuck, see `../triage/workflow-stuck.md`. + +--- + +## 1. List Filter fundamentals + +`temporal workflow list` accepts an optional `--query` (`-q`) flag whose value is an SQL-like List Filter string. + +### Supported operators + +`=, !=, >, >=, <, <=` | `AND, OR, ()` | `BETWEEN ... AND` | `IN` | `STARTS_WITH` + +Additional filter expressions: `IS NULL`, `IS NOT NULL` + +> **ORDER BY is not supported in Temporal Cloud.** +> The default ordering is `ClosedTime DESC NULL FIRST`, `StartTime DESC`. + +### Key Search Attributes for health queries + +All of the following are default (built-in) Search Attributes: + +| Attribute | Type | Notes | +|-----------|------|-------| +| `ExecutionStatus` | Keyword | Current state. Values: `Running`, `Completed`, `Failed`, `Canceled`, `Terminated`, `ContinuedAsNew`, `TimedOut` | +| `WorkflowType` | Keyword | The type of Workflow | +| `WorkflowId` | Keyword | Identifies the Workflow Execution | +| `TaskQueue` | Keyword | Task Queue used by Workflow Execution | +| `StartTime` | Datetime | Time the Workflow Execution started | +| `CloseTime` | Datetime | Time the Workflow Execution completed (closed workflows only) | +| `ExecutionTime` | Datetime | Actual begin time; differs from `StartTime` for cron/retry | +| `ExecutionDuration` | Int | Time to run in nanoseconds (closed workflows only) | +| `HistoryLength` | Int | Event count (closed workflows only) | +| `HistorySizeBytes` | Long | Size of Event History | +| `StateTransitionCount` | Int | Number of state persists (closed workflows only) | + +Search Attribute names are case sensitive. + +Datetime attributes accept RFC3339Nano strings (e.g. `"2024-01-15T10:00:00Z"`) or epoch-nanosecond integers. + +`ExecutionDuration` accepts nanosecond integers, Golang duration format, or `"hh:mm:ss"` format. + +--- + +## 2. Listing workflows by status + +### All running workflows + +``` +temporal workflow list \ + --query "ExecutionStatus = 'Running'" +``` + +### All failed workflows + +``` +temporal workflow list \ + --query "ExecutionStatus = 'Failed'" +``` + +### All timed-out workflows + +``` +temporal workflow list \ + --query "ExecutionStatus = 'TimedOut'" +``` + +### All terminated workflows + +``` +temporal workflow list \ + --query "ExecutionStatus = 'Terminated'" +``` + +### Non-running workflows (any closed status) + +``` +temporal workflow list \ + --query "ExecutionStatus != 'Running'" +``` + +--- + +## 3. Finding stuck/long-running workflows + +### Running longer than a time threshold + +Workflows that started more than 24 hours ago and are still running: + +``` +temporal workflow list \ + --query "ExecutionStatus = 'Running' AND StartTime < '2024-01-14T00:00:00Z'" +``` + +Replace the timestamp with the appropriate cutoff for your use case. + +### Running workflows on a specific Task Queue + +``` +temporal workflow list \ + --query "ExecutionStatus = 'Running' AND TaskQueue = 'my-task-queue'" +``` + +### Running workflows of a specific type + +``` +temporal workflow list \ + --query "ExecutionStatus = 'Running' AND WorkflowType = 'MyWorkflow'" +``` + +### Combining conditions + +``` +temporal workflow list \ + --query "WorkflowType = 'OrderWorkflow' AND ExecutionStatus = 'Running' AND StartTime < '2024-01-14T00:00:00Z'" +``` + +### Workflows started in a time window + +``` +temporal workflow list \ + --query "StartTime BETWEEN '2024-01-01T00:00:00Z' AND '2024-01-02T00:00:00Z'" +``` + +### Matching Workflow IDs by prefix + +``` +temporal workflow list \ + --query "WorkflowId STARTS_WITH 'order-'" +``` + +`STARTS_WITH` is only available for Keyword Search Attributes. + +--- + +## 4. Counting workflows + +`temporal workflow count` returns a count of Workflow Executions regardless of execution state. Use `--query` to filter: + +``` +temporal workflow count \ + --query "ExecutionStatus = 'Running'" +``` + +``` +temporal workflow count \ + --query "ExecutionStatus = 'Failed'" +``` + +``` +temporal workflow count \ + --query "ExecutionStatus = 'Running' AND TaskQueue = 'my-task-queue'" +``` + +Use count to detect anomalies: a rising count of `Running` workflows with a stable `Failed` count may indicate workers are not processing tasks (check poller status below). + +--- + +## 5. Describing a specific workflow + +`temporal workflow describe` displays information about a specific Workflow Execution: + +``` +temporal workflow describe \ + --workflow-id YourWorkflowId +``` + +Key flags: + +| Flag | Description | +|------|-------------| +| `--workflow-id`, `-w` | **(required)** Workflow ID | +| `--run-id`, `-r` | Run ID (optional, defaults to latest run) | +| `--reset-points` | Show auto-reset points only | +| `--raw` | Print properties without format changes | + +The output includes execution status, start/close times, task queue, workflow type, search attributes, and pending activities/child workflows. + +### Viewing Event History + +`temporal workflow show` displays the full Event History: + +``` +temporal workflow show \ + --workflow-id YourWorkflowId +``` + +Key flags: + +| Flag | Description | +|------|-------------| +| `--workflow-id`, `-w` | **(required)** Workflow ID | +| `--run-id`, `-r` | Run ID | +| `--follow`, `-f` | Follow progress in real time (not for JSON output) | +| `--reverse` | Fetch newest events first (cannot combine with `--follow`) | +| `--detailed` | Display events as detailed sections | +| `--output json` | JSON output (usable for SDK replay) | + +Export history for replay: + +``` +temporal workflow show \ + --workflow-id YourWorkflowId \ + --output json +``` + +--- + +## 6. Stack trace + +Get the current stack trace of a running Workflow's threads/routines: + +``` +temporal workflow stack \ + --workflow-id YourWorkflowId +``` + +This performs a `__stack_trace`-type Query on the Workflow Execution. + +Flags: + +| Flag | Description | +|------|-------------| +| `--workflow-id`, `-w` | **(required)** Workflow ID | +| `--run-id`, `-r` | Run ID | +| `--reject-condition` | Reject based on Workflow state. Values: `not_open`, `not_completed_cleanly` | + +--- + +## 7. Task Queue poller status + +`temporal task-queue describe` displays active Workers that have recently polled a Task Queue. + +``` +temporal task-queue describe \ + --task-queue YourTaskQueue +``` + +**Interpreting poller results:** + +- The Temporal Server records each poll request time. +- A `LastAccessTime` over one minute may indicate the Worker is at capacity or has shut down. +- Workers are removed if 5 minutes have passed since the last poll request. + +### Workflow vs. Activity pollers + +Workflow and Activity polling use separate Task Queues. Specify the type to check Activity pollers: + +``` +temporal task-queue describe \ + --task-queue YourTaskQueue \ + --task-queue-type "activity" +``` + +The `--task-queue-type` flag accepts: `workflow`, `activity`, `nexus`. If not specified, all types are reported. + +### Backlog statistics + +The describe output includes the following statistics: + +| Statistic | Description | +|-----------|-------------| +| `ApproximateBacklogCount` | Approximate tasks backlogged. May count expired tasks but eventually converges. | +| `ApproximateBacklogAge` | Approximate age of the oldest backlogged task (seconds), based on creation time. | +| `TasksAddRate` | Approximate tasks added per second, averaged over the last 30 seconds. Includes sync-matched tasks. | +| `TasksDispatchRate` | Approximate tasks dispatched per second, averaged over the last 30 seconds. Includes sync-matched tasks. | +| `BacklogIncreaseRate` | Approximate rate of backlog growth (positive) or shrinkage (negative), in tasks per second. Roughly `TasksAddRate - TasksDispatchRate`. | + +> **Note:** `TasksAddRate` and `TasksDispatchRate` may differ from actual rates because eagerly dispatched or sticky tasks are not counted. The derived `BacklogIncreaseRate` is accurate for backlogs older than a few seconds. + +To disable statistics and show only poller info, use `--disable-stats`. + +### What to look for + +- **No pollers**: No Workers are running (or recently running) for this Task Queue. Workflows on this queue will not make progress. +- **Stale `LastAccessTime`**: Workers may be overloaded or shutting down. +- **Growing `ApproximateBacklogCount` / positive `BacklogIncreaseRate`**: Workers cannot keep up with the incoming task rate. Scale up Workers or investigate slow activities. + +--- + +## 8. Workflow tracing + +Display progress of a Workflow Execution and its child workflows in real time: + +``` +temporal workflow trace \ + --workflow-id YourWorkflowId +``` + +Key flags: + +| Flag | Description | +|------|-------------| +| `--workflow-id`, `-w` | **(required)** Workflow ID | +| `--depth` | Depth for child Workflow fetches. `-1` fetches all depths. | +| `--fold` | Fold away child Workflows with specified statuses. Values: `running`, `completed`, `failed`, `canceled`, `terminated`, `timedout`, `continueasnew`. | +| `--no-fold` | Disable folding; fetch and display all child Workflows within depth. | +| `--concurrency` | Number of Workflow Histories to fetch concurrently. | + +--- + +## 9. Pagination and output control + +### Limiting results + +``` +temporal workflow list \ + --query "ExecutionStatus = 'Running'" \ + --limit 50 +``` + +### Page size + +``` +temporal workflow list \ + --query "ExecutionStatus = 'Running'" \ + --page-size 100 +``` + +### JSON output + +Use `--output json` or `--output jsonl` on any command for machine-readable output. + +### Archived workflows + +``` +temporal workflow list --archived +``` + +This is an experimental feature. + +--- + +## 10. Common health-check patterns + +### Pattern: "Are any workflows stuck?" + +```bash +# Count running workflows that started more than 1 hour ago +temporal workflow count \ + --query "ExecutionStatus = 'Running' AND StartTime < '2024-01-15T09:00:00Z'" + +# If count > 0, list them +temporal workflow list \ + --query "ExecutionStatus = 'Running' AND StartTime < '2024-01-15T09:00:00Z'" \ + --limit 20 +``` + +### Pattern: "Are workers healthy?" + +```bash +# Check pollers on the task queue +temporal task-queue describe \ + --task-queue my-task-queue + +# Check activity pollers separately +temporal task-queue describe \ + --task-queue my-task-queue \ + --task-queue-type activity +``` + +Look for: active pollers present, `LastAccessTime` within the last minute, no growing backlog. + +### Pattern: "What failed recently?" + +```bash +temporal workflow list \ + --query "ExecutionStatus = 'Failed' AND CloseTime > '2024-01-15T00:00:00Z'" + +temporal workflow count \ + --query "ExecutionStatus = 'Failed' AND CloseTime > '2024-01-15T00:00:00Z'" +``` + +To break failures down by Workflow Type, add a `WorkflowType` filter to the same query. Run a count per type across your known types: + +```bash +temporal workflow count \ + --query "ExecutionStatus = 'Failed' AND WorkflowType = ''" +``` + +This identifies which Workflow Type is contributing the most failures rather than returning a flat list. `GROUP BY` in the Count API only supports grouping by `ExecutionStatus`, not by `WorkflowType` or other attributes — use per-type filtered counts instead. + +### Pattern: "Workflows approaching history limits" + +The server terminates a Workflow Execution when its Event History exceeds 51,200 events, contains more than 2,000 Updates, or more than 10,000 Signals. The `HistoryLength` Search Attribute surfaces the event count for running workflows. + +```bash +temporal workflow list \ + --query "ExecutionStatus = 'Running' AND HistoryLength > 40000" +``` + +Long-lived workflows that grow history without using Continue-As-New will eventually hit these limits. This is common in agent-loop or orchestrator patterns. Workflows returned by this query need either Continue-As-New or a redesign to bound history growth. + +### Pattern: "Drill into a specific stuck workflow" + +```bash +# 1. Describe it (status, pending activities, search attributes) +temporal workflow describe --workflow-id + +# 2. View its event history (newest first) +temporal workflow show --workflow-id --reverse + +# 3. Get its stack trace (running workflows only) +temporal workflow stack --workflow-id +``` + +For diagnosing *why* a workflow is stuck, see `../triage/workflow-stuck.md`. + +--- + +## Quick reference: List Filter examples from docs + +The following examples are taken directly from the docs: + +```sql +WorkflowType = "main.YourWorkflowDefinition" and ExecutionStatus != "Running" and (StartTime > "2021-06-07T16:46:34.236-08:00" or CloseTime > "2021-06-07T16:46:34-08:00") +``` + +```sql +WorkflowId = '' +``` + +```sql +WorkflowId = '' or WorkflowId = '' +``` + +```sql +WorkflowId IN ('', '') +``` + +```sql +WorkflowId = '' and ExecutionStatus = 'Running' +``` + +```sql +WorkflowId = '' and StartTime > '2021-08-22T15:04:05+00:00' +``` + +```sql +ExecutionTime between '2021-08-22T15:04:05+00:00' and '2021-08-28T15:04:05+00:00' +``` + +```sql +WorkflowType STARTS_WITH '' +``` diff --git a/plugins/temporal/skills/temporal-ops/references/triage/authentication.md b/plugins/temporal/skills/temporal-ops/references/triage/authentication.md new file mode 100644 index 0000000..2c1cbd7 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/triage/authentication.md @@ -0,0 +1,312 @@ +# Authentication + +Diagnose failures that happen *after* TLS has completed — the TCP connection is up, the handshake finished, and the peer returned a gRPC error about who you are or what you can do. This file covers layer 4 of the [diagnostic ladder](diagnostic-ladder.md). + +Prerequisite: rule out layers 1–3 first. If `openssl s_client` (see [certificates.md → openssl recipes](certificates.md#openssl-recipes)) does not print `Verify return code: 0 (ok)`, the problem is not in this file. + +Out of scope here: +- DNS / TCP / endpoint family → [connectivity.md](connectivity.md) (layers 1–2) +- TLS handshake / x509 / SNI override → [certificates.md](certificates.md) (layer 3) +- gRPC `RESOURCE_EXHAUSTED` → [rate-limits.md](rate-limits.md) +- `context deadline exceeded` (ambiguous) → [runtime-errors.md](runtime-errors.md) + +## Table of Contents + +- [Authentication vs authorization](#authentication-vs-authorization) +- [UNAUTHENTICATED vs PERMISSION_DENIED](#unauthenticated-vs-permission_denied) +- [API-key authentication](#api-key-authentication) +- [mTLS authentication after TLS completes](#mtls-authentication-after-tls-completes) +- [Cloud role and permission model](#cloud-role-and-permission-model) +- [Quick routing](#quick-routing) + +## Authentication vs authorization + +Temporal Cloud distinguishes authentication from authorization, but **API keys and mTLS do not share the same authorization model**. + +### API keys + +- **Authentication** — *who you are*. The API key is valid and active. Failure → `UNAUTHENTICATED`. +- **Authorization** — *what you can do*. Key → User or Service Account → Cloud RBAC. Failure → `PERMISSION_DENIED`. + +The authorization pathway for API keys is documented as "API key (authentication) → Identity (user or Service Account) → RBAC (authorization)". + +### mTLS + +- **Authentication (connection)** — client cert chains to an accepted Namespace CA. Failure is usually a **TLS-layer** error (`tls:` / `x509:`), not gRPC `UNAUTHENTICATED`. See [certificates.md](certificates.md). +- **Authorization (Namespace access)** — after a clean handshake: no certificate filters → full Namespace data-plane access; with filters → leaf must match or the connection is rejected (exact wire signal is release-dependent — see [certificates.md](certificates.md)). + +**mTLS is not tied to Cloud RBAC identities** (User / Service Account / Namespace Read|Write|Admin). Access is CA trust + optional filters. Do not triage mTLS `PERMISSION_DENIED` via `tcld user get`. Check CA, filters, Namespace name (`.`), and Namespace auth method. + +## UNAUTHENTICATED vs PERMISSION_DENIED + +The two gRPC codes in this file, per the gRPC spec: + +| Code | Meaning in the spec | What it tells you on Temporal Cloud | +|---|---|---| +| `UNAUTHENTICATED` | The request does not have valid authentication credentials for the operation. | Almost always an **API key** problem: missing, typo, disabled, deleted, expired, or wrong auth method for the Namespace. Untrusted mTLS certs usually fail at TLS (`UNAVAILABLE` / `x509:`), not here. | +| `PERMISSION_DENIED` | The caller does not have permission to execute the specified operation. Distinct from `UNAUTHENTICATED`. | **API key:** identity known but lacks the Namespace-level permission (`Read` / `Write` / `Admin`) or the account-level role required for the action (including Owner/Admin inheritance). **mTLS:** often cert-filter mismatch, wrong Namespace, or auth-method mismatch — not missing Cloud RBAC. | + +Two traps this distinction prevents: + +- **`UNAVAILABLE` is not an auth code.** If your error is gRPC `UNAVAILABLE`, TLS may have failed before any auth happened. Peel the wrapped cause; if it starts with `tls:`, `x509:`, or `remote error: tls:`, jump to [certificates.md](certificates.md). The troubleshooting guide names an expired TLS certificate as a common root cause of "looks like auth but isn't." +- **`RESOURCE_EXHAUSTED` is not `PERMISSION_DENIED`.** A rate-limited caller is *allowed* to make the call but is being throttled. See [rate-limits.md](rate-limits.md). + +Do not use `FORBIDDEN` or `FAILED_PRECONDITION` as auth codes. `FORBIDDEN` is not a gRPC code; `FAILED_PRECONDITION` is a gRPC code but is unrelated to Cloud auth decisions. + +## API-key authentication + +### How the Temporal CLI and SDKs pick up the key + +The Temporal CLI reads the API key either from the `--api-key` flag or from the `TEMPORAL_API_KEY` environment variable. The Cloud docs: "The CLI automatically picks up the `TEMPORAL_API_KEY` environment variable from your shell." + +`tcld` and the Terraform provider use a **different** env var: `--api-key` or `TEMPORAL_CLOUD_API_KEY` (confirmed in `tcld` source `app/flags.go`). Do not tell a `tcld` user to set `TEMPORAL_API_KEY` — and do not tell an SDK / `temporal` CLI user that `TEMPORAL_CLOUD_API_KEY` is enough. Public api-keys docs that claim `tcld` reads `TEMPORAL_API_KEY` are stale vs current `tcld`. + +This section covers data-plane API-key auth (`TEMPORAL_API_KEY` / `--api-key` on `temporal` CLI and SDKs). + +### Address form for API-key connections + +For **API-key-only** Namespaces, the Cloud API-keys guide and SDK develop docs recommend the **Namespace Endpoint**: `..tmprl.cloud:7233`. + +Use the **API Regional Endpoint** (`..api.temporal.io:7233`) when: + +- The client needs an explicit region pin +- Private connectivity without private DNS (SNI / server name may need the regional API hostname — see [connectivity.md](connectivity.md#endpoint-formats)) +- The Namespace uses **dual auth** (`api_key_or_mtls`, pre-release): API keys cannot use the Namespace Endpoint on these Namespaces — enable via Support; HA is not supported in this mode + +If an API key fails on the Namespace Endpoint, try Regional before assuming the key is bad (especially if the Namespace allows both auth methods). See the endpoint table in [connectivity.md → endpoint formats](connectivity.md#endpoint-formats) for the full comparison. + +### Things to check when `UNAUTHENTICATED` is returned with an API key + +Per the Cloud troubleshooting note: "Invalid API key errors: Check that you copied the key correctly and that it hasn't been revoked or expired." + +- **Key not delivered to the process.** Inside the failing environment (pod/container/host), confirm `TEMPORAL_API_KEY` is set: `env | grep -i TEMPORAL_API_KEY`. A shell-level export on the developer's laptop is not inherited by a container. +- **Key typo or truncation.** Leading/trailing whitespace, a trailing newline from a copy-paste, or a shell that split the key on whitespace will all produce `UNAUTHENTICATED`. +- **Key disabled.** A disabled key cannot authenticate — per the Cloud docs: "When disabled, an API key cannot authenticate with Temporal Cloud." Check with `tcld apikey list` or `tcld apikey get --id `. +- **Key deleted.** Per the Cloud docs: "Deleting an API key stops it from authenticating with Temporal Cloud." +- **Key expired.** API keys expire based on the `--duration` or `--expiry` set at creation time. The FAQ caps expiry at 2 years. +- **Wrong address / auth combo.** See above — default to Namespace Endpoint for API-key-only; use API Regional when pinning, private connectivity, or a dual-auth (`api_key_or_mtls`) Namespace requires it. +- **API keys disabled at the account level.** A Global Administrator or Account Owner can disable the *creation* of new API keys with the **Disable Create API Keys** control; existing keys continue to work until disabled, deleted, or expired. This does not on its own turn a working key into `UNAUTHENTICATED`. + +### API-key lifecycle commands + +Every command here is grounded in `docs/cloud/tcld/apikey.mdx`. The `tcld apikey` group alias is `ak`. + +```bash +# Create +tcld apikey create --name +# Command and required flag: +# Optional: --description, --duration, --expiry, --request-id +# --description: +# --duration: +# --expiry: +# --request-id: + +# List (to find the ID for disable/enable/delete) +tcld apikey list +# Command: + +# Inspect a specific key +tcld apikey get --id +# Command: + +# Disable / enable +tcld apikey disable --id +tcld apikey enable --id +# disable: +# enable: + +# Delete +tcld apikey delete --id +# Command: +``` + +### Rotating without downtime + +The Cloud docs prescribe this sequence: create a new key; verify both keys work; switch clients to the new key; delete the old key. + +Key behavioral note: "Deleting or disabling a key removes its ability to authenticate into Temporal Cloud. If you delete or disable an API key being used by Workers to run a Workflow, those Workers will be unable to connect to Temporal until a new API key secret is created and configured." That is why the rotation order is "deploy new, verify, then delete old" and not the reverse. + +### Discriminating with a CLI smoke test + +Prefer Namespace Endpoint for API-key-only Namespaces. Use Regional for dual-auth Namespaces or when Namespace Endpoint fails: + +```bash +# Preferred for API-key-only Namespaces +temporal workflow list --limit 1 \ + --address ..tmprl.cloud:7233 \ + --namespace . \ + --api-key "$TEMPORAL_API_KEY" + +# Fallback: Regional Endpoint (dual-auth API-key clients, region pin, private connectivity) +temporal workflow list --limit 1 \ + --address ..api.temporal.io:7233 \ + --namespace . \ + --api-key "$TEMPORAL_API_KEY" +# --address / --namespace / --api-key flags: docs/cli/setup-cli.mdx +``` + +Interpret the result: + +| CLI result | Where the chain broke | Where to go | +|---|---|---| +| Returns a list (possibly empty) | Auth and authorization both succeeded | Issue is elsewhere — look at the SDK / worker config | +| `UNAUTHENTICATED` | Key itself is rejected (disabled, deleted, expired, typo, wrong env var, wrong endpoint / auth combo) | Re-run the [things-to-check list above](#things-to-check-when-unauthenticated-is-returned-with-an-api-key) | +| `PERMISSION_DENIED` | Key is authenticated but the identity lacks Namespace permission | [Cloud role and permission model](#cloud-role-and-permission-model) | +| API key rejected on the Namespace Endpoint (mTLS-only or dual-auth Namespace) | Wrong endpoint for the auth combo | Try the API Regional Endpoint | +| An `x509:` or `tls:` error | Not an auth issue | [certificates.md](certificates.md) | + +A lighter probe is `temporal operator cluster health --address ` — if it returns `SERVING`, the client can reach the frontend service. + +## mTLS authentication after TLS completes + +If the TLS handshake succeeded (the peer did not send a `tls:` alert and the client did not emit an `x509:` error — see [certificates.md](certificates.md) for those), remaining mTLS failures are usually: + +1. **Certificate filter mismatch** — leaf does not match any filter. See [Certificate filters](#certificate-filters) below. +2. **Wrong Namespace** (missing `.` suffix, or CA not configured on the target Namespace) or **wrong auth method** (API-key-only Namespace). + +Case (1) is an authorization rejection *after* a clean TLS handshake. The exact wire signal is release-dependent — you may see a TLS-layer alert (`remote error: tls: bad certificate`) or a post-handshake gRPC `PERMISSION_DENIED`; read the observed error rather than pinning a string. An untrusted CA fails earlier, at the handshake (`remote error: tls: unknown certificate authority`) → [certificates.md → Accepted client CA set](certificates.md#accepted-client-ca-set-mtls-cloud). + +### Certificate filters + +Cloud Namespace certificate filters are configured at the Namespace level and restrict which end-entity (leaf) certificates may authenticate, even when the issuing CA is in the accepted-client-ca set. Per the Cloud docs: "To limit access to specific end-entity certificates, create certificate filters. Each filter contains values for one or more of the following fields: commonName (CN), organization (O), organizationalUnit (OU), subjectAlternativeName (SAN)." "Corresponding fields in the client certificate must match every specified value in the filter." + +Matching rules worth knowing when diagnosing a filter mismatch: + +- Values are case-insensitive. +- Without wildcards, each value must match exactly. +- A single `*` wildcard may appear at the beginning or end of a value (but not both, and not alone). +- Maximum 25 filters per Namespace. + +Inspect the DN fields on the client cert: + +```bash +openssl x509 -in client.pem -noout -subject +# -subject prints the cert's Subject (CN, O, OU, etc.) — see certificates.md openssl recipes +``` + +Inspect / change filters with `tcld`: + +```bash +# View current filters +tcld namespace certificate-filters export \ + --namespace \ + --certificate-filter-file +# Command: + +# Clear all filters (allows any cert that chains to an accepted CA) +tcld namespace certificate-filters clear \ + --namespace +# Command: + +# Replace filters with a JSON file +tcld namespace certificate-filters import \ + --namespace \ + --certificate-filter-file +# Command: + +# Add additional filters +tcld namespace certificate-filters add \ + --namespace \ + --certificate-filter-file +# Command: +``` + +Cloud UI path is documented alongside these commands. + +Caution on clearing: "Using this command allows _any_ client certificate that chains up to a configured CA certificate to connect to the Namespace." + +### mTLS does not use Cloud roles + +After CA (+ optional filter) acceptance, the data plane grants full Namespace access for WorkflowService APIs. There is no User/Service Account principal with Read/Write/Admin RBAC for the cert. + +Clean `openssl s_client` (`Verify return code: 0 (ok)`) + a `PERMISSION_DENIED` (or TLS-layer rejection) on the first data-plane call → check filters, Namespace name, or auth method — not `tcld user get`. Cloud RBAC applies to **API keys** (next section), not mTLS data-plane callers. + +## Cloud role and permission model + +Applies to **API keys** (and UI/SSO users / Service Accounts), not to mTLS data-plane callers. + +Cloud access is governed on two axes — account-level roles (who can do account operations) and Namespace-level permissions (who can do data-plane operations in a given Namespace) — with important coupling via inheritance. Reference: `docs/cloud/manage-access/roles-and-permissions.mdx`. + +### Account-level roles + +The CLI `--account-role` flag on `tcld user invite` and `tcld user set-account-role` accepts: + +> `[Admin Developer FinanceAdmin MetricsRead Owner Read]` + +Concept docs use display names (Account Owner, Global Admin, Developer, Finance Admin, Read-Only, Metrics Read). Map CLI `Admin` ↔ Global Admin. + +**Inheritance:** Account Owner and Global Admin automatically have Namespace Admin on every Namespace in the account. Developers get Namespace Admin on Namespaces they create (revocable). + +### Namespace-level permissions + +Namespace permissions are set via `--namespace-permission =` on `tcld user invite` and `tcld user set-namespace-permissions`. The enum is: + +> Available namespace permissions: `Admin` | `Write` | `Read`. + +The values are case-sensitive (the docs write them capitalized; the invite example uses `ns1=Admin --namespace-permission ns2=Write`). + +What each permission grants is summarized in the concept table: Read observes activity; Write starts / signals / cancels / terminates / resets Workflows and polls Task Queues; Namespace Admin does all of that plus Namespace administration. + +Also check **Service Account** ownership of the API key (`tcld service-account get`, Cloud UI). Keys inherit the owner's roles. + +### Inviting and adjusting users with tcld + +```bash +# Invite a user with account role + per-namespace permissions +tcld user invite \ + --user-email \ + --account-role Developer \ + --namespace-permission =Admin \ + --namespace-permission =Write +# Command: +# --user-email (required, NOT --email): +# --account-role (required), enum Admin|Developer|FinanceAdmin|MetricsRead|Owner|Read: +# --namespace-permission (repeatable), format =: + +# Change a user's account role after the fact +tcld user set-account-role \ + --user-email \ + --account-role Developer +# Command: + +# Change a user's Namespace permissions +tcld user set-namespace-permissions \ + --user-email \ + --namespace-permission =Write +# Command: + +# Look up what a user has +tcld user get --user-email +# Command: + +# List users scoped to a Namespace +tcld user list --namespace +# Command: +``` + +User groups follow a parallel shape but a different namespace-role syntax — `-` with `` in `admin | read | write`, instead of the `=` used on `tcld user`. The user-group account-role enum is also broader than the `tcld user` one. If a triage turns up `PERMISSION_DENIED` for a principal that inherits its access from a group, reach for `tcld user-group get --group-id ` rather than `tcld user get`. + +### Discriminating with a read-vs-write smoke test + +If a user can do one operation but not another in the same session, the question is almost always which Namespace permission they hold (API keys / users / Service Accounts only): + +- `temporal workflow list --limit 1 ...` — requires `Read` or higher (or Owner/Admin inheritance). Success implies the identity is authenticated and has at least `Read` on the Namespace. +- `temporal workflow start ...` or `temporal workflow signal ...` — requires `Write` or higher. +- `tcld namespace get --namespace ` is control plane and needs Namespace **Read** (or Owner/Admin inheritance) — not Admin-only. A control-plane-only failure with data-plane access intact still points at account role / missing Namespace grant, not broken data-plane auth. + +If the read smoke test returns data and the write smoke test returns `PERMISSION_DENIED`, the diagnosis is "identity authenticated, permission insufficient" rather than "broken auth." + +## Quick routing + +| Error text shape | Layer | Go to | +|---|---|---| +| gRPC `UNAUTHENTICATED` with API key | 4 | [API-key authentication](#api-key-authentication) | +| API key rejected on Namespace Endpoint (mTLS-only / dual-auth Namespace) | 4 | Try the API Regional Endpoint | +| gRPC `UNAUTHENTICATED` with mTLS | 3–4 boundary | First confirm TLS with `openssl s_client` per [certificates.md](certificates.md); if TLS is clean, this is a post-TLS rejection — see [mTLS authentication after TLS completes](#mtls-authentication-after-tls-completes) | +| gRPC `PERMISSION_DENIED` on data-plane (API key) | 4 | [Cloud role and permission model](#cloud-role-and-permission-model) | +| gRPC `PERMISSION_DENIED` on data-plane (mTLS, TLS clean) | 4 | [Certificate filters](#certificate-filters) / wrong Namespace / auth method — **not** Cloud RBAC | +| gRPC `PERMISSION_DENIED` on `tcld` control-plane call | 4 (control plane) | Account-level role — see [Account-level roles](#account-level-roles) | +| `remote error: tls: bad certificate` | 3–4 | [certificates.md → Accepted client CA set](certificates.md#accepted-client-ca-set-mtls-cloud) — rejected cert (untrusted CA or filter mismatch) | +| `remote error: tls: unknown certificate authority` | 3 | [certificates.md → Accepted client CA set](certificates.md#accepted-client-ca-set-mtls-cloud) | +| gRPC `UNAVAILABLE` | 2–3 | Not an auth code — peel the wrapped cause; see [connectivity.md](connectivity.md) and [certificates.md](certificates.md) | +| gRPC `RESOURCE_EXHAUSTED` | 4+ | Not an auth code — see [rate-limits.md](rate-limits.md) | diff --git a/plugins/temporal/skills/temporal-ops/references/triage/blob-size-limits.md b/plugins/temporal/skills/temporal-ops/references/triage/blob-size-limits.md new file mode 100644 index 0000000..fd9445b --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/triage/blob-size-limits.md @@ -0,0 +1,88 @@ +# Blob Size Limits + +Temporal enforces size limits on the data that passes between the Temporal Client, Workers, and the Temporal Service. There are two distinct limits, each producing different error messages and behaviors. + +--- + +## Payload size limit (2 MB) + +The Temporal Service enforces a size limit on individual payloads. This limit is **2 MB** on Temporal Cloud, and is configurable on self-hosted deployments with a default of 2 MB. + +A payload represents the serialized binary data for the input and output of Workflows and Activities. + +### Error messages + +- `WORKFLOW_TASK_FAILED_CAUSE_PAYLOADS_TOO_LARGE` +- `[TMPRL1103] Attempted to upload payloads with size that exceeded the error limit.` +- `BadScheduleActivityAttributes: ScheduleActivityTaskCommandAttributes.Input exceeds size limit` +- `Complete result exceeds size limit` +- `CompleteWorkflowExecutionCommandAttributes.Result exceeds size limit` +- `WORKFLOW_TASK_FAILED_CAUSE_BAD_UPDATE_WORKFLOW_EXECUTION_MESSAGE` + +### Error behavior by SDK + +**Python SDK 1.23.0+:** The SDK fails the Workflow Task with cause `WORKFLOW_TASK_FAILED_CAUSE_PAYLOADS_TOO_LARGE`. The Workflow is not terminated and remains open, so you can deploy a fix and allow the Workflow to continue. + +**All other SDK versions:** The behavior depends on whether the oversized payload is an input or a result: + +- **Inputs (Workflow input, Activity input):** The Temporal Service rejects the command and terminates the Workflow. You need to resolve the issue and restart the Workflow. +- **Activity result:** The Temporal Service rejects the Activity completion and the Activity fails with an error. +- **Workflow result:** The Workflow gets stuck in a retry loop. The server rejects the `CompleteWorkflowExecution` command, and replay produces the same oversized result. + +### How to resolve + +1. **Claim check pattern** (recommended): Offload large payloads to an object store. Pass references to stored payloads within the Workflow instead of the actual data. Retrieve the payloads from the object store when needed. + + The claim check pattern is built into the SDKs as **External Storage**, currently in **Pre-release**. Or implement your own via a custom Payload Codec. + +2. **Compression**: Use compression with a custom Payload Codec. This may address the immediate issue, but if payload sizes continue to grow, the problem can arise again. + +--- + +## gRPC message size limit (4 MB) + +All communication between the Temporal Client, Workers, and the Temporal Service uses gRPC, which enforces a **4 MB** limit on each request. This limit is **fixed at 4 MB on Temporal Cloud**. On self-hosted it is technically configurable, but raising it requires changes across multiple layers (gRPC server config, event history limits, and the underlying Postgres row-size limit), so the recommended path is to fix the workflow design rather than raise the limit. + +A Workflow can hit this limit even when every individual payload is under 2 MB. Scheduling several Activities with moderate-sized inputs, or hundreds of Activities with tiny inputs in the same Workflow Task, can push the combined request past 4 MB. Activity results are also subject to this limit. + +### Error messages + +- `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE` +- `ScheduleToCloseTimeout` (Activities only, see error behavior below) + +### Error behavior by SDK + +**Python SDK 1.23.0+:** The SDK fails the Workflow Task with cause `WORKFLOW_TASK_FAILED_CAUSE_PAYLOADS_TOO_LARGE`. The Workflow is not terminated and remains open. For Activities, the Activity fails with an explicit error instead of timing out silently. + +**All other SDK versions:** + +- **Workflow Tasks:** The Workflow gets stuck in a retry loop that is not visible in the Event History. The Worker sends all commands back to the Temporal Service; if the combined size exceeds 4 MB, the SDK catches the gRPC error and sends a failed Workflow Task response with cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`. Replay produces the same oversized request every time, so the Workflow never makes progress. +- **Activity Tasks:** The Activity gets stuck in a retry loop or exits with a `ScheduleToCloseTimeout`. The Activity executes successfully, but the Worker cannot deliver the oversized result over gRPC. The server never receives the completion, so it retries the Activity. Each retry completes but fails to deliver the result. The Activity retries until `ScheduleToCloseTimeout` expires. If no `ScheduleToCloseTimeout` is set, it retries indefinitely until the Workflow is manually terminated. The `ResourceExhausted` gRPC error only appears in Worker logs. + +### How to resolve + +1. **Break larger batches into smaller sizes:** + - **Workflow-level batching:** Process Activities or Child Workflows in smaller batches. Iterate through each batch, waiting for completion before the next. + - **Workflow Task-level batching:** Execute Activities in smaller batches within a single Workflow Task. Introduce brief pauses or sleeps between batches. + +2. If the request is large because of payload sizes (not the number of commands), refer to the Payload size limit solutions above. + +--- + +## Quick diagnostic table + +| Symptom | Likely limit | Next step | +|---|---|---| +| `WORKFLOW_TASK_FAILED_CAUSE_PAYLOADS_TOO_LARGE` | Payload (2 MB) | Check individual payload sizes; implement claim check pattern | +| `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE` | gRPC (4 MB) | Reduce batch size or number of concurrent commands per Workflow Task | +| Workflow stuck in invisible retry loop | gRPC (4 MB) on Workflow Task | Check Worker logs for `ResourceExhausted`; reduce batch size | +| Activity retries indefinitely with no visible error | gRPC (4 MB) on Activity result | Check Worker logs for `ResourceExhausted`; implement claim check for result | +| `ScheduleToCloseTimeout` on Activity that completes successfully | gRPC (4 MB) on Activity result | Check Worker logs for `ResourceExhausted`; reduce result size | +| `CompleteWorkflowExecutionCommandAttributes.Result exceeds size limit` | Payload (2 MB) on Workflow result | Reduce Workflow result size or use claim check pattern | + +--- + +## Sibling skill pointers + +- For implementing the claim check pattern or External Storage in SDK code, see the developer skill (`skill-temporal-developer`). +- For worker tuning to manage batch sizes and concurrency, see the worker tuning skill (`skill-temporal-workertuning`). diff --git a/plugins/temporal/skills/temporal-ops/references/triage/certificates.md b/plugins/temporal/skills/temporal-ops/references/triage/certificates.md new file mode 100644 index 0000000..6eb47e1 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/triage/certificates.md @@ -0,0 +1,548 @@ +# Certificates + +Diagnose TLS and x509 failures against Temporal Cloud or a self-hosted frontend. This file covers layer 3 of the [diagnostic ladder](diagnostic-ladder.md). + +Prerequisite: before reading this file, rule out layers 1 and 2 (DNS, TCP). TLS errors can masquerade as connectivity errors when a middlebox drops packets mid-handshake, so confirm TCP reachability first via [connectivity.md](connectivity.md#connection-refused). The reverse also happens: when a TLS handshake completes but the cert is invalid, the Go client emits cert-shaped errors that are firmly in layer 3. + +**Ports referenced in this file:** SDK and CLI gRPC traffic to a Namespace (Cloud or self-hosted frontend) uses **7233**, for both mTLS and API-key Namespaces. The web UI uses **443**. All examples below assume 7233 unless stated. + +**Flexible Auth (mTLS + API key on the same Namespace):** When a Namespace has both auth methods enabled concurrently, a TLS-layer failure on the mTLS leg can be masked by a successful API-key path (and vice versa). Before triaging, confirm which auth method the failing client is using. If the same client is configured for both, disable one at a time to isolate which leg is broken. The recipes in this file apply to the mTLS leg only; API-key auth failures live in [authentication.md](authentication.md). + +Out of scope here: +- DNS / TCP / endpoint / firewall → [connectivity.md](connectivity.md) (layers 1-2) +- gRPC `UNAUTHENTICATED` after a successful TLS handshake → [authentication.md](authentication.md) (layer 4) +- API-key auth semantics (API-key connections still ride TLS, so TLS-level issues in this file apply — but the API-key authorization check itself is not a TLS issue) → [authentication.md](authentication.md) +- gRPC `RESOURCE_EXHAUSTED` → [rate-limits.md](rate-limits.md) + +## Table of Contents + +- [Error-string origin cheat sheet](#error-string-origin-cheat-sheet) +- [Handshake failure](#handshake-failure) +- [Expired or not-yet-valid](#expired-or-not-yet-valid) +- [Unknown authority](#unknown-authority) +- [Hostname mismatch](#hostname-mismatch) +- [Server name override](#server-name-override) +- [Key does not match cert](#key-does-not-match-cert) +- [Accepted client CA set (mTLS Cloud)](#accepted-client-ca-set-mtls-cloud) + - [Certificate filters](#certificate-filters) +- [Certificate requirements (Cloud mTLS)](#certificate-requirements-cloud-mtls) +- [Rotation and expiry notifications](#rotation-and-expiry-notifications) +- [Private key handling](#private-key-handling) +- [openssl recipes](#openssl-recipes) +- [TLS / cert error reference](#tls--cert-error-reference) + +## Error-string origin cheat sheet + +Strings in this file come from the Go standard library (client side) or the TLS peer (alert descriptions). Every quoted string below is tagged with its source so you can tell a Go-local complaint from a peer-originated alert. + +- **`x509: ...`** — emitted by Go's `crypto/x509` when the client rejects a peer cert locally. Source: `src/crypto/x509/verify.go`. +- **`tls: ...`** — emitted by Go's `crypto/tls` during the handshake (typically a local protocol-level error). Source: `src/crypto/tls/`. +- **`remote error: tls: `** — a TLS alert received from the *peer*, formatted by Go. The alert description words (`handshake failure`, `bad certificate`, `unknown certificate authority`, `expired certificate`, `internal error`, etc.) come from `src/crypto/tls/alert.go`; the `remote error: tls: ` prefix is added by Go when it surfaces the peer's alert. +- **`Failed reaching server: last connection error`** — surfaced by the Temporal Cloud connection path; the troubleshooting guide identifies an expired TLS certificate as a common root cause. + +If you have an error string that doesn't start with `x509:`, `tls:`, or `remote error: tls:`, it is probably not a TLS-layer error — re-check the layer above (connectivity.md) or below (authentication.md). + +## Handshake failure + +**Symptom shape:** +- Client side: `tls: handshake failure` or a gRPC `UNAVAILABLE` whose wrapped cause starts with `tls:` or `remote error: tls:`. +- Peer-alerted: `remote error: tls: handshake failure` (the word `handshake failure` is the TLS alert description emitted by the other side). + +**What it means:** the two sides did not agree on protocol version, cipher, or client authentication. On its own, `handshake failure` is not specific — it is the generic TLS alert when the peer cannot continue. Use `openssl s_client` (see [openssl recipes](#openssl-recipes)) to get a more specific line such as `certificate has expired`, `unknown certificate authority`, or a hostname-mismatch error. + +**First check:** reproduce the handshake from the same host that saw the failure: + +```bash +openssl s_client -connect :7233 -servername :7233 \ + -servername \ + -cert client.pem -key client.key \ + -showcerts -showcerts -cert ... -key ...` form is used by the Temporal troubleshooting guide.) + +Interpret by looking at the last few lines of output: + +| Output line | Interpretation | Where to go | +|---|---|---| +| `Verify return code: 0 (ok)` and certificate info printed | TLS succeeded | Not a TLS problem. Move to [authentication.md](authentication.md). | +| `Verify return code: 10 (certificate has expired)` or `x509: certificate has expired or is not yet valid` on the client | Cert past validity on server or client chain | [Expired or not-yet-valid](#expired-or-not-yet-valid) | +| `Verify return code: 19 (self signed certificate in certificate chain)` or `Verify return code: 20/21` | Client doesn't trust the server chain | [Unknown authority](#unknown-authority) | +| `tlsv1 alert unknown ca` / `remote error: tls: unknown certificate authority` | Server rejected the client CA | [Accepted client CA set](#accepted-client-ca-set-mtls-cloud) | +| `tlsv1 alert bad certificate` / `remote error: tls: bad certificate` | Server rejected the client cert (cert-filter mismatch, malformed cert, or wrong cert presented) | [Accepted client CA set](#accepted-client-ca-set-mtls-cloud) and [certificate filters in `docs/cloud/certificates.mdx`](https://docs.temporal.io/cloud/certificates#manage-certificate-filters) | +| Handshake opens TCP but closes with no TLS alert bytes | Middlebox dropping / TLS-inspecting proxy | Back off to [connectivity.md → firewall and proxy](connectivity.md#firewall-and-proxy) | + +## Expired or not-yet-valid + +**Symptom shapes:** +- Go client: `x509: certificate has expired or is not yet valid: ` followed by detail +- Peer alert: `remote error: tls: expired certificate` +- Temporal Cloud: `Failed reaching server: last connection error` — the troubleshooting guide names an expired TLS certificate as a common root cause. +- Workers that were fine yesterday stopped connecting overnight with no deploy. + +**What to check first — exact expiry times of each cert in play:** + +```bash +# Local cert file +openssl x509 -enddate -noout -in client.pem +# -enddate: print notAfter field +# -noout: no encoded output +# -in file: input file + +# Both notBefore and notAfter +openssl x509 -dates -noout -in client.pem +# -dates: Both Before and After dates + +# Server cert fetched from a live endpoint +openssl s_client -connect :7233 -servername /dev/null \ + | openssl x509 -enddate -noout +``` + +For the accepted-client-ca set on a Cloud Namespace, the troubleshooting guide lists expiry via `tcld namespace accepted-client-ca list` with `jq`: + +```bash +tcld namespace accepted-client-ca list \ + --namespace . \ + | jq -r '.[0].notAfter' +# tcld namespace accepted-client-ca list: +# --namespace (-n) modifier: +# Recipe exactly as written in: +``` + +**Clock skew — the "not yet valid" variant:** the same Go error string covers both "expired" and "not yet valid" (`x509: certificate has expired or is not yet valid: `). If `date -u` on the client disagrees with a time authority by more than the cert's overlap window (common in containers with no NTP, on appliances with a dead RTC battery, or right after a host boot), a cert that is in fact valid will still fail verification. Verify with `date -u` before regenerating anything. + +**Classification:** + +- **End-entity (leaf) cert expired, CA still valid:** regenerate the leaf against the existing CA. See [openssl recipes → issue a new leaf with tcld](#issue-a-new-leaf-with-tcld). +- **CA cert expired (so all leaves under it fail):** regenerate CA + leaf, upload the new CA *alongside* the existing one before removing the old one, then distribute new leaves and restart clients. See [Rotation and expiry notifications](#rotation-and-expiry-notifications). The Cloud docs describe this as a "rollover process" that "enables your Namespace to serve both CA certificates for a period of time until traffic to your old CA certificate ceases." +- **Temporal Cloud server-side cert expired:** you don't manage the server side on Cloud. Open a support ticket per `docs/troubleshooting/last-connection-error.mdx`. +- **Self-hosted server cert expired:** rotate the frontend's server TLS cert on your deployment. No tcld involvement. + +:::caution +An expired root CA certificate invalidates all downstream certificates, per `docs/cloud/get-started/certificates.mdx:31`. +::: + +## Unknown authority + +**Symptom shapes:** +- Go client: `x509: certificate signed by unknown authority` +- Go client (verify chain): `x509: no valid chains built` or `x509: failed to load system roots and no roots provided` +- Peer alert: `remote error: tls: unknown certificate authority` +- Peer alert: `remote error: tls: bad certificate` + +**Two directions this error travels — check which side is complaining before you act:** + +1. **Client does not trust server (client-side `x509: ...`).** The client cannot validate the server certificate against any root it knows about. One thing to check is whether the client's trust store is populated — minimal container images often ship without system CAs — and whether the client is being pointed at a non-Temporal endpoint (TLS-inspecting proxy). The Go client does not ship its own root bundle; it relies on the host's trust store or an explicit `--tls-ca-path` / `TEMPORAL_TLS_CA`. +2. **Server does not trust client (peer-alert `remote error: tls: unknown certificate authority`).** For mTLS, the Namespace's accepted-client-ca set does not contain the CA that signed the client cert. Fix in [Accepted client CA set](#accepted-client-ca-set-mtls-cloud). + +**Verify locally that the client cert chains to the CA you think it chains to:** + +```bash +openssl verify -CAfile ca.pem client.pem +# -CAfile file: Certificate Authority file +``` + +If intermediates exist, supply them via `-untrusted`: + +```bash +openssl verify -CAfile root-ca.pem -untrusted intermediate.pem client.pem +# -untrusted file: untrusted certificates file +``` + +## Hostname mismatch + +**Symptom shapes:** +- `x509: certificate is valid for , not ` +- `x509: certificate is not valid for any names, but wanted to match ` +- `x509: cannot validate certificate for ` +- Go client may also emit: `x509: certificate relies on legacy Common Name field, use SANs instead` when a server cert has no SAN and only a CN. Rare on modern clients: this string is from the deprecated `GODEBUG=x509ignoreCN=0` path, which Go 1.17 removed. Go 1.17+ rejects the cert with a hostname-mismatch error instead. Treat the legacy-CN string as a Go 1.15–1.16 era signal; if you see it on a current build, the client is pinned to an older Go runtime. + +**What it means:** the hostname the client asked for does not match any Subject Alternative Name (or DNSName) on the server certificate the peer presented. + +**First check:** which hostname is the client asking for? + +```bash +# Inspect server cert SANs as actually served: +openssl s_client -connect :7233 -servername /dev/null \ + | openssl x509 -text -noout \ + | grep -A1 "Subject Alternative Name" +# -text: print certificate in text form +``` + +**Common cause on Temporal Cloud:** connecting by VPC-endpoint DNS name (PrivateLink) or GCP PSC IP, or by Regional Endpoint, without overriding the TLS server name. The TLS peer serves the Namespace's certificate, whose SAN is the Namespace Endpoint hostname — the VPC-endpoint hostname is not in it. Fix in the next section. + +## Server name override + +**Symptom:** TLS handshake fails with hostname-mismatch errors (see [Hostname mismatch](#hostname-mismatch) above) when connecting through a PrivateLink VPC endpoint, a GCP Private Service Connect IP, or a Cloud Regional Endpoint. + +**What the Cloud docs say:** + +- When a client uses a PrivateLink / PSC endpoint instead of the Namespace Endpoint DNS name, the docs instruct you to "Set TLS configuration to override the TLS server name (e.g., my-namespace.my-account.tmprl.cloud)." +- When a client uses a Regional Endpoint with mTLS, the docs say: "the Temporal Client must set the `server_name` property to `` in its request to the value of the Namespace endpoint. This tells the client to expect a different SNI header during the TLS handshake, since the request to the regional endpoint is redirected to the specific Namespace." +- Temporal recommends configuring private DNS instead, so the Namespace Endpoint hostname resolves to the VPC endpoint directly and no server-name override is needed. + +**How to override on each client — verified forms only:** + +`temporal` CLI flag (value: the Namespace Endpoint, e.g. `..tmprl.cloud`): + +```bash +temporal workflow list \ + --address vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233 \ + --namespace . \ + --tls-cert-path client.pem --tls-key-path client.key \ + --tls-server-name ..tmprl.cloud +# --tls-server-name: Overrides the target TLS server name +# --tls-cert-path: Path to x509 certificate +# --tls-key-path: Path to private certificate key +``` + +`temporal` CLI env var equivalent: + +```bash +export TEMPORAL_ADDRESS=vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233 +export TEMPORAL_TLS_CERT=/path/to/cert.pem +export TEMPORAL_TLS_KEY=/path/to/cert.key +export TEMPORAL_TLS_SERVER_NAME=..tmprl.cloud +# TEMPORAL_TLS_SERVER_NAME: Override for target TLS server name +# TEMPORAL_TLS_CERT: Path to x509 certificate +# TEMPORAL_TLS_KEY: Path to private certificate key +# TEMPORAL_ADDRESS: Host and port for the Temporal Frontend Service + +temporal workflow list --namespace . +``` + +The exact form above — `TEMPORAL_ADDRESS=vpce-...:7233` paired with `TEMPORAL_TLS_SERVER_NAME=my-namespace.my-account.tmprl.cloud` — is written out in the Cloud connectivity guide. + +`grpcurl` (useful as a SDK-independent probe) — exactly the form the Cloud docs give: + +```bash +grpcurl \ + -servername ..tmprl.cloud \ + -cert path/to/cert.pem \ + -key path/to/cert.key \ + vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233 \ + temporal.api.workflowservice.v1.WorkflowService/GetSystemInfo +# Recipe as written in: +``` + +**For SDK clients:** the SDK concept is the same — set the TLS `ServerName` (Go / Java / .NET / Python / TypeScript) to the Namespace Endpoint hostname. The exact property name varies by SDK; refer to each SDK's client-connection doc linked from `docs/cloud/certificates#configure-clients-to-use-client-certificates` and cross-check against `skill-temporal-developer`. This triage file deliberately does not spell SDK APIs out, to avoid drift. + +## Key does not match cert + +**Symptom shapes:** +- `tls: failed to find any PEM data in certificate input` (the file is empty, wrong format, or the wrong file) +- Go's `tls.LoadX509KeyPair` surfaces a keypair mismatch at client start. The precise string varies across Go versions, so don't pattern-match on it; the *shape* is an error at client dial saying the cert and key don't pair. + +**Verify by comparing modulus hashes of the cert and the private key:** + +```bash +# RSA +openssl x509 -modulus -noout -in client.pem | shasum -a 256 +openssl rsa -modulus -noout -in client.key | shasum -a 256 +# -modulus: print the RSA key modulus +# The two digests must be identical. +``` + +If the modulus hashes disagree, the cert and key file are from different keypairs. Find the key that was generated alongside this cert — commonly in the same directory — or regenerate both together. + +(For ECDSA keys, `-modulus` does not apply; compare public keys with `openssl pkey -in client.key -pubout` against `openssl x509 -in client.pem -pubkey -noout`.) + +## Accepted client CA set (mTLS Cloud) + +On Temporal Cloud, an mTLS Namespace authenticates a client by validating the client cert against the CA set configured on the Namespace. The server-side error when the CA is not trusted is `remote error: tls: unknown certificate authority`. When a CA *is* trusted but the end-entity cert fails other checks (certificate filter mismatch, malformed cert), the server sends `remote error: tls: bad certificate`. + +### Certificate filters + +Certificate filters are SAN/CN-based allow rules configured on the Namespace that further restrict which client certs are accepted, even when the signing CA is in the accepted set. A filter specifies allowed values for the leaf's Common Name, Subject Organization, Subject Organizational Unit, or SANs (DNS names, URIs, emails); a client cert that chains to a trusted CA but does not match any filter is rejected with `remote error: tls: bad certificate`. + +Filters are the common cause of "the CA is trusted, the cert looks fine, but the handshake still fails." If you see `remote error: tls: bad certificate` and the local `openssl verify -CAfile ...` check passes, suspect a filter mismatch before regenerating certs. + +```bash +# Inspect what filters (if any) are configured on the Namespace +tcld namespace certificate-filters export \ + --namespace . +# Reference: + +# Inspect the SANs / CN / OU on the client cert that's failing +openssl x509 -in client.pem -noout -subject -ext subjectAltName +``` + +If the filter set is non-empty, the leaf cert's subject fields and SANs must match at least one filter entry. Either re-issue the leaf with subject/SAN values that match an existing filter, or update the filter set to allow the new cert. + +**List what the Namespace currently accepts:** + +```bash +tcld namespace accepted-client-ca list \ + --namespace . +# Command: +# --namespace: +``` + +**Add a new CA to the accepted set:** + +`add` **appends** to the existing set without removing other CAs, so it is the safe verb for the upload-alongside step of a rollover. + +```bash +tcld namespace accepted-client-ca add \ + --namespace . \ + --ca-certificate-file +# Command: +# --ca-certificate-file: +``` + +**Remove a CA (by fingerprint, safer than by PEM):** + +```bash +tcld namespace accepted-client-ca remove \ + --namespace . \ + --ca-certificate-fingerprint +# Command: +# --ca-certificate-fingerprint (--fp): +``` + +**Set the whole bundle at once (zero-downtime rollover):** + +The Cloud docs describe a concat-old-plus-new-then-set pattern for rolling over CA certs without dropping traffic: + +```bash +# 1. Create a file with old + new CA PEM blocks concatenated. +# 2. Run: +tcld namespace accepted-client-ca set \ + --namespace . \ + --ca-certificate-file +# Command and rollover procedure: +# Same procedure in: +# 3. Wait until all clients present leaves signed by the new CA (operator-confirmed; Cloud shows no drain signal). +# 4. Create a file with only the new CA and run the set command again. +``` + +:::caution +`set` **replaces the entire accepted-CA bundle** with exactly what you pass, and it does not prompt for confirmation. Passing only the new CA silently drops every other trusted CA and locks out any client still presenting a leaf under them. Use `add` to append a CA during a rollover; use `set` only with a deliberately concatenated old+new bundle. Trust-changing commands (`set`, `remove`, rotation) are high-consequence: run them with operator confirmation of the blast radius, not autonomously. +::: + +**Verify locally before uploading** that the client cert chains to the CA you're about to upload: + +```bash +openssl verify -CAfile new-ca.pem client.pem +# Expected: client.pem: OK +``` + +If this fails locally, it will fail at the peer too. Fix before touching the Namespace. + +## Certificate requirements (Cloud mTLS) + +The docs state hard requirements for any CA or leaf cert you upload to a Cloud Namespace. These are where subtle TLS rejections come from when a cert "looks fine" but the peer sends `remote error: tls: bad certificate` anyway. Quoting the requirements as listed at `docs/cloud/get-started/certificates.mdx:62-96`: + +**CA certificates:** +- X.509v3. +- Each cert in a bundle is either a root or issued by another cert in the bundle. +- Each cert includes `CA: true`. +- Cannot be a well-known CA (DigiCert, Let's Encrypt, etc.) unless the user also specifies certificate filters. +- Signing algorithm must be RSA or ECDSA and must include SHA-256 or stronger. SHA-1 and MD5 cannot be used. +- Cannot be generated with a passphrase. +- Bundle ≤ 16 CA certs, ≤ 32 KB before base64 encoding. +- In a full end-entity → root chain, each certificate must have a unique Distinguished Name (DN comparison is case-insensitive). + +**End-entity (leaf) certificates:** +- X.509v3. +- Basic constraints must include `CA: false`. +- Key usage must include Digital Signature. +- Signing algorithm: RSA or ECDSA with SHA-256 or stronger. + +**Certificate Revocation Lists:** Temporal does not support or check CRLs (or OCSP); customers are expected to keep certificates up to date. Because there is no revocation list, plan for revocation by other means: + +- **Short certificate lifetimes** are the primary control: a short-lived leaf bounds how long a stolen key is usable. Set leaves to expire before their issuing CA. +- **Removing a CA from the accepted set** (`tcld namespace accepted-client-ca remove`) is the hard kill switch. This rejects *all* leaves signed by that CA, so confirm the blast radius before running it. +- **Certificate filters** scope acceptance to specific leaf identities (CN/OU/Subject Organization/SAN); tightening or removing a filter cuts off specific certs without rotating the CA. +- There is **no per-leaf revocation**: to cut off a single leaf before it expires, you must rotate or remove its CA. Size leaf lifetimes accordingly. + +**Algorithm choice when generating with tcld:** `tcld gen ca` defaults to ECDSA P-384; `--rsa-algorithm` (alias `--rsa`) switches to a 4096-bit RSA key pair. + +The accepted set above (RSA or ECDSA, SHA-256+) is a floor, not a recommendation: Temporal does not mandate a specific key size or curve for customer CAs. Match key strength to certificate lifetime. RSA-2048 (~112-bit) is acceptable for short-lived certs today, but NIST (SP 800-131A Rev. 2) deprecates 112-bit strength after 2030, so a long-lived CA root that must stay trusted past then should use ECDSA (P-256/P-384) or RSA-3072 or larger (4096 preferred; the `--rsa` flag emits 4096). The tcld default, ECDSA P-384, is a good choice for most cases. + +**Duration caps when generating with tcld:** `tcld gen ca` has a maximum duration of 1 year (`-d 1y`). You must set an end-entity cert to expire before its root CA. + +## Rotation and expiry notifications + +**Notifications.** Temporal Cloud sends email notifications before CA expiry. The notifications doc lists "Certificate Expiring in 15 days" as an admin notification sent to Global Administrators, Namespace Administrators, and Account Owners. The certificates guide states: "Temporal Cloud begins sending notifications 15 days before expiration." + +**Rollover strategy.** The Cloud docs prescribe a zero-downtime rollover pattern: add the new CA alongside the existing one (`accepted-client-ca add`), wait until every client has rolled to leaves signed by the new CA, then remove the old CA. Cloud exposes no signal for when traffic has fully shifted, so treat this as operator-driven: confirm your worker fleet is on new-CA leaves before removing the old CA. The same shape applies whether you use the UI or `tcld`. + +**Rotation command sequence (issue with tcld, upload with tcld):** the generated `*.key` files are secrets; see [Private key handling](#private-key-handling). + +```bash +# Issue a new CA cert (if rotating CA). Default is ECDSA P-384. +tcld generate-certificates certificate-authority-certificate \ + --organization \ + --validity-period 1y \ + --ca-certificate-file new-ca.pem \ + --ca-key-file new-ca.key +# Command (alias: tcld gen ca): +# --organization (--org): +# --validity-period (-d): +# --ca-certificate-file (--ca-cert): +# --ca-key-file (--ca-key): + +# Issue a new end-entity (leaf) cert against a CA +tcld generate-certificates end-entity-certificate \ + --organization \ + --validity-period 364d \ + --ca-certificate-file new-ca.pem \ + --ca-key-file new-ca.key \ + --certificate-file client.pem \ + --key-file client.key +# Command (alias: tcld gen leaf): +# --certificate-file (--cert): +# --key-file (--key): + +# Upload concatenated old+new CA bundle to the Namespace, then (after drain) upload new-only bundle. +tcld namespace accepted-client-ca set \ + --namespace . \ + --ca-certificate-file +# Procedure: +# --namespace: +``` + +**When the CA itself is already expired** (the rollover window was missed), a new CA must be uploaded before any client can reconnect. This is the "cert expired at 3 a.m." shape; see [recipes.md](recipes.md) for the full step-by-step. + +## Private key handling + +The recipes in this file generate private keys (`new-ca.key`, `client.key`). Treat them as secrets, and treat the CA private key as a root credential: it can mint client certificates the Namespace will accept, and there is no revocation backstop if it leaks (see [Certificate Revocation Lists](#certificate-requirements-cloud-mtls)). + +- **Never print, echo, `cat`, or log private-key contents.** To check whether a key matches a cert, use the modulus/fingerprint comparisons in [Compare a keypair](#compare-a-keypair) and [Key does not match cert](#key-does-not-match-cert) — those expose no secret material. +- **The key is plaintext on disk.** Cloud requires CA keys to be generated without a passphrase, so the file itself is the secret. Restrict permissions (`chmod 600`), never commit it to source control (add `*.key` to `.gitignore`), and keep the long-term CA key in a secrets manager or offline storage. +- **Don't leave key material in working or shared directories.** Where a key should live long-term depends on the environment, so flag leftover key files to the operator to place in durable secret storage rather than guessing or deleting them. + +:::caution +Generating or rotating keys locally is safe, but anything that changes a Namespace's trust (`accepted-client-ca set`/`remove`, rotation) or deletes key material is high-consequence: it can lock out every worker on the Namespace or destroy an unrecoverable CA key. Run those steps with operator confirmation, not autonomously. +::: + +## openssl recipes + +These are reference-card forms; each flag is cross-linked to the openssl usage output. + +### Inspect a local cert + +```bash +# Full text +openssl x509 -in cert.pem -noout -text +# -in / -noout / -text + +# Subject and issuer +openssl x509 -in cert.pem -noout -subject -issuer +# -subject / -issuer + +# Validity dates +openssl x509 -in cert.pem -noout -dates +# -dates: both Before and After + +# Fingerprint (for tcld --ca-certificate-fingerprint) +openssl x509 -in cert.pem -noout -fingerprint +# -fingerprint +``` + +Note: LibreSSL (the `openssl` shipped with macOS as of LibreSSL 3.x) does not support the upstream OpenSSL `-ext ` filter on `openssl x509`. To see SANs on LibreSSL, use `openssl x509 -text -noout` and read the `X509v3 Subject Alternative Name` section, or use grep as in [Hostname mismatch](#hostname-mismatch). + +### Verify a chain + +```bash +# Leaf + root +openssl verify -CAfile ca.pem client.pem + +# Leaf + root + intermediate +openssl verify -CAfile root-ca.pem -untrusted intermediate.pem client.pem +# -CAfile / -untrusted +``` + +### Test a live endpoint + +```bash +# Public-internet or mTLS Namespace, inspect the peer's cert chain +openssl s_client -connect :7233 \ + -servername \ + -showcerts /dev/null \ + | openssl x509 -text -noout + +# Full mTLS handshake (what the Temporal troubleshooting guide uses) +openssl s_client -connect ..tmprl.cloud:7233 \ + -showcerts \ + -cert client.pem -key client.key \ + -tls1_2 `; the alert description is one of the words listed in `src/crypto/tls/alert.go` (`handshake failure`, `bad certificate`, `unknown certificate authority`, `expired certificate`, etc.). + +### Compare a keypair + +```bash +openssl x509 -in cert.pem -modulus -noout | shasum -a 256 +openssl rsa -in key.pem -modulus -noout | shasum -a 256 +# Digests must match for the files to be from the same keypair. +``` + +### Self-signed cert workflow (self-hosted or temporary) + +The troubleshooting guide suggests using `temporal namespace describe` with explicit TLS flags when working with self-signed certs: + +```bash +temporal namespace describe \ + --namespace . \ + --address \ + --tls-cert-path \ + --tls-key-path +# Recipe as written in: +``` + +### Issue a new leaf with tcld + +```bash +tcld gen leaf \ + --org \ + -d 364d \ + --ca-cert ca.pem --ca-key ca.key \ + --cert client.pem --key client.key +# tcld gen leaf = tcld generate-certificates end-entity-certificate +# Aliases listed in: +# Example as written in: +``` + +## TLS / cert error reference + +Each row is tagged with the string's origin. When an error doesn't fit any row here, re-read [Error-string origin cheat sheet](#error-string-origin-cheat-sheet) to decide whether it is really a TLS-layer error at all. + +| Error text (exact, as emitted) | Origin | One thing to check | +|---|---|---| +| `x509: certificate has expired or is not yet valid: ` | Go x509 | [Expired or not-yet-valid](#expired-or-not-yet-valid); also check `date -u` for clock skew | +| `x509: certificate signed by unknown authority` | Go x509 | Client doesn't trust peer's root. [Unknown authority](#unknown-authority) | +| `x509: certificate is valid for , not ` | Go x509 | [Server name override](#server-name-override) | +| `x509: cannot validate certificate for ` | Go x509 | [Hostname mismatch](#hostname-mismatch) | +| `x509: no valid chains built` | Go x509 | Chain does not reach a trusted root; verify with `openssl verify -CAfile ...` | +| `x509: a root or intermediate certificate is not authorized to sign for this name: ` | Go x509 | Name-constraints extension rejects the leaf | +| `x509: certificate is not authorized to sign other certificates` | Go x509 | Cert with `CA: false` is being used as an issuer | +| `x509: failed to load system roots and no roots provided` | Go x509 | Minimal container missing `ca-certificates`; or `TEMPORAL_TLS_CA` / `--tls-ca-path` not set | +| `x509: certificate relies on legacy Common Name field, use SANs instead` | Go x509 (Go 1.15–1.16 era) | Peer cert has no SAN, only CN; re-issue with SANs. Go 1.17+ emits hostname-mismatch instead; if you see this on current Go, the client is pinned to an older runtime | +| `tls: handshake failure` | Go tls (local) | See [Handshake failure](#handshake-failure) and reproduce with `openssl s_client` | +| `remote error: tls: handshake failure` | peer alert | Peer rejected the handshake; reproduce with `openssl s_client` for the alert description | +| `remote error: tls: bad certificate` | peer alert | [Accepted client CA set](#accepted-client-ca-set-mtls-cloud); also check certificate filters | +| `remote error: tls: unknown certificate authority` | peer alert | [Accepted client CA set](#accepted-client-ca-set-mtls-cloud) | +| `remote error: tls: expired certificate` | peer alert | [Expired or not-yet-valid](#expired-or-not-yet-valid) | +| `remote error: tls: internal error` | peer alert | Peer-side failure, not a cert problem on this end; retry and, if persistent on Cloud, open a support ticket | +| `Failed reaching server: last connection error` | Temporal client diagnostic | Often an expired TLS cert per the troubleshooting guide; run the expiry checks in [Expired or not-yet-valid](#expired-or-not-yet-valid) | + +For TLS errors that only appear after the handshake (gRPC `UNAUTHENTICATED`, `PERMISSION_DENIED`), jump to [authentication.md](authentication.md). For `UNAVAILABLE` without a TLS-layer cause, back off to [connectivity.md](connectivity.md). diff --git a/plugins/temporal/skills/temporal-ops/references/triage/connectivity.md b/plugins/temporal/skills/temporal-ops/references/triage/connectivity.md new file mode 100644 index 0000000..ff191d7 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/triage/connectivity.md @@ -0,0 +1,218 @@ +# Connectivity + +Diagnose failures to reach the Temporal frontend at the network layer — DNS resolution, TCP reachability, wrong endpoint, firewall/proxy drops, PrivateLink / Private Service Connect routing. This file covers layers 1 and 2 of the [diagnostic ladder](diagnostic-ladder.md). + +Out of scope here: +- TLS handshake / x509 / server-name mismatch → [certificates.md](certificates.md) (layer 3) +- gRPC `UNAUTHENTICATED` / `PERMISSION_DENIED` → [authentication.md](authentication.md) (layer 4) +- gRPC `RESOURCE_EXHAUSTED` → [rate-limits.md](rate-limits.md) +- `context deadline exceeded` (ambiguous) → [runtime-errors.md](runtime-errors.md) + +## Table of Contents + +- [Connection refused](#connection-refused) +- [DNS](#dns) +- [Endpoint formats](#endpoint-formats) +- [Firewall and proxy](#firewall-and-proxy) +- [PrivateLink and PSC](#privatelink-and-psc) +- [Quick diagnostic scripts](#quick-diagnostic-scripts) +- [Connection error reference](#connection-error-reference) + +## Connection refused + +**Symptom shape:** OS-level `connection refused` reported inside a gRPC `UNAVAILABLE` status, or by tools like `nc` / `curl` before any gRPC is attempted. The string `connection refused` is emitted by the Go `net` package wrapping `ECONNREFUSED`. + +**What it means:** a TCP SYN reached the host but nothing was listening on that port (RST response). This is layer 2 — the host is up and routable; the process you wanted is not. + +**Things to check (not attributed causes — verify each one):** +- **Dev server not running.** The local dev server is `temporal server start-dev`, which listens on `localhost:7233` by default . +- **Wrong port.** Temporal Cloud's Namespace gRPC endpoint is on port `7233`. The Cloud Ops / control-plane endpoint `saas-api.tmprl.cloud` is on port `443`, not `7233` — pointing a worker or `temporal` CLI data-plane command at `saas-api.tmprl.cloud:7233` will not connect. +- **Self-hosted frontend not accepting connections.** Verify from the process/pod perspective that the frontend is listening on the configured port. + +**First check:** + +```bash +nc -zvw10 7233 +``` + +On macOS/BSD `nc`, `-z` scans without sending data, `-v` is verbose, `-w ` sets the idle/connection timeout in seconds. If this fails with "Connection refused" no higher layer (TLS, gRPC, auth) can succeed. + +## DNS + +**Symptom shape:** errors containing `no such host` (Go's `net` package wrapping `EAI_NONAME` / NXDOMAIN), or resolver-specific messages from the libc resolver (`nodename nor servname provided`, `Temporary failure in name resolution`) bubbled up through Go. + +**What it means:** the resolver could not produce an IP for the hostname. No TCP attempt happens. + +**Things to check:** +- **Namespace endpoint format.** Temporal Cloud Namespace Endpoint is `..tmprl.cloud:7233`. A short namespace name without the `.` suffix will not resolve. +- **Typo in the namespace or account ID.** The hostname must match the Namespace page in the Cloud UI exactly. +- **Broken resolver in the client environment.** Container with missing/empty `/etc/resolv.conf`, CoreDNS not running in the cluster, split-tunnel VPN that isn't forwarding DNS for `*.tmprl.cloud`. +- **PrivateLink / PSC private DNS not set up.** When a namespace is served via a private connection, public DNS still returns a public IP; without a private hosted zone the client may resolve an address it cannot route. See [PrivateLink and PSC](#privatelink-and-psc) below and `/cloud/connectivity` for the required DNS configuration. + +**First check:** + +```bash +nslookup ..tmprl.cloud +dig ..tmprl.cloud +dig +short ..tmprl.cloud # short-form answer +``` + +Interpreting results: + +- For a **public-internet Namespace**, the A record resolves to a public IP (CNAME chain through `-.region.tmprl.cloud`). +- For a **PrivateLink/PSC Namespace with private DNS configured**, the Namespace hostname should resolve to the VPC endpoint address (AWS VPCE DNS name) or the PSC internal IP (GCP) from inside the VPC. +- For an HA (multi-region) Namespace, the Namespace record is a CNAME to `-.region.tmprl.cloud` where `` is the currently active region. + +## Endpoint formats + +Using the wrong endpoint family is one of the most common causes of "cannot connect" reports. + +| Purpose | Endpoint pattern | Port | Source | +|---|---|---|---| +| Cloud Namespace Endpoint (recommended default for workers, SDKs, and `temporal` CLI data-plane — mTLS and API-key-only) | `..tmprl.cloud` | 7233 | | +| Cloud API Regional Endpoint (explicit region pin; dual-auth API-key path; some private-connectivity setups) | `..api.temporal.io` | 7233 | | +| Cloud HA Regional Endpoint (pin to a specific HA replica region) | `-.region.tmprl.cloud` | 7233 | | +| Cloud control-plane (Cloud Ops API, `tcld`, Terraform provider) | `saas-api.tmprl.cloud` | 443 | | +| Self-hosted frontend | `` | `7233` default | deployment-specific | +| Local dev server | `localhost` | `7233` default | `temporal server start-dev` | + +Notes: + +- The **Namespace Endpoint** is the recommended default for Temporal Clients (SDK, workers, `temporal` CLI) for both mTLS and API-key-only Namespaces, because it transparently follows HA failovers without a client change . +- The **API Regional Endpoint** is for explicit region pinning, some private-connectivity setups, and **dual-auth pre-release** (which does not support API key auth to a Namespace Endpoint). When using **mTLS** against an API Regional, HA Regional, or VPCE address, the client must set the TLS server name to the Namespace Endpoint value (see [certificates.md → server name override](certificates.md#server-name-override)). +- Do not conflate the API Regional form (`*.api.temporal.io`) with the HA Regional form (`*.region.tmprl.cloud`) — both are “regional” in docs, but they are different hostnames. +- `saas-api.tmprl.cloud` is **not** a workflow data-plane endpoint — pointing a worker or `temporal workflow …` command at it will not work. +- The `--address` flag (env `TEMPORAL_ADDRESS`) takes `host:port`, not a URL. + +**Private connectivity (PrivateLink / PSC):** When using private endpoints without private DNS, the TLS server name override varies by auth method: + +| Auth method | TLS server name | +|---|---| +| mTLS (single-region) | Namespace Endpoint, e.g. `my-namespace.my-account.tmprl.cloud` | +| API key (single-region) | Regional API endpoint, e.g. `us-east-1.aws.api.temporal.io` (or `us-central1.gcp.api.temporal.io`) | +| Multi-region (mTLS or API key) | Active region endpoint, e.g. `aws-us-east-1.region.tmprl.cloud` | + +For full private connectivity setup (PrivateLink, PSC, connectivity rules), see [cloud-connectivity.md](../ops/cloud-connectivity.md). + +See also [cli-conventions.md → Connection and identity](../ops/cli-conventions.md#connection-and-identity) for the endpoint / env-var reference. + +## Firewall and proxy + +**Symptom shape:** `nc -zvw10` hangs and then reports failure; or the TCP connection completes but no bytes are returned during TLS handshake. Depending on the device, a middlebox may silently drop packets (timeout), send TCP RST (looks like `connection refused` late in the session), or terminate and re-originate TLS (breaks mTLS). + +**Things to check:** +- **Egress allowlist.** Corporate proxy, AWS security group / NACL, or Kubernetes NetworkPolicy permitting TCP egress to the Temporal endpoint on the correct port. For AWS PrivateLink, the VPC-endpoint security group must accept TCP ingress to port 7233. +- **TLS-inspecting proxy in the path.** A device that terminates and re-originates TLS will break mTLS client-certificate authentication because the proxy presents its own certificate downstream and can't forward the client's private key. Detection heuristic: `nc -zvw10` succeeds (TCP works) but an `openssl s_client` against the same host returns an unexpected certificate chain or is closed without a TLS alert. + +**First check:** +- Run `nc -zvw10 7233` from the failing environment. A connect with no data is enough to confirm TCP reachability; that isolates the problem above layer 2. +- For TLS-inspection hypotheses, compare the server certificate returned by `openssl s_client -connect :7233 -servername ` (see [certificates.md → OpenSSL recipes](certificates.md#openssl-recipes)) against the expected Temporal Cloud CA/issuer. + +**Fix direction:** +- Allow egress on TCP/7233 to the Temporal endpoint (or the VPCE / PSC IP when using private connectivity). +- Bypass TLS inspection for the Temporal endpoint. Re-signing TLS is incompatible with mTLS client-cert auth. + +## PrivateLink and PSC + +Temporal Cloud supports private connectivity via AWS PrivateLink and GCP Private Service Connect in addition to the default public internet endpoints. This section is layer 1–2 diagnosis only; the full setup reference is in `docs/cloud/connectivity/`. + +Classification of common layer-1/2 failures after PrivateLink/PSC is supposed to be in use: + +- **DNS resolves to a public IP, VPC cannot route it.** Private DNS (Route 53 PHZ in AWS, Cloud DNS private zone in GCP) is missing or not attached to the workers' VPC. Without it, the client gets the public A record, and the VPC has no egress to the internet. See the private-DNS setup in `/cloud/connectivity/aws-connectivity` or `/cloud/connectivity/gcp-connectivity`. +- **DNS resolves to the private endpoint, port unreachable.** Likely the VPC-endpoint security group is not permitting TCP/7233 from the client subnet. +- **Connection succeeds but TLS fails with a server-name mismatch.** This is layer 3, not layer 2. Clients connecting by VPC-endpoint DNS name must set the TLS server name (SNI override) to the Namespace Endpoint (`..tmprl.cloud`). Details in [certificates.md → server name override](certificates.md#server-name-override). +- **Works before failover, breaks after failover (HA Namespaces).** For multi-region Namespaces, the Namespace record CNAMEs to `-.region.tmprl.cloud`, and on failover the CNAME is updated to point to the new active region. If private DNS only overrides the old region, workers lose the path on failover. The `region.tmprl.cloud` private zone must cover every region the Namespace can fail over to. Note: automatic DNS-based failover is not supported for GCP PSC — manual worker reconfiguration is required. + +**Quick reachability check from inside the client's VPC:** + +```bash +# AWS VPCE DNS name, or GCP PSC IP +nc -zvw10 vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com 7233 +``` + +. + +## Quick diagnostic scripts + +Run from the failing environment (the pod, container, or host where the problem reproduces). These scripts chain the layer-1/2 checks with a final `temporal` call to confirm the whole stack end-to-end. + +### mTLS variant + +```bash +#!/bin/bash +NS="your-namespace.account-id" +CERT="client.pem" +KEY="client.key" +HOST="$NS.tmprl.cloud" +ADDRESS="$HOST:7233" + +echo "=== DNS resolution ===" +nslookup "$HOST" + +echo "=== TCP reachability ===" +nc -zvw10 "$HOST" 7233 + +echo "=== TLS handshake (see certificates.md for deeper TLS diagnosis) ===" +openssl s_client -connect "$ADDRESS" \ + -servername "$HOST" \ + -cert "$CERT" -key "$KEY" &1 | head -20 + +echo "=== Temporal CLI end-to-end ===" +temporal workflow list --limit 1 \ + --address "$ADDRESS" \ + --namespace "$NS" \ + --tls-cert-path "$CERT" \ + --tls-key-path "$KEY" +# --address: docs/cli/cmd-options.mdx:137 +# --namespace: docs/cli/cmd-options.mdx:420 +# --tls-cert-path: docs/cli/cmd-options.mdx:661 +# --tls-key-path: docs/cli/cmd-options.mdx:673 +# --limit: docs/cli/cmd-options.mdx:379 +``` + +### API-key variant + +For API-key-only Namespaces, default to the **Namespace Endpoint**. Use the API Regional form for region pin / dual-auth / some private-connectivity setups. + +```bash +#!/bin/bash +NS="your-namespace.account-id" +ADDRESS="..tmprl.cloud:7233" # default for API-key-only +# ADDRESS="..api.temporal.io:7233" # region pin / dual-auth +HOST="${ADDRESS%:*}" +export TEMPORAL_API_KEY="..." + +echo "=== DNS resolution ===" +nslookup "$HOST" + +echo "=== TCP reachability ===" +nc -zvw10 "$HOST" 7233 + +echo "=== TLS handshake ===" +openssl s_client -connect "$ADDRESS" \ + -servername "$HOST" &1 | head -20 + +echo "=== Temporal CLI end-to-end ===" +temporal workflow list --limit 1 \ + --address "$ADDRESS" \ + --namespace "$NS" \ + --api-key "$TEMPORAL_API_KEY" +``` + +A lighter end-to-end probe, with fewer moving parts, is `temporal operator cluster health --address
` — if it returns `SERVING`, the client reached a Temporal frontend through all of DNS, TCP, TLS, and gRPC. + +## Connection error reference + +Each row's error text comes from outside Temporal — either the Go `net` package (syscall wrapping) or the gRPC runtime. Tagged accordingly. + +| Error text (shape) | Origin | Layer | One thing to check | +|---|---|---|---| +| `no such host` | Go net / libc resolver | 1 (DNS) | Endpoint hostname typo; missing `.` suffix; resolver / VPN / private DNS configuration | +| `connection refused` | Go net (ECONNREFUSED) | 2 (TCP) | Wrong port, dev server not running, data-plane command pointed at `saas-api.tmprl.cloud` | +| `i/o timeout` | Go net | 2 (TCP) | Silent drop by firewall / NACL / NetworkPolicy | +| `network is unreachable` | Go net (ENETUNREACH) | 2 (routing) | No route from client subnet to target | +| `UNAVAILABLE` with a connection-layer cause | gRPC status | 2–3 | Peel back to the wrapped `net` error; `UNAVAILABLE` on its own can also come from TLS and gRPC layers | +| `context deadline exceeded` | Go context | ambiguous | Not diagnostic on its own; see [runtime-errors.md](runtime-errors.md) | +| `Failed reaching server: last connection error` | Temporal Cloud client diagnostics | usually 3 (TLS) | Most often an expired TLS cert per the troubleshooting guide — jump to [certificates.md](certificates.md) | + +For TLS-layer errors that surface through `UNAVAILABLE` or masquerade as connectivity, jump to [certificates.md](certificates.md). For `UNAUTHENTICATED` / `PERMISSION_DENIED`, [authentication.md](authentication.md). For `RESOURCE_EXHAUSTED`, [rate-limits.md](rate-limits.md). diff --git a/plugins/temporal/skills/temporal-ops/references/triage/diagnostic-ladder.md b/plugins/temporal/skills/temporal-ops/references/triage/diagnostic-ladder.md new file mode 100644 index 0000000..2a71cb6 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/triage/diagnostic-ladder.md @@ -0,0 +1,210 @@ +# Diagnostic Ladder + +The bottom-up seven-layer model every other file in this skill hangs off. A failure at a lower layer breaks every layer above it, so a signature you think you recognize at layer 5 may really be layer 3 with a misleading wrapper. + +This file is a table of contents, not a recipe book: each layer gives one check and one minimal healthy signal, then links to the sibling file that owns the failure diagnosis. If you want the full OpenSSL recipe, it's in [certificates.md](certificates.md); if you want the full worker-polling discrimination, it's in [worker-health.md](worker-health.md). Come here to choose *which* layer to investigate. + +## Table of Contents + +- [How to use the ladder](#how-to-use-the-ladder) +- [Layer 1: DNS / network path](#layer-1-dns--network-path) +- [Layer 2: TCP / port reachability](#layer-2-tcp--port-reachability) +- [Layer 3: TLS handshake](#layer-3-tls-handshake) +- [Layer 4: Authentication and authorization](#layer-4-authentication-and-authorization) +- [Layer 5: gRPC / Temporal frontend health](#layer-5-grpc--temporal-frontend-health) +- [Layer 6: Namespace, task queues, and workers](#layer-6-namespace-task-queues-and-workers) +- [Layer 7: Workflow code](#layer-7-workflow-code) +- [Quick per-layer commands](#quick-per-layer-commands) + +## How to use the ladder + +1. **Run diagnostics from the failing environment.** A pod in a production VPC and a laptop on a home network do not share DNS, egress paths, or CA bundles. Always reproduce from where the problem occurs. (Skill convention.) +2. **Start at the lowest layer that could plausibly be broken.** For a fresh cert-rotation incident, start at layer 3. For a "workflow stuck" report where the client still works, start at layer 6. +3. **Move up only after the current layer is proven healthy.** "Healthy" means the check in that layer returned the signal described — not that the command didn't crash. +4. **If a layer fails, layers above it are unknown.** An `UNAUTHENTICATED` reported by a client whose TLS handshake is actually failing will not be fixed by rotating API keys — see the wrapped-cause trap in [authentication.md → UNAUTHENTICATED vs PERMISSION_DENIED](authentication.md#unauthenticated-vs-permission_denied). +5. **Do not skip layers.** Skill convention, not a Temporal contract — but every bug-report-chased-at-the-wrong-layer in this skill's scope would have been caught by it. + +## Layer 1: DNS / network path + +**Question:** Can this host resolve the Temporal endpoint to an address it can route? + +**Minimal check:** + +```bash +nslookup ..tmprl.cloud +dig +short ..tmprl.cloud +``` + +**Healthy signal:** an A record (self-hosted) or a CNAME chain through `.region.tmprl.cloud` ending in A records (Cloud). See [connectivity.md → DNS](connectivity.md#dns) for the exact shapes per namespace type. + +**Failure signatures** (each routes to [connectivity.md](connectivity.md) for remediation): + +- `no such host` wrapped in a gRPC `UNAVAILABLE` — resolver returned NXDOMAIN or the resolver is unreachable. See [connectivity.md → DNS](connectivity.md#dns). +- Namespace hostname format wrong (missing `.` suffix, typo) — the endpoint table in [connectivity.md → Endpoint formats](connectivity.md#endpoint-formats) shows the valid patterns. +- Returns a private IP unexpectedly, or a public IP when PrivateLink is supposed to be in use — split-horizon / missing private hosted zone. See [connectivity.md → PrivateLink and PSC](connectivity.md#privatelink-and-psc). + +**What failure here means above:** every higher layer fails. Do not attempt TLS or auth diagnosis until DNS works. + +## Layer 2: TCP / port reachability + +**Question:** Can this host open a TCP connection on the Temporal frontend port to the resolved IP? + +**Minimal check:** + +```bash +nc -zvw10 7233 +``` + +The Cloud Namespace gRPC endpoint is on TCP/7233 (see [connectivity.md → Connection refused](connectivity.md#connection-refused), which cites the Namespace Endpoint port). On BSD `nc`, `-z` scans without sending data, `-v` is verbose, `-w` sets the idle timeout. + +**Healthy signal:** `succeeded!` (or the BSD `nc` equivalent line). Anything beyond a successful connect is out of scope for layer 2. + +**Failure signatures:** + +- `Connection refused` — TCP reached the host but nothing is listening. See [connectivity.md → Connection refused](connectivity.md#connection-refused) for the top causes (dev server not running; wrong port — e.g. pointing at `saas-api.tmprl.cloud:7233` when that endpoint is port 443; self-hosted frontend not accepting connections). +- Hangs for the full `-w` timeout — firewall silently dropping packets. See [connectivity.md → Firewall and proxy](connectivity.md#firewall-and-proxy). +- TCP succeeds but no bytes returned at TLS time — often a TLS-inspecting middlebox. Same section: [connectivity.md → Firewall and proxy](connectivity.md#firewall-and-proxy). +- PrivateLink / PSC: DNS resolves to the private endpoint but port unreachable — VPC-endpoint security group not permitting TCP/7233. See [connectivity.md → PrivateLink and PSC](connectivity.md#privatelink-and-psc). + +**What failure here means above:** TLS and everything above cannot complete. + +## Layer 3: TLS handshake + +**Question:** Does the TLS handshake complete and (for mTLS) does the server accept the client cert? + +**Minimal check** (API-key or server TLS only): + +```bash +openssl s_client -connect :7233 -servername , not ` — [certificates.md → Hostname mismatch](certificates.md#hostname-mismatch). Common on PrivateLink / Regional-Endpoint clients that didn't override SNI — fix per [certificates.md → Server name override](certificates.md#server-name-override). +- TCP opens, closes without a TLS alert — typically a middlebox. Back off to layer 2, [connectivity.md → Firewall and proxy](connectivity.md#firewall-and-proxy). + +See also [certificates.md → TLS / cert error reference](certificates.md#tls--cert-error-reference) for the full error-string table. + +**What failure here means above:** depending on the SDK / client, the gRPC client will surface this as `UNAVAILABLE` with a wrapped `tls:` / `x509:` cause, or — confusingly — as `UNAUTHENTICATED`. Do not debug layer 4 until TLS is clean. The wrapped-cause trap is documented in [authentication.md → UNAUTHENTICATED vs PERMISSION_DENIED](authentication.md#unauthenticated-vs-permission_denied). + +## Layer 4: Authentication and authorization + +**Question:** Does Temporal accept the presented credentials, and does the resulting identity permit this operation on this namespace? + +**Minimal check** (API-key variant; mTLS variant is in the sibling): + +```bash +temporal workflow list --limit 1 \ + --address ..tmprl.cloud:7233 \ + --namespace . \ + --api-key "$TEMPORAL_API_KEY" +# Command form: authentication.md → Discriminating with a CLI smoke test +# Use API Regional Endpoint when dual-auth / region pin requires it +``` + +The full form, including flag citations and the mTLS variant, is in [authentication.md → Discriminating with a CLI smoke test](authentication.md#discriminating-with-a-cli-smoke-test). API-key-only Namespaces default to the Namespace Endpoint; API Regional is for pin / dual-auth / some private-connectivity cases — see [authentication.md → Address form for API-key connections](authentication.md#address-form-for-api-key-connections). + +**Healthy signal:** the command returns a list (possibly empty) without error. + +**Failure signatures:** + +- `UNAUTHENTICATED` — credentials rejected (missing, typo, disabled, deleted, expired key; untrusted mTLS cert; wrong endpoint family). [authentication.md → API-key authentication](authentication.md#api-key-authentication) and [authentication.md → mTLS authentication after TLS completes](authentication.md#mtls-authentication-after-tls-completes). +- `PERMISSION_DENIED` — credentials valid but the identity lacks account-role / namespace-permission / cert-filter-derived identity for the action. [authentication.md → Cloud role and permission model](authentication.md#cloud-role-and-permission-model). +- `INVALID_ARGUMENT` with a "namespace not found" suffix — namespace string does not exist in this account, or the Namespace Endpoint form is wrong. See [runtime-errors.md → `INVALID_ARGUMENT`](runtime-errors.md#invalid_argument) and verify against the endpoint table in [connectivity.md → Endpoint formats](connectivity.md#endpoint-formats). + +**What failure here means above:** every gRPC call from this principal fails the same way. No point inspecting task queues or workflows. + +## Layer 5: gRPC / Temporal frontend health + +**Question:** Is the Temporal frontend reachable and reporting itself healthy over gRPC? + +**Minimal check:** + +```bash +temporal operator cluster health +``` + +Supply whatever `--address`, `--namespace`, and auth flags you established at layer 4. The command calls `grpc.health.v1.Health/Check`. + +**Self-hosted:** use `temporal operator cluster health` directly. **Cloud:** use `temporal workflow list --limit 1` as the frontend-reachability smoke test instead — `cluster health` is scoped to self-hosted in the docs. + +**Healthy signal:** `SERVING` (self-hosted) or a successful list response (Cloud). The Temporal troubleshooting guide uses `cluster health` as the first "is the frontend up?" probe on self-hosted. + +**Failure signatures:** + +- `NOT_SERVING` or an unhealthy status — the frontend is up but reports itself unhealthy. On self-hosted, the troubleshooting guide points at `grpc-health-probe` to test Frontend, Matching, and History individually; on Cloud, open a ticket. See [runtime-errors.md → Deadline exceeded](runtime-errors.md#deadline-exceeded) for the self-hosted vs. Cloud routing. +- Long timeouts or `DEADLINE_EXCEEDED` with no specific cause — overload or upstream saturation. A `resource_exhausted` condition can surface as a deadline — see [runtime-errors.md → Deadline exceeded](runtime-errors.md#deadline-exceeded) and [rate-limits.md → Identifying which limit was hit](rate-limits.md#identifying-which-limit-was-hit). +- `RESOURCE_EXHAUSTED` at the frontend — rate limit or capacity. [rate-limits.md → What RESOURCE_EXHAUSTED means and what it does not](rate-limits.md#what-resource_exhausted-means-and-what-it-does-not). + +**What failure here means above:** task-queue and workflow operations will succeed intermittently or not at all. Check this layer before blaming workers or workflow code. + +## Layer 6: Namespace, task queues, and workers + +**Question:** Are workers polling the expected task queue in the expected namespace on a recent timescale? + +**Minimal check:** + +```bash +temporal task-queue describe --task-queue +# Command form and statistics: worker-health.md → Inspecting a Task Queue +``` + +The full invocation, including `--task-queue-type` filtering and the statistics the call returns, is in [worker-health.md → Inspecting a Task Queue with `temporal task-queue describe`](worker-health.md#inspecting-a-task-queue-with-temporal-task-queue-describe). + +**Healthy signal:** a non-empty pollers list whose `LastAccessTime` values are recent. Per the CLI docs, "A `LastAccessTime` over one minute may indicate the Worker is at capacity or has shut down. Temporal Workers are removed if 5 minutes have passed since the last poll request" (quoted and cited in [worker-health.md → Reading the pollers field](worker-health.md#reading-the-pollers-field)). + +**Failure signatures:** + +- Empty poller list — no worker reached the frontend for this queue+type within the 5-minute window. [worker-health.md → What "no pollers" looks like](worker-health.md#what-no-pollers-looks-like). +- Pollers present but stale `LastAccessTime` (> ~1 min, no tasks moving) — worker at capacity or shut down. [worker-health.md → Reading the pollers field](worker-health.md#reading-the-pollers-field). +- Pollers present but the Build IDs / versions don't match the worker fleet you expect — versioning routing issue. [worker-health.md → Reachability and versioning](worker-health.md#reachability-and-versioning). +- Backlog growing despite fresh pollers — slot exhaustion or schedule-to-start saturation. [worker-health.md → Schedule-to-start latency](worker-health.md#schedule-to-start-latency) and [worker-health.md → Worker task slots](worker-health.md#worker-task-slots). + +**What failure here means above:** workflows enqueue tasks that never get picked up, so a workflow will appear stuck at the first `ActivityTaskScheduled` or `WorkflowTaskScheduled` event. Do not start diagnosing layer 7 until pollers are healthy. + +## Layer 7: Workflow code + +**Question:** When a worker picks up a Workflow Task, does the workflow code execute successfully against the recorded Event History? + +**Minimal check:** + +```bash +temporal workflow describe --workflow-id +# Command form and output schema: workflow-stuck.md → The primary inspection command +``` + +For the full inspection flow — describe output shape, status interpretation, pending-sections routing, and the companion `temporal workflow show` for Event History — see [workflow-stuck.md → The primary inspection command: `temporal workflow describe`](workflow-stuck.md#the-primary-inspection-command-temporal-workflow-describe) and [workflow-stuck.md → Inspecting the Event History: `temporal workflow show`](workflow-stuck.md#inspecting-the-event-history-temporal-workflow-show). + +**Healthy signal:** `workflowExecutionInfo.status` is `Running` and the last meaningful event is one that legitimately blocks progress (an intentional `TimerStarted`, an awaited signal, an in-flight activity with a running retry state), or the workflow has progressed since the previous check. The full status table (Running / Completed / Failed / Canceled / Terminated / ContinuedAsNew / TimedOut) is in [workflow-stuck.md → Workflow Execution Status values](workflow-stuck.md#workflow-execution-status-values). + +**Failure signatures:** + +- `WorkflowTaskFailed` events recurring with `cause` = Nondeterminism — [non-determinism.md → The WFT-failure signature of non-determinism](non-determinism.md#the-wft-failure-signature-of-non-determinism). Reproduce locally via [replay.md](replay.md). +- `WorkflowTaskFailed` loops with a non-Nondeterminism cause (e.g. Workflow Worker Unhandled Failure) — [workflow-stuck.md → Pending Workflow Task and WorkflowTaskFailed loops](workflow-stuck.md#pending-workflow-task-and-workflowtaskfailed-loops). +- `ActivityTaskScheduled` with no matching retry / terminal event after describe — loop back to layer 6; the task was never picked up. [workflow-stuck.md → Pending activities](workflow-stuck.md#pending-activities). +- Pending Activity with climbing attempts and `last_failure` populated — the Activity is running and failing; fix the Activity or its retry policy. Same section. +- Workflow `Running` with `historyLength` flat and no pending sections — a timer-based wait, covered in [workflow-stuck.md → Timer-based waits](workflow-stuck.md#timer-based-waits). +- Ambiguous `DEADLINE_EXCEEDED` or `Workflow is busy` lock contention on signals/updates/queries — [runtime-errors.md → Deadline exceeded](runtime-errors.md#deadline-exceeded) and [runtime-errors.md → Workflow lock contention (BusyWorkflow)](runtime-errors.md#workflow-lock-contention-busyworkflow). + +## Quick per-layer commands + +Each command below is the minimal check for its layer. Full invocations with all flags and citations live in the sibling file linked on the right. + +| Layer | Command | Healthy signal | Owner | +|---|---|---|---| +| 1. DNS | `nslookup ` | A record or CNAME chain returned | [connectivity.md → DNS](connectivity.md#dns) | +| 2. TCP | `nc -zvw10 7233` | `succeeded!` | [connectivity.md → Connection refused](connectivity.md#connection-refused) | +| 3. TLS | `openssl s_client -connect :7233 -servername ` | `Verify return code: 0 (ok)` | [certificates.md → Handshake failure](certificates.md#handshake-failure) | +| 4. Auth | `temporal workflow list --limit 1 …` | list returns (possibly empty) | [authentication.md → Discriminating with a CLI smoke test](authentication.md#discriminating-with-a-cli-smoke-test) | +| 5. Frontend | Self-hosted: `temporal operator cluster health`; Cloud: `temporal workflow list --limit 1` | `SERVING` (self-hosted) or successful response (Cloud) | [runtime-errors.md → Deadline exceeded](runtime-errors.md#deadline-exceeded) | +| 6. Workers | `temporal task-queue describe --task-queue ` | pollers listed with recent `LastAccessTime` | [worker-health.md → Inspecting a Task Queue with `temporal task-queue describe`](worker-health.md#inspecting-a-task-queue-with-temporal-task-queue-describe) | +| 7. Workflow | `temporal workflow describe --workflow-id ` | status `Running` with a legitimate pending reason | [workflow-stuck.md → The primary inspection command: `temporal workflow describe`](workflow-stuck.md#the-primary-inspection-command-temporal-workflow-describe) | + +If every layer is healthy and the user still reports a problem, the diagnosis narrows to workflow code, SDK configuration, or workload pressure — handed off to [workflow-stuck.md](workflow-stuck.md), [non-determinism.md](non-determinism.md), [runtime-errors.md](runtime-errors.md), or [rate-limits.md](rate-limits.md) depending on the shape of the symptom. HA-failover-specific symptoms (CNAME didn't update, PrivateLink breaks after failover) are in [ha-failover.md](ha-failover.md); the worker-placement triage pointer is at [ha-failover.md → Worker placement — triage-layer pointer](ha-failover.md#worker-placement--triage-layer-pointer). diff --git a/plugins/temporal/skills/temporal-ops/references/triage/ha-failover.md b/plugins/temporal/skills/temporal-ops/references/triage/ha-failover.md new file mode 100644 index 0000000..9bbcc55 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/triage/ha-failover.md @@ -0,0 +1,191 @@ +# HA Failover + +Diagnose Temporal Cloud Multi-region / Multi-cloud Namespace failover symptoms: clients not following the CNAME swap, PrivateLink breaking after the active region changes, and failovers that don't appear to have taken effect. Cloud-only — self-hosted HA is out of scope. + +Citations point at stable doc anchors (slug + heading), e.g. `/cloud/high-availability/ha-connectivity#namespace-endpoint-recommended`. They are public URL fragments, not line numbers — durable across doc edits. + +Out of scope here: +- DNS / TCP reachability in general (before HA is even a hypothesis) → [connectivity.md](connectivity.md) (layers 1-2), which also carries the PrivateLink / CNAME facts in [§ PrivateLink and PSC](connectivity.md#privatelink-and-psc). +- Worker placement architecture decisions (cost, latency, pattern selection) → `skill-temporal-deploy`. This file gives only a triage-layer pointer. +- Enabling HA / choosing replica regions / pricing at setup time → `/cloud/high-availability/enable`. +- `tcld` flag semantics in depth → `skill-temporal-cli` → `references/core/cloud-control-plane.md`. + +## Table of Contents + +- [Start here: establish ground truth](#start-here-establish-ground-truth) +- [How Cloud HA routing works (minimum needed for triage)](#how-cloud-ha-routing-works-minimum-needed-for-triage) +- [Symptom: clients did not follow the failover](#symptom-clients-did-not-follow-the-failover) +- [Symptom: Serverless Workers kept running in the old region after failover](#symptom-serverless-workers-kept-running-in-the-old-region-after-failover) +- [Symptom: PrivateLink / PSC stopped working after failover](#symptom-privatelink--psc-stopped-working-after-failover) +- [Symptom: failover was requested but never happened](#symptom-failover-was-requested-but-never-happened) +- [Symptom: Workflows are rejected during handover](#symptom-workflows-are-rejected-during-handover) +- [Worker placement — triage-layer pointer](#worker-placement--triage-layer-pointer) +- [RPO / RTO reference](#rpo--rto-reference) + +## Start here: establish ground truth + +Every failover symptom funnels through one question: **what does the control plane think is active, and what does the client's DNS resolve to?** Get both before forming any hypothesis — most "failover is broken" reports are a disagreement between these two views. + +**Control-plane view:** + +```bash +tcld namespace get --namespace . +``` + +Returns the Namespace record including its regions. For the exact field reporting the active region or replica state, read the live response — the active region is reflected in the target of the Namespace Endpoint's CNAME record. + +**DNS view, from the failing environment:** + +```bash +dig +short ..tmprl.cloud +nslookup ..tmprl.cloud +``` + +The CNAME target encodes the active region, e.g. `aws-us-east-1.region.tmprl.cloud`. + +**Corroborating signals:** failovers are written to the audit log as `"operation": "FailoverNamespace"`, shown on the Namespace detail page in the Web UI, and emailed to account admins. + +| Observation | Reading | Go to | +|---|---|---| +| Both views agree, CNAME points at the expected region | Healthy. The failover itself is fine. | If traffic still fails, the problem is below HA → [connectivity.md](connectivity.md) | +| Control plane moved, client DNS still shows old region | Resolver on the client path cached the old CNAME | [clients did not follow](#symptom-clients-did-not-follow-the-failover) | +| Control plane still shows old region, no `FailoverNamespace` audit entry | The failover never executed | [failover was requested but never happened](#symptom-failover-was-requested-but-never-happened) | +| DNS resolves to a public IP a private VPC can't reach | Private DNS doesn't cover the new region | [PrivateLink / PSC stopped working](#symptom-privatelink--psc-stopped-working-after-failover) | + +## How Cloud HA routing works (minimum needed for triage) + +An HA Namespace keeps a primary and a replica in separate isolation domains. On failover, Temporal changes which is active and updates DNS so the Namespace Endpoint routes to the new active region. Two DNS names drive every symptom below: + +| Name | Form | Behavior on failover | +|---|---|---| +| **Namespace Endpoint** (what clients use) | `..tmprl.cloud:7233` | Hostname never changes; it is a CNAME whose target Temporal repoints. | +| **Regional record** (CNAME target) | `-.region.tmprl.cloud` (e.g. `aws-us-west-2.region.tmprl.cloud`) | Temporal updates the CNAME from the old region to the new one. | + +Two timing facts the triage logic depends on: + +- Namespace DNS records carry a **15-second TTL**; clients converge to the new region within ~30 seconds (about 2× the TTL), assuming their resolver honors the TTL. +- Temporal Cloud enforces a **5-minute maximum connection lifetime**, forcing long-lived connections to re-resolve DNS. + +## Symptom: clients did not follow the failover + +**Symptom:** after a failover, workflow starts or worker polls still hit the old region, connections time out, or the control-plane and client-side DNS views disagree about the active region. + +**Discriminate** (re-resolve from the failing environment and compare against `tcld namespace get` first): + +1. **DNS resolver caching on the client path.** A cache should re-resolve within the 15-second TTL and converge within ~30 seconds. A resolver that holds the old CNAME longer (NodeLocal DNSCache, a local `dnsmasq`, a VM stub resolver) shows a stale target on repeated `dig +short`. No doc-authoritative list of which resolvers honor the TTL exists — verify empirically. +2. **Long-lived connection not re-resolving.** The 5-minute connection cap exists to force re-resolution. A worker wedged on a connection that outlives the window won't move; restarting the worker pod forces fresh resolution. +3. **Application-level address caching.** A caller that resolved the hostname to an IP at startup and reused it won't follow a CNAME swap. Pass the hostname to the client config, never a pre-resolved IP. +4. **GCP Private Service Connect.** PSC has no DNS-based automatic failover — workers must be manually repointed to the new region's PSC endpoint. See [PrivateLink / PSC stopped working](#symptom-privatelink--psc-stopped-working-after-failover). +5. **Private DNS override covers only one region.** Same section. +6. **Serverless Workers (AWS Lambda).** The Worker Controller Instance keeps invoking Workers in the compute provider's originally configured region because compute-provider configuration is region-scoped and the WCI has no failover-detection mechanism. This is a distinct failure mode from long-lived Worker DNS caching — see [Serverless Workers kept running in the old region](#symptom-serverless-workers-kept-running-in-the-old-region-after-failover). + +**Fix:** clear/await the offending cache, restart wedged workers, or repoint PSC workers per the discriminator that matched. + +**Verify:** + +```bash +dig +short ..tmprl.cloud +# CNAME target should now match the active region from `tcld namespace get`. +``` + +Then re-run the operation that was failing. + +## Symptom: Serverless Workers kept running in the old region after failover + +**Symptom:** the Namespace failed over successfully, but Serverless Workers (AWS Lambda, Public Preview) are still being invoked in the old region. Silent while that region is healthy; degraded throughput, latency, or a stall once it is not. + +Nothing in your infrastructure polls, so there is no DNS to re-resolve. The Worker Controller Instance invokes the compute provider configured on a Worker Deployment Version, that configuration is scoped to a single region (for example, a Lambda ARN), and the WCI has no mechanism to detect a failover or redirect invocations into the new active region. Applies to Multi-region and Multi-cloud Replication alike. See `/cloud/high-availability#serverless-workers` and the High Availability row of `/serverless-workers#constraints`. + +**Discriminate:** confirm the new active region (`tcld namespace get --namespace .` plus the `FailoverNamespace` audit entry) and compare it against the Lambda ARN on the Version serving the affected Task Queue. Long-lived Workers on other Task Queues recover on their own, so a mixed fleet recovers partially — which reads like a regional outage rather than a configuration constraint. + +**Fix:** `tcld namespace failover` moves the Namespace only, and tcld has no compute-provider surface. Remediation is to repoint the existing Worker Deployment Version's compute provider at a function in the new active region — an in-place update, not a new Version. Hand it to `skill-temporal-serverless`; it changes where production Workers are invoked, so propose it before running. + +**Prevent:** publish the function in every region the Namespace can fail over to, so the repoint is a single command instead of a provisioning exercise under time pressure. + +**Verify:** the affected Task Queue drains, and invocations land on the new region's function in the provider's logs. + +## Symptom: PrivateLink / PSC stopped working after failover + +**Symptom:** pre-failover the Namespace was reachable via a private VPC Endpoint; post-failover DNS resolves the Namespace Endpoint to a public IP the VPC can't reach, or to nothing. + +**Discriminate:** + +1. **Private DNS covered only the old region.** The private hosted zone overrode only the old active region's `-.region.tmprl.cloud` record. After the CNAME flips, the new region has no private entry, so the client falls back to public DNS or dead-ends in a no-egress VPC. This is the common cause. +2. **Workers can't reach the new region.** Even with DNS fixed, a single-region worker fleet needs a network path to the now-active region. +3. **Direct-VPCE targeting without a worker in every region.** Direct VPCE works with HA, but it doesn't follow the CNAME — each worker reaches Temporal only through its own region's VPC Endpoint (same `ServerName` override, a different VPCE address per region). If you deployed a worker + VPCE in just one region, cross-region forwarding keeps it working while that region is passive, but a full outage of that region leaves no path. +4. **GCP PSC.** No automatic DNS failover; workers must be manually repointed. + +**Fix:** the `region.tmprl.cloud` private hosted zone must map **every** region the Namespace can fail over to, each `-.region.tmprl.cloud` → that region's VPC Endpoint. Give workers a path to the new region — run workers in both regions, or link the VPCs (Transit Gateway / VPC Peering). For direct-VPCE, deploy a worker plus VPC Endpoint in each region (same `ServerName`, different VPCE per region). For GCP PSC, repoint workers to the active region's PSC endpoint manually on failover. + +**Verify:** from inside the client VPC, `dig +short` the Namespace Endpoint and confirm it resolves to the new region's VPC Endpoint, then `nc -zvw10 7233`. + +## Symptom: failover was requested but never happened + +**Symptom:** a failover was initiated (Web UI, `tcld`, or Cloud Ops API) but `tcld namespace get` still shows the old active region, the audit log has no `FailoverNamespace` entry, and traffic hasn't shifted. + +**Discriminate:** + +1. **Failover is in progress.** Check whether the user received an async operation ID from the failover request. If they did, the failover was accepted and is guaranteed to complete (see Verify below) - wait for it to finish, then verify with the [ground truth](#start-here-establish-ground-truth) steps. +2. **Replica is in a failed state.** If the replica shows a failed state, the failover was attempted but did not succeed. Temporal on-call has been paged and will reach out. Inform the user that Temporal is aware and actively working on remediation. No user action is needed. +3. **Manual `tcld` invocation was malformed.** The command is: + ```bash + tcld namespace failover \ + --namespace . \ + --region + ``` + `--namespace` and `--region` are required. With API-key auth, `--api-key` must come immediately after `tcld`, before `namespace failover`. +4. **Target region isn't an `Activated` replica.** The target must be a region holding a replica that is ready to be failed over to (state `Activated`). An unhealthy replica makes the Web UI disable "Trigger a failover"; common causes are data-sync issues, replication lag, network issues, and failed health checks. +5. **The Namespace can't support this failover (constraint).** See the constraints table below — a missing replica, region eligibility, or a replication-type conflict can make the failover impossible to request. +6. **Permissions.** `FailoverNamespaceRegion` requires Namespace Admin. Account Owner and Global Admin hold Namespace Admin on all Namespaces. +7. **Automatic Failover didn't fire.** Automatic Failover is driven by Temporal Cloud health checks on error rates, latencies, and infrastructure indicators. If it's disabled (`tcld namespace update-high-availability --disable-auto-failover=true`), Temporal won't initiate failovers — the user must trigger manually, and the published Temporal Cloud RTO does not apply. +8. **Expecting an automatic failback that won't come.** After a user-triggered failover Temporal does *not* fail back automatically; the user must trigger it. Automatic failback only follows an Automatic Failover. + +**Constraints that can block a failover from being possible:** + +| Constraint | Effect | Source | +|---|---|---| +| Namespace has no replica | Must be upgraded with HA (`tcld namespace add-region` or Web UI) before any failover | | +| Replica must be on the same continent as the primary; `sa-east-1` is the only region on its continent, so it has no eligible Multi-region replica | No replica region to fail over to | | +| Only one replica may be added per Namespace, so it is either Multi-region (same cloud, different region) or Multi-cloud (different cloud provider) - not both | Limits which replica topologies exist to fail over to | | +| Replica must be in `Activated` state to fail over to | A replica that is still activating, is in a failed state, or is in any other non-`Activated` state cannot be a failover target | | +| 7-day wait after `tcld namespace delete-region` before re-enabling HA in that region | A just-removed region can't be re-added as a failover target yet | | + +**Fix:** correct the matched discriminator (command form, target state, permissions, or auto-failover setting), or resolve the blocking constraint. + +**Verify — and an important guarantee:** once a failover request returns an async operation ID, the failover **is guaranteed to complete** — there is no case where an accepted failover silently fails to execute. Temporal retries the failover Workflow internally and pages on-call to force it through on any internal error. So if you received an operation ID but traffic hasn't shifted, the failover succeeded — look to DNS propagation (the [clients did not follow](#symptom-clients-did-not-follow-the-failover) path), not the failover request. + +## Symptom: Workflows are rejected during handover + +**Symptom:** during the failover, clients see a brief window of retryable "Service unavailable" errors and start/signal requests are rejected. + +**What it is (expected behavior):** the failover is a single hybrid strategy. Temporal first attempts a *graceful failover* — pause traffic, drain in-flight replication, switch with no data conflicts. If that doesn't complete within 10 seconds, it falls back to a *forced failover* that immediately activates the replica; unreplicated events undergo conflict resolution when the original region returns. It **proceeds to the forced failover — it does not revert.** Operations pause briefly and SDKs receive a retryable "Service unavailable" error they retry automatically. + +**Discriminate** (only if the window is unusually long or the error isn't retried): + +1. **Raw gRPC client, not an SDK.** SDK retries cover this window by design; a raw client must retry `UNAVAILABLE` itself. +2. **Large replication lag.** A forced failover with significant lag is more likely to roll back Workflow progress; always check lag before failing over. Lag is exposed as `temporal_cloud_v1_replication_lag_p50` / `_p95` / `_p99`. + +**Fix:** ensure callers use a Temporal SDK (or add `UNAVAILABLE` retries to raw clients); check and wait out replication lag before manual failovers. + +## Worker placement — triage-layer pointer + +The triage concern is narrow: confirm workers can reach whichever region is currently active and that they follow the CNAME rather than hard-coding a Regional Endpoint. + +- Enabling HA requires no special worker configuration; the DNS redirection is invisible to workers using the Namespace Endpoint. +- Two supported configurations: run workers in both regions continuously, or establish cross-region connectivity (Transit Gateway / VPC Peering) so a single-region fleet can reach the newly active region. +- In a full regional outage, workers in that region may fail alongside the primary; a second fleet in the replica's region keeps Workflows moving. + +Pattern selection (cost, latency, operational complexity) is a `skill-temporal-deploy` concern, not triage. + +## RPO / RTO reference + +Full reference: `/cloud/rpo-rto`. Triage-relevant facts only: + +| Fact | Value | Source | +|---|---|---| +| HA target (cell / regional / Multi-cloud cloud-wide outages) | sub-1-minute RPO, 20-minute RTO | | +| AZ outages (all Namespaces, not just HA) | zero RPO, near-zero RTO via 3-AZ replication | | +| Automatic Failover disabled | published RTO does not apply (Temporal can't control when the user triggers) | | +| Failback responsibility | automatic after an Automatic Failover; user's responsibility after a user-triggered failover | | + +For manual-failover sequencing (why an operator might trigger faster than Temporal, how to order application-side vs Namespace failover), see `/cloud/rpo-rto` § "Tips for a lower Recovery Time". Region health: `https://status.temporal.io`. diff --git a/plugins/temporal/skills/temporal-ops/references/triage/non-determinism.md b/plugins/temporal/skills/temporal-ops/references/triage/non-determinism.md new file mode 100644 index 0000000..80f2785 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/triage/non-determinism.md @@ -0,0 +1,221 @@ +# Non-Determinism + +This file scopes to one specific Workflow Task failure subtype: the one whose `WorkflowTaskFailedCause` is the Nondeterminism cause — i.e. the Workflow Task failed because replaying the recorded Event History against the currently-loaded Workflow code produced a Command that did not match the next Event. + +Non-determinism is a subtype of the WFT-failure loop documented in [workflow-stuck.md](workflow-stuck.md#pending-workflow-task-and-workflowtaskfailed-loops). For the taxonomy of *other* WFT-failure causes (Unhandled Command, Pending Activities Limit Exceeded, Bad Search Attributes, Workflow Worker Unhandled Failure), start there. + +Out of scope here: +- Workflow stuck in general, including other WFT-failure causes → [workflow-stuck.md](workflow-stuck.md) +- Replaying an Event History locally under a debugger → [replay.md](replay.md) +- Worker not polling the Workflow Task Queue at all → [worker-health.md](worker-health.md) +- gRPC `RESOURCE_EXHAUSTED` on the client that tries to describe or reset → [rate-limits.md](rate-limits.md) +- The bottom-up layer model for routing between files → [diagnostic-ladder.md](diagnostic-ladder.md) + +## Table of Contents + +- [What determinism means in Temporal](#what-determinism-means-in-temporal) +- [How replay detection works](#how-replay-detection-works) +- [The WFT-failure signature of non-determinism](#the-wft-failure-signature-of-non-determinism) +- [Why ND does not fail the Workflow](#why-nd-does-not-fail-the-workflow) +- [What the docs call out as ND-inducing patterns](#what-the-docs-call-out-as-nd-inducing-patterns) +- [Per-SDK error shape](#per-sdk-error-shape) +- [Identifying ND from the Event History](#identifying-nd-from-the-event-history) +- [Reproducing ND locally via replay](#reproducing-nd-locally-via-replay) +- [Remediation: Worker Versioning (preferred)](#remediation-worker-versioning-preferred) +- [Remediation: per-SDK patching](#remediation-per-sdk-patching) +- [Remediation: fix and redeploy, or reset past the divergence](#remediation-fix-and-redeploy-or-reset-past-the-divergence) +- [Quick routing](#quick-routing) + +## What determinism means in Temporal + +The Temporal Platform requires that Workflow code is deterministic. A Replay "recreates the exact state of a Workflow Execution" by running the Workflow code against the recorded Event History, and "Replay succeeds only if the Workflow Definition is compatible with the provided history from a deterministic point of view." + +Per the encyclopedia's formal definition: "The use of certain Workflow APIs in the function is what generates Commands. Commands tell the Temporal Service which Events to create and add to the Workflow Execution's Event History. When the Workflow's code replays, the Commands that are emitted are compared with the existing Event History. If a corresponding Event already exists within the Event History that matches that command, then the Execution progresses." + +"If a generated Command doesn't match what it needs to in the existing Event History, then the Workflow Execution returns a non-deterministic error." + +Two reasons a Command may not match: (1) code changes to a Workflow Definition that is in use by a running Workflow Execution, or (2) intrinsic non-deterministic logic such as inline random branching. + +## How replay detection works + +Replay is "the method by which a Workflow Execution resumes making progress. During a Replay the Commands that are generated are checked against an existing Event History." Replay happens whenever a Worker picks up a Workflow Task for a Workflow Execution whose in-memory state is not already present on that Worker — i.e. after a Worker restart, after a sticky-cache eviction, or when a different Worker picks up the next Workflow Task. + +At Command-emission time, the Worker compares the Command it is about to emit against the next non-bookkeeping Event in the recorded history. A mismatch at any point (wrong Command type, wrong attributes like Activity name, or a Command appearing where none was recorded) surfaces as a non-determinism error inside the Worker. The Worker reports the Workflow Task as failed; the server appends a `WorkflowTaskFailed` Event with `cause` set to the Nondeterminism value of the `WorkflowTaskFailedCause` enum. + +## The WFT-failure signature of non-determinism + +Each `WorkflowTaskFailed` Event corresponds to a value of the `WorkflowTaskFailedCause` enum, exposed in the Event's `workflow_task_failed_event_attributes` as `cause`. The Nondeterminism cause is documented in `errors.mdx` as: + +> The Workflow Task failed due to a nondeterminism error. + +The event attributes schema — `scheduled_event_id`, `started_event_id`, `failure`, `identity`, `base_run_id`, `new_run_id`, `fork_event_version`, `binary_checksum` — is shared with every other `WorkflowTaskFailed` cause. The Nondeterminism-specific detail (which Command diverged from which Event) is carried inside the `failure` field, produced by the SDK. + +## Why ND does not fail the Workflow + +Workflow Task failures do not fail the Workflow Execution. The server retries failed Workflow Tasks so that, once the Worker code is fixed and redeployed, the Workflow resumes. The observable effect of a Nondeterminism WFT failure is: + +- The Workflow Execution stays in the `Running` status. +- `pendingWorkflowTask.attempt` (shown by `temporal workflow describe`) climbs. +- The Event History accumulates repeating `WorkflowTaskFailed` Events with `cause` = Nondeterminism. +- Worker logs repeat an SDK-specific non-determinism exception (see [Per-SDK error shape](#per-sdk-error-shape)). + +This "the Workflow is durable, the bad deploy is not" behavior is intentional: fixing the Worker code and redeploying, or resetting the Workflow past the divergence point, resumes the execution without losing its progress to that point. + +Contrast with ordinary Workflow failures. An unhandled exception inside an Activity produces `ActivityTaskFailed` and, depending on the Activity's Retry Policy, either retries or propagates to the Workflow. An unhandled exception inside the Workflow function itself produces the distinct `Workflow Worker Unhandled Failure` WFT cause, which is routed out of this file (worker logs for stack trace, fix and redeploy; see [workflow-stuck.md](workflow-stuck.md#pending-workflow-task-and-workflowtaskfailed-loops)). Neither of those is a Nondeterminism error — the classification lives in the `cause` field. + +## What the docs call out as ND-inducing patterns + +Two categories, per the encyclopedia: + +### Code changes to a Workflow Definition in use + +"The Workflow Definition can change in very limited ways once there is a Workflow Execution depending on it." The canonical example in the docs: a Workflow that was started under a definition of *Timer then Activity* cannot be migrated to a definition of *Activity then Timer* without versioning; on the next Workflow Task, "the first Command the Worker sees would be ScheduleActivityTask Command, which wouldn't match up to the expected TimerStarted Event. The Workflow Execution would fail and return a nondeterminism error." + +Minor changes that do *not* cause ND on replay of histories that already contain the corresponding Events: + +- Changing the duration of a Timer, with language-specific exceptions: "In Java, Python, and Go, changing a Timer's duration from or to 0 is a non-deterministic behavior. In .NET, changing a Timer's duration from or to -1 (which means 'infinite') is a non-deterministic behavior." +- Changing arguments to Activity Options, Child Workflow Options, or the Signal External Workflow Execution call. +- Adding a Signal Handler for a Signal Type that has not been sent to this Workflow Execution. + +### Intrinsic non-determinism in the Workflow function + +"Intrinsic non-determinism is when a Workflow Function Execution might emit a different sequence of Commands on re-execution, regardless of whether all the input parameters are the same." "A Workflow Definition can not have inline logic that branches (emits a different Command sequence) based off a local time setting or a random number." + +Each SDK exposes replay-safe alternatives. The per-SDK constraint lists documented in `develop//workflows/basics.mdx`: + +- **Go.** "Iterate over maps using `range`, because with `range` the order of the map's iteration is randomized." Use sorted keys, a Side Effect, or an Activity instead. Direct calls to external APIs, file I/O, or other services must be wrapped in Activities. Replay-safe substitutes include `workflow.Now()` (for `time.Now()`), `workflow.Sleep()`, `workflow.GetLogger()`, `workflow.Go()` (for the `go` statement), `workflow.Channel` (for native `chan`), `workflow.Selector` (for `select`), and `workflow.Context`. Go's map iteration order is not specified by the language spec. +- **Python.** "Workflow code must be deterministic because the Temporal Server may replay your Workflow to reconstruct its state. This means: no threading, no randomness, no external calls to processes, no network I/O, no global state mutation, no system date or time." Replay-safe substitutes: `workflow.random()` (for `random.random()`), `workflow.uuid4()` (for `uuid.uuid4()`), `workflow.now()` (for `datetime.now()` / `time.time()`), `workflow.logger` (for `print()` / `logging`). +- **TypeScript, Java, .NET, Ruby, PHP.** Per-SDK constraint lists and replay-safe APIs are linked from the encyclopedia. + +Also documented: changes to "Patched or GetVersion calls for Versioning (although they may be added or removed according to the patching rules)" are among the minor allowed changes. In other words, patching and removing patches have documented rules; ad-hoc additions of non-patched activity calls in the middle of a Workflow do not. + +What the docs do *not* prescribe here: a ranked list of "most common causes." Treat the two categories above as exhaustive framings. Any diagnosis that does not map to one of them should be escalated to a VERIFY. + +## Per-SDK error shape + +The error class or message that the Worker emits when replay diverges differs per SDK. Only the names actually verified in the docs are listed: + +- **TypeScript.** "When an Event History is replayed and non-determinism is detected (that is, the Workflow code is incompatible with the History), `DeterminismViolationError` is thrown. If replay fails for any other reason, `ReplayError` is thrown." +- **Go.** The Go SDK emits a non-determinism error through the Workflow Task failure path; the docs for versioning describe the condition as "cause the Workflow to fail with a nondeterminism error" without pinning a public class name. +- **Java.** The Java versioning doc describes the condition as "This would cause the Workflow to fail with a nondeterminism error" without pinning a public exception class. +- **Python.** The Python versioning doc describes the condition as "cause a nondeterminism error" without pinning a public exception class. +- **.NET, Ruby, PHP.** Per-SDK class names are not documented in the files consulted. + +Server-side, regardless of SDK, the cause surfaces as the Nondeterminism `WorkflowTaskFailedCause` in the Event History. Rely on that server-side cause for cross-SDK classification; rely on the per-SDK error class only when debugging from worker logs. + +## Identifying ND from the Event History + +The server-side signal that unambiguously marks ND is a `WorkflowTaskFailed` Event whose attributes carry the Nondeterminism `cause`. Extract via: + +```bash +temporal workflow show \ + --workflow-id YourWorkflowId \ + --output json > history.json +``` + + Then filter the JSON for `WorkflowTaskFailed` events and inspect each one's `workflowTaskFailedEventAttributes.cause` and `workflowTaskFailedEventAttributes.failure` fields. The exact JSON field names come from the server proto; the encyclopedia identifies them as `workflow_task_failed_event_attributes`. + +`temporal workflow describe` also surfaces a `pendingWorkflowTask` with a rising `attempt` when WFT failures are looping. A rising attempt count plus `WorkflowTaskFailed` Events with Nondeterminism cause is the confirming two-signal match. + +If `describe` shows WFT failures but the history JSON does not report Nondeterminism as the cause, the WFT is failing for a different reason — return to [workflow-stuck.md §Pending Workflow Task and WorkflowTaskFailed loops](workflow-stuck.md#pending-workflow-task-and-workflowtaskfailed-loops) and match the actual cause. + +## Reproducing ND locally via replay + +Running the recorded history through a local replayer against a Worker source tree is the canonical reproducer. "Replay recreates the exact state of a Workflow Execution. You can replay a Workflow from the beginning of its Event History. Replay succeeds only if the Workflow Definition is compatible with the provided history from a deterministic point of view." + +Per-SDK replay APIs (names transcribed from the testing-suite pages; cross-check against the SDK version in use): + +- **Go.** `worker.NewWorkflowReplayer()` + `replayer.ReplayWorkflowHistory(logger, history)`. +- **Python.** `Replayer(workflows=[YourWorkflow])` + `replayer.replay_workflow(WorkflowHistory.from_json(history_json_str))`, or `replayer.replay_workflows(histories)` for bulk. +- **TypeScript.** `Worker.runReplayHistory(options, history)` for single histories, `Worker.runReplayHistories(...)` for bulk. +- **Java.** `WorkflowReplayer.replayWorkflowExecution(file, MyWorkflow.class)` for single histories, `WorkflowReplayer.replayWorkflowExecutions(...)` for bulk. + +For an interactive reproducer with breakpoints over the same replayer APIs, see [replay.md](replay.md). + +Fetch the history with the SDK client in the same test that replays it — per-SDK calls in [replay.md → Step 1](replay.md#step-1--get-the-event-history). Where a file is needed instead: + +```bash +temporal workflow show \ + --workflow-id YourWorkflowId \ + --run-id YourRunId \ + --output json > history.json +``` + +If local replay *succeeds* against the checked-out source but production Workers keep failing, the deployed Worker code is different from the local checkout. Find the deployed build and either roll it back or patch it forward. + +If local replay *fails* with the same error, iterate on the Workflow source until replay succeeds. + +## Remediation: Worker Versioning (preferred) + +"For most teams, Worker Versioning should be the default recommendation for deploying Workflow code changes in production. If you can run versioned worker deployments, prefer Worker Versioning over patching." + +The mechanism: "Worker Versioning introduces Workflow Pinning. For pinned Workflow Types, each execution runs entirely on the Worker Deployment Version where it started. You need not worry about making breaking code changes to running, pinned Workflows." "Pinned Workflows don't need to be patched, as they run on the same worker and build until they complete." + +Minimum versions for Worker Versioning (Public Preview as of the docs snapshot): Go v1.35.0, Python v1.11, Java v1.29, TypeScript v1.12, .NET v1.7.0, Ruby v0.5.0; Temporal CLI v1.4.1, Server v1.29.1, UI v2.38.0. + +Adoption in the middle of triaging an ND incident is not a remediation — it is a prevention for future changes. For Workflows that are *already* in the ND loop and were started before Worker Versioning was enabled for their Workflow Type, use patching or reset. + +## Remediation: per-SDK patching + +Patching lets Workflow code branch on whether it is replaying history recorded before or after a specific change. Patching APIs per SDK (each citation links to the canonical docs page; only names present in those pages are listed): + +- **Go.** `workflow.GetVersion(ctx, "change-id", workflow.DefaultVersion, 1)`. "When `workflow.GetVersion()` is run for the new Workflow Execution, it records a marker in the Event History so that all future calls to `GetVersion` for this change Id ... will always return the given version number." A Workflow that has already passed this `GetVersion()` call before it was introduced returns `DefaultVersion`. +- **TypeScript.** `patched('change-id')` to branch, `deprecatePatch('change-id')` to mark a patch as deprecated. "Using `patched` inserts a marker into the Workflow History. During Replay, if a Worker encounters a history with that marker, it will fail the Workflow task when the Workflow code doesn't produce the same patch marker." +- **Java.** `Workflow.getVersion("change-id", Workflow.DEFAULT_VERSION, 1)`. "Each call to `Workflow.getVersion` automatically upserts the `TemporalChangeVersion` Search Attribute with a keyword list of `"-"` entries." +- **Python.** `workflow.patched('change-id')` to branch, `workflow.deprecate_patch('change-id')` to mark deprecated. +- **.NET.** `Workflow.Patched("change-id")`, `Workflow.DeprecatePatch("change-id")`. + +The three-phase lifecycle, per each SDK's versioning page: + +1. Patch in new code using the branching API alongside the old code. +2. Once no open Workflow Executions are on the old code, switch to the deprecate-patch API. +3. Once all patched Workflows have left retention, remove the patch call entirely. + +The patching behavior at replay time is documented in detail in `docs/encyclopedia/workflow/patching.mdx`. Key consequence: if the execution hits a `patched()` call during replay but the marker is *after* the current execution point in history, "it will throw a non-deterministic exception because the replay and original event histories don't match." Put the newest code at the top of patch blocks: "when patching in new code, always put the newest code at the top of an if-patched-block." + +## Remediation: fix and redeploy, or reset past the divergence + +Because ND WFT failures retry server-side, there are two live-incident paths: + +### Fix the Worker code and redeploy + +If the divergence was introduced by a recent deploy and the broken Worker can be replaced with one whose code matches the recorded history (either by reverting the change or by adding a patch branch), redeploy and let the server-retried Workflow Tasks succeed on the new Workers. The Workflow resumes from the point where it started failing; history up to that point is intact. + +### Reset the Workflow past the divergence + +If the divergence is already embedded in the history (e.g. new workflows have executed the broken path and recorded it) and a simple redeploy cannot recover them, `temporal workflow reset` rewinds a Workflow to an earlier Event ID, terminates the current run, and starts a new run with history copied up to the reset point. + +```bash +temporal workflow reset \ + --workflow-id YourWorkflowId \ + --event-id YourLastGoodEventId +``` + + Valid reset points per the Event encyclopedia are "`WorkflowTaskStarted`, `WorkflowTaskCompleted`, `WorkflowTaskTimedOut`, and `WorkflowTaskFailed`." Choose a `WorkflowTaskCompleted` immediately before the first failing WFT as the reset target: replay from that point against the fixed code avoids the divergence. + +For batch resets, the `--type` values permitted are `FirstWorkflowTask`, `LastWorkflowTask`, and `BuildId`. The single-run reset example in the docs uses `LastContinuedAsNew`. + +```bash +temporal workflow reset \ + --workflow-id YourWorkflowId \ + --type LastContinuedAsNew +``` + + `--reapply-type` controls which Events are reapplied after the reset point; accepted values are `Signal, None`. + +Confirm with the business owner before resetting: the Events after the reset point are re-executed, which means any activity that was run post-reset-point will be re-scheduled on replay — fine for idempotent operations, hazardous for irreversible external side effects. + +## Quick routing + +| Evidence | Go to | +|---|---| +| `WorkflowTaskFailed` Events with `cause` = Nondeterminism in the Event History | This file | +| `pendingWorkflowTask.attempt` climbing, but `WorkflowTaskFailed` Events show a different `cause` | [workflow-stuck.md §Pending Workflow Task and WorkflowTaskFailed loops](workflow-stuck.md#pending-workflow-task-and-workflowtaskfailed-loops) | +| Worker logs show a class like `DeterminismViolationError` (TypeScript) or a nondeterminism error (other SDKs) | [Per-SDK error shape](#per-sdk-error-shape), [Reproducing ND locally](#reproducing-nd-locally-via-replay) | +| Need to reproduce the ND locally with breakpoints | [replay.md](replay.md) | +| Need a CI regression test around a fixed ND bug | [Reproducing ND locally via replay](#reproducing-nd-locally-via-replay) — use the SDK's bulk replayer | +| Planning a future-proof deploy strategy | [Remediation: Worker Versioning (preferred)](#remediation-worker-versioning-preferred) | +| Live incident, pre-existing Workflows are stuck, deploy-the-fix is not enough | [Remediation: fix and redeploy, or reset past the divergence](#remediation-fix-and-redeploy-or-reset-past-the-divergence) | +| The describe call itself fails (cannot reach/auth/authorize) | [connectivity.md](connectivity.md), [certificates.md](certificates.md), [authentication.md](authentication.md), [rate-limits.md](rate-limits.md) | + +For the whole-stack picture and where this file sits in the diagnostic order, see [diagnostic-ladder.md](diagnostic-ladder.md). diff --git a/plugins/temporal/skills/temporal-ops/references/triage/performance-bottlenecks.md b/plugins/temporal/skills/temporal-ops/references/triage/performance-bottlenecks.md new file mode 100644 index 0000000..f558383 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/triage/performance-bottlenecks.md @@ -0,0 +1,242 @@ +# Performance Bottlenecks + +Diagnose performance bottlenecks in Temporal Workers and Clients using SDK metrics. This reference covers key latency metrics, root causes, and diagnostic steps. + +--- + +## Task processing metrics + +### `temporal_workflow_task_schedule_to_start_latency` spike + +Time between when a Workflow Task is scheduled (enqueued) and when it is picked up by a Worker. P95 higher than one second is a concern. + +Potential causes: + +- **Insufficient Worker capacity:** Not enough Workers or overloaded Workers cannot pick up Tasks quickly enough. +- **Worker configuration issues:** Too few pollers or Task slots. +- **High Workflow lock latency:** Many updates to a single execution cause lock contention. Reduce the rate of Signals. +- **Network latency:** Workers in a different region from the Temporal cluster, or large payload sizes. + +Diagnostic steps: + +1. Check Worker CPU and memory usage. +2. Review Worker configuration (number of pollers, Task slots, etc.). +3. Look for spikes in Workflow or Activity starts. +4. Ensure Workers are in the same region as the Temporal cluster. + +### `temporal_activity_schedule_to_start_latency` spike + +Time between when an Activity Task is scheduled and when it is picked up by a Worker. P95 higher than one second is a concern. + +Potential causes: + +- **Insufficient Worker capacity.** +- **Worker configuration issues:** Too few pollers or Task slots. +- **Task Queue configuration:** `TaskQueueActivitiesPerSecond` set too low. +- **Network latency.** + +Diagnostic steps: + +1. Check Worker CPU and memory usage. +2. Review Worker configuration. +3. Look for spikes in Workflow or Activity starts. +4. Ensure Workers are in the same region. + +### `temporal_workflow_endtoend_latency` spike + +Total Workflow Execution time from Schedule to closure for a single Workflow Run. Normal ranges depend on the use case. + +Potential causes: + +- **Complex Workflows:** Many Activities or slow Activity execution. +- **Workflow and Activity retries:** Frequent failures with retry delays. +- **Worker capacity and configuration.** +- **External dependencies:** Slow databases, APIs, or services. +- **Network latency.** + +Diagnostic steps: + +1. Review Workflow and Activity designs for efficiency. +2. Monitor Worker capacity (CPU, memory). +3. Monitor external dependencies. +4. Ensure Workers are co-located with the Temporal cluster. + +### High `temporal_workflow_task_execution_latency` + +Time taken by a Worker to execute a Workflow Task. The SDK raises a "Deadlock detected during Workflow run" error or TMPRL1101 when a Workflow Task takes more than one or two seconds. + +Potential causes: + +- **CPU-intensive work in Workflow Task.** +- **Slow Local Activities** (execution time included in the Workflow Task). +- **Slow Workflow replay** (see `workflow_task_replay_latency`). +- **Worker resource constraints:** High CPU usage on Worker pods. +- **Infinite loops or blocking calls in Workflow code.** +- **Slow data conversion:** Custom Data Converter taking too long. + +Diagnostic steps: + +1. Monitor Worker CPU and memory utilization. +2. Ensure Workers have adequate resources and scaling. +3. Run Workflow code in a profiler using a replayer. +4. Review Workflow code for optimizations or blocking operations. +5. For Data Converter: disable deadlock detection (Go: `workflow.DataConverterWithoutDeadlockDetection`; Java: `WorkflowUnsafe.deadlockDetectorOff`). This removes the error but does not reduce latency. + +### High `workflow_task_replay_latency` + +Time to replay a Workflow Task by re-executing the Workflow code from the beginning using the recorded Event History. High if it exceeds a few milliseconds. + +Potential causes: + +- **Large Event Histories.** +- **Slow Data Converters** (especially encryption or external services). +- **Large payloads** in Activities or Signals. +- **Complex Workflow logic** (many concurrent child Workflows or Activities). +- **Frequent cache evictions** (memory constraints or frequent restarts). +- **Worker resource constraints.** + +Diagnostic steps: + +1. Monitor `temporal_workflow_task_replay_latency`. +2. Analyze Workflow History size; consider Continue-As-New for long-running Workflows. +3. Optimize Data Converters. +4. Review payload sizes. +5. Profile Workflow code. +6. Tune Worker cache size and eviction policies. + +### `temporal_activity_execution_latency` spike + +Time from when a Worker starts processing an Activity Task until it reports completion or failure. + +Potential causes: + +- **Activity implementation:** Time-consuming operations or slow external API calls. +- **External dependencies:** Shared external resources causing contention. +- **Worker resource constraints.** +- **Network latency** between Workers and external services. + +Diagnostic steps: + +1. Monitor `activity_execution_latency` (filter by Activity type and Task Queue). +2. Optimize Activity implementation. +3. Check Worker CPU and memory. +4. Check Worker configuration: `(Max)ConcurrentActivityExecutionSize` and `(Max)WorkerActivitiesPerSecond`. + +--- + +## Task slot depletion + +### `temporal_worker_task_slots_available{worker_type="WorkflowWorker"}` at zero + +Available slots for executing Workflow Tasks on a Worker. + +Potential causes: + +- **High Workflow Task load** exceeding concurrent capacity. +- **Worker configuration:** `MaxConcurrentWorkflowTaskExecutionSize` set too low. +- **High `temporal_workflow_task_execution_latency`** and `workflow_task_replay_latency`. + +Resolution: + +1. Monitor Worker CPU and Memory while increasing `(Max)ConcurrentWorkflowTaskExecutionSize`. +2. Scale Workers vertically (CPU, Memory) and horizontally (more instances). + +### `temporal_worker_task_slots_available{worker_type="ActivityWorker"}` at zero + +Available slots for executing Activity Tasks on a Worker. + +Potential causes: + +- **Blocked Activities and Zombie Activities:** Activities blocked on downstream services or infinite loops. Zombie Activities occur when an Activity times out (`StartToClose` or `HeartbeatTimeout`) but continues running, occupying slots as retries occur. +- **Resource utilization:** High CPU or memory causing Activities to block. + +Resolution: + +1. Monitor Worker CPU and Memory while increasing `(Max)ConcurrentActivityExecutionSize`. +2. Add client-side timeout to downstream API clients. +3. Review Task code to ensure completion within reasonable time. + +--- + +## Network request metrics + +### High `temporal_long_request_failure` + +Counts failed RPC long poll requests for `PollWorkflowTaskQueue`, `PollActivityTaskQueue`, and `GetWorkflowExecutionHistory`. + +Potential causes: + +- **Network issues** between Client and Server. +- **Rate limiting** (`ResourceExhausted` status code). +- **Server errors.** + +Diagnostic steps: + +1. Check the `operation` and `status`/`code` tag. +2. For `ResourceExhausted`, review rate limits. +3. Check the network connection. + +### High `temporal_request_failure_total` + +Counts total failed RPC requests. + +Potential causes: + +- **Network issues.** +- **Client errors** (misconfiguration, resource exhaustion). +- **Operation errors** (acting on a closed Workflow past retention time). +- **Rate limiting** (`ResourceExhausted` status code). +- **Request size limit** (blob size limit of 2 MB). +- **Server errors.** + +Diagnostic steps: + +1. Check the `status`/`code` tag. +2. Check the `operation` tag. +3. Monitor Server and Client logs. +4. Check the network connection. + +### High `temporal_request_latency` + +Latency of gRPC requests made by the Temporal Client. + +Potential causes: + +- **Network latency** (distance, conditions). +- **Network transfer time** (large payloads). +- **Resource exhaustion** (CPU, memory on client or server). +- **Client configuration** (thread pool sizes, memory constraints). +- **Server load.** + +Diagnostic steps: + +1. Monitor `temporal_request_latency` for spike timing and location. +2. Check the network connection. +3. Monitor resource usage on Client and Server. +4. Review Client configuration. +5. For Temporal Cloud, check `service-latency` metric and contact Support. + +--- + +## Caching metrics + +### `temporal_sticky_cache_size` + +Number of Workflow executions cached in a Worker's memory. Sticky cache keeps Workflow state in memory, reducing the need to replay from Event History. + +Monitor alongside Worker memory usage. A sudden increase correlates with increased memory consumption. + +### `temporal_sticky_cache_hit_total` and `temporal_sticky_cache_miss_total` + +A "hit" means the Worker found the Workflow in its cache; a "miss" means the Worker must fetch the Event History and replay. High hit rate with low miss rate indicates efficient scheduling. + +### `temporal_sticky_cache_total_forced_eviction_total` + +Counts Workflow Executions forcibly evicted from the sticky cache (cache was full). A high eviction rate may indicate the cache size is too small; increase `WorkflowCacheSize` if resources allow. + +--- + +## Sibling skill pointers + +- For detailed Worker tuning recommendations (slot counts, poller counts, cache sizes), see the worker tuning skill (`skill-temporal-workertuning`). +- For metrics collection and dashboard setup, see the observability skill (planned: `skill-temporal-observability`). diff --git a/plugins/temporal/skills/temporal-ops/references/triage/rate-limits.md b/plugins/temporal/skills/temporal-ops/references/triage/rate-limits.md new file mode 100644 index 0000000..5366576 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/triage/rate-limits.md @@ -0,0 +1,159 @@ +# Rate Limits + +Diagnose gRPC `RESOURCE_EXHAUSTED` against Temporal — what budget was exceeded, which party enforced it, and how to tell from the error text and Cloud signals. This file covers one specific gRPC status code; it does not cover every "things feel slow" symptom. + +Prerequisite: `RESOURCE_EXHAUSTED` is returned *after* the connection, TLS, and auth layers have succeeded. If you do not yet know that the caller is reaching the frontend, rule out layers 1–3 first via [connectivity.md](connectivity.md), [certificates.md](certificates.md), and [authentication.md](authentication.md). + +Out of scope here: +- DNS / TCP / endpoint → [connectivity.md](connectivity.md) +- TLS / x509 → [certificates.md](certificates.md) +- `UNAUTHENTICATED` / `PERMISSION_DENIED` → [authentication.md](authentication.md) +- Task-queue backlog, poller count, worker health → [worker-health.md](worker-health.md) +- Workflow stuck in a pending state (pending activity / pending child / pending signal) → [workflow-stuck.md](workflow-stuck.md) +- Ambiguous `context deadline exceeded` that callers sometimes confuse with throttling → [runtime-errors.md](runtime-errors.md) + +## Table of Contents + +- [What RESOURCE_EXHAUSTED means and what it does not](#what-resource_exhausted-means-and-what-it-does-not) +- [Temporal Cloud: APS, RPS, OPS and capacity modes](#temporal-cloud-aps-rps-ops-and-capacity-modes) +- [Self-hosted: service-level RPS via dynamic configuration](#self-hosted-service-level-rps-via-dynamic-configuration) +- [Server throttled me vs. client throttled itself](#server-throttled-me-vs-client-throttled-itself) +- [Identifying which limit was hit](#identifying-which-limit-was-hit) +- [What RESOURCE_EXHAUSTED is not](#what-resource_exhausted-is-not) +- [Quick routing](#quick-routing) + +## What RESOURCE_EXHAUSTED means and what it does not + +`RESOURCE_EXHAUSTED` is one of the 17 canonical gRPC status codes. In the spec it means "Some resource has been exhausted, perhaps a per-user quota, or perhaps the entire file system is out of space." On Temporal, it is the code the frontend returns when a request is **throttled against a configured rate limit** rather than rejected for identity or input reasons. + +Key scoping facts: + +- It is not `UNAVAILABLE` (connectivity / transient peer failure), not `DEADLINE_EXCEEDED` (the caller's deadline fired), and not `PERMISSION_DENIED` (identity known, action forbidden). Mapping a rate-limit symptom onto one of those codes is a common triage mistake — see [authentication.md → UNAUTHENTICATED vs PERMISSION_DENIED](authentication.md#unauthenticated-vs-permission_denied) for the boundary. +- Temporal Cloud documents the server-returned value as `ResourceExhausted`: "When throttled, the server returns a `ResourceExhausted` gRPC error." The gRPC wire name is `RESOURCE_EXHAUSTED`; client SDKs surface it as an `ResourceExhausted`-named exception. Both refer to the same code. +- On self-hosted, the same code is emitted when the frontend's RPS budget is exceeded: "Exceeding these limits results in `ResourceExhaustedError`." + +## Temporal Cloud: APS, RPS, OPS and capacity modes + +Cloud enforces three **per-Namespace** throughput limits, each of which can independently produce `RESOURCE_EXHAUSTED`: + +| Limit | Default | Scope | What it counts | +|---|---|---|---| +| **Actions per second (APS)** | 500 APS | Namespace | Billable [Actions](https://docs.temporal.io/cloud/actions) — starts, Signals, Updates, Activity starts / retries / heartbeats, Timers, etc. | +| **Requests per second (RPS)** | 2000 RPS | Namespace | gRPC requests to the frontend. A lower-level measure of load at the service level. | +| **Operations per second (OPS)** | 4000 OPS | Namespace | Operations (user-driven or Temporal-internal) that hit the Temporal Server on the user's behalf. See the [operations list](https://docs.temporal.io/references/operation-list) for the set. | + +All three have the same **automatic scaling** behavior under the default **On-Demand Capacity** mode: the limit "automatically increases (and decreases) based on the last 7 days of [APS/RPS/OPS] usage. Will never go below the default limit." The On-Demand formula is documented as "the lesser of 4 × APS Average or 2 × APS P90 over the past 7 days." + +**Provisioned Capacity** replaces the automatic envelope with a fixed allocation, expressed in Temporal Resource Units (TRUs); each TRU supplies 500 APS / 1500 RPS / 4000 OPS. TRUs are set via UI, `tcld namespace capacity update`, or the `UpdateNamespace` API. Adjustments are hourly. + +There are additional, narrower Namespace-scoped rate limits that also surface as `RESOURCE_EXHAUSTED`: + +- **Schedules rate limit**: 10 schedule requests per second, per Namespace, not configurable via UI — raise via a support ticket. The Cloud metric `temporal_cloud_v1_schedule_rate_limited_count` tracks workflows delayed due to this limit. +- **Visibility API rate limit**: 30 Visibility API calls per second per Namespace; not configurable. "All read calls are subject to the Visibility API rate limit." +- **Concurrent Task pollers**: 20,000 Activity pollers and 20,000 Workflow Task pollers per Namespace concurrently. Per-Namespace poll saturation falls under poller health — see [worker-health.md](worker-health.md). + +### Throttling behavior (Cloud) + +From the Cloud limits reference: + +- **Priority-based throttling.** "Low-priority operations are throttled first. Higher-priority operations like `StartWorkflowExecution`, `SignalWorkflowExecution`, and `UpdateWorkflowExecution` continue to go through when possible." +- **Throttling latency.** "Rate limiting is not instantaneous, so usage may briefly exceed your limit before throttling takes effect." +- **SDK retry by default.** "SDK clients automatically retry these based on the default gRPC retry policy." +- **Persistent throttling fails the call.** "If throttling persists beyond the SDK's retry limit, client calls fail." +- **Per the Cloud docs, Actions that are external to the core Temporal service do not contribute to APS** — e.g. [Export](https://docs.temporal.io/cloud/export) and Capacity-related Actions. + +## Self-hosted: service-level RPS via dynamic configuration + +On a self-hosted cluster, per-Namespace and per-host rate limits are dynamic-configuration keys. The docs name these explicitly as producers of `ResourceExhaustedError`: "Exceeding these limits results in `ResourceExhaustedError`." + +Commonly referenced frontend keys (defaults from `docs/references/dynamic-configuration.mdx`, Temporal server v1.21; verify against the version you run): + +| Key | Default | What it limits | +|---|---|---| +| `frontend.rps` | 2400 | Requests per second accepted by each Frontend Service host | +| `frontend.namespaceRPS` | 2400 | Requests per second per Namespace, per Frontend host | +| `frontend.globalNamespaceRPS` | 0 (disabled) | Cluster-wide per-Namespace RPS, distributed across Frontend hosts. When set, overrides `frontend.namespaceRPS`. | +| `history.rps` | 3000 | Per History Service host | +| `matching.rps` | 1200 | Per Matching Service host | + +Defaults shift across server versions. Before quoting a number to a user, check their server version against `docs/references/dynamic-configuration.mdx` or the release notes. + +Persistence-store QPS keys (`frontend.persistenceMaxQPS`, `history.persistenceMaxQPS`, etc.) are evaluated synchronously and produce latency / timeouts rather than `RESOURCE_EXHAUSTED` to the client: "If the number of queries made to the Persistence store exceeds the dynamic configuration value, you will see latencies and timeouts on your tasks." Persistence saturation therefore usually surfaces as `DEADLINE_EXCEEDED`, not `RESOURCE_EXHAUSTED` — see [runtime-errors.md](runtime-errors.md). + +## Server throttled me vs. client throttled itself + +`RESOURCE_EXHAUSTED` on the wire is always server-emitted. But "why is my caller falling behind?" can also be the caller throttling *itself* — those two look alike from a dashboard but need different fixes. + +- **Server-enforced limit hit → `RESOURCE_EXHAUSTED` returned to the caller.** Cloud APS/RPS/OPS; self-hosted `frontend.rps`/`frontend.namespaceRPS`; Cloud Schedules limit; Visibility API limit. The SDK default retry policy retries the RPC with exponential backoff until its retry budget is exhausted, after which the call fails to the application. +- **Client-side throttling (no `RESOURCE_EXHAUSTED` on the wire).** A worker may be limiting its own rate via SDK settings — e.g. `TaskQueueActivitiesPerSecond` or `(Max)WorkerActivitiesPerSecond`. These produce Activity schedule-to-start latency, not `RESOURCE_EXHAUSTED`: "Setting `TaskQueueActivitiesPerSecond` too low can limit the rate at which Activities are started, leading to increased Schedule-to-start latency." If the reported error is high `temporal_activity_schedule_to_start_latency` without `RESOURCE_EXHAUSTED`, the problem is worker-side — go to [worker-health.md](worker-health.md). + +A quick discriminator: inspect the failed RPC's gRPC status. If the code is literally `RESOURCE_EXHAUSTED`, the server throttled it. If the RPC never left the worker or returned some other status, the ceiling is client-side. + +Rule of thumb: **retries themselves count against the budget**. A caller that retries `RESOURCE_EXHAUSTED` without backoff makes the situation worse. The default gRPC retry policy used by the SDKs already includes backoff; custom clients that reimplement retry need to do the same. + +## Identifying which limit was hit + +Two sources of signal: the gRPC error itself and the Cloud metrics endpoint. + +### From the error + +`RESOURCE_EXHAUSTED` carries a free-text message whose wording varies by server version and by which internal limiter fired, so rely on the gRPC code plus the metric label rather than pattern-matching the text. Which label depends on the metric family: + +| Environment | Metric | Label to split by | +|---|---|---| +| Cloud, v1 metrics | `temporal_cloud_v1_resource_exhausted_error_count` | `operation` | +| Cloud, v0 metrics | `temporal_cloud_v0_resource_exhausted_error_count` | `resource_exhausted_cause`, "Cause for resource exhaustion" | +| Self-hosted | `service_errors_resource_exhausted` | `resource_exhausted_cause` | + +Self-hosted: the `deadline-exceeded` troubleshooting page recommends `sum(rate(service_errors_resource_exhausted{}[1m])) by (resource_exhausted_cause)` to check for `RpsLimit`, `ConcurrentLimit`, and `SystemOverloaded` causes. + +One documented message string does exist: per-Workflow lock contention appears in Service logs as `Workflow is busy.` That is a *different condition* from account-limit throttling and is not fixed by raising limits — see [runtime-errors.md → Workflow lock contention (BusyWorkflow)](runtime-errors.md#workflow-lock-contention-busyworkflow). + +### From Cloud metrics + +Temporal Cloud exposes an OpenMetrics endpoint whose limit / count / throttle triples let you tell "which budget is saturating" without guessing. + +| Budget | Limit metric | Count metric | Throttle metric | +|---|---|---|---| +| Actions | `temporal_cloud_v1_action_limit` | `temporal_cloud_v1_total_action_count` | `temporal_cloud_v1_total_action_throttled_count` | +| Frontend gRPC requests | `temporal_cloud_v1_service_request_limit` | `temporal_cloud_v1_service_request_count` | `temporal_cloud_v1_service_request_throttled_count` | +| Operations | `temporal_cloud_v1_operations_limit` | `temporal_cloud_v1_operations_count` | `temporal_cloud_v1_operations_throttled_count` | + +The v1 metrics are pre-computed per-second rates aggregated over a 1-minute window, so a sustained non-zero value on a `*_throttled_count` metric is the definitive Cloud signal that a specific budget is being hit. + +**Do not read `temporal_cloud_v1_resource_exhausted_error_count` as the throttling signal.** The throttle metrics in the table above are. "Tracking trends against your account limits is the most important throttling signal to monitor. Unlike Resource Exhaustion, which usually self-heals through retries, hitting a limit slows or stalls progress until the workload backs off or your capacity is increased." The v1 resource-exhausted metric explicitly "does not include throttling due to Namespace limits"; it counts bursts against a single resource — a Namespace, Task Queue, or Workflow ID — that the resource could not absorb in the moment, and persistent non-zero values indicate a hot resource. Route those to [runtime-errors.md → Workflow lock contention (BusyWorkflow)](runtime-errors.md#workflow-lock-contention-busyworkflow), which is the most common cause. + +For the v0 metric family, the equivalent is `temporal_cloud_v0_resource_exhausted_error_count`, "gRPC requests received that were rate-limited by Temporal Cloud, aggregated by cause." + +For Provisioned Capacity namespaces, the *envelope* metrics (`temporal_cloud_v1_action_on_demand_envelope_limit`, `temporal_cloud_v1_operations_on_demand_envelope_limit`, `temporal_cloud_v1_service_request_on_demand_envelope_limit`) show what the limit would be under On-Demand. Compare these against the currently provisioned limit metric to evaluate whether the provisioned allocation is too tight or too loose. + +### Without metrics (UI-only) + +When the caller has no metrics pipeline, the Cloud UI shows a recent APS usage summary on the Namespace's *Manage Capacity* panel, which can confirm whether the Namespace is actually running hot. + +## What RESOURCE_EXHAUSTED is not + +These are the common false positives — they look superficially similar but are **not** `RESOURCE_EXHAUSTED`: + +- **Worker task-queue backlog.** If Activity Tasks or Workflow Tasks sit in the queue without being picked up, the symptom is high `temporal_workflow_task_schedule_to_start_latency` or `temporal_activity_schedule_to_start_latency`. These are caused by "insufficient Worker capacity" or "worker configuration issues (too few pollers or task slots)" , not by server rate limiting. Diagnosis: see [worker-health.md](worker-health.md). +- **Workflow stuck with pending activities / children / signals.** The workflow made a Command (`ScheduleActivityTask`, `StartChildWorkflowExecution`, etc.), but nothing moves forward. This is a worker / queue / dependency issue, not a RESOURCE_EXHAUSTED condition. See [workflow-stuck.md](workflow-stuck.md). +- **`context deadline exceeded`.** The caller's own deadline fired before the server responded. This can *coincide* with a saturated namespace (the server-side `temporal_cloud_v0_resource_exhausted_error_count` may be non-zero), but the wire code received by the client is `DEADLINE_EXCEEDED`, not `RESOURCE_EXHAUSTED`. See [runtime-errors.md](runtime-errors.md) and the dedicated Cloud-side guidance. +- **`temporal_long_request_failure` spikes on poll RPCs.** The performance-bottlenecks guide calls out that high `temporal_long_request_failure` may be caused by rate limiting ("often indicated by a `ResourceExhausted` status code"). But that metric is counted on the client side and aggregates all causes; confirm via the server-side throttle metric above before concluding the limiter is the root cause. +- **Per-Workflow concurrency caps.** A single Workflow Execution hitting the 2000 incomplete Activities / Signals / Child Workflows / external Workflow Cancellation requests limit fails the Command at the programming-model level; it is not a Namespace-level rate limit and not diagnosed via this file. + +Per the managing-APS guide: "In Temporal Cloud, the effect of rate limiting is increased latency, not lost work. Workers might take longer to complete Workflows." Combined with the Cloud limits note about failures if throttling persists beyond the SDK's retry budget, the distinction matters: short bursts of `RESOURCE_EXHAUSTED` are normal and self-recovering; sustained throttling plus application-visible failures is what warrants capacity action. + +## Quick routing + +| Symptom | Go to | +|---|---| +| gRPC `RESOURCE_EXHAUSTED` on a client call to Cloud | [Temporal Cloud: APS, RPS, OPS and capacity modes](#temporal-cloud-aps-rps-ops-and-capacity-modes) and [Identifying which limit was hit](#identifying-which-limit-was-hit) | +| gRPC `RESOURCE_EXHAUSTED` from a self-hosted frontend | [Self-hosted: service-level RPS via dynamic configuration](#self-hosted-service-level-rps-via-dynamic-configuration) | +| `temporal_cloud_v0_resource_exhausted_error_count` / `temporal_cloud_v1_*_throttled_count` non-zero | Identify the specific budget via the metric triples in [Identifying which limit was hit](#identifying-which-limit-was-hit) | +| Activity / Workflow Task `schedule_to_start_latency` rising but no `RESOURCE_EXHAUSTED` | [worker-health.md](worker-health.md) | +| Workflow stuck with pending activities / children / signals | [workflow-stuck.md](workflow-stuck.md) | +| `context deadline exceeded` on caller | [runtime-errors.md](runtime-errors.md) | +| `UNAUTHENTICATED` / `PERMISSION_DENIED` | [authentication.md](authentication.md) | +| `UNAVAILABLE` with no rate-limit context | [connectivity.md](connectivity.md) and [certificates.md](certificates.md) | + +See the whole-stack picture in [diagnostic-ladder.md](diagnostic-ladder.md). For concrete commands to pull these metrics and confirm a throttling hypothesis, see [recipes.md](recipes.md). diff --git a/plugins/temporal/skills/temporal-ops/references/triage/recipes.md b/plugins/temporal/skills/temporal-ops/references/triage/recipes.md new file mode 100644 index 0000000..84e5123 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/triage/recipes.md @@ -0,0 +1,272 @@ +# Recipes + +End-to-end triage walkthroughs. Each recipe stitches sibling files into a single narrative for a common incident shape. Commands below are copied forward from the sibling files with pointers back to where the flag set and citations live; this file is not independent ground truth for command syntax. + +Confidence checkpoints follow the skill convention in [runtime-errors.md](runtime-errors.md#why-these-errors-are-hard): label a proposed root cause low/medium/high based on corroborating signals, not feel. + +## Table of Contents + +- [Stuck workflow at 3am](#stuck-workflow-at-3am) +- [Cert expired, workers offline](#cert-expired-workers-offline) +- [Task-queue backlog mystery](#task-queue-backlog-mystery) +- [Non-determinism caught in prod](#non-determinism-caught-in-prod) + +## Stuck workflow at 3am + +**Page text:** "Workflow `order-1234` is stuck, status Running, no progress for 2 hours." + +**Goal:** classify the stuck-ness shape (pending operation vs. WFT-failure loop vs. timer) and apply the matching recovery. + +**Steps:** + +1. **Confirm the status is Open.** Run the primary inspection command — see [workflow-stuck.md → The primary inspection command: `temporal workflow describe`](workflow-stuck.md#the-primary-inspection-command-temporal-workflow-describe) for the full flag table: + + ```bash + temporal workflow describe \ + --workflow-id order-1234 \ + --namespace + ``` + + If `workflowExecutionInfo.status` is anything other than `Running`, the workflow is Closed and this recipe does not apply — see [workflow-stuck.md → Workflow Execution Status values](workflow-stuck.md#workflow-execution-status-values). + +2. **Rule out the worker layer.** Most 3am "stuck workflow" pages are really worker outages. Check pollers on the Workflow's Task Queue — full discrimination in [worker-health.md → Inspecting a Task Queue with `temporal task-queue describe`](worker-health.md#inspecting-a-task-queue-with-temporal-task-queue-describe): + + ```bash + temporal task-queue describe \ + --task-queue \ + --namespace + ``` + + For deeper worker-level insights, use `temporal worker describe` to see individual worker status, build IDs, and deployment info: + + ```bash + temporal worker describe \ + --task-queue \ + --namespace + ``` + + If no recent pollers (`LastAccessTime` aged out after 5 minutes per [worker-health.md → What "no pollers" looks like](worker-health.md#what-no-pollers-looks-like)), stop here and route to worker-health.md. If `describe` itself fails, back off to [connectivity.md](connectivity.md), [certificates.md](certificates.md), [authentication.md](authentication.md), or [rate-limits.md](rate-limits.md). + +3. **Read the pending sections on `describe`.** Per [workflow-stuck.md → What to look at first](workflow-stuck.md#the-primary-inspection-command-temporal-workflow-describe), the partitioning checks after status are: which pending sections are present (`Pending Activities`, `Pending Child Workflows`, `Pending Nexus Operations`, `pendingWorkflowTask`), and whether `historyLength` is climbing between successive describes. + +4. **Export the Event History if you need the last meaningful event.** Command form from [workflow-stuck.md → Inspecting the Event History: `temporal workflow show`](workflow-stuck.md#inspecting-the-event-history-temporal-workflow-show): + + ```bash + temporal workflow show --workflow-id order-1234 --namespace --output json > history.json + ``` + + To locate the scheduling event with no matching terminal event, follow [workflow-stuck.md → Finding the last meaningful event](workflow-stuck.md#inspecting-the-event-history-temporal-workflow-show). That section warns that `ActivityTaskStarted` is only written *with* its terminal event — so a gap between `ActivityTaskScheduled` and a terminal event does not imply the Activity is unpicked. Read the Pending Activities block on `describe` for retry state. + +5. **Classify and route.** Map the evidence to the owning section: + + | Evidence on `describe` / history | Route | + |---|---| + | Pending Activity, attempts climbing, `LastAttemptFailure` present | [workflow-stuck.md → Pending activities](workflow-stuck.md#pending-activities) | + | Pending Activity, no attempts yet, only `ActivityTaskScheduled` in history | [worker-health.md → What "no pollers" looks like](worker-health.md#what-no-pollers-looks-like) (back to step 2) | + | Pending Child Workflow | [workflow-stuck.md → Pending child workflows](workflow-stuck.md#pending-child-workflows) — recurse into the child | + | Waiting on a Signal that never arrives | [workflow-stuck.md → Pending signals, cancellations, and updates](workflow-stuck.md#pending-signals-cancellations-and-updates) | + | Pending Nexus Operation, `State: Blocked` or `BackingOff` | [workflow-stuck.md → Pending Nexus Operations](workflow-stuck.md#pending-nexus-operations) | + | `pendingWorkflowTask.attempt` > 1 and `WorkflowTaskFailed` events accumulating | [workflow-stuck.md → Pending Workflow Task and WorkflowTaskFailed loops](workflow-stuck.md#pending-workflow-task-and-workflowtaskfailed-loops); if cause is Nondeterminism, [non-determinism.md](non-determinism.md) | + | Last non-bookkeeping event is `TimerStarted` with a future fire time | [workflow-stuck.md → Timer-based waits](workflow-stuck.md#timer-based-waits) — not stuck | + +6. **Apply recovery.** Use the commands in [workflow-stuck.md → Recovery commands](workflow-stuck.md#recovery-commands): + - Waiting on a Signal that was never sent: `temporal workflow signal` with the correct name and Workflow ID. + - Long-running Activity wedged without heartbeat: `temporal activity pause` / `unpause` / `reset`. + - Reset past a bad deploy: `temporal workflow reset --event-id ` (valid reset points are `WorkflowTaskStarted`, `WorkflowTaskCompleted`, `WorkflowTaskTimedOut`, `WorkflowTaskFailed`). + - Unrecoverable: `temporal workflow terminate --reason `. + +7. **Verify.** Re-run step 1; new events should be arriving, or the Workflow should be Closed as intended. + +**Confidence:** high when the pending-section evidence and the last meaningful event agree on a single route. Drop confidence when step 2 passes (fresh pollers) but the history still shows only `ActivityTaskScheduled` with no terminal event and no retry attempts — per [workflow-stuck.md → Pending activities](workflow-stuck.md#pending-activities), that mismatch hints at a Task Queue name / Build ID routing issue rather than a missing-pollers outage. (Confidence framing is a skill convention; see [runtime-errors.md](runtime-errors.md#why-these-errors-are-hard).) + +## Cert expired, workers offline + +**Page text:** "Workers across the fleet disconnected overnight. Logs repeat `x509: certificate has expired or is not yet valid`." + +**Goal:** rotate to a valid cert without cutting live traffic; remove the expired trust material only after the new path is verified. + +**Steps:** + +1. **Confirm the failure is TLS-layer cert expiry.** Reproduce the handshake from an affected host using the canonical `openssl s_client` form in [certificates.md → openssl recipes → Test a live endpoint](certificates.md#openssl-recipes): + + ```bash + openssl s_client -connect ..tmprl.cloud:7233 \ + -servername ..tmprl.cloud \ + -showcerts -cert client.pem -key client.key \ + -tls1_2 \ + --validity-period 364d \ + --ca-certificate-file ca.pem --ca-key-file ca.key \ + --certificate-file new-certs/client.pem --key-file new-certs/client.key + # Command and modifiers: + ``` + + Distribute `new-certs/client.pem` and `new-certs/client.key` to workers via the existing secret-distribution path. Restart workers. Skip to step 6. + +4. **CA + leaf rotation (CA expired or rolling over).** Use the zero-downtime pattern documented in [certificates.md → Rotation and expiry notifications](certificates.md#rotation-and-expiry-notifications): + + ```bash + # New CA — default is ECDSA P-384; add --rsa to switch to RSA 4096 + tcld generate-certificates certificate-authority-certificate \ + --organization \ + --validity-period 1y \ + --ca-certificate-file new-ca/ca.pem --ca-key-file new-ca/ca.key + # Command and modifiers: + + # Add the new CA to the Namespace *before* removing the old one. + # Full flag set in certificates.md → Accepted client CA set. + tcld namespace accepted-client-ca add \ + --namespace . \ + --ca-certificate-file new-ca/ca.pem + + # New leaf signed by the new CA + tcld generate-certificates end-entity-certificate \ + --organization \ + --validity-period 364d \ + --ca-certificate-file new-ca/ca.pem --ca-key-file new-ca/ca.key \ + --certificate-file new-certs/client.pem --key-file new-certs/client.key + ``` + + For a bundle-based rollover (concat old + new, then later new-only) use `tcld namespace accepted-client-ca set` as documented in [certificates.md → Accepted client CA set (mTLS Cloud)](certificates.md#accepted-client-ca-set-mtls-cloud). + +5. **Verify the chain locally before distribution** — see [certificates.md → openssl recipes → Verify a chain](certificates.md#openssl-recipes): + + ```bash + openssl verify -CAfile new-ca/ca.pem new-certs/client.pem # expect: OK + ``` + + If this fails locally it will fail at the peer. Do not distribute. + +6. **Distribute and confirm workers reconnect.** Push new leaf (and new CA if rotated) to the worker secret store and restart. Re-probe a critical queue per [worker-health.md → Inspecting a Task Queue with `temporal task-queue describe`](worker-health.md#inspecting-a-task-queue-with-temporal-task-queue-describe): + + ```bash + temporal task-queue describe \ + --task-queue \ + --namespace . + ``` + + Fresh pollers with recent `LastAccessTime` values mean the new credentials are accepted — see the interpretation matrix in that section. + +7. **Remove the old CA only after everything is green.** List first, then remove by fingerprint (safer than by PEM); `--fp` is the alias for `--ca-certificate-fingerprint`: + + ```bash + tcld namespace accepted-client-ca list --namespace . + # Command: + + tcld namespace accepted-client-ca remove \ + --namespace . \ + --fp + # Command: + # --ca-certificate-fingerprint / --fp: + ``` + + Get the fingerprint with `openssl x509 -in old-ca.pem -noout -fingerprint` (see [certificates.md → openssl recipes](certificates.md#openssl-recipes)). + +8. **Post-incident.** Cloud sends "Certificate Expiring in 15 days" notifications per [certificates.md → Rotation and expiry notifications](certificates.md#rotation-and-expiry-notifications); verify recipients are current and add an internal `openssl x509 -enddate` cron against the certs in use. + +**Confidence:** high once step 1 prints an expiry matching the incident, step 5 returns `OK`, and step 6 shows fresh pollers. If pollers are still stale after redeploy, peel the wrapped cause: `UNAVAILABLE` with a `tls:` cause is still layer 3; `UNAUTHENTICATED` after a clean handshake is a post-TLS rejection (certificate filter or role) — route to [authentication.md → mTLS authentication after TLS completes](authentication.md#mtls-authentication-after-tls-completes). + +## Task-queue backlog mystery + +**Page text:** "Task queue `payments-v2` has a growing backlog. Workers report as up." + +**Goal:** decide whether the backlog is a worker-polling problem, a versioning / Build ID routing problem, an auth or rate-limit regression, or a genuine capacity shortfall. + +**Steps:** + +1. **Read the Task Queue statistics and poller list** — one call, interpretation matrix in [worker-health.md → Inspecting a Task Queue](worker-health.md#inspecting-a-task-queue-with-temporal-task-queue-describe): + + ```bash + temporal task-queue describe --task-queue payments-v2 --namespace + ``` + + The statistics block (`ApproximateBacklogCount`, `ApproximateBacklogAge`, `TasksAddRate`, `TasksDispatchRate`, `BacklogIncreaseRate`) quantifies how fast the queue is growing; the pollers block covers no pollers, stale `LastAccessTime`, and identity mismatches. + +2. **No pollers** → [worker-health.md → What "no pollers" looks like](worker-health.md#what-no-pollers-looks-like). Recipe ends. + +3. **If pollers look fine, check Build ID / versioning routing.** Re-run `describe` with versioning flags per [worker-health.md → Reachability and versioning](worker-health.md#inspecting-a-task-queue-with-temporal-task-queue-describe): + + ```bash + temporal task-queue describe --task-queue payments-v2 --namespace \ + --select-build-id "" --report-reachability + ``` + + If the Build ID Workflows route to is not listed as reachable (e.g. a decommissioned cohort), tasks are being sent to a version that no longer has Workers. worker-health.md also covers the Worker-Deployments alternative `temporal worker deployment describe-version`. + +4. **Scan worker logs for layer regressions.** Routing table in [worker-health.md → Worker log signatures](worker-health.md#worker-log-signatures): + + | Worker log shape | Route | + |---|---| + | gRPC `UNAUTHENTICATED` / `PERMISSION_DENIED` | [authentication.md](authentication.md) | + | gRPC `RESOURCE_EXHAUSTED` | [rate-limits.md → Identifying which limit was hit](rate-limits.md#identifying-which-limit-was-hit) | + | `x509:` / `tls:` | [certificates.md](certificates.md) | + | Repeating `WorkflowTaskFailed` on one Workflow | Poison task — see step 5 | + +5. **Check for a poison task.** If the same Workflow keeps failing its WFT, read its history per [workflow-stuck.md → Pending Workflow Task and WorkflowTaskFailed loops](workflow-stuck.md#pending-workflow-task-and-workflowtaskfailed-loops). If the cause is Nondeterminism, escalate to the non-determinism recipe below. + +6. **Pollers fresh, no auth/rate-limit/poison signals, schedule-to-start latency high** → the ceiling is the Worker, not the server. See [worker-health.md → Schedule-to-start latency](worker-health.md#schedule-to-start-latency) and [→ Worker task slots](worker-health.md#worker-task-slots). Sizing / tuning lives in `skill-temporal-deploy`; this skill has done its job by proving the earlier layers healthy. + +**Confidence:** high if one signal from steps 3–5 fires and matches the backlog's start time. Low if steps 1–5 are all clean — that pattern is almost always "not enough workers," and worker-health.md flags it: low Poll Success Rate + low schedule-to-start latency + low host utilization can even indicate *too many* Workers (see [worker-health.md → Cloud Namespace-level poller limits](worker-health.md#cloud-namespace-level-poller-limits)). + +## Non-determinism caught in prod + +**Page text:** "Workers log repeated non-determinism errors. Workflow `reconcile-9999` has many `WorkflowTaskFailed` events in a row." + +**Goal:** confirm the WFT-failure cause is Nondeterminism, reproduce locally against the right source commit, and pick a remediation path. + +**Steps:** + +1. **Confirm the cause is Nondeterminism.** Export the history with the form from [workflow-stuck.md → Inspecting the Event History](workflow-stuck.md#inspecting-the-event-history-temporal-workflow-show), then follow [non-determinism.md → Identifying ND from the Event History](non-determinism.md#identifying-nd-from-the-event-history) to inspect each `WorkflowTaskFailed` event's `workflowTaskFailedEventAttributes.cause`: + + ```bash + temporal workflow show --workflow-id reconcile-9999 --namespace --output json > history.json + ``` + + The cross-SDK cause value is "Nondeterminism" on the `WorkflowTaskFailedCause` enum. If the cause is something else, route via [workflow-stuck.md → Pending Workflow Task and WorkflowTaskFailed loops](workflow-stuck.md#pending-workflow-task-and-workflowtaskfailed-loops). + +2. **Capture the worker error text.** Per [non-determinism.md → Per-SDK error shape](non-determinism.md#per-sdk-error-shape), only TypeScript has a doc-pinned class (`DeterminismViolationError`); other SDKs emit per-SDK errors. The message usually names the offending Command vs. expected Event — that's what pins the divergence. + +3. **Reproduce locally against the *deployed* commit** (not `main`) — per [replay.md → Prerequisites](replay.md#prerequisites), replaying newer source against an older recording can produce divergence for a different reason than the bug being triaged. + + - **Interactive (TypeScript only):** VS Code extension — [replay.md → The VS Code extension](replay.md#the-vs-code-extension-typescript-only-interactive). Point it at `history.json`. + - **Headless / CI (any supported SDK):** SDK replayer — [replay.md → Step 2](replay.md#step-2--run-the-sdk-replayer-all-supported-sdks). For Go/Java, set `TEMPORAL_DEBUG=true` while stepping ([replay.md → TEMPORAL_DEBUG](replay.md#temporal_debug-suppress-the-deadlock-detector-while-stepping)). + + Interpret per [replay.md → Interpreting a replay that diverges](replay.md#interpreting-a-replay-that-diverges) / [→ succeeds](replay.md#interpreting-a-replay-that-succeeds). Replay against the deployed commit fails with the same error → diagnosis stands. Replay succeeds → deployed Workers are on different code; find them before fixing. + +4. **Identify the change.** Diff the deployed commit against the previous known-working commit. Canonical divergence shapes are listed in [non-determinism.md → What the docs call out as ND-inducing patterns](non-determinism.md#what-the-docs-call-out-as-nd-inducing-patterns) — command-order changes, Activity name changes, Timer-duration changes to/from zero (per-SDK), and intrinsic ND (random branches, map iteration order, wall-clock reads). + +5. **Pick a remediation path** from [non-determinism.md](non-determinism.md): + + | Situation | Remediation | + |---|---| + | Worker Versioning already in place and this Workflow Type can be pinned going forward | [→ Worker Versioning (preferred)](non-determinism.md#remediation-worker-versioning-preferred). Adopting Versioning mid-incident does not fix already-looping Workflows. | + | In-flight Workflows must complete under both old and new behavior | [→ per-SDK patching](non-determinism.md#remediation-per-sdk-patching) — `GetVersion` / `patched` branch. SDK API details in `skill-temporal-developer`. | + | Bad commit cleanly revertible, divergences not yet widespread | [→ Fix the Worker code and redeploy](non-determinism.md#remediation-fix-and-redeploy-or-reset-past-the-divergence) — revert, let the server-side WFT-retry loop succeed on new Workers. | + | Workflows already wedged past the divergence | [→ Reset the Workflow past the divergence](non-determinism.md#remediation-fix-and-redeploy-or-reset-past-the-divergence) — `temporal workflow reset --event-id `. Valid reset points: `WorkflowTaskStarted`, `WorkflowTaskCompleted`, `WorkflowTaskTimedOut`, `WorkflowTaskFailed`. Confirm with the business owner — events after the reset point are re-executed. | + +6. **Verify.** Local replay of the failing history should now succeed. After deploy, re-run `temporal workflow describe`; `pendingWorkflowTask.attempt` should stop climbing and new non-bookkeeping events should arrive. + +7. **Post-incident: add a replay regression test.** Put each production history into a bulk replayer in CI (`WorkflowReplayer.replayWorkflowExecutions` / `Worker.runReplayHistories` / `Replayer.replay_workflows`) per [replay.md → Step 2](replay.md#step-2--run-the-sdk-replayer-all-supported-sdks), so the next regression fails CI. + +**Confidence checkpoints:** +- After step 1: high that it's ND *iff* the JSON inspection returns the Nondeterminism cause on at least one `WorkflowTaskFailed` event. Without that, do not prescribe patching or reset — the WFT is failing for a different reason. +- After step 3: if local replay reproduces against the deployed commit, high that the deployed code is the cause. If replay *succeeds* against that commit, the deployed Workers are on different code; find the real build before picking a remediation. + +See the whole-stack picture in [diagnostic-ladder.md](diagnostic-ladder.md). diff --git a/plugins/temporal/skills/temporal-ops/references/triage/replay.md b/plugins/temporal/skills/temporal-ops/references/triage/replay.md new file mode 100644 index 0000000..a0c53ca --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/triage/replay.md @@ -0,0 +1,182 @@ +# Replay a Workflow Execution locally + +This file is about the *tooling* that reproduces a recorded Workflow Execution in a debugger. For what replay divergence *means* as a concept — why the Worker Task fails, how the server classifies the cause, and how to remediate — see [non-determinism.md](non-determinism.md). + +A Replay is "the method by which a Workflow Execution resumes making progress. During a Replay the Commands that are generated are checked against an existing Event History." Running a recorded history through a local replayer against your Worker source tree replays the Workflow Execution "to replicate errors" — how you reproduce a non-determinism error under a debugger and, via the bulk replayer, pin a CI regression test. + +The **SDK replayer** is the general-purpose tool — documented for every supported SDK, it runs headless, attaches to any debugger, and drops into CI as a regression guard. The **VS Code extension** is a TypeScript-only convenience wrapper around the same replayer. + +Out of scope: + +- What non-determinism means, how to identify it in an Event History, and remediation options → [non-determinism.md](non-determinism.md) +- Worker not polling the Task Queue at all — nothing to replay, fix the Worker first → [worker-health.md](worker-health.md) +- Can't reach the server to fetch the history → [connectivity.md](connectivity.md), [authentication.md](authentication.md) +- The bottom-up layer model for routing between files → [diagnostic-ladder.md](diagnostic-ladder.md) + +## Table of Contents + +- [Prerequisites](#prerequisites) +- [Step 1 — Get the Event History](#step-1--get-the-event-history) +- [Step 2 — Run the SDK replayer (all supported SDKs)](#step-2--run-the-sdk-replayer-all-supported-sdks) +- [TEMPORAL_DEBUG: suppress the deadlock detector while stepping](#temporal_debug-suppress-the-deadlock-detector-while-stepping) +- [Interpreting a replay that diverges](#interpreting-a-replay-that-diverges) +- [Interpreting a replay that succeeds](#interpreting-a-replay-that-succeeds) +- [The VS Code extension (TypeScript only, interactive)](#the-vs-code-extension-typescript-only-interactive) + +## Prerequisites + +- The Workflow source tree at the commit that was deployed when the recorded Workflow Execution ran. Replaying current `main` against an older recording can produce divergence *for a different reason than the bug you're triaging*. +- The SDK installed and importable in that workspace (the replayer is part of the SDK, not a standalone binary). +- A client connection from that workspace to the Namespace holding the run, with read access to its history (see [authentication.md](authentication.md)) — or, failing that, a history file exported by someone who has it. + +## Step 1 — Get the Event History + +The replayer takes an Event History object, and every SDK client can fetch one. Fetch it in the same test that replays it. + +| SDK | Fetch from the server | +|---|---| +| Go | `c.GetWorkflowHistory(ctx, id, runID, false, enums.HISTORY_EVENT_FILTER_TYPE_ALL_EVENT)` returns an iterator; accumulate `iter.Next()` into a `history.History` | +| Python | `client.list_workflows().map_histories()` | +| TypeScript | `client.workflow.getHandle().fetchHistory()`; `client.workflow.list({ query }).intoHistories()` for many | +| Java | `service.blockingStub().getWorkflowExecutionHistory()`, wrapped as `new WorkflowExecutionHistory(response.getHistory(), )` | +| .NET | `client.ListWorkflowHistoriesAsync()` | +| Ruby | `client.list_workflows()` | +| PHP | `$replayer->replayFromServer(workflowType: ..., execution: ...)` fetches and replays in one call | + +The list-based fetches require Advanced Visibility on the server. + +Export a JSON file instead when the workspace has no client access to the Namespace, the history is being committed as a CI fixture, or the run has aged out of retention: + +```bash +temporal workflow show \ + --workflow-id YourWorkflowId \ + --run-id YourRunId \ + --output json > history.json +``` + +`--output` accepts `text, json, jsonl, none` (default `text`); replay requires `json`. `--run-id` is not required. + +If the run id is unknown, find it via `temporal workflow describe --workflow-id YourWorkflowId` (see [workflow-stuck.md](workflow-stuck.md)) or the Web UI. + +For Cloud or any non-default target, add the usual connection flags (`--address`, `--namespace`, and either `--api-key` or the mTLS `--tls-*` flags). See [authentication.md](authentication.md). + +## Step 2 — Run the SDK replayer (all supported SDKs) + +Each SDK's testing-suite page documents a replayer. Names and signatures below are transcribed from those pages; they are what the docs state, not what the runtime type system exports at any given version. The history argument is whatever step 1 produced — a fetched object or a parsed file. + +All of these run headless and the failure surfaces as a thrown error, which is enough to identify the divergence. A debugger is only needed to step through Workflow code; recommend that to the user rather than treating it as the default path. + +### Go + +```go +replayer := worker.NewWorkflowReplayer() +replayer.RegisterWorkflow(YourWorkflow) +err := replayer.ReplayWorkflowHistory(nil, hist) +``` + + Use `worker.WorkflowReplayer` to "replay an existing Workflow Execution from its Event History to replicate errors." "If a noticeably different code path was followed or some code caused a deadlock, it will be returned in the error code." + +To step through the Workflow function, attach Delve or an IDE debugger to the test process that calls `ReplayWorkflowHistory`, with `TEMPORAL_DEBUG=true` (see below). + +### Python + +```python +replayer = Replayer(workflows=[YourWorkflow]) +await replayer.replay_workflows(histories) +``` + + `replay_workflows` takes the iterator from step 1; if any replay fails, it raises. Set `fail_fast` to `false` to replay every history before reporting. From a file, the single-history form is `replayer.replay_workflow(WorkflowHistory.from_json(history_json_str))`. Histories fetched from the server or exported can be protobuf-encoded (`bytes`) while the `Replayer` works with decoded histories (like a `dict`); a `dict`-vs-`bytes` `TypeError` during replay means the history needs decoding first. + +### TypeScript + +```ts +const history = JSON.parse(await fs.promises.readFile('./history.json', 'utf8')); +await Worker.runReplayHistory( + { workflowsPath: require.resolve('./your/workflows') }, + history, +); +``` + + For bulk replay, use `Worker.runReplayHistories`. + +Two error classes are documented: "When an Event History is replayed and non-determinism is detected (that is, the Workflow code is incompatible with the History), `DeterminismViolationError` is thrown. If replay fails for any other reason, `ReplayError` is thrown." + +### Java + +```java +File file = new File("history.json"); +WorkflowReplayer.replayWorkflowExecution(file, MyWorkflow.class); +``` + + Use `WorkflowReplayer` from the `temporal-testing` package. For bulk replay, `WorkflowReplayer.replayWorkflowExecutions`. "In both examples, if Event History is non-deterministic, an error is thrown. You can choose to wait until all histories have been replayed with `replayWorkflowExecutions` by setting the `failFast` argument to `false`." + +### .NET + +```csharp +var replayer = new WorkflowReplayer( + new WorkflowReplayerOptions().AddWorkflow()); +await replayer.ReplayWorkflowAsync( + WorkflowHistory.FromJson("my-workflow-id", historyJson)); +``` + + For bulk replay, iterate `replayer.ReplayWorkflowsAsync(...)` and check each `result.ReplayFailure`. + +### Ruby + +```ruby +replayer = Temporalio::Worker::WorkflowReplayer.new(workflows: [MyWorkflow]) +replayer.replay_workflow(history) +``` + + For bulk replay, pass `client.list_workflows(...)` to `replayer.replay_workflows(...)`; set `raise_on_replay_failure: true`, or inspect each `result.replay_failure`. + +### PHP + +The replayer is `\Temporal\Testing\Replay\WorkflowReplayer`. Replay from a running server with `replayFromServer(...)`, from an exported JSON file with `replayFromJSON(...)`, or from an in-memory history with `replayHistory($history)`. A non-deterministic replay throws `\Temporal\Testing\Replay\Exception\ReplayerException`. + +## TEMPORAL_DEBUG: suppress the deadlock detector while stepping + +Two SDKs explicitly document a debug-mode env var. Without it, pausing on a breakpoint for more than a second can cause the Worker's deadlock detector to fail the Workflow Task *during your debugging session*: + +- **Go.** "The Temporal Go SDK includes deadlock detection which fails a Workflow Task in case the code blocks over a second without relinquishing execution control. Because of this you can often encounter a `PanicError: Potential deadlock detected` while stepping through Workflow Definitions during debugging. To alleviate this issue, you can set the `TEMPORAL_DEBUG` environment variable to `true` before debugging your Workflow Definition." +- **Java.** "The Temporal Java SDK includes deadlock detection which fails a Workflow Task in case the code blocks over a second without relinquishing execution control. Because of this you can often encounter the `PotentialDeadlockException` Exception while stepping through Workflow code during debugging. To alleviate this issue, you can set the `TEMPORAL_DEBUG` environment variable to true before debugging your Workflow code." + +Both pages add the same warning: "Make sure to set `TEMPORAL_DEBUG` to true only during debugging." + +Python and TypeScript debugging pages do not document a `TEMPORAL_DEBUG` env var in the sources consulted. + +## Interpreting a replay that diverges + +The documented behavior when replay detects non-determinism: + +- **TypeScript**: throws `DeterminismViolationError`; any other replay failure throws `ReplayError`. +- **Go**: the replayer returns an error from `ReplayWorkflowHistory`; the docs describe the condition as "cause the Workflow to fail with a nondeterminism error" without pinning a public type name. +- **Java**: `WorkflowReplayer.replayWorkflowExecution` throws; the versioning doc describes the condition as "This would cause the Workflow to fail with a nondeterminism error." +- **Python**: `Replayer.replay_workflow` raises; if any replay fails, the code raises an exception. + +The error is raised where the SDK detected the mismatch — typically inside *the Worker machinery*, not on the Workflow line that emitted the bad Command. A debugger halts at the same place. To locate the offending Workflow line, compare: + +- The last Command the code was about to emit (the frame just below the SDK entry in the stack), and +- The next non-bookkeeping Event in the recorded history (see [non-determinism.md §Identifying ND from the Event History](non-determinism.md#identifying-nd-from-the-event-history)). + +The divergence is the mismatch between those two. The encyclopedia frames this as: "If a generated Command doesn't match what it needs to in the existing Event History, then the Workflow Execution returns a non-deterministic error." + +For remediation paths (Worker Versioning, per-SDK patching, reset past the divergence), return to [non-determinism.md §Remediation](non-determinism.md#remediation-worker-versioning-preferred). + +## Interpreting a replay that succeeds + +If replay succeeds locally against the checked-out source, but production Workers keep failing with the same Workflow Execution, the deployed Worker code almost certainly differs from the local checkout. Find the deployed build (via Worker Versioning metadata if used, or your deploy system) and reproduce from that commit. See [non-determinism.md §Reproducing ND locally via replay](non-determinism.md#reproducing-nd-locally-via-replay). + +A local-replay success on the *current* source with production still failing is also the classic case Worker Versioning is meant to prevent. See [non-determinism.md §Remediation: Worker Versioning (preferred)](non-determinism.md#remediation-worker-versioning-preferred). + +## The VS Code extension (TypeScript only, interactive) + +Temporal publishes a VS Code extension that wraps the TypeScript SDK's replayer in the VS Code debugger UI — one-click "open history → set breakpoints → step through." For Workflows authored in any other SDK, skip it and use the SDK replayer above under your IDE's native debugger; the observability is the same (a stack trace at the point the SDK detected divergence) and the replayer APIs are first-party and doc-backed. + +Claims in this section come from the extension's marketplace listing; the Temporal docs themselves do not document the extension. + +- **Install:** search the VS Code Marketplace for "Temporal" and install the extension published by **Temporal Technologies Inc.** (`temporal-technologies.temporalio`). As of the listing consulted, it debugs **TypeScript workflows only**. +- **Configure:** the replayer is driven by a TypeScript entrypoint that calls `startDebugReplayer` with a `workflowsPath`; the extension reads the path from the `temporal.replayerEntrypoint` setting (default `src/debug-replayer.ts`). +- **Run:** **Temporal: Open Panel** from the Command Palette → enter a Workflow ID (fetched from the configured server, default `localhost:7233`) or pick a history JSON file → **Start** → set breakpoints in Workflow source or on history events, then step through. For Cloud, supply the Cloud gRPC endpoint plus the credentials in [authentication.md](authentication.md) and [certificates.md](certificates.md). + +For anything the marketplace page does not spell out (a supported-SDK matrix beyond TypeScript, the full command list, `launch.json` recipes, OS requirements, mTLS-vs-API-key support), consult the extension's README for the version you installed — the citations here are a point-in-time snapshot. diff --git a/plugins/temporal/skills/temporal-ops/references/triage/runtime-errors.md b/plugins/temporal/skills/temporal-ops/references/triage/runtime-errors.md new file mode 100644 index 0000000..2296ca4 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/triage/runtime-errors.md @@ -0,0 +1,149 @@ +# Runtime Errors + +Disambiguate Temporal errors whose text does not by itself identify the layer at fault. Primary focus: `context deadline exceeded` (`DEADLINE_EXCEEDED`) and Workflow lock contention (`BusyWorkflow`). Secondary: where to route `no pollers`, `INVALID_ARGUMENT`, and unspecified `UNAVAILABLE` when the layer isn't obvious from the message alone. + +Unambiguous errors are covered in the layer-specific files — link, don't duplicate: +- DNS / TCP / endpoint / PrivateLink → [connectivity.md](connectivity.md) +- TLS / x509 / mTLS alerts → [certificates.md](certificates.md) +- `UNAUTHENTICATED` / `PERMISSION_DENIED` → [authentication.md](authentication.md) +- `RESOURCE_EXHAUSTED` rate-limit anatomy → [rate-limits.md](rate-limits.md) +- Pollers / schedule-to-start / sticky cache → [worker-health.md](worker-health.md) +- Pending activities / children / signals / Workflow Task failures → [workflow-stuck.md](workflow-stuck.md) +- Blob-size / history-size limits → `docs/troubleshooting/blob-size-limit-error.mdx` (not ambiguous; the error names itself) +- Performance-bottlenecks deep dive → `docs/troubleshooting/performance-bottlenecks.mdx` + +## Table of Contents + +- [Why these errors are hard](#why-these-errors-are-hard) +- [Deadline exceeded](#deadline-exceeded) +- [Workflow lock contention (BusyWorkflow)](#workflow-lock-contention-busyworkflow) +- [Other frequently ambiguous errors](#other-frequently-ambiguous-errors) +- [Triage protocol](#triage-protocol) + +## Why these errors are hard + +`context deadline exceeded` is emitted by the Go `context` package and propagates through gRPC as `DEADLINE_EXCEEDED`. It tells you only that the caller gave up waiting — not why the response did not arrive. The Temporal troubleshooting guide lists "network interruptions, timeouts, server overload, and Query errors" as causes in the same breath, and the fix catalog spans clock skew, Frontend Service reachability, rate-limit saturation, client/worker configuration, and connection-age tuning. + +`ResourceExhausted` (`RESOURCE_EXHAUSTED`) is ambiguous for a different reason: one gRPC code covers both throttling against your account limits and per-Workflow lock contention, and these are different problems with different fixes. The Cloud docs are explicit that lock contention "is contention on a single Execution, not an account limit. Increasing your Actions, Requests, or Operations per second limits does not resolve it." The discriminator is the metric breakdown, not the error text. + +Treating either error as a single-layer failure is the most common triage mistake in this category. Identify the operation and the layer before prescribing a fix. Give the proposed root cause an explicit confidence label — "low confidence, next discriminating check is X" beats a guess dressed up as a diagnosis. (Confidence framing is a skill convention, not a Temporal contract.) + +## Deadline exceeded + +**Verbatim error shapes seen in the wild:** +- `Context: deadline exceeded` — surfaced by the Temporal troubleshooting guide. +- `rpc error: code = DeadlineExceeded desc = context deadline exceeded` — Go gRPC client. +- `Error: 4 DEADLINE_EXCEEDED: context deadline exceeded` — the `@grpc/grpc-js` TypeScript client form documented in the Temporal TS debugging guide. + +**What it means:** the caller's deadline fired before a response arrived. The code is `DEADLINE_EXCEEDED` regardless of which layer failed to respond. + +**Causes documented in the Temporal troubleshooting guide for `deadline-exceeded`:** + +- **Clock skew** between a Worker and the Temporal Service exceeding an Activity's Start-To-Close Timeout — produces `Activity complete after timeout` alongside `Context: deadline exceeded`. Resolution: sync to NTP. +- **Frontend Service not reachable.** OSS users can check with `temporal operator cluster health --address 127.0.0.1:7233`; `grpc-health-probe` lets you probe Frontend, Matching, and History individually. Cloud users cannot access these logs directly — the guide directs them to open a support ticket with Namespace Name and sample Workflow IDs. +- **Resource-exhausted backpressure masquerading as deadline.** "A `resource exhausted` error can cause your client request to fail, which prompts the `deadline exceeded` error." Discriminator query (self-hosted metrics): `sum(rate(service_errors_resource_exhausted{}[1m])) by (resource_exhausted_cause)`, watching for `RpsLimit`, `ConcurrentLimit`, `SystemOverloaded`. For the Cloud equivalent and the `RESOURCE_EXHAUSTED` anatomy see [rate-limits.md](rate-limits.md#identifying-which-limit-was-hit). +- **Invalid client/worker configuration** (wrong server name, address, or certificate). These produce `connection refused` alongside `deadline exceeded`; the guide explicitly pairs the two. Re-run the [connectivity ladder](connectivity.md) before anything else. +- **Service just restarted / roles not yet initialized.** Wait and retry; review Workflow Execution history and server logs if it persists. +- **Self-hosted: `frontend.keepAliveMaxConnectionAge` too short** for in-flight requests — increase it and monitor server load. + +**TypeScript SDK guide adds two concrete triggers for a `context deadline exceeded` at call time:** + +- Network hiccup, timeout that's too short, or overloaded server. +- "Querying a Workflow Execution whose query handler causes an error can result in the query call timing out." + +**Nexus-specific:** "If a Nexus handler doesn't process a start or cancel request within 10 seconds, it will receive a context deadline exceeded error, and the caller will retry, with an exponential backoff, for the ScheduleToClose duration for the overall Nexus Operation." + +### Discriminating by where the call was made + +**First question to ask the reporter:** which operation emitted this — workflow start, signal/update, query, a worker poll, a Nexus start/cancel, or an un-attributed log line? The answer narrows the layer. + +| Operation | Likely-first check | +|---|---| +| Workflow start / signal / update / describe (RPC from a client) | Confirm DNS / TCP / TLS / auth succeed for the same endpoint from the failing environment. Start at [connectivity.md → Quick diagnostic scripts](connectivity.md#quick-diagnostic-scripts). If layers 1–4 pass, move to Frontend health and the `resource_exhausted_cause` metric. | +| Workflow start with a large input | Rule out `BlobSizeLimitError` (2 MB per individual payload, 4 MB per gRPC request). That error is self-naming, not a bare `deadline exceeded`, but large payloads also inflate request latency (`temporal_request_latency`) per the bottlenecks guide. | +| Query | The Workflow's Query handler may itself be erroring out. Queries run in the Worker and the call is synchronous. Check Worker logs for an exception raised inside the handler; fix and redeploy. | +| Worker long-poll (`PollWorkflowTaskQueue`, `PollActivityTaskQueue`) | `temporal_long_request_failure` is counted against these poll RPCs; the bottlenecks guide lists network issues, rate limiting (often indicated by `ResourceExhausted`), and server errors as the three cause classes. Jump to [worker-health.md → Worker log signatures](worker-health.md#worker-log-signatures). | +| Nexus handler call | 10-second handler cap; see `docs/evaluate/temporal-cloud/limits.mdx` for Nexus timeout semantics. | + +### PrivateLink-specific deadline exceeded + +`context deadline exceeded` from a Worker or CLI going through AWS PrivateLink or GCP Private Service Connect has a common set of layer-1–3 causes. The commands below are the ones already grounded in the sibling files — reuse them rather than re-deriving. + +1. **VPC-endpoint path unreachable from the client subnet.** + ```bash + nc -zvw10 vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com 7233 + ``` + If this times out, the VPC-endpoint security group is not permitting TCP/7233 from the client subnet. The exact probe command form is the one used in the Cloud connectivity guide. See [connectivity.md → PrivateLink and PSC](connectivity.md#privatelink-and-psc). +2. **TLS handshake fails because SNI is not overridden.** When connecting by the VPC-endpoint DNS name (i.e. not via private DNS), the client must set the TLS server name to the Namespace Endpoint — `..tmprl.cloud`. The Cloud connectivity guide gives the exact env-var form: `TEMPORAL_ADDRESS=vpce-...:7233` paired with `TEMPORAL_TLS_SERVER_NAME=my-namespace.my-account.tmprl.cloud`. Full details: [certificates.md → Server name override](certificates.md#server-name-override). +3. **Private DNS missing for the region the Namespace is currently in (HA Namespaces).** After a failover, the `region.tmprl.cloud` private hosted zone must cover every region the Namespace can fail over to. Details: [ha-failover.md → PrivateLink stopped working after failover](ha-failover.md#symptom-privatelink--psc-stopped-working-after-failover). +4. **PrivateLink not enabled on the Namespace.** Verify connectivity configuration on the Namespace; if the Namespace is not configured for PrivateLink, public DNS will route the caller somewhere the VPC cannot reach. + +## Workflow lock contention (BusyWorkflow) + +**Error shape:** every operation that mutates a single Workflow Execution — starting it, sending a Signal, and so on — is serialized under a per-Workflow lock. When operations reach one Execution faster than that lock can be acquired, the Service rejects the excess with a `ResourceExhausted` error (`RESOURCE_EXHAUSTED`). In Service logs it appears as `Workflow is busy.` The Cloud docs name the condition **Workflow lock contention (BusyWorkflow)**; use that term when explaining it, and treat a caller's "workflow is busy" report as pointing at it. + +The Cloud metric is `temporal_cloud_v1_resource_exhausted_error_count`, which increments when "a single resource (a Namespace, Task Queue, or Workflow ID) receives a burst of operations larger than that resource can absorb in the moment." Lock contention is the most common cause of resource exhaustion. + +**What this is not:** +- **Not an account limit.** "This is contention on a single Execution, not an account limit. Increasing your Actions, Requests, or Operations per second limits does not resolve it." Account-limit throttling is APS / RPS / OPS on Cloud or `frontend.rps` / `frontend.namespaceRPS` self-hosted — see [rate-limits.md](rate-limits.md). The two are distinct conditions that share a gRPC code. +- **Not a Workflow failure.** A `ResourceExhausted` on a signal/update does not fail the Workflow Execution; the SDK's default gRPC retry policy retries the RPC with backoff. +- **Not "the Workflow is blocked in a useful sense."** The Workflow may be perfectly healthy; the pressure is on the lock, from the caller's side. +- **Not always worth chasing.** "At low, brief rates this error is benign because clients retry it and no progress is lost." Investigate when the rate is sustained or correlates with rising latency on the affected operations. + +### Confirming lock contention + +The Cloud service-health guide's protocol: + +1. **Rule out account-limit throttling first.** If the throttle metrics are elevated, address that throttling before looking at lock contention — limits-driven throttling slows or stalls a workload, so it is the more important signal. See [rate-limits.md → Identifying which limit was hit](rate-limits.md#identifying-which-limit-was-hit). +2. **If you are within limits but `temporal_cloud_v1_resource_exhausted_error_count` is still non-zero, break it down by the `operation` label.** "Lock contention concentrates on operations that target individual executions." +3. **Match the operation to the guidance below.** + +**Mind the label, it differs by metric family.** The v1 Cloud metric carries only `operation`. The v0 metric carries `resource_exhausted_cause`. Self-hosted uses `resource_exhausted_cause` on `service_errors_resource_exhausted`. Whichever family the user is on, classify from the label, not the free-text message. See [rate-limits.md → From the error](rate-limits.md#from-the-error). + +### Per-operation guidance + +Mapped from the Cloud service-health table. + +| `operation` | What it indicates | What to do | +|---|---|---| +| `StartWorkflowExecution`, `SignalWithStartWorkflowExecution` | The same Workflow ID was started again inside the de-duplication window (about one second). The first start succeeded; the duplicate was rejected. | Usually safe to ignore. Don't retry aggressively. Look for a client path firing the duplicate start. | +| `SignalWorkflowExecution` | Signal rate to one Execution is too high. | Batch or coalesce Signals (one per N events), shard work across more Executions, or buffer Signals and drain them in the main Workflow loop. | +| `UpdateWorkflowExecution` | More than the per-execution in-flight Update limit (10) are outstanding. | Cap concurrent in-flight Updates client-side, then back off and retry. | +| `RecordActivityTaskHeartbeat` | Too many Activities heartbeating into the same Execution. | Raise the heartbeat timeout and interval; reduce how many Activities heartbeat into one Execution concurrently. | +| `RespondWorkflowTaskCompleted` | One Workflow schedules a large batch of Activities or Child Workflows in parallel, each taking the lock. | Keep concurrent operations at 500 or fewer per Execution. Process the batch in smaller groups (sliding-window or plain batching) instead of scheduling everything at once. | +| `QueryWorkflow` | Too many concurrent Queries against one Execution, or fallout from repeated Workflow Task retries. | Reduce concurrent Queries to that Execution. If it correlates with Workflow Task failures or timeouts, resolve those first. | + +The per-execution ceilings referenced above are in the Cloud limits page: 10 in-flight Updates per Execution, and 2,000 incomplete Activities / Signals / Child Workflows / external-cancellation requests, with 500 or fewer recommended for optimal performance. + +### Related signals + +- **Schedule-to-start latency rising alongside it.** "High Workflow lock latency. If many updates are made to a single execution, this can cause Workflow lock latency, which in turn affects the Schedule-to-start latency. Reduce the rate of Signals." This is the same hot-execution pressure seen from the latency side; see [performance-bottlenecks.md](performance-bottlenecks.md). +- **Is the caller retrying without backoff?** Retries count against the budget. A raw gRPC client reimplementing retry must use exponential backoff. +- **Pending state on the Execution.** `pendingWorkflowTask` in `temporal workflow describe` output shows in-flight Workflow Task state for the Execution. + +## Other frequently ambiguous errors + +### `no pollers` + +This phrase is a clue, not a diagnosis. Common shapes: no Worker reached the frontend for the queue+type within the last 5 minutes, Workers are polling a different queue or Namespace, or Workers connect but fail before the poll loop. Verify from `temporal task-queue describe`, not from cached metrics. Full protocol: [worker-health.md → What "no pollers" looks like](worker-health.md#what-no-pollers-looks-like). + +### `INVALID_ARGUMENT` + +`INVALID_ARGUMENT` is a catch-all. The suffix is the useful part: + +- `namespace not found` — the namespace string does not exist in this account. Confirm via `tcld namespace list` (see [connectivity.md → Endpoint formats](connectivity.md#endpoint-formats) for the correct Namespace Endpoint form, which is a common source of this shape). +- Field-specific validation errors — fix the input; the suffix names the bad field. + +### Unspecified `UNAVAILABLE` + +`UNAVAILABLE` on its own does not pin a layer. Peel the wrapped cause: `tls:` / `x509:` / `remote error: tls:` → [certificates.md](certificates.md); `connection refused` / `no such host` / `i/o timeout` → [connectivity.md](connectivity.md). If the wrapped cause is absent, run the [diagnostic ladder](diagnostic-ladder.md) from layer 1. + +## Triage protocol + +For any ambiguous runtime error: + +1. **Demand the exact text.** Copy-paste, not paraphrase. An `UNAVAILABLE` with `x509:` wrapped inside is a TLS problem, not a network one. +2. **Identify which operation produced it** — start / signal / update / query / poll / Nexus handler / internal. The operation narrows the candidate layers. +3. **Identify which environment produced it** — local dev, self-hosted, Cloud. Different error catalogs (e.g. Cloud's `resource_exhausted_cause` labels versus self-hosted `service_errors_resource_exhausted`). +4. **Walk the diagnostic ladder** up to the layer the evidence implicates. See [diagnostic-ladder.md](diagnostic-ladder.md). +5. **Attach confidence.** Below 6, the next action is a discriminating check (DNS lookup, `openssl s_client`, `temporal operator cluster health`), not a fix. Confidence framing is a skill-level triage norm, not a documented Temporal property — but it's how this skill avoids prescribing fixes on the strength of ambiguous evidence. diff --git a/plugins/temporal/skills/temporal-ops/references/triage/schedule-missed.md b/plugins/temporal/skills/temporal-ops/references/triage/schedule-missed.md new file mode 100644 index 0000000..6326c94 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/triage/schedule-missed.md @@ -0,0 +1,131 @@ +# Missed Schedule Actions + +When a Schedule does not start a Workflow Execution at its expected time, the Action was either skipped intentionally (paused, overlap policy, end time reached) or the Temporal Service could not take the Action within the Catchup Window. This guide covers the second case. + +--- + +## Alert on missed catchup window + +The Temporal Service emits a counter each time it skips a scheduled Action because it could not run it within the configured Catchup Window. Alert on any non-zero value. + +### Temporal Cloud + +Alert on `temporal_cloud_v1_schedule_missed_catchup_window_count` grouped by `temporal_namespace`. + +Example PromQL: + +```promql +sum by (temporal_namespace) ( + increase(temporal_cloud_v1_schedule_missed_catchup_window_count[5m]) +) > 0 +``` + +### Self-hosted + +Alert on `schedule_missed_catchup_window` grouped by `namespace`. + +Example PromQL: + +```promql +sum by (namespace) ( + increase(schedule_missed_catchup_window[5m]) +) > 0 +``` + +The metric is scoped to the Namespace, not to individual Schedules. A non-zero value tells you that at least one Schedule in the Namespace missed an Action, but not which one. + +--- + +## Investigate which Schedule missed an Action + +### Step 1: List Schedules in the Namespace + +```bash +temporal schedule list --namespace +``` + +`ListSchedules` returns Schedule IDs and summary information. It does not return per-Schedule miss counters; use it only to produce the set of Schedule IDs to inspect. + +### Step 2: Describe each Schedule + +```bash +temporal schedule describe \ + --schedule-id \ + --namespace +``` + +`DescribeSchedule` returns full Schedule state, including the `info` block with cumulative counters. + +Relevant fields: + +| Field | Meaning | +|---|---| +| `missedCatchupWindow` | Actions skipped because they could not run within the Catchup Window. Non-zero identifies the Schedule responsible for the alert. | +| `overlapSkipped` | Actions skipped because the previous run was still in progress and the Overlap Policy is `Skip`. | +| `bufferDropped` | Buffered Actions dropped because the buffer was full under `BufferOne` or `BufferAll`. | +| `bufferSize` | Current depth of the Action buffer. | +| `recentActions` | Most recent Action times and results. | +| `runningWorkflows` | Workflow Executions currently running for this Schedule. | + +Scripting the fan-out against the JSON output (`temporal schedule describe -o json`) is usually faster than inspecting each Schedule interactively. + +--- + +## Interpret the result + +### Assess impact + +- Compare `recentActions` to the Schedule's Spec to determine how many Actions were skipped and over what time period. +- If the Schedule uses the `Skip` Overlap Policy and the preceding run was long-running, the miss may reflect that run exceeding the Catchup Window, not a Service outage. +- For business-critical Schedules, Backfill the skipped interval once the underlying cause is resolved. + +### Common root causes + +- **Service or Namespace outage longer than the Catchup Window.** The default Catchup Window is **one year**, so a miss typically means the Schedule is configured with a tighter window (minimum ten seconds) and the outage exceeded it. +- **Namespace rate limiting.** If scheduled starts are throttled, Actions can queue past the Catchup Window. Cross-check `temporal_cloud_v1_schedule_rate_limited_count` (Cloud) or `schedule_rate_limited` (self-hosted) in the same time range. +- **Buffer overruns under `BufferAll`.** Long-running Workflow Executions under `BufferAll` can push buffered Actions past the Catchup Window. Cross-check `temporal_cloud_v1_schedule_buffer_overruns_count` (Cloud) or `schedule_buffer_overruns` (self-hosted) and examine `bufferSize`. + +### Remediation + +- Widen the Catchup Window if the current value is tighter than the Service's worst-case unavailability. The trade-off is more late Actions during recovery. +- Revisit the Overlap Policy if runs routinely exceed the Spec interval. `BufferAll` and `Skip` have different failure modes under sustained delay. +- Increase Namespace throughput limits if rate limiting is the contributing factor. +- Backfill the missed interval if the skipped Actions need to run. + +--- + +## Backfill + +Batch-execute actions that would have run during a specified time interval. Use `BufferAll` or `AllowAll` overlap policies for backfills to avoid skipping Workflow Executions. + +```bash +temporal schedule backfill \ + --schedule-id "YourScheduleId" \ + --start-time "2022-05-01T00:00:00Z" \ + --end-time "2022-05-31T23:59:59Z" \ + --overlap-policy BufferAll +``` + +--- + +## Overlap policies reference + +The overlap policy controls what happens when a new scheduled Action fires while a previous run is still in progress. + +| Policy | Behavior | +|---|---| +| `Skip` | If a previous Workflow Execution is still running, discard new Workflow Executions. | +| `BufferOne` | Same as Skip but buffer a single Workflow Execution to run after the previous completes. Discard others. | +| `BufferAll` | Buffer all incoming Workflow Executions while waiting for the running one to complete. | +| `CancelOther` | Cancel the running Workflow Execution and replace it with the incoming new one. | +| `TerminateOther` | Terminate the running Workflow Execution and replace it with the incoming new one. | +| `AllowAll` | Allow unlimited concurrent Workflow Executions. Significantly speeds up backfilling. Ensure running Workflows do not interfere with each other. | + +There are exactly 6 overlap policies. + +--- + +## Sibling skill pointers + +- For metrics collection and alerting infrastructure, see the observability skill (planned: `skill-temporal-observability`). +- For Schedule CRUD operations (`temporal schedule create`, `temporal schedule update`, etc.) and the time-spec forms, see [`cli-conventions.md`](../ops/cli-conventions.md#schedule-time-spec-forms) in the ops reference files. diff --git a/plugins/temporal/skills/temporal-ops/references/triage/sdk-snippet-review.md b/plugins/temporal/skills/temporal-ops/references/triage/sdk-snippet-review.md new file mode 100644 index 0000000..fd4d7b1 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/triage/sdk-snippet-review.md @@ -0,0 +1,161 @@ +# SDK Snippet Review + +A Layer-0 config check that runs **before** the [diagnostic ladder](diagnostic-ladder.md). When a user pastes SDK connection code (or just the address / namespace / auth fields), the snippet itself is frequently the whole bug — wrong endpoint family, short namespace, auth method mismatched to the Namespace. Running DNS / TCP / TLS probes at that point "fixes" nothing, because nothing lower is broken. + +This file is a **cross-SDK** checklist for **Temporal Cloud** client/worker connection config. SDK-specific field names (`tls.Config{}` in Go, `Connection.connect({ tls })` in TypeScript, `TLSConfig` in Python) belong to `skill-temporal-developer`. Stay at the level of: endpoint, namespace, auth method, TLS expectations, env vars. + +Out of scope here: +- Network-layer probes (DNS / TCP / TLS / auth) → [diagnostic-ladder.md](diagnostic-ladder.md) +- Endpoint-family rationale and DNS failure shapes → [connectivity.md → Endpoint formats](connectivity.md#endpoint-formats) +- API-key lifecycle and address guidance → [authentication.md → Address form for API-key connections](authentication.md#address-form-for-api-key-connections) +- mTLS certificate validation → [certificates.md](certificates.md) + +## Table of Contents + +- [When to use this check](#when-to-use-this-check) +- [Review order](#review-order) +- [Endpoint form](#endpoint-form) +- [Namespace format](#namespace-format) +- [Auth method](#auth-method) +- [TLS expectations](#tls-expectations) +- [Environment variables](#environment-variables) +- [Common misconfigurations](#common-misconfigurations) +- [Quick routing](#quick-routing) + +## When to use this check + +Run snippet review when the user pastes SDK connection code **and** reports any of: + +- Cannot connect to Cloud, `UNAVAILABLE` on first attempt +- `UNAUTHENTICATED` / `PERMISSION_DENIED` on first attempt +- `context deadline exceeded` with no prior-success baseline +- `namespace not found` / `INVALID_ARGUMENT` +- "This worked yesterday" **after** a config change + +Skip snippet review (go straight to the ladder) when the user has an established, previously-working config and the symptom is new — the snippet is not the culprit, something in the environment changed. + +## Review order + +Check in this order. Each step is cheaper than the one below it, and a failure at a higher step makes the lower ones moot. + +1. **Auth method** — API key vs mTLS. This determines everything else. +2. **Endpoint form** — must be a plausible Cloud data-plane address for that auth method (see below). +3. **Namespace** — full `.` form. +4. **TLS expectations** — principle-level, per auth method. +5. **Env-var propagation** — are the `TEMPORAL_*` / `TEMPORAL_CLOUD_*` values actually reaching the client? + +If any of steps 1–3 is wrong, fix that first. Do not start the diagnostic ladder until endpoint, namespace, and auth method are plausible — the ladder will produce misleading results. + +## Endpoint form + +Cloud data-plane endpoints (port `7233`) plus the control-plane endpoint: + +| Purpose | Form | Typical use | +|---|---|---| +| Namespace Endpoint (recommended default) | `..tmprl.cloud:7233` | mTLS and API-key-only Namespaces; follows HA failovers | +| API Regional Endpoint | `..api.temporal.io:7233` | Explicit region pin; private connectivity without private DNS; dual-auth pre-release (API key cannot use Namespace Endpoint) | +| HA Regional Endpoint | `-.region.tmprl.cloud:7233` | Pin to a specific HA replica region | +| Control plane (`tcld`, Cloud Ops API, Terraform) | `saas-api.tmprl.cloud:443` | **Not** a workflow / worker endpoint | + +The Namespace Endpoint follows HA failovers transparently. When a client pins to an API Regional or HA Regional Endpoint (or a PrivateLink/PSC DNS name) with **mTLS**, the client **must** override the TLS `server_name` to the Namespace Endpoint value — see [certificates.md → Server name override](certificates.md#server-name-override). + +**Snippet smells:** + +- Short hostname without `.` or `.tmprl.cloud` / `.api.temporal.io` — stale docs / wrong form. +- `saas-api.tmprl.cloud` as the data-plane address — that's the control plane on port 443, not a workflow endpoint. Pointing a worker or `temporal` CLI data-plane command at it will not connect. +- Dual-auth (`api_key_or_mtls`) Namespace + API key + Namespace Endpoint — pre-release dual-auth does not support API key auth to the Namespace Endpoint; use the API Regional Endpoint. +- mTLS + Regional / VPCE address **without** SNI override — the TLS handshake will fail with `x509: certificate is valid for , not `. See [certificates.md → Hostname mismatch](certificates.md#hostname-mismatch). +- Empty `HostPort` / address with a Cloud namespace set — no explicit Cloud endpoint; the client will attempt a local-dev default that won't reach Cloud. +- URL form (`https://…`) in `--address` / `TEMPORAL_ADDRESS` — the flag takes `host:port`, not a URL. + +## Namespace format + +Cloud namespace format is `.`. The account suffix is visible in the Cloud UI and in `tcld namespace list`. + +**Snippet smells:** + +- Short name (`payments`) without the account suffix — fails resolution, or produces `INVALID_ARGUMENT: namespace not found`. +- Namespace set in the address but not in the namespace field (or vice versa) — both values are required. +- Case mismatch — namespace strings are case-sensitive. Match the Cloud UI exactly. +- Namespace from the wrong account — looks correct but the user isn't logged into that account. Verify with `tcld account get` (see [authentication.md → Cloud role and permission model](authentication.md#cloud-role-and-permission-model)). + +## Auth method + +One auth method per client connection. A snippet that sets both mTLS cert flags **and** an API key is a configuration smell — some SDKs apply both (API key credentials plus client certs), which is ambiguous and hard to debug. Prefer exactly one. + +**Snippet smells:** + +- Both `tls-cert-path` / `tls-key-path` **and** `api-key` set — pick one. +- API key in the snippet but the Namespace auth method is `mtls` only (or vice versa). Check with `tcld namespace auth-method get --namespace `. Migrate via Support or `tcld namespace auth-method set` (`mtls`, `api_key`, or pre-release `api_key_or_mtls`) — recreate is not the only path. +- Auth method mismatched against endpoint constraints (especially dual-auth + Namespace Endpoint for API keys — see [Endpoint form](#endpoint-form)). + +## TLS expectations + +Principle-level only. Do not diagnose SDK-specific struct fields here — that belongs to `skill-temporal-developer`. + +- **API key auth.** TLS is **required**. The client opens a TLS connection (server TLS only, no client cert) and presents the API key as a bearer credential at the gRPC layer. An API-key snippet with TLS explicitly disabled will not connect. +- **mTLS auth.** Client certificate and private key are **required**. The certificate must chain to a CA that the namespace accepts — verify with `tcld namespace accepted-client-ca list --namespace ` (see [certificates.md → Accepted client CA set (mTLS Cloud)](certificates.md#accepted-client-ca-set-mtls-cloud)). The key file must match the cert; see [certificates.md → Key does not match cert](certificates.md#key-does-not-match-cert). + +**Snippet smells:** + +- API key connection with TLS disabled or with an `insecure` flag set — the server will close the connection. +- mTLS snippet without a key file, or key and cert from different generations. +- `--tls-disable-host-verification` / `TEMPORAL_TLS_DISABLE_HOST_VERIFICATION=true` in a snippet the user expects to work in production — this masks hostname-mismatch bugs rather than fixes them. + +## Environment variables + +Two families. Do not mix them up. + +### `temporal` CLI (and common SDK / envconfig names) + +| Variable | Purpose | CLI flag | +|---|---|---| +| `TEMPORAL_ADDRESS` | `host:port` (not a URL) | `--address` | +| `TEMPORAL_NAMESPACE` | `.` | `--namespace` | +| `TEMPORAL_API_KEY` | API-key secret for `temporal` CLI / SDKs | `--api-key` | +| `TEMPORAL_TLS_CA` | Server CA certificate path | `--tls-ca-path` | +| `TEMPORAL_TLS_CERT` | Client x509 certificate path (CLI legacy name) | `--tls-cert-path` | +| `TEMPORAL_TLS_KEY` | Client private key path (CLI legacy name) | `--tls-key-path` | +| `TEMPORAL_TLS_SERVER_NAME` | SNI override (regional / VPCE + mTLS) | `--tls-server-name` | +| `TEMPORAL_TLS_DISABLE_HOST_VERIFICATION` | Default `false` | `--tls-disable-host-verification` | + +SDK envconfig also accepts `TEMPORAL_TLS_CLIENT_CERT_PATH` / `TEMPORAL_TLS_CLIENT_KEY_PATH` (and `*_DATA` variants). Those are real for SDKs — not typos. The CLI's legacy names are `TEMPORAL_TLS_CERT` / `TEMPORAL_TLS_KEY`. + +### `tcld` / Terraform (control plane) + +| Variable | Purpose | +|---|---| +| `TEMPORAL_CLOUD_API_KEY` | API key for `tcld` and the Terraform provider | + +`TEMPORAL_API_KEY` is **not** what `tcld` reads. Public Cloud docs that say otherwise are wrong vs current `tcld` source. + +**Snippet smells:** + +- Snippet hardcodes a value that an env var also sets. Precedence is SDK-specific; the effective value may not be what the snippet shows. Ask the user to echo the env var from the exact shell / container the client runs in. +- Env vars set in the user's interactive shell but the worker runs in a different shell / container / pod where they are absent. +- Wrong tool's API-key env var — `TEMPORAL_CLOUD_API_KEY` for an SDK/`temporal` CLI client, or `TEMPORAL_API_KEY` for `tcld`. Symptom: "my config doesn't take effect." +- Typo that truly does not exist — e.g. `TEMPORAL_TLS_CERT_FILE`. Unknown variables are ignored. +- Mixing env vars and explicit flags across auth methods — e.g. `TEMPORAL_API_KEY` set in the environment, but the snippet also passes `--tls-cert-path`. See [Auth method](#auth-method) above. + +## Common misconfigurations + +| Snippet shape | Likely root cause | First fix | +|---|---|---| +| API key + short / wrong hostname | Missing Namespace Endpoint or wrong family | Use `..tmprl.cloud:7233` (default) or the API Regional form when pinning / dual-auth requires it | +| Dual-auth Namespace + API key + `*.tmprl.cloud` | Dual-auth pre-release blocks API key on Namespace Endpoint | Switch to `..api.temporal.io:7233` | +| mTLS + `*.api.temporal.io` / `*.region.tmprl.cloud` / VPCE without SNI | Regional / private address needs `server_name = ..tmprl.cloud` | Add SNI override, or switch to Namespace Endpoint | +| Short namespace (`payments`) | Missing account suffix | Use `.` from `tcld namespace list` | +| Empty HostPort / address, Cloud namespace set | No explicit Cloud endpoint | Add Namespace Endpoint (or Regional when required) | +| API key **and** mTLS cert both configured | Ambiguous; may present both | Remove whichever you are not using | +| `saas-api.tmprl.cloud` as address | Control plane mistaken for data plane | Switch to Namespace or Regional Endpoint | +| `https://…` in `--address` | Flag takes `host:port`, not a URL | Strip the scheme | +| SDK/`temporal` CLI with only `TEMPORAL_CLOUD_API_KEY` | Wrong env var for that client | Set `TEMPORAL_API_KEY` | +| `tcld` with only `TEMPORAL_API_KEY` | Wrong env var for tcld | Set `TEMPORAL_CLOUD_API_KEY` | +| CLI expected, but only `TEMPORAL_TLS_CLIENT_CERT_PATH` set | CLI reads `TEMPORAL_TLS_CERT` | Rename for CLI, or keep `_CLIENT_` names for SDK envconfig | + +## Quick routing + +- Snippet looks plausible on all five review points → proceed to [diagnostic-ladder.md](diagnostic-ladder.md). +- Snippet has an obvious wrong endpoint, namespace, or auth method → fix that first; do not run network-layer probes until it's corrected. +- Snippet fix applied and symptom persists → descend the ladder from layer 1 (DNS). The environment may have a second, independent problem. +- Snippet references SDK-specific struct fields, connection-builder objects, or runtime-specific TLS APIs that aren't covered here → hand off to `skill-temporal-developer`. diff --git a/plugins/temporal/skills/temporal-ops/references/triage/worker-health.md b/plugins/temporal/skills/temporal-ops/references/triage/worker-health.md new file mode 100644 index 0000000..76cc923 --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/triage/worker-health.md @@ -0,0 +1,236 @@ +# Worker Health + +Diagnose worker-side failures that surface as tasks not being picked up, picked up slowly, or dropped because no worker is actually connected to the right task queue. This file scopes to the worker-polling surface — the gap between "a task is enqueued on a Task Queue" and "a Worker Process dequeues it" — and to the direct symptoms of worker saturation (task slots, sticky cache, heartbeat). + +Prerequisite: layers 1–4 already succeed. If the worker cannot connect or cannot authenticate, its pollers will never register and every symptom here is secondary — rule those out via [connectivity.md](connectivity.md), [certificates.md](certificates.md), [authentication.md](authentication.md) first. + +Out of scope here: +- Pending activity / pending child / pending signal inside a specific Workflow Execution → [workflow-stuck.md](workflow-stuck.md) +- Non-determinism / replay errors / `WorkflowTaskFailed` loops → [non-determinism.md](non-determinism.md) +- gRPC `RESOURCE_EXHAUSTED` / server-side throttling → [rate-limits.md](rate-limits.md) +- DNS / TCP / endpoint / TLS / auth → earlier layers +- SDK-specific worker knob names and tuning targets — described here as concepts only; ground-truth for the API names lives in each SDK's worker docs, not in this skill + +## Table of Contents + +- [What "no pollers" looks like](#what-no-pollers-looks-like) +- [Inspecting a Task Queue with `temporal task-queue describe`](#inspecting-a-task-queue-with-temporal-task-queue-describe) +- [Schedule-to-start latency](#schedule-to-start-latency) +- [Worker task slots](#worker-task-slots) +- [Sticky Execution and sticky cache](#sticky-execution-and-sticky-cache) +- [Worker heartbeating, restarts, and disconnects](#worker-heartbeating-restarts-and-disconnects) +- [Cloud Namespace-level poller limits](#cloud-namespace-level-poller-limits) +- [Worker log signatures](#worker-log-signatures) +- [Quick routing](#quick-routing) + +## What "no pollers" looks like + +**Symptom shape:** a workflow or activity has been enqueued on a Task Queue, but no Worker picks it up. The classic server-side confirmation is that `temporal task-queue describe` returns no recent pollers for that queue and type. + +The server's view of liveness for a Worker polling a Task Queue is time-based: "The Temporal Server records each poll request time. A `LastAccessTime` over one minute may indicate the Worker is at capacity or has shut down. Temporal Workers are removed if 5 minutes have passed since the last poll request." So "no pollers" can mean any of: + +- No worker process is reaching the frontend on this Task Queue at all. +- Workers reached the frontend historically but have been silent > 5 minutes, so their poller entries have aged out. +- Workers are polling a *different* queue (name mismatch: `foo` vs `foo-prod` vs `foo ` with trailing whitespace) or a different Namespace. +- Workers connect and authenticate, then fail before the poll loop — crash on startup, deadlock, or TLS/auth rejection silently producing a reconnect loop. + +This is a discrimination problem: the triage task is to separate "no worker was ever started" from "the worker is running but pointed somewhere else" from "the worker is running but blocked." The tools below address each. + +## Inspecting a Task Queue with `temporal task-queue describe` + +The authoritative worker-side view from outside the worker process is `temporal task-queue describe`, which reports "a list of active Workers that have recently polled a Task Queue." + +### Minimal invocation + +```bash +temporal task-queue describe \ + --task-queue YourTaskQueue +``` + +Workflow and Activity polling happen on separate queues internally; filter with `--task-queue-type` (`workflow`, `activity`, or `nexus`): + +```bash +temporal task-queue describe \ + --task-queue YourTaskQueue \ + --task-queue-type "activity" +``` + + Flag: `--task-queue-type` accepts `workflow`, `activity`, `nexus`, and defaults to reporting all types. + +### Statistics the describe call returns + +Per the CLI docs, a describe call without `--disable-stats` returns the following Task Queue statistics: + +- `ApproximateBacklogCount` — approximate number of tasks backlogged. "May count expired tasks but eventually converges to the right value." +- `ApproximateBacklogAge` — approximate age of the oldest backlog task in seconds. +- `TasksAddRate` — approximate rate at which tasks are being added, averaged over the last 30 seconds. +- `TasksDispatchRate` — approximate rate at which tasks are being dispatched, averaged over the last 30 seconds. +- `BacklogIncreaseRate` — roughly `TasksAddRate − TasksDispatchRate`; docs note this is "accurate for backlogs older than a few seconds" even though the component rates can be off because of sync-matched and sticky dispatch. + +These statistics are the server's view of matching-service throughput; combine them with the poller list (below) to answer "is there a worker attached, and is it keeping up?" + +### Reading the pollers field + +The textual output includes a list of Worker identities that have recently polled the queue. The identifier Temporal reports is a Worker Identity, and by default "Temporal SDKs set a Worker Identity to `${process.pid}@${os.hostname()}`". A "recent" poller is one whose `LastAccessTime` is within 5 minutes. + +Interpretation matrix: + +| Observation | What it suggests | +|---|---| +| No pollers at all | No Worker reached the frontend for this queue+type within the last 5 minutes. | +| Pollers present, `LastAccessTime` > ~1 minute, no tasks moving | Docs note this "may indicate the Worker is at capacity or has shut down." | +| Pollers present, recent, backlog still growing | Sizing or per-task-failure issue — see [Schedule-to-start latency](#schedule-to-start-latency) and [Worker task slots](#worker-task-slots) below. | +| Pollers present but identities don't match the worker fleet the user expects | Worker versioning / Build ID routing — re-run with versioning-aware flags (below). | + +### Reachability and versioning + +On the same `describe` call you can request reachability information, which reports whether a given Build ID can be reached by new Workflows, only by closed Workflows, or is unreachable: + +```bash +temporal task-queue describe \ + --task-queue YourTaskQueue \ + --select-build-id "YourBuildId" \ + --report-reachability +``` + + Relevant flags for versioning discrimination: + +| Flag | Purpose | +|---|---| +| `--select-build-id` | Filter the Task Queue describe results to one or more Build IDs. | +| `--select-unversioned` | Include the unversioned queue. | +| `--select-all-active` | Include all "active" versions (recent polls or new tasks). | +| `--report-reachability` | Display task reachability. Docs note "task reachability status is deprecated in favor of Drainage Status (ie. of a Drained or Draining Worker Deployment Version) and will be removed in a future release." | +| `--report-config` | Include Task Queue rate-limit configuration. | +| `--legacy-mode` | Fallback for servers that do not support rules-based worker versioning; "only provides pollers info." | +| `--disable-stats` | Suppress the statistics listed above. | + +If the user is running Worker Deployments, the per-version worker view is `temporal worker deployment describe-version`, which reports "the task queues polled by workers in this Deployment Version, or drainage information required to safely decommission workers". Adding `--report-task-queue-stats` reports per-task-queue stats for that version. + +### Worker-level describe and list + +Separate from Task Queue introspection, two worker-centric CLI commands surface individual Worker instances connected to the server: + +```bash +temporal worker list --namespace YourNamespace --query 'TaskQueue="YourTaskQueue"' +temporal worker describe --namespace YourNamespace --worker-instance-key YourKey +``` + + These depend on Worker heartbeating being enabled (see [Worker heartbeating](#worker-heartbeating-restarts-and-disconnects) below); if heartbeating is disabled, "features that provide the list of active Workers and information about those Workers to show missing or inaccurate information." + +## Schedule-to-start latency + +The primary *latency* signal for "tasks are being enqueued but not picked up promptly" is the SDK's schedule-to-start metric, split by task kind: + +- `temporal_workflow_task_schedule_to_start_latency` — "the time between when a [Workflow Task](https://docs.temporal.io/tasks#workflow-task) is scheduled (enqueued) and when it is picked up by a Worker for processing." +- `temporal_activity_schedule_to_start_latency` — "the time between when an [Activity Task](https://docs.temporal.io/tasks#activity-task) is scheduled (enqueued) and when it is picked up by a Worker for processing." + +Both are Worker-reported SDK metrics, available as histograms (`_bucket`, `_sum`, `_count`). Per the Cloud worker-health guide, "This latency should be very low, close to zero. Any higher value indicates a bottleneck", with example alert thresholds of ">200ms for your p99 value" and ">100ms for your p95 value." + +### Causes documented for high schedule-to-start latency + +The performance-bottlenecks troubleshooting page lists, for both the Workflow-Task and Activity-Task variants: + +- **Insufficient Worker capacity.** "If there aren't enough Workers or if the Workers are overloaded, they may not be able to pick up Tasks quickly enough." +- **Worker configuration issues.** "Improperly configured Workers, such as having too few pollers or Task slots, can lead to increased latency." +- **Network latency.** "Workers in a different region from the Temporal cluster, or large payload size, can introduce additional latency." +- Activity-specific: **Task Queue configuration.** "Setting `TaskQueueActivitiesPerSecond` too low can limit the rate at which Activities are started, leading to increased Schedule-to-start latency." This is client/worker-imposed throttling, not server throttling — the symptom is latency, not `RESOURCE_EXHAUSTED`. The discrimination against server-side throttling is covered in [rate-limits.md → Server throttled me vs. client throttled itself](rate-limits.md#server-throttled-me-vs-client-throttled-itself). +- Workflow-specific: **High Workflow lock latency.** "If many updates are made to a single execution, this can cause Workflow lock latency, which in turn affects the Schedule-to-start latency. Reduce the rate of Signals." + +### Cloud-side counterparts + +On Cloud, two server-visible metric families let you cross-check without pulling SDK metrics: + +- `temporal_cloud_v1_approximate_backlog_count` — "Approximate number of tasks pending in a task queue. Started Activities are not included in the count as they have been dequeued from the task queue." +- No direct Cloud-side "no poller" metric exists. Use `temporal_cloud_v1_approximate_backlog_count` rising while poll success rate is zero as the proxy signal for the "no poller" case in [What "no pollers" looks like](#what-no-pollers-looks-like). The self-hosted equivalent is `no_poller_tasks`. +- Sync-match ratio: `temporal_cloud_v1_poll_success_sync_count / temporal_cloud_v1_poll_success_count`. The Cloud worker-health page targets ">95%, but preferably >99%"; a drop indicates workers are not on hand to sync-match new tasks. + +See [Cloud Namespace-level poller limits](#cloud-namespace-level-poller-limits) for the related *greedy-worker* signal (poll timeouts). + +## Worker task slots + +A Worker Entity has a bounded number of concurrent Task execution slots per task kind. When those slots saturate, even a well-connected Worker stops picking up new Tasks, and the symptom is identical to "too few Workers": rising schedule-to-start latency. + +The worker-side gauge is `temporal_worker_task_slots_available`: "The total number of Workflow, Activity, Local Activity, or Nexus Task execution slots that are currently available. Use the `worker_type` key to differentiate execution slots." Cloud worker-health documents the monitoring target: "The `temporal_worker_task_slots_available` metric should always be >0." + +Docs call out two common depletion patterns: + +- **Workflow-Worker slot depletion** — usually driven by inbound Workflow Task load exceeding the configured cap, or by elevated `temporal_workflow_task_execution_latency` / `workflow_task_replay_latency` holding slots longer. +- **Activity-Worker slot depletion** — docs' first-listed cause is "Blocked Activities and Zombie Activities ... when an Activity times out (hits its `StartToClose` or `HeartbeatTimeout` timeout) and has stopped Heartbeating but continues to run, occupying some or all the slots as more retries occur." A companion gauge, `temporal_worker_task_slots_used`, surfaces how many slots are in use. + +The *configuration* knobs that set these slot caps differ by SDK; the Cloud worker-health page refers to `maxConcurrentWorkflowTaskExecutionSize` and `maxConcurrentActivityExecutionSize` as the generic names, and the Worker tuning reference documents the per-SDK defaults (e.g., Go 1,000; Java 200; TypeScript 40 / 100; Python 100; .NET 100). For SDK-specific API names and tuning, follow `/develop/worker-performance` and `/develop/worker-tuning-reference` rather than guessing. + +The triage-layer conclusion is: confirm slot availability is non-zero over the period the task was stuck; if it is chronically zero, the ceiling is the Worker, not the server. + +## Sticky Execution and sticky cache + +**Concept:** "Workers cache the state of the Workflow they execute. ... Temporal employs a performance optimization known as 'Sticky Execution', which directs Workflow Tasks to the same Worker that previously processed tasks for a specific Workflow Execution." Each Worker gets an "automatically-generated" Sticky Queue name exclusive to that Worker. + +**Failure mode relevant to triage:** if the Worker fails to start a queued Workflow Task in its Sticky Queue "shortly after it's scheduled (within five seconds by default), the Temporal Service disables stickiness for that Workflow Execution" and reschedules on the original Task Queue. A related trigger: if a Workflow Task fails, "the Worker removes that Workflow Execution from its cache ... which invalidates the Sticky Execution." + +Key metrics for cache-pressure diagnosis: + +- `temporal_sticky_cache_size` — "Current cache size, expressed in number of Workflow Executions." +- `temporal_sticky_cache_total_forced_eviction` / `_total` — counter for forced evictions. "A 'forced eviction' ... means that a Workflow Execution was removed from the cache before it completed, typically because the cache was full and needed to make room for other Workflow Executions." Available in the Go SDK and the Java SDK. +- `temporal_sticky_cache_hit_total` / `temporal_sticky_cache_miss_total` — sticky match versus Event-History-replay. + +The Cloud worker-health guidance: "The `sticky_cache_size` should report less than or equal to your `WorkflowCacheSize` value. Also, `sticky_cache_total_forced_eviction` should not be reporting high numbers (relative)." The `WorkflowCacheSize` / `StickyWorkflowCacheSize` per-SDK defaults are in the Worker tuning reference (Go 10,000; Java 600; TypeScript dynamic; Python 1,000; .NET 10,000). + +Triage-layer conclusions: + +- A workflow that keeps moving but with elevated schedule-to-start latency and a high sync-match-miss or a high forced-eviction rate is starving the cache, not a pollers-missing problem. +- A workflow that was Sticky to a Worker that later died or restarted will wait up to five seconds before the Sticky-queue scheduler gives up and the Task returns to the shared queue. + +## Worker heartbeating, restarts, and disconnects + +Temporal Workers send a heartbeat to the Server on an interval. Per the Cloud worker-health guide (the feature is "in Public Preview" as documented): "Workers send a heartbeat to Temporal Server every 60 seconds by default. This heartbeat serves to provide liveness and configuration data from the Worker to the Server." Heartbeats feed the `temporal worker list` / `temporal worker describe` views; "Disabling the Worker heartbeat will cause features that provide the list of active Workers and information about those Workers to show missing or inaccurate information." + +Heartbeating is distinct from Task Queue polling. A Worker process can stop polling (or start failing polls) for reasons that do not appear in its heartbeat. Consequently: + +- If `temporal task-queue describe` shows no pollers but `temporal worker list --query 'TaskQueue="..."'` shows recent Workers, the Workers are alive and heartbeating but not polling this queue (name mismatch, task-queue-type mismatch, or they've exhausted slots). +- If `temporal worker list` is empty for a Namespace where the user insists workers are running, the likeliest triage question is "is heartbeating disabled?" or "do the workers predate an SDK version with heartbeating available?" Per the Cloud worker-health page, heartbeating is available in specific SDK versions: Go SDK v1.41.0+, Java SDK v1.35.0+, Python SDK v1.20.0+, TypeScript SDK v1.14.0+, .NET SDK v1.10.0+, Ruby SDK v1.1.0+. + +Restart-loop / container-kill failure shapes are operating-system and orchestrator concerns, not Temporal-specific errors — the triage-layer observation is that a Worker in a restart loop will never sustain the 5-minute `LastAccessTime` window the server needs to keep it in the poller list. If the Worker container is crash-looping, the symptom will alternate between "pollers present briefly" and "pollers absent" at the restart cadence. + +## Cloud Namespace-level poller limits + +Cloud caps the *total* pollers a Namespace can hold concurrently: "Temporal Cloud limits each Namespace to 20,000 Activity pollers and 20,000 Workflow Task pollers concurrently." Per the same section, "Each SDK offers a way to configure Workers for per-Worker maximum Activity and Workflow Task pollers. Those values do not affect the global Namespace limit." Saturating this cap is a Namespace-level policy limit — it falls under [rate-limits.md](rate-limits.md) in terms of gRPC surface behavior. + +A related symptom is *too many* pollers for the load: the Poll Success Rate, computed as `temporal_cloud_v1_poll_success_count / (temporal_cloud_v1_poll_success_count + temporal_cloud_v1_poll_timeout_count)`, falling well below the documented target of ">90% in most cases of systems with a steady load. For high volume and low latency, try to target >95%." The Cloud guide's interpretation: low poll success + low schedule-to-start latency + low Worker host utilization together suggest "you might have too many Workers." + +Self-hosted has no equivalent documented Namespace-wide poller cap in this skill's primary sources; see `docs/references/dynamic-configuration.mdx` for per-service RPS and capacity keys and [rate-limits.md → Self-hosted: service-level RPS via dynamic configuration](rate-limits.md#self-hosted-service-level-rps-via-dynamic-configuration). + +## Worker log signatures + +Patterns to watch for in the Worker's own process logs when correlating with `task-queue describe` output. Each row points to the file that actually owns the diagnostic. + +| Log pattern (shape — exact wording is SDK/version-specific) | Interpretation | Route | +|---|---|---| +| gRPC `UNAUTHENTICATED` / `PERMISSION_DENIED` at worker startup or on poll RPCs | Identity or authorization rejected after TLS; Worker never registers as a poller | [authentication.md](authentication.md) | +| `x509: ...` / `tls: ...` | TLS handshake or cert validation failed before gRPC | [certificates.md](certificates.md) | +| `no such host` / `connection refused` / `i/o timeout` | DNS / TCP problem; poll loop never engages | [connectivity.md](connectivity.md) | +| gRPC `RESOURCE_EXHAUSTED` | Server throttling the Worker's RPCs (polls or other) | [rate-limits.md](rate-limits.md) | +| `temporal_long_request_failure` elevated on `PollWorkflowTaskQueue` / `PollActivityTaskQueue` | Long-poll RPCs failing; docs note this "is often indicated by a `ResourceExhausted` status code" when caused by rate limiting, but also covers network issues and server errors | [rate-limits.md](rate-limits.md) / [connectivity.md](connectivity.md) depending on the `status` / `code` tag on the metric | +| "Deadlock detected during Workflow run" / `TMPRL1101` | Workflow Task ran longer than the SDK's deadlock-detection threshold | [non-determinism.md](non-determinism.md) and the relevant SDK docs | +| Replay-divergence messages from the SDK (non-determinism) | Re-execution of recorded history no longer matches the compiled Workflow | [non-determinism.md](non-determinism.md) | +| Unhandled exception / panic / OOM kill at process level, no Temporal-specific error | Runtime-level failure, not a Temporal error string; the Worker will drop out of the poller list until restarted | OS / runtime tooling; not covered in this skill | + +Correlate these with `task-queue describe` output. Agreement (log says "polling" and describe shows a fresh poller) confirms the Worker is healthy at the server's view. Disagreement is itself a diagnostic signal — e.g., log says "connected" but `task-queue describe` is empty → likely a Namespace or Task Queue name mismatch, or heartbeating is disabled so `worker list` is silent while polling still works. + +## Quick routing + +| Symptom | Go to | +|---|---| +| `temporal task-queue describe` returns no pollers | [What "no pollers" looks like](#what-no-pollers-looks-like) | +| `temporal_cloud_v1_approximate_backlog_count` rising with zero poll success on Cloud | [Schedule-to-start latency](#schedule-to-start-latency) (proxy for no-poller on Cloud; no direct no-poller metric exists) | +| Pollers are present but `LastAccessTime` stale | [What "no pollers" looks like](#what-no-pollers-looks-like) and [Worker heartbeating](#worker-heartbeating-restarts-and-disconnects) | +| `temporal_workflow_task_schedule_to_start_latency` or `temporal_activity_schedule_to_start_latency` spike | [Schedule-to-start latency](#schedule-to-start-latency) | +| `temporal_worker_task_slots_available` hits 0 | [Worker task slots](#worker-task-slots) | +| Elevated `temporal_sticky_cache_total_forced_eviction` | [Sticky Execution and sticky cache](#sticky-execution-and-sticky-cache) | +| `temporal worker list` is empty but workers are running | [Worker heartbeating](#worker-heartbeating-restarts-and-disconnects) | +| Low Poll Success Rate with low schedule-to-start latency and low host utilization | [Cloud Namespace-level poller limits](#cloud-namespace-level-poller-limits) — likely too many Workers | +| `RESOURCE_EXHAUSTED` on poll RPCs | [rate-limits.md](rate-limits.md) | +| Workflow Execution is stuck with a pending activity / child / signal | [workflow-stuck.md](workflow-stuck.md) | +| Replay errors / non-determinism in Worker logs | [non-determinism.md](non-determinism.md) | + +See the whole-stack picture in [diagnostic-ladder.md](diagnostic-ladder.md). diff --git a/plugins/temporal/skills/temporal-ops/references/triage/workflow-stuck.md b/plugins/temporal/skills/temporal-ops/references/triage/workflow-stuck.md new file mode 100644 index 0000000..d51a95b --- /dev/null +++ b/plugins/temporal/skills/temporal-ops/references/triage/workflow-stuck.md @@ -0,0 +1,351 @@ +# Workflow Stuck + +A Workflow Execution exists, is reachable, reports an Open status, but is not making progress. This file scopes to what the server sees when that is true: the execution's own state as surfaced by `temporal workflow describe`, the append-only Event History as surfaced by `temporal workflow show`, and the pending-operation sub-structures (pending activities, pending child workflows, pending nexus operations, pending workflow task) that describe exposes. + +Prerequisite: the client can reach the Temporal Service, authenticate, and issue data-plane RPCs. If `temporal workflow describe` itself fails, the problem is not in this file — rule out earlier layers via [connectivity.md](connectivity.md), [certificates.md](certificates.md), [authentication.md](authentication.md), and [rate-limits.md](rate-limits.md). + +Out of scope here: +- Task Queue has no pollers / workers not polling → [worker-health.md](worker-health.md) +- Replaying recorded history diverges from the compiled Workflow (non-determinism) → [non-determinism.md](non-determinism.md) +- Replaying an Event History locally under a debugger → [replay.md](replay.md) +- gRPC `RESOURCE_EXHAUSTED` when the client call to describe/signal/query was itself rate-limited → [rate-limits.md](rate-limits.md) +- `context deadline exceeded` on the client side → [runtime-errors.md](runtime-errors.md) + +## Table of Contents + +- [What "stuck" means](#what-stuck-means) +- [Workflow Execution Status values](#workflow-execution-status-values) +- [The primary inspection command: `temporal workflow describe`](#the-primary-inspection-command-temporal-workflow-describe) +- [Inspecting the Event History: `temporal workflow show`](#inspecting-the-event-history-temporal-workflow-show) +- [Pending activities](#pending-activities) +- [Pending child workflows](#pending-child-workflows) +- [Pending signals, cancellations, and updates](#pending-signals-cancellations-and-updates) +- [Pending Nexus Operations](#pending-nexus-operations) +- [Pending Workflow Task and WorkflowTaskFailed loops](#pending-workflow-task-and-workflowtaskfailed-loops) +- [Timer-based waits](#timer-based-waits) +- [Pending-operation per-Workflow limits](#pending-operation-per-workflow-limits) +- [Recovery commands](#recovery-commands) +- [Quick routing](#quick-routing) + +## What "stuck" means + +"Stuck" is not a Temporal-defined state. It is a user observation that requires server evidence to classify. The three shapes of evidence that reliably separate "stuck" from "running as designed": + +- **Event History has grown recently, but the last non-bookkeeping event is a scheduling event with no corresponding terminal event.** For example, `ActivityTaskScheduled` appears with no `ActivityTaskCompleted` / `ActivityTaskFailed` / `ActivityTaskTimedOut` appended for an interval longer than expected. +- **`temporal workflow describe` shows a pending activity, child workflow, nexus operation, or workflow task whose attempt count is climbing and whose last failure reason is reported.** Per the Activity Operations observability guide: "`temporal workflow describe` shows the current state of each pending Activity, including whether it's Paused, its current attempt count, and last failure." +- **Event History has not grown for longer than the longest timer or timeout currently outstanding.** A workflow can legitimately wait years on a Timer or indefinitely on a signal. Absence of progress is only evidence of stuck-ness when it exceeds the longest outstanding deadline the Workflow itself established. + +All three rely on comparing the Workflow's own records (history + describe output) against expectations. Neither the SDK nor the server synthesises a single "Workflow is stuck" signal — the evidence lives in the fields below. + +## Workflow Execution Status values + +A Workflow Execution is either _Open_ or _Closed_. The full canonical set of statuses surfaced through the `ExecutionStatus` Search Attribute is: `Running, Completed, Failed, Canceled, Terminated, ContinuedAsNew, TimedOut`. + +| Status | Open/Closed | Meaning | Source | +|---|---|---|---| +| `Running` | Open | "The only Open status for a Workflow Execution. When the Workflow Execution is Running, it is either actively progressing or is waiting on something." | | +| `Completed` | Closed | "The Workflow Execution has completed successfully." | | +| `Failed` | Closed | "The Workflow Execution returned an error and failed." | | +| `Canceled` | Closed | "The Workflow Execution successfully handled a cancellation request." | | +| `Terminated` | Closed | "The Workflow Execution was terminated." | | +| `ContinuedAsNew` | Closed | "The Workflow Execution Continued-As-New." | | +| `TimedOut` | Closed | "The Workflow Execution reached a timeout limit." | | + +A "stuck" Workflow is almost always in `Running`. The phrase *"either actively progressing or waiting on something"* is the core ambiguity this file exists to resolve: `Running` alone cannot distinguish "working" from "wedged." The describe output and history are what distinguish them. + +Transitions terminate the run: + +- A Workflow Execution retry caused by its Retry Policy ends the failed run with `Failed` (status `Failed`, `retryState=IN_PROGRESS`, `newExecutionRunId` set) and spawns a new run. +- Continue-As-New closes the current run with `ContinuedAsNew` and starts a new run in the same Workflow Execution Chain. +- A Workflow Execution Timeout produces status `TimedOut`. +- The Event History reaching 51,200 events, 2,000 Updates, or 10,000 Signals terminates the Workflow Execution. + +If the user reports a stuck workflow whose describe output shows a Closed status, they are not describing a stuck workflow — they are describing a completed or failed one, and the rest of this file does not apply. + +## The primary inspection command: `temporal workflow describe` + +`temporal workflow describe` is the one command that reports the server's current view of a single Workflow Execution's state. This file's diagnosis rests on the shape of its output. + +```bash +temporal workflow describe \ + --workflow-id YourWorkflowId +``` + + Flags: + +| Flag | Required | Purpose | +|---|---|---| +| `--workflow-id`, `-w` | Yes | Workflow ID. | +| `--run-id`, `-r` | No | Run ID. If omitted, the most recent run in the Workflow Execution Chain is described. | +| `--reset-points` | No | "Show auto-reset points only." | +| `--raw` | No | "Print properties without changing their format." | +| `--output`, `-o` | No (global) | Output format. Accepted values: `text, json, jsonl, none`. | + +### Reading the output + +The text example shown in the CLI docs for a minimal Running execution exposes these top-level fields: `executionConfig`, `workflowExecutionInfo` (containing `execution.workflowId`, `execution.runId`, `type.name`, `startTime`, `status`, `historyLength`, `executionTime`, `memo`, `autoResetPoints`, `stateTransitionCount`), and `pendingWorkflowTask` (containing `state`, `scheduledTime`, `originalScheduledTime`, `attempt`). + +The Nexus-operations and pending-activities text examples in the encyclopedia show an alternative tabular shape under the same command — `Pending Activities: N`, `Pending Child Workflows: N`, `Pending Nexus Operations: N`, each followed by an indented block. + +Which sections appear depends on the Workflow's current state: a Workflow with no pending operations simply won't show those sections. + +### What to look at first + +Per the intake shape above, the three checks that partition the problem: + +1. **`workflowExecutionInfo.status`** — if it's anything other than `Running`, the workflow is Closed, not stuck. See [Workflow Execution Status values](#workflow-execution-status-values). +2. **Pending sections present** — are there `Pending Activities`, `Pending Child Workflows`, `Pending Nexus Operations`, or a `pendingWorkflowTask`? Each maps to a section below. +3. **`historyLength`** — stored in `workflowExecutionInfo.historyLength`. If it is climbing between successive describes but the pending sections don't resolve, events are being written but the Workflow is cycling. If it is flat with no pending operations, the Workflow is asleep on something with no visible pending work (typically a Timer — see [Timer-based waits](#timer-based-waits)). + +## Inspecting the Event History: `temporal workflow show` + +`temporal workflow show` returns the Event History, the append-only log of Events that drives replay. + +```bash +temporal workflow show \ + --workflow-id YourWorkflowId \ + --output json +``` + + Flags: + +| Flag | Required | Purpose | +|---|---|---| +| `--workflow-id`, `-w` | Yes | Workflow ID. | +| `--run-id`, `-r` | No | Run ID. | +| `--detailed` | No | "Display events as detailed sections instead of table. Does not apply to JSON output." | +| `--follow`, `-f` | No | "Follow the Workflow Execution progress in real time. Does not apply to JSON output." | +| `--output json` | No (global) | Emits the JSON shape an SDK would replay. | + +Event types are enumerated in the [Events reference](https://docs.temporal.io/references/events); the canonical list is the `EventType` proto enum. A Command issued by the Workflow code produces a corresponding Event on the server — the Command/Event mapping is documented in the [Commands reference](https://docs.temporal.io/references/commands). + +### Finding the last meaningful event + +Within each Workflow Task execution, the server records `WorkflowTaskScheduled`, `WorkflowTaskStarted`, and `WorkflowTaskCompleted` bookkeeping around the state changes the Workflow code produced. When diagnosing stuck-ness, the event type that exposes *why* the workflow is waiting is the last non-bookkeeping event — the scheduling event (e.g., `ActivityTaskScheduled`, `StartChildWorkflowExecutionInitiated`, `TimerStarted`, `NexusOperationScheduled`) with no matching terminal event yet. + +Asymmetric recording: per the event encyclopedia, "While the Activity is running and retrying, `ActivityTaskScheduled` is the only Activity-related Event in History: `ActivityTaskStarted` is written along with a terminal Event like `ActivityTaskCompleted` or `ActivityTaskFailed`." So a long gap between `ActivityTaskScheduled` and its terminal event does *not* imply the Activity is unpicked — it may be running and retrying. Use `temporal workflow describe`'s Pending Activities section to see retry state, not the Event History alone. + +## Pending activities + +A Pending Activity is an Activity whose `ActivityTaskScheduled` event has been appended but whose terminal event has not. The retry state of the in-flight Activity Execution (attempt count, last failure, paused status) is only visible via `temporal workflow describe`; it is not re-written to Event History on each retry. + +### Fields surfaced by describe for a pending activity + +Docs state that describe surfaces "the current state of each pending Activity, including whether it's Paused, its current attempt count, and last failure." A Pending Activity block's exact text-output field names are not enumerated in a single docs location in this repo's snapshot. Treat the fields below as conceptual checkpoints for the diagnosis, and read the live describe output for exact field names. + +Things to check on a pending activity: + +- **Attempt count.** If retries are climbing and the Activity has a Retry Policy with a finite `Maximum Attempts`, the Activity will eventually be recorded as `ActivityTaskFailed` and the Workflow will proceed (or fail) accordingly. If Maximum Attempts is unlimited and each attempt is failing for the same reason, the Workflow will loop indefinitely on this Activity. +- **Last failure.** The `last_failure` field of `ActivityTaskStarted` carries "Details from the most recent failure Event. Only assigned values if the Task has previously failed and been retried." The describe output exposes the equivalent information for pending activities. +- **Scheduled vs. Started.** If the Activity has never been picked up (only `ActivityTaskScheduled` in the history, no retry attempts in the pending block), the problem is almost certainly worker-side — no Worker is polling the Task Queue the Activity was scheduled on, or pollers exist but none match the Build ID routing the Workflow expects. Route to [worker-health.md](worker-health.md). +- **Timeouts on the scheduling attributes.** The `activityTaskScheduledEventAttributes` record includes `schedule_to_close_timeout`, `schedule_to_start_timeout`, `start_to_close_timeout`, and `heartbeat_timeout`. A long-running Activity without heartbeats and a missing `start_to_close_timeout` or `heartbeat_timeout` can block indefinitely: "The Temporal Server doesn't detect failures when a Worker loses communication with the Server or crashes. Therefore, the Temporal Server relies on the Start-To-Close Timeout to force Activity retries." +- **Paused state.** If the Activity was explicitly paused via `temporal activity pause`, retries are server-side suspended. Unpause via `temporal activity unpause`. + +### Heartbeating + +For long-running activities, heartbeating is the only mechanism by which the server detects a dead worker. A missed heartbeat within the `heartbeat_timeout` results in `ActivityTaskTimedOut` and a retry. Activities without heartbeating cannot be killed server-side mid-execution; they run until `start_to_close_timeout` (or forever, if neither start-to-close nor schedule-to-close is set). + +## Pending child workflows + +A Pending Child is one whose `StartChildWorkflowExecutionInitiated` or `ChildWorkflowExecutionStarted` event has been appended but whose terminal event (`ChildWorkflowExecutionCompleted`, `ChildWorkflowExecutionFailed`, `ChildWorkflowExecutionCanceled`, `ChildWorkflowExecutionTimedOut`, `ChildWorkflowExecutionTerminated`) has not. + +Things to check: + +- **Extract the child's Workflow ID and Run ID from the parent's describe output**, then recurse into describe on the child. The parent is only stuck in the sense that the child is — the root cause lives in the child's describe / history. +- **The child may be in a different Namespace.** Child Workflows can cross Namespaces; if the parent's describe shows a child that doesn't appear in the current Namespace's `temporal workflow list`, re-run describe with the correct `--namespace`. +- **Parent Close Policy.** When a parent closes, Temporal "propagates Cancellation Requests or Terminations to Child Workflow Executions depending on the Child's Parent Close Policy." A child that is stuck because its parent was terminated but used the `Abandon` close policy is a different shape — the child is independently running. + +## Pending signals, cancellations, and updates + +A Workflow's handling of Signals and Updates is Workflow-code-directed — the SDK's `await` / `workflow.wait_condition` APIs block the workflow until the condition becomes true. "Pending Signals" (server-side) are distinct from "Signals the Workflow is waiting for" (SDK / code-side): + +- **`SignalExternalWorkflowExecutionInitiated`** is an Event type recorded when *this* Workflow signals *another*. The terminal events are `ExternalWorkflowExecutionSignaled` (success) and `SignalExternalWorkflowExecutionFailed`. A stalled outgoing signal is a pending operation from the caller's perspective. +- **`WorkflowExecutionSignaled`** is the Event type recorded when an external party (client, another Workflow, `temporal workflow signal`) sends a signal *to* this Workflow. If a Workflow is blocked waiting on a signal that has never been sent, no `WorkflowExecutionSignaled` event will appear — *absence* of the event is the symptom. +- **`WorkflowExecutionUpdateAcceptedEvent`** / **`WorkflowExecutionUpdateCompletedEvent`** record Update lifecycle. A pending Update (accepted but not completed) keeps the Workflow responsible for resolving it. + +Things to check for "waiting on a signal" stuck-ness: + +- **Inspect the call stack.** `temporal workflow stack` runs the `__stack_trace` Query and returns "a stack trace of the threads and routines currently in use by the Workflow for troubleshooting." This works only when a Worker is running and available to respond to queries. The stack trace will show which `await` or `wait_condition` the Workflow is blocked on, which identifies the signal/update name. +- **Send a test signal** to confirm the handler is wired correctly: `temporal workflow signal --workflow-id YourWorkflowId --name YourSignal --input '{"key": "value"}'`. +- **Check the sender side.** If the signal is being sent but not received, three app-code patterns account for most cases. All three are diagnosable from describe/show output: + + 1. **Stale Run ID after Continue-As-New.** The sender targets a specific Run ID that belongs to a now-closed run (status `ContinuedAsNew`). The server accepts the signal — it is valid for the closed run — but the new run never sees it because it has a different Run ID. Symptom: `WorkflowExecutionSignaled` appears in the *old* run's history but not the current run's. Fix: the sender should omit the Run ID (letting the server route to the latest run in the chain) or re-resolve the Workflow ID to the current Run ID before signaling. + 2. **Continue-As-New fires before in-flight handlers finish.** The Workflow calls Continue-As-New without waiting for `all_handlers_finished` (Python) / the SDK-equivalent guard. In-flight signal and update handlers on the closing run are silently dropped — they never run to completion, and the new run has no record of them. Symptom: the sender sees success (the signal was delivered), but the Workflow behaves as if the signal never arrived. The old run's history shows `WorkflowExecutionSignaled` followed by `WorkflowExecutionContinuedAsNew` with no evidence the handler acted. Fix: the Workflow must await the all-handlers-finished condition before issuing Continue-As-New. + 3. **Handler name mismatch.** The sender uses a signal name that doesn't match any registered handler (e.g., `order_shipped` vs. `orderShipped`). The server accepts the signal — it does not validate handler names — so `WorkflowExecutionSignaled` appears in history, but no handler runs. The Workflow blocks forever on a condition that will never be satisfied. Symptom: history contains the `WorkflowExecutionSignaled` event with the signal name, but `temporal workflow stack` shows the Workflow still blocked on the same `wait_condition`. Fix: align the signal name in the sender with the name registered in the Workflow Definition. + +The workflow has per-Workflow-Execution pending-signal limits documented in [Pending-operation per-Workflow limits](#pending-operation-per-workflow-limits). + +## Pending Nexus Operations + +A pending Nexus Operation is a Nexus call issued from this Workflow that has been scheduled (`NexusOperationScheduled`) and may have been started (`NexusOperationStarted`) but has not reached a terminal event (`NexusOperationCompleted`, `NexusOperationFailed`, `NexusOperationTimedOut`, `NexusOperationCanceled`). + +`temporal workflow describe` surfaces pending Nexus Operations with these fields (from the docs' own example): `Endpoint`, `Service`, `Operation`, `OperationToken`, `State`, `Attempt`, `ScheduleToCloseTimeout`, `NextAttemptScheduleTime`, `LastAttemptCompleteTime`, `LastAttemptFailure`, and — when the per-destination circuit breaker is open — `BlockedReason`. + +Things to check: + +- **`State`.** The docs' two text examples show `BackingOff` (retrying after a retryable error) and `Blocked` (circuit breaker open for the destination pair). `Blocked` with a `BlockedReason` of "The circuit breaker is open" means the handler side (or the network path to it) has produced enough consecutive timeouts / retryable errors to trip the breaker; handler Workers must recover before pending Operations resume. +- **`LastAttemptFailure`.** Non-retryable errors resolve the Operation with a `NexusOperationFailed` / `NexusOperationTimedOut` / `NexusOperationCanceled` event. Retryable errors surface in the pending Operation with `LastAttemptFailure` filled in. +- **Pending Callbacks.** Separate from Pending Operations, Nexus completion callbacks have their own pending view, surfaced as `Callbacks: N` with `URL`, `Trigger`, `State`, `Attempt`, `RegistrationTime`. + +## Pending Workflow Task and WorkflowTaskFailed loops + +A Pending Workflow Task is a Workflow Task scheduled for this Workflow but not yet completed by a Worker. The JSON describe example shows the shape: `pendingWorkflowTask.state`, `pendingWorkflowTask.scheduledTime`, `pendingWorkflowTask.originalScheduledTime`, `pendingWorkflowTask.attempt`. + +### Distinguishing pending Workflow Task shapes + +- **`pendingWorkflowTask.attempt` = 1, recently scheduled, no `WorkflowTaskFailed` in history.** A Worker has not yet picked up the task. If this persists, route to [worker-health.md](worker-health.md): confirm pollers on the Workflow Task Queue. +- **`pendingWorkflowTask.attempt` > 1 and the Event History contains repeating `WorkflowTaskFailed` events.** The task is being retried server-side after worker-reported failures. +- **`pendingWorkflowTask` absent while history has a recent `WorkflowTaskCompleted`.** The last Workflow Task completed cleanly — the Workflow is not blocked on running Workflow code, it is blocked on an awaitable from that completed task (Activity, Child, Signal, Update, Timer, Nexus). + +### The WorkflowTaskFailed retry loop + +`WorkflowTaskFailed` "indicates that the Workflow Task encountered a failure." The event's `workflow_task_failed_event_attributes` include `scheduled_event_id`, `started_event_id`, `failure`, `identity`, and (for Reset-initiated failures) `base_run_id`, `new_run_id`, `fork_event_version`, and `binary_checksum`. The `failure` field carries the detail of the failure. + +Each `WorkflowTaskFailed` corresponds to a value of the `WorkflowTaskFailedCause` enum (referenced in Event attributes as `cause`). The error reference enumerates the causes; these are the ones relevant to stuck-ness: + +| Cause (in the `errors.mdx` reference) | Interpretation | Route | +|---|---|---| +| Nondeterminism Error | Replay diverged from recorded history | [non-determinism.md](non-determinism.md) | +| Workflow Worker Unhandled Failure | An unhandled failure (panic, exception) from the Workflow Definition | Worker logs for stack trace; fix and redeploy | +| Unhandled Command | The Workflow attempted to close without handling new Events; can occur under Signal load | Drain signal channel; inspect Worker logs | +| Pending Activities Limit Exceeded | Workflow reached the per-Workflow pending-Activity cap | [Pending-operation per-Workflow limits](#pending-operation-per-workflow-limits) | +| Pending Child Workflows Limit Exceeded | Reached pending-Child cap | Same as above | +| Pending Signals Limit Exceeded | Reached pending external-Signal cap | Same as above | +| Pending Nexus Operations Limit Exceeded | Reached pending Nexus Operation cap | Same as above | +| Bad Search Attributes | Invalid Search Attributes cause Workflow Tasks "to continue to retry without success." | Fix Workflow code or attribute definitions | + +### Detecting WFT-failure loops at scale with Task Issue detection + +Rather than inspecting individual Workflows, use the `TemporalReportedIssue` search attribute to find all Workflows currently experiencing Workflow Task failures: + +```bash +temporal workflow list \ + --query "TemporalReportedIssue IS NOT NULL" \ + --namespace +``` + +The `TemporalReportedIssue` search attribute is automatically set by the server when it detects a Workflow Task issue (e.g., repeated `WorkflowTaskFailed` events). This surfaces problems proactively without needing to inspect each Workflow individually. + +### Why a Workflow stays Running through WFT-failure loops + +Workflow Tasks do not fail the Workflow Execution. The server retries failed Workflow Tasks so that once the Worker code is fixed and redeployed, the Workflow resumes. This is by design: the Event History is durable, and a bad deploy that causes a WFT to fail can be rolled back or patched without losing the Workflow's progress. The visible effect is that the Workflow stays `Running`, the Event History grows a tail of `WorkflowTaskFailed` events, and the attempt count climbs. + +## Timer-based waits + +Timers are persisted server-side: "even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service become available, the call that is awaiting the Timer in your Workflow code will resolve." A Workflow that is waiting on a timer is not stuck; it is asleep. Fire time is deterministic from history. + +The `TimerStarted` event carries `timer_id`, `start_to_fire_timeout`, and `workflow_task_completed_event_id`. The fire time is the `eventTime` of the `TimerStarted` event plus `start_to_fire_timeout`. If that fire time is in the future, the Workflow is working as designed; report the resume time rather than treating the Workflow as stuck. + +"The duration of a Timer is fixed, and your Workflow might specify a value as short as one second or as long as several years." Multi-year timer waits are legitimate. + +The terminal events are `TimerFired` (with `timer_id` and `started_event_id`) and `TimerCanceled`. + +## Pending-operation per-Workflow limits + +The server enforces per-Workflow caps on the number of incomplete operations of each kind. Exceeding a cap fails the Workflow Task that attempted to add another operation of that kind — the Workflow Execution does not close, but the WFT will fail and retry until the pending count drains. + +| Operation | Default limit | Source | +|---|---|---| +| Pending Activities | 2,000 | | +| Pending Child Workflow Executions | 2,000 | | +| Pending Signals (external) | 2,000 | | +| Pending Cancel Requests | 2,000 | | +| Pending Nexus Operations | 30 | | + +The Event-History caps are a separate ceiling that terminates the Workflow when crossed: "The Workflow Execution is terminated when the Event History exceeds 51,200 Events, contains more than 2000 Updates, or contains more than 10000 Signals." A stuck-looking Workflow approaching these limits will not stay Running — it will close with `Terminated` when it crosses them. + +## Recovery commands + +These commands alter the Workflow's state or flow. Use them once the diagnosis identifies the shape of the stuck-ness. + +### `temporal workflow signal` + +Send a Signal to unblock a Workflow that is waiting on one. + +```bash +temporal workflow signal \ + --workflow-id YourWorkflowId \ + --name YourSignal \ + --input '{"key": "value"}' +``` + +### `temporal workflow terminate` + +Terminate a Workflow that cannot be recovered. Terminal events become the closing event of the Execution History. "Workflow code cannot see or respond to terminations. To perform clean-up work in your Workflow code, use `temporal workflow cancel` instead." + +```bash +temporal workflow terminate \ + --workflow-id YourWorkflowId \ + --reason YourReason +``` + +### `temporal workflow cancel` + +Request cancellation. A `WorkflowExecutionCancelRequested` event is appended, and the Workflow runs any cleanup paths its code supports. + +```bash +temporal workflow cancel \ + --workflow-id YourWorkflowId +``` + +### `temporal workflow reset` + +Reset the Workflow to a point in its Event History so it can resume from there without losing progress up to that point. Valid reset points per the Event encyclopedia are "`WorkflowTaskStarted`, `WorkflowTaskCompleted`, `WorkflowTaskTimedOut`, and `WorkflowTaskFailed`." A Reset terminates the current Run and creates a new Run with history copied up to the reset point. + +```bash +temporal workflow reset \ + --workflow-id YourWorkflowId \ + --event-id YourLastEvent +``` + + To reset to where the current Run continued-as-new: + +```bash +temporal workflow reset \ + --workflow-id YourWorkflowId \ + --type LastContinuedAsNew +``` + + The `--type` flag's `LastContinuedAsNew` value appears in the reset example. The batch-reset guidance in the same section mentions `FirstWorkflowTask`, `LastWorkflowTask`, and `BuildId` as the only `--type` values permitted for batch resets. A companion flag `--reapply-type` controls which Events are reapplied after the reset point; accepted values are `Signal, None`. + +Because the new Run replays history only up to the reset point, everything after it +*happens again* — +Activities with external side effects re-execute, so a reset past a payment, an +email, a provisioning call, or a non-idempotent write performs it a second time. +`--reapply-type None` suppresses Signal reapplication; it does not prevent Activity +re-execution. Establish where the side effects are relative to the candidate reset +point, say so in the proposal, and prefer the latest reset point that still clears +the problem. + +### `temporal workflow pause` / `unpause` + +Experimental. + +```bash +temporal workflow pause --workflow-id YourWorkflowId --reason YourReason +temporal workflow unpause --workflow-id YourWorkflowId --reason YourReason +``` + +### `temporal activity pause` / `unpause` / `reset` + +For stuck pending *activities*, Activity Operations are the surgical tool: pause to stop retries, unpause to resume, reset to clear attempt state. These operations are in Public Preview (Server v1.28.0+). Activity Operations don't produce Event History events — they leave no audit trail beyond the current describe output. + +## Quick routing + +| Evidence | Go to | +|---|---| +| `workflowExecutionInfo.status` is not `Running` | [Workflow Execution Status values](#workflow-execution-status-values) — the workflow is Closed, not stuck | +| Pending activity present, no attempts yet, Event History shows `ActivityTaskScheduled` only | [worker-health.md](worker-health.md) — check pollers on the Activity's Task Queue | +| Pending activity present, attempts climbing, `LastAttemptFailure` reported | [Pending activities](#pending-activities) | +| Pending child workflow present | [Pending child workflows](#pending-child-workflows) — recurse into the child | +| Waiting on a signal that never arrives | [Pending signals, cancellations, and updates](#pending-signals-cancellations-and-updates); use `temporal workflow stack` | +| Pending Nexus Operation with `State: Blocked` or `BackingOff` | [Pending Nexus Operations](#pending-nexus-operations) | +| `pendingWorkflowTask.attempt` > 1, `WorkflowTaskFailed` events accumulating | [Pending Workflow Task and WorkflowTaskFailed loops](#pending-workflow-task-and-workflowtaskfailed-loops) | +| `TemporalReportedIssue` search attribute set on Workflow(s) | [Detecting WFT-failure loops at scale](#detecting-wft-failure-loops-at-scale-with-task-issue-detection) | +| WFT failure cause is `NonDeterministicError` | [non-determinism.md](non-determinism.md) | +| Last non-bookkeeping event is `TimerStarted`, fire time is in the future | [Timer-based waits](#timer-based-waits) — not stuck | +| WFT fails with `Pending Activities Limit Exceeded` / `Pending Child Workflows Limit Exceeded` / `Pending Signals Limit Exceeded` / `Pending Nexus Operations Limit Exceeded` | [Pending-operation per-Workflow limits](#pending-operation-per-workflow-limits) | +| Need to reproduce the failure locally under a debugger | [replay.md](replay.md) | +| `temporal workflow describe` itself fails (cannot reach / authenticate / authorize) | [connectivity.md](connectivity.md), [certificates.md](certificates.md), [authentication.md](authentication.md), [rate-limits.md](rate-limits.md) | + +See the whole-stack picture in [diagnostic-ladder.md](diagnostic-ladder.md). diff --git a/plugins/temporal/skills/temporal-serverless/SKILL.md b/plugins/temporal/skills/temporal-serverless/SKILL.md new file mode 100644 index 0000000..54ece14 --- /dev/null +++ b/plugins/temporal/skills/temporal-serverless/SKILL.md @@ -0,0 +1,232 @@ +--- +name: temporal-serverless +description: 'Deploy and operate Temporal Workers on serverless compute (AWS Lambda) driven by the Worker Controller Instance (WCI). Use when the user mentions: "serverless worker", "Temporal serverless", "Worker Controller Instance", "WCI", "deploy Temporal worker on Lambda", "Lambda packaging", "Lambda timeout", "WCI inspection", "CloudFormation Temporal".' +version: 0.6.0 +disable-model-invocation: true +--- + +# Skill: temporal-serverless + +## Overview + +This skill helps users deploy and operate Temporal Workers on serverless compute. Instead of a long-lived process, Temporal invokes the Worker on demand through the Worker Controller Instance (WCI); the Worker processes available Tasks and shuts down, scaling to zero when idle. The skill produces Worker code, deployment configuration, connection configs, and packaging steps for the chosen SDK, and walks users through troubleshooting when serverless Workers aren't picking up Tasks. + +## Supported compute providers + +| Cloud provider | Compute service | Support | Reference directory | +|---|---|---|---| +| AWS | Lambda | Supported — Public Preview, open to all Temporal Cloud customers | `references/aws-lambda/` | +| GCP | Cloud Run | Not supported | — | + +Only a provider marked Supported is covered. If a request names another, say it is not supported and stop; do not adapt a supported provider's material to it. **Never let the provider be an unstated assumption:** when the request does not name one, it is confirmed in the step 1 questions, not silently defaulted. + +Every supported provider's directory carries the same layout — `setup.md`, `iam.md`, `versioning.md`, `diagnostics.md`, `observability.md`, `self-hosted.md`. Paths below are written `references//…`; substitute the directory from the table. Provider-specific commands, templates, permissions, and defaults live there — this file stays at the workflow level. When a step needs concrete commands, go to the reference file named at the end of that step. + +**Public Preview is not GA.** The APIs are still evolving and may change: pin SDK and CLI versions for anything long-lived, and read the installed package's actual API surface rather than writing from memory. + +## Deployment workflow + +Follow these steps in order. Each step is provider-neutral; the concrete commands, templates, and options live in the reference file named at the end of the step. + +**Open a new deployment with a plain-language summary of the run.** Before the step 1 questions, tell the user in a few sentences what is about to happen: that this creates real resources in their cloud account which cost money for as long as they exist; that you will ask about a handful of things, then show an exact list of what you are about to create and wait for approval, and that nothing is created before that approval; that the middle of the run is unattended; and that it ends with a Workflow they can watch execute, an inventory of everything created, and an offer to remove it all. Name the five stages below in ordinary words. Do not explain Temporal or serverless compute; keep it short enough to read at a glance. + +**Lay it out as bullets, with the five stages as sub-bullets under "How it goes" — one stage per line, never chained into a single run-on bullet.** Follow this shape: + +> Here's what's about to happen, before I ask anything: +> +> - This creates real resources in your cloud account — the compute unit that runs your Worker, roles, an infrastructure stack, logs. They're live and billable for as long as they exist. +> - **How it goes.** Five stages: +> - **Scope** — a handful of questions, below. +> - **Access** — check credentials and permissions on both sides, then show you an exact list of what I'm about to create and wait for your approval. +> - **Build** — write, package, deploy the Worker. +> - **Connect** — bind the Task Queue, set the version current. +> - **Verify and hand back.** +> - Nothing gets created before you approve that list. After approval the middle stretch runs unattended. +> - At the end you get a Workflow you can watch execute, a full inventory of everything created, and an offer to remove it all. + +**Write the summary provider-neutral, because at that point you do not know the provider.** It is one of the things step 1 asks. Say "your cloud account", never the name of a provider you have not been told. The same applies to the account, Namespace, and region: if a cheap read-only call has already told you (see step 1), name what you actually found; otherwise leave it out rather than filling it in with a plausible guess. + +Skip the summary for troubleshooting, inspection, and configuration-change tasks. Someone whose Worker is not being invoked does not need an overview of a deployment they have already done. + +**Then track the run on a checklist, and reprint it every time a step completes.** The eight steps group into the five stages below. Create one item per step, grouped under its stage, and build the checklist as soon as step 1's answers land, so items can name the confirmed provider and the agreed prefix instead of hedging. + +**Reprint the whole checklist at each step boundary — not just the item that changed, and not a sentence saying the stage is done.** Mark finished items ✅, the one you are starting ⏳, and the rest ⬜. Use the bare marker with nothing in front of it — `✅ Confirm SDK`, not `- [x] Confirm SDK` — and put each item on its own line. A narrated "Access complete, now Build" is not a substitute: it says where you are but not what remains, and the user cannot see it without scrolling back to a checklist printed twenty commands ago. Reprint during Scope and Access too — those stages end in a user decision, and the reprint is what shows the decision landed and what it unblocked. + +Where the harness has a todo list, use it *in addition to* the printed checklist, not instead of it. It is not part of the transcript the user reads back. + +**Word each item as plain language about what happens, not as a compressed step title,** and name both sides concretely — the confirmed compute provider and Temporal, never "both sides." Follow this shape: + +> **Scope** +> ✅ Confirm SDK (Go), compute provider (AWS Lambda), Namespace (``), and naming prefix (``) +> +> **Access** +> ⏳ Check credentials and permissions on AWS and on Temporal, then show the exact list of resources to be created and wait for your approval +> +> **Build** +> ⬜ Write the Worker against the installed package's real API +> ⬜ Cross-compile, package, deploy the compute unit, wait for it to report ready +> +> **Connect** +> ⬜ Create the role Temporal assumes to invoke the Worker +> ⬜ Register the Worker Deployment Version, confirm the validation invocation bound the Task Queue, set it current +> +> **Verify and hand back** +> ⬜ Start a Workflow and confirm it executes, from both the Temporal side and the provider's logs +> ⬜ Deliver the inventory of everything created, then offer teardown + +| Stage | Steps | Complete when | +|---|---|---| +| Scope | 1 | SDK, compute provider, Namespace, and naming prefix are all confirmed by the user. | +| Access | 2 | Compute provider and Temporal both authenticated, permissions confirmed, and the list of resources to create approved. | +| Build | 3–4 | The compute unit is deployed and reports ready, built for the architecture it runs on. | +| Connect | 5–6 | The Task Queue is bound and the version is current. | +| Verify and hand back | 7–8 | A Workflow completed, two independent signals agree, the inventory is delivered, and teardown has been offered. | + +**A step is complete when its verification passed — not when its command exited zero.** Several commands in this workflow exit clean having done nothing: the traffic-shifting and key-revocation commands no-op when their confirmation prompt goes unanswered, and providers return from create and update calls while the resource is still settling. Check an item off against state you read back, not against an exit code. When a step's verification fails, say which step you are on and what it is blocked on rather than moving down the list. + +1. **Scope the task.** Identify the SDK language (Go, Python, or TypeScript), the deployment target (Temporal Cloud or self-hosted — self-hosted has its own server prerequisites), the compute provider, and whether this is a new setup, a configuration change, or troubleshooting. Confirm the deployment target is compatible with the chosen provider — see "A Namespace on the target cloud provider is required" under Provider-neutral principles. Ensure a Temporal client/CLI is available and authenticated to the target. Each changes the specifics. → `references/concepts.md` for what the user is building; `references//setup.md` for the compatibility and client-setup details. + + **Put the compute provider in that batch of questions as a confirmable default, not a free choice.** Pre-select the supported provider from the table above and carry its support status in the option's description. The user confirms rather than chooses, so it costs no extra turn, but the provider is never something they were assumed into. Skip the question only when the request already names a provider. Do not restate any of this in a paragraph before the questions; the option description is where it belongs. + + **Let the user pick the Namespace from a list; never make them retype one.** Namespace names are long and error-prone — a generated suffix on an account ID, `-.`. Where control-plane access is available, `tcld namespace list` returns the full Namespace objects, so one call gives every name with its region — and a region ID is provider-prefixed (`aws-…`, `gcp-…`), so the same response tells you each Namespace's provider. Only the prefix carries meaning; the region itself imposes no constraint. + + Present it like this: + + - **Offer the eligible Namespaces as the options**, each labelled with its region. + - **Summarize the ineligible ones in a single line** — "you also have 2 Namespaces on \, which this skill does not support" — rather than listing them individually or hiding them. A user who knows they have a Namespace and cannot find it in the list concludes the tool is broken; one line keeps them informed and explains the constraint. + - **Name the account you are listing from and confirm it is the intended one** before showing anything. A stale credential lists a real account that is not the one the user means to deploy into, and every option under it looks authoritative. + - **If more Namespaces are eligible than the question format can hold, print the labelled list and ask the user to name one.** Do not silently show only the first few. + + This also settles the compute-provider answer, since a Namespace can only be served by compute on its own cloud provider — so a mismatch is caught here rather than at connection time, several steps later. + + **Degrade gracefully if `tcld` is not authenticated.** Ask the user for the Namespace name rather than stopping to fix the login — they can copy it from the Cloud UI, where it appears on the Namespace page and in the URL. Ask for its region in the same batch of questions: the name alone does not tell you the provider, and a mismatch missed here surfaces at connection time instead. + + **Never source a Namespace, account, or resource identifier from shell history.** History is stale by construction — it is full of last quarter's accounts — and reading it to guess a deployment target produces confident, wrong answers. Take identifiers from the user or from an authenticated API call, and nowhere else. + + **Agree a resource-naming prefix in this same batch of questions, and propose a default so the user can accept without thinking about it.** Assume the account and the Namespace are shared — unprefixed names like `temporal-serverless-worker` collide with, or quietly shadow, another team's deployment. Naming is not a late cosmetic choice you can patch on the provider side: the deployment name, build ID, and Task Queue are compiled into the Worker binary, so changing them after step 3 means editing code, rebuilding, repackaging, and cleaning up whatever was already created under the old names. Once agreed, apply the prefix to everything you create on both sides — compute unit, roles, infrastructure stacks, log groups, deployment name, and Task Queue. + + **The prefix you propose must be identifying** — derived from the user, their team, or the project. A generic word like `demo`, `test`, or `temporal` collides about as readily as no prefix at all, so never offer one as the safe choice. + + Offer exactly two options plus the free-text escape: the identifying prefix, and "no prefix" — some users genuinely own the account. Do not offer a second prefix string; the consequential choice is prefix versus none, and anything else goes in free text. When you offer "no prefix," say what it risks in the same breath: unprefixed names can collide with or shadow an existing deployment, and that surfaces as another team's Worker behaving oddly rather than as an error you will see. + +2. **Confirm you can make the required changes — before making any.** Determine which credentials are available (for the compute provider and for Temporal) and confirm the active identity actually has permission to make the changes the task needs — creating or updating compute resources, creating roles, registering deployment versions. Verify *both* sides: the compute provider AND Temporal access. Do not run account-mutating commands and let them fail partway. **If access is missing or unconfirmed, stop and ask the user how they want to proceed** — extend their identity's permissions, have an administrator make the change and hand back the result, or generate the commands for the user to run under a privileged identity. Changing a user's cloud account is consequential; confirm authorization and the preferred method first. → `references//iam.md` (exact permissions, compute-provider preflight) and `references//setup.md` (Temporal connection preflight). + + **Classify an authentication failure before acting on it — "not signed in" and "not permitted" have different fixes.** A failed preflight does not automatically mean the bottom row of the table below. An absent or expired credential is usually recoverable in this session, in under a minute. A caller that resolves but is denied a specific action is a real permissions problem. Never collect credentials in the conversation: no interactive credential-configuration wizards, and never ask the user to paste access keys, API keys, or session tokens. → `references//iam.md` (credential recovery). + + **Then ask the user which way they want to go, and do not choose for them:** + + - **Fix the CLI** — you run the login flow, surface the verification URL for them to open, wait for it to complete, re-run the preflight, and continue with full automation. + - **Work in the browser** — the user makes the changes in the Cloud UI and their cloud provider's console while you give the instructions step by step, naming the exact path for each one, and they report the result back. + + Both paths reach the same end state, so present them as equals rather than as a preference and a fallback — every control-plane step in this workflow exists in the Cloud UI (see `references//setup.md`). Ask once, then commit to the answer — do not re-offer the login at every subsequent step, and never start an identity-provider login as a silent side effect of a preflight. + + Adapt to what is available — the skill is valuable at every level: + + | Compute-provider access | Temporal access | Behavior | + |---|---|---| + | Authenticated | Authenticated | Full workflow — run commands, verify results, register deployment versions. | + | Authenticated | None | Deploy compute infrastructure; walk the user through the Temporal steps in the Cloud UI, or generate the commands for them to run. | + | None | Authenticated | Write Worker code and configs; walk the user through the compute steps in their provider's console, or generate the deploy commands; run Temporal commands and verify WCI state. | + | None | None | Write Worker code, deploy templates, permission policies, connection configs, packaging scripts; provide all commands with placeholder values. | + + **Do not self-select a row.** Drop to a lower one only after the choice above has been put to the user and the browser path chosen, or the login attempted and failed. When you hand off a runbook, say the offer stands — if the user authenticates and comes back, take the work over rather than leaving them to run the steps by hand. + + **Before the first account-mutating command, list what you are about to create — with final names — and get approval.** Name the target account and region, then every resource: compute unit, execution role, infrastructure stack, log group, deployment name, and Task Queue. Say plainly that they are live and billable. This is the mirror of the inventory in step 8, and it is worth more here than there: it makes the naming prefix concrete while changing it is still free, and the deployment name, build ID, and Task Queue become expensive to change once step 3 compiles them into the Worker. Skip it only when nothing will be created — a troubleshooting or inspection task. + +3. **Author the Worker.** *Install the SDK's serverless Worker package before writing any code* — it is often shipped separately from the main SDK, with its own version line, so having the base SDK installed does not mean it is importable. Then read the installed package's actual API surface and write against that; these are Public Preview APIs that drift between versions, and generating code from memory costs a build cycle. Every Workflow must declare a versioning behavior (`Pinned` or `AutoUpgrade`), per-Workflow or as a Worker-level default — code without it fails at runtime. → `references/sdk-configuration.md` (package, install, entry point, tuned defaults) and `references//setup.md` (install commands, API-inspection recipes, handler shape). + +4. **Package and deploy the compute unit.** Build and package per SDK, deploy the compute unit, and set the invocation deadline high enough for the Worker to start, connect, register the Task Queue, and shut down gracefully. Match the build's target architecture to the deployed compute unit's — a mismatch fails only at invocation time, not at build time. After a create or update, wait for the compute unit to reach a ready state before the next step; providers return from these calls while the unit is still settling. → `references//setup.md`. + +5. **Grant Temporal permission to invoke the Worker.** Configure the compute provider's access so Temporal can invoke and inspect the Worker. This access is separate from the compute unit's own execution role — do not confuse the two. Two things to get right before you create anything: (a) this grant is **shared, account-wide infrastructure** that a previous deployment may already have created — look for an existing one and extend it to cover your new Worker rather than creating a parallel copy, and never delete or repurpose one you did not create without asking; (b) scope the grant so that *future* immutable builds are covered, not just today's — a grant pinned to one build breaks the next release in a way that surfaces later as an unrelated-looking invocation failure. → `references//iam.md`. + +6. **Register the Worker Deployment Version, verify the validation invocation, then set it current.** Create the Worker Deployment Version with the compute provider configured; the deployment name and build ID must exactly match the values in the Worker code. Creating it triggers one validation invocation — **check that it bound the Task Queue before going further.** If the Task Queue is bound, the permission grant, package, config, and deadline are all provably correct, and any later failure is downstream; if it is not, setting the version current will not fix it. Then set it current: through the UI this happens automatically, through the CLI it is a separate step, without which Tasks never route to the version. → `references//setup.md`. + +7. **Verify.** Start a Workflow on the Task Queue and confirm Temporal invokes the Worker — check the Workflow history in the Temporal UI and the compute provider's logs. If it does not progress, → `references//diagnostics.md`. + +8. **Hand back the inventory first; offer teardown as the closing note.** The order is inventory → offer, never the reverse. Close with what now exists — compute unit and published build identifiers, roles, infrastructure stacks, region, deployment name and build ID — and what the run actually did, including anything you worked around or deviated from. Say plainly that it is live and billable. These names are only knowable from the run that created them, and reconstructing them later means scanning the user's account. + + **Do not write a teardown script before the user asks for one.** Generating it unprompted buries the inventory under a file they did not request, and the inventory is what they need in order to decide. End with a single line — *"Let me know if you want a teardown script to remove these resources"* — and stop there. Write the script, or run the teardown, when they take you up on it. → `references//setup.md` (Teardown). + +## Working practices + +How to move through the workflow above. + +- **Say what a command will do in your own text, above the command.** The user sees a collapsed "Ran 6 shell commands" in the transcript, not the commands themselves, so an unannounced batch is opaque at exactly the moments that matter. State it in one line before the tool call: what it does and to what — the resource, and the account or Namespace it touches. + + **The tool's own description field does not count.** It renders at the bottom of the command block, underneath the command it is describing, where the user has to go looking for it. The summary belongs above the block, as ordinary message text. + + **Set it apart on its own line so it is visibly not narrative prose** — bold, and nothing else on the line: + + > **Checking for the Temporal CLIs, AWS credentials, and which AWS account and Temporal account they resolve to** + > + > **Creating the invocation role Temporal assumes, with a generated External ID** + + For anything that creates, updates, or deletes, name the resource and the target account or Namespace explicitly — an approval prompt should arrive with its justification already on screen, not after it. +- **Read the current state instead of recalling it.** Check the installed package's API, the CLI's own `--help` for the flags you are about to pass, the compute unit's reported state, and the CLI version. Each of these has drifted in practice: a Public Preview SDK whose fields moved, a CLI too old to have the serverless subcommand at all, a resource that reports success while still settling. +- **Do not chain `cd` with commands that create or modify files.** A compound `cd && ` triggers a manual approval prompt no matter how the user's permissions are configured, so scaffolding a project this way asks for approval on every run. Use absolute paths, or the tool's own directory flag (`go -C …`), and rely on the shell's working directory persisting between calls — the `cd` buys nothing and costs a prompt. Keep the command count down for the same reason: one `go get` covering both packages beats two. +- **Verify each step before building the next on top of it.** Compile the Worker before packaging it, confirm the package's target architecture before uploading, wait for the compute unit to be ready before publishing a build, and confirm the Task Queue is bound before shifting traffic. Deployment failures here surface far from their cause — an architecture or dependency mismatch appears only at first invocation, and a first-invocation failure appears as "the Worker is never invoked", several steps later. +- **When something fails, read the actual error before changing anything.** Fetch the failure reason from the provider (deployment events, logs, status fields) and fix that. Do not retry the same command with variations, and do not start editing permissions or trust policies on the theory that the problem might be access — most first-invocation failures are not permission problems, and some failures are on Temporal's side and will reproduce no matter what you change. +- **Treat the user's account as shared and pre-existing.** Assume other deployments, roles, and stacks are already there. Look before creating, extend rather than duplicate, and never delete or repurpose something you did not create without asking. When you do work around existing infrastructure — a different name, a reused role — say so explicitly in your summary rather than leaving it as a silent deviation. +- **Confirm the end state from two independent signals.** A Workflow that completes in the Temporal UI *and* the Worker's own logs showing startup, Task Queue registration, and Task execution. One signal alone can mislead: a system Workflow that exists and is running proves nothing about invocation health, and a command that exits zero may have done nothing at all if it was waiting on a confirmation prompt. +- **Account for what you created.** Keep the inventory as you go rather than reconstructing it at the end, say plainly that the resources are live and billable, and offer to tear them down (step 8). + +## Never create or manage the WCI + +Temporal creates the WCI automatically once a Worker Deployment Version has a compute provider. You never create, start, or manage it. A WCI that exists or is running is *not* evidence that invocation works — it continue-as-news and keeps running even while its Activities fail. Diagnose from Temporal's own signals: read the WCI Workflow history and look for Activity failures. Do not enumerate compute resources across regions or scan the account to reverse-engineer state. → `references/concepts.md`, `references//diagnostics.md`. + +## Provider-neutral principles + +Surface these early — they apply regardless of compute provider: + +- **A Namespace on the target cloud provider is required.** A Serverless Worker runs only on the cloud provider that hosts its Temporal Cloud Namespace — there is no cross-cloud pairing. Confirm the user has a Namespace on the provider they intend to run compute on *before* building anything; without one, the work stops there and they need either a Namespace on that provider or a different provider. A mismatch is not caught at deploy time — it fails later, at connection time. **Regions do not have to match:** a Namespace in one region can drive a compute unit in another, so never tell a user to move or re-create a Namespace to line up regions. +- **Use `tcld` for every Temporal Cloud control-plane operation** — accounts, Namespaces, API keys, users, service accounts. Do not use the unified CLI's `temporal cloud …` subcommands for them. Worker Deployments and Workflows are *not* control-plane operations: they live on the Namespace frontend, have no `tcld` equivalent, and use `temporal worker deployment …`. → `references//setup.md`. +- **Versioning behavior is mandatory.** Every Workflow needs `Pinned` or `AutoUpgrade`, or the Worker sets a default. +- **Deployment name and build ID must match exactly** between the Worker code and the Worker Deployment Version. A mismatch causes an invocation loop (Temporal invokes → Worker polls with the wrong version → Task not processed → invoke again). Signature: rapid repeated invocations with no Workflow progress. +- **Set the invocation deadline high enough.** Providers often default to a very short timeout. If the first invocation times out before the Worker registers the Task Queue, the binding is never created and the Worker is never invoked again. → `references//setup.md` for the exact default. +- **Use an immutable, versioned build per Build ID in production.** Pointing the provider at a mutable "latest" target lets code change under in-flight Workflows and cause non-determinism errors, even for Pinned Workflows. Keep a 1-to-1 mapping between each Build ID and one immutable build. → `references//versioning.md`. +- **Tune the timeout triple together for long-running Activities:** (1) worker stop timeout > longest Activity runtime, (2) shutdown deadline buffer > worker stop timeout + shutdown hook time, (3) invocation deadline > longest Activity runtime + shutdown deadline buffer. Raising one alone does not help. If the longest Activity exceeds half the maximum invocation deadline, recommend Activity Heartbeats. → `references/concepts.md`, `references/sdk-configuration.md`. +- **Eager Activities are always disabled** — serverless invocations don't maintain persistent connections. Don't suggest them as an optimization. +- **Activities are bounded by the invocation limit** (minus the shutdown deadline buffer); Workflow duration is unbounded and can span many invocations. Flag Activities that approach the provider's limit early. → `references/concepts.md`. +- **Mixed serverless + long-lived Workers on one Task Queue:** do not enable dynamic scaling on the long-lived Workers — the two groups can't coordinate scaling and will cause unnecessary invocations. +- **Secrets belong in a secret store**, not plaintext environment variables. Provider docs and quickstarts commonly pass the API key or TLS key as a plaintext environment variable; that is acceptable in a throwaway development walkthrough *only if you say so explicitly at the time*. Anything the user describes as production, shared, or long-lived gets the secret store, loaded at cold start. Either way, keep key material out of shell history and command echoes. +- **Both CLIs prompt for confirmation before mutating state, and their flags differ.** Setting the current or ramping version, and revoking an API key, all ask interactively; run non-interactively without the flag, the command exits having done nothing, which reads as success. `temporal worker deployment …` takes `--yes`; `tcld` takes the global `--auto_confirm`. Pass the right one in scripts, CI, and agent shells, and confirm the resulting state rather than trusting the exit code. → `references//setup.md`. + +## Troubleshooting + +Start by determining whether the Worker is being invoked at all. Then, in priority order: (1) **Validate Connection** in the Temporal UI (Workers > Deployments > select > Actions > Validate Connection) — checks credentials, role assumption, and reachability in one step; (2) check whether the version's **Task Queue is bound** — if it is, invocation and Worker startup provably work and the fault is downstream, which rules out most of the surface in one command; (3) confirm the version is **current** (CLI-created versions are not automatic, and a confirmation-prompted command may have silently done nothing); (4) check the compute provider's logs for connection, auth, or TLS errors; (5) if rapid repeated invocations show no progress, check the deployment name/build ID match. Distinguish a Temporal-side failure (reproduces no matter what you change on the provider side) from a genuine user-permission problem before editing anything. → `references//diagnostics.md`, `references/concepts.md`. + +## Common Pitfalls + +High-impact mistakes — warn the user proactively. Each is a symptom → cause → fix. + +1. **Deployment name / build ID mismatch → invocation loop.** *Symptom:* rapid, repeated invocations with no Workflow progress. *Cause:* the name or build ID in the Worker code doesn't match the Worker Deployment Version, so the Worker polls with the wrong version, the Task isn't processed, and Temporal invokes again. *Fix:* make the values in code exactly match the version configuration. +2. **Version not set as current.** A version created through the CLI is not automatically current; without it, Tasks don't route to the version and the Worker is never invoked. *Fix:* set it current as a separate step (the UI does this automatically). +3. **Failed first invocation.** When a version is created, the WCI invokes the Worker once to validate. If that invocation fails — missing env vars, bad TLS/auth config, missing dependencies, or an invocation deadline too short for the Worker to start and register the Task Queue — the Worker never connects, never polls, the binding is never created, and the Worker is never automatically invoked again. *Fix:* diagnose by manually invoking the compute unit, and confirm the invocation deadline is set high. +4. **Confusing the two roles.** The compute unit's execution role (grants the function permission to run) is separate from the access Temporal uses to invoke it. Never describe one as the other. → `references//iam.md`. +5. **Timeout tuning mismatch.** Raising only the shutdown deadline buffer makes the Worker stop polling earlier but gives in-flight Activities no more time; raising only the worker stop timeout doesn't make it stop polling earlier, so the provider may terminate the Worker first. *Fix:* tune the three values together (see the timeout triple above). +6. **Mutable "latest" build reference in production.** Pointing the provider at a mutable/unqualified target means the code changes on every redeploy; deploying replay-unsafe code then causes non-determinism errors for in-flight Workflows, even Pinned ones. *Fix:* publish an immutable versioned build and keep a 1-to-1 mapping between each Build ID and one build. → `references//versioning.md`. +7. **Re-creating shared permission infrastructure that already exists.** *Symptom:* the infrastructure deployment fails outright and rolls back, or it succeeds and leaves a second, redundant grant behind. *Cause:* the permission grant Temporal assumes is account-wide with a fixed default name, so a previous serverless deployment already owns it. *Fix:* check whether it exists and what owns it *before* creating; extend the existing one to cover the new Worker, and fall back to a distinctly named parallel one only when the existing infrastructure is not yours to change — saying why when you do. A failed-and-rolled-back deployment must be deleted before the name can be reused; a successful one is live infrastructure and must not be. → `references//iam.md`. +8. **Invoke permission scoped to a single build.** *Symptom:* the deployment works, then the *next* release cannot be invoked, with an error that looks like a connection or configuration problem rather than a permissions one. *Cause:* the grant named one immutable build, and the new release is a different resource. *Fix:* scope the grant to cover the base resource and all its published builds. → `references//iam.md`. + +## Routing to reference files + +Most questions need 2–3 reference files. + +| User intent | Reference file(s) | +|---|---| +| What is a Serverless Worker / the WCI? How do invocation and autoscaling work? What are the constraints? Serverless vs long-lived Workers? | `references/concepts.md` | +| Deploy a Serverless Worker (happy path): write code, package, deploy, register + set-current version, verify, tear down. | `references//setup.md` (+ `references/concepts.md`) | +| Operator permissions and preflight; execution role vs Temporal invocation role; CloudFormation (Cloud + self-hosted). | `references//iam.md` | +| Update or redeploy; version the build, use a qualified ARN, roll back. | `references//versioning.md` (+ `references/concepts.md`) | +| Self-hosted server enablement (dynamic config, WCI, server AWS credentials). | `references//self-hosted.md` (+ `references//iam.md`) | +| SDK-specific options and tuned defaults, which package to install and how it is distributed, imports, versioning-behavior configuration, connection config (TOML, env vars). Reduce cold start / pre-bundle Workflow code. | `references/sdk-configuration.md` | +| Add OpenTelemetry observability, collector config, tracing. | `references//observability.md` | +| Worker not invoked, Workflows not progressing, inspect the WCI. | `references//diagnostics.md` (+ `references/concepts.md`) | +| Long-running Activities and timeout relationships. Isolate Activities from resource exhaustion. | `references/concepts.md` (+ `references/sdk-configuration.md`) | + +## Out of Scope + +- **General SDK development patterns** (Workflows, Activities, signals, queries, Worker Versioning concepts): see `skill-temporal-developer`. +- **Traditional Worker tuning** (slot suppliers, tuners, poller autoscaling, resource-based tuning): see `skill-temporal-workertuning`. +- **Temporal Cloud administration** (Namespaces, users, certificates, billing): see `skill-temporal-ops`. +- **CLI command reference** (beyond the serverless-specific flags): see `skill-temporal-cli`. diff --git a/plugins/temporal/skills/temporal-serverless/agents/openai.yaml b/plugins/temporal/skills/temporal-serverless/agents/openai.yaml new file mode 100644 index 0000000..0b274c4 --- /dev/null +++ b/plugins/temporal/skills/temporal-serverless/agents/openai.yaml @@ -0,0 +1,6 @@ +interface: + display_name: "Temporal Serverless" + short_description: "Deploy and operate Temporal Workers on serverless compute" + +policy: + allow_implicit_invocation: false diff --git a/plugins/temporal/skills/temporal-serverless/assets/temporal-cloud-serverless-worker-role.yaml b/plugins/temporal/skills/temporal-serverless/assets/temporal-cloud-serverless-worker-role.yaml new file mode 100644 index 0000000..8fe5571 --- /dev/null +++ b/plugins/temporal/skills/temporal-serverless/assets/temporal-cloud-serverless-worker-role.yaml @@ -0,0 +1,99 @@ +# CloudFormation template for creating an IAM role that Temporal Cloud can assume to invoke Lambda functions. +# Transcribed verbatim from docs/production-deployment/worker-deployments/serverless-workers/aws-lambda.mdx:404-501 +AWSTemplateFormatVersion: '2010-09-09' +Description: + Creates an IAM role that Temporal Cloud can assume to invoke multiple Lambda functions for Serverless Workers. + +Parameters: + AssumeRoleExternalId: + Type: String + Description: A string you choose. Can be any value. + AllowedPattern: '[a-zA-Z0-9_+=,.@-]*' + MinLength: 5 + MaxLength: 45 + + LambdaFunctionARNs: + Type: CommaDelimitedList + Description: >- + Comma-separated list of Lambda function ARNs to invoke (e.g., + arn:aws:lambda:us-west-2:123456789012:function:worker-1,arn:aws:lambda:us-west-2:123456789012:function:worker-2) + + RoleName: + Type: String + Default: 'Temporal-Cloud-Serverless-Worker' + +Metadata: + AWS::CloudFormation::Interface: + ParameterGroups: + - Label: + default: 'Temporal Cloud Configuration' + Parameters: + - AssumeRoleExternalId + - Label: + default: 'Lambda Configuration' + Parameters: + - LambdaFunctionARNs + - RoleName + ParameterLabels: + AssumeRoleExternalId: + default: 'External ID' + LambdaFunctionARNs: + default: 'Lambda Function ARNs (comma-separated list)' + RoleName: + default: 'IAM Role Name' + +Resources: + TemporalCloudServerlessWorker: + Type: AWS::IAM::Role + Properties: + RoleName: !Ref RoleName + AssumeRolePolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Principal: + AWS: + [ + arn:aws:iam::902542641901:role/wci-lambda-invoke, + arn:aws:iam::160190466495:role/wci-lambda-invoke, + arn:aws:iam::819232936619:role/wci-lambda-invoke, + arn:aws:iam::829909441867:role/wci-lambda-invoke, + arn:aws:iam::354116250941:role/wci-lambda-invoke, + ] + Action: sts:AssumeRole + Condition: + StringEquals: + 'sts:ExternalId': [!Ref AssumeRoleExternalId] + Description: 'The role Temporal Cloud uses to invoke Lambda functions for Serverless Workers' + MaxSessionDuration: 3600 # 1 hour + + TemporalCloudLambdaInvokePermissions: + Type: AWS::IAM::Policy + DependsOn: TemporalCloudServerlessWorker + Properties: + PolicyName: 'Temporal-Cloud-Lambda-Invoke-Permissions' + PolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Action: + - lambda:InvokeFunction + - lambda:GetFunction + Resource: !Ref LambdaFunctionARNs + Roles: + - !Ref RoleName + +Outputs: + RoleARN: + Description: The ARN of the IAM role created for Temporal Cloud + Value: !GetAtt TemporalCloudServerlessWorker.Arn + Export: + Name: !Sub '${AWS::StackName}-RoleARN' + + RoleName: + Description: The name of the IAM role + Value: !Ref RoleName + + LambdaFunctionARNs: + Description: The Lambda function ARNs that can be invoked + Value: !Join [', ', !Ref LambdaFunctionARNs] diff --git a/plugins/temporal/skills/temporal-serverless/assets/temporal-self-hosted-serverless-worker-role.yaml b/plugins/temporal/skills/temporal-serverless/assets/temporal-self-hosted-serverless-worker-role.yaml new file mode 100644 index 0000000..afb3e54 --- /dev/null +++ b/plugins/temporal/skills/temporal-serverless/assets/temporal-self-hosted-serverless-worker-role.yaml @@ -0,0 +1,76 @@ +# CloudFormation template for creating an IAM role that a self-hosted Temporal Service can assume to invoke Lambda functions. +# Transcribed verbatim from docs/production-deployment/worker-deployments/serverless-workers/self-hosted-setup.mdx:136-209 +AWSTemplateFormatVersion: '2010-09-09' +Description: + Creates an IAM role that a self-hosted Temporal Service can assume to invoke Lambda functions for Serverless Workers. + +Parameters: + TemporalIamRoleArn: + Type: String + Description: The ARN of the IAM role or user that the Temporal Service runs as. + + AssumeRoleExternalId: + Type: String + Description: A unique identifier to prevent confused deputy attacks. + AllowedPattern: '[a-zA-Z0-9_+=,.@-]*' + MinLength: 5 + MaxLength: 45 + + LambdaFunctionARNs: + Type: CommaDelimitedList + Description: >- + Comma-separated list of Lambda function ARNs to invoke (e.g., + arn:aws:lambda:us-west-2:123456789012:function:worker-1,arn:aws:lambda:us-west-2:123456789012:function:worker-2) + + RoleName: + Type: String + Default: 'Temporal-Serverless-Worker' + +Resources: + TemporalServerlessWorker: + Type: AWS::IAM::Role + Properties: + RoleName: !Ref RoleName + AssumeRolePolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Principal: + AWS: [!Ref TemporalIamRoleArn] + Action: sts:AssumeRole + Condition: + StringEquals: + 'sts:ExternalId': [!Ref AssumeRoleExternalId] + Description: 'The role the Temporal Service uses to invoke Lambda functions for Serverless Workers' + MaxSessionDuration: 3600 + + TemporalLambdaInvokePermissions: + Type: AWS::IAM::Policy + DependsOn: TemporalServerlessWorker + Properties: + PolicyName: 'Temporal-Lambda-Invoke-Permissions' + PolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Action: + - lambda:InvokeFunction + - lambda:GetFunction + Resource: !Ref LambdaFunctionARNs + Roles: + - !Ref RoleName + +Outputs: + RoleARN: + Description: The ARN of the IAM role created for the Temporal Service + Value: !GetAtt TemporalServerlessWorker.Arn + Export: + Name: !Sub '${AWS::StackName}-RoleARN' + + RoleName: + Description: The name of the IAM role + Value: !Ref RoleName + + LambdaFunctionARNs: + Description: The Lambda function ARNs that can be invoked + Value: !Join [', ', !Ref LambdaFunctionARNs] diff --git a/plugins/temporal/skills/temporal-serverless/references/aws-lambda/diagnostics.md b/plugins/temporal/skills/temporal-serverless/references/aws-lambda/diagnostics.md new file mode 100644 index 0000000..f8f20d9 --- /dev/null +++ b/plugins/temporal/skills/temporal-serverless/references/aws-lambda/diagnostics.md @@ -0,0 +1,179 @@ +# AWS Lambda — Diagnostics & troubleshooting + + + +## Invocation flow (when working correctly) + +When a Serverless Worker invocation works correctly, the following sequence happens: + +1. You deploy the Worker function on Lambda. +2. You configure a Worker Deployment Version with a compute provider. This starts a Worker Controller Instance (WCI) Workflow and a validation invocation of the Lambda function. +3. The Lambda polls the Temporal Service successfully, binding the Task Queue configured on the Worker to the Worker Deployment Version. +4. The WCI continuously monitors the associated Task Queue on a schedule. The Matching Service also notifies the WCI Workflow of sync match failures immediately as they happen. +5. A Task arrives on the Task Queue and the WCI detects the backlog. +6. The WCI invokes the Lambda function. +7. The Lambda function starts, the Worker connects to Temporal and polls the Task Queue. +8. The Worker processes Tasks and shuts down gracefully. + +## Diagnostic decision tree + +Start by determining whether the Lambda function is being invoked at all, then narrow down from there. + +Check the Lambda function's CloudWatch metrics or invocation logs. In the AWS Console, go to **Lambda > Functions > your function > Monitor**. Look for recent invocations in the **Invocations** graph. You can also check **CloudWatch > Log groups > /aws/lambda/your-function-name** for execution logs. + +- If there are no invocations: see [Lambda is not being invoked](#lambda-not-invoked). +- If the Lambda is being invoked but Workflows are not progressing: see [Lambda is invoked but Tasks are not completing](#lambda-invoked-not-completing). + +--- + +## Lambda is not being invoked {#lambda-not-invoked} + +Work through the following checks in order. + +### 1. Validate the connection to Lambda + +Go to **Workers > Deployments > select your deployment**, open the **Actions** menu on the version, and click **Validate Connection**. + +A successful validation confirms that: +- The Worker Deployment Version has a compute provider configured. +- Temporal can assume the invocation role. +- The Lambda function can be invoked. + +If validation fails: +- Verify the Lambda function ARN and invocation role ARN in the Worker Deployment Version configuration are correct. +- Verify the invocation role was created using the CloudFormation template and that the External ID matches the value in the Worker Deployment Version configuration. + +**Not every validation failure is on your side.** If the error is Temporal failing to obtain its **own** base AWS credentials (for example, `no EC2 IMDS role found`) while assuming the `wci-lambda-invoke` role, that is a Temporal-Cloud-side credential problem occurring *before* Temporal ever reaches your invocation role or Lambda. Signature: it reproduces identically no matter what you change on the AWS side, and (for Cloud) is independent of your trust policy or External ID. Do not keep editing your AWS config — verify your artifacts once (Lambda `Active` with the correct runtime; trust policy lists all five `wci-lambda-invoke` principals plus the `sts:ExternalId` condition; invoke permissions scoped to the function ARN), then escalate to Temporal support. Expect infrastructure-side rough edges while the feature is in Public Preview. + +If the Worker Deployment Version does not have a compute provider configured, no WCI Workflow exists and the Lambda is never automatically invoked. + +**Common cause:** Manually invoking the Lambda function before creating the Worker Deployment Version in the UI or CLI. When the Lambda runs, the Worker connects to Temporal and polls the Task Queue. That polling registers the Worker Deployment Version and binds the Task Queue on the server, but the version has no compute provider. To fix, create or update the Worker Deployment Version with the compute provider flags. + +**Recovering a provider-less version via the CLI.** In practice there is no `update-provider` command, and `describe-version` does **not** show whether a compute provider is attached, so you cannot confirm the provider-less state by inspecting the version — you infer it because `create-version` reports the version already exists. Re-running `create-version` fails for the same reason. The reliable fix is to **delete the provider-less version and recreate it** with the compute-provider flags. This is clean as long as the version was never set as current. + +```bash +temporal worker deployment delete-version \ + --deployment-name \ + --build-id +# then re-run: temporal worker deployment create-version --aws-lambda-function-arn ... (see setup.md, Step 4) +``` + +`delete-version` requires the version to not be Current, Ramping, or have active pollers; add `--skip-drainage` to ignore the draining restriction. + +### 2. Check that the version is set as current + +The Worker Deployment Version must be set as the current version for new Tasks to route to it. If you created the version through the CLI, you need to set it as current. + +Verify the current version with: + +```bash +temporal worker deployment describe --name +``` + +**A `set-current-version` that ran without `--yes` may have done nothing.** The command prompts for confirmation; run non-interactively (script, CI, agent shell) it exits without applying the change, which is easy to mistake for success. If `describe` does not show your build as current, re-run `set-current-version` with `--yes` rather than looking for a deeper cause. + +### 3. Check that the WCI is detecting Tasks + +If the connection validates successfully but the Lambda is still not being invoked, the WCI may not be detecting Tasks on the Task Queue. + +Check which Task Queues are bound to the Worker Deployment Version and whether there is a backlog: + +```bash +temporal worker deployment describe-version \ + --namespace \ + --deployment-name \ + --build-id \ + --report-task-queue-stats +``` + + +If no Task Queues are listed, the binding has not been established. The server binds a Task Queue to a Worker Deployment Version when a Worker with that deployment version successfully connects and polls the Task Queue. + +**What a healthy version looks like.** The inverse of the check above is the single most useful positive signal in the whole setup: the Task Queue appearing here, for both workflow and activity task types, proves that Temporal assumed the invocation role, invoked the Lambda, the package loaded, the env vars were right, the Worker authenticated to the Namespace, and the timeout was long enough to reach a poll. Getting this output means every remaining failure is downstream (routing, versioning-behavior, or application code), so check it immediately after `create-version` and before touching anything else. `describe-version` still does **not** report whether a compute provider is attached, so it cannot rule that in or out. + +#### Failed first invocation + +A common cause of missing Task Queue bindings is a failed first invocation. When you create a Worker Deployment Version, the WCI invokes the Lambda to validate the configuration. If that first invocation fails (for example, due to missing environment variables, incorrect TLS configuration, missing dependencies, or a Lambda timeout that is too short), the Worker never connects to Temporal and never polls. Without a successful poll, the Task Queue binding is never created. + +Pay particular attention to the Lambda timeout. AWS Lambda functions default to a 3-second timeout, which is often too short for the Worker to start, connect to Temporal, and register the Task Queue before AWS terminates the invocation. If the function times out during this initial invocation, the binding is never established and the Lambda is not invoked again. + +To diagnose: invoke the Lambda function manually from the AWS Console. The console displays the execution result and any errors directly, making it easier to identify configuration issues than searching through CloudWatch logs. Once the Lambda runs successfully and the Worker connects to Temporal, the Task Queue binding is established. + +**A manual invoke runs for nearly the whole timeout.** A serverless Worker keeps polling until its shutdown-deadline buffer, so a synchronous manual invoke runs for roughly the full Lambda timeout (e.g. ~590s of a 600s timeout), not a few seconds. From the AWS CLI (`aws lambda invoke`) this trips the default **60-second client read timeout** with a "Read timeout on endpoint" error — that is expected, **not** a Worker crash. Add `--cli-read-timeout 0` (or use `--invocation-type Event` for an async invoke), and judge health from the CloudWatch startup logs (Worker connected + polling), not from the CLI's exit. Remember this successful poll also auto-registers a provider-less version — see "Common cause" above. + +**A running WCI is not proof anything works.** The WCI Workflow continue-as-news and keeps running even while its invocation/scaling Activities are failing, so its existence tells you nothing about invocation health. Read the WCI's history (below) and look for **Activity failures** to find the real error. Never infer health from the WCI merely existing or running. + + +--- + +## Lambda is invoked but Tasks are not completing {#lambda-invoked-not-completing} + +If CloudWatch shows Lambda invocations but Workflows are not progressing, the problem is in the Worker's execution within the Lambda function. + +### Check Lambda execution logs + +Check CloudWatch logs for errors during Worker startup. In the AWS Console, go to **CloudWatch > Log groups > /aws/lambda/your-function-name** and look for recent error messages. + +Common errors include: + +- **Connection failures**: The Worker cannot reach the Temporal Service. Check that the `TEMPORAL_ADDRESS` and `TEMPORAL_API_KEY` environment variables (or `temporal.toml` config file) are correctly set on the Lambda function. For self-hosted deployments, verify network reachability. +- **TLS errors**: The TLS certificate or key is missing, expired, or does not match the Namespace. +- **Authentication errors**: The API key is invalid or does not have access to the Namespace. + +### Check for Lambda timeout + +If the Lambda function reaches its configured timeout before the Worker finishes processing, AWS terminates the invocation. + +The Worker begins graceful shutdown before the Lambda deadline. If Activities take longer than the available execution window, the Activities are abandoned mid-execution and retried on the next invocation. + +For long-running Activities, increase the Lambda timeout and the Worker's shutdown buffer together. + +### Check that the deployment name and build ID match + +If CloudWatch shows rapid, repeated invocations with no Workflow progress, the deployment name or build ID in the Worker code may not match the Worker Deployment Version configuration. + +The deployment name and build ID in your Lambda function code must exactly match the values you used when creating the Worker Deployment Version. Compare the values in your code against the WCI Workflow ID (`temporal-sys-worker-controller-instance::`) and the output of `temporal worker deployment describe`. + +A mismatch causes an invocation loop: the WCI invokes the Lambda, the Worker starts and polls with a different deployment version than the WCI expects, the Task is not processed, and the WCI invokes the Lambda again. + +To fix the loop, update the deployment name and build ID in the Worker code to match the Worker Deployment Version, then redeploy the Lambda function. + +--- + +## WCI Workflow inspection + +You never create or manage a WCI Workflow yourself — Temporal creates one automatically for each Worker Deployment Version that has a compute provider, and it lives in a `temporal-sys-*` Namespace division. Diagnose from Temporal's own signals (the WCI history) **before** touching AWS; do not enumerate Lambdas across regions or scan the AWS account to reverse-engineer state, which is invasive, slow, and unnecessary. + +List WCI Workflows in your Namespace: + +```bash +temporal workflow list \ + --namespace \ + --query 'TemporalNamespaceDivision = "TemporalWorkerControllerInstance"' +``` + + +WCI Workflow IDs follow the pattern `temporal-sys-worker-controller-instance::`. + +Inspect a WCI Workflow's history to see its recent Activity results: + +```bash +temporal workflow show \ + --namespace \ + --workflow-id 'temporal-sys-worker-controller-instance::' +``` + + +Describe a Worker Deployment Version and check Task Queue stats: + +```bash +temporal worker deployment describe-version \ + --namespace \ + --deployment-name \ + --build-id \ + --report-task-queue-stats +``` + diff --git a/plugins/temporal/skills/temporal-serverless/references/aws-lambda/iam.md b/plugins/temporal/skills/temporal-serverless/references/aws-lambda/iam.md new file mode 100644 index 0000000..0e58d2a --- /dev/null +++ b/plugins/temporal/skills/temporal-serverless/references/aws-lambda/iam.md @@ -0,0 +1,264 @@ +# AWS Lambda — IAM & permissions + + + +This file covers three distinct identities: the **operator** (whose credentials run the deploy commands), the **execution role** (grants the function permission to run), and the **Temporal invocation role** (grants Temporal permission to invoke the function). The deploy steps themselves are in `setup.md`; self-hosted server enablement is in `self-hosted.md`. + +## Operator AWS permissions + +The identity whose credentials run the `aws`/CloudFormation commands (local profile, EC2/ECS instance role, CI role, or CloudShell console identity) needs the actions below. These are separate from the execution and invocation roles created in Step 3. + +| Deployment step | Operator actions required | +|---|---| +| Deploy / update the Lambda (`setup.md`, Step 2) | `lambda:CreateFunction`, `lambda:UpdateFunctionCode`, `lambda:GetFunction`, `lambda:PublishVersion`, `lambda:InvokeFunction`, and **`iam:PassRole`** on the execution role. | +| Create the execution role directly (if not using CloudFormation) | `iam:CreateRole`, `iam:AttachRolePolicy`. | +| Create the invocation role via CloudFormation (Step 3) | `cloudformation:CreateStack`, `cloudformation:DescribeStacks`, plus IAM write (`iam:CreateRole`, `iam:PutRolePolicy`/`iam:AttachRolePolicy`, `iam:GetRole`, `iam:DeleteRole` for rollback) — hence `--capabilities CAPABILITY_NAMED_IAM`. | +| Read Worker logs (`setup.md` verify + `diagnostics.md`) | `logs:FilterLogEvents`, `logs:GetLogEvents`, `logs:DescribeLogGroups`, `logs:DescribeLogStreams`. | + +`iam:PassRole` is required separately from `lambda:CreateFunction`: `create-function` attaches the execution role to the function, and without `iam:PassRole` on that role the deploy fails with `AccessDenied ... not authorized to perform: iam:PassRole` even when `CreateFunction` is allowed. Scope it to the execution role ARN. + +If the operator cannot have IAM write permissions, have an administrator run the Step 3 CloudFormation stack once and hand back the `RoleARN` output (and pre-create the execution role if needed). The operator then needs only the Lambda actions, `iam:PassRole` on the pre-created execution role, and CloudWatch Logs read. + +### Preflight check + +Run before any command that creates or modifies AWS resources. None should print `DENIED`: + +```bash +aws sts get-caller-identity +aws lambda list-functions --max-items 1 >/dev/null 2>&1 && echo "lambda: ok" || echo "lambda: DENIED" +aws cloudformation describe-stacks >/dev/null 2>&1 && echo "cloudformation: ok" || echo "cloudformation: DENIED" +aws iam list-roles --max-items 1 >/dev/null 2>&1 && echo "iam read: ok" || echo "iam: limited (role creation may be blocked)" +aws logs describe-log-groups --limit 1 >/dev/null 2>&1 && echo "logs: ok" || echo "logs: DENIED" +``` + +The calls above cannot exercise `iam:PassRole`. To check it (and any other specific action) authoritatively, use the policy simulator against the caller's ARN: + +```bash +CALLER_ARN=$(aws sts get-caller-identity --query Arn --output text) +aws iam simulate-principal-policy \ + --policy-source-arn "$CALLER_ARN" \ + --action-names lambda:CreateFunction iam:PassRole cloudformation:CreateStack iam:CreateRole \ + --query 'EvaluationResults[].{action:EvalActionName,decision:EvalDecision}' --output table +``` + +For an assumed-role session, rewrite the ARN (`arn:aws:sts::…:assumed-role/Name/session`) to the underlying role ARN (`arn:aws:iam::…:role/Name`) for `simulate-principal-policy`. + +### When the preflight cannot authenticate + +`aws sts get-caller-identity` failing does not mean the user has no access — most often no profile is selected, or an SSO token has expired. Classify before concluding anything: + +| Error | Meaning | Action | +|---|---|---| +| `Unable to locate credentials` | Nothing configured or selected | Check for existing profiles before concluding there are none | +| `ExpiredToken`, `The SSO session associated with this profile has expired` | Profile is fine, token is stale | Refresh it, below | +| `InvalidClientTokenId`, `SignatureDoesNotMatch`, `UnrecognizedClientException` | Credentials exist but are invalid — stale static keys, a deleted user, or the wrong account | Do not retry; re-authenticate, or ask which profile is intended. Report the account you *did* resolve, if any | +| `AccessDenied` on an action while `get-caller-identity` succeeds | Authenticated, insufficient permissions | Genuine permissions problem — see the operator table above | + +Look at what is already configured before asking the user for anything: + +```bash +echo "AWS_PROFILE=${AWS_PROFILE:-}" +aws configure list-profiles +``` + +Then pick the login command by what that turned up. These are two different commands, not aliases — do not substitute one for the other: + +**A profile already exists (token is just stale)** — refresh the IAM Identity Center token: + +```bash +aws sso login --profile --no-browser +``` + +`--no-browser` prints the verification URL and user code instead of opening a browser. Auto-open is unreliable over SSH, containers, and remote sessions, and the user needs the URL in the conversation either way. + +**Nothing is configured at all** — `aws login` needs no prior setup, acquiring temporary credentials from a Management Console session plus a refresh token that the CLI renews automatically: + +```bash +aws login +``` + +Two limits on it. AWS scopes this command to *local development*, so do not use it as the identity for a CI or production deployment — those need a configured profile or an assumed role. And it is recent enough that older CLIs do not have it: confirm with `aws login help` before offering it, and fall back to asking the user to run `aws configure sso` themselves if it is absent. Avoid its `--remote` flag in an agent shell — unlike `aws sso login --no-browser`, it prompts for an authorization code on stdin; on a remote host, prefer configuring an SSO profile and using `aws sso login --no-browser` instead. + +Either command blocks until the browser flow finishes, so run it in a background shell, surface the URL (and code, if any) to the user, and poll until credentials land: + +```bash +aws sts get-caller-identity +``` + +Re-run the preflight above and continue the deployment. + +If the organization wraps credentials in its own tool (`aws-vault`, `granted`/`assume`, `saml2aws`), use that instead — ask which, rather than guessing. + +**Never run `aws configure`/`aws configure sso` interactively, and never ask the user to paste access keys or session tokens into the conversation** — the wizards block an agent shell on stdin, and pasted keys land in the transcript and shell history. If neither login command applies, ask the user to configure a profile or export credentials in their own shell, then tell you when to re-run the preflight. + +## Execution role + +The Lambda execution role grants the function permission to run. It is trusted by `lambda.amazonaws.com` and must have at least the `AWSLambdaBasicExecutionRole` managed policy attached (which includes the CloudWatch Logs permissions the Worker needs). This is separate from the Temporal invocation role below. Pass its ARN as `--role` when you create the function (`setup.md`, Step 2). + +## Step 3: Configure IAM for Temporal invocation + +### Temporal Cloud + +This section applies to Temporal Cloud. For self-hosted, see the self-hosted section below. + +Temporal needs permission to invoke your Lambda function and check its status. The Temporal server assumes an IAM role in your AWS account with a handful of Lambda permissions scoped to your Worker functions. The trust policy on the role includes an External ID condition to prevent confused deputy attacks. + +#### CloudFormation template parameters + + + +| Parameter | Description | +|---|---| +| `AssumeRoleExternalId` | A string you choose to prevent confused deputy attacks. Can be any value. Use the same value when creating the Worker Deployment Version. | +| `LambdaFunctionARNs` | Comma-separated list of Lambda function ARNs that Temporal may invoke. To allow invocation of any published version of a function, use a wildcard suffix (for example, `arn:aws:lambda:...:function:my-temporal-worker:*`). One role can authorize multiple Worker Lambdas. | +| `RoleName` | Base name for the created IAM role. Defaults to `Temporal-Cloud-Serverless-Worker`. Provide a new role name if creating more than one stack. | + +#### Before creating the stack: check whether the role already exists + +**The default `RoleName` collides, and the failure mode is a full stack rollback.** The template creates a *named* IAM role, and IAM role names are unique per account. Any earlier serverless deployment in the same account already holds `Temporal-Cloud-Serverless-Worker`, so a second `create-stack` with default parameters fails with `already exists` and rolls the whole stack back. Check first: + +```bash +ROLE=Temporal-Cloud-Serverless-Worker # or whatever name you intend to use + +# Does the role already exist? +aws iam get-role --role-name "$ROLE" --query 'Role.Arn' --output text 2>/dev/null \ + || echo "not present — safe to create" + +# If it does: which stack owns it? CloudFormation tags the resources it creates. +# Empty output means the role is orphaned or hand-created, not stack-managed. +aws iam list-role-tags --role-name "$ROLE" \ + --query 'Tags[?Key==`aws:cloudformation:stack-name`].Value' --output text + +# Cross-check against the stack list if the tag lookup comes back empty +aws cloudformation describe-stacks --region \ + --query 'Stacks[?StackStatus!=`DELETE_COMPLETE`].{Name:StackName,Status:StackStatus}' \ + --output table +``` + +Then pick a path — **prefer reuse over a parallel role.** The invocation role is account-wide shared infrastructure, not per-function, and one role can authorize many Worker Lambdas: + +| Situation | Action | +|---|---| +| No role, no stack | Create the stack as documented below. | +| A stack owns the role, and this deployment is a legitimate addition | **Update that stack**, adding your function's ARNs to `LambdaFunctionARNs` (`aws cloudformation update-stack` with the full comma-separated list — CloudFormation replaces the list, it does not merge). Reuse its `RoleARN` output. Do not create a second role. | +| A stack owns the role but is someone else's infrastructure you must not touch | Create a *separate* stack with an explicit distinct `RoleName` (for example `Temporal-Cloud-Serverless-Worker-`) and say why in your summary. Two roles is the fallback, not the default. | +| The role exists but no stack owns it (orphaned or hand-created) | Do not delete it blind. Either reuse it after confirming its trust policy and permissions, or create a new stack with a distinct `RoleName`. Ask the user before removing an unowned IAM role. | + +**Recovering from a rolled-back stack.** A stack in `ROLLBACK_COMPLETE` cannot be updated or re-created under the same name — delete it first, then create again with corrected parameters. Read the actual cause before changing anything: + +```bash +aws cloudformation describe-stack-events --stack-name --region \ + --query 'StackEvents[?ResourceStatus==`CREATE_FAILED`].[LogicalResourceId,ResourceStatusReason]' --output table +aws cloudformation delete-stack --stack-name --region +aws cloudformation wait stack-delete-complete --stack-name --region +``` + +Deleting a `ROLLBACK_COMPLETE` stack is safe — it created nothing that survived. Deleting a stack in any *successful* state is not; that is live infrastructure. + +#### Which ARNs to authorize + +**`function:name:*` does not match the unqualified `function:name`.** In IAM, the wildcard-version ARN covers published versions and aliases only; the unqualified function ARN is a distinct resource. Temporal needs `lambda:GetFunction` and `lambda:InvokeFunction` to resolve for whatever ARN form you register on the Worker Deployment Version, so authorize **both** forms and you are covered either way: + +```bash +FN=arn:aws:lambda:::function:my-temporal-worker +# The CLI shorthand needs the literal double quotes to survive, so that the +# embedded comma is read as a list separator and not as the next ParameterKey: +ARNS="\"$FN,$FN:*\"" +# ... --parameters ParameterKey=LambdaFunctionARNs,ParameterValue="$ARNS" +``` + +The `:*` form is also what makes future `publish-version` builds work without an IAM change — authorizing a single fixed version (`function:my-temporal-worker:1`) means the next published version cannot be invoked, which surfaces later as a validation/invocation failure that looks unrelated to IAM. See `versioning.md`. + +#### Trust policy principals + +The Cloud template trusts five Temporal Cloud AWS account IDs with the role `wci-lambda-invoke`: + +- `arn:aws:iam::902542641901:role/wci-lambda-invoke` +- `arn:aws:iam::160190466495:role/wci-lambda-invoke` +- `arn:aws:iam::819232936619:role/wci-lambda-invoke` +- `arn:aws:iam::829909441867:role/wci-lambda-invoke` +- `arn:aws:iam::354116250941:role/wci-lambda-invoke` + +The IAM policy grants `lambda:InvokeFunction` and `lambda:GetFunction` on the specified Lambda function ARNs. + +#### Deploy the CloudFormation stack + +This skill ships the complete, ready-to-deploy template at `assets/temporal-cloud-serverless-worker-role.yaml` (transcribed verbatim from the docs). Copy it into your working directory, or point `--template-body` at the skill's copy — no need to author it by hand. + +```bash +aws cloudformation create-stack \ + --stack-name \ + --template-body file://temporal-cloud-serverless-worker-role.yaml \ + --parameters \ + ParameterKey=AssumeRoleExternalId,ParameterValue= \ + ParameterKey=LambdaFunctionARNs,ParameterValue='",:*"' \ + ParameterKey=RoleName,ParameterValue= \ + --capabilities CAPABILITY_NAMED_IAM \ + --region +``` + + +Notes on the parameters above: + +- **Set `RoleName` explicitly** rather than relying on the default, so the name reflects this deployment and does not collide with an existing one (see the collision check above). +- **`LambdaFunctionARNs` is a `CommaDelimitedList`.** The `'"a,b"'` quoting is required: the AWS CLI's `ParameterKey=,ParameterValue=` shorthand otherwise reads the embedded comma as the start of a new key. Keep the literal double quotes inside the single quotes. +- **Generate the External ID rather than choosing a memorable string** — it is a confused-deputy guard. The template accepts 5–45 characters matching `[a-zA-Z0-9_+=,.@-]*`. Record it: you need the identical value when registering the Worker Deployment Version, and it is not recoverable from the version afterward. + ```bash + EXTERNAL_ID=$(openssl rand -hex 16) + ``` + +Retrieve the IAM role ARN from the stack outputs: + +```bash +aws cloudformation describe-stacks --stack-name --query 'Stacks[0].Outputs[?OutputKey==`RoleARN`].OutputValue' --output text --region +``` + + +### Self-hosted invocation role + +For self-hosted server enablement (dynamic config, WCI, and the server's own AWS credentials), see `self-hosted.md`. The invocation role itself is below. + +Self-hosted Serverless Workers require Temporal Service v1.31.0 or later. + +#### Create the Lambda invocation role (self-hosted) + +Temporal invokes Lambda functions by assuming an IAM role in your AWS account. This role needs `lambda:GetFunction` and `lambda:InvokeFunction` permission on your Worker Lambda functions, and a trust policy that allows the Temporal server's identity to assume it. + +This skill ships the complete self-hosted template at `assets/temporal-self-hosted-serverless-worker-role.yaml` (verbatim from the docs). Copy it locally or point `--template-body` at the skill's copy. + +```bash +aws cloudformation create-stack \ + --stack-name temporal-serverless-worker \ + --template-body file://temporal-self-hosted-serverless-worker-role.yaml \ + --parameters \ + ParameterKey=TemporalIamRoleArn,ParameterValue= \ + ParameterKey=AssumeRoleExternalId,ParameterValue= \ + ParameterKey=LambdaFunctionARNs,ParameterValue='""' \ + --capabilities CAPABILITY_NAMED_IAM \ + --region +``` + + +| Parameter | Description | +|---|---| +| `TemporalIamRoleArn` | ARN of the IAM role or user that the Temporal Service runs as (the identity used to call `sts:AssumeRole`). Run `aws sts get-caller-identity` in the server's environment to find it. | +| `AssumeRoleExternalId` | A unique string to prevent confused deputy attacks. Use the same value when creating the Worker Deployment Version. | +| `LambdaFunctionARNs` | Comma-separated list of Lambda function ARNs that Temporal may invoke. To allow any published version, use a wildcard suffix (for example, `arn:aws:lambda:...:function:my-temporal-worker:*`). | +| `RoleName` | Base name for the created IAM role. Defaults to `Temporal-Serverless-Worker`. Provide a new role name if creating more than one stack. | + + +Retrieve the role ARN: + +```bash +aws cloudformation describe-stacks \ + --stack-name temporal-serverless-worker \ + --query 'Stacks[0].Outputs[?OutputKey==`RoleARN`].OutputValue' \ + --output text \ + --region +``` + + +**Key distinction:** The Lambda execution role (trusted by `lambda.amazonaws.com`) is separate from the Temporal invocation role (trusted by Temporal's `wci-lambda-invoke` principals for Cloud, or the Temporal Service's own IAM identity for self-hosted). The execution role grants the function permission to run. The invocation role grants Temporal permission to invoke the function. diff --git a/plugins/temporal/skills/temporal-serverless/references/aws-lambda/observability.md b/plugins/temporal/skills/temporal-serverless/references/aws-lambda/observability.md new file mode 100644 index 0000000..af52f8f --- /dev/null +++ b/plugins/temporal/skills/temporal-serverless/references/aws-lambda/observability.md @@ -0,0 +1,225 @@ +# Observability for Serverless Workers + + + +## Overview + +Each SDK provides an OpenTelemetry integration package with defaults configured for the AWS Distro for OpenTelemetry (ADOT) Lambda layer. When enabled, the Worker emits SDK metrics and distributed traces for Workflow and Activity executions. The ADOT Lambda layer collects this telemetry and can forward traces to AWS X-Ray and metrics to Amazon CloudWatch. + +## Go SDK + +### OTel package + +Import: `otel "go.temporal.io/sdk/contrib/aws/lambdaworker/otel"` + +### OTel functions + +- `otel.ApplyDefaults` — configures both metrics and tracing. +- `otel.ApplyMetrics` — configures metrics only. +- `otel.ApplyTracing` — configures tracing only. + +Usage in the configure callback: + +```go +if err := otel.ApplyDefaults(opts, &opts.ClientOptions, otel.Options{}); err != nil { + return err +} +``` + +By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. + +### ADOT layer setup (Go) + +Attach the ADOT Collector layer to your Lambda function. +Go does not need a language-specific ADOT layer because the OTel SDK is compiled into the binary. + +### Collector config env var (Go) + +`OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml` + +--- + +## Python SDK + +### OTel package + +Import: `from temporalio.contrib.aws.lambda_worker.otel import apply_defaults` + +To install with OTel support: `pip install temporalio[lambda-worker-otel]` + +### OTel functions + +- `apply_defaults` — configures both metrics and tracing. +- `build_metrics_telemetry_config` — configures metrics only. +- `apply_tracing` — configures tracing only. + +Usage in the configure callback: + +```python +def configure(config: LambdaWorkerConfig) -> None: + config.worker_config["task_queue"] = TASK_QUEUE + config.worker_config["workflows"] = [SampleWorkflow] + config.worker_config["activities"] = [hello_activity] + apply_defaults(config) +``` + +By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. + +### ADOT layer setup (Python) + +Attach the ADOT Python Lambda layer to your Lambda function. The layer includes both auto-instrumentation and an OpenTelemetry Collector that receives telemetry on `localhost:4317` and forwards traces to AWS X-Ray and metrics to Amazon CloudWatch. + +### Collector config env var (Python) + +`OPENTELEMETRY_COLLECTOR_CONFIG_FILE=/var/task/otel-collector-config.yaml` + +Note: Python uses `_FILE` while Go and TypeScript use `_URI`. + +--- + +## TypeScript SDK + +### OTel package + +Import: `import { applyDefaults } from '@temporalio/lambda-worker/otel'` + +### OTel functions + +- `applyDefaults` — registers Temporal SDK interceptors for tracing and configures the Core SDK to export metrics via OTLP. +- `makeOtelPlugin` — returns a plugin for pre-bundling Workflow code that includes Workflow interceptor modules. + +Usage in the configure callback: + +```typescript +export const handler = runWorker({ deploymentName: 'sdk-demo', buildId: 'v1' }, (config) => { + config.workerOptions.taskQueue = TASK_QUEUE; + config.workerOptions.workflowBundle = { + codePath: require.resolve('./workflow-bundle.js'), + }; + config.workerOptions.activities = activities; + applyDefaults(config); +}); +``` + +By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. + +### Pre-bundling with OTel + +When pre-bundling Workflow code, pass the plugin from `makeOtelPlugin()` so that Workflow interceptor modules are included in the bundle: + +```typescript +import { bundleWorkflowCode } from '@temporalio/worker'; +import { makeOtelPlugin } from '@temporalio/lambda-worker/otel'; + +const { plugin } = makeOtelPlugin(); +const { code } = await bundleWorkflowCode({ + workflowsPath: require.resolve('./workflows'), + plugins: [plugin], +}); +``` + + +### ADOT layer setup (TypeScript) + +Attach two ADOT Lambda layers: + +1. The ADOT JavaScript layer for Node.js-side auto-instrumentation and trace export. +2. The ADOT Collector layer (`aws-otel-collector-amd64`) to run the OTel Collector as a Lambda extension, receiving telemetry via OTLP on `localhost:4317` and forwarding traces to X-Ray and metrics to CloudWatch. + +### Collector config env var (TypeScript) + +`OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml` + +--- + +## Common across all SDKs + +### Custom Collector configuration required + +The default ADOT Collector configuration does not route OpenTelemetry Protocol (OTLP) data to the traces pipeline. You must provide a custom Collector configuration that wires the OTLP receiver to both the traces and metrics pipelines. + +Example `otel-collector-config.yaml` (bundle in your Lambda deployment package): + +```yaml +receivers: + otlp: + protocols: + grpc: + endpoint: "localhost:4317" + http: + endpoint: "localhost:4318" + +exporters: + debug: + awsxray: + region: us-west-2 + awsemf: + namespace: TemporalWorkerMetrics + log_group_name: /aws/lambda/ + region: us-west-2 + dimension_rollup_option: NoDimensionRollup + resource_to_telemetry_conversion: + enabled: true + +service: + pipelines: + traces: + receivers: [otlp] + exporters: [awsxray, debug] + metrics: + receivers: [otlp] + exporters: [awsemf] + telemetry: + logs: + level: debug + metrics: + address: localhost:8888 +``` + + +### Enable X-Ray active tracing + +```bash +aws lambda update-function-configuration \ + --function-name \ + --tracing-config Mode=Active +``` + + +### Required IAM permissions + +The Lambda execution role must have permissions to write to X-Ray and CloudWatch: + +- `xray:PutTraceSegments` +- `xray:PutTelemetryRecords` +- `cloudwatch:PutMetricData` + +Without these permissions, the Collector fails silently and no telemetry appears. + +For Python, the `AWSXRayDaemonWriteAccess` managed policy can be attached instead. + +### Collector config env var summary + + + +| SDK | Environment variable | +|---|---| +| Go | `OPENTELEMETRY_COLLECTOR_CONFIG_URI` | +| Python | `OPENTELEMETRY_COLLECTOR_CONFIG_FILE` | +| TypeScript | `OPENTELEMETRY_COLLECTOR_CONFIG_URI` | + +### ADOT layer summary + +| SDK | Layers needed | +|---|---| +| Go | ADOT Collector layer only (no language-specific layer; OTel SDK is compiled into the binary) | +| Python | ADOT Python Lambda layer (includes collector and auto-instrumentation) | +| TypeScript | ADOT JavaScript layer + ADOT Collector layer (`aws-otel-collector-amd64`) | + + + + diff --git a/plugins/temporal/skills/temporal-serverless/references/aws-lambda/self-hosted.md b/plugins/temporal/skills/temporal-serverless/references/aws-lambda/self-hosted.md new file mode 100644 index 0000000..d6508e2 --- /dev/null +++ b/plugins/temporal/skills/temporal-serverless/references/aws-lambda/self-hosted.md @@ -0,0 +1,60 @@ +# AWS Lambda — Self-hosted Temporal Service + + + +Self-hosted Serverless Workers require Temporal Service v1.31.0 or later. + +Complete this server enablement before following `setup.md`. For the Lambda invocation role Temporal assumes (the CloudFormation stack and trust policy), see `iam.md`. + +## Network reachability + +The Temporal Service frontend must be reachable from the Lambda execution environment. If the Temporal Service runs on a private network, you may need VPC access for Lambda, VPC peering, or a similar mechanism. + +## Enable the Worker Controller Instance (WCI) + +WCI is disabled by default and must be enabled through dynamic configuration. + +Add the following keys to your dynamic config file: + +```yaml +workercontroller.enabled: + - value: true + +workercontroller.compute_providers.enabled: + - value: + - aws-lambda + +workercontroller.scaling_algorithms.enabled: + - value: + - no-sync +``` + + +To enable WCI for specific Namespaces instead of globally, add a `constraints` section with the Namespace name under `workercontroller.enabled`: + +```yaml +workercontroller.enabled: + - value: true + constraints: + namespace: 'your-namespace' +``` + + +The Temporal Service watches the dynamic config file for changes and applies updates without a restart. + +## Configure AWS credentials + +The Temporal Service needs AWS credentials to assume an IAM role that invokes Lambda functions. + +**On AWS infrastructure (EC2, ECS, EKS):** The server uses the attached instance role, task role, or pod role automatically. No additional credential configuration is needed. The attached role must have `sts:AssumeRole` permission for the Lambda invocation role. + +**Outside AWS:** Use IAM Roles Anywhere, or configure static AWS credentials in the server's environment (not recommended). These credentials must belong to an IAM user or role that has `sts:AssumeRole` permission for the Lambda invocation role. + +``` +AWS_ACCESS_KEY_ID= +AWS_SECRET_ACCESS_KEY= +AWS_REGION= +``` + diff --git a/plugins/temporal/skills/temporal-serverless/references/aws-lambda/setup.md b/plugins/temporal/skills/temporal-serverless/references/aws-lambda/setup.md new file mode 100644 index 0000000..a026a84 --- /dev/null +++ b/plugins/temporal/skills/temporal-serverless/references/aws-lambda/setup.md @@ -0,0 +1,639 @@ +# AWS Lambda — Setup (happy path) + + + +This is the end-to-end golden path: connect, write the Worker, package and deploy, register a Worker Deployment Version, set it current, and verify. For the operator permissions and preflight, execution/invocation roles, and CloudFormation, see `iam.md`. For production build versioning (`publish-version`, qualified ARNs, rollback), see `versioning.md`. For self-hosted server enablement, see `self-hosted.md`. If it doesn't work, see `diagnostics.md`. + +## Prerequisites + +- A Temporal Cloud account with an AWS-hosted Namespace, or a self-hosted Temporal Service v1.31.0 or later. +- The Namespace's cloud provider must match the serverless compute provider. +- For self-hosted deployments, complete the self-hosted setup before following the deployment guide. +- Every Workflow must declare a versioning behavior, or the Worker must set a default versioning behavior. +- An AWS account with permissions to create and invoke Lambda functions and create IAM roles. For the exact operator actions and a preflight check, see `iam.md`. +- The AWS-specific steps require the `aws` CLI installed and configured with your AWS credentials. You may also use the AWS Console or the AWS SDKs. +- The Go SDK, Python SDK, or TypeScript SDK, depending on your language. +- The `temporal` CLI, authenticated to the target Temporal Service — Steps 4–6 and the CLI troubleshooting paths use it. See "Temporal CLI and Cloud connection" below. + +Sample projects: +- Go: [Go Lambda Worker sample](https://github.com/temporalio/samples-go/tree/main/lambda-worker) +- Python: [Python Lambda Worker sample](https://github.com/temporalio/samples-python/tree/main/lambda_worker) +- TypeScript: [TypeScript Lambda Worker sample](https://github.com/temporalio/samples-typescript/tree/main/lambda-worker) + +## Temporal CLI and Cloud connection + +Steps 4–6 and the CLI troubleshooting paths use the `temporal` CLI. Install it and authenticate it to the target Temporal Service before those steps, or commands default to `localhost:7233` and fail against Temporal Cloud. The serverless `worker deployment create-version` subcommand and its `--aws-lambda-*` flags also require a recent CLI build — see "Check the CLI version" in Step 4. + +**Authenticate to Temporal Cloud (API key).** Export environment variables (the CLI and the serverless Worker packages both read these): + +```bash +export TEMPORAL_ADDRESS="..tmprl.cloud:7233" +export TEMPORAL_NAMESPACE="." +export TEMPORAL_API_KEY="" +``` + +or configure a profile and pass `--profile prod` on each command: + +```bash +temporal --profile prod config set --prop address --value "..tmprl.cloud:7233" +temporal --profile prod config set --prop namespace --value "." +temporal --profile prod config set --prop api_key --value "" +``` + + +or configure an environment and pass `--env prod` (or set `TEMPORAL_ENV`): + +```bash +temporal env set --env prod --key address --value "..tmprl.cloud:7233" +temporal env set --env prod --key namespace --value "." +temporal env set --env prod --key api-key --value "" +``` + +**Do not assume which of the three a user has, and do not migrate them.** `--env` (YAML, `temporal env`) is the long-standing mechanism; `--profile` (TOML, `temporal config`) is newer and the CLI still marks it EXPERIMENTAL. Both are supported — work with whichever is already configured. Read the existing values rather than asking the user to re-enter them: + +```bash +temporal env get --env prod # --env mechanism +temporal config get --prop address # --profile mechanism +``` + +- For Temporal Cloud the Namespace is the fully-qualified `.`, not the bare name. +- Supplying an API key auto-enables TLS; no cert flags are needed for API-key auth. +- The `temporal ...` commands in Steps 4–6 assume this is configured. To create an API key, see `skill-temporal-ops`. + +**Temporal-side preflight.** Confirm the CLI can reach the Namespace before deploying — this is the Temporal side of the pre-deploy access check. It should list (empty is fine) without an auth or connection error: + +```bash +temporal worker deployment list +``` + +If this fails with an auth error, note first that this is a **frontend** call — it needs address, Namespace, and an API key. A control-plane login does not provide any of them; the two planes are separate: + +| | Control plane (accounts, Namespaces, API keys) | Namespace frontend (Workflows, Worker Deployments) | +|---|---|---| +| Interactive | `tcld login` | `temporal ...` with address + namespace | +| Headless | `--api-key` / `TEMPORAL_CLOUD_API_KEY` | `TEMPORAL_API_KEY` | + +**Use `tcld` for every Temporal Cloud control-plane operation** — accounts, Namespaces, API keys, users, service accounts. Do not use the unified CLI's `temporal cloud …` subcommands for them. + +**Worker Deployments and Workflows are not control-plane operations.** They live on the Namespace frontend and have no `tcld` equivalent: Steps 4–6 use `temporal worker deployment …`, authenticated with address, Namespace, and `TEMPORAL_API_KEY`. + +The two API-key variables are **different**: `TEMPORAL_CLOUD_API_KEY` authenticates `tcld`, `TEMPORAL_API_KEY` authenticates the frontend and is the one the Worker needs. Do not set one expecting the other. + +Two `tcld` mechanics worth knowing before you run it in an agent shell: + +- `tcld login --disable-pop-up` prints the URL instead of opening a browser. Auto-open is unreliable over SSH, in containers, and in remote sessions, and the user needs the URL in the conversation either way. +- `tcld` prompts for confirmation before mutating operations. Non-interactively, pass the global `--auto_confirm` (note the underscore) or set `AUTO_CONFIRM=true`, then read the resulting state back — without it the command exits clean having changed nothing. + +**Go to the API key first.** It requires no CLI login, no browser handshake, and works on every account type: + +```bash +export TEMPORAL_ADDRESS="..tmprl.cloud:7233" +export TEMPORAL_NAMESPACE="." +export TEMPORAL_API_KEY="" +``` + +Have the user create the key in the Cloud UI, signing in however they normally do, and confirm the address against the endpoint shown on the Namespace page — some Namespaces have regional endpoints that do not follow the pattern above. Never ask them to paste the key into the conversation. + +**A control-plane login is a convenience, not a prerequisite.** When it is available it saves asking: + +```bash +tcld namespace list # full Namespace objects — every name with its region and endpoint +tcld namespace get -n # one Namespace +tcld apikey create --name --duration +``` + +`apikey create` mints a key for the calling user and creates a long-lived credential in their account — offer it and get explicit approval, never silently. `tcld` is not guaranteed present: check `command -v tcld`, and read `tcld --help` for the flags you are about to pass. + +**When a control-plane login fails, stop — do not debug it, retry it, or install another CLI.** Some accounts cannot complete a `tcld` login at all, and no flag, plugin upgrade, or alternate CLI changes that. Retrying burns turns without converging, and the browser path below reaches the same end state anyway. + +**Then put the choice to the user rather than deciding for them:** fix the CLI, or work in the browser while you give the instructions. Only the control-plane steps move — the frontend work needs address, Namespace, and an API key, and no control-plane login at all, so it continues either way. In the Cloud UI, the control-plane steps are Namespace names, regions, and endpoints, and API key creation; the frontend steps are creating the Worker Deployment Version with its compute provider, and setting a version current, which the UI does automatically and the CLI does not. Where the login cannot complete, say so plainly instead of sending the user back to retry it. + +Do not proceed to Steps 4–6 on the assumption auth will work — re-run this command and confirm. + +## Step 1: Write Worker code + +The Worker handles the per-invocation lifecycle: connecting to Temporal, polling for tasks, and gracefully shutting down before the invocation deadline. + +### Install the serverless Worker package first + +**The serverless Worker package is not always part of the main SDK.** Install it explicitly before writing code — do not assume an `import` resolves just because the base SDK is present. Scaffolding a project and discovering only at build time that the package lives in its own module means backing out and redoing the module setup. + +| SDK | Install | Packaging | +|---|---|---| +| Go | `go get go.temporal.io/sdk/contrib/aws/lambdaworker` | **Separate Go module** from `go.temporal.io/sdk`, with its own version line (`v0.1.1` at the time of writing). It is *not* pulled in by the main SDK — `go get` it directly, then `go mod tidy`. | +| Python | `pip install temporalio` | `temporalio.contrib.aws.lambda_worker` ships inside the main `temporalio` package. Use `temporalio[lambda-worker-otel]` to add OpenTelemetry. | +| TypeScript | `npm install @temporalio/lambda-worker` | Separate npm package from `@temporalio/worker`, versioned independently. | + +### Verify the installed API before generating code + +These are Public Preview APIs and signatures drift between versions. Read the real surface of the version you just installed rather than writing from memory — a wrong field name costs a build cycle: + +```bash +# Go — list the exported API of the installed module version +go doc go.temporal.io/sdk/contrib/aws/lambdaworker +go doc go.temporal.io/sdk/contrib/aws/lambdaworker.Options + +# Python +python -c "import temporalio.contrib.aws.lambda_worker as m; help(m.LambdaWorkerConfig)" + +# TypeScript — check the installed version, then read its type declarations +npm ls @temporalio/lambda-worker +``` + +Specifics worth confirming this way, because they differ by SDK and are easy to get wrong from memory: + +- **Where the Task Queue lives.** In Go it is a direct field on the options object (`opts.TaskQueue`). In Python it goes through the worker-config mapping (`config.worker_config["task_queue"]`), and in TypeScript through worker options (`config.workerOptions.taskQueue`). Do not carry one shape over to another language. +- **Where registration happens.** In Go the `Register*` methods hang off the same options object; Python and TypeScript pass Workflow and Activity collections into the worker config. +- **You do not construct a client.** Connection details (address, namespace, API key) load automatically from the process environment, so `TEMPORAL_*` variables set on the function flow straight through with no client code. In a Lambda that means the `--environment` block at deploy time: no config file is bundled unless you put one there, and the operator's own CLI configuration never reaches the function (see "Operator CLI config does not reach the function" below). +- **Worker Versioning is always on.** The run-worker entry point enables it, so the only remaining decision is `Pinned` vs `AutoUpgrade` per Workflow (or a Worker-level default). + +**Fastest path:** start from the language sample linked in Prerequisites — it has a working Worker, Workflow, and Activity already wired together. The handler examples below import the Workflow and Activity from separate modules (`my_workflows`, `my_activities`). When writing from scratch, create those modules with at least one registered Workflow (declaring a versioning behavior) and one Activity, and name the entry-point file to match the `--handler` you deploy (for example, `lambda_function.py` → `--handler lambda_function.lambda_handler`). + +### Go + +Use the Go SDK's `lambdaworker` package. + +```go +package main + +import ( + lambdaworker "go.temporal.io/sdk/contrib/aws/lambdaworker" + "go.temporal.io/sdk/worker" + "go.temporal.io/sdk/workflow" +) + +func main() { + lambdaworker.RunWorker(worker.WorkerDeploymentVersion{ + DeploymentName: "my-app", + BuildID: "build-1", + }, func(opts *lambdaworker.Options) error { + opts.TaskQueue = "my-task-queue" + + opts.RegisterWorkflowWithOptions(MyWorkflow, workflow.RegisterOptions{ + VersioningBehavior: workflow.VersioningBehaviorPinned, + }) + opts.RegisterActivity(MyActivity) + + return nil + }) +} +``` + + +Versioning behavior: set per-Workflow at registration time with `workflow.VersioningBehaviorPinned` or `workflow.VersioningBehaviorAutoUpgrade`, or set a Worker-level default with `DefaultVersioningBehavior` in `DeploymentOptions`. + +### Python + +Use the Python SDK's `lambda_worker` contrib package. + +```python +from temporalio.common import WorkerDeploymentVersion +from temporalio.contrib.aws.lambda_worker import LambdaWorkerConfig, run_worker + +from my_workflows import MyWorkflow +from my_activities import my_activity + + +def configure(config: LambdaWorkerConfig) -> None: + config.worker_config["task_queue"] = "my-task-queue" + config.worker_config["workflows"] = [MyWorkflow] + config.worker_config["activities"] = [my_activity] + + +lambda_handler = run_worker( + WorkerDeploymentVersion( + deployment_name="my-app", + build_id="build-1", + ), + configure, +) +``` + + +Versioning behavior: set per-Workflow in the `@workflow.defn` decorator with `VersioningBehavior.PINNED` or `VersioningBehavior.AUTO_UPGRADE`, or set a Worker-level default with `default_versioning_behavior` in the worker config. + +```python +from temporalio import workflow +from temporalio.common import VersioningBehavior + + +@workflow.defn(versioning_behavior=VersioningBehavior.PINNED) +class MyWorkflow: + @workflow.run + async def run(self, input: str) -> str: + ... +``` + + +### TypeScript + +Use the `@temporalio/lambda-worker` package. + +```typescript +import { runWorker } from '@temporalio/lambda-worker'; +import * as activities from './activities'; + +export const handler = runWorker({ deploymentName: 'my-app', buildId: 'build-1' }, (config) => { + config.workerOptions.taskQueue = 'my-task-queue'; + config.workerOptions.workflowBundle = { + codePath: require.resolve('./workflow-bundle.js'), + }; + config.workerOptions.activities = activities; + config.workerOptions.workerDeploymentOptions!.defaultVersioningBehavior = 'PINNED'; +}); +``` + + +Use `workflowBundle` with pre-bundled code instead of `workflowsPath` to avoid webpack bundling overhead on Lambda cold starts. + +Versioning behavior: set per-Workflow with `setWorkflowOptions` in the Workflow file, or set a default for all Workflows with `defaultVersioningBehavior` in the configure callback. Values are `'AUTO_UPGRADE'` or `'PINNED'`. + +## Step 2: Deploy Lambda function + +### Build and package + +#### Go + +Cross-compile for Lambda's Linux runtime: + +```bash +GOOS=linux GOARCH=amd64 go build -tags lambda.norpc -o bootstrap ./worker +``` + + +Package the binary into a zip file: + +```bash +zip function.zip bootstrap +``` + + +**Add `CGO_ENABLED=0`, and match the architecture you deploy.** The `provided.al2023` runtime expects a self-contained binary; building with cgo enabled links against host libraries that may not resolve inside the runtime. Set `CGO_ENABLED=0` for a statically linked binary, and keep `GOARCH` consistent with the function's `--architectures` (`amd64` ↔ `x86_64`, `arm64` ↔ `arm64`). Also adjust the trailing package path to your layout — `.` when `main` is in the repo root, `./worker` when it is in a `worker/` subdirectory. A reusable script: + +```bash +#!/usr/bin/env bash +set -euo pipefail +go vet ./... # catches a missing import before the cross-compile +CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -tags lambda.norpc -o bootstrap . +zip -q function.zip bootstrap +file bootstrap # expect: ELF 64-bit ... statically linked +``` + +Run `go vet` (or a plain `go build ./...`) before the packaging build. The three-package import block above — `lambdaworker`, `worker` for `WorkerDeploymentVersion`, and `workflow` for the versioning-behavior constants — is easy to write short by one entry, and catching that locally is faster than discovering it in the cross-compile step. + +An architecture mismatch surfaces only at invocation time as an `Runtime.InvalidEntrypoint`/exec-format error, not at build or package time — the same failure class as the Python wheel mismatch below. + +A typical Go Worker zip lands around 10–15 MB, well under the 50 MB direct-upload limit. + +#### Python + +Install dependencies into a local directory for packaging, using `--platform` for Linux-compatible binaries: + +```bash +pip install --target ./package --platform manylinux2014_x86_64 --only-binary=:all: temporalio +``` + + +**Pin the download to the Lambda runtime's Python version and architecture, not your local interpreter's.** If they differ (e.g. local `3.14` vs the function's `python3.13`), add `--python-version 3.13` alongside `--only-binary=:all:` so pip fetches runtime-matching wheels, and keep `--platform` (`manylinux2014_x86_64` for `x86_64`, `manylinux2014_aarch64` for `arm64`) consistent with the function's `--architectures`. Mismatches surface as import errors only at invocation time, not at package time. + +To include OpenTelemetry support, install `temporalio[lambda-worker-otel]` instead. + +Package dependencies and application code: + +```bash +cd package && zip -r ../function.zip . && cd .. +zip function.zip lambda_function.py my_workflows.py my_activities.py +``` + + +#### TypeScript + +Build the Workflow bundle and compile the project: + +```bash +npx ts-node src/scripts/build-workflow-bundle.ts +npx tsc +``` + + +Install production dependencies and package everything: + +```bash +npm install --omit=dev +zip -r function.zip lib/ node_modules/ workflow-bundle.js +``` + + +### Deploy the Lambda function + +**A freshly created execution role may not be assumable immediately.** `create-function` can fail with an assume-role / "cannot be assumed by Lambda" error because of IAM propagation delay. Wait a few seconds and retry; it is not a policy error, so do not start rewriting the trust policy. + +**Operator CLI config does not reach the function.** All three CLI mechanisms above — exported `TEMPORAL_*` variables, `--profile`, and `--env` — configure the `temporal` CLI on the operator's machine only. The function reads its own environment, set by the `--environment` block below (or a secret store). A user with a working `--env prod` or `--profile prod` still needs every value written into that block; nothing is inherited. Treat their CLI configuration as the *source* of the values, not a substitute for setting them. + +**Resolve the values before building the block, and check they are not empty.** The heredoc below expands shell variables, which hold values only under the env-var mechanism. Under `--env` or `--profile` they are unset, and an unset variable expands to an empty string: the JSON stays valid, `create-function` succeeds, and the function deploys with `"TEMPORAL_ADDRESS":""` — failing at first invocation with a connection error that looks nothing like its cause. Populate them from whichever mechanism the user actually has (`temporal env get --env prod`, `temporal config get --prop address`), then guard: + +```bash +: "${TEMPORAL_ADDRESS:?resolve this before deploying}" +: "${TEMPORAL_NAMESPACE:?resolve this before deploying}" +: "${TEMPORAL_API_KEY:?resolve this before deploying}" +``` + +**Pass the API key through a file or shell variable, not inline in the shell history.** Building the `--environment` block inline puts the API key into shell history and into any command echo. Reference it from the environment and pass the block via a file instead: + +```bash +cat > /tmp/lambda-env.json < \ + --zip-file fileb://function.zip \ + --timeout 600 \ + --memory-size 256 \ + --environment '{"Variables":{"HOME":"/tmp","TEMPORAL_ADDRESS":":7233","TEMPORAL_NAMESPACE":"","TEMPORAL_API_KEY":""}}' +``` + + +- `--runtime`: `provided.al2023` for custom Go binaries. +- `--handler`: `bootstrap` when using the `provided.al2023` custom runtime. + +#### Python + +```bash +aws lambda create-function \ + --function-name my-temporal-worker \ + --runtime python3.13 \ + --handler lambda_function.lambda_handler \ + --role \ + --zip-file fileb://function.zip \ + --timeout 600 \ + --memory-size 256 \ + --environment '{"Variables":{"TEMPORAL_ADDRESS":":7233","TEMPORAL_NAMESPACE":"","TEMPORAL_API_KEY":""}}' +``` + + +- `--runtime`: `python3.13` (or another supported Python version). +- `--handler`: `lambda_function.lambda_handler` (entry point in `module.function` format, must point to the handler returned by `run_worker`). + +#### TypeScript + +```bash +aws lambda create-function \ + --function-name my-temporal-worker \ + --runtime nodejs22.x \ + --handler lib/index.handler \ + --role \ + --zip-file fileb://function.zip \ + --timeout 600 \ + --memory-size 256 \ + --environment '{"Variables":{"HOME":"/tmp","TEMPORAL_ADDRESS":":7233","TEMPORAL_NAMESPACE":"","TEMPORAL_API_KEY":""}}' +``` + + +- `--runtime`: `nodejs22.x` (or another supported Node.js version, 20+). +- `--handler`: `lib/index.handler` (entry point in `module.export` format, must point to the handler exported by `runWorker`). + +### Wait for the function to become Active + +`create-function` returns immediately with `"State": "Pending"`. The function cannot be invoked and `publish-version` fails while it is pending, so block on the state transition before the next step rather than sleeping a guessed interval: + +```bash +aws lambda wait function-active-v2 --function-name my-temporal-worker +``` + +Use `aws lambda wait function-updated-v2` after `update-function-code` for the same reason (see `versioning.md`). The `-v2` suffix is the AWS CLI v2 waiter name; on AWS CLI v1 the waiters are `function-active` and `function-updated`. If neither resolves, poll instead: + +```bash +aws lambda get-function --function-name my-temporal-worker \ + --query 'Configuration.[State,LastUpdateStatus]' --output text +``` + +### Common parameters (all SDKs) + + + +| Parameter | Description | +|---|---| +| `--role` | ARN of the Lambda execution role, which grants the function permission to run (trusted principal: `lambda.amazonaws.com`). This is separate from the role Temporal uses to invoke the function. The role must have at least the `AWSLambdaBasicExecutionRole` managed policy attached. (See `iam.md` for the execution role.) | +| `--zip-file` | Path to your packaged deployment zip. | +| `--timeout` | Invocation deadline in seconds. Maximum time each Lambda invocation can run before AWS terminates it. Set high enough for the Worker to start, process Tasks, and shut down gracefully. | +| `--memory-size` | Memory in MB allocated to each invocation. | + +**Caution:** AWS Lambda functions default to a 3-second timeout, which is too short for the Worker to start, connect to Temporal, and register the Task Queue. If the first invocation times out before the Worker polls, the Task Queue binding is never created and the Lambda is never invoked again. Always set `--timeout` high enough for the Worker to start, process Tasks, and shut down gracefully. + +### Environment variables + + + +| Variable | Description | +|---|---| +| `HOME` | Set to `/tmp` in the Go and TypeScript examples above. Lambda's filesystem is read-only outside `/tmp`, so anything the runtime or config loader resolves relative to the home directory needs a writable target. The docs omit it from the Python example; including it there is harmless. | +| `TEMPORAL_ADDRESS` | Temporal frontend address (e.g., `..tmprl.cloud:7233`). | +| `TEMPORAL_NAMESPACE` | Temporal Namespace. For Temporal Cloud, the fully-qualified `.`, not the bare name. | +| `TEMPORAL_TASK_QUEUE` | Task Queue name. Overrides the value set in code. | +| `TEMPORAL_TLS_CLIENT_CERT_PATH` | Path to the TLS client certificate file for mTLS authentication. | +| `TEMPORAL_TLS_CLIENT_KEY_PATH` | Path to the TLS client key file for mTLS authentication. | +| `TEMPORAL_API_KEY` | API key for API key authentication. Supplying it auto-enables TLS; mTLS cert paths are not needed. | + +The serverless Worker packages read environment variables and configuration files automatically at startup. For the full list of supported environment variables, config file format, and profiles, see the Environment configuration docs (`/develop/environment-configuration`). + +Sensitive values like TLS keys and API keys should be encrypted at rest. + +The `--environment` examples above pass `TEMPORAL_API_KEY` inline for brevity — **that is acceptable for development only.** For production, store the API key (or TLS private key) in AWS Secrets Manager or SSM Parameter Store, grant the *execution* role `secretsmanager:GetSecretValue` (or `ssm:GetParameter`), and load it at cold start before the Worker initializes — for example, at module scope in the handler file, fetch the secret and set `os.environ["TEMPORAL_API_KEY"]` so the serverless Worker package reads it at startup. Do not commit key values into the `--environment` block for production functions. + +For updating the function code and publishing immutable versions, see `versioning.md`. + +## Step 3: Configure IAM for Temporal invocation + +Step 3 (execution role, Temporal invocation role, and CloudFormation for Temporal Cloud and self-hosted) lives in `iam.md`. Complete it before Step 4. + +## Step 4: Create Worker Deployment Version + +Create a Worker Deployment Version with a compute provider that points to your Lambda function. The compute configuration tells Temporal how to invoke your Worker: the provider type (`aws-lambda`), the Lambda function ARN, and the IAM role to assume. The deployment name and build ID must match the values in your Worker code. + +### Using Temporal UI + + + +1. In the Temporal UI, open your Namespace. +2. In the left pane, select **Workers**. +3. Click **Create Worker Deployment** in the upper right corner. +4. Under **Configuration**, enter a **Name** and **Build ID** (must match `DeploymentName` and `BuildID` in your Worker code). +5. Under **Compute**, select **AWS Lambda** and provide: + - **Lambda ARN**: the ARN of your Lambda function. + - **IAM Role ARN**: the ARN of the role Temporal assumes to invoke your Lambda function (the `RoleARN` output from the CloudFormation stack). This is not the Lambda execution role or your own IAM user/role. + - **External ID**: the same value passed to the CloudFormation template. +6. Click **Save**. + +When you create a version through the UI, the version is automatically set as current. + +### Using Temporal CLI + +Use the CLI for manual setup, shell scripts, and CI/CD pipelines. When you create a version through the CLI, you must set the version as current as a separate step. + +**Check the CLI version before relying on these commands.** The `worker deployment create-version` subcommand and its `--aws-lambda-*` flags only exist in recent Temporal CLI builds, and a CLI old enough to lack them fails in a way that looks like a syntax mistake: + +```bash +temporal --version +temporal worker deployment create-version --help +``` + +If the subcommand or the flags are missing, upgrade. Observed bounds: v1.5.0 (Homebrew) lacked `create-version` entirely; v1.8.0 (standalone) had the serverless flags. The exact minimum version is unconfirmed against the CLI changelog, so treat those as bounds rather than a threshold. A current standalone build can be installed alongside a package-managed one without disturbing it — worth doing rather than upgrading a CLI the user may depend on elsewhere. + +First, create the Worker Deployment if it does not already exist: + +```bash +temporal worker deployment create \ + --namespace \ + --name my-app +``` + + +Then create the version with the compute provider configuration: + +```bash +temporal worker deployment create-version \ + --namespace \ + --deployment-name my-app \ + --build-id build-1 \ + --aws-lambda-function-arn \ + --aws-lambda-assume-role-arn \ + --aws-lambda-assume-role-external-id +``` + + +| Flag | Description | +|---|---| +| `--deployment-name` | Worker Deployment name. Must match `DeploymentName` in your Worker code. | +| `--build-id` | Worker Deployment Version build ID. Must match `BuildID` in your Worker code. | +| `--aws-lambda-function-arn` | Qualified versioned ARN of the Lambda function Temporal invokes for this version (for example, `function:my-worker:5`). An unqualified ARN is also accepted for development. | +| `--aws-lambda-assume-role-arn` | IAM role Temporal assumes to invoke the function. This is the `RoleARN` output from the CloudFormation stack. This is not the Lambda execution role or your own IAM user/role. | +| `--aws-lambda-assume-role-external-id` | External ID configured in the IAM role trust policy. | + + +### Validate connection + +Go to **Workers** > **Deployments** > select your deployment > open the **Actions** menu on the version and click **Validate Connection**. This checks that Temporal can assume the IAM role and invoke the function. + +### Checkpoint: confirm the validation invocation actually bound the Task Queue + +**Do this before Step 5.** Creating the version triggers one validation invocation of the Lambda. If it succeeded, the Worker connected and registered its Task Queue, and `describe-version` lists that Task Queue for both workflow and activity types: + +```bash +temporal worker deployment describe-version \ + --deployment-name my-app --build-id build-1 --report-task-queue-stats +``` + +Task Queues listed = the invocation role, the Lambda package, the env vars, and the timeout are all working end-to-end. This is the cheapest early signal in the whole setup, and it isolates a first-invocation failure to Step 2/3 *before* current-version routing adds another variable. If no Task Queues are listed, stop here and go to `diagnostics.md` ("Failed first invocation") — setting the version current will not fix it. + +## Step 5: Set version as current + +If you created the version through the Temporal UI, the version is already current — skip this step. + +If you used the CLI, set the version as current. Without this step, tasks on the Task Queue will not route to the version, and Temporal will not invoke the Lambda function. + +```bash +temporal worker deployment set-current-version \ + --deployment-name my-app \ + --build-id build-1 \ + --yes +``` + + +**`set-current-version` asks for interactive confirmation.** Without `--yes` (`-y`) it prompts, and run non-interactively (scripts, CI, or an agent shell) it exits without applying the change — which reads as a silent no-op: the command appears to succeed but the version never becomes current. Pass `--yes` for any non-interactive use. `set-ramping-version` behaves the same way. Note that `delete-version` does *not* take `--yes` — its gating flag is `--skip-drainage`. + +Confirm it took effect before moving on: + +```bash +temporal worker deployment describe --name my-app +``` + +## Step 6: Verify deployment + +Start a Workflow on the same Task Queue to confirm that Temporal invokes your Lambda Worker. + +```bash +temporal workflow start \ + --task-queue my-task-queue \ + --type MyWorkflow \ + --input '"Hello, serverless!"' +``` + + +Verify the invocation by checking: + +- **Temporal UI:** The Workflow execution should show task completions in the event history. +- **AWS CloudWatch Logs:** The Lambda function's log group (`/aws/lambda/my-temporal-worker`) should show invocation logs with the Worker startup, task processing, and graceful shutdown. Requires the execution role to have CloudWatch Logs permissions (included in `AWSLambdaBasicExecutionRole`). + +If the Workflow does not progress or the Lambda is not invoked, see `diagnostics.md`. + +## Teardown + +**Record what you create, as you create it.** These are live, billable AWS resources spread across three services plus Temporal Cloud, and their names are only knowable from the run that created them. Keep a running inventory — function name and published version numbers, execution role name, CloudFormation stack name and role name, region, deployment name and build ID — and hand it to the user at the end. Deliver the inventory before offering teardown, and do not write a teardown script until they ask for one. Reconstructing the inventory later means scanning the account, which the skill otherwise tells you not to do. + +To remove a serverless Worker deployment (for example, after an evaluation), tear down in this order so nothing is left invoking or being invoked. + +**The Lambda function must go before the Worker Deployment Version, not after.** The intuitive order — Temporal first, so nothing is left invoking — deadlocks, because the version refuses to delete while pollers are active and the pollers *are* the still-running Lambda. Deleting the function is what drains them. Expect the Temporal side to be split across the sequence for this reason. + +1. Unset the current version. A Current version cannot be deleted, and a Worker Deployment with versions cannot be deleted either — so this deadlock has to be broken first. `--skip-drainage` does not help here; it waives the draining check, not the Current restriction. + ```bash + temporal worker deployment set-current-version \ + --deployment-name my-app --unversioned --yes + ``` + If other versions exist and should keep serving, set one of them current instead of `--unversioned`. +2. Delete the Lambda function. This removes all published versions, and ends the invocation that is still polling: + ```bash + aws lambda delete-function --function-name my-temporal-worker + ``` +3. Wait for the version to drain, then delete it. Poller registration is server-side and expires on a TTL after the Worker stops, so `delete-version` can still fail with active pollers for a while after the function is gone — and the poller list can read empty before drainage has actually completed. Poll for `DrainageStatus: drained` rather than retrying blind: + ```bash + temporal worker deployment describe-version \ + --deployment-name my-app --build-id build-1 + temporal worker deployment delete-version \ + --deployment-name my-app --build-id build-1 + ``` +4. Delete the Worker Deployment itself, once it has no versions left. This stops its WCI (one WCI runs per version with a compute provider); confirm the WCI has moved to `Completed`, and that any *other* deployment's WCI is still `Running`. + ```bash + temporal worker deployment delete --name my-app + ``` + The deployment may still appear in `list` output immediately afterward — index lag, not a failed delete. Confirm with `describe`. +5. Delete the CloudFormation stack that created the Temporal invocation role — **only if this deployment created it.** One invocation role can authorize several Worker Lambdas, so a pre-existing stack may still be in use by another deployment; in that case remove just this function's ARN from its `LambdaFunctionARNs` instead (see `iam.md`). + ```bash + aws cloudformation delete-stack --stack-name --region + ``` +6. If you created a dedicated execution role, delete it — **detach its managed policies first**, or `delete-role` fails with `DeleteConflict: Cannot delete entity, must detach all policies first`: + ```bash + ROLE= + aws iam list-attached-role-policies --role-name "$ROLE" \ + --query 'AttachedPolicies[].PolicyArn' --output text \ + | tr '\t' '\n' | while read -r P; do aws iam detach-role-policy --role-name "$ROLE" --policy-arn "$P"; done + aws iam delete-role --role-name "$ROLE" + ``` + Skip this if the execution role predates your deployment or is shared with other functions. +7. Delete the CloudWatch log group. **`delete-function` does not remove it** — the log group and its retained events survive the function and keep accruing storage charges: + ```bash + aws logs delete-log-group --log-group-name /aws/lambda/my-temporal-worker + ``` +8. **Ask whether to revoke the Temporal Cloud API key** — do not revoke it as a matter of course. The key is account-scoped, not deployment-scoped: tearing this deployment down does not mean the user is finished with Temporal Cloud, and if the key was created during this run (see `skill-temporal-ops`) it is the one they now need for their next deploy, for `tcld`, and for every other Worker in the Namespace. Keep it unless they say otherwise. On a yes, revoke it **last** — it is the credential authenticating every Temporal command above it. Like `set-current-version`, `tcld apikey delete` prompts for confirmation and, run non-interactively, exits without deleting anything; the exit code looks clean while the key is still live. Pass `--auto_confirm` and confirm from the `list` output that the key is gone rather than trusting the exit code. + ```bash + tcld apikey delete --id --auto_confirm + tcld apikey list + ``` diff --git a/plugins/temporal/skills/temporal-serverless/references/aws-lambda/versioning.md b/plugins/temporal/skills/temporal-serverless/references/aws-lambda/versioning.md new file mode 100644 index 0000000..1959d63 --- /dev/null +++ b/plugins/temporal/skills/temporal-serverless/references/aws-lambda/versioning.md @@ -0,0 +1,64 @@ +# AWS Lambda — Versioning, updates & rollback + + + +For the conceptual model — Temporal Worker Deployment Versions vs. Lambda function versions, and Pinned vs. Auto-Upgrade behavior — see `../concepts.md` ("Worker Versioning with Serverless Workers"). + +## The full redeploy sequence + +Shipping a change touches code, AWS, and Temporal in a fixed order. Skipping or reordering a step is the most common way to end up with a version that is never invoked: + +1. Bump the Build ID in the Worker code (`build-1` → `build-2`). It must match the `--build-id` you will register in step 5. +2. Rebuild and repackage (`setup.md`, Step 2). +3. `aws lambda update-function-code`, then `aws lambda wait function-updated-v2`. +4. `aws lambda publish-version` → note the new qualified ARN (`...:function:my-temporal-worker:2`). +5. `temporal worker deployment create-version` with the new build ID **and** the new qualified ARN. +6. Confirm the validation invocation bound the Task Queue (`describe-version --report-task-queue-stats`). +7. `temporal worker deployment set-current-version --yes` to shift traffic. + +Two things that bite here: the Build ID lives in **three** places that must agree (Worker code, `--build-id`, and the Lambda version description if you use one), and the invocation role must authorize the *new* qualified ARN — which it does automatically only if you granted the `:*` wildcard form (see `iam.md`). A role scoped to `function:my-temporal-worker:1` silently fails to invoke `:2`. + +## Update existing function + +```bash +aws lambda update-function-code \ + --function-name my-temporal-worker \ + --zip-file fileb://function.zip +``` + + +**Wait for the update to settle before publishing.** `update-function-code` returns while `LastUpdateStatus` is still `InProgress`, and `publish-version` in that window either fails (`ResourceConflictException`) or, worse, snapshots the *previous* code. Block on the transition: + +```bash +aws lambda wait function-updated-v2 --function-name my-temporal-worker +``` + +(`function-updated` on AWS CLI v1; see `setup.md` for the polling fallback.) + +After updating, increment the Build ID in your Worker code and publish a new Lambda function version. + +## Publish a Lambda function version + +For production, create an immutable snapshot of your Lambda code after creating the function and after each `update-function-code`, and maintain a one-to-one mapping between each Lambda function version and each Temporal Worker Deployment Version Build ID. + +```bash +aws lambda publish-version \ + --function-name my-temporal-worker \ + --description "Build ID build-5" +``` + + +The command prints the `FunctionArn` for the new version, for example `arn:aws:lambda:us-east-1:123456789012:function:my-temporal-worker:5`. Use this qualified versioned ARN when you create the Worker Deployment Version. + +For development or non-critical workloads, you can skip `publish-version` and use an unqualified ARN to iterate faster. + +To roll back, revert the Temporal Current Version with `temporal worker deployment set-current-version`. The previous Worker Deployment Version still points at its original Lambda function version and is ready to receive traffic again. + +```bash +temporal worker deployment set-current-version \ + --deployment-name my-app --build-id build-1 --yes +``` + +This is the whole rollback — one Temporal-side command, no AWS change. Because each Build ID is pinned to its own immutable Lambda version, the old code is still deployed and invocable; nothing needs to be rebuilt or re-uploaded. Keep the previous version registered (do not `delete-version` right after a release) so this path stays available. Two caveats: `--yes` is required non-interactively or the command silently does nothing, and rolling back the Current Version does not move Workflows already pinned to the new build — it only redirects *new* Workflow executions. diff --git a/plugins/temporal/skills/temporal-serverless/references/concepts.md b/plugins/temporal/skills/temporal-serverless/references/concepts.md new file mode 100644 index 0000000..f5d141d --- /dev/null +++ b/plugins/temporal/skills/temporal-serverless/references/concepts.md @@ -0,0 +1,244 @@ +# Serverless Workers — Concepts + + + +## Release status + +**AWS Lambda — Public Preview since July 30, 2026.** Open to all Temporal Cloud customers. There is no access request, no support ticket, and no manual toggle to enable: a customer selects "AWS Lambda (Public Preview)" as the compute provider in the UI and sets up their Worker Deployment directly. Never route a user to support to "get access" for Lambda. + +AWS Lambda is the only compute provider this skill supports. Do not adapt the Lambda material to any other provider. + +Public Preview is not General Availability. APIs are still evolving and may be subject to backwards-incompatible changes between versions — pin SDK and CLI versions for anything long-lived, and read the installed package's real API surface rather than writing from memory. + +## What is a Serverless Worker? + +A Serverless Worker is a Temporal Worker that runs on serverless compute instead of a long-lived process. +There is no always-on infrastructure to provision or scale. Temporal invokes the Worker when Tasks arrive on a Task Queue, and the Worker shuts down when the work is done. + +A Serverless Worker uses the same Temporal SDKs as a traditional long-lived Worker. It registers Workflows and Activities the same way. The difference is in the lifecycle: instead of the Worker starting and polling continuously, Temporal invokes the Serverless Worker on demand, the Worker starts, processes available Tasks, and then shuts down. + +Serverless Workers require Worker Versioning. Each Serverless Worker must be associated with a Worker Deployment Version that has a compute provider configured. + +Each Workflow must have an `AutoUpgrade` or `Pinned` versioning behavior, set per-Workflow or as a Worker-level default. + +## How Serverless invocation works + +With long-lived Workers, the Worker process starts, connects to Temporal, and polls a Task Queue for work. Temporal does not need to know anything about the Worker's infrastructure. + +With Serverless Workers, Temporal starts the Worker. + +### Worker Controller Instance (WCI) + +The Worker Controller Instance (WCI) is a system Workflow that scales Serverless Workers based on Task Queue conditions. +One WCI Workflow runs per Worker Deployment Version that has a compute provider configured. The WCI runs in the same Namespace as your Worker Deployment. + +The WCI responds to two triggers: sync match failures and Task Queue backlog. When either trigger fires, the WCI produces a scaling action, such as invoking the configured compute provider (for example, calling AWS Lambda's `InvokeFunction` API) to start new Workers. + +You can list WCI Workflows in your Namespace: + +```bash +temporal workflow list \ + --namespace \ + --query 'TemporalNamespaceDivision = "TemporalWorkerControllerInstance"' +``` + + +WCI Workflow IDs follow the pattern `temporal-sys-worker-controller-instance::`. + +You can inspect a WCI Workflow's history to see its recent Activity results: + +```bash +temporal workflow show \ + --namespace \ + --workflow-id 'temporal-sys-worker-controller-instance::' +``` + + +### Invocation flow + +The invocation flow works as follows: + +1. A Task is submitted (for example, `StartWorkflow` or `ScheduleActivity`). +2. The Matching Service attempts to route the Task directly to an available Worker (a sync match). +3. If a Worker is available, the Task is routed to that Worker. +4. If no Worker is available (sync match fails), the Matching Service pushes a signal to the WCI, and the WCI invokes the configured compute provider. +5. The Serverless Worker starts, creates a Temporal Client, and begins polling the Task Queue. +6. The Worker processes available Tasks until it exits (see Worker lifecycle). + +Each invocation is independent. The Worker creates a fresh client connection on every invocation. There is no connection reuse or shared state across invocations. + +## Autoscaling + +The WCI automatically scales Serverless Workers based on Task Queue signals. When Tasks arrive and no Worker is available, the WCI invokes new Workers. When the Tasks are done, Workers exit and scale to zero. + +The WCI uses two signals to decide when to invoke new Workers: + +### Sync match failure + +When a Task is submitted, the Matching Service attempts to route it directly to an available Worker. If no Worker is available, the sync match fails, and the Matching Service pushes a signal to the WCI. The WCI then invokes a new Worker. This is the primary scaling path. + +Because the Matching Service pushes match failures to the WCI as they happen rather than the WCI polling on a timer, latency stays low and scaling is responsive. + +### Task Queue backlog + +The WCI monitors Task Queue metadata to determine whether pending Tasks exist without enough Workers to process them. If there are Tasks on the queue and not enough Workers, the WCI invokes additional Workers. + +## Scaling with long-lived Workers + +Serverless Workers can share a Task Queue with long-lived Workers. Because Serverless Workers are only invoked on sync match failure, Serverless Workers only pick up Tasks that no long-lived Worker was available to handle. In practice, the Serverless Workers act as spillover capacity for the long-lived fleet. + +**Warning:** If you configure Serverless and long-lived Workers on the same Task Queue, do not enable dynamic scaling on the long-lived Workers. The two groups cannot coordinate their scaling behavior. If both scale dynamically, the long-lived Workers may scale up to handle the same Tasks that Temporal is simultaneously invoking Serverless Workers for, leading to unnecessary invocations and unpredictable scaling. + +## Worker lifecycle + +A single Serverless Worker invocation has three phases: init, work, and shutdown. + +### Init phase + +The Worker initializes and establishes a client connection to Temporal. + +### Work phase + +The Worker polls the Task Queue and processes Tasks. + +### Shutdown phase + +The Worker stops polling, waits for in-flight Tasks to finish, and runs any shutdown hooks (for example, OpenTelemetry telemetry flushes). Shutdown begins before the invocation deadline so the Worker can exit cleanly before the compute provider forcibly terminates the execution environment. + +### Tuning for long-running Activities + +If your Worker handles long-running Activities, set these three values together: + +- **Worker stop timeout > longest Activity runtime.** Gives in-flight Activities enough time to finish after polling stops. +- **Shutdown deadline buffer > Worker stop timeout + shutdown hook time.** Ensures the drain and any shutdown hooks complete before the compute provider terminates the environment. +- **Invocation deadline > longest Activity runtime + shutdown deadline buffer.** Set on the compute provider to give each invocation enough total runtime. + +If your longest-running Activity runs longer than half the maximum invocation deadline, use Activity Heartbeats to record the state of the Activity execution so that the next retry can pick up where it left off. + +Example: if your longest Activity runtime is 5 minutes, and your shutdown hooks take 3 seconds, set the Worker stop timeout to more than 5 minutes, and the shutdown deadline buffer to more than 303 seconds (5 minutes + 3 seconds). Set your invocation deadline to at least 10 minutes and 3 seconds. + +The Worker stop timeout controls how long the Worker waits for in-flight Tasks to finish after it stops polling. The shutdown deadline buffer controls how much time before the invocation deadline the Worker stops polling for Tasks. + +Raising only the shutdown deadline buffer makes the Worker stop polling earlier, but does not give in-flight Tasks any more time to complete. + +Raising only the Worker stop timeout does not make the Worker stop polling earlier, which means the compute provider might terminate the Worker before the full stop timeout completes. + +## Failure handling + +Serverless Workers rely on Temporal's standard retry and timeout semantics to recover from failures. + +### Worker crash + +If a Worker invocation crashes (out of memory, unhandled exception, etc.): + +- The Activity Timeout fires after the configured duration. +- Temporal retries the Activity on a different Worker invocation. +- No manual intervention is required. + +### Provider concurrency limit + +If the compute provider's concurrency limit is reached (for example, AWS Lambda account concurrency): + +- Further invocations from the WCI fail. +- Tasks remain in the Task Queue backlog. No data loss occurs. +- Processing slows until concurrency frees up. + +### Resource exhaustion across Activity slots + +By default, a single Worker invocation may run multiple Activity slots. A crash or resource exhaustion in one Activity can affect other Activities running in the same invocation. + +To isolate Activities from each other: + +- Split Workflow and Activity Workers into separate compute functions. +- Set Activity slots to 1 per invocation. + +With single-slot configuration, each Activity gets a dedicated execution environment. + +## Constraints + + + +| Constraint | Detail | +|---|---| +| Activity duration | Must complete within the compute provider's invocation limit (minus shutdown deadline buffer). For AWS Lambda, the maximum is 15 minutes. | +| Workflow duration | No limit. Workflows of any duration work, regardless of the invocation timeout. A Workflow runs across as many invocations as needed. | +| Worker code | Same Temporal SDK Worker code, using the serverless Worker package for your SDK. | +| Versioning | Worker Versioning is required. Each Workflow must have an `AutoUpgrade` or `Pinned` behavior, set per-Workflow or as a Worker-level default. | + +## Worker Versioning with Serverless Workers + +Serverless Workers require Worker Versioning, and the compute provider must invoke a stable, immutable build for each Worker Deployment Version. With AWS Lambda, this means aligning two versioning systems: + +- **Temporal Worker Deployment Versions** — identified by deployment name and Build ID. Each Workflow runs against a specific Worker Deployment Version (Pinned) or moves between them on routing changes (Auto-Upgrade). +- **AWS Lambda function versions** — immutable numbered snapshots of your Lambda function code (`1`, `2`, `3`, ...). + +For production workloads, map each Worker Deployment Version to exactly one Lambda function version, and configure the compute provider with the qualified versioned ARN for that Lambda version (for example, `arn:aws:lambda:us-east-1:123:function:my-worker:5`). + +For development or non-critical workloads, you can use an unqualified ARN to iterate without publishing a new Lambda function version each time. + +**Caution:** An unqualified ARN (no version suffix) points at `$LATEST`, which changes on every redeploy. Without a versioned ARN, deploying replay-unsafe code causes non-determinism errors for in-flight Workflows, even for Workflows annotated as Pinned. + +The choice of Pinned or Auto-Upgrade controls how Workflows move between Worker Deployment Versions in Temporal. It does not change how a Worker Deployment Version targets Lambda. Both behaviors expect a versioned ARN that points at one immutable Lambda function version. + +| Versioning Behavior | With versioned Lambda ARN | Without versioned Lambda ARN | +|---|---|---| +| **Pinned** | Existing Workflows stay on their original Lambda function version until they complete. | Existing Workflows stay on their original Worker Deployment Version, but the underlying Lambda code has already changed since `$LATEST` updated at redeploy. The new code must be replay-compatible. | +| **Auto-Upgrade** | Existing Workflows move to the new Worker Deployment Version and its new Lambda function version at the next Workflow Task after you move the Current Version. | The Lambda redeploy already changed the code for all versions. Setting the Current Version only changes routing, not which code runs. | + + +See `aws-lambda/versioning.md` for the step-by-step `aws lambda publish-version` workflow and `aws-lambda/setup.md` (Step 4) for how to configure the compute provider with a versioned ARN. + +## Compute providers + +A compute provider is the configuration that tells Temporal how to invoke a Serverless Worker. The compute provider is set on a Worker Deployment Version and specifies the provider type, the invocation target, and the credentials Temporal needs to trigger the invocation. + +For example, an AWS Lambda compute provider includes the Lambda function ARN and the IAM role that Temporal assumes to invoke the function. + +Compute providers are only needed for Serverless Workers. Traditional long-lived Workers do not require a compute provider because the Worker process lifecycle is not managed by the Temporal server. + +### Supported providers + + + +| Provider | Description | +|---|---| +| AWS Lambda | Temporal assumes an IAM role in your AWS account to invoke a Lambda function. | + +## Why use Serverless Workers? + + + +- **Reduce operational overhead.** No always-on infrastructure to manage and no autoscaling policies to tune. Temporal and the compute provider handle invocation and scaling. +- **Get started faster.** Deploying a Worker is as simple as deploying a function. No Kubernetes, container orchestration, or scaling strategy required. +- **Scale automatically.** The compute provider handles scaling natively. When traffic drops, instances scale down. When there is no work, there is no compute running. +- **Pay only for what you use.** Workers run only when Tasks are available. For low or intermittent volume workloads, this pay-per-invocation model can significantly reduce compute costs. + +## When to use Serverless Workers + + + +Good fit when: + +- Workloads are bursty or event-driven (order processing, notifications, webhook handlers). +- Traffic is low or intermittent. +- You want a simpler getting-started path. +- Your organization has standardized on serverless. +- You serve multiple tenants with infrequent workloads. + +May not be ideal when: + + + +- Activities are long-running and cannot be interrupted. AWS Lambda has a 15-minute execution limit. Activities that run longer and cannot be broken into smaller steps need a different hosting strategy or a provider with longer limits. +- Workloads require sustained high throughput. Long-lived Workers on dedicated compute may be more cost-effective and performant. +- You need persistent connections. Some features require a persistent connection between the Worker and Temporal, which serverless invocations do not maintain. + +## How Serverless Workers compare to long-lived Workers + + + +| | Long-lived Worker | Serverless Worker | +|---|---|---| +| **Lifecycle** | Long-lived process that runs continuously. | Invoked on demand. Starts and stops per invocation. | +| **Scaling** | You manage scaling (Kubernetes HPA, instance count, etc.). | Temporal invokes additional instances as needed, within the compute provider's concurrency limits. | +| **Connection** | Persistent connection to Temporal. | Fresh connection on each invocation. | diff --git a/plugins/temporal/skills/temporal-serverless/references/sdk-configuration.md b/plugins/temporal/skills/temporal-serverless/references/sdk-configuration.md new file mode 100644 index 0000000..c0ccca4 --- /dev/null +++ b/plugins/temporal/skills/temporal-serverless/references/sdk-configuration.md @@ -0,0 +1,221 @@ +# SDK Configuration for Serverless Workers + + + +## Go SDK + +### Package + +Import: `lambdaworker "go.temporal.io/sdk/contrib/aws/lambdaworker"` + +Install: `go get go.temporal.io/sdk/contrib/aws/lambdaworker` — **this is a separate Go module** from `go.temporal.io/sdk`, versioned independently (`v0.1.1` at the time of writing). Having the main SDK in `go.mod` does not make it importable; add it explicitly, then `go mod tidy`. Verify the installed surface with `go doc go.temporal.io/sdk/contrib/aws/lambdaworker` before generating code — the API is Public Preview and drifts. + +### Entry point + +`lambdaworker.RunWorker` — starts a Lambda-based Worker. Pass a `WorkerDeploymentVersion` and a callback that registers Workflows and Activities. + +### Configure callback + +The `Options` callback gives access to the same registration methods as a traditional Worker: `RegisterWorkflow`, `RegisterWorkflowWithOptions`, `RegisterActivity`, `RegisterActivityWithOptions`, and `RegisterNexusService`. + +### Versioning behavior + +Set per-Workflow at registration time with `workflow.VersioningBehaviorPinned` or `workflow.VersioningBehaviorAutoUpgrade`. +Or set a Worker-level default with `DefaultVersioningBehavior` in `DeploymentOptions`. + +### Lambda-tuned defaults + + + +| Setting | Lambda default | +|---|---| +| `MaxConcurrentActivityExecutionSize` | 2 | +| `MaxConcurrentWorkflowTaskExecutionSize` | 10 | +| `MaxConcurrentLocalActivityExecutionSize` | 2 | +| `MaxConcurrentNexusTaskExecutionSize` | 5 | +| `MaxConcurrentActivityTaskPollers` | 1 | +| `MaxConcurrentWorkflowTaskPollers` | 2 | +| `MaxConcurrentNexusTaskPollers` | 1 | +| `WorkerStopTimeout` | 5 seconds | +| `DisableEagerActivities` | Always true | +| Sticky cache size | 100 | +| `ShutdownDeadlineBuffer` | 7 seconds | + +These are the same `worker.Options` available to any Temporal Worker, just with lower values for Lambda's constrained environment. Except for `ShutdownDeadlineBuffer`, which is specific to the `lambdaworker` package. + +`DisableEagerActivities` is always true and cannot be overridden. Eager Activities require a persistent connection, which Lambda invocations don't maintain. + +`ShutdownDeadlineBuffer` controls how much time before the Lambda deadline the Worker begins its graceful shutdown. The default is `WorkerStopTimeout` + 2 seconds. + +If your Worker handles long-running Activities, increase `WorkerStopTimeout`, `ShutdownDeadlineBuffer`, and the Lambda invocation deadline (`--timeout`) together. + +### Connection configuration + +The `lambdaworker` package automatically loads Temporal client configuration from a TOML config file and environment variables (see the Environment Configuration docs, `/develop/environment-configuration`). + +TOML config file resolution order: + +1. `TEMPORAL_CONFIG_FILE` environment variable, if set. +2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). +3. `temporal.toml` in the current working directory. + +The file is optional. If absent, only environment variables are used. + +--- + +## Python SDK + +### Package + +Import: `from temporalio.contrib.aws.lambda_worker import LambdaWorkerConfig, run_worker` + +Install: `pip install temporalio` — the contrib module ships inside the main package here (unlike Go and TypeScript, which need a separate dependency). Use `temporalio[lambda-worker-otel]` for OpenTelemetry support. + +### Entry point + +`run_worker` — takes a `WorkerDeploymentVersion` and a configure callback, returns a Lambda handler. + +### Configure callback + +The `configure` callback receives a `LambdaWorkerConfig` dataclass with fields pre-populated with Lambda-appropriate defaults. Set the Task Queue, Workflows, and Activities through `worker_config`, which accepts the same keyword arguments as the `Worker` constructor. + +### Versioning behavior + +Set per-Workflow in the `@workflow.defn` decorator: `VersioningBehavior.PINNED` or `VersioningBehavior.AUTO_UPGRADE`. +Or set a Worker-level default with `default_versioning_behavior` in the worker config. + +### Lambda-tuned defaults + + + +| Setting | Lambda default | +|---|---| +| `max_concurrent_activities` | 2 | +| `max_concurrent_workflow_tasks` | 10 | +| `max_concurrent_local_activities` | 2 | +| `max_concurrent_nexus_tasks` | 5 | +| `workflow_task_poller_behavior` | `SimpleMaximum(2)` | +| `activity_task_poller_behavior` | `SimpleMaximum(1)` | +| `nexus_task_poller_behavior` | `SimpleMaximum(1)` | +| `graceful_shutdown_timeout` | 5 seconds | +| `max_cached_workflows` | 30 | +| `disable_eager_activity_execution` | Always `True` | +| `shutdown_deadline_buffer` | 7 seconds | + +`disable_eager_activity_execution` is always `True` and cannot be overridden. Eager Activities require a persistent connection, which Lambda invocations don't maintain. + +`shutdown_deadline_buffer` is specific to the `lambda_worker` package. It controls how much time before the Lambda deadline the Worker begins its graceful shutdown. The default is `graceful_shutdown_timeout` + 2 seconds. + +If your Worker handles long-running Activities, increase `graceful_shutdown_timeout`, `shutdown_deadline_buffer`, and the Lambda invocation deadline (`--timeout`) together. + +### Connection configuration + +The `lambda_worker` package automatically loads Temporal client configuration from a TOML config file and environment variables (see the Environment Configuration docs, `/develop/environment-configuration`). + +TOML config file resolution order: + +1. `TEMPORAL_CONFIG_FILE` environment variable, if set. +2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). +3. `temporal.toml` in the current working directory. + +The file is optional. If absent, only environment variables are used. + +--- + +## TypeScript SDK + +### Package + +Import: `import { runWorker } from '@temporalio/lambda-worker'` + +Install: `npm install @temporalio/lambda-worker` — a separate npm package from `@temporalio/worker`, versioned independently. + +### Entry point + +`runWorker` — creates a Lambda handler that runs a Temporal Worker. Pass a deployment version and a configure callback. + +### Configure callback + +Set Worker options via `config.workerOptions`. For Workflow code, use `workflowBundle` with pre-bundled code instead of `workflowsPath` to avoid webpack bundling overhead on Lambda cold starts. + +### Pre-bundling Workflow code + +Build the bundle as a separate build step: + +```typescript +import { bundleWorkflowCode } from '@temporalio/worker'; +import { writeFile } from 'fs/promises'; + +const { code } = await bundleWorkflowCode({ + workflowsPath: require.resolve('./workflows'), +}); +await writeFile('./workflow-bundle.js', code); +``` + + +Then reference the bundle in your handler with `workflowBundle: { codePath: require.resolve('./workflow-bundle.js') }`. + +### Versioning behavior + +Set per-Workflow with `setWorkflowOptions` in the Workflow file, or set a default for all Workflows with `defaultVersioningBehavior` in the configure callback. +Values are `'PINNED'` or `'AUTO_UPGRADE'`. The default versioning behavior is `PINNED`. + +Access via: `config.workerOptions.workerDeploymentOptions!.defaultVersioningBehavior = 'PINNED'` + +### Lambda-tuned defaults + + + +| Setting | Lambda default | +|---|---| +| `maxConcurrentActivityTaskExecutions` | 2 | +| `maxConcurrentWorkflowTaskExecutions` | 10 | +| `maxConcurrentLocalActivityExecutions` | 2 | +| `maxConcurrentNexusTaskExecutions` | 5 | +| `workflowTaskPollerBehavior` | `SimpleMaximum(2)` | +| `activityTaskPollerBehavior` | `SimpleMaximum(1)` | +| `nexusTaskPollerBehavior` | `SimpleMaximum(1)` | +| `shutdownGraceTime` | 5 seconds | +| `maxCachedWorkflows` | 30 | +| `shutdownDeadlineBufferMs` | 7000 | + +Eager Activities are not supported. Lambda invocations don't maintain persistent connections. + +`shutdownDeadlineBufferMs` is specific to the `@temporalio/lambda-worker` package. It controls how much time before the Lambda deadline the Worker begins its graceful shutdown. The default is `shutdownGraceTime` (5s) + 2s. + +If your Worker handles long-running Activities, increase `shutdownGraceTime`, `shutdownDeadlineBufferMs`, and the Lambda invocation deadline (`--timeout`) together. + +### Connection configuration + +The `@temporalio/lambda-worker` package automatically loads Temporal client configuration from a TOML config file and environment variables (see the Environment Configuration docs, `/develop/environment-configuration`). + +TOML config file resolution order: + +1. `TEMPORAL_CONFIG_FILE` environment variable, if set. +2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). +3. `temporal.toml` in the current working directory. + +The file is optional. If absent, only environment variables are used. + +--- + +## Cross-SDK comparison: Lambda-tuned defaults + +| Concept | Go | Python | TypeScript | +|---|---|---|---| +| Max concurrent activities | `MaxConcurrentActivityExecutionSize` = 2 | `max_concurrent_activities` = 2 | `maxConcurrentActivityTaskExecutions` = 2 | +| Max concurrent workflow tasks | `MaxConcurrentWorkflowTaskExecutionSize` = 10 | `max_concurrent_workflow_tasks` = 10 | `maxConcurrentWorkflowTaskExecutions` = 10 | +| Sticky cache size | 100 | `max_cached_workflows` = 30 | `maxCachedWorkflows` = 30 | +| Worker stop timeout | `WorkerStopTimeout` = 5s | `graceful_shutdown_timeout` = 5s | `shutdownGraceTime` = 5s | +| Shutdown deadline buffer | `ShutdownDeadlineBuffer` = 7s | `shutdown_deadline_buffer` = 7s | `shutdownDeadlineBufferMs` = 7000 | +| Eager activities | `DisableEagerActivities` always true | `disable_eager_activity_execution` always `True` | Not supported | + + + + + +Note: Go sticky cache size is 100, while Python and TypeScript are 30. These values come from each SDK's own docs and are not interchangeable.