From d9810c7d02e448aa9ba3077ee9dea3cb5301d7a3 Mon Sep 17 00:00:00 2001 From: Brian Cooper Date: Fri, 19 Jun 2026 00:03:06 -0700 Subject: [PATCH] docs(vortex): Herald replaces the prior email provider Email moved to Herald; update Vortex docs (config vars, email step/trigger, integration catalog) to reference Herald instead of the prior provider. --- content/docs/core/vortex/integrations.mdx | 2 +- content/docs/grid/vortex/configuration.mdx | 2 +- content/docs/grid/vortex/execution.mdx | 2 +- content/docs/grid/vortex/index.mdx | 2 +- content/docs/grid/vortex/integrations.mdx | 2 +- content/docs/grid/vortex/self-hosting.mdx | 63 +-- content/docs/grid/vortex/step-types.mdx | 417 +++++++++++++++++++ content/docs/grid/vortex/triggers.mdx | 4 +- content/docs/grid/vortex/troubleshooting.mdx | 225 ++++++++++ 9 files changed, 652 insertions(+), 67 deletions(-) create mode 100644 content/docs/grid/vortex/step-types.mdx create mode 100644 content/docs/grid/vortex/troubleshooting.mdx diff --git a/content/docs/core/vortex/integrations.mdx b/content/docs/core/vortex/integrations.mdx index b4d14a9..2b6f175 100644 --- a/content/docs/core/vortex/integrations.mdx +++ b/content/docs/core/vortex/integrations.mdx @@ -19,7 +19,7 @@ Vortex supports 500+ integrations via the ActivePieces connector ecosystem, plus | **HubSpot** | CRM | Contact management, marketing automation | | **Stripe** | Payments | Payment processing, subscription management | | **SendGrid** | Email | Transactional and marketing email delivery | -| **Resend** | Email | Transactional email delivery | +| **Herald** | Email | Transactional email delivery | | **Twilio** | SMS | Send and receive SMS messages | | **Anthropic** | AI | Claude models for text generation and analysis | | **OpenAI** | AI | GPT models, DALL-E, embeddings | diff --git a/content/docs/grid/vortex/configuration.mdx b/content/docs/grid/vortex/configuration.mdx index 1700153..2631d6f 100644 --- a/content/docs/grid/vortex/configuration.mdx +++ b/content/docs/grid/vortex/configuration.mdx @@ -163,7 +163,7 @@ The worker executes workflow DSL definitions, manages trigger adapters, and runs | `CHRONICLE_API_URL` | Chronicle audit logging URL | | `MANTLE_API_URL` | Mantle notification service URL | | `MANTLE_SERVICE_KEY` | Mantle service key | -| `RESEND_API_KEY` | Resend API key (for email trigger adapter) | +| `HERALD_API_KEY` | Herald API key (for email trigger adapter) | | `AUTH_API_URL` | Auth API URL (for email rendering) | | `EMAIL_RENDER_SECRET` | Email rendering secret | | `MEILISEARCH_URL` | Meilisearch URL | diff --git a/content/docs/grid/vortex/execution.mdx b/content/docs/grid/vortex/execution.mdx index 44cf466..3d6b1ed 100644 --- a/content/docs/grid/vortex/execution.mdx +++ b/content/docs/grid/vortex/execution.mdx @@ -112,7 +112,7 @@ Plugins provide infrastructure capabilities that don't require external connecto | **Rate Limit** | Token bucket, sliding window, fixed window algorithms | | **Queue** | Push/pull with memory, Valkey, SQS, or RabbitMQ backends | | **Database** | Query, insert, update, delete with parameterized SQL | -| **Email** | Send emails via Resend | +| **Email** | Send emails via Herald | | **File** | Read, write, list, delete files | | **HTTP** | Make HTTP requests with auth and retry | | **Hash** | MD5, SHA-256, bcrypt, HMAC | diff --git a/content/docs/grid/vortex/index.mdx b/content/docs/grid/vortex/index.mdx index 31064f4..e1daa96 100644 --- a/content/docs/grid/vortex/index.mdx +++ b/content/docs/grid/vortex/index.mdx @@ -209,7 +209,7 @@ Vortex ships with 600+ built-in integrations (Slack, GitHub, Stripe, Google Shee | **Rate Limit** | Token bucket, sliding window, fixed window algorithms | | **Queue** | Push/pull with memory, Valkey, SQS, or RabbitMQ backends | | **Database** | Query, insert, update, delete with parameterized SQL | -| **Email** | Send emails via Resend | +| **Email** | Send emails via Herald | | **File** | Read, write, list, delete files | | **HTTP** | Make HTTP requests with auth and retry | | **Hash** | MD5, SHA-256, bcrypt, HMAC | diff --git a/content/docs/grid/vortex/integrations.mdx b/content/docs/grid/vortex/integrations.mdx index ab09e01..f730a0c 100644 --- a/content/docs/grid/vortex/integrations.mdx +++ b/content/docs/grid/vortex/integrations.mdx @@ -13,7 +13,7 @@ Vortex ships with 600+ built-in integrations powered by the [ActivePieces](https | **Communication** | Slack, Discord, Microsoft Teams, Twilio | | **CRM** | HubSpot, Salesforce, Pipedrive | | **Developer** | GitHub, GitLab, Linear, Jira | -| **Email** | Gmail, Resend, SendGrid, Mailchimp | +| **Email** | Gmail, Herald, SendGrid, Mailchimp | | **Productivity** | Notion, Airtable, Google Sheets, Asana | | **Cloud** | AWS S3, Azure Blob Storage, Google Cloud Storage | | **eCommerce** | Shopify, Stripe, WooCommerce | diff --git a/content/docs/grid/vortex/self-hosting.mdx b/content/docs/grid/vortex/self-hosting.mdx index 873739e..01db2e1 100644 --- a/content/docs/grid/vortex/self-hosting.mdx +++ b/content/docs/grid/vortex/self-hosting.mdx @@ -1,9 +1,9 @@ --- title: Self-Hosting -description: Deploy Vortex on your own infrastructure with Docker Compose or Kubernetes +description: Deploy Vortex on your own infrastructure with Docker Compose --- -Vortex is fully open source (Apache 2.0) and designed to run on your own infrastructure. Two deployment options are supported: Docker Compose for simpler setups and Kubernetes (via Helm) for production-grade orchestration. +Vortex is fully open source (Apache 2.0) and designed to run on your own infrastructure. Docker Compose is the supported path for self-hosting. A Helm chart is not currently published; Kubernetes operators who want to wire their own manifests can use the compose file as the source of truth for service topology, env vars, and health checks. ## Docker Compose @@ -81,7 +81,7 @@ Omni platform integrations (optional, omit to run Vortex standalone): |----------|-------------| | `MANTLE_API_URL`, `MANTLE_SERVICE_KEY` | Mantle notification service | | `AUTH_API_URL` | HIDRA identity provider | -| `RESEND_API_KEY`, `EMAIL_RENDER_SECRET` | Email trigger adapter | +| `HERALD_API_KEY`, `EMAIL_RENDER_SECRET` | Email trigger adapter | ### Architecture @@ -100,63 +100,6 @@ All services include health checks and proper dependency ordering. The worker de When billing is not configured, the pricing page shows "All Features" mode, and all entitlements are unlocked. Self-hosted deployments get the full feature set with no artificial limits: unlimited workflows, unlimited executions, all integrations, custom plugins, and SSO. -## Kubernetes (Helm) - -For production deployments, Vortex ships a Helm chart with configurable replicas, resource limits, ingress, TLS, and Pod Disruption Budgets. - -### Prerequisites - -The Helm chart deploys the three Vortex services (API, app, worker) and optionally a PostgreSQL subchart. The following external dependencies must be provisioned separately: - -- **Valkey / Redis** for caching and distributed locks. Configure via `externalCache` values, or deploy your own Valkey instance alongside the chart. -- **[Apache Iggy](https://iggy.rs)** for event streaming. The worker and API both require a reachable Iggy server. -- **[Hatchet](https://docs.hatchet.run/self-hosting)** for workflow execution. See the Hatchet self-hosting docs for setup instructions. - -### Install - -```sh -helm install vortex ./charts/vortex \ - --namespace vortex --create-namespace \ - -f charts/vortex/values.yaml \ - --set secrets.dbPassword= \ - --set secrets.authSecret= \ - --set secrets.encryptionKey= \ - --set secrets.internalApiSecret= \ - --set secrets.hatchetClientToken= \ - --set secrets.iggyUsername= \ - --set secrets.iggyPassword= \ - --set secrets.cachePassword= -``` - -| Secret | Description | Generate with | -|--------|-------------|---------------| -| `secrets.dbPassword` | PostgreSQL password | `openssl rand -hex 16` | -| `secrets.authSecret` | Session signing secret | `openssl rand -hex 16` | -| `secrets.encryptionKey` | Data encryption key | `openssl rand -base64 32` | -| `secrets.internalApiSecret` | Service-to-service auth | `openssl rand -hex 16` | -| `secrets.hatchetClientToken` | Hatchet API token | From Hatchet dashboard | -| `secrets.iggyUsername` | Iggy streaming username | Any value | -| `secrets.iggyPassword` | Iggy streaming password | Any value | -| `secrets.cachePassword` | Valkey/Redis password | `openssl rand -hex 16` | - -See `charts/vortex/values.yaml` for the full list of configurable values, and `charts/vortex/values.prod.yaml` for a production reference. - -### External database - -To use an external PostgreSQL instance instead of the bundled subchart: - -```yaml -postgresql: - enabled: false - -externalDatabase: - host: your-db-host - port: 5432 - database: vortex - username: postgres - password: -``` - ## Upgrading Pull the latest changes and rebuild: diff --git a/content/docs/grid/vortex/step-types.mdx b/content/docs/grid/vortex/step-types.mdx new file mode 100644 index 0000000..885c2f2 --- /dev/null +++ b/content/docs/grid/vortex/step-types.mdx @@ -0,0 +1,417 @@ +--- +title: Step Types +description: Reference for every step type in the Vortex DSL +--- + +This page documents every step type the worker recognizes. Each entry includes purpose, key fields, and a minimal example. For shared fields (`id`, `name`, `position`, `onError`), see [Concepts](./concepts) and [DSL](./dsl). + +## Control flow + +### `trigger` + +The entry point of every workflow. Wraps a [trigger configuration](./triggers). + +```json +{ + "id": "trigger_1", + "type": "trigger", + "name": "Webhook", + "position": { "x": 0, "y": 0 }, + "trigger": { "type": "webhook", "config": {} } +} +``` + +### `condition` + +Two-way branch on a boolean expression. Edges from this step use `sourceHandle: "true"` or `"false"`. + +```json +{ + "type": "condition", + "condition": { + "expression": "$.total > 100", + "trueBranch": "discount", + "falseBranch": "skip" + } +} +``` + +### `switch` + +Multi-way branch on a value, with named cases and an optional default. + +```json +{ + "type": "switch", + "switch": { + "expression": "$.status", + "cases": [ + { "value": "paid", "label": "Paid", "next": "fulfill" }, + { "value": "refunded", "label": "Refunded", "next": "credit" } + ], + "default": "review" + } +} +``` + +### `loop` + +Iterate with `forEach`, `while`, or `times`. `body` is an array of step IDs that run per iteration. + +```json +{ + "type": "loop", + "loop": { + "type": "forEach", + "collection": "$.items", + "itemVariable": "item", + "indexVariable": "i", + "body": ["process_item"] + } +} +``` + +### `parallel` + +Run multiple branches concurrently. `waitFor` accepts `"all"`, `"any"`, or a number. + +```json +{ + "type": "parallel", + "parallel": { + "branches": [["send_email"], ["send_slack"], ["log_event"]], + "waitFor": "all" + } +} +``` + +### `race` + +Run branches in parallel, the first to complete wins. + +```json +{ + "type": "race", + "race": { + "branches": [["primary_api"], ["fallback_api"]], + "timeout": "10s", + "output": "result", + "winnerIndex": "winner" + } +} +``` + +### `try_catch` + +Run a `tryBranch`. On failure, optionally retry, then run `catchBranch` with the error available as `errorOutput`. + +```json +{ + "type": "try_catch", + "tryCatch": { + "tryBranch": ["charge_card"], + "catchBranch": ["notify_failure"], + "errorOutput": "chargeError", + "retries": 2, + "retryDelay": "1s" + } +} +``` + +### `delay`, `sleep`, `wait` + +`delay` pauses for `duration` and `unit` (`seconds`, `minutes`, `hours`, `days`). `sleep` pauses in `ms`, `seconds`, or `minutes`. `wait` blocks until a webhook arrives, a named event fires, or a timeout elapses. + +```json +{ + "type": "wait", + "wait": { + "resumeOn": "webhook", + "webhookSuffix": "approval", + "timeout": 86400, + "timeoutUnit": "seconds", + "timeoutAction": "continue" + } +} +``` + +### `subworkflow` + +Invoke another workflow by ID. Pass inputs in, map outputs back out. + +```json +{ + "type": "subworkflow", + "subworkflow": { + "workflowId": "wf_audit_log", + "inputs": { "event": "{{event}}" }, + "waitForCompletion": true + } +} +``` + +### `stop`, `noop`, `comment` + +`stop` terminates the run with `success`, `failure`, or `cancelled`. `noop` is a passthrough. `comment` is a visual-only annotation skipped by the executor. + +### `retry`, `timeout` + +`retry` re-executes a referenced `stepId` with backoff. `timeout` enforces a deadline on a referenced step and can `error`, `continue`, or run a `fallback`. + +## Data + +### `set` + +Assign workflow variables. + +```json +{ + "type": "set", + "set": { "variables": { "approved": true, "reviewer": "{{user.id}}" } } +} +``` + +### `map`, `reduce`, `filter`, `sort`, `unique`, `group`, `flatten`, `chunk`, `zip` + +Standard collection operations. Each takes a `source` JSONPath and an `outputVariable`. + +```json +{ + "type": "filter", + "filter": { + "source": "$.orders", + "expression": "$.total > 50", + "outputVariable": "largeOrders" + } +} +``` + +### `merge`, `split` + +`merge` combines objects or arrays from multiple sources with a `conflictStrategy` (`first`, `last`, or `error`). `split` divides an array into batches. + +### `template`, `parse`, `format`, `validate` + +`template` renders a string with variable interpolation. `parse` reads `json`, `xml`, `csv`, `yaml`, or `querystring` input. `format` formats `date`, `number`, `currency`, `percentage`, or `bytes`. `validate` checks input against a schema. + +```json +{ + "type": "parse", + "parse": { + "input": "{{triggerData.body}}", + "format": "json", + "outputVariable": "payload" + } +} +``` + +### `aggregate`, `diff`, `change_detector`, `debounce`, `time_window` + +Stateful data steps. `aggregate` rolls up arrays (`collect`, `merge`, `sum`, `first`, `last`). `diff` reports `added`, `removed`, and `changed`. `change_detector` skips the run if the value matches the previous run (`hash` or `deep_equal`). `debounce` coalesces rapid triggers using a cache key and `windowMs`. `time_window` proceeds only when the current time falls inside a window. + +### `cache`, `state_get`, `state_set`, `state_wait` + +`cache` is per-organization with TTL and supports `get`, `set`, `delete`, `getOrSet`. The `state_*` steps operate on a cross-workflow KV store, useful for coordinating runs. + +## Integration + +### `action` + +Run a connector operation. The connector executor in `vortex-worker` dispatches on `integrationId` and `operation`. + +```json +{ + "type": "action", + "action": { + "integrationId": "slack", + "operation": "sendMessage", + "inputs": { "channel": "#alerts", "text": "{{message}}" } + } +} +``` + +### `database` + +Execute SQL through an MCP database server. `operation` is `query`, `insert`, `update`, or `delete`. + +### `email` + +Send an email through Herald (or another configured provider). Supports `to`/`cc`/`bcc` (single or array), HTML or text body, and attachments. + +### `file`, `queue` + +`file` supports `local`, `s3`, `gcs`, and `azure` providers with `read`, `write`, `delete`, `list`, `exists`. `queue` operates on `memory`, `valkey`, `sqs`, or `rabbitmq` queues with `push`, `pull`, `peek`, `ack`, `nack`. + +### `spreadsheet`, `googleSheets`, `pdf` + +Office-style operations. `spreadsheet` handles `xlsx`, `xls`, `csv`. `googleSheets` uses a stored connection ID. `pdf` supports `create`, `parse`, `merge`, `split`, `watermark`, `extractPages`, `info`. + +### `webhookResponse`, `webhookVerify` + +`webhookResponse` returns a status code and body to a synchronous webhook caller. `webhookVerify` validates incoming signatures for `stripe`, `github`, `slack`, `twilio`, `shopify`, `sendgrid`, `paddle`, `linear`, or a `custom` HMAC scheme. + +### `notification` + +Send a notification through `email`, `slack`, `webhook`, or `push` channel with a `priority` (`low`, `normal`, `high`, `urgent`). + +## AI + +### `llm` + +Call an LLM via an MCP server with optional memory and tool use. Supports `images` for vision models. + +```json +{ + "type": "llm", + "llm": { + "serverId": "openai-mcp", + "model": "gpt-4", + "systemPrompt": "You are a helpful assistant", + "userPrompt": "Summarize: {{document}}", + "temperature": 0.7, + "outputs": { "summary": "summary" } + } +} +``` + +### `prompt`, `chat`, `summarize`, `classify` + +Higher-level wrappers. `prompt` runs a template through a model. `chat` keeps a message history. `summarize` produces `brief`, `detailed`, or `bullets` output. `classify` returns one or many labels. + +### `agent`, `rag`, `vision`, `audio` + +`agent` executes an autonomous loop with `maxIterations`. `rag` queries a vector collection then prompts the model. `vision` performs `describe`, `ocr`, `detect`, or `classify` on an image. `audio` does `transcribe` or `synthesize`. + +### `embedding`, `vectorSearch` + +`embedding` generates vectors through an MCP server. `vectorSearch` queries `pinecone`, `pgvector`, `qdrant`, `weaviate`, or `chroma` with a `topK` and optional `minScore`. + +### `modelRegistry` + +BYOK access to `openai`, `anthropic`, `huggingface`, `ollama`, `replicate`, `together`, `groq`, `mistral`, `cohere`. Use a stored `connectionId` or an inline `apiKey`. + +### `ai_transform`, `ai_guardrails`, `rivet` + +`ai_transform` sends a value through a prompt template for structured output. `ai_guardrails` validates output against `regex`, `contains`, `not_contains`, `max_length`, or `json_schema` rules. `rivet` executes a Rivet AI agent graph by `graphId` or inline JSON. + +## Security + +### `encrypt`, `decrypt`, `sign`, `hash`, `jwt` + +`encrypt` and `decrypt` use AES-GCM or AES-CBC. `sign` produces HMAC signatures (SHA-256/384/512). `hash` accepts `md5`, `sha1`, `sha256`, `sha512`, `xxhash`. `jwt` creates, verifies, or decodes tokens with HS256/HS384/HS512. + +## Human in the loop + +### `approval`, `input`, `gate` + +`approval` blocks until approvers respond, with a `timeoutAction` of `approve`, `reject`, or `error`. `input` collects form data from `assignees` (`text`, `number`, `email`, `textarea`, `select`, `checkbox` fields). `gate` is a lower-level pause that resumes on approval, named signal, or manual continue. + +```json +{ + "type": "approval", + "approval": { + "title": "Refund over $1000", + "message": "Approve refund for order {{orderId}}", + "approvers": ["user_admin_1"], + "timeout": 86400, + "timeoutAction": "reject" + } +} +``` + +## Plugins and code + +### `plugin` + +Execute a registered plugin (built-in or WASM) with `pluginId`, `function`, and `inputs`. Supports `timeout` and `memoryLimit` for sandboxing. + +### `code` + +Run JavaScript or TypeScript. `sandbox` picks the execution environment: `mcp` (default), `worker` (isolated Bun Worker), or `wasm` (QuickJS through Extism). MCP sandbox can install npm `dependencies` before execution. + +### `mcp` + +Call any tool exposed by a registered MCP server. + +## Events and orchestration + +### `event` + +Emit an event to the Iggy stream. `operation` is `emit`. + +```json +{ + "type": "event", + "event": { + "operation": "emit", + "eventName": "order.processed", + "payload": { "orderId": "{{orderId}}" } + } +} +``` + +### `collect` + +Pause the workflow until matching cross-service events arrive. Useful for sagas that fan out across products. See [Events](./events). + +```json +{ + "type": "collect", + "collect": { + "events": [ + { "name": "billing", "sourcePattern": "omni.aether", "typePattern": "subscription.created" }, + { "name": "identity", "sourcePattern": "omni.gatekeeper", "typePattern": "user.created" } + ], + "correlationKey": "data.userId", + "timeout": "5m", + "mode": "all" + } +} +``` + +### `window` + +Collect events into `tumbling`, `sliding`, or `session` windows. Emit `onClose` or `onEach`. + +### `saga` + +Distributed transaction with execute and compensate pairs per step. Set `parallel: true` to run steps concurrently. + +```json +{ + "type": "saga", + "saga": { + "steps": [ + { + "name": "reserveInventory", + "execute": { "type": "http", "url": "https://inv.example.com/reserve", "method": "POST" }, + "compensate": { "type": "http", "url": "https://inv.example.com/release", "method": "POST" } + } + ] + } +} +``` + +## Observability + +### `log`, `assert`, `error` + +`log` writes structured records at `debug`, `info`, `warn`, `error`. `assert` evaluates an expression and fails (or soft-fails) if false. `error` raises a typed error with optional `fatal` to terminate the workflow. + +### `rateLimit` + +Apply `token_bucket`, `sliding_window`, or `fixed_window` rate limits keyed by a string. Operations: `acquire`, `check`, `reset`, `throttle`, `configure`. + +## Next steps + + + + Workflow templates that combine these steps + + + Configure how workflows start + + + When to reach for which step + + diff --git a/content/docs/grid/vortex/triggers.mdx b/content/docs/grid/vortex/triggers.mdx index eb9990f..042977c 100644 --- a/content/docs/grid/vortex/triggers.mdx +++ b/content/docs/grid/vortex/triggers.mdx @@ -323,7 +323,7 @@ A higher-level trigger that subscribes to structured events from any Omni produc ## Email -Fires when an inbound email is received via Resend webhook. Supports glob-pattern filters on sender address and subject line. +Fires when an inbound email is received via Herald webhook. Supports glob-pattern filters on sender address and subject line. ```json { @@ -332,7 +332,7 @@ Fires when an inbound email is received via Resend webhook. Supports glob-patter "type": "email", "config": {}, "address": "workflows@omni.dev", - "provider": "resend", + "provider": "herald", "filters": { "from": "*@stripe.com", "subject": "Invoice*" diff --git a/content/docs/grid/vortex/troubleshooting.mdx b/content/docs/grid/vortex/troubleshooting.mdx new file mode 100644 index 0000000..da123ba --- /dev/null +++ b/content/docs/grid/vortex/troubleshooting.mdx @@ -0,0 +1,225 @@ +--- +title: Troubleshooting +description: Diagnose common errors in workflow execution, webhooks, and integrations +--- + +This page lists the failures we see most often and how to resolve each one. Most errors surface in the dashboard's run inspector and in the `workflow_run` and `workflow_run_step` tables. + +## Execution limit reached + +**Symptom:** New workflow runs return `403` with a message like `Plan limit reached: executions (2500/2500). Upgrade your plan to continue.` + +**Cause:** Each plan has a `max_executions_per_month` quota enforced by the API. Free tier is 2500. The check runs every time a workflow is about to dispatch. + +**Resolution:** + +- Check `/api/v1/billing/usage` to see current consumption and reset window. +- Disable workflows that are firing more often than needed. Cron expressions like `* * * * *` and unbounded webhook handlers are common culprits. +- Add a [`debounce`](./step-types) step to coalesce rapid triggers. +- Add a [`change_detector`](./step-types) step early in the workflow so noop runs do not consume the budget. +- Upgrade the plan if real usage outgrows the quota. + +Other plan limits to watch: `max_workflows`, `max_subscriptions`, `max_integrations`, `max_routing_rules`, `max_event_schemas`, `max_mcp_servers`, `max_functions`, `max_plugins`. Each returns a similar `403` from the matching route. + +## Webhook signature verification fails + +**Symptom:** Outbound subscription deliveries always return `401` from your receiver, even though the URL is correct. + +**Cause:** Almost always one of: + +1. Comparing against a `sha256=` prefix that Vortex does not emit. +2. Parsing the JSON body before verifying, so the bytes used to compute the signature differ from the bytes Vortex signed. +3. Using a string compare (`==`) instead of a constant-time compare. +4. Wrong header name. Vortex uses the configured `signatureHeader` (default `x-vortex-signature`), not `X-Hub-Signature` or `Stripe-Signature`. + +**Resolution:** + +- Use the **raw request body bytes**, not the parsed object. +- Expect a plain lowercase hex digest with no prefix. +- Use `crypto.timingSafeEqual`, `hmac.Equal`, or `hmac.compare_digest`. +- Verify the secret matches what was returned at subscription creation. Rotate through `PATCH /api/v1/subscriptions/:id` if unsure. +- Send a test delivery with `POST /api/v1/subscriptions/:id/test` and compare what arrives against what you compute. See [Webhooks](./webhooks#signature-verification). + +## Inbound webhook returns 404 + +**Symptom:** `POST /webhooks/workflow/:id/:secret` returns `404 Not Found`. + +**Cause:** Either the workflow does not exist, the secret was rotated, or the workflow is disabled. + +**Resolution:** + +- Confirm the workflow ID in the dashboard URL bar matches the path. +- Re-fetch the webhook URL from the workflow detail page. Rotation generates a new secret. +- Make sure the workflow's `enabled` flag is true. Disabled workflows reject inbound webhooks. + +## Hatchet worker offline + +**Symptom:** Runs sit in `pending` indefinitely and never progress, or fail to dispatch with `Failed to push event to Hatchet`. + +**Cause:** The worker process is not connected to the Hatchet server, or `HATCHET_CLIENT_TOKEN` is missing or invalid. + +**Resolution:** + +- Check the worker container logs for `Hatchet client connected` on startup. The worker logs a warning on boot if Hatchet credentials are absent. +- Verify `HATCHET_CLIENT_TOKEN` is set in the worker environment. +- Confirm network reachability from the worker to the Hatchet host. +- For self-hosted Hatchet, ensure its database is healthy and migrations have run. +- For local development, the worker auto-bootstraps Hatchet through Docker Compose. In production, the [feedback note about auto-setup](./self-hosting) still applies: do not run dev mode against a production Hatchet instance. + +## Schema validation error on workflow save + +**Symptom:** `POST /api/v1/workflows` returns `400` with a Zod validation error pointing at `steps[N]` or `trigger`. + +**Cause:** The DSL JSON does not match the discriminated union defined in `vortex-worker/src/dsl/types.ts`. Common mistakes: + +- `type: "trigger"` step missing the inner `trigger` config object. +- Switch step with no `cases` array. +- Loop step omitting `body`. +- Action step with neither `integrationId` nor `pluginId` set. + +**Resolution:** + +- Read the Zod error path carefully. It tells you exactly which step and field is wrong. +- Cross-check against [Step types](./step-types) which mirrors the schema. +- Open the workflow in the visual editor. The editor refuses to save invalid graphs, so any persistable graph is structurally valid. + +## OAuth integration broken + +**Symptom:** An `action` step fails with `Integration credentials expired` or `Token refresh failed`. + +**Cause:** The stored OAuth token has expired and the refresh attempt failed. This usually happens because the third-party provider revoked the token (password change, app permissions revoked, scope change) or the refresh token was never issued. + +**Resolution:** + +- Open the **Integrations** page in the dashboard. +- Find the affected integration and click **Reconnect**. +- Re-authorize through the provider, accepting any new scopes. +- Re-run the failed workflow. New runs use the refreshed credentials immediately. + +For a workflow that runs frequently, wrap the action in a [`try_catch`](./step-types) so a single bad refresh does not stop scheduled fan-outs. + +## Slow workflow runs + +**Symptom:** Runs take many seconds or minutes longer than the underlying step durations suggest. + +**Diagnosis:** + +- Open the run detail in the dashboard. Each step shows `startedAt` and `completedAt`. Identify gaps between consecutive steps (queue delay) versus long step durations (work itself is slow). +- Check the worker logs for `connector executor` timings. +- For LLM steps, large `maxTokens` and `messages` history dominate latency. + +**Common causes and fixes:** + +- **Sequential HTTP calls in a loop.** Convert to a `parallel` step or use `map` with a sub-action if the connector supports batch operations. +- **Cold MCP server.** First call to an MCP server pays connection cost. Keep it warm with a periodic health check workflow. +- **Cache misses.** Wrap expensive lookups in a `cache` step with `getOrSet` and a sensible TTL. +- **Subworkflow with `waitForCompletion: true`.** Set it to `false` for fire-and-forget when the parent does not need the child's result. +- **Hatchet queue saturation.** Scale worker replicas or increase concurrency on the Hatchet side. + +## Trigger never fires + +**Symptom:** A workflow is enabled but never receives data from its configured trigger. + +**By trigger type:** + +| Trigger | Check | +|---|---| +| Webhook | Curl the URL with a small body and look at the response status | +| Cron | Confirm the expression is in UTC and the workflow is enabled | +| Event | Check the routing rule matches the expected `typePattern` and `sourcePattern` | +| Polling | Worker logs show the polling adapter starting; confirm the URL is reachable | +| Kafka/SQS/AMQP/MQTT/NATS/Redis/SSE/WebSocket | Worker logs show the adapter connecting; confirm broker credentials | +| Email | Confirm the Herald webhook is configured to point at Vortex's email endpoint | +| Omni / Event | Confirm the publishing service is emitting; query the event log | + +Streaming triggers (MQTT, NATS, Kafka, AMQP, WebSocket, Redis, SSE, gRPC) require the worker process to be running. If the worker is restarted, adapters reconnect automatically on startup. + +## Routing rule does not match + +**Symptom:** Events flow through the Iggy stream but no workflow runs. + +**Resolution:** + +- Open the **Events** page and click an arrived event to inspect its `type` and `source`. +- Compare against your routing rule's `typePattern` (glob) and `sourcePattern`. +- If the rule has a `condition` (JSONPath), confirm it returns a non-empty value for the event in question. Conditions are presence checks, not boolean checks. +- If `celCondition` is set, it overrides `condition`. Verify the CEL expression evaluates to `true`. +- Higher-priority rules can shadow lower-priority ones if they consume an event first. Check the rule list ordering. + +See [Events](./events) for routing semantics. + +## State not persisting across runs + +**Symptom:** A `state_set` step succeeds, but a later run's `state_get` returns `null`. + +**Cause:** + +- TTL expired between runs. +- The key includes a per-run variable like `{{runId}}`, so each run writes under a different key. +- The keys differ in organizationId scope (state is per-organization). + +**Resolution:** + +- Use a stable key derived from a business identifier (`order:{{orderId}}`), not a run identifier. +- Set a long enough TTL or omit it for indefinite retention. + +## Duplicate runs from the same event + +**Symptom:** A single event triggers the workflow twice. + +**Cause:** The event has no `correlationId`, so the deduplication SETNX in Redis cannot recognize the duplicate. + +**Resolution:** + +- Publish events with a `correlationId` (a stable hash of the operation works well). +- Vortex deduplicates within a 24-hour window per organization. See [Events](./events#idempotency). + +## Subscription delivery in DLQ + +**Symptom:** Deliveries pile up with `status: "dlq"` in `GET /api/v1/subscriptions/:id/deliveries?status=dlq`. + +**Common causes:** + +- Target endpoint consistently returns non-2xx. +- Target endpoint is unreachable (DNS, TLS, firewall). +- Target endpoint is on a blocked private network (SSRF guard rejects it). +- `maxRetries` reached. + +**Resolution:** + +- Inspect the `error` and `httpStatus` columns on the DLQ rows. +- Fix the receiver, then replay the DLQ entries through the events replay API or by sending fresh test deliveries. +- Increase `maxRetries`, `initialBackoffMs`, or `backoffMultiplier` on the subscription if transient errors are common. + +## Connector or plugin not found + +**Symptom:** `IntegrationError: Connector "foo" not found` or `PluginError: Plugin "bar" not installed`. + +**Cause:** The integration or plugin is referenced in the DSL but is not registered on the worker. + +**Resolution:** + +- Confirm the `integrationId` or `pluginId` matches a record in `integration_definition` or the plugin registry. +- For OAuth integrations, the user must connect their account first. +- For custom plugins, the WASM or built-in plugin must be loaded on worker startup. + +## When all else fails + +- Tail the worker logs (`docker compose logs -f vortex-worker`). Every step logs structured records at `info` and `error`. +- Drop a [`log`](./step-types) step early in the workflow with `data: { ctx: "{{triggerData}}" }` to see exactly what the executor received. +- Reproduce the failure with a minimal workflow before changing the production one. +- File an issue at [github.com/omnidotdev/vortex](https://github.com/omnidotdev/vortex) with the workflow ID and a redacted run trace. + +## Next steps + + + + Design patterns that avoid these errors + + + Environment variables and service tuning + + + Deployment topology + +