feat(events): pub/sub event system

.changeset/event-dead-letter-queue.md

@@ -0,0 +1,6 @@
+---
+"@trigger.dev/core": minor
+"trigger.dev": minor
+---
+
+Add dead letter queue for failed event-triggered task runs

.changeset/event-docs-cli-dx.md

@@ -0,0 +1,10 @@
+---
+"@trigger.dev/sdk": minor
+"trigger.dev": minor
+---
+
+Add event system documentation, CLI commands, and developer experience improvements.
+New `rules/4.4.0/events.md` documentation covering all event features.
+CLI `trigger events list` and `trigger events publish` commands.
+JSDoc on `createEvent()` and `event()` with usage examples.
+Reference project `references/event-system/` demonstrating all event patterns.

.changeset/event-observability-dx.md

@@ -0,0 +1,10 @@
+---
+"@trigger.dev/core": minor
+"@trigger.dev/sdk": minor
+"@internal/clickhouse": minor
+"apps-webapp": minor
+---
+
+Add observability and developer experience improvements to the event system.
+New stats API endpoint aggregates event publish counts and fan-out metrics from
+ClickHouse. SDK gains a `validate()` method for pre-flight payload validation.

.changeset/event-ordering-consumer-groups.md

@@ -0,0 +1,7 @@
+---
+"@trigger.dev/core": minor
+"@trigger.dev/sdk": minor
+"trigger.dev": minor
+---
+
+Add ordering keys and consumer groups for event subscriptions

.changeset/event-persistence-replay.md

@@ -0,0 +1,6 @@
+---
+"@trigger.dev/core": minor
+"trigger.dev": minor
+---
+
+Add event persistence in ClickHouse and replay API for pub/sub events

.changeset/event-publish-and-wait.md

@@ -0,0 +1,11 @@
+---
+"@trigger.dev/core": minor
+"@trigger.dev/sdk": minor
+"apps-webapp": minor
+---
+
+Add publishAndWait support to the event system. Events can now be published
+with parentRunId to create waitpoints for each subscriber run, enabling
+fan-out / fan-in patterns. The SDK exposes `event.publishAndWait()` which
+publishes, blocks the parent run, and returns aggregated results from all
+subscriber completions.

.changeset/event-rate-limiting.md

@@ -0,0 +1,11 @@
+---
+"@trigger.dev/core": minor
+"@trigger.dev/sdk": minor
+"@trigger.dev/database": minor
+"apps-webapp": minor
+---
+
+Add per-event rate limiting to the pub/sub system. Events can now be configured
+with a `rateLimit: { limit, window }` option that limits how many times they can
+be published within a sliding time window. When exceeded, the API returns 429
+with `x-ratelimit-limit`, `x-ratelimit-remaining`, and `retry-after` headers.

.changeset/event-schema-registry.md

@@ -0,0 +1,7 @@
+---
+"@trigger.dev/core": minor
+"@trigger.dev/sdk": minor
+"trigger.dev": minor
+---
+
+Add event schema registry with versioning, validation, and discovery API endpoints

.changeset/event-smart-routing.md

@@ -0,0 +1,7 @@
+---
+"@trigger.dev/core": minor
+"@trigger.dev/sdk": minor
+"trigger.dev": minor
+---
+
+Add smart routing for events: content-based filters and wildcard pattern subscriptions

.claude/projects/-Users-terac-repos-trigger-dev/memory/MEMORY.md

@@ -0,0 +1,33 @@
+# Memory Index
+
+## Pub/Sub Event System
+- [Roadmap & Status](pubsub-roadmap.md) — phases 0-8 complete, pending items identified
+- [Detailed Progress](pubsub-progress.md) — per-phase notes, commits, decisions
+- [Pending Items](pubsub-pending.md) — Redis rate limiter, consumer groups, dashboard, etc.
+- [Audit Fix Plan](pubsub-audit-fixes.md) — Phases 10-13: CRITICAL/HIGH/MEDIUM/LOW fixes + test coverage
+- Repo conventions: [repo-conventions.md](repo-conventions.md)
+- Branch: `feat/pubsub-event-system`
+
+## Repo Quick Reference
+
+- Build: `pnpm run build --filter <pkg>`, Test: `pnpm run test --filter <pkg>`
+- Build order: core → sdk → cli → run-engine → webapp
+- Services extend `WithRunEngine`, use `traceWithEnv()`, throw `ServiceValidationError`
+- API routes use `createActionApiRoute()` builder
+- Tests use testcontainers (never mocks), vitest
+- Import `@trigger.dev/core` subpaths only, never root
+- Migrations: clean extraneous lines, indexes need CONCURRENTLY in separate files
+- Changesets required for `packages/*` changes (default: patch)
+- Tags in integration tests: avoid `tags` option in trigger calls — `createTag` uses global prisma mock `{}`
+
+## Rate Limiting Patterns in Codebase
+- `apps/webapp/app/services/rateLimiter.server.ts` — Upstash `@upstash/ratelimit` wrapper (sliding window, token bucket, fixed window)
+- `apps/webapp/app/v3/GCRARateLimiter.server.ts` — Custom GCRA with Redis Lua scripts
+- Both use dedicated Redis connection (`RATE_LIMIT_REDIS_HOST` env vars)
+- Good reference implementations: `mfaRateLimiter.server.ts`, `magicLinkRateLimiter.server.ts`
+
+## User Preferences
+
+- Documentation and roadmap files must be written in English
+- Commit frequently (per sub-step)
+- Never commit broken code

.claude/projects/-Users-terac-repos-trigger-dev/memory/pubsub-audit-fixes.md

@@ -0,0 +1,229 @@
+# Pub/Sub Event System — Audit Fix Plan
+
+Findings from 5-agent parallel audit (2026-03-03). Organized by priority.
+Follow [Implementation Process & Guidelines](pubsub-roadmap.md#implementation-process--guidelines) for each phase.
+
+---
+
+## Phase 10: Audit Fixes — CRITICAL + HIGH
+
+### 10.1 — Fix `expireTtlRuns` Lua: global concurrency slot leak (CRITICAL)
+
+**Found by**: Redis auditor
+**Severity**: CRITICAL — permanent slot leak
+**File**: `internal-packages/run-engine/src/run-queue/index.ts:2633-2726`
+
+**Problem**: The `expireTtlRuns` Lua script removes from `queueCurrentConcurrency`, `queueCurrentDequeued`, `envCurrentConcurrency`, `envCurrentDequeued` but does NOT remove from `globalCurrentConcurrency`. When a run with an ordering key expires via TTL, the global concurrency slot is permanently leaked, eventually starving the entire queue.
+
+**Fix**:
+1. Add `globalCurrentConcurrencyKey` as a new KEYS parameter to the `expireTtlRuns` Lua
+2. Add `redis.call('SREM', globalCurrentConcurrencyKey, messageId)` alongside existing SREMs
+3. Update `numberOfKeys`, type declaration, and caller to pass the key
+4. Update the caller that invokes `expireTtlRuns` to compute and pass `queueGlobalCurrentConcurrencyKeyFromQueue`
+
+**Verify**: Run existing run-engine TTL tests
+
+### 10.2 — Fix `clearMessageFromConcurrencySets` bare queue name (HIGH)
+
+**Found by**: Redis auditor
+**Severity**: HIGH — SREM to wrong key, slot never released
+**File**: `internal-packages/run-engine/src/engine/index.ts:2240-2243`
+
+**Problem**: `clearMessageFromConcurrencySets` is called with `taskRun.queue` which is a bare queue name (e.g. `"my-task"`), not a full Redis key. `queueGlobalCurrentConcurrencyKeyFromQueue()` expects a full key like `{org:X}:proj:Y:env:Z:queue:my-task` and produces a nonsense key from a bare name.
+
+**Fix**: Trace how other callers of similar methods get the full queue key (likely from the `message.queue` field which includes the full path). Ensure `clearMessageFromConcurrencySets` either:
+- Receives the full queue key, or
+- Has access to the env/org/project context to construct it
+
+**Verify**: Check that the same issue exists for the existing per-key `queueCurrentConcurrencyKeyFromQueue` call (it probably does but SREM on a wrong key is a no-op, not a crash).
+
+### 10.3 — Add `.max()` to batch publish items array (HIGH)
+
+**Found by**: Security auditor
+**Severity**: HIGH — potential DoS
+**File**: `packages/core/src/v3/schemas/api.ts` — `BatchPublishEventRequestBody`
+
+**Fix**: Add `.max(100)` (or similar) to the `items` array in `BatchPublishEventRequestBody`. Matches the pattern of existing batch trigger which has limits.
+
+### 10.4 — Fix publishAndWait schema: parentRunId required but options optional (HIGH)
+
+**Found by**: API auditor
+**Severity**: HIGH — schema mismatch causes runtime 400 instead of Zod validation error
+**File**: `packages/core/src/v3/schemas/api.ts:1658-1671`
+
+**Fix**: Either:
+- Make `options` required in `PublishAndWaitEventRequestBody`, or
+- Move `parentRunId` to be a top-level required field outside of `options`
+
+### 10.5 — Fix ClickHouse interval string interpolation (HIGH)
+
+**Found by**: Security auditor
+**Severity**: HIGH — fragile pattern
+**File**: `apps/webapp/app/routes/api.v1.events.$eventId.stats.ts:54`
+
+**Fix**: Use parameterized query or keep the whitelist validation but use a safer pattern (map from allowed period to interval string rather than interpolating user input).
+
+### 10.6 — Add missing index for pattern subscription query (HIGH)
+
+**Found by**: DB auditor
+**Severity**: HIGH — full table scan on every publish
+**File**: `internal-packages/database/prisma/schema.prisma` — `EventSubscription`
+
+**Fix**:
+1. Add `@@index([projectId, environmentId, enabled])` to EventSubscription model
+2. Create migration with `CREATE INDEX CONCURRENTLY` in its own file
+3. Run `pnpm run db:migrate:deploy && pnpm run generate`
+
+### 10.7 — Fix batch publish partial failure semantics (HIGH)
+
+**Found by**: API auditor
+**Severity**: HIGH — client can't determine which items succeeded
+**File**: `apps/webapp/app/routes/api.v1.events.$eventId.batchPublish.ts:40-57`
+
+**Fix**: Two options:
+- **Option A**: Validate ALL items upfront before triggering any (current approach fails mid-batch)
+- **Option B**: Return partial results with per-item status (more complex but more resilient)
+
+Recommend Option A — validate schema + rate limits for all items first, then trigger.
+
+---
+
+## Phase 11: Audit Fixes — MEDIUM
+
+### 11.1 — Fix N+1 in DLQ retryAll
+
+**File**: `apps/webapp/app/v3/services/events/deadLetterManagement.server.ts:126-148`
+**Fix**: Remove redundant re-fetch in `retry()` when called from `retryAll()`, or batch the operations.
+
+### 11.2 — Add payload size check before fan-out
+
+**File**: `apps/webapp/app/v3/services/events/publishEvent.server.ts`
+**Fix**: Check payload byte size before triggering subscribers. Return 413 if over limit and object store is not configured.
+
+### 11.3 — Fix inconsistent error handling in routes
+
+**Files**: `api.v1.events.dlq.retry-all.ts`, `api.v1.events.ts`
+**Fix**: Add try/catch with ServiceValidationError handling, matching other routes.
+
+### 11.4 — Add CLI publish options support
+
+**File**: `packages/cli-v3/src/commands/events/publish.ts`
+**Fix**: Add `--delay`, `--tags`, `--idempotency-key`, `--ordering-key` options.
+
+### 11.5 — Fix schema validation silent pass on compilation error
+
+**File**: `apps/webapp/app/v3/services/events/schemaRegistry.server.ts:198-201`
+**Fix**: Log a warning when ajv compilation fails, and optionally reject the publish.
+
+### 11.6 — Add stale subscription cleanup
+
+**File**: `apps/webapp/app/v3/services/events/publishEvent.server.ts`
+**Fix**: When a subscriber trigger fails consistently, log a warning and optionally disable the subscription after N consecutive failures.
+
+### 11.7 — Add data cleanup mechanism
+
+**Fix**: Add a periodic cleanup job (or TTL-based approach) for:
+- Disabled EventSubscriptions older than 30 days
+- Processed DeadLetterEvents (RETRIED/DISCARDED) older than 30 days
+- Deprecated EventDefinitions with no active subscriptions
+
+---
+
+## Phase 12: Test Coverage
+
+### 12.1 — Tests for ReplayEventsService
+
+**File**: `apps/webapp/test/engine/replayEvents.test.ts` (new)
+**Tests**:
+- Replay with date range filter
+- Replay with task filter
+- Replay dry run (count only)
+- Replay with idempotency (no duplicate triggers)
+- Replay when ClickHouse is unavailable (graceful error)
+
+Note: These require ClickHouse in testcontainers or mocking.
+
+### 12.2 — Tests for DeadLetterService
+
+**File**: `apps/webapp/test/engine/deadLetterService.test.ts` (new)
+**Tests**:
+- Failed event-triggered run creates DLQ entry
+- Non-event run does NOT create DLQ entry
+- DLQ entry has correct eventType, payload, error
+- Multiple failures create separate DLQ entries
+
+### 12.3 — Tests for DeadLetterManagementService
+
+**File**: `apps/webapp/test/engine/deadLetterManagement.test.ts` (new)
+**Tests**:
+- List DLQ entries with pagination
+- List with eventType filter
+- List with status filter
+- Retry creates new run with correct payload
+- Retry marks DLQ entry as RETRIED
+- Discard marks entry as DISCARDED
+- RetryAll processes up to 1000 items
+- Retry/discard nonexistent ID returns error
+
+### 12.4 — Tests for RedisEventRateLimitChecker
+
+**File**: `apps/webapp/test/engine/eventRateLimiter.test.ts` (extend)
+**Tests**:
+- Redis checker allows under limit
+- Redis checker blocks over limit
+- Redis checker returns correct remaining/retryAfter
+- Different configs get separate Ratelimit instances
+
+Note: Requires Redis in testcontainers.
+
+### 12.5 — Tests for SchemaRegistryService.checkCompatibility
+
+**File**: extend existing SchemaRegistryService tests
+**Tests**:
+- Compatible schema change (add optional field)
+- Incompatible change (remove required field)
+- Incompatible change (change field type)
+
+---
+
+## Phase 13: LOW Priority Fixes
+
+### 13.1 — Add LRU bounds to caches
+- `validatorCache` in SchemaRegistryService: max 1000 entries
+- `patternCache`/`filterCache` in core evaluators: max 1000 entries
+- `InMemoryEventRateLimitChecker.windows`: evict entries older than 2x window
+
+### 13.2 — Tighten Zod schemas
+- `payload: z.any()` → `payload: z.unknown()`
+- `metadata: z.any()` → `metadata: z.record(z.unknown())`
+- Add `.max(256)` to idempotencyKey
+- Add DLQ status validation with Zod instead of `as` cast
+
+### 13.3 — Remove dead code
+- Unused `compileFilter`/`evaluateFilter` exports from core filterEvaluator
+
+### 13.4 — Fix batchPublish URL naming
+- Current: `/api/v1/events/:id/batchPublish` (camelCase)
+- Consider: `/api/v1/events/:id/batch-publish` or keep for consistency
+
+---
+
+## Execution Order
+
+```
+Phase 10 (CRITICAL+HIGH) → Phase 12 (Tests) → Phase 11 (MEDIUM) → Phase 13 (LOW)
+```
+
+Phase 10 first because it contains a CRITICAL bug (permanent slot leak).
+Phase 12 second because tests validate the fixes and catch regressions.
+Phase 11 and 13 are improvements, not blockers.
+
+## Verification per phase
+
+Same as roadmap guidelines:
+1. `pnpm run build --filter @internal/run-engine --filter webapp --filter @trigger.dev/core --filter @trigger.dev/sdk`
+2. `cd internal-packages/run-engine && pnpm run test --run` (run-engine: 236+ must pass)
+3. `cd apps/webapp && pnpm run test ./test/engine/publishEvent.test.ts --run` (24+ must pass)
+4. `cd apps/webapp && pnpm run test ./test/engine/eventRateLimiter.test.ts --run` (11+ must pass)
+5. New test files must pass
+6. Commit after each sub-step: `feat(events): phase X.Y — <description>`