Tracking: move email to Resend behind a pluggable provider #418

Goal

Move production email to Resend, and keep the email provider a small pluggable module. Self-hosters must still be able to run SMTP or add their own provider.

At the end:

• Product mail (digests, notifications, generic sends) goes out through Resend, with safe retries and no double sends.
• Sign-in mail (magic links) goes out through Resend's SMTP relay, set in Supabase Auth.
• Resend reports hard bounces and complaints back to us through a signed webhook.
• An admin Email setup page shows whether email is set up correctly and can send a test email.
• .env stays the only source of the provider choice and of every secret.

How to execute this

1. Read this issue in full. The Design section is the contract for every sub-issue.
2. Work the sub-issues in the order listed under Sequence. Each one names its owner, entry gate, exit evidence and revert. Tick it in Sequence when it closes.
3. A sub-issue is done only when every acceptance box is checked and its exit evidence is posted as a comment on that sub-issue.
4. Get the maintainer's answer to each Decision before the step it blocks.

Rulings are posted as comments headed Maintainer ruling, <date>. A later change uses Maintainer ruling update, <date>. The newest ruling wins over this body.

Volume numbers were measured on production on 2026-10-06 (see Measured volume). Other numbers were read from code or from vendor docs. Resend facts marked (unverified) come from search excerpts, because the review sandbox could not reach resend.com. Check each one in the Resend dashboard before relying on it. Paths are from the repository root. server/ is short for apps/hocuspocus.server/.

Decisions

The default applies until the maintainer rules. D1 and D2 cannot change later, so #420 (Ops 0) waits for them.

D1. Which Resend sending region?
Recommended: eu-west-1. It sends from EU infrastructure close to Supabase production (eu-west-2) and most users. It does not give EU data residency: Resend stores account data, logs and message content in the US in every region (unverified). The transfer basis is the UK Extension to the EU-US Data Privacy Framework plus SCCs (see the privacy step in #424). A domain's region cannot change after it is created. Blocks: #420 (Ops 0).

D2. Which Resend teams and domains?
Recommended: three teams, one domain each.

Why: suppression, quota and account standing are per team (Resend suppressions). With one team, a digest complaint would block that person's magic links. A sign-in flood could pause product mail. Staging would also use prod reputation and quota. One domain per team also fits the Free plan's domain limit, whatever it is today. Check that Resend's terms allow several Free teams under one account (unverified). Fallback: one team holding mail. and auth., plus a staging team. Blocks: #420 (Ops 0).

D3. Supabase Auth email rate limit after the switch?
Recommended: 100 per hour, plus CAPTCHA before any launch push. Saving custom SMTP settings resets the limit to 30 per hour (Supabase SMTP guide). Blocks: #425 (Ops B).

D4. Cutover thresholds?
Recommended: no hard bounce and no complaint from the opted-in list over 7 days. Investigate any single event; do not auto-fail on it. At about 10 product sends a week (see Measured volume), one bounce is a 10 % rate, so rate limits carry no meaning. Switch to rates (bounce under 2 %, complaint under 0.1 %) above 1,000 sends a week. Blocks: the #424 (Ops A) exit.

D5. Record the rule in root CLAUDE.md §Settled?
Recommended: yes. Text: "Email secrets come from .env only. No UI write, no secrets in Postgres." It stops a later agent from proposing a credential form again. Blocks: the closure checks in Done means.

D6. Which Resend plan?
Recommended: Free for all three D2 teams now. Product mail is about 30 a month, at most 7 a day. Sign-in mail is at most 4 to 9 a day. Both fit Free's 3,000 a month and 100 a day (unverified) many times over. Upgrade the product team to Pro when product mail passes 50 a day, or email opt-ins pass 100 users. Pro is about $20 a month with no daily cap (unverified). Blocks: #420 (Ops 0).

Already decided. A ruling can overturn these:

• SendGrid is deleted. Its users move to the SendGrid SMTP relay with EMAIL_PROVIDER=smtp.
• A suppression check on other send paths comes later.
• The webhook and the setup page are two small modules.
• Mail that waits on an invalid config has no special expiry. The general 24 h age limit covers it.
• The admin page is called Email setup, not "connector". In CONTEXT.md, "connector" already means the MCP connector at /api/mcp.

Terms

Why: current state

Keep:

• one send path through server/src/lib/email/sender.ts;
• stable pgmq job ids and the sent log (EmailSentLog, server/prisma/schema.prisma:110-117);
• the DLQ condition at server/src/lib/email/queue.ts:180;
• templates with no provider knowledge;
• record_email_bounce, which already turns off email on hard and complaint (packages/supabase/scripts/07-5-email-notifications-pgmq.sql:226-245);
• email secrets from env_file only (docker-compose.prod.yml:428-431).

Measured volume

Read-only aggregate queries on Supabase prod, 2026-10-06. Counts only; no personal data was read.

Limits of the data: cleanup_email_queue deletes sent and skipped rows after 30 days (07-5…sql:811-825), so there is no longer history. EmailSentLog lives in the server database, not in Supabase. Staging was not measured.

Design

Layers

Each layer imports only from layers above it. Each concern has one owner.

Shared helpers: server/src/lib/maskEmail.ts (same output as SQL internal.mask_email, 07-5…sql:164-175), server/src/lib/email/bounces.ts (recordEmailBounce, used by /bounce and the webhook).

Every new pure file is proved safe by importing it with REDIS_HOST, REDIS_PORT and DATABASE_URL unset (server/CLAUDE.md §Testability Seams).

Config

resolveEmailConfig(env) in server/src/config/email.ts is pure and never throws. It imports the Env type only, because env.schema.ts validates and may exit at import. EMAIL_PROVIDER stays z.string(), so an old value cannot crash the boot. SMTP_SECURE becomes optional 'true' | 'false' with no default.

// server/src/config/email.ts
export type EmailConfig =
  | { status: 'ready'; from: string; provider: EmailProviderConfig; keyNamespace: string }
  | { status: 'off' }
  | { status: 'invalid'; problems: string[] }

/** Presence only. No secret value enters this object. */
export interface EmailEnvFacts {
  from: string | null
  smtpHost: string | null
  smtpPort: number | null
  set: Record<'RESEND_API_KEY' | 'RESEND_WEBHOOK_SECRET' | 'SMTP_PASS' | 'EMAIL_UNSUBSCRIBE_SECRET' | 'PUBLIC_RESTAPI_URL', boolean>
}

• EMAIL_FROM is required and accepts Name <addr>. Both fallbacks, SMTP_FROM_NAME, and the dead config.email.smtp, fromEmail and fromName are deleted.
• SMTP needs only SMTP_HOST. auth exists only when SMTP_USER is set. An explicit SMTP_SECURE wins; otherwise secure is port === 465.
• keyNamespace is the host of PUBLIC_RESTAPI_URL, or of the app URL when that is unset. Staging and prod then never share idempotency keys. Every product mail also carries it as the ns tag, so the webhook can ignore events from another environment.

Status behavior (EmailGatewayService)

• ready: normal. One info line email config ready with provider.
• off: the pgmq consumer and the BullMQ worker both settle rows skipped with email not configured. One info line at boot.
• invalid: neither the pgmq consumer nor the BullMQ email worker starts, so mail waits. Push has its own consumer and is not affected. The worker /health stays healthy, because server/src/hocuspocus.worker.ts:178 would otherwise fail every deploy. The body adds workers.email.configStatus: 'ready' | 'off' | 'invalid'. The worker logs email config invalid with the problem list at error level every 5 min, and that line pages.

Self-hoster .env

# Resend
EMAIL_PROVIDER=resend
EMAIL_FROM="Acme Docs <notify@mail.acme.com>"
RESEND_API_KEY=re_...
RESEND_WEBHOOK_SECRET=whsec_...   # optional: bounces and complaints

# or SMTP
EMAIL_PROVIDER=smtp
EMAIL_FROM="Acme Docs <notify@acme.com>"
SMTP_HOST=smtp.acme.com
SMTP_PORT=465
SMTP_USER=...                     # omit both for Mailpit or an open relay
SMTP_PASS=...

Both also need EMAIL_UNSUBSCRIBE_SECRET and PUBLIC_RESTAPI_URL. Without that URL, the one-click List-Unsubscribe header is dropped (sender.ts:74).

Provider contract

server/src/lib/email/providers/types.ts is the only home of these types. It ships in #419 (PR 1). The old EmailProvider string union becomes EmailProviderName, and the old EmailProviderInterface becomes EmailProvider. Update the re-export at server/src/lib/email/index.ts:25.

export type EmailProviderConfig =
  | { name: 'resend'; apiKey: string }
  | { name: 'smtp'; host: string; port: number; secure: boolean; auth?: { user: string; pass: string } }

export type EmailProviderName = EmailProviderConfig['name']

/** What to send. No attachments, cc or bcc: no sender needs them yet. */
export interface EmailMessage {
  from: string
  to: string[]
  subject: string
  html: string
  text: string
  replyTo?: string
  headers?: Record<string, string>
  tags?: Record<string, string> // { job_id, email_type, ns }
}

/** How this one call runs. */
export interface SendOptions {
  signal: AbortSignal
  idempotencyKey: string
}

export interface SentEmail {
  /** Opaque. Resend: the API email id. SMTP: the Message-ID header nodemailer wrote. */
  messageId: string
}

export type ConnectionCheck =
  | { ok: true; note?: 'send_only_key' }
  | { ok: false; kind: EmailErrorKind; code: string; message: string }

export interface EmailProvider {
  readonly name: EmailProviderName
  /**
   * Throws EmailSendError on any failure. An adapter that cannot cancel
   * ignores `signal` and relies on its own transport timeouts.
   */
  send(message: EmailMessage, options: SendOptions): Promise<SentEmail>
  checkConnection(options: { signal: AbortSignal }): Promise<ConnectionCheck>
  close(): Promise<void>
}

/** What makes the send succeed: time, nothing, or an operator. */
export type EmailErrorKind = 'transient' | 'permanent' | 'operator'

export class EmailSendError extends Error {
  readonly responseCode?: number
  constructor(
    readonly kind: EmailErrorKind,
    readonly code: string, // the provider's own code, verbatim: 'EAUTH', 'rate_limit_exceeded'
    message: string,
    options?: ErrorOptions & { responseCode?: number }
  ) {
    super(message, options)
    this.name = 'EmailSendError'
    this.responseCode = options?.responseCode
  }
}

kind, code and responseCode are own properties, so pino.stdSerializers.err (server/src/lib/logger.ts:36) emits them as err_kind, err_code and err_responseCode. The existing incident-email-broken query (rules-app-logs.yml:153) keeps working unchanged.

Adding a provider (for example Postmark):

1. Add providers/postmark.ts exporting createPostmarkProvider(cfg): EmailProvider. Map its errors to the three kinds.
2. Add its branch to EmailProviderConfig, and its checks to resolveEmailConfig.
3. Add one case to createEmailProvider.
4. Add its rows to server/ENV.md and .env.example.

There is no registry and no loader.

Job identity and the same body on every retry

• queueEmail always sets a job id: the caller's deterministic pgmq id, else randomUUID(). BullMQ counter ids are never used.
• jobIdentity.ts derives both keys from that id: sentLogKey(jobId) is email:<jobId>, and providerIdempotencyKey(namespace, jobId) is <namespace>/email/<jobId>. A send with no job (no-Redis mode, new-document mail) uses <namespace>/inline/<uuid>. The test email uses <namespace>/test/<uuid>.
• buildEmailMessage signs the unsubscribe token with nowSeconds from the job's created_at (server/src/types/email.types.ts:64), and passes the same time to the template year. The digest "ago" line is already stable: it reads period_end, set at enqueue (pgmqConsumer.ts:244).
• buildEmailMessage strips control characters from the subject and cuts it to 150 characters. A user-set name forms part of some subjects (packages/email-templates/src/engine.ts:119-124).
• Resend keeps a key for 24 h and answers a repeat with the first result (Resend idempotency keys).

Failures and retries

deliverEmail rethrows EmailSendError. The worker passes any error to decideEmailFailure(err, attemptsMade, attempts): EmailFailureAction, where EmailFailureAction = 'retry' | 'dead-letter'. An error that is not an EmailSendError, such as a template or Prisma error, counts as transient. On 'dead-letter' before the last attempt, the worker throws BullMQ UnrecoverableError (exported by bullmq 6.3.4).

Retry policy: 6 attempts, 30 s exponential backoff: 30 + 60 + 120 + 240 + 480 s, about 15 min.

Rate: the BullMQ limiter stays at EMAIL_RATE_LIMIT_MAX=50 per EMAIL_RATE_LIMIT_DURATION=60000 (server/src/config/env.schema.ts:133-135), about 0.8 a second. That is far below Resend's 10 requests a second per team (unverified). A retry reuses its key, so it adds requests, not billed mail.

Deadlines: deliverEmail owns EMAIL_SEND_TIMEOUT_MS = 15_000, below lockDuration: 60000 (queue.ts:211). checkConnection callers use EMAIL_CHECK_TIMEOUT_MS = 10_000. Adapters never start their own timer. SMTP takes no AbortSignal, so it ignores the signal and relies on connectionTimeout and greetingTimeout (10 s each, already set) plus a new socketTimeout: 30_000. Never race an SMTP send against the signal: SMTP has no idempotency key, so a race plus a retry sends twice.

Resend error map, keyed by the error name in the body. An unknown name falls back to the status code. Source: RESEND_ERROR_CODE_KEY in the resend-node SDK, and the Resend errors page.

SMTP error map:

• A 4xx response code is transient, even with EAUTH. nodemailer stamps EAUTH on a transient 454 too (rules-app-logs.yml:148).
• An enhanced status 5.1.x is permanent.
• EAUTH with a 5xx or no code, and any other 5xx, is operator.

Age limit. In processMessage (pgmqConsumer.ts:365), a message whose enqueued_at is more than 24 h old (EMAIL_MAX_AGE_MS) settles skipped with stale. The SQL checks read-state only at enqueue (07-5…sql:455). An invalid config or a worker outage can stop the consumer for days. Without this limit, it would then mail people about things they already read.

Adapters

• Resend uses plain fetch, not the SDK. The SDK adds no retry or timeout logic we need, and SMTP-only installs then carry no extra package. It sends POST /emails with the Idempotency-Key header. Tags go as [{ name, value }], each value cleaned to [A-Za-z0-9_-], at most 256 characters.
• Resend checkConnection calls GET /domains. A 401 restricted_api_key answer means a valid send-only key: { ok: true, note: 'send_only_key' }.
• SMTP checkConnection calls nodemailer transporter.verify(). SMTP close closes the pool. Resend close does nothing.
• Open and click tracking stay off. Click tracking would rewrite the signed unsubscribe link.

Smaller fixes

• Mask every address in logs with maskEmail, in the field to. No raw address may appear under any key.
• The DLQ route returns a summary, not the job data, for both queues. It keeps the response key failedReason and fills it from data.failureReason ?? failedReason, so the admin page needs no change. New DLQ fields: failureKind, failureCode.
• In no-Redis mode an inline failure is final, so the service never sends a second time.
• replyTo passes through.

Dead-letter drain

drainEmailDeadLetterQueue({ apply }) lives in server/src/lib/email/queue.ts, beside the DLQ it owns. server/scripts/drain-email-dlq.ts only parses --apply, calls it and prints one line per entry, like server/scripts/drain-store-dlq.ts. It is a dry run by default.

judgeEmailDlqEntry(facts, now): EmailDlqDisposition, with EmailDlqDisposition = 'replay' | 'discard' | 'unresolved', follows judgeStoreDlqEntry (server/src/lib/storeDlqDisposition.ts:9, :45).

Replay goes through queueEmail with the original job id and data, so it gets the full retry ladder. The sent log still blocks a double send. Do not use job.retry(): it keeps attemptsMade unless told otherwise.

Resend webhook

Module server/src/modules/email-webhooks/, mounted in server/src/index.ts at /api/email/webhooks only when config.email.webhookSecret is set. The route is POST /api/email/webhooks/resend.

A second provider adds domain/<name>/ and one route. No Resend file changes.

Input rules: svix-id must match ^msg_[A-Za-z0-9]{1,64}$. Reject a timestamp that is not a number or is more than 5 min off in either direction. Every rejection gets the same 401 body. resolveWebhookSecret(raw) in server/src/config/email.ts treats a RESEND_WEBHOOK_SECRET that does not decode to at least 24 bytes as unset. The route is then not mounted, and the boot logs email webhook secret invalid at error level. Product mail is never held for it.

Environment check: a product-mail event whose ns tag is not this server's keyNamespace maps to ignored.

Dedupe, key email:webhook:<svix-id>. withRedisLease is deliberately not used: it runs work unleased on a Redis error and has no done state.

• Absent: set pending (NX, 60 s), record the event, then set done (24 h) and answer 200.
• pending: answer 409, so Svix retries later.
• done: answer 200 with no write.
• Record failure: delete the key and answer 5xx.
• No Redis: no dedupe. A duplicate adds only a second system_alert row.

Bad signature: 401. Ignored event: 200. Record failure: 5xx.

Supabase Auth email (magic link)

Magic links do not pass through our code. apps/webapp/src/components/auth/SignInForm.tsx:116 calls supabaseClient.auth.signInWithOtp, and Supabase Auth sends the mail with its own SMTP settings. Production already uses some custom SMTP, because the built-in sender refuses addresses outside the project team.

Use the custom SMTP relay, not the Send Email Hook. The hook would make sign-in depend on rest-api being up, and email links are the only sign-in besides Google. Settings: host smtp.resend.com, port 465, user resend, password a separate send-only key from the D2 auth team. Use the auth domain, with tracking off. Link tracking rewrites the confirmation link and breaks it.

Email setup page

Module server/src/modules/email-setup/; page apps/admin-dashboard/src/pages/email.tsx; nav label Email setup.

Routes, mounted in server/src/api/routers/admin.router.ts with admin.route('/email/setup', emailSetup.init(deps).router). This is a deliberate exception to "src/index.ts mounts modules": the admin namespace owns one auth boundary (admin.use('*', adminAuthMiddleware), admin.router.ts:31), and it already hosts /email/digest-grouping. Mounting elsewhere would run is_admin twice.

• GET /api/admin/email/setup
• POST /api/admin/email/setup/test-send

toEmailSetupView(delivery, facts, extras): EmailSetupView is a pure projection and the only code that builds the page data. Its inputs are config.email.delivery and config.email.facts, so no secret string ever enters the module.

It shows:

• Provider and config status. For invalid, each problem in plain words.
• EMAIL_FROM in full.
• set or missing for each secret in EmailEnvFacts.set.
• SMTP_HOST and SMTP_PORT.
• A live checkConnection on each load.
• Whether the webhook secret is set, and the newest row of get_email_bounces (admin-audit.controller.ts:195-199). The row shows a masked address and no username. Its reason is cut to 80 characters.
• The email DLQ depth, and a link to the failure summary.
• A copy-ready .env block with a blank line for each missing value, and "Edit the host env file, then redeploy".

One action, Send test email:

• It goes only to the signed-in admin's own address and takes no body.
• It calls deliverEmail in rest-api, so it skips only the queue.
• It shows the message id, or the error kind and code.
• Limit: 1 per minute per admin, through RateLimiterRedis (keyPrefix: 'email-test-send') and consumeWithin (server/src/lib/consumeWithin.ts), as server/src/modules/mcp/infra/toolBudget.ts does. It is refused when Redis is absent or unavailable.

The page reads only from rest-api, so the admin-dashboard container needs no email env (compatible with #403).

Why the UI never writes credentials:

• Each deploy copies the host env file over .env.production (prod.docs.plus.yml:537), and containers do not mount the host file (server/docker/Dockerfile.bun:99), so a write is lost.
• A write would not reach the second worker replica.
• It would turn a stolen admin session into a lasting credential change.
• It adds a second writer, which is the drift class behind the 535 outage (docker-compose.prod.yml:428-431).
• Secrets in Postgres would end ".env is the only source", and would need an encryption key and cache invalidation.

Observability

Logs: one line per send from deliverEmail: jobId, provider, messageId, durationMs, to (masked). A failure logs { err }, so Loki reads err_kind, err_code and err_responseCode. No new log field names, no new Prometheus metrics.

Alerts in rules-app-logs.yml:

Sequence

Work these in order. Each sub-issue has the full steps.

• Email 1/7 (PR 1): explicit provider config, the provider contract and job identity #419 Email 1/7 (PR 1): explicit provider config, the provider contract and job identity
• Email 2/7 (Ops 0): Resend teams, domains, records and staging credentials #420 Email 2/7 (Ops 0): Resend teams, domains, records and staging credentials
• Email 3/7 (PR 2): typed failures, delivery and a safe DLQ drain #421 Email 3/7 (PR 2): typed failures, delivery and a safe DLQ drain
• Email 4/7 (PR 3): Resend webhook for bounces and complaints #422 Email 4/7 (PR 3): Resend webhook for bounces and complaints
• Email 5/7 (PR 4): admin Email setup page with a test email #423 Email 5/7 (PR 4): admin Email setup page with a test email
• Email 6/7 (Ops A): production product mail cutover to Resend #424 Email 6/7 (Ops A): production product mail cutover to Resend
• Email 7/7 (Ops B): Supabase Auth magic links through Resend SMTP #425 Email 7/7 (Ops B): Supabase Auth magic links through Resend SMTP

Accountability

Rules for every PR

• Each PR changes production when it deploys. The provider stays SMTP until Email 6/7 (Ops A): production product mail cutover to Resend #424 (Ops A).
• Docs, alerts and env templates ship in the same PR as the code they describe.
• Five new test files, no more (AGENTS.md §Test Policy): server/src/config/__tests__/email.test.ts, server/src/lib/email/__tests__/failureDecision.test.ts, server/src/lib/email/__tests__/dlqDisposition.test.ts, server/src/modules/email-webhooks/__tests__/unit/verifySignature.test.ts, server/src/modules/email-setup/__tests__/unit/toEmailSetupView.test.ts. Each mirrors its source file name. Exit evidence includes bun test output for each, observed green.
• CHANGELOG entries go under ## [Unreleased] in server/CHANGELOG.md, using ### Breaking, ### Migration, ### Changed, ### Fixed, ### Removed (RELEASE_POLICY.md:203, :224).
• Prose goes through the tech-writer skill and its Simplified English standard. Use American spelling.
• Each sub-issue names its exact deploy subject (see Terms).
• After each deploy, the maintainer watches the email DLQ depth and the err_kind logs for 1 h. Revert means a code revert plus the rolling deploy.
• Exit evidence is public. Mask every email address with maskEmail's format (j***@example.com), and check each screenshot before posting it. Never paste a key or a secret.

Done means

• Every sub-issue is closed with its exit evidence posted.
• Email 6/7 (Ops A): production product mail cutover to Resend #424 (Ops A) exit signals held for 7 days, and the day-30 SMTP decision is posted here as a comment.
• Closure checks pass:
  • git grep -n -i -E "sendgrid|SENDGRID_API_KEY|SMTP_FROM_NAME" finds only CHANGELOG history.
  • git grep -n "noreply@docs.plus" finds nothing.
  • git grep -n -E "email_kind|email_code|connector-test" finds nothing.
  • GET /openapi.json lists POST /api/email/webhooks/resend, GET /api/admin/email/setup and POST /api/admin/email/setup/test-send.
  • A fresh reader follows docs/self-hosting/install.md with Resend only and gets a test email from the Email setup page.
  • If D5 is yes: the root CLAUDE.md §Settled bullet is added.
  • A follow-up issue is filed to delete POST /api/email/send-generic, or to limit it to admin addresses. It has no caller in the repo; staging checks are its only use.
  • The old SMTP provider is offboarded: account closed or stored mail deleted, and its record-of-processing row removed (Email 6/7 (Ops A): production product mail cutover to Resend #424 Day 30).

Not touched: scripts/observability/.env.example:12 is GlitchTip's own SMTP. Files under docs/roadmap/ are history.

Out of scope

Related

• Stop REST /health from leaking error text or taking every /api route down #405 public REST /health. Email 5/7 (PR 4): admin Email setup page with a test email #423 (PR 4) applies the same rule to /api/email/health: public routes return status only.
• Keep server secrets out of the webapp and admin-dashboard containers #403 removes env_file from the admin-dashboard container. This plan needs no email env there.

Source

Local plan, team reports and the decision log: Notes/local-docs/email-resend/ (gitignored, maintainer machine). This issue is complete without them. The architecture review and the security, compliance and cost review of 2026-10-06 are summarized in comments on this issue. Security details stay private, per SECURITY.md.