Email 3/7 (PR 2): typed failures, delivery and a safe DLQ drain #421

Part of #418. Read the parent first. Its Design sections Failures and retries and Dead-letter drain are the contract here. Step 3 of 7. server/ is short for apps/hocuspocus.server/.

Goal

Classify every failure so that only real outages retry, operator problems page, and bad recipients stop at once. Give the delivery path one owner, and give operators a safe drain for the dead-letter queue.

Owner and gate

• Builds it: an agent or engineer.
• Staging checks: the maintainer runs them, because they need the Resend staging key.
• Gate to merge:
  • Email 1/7 (PR 1): explicit provider config, the provider contract and job identity #419 (PR 1) has run on prod for 48 h with no email alert.
  • Email 2/7 (Ops 0): Resend teams, domains, records and staging credentials #420 (Ops 0) is closed.

Steps

1. Split the sender (server/src/lib/email/sender.ts:78-196).
   • buildEmailMessage(data): EmailMessage renders the mail from the job data and its created_at only. It strips control characters from the subject and cuts it to 150 characters.
   • deliverEmail(message, { idempotencyKey, jobId }): Promise<SentEmail> calls getEmailProvider() with AbortSignal.timeout(EMAIL_SEND_TIMEOUT_MS). It logs one line per send (jobId, provider, messageId, durationMs, to) and rethrows EmailSendError. It never turns an error into a string.
   • The worker, the no-Redis path and server/src/lib/email/document-notification.ts:80 call these two. sendEmailViaProvider is deleted.
2. Classification. In providers/resend.ts and providers/smtp.ts, map errors with the parent's Resend error map table and SMTP error map rules. Keep code verbatim and set responseCode.
3. server/src/lib/email/failureDecision.ts (create). Pure, with no import of queue.ts.
   • decideEmailFailure(err, attemptsMade, attempts): EmailFailureAction, where EmailFailureAction = 'retry' | 'dead-letter'.
   • An error that is not an EmailSendError counts as transient.
4. server/src/lib/email/dlqDisposition.ts (create). Pure.
   • judgeEmailDlqEntry(facts: EmailDlqFacts, now: number): EmailDlqDisposition, where EmailDlqDisposition = 'replay' | 'discard' | 'unresolved', per the parent's drain table. Follow server/src/lib/storeDlqDisposition.ts.
5. Worker (server/src/lib/email/queue.ts:174-199).
   • Catch any error and call decideEmailFailure. On 'dead-letter' before the last attempt, throw BullMQ UnrecoverableError.
   • The DLQ entry gets failureKind and failureCode beside failureReason (server/src/types/email.types.ts:67-71).
   • Retry policy (queue.ts:28-31): attempts: 6, backoff: { type: 'exponential', delay: 30_000 }.
6. Timeouts. EMAIL_SEND_TIMEOUT_MS = 15_000 beside deliverEmail. SMTP adds socketTimeout: 30_000 and does not race the signal.
7. Resend request. Tags are { job_id, email_type, ns }, where ns is config.email.delivery.keyNamespace. They are sent as [{ name, value }], each value cleaned to [A-Za-z0-9_-], at most 256 characters. Pass reply_to through.
8. Masking. Create server/src/lib/maskEmail.ts with maskEmail(address). Its output matches SQL internal.mask_email (packages/supabase/scripts/07-5-email-notifications-pgmq.sql:164-175): j***@example.com. Every email log line puts the masked address in the field to. No raw address may appear under any key.
9. No-Redis path. queue.ts:244-249 and server/src/lib/email/service.ts:59, :73, :87: an inline failure is final. The service must not send a second time.
10. DLQ route (server/src/api/controllers/admin-audit.controller.ts:268-285), for both push and email:
    • Build the queue with EMAIL_DLQ_NAME from jobIdentity.ts, not a string literal (:255).
    • Return a summary (id, name, timestamp, failureKind, failureCode, type, masked to), not data.
    • Fill the response key failedReason from j.data?.failureReason ?? j.failedReason. Keep the key name, so the admin page needs no change.
11. Drain.
    • Add drainEmailDeadLetterQueue({ apply }) to queue.ts. Apply judgeEmailDlqEntry to each entry. Replay goes through queueEmail with the original job id and data. Do not use job.retry(), which keeps attemptsMade.
    • Create server/scripts/drain-email-dlq.ts, a thin script like server/scripts/drain-store-dlq.ts. It is a dry run by default, acts only with --apply, and prints one line per entry: id, failureKind, disposition.
12. Alerts in scripts/observability/grafana/provisioning/alerting/rules-app-logs.yml: add incident-email-operator and incident-email-transient-surge, per the parent's alert table. Keep incident-email-broken. In rules-workers.yml:137, change the worker-dlq-depth summary to name the email drain.

Tests (two files)

• server/src/lib/email/__tests__/failureDecision.test.ts: each kind on the last and a non-last attempt; a non-EmailSendError; a 422 validation_error naming subject maps to permanent.
• server/src/lib/email/__tests__/dlqDisposition.test.ts: each disposition, including a transient entry at 23 h and at 25 h, and an entry with no failureKind.

The unit tests are the only proof of the permanent path. Staging cannot produce it, because send-generic refuses a malformed address before it is queued (server/src/schemas/email.schema.ts:4).

Docs (same PR)

• server/CLAUDE.md:
  • the provider contract and the three kinds;
  • failureDecision.ts and dlqDisposition.ts as lifted pure decisions;
  • the same-body rule and jobIdentity.ts;
  • "never write .env from a web route".
• docs/RUNBOOK-backend.md: extend §Dead-letter queue not empty with the email drain. Add a §Email section with these entries:
  • Key revoked: edit the host env file, redeploy, run the drain.
  • Quota reached: sends fail as operator, or Resend fails them later with email.failed. Raise the plan or wait for the reset, then run the drain.
  • Config invalid: the Email setup page and the worker log name the variable. Fix it and redeploy. Messages older than 24 h settle stale.
  • "I get no mail": check users.notification_preferences, then email_bounces, then the Resend log by the job_id tag.
  • "Email sign-in does nothing": check Resend Suppressions for the address in the auth team.
  • Remove a suppression: remove it in Resend (Dashboard > Suppressions) first. Then reset email_bounces and notification_preferences.email_enabled. In the other order, the next send triggers email.suppressed and turns email off again.
• server/CHANGELOG.md under ## [Unreleased]: ### Changed for the retry window, ### Fixed for the DLQ Reason and the no-Redis double send.

Checks before push

• bun run check at the repo root. It includes check:agent-docs, which this PR's CLAUDE.md edit must pass.
• bun test in server/.
• Import failureDecision.ts, dlqDisposition.ts and maskEmail.ts with REDIS_HOST, REDIS_PORT and DATABASE_URL unset.

Deploy

The deploy subject is exactly (build): back observability. The app deploys before the alert rules.

Staging checks (maintainer)

Set EMAIL_PROVIDER=resend and an EMAIL_FROM on the verified domain in the staging host env file, then redeploy staging.

• Send with EMAIL_FROM on a domain that is not verified. Resend answers 403 validation_error. The job dead-letters with failureKind: operator and no retries, and incident-email-operator fires. Restore EMAIL_FROM.
• Kill the worker right after a send (docker kill during the job). The retry gets the same Resend message id, and the Resend log shows one mail.
• Set a wrong RESEND_API_KEY and send. It dead-letters as operator. Restore the key. The drain dry run lists the entry as replay. --apply sends it once.
• The Notifications DLQ table shows a Reason.

Acceptance

• The staging checks above pass.
• bun test is green, including both new files.
• Every doc listed above is updated in this PR.

Exit evidence

Post as a comment here:

• bun run check and bun test output.
• The DLQ entry JSON with failureKind, with every address masked.
• The Resend log line for the killed-worker retry.
• The drain dry-run output.

Revert

git revert the merge, then redeploy. Jobs already queued keep 6 attempts, which is harmless. The DLQ keeps its entries.