Skip to content

Email 6/7 (Ops A): production product mail cutover to Resend #424

Open

Description

@HMarzban

Part of #418. Read the parent first. Step 6 of 7. No code.

Goal

Switch production product mail from SMTP to Resend, with a tested rollback and a clear definition of done.

Owner and gate

Steps

  1. Privacy page (apps/webapp/src/pages/privacy.tsx), through the tech-writer skill:

    • Name Resend as the email provider for sign-in and notification mail.
    • Add "Where your data goes": Resend stores data in the US. The safeguard is the UK Extension to the EU-US Data Privacy Framework, plus SCCs with the UK Addendum. Confirm the wording with counsel.
    • Say that notification and digest mail include short extracts of document and chat text. Say that the provider keeps sent mail and logs for 30 days.
    • Bump LEGAL_UPDATED in apps/webapp/src/components/pages/legal/legalMetadata.ts:1, and deploy.
  2. Record the current values in a comment here:

    • the current EMAIL_PROVIDER and EMAIL_FROM lines from /opt/projects/prod.docs.plus/.env;
    • the current email DLQ depth, from the Email setup page;
    • the name of the current SMTP provider, for the Day 30 offboarding.

    Rollback restores both lines. Restoring EMAIL_PROVIDER alone would leave the old relay sending as the Resend domain, and that fails.

  3. In the Resend product team, create a prod send-only key for mail.docs.plus.

  4. Edit /opt/projects/prod.docs.plus/.env. Keep every SMTP_* line.

    EMAIL_PROVIDER=resend
    EMAIL_FROM="docs.plus <notify@mail.docs.plus>"   # the D2 product domain
    RESEND_API_KEY=<prod send-only key>
    
  5. Redeploy (see Terms on Tracking: move email to Resend behind a pluggable provider #418).

  6. In Resend, add the prod webhook for https://<prod rest-api host>/api/email/webhooks/resend, with events email.bounced, email.complained, email.suppressed and email.failed.

    • Add RESEND_WEBHOOK_SECRET=<its whsec_> to the prod host env file and redeploy again.
    • The webhook comes after the first deploy on purpose. Adding it earlier only collects 404 retries.
  7. Smoke test.

    • Open the Email setup page. Status is ready, and the connection check shows send_only_key.
    • Send a test email and confirm it arrives.
    • In Gmail, open a real digest or notification and confirm that the one-click Unsubscribe link appears.

Roll back at once if any of these happens in the first hour

  • incident-email-operator or incident-email-config-invalid fires;
  • the email DLQ depth rises above the step 1 value;
  • the smoke test fails.

Rollback

  1. Restore both lines recorded in step 1.
  2. Keep RESEND_API_KEY, RESEND_WEBHOOK_SECRET and the Resend webhook. Mail that Resend already accepted keeps producing bounces and complaints for days.
  3. Redeploy.
  4. Send a test email from the Email setup page and confirm it arrives.
  5. Only then run the drain inside a hocuspocus-worker container: bun scripts/drain-email-dlq.ts first (dry run), then bun scripts/drain-email-dlq.ts --apply.
  6. Post what happened here.

Exit

Close this issue when all of these hold for 7 days:

  • No email alert fired.
  • worker-dlq-depth stayed silent.
  • The Resend dashboard met the D4 rule: no hard bounce and no complaint, or each one investigated and explained here.
  • Each day, the maintainer requested one magic link and it arrived. Sign-in mail is unchanged until Email 7/7 (Ops B): Supabase Auth magic links through Resend SMTP #425 (Ops B), so this checks that nothing else broke.

Day 30 (maintainer)

Post on #418 one of these:

  • "Removed SMTP_* from the prod host env file. Revoked the old relay credential. Offboarded the old provider: account closed or stored mail deleted, and its record-of-processing row removed."
  • "Extended because ."

Exit evidence

Post as a comment here:

  • The step 1 values.
  • A link to the deployed privacy page change.
  • A screenshot of the Email setup page after step 6, checked for unmasked addresses.
  • The 7-day list above, ticked, with the 7 dates of the magic-link checks and the Resend bounce and complaint counts.

Activity

  1. self-assigned this
    on Oct 6, 2026
  2. added theissue type on Oct 6, 2026
  3. changed the title [-]Email 6/7 (Ops A): switch production product mail to Resend[/-] [+]Email 6/7 (Ops A): production product mail cutover to Resend[/+] on Oct 6, 2026
  4. HMarzban commented on Oct 9, 2026

    @HMarzban
    CollaboratorAuthor

    Status, 2026-10-09: cutover live

    • Deployed in run 37906965365 (deploy job finished 09:28:41Z). GET /api/email/health reports "provider":"resend".
    • Prod host env: EMAIL_PROVIDER=resend and EMAIL_FROM="docs.plus <notify@mail.docs.plus>". The SMTP_* lines are kept, and dated .env.bak-* backups hold the old values for rollback.
    • Smoke test: the new-document notice was Delivered in the Resend dashboard.

    Still open: step 0, the privacy page, is being built today. Step 5, the webhook, follows #422. The 7-day exit window started at 09:28:41Z today.

  5. HMarzban commented on Oct 12, 2026

    @HMarzban
    CollaboratorAuthor

    Resend webhook live, 2026-10-12

    • The maintainer created the Resend webhook for https://prodback.docs.plus/api/email/webhooks/resend, with the events email.bounced, email.complained, email.failed and email.suppressed.
    • RESEND_WEBHOOK_SECRET is set in the host env file. Deploy run 38239392350 (51a9be84f) finished green at 11:14:12Z.
    • An unsigned POST to the webhook now answers 401, not 404. The route is mounted, and it refuses calls without a valid signature.

    From today, bounces and complaints land in public.email_bounces. Before today that table could not record anything, so the #424 exit check for 2026-10-09 to 2026-10-12 must read the Resend log instead.

    Earliest close: 2026-10-16, if the Resend log and email_bounces show no hard bounce and no complaint.

  6. HMarzban commented on Oct 15, 2026

    @HMarzban
    CollaboratorAuthor

    Related issues


    Cross-link review, 2026-10-15. Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions