Skip to content

fix(sequences): stop retry floods when sending is blocked and warn owners early - #190

Open
s3rot0n1n wants to merge 3 commits into
mainfrom
fix/sequence-sending-paused-loop
Open

s3rot0n1n wants to merge 3 commits into
mainfrom
fix/sequence-sending-paused-loop

Conversation

@s3rot0n1n

Copy link
Copy Markdown
Contributor

Problem

In production, one team's sequence logged processOngoingSequence failed with sending_paused about 25 times every 40 seconds. Its organization had reached 100 sends on SendLit Cloud without a bounce/complaint feedback connection for its SES ESP, so assertSendingEligibility stopped sending (feedback_connection_required). That check is correct. The bug was that it throws in resolvePinnedDeliverySource, before the code that reschedules a failed send. Every scheduler tick retried the same 17 deliveries, logged an error and reported to PostHog again, and nobody told the owners why the sequence had stopped.

Changes

No more retry floods

  • Sending paused for billing or abuse reasons: queued sends wait and check again every 15–20 minutes, logging only at info level. They resume by themselves once the stop is lifted.
  • Any other failure before the send (for example, the ESP was retired) waits 5 minutes and counts toward the existing 5-in-a-row auto-pause. It doesn't use up the recipient's retries, so a short database outage can't mark recipients as failed.
  • Failed sends now back off between retries (5 minutes or more, doubling, at most an hour) instead of retrying on the next tick.

Owners are told

  • Stops only the team can fix (feedback_connection_required, verified_owner_email_required) auto-pause the sequence with the new sending_blocked reason and email the owners once. The dashboard banner on the paused sequence explains it.
  • Owners are emailed at 50%, 90% and 100% of the cloud test volume while one of the organization's ESPs has no healthy feedback connection. Each email goes out once per organization, tracked in organizations.feedback_notice_percent (migration 0032, additive). Nothing is sent on self-hosted.
  • New endpoint GET /settings/feedback-requirement and MCP tool get_feedback_requirement, with its parity entry and scope. They power a notice in Settings → ESP and in the new-ESP dialog that shows how many of the 100 emails the organization has sent. openapi.json is regenerated.

Testing

  • New tests: sends wait while sending is paused; the sequence pauses and the owners are emailed once when sending is blocked; a persistent failure before the send stops after 5 attempts; the 50/90/100% emails go out once each, including under concurrent sends, and not at all when feedback is connected or on self-hosted.
  • Updated tests: a retired ESP and a transient render error now back off and stay queued, instead of being retried on the next tick.
  • Lint passes, and the API and web app typecheck.
  • Full API suite: 1040 pass, 15 fail. None of the 15 are in changed code. 5 were confirmed to fail on main too. The other 10 are in tracking-link, tracking-domain and web-page tests and look like local setup problems (tracking host, HTTPS), but haven't been rerun on main.

Follow-ups (not in this PR)

  • The feedback check looks for a connection tied to the sending team. Organization-level ESPs whose feedback connection belongs to the organization may get blocked after 100 sends even with feedback connected.
  • The 100-send rule doesn't check whether the provider supports feedback (for example, generic SMTP).
  • Error-level logs show up in PostHog as info.

🤖 Generated with Claude Code

…ners early

When an organization's sending was stopped, the sequence worker hit the
stop before the code that reschedules a failed send, so every scheduler
tick retried the same deliveries and logged and reported the error again.
Production saw ~25 "sending_paused" errors every 40 seconds for one team.

- Sending paused for billing or abuse reasons: queued sends wait and
  check again every 15-20 minutes, resuming once the stop lifts.
- Stops only the team can fix (no feedback connection, unverified
  owner): the sequence is auto-paused and the owners are emailed once.
- Any other failure before the send backs off 5 minutes and counts
  toward the 5-in-a-row auto-pause, without using up a recipient's
  retries. Failed sends back off between retries.
- Owners are emailed at 50%, 90% and 100% of the cloud test volume
  while an ESP has no healthy feedback connection (migration 0032).
- GET /settings/feedback-requirement (and MCP get_feedback_requirement)
  powers a notice in Settings -> ESP and the new-ESP dialog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-10T18:37:29.295576Z ebd3761 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ebd37617f7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +120 to +122
const [claimed] = await db
.update(organizations)
.set({ feedbackNoticePercent: reached })

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep milestone claims retryable when owner email fails

If notifyOrganizationOwners fails due to a transient SMTP error, the catch below records it but leaves feedbackNoticePercent advanced. Later sends then skip this milestone, so the 50%, 90%, or 100% warning is permanently lost; retry the claim or otherwise preserve a retry path when notification delivery fails.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in abae676: if emailing the owners fails, the milestone claim is released back to its previous value, so the next accepted send tries that warning again (covered by a new test). At 100% no further sends are accepted, so a failed 100% warning is not retried, but the owners still get the email when the sequence auto-pauses with sending_blocked.

switch (code) {
case "feedback_connection_required":
return {
why: "its email provider isn't reporting bounces and spam complaints back to SendLit, which is required after your first 100 emails",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use the configured volume in the blocked-send notice

When BILLING_TEST_VOLUME_THRESHOLD is set to a value other than 100, the feedback gate uses that configured threshold but this owner notification—and the matching dashboard banner—still says the requirement starts after 100 emails. Owners can therefore be told the wrong limit when the configured threshold is used.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in abae676: the sending_blocked pause now records the configured testVolumeThreshold(), and both the owner email and the dashboard banner use it instead of a hard-coded 100. The test sets the threshold to 250 and checks that the email says so.

Rajat and others added 2 commits November 11, 2026 00:22
- Release a 50/90/100% milestone claim when emailing the owners fails,
  so the next accepted send tries that warning again.
- Record the configured test volume on a sending_blocked pause and use
  it in the owner email and dashboard banner instead of a hard-coded 100.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_feedback_requirement imported team/queries and the requirement
module, which load the real database client, so the mocked MCP test
crashed in CI without DB_CONNECTION_STRING. Look the organization up
inside getFeedbackRequirementForTeam, mock that one module in the test,
and cover the tool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant