Harden the Datamailer send boundary so misconfiguration fails closed

[alexeygrigorev]

Harden the Datamailer send boundary so misconfiguration fails closed

PM-groomed specification

Reviewed 2026-09-02 against origin/main at 2d3c28d1b8d1c2f3844f47f664945b42849d5eb9.
This issue is a P0 security/operations hardening slice for the existing course-platform
Datamailer client. It does not authorize a provider, a send, a migration, or production
configuration change.

The issue is groomed for two independently testable slices:

• H1 is a test-only deployment-contract correction and can be implemented independently.
• H2 is a client-boundary correction and can be implemented independently once the owner
  decisions below are recorded.
• H3 is narrowed/deferred. The raw four-directory rule cannot be accepted on current main:
  accounts/views/account_settings.py and accounts/views/email_preferences.py still import the
  legacy Datamailer preference adapter. The website-native preference migration belongs to
  #227, and the complete source-surface
  inventory belongs to #290. This issue must
  not claim either migration.

Normative authority and observed gap

• _docs/PROCESS.md defines the groom → engineer → independent tester →
  PM acceptance → focused commit → local merge/push → on-call lifecycle.
• Spec 01 requires network side effects after commit
  through durable jobs, makes email_app the logical delivery boundary, and prohibits direct
  Datamailer/SES calls from website request/service/job code.
• Spec 06 makes email management Relay-proxied and
  gives the website no provider adapter, renderer, provider store, or direct-send fallback.
• Spec 07 requires provider credentials to stay
  outside the website, safe redacted failures, and rollback that never re-enables Datamailer.
• Spec 09 requires every outbound path to be
  disabled during rehearsal/import and treats Datamailer as read-only migration/history input.

At this base, deploy/task_definitions.py pins five disabled safety values for web, worker, and
migration tasks, including empty DATAMAILER_URL and DATAMAILER_API_KEY. The release test's
metadata-normalization scenario asserts only DATAMAILER_TRANSACTIONAL_DRY_RUN == "1", so the
load-bearing URL/API-key pins could be removed without that scenario failing. Separately,
course_management/datamailer/client.py constructs a configured requests.Session request for
any non-empty URL/API key/client/audience; the transactional dry-run flag does not cover all
certificate, recipient-list, reminder, campaign, contact/list, and preference paths.

In scope

H1 — assert the complete deployed safety contract

Extend the deployment-release test so the web, worker, and migration task definitions each contain
the exact reviewed SAFETY_ENVIRONMENT mapping after release metadata normalization:

DATAMAILER_SYNC_ON_USER_CREATE=0
DATAMAILER_OUTBOX_DISPATCH_IMMEDIATELY=0
DATAMAILER_TRANSACTIONAL_DRY_RUN=1
DATAMAILER_URL=
DATAMAILER_API_KEY=

The test must reject omission or mutation of each safety key, specifically URL and API key. Keep
the existing builder validation unchanged unless a test-only coverage gap requires a narrowly
justified adjustment. H1 adds no secret, environment override, URL, or provider configuration.

H2 — fail closed before network construction

Add one code-level disabled boundary at DatamailerClient.request() (or a shared configuration
boundary that every request unconditionally reaches). Under the disabled deployed contract, the
boundary runs before URL/header construction, requests.Session construction, request creation,
DNS, or socket access. If the client currently constructs a default session in __init__, make
that construction lazy or otherwise prove that disabled construction has no transport side effect.

The resulting behavior must satisfy all of these conditions:

1. Supplying a non-empty URL/API key/client/audience cannot enable a Datamailer request in a
   deployed web, worker, or migration process.
2. DATAMAILER_TRANSACTIONAL_DRY_RUN=1 remains distinct from disablement and cannot authorize a
   non-transactional send endpoint.
3. Disabled execution produces a deterministic typed outcome with a stable safe reason/category.
   The reason must not contain URL, API key/bearer value, recipient, payload, response body, or
   other secret/PII.
4. Existing callers keep their established safe behavior or the explicitly chosen typed behavior:
   no public traceback/500, no acknowledged provider work, and no retry loop that treats a
   disabled call as a provider attempt. The outbox path must classify the typed disabled outcome
   as terminal/disabled (or leave it safely unclaimed according to the chosen contract), never as
   a successful acknowledgement or ordinary transient retry.
5. There is one choke point. No second client/helper may bypass it.

The recommended method boundary is block every Datamailer HTTP method in deployed runtime
(GET, POST, PUT, and DELETE). Datamailer history/status reads remain migration/reconciliation
input and are not an operational website-provider exception. A GET-only exception is allowed only
if the owner explicitly approves an exact path allowlist with tests proving no mutation and no send
endpoint reachability. “Block write verbs” alone is not an accepted boundary because contact/list,
preference, campaign, and transactional routes must all be classified.

H2 does not remove or port any current send path. The Relay delivery lifecycle and Datamailer
freeze/drain/retirement remain owned by #49,
#50, and
#74.

H3 — narrow static boundary only; strict rule deferred

The strict raw rule over email_app/, events/, accounts/, and jobs/ is not an acceptance
criterion for this issue. The known accounts preference imports remain an explicit exception
owned by #227; the complete source inventory and adopted-path policy remain owned by #290.

The owner may choose one of these dispositions before engineering:

• Narrow H3 (recommended): a standard-library/AST-only check for newly owned boundary
  directories email_app/, events/, and jobs/, with every exception listed by exact path,
  owning issue, and reason; or
• Defer H3: no new static checker in Harden the Datamailer send boundary so misconfiguration fails closed #305; consume Pin the adopted Datamailer source surface in a static inventory #290's reconciled manifest later.

Neither disposition permits a blanket legacy exemption or a claim that accounts is Datamailer-free.

Explicit non-goals

• No Relay client, EmailDelivery model, durable delivery job, callback/reconciliation projection,
  template catalog, sender routing, preference migration, or replacement notification flow.
• No deletion, movement, refactor, or retirement of the copied Datamailer package.
• No removal or rewrite of existing CMP/course send call sites; their replacement is owned by the
  purpose and Relay issues.
• No change to certificate, enrollment, scoring, event, account, campaign, contact, or recipient-list
  business semantics beyond safe disabled failure at the existing provider boundary.
• No deployment secret injection, live URL/API key, provider credential, DNS/network mutation, queue
  drain, import/migration, production canary, sender activation, or real email.
• No claim of Build the durable Relay delivery intent and status lifecycle #49/Wire approved messages through Relay and retire Datamailer #50/Preserve graduates, certificates, and historical Wrapped records #58/Execute production cutover, monitor parity, and retire legacy writes safely #74/Extend account email preferences for event and newsletter delivery #227/Pin the adopted Datamailer source surface in a static inventory #290 completion, Relay readiness, migration completeness, or
  one-active-sender evidence.

Dependencies and ownership

Required before H1/H2 engineering dispatch

• PM records the owner decisions below and freezes the base/tree and affected request categories.
• All verification uses synthetic fakes and denied transport. No Relay, Datamailer, SES, Mailchimp,
  AWS, DNS, or production authority is available or used.
• The known scheduled-regression baseline is recorded separately; Harden the Datamailer send boundary so misconfiguration fails closed #305 evidence may not relabel
  unrelated failures as this issue's result.

Coordination, not H1/H2 blockers

• Build the durable Relay delivery intent and status lifecycle #49 owns website logical delivery intent/status and Relay delivery hardening.
• Wire approved messages through Relay and retire Datamailer #50 owns purpose routing, send-disabled history import, Datamailer freeze/drain/retirement
  preparation.
• Preserve graduates, certificates, and historical Wrapped records #58 owns certificate lifecycle/domain semantics; Remove request-time certificate notification fanout #233 owns later removal of request-time
  certificate notification fanout.
• Extend account email preferences for event and newsletter delivery #227 owns website-native account preferences and removal of the known account imports.
• Pin the adopted Datamailer source surface in a static inventory #290 owns the deterministic adopted Datamailer source-surface inventory.
• Execute production cutover, monitor parity, and retire legacy writes safely #74 owns authorized production cutover/retirement.

Owner decisions required before dispatch

These are open decisions, not implied adoption. Engineering must not infer a different behavior.

1. HTTP method boundary: approve the recommended all-method deployed block, or record the exact
   GET-only history/status path allowlist and its non-mutation/send-exclusion tests.
2. Disabled-call typing/handling: approve a stable typed disabled outcome. Recommended shape is
   a redacted DatamailerDisabledError (compatible with existing safe request-error handling) with
   a stable reason code; non-strict callers preserve their safe no-op/audit behavior, while strict
   worker/command paths fail safely without acknowledgement or blind retry. If a safe response
   status/result is preferred, record the mapping for public, job, and management-command callers.
3. Fake-only test opt-in: approve endpoint-shape tests using an injected synthetic fake session or
   equivalent explicit test transport. It must never be read from deployment environment/configuration
   and must fail if it reaches a real network. No operational enable switch is requested.
4. No-provider gates: approve zero-network evidence as mandatory: fake session request() is a
   failure sentinel, URL/header/session construction is checked to occur after the guard, and
   tests use only synthetic credentials/recipients/payloads. No external provider, DNS, socket,
   AWS, browser, or production data is permitted.
5. H3 disposition: choose narrow email_app/events/jobs AST coverage or defer H3 to Pin the adopted Datamailer source surface in a static inventory #290;
   strict accounts coverage remains deferred to Extend account email preferences for event and newsletter delivery #227/Pin the adopted Datamailer source surface in a static inventory #290 either way.

Allowed files and tests

Preferred implementation is limited to:

• core/tests/test_deployment_release.py for H1;
• course_management/datamailer/client.py and, only if required for the typed outcome,
  course_management/datamailer/client_types.py or a single adjacent exception module for H2;
• course_management/datamailer_outbox_dispatch.py and/or
  course_management/datamailer_outbox_retry.py only if required to keep a disabled outbox event
  from being acknowledged or retried; this is boundary handling, not a new delivery workflow;
• course_management/settings.py and/or deploy/task_definitions.py only if an explicit disabled
  setting must be defined/pinned; no live values or secret references;
• focused existing Datamailer tests under courses/tests/, data/tests/, and
  studio_courses/tests/, plus a narrowly scoped new test file if no existing home is suitable;
• for narrow H3 only, a standard-library AST checker and its test under scripts//scripts/tests/.

Do not edit callers, templates, migrations, provider packages, Relay code, or unrelated deployment
workflows. Any need outside this list stops implementation and returns the issue for re-grooming.

Required evidence:

• H1 per-workload exact safety-map assertions plus mutation/omission cases;
• H2 fake-session/no-network tests covering transactional, recipient-list, certificate, deadline,
  campaign, contact/list, and preference request shapes; incomplete config; dry-run-only behavior;
  one representative public caller; and one outbox worker path;
• typed reason redaction assertions and proof that disabled work is not acknowledged or blindly
  retried;
• narrow H3 AST checks only if adopted;
• applicable uv-backed Make migration, quality, type, CI, and container checks. No migration is
  expected; discovering a model change is out of scope and stops the issue.

H2 has no intended rendered-template change. The independent tester recomputes the change-selective
plan, runs the graph-selected backend Playwright tier, and marks screenshots not_applicable only
when the graph proves no changed page/error surface requires a screenshot. If an error/page surface
changes, desktop and mobile screenshots are required.

Acceptance and lifecycle gates

The implementation starts only after the owner decisions above are recorded. The engineer works
uncommitted in an isolated worktree and posts the frozen base/head, plan digest, exact file list,
tests, and no-provider evidence. A separate tester independently recomputes the plan, verifies every
H1/H2 criterion and security boundary, runs the focused Django and graph-selected Playwright checks,
and posts a terminal tester-final PASS with required screenshots or an evidenced not_applicable.

The PM then evaluates the tester evidence and scope, and posts ## Product Acceptance with ACCEPT
or REJECT. Only after tester PASS and PM ACCEPT may the engineer create a focused commit whose body
contains Closes #305; the orchestrator performs the local --no-ff merge/push, and on-call alone
observes the resulting CI. No pull request, deployment, provider enablement, or live email is part
of #305.

[alexeygrigorev]

Grooming Complete

PM grooming is complete against origin/main @ 2d3c28d1b8d1c2f3844f47f664945b42849d5eb9. The issue is now labelled decision and is ready for owner decisions, but engineering is not dispatched yet.

Approved issue boundary

• H1 (independent): strengthen core/tests/test_deployment_release.py so web, worker, and migration tasks assert the exact five-entry SAFETY_ENVIRONMENT map after metadata normalization, including empty DATAMAILER_URL and DATAMAILER_API_KEY; mutation/omission cases must fail. No secret or deployment value is added.
• H2 (independent after decisions): put one code-enforced disabled boundary at DatamailerClient.request() (with lazy/no-side-effect transport construction) before URL/header/session/DNS/socket work. Non-empty test configuration must not enable deployed runtime; dry-run remains distinct; the typed disabled outcome is redacted and cannot acknowledge or ordinarily retry an outbox event.
• H3: the raw strict rule over email_app/, events/, accounts/, and jobs/ is deferred. The known account preference imports are owned by #227, and the complete source inventory by #290. The owner may retain a narrow AST-only check for email_app/, events/, and jobs/, with exact owned exceptions, or defer H3 entirely to Pin the adopted Datamailer source surface in a static inventory #290. No blanket legacy exemption is allowed.

Non-goals and dependencies

This issue does not build Relay/EmailDelivery, alter send callers or business semantics, delete/refactor Datamailer, migrate preferences, drain queues, inject credentials, enable a provider, run a live send, or claim #49/#50/#58/#74/#227/#290 completion. #49/#50 own the Relay lifecycle and Datamailer retirement; #58/#233 own certificate lifecycle/fanout; #227/#290 own the deferred boundaries.

Owner decisions required (do not infer adoption)

1. Approve the recommended all-method deployed block (GET/POST/PUT/DELETE), or record an exact GET-only history/status allowlist with non-mutation/send-exclusion tests. “Write verbs only” is not sufficient.
2. Approve disabled-call typing/handling. Recommended: a redacted stable DatamailerDisabledError compatible with existing safe request-error handling; non-strict callers preserve safe no-op/audit behavior, while strict worker/command paths fail safely without acknowledgement or blind retry. Record any alternate public/job/command mapping.
3. Approve fake-only test opt-in via injected synthetic transport; no deployment environment flag may enable it, and the fake must fail on real network access.
4. Approve mandatory no-provider gates: a failing fake-session request sentinel; proof the guard precedes URL/header/session construction; synthetic credentials/recipients/payloads only; no Datamailer, Relay, SES, Mailchimp, AWS, DNS, socket, browser-provider, or production-data calls.
5. Choose narrow H3 or defer H3; strict accounts coverage remains with Extend account email preferences for event and newsletter delivery #227/Pin the adopted Datamailer source surface in a static inventory #290 either way.

Allowed files/evidence

Preferred files are core/tests/test_deployment_release.py; course_management/datamailer/client.py plus one adjacent typed-error/types module if needed; course_management/settings.py/deploy/task_definitions.py only for an explicit disabled setting/pin; existing focused Datamailer tests under courses/tests/, data/tests/, and studio_courses/tests/; and, only if narrow H3 is chosen, one AST checker plus tests under scripts/. Outbox dispatch/retry files may change only to classify disabled outcomes as non-acknowledged/non-retryable. Callers, templates, migrations, Relay code, provider packages, and unrelated workflows are out of scope.

Evidence must cover exact H1 safety-map mutation/omission cases; H2 request shapes for transactional, recipient-list, certificate, deadline, campaign, contact/list, and preference paths; incomplete config and dry-run-only cases; public and outbox safe failure; reason redaction; zero transport activity; and applicable uv-backed Make migration/quality/type/CI/container checks. No migration is expected. H2 has no render impact unless the tester finds a changed error/page surface; the independent tester recomputes the graph-selected Playwright tier and supplies required desktop/mobile screenshots or an evidenced not_applicable.

Lifecycle gate

After the owner decisions are recorded, one engineer works uncommitted in an isolated worktree and posts the frozen base/head, plan digest, file list, tests, and no-provider evidence. A separate tester independently recomputes and verifies the plan, focused Django/Playwright checks, and screenshots. PM then posts ## Product Acceptance with ACCEPT or REJECT. Only tester PASS + PM ACCEPT permit a focused Closes #305 commit, local --no-ff merge/push, and on-call CI observation. No pull request, deployment, provider enablement, or live email belongs to this issue.

[alexeygrigorev]

PM audit — decision gate remains (2026-09-02)

I re-audited #305 against the current website tree, the normative email/operations specifications, the adopted CMP contract, and the latest scheduled regression. The issue is correctly scoped, but it is not dispatchable yet: the five owner decisions already listed in the groomed specification are still not recorded. I have not changed code, configuration, credentials, or provider state.

Evidence and boundary

• The website remains pinned to CMP 98a235283904b4ef9ad29e196298540756cf1bcc. CMP main is currently 6d3cc0e0e0b9a1d28be1a8d53209555269e1d3fe; its post-pin changes are campaign payload/queue changes, with no DatamailerClient request-boundary or outbox retry contract change to port into Harden the Datamailer send boundary so misconfiguration fails closed #305.
• course_management/datamailer/client.py currently creates requests.Session() in DatamailerClient.__init__, then request() constructs the URL and bearer header and permits every currently exposed CMP method. The specialized clients cover transactional, recipient-list, contact/list, preference, campaign, status/history, and template request shapes. There is no deployed all-method guard.
• course_management/datamailer_outbox_dispatch.py claims a retryable event before dispatch. It catches only requests.RequestException; course_management/datamailer_outbox_retry.py classifies ordinary request/network failures as retryable until the attempt limit. A disabled typed result therefore needs an explicit terminal/non-acknowledged mapping; it must not be treated as a successful response or ordinary retry.
• deploy/task_definitions.py already defines the reviewed five-entry empty/disabled SAFETY_ENVIRONMENT map for web, worker, and migration. The release test currently checks the complete map only as a builder subset and directly asserts only DATAMAILER_TRANSACTIONAL_DRY_RUN, which is the H1 coverage gap.
• Specs 01, 05, 06, 07, 09, and 10 consistently require Relay to own future delivery, prohibit new direct Datamailer/SES sends, keep Datamailer read-only migration/history input, require safe redacted failures, and never restore Datamailer during rollback. These contracts do not select the concrete disabled exception/status mapping for this copied CMP surface.

The scheduled run 33673075817 is not #305 evidence: it failed on the known 102-finding development-terminology inventory, the corresponding Django assertion, and a cancelled Playwright component. No Datamailer-specific failure was reported.

Exact blockers requiring owner response

Please record one explicit answer for each item before an engineer is dispatched:

1. HTTP boundary: approve blocking GET, POST, PUT, and DELETE in deployed runtime, or provide the exact GET-only history/status path allowlist plus proof that it cannot reach mutation/send endpoints. A write-verb-only block is insufficient.
2. Disabled outcome: approve the stable redacted typed outcome (recommended DatamailerDisabledError plus a stable reason code) and its mapping for public callers, management commands, and outbox workers. State whether disabled outbox work becomes terminal/disabled or remains safely unclaimed; it must never be acknowledged or blindly retried.
3. Test transport: approve injected synthetic fake-session tests only; no deployment setting or non-empty environment may enable Datamailer, and the fake must fail on real network access.
4. No-provider gate: approve mandatory proof that the guard runs before URL/header/session construction, with sentinel transport calls, synthetic recipients/credentials/payloads, and no Datamailer/Relay/SES/Mailchimp/AWS/DNS/socket/browser-provider or production-data access.
5. Static H3: choose the narrow stdlib/AST check for email_app/, events/, and jobs/ with exact owned exceptions, or defer H3 to Pin the adopted Datamailer source surface in a static inventory #290. The known accounts preference imports remain Extend account email preferences for event and newsletter delivery #227/Pin the adopted Datamailer source surface in a static inventory #290 scope either way.

After those answers, PM can freeze the exact current base and request categories. The engineer must then work uncommitted, produce the versioned plan and no-provider evidence, and stop/re-groom if the fix needs callers, templates, migrations, provider code, or unrelated deployment changes. The independent tester and PM acceptance gates remain mandatory before any commit, merge, push, or on-call observation.

[alexeygrigorev]

PM decision packet — safe default and unanswered questions

What this controls: whether legacy Datamailer can make any runtime HTTP request, how a disabled call is represented, and whether disabled outbox work can be retried or acknowledged. It protects the boundary while Relay becomes the future delivery owner.

Recommended safe default: fail closed before URL, header, session, or network construction for every HTTP verb in deployed runtime. Return a stable redacted disabled outcome, classify outbox work as terminal and unacknowledged rather than successful, use only injected network-failing fake transports in tests, and require a pre-provider no-network proof. Keep the static import check narrow and explicit.

Exact unanswered owner question: approve or amend each of the five decisions: all-verb versus GET-only history allowlist; disabled exception/reason and worker mapping; fake transport policy; mandatory pre-provider/no-network gate; and the exact static H3 scope or deferral to #290. This issue remains open and is not dispatchable.