[the-gigi-pplx]

Summary

A sandbox placement could commit onto a node that had already been moved to drained or terminating, so the sandbox died with the instance. The phase writers lock the nodes row, but sandboxes.active_node_id has no foreign key to nodes, and its writers never took that lock. The pre-terminate recount in handle_terminating_node narrowed the window but couldn't close it: a placement could still land between the recount and the provider call.

This PR orders the two sides on the nodes row. Every live writer of active_node_id now takes the placement half of a termination fence, and the terminate path takes the other half, so a placement either commits first and is counted, or reads the committed phase and refuses.

The fence orders database placement commits against termination. It doesn't close runtime admission: a pod or VM can already be running before its placement is recorded, which is why a refused placement has to tear down what it can't record.

The mechanism

The whole protocol lives in crates/libs/common/svc/src/sql/node_placement_fence.rs:

Side	Lock	Result
Placement (fence_node_for_placement)	SELECT drain_phase … FOR KEY SHARE	Waits for a concurrent phase write, then reads the committed phase before writing active_node_id.
Drain and terminate (lock_node_for_phase_write)	SELECT 1 … FOR UPDATE in its own statement, with a transaction-local lock_timeout, then the guarded phase write	Waits for earlier placements, then recounts residency from a fresh snapshot.

• FOR KEY SHARE is the weakest lock that works. It conflicts with a phase write's FOR UPDATE and with node deletion, but not with other placements or with heartbeat and inventory traffic.
• The placement half is two statements, not one. FOR KEY SHARE re-reads the locked nodes tuple fresh after a wait, but evaluates the rest of its statement against the pre-wait snapshot. A carve-out subquery over sandboxes in the same statement would be stale.
• The fence refuses drained, terminating, an unrecognized phase, and a missing node row. It leaves draining open: by then a pod may already exist, and recording it is what makes the sandbox visible to node-drain's evacuation.
• A placement that only restates the row's current node passes through, so a retry doesn't tear down a live sandbox just because the node advanced after the original attach. Pod placement also requires the active Kubernetes execution to name the same node, so a replacement execution can't borrow its predecessor's pass.

What changed

• Six fenced writers: pooled_post_wake (controllers/sandbox/responses.rs), post_create_placement and create_finalize (controllers/sandbox/queries.rs), pooled_create (insert_pooled_sandbox, which fences before calling pplx_space_sandbox_store::insert_pooled), direct_restore (controllers/substrate/direct.rs), and pod_placement (controllers/substrate/kubernetes/store/pod_placement.rs). Writer-side policy and the refusal metric live in controllers/placement_fence.rs.
• Terminate half: node_drain::store's mark_terminating path takes the shared preamble and treats lock contention as a one-tick deferral. drained_node_terminator and orphaned_nodes take the same lock before their phase writes and deletes.
• A refusal tears down what it can't record.
  • Pooled create rolls its transaction back and returns the warm slot.
  • Post-create placement and create finalization roll the created sandbox back.
  • Direct restore and pod placement return RestoreInFlight, which keeps cleanup ownership with the caller.
  • A Kubernetes create whose placement fails claims the row as deleting before it returns, including a redrive that joined an abandoned activation.
    • Claim confirmed: the cancel guard deletes the pod (UID-fenced) and waits until the apiserver no longer has it, then retires the tracking and deletes the never-succeeded row. A pod still present at the deadline keeps its tracking. Create cleanup leaves a deleting row to it, and doesn't offer the client a retry of that sandbox.
    • Fork restore (restoring): the claim doesn't cover it. On the request path, before returning, the guard deletes the pod (UID-fenced), waits until the apiserver no longer has it, and only then retires the claim, so create cleanup deletes the dead shell. A pod that outlives its delete keeps the claim, and the create reports RestoreInFlight instead of the placement error, so create cleanup keeps the row with it for the transient-lifecycle sweeper rather than leaving a live pod with neither.
    • Claim not confirmed: either ownership moved on or the claim's write failed. The error comes back as RestoreInFlight, so create cleanup keeps the row the pod's cleanup goes through. When ownership moved on, the guard leaves the pod alone.
    • Ordinary cancellation: keeps main's behavior. It retires the claim with its pod and leaves the creating row for a retried create.
• Create compensation is a durable operation of its own kind.
  • rollback_created_sandbox moves the row to deleting and queues a compensate_create_sandbox operation, already claimed, in one transaction, before touching the substrate. The request then runs the first attempt itself, through the driver and executor the dispatcher uses, and waits for it as main waited for its teardown. So the teardown also runs with SPACE_PUBLIC_OPERATION_DISPATCHER_ENABLED=false. The dispatcher retries a failed attempt and reclaims one whose process died.
  • The payload pins the exact execution (the Direct route identity, or the Kubernetes generation, attempt, and pod UID), so a delayed retry refuses a successor execution that reused the sandbox id.
  • The teardown requeues on an unconfirmed node or pod delete instead of applying the user DELETE contract, and gets a larger but finite retry budget. A Kubernetes pod counts as deleted only once the apiserver no longer has it, not when it accepts the delete, and its tracking is kept until then.
  • It checks ownership before the teardown: a row that has left deleting belongs to whatever took it over, and the compensation declines without touching the substrate.
  • It runs even when the row is already gone, since a best-effort user DELETE can remove the row while the guest still runs. On Kubernetes it then deletes the pod by its persisted UID, though no execution row names it any more. It leaves alone a successor's route receipt.
  • Claims select by kind, so a controller that predates compensate_create_sandbox never claims one. The new create_rollback_fence payload key is skipped when absent, so user lifecycle payloads serialize as before.
  • The kind is internal: OperationKind lists it after an internal marker, which keeps it out of the published schema, so the M1 contract is unchanged.
  • The retry budget for this kind lives in public_operation_driver. Every requeue spends it, whatever the reason, including the workflow-progress and backpressure requeues that don't for other kinds. Compensation isn't an M1 kind and has no deadline, so the budget is what guarantees it reaches a terminal state.
  • On the direct substrate, an attempt records on its operation that its node delete ran, before anything can lose that fact: atomically with its receipt clear, or on its own when there is no receipt to clear. The write is guarded by the attempt's claim, so an attempt whose claim went stale records nothing and clears nothing, and the attempt that holds the claim repeats the fenced delete. Every later attempt that finds the record only verifies: a receipt present then is a successor's and a conflict, and otherwise the node decides. It never repeats the delete, which names the sandbox and not the execution.
  • Verification asks the node after its delete has returned. An explicit "missing" confirms the teardown and the row is cleaned up; a sandbox the node still has landed after the delete and is a conflict. Under a claim, an answer that doesn't come is retried as an operation, which is safe because the retry only verifies. Without one (the claimed immediate teardown, which has no operation to retry it), it stays a conflict.
  • Needs PR [db] drop public_operations_kind_check; the app owns the kind set #558, now merged, which drops public_operations_kind_check.
  • If the operation can't be queued, the rollback makes the same claim in a transaction of its own (row lock, lifecycle check, and the check that the create's execution is still the active one) and tears down immediately. A stop that commits first, or a successor whose receipt or placement shows it, makes it decline instead of tearing down a sandbox it doesn't own. A direct successor on the same node that hasn't persisted its receipt yet is still claimed.
• Claim renewal never waits on the operation's row lock. The driver renews with the execution suspended, and an execution can hold that lock in an open transaction (a compensation's teardown record, or the fork executor's payload writes), so a blocking renewal could wait on a transaction that can't progress. A locked row now counts as still owned for that tick when a plain read still shows the claim.
• orphaned_sandboxes claims abandoned creates reached through the Forgotten guard as well as the pod-name guard.
• Kill switches, both on by default, documented in docs/configuration.md, and turned off only by a recognized false value. Set each on the API and the cluster controller together.
  • SPACE_PLACEMENT_TERMINATION_FENCE: disabling removes the row lock as well as the check.
  • SPACE_CREATE_COMPENSATION_QUEUE: off, a failed create claims its sandbox and tears it down immediately, with no operation to retry it, and writes no compensate_create_sandbox operation. A teardown that fails then leaves the row deleting, and on the direct substrate nothing else collects it or its guest. Both processes run creates, which is why both need it.
• Metrics: placement_fence.refusals{writer,verdict} counts refusals, and placement_fence.termination_contention_deferrals counts terminate-side deferrals after a bounded lock wait. Compensation's confirmed pod deletes report through the existing blocking-delete latency metric with operation create_compensation.
• Error classification: the direct-restore and pod-placement fence reads map SQL errors through DbContext, so a lost lock race there stays the retryable LockContention.
• Architecture baseline: raises three sandbox_writes caps in architecture-baseline.toml: controllers/sandbox/queries.rs 4 to 5, kubernetes/store/pod_tracking.rs 5 to 7, and orphaned_sandboxes.rs 5 to 7. That's the ratchet's documented exception for another write of the sandboxes row in a listed file, and archcheck reports it as advisory. The new writes are create cleanup's shell delete, the cancelled-create claim and its row delete, and orphaned_sandboxes' claims of abandoned creates. They stay in those files rather than moving into pplx_space_sandbox_store in this PR.
• CI guard: scripts/source_guards/placement_fence_writers.py finds every production SQL statement that assigns a non-NULL sandboxes.active_node_id, requires a fence call in the same function, and pins the site count at six.
  • A writer that only receives an executor counts as fenced through its single named caller (FENCED_BY_CALLER), and fails if a second caller appears.
  • The one exemption is the dormant completion_sql in crates/libs/placement/machine, which has no production caller.
  • The guard runs through scripts/tests/test_placement_fence_writers_guard.py in the repo's Python test discovery.

Dependencies

• [db] drop public_operations_kind_check; the app owns the kind set #558, merged. It dropped public_operations_kind_check, which listed the operation kinds and not compensate_create_sandbox; with it in place, every compensation insert would fail. This PR targets main and is rebased onto that merge.

Rollout order

1. PR [db] drop public_operations_kind_check; the app owns the kind set #558's migration, which drops the operation-kind CHECK.
2. The cluster controller.
3. The API.

Deploying the API ahead of the controller opens two windows that the reverse order doesn't:

• An older controller doesn't claim compensate_create_sandbox, so a compensation whose inline attempt fails waits in deleting with its guest running.
• An older controller's node-reaping DELETEs don't take the phase lock first. A delete that waits behind a fenced placement then evaluates its residency check on the snapshot from before the wait.

An older API can't decode a compensate_create_sandbox row either. On an older replica during rollout, or after a rollback, operation fetch and the admin listing fail while such a row exists, terminal rows included. Claims aren't affected, since they select by kind.

Rolling back

A rollback has to drain the new operation kind before it removes the code that runs it, and PR #558's down migration refuses while any compensate_create_sandbox row exists. In order:

1. Set SPACE_CREATE_COMPENSATION_QUEUE=false on the API and the cluster controller. Failed creates then tear down immediately and write no new rows of the kind.
2. Keep this cluster controller running until every queued compensation is terminal. It still claims the kind, and each one ends within its retry budget.
3. Wait for retention to delete the terminal rows. gc_after is set when the operation is inserted, 7 days after its creation, not after it completes.
4. Roll back the API and the cluster controller, then run PR [db] drop public_operations_kind_check; the app owns the kind set #558's down migration.

Rolling the cluster controller back before step 2 finishes strands the queued compensations. An older controller never claims the kind, and on the direct substrate nothing else tears down their guests.

Deliberately not covered

• On a direct compensation whose route receipt disappeared during the teardown, a successor that lands on the node after the compensation's state check isn't seen. With the receipt gone and the node empty at the check, the compensation reports its teardown done.

• When two attempts of one compensation overlap, the one that holds the claim clears the receipt and deletes the row, even when the stale one found a sandbox on the node and reported a conflict. The stale one's result is dropped. A successor that landed between the two deletes can be left without a row.

• An attempt cancelled after its node delete returns but before its teardown record commits leaves nothing recorded. With a receipt, the next attempt finds it still current and repeats the fenced delete. Without one, the next attempt repeats the sandbox-id delete, as before this change.

• A created pod whose create response carries no valid UID can't be named for a fenced delete. Its row and execution claim are kept for reconciliation, but the create returns a retryable 503.

• A direct compensation's node delete names the sandbox, not the execution. A successor activation that landed on the same node before that call is deleted with it. The compensation then leaves the successor's receipt and row alone and stops with a conflict, so they point at a guest that's gone.

• Several paths leave the last step to the orphaned-sandbox reconciler, and that reconciler acts only with SPACE_API_RECONCILER_DRY_RUN=false; the default is dry-run. The paths:

  • durable cleanup interrupted by a process crash;
  • a deleting row an earlier placement already attached to a node;
  • a joined redrive's claimed cleanup;
  • a Forgotten creating shell;
  • an expired create whose pod is still present.

  In dry-run, these stay deleting or creating until someone acts by hand.

• A pod scheduled before the drain began keeps running, since cordon only stops new scheduling. Until its placement persists, the row carries a NULL active_node_id and is invisible to every occupancy term. The fence makes the later placement refuse instead of recording a healthy sandbox on a dying node, but the guest on that node is lost either way.

• A create refused for a missing node row (Direct substrate) can't reach the node to tear down, so its compensation retries to budget and leaves the row in deleting.

• An action already in flight can still overwrite deleting when its result lands during the compensation's teardown call. Compensation checks ownership before the call, but no lock spans it. That post-result write replaces cross-intent lifecycles in general and is characterized as open in lifecycle/characterization_tests.rs.

• With the dispatcher disabled, a compensation gets only the attempt its request runs. A failed attempt stays queued, with its row in deleting.

• A Kubernetes pod that stays Terminating (a finalizer, or a kubelet that never reports back) fails every confirmed delete, so its compensation retries to budget and leaves the row in deleting with its execution still tracked.

• An ordinary cancel guard's claim retirement is fenced on generation and attempt, not lifecycle, as on main. The lifecycle check applies only on the durable path.

• The four controller-side writers don't classify a lock-timeout error (55P03) from the fence; the two substrate writers do. This has no effect while the app role has no lock_timeout. The coupling is recorded at the statement.

Test plan

• cargo fmt --all -- --check
• cargo clippy --workspace --all-targets --locked -- -D warnings
• cargo nextest run -p pplx-space-api -p pplx-space-cluster-controller-server -p pplx-space-common-svc -p pplx-space-metrics -p pplx-space-public-api -p pplx-space-db-test-fixtures: 3314 passed, 1 skipped, against Postgres with the PR [db] drop public_operations_kind_check; the app owns the kind set #558 migration applied.
• Compensation regression tests, each failing with its mechanism reverted: compensation_leaves_a_sandbox_that_left_deleting_untouched, create_rollback_tears_down_without_a_dispatcher, create_rollback_falls_back_to_a_claimed_teardown_when_queueing_fails, compensation_claim_declines_a_successor_execution, create_compensation_deletes_its_pod_by_uid_after_the_row_is_gone, create_compensation_releases_tracking_only_once_the_pod_is_gone, cancel_guard_durable_cleanup_retires_tracking_only_once_the_pod_is_gone, owned_restore_fence_contention_is_retryable, overlapping_compensation_attempts_finish_the_teardown, create_rollback_queues_nothing_with_the_queue_switch_off, and compensation_requeues_always_spend_the_retry_budget, an_unconfirmed_teardown_stays_retryable_and_is_only_verified, a_compensation_cancelled_after_its_teardown_is_only_verified_on_reclaim, a_stale_compensation_claim_clears_no_receipt, a_legacy_teardown_records_itself, claim_renewal_does_not_wait_on_the_operations_row_lock, and cancel_guard_release_now_retires_before_returning. The second and third run through the operation driver with the production store and executor: one forces a teardown whose probes fail (the worker is stopped mid-attempt) and asserts the operation is requeued with its teardown recorded, then that a retry succeeds once the node answers; the other cancels an attempt between its teardown and its row cleanup and reclaims it. Each has a control where a successor landed on the node and must survive the retry. overlapping_compensation_attempts_finish_the_teardown forces the interleaving: it holds the route receipt row, so one attempt passes its receipt check and node delete and then waits in its receipt clear, and the test commits the other attempt's clear while it waits. Its control has the node still holding the sandbox, which must stay a conflict. a_compensation_cancelled_before_its_row_cleanup_is_finished_by_the_next_attempt covers an attempt cancelled between its receipt clear and its row delete. That case takes the absent-receipt arm, which already handled it, so the test guards that path rather than this round's probe. The fallback's identity check is tested through the claim it shares with the queued path, not through a successor racing the fallback itself. The pod-placement fence's classification has no test: that writer locks the node row before the fence reads it, so the read doesn't wait on a lock there.
• The ordering tests in node_drain/store/placement_fence_tests.rs interleave two live transactions, with pg_blocking_pids as the barrier. They cover a placement committing first, a phase write committing first, and a node delete committing mid-placement.
• Per-writer refusal tests in controllers/sandbox/queries/placement_fence_tests.rs pair each refused case with an open-node case, so a writer that stopped working can't pass as fenced.
• python -m pytest scripts/tests/test_placement_fence_writers_guard.py: 35 passed. Each new guard test fails with its mechanism reverted.
• The routes/public_docs M1 contract snapshot is unchanged.

[ac-pplx]

[P2] Keep ambiguous receipt disappearance retryable during Direct compensation

In direct.rs, Absent with receipt_matched_before = true is treated like a successor replacing the receipt. Another attempt of the same compensation can also remove it.

If attempt A remains active after its lease becomes reclaimable, replacement B can read the same matching receipt and issue the same idempotent teardown. A can then clear the receipt but lose its claim before parent/reconcile cleanup. B observes Absent, returns Conflict, and the operation executor records a terminal failure. The sandbox row remains deleting without a retry owner; Direct has no Kubernetes orphan-cleanup backstop.

Please preserve successor protection while making this ambiguous outcome retryable, or persist evidence that the same compensation completed teardown. Add a regression covering overlapping claims and cancellation after receipt removal but before parent cleanup. Unconditionally treating absence as success would weaken successor protection.

The overlapping substrate behavior was reproduced with PostgreSQL and a simulated worker. Lease-loss cancellation and terminal error handling were separately verified in source.

Review-Key: #5595e4129b

[the-gigi-pplx]

Thanks — the ambiguous case was real: a second attempt of the same compensation (a stale claim reclaimed while the first still ran) clears the receipt, and the old code made that terminal with the row left deleting. Fixed in c72bee1.

After its delete RPC returns, the attempt now asks the node:

• an explicit "missing" confirms the teardown, and the row is cleaned up;
• a sandbox still on the node landed after the delete, and stays a conflict;
• a node that doesn't answer is probed again a bounded number of times, then ends as the conflict it was before. Retrying that case as an operation would let the next attempt, which finds no receipt, delete whatever the node holds.

Regression coverage is Postgres-backed with a simulated worker. overlapping_compensation_attempts_finish_the_teardown forces the overlap: it holds the receipt row so one attempt passes its check and node delete and then waits in its clear while the other attempt's clear commits, with a control where the node still has the sandbox. It fails against the previous behavior. a_compensation_cancelled_before_its_row_cleanup_is_finished_by_the_next_attempt covers an attempt cancelled between its receipt clear and its row delete; that case takes the absent-receipt arm, which already handled it, so the test guards that path.

The same push also makes every compensation requeue spend its retry budget, so a compensation always reaches a terminal state, and adds SPACE_CREATE_COMPENSATION_QUEUE so a rollback can stop new compensations while the current controller drains the queued ones. The description's "Rolling back" section has the order.

[the-gigi-pplx]

Addressed in c72bee1. That head postdates the commit this review was posted against, 5e4129b.

delete_created_execution no longer treats a receipt that matched and then went missing as a successor. After its own delete RPC it probes the node. If the node reports the sandbox Missing, the teardown is confirmed and the compensation finishes and deletes the row. If the node still has the sandbox, something landed after this attempt's delete, so it stays a Conflict. That keeps successor protection, and the attempt that cleared the receipt owns the row. A node that never answers is also a Conflict, because a later attempt would find no receipt and delete by sandbox id.

Regression coverage, in controllers/sandbox/updates/compensation_tests.rs:

• overlapping_compensation_attempts_finish_the_teardown forces the interleaving against Postgres and a simulated worker. It holds the receipt row so this attempt passes its receipt check and its node delete, then waits in its clear. The other attempt deletes the receipt meanwhile. Without a successor, the compensation ends TornDown, with the guest and the row gone. As a control, a successor landed on the node yields Conflict, the guest stays and the row stays deleting.
• a_compensation_cancelled_before_its_row_cleanup_is_finished_by_the_next_attempt covers cancellation after the receipt clear but before the row cleanup: the next attempt confirms absence on the node and deletes the row.

With the 5e4129b behaviour restored (Absent after a match returns Conflict), the overlap test fails with the 409 described here. At c72bee1, both tests pass.

[ac-pplx]

[P2] Persist verification progress so failed probes retain cleanup ownership

Rechecked c72bee1. The explicit Missing response now fixes the healthy overlap case, and the new regression passes. The unanswered-probe branch in direct.rs still leaves part of the previous finding unresolved.

If the attempt that removed the receipt stops before parent cleanup, the replacement attempt must finish that cleanup. A temporary worker outage during its three state probes returns Conflict, which the executor maps to terminal failure. The parent stays deleting, and the failed operation is no longer claimable. Direct has no Kubernetes orphan-cleanup backstop. The guest can already be gone; the remaining defect is abandoned cleanup and loss of automatic recovery.

I agree that simply making this error retryable is unsafe: the next attempt sees no receipt and can issue another sandbox-ID delete against a successor. Please persist enough teardown/verification progress, or transfer ownership to an equivalent durable cleanup mechanism, so recovery can continue verification and parent cleanup without repeating that delete. Cancellation during the probe needs the same recovery contract. More in-attempt probes do not preserve this distinction across attempts.

Validation: 21 focused tests passed with PostgreSQL/Redis and simulated workers, without dependency skips. A temporary harness using the production operation store, executor, and driver reproduced three failed probes followed by terminal failed/conflict, a deleting parent, no receipt, and no reclaimable operation. It used the checked-in overlap test's locked-row stand-in for the competing attempt, then stopped the simulated worker. Aborting after the first failed probe also allowed reclaim with the original fence but no verification-progress marker. Actual simultaneous lease-loss scheduling and successor deletion were not reproduced.

Please add regression coverage for failed probes and cancellation during verification. This is an incomplete repair of the existing P2, not a new regression introduced by c72bee1.

Review-Key: #559c72bee1

[nikhilgupta-pplx]

+1 to ac's latest review. The placement fencing looks sound, but the compensation handoff still loses ownership after teardown. If an overlapping attempt clears the receipt and is cancelled before parent cleanup, failed worker probes permanently fail the operation and leave the Direct row in deleting; cancellation also forgets that a retry must only verify. Please make that phase durable, keep inconclusive probes retryable, and cover failure and cancellation through the operation driver.

Separately for rollout: #558 needs to land first, and since old API replicas can fail reads once new-kind rows exist (including terminal history), the rollback boundary needs to be explicit rather than relying on controller-first ordering alone.

[nikhilgupta-pplx]

[P2] Keep ownership of the verification-only cleanup durably. If an overlapping attempt clears the receipt and is cancelled before deleting the parent row, three unavailable worker probes here become Conflict, which permanently fails the compensation operation and strands the Direct row in deleting. Cancellation also loses receipt_matched_before, so a reclaimed attempt takes the sandbox-ID delete path again. Please persist the post-teardown verification phase, retry inconclusive probes, and cover both failed probes and cancellation during verification through the operation driver.

[nikhilgupta-pplx]

The durable teardown marker addresses my blocker: it commits with receipt removal under the current operation claim, and reclaimed attempts only verify. The driver tests cover retry, cancellation, and preserving a successor receipt.

LGTM. Keep the documented rollout/rollback ordering, and get ac’s re-review before landing.

[the-gigi-pplx]

Thanks for the driver-level reproduction. This is addressed at 1c30bc2: verification progress is now durable, so a replacement attempt continues verification and parent cleanup without repeating the delete, and an unanswered probe is retryable.

Persisted progress. After its node delete returns, an attempt records create_rollback_teardown_issued on its operation's payload, guarded by its claim: operation id, claim token, claim generation and state = 'running' (public_operations.rs). The record commits in the same transaction as the attempt's receipt clear, or as its finding the receipt already gone (direct.rs). On the legacy no-receipt path it commits on its own (direct.rs). A stale attempt records and clears nothing (ClaimLost), so the attempt that holds the claim still finds the receipt and repeats the fenced delete instead of inferring a teardown from its absence.

Recovery only verifies. The executor passes the claim and the recorded flag through (public_sandbox_operations.rs; dispatcher). An attempt that finds the record takes verify_recorded_create_rollback and never deletes (direct.rs, L1728-L1773): any receipt then is a successor's and a conflict; otherwise the node decides, and an answer that doesn't come is Unavailable, so the operation is requeued rather than failed. The same holds on the first attempt's own probe once it has recorded (direct.rs).

Cancellation during verification. The record always commits before any probe, so an attempt cancelled during its probe leaves the record, no receipt, and the row in deleting, and the reclaiming attempt verifies.

Regression coverage, through the operation driver with the production store and executor, Postgres and simulated workers (compensation_tests.rs):

• an_unconfirmed_teardown_stays_retryable_and_is_only_verified: the attempt passes its node delete, the receipt is cleared under it, and the worker is stopped so every probe fails. The operation is requeued with the record set, not failed. Once the node is back, the retry succeeds and the row is gone.
• a_compensation_cancelled_after_its_teardown_is_only_verified_on_reclaim: the attempt is aborted after its teardown is recorded and before its row cleanup, then reclaimed. That's the same durable state a cancellation mid-probe leaves; the test doesn't abort mid-probe itself.
• Both have a control where a successor landed on the node; it must survive the retry, and both fail against the previous behavior.
• a_stale_compensation_claim_clears_no_receipt covers the claim guard.

What's left: an attempt that ends between its node delete returning and that record committing leaves nothing recorded. With a receipt, the next attempt finds it still current and repeats the fenced delete; on the legacy path it repeats the sandbox-id delete, as before. It's listed in the description.