Skip to content

fix: allow safe Boat destroy after suspension - #207

Merged
dnth merged 2 commits into
mainfrom
fm/fm-boat-destroy-suspended
Oct 5, 2026
Merged

dnth merged 2 commits into
mainfrom
fm/fm-boat-destroy-suspended

Conversation

@dnth

@dnth dnth commented Oct 5, 2026

Copy link
Copy Markdown
Owner

Intent

Fix Boat F3: bin/fm-boat.sh destroy on a suspended remote placement fails with SSH 255 before it reaches the provider operation. The wrapper's remote checks (sleep-reconcile and children via fm-on.sh) require a reachable host, and a stopped sandbox is unreachable. The fail-closed outcome is correct but the UX is wrong: sleep then destroy requires a wake first. In the destroy branch, when the recorded lifecycle is suspended or provisioned and the sleep-time checks already proved the route idle, skip the remote reachability checks. Keep them for a running placement, and keep every other destroy safety check (live children, unlanded work, unresolved decisions, the --yes confirmation). Evidence: boat-lifecycle-proof report section 3, F3 row, recommendation 3. Tests must use a fake or stub provider and SSH; never create, wake, or destroy a real Boat sandbox.

Binding firstmate tightening (supersedes the plain lifecycle check): lifecycle=suspended alone does NOT prove the sleep-time quiescence/work checks ran - fm-boat.py compensate() also sets suspended after a FAILED wake of an ever-ready sandbox, without those checks, and sleep(..., destroy=True) then deletes directly. So bypass the remote SSH checks in destroy ONLY when there is durable proof that a successful sleep-time quiescence/work check completed for the current generation (record such proof at sleep, since none existed), and refuse otherwise with a clear message. Implementation choice: sleep records sleep_quiesced=1 in the Boat record only when the wrapper's remote checks ran and passed (FM_BOAT_SLEEP_QUIESCED=1); wake clears it before transitioning, so failed-wake compensation leaves suspended without proof; the wrapper refuses an unproven dormant destroy before any remote call ("wake it so destroy can run remote checks"); fm-boat.py re-verifies the proof under its lock (FM_BOAT_DESTROY_DORMANT=1) and refuses if the placement changed.

Acceptance criteria:

  • AC1: a regression test proves destroy on a suspended placement with an unreachable host proceeds to the provider delete (fails on the parent commit, passes after the fix). Plus an AC4-style regression: a failed-wake-compensation suspended placement WITHOUT the proof must still refuse destroy and not reach the provider delete.
  • AC2: a test proves destroy on a RUNNING placement still performs the remote checks and still refuses when they fail.
  • AC3: changed tests green and bin/fm-lint.sh clean; the PR's full GitHub CI suite is the broad-regression gate.

Firstmate-Validation-Generation: 6c7d0ce5a7622acefad52f680469db1f

What Changed

  • Allow dormant Boat placements with durable sleep-time quiescence proof to be destroyed without remote SSH checks; refuse unproven dormant deletion and retain remote checks for running placements.
  • Record proof after checked sleep, clear it before wake, and recheck lifecycle and proof under the deletion lock.
  • Document the behavior and add fake-provider/SSH regressions for suspended deletion, failed-wake refusal, running-placement checks, and required confirmation.

Risk Assessment

✅ Low: The change is bounded, preserves existing safety checks, and revalidates dormant deletion proof under the lifecycle lock.

Testing

Changed routing tests and focused lifecycle tests passed. Isolated real-CLI checks with fake provider/SSH confirmed dormant deletion and safety refusals, with product transcripts captured; parent replay reproduced SSH 255. No real sandbox was contacted. Lint and broad CI were left to their assigned phases; disposable files were removed.

  • Live validation: ✅ go - 7 of 7 scenarios driven live against the product
Scenario Result Live Evidence
Destroy a checked suspended placement with unreachable SSH: provider deletes it and the placement record disappears ✅ pass live CLI behavior and provider-state transcript; Parent commit reproduces SSH 255
Destroy a running placement with unreachable SSH or active remote children: deletion is refused ✅ pass live CLI behavior and provider-state transcript
Attempt sleep with active remote children: sleep refuses and records no quiescence proof ✅ pass live CLI behavior and provider-state transcript
Fail wake after a checked sleep, then destroy: compensation clears proof and destroy refuses before SSH or provider deletion ✅ pass live CLI behavior and provider-state transcript
Destroy a proven dormant placement with pending delivery, an unresolved decision, or missing --yes: deletion remains refused ✅ pass live CLI behavior and provider-state transcript
Destroy a provisioned placement: unproven dormancy refuses; checked sleep enables deletion without SSH ✅ pass live CLI behavior and provider-state transcript
Request dormant deletion after wake changes the placement or unchecked sleep removes proof: the provider-side guard refuses ✅ pass live Provider-side proof revalidation
Evidence: CLI behavior and provider-state transcript

$ fm-boat.sh provision ios --identity ~/.no-mistakes/worktrees/dd71c22cc6d7/01M45X42MBT8DXPE5S08335KCW/.validation-tmp/fm-boat-routing.xnzdOQ/world/identity --alias fm-sm-ios-runpod --model openai-codex/fixture
provisioned: ios
exit=0
provider calls: boat new,curl --config,curl --config,boat info,boat stop,boat info,
remote calls: 
{"state":"archived","fail":[]}
lifecycle=provisioned
ever_ready=0

$ fm-boat.sh wake ios
ready: ios
exit=0
provider calls: boat resume,curl --config,curl --config,ssh -o,ssh -o,
remote calls: 
{"state":"ready","fail":[]}
lifecycle=ready
ever_ready=1

$ fm-boat.sh destroy ios --yes
exit=255
provider calls: 
remote calls: 
{"state":"ready","fail":[]}
lifecycle=ready
ever_ready=1

$ fm-boat.sh destroy ios --yes
reconciled=0
error: remote child work is active or unknown
exit=1
provider calls: 
remote calls: fm-sm-ios-runpod fm-remote-secondmate-control.sh sleep-reconcile,fm-sm-ios-runpod fm-remote-secondmate-control.sh children,
{"state":"ready","fail":[]}
lifecycle=ready
ever_ready=1

$ fm-boat.sh sleep ios
reconciled=0
error: remote child work is active or unknown
exit=1
provider calls: 
remote calls: fm-sm-ios-runpod fm-remote-secondmate-control.sh sleep-reconcile,fm-sm-ios-runpod fm-remote-secondmate-control.sh children,
{"state":"ready","fail":[]}
lifecycle=ready
ever_ready=1

$ fm-boat.sh sleep ios
reconciled=0
retired: remote-reply-ios
suspended: ios
exit=0
provider calls: boat info,boat stop,boat info,
remote calls: fm-sm-ios-runpod fm-remote-secondmate-control.sh sleep-reconcile,fm-sm-ios-runpod fm-remote-secondmate-control.sh children,
{"state":"archived","fail":[]}
lifecycle=suspended
ever_ready=1
sleep_quiesced=1

Dormant local safety guards:

$ fm-boat.sh destroy ios --yes
error: undelivered backlog handoff
exit=1
provider calls: 
remote calls: 
{"state":"archived","fail":[]}
lifecycle=suspended
ever_ready=1
sleep_quiesced=1

$ fm-boat.sh destroy ios --yes
error: unresolved decisions
exit=1
provider calls: 
remote calls: 
{"state":"archived","fail":[]}
lifecycle=suspended
ever_ready=1
sleep_quiesced=1

$ fm-boat.sh destroy ios --yes
exit=1
provider calls: 
remote calls: 
{"state":"archived","fail":[]}
lifecycle=suspended
ever_ready=1
sleep_quiesced=1

$ fm-boat.sh destroy ios
retired: remote-reply-ios
usage: fm-boat.py destroy [-h] --yes id
fm-boat.py destroy: error: the following arguments are required: --yes
skipped: remote-reply-ios suspended
exit=1
provider calls: 
remote calls: 
{"state":"archived","fail":[]}
lifecycle=suspended
ever_ready=1
sleep_quiesced=1

$ fm-boat.sh wake ios
error: Boat resume refused; compensation confirmed
exit=1
provider calls: boat resume,boat info,
remote calls: 
{"state":"archived","fail":["resume"]}
lifecycle=suspended
ever_ready=1

$ fm-boat.sh destroy ios --yes
error: dormant placement has no sleep-time quiescence proof; wake it so destroy can run remote checks
exit=1
provider calls: 
remote calls: 
{"state":"archived","fail":["resume"]}
lifecycle=suspended
ever_ready=1

$ fm-boat.sh wake ios
ready: ios
exit=0
provider calls: boat resume,curl --config,boat exec,curl --config,ssh -o,
remote calls: 
{"state":"ready","fail":[]}
lifecycle=ready
ever_ready=1

$ fm-boat.sh sleep ios
reconciled=0
retired: remote-reply-ios
suspended: ios
exit=0
provider calls: boat info,boat stop,boat info,
remote calls: fm-sm-ios-runpod fm-remote-secondmate-control.sh sleep-reconcile,fm-sm-ios-runpod fm-remote-secondmate-control.sh children,
{"state":"archived","fail":[]}
lifecycle=suspended
ever_ready=1
sleep_quiesced=1

$ fm-boat.sh destroy ios --yes
retired: remote-reply-ios
deleted: ios
exit=0
provider calls: boat info,boat delete,
remote calls: 
{"state":"deleted","fail":[]}
placement record absent

$ fm-boat.sh provision ios --identity ~/.no-mistakes/worktrees/dd71c22cc6d7/01M45X42MBT8DXPE5S08335KCW/.validation-tmp/fm-boat-routing.xnzdOQ/world/identity --alias fm-sm-ios-runpod --model openai-codex/fixture
provisioned: ios
exit=0
provider calls: boat new,curl --config,curl --config,boat info,boat stop,boat info,
remote calls: 
{"state":"archived","fail":[]}
lifecycle=provisioned
ever_ready=0

$ fm-boat.sh destroy ios --yes
error: dormant placement has no sleep-time quiescence proof; wake it so destroy can run remote checks
exit=1
provider calls: 
remote calls: 
{"state":"archived","fail":[]}
lifecycle=provisioned
ever_ready=0

$ fm-boat.sh sleep ios
reconciled=0
retired: remote-reply-ios
suspended: ios
exit=0
provider calls: boat info,
remote calls: fm-sm-ios-runpod fm-remote-secondmate-control.sh sleep-reconcile,fm-sm-ios-runpod fm-remote-secondmate-control.sh children,
{"state":"archived","fail":[]}
lifecycle=provisioned
ever_ready=0
sleep_quiesced=1

$ fm-boat.sh destroy ios --yes
retired: remote-reply-ios
deleted: ios
exit=0
provider calls: boat info,boat delete,
remote calls: 
{"state":"deleted","fail":[]}
placement record absent
Evidence: Parent commit reproduces SSH 255

$ fm-boat.sh provision ios --identity ~/.no-mistakes/worktrees/dd71c22cc6d7/01M45X42MBT8DXPE5S08335KCW/.validation-tmp/fm-boat-routing.wi2lWm/world/identity --alias fm-sm-ios-runpod --model openai-codex/fixture
provisioned: ios
exit=0
provider calls: boat new,curl --config,curl --config,boat info,boat stop,boat info,
remote calls: 
{"state":"archived","fail":[]}
lifecycle=provisioned
ever_ready=0

$ fm-boat.sh wake ios
ready: ios
exit=0
provider calls: boat resume,curl --config,curl --config,ssh -o,ssh -o,
remote calls: 
{"state":"ready","fail":[]}
lifecycle=ready
ever_ready=1

$ fm-boat.sh sleep ios
reconciled=0
retired: remote-reply-ios
suspended: ios
exit=0
provider calls: boat info,boat stop,boat info,
remote calls: fm-sm-ios-runpod fm-remote-secondmate-control.sh sleep-reconcile,fm-sm-ios-runpod fm-remote-secondmate-control.sh children,
{"state":"archived","fail":[]}
lifecycle=suspended
ever_ready=1

$ fm-boat.sh destroy ios --yes
exit=255
provider calls: 
remote calls: 
{"state":"archived","fail":[]}
lifecycle=suspended
ever_ready=1
Evidence: Provider-side proof revalidation

$ fm-boat.sh provision ios --identity ~/.no-mistakes/worktrees/dd71c22cc6d7/01M45X42MBT8DXPE5S08335KCW/.validation-tmp/fm-boat-routing.PPOZ4s/world/identity --alias fm-sm-ios-runpod --model openai-codex/fixture
provisioned: ios
exit=0
provider calls: boat new,curl --config,curl --config,boat info,boat stop,boat info,
remote calls: 
{"state":"archived","fail":[]}
lifecycle=provisioned
ever_ready=0

$ fm-boat.sh wake ios
ready: ios
exit=0
provider calls: boat resume,curl --config,curl --config,ssh -o,ssh -o,
remote calls: 
{"state":"ready","fail":[]}
lifecycle=ready
ever_ready=1

$ fm-boat.sh sleep ios
reconciled=0
retired: remote-reply-ios
suspended: ios
exit=0
provider calls: boat info,boat stop,boat info,
remote calls: fm-sm-ios-runpod fm-remote-secondmate-control.sh sleep-reconcile,fm-sm-ios-runpod fm-remote-secondmate-control.sh children,
{"state":"archived","fail":[]}
lifecycle=suspended
ever_ready=1
sleep_quiesced=1

$ fm-boat.sh wake ios
ready: ios
exit=0
provider calls: boat resume,curl --config,boat exec,curl --config,ssh -o,
remote calls: 
{"state":"ready","fail":[]}
lifecycle=ready
ever_ready=1

$ FM_BOAT_DESTROY_DORMANT=1 fm-boat.py destroy ios --yes (placement is now running)
error: placement changed after dormant destroy checks; retry destroy
exit=1
{"state":"ready"}

$ FM_BOAT_SLEEP_QUIESCED=0 fm-boat.py sleep ios
suspended: ios

$ FM_BOAT_DESTROY_DORMANT=1 fm-boat.py destroy ios --yes (dormant without proof)
error: placement changed after dormant destroy checks; retry destroy
exit=1
{"state":"archived"}
provider calls: 

$ fm-boat.sh wake ios
ready: ios
exit=0
provider calls: boat resume,curl --config,boat exec,curl --config,ssh -o,
remote calls: 
{"state":"ready","fail":[]}
lifecycle=ready
ever_ready=1

$ fm-boat.sh destroy ios --yes
reconciled=0
retired: remote-reply-ios
deleted: ios
exit=0
provider calls: boat info,boat stop,boat info,boat delete,
remote calls: fm-sm-ios-runpod fm-remote-secondmate-control.sh sleep-reconcile,fm-sm-ios-runpod fm-remote-secondmate-control.sh children,
{"state":"deleted","fail":[]}
placement record absent

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 7 of 7 scenarios driven live against the product
Scenario Result Live Evidence
Destroy a checked suspended placement with unreachable SSH: provider deletes it and the placement record disappears ✅ pass live CLI behavior and provider-state transcript; Parent commit reproduces SSH 255
Destroy a running placement with unreachable SSH or active remote children: deletion is refused ✅ pass live CLI behavior and provider-state transcript
Attempt sleep with active remote children: sleep refuses and records no quiescence proof ✅ pass live CLI behavior and provider-state transcript
Fail wake after a checked sleep, then destroy: compensation clears proof and destroy refuses before SSH or provider deletion ✅ pass live CLI behavior and provider-state transcript
Destroy a proven dormant placement with pending delivery, an unresolved decision, or missing --yes: deletion remains refused ✅ pass live CLI behavior and provider-state transcript
Destroy a provisioned placement: unproven dormancy refuses; checked sleep enables deletion without SSH ✅ pass live CLI behavior and provider-state transcript
Request dormant deletion after wake changes the placement or unchecked sleep removes proof: the provider-side guard refuses ✅ pass live Provider-side proof revalidation
  • TMPDIR="$PWD/.validation-tmp" bash tests/fm-boat-routing.test.sh
  • TMPDIR="$PWD/.validation-tmp" uv run --no-project tests/boat-lifecycle-cases.py "$PWD" LifecycleTests.test_dormant_destroy_requires_sleep_time_quiescence_proof LifecycleTests.test_wake_invalidates_sleep_time_quiescence_proof LifecycleTests.test_sleep_guards_refuse_unresolved_reply_and_decision -v
  • TMPDIR="$PWD/.validation-tmp" bash .validation-tmp/driver.sh — real CLI scenarios with isolated records and fake provider/SSH
  • TMPDIR="$PWD/.validation-tmp" bash .validation-tmp/lock-driver.sh — executable provider-side guard checks
  • TMPDIR="$PWD/.validation-tmp" CODE_ROOT="$PWD/.validation-tmp/parent" BASELINE=1 bash .validation-tmp/driver.sh — parent-commit regression replay
  • Removed disposable fixtures, parent copy, drivers, and generated Python cache; verified clean git status.
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

dnth added 2 commits October 5, 2026 19:28
Destroy on a suspended remote placement ran sleep-reconcile and children
through fm-on.sh, which needs a reachable host; a stopped sandbox returns
SSH 255, so sleep then destroy required a wake first.

Sleep now records sleep_quiesced=1 when its remote checks passed, and wake
clears it, so a failed-wake compensation that ends suspended carries no
proof. Destroy skips the remote checks only for a suspended or provisioned
record with that proof, refuses an unproven dormant record before any
remote call, and fm-boat.py re-checks the proof under its lock. Running
placements keep the remote checks; children, unlanded work, decisions and
--yes guards are unchanged.
@dnth
dnth merged commit 095b651 into main Oct 5, 2026
17 checks passed
@dnth
dnth deleted the fm/fm-boat-destroy-suspended branch October 5, 2026 13:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant