Skip to content

Startup convergence: re-invoke the reclaim verb, and boot before a resumed hibernate - #59

Merged
CMGS merged 4 commits into
mainfrom
fix/watchbusycreate-reinvoke-verb
Jul 29, 2026
Merged

Startup convergence: re-invoke the reclaim verb, and boot before a resumed hibernate#59
CMGS merged 4 commits into
mainfrom
fix/watchbusycreate-reinvoke-verb

Conversation

@tonicmuroq

@tonicmuroq tonicmuroq commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Fault-injection follow-up to #55/#56/#57, both reproduced on a real node.

watchBusyCreate only called Runtime.Inspect. A creating record reads the same
whether its owner is still cloning or died holding the name, so a clone whose
owner was SIGKILLed was never reclaimed: the loop polled until the 30-minute
budget expired while every CreatePod retry failed on reserve VM record: vm name ... already exists. The verb is the only thing that tells those apart, so
the loop now re-invokes it each tick, short-circuiting on collected/not-found.
It stays additive: a record that reaches running is still adopted whatever the
verb answers, so a verb that keeps failing cannot strand a committed clone.
The Inspect classification is now shared with reconcileStaleCreates.

dispatchResume gated its Start on the VM state carried in the startup List
snapshot. That is a stored record field, and it does not reliably describe
liveness -- runningVMClientWithRecord's pid %d not cloud-hypervisor branch is
only reachable when the record still reads running, so a hibernate owed on a VM
whose VMM had been killed could skip the boot and then fail every step with
vm is not running. Start is idempotent (PrepareStart no-ops on a live VM and
converges a crashed one via convergeCrashedStart), so it is now unconditional.

Verified on a testbed node:

  • kill -KILL vk with cocoon vm clone in flight, RestartSec shortened so vk is
    back while the orphaned child still holds the name: stale_create_reconcile
    goes busy=2 -> collected=1 and the record resolves in 6s, where the old build
    logged busy once and then went quiet for the full budget.
  • stop vk mid-hibernate then SIGKILL the VMM: startup_resume_total{hibernate}=1,
    the VM boots, and the hibernate completes (netresize/snapshot/push/remove all
    ok). A marker file written into the guest before each hibernate was present
    after every wake, so no guest state was lost on either path.

Closes part of #58.

…sumed hibernate

Fault-injection follow-up to #55/#56/#57, both reproduced on a real node.

watchBusyCreate only called Runtime.Inspect. A creating record reads the same
whether its owner is still cloning or died holding the name, so a clone whose
owner was SIGKILLed was never reclaimed: the loop polled until the 30-minute
budget expired while every CreatePod retry failed on `reserve VM record: vm
name ... already exists`. The verb is the only thing that tells those apart, so
the loop now re-invokes it each tick, short-circuiting on collected/not-found.
It stays additive: a record that reaches running is still adopted whatever the
verb answers, so a verb that keeps failing cannot strand a committed clone.
The Inspect classification is now shared with reconcileStaleCreates.

dispatchResume gated its Start on the VM state carried in the startup List
snapshot. That is a stored record field, and it does not reliably describe
liveness -- runningVMClientWithRecord's `pid %d not cloud-hypervisor` branch is
only reachable when the record still reads running, so a hibernate owed on a VM
whose VMM had been killed could skip the boot and then fail every step with
`vm is not running`. Start is idempotent (PrepareStart no-ops on a live VM and
converges a crashed one via convergeCrashedStart), so it is now unconditional.

Verified on internal-cocoon-node-7:

- kill -KILL vk with `cocoon vm clone` in flight, RestartSec shortened so vk is
  back while the orphaned child still holds the name: stale_create_reconcile
  goes busy=2 -> collected=1 and the record resolves in 6s, where the old build
  logged busy once and then went quiet for the full budget.
- stop vk mid-hibernate then SIGKILL the VMM: startup_resume_total{hibernate}=1,
  the VM boots, and the hibernate completes (netresize/snapshot/push/remove all
  ok). A marker file written into the guest before each hibernate was present
  after every wake, so no guest state was lost on either path.
@CMGS
CMGS merged commit 4d45cc2 into main Jul 29, 2026
2 checks passed
@CMGS
CMGS deleted the fix/watchbusycreate-reinvoke-verb branch July 29, 2026 10:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants