Startup convergence: re-invoke the reclaim verb, and boot before a resumed hibernate - #59
Merged
Merged
Conversation
…sumed hibernate Fault-injection follow-up to #55/#56/#57, both reproduced on a real node. watchBusyCreate only called Runtime.Inspect. A creating record reads the same whether its owner is still cloning or died holding the name, so a clone whose owner was SIGKILLed was never reclaimed: the loop polled until the 30-minute budget expired while every CreatePod retry failed on `reserve VM record: vm name ... already exists`. The verb is the only thing that tells those apart, so the loop now re-invokes it each tick, short-circuiting on collected/not-found. It stays additive: a record that reaches running is still adopted whatever the verb answers, so a verb that keeps failing cannot strand a committed clone. The Inspect classification is now shared with reconcileStaleCreates. dispatchResume gated its Start on the VM state carried in the startup List snapshot. That is a stored record field, and it does not reliably describe liveness -- runningVMClientWithRecord's `pid %d not cloud-hypervisor` branch is only reachable when the record still reads running, so a hibernate owed on a VM whose VMM had been killed could skip the boot and then fail every step with `vm is not running`. Start is idempotent (PrepareStart no-ops on a live VM and converges a crashed one via convergeCrashedStart), so it is now unconditional. Verified on internal-cocoon-node-7: - kill -KILL vk with `cocoon vm clone` in flight, RestartSec shortened so vk is back while the orphaned child still holds the name: stale_create_reconcile goes busy=2 -> collected=1 and the record resolves in 6s, where the old build logged busy once and then went quiet for the full budget. - stop vk mid-hibernate then SIGKILL the VMM: startup_resume_total{hibernate}=1, the VM boots, and the hibernate completes (netresize/snapshot/push/remove all ok). A marker file written into the guest before each hibernate was present after every wake, so no guest state was lost on either path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fault-injection follow-up to #55/#56/#57, both reproduced on a real node.
watchBusyCreate only called Runtime.Inspect. A creating record reads the same
whether its owner is still cloning or died holding the name, so a clone whose
owner was SIGKILLed was never reclaimed: the loop polled until the 30-minute
budget expired while every CreatePod retry failed on
reserve VM record: vm name ... already exists. The verb is the only thing that tells those apart, sothe loop now re-invokes it each tick, short-circuiting on collected/not-found.
It stays additive: a record that reaches running is still adopted whatever the
verb answers, so a verb that keeps failing cannot strand a committed clone.
The Inspect classification is now shared with reconcileStaleCreates.
dispatchResume gated its Start on the VM state carried in the startup List
snapshot. That is a stored record field, and it does not reliably describe
liveness -- runningVMClientWithRecord's
pid %d not cloud-hypervisorbranch isonly reachable when the record still reads running, so a hibernate owed on a VM
whose VMM had been killed could skip the boot and then fail every step with
vm is not running. Start is idempotent (PrepareStart no-ops on a live VM andconverges a crashed one via convergeCrashedStart), so it is now unconditional.
Verified on a testbed node:
cocoon vm clonein flight, RestartSec shortened so vk isback while the orphaned child still holds the name: stale_create_reconcile
goes busy=2 -> collected=1 and the record resolves in 6s, where the old build
logged busy once and then went quiet for the full budget.
the VM boots, and the hibernate completes (netresize/snapshot/push/remove all
ok). A marker file written into the guest before each hibernate was present
after every wake, so no guest state was lost on either path.
Closes part of #58.