Reclaim creating-state stale VMs safely; document dense-node host tuning - #60
Merged
Conversation
… verb The startup sweep force-removed every unowned sbx- VM, but rm --force queues on the VM ops lock: against a clone still in flight from a previous life it waits, then deletes the VM the clone just produced. cocoon v0.5.8's vm reconcile-stale-create collects a creating record only when that lock is free, so the sweep now routes creating-state records through it — busy records are left to their owner (cocoon's GC collects them on its own cadence), and a verb error (older cocoon) falls back to the forced remove. The verb call is cancellation-immune like removeVM: a canceled ctx must not skip the busy check and fall through to the remove it guards.
Record the fleet recipe dense-node measurement rounds settled on: quiet kernel console (bridge-port printk holds rtnl), udev kept off sandbox taps, an explicit ~64 refill_concurrency for egress-heavy nodes (rtnl collapses rather than plateaus), multi-daemon sharding behind the mesh, and cocoon's post-v0.5.8 cgroup_cpus fence keeping reserved cores free for the control plane. Also note the startup sweep's stale-create path in Running.
Batch-end /code+/simplify+comment pass over the branch (whole-repo scans against the 2026-08-02 clean baseline). Applied: two misattached doc comments re-anchored to their declarations (sdk files_test fakeSandbox, silkdtest readChunk/Fake split); interface-label and restating comments deleted (peer Pull/pullFrom, pty_test); two edit-narration test docs compressed to their invariants (peer_test dedup, checkpoint_test redirect); AppendBulkRequest godoc drops the perf-benefit phrase; sysutil.rs module doc no longer overclaims that all unsafe lives there; retryRemoval adopts the package's switch-with-init form; the busy log line says queued-for-retry, which is what now happens; the two identical stale-create test tables share one row type.
Three approved cuts, net -14 prod: claimFollow extracts the claim->redirect->no_redirect-retry skeleton Client.New and Checkpoint.New each hand-rolled; doJSONPtr collapses the four doJSON-then-pointer tails (info, drain x2, pools); logSweepResult single-sources the benign-sweep switch-log block copy-pasted across archive/idle/reap — whose copies had already drifted (the reap-archive failure line lost the word sandbox its success line carries).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two robustness items consuming the cocoon v0.5.8 line:
Startup sweep:
vm reconcile-stale-createfor creating-state records. The reconcile sweep force-removed every unownedsbx-VM, butrm --forcequeues on the VM ops lock — against a clone still in flight from a previous daemon life it waits, then deletes the VM the clone just produced. Creating-state records now go through cocoon's stale-create verb (vm: add reconcile-stale-create verb over a shared ownerless-create reclaim cocoon#173), which collects only when the ops lock is free:collected/not-found→ gone, no forced remove;busy→ left to its in-flight owner and queued for the reap tick, which re-runs the verb until it converges (collected/not-found, or not-creating → normal removal); its tap stays out of the reconcile-time egress table sweep meanwhile;not-creating→ the record moved on under the lock; normal removal;The verb call is cancellation-immune like
removeVM: a SIGTERM landing mid-reconcile must not skip the busy check and fall through to the forced remove it guards (mutation-verified byTestRemoveStaleVMCanceledCtxStillChecksBusy).docs/deploy.md: dense-node host tuning + control-plane CPU fence. Records what the dense-node measurement rounds settled on: quiet kernel console (bridge-port printk runs synchronously to every registered console while holding rtnl — one attach 44 ms noisy vs 3 ms quiet), udev kept off sandbox taps (~4x fill), an explicit ~64
refill_concurrencyfor egress-heavy nodes (rtnl collapses rather than plateaus; the auto default overshoots), and cocoon's post-v0.5.8cgroup_cpuscpuset fence + per-VM cgroup scopes keeping reserved cores free for sandboxd, its cocoon invocations, and the OS.Hot-path cost
Zero. The verb runs only in the startup sweep, one call per creating-state stale VM (a crash-recovery-only condition), inside the existing
runBoundedbudget. The docs change is documentation.Evidence
make go-testgreen across all modules;make go-lint0 issues on both GOOS after a lint cache clean;asl ./...clean on both GOOS for the sandboxd and e2e modules.WithoutCancelis removed — verified by mutation), running-state orphans never touch the verb.git tag --contains), the cgroup work is post-v0.5.8 (untagged),cgroup_cpusis a real config key, and the autorefill_concurrencyformula yields 256 on 384 cores as stated.