Efficiency follow-ups: shared scrape sample, parallel startup probes and status reconcile - #57
Merged
Conversation
…s reconcile Follow-ups from the whole-repo efficiency audit. The three scrape consumers (stats summary, metrics resource, Prometheus collector) each paid their own /proc+statfs sweep per VM; they now share one 2s-TTL sample, and CPU+RSS come from a single /proc/<pid>/stat read (RSS field 24 replaces the separate status VmRSS read — same resident-set definition). StartupReconcile's adopted-pod probe starts move to a bounded fan-out: each first probe is synchronous with a 3s worst case and the loop gates node registration. reconcilePodStatuses gets the same bounded fan-out; the three reconcile paths share reconcileFanOut.
cocoon-common's typed manifest 404 (cocoonstack/cocoon-common#13) lands via the pseudo-version bump, so fetchHibernateManifest drops its HasManifest HEAD: absence comes back as ErrManifestNotFound, every other error keeps failing closed. One registry round trip per wake and per create-time evidence check instead of two.
fanOut collapses the two policy-identical bounded fan-outs (probe starts inline at the call site, status reconcile's closure drops its error ceremony); the shared constant splits by lifecycle into startupFanOut and statusReconcileFanOut so steady-state apiserver concurrency tunes independently of the boot gate; buildNetworkStats drops the disjunct the upstream sample already implies; narration trimmed from the sample/parse/evidence comments. Skipped as adjudicated: Manager.StartBatch (single-consumer API), a lazily-split sample cache (codex round-1 rebuttal), a generic TTL-cache type (one instantiation).
CMGS
added a commit
that referenced
this pull request
Jul 29, 2026
…sumed hibernate (#59) * Startup convergence: re-invoke the reclaim verb, and boot before a resumed hibernate Fault-injection follow-up to #55/#56/#57, both reproduced on a real node. watchBusyCreate only called Runtime.Inspect. A creating record reads the same whether its owner is still cloning or died holding the name, so a clone whose owner was SIGKILLed was never reclaimed: the loop polled until the 30-minute budget expired while every CreatePod retry failed on `reserve VM record: vm name ... already exists`. The verb is the only thing that tells those apart, so the loop now re-invokes it each tick, short-circuiting on collected/not-found. It stays additive: a record that reaches running is still adopted whatever the verb answers, so a verb that keeps failing cannot strand a committed clone. The Inspect classification is now shared with reconcileStaleCreates. dispatchResume gated its Start on the VM state carried in the startup List snapshot. That is a stored record field, and it does not reliably describe liveness -- runningVMClientWithRecord's `pid %d not cloud-hypervisor` branch is only reachable when the record still reads running, so a hibernate owed on a VM whose VMM had been killed could skip the boot and then fail every step with `vm is not running`. Start is idempotent (PrepareStart no-ops on a live VM and converges a crashed one via convergeCrashedStart), so it is now unconditional. Verified on internal-cocoon-node-7: - kill -KILL vk with `cocoon vm clone` in flight, RestartSec shortened so vk is back while the orphaned child still holds the name: stale_create_reconcile goes busy=2 -> collected=1 and the record resolves in 6s, where the old build logged busy once and then went quiet for the full budget. - stop vk mid-hibernate then SIGKILL the VMM: startup_resume_total{hibernate}=1, the VM boots, and the hibernate completes (netresize/snapshot/push/remove all ok). A marker file written into the guest before each hibernate was present after every wake, so no guest state was lost on either path. * review: tighten comments, share stale-create metric recording, drop dead test knobs * docs: describe stale-create watcher retries * test: deliver mid-wait hibernate as a fresh tracked snapshot, not an in-place mutation --------- Co-authored-by: CMGS <ilskdw@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Applies all four follow-ups from the whole-repo efficiency audit that closed out #56. Items 1, 2, and 4 are implemented directly here; item 3 is included through the pinned cocoon-common change from cocoonstack/cocoon-common#13.
Shared scrape sample (item 2)
The three consumers previously sampled overlapping data independently:
GetStatsSummaryread CPU, RSS, and network counters;GetMetricsResourceread CPU and RSS; and the Prometheus collector also read network, COW, and node-storage data. They now share one complete sample behind a two-second TTL (sampleStats), so overlapping scrapes reuse the same/procand filesystem reads. A scrape that starts a fresh sample collects the complete set, and reported values may trail the live sources by up to two seconds.Per-VM CPU and RSS now come from one
/proc/<pid>/statread. RSS field 24 replaces the separate/proc/<pid>/statusVmRSSread.parseProcStatis extracted and tested with spaces and parentheses incomm.Parallel startup probe starts (item 1)
Every adopted pod's first probe runs synchronously with a three-second timeout, and
StartupReconcilegates node registration. Adopted pods are now collected and passed throughfanOut(startupFanOut, ...), reducing worst-case probe startup latency from N x 3 seconds to ceil(N/8) x 3 seconds.Parallel status drift reconcile (item 4)
The 30-second
reconcilePodStatusespass previously performed its apiserverGetand status derivation serially. It now usesfanOut(statusReconcileFanOut, ...); workers continue to log their own failures. Startup probe and stale-create work sharestartupFanOut, while steady-state status reconciliation has its own concurrency constant.Single-request hibernate evidence (item 3)
fetchHibernateManifestnow uses oneGetManifestcall. The pinned cocoon-common implementation maps an authoritative manifest 404 tosnapshot.ErrManifestNotFound; all other registry errors still fail closed.Verification
go test -race ./...passes; new coverage includesTestParseProcStat, malformed stat input, andTestSampleStatsServesCachedWithinTTL.make lintreports zero issues on Linux and Darwin;aslis clean on both.