Skip to content

feat: keep local snapshots on a seat-release delete - #61

Merged
CMGS merged 6 commits into
mainfrom
feat/keep-snapshot-on-seat-release
Jul 31, 2026
Merged

feat: keep local snapshots on a seat-release delete#61
CMGS merged 6 commits into
mainfrom
feat/keep-snapshot-on-seat-release

Conversation

@tonicmuroq

@tonicmuroq tonicmuroq commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Depends on cocoonstack/cocoon-common#14 — merged; pinned here as v0.2.9-0.20260731042413-9ca4f1c8fc0f (cocoon-common main HEAD).

Why

DeletePod GC'd the local snapshot on every delete — including the one the operator issues to free a scheduling seat under hibernatePolicy: release. That threw away the warm-wake cache: a wake landing back on the same node (the operator soft-prefers it at weight 100 via hibernated-on-node) had to cold-pull from the registry state that was still sitting on local disk.

Note the wake path was already local-first — resolveWakeSource prefers a verified local snapshot and only pulls on a miss, and cleanupWakeImport explicitly keeps the local copy "live for the next wake". retain is fast today precisely because its pod is never deleted, so its snapshot is never GC'd. This closes the gap for release, which frees the seat.

What

  • Honor vm.cocoonstack.io/keep-snapshot-on-delete and skip removeLocalSnapshots in both branches: the forgotten-VM path a completed hibernate leaves behind, and the live-VM path an interrupted hibernate can leave (the VM is still removed there — only the snapshot survives).
  • The seat-release delete reports reason=seat_release instead of no_vm, so the cache-keep rate is countable at the source.
  • Invalidate fork-<vm> on every fresh (non-restore) bring-up, generalizing the run-mode-only invalidation. The kept fork can outlive a real lineage break — a set deleted while released never gets another DeletePod, and a same-name recreate would hand the dead incarnation's fork to new sub-agents via ensureForkSnapshot's create-once reuse. A hibernate restore continues the lineage and keeps the fork; the operator creates sub-agents only after the main is Ready, so pre-boot invalidation has no race window. Pair it with snapshot_verify_total{result=ok} and snapshot_pull_total to see the pulls actually avoided.

Safety

The delete.go comment this replaces justified the GC as "so a later restore cannot prefer stale local state over the registry tag" — but verifyLocalSnapshot already defends that more strongly: it compares the local snapshot's ID against the SnapshotID in the :hibernate tag's config blob, judges a missing tag stale, and fails closed when the registry is unreachable. The tag is the sole authority; the local copy is strictly a cache keyed by it. A missing flag only costs a pull, never correctness.

Follow-up (not in this PR)

Nothing reclaims a kept snapshot — the orphan policy covers VMs, not snapshots. This needs node-level cocoon gc --snapshot --snapshot-size/-age to bound the cache, and vk's storage allocatable (read once at startup) to stop hiding it from the scheduler, or it trades silent disk oversubscription for memory oversubscription.

Test

Two new cases in delete_test.go (forgotten-VM and live-VM seat release). go build ./..., go test ./..., make lint (linux + darwin) all green.

DeletePod GC'd the local snapshot on every delete, including the one the
operator issues to free a scheduling seat under hibernatePolicy=release.
That threw away the warm-wake cache: a wake landing back on the same node
(the operator soft-prefers it at weight 100) had to cold-pull state that
was still sitting on local disk.

Honor the operator's keep-snapshot-on-delete flag and skip the GC in both
branches — the forgotten-VM path a hibernate leaves behind, and the
live-VM path an interrupted hibernate can leave. The seat-release delete
now reports reason=seat_release so the cache-keep rate is countable at
the source.

Safety is unchanged: resolveWakeSource verifies any local copy against
the SnapshotID in the :hibernate tag's config blob, so a stale copy is
discarded rather than restored. A missing flag only costs a pull.
CMGS added 5 commits July 31, 2026 12:33
The seat-release delete now keeps fork-<vm>, so it can outlive a real
lineage break: a set deleted while released never gets another DeletePod
(the pod is already gone), and a same-name recreate on the same node
would hand the dead incarnation's fork to new sub-agents via
ensureForkSnapshot's create-once reuse. The operator already GCs the
:hibernate tag at teardown so recreates start clean; this restores the
node-local half of that guarantee.

Hoist the run-mode fork invalidation to cover every fresh (non-restore)
bring-up arm, before boot. A hibernate restore continues the lineage and
keeps the fork; the operator creates sub-agents only after the main is
Ready, so pre-boot invalidation has no race window.
@CMGS
CMGS merged commit 1d4ae56 into main Jul 31, 2026
2 checks passed
@CMGS
CMGS deleted the feat/keep-snapshot-on-seat-release branch July 31, 2026 06:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants