fix(envd): guard MMDS-poll goroutine with CAS to prevent unbounded accumulation on /init retries - #3560
Open
AdaAibaby wants to merge 1 commit into
Open
Conversation
…cumulation PostInit unconditionally spawned a new PollForMMDSOpts goroutine on every call. The orchestrator calls /init with infinite retries until the sandbox responds, so during normal Firecracker boot (MMDS fills asynchronously) many retries land before MMDS is ready. Each goroutine lives 60 s and fires up to 1 200 HTTP requests, producing O(N) goroutine stacks and O(N*1200) MMDS connections in the VM for N retries. Fix: add mmdsPollRunning atomic.Bool to API and use CompareAndSwap so at most one polling goroutine is in flight at any time. The flag resets to false when the goroutine exits (success or timeout), allowing a fresh poll goroutine to start on the next /init if MMDS was not yet ready the first time. Fixes: e2b-dev#3559
AdaAibaby
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
August 13, 2026 10:13
leonmeijer
reviewed
Aug 16, 2026
leonmeijer
left a comment
There was a problem hiding this comment.
packages/envd/internal/api/init.go changes retry and MMDS polling behavior but leaves packages/envd/pkg/version.go at 0.6.13. The repository requires every envd behavior change to bump that version; otherwise X-Envd-Version and rollout compatibility logic cannot distinguish binaries with and without this fix. Please bump the envd version in this PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Closes #3559
PostInitunconditionally spawns a newPollForMMDSOptsgoroutine on every call:The orchestrator retries
/initwith infinite backoff until the sandbox responds. During normal Firecracker boot MMDS fills asynchronously, so several retries arrive before MMDS is ready. Each goroutine:With N concurrent retries this produces O(N) goroutine stacks and O(N × 1200) MMDS connections accumulating silently inside the VM until they time out.
Fix
Add
mmdsPollRunning atomic.BooltoAPI(alongside the existingisMountingNFSandinitializedatomics) and gate the goroutine spawn withCompareAndSwap:Behaviour preserved:
/init— CAS succeeds, goroutine starts, MMDS opts delivered normally.false, the next/initretry starts a fresh poll.Changes
packages/envd/internal/api/store.gommdsPollRunning atomic.Boolfield with explaining commentpackages/envd/internal/api/init.goCompareAndSwap; reset flag on exitTesting
The fix is a pure atomicity change on an existing
atomic.Boolpattern already used forisMountingNFSin the same struct. No new dependencies.