Skip to content

fix(envd): guard MMDS-poll goroutine with CAS to prevent unbounded accumulation on /init retries - #3560

Open
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/envd-mmds-goroutine-accumulation
Open

fix(envd): guard MMDS-poll goroutine with CAS to prevent unbounded accumulation on /init retries#3560
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/envd-mmds-goroutine-accumulation

Conversation

@AdaAibaby

Copy link
Copy Markdown
Contributor

Problem

Closes #3559

PostInit unconditionally spawns a new PollForMMDSOpts goroutine on every call:

go func() { //nolint:contextcheck // TODO: fix this later
    ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
    defer cancel()
    host.PollForMMDSOpts(ctx, a.mmdsChan, a.defaults.EnvVars)
}()

The orchestrator retries /init with infinite backoff until the sandbox responds. During normal Firecracker boot MMDS fills asynchronously, so several retries arrive before MMDS is ready. Each goroutine:

  • lives for up to 60 seconds
  • calls MMDS every 50 ms for both token and opts = up to 1 200 HTTP requests

With N concurrent retries this produces O(N) goroutine stacks and O(N × 1200) MMDS connections accumulating silently inside the VM until they time out.

Fix

Add mmdsPollRunning atomic.Bool to API (alongside the existing isMountingNFS and initialized atomics) and gate the goroutine spawn with CompareAndSwap:

if a.mmdsPollRunning.CompareAndSwap(false, true) {
    go func() { //nolint:contextcheck
        ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
        defer cancel()
        host.PollForMMDSOpts(ctx, a.mmdsChan, a.defaults.EnvVars)
        a.mmdsPollRunning.Store(false)  // reset so next /init can re-poll if needed
    }()
}

Behaviour preserved:

  • First /init — CAS succeeds, goroutine starts, MMDS opts delivered normally.
  • Subsequent retries while goroutine is running — CAS fails, no new goroutine; the already-running goroutine delivers the opts.
  • If goroutine times out before MMDS is ready — flag resets to false, the next /init retry starts a fresh poll.

Changes

File Change
packages/envd/internal/api/store.go Add mmdsPollRunning atomic.Bool field with explaining comment
packages/envd/internal/api/init.go Wrap goroutine spawn in CompareAndSwap; reset flag on exit

Testing

The fix is a pure atomicity change on an existing atomic.Bool pattern already used for isMountingNFS in the same struct. No new dependencies.

…cumulation

PostInit unconditionally spawned a new PollForMMDSOpts goroutine on
every call. The orchestrator calls /init with infinite retries until
the sandbox responds, so during normal Firecracker boot (MMDS fills
asynchronously) many retries land before MMDS is ready. Each goroutine
lives 60 s and fires up to 1 200 HTTP requests, producing O(N) goroutine
stacks and O(N*1200) MMDS connections in the VM for N retries.

Fix: add mmdsPollRunning atomic.Bool to API and use CompareAndSwap so
at most one polling goroutine is in flight at any time. The flag resets
to false when the goroutine exits (success or timeout), allowing a
fresh poll goroutine to start on the next /init if MMDS was not yet
ready the first time.

Fixes: e2b-dev#3559

@leonmeijer leonmeijer left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

packages/envd/internal/api/init.go changes retry and MMDS polling behavior but leaves packages/envd/pkg/version.go at 0.6.13. The repository requires every envd behavior change to bump that version; otherwise X-Envd-Version and rollout compatibility logic cannot distinguish binaries with and without this fix. Please bump the envd version in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(envd): PostInit spawns unbounded MMDS-poll goroutines — one per /init retry

3 participants