fix(orchestrator): close TOCTOU race in Cleanup.Add/AddPriority - #3558
Open
AdaAibaby wants to merge 1 commit into
Open
fix(orchestrator): close TOCTOU race in Cleanup.Add/AddPriority#3558AdaAibaby wants to merge 1 commit into
AdaAibaby wants to merge 1 commit into
Conversation
Cleanup.run() set hasRun before acquiring mu, while Add/AddPriority read hasRun before acquiring mu. This created a window where a goroutine could observe hasRun==false, lose the race to run(), and then append a cleanup function into an already-drained slice — silently discarding it. Fix: - Move hasRun.Store(true) inside mu in run(), so the flag is only set while the slice is still accessible under the lock. - Add a double-check of hasRun after acquiring mu in Add/AddPriority, so any function that loses the race is executed immediately rather than silently dropped. The optimistic fast-path Load() before the lock is kept for the common post-run case (avoids lock contention when the sandbox is fully torn down), but is no longer the only gate. Fixes e2b-dev#3557
AdaAibaby
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
August 13, 2026 08:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Closes #3557
Cleanup.AddandCleanup.AddPriorityhad a TOCTOU race againstCleanup.runthat silently discarded cleanup functions, causing permanent resource leaks on the host.The race window:
sync.OnceinRun()preventsrun()from firing a second time, sofis permanently lost — no log, no error, no signal.With 28 call sites in the sandbox startup path registering operations like closing overlay filesystems, releasing network slots, removing Firecracker/UFFD sockets, and stopping the Firecracker process, any one of these dropped in a concurrent teardown leaves a leaked resource until the orchestrator restarts.
Fix
Two changes to
cleanup.go:1. Move
hasRun.Store(true)inside the lock inrun()2. Add a double-check of
hasRunafter acquiringmuinAdd/AddPriorityThe optimistic fast-path
Load()before the lock is kept to avoid lock contention in the common post-run case (already-torn-down sandbox), but is no longer the only gate.Test plan
AddandRuncalls no longer drop functions (race detector:go test -race ./packages/orchestrator/pkg/sandbox/...)/cc @jakubno @dobrac @ValentaTomas @arkamar @tvi @tomassrnka Looking forward to your code review.